ECG-Reasoning-Benchmark: A Benchmark for Evaluating Clinical Reasoning Capabilities in ECG Interpretation

ArXiv: 2603.14326

🎯 Pitch

Current MLLMs appear to interpret ECGs by memorizing shortcuts rather than performing actual clinical reasoning—when given the correct step-by-step reasoning chain, their diagnostic accuracy drops from ~85% to ~23%. The benchmark reveals that these models can recite textbook criteria but fail catastrophically at pinpointing where that evidence actually appears in the signal, with near-zero rates of completing a coherent reasoning chain.

---

1. Executive Summary

This paper introduces ECG-Reasoning-Benchmark, a multi-turn evaluation framework with over 6,400 samples across 17 core ECG diagnoses designed to assess whether Multimodal Large Language Models (MLLMs) perform genuine step-by-step clinical reasoning or merely rely on superficial pattern matching. The benchmark enforces a 4-stage verification loop — Criterion Selection (identifying which diagnostic criterion to evaluate, e.g., choosing "Prolonged QRS duration" over distractors for complete left bundle branch block), Finding Identification (detecting whether the criterion is present in the signal), ECG Grounding (pinpointing the exact leads, waveform segments, and measurement ranges that support the finding), and Diagnostic Decision (determining whether sufficient evidence exists) — to systematically probe the reasoning trajectory. Across state-of-the-art models including ECG-specific, medical-domain, and proprietary systems, the paper reveals that while models possess the medical knowledge to select appropriate criteria, they exhibit near-zero success rates in maintaining complete reasoning chains, with maximum Completion scores reaching only ~6% (Gemini-3-Flash on PTB-XL) and average Depth scores rarely exceeding 2.0, establishing that current MLLMs can retrieve textbook knowledge but critically fail to ground that knowledge to actual visual evidence in the ECG signal — a failure most starkly demonstrated by ECG-specific models whose diagnostic accuracy collapses from ~85% to ~23% when provided with the correct reasoning trajectory, exposing their reliance on superficial signal-to-label shortcuts rather than valid clinical deduction.

2. Context and Motivation

The Core Problem: We Don't Know If ECG AI Models Actually "Think" or Just Pattern-Match

The fundamental question this paper tackles is deceptively simple: when a multimodal AI model interprets an ECG and says "this shows left bundle branch block," is it actually examining the signal for prolonged QRS duration, notched R waves in lateral leads, and dominant S waves in V1/V2 — or is it just recognizing a global visual pattern and regurgitating the associated diagnosis? This distinction matters enormously because the two behaviors look identical in terms of final output accuracy but have radically different implications for safety, trustworthiness, and clinical deployment.

The paper argues that current evaluation paradigms cannot distinguish between these two modes of operation. A model can achieve high diagnostic accuracy by learning superficial correlations (e.g., "this particular lead configuration usually means LVH") without ever performing the step-by-step clinical deduction that a cardiologist would demand. In the high-stakes domain of healthcare, this is a dangerous blind spot: a model that relies on shortcuts rather than genuine reasoning will fail silently and unpredictably when those shortcuts break — for instance, when encountering a rare presentation, a comorbidity that alters the expected signal pattern, or an artifact that superficially resembles a pathological feature.

This gap is significant for several practical reasons the paper highlights (Section 1):

• Clinical adoption barrier: Despite deep learning models achieving diagnostic accuracy comparable to or surpassing human cardiologists in classification tasks (Pyakillya et al., 2017; Liu et al., 2021), clinical practice remains hesitant. The paper explicitly states that "in the high-stakes domain of healthcare, a 'black-box' prediction is insufficient. Clinicians require not just a diagnostic label, but the clinical reasoning and evidence that justify the conclusion to make informed decisions." This is not just about trust — it's about liability, second-opinion workflows, and the ability to catch errors before they cause harm.
• Hallucination risk in generative models: The recent pivot toward Multimodal Large Language Models (MLLMs) for ECG interpretation — PULSE, GEM, OpenTSLM, ECG-R1 — has introduced a new failure mode. These models generate fluent, medically plausible explanations, but the paper argues that this fluency masks a dangerous dependency on synthetic training data. Because the training rationales are often generated by an LLM (like GPT-4) given only the diagnostic label and machine-generated reports — without access to the actual ECG signal — the resulting models learn to produce "medically fluent justifications that recite textbook descriptions associated with the diagnosis, regardless of what the underlying signal actually shows" (Section 1).
• Evaluation methodology is part of the problem: The predominant evaluation framework — LLM-as-a-Judge (Zheng et al., 2023) — compares generated interpretations against reference responses that are themselves synthetically created by an LLM. The paper identifies the circularity: "measuring the alignment between a model's output and these references primarily assesses how well the model mimics the linguistic style of the text generator." The judge-LLM never examines the actual ECG image to verify whether the interpretation is grounded. This means a model optimized for high LLM-as-a-Judge scores is being optimized to produce text that sounds like valid clinical reasoning, not text that is valid clinical reasoning anchored to signal evidence.

The Deeper Structural Problem: Training Data Contamination

The paper identifies a specific mechanism by which current ECG-MLLMs become disconnected from the underlying signal. In Section 1, the authors explain that existing training datasets — ECGInstruct for PULSE, ECG-Grounding for GEM, ECG-Protocol-Guided-Grounding-CoT for ECG-R1, and ECG-QA-CoT for OpenTSLM — share a common flaw:

"the training explanations are synthetically generated by providing an LLM like GPT-4 with the final diagnostic labels and machine-generated reports, typically without direct exposure to the actual ECG signal."

This is a nuanced point that deserves careful unpacking. The pipeline works as follows: (1) take an ECG recording that has a human-annotated diagnostic label (e.g., "first-degree AV block"); (2) optionally run the ECG through a traditional signal processing tool to extract some measurements; (3) give the label and measurements to GPT-4 and ask it to generate a step-by-step reasoning chain; (4) train the MLLM on these text-generated rationales paired with the ECG images. The problem is that step (3) is structurally incapable of producing genuine visual grounding because GPT-4 never sees the ECG. The generated rationale might say "the PR interval is prolonged at 240 ms" based on a measurement fed into the prompt, but the rationale's language patterns, the way it structures its reasoning, and the confidence with which it asserts findings are all artifacts of the text generator's style, not a genuine response to visual evidence.

The downstream consequence: models trained on this data learn to associate global ECG patterns with the corresponding text-derived rationales. They become very good at recognizing that this kind of ECG typically comes with that kind of explanation, but they do not necessarily learn to verify individual findings. As the paper demonstrates in its results (Section 5.2), ECG-specific models like ECG-R1-RL achieve 85.41% Initial Diagnosis Accuracy on PTB-XL — suggesting they can often guess the right answer — but collapse to 22.70% GT-Reasoning-Based Diagnosis Accuracy when provided with the correct reasoning chain, indicating they cannot even recognize valid reasoning when it is presented to them.

Conflicting Signals: High Accuracy vs. Zero Reasoning

The paper is motivated by a genuine paradox in the literature. On one side, ECG-MLLMs report impressive performance: PULSE, GEM, and ECG-R1 all claim strong results on ECG interpretation tasks using LLM-as-a-Judge evaluation. On the other side, the authors' pilot observations (implicit in the benchmark design) suggested that these same models fail at elementary visual grounding tasks — identifying which lead shows a particular waveform, or measuring whether a QRS complex exceeds 120 ms.

These are not contradictory findings; they reflect different things being measured. The LLM-as-a-Judge framework measures whether the model's output is plausible — does it use the right medical terminology, follow the expected rhetorical structure, and arrive at a diagnosis consistent with the reference? The benchmark this paper introduces measures whether the model's reasoning is correct — does it identify the right criteria, detect their presence in the signal, and localize them to specific temporal segments and leads?

A model can score highly on plausibility while scoring near zero on correctness. This is the paper's central insight, and it explains why prior work reached optimistic conclusions while this paper's evaluation reveals catastrophic failures. The optimism was an artifact of an evaluation framework that prioritized linguistic fluency over evidentiary grounding.

Where Existing Approaches Fall Short

The paper identifies specific limitations in prior work along three axes:

1. Training data is structurally disconnected from visual evidence (Section 2). The paper surveys the main ECG-MLLM datasets:

• ECGInstruct (PULSE): Instruction-response pairs synthesized by Llama-3-70B-Instruct without direct exposure to actual signals. The model learns to generate instruction-following responses about ECGs, but the "reasoning" in those responses is text-derived, not visually derived.
• ECG-Grounding (GEM) and ECG-Protocol-Guided-Grounding-CoT (ECG-R1): These datasets attempted to incorporate physical measurements extracted from external tools (Hong et al., 2017, 2019) to provide structural grounding. However, the paper argues this is a partial fix: the measurements are fed into an LLM to generate rationales, so the language of the rationale is still synthetic and the model still learns to map global signals to text rather than to perform step-by-step verification. The paper notes this represents "a step forward" but "still fundamentally relies on synthetic data generation processes."
• ECG-QA-CoT (OpenTSLM): Chain-of-Thought trajectories generated by GPT-4o from question-answer pairs in the ECG-QA dataset (Oh et al., 2023). The same structural problem: GPT-4o never sees the ECG, so its chain-of-thought is speculative reconstruction from the QA pair.

A subtle but critical point the paper makes: even when these datasets incorporate real measurements, the MLLM trained on them is not necessarily learning to extract those measurements from the signal. Instead, it may learn to recognize the signal pattern that correlates with those measurements appearing in the training text. This is a form of Clever Hans effect — the model appears to measure QRS duration because it can identify ECGs where prolonged QRS is mentioned in the training rationales, not because it can actually measure duration from the waveform.

2. Evaluation methodology cannot detect grounding failures (Section 1). The LLM-as-a-Judge framework has several structural weaknesses:

• Circular reference generation: The "ground truth" reference explanations are generated by the same or similar LLMs that produce the model's output. High similarity scores indicate shared training distributions, not clinical accuracy.
• No signal-level verification: The judge-LLM evaluates text against text. It never checks whether the claimed findings (e.g., "ST elevation in leads II, III, and aVF") are actually present in the ECG image. A model could hallucinate entire findings and the evaluation would not catch it if the hallucination is medically plausible.
• Fluency confounds evaluation: LLM judges are known to be biased toward well-structured, confident-sounding responses regardless of factual accuracy. In the ECG domain, where the vocabulary and reasoning structures are highly formulaic, this bias is particularly dangerous — formulaic correctness looks like genuine expertise.

3. No framework exists for verifying step-by-step reasoning. Prior to this work, there was no publicly available benchmark that required models to demonstrate explicit, verifiable clinical reasoning on ECGs. Existing ECG QA datasets (ECG-QA) test factual knowledge and basic finding recognition but do not enforce multi-step logical deduction where each step must be grounded in specific signal features. The paper positions ECG-Reasoning-Benchmark as filling this gap: a test of clinical reasoning, not clinical knowledge.

How This Paper Positions Itself

The paper positions its contribution as a paradigm shift in evaluation rather than a new model or training method. The key framing (Section 1):

"We posit that evaluating an ECG-MLLM should not be a test of fluency, but a rigorous 'clinical reasoning exam' that probes the model's intrinsic reasoning capabilities."

This positioning has several implications:

• The benchmark is the contribution. The paper is not proposing a better MLLM architecture or training recipe. It is proposing a better way to measure whether any architecture or training recipe actually produces clinical reasoning, as opposed to pattern-matching fluency. This makes the work foundational: it establishes a standard against which future models can be judged, and it reveals failures that must be addressed before ECG-MLLMs can be considered clinically reliable.
• The automated analysis pipeline is a necessary prerequisite. To create a ground truth that supports step-by-step verification, the paper had to build a system that extracts explicit diagnostic features from raw 12-lead signals — wave boundaries, interval measurements, morphological classifications — and maps them to discrete clinical findings via expert-validated logic diagrams. This pipeline (Section 3) is itself a significant engineering contribution, enabling objective verification without requiring expensive human annotation of every reasoning step.
• The evaluation design intentionally exposes the gap between knowledge and grounding. The 4-stage verification loop is not arbitrary; it is designed to isolate specific failure points. Criterion Selection tests medical knowledge (does the model know what to look for?). Finding Identification tests perceptual ability (does the model see what it claims to see?). ECG Grounding tests spatial and temporal localization (can the model point to exactly where in the signal it sees the finding?). Diagnostic Decision tests logical integration (can the model combine findings into a coherent diagnosis?). The paper's finding that models succeed at Stage 1 but fail catastrophically at Stages 2–3 precisely quantifies the nature of the reasoning gap.
• The GT-RDA metric is a clever diagnostic tool. By comparing Initial Diagnosis Accuracy (IDA) with GT-Reasoning-Based Diagnosis Accuracy (GT-RDA) — accuracy when the model is given the correct reasoning chain and only asked to make the final diagnostic decision — the paper distinguishes between models that fail at diagnosis versus models that fail at recognizing valid evidence. The collapse of ECG-specific models on GT-RDA (from ~85% to ~23%) is the paper's strongest evidence that these models never learned genuine clinical reasoning in the first place: they cannot even comprehend the correct reasoning when it is handed to them, presumably because their training optimized for a direct signal-to-diagnosis shortcut that bypasses intermediate reasoning entirely.

The paper explicitly connects its methodology to CXReasonBench (Lee et al., 2025), which introduced a similar structured reasoning evaluation for chest X-rays, but notes that ECG-Reasoning-Benchmark extends this approach to the more technically demanding domain of time-series signal interpretation with precise measurement grounding requirements. The ECG domain requires not just visual localization (which lead? which segment?) but also quantitative verification (which measurement range?), adding a layer of rigor absent from image-only benchmarks.

Why This Matters Beyond ECGs

While the paper focuses specifically on ECG interpretation, the problem it identifies generalizes to any domain where MLLMs are being trained on synthetic rationales and evaluated on text similarity. The core dynamic — training models on LLM-generated explanations, then evaluating them with LLM judges, creating a closed loop that optimizes for linguistic style rather than factual grounding — is not unique to ECGs. It appears in radiology, pathology, legal reasoning, and any field where domain experts produce structured reasoning that is expensive to annotate but easy to simulate with a large language model.

The paper's contribution is demonstrating, with rigorous quantitative evidence, just how completely this closed loop can fail. The near-zero Completion rates and the GT-RDA collapse are not incremental findings — they represent a fundamental indictment of current training and evaluation paradigms. This positions ECG-Reasoning-Benchmark as a cautionary tale and a methodological template: before claiming that an AI model can "reason" about a domain, we must demand evidence that its reasoning is grounded in the actual data, not in the linguistic patterns of its training text.

3. Technical Approach

3.1 Reader Orientation

This paper builds a benchmark and automated ground-truth pipeline — not a new model — that functions as a clinical reasoning exam for ECG-interpreting AI systems. The problem it solves is that current evaluation methods cannot distinguish between an MLLM that performs genuine step-by-step clinical deduction (identifying diagnostic criteria, detecting them in the signal, and grounding them to specific waveform features) and one that simply pattern-matches the global ECG appearance to a memorized diagnosis. The solution takes the shape of a multi-turn, progressively deepening verification loop that forces the model to demonstrate explicit, verifiable reasoning at each stage, backed by an automated analysis pipeline that extracts objective ground truth directly from raw 12-lead signals using waveform delineation, quantitative measurement, and expert-validated diagnostic logic diagrams.

3.2 Big-Picture Architecture

The system has five major components that operate in two phases — an offline ground-truth construction phase and an online evaluation phase:

1. Automated ECG Analysis Pipeline (offline) — a signal processing and rule-based system that takes raw 12-lead ECG recordings and produces structured ground-truth annotations: waveform boundaries, quantitative measurements, discrete clinical findings, and final diagnoses. This is the "answer key" that makes objective evaluation possible without expensive human annotation.

2. Diagnostic Logic Diagrams (offline) — 17 hand-crafted, hierarchical decision trees, one per core diagnosis (e.g., Complete Left Bundle Branch Block, First Degree AV Block), codified from authoritative cardiology textbooks and validated by three board-certified internal medicine specialists. These diagrams specify exactly which clinical findings must be verified, in what order, and with what grounding evidence (which leads, which waveform segments, which measurement ranges), for a diagnosis to be confirmed.

3. Sampling and Filtering Module (offline) — selects balanced sets of 100 positive and 100 negative ECG samples per diagnosis from PTB-XL and MIMIC-IV-ECG, ensuring even coverage across all valid reasoning paths within each diagnosis and verifying alignment between human-provided labels and the automated pipeline's conclusions.

4. Multi-Turn Evaluation Engine (online) — conducts the actual conversation with the model under test. For each sample, it begins with an Initial Diagnostic Question, then iteratively executes the 4-stage verification loop (Criterion Selection, Finding Identification, ECG Grounding, Diagnostic Decision) for every clinical finding required by the relevant logic diagram, accumulating the model's responses for metric computation.

5. Metrics Computation Module (online) — calculates four quantitative scores (Initial Diagnosis Accuracy, Completion, Depth, GT-Reasoning-Based Diagnosis Accuracy) from the model's multi-turn response history, using a separate LLM (Gemini-3-Flash) as a semantic consistency checker rather than a quality judge.

Information flows as follows: raw ECG signals enter the Automated Analysis Pipeline → wave boundaries, measurements, and findings are extracted → the Diagnostic Logic Diagrams are applied to determine which findings are present and which diagnosis is confirmed → the Sampling Module selects balanced, verified cases → for each selected case, the Evaluation Engine presents the ECG image (or time-series) to the model and conducts the multi-turn verification conversation → the model's responses at each turn are recorded → the Metrics Module scores the complete response history, computing Completion, Depth, IDA, and GT-RDA.

3.3 Roadmap for the Deep Dive

• First, the design philosophy and formal structure of the 4-stage verification loop, because every metric and finding in the paper originates from this loop's architecture.
• Second, the Automated ECG Analysis Pipeline — wave detection, feature extraction, finding mapping, and diagnosis derivation — because this pipeline generates the ground truth against which all model responses are evaluated.
• Third, the Diagnostic Logic Diagrams, because they define the exact reasoning paths, required findings, and grounding sub-tasks for each of the 17 diagnoses, and understanding them is essential to understanding what Depth and Completion actually measure.
• Fourth, the sampling strategy and dataset construction, because the benchmark's claim to be "unbiased" and "rigorous" rests on how cases were selected, balanced, and verified.
• Fifth, the evaluation protocol and metrics — the multi-turn conversation structure, the semantic consistency checking, and the four quantitative scores — because these operationalize the paper's central claim that reasoning (not fluency) is being measured.
• Sixth, the model configurations and prompting strategy, because the paper's experimental conclusions depend on how models were instructed and configured.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a benchmark and evaluation methodology paper whose core idea is that verifying step-by-step clinical reasoning requires a structured, multi-turn protocol that independently tests three capabilities — medical knowledge retrieval, perceptual detection, and visual grounding — and that current MLLMs catastrophically fail at the latter two despite appearing competent at the first.

---

Design Philosophy of the 4-Stage Verification Loop

The paper conceptualizes ECG interpretation as a multi-stage deduction process with three distinct cognitive operations: (1) knowledge retrieval — knowing which diagnostic criteria are relevant to a suspected condition, (2) perceptual detection — visually identifying whether those criteria are present in the specific ECG under examination, and (3) visual grounding — precisely localizing the evidence in space (which leads) and time (which waveform segments) while quantifying key measurements that determine whether a threshold is exceeded.

The 4-stage verification loop is explicitly designed to isolate and test each of these operations independently. A model that possesses medical knowledge but fails at perceptual detection will succeed at Stage 1 (Criterion Selection) but fail at Stage 2 (Finding Identification). A model that can detect findings but cannot localize them will succeed at Stages 1–2 but fail at Stage 3 (ECG Grounding). A model that can do everything but prematurely jumps to conclusions will succeed at all stages but fail at Stage 4 (Diagnostic Decision) by not recognizing that further findings are required. This decomposition is what enables the paper to claim not just that models fail, but where in the reasoning chain they fail.

The loop is applied iteratively for every clinical finding required by the relevant diagnostic logic diagram. For a diagnosis like Complete Left Bundle Branch Block (Figure 9 in Appendix A.2), which requires verification of four sequential findings — Prolonged QRS duration, Dominant S waves in V1/V2, Positive monophasic QRS without Q waves in lateral leads, and Notched R waves in lateral leads — the model must navigate the full 4-stage loop four times in succession. Each iteration builds on the previous one: the Diagnostic Decision step of loops 1–3 offers the option "further findings are required to confirm the diagnosis," and only the final loop asks for a definitive yes/no decision. This enforces a disciplined diagnostic process where conclusions are withheld until all evidence is gathered — the opposite of the shortcut-based pattern matching the paper suspects current models employ.

A crucial design choice: the evaluation advances to the step-wise verification process regardless of whether the model answers the Initial Diagnostic Question correctly. This means the benchmark measures reasoning capacity independently from diagnostic accuracy. A model that guesses the right diagnosis by chance but fails every reasoning step will have a high IDA but near-zero Completion and Depth. Conversely, a model that misses the initial diagnosis but correctly navigates the reasoning chain would — in principle — demonstrate genuine reasoning ability despite the initial error. This separation ensures that the reasoning metrics are not confounded by the model's baseline diagnostic accuracy.

Distractor design in Criterion Selection. The paper employs two specific types of distractors in the multiple-choice options for Criterion Selection (Section 4.1):

• Category-based distractors: incorrect options drawn from the same clinical category as the correct finding. For example, if the correct criterion is "Prolonged PR interval," a category-based distractor would be "Normal PR interval" — both are about the PR interval, so the model cannot simply pattern-match on the clinical concept being discussed; it must know the specific threshold that defines the abnormality.
• Presence-based distractors: findings that are actually present in the current ECG recording but are clinically irrelevant to the diagnosis in question. For example, if the ECG shows both a prolonged QRS duration and ST elevation, but the diagnosis being evaluated is Complete Left Bundle Branch Block (which requires prolonged QRS), then "ST elevation" would be a presence-based distractor — something the model can see in the signal but that is not part of the diagnostic criteria for CLBBB. This tests whether the model can distinguish relevant findings from incidental ones.

This dual-distractor design is non-trivial because it prevents models from exploiting simple heuristics. A model that always picks the option mentioning a finding it detects in the signal would be fooled by presence-based distractors. A model that always picks the option that sounds "abnormal" would be fooled by category-based distractors where the abnormal option is incorrect. The model must possess both knowledge of diagnostic criteria and the ability to filter out irrelevant observations.

ECG Grounding as anti-hallucination mechanism. The third stage of the verification loop — ECG Grounding — is the paper's most important technical contribution to evaluation design. Unlike previous benchmarks that ask only whether a finding is present, ECG-Reasoning-Benchmark demands that the model specify exactly where in the signal it sees the evidence. This grounding requirement operates at three granularities:

• Lead Grounding: For findings associated with specific anatomical regions, the model must identify which of the 12 leads exhibit the abnormality. For example, "notched R waves" in CLBBB must be localized to the lateral leads (I, aVL, V5, V6), and the model must select the correct subset from a multi-select list.
• Wave Grounding: The model must temporally locate the relevant waveform within the 10-second recording. Options are presented as time ranges (e.g., "[1.45s – 1.61s]") corresponding to specific QRS complexes or cardiac cycles. This forces the model to demonstrate that it is attending to a particular region of the signal rather than making a global assertion.
• Measurement Grounding: For findings that depend on quantitative thresholds (e.g., QRS duration > 120 ms for prolonged QRS), the model must select the correct measurement range (e.g., "[150ms – 160ms]") from a set of discretized bins. This verifies not just that the model sees the abnormality, but that it correctly quantifies it — a level of precision that simple yes/no questions cannot assess.

The measurement grounding component is scored fractionally within the Depth metric: if a finding requires N grounding sub-tasks (typically 1–3), each correctly answered sub-task contributes 1/N to the depth score for that stage. This granular scoring allows the paper to distinguish between models that can localize findings but cannot measure them versus models that fail at both, providing a more nuanced picture than binary pass/fail.

---

The Automated ECG Analysis Pipeline

The foundation of the benchmark is the automated pipeline that constructs verifiable ground-truth annotations directly from raw 12-lead signals. Without this pipeline, the paper would need human experts to annotate every waveform boundary, measurement, and finding for thousands of ECGs — an economically infeasible prospect. The pipeline makes the benchmark scalable while maintaining objectivity, since every annotation is deterministically derived from the signal via well-defined algorithms and expert-validated rules.

Stage 1: Wave Detection and Segmentation. The pipeline begins with a U-Net3+ architecture (Joung et al., 2024) that performs pixel-level segmentation of the ECG signal. For a given 12-lead ECG, the model processes each lead independently, generating separate probability maps for four classes: P wave, QRS complex, T wave, and the isoelectric background (everything else). The output is a per-time-point classification indicating which waveform (if any) is present at each sample.

The raw U-Net3+ outputs are then refined through three context-aware post-processing algorithms designed to correct known failure modes of deep learning-based segmentation:

• P-wave recovery via template matching. The authors observed that deep-learning models often miss non-conducted P waves — P waves that occur without a subsequent QRS complex, as seen in high-degree AV blocks. To recover these missed detections, the pipeline performs a secondary search within RR intervals where no P waves were initially identified. It uses SciPy's peak detection algorithm guided by a "P wave template" — the average duration and amplitude of successfully detected P waves within the same lead. Candidate peaks are validated against two criteria: (1) physiological constraints requiring a minimum duration of 60 ms and amplitude exceeding 5% of the adjacent QRS amplitude (a noise threshold), and (2) morphological similarity to the template (matching the deflection pattern — positive, negative, or biphasic). This recovery step is clinically important because non-conducted P waves are a key finding in second-degree AV block, and missing them would break the diagnostic logic.

• Physiological constraint enforcement. Biological rules are applied to eliminate artifacts. For instance, each cardiac cycle must contain exactly one T-wave following a QRS complex. If multiple T-wave candidates exist within a single RR interval, the algorithm selects the most probable peak based on its timing relative to the QT interval. This prevents segmentation errors where noise or U-waves are mistaken for T-waves.

• Multi-lead consensus alignment. To account for lead-specific noise or poor signal quality, the pipeline requires that a wave be detected at a consistent temporal location in at least 4 of the 12 leads before it is validated. Once validated, global boundaries (onset and offset) are defined by the earliest onset and latest offset across all contributing leads, capturing the full duration of the waveform. This consensus mechanism exploits the redundancy of the 12-lead system — a genuine P wave should appear simultaneously (to within a small tolerance) across multiple leads, while noise artifacts are typically lead-specific.

The paper evaluates this detection module on the Lobachevsky University Electrocardiography Database (LUDB), a publicly available dataset with expert-annotated waveform boundaries. The results (Table 2 in Appendix A.1) report mean recall and precision across three random train/validation/test splits:

The detection tolerance follows the Association for the Advancement of Medical Instrumentation (AAMI) standard: a predicted onset or offset is considered correct if it falls within 150 ms of the expert annotation. The near-perfect QRS detection (1.000 recall and precision) is expected because QRS complexes are the highest-amplitude, most visually distinct feature in the ECG. The P-wave performance shows a notable gap between recall (0.978) and precision (0.934), which the paper explains as an artifact of the LUDB annotation policy: LUDB systematically omits annotations for non-conducted P waves and P waves during AV dissociation. Since the pipeline is explicitly designed to detect these unannotated waves, its correct detections are counted as false positives, artificially lowering precision. Figure 3 in Appendix A.1 provides visual evidence: circled regions show clearly visible P waves with no corresponding ground-truth annotation, which the raw U-Net3+ misses (adhering to the annotation pattern) but the post-processed pipeline recovers.

Stage 2: Feature Extraction and Quantification. With waveforms precisely delineated, the pipeline extracts quantitative measurements that form the basis of clinical decision-making. These measurements are computed on a per-beat basis and then aggregated:

• Temporal measurements: Duration of P waves, QRS complexes, and T waves (in milliseconds), plus physiological intervals — PR interval (from P onset to QRS onset), RR interval (from one QRS onset to the next QRS onset), and QT interval (from QRS onset to T offset). These are fundamental because many ECG diagnoses are defined by threshold crossings: a PR interval exceeding 200 ms indicates first-degree AV block; a QRS duration exceeding 120 ms is required for bundle branch block diagnoses.
• Amplitude measurements: Peak heights of P, QRS, and T waves relative to the isoelectric line (in millivolts), plus ST-segment deviation measured at the J-point (the junction between QRS end and ST segment beginning). Amplitude criteria are central to hypertrophy diagnoses — for example, the Sokolow-Lyon criterion for left ventricular hypertrophy requires the sum of S in V1 and R in V5 or V6 to exceed 3.5 mV.
• Morphological analysis: The pipeline classifies QRS configurations (e.g., qR, rS, RSR' patterns) by analyzing the sequence of positive and negative deflections within each QRS complex. It also explicitly verifies the presence of pathological Q waves, defined as Q waves exceeding 40 ms in duration or 25% of the subsequent R-wave amplitude. Morphological patterns are critical for conduction disturbance diagnoses — the RSR' pattern in V1 is the hallmark of right bundle branch block.
• Axis computation: The frontal plane electrical axis is calculated for each beat based on the net area under the QRS complexes in leads I and aVF. Axis deviation (left axis < -30°, right axis > +90°) is required for fascicular block diagnoses — left anterior fascicular block requires left axis deviation, while left posterior fascicular block requires right axis deviation.

Stage 3: Finding Extraction. The quantitative measurements are then mapped to discrete clinical findings by applying established clinical criteria. This is a thresholding and classification layer that transforms continuous physiological values into categorical medical statements. For example:

• A PR interval exceeding 200 ms in the majority of detected beats → "Prolonged PR interval"
• A QRS duration > 120 ms in the majority of beats → "Prolonged QRS duration"
• ST-segment deviation at the J-point exceeding 0.1 mV in at least two contiguous leads → "ST elevation" (with lead group specified)
• QRS axis between -30° and -90° → "Left axis deviation"

This mapping is deterministic and rule-based, using thresholds from the ECG Core Curriculum (Zimmerman, 2023). The "majority of beats" criterion for interval-based findings is a design choice that adds robustness: a single aberrant beat (e.g., a premature complex) won't trigger a pathological finding unless it's consistent across the recording. This mirrors clinical practice, where findings are typically reported based on the dominant rhythm.

Stage 4: Diagnosis Derivation. The final stage combines identified findings into diagnostic conclusions using hierarchical logic diagrams. These diagrams (provided in Appendix A.2, Figures 6–21) are decision trees that encode the necessary and sufficient conditions for each of the 17 core diagnoses. For instance, the logic diagram for First Degree AV Block (Figure 6) requires: (1) every P wave (excluding those following premature beats) must be consistently followed by a QRS complex — a necessary condition for any AV conduction diagnosis; (2) the PR interval must be prolonged (> 200 ms) — the defining criterion. Only when both conditions are met is 1AVB confirmed. The diagram also encodes negation paths: if the PR interval is not prolonged, the diagnosis is "Not 1AVB," regardless of the first criterion.

The diagrams were codified from authoritative guidelines (ECG Core Curriculum; Zimmerman, 2023) and validated by three board-certified internal medicine specialists, providing a clinical authority layer that distinguishes the benchmark from purely automated or LLM-generated reasoning structures. The hierarchical arrangement — findings feeding into sub-decisions feeding into final diagnoses — mirrors the systematic approach taught in medical training, which is precisely what the benchmark aims to test.

Why build an automated pipeline rather than use human annotations? The paper's approach represents a deliberate tradeoff between coverage and annotation cost. Manual expert annotation of waveform boundaries for thousands of ECGs would be prohibitively expensive (the benchmark comprises over 6,400 multi-turn cases derived from 2,868 PTB-XL and 3,316 MIMIC-IV-ECG recordings). The automated pipeline achieves near-perfect QRS detection and strong P and T wave segmentation (Table 2), making it reliable enough to serve as ground truth for evaluation purposes. The tradeoff is that the pipeline's errors (e.g., the P-wave precision gap due to unannotated waves in LUDB) propagate into the benchmark. The paper mitigates this by filtering samples to include only those where the pipeline's diagnosis matches the human-provided label (Section 4.2), creating a consensus ground truth that combines algorithmic objectivity with human expertise.

---

Diagnostic Logic Diagrams: Structure and Coverage

The 17 core diagnoses are organized into six clinical groups (Section A.2): AV Block (3 diagnoses), Conduction Disturbance (4), Hypertrophy (2), Ectopic Beat (2), Myocardial Infarction (3), and Ischemia (3). Each diagnosis has a dedicated logic diagram that specifies:

1. The sequence of clinical findings that must be verified, in a specific order. For example, the CLBBB diagram (Figure 9) proceeds: Prolonged QRS duration → Dominant S waves in V1/V2 → Positive monophasic QRS without Q waves in lateral leads → Notched R waves in lateral leads. The order is clinically motivated: Prolonged QRS is the most fundamental criterion for any bundle branch block, so it comes first; if QRS duration is normal, the diagnosis can be immediately excluded without checking the other criteria.

2. The grounding sub-tasks required for each finding. Not every finding requires all three grounding types. The CLBBB diagram specifies that "Prolonged QRS duration" requires Wave Grounding (locating the QRS complex) and Measurement Grounding (selecting the duration range), but not Lead Grounding (since QRS duration is a global measurement). Conversely, "Notched R waves in lateral leads" requires Lead Grounding (which of I, aVL, V5, V6) and Wave Grounding (which temporal segment), but not Measurement Grounding (since notch presence is a qualitative finding).

3. Branching logic for both positive and negative paths. When a finding is absent (the model answers "No" at Finding Identification), the diagram specifies whether the diagnosis is immediately excluded or whether alternative reasoning paths exist. For diagnoses with multiple possible presentations — e.g., Complete Right Bundle Branch Block (Figure 10), which can present with either an RSR' pattern or a broad monophasic R wave in V1 — the diagram encodes the different paths and their respective criteria.

4. The "Abort Diagnosis" node for conditions that preclude the target diagnosis. The LAFB diagram (Figure 11) includes an explicit check for left bundle branch block before confirming LAFB, because LBBB distorts the QRS axis and makes fascicular block diagnosis unreliable. This kind of clinical nuance — where one diagnosis invalidates another — is encoded directly in the logic structure.

Multiple reasoning paths per diagnosis. Crucially, a single diagnosis can be confirmed through multiple combinations of clinical findings. For example, Left Ventricular Hypertrophy (Figure 13) can be diagnosed through several independent voltage criteria (R in aVL > 1.1 mV, Sokolow-Lyon sum > 3.5 mV, Cornell Product > 0.244 mV·s), plus a strain pattern criterion. The diagram encodes seven distinct reasoning paths (four paths with strain pattern when any one voltage criterion is met, plus three paths without strain pattern when at least two voltage criteria are met). The sampling strategy ensures even coverage across all paths (Section 4.2), preventing models from exploiting path-specific shortcuts — a model that only recognizes the Sokolow-Lyon criterion would fail on samples where LVH is diagnosed via the Cornell Product alone.

Prevalence coverage. The paper analyzes the clinical relevance of its 17 diagnoses by examining their prevalence in PTB-XL and MIMIC-IV-ECG (Figure 4). The 17 core diagnoses directly cover 56.05% of PTB-XL samples and 82.03% of MIMIC-IV-ECG samples. When including diagnoses that can be logically derived from the 17 core categories (e.g., "inferolateral MI" from combining inferior and lateral MI criteria), coverage expands to 76.85% (PTB-XL) and 96.18% (MIMIC-IV-ECG). This is not just a statistical detail — it establishes that the benchmark evaluates conditions that a deployed ECG AI would encounter frequently, making the findings practically relevant rather than academic.

---

Sampling Strategy and Dataset Construction

The benchmark is constructed from two source datasets: PTB-XL (Wagner et al., 2022) and MIMIC-IV-ECG (Gow et al., 2022), both large public ECG databases with human-annotated diagnostic labels. The sampling strategy has several carefully designed properties:

Balanced positive and negative cases. For each of the 17 diagnoses, the benchmark includes 100 positive samples (where the diagnosis is confirmed) and 100 negative samples (where the diagnosis is absent), with two exceptions: Second Degree AV Block on PTB-XL has only 3 positives due to natural scarcity, and Third Degree AV Block on PTB-XL has only 9 positives, similarly limited. The balanced design prevents metrics from being skewed by class prevalence. A model that always answers "No" would achieve 50% accuracy on this benchmark, providing a meaningful baseline.

Uniform coverage across reasoning paths. The paper explicitly states that "we ensured that the selected samples were evenly distributed across the various logical paths defined in our logic diagrams." The detailed breakdown in Table 4 (Appendix B) quantifies this: for Left Ventricular Hypertrophy on PTB-XL, the 100 positives are distributed as 4 + 16 + 16 + 16 + 16 + 16 + 16 across the seven reasoning paths. The non-uniform first bucket (4) likely reflects the natural rarity of that particular path in the source dataset. For diagnoses where the target of 100 is not perfectly divisible by the number of paths, the total is slightly increased to maintain exact balance — e.g., Third Degree AV Block on PTB-XL has 3 reasoning paths for negatives, yielding 102 cases (34 per path) rather than 100.

This uniform path coverage is methodologically important because it prevents a model from gaming the benchmark by specializing in only the most common presentation. A model that learns to recognize LVH only through the Sokolow-Lyon criterion would succeed on samples following that path but fail on samples following the Cornell Product path. Since all paths are equally represented, a high Completion score requires competence across all valid diagnostic pathways.

Consensus filtering. The paper applies a strict filter: only samples where the human-provided label (from PTB-XL or MIMIC-IV-ECG) aligns with the automated pipeline's diagnosis are included. This serves two purposes. First, it eliminates cases where either the human label or the automated pipeline might be erroneous, since disagreement suggests uncertainty. Second, it ensures that the ground truth used for evaluation is supported by explicit, verifiable signal features — the automated pipeline's diagnosis is always traceable to specific waveform measurements, so every included sample has a complete, auditable reasoning chain.

Expert validation subset. To establish a reliability baseline, three board-certified internal medicine specialists validated 143 representative samples — one instance per unique reasoning path from both PTB-XL and MIMIC-IV-ECG (excluding one rare path unavailable in PTB-XL). This expert review serves as a quality check on the logic diagrams and the automated pipeline's output, though the paper does not report quantitative inter-rater agreement statistics. Following the specialist validation, all authors manually reviewed the extracted reasoning path for every single sample under specialist supervision, providing an additional layer of human oversight.

Final dataset scale. The benchmark comprises 3,076 samples from PTB-XL and 3,355 from MIMIC-IV-ECG, totaling over 6,400 multi-turn evaluation cases. Each case generates multiple question-answer pairs — 24,097 total QA pairs for PTB-XL (average 7.81 reasoning turns per case) and 27,025 for MIMIC-IV-ECG (average 8.05 turns). This scale enables statistically meaningful per-diagnosis analysis (100+ samples per diagnosis per dataset) while the multi-turn depth (average ~8 turns) ensures that the reasoning assessment is non-trivial.

---

Evaluation Protocol and Metrics

Multi-turn conversation format. The evaluation is conducted as a structured dialogue between the evaluation engine and the model under test. Each turn presents a multiple-choice question, and the model must select one or more options (for multi-select grounding questions) from the provided list. This format — rather than free-text generation — is chosen deliberately to enable objective, deterministic scoring. Free-text responses would require subjective interpretation of whether the model's answer matches the ground truth, introducing the same LLM-as-a-Judge confounds the paper critiques. Multiple choice eliminates this ambiguity: the model's selection either matches the correct answer(s) or it doesn't.

Semantic consistency checking with Gemini-3-Flash. The paper does not use rigid string matching to compare model responses to ground truth, recognizing that "correct answers formulated in different styles" should not be penalized. Instead, it employs Gemini-3-Flash as a semantic consistency checker that verifies whether the model's response conveys the same meaning as the ground truth answer. This is a subtly different use of an LLM from the LLM-as-a-Judge paradigm the paper criticizes: rather than evaluating quality or fluency, the checker performs a narrow semantic equivalence task ("does response A mean the same thing as answer B?"), which is less susceptible to stylistic bias. The paper does not report the accuracy of this checking mechanism, which is a limitation — errors in semantic equivalence checking would propagate into all reported metrics.

Metric 1: Initial Diagnosis Accuracy (IDA). This is the model's accuracy on the Initial Diagnostic Question — the very first question of each evaluation case, asked before any step-wise reasoning. The question takes the form "Does this ECG suggest the presence of [diagnosis]?" with yes/no options. IDA measures baseline diagnostic capability prior to engaging in structured reasoning. It serves as a reference point: if a model achieves high IDA but near-zero Completion, it demonstrates that the model can guess the correct diagnosis without being able to justify it — evidence of pattern-matching without reasoning.

Metric 2: Completion (%). This is the strictest metric, defined as "the proportion of samples where the model correctly answers every constituent multi-turn reasoning question" (Section 5.1). The evaluation for this metric terminates at the first incorrect response at any reasoning step. If a model fails at Step 2 of Loop 1, the Completion for that sample is 0, regardless of how many subsequent questions it might have answered correctly. This makes Completion a conjunctive metric — it measures the probability that the entire reasoning chain is flawless. The paper's central finding is that this metric is near zero (< 6%) across all models and both datasets.

The stringency of Completion is intentional: in clinical practice, an error at any step of a diagnostic reasoning chain can lead to misdiagnosis. A model that correctly identifies the diagnostic criteria but incorrectly detects whether they are present is just as dangerous as a model that fails at criterion selection — both produce wrong conclusions. Completion captures this "no room for error" requirement.

Metric 3: Depth (0–4). This metric is more forgiving and more informative than Completion. It quantifies "the model's average progression through the 4-step verification sequence, evaluated independently per finding" (Section 5.1). The scoring works as follows:

• For each finding loop, the model earns +1 for each correctly completed step (Criterion Selection, Finding Identification, ECG Grounding, Diagnostic Decision).
• Within Step 3 (ECG Grounding), scoring is fractional: if the finding requires N grounding sub-tasks (e.g., N=2 for Lead + Wave grounding), each correctly answered sub-task contributes 1/N to the depth score.
• If the model fails at any step within a finding loop, its depth score for that finding is recorded up to the point of failure, and the evaluation proceeds to the next required finding by injecting the ground-truth history of all preceding steps into the prompt.

This "recovery" mechanism is crucial: it ensures that a failure in one finding loop does not contaminate the evaluation of subsequent loops. Without this, a model that fails at the first finding would get Depth=0 for all remaining findings, making it impossible to determine whether the failure was localized (one specific finding it couldn't detect) or global (it cannot do any reasoning). By providing the correct history and continuing, the benchmark isolates performance on each finding independently.

• All finding-level depth scores are pooled across all samples to compute the global average. This micro-averaging means that each clinical finding contributes equally, rather than each sample contributing equally.

The concrete example in Appendix C.1 illustrates this: for a CLBBB sample with four finding loops, if the model achieves depth scores of 2.5, 4.0, 1.0, and 2.0 across the four loops, the micro-averaged depth is (2.5 + 4.0 + 1.0 + 2.0) / 4 = 2.375. This average of 2.375 tells us that the model can, on average, complete Criterion Selection (Step 1) and Finding Identification (Step 2) but rarely progresses far into ECG Grounding (Step 3) — precisely the pattern the paper reports.

The paper's key finding (Section 5.2) is that average depth scores "rarely exceed 2.0 across the board," meaning that after Criterion Selection and sometimes Finding Identification, models consistently fail at the grounding stages. A depth of ~2.0 is the threshold between "has medical knowledge" (Stage 1) and "can apply it to visual evidence" (Stages 2–3).

Metric 4: GT-Reasoning-Based Diagnosis Accuracy (GT-RDA). This metric measures diagnostic accuracy under perfect guidance. Regardless of any prior failures at earlier steps, the evaluation engine explicitly provides the complete ground-truth reasoning history up to the final Diagnostic Decision step in the prompt, then asks the model to make the diagnosis. The metric is the accuracy of this final diagnostic decision.

GT-RDA is a diagnostic tool for distinguishing between two types of failure:

• Comprehension failure: The model cannot understand or utilize valid clinical reasoning even when it is presented. A model with low GT-RDA falls into this category — it fails to make the correct diagnosis even after being shown exactly which findings are present and why they matter.
• Generation failure: The model can utilize valid reasoning when provided but cannot generate it independently. A model with high GT-RDA but low Completion falls into this category — it understands clinical logic but cannot execute the perceptual tasks required to construct the reasoning chain.

The paper's most striking finding (Section 5.2) is that ECG-specific models show a GT-RDA collapse: ECG-R1-RL drops from 85.41% IDA to 22.70% GT-RDA on PTB-XL, and PULSE drops from 80.93% to 35.18%. This is interpreted as evidence that these models never learned clinical reasoning — they learned to map global signal patterns directly to diagnostic labels, and when presented with the correct intermediate reasoning (findings + measurements), they cannot integrate it because their training optimized for a shortcut that bypassed the reasoning process entirely. In contrast, non-ECG-specific models like Hulu-Med show the expected pattern: IDA near random guessing (~50–57%) but GT-RDA soaring to 86–99%, demonstrating latent capacity to utilize clinical reasoning when externally provided.

Why metrics are computed per-finding rather than per-sample. The Depth metric's finding-level pooling is a deliberate design choice that reflects the benchmark's goal of diagnosing where reasoning fails. If scoring were per-sample, a single early failure would mask all subsequent performance, and the benchmark would lose granularity. By computing finding-level depths independently — providing ground-truth history between loops — the benchmark can answer questions like "do models fail equally on all types of findings, or do certain findings (e.g., measurement-dependent ones) cause disproportionate failures?" This granularity is what enables the paper's claim that the primary bottleneck is visual grounding, not knowledge retrieval.

---

Model Configurations and Prompting Strategy

Evaluated models. The paper evaluates 19 model configurations spanning four categories (Section 5.1, Appendix C.2):

• ECG-specific models (5): PULSE (7B), GEM (7B), ECG-R1-SFT (8B), ECG-R1-RL (8B), OpenTSLM (3B) — models specifically fine-tuned on ECG interpretation data.
• Medical-domain models (5): Hulu-Med (7B, 32B), MedGemma (4B, 27B), MedGemma-1.5 (4B) — models optimized for broad biomedical tasks.
• Open-weight general domain models (4): Qwen3-VL (8B, 32B), Llama-3.2-Vision (11B, 90B) — general-purpose vision-language models without specific medical fine-tuning.
• Proprietary models (5): Gemini-2.5-Flash, Gemini-2.5-Pro, Gemini-3-Flash, GPT-5-Mini, GPT-5.2 — accessed via official APIs.

Input modality. The paper accommodates two input formats depending on model architecture (Section 5.1). OpenTSLM, which is natively designed for time-series processing, receives the ECG as a 100 Hz, 12-channel time-series array. All other models receive standard 12-lead 2D ECG images generated using the ecg-plot Python package, which renders the 1D signals as the familiar grid-based visualization used in clinical practice. This dual-modality design ensures that time-series-native models are not disadvantaged by being forced into an image format, while vision-language models receive the input format they were trained on.

Deterministic decoding. The paper configures all models for deterministic output to eliminate randomness in the evaluation. Specifically, "the decoding temperature was set to 0 across all generation pipelines" (Appendix C.2). This ensures that each model produces its most probable response consistently, making the benchmark results reproducible. The tradeoff is that temperature=0 may disadvantage models that benefit from stochastic decoding for reasoning tasks, but the paper prioritizes reproducibility over potential performance gains from sampling.

System prompt design. The paper uses a standardized system prompt (Figure 5 in Appendix C.3) that instructs models to adopt the persona of "an expert cardiologist specializing in advanced electrocardiography" participating in "a rigorous clinical reasoning examination." The prompt contains four explicit instructions:

1. Read the question and provided options.
2. "Analyze the ECG systematically to answer the question."
3. Select answers from the given options that correspond to findings "visible in the ECG image" or the correct diagnostic criterion.
4. Base judgment "strictly on established diagnostic criteria defined in standard textbooks" and "finalize your diagnosis only when sufficient evidence exists," while acknowledging that "further findings are required if only necessary conditions are met."

This prompt is deliberately designed to align the model's behavior with the benchmark's expectations: it mandates systematic analysis, requires visual grounding ("visible in the ECG image"), invokes textbook criteria (matching the logic diagrams' derivation), and enforces the "further findings required" discipline that the multi-loop evaluation depends on. The paper states that this "prevents models from being unfairly penalized for premature diagnoses" — without this instruction, a model might legitimately answer "Yes" to a Diagnostic Decision question after the first finding, based on clinical heuristics, and be marked wrong because the benchmark requires completing all loops before confirmation.

Version pinning for reproducibility. The paper pins exact model identifiers for proprietary APIs (e.g., gpt-5-mini-2025-08-07, gemini-3-flash-preview) and specifies Hugging Face repository paths for open-weight models (e.g., PULSE-ECG/PULSE-7B, PKUDigitalHealth/ECG-R1-8B-RL). This level of detail is essential for benchmark reproducibility, as proprietary models are frequently updated and model behavior can shift between versions even when the model name remains the same.

---

Summary of Design Choices and Their Justifications

• 4-stage verification loop over single-question evaluation: isolates knowledge retrieval, perceptual detection, and visual grounding into separately measurable components, enabling fine-grained diagnosis of failure modes rather than a binary pass/fail.
• Automated analysis pipeline over manual annotation: trades perfect annotation accuracy for scalability, enabling thousands of cases with complete waveform-level ground truth that would be economically infeasible to annotate manually. The consensus filter (pipeline diagnosis must match human label) mitigates pipeline errors.
• Multiple-choice format over free-text generation: enables deterministic, objective scoring without relying on LLM-as-a-Judge quality assessment, which the paper argues is structurally incapable of verifying grounding. The semantic consistency checker (Gemini-3-Flash) is used only for narrow equivalence verification, not quality judgment.
• Dual-distractor design in Criterion Selection: prevents models from exploiting simple heuristics (pick the "abnormal" option or pick the option matching a detected finding) and forces genuine diagnostic knowledge.
• Iterative loop structure with ground-truth injection between loops: prevents early failures from contaminating later evaluations, enabling per-finding Depth computation that isolates where in the reasoning chain models fail.
• Uniform sampling across reasoning paths: prevents models from specializing in common presentations and requires competence across all valid diagnostic pathways for a given diagnosis.
• GT-RDA as a diagnostic metric: distinguishes between models that fail to generate reasoning (low Completion, high GT-RDA) and models that fail to comprehend reasoning (low GT-RDA), with the ECG-specific model collapse revealing training-induced shortcut learning.
• Temperature=0 decoding: prioritizes reproducibility over potential performance gains from stochastic sampling, consistent with the benchmark's goal of evaluating capability rather than optimizing for score.
• System prompt mandating textbook criteria and evidence-gathering discipline: aligns model behavior with the benchmark's structured evaluation without giving away specific answers, ensuring fair comparison across models with different default behaviors.

4. Key Insights and Innovations

Innovation 1: Shifting Evaluation from Fluency Judgment to Evidence-Grounded Verification

The paper's most fundamental contribution is not a new model or training recipe but a paradigm-level reframing of what it means to evaluate an ECG-interpreting AI. Prior work — PULSE, GEM, ECG-R1, OpenTSLM — all evaluated their models using the LLM-as-a-Judge framework (Zheng et al., 2023), which compares generated text against reference explanations that are themselves synthetically produced by an LLM. The paper identifies this as a structurally circular evaluation loop: the model is trained on LLM-generated rationales, tested against LLM-generated references, and scored by an LLM that never examines the actual ECG signal. High scores in this regime primarily measure stylistic alignment with the text generator's output distribution, not clinical correctness.

What makes the paper's reframing distinctive is that it treats evaluation as an adversarial probe rather than a similarity measurement. The 4-stage verification loop is not designed to assess whether the model's output sounds like valid reasoning — it is designed to test whether the model can perform the specific cognitive operations that constitute valid reasoning, with each stage isolating a different capability. Criterion Selection tests knowledge retrieval independent of perception. Finding Identification tests perception independent of localization. ECG Grounding demands spatial-temporal evidence that cannot be faked through linguistic fluency — the model must point to specific leads, waveform segments, and measurement ranges where it claims to see the finding. Diagnostic Decision tests whether the model can integrate partial evidence without premature closure.

This adversarial framing is a substantial departure from the field's dominant assumption that fluent, medically plausible text is evidence of genuine understanding. The paper's results vindicate the reframing: ECG-specific models like ECG-R1-RL achieve 85.41% Initial Diagnosis Accuracy, demonstrating that they can often guess the correct answer, yet their Completion rates are ~5.90% (Table 1) and their GT-RDA collapses to 22.70%. This is the empirical signature of a model that has learned to produce the outputs of reasoning without performing the process of reasoning. The LLM-as-a-Judge framework would have given these models high scores; ECG-Reasoning-Benchmark exposes them as fundamentally unreliable.

This reframing has implications beyond ECG interpretation. It provides a template for evaluating any domain where MLLMs are being trained on synthetic rationales: demand evidence that the model can localize its claims to specific features in the input, test each cognitive sub-skill independently, and use adversarial distractors that probe whether the model is pattern-matching or reasoning. The paper demonstrates that without such adversarial probing, high fluency scores create a dangerous illusion of competence.

Innovation 2: The GT-RDA Metric as a Diagnostic Tool for Distinguishing Shortcut Learning from Genuine Reasoning

The GT-Reasoning-Based Diagnosis Accuracy metric represents a genuinely novel diagnostic concept in AI evaluation. The idea is deceptively simple: give the model the complete ground-truth reasoning chain — all the clinical findings, their lead and wave localizations, their quantitative measurements — and then ask only the final diagnostic question. Compare this accuracy against the model's unaided Initial Diagnosis Accuracy.

Prior evaluation paradigms had no mechanism to distinguish between two very different failure modes: (a) a model that understands clinical reasoning but cannot generate it independently due to perceptual limitations, versus (b) a model that never learned clinical reasoning in the first place and instead relies on shortcuts. Both failure modes look identical under standard accuracy metrics — the model gets the diagnosis wrong. GT-RDA teases them apart.

The paper's most striking finding — and arguably its most important contribution to the broader conversation about MLLM evaluation — is the GT-RDA collapse in ECG-specific models (Table 1). ECG-R1-RL drops from 85.41% IDA to 22.70% GT-RDA on PTB-XL. PULSE drops from 80.93% to 35.18%. These models are worse at making a diagnosis when given the correct reasoning than when guessing from the ECG alone. This is not a random fluctuation — it is a systematic effect that reveals something profound about how these models were trained.

The paper's interpretation (Section 5.2) is that these models optimized for a direct signal-to-diagnosis shortcut during training. The training data paired ECG images with text-derived rationales, but the model discovered it could achieve low training loss by learning global image-to-label mappings without attending to the reasoning content. When presented with explicit reasoning at test time — reasoning that contradicts the shortcut the model learned — the model cannot integrate it because its internal representations never encoded the intermediate reasoning steps. The GT-RDA collapse is evidence that the model's "knowledge" of ECG interpretation is not compositional — it cannot use the pieces of clinical reasoning because it never learned them as separable components.

This finding fundamentally challenges the dominant paradigm of training MLLMs on LLM-generated rationales. The standard assumption is that training on reasoning traces will teach the model to reason. The paper's evidence suggests the opposite can happen: if the reasoning traces are not grounded in the input modality (because the LLM generating them never saw the ECG), the model may learn to bypass the traces entirely and map directly from input patterns to output labels. The reasoning traces become decorative text that the model learns to ignore. GT-RDA provides the diagnostic tool that reveals this failure — without it, the high IDA scores would be interpreted as success.

In contrast, non-ECG-specific models show the expected pattern: Hulu-Med (32B) goes from 57.49% IDA (near random) to 99.42% GT-RDA on PTB-XL. These models lack the domain-specific shortcuts that ECG-specific models developed, so they fall back on genuine language understanding, and they can use the provided reasoning effectively. The asymmetry between ECG-specific and general models on GT-RDA is the paper's strongest empirical argument that ECG-specific fine-tuning on synthetic rationales is currently doing more harm than good for reasoning capability.

Innovation 3: The Automated Analysis Pipeline as Ground-Truth Infrastructure for Scalable Reasoning Evaluation

While the automated ECG analysis pipeline is described in Section 3 as a technical mechanism, its role in the paper's contribution is better understood as infrastructure that enables a new class of evaluation. The paper faced a chicken-and-egg problem: to evaluate whether an MLLM's reasoning is grounded in signal evidence, you need ground truth that specifies exactly which findings are present, where they are located in the signal, and what their quantitative values are. Creating this ground truth manually for thousands of ECGs would require expert cardiologists to annotate every waveform boundary, interval measurement, and finding — an economically infeasible prospect that explains why no prior benchmark attempted step-by-step reasoning verification at scale.

The automated pipeline solves this by extracting ground truth deterministically from the raw signal using well-defined algorithms and expert-validated rules. The key design insight is that the pipeline does not need to be perfect — it needs to be reliable enough that its errors can be controlled through consensus filtering with human labels. The paper's filtering strategy (only include samples where the pipeline's diagnosis matches the human-provided label) creates a high-confidence subset where both the algorithmic and human perspectives agree. The expert validation of 143 representative samples provides additional quality assurance, establishing that the pipeline's reasoning chains are clinically valid for the paths they cover.

What makes this infrastructure contribution significant is that it decouples evaluation from annotation cost, enabling reasoning benchmarks to scale to domains where expert annotation is expensive. The LUDB evaluation (Table 2) demonstrates that the pipeline's waveform detection is near-perfect for QRS complexes (1.000 recall and precision) and strong for P and T waves (0.978 and 0.997 recall respectively), making it suitable as ground truth for evaluation purposes. The pipeline is not a clinical-grade diagnostic tool — the paper does not claim it should replace human interpretation — but it is sufficiently accurate to serve as an objective reference against which to test whether models can perform the elementary operations of ECG analysis.

This infrastructure approach generalizes beyond ECGs. Any domain with established quantitative criteria for diagnosis — radiology measurements, pathology quantification, laboratory value interpretation — could potentially adopt a similar pipeline: build an automated analysis system that extracts verifiable features from the raw data, encode expert-validated diagnostic logic as rule-based decision trees, and use the combined system to generate ground-truth reasoning chains for evaluation. The paper demonstrates that this approach is viable at scale: over 6,400 multi-turn cases constructed across two source datasets with 24,000+ total QA pairs (Table 3).

Innovation 4: Empirical Demonstration That Current Training Paradigms Produce Models Incapable of Multi-Step Reasoning — And That the Failures Are Systematic, Not Random

The paper's quantitative results do more than just show low scores — they reveal a systematic pattern of failure that diagnoses the nature of the reasoning gap. Prior critiques of MLLM reasoning capabilities (e.g., Huang et al., 2023 on self-correction) tended to report aggregate failure rates without decomposing where in the reasoning chain the failure occurs. The paper's Depth metric provides this decomposition, and the results (Table 1, Section 5.2) reveal a consistent pattern across nearly all evaluated models: Depth scores cluster in the range of ~1.0–2.0, with only the top proprietary models (Gemini-3-Flash, Gemini-2.5-Pro) reaching ~2.0.

The interpretation of Depth ~1.0–2.0 is specific and informative. A score of 1.0 means the model can successfully complete Criterion Selection — it retrieves the correct diagnostic criteria from medical knowledge — but fails at Finding Identification, the point where it must examine the actual ECG to determine whether the criterion is met. A score of 2.0 means the model can sometimes detect whether a finding is present but rarely progresses far into the ECG Grounding stage where it must localize and quantify the evidence. The bottleneck is not knowledge (Stage 1) but perception and grounding (Stages 2–3).

This finding is more significant than a generic "models fail at reasoning" claim because it identifies what kind of reasoning is missing. The models possess declarative medical knowledge — they can tell you that Complete Left Bundle Branch Block requires prolonged QRS duration, dominant S waves in V1/V2, and notched R waves in lateral leads. What they cannot do is examine a specific ECG and determine whether those findings are present, where they are located, and what their quantitative values are. This is the difference between knowing the textbook definition of a disease and being able to diagnose it from patient data.

The pattern is not explained by model scale or domain specialization. MedGemma (27B) achieves Depth=1.51 on PTB-XL, only marginally better than MedGemma (4B) at Depth=0.80. ECG-specific models like ECG-R1-RL (8B) reach Depth=1.79, slightly above Qwen3-VL (8B) at Depth=1.53. The proprietary models (Gemini-3-Flash at Depth=2.09, GPT-5.2 at Depth=1.97) are better but still fail to consistently ground their reasoning. Increasing model capacity improves knowledge retrieval but does not substantially improve perceptual grounding — suggesting that the problem is structural (how models are trained) rather than scalar (how large they are).

The Completion metric reinforces this interpretation. Completion measures the probability that a model executes an entire reasoning chain without a single error. The near-zero rates (< 6% maximum) mean that even the best models are essentially incapable of completing the multi-step verification required for a typical diagnosis. Given that the average case requires ~8 reasoning turns (Table 3), a model with a per-step accuracy of even 80% would have a Completion rate of 0.80^8 ≈ 17%. The observed Completion rates of ~5–6% imply per-step accuracies substantially below 80% once the model moves beyond Criterion Selection. This quantifies just how fragile current MLLM reasoning is: the probability of a flawless chain is vanishingly small because each perceptual step is unreliable.

The significance of this finding lies in its actionable specificity. It tells the field exactly what needs to be fixed: not medical knowledge acquisition, but visual grounding — the ability to map from diagnostic criteria to specific signal features. Future training approaches should focus on teaching models to localize and measure ECG features, possibly through explicit supervision on waveform segmentation, interval measurement, and lead-specific finding detection, rather than through more text-derived rationales.

5. Experimental Analysis

Evaluation Methodology

• Dataset. All experiments use ECG-Reasoning-Benchmark, constructed from two source datasets: PTB-XL (2,868 unique ECGs yielding 3,076 multi-turn cases) and MIMIC-IV-ECG (3,316 unique ECGs yielding 3,355 multi-turn cases), totaling over 6,400 evaluation cases. For each of the 17 core diagnoses, the benchmark includes 100 positive and 100 negative samples (with rare exceptions for scarce conditions like Second Degree AV Block on PTB-XL, which has only 3 positives), with cases evenly distributed across all valid reasoning paths within each diagnosis. The evaluation generates 24,097 total QA pairs from PTB-XL (average 7.81 reasoning turns per case) and 27,025 from MIMIC-IV-ECG (average 8.05 turns).

• Base model(s). The paper evaluates 19 model configurations spanning four categories: ECG-specific models — PULSE (7B), GEM (7B), ECG-R1-SFT (8B), ECG-R1-RL (8B), OpenTSLM (3B); medical-domain models — Hulu-Med (7B, 32B), MedGemma (4B, 27B), MedGemma-1.5 (4B); open-weight general domain models — Qwen3-VL (8B, 32B), Llama-3.2-Vision (11B, 90B); and proprietary models accessed via official APIs — Gemini-2.5-Flash, Gemini-2.5-Pro, Gemini-3-Flash, GPT-5-Mini, and GPT-5.2. The ECG-specific models were chosen because they represent the current state-of-the-art in ECG-MLLM development and allow direct testing of the paper's central hypothesis that models trained on synthetic rationales learn shortcuts rather than genuine reasoning. The general-domain and medical-domain models provide baselines for how much ECG reasoning capability emerges from broad training versus domain-specific fine-tuning.

• Metrics. Four quantitative metrics are computed from the model's multi-turn response history:

  • Initial Diagnosis Accuracy (IDA) (%) — the accuracy on the Initial Diagnostic Question ("Does this ECG suggest the presence of [diagnosis]?") asked before any step-wise reasoning begins. This measures baseline diagnostic capability without structured reasoning support.
  • Completion (%) — the proportion of samples where the model correctly answers every constituent multi-turn reasoning question. Evaluation terminates upon the first incorrect response at any reasoning step, making this a strict conjunctive metric that requires flawless execution of the entire reasoning chain for a sample to count as a success.
  • Depth (0–4) — the model's average progression through the 4-step verification sequence, evaluated independently per clinical finding. Each correctly completed step (Criterion Selection, Finding Identification, ECG Grounding, Diagnostic Decision) contributes +1, with Step 3 scored fractionally based on the number of required grounding sub-tasks (each correctly answered sub-task contributes 1/N where N is the number of sub-tasks for that finding). If a model fails at any step within a finding loop, its depth score for that finding is recorded up to the failure point, and the ground-truth history of all preceding steps is injected into the prompt before proceeding to the next required finding. All finding-level depths are micro-averaged across all samples to compute the global metric.
  • GT-Reasoning-Based Diagnosis Accuracy (GT-RDA) (%) — diagnostic accuracy when the model is explicitly provided with the complete ground-truth reasoning history up to the final Diagnostic Decision step, regardless of prior failures. By comparing this against IDA, the metric distinguishes between models that fail to comprehend valid reasoning (low GT-RDA) versus models that can utilize reasoning when provided but cannot generate it independently (high GT-RDA despite low Completion).

• Baselines. The primary comparisons are between model categories (ECG-specific vs. medical-domain vs. general-domain vs. proprietary) on the four reasoning metrics. There is no single "best-of-N" or "majority voting" baseline in the traditional sense, because the benchmark does not involve stochastic sampling — all models are evaluated with temperature=0 for deterministic output. The paper uses IDA as an internal baseline for each model: the gap between IDA and GT-RDA reveals whether a model's diagnostic behavior is consistent with genuine reasoning (GT-RDA should exceed IDA if reasoning helps) or indicative of shortcut learning (GT-RDA below IDA, as seen in ECG-specific models).

• Generation budget / compute accounting. The paper does not measure compute in FLOPs or generation counts, as the evaluation is deterministic (temperature=0, single-pass) and does not involve search or sampling. All models are evaluated with the same multi-turn protocol applied uniformly across all samples. The implicit "cost" is the number of evaluation turns required per case (average ~8 turns), which is consistent across models. The paper pins exact model versions for proprietary APIs (e.g., gpt-5-mini-2025-08-07, gemini-3-flash-preview) to ensure reproducibility but does not report inference costs or latency.

• Cross-validation / statistical protocol. The paper does not employ cross-validation in the traditional machine learning sense, as it is evaluating fixed, pre-trained models rather than training new ones. Instead, it relies on design-level robustness mechanisms: (1) the benchmark includes both PTB-XL and MIMIC-IV-ECG as independent source datasets, providing a form of external validation — consistent patterns across both datasets strengthen confidence that findings are not dataset-specific artifacts; (2) the balanced sampling strategy (100 positive, 100 negative per diagnosis) and uniform reasoning path coverage prevent distributional confounds; (3) the expert validation of 143 representative samples by three board-certified internal medicine specialists, combined with manual review of every extracted reasoning path by all authors under specialist supervision, serves as a quality assurance protocol for the ground truth rather than a statistical protocol. The paper does not report confidence intervals or standard errors on any metric, which is a limitation — with ~100 samples per diagnosis quintile and a 500-question test set from PTB-XL, the per-diagnosis Completion rates have substantial binomial uncertainty that is not quantified.

Main Quantitative Results

Overall Reasoning Completion Rates

The central quantitative finding appears in Table 1, which reports IDA, Completion, Depth, and GT-RDA for all 19 model configurations across both PTB-XL and MIMIC-IV-ECG. The headline result is the near-universal failure on the Completion metric:

"Across all evaluated models and both datasets, performance on the Completion metric is remarkably poor, with maximum success rates reaching only around 6%."

On PTB-XL, Gemini-3-Flash achieves the highest Completion at 6.26%, followed by GEM at 6.13% and ECG-R1-RL at 5.90%. Proprietary models GPT-5.2 and Gemini-2.5-Flash reach 5.84% and 5.12% respectively. Among open models, Qwen3-VL (8B) reaches 5.67% and Qwen3-VL (32B) reaches 5.71%. On MIMIC-IV-ECG, the ranking is similar: ECG-R1-RL achieves 5.81%, Gemini-3-Flash achieves 5.72%, and ECG-R1-SFT and Qwen3-VL (8B) both achieve 5.45%.

At the bottom of the Completion distribution, several models effectively fail entirely. OpenTSLM (3B) achieves 0.65% on PTB-XL and 0.36% on MIMIC-IV-ECG. MedGemma (4B) reaches 0.58% and 0.83%, while MedGemma-1.5 (4B) reaches 0.68% and 1.01%. The paper attributes these near-zero rates to these models' limited parameter capacity:

"A qualitative review of their responses reveals that these compact models frequently lose contextual focus during the multi-turn verification loops, generating irrelevant or repetitive responses. This suggests that their limited parameter capacity restricts their ability to maintain sustained logical reasoning."

A notable outlier in the small-model category is Llama-3.2-Vision (11B), which achieves only 0.49% Completion on PTB-XL and 0.74% on MIMIC-IV-ECG — substantially worse than Qwen3-VL (8B) at 5.67% and 5.45%. The paper speculates that "the failure of this model stems from a lack of exposure to ECG or medical contexts during its training phase. Conversely, the robust performance of Qwen3-VL implies that its training corpus encompassed substantial medical or ECG-specific data."

The maximum Completion rate of ~6% means that even the best models fail to execute a complete, error-free reasoning chain on roughly 94% of cases. Given that the average case requires approximately 8 reasoning turns (Table 3), this translates to a per-step error rate that makes flawless chains vanishingly improbable. The finding is consistent across both datasets, with PTB-XL and MIMIC-IV-ECG showing highly correlated rankings — models that perform well on one dataset tend to perform well on the other, though MIMIC-IV-ECG Completion rates are consistently slightly lower (e.g., Gemini-3-Flash drops from 6.26% to 5.72%), suggesting MIMIC-IV-ECG may be marginally more challenging.

Depth Analysis: Pinpointing Where Reasoning Collapses

The Depth metric provides the critical decomposition that reveals where in the reasoning chain models fail. Table 1 reports Depth on the 0–4 scale:

"Consistently across both datasets, with the exception of the smallest models (OpenTSLM (3B), MedGemma (4B), MedGemma-1.5 (4B)), nearly all evaluated models achieve an average depth greater than 1.0. This indicates a broad success in the first verification stage: Criterion Selection."

On PTB-XL, Gemini-3-Flash achieves the highest Depth at 2.09, followed by Gemini-2.5-Pro at 2.01 and GPT-5.2 at 1.97. Among open models, ECG-R1-SFT reaches 1.80 and ECG-R1-RL reaches 1.79, while Qwen3-VL (32B) reaches 1.63. The Depth metric for the smallest models reflects their near-total failure: OpenTSLM at 0.17 and 0.13 (PTB-XL and MIMIC-IV-ECG), indicating these models cannot even consistently select the correct diagnostic criterion.

The critical interpretive threshold is Depth = 2.0. The paper states:

"However, average depth scores rarely exceed 2.0 across the board, pinpointing the critical bottleneck at the subsequent Finding Identification and ECG Grounding stages."

A Depth score of 2.0 corresponds to successful Criterion Selection (Step 1, +1) and Finding Identification (Step 2, +1), with zero or minimal progress into ECG Grounding (Step 3). The fact that only the top two proprietary models (Gemini-3-Flash, Gemini-2.5-Pro) cross this threshold — and barely, at 2.09 and 2.01 — means that even the best available models fail to provide verifiable grounding evidence for the findings they claim to detect. The paper's interpretation:

"Therefore, we suspect that while the current generation of AI models possesses the medical knowledge required for diagnosis, they lack the crucial ability to reliably ground that logic to the granular visual evidence of the ECG signal. This observation suggests that the primary limitation in automated ECG interpretation is not a knowledge deficit, but a profound reasoning gap in linking diagnostic criteria to the actual detection and measurement of specific findings within the signal."

An important nuance: the Depth metric uses a recovery mechanism where ground-truth history is injected after a failure, allowing independent evaluation of each finding loop. This means the Depth score reflects the model's independent capability on each finding type, uncontaminated by cascading errors. If a model fails at Finding Identification on Loop 1, it still gets a chance at Criterion Selection on Loop 2 after being shown the correct answer for Loop 1. The fact that Depth still plateaus around 2.0 despite this recovery mechanism indicates that the failure is not due to cascading error propagation — it is due to a fundamental inability to perform the perceptual and grounding tasks required at Steps 2–3, even when given a clean slate for each new finding.

The paper does not report per-diagnosis or per-finding-type Depth breakdowns, which limits the granularity of the analysis. It would be informative to know, for example, whether models fail more on measurement-intensive findings (e.g., QRS duration thresholds) versus qualitative findings (e.g., ST elevation presence), or whether certain diagnostic categories (e.g., hypertrophy with its complex voltage criteria) are disproportionately difficult. This remains a gap in the reported results.

IDA vs. GT-RDA: The Shortcut Learning Signature

The comparison between Initial Diagnosis Accuracy and GT-Reasoning-Based Diagnosis Accuracy produces the paper's most diagnostically revealing finding. The pattern differs dramatically between ECG-specific models and non-ECG-specific models, and this asymmetry is the primary evidence for the claim that ECG-specific fine-tuning on synthetic rationales produces shortcut learning.

Non-ECG-specific models show the expected pattern: GT-RDA substantially exceeds IDA. On PTB-XL:

• Hulu-Med (32B): IDA = 57.49%, GT-RDA = 99.42% (a 41.93 percentage point increase)
• Hulu-Med (7B): IDA = 55.87%, GT-RDA = 86.87% (+31.00 pp)
• MedGemma-1.5 (4B): IDA = 42.83%, GT-RDA = 85.02% (+42.19 pp)
• Qwen3-VL (32B): IDA = 59.73%, GT-RDA = 89.46% (+29.73 pp)
• Llama-3.2-Vision (90B): IDA = 56.84%, GT-RDA = 91.21% (+34.37 pp)

The pattern is consistent on MIMIC-IV-ECG: Hulu-Med (32B) rises from 51.62% to 97.20%, Qwen3-VL (8B) from 53.53% to 86.19%, and Llama-3.2-Vision (90B) from 52.52% to 88.39%. The paper's interpretation:

"As a baseline observation, excluding ECG-specific models, most evaluated models achieve an IDA of approximately 50%, indicating they are randomly guessing the initial diagnosis. However, for these non-ECG-specific models, providing the ground-truth reasoning trajectory yields the expected result, which is a massive surge in diagnostic accuracy across both datasets. This demonstrates their latent capacity to utilize valid evidence when perfectly guided."

The near-50% IDA for non-ECG-specific models is consistent with the balanced positive/negative sampling design: random guessing on a binary diagnostic question would yield 50% accuracy. These models have not learned ECG-specific shortcuts, so their unaided performance is at chance level. But when provided with explicit reasoning — "here are the findings, here is where they appear, here are their measurements" — they can integrate this information and make the correct diagnostic decision. This is evidence of genuine (if latent) reasoning capacity: the models understand clinical logic and can apply it when given the evidence.

ECG-specific models show the opposite pattern: GT-RDA collapses below IDA. This is the paper's most important empirical result. On PTB-XL:

• ECG-R1-RL: IDA = 85.41%, GT-RDA = 22.70% (a 62.71 percentage point decrease)
• ECG-R1-SFT: IDA = 85.12%, GT-RDA = 21.76% (−63.36 pp)
• PULSE: IDA = 80.93%, GT-RDA = 35.18% (−45.75 pp)
• GEM: IDA = 84.37%, GT-RDA = 65.66% (−18.71 pp)

On MIMIC-IV-ECG:

• ECG-R1-RL: IDA = 80.17%, GT-RDA = 22.95% (−57.22 pp)
• ECG-R1-SFT: IDA = 79.58%, GT-RDA = 22.12% (−57.46 pp)
• PULSE: IDA = 72.10%, GT-RDA = 30.01% (−42.09 pp)
• GEM: IDA = 76.27%, GT-RDA = 57.01% (−19.26 pp)

The paper's interpretation is direct:

"This unexpected degradation exposes a critical flaw in current ECG-specific MLLMs, in that they have not learned to interpret ECGs through valid clinical reasoning. Instead, they appear to have optimized for a superficial pattern that links global signal patterns directly to high-level diagnostic conclusions."

The mechanism proposed is that these models learned a direct mapping from ECG appearance to diagnostic label during training, bypassing the intermediate reasoning steps that were present in the training data. The synthetic rationales in datasets like ECGInstruct and ECG-Protocol-Guided-Grounding-CoT provided reasoning traces that the model learned to ignore in favor of a shorter path: signal pattern → diagnosis. When the correct reasoning is provided at test time and it points to a different diagnosis than the model's shortcut predicted, the model cannot reconcile the contradiction because its internal representations never encoded the finding-to-diagnosis mapping. The result is worse-than-chance performance under explicit guidance.

GEM is a partial exception: its GT-RDA of 65.66% on PTB-XL is substantially higher than ECG-R1 and PULSE, though still below its IDA of 84.37%. The paper does not discuss this specifically, but GEM's training data (ECG-Grounding) was designed to incorporate more structural grounding than ECGInstruct, which may explain why its reasoning capability, while still impaired, is less catastrophically broken than the other ECG-specific models.

OpenTSLM (3B) shows intermediate behavior: IDA = 54.77% on PTB-XL, GT-RDA = 37.58%. Its IDA is near chance level (unlike the other ECG-specific models), suggesting it did not learn strong shortcuts, but its GT-RDA is also low, suggesting it cannot utilize provided reasoning — likely due to its small parameter capacity (3B) being insufficient for complex multi-turn reasoning, as noted in the Completion analysis.

The GT-RDA collapse is the paper's strongest evidence against the current paradigm of training on LLM-generated rationales. It demonstrates not just that ECG-specific models fail at generating reasoning (which the low Completion rates already showed), but that they fail at comprehending reasoning — a much more fundamental deficit that implicates the training process itself, not just the inference-time strategy.

Proprietary vs. Open Models: Modest Advantages at Scale

The proprietary models (Gemini-3-Flash, Gemini-2.5-Pro, GPT-5.2) generally achieve the highest Depth scores and among the highest Completion rates, but the margins over the best open models are modest. On PTB-XL Completion, Gemini-3-Flash (6.26%) edges out Qwen3-VL (32B) (5.71%) by only 0.55 percentage points. On Depth, Gemini-3-Flash (2.09) leads Qwen3-VL (32B) (1.63) by a more substantial 0.46 points. On IDA, GPT-5.2 (67.80%) and Gemini-3-Flash (65.86%) show notably higher baseline diagnostic accuracy than open models (Qwen3-VL 32B at 59.73%, Llama-3.2-Vision 90B at 56.84%), suggesting that proprietary models have stronger pattern-recognition capabilities even if their reasoning chains are similarly fragile.

Notably, no model — proprietary or open, small or large — achieves a Completion rate above ~6% or a Depth above ~2.1. The ceiling appears to be determined by a capability that current training paradigms fundamentally fail to impart, rather than by model scale. The paper does not explore whether further scaling (e.g., 405B parameter models) would break through this ceiling, but the relatively flat relationship between model size and reasoning metrics within the evaluated range (compare Qwen3-VL 8B vs. 32B, or Llama-3.2-Vision 11B vs. 90B) suggests diminishing returns from scale alone.

Ablation Studies and Robustness Checks

The paper does not contain traditional ablation studies in the sense of removing components from a proposed model and measuring degradation. However, several design choices and comparisons function as implicit ablations that test the robustness of the benchmark's findings:

ECG-specific vs. general-domain models as an ablation of training data influence. The comparison between ECG-specific models (PULSE, GEM, ECG-R1) and general-domain models (Qwen3-VL, Llama-3.2-Vision) functions as an ablation of ECG-specific fine-tuning. The result — that ECG-specific models achieve higher IDA but lower GT-RDA, and comparable Completion — demonstrates that domain-specific fine-tuning on synthetic rationales improves pattern-matching diagnostic accuracy while degrading or failing to improve genuine reasoning capability. This is reported in Table 1.

Model scale within the same family. The paper includes pairs of models at different scales within the same family: Qwen3-VL (8B vs. 32B), Llama-3.2-Vision (11B vs. 90B), Hulu-Med (7B vs. 32B), MedGemma (4B vs. 27B). Across all pairs, the larger model shows modest improvements in Depth and IDA but negligible improvements in Completion. For instance, Qwen3-VL Completion: 5.67% (8B) vs. 5.71% (32B) on PTB-XL. Hulu-Med Completion: 4.96% (7B) vs. 3.21% (32B) on PTB-XL — the larger model actually performs worse on Completion, though better on Depth (1.20 vs. 1.62) and dramatically better on GT-RDA (86.87% vs. 99.42%). This pattern — scale helps knowledge utilization (GT-RDA) but not reasoning chain execution (Completion) — reinforces the paper's core claim that the bottleneck is not model capacity but a fundamental capability gap in visual grounding that scale alone does not address.

Temperature=0 deterministic decoding as an implicit ablation of stochastic generation. The paper configures all models with temperature=0, producing deterministic outputs. This contrasts with the stochastic sampling used in many MLLM evaluations and in the best-of-N paradigm critiqued in the paper's related work. While not presented as an ablation, this choice implicitly tests whether models' most likely responses contain genuine reasoning (they don't, as shown by the Completion rates) — a stronger test than whether reasoning can sometimes emerge from stochastic sampling. The paper does not report results with non-zero temperature, which would be informative: if models could occasionally produce correct reasoning chains via sampling, it would suggest the capability exists latently but is not expressed in the model's preferred output.

Semantic consistency checking via Gemini-3-Flash as an implicit robustness check. The paper uses Gemini-3-Flash to verify semantic equivalence between model responses and ground-truth answers, rather than rigid string matching. This is a robustness design choice rather than an ablation: it ensures that models are not penalized for phrasing correct answers differently than the reference. However, the paper does not report the accuracy of this checker, either by measuring its agreement with human judgment on a subset of responses or by comparing against string-matching baselines. If the semantic checker has systematic errors (e.g., accepting incorrect answers as semantically equivalent), those errors would propagate into all reported metrics. This is a significant limitation.

Multi-lead consensus alignment and P-wave recovery in the automated pipeline as design-level robustness. The pipeline's post-processing steps (Section 3.1, Appendix A.1) function as implicit ablations of raw deep learning segmentation. The paper reports (Table 2) that the raw U-Net3+ achieves P-wave recall of 0.972 and precision of 0.942, while the full pipeline with post-processing achieves recall of 0.978 and precision of 0.934. The precision decrease is explained as an artifact of detecting unannotated P waves in LUDB, not genuine errors. This demonstrates that the pipeline's design choices improve detection of clinically important features (non-conducted P waves) even when strict evaluation against incomplete annotations penalizes it.

Consensus filtering as a data quality ablation. The paper implicitly ablates the effect of pipeline errors by filtering the benchmark to include only samples where the automated pipeline's diagnosis matches the human-provided label (Section 4.2). Without this filter, pipeline errors would contaminate the ground truth and potentially create "impossible" reasoning chains where the ground-truth findings don't actually support the ground-truth diagnosis. The paper does not report what fraction of samples were excluded by this filter, nor does it analyze whether excluded samples differ systematically from included ones — both would be informative for understanding potential selection bias.

Dual-distractor types in Criterion Selection as a diagnostic ablation. The paper describes (Section 4.1) but does not separately evaluate the two distractor types (category-based and presence-based). An ablation analyzing model performance on questions with each distractor type would reveal whether models fail because they don't know the criteria (category-based distractors) or because they can't ignore irrelevant findings (presence-based distractors). This analysis is not present in the paper but would strengthen the diagnostic granularity of the Depth metric.

Critical Assessment

Claim 1: Current MLLMs exhibit near-zero success rates in maintaining complete reasoning chains.

This claim is strongly supported by the Completion metric results in Table 1, with important qualifications about what "complete reasoning chain" means in this benchmark.

The maximum Completion rate across all models and datasets is 6.26% (Gemini-3-Flash on PTB-XL), which is unambiguously a near-zero success rate for a task that requires ~8 correct sequential decisions. The finding is consistent across both datasets (PTB-XL and MIMIC-IV-ECG) and across model categories (ECG-specific, medical, general, proprietary), with all models clustering in a narrow band from ~0% to ~6%. This rules out the possibility that the failure is specific to a particular model architecture, training procedure, or dataset.

However, the Completion metric's stringency deserves careful examination. Completion requires flawless execution of the entire reasoning chain — a single error at any step flips the sample from success to failure. This is a valid clinical standard (one reasoning error can lead to misdiagnosis), but it makes Completion highly sensitive to per-step error rates in a way that may overstate the practical severity of the problem. For example, a model that achieves a per-step accuracy of 80% on 8-step chains would have a Completion rate of only ~17%. If per-step accuracy were 70%, Completion would be ~5.7% — matching the observed ceiling. This means the Completion metric cannot distinguish between a model that makes one error per chain (which might still arrive at the correct diagnosis through other reasoning paths) and a model that makes multiple errors per chain (which is genuinely dangerous). The Depth metric partially addresses this by tracking where failures occur, but it does not quantify how many errors accumulate per chain.

A missing analysis that would clarify this: reporting the distribution of the number of correct steps per chain, or equivalently, the Completion rate for chains of different lengths (different diagnoses have different numbers of required findings, as shown in the logic diagrams). If models perform better on shorter chains, it would suggest that chain length is a meaningful variable and that diagnoses with fewer required findings are less affected by the reasoning gap.

Claim 2: Models possess the medical knowledge to select appropriate criteria (Criterion Selection) but fail at grounding knowledge to visual evidence (Finding Identification and ECG Grounding).

This claim is supported by the Depth metric in Table 1 but with limited per-stage granularity in the reported results.

The Depth scores in Table 1 cluster around 1.0–2.0, which the paper interprets as evidence of success at Stage 1 (Criterion Selection) and failure at Stages 2–3 (Finding Identification and ECG Grounding). The interpretation is reasonable: a Depth of ~1.5 means the model can, on average, complete Criterion Selection and get partly through Finding Identification but rarely progresses into ECG Grounding.

However, the paper does not report per-stage accuracy rates — what fraction of Criterion Selection questions are answered correctly, what fraction of Finding Identification questions are answered correctly, etc. — which would provide a more direct test of the claim. The Depth metric aggregates across stages with fractional scoring, making it somewhat opaque. For example, a Depth of 1.79 (ECG-R1-RL on PTB-XL) could mean the model consistently passes Criterion Selection (~100%), passes Finding Identification ~79% of the time, and never passes ECG Grounding — or it could mean more complex patterns where the model sometimes passes all three stages but more frequently fails at Stage 1. Without per-stage breakdowns, the claim that "models succeed at knowledge retrieval but fail at perception" is plausible but not directly verified by the reported numbers.

The Depth metric's recovery mechanism (injecting ground-truth history after failures) is methodologically sound for isolating per-finding performance, but it introduces a subtle confound: the model is receiving more information in later loops (the injected history of earlier findings) than it would in a natural reasoning scenario. If models perform better on later loops because the injected history provides useful context, Depth scores may overestimate natural reasoning capability. Conversely, if the injected history confuses models (as the GT-RDA collapse suggests for ECG-specific models), Depth may underestimate it. The paper does not analyze whether Depth varies systematically by loop position within a diagnosis.

Claim 3: ECG-specific models exhibit a performance collapse under explicit reasoning guidance (GT-RDA), revealing reliance on superficial pattern matching.

This claim is the strongest and most convincingly supported finding in the paper, but the interpretation deserves scrutiny.

The GT-RDA collapse is dramatic and consistent: ECG-R1-RL drops from 85.41% to 22.70% on PTB-XL, and from 80.17% to 22.95% on MIMIC-IV-ECG. PULSE drops from 80.93% to 35.18% (PTB-XL) and 72.10% to 30.01% (MIMIC-IV-ECG). These are large, robust effects that replicate across datasets and model versions (both SFT and RL variants of ECG-R1 show the collapse). The asymmetry with non-ECG-specific models, which show the expected IDA-to-GT-RDA increase, strengthens the causal interpretation that ECG-specific training on synthetic rationales is responsible for the deficit.

The paper's interpretation — that these models learned direct signal-to-diagnosis shortcuts and cannot integrate explicit reasoning — is the most parsimonious explanation consistent with the data. However, alternative explanations are not explored:

• Distribution shift in the GT-RDA prompt format. The GT-RDA condition presents the model with a long context containing the full reasoning history (findings, lead localizations, wave segments, measurements) followed by the diagnostic question. This context format is likely different from anything the model saw during training, especially for ECG-specific models trained on specific instruction formats. The collapse could reflect poor out-of-distribution generalization rather than lack of reasoning capability. A useful control would be to test models on GT-RDA with ablated reasoning histories of varying lengths — if performance degrades monotonically with context length, the issue may be context processing rather than reasoning integration.

• The benchmark's reasoning chain format may be incompatible with the models' training format. ECG-R1 was trained on a 6-step analysis protocol (as stated in Appendix C.2: "Reinforcement Learning to enforce a 6-step analysis protocol for structured ECG interpretation"). The benchmark's 4-stage verification loop with multiple-choice questions presents a different interaction format. The model may be failing at GT-RDA because it expects to generate reasoning, not read it, or because the multiple-choice format conflicts with its training. This would not rescue the model — it would still mean the model cannot utilize provided reasoning — but it would localize the failure to format incompatibility rather than fundamental reasoning incapacity.

• GEM's partial exception is informative. GEM shows a smaller GT-RDA drop (84.37% → 65.66% on PTB-XL, 76.27% → 57.01% on MIMIC-IV-ECG) than ECG-R1 or PULSE. GEM's training data (ECG-Grounding) was designed with more structural grounding than ECGInstruct or ECG-Protocol-Guided-Grounding-CoT. This suggests that the severity of the GT-RDA collapse correlates with the degree of synthetic vs. grounded training data, providing a gradient of evidence that supports the paper's interpretation. However, the paper does not analyze this gradient explicitly.

Claim 4: The automated ECG analysis pipeline provides objective, verifiable ground truth at scale.

This claim is partially supported by the LUDB evaluation (Table 2) and the expert validation (143 samples), but the paper provides limited quantitative evidence about the pipeline's end-to-end accuracy on the benchmark datasets.

The LUDB evaluation (Table 2) demonstrates strong waveform detection performance: QRS recall/precision of 1.000/1.000, P-wave recall of 0.978/0.979, T-wave recall of 0.997/0.995. These numbers justify confidence in the pipeline's segmentation capability on a held-out test set. However, LUDB is a specific dataset of 200 recordings with expert annotations, and it is not guaranteed that the pipeline generalizes equally well to PTB-XL and MIMIC-IV-ECG, which may have different signal characteristics, noise levels, or lead configurations. The paper does not report any evaluation of the pipeline on samples from its target datasets.

The expert validation of 143 samples by three board-certified specialists provides clinical credibility, but the paper does not report quantitative results from this validation — no inter-rater agreement statistics, no error rate of the pipeline's reasoning chains relative to expert judgment, and no analysis of disagreement cases. Without these, the expert validation functions more as a quality assurance narrative than as a rigorous validation study. A reader cannot assess whether the pipeline's errors (which certainly exist, as with any automated system) are systematic, random, or concentrated in particular diagnoses.

The consensus filtering (requiring pipeline diagnosis to match human label) provides a form of implicit error control but introduces potential selection bias. Cases where the pipeline and human label agree may be systematically easier or more prototypical than cases of disagreement. If models also perform better on easier cases, the benchmark may overestimate reasoning capability relative to a deployment scenario with more challenging ECGs. The paper does not report the agreement rate between pipeline and human labels, nor does it characterize excluded samples.

Missing Experiments That Would Strengthen the Paper

Per-stage accuracy breakdowns. Reporting accuracy separately for Criterion Selection, Finding Identification, each ECG Grounding sub-task (lead, wave, measurement), and Diagnostic Decision would provide direct evidence for the paper's central claim about where models fail, without requiring interpretation of the composite Depth metric.

Per-diagnosis analysis. The 17 diagnoses vary substantially in complexity (number of required findings, types of grounding required, measurement vs. qualitative criteria). Reporting Completion and Depth per diagnosis would reveal whether certain diagnostic categories are disproportionately responsible for the low aggregate scores and whether the reasoning gap is uniform or concentrated.

Confidence interval reporting. With ~100 samples per diagnosis and sample sizes of 3,076 (PTB-XL) and 3,355 (MIMIC-IV-ECG) for aggregate metrics, binomial confidence intervals for Completion rates would help assess whether differences between models (e.g., Gemini-3-Flash at 6.26% vs. GPT-5.2 at 5.84%) are statistically reliable or within sampling noise. The absence of any uncertainty quantification limits the interpretability of model comparisons.

Non-zero temperature evaluation. All experiments use temperature=0, which tests the model's most likely response. Evaluating with non-zero temperature and multiple samples would test whether reasoning capability exists latently and can be surfaced through stochastic decoding. If some samples yield correct reasoning chains under sampling (even if the model's preferred output is wrong), it would change the interpretation from "models cannot reason" to "models do not reliably express their reasoning capability."

Synthetic rationale ablation for non-ECG-specific models. To strengthen the causal claim that training on synthetic rationales degrades reasoning, the paper could evaluate a general-domain model (e.g., Qwen3-VL) before and after fine-tuning on ECGInstruct-style data, measuring whether GT-RDA decreases. This would provide direct experimental evidence for the training paradigm's causal role rather than relying on cross-model comparisons.

Semantic checker accuracy. Reporting the agreement between Gemini-3-Flash's semantic equivalence judgments and human judgments on a subset of model responses would establish the reliability of the evaluation pipeline. If the checker has systematic biases (e.g., accepting fluent but incorrect answers), those biases could inflate or deflate reported metrics.

Despite these limitations, the paper's experimental results are internally consistent, replicate across two independent datasets, and produce an effect (GT-RDA collapse) that is large enough to be robust to plausible measurement error. The core finding — that current MLLMs cannot execute multi-step visually grounded reasoning on ECGs — is well-supported by the reported metrics, even with the acknowledged gaps in granularity and statistical rigor.

6. Limitations and Trade-offs

The Cost of Difficulty Estimation Is Unaccounted For

The assumption or constraint. The benchmark construction depends on an automated analysis pipeline that produces ground-truth reasoning chains — waveform boundaries, quantitative measurements, discrete findings, and diagnostic conclusions — for every sample in the dataset. The pipeline as described in Section 3 and Appendix A.1 is a complete offline system: U-Net3+ segmentation, multi-lead consensus alignment, P-wave template matching, feature extraction, finding mapping, and diagnosis derivation via 17 expert-validated logic diagrams. The paper acknowledges that the pipeline is not intended to replace human interpretation but does not quantitively characterize its failure modes on the target datasets (PTB-XL and MIMIC-IV-ECG) as opposed to the held-out LUDB evaluation set. The paper's expert validation of 143 representative samples (Section 4.2) establishes plausibility but is not a systematic error analysis:

"three board-certified internal medicine specialists validated 143 representative samples to establish a reliable baseline for data quality... Following this expert verification, all authors manually reviewed the extracted reasoning path for every single sample, under the supervision of the specialists to ensure the dataset integrity."

The consequence. A practitioner using this benchmark to evaluate their ECG-MLLM is trusting that the ground-truth reasoning chains are clinically correct. If the automated pipeline makes systematic errors — for example, misclassifying a borderline QRS duration as prolonged when it falls just above threshold, or missing subtle T-wave inversions in a noisy lead — then a model that correctly identifies the actual finding would be marked wrong, artificially depressing Completion and Depth scores. Conversely, if the pipeline hallucinates findings that are not visually present, the ground-truth reasoning chain would demand that the model "ground" evidence that does not exist, creating an impossible task. Either direction of pipeline error biases the benchmark against the model being evaluated. The paper's consensus filtering (requiring pipeline diagnosis to match human labels) controls for errors that change the final diagnosis but does not control for errors in intermediate findings — a pipeline could arrive at the correct diagnosis through wrong findings (compensating errors), or could specify correct findings with wrong localizations, and neither would be caught by the filtering criterion.

What evidence exists in the paper. The LUDB evaluation (Table 2, Appendix A.1) reports strong waveform detection performance: QRS recall/precision of 1.000, P-wave recall ~0.978 and precision ~0.937, T-wave recall ~0.996 and precision ~0.992. However, LUDB is a specific dataset of 200 recordings, and the paper provides no quantitative evaluation on PTB-XL or MIMIC-IV-ECG — the actual datasets from which the benchmark is constructed. The precision drop for P-waves (0.942 raw U-Net3+ → 0.934 with post-processing) is explained as an artifact of unannotated P-waves in LUDB, but this explanation cannot be verified on the target datasets without similar expert annotations. The paper also does not report: (a) what fraction of samples were excluded by the consensus filter, (b) the agreement rate between pipeline and human labels per diagnosis, (c) inter-rater agreement statistics from the expert validation, or (d) any analysis of systematic pipeline errors (e.g., does it systematically under- or over-estimate QRS duration in certain lead configurations?). Without these, a practitioner cannot calibrate how much of a model's low Completion score reflects model failure versus ground-truth noise.

Mitigation status. The paper partially addresses this through the consensus filter and expert validation, but does not quantify the pipeline's error rate on the benchmark datasets themselves. The authors acknowledge the validation as establishing a "reliable baseline for data quality" (Section 4.2) but do not frame this as a limitation requiring future work. A user of this benchmark should supplement it with their own spot-checking of pipeline outputs, particularly for diagnoses (like hypertrophy) where quantitative measurement accuracy is critical.

The Benchmark Evaluates Only 17 Diagnoses on Two Datasets from One Modality

The assumption or constraint. The benchmark covers 17 core ECG diagnoses organized into six clinical groups (Section A.2): AV blocks (3), conduction disturbances (4), hypertrophy (2), ectopic beats (2), myocardial infarction (3), and ischemia (3). The paper reports that these 17 diagnoses directly cover 56.05% of PTB-XL samples and 82.03% of MIMIC-IV-ECG samples (Figure 4), and that coverage expands to 76.85% and 96.18% respectively when including indirectly derivable conditions. However, the remaining diagnoses — including clinically important categories like atrial fibrillation, atrial flutter, ventricular tachycardia, electrolyte abnormalities, and many rarer conditions — are not evaluated. Furthermore, the benchmark is constructed exclusively from ECG signals (time-series or rendered images) and does not test reasoning in other medical imaging modalities (chest X-ray, CT, MRI) or non-imaging clinical reasoning tasks.

The consequence. A practitioner who reads this paper and concludes that "current MLLMs cannot perform step-by-step clinical reasoning" must qualify this statement carefully. What the paper demonstrates is that current MLLMs cannot perform step-by-step visually grounded ECG reasoning on 17 specific diagnoses as defined by textbook logic diagrams. The failure mode the paper identifies — models retrieve diagnostic criteria from knowledge but cannot perceptually detect or localize those criteria in the signal — is likely to generalize to other ECG diagnoses that require similar perceptual operations (e.g., detecting P-wave morphology in atrial enlargement, measuring QT intervals in long QT syndrome). However, the paper provides no evidence about whether the failure mode generalizes to other medical domains (radiology, pathology) or to non-perceptual reasoning tasks (differential diagnosis from lab values, treatment planning from clinical guidelines). A model that fails at visually grounding ECG findings might succeed at reasoning tasks that do not require perceptual grounding, or it might fail at perceptual grounding in chest X-rays in different ways. The single-modality, single-task-family scope means the paper's conclusions cannot safely be extrapolated to clinical AI reasoning in general.

Additionally, the coverage statistics should not be overinterpreted. That the 17 core diagnoses cover 82% of MIMIC-IV-ECG samples means that 82% of ECGs in that dataset have at least one of these 17 diagnoses in their label set. It does not mean that the benchmark evaluates 82% of the reasoning that a deployed ECG AI would need to perform — many ECGs have multiple co-occurring diagnoses, and reasoning in the presence of comorbidities (e.g., left bundle branch block with superimposed ischemia) requires more complex logic than the single-diagnosis evaluation the benchmark performs. The logic diagrams (Figures 6–21) encode decision trees for individual diagnoses but do not specify how to reason when findings are consistent with multiple diagnoses simultaneously.

What evidence exists in the paper. The single-diagnosis scope is explicit in the benchmark construction (Section 4.2): "we curated a balanced set of 100 positive and 100 negative samples for each of the 17 core diagnoses." The prevalence analysis (Figure 4) quantifies coverage of diagnostic labels but does not address comorbidity or reasoning complexity. There is no evaluation on other modalities, other ECG diagnoses, or other clinical reasoning tasks. This limitation is not discussed in the paper's Discussion (Section 6), which focuses instead on diagnostic uncertainty and structured verification vs. clinical heuristics.

Mitigation status. Not addressed. The paper presents the benchmark as comprehensive within its scope but does not acknowledge the scope limitation or discuss generalization to other diagnoses, other modalities, or comorbid presentations. Future work on extending the logic diagrams to additional diagnoses or on evaluating reasoning in multi-diagnosis contexts would directly address this gap.

The Benchmark Cannot Distinguish Model Failure from Format Incompatibility

The assumption or constraint. The evaluation protocol uses a highly structured, multi-turn multiple-choice format: the model is asked a series of questions with pre-specified options, must select from those options at each turn, and is evaluated by a semantic consistency checker (Gemini-3-Flash) that compares the model's selection to the ground-truth answer. The system prompt (Figure 5, Appendix C.3) instructs the model to adopt an expert cardiologist persona and follow textbook diagnostic criteria. The paper configures all models with temperature=0 for deterministic output (Appendix C.2). This rigid format is deliberately chosen to enable objective, reproducible scoring — the authors correctly identify that free-text generation would require subjective evaluation — but it introduces a format confound: a model might possess the capability to reason about an ECG but fail to express that reasoning correctly in the specific multiple-choice interaction format the benchmark demands.

The consequence. Consider two different failure scenarios for a model that scores low Completion on the benchmark. In Scenario A, the model genuinely cannot perceive the relevant ECG features — it looks at a prolonged QRS complex and cannot determine that the duration exceeds 120 ms. In Scenario B, the model can perceive the feature and would correctly describe it in a free-text report ("QRS duration is approximately 155 ms, consistent with prolongation"), but fails in the benchmark because: (a) the measurement grounding question offers discrete bins (e.g., "150ms–160ms") and the model's internal estimate is 148 ms (just below the bin boundary), (b) the wave grounding question asks it to select a temporal segment from a list of options and the model's attention is drawn to a different cardiac cycle that also shows the finding, or (c) the multi-turn format causes the model to lose track of which finding is currently being verified, and it selects an option that would be correct for a different step of the reasoning chain. In Scenario A, the model is clinically unreliable. In Scenario B, the model's reasoning capability is real but its interface with the evaluation format is broken. The benchmark would assign identically low scores to both scenarios, and the paper's interpretation ("models lack the crucial ability to reliably ground that logic to the granular visual evidence") would be valid for Scenario A but misleading for Scenario B.

This concern is amplified by the GT-RDA collapse in ECG-specific models (Table 1). The paper interprets the collapse as evidence that these models never learned clinical reasoning and instead rely on shortcuts. An alternative interpretation is that these models did learn clinical reasoning but in a format (free-text chain-of-thought generation) that is incompatible with the benchmark's multiple-choice interaction. When presented with the ground-truth reasoning in the benchmark's format, the model cannot map it onto its internal reasoning representations, leading to worse-than-chance performance. This alternative hypothesis may be less parsimonious than the shortcut-learning explanation, but the paper provides no evidence to distinguish between them.

What evidence exists in the paper. The paper does not include any ablation that tests format sensitivity: no comparison of multiple-choice vs. free-text evaluation, no experiment varying the number of options or the phrasing of distractor choices, and no analysis of whether errors cluster at particular question formats (e.g., do models fail more on measurement grounding with numeric bins than on lead grounding with lead-name options?). The qualitative review mentioned for small models ("these compact models frequently lose contextual focus during the multi-turn verification loops, generating irrelevant or repetitive responses") provides anecdotal evidence of format-related failure but does not quantify how much of the Completion deficit is attributable to format vs. capability. The decision to use temperature=0 means the paper tests only the model's single most likely response; evaluation with multiple samples per question (which could reveal whether correct answers are present in the model's distribution but not in its argmax) is not performed.

Mitigation status. Not addressed. The paper's design choices (multiple-choice, temperature=0, system prompt) are well-justified for reproducibility and objectivity, but the absence of format sensitivity analysis means a practitioner cannot determine whether a low Completion score indicates fundamental reasoning incapacity or interface incompatibility. Future work that compares benchmark performance against free-text expert evaluation on a subset of samples would help calibrate this confound.

The Benchmark Excludes Diagnostic Uncertainty and Clinical Heuristics

The assumption or constraint. The benchmark's logic diagrams (Appendix A.2, Figures 6–21) encode binary decision trees: each finding is either present or absent, each measurement either exceeds a threshold or does not, and the diagnosis is either confirmed or excluded. The paper explicitly acknowledges this limitation in the Discussion (Section 6):

"In real-world clinical practice, ECG interpretation inherently involves a degree of diagnostic uncertainty and ambiguity. However, to establish an indisputable ground truth for our benchmark, we intentionally excluded borderline or ambiguous samples through rigorous manual review to ensure the absolute reliability of the evaluation metrics."

Additionally, the benchmark enforces a strict sequential verification process: every required finding must be explicitly verified before a diagnosis can be confirmed, and early termination (making a diagnosis after a subset of findings) is penalized even if clinically justifiable. The paper acknowledges this as well:

"In practice, physicians frequently rely on clinical heuristics rather than exhaustively verifying every single criterion. For instance, if a diagnosis requires three distinct findings, but one finding presents with absolute certainty, a clinician might confidently confirm the diagnosis without explicitly evaluating the remaining findings."

The consequence. A model evaluated on this benchmark is being tested on a sanitized version of clinical reasoning — one where findings are unambiguous, thresholds are crisp, and the required evidence is specified in advance. This is a valid test of logical deduction (can the model chain criteria together correctly?) but not a valid test of clinical judgment (can the model recognize when evidence is sufficient despite ambiguity, integrate conflicting findings, or express appropriate diagnostic doubt?). A model that scores perfectly on this benchmark might still be clinically unsafe if it cannot handle borderline cases — for example, confidently diagnosing left ventricular hypertrophy when voltage criteria are barely met and the patient is young and thin (where high voltage can be normal), or failing to recognize that a "prolonged PR interval" of 205 ms in an otherwise healthy young adult is clinically insignificant despite meeting the textbook threshold.

The exclusion of borderline cases through "rigorous manual review" also means the benchmark's ground truth is constructed from prototypical presentations. If future models improve on perceptual grounding and achieve higher Completion scores, the next bottleneck will be handling ambiguous findings — but the benchmark provides no signal about model performance in that regime. A practitioner deploying an ECG-MLLM in a real clinical setting, where borderline and ambiguous ECGs are common (especially in emergency departments and outpatient screening), cannot use this benchmark to assess the model's behavior on those cases.

The decision to penalize heuristic shortcuts — where a model makes a confident diagnosis before verifying all findings — is a deliberate evaluation choice that the paper defends as necessary for testing systematic reasoning. However, it creates a potential validity-efficiency tradeoff: a model that reasons like an experienced clinician (recognizing a pathognomonic finding and stopping) appears worse on this benchmark than a model that mechanically verifies every criterion, even though the former may be more clinically efficient and equally accurate. The paper's proposed future work on "dynamic reasoning diagrams" with "severity and certainty weightings" and "early termination evaluation scheme" (Section 6) acknowledges this limitation but does not resolve it in the current benchmark.

What evidence exists in the paper. The paper's Table 1 reports models' Completion and Depth without distinguishing between failure modes caused by perceptual errors versus those caused by heuristic reasoning that the benchmark penalizes. The qualitative review of model responses (mentioned for small models losing "contextual focus") does not analyze whether any model errors were clinically reasonable heuristic shortcuts rather than genuine mistakes. The expert validation of 143 samples (Section 4.2) establishes that the ground-truth reasoning chains are clinically valid for the included samples but does not characterize what fraction of real-world ECGs would be excluded as borderline or ambiguous.

Mitigation status. Partially addressed through explicit acknowledgment and future work proposals in the Discussion (Section 6). The paper plans to "formalize the definition of uncertainty, allowing for the reintroduction of borderline cases" and to "evolve our static reasoning pathways into dynamic reasoning diagrams by incorporating severity and certainty weightings." These are sensible directions but are not implemented in the current benchmark. A practitioner using the benchmark should understand that it evaluates rigorous deductive reasoning under idealized conditions, not clinical judgment under uncertainty.

No Measurement of Inter-Rater Reliability or Semantic Checker Accuracy

The assumption or constraint. Two critical components of the evaluation pipeline rely on judgments whose reliability is not quantified. First, the ground-truth reasoning chains are derived from logic diagrams that were "validated by three board-certified internal medicine specialists" (Section 3.2), but the paper reports no quantitative measure of this validation: no inter-rater agreement statistics, no rate of disagreement between specialists and pipeline, and no characterization of which diagnoses or finding types generated disagreements. Second, the evaluation uses Gemini-3-Flash as a semantic consistency checker to verify whether model responses match ground-truth answers (Section 5.1), but the paper reports no accuracy measurement for this checker — no comparison against human judgment on a subset of model responses, no analysis of checker errors (false positives where semantically different answers are accepted, false negatives where semantically equivalent answers are rejected), and no assessment of whether checker performance varies by question type (multiple-choice single-select vs. multi-select vs. yes/no).

The consequence. The paper's headline metrics — Completion, Depth, IDA, GT-RDA — are the product of a measurement pipeline whose error characteristics are unknown. If the semantic checker has a false-positive rate of even 5% (accepting an incorrect model response as equivalent to the ground truth), the reported Completion rates of ~5–6% could be inflated — a model whose true Completion is near zero could appear at 5% due to checker errors alone. Conversely, if the checker has a false-negative rate of 5% (rejecting a correct model response), the reported Completion rates could be underestimates. The fact that the maximum Completion across all models is ~6% means that even small checker error rates have large relative effects on the metrics.

The absence of inter-rater reliability data for the expert validation of logic diagrams is similarly consequential. If the three specialists disagreed on, say, 10% of findings across the 143 reviewed samples, then even the "indisputable ground truth" has an irreducible uncertainty of ~10% — and a model that achieves 90% Completion would be performing at the level of expert agreement. If specialist agreement was perfect (0% disagreement), that would strengthen confidence that the ground truth is genuinely unambiguous. The paper provides neither number, making it impossible to calibrate the benchmark's ceiling.

The semantic checker reliability concern is particularly acute for the benchmark's multi-select grounding questions (lead grounding, wave grounding) and measurement range selection, where the model must select one or more options and semantic equivalence is less straightforward than for binary yes/no questions. A model that selects three of four correct leads might be semantically close to correct (it found most of the evidence) but the checker would need to decide whether "close" counts as equivalent — a decision that could swing the grounding sub-task scores that feed into the Depth metric.

What evidence exists in the paper. The paper provides no quantitative evaluation of semantic checker accuracy and no inter-rater reliability statistics for the expert validation. The description of expert validation (Section 4.2) states that "three board-certified internal medicine specialists validated 143 representative samples" and "all authors manually reviewed the extracted reasoning path for every single sample... under the supervision of the specialists to ensure the dataset integrity." The description of the semantic checker (Section 5.1) states only that "we employ Gemini-3-Flash to verify the semantic consistency between the models' response and the ground truth answer at every step." Neither description includes quantitative reliability measures.

Mitigation status. Not addressed. The paper relies on the clinical authority of the specialist validators and the general capability of Gemini-3-Flash as a language model without empirically verifying either. A minimal mitigation would be to report human-checker agreement on a random subset of model responses (e.g., 200 QA pairs), establishing an upper bound on checker-induced metric error. The paper's future work discussion (Section 6) does not mention improving or validating the semantic checker, suggesting this limitation may not have been recognized as significant by the authors. A practitioner relying on these metrics should conduct their own spot-checking of semantic checker decisions, particularly for open-ended or multi-select questions where equivalence judgments are most subjective.

The Benchmark Only Tests Single-Diagnosis Reasoning, Not Comorbid Presentations

The assumption or constraint. The benchmark is constructed as a set of independent per-diagnosis evaluations: for each of the 17 diagnoses, 100 positive and 100 negative samples are selected, where a "positive" sample means the ECG exhibits that specific diagnosis and a "negative" sample means it does not (Section 4.2). The logic diagrams (Figures 6–21) encode reasoning for individual diagnoses in isolation. The evaluation protocol asks the Initial Diagnostic Question for a single target diagnosis (e.g., "Does this ECG suggest the presence of complete left bundle branch block?"), and the subsequent verification loops test only findings relevant to that diagnosis. The paper does not evaluate scenarios where an ECG exhibits multiple diagnoses simultaneously, where the findings for one diagnosis might mimic, mask, or contradict the findings for another.

The consequence. Real-world ECGs frequently exhibit multiple abnormalities that interact in clinically significant ways. A patient with an inferior myocardial infarction may also have first-degree AV block (due to ischemia affecting the AV node). A patient with left ventricular hypertrophy may also have left anterior fascicular block (the hypertrophy voltage criteria may be met, and the axis deviation may be present, but the combination changes the diagnostic interpretation). A patient with complete left bundle branch block presents with ST-segment and T-wave changes that are "secondary" to the conduction abnormality — but if the same patient also has acute ischemia, the ST changes become diagnostically ambiguous, and the reasoning required to disentangle primary from secondary changes is substantially more complex than either diagnosis in isolation.

The single-diagnosis benchmark design means that a model that performs well on Completion and Depth might still fail catastrophically when presented with a comorbid ECG, because: (a) the model must identify which of multiple concurrent abnormalities are present, (b) it must determine which findings are attributable to which diagnosis, and (c) it must recognize when one diagnosis modifies or invalidates the criteria for another (e.g., LBBB makes the diagnosis of LVH by voltage criteria unreliable, and makes ST-segment analysis for ischemia nearly impossible by standard criteria). None of these skills are tested by the current benchmark.

The paper's logic diagrams do include some cross-diagnosis checks — for example, the LAFB diagram (Figure 11) includes an explicit check for left bundle branch block before confirming LAFB, because LBBB distorts the QRS axis. But this is an exclusion criterion (if LBBB is present, abort the LAFB diagnosis) rather than a comorbidity reasoning test — the benchmark would present an ECG with LBBB as a negative sample for LAFB, not as a sample requiring the model to explain why LAFB cannot be diagnosed in the presence of LBBB.

What evidence exists in the paper. The benchmark construction (Section 4.2, Table 4) explicitly describes a balanced sampling of 100 positive and 100 negative cases per diagnosis, with uniform coverage across reasoning paths within each diagnosis. There is no mention of comorbid sampling, no analysis of how frequently the selected ECGs have multiple diagnoses, and no evaluation of model performance on multi-diagnosis cases. The prevalence analysis (Figure 4) reports per-diagnosis label frequencies but does not report co-occurrence statistics. The paper's Discussion (Section 6) focuses on uncertainty and heuristics but does not mention comorbidity as a limitation.

Mitigation status. Not addressed. The benchmark's single-diagnosis design is a reasonable starting point for establishing baseline reasoning capability — a model that cannot handle single diagnoses in isolation certainly cannot handle comorbid presentations — but a practitioner deploying an ECG-MLLM should recognize that success on this benchmark is a necessary but not sufficient condition for clinical reliability. The benchmark provides no signal about a model's ability to reason about interacting diagnoses, which is a core requirement for real-world ECG interpretation. Future benchmark extensions that include multi-label ECGs with ground-truth reasoning chains that explicitly model diagnostic interactions would address this gap.

7. Implications and Future Directions

How This Work Changes the Landscape

A paradigm shift in evaluation philosophy for medical AI reasoning. This paper is not an incremental improvement to ECG-MLLM evaluation — it is a fundamental reframing of what it means to test whether a model "reasons." The dominant evaluation paradigm in the field — LLM-as-a-Judge comparing generated text against synthetic references — is revealed as structurally incapable of detecting whether a model's diagnostic conclusions are grounded in signal evidence or merely reflect learned linguistic correlations. The paper's central finding — that models with 85% diagnostic accuracy collapse to 23% when asked to use correct reasoning — is a direct empirical demonstration that fluency-based evaluation creates a dangerous illusion of competence. This is not a subtle methodological concern; it is evidence that current evaluation frameworks are answering the wrong question (does the output sound right? rather than: is the output derived from the signal?).

The shift is from plausibility-based evaluation (does the model produce text that a human evaluator or judge-LLM finds medically convincing?) to evidence-grounding evaluation (can the model demonstrate that its conclusions are causally connected to specific features in the input data?). This reframing has implications beyond ECGs: any domain where MLLMs are trained on synthetic rationales and evaluated on text similarity — radiology, pathology, legal reasoning, scientific QA — faces the same risk that high fluency scores mask zero grounding capability. The paper provides both a cautionary tale (the GT-RDA collapse is the smoking gun) and a methodological template (multi-turn verification with spatial-temporal grounding requirements) for conducting genuine reasoning evaluations.

Reconciling the contradiction between high reported accuracy and clinical distrust. The paper provides a parsimonious explanation for a tension that has plagued automated ECG interpretation: why do deep learning models achieve cardiologist-level classification accuracy (Pyakillya et al., 2017; Liu et al., 2021) yet fail to gain clinical adoption? The answer, per this paper's findings, is that high accuracy scores measure pattern-matching capability (can the model map global ECG appearance to diagnostic labels?) while clinicians demand justified accuracy (can the model explain what specific findings support its conclusion and where in the signal those findings appear?). The paper's decomposition — models succeed at Criterion Selection (Depth consistently >1.0) but fail at Finding Identification and ECG Grounding (Depth rarely exceeding 2.0) — reveals that current ECG-MLLMs possess declarative knowledge without perceptual grounding. They know that prolonged QRS duration is required for bundle branch block, but they cannot determine whether a specific ECG exhibits it. This explains why clinicians remain skeptical despite impressive published accuracy numbers: the models cannot provide the evidence trail that clinical decision-making requires.

Redirecting research priorities from knowledge acquisition to perceptual grounding. Prior to this work, the dominant approach to improving ECG-MLLMs was to enrich the training data with more elaborate reasoning traces — longer chain-of-thought, more detailed explanations, protocol-guided structures (as in ECG-R1's 6-step protocol). The paper's results suggest this approach is counterproductive. ECG-specific models trained on the most elaborate reasoning datasets (ECG-R1, trained on Protocol-Guided-Grounding-CoT with reinforcement learning) show the worst GT-RDA collapse: 85.41% IDA → 22.70% GT-RDA on PTB-XL. This implies that training on synthetic rationales actually degrades the model's ability to utilize genuine reasoning, presumably by teaching it to bypass the reasoning content in favor of signal-to-label shortcuts. The implication is clear: the research priority should shift from generating better text rationales to developing training methods that force models to ground their reasoning in perceptual operations — explicit waveform segmentation, interval measurement, lead localization — rather than in text patterns.

The paper also implicitly redirects attention from model scale to training methodology. The near-flat relationship between model size and Completion/Depth within model families (Qwen3-VL 8B vs. 32B: Completion 5.67% vs. 5.71%; Llama-3.2-Vision 11B vs. 90B: 0.49% vs. 4.47% — the only substantial gain) and the absolute ceiling at ~6% Completion regardless of scale suggest that making models larger yields diminishing returns on reasoning capability. The bottleneck is not capacity but a training paradigm that never teaches models to connect perceptual observations to diagnostic conclusions. This finding makes research on training methodology (explicit visual grounding supervision, multi-task learning with segmentation objectives, reinforcement learning with grounding rewards) more attractive, and research on simply scaling up existing synthetic-rationale pipelines less attractive.

Establishing a new standard for what "explainable medical AI" must demonstrate. The paper's ECG Grounding requirement — that models must specify which leads, which waveform segments, and which measurement ranges contain the evidence for each claimed finding — sets a concrete bar for what constitutes a genuine explanation in medical AI. This is a departure from the prevalent notion of "explainability" as post-hoc attention maps or free-text justifications. Attention maps can highlight regions the model looked at but cannot verify whether the model correctly interpreted what it saw. Free-text justifications can be fluent but hallucinated. The paper demands spatial-temporal localization with verifiable correctness: if the model says "notched R waves are present in leads I and aVL at [1.45s – 1.61s]," the benchmark checks this against objective waveform annotations. This operationalizes explainability as evidentiary precision rather than linguistic plausibility, and it provides a template for how other medical AI benchmarks (chest X-ray finding localization, pathology slide region annotation, ultrasound measurement verification) could adopt similar grounding requirements.

Follow-Up Research This Work Enables

1. Explicit visual grounding supervision during MLLM fine-tuning. The paper identifies the primary bottleneck as models' inability to ground diagnostic criteria to specific signal features (Depth plateaus at ~2.0, meaning models rarely progress beyond Criterion Selection and Finding Identification into the ECG Grounding stage). A direct follow-up would be to fine-tune an existing ECG-MLLM (e.g., Qwen3-VL, which shows strong general performance at Depth=1.53–1.63) with supervision that explicitly teaches perceptual grounding. The training data would consist of ECG images paired not with text rationales but with structured grounding annotations: for each training sample, provide the bounding box coordinates (lead, temporal segment) of key waveforms and the measurements of relevant intervals. The training objective would include auxiliary losses: a waveform detection loss (predict P/QRS/T boundaries), an interval measurement regression loss (predict QRS duration, PR interval, QT interval in milliseconds), and a finding-to-lead localization loss (for each clinical finding, predict which leads exhibit it). The evaluation would measure whether the grounded model exceeds the ~6% Completion ceiling and ~2.0 Depth ceiling that the paper establishes as the current state of the art. A strong positive result (Completion >20%, Depth >3.0) would validate the paper's implicit hypothesis that the reasoning gap is a training problem, not an architectural limitation. A null result (no improvement despite grounding supervision) would suggest that current MLLM architectures fundamentally cannot perform the type of fine-grained visual analysis that ECG interpretation requires, redirecting attention toward architectural innovations.

2. Multi-diagnosis comorbidity reasoning benchmark extension. The paper acknowledges (implicitly, through its single-diagnosis design) that the current benchmark evaluates isolated diagnostic reasoning. A natural extension is to construct a comorbid subset: select ECGs from PTB-XL and MIMIC-IV-ECG that carry multiple labels from the 17 core diagnoses (e.g., first-degree AV block + left anterior fascicular block, or inferior MI + sinus bradycardia). The extended benchmark would require the automated pipeline to generate ground-truth reasoning chains that explicitly model diagnostic interactions: when two diagnoses share overlapping findings (e.g., ST-segment changes in LBBB vs. ischemia), the reasoning chain must specify how findings are attributed and when diagnostic criteria are modified by the presence of another condition. The evaluation would measure: (a) whether models can identify both diagnoses when present, (b) whether they incorrectly apply criteria that are invalidated by the comorbid condition (e.g., diagnosing LVH by voltage criteria in the presence of LBBB, which is clinically inappropriate), and (c) whether the Depth bottleneck shifts when findings are attributable to multiple conditions. The paper's coverage analysis (Figure 4) showing that directly and indirectly derivable diagnoses cover 76.85% of PTB-XL and 96.18% of MIMIC-IV-ECG provides a large pool of candidate samples. A finding that models perform substantially worse on comorbid cases (Completion near 0% even for models that achieve 6% on single-diagnosis cases) would demonstrate that the reasoning gap is even more severe than the current paper suggests, while any model that handles comorbidity well would represent a qualitative advance.

3. Training data ablation study: synthetic rationales vs. structurally grounded rationales. The paper's most important causal claim — that training on LLM-generated synthetic rationales degrades reasoning capability — is supported by cross-model comparison (ECG-specific models vs. general models) but not by a controlled experiment. A direct follow-up would take a single base model (e.g., Qwen3-VL 8B) and fine-tune three variants on matched ECG interpretation data that differ only in the nature of the reasoning traces: (a) synthetic rationales generated by GPT-4 from diagnostic labels without ECG access (replicating the ECGInstruct / ECG-Protocol-Guided-Grounding-CoT paradigm), (b) structurally grounded rationales generated by converting the automated pipeline's output into natural language ("The QRS duration is 155 ms, which exceeds the 120 ms threshold, indicating prolonged QRS. The prolonged QRS is visible in leads I and V5 at [1.45s–1.61s]..."), and (c) no rationales — fine-tune only on diagnostic labels. All three variants would be evaluated on ECG-Reasoning-Benchmark, measuring IDA, Completion, Depth, and GT-RDA. The paper's framework predicts: variant (a) should show high IDA but low GT-RDA (the shortcut-learning signature), variant (b) should show lower IDA but substantially higher Completion, Depth, and GT-RDA (genuine reasoning), and variant (c) should show chance-level IDA but the highest GT-RDA (the model learns nothing about ECGs but retains its general reasoning capacity). If this pattern holds, it would provide causal evidence for the paper's central critique of synthetic rationale training and establish a recipe for effective ECG-MLLM training. If variant (b) fails to improve reasoning (Completion remains near 6%), it would suggest that the grounding problem is deeper than training data composition — perhaps requiring architectural modifications to enable fine-grained visual analysis — which would be a crucial negative result redirecting the research agenda.

4. Perceptual capability benchmarking: isolate visual detection from medical knowledge. The paper's Depth metric conflates two distinct capabilities: medical knowledge retrieval (knowing which criterion to test) and perceptual detection (seeing whether the criterion is met in the signal). A follow-up study could design a two-phase evaluation that cleanly separates these: Phase 1 tests pure perceptual capability by asking models to perform elementary visual tasks on ECGs without requiring medical knowledge — locate all QRS complexes (temporal bounding boxes), count P waves, identify which lead has the largest R-wave amplitude, measure the distance between two marked points. These tasks require visual analysis but no clinical interpretation. Phase 2 tests medical knowledge application by providing the model with pre-extracted measurements and findings (from the automated pipeline) and asking diagnostic questions — this is essentially the GT-RDA condition but across all finding types. This decomposition would reveal whether the Depth bottleneck at ~2.0 is primarily a vision problem (models cannot visually parse ECG waveforms with sufficient precision) or a reasoning problem (models cannot apply clinical logic to perceptual observations). If Phase 1 performance is also poor (models fail at basic waveform localization), the research priority shifts toward improving MLLM visual acuity for time-series data — potentially through higher-resolution image encoding, pre-training on waveform segmentation, or architectural changes for fine-grained spatial attention. If Phase 1 performance is strong but Phase 2 performance is weak, the bottleneck is in the reasoning architecture, not the visual encoder, and research should focus on reasoning chain training. The paper's ECG Grounding questions (wave grounding, measurement grounding) provide a template for Phase 1 task design; the GT-RDA condition provides a template for Phase 2.

5. Dynamic reasoning evaluation with adaptive question selection. The current benchmark uses static logic diagrams — the sequence of questions for each diagnosis is fixed in advance. This is a valid first step but does not test whether models can reason adaptively when the evidence points in unexpected directions. A follow-up would implement an adaptive evaluation protocol where the questions asked depend on the model's previous answers and the actual findings present in the ECG. For example, if the pipeline detects that QRS duration is normal on an ECG being evaluated for CLBBB, the adaptive protocol would skip the remaining CLBBB findings (since the diagnosis is immediately excluded) and pivot to asking about alternative diagnoses that could explain any other abnormalities present. This would test the model's ability to perform differential diagnosis — dynamically updating its diagnostic hypotheses as evidence accumulates — rather than mechanically verifying a pre-specified list. The evaluation would measure: (a) diagnostic accuracy under the adaptive protocol, (b) the number of questions required to reach a correct diagnosis (testing efficiency), and (c) whether the model pursues appropriate diagnostic paths versus getting stuck on irrelevant questions. The paper's automated pipeline enables this because it provides complete ground truth for all findings across all diagnoses, allowing the adaptive evaluator to know what questions are clinically relevant at each step. This direction also addresses the paper's acknowledged limitation about "clinical heuristics" — the adaptive protocol could reward efficient diagnosis (fewer questions) when the initial evidence strongly supports a particular conclusion.

6. Cross-modal generalization study: does the reasoning gap replicate in radiology? The paper identifies a specific failure pattern (success at Criterion Selection, failure at perceptual grounding) in the ECG domain. A critical question for the field is whether this pattern is unique to time-series signal interpretation or whether it generalizes to other medical imaging modalities where MLLMs are also being trained on synthetic rationales and evaluated with LLM judges. A follow-up study could adapt the ECG-Reasoning-Benchmark methodology to chest X-ray interpretation, building on CXReasonBench (Lee et al., 2025), which the paper cites as inspiration. The adapted benchmark would define logic diagrams for common radiographic findings (pneumonia, pneumothorax, pleural effusion, cardiomegaly) using established radiological criteria (silhouette sign, air bronchogram, deep sulcus sign) and require models to ground their findings to specific anatomical regions in the image (right upper lobe, costophrenic angle) rather than temporal segments. The evaluation would measure the same metrics: IDA (baseline diagnostic accuracy), Completion (flawless reasoning chains), Depth (where reasoning collapses), and GT-RDA (shortcut-learning detection). If radiologist-specific models (analogous to ECG-specific models) show the same GT-RDA collapse pattern, it would establish that the synthetic-rationale training problem is a general phenomenon in medical AI, not an ECG-specific quirk. If the pattern does not replicate — if radiology models show strong grounding capability — it would suggest that the ECG domain's difficulty stems from the temporal/quantitative nature of the signal (precise millisecond-level measurements) rather than from the training paradigm, and that visual grounding in static images is an easier problem that current MLLMs can already handle.

Practical Applications and Downstream Use Cases

1. Regulatory-grade model auditing for clinical deployment clearance. Before an ECG-MLLM can be deployed in a clinical setting, regulatory bodies (FDA, EMA) require evidence of safety and efficacy. Current evaluation paradigms based on aggregate diagnostic accuracy or LLM-as-a-Judge scores do not provide the granular failure-mode analysis that regulators need to assess risk. ECG-Reasoning-Benchmark provides a concrete auditing framework: a model seeking regulatory clearance could be required to achieve a minimum Completion rate (e.g., >80%) and Depth score (e.g., >3.5) on this benchmark, demonstrating that it can execute complete, evidence-grounded reasoning chains for the 17 core diagnoses covering 77–96% of real-world ECG presentations. The GT-RDA metric could serve as a specific screen for shortcut learning: any model showing a GT-RDA significantly below its IDA (the signature of synthetic-rationale training damage) would be flagged for additional review. The paper's separation of Criterion Selection, Finding Identification, and ECG Grounding stages would allow regulators to specify performance thresholds per cognitive sub-skill — for instance, requiring >95% accuracy on Criterion Selection (knowledge retrieval) but also >90% on Finding Identification (perceptual detection) and >85% on Measurement Grounding (quantitative precision). This granularity is absent from current regulatory frameworks for AI/ML-enabled medical devices, which typically assess only end-to-end diagnostic accuracy. The benchmark's scale (6,400+ cases across two independent datasets) provides sufficient statistical power for per-diagnosis auditing, and the expert-validated logic diagrams provide a defensible clinical standard against which to measure performance.

2. Training data quality assurance for ECG-MLLM developers. Organizations building ECG-interpreting AI systems (hospitals, medical device companies, AI startups) currently have no systematic way to evaluate whether their training data — increasingly composed of LLM-generated rationales — actually teaches models to reason rather than to pattern-match. ECG-Reasoning-Benchmark can serve as an off-policy evaluation tool during model development: after each training iteration, evaluate the model on the benchmark and track not just overall accuracy but the IDA-to-GT-RDA gap. If the gap widens (IDA increases while GT-RDA decreases), it signals that the training is inducing shortcut learning — the model is getting better at guessing diagnoses from global patterns while losing the ability to utilize explicit reasoning. This would trigger a review of the training data composition (are the rationales grounded in actual signal features or generated from labels alone?) and training procedure (is the loss function inadvertently rewarding shortcut behavior?). The paper's finding that ECG-R1-RL's GT-RDA is 22.70% while its IDA is 85.41% — the widest gap among all models — would serve as a concrete warning threshold: any model approaching a 60+ percentage point IDA-to-GT-RDA gap should be considered fundamentally broken in its reasoning capability, regardless of its diagnostic accuracy. This use case does not require the benchmark to be a perfect reasoning test; it only requires that the benchmark consistently detects the shortcut-learning signature, which Table 1 demonstrates across multiple ECG-specific models.

3. Differential diagnosis support system with verifiable evidence trails. While the paper demonstrates that current MLLMs fail at the benchmark, the benchmark's structure can be productively combined with the automated analysis pipeline to create a human-in-the-loop diagnostic support tool that does not rely on black-box model reasoning. The workflow: (1) the automated pipeline analyzes the ECG, extracts waveform boundaries, quantitative measurements, and discrete findings using the expert-validated logic diagrams; (2) the system presents the clinician with a structured reasoning report — for each potential diagnosis, it lists which findings are present, where they are located (leads, temporal segments), and what the measured values are, with abnormal values highlighted; (3) the clinician reviews the evidence and makes the final diagnostic decision. This approach leverages the pipeline's near-perfect QRS detection (1.000 recall/precision) and strong P and T wave segmentation (0.978 and 0.997 recall) to automate the perceptual grounding that MLLMs currently cannot perform, while leaving clinical judgment (diagnostic integration, uncertainty handling, comorbidity reasoning) to the human expert. The MLLM's role shifts from primary diagnostician to evidence summarizer — a task that even current models with Depth ~2.0 could perform, since they succeed at Criterion Selection (identifying which criteria are relevant) and the pipeline provides the grounded findings they fail to extract independently. This is a more immediately deployable application than fully autonomous ECG interpretation, and it addresses the paper's identified limitation about "clinical heuristics" and "diagnostic uncertainty" — the human clinician handles the nuanced judgment that the benchmark specifically excludes from evaluation.

When to Prefer This Method

The paper does not propose a method to be preferred over alternatives — it proposes an evaluation framework to be used instead of the dominant LLM-as-a-Judge paradigm. The framing is not "use our 4-stage verification loop rather than some other evaluation metric," but rather "measure reasoning capability through evidence-grounded verification rather than through text similarity to synthetic references." Within that narrow scope, the decision rule emerging from the paper is:

• Prefer ECG-Reasoning-Benchmark as your evaluation framework when you need to determine whether an ECG-MLLM performs genuine clinical reasoning versus pattern-matching, and you are willing to accept the constraints of multiple-choice format, single-diagnosis scope, and prototypical (non-borderline) ECG presentations. The benchmark is specifically designed for development-stage auditing and research evaluation, not for real-time clinical deployment monitoring, given the static logic diagrams and offline automated pipeline.
• Do not use this benchmark when (a) you need to evaluate model performance on borderline/ambiguous ECGs typical of emergency department screening, (b) you need to assess multi-diagnosis comorbidity reasoning, (c) you need to measure clinical judgment under uncertainty rather than rigorous deductive logic, or (d) you rely on free-text generation as the primary model output format and suspect format incompatibility with multiple-choice questions. In these scenarios, the benchmark's deliberate simplifications (excluded borderline cases, single-diagnosis logic, multiple-choice structure) make it an inappropriate evaluation tool, and the paper acknowledges these as limitations requiring future benchmark extensions.

The paper does not provide a decision rule comparing its benchmark to any named alternative evaluation framework, nor does it claim superiority over a specific competing benchmark — it claims superiority over an approach (LLM-as-a-Judge with synthetic references) that it argues is fundamentally incapable of measuring what matters. The "preference" is therefore categorical: when your goal is to verify reasoning rather than to assess fluency, evidence-grounded verification is the right class of methods, and ECG-Reasoning-Benchmark is a concrete implementation of that class for the ECG domain.