Instruction-Guided Lesion Segmentation for Chest X-rays with Automatically Generated Large-Scale Dataset

ArXiv: 2511.15186

🎯 Pitch

A visual-language model can now segment chest X-ray lesions from simple, natural language instructions, a task where prior models failed almost completely (71% vs. <24% accuracy). This leap comes from MIMIC-ILS, a fully automated pipeline that mined 192,000 images to generate 91,000 pixel-perfect lesion masks without any human annotation. Matching a radiologist’s workflow, the model can also state when an image is clean, correctly identifying empty scans over 90% of the time.

---

1. Executive Summary

This paper introduces instruction-guided lesion segmentation (ILS), a medical-domain adaptation of referring image segmentation that enables models to segment diverse CXR lesions based on simple, user-friendly text instructions. The authors construct MIMIC-ILS, the first large-scale instruction-answer dataset for CXR lesion segmentation, containing 1.1M instruction-answer pairs derived from 192K images and 91K unique segmentation masks across seven lesion types, all generated through a fully automated multimodal pipeline that requires no human annotation. Fine-tuning LISA on MIMIC-ILS yields ROSALIA, a VLM that achieves 71.2% gIoU and 91.8% empty-target accuracy on the ILS task — dramatically outperforming all general-domain and medical-domain baselines, which struggle to exceed 24% gIoU and near-zero empty-target accuracy. The automated data generation pipeline leverages cross-modal consistency between radiology reports and pretrained vision models — report structuring with LLMs, anomaly map extraction via diffusion-based editing, and lesion mask generation through anatomy-aware box filtering — to produce masks that medical experts accept at a 96.4% rate across 10.7K reviewed samples, establishing that high-quality pixel-level CXR lesion grounding data can be generated at scale without manual annotation.

2. Context and Motivation

The Core Problem: CXR Lesion Segmentation Models Are Brittle and Inaccessible

The fundamental problem this paper tackles is that current automated lesion segmentation models for chest X-rays are simultaneously too narrow in scope and too difficult to use. Specifically, they suffer from two intertwined limitations: (1) they can only segment one or a small handful of lesion types, and (2) they require users to provide expert-level, image-specific text descriptions that assume the user has already examined and interpreted the CXR. This creates a paradox: the people who might benefit most from automated segmentation — clinicians seeking to reduce their workflow burden, or non-experts who cannot interpret CXRs at all — are precisely the people least able to provide the detailed inputs these models demand.

This gap matters for several concrete reasons the paper establishes (Section 1):

• Clinical workflow burden: Radiologists spend substantial time precisely delineating lesion boundaries, a task that "demands substantial clinical expertise and analytical precision" (Section 1). A model that could respond to simple, intuitive instructions ("Segment the pneumonia in the right lung") rather than detailed radiological descriptions ("Bilateral pulmonary infection, two infected areas, upper right lung and upper left lung") would integrate far more naturally into clinical workflows.

• Accessibility for non-experts: In settings where radiologists are scarce, other healthcare workers — or even patients themselves — might wish to query an AI system about CXRs. Current models are essentially unusable for this population because they require the user to already know what is in the image. The ILS task flips this relationship: the user asks a question, and the model provides both localization and explanation.

• Scalability of training data: Building segmentation models that cover multiple lesion types traditionally requires radiologists to manually annotate each lesion mask — an expensive, slow process that has limited existing CXR segmentation datasets to at most a few thousand annotated images (Table 1). This paper asks whether a large-scale dataset can be generated automatically, potentially unlocking the same kind of scaling benefits that general-domain vision-language models have enjoyed.

Conflicting Demands in Existing Approaches

The paper identifies a specific tension in prior work that their ILS formulation resolves. On one side, there are classical lesion segmentation models trained on datasets like SIIM-ACR (pneumothorax only) or QaTa-COVID (COVID-19 only) (Section 2.1). These models achieve reasonable accuracy but are laser-focused on a single lesion type — they cannot generalize to the diverse abnormalities that appear in real clinical practice. On the other side, there are recent VLM-based medical segmentation models like RecLMIS (Huang et al., 2024) and LViT (Li et al., 2023) that accept free-form text prompts (Section 2.2). These models can in principle handle diverse instructions, but the text inputs they were designed for are fundamentally different from what the ILS task envisions.

The distinction is crucial and worth examining carefully. RecLMIS and LViT expect the user to provide descriptive text that reflects having already examined the image — phrases like "Bilateral pulmonary infection, two infected areas, upper right lung and upper left lung." This is not a natural query format. A radiologist encountering a new CXR would not describe the findings before seeing them; they would ask "Is there consolidation in the right lower lobe?" or "Show me any opacities." The prior models invert the information flow: the text prompt describes the answer rather than the question. This makes them impractical "not only for physicians who aim to segment diverse lesion types across various anatomical subregions before closely reviewing the image themselves, but especially for non-experts who can hardly interpret CXR images at all" (Section 1).

The ILS task re-establishes the correct direction of information: the instruction specifies what to look for and where, and the model returns what it finds. This is not merely a cosmetic change in prompt phrasing. It fundamentally changes what the model must learn: it must now perform grounded visual reasoning — parsing the instruction, locating the relevant anatomy, determining whether the lesion is present, and if so, producing the mask — rather than simple text-to-mask alignment where the text already describes the target in detail.

The Data Bottleneck

Even if one wanted to train a model for the ILS task, no suitable training dataset exists. The paper's Table 1 lays out the landscape starkly. Existing CXR datasets with spatial annotations fall into several categories, none of which support the ILS objective:

• Bounding-box datasets (VinDr-CXR, Padchest-GR, MS-CXR, TBX-11K): These provide coarse localization but not pixel-level segmentation masks. A bounding box cannot delineate the irregular, diffuse boundaries typical of CXR lesions (e.g., an infiltrate or effusion). The paper's goal is segmentation, requiring mask-level precision.

• Single-lesion mask datasets (SIIM-ACR for pneumothorax, QaTa-COVID for COVID-19): These provide masks but are restricted to one disease. A model trained only on pneumothorax masks cannot segment pneumonia, atelectasis, or effusion. Real clinical practice involves multiple lesion types within the same image — the paper explicitly notes that their dataset must support "multi-lesion" scenarios (Table 1).

• Small-scale multi-lesion mask datasets (Danilov et al.): With only 1.4K images and 600 annotations, this is too small for training modern VLMs that require hundreds of thousands of examples to achieve robust instruction-following behavior.

Crucially, none of these datasets contain instruction-answer pairs. They provide masks paired with class labels or bounding boxes, but not with natural language instructions of the form "Segment the X in the Y." This means that even if one were to repurpose existing masks, substantial engineering would be required to construct the instruction-answer pairs — and the resulting dataset would still be small, covering at most a few thousand examples.

This gap is what the paper means by "a dataset to support such a versatile task has been unavailable" (Section 1). The bottleneck is not just the absence of annotations — it is the absence of any scalable method for creating such annotations. Expert radiologists are too expensive and slow to annotate hundreds of thousands of images at the pixel level. Semi-automated approaches (like QaTa-COVID's method for COVID-19) still require expert verification and are designed for single-lesion detection. The central methodological challenge the paper takes on is whether a fully automated pipeline — using only routinely collected image-report pairs that already exist in repositories like MIMIC-CXR — can produce instruction-answer data of sufficient quality to train a competent ILS model.

Why Existing VLMs Fail on ILS Without Adaptation

The paper's experimental results (Table 4, Section 6.2) demonstrate a finding that might surprise readers unfamiliar with medical imaging: general-domain VLMs with segmentation capabilities perform catastrophically on the ILS task, even when they handle general-domain referring segmentation well. LISA-7B and LISA-13B — the same architecture that ROSALIA is built on — achieve only 8.3% and 8.9% gIoU respectively on the MIMIC-ILS test set, with near-zero empty-target accuracy (0.7% and 0.0%). PixelLM and Text4Seg are similarly ineffective.

This is not simply a domain gap. The medical-domain baselines — BiomedParse, RecLMIS, IMIS-Net — fare somewhat better (22–24% gIoU) but still fall dramatically short of practical utility. More importantly, their near-zero empty-target accuracy (0.0–0.6%, with the exception of IMIS-Net at 21.6%) reveals a deeper problem: these models essentially never produce "there is no lesion" as an answer. They appear to assume a lesion is always present, which is fundamentally incompatible with the ILS task's requirement that the model "reliably report [the lesion's] absence" when requested (Section 1).

Why does this happen? The paper does not provide a detailed failure analysis of each baseline, but the qualitative examples in Figure 6 are instructive. The general-domain models tend to produce masks that cover entire anatomical structures (e.g., the whole left or right lung) rather than the specific lesion. This suggests that while they can loosely associate anatomical terms in the instruction with image regions, they lack the fine-grained lesion detection capability needed to isolate the abnormal tissue. The medical-domain baselines are better at identifying diseased areas but fail to follow instructions about specific locations or lesion types — they produce reasonable-looking masks that simply do not correspond to what was requested. None of them handle negative instructions because they were never trained on data that included explicit "there is no lesion" examples.

The MIMIC-CXR Opportunity

The paper's approach exploits a unique property of the MIMIC-CXR dataset (Johnson et al., 2019) that prior work had not leveraged for segmentation: each image comes with a detailed radiology report written by a board-certified radiologist. These reports are not simple class labels — they are multi-sentence narratives that describe which abnormalities are present, where they are located, their severity, and sometimes their likely etiology. A sentence like "Bibasilar pulmonary opacities are increasing from the prior examination and are likely related to increasing atelectasis" (from Appendix E, Figure 11) contains rich information: the finding type (opacity/atelectasis), the location (bibasilar = both lung bases), the severity trend (increasing), and a differential diagnosis.

The key insight driving this paper is that these reports contain sufficient information to supervise both mask generation and instruction-answer pairing, provided one can solve two sub-problems: (1) accurately extracting structured information (lesion type, location, certainty) from free-text reports, and (2) reliably locating those reported lesions in the corresponding image. The first sub-problem is a natural language processing task that modern LLMs can handle with appropriate prompting. The second sub-problem is a computer vision challenge that requires combining multiple pretrained models — anomaly detection, anatomy segmentation, and lesion detection — and enforcing cross-modal consistency between the textual and visual signals.

The MIMIC-CXR dataset contains approximately 377K images, orders of magnitude more than any existing annotated CXR segmentation dataset. If even a fraction of these could be converted into high-quality instruction-answer pairs, it would represent a transformative resource for the field — analogous to how large-scale image-caption datasets (SBU Captions, CC12M) enabled the general-domain vision-language revolution.

How This Paper Positions Itself

The paper frames its contribution not as a single new model but as a complete pipeline that addresses three interconnected gaps: (1) the absence of an ILS task formulation, (2) the absence of a suitable training dataset, and (3) the absence of a method for generating such a dataset at scale without human annotation. The contributions are explicitly threefold (Section 1):

1. A novel automated pipeline that generates lesion masks and corresponding instructions from unannotated CXR–report pairs, eliminating the need for manual processing.
2. The MIMIC-ILS dataset, constructed by applying this pipeline to MIMIC-CXR and validated by medical experts, serving as the first resource for training and evaluating ILS models.
3. ROSALIA, a fine-tuned LISA model that demonstrates the utility of the dataset by achieving strong performance on the ILS task, establishing a baseline for future work.

This positions the paper at the intersection of three research communities. From the medical image analysis community, it inherits the problem of CXR lesion segmentation and the toolset of pretrained anatomy and lesion detection models. From the vision-language community, it adopts the LISA architecture and the referring image segmentation framework. From the large language model community, it leverages LLMs for structured information extraction from radiology reports. The novelty lies not in any individual component — the paper is explicit that its building blocks (RadEdit, CXAS, YOLO, LISA, Mistral-Small) are all prior work — but in the integration architecture that makes them work together to produce instruction-answer data with no human in the loop.

The Significance of the Empty-Target Problem

One aspect of the ILS task that deserves emphasis is the requirement to reliably report lesion absence. This is not a minor feature — it is what makes the task clinically meaningful. In real diagnostic workflows, confirming that a finding is not present is just as important as detecting one. A model that cannot say "There is no pneumonia" is not just incomplete — it is potentially dangerous, because it might hallucinate a mask where no lesion exists, or might fail to flag a negative search for clinical documentation.

The paper's N-Acc. metric (empty-target accuracy) operationalizes this requirement. The fact that baselines achieve 0–22% on this metric while ROSALIA achieves 91.8% (Table 4) demonstrates how thoroughly prior models were unprepared for this capability. The negative samples in MIMIC-ILS — 930K out of 1.1M total pairs — are not just filler; they are essential for teaching the model the concept of "requested lesion not found." The paper's instruction-answer generation framework carefully constructs these negative samples using two strategies (Section 3.2): querying lesion types absent from the report, and querying empty locations for present lesion types. This deliberate design is what enables ROSALIA's robust performance on the full ILS task.

3. Technical Approach

3.1 Reader Orientation

This paper is primarily a data engineering and systems paper whose core idea is that high-quality instruction-answer pairs for CXR lesion segmentation can be generated fully automatically by exploiting cross-modal consistency between radiology reports (which describe lesions in structured language) and pretrained vision models (which detect anatomical structures and abnormalities in images). The system being built is not a single model but a pipeline that takes raw chest X-ray images and their corresponding free-text radiology reports, extracts structured lesion information from the text, generates segmentation masks by combining signals from multiple vision models, verifies that the generated masks align with the reported findings, and finally constructs diverse instruction-answer pairs suitable for training a vision-language model. The problem it solves is the chicken-and-egg deadlock in medical image segmentation: training a model that can segment diverse lesions based on natural language instructions requires a large dataset of (image, instruction, mask) triplets, but no such dataset exists, and manual annotation is prohibitively expensive. The shape of the solution is a two-stage automated pipeline — first generating grounded lesion masks from image-report pairs, then constructing instruction-answer pairs from the grounded information — that bootstraps a million-scale training dataset from existing unannotated data, validated post-hoc by medical experts.

3.2 Big-Picture Architecture (Diagram in Words)

The system has two major stages, each composed of multiple sub-components:

Stage 1 — Grounded Lesion Mask Generation (Section 3.1): This stage takes a CXR image and its radiology report as input and produces a segmentation mask for each lesion mentioned in the report, along with structured metadata about what was found and where. It has four sequential steps:

• Report Structuring and Location Mapping: An LLM converts free-text radiology reports into structured six-element tuples (entity, sentence index, presence, certainty, location, predicted lesion type). The extracted location text is then mapped to standardized anatomical labels using a second LLM.
• Spatial Information Extraction: Three pretrained vision models process the CXR image independently: RadEdit (diffusion-based editing) generates an anomaly map by removing lesions and computing the difference from the original; CXAS (anatomy segmentation) produces masks for the anatomical regions mentioned in the structured report; YOLO (lesion detection) produces bounding boxes with confidence scores for potential lesions.
• Lesion Mask Generation: An algorithm (Algorithm 1) fuses these three visual signals — anomaly map, anatomy masks, and lesion box masks — to produce a refined segmentation mask for each reported lesion, filtering out noise and false detections through four explicit conditions.
• Location Verification: The system classifies anatomical locations into three categories: reported (mentioned in the report), grounded (successfully paired with a generated mask), and empty (lung regions with no reported lesions), enabling downstream negative sample generation.

Stage 2 — Instruction-Answer Pair Generation (Section 3.2): This stage takes the structured information from Stage 1 (grounded lesion masks, lesion types, locations, certainty levels, presence/absence flags, and empty location sets) and constructs diverse instruction-answer pairs following predefined templates:

• Positive instructions include basic instructions (specify both lesion type and location), global instructions (specify only lesion type, for lesions where the generated mask covers all reported instances), and lesion inference instructions (ask the model to predict the lesion type from an opacity).
• Negative instructions are generated for lesions absent from the report or for present lesions queried at empty locations.
• Answers pair each instruction with the appropriate ground-truth segmentation mask (or empty mask for negative samples) and a textual description that includes the lesion's location, type, and certainty level.

The output is the MIMIC-ILS dataset: 1.1M instruction-answer pairs from 192K images, covering seven lesion types with 91K unique masks.

3.3 Roadmap for the Deep Dive

• First, the report structuring and location mapping process (Section 3.1, step 1), because all downstream steps depend on knowing what lesions are mentioned and where the report claims they are located — this is the textual anchor that the visual pipeline must ground.
• Second, the three vision models used for spatial information extraction (Section 3.1, step 2) — RadEdit for anomaly maps, CXAS for anatomy masks, and YOLO for lesion detection — explaining what each produces, how they work, and why all three are needed rather than any single one.
• Third, the lesion mask generation algorithm (Algorithm 1) that fuses these visual signals, including the four filtering conditions, the threshold values, and the post-processing steps that refine the raw anomaly map into clean lesion masks.
• Fourth, the location verification step and its classification of locations into reported/grounded/empty, since the empty locations are essential for negative sample generation.
• Fifth, the instruction-answer pair generation framework, covering the three instruction types (basic, global, lesion inference), the rules for deciding which instructions are valid for each lesion, the construction of negative instructions, and the answer generation templates.
• Sixth, the quality control and dataset statistics, including the cardiothoracic ratio filtering for cardiomegaly negatives, the cross-model lung/heart mask verification, and the final dataset composition across splits.

3.4 Detailed, Sentence-Based Technical Breakdown

Report Structuring and Location Mapping

The pipeline begins by extracting structured information from the free-text radiology reports that accompany each CXR image in MIMIC-CXR. This step is critical because the subsequent mask generation must know which abnormalities to search for and where the radiologist described them — without this textual grounding, the vision models would have no way to associate detected anomalies with specific lesions of interest.

Report pre-processing. Before any structuring occurs, the pipeline extracts the "findings," "impression," and "last paragraph" sections from each raw report using the official MIMIC report pre-processing code. It then applies a hierarchical fallback logic: the impression section is used if the findings section is missing, and the last paragraph is used if the impression is also absent. Studies lacking all three sections are excluded. This ensures that a single representative text segment is selected for each study, avoiding redundant processing of overlapping report sections (Appendix A.1).

LLM-based report structuring. The pre-processed report section is passed to Mistral-Small-3.1-24B-Instruct-2503, a 24B-parameter instruction-tuned language model, using the detailed prompt shown in Appendix A.2 (Figure 9). The prompt instructs the model to extract six-element tuples for each disease or finding mentioned in the report:

• Entity: the clinical finding name (e.g., "consolidation," "pleural effusion," "pneumothorax").
• Sentence Index: the 1-indexed position of the sentence containing this finding within the selected report section.
• Presence: either positive (the finding is present or stable) or negative (the finding is explicitly stated to be absent or resolved). The prompt gives specific rules: "No change in pneumothorax" counts as positive, while "Pneumothorax has been resolved" counts as negative. General terms like "clear," "unremarkable," or "normal" are explicitly excluded from negative extraction — only findings with specific disease names being negated (e.g., "No pneumothorax") are captured.
• Certainty: either definitive (a clear, confident assertion) or tentative (uncertainty expressed, e.g., "possible pneumonia," "suggests effusion," "cannot exclude pneumothorax").
• Location: the anatomical position(s) of the finding, extracted from descriptive text. If multiple locations are mentioned, all are included. If an anatomical term is embedded in the entity phrase (e.g., "hilar adenopathy"), it is moved to the Location field and removed from the Entity (e.g., Entity: "adenopathy," Location: "hilar").
• Predicted Lesion Type: when a non-specific finding like "opacity" is described as representing a specific disease (e.g., "opacity reflects pneumonia"), the specific disease is extracted here. If no such inference is present, this field is set to None.

The prompt also includes extraction rules: medical devices (endotracheal tubes, central lines, pacemakers) are ignored; only diseases and findings are extracted. If no disease or finding exists in the report section, the model returns None.

The choice of Mistral-Small-3.1-24B-Instruct-2503 for this task reflects several practical considerations. The 24B parameter scale provides sufficient reasoning capability for the nuanced extraction rules (distinguishing positive-stable from negative-resolved, handling embedded location terms), while being small enough to run locally on the authors' GPU infrastructure in compliance with PhysioNet's credentialed data use agreement for MIMIC-CXR (Appendix A.2). Running the model locally rather than via API is required because MIMIC-CXR data cannot be transmitted to external services under the data use agreement.

Location mapping to standardized anatomical labels. The raw location text extracted by the report structuring step can take many forms — "right lower lobe," "bibasilar," "left upper and middle lobe" — and must be converted into a standardized set of anatomical labels that the downstream anatomy segmentation model (CXAS) can recognize. The pipeline uses a second LLM, medgemma-27b-text-it, which is specifically fine-tuned for the medical domain, following a two-step mapping process (Appendix A.2, Figure 10).

Step 1 maps the raw anatomical term to one of 17 broad anatomical categories: thoracic spine, cervical spine, lumbar spine, clavicle, scapula, rib, sternum, diaphragm, mediastinum, abdomen, heart, breast, trachea, vessels, lung, pleura, or lateral view location. The mapping rules include commonsense heuristics (e.g., "left," "right," or "bilateral" without further specification defaults to lung; terms visible only in lateral views must map to lateral view location).

Step 2, applied only when the Step 1 output is lung, maps the term to one of 14 fine-grained lung region labels: right upper zone lung, right mid zone lung, right lung base, right apical zone lung, left upper zone lung, left mid zone lung, left lung base, left apical zone lung, lung lower lobe left, lung upper lobe left, lung lower lobe right, lung middle lobe right, lung upper lobe right, right lung, left lung. If the term overlaps multiple categories, multiple labels are selected. If no category matches, None is returned.

For example, if the original report states "bibasilar atelectasis," Step 1 maps "bibasilar" to lung, and Step 2 maps it to right lung base and left lung base. This decomposition ensures compatibility with CXAS, which segments specific lung zones individually.

What the structured output enables. After these two LLM processing steps, each radiology report has been converted into a set of structured tuples, each representing one clinical finding with a standardized anatomical location. This structured information serves as the textual ground truth for the visual pipeline: the lesion mask generation stage will attempt to produce a mask for each (entity, location) pair, and the location verification stage will check whether the generated mask actually overlaps with the claimed anatomy. The structured tuples also directly populate the instruction-answer generation templates, where entity becomes the segmentation target, location becomes the target region, and certainty determines whether the instruction uses the specific lesion name or the generic "opacity" term.

Spatial Information Extraction from Vision Models

With the textual information extracted, the pipeline turns to the CXR image itself. Three distinct pretrained vision models process the same input image, each contributing a different type of spatial signal. The design rationale is that no single model provides all the necessary information: anomaly detection identifies what is abnormal, anatomy segmentation identifies where in the body it is, and lesion detection provides candidate locations with confidence scores. The fusion of these signals in Algorithm 1 is what enables high-quality mask generation without any manual annotation.

RadEdit for anomaly map generation. RadEdit (Pérez-García et al., 2024) is a diffusion-based image editing model trained to modify chest X-rays according to text prompts. The pipeline uses it with the standardized prompt "No acute cardiopulmonary process" — the same prompt on which RadEdit was trained — to generate an edited version of the input image from which lesions have been removed.

The process works as follows. The input image $x \in \mathbb{R}^{H \times W}$ (an 8-bit grayscale image with pixel values in $[0, 255]$) is provided to RadEdit along with the text prompt and a binary mask specifying the editing region. The editing region is defined as the union of the right lung mask, left lung mask, and heart mask — all produced by a pretrained HybridGNet model (Gaggion et al., 2022, 2024; trained on the CheXMask dataset) that segments these three organ regions. Constraining RadEdit to only edit within these organ boundaries prevents it from modifying irrelevant background regions. RadEdit outputs an edited image $\hat{x}$ from which the model has attempted to remove all lesions, producing what the image would look like if the patient had a normal cardiopulmonary exam.

Crucial implementation detail: the pipeline uses the original MIMIC-CXR DICOM files rather than the MIMIC-CXR-JPG version. The paper notes that "RadEdit was trained on the original MIMIC-CXR, and we observed that inputting the histogram-equalized MIMIC-CXR-JPG images significantly degraded the quality of the edited image" (Appendix A.3). This is a critical design choice: the histogram equalization applied in the JPG version alters the pixel intensity distribution that RadEdit was trained on, causing diffusion artifacts.

The anomaly map $\mathcal{A}$ is derived as:

$x_{\text{ano}} = \frac{x - \hat{x}}{I_{\text{max}}}$

where $I_{\text{max}} = 255$ is the maximum possible pixel intensity for an 8-bit image. This produces a difference image $x_{\text{ano}} \in [0, 1]^{H \times W}$ where pixels with high values correspond to regions that RadEdit removed during editing — i.e., abnormal hyperintense regions.

From $x_{\text{ano}}$, the anomaly map $\mathcal{A}$ is defined as:

$\mathcal{A} = \{(i, j) \mid (x_{\text{ano}})_{i,j} \geq \tau_{\text{ano}}\}$

where $(i, j)$ represents a pixel coordinate and $\tau_{\text{ano}}$ is a threshold for anomaly pixels.

What this computes: Each pixel in the difference image $x_{\text{ano}}$ represents the intensity change that RadEdit applied when attempting to "heal" the image to a normal state. Pixels where RadEdit made a large change (above the threshold $\tau_{\text{ano}}$) are considered part of the anomaly map. The result $\mathcal{A}$ is a set of pixel coordinates forming a binary mask of all hyperintense lesions in the image — opacities, consolidations, effusions, edema, and atelectasis, which all appear brighter than normal lung tissue on CXR.

Why this form: The difference-based formulation $\frac{x - \hat{x}}{I_{\text{max}}}$ captures morphological information about lesions — their shape, size, and internal structure — rather than just their presence or bounding box location. This is critical because the final output needs to be a precise segmentation mask, not a detection box. The normalization by $I_{\text{max}}$ converts the difference to a $[0,1]$ range independent of image bit depth, making the thresholding consistent across images. The alternative — using a dedicated lesion segmentation model — would require exactly the kind of pixel-level annotations that the paper is trying to avoid generating manually.

The threshold $\tau_{\text{ano}}$ is set to $0.10$ for general lesions and $0.01$ for edema (Table 7). Edema receives a lower threshold because "it tends to spread widely throughout the lungs" (Appendix A.4), producing more subtle intensity changes that a higher threshold would miss. This lesion-specific thresholding is a pragmatic concession to the different radiologic characteristics of different disease processes.

CXAS for anatomy segmentation. CXAS (Seibold et al., 2023) is an anatomy segmentation model designed for CXRs, capable of segmenting 159 distinct anatomical region classes. The pipeline applies CXAS to the RadEdit-processed image $\hat{x}$ rather than the original $x$, because "CXAS tends to produce lower-quality anatomy masks for patients with significant opacities" (Appendix A.3). By first removing opacities via RadEdit, CXAS receives a "cleaner" image where anatomical boundaries are not obscured by disease, improving segmentation accuracy.

CXAS produces anatomy masks $\{\mathcal{M}_i\}_{i=1}^n$ where $n$ is the number of anatomical labels mapped in the location mapping step. Each $\mathcal{M}_i$ is a set of pixel coordinates corresponding to a specific lung zone or broad region (e.g., right lung base, left upper zone lung). For example, if the report states "bibasilar atelectasis" and the location mapping produces [right lung base, left lung base], CXAS generates masks for both the right and left lung bases.

What this computes: Each $\mathcal{M}_i$ is a binary mask covering the specific anatomical region mentioned in the structured report. These masks serve as spatial constraints in the subsequent lesion mask generation: a detected anomaly is only considered a valid lesion candidate if it overlaps sufficiently with the anatomy masks corresponding to the locations reported by the radiologist.

Why this design: Using anatomy masks as constraints rather than simply trusting the anomaly map alone addresses a fundamental ambiguity in CXR interpretation. An opacity in the right lung base could be pneumonia, atelectasis, or effusion — the visual appearance alone is often insufficient to distinguish them without location context. By requiring that the generated mask overlaps with the specific anatomical region described in the report, the pipeline ties the visual evidence to the textual description, resolving ambiguities that pure image-based methods would struggle with.

Pretrained YOLO for lesion detection. The third vision model is a YOLO object detector, specifically the checkpoint from a submitted solution to the VinBigData Chest X-ray Abnormalities Detection competition (Nguyen et al., 2020), which was trained to detect 14 lesion types on the VinDr-CXR dataset. Although the model can detect diverse abnormalities (aortic enlargement, atelectasis, calcification, consolidation, ILD, infiltration, lung opacity, nodule/mass, other lesion, pleural effusion, pleural thickening, pneumothorax, pulmonary fibrosis), the pipeline filters its outputs to retain only findings considered hyperintense lesions — explicitly excluding aortic enlargement, other lesion, and pneumothorax (Appendix A.3).

The YOLO model outputs two things for each detection: a bounding box specifying the pixel coordinates of the detected lesion, and a confidence score $conf_{\mathcal{B}_j} \in [0, 1]$. From these bounding boxes, the pipeline constructs a set of lesion box masks $\{\mathcal{B}_j\}_{j=1}^m$, where $m$ is the number of detected boxes and each $\mathcal{B}_j$ is the set of pixel coordinates enclosed by one bounding box.

What this computes: The lesion box masks provide candidate regions where lesions might exist, each with an associated confidence. These boxes serve as seeds for the anomaly map decomposition: rather than trying to separate overlapping anomalies in $\mathcal{A}$ from scratch, the pipeline uses the YOLO boxes to propose candidate locations, then refines each candidate by extracting the connected components of $\mathcal{A}$ that intersect with it.

Why YOLO alone is insufficient: YOLO object detection produces bounding boxes, not segmentation masks. A bounding box around a diffuse opacity would include large amounts of normal lung tissue within its rectangle, making it unsuitable as a final lesion mask. The YOLO output identifies where to look in the anomaly map, while the anomaly map provides the precise boundary within that region. This division of labor — YOLO for coarse localization, RadEdit for fine boundary delineation — leverages the complementary strengths of the two models.

Summary of spatial information extraction. After this step, the pipeline has three visual cues for a single input image:

• $\mathcal{A}$: a binary mask of all hyperintense anomalies (from RadEdit).
• $\{\mathcal{M}_i\}_{i=1}^n$: binary masks for each anatomical region mentioned in the report (from CXAS).
• $\{\mathcal{B}_j\}_{j=1}^m$: bounding box masks with confidence scores for candidate lesions (from YOLO).

These three signals are complementary and mutually constraining. $\mathcal{A}$ alone is a composite signal that cannot distinguish between different lesions. $\{\mathcal{B}_j\}$ alone provides coarse locations but not precise boundaries. $\{\mathcal{M}_i\}$ alone provides anatomical context but no information about whether a lesion actually exists. The fusion algorithm in the next step combines all three to produce high-quality masks.

Lesion Mask Generation (Algorithm 1)

The core of the mask generation pipeline is Algorithm 1, which takes the three visual cues from the previous step and produces a refined segmentation mask for each reported lesion. The algorithm is designed as a filtering and decomposition process: it starts with all YOLO-detected bounding boxes, filters out those that don't align with the reported anatomy or don't contain sufficient anomaly signal, then extracts connected components from the anomaly map within the surviving boxes and applies post-processing to produce clean masks.

Inputs to the algorithm:

• $\mathcal{A}$: the anomaly map (set of pixel coordinates of hyperintense regions).
• $\{\mathcal{M}_i\}_{i=1}^n$: anatomy masks for the reported locations (e.g., right lung base, left lung base).
• $\{\mathcal{B}_j\}_{j=1}^m$: lesion box masks from YOLO, each with a confidence score $conf_{\mathcal{B}_j}$.
• $L_r$ and $L_l$: right and left lung masks from the pretrained HybridGNet.

Step 1: Union of anatomy masks. The algorithm first constructs:

$\mathcal{M}_{\text{union}} = \bigcup_{i=1}^n \mathcal{M}_i$

This is the union of all anatomy masks corresponding to the reported locations. For a report mentioning "bibasilar atelectasis," $\mathcal{M}_{\text{union}}$ would be the combined mask of the right and left lung bases. This union serves as the spatial reference for the first filtering condition.

Step 2: Box-level filtering with four conditions. For each detected lesion box $\mathcal{B}_j$, the algorithm evaluates four binary conditions. A box must satisfy all four to be retained; any box failing any condition is discarded.

Condition $c_1$ — Anatomy overlap:

$c_1 \leftarrow \frac{|\mathcal{B}_j \cap \mathcal{M}_{\text{union}}|}{|\mathcal{B}_j \cup \mathcal{M}_{\text{union}}|} \geq \tau_{\text{anatomy}}$

where $|\cdot|$ denotes the number of pixels in the set.

What this computes: the Intersection-over-Union (IoU) between the detected bounding box and the union of anatomy masks corresponding to the reported locations. If the IoU exceeds $\tau_{\text{anatomy}}$, the box is considered to overlap sufficiently with the anatomy described in the report.

Why this condition: This enforces that the detected lesion is actually in the location the radiologist described. Without this condition, a YOLO detection of pneumonia in the left lung would be incorrectly associated with a report mentioning atelectasis in the right lung base. The condition creates a spatial association between textual descriptions ("right lung base atelectasis") and visual detections. The threshold $\tau_{\text{anatomy}} = 0.25$ is deliberately permissive — it doesn't require the box to be entirely within the anatomy, only that there is meaningful overlap, since bounding boxes around lesions near anatomical boundaries may extend beyond the labeled region.

Condition $c_2$ — Confidence score:

$c_2 \leftarrow conf_{\mathcal{B}_j} \geq \tau_{\text{conf}}$

This is a straightforward threshold on the YOLO model's confidence score, set to $\tau_{\text{conf}} = 0.20$ for general lesions and $0.01$ for edema.

Why the low threshold for edema: Edema produces diffuse, bilateral opacities that are more difficult for a bounding-box detector to localize with high confidence. Lowering the threshold ensures that edema detections are not filtered out prematurely by the confidence condition, while the other three conditions provide sufficient filtering power to prevent false positives.

Condition $c_3$ — Internal anomaly signal ratio:

$c_3 \leftarrow \frac{|\mathcal{B}_j \cap \mathcal{A}|}{|\mathcal{B}_j|} \geq \tau_{\text{signal}}$

where $\tau_{\text{signal}} = 0.20$.

What this computes: the fraction of pixels within the bounding box that are also in the anomaly map. A high value means the box contains substantial hyperintense signal; a low value means the box is mostly normal tissue.

Why this condition: The anomaly map $\mathcal{A}$ captures the actual morphological extent of lesions, but can contain false negatives — pixels that belong to a lesion but weren't flagged as anomalous by RadEdit. This condition ensures that the boxes retained have enough anomaly signal to be meaningful, but doesn't require perfect coverage. Without this condition, noisy YOLO detections that don't correspond to any actual hyperintense region would produce meaningless or empty masks after anomaly map extraction. The threshold $0.20$ means a box can miss up to 80% of its area in the anomaly map and still pass, which accommodates the known false-negative tendency of RadEdit.

Condition $c_4$ — Size relative to lung:

$c_4 \leftarrow \left( \frac{|\mathcal{B}_j \cap L_r|}{|\mathcal{B}_j \cup L_r|} \geq \tau_{\text{size}} \right) \vee \left( \frac{|\mathcal{B}_j \cap L_l|}{|\mathcal{B}_j \cup L_l|} \geq \tau_{\text{size}} \right)$

where $\tau_{\text{size}} = 0.10$ and $\vee$ is logical OR.

What this computes: the IoU between the bounding box and either the right or left lung mask. The box must occupy at least 10% of the area of either lung.

Why this condition: This eliminates very small detections that are likely noise — tiny specks of anomaly signal, imaging artifacts, or YOLO false positives on irrelevant structures. A lesion that is clinically meaningful should occupy a non-trivial fraction of a lung field. The OR between right and left lung accommodates cases where the box straddles both lungs or where the laterality of the detection is ambiguous.

Step 3: Connected component extraction. For each box $\mathcal{B}_j$ that passes all four conditions, the algorithm extracts the connected components of the anomaly map that intersect with the box:

$C \leftarrow \text{FindIntersectingComponent}(\mathcal{B}_j, \mathcal{A})$

A "connected component" in this context is a contiguous island of anomaly pixels — essentially one blob of abnormality — that overlaps with the bounding box. If the anomaly map contains multiple disconnected blobs, only those touching the bounding box are retained.

Step 4: Refinement and accumulation. If the extracted component $C$ is not empty, it undergoes a refinement step:

$\mathcal{M}_{\text{new}} \leftarrow \text{Refine}(C)$

The refinement includes the post-processing operations described in Appendix A.5: sequential erosion and dilation to remove small scattered noise, expansion to include adjacent pixels with similar intensity values (capturing lesion boundaries that the anomaly map might have missed), and, for effusions at the lung base specifically, incorporation of the lower portion of the lung masks from HybridGNet "to ensure clean coverage extending to the costophrenic angle."

What this computes: The final $\mathcal{M}_{\text{lesion}}$ is the union of all refined connected components from all boxes that survived filtering. Each component corresponds to one spatially distinct occurrence of the reported lesion.

Why this multi-step filtering process: The central challenge in automated mask generation from unannotated images is distinguishing true lesion pixels from noise and artifacts, while simultaneously associating each lesion with the correct report finding. Algorithm 1 addresses this through redundant filtering across independent modalities: the anatomy overlap condition ties the detection to the report text, the confidence condition uses the YOLO model's learned discriminative ability, the signal ratio condition validates against the RadEdit anomaly map, and the size condition eliminates morphological implausibilities. A detection that survives all four conditions has been independently verified by the report text (via anatomy masks), the object detector (via confidence score), the image editor (via signal ratio), and anatomical knowledge (via size constraint). This cross-modal consensus is what enables high-quality mask generation without any human verification in the loop.

Threshold values and their calibration. All thresholds (Table 7) were "carefully calibrated to ensure maximum mask quality" through an "iterative process involving multiple quality checks by a physician, who identified the settings that maximized the yield of high-quality masks" (Appendix A.4). The separate thresholds for edema reflect its distinct radiologic appearance: diffuse bilateral opacities are harder to detect with high confidence ($\tau_{\text{conf}} = 0.01$ vs. $0.20$) and produce subtler intensity changes ($\tau_{\text{ano}} = 0.01$ vs. $0.10$), so the thresholds are lowered to avoid missing these findings entirely.

Sample discarding and pipeline recall. The pipeline does not attempt to generate masks for all reported lesions. Samples that do not meet the strict cross-model consistency criteria are excluded. The recall rate — the fraction of reported lesions for which a mask is successfully generated — is approximately 28.3% on average across lesion types with the default thresholds, varying from 13.4% for atelectasis to 58.0% for edema (Table 8). The paper explicitly prioritizes precision over recall:

"While the recall can be flexibly increased by relaxing these criteria, our primary goal is to generate high-confidence samples rather than to maximize recall. Since training the LISA on a higher-recall dataset led to degraded performance on the MIMIC-ILS test set (gIoU: 54.3%, cIoU: 61.4%, N-Acc: 95.8%), we opted for a high-threshold setting." (Appendix A.7)

This is a crucial design decision. The pipeline is optimizing for training data quality, not detection coverage. A mask that is generated with high confidence but imperfect recall is more valuable for training a downstream model than a higher-recall dataset contaminated with noisy or incorrect masks. The degraded test-set performance when training on higher-recall data (gIoU dropping from 71.2% to 54.3%) empirically validates this choice.

Location Verification and Empty Location Extraction

After lesion masks are generated, the pipeline explicitly verifies whether each reported finding has been successfully grounded to a visual mask, and extracts information about locations where no lesions were found — both essential for the downstream instruction-answer generation.

Three location categories. For each finding extracted from the structured report, the pipeline defines three sets of anatomical locations:

• Reported location ($l_{\text{reported}}$): The complete set of anatomical labels extracted from the location mapping step. For "bibasilar atelectasis," this would be {right lung base, left lung base}. This represents where the radiologist claimed the lesion exists.

• Grounded location ($l_{\text{grounded}}$): A subset of the reported location — those anatomical labels for which a generated lesion mask actually spatially overlaps with the corresponding CXAS anatomy mask. For example, if the pipeline successfully produced a mask overlapping with the right lung base anatomy but not the left lung base, the grounded location would be {right lung base}. This is determined by checking which of the anatomy masks $\{\mathcal{M}_i\}_{i=1}^n$ intersect with the selected lesion box masks during Algorithm 1.

• Empty location ($l_{\text{empty}}$): Lung regions where a specific lesion is definitively not present. A lung region is classified as empty if it does not overlap at all with any of the anatomy masks corresponding to any reported lesion location (not just the lesion being queried — this check is performed across all seven major lesion types plus any other lesions mentioned in the report). For "bibasilar atelectasis" where both bases are reported locations, the left lung apex would be an empty location (assuming no other findings mention it).

What this enables. The grounded location determines which positive instructions can be generated: a basic instruction "Segment the atelectasis in the right lung base" is valid because the grounded location includes right lung base, while "Segment the atelectasis in the left lung base" would be invalid (no mask was generated there). The empty locations are used to generate negative instructions: "Segment the atelectasis in the left lung apex" becomes a negative sample because atelectasis is not present at that location. This three-way classification is what makes the instruction-answer generation fully automatic — the system knows not only what masks were generated but also where they were and weren't generated relative to the report, enabling it to construct valid positive and negative instructions without human guidance.

Instruction-Answer Pair Generation

With grounded lesion masks, structured report information, and location verification data in hand, the second major stage constructs the final instruction-answer pairs. This stage transforms the pipeline's internal representations into a format suitable for training a VLM.

Lesion types covered. The dataset targets seven major lesion types identified through discussions with board-certified physicians (Appendix B): cardiomegaly (heart enlargement), pneumonia (lung infection), atelectasis (lung collapse), opacity (general hyperintense finding), consolidation (dense opacity filling airspaces), edema (fluid in lung tissue), and effusion (fluid in pleural space). These categories are widely used in CXR-related research (CheXpert, ChestX-ray8, Indiana) and represent both broad descriptive terms (opacity, consolidation) and specific diagnoses (pneumonia, atelectasis).

Three instruction types. The framework supports three types of segmentation instructions with specific templates (Table 2):

Basic instructions specify both the segmentation target and its location. The template is:

"Segment the [Target] in the [Location]."

The target is the lesion type (e.g., "pneumonia," "atelectasis," "effusion"). The location can be a broad region (right lung, left lung), one of eight specific lung zones (right apical zone lung, right upper zone lung, right mid zone lung, right lung base, left apical zone lung, left upper zone lung, left mid zone lung, left lung base), or a combination of multiple such regions (Appendix C, Table 9). For positive samples, the answer template is the special token [SEG] alone (indicating "here is the mask"), while for negative samples the answer template is:

"[SEG] There is no [Target] in the [Location]."

Global instructions specify only the segmentation target, asking the model to find all instances of a lesion across the entire image:

"Segment the [Target]."

For positive samples, the answer includes location information:

"[SEG] It is located in the [Location]."

For negative samples:

"[SEG] There is no [Target]."

Lesion inference instructions ask the model to identify what type of opacity is present at a given location:

"Segment the opacity in the [Location] and predict its type."

For positive samples, the answer template incorporates certainty:

"[SEG] It is highly suggestive of [Lesion]." (for definitive certainty) "[SEG] It possibly reflects [Lesion]." (for tentative certainty)

For negative samples:

"[SEG] There is no opacity in the [Location]."

Rules for instruction validity. Not all instruction types can be generated for every lesion. The framework enforces specific validity rules to ensure that the generated ground-truth mask is appropriate for the instruction:

Global instruction validity: A global instruction is only generated when the grounded location and the reported location are identical. This ensures that the generated mask covers all instances of the lesion that the radiologist described. If the pipeline only generated a mask for the right lung base but the report mentioned bibasilar (both bases), a global instruction would be inappropriate because the mask is incomplete — it only shows part of the lesion. For cardiomegaly, global instructions are always generated because "cardiomegaly represents a condition of the heart itself, rather than a lesion that can appear in variable locations" (Appendix C.1).

Lesion inference target selection: Lesion inference instructions are generated only for pneumonia, atelectasis, and edema. These three lesions were selected because they are specific types of "opacity" — the fundamental visual concept — and radiologists commonly describe them using inferential language in reports ("opacity reflects pneumonia"). In contrast, other lesions like effusion and cardiomegaly are typically stated directly in reports rather than inferred from opacity. Even for these three target types, if the original report's finding has tentative certainty, the basic instruction already uses the term "opacity" instead of the specific lesion name (to avoid claiming definitive presence of an uncertain finding), but the lesion inference instruction is still generated because "the certainty is instead reflected in the ground-truth text description" — the "highly suggestive" vs. "possibly reflects" distinction in the answer template handles the certainty communication.

Negative instruction generation. Negative instructions are generated through two mechanisms (Appendix C.2):

1. Absent lesion types: For each of the seven lesion types that is either never mentioned or explicitly negated (e.g., "no pneumonia") in the report, a negative instruction is created. This can be a basic type with a randomly assigned lung region ("Segment the pneumonia in the left lung.") or a global type ("Segment the pneumonia"). To prevent dataset explosion, at most one negative instruction per lesion type per study is generated.

2. Empty location substitution: For a lesion type that is present in the image but not at all reported locations, a basic instruction is generated by substituting one of the empty locations in place of the reported location. For example, if atelectasis is present only in the right lung base (grounded location) but the left lung apex is empty, the instruction "Segment the atelectasis in the left lung apex" becomes a valid negative sample.

Answer generation and mask pairing. Each instruction is paired with an appropriate answer consisting of a [SEG] token (which the VLM will learn to map to the predicted mask embedding) and a textual description. The mask paired with the instruction depends on the lesion type:

• Organ-level abnormalities (cardiomegaly): The ground-truth mask is the heart mask from the pretrained HybridGNet, not a generated lesion mask from Algorithm 1. This is because cardiomegaly is defined by the state of the heart organ itself — an enlarged heart — rather than by a localized abnormality within the lung. The heart mask serves as the complete segmentation target.

• Localized abnormalities (pneumonia, atelectasis, opacity, consolidation, edema, effusion): The ground-truth mask is the lesion mask generated in Stage 1. Since these lesions can appear in variable locations within the lungs, the mask captures only the specific occurrence identified by the pipeline.

• Negative samples: An empty mask (zero pixels) is used, and the textual description confirms the absence.

Certainty handling in instruction generation. When a lesion finding has tentative certainty (the radiologist was not sure), the basic instruction substitutes the generic term "opacity" for the specific lesion name. For example, if the report states "possible pneumonia in the right lung base," the basic instruction becomes "Segment the opacity in the right lung base" rather than "Segment the pneumonia in the right lung base." This prevents the model from learning to associate uncertain findings with definitive labels. The certainty information is preserved in the lesion inference instruction's textual answer ("possibly reflects pneumonia" vs. "highly suggestive of pneumonia").

Dataset composition. This generation process yields 1.1M total instruction-answer pairs from 192K images, of which approximately 135K are positive samples (contain a lesion mask) and 930K are negative samples (contain an empty mask). The heavy skew toward negatives (roughly 7:1 ratio) reflects clinical reality: most lesion types are absent from most images. The training maintains a 1:1 positive-to-negative ratio in each mini-batch (Section 5), preventing the model from learning a degenerate "always predict empty" strategy.

The final dataset is split following the official MIMIC-CXR partition: approximately 1M training samples, 8.2K validation samples, and 12K test samples. The validation and test splits are substantially smaller because they only include samples that survived the expert quality assessment filtering (described below). The distribution across lesion types is shown in Figure 4, with the most common lesions in the training set being edema (182K pairs), atelectasis (167K pairs), and effusion (163K pairs), while cardiomegaly (63K pairs) is least represented due to its lower prevalence in the MIMIC-CXR population.

Quality Control Measures

Several quality control mechanisms operate throughout the pipeline to ensure the resulting dataset is suitable for training, even though the entire generation process is fully automated.

Image quality filtering. The pipeline uses only Posteroanterior (PA) and Anteroposterior (AP) view images from MIMIC-CXR. Additionally, it leverages metadata from CXReasonBench (Lee et al., 2025) — a dataset constructed from verified high-quality frontal-view images within MIMIC-CXR — to pre-filter images: the count of extractable CXAS anatomy masks and indicators of full chest visibility are used to identify and exclude mislabeled lateral views, non-chest X-rays, and images with severe anatomical truncation before any mask generation is attempted (Appendix D.1).

Lung and heart mask cross-validation. The HybridGNet and CXAS models both segment the lungs and heart, but can produce erroneous results (HybridGNet occasionally generates abnormal masks; CXAS applied to RadEdit-processed images can also fail). To catch these failures, the pipeline cross-references the masks from both models and excludes cases with significant discrepancies, "interpreting this as a failure in either the HybridGNet or CXAS segmentation." Specifically, "large differences in the outermost x-coordinates of the lung masks or the lowermost y-coordinates of the heart masks" trigger exclusion, as these "would cause problems for subsequent grounded lesion mask generation" (Appendix D.2).

Cardiomegaly negative filtering. To ensure that negative cardiomegaly samples are genuinely negative, the pipeline computes the cardiothoracic ratio (CTR) — the ratio of heart width to thoracic width — using the right lung, left lung, and heart masks from HybridGNet. Only images with a CTR of 0.45 or less are included as negative cardiomegaly samples (Appendix D.3). This 0.45 threshold was "calibrated by a physician who analyzed the distribution of CXRs across different CTR intervals to establish a clinically sound cutoff," ensuring that the model doesn't learn to label borderline cardiomegaly cases as negative examples.

Expert evaluation of the test set. The most rigorous quality control is applied to the test set. Four radiation oncologists (two board-certified specialists with 9 and 7 years of experience, and two residents with 6 years each) reviewed each sample classified as either "acceptable" or "unacceptable" based on mask quality. The reviewers were given the CXR image, the lesion type, the mapped anatomical location text from the pipeline, and the corresponding ground-truth radiology report. Any sample judged unacceptable by at least one expert was excluded from the final test set (Appendix E.2-E.3).

The results (Table 3) show that across 10,701 initially reviewed samples, the overall acceptance rate was 96.4%, with individual expert rates ranging from 96.1% to 98.7%. Positive samples (with actual lesion masks) had a lower acceptance rate (90.1% overall) than negative samples (97.7%), reflecting the greater difficulty of generating accurate lesion masks versus correctly identifying empty locations. The lesion-level breakdown (Table 13 in Appendix E.4) shows that effusion and pneumonia had the lowest positive acceptance rates (roughly 89–90% for some experts), while cardiomegaly and consolidation were most consistently accepted (97–100%).

After excluding all rejected samples, the final test set contains 12,235 instruction-answer pairs (Table 12), with 852 positive basic, 1,049 positive global, and 453 positive lesion inference samples, plus 8,316 negative basic, 796 negative global, and 769 negative lesion inference samples. This expert-verified test set serves as the evaluation benchmark for all experiments in Section 6, ensuring that reported performance numbers reflect quality on clinically validated ground truth.

Model Training on MIMIC-ILS

The final technical component is the training of ROSALIA on the MIMIC-ILS dataset. The paper adopts the LISA architecture (Lai et al., 2024) without architectural modifications — the contribution is the dataset and the demonstration that it enables effective ILS training, not a novel model design.

Architecture. ROSALIA uses the LISA-7B configuration: LLaVA (Liu et al., 2023) as the VLM backbone with 7B parameters, integrated with SAM-H (Kirillov et al., 2023) — the largest variant of the Segment Anything Model — as the mask decoder. The VLM processes the CXR image and instruction text to produce two outputs: a hidden embedding for the special [SEG] token and an autoregressively generated textual description. The [SEG] token embedding is then passed to SAM's mask decoder alongside image embeddings from SAM's frozen image encoder, producing the final predicted segmentation mask.

Training configuration. Training starts from the original LISA-7B checkpoint — not from scratch — and uses:

• Fine-tuning strategy: LoRA (Hu et al., 2022) with rank 128 and alpha 256 applied to the VLM backbone, while SAM's mask decoder is fully fine-tuned. This partially-frozen approach preserves the VLM's general language-and-vision capabilities while adapting the mask decoder to the CXR domain.
• Optimization: AdamW optimizer (Loshchilov and Hutter, 2017) with initial learning rate 0.0003, WarmupDecayLR scheduler with 100 warmup steps and minimum learning rate 0.
• Training duration: 15 epochs with total batch size 256. Training takes approximately 2.5 days on two NVIDIA H100 GPUs using DeepSpeed Stage-2 configuration (Appendix F).
• Batch composition: The ratio of positive to negative samples is maintained at 1:1 in each mini-batch, despite the severe class imbalance in the full dataset. This prevents the model from collapsing to always predicting empty masks.
• Data augmentation: Each input image has a 50% chance of being processed with histogram equalization during training (Appendix F).

Loss function. The total loss combines two components:

$\mathcal{L} = \lambda_{\text{txt}} \mathcal{L}_{\text{txt}} + \mathcal{L}_{\text{mask}}$

$\mathcal{L}_{\text{mask}} = \lambda_{\text{bce}} \mathcal{L}_{\text{bce}} + \lambda_{\text{dice}} \mathcal{L}_{\text{dice}}$

where $\mathcal{L}_{\text{txt}}$ is the autoregressive cross-entropy loss for the generated answer text, $\mathcal{L}_{\text{bce}}$ is the binary cross-entropy loss between the predicted foreground probability map and the ground-truth mask, and $\mathcal{L}_{\text{dice}}$ is the DICE loss (Milletari et al., 2016). The coefficients are $\lambda_{\text{txt}} = 0.5$, $\lambda_{\text{bce}} = 5$, and $\lambda_{\text{dice}} = 1$. The DICE loss is computed only for positive samples (those with actual lesion masks), since DICE is undefined for empty targets.

What this loss computes: The language loss $\mathcal{L}_{\text{txt}}$ encourages the model to generate correct textual descriptions ("There is no pneumonia in the left lung," "It is highly suggestive of atelectasis"). The mask loss $\mathcal{L}_{\text{mask}}$ encourages the predicted segmentation mask to match the ground truth. The BCE component penalizes pixel-level classification errors, while the DICE component directly optimizes the overlap metric that will be used for evaluation.

Why this weighting: The higher $\lambda_{\text{bce}} = 5$ relative to $\lambda_{\text{dice}} = 1$ suggests that pixel-level accuracy is prioritized over region-level overlap during training. The lower $\lambda_{\text{txt}} = 0.5$ balances the language loss contribution against the mask losses, preventing the text generation objective from dominating the segmentation objective. Computing DICE loss only on positive samples avoids the mathematical degeneracy of DICE computation on empty masks (where both prediction and target have zero foreground pixels, making DICE either 1.0 or undefined depending on implementation).

Inference. During inference, ROSALIA processes an instruction and image in approximately 20 minutes for segmentation-only evaluation on the 12K-sample test set, or 1.5 hours when generating both masks and textual descriptions (Appendix F). The pipeline returns a segmentation mask and optionally a textual description, depending on the instruction type.

4. Key Insights and Innovations

Innovation 1: Reframing CXR Lesion Segmentation as a Two-Way Instructional Interaction Rather Than a One-Way Descriptive Prompt

The paper's most fundamental conceptual move is redefining the human–model interaction for CXR lesion segmentation from a descriptive paradigm to an instructional one. In prior work on text-guided medical segmentation — including RecLMIS (Huang et al., 2024) and LViT (Li et al., 2023) — the text input functions as a detailed description of what the user has already observed in the image: "Bilateral pulmonary infection, two infected areas, upper right lung and upper left lung." This is effectively a verbal annotation. The model's job is to align that description with image regions, producing a mask that matches the provided description. The information flows from user to model: the user tells the model what is there, and the model confirms it.

The ILS task inverts this. The user provides a query — "Segment the pneumonia in the right lung" — and the model must determine whether pneumonia exists at that location, produce a mask if it does, and report its absence if it does not. The information now flows bidirectionally: the user specifies what to search for, and the model reports back what it found. This is not a cosmetic rephrasing of prompts. It fundamentally changes what the model must learn. Under the descriptive paradigm, the model performs text-to-mask alignment: given a description of a finding, locate it. Under the instructional paradigm, the model must perform grounded visual querying: parse the instruction, attend to the specified anatomical region, determine whether the requested lesion type is present there, and produce the appropriate output (mask + location description, or absence confirmation).

The distinction becomes clearest when examining what baseline models cannot do. Table 4 shows that general-domain VLMs (LISA, PixelLM, Text4Seg) and medical-domain models (BiomedParse, RecLMIS) achieve 0–0.7% empty-target accuracy — they essentially never produce "there is no lesion" as an answer. This is not a minor deficiency; it reveals that these models were never designed to handle the query semantics of the ILS task. They were trained under the assumption that the text prompt describes something present in the image, and when that assumption is violated, they hallucinate masks rather than reporting absence. The instructional reframing requires the model to internalize the concept that a valid response to a segmentation instruction can be "the requested finding does not exist here." The fact that prior work never even measured this capability underscores how thoroughly the field had conflated "text-guided segmentation" with "description-guided segmentation."

This reframing also connects the ILS task to broader trends in AI interaction design. Just as large language models shifted from command syntax ("find restaurants near me with outdoor seating") to conversational querying, the ILS task moves CXR segmentation from requiring expert-formatted inputs to accepting natural, pre-examination questions. A clinician who has not yet examined the image can ask "Is there consolidation in the right lower lobe?" — exactly the kind of question that initiates the diagnostic workflow — rather than having to first identify the consolidation visually and then type a description of it. This makes the model a search tool rather than a confirmation tool, which is a far more natural fit for how radiologists actually work.

The significance of this reframing extends beyond the specific application. It identifies a hidden assumption in the referring image segmentation literature — that the referring expression describes the target — and shows that removing this assumption changes the task requirements so fundamentally that existing models collapse entirely (single-digit gIoU, near-zero N-Acc.). The ILS formulation points toward a more general class of medical vision-language tasks where the model must reason about presence/absence in addition to localization, a capability that will be essential for any system that aims to assist rather than merely confirm.

---

Innovation 2: Cross-Modal Consistency as a Substitute for Manual Annotation at Scale

The paper's central technical innovation is not any single model or algorithm but a systems-level insight about how to generate high-quality segmentation training data without human annotators: exploit the consistency between two independently generated signals — radiology reports (text) and pretrained vision models (images) — and discard samples where these signals disagree. This is a fundamentally different approach to automated annotation than prior work in medical imaging, and its implications extend well beyond the CXR domain.

To appreciate what is distinctive here, consider the dominant approaches to automated or semi-automated medical image annotation. The most common is weak supervision from image-level labels: use class activation maps or attention-based methods to derive pixel-level pseudo-labels from classification models trained on global labels (common in ChestX-ray8 and CheXpert derivatives). The problem is that these pseudo-labels are notoriously imprecise — they capture the most discriminative region for classification, not the full morphological extent of the lesion. A pneumonia classifier might attend to the most opaque part of a consolidation while missing its boundaries entirely. Another approach is semi-automated segmentation with expert verification: a model proposes masks, and a radiologist corrects them (as in QaTa-COVID for COVID-19). This reduces annotation burden but still requires expert review at scale, which is exactly what MIMIC-ILS avoids.

The paper's approach is closer in spirit to multi-modal self-supervision, but with a crucial difference. Rather than training a model to predict alignment between modalities (e.g., contrastive image-text pretraining), the pipeline uses pretrained models from both modalities as independent "sensors" measuring the same underlying phenomenon — the presence and location of a lesion — and then applies a strict consistency filter. If the radiology report says "bibasilar atelectasis" and the visual pipeline produces a mask overlapping with the right lung base anatomy but not the left lung base, the pipeline records the grounded location as {right lung base} and the left lung base becomes an empty location. The mask is generated only where text and vision agree; where they disagree, the sample is either modified (negative instruction for empty location) or discarded entirely.

The intellectual move is treating disagreement between modalities as information rather than as error. Prior automated annotation efforts typically try to maximize recall — generate as many annotations as possible — and then handle noise through robust training or loss reweighting. This paper inverts that logic: it prioritizes precision over recall, explicitly discarding samples where cross-modal consistency cannot be established, and empirically demonstrates that this produces better training data. The recall rate of 28.3% on average (Table 8) means the pipeline throws away ~72% of potential annotations. When the authors trained on a higher-recall (47.3%) version of the dataset generated by relaxing thresholds, performance on the test set dropped from 71.2% gIoU to 54.3% — a 24% relative decline. The discarded samples were not just missing annotations; they were actively harmful as training data because they introduced noisy or incorrect masks.

This finding has significance beyond the specific pipeline. It suggests a general principle for automated dataset construction in domains where multiple independent signals about the same phenomenon are available: precision-filtered cross-modal agreement may be a more scalable path to high-quality training data than attempting to maximize annotation coverage. This is a counterpoint to the prevailing "bigger datasets are better" ethos and provides an empirical calibration point: for this task and this pipeline, a 28% recall rate with high precision outperforms a 47% recall rate with lower precision. The principle would apply in any domain with paired text and image data where the text describes specific visual entities — pathology reports with histology slides, echocardiogram reports with ultrasound images, or even non-medical domains like instructional manuals with photographs.

The expert validation results (Table 3) provide the crucial empirical anchor. A 96.4% acceptance rate across 10.7K expert-reviewed samples, with positive samples (the harder case) at 90.1%, demonstrates that the cross-modal consistency approach can approach human-annotation quality without any human in the loop. This is not a claim that automated annotation matches expert annotation — the 3.6% rejection rate and the 9.9% rejection rate for positive samples demonstrate there is a quality gap. But the gap is small enough that the resulting dataset serves as effective training data, and the fully automated nature of the pipeline means it can scale to 1.1M samples — roughly 100× more than any manually annotated CXR segmentation dataset.

---

Innovation 3: Identifying Empty-Target Handling as the Critical Missing Capability in Medical Vision-Language Segmentation

The paper conducts what is effectively a diagnostic experiment that reveals a systematic failure mode across all existing medical and general-domain vision-language segmentation models: the inability to say "the requested finding is not present." The near-zero empty-target accuracy of all baselines (0.0–0.7% for six of eight models, with only IMIS-Net reaching 21.6% — Table 4) is not presented as merely a performance gap but as evidence of a fundamental capability that existing architectures and training paradigms do not confer.

This insight is significant because it identifies a capability that has been entirely overlooked in the evaluation of medical segmentation models. Prior work on text-guided CXR segmentation (RecLMIS, LViT) reported only positive-case segmentation metrics (gIoU, cIoU, Dice). The question "does this model correctly identify when the requested lesion is absent?" was never asked, because the descriptive paradigm — where the text prompt describes an observed finding — makes the question ill-posed. If the prompt describes a lesion that the user has already seen, the model does not need to reason about absence; it just needs to locate what the prompt describes. The ILS task makes absence reasoning explicit and necessary, and the baseline results show that no existing model was prepared for it.

The paper's response to this diagnostic finding is not to propose a new architectural mechanism for empty-target detection but to demonstrate that the data itself can teach this capability when properly constructed. MIMIC-ILS contains 930K negative samples — roughly 7× the number of positive samples — generated through two complementary strategies: querying lesion types entirely absent from the report, and querying present lesion types at empty anatomical locations (Appendix C.2). These negative samples are paired with explicit textual answers of the form "There is no [Target] in the [Location]." After training on this data, ROSALIA achieves 91.8% N-Acc., demonstrating that the empty-target problem is primarily a data problem, not an architecture problem — the same LISA architecture that achieves 0.7% N-Acc. without MIMIC-ILS training achieves 91.8% with it.

This is a conceptually important finding because it suggests that the catastrophic failure of existing models on empty-target cases is not due to an inherent limitation of the VLM segmentation paradigm but due to the absence of negative examples in their training data. General-domain RIS datasets are constructed entirely from images that contain the referred object — RefCOCO, RefCOCO+, and RefCOCOg all provide referring expressions for objects that are definitely present. Medical datasets follow the same pattern: if a mask is annotated, the lesion is present. The concept of "query for absence" simply never appears in the training distribution. The paper's contribution is identifying this blind spot explicitly, constructing a dataset that fills it, and demonstrating that the capability emerges from the data without architectural changes.

The broader implication is that presence/absence reasoning may be a missing capability across many medical AI tasks, not just segmentation. A diagnostic model that can only report positive findings but cannot confirm the absence of disease is of limited clinical utility — ruling out conditions is a core part of the diagnostic process. The paper's framework for generating negative examples — using structured report information to identify what is not mentioned, combined with anatomical location reasoning to identify where present lesions are not located — provides a template for extending this capability to other medical vision-language tasks.

---

Innovation 4: The Diagnostic Hierarchy of Lesion Types as a Scaffold for Instruction Design

A subtler but significant conceptual contribution is the paper's use of medical diagnostic reasoning structure — specifically, the relationship between broad descriptive findings (opacity, consolidation) and specific disease entities (pneumonia, atelectasis, edema) — as a scaffold for generating diverse instruction types. This is not merely a template design choice; it reflects a deliberate exploitation of how radiologists actually reason about CXRs, and it enables the dataset to cover multiple levels of diagnostic specificity within a unified framework.

The key move is the treatment of "opacity" as a superordinate visual concept from which specific diagnoses are inferred. In clinical practice, a radiologist seeing a hyperintense region on a CXR first identifies it as an opacity (a descriptive finding), then uses additional features — location, shape, borders, associated findings — to infer the specific disease process (pneumonia, atelectasis, edema). The paper's instruction types mirror this hierarchy: basic instructions specify the precise lesion type ("Segment the pneumonia"), while lesion inference instructions ask the model to perform the inference step ("Segment the opacity in the right lung base and predict its type"). The global instruction type occupies an intermediate level, asking for all instances of a named finding across the entire image.

What makes this more than just template engineering is that the hierarchy is grounded in the radiology reports themselves. The report structuring LLM extracts both the entity and the predicted lesion type (e.g., an entity of "opacity" with a predicted lesion type of "atelectasis"), preserving the inferential relationship from the original clinical text. The certainty field further enriches this hierarchy: when certainty is tentative, the basic instruction uses "opacity" rather than the specific disease name, reflecting the fact that the radiologist was not confident enough to commit to a diagnosis. The lesion inference answer template distinguishes between "highly suggestive of" (definitive) and "possibly reflects" (tentative), encoding the certainty information in the model's expected output.

This design enables ROSALIA to learn not just where lesions are but also something about the evidential relationship between visual features and diagnostic labels. When the model is asked to "predict its type" for an opacity, it is being trained to perform a simplified version of differential diagnosis — mapping visual patterns to disease categories with associated certainty levels. Table 6 shows that this is the hardest instruction type (84.8% text accuracy overall, vs. 96.8% for basic), which is consistent with the clinical reality that CXR alone often cannot definitively distinguish between pneumonia, atelectasis, and edema without clinical context. The model's 75.1% positive accuracy on lesion inference reflects this inherent ambiguity.

The significance of this design choice is that it demonstrates how domain knowledge about diagnostic reasoning can be encoded into dataset construction, yielding a training resource that teaches not just segmentation but also a simplified form of structured reporting. This is a different philosophy from the "scale and let the model figure it out" approach common in general-domain vision-language work. The instruction types are not arbitrary templates — they correspond to distinct clinical question types that a radiologist might actually ask. Basic instructions correspond to "show me finding X in location Y" (confirmation). Global instructions correspond to "find all instances of X" (search). Lesion inference corresponds to "what is this opacity?" (diagnostic reasoning). By structuring the dataset around these clinically meaningful question types, the paper ensures that the model learns capabilities that transfer to real diagnostic workflows rather than just optimizing for segmenting whatever the prompt describes.

5. Experimental Analysis

Evaluation Methodology

• Dataset. All experiments use the MIMIC-ILS test split: 12,235 instruction-answer pairs across seven lesion types, filtered through expert review where any sample judged unacceptable by at least one of four radiation oncologists was excluded. The test set contains 852 positive basic, 1,049 positive global, 453 positive lesion inference, 8,316 negative basic, 796 negative global, and 769 negative lesion inference samples (Appendix E, Table 12). The training set is approximately 1M samples, with a held-out validation set of 8.2K samples used for hyperparameter monitoring during training.

• Base model. ROSALIA is built on the LISA-7B architecture (Lai et al., 2024), which integrates LLaVA (Liu et al., 2023) as the VLM backbone with SAM-H (Kirillov et al., 2023) as the mask decoder. The model is fine-tuned from the original LISA-7B checkpoint — not trained from scratch — using LoRA (rank 128, alpha 256) on the VLM backbone and full fine-tuning on SAM's mask decoder. The paper also evaluates LISA-13B as a baseline. The choice of the 7B scale (rather than larger variants) is practical: training completes in approximately 2.5 days on two NVIDIA H100 GPUs, and the authors' primary contribution is the dataset, not architectural innovation — the standard LISA architecture suffices to demonstrate MIMIC-ILS's utility.

• Metrics. Three metrics evaluate model performance. For positive samples (those with actual lesion masks), the paper uses gIoU (global intersection-over-union) and cIoU (cumulative intersection-over-union), following the evaluation convention established by LISA (Lai et al., 2024). gIoU is the average IoU across all positive samples: compute the IoU for each prediction–ground-truth mask pair, then average. cIoU is the ratio of total intersection to total union across the entire dataset: sum all intersection pixels across all samples, divide by the sum of all union pixels. The cIoU metric is more sensitive to performance on large masks, while gIoU weights all samples equally regardless of lesion size. For negative samples (where the ground-truth mask is empty), the paper uses N-Acc. (empty-target accuracy), defined as "the proportion of samples correctly predicted to have no masks" (Section 6.1). A negative prediction is considered correct only if the model's output mask is entirely empty; any non-zero predicted mask on a negative sample counts as incorrect. The paper also evaluates text response accuracy (Table 6): a response is correct only when both the template and all variables (words in square brackets in Table 2) exactly match the structured ground-truth information — a strict exact-match criterion with no partial credit.

• Baselines. Since MIMIC-ILS is the first dataset for the ILS task, no existing model has been directly trained on this exact objective. The paper evaluates eight baselines spanning both general-domain and medical-domain vision-language segmentation models, all of which can take an image and text as input to produce a segmentation output. The general-domain baselines are: LISA-7B and LISA-13B (Lai et al., 2024) — the same architecture as ROSALIA but using the original general-domain checkpoint without fine-tuning on MIMIC-ILS; Text4Seg (Lan et al., 2024) — which reimagines segmentation as text generation; and PixelLM-7B and PixelLM-13B (Ren et al., 2024) — pixel-level reasoning models using large multimodal models. The medical-domain baselines are: BiomedParse (Zhao et al., 2024) — a biomedical foundation model for image parsing; RecLMIS (Huang et al., 2024) — a cross-modal conditioned reconstruction model for language-guided medical image segmentation; and IMIS-Net (Cheng et al., 2025) — an interactive medical image segmentation model. All baselines are evaluated on the MIMIC-ILS test set without fine-tuning, representing their off-the-shelf or zero-shot performance on the novel ILS task.

• Generation budget / compute accounting. The paper does not employ a generation budget or compute-matched comparison framework in the style of scaling-law analyses. The primary comparison is performance-on-test-set rather than compute-efficiency. Training ROSALIA uses two NVIDIA H100 GPUs for 15 epochs with a total batch size of 256 and a 1:1 positive-to-negative ratio in each mini-batch (Section 5, Appendix F). Inference speed is reported for reference: segmentation-only evaluation on the 12K-sample test set takes approximately 20 minutes, while segmentation with text outputs requires approximately 1.5 hours (Appendix F). There is no FLOPs-matched comparison between baselines or between training pipeline variants.

• Cross-validation / statistical protocol. No cross-validation or statistical significance testing is reported in the main experiments. The test set is a single fixed split following the official MIMIC-CXR partition, filtered through expert review. The expert review itself is conducted independently by four physicians who cannot access each other's evaluation results (Appendix E.2). For the recall-rate experiment (Appendix A.7), the paper compares training ROSALIA on the default-threshold dataset (recall 28.3%) against training on a higher-recall variant (recall 47.3%), with both evaluated on the same MIMIC-ILS test set, providing a single before-after comparison without multiple runs or error bars.

---

Main Quantitative Results

Aggregate Performance Comparison (Table 4)

The headline result is that ROSALIA dramatically outperforms all baselines across all three metrics. On the MIMIC-ILS test set, ROSALIA achieves 71.2% gIoU, 75.6% cIoU, and 91.8% N-Acc. (Table 4). The best general-domain baseline (PixelLM-13B) achieves only 12.8% gIoU, 15.4% cIoU, and 0.0% N-Acc. — meaning it never correctly predicts an empty mask. The best medical-domain baseline on segmentation quality (BiomedParse) reaches 23.8% gIoU and 18.5% cIoU but only 0.6% N-Acc., while the best medical-domain baseline on empty-target accuracy (IMIS-Net) achieves 21.6% N-Acc. but only 9.8% gIoU and 11.8% cIoU. No baseline achieves both reasonable segmentation quality and reasonable empty-target handling simultaneously.

The gap between ROSALIA and the baselines is not a matter of incremental improvement — it is a qualitative regime change. ROSALIA's gIoU is approximately 3× higher than the best medical-domain baseline (71.2% vs. 23.8%) and 5.5× higher than the best general-domain baseline (71.2% vs. 12.8%). The N-Acc. gap is even starker: 91.8% vs. the next-best of 21.6% (IMIS-Net), with six of eight baselines effectively at zero. This pattern — baselines achieving non-trivial IoU on positive cases but catastrophically failing on negative cases — confirms the paper's central argument that existing models were never designed or trained to handle the query semantics of the ILS task.

The performance of the untuned LISA-7B and LISA-13B baselines is particularly informative because these share the identical architecture as ROSALIA. LISA-7B achieves 8.3% gIoU, 12.8% cIoU, and 0.7% N-Acc.; LISA-13B achieves 8.9% gIoU, 12.2% cIoU, and 0.0% N-Acc. The 13B variant's performance is not meaningfully better than the 7B variant — and is slightly worse on cIoU — indicating that simply scaling the VLM backbone does not address the fundamental training data gap. The ~63 percentage-point gIoU improvement from LISA-7B (0-shot) to ROSALIA (fine-tuned) is driven entirely by exposure to MIMIC-ILS training data, not by architectural differences.

Per-Lesion Segmentation Performance (Table 5)

Breaking down ROSALIA's performance by lesion type (Table 5) reveals substantial variation in segmentation quality across diseases. Cardiomegaly is the standout at 89.0% gIoU and 89.0% cIoU — significantly higher than any other lesion type. This is expected because cardiomegaly segmentation uses the HybridGNet heart mask as ground truth, which is inherently more precise than the RadEdit-derived anomaly masks used for lung lesions. The heart is a well-defined organ with clear boundaries; segmenting it is fundamentally easier than segmenting diffuse, irregularly shaped lung opacities.

The remaining lesion types cluster in the 57–65% gIoU range: edema (64.8%), consolidation (61.9%), opacity (60.5%), effusion (60.3%), atelectasis (60.2%), and pneumonia (57.2%). The ranking is consistent with the known radiologic characteristics of these findings: edema tends to produce diffuse bilateral opacities that may be easier for the anomaly map to capture broadly, while pneumonia and atelectasis can appear as irregular, subtle opacities that are harder to delineate precisely. The N-Acc. for these lesion types is consistently high (85.0–97.1%), with pneumonia showing the best empty-target accuracy at 97.1% and opacity showing the lowest at 85.0%. This suggests that the model is more reliable at confirming the absence of well-defined lesions (pneumonia) than at confirming the absence of the broad, ambiguous "opacity" category.

A subtle detail: the discrepancy between gIoU and cIoU varies by lesion type. For effusion, gIoU (60.3%) is higher than cIoU (59.6%), suggesting that smaller effusions are segmented more precisely than larger ones. For edema, cIoU (66.6%) slightly exceeds gIoU (64.8%), implying that larger edema masks — which contribute more to the cumulative metric — are segmented somewhat better. These differences are small (1–2 percentage points) but hint at size-dependent accuracy patterns that the paper does not further analyze.

Text Response Accuracy Analysis (Tables 6, 14)

The paper evaluates text response accuracy using a strict exact-match criterion: both the answer template and all variables (words in square brackets in Table 2, such as lesion type, location, certainty) must exactly match the structured ground-truth information. Under this rigorous standard, ROSALIA achieves 94.4% overall text accuracy across all question types and all lesions (Table 6).

Breaking down by instruction type, performance varies systematically. Basic instructions are easiest at 96.8% overall accuracy (95.4% positive, 96.9% negative). These instructions require the simplest textual response: the positive answer is just the [SEG] token (no descriptive text), while the negative answer follows the template "There is no [Target] in the [Location]." The high accuracy reflects that the model has reliably learned to produce the correct absence confirmation text for the specified lesion and location.

Global instructions achieve 88.8% overall accuracy, with a notable asymmetry: 93.7% accuracy on positive cases but only 82.3% on negative cases. The positive answer template requires the model to state the lesion's location ("It is located in the [Location]"), which it does reliably. The negative template ("There is no [Target]") is simpler but achieves lower accuracy. This counterintuitive pattern may reflect that global negative instructions are less diverse than basic negative instructions (the location is not specified, so the model only needs to correctly identify the lesion type), but the paper does not provide a detailed error analysis. One possibility: the model may confuse which lesion types are globally absent when multiple lesions are present in the image — for instance, correctly identifying that there is no cardiomegaly while incorrectly stating there is no opacity when an opacity exists elsewhere.

Lesion inference instructions are the hardest at 84.8% overall, and the positive/negative asymmetry is reversed: 75.1% positive accuracy vs. 90.6% negative accuracy. The positive answer templates are the most complex in the dataset — they require the model to infer the lesion type from an opacity ("It is highly suggestive of [Lesion]" or "It possibly reflects [Lesion]") with the correct certainty level. The 75.1% positive accuracy reflects the inherent difficulty of this task: distinguishing pneumonia from atelectasis from edema on CXR alone is challenging even for radiologists, as the paper acknowledges when it notes that "CXR alone typically cannot provide a definitive diagnosis and often requires additional examinations" (Appendix G.1). The high negative accuracy (90.6%) is expected because confirming the absence of any opacity is a simpler visual task than classifying its specific etiology.

Table 14 provides the per-lesion breakdown of text accuracy. Cardiomegaly achieves 96.0% overall accuracy with only global instructions (basic and lesion inference instructions are not generated for cardiomegaly, as shown in Table 10). For the three lesion inference target types, positive accuracy varies dramatically: edema at 85.5%, atelectasis at 69.1%, and pneumonia at only 36.7%. The pneumonia lesion inference accuracy is the single weakest result in the entire evaluation — roughly one-third of cases — and is described by the paper as reflecting that "radiologists generally provide only a differential diagnosis based on visual findings" and that "improvements in accuracy for lesion-inference questions are naturally limited" (Appendix G.1). This is a significant limitation that the paper acknowledges but does not attempt to resolve through architectural changes or additional training strategies.

Qualitative Results (Figures 6, 7, 12, 13)

Figure 6 presents side-by-side qualitative comparisons between ROSALIA and the baselines on four example cases (three positive, one negative). The baseline models consistently fail in characteristic ways: general-domain models (LISA, PixelLM) tend to segment entire anatomical regions — the whole left or right lung — rather than the specific lesion location. This behavior is consistent with the descriptive-to-instructional paradigm shift discussed earlier: these models were trained to associate text descriptions with broad image regions, not to perform precise instruction-following segmentation. Medical-domain models (BiomedParse, RecLMIS) produce masks that look more lesion-like but do not correspond to the requested lesion type or location. For instance, a baseline might output a mask in the correct general area but of the wrong finding, or might segment a lesion that exists in the image but was not the one requested. None of the baselines handle the negative case correctly — they all produce non-empty masks when asked to segment a lesion that is absent.

In contrast, ROSALIA's outputs shown in Figure 6 demonstrate precise adherence to the instruction: the mask covers only the specified lesion in the specified anatomical region, and the negative case correctly produces an empty mask. Figure 7 demonstrates ROSALIA's within-image selectivity: when the same CXR contains multiple lesions, different instructions applied to the same image produce different, correctly targeted masks. For example, an image containing both pneumonia and effusion yields different segmentation outputs when queried for "pneumonia in the right lung" versus "effusion in the left lung base." This demonstrates that ROSALIA has learned to associate specific lesion types with specific anatomical locations as requested, rather than simply segmenting all abnormalities in the image.

Figures 12 and 13 in the appendix provide extended qualitative examples. Figure 12 shows additional baseline comparisons across diverse lesion types and locations, reinforcing the pattern that baselines fail through over-segmentation or wrong-target segmentation rather than through complete failure to produce any output. Figure 13 shows ROSALIA outputs that include both textual descriptions and segmentation masks, with all text responses rated as correctly matching the ground truth by medical experts. These examples demonstrate the full ILS capability — the model not only produces the correct mask but also generates accurate structured descriptions of what it found (or didn't find), integrating visual grounding with textual reporting.

---

Ablation Studies and Robustness Checks

Recall rate vs. training data quality (Appendix A.7, Table 8): The paper compares ROSALIA models trained on datasets generated with two different threshold configurations. The default thresholds (Table 7) produce an average recall of 28.3% across lesion types (ranging from 13.4% for atelectasis to 58.0% for edema). Halving all thresholds approximately doubles the recall to 47.3% (ranging from 28.7% for atelectasis to 82.1% for edema). However, training on the higher-recall dataset leads to degraded test performance: gIoU drops from 71.2% to 54.3%, cIoU drops from 75.6% to 61.4%, while N-Acc. actually improves slightly from 91.8% to 95.8%. This is the paper's most informative ablation because it directly tests the core design principle — that precision-filtered cross-modal agreement produces better training data than maximizing coverage. The substantial gIoU decline (16.9 percentage points) demonstrates that the additional 19% of recalled samples are sufficiently noisy to harm model performance, validating the pipeline's high-threshold, low-recall design. The slight N-Acc. improvement at higher recall is interesting but not explained: one possibility is that the additional training samples, even if noisy, provide more diverse negative examples that improve empty-target discrimination.

Data augmentation with histogram equalization (Appendix F): The paper mentions that during training, "each input image had a 50% chance of being processed with histogram equalization." No ablation is provided comparing training with and without this augmentation, so its contribution to final performance is unknown. However, the fact that MIMIC-CXR-JPG images (which are histogram-equalized) significantly degrade RadEdit's performance (Appendix A.3) while histogram equalization is used as training augmentation for ROSALIA suggests that the downstream model may benefit from exposure to contrast variation that the upstream pipeline cannot handle.

Paraphrased instruction robustness (Appendix C.3): To assess whether MIMIC-ILS's instruction diversity is sufficient or whether the model overfits to the specific templates, the authors used Qwen3-Next-80B-A3B-Instruct to paraphrase each original instruction into nine variants reflecting three user personas (medical experts, laypersons, AI developers). Training LISA on this enriched dataset still demonstrates strong performance: 67.3% gIoU, 73.1% cIoU, and 96.5% N-Acc. on the paraphrased test set. The gIoU slightly declines from 71.2% (Table 4) to 67.3%, while cIoU is roughly stable (75.6% to 73.1%) and N-Acc. improves (91.8% to 96.5%). This is an important robustness demonstration because it addresses the concern that the ILS task might be trivially solvable by memorizing template patterns rather than learning genuine instruction-following. The fact that performance degrades only modestly under substantial linguistic variation (paraphrasing across personas) suggests that the model has learned the underlying task semantics, not just surface-level template matching. The N-Acc. improvement with paraphrased training data is particularly encouraging — it suggests that linguistic diversity in training instructions helps the model generalize the concept of "absence confirmation" beyond the specific phrasing used in MIMIC-ILS.

Impact of DICE loss computation on negative samples (Section 5): The paper states that "the DICE loss was computed only for positive samples" — an implicit ablation of sorts, since computing DICE on negative samples would be mathematically problematic (DICE of two empty masks is either 1.0 or undefined depending on implementation). No explicit comparison is provided between computing DICE only on positives versus computing it on all samples with a masking mechanism, so this design choice is based on mathematical necessity rather than empirical ablation.

---

Critical Assessment

Does the paper demonstrate that MIMIC-ILS enables effective training for the ILS task?

Yes, with qualifications. The main results (Table 4) convincingly show that fine-tuning LISA on MIMIC-ILS (yielding ROSALIA) produces a model that dramatically outperforms all baselines on all three metrics. The 71.2% gIoU and 91.8% N-Acc. represent a qualitative capability that no existing model — general-domain or medical-domain — approaches. This is not a marginal improvement; it is the difference between a non-functional system (single-digit gIoU, zero empty-target accuracy) and a potentially useful one. The qualitative examples (Figures 6, 7) reinforce this by showing that ROSALIA's outputs are not just numerically better but qualitatively appropriate — the model produces masks in the right locations for the right lesions and correctly identifies negative cases, while baselines produce masks that are either anatomically implausible or inappropriate to the instruction.

However, the claim is narrower than it might appear. The paper does not demonstrate that MIMIC-ILS is necessary for ILS performance — only that it is sufficient. An experiment comparing ROSALIA against the same LISA architecture trained on a different CXR segmentation dataset (e.g., combining SIIM-ACR, QaTa-COVID, and VinDr-CXR with manually constructed instruction-answer pairs) would test whether the automated pipeline produces uniquely effective training data or whether any large enough CXR segmentation dataset would yield similar results. The paper also does not compare against a strong baseline trained on a subset of MIMIC-ILS with different quality characteristics (e.g., only the highest-confidence masks, or only samples where all four experts agreed). Without such comparisons, it is difficult to attribute ROSALIA's performance specifically to the pipeline's design choices (cross-modal consistency, high-threshold filtering) rather than to the sheer scale of the dataset (1.1M instruction-answer pairs).

The recall-rate ablation (Appendix A.7) partially addresses this concern by showing that higher recall (47.3% vs. 28.3%) actually hurts performance, which does support the claim that precision matters. But a cleaner experiment would compare: (1) MIMIC-ILS (default thresholds), (2) a dataset of the same size but with random-mask replacement (to test whether any masks at this scale suffice), and (3) a dataset with the same high-precision masks but fewer total samples (to test whether scale or quality drives the result). The current experiments show that precision beats recall, but don't disentangle precision from scale.

Does the paper validate MIMIC-ILS's quality independently of downstream model performance?

Yes, through expert evaluation, but with important limitations. The expert evaluation (Table 3) showing 96.4% overall acceptance rate across 10.7K reviewed samples is the paper's strongest evidence for dataset quality independent of model training. This is a substantial validation effort: four radiation oncologists (two board-certified, two residents) independently reviewing masks. The methodology is rigorous — any sample rejected by any expert was excluded from the test set, and reviewers could not see each other's evaluations.

However, several aspects of the expert evaluation deserve scrutiny. First, the evaluation was conducted only on the test set, not on the training or validation sets. The paper is transparent about this: "for the test set samples, clinicians classified each case" (Section 4). This means the training set — which contains approximately 1M samples — has not been expert-verified. The model was trained on auto-generated masks that may include errors, and the fact that it still performs well on the expert-verified test set is evidence of robustness, but it also means we don't know the true quality of the training data that actually drove model learning. If training masks have, say, a 15% error rate, then the model's ability to achieve 71.2% gIoU on a clean test set indicates it has learned to overcome training noise, but the paper cannot claim the training data is 96.4% accurate — only that the test data is.

Second, the positive-sample acceptance rate (90.1% overall) is notably lower than the negative-sample rate (97.7%). This makes sense: generating accurate lesion masks is harder than confirming absence. But the per-lesion breakdown (Table 13) shows substantial expert disagreement. For atelectasis, Expert B's positive acceptance rate (79.9%) is far below the other experts (97.2%, 100.0%, 99.3%). For effusion, Expert A (89.8%) and Expert B (89.3%) are markedly more stringent than Expert C (99.5%). For edema, Expert D (89.8%) differs from Expert C (100.0%) by over 10 percentage points. These discrepancies suggest that even among trained radiation oncologists, the judgment of "acceptable mask quality" has substantial inter-rater variability — a finding the paper reports but does not analyze in depth. If expert agreement is this variable, the test set's "ground truth" is itself somewhat noisy, and the 96.4% acceptance rate should be understood as a consensus threshold rather than an objective quality measure.

Third, the paper reports the final test set size (12,235 pairs) after excluding all rejected samples, but does not report what fraction of the original test set was excluded. Tables 10–12 show that the number of negative basic instruction pairs in the test set (8,316) is roughly consistent with what one would expect from the train-validation proportions, suggesting the exclusion rate was modest. But a precise exclusion rate by sample type would help readers understand how much filtering was applied.

Does ROSALIA's performance represent genuine instruction-following or template memorization?

The evidence leans toward genuine instruction-following, but with caveats. The paraphrased instruction experiment (Appendix C.3) provides the strongest evidence against template memorization: when instructions are linguistically varied across three user personas, performance drops only modestly (67.3% vs. 71.2% gIoU), and empty-target accuracy actually improves (96.5% vs. 91.8%). If ROSALIA were merely memorizing the specific template strings from MIMIC-ILS, performance would collapse under paraphrasing. The fact that it doesn't suggests the model has learned the underlying task semantics — mapping instruction intent to visual grounding — rather than surface-level pattern matching.

However, the paraphrasing experiment has a significant confound: the paraphrased instructions were used to train the model (the enriched dataset), not just to test it. The reported 67.3% gIoU is for a model trained on the paraphrased data and evaluated on the paraphrased test set. This tests whether the model can learn from diverse instructions, but it doesn't test whether the original ROSALIA model (trained only on MIMIC-ILS templates) generalizes to paraphrased instructions at test time. A cleaner experiment would be: train on MIMIC-ILS (template instructions only), test on paraphrased instructions. The paper does not report this experiment, which is the more direct test of template memorization. The current experiment demonstrates that MIMIC-ILS's functional scope "is driven primarily by the diversity of disease-anatomy combinations rather than by the number of instruction templates" (Appendix C.3), but falls short of demonstrating that a template-trained model generalizes.

Is the 71.2% gIoU practically meaningful?

Partially, but the clinical utility bar is not established. 71.2% gIoU means that on average, the predicted mask overlaps with the ground-truth mask by roughly 71% relative to their union. Whether this is "good enough" depends on the application. For cardiomegaly (89.0% gIoU), the segmentation is likely clinically useful — the heart is a well-defined structure, and 89% overlap represents high precision. For pneumonia (57.2% gIoU), the segmentation is substantially coarser — roughly 57% overlap means significant portions of the lesion are either missed or over-segmented. The paper does not provide a clinical utility analysis: no radiologist evaluated whether ROSALIA's masks would change diagnostic decisions or save time compared to manual segmentation. The expert evaluation assessed mask quality as "acceptable" or "unacceptable" but did not assess whether "acceptable" masks are sufficiently accurate for downstream tasks (lesion measurement, progression tracking, treatment planning).

The paper also does not report per-mask metrics like false positive rate, false negative rate, or boundary distance (e.g., Hausdorff distance), which would provide a more clinically interpretable picture of segmentation quality. IoU can be high even when the boundary is inaccurate if the lesion is large, because the overlapping region dominates the computation. A small missed extension of a pneumonia could be clinically significant even if IoU remains acceptable.

Does the paper adequately benchmark against alternative approaches?

No — several relevant baselines are missing. The paper evaluates eight baselines, but all are evaluated in a zero-shot setting without any fine-tuning on MIMIC-ILS (or any CXR segmentation data, in some cases). This is appropriate for establishing that the ILS task requires specialized training data, but it leaves open the question of how much of ROSALIA's advantage comes from the quality of MIMIC-ILS versus simply from being fine-tuned on a large CXR dataset. Relevant missing baselines include:

• LISA fine-tuned on existing CXR segmentation datasets. If one were to construct instruction-answer pairs from SIIM-ACR (pneumothorax masks), QaTa-COVID (COVID masks), and VinDr-CXR (bounding boxes converted to pseudo-masks), how would a LISA model trained on this smaller but expert-annotated dataset compare? This would test whether scale (1.1M auto-generated pairs) or quality (expert annotations) matters more.

• A simple non-instruction-following segmentation model. Would a standard U-Net trained on the collective masks from MIMIC-ILS (ignoring instructions) and queried with class labels rather than instructions achieve comparable performance? This would test whether the instruction-following capability adds value beyond multi-class segmentation.

• ROSALIA with only positive training samples. The paper emphasizes the importance of negative samples (930K of 1.1M pairs) and maintains a 1:1 positive-to-negative ratio in training. An ablation showing ROSALIA trained without negative samples would directly test the claim that negative samples are essential for empty-target accuracy.

• An upper-bound sanity check. What is human-level performance on the ILS task? If a radiologist were shown the same instruction and image and asked to produce a mask, what gIoU and N-Acc. would they achieve? Without a human baseline, it is impossible to interpret 71.2% gIoU as "approaching expert performance" or "still far from expert performance."

Does the paper establish generalizability beyond MIMIC-CXR?

No — all experiments use a single dataset from a single institution. MIMIC-CXR is derived from Beth Israel Deaconess Medical Center emergency department visits. The paper does not evaluate ROSALIA on external CXR datasets (e.g., CheXpert, Padchest, VinDr-CXR, or a held-out hospital's data), so the generalizability of both the dataset generation pipeline and the trained model to different patient populations, imaging equipment, or acquisition protocols is unknown. The pipeline relies on several pretrained models (RadEdit, CXAS, YOLO) that were also trained primarily on MIMIC-CXR or similar Western hospital datasets. It is plausible that the pipeline would produce lower-quality masks on CXRs from different demographic populations or with different image characteristics (e.g., portable AP vs. standard PA views, pediatric vs. adult patients). The paper's quality control filtering (Appendix D.1) explicitly excludes certain image types (non-frontal views, images with severe anatomical truncation), which limits the pipeline's applicability to a subset of the MIMIC-CXR data — and likely an even smaller subset of more heterogeneous external datasets.

Summary of Strengths and Weaknesses in the Experimental Design

Strengths:

• The expert evaluation of the test set (10.7K samples, four independent reviewers) is a substantial and rigorous validation effort that anchors the paper's quality claims.
• The baseline comparison against eight models from both general and medical domains establishes a clear performance gap that demonstrates the novelty of the ILS task.
• The per-lesion and per-instruction-type breakdowns (Tables 5, 6, 14) provide granular insight into where the model succeeds and struggles.
• The recall-rate ablation (Appendix A.7) provides empirical validation of the paper's core design principle (precision over recall).
• The paraphrasing experiment (Appendix C.3) partially addresses the template memorization concern.

Weaknesses:

• No evaluation on external CXR datasets — generalizability is unestablished.
• No human performance baseline — the clinical utility bar is unknown.
• No ablation isolating the contribution of negative samples or specific pipeline components to final model performance.
• No ablation comparing MIMIC-ILS against a same-scale dataset of comparable masks from expert annotation or alternative automated methods.
• The expert evaluation covers only the test set, not the training data that actually drove model learning.
• The paraphrasing experiment tests a model trained on paraphrased data rather than a model trained on templates and tested on paraphrases.
• No confidence intervals, error bars, or statistical significance tests on the main results.
• No analysis of failure modes — the 28.8% of cases where ROSALIA's gIoU is below 100% are not characterized (does the model fail on small lesions, ambiguous boundaries, overlapping findings, edge cases?).

The experiments convincingly demonstrate that the automated pipeline can produce a dataset that, when used to fine-tune LISA, yields a model capable of the ILS task at performance levels far beyond existing baselines. They also convincingly demonstrate that precision-filtered cross-modal consistency produces better training data than higher-recall alternatives. However, they leave open several important questions about whether the specific pipeline design choices are optimal, whether the resulting model generalizes beyond the training distribution, and whether the achieved performance is sufficient for clinical deployment. These are natural limitations for a paper introducing a new task and dataset, but they represent important directions for future work rather than resolved questions.

6. Limitations and Trade-offs

1. Difficulty Estimation Cost Is Not Accounted for in the Pipeline

The assumption or constraint. The entire dataset construction pipeline depends on being able to verify that generated lesion masks align with reported findings, but the verification process itself requires generating anomaly maps, anatomy masks, and lesion detection boxes — all computationally expensive operations. The paper focuses on the downstream training data quality and model performance but never accounts for the computational cost of running the generation pipeline itself. Specifically, each image must pass through three separate pretrained models (RadEdit, CXAS, YOLO) plus two LLM inference calls (Mistral-Small-3.1-24B for report structuring, medgemma-27b for location mapping), plus the post-processing logic in Algorithm 1. For a dataset of 192K images, this represents a substantial compute investment that the paper does not quantify.

The consequence. A practitioner seeking to apply this pipeline to a new dataset — say, CheXpert or a proprietary hospital archive — has no way to estimate the computational budget required. The pipeline is described as "fully automated" and requiring "no human intervention" (Section 1), but the cost of that automation in GPU-hours, model inference latency, and storage for intermediate outputs is entirely opaque. If the pipeline costs, for example, 100 GPU-days to process 200K images, the headline claim of "free" annotation (via automation rather than manual labeling) becomes misleading — the annotations are not free, they just shift cost from human expert time to compute time. The decision about whether this tradeoff is worthwhile depends on local compute availability and expert annotation costs, but the paper provides no data to inform that decision.

What evidence exists in the paper. The paper reports training cost for ROSALIA (2.5 days on 2 H100 GPUs; Appendix F) and inference cost (20 minutes for segmentation-only on 12K samples; Appendix F), but the dataset construction cost is never reported. The recall rate of 28.3% (Table 8) and the image filtering criteria (Section 3.1, Appendix D) imply that a large fraction of images and candidate masks are processed and then discarded, meaning the effective cost per accepted mask is even higher than the per-image cost. The paper also mentions running multiple models locally on dedicated GPU hardware due to PhysioNet data use agreement constraints (Appendix A.2), but does not translate this into concrete compute figures.

Mitigation status. Not addressed. The paper does not suggest future work on cheaper alternatives to RadEdit, CXAS, or the LLM-based report structuring, nor does it discuss the cost-quality tradeoff of using lighter-weight models for the generation pipeline. The pipeline's efficiency is implicitly assumed to be acceptable because it runs "fully automated," but this conflates automation with cost-effectiveness.

---

2. Single-Dataset, Single-Institution Evaluation — Generalizability Is Unproven

The assumption or constraint. All experiments — dataset construction, model training, and evaluation — use MIMIC-CXR, a dataset from a single institution (Beth Israel Deaconess Medical Center). The pipeline is designed for and validated on this specific data distribution. The paper does not evaluate ROSALIA on any external CXR dataset (CheXpert, Padchest, VinDr-CXR, or a multi-institutional collection), nor does it test whether the dataset construction pipeline produces comparable-quality masks when applied to CXRs from different hospitals, demographic populations, or imaging protocols.

The consequence. The paper's core claims — that MIMIC-ILS is a "foundational resource for pixel-level CXR lesion grounding" (Section 1) and that ROSALIA demonstrates "comprehensive ability to generate accurate lesion segmentations and textual responses across diverse user instructions" (Section 7) — are only established for one data distribution. Several pipeline components carry implicit assumptions that may not hold at other institutions. RadEdit was trained on MIMIC-CXR; its anomaly removal quality on images from different scanners or with different post-processing pipelines is unknown. The YOLO lesion detector was trained on VinDr-CXR (a Vietnamese hospital dataset) and may have systematic performance differences on Western vs. Asian patient populations. The LLM prompts for report structuring (Appendix A.2, Figure 9) were designed for MIMIC-CXR's reporting style; different hospitals use different report formats, abbreviations, and terminology. CXAS was trained on a multi-institutional dataset but the paper notes that it "tends to produce lower-quality anatomy masks for patients with significant opacities" (Appendix A.3) — a property that may interact with disease prevalence differences across institutions.

In the worst case, a hospital deploying the pipeline on their own data might discover that cross-modal consistency rates are much lower, recall drops below 10%, and the resulting training data is too sparse or too noisy to train a useful model. The paper provides no evidence about whether this worst case is likely.

What evidence exists in the paper. None. The paper contains no external evaluation, no domain shift experiments, and no analysis of how pipeline component performance varies across subpopulations within MIMIC-CXR itself (e.g., portable vs. standard views, ICU vs. outpatient, different age groups). Table 1 positions MIMIC-ILS as dramatically larger than existing datasets but does not compare its domain diversity to multi-institutional collections.

Mitigation status. Not addressed. The paper acknowledges in the introduction that MIMIC-CXR is the data source and does not claim generalizability beyond it, but equally does not flag this as a limitation or suggest multi-institutional validation as future work. Given that the dataset and model are released publicly with the explicit goal of advancing "research on fine-grained lesion grounding in the CXR domain" (Section 7), the unknown generalizability is a significant gap between the paper's ambition and its empirical support.

---

3. No Human Performance Baseline or Clinical Utility Assessment

The assumption or constraint. The paper evaluates ROSALIA against other automated models (Table 4) and validates the test set masks through expert review (Table 3), but never establishes what level of performance would be clinically useful. The 71.2% gIoU and 91.8% N-Acc. are presented as strong results relative to baselines, but there is no comparison to human performance on the same task: if a radiologist were asked to produce a segmentation mask given the same instruction and image, what gIoU and N-Acc. would they achieve? Is 71.2% gIoU approaching expert-level, or is it still far below? The expert evaluation rated masks as "acceptable" or "unacceptable" for inclusion in the test set — a binary quality gate — rather than assessing whether ROSALIA's predictions would be clinically actionable.

The consequence. Without a human baseline, a practitioner cannot determine whether ROSALIA is ready for deployment, requires further improvement, or is fundamentally limited by the quality of its training data. The per-lesion gIoU variation (57.2% for pneumonia to 89.0% for cardiomegaly; Table 5) further complicates this assessment — a model that segments cardiomegaly at near-human levels but pneumonia at modest levels might be clinically useful for some findings but not others, but the paper provides no guidance on where the threshold for utility lies. The 75.1% positive text accuracy on lesion inference (Table 6) and the 36.7% pneumonia lesion inference accuracy (Table 14) raise specific clinical concerns: if the model confidently misclassifies a pneumonia as atelectasis, a clinician acting on that output could pursue the wrong treatment pathway. The paper acknowledges the difficulty of lesion inference ("CXR alone typically cannot provide a definitive diagnosis"; Appendix G.1) but does not assess whether the achieved accuracy is above or below what a non-radiologist clinician would achieve, which is the relevant comparison for the "accessibility for non-experts" motivation stated in Section 1.

What evidence exists in the paper. The expert evaluation (Table 3) provides ground-truth quality metrics but not human performance on the ILS task itself. The paper reports that 96.4% of test set masks were "acceptable" and that positive samples had a lower acceptance rate (90.1%) than negative samples (97.7%), but these figures describe the data quality, not the model quality. Table 4 compares ROSALIA to automated baselines, none of which are near human-level. No experiment asks radiologists to perform the ILS task.

Mitigation status. Not addressed. The paper frames ROSALIA as demonstrating "the value of our dataset and model in advancing fine-grained lesion grounding for CXR analysis" (Section 1), which is a research contribution claim rather than a clinical deployment claim. However, the introduction's motivation emphasizes practical utility — reducing physician workload, enabling use by non-experts — and without a human baseline, the gap between research demonstration and practical utility remains unquantified. The paper does not suggest human-baseline experiments as future work.

---

4. The Pipeline's Low Recall (~28%) Means Dataset Coverage Is Sparse and Potentially Biased

The assumption or constraint. The grounded lesion mask generation pipeline achieves an average recall of only 28.3% across lesion types, with substantial variation: 13.4% for atelectasis, 28.5% for pneumonia, 58.0% for edema (Table 8). This means the pipeline successfully generates a mask for only about one in four reported lesions overall, and for some lesion types fewer than one in seven. The paper explicitly acknowledges this: "the recall can be flexibly increased by relaxing [threshold] criteria, [but] our primary goal is to generate high-confidence samples rather than to maximize recall" (Appendix A.7). The paper frames this as a feature — precision-filtered training data — rather than a bug.

The consequence. The low recall means MIMIC-ILS is a sparse and potentially biased sample of the lesions actually present in MIMIC-CXR. The lesions for which the pipeline fails to generate masks — the ~72% of reported findings that are discarded — are not randomly distributed. They are precisely the cases where cross-modal consistency could not be established: cases where the radiology report described a finding but the vision models could not locate it with sufficient confidence, or where the vision models detected something the report did not describe. These are likely to be the most challenging cases — subtle lesions, atypical presentations, findings at the boundary of detectability, or cases where the report language is imprecise or ambiguous. If the pipeline systematically misses subtle or ambiguous lesions, then MIMIC-ILS overrepresents clear, unambiguous, easily-detectable lesions, and a model trained on it (ROSALIA) may inherit this bias — performing well on the kinds of lesions the pipeline captured and poorly on the kinds it missed.

The per-lesion recall variation (13.4% to 58.0%) compounds this problem. The training data is heavily skewed toward lesion types the pipeline handles well (edema at 58.0% recall) and away from those it handles poorly (atelectasis at 13.4%). The per-lesion gIoU results (Table 5) should be interpreted in this light: the 60.2% gIoU for atelectasis may reflect not just the inherent difficulty of segmenting atelectasis but also the sparsity and potential bias of atelectasis training examples. The paper's decision to accept 28.3% recall trades coverage for quality, but the nature of the coverage loss — which specific lesion characteristics are systematically excluded — is not analyzed.

What evidence exists in the paper. Table 8 reports the recall rates. The recall-rate ablation (Appendix A.7) shows that doubling recall to 47.3% by halving thresholds reduces downstream model gIoU from 71.2% to 54.3%. This validates the precision-over-recall choice for model training but does not characterize what is lost. The paper does not analyze the properties of recalled vs. discarded lesions: are discarded lesions smaller on average? More diffuse? Located in harder-to-segment anatomical regions? Associated with more uncertain or ambiguous report language? Without this analysis, the potential for systematic bias in the training data remains an unexamined risk.

Mitigation status. Partially addressed through the recall-rate ablation, which demonstrates that including lower-quality masks harms model performance — a pragmatic justification for low recall. But the paper does not attempt to characterize the distributional shift between recalled and discarded lesions, does not suggest methods for debiasing the training data, and does not flag coverage bias as a limitation. The paper treats recall purely as a quality-control parameter rather than as a source of potential systematic bias in the resulting dataset. Future work on understanding which lesions the pipeline misses and why is not suggested.

---

5. The Revision Model Architecture (LISA) Is Adopted Without Adaptation — The Contribution Is the Dataset, Not the Model

The assumption or constraint. ROSALIA is LISA-7B fine-tuned on MIMIC-ILS with no architectural modifications. The paper is explicit about this: "The model adopts the architecture of LISA" (Section 5), and the training section describes standard LoRA fine-tuning with SAM's mask decoder fully fine-tuned. The contribution is positioned as the dataset (MIMIC-ILS) and the demonstration that it enables effective ILS training, not a novel model design.

The consequence. This is primarily a scope limitation rather than a flaw, but it constrains what conclusions can be drawn from the results. ROSALIA's strong performance relative to baselines (Table 4) demonstrates that MIMIC-ILS provides effective training data, but it does not demonstrate that the LISA architecture is optimal for the ILS task, or that MIMIC-ILS would benefit other architectures equally. There may be ILS-specific architectural improvements — attention mechanisms that better handle negative instructions, mask decoders that leverage the textual answer for refinement, or uncertainty quantification modules suited to the lesion inference sub-task — that could extract more value from MIMIC-ILS than the off-the-shelf LISA design. The paper's results establish a lower bound on what MIMIC-ILS enables; the upper bound is unknown.

More practically, the reliance on LISA-7B means ROSALIA inherits all of LISA's limitations. The model requires the full LISA inference stack (LLaVA for text generation + SAM for mask prediction), which is computationally heavy relative to a pure segmentation model. The inference time of ~1.5 hours for 12K samples with text output (Appendix F) translates to approximately 0.45 seconds per sample — acceptable for batch processing but potentially high for interactive use where a clinician issues sequential instructions and expects near-instantaneous mask updates. The paper does not profile latency or discuss deployment considerations, which are relevant to the clinical utility motivation in the introduction.

What evidence exists in the paper. The architecture description (Section 5, Figure 5) and the training details (Appendix F) confirm no architectural modifications. The paraphrase experiment (Appendix C.3) demonstrates that LISA trained on MIMIC-ILS handles linguistic variation, but this is evidence about the dataset's instruction diversity, not about architectural suitability. No architectural ablation (e.g., comparing LISA-7B vs. LISA-13B on MIMIC-ILS training, or comparing different mask decoder configurations) is reported. The baselines in Table 4 include LISA-7B and LISA-13B evaluated zero-shot, but not fine-tuned on MIMIC-ILS — so the scaling behavior of the architecture when given access to training data is not characterized.

Mitigation status. Not addressed. The paper does not claim architectural novelty, so this is not an unacknowledged limitation in the sense of an overclaim. However, the paper also does not discuss architecture-specific limitations or suggest that alternative architectures might yield better results on MIMIC-ILS. A practitioner deciding whether to invest in the dataset would benefit from knowing whether the reported ROSALIA performance represents a ceiling or a floor for what is achievable with MIMIC-ILS, but the paper provides no evidence either way.

---

6. Empty-Target Handling Depends on a Dataset Characteristic (7:1 Negative-to-Positive Ratio) That May Not Transfer to Other Tasks

The assumption or constraint. MIMIC-ILS contains approximately 930K negative samples and 135K positive samples — a roughly 7:1 ratio that reflects the clinical reality that most lesion types are absent from most images. During training, this ratio is artificially rebalanced to 1:1 in each mini-batch (Section 5) to prevent the model from learning a degenerate "always predict empty" strategy. The strong empty-target accuracy (91.8% N-Acc.; Table 4) is one of the paper's headline results and a key differentiator from baselines, which achieve 0–22% N-Acc.

The consequence. The 1:1 training ratio is a hyperparameter choice that was not ablated. The paper does not report experiments with different positive-to-negative ratios (e.g., 2:1, 1:2, 1:7 matching the natural distribution). This matters because the ratio controls a fundamental tradeoff: more negative samples improve empty-target discrimination but risk causing the model to become overly conservative (predicting empty when a subtle lesion is actually present); more positive samples improve segmentation quality on present lesions but risk false positive masks on negative cases. The 1:1 ratio is a reasonable default, but without an ablation, a practitioner cannot know whether it is near-optimal or whether small changes would significantly shift the gIoU/N-Acc. balance.

More importantly, the dependence of empty-target accuracy on training data composition means that ROSALIA's N-Acc. is fragile with respect to the instruction distribution at test time. If a deployed system receives a different distribution of positive-to-negative queries than what was seen during training (e.g., a clinician who only queries when they suspect a finding is present, yielding a high positive-to-negative ratio), the model's empty-target behavior may degrade — it may become either over-eager to predict absence (if the deployment distribution has fewer negatives than training) or prone to hallucinating masks (if the deployment distribution has more negatives). The paper provides no analysis of calibration or distribution shift robustness for the N-Acc. metric.

This limitation extends to generalization of the ILS task framework to other medical imaging domains. The paper's approach to negative sample generation depends on having access to structured radiology reports that explicitly mention which findings are and are not present. In domains without such reports — for example, datasets where only image-level labels are available without detailed textual descriptions — the negative sample generation strategy (querying absent lesion types, empty location substitution) would not be directly applicable. The ILS task formulation, with its emphasis on presence/absence reasoning, may require richer textual supervision than is available in many medical imaging contexts.

What evidence exists in the paper. The training configuration (Section 5) states the 1:1 ratio in each mini-batch but does not justify it or ablate alternatives. The N-Acc. metric is reported for the main results (Table 4), per-lesion results (Table 5), and the paraphrasing experiment (Appendix C.3), but there is no sensitivity analysis. The recall-rate ablation (Appendix A.7) incidentally reports N-Acc. changes (91.8% to 95.8% when trained on higher-recall data), but this confounds changes in mask quality with changes in sample composition, making it impossible to isolate the effect of the positive-to-negative ratio.

Mitigation status. Not addressed. The paper does not discuss the positive-to-negative ratio as a hyperparameter requiring tuning, does not ablate it, and does not suggest calibration studies under distribution shift as future work. Given the centrality of empty-target handling to the paper's claims and its novelty relative to prior work, the lack of robustness analysis around this aspect is a significant gap. A practitioner deploying ROSALIA would need to monitor N-Acc. in their specific use case and potentially retrain with a different ratio, but the paper provides no guidance on how to determine the appropriate ratio for a target deployment distribution.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the CXR lesion segmentation field from a model-centric paradigm—where progress depends on better architectures trained on whatever small, manually annotated datasets happen to be available—to a data-centric paradigm where the primary bottleneck (lack of large-scale, diverse, instruction-paired segmentation data) is addressed through automated cross-modal supervision. This is an infrastructural contribution more than a conceptual one: MIMIC-ILS provides the CXR community with a resource analogous to what SBU Captions and CC12M provided for general-domain vision-language models—a large-scale training corpus that enables models to learn capabilities (instruction following, presence/absence reasoning) that were previously inaccessible.

The magnitude of this shift should be understood precisely. It is not a paradigm shift in the sense of introducing a fundamentally new learning algorithm or architectural principle. The LISA architecture ROSALIA uses is unchanged from prior work, and the training procedure is standard LoRA fine-tuning. Rather, the paper demonstrates that a capability which appeared to require a fundamentally different model design—the ability to respond to instructional queries about lesion presence, location, and type, including confirming absence—is actually latent in existing architectures and emerges when appropriate training data is provided. The fact that LISA-7B achieves 0.7% N-Acc. zero-shot but 91.8% after MIMIC-ILS fine-tuning (Table 4) is the clearest evidence: the architecture was capable all along; the missing ingredient was data that teaches the distinction between "find X" and "confirm X is absent."

This reframes the research agenda for medical vision-language models. Before this paper, the dominant question was: "How do we design models that can follow medical instructions for segmentation?" The response was architectural—build bigger VLMs, add specialized medical pretraining, design better cross-modal attention mechanisms. After this paper, the question becomes: "How do we generate training data that teaches presence/absence reasoning for diverse medical findings, and how do we validate that data at scale without expert annotation?" This shifts attention from model architects to data engineers, and from training algorithms to data generation pipelines.

The paper also resolves a latent tension in the medical image segmentation literature that had not been explicitly articulated. On one side, works like RecLMIS and LViT demonstrated that VLMs can produce masks in response to text prompts, suggesting that natural language interfaces for medical image analysis are feasible. On the other side, these models required prompts that described already-observed findings—effectively verbal annotations rather than genuine queries—making them impractical for the workflow they ostensibly served. The paper identifies this tension explicitly through the instructional-vs-descriptive distinction (Section 1) and resolves it by demonstrating that the instructional capability is trainable, not an inherent limitation of VLM architectures. This resolution is significant because it validates the vision of natural language medical image interfaces while also explaining why prior implementations fell short—they were trained on the wrong kind of text-image pairs.

The paper also makes certain research directions less attractive. The line of work on designing increasingly complex text-guided medical segmentation models (specialized cross-modal fusion, medical-domain pretraining objectives) now appears to be addressing the wrong bottleneck. The baseline results in Table 4 show that medical-domain models (BiomedParse, RecLMIS) indeed outperform general-domain models on this task (23.8% gIoU vs. 8.3% gIoU), but the improvement is from "completely unusable" to "still unusable." The 71.2% gIoU of ROSALIA comes not from architectural sophistication but from dataset scale and negative-sample engineering. This suggests that for the specific capability of instruction-guided lesion segmentation, dataset construction methodology may be more impactful than model architecture improvements—at least at the current scale of available architectures.

Equally important is the paper's implicit argument about evaluation scope. Prior work on text-guided CXR segmentation reported only positive-case segmentation metrics (Dice, IoU). By introducing N-Acc. as a co-equal metric and demonstrating that all baselines effectively score zero on it, the paper reveals that the field's evaluation methodology was blind to a critical capability—one that becomes obvious once the ILS task is properly formulated. This will likely influence evaluation practices in the broader medical VLM community, encouraging researchers to measure not just how well models segment what users describe, but whether models can correctly report when those descriptions correspond to nothing. The fact that the best medical-domain baseline on segmentation (BiomedParse, 23.8% gIoU) achieves 0.6% N-Acc. while the best on N-Acc. (IMIS-Net, 21.6%) achieves 9.8% gIoU—and no model achieves both—demonstrates that positive-case segmentation and absence reasoning are distinct capabilities that must be evaluated and optimized jointly.

Follow-Up Research This Work Enables

Cross-institutional validation and domain shift characterization. The single most important follow-up is evaluating both the dataset generation pipeline and ROSALIA on CXRs from institutions other than Beth Israel Deaconess Medical Center. A strong follow-up would apply the pipeline to CheXpert (Stanford), Padchest (Spain), and VinDr-CXR (Vietnam), measuring three quantities: (1) the recall rate and expert acceptance rate of the generated masks on each dataset, to determine whether the pipeline's quality degrades on different patient populations or imaging protocols; (2) ROSALIA's gIoU and N-Acc. when tested directly on these external datasets (zero-shot cross-institutional transfer); and (3) the performance of a ROSALIA variant fine-tuned on the union of MIMIC-ILS and automatically generated external datasets. If the pipeline produces comparable-quality data at other institutions, the approach generalizes and becomes a practical tool for any hospital with archived image-report pairs. If quality degrades substantially, the analysis should identify which pipeline components (RadEdit editing quality, YOLO detection reliability, LLM report structuring accuracy) are most sensitive to domain shift, informing where effort should be invested for robustness. This experiment is tractable: the models are public, the external datasets are public, and the pipeline's code is released.

Human performance baselines and clinical utility thresholds. The paper establishes that ROSALIA dramatically outperforms automated baselines but provides no comparison to human performance on the ILS task. A direct follow-up would present the same 12,235 test-set instruction-image pairs to a panel of radiologists (ideally both specialists and general practitioners) and measure their gIoU, cIoU, and N-Acc. under the same evaluation protocol. This would establish: (1) the performance ceiling—what gIoU is achievable by experts on ambiguous or subtle lesions; (2) the clinical utility threshold—at what gIoU level does ROSALIA's output become preferable to a non-radiologist clinician's unassisted judgment, which is the relevant comparison for the paper's "accessibility for non-experts" motivation; and (3) the calibration of expert confidence—do radiologists make the same kinds of errors as ROSALIA (e.g., lower accuracy on pneumonia lesion inference, Table 14), suggesting that the task itself has inherent ambiguity independent of model or human skill? A particularly informative sub-experiment would measure whether ROSALIA + human (human reviews and corrects model predictions) outperforms human alone on time-to-completion and accuracy, testing the paper's clinical workflow burden reduction claim directly.

Architecture optimization for presence/absence reasoning and multi-lesion disambiguation. The paper demonstrates that off-the-shelf LISA trained on MIMIC-ILS achieves strong performance, but leaves open whether the architecture can be improved for the specific demands of the ILS task. Two capabilities seem particularly worth optimizing. First, explicit presence/absence classification: rather than relying on the text generation module to output "There is no X in Y" and the mask decoder to output an empty mask as two independent predictions, an architecture could include a dedicated presence-detection head that makes an explicit binary decision about whether the requested lesion exists at the queried location, and conditions the mask decoder on this decision. This could improve both N-Acc. (by making absence predictions more deliberate) and segmentation quality on positive cases (by allowing the mask decoder to assume presence when the detection head signals positive). Second, lesion type disambiguation: the 36.7% pneumonia lesion inference accuracy (Table 14) is the single weakest result and arises from a known clinical ambiguity—pneumonia, atelectasis, and edema can appear similar on CXR. An architecture incorporating clinical prior knowledge (e.g., typical anatomical distributions for each lesion type, known co-occurrence patterns) could improve inference accuracy by providing the model with the same contextual information radiologists use. A strong experiment would compare ROSALIA against a variant that receives the patient's clinical history (fever, cough, post-operative status) as additional conditioning, testing whether the lesion inference gap can be closed by adding the non-imaging context that radiologists rely on.

Thin-slice reverse-engineering of negative sample necessity. The paper's central experimental claim is that negative samples are essential for empty-target accuracy—baselines achieve 0–22% N-Acc. while ROSALIA achieves 91.8%. But the evidence is circumstantial: the baselines differ from ROSALIA not just in training data composition but in architecture, pretraining, and domain. A clean ablation would train ROSALIA variants on MIMIC-ILS with varying fractions of negative samples removed: 0% negatives (positive-only), 25%, 50%, 75%, and 100% (the default). Plot gIoU and N-Acc. as a function of negative sample fraction. The prediction from the paper's framing is that N-Acc. should increase monotonically with negative fraction while gIoU remains stable (since positive samples are unchanged). If N-Acc. saturates at, say, 25% negative samples, the 7:1 ratio in MIMIC-ILS is overkill and training could be more efficient. If N-Acc. continues improving all the way to 100%, the negative sample engineering is genuinely load-bearing. Additionally, ablate the two negative generation strategies separately: absent-lesion negatives vs. empty-location negatives. These teach conceptually different things—global absence of a disease vs. localized absence at a specific anatomical site—and may contribute differently to N-Acc. on global vs. basic negative instructions (Table 6 already hints at this: 82.3% negative global accuracy vs. 96.9% negative basic accuracy).

Extension to longitudinal and multi-finding instructions. The ILS task as formulated handles single-lesion, single-timepoint queries. Clinical practice often involves comparing current and prior images ("Segment the pneumonia that is new since the previous exam") or handling multiple interacting findings ("Segment the effusion and the adjacent atelectasis"). Extending MIMIC-ILS and ROSALIA to these scenarios would test whether the cross-modal consistency paradigm scales beyond single-finding queries. For longitudinal queries, MIMIC-CXR contains paired prior exams with corresponding reports that describe temporal changes ("opacity has increased," "effusion is resolved"). A pipeline extension could generate paired instruction-answer data where the instruction references both the current and prior image, and the answer identifies the changed region. The experiment would measure whether a model trained on such data can identify which lesions are new, resolved, or stable—a capability that would directly address the clinical workflow of comparing serial CXRs. For multi-finding queries, the challenge is decomposing a compound instruction ("Segment the pneumonia and the effusion") into independent sub-queries while ensuring that overlapping or adjacent findings are correctly separated. This is non-trivial because the pipeline's current mask generation independently processes each reported finding, and the model must learn to output multiple distinct masks rather than one merged mask.

Negative finding: sensitivity analysis of the low-recall design under resource constraints. The paper's recall-rate ablation (Appendix A.7) demonstrates that 28.3% recall produces better downstream model performance than 47.3% recall, validating the high-precision, low-recall design. But this experiment keeps the dataset size roughly constant by including more images in the higher-recall condition. A different question—relevant to resource-constrained deployments—is: at what recall level does the dataset become too small to train a useful model? Systematically vary the thresholds from the default values down to the minimum that produces any masks at all, generating datasets with recall rates ranging from ~5% to the paper's 28.3%. Train ROSALIA on each and measure test-set performance. The hypothesis: there should be a critical recall rate below which the number of training samples is insufficient for the model to learn the instruction-following capability, and performance should degrade sharply. Identifying this threshold would provide practical guidance for applying the pipeline to smaller datasets than MIMIC-CXR (e.g., a single hospital's archive of 20K images): how many images are needed before the pipeline produces enough high-quality masks to train a useful model? The result would also distinguish whether the paper's 28.3% recall is near the knee of the precision-recall tradeoff curve or whether a much lower recall rate would suffice.

Practical Applications and Downstream Use Cases

Automated preliminary CXR screening with natural language querying. In settings where radiologist review is the rate-limiting step—emergency departments with long turnaround times, outpatient clinics sending CXRs for overnight reading, or low-resource settings with few or no radiologists—ROSALIA (or a model trained on MIMIC-ILS) could serve as a preliminary screening tool that non-radiologist clinicians interact with through natural language. A triage nurse or general practitioner could query the system with simple instructions ("Segment any opacities in the right lung," "Is there cardiomegaly?") and receive both a visual mask overlay and a structured text response. The 91.8% N-Acc. means the system reliably reports when findings are absent, reducing false alarms. The 71.2% gIoU means the system provides reasonable—though not perfect—delineation of any lesions found. Critically, the system can be deployed without requiring the user to interpret the CXR first: the instruction-to-mask direction means a clinician can ask whether something is present rather than describing what they see. The efficiency gain comes from reducing the time between image acquisition and initial clinical interpretation, particularly in off-hours when radiologists are not immediately available. The paper's inference time of ~0.45 seconds per query (Appendix F) supports interactive use, though deployment would require integration with PACS systems.

Training data generation for self-improving CXR AI pipelines. MIMIC-ILS is valuable not just as a training dataset for instruction-following models, but as a template for bootstrapping annotated data from raw image-report pairs in any institution with an archive of paired CXRs and radiology reports. A hospital system with 500K historical CXRs and corresponding reports could run the pipeline (accepting the 28.3% recall) to generate an institution-specific version of MIMIC-ILS. This dataset could then be used to fine-tune ROSALIA for that institution's specific patient population, imaging protocols, and reporting conventions. More ambitiously, the pipeline could be integrated into a continuous learning loop: as new CXRs are acquired and reported, the pipeline automatically generates training pairs, the model is periodically retrained, and the improved model assists radiologists who then produce reports that feed the next iteration. The key enabler is the fully automated nature of the pipeline—no human annotation is required at any step, so the cost of generating new training data is purely computational. The paper's demonstration that higher recall (47.3%) degrades performance (Appendix A.7) provides an important caution: such a loop must maintain strict quality thresholds or risk model degradation from noisy self-supervision.

Lesion burden quantification and progression tracking. Beyond detection and segmentation, ROSALIA's per-lesion masks could be used to quantify disease burden—for example, computing the percentage of lung volume occupied by edema or effusion—and tracking changes over serial CXRs. The per-lesion gIoU values (Table 5) indicate where this is most feasible: cardiomegaly at 89.0% gIoU enables reliable heart size measurement for tracking progression; edema at 64.8% gIoU provides a reasonable but noisier quantification; pneumonia at 57.2% gIoU is likely too coarse for precise burden measurement but could still support coarse trending (improving, stable, worsening). The empty-target capability is essential here: a quantification tool that hallucinates lesions when they are absent would produce dangerously misleading trend data. ROSALIA's 91.8% N-Acc. provides confidence that "lesion not present" reports are reliable. This application would require extending the ILS task to handle explicit measurement queries ("What percentage of the right lung is occupied by effusion?") and would benefit from the longitudinal multi-finding instruction extension described above. The key practical benefit is automating a task that is currently performed manually—radiologists visually estimate lesion size and describe changes qualitatively ("slightly increased") rather than quantitatively—potentially enabling more objective and reproducible disease monitoring.