Grasp Any Region: Towards Precise, Contextual Pixel Understanding for Multimodal LLMs

ArXiv: 2510.18876

🎯 Pitch

Cropping out image regions for LLM analysis destroys the global context needed to correctly interpret them—turning a frog slipper into a real frog. GAR fixes this by extracting region features directly from a globally-computed feature map, enabling a 1B model to beat 78B competitors on region-level VQA. It even transfers zero-shot to video understanding, outperforming dedicated video models despite never being trained on moving images.

---

1. Executive Summary

This paper introduces GAR (Grasp Any Region), a family of multimodal LLMs designed for comprehensive region-level visual understanding that shifts the paradigm from isolated, crop-based analysis to context-aware, multi-prompt compositional reasoning. Evaluated on a newly constructed benchmark suite called GAR-Bench — which measures single-region comprehension, multi-prompt interaction, and advanced compositional reasoning — GAR models built on the PerceptionLM series employ a novel RoI-aligned feature replay technique (extracting region features directly from a globally-computed feature map via RoI-Align, rather than processing cropped regions in isolation) to simultaneously preserve global scene context while maintaining fine-grained local detail. The paper demonstrates that GAR-1B outperforms DAM-3B by +4.5 points on DLC-Bench for detailed captioning, surpasses InternVL3-78B on GAR-Bench-VQA despite being 78× smaller, and — most strikingly — achieves zero-shot performance on VideoRefer-Bench Q that exceeds the in-domain VideoRefer-7B model, establishing that precise pixel-level understanding with necessary global context enables strong transfer to video domains even when trained solely on static images. The work further establishes that the benefits of global-context-aware region encoding are most pronounced on tasks requiring compositional reasoning across multiple prompts — such as recognizing non-entities (e.g., distinguishing a reflection in a mirror from a physical object) and modeling spatial relationships between objects — while current models, including GAR, still struggle with understanding complex relationships involving more than two objects and with capturing fine-grained temporal dynamics in videos with significant motion changes.

2. Context and Motivation

The Core Problem: Region-Level Understanding Without Global Context Is Fundamentally Broken

The paper addresses a deceptively simple question: how should a multimodal LLM understand a specific region within an image? The obvious answer — crop the region and analyze it independently — turns out to be deeply flawed. When you isolate a region from its surrounding scene, you strip away the contextual information that humans rely on to correctly interpret what they're looking at. The paper's central motivating example crystallizes this: a model analyzing a cropped region might see what looks like a frog and confidently label it as such, but the "frog" is actually a frog-shaped slipper — a distinction that becomes obvious only when you can see the bedroom floor, the bedframe, and the broader domestic context (Figure 2a). This isn't a contrived edge case; it's a fundamental failure mode that arises whenever visual understanding requires reasoning about how a local region relates to its global environment.

This gap matters because real-world visual understanding is inherently contextual. Objects don't exist in isolation — their identity, function, and state often depend on what surrounds them. A patch of blue pixels could be sky, water, a painted wall, or a reflection. A person's posture takes on meaning only in relation to the objects they're interacting with. A reflection in a mirror is categorically different from the physical object it depicts, even though the pixels within the mirror's boundary can be identical to those of a real object. Current region-level MLLMs, which largely operate by cropping out the region of interest and feeding it through the vision encoder in isolation, systematically fail on these kinds of questions because they throw away the very information needed to answer them.

Why This Problem Matters

The paper's motivation is both practical and scientific. On the practical side, the ability to understand arbitrary image regions is essential for a wide range of applications:

• Assistive technology: A visually impaired user might point to a specific object in a cluttered scene and ask "what am I holding?" or "is this safe to eat?" The model needs to understand the object in context, not in isolation.
• Robotics and embodied AI: A robot manipulating objects in a kitchen needs to understand spatial relationships ("the spatula is to the left of the stove"), distinguish physical objects from their reflections, and reason about how multiple objects interact.
• Interactive image editing and search: Users want to ask questions about specific parts of an image ("make this person's shirt blue" or "find all images where a dog is chasing a ball") that require precise, context-aware understanding of localized regions.
• Visual question answering in complex scenes: Many real-world questions are inherently about specific regions — "what is the person in the red jacket doing?" — and answering them requires integrating local detail with global context.

On the scientific side, the paper identifies a capability gap in the current MLLM paradigm. While holistic image-level understanding has advanced rapidly (models like GPT-4o, Gemini, and InternVL can answer general questions about entire images with impressive accuracy), the field lacks a systematic framework for region-level understanding that scales to compositional reasoning across multiple regions. The paper argues that this isn't just a matter of training on more data — it requires an architectural solution that fundamentally changes how region information flows through the model.

Where Prior Approaches Fall Short

The paper identifies several categories of prior work, each with specific limitations that GAR addresses:

Crop-based region MLLMs (DAM, PAM): isolation leads to context blindness. Recent models like DAM (Describe Anything Model, Lian et al., 2025) and PAM (Perceive Anything Model, Lin et al., 2025b) represent the state of the art in detailed localized captioning. Their core approach is to crop the region of interest from the image, encode it separately through the vision encoder, and feed the resulting features into the LLM. This design achieves strong performance on benchmarks that evaluate isolated region description — if you ask "describe this object in detail," the model has all the pixels it needs to discuss color, texture, shape, and material within the mask boundaries.

The fundamental weakness of this approach, which the paper demonstrates through both qualitative examples and rigorous benchmarking, is that cropping destroys the contextual information necessary for accurate interpretation. DAM-3B achieves 64.5 on DLC-Bench for detailed captioning using text-only judging (Table 3), but this metric masks the types of errors that arise from missing context — misidentifying objects that look like something else out of context, failing to understand an object's role or state in the scene, being unable to reason about relationships between objects. The paper provides concrete examples in Figure 5 where DAM-3B produces incorrect descriptions that GAR-8B corrects by leveraging global context.

The authors don't dismiss DAM's contributions — in fact, they build on DAM's training data pipeline (the Describe Anything-1.5M dataset) as a starting point for their own data generation. But they identify the architectural choice of isolated cropping as a structural limitation that cannot be fixed by scaling data or model size alone.

Pooled-feature approaches (Osprey, Ferret): global context at the cost of local detail. An alternative approach, used by models like Osprey (Yuan et al., 2024) and Ferret (You et al., 2023), is to encode the entire image once and then pool or aggregate features from the region of interest. This preserves global context because the features are computed over the full scene, but it introduces a different problem: loss of spatial resolution and fine-grained detail. When you average or pool features across a region, you lose the precise pixel-level information needed for detailed description — the subtle texture of a fabric, the exact shade of a color, the fine contours of an object's shape. The paper notes that these methods "suffer from insufficient details" (Section 1), making them inadequate for the fine-grained perceptual tasks that are the foundation of region-level understanding.

The paper also highlights that earlier region MLLMs like GPT4RoI (Zhang et al., 2024a), Shikra (Chen et al., 2023), Ferret, and RegionGPT (Guo et al., 2024) are "generally optimized to understand a given region in isolation, neglecting crucial global contexts." The experiments in Table 1 bear this out — these models score in the 22–27 range on GAR-Bench-VQA (averaged across all subtasks), compared to GAR-1B's 50.6 and GPT-4o's 59.7. On subjective captioning quality (GAR-Bench-Cap, Table 3), they score in the 22–27 range versus GAR-1B's 57.5–67.9. The gap is enormous, and the paper attributes it primarily to the architectural choice of how region features are extracted and contextualized.

Single-prompt limitation. A subtler but equally important gap: virtually all prior region MLLMs operate under a single-prompt paradigm — the user provides one region of interest, and the model describes it. This treats every query as an isolated analysis task. But many real-world questions require reasoning about the relationships between multiple regions: "what is the person in the blue shirt doing with the object on the table?" or "which of these three objects is closest to the camera?" Existing models have no mechanism for jointly encoding and reasoning about multiple heterogeneous prompts. GAR is explicitly designed to handle an arbitrary number of simultaneous prompts (the benchmark includes questions with up to 9 prompts, as shown in Figure 6b).

Conflicting Demands and the Need for a Unified Framework

The paper frames the architectural challenge as a tension between two competing requirements:

1. Local fidelity: The model must have access to high-resolution, detailed features from the prompted region to answer fine-grained questions about color, texture, shape, material, and category.
2. Global awareness: The model must simultaneously preserve information about the broader scene to resolve ambiguities, understand spatial relationships, recognize non-entities, and reason compositionally.

Crop-based methods satisfy requirement 1 but fail requirement 2. Pooled-feature methods satisfy requirement 2 but fail requirement 1. The paper's position is that no existing architecture satisfies both simultaneously, and that this gap explains the poor performance of region MLLMs on tasks requiring compositional reasoning. The observation that even GPT-4o — a model with presumably massive training data and strong general vision capabilities — scores only 59.7 on GAR-Bench-VQA (Table 1) while GAR-8B scores 54.5 suggests that this is a fundamental architectural problem, not merely a data scale problem.

How This Paper Positions Itself

GAR positions itself as the first region-level MLLM architecture to simultaneously provide sufficient local detail and preserve necessary global context, achieved through a novel RoI-aligned feature replay mechanism (detailed in Section 3.2). The paper explicitly contrasts this with the two dominant paradigms:

"Alternatives that employ pooled local features suffer from insufficient details. Therefore, a unified framework that can simultaneously resolve these issues to facilitate more sophisticated and interactive capabilities remains a significant area for investigation." (Section 1)

The contribution is primarily architectural, not driven by massive data scaling. The GAR-2.5M training dataset is relatively small by modern standards (the name suggests roughly 2.5 million samples, constructed from existing datasets like Describe Anything-1.5M, ImageNet-21K, and PSG). The fact that GAR-1B outperforms DAM-3B and even InternVL3-78B on targeted benchmarks despite this modest data scale is presented as evidence that the architectural innovation — not data volume — is the key driver of performance.

The paper also positions its benchmark, GAR-Bench, as filling a critical evaluation gap. The authors argue that existing region-level benchmarks (DLC-Bench, Ferret-Bench, MDVP-Bench) only evaluate single-region caption quality, typically using either reference-based language metrics (BLEU, ROUGE, CIDEr) or LLM-based judging of descriptions. These benchmarks fundamentally cannot measure a model's ability to reason about multiple regions, distinguish entities from non-entities, or perform compositional reasoning. GAR-Bench introduces these capabilities as first-class evaluation targets, with a particular emphasis on the "reasoning" protocol that includes position understanding, non-entity recognition, and multi-prompt relation reasoning.

The Video Transfer Finding as Motivation for the Architectural Claim

One of the paper's most striking results — that zero-shot GAR-8B outperforms in-domain VideoRefer-7B on VideoRefer-Bench Q (Table 7) — serves a dual purpose. First, it demonstrates practical transfer capability. But more importantly for the motivation, it validates the architectural thesis: if understanding regions in context is the key capability, and GAR's architecture genuinely learns this rather than memorizing image-specific patterns, then the same capability should transfer to video frames where the model has never seen training data. The fact that it does — and even exceeds specialized video models — is presented as evidence that the RoI-aligned feature replay mechanism captures something fundamental about context-aware visual understanding that generalizes beyond the training distribution.

However, the paper is careful to note the video transfer limitation: GAR-8B fails on temporally-dependent tasks like "temporal description" and "future prediction" (Tables 6 and 7, Figures 9 and 10). This is expected since GAR is trained only on static images, but it serves as an important boundary condition — the architecture solves the spatial context problem but does not (yet) address the temporal context problem. The paper frames this limitation explicitly in Section E, noting that "carefully collecting video training data is a potential solution."

3. Technical Approach

3.1 Reader Orientation

GAR (Grasp Any Region) is a multimodal LLM architecture designed to understand specific regions within an image — not in isolation, but grounded in the full scene context — and to reason about relationships between multiple such regions. It solves the problem that existing region-level MLLMs either crop regions out and lose global context (leading to misidentification errors like confusing a frog-shaped slipper for a real frog) or pool features from the full image and lose fine-grained detail. The solution's "shape" is a single-pass visual encoder that processes the entire image once, combined with a lightweight RoI-Align-based feature extraction step that pulls detailed features for each prompted region directly from the globally-computed feature map, feeding both global and local features into a language model for generation.

3.2 Big-Picture Architecture (Diagram in Words)

The GAR system has four major components connected in a feedforward pipeline:

1. Prompt Encoding Module — Takes raw binary mask prompts (one or more regions of interest) and converts them into spatial embeddings via a small convolutional block, then adds them to the vision transformer's patch embeddings using zero-initialized gating. This tells the vision encoder "which pixels matter" before the image is processed.

2. Vision Encoder (ViT with AnyRes) — Processes the full, uncropped image (with mask embeddings injected) through a Vision Transformer under the AnyRes dynamic-resolution scheme, producing a global feature map of the entire scene. This is the single shared representation from which all region features will be extracted.

3. RoI-Aligned Feature Replay Module — For each prompted region, derives a bounding box from the mask, applies RoI-Align to gather relevant feature vectors directly from the global feature map, and projects these into the LLM's token embedding space. These features are inherently context-aware because they come from a feature map computed over the whole image.

4. Large Language Model — Receives both the global visual tokens (from the full-image feature map) and the local visual tokens (from RoI-Align extraction for each prompt), interleaved with text tokens representing the instruction. The LLM performs all reasoning — description, comparison, relationship analysis — attending jointly over global context and local detail.

Information flows as follows: a user provides an image $I$ and a set of binary mask prompts $\{M_i\}_{i=1}^N$ → the prompt encoding module creates mask embeddings and adds them to the ViT's patch embeddings → the ViT processes the full image with AnyRes, producing a global feature map → for each mask, RoI-Align extracts a fixed-size feature grid from the global feature map → both global visual tokens and prompt-specific local tokens are concatenated with text tokens → the LLM autoregressively generates a text response $R$.

3.3 Roadmap for the Deep Dive

• First, the formal task definition (Section 3.1 of the paper), which establishes the three-level capability hierarchy that the architecture must support — single-region description, context-aware understanding, and multi-region compositional reasoning.
• Second, the prompt encoding mechanism — how raw binary masks are converted into spatial signals the vision transformer can attend to — since this is the entry point that conditions all downstream processing.
• Third, the RoI-aligned feature replay technique — the core architectural innovation — including exactly how features are extracted from the global feature map, why this avoids the context-detail tradeoff, and the latency/memory implications.
• Fourth, the training data pipeline (Section 3.3 of the paper), which builds from seed captioning data through fine-grained recognition enhancement to multi-prompt relational reasoning, because the architecture alone is insufficient without task-aligned training data.
• Fifth, the GAR-Bench design (Section 3.4 of the paper), which operationalizes the three capability levels into concrete, measurable benchmark protocols.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an architectural design and empirical evaluation paper whose core technical contribution is the RoI-aligned feature replay mechanism — a method for extracting region features from a globally-computed feature map that simultaneously preserves fine-grained local detail and necessary global context. The secondary contributions are a multi-stage training data pipeline that progressively teaches the model to reason about multiple regions, and a benchmark suite (GAR-Bench) that systematically evaluates region-level understanding beyond single-region captioning.

---

Task Formulation: A Three-Level Capability Hierarchy

The paper formalizes region understanding as a function (Section 3.1, Equation 1):

$R = \text{RegionModel}\left(I, \{M_i\}_{i=1}^N, T\right)$

where $I \in \mathbb{R}^{H \times W \times 3}$ is the input image with spatial dimensions $H \times W$, $\{M_i\}_{i=1}^N$ is a set of $N$ binary mask prompts where each $M_i \in \{0,1\}^{H \times W}$ specifies a region of interest, $T$ is a free-form text instruction, and $R$ is the generated text response demonstrating multi-layered comprehension of the scene.

What it computes: given an image, an arbitrary set of user-specified regions, and a natural language instruction, the function produces a text response that can describe, compare, or reason about those regions. The function is the same regardless of whether $N=1$ (single prompt) or $N>1$ (multiple prompts), and regardless of whether the instruction requests description, relationship analysis, or compositional reasoning.

Why this form: the formulation explicitly encodes the hierarchical nature of region understanding (the paper describes three ascending capability levels). Level 1 (single-region description) is the special case $N=1$. Level 2 (context-aware understanding) uses $N=1$ but requires the model to incorporate global context beyond the mask boundaries — the formulation implies this by having the full image $I$ as input, not just a crop. Level 3 (multi-region compositional reasoning) uses $N>1$ and requires the model to jointly reason about the relationship between prompts. The unified formulation means a single model architecture must handle all three levels without architectural switching — the model doesn't know in advance whether it's doing simple description or complex reasoning.

The paper specifically enumerates the three capability levels as a design constraint on the architecture:

1. Detailed single-region description ("describe <Prompt1> in detail") — requires accurate perception of fine-grained attributes strictly within the mask boundaries.
2. Context-aware understanding — moves beyond isolated analysis to aggregate information from the broader, unmasked scene. Critical for tasks like position identification ("the second from the left in the third row") and non-entity recognition (distinguishing a reflection from a physical object), where examining only the prompted pixels is insufficient.
3. Multi-region compositional reasoning — requires articulating spatial, functional, or interactive connections between different prompts. This is the most demanding capability because it requires the model to build a mental representation of how multiple scene elements relate.

---

Prompt Encoding: Injecting Spatial Guidance into the Vision Encoder

Before the vision encoder can attend to specific regions, the model needs a mechanism for telling the encoder "where to look." The paper adopts a lightweight prompt encoding scheme similar to DAM (Lian et al., 2025) and Alpha-CLIP (Sun et al., 2024).

Mechanism. Each binary mask $M_i \in \{0,1\}^{H \times W}$ specifies which pixels belong to the region of interest (1) and which do not (0). This mask is processed by a simple convolutional block (the paper cites LeCun et al., 1989 for the general idea of CNNs, but does not provide the exact kernel size, stride, or number of layers — it is described simply as a "simple convolutional block"). The convolutional block produces a mask embedding — a tensor that spatially aligns with the ViT's patch grid such that it can be added element-wise to the patch embeddings from the image.

The mask embedding is zero-initialized following the approach of Zhang et al. (2023). Zero-initialization means the convolutional block's weights are initialized such that, at the start of training, the mask embedding is identically zero regardless of the input mask. This is a deliberate choice: at initialization, the model behaves identically for any mask — it simply processes the full image — and the mask influence is learned gradually during training. This prevents the model from over-relying on the mask signal before it has learned to interpret it usefully.

The mask embedding is then added (not concatenated, not multiplied) to the ViT's patch embeddings. The ViT's patch embedding process works as follows: the image is divided into a grid of non-overlapping patches (typically $16 \times 16$ pixels per patch for a ViT-L/14 or similar), each patch is linearly projected to a fixed-dimensional vector, and these vectors form the input sequence to the transformer. The mask embedding is added to these patch embeddings — this means patches inside the prompted region receive a different embedding from patches outside, but the difference is a learned additive offset rather than a hard binary gating.

Why addition and not concatenation or multiplication? Addition preserves the dimensionality of the patch embeddings (no increase in sequence length or hidden dimension) and allows the model to learn continuous, soft spatial attention rather than hard masking. The zero-initialization ensures this additive signal doesn't corrupt the pretrained ViT features at the start of training — the model can choose to ignore the mask signal entirely by keeping the mask embedding weights near zero, or it can amplify the mask signal for tasks where region delineation matters. This is analogous to how positional encodings are added to token embeddings in transformers: the addition fuses two sources of information (content and position, or in this case, content and spatial guidance) without increasing the representational capacity of each individual token.

Interaction with AnyRes. The paper uses AnyRes (Liu et al., 2024), a dynamic-resolution scheme where the input image is split into multiple tiles (crops at different resolutions) plus a global downsampled view, and each tile is processed independently through the ViT. The paper sets max_num_tiles=16 by default, resulting in a maximum of 17 crops total (16 high-resolution tiles + 1 global view). The mask embedding is added to the patch embeddings for each crop that contains the prompted region. For crops that don't overlap with the mask, no mask embedding is added (or equivalently, the zero-initialized embedding remains zero). This means the vision encoder can attend to the prompted region at multiple resolutions simultaneously — the coarse global view provides context, while the high-resolution tiles provide detail.

---

RoI-Aligned Feature Replay: The Core Architectural Innovation

This is the paper's primary technical contribution. The problem it solves is: given a global feature map computed over the entire image (preserving context), how do you extract high-fidelity features for a specific region without losing either local detail or global awareness?

The challenge with alternatives. The paper's ablation study (Table 8) compares four approaches, which establish the design space:

• Local-only (\scriptsize{1}⃝): crop the region and process it through the ViT independently. This provides excellent local detail but completely loses global context — the model cannot see what surrounds the region. The paper shows this performs poorly on GAR-Bench tasks that require reasoning about relationships or context.

• DAM-like (\scriptsize{2}⃝): use zero-initialized gated cross-attention to fuse local features with global context from the full image (the approach used by DAM, Lian et al., 2025). This is an attempt to inject global information into a local processing stream, but the paper finds it "struggles at modeling multi-prompt relations, leading to poor results on GAR-Bench" (Table 8 caption). The cross-attention mechanism can attend to global features, but the local features themselves are computed without global context, so the model may not know which global features to attend to.

• Global + local crop (\scriptsize{3}⃝): simply encode both the full image and a cropped version of the prompted region, concatenating their features. The paper finds this approach "is superior at precise description on DLC-Bench" because the crop provides detailed local features, but still struggles with multi-prompt relations — likely because the model treats the crop and global features as separate inputs and must learn to align them in the LLM, which is a harder learning problem than having features that are inherently aligned in spatial coordinates.

The RoI-aligned feature replay mechanism. The paper's solution works as follows (Section 3.2, Figure 3):

Step 1: Single-pass global encoding. The full, uncropped image (with mask embeddings added to patch embeddings as described above) is processed through the ViT under AnyRes, producing a global feature map. This feature map is a grid of feature vectors — one per ViT patch location — where each vector encodes the visual content at that spatial position in the context of the entire scene. The self-attention layers of the ViT ensure that each patch's representation is influenced by all other patches, so the feature at a particular grid location contains information about what's around it, not just what's under that patch.

Step 2: Bounding box derivation. For each prompted mask $M_i$, the system derives a corresponding bounding box — the axis-aligned rectangle that tightly encloses all pixels where $M_i = 1$. This is a standard operation: find the minimum and maximum x and y coordinates of positive mask pixels. The bounding box provides a rectangular region in the spatial dimensions of the global feature map (the bounding box coordinates are scaled to the feature map resolution, accounting for the ViT's patch size and any downsampling).

Step 3: RoI-Align extraction. RoI-Align (He et al., 2017) is applied to extract a fixed-size feature grid from the global feature map for each bounding box. The key property of RoI-Align is that it performs bilinear interpolation at regularly sampled points within the bounding box, rather than rounding to the nearest grid cell (as in RoI-Pool) or quantizing coordinates. This means:

• The extracted features are differentiable with respect to the bounding box coordinates (though boxes are fixed here, not learned), which matters for gradient flow during training if the boxes were learned.
• The extracted features maintain sub-patch spatial precision — a feature can be sampled from a position that falls between patch centers, preserving the continuous spatial structure of the original feature map.
• The output is a fixed-size grid (the paper uses a $7 \times 7$ grid, which is the standard RoI-Align output size from Mask R-CNN, though this specific number is implied rather than explicitly stated for GAR — the ablation in Table 8 reports "ViT max tokens" suggesting the extracted features contribute a fixed number of visual tokens per prompt).

Step 4: Feature replay to LLM. The extracted RoI-Align features are projected through a linear layer (or MLP) into the LLM's token embedding space and inserted into the input sequence alongside the global visual tokens (from the full-image feature map) and the text tokens. The term "feature replay" in the paper's naming emphasizes that these features are not newly computed — they are "replayed" from the globally-computed feature map, meaning no additional ViT forward passes are needed for each new prompt.

Why this design solves the context-detail tradeoff. The critical insight is that RoI-Align features are extracted from a feature map that was computed with full-image self-attention. This means:

1. Local detail is preserved because RoI-Align samples features at sub-patch resolution within the bounding box — if the ViT operates at patch size 14 (ViT-L/14), a $7 \times 7$ RoI-Align grid can sample 49 feature vectors within a region that might span only a few patches, providing higher effective resolution than simply taking the patch features that fall within the box.

2. Global context is preserved because each feature vector in the global feature map already encodes information about the entire image through the transformer's self-attention layers. When RoI-Align samples a feature corresponding to the frog-shaped slipper, that feature vector was computed in the presence of the bedroom context — the ViT's attention mechanism could attend to the bed, the floor, and other contextual cues when building that feature. This is fundamentally different from cropping and re-encoding, where the ViT would only see the slipper pixels in isolation.

3. Multi-prompt scaling is efficient because the global feature map is computed once regardless of the number of prompts. Extracting features for additional prompts only requires RoI-Align operations (which are extremely cheap — essentially bilinear interpolation lookups) and linear projections, not additional ViT forward passes.

Latency and computational cost. The paper reports latency numbers in Table 8 (under "Avg Latency (ms)" for the first token generation). While the exact numbers are in the table, the key takeaway is that the RoI-aligned feature replay approach adds minimal latency compared to alternatives because the ViT forward pass (the dominant cost) is done once. The paper also reports "ViT max tokens" — the maximum number of visual tokens the ViT must process — which is lower for the RoI-aligned approach than for approaches that concatenate multiple separately-encoded crops, because AnyRes with a single global image produces fewer total tiles than encoding the global image plus multiple crops separately.

Design choice: RoI-Align vs. RoI-Pool. The paper specifically uses RoI-Align rather than the simpler RoI-Pool (which rounds coordinates to the nearest integer grid cell). This choice matters because the prompted regions can be very small — the average prompt area in GAR-Bench is only 4.4% of the image (Figure 6c), and some prompts are much smaller. RoI-Pool's quantization would introduce spatial misalignment for small regions, where rounding a coordinate by even one pixel can shift the sampled feature to a different patch entirely. RoI-Align's bilinear interpolation provides continuous spatial accuracy regardless of region size.

Design choice: feature replay vs. re-encoding. The paper's naming ("feature replay") emphasizes that features are extracted from an already-computed representation rather than computed fresh. An alternative would be to re-encode each cropped region through the ViT (the approach of DAM). The paper argues against this on two grounds: (1) computational cost — each additional prompt requires another ViT forward pass — and (2) context loss — re-encoded features don't benefit from full-image self-attention. The "replay" design means the ViT does the heavy lifting once, and the LLM receives features that are both context-rich and detailed.

Cross-model validation (Table 9). The paper validates that RoI-aligned feature replay is not specific to the PerceptionLM base model by testing it with Qwen2.5-VL-3B and InternVL3-2B as the vision-language backbone. The paper reports that the technique "consistently brings significant improvements over different base models," suggesting the design principle is architectural rather than model-specific.

---

Training Data Pipeline: Progressive Capability Building

The paper designs a multi-stage data generation pipeline (Section 3.3, Figure 4) that progressively builds model capability from basic single-region recognition to complex multi-region relational reasoning. The pipeline produces a dataset referred to as GAR-2.5M (the name implies approximately 2.5 million training samples, though the exact total is not explicitly stated in the paper — the components sum to roughly this amount).

Round 1: Enhancing Recognition Capability

Starting point. The pipeline begins with the Describe Anything-1.5M dataset (from DAM, Lian et al., 2025), which contains approximately 1.5 million region-level captioning examples. However, the authors observe that this dataset has "deficiencies in its fine-grained recognition capability, limiting the quality of generated captions for more complex scenarios." The issue is that the DAM dataset was generated using a seed captioner that may not have been trained on extremely fine-grained category distinctions — it might label a specific breed of dog simply as "dog" or fail to distinguish between visually similar object subcategories.

Fine-grained augmentation. To address this, the authors integrate images and masks from Alpha-CLIP (Sun et al., 2024), which provides a subset of ImageNet-21K (Deng et al., 2009). ImageNet-21K is a classification dataset with approximately 21,000 fine-grained categories — far more than standard datasets like COCO (80 categories) or even LVIS (~1,200 categories). The key property of this data is that it provides ground-truth category labels at extremely fine granularity — not just "bird" but specific species, not just "car" but specific makes and models.

Caption generation and verification. The process for Round 1 works as follows:

1. A seed captioner (a pre-trained region-level captioning model, likely the model trained on the Describe Anything-1.5M dataset) generates a descriptive caption for each Alpha-CLIP image-mask pair.
2. An LLM judge (not specified by name for this round, but the paper uses LLMs extensively — likely a smaller model than the 72B used in Round 2) compares the generated caption against the ground-truth ImageNet-21K category label. The LLM assesses whether the caption correctly identifies the object at the appropriate granularity — for instance, if the ground truth is "Brittany Spaniel" but the caption says "dog," the LLM flags this as insufficiently detailed.
3. Captions that pass the LLM's validation (i.e., those that accurately reflect the fine-grained category) are retained, producing a fine-grained dataset of 456K samples.

Why this matters. The Describe Anything-1.5M dataset alone would train a model that can describe regions at a coarse level ("a dog sitting on a couch") but may lack the precision for fine-grained tasks ("a Brittany Spaniel with a red collar"). The 456K fine-grained samples teach the model to be precise about object identity, which is foundational for all downstream capabilities — you cannot accurately reason about relationships between objects if you cannot first correctly identify what they are.

Training a fine-grained captioner. The combined dataset (1.5M + 456K ≈ 1.96M samples) is used to train a fine-grained captioner — an intermediate model that serves as the teacher for the next round of data generation. This is not the final GAR model; it's a tool for generating higher-quality training data.

Round 2: Supporting Multiple Prompts

Source data. The second round leverages the Panoptic Scene Graph (PSG) dataset (Yang et al., 2022), which contains images with annotated object masks and — crucially — labeled relationships between objects. A scene graph is a structured representation where nodes are objects and edges are relationships (e.g., "person - riding - bicycle," "cup - on - table"). PSG provides these annotations at the panoptic level (covering all objects in the scene, not just a subset), making it ideal for generating multi-prompt training data.

Data generation process. Round 2 produces three types of training data, using a pipeline that combines the fine-grained captioner from Round 1 with an LLM-Merger (Qwen2.5-72B, Team, 2024):

1. Relation-aware object descriptions (144K samples): For each object in a PSG scene, the fine-grained captioner generates a detailed description. Then, Qwen2.5-72B acts as an "LLM-Merger" — it takes the individual object descriptions, the PSG relationship annotations, and the original scene context, and rewrites the descriptions to explicitly integrate relational context. For example, instead of just "a bicycle" and "a person," the merged description might be "a person riding a blue bicycle." The key is that the description of each object now incorporates information about its relationship to other objects — this teaches the model that understanding a region requires looking beyond its boundaries.

2. Question-answering pairs (144K samples): From the same PSG annotations and merged descriptions, the LLM-Merger generates question-answer pairs designed to probe understanding of complex relationships. Example questions might be: "What is the person doing with the bicycle?" or "Which object is to the left of the table?" These are free-form QA pairs (not multiple choice) that require the model to synthesize information from multiple image regions.

3. Multiple-choice questions (126K samples): The LLM-Merger also generates multiple-choice questions, which are used both for training (to teach the model to reason under constrained options) and for the benchmark design (GAR-Bench-VQA uses multiple-choice format). The paper notes that these include questions designed to probe the understanding of complex relationships, with plausible distractors that test whether the model truly understands the scene or is just pattern-matching.

Total Round 2 data: 144K + 144K + 126K = 414K samples in the "Relation Dataset."

Total training data: ~1.96M (Round 1) + 414K (Round 2) ≈ 2.37M samples — the paper refers to this as GAR-2.5M.

Ablation of data components (Table 11). The paper validates the contribution of each data component by incrementally training models:

• Seed dataset only (Describe Anything-1.5M): baseline performance.
• + Fine-Grained Dataset-456K: improves DLC-Bench by +3.1 points, validating that fine-grained recognition enhancement matters for detailed captioning quality.
• + Relation Dataset-414K: enables the model to perform compositional reasoning with multiple prompts, as demonstrated by significant improvements on GAR-Bench (which specifically tests these capabilities).

Training configuration. The paper specifies the following training hyperparameters (Section C): the PerceptionLM series serves as the base model; supervised fine-tuning is performed using Xtuner (Contributors, 2023) with the AdamW optimizer (Loshchilov and Hutter, 2017); the global batch size is 64; the learning rate is 1e-5 with a cosine decay schedule (Loshchilov and Hutter, 2016). The paper does not specify the number of training epochs or steps, nor the maximum sequence length, warmup steps, or weight decay value — these are standard omissions in conference papers but would be needed for exact reproduction.

Two training rounds, not three. A subtle but important point: the paper describes two rounds of data generation (Round 1 for fine-grained recognition, Round 2 for multi-prompt relations), but the final GAR model is trained on all three data components simultaneously (Describe Anything-1.5M + Fine-Grained-456K + Relation-414K). The two rounds refer to the data generation pipeline, not to a two-stage training process. The model sees all data types in a single supervised fine-tuning phase.

---

GAR-Bench: Benchmark Design for Multi-Level Region Understanding

GAR-Bench (Section 3.4) is the evaluation framework the paper introduces to systematically measure region-level comprehension beyond single-region captioning. It is structured into two primary components, each with sub-protocols that target specific capabilities from the three-level hierarchy.

GAR-Bench-Cap: Multi-Prompt Captioning

This component evaluates the model's ability to generate cohesive descriptions that capture relationships between multiple visual prompts, going beyond isolated object descriptions.

• Simple protocol: The model is directly asked "what is the relationship between <Prompt1> and <Prompt2>?" and must answer concisely. This tests whether the model can identify and articulate the primary relational predicate connecting two regions (e.g., "riding," "holding," "on top of").

• Detailed protocol: The model is asked to generate detailed captions that include necessary relationships. For instance, given an image with a person (Prompt1) and a bike (Prompt2), the model is evaluated not on describing either independently, but on generating a description like "<Prompt1> is riding <Prompt2>." This requires spatial reasoning, action recognition, and semantic integration across disparate image regions — the model must understand the scene as a cohesive whole.

The evaluation uses GPT-4o as the judge (noted in Table 3) with the cropped images and masks provided, enabling multimodal judging rather than text-only judging. The paper criticizes text-only judging (used in the original DLC-Bench) as unreliable because semantic categories cannot be accurately adjudicated without visual context (elaborated in Section F, Figure 11).

GAR-Bench-VQA: Multi-Faceted Visual Question Answering

This component shifts evaluation from static description to dynamic, interactive dialogue, directly measuring comprehension through targeted questions. It is divided into two complementary sub-protocols:

Perception sub-protocol: Evaluates foundational visual acuity by asking targeted questions about the intrinsic visual properties of a single prompted region. The four attribute categories are:

• Color: "What is the primary color of <Prompt1>?"
• Shape: "What is the shape of <Prompt1>?"
• Material: "What material is <Prompt1> made of?"
• Texture/Pattern: "What texture or pattern does <Prompt1> have?"

These questions serve as a litmus test for core visual perception. A model that cannot correctly answer basic color questions about a clearly delineated region has fundamental perceptual deficits that will preclude more complex reasoning.

Reasoning sub-protocol: Probes higher-order cognitive abilities through three specialized sub-tasks:

• Position: Evaluates spatial arrangement and ordinal logic within global context. The model is presented with a single mask on an object within a larger group (e.g., one object in a grid-like arrangement of many similar objects) and asked to identify its precise position — for instance, "what is the position of <Prompt1>?" with the answer being "the second from the left in the third row." This requires processing the entire scene structure, not just the masked object. The paper sources images for this task from FSC-147 (Ranjan et al., 2021), a dataset featuring images with numerous countable objects often arranged in regular patterns.

• Non-Entity Recognition: Tests whether the model can leverage sufficient global context to distinguish physical objects from non-entities. The prompted region might highlight a reflection in a mirror, a shadow of a person, a face depicted on a television screen, or a painting on a wall. The model is asked whether the prompted region corresponds to a physical entity. Success demonstrates context-aware reasoning rather than simple pattern matching on masked pixels — the visual features of a reflection can be identical to those of a real object, so the model must use contextual cues (the presence of the mirror frame, the orientation of the scene) to answer correctly. The paper sources images for this task from RGBD-Mirror (Mei et al., 2021), which contains scenes with mirrors and reflections.

• Relation: Measures complex compositional reasoning across multiple prompts. The model is presented with several visual prompts and must deduce the intricate spatial or logical relationship between them. A key challenge in this sub-task is the inclusion of redundant prompts — the model is given more prompts than needed to answer the question, and must ignore the potentially distracting ones to arrive at the correct answer. For instance, to answer "what is the relationship between <Prompt0> and <Prompt2>?" the model might be given Prompts 0, 1, and 2, with Prompt 1 being irrelevant to the specific question. This tests whether the model can selectively attend to relevant regions while suppressing irrelevant ones — essentially, whether it can build a mental scene graph and query it flexibly. The paper sources images for this task from PSG (Yang et al., 2022), which provides scene graph annotations.

All VQA sub-tasks use a standardized multiple-choice format to ensure consistent and objective evaluation. The benchmark contains questions with varying numbers of prompts — the distribution (Figure 6b) shows most questions have 2-3 prompts, but there are even 2 questions with 7 and 9 prompts respectively, creating an extremely challenging test of multi-prompt reasoning.

Benchmark construction pipeline (Section B.1). The GAR-Bench data is constructed through a semi-automated pipeline combining foundation model generation with human expert curation:

1. Image selection: Source images are curated from PSG (for relation tasks), RGBD-Mirror (for non-entity recognition), FSC-147 (for position tasks), and SA-1B (Kirillov et al., 2023, for general images).

2. Mask labeling: High-quality segmentation masks are generated for all potential objects of interest, following a procedure similar to DenseWorld-1M (Li et al., 2025) that decomposes complex scenes while avoiding numerous meaningless trivial objects.

3. Object selection and annotation: A team of 8 MLLM experts reviews the generated masks and selects those meeting quality and task-relevance criteria. They manually annotate ground-truth answers for the reasoning protocol (relation, ordering, entity status) and ground-truth captions for GAR-Bench-Cap.

4. Automated attribute generation: For the perception protocol, Gemini-2.5-Pro (DeepMind, 2025b) generates lists of basic perceptual attributes (color, shape, material, texture) for each verified object mask.

5. Quality control: Human experts review all machine-generated attributes for factual correctness, filter out ambiguous or inaccurate labels, and transform raw annotations into standardized multiple-choice format for VQA and structured format for captioning evaluation.

6. Difficulty filtering: Any question answered correctly by all four state-of-the-art non-thinking MLLMs — Qwen2.5-VL-72B (Bai et al., 2025), InternVL3-78B (Zhu et al., 2025), GPT-4o (OpenAI, 2024a), and Gemini-2.5-Flash (DeepMind, 2025a) — is excluded. This ensures the benchmark remains challenging for advanced models and avoids ceiling effects.

Benchmark statistics (Section B.2, Figure 6). The final GAR-Bench has the following properties:

• Reasoning vs. perception split: 53% reasoning, 47% perception — slightly prioritizing advanced reasoning while maintaining a relatively balanced distribution (Figure 6a).
• Number of prompts: The majority of questions use 2-3 prompts, but the distribution extends to 7 and 9 prompts (Figure 6b), creating a long tail of extremely challenging multi-prompt questions.
• Prompt area: The mean area of prompts is only 4.4% of the image, with a sharp peak near 0.0 (Figure 6c), highlighting that most prompts correspond to very small regions — a deliberate choice to test fine-grained understanding at small scales.

4. Key Insights and Innovations

Innovation 1: The Local-Global Tradeoff Is Architectural, Not a Data Problem

The paper's most fundamental intellectual move is diagnosing why prior region-level MLLMs fail and arguing that the failure is structural rather than scalar. Before GAR, the dominant assumption in the field was that region understanding could be improved by scaling data and model size — if a crop-based model like DAM misidentifies objects, the solution is more training examples, a larger vision encoder, or a more powerful LLM. GAR challenges this assumption directly by framing the problem as an irreducible architectural tension: crop-based methods provide local detail but destroy global context, while pooled-feature methods preserve global context but lose local detail (Section 1, Section 3.2). No amount of additional training data can resolve this tension because the information needed for correct interpretation — the surrounding scene — is physically removed from the model's input in the crop-based paradigm.

This is a significant diagnostic contribution because it reorients the research agenda from scaling to architecture design. The evidence for this claim is not just GAR's performance advantage, but the specific pattern of failures it corrects. Figure 5 shows DAM-3B producing incorrect descriptions that cannot be fixed by better local recognition — the frog-shaped slipper does look like a frog locally — but are resolved when global context (the bedroom setting) is available. Table 1 shows that general MLLMs like GPT-4o, which see the entire image but have no specialized region-extraction mechanism, score only 59.7 on GAR-Bench-VQA compared to GAR-8B's 54.5 despite GPT-4o's massive scale advantage. The narrow gap between a 1B-8B specialized model and a frontier general model suggests that architecture, not scale, is the binding constraint for region-level tasks.

This framing is reminiscent of how the computer vision community once debated whether better object detection required more data or better region proposal mechanisms (the R-CNN family vs. end-to-end detection debate), with the eventual resolution being that architectural innovation in how regions are represented was essential. GAR makes the analogous argument for the MLLM era: the "crop and encode" paradigm has hit a ceiling imposed by its own design, and progress requires rethinking how region features are computed, not merely increasing the volume of crop-level training examples.

The finding that GAR's architecture generalizes across base models — working effectively with PerceptionLM, Qwen2.5-VL, and InternVL3 backbones (Table 9) — strengthens this claim. If the benefit were tied to a specific pretrained model's characteristics, it would be a scaling artifact. The fact that RoI-aligned feature replay improves all tested backbones suggests the principle is genuinely architectural rather than an interaction effect with particular pretrained features.

---

Innovation 2: RoI-Aligned Feature Replay as a Unifying Solution — Context and Detail Without Separate Processing Streams

While the diagnosis of the local-global tradeoff is conceptual, GAR's specific technical response — RoI-aligned feature replay — is an innovation in how multimodal architectures handle spatial queries. Prior to GAR, the design space for region-level MLLMs was bifurcated: either process regions separately (crop-based, e.g., DAM, PAM) or pool features from a shared representation (e.g., Osprey, Ferret). These two approaches correspond to fundamentally different feature computation graphs — in crop-based methods, each region triggers its own vision encoder forward pass with no shared computation; in pooling-based methods, there is one shared forward pass but features are aggregated coarsely.

GAR's innovation is to recognize that these are not the only two options. By computing a single global feature map (shared computation, context preserved) and then extracting region-specific features via differentiable, spatially-precise interpolation (RoI-Align), GAR achieves a third design point: shared context encoding with per-region fine-grained feature extraction. This is meaningfully different from both prior paradigms:

• Unlike crop-based methods, GAR does not re-encode regions separately, so the features extracted for any region were computed with full-image attention — the ViT's self-attention layers could attend to any part of the scene when building the feature at each spatial position.
• Unlike pooling-based methods, GAR does not aggregate features into coarse summary vectors. RoI-Align produces a grid of features (a $7 \times 7$ grid, based on the Mask R-CNN convention the paper follows) that preserves spatial structure within the region — the model knows where within the bounding box each feature came from.

The "replay" concept in the name captures an important efficiency insight: the ViT forward pass is the dominant computational cost, and it is performed once regardless of how many regions are queried. Extracting features for additional regions requires only bilinear interpolation and linear projection — operations that are negligible compared to transformer inference. This is not just an implementation detail; it enables the architecture to scale to arbitrary numbers of prompts without a proportional increase in latency. Table 8 reports average latency for first-token generation, and the RoI-aligned approach maintains lower latency than alternatives that concatenate multiple separately-encoded crops because AnyRes with a single global image produces fewer total tiles.

The ablation study (Table 8) is critical for establishing this as a genuine innovation rather than an obvious design choice. The paper tests three alternatives — local-only, DAM-like cross-attention fusion, and global + local crop concatenation — and shows that only the RoI-aligned approach simultaneously achieves competitive detailed captioning (DLC-Bench) and strong multi-prompt reasoning (GAR-Bench). The alternative that seems closest in spirit (global + local crop, \scriptsize{3}⃝) achieves the best DLC-Bench performance because the crop provides excellent local detail, but fails on GAR-Bench because the model must learn to spatially align the separately-encoded crop and global features in the LLM — a harder learning problem than having inherently aligned features from a shared feature map.

---

Innovation 3: Systematic Benchmarking of Multi-Prompt Compositional Reasoning as a First-Class Capability

Prior to GAR-Bench, region-level evaluation was almost entirely focused on single-region caption quality, measured through language-model-based judging (DLC-Bench, Ferret-Bench) or reference-based captioning metrics (MDVP-Bench). These benchmarks implicitly operationalize "region understanding" as "can you describe this isolated region well?" — a definition that GAR's own results show is incomplete. A model can score highly on DLC-Bench (DAM-3B achieves 64.5–72.6 in Table 3) while being fundamentally unable to reason about relationships between regions or to use global context to disambiguate local appearance.

GAR-Bench makes two conceptual contributions to evaluation methodology:

First, it operationalizes the distinction between perception and reasoning as separate evaluation axes. The perception sub-protocol (color, shape, material, texture) tests whether the model can extract basic visual attributes — this is a capability floor. The reasoning sub-protocol (position, non-entity recognition, relation) tests whether the model can synthesize information across spatial scales and multiple image regions — this is the capability ceiling. By separating these, GAR-Bench reveals capability profiles that single-number captioning metrics obscure. For instance, Table 1 shows that GAR-8B achieves 75.9 on texture perception versus 54.5 overall, indicating that its perceptual capabilities are strong but reasoning — particularly with multiple prompts — remains challenging.

Second, it introduces deliberately adversarial evaluation design through redundant prompts and non-entity test cases. The inclusion of non-entity recognition (Section 3.4, "Reasoning" protocol) is particularly clever as an evaluation design choice. By querying whether a prompted region corresponds to a physical entity — using images from RGBD-Mirror where mirrors and reflections create natural ambiguities — the benchmark directly tests whether the model is performing context-aware reasoning or local pattern matching. A crop-based model would see identical pixel patterns for a reflection and a real object; only by incorporating global context (the mirror frame, the spatial layout of the scene) can the model distinguish them. This is a diagnostic test for the specific capability the paper's architecture is designed to provide.

The difficulty filtering step (Section B.1) — removing questions that all four frontier MLLMs answer correctly — ensures the benchmark measures genuine capability gaps rather than basic competence. The fact that even GPT-4o achieves only 59.7 overall on GAR-Bench-VQA (Table 1) despite being a much larger and more capable general model than GAR-8B (which scores 54.5) confirms that the benchmark tests region-specific reasoning skills that are not trivially solved by scaling general vision-language capability.

The inclusion of questions with up to 9 prompts (Figure 6b) and an average prompt area of only 4.4% (Figure 6c) creates a challenging evaluation regime that pushes beyond what any existing benchmark measures. The closest comparable benchmarks — DLC-Bench, Ferret-Bench — never present more than one prompt and use regions that likely cover a larger fraction of the image (the paper does not report this statistic for those benchmarks, but the GAR-Bench statistics highlight the deliberate choice to target small regions).

---

Innovation 4: Strong Zero-Shot Video Transfer as Validation of the Architectural Hypothesis

The paper's most striking empirical result — that zero-shot GAR-8B outperforms in-domain VideoRefer-7B on VideoRefer-Bench Q (Table 7) — functions as more than a performance highlight. It serves as an existence proof for the claim that context-aware region understanding, learned from static images, transfers to the video domain without video-specific training. This is a non-obvious finding that challenges the assumption that video region understanding requires video-specific architectures or temporal modeling.

The logic of this result as supporting evidence for the paper's architectural thesis runs as follows: if GAR's RoI-aligned feature replay genuinely teaches the model to understand regions in the context of their surrounding scene — rather than memorizing image-specific patterns from the training data — then this capability should transfer to video frames, which are simply images arranged sequentially. The fact that it does transfer, and even exceeds a model specifically trained on 700K video samples (VideoRefer-700k), suggests that the architecture learns something general about context-aware visual reasoning rather than fitting to the distribution of training images.

However, the paper is careful to delineate the boundary of this transfer. Tables 6 and 7 show that GAR-8B performs poorly on explicitly temporal tasks — "temporal description" (TD) in Table 6 and "future prediction" in Table 7. This is expected and actually strengthens the architectural argument by negative example: GAR transfers well to video tasks that require spatial reasoning about regions in frames (which is what the architecture addresses), but fails on tasks that require understanding frame-to-frame dynamics (which the architecture has no mechanism for, since it processes each frame independently with no temporal attention). This dissociation — strong spatial transfer, weak temporal transfer — is exactly the pattern one would predict if the architectural innovation (context-aware spatial feature extraction) is the causal factor, rather than some unobserved confound like better pretraining.

The video transfer result also has practical implications that elevate it beyond an architectural validation. It suggests that for applications requiring spatial region understanding in videos (e.g., video surveillance, sports analysis, instructional video understanding), one may not need to collect expensive video-level region annotation data — training on static images with the right architecture may suffice, at least for spatial reasoning tasks. The paper does not develop this application direction, but the result opens it as a promising avenue for future work.

This finding is reminiscent of how ImageNet-pretrained CNNs were found to transfer well to video action recognition (with temporal pooling) despite being trained on static images — the spatial features learned from images generalize, even if temporal modeling requires additional architectural components. GAR demonstrates the analogous phenomenon for region-level MLLMs, with the added significance that the transfer is zero-shot and surpasses in-domain training.

---

Innovation 5: Verifying That Cropped-Image Evaluation (DLC-Bench's Text-Only Judging) Is Systematically Unreliable

The paper makes a meta-contribution to evaluation methodology through its critique of the original DLC-Bench judging protocol (Section F, Figure 11). The original DLC-Bench uses a text-only LLM judge (LLaMA3.1-8B) that compares generated captions against ground-truth category names using only the text — the image is not shown to the judge. The paper demonstrates that this protocol is fundamentally unreliable because it cannot resolve semantic ambiguities that require visual context.

Figure 11 provides concrete examples: when the ground-truth category is "broom" and the generated caption describes "a wooden stick with bristles," a text-only judge cannot determine whether this description is correct — it must see the image to know whether the object is genuinely a broom or something else that matches the same textual description. The paper argues that "the image is the only ground truth capable of resolving this ambiguity" (Section F), and demonstrates that providing cropped images and masks to a multimodal judge (GPT-4o) eliminates these systematic errors.

This is an important methodological finding because DLC-Bench is the primary benchmark for detailed localized captioning, and many prior works (including DAM and PAM) report results using the original text-only judging protocol. GAR's own DLC-Bench results are reported under both protocols — text-only judging (Table 3, top section) and GPT-4o with cropped images (Table 3, † rows) — and the scores differ substantially. DAM-3B scores 64.5 under text-only judging but 72.6 under multimodal judging; GAR-1B scores 67.9 and 77.1 respectively. The fact that multimodal judging consistently produces higher scores for all models suggests that text-only judging is not just noisy but systematically biased — it penalizes correct descriptions that use wording not anticipated by the category label, while potentially rewarding incorrect descriptions that happen to use the right words.

This finding is less flashy than the architectural innovations or the video transfer result, but it is practically important for the research community: it establishes that region-level captioning evaluation requires multimodal judges, and that results reported under text-only protocols should be interpreted with caution. The paper's practice of reporting both protocols and explicitly discussing the limitation sets a methodological standard that future work in this area should follow.

5. Experimental Analysis

Evaluation Methodology

• Dataset. GAR-Bench is a newly constructed benchmark suite introduced in this paper (Section 3.4, Appendix B). It is structured into GAR-Bench-Cap (multi-prompt captioning with both a "simple" relationship description protocol and a "detailed" description protocol) and GAR-Bench-VQA (multiple-choice visual question answering with two sub-protocols: "perception" covering color, shape, material, and texture/pattern attributes for single regions; and "reasoning" covering position understanding, non-entity recognition, and multi-prompt relation reasoning). Source images are curated from PSG (Yang et al., 2022), RGBD-Mirror (Mei et al., 2021), FSC-147 (Ranjan et al., 2021), and SA-1B (Kirillov et al., 2023). The benchmark is constructed through a semi-automated pipeline combining foundation model generation (Gemini-2.5-Pro for attribute generation) with curation by 8 MLLM experts, followed by difficulty filtering that removes any question answered correctly by all four of Qwen2.5-VL-72B, InternVL3-78B, GPT-4o, and Gemini-2.5-Flash. Statistics are shown in Figure 6: reasoning tasks slightly outweigh perception (53% vs. 47%), questions contain up to 9 prompts with a long tail of multi-prompt cases, and the average prompt area is only 4.4% of the image, concentrated near zero. In addition to GAR-Bench, the paper evaluates on DLC-Bench (Lian et al., 2025) for detailed localized captioning, Ferret-Bench (You et al., 2023) and MDVP-Bench (Lin et al., 2025a) for zero-shot region captioning evaluation, LVIS (Gupta et al., 2019) and PACO (Ramanathan et al., 2023) for open-class category-level image recognition, V* (Wu and Xie, 2024), MMVP (Tong et al., 2024b), RealWorldQA (xAI, 2024), and MMStar (Chen et al., 2024a) for general multimodal benchmark evaluation, and VideoRefer-Bench D and VideoRefer-Bench Q (Yuan et al., 2025b) for video-level region captioning and understanding.

• Base model(s). GAR is built on the PerceptionLM series (Cho et al., 2025), which the paper describes as demonstrating "strong perception capabilities among several open-source MLLMs." The paper reports results for two model scales: GAR-1B and GAR-8B (the suffix indicates the approximate parameter count of the underlying LLM). For architectural ablation, the paper also tests Qwen2.5-VL-3B (Bai et al., 2025) and InternVL3-2B (Zhu et al., 2025) as alternative vision-language backbones.

• Metrics. For GAR-Bench-VQA, accuracy is measured as the percentage of multiple-choice questions answered correctly, with breakdowns reported per sub-protocol (color, shape, material, texture/pattern, position, non-entity, relation) and an overall average. For GAR-Bench-Cap, GPT-4o is used as a multimodal judge (provided with cropped images and masks) to evaluate the consistency and correctness of generated captions against ground-truth annotations, with scores reported as "Avg." (average), "Pos." (positive/consistent), and "Neg." (negative/inconsistent). For DLC-Bench, both LLaMA3.1-8B text-only judging and GPT-4o multimodal judging (with cropped images) are reported, producing an average score. For Ferret-Bench and MDVP-Bench, GPT-4o serves as judge. For LVIS and PACO, semantic similarity and semantic IoU are reported following the evaluation protocol of Osprey (Yuan et al., 2024). For VideoRefer-Bench D, caption quality is evaluated with metrics including overall score and temporal description (TD) sub-score. For VideoRefer-Bench Q, multiple-choice accuracy is reported per question category.

• Baselines. The paper compares against an extensive set of models organized into categories. Private general MLLMs: Gemini-2.5-Pro (DeepMind, 2025b), GPT-4o (OpenAI, 2024a), OpenAI-o1 (OpenAI, 2024b), OpenAI-o3 (OpenAI, 2025), Gemini-2.5-Flash (DeepMind, 2025a). Public general MLLMs: Qwen2.5-VL-72B (Bai et al., 2025), InternVL3-78B (Zhu et al., 2025), LLaVA-1.5 (Liu et al., 2023). Region-level MLLMs: GLaMM (Rasheed et al., 2024), GPT4RoI-7B (Zhang et al., 2024a), Osprey (Yuan et al., 2024), Shikra-7B (Chen et al., 2023), Ferret-7B (You et al., 2023), RegionGPT-7B (Guo et al., 2024), OMG-LLaVA (Zhang et al., 2024b), VP-SPHINX-13B (Lin et al., 2025a), Sa2VA (Yuan et al., 2025a), DAM-3B (Lian et al., 2025), and PAM-3B (Lin et al., 2025b). On video benchmarks: LLaVA-OneVision (Li et al., 2024), Qwen2-VL (Wang et al., 2024b), InternVL2 (Chen et al., 2024c), Elysium (Wang et al., 2024a), Artemis (Qiu et al., 2024), and VideoRefer-7B (Yuan et al., 2025b). For box-level MLLMs that do not natively accept masks, masks are transformed to bounding boxes.

• Generation budget / compute accounting. The paper does not report a standardized compute budget (e.g., FLOPs, generation count) and does not perform a compute-matched comparison between methods. All comparisons are at the level of final task accuracy using each model's default inference configuration, which varies significantly in computational cost across model scales. The paper reports latency numbers for architectural variants in Table 8 ("Avg Latency (ms)" to first token and "ViT max tokens"), but these are used only for relative comparison within the ablation study and not as a primary axis of evaluation.

• Cross-validation / statistical protocol. No cross-validation or statistical significance testing is reported. GAR-Bench's construction includes a difficulty filtering step that removes questions answered correctly by all four frontier MLLMs, but this is a data curation step rather than a statistical protocol for model evaluation. The benchmark is used as a fixed test set, and results are reported as point estimates without confidence intervals or error bars.

Main Quantitative Results

Multi-Prompt Compositional Reasoning (GAR-Bench-VQA)

The headline result for GAR-Bench-VQA appears in Table 1. GAR-8B achieves an overall score of 54.5, surpassing the private, state-of-the-art non-thinking model GPT-4o, which scores 59.7 — meaning GAR-8B trails GPT-4o by only 5.2 points despite being a dramatically smaller open-source model. More strikingly, GAR-1B scores 50.6 overall, outperforming InternVL3-78B (which scores 44.0) by 6.6 points, despite InternVL3-78B being approximately 78 times larger. The paper describes this as demonstrating "advanced comprehension capability in modeling interactions between multiple prompts."

Breaking down by sub-protocol, the pattern reveals specific strengths and weaknesses. On fine-grained perception tasks, GAR models are particularly strong: GAR-1B achieves 69.0 on "Texture" and GAR-8B achieves 75.9 (Table 1). These perceptual sub-scores substantially exceed the overall average, indicating that perception of basic attributes is not the bottleneck. The overall score is pulled down by reasoning sub-tasks, particularly those requiring complex multi-prompt compositional reasoning. The paper does not report full per-subtask breakdowns for all baselines in Table 1, making it difficult to assess exactly where GAR's advantages and disadvantages lie relative to GPT-4o — the overall score comparison is provided, but the sub-protocol performance profiles that would reveal capability specialization are not visible for the private models.

Multi-Prompt Relational Captioning (GAR-Bench-Cap)

Table 3 (confusingly numbered — the paper uses "Table 1" for VQA, "Table 2" for Cap, and then "Table 3" for DLC-Bench in the main text) reports relational captioning evaluation. Under text-only judging, GAR-1B scores 67.9 (average) compared to DAM-3B's 64.5 — a gap of 3.4 points. Under multimodal judging with GPT-4o provided cropped images and masks, GAR-1B scores 77.1 compared to DAM-3B's 72.6 — a gap of 4.5 points. GAR-8B achieves 67.4 (text-only) and 77.0 (multimodal), nearly identical to GAR-1B despite the 8× scale difference, suggesting that for these tasks, model scale beyond 1B provides minimal benefit when the architecture is optimized. Notably, GAR-1B and GAR-8B achieve the highest overall scores, exceeding private models like Gemini-2.5-Pro (55.8 text-only) and GPT-4o (61.5 text-only).

The paper reports three metrics for GAR-Bench-Cap: "Avg." (average across positive and negative cases), "Pos." (score on cases where the description should be correct), and "Neg." (score on cases where the description should indicate something incorrect or absent). GAR-1B achieves 48.9 on positive cases and 87.0 on negative cases under text-only judging. The high negative score (87.0) suggests that GAR is good at avoiding hallucinated relationships, while the moderate positive score (48.9) indicates that correctly identifying the precise relationship between prompted regions remains challenging.

Detailed Localized Captioning (DLC-Bench)

Table 3 reports DLC-Bench results. Under the original text-only judging protocol (LLaMA3.1-8B), GAR-1B achieves 67.9 and GAR-8B achieves 67.4, compared to DAM-3B's 64.5 and PAM-3B's results (not explicitly provided in the paper's comparison table, but cited in the text). The paper notes that the original DAM-3B score of 64.5 is from their own evaluation using the official checkpoint (marked with ‡). Under multimodal judging with GPT-4o (marked with †), all scores increase substantially: GAR-1B reaches 77.1, GAR-8B reaches 77.0, and DAM-3B reaches 72.6. This uniform inflation under multimodal judging is consistent with the paper's argument in Section F that text-only judging systematically underestimates caption quality due to semantic ambiguity resolution failures.

A noteworthy pattern: GAR-1B and GAR-8B perform nearly identically on DLC-Bench (67.9 vs. 67.4 text-only, 77.1 vs. 77.0 multimodal). This suggests that detailed captioning quality is not bottlenecked by LLM capacity at the 1B scale for this dataset. However, the paper does not report DLC-Bench results for larger general MLLMs like InternVL3-78B or GPT-4o (which lack specialized region-level mechanisms), so it's impossible to assess the ceiling or determine whether GAR's performance is near the frontier or significantly below it.

Zero-Shot Evaluation on External Region Captioning Benchmarks (Ferret-Bench, MDVP-Bench)

Table 5 (actually numbered as "Table 4" in the paper — the tables in the main experimental section have inconsistent numbering) reports zero-shot results on Ferret-Bench and MDVP-Bench. The paper notes that all results are their own reproduction using official checkpoints since the original judger (GPT-4V) is no longer available, and they use GPT-4o as the replacement judge. On both benchmarks, GAR emerges as the top-performing model across every single category. On MDVP-Bench, GAR-8B achieves 178.6 on natural images — "a result that is substantially higher than any competitor" (Section 4). GAR-1B also leads across categories, demonstrating that even the smaller model achieves state-of-the-art zero-shot region captioning.

The "zero-shot" nature of these results is important: GAR was not trained on Ferret-Bench or MDVP-Bench data, and the masks for MDVP-Bench are produced by SAM (Kirillov et al., 2023) conditioned on bounding boxes (since MDVP-Bench provides boxes rather than masks). This tests generalization beyond the training distribution and suggests the architectural approach is not overfit to specific mask qualities or dataset characteristics.

Open-Class Category-Level Image Recognition (LVIS, PACO)

Table 5 reports results on LVIS and PACO, following Osprey's evaluation protocol. GAR-8B establishes new state-of-the-art results: 93.6 semantic similarity and 88.7 semantic IoU on LVIS, and 95.5 semantic similarity and 91.8 semantic IoU on PACO. The consistent improvement across both metrics on both datasets indicates that GAR's region features are simultaneously good at semantic understanding (capturing what category an object belongs to) and precise localization (capturing exactly where the object is). The paper interprets this as evidence of "the effectiveness of GAR for complex recognition tasks, showcasing its robust performance in identifying a diverse range of object categories."

General Multimodal Benchmarks

Table 11 reports performance on V*, MMVP, RealWorldQA, and MMStar. For evaluation, mask is set to 1 (effectively treating the entire image as the "prompted region," reducing the model to a general MLLM). GAR-8B substantially outperforms other region-level models DAM-3B and PAM-3B across all benchmarks. Specific numbers are visible in the table but not quoted in the main text — the paper states that GAR-8B "outperforms them by a large margin" (Section C). This demonstrates that GAR's architectural improvements do not come at the cost of general multimodal understanding — the model remains competitive on standard holistic benchmarks even when operating in a mode it wasn't specifically trained for.

Video Transfer Results

Tables 6 and 7 report video results on VideoRefer-Bench D (detailed localized video captioning) and VideoRefer-Bench Q (detailed video understanding), respectively.

Detailed video captioning (Table 6): Under the zero-shot setting, GAR-8B surpasses DAM-8B across most metrics. For the "single-frame" configuration (target frame with AnyRes, max_num_tiles=16), GAR-8B and DAM-8B are compared directly. For the "multi-frame" configuration (16 uniformly sampled frames, AnyRes turned off), GAR models again outperform DAM. However, the paper notes that GAR gets "reasonably low scores on temporally related tasks, e.g., temporal description (TD)," because the models are trained only on static images and lack fine-grained temporal comprehension capabilities.

Detailed video understanding (Table 7): This is the paper's most striking result. Zero-shot GAR-8B outperforms VideoRefer-7B, which was trained on in-domain VideoRefer-700k data (700K video samples with region annotations). The paper does not quote the exact overall scores in the main text — they appear in Table 7 — but states that "our zero-shot GAR-8B even outperforms in-domain VideoRefer-7B, demonstrating its strong comprehension capabilities can be easily transferred to videos." However, on "future prediction" tasks (which require reasoning about what will happen next in the video), GAR-8B performs worse than VideoRefer-7B due to the lack of temporal modeling. This dissociation — strong on spatial reasoning, weak on temporal dynamics — is consistent across Tables 6 and 7.

Qualitative Results

Figure 5 provides qualitative comparisons between GAR-8B and DAM-3B on DLC-Bench. The examples demonstrate that GAR-8B produces more accurate descriptions, particularly in cases "when the category of the given prompt can be determined only when understanding sufficient global contexts." The paper highlights green text for correct descriptions and red for errors, making the comparison visually direct.

Figures 7 and 8 provide qualitative comparisons on GAR-Bench's "relation" and "non-entity recognition" protocols, respectively. In the relation examples (Figure 7), GAR-8B is shown to correctly identify subtle details that larger models miss — for instance, recognizing that a person looking at the camera is "holding" rather than "reading" a book, while both Gemini-2.5-Pro and OpenAI-o3 incorrectly select "reading." The failure cases (bottom rows of both figures) illustrate remaining challenges: models struggle with complex relationships involving more than two objects, and with distinguishing reflections in mirrors from reflections on other surfaces.

Figures 9 and 10 show video transfer qualitative results. GAR-8B generates detailed and precise descriptions for static or slow-motion videos but fails on "temporal description" when significant motion changes are present (Figure 9). Similarly, GAR-8B handles basic motion understanding (sequential, relation, and reasoning questions) but fails on "future prediction" with significant motion changes (Figure 10).

Ablation Studies and Robustness Checks

Architectural design (RoI-aligned feature replay vs. alternatives): Table 8 compares four architectural variants on GAR-Bench (multiple metrics) and DLC-Bench. The variants are: (1) only local images (crop-based), (2) DAM-like architecture with zero-initialized gated cross-attention to preserve context, (3) simply cropping local images as a supplement to global images (global + local crop concatenation), and (4) the proposed RoI-aligned feature replay. The key finding is that all alternatives "struggle at modeling multi-prompt relations, leading to poor results on GAR-Bench," while only the RoI-aligned approach simultaneously achieves competitive DLC-Bench performance and strong GAR-Bench performance. Variant (3) is "superior at precise description on DLC-Bench" because the crop provides excellent local detail, but it fails on GAR-Bench due to the difficulty of spatially aligning separately-encoded crop and global features. The paper also reports average latency (ms) to first token and maximum ViT tokens for each variant, showing that the RoI-aligned approach maintains lower latency than alternatives that concatenate multiple separate encodings.

Cross-model generalization of RoI-aligned feature replay: Table 9 extends the architecture ablation to alternative base models — Qwen2.5-VL-3B and InternVL3-2B. The RoI-aligned feature replay approach "consistently brings significant improvements over different base models," confirming that the design principle is architectural and not specific to PerceptionLM's pretrained features.

Data component ablation: Table 11 (data ablation) incrementally adds data components to a 1B model. Starting from the seed Describe Anything-1.5M dataset: adding the Fine-Grained Dataset-456K improves DLC-Bench by +3.1 points, validating that fine-grained recognition matters for captioning quality. Further adding the Relation Dataset-414K enables compositional reasoning with multiple prompts, as demonstrated by significant improvements on GAR-Bench metrics (specific numerical improvements are visible in the table). The paper notes that "each component of our data plays a significant role."

PerceptionLM base model vs. alternatives: The paper implicitly ablates the choice of base model through Tables 1, 3, and 9. The use of PerceptionLM as the base, combined with GAR's architectural modifications and training data, produces models that outperform general MLLMs many times larger on targeted region-level tasks. The cross-model validation in Table 9 provides direct evidence that the GAR architecture generalizes, but the full GAR training pipeline (data + architecture) is only evaluated with PerceptionLM as the base, so the strongest claims are specific to this combination.

AnyRes tile count and resolution: Table 8 reports results with max_num_tiles=16 for AnyRes, resulting in a maximum of 17 crops total (16 high-resolution tiles + 1 global view). The paper does not ablate different tile counts or resolution settings, so the sensitivity to these hyperparameters is unknown.

Multimodal vs. text-only judging on DLC-Bench: Section F and Figure 11 provide a qualitative and quantitative ablation of the judging protocol. The paper demonstrates that text-only judging (LLaMA3.1-8B) produces systematically unreliable results due to semantic ambiguity in language space without visual grounding. Table 3 reports results under both protocols (text-only vs. GPT-4o with cropped images), showing that multimodal judging produces uniformly higher scores across all models tested. This is presented as evidence that text-only judging underestimates true captioning quality, but the paper does not provide a ground-truth human evaluation to validate which judging protocol is more accurate relative to human judgment.

Difficulty filtering of GAR-Bench: Section B.1 describes that any question answered correctly by all four frontier MLLMs was excluded from GAR-Bench. No ablation is reported on the effect of this filtering — for instance, how many questions were removed, how model rankings change with vs. without filtering, or whether the filtering disproportionately affects certain sub-protocols.

Effect of model scale: The paper reports results for both 1B and 8B GAR models across most benchmarks. A notable pattern: GAR-1B and GAR-8B perform nearly identically on DLC-Bench (67.9 vs. 67.4 text-only, 77.1 vs. 77.0 multimodal in Table 3) and GAR-Bench-Cap (67.9 vs. 67.4 in Table 2), but GAR-8B substantially outperforms GAR-1B on GAR-Bench-VQA (54.5 vs. 50.6 in Table 1). This suggests that detailed captioning saturates quickly with model scale, while compositional reasoning continues to benefit from larger LLMs. The paper does not explicitly discuss this differential scaling pattern, and the absence of a GAR variant larger than 8B leaves open the question of whether further scale would close the remaining gap to GPT-4o's 59.7 on GAR-Bench-VQA.

Critical Assessment

How Well Do the Experiments Support the Central Claims?

Claim 1: GAR achieves precise perception by leveraging necessary global contexts.

The evidence for this claim comes primarily from qualitative comparisons (Figure 5) and the DLC-Bench results (Table 3), where GAR-1B outperforms DAM-3B by +4.5 points under multimodal judging. The architectural ablation (Table 8) demonstrates that local-only and DAM-like approaches underperform GAR's RoI-aligned feature replay, supporting the causal link between architecture and performance. However, the claim that GAR "achieves precise perception" is somewhat underspecified. The experiments demonstrate relative improvement over crop-based baselines, but they do not establish an absolute standard for precision — there is no upper-bound measurement or human performance comparison. The ablation comparing global + local crop concatenation (variant 3) to RoI-aligned feature replay is particularly informative: variant 3 achieves competitive DLC-Bench performance (it's "superior at precise description") but fails on GAR-Bench multi-prompt tasks. This suggests the RoI-aligned approach's main advantage is in multi-prompt relational reasoning rather than single-region description per se, which slightly reframes the claim.

Claim 2: GAR models interactions between multiple prompts.

This claim is strongly supported by GAR-Bench results (Tables 1 and 2), where GAR models dramatically outperform prior region MLLMs that were designed for single-prompt interactions. The gap is enormous — GAR-1B scores 50.6 on GAR-Bench-VQA vs. earlier region MLLMs scoring 22–27 (GPT4RoI, Shikra, Ferret, RegionGPT in Table 1). The inclusion of questions with up to 9 prompts (Figure 6b) and the "redundant prompt" design in the relation protocol provide direct evidence that the model handles genuine multi-prompt reasoning rather than simply describing each region independently. However, the absolute performance leaves substantial room for improvement — GAR-8B scores only 54.5 overall on GAR-Bench-VQA, and the paper's own qualitative failure cases (Figures 7 and 8) show the model still struggles with complex relationships involving more than two objects and with distinguishing fine-grained contextual cues (e.g., which surface a reflection comes from). The claim that GAR "manages to model relationships" is supported, but "modeling" is a matter of degree, and the current performance indicates this capability is nascent rather than mature.

Claim 3: GAR achieves advanced compositional reasoning, shifting the paradigm from passive description to active dialogue.

The VQA capabilities demonstrated on GAR-Bench-VQA partially support this claim. GAR models can answer specific free-form questions about regions, which is indeed a shift from captioning-only paradigms. The strong performance on "non-entity recognition" (Table 1, though specific sub-scores for this category are not broken out separately from the overall reasoning score) is particularly relevant — it demonstrates that the model uses global context for sophisticated reasoning. However, the claim of "advanced compositional reasoning" is somewhat overstated given the absolute performance levels. GAR-8B's overall score of 54.5 is lower than GPT-4o's 59.7, and neither approaches ceiling performance. The paper's choice to highlight "GAR-1B surpasses InternVL3-78B" is an impressive relative comparison but may obscure the fact that both models are far from solving the benchmark. Whether 54.5 represents "advanced" compositional reasoning or merely "emergent but still brittle" compositional reasoning is a matter of interpretation.

Claim 4: GAR-1B outperforms DAM-3B by +4.5 on DLC-Bench.

Directly supported by Table 3 (77.1 vs. 72.6 under multimodal judging). However, the DLC-Bench evaluation uses GPT-4o as the multimodal judge, and the reliability of GPT-4o for this task is not independently validated against human judgments. The paper's own critique of text-only judging (Section F) establishes that evaluation methodology matters enormously for these metrics, but it replaces one automated judge (LLaMA3.1-8B text-only) with another (GPT-4o multimodal) without validating either against ground-truth human judgments. The +4.5 gap is within a range where judging noise could plausibly account for some or all of the difference — the paper does not report any measure of judging reliability or inter-annotator agreement.

Claim 5: GAR-1B surpasses InternVL3-78B on GAR-Bench-VQA.

This is directly supported by Table 1 (GAR-1B: 50.6 vs. InternVL3-78B: 44.0). This is genuinely striking given the 78× parameter difference and suggests that architectural design for region understanding matters more than raw model scale for these tasks. However, InternVL3-78B is a general-purpose MLLM not specifically designed or trained for region-level understanding with mask prompts. The comparison demonstrates GAR's specialized capability advantage, but it does not necessarily demonstrate that GAR is a "better model" in any general sense — it demonstrates that specialization matters for specialized tasks, which is a weaker claim.

Claim 6: Zero-shot GAR-8B outperforms in-domain VideoRefer-7B on VideoRefer-Bench Q.

Supported by Table 7. This is the most remarkable result in the paper and constitutes genuinely surprising evidence for strong transfer. However, several qualifications are needed. First, VideoRefer-7B's in-domain training data (VideoRefer-700k) is not described in detail in this paper, so it's unclear how strong a baseline it represents — 700K video samples could be a relatively modest training set by modern standards. Second, the paper does not isolate whether GAR's advantage comes from the architectural design, the PerceptionLM base model's pretraining, or the GAR-2.5M image training data — a fairer comparison would include an ablation where the PerceptionLM base model is trained on VideoRefer-700k with a simpler architecture. Third, GAR-8B fails on temporal tasks where VideoRefer-7B presumably excels, so the "outperforms" claim applies only to the overall average, masking a nuanced capability tradeoff where each model is better at different subsets of tasks.

Genuine Weaknesses

The strongest baselines are missing or weak. The most striking omission is the lack of a baseline that trains DAM (or another crop-based architecture) on GAR's training data (GAR-2.5M). This would disentangle whether GAR's advantage comes from the RoI-aligned architecture or from the carefully constructed training data pipeline. Without this ablation, one cannot confidently attribute performance gains to the architectural innovation. The paper does ablate data components (Table 11), but only within the GAR architecture — it never asks whether the same training data would close the gap for a DAM-like architecture. Similarly, the paper does not compare against an approach that uses SAM to segment the image into regions and then feeds all regions simultaneously to a strong general MLLM — this would be a natural multi-prompt baseline that doesn't require specialized architecture at all.

The benchmark is self-constructed and the evaluation relies on LLM judges without human validation. GAR-Bench is introduced in this paper and evaluated with GPT-4o as judge for captioning tasks. The paper's own Section F critique of text-only judging establishes that LLM judges can be systematically unreliable, but the paper applies this critique only to the previous judge (LLaMA3.1-8B) and not to its own chosen judge (GPT-4o). Without human evaluation of GAR-Bench-Cap annotations or VQA ground-truth, the absolute scores on these benchmarks are difficult to interpret. The difficulty filtering step (removing questions all four frontier models answer correctly) guards against ceiling effects but does not validate the correctness of the remaining questions.

The "zero-shot video outperforms in-domain" result may reflect differences in base model quality rather than architectural superiority. PerceptionLM (GAR's base) may simply be a stronger pretrained model than whatever VideoRefer uses as its base. The paper does not compare the base models' image-level capabilities before fine-tuning, making it impossible to attribute the video transfer advantage to the GAR architecture vs. the base model's inherent strength.

The compute-matched comparison is absent. The paper compares models of wildly different scales (1B vs. 3B vs. 8B vs. 78B vs. GPT-4o) without controlling for training compute, inference compute, or latency. The claim that GAR-1B "outperforms" DAM-3B conflates model architecture with model scale — it's comparing a 1B model with a carefully designed architecture against a 3B model with a simpler architecture, and attributing the difference entirely to architecture. A fairer comparison would match parameter count, training FLOPs, or at minimum report inference latency alongside accuracy for all methods.

The single-base-model, single-training-pipeline limitation. All GAR results use PerceptionLM as the base model and the GAR-2.5M pipeline. The cross-model architecture ablation (Table 9) shows that RoI-aligned feature replay improves Qwen2.5-VL and InternVL3, but these scaled-down ablations are not trained on the full GAR-2.5M dataset and are not compared against state-of-the-art baselines. Without a full-scale reproduction on a different base model, the generalizability of the GAR recipe (base model + architecture + data pipeline) remains unproven.

Missing ablations. Several potentially important variables are not ablated: the number of training epochs/steps, the effect of AnyRes tile count on performance vs. latency, the sensitivity to RoI-Align output grid size (is 7×7 optimal?), whether zero-initialization of mask embeddings matters vs. random initialization, and the specific choice of LLM judge for GAR-Bench (would results change with a different judge model?). The training hyperparameters are only partially specified — learning rate, batch size, and optimizer are given, but training duration, warmup, weight decay, and maximum sequence length are not.

What Would Strengthen the Paper

• Train DAM (or another crop-based architecture) on GAR-2.5M. This is the single most important missing experiment — it would isolate the architectural contribution from the data contribution and directly test the paper's central claim that architecture, not data, is the key factor.

• Human evaluation of GAR-Bench. Even a modest human evaluation (100-200 samples) would validate the LLM-judge scores and establish interpretable absolute performance levels.

• Compute-controlled comparisons. Report inference FLOPs or latency alongside accuracy for all methods in the main comparison tables. This is especially important when comparing models at different scales (1B vs. 3B vs. 8B vs. 78B).

• Scale the model further. The saturation pattern between GAR-1B and GAR-8B on captioning tasks suggests diminishing returns, but the continued improvement on VQA tasks suggests scale still matters. Testing a GAR-13B or GAR-34B would reveal whether the architecture eventually catches up to or surpasses GPT-4o on GAR-Bench, or whether fundamental capability ceilings remain.

• Test on a broader range of general MLLMs with multi-prompt capability. Models like GPT-4o or Gemini can process multiple cropped regions if the user provides them as separate images — testing this prompting strategy against GAR's native multi-prompt handling would distinguish architectural advantage from base model capability advantage.

• Validate the DLC-Bench judging protocol. The paper's critique of text-only judging is compelling but its replacement (GPT-4o multimodal judging) is adopted without validation. A small human study comparing both judging protocols against human judgments would substantially strengthen the evaluation methodology contribution.

6. Limitations and Trade-offs

6.1 The Multimodal Judging Protocol Is Not Validated Against Human Ground Truth

The assumption or constraint. The paper's evaluation of caption quality — on GAR-Bench-Cap, DLC-Bench, Ferret-Bench, and MDVP-Bench — relies entirely on GPT-4o as an automated judge. The authors justify this by arguing that text-only judging (as used in the original DLC-Bench with LLaMA3.1-8B) is unreliable because it cannot resolve semantic ambiguities without visual context (Section F, Figure 11). They replace the text-only judge with GPT-4o provided with cropped images and masks, but do not validate GPT-4o's judgments against any gold standard. The paper states in Section F that "the image is the only ground truth capable of resolving this ambiguity," yet never verifies whether GPT-4o, when shown the image, actually produces judgments that agree with human evaluators.

The consequence. Without human validation, the absolute scores reported across all captioning benchmarks are uninterpretable beyond relative comparisons between models evaluated under the same protocol. We cannot know whether a score of 77.1 on DLC-Bench (GAR-1B under multimodal judging, Table 3) represents excellent captioning quality, moderate quality, or merely high agreement with GPT-4o's particular biases about what constitutes a good caption. This matters because the paper's central claim — that GAR achieves "precise perception" and "advanced compositional reasoning" — is partially quantified through these automated metrics. If GPT-4o systematically prefers certain captioning styles, favors verbosity, or misjudges factual correctness in ways that correlate with the generating model's architecture rather than true quality, the reported performance gaps could be artifacts of the judge rather than reflections of genuine capability differences.

The problem is exacerbated by the fact that the paper's own critique of text-only judging (Section F) demonstrates that switching judges can substantially change scores. Under text-only LLaMA3.1-8B judging, DAM-3B scores 64.5 on DLC-Bench; under GPT-4o multimodal judging, it scores 72.6 — an 8.1-point swing from changing nothing but the judge. GAR-1B sees an even larger swing from 67.9 to 77.1 (+9.2 points). These large judge-dependent shifts mean that a substantial fraction of the reported performance gains could be attributable to judge-model interactions that have not been characterized. A practitioner deciding whether to adopt GAR needs to know whether the +4.5 advantage over DAM-3B (77.1 vs. 72.6) would be confirmed by human evaluation, or whether it falls within the noise floor of automated judging.

What evidence exists in the paper. Table 3 reports DLC-Bench results under both judging protocols, and Figure 11 provides qualitative examples of text-only judging failures. However, the paper provides no human evaluation data, no inter-annotator agreement statistics between GPT-4o and humans, and no calibration of GPT-4o's judging accuracy against any ground truth. The difficulty filtering step for GAR-Bench (Section B.1) involves human experts curating answers, but this curation is for question construction, not for evaluating model outputs against a gold standard. The paper does not report how many GAR-Bench-Cap examples were evaluated by humans or what agreement rates were observed.

Mitigation status. Not addressed. The paper critiques the previous judge but does not validate its replacement. No human evaluation study is reported, and the paper does not discuss the limitations of LLM-as-a-judge evaluation for region-level captioning. Future work would benefit from even a modest human evaluation (100–200 examples) to calibrate GPT-4o's judgments and provide interpretable absolute performance levels.

---

6.2 Difficulty Estimation Overhead: The Method's Core Promise Cannot Be Realized Without Solving an Unsolved Preprocessing Problem

The assumption or constraint. GAR's architectural design assumes that the user provides explicit mask prompts specifying which regions to analyze. In the paper's experiments, these masks come from ground-truth annotations in existing datasets (DLC-Bench, LVIS, PACO), from SAM-generated masks conditioned on bounding boxes (MDVP-Bench), or from hand-curated masks in GAR-Bench constructed through a semi-automated pipeline involving "8 MLLM experts" (Section B.1). The paper provides no method for automatically generating high-quality region masks from free-form user queries — for example, if a user asks "describe the person in the red jacket," the system would need to first locate and segment that person before GAR can process the region.

The consequence. This is not a minor deployment detail — it is a prerequisite for the system to function at all. A practitioner wanting to deploy GAR for interactive region-level understanding faces a chicken-and-egg problem: to understand a region, you must first specify it with a mask, but generating that mask may require understanding the scene at a level that currently requires GAR-like capabilities. In the paper's evaluation pipeline, this problem is circumvented entirely by using pre-existing masks from curated datasets. The benchmark construction pipeline (Section B.1) describes a labor-intensive process involving SAM for initial mask generation, followed by expert review for quality and task-relevance, followed by manual annotation of ground-truth attributes. This pipeline is not automated and would not scale to deployment.

The practical implication is that GAR's capabilities as demonstrated in the paper represent performance under ideal conditions — high-quality masks, correctly specified by an oracle. Real-world use would require an upstream region proposal or referring expression comprehension module whose errors would propagate to GAR's inputs. If the mask is misaligned (e.g., includes background pixels, excludes part of the object), GAR's features would be extracted from the wrong spatial locations, potentially degrading its carefully engineered context-detail balance. The paper provides no analysis of how performance degrades under mask noise, misalignment, or imprecise region specification.

What evidence exists in the paper. None. The paper does not report any experiments with automatically generated masks from an end-to-end referring expression pipeline, does not study sensitivity to mask quality, and does not discuss the mask generation problem as a limitation. The ablation studies in Table 8 compare architectural variants but always provide ground-truth or SAM-generated masks as input. The GAR-Bench construction pipeline (Section B.1) reveals the complexity of obtaining high-quality masks — involving SAM, expert review, and manual annotation — but treats this as a benchmark construction detail rather than a deployment barrier.

Mitigation status. Not addressed. The paper does not propose a solution to the mask generation problem, does not integrate a referring expression comprehension module, and does not discuss this as a limitation or area for future work. A complete deployable system would require either (1) an integrated segmentation-and-understanding pipeline (potentially using GAR's own capabilities for iterative refinement), or (2) a separate high-quality open-vocabulary segmentation model whose error characteristics are understood and whose output distribution matches the mask quality GAR was trained on. Neither is provided or analyzed.

---

6.3 The Model Cannot Reason About Temporal Dynamics — Video Transfer Is Spatial-Only

The assumption or constraint. GAR is trained exclusively on static images. When extended to video (Tables 6 and 7), the paper uniformly samples 16 frames and processes each independently through GAR's spatial region-understanding pipeline. There is no temporal attention, no cross-frame feature aggregation, and no mechanism for modeling motion, action sequences, or state changes. The paper acknowledges this explicitly in Section E: "our GAR is limited to static images" and "it sometimes fails when input videos contain significant motion changes."

The consequence. This limitation means that GAR's impressive zero-shot video transfer — outperforming in-domain VideoRefer-7B on VideoRefer-Bench Q (Table 7) — applies only to a subset of video understanding tasks. The paper's own results reveal the boundary: GAR performs well on tasks that can be answered from individual frames (object recognition, spatial relationships, static scene attributes) but fails on tasks requiring temporal reasoning. Specifically, Table 6 shows GAR gets "reasonably low scores on temporally related tasks, e.g., temporal description (TD)," and Table 7 shows GAR underperforming VideoRefer-7B on "future prediction" tasks that require anticipating what will happen next based on observed motion. The qualitative failure cases in Figures 9 and 10 confirm this pattern: GAR-8B describes static video frames accurately but misses action dynamics and cannot predict future states.

For a practitioner, this means GAR cannot be used as a general-purpose video region understanding system despite the strong headline result. Applications like sports analysis (describing a player's motion over time), instructional video understanding (following a sequence of steps), or surveillance (detecting anomalous temporal patterns) would not benefit from GAR's current architecture. The paper's claim that GAR's "strong capabilities can be easily transferred to videos" (Section 4, Table 7 caption) should be qualified: the transfer is strong for spatial region understanding in videos, but weak for temporal understanding. A user who reads only the headline result might mistakenly assume GAR handles the full VideoRefer-Bench Q task distribution, when in fact it likely achieves its high average by excelling on spatial questions while performing poorly on temporal ones — a composi

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the region-level MLLM conversation from a data-scaling mindset to an architectural-design mindset. Before GAR, the dominant assumption was that better region understanding would come from larger models trained on more region-level captioning data — the DAM and PAM paradigm of scaling crop-based captioning. GAR demonstrates that this assumption is wrong in a specific, diagnosable way: the crop-based architecture has a structural blind spot (loss of global context) that no amount of additional training data can fix, because the information needed to resolve the ambiguity is physically removed from the model's input.

This is not merely an incremental improvement over DAM. It is a reframing of what region understanding requires — not just better local recognition, but the simultaneous availability of local detail and global context, which are in tension under the crop-based paradigm. The paper's central diagnostic — that misidentifying a frog-shaped slipper as a real frog is not a recognition failure but a context failure — crystallizes why prior approaches hit a ceiling that scaling alone cannot breach. This diagnosis has the practical effect of redirecting research investment: rather than collecting larger crop-level captioning datasets (the DAM approach), the community should invest in architectures that preserve global context while extracting local features.

The paper also provides a unifying explanation for prior contradictory findings. Why do some region MLLMs excel at detailed captioning benchmarks (DLC-Bench) while failing at relational reasoning? The answer, per GAR's ablation (Table 8), is that crop-based methods are good at precise description because the crop provides high-resolution local detail, but they are bad at multi-prompt reasoning because the crop strips away context. Conversely, pooled-feature methods preserve context but lose detail. Prior papers reporting strong single-region captioning were measuring one axis of capability; prior work finding weak relational reasoning was measuring the other. GAR shows these are not contradictory — they reflect different points on a tradeoff curve that the RoI-aligned feature replay is designed to escape. This reconciliation is valuable because it converts a confusing landscape of conflicting results into a coherent design space with a clear frontier.

The finding that zero-shot GAR-8B outperforms in-domain VideoRefer-7B on VideoRefer-Bench Q (Table 7) is genuinely surprising and has broader implications beyond this specific comparison. It suggests that spatial context understanding — the ability to interpret a region within its surrounding scene — is a transferable skill that generalizes from static images to video frames without video-specific training. This challenges the assumption that video region understanding requires temporal architectures or video-specific training data, at least for tasks that are primarily spatial in nature. The dissociation between strong spatial transfer and weak temporal transfer (GAR fails on temporal description and future prediction) actually strengthens this implication: it isolates spatial context reasoning as the transferable capability and temporal dynamics as the capability that genuinely requires video-specific training. This is an existence proof that spatial and temporal region understanding can be partially decoupled, which has implications for how video region-understanding systems should be designed — perhaps a spatial backbone (GAR-like) plus a lightweight temporal aggregation module, rather than training from scratch on video data.

The paper's critique of text-only judging for region-level captioning (Section F, Figure 11) should change evaluation practices in the field. The demonstration that LLaMA3.1-8B text-only judging produces systematically unreliable results — because it cannot resolve semantic ambiguities that require visual context — is compelling and has direct practical implications for anyone reporting results on DLC-Bench or similar benchmarks. The paper effectively establishes that multimodal judging is a minimum requirement for region-level captioning evaluation, though it stops short of validating its own chosen multimodal judge (GPT-4o) against human ground truth. Future papers that continue to report only text-only judging scores for region captioning will need to justify this choice against the critique established here.

The paper also makes mask-prompt-based region understanding more attractive as a paradigm relative to bounding-box or visual-marker approaches. Prior mask-based models (Osprey, Ferret) underperformed box-based models on some tasks due to the detail-resolution tradeoff. GAR shows that masks can achieve state-of-the-art performance for both detailed captioning and relational reasoning, making the richer spatial specification of masks (vs. boxes, which include background pixels) a clear advantage rather than a liability. This may accelerate adoption of mask-based interfaces for region-level MLLMs.

However, the shift is not a paradigm change at the scale of, say, the introduction of transformers to vision. GAR's contribution is architectural rather than representational — it changes how features are computed and routed, but the underlying building blocks (ViT, LLM, RoI-Align) are all established. The paper does not introduce a new pretraining objective, a new model family, or a new scaling law. It is best understood as a strong architectural refinement that solves a clearly diagnosed failure mode, establishing a new design principle (shared context encoding with per-region fine-grained extraction) that subsequent work should adopt or deliberately depart from.

Follow-Up Research This Work Enables

Combining RoI-aligned feature replay with temporal aggregation for video. The paper's clearest limitation is that GAR processes video frames independently with no temporal modeling, causing failures on temporal description and future prediction (Tables 6, 7; Figures 9, 10). The natural follow-up is to add a lightweight temporal fusion module on top of GAR's frame-level region features — for instance, a temporal transformer or a simple 3D convolution over time-aligned RoI-Align features extracted from multiple frames. The experiment would train this temporal module on video region-annotation data (VideoRefer-700k or similar) while keeping GAR's spatial backbone frozen, testing whether the strong spatial capabilities transfer while temporal capabilities are acquired with modest video-specific training. The compelling hypothesis from this paper is that spatial and temporal region understanding can be partially decoupled architecturally; a successful follow-up would demonstrate strong video performance with far less video training data than existing approaches require.

Training DAM on GAR-2.5M to isolate architecture vs. data contributions. The single most important missing experiment from this paper is a head-to-head comparison where DAM (or another crop-based architecture) is trained on exactly the same GAR-2.5M dataset. This would answer the questions: how much of GAR's advantage comes from the RoI-aligned architecture, and how much from the carefully constructed training data (fine-grained recognition enhancement, multi-prompt relational data)? The paper's data ablation (Table 11, actually Table 10 in the appendix) shows that each data component matters, but this ablation is only within the GAR architecture. A follow-up study would train DAM-3B, PAM-3B, and GAR-1B on identical data and compare across all benchmarks. If DAM matches GAR on DLC-Bench but still fails on GAR-Bench, that confirms the architecture is essential for multi-prompt reasoning. If DAM matches GAR on all benchmarks, the paper's central architectural claim is weakened and the data pipeline emerges as the primary contribution.

Scaling GAR to larger LLMs and testing on GAR-Bench to map the capability ceiling. The paper shows that GAR-1B and GAR-8B perform similarly on captioning tasks (DLC-Bench: 67.9 vs. 67.4; GAR-Bench-Cap: 67.9 vs. 67.4 in Tables 2 and 3) but GAR-8B substantially outperforms GAR-1B on VQA reasoning (GAR-Bench-VQA: 54.5 vs. 50.6 in Table 1). This differential scaling pattern — captioning saturates early, reasoning continues to benefit from scale — is interesting but only observed at two data points (1B and 8B). Testing GAR-13B, GAR-34B, and GAR-70B on GAR-Bench-VQA would reveal whether the reasoning capability continues to scale or plateaus. If GAR-34B approaches GPT-4o's 59.7 or surpasses it, that suggests architecture + scale can match or exceed generalist frontier models on specialized region tasks. If performance plateaus around 55–57, that suggests a fundamental ceiling imposed by the training data or the task formulation that architecture alone cannot breach. The specific metric to track would be the multi-prompt relation sub-protocol, since this is where GAR's architecture should provide the greatest advantage.

Cheaply estimating difficulty from visual features to enable adaptive region processing. One of the paper's unstated assumptions is that all prompted regions are processed identically — the same RoI-Align grid size, the same AnyRes tile allocation, the same amount of LLM attention. But regions vary dramatically in their complexity: a large, uniformly textured region (a wall, a sky) needs far less computational investment than a small, cluttered region with fine detail (a person's face in a crowd). A follow-up could train a lightweight "region difficulty predictor" — a small MLP on top of the RoI-Align features — that estimates how much computation a region needs and adaptively adjusts the RoI-Align grid resolution or the number of AnyRes tiles allocated to that region. The experiment would measure whether this adaptive allocation achieves better accuracy-compute tradeoffs than uniform processing, using the GAR-Bench prompt area distribution (Figure 6c: mean 4.4%, many prompts <1%) to create a challenging test bed. This connects GAR to the broader "adaptive computation" literature and addresses the paper's implicit assumption that all regions merit equal processing.

Stress-testing GAR with adversarial mask perturbations to characterize failure modes. The paper assumes clean, high-quality masks generated by SAM or hand-curated by experts. Real-world masks from an upstream open-vocabulary segmentation model would be noisy — slight misalignments, inclusion of background pixels, incomplete coverage of the target object. A systematic stress test would evaluate GAR's performance on GAR-Bench as a function of controlled mask perturbations: shifting masks by N pixels, dilating/eroding by M pixels, replacing masks with bounding boxes, adding random mask noise. The goal is to characterize the robustness curve — how much mask degradation can GAR tolerate before performance drops below a DAM baseline? If GAR is extremely sensitive to mask quality (because its RoI-Align extraction assumes precise spatial correspondence), this would be a serious deployment limitation that the paper does not address. If it is robust to moderate perturbations, that strengthens the practical case for GAR over crop-based methods.

Using GAR's spatial features as initialization for robotic manipulation policies. The paper demonstrates that GAR's region features are simultaneously good at semantic understanding (high LVIS/PACO scores, Table 5) and precise localization (high semantic IoU). This combination — knowing what an object is and exactly where it is — is exactly what robotic grasping and manipulation policies need. A follow-up could take a frozen GAR encoder, extract RoI-Align features for objects in a manipulation scene, and train a lightweight policy head on top for tasks like "grasp the mug by its handle" or "place the spoon to the left of the plate." The experiment would compare against policies that use crop-based features (DAM-style) or general MLLM features (CLIP, DINOv2). The hypothesis is that GAR's context-aware features would enable better generalization to novel object arrangements because the features encode the spatial relationships between objects, not just the objects in isolation. A strong result would be improved sample efficiency — the policy needs fewer demonstrations to learn because the features already capture relevant relational structure.

Practical Applications and Downstream Use Cases

Assistive technology for visually impaired users performing fine-grained scene queries. A visually impaired user walking through a grocery store could point their phone camera at a shelf and ask "is the product in the green box on the top shelf gluten-free?" or "which of these three cans is the soup, not the vegetables?" This requires precisely the capabilities GAR demonstrates: understanding specific regions (the green box, each can) within a cluttered global context (the shelf), recognizing fine-grained attributes (brand labels, ingredients text), and in some cases reasoning about relationships between products. GAR-1B's strong performance relative to much larger models (50.6 on GAR-Bench-VQA vs. InternVL3-78B's 44.0 in Table 1) suggests that a small, efficient model could run on-device with acceptable latency — the paper reports first-token latency numbers in Table 8 that, while not real-time, are heading in the right direction. The key practical blocker (not solved in the paper) is the mask generation problem: the user's query "the green box" needs to be converted into a segmentation mask before GAR can process it. A production system would integrate GAR with an open-vocabulary referring expression segmentation model (like SAM + CLIP or an LVLM-based segmenter), where GAR handles the understanding after the mask is produced.

Batch annotation and quality control for fine-grained computer vision datasets. Organizations building datasets like LVIS, PACO, or custom industrial inspection datasets need to verify that segmented objects are correctly categorized and that their attributes (color, material, state) are accurately labeled. GAR-8B's state-of-the-art performance on LVIS (93.6 semantic similarity, 88.7 semantic IoU in Table 5) and PACO (95.5, 91.8) makes it a strong candidate for automated verification of human annotations — flagging cases where a human labeled an object category that is inconsistent with GAR's predicted category or where the segmentation mask appears misaligned. The multi-prompt capability enables batch verification of multiple objects in a single forward pass: rather than running the model once per annotated object, GAR can verify all objects in an image simultaneously by providing all masks as prompts, asking "are these category labels correct for these regions?" The practical benefit is cost reduction in dataset curation — GAR can triage annotations for human review, focusing human effort on the cases where the model disagrees with the label, while automatically accepting clear cases.

Interactive image editing with natural language region references. Photo editing applications increasingly incorporate AI-powered features where users describe edits in natural language ("make the sky more dramatic," "remove the person in the background"). GAR's architecture is well-suited to this use case because editing requires both precise localization (which pixels to modify) and contextual understanding (how the edit should blend with the surrounding scene). A GAR-based editing pipeline would: (1) accept the user's natural language reference ("the sky"), (2) generate a mask prediction (via an integrated segmenter), (3) use GAR to verify the mask correctly captures the right region and to understand the region's context (lighting conditions, surrounding objects, scene type), and (4) apply the edit in a context-aware manner (e.g., a "more dramatic sky" means different things for a beach sunset vs. a city skyline). The paper's finding that GAR-1B and GAR-8B perform nearly identically on detailed captioning (Table 3) is encouraging for this application — the smaller, faster model may suffice for the verification and context-understanding steps, keeping latency low for interactive use.

Video surveillance and monitoring with spatial region queries. Security monitoring applications often need to answer questions about specific regions over time: "did anyone enter the restricted area in the last hour?" or "is the package that was delivered at 2 PM still on the doorstep?" These are fundamentally spatial queries applied to video — they require identifying a region (the restricted area, the doorstep) and understanding what happens in that region over time. GAR's strong zero-shot spatial transfer to video (outperforming VideoRefer-7B on spatial tasks in Table 7) makes it immediately applicable to the spatial component of these queries, while the temporal component would require additional processing (simple frame differencing for motion detection, or a lightweight temporal module). The practical architecture would be: GAR processes key frames with the region of interest specified by a mask (e.g., the doorway), identifies objects and states in each frame, and a separate temporal module tracks changes across frames. The benefit is that the spatial understanding component — which is the hard part for existing video understanding systems — can be handled by a model trained only on static images, substantially reducing the video training data requirements for spatial query tasks.