Sa2VA: Marrying SAM2 with LLaVA for Dense Grounded Understanding of Images and Videos

ArXiv: 2501.04001

🎯 Pitch

Existing grounding MLLMs forget how to chat—catastrophically. Sa2VA solves this by unifying SAM-2 and advanced vision-language models (Qwen-VL, Intern-VL) into a single token space, simply generating [SEG] tokens to prompt mask decoders, and with minimal one-shot instruction tuning it becomes the first model to simultaneously achieve strong pixel grounding and competitive image/video conversation across 15+ benchmarks.

---

1. Executive Summary

This work introduces Sa2VA, the first unified framework that marries SAM-2—a foundation video segmentation model—with multi-modal large language models (MLLMs) to achieve dense grounded understanding of both images and videos through a single end-to-end model. The system formulates referring segmentation, visual question answering, and grounded caption generation as a unified one-shot instruction tuning process, using a special token mechanism (the [SEG] token whose hidden states serve as spatial-temporal prompts for SAM-2's decoder) to bridge the LLM's language outputs with precise mask generation. Sa2VA achieves strong performance across more than 15 benchmarks, including cIoU scores of 81.9, 76.5, and 78.9 on RefCOCO/RefCOCO+/RefCOCOg while simultaneously maintaining competitive image and video chat capabilities—a balance prior grounding MLLMs fail to achieve, with previous methods showing catastrophic forgetting (e.g., LISA-7B scores 1/1 on MME perception/cognition versus Sa2VA-8B's 1651/578). The paper also introduces Ref-SAV, a challenging referring video object segmentation benchmark with 1,147 videos and 1,945 manually validated object expressions featuring long text descriptions, heavy occlusions, and large camera motion, establishing that test-time compute from additional training data yields significant gains only when the base model already possesses video understanding capacity—the hardest Ref-VOS scenarios remain at roughly 50% J&F even with the full Ref-SAV training set.

2. Context and Motivation

The Core Problem: No Single Model Handles Both Fine-Grained Pixel Grounding AND Open-Ended Understanding Across Images and Videos

The fundamental gap this paper addresses is deceptively simple: there exists no unified model that can simultaneously perform precise pixel-level segmentation (driven by language descriptions) AND open-ended conversational understanding, across both static images and dynamic videos. This is not merely a convenience problem—it represents a genuine architectural and training divide in the field that prevents building the kind of interactive, promptable visual assistants that real-world applications demand.

The paper motivates this through a concrete use case (Section 1, Figure 1a): imagine a user watching a video and wanting to both segment and track "the person wearing sunglasses on the street" while also asking "what is the weather now?" and getting the answer "it is foggy." This requires a single model to handle referring video object segmentation (RVOS), video question answering (Video-QA), and visual prompt understanding simultaneously. No existing system—whether a pure perception model or a pure MLLM—can do this.

Why This Matters: The Divide Between Perception Models and Language Models

The importance of bridging this gap extends beyond academic novelty. The paper identifies four concrete application domains where unified dense grounded understanding would be transformative:

1. Short video editing (Section 1). Modern video editing pipelines increasingly rely on AI-driven object selection and manipulation. A unified model that can both understand a natural language instruction ("remove the red car in the background") and execute precise pixel-level tracking across frames would dramatically simplify editing workflows. Current approaches require stitching together separate models—one for language understanding, another for segmentation, another for tracking—with no guarantee of semantic alignment between them.

2. Robot navigation (Section 1). The paper cites work on video object segmentation for visual servo control and terrain segmentation for outdoor navigation (Griffin et al., 2020; Guan et al., 2022, 2023). A robot navigating unstructured environments needs to segment relevant objects (paths, obstacles, landmarks) while also answering compositional queries about its surroundings. A model that collapses perception and reasoning into a single forward pass is inherently more efficient and less prone to cascading errors than a pipeline of specialists.

3. Surveillance analysis (Section 1). Monitoring applications require tracking specific entities ("the person in the blue jacket") across extended video sequences while also generating descriptive summaries of events. Unified models could enable natural language queries over surveillance footage without requiring separate retrieval, tracking, and captioning modules.

4. Interactive visual assistants. The paper's framing of "promptable fine-grained analysis" where "the user is in the loop" (Section 1) points toward a broader vision: AI systems that users can converse with about visual content at multiple levels of granularity—from holistic scene descriptions down to individual pixel masks—without switching between different tools or models.

The Two Sides of the Divide: What Each Paradigm Can and Cannot Do

The paper carefully characterizes the limitations of both existing approaches, establishing that this is not a case of "either side is sufficient"—each paradigm is fundamentally missing capabilities the other provides.

What Video Perception Models Can Do (and Cannot)

State-of-the-art models like SAM-2 (Ravi et al., 2024) represent the pinnacle of promptable video segmentation. SAM-2 can perform interactive segmentation and tracking given visual prompts (points, boxes, or masks), and it's trained on more than 1 billion masks, giving it robust spatial-temporal perception capabilities.

However, the paper identifies critical limitations (Section 1):

"SAM-2 model can perform promptable segmentation and tracking, [but] it cannot handle text-aware tasks, such as understanding language expressions or video conversation."

In other words, SAM-2 has no language understanding. Given a natural language query like "segment the blonde-haired girl dancing in a blue dress," SAM-2 cannot interpret the text itself—it requires an external system to convert language into visual prompts (points, boxes) that it can process. It operates purely in the visual domain.

The paper also notes that video perception models more broadly are "limited by constrained semantic concepts and lack open-ended capabilities, such as video question answering." Models like OMG-Seg (Li et al., 2024) unify multiple segmentation tasks but operate within a closed vocabulary of visual concepts. They cannot answer "what is the person holding?" because they have no generative language capacity.

What Video MLLMs Can Do (and Cannot)

Conversely, video multi-modal large language models like LLaVA-OneVision (Li et al., 2024) and LLaMA-VID (Li et al., 2024) can understand video content and perform Video-QA. The latest LLaVA variants achieve strong results on benchmarks like Video-MME and MMBench-Video, demonstrating sophisticated temporal reasoning about dynamic scenes.

But the paper is explicit about their limitation:

"[Video MLLMs] cannot carry out perception tasks nor understand visual prompts."

This means they cannot produce pixel-level outputs. They can tell you what is happening in a video, but they cannot segment, track, or manipulate individual objects at the mask level. When asked "segment the person wearing sunglasses," they have no mechanism to output segmentation masks.

Prior Attempts at Bridging the Divide: Partial Solutions, Critical Failures

The paper acknowledges that several prior works have explored combining perception models with MLLMs (Section 1):

"Several works [24, 75, 94, 100, 114] have explored the combination of perception models and MLLMs."

However, the authors identify three specific failure modes in these existing approaches:

Failure Mode 1: Modality Restriction

Existing integrated models focus predominantly on images only or one specific video task, never both modalities across a full task spectrum. As the paper states:

"existing works mainly focus on exploring image tasks [24, 107, 114] or trying to solve one specific video task [4, 41, 70, 100]."

For example, LISA (Lai et al., 2024) and GLaMM (Rasheed et al., 2024) are image-only referring segmentation systems built on top of MLLMs. They handle image-level chat and segmentation, but have no video capability whatsoever. On the video side, VISA (Yan et al., 2024) addresses referring video object segmentation via LLMs, but the paper notes it "lacks comprehensive training, which restricts their capability in other tasks and scaling up" (Section 2). VideoLISA (Bai et al., 2024) extends LISA's approach to videos, but remains a specialist—it handles video referring segmentation but drops image chat and QA capabilities.

The paper summarizes this fragmentation in a striking way (the "To the best of our knowledge" statement in Section 1):

"no work has successfully integrated the strengths of both sides to build a unified model that supports both image and video inputs across a diverse range of tasks."

Failure Mode 2: Catastrophic Forgetting in Conversation

Perhaps the most damning evidence the paper presents is the systematic degradation of conversation capabilities when MLLMs are fine-tuned for grounding tasks. This is not a minor regression—it represents a near-total collapse of language capabilities in some cases.

The authors cite prior findings (Section 2, citing Zhang et al., 2024, and Wu et al., 2024):

"existing methods [95, 114] find that conversation tasks are drastically degraded when grounding tasks are performed. The models are thus degraded to specialists."

The evidence in Table 6 and Table 7 of the paper quantifies this failure:

• LISA-7B (Lai et al., 2024), which fine-tunes LLaVA for reasoning segmentation, achieves 1/1 on MME perception/cognition scores—effectively zero conversational ability. The model has become a pure segmentation specialist.
• GLaMM-7B (Rasheed et al., 2024) scores a similarly dismal 14/9 on MME (perception/cognition) and only 36.8 on MMBench.
• LaSagnA (Wei et al., 2024) scores 0/0 on MME—complete loss of conversational capability.

These models can produce excellent segmentation masks, but they have lost the open-ended understanding that makes MLLMs valuable in the first place. They cannot chat, answer questions, or describe scenes in natural language.

What makes this particularly insidious is that these models don't just fail at conversation while succeeding at grounding—the paper implies there is an inherent task conflict that the training paradigm hasn't resolved. The models are being asked to simultaneously optimize for mask prediction (a dense, pixel-level output) and text generation (a sparse, token-level output), and naive co-training causes one objective to dominate and wipe out the other.

Failure Mode 3: Knowledge Inheritance Barriers

The paper also identifies a subtler problem: how to leverage pre-trained knowledge from both foundation models without interference. SAM-2 has been trained on over 1 billion masks, representing enormous investment in visual perception capabilities. MLLMs have been trained on massive instruction-following datasets, representing enormous investment in language understanding. A naive combination that forces both models to co-adapt would likely lose the specialized knowledge that makes each component valuable.

The authors note (Section 1) that "both model families are evolving rapidly, and a flexible framework is crucial to more easily leverage their future progress." Any architecture that tightly couples SAM-2 and the MLLM would need to be redesigned every time a better version of either component is released. This practical consideration—the need to ride the wave of progress in both the segmentation and language modeling communities—motivates a more modular design.

The Data Gap: Why Existing Video Referring Segmentation Datasets Are Insufficient

The paper identifies a second, parallel gap beyond model architecture: the inadequacy of existing referring video object segmentation (Ref-VOS) benchmarks and training data.

The authors empirically observe (Section 1) that:

"existing video segmentation datasets are limited to small-scale collections of short clips with a limited number of occlusions."

Table 4 quantifies this claim with stark numbers:

The existing benchmarks share three critical weaknesses:

1. Short text expressions. Ref-YTVOS averages under 10 words per expression, and MeViS averages only 7.07. These are simple, direct descriptions ("the red car," "the person on the left") that don't require sophisticated language understanding. Real-world applications—particularly interactive assistants where users speak naturally—require handling much longer, more compositional descriptions.

2. Limited challenging conditions. Table 3 provides a qualitative comparison of existing benchmarks across five dimensions: short text support, long text support, large object motion, large camera motion, and heavy occlusion. The pattern is clear:

"The previous benchmarks only contain partial aspects of these five challenging cases."

Ref-DAVIS17 has large camera motion but limited occlusion; Ref-YTVOS has large object motion but no long text; no prior benchmark has all five challenging characteristics simultaneously. This means models trained and evaluated on existing benchmarks are systematically tested on easier scenarios than they would encounter in deployment.

3. Small scale. Existing Ref-VOS training datasets collectively contain only ~5.8K videos with referring expressions (Ref-YouTubeVOS + MeViS + ReVOS). This is orders of magnitude smaller than what modern deep learning systems typically require for robust performance. By comparison, SAM-2 was trained on over 1 billion masks—the scale mismatch suggests existing Ref-VOS models are severely data-starved.

The consequence of these weaknesses is that current Ref-VOS models achieve poor zero-shot performance on realistic, complex videos. The paper's Ref-SAV benchmark results in Table 10 demonstrate this directly: state-of-the-art models like UNINEXT (Yan et al., 2023) achieve only 7.6 J&F, and VISA (Yan et al., 2024) achieves only 11.8 J&F when tested zero-shot on Ref-SAV's long-expression set. These are not bad models—they represent the state of the art on existing benchmarks. They simply haven't been trained or evaluated on data that reflects real-world complexity.

How the Paper Positions Itself

The paper positions Sa2VA as addressing both gaps simultaneously—the model architecture gap AND the data gap—through three key design principles that directly respond to the failures identified above:

1. Unified task formulation through token-space unification. Rather than building separate pipelines for different modalities, Sa2VA treats all inputs—text, images, video frames, visual prompts—as tokens fed into a single LLM (Equation 1 in Section 3.1). This directly addresses Failure Mode 1 (modality restriction) by providing a single interface for all tasks. The paper argues this is possible because "we can leverage LLMs to manage various visual tokens."

2. Co-training with careful data balancing to prevent catastrophic forgetting. The paper explicitly acknowledges the catastrophic forgetting problem and addresses it through joint training with all four data types: image QA, video QA, image segmentation, and video segmentation (Table 2, Section 4). The ablation study in Table 9 serves as direct evidence that this co-training strategy works: removing any single data type causes significant degradation in the corresponding capability, but keeping all four maintains balanced performance. This directly addresses Failure Mode 2.

3. Decoupled design that freezes SAM-2's encoder and memory module. To address Failure Mode 3 (knowledge inheritance), Sa2VA adopts a design where SAM-2's spatial-temporal perception components are frozen during training (Table 5, Section 3.2). Only the SAM-2 decoder is fine-tuned, learning to interpret the [SEG] token's hidden states as a novel spatial-temporal prompt. This preserves SAM-2's billion-mask pre-training while allowing the MLLM side to evolve independently. The paper explicitly states:

"We adopt a decoupled design in which SAM-2's encoder and memory module are frozen, allowing us to retain its tracking capabilities. This design also makes our method a flexible framework, enabling our model to evolve with increasingly powerful MLLMs."

4. A new challenging benchmark (Ref-SAV) with an automatic annotation pipeline. To address the data gap, the paper introduces both a training dataset and an evaluation benchmark. The training set (37K videos, 72K expressions) is created through a fully automatic three-stage pipeline using InternVL2-76B and Qwen2-72B (Section 3.3), eliminating the need for expensive human annotation. The evaluation benchmark (1,147 videos, 1,945 expressions) is manually validated by human annotators and deliberately designed to stress-test models on long text, heavy occlusion, large camera motion, and large object motion simultaneously.

The paper is careful to position the Ref-SAV benchmark not as a solved problem but as a revealer of remaining gaps. Even Sa2VA-8B with full Ref-SAV training achieves only 50.0 J&F overall (Table 10)—a score that, while more than 4× better than the next-best zero-shot model, still leaves enormous room for improvement. This honest assessment of remaining challenges is consistent with the paper's framing of the work as establishing "a new strong baseline" rather than claiming to have solved the problem.

The Intellectual Lineage

The paper situates itself at the convergence of two research trajectories:

From the MLLM side: The lineage runs from LLaVA (Liu et al., 2023)—which pioneered visual instruction tuning by treating image features as visual tokens—through LISA (Lai et al., 2024) and GLaMM (Rasheed et al., 2024), which added segmentation capabilities by introducing special [SEG] tokens decoded by SAM. The paper extends this lineage to video by replacing SAM with SAM-2 and adding the temporal dimension to the [SEG] token mechanism.

From the video segmentation side: The lineage runs from early referring video object segmentation methods (ReferFormer, Wu et al., 2022) through universal perception models (UNINEXT, Yan et al., 2023) to foundation models like SAM-2 that unify image and video segmentation under a single architecture. The paper's key contribution from this perspective is giving SAM-2 language understanding capabilities through the MLLM bridge.

What distinguishes Sa2VA from both lineages is the insistence on maintaining capabilities rather than trading them off. LISA gained segmentation but lost chat. ReferFormer gained video referring but stayed within closed visual vocabularies. Sa2VA aims to keep everything—image chat, video chat, image segmentation, video segmentation, visual prompt understanding, grounded caption generation—in one model with one set of weights. Whether this ambition is fully realized is an empirical question the paper addresses through extensive benchmarking, but the motivation is clear: real-world interactive visual assistants need all of these capabilities simultaneously.

3. Technical Approach

3.1 Reader Orientation

Sa2VA is a single neural network that takes text, images, and video as input and produces both natural language responses and pixel-precise segmentation masks as output. The system solves the problem of building a unified visual assistant that can simultaneously chat about visual content AND segment and track objects based on language descriptions, by treating all visual modalities as tokens fed into a large language model and using a special token mechanism to bridge language outputs with a frozen segmentation foundation model.

3.2 Big-Picture Architecture (Diagram in Words)

The Sa2VA architecture consists of four major components connected in a feedforward pipeline:

Component 1: The Multi-Modal Large Language Model (MLLM). This is the central reasoning engine. It contains three sub-parts: (a) a visual encoder that converts images and video frames into visual feature vectors, (b) a visual projection layer that maps those features into the LLM's token embedding space, and (c) a large language model that processes both text tokens and projected visual tokens through its transformer layers and autoregressively generates output text tokens.

Component 2: SAM-2, the Frozen Perception Backbone. SAM-2 has three sub-parts: (a) an image/video encoder that produces dense spatial-temporal features from raw pixels and stores them in a memory bank, (b) a memory module that maintains and updates object-specific representations across video frames via cross-attention, and (c) a mask decoder that takes a prompt embedding and the encoded features, and produces binary segmentation masks. During Sa2VA training, the encoder and memory module are frozen—their weights never change. Only the mask decoder is fine-tuned.

Component 3: The [SEG] Token Bridge. This is the critical interface between the MLLM and SAM-2. When the MLLM needs to produce a segmentation mask, it generates a special text token called [SEG]. The hidden state (the LLM's internal vector representation at that token position) is extracted and treated as a spatial-temporal prompt embedding for SAM-2's mask decoder. This hidden state flows through a learned linear projection layer, then enters SAM-2's decoder alongside the visual features from SAM-2's encoder. The mask decoder produces the actual binary mask.

Component 4: The Output Decoders. Two separate decoders consume the LLM's output: (a) the LLM's standard prediction head (typically a linear layer followed by softmax over the vocabulary) produces text responses, and (b) SAM-2's mask decoder produces segmentation masks. The model routes information to the appropriate decoder based on which tokens the LLM generates—regular text tokens go to the language head, while [SEG] tokens route their hidden states to the mask decoder.

Information Flow for a Referring Segmentation Query:

1. The prompt text ("Please segment the person wearing sunglasses") is tokenized into text tokens.
2. The input image or video key frames are encoded by both the MLLM's visual encoder (producing visual tokens for the LLM) AND SAM-2's encoder (producing features for mask generation). These are two separate encoding pipelines that run in parallel.
3. The LLM receives the concatenation of visual tokens and text tokens as its input sequence.
4. The LLM autoregressively generates output tokens, including a [SEG] token.
5. The hidden state of the [SEG] token is extracted, projected through a linear layer, and fed as a prompt to SAM-2's mask decoder.
6. SAM-2's decoder combines this prompt with the pre-computed image/video features to produce a binary mask.
7. For video referring segmentation, this process runs on key frames first, then SAM-2's memory module propagates the mask to remaining frames using cross-attention between stored features and the newly decoded masks.

3.3 Roadmap for the Deep Dive

• First, I explain the unified task formulation (Section 3.1), which defines how fundamentally different tasks—image referring segmentation, video referring segmentation, chat, grounded caption generation, visual prompt understanding—are represented in a single mathematical framework using a shared LLM token space. Understanding this unification is essential because everything else in the architecture follows from the decision to treat all visual inputs as tokens.
• Second, I walk through the Sa2VA framework architecture (Section 3.2) in exhaustive detail: the pre-trained MLLM components, the decoupled SAM-2 integration, the [SEG] token mechanism for bridging language and vision, and the video inference pipeline that handles temporal propagation. This is the core technical contribution—how exactly the marriage of SAM-2 and MLLMs works at the implementation level.
• Third, I explain the Ref-SAV data annotation pipeline (Section 3.3), which is a three-stage automated process using InternVL2-76B and Qwen2-72B to generate long, detailed referring expressions for objects in complex videos. This is important because the dataset itself is a contribution, and the pipeline design reveals how the authors handle the challenges of automatic annotation quality control.
• Fourth, I cover the training procedure (Section 3.4) including the loss functions, the co-training data mixture, the hyperparameter choices, and the specific freezing/fine-tuning strategy. This section answers the question: what exactly happens during that "one-shot instruction tuning" and why does it prevent catastrophic forgetting?

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an architecture and systems paper whose core idea is that dense grounded understanding across images and videos can be achieved by treating a frozen video segmentation foundation model (SAM-2) as a mask decoder controlled by a special token emitted by a multi-modal large language model, with all components connected through a unified token-space interface and trained jointly on a mixture of chat, segmentation, and grounding data in a single instructional fine-tuning stage.

---

Unified Task Formulation: One Token Space to Rule Them All

The paper's foundational insight is that four seemingly distinct task categories—referring segmentation, visual question answering, grounded caption generation, and visual prompt understanding—can all be expressed as instances of a single generative process where an LLM takes a mixture of text tokens, image tokens, video tokens, and visual prompt tokens as input, and produces both text tokens and mask tokens as output.

The authors begin by examining each task type in isolation and showing that they differ only in which subsets of inputs are provided and which subsets of outputs are expected.

Referring Image Segmentation (RIS). Given input text tokens representing a referring expression (e.g., "the person wearing a red jacket") and an input image, the model must output a binary segmentation mask that precisely delineates the referred object. The paper formalizes this as: given text tokens $T_i \in \mathbb{R}^{N \times D}$ (where $N$ is the number of text tokens and $D$ is the token embedding dimension) and an image $I_i \in \mathbb{R}^{H \times W \times 3}$, the model produces a binary mask $M_o \in \{0, 1\}^{H \times W}$ that aligns with the text description.

Referring Video Object Segmentation (Ref-VOS). This extends RIS into the temporal domain. Given text tokens $T_i$ and a video $V_i \in \mathbb{R}^{T \times H \times W \times 3}$ (where $T$ is the number of frames), the model produces a spatio-temporal binary masklet $M_o \in \{0, 1\}^{T \times H \times W}$—essentially a binary mask for every frame of the video, all corresponding to the same referred object tracked across time.

Image/Video Chat and Grounded Caption Generation (GCG). For pure chat tasks, given text tokens $T_i$ and corresponding images $I_i$ or videos $V_i$, the model produces answer text $T_o$—no masks needed. For grounded caption generation, the model simultaneously outputs both masks $M_o$ and aligned text $T_o$, where the text describes specific regions and the masks identify which pixels correspond to which descriptions. This is a harder task because it requires the model to jointly reason about what to say and where the thing being described is located.

Visual Prompt Understanding. In addition to text tokens $T_i$ and images $I_i$, the model receives visual prompt tokens $VP_i$—these could be encoded representations of points, bounding boxes, or scribbles that a user has drawn on the image to indicate regions of interest. The model must then produce corresponding masks $M_o$ and aligned text answers $T_o$ that describe or reason about the indicated regions.

The Unification. The paper's key move is to observe that all these tasks can be collapsed into a single functional form:

$T_o, M_o = \text{LLM}(\{I_i, V_i, VP_i\}, T_i)$

where $\{I_i, V_i, VP_i\}$ is the set of all visual inputs provided (some may be empty depending on the task), $T_i$ is the input text tokens, and the LLM produces output text tokens $T_o$ and output masks $M_o$.

What this equation means operationally: The LLM receives a concatenated sequence of token embeddings from all input sources—visual embeddings from images, video frames, and visual prompts are projected into the LLM's token embedding space and interspersed with text token embeddings. The LLM processes this entire sequence through its transformer layers and generates output tokens autoregressively. Some of those output tokens are regular vocabulary tokens that form the text response; others are special [SEG] tokens that signal "produce a mask here." The hidden states of the [SEG] tokens are routed to SAM-2's decoder to generate the actual binary masks. The text tokens are decoded by the LLM's standard language modeling head.

Why this form matters: This formulation is powerful because it eliminates the need for task-specific architectures or training procedures. If a new task can be expressed by specifying which inputs to provide and whether to expect masks in the output, it can be handled by the same model with no architectural changes. The LLM's natural ability to handle variable-length sequences and mixed-modality inputs (a capability inherited from the base MLLM's training) means the system doesn't need separate encoders, decoders, or routing logic for different tasks. Everything flows through the LLM's token processing pipeline.

A critical subtlety: The paper is not claiming that the LLM generates masks directly. The LLM generates [SEG] tokens, and the hidden states of those tokens condition SAM-2's decoder to produce masks. But from the perspective of the training objective and the architectural interface, the system behaves as if it takes tokens in and produces tokens and masks out. The LLM learns to emit [SEG] tokens at the right positions in its output sequence, and the gradient from the mask loss flows back through the [SEG] token's hidden states into the LLM's parameters, teaching it what information to encode in those hidden states so that SAM-2's decoder can produce accurate masks.

---

The Sa2VA Framework: Marrying SAM-2 with MLLMs

The framework has two major components that are architecturally decoupled but connected through a single token-level interface. I'll describe each in detail, then explain how they interact.

######## Pre-trained Multi-Modal Large Language Models

The paper uses off-the-shelf pre-trained MLLMs without modifying their internal architecture. These models, including InternVL2 (Chen et al., 2024), InternVL2.5 (Chen et al., 2024), InternVL3 (Zhu et al., 2025), Qwen2.5-VL (Bai et al., 2025), and Qwen3-VL (Xu et al., 2025), all share the same high-level structure:

Visual Encoder. This is typically a Vision Transformer (ViT) or similar architecture that takes raw images, video frames, or sub-image crops as input and produces a sequence of visual feature vectors. For both image and video conversation tasks, Sa2VA uses "the same pipeline as the original pre-trained MLLMs" (Section 3.2)—meaning the visual encoding strategy, including how videos are sampled into frames and how multiple images are handled, is inherited directly from whichever base MLLM is being used.

Visual Projection Layer. This is typically a simple linear transformation or a small MLP that maps visual features from the encoder's output dimension to the LLM's token embedding dimension. The paper does not describe this in detail because it's inherited from the base MLLM, but its role is critical: it converts visual information into a format that the LLM can process alongside text tokens.

Large Language Model. This is the autoregressive transformer that forms the reasoning core. It takes a sequence of token embeddings (some from text, some from the visual projection layer, plus positional embeddings) and generates output token predictions one at a time. The paper uses models ranging from 0.5B to 20B parameters as the LLM backbone, depending on the specific MLLM variant. The LLM processes all tokens—text, image, video—through the same self-attention mechanism, enabling cross-modal reasoning through attention between text and visual tokens.

Inheritance, not redesign. The paper emphasizes that Sa2VA does not modify the MLLM architecture: "we utilize pre-trained MLLMs... to harness their powerful capabilities." This is a deliberate design choice that allows Sa2VA to "evolve with increasingly powerful MLLMs" as the open-source community releases better models. When InternVL2.5 becomes available, one can swap it in for InternVL2 without changing any other part of the Sa2VA framework.

######## The Decoupled SAM-2 Integration

The most important architectural decision in Sa2VA is what the authors call the "decoupled design": SAM-2's encoder and memory module are kept completely separate from the MLLM's processing pipeline, and SAM-2's output tokens are never fed back into the LLM.

What "decoupled" means concretely. When processing a referring segmentation query, the input image or video frames are encoded twice: once by the MLLM's visual encoder (producing tokens that the LLM can attend to for language understanding) and once by SAM-2's encoder (producing dense spatial-temporal features for mask generation). These two encodings serve different purposes and do not interact. The MLLM never sees SAM-2's internal features, and SAM-2 never sees the MLLM's visual features. The only connection point is the [SEG] token hidden state.

Why decouple? Three explicit reasons from the paper (Section 3.2):

1. Computational simplicity. "We aim to simplify the combination without incurring additional computation costs." If SAM-2's output features were fed back into the LLM, the LLM would need to process additional tokens (SAM-2 features for every frame, for every object), dramatically increasing the sequence length and the quadratic self-attention cost. By keeping SAM-2's features separate, the LLM's computational graph is unchanged from the base MLLM.

2. Knowledge preservation. "Adding extra tokens will impede knowledge inheritance from MLLMs because MLLMs and SAM-2 are misaligned." The MLLM has been pre-trained to process visual tokens from its own visual encoder, which have a specific statistical distribution and semantic content. SAM-2's features come from a different architecture trained on a different objective (mask prediction rather than vision-language alignment), and their distribution and semantics would differ substantially. Forcing the LLM to process these alien tokens would likely degrade its language and reasoning capabilities—the exact catastrophic forgetting problem the paper is trying to avoid.

3. Framework flexibility. "With this design, we can fully convert our work into a flexible framework that utilizes pre-trained, evolving MLLMs, as the MLLM community progresses rapidly." Because the only interface between the MLLM and SAM-2 is the [SEG] token hidden state (a fixed-dimensional vector), the two components can evolve independently. A better MLLM can be swapped in as long as it can generate tokens; a better SAM-2 can be swapped in as long as it accepts prompt embeddings. There is no tight architectural coupling that would break when either component changes.

What gets frozen and what gets trained. Table 5 (Section 4) specifies the training strategy:

• MLLM: Trained using LoRA (Low-Rank Adaptation) with rank $r = 256$. This means the original MLLM weights are frozen, and only low-rank adapter matrices are trained, which dramatically reduces the number of parameters being updated and helps prevent catastrophic forgetting of the base MLLM's pre-trained capabilities.

• SAM-2 Encoder: Frozen completely. The visual features produced by SAM-2's encoder are used as-is, with no gradient flowing back into them during Sa2VA training.

• SAM-2 Decoder: Fully fine-tuned. This is the component that learns to interpret the [SEG] token hidden states as mask prompts and produce accurate segmentation masks.

• SAM-2 Memory Module: Frozen completely. The cross-attention mechanism that maintains and updates object representations across video frames is preserved exactly as pre-trained by SAM-2's original training on 1 billion masks.

Why this specific freezing strategy? The SAM-2 encoder and memory module encapsulate the spatial-temporal perception capabilities learned from massive pre-training—capabilities the authors want to preserve intact. The SAM-2 decoder, however, was originally trained to accept visual prompts (points, boxes, masks) rather than language-derived embeddings. Fine-tuning the decoder teaches it to interpret the MLLM's [SEG] hidden states as a new type of prompt, effectively adding a language-to-visual-prompt translation capability without disturbing the underlying perception machinery.

######## The [SEG] Token Mechanism: Bridging Language and Vision

This is the linchpin of the entire architecture. The paper adapts a technique previously used in image-only models like LISA (Lai et al., 2024) and extends it to handle video through SAM-2's temporal capabilities.

What the [SEG] token is. The [SEG] token is a special token added to the LLM's vocabulary. It is not a word or concept—it is a signal that says "produce a segmentation mask for the object being discussed at this point in the output sequence." When the LLM generates a [SEG] token, the internal hidden state vector at that position in the transformer's output is extracted.

How the hidden state becomes a mask prompt. The extracted hidden state $h_{[SEG]} \in \mathbb{R}^{D_{LLM}}$ (where $D_{LLM}$ is the LLM's hidden dimension) is passed through a learned linear projection layer:

$p = W_{\text{proj}} \cdot h_{[SEG]} + b_{\text{proj}}$

where $W_{\text{proj}} \in \mathbb{R}^{D_{SAM} \times D_{LLM}}$ maps from the LLM's hidden dimension to SAM-2's prompt embedding dimension, and $b_{\text{proj}}$ is a bias term. The projected vector $p$ is the spatial-temporal prompt embedding that enters SAM-2's mask decoder.

Why a linear projection? The LLM's hidden space and SAM-2's prompt space are in different dimensionalities and have different semantic structures. A simple linear layer provides enough flexibility to align these spaces without introducing enough parameters to cause overfitting or interfere with the LLM's primary training objective. More complex projections (e.g., MLPs, attention layers) would risk the same catastrophic forgetting problem the decoupled design avoids.

How the mask is generated. SAM-2's mask decoder takes two inputs: (1) the prompt embedding $p$ from the [SEG] token, and (2) the image or video frame features $F$ from SAM-2's frozen encoder. The decoder's architecture (inherited from SAM-2's original design) uses cross-attention between the prompt embedding and the spatial features, followed by upsampling layers, to produce a binary mask $M_o$ at the original image resolution.

What this enables. The LLM learns to encode rich semantic information into the [SEG] token's hidden state—information about which object is being referred to, its shape, its location, its relationship to other objects, etc.—such that when this hidden state is projected into SAM-2's prompt space, SAM-2's decoder can identify the correct object and produce an accurate mask. The gradient from the mask loss flows back through the decoder, through the linear projection, through the [SEG] token's hidden state, and into the LLM's parameters (specifically the LoRA adapter weights), teaching the LLM what information to encode.

The paper describes this compactly (Section 3.2):

"The hidden states of the '[SEG]' token serve as a new kind of prompt, which are fed into SAM-2's Decoder, where they are decoded into segmentation masks. The hidden states of '[SEG]' can be viewed as a novel spatial-temporal prompt for SAM-2."

Extension to video: the single vs. multiple token question. A subtle design choice arises for video: should one [SEG] token represent the object across all frames, or should each frame get its own token? The paper explores three strategies (Table 12):

• Single token: The LLM outputs one [SEG] token for the entire video. Its hidden state is used as the prompt for all frames. This is the default and best-performing strategy.
• Repetitive tokens: The LLM outputs the same [SEG] token $N$ times for $N$ video frames. Each frame's mask is decoded from the hidden state of its corresponding token position.
• Unique tokens: The LLM outputs distinct tokens like [SEG 1], [SEG 2], etc. for different frames.

The single token strategy works best because "it enables knowledge sharing with the image segmentation task. The model is typically pre-trained to associate a single, generic [SEG] token with the concept of segmentation." The paper notes that introducing unique tokens creates a mismatch between image and video processing that "blocks this knowledge transfer, forcing the model to learn the meaning of each new segmentation token independently."

Ablation results (Table 12, Sa2VA-1B):

• Single token: RefCOCO 77.4, RefCOCO+ 69.9, RefCOCOg 72.3, DAVIS 72.3, MeViS(val_u) 50.8
• Repetitive: RefCOCO 77.3, RefCOCO+ 70.2, RefCOCOg 72.5, DAVIS 71.1, MeViS(val_u) 49.6
• Unique: RefCOCO 77.6, RefCOCO+ 70.3, RefCOCOg 72.4, DAVIS 68.6, MeViS(val_u) 46.3

The unique token strategy shows a dramatic 4.5 J&F drop on MeViS compared to the single token approach, confirming that knowledge sharing across the temporal dimension is crucial for video performance.

######## The Video Referring Segmentation Inference Pipeline

For referring video object segmentation (Ref-VOS), Sa2VA uses a two-stage inference procedure described in Algorithm 1 of the paper.

Stage 1: Key frame segmentation via MLLM. The system extracts the first $M$ frames from the video (where $M$ is typically 5 for the default configuration). These key frames are encoded by both the MLLM's visual encoder (producing visual tokens for the LLM) and SAM-2's encoder (producing features for mask generation). The LLM processes the visual tokens and text tokens, generating output that includes one or more [SEG] tokens. For each key frame, the hidden state of the [SEG] token is projected and fed to SAM-2's decoder to produce a mask.

Stage 2: Propagation to remaining frames via SAM-2's memory. After processing the key frames, SAM-2's memory module has accumulated object-specific representations through cross-attention between the decoded masks and the stored frame features. For each subsequent frame $i > M$, the process is:

1. SAM-2's encoder processes the frame to produce features $F_i$.
2. SAM-2's decoder takes the memory representation (which encodes the object's appearance and motion from previous frames) and the current frame's features $F_i$.
3. Cross-attention between the memory and the features identifies where the object is in the current frame.
4. The decoder produces the mask $M_i$.
5. The memory is updated with the new mask and features via cross-attention.

This leverages SAM-2's pre-trained video tracking capabilities: the model was trained on over 1 billion masks across videos and learned to propagate object identities through time using its memory mechanism. Sa2VA doesn't need to retrain this capability; it simply provides the initial mask on key frames and lets SAM-2 handle the temporal propagation.

Key frame sampling strategy. The paper's default uses the first 5 frames, chosen "for simplicity and consistency" (Section 4.4). However, Table 20 compares different strategies:

• First 1 frame: 55.1 J&F on MeViS(val_u)
• First 3 frames: 58.7 J&F
• First 4 frames: 59.5 J&F
• First 5 frames: 58.9 J&F
• Uniform 5 frames (spread across entire video): 62.9 J&F

The uniform sampling strategy significantly outperforms the default, but the authors note they "would like to keep Sa2VA as simple as possible and leave designing advanced techniques on sampling as future work." The performance gap (62.9 vs. 58.9 J&F) with identical inference time suggests that more sophisticated key frame selection is a promising direction for future improvement.

A important detail about token and feature handling. The paper notes that different base MLLMs have different image tokenization strategies. When using the InternVL2 family, the MLLM input image size is 448×448 (producing a certain number of visual tokens), while SAM-2 operates at 1024×1024 input resolution. These two pipelines are completely independent—the MLLM's visual tokens capture global scene context at moderate resolution for language understanding, while SAM-2's features capture fine spatial detail at high resolution for precise mask boundaries.

Inference speed analysis (Table 19). The computational cost is dominated by the autoregressive MLLM text generation, not SAM-2's mask decoding. On an NVIDIA H100 GPU with fixed 5-frame + 1280-token input:

• Sa2VA-1B: 0.123 seconds per sample
• Sa2VA-4B: 0.282 seconds per sample
• Sa2VA-8B: 0.201 seconds per sample
• Sa2VA-26B: 0.463 seconds per sample

The paper notes that SAM-2 is "relatively lightweight at 220M parameters... achieving 39.5 FPS on vision tasks with only a single forward pass per image." The discrepancy between the 4B and 8B inference times (0.282s vs. 0.201s) is attributed to "the different base LLMs (Qwen vs. InternLM) employed" and optimization differences, highlighting that inference speed depends as much on the base model choice as on the Sa2VA architectural additions.

---

The Ref-SAV Dataset: Automated Annotation Pipeline

The paper introduces Ref-SAV, a large-scale referring video object segmentation dataset, created through a fully automatic pipeline that leverages existing powerful vision-language models (InternVL2-76B and Qwen2-72B) to generate detailed referring expressions without human annotation cost.

######## The Three-Stage Annotation Pipeline

The pipeline, illustrated in Figure 3 of the paper, proceeds through three sequential stages, each adding a layer of descriptive detail:

Stage 1: Object-Level Annotation. For each annotated object in the SA-V dataset (SAM-2's training data), the system selects the video frame where the object occupies the largest area (making it most visible and easiest to describe). Two versions of this frame are prepared: a "cropped" version where all pixels outside the object mask are removed (showing only the object against a blank background), and a "full" version where the object is highlighted with its mask applied (showing the object in context, with everything else blacked out). These two complementary views are separately fed into InternVL2-76B to generate detailed descriptions of the object's appearance.

The descriptions from the cropped and full views are then processed by Qwen2-72B for consistency checking. The paper states that "conflicting descriptions are discarded." This is a crucial quality control step: if the two views produce contradictory descriptions (e.g., one description says the object is "blue" and the other says "green"), the entire annotation is dropped. This filtering mechanism allows the automatic pipeline to "filter out error-prone cases" without any human intervention.

Stage 2: Scene-Level Annotation. The full image (without masking) and the object-level description from Stage 1 are fed together into InternVL2-76B. The model is prompted to generate a description that enriches the object description with contextual information: "relationships to the scene and surrounding objects." For example, an object-level description might say "a black goat wearing a blue collar" while the scene-level annotation adds "it is moving around in a natural outdoor environment with grass and trees."

This stage addresses a key limitation of existing Ref-VOS datasets: their referring expressions typically describe only the object itself, without situating it in the visual context. Real-world referring expressions often include relational information ("the person next to the red car," "the bird on the branch in the upper left"), and the scene-level annotation captures these relationships.

Stage 3: Video-Level Annotation. The system samples 8 frames uniformly from the video. In each frame, the target object is visually highlighted with a yellow border (the paper states "applying yellow borders to emphasize the object in each frame"). These annotated frames, together with the scene-level description from Stage 2, are processed by InternVL2-76B to generate a video-level description that "captures the object's movement and actions."

This stage adds temporal dynamics that are completely absent from image-only referring datasets. The video-level annotation might describe how the object moves ("the goat walks across the field"), its actions ("the car turns left at the intersection"), or its state changes ("the person sits down and then stands up"). This temporal information is critical for disambiguating between multiple similar objects in a video—two people wearing similar clothing might be distinguished by their motion patterns.

######## Dataset Statistics and Comparison to Prior Work

The resulting Ref-SAV training set (Table 4) contains 37,311 videos with 72,509 object expressions, totaling approximately 6.0 million mask annotations. The average referring expression length is 83.6 words—roughly 8–12 times longer than existing datasets (Ref-YTVOS averages 9.68 words, MeViS averages 7.07 words, ReVOS averages 10.5 words).

Why such long expressions? The pipeline's three-stage design naturally produces long, compositional descriptions. An object-level description (e.g., 30 words about appearance) plus scene-level context (e.g., 30 words about surroundings) plus video-level dynamics (e.g., 25 words about motion) combine into a comprehensive ~85-word expression. This is deliberate: the paper wants to test whether models can handle the kind of detailed, multi-faceted descriptions that users might naturally produce when interacting with a visual assistant, rather than the terse, simplified expressions common in existing benchmarks.

The Ref-SAV evaluation benchmark. The paper creates a separate validation benchmark by selecting a subset of SA-V videos not used in Ref-SAV training. This benchmark has two parts:

1. Long-expression set: 1,694 expressions generated by the automatic pipeline and "carefully filtered by human annotators." Human filtering at this stage is important—it ensures that the expressions in the evaluation set are actually correct and refer to identifiable objects, providing a clean measurement of model capability.

2. Short-expression set: 251 expressions that are manually annotated from scratch by human annotators. These provide a contrast to the long expressions, testing whether models can also handle concise, direct referring expressions.

The benchmark totals 1,147 videos and 1,945 object expressions, making it substantially larger than prior Ref-VOS validation sets.

What makes Ref-SAV challenging (Table 3). The paper characterizes the benchmark across five dimensions:

• Heavy occlusion: Objects are frequently partially or fully occluded by other objects, requiring models to track through periods of invisibility. No prior benchmark systematically includes heavy occlusion.
• Long text: Expressions averaging 83.6 words require models to parse and integrate multi-faceted descriptions. No prior benchmark exceeds ~10 words on average.
• Large object motion: Objects move significantly across frames, testing tracking robustness.
• Large camera motion: Camera movement (panning, zooming, handheld shake) changes the entire scene, testing whether models can re-identify objects after viewpoint changes.
• Short text capability: The benchmark also tests short expressions (the 251 manually annotated ones) to ensure models aren't overfitting to long-form descriptions.

The paper's claim is that Ref-SAV is the first benchmark to simultaneously stress-test all five of these dimensions. The results in Table 10 support this: state-of-the-art Ref-VOS models achieve only 7.6–11.8 J&F zero-shot on Ref-SAV (versus much higher scores on existing benchmarks), suggesting Ref-SAV indeed captures failure modes that prior evaluations miss.

Effectiveness of the training data. Table 17 demonstrates that training on Ref-SAV improves a representative model (UniRef++) from 10.5 to 14.6 overall J&F—a 39% relative improvement. This validates that the automatically generated annotations, despite having no human labeling, contain useful training signal. The paper also shows (Table 13) that adding Ref-SAV training data to Sa2VA-1B improves MeViS performance by 1.7 J&F (from 50.8 to 52.5) while having negligible impact on image segmentation scores (RefCOCO drops only 0.2 cIoU), suggesting the video-specific knowledge transfers well without interfering with image capabilities.

---

Sa2VA Training: One-Shot Instruction Tuning

The paper's training procedure is notable for what it doesn't include: there is no pre-training phase, no multi-stage curriculum, no task-specific fine-tuning. The model is trained in a single supervised fine-tuning stage ("one-shot instruction tuning") on a mixture of four data types.

######## The Training Objective

The overall loss function combines two terms:

$\mathcal{L}_{\text{instruction}} = \mathcal{L}_{\text{text}} + \mathcal{L}_{\text{mask}}$

where $\mathcal{L}_{\text{text}}$ is the standard autoregressive language modeling loss (cross-entropy between predicted and ground-truth next tokens, applied to text outputs), and $\mathcal{L}_{\text{mask}}$ is the mask prediction loss, defined as:

$\mathcal{L}_{\text{mask}} = \mathcal{L}_{\text{CE}} + \mathcal{L}_{\text{DICE}}$

where $\mathcal{L}_{\text{CE}}$ is pixel-wise binary cross-entropy between the predicted mask and the ground-truth binary mask, and $\mathcal{L}_{\text{DICE}}$ is the Dice loss, a common segmentation metric that measures overlap between predicted and ground-truth masks.

What this computes operationally: For each training example, the model produces text outputs (for which $\mathcal{L}_{\text{text}}$ applies only to the text tokens) and, if the example includes segmentation ground truth, mask outputs (for which $\mathcal{L}_{\text{mask}}$ applies pixel-by-pixel). The two loss terms are summed with equal weight (implicitly, since no weighting coefficients are specified). The gradient from $\mathcal{L}_{\text{mask}}$ flows back through SAM-2's decoder (fully trained), through the [SEG] token linear projection (trained), through the [SEG] token's hidden state, and into the LLM's LoRA adapter weights. The gradient from $\mathcal{L}_{\text{text}}$ flows back through the LLM's LoRA adapter weights and the visual projection layer.

Why Dice loss alongside cross-entropy? In segmentation, cross-entropy alone tends to produce acceptable results on average but can miss fine boundary details. The Dice loss directly optimizes region overlap (intersection over union), which is more sensitive to boundary accuracy and handles class imbalance better (most pixels in a segmentation mask are background, so cross-entropy can achieve low loss by simply predicting "background" everywhere). Using both losses combines the calibration benefits of cross-entropy with the overlap-sensitivity of Dice.

######## Training Data Composition

The paper trains on approximately 1.1 million image-text or video-text pairs drawn from four categories (Table 2):

Image QA: 665K examples from LLaVA 1.5. The paper notes: "Since InternVL2 has been trained with a large amount of image QA and video QA data, we only used 665K LLaVA 1.5 and 100K Video-ChatGPT data to prevent the MLLM from forgetting its image and video QA capabilities." This is a crucial insight: Sa2VA doesn't need to relearn general visual understanding from scratch because the base MLLM already has it. The image QA data serves as a "reminder" to maintain capabilities rather than a primary training signal.

Image Segmentation: 270K examples. This includes 56K referring expression data from RefCOCO (17K), RefCOCO+ (17K), and RefCOCOg (22K), plus 214K grounded conversation generation data from Grand-f (the GLaMM dataset). The referring expression data provides standard image referring segmentation training; the GCG data teaches the model to jointly produce masks and descriptions.

Video QA: 100K examples from Video-ChatGPT. Similar to image QA, these maintain video understanding capabilities inherited from the base MLLM.

Video Segmentation: 42.8K examples. This includes 3.5K from Ref-YTVOS, 0.6K from MeViS, 1.7K from ReVOS (the existing Ref-VOS datasets, totaling 5.8K), plus 37K long-text referring VOS data from Ref-SAV (the automatically annotated dataset). The Ref-SAV data dominates the video segmentation training portion (37K out of 42.8K, or ~86%), which is deliberate: the paper wants to teach the model to handle long, complex referring expressions that existing datasets don't cover.

The balance between data types. The paper's co-training strategy is designed to prevent the catastrophic forgetting observed in prior works. By including all four data types simultaneously in training, the model is forced to maintain performance on all tasks. The ablation study in Table 9 (discussed in the experimental section) confirms that removing any single data type causes significant degradation in the corresponding capability.

######## Training Hyperparameters and Configuration

The paper provides detailed training specifications (Section 4):

Optimization: AdamW optimizer with initial learning rate $4 \times 10^{-5}$ and weight decay $0.05$. The paper uses a 5% warmup ratio (the learning rate linearly increases from 0 to $4 \times 10^{-5}$ over the first 5% of training steps, then decays according to a cosine schedule). Gradient clipping is applied with a maximum gradient norm of 1.0 to prevent training instabilities.

LoRA parameters: Rank $r = 256$, applied to the LLM's attention layers. LoRA (Low-Rank Adaptation) adds trainable low-rank matrices to the frozen pre-trained weights: for a weight matrix $W \in \mathbb{R}^{d \times k}$, LoRA adds $\Delta W = B A$ where $B \in \mathbb{R}^{d \times r}$ and $A \in \mathbb{R}^{r \times k}$ are trainable, and $r \ll \min(d, k)$. The rank $r=256$ is relatively high for LoRA (common values are 8–64), suggesting the model needs substantial adaptation capacity to learn the segmentation capabilities while maintaining language performance.

Input processing: The MLLM processes images at 448×448 resolution (producing visual tokens for the LLM). SAM-2 processes images at 1024×1024 resolution (producing high-resolution features for mask decoding). These different resolutions reflect their different roles: the MLLM needs enough resolution for scene-level language understanding; SAM-2 needs higher resolution for precise mask boundaries.

Sequence length: The LLM's maximum sequence length is set to 8,192 tokens. This accommodates the concatenation of visual tokens (from multiple images or video frames) plus text tokens plus the generated output.

Training regime: One epoch of training with a total batch size of 256. This is achieved using 16 NVIDIA H100 80GB GPUs with a per-device batch size of 2 and 8 gradient accumulation steps (so each device processes 2 examples, and gradients from 8 forward/backward passes are accumulated before an optimizer step, giving an effective batch size of $16 \times 2 \times 8 = 256$). Training uses bfloat16 (bf16) mixed precision for memory efficiency. The instruction tuning stage lasts approximately 24 hours.

Why one epoch? The paper's datasets total ~1.1 million examples, and the model is being fine-tuned (not trained from scratch) with LoRA (which adapts only a fraction of parameters). One epoch provides sufficient training signal for adaptation without overfitting. More epochs would risk the model memorizing segmentation examples at the expense of general conversation ability—exactly the catastrophic forgetting the co-training strategy aims to avoid.

Training components (Table 5):

• MLLM: LoRA training (rank 256)
• SAM-2 Encoder: Frozen
• SAM-2 Decoder: Fully fine-tuned
• SAM-2 Memory: Frozen

What this training strategy implies. The model is learning a specific skill: how to encode object identity, location, shape, and context information into the [SEG] token's hidden state such that SAM-2's decoder can reliably produce the correct mask. The SAM-2 decoder is learning to interpret these language-derived prompts as a new type of conditioning signal. Everything else—visual feature extraction, temporal tracking, general vision-language understanding—is preserved from pre-training.

######## Testing Protocol: One Model for All Tasks

The paper emphasizes that Sa2VA uses the same model weights for all tasks with no task-specific fine-tuning. During inference:

"we encode the necessary task requirements, such as text prompts, visual prompts, image features, and video features, into tokens to input into the LLM. The output tokens from the LLM are then decoded into text responses (LLM prediction head), segmentation masks (SAM-2 decoder), and responses from the SAM-2 mask tracking module according to the task definition."

The model determines what to output based on the input format and prompt structure. If the prompt asks a question ("What is the weather?"), the model generates only text tokens. If the prompt asks for segmentation ("Please segment the person"), the model generates both text tokens (e.g., "Sure, it is [SEG]") and a [SEG] token that triggers mask generation. If the prompt asks for grounded captioning, the model generates interleaved text and [SEG] tokens, with masks produced for each [SEG] and text describing each masked region.

This "one for all" testing protocol is a direct consequence of the unified task formulation in Equation 1. By reducing all tasks to the same input-output interface, the same model can handle any task without architectural switching, task-specific heads, or routing logic.

######## Extension to Different Base MLLMs

A key claim of the paper is that Sa2VA is a "flexible framework" that works with different base models. Table 14 demonstrates this by reporting results with seven different base MLLMs: InternVL2, InternVL2.5, InternVL3 (at 2B, 8B, and 14B scales), Qwen2.5-VL (3B and 7B), and Qwen3-VL (4B). The consistent strong performance across all these variants validates the framework's modularity.

What changes and what stays the same. The paper doesn't describe model-specific adaptations in detail, but the architecture implies that swapping base MLLMs requires: (1) using the new MLLM's visual encoder and projection layer for producing visual tokens, (2) using the new MLLM's LLM for token processing, (3) potentially adjusting the [SEG] token's linear projection layer to match the new LLM's hidden dimension, (4) retraining the LoRA adapters (which are attached to the new LLM's specific weight matrices). SAM-2, its encoder, its memory module, and the mask loss remain unchanged regardless of the MLLM choice.

The results show that model choice matters: InternVL3-14B achieves the best segmentation scores (83.6 cIoU on RefCOCO, 59.2 J&F on MeViS), while Qwen3-VL-4B achieves the best MMBench score (86.3). This variation reflects inherent differences in the base models' capabilities—InternVL3 appears stronger at visual grounding, while Qwen3-VL appears stronger at general multimodal understanding. The Sa2VA framework simply inherits these differences without exacerbating or mitigating them.

4. Key Insights and Innovations

Innovation 1: The Decoupled Architecture as a Fundamental Reframing of How to Combine Vision Foundation Models with LLMs

The dominant approach for combining segmentation models with MLLMs prior to Sa2VA was to feed segmentation features back into the LLM—either by concatenating mask-derived tokens into the LLM's input sequence or by using the LLM as a routing agent that explicitly calls external segmentation APIs. LISA (Lai et al., 2024) used SAM's encoded features as additional tokens processed by the LLM. GLaMM (Rasheed et al., 2024) introduced region-level tokens that flow through the LLM's self-attention layers alongside text tokens. VideoGLaMM (Munasinghe et al., 2024) extended this to video by having the LLM process temporal features from the segmentation backbone. The implicit assumption across all these works was that for the LLM to produce good masks, it must directly process the visual features that will be used for mask generation.

Sa2VA's decoupled design is a conceptual rejection of this assumption. The key insight is counterintuitive: the LLM does not need to see SAM-2's features to produce accurate segmentation prompts—it only needs to encode the identity of the object being referred to in a form SAM-2's decoder can interpret. The LLM processes the scene through its own visual encoder (at moderate resolution, optimized for language understanding), reasons about which object is being referred to, and compresses this reasoning into a single vector (the [SEG] token hidden state). SAM-2, processing the same scene through its own high-resolution encoder (optimized for precise boundaries), takes this identity vector and localizes it spatially.

The three explicit justifications the paper gives for decoupling—computational simplicity, knowledge preservation, and framework flexibility—are engineering motivations. But the deeper conceptual contribution is the separation of semantic reasoning from spatial precision. The LLM handles who or what is being referred to (requiring language understanding, world knowledge, and relational reasoning); SAM-2 handles where the referred entity is in pixel space (requiring fine-grained visual perception and tracking). Neither component needs to do the other's job, and forcing them to do so (by feeding SAM-2 features into the LLM) introduces a difficult alignment problem: the LLM must learn to extract spatial information from features it was never pre-trained to process, while simultaneously maintaining its language capabilities.

This reframing is fundamental rather than incremental because it changes the design philosophy for future multi-modal systems. It suggests that when combining models pre-trained on different objectives, the interface between them should be as narrow as possible—a single information bottleneck that forces each component to abstract away the details the other doesn't need. Prior work assumed rich communication between components; Sa2VA shows that minimal communication can work better.

The evidence for this claim is both architectural (the framework works across seven different base MLLMs with no model-specific adaptations, Table 14) and empirical (the catastrophic forgetting that plagues LISA and GLaMM is avoided, with MME scores of 1651/578 for Sa2VA-8B versus 1/1 for LISA-7B, Table 6). The fact that a simple linear projection from the LLM's hidden state to SAM-2's prompt space is sufficient—no attention, no multi-layer transformation, just a learned dimensionality reduction—is strong evidence that the [SEG] token is primarily encoding semantic identity rather than spatial coordinates.

Innovation 2: Co-Training as a Solution to the Grounding-Chat Trade-off, Not as a Mere Training Recipe

The field's experience prior to Sa2VA strongly suggested an inherent tension between dense grounding and open-ended conversation. LISA-7B achieved 74.1 cIoU on RefCOCO but scored 1/1 on MME—effectively zero conversational ability. GLaMM-7B scored 14/9 on MME. LaSagnA scored 0/0. The common interpretation, implied by Zhang et al. (2024) and Wu et al. (2024), was that fine-tuning an MLLM for segmentation unavoidably destroys its language capabilities—that the high-dimensional spatial reasoning required for pixel-level prediction fundamentally interferes with the token-level sequence modeling that produces coherent text.

Sa2VA's key intellectual move is to reframe this not as an intrinsic trade-off but as a data imbalance problem. The hypothesis is that prior works lost conversational ability not because segmentation and conversation are fundamentally incompatible, but because their training data was overwhelmingly skewed toward segmentation (thousands of segmentation examples with few or no conversational examples). The model, optimizing a single loss function over an imbalanced dataset, simply forgot how to do the things it wasn't being asked to do.

The evidence for this reframing is the ablation study in Table 9. Removing image QA data from training causes MME to drop by 83 points (from 1381 to 1298 perception) and MMBench to drop by 4.9 points. Removing video QA data causes MMBench-Video to drop by 34%. These are substantial degradations, but more importantly, they are selective—removing segmentation data doesn't dramatically boost chat performance, and removing chat data doesn't dramatically boost segmentation performance. The capabilities are maintained independently, suggesting they don't interfere with each other as long as both are presented during training.

This is a fundamentally different claim than "we found the right balance of losses" or "we added a regularization term." The paper is arguing that the previously observed trade-off was an artifact of experimental design (monolithic segmentation training without conversational data) rather than an intrinsic limitation of neural architectures. This has significant implications for the field because it suggests that any MLLM can be extended to handle segmentation while preserving its original capabilities, as long as the training mixture includes representative examples of all the desired tasks.

The paper's choice to include only 665K image QA examples (from LLaVA 1.5) and 100K video QA examples (from Video-ChatGPT)—relatively small amounts compared to the base MLLM's original pre-training—is telling. The base model (InternVL2) was pre-trained on a much larger corpus, including massive amounts of image and video QA data. The 765K QA examples in Sa2VA's fine-tuning serve as a "reminder" rather than a source of new knowledge—they prevent the model's language capabilities from being overwritten during the segmentation-focused adaptation. This is conceptually aligned with the LoRA fine-tuning strategy (rank 256, frozen base weights), where the model can only make low-rank adjustments to its pre-trained knowledge, making it harder to completely erase capabilities.

The significance of this innovation is not in the specific data mixture (which is domain- and model-dependent) but in establishing that the grounding-chat trade-off is solvable through data engineering rather than requiring architectural innovations. Prior work that accepted catastrophic forgetting as inevitable would have pursued fundamentally different solutions (e.g., separate models, task-specific routing, architectural isolation). Sa2VA shows these are unnecessary.

Innovation 3: The [SEG] Token as a Learned Spatial-Temporal Prompt — Introducing a New Modality of Model Communication

The technique of using special tokens to trigger external model calls was established before Sa2VA—LISA (2024) introduced the [SEG] token to prompt SAM for image segmentation, and VISA (2024) and VideoLISA (2024) extended this to video via SAM. What makes Sa2VA's treatment of the [SEG] token distinctive is the conceptual elevation from an implementation trick to a principled communication channel: the [SEG] token hidden state is treated as a learned spatial-temporal prompt that carries object identity information across the temporal dimension.

This reframing addresses a non-obvious problem that prior work didn't fully engage with: how should a single token represent an object across multiple frames of a video? LISA and VISA used the [SEG] token frame-by-frame, essentially treating each frame as an independent referring segmentation task. This works for short clips but fails to leverage SAM-2's temporal tracking capabilities. Sa2VA's insight is that a single [SEG] token should encode an object's identity (which is stable across time) while SAM-2's memory mechanism handles the temporal dynamics (where the object moves, how it changes appearance, when it's occluded). The [SEG] token hidden state is a prompt not just for segmentation but for tracking—it tells SAM-2 "track the object with this identity across all frames."

The ablation in Table 12 provides direct evidence for this interpretation. The single-token strategy achieves 50.8 J&F on MeViS(val_u); the "unique tokens" strategy (different [SEG] tokens per frame) achieves only 46.3 J&F. The paper attributes this gap to knowledge transfer from image segmentation: the model learns that [SEG] means "the object we're talking about" through image training, and this semantic association transfers to video. Unique tokens break this transfer because each new token has to be learned from scratch. But there's a deeper reason: unique tokens would require the LLM to encode motion information in the token hidden states (so that [SEG 1] identifies the object in the first frame, [SEG 2] in the second, etc.), which is fundamentally harder than encoding identity and letting SAM-2 handle motion. The single-token design reflects a clean division of labor: the LLM handles what; SAM-2 handles where and when.

The design choice also connects to the broader debate about how multimodal models should represent time. Should a language model reason about temporal dynamics explicitly by generating token-by-token frame descriptions, or should it delegate temporal processing to a specialized module? Sa2VA's answer is decisive: delegate. The LLM provides a stable identity signal, and SAM-2's pre-trained temporal machinery (cross-attention memory, learned from 1 billion masks) handles everything time-varying. This is efficient (the LLM doesn't need to process temporal features) and effective (SAM-2's tracking is far more capable than anything an LLM could learn from limited fine-tuning data).

This innovation is significant because it introduces a new design pattern for LLM-expert model communication: a single information bottleneck token whose hidden state encodes a semantic query, with the expert model responsible for interpreting that query across a complex input space. This pattern is more general than referring segmentation—it could apply to any setting where an LLM needs to select or manipulate entities in a rich input space (audio segments, 3D objects, database records) without processing the full complexity of that space itself.

Innovation 4: The Ref-SAV Benchmark as a Diagnostic Tool Revealing That Current Ref-VOS Models Fail on Compositional Complexity, Not Just Visual Difficulty

On the surface, Ref-SAV is a dataset contribution—a new benchmark with more videos, longer text, and more challenging conditions than prior Ref-VOS evaluations. But the paper's deeper contribution is using Ref-SAV to reveal which specific failure modes are the bottleneck for current models, distinguishing between visual difficulty (can the model see the object?) and linguistic compositional difficulty (can the model understand what's being referred to?).

Prior Ref-VOS benchmarks conflated these two dimensions. Ref-YTVOS has short, simple expressions but relatively clean videos—models could fail because they couldn't track through motion, or because they couldn't parse the (simple) language, but you couldn't tell which. Ref-DAVIS17 has challenging visual conditions (camera motion, viewpoint changes) but also simple referring expressions. The field lacked a benchmark that systematically increased linguistic complexity while holding visual complexity constant (or vice versa), making it impossible to diagnose whether a model's failures were perceptual or semantic.

Ref-SAV's design—with separate long-expression and short-expression subsets on the same videos—enables this diagnosis. If a model performs well on short expressions but poorly on long ones for the same videos, the bottleneck is linguistic processing. If it performs poorly on both, the bottleneck is visual tracking. The results in Table 10 show a clear pattern: Sa2VA-8B achieves 49.3 J&F on long expressions and only 33.3 J&F on short expressions. This is puzzling at first glance—shouldn't shorter expressions be easier? The paper doesn't explicitly analyze this counterintuitive result, but it reveals something important: the automatically generated long expressions may be more precise than the manually written short ones. A long expression like "the object is a black goat wearing a blue collar, moving around in a natural outdoor environment with grass and trees, walking across the field from left to right" provides multiple disambiguating cues (color, context, motion direction), while a short expression like "the goat" provides only the object category. If there are multiple goats in the video, the long expression actually makes the task easier by providing more grounding information.

This diagnostic power—the ability to attribute failures to specific aspects of the task—is what elevates Ref-SAV from a dataset contribution to an intellectual one. The benchmark doesn't just say "this is hard"; it provides differential diagnosis of model capabilities. The finding that prior state-of-the-art models (UNINEXT, VISA) achieve only 7.6–11.8 J&F on Ref-SAV suggests that their strong performance on existing benchmarks came primarily from handling simple, unambiguous referring expressions in visually stable videos, not from genuine visiolinguistic understanding. Ref-SAV exposes this by demanding both dimensions simultaneously.

The automatic annotation pipeline (the three-stage process using InternVL2-76B and Qwen2-72B) is a practical contribution, but its conceptual significance is that it demonstrates scalable generation of linguistically complex visual grounding data without human annotation. The quality control mechanism—comparing descriptions from cropped versus full-image views and discarding conflicting ones—is an elegant solution to the hallucination problem in LLM-generated annotations. This matters because it suggests a path toward arbitrarily large, linguistically rich visual grounding datasets, breaking the bottleneck that has historically limited this field to small, human-annotated benchmarks with terse expressions.

Innovation 5: Establishing the "Amplification Not Creation" Principle for Test-Time Compute in Video Understanding

The paper's most significant negative result—and arguably its most important conceptual contribution—is the demonstration that additional training data (Ref-SAV) and test-time compute (more frames, larger models) cannot overcome fundamental capability gaps. The hardest Ref-VOS scenarios remain at roughly 50% J&F even with Sa2VA-8B trained on the full Ref-SAV dataset (Table 10, overall J&F of 50.0 for the trained model versus 41.3 for the zero-shot model). While this is more than 4× better than the next-best zero-shot approach (VISA at 11.8 J&F), it is far from solved.

This result establishes what might be called the "amplification not creation" principle: test-time strategies (more data, larger models, better training) can amplify existing capabilities—Sa2VA dramatically outperforms prior models on Ref-VOS because it inherits SAM-2's tracking and the MLLM's language understanding—but they cannot create capabilities that the base components lack. The residual failures likely stem from fundamental challenges (objects that are entirely occluded for extended periods, ambiguous language descriptions where even humans would struggle, extremely fine-grained distinctions between similar objects) that neither better training data nor larger models can address with the current architecture.

This principle has direct implications for how the field should allocate research effort. If additional data and scale were sufficient, the optimal strategy would be to build ever-larger Ref-VOS datasets and ever-larger models within the Sa2VA framework. But the plateau suggests that architectural innovations are needed—perhaps more robust memory mechanisms for long-term occlusion handling, or explicit reasoning about which parts of a long expression disambiguate which moment in the video. The paper's failure case analysis (Section 5, Figure 8) identifies specific scenarios—long videos with hard-to-distinguish referring examples—where Sa2VA fails, providing concrete targets for future architectural work.

The parallel to the broader LLM scaling literature is instructive. Just as scaling laws (Hoffmann et al., 2022) established predictable relationships between compute, data, and performance for language modeling, the Ref-SAV results suggest a saturation curve for test-time compute in video understanding: initial gains are dramatic (going from 11.8 to 41.3 J&F by adding a capable MLLM and SAM-2), subsequent gains from data addition are meaningful but diminishing (41.3 to 50.0 J&F from 37K additional training videos), and further gains likely require qualitatively different approaches rather than more of the same. This is not a failure of Sa2VA—it's an honest characterization of the problem's difficulty, and one that provides useful guidance for the field.

5. Experimental Analysis

Evaluation Methodology

• Dataset. Sa2VA is evaluated on more than 15 benchmarks spanning four task categories. For image referring segmentation: RefCOCO (17K test), RefCOCO+ (17K test), RefCOCOg (22K test), all drawn from the standard referring expression datasets (Kazemzadeh et al., 2014; Yu et al., 2016). For video referring segmentation: MeViS (2,006 videos, val_u split, Ding et al., 2023), Ref-DAVIS17 (30 videos, Khoreva et al., 2018), Ref-YTVOS (3,978 videos, Seo et al., 2020), ReVOS (1,042 videos, Yan et al., 2024), and the newly proposed Ref-SAV (1,147 videos, 1,945 expressions with both long and short subsets). For image chat: MME (Fu et al., 2023), MMBench (Liu et al., 2024), SEED-Bench (Li et al., 2024), AI2D (Kembhavi et al., 2016), MMStar (Chen et al., 2024), MMMU (Yue et al., 2024), and ScienceQA test (Lu et al., 2022). For video chat: Video-MME (Fu et al., 2024) and MMBench-Video (Fang et al., 2024). For grounded caption generation: GCG benchmark from GLaMM (Rasheed et al., 2024), evaluated on RefCOCOg.

• Base model(s). The primary experiments use InternVL2 (Chen et al., 2024) at four scales—1B, 4B, 8B, and 26B parameters—paired with SAM-2 (Ravi et al., 2024) as the frozen segmentation backbone. The authors choose InternVL2 as the default because it represents a strong open-source MLLM with established image and video QA capabilities, providing a clean baseline for measuring how much segmentation performance can be added without degrading language abilities. Additional experiments swap in InternVL2.5 (Chen et al., 2024), InternVL3 (Zhu et al., 2025), Qwen2.5-VL (Bai et al., 2025), and Qwen3-VL (Xu et al., 2025) to demonstrate framework flexibility. For the Ref-SAV data annotation pipeline, InternVL2-76B and Qwen2-72B are used as off-the-shelf annotators, not as Sa2VA base models.

• Metrics. Image referring segmentation uses cIoU (cumulative Intersection over Union), which computes IoU for each predicted mask against its ground truth and averages across all test samples. The paper states this metric "balances the large and small objects" (Section 4). Video referring segmentation uses J&F, the standard metric averaging region similarity J (Jaccard index, equivalent to IoU) and contour accuracy F (boundary F-measure). Image and video chat use benchmark-specific metrics: MME reports perception (A) and cognition (B) subscores with total C = A + B; MMBench and SEED-Bench report accuracy percentages; Video-MME and MMBench-Video report aggregate scores on their respective scales; MMMU and ScienceQA report accuracy. Grounded caption generation uses METEOR, a machine translation metric that evaluates n-gram overlap between generated and reference captions. Ref-SAV reporting separates J and F on long, short, and overall subsets.

• Baselines. The paper compares against three categories of prior work. Image/video chat MLLMs without segmentation: LLaVA-1.5-13B (Liu et al., 2024), Video-LLaVA-7B (Lin et al., 2024), LLaMA-VID-7B (Li et al., 2024), InternVL2-8B without modifications, and mPLUG-Owl3-8B (Ye et al., 2024). Image grounding MLLMs that lose chat capability: LISA-7B (Lai et al., 2024), PixelLM-7B (Ren et al., 2024), LaSagnA-7B (Wei et al., 2024), GLaMM-7B (Rasheed et al., 2024), OMG-LLaVA-7B (Zhang et al., 2024), LLaVA-G-7B (Zhang et al., 2024), GSVA-13B (Xia et al., 2024). Video referring segmentation specialists: VideoLISA-3.8B (Bai et al., 2024), VISA-13B (Yan et al., 2024), PG-Video-LLaVA (Munasinghe et al., 2023), VideoGLaMM-3.8B (Munasinghe et al., 2024), HyperSeg-3B (Wei et al., 2025), InstructSeg (Wei et al., 2024), and vision expert models including LAVT (Yang et al., 2022), ReferFormer (Wu et al., 2022), UniRef++ (Wu et al., 2023), EVF-SAM (Zhang et al., 2024), LMPM (Ding et al., 2023), and UniVS (Li et al., 2024). For Ref-SAV specifically, zero-shot baselines include UniRef++, UNINEXT (Yan et al., 2023), MeViS, and VISA.

• Generation budget / compute accounting. The paper does not frame compute in terms of a "budget" allocated across strategies—there is no best-of-N, beam search, or revision depth to sweep. Instead, compute is characterized as model scale (1B, 4B, 8B, 26B parameter variants), training data scale (~1.1M examples with ablations on subsets), and key frame count for video inference (1 to 5 frames, different sampling strategies). Inference time is reported in Table 19 as controlled measurements on H100 GPUs with fixed 5-frame + 1280-token inputs, ranging from 0.123s (Sa2VA-1B) to 0.463s (Sa2VA-26B). The paper notes that SAM-2 alone achieves 39.5 FPS on vision tasks, so the dominant cost is the autoregressive MLLM text generation.

• Cross-validation / statistical protocol. No cross-validation or statistical significance testing is reported. Results are single-point evaluations on standard test splits. The paper does not report confidence intervals, standard deviations, or multiple random seeds. For the compute-optimal strategy selection that would warrant cross-validation (as in the example paper's difficulty-bin analysis), Sa2VA does not perform such selection—it trains one set of weights on the full ~1.1M example mixture and evaluates once on each benchmark. The Ref-SAV validation benchmark creation uses a held-out subset of SA-V videos separate from the Ref-SAV training set, providing a clean train-test split for that dataset, but this is standard practice rather than a formal cross-validation protocol.

Main Quantitative Results

Sa2VA Against State-of-the-Art Grounding MLLMs: Segmentation Without Sacrificing Conversation

The headline result from Table 6 is that Sa2VA-8B simultaneously achieves competitive or state-of-the-art referring segmentation performance while maintaining image and video chat capabilities that prior grounding MLLMs completely lose.

Image referring segmentation. On RefCOCO, RefCOCO+, and RefCOCOg, Sa2VA-8B achieves cIoU scores of 81.9, 76.5, and 78.9 respectively (Table 6). Compared to the strongest prior grounding MLLM, GLaMM-7B (which scores 79.5, 72.6, 74.2), Sa2VA-8B improves by 2.4, 3.9, and 4.7 cIoU—a substantial margin, particularly on the harder RefCOCO+ and RefCOCOg benchmarks where compositional language understanding matters more. The 26B variant pushes these to 82.5, 78.8, and 79.7, demonstrating positive scaling with model size.

What makes this significant is not the absolute numbers (vision expert models like EVF-SAM achieve 82.4, 76.5, 78.2 on the same benchmarks, as shown in Table 16) but the fact that Sa2VA achieves them while retaining general multimodal understanding. LISA-7B also achieves reasonable segmentation (74.1, 62.4, 66.4) but at the cost of near-zero conversational ability.

Video referring segmentation. Sa2VA-8B achieves 46.9 J&F on MeViS, 75.2 J&F on Ref-DAVIS17, and 57.6 J&F on ReVOS (Table 6). Compared to the prior state-of-the-art VISA-13B (44.5, 70.4, 50.9), this represents improvements of 2.4, 4.8, and 6.7 J&F respectively. The ReVOS improvement of 6.7 J&F is particularly notable—ReVOS is the most recent and challenging of the existing Ref-VOS benchmarks, involving longer expressions and more diverse scenes, and Sa2VA's advantage widens on the harder task.

Compared to even more recent video MLLM approaches (Table 15), Sa2VA-8B's 46.9 J&F on MeViS exceeds VideoGLaMM-3.8B (45.2), VideoLISA-3.8B (44.4), and the GLaMM+SAM2 combined baseline (38.7). On ReVOS, Sa2VA-8B's 57.6 J&F exceeds InstructSeg (54.5) and HyperSeg-3B (55.7). On Ref-DAVIS17, Sa2VA-8B's 75.2 J&F exceeds all video MLLM competitors.

Image and video chat. This is where prior grounding MLLMs catastrophically fail and Sa2VA distinguishes itself. On MME, LISA-7B scores 1/1 (perception/cognition), GLaMM-7B scores 14/9, LaSagnA scores 0/0—all essentially non-functional for conversation. Sa2VA-8B scores 1651/578, which is actually higher than the InternVL2-8B baseline's reported MME scores (the InternVL2-8B row in Table 6 shows dashes, but the context of prior InternVL2 results indicates strong conversation performance). On MMBench, Sa2VA-8B achieves 82.4 (versus OMG-LLaVA-7B's 47.9 and GLaMM-7B's 36.8). On SEED-Bench, Sa2VA-8B achieves 75.5 (versus OMG-LLaVA-7B's 56.5). On video benchmarks, Sa2VA-8B achieves 52.1 on Video-MME and 1.34 on MMBench-Video, which the paper notes "does not achieve dominant results on video chat tasks, as it has both video segmentation and understanding capabilities, which current methods are not equipped to handle simultaneously." This is a fair characterization—Sa2VA's video chat scores are competitive but not category-leading, representing a trade-off between segmentation and conversation that the co-training strategy mitigates but does not eliminate.

Grounded caption generation. On the GCG benchmark evaluated on RefCOCOg, Sa2VA-4B achieves 17.3 METEOR (Table 18), outperforming OMG-LLaVA (15.3), Osprey (16.6), GLaMM (16.2), GRIT (15.2), and Kosmos-2 (14.1). This task requires the model to jointly produce masks and aligned descriptions—Sa2VA's advantage suggests the unified framework handles the joint mask-text generation better than prior specialized approaches.

Detailed Image Benchmark Results: Evidence Against Catastrophic Forgetting

Table 7 provides the most comprehensive evidence for the paper's central claim about maintaining chat capabilities. On MME, Sa2VA-4B achieves 1553/540 and Sa2VA-8B achieves 1651/578, compared to LLaVA-1.5-13B's 1531(+) (which is a chat-only model with no segmentation capability). The fact that Sa2VA-8B exceeds the chat-only baseline on MME despite also handling segmentation is the strongest single piece of evidence that the co-training strategy resolves the catastrophic forgetting problem.

On MMBench, Sa2VA-8B scores 82.4 (versus LLaVA-1.5-13B's 68.8). On SEED-Bench, Sa2VA-8B scores 75.5 (versus LLaVA-1.5-13B's 70.1—the paper reports LLaVA's MME as "1531(+)" where the (+) notation likely indicates the total, while other MLLMs report separate perception/cognition, making direct comparison slightly ambiguous). On AI2D, Sa2VA-8B achieves 82.1; on MMStar, 60.3; on MMMU, 44.7; on ScienceQA test, 96.8. These are strong, general-purpose multimodal understanding scores that no prior grounding MLLM comes close to matching. For comparison, GLaMM-7B scores 36.8 on MMBench and 28.2 on AI2D; OMG-LLaVA-7B scores 47.9 on MMBench and 42.9 on AI2D.

The image segmentation scores for these same models: Sa2VA-4B achieves 80.4, 74.3, 76.7 on RefCOCO/+/g; Sa2VA-8B achieves 81.9, 76.5, 78.9. These are higher than all prior grounding MLLMs (GLaMM-7B at 79.5, 72.6, 74.2; OMG-LLaVA-7B at 78.0, 69.1, 72.9). The simultaneous superiority in both segmentation and chat is the paper's most important experimental result.

Scaling Behavior: Bigger Base MLLMs Yield Monotonic Improvements

Table 8 reports results when upgrading from InternVL2 to InternVL2.5 as the base MLLM, at five scales: 1B, 4B, 8B, and 26B. The pattern is consistently monotonic for segmentation: on RefCOCO, scores progress from 79.6 (1B) to 82.4 (4B) to 82.6 (8B) to 82.9 (26B). On MeViS, from 53.4 to 55.9 to 58.9 to 61.8 J&F. On Ref-DAVIS17, from 69.5 to 73.7 to 75.9 to 78.6 J&F.

Chat benchmarks also scale positively: MME total increases from 1938 (1B: 1504+434) to 2301 (4B: 1691+610) to 2300 (8B: 1690+610) to 2351 (26B: 1698+653). MMBench increases from 71.9 to 81.8 to 84.4 to 85.8. AI2D increases from 69.2 to 81.4 to 82.7 to 85.7.

The InternVL2.5 variants consistently outperform their InternVL2.0 counterparts at the same scale. For instance, InternVL2.5-8B achieves 82.6 cIoU on RefCOCO, 78.0 on RefCOCO+, 80.3 on RefCOCOg, compared to InternVL2.0-8B's 81.9, 76.5, 78.9—improvements of 0.7, 1.5, and 1.4 cIoU respectively. On video segmentation, InternVL2.5-8B achieves 58.9 J&F on MeViS (versus 57.0), 72.3 on Ref-YTVOS (versus 70.7), and 75.9 on Ref-DAVIS17 (versus 75.2).

This scaling behavior is important because it demonstrates that Sa2VA's architecture does not bottleneck the MLLM's representational capacity—as the base model improves, Sa2VA's performance improves correspondingly, suggesting the framework genuinely inherits advances in both the MLLM and segmentation foundation model communities.

Ref-SAV Benchmark Results: Diagnostic Evidence of Failure Modes

Table 10 presents the most diagnostically revealing results in the paper. On the long-expression subset of Ref-SAV (1,694 expressions with average length ~83.6 words), prior state-of-the-art models achieve:

• UNINEXT (zero-shot): 10.0 J&F
• UniRef++ (zero-shot): 12.5 J&F
• MeViS model (zero-shot): 11.3 J&F
• VISA (zero-shot): 14.1 J&F

Sa2VA-8B zero-shot achieves 49.3 J&F on the same long-expression set—a 3.5× improvement over VISA. With full Ref-SAV training, Sa2VA-8B reaches 58.7 J&F on long expressions.

On the short-expression subset (251 manually annotated expressions), the gap narrows but remains large: VISA zero-shot achieves 9.2 J&F, Sa2VA-8B zero-shot achieves 33.3 J&F, and Sa2VA-8B with Ref-SAV training achieves 41.2 J&F.

The overall J&F (weighted by expression count) for Sa2VA-8B zero-shot is 41.3, and with Ref-SAV training it rises to 50.0. This 8.7 J&F improvement from adding 37K training videos is meaningful but modest relative to the 29.5 J&F gap between Sa2VA zero-shot and the next-best prior model (VISA at 11.8). Most of Sa2VA's advantage comes from architecture (MLLM + SAM-2), not from the additional training data.

A counterintuitive pattern emerges: Sa2VA performs better on long expressions than on short ones (49.3 vs. 33.3 J&F zero-shot; 58.7 vs. 41.2 with training). The paper does not directly analyze this inversion, but it likely reflects the precision-disambiguation trade-off: long expressions provide multiple grounding cues (color, context, motion) that help the MLLM identify the correct object, while short expressions like "the goat" are ambiguous when multiple similar objects are present. This interpretation aligns with the paper's description of Ref-SAV's long expressions as compositional descriptions spanning object appearance, scene context, and temporal dynamics—each component adds discriminative information.

Comparison with Specific-Task Fine-Tuned Models

Table 11 addresses a practical question: can Sa2VA be further improved by fine-tuning on individual segmentation datasets after the general co-training? The answer is yes, but modestly: fine-tuned Sa2VA-8B achieves 82.3, 77.3, and 79.3 cIoU on RefCOCO/RefCOCO+/RefCOCOg, compared to 81.9, 76.5, 78.9 for the co-trained model—improvements of only 0.4, 0.8, and 0.4 cIoU. The paper argues that "fine-tuning on a single dataset does not yield much benefit" and advocates "for more general models like Sa2VA for their versatility and convenience." This is a significant practical claim: the one-shot instruction-tuned model is nearly Pareto-optimal relative to per-dataset fine-tuning, making task-specific adaptation largely unnecessary.

However, this comparison has a limitation: the fine-tuned models in Table 11 are Sa2VA models fine-tuned on each dataset separately (presumably initialized from the co-trained checkpoint), while the other models listed (LAVT, GLaMM, OMG-LLaVA) were trained with different procedures. A fairer comparison would show whether GLaMM fine-tuned on RefCOCO also sees a small improvement, to determine whether the diminishing returns from fine-tuning are specific to Sa2VA or general to the task.

Results with Different Base MLLMs: Framework Flexibility Demonstrated

Table 14 provides the most comprehensive evidence for Sa2VA's claimed framework flexibility. Seven base MLLM variants are evaluated across 12 benchmarks:

• Sa2VA-8B (InternVL2.5-8B): the default, achieving 81.9 cIoU on RefCOCO, 57.0 J&F on MeViS, 1651/578 on MME, 82.4 on MMBench.
• Sa2VA-InternVL3-2B: despite being only 2B parameters, achieves 81.4 on RefCOCO and 53.9 on MeViS, with 1631/559 on MME.
• Sa2VA-InternVL3-8B: 83.3 on RefCOCO, 56.4 on MeViS, 1743/633 on MME.
• Sa2VA-InternVL3-14B: the strongest overall, achieving 83.6 on RefCOCO, 83.6 on RefCOCOg, 59.2 on MeViS, 76.6 on Ref-DAVIS17, 1746/724 on MME, 84.3 on MMBench.
• Sa2VA-Qwen2.5VL-3B: 79.6 on RefCOCO, 51.6 on MeViS, 1533/572 on MME, 78.4 on MMBench.
• Sa2VA-Qwen2.5VL-7B: 82.4 on RefCOCO, 56.4 on MeViS, 1552/676 on MME, 84.5 on MMBench.
• Sa2VA-Qwen3VL-4B: 81.7 on RefCOCO, 57.1 on MeViS, 1660/655 on MME, 86.3 on MMBench (the highest MMBench score among all variants).

The results reveal that different base models have different strengths. InternVL3 variants excel at segmentation (14B achieves 83.6 on both RefCOCO and RefCOCOg), while Qwen3VL-4B excels at multimodal understanding benchmarks (86.3 MMBench, 66.3 MMStar, 91.6 ScienceQA). The Qwen2.5VL-7B variant achieves the highest Ref-DAVIS17 score (79.4 J&F). This heterogeneity validates the paper's decision to keep the framework modular—users can select the base MLLM that best matches their deployment priorities (segmentation quality vs. conversational ability vs. parameter efficiency).

Effect of the Ref-SAV Training Data on a Representative Model

Table 17 isolates the impact of the Ref-SAV training set by evaluating UniRef++, a vision expert model, with and without fine-tuning on Ref-SAV. In the zero-shot setting, UniRef++ achieves 10.5 overall J&F on the Ref-SAV validation benchmark. After fine-tuning on the Ref-SAV training set, overall J&F rises to 14.6—a 39% relative improvement. The gains are consistent across both the long-expression subset (12.5 to 17.2 J&F, a 38% relative improvement) and the short-expression subset (8.6 to 12.0 J&F, a 40% relative improvement).

This result serves two purposes. First, it validates that the automatically generated annotations contain genuine training signal—a model that has no MLLM component, no language understanding beyond its original training, still benefits substantially from the Ref-SAV data. Second, it establishes that Ref-SAV's difficulty is not simply a matter of the data being out-of-distribution for existing models (which would mean fine-tuning helps regardless of data quality); the consistent improvement across both expression types, achieved with automatically generated annotations, suggests the pipeline produces meaningful referring expressions.

Ablation Studies and Robustness Checks

Co-training data mixture ablation (Table 9): Removing image QA data from training causes MME to drop from 1381/405 to 1298/359 (perception drops by 83, cognition by 46) and MMBench to drop from 68.3 to 63.4. Image segmentation scores remain stable (RefCOCO actually increases slightly from 77.4 to 78.0). This demonstrates that image QA data is necessary specifically for maintaining conversational ability, not for segmentation. Removing image segmentation data causes segmentation scores to collapse (RefCOCO drops from 77.4 to 20.2, MeViS drops from 50.8 to 38.0) while chat scores remain stable (MME stays at 1393/408, MMBench increases slightly to 70.1). This confirms that segmentation and conversation capabilities are independently maintained through their respective data sources. Removing video QA data causes MMBench-Video to drop from 1.07 to 0.71 (a 34% relative decrease) and Video-MME to drop from 39.9 to 41.3, with minimal impact on other metrics. Removing video segmentation data causes MeViS to drop from 50.8 to 44.4 J&F and Ref-DAVIS17 to drop from 72.3 to 69.0 J&F. The clean separability of these effects is the paper's strongest evidence that the four data types maintain independent capabilities without destructive interference.

Segmentation token design ablation (Table 12): Comparing single [SEG] token (the default), repetitively generated [SEG] tokens (same token output N times for N frames), and unique frame-specific tokens ([SEG 1], [SEG 2], etc.), the single token strategy achieves the best video segmentation: 50.8 J&F on MeViS(val_u) versus 49.6 for repetitive and 46.3 for unique. The image segmentation scores are nearly identical across all three designs (RefCOCO: 77.4, 77.3, 77.6), indicating the token design only affects temporal knowledge sharing. The unique token design's 4.5 J&F deficit on MeViS relative to the single token is the paper's clearest evidence that knowledge transfer from image segmentation (where a single [SEG] means "the object being referred to") is crucial for video performance.

Additional data scaling ablation (Table 13): Adding 3M image-QA samples from Infinity-MM (Gu et al., 2024) to the baseline 1.2M training examples improves MMBench by 2.1 points (from 68.3 to 70.4) but slightly degrades MME cognition (405 to 346, a drop of 59) and leaves segmentation essentially unchanged (RefCOCO: 77.4 to 77.1). Adding the 37K Ref-SAV training videos improves MeViS by 1.7 J&F (from 50.8 to 52.5) with negligible impact on image segmentation (RefCOCO: 77.4 to 77.2) and image chat (MMBench: 68.3 to 68.2). The near-zero cross-task interference when adding modality-specific data is a robustness check on the co-training strategy: the model absorbs new information in the targeted domain without disrupting other capabilities.

Base MLLM scaling (Table 8): Upgrading from InternVL2.0 to InternVL2.5 produces consistent, monotonic improvements across all benchmarks at all model scales, with no observed regressions or diminishing returns within the tested range. The InternVL2.5-26B variant achieves the paper's strongest results: 82.9 on RefCOCO, 79.3 on RefCOCO+, 81.2 on RefCOCOg, 61.8 on MeViS, 75.1 on Ref-YTVOS, 78.6 on Ref-DAVIS17, 1698/653 on MME, 85.8 on MMBench, 78.3 on SEED-Bench, 85.7 on AI2D, 67.0 on MMStar, and 98.4 on ScienceQA test.

Key frame sampling strategy ablation (Table 20): Using Sa2VA-8B on MeViS(val_u), performance varies substantially with key frame selection: 55.1 J&F for the first frame only, 58.7 for first 3 frames, 59.5 for first 4 frames, 58.9 for first 5 frames (the default strategy), and 62.9 for uniform 5-frame sampling (frames spread across the entire video). The 4.0 J&F gap between the default "first 5" strategy and the uniform strategy, with identical inference time and token count, reveals that the default strategy is suboptimal—objects appearing later in videos or scenes with significant temporal variation benefit from temporal coverage in the key frames. The paper treats this as a known limitation and leaves optimization to future work.

Fine-tuning vs. co-training comparison (Table 11): Fine-tuning the co-trained Sa2VA-8B separately on each referring segmentation dataset yields only 0.4–0.8 cIoU improvements (RefCOCO: 81.9 to 82.3; RefCOCO+: 76.5 to 77.3; RefCOCOg: 78.9 to 79.3). This near-saturation suggests the co-trained model already extracts most of the available signal from these datasets, and dataset-specific fine-tuning provides minimal marginal benefit.

Ref-SAV training data benefit on expert model (Table 17): Fine-tuning UniRef++ on the Ref-SAV training set improves overall J&F from 10.5 to 14.6 (39% relative), with consistent gains on both long (12.5 to 17.2) and short (8.6 to 12.0) expressions. This ablation demonstrates that Ref-SAV's training signal is not specific to Sa2VA's architecture.

Critical Assessment

How Strongly Do the Experiments Support the Paper's Primary Claims?

Claim: Sa2VA is the first unified model supporting both image and video referring segmentation AND multimodal conversation.

What the experiments demonstrate: Tables 6 and 7 show that Sa2VA-8B achieves competitive or superior performance on image referring segmentation (81.9/76.5/78.9 cIoU on RefCOCO/+/g), video referring segmentation (46.9/75.2/57.6 J&F on MeViS/Ref-DAVIS17/ReVOS), image chat (1651/578 on MME, 82.4 on MMBench, 75.5 on SEED-Bench), and video chat (52.1 on Video-MME, 1.34 on MMBench-Video) simultaneously. No prior model reports results on all of these benchmarks with a single set of weights.

Limitations: The "first unified model" claim is technically accurate based on the comparison table (Table 1), but the evaluation is restricted to a specific set of benchmarks that the paper itself curated. Does Sa2VA support arbitrary video referring segmentation tasks, or only those that align with the Ref-VOS paradigm (language-driven, single-object, with ground-truth masks for evaluation)? The paper does not test on video instance segmentation, panoptic segmentation, interactive segmentation with visual prompts, or open-vocabulary segmentation in video—all tasks that a "comprehensive unified model for dense grounded understanding" might be expected to handle. The model's scope is broader than any prior work, but still narrower than the framing suggests.

Missing experiments: No evaluation on out-of-distribution referring expressions (e.g., adversarial descriptions, negations, counterfactuals), no testing on domains outside natural images/videos (medical, satellite, document), no evaluation of the model's calibration or confidence estimation for segmentation outputs.

Claim: Sa2VA resolves the catastrophic forgetting problem that causes prior grounding MLLMs to lose conversation ability.

What the experiments demonstrate: The comparison in Tables 6 and 7 is stark: LISA-7B scores 1/1 on MME, GLaMM-7B scores 14/9, LaSagnA scores 0/0. Sa2VA-8B scores 1651/578, exceeding the InternVL2-8B baseline (which is a chat-only model) on MME and achieving competitive scores across all chat benchmarks. The ablation in Table 9 shows that removing QA data from training causes selective degradation of chat capabilities without improving segmentation, suggesting the co-training strategy genuinely maintains capabilities rather than finding a different point on a trade-off curve.

Limitations: The chat evaluations are on standard benchmarks designed for chat-only MLLMs, not for systems that simultaneously produce segmentation masks. A model that generates [SEG] tokens in chat contexts (e.g., unnecessarily segmenting objects when asked a pure QA question) might score well on QA accuracy while exhibiting undesirable behavior. The paper does not analyze whether Sa2VA appropriately suppresses mask generation in chat-only contexts or whether it sometimes produces spurious segmentations. Additionally, the comparison is between Sa2VA (co-trained on 1.1M examples including both segmentation and QA data) and prior models that were trained primarily on segmentation data. A fairer comparison would test whether LISA or GLaMM could recover conversation ability through the same co-training strategy—the paper leaves open the possibility that the co-training recipe, not the Sa2VA architecture, is the critical factor.

Missing experiments: No evaluation of conversation quality beyond benchmark scores (e.g., human evaluation of fluency, relevance, hallucination rate). No analysis of whether Sa2VA sometimes confuses segmentation and conversation contexts. No controlled experiment where the same base MLLM is trained with the same data mixture but using a different architecture (e.g., feeding SAM-2 features into the LLM), to isolate the architectural contribution from the data mixture contribution.

Claim: Sa2VA's decoupled design enables flexible integration with any modern MLLM.

What the experiments demonstrate: Table 14 evaluates seven base MLLM variants (InternVL2.5, InternVL3 at three scales, Qwen2.5-VL at two scales, Qwen3-VL) and Table 8 evaluates InternVL2.5 at four scales. All variants achieve strong performance across segmentation and chat benchmarks, with consistent positive scaling as the base model improves.

Limitations: All tested models are from two model families (InternVL and Qwen-VL), which share architectural similarities (both use ViT-based visual encoders with LLM backbones). The paper does not test with radically different architectures (e.g., models using CNN-based visual encoders, models without explicit visual projection layers, models using different modality fusion strategies). The "any modern MLLM" claim is supported for the specific families tested but extrapolation to arbitrary architectures is unverified. Additionally, the paper does not report how much engineering effort was required to adapt Sa2VA to each new base model—the modularity claim would be stronger if it included the number of lines of code changed or hours of adaptation required.

Missing experiments: Testing with models from other families (LLaVA-NeXT, CogVLM, Fuyu, etc.), testing with models that use different image tokenization strategies (e.g., dynamic resolution, any-resolution), testing with video-native MLLMs that have fundamentally different temporal processing pipelines.

Claim: The Ref-SAV benchmark is more challenging than prior Ref-VOS benchmarks and reveals failure modes that existing evaluations miss.

What the experiments demonstrate: Prior state-of-the-art models achieve only 7.6–11.8 J&F on Ref-SAV zero-shot (Table 10), versus much higher scores on existing benchmarks (e.g., VISA achieves 44.5 on MeViS, 70.4 on Ref-DAVIS17). This 4–6× performance gap between existing benchmarks and Ref-SAV validates that Ref-SAV captures difficulty dimensions absent from prior evaluations.

Limitations: The benchmark was created from SA-V videos, which are themselves drawn from a specific distribution (the videos used to train SAM-2). The automatic annotation pipeline uses InternVL2-76B and Qwen2-72B, which may introduce systematic biases in the language expressions (e.g., a preference for certain descriptive patterns, underrepresentation of certain types of referring expressions). The human validation is described as "carefully filtered" but the paper does not report inter-annotator agreement, the exact filtering criteria, or how many expressions were rejected. The benchmark may encode the biases of its annotation pipeline rather than representing a natural distribution of referring expressions.

Missing experiments: Analysis of what specific failure modes cause the performance gap (is it occlusion, camera motion, expression length, vocabulary complexity, or some interaction?), correlation between individual video characteristics and model performance, comparison of Ref-SAV difficulty against a human performance baseline, evaluation of whether Ref-SAV-trained models transfer to other Ref-VOS benchmarks (positive transfer would validate the benchmark's relevance; negative transfer would suggest it's a specialized distribution).

Additional Weaknesses Not Tied to Specific Claims

No statistical significance reporting. All results are single-point evaluations with no confidence intervals, standard deviations, or multiple random seeds. The differences between methods, particularly in the fine-tuning ablation (Table 11: 0.4–0.8 cIoU improvements), may not be statistically significant. The 500-question MATH test set in the example paper analogously would report cross-validation folds; Sa2VA has no comparable protocol.

The Ref-SAV training set benefit is modest relative to the architectural contribution. Sa2VA-8B zero-shot achieves 41.3 overall J&F on Ref-SAV; adding Ref-SAV training brings it to 50.0 (Table 10). The 8.7 J&F improvement is real but represents only a 21% relative gain, compared to the 250% relative gain from switching from VISA to Sa2VA zero-shot (11.8 to 41.3). This suggests the architecture contributes substantially more than the dataset, which complicates the paper's presentation of Ref-SAV as a major contribution.

No ablation on the LoRA rank. The paper uses LoRA with rank r=256, which is unusually high (common values are 8–64). There is no ablation comparing different LoRA ranks, leaving open the question of whether the high rank is necessary for segmentation performance or whether a lower rank would suffice and improve training efficiency.

No ablation on the freezing strategy. The paper freezes SAM-2's encoder and memory module while fine-tuning its decoder. There is no experiment testing whether fine-tuning the encoder or memory would improve or degrade performance, making the freezing choices appear as design decisions rather than empirically validated optimizations.

No comparison to simpler baselines. Could the same functionality be achieved by training an MLLM to output bounding boxes and using SAM-2 to segment within those boxes? This two-stage approach would be simpler and might achieve comparable performance. The paper does not compare against such a baseline.

Limited video chat evaluation. While Sa2VA's video chat performance is reasonable (52.1 on Video-MME, 1.34 on MMBench-Video), it is not state-of-the-art compared to video-only MLLMs (mPLUG-Owl3-8B achieves 53.5 and 1.35 respectively). The paper acknowledges this ("Sa2VA does not achieve dominant results on video chat tasks") but does not analyze whether this represents an inherent trade-off or could be addressed through data scaling.

6. Limitations and Trade-offs

The Ref-SAV Benchmark Is Built on a Single Video Distribution and May Encode Systematic Annotation Biases

The assumption or constraint. The Ref-SAV benchmark is constructed entirely from SA-V videos—the dataset used to train SAM-2. The annotation pipeline relies on InternVL2-76B and Qwen2-72B as annotators, and employs a consistency-checking mechanism that discards descriptions from the cropped-object and full-image views when they conflict. This means the benchmark inherits both the visual distribution of SA-V and the linguistic distribution of the annotator models. The paper acknowledges the annotation pipeline's automated nature but does not systematically analyze what biases this introduces.

The consequence. Three failure modes arise. First, distributional overfitting: a model trained on Ref-SAV (as Sa2VA is) may perform well on Ref-SAV's evaluation set not because it has learned generalizable referring video understanding, but because it has adapted to the specific descriptive patterns, vocabulary choices, and video characteristics of the SA-V domain. The paper provides no cross-domain evaluation—for example, testing Ref-SAV-trained models on Ref-YTVOS or MeViS to check for positive transfer—so the generality of Ref-SAV-trained capabilities is unknown.

Second, linguistic homogenization: the consistency-checking step filters out cases where the cropped and full-image descriptions conflict. While this removes error-prone annotations, it may also systematically exclude objects that are genuinely ambiguous or context-dependent—precisely the kinds of cases that would stress-test a model's reasoning capabilities. A referring expression like "the person in the background" might generate conflicting cropped vs. full-image descriptions because the object-level description (from the cropped view) lacks scene context, while the full-image description correctly identifies the person as background. If such cases are filtered out, Ref-SAV's long-expression set may be biased toward unambiguously identifiable objects, making the benchmark easier in a specific way that does not generalize.

Third, model-specific annotation artifacts: InternVL2-76B and Qwen2-72B are from the same model ecosystem (InternVL, Qwen) that Sa2VA uses as base MLLMs. If these annotators share systematic tendencies—preferences for certain spatial relationship phrasings, color descriptions, or motion characterizations—then Ref-SAV's expressions may align better with Sa2VA's language understanding than with a model from a different family. This would inflate Sa2VA's apparent performance relative to what a truly model-agnostic benchmark would show.

What evidence exists in the paper. Table 17 provides indirect evidence by showing that UniRef++, a vision expert model with no MLLM architecture, benefits from Ref-SAV fine-tuning (10.5 to 14.6 J&F, a 39% relative improvement). This suggests the annotations contain genuine signal that transfers beyond the annotator model family. However, this is a single data point and does not address whether Ref-SAV's difficulty ranking of models generalizes. The paper does not evaluate Ref-SAV-trained Sa2VA on existing benchmarks to test for positive or negative transfer. The human filtering on the evaluation set is described only as "carefully filtered by human annotators" with no inter-annotator agreement metrics, filtering criteria, or rejection rates provided.

Mitigation status. The paper acknowledges none of these concerns explicitly. The future work discussion (Section 5) focuses on architectural improvements for handling "long video with hard, distinguished referring examples" and balancing VQA with segmentation, not on dataset bias or cross-domain generalization. The annotation pipeline's quality control (cropped vs. full-image consistency) is presented as a strength, and its potential to introduce systematic biases is not discussed.

---

The Difficulty Estimation and Strategy Selection Framework That Enabled the 4× Efficiency Gains in Prior Work Is Entirely Absent—All Problems Receive Identical Inference-Time Treatment

The assumption or constraint. Sa2VA applies the same inference procedure—same number of key frames, same [SEG] token mechanism, same SAM-2 propagation—to every video regardless of its length, complexity, occlusion level, or language description ambiguity. There is no difficulty estimation, no adaptive allocation of inference compute, and no mechanism for the model to "try harder" on challenging cases. The paper's ablation on key frame sampling strategies (Table 20) shows that different strategies produce different results (first 5 frames: 58.9 J&F; uniform 5 frames: 62.9 J&F), but the choice is fixed globally rather than adapted per-video.

The consequence. This is a missed opportunity for efficiency and accuracy that the paper's own analysis implicitly identifies. Table 10 shows a 17.4 J&F gap between Sa2VA-8B's long-expression performance (49.3 J&F zero-shot) and short-expression performance (33.3 J&F zero-shot) on the same videos. If the model could detect when a referring expression is ambiguous (short, lacking discriminative cues) and allocate additional computation—perhaps by processing more key frames, using a more sophisticated temporal sampling strategy, or generating multiple [SEG] hypotheses and selecting among them—it might close this gap without increasing average inference cost.

More fundamentally, the uniform treatment means Sa2VA cannot exploit the phenomenon that makes the example paper's compute-optimal scaling so powerful: the observation that easy problems need less compute and hard problems need more. A video where the referred object is uniquely identifiable from a single frame and tracked trivially by SAM-2's pre-trained memory consumes the same 0.2 seconds (for Sa2VA-8B) as a video with heavy occlusion, large camera motion, and an ambiguous long-text description. In high-throughput deployment scenarios (surveillance, batch video processing), this uniform allocation wastes compute on easy cases while potentially under-serving hard ones.

What evidence exists in the paper. Table 20 directly demonstrates that key frame sampling strategy matters: a 4 J&F gap between the default and uniform strategies at identical inference cost. Table 10 shows the substantial performance gap between long and short expressions. The paper does not report performance as a function of video length, occlusion level, or camera motion magnitude, so the relationship between video characteristics and accuracy is unknown. The inference time measurements in Table 19 are for a fixed configuration (5 frames, 1280 tokens) and do not explore variable-cost inference strategies.

Mitigation status. The paper explicitly states it defers this: "we would like to keep Sa2VA as simple as possible and leave designing advanced techniques on sampling as future work" (Section 4.4). This is a reasonable scoping decision for an initial system paper, but it means that Sa2VA's reported performance numbers reflect a fixed-strategy upper bound—better results are likely achievable through adaptive inference, but the paper provides no guidance on what those strategies should be or how much improvement to expect.

---

Short Referring Expressions Are Handled Worse Than Long Ones, Inverting the Expected Difficulty Ordering and Limiting Practical Utility for Concise User Queries

The assumption or constraint. Sa2VA's architecture and training data emphasize long, detailed referring expressions. The Ref-SAV training set consists overwhelmingly of 83.6-word average expressions generated by the three-stage annotation pipeline. The existing Ref-VOS training datasets (Ref-YTVOS, MeViS, ReVOS) contain short expressions (7–10 words average) but contribute only 5.8K training examples, dwarfed by Ref-SAV's 37K. The model is thus trained primarily on the task of parsing compositional, multi-cue descriptions, not on resolving brief, potentially ambiguous references.

The consequence. On Ref-SAV's evaluation benchmark, Sa2VA-8B achieves 49.3 J&F on long expressions but only 33.3 J&F on short expressions—a 16 J&F gap in the wrong direction (Table 10). This is not merely a benchmark artifact; it reflects a genuine capability limitation. Real users interacting with a visual assistant are more likely to issue concise queries ("segment the goat," "where is the red car?") than 85-word multi-sentence descriptions. A model that performs well on verbose, disambiguated descriptions but poorly on terse, natural queries is solving a distribution that does not match typical deployment scenarios.

Why does this inversion occur? The paper does not analyze the mechanism, but the likely explanation is that long expressions provide multiple independent grounding cues. An 85-word description might mention the object's color, category, relationship to other objects, location in the frame, motion pattern, and temporal context. If the MLLM fails to parse one cue (e.g., misidentifies the color), it can still localize the object from other cues (category, motion, context). A 7-word expression like "the goat" provides only the category—if multiple goats are present, the model must rely on implicit context (e.g., saliency, recency of mention) that it was not explicitly trained to use. Sa2VA's strong long-expression performance and weak short-expression performance suggest the model is a good integrator of multiple cues but a poor resolver of ambiguity—it can combine information but cannot make decisions under uncertainty.

What evidence exists in the paper. Table 10 provides the direct comparison, with the gap clearly visible in both zero-shot (49.3 vs. 33.3) and trained (58.7 vs. 41.2) settings. Table 4 shows the extreme imbalance in training expression lengths: Ref-SAV averages 83.6 words versus 7–10 words for all prior datasets. The Ref-SAV benchmark's short-expression subset is only 251 expressions (versus 1,694 long), so the evaluation is also imbalanced—the short-expression results have higher variance and less statistical reliability.

Mitigation status. The paper acknowledges the short-expression subset exists and reports results on it, but does not treat the inverted difficulty as a finding requiring explanation or mitigation. The imbalance in training data and the architectural bias toward compositional cue integration are not discussed. No experiment tests whether balancing the training mixture (e.g., by generating both short and long expressions for the same objects) would close the gap.

---

The Computational Cost of Online Video Processing Is Dominated by the Autoregressive MLLM, Making Latency-Sensitive Deployment Impractical for Many Real-World Applications

The assumption or constraint. Sa2VA's inference pipeline requires the MLLM to generate text tokens sequentially before any mask can be produced—the [SEG] token's hidden state is only available after the LLM has autoregressively decoded the entire response up to and including that token. SAM-2's mask decoding runs in a single forward pass per frame and is "relatively lightweight at 220M parameters... achieving 39.5 FPS" (Section 4.4). But the end-to-end latency is gated by the MLLM's text generation speed, which is inherently sequential and scales with both model size and output length.

For referring video object segmentation, this means the system cannot begin mask generation until the MLLM has finished producing its text response. For interactive applications—a user watching a video, clicking on an object, and asking "what is this?" while expecting immediate segmentation and description—the 0.2–0.5 second latency reported in Table 19 represents only the processing time for a fixed 5-frame, fixed-prompt configuration. Real interactive use would involve variable-length prompts, potential multi-turn clarification, and user expectation of sub-100ms response times.

The consequence. Three deployment scenarios are affected:

1. Real-time video applications (robot navigation, augmented reality, live video editing): Sa2VA's latency, while impressive for an MLLM-based system, is likely too high for closed-loop control where decisions must be made within tens of milliseconds. The autoregressive bottleneck cannot be parallelized away—each token depends on all previous tokens—so latency scales linearly with output length regardless of available compute.

2. High-throughput batch processing (processing thousands of videos, generating training data): The 0.2–0.5 seconds per video for Sa2VA-8B is competitive, but the cost is heavily dependent on the prompt length and required output verbosity. A simple "segment the car" with a single [SEG] token costs less than a full grounded caption generation task that produces interleaved text and masks. The paper provides no throughput benchmarks or cost scaling analysis.

3. On-device deployment: The smallest Sa2VA variant (1B parameters) requires 0.123 seconds per sample on an H100 GPU—a datacenter-class accelerator. On mobile or edge hardware, inference times would be substantially higher, potentially rendering the model unusable for real-time applications.

What evidence exists in the paper. Table 19 provides inference time measurements for Sa2VA variants on H100 GPUs under a fixed configuration (5 frames, 1280 tokens, fixed prompt). The times range from 0.123s (1B) to 0.463s (26B). The paper notes that "inference speed is largely dependent on optimization" and attributes the 4B vs. 8B discrepancy (0.282s vs. 0.201s) to different base LLMs (Qwen vs. InternLM). No latency breakdown between MLLM generation and SAM-2 decoding is provided, though the paper states "the majority of the model's computational cost stems from the auto-regressive MLLM component."

Mitigation status. The paper identifies this as partially addressed by architectural design ("the decoupled design... without incurring additional computation costs") but does not propose solutions for the fundamental autoregressive bottleneck. The inference time measurements are provided for transparency but not analyzed for deployment implications. Potential mitigations—speculative decoding, early exit for simple queries, caching of visual features for repeated queries on the same video—are not discussed.

---

The Freezing Strategy for SAM-2's Components Is Empirically Unvalidated, and the Consequences of Violating It Are Unknown

The assumption or constraint. Table 5 specifies that SAM-2's encoder and memory module are frozen during Sa2VA training, while the decoder is fully fine-tuned. The paper provides three conceptual justifications for this choice (Section 3.2): computational simplicity, preventing feature misalignment between MLLM and SAM-2 token spaces, and enabling independent evolution of the two components. However, there is no ablation experiment testing whether this specific freezing configuration is optimal—or even whether freezing is necessary at all.

The consequence. The freezing strategy is presented as a design principle rather than an empirically validated choice. This matters for two reasons:

First, it may be overly conservative. SAM-2 was trained on 1 billion masks for promptable segmentation and tracking, but those prompts were visual (points, boxes, masks), not language-derived embeddings. Fine-tuning SAM-2's encoder to better handle language-conditioned features—or fine-tuning the memory module to better retain object identity when prompted by an MLLM-generated [SEG] token rather than a visual prompt—might improve performance, particularly on challenging cases like heavy occlusion or long-term tracking. The paper never tests this, so the performance ceiling with a fully fine-tuned SAM-2 is unknown.

Second, it limits the framework's applicability to future SAM variants. The paper claims Sa2VA is a "flexible framework" that can "evolve with increasingly powerful" models. But if a future SAM-3 introduces architectural changes that make the decoder interface incompatible with Sa2VA's current [SEG] token projection, the freezing strategy provides no guidance on how to adapt. More importantly, if fine-tuning SAM-2's components would unlock substantial performance gains, then the freezing strategy is leaving capability on the table in exchange for flexibility that may not be necessary.

What evidence exists in the paper. None directly. The paper provides no ablation that varies which SAM-2 components are frozen versus trained. The ablation studies cover data mixture (Table 9), [SEG] token design (Table 12), key frame sampling (Table 20), base MLLM choice (Tables 8, 14), and additional data (Table 13), but never touch the freezing strategy. The justifications in Section 3.2 are architectural arguments, not empirical findings.

Mitigation status. Not addressed. The paper treats the freezing strategy as a fixed design decision and does not list it as a direction for future investigation. This is a particularly notable omission given that the decoder fine-tuning works well—if relaxing one freezing constraint (decoder) helps, testing whether relaxing others (encoder, memory) helps further would be a natural ablation.

7. Implications and Future Directions

How This Work Changes the Landscape

Sa2VA represents a reframing of the relationship between perception models and multimodal LLMs, not a paradigm shift in either individually. The fundamental contribution is conceptual: the demonstration that dense pixel-level grounding and open-ended language understanding are not inherently in tension—the catastrophic forgetting observed in LISA, GLaMM, and OMG-LLaVA was an artifact of data imbalance, not an architectural necessity. This reframing matters because it redirects the field away from designing increasingly elaborate architectural isolation mechanisms (separate vision towers, task-specific routing, frozen perception experts with complex adapters) and toward a simpler principle: maintain balanced co-training data, and the model will maintain balanced capabilities.

The evidence for this reframing is the stark contrast in Table 7: LISA-7B scoring 1/1 on MME while Sa2VA-8B scores 1651/578, despite both using SAM-family backbones and special token mechanisms. The difference is not architectural—it is that Sa2VA includes 665K image QA examples in its training mixture, while LISA's fine-tuning consisted almost entirely of segmentation data. This is simultaneously encouraging (the problem is solvable without exotic architectures) and humbling (the field spent years designing complex solutions to a data engineering problem).

The paper also resolves a latent contradiction in the video understanding literature. On one side, video MLLMs (LLaVA-OneVision, LLaMA-VID, mPLUG-Owl3) achieved strong Video-QA but no pixel grounding. On the other, video perception models (UNINEXT, UniRef++, SAM-2) achieved strong tracking but no language understanding. The implicit assumption was that these capabilities required different architectural primitives—dense feature pyramids for segmentation, token-level attention for language. Sa2VA demonstrates that a single autoregressive transformer can coordinate both, provided it has access to a frozen perception backbone through a narrow information bottleneck. The [SEG] token mechanism is the minimal viable interface: one vector, one linear projection, one decoder call. Everything else is standard MLLM autoregressive generation.

For the broader multimodal systems community, this work establishes an amplification-not-creation boundary that refines expectations about what combining foundation models can achieve. Sa2VA dramatically amplifies the referring video segmentation capabilities of SAM-2 (from 11.8 J&F for VISA to 41.3 J&F for Sa2VA zero-shot on Ref-SAV, Table 10), but it does not create fundamentally new perception capabilities—the hardest Ref-VOS scenarios remain at only 50% J&F even with full training. This boundary is valuable because it tells researchers where to invest: if you need better tracking through heavy occlusion, improve SAM-2 (or its successor). If you need better disambiguation of complex referring expressions, improve the MLLM's language understanding or the co-training data mixture. Don't expect the combination to solve problems that neither component solves individually.

The Ref-SAV benchmark also introduces a diagnostic capability that the field previously lacked. Prior Ref-VOS evaluations confounded visual difficulty, linguistic complexity, and temporal challenge into a single score. Ref-SAV's separate long-expression and short-expression subsets on the same videos enable differential diagnosis: the 16 J&F gap between Sa2VA's long-expression (49.3) and short-expression (33.3) zero-shot performance (Table 10) reveals that the model struggles with ambiguity resolution, not perceptual grounding. This diagnostic framework is perhaps more valuable than the benchmark itself—it provides a template for how to construct evaluations that reveal why models fail, not just that they fail.

Finally, the paper shifts what "state-of-the-art" means for multimodal models. Prior to Sa2VA, a model could claim SOTA by excelling at either segmentation or conversation, but not both simultaneously. The fact that Sa2VA-8B achieves 81.9 cIoU on RefCOCO while scoring 1651/578 on MME establishes a new baseline: from this point forward, a multimodal system that claims comprehensive visual understanding must demonstrate competence across both dense grounding and open-ended conversation. Specialists that sacrifice one for the other are no longer competitive for the "general visual assistant" goal.

Follow-Up Research This Work Enables

Adaptive inference strategies that allocate key frames and MLLM decode length based on estimated video difficulty. The paper's own ablation (Table 20) reveals a 4 J&F gap between naive first-5-frame sampling and uniform 5-frame sampling, with identical inference cost. This implies substantial headroom for difficulty-aware allocation: easy videos (single object, no occlusion, static camera) might need only 1–2 key frames and a short MLLM response, while hard videos (multiple similar objects, heavy occlusion, camera motion) might benefit from 10+ uniformly sampled key frames and more verbose MLLM reasoning. A concrete experiment would train a lightweight difficulty predictor (perhaps the MLLM's own confidence score on an initial fast pass) and use it to allocate a fixed total inference budget across a batch of videos, then measure whether budget-aware allocation outperforms uniform allocation at the same average cost. The Ref-SAV benchmark's long/short split provides natural test cases: easy cases (long expressions with multiple discriminative cues) should receive less budget than hard cases (short ambiguous expressions), inverting the paper's current uniform treatment.

Investigating whether the inverted difficulty curve—better performance on long expressions than short—can be corrected through data balancing. The finding that Sa2VA-8B achieves 49.3 J&F on 83.6-word expressions but only 33.3 J&F on 7–10 word expressions (Table 10) is the paper's most puzzling and practically consequential result. It suggests the model has learned to integrate multiple grounding cues but has not learned to resolve ambiguity when cues are sparse. A direct follow-up would generate paired training data: for each object in Ref-SAV, produce both a long expression (via the three-stage pipeline) and a short expression (by prompting the annotator LLM to summarize, or by using only the object-level description stage). Training on this balanced mixture—and measuring whether the short-expression gap closes without sacrificing long-expression performance—would distinguish between two hypotheses: (1) short expressions are fundamentally harder and the gap is irreducible with current architectures, or (2) the gap is purely a training data imbalance, and balanced training eliminates it. The second hypothesis is consistent with the paper's co-training philosophy; the first would indicate a genuine architectural limitation in how Sa2VA handles uncertainty.

Replacing the frozen SAM-2 encoder with a fine-tuned variant that learns to produce features optimized for language-prompted mask decoding. The paper freezes SAM-2's encoder and memory module (Table 5) for architectural flexibility and to preserve pre-trained tracking capabilities, but provides no ablation testing whether this choice is optimal. A straightforward experiment would fine-tune SAM-2's encoder jointly with the decoder during Sa2VA training, using the same co-training data mixture, and measure the impact on Ref-SAV's hardest cases (heavy occlusion subset, if segmentable by occlusion level; or the long-expression subset where perceptual precision matters most). If fine-tuning improves performance without degrading tracking on standard benchmarks (Ref-DAVIS17, Ref-YTVOS), it would suggest the freezing strategy is overly conservative. If fine-tuning degrades tracking, it would validate the paper's decoupled design and establish an important negative result: that language-conditioned and visually-prompted segmentation require different feature representations. A more targeted variant would fine-tune only SAM-2's memory module, testing whether the cross-attention mechanism can learn to better retain language-identified objects through occlusions when the prompt embedding carries richer semantic information than a point or box.

Extending the [SEG] token mechanism to multi-object and hierarchical segmentation tasks. Sa2VA's current design assumes one [SEG] token per referred object—sufficient for referring segmentation but inadequate for tasks requiring simultaneous segmentation of multiple interacting objects (panoptic segmentation, video instance segmentation, scene understanding with multiple grounded entities). A natural extension would introduce multiple distinct [SEG] tokens (e.g., [SEG_0], [SEG_1], etc.) where each token's hidden state encodes a different object identity, and the MLLM learns to emit the correct token for each object in a scene description. The challenge—and the research contribution—would be in designing the training data and loss function to ensure that different tokens consistently correspond to the same objects across examples, avoiding the permutation problem that plagues multi-object detection. Ref-SAV videos with multiple annotated objects (Table 4 shows 72,509 expressions for 37,311 videos, averaging ~1.94 objects per video) provide natural training data. A concrete experiment would evaluate on a multi-object variant of Ref-SAV where the model must simultaneously segment 2–5 objects per video, measuring whether the token identity remains stable across frames and whether performance degrades relative to single-object segmentation.

Stress-testing the Ref-SAV annotation pipeline's biases by comparing model performance on human-written versus LLM-generated expressions describing the same objects. The paper acknowledges that Ref-SAV's long expressions are automatically generated and human-filtered (Section 3.3), but does not characterize the linguistic differences between automatically generated and human-written referring expressions. A controlled experiment would take the 251 manually annotated short expressions from Ref-SAV's validation set and, for the same objects, generate long expressions using the three-stage pipeline. Then evaluate Sa2VA and other models on both expression sets, measuring whether performance differs systematically. If models perform better on LLM-generated expressions than human-written ones for the same objects, it would indicate an annotation bias—the models are overfitting to the annotator LLM's linguistic patterns (which may be shared with the MLLM component of Sa2VA). If performance is comparable, it validates the pipeline's ecological validity. This experiment would also inform whether future dataset construction should invest in expensive human annotation or can rely on scalable automatic pipelines.

Exploring whether the compute-optimal allocation principles from test-time scaling literature apply to Sa2VA's multi-frame video inference. The paper demonstrates (Table 20) that allocating more key frames improves performance at higher compute cost, but does not explore whether the distribution of those frames can be optimized per-video. A follow-up could treat the key frame selection problem as a budgeted optimization: given a total frame budget N, which N frames maximize the probability of correct segmentation? This is structurally analogous to the compute-optimal test-time scaling problem from the example paper, where different difficulty levels demanded different search strategies. For Sa2VA, "easy" videos (single prominent object, simple motion) might achieve ceiling performance with N=1 frame at a fraction of the cost, while "hard" videos need N=10+ frames with uniform or adaptive sampling. A concrete setup would use the Ref-SAV benchmark, train a per-video difficulty predictor from the MLLM's initial confidence scores, and define a policy that allocates the N budget non-uniformly across a video batch. The metric would be overall J&F at fixed average inference cost, compared to uniform allocation. Positive results would establish that the same difficulty-conditioned allocation principles that produced 4× efficiency gains in the example paper apply to video segmentation.

Practical Applications and Downstream Use Cases

Interactive video editing with natural language object selection. Modern video editing tools (Adobe Premiere, DaVinci Resolve, Final Cut Pro) require editors to manually rotoscope objects—drawing masks frame by frame—to apply effects, remove backgrounds, or replace elements. Sa2VA enables a workflow where the editor types "select the person in the red jacket walking from left to right" and receives a pixel-accurate masklet across the entire clip, produced in approximately 0.2 seconds (for Sa2VA-8B on an H100 GPU, Table 19). The 75.2 J&F on Ref-DAVIS17 and 70.7 J&F on Ref-YTVOS (Table 6) indicate that the initial automatic masks are strong enough to serve as a starting point, dramatically reducing the rotoscoping burden. The 57.6 J&F on ReVOS (Table 6) suggests that even for complex, recently-collected videos, Sa2VA provides useful first-pass masks. The Ref-SAV-trained variant's 58.7 J&F on long expressions with heavy occlusions (Table 10) indicates that the system handles realistic editing scenarios where objects are partially obscured.

Surveillance and monitoring systems with compositional natural language queries. Security operations centers monitor dozens or hundreds of video feeds simultaneously, with operators needing to quickly locate specific events or individuals. Sa2VA enables queries like "show me every person who entered through the north gate between 2:00 and 2:30 PM wearing a blue jacket" without requiring the operator to manually review footage or use rigid query interfaces. The model's ability to handle 83.6-word average expressions (Ref-SAV, Table 4) means it can process detailed suspect descriptions directly from witness reports, rather than requiring operators to translate natural language into structured database queries. The real-time performance—0.2 seconds per query on datacenter hardware (Table 19)—means the system could process archived footage at interactive speeds. The key limitation for deployment is the short-expression difficulty (33.3 J&F on Ref-SAV's short subset, Table 10): concise queries like "show me the person in the blue jacket" may fail when multiple people wear similar clothing. Deployment in this context would benefit from the follow-up research on data balancing to close the short-expression gap.

Automated video description for accessibility and content indexing. Generating dense, grounded descriptions of video content is critical for accessibility (describing videos for visually impaired users) and content indexing (making video archives searchable). Sa2VA's grounded caption generation capability—17.3 METEOR on the GCG benchmark, exceeding prior models including Osprey (16.6) and GLaMM (16.2) per Table 18—means it can produce descriptions that are not only accurate but explicitly linked to pixel regions, enabling downstream applications like "click on any part of this description to jump to that moment in the video." The 70.7 J&F on Ref-YTVOS and 57.6 J&F on ReVOS (Table 6) indicate that the grounding is reliable enough for production use on standard video domains. The Ref-SAV benchmark's long-expression capability (58.7 J&F trained, Table 10) demonstrates that the system can generate and ground detailed multi-sentence descriptions covering object appearance, scene context, and motion dynamics simultaneously—producing the kind of rich video understanding that current accessibility tools lack. The primary practical bottleneck is the autoregressive generation latency: a full grounded caption for a multi-minute video with many objects would require many MLLM decoding steps, each at ~0.2 seconds (Table 19), making real-time description generation impractical on current hardware for long-form content.

When to Prefer This Method

The paper implicitly positions Sa2VA against two alternatives: (1) specialist models that excel at either segmentation or conversation but not both, and (2) prior grounding MLLMs (LISA, GLaMM, OMG-LLaVA) that attempt both but suffer catastrophic forgetting of conversational ability. The decision rule emerging from the experimental results is:

• Prefer Sa2VA over specialist perception models (SAM-2, UNINEXT, UniRef++) when: the deployment requires natural language interaction with visual content—users will issue free-form referring expressions, ask questions about segmented objects, or expect the system to describe what it sees while simultaneously producing masks. Sa2VA-8B achieves 81.9 cIoU on RefCOCO (Table 6) while maintaining conversation, compared to pure perception models that require external language understanding. The trade-off is inference cost: Sa2VA-8B requires 0.201 seconds per sample (Table 19), dominated by LLM generation, versus SAM-2's 39.5 FPS single-pass inference. For high-throughput, language-free applications (e.g., batch video segmentation with fixed categories), the specialist model's speed advantage outweighs Sa2VA's language capability.

• Prefer Sa2VA over prior grounding MLLMs (LISA, GLaMM) when: maintaining conversational ability is a hard requirement—the system must chat about images and videos while also segmenting. The evidence is unambiguous: LISA-7B scores 1/1 on MME (Table 7), rendering it useless for any task requiring language generation beyond "here is the mask." Sa2VA-8B scores 1651/578 on MME, exceeding the InternVL2-8B chat baseline, meaning it can be used as a general visual assistant without sacrificing segmentation quality. The data requirement to achieve this balance is modest: 665K image QA examples plus 100K video QA examples (Table 2), co-trained with the segmentation data. This is a small fraction of the base MLLM's original pre-training data and should be feasible to collect or generate for any deployment domain.

• Prefer scaling the base MLLM over architectural modifications when: seeking improved Sa2VA performance. Table 8 shows consistent monotonic improvements when upgrading from InternVL2.0 to InternVL2.5 at all scales: RefCOCO improves from 81.9 to 82.6 cIoU at 8B, MeViS from 57.0 to 58.9 J&F, MMBench from 82.4 to 84.4. Table 14 shows further gains with InternVL3-14B (83.6 cIoU on RefCOCO, 84.3 on MMBench). The framework's modularity means these improvements come "for free" with the next generation of open-source MLLMs—no architectural redesign needed. Investment in custom perception-model integration (e.g., feeding SAM-2 features back into the LLM) would need to substantially exceed these baseline improvements to be worthwhile, and the paper's decoupled design suggests such integration is unnecessary and potentially harmful (as evidenced by prior work's catastrophic forgetting).