SAM 2: Segment Anything in Images and Videos

🎯 Pitch

SAM 2 takes the 'segment anything' idea from images to video, and in doing so actually becomes a better image segmentor: it's 6× faster than the original SAM while being more accurate, because its streaming memory architecture—designed for tracking objects across frames—generalizes back to static images by simply leaving the memory empty. For video, it needs 3× fewer user interactions to achieve higher accuracy than prior approaches, powered by a new dataset with 53× more masks than any existing video segmentation benchmark.

1. Executive Summary

This paper introduces Segment Anything Model 2 (SAM 2), a unified foundation model for promptable visual segmentation that extends the Segment Anything (SA) task from static images to videos. The model operates on a new Promptable Visual Segmentation (PVS) task — accepting clicks, boxes, or masks on any frame to define and refine spatio-temporal masklets — using a streaming transformer architecture with a memory bank (a FIFO queue of spatial feature maps and object pointer vectors from past frames) that conditions current-frame predictions on previously observed object context, enabling interactive refinement via minimal prompts. SAM 2 is trained on the newly collected Segment Anything Video (SA-V) dataset, built through a three-phase data engine where SAM 2 in the loop achieved 8.4× faster annotation than per-frame SAM, yielding 35.5M masks across 50.9K videos (53× more masks than any prior video segmentation dataset). In interactive video evaluation, SAM 2 delivers better accuracy while using 3× fewer interactions than SAM-plus-tracker baselines, and on the conventional semi-supervised VOS task (first-frame ground-truth mask input) it achieves 90.2 J &F on DAVIS 2017 and 76.6 on MOSE with a Hiera-B+ encoder at 43.8 FPS. On image segmentation, SAM 2 is more accurate than the original SAM (58.9 vs. 58.1 1-click mIoU on 23 zero-shot benchmarks) while running 6× faster, establishing that a streaming memory architecture originally designed for video can improve static-image segmentation when the memory is empty and the model naturally reverts to SAM-like behavior.

2. Context and Motivation

The Core Problem: Image Segmentation Alone Cannot Capture the Visual World

Segment Anything (Kirillov et al., 2023) established a foundation model for promptable image segmentation — given a click, box, or mask on a static image, SAM outputs a valid segmentation mask for the referenced object. This capability enabled an explosion of downstream applications, from medical imaging (Ma et al., 2024; Deng et al., 2023) to remote sensing (Chen et al., 2024) to camouflaged object detection (Tang et al., 2023). However, an image is fundamentally a frozen moment — a single temporal slice of a world where objects move, deform, become occluded, and reappear. As the authors state:

"an image is only a static snapshot of the real world in which visual segments can exhibit complex motion, and with the rapid growth of multimedia content, a significant portion is now recorded with a temporal dimension, particularly in video data."

This gap between static and temporal understanding is not merely an incremental concern. Many of the most important applications of visual perception — AR/VR, robotics, autonomous vehicles, video editing — require understanding when an object exists in a video, not just where it is in a single frame. A system that can segment a person's hat in a photo but loses track of it when the person turns their head in a video is incomplete. The paper frames this explicitly: "We believe a universal visual segmentation system should be applicable to both images and videos."

Why Video Segmentation Is Fundamentally Harder

Video segmentation presents challenges that go qualitatively beyond the image case. The paper identifies several specific difficulties (Section 3):

Appearance variation: Objects change dramatically across frames due to motion, deformation, lighting shifts, and camera angle changes. The same object can look entirely different at frame 1 and frame 50.
Occlusion and re-appearance: Objects disappear behind other objects and later reappear. The model must understand object persistence — that the occluded object is "still there" — rather than treating re-appearance as a new object. The paper measures this via the disappearance rate, reporting that 42.5% of manually annotated masklets in SA-V disappear in at least one frame and then re-appear (Table 3).
Degraded quality: Videos often have lower quality than images due to motion blur, lower resolution, and compression artifacts. A segmentation model must be robust to these degradations.
Temporal scale: Videos can be arbitrarily long (the paper evaluates on LVOSv2 with videos averaging 1.14 minutes). Processing thousands of frames efficiently while maintaining object identity is a computational and representational challenge that does not exist for static images.

These challenges collectively mean that naively applying an image segmentation model per-frame — even a strong one like SAM — cannot solve video segmentation. Objects must be tracked, not just detected, and the tracking must be robust to all of the above failure modes.

The Landscape of Existing Approaches

The paper positions itself against three distinct strands of prior work, each with specific limitations.

Interactive Video Object Segmentation (iVOS)

The iVOS task aims to efficiently obtain object segmentations across video frames with user guidance — typically scribbles, clicks, or bounding boxes. Early approaches used graph-based optimization (Wang et al., 2005; Bai & Sapiro, 2007; Fan et al., 2015) that propagated user annotations through spatio-temporal volumes. More recent methods adopt a modular design: convert user input into a mask on one frame, then propagate it to other frames using a separate tracking module (Heo et al., 2020; Cheng et al., 2021b; Delatolas et al., 2024).

The most recent generation — and the strongest baselines the paper compares against — combine SAM for image-level mask generation with specialized video trackers. For instance, SAM+Cutie uses SAM to produce masks from clicks or boxes, then feeds those masks to Cutie (a state-of-the-art VOS tracker) for propagation across frames. SAM+XMem++ follows the same pattern with the XMem++ tracker. Related approaches like Track-Anything (Yang et al., 2023) and Segment-and-Track-Anything (Cheng et al., 2023c) deploy similar decoupled architectures.

The paper identifies three critical limitations of these decoupled approaches (Section 2):

"the tracker may not work for all objects, SAM may not perform well on video frames, and there is no mechanism to interactively refine a model's mistakes, other than re-annotating using SAM in each frame and restarting the tracking from there."

This last point is particularly important. In a decoupled system, if the tracker loses the object at frame 50, the user must re-segment frame 50 from scratch with SAM (potentially requiring multiple clicks to define the object boundary), then restart the tracker from that frame. The prior context — what the object looked like in frames 1–49, what clicks the user already provided — is discarded. There is no memory of the interaction history.

Semi-Supervised Video Object Segmentation (VOS)

The VOS task (Pont-Tuset et al., 2017) provides a ground-truth mask on the first frame and requires tracking that object through the entire video. This is a special case of the PVS task — it is PVS with exactly one mask prompt on frame 1 and no further interaction.

Early deep learning approaches to VOS used online fine-tuning: take a pretrained model and fine-tune it on the first frame's mask at inference time to adapt to the specific object (Caelles et al., 2016; Perazzi et al., 2016; Maninis et al., 2017; Bhat et al., 2020). While effective, this is slow — each video requires its own training loop.

Faster methods emerged that avoid per-video fine-tuning. Some condition only on the first frame (Hu et al., 2018b; Chen et al., 2018), while others integrate the previous frame's prediction (Oh et al., 2018; Yang et al., 2018, 2020). The most successful recent architectures — XMem (Cheng & Schwing, 2022), Cutie (Cheng et al., 2023a), AOT/DeAOT (Yang et al., 2021b; Yang & Yang, 2022) — use transformer-based memory networks that maintain and update a representation of the tracked object across frames. XMem introduced an Atkinson-Shiffrin-inspired memory model with short-term, long-term, and working memory. Cutie improved object representation by distinguishing foreground object features from background context. DeAOT used hierarchical feature propagation for efficiency.

These methods are highly specialized for the VOS task. They accept only mask inputs (not clicks or boxes), can only receive prompts on the first frame, and offer no mechanism for interactive refinement. The paper notes a practical concern:

"annotating the required high-quality object mask in the first frame in VOS is practically challenging and time-consuming for inference."

In real-world usage, drawing a precise pixel-level mask on the first frame is far more burdensome than providing a few clicks or a bounding box. SAM 2 is designed to accept the full spectrum of prompt types and allow refinement on any frame.

Video Segmentation Datasets: Insufficient Coverage for "Segment Anything"

A central motivation for the paper is that existing video segmentation datasets are too narrow to support the goal of "segmenting anything in videos." The authors analyze this limitation along several dimensions (Section 2, Table 3):

Scale. The largest prior dataset is YouTube-VOS (Xu et al., 2018b) with 197K masks across 4.5K videos. MOSE (Ding et al., 2023) provides 431K masks across 2.1K videos, and BURST (Athar et al., 2022) provides 600K masks across 2.9K videos. SA-V provides 35.5M masks — 53× more than any existing dataset — making it possible to train models that encounter sufficient diversity of objects, motions, and scenarios.

Object coverage. Existing datasets are overwhelmingly restricted to specific object categories: people, vehicles, animals, and common household objects. They annotate whole objects — a person, a car, a dog — but rarely parts and subparts — a person's hat, a car's side mirror, a dog's ear. The SA data philosophy, carried forward from the original SAM, is to segment any object with a valid visual boundary, including parts. The paper states:

"We find that current video segmentation datasets lack sufficient coverage to achieve the capability of 'segmenting anything in videos'. Their annotations typically cover entire objects (not parts) and datasets are often centered around specific object classes."

Figure 10a shows that over 88% of SA-V masks have a normalized mask area less than 0.1 (i.e., occupy less than 10% of the frame), indicating a heavy emphasis on small objects and parts — precisely the regime where prior datasets are sparse.

Challenging scenarios. Later datasets like VOST (Tokmakov et al., 2022) focused on extreme transformations (e.g., an egg being broken, paper being torn), MOSE (Ding et al., 2023) emphasized occlusions in complex scenes, and LVOS (Hong et al., 2023, 2024) targeted long videos. Each of these captures one dimension of difficulty, but none combines sufficient scale with broad object coverage and interactive annotation capabilities.

Where the Decoupled Approach Breaks Down

The paper illustrates the failure mode of decoupled SAM+tracker approaches with a concrete example (Figure 2, and the accompanying description):

Imagine segmenting a person's tongue in a video. The user provides positive and negative clicks on frame 1 to define the tongue (not the whole face, not the mouth — just the tongue). SAM produces the mask, and the tracker propagates it forward. By frame 3, the tracker has lost the object (perhaps the tongue briefly retracted and reappeared, or the tracker drifted to the lips).

In a SAM+tracker system, the user must now re-annotate frame 3 from scratch with SAM — providing multiple clicks to re-define the tongue's boundary — and restart the tracker. There is no memory of the frame 1 annotation. With SAM 2, a single click on frame 3 is sufficient: the model's memory retains the context of what was being tracked, recognizes that the click refers to the same tongue, and recovers the correct segmentation. This is the practical consequence of having a unified model with memory rather than a decoupled pipeline.

How SAM 2 Positions Itself

SAM 2 is explicitly positioned not as a specialized VOS model nor as a SAM replacement, but as a unification. The paper describes SAM 2 as "a natural generalization of SAM to the video domain" (Section 4). The PVS task (Section 3) is designed as a superset: the SA task (segment anything in images) is PVS with a single-frame video, and semi-supervised VOS (track an object given a first-frame mask) is PVS with exactly one mask prompt on frame 1 and no further interaction.

This unification has architectural implications. Rather than designing separate models for images and videos, SAM 2 uses one model. When the memory bank is empty (as it is for single-image inputs), the memory attention has nothing to attend to, and the model naturally behaves like SAM. The authors leverage this by training jointly on image and video data (Section 4), which improves performance on both modalities — video data helps the image segmentation accuracy on video-derived benchmarks, and image data (SA-1B) ensures the video model doesn't lose its spatial segmentation capability.

The paper's position is thus threefold:

Task unification: PVS subsumes SA, iVOS, and VOS, providing a single interface for all promptable segmentation.
Model unification: A single streaming architecture with memory handles both images (memory empty) and videos (memory populated), avoiding the decoupled SAM+tracker approach and its interaction-reset problem.
Data unification: The SA-V dataset, collected through an iterative model-in-loop data engine, provides the scale and diversity needed for "segment anything in videos" — covering whole objects, parts, occlusions, re-appearances, and diverse geographic and demographic sources — at a scale 53× larger than prior datasets.

The Data Engine as a Self-Reinforcing Loop

A subtle but important aspect of the paper's positioning is the role of the data engine (Section 5.1) as a proof of concept for self-improving systems. The engine went through three phases:

Phase 1 (per-frame SAM): 37.8 seconds per frame, no temporal assistance. Produced 16K masklets but was too slow to scale.
Phase 2 (SAM + SAM 2 Mask): 5.1× faster than Phase 1, with a mask-only version of SAM 2 propagating annotations temporally. However, annotators still had to re-annotate intermediate frames from scratch when the propagation failed — they couldn't provide refinement clicks; they had to draw new masks.
Phase 3 (fully-featured SAM 2): 8.4× faster than Phase 1, with annotators providing only occasional refinement clicks. The model's memory preserved the context of all previous interactions, allowing a single click to correct errors that would require a full re-annotation in earlier phases.

This progression demonstrates that the unified memory architecture isn't just an academic improvement — it fundamentally changes the annotation experience, reducing effort while maintaining quality (Table 1: Phase 3 achieves 89.1% Mask Alignment Score vs. Phase 1, meaning 89.1% of Phase 3 masks have IoU > 0.75 with the corresponding high-quality Phase 1 masks). The data engine is thus both a means to collect training data and a validation of the interaction paradigm that SAM 2 enables.

3. Technical Approach

3.1 Reader orientation (approachable technical breakdown)

SAM 2 is a unified neural network that takes a video and user-provided clicks, boxes, or masks on any frame, and outputs a spatio-temporal segmentation mask (a “masklet”) tracking the specified object across all frames, while maintaining a memory of past prompts and predictions to enable single-click refinements when the model makes mistakes. The system solves the fundamental problem that decoupled SAM-plus-tracker approaches completely discard interaction history when corrections are needed, forcing users to re-annotate from scratch — SAM 2’s streaming memory architecture instead preserves the full context of all previous interactions, allowing minimal, incremental corrections that propagate forward through time.

3.2 Big-picture architecture (diagram in words)

SAM 2 consists of five major components connected in a streaming pipeline:

Image encoder (a MAE-pretrained Hiera hierarchical vision transformer) — processes each video frame exactly once as it arrives, producing multiscale feature maps that serve as the foundational visual representation for all downstream components.
Memory bank (a FIFO queue storing spatial feature maps from up to N recent frames and M prompted frames, plus lightweight object pointer vectors) — retains compressed representations of past predictions and prompt locations, providing the temporal context that the model conditions on to maintain object identity across frames.
Memory attention (a stack of L transformer blocks performing self-attention on current frame features followed by cross-attention to the memory bank) — infuses the current frame’s image features with information about what the object looked like in previous frames and where it was prompted, producing a temporally-conditioned embedding for the current frame.
Prompt encoder and mask decoder (nearly identical to SAM’s design, with additional occlusion prediction and skip connections from the hierarchical image encoder) — takes the temporally-conditioned frame embedding and any user prompts (clicks, boxes, masks) on the current frame, and outputs a segmentation mask, a predicted IoU score, and an occlusion probability for that frame.
Memory encoder (a lightweight convolutional module) — fuses the mask decoder’s output mask with the unconditioned image encoder features to produce a compressed spatial memory that is pushed into the memory bank for use in subsequent frames.

Information flows as follows: a video frame enters the image encoder → the image encoder produces unconditioned feature maps → the memory attention conditions these features on the memory bank’s stored representations from past frames → the mask decoder ingests the temporally-conditioned features and any user prompts for the current frame, outputting a mask, IoU score, and occlusion prediction → the memory encoder fuses the predicted mask with the image features and pushes the result into the memory bank → the next frame arrives and the cycle repeats, with the memory bank now containing the just-computed representation.

3.3 Roadmap for the deep dive

First, the formal PVS task definition, establishing exactly what inputs the model receives, what outputs it must produce, and how ambiguity across video frames is handled — this is the contract that the architecture must satisfy.
Second, the image encoder — the Hiera backbone, why it is hierarchical and how multiscale features are fused — since all downstream components depend on the quality and structure of these feature maps.
Third, the memory bank and its contents (spatial memories, object pointers, temporal encoding, occlusion embeddings) — because understanding what is stored and how it is structured is prerequisite to understanding memory attention.
Fourth, the memory attention mechanism — how cross-attention to the memory bank works, the role of self-attention on current frame features, and the positional encoding choices — since this is the central architectural novelty that distinguishes SAM 2 from SAM.
Fifth, the prompt encoder and mask decoder — how prompts are represented, the two-way transformer design, multi-mask output for ambiguity resolution, the occlusion prediction head, and the skip connections from the hierarchical encoder — since this is where the actual segmentation prediction happens.
Sixth, the memory encoder — how predictions are compressed into storable memories — completing the loop from prediction back to memory for subsequent frames.
Seventh, the training procedure — data mixture, interactive simulation, loss functions, and the two-stage pretraining then full-training schedule — since the training protocol is as important as the architecture for achieving the reported results.
Eighth, the data engine — the three-phase annotation pipeline and auto masklet generation — since the training data’s scale and quality are inseparable from the model’s capabilities.

3.4 Detailed, sentence-based technical breakdown

This is primarily a systems and data engineering paper whose core technical idea is that a streaming transformer architecture with a FIFO memory bank storing spatial feature maps and learned object pointers from past frames can unify interactive image segmentation with video object tracking under a single model, eliminating the interaction-reset problem that plagues decoupled SAM-plus-tracker pipelines while remaining real-time (43.8 FPS with the B+ encoder). The architecture is not a radical departure from SAM but rather a careful extension: the image encoder is replaced with a more efficient hierarchical backbone, the mask decoder gains skip connections and an occlusion head, and — most critically — a memory attention module is inserted between the image encoder and mask decoder to condition current-frame features on temporally distant context.

3.4.1 The Promptable Visual Segmentation (PVS) Task

The PVS task (Section 3) defines the input-output contract that SAM 2 must satisfy. Understanding this contract is essential because several architectural choices — multi-mask output, occlusion prediction, the memory bank’s preservation of prompted frames — are direct responses to task requirements.

Input. The model receives a video (an ordered sequence of frames) and, at any point during interaction, prompts on any frame. Prompts can be of three types:

Positive clicks: the user indicates a point on the object to segment.
Negative clicks: the user indicates a point that is not part of the object (background).
Bounding boxes: the user draws a box around the object.
Masks: the user provides a coarse or precise pixel-level mask.

Prompts are not restricted to the first frame. The user can prompt frame 1 to define the object, then later prompt frame 50 to correct a tracking failure, then prompt frame 30 to refine a boundary — in any order and on any frame. The paper emphasizes that prompted frames can “come from the future relative to the current frame,” meaning the model must be able to incorporate prompts from frames that appear later in the video timeline.

Output. Upon receiving a prompt on any specific frame, the model must immediately produce a valid segmentation mask for that frame — the interactive response. Simultaneously, it must propagate the accumulated set of prompts to produce the complete masklet: the segmentation mask of the target object on every frame of the video.

Refinement. If the predicted masklet contains errors (the model loses the object, includes too much background, misses a part), the user can provide additional prompts on any frame. The model must incorporate these new prompts and update the masklet across all frames — not just frames after the correction. This is the key distinction from online-only evaluation where corrections only affect future frames. In the full PVS setting, adding a prompt to frame 50 should improve segmentation on frames 1–49 as well, because the model can now retrospectively understand that what it saw earlier belonged to the same object.

Ambiguity across frames. SAM introduced the ability to output multiple valid masks when a prompt is ambiguous (e.g., a click on a bicycle tire could refer to just the tire or the entire bicycle). In video, this ambiguity extends temporally. The paper gives this example:

“For example, if in one frame only the tire is visible, a click on the tire might relate to just the tire, or as more of the bike becomes visible in subsequent frames, this click could have been intended for the entire bike.”

SAM 2 handles this by predicting multiple masks at each step of the video. If subsequent prompts do not resolve the ambiguity, the model selects the mask with the highest predicted IoU for the current frame and propagates only that mask forward.

Occlusion. Unlike the image segmentation setting where there is always a valid object given a positive prompt, in video the object may disappear (become fully occluded or leave the frame) and later reappear. The model must predict that the object is absent on those frames rather than hallucinating a mask.

Relationship to prior tasks. PVS is a superset: the SA task (Kirillov et al., 2023) is PVS with a single-frame video; semi-supervised VOS (Pont-Tuset et al., 2017) is PVS with exactly one mask prompt on frame 1 and no further interaction; interactive VOS is PVS with scribbles or masks on multiple frames. SAM 2 is designed to handle all of these cases within the same architecture.

3.4.2 The Image Encoder: Hiera with Multiscale Feature Extraction

The image encoder is responsible for producing rich, unconditioned feature representations of each video frame. It runs exactly once per frame for the entire interaction (Section 4: “The image encoder is only run once for the entire interaction and its role is to provide unconditioned tokens (feature embeddings) representing each frame”).

Backbone choice: Hiera. The authors use a Hiera (Ryali et al., 2023; Bolya et al., 2023) hierarchical vision transformer, pretrained with MAE (He et al., 2022). Hiera is a “hierarchical vision transformer without the bells-and-whistles” — it eschews many of the architectural complexities that have accumulated in modern ViT designs (relative positional biases, convolutional patch embeddings, etc.) in favor of a simpler, faster, and more scalable architecture. The authors ablate different sizes — Tiny (T), Small (S), Base+ (B+), and Large (L) — with the B+ variant used as the default for the main results (43.8 FPS), and the L variant providing the highest accuracy (30.2 FPS).

Why hierarchical matters. A standard ViT produces a single-resolution feature map (e.g., 16×16 patches), but segmentation benefits from features at multiple scales: coarse features capture object-level semantics, while fine features capture boundary details. Hiera is hierarchical because it progressively downsamples spatial resolution across stages (like a CNN), producing feature maps at strides 4, 8, 16, and 32 relative to the input. The paper uses strides 16 and 32 for the main processing pipeline, and strides 4 and 8 as skip connections to the mask decoder for high-resolution boundary refinement.

Feature pyramid network (FPN). The image encoder does not directly output Hiera’s raw stage features. Instead, an FPN (Lin et al., 2017) fuses the stride-16 and stride-32 features:

“We use a feature pyramid network (Lin et al., 2017) to fuse the stride 16 and 32 features from Stages 3 and 4 of the Hiera image encoder respectively to produce the image embeddings for each frame.”

An FPN works by taking the coarser (lower-resolution, semantically richer) features, upsampling them, and combining them with the finer (higher-resolution, spatially precise) features via lateral connections. The result is a single feature map — the “image embedding” — that serves as the input to the memory attention and mask decoder. This fused embedding balances semantic understanding with spatial precision.

Stride 4 and 8 skip connections. The stride-4 and stride-8 features from Hiera’s Stages 1 and 2 are not passed through the memory attention. Instead, they bypass it entirely and are injected directly into the mask decoder’s upsampling layers. This is a design choice motivated by two considerations:

Efficiency: memory attention operates on lower-resolution features (stride 16/32), keeping the cross-attention computation manageable. Adding high-resolution features to the memory attention would dramatically increase memory and compute costs.
Semantic appropriateness: the high-resolution features contain fine spatial details (edges, textures) that are important for precise mask boundaries but are less relevant for the high-level object matching that memory attention performs. The memory attention benefits from coarser, more semantic features for determining “is this the same object?”

Positional encoding. The original SAM used Relative Positional Bias (RPB) in its ViT image encoder. Bolya et al. (2023) improved efficiency by removing RPB from most layers and using “absolute-win” positional encoding. SAM 2 goes further:

“We improve upon this further by removing all RPB from the image encoder, with no performance regression on SA-23 and minimal regression on video benchmarks... while giving a significant speed boost at 1024 resolution.”

This elimination of RPB is critical for enabling FlashAttention-2 (Dao, 2023), a highly optimized attention kernel that does not support RPB. The removal also simplifies the architecture. Positional information is instead provided by windowed absolute positional embeddings (following Bolya et al., 2023) with global positional embeddings interpolated to span across windows.

Global attention layers. Not all layers in the Hiera encoder use full global attention. Following Li et al. (2022b), global attention is used in only a subset of layers, with the remaining layers using more efficient windowed attention. The specific layers with global attention vary by model size (Table 12): for B+, global attention is applied in blocks 12, 16, and 20 (out of more layers in total). This hybrid approach balances the global context needed for object understanding with the efficiency of local attention.

Input resolution and processing. The default training and inference resolution is 1024×1024 (square). Images are resized to this resolution, with horizontal flip augmentation applied during training. The paper ablates resolution (Table 9a) and finds substantial gains from higher resolution: moving from 512² to 1024² improves MOSE dev J &F from 73.0 to 77.0 (3-click, first frame), at the cost of 4.5× slower encoding.

3.4.3 The Memory Bank: What Is Stored and How It Is Organized

The memory bank is the central data structure that gives SAM 2 its temporal reasoning capability. It is not a learned neural module but rather a FIFO (First-In-First-Out) queue — a fixed-capacity buffer that discards the oldest entries when new ones are added — storing compressed representations of past frames and interactions.

Two types of stored memories. The memory bank maintains two separate FIFO queues (Section 4):

Recent frame memories (up to N frames): spatial feature maps produced by the memory encoder for the most recent N frames. By default, N = 6 (Table 9c). These provide short-term temporal context — what the object looked like in the immediate past, how it moved, how its appearance changed.
Prompted frame memories (up to M frames): spatial feature maps for frames where the user provided prompts. These are retained regardless of temporal distance — a prompted frame from the beginning of a long video stays in the memory bank even as recent frames cycle through. This ensures that the initial object definition and any corrections are never forgotten.

For the semi-supervised VOS setting where the only prompt is the first-frame mask, the memory bank “consistently retains the first frame’s memory along with memories of up to N recent (unprompted) frames.” The first frame is always present, serving as the permanent reference for the target object.

Spatial memory format. Each memory is stored as a spatial feature map — a tensor with spatial dimensions (reduced resolution relative to the input frame) and a channel dimension. The paper ablates the channel dimension (Table 9d) and finds that 64 channels are sufficient:

“Using fewer channels for memories does not cause much performance regression... while making the memory required for storage 4× smaller.”

This is a deliberate efficiency choice: the memory bank must store representations for multiple frames, and reducing the channel dimension from 256 to 64 cuts storage by 4× with minimal accuracy loss.

Object pointers. In addition to the spatial feature maps, the memory bank stores a list of “object pointers” — lightweight vectors derived from the mask decoder’s output token for each frame. Specifically:

“We use the mask token corresponding to the output mask as the object pointer token for the frame, which is placed in the memory bank.”

These object pointers are 256-dimensional vectors. They are split into 4 tokens of 64 dimensions for cross-attention, matching the 64-dim memory channel dimension. Object pointers provide high-level semantic information about the object — a compact summary of “what kind of thing is being tracked” — that complements the spatial feature maps. The ablation in Table 11 shows that object pointers do not improve average performance on the 9 zero-shot datasets but “significantly boosts performance on SA-V val dataset as well as on the challenging LVOSv2 benchmark,” suggesting they are particularly valuable for long-term tracking and part-level segmentation.

Temporal position encoding. The memories of the N recent frames are embedded with temporal position information, allowing the memory attention to represent short-term object motion (e.g., the object is moving leftward). Prompted frame memories do not receive temporal encoding:

“because the training signal from prompted frames is sparser and it is more difficult to generalize to the inference setting where prompted frames may come from a very different temporal range than seen during training.”

This is a nuanced generalization argument: during training, prompted frames appear at specific temporal positions within the 8-frame training sequences. If the model learned to associate temporal position with prompted-frame status, it might fail at inference when prompted frames appear at positions not seen during training (e.g., frame 200 of a 1000-frame video). By omitting temporal encoding from prompted frames, the model treats them as “timeless” references, generalizing to any temporal position.

Occlusion embeddings. When the occlusion prediction head determines that the object is not present on a frame, a learned occlusion embedding is added to that frame’s memory features in the memory bank. This explicitly signals to subsequent frames that the object was absent — a form of “negative information” that prevents the model from trying to match to an occluded frame’s features.

Memory bank capacity and computational cost. The number of memories stored (N recent + M prompted) directly affects the cost of cross-attention, which attends to all stored memories. The default N = 6 provides sufficient temporal context (about 0.25 seconds at 24 FPS) while keeping cross-attention manageable. Table 9c shows that increasing N from 4 to 6 to 8 provides diminishing returns (MOSE dev J &F: 73.5, 73.0, 73.2), and the authors choose N = 6 as the balance point.

3.4.4 Memory Attention: Conditioning Current Features on Temporal Context

Memory attention is the architectural component that distinguishes SAM 2 from SAM. It sits between the image encoder and the mask decoder, transforming unconditioned image features into temporally-conditioned features by attending to the memory bank.

Architecture: stack of transformer blocks. Memory attention consists of L transformer blocks (default L = 4; Table 9e). Each block performs three operations in sequence:

Self-attention on the current frame’s image features (the FPN-fused embedding from the image encoder). This allows spatial positions within the current frame to attend to each other — for instance, the region containing the object can integrate context from surrounding regions.
Cross-attention to the memory bank: the current frame features (as queries) attend to the stored spatial memories and object pointers (as keys and values) from past frames and prompted frames. This is where temporal conditioning happens — the current frame’s representation is infused with information about what the object looked like previously.
MLP (feed-forward network): a standard point-wise fully-connected layer, applied independently to each spatial position.

The L = 4 blocks are stacked sequentially, with the output of one block serving as input to the next. The first block takes the raw image embedding as input; the final block outputs the temporally-conditioned frame embedding that is passed to the mask decoder.

Vanilla attention operations. The paper uses standard scaled dot-product attention for both self-attention and cross-attention — no specialized memory-efficient variants like linear attention or performant approximations. The rationale is compatibility:

“We use vanilla attention operations for self- and cross-attention, allowing us to benefit from recent developments in efficient attention kernels (Dao, 2023).”

Specifically, FlashAttention-2 provides highly optimized implementations of exact attention that are faster and more memory-efficient than naive implementations while producing identical outputs. By using vanilla attention, SAM 2 can directly leverage these kernels without modification.

Positional encoding in memory attention. Two forms of positional encoding are used:

Sinusoidal absolute positional embeddings: standard fixed sinusoidal encodings added to the input features, providing each spatial position with a unique encoding that depends on its coordinates.
2D Rotary Positional Embedding (RoPE): applied in both self-attention and cross-attention layers. RoPE (Su et al., 2021; Heo et al., 2024) encodes relative position by rotating the query and key vectors by an angle proportional to their spatial separation. Unlike absolute encodings, RoPE naturally captures relative spatial relationships — for example, that a pixel 5 positions to the right of another pixel has a consistent positional relationship regardless of absolute location. 2D RoPE extends this to two spatial dimensions simultaneously.

Object pointer tokens are excluded from RoPE because they are semantic summaries, not spatially localized features — they don’t have a specific spatial position to encode.

The ablation in Table 10 compares three configurations: RPB in image encoder (SAM’s original approach), RPB removed + 2D-RoPE in memory attention, and RPB removed + no RoPE. Adding 2D-RoPE to memory attention improves MOSE dev J &F from 72.8 to 73.6 and LVOSv2 J &F from 70.3 to 71.0, while being slightly slower (0.93× vs. 1.04× speed). The authors adopt the configuration with 2D-RoPE.

Cross-attention mechanics. During cross-attention, the current frame’s features (after self-attention) serve as queries. The keys and values come from the concatenated memory bank contents:

Spatial feature maps from up to N recent frames (each of spatial dimension H×W, 64 channels).
Spatial feature maps from up to M prompted frames (same format).
Object pointer tokens (4 tokens of 64 dimensions each per frame, stored for all frames in the memory bank).

The cross-attention computes attention weights between every spatial position in the current frame and every stored memory element (spatial positions in past frames plus object pointers). This is computationally intensive — for N = 6, a typical feature map size, and M prompted frames, the attention matrix can be large. The reduced channel dimension (64) and the relatively small N keep this tractable while enabling FlashAttention-2’s optimizations.

What memory attention accomplishes functionally. After memory attention, each spatial position in the current frame’s feature map contains information not just about what that position looks like in the current frame, but also about how similar it is to the tracked object’s appearance in past frames and whether it corresponds to prompted locations. If the object was a red ball in frame 1, and memory attention at frame 50 encounters a red circular region, the cross-attention to frame 1’s memory will produce high attention weights, and the resulting conditioned features will encode “this region looks like the object being tracked.” The mask decoder can then use these features to produce a high-confidence mask for that region.

3.4.5 Prompt Encoder and Mask Decoder

The prompt encoder and mask decoder together form the segmentation head — the component that takes temporally-conditioned features and user prompts and produces the actual mask prediction. Their design largely follows SAM, with targeted modifications for video.

Prompt encoder. The prompt encoder is identical to SAM’s design (Section 4: “Our prompt encoder is identical to SAM’s”). It handles three types of sparse prompts and one type of dense prompt:

Sparse prompts (clicks and boxes): each click (positive or negative) is represented by a positional encoding (encoding its (x,y) coordinates) summed with a learned embedding that indicates the prompt type (positive click, negative click, or bounding box corner). Bounding boxes are represented by their top-left and bottom-right corners, encoded as a pair of positional encodings summed with learned embeddings. The resulting set of prompt tokens has variable length depending on how many clicks/boxes the user provides.
Dense prompts (masks): a user-provided mask (or a mask from a previous model prediction used as a prompt) is processed through convolutional layers to produce a feature map at the same spatial resolution as the frame embedding. This mask embedding is summed element-wise with the frame embedding. This means mask information is directly fused into the spatial features before the transformer processing, providing a strong spatial prior for where the object is located.

Mask decoder architecture. The mask decoder follows SAM’s “two-way” transformer design — a stack of transformer blocks where prompt tokens attend to image features (“token-to-image attention”) and image features attend to prompt tokens (“image-to-token attention”), creating a bidirectional information flow. The specific operations in each block (Figure 8) are:

Self-attention on the prompt tokens (including output tokens for masks, IoU, and occlusion).
Token-to-image cross-attention: prompt tokens attend to the temporally-conditioned frame embedding, gathering spatial information about where the prompted object appears.
MLP on prompt tokens.
Image-to-token cross-attention: the temporally-conditioned frame embedding attends to the (now updated) prompt tokens, infusing spatial features with information about which object is being queried.
MLP on image features.

After the transformer blocks, the output tokens are used to produce predictions (see below), and the updated image features are passed through upsampling layers to produce the final mask at the original image resolution.

Skip connections from image encoder. A novel modification to SAM’s design: the stride-4 and stride-8 features from Hiera’s Stages 1 and 2 are injected into the mask decoder’s upsampling pathway (Figure 8). Specifically, after the two-way transformer blocks, the decoder upsamples the (still relatively low-resolution) frame features. At each upsampling stage, the corresponding high-resolution features from the image encoder are concatenated or added, providing fine spatial details — edges, textures, thin structures — that were lost during the downsampling in the image encoder and were never passed through the memory attention.

This is critical for segmenting small objects and parts, which form the majority of SA-V annotations (88% of masks have normalized area < 0.1). Without these skip connections, the decoder would have to reconstruct fine boundary details from the relatively coarse temporally-conditioned features, which is especially difficult for thin or fast-moving objects.

Output predictions. The mask decoder produces three types of outputs from dedicated output tokens (Figure 8):

Segmentation masks (multiple): a dot product between the mask output tokens and the upsampled image features produces per-pixel mask logits. Multiple masks are produced simultaneously (typically 3), each from a different mask output token. This handles ambiguity — the model can output “just the tire,” “the whole wheel,” and “the entire bicycle” from a single click, allowing the user or downstream selection to resolve the ambiguity later.
IoU scores: an MLP head applied to a dedicated IoU output token predicts a scalar score for each mask, estimating the model’s confidence in that mask’s quality (predicted IoU with the ground truth). During multi-mask output, these scores are used to select the highest-confidence mask for propagation if no further prompts resolve the ambiguity.
Occlusion score: an MLP head applied to a dedicated occlusion output token predicts a scalar between 0 and 1 indicating whether the object of interest is visible in the current frame. This is a new head not present in SAM. It is trained with cross-entropy loss against ground-truth presence/absence labels (is there a valid mask for this object in this frame?).

How ambiguity resolution works in video. When the model receives an ambiguous prompt (e.g., a single click), it predicts multiple masks on that frame. If the user does not provide additional prompts to disambiguate (e.g., a negative click to exclude the whole bicycle and keep only the tire), the model:

“selects the mask with the highest predicted IoU for the current frame for further propagation in the video.”

This is a local decision: the highest-IoU mask for the current frame is propagated forward. However, ambiguity can also be resolved temporally — if in later frames it becomes clear that the object was the whole bicycle (because the entire bicycle becomes visible and the model’s features consistently match that interpretation), the propagation may effectively “switch” to the correct interpretation even if the initial frame’s highest-IoU mask was wrong. The paper doesn’t detail this mechanism explicitly, but the multi-mask prediction at each step combined with the memory’s temporal consistency pressure provides an implicit disambiguation signal.

Handling multiple objects. SAM 2 processes each object independently (Section D.1: “When applying SAM 2 to segment multiple objects in the same video... we perform inference on each object independently”). The image encoder output is shared across all objects (run once per frame, regardless of the number of objects), but all subsequent components — memory attention, memory bank, mask decoder — are run separately for each object. There is no inter-object communication or shared object-level context. This design choice prioritizes simplicity and modularity over potential efficiency gains from joint processing.

3.4.6 The Memory Encoder: Compressing Predictions into Storable Memories

After the mask decoder produces a mask for the current frame, the memory encoder transforms this prediction into a compressed spatial memory that can be stored in the memory bank and attended to by future frames.

Motivation for reusing image encoder features. The memory encoder does not use a separate image encoder to process the current frame. Instead, it reuses the unconditioned image embeddings from the Hiera encoder — the same features that were also passed to the memory attention and mask decoder. The paper explains:

“This design allows the memory features to benefit from the strong representations produced by the image encoder (especially when we scale the image encoder to a larger size).”

By sharing the image encoder between the forward pass (memory attention → mask decoder) and the memory creation pathway, the model avoids duplicating expensive vision backbone computation. Moreover, the representations that are useful for segmenting objects are also useful for storing memories of those objects.

Memory creation pipeline (Section 4, Figure 3):

Mask downsampling: the predicted mask (at the original image resolution or a higher resolution produced by the decoder’s upsampling pathway) is downsampled to match the spatial resolution of the image encoder’s feature map using a convolutional module. This produces a spatial feature map where each position encodes local mask information — essentially, a compressed representation of “where the object was in this frame.”
Element-wise summation: the downsampled mask features are summed element-wise with the unconditioned image embedding from the image encoder (specifically, the FPN-fused stride 16/32 features — the same features that serve as input to the memory attention, though the paper notes “not shown in Fig. 3” for the image encoder connection). This fuses appearance information (what the frame looks like) with segmentation information (where the object was) into a single spatial feature map.
Lightweight convolutional layers: the fused features are passed through additional light-weight convolutional layers to further process and refine the combined representation.
Channel projection: the resulting features are projected to the memory channel dimension (64 by default) for storage efficiency.

What gets stored in the memory bank. The output of the memory encoder is a spatial feature map with dimensions (height × width of the downsampled feature map, 64 channels). This is pushed into the FIFO queue of recent frame memories. If the frame was prompted (the user provided clicks, boxes, or masks on this frame), the memory is also added to the prompted frame memory queue. The object pointer (256-dim, split into 4×64-dim tokens from the mask output token) is also stored.

GRU ablation (Table 11). The paper explores whether feeding the memory features through a GRU (Gated Recurrent Unit) before storage provides benefits, motivated by prior work that used recurrent states for memory incorporation (Cho et al., 2014; common in tracking literature). The results are negative:

“While prior works have commonly employed GRU states as a means of incorporating memory into the tracking process, our findings in Table 11 suggest that this approach does not provide an improvement (except slightly on LVOSv2). Instead, we find it sufficient to directly store the memory features in the memory bank, which is both simpler and more efficient.”

This is an important simplification: the memory bank is purely a key-value store with no learned state transitions between memories. All temporal processing happens in the memory attention (the cross-attention from current frame to past frames), not in the memory storage mechanism.

Occlusion handling in memory. When the occlusion prediction head determines that the object is not visible on a frame, a learned occlusion embedding is added to that frame’s memory features in the bank. This explicitly signals “this frame contains no object” to future frames during cross-attention. Without this embedding, the memory attention might attempt to match current frame features to an occluded frame’s features, potentially causing tracking errors when the object reappears at a different location.

3.4.7 Training Procedure: Pretraining, Full Training, and Fine-Tuning

SAM 2’s training is a three-stage process: pretraining on images, full training on a mixture of image and video data with interactive simulation, and optional fine-tuning on longer sequences. The training protocol is as critical as the architecture for the model’s performance — this section details the data mixtures, loss functions, interactive simulation, and hyperparameters.

3.4.7.1 Stage 1: Pretraining on SA-1B (Static Images)

Before seeing any video data, SAM 2 is pretrained on the SA-1B image dataset (Kirillov et al., 2023) — 1 billion masks across 11 million images — using the standard interactive segmentation setup from SAM. This stage teaches the model the fundamental promptable segmentation capability: given clicks, boxes, or masks on an image, produce a valid segmentation mask.

Training setup (Table 12a). The key hyperparameters are:

Data: SA-1B, filtering masks covering more than 90% of the image and restricting to 64 randomly sampled masks per image.
Steps: approximately 90K iterations.
Resolution: 1024×1024 (square), with horizontal flip augmentation.
Optimizer: AdamW (Loshchilov & Hutter, 2019) with β₁ = 0.9, β₂ = 0.999.
Learning rate: 4×10⁻⁴, following a reciprocal square-root schedule (Zhai et al., 2022) with a timescale of 1000 and 1000 iterations of linear warmup followed by 5000 iterations of linear cooldown.
Weight decay: 0.1.
Gradient clipping: ℓ₂ norm, maximum 0.1.
Batch size: 256.
Layer-wise decay: applied to the image encoder (Hiera), with decay rates of 0.8 (T, S), 0.9 (B+), 0.925 (L). This means earlier layers have their learning rates decayed more strongly than later layers, a technique from Clark et al. (2020) that stabilizes fine-tuning of pretrained backbones.
Drop path: stochastic depth (dropping entire layers during training) with rates 0.1 (T, S), 0.2 (B+), 0.3 (L).

Image encoder initialization. The Hiera image encoder is initialized from MAE pretrained weights (He et al., 2022; Ryali et al., 2023). MAE pretraining involves masking random patches of the input image and training the model to reconstruct the missing patches, which produces strong visual representations that transfer well to downstream tasks.

Interactive simulation during pretraining. Following SAM, the training simulates the interactive user experience:

Initial prompts are sampled: ground-truth mask (probability 0.5), a positive click from the ground-truth mask (0.25), or a bounding box (0.25).
Up to 7 correction clicks are iteratively added, sampled from the center of the error region between the model’s current prediction and the ground truth.
An important difference from SAM: “In SAM, during iterative sampling of points, two iterations were inserted with no additional prompts (only feeding the previous mask logits) – we do not add such iterations during our training.” This means each training iteration always includes new click information, making the training signal stronger per step.

Loss functions (pretraining). The loss is a weighted combination:

Focal loss (weight 20) and dice loss (weight 1) for mask prediction. Focal loss is a modification of binary cross-entropy that down-weights easy examples, focusing training on hard-to-segment regions. Dice loss directly optimizes the overlap between predicted and ground-truth masks.
ℓ₁ loss (weight 1) for IoU prediction. The paper notes a modification from SAM: “we found it beneficial to use an ℓ₁ loss to more aggressively supervise the IoU predictions and to apply a sigmoid activation to the IoU logits to restrict the output into the range between 0 and 1.” The sigmoid ensures IoU predictions are calibrated probabilities rather than unbounded logits.
Multi-mask supervision: when predicting multiple masks (first click), all masks’ IoU predictions are supervised (to learn when a mask might be bad), but only the mask with the lowest segmentation loss (focal + dice) receives mask supervision. This prevents the model from being penalized for producing alternative valid masks.

3.4.7.2 Stage 2: Full Training on Image and Video Data

After pretraining, SAM 2 is trained on the full data mixture: SA-V (the newly collected dataset), Internal (internally licensed video data), a 10% subset of SA-1B, and optionally open-source VOS datasets (DAVIS, MOSE, YouTubeVOS). The released model is trained on SA-V manual, Internal, and SA-1B (without the open-source datasets).

Data mixture and sampling. Training is done jointly on image and video data using an alternating strategy:

“In each training iteration, we sample a full batch either from the image or video dataset, with their sampling probabilities proportional to the size of each data source.”

This means in each training step, the model either sees a batch of video sequences (8 frames each) or a batch of static images (1 frame each), with the probability of choosing each data source proportional to its size. The released model’s training mixture is approximately 70% SA-V, 14.8% Internal, and 15.2% SA-1B. When open-source datasets are included, the proportions shift to approximately 49.5% SA-V, 15.1% Internal, 15.5% SA-1B, 9.4% MOSE, 9.2% YouTubeVOS, and 1.3% DAVIS.

This alternating strategy serves two purposes:

Balanced exposure: the model learns both the video task (temporal propagation, memory usage) and the image task (spatial segmentation fundamentals) without either dominating.
Different batch sizes: image and video data can use different batch sizes to maximize GPU memory utilization. Video sequences (8 frames) consume more memory per sample than single images, so the video batch size can be smaller while the image batch size is larger.

Video training: interactive simulation. For video data, the training simulates the interactive PVS experience:

Sequence sampling: 8-frame sequences are randomly sampled from training videos. The temporal order is reversed with 50% probability to help generalization to bi-directional propagation (since at inference, prompts can come from any frame, requiring both forward and backward propagation).
Prompt frame selection: up to 2 frames (including the first) are randomly selected for corrective clicks.
Initial prompts: ground-truth mask (50%), a positive click from the ground-truth mask (25%), or a bounding box (25%).
Correction clicks: when corrective clicks are sampled, with 10% probability the clicks are randomly sampled from the ground-truth mask irrespective of the model prediction. This introduces “exploration” — the model learns to refine masks even when the clicks don’t correspond to obvious error regions.
Maximum masklets per sequence: restricted to 3 randomly chosen ones per 8-frame sequence.

Image training during full training. The image training follows the same protocol as pretraining, with the same initial prompt probabilities and correction click sampling.

Training hyperparameters (Table 12b). Key differences from pretraining:

Learning rate: image encoder gets 6×10⁻⁵, all other parameters get 3.0×10⁻⁴. The lower learning rate for the image encoder reflects that it was already well-trained during pretraining and should change more slowly.
Learning rate schedule: cosine decay (instead of the reciprocal square-root used in pretraining), with linear warmup for 7500 iterations.
Steps: approximately 150K iterations.
Batch size: 256.
Video augmentations: horizontal flip, random affine transforms (rotation up to 25 degrees, shear up to 20 degrees), random color jittering (brightness 0.1, contrast 0.03, saturation 0.03, hue null), random grayscale (5% probability), per-frame color jittering with different parameters for each frame (brightness 0.1, contrast 0.05, saturation 0.05, hue null), and a mosaic transform (described below).
Image augmentations: horizontal flip, resize to 1024×1024 square.

Mosaic transform. A specialized augmentation designed to train the model to handle challenging scenarios with multiple similar-looking objects:

“with 10% probability, we tile the same training video into a 2×2 grid and select a masklet from one of the 4 quadrants as the target object to segment.”

When this transform is applied, the same video is repeated four times in a 2×2 grid. The model must segment the target object in one quadrant while ignoring three identical copies in the other quadrants. This forces the model to rely on cues beyond appearance — such as motion continuity, temporal context from memory, and relative position — to distinguish the target from distractors. It also reduces the effective object size (each quadrant is half the original width and height), helping the model learn to segment small objects.

Loss functions (full training). The loss is extended from pretraining:

$\mathcal{L}_{\text{total}} = 20 \cdot \mathcal{L}_{\text{focal}} + 1 \cdot \mathcal{L}_{\text{dice}} + 1 \cdot \mathcal{L}_{\text{MAE}} + 1 \cdot \mathcal{L}_{\text{CE}}$

where $\mathcal{L}_{\text{focal}}$ is the focal loss for mask prediction, $\mathcal{L}_{\text{dice}}$ is the dice loss for mask prediction, $\mathcal{L}_{\text{MAE}}$ is the mean absolute error loss for IoU prediction, and $\mathcal{L}_{\text{CE}}$ is the cross-entropy loss for the occlusion prediction head.

What each loss component computes:

Focal loss (weight 20): a spatially-weighted binary classification loss where each pixel is classified as “object” or “background.” Focal loss reduces the contribution of easy pixels (those the model already classifies correctly with high confidence), focusing gradient on difficult edge cases. The weight of 20 makes this the dominant training signal.
Dice loss (weight 1): directly measures the overlap coefficient between the predicted mask and the ground-truth mask:

$\mathcal{L}_{\text{dice}} = 1 - \frac{2 \cdot |\text{pred} \cap \text{gt}|}{|\text{pred}| + |\text{gt}|}$

where $|\text{pred} \cap \text{gt}|$ is the number of correctly predicted object pixels, $|\text{pred}|$ is the total number of predicted object pixels, and $|\text{gt}|$ is the total number of ground-truth object pixels. The result is a scalar between 0 (perfect overlap) and 1 (no overlap). Unlike focal loss, dice loss directly optimizes the Jaccard-like metric used at evaluation time, providing a complementary training signal.

MAE loss (weight 1): the absolute difference between the predicted IoU score and the actual IoU between the predicted mask and ground truth. The sigmoid activation on the IoU logits ensures predicted values are in [0, 1].
Cross-entropy loss (weight 1): standard binary cross-entropy for the occlusion prediction head, comparing the predicted occlusion probability to the ground-truth “object present” vs. “object absent” label.

Why this loss combination and weighting: Focal loss provides strong per-pixel supervision, especially for hard-to-classify boundary pixels. Dice loss provides global shape supervision that is invariant to object size (unlike per-pixel losses that can be dominated by large background regions). The weight ratio of 20:1 ensures the primary optimization target is accurate per-pixel classification, with dice loss acting as a regularizing shape prior. The IoU and occlusion losses are secondary objectives that improve the model’s confidence calibration and its ability to handle disappearances — critical for the interactive experience but less important than mask quality.

Handling frames without ground-truth masks. If the ground-truth does not contain a mask for a frame (the object is occluded or out of frame), no mask loss is applied for that frame (the model is free to predict anything). However, the occlusion prediction head is always supervised — it must learn to predict “object absent” for frames without a ground-truth mask. This is crucial for the occlusion prediction to work correctly.

Multi-mask supervision. As in pretraining, for multi-mask predictions only the mask with the lowest segmentation loss receives mask supervision, while all masks’ IoU predictions are supervised.

3.4.7.3 Stage 3: Fine-Tuning on Longer Sequences

A potential shortcoming of the main training is that the model only sees 8-frame sequences, which is short compared to typical video lengths (the SA-V test videos average 14 seconds at 6 FPS annotation rate, roughly 84 frames; LVOSv2 videos average 1.14 minutes). To improve performance on long videos, an additional fine-tuning stage is introduced.

Fine-tuning setup:

Sequence length: 16 frames (doubled from 8).
Data selection: only the top 50% most edited masklets — those where annotators made the most corrections during data engine annotation — are used for SA-V and Internal datasets. This focuses training on the most challenging cases. Open-source datasets (DAVIS, MOSE, YouTubeVOS) are kept complete.
Iterations: 50K (one-third of the full training schedule).
Learning rate: half of the original learning rate (image encoder: 3×10⁻⁵, others: 1.5×10⁻⁴).
Image encoder: frozen (not updated) to fit the 16-frame sequences into the 80 GB memory of an A100 GPU. The longer sequences consume more memory, and freezing the largest component (the image encoder) reduces memory usage.

Why freeze the image encoder: The image encoder is the largest component of SAM 2. By freezing it during this fine-tuning stage, the model can process longer sequences (16 frames instead of 8) within the same GPU memory budget — the memory savings from not storing image encoder gradients are redirected to storing and processing more frames. The rationale is that the image encoder’s spatial feature extraction is already mature after pretraining and full training; fine-tuning focuses on improving the temporal components (memory attention, memory encoder) for longer-range propagation.

3.4.8 The Data Engine: Three-Phase Annotation with Model in the Loop

The SA-V dataset was collected through a three-phase data engine where increasingly capable versions of SAM 2 were deployed to assist human annotators. Understanding this process is essential because the data quality, diversity, and scale are inseparable from the model’s final capabilities — and because the data engine itself validates SAM 2’s interaction paradigm.

Phase 1: Per-frame SAM (no video assistance).

Annotators used the original image-based SAM to segment each frame independently at 6 FPS, with pixel-precise manual editing tools (brush, eraser). There was no temporal propagation — every frame required a complete re-annotation from scratch. This was slow (37.8 seconds per frame on average) but produced high-quality spatial annotations per frame because annotators carefully refined each mask individually.

Phase 1 produced 16K masklets across 1.4K videos. It also served as the annotation protocol for the SA-V validation and test sets, ensuring evaluation data is independent of SAM 2’s biases. The Phase 1 data is used as the quality reference standard: the “Phase 1 Mask Alignment Score” measures what percentage of masks from later phases have IoU > 0.75 with their Phase 1 counterparts.

Phase 2: SAM + SAM 2 Mask (mask-only propagation).

A preliminary version of SAM 2 that only accepted mask prompts (called “SAM 2 Mask”) was introduced. Annotators used SAM to create a spatial mask on the first frame, then used SAM 2 Mask to propagate it temporally. If the propagation failed, annotators re-annotated the problematic frame from scratch with SAM (no click-based refinement — they had to draw a new mask), then re-propagated.

SAM 2 Mask was initially trained on Phase 1 data and publicly available datasets. During Phase 2, it was retrained and updated twice with newly collected data. This phase produced 63.5K masklets at 7.4 seconds per frame — a 5.1× speed-up over Phase 1.

The key limitation: annotators had to create full masks from scratch for intermediate frame corrections because SAM 2 Mask had no memory of previous interactions and no mechanism for click-based refinement. This motivated the development of the fully-featured SAM 2.

Phase 3: Fully-featured SAM 2 (clicks, boxes, masks, and memory).

The complete SAM 2 — with memory, interactive prompting (clicks, boxes), and memory-based refinement — was deployed. Annotators used SAM 2 to segment objects, providing initial prompts and occasional refinement clicks. Because SAM 2 retained memory of all previous prompts and predictions, a single click could correct tracking errors that would require a full SAM re-annotation in Phase 2.

During Phase 3, SAM 2 was retrained and updated five times with newly collected annotations — the model and data co-evolved, with each iteration producing a better model that made annotation easier, which produced more data that further improved the model. Phase 3 produced 197.0K masklets at 4.5 seconds per frame — an 8.4× speed-up over Phase 1.

The controlled experiment in Table 1 (169 videos, 452 masklets annotated by three separate annotator groups, one per phase protocol) quantifies the improvements:

Time per frame: 37.8s (Phase 1) → 7.4s (Phase 2) → 4.5s (Phase 3).
Edited frames: 100% (every frame edited in Phase 1) → 23.25% (Phase 2) → 19.04% (Phase 3).
Clicks per clicked frame: 4.80 (Phase 1) → 3.61 (Phase 2) → 2.68 (Phase 3).
Mask Alignment Score (IoU > 0.75 with Phase 1): Phase 2 achieves 86.4% overall (71.3% for small objects); Phase 3 achieves 89.1% overall (72.8% for small objects). Critically, Phase 3 is both faster and higher quality than Phase 2 — the unified memory model not only reduces annotation effort but also improves annotation consistency.

Quality verification. Every masklet (manual and automatic) goes through a verification step where a separate set of annotators (not the ones who created the masklet) rate it as “satisfactory” (correctly and consistently tracking the target across all frames) or “unsatisfactory” (target is well-defined but masklet is incorrect or inconsistent). Unsatisfactory masklets are sent back for refinement. Masklets tracking not-well-defined objects (objects without clear visual boundaries) are rejected entirely. This verification step is a quality gate that ensures only high-quality annotations enter the training set.

Auto masklet generation. To increase annotation diversity and coverage beyond what human annotators naturally focus on (which tends to be salient, central objects), SAM 2 was used to automatically generate candidate masklets:

“To generate auto masklets, we prompt SAM 2 with a regular grid of points in the first frame and generate candidate masklets.”

Three grid densities are used: a 32×32 grid on the full frame, a 16×16 grid on four zoomed crops (from a 2×2 overlapped window), and a 4×4 grid on sixteen zoomed crops (from a 4×4 overlapped window). The zoomed crops target smaller objects that the full-frame grid might miss. Automatically generated masklets go through the same verification step as manual ones: those rated “satisfactory” are added to the dataset; those rated “unsatisfactory” (model failure cases) are sampled and presented to annotators for manual refinement in Phase 3.

Two post-processing steps are applied: tiny disconnected components (areas < 200 pixels) are removed, and small holes in masks (areas < 200 pixels) are filled. These clean up common artifacts without affecting the annotation’s structural validity.

Figure 9 visually demonstrates how automatic masklets increase coverage: the manual-only overlay shows sparse annotations primarily on salient objects; the manual+auto overlay shows dense annotations across the entire frame, including small background objects and parts.

Annotation protocol structure. The annotation task was divided into five steps, each performed by different annotators (Figure 11):

Steps 1–2: Object selection (identifying challenging, well-defined objects to annotate).
Steps 3–4: Masklet tracking (the actual annotation with SAM 2 in the loop).
Step 5: Quality verification (independent review).

This separation of concerns — selectors choose objects, annotators track them, verifiers check quality — prevents bias where the same person both creates and judges annotations. It also allows specialization: selectors are trained to find challenging, diverse objects (including small parts and occluded objects), while annotators focus on precise tracking.

Challenging object focus. To improve the model’s ability to “segment anything” — including difficult cases — annotators were instructed to find objects where SAM 2 required at least 2 edited frames. The number of edited frames serves as a proxy for difficulty (validated in Table 8: filtering for most-edited masklets yields better training data than random sampling). Annotators were also shown videos pre-filled with verified satisfactory automatic masklets and asked to find un-annotated challenging objects, ensuring that the manual annotations cover cases the automatic system missed.

Geographic and demographic diversity. The SA-V videos were recorded by crowdworkers across 47 countries (Figure 10b). The crowdworkers’ self-reported demographics (Figure 10c) show participation from 274 male and 236 female workers, with age distribution spanning 18–24 (109), 25–40 (305), and 41–64 (88). This geographic and demographic diversity is, as the paper states, intended to make the dataset “more representative on these factors than most of the publicly existing datasets of its kind at this time.”

4. Key Insights and Innovations

Innovation 1: Unifying Image and Video Segmentation Through Streaming Memory Eliminates the Interaction-Reset Problem

The distinction between segmenting an object in a frame and tracking it across frames has historically been treated as two separate problems requiring two separate architectures. Prior work on promptable video segmentation (Section 2) took a decoupled approach: SAM for per-frame mask generation, plus a specialized video tracker (XMem++, Cutie) for temporal propagation. The dominant assumption — implicit in the SAM+tracker baseline design — was that these two capabilities are sufficiently independent that they can be composed without loss.

SAM 2 challenges this assumption at an architectural level. By inserting a memory attention module between the image encoder and mask decoder, the model makes the segmentation decision conditional on interaction history — not just the current frame's pixels and the current frame's prompts, but the full context of what was previously prompted and what the object looked like in prior frames. The practical consequence is that refinement becomes incremental rather than restart-based. Figure 2 illustrates this concretely: a decoupled system requires re-annotating a lost object from scratch with multiple clicks; SAM 2 recovers it with a single click because the memory bank preserves the prior context that a click refers to the same tongue, not a new object.

This is more than an engineering convenience. It fundamentally reframes the interaction model for video segmentation. In the decoupled paradigm, the user's mental model must track two systems — "I am using SAM to define masks" and "I am using the tracker to propagate them" — with no continuity between correction events. In SAM 2's unified paradigm, the user interacts with a single system that maintains object identity across interactions. The paper's data engine results (Table 1) validate that this is not just conceptually cleaner but practically transformative: Phase 3 (unified SAM 2 with memory) achieves higher mask alignment scores (89.1% vs. 86.4%) than Phase 2 (decoupled SAM + SAM 2 Mask) while requiring fewer edited frames (19.04% vs. 23.25%) and fewer clicks per frame (2.68 vs. 3.61). The unified architecture produces better annotations with less human effort — a strong signal that the interaction paradigm itself, not just the underlying accuracy, has improved.

This innovation is best characterized as a fundamental architectural reframing rather than an incremental improvement. It does not add a tracking head to SAM or a prompt encoder to a VOS model; it reconceives the segmentation process as inherently memory-dependent, with image segmentation emerging naturally as the special case of an empty memory bank. The evidence that this reframing works bidirectionally — video training improves image segmentation (Table 5: 61.9 vs. 58.9 1-click mIoU on SA-23 when trained with the full data mix) — confirms that memory conditioning is not a video-specific add-on but a general mechanism that benefits spatial understanding even when the memory is unused.

Innovation 2: A Two-Level Memory Architecture Separates Short-Term Motion from Persistent Object Identity

Prior memory-based VOS models — XMem (Cheng & Schwing, 2022), AOT/DeAOT (Yang et al., 2021b; Yang & Yang, 2022), Cutie (Cheng et al., 2023a) — maintain a single type of memory that mixes recent-frame features with first-frame reference features, treating all stored frames as a homogeneous set for attention. SAM 2 introduces a conceptual distinction between two qualitatively different types of stored information, implemented through separate FIFO queues with different retention policies and different positional encoding treatments.

The first type — recent frame memories (up to N=6 frames, with temporal position encoding) — captures short-term motion and appearance continuity. These memories arrive with temporal metadata that allows the memory attention to learn motion patterns (the object is moving leftward, growing larger, rotating). They are transient: when a new frame arrives, the oldest recent memory is evicted.

The second type — prompted frame memories (up to M frames, without temporal position encoding) — captures what the object is, independent of when it was seen. These memories are retained indefinitely, serving as permanent reference points for object identity. The deliberate omission of temporal encoding from prompted frames is a subtle insight: if the model learned to associate "this is what the object looks like at time t=0" with a specific temporal position, it would fail when prompts arrive at arbitrary times during inference. By stripping temporal information from prompted memories, the model treats them as timeless exemplars that can be matched against regardless of temporal distance.

The third memory component — object pointers — adds a further level of abstraction. These are compact 256-dimensional semantic vectors from the mask decoder's output token, summarizing high-level "what kind of thing" information that complements the spatial feature maps. Their value is validated in the ablation (Table 11): they don't help on average across the 9 zero-shot benchmarks (where objects tend to be whole, salient entities) but significantly boost performance on SA-V (where many targets are parts and subparts) and LVOSv2 (where long videos tax identity preservation). This suggests object pointers are particularly valuable when the tracking task requires distinguishing semantically similar but distinct entities — a person's hat vs. their shirt — where spatial features alone might be ambiguous.

This is an architectural design innovation with theoretical motivation. The two-memory-type distinction maps onto a cognitively meaningful separation between "where the object is going" (recent frames, motion) and "what the object is" (prompted frames, identity). The negative GRU result (Table 11: adding a recurrent state transition provides no improvement) reinforces that the value comes from what is stored and how it's organized, not from learned dynamics over the memory state. This suggests future memory architectures should focus on structural priors (what to remember and for how long) rather than learned update rules.

Innovation 3: Test-Time Compute Substitution — 6× Faster Image Segmentation While Improving Accuracy

Scaling laws in computer vision have overwhelmingly favored larger models: ViT-H outperforms ViT-B, and scaling parameters is the default path to better accuracy. The original SAM used ViT-H as its largest and most accurate variant, achieving 58.1 1-click mIoU on the SA-23 benchmark at 21.7 FPS. A natural expectation would be that SAM 2, adding video capability and memory components, would be slower than SAM at image segmentation — more components to run, even if the memory is empty.

SAM 2 (Hiera-B+) inverts this expectation: it achieves 58.9 1-click mIoU on SA-23, exceeding SAM (ViT-H) by 0.8 points, while running at 130.1 FPS — a 6× speed improvement (Table 5). When trained on the full data mix including video data, the gap widens to 61.9 vs. 58.1 (3.8 points). The Hiera-L variant pushes accuracy further to 63.6 while still being 3.4× faster than SAM (ViT-H).

This is significant beyond the numbers because it challenges the implicit assumption that video-capable models must trade image accuracy for temporal capability. SAM 2 is not "SAM plus video" — it is a more efficient architecture for image segmentation even when the video components are idle. The speed gains come primarily from architectural improvements that are independent of the memory system: replacing ViT with Hiera, removing relative positional biases to enable FlashAttention-2, and using a hierarchical design that processes features more efficiently. The accuracy gains come partially from these architectural improvements and partially from the training data: video data provides additional supervision that improves segmentation on image benchmarks derived from video domains (Table 5: SA-23 Video improves from 56.4 to 60.1 when trained on the full mix; the 14 new video datasets improve from 56.6 to 69.6).

This is a practical efficiency innovation with a broader methodological implication: training on video data improves static image segmentation, especially for video-domain images. The joint image-video training strategy (alternating batches proportional to dataset sizes) is an implementation detail, but the finding that this improves both modalities suggests that temporal understanding and spatial understanding are not competing objectives — they are complementary, with temporal supervision providing a form of data augmentation that improves spatial feature quality.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper uses three distinct evaluation suites selected for different purposes. For promptable video segmentation (the PVS task), 9 densely annotated zero-shot datasets are used: EndoVis 2018, ESD, LVOSv2, LV-VIS, UVO, VOST, PUMaVOS, Virtual KITTI 2, and VIPSeg — all of which provide object segments annotated for every video frame (§F.1.1). For semi-supervised VOS evaluation, the benchmark expands to 17 zero-shot datasets, adding 8 sparsely annotated datasets: Wildfires, VISOR, FBMS, Ego-Exo4D, Cityscapes, Lindenthal Camera, HT1080WT Cells, and Drosophila Heart (§F.1.1). For image segmentation, the benchmark comprises 37 datasets — 23 previously used by SAM for zero-shot evaluation (including LVIS, ADE20K, Cityscapes, Hypersim, and 20 others) plus 14 new datasets sampled from video benchmarks like VOST, LV-VIS, FBMS, and Virtual KITTI 2 (§F.3.1). Additionally, SAM 2 is evaluated on established VOS benchmarks: MOSE val, DAVIS 2017 val and test, LVOS val, LVOSv2 val, YouTubeVOS 2019 val, and the newly introduced SA-V val and test sets (Table 6). The SA-V test set contains 278 masklets across 150 videos, and the SA-V val set contains 293 masklets across 155 videos, both annotated at 6 FPS using the Phase 1 data engine setup to ensure independence from SAM 2's biases (§5.2).
Base model(s). All primary experiments use SAM 2 with a Hiera-B+ image encoder at 1024×1024 resolution, trained on the full data mixture (SA-V manual + Internal + SA-1B, optionally with open-source VOS datasets). The Hiera-B+ variant is the default because it provides "a reasonable balance for speed and accuracy" (Table 9f), achieving 43.8 FPS on video tasks (Table 6). For speed-accuracy tradeoff analysis, four Hiera sizes are evaluated: Tiny (T), Small (S), Base+ (B+), and Large (L) — with the L variant providing the highest accuracy at 30.2 FPS (Table 6). The Hiera image encoder is initialized from MAE pretrained weights (He et al., 2022; Ryali et al., 2023). For baselines, the original SAM uses ViT-B and ViT-H backbones; HQ-SAM uses the same ViT variants; XMem++ and Cutie (the VOS trackers) use their released checkpoints; and the decoupled SAM+tracker baselines use SAM (ViT-H) for mask generation with XMem++ or Cutie for propagation (§6.1).
Metrics. For video segmentation, the primary metric is J &F (Pont-Tuset et al., 2017): the average of the Jaccard index J (intersection-over-union of predicted and ground-truth masks, measuring region similarity) and the F-measure F (measuring contour accuracy, the harmonic mean of contour precision and recall). J &F is reported on a per-dataset basis and averaged across datasets for aggregate comparisons. For YouTubeVOS 2019, the G metric is reported following that benchmark's protocol. For VOST, only the J metric is reported, following its official evaluation protocol. For LVOSv2, the paper reports Js, Fs, Ju, Fu (seen and unseen categories). For image segmentation, the metric is mIoU (mean intersection-over-union, averaged across all ground-truth instances in a dataset), reported at 1-click and 5-click settings. For the DAVIS interactive benchmark, AUC-J &F (area under the J &F vs. time curve) and J &F@60s (J &F at 60 seconds of annotation time) are reported. For the data engine phase comparison, Phase 1 Mask Alignment Score is defined as the percentage of masks whose IoU compared to the corresponding Phase 1 masks exceeds 0.75, stratified by mask size (small: 1 to 32² pixels, medium: 32² to 96², large: ≥ 96²).
Baselines. Five baseline categories are compared against:
1. SAM (Kirillov et al., 2023): the original Segment Anything model, evaluated with both ViT-B and ViT-H backbones, trained on SA-1B.
2. HQ-SAM (Ke et al., 2024): a SAM variant with a High-Quality output token trained on fine-grained masks (HQSeg-44K dataset), evaluated with ViT-B and ViT-H.
3. SAM + XMem++: a decoupled interactive baseline where SAM (ViT-H) converts click/box prompts to masks, and XMem++ (Bekuzarov et al., 2023) propagates them temporally. For multi-frame correction, SAM re-annotates the corrected frame using the XMem++ output mask as initialization before incorporating new clicks, following the strategy in EVA-VOS (Delatolas et al., 2024).
4. SAM + Cutie: analogous to SAM+XMem++ but using Cutie (Cheng et al., 2023a) as the tracker, modified to accept mask inputs on multiple frames.
5. State-of-the-art VOS models: XMem (Cheng & Schwing, 2022), Cutie-base and Cutie-base+ (Cheng et al., 2023a), SwinB-AOT (Yang et al., 2021b), SwinB-DeAOT (Yang & Yang, 2022), RDE (Li et al., 2022a), STCN (Cheng et al., 2021a), SimVOS-B (Wu et al., 2023b), JointFormer (Zhang et al., 2023b), ISVOS (Wang et al., 2022), DEVA (Cheng et al., 2023b), and DDMemory (Hong et al., 2023) — all evaluated on standard VOS benchmarks with ground-truth mask input on the first frame (Table 6, Table 17).
Generation budget / compute accounting. For interactive video evaluation, the budget is measured in number of interacted frames (Nframe) and number of clicks per frame (Nclick). The offline evaluation makes up to 8 passes through the video, selecting the frame with the lowest IoU for corrective prompts in each pass, with 3 clicks per frame (Figures 5, 12). The online evaluation makes a single forward pass, pausing when IoU < 0.75 to add 3 corrective clicks, up to 8 prompted frames (Figures 5, 13). For semi-supervised VOS, the budget is 1, 3, or 5 clicks on the first frame, or a bounding box, or a ground-truth mask (Table 4). The paper also reports annotation time, modeled as Texam·(L/300) + (Tloc + Tclick·Nclick)·Nframe where Texam = 30 sec for a 300-frame video to examine results at 10 FPS, Tloc = 1 sec to visually locate an object, and Tclick = 1.5 sec per click, with L being video length (§F.1.2). For image segmentation, the budget is 1-click and 5-click settings, with FPS measured on a single A100 GPU using batch size 10 (Table 5, following the SA task protocol). For data engine comparison, annotation time per frame is measured directly in a controlled experiment with 169 videos and 452 masklets (Table 1). No FLOPs-matched pretraining vs. inference comparison is conducted in this paper — the compute accounting is purely for inference-time efficiency, not training-inference tradeoffs.
Cross-validation / statistical protocol. The paper does not employ k-fold cross-validation for strategy selection as in the reference example; instead, results are reported as direct comparisons on fixed test sets. For the SA-V benchmark, the test and validation sets were created by splitting based on video authors and geographic locations "to ensure minimal overlap of similar objects" (§5.2). For the data engine phase comparison (Table 1), a controlled experiment was conducted: the same set of 169 videos and 452 masklets was annotated by three separate groups of annotators, each using the protocol of one phase, enabling direct comparison of annotation time, click count, and mask quality. No confidence intervals or statistical significance tests are reported. The speed benchmarking (§D.3) is conducted on a single A100 GPU using PyTorch 2.3.1 and CUDA 12.1 with automatic mixed precision (bfloat16) and torch.compile for the image encoder, with batch size 10 for images and batch size 1 for videos (following common video segmentation protocol).

Main Quantitative Results

5.1 Promptable Video Segmentation: Interactive Offline and Online Evaluation

Headline result: SAM 2 achieves better segmentation accuracy while using more than 3× fewer interactions than SAM-plus-tracker baselines, and the advantage holds across all 9 densely annotated zero-shot datasets in both offline and online settings.

Figure 5a (offline evaluation, 3 clicks per frame) shows SAM 2 starting at approximately 75 J &F (averaged across 9 datasets) with 1 annotated frame, rising to approximately 80 J &F at 8 annotated frames. SAM+XMem++ starts at approximately 68 J &F (1 frame) and rises to 72 J &F (8 frames). SAM+Cutie starts at approximately 70 J &F and rises to 75 J &F. The gap between SAM 2 and the best baseline (SAM+Cutie) is approximately 5 J &F points at 1 frame and grows to approximately 5–6 points at 8 frames — meaning SAM 2 with 1 annotated frame roughly matches SAM+Cutie with 3–4 annotated frames, representing a 3–4× reduction in required interactions.

Figure 5b (online evaluation) shows a similar pattern: SAM 2 starts at approximately 72 J &F (1 frame), rising to 80 J &F (8 frames), compared to SAM+Cutie at approximately 68 J &F (1 frame) and 74 J &F (8 frames). The advantage is slightly larger in online mode, likely because SAM 2's memory preserves context across the forward pass, whereas decoupled baselines have no mechanism to refine earlier frames that have already been passed.

Per-dataset results (Figures 12 and 13) demonstrate that SAM 2's advantage is not driven by a few datasets — it outperforms both baselines on all 9 datasets at 8 interacted frames, with the largest margins on the most challenging benchmarks:

VOST (large transformations): SAM 2 achieves 67.5 J &F vs. 54.3 for SAM+Cutie and 46.6 for SAM+XMem++ in offline mode (Figure 12b); 65.0 vs. 54.6 vs. 52.7 in online mode (Figure 13b).
LVOSv2 (long videos): SAM 2 achieves 87.9 vs. 82.1 (SAM+Cutie) and 72.1 (SAM+XMem++) offline.
PUMaVOS (parts): SAM 2 achieves 68.5 vs. 59.4 (SAM+Cutie) and 60.2 (SAM+XMem++) offline.
On easier datasets like VIPSeg and LV-VIS, the margins are smaller (2–4 points) but consistently in SAM 2's favor.

The annotation-time plots (Figures 12a and 13a) translate these frame-count advantages into estimated time savings: SAM 2 reaches 75 J &F at approximately 60 seconds on LVOSv2, while SAM+Cutie requires approximately 120 seconds — a 2× time saving for equivalent accuracy.

5.2 Semi-Supervised Video Object Segmentation (VOS)

Headline result: With only first-frame prompts, SAM 2 outperforms specialized VOS models on established benchmarks, and the performance gap is largest on the new SA-V benchmark, demonstrating that prior models are far from "segment anything in videos" capability.

Table 4 reports average J &F across 17 zero-shot video datasets under five prompt types:

Prompt type	SAM+XMem++	SAM+Cutie	SAM 2
1-click	56.9	56.7	64.7
3-click	68.4	70.1	75.3
5-click	70.6	72.2	77.6
Bounding box	67.6	69.4	74.4
Ground-truth mask	72.7	74.1	79.3

SAM 2's advantage is largest under the sparsest supervision (1-click: +7.8 points over the best baseline), and the gap narrows but remains substantial with full mask supervision (ground-truth mask: +5.2 points over Cutie). This pattern — larger advantage with weaker prompts — directly reflects the value of the unified architecture: decoupled baselines rely on SAM to produce a high-quality mask from sparse clicks before tracking, and SAM's click-to-mask accuracy on video frames limits their performance. SAM 2's memory attention can use temporal context to resolve click ambiguity, producing better masks from fewer clicks.

Per-dataset results (Figure 14) show SAM 2 outperforming on the majority of the 17 datasets across all prompt types. The exceptions tend to be on datasets with specific characteristics where specialized trackers have advantages — for instance, on some driving datasets where Cutie was heavily optimized.

Table 6 reports results on standard VOS benchmarks with ground-truth mask input on the first frame:

Method	MOSE val	DAVIS 2017 val	LVOS val	SA-V val	SA-V test	YTVOS 2019 val
Best prior (varies)	71.7 (Cutie+)	90.1 (JointFormer)	66.0 (Cutie)	61.4 (SwinB-DeAOT)	62.8 (Cutie+)	87.5 (Cutie+)
SAM 2 (Hiera-B+)	76.6	90.2	78.0	76.8	77.0	88.6
SAM 2 (Hiera-L)	77.9	90.7	78.0	77.9	78.4	89.3

The most striking result is on the SA-V benchmark: prior methods cluster around 60–63 J &F on SA-V val and 50–63 on SA-V test, while SAM 2 (Hiera-B+) achieves 76.8 and 77.0 respectively — a ~15-point improvement over the best prior method. This gap quantifies the distance between specialized VOS models (trained on restricted object categories) and the "segment anything in videos" capability that SAM 2 targets. The SA-V benchmark contains small objects, parts, and diverse categories not present in standard VOS training data, and prior models — even state-of-the-art ones — fail to generalize to this open-world setting.

On DAVIS 2017, SAM 2 (Hiera-B+) achieves 90.2 J &F, slightly exceeding JointFormer's 90.1 — notable because DAVIS is the most saturated benchmark where prior work has pushed performance near ceiling. On MOSE, SAM 2 achieves 76.6, a +4.9 point improvement over Cutie-base+ (71.7). On LVOS (long videos), SAM 2 achieves 78.0 vs. Cutie's 66.0 — a +12 point improvement, suggesting the memory architecture is particularly effective for long-term tracking.

Table 17 provides detailed breakdowns including J and F components separately, as well as additional metrics for LVOSv2 (Js, Fs for seen categories; Ju, Fu for unseen) and YouTubeVOS (G metric). Across all sub-metrics, SAM 2 variants consistently outperform prior work. The "‡" variants in Table 17 represent models trained only on SA-1B, SA-V, and Internal data (without open-source VOS datasets), showing that the proprietary dataset alone is sufficient to achieve state-of-the-art results — the inclusion of DAVIS, MOSE, and YouTubeVOS in training (the main SAM 2 rows) provides additional gains but is not necessary for superiority.

5.3 Image Segmentation

Headline result: SAM 2 is more accurate than the original SAM on image segmentation while being 6× faster, and training on video data further improves image segmentation accuracy, particularly on video-domain images.

Table 5 reports 1-click and 5-click mIoU across 37 datasets, grouped by domain:

Model	Training Data	SA-23 All	SA-23 Image	SA-23 Video	14 new Video	FPS
SAM (ViT-H)	SA-1B	58.1 (81.3)	60.8 (82.1)	54.5 (80.3)	59.1 (83.4)	21.7
HQ-SAM (ViT-H)	HQSEG-44k	59.1 (79.8)	61.8 (80.5)	55.7 (78.9)	58.9 (81.6)	21.4
SAM 2 (Hiera-B+)	SA-1B only	58.9 (81.7)	60.8 (82.1)	56.4 (81.2)	56.6 (83.7)	130.1
SAM 2 (Hiera-L)	SA-1B only	60.0 (81.8)	62.0 (82.2)	57.4 (81.2)	58.5 (83.8)	61.4
SAM 2 (Hiera-B+)	full mix	61.9 (83.5)	63.3 (83.8)	60.1 (83.2)	69.6 (85.8)	130.1
SAM 2 (Hiera-L)	full mix	63.6 (83.5)	64.7 (83.7)	62.2 (83.2)	71.1 (85.7)	61.4

Several patterns emerge:

Image-only comparison (SA-1B training, rows 1–4): SAM 2 (Hiera-B+) achieves 58.9 1-click mIoU on SA-23 vs. SAM (ViT-H)'s 58.1 — a +0.8 point improvement while being 6× faster. On 5-click, SAM 2 achieves 81.7 vs. 81.3. The advantage comes primarily from the more efficient Hiera backbone and architectural improvements (removal of RPB, hierarchical design) rather than from video training — this row shows pure architectural efficiency gains. SAM 2 (Hiera-L) further improves to 60.0 1-click mIoU (still 2.8× faster than SAM ViT-H at 61.4 FPS).

Video data benefit (rows 4–5 vs. rows 2–3): Training on the full data mix (SA-V + Internal + SA-1B) provides an additional +3.0 points for Hiera-B+ (58.9 → 61.9 on SA-23 All) and +3.6 points for Hiera-L (60.0 → 63.6). This improvement is most dramatic on video-domain images: SA-23 Video improves from 56.4 → 60.1 (+3.7) and the 14 new video datasets improve from 56.6 → 69.6 (+13.0). This demonstrates that video training data provides a form of domain adaptation — images sampled from video distributions (with motion blur, lower resolution, compression artifacts) benefit substantially from having seen similar frames during video training. The 14 new video datasets show the largest gain because SAM 2's training distribution (video frames) matches their test distribution.

Per-dataset deltas (Figure 15): SAM 2 (Hiera-B+, full mix) outperforms SAM (ViT-H) on 29 out of 37 datasets. The largest improvements are on video-derived datasets: EndoVis 2018 (+41.1 1-click mIoU — from surgical videos), Drosophila Heart OCM (+19.7 — microscopy), FBMS (+18.1 — motion segmentation). The few datasets where SAM 2 underperforms SAM show small deltas: PPDLS (-6.6), NDISPark (-4.8), HT108WT (-3.4), Cityscapes (-1.7), TimberSeg (-0.6). These are primarily static image datasets with characteristics well-covered by SA-1B — the video training provides less benefit and may slightly shift the feature distribution away from these domains.

Speed comparison: The FPS measurements use batch size 10 on a single A100 GPU with torch.compile and bfloat16 precision (§D.3). SAM 2 (Hiera-B+) at 130.1 FPS is 6.0× faster than SAM (ViT-H) at 21.7 FPS. SAM 2 (Hiera-L) at 61.4 FPS is 2.8× faster than SAM (ViT-H). Even SAM 2 (Hiera-L) with the full data mix achieves 63.6 mIoU at 61.4 FPS — higher accuracy at nearly 3× the speed of SAM's best model.

Table 15 extended comparison: Includes HQ-SAM baselines and a broader sweep of SAM 2 variants. HQ-SAM (ViT-H) achieves 59.1 1-click mIoU on SA-23 — slightly higher than SAM (58.1) and SAM 2 image-only (58.9) but lower than SAM 2 with full mix (61.9). On 5-click, HQ-SAM underperforms (79.8 vs. SAM's 81.3 and SAM 2's 83.5), likely because HQ-SAM's fine-grained mask training improves first-click accuracy on small objects but does not benefit iterative refinement as much. The trend is clear: SAM 2 with video training dominates in both 1-click and 5-click settings.

5.4 Data Engine Efficiency and Scale

Headline result: SAM 2 in the loop achieves 8.4× faster annotation than per-frame SAM while producing higher-quality masks, and the SA-V dataset is 53× larger than any prior video segmentation dataset by mask count.

Table 1 reports the controlled experiment comparing the three data engine phases on the same 169 videos and 452 masklets:

Phase	Method	Time/Frame	Edited Frames	Clicks/Clicked Frame	Mask Alignment (All / Small / Medium / Large)
1	SAM only	37.8 s	100.00%	4.80	— (reference)
2	SAM + SAM 2 Mask	7.4 s	23.25%	3.61	86.4% / 71.3% / 80.4% / 97.9%
3	SAM 2	4.5 s	19.04%	2.68	89.1% / 72.8% / 81.8% / 100.0%

Phase 3 with SAM 2 is 8.4× faster than Phase 1 (4.5 vs. 37.8 seconds per frame) while achieving 89.1% Mask Alignment (89.1% of masks have IoU > 0.75 with the high-quality Phase 1 reference). Critically, Phase 3 is both faster and higher quality than Phase 2 — disproving any speed-quality tradeoff. The reduction in edited frames (23.25% → 19.04%) and clicks per frame (3.61 → 2.68) quantifies the interaction efficiency gain: annotators need to correct fewer frames, and when they do correct, they use fewer clicks because SAM 2's memory preserves context.

The mask alignment is stratified by object size, showing that the largest improvement from Phase 2 to Phase 3 occurs for small objects (71.3% → 72.8%) and medium objects (80.4% → 81.8%), while large objects are already near-ceiling (97.9% → 100.0%). This aligns with the model architecture: memory attention and object pointers are most valuable for challenging cases (small, easily confused objects), while large salient objects are successfully tracked by simpler propagation mechanisms.

Training data scaling (Table 2): Adding data from each phase progressively improves accuracy on both in-domain (SA-V val) and out-of-domain (9 zero-shot) benchmarks:

Training Data	SA-V val J &F	9 zero-shot J &F
Baseline (VOS datasets + SA-1B)	50.0	62.5
+ Phase 1 (16K masklets)	53.0 (+3.0)	66.9 (+4.4)
+ Phase 2 (63.5K masklets)	58.8 (+5.8)	70.9 (+4.0)
+ Phase 3 (197.0K masklets)	62.5 (+3.7)	71.2 (+0.3)
+ Auto (451.7K masklets)	63.2 (+0.7)	71.5 (+0.3)

The diminishing returns from Phase 3 and Auto masklets (smaller incremental gains) suggest that the data scaling might be approaching saturation for the Hiera-B+ model size at this training duration (200K iterations fixed across rows). However, Figure 6 shows that when training data quantity is varied more finely (subsampling SA-V to different sizes), a power-law relationship emerges between data quantity and video segmentation accuracy on all three benchmarks (SA-V val, zero-shot, MOSE dev), with no clear saturation at the maximum data size. This suggests that the fixed 200K iteration budget, not the data quantity, may be the limiting factor in Table 2 — with more training, the larger datasets would likely yield larger gains.

Data quality ablation (Table 8): Training on 50K randomly sampled masklets vs. 50K most-edited masklets (with the most annotator corrections) vs. all 190K SA-V masklets:

Setting	SA-V val J &F	MOSE dev J &F	9 zero-shot J &F
50K random	63.7	72.3	68.7
50K most edited	66.2 (+2.5)	72.5 (+0.2)	69.2 (+0.5)
Full 190K	69.9 (+3.7)	73.9 (+1.4)	70.8 (+1.6)

The most-edited subset of 50K (25% of the data) significantly outperforms random 50K, particularly on SA-V val (+2.5 J &F). This validates the annotator strategy of focusing on challenging objects — the "edited frames" count is an effective proxy for sample difficulty, and training on harder examples yields better generalization. However, using all 190K masklets provides substantial additional gains over the filtered subset, indicating that easy examples also contribute useful training signal (perhaps for learning basic propagation before tackling hard cases).

Ablation Studies and Robustness Checks

All ablations in this section use a reduced setup: 512² spatial resolution, trained on SA-V manual and a 10% subset of SA-1B, with a smaller model configuration unless otherwise specified. The default configuration for each ablation is highlighted in gray in Tables 9–11. Results are reported on SA-V val, Internal-test, MOSE dev (200 randomly sampled videos from MOSE training split, excluded from SAM 2's training data), 9 zero-shot video datasets, and SA-23 (image benchmarks). The primary metric is J &F under 3-click input on the first frame for video tasks and 1-click mIoU for image tasks (§A).

Data Ablations

Data mixture (Table 7): Training on existing VOS datasets alone (DAVIS, MOSE, YouTubeVOS — row 1) produces strong in-domain MOSE dev performance (76.9 J &F) but poor generalization to SA-V val (48.1), Internal-test (60.2), and 9 zero-shot benchmarks (59.7). Adding SA-V data (row 3 vs. row 1) improves SA-V val by +14.9 points (48.1 → 63.0) and 9 zero-shot by +10.0 points (59.7 → 69.7). The best results come from combining all sources: VOS + SA-V + Internal + SA-1B (row 12) achieves 63.1 on SA-V val, 73.7 on Internal-test, 79.0 on MOSE dev, 71.6 on 9 zero-shot, and 58.9 mIoU on SA-23. Notably, SA-1B images improve image segmentation without degrading video performance (row 11 vs. 12: SA-23 improves from 55.7 to 58.9, while video metrics are essentially unchanged — 61.8 vs. 63.1 on SA-V val, 71.8 vs. 71.6 on 9 zero-shot). Training only on SA-V and SA-1B (row 4) achieves strong performance everywhere except MOSE (73.6 vs. 79.0 with VOS data), confirming that specific object categories in VOS datasets (people, vehicles, animals) provide complementary coverage to the open-world SA-V annotations.

Data quantity scaling (Figure 6): The three plots show J &F as a function of SA-V masklet count (log scale, from 5K to 200K) on SA-V val, 9 zero-shot, and MOSE dev. All three exhibit consistent power-law scaling: accuracy improves linearly with log data quantity, with no visible plateau. For SA-V val, J &F rises from approximately 58 at 5K masklets to approximately 68 at 200K. For 9 zero-shot, from approximately 66 to approximately 73. For MOSE dev, from approximately 69 to approximately 75. The slopes are similar across benchmarks, suggesting the scaling behavior is a property of the model architecture and training procedure rather than dataset-specific.

Model Architecture Ablations

Input resolution (Table 9a): Increasing resolution from 512² to 768² to 1024² consistently improves accuracy: MOSE dev J &F goes from 73.0 → 76.1 → 77.0, and SA-23 mIoU from 59.7 → 61.0 → 61.5. The 9 zero-shot average improves from 70.7 → 72.5 at 768² but slightly decreases to 72.3 at 1024², suggesting some zero-shot datasets may not benefit from higher resolution (perhaps due to low native video resolution). Speed drops proportionally: 0.43× at 768² and 0.22× at 1024² relative to 512².

Number of training frames (Table 9b): Increasing from 4 to 8 frames substantially improves video metrics: SA-V val from 60.0 → 68.3 (+8.3 J &F), 9 zero-shot from 67.7 → 70.7 (+3.0). Further increasing to 10 frames provides marginal additional gains (MOSE: 74.5 vs. 73.0, SA-V val: 68.1 vs. 68.3) without speed penalty (all at 1.00×). Image segmentation (SA-23 mIoU) is unaffected by frame count (59.7–60.1 across 4–10 frames), confirming the modularity of image and video components.

Number of memories N (Table 9c): Memory bank capacity shows diminishing returns: N=4 achieves 73.5 MOSE dev J &F vs. 73.0 for N=6 and 73.2 for N=8. SA-V val peaks at N=8 (69.0 vs. 68.3/68.6) but the gain is small. Speed decreases from 1.01× (N=4) to 0.93× (N=8). The default N=6 balances temporal context with computational cost.

Memory channel dimension (Table 9d): Reducing memory channels from 256 to 64 has minimal impact on video metrics (MOSE: 73.4 vs. 73.0; 9 zero-shot: 70.0 vs. 70.7; SA-V val: 66.4 vs. 68.3) while making storage 4× smaller and speed nearly identical (0.92× vs. 1.00×). This is a critical efficiency finding: the memory attention can extract sufficient information from low-dimensional memory features, and the channel reduction is essentially "free" in accuracy terms.

Memory attention capacity (Table 9e): Varying the number of self-attention and cross-attention blocks (#sa, #ca): (4, 4) — the default — achieves 73.0 MOSE dev, 68.3 SA-V val, 70.7 9 zero-shot. Reducing to (3, 2) drops SA-V val to 64.1 (−4.2) and 9 zero-shot to 69.5 (−1.2). Reducing further to (2, 2) — no cross-attention reduction but fewer self-attention layers — drops SA-V val to 67.3 (−1.0). Image segmentation (SA-23) is unaffected (59.7–60.0 across all settings), confirming that memory attention only affects video tasks. The (4, 4) default is chosen for best video performance at acceptable speed (1.00×).

Image encoder size (Table 9f): Scaling the Hiera encoder from S → B+ → L consistently improves accuracy: MOSE dev from 70.9 → 73.0 → 75.0; SA-23 from 57.8 → 59.7 → 61.1. SA-V val shows a non-monotonic pattern (65.5 → 68.3 → 66.3), with the L variant surprisingly underperforming B+ on SA-V val (66.3 vs. 68.3). The 9 zero-shot improves monotonically (69.4 → 70.7 → 71.9). Speed decreases from 1.33× (S) to 1.00× (B+) to 0.60× (L). The B+ encoder is chosen as the default for the best speed-accuracy balance.

Relative positional encoding (Table 10): Three configurations compared: RPB in image encoder (SAM's original approach, row 1), RPB removed + 2D-RoPE in memory attention (row 2), and RPB removed + no RoPE (row 3). Row 2 (the default) achieves the best video metrics: MOSE 73.6, 9 zero-shot 71.5, LVOSv2 71.0, at 0.93× speed. Row 3 (no RoPE) drops MOSE to 72.8 (−0.8), 9 zero-shot to 70.3 (−1.2), LVOSv2 to 70.3 (−0.7) but is faster (1.04×). Row 1 (RPB + 2D-RoPE) achieves 73.0 MOSE, 68.3 SA-V val, 71.6 LVOSv2 at 1.00× speed. The removal of RPB from the image encoder is critical for enabling FlashAttention-2, which provides significant speedup at 1024² resolution — the speed gap between rows 1 and 2 is small at 512² (this ablation) but large at 1024² (the final training resolution).

Recurrent memory (GRU) and object pointers (Table 11): Two memory architecture ablations:

GRU (row 2 vs. row 1): Adding a GRU to process memory features before storage provides no overall improvement: MOSE drops from 73.1 to 72.3 (−0.8), 9 zero-shot drops from 70.9 to 70.5 (−0.4), with a slight speed penalty (0.97×). LVOSv2 improves from 67.0 to 68.9 (+1.9), suggesting GRU might help long-term tracking but at the cost of general performance. The authors choose the simpler non-GRU design.
Object pointers (row 3 vs. row 1): Adding object pointer cross-attention improves SA-V val from 64.5 to 68.3 (+3.8) and LVOSv2 from 67.0 to 71.6 (+4.6), while having minimal impact on MOSE (73.0 vs. 73.1) and 9 zero-shot (70.7 vs. 70.9). This confirms that object pointers are particularly valuable for part-level and long-term segmentation — exactly the scenarios where semantic identity information supplements spatial features. Speed is unchanged (1.00×).

Fairness Evaluation

Demographic fairness (Table 13): Evaluated on 20-second clips from the Ego-Exo4D dataset (Grauman et al., 2023), which contains self-reported demographic information from the video subjects. The evaluation uses 1-click, 3-click, and ground-truth mask on the first frame, measuring J &F:

	1-click	3-click	Mask
Male	81.9	95.1	95.9
Female	75.1	94.1	95.2
Age 18–26	77.2	95.0	95.7
Age 26–50	76.7	94.7	95.8
Age 50+	81.4	95.1	96.2

At 3 clicks and with ground-truth mask prompts, discrepancies are minimal (≤1.0 J &F difference across all groups). At 1 click, a larger gap appears between male (81.9) and female (75.1), a 6.8-point difference. The paper investigates this: "We manually inspect 1 click predictions, and find the model frequently predicts the mask for a part instead of the person." When limiting the comparison to clips where the person is correctly segmented (rather than a body part), the gap shrinks substantially: male 94.3 vs. female 92.7 J &F (a 1.6-point difference). This suggests the 1-click discrepancy is primarily attributable to ambiguity resolution — given a single click on a person, the model sometimes segments a part (arm, clothing) rather than the whole person, and this tendency may correlate with demographic attributes (e.g., clothing differences, pose differences). At 3 clicks, the additional clicks resolve the ambiguity, equalizing performance. This is a nuanced fairness finding: the model's underlying segmentation capability is equitable, but its default ambiguity resolution (which mask to select when multiple are valid) shows demographic variation.

Critical Assessment

Claim 1: "SAM 2 delivers better accuracy using 3× fewer interactions than prior approaches"

What was tested: Interactive offline and online evaluation on 9 zero-shot datasets, comparing SAM 2 against SAM+XMem++ and SAM+Cutie with 3 clicks per interacted frame, measuring J &F at 1–8 annotated frames (Figure 5, Figures 12–13).

What the experiments show: SAM 2 with 1 annotated frame (~75 J &F offline) roughly matches SAM+Cutie with ~3 annotated frames, and SAM 2 with 2 annotated frames (~77 J &F) exceeds SAM+Cutie at 8 annotated frames (~75 J &F). This approximately supports the "3× fewer interactions" claim when interpreting "interactions" as annotated frames. The advantage is consistent across all 9 datasets, with the largest margins on challenging benchmarks (VOST, LVOSv2, PUMaVOS).

What limits the evidence:

The "3×" figure is a rough aggregate over interactive rounds, not a precise metric. The exact ratio depends on which accuracy level is targeted and which baseline is compared against. For example, achieving 80 J &F (offline) requires ~6 annotated frames for SAM 2 vs. >8 for baselines — closer to 1.3× fewer, not 3×. The claim is best understood as qualitative ("substantially fewer interactions") rather than a fixed multiplier.
The baselines (SAM+XMem++, SAM+Cutie) use SAM (ViT-H) for mask generation, the strongest SAM variant. However, SAM 2 uses Hiera-B+ or Hiera-L — the comparison includes both model architecture improvements and the memory mechanism. An ablation using SAM 2's image encoder in the decoupled baseline (e.g., SAM 2 image-only + XMem++) is not reported, making it impossible to separate how much of the gain comes from the better image encoder vs. the unified memory architecture.
The click strategy uses 3 clicks per frame, sampled from error region centers. Real users might distribute clicks differently (e.g., one click per frame on more frames rather than three on fewer frames). The 3-click protocol matches prior work (Delatolas et al., 2024) but may not represent all interaction patterns.
The annotation time model (§F.1.2) assumes 10 FPS examination speed, 1 sec for visual object localization, and 1.5 sec per click — these are reasonable estimates but not empirically validated with user studies. Real annotation time includes cognitive decision-making (deciding which frame to correct, assessing whether the current masklet is correct) that this model abstracts away.

Claim 2: "SAM 2 outperforms prior work in established video object segmentation benchmarks"

What was tested: Semi-supervised VOS on MOSE val, DAVIS 2017 val/test, LVOS val, LVOSv2 val, YouTubeVOS 2019 val, and SA-V val/test, with first-frame ground-truth mask input (Tables 6, 17).

What the experiments show: SAM 2 (Hiera-B+) achieves state-of-the-art on all benchmarks, with margins ranging from +0.1 J &F on DAVIS 2017 val (vs. JointFormer's 90.1) to +15.7 on SA-V val (vs. SwinB-DeAOT's 61.4). The improvement is largest on SA-V, which contains open-world objects and parts, and on LVOS/LVOSv2 (long videos), suggesting SAM 2's advantages are most pronounced in regimes outside standard VOS training distributions.

What limits the evidence:

Comparing against methods with different training data. SAM 2 is trained on SA-V (35.5M masks) plus Internal data, while prior VOS methods are trained on standard VOS datasets (DAVIS, MOSE, YouTubeVOS — collectively a few hundred thousand masks). The SA-V benchmark results (76.8 vs. 61.4) primarily measure the value of the new training data, not architectural superiority. The paper partially addresses this by showing that SAM 2 trained only on standard VOS datasets (Table 7, row 1) achieves strong MOSE dev performance (76.9) but no SA-V comparison is reported for this configuration — it would likely underperform, confirming that the SA-V benchmark is heavily data-dependent.
The DAVIS 2017 result (90.2 vs. 90.1) is within noise. DAVIS 2017 val has 30 videos; a 0.1 J &F difference on this small set is not statistically robust. The more meaningful DAVIS result is on the test set (Table 17d): SAM 2 (Hiera-L) achieves 88.9 vs. Cutie-base+'s 88.1 — a more decisive but still modest +0.8 margin on this saturated benchmark.
JointFormer achieves 90.1 on DAVIS 2017 val (Table 6) vs. SAM 2's 90.2 — but JointFormer also achieves 87.4 on YouTubeVOS val G metric, close to SAM 2's 88.6. The comparison to JointFormer is limited because JointFormer results are not available for most other benchmarks (Table 6 shows "—" for MOSE, LVOS, SA-V, SA-V test), preventing a comprehensive head-to-head.
No evaluation on recent VOS challenge sets. Benchmarks like the DAVIS 2020/2021 challenge, the YouTubeVOS 2021/2022 challenge, or the VOT (Visual Object Tracking) benchmarks are absent. These newer benchmarks often include more challenging scenarios (multi-object interaction, long-term tracking) and could reveal failure modes not visible on the standard 2017/2019 sets.

Claim 3: "SAM 2 is more accurate and 6× faster than SAM on image segmentation"

What was tested: 1-click and 5-click mIoU on 37 zero-shot image datasets (Table 5, Table 15, Figure 15), with FPS measured on a single A100 GPU using batch size 10, torch.compile, and bfloat16.

What the experiments show: SAM 2 (Hiera-B+), trained on SA-1B only, achieves 58.9 1-click mIoU vs. SAM (ViT-H)'s 58.1, at 130.1 FPS vs. 21.7 FPS — 6× faster. Trained on the full data mix, SAM 2 achieves 61.9 mIoU (+3.8 over SAM). The speed comparison is clean (same hardware, same batch size, same precision, both models compiled with torch.compile).

What limits the evidence:

The image encoder comparison is Hiera vs. ViT, not SAM architecture vs. SAM 2 architecture. SAM 2's image segmentation uses an empty memory bank — the memory attention and memory encoder are idle. The image segmentation accuracy difference is almost entirely attributable to the Hiera backbone being more efficient than ViT, not to video-specific architectural innovations. This doesn't invalidate the claim (SAM 2 is faster and more accurate), but it reframes it: the improvement comes from a better image encoder design, not from the unified image-video architecture.
The "6× faster" includes torch.compile. The paper notes (§D.3): "We compile the image encoder with torch.compile for all SAM 2 models and do the same for SAM and HQ-SAM for direct comparison." This is a fair comparison (all models get compilation), but the absolute FPS numbers reflect compiled rather than eager-mode inference. For deployment scenarios where compilation is unavailable or impractical, the speed ratios might differ.
Batch size 10 for images. The FPS measurement uses batch size 10 for the image task (to maximize throughput), but batch size 1 for video tasks (following convention). For interactive image segmentation where the user clicks and waits, batch size 1 latency might be more relevant than batch size 10 throughput. The paper does not report batch-1 image FPS, making the "6× faster" measurement representative of batched throughput rather than single-image interactive latency.
The video training benefits are distribution-matched, not universal. The largest image segmentation gains from video training are on video-derived image benchmarks (69.6 vs. 56.6 on the 14 new video datasets), while standard image benchmarks see more modest gains (63.3 vs. 60.8 on SA-23 Image). On some purely static image datasets, SAM 2 underperforms SAM (Figure 15: TimberSeg -0.6, NDD20 +4.0, DRAM +11.7 — the distribution is wide). The "more accurate" claim holds on average (29/37 datasets) but not universally — users with specific static-image domains should verify whether SAM 2's video training helps or hurts their use case.

Claim 4: "SA-V is 53× larger than any existing video segmentation dataset by mask count"

What was tested: Comparison of SA-V (Manual+Auto) statistics against open-source VOS datasets (Table 3).

What the experiments show: SA-V Manual+Auto contains 35.5M masks. The largest prior dataset (BURST) contains 600K masks. The ratio is 35.5M / 600K ≈ 59× — but the paper cites 53× comparing against 667K (UVO-dense). The claim is numerically accurate under the specific comparison chosen.

What limits the evidence:

Counting masks vs. mask quality. The SA-V auto masklets (451.7K masklets, 25.5M masks) are generated automatically and verified by annotators as "satisfactory." However, the verification step only classifies masklets as satisfactory/unsatisfactory — it does not measure mask quality on a continuous scale. An automatically generated masklet rated "satisfactory" may still be lower quality than a manually annotated masklet, but it counts equally in the mask count.
Only 15× larger without auto annotations. SA-V Manual alone contains 190.9K masklets and 10.0M masks — still 15× larger than UVO-dense (667K masks) by mask count, but the gap is smaller by masklet count (190.9K vs. 16.1K for BURST). The paper is transparent about this distinction (Table 3 labels it separately), but the headline "53×" figure depends on auto-generated masks.
The Internal dataset is larger but not released (62.9K videos, 5.4M masks). The Internal data provides additional training signal but is not available for reproduction. The paper reports results with and without Internal data (§A.1), showing that the released SA-V + SA-1B combination already achieves strong performance, but the best reported numbers (the main results in Sections 6–7) include Internal data. This limits full reproducibility.

Missing Experiments That Would Strengthen the Paper

User study with real annotators. All interactive evaluation is simulation-based: clicks are sampled algorithmically from error regions. A user study with real annotators completing segmentation tasks would validate whether the simulated interaction protocol accurately reflects real-world usage, and whether the "3× fewer interactions" claim translates to actual time savings and user satisfaction.
SAM 2 image-only training without video components. The image segmentation comparison (Table 5) trains SAM 2 on SA-1B with the full architecture (including memory attention and memory encoder), even though those components are idle for images. Training a version without memory attention/encoder (purely the Hiera backbone + SAM-style mask decoder) would isolate whether the architectural overhead of video components penalizes image training — or whether, as the paper implies, the empty memory attention effectively becomes an identity function that doesn't hurt.
Scaling beyond Hiera-L. The paper scales the image encoder from T → S → B+ → L but does not explore larger models (e.g., Hiera-H or ViT-based backbones). The data scaling curves (Figure 6) show no saturation at 200K masklets, suggesting that model scaling might further improve performance. An experiment with a larger backbone on the full dataset would indicate whether the current models are data-limited or capacity-limited.
Combined PRM search with revisions analog. The reference example paper combined test-time compute strategies and found complementary benefits. SAM 2 could similarly combine with dedicated post-processing (CRF refinement, test-time augmentation, ensemble of memory attention variants) — but no such combinations are explored. The memory attention's sensitivity to hyperparameters (N, channel dim, number of layers) suggests there may be inference-time strategies that could boost accuracy without retraining.
Ablation on prompted frame memory capacity M. The paper ablates N (recent frames) but not M (prompted frames). The number of prompted frames directly affects memory bank size and cross-attention cost, and in long interactive sessions with many corrections, M determines whether early prompts are forgotten. Understanding the sensitivity to M would inform deployment decisions for different interaction patterns.
Failure mode analysis. Section C lists qualitative limitations (struggles with shot changes, crowded scenes, long occlusions, fast-moving thin objects, similar-looking nearby objects, extended videos) but provides no quantitative breakdown of failure frequency. An analysis showing what percentage of errors fall into each category, and how these percentages vary across benchmarks, would help users understand where SAM 2 can be trusted and where it needs human oversight.
Multi-object tracking efficiency. The paper notes that SAM 2 processes multiple objects independently, sharing only the image encoder. A comparison of throughput (FPS per object) as the number of objects increases would clarify the scalability of this approach for dense segmentation tasks like video instance segmentation on UVO or VIPSeg, where dozens of objects may need tracking per frame.

Overall, the experimental section provides comprehensive evidence for the paper's core claims — SAM 2 is a substantially more capable and efficient system than prior approaches for promptable video segmentation — with the caveat that the largest improvements are driven by data scale (SA-V is 53× larger than prior datasets) and image encoder efficiency (Hiera vs. ViT) as much as by the memory architecture itself. The ablation studies do a commendable job of isolating the contribution of individual architectural components, showing that the two-level memory bank, object pointers, and the removal of recurrent state transitions each contribute non-trivially to video performance without affecting image segmentation. The fairness evaluation, while limited in scope (one dataset, one object category), demonstrates a responsible approach to demographic bias analysis that goes beyond most computer vision papers' evaluation protocols.

6. Limitations and Trade-offs

6.1 Unaccounted Difficulty Estimation Cost in Real Deployment

The assumption. The SA-V annotation pipeline, particularly Phase 3, relied on annotators identifying "challenging objects" that SAM 2 struggled with — those requiring at least 2 edited frames — as a proxy for focusing annotation effort on model failure cases (§E.2.1). Annotators were shown videos "pre-filled with verified satisfactory automatic masklets and asked to find un-annotated challenging objects." This implicitly assumes that a separate verification step can cheaply identify which masklets are correct and which need human correction. The paper also assumes that automatic masklets rated "satisfactory" by verifiers are of sufficient quality to be added directly to the training set without human refinement (§5.1, Auto masklet generation).

The consequence. This assumption moves significant quality-assessment burden onto human verifiers and onto the model's own auto-generation quality. The data engine's cost model (Table 1: 4.5 seconds per frame in Phase 3) measures active annotation time — the time spent by annotators providing prompts and clicks. It does not include the time spent by the separate verification step (Step 5 in Figure 11), where a different set of annotators examined each masklet and rated it satisfactory or unsatisfactory. If verification takes, say, even 1 second per frame, the total annotation cost per frame would increase by 22%, and the Phase 3 speed advantage over Phase 1 would shrink accordingly (from 8.4× to perhaps 6–7×, depending on the verification overhead). More fundamentally, the verification step assumes a binary quality signal (satisfactory/unsatisfactory) that is itself imperfect — verifiers may miss subtle tracking errors in fast-moving or small objects, allowing incorrect masklets into the training set and potentially degrading model quality in ways that are hard to measure.

What evidence exists in the paper. The paper does not report the time or cost of the verification step. Table 1 reports only annotation time, edited frames, clicks, and mask alignment for the annotation phases, with no corresponding metrics for verification. The masklet verification protocol is described qualitatively (§5.1, Quality verification and §E.2.1) but no quantitative analysis of verifier accuracy (e.g., inter-annotator agreement, false positive/negative rates of the satisfactory rating) is provided. The auto masklet generation description (§E.1) notes that automatic masklets rated "unsatisfactory" are sent back for refinement, and those rated "satisfactory" are added directly to the dataset — but the paper does not report what fraction of auto-generated masklets are rated satisfactory, nor whether those masklets achieve comparable quality to manual annotations on any metric besides the binary verification label.

Mitigation status. Not addressed. The paper frames the verification step as a quality gate but does not analyze its cost, accuracy, or potential failure modes. Future annotation pipelines that adopt the SAM 2 data engine approach would need to characterize this overhead independently to understand the true annotation cost. The auto masklet pipeline specifically carries a risk: if SAM 2 has systematic failure modes (e.g., confusing similar-looking objects, losing track after occlusions), those failures may propagate into the automatically generated data and go undetected by verifiers who make the same mistakes for the same reasons. Training on such data could amplify rather than correct the model's biases.

6.2 The 6× Speed Improvement on Images Comes from the Image Encoder, Not the Unified Architecture

The assumption. The paper claims that "SAM 2 is more accurate and 6× faster than the Segment Anything Model (SAM)" on image segmentation (Section 1, Abstract). This claim is supported in Section 6.3 and Table 5, where SAM 2 (Hiera-B+) trained on SA-1B achieves 58.9 1-click mIoU at 130.1 FPS vs. SAM (ViT-H)'s 58.1 at 21.7 FPS. The presentation positions this as a benefit of SAM 2 as a system, implying that the unified image-video architecture contributes to the improvement.

The consequence. This framing conflates two independent sources of improvement: (1) the Hiera image encoder being more efficient than the ViT used in the original SAM, and (2) any benefit from the unified architecture or video training. For static image segmentation with an empty memory bank, SAM 2's memory attention and memory encoder are idle — the model functions as a Hiera backbone with a SAM-style mask decoder. The "6× faster" claim is therefore attributable almost entirely to Hiera's architectural efficiency (hierarchical design, removal of relative positional biases enabling FlashAttention-2, efficient global attention scheduling), reinforced by torch.compile, not to any video-specific innovation. A practitioner choosing between SAM and SAM 2 for a pure image segmentation deployment is selecting a better backbone, not a different class of model. The paper does train a version of SAM 2 on SA-1B only (Table 5, row 2) to isolate the architecture's contribution, but the framing throughout the paper attributes the speed and accuracy gains to "SAM 2" without emphasizing this decomposition.

What evidence exists in the paper. The evidence is clear once examined carefully: Table 5 row 2 (SAM 2 trained only on SA-1B) achieves 58.9 mIoU vs. SAM's 58.1, with identical training data. The only difference is the architecture (Hiera-B+ vs. ViT-H) and decoder modifications (skip connections, sigmoid IoU activation, no "no-prompt" iterations). The ablation in Table 9f confirms that image encoder size scaling (S → B+ → L) improves image segmentation (SA-23 mIoU: 57.8 → 59.7 → 61.1) independently of the memory system. The removal of RPB from the image encoder, which enables FlashAttention-2 and is a major contributor to the speedup, is motivated by video processing efficiency but benefits images equally. The absence of an ablation comparing a Hiera-based pure-image model (without memory attention) against the full SAM 2 on images means the paper provides no direct evidence that the unified architecture helps (or hurts) image segmentation relative to a same-backbone image-only model.

Mitigation status. Partially addressed by transparency. The paper does report the SA-1B-only row in Table 5, and the architectural ablations (Tables 9–11) clearly separate image and video components. However, the paper never explicitly states that the image speed and accuracy improvements are primarily encoder-driven and not a consequence of the memory architecture. This is a communication limitation rather than a technical flaw: the results are correct and well-documented, but a practitioner skimming the abstract or introduction could reasonably infer that SAM 2's memory design somehow improves image segmentation, which is not the case. The improved image accuracy with video training data (58.9 → 61.9) is a genuine contribution of the unified training, but the 6× speed is not.

6.3 The Interactive Evaluation Is Entirely Simulation-Based with No Human Study

The assumption. The paper's interactive video segmentation evaluation (Section 6.1, §F.1.2) simulates user behavior algorithmically: initial clicks are placed at "the center of the object's ground-truth mask" (defined as the mask pixel with the largest Euclidean distance to the mask boundary), and corrective clicks are placed at "the center of the error region" between the predicted and ground-truth masks. The annotation time model assumes fixed per-action costs: 1 second to visually locate an object, 1.5 seconds per click, and 30 seconds per 300-frame video to examine results at 10 FPS (§F.1.2). The paper uses these assumptions to convert from "number of interacted frames" to "annotation time" in Figures 12a and 13a, and to claim that SAM 2 requires "3× fewer interactions" than baselines.

The consequence. Algorithmic click placement fundamentally differs from human click placement in at least three ways: (1) humans do not have access to ground-truth error region centers — they see the model's output overlaid on the video and must visually identify where the error is, which is slower and less precise, especially for small objects or subtle boundary errors; (2) human click placement has spatial variance — clicks land near but not exactly at the intended location, and this noise can affect model behavior differently for SAM 2 (which has memory context to disambiguate imprecise clicks) vs. decoupled baselines (where SAM must interpret each click without temporal context); (3) humans may choose to correct different frames than the algorithm — the offline evaluation selects the frame with the lowest IoU against ground truth, but humans select the frame where the error is most visually apparent, which may not be the same frame (an error that is large in area may be easy to spot, while a small but semantically important error — like the model including an extra object — may be harder to notice). The fixed time-per-action assumptions further abstract away cognitive load: annotators may take longer to decide where to click when tracking failures are complex (e.g., the object is partially occluded and the model's mask is ambiguous), whereas algorithmic clicks require no decision time.

What evidence exists in the paper. The simulation protocol is described in detail (§F.1.2), including the click sampling strategy and the annotation time model parameters. The comparison of data engine phases (Table 1) does provide real-human annotation metrics (time per frame, edited frames, clicks per frame) from a controlled experiment with 169 videos and 452 masklets, annotated by separate groups of annotators using each phase's protocol. This is real human data, and it validates the relative efficiency improvement (Phase 3 is 8.4× faster than Phase 1 with real annotators). However, this data comes from the annotation pipeline, not from the interactive evaluation benchmarks used in Section 6.1 — the datasets, annotation tooling, and annotator training differ between these settings. The DAVIS interactive benchmark evaluation (Table 14) uses the DAVIS evaluation server's scribble protocol, which is a standardized but still automated evaluation. The paper does not conduct a user study comparing SAM 2 against baselines on the zero-shot video datasets with real annotators performing the same task as in the simulation.

Mitigation status. Not addressed. The paper relies entirely on simulation for the interactive evaluation claims (Section 6.1), and while the data engine experiment (Table 1) provides real-human validation of the annotation efficiency gains, it evaluates a different task (data annotation with model assistance) under different conditions (annotation tooling, trained annotators, specific datasets). A user study on the zero-shot benchmarks would substantially strengthen the claim that SAM 2 requires 3× fewer interactions in practice, but no such study is reported. The algorithmic click placement strategy is reasonable and follows prior work (Delatolas et al., 2024; Sofiiuk et al., 2022), and the fixed time-per-action model is clearly stated with its assumptions, so the results are reproducible and interpretable — but their external validity for real user behavior remains unvalidated.

6.4 Hard Cases — Shot Changes, Long Occlusions, Crowded Scenes — Remain Effectively Unsolved

The assumption. The paper states in Appendix C that "SAM 2 demonstrates strong performance in both static image and video domains, yet it encounters difficulties in certain scenarios. The model may fail to segment objects across shot changes and can lose track of or confuse objects in crowded scenes, after long occlusions or in extended videos." The proposed mitigation is the ability to prompt SAM 2 in any frame: "if the model loses the object or makes an error, refinement clicks on additional frames can quickly recover the correct prediction in most cases." This assumes that the model's failures are recoverable — that the user can identify when tracking is lost and provide corrective prompts that restore correct tracking.

The consequence. The paper does not quantify how often these failures occur, how long it takes to recover from them, or whether recovery is always possible with a reasonable number of clicks. On the hardest evaluation settings — VOST (large transformations), LVOSv2 (long videos), and the SA-V benchmark (open-world objects and parts) — SAM 2 substantially outperforms baselines (Table 6: 76.8 on SA-V val vs. 61.4 best prior), but the absolute performance leaves substantial room for improvement: 23.2% of SA-V val J &F is "lost" relative to perfect tracking. It is unclear whether this gap is distributed across many small, correctable errors (which the interactive refinement paradigm could address with a few extra clicks) or concentrated in catastrophic failures (where the model loses the object entirely and requires extensive re-annotation). If failures are catastrophic — the model confuses the tracked object with a similar nearby object, or loses track after a shot change and cannot recover even with prompts — then the interactive refinement paradigm breaks down: the user must effectively re-annotate from scratch, erasing the interaction efficiency advantage.

The paper also notes that "SAM 2 also struggles with accurately tracking objects with very thin or fine details especially when they are fast-moving," and that "incorporating more explicit motion modeling into SAM 2 could mitigate errors in such cases." The absence of explicit motion modeling (optical flow, trajectory prediction, motion-based feature matching) is an architectural choice — SAM 2 relies entirely on appearance-based memory matching with temporal position encoding providing only implicit motion cues through the ordering of recent frames. This means that when appearance changes dramatically (due to fast motion, motion blur, or deformation), the cross-attention to memory features may fail because no stored frame looks sufficiently similar to the current frame, even though a motion model could predict where the object should be.

What evidence exists in the paper. No quantitative breakdown of failure modes is provided. Appendix C lists failure scenarios qualitatively. The per-dataset results in Figures 12 and 13 show that SAM 2 achieves lower absolute accuracy on VOST (67.5 offline, 65.0 online at 8 frames) — the dataset specifically designed for large object transformations — than on other benchmarks, but the gap relative to baselines is large, indicating that while SAM 2 struggles on these cases, prior methods struggle more. The LVOSv2 results (Table 17c: SAM 2 Hiera-B+ achieves 78.7 J &F overall, but only 69.0 Ju and 77.8 Fu on unseen categories) reveal that performance drops substantially on object categories not seen during training — these are likely to be where the appearance-based memory matching is least reliable. The SA-V benchmark (Table 6) shows a 23.2-point gap between SAM 2 and perfection, but the distribution of errors across easy vs. hard frames within videos is not analyzed. The paper does not provide occlusion-length-specific analysis (e.g., accuracy after 5, 10, 20 frames of full occlusion), which would directly measure the recovery capability.

Mitigation status. Partially addressed by design. The interactive refinement capability is explicitly intended to handle these failure cases — the model is not expected to be perfect, but to fail gracefully and recover quickly. The quantitative results show that when interactive refinement is allowed (Figures 5, 12, 13), SAM 2's accuracy improves with additional annotated frames, confirming that corrections do help. However, the paper does not demonstrate that recovery is efficient for the hardest failure modes — the per-frame accuracy improvements in the interactive plots (Figures 5, 12) may be driven primarily by easy corrections on medium-difficulty frames rather than recovery from catastrophic failures. The suggestion of incorporating explicit motion modeling is a future work direction, not an implemented solution.

6.5 Training Data Scale Is a Confounding Variable in the Comparison to Prior VOS Methods

The assumption. The paper compares SAM 2 against prior state-of-the-art VOS methods (XMem, Cutie, DeAOT, JointFormer, etc.) on standard benchmarks (DAVIS, MOSE, YouTubeVOS, LVOS) and the new SA-V benchmark (Table 6, Table 17). The comparison protocol for these prior methods uses their released checkpoints, which were trained on standard VOS datasets (DAVIS, MOSE, YouTubeVOS — collectively a few hundred thousand masks). SAM 2 is trained on SA-V (35.5M masks — 53× more than any prior dataset) plus Internal data (5.4M masks) plus SA-1B (1B masks on images). The training data size difference is approximately two orders of magnitude.

The consequence. The performance gaps reported in Table 6 — particularly the +15.7 J &F improvement on SA-V val (76.8 vs. 61.4 best prior) and the +12.0 improvement on LVOS (78.0 vs. 66.0) — conflate two separate effects: (1) architectural improvements (the unified memory design, Hiera backbone, object pointers) and (2) training data scale and diversity. The SA-V benchmark result is the most affected: prior methods were never trained on SA-V-like data (open-world objects, parts, diverse scenarios), so their poor performance on SA-V primarily measures domain gap, not architectural inferiority. A reader might incorrectly conclude that SAM 2's architecture is 15+ J &F points better than XMem/Cutie for video segmentation, when a substantial portion of that gap would likely be closed if prior methods were retrained on SA-V — an experiment the paper does not conduct.

This is not to say the comparison is invalid: SAM 2's ability to be trained on SA-V is itself a contribution (the data engine enabled the dataset), and the final system includes both architecture and data. However, the causal attribution of the performance gains is unclear. The data ablation in Table 7 helps separate these effects: training SAM 2 on only existing VOS datasets (row 1) yields 76.9 J &F on MOSE dev — still competitive with prior methods. But this ablation is at 512² resolution and 200K iterations, not the full 1024² training, and it evaluates on MOSE dev (200 videos from the training split), not the full benchmarks. A direct comparison at full training scale (SAM 2 trained on VOS-only data vs. prior methods) is absent.

What evidence exists in the paper. Table 7 (data mixture ablation) and Table 2 (data scaling by engine phase) provide the basis for estimating the data contribution. Table 2 shows that going from the baseline (VOS datasets + SA-1B, which achieves 50.0 J &F on SA-V val) to the full SA-V data (63.2 J &F) adds +13.2 points — the majority of the final SA-V val performance comes from SA-V training data, not from the baseline knowledge. The data scaling curve in Figure 6 shows consistent improvement as SA-V data quantity increases, with no saturation, confirming that data is a major driver of performance. However, these ablations are all within the SAM 2 architecture — they show how SAM 2 benefits from more data, but they do not show whether prior methods would benefit similarly. The paper acknowledges that the SA-V dataset is "a substantial resource for future work" (Section 5.2), implicitly inviting retraining of prior methods on SA-V.

Mitigation status. Partially addressed by the inclusion of Table 7 and the data scaling analysis, but the central comparison table (Table 6) does not control for training data. A more rigorous comparison would include at least one prior method retrained on SA-V (or a comparable data mixture), or would report results for SAM 2 trained on the same VOS-only data as prior methods at full resolution and training schedule. The paper's framing emphasizes the system (model + data) rather than isolating architectural contributions, which is a valid design choice for a foundation model paper, but leaves the architectural contribution ambiguous. The substantial release of SA-V with permissive licensing (§1: CC by 4.0) partially mitigates this by enabling future researchers to run the controlled comparison.

6.6 The Multi-Object Tracking Strategy Is Inefficient and Lacks Inter-Object Communication

The assumption. SAM 2 processes each object independently: "When applying SAM 2 to segment multiple objects in the same video (such as multi-object tracking in the semi-supervised VOS evaluation), we perform inference on each object independently. More specifically, we share the visual features from the image encoder between all the objects in the video but run all the other model components (such as the memory bank and the mask decoder) separately for each object" (§D.1). This design assumes that objects can be segmented independently without inter-object communication, and that the computational cost of running memory attention, memory encoder, and mask decoder separately for each object is acceptable.

The consequence. The computational cost scales linearly with the number of objects. For a video with K objects, memory attention and mask decoding are performed K times per frame, while the image encoder is run once. On benchmarks with many objects per video — UVO (open-world segmentation, averaging ~6 objects per video in the validation set), VIPSeg (panoptic segmentation, averaging ~10 objects), BURST (averaging ~5.5 objects), or SA-V (averaging 3.8 manual + 8.9 auto masklets per video) — this linear scaling can make inference substantially slower than 43.8 FPS (the single-object speed). If a video has 10 objects, the effective throughput drops to approximately 4.4 FPS, which may be below real-time requirements for applications like autonomous driving or live video editing.

Beyond computational cost, the independent processing means there is no mechanism for objects to compete for or share pixel assignments. In crowded scenes where objects overlap or occlude each other, independent processing can lead to inconsistent masks: two objects may both claim the same pixel, or one object's mask may "leak" into another's region, because neither model component is aware of the other object's predictions. Standard multi-object tracking methods often include a spatial exclusion loss or a joint decoding step to enforce non-overlapping masks, but SAM 2 lacks any such mechanism. The paper acknowledges this: "incorporating shared object-level contextual information could aid in improving efficiency" (Section C), but no quantitative analysis of multi-object consistency or scalability is provided.

What evidence exists in the paper. The paper evaluates on multi-object benchmarks (MOSE, UVO, VIPSeg, BURST, YouTubeVOS) and reports aggregate metrics that include multiple objects per video, so the independent processing strategy is implicitly validated: the accuracy numbers in Tables 4 and 6 demonstrate that independent processing produces competitive results. However, no analysis is provided on:

Scaling of FPS with object count. The FPS measurements (43.8 for Hiera-B+, 30.2 for Hiera-L) are for single-object tracking. Multi-object throughput is not reported.
Mask overlap/consistency. There is no metric for inter-object mask consistency (e.g., percentage of pixels claimed by multiple objects, or IoU between object masks that should be disjoint).
Comparison to joint decoding methods. Some prior VOS methods (e.g., STCN, AOT/DeAOT) process multiple objects jointly in a single forward pass, sharing computation across objects. The paper does not compare SAM 2's multi-object speed against these methods, nor does it analyze whether independent processing is faster or slower than joint processing at various object counts.

Mitigation status. Acknowledged as a limitation in Appendix C: "While SAM 2 can track multiple objects in a video simultaneously, SAM 2 processes each object separately, utilizing only shared per-frame embeddings without inter-object communication. While this approach is simple, incorporating shared object-level contextual information could aid in improving efficiency." This is presented as a future work direction, not as a current capability. For practitioners deploying SAM 2 on dense multi-object segmentation tasks, the linear scaling in object count is a practical constraint that the paper does not quantify, requiring independent benchmarking for specific deployment scenarios.

7. Implications and Future Directions

How This Work Changes the Landscape

SAM 2 represents a paradigm unification rather than a paradigm shift — it does not introduce a fundamentally new mechanism for video understanding, but it collapses the previously separate problems of image segmentation and video object tracking into a single architectural framework with a single training objective and a single interaction protocol. This is consequential not because it overturns any theoretical understanding, but because it eliminates a fragmentation that had practical costs: prior to SAM 2, building a system that could segment objects from clicks and track them through video required composing two distinct models (SAM plus a tracker) with incompatible interfaces and no shared memory. The evidence that this unification matters is not theoretical but empirical: in the controlled data engine experiment (Table 1), the unified Phase 3 system is simultaneously faster (4.5 vs. 7.4 seconds per frame), requires fewer corrections (19.04% vs. 23.25% edited frames), and produces higher-quality masks (89.1% vs. 86.4% mask alignment) than the decoupled Phase 2 system annotating the same objects. The improvement is not from better components but from removing the interface between them.

The paper's most durable conceptual contribution may be the two-level memory taxonomy — separating recent frames with temporal encoding (short-term motion) from prompted frames without temporal encoding (persistent object identity), supplemented by compact object pointers for semantic disambiguation. Prior memory-based VOS models (XMem, AOT, Cutie) treated all stored frames as a homogeneous set for attention, without distinguishing what type of information each frame carries or how it should be retained. SAM 2's design makes explicit that not all memories are equivalent: some are transient motion cues, others are permanent identity anchors, and the model benefits from knowing which is which. The ablation evidence (Table 11: object pointers add +3.8 J &F on SA-V val and +4.6 on LVOSv2 but nothing on the 9 zero-shot average) suggests this taxonomy is particularly valuable when objects are semantically ambiguous (parts vs. wholes) or need to be tracked across long durations — precisely the scenarios where prior datasets provided insufficient coverage. This reframes the memory design problem from "how much to remember" to "what to remember and for what purpose" — a shift in emphasis that future memory architectures are likely to adopt.

The paper also resolves a tacit contradiction in the interactive segmentation literature. Prior work on decoupled SAM+tracker systems (Cheng et al., 2023b; Yang et al., 2023; Rajič et al., 2023) demonstrated that composing SAM with a VOS tracker produced functional interactive video segmentation, implying that unification was unnecessary. Other work on interactive VOS (Heo et al., 2020; Delatolas et al., 2024) built specialized architectures that integrated interaction and propagation but lacked the scale and generality of SAM's training data. SAM 2 demonstrates that both approaches were partially right: decoupled systems work adequately (SAM+Cutie achieves 74.7 J &F offline at 8 frames), but a unified system with memory works substantially better (SAM 2 achieves 80.3 J &F at the same budget, Figure 12b) because it preserves interaction context that decoupled systems discard. The contradiction is resolved by quantifying the gap: the decoupled approach is a reasonable engineering compromise, but the unified approach is the better architecture given sufficient training data.

The release of the SA-V dataset (50.9K videos, 35.5M masks under CC BY 4.0, with geographically diverse video capture across 47 countries and an analyzed fairness profile) is a landscape-changing contribution independent of the model. Prior to SA-V, the largest video segmentation dataset (BURST, 600K masks) was approximately 1/59th the size. The dataset's emphasis on small objects and parts (88% of masks with normalized area < 0.1, Figure 10a) and its high disappearance rate (42.5% for manual annotations, Table 3) define a new standard for evaluation difficulty. The power-law scaling relationship in Figure 6 — where video segmentation accuracy on SA-V val, 9 zero-shot benchmarks, and MOSE dev all improve log-linearly with SA-V masklet count up to 200K masklets without saturation — strongly implies that the field was previously data-limited rather than architecture-limited for the "segment anything in videos" capability. By releasing SA-V, the paper enables the next generation of video segmentation models to be trained at this data scale, and the SA-V benchmark provides a test bed where prior methods cluster around 60–63 J &F (Table 6) while SAM 2 achieves 76.8–77.9, leaving substantial headroom for architectural innovation.

Finally, the paper makes speed-accuracy tradeoffs a first-class design consideration in a domain where they had been an afterthought. The image segmentation result — 6× faster than SAM while more accurate — is primarily an architectural efficiency finding (Hiera vs. ViT, removal of relative positional biases enabling FlashAttention-2), but its presentation alongside video results frames speed as a core design objective rather than a constraint to be relaxed for accuracy. The ablation tables systematically report speed relative to a default configuration (Tables 9–11), making it possible to trace the performance cost of each design choice. At 43.8 FPS for video (Hiera-B+) and 130.1 FPS for images, SAM 2 demonstrates that real-time interactive segmentation is feasible without accuracy compromises — a practical threshold that makes the technology deployable in latency-sensitive applications like AR/VR and live video editing.

Follow-Up Research This Work Enables

Retraining prior VOS methods on SA-V to isolate architectural vs. data contributions. The central confound in SAM 2's evaluation is that it is trained on 53× more video masks than any prior method, making it impossible to determine how much of the +15.7 J &F improvement on SA-V val (76.8 vs. 61.4) comes from the memory architecture and how much from data scale. A direct follow-up would retrain Cutie, XMem, and DeAOT on the full SA-V dataset (plus SA-1B images for parity) using their original training recipes, then evaluate on the SA-V benchmark at matched training iterations. If these methods close most of the gap — achieving, say, 72–75 J &F on SA-V val — then SAM 2's architectural contribution is modest (~3–5 points) and the primary value is the dataset. If they improve only modestly (to 65–68 J &F), then the unified memory architecture genuinely enables better use of large-scale training data in a way that decoupled or specialized VOS architectures cannot. The controlled experiment should also measure whether prior methods' performance on standard benchmarks (DAVIS, MOSE) degrades when trained primarily on SA-V, testing whether SA-V's open-world distribution is complementary or detrimental to established VOS tasks.

Measuring and mitigating the correct-to-incorrect reversion rate in memory-based tracking. SAM 2's memory encoder fuses the predicted mask with image features and pushes the result into the FIFO queue for future cross-attention. This creates a feedback loop: if the model makes an error on frame t, the erroneous memory enters the bank and can corrupt tracking on frames t+1, t+2, etc. The paper documents a qualitatively similar phenomenon in the context of revision models (the reference example's "38% correct-to-incorrect reversion rate" for language model revisions), but no analogous measurement exists for SAM 2. A targeted experiment would introduce synthetic perturbations to the predicted mask on a single frame (replacing it with a shifted or scaled version, or a mask of a different object) and measure the "error propagation length" — how many subsequent frames are affected before the memory attention recovers, and whether object pointers or prompted frame memories accelerate recovery. If errors propagate for 10+ frames before self-correcting, this identifies a fundamental stability issue that may be addressable through memory gating mechanisms (only storing memories when the IoU prediction exceeds a confidence threshold) or through test-time memory verification (comparing the current prediction against stored prompted-frame memories and flagging inconsistencies). If errors self-correct within 2–3 frames due to the prompted frame memories serving as stable anchors, the architecture is more robust than it appears.

Explicit motion modeling as a complementary cue to appearance-based memory matching. The limitations section (Appendix C) suggests that "incorporating more explicit motion modeling into SAM 2 could mitigate errors" for fast-moving thin objects and similar-looking distractors. A concrete experiment would add an optical flow branch to SAM 2: compute flow between consecutive frames using an off-the-shelf method (e.g., RAFT), warp the previous frame's mask according to the flow to produce a motion-predicted mask, and fuse this with the memory-attention-conditioned features in the mask decoder (e.g., as an additional input channel or through a learned gating mechanism). The test would target VOST (where SAM 2 achieves 67.5 J &F offline at 8 frames — the lowest among the 9 zero-shot datasets), measuring whether flow-based mask warping improves tracking of deforming objects. The critical negative result to watch for: flow-based warping may hurt on static-camera scenes where appearance-based matching is already sufficient, if the flow network produces noisy estimates that confuse the mask decoder. A successful result would demonstrate improvement specifically on fast-motion subsets of VOST and LVOSv2 while leaving performance unchanged on DAVIS and MOSE.

Dynamic memory bank management: learning when to forget. SAM 2's memory bank uses a fixed FIFO policy with N=6 recent frames — the oldest frame is evicted when a new one arrives, regardless of content. This is suboptimal: some frames contain critical information (the moment the object emerges from occlusion, a rare viewing angle) that should be retained longer, while others are redundant (consecutive frames where the object barely moves) and could be discarded sooner. A follow-up could train a lightweight "memory importance" scorer that takes the memory encoder output and predicted IoU as input, and outputs a retention priority score used to decide which memory to evict when the bank is full. The training signal could come from downstream mask accuracy: evict memories whose removal causes the least degradation in tracking performance on subsequent frames. The experiment would measure whether learned eviction outperforms FIFO at fixed memory capacity N=6, and more importantly, whether it matches the performance of N=8 or N=12 with a smaller memory budget (e.g., N=4 with learned retention matching N=6 FIFO). If successful, this would improve efficiency on long videos (LVOSv2) where the fixed window may span only a fraction of the video. The ablation in Table 9c already shows diminishing returns from larger N (MOSE dev: 73.5 for N=4, 73.0 for N=6, 73.2 for N=8), suggesting that raw capacity is not the bottleneck — which frames are retained matters more than how many.

Inter-object communication for multi-object consistency and computational efficiency. The paper acknowledges that processing each object independently "without inter-object communication" is a limitation, particularly for crowded scenes and efficiency. A concrete architectural extension would add a lightweight cross-object attention module that runs after the per-object memory attention but before the mask decoder: for K objects being tracked, the K temporally-conditioned feature maps attend to each other (or to a shared compact representation summarizing all objects) to resolve spatial conflicts before mask prediction. The experiment would measure two outcomes: (1) mask consistency (reduction in pixel overlap between objects that should be disjoint, measured on MOSE and UVO where multiple objects frequently interact), and (2) computational scaling (whether the additional cross-object attention cost is offset by the ability to reduce per-object capacity, e.g., using fewer memory attention layers when cross-object context is available). A simpler baseline to test first: using the occlusion prediction head to explicitly inhibit pixels that another object already claimed, a lightweight form of spatial exclusion that requires no architectural changes. The negative result to watch for: cross-object attention may cause objects to "steal" pixels from each other when their appearances are similar, reducing per-object accuracy even as consistency improves.

Difficulty-conditioned inference: adaptive memory bank size based on video characteristics. The reference example paper established that compute-optimal test-time strategies should adapt to estimated prompt difficulty. An analogous idea applies to SAM 2: easy videos (slow motion, distinct object, minimal occlusion) may achieve ceiling performance with N=2 recent frames and no object pointers, while hard videos (fast motion, similar distractors, long occlusions) benefit from N=8 and full pointer attention. A follow-up could train a lightweight difficulty estimator that takes the first few frames and the initial prompt as input and predicts the optimal N, memory channel dimension, and number of memory attention layers for that specific video, then evaluate whether adaptive resource allocation matches the accuracy of using the maximum configuration everywhere while reducing average FLOPs per frame. The difficulty signal could come from the model's own uncertainty: if the IoU prediction drops sharply between consecutive frames, the video may be in a "hard" regime requiring more memory. The DAVIS and MOSE benchmarks would serve as the "easy" and "hard" test cases respectively, with SA-V providing the mixed-difficulty distribution where adaptation should provide the largest efficiency gains.

Practical Applications and Downstream Use Cases

Cost-efficient video annotation for training data generation. The SA-V data engine demonstrated that SAM 2 in the loop reduces annotation time from 37.8 to 4.5 seconds per frame (8.4×), with the majority of that improvement coming from the shift from per-frame re-annotation to click-based refinement enabled by memory. For organizations building custom video segmentation datasets — medical imaging (surgical tool tracking), agriculture (crop growth monitoring), wildlife conservation (animal behavior tracking), or sports analytics (player tracking) — deploying SAM 2 with domain-specific prompting can reduce annotation costs proportionally to the fraction of frames requiring correction (~19% in the Phase 3 experiment) vs. full re-annotation (100% in Phase 1). The concrete workflow: annotators provide initial prompts on frame 1, SAM 2 propagates to all frames, annotators verify at 10 FPS (~3 seconds for a 300-frame video at the 10 FPS examination rate from §F.1.2) and add refinement clicks on frames with visible errors, with SAM 2 updating the full masklet after each correction. At 4.5 seconds per frame average from the controlled experiment, annotating the 4.2M frames in SA-V would require ~5,250 annotator-hours — compared to ~44,100 hours with per-frame SAM, a savings of ~38,850 hours that translates directly to faster and cheaper dataset creation.

Real-time interactive video editing with object-aware selection. Video editing tools (Adobe Premiere, DaVinci Resolve, Final Cut Pro) currently require rotoscoping — manually drawing masks frame-by-frame — to apply effects to specific objects (color grading a person's shirt, blurring a license plate, replacing a background behind a moving subject). SAM 2 at 43.8 FPS on a single A100 GPU (Hiera-B+) processes video faster than real-time for typical 24–30 FPS content, enabling an editing workflow where the user clicks the object once, SAM 2 generates the masklet across the entire clip in less time than the clip's duration, and the user scrubs through to add refinement clicks on frames where the mask is imperfect. The interactive evaluation results (Figure 5b, online mode) show SAM 2 achieving ~80 J &F at 8 annotated frames across 9 zero-shot datasets, and Table 1 shows that annotators needed to edit only 19% of frames when using SAM 2. For a 10-second clip at 24 FPS (240 frames), this translates to ~46 frames requiring a click correction — a task completable in under a minute, compared to hours for frame-by-frame manual rotoscoping. The 130.1 FPS image segmentation speed means that even when the user does need to refine a mask on a specific frame, the interactive response is instantaneous by human perception standards.

Video instance segmentation for autonomous vehicle perception. Autonomous vehicles must detect, segment, and track all objects in a scene (vehicles, pedestrians, cyclists, traffic signs) across video frames to predict motion and plan trajectories. SAM 2's semi-supervised VOS performance on driving datasets (Virtual KITTI 2: 74.1 J &F offline at 8 frames with 3 clicks, Table 12b; Cityscapes: evaluated in the zero-shot image and video benchmarks) demonstrates competence on this domain, but the independent-per-object processing strategy means that tracking 30+ objects in a busy urban scene would reduce throughput from 43.8 FPS to ~1.5 FPS — below real-time requirements. The practical deployment path is therefore not direct use of SAM 2 as a perception module, but rather using SAM 2 to auto-label driving datasets at scale: run SAM 2 once on each object in each video (offline, with verification), producing dense segmentation labels that can then train a lightweight multi-object tracker optimized for real-time inference. The SA-V pipeline's auto masklet generation and verification workflow (§E.1) provides a template for this: use SAM 2 with grid prompting on the first frame to generate candidate object masks, propagate them temporally, verify quality (potentially with an automated consistency checker rather than human verifiers), and use the verified masklets as training data for a domain-specific model. The 53× mask increase from SA-V over prior datasets suggests that this approach could similarly increase the scale of driving-specific video segmentation datasets by an order of magnitude.

Long-term wildlife monitoring with re-identification across occlusions. Camera traps and wildlife monitoring stations capture extended videos where animals appear, disappear behind vegetation, and reappear minutes or hours later. The LVOSv2 evaluation (videos averaging 1.14 minutes) shows SAM 2 achieving 78.7 J &F (Hiera-B+, Table 17c), including 69.0 on unseen object categories — suggesting that the object pointer mechanism provides some re-identification capability even when the object's appearance changes. For wildlife monitoring specifically, the Lindenthal Camera Traps dataset evaluation (included in the 17 zero-shot video datasets, §F.1.1) provides a direct test, though per-dataset results for this benchmark are aggregated into the average. A deployment scenario would run SAM 2 continuously on a video stream, using the occlusion prediction head to detect when the tracked animal has left the frame, and using the prompted-frame memory (which is retained indefinitely — the paper does not specify an M limit) to re-identify the same individual when it returns, based on matching the stored object pointer against the new detection. The key practical metric is the re-identification precision: what fraction of "reappearances" does the model correctly link to the original track vs. treating as a new animal? The disappearance rate of 42.5% on SA-V manual annotations (Table 3) indicates that this is a common challenge in the training distribution, but no direct re-identification experiment is reported.

When to Prefer This Method

The paper articulates a clear tradeoff between (a) SAM 2 as a unified model for promptable video segmentation and (b) decoupled SAM+tracker pipelines. The decision rule emerges from the architectural and experimental evidence:

Prefer SAM 2 when:

The interaction involves refinement across multiple frames. If the user expects to provide prompts on more than one frame (e.g., initial segmentation on frame 1, corrections on frames 10 and 50), SAM 2's memory bank retains all prompted frames indefinitely, enabling single-click corrections that leverage prior context. Decoupled SAM+tracker baselines restart SAM from scratch on each correction frame, discarding interaction history. The controlled experiment (Table 1) quantifies this advantage: Phase 3 (unified SAM 2) requires 2.68 clicks per corrected frame vs. 3.61 for Phase 2 (decoupled), and edits only 19.04% of frames vs. 23.25%.
The objects are small, are parts rather than wholes, or have ambiguous boundaries. The SA-V benchmark evaluations show prior VOS methods clustering at 60–63 J &F (Table 6) on open-world parts and small objects, while SAM 2 achieves 76.8–77.9. The object pointer ablation (Table 11) shows +3.8 J &F specifically on SA-V val from object pointers, confirming that semantic disambiguation through compact object-level representations is valuable for part-level segmentation.
Long videos with occlusions are expected. The LVOS and LVOSv2 results (78.0 and 78.7 J &F, respectively) substantially exceed prior methods (Cutie: 66.0 on LVOS) on long-duration tracking. The disappearance rate of 42.5% in SA-V Manual suggests the training distribution specifically covers occlusion and re-appearance.
Latency for the interactive response on a single frame is critical. At 130.1 FPS for image segmentation (batch size 10, A100), the per-frame interactive response is well below the threshold of human perception, even without batched processing.

Prefer decoupled SAM+tracker when:

Only a single-frame prompt is needed and no refinement is expected. In the semi-supervised VOS setting with a ground-truth mask on frame 1 (Table 6, mask column), SAM 2 outperforms Cutie and XMem, but the margin on saturated benchmarks is modest: SAM 2 achieves 90.2 on DAVIS 2017 val vs. JointFormer's 90.1. If the use case is pure tracking from a high-quality first-frame mask with no user interaction, a specialized VOS model may be sufficient and simpler to deploy.
The video domain matches standard VOS training distributions (people, vehicles, animals) and the objects are whole, salient entities. On these distributions, prior methods are competitive: on YouTubeVOS 2019 val, Cutie-base+ achieves 87.5 G vs. SAM 2's 88.6 — a 1.1-point gap. If SA-V-scale training data is not available for fine-tuning, the specialized VOS models trained on existing datasets may generalize adequately.
Training data scale is constrained. SAM 2's performance is data-hungry: the power-law scaling in Figure 6 shows log-linear improvement up to 200K masklets without saturation. If only standard VOS datasets (a few hundred thousand masks total) are available for training, SAM 2 trained on this limited data achieves 50.0 J &F on SA-V val (Table 7, row 7), while existing specialized methods may perform comparably on their target benchmarks — though the paper does not provide this direct comparison for SAM 2, only for the baselines.
Explainability of tracking failures is important. SAM 2's memory attention is a black box: when tracking is lost, it is difficult to diagnose whether the failure came from the image encoder features, the memory cross-attention, or the mask decoder. In a decoupled pipeline, failures can be attributed to SAM (spatial mask quality) or the tracker (temporal propagation quality) separately, simplifying debugging. Applications where tracking failures have high consequence (medical, autonomous driving safety) may prefer the debuggability of modular systems until SAM 2's failure modes are better characterized.

These preferences are not absolute — the paper demonstrates that SAM 2 achieves state-of-the-art in all evaluated settings, and the "prefer decoupled" scenarios represent cases where the gap is small enough that other factors (simplicity, debuggability, existing deployment infrastructure) may dominate the decision. The release of SAM 2 under Apache 2.0 and SA-V under CC BY 4.0 lowers the barrier to trying both approaches.

🎯 Pitch

1. Executive Summary

2. Context and Motivation

The Core Problem: Image Segmentation Alone Cannot Capture the Visual World

Why Video Segmentation Is Fundamentally Harder

The Landscape of Existing Approaches

Interactive Video Object Segmentation (iVOS)

Semi-Supervised Video Object Segmentation (VOS)

Video Segmentation Datasets: Insufficient Coverage for "Segment Anything"

Where the Decoupled Approach Breaks Down

How SAM 2 Positions Itself

The Data Engine as a Self-Reinforcing Loop

3. Technical Approach

3.1 Reader orientation (approachable technical breakdown)

3.2 Big-picture architecture (diagram in words)

3.3 Roadmap for the deep dive

3.4 Detailed, sentence-based technical breakdown

3.4.1 The Promptable Visual Segmentation (PVS) Task

3.4.2 The Image Encoder: Hiera with Multiscale Feature Extraction

3.4.3 The Memory Bank: What Is Stored and How It Is Organized

3.4.4 Memory Attention: Conditioning Current Features on Temporal Context

3.4.5 Prompt Encoder and Mask Decoder

3.4.6 The Memory Encoder: Compressing Predictions into Storable Memories

3.4.7 Training Procedure: Pretraining, Full Training, and Fine-Tuning

3.4.7.1 Stage 1: Pretraining on SA-1B (Static Images)

3.4.7.2 Stage 2: Full Training on Image and Video Data

3.4.7.3 Stage 3: Fine-Tuning on Longer Sequences

3.4.8 The Data Engine: Three-Phase Annotation with Model in the Loop

4. Key Insights and Innovations

Innovation 1: Unifying Image and Video Segmentation Through Streaming Memory Eliminates the Interaction-Reset Problem

Innovation 2: A Two-Level Memory Architecture Separates Short-Term Motion from Persistent Object Identity

Innovation 3: Test-Time Compute Substitution — 6× Faster Image Segmentation While Improving Accuracy

5. Experimental Analysis

Evaluation Methodology

Main Quantitative Results

5.1 Promptable Video Segmentation: Interactive Offline and Online Evaluation

5.2 Semi-Supervised Video Object Segmentation (VOS)

5.3 Image Segmentation

5.4 Data Engine Efficiency and Scale

Ablation Studies and Robustness Checks

Data Ablations

Model Architecture Ablations

Fairness Evaluation

Critical Assessment

Claim 1: "SAM 2 delivers better accuracy using 3× fewer interactions than prior approaches"

Claim 2: "SAM 2 outperforms prior work in established video object segmentation benchmarks"

Claim 3: "SAM 2 is more accurate and 6× faster than SAM on image segmentation"

Claim 4: "SA-V is 53× larger than any existing video segmentation dataset by mask count"

Missing Experiments That Would Strengthen the Paper

6. Limitations and Trade-offs

6.1 Unaccounted Difficulty Estimation Cost in Real Deployment

6.2 The 6× Speed Improvement on Images Comes from the Image Encoder, Not the Unified Architecture

6.3 The Interactive Evaluation Is Entirely Simulation-Based with No Human Study

6.4 Hard Cases — Shot Changes, Long Occlusions, Crowded Scenes — Remain Effectively Unsolved

6.5 Training Data Scale Is a Confounding Variable in the Comparison to Prior VOS Methods

6.6 The Multi-Object Tracking Strategy Is Inefficient and Lacks Inter-Object Communication

7. Implications and Future Directions

How This Work Changes the Landscape

Follow-Up Research This Work Enables

Practical Applications and Downstream Use Cases

When to Prefer This Method

Adjacent in domain

LLaVA-OneVision: Easy Visual Task Transfer

Building and better understanding vision-language models: insights and future directions