SetCon: Towards Open-Ended Referring Segmentation via Set-Level Concept Prediction

ArXiv: 2605.20110

🎯 Pitch

Instead of outputting a flat list of indistinguishable mask tokens for multi-target queries, this paper shows that explicitly predicting a shared set-level concept (e.g., 'summer salad staples') and then decomposing it into fine-grained sub-concepts (e.g., 'green cucumber', 'red tomato') dramatically improves mask completeness and distinctness, with gains widening as the number of referred targets grows.

---

1. Executive Summary

This paper proposes Set-Concept Segmentation (SETCON), a framework that reformulates open-ended referring segmentation as explicit set-level concept prediction, replacing the prevailing paradigm of indistinguishable [SEG] tokens with an interpretable language-grounded concept interface organized through hierarchical semantic decomposition (a coarse-to-fine structure where a shared set-level concept defines the overall target scope and then decomposes into fine-grained sub-category concepts aligned with target subsets). Evaluated across six image and seven video referring segmentation benchmarks using Qwen3-VL-8B-Instruct as the LVLM backbone and SAM 3 as the segmentation model, SETCON achieves state-of-the-art results, including +12.1 gIoU on MUSE and +3.3 gIoU on gRefCOCO for images—with the margin widening as the number of referred targets grows—and +10.9 J&F on MeViS and +12.4 J&F on Ref-SeCVOS for video, establishing that explicit semantic anchoring provides disproportionate benefits in multi-target and cross-category settings while remaining competitive on conventional single-target benchmarks, though the approach remains bounded by the model's capability to disambiguate open-ended queries where target boundaries are inherently ambiguous.

2. Context and Motivation

The Gap: Open-Ended Referring Segmentation Remains Unaddressed

Referring segmentation — the task of producing pixel-level masks for objects described by natural-language queries — has seen substantial progress, particularly with the integration of Large Vision-Language Models (LVLMs). However, as the paper identifies, this progress has been concentrated on a narrow slice of the problem space: single-target, single-category queries. The domain benchmarks that drove this progress (RefCOCO, RefCOCO+, RefCOCOg) predominantly feature queries that refer to a single instance or a small set of identical instances (e.g., "the red car on the left"), where the model's job is to ground one linguistic referent to one or a few visually similar masks.

The gap the paper addresses is the extension of referring segmentation to open-ended scenarios — queries that ask about heterogeneous sets of targets spanning multiple categories, where the cardinality, identity, and semantic scope of the target set are not predetermined by a simple template but must be dynamically inferred from the query and visual context. Examples include "If you wanted to make a refreshing summer vegetable salad, which objects in the image would be the best to pick first?" or "Which visible objects together form the main entertaining setup?" These are not single-"where is the X?" questions. They require the model to:

1. Parse a compositional intent: the query describes a role, function, or scenario that a set of objects must collectively satisfy, not a list of named instances.
2. Determine the target set's boundaries: which objects belong in the set and which do not, based on their visual properties, spatial relationships, and functional roles.
3. Segregate a heterogeneous collection into coherent subsets: the answer often involves multiple distinct categories (e.g., cutting board, knife, vegetables for a "salad preparation station"), each requiring separate masks.
4. Enforce set-level properties implicitly: the model must avoid including irrelevant objects, must not miss relevant ones, and must not duplicate masks for the same instance — properties of completeness and mutual exclusivity that are natural when the answer is treated as a set rather than as a list of independent predictions.

Prior benchmarks that touch on multi-target settings, such as gRefCOCO and MUSE, exist, but the paper argues that the prevailing architectural paradigm in LVLM-based segmentation is fundamentally misaligned with this kind of set-level reasoning. Understanding why requires examining the dominant design pattern in recent work.

The Prevailing Paradigm: The Implicit [SEG] Token Interface

Starting with LISA, the dominant approach for LVLM-based referring segmentation has been to insert a special [SEG] token into the LVLM's vocabulary. The model generates one or more of these tokens in its textual response, and the hidden state of each [SEG] token is separately fed into a mask decoder to produce a mask. This is an implicit interface: the semantic content of what is being segmented is not stated in the model's own words but rather compressed into a learned, non-linguistic representation — the hidden state of a newly initialized token.

For single-target queries, this design is adequate. The model generates a sentence like "The red car is [SEG]" and the single [SEG] hidden state carries enough information — combined with the surrounding linguistic context — to decode the correct mask. But the paper identifies two interconnected problems that emerge when this paradigm is stretched to multi-target, cross-category scenarios.

Problem 1: Performance Degradation with Target Count

The paper conducts a pilot study (Section 3.1) that is central to its motivation. They take a representative LVLM-based segmentation model, Sa2VA, extend it to the multi-target setting by having it sequentially predict multiple [SEG] tokens in a single response, fine-tune it on MUSE, and evaluate performance stratified by the number of referred targets. The finding is stark: performance degrades sharply as the target count grows (Figure 2a). What works passably for one or two targets collapses as the model must handle four, five, or more.

This degradation is not a training data issue — the model was trained on multi-target data. It is an architectural mismatch. The per-token formulation treats each target as an independent output: predict [SEG], decode mask for target 1; predict another [SEG], decode mask for target 2; and so on. There is no mechanism forcing these independent predictions to respect set-level constraints. The model can:

• Miss targets entirely (incomplete coverage): because each [SEG] is predicted sequentially without a global view of what has already been covered.
• Duplicate masks (lack of mutual exclusivity): because two [SEG] tokens with similar hidden states may decode overlapping or identical regions.
• Produce an unstable target-to-mask assignment: because the correspondence between the sequence position of a [SEG] token and a specific target is learned implicitly and can vary unpredictably.

Problem 2: [SEG] Embeddings Encode Spatial Layout More Than Semantics

The paper's second critical observation from the pilot study is an analysis of what the [SEG] hidden states actually represent. They apply t-SNE to the projected [SEG] representations from the Sa2VA baseline on a deduplicated, class-balanced subset of RefCOCO, and color the embeddings by two attributes: (1) the semantic category of the referred object (e.g., "person," "car," "chair") and (2) the 2D spatial position of the object in the image.

The result, shown in Figure 2b, reveals that the [SEG] embeddings cluster predominantly by spatial position rather than by semantic category. Spatially adjacent regions remain close in the embedding space, while semantically similar objects located in different parts of the image are scattered apart. This is a critical diagnostic finding. It means the implicit token interface — intended to bridge language reasoning and pixel prediction — is effectively encoding where to look rather than what to look for.

For single-target queries, this can suffice: the linguistic description already specifies the target semantics ("the red car"), and the [SEG] token provides a spatial coordinate to decode the mask. But for open-ended, multi-target, cross-category queries, the model needs to reason about semantic structure — distinguishing a cucumber from a tomato, grouping both under "salad ingredients," while excluding a cutting board if the query is specifically about edible items. An interface that merges all this semantic information into a spatial-position-dominated representation loses the very discriminative power needed to make these fine-grained distinctions.

The paper frames this concisely in Section 3.1:

"it is highly counter-intuitive to encode the distinct semantic and spatial properties of multiple diverse targets into the hidden states of identical, repetitively generated [SEG] tokens. Such a trivial design inevitably limits the representation capacity and risks feature ambiguity."

The key words are "identical" and "repetitively generated." Two apples and an orange, all referred to by the same query, would each be represented by a [SEG] token. These tokens are the same symbol in the model's vocabulary, distinguished only by their position in the output sequence and the context leading up to them. The model must learn to pack all the discriminative information — category, appearance, spatial location, and relation to other targets — into a single dense vector that is then decoded independently for each target. The t-SNE evidence shows that, in practice, the model settles on spatial location as the dominant organizing principle, leaving semantic distinctions under-encoded.

Why This Problem Matters

The paper's motivation extends beyond a methodological critique of the [SEG] token interface. The inability to handle open-ended, multi-target, cross-category referring segmentation has concrete downstream consequences:

Real-world queries are rarely single-target and single-category. In interactive image editing ("select all the ingredients for this recipe"), embodied AI ("pick up the items needed to set the table"), industrial inspection ("identify all safety hazards in this workspace"), or AR/VR interaction ("highlight the objects that form the entertainment setup"), users naturally refer to heterogeneous sets of objects defined by functional roles or scenarios, not by individual category labels. A system that degrades with each additional target is fundamentally limited in these deployed settings.

The LVLM's pretrained knowledge is underutilized. Modern LVLMs like Qwen3-VL possess rich semantic knowledge — they can reason about object categories, attributes, functional roles, and cross-object relationships. The [SEG] token paradigm circumvents this knowledge by introducing a non-linguistic interface trained from scratch. The model reasons in language about what to segment, but then compresses that reasoning into a newly initialized token that cannot leverage the pretrained semantic geometry of the model's own vocabulary. This is a missed opportunity: if the LVLM can name what it is segmenting, those names come embedded in a rich representational space that already encodes semantic similarity, contrast, and hierarchical relationships.

Set-level properties are left implicit and unenforced. The per-token formulation provides no architectural incentive for the model to produce a coherent set of masks. Completeness (are all relevant objects covered?) and mutual exclusivity (are any objects masked twice?) are emergent properties at best, and the training objective — typically a DETR-style set-prediction loss with bipartite matching — only weakly enforces them through label assignment. The model never receives an explicit signal that "this set of masks should be complete and non-overlapping for this query."

Prior Approaches and Their Shortcomings

The paper situates itself within a broader landscape of approaches, each of which addresses parts of the open-ended referring segmentation problem but leaves key gaps:

LVLM + [SEG] token variants (LISA, GSVA, PixelLM, SAM4MLLM, Sa2VA, etc.). These methods are the direct precursors and the primary targets of the paper's critique. They established the viability of LVLM-based referring segmentation but remain anchored to the per-token implicit interface. Extensions to multi-target settings (e.g., PixelLM predicting multiple [SEG] tokens, GSVA predicting [SEG] sequences) work by simply predicting more tokens, but as the pilot study demonstrates, this does not scale to complex sets. The paper identifies the root cause as the representation bottleneck — identical tokens encoding primarily spatial information — rather than a training data or loss function deficiency that could be fixed within the same paradigm.

Open-vocabulary grounding and detection (Grounding DINO, GLIP, GLEE). These methods support concept-level localization, where a text prompt (e.g., "red car") is used to detect all instances of that concept. They are inherently category-conditional: you specify the category, and the model finds all instances. In open-ended referring segmentation, however, the target categories are not specified; they must be inferred from the query. Moreover, the query often describes a functional role or scenario ("objects for a summer salad") that does not map cleanly to a single object category. These methods lack the language understanding and reasoning capability to decompose a scene-level query into the appropriate set of visual concepts.

Text-as-segmentation approaches (Text4Seg). A recent direction reformulates segmentation as text generation, where the model outputs category labels alongside mask tokens. This is closer in spirit to SETCON's explicit concept prediction, but it typically operates with a fixed or closed vocabulary and treats each mask independently. It does not organize predictions into a hierarchical set structure or enforce set-level coherence.

Reasoning-enhanced segmentation (VisionReasoner, Seg-Zero, RSVP). These methods incorporate explicit reasoning steps — chain-of-thought, geometric prompting, or reinforcement learning — to improve segmentation accuracy. While they can handle complex queries, they still output masks via implicit interfaces ([SEG] tokens or external geometric prompts) and do not model the target set as a structured, hierarchical semantic entity. Their contributions are orthogonal to the representation format: better reasoning before mask decoding, rather than a different interface for mask decoding itself.

Referring video object segmentation (VideoLISA, VISA, Sa2VA, GLUS). In the video domain, the challenge is compounded by the need for temporal consistency. Methods that treat each frame independently (or with limited cross-frame communication) struggle to maintain identity consistency for long sequences. The [SEG] token approach is particularly brittle here because the token's hidden state — already dominated by spatial information — varies from frame to frame as objects move, making identity association across time difficult. Some methods (e.g., Sa2VA for video) address this with memory-based propagation, but the semantic anchors for tracking remain implicit.

Promptable foundation models (SAM, SAM 2, SAM 3). SAM 3, which SETCON builds on, introduces the notion of "concepts" — text-based prompts that specify what to segment. This is an important step toward explicit semantic grounding. However, SAM 3's concept interface is designed for user-provided prompts in an interactive setting, not for model-generated concepts derived from open-ended queries. It does not include a mechanism for hierarchical set-level decomposition or for jointly decoding a mask set from multiple related concepts. SETCON's contribution can be understood as operationalizing SAM 3's concept interface within an LVLM reasoning framework, making it automatic and structurally organized for complex queries.

How SETCON Position Itself

The paper positions SETCON as a paradigm shift from implicit token generation to explicit semantic grounding. The core thesis, stated in Section 3.2, is:

"Instead of relying on implicit [SEG] tokens as a latent signal for mask decoding, SETCON reads the target concepts directly from its own textual output."

This is not a minor modification to the LISA architecture; it is a fundamental reformulation of the interface between language reasoning and pixel prediction. The key design choices that flow from this reformulation are:

1. Concepts are natural-language phrases, not learned tokens. Each referred concept is expressed as a free-form language phrase delimited by special markers (<ref>...</ref>). The hidden states of the tokens within this phrase — not a single [SEG] token — are projected into the mask decoder. This means the model uses its own pretrained vocabulary to name what it is segmenting. The embeddings of these concept words already live in the LVLM's semantic space, where "cucumber" and "tomato" are closer to each other than either is to "chair," and where "salad ingredients" summarizes a superordinate category. This side-steps the representation bottleneck identified in the pilot study: the model no longer needs to compress all semantic information into a single dense vector because the semantic information is distributed across the tokens of a natural-language phrase.

2. Concepts are hierarchical, not flat. The paper introduces a hierarchical semantic decomposition where the model first emits a set-level concept (e.g., "essential gear for a baseball game") that defines the overall scope, and then decomposes it into fine-grained sub-category concepts (e.g., "baseball bat," "baseball helmet," "catcher's mitt"). This coarse-to-fine structure is explicitly designed to address the set-level properties — completeness and mutual exclusivity — that the per-token formulation neglects. The set-level concept delineates the boundary of what belongs; the sub-category concepts discriminate within that boundary. During mask decoding, each sub-category representation is fused with the set-level representation, conditioning fine-grained predictions on the shared semantic scope.

3. The mask set is decoded jointly, not independently. Rather than decoding one mask per concept in isolation, SETCON produces a primary mask set conditioned on the set-level concept and then assigns fine-grained labels from the sub-category concepts. This joint decoding, combined with DETR-style set-prediction losses, provides architectural incentives for the masks to form a coherent, non-overlapping set.

4. The framework is video-native, not image-first. For video, the LVLM is invoked once per clip to produce motion-aware concepts, which are then shared across all frames as persistent semantic anchors. This is fundamentally different from per-frame [SEG] token prediction, where the hidden states drift as objects move. The concept "yellow headband" remains the same linguistic entity across 200 frames, providing a stable reference for the tracker to associate. This leverages the detect-and-track paradigm of SAM 3 while replacing its interactive concept interface with an automatic, query-driven one.

The paper's ambition, as stated in the concluding paragraph of the introduction, is to provide "a simple yet effective step toward more semantic, scalable, and general-purpose segmentation systems." The emphasis is on the interface — how language reasoning connects to pixel prediction — as the critical design dimension, rather than on improving the individual components (better language model, better mask decoder, better tracker). By making that interface explicit, interpretable, and hierarchically structured, SETCON aims to unlock capabilities that are architecturally inaccessible to the prevailing paradigm, particularly in the open-ended, multi-target, cross-category scenarios where that paradigm demonstrably struggles.

3. Technical Approach

SETCON is primarily a systems and representation-design paper whose core idea is that open-ended referring segmentation should be formulated as explicit set-level concept prediction, where an LVLM generates interpretable natural-language concepts organized hierarchically, and these concepts serve as semantic conditions for joint mask-set decoding — replacing the implicit, non-interpretable [SEG] token interface that encodes spatial position more than semantic category and degrades sharply as the number of referred targets increases.

3.1 Reader Orientation

SETCON builds an end-to-end framework that takes an image (or video clip) and an open-ended natural-language query (e.g., "Which objects in the image would be the best to pick first for a summer salad?") and produces a set of pixel-level masks with explicit semantic labels, where each mask corresponds to a coherent subset of the referred targets. The system solves the problem of open-ended, multi-target, cross-category referring segmentation by replacing the dominant architectural pattern — indistinguishable [SEG] tokens decoded independently — with an interpretable, language-grounded concept interface that organizes predictions hierarchically: a shared set-level concept defines what belongs in the answer, and fine-grained sub-category concepts discriminate between subsets, enabling joint decoding that respects set-level properties of completeness and mutual exclusivity.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components connected in a feedforward pipeline:

1. Large Vision-Language Model (LVLM) — receives the visual input (image or video frame) and the natural-language query, performs language reasoning, and generates a textual response containing hierarchically organized concepts. This is the "brain" that decides what to segment and how to name it.

2. Concept Delimitation and Extraction — identifies the spans of the textual response that correspond to concepts using special markers <ref>...</ref>, extracts the hidden states of the tokens within these spans, and routes them to the next stage as semantic representations. This is the "interface" between language reasoning and pixel prediction.

3. Projection Module — projects the LVLM's hidden states (which live in the language model's embedding space) into the feature space expected by the segmentation decoder. This is a learned mapping that aligns two different representational spaces.

4. Segmentation Model (SAM 3) — takes the visual input and the projected concept embeddings as semantic conditions, and jointly decodes a mask set. The image encoder and detector decoder are frozen; only the multimodal decoder is trainable. This is the "hand" that produces the actual pixel-level masks.

5. Concept-Mask Assignment and Labeling — takes the decoded masks and the sub-category concepts, performs Hungarian matching to align masks with concepts, and attaches interpretable semantic labels to each mask. For video, this component also handles temporal propagation via SAM 3's tracker.

Information flows as follows: Visual input + query → LVLM generates concept-bearing response → concept spans are extracted and their hidden states projected → projected concept embeddings condition the segmentation model → mask set is decoded jointly → masks are matched to sub-category concepts and labeled. For video, the LVLM is invoked once per clip; its concepts are broadcast to all frames; and SAM 3's memory-based tracker handles temporal propagation.

3.3 Roadmap for the Deep Dive

• First, the explicit set-level concept prediction mechanism — how the LVLM generates concepts, how they are extracted from the response, and how they serve as semantic conditions — because this is the core representation innovation that distinguishes SETCON from all prior work.
• Second, the hierarchical semantic decomposition — the coarse-to-fine structure of set-level and sub-category concepts, how they interact during decoding, and why this structure matters for completeness and discrimination — because this is the organizing principle that enables the model to handle multi-target, cross-category queries.
• Third, the training procedure — the loss functions, the optimization configuration, the handling of no-target samples, the video sampling strategy, and the LoRA adapter choices — because these design decisions determine whether the concept interface can be learned effectively.
• Fourth, the inference procedure — how the mask sets are decoded, how sub-category labels are assigned, and how the video tracking pipeline works — because the inference-time orchestration reveals what the trained system actually does.
• Fifth, the hierarchical semantic annotation pipeline — the two-stage process for creating training supervision, the quality control mechanisms, and the resulting dataset statistics — because this dataset enables the concept-based training and its properties shape what the model can learn.
• Sixth, the DETR-style set-prediction loss — the formal loss function that governs mask decoding — because this is the mathematical mechanism by which set-level properties (completeness, mutual exclusivity) are enforced during training.

3.4 Detailed, Sentence-Based Technical Breakdown

3.4.1 Explicit Set-Level Concept Prediction

The fundamental design departure in SETCON is the replacement of the implicit [SEG] token interface with an explicit language-grounded concept interface. In the prevailing paradigm (LISA, Sa2VA, PixelLM, GSVA), the LVLM generates one or more special [SEG] tokens in its textual output. These tokens are newly added to the vocabulary — they are initialized randomly and trained from scratch during fine-tuning. Each [SEG] token's hidden state h_seg is a single dense vector that is separately fed into a mask decoder D to produce a mask m = D(V, h_seg), where V is the visual input. The semantic content of what is being segmented is not linguistically expressed; it is compressed into a learned, non-interpretable vector.

SETCON replaces this with a process where the model explicitly names each referred target group using its own vocabulary. Specifically, the LVLM generates a textual response R in which each referred concept C is expressed as a free-form natural-language phrase and delimited by special markers <ref>...</ref>. For example, the model might generate:

"Essential gear for a baseball game <ref>( includes baseball bat , baseball helmet and catchers mitt )</ref>"

where the content between the markers is the concept phrase. Crucially, each concept is bound to an entire set of semantically coherent instances, not to a single target. This means the model's output text explicitly encodes the granularity of set-level mask prediction — one concept = one mask set.

How concepts are extracted and used. The hidden states of the tokens within each <ref>...</ref> span are collected. These are not single vectors but sequences of token-level hidden states from the LVLM's final layer. Let H = [h_1, h_2, ..., h_T] be the sequence of hidden states for the T tokens in a concept phrase. These hidden states are then projected into the segmentation decoder's feature space through a learned projection module to produce \tilde{H} = [\tilde{h}_1, \tilde{h}_2, ..., \tilde{h}_T]. The projected features \tilde{H} serve as semantic conditions — the decoder uses them to determine what visual content to mask, analogous to how a text prompt conditions a text-to-image model's generation, but here applied to mask prediction.

The key difference from [SEG] tokens. The concept interface is not merely a cosmetic change from one token to many. It provides three architectural advantages over the [SEG] token design:

1. Interpretable semantic anchors. Every mask is grounded in a human-readable concept phrase. If the model segments a "baseball bat," the concept phrase literally says "baseball bat." This makes the model's reasoning transparent: you can inspect the generated text to understand why the model segmented what it did. In contrast, a [SEG] token provides no such interpretability — it is an opaque vector.

2. Reuse of pretrained semantic geometry. The embeddings of the concept words ("baseball", "bat", "helmet", "catcher's", "mitt") already exist in the LVLM's pretrained embedding space, where they are organized by semantic similarity. Words for sports equipment are closer to each other than to words for food or furniture. When these embeddings are projected into the decoder, the decoder inherits this semantic structure without having to learn it from scratch. In contrast, [SEG] tokens are newly initialized — their embeddings start random and must learn all semantic structure from the fine-tuning data alone, which is why the pilot study (Figure 2b) found that they settle on spatial position (an easier signal to learn) rather than semantic category.

3. Variable-length, compositional representations. A concept phrase can be a single word ("catcher's mitt") or a short descriptive phrase ("ripe, red tomatoes"), and the representation is distributed across multiple token positions — the decoder receives not one vector but a sequence. This provides richer representational capacity than a single [SEG] token vector, which must compress all semantic and spatial information into one fixed-dimensional point. The distributed representation can allocate different aspects of the concept (category, attributes, relations) to different token positions.

Concept expressiveness for open-ended scenarios. A crucial design property is that the concept phrases are free-form natural language, not a fixed taxonomy. The paper states that concepts "average 3.5 words" and describes them as "natural-language expressions rather than closed-set labels." This means the model is not constrained to a predetermined vocabulary of object categories (e.g., the 80 COCO classes). It can invent concept descriptions on the fly that match the query's semantics and the visual context. For instance:

• "the fresh, green cucumber" (category + visual attributes)
• "essential gear for a baseball game" (functional role)
• "objects that combine heat-producing equipment, fuel or compressed gas, and ventilation-related infrastructure" (compositional functional description)

This expressiveness is what makes the framework suitable for open-ended, cross-category queries. The target set's identity is determined by the query and the image content, not by a pre-specified class list. The LVLM, by virtue of its pretraining on vast text corpora, can generate concept descriptions that capture fine-grained, context-dependent semantics that no fixed taxonomy could enumerate.

How concepts relate to masks: set-level binding. The paper makes a deliberate and non-obvious design choice: each concept is bound not to a single instance mask, but to an entire set of semantically coherent instances. For example, the concept "baseball bat" might correspond to two masks (if there are two bats in the image), and the concept "catcher's mitt" might correspond to one mask. This set-level binding is important for two reasons:

• Alignment with query granularity. Open-ended queries refer to groups of objects, not individual instances. "The ingredients for a salad" is a group, and within that group, "tomatoes" is a subset that may contain multiple instances. Binding concepts to sets rather than instances means the model's output structure mirrors the query's semantic structure.

• Efficient representation. Instead of generating a separate [SEG] token (or a separate concept phrase) for every individual instance — which would require the model to count instances and produce a variable-length sequence — the model generates one concept per semantic subset, and the mask decoder produces however many masks that subset requires. The cardinality is handled in the pixel space (where the decoder can easily produce multiple disconnected masks) rather than in the token space (where predicting the exact count is an error-prone autoregressive process).

3.4.2 Hierarchical Semantic Decomposition

The explicit concept prediction provides semantic anchors for mask sets, but for complex queries spanning multiple sub-categories, a flat list of concepts is insufficient. The paper identifies a specific failure mode: "Enumerating every sub-category as a flat sequence of concepts tends to yield incomplete target sets." The reason is that a flat list has no mechanism to define the boundary of the target set — which objects belong in the answer at all, versus which are distractors. Each sub-category concept in isolation ("cucumber," "tomato," "bell pepper") tells the model what to include within that subset, but nothing tells it that these three subsets collectively exhaust the answer. The model might miss the "bell pepper" because nothing in the architecture signals that it must comprehensively cover all subsets implied by the query.

SETCON addresses this through a hierarchical decomposition that organizes concepts into two levels:

• Set-level concept: A single, global concept C_0 that summarizes the shared semantics of the entire target group. The paper describes it as "an 8–15 word free-form description of the scenario, behavior, or purpose that the targets collectively express" (Appendix B). Examples from the paper include: "essential gear for a baseball game," "the main entertaining setup," "items that provide the necessary means to first pinch and strip the protective insulation off an electrical cable." This concept answers the question: "What is the overall scope of things I should be looking for?"

• Sub-category concepts: A list of fine-grained phrases {C_1, C_2, ..., C_N} that decompose the set-level concept into semantically coherent subsets. Examples decomposing "essential gear for a baseball game": "baseball bat," "baseball helmet," "catcher's mitt." These answer the question: "Within that scope, what are the distinct types of things?"

How the two levels interact during decoding. This is the most architecturally nuanced part of SETCON and where the hierarchical structure becomes more than just a labeling scheme. The set-level concept and sub-category concepts are used differently in the mask decoding pipeline:

1. The set-level concept's hidden states H_0 (after projection) serve as the global semantic condition \tilde{H}_0. This is the representation that defines the overall target scope. It is used to produce a primary mask set M_0 — a set of binary masks that collectively covers all targets that satisfy the query, without yet distinguishing between sub-categories. Essentially, M_0 is the answer to "which pixels belong to any referred target?"

2. Each sub-category concept's hidden states H_i are fused with \tilde{H}_0 to produce the fused condition \tilde{H}_i^{\text{fused}}. The paper states: "each sub-category representation is fused with \tilde{H}_0 so that fine-grained predictions are conditioned on the shared semantic scope." The fusion operation is not detailed explicitly in the paper (whether it is concatenation, addition, cross-attention, or a learned combination), but the principle is clear: the sub-category concept tells the decoder which subset of the total target set to output, while the set-level context prevents it from including objects outside the intended scope.

3. The fused conditions {\tilde{H}_i^{\text{fused}}}_{i=1}^N are used to produce sub-category mask sets {M_i}_{i=1}^N, where each M_i is the set of masks belonging to the i-th sub-category. These mask sets are mutually exclusive (each target belongs to exactly one sub-category) and collectively exhaustive of M_0.

4. During training: The primary mask set M_0 and the sub-category mask sets {M_i} are all supervised. M_0 is trained to match the union of all ground-truth target masks (the full referred set). Each M_i is trained to match the ground-truth masks of the corresponding sub-category. This joint supervision ensures that the set-level concept learns to define the correct scope, and the sub-category concepts learn to partition it correctly.

5. During inference: The segmentation model D separately decodes \tilde{H}_0 and the fused \tilde{H}_1, ..., \tilde{H}_N. The sub-category mask sets are unioned: \cup_i M_i. This union is then aligned with M_0 via mask-level Hungarian matching — a bipartite matching algorithm that finds the optimal one-to-one correspondence between the masks in \cup_i M_i and the masks in M_0. After matching, each primary mask inherits the sub-category label from its matched partner. Masks in \cup_i M_i that have no good match in M_0 are discarded; masks in M_0 that have no match in \cup_i M_i remain unlabeled. This alignment step is the operational mechanism that enforces completeness (the union of sub-category masks should cover M_0) and mutual exclusivity (each mask in M_0 is assigned to exactly one sub-category).

Why the hierarchical structure matters. The paper provides empirical evidence in Figure 4(b): when they ablate the hierarchical decomposition and instead use a flat list of concepts (no set-level concept, each sub-category decoded independently), the F1@0.5 score drops on MUSE, with the gap widening as the number of referred categories increases. For queries with five categories, the hierarchical variant substantially outperforms the flat variant. This demonstrates that the set-level concept provides a shared semantic context that becomes increasingly important as the target set grows more complex — without it, the model struggles to maintain coherent coverage of all subsets and avoid including distractor objects.

The hierarchical structure also has a representational interpretation in terms of the concept embeddings. The set-level concept embedding \tilde{H}_0 anchors the sub-category embeddings \tilde{H}_i in a common semantic region of the embedding space. In the ablation without hierarchy, the sub-category embeddings are unanchored — each independently decoded — and there is no architectural mechanism pushing them toward a consistent scope. With hierarchy, the fusion with \tilde{H}_0 creates a soft constraint that the decoded masks for different sub-categories should collectively respect the set-level boundary.

3.4.3 Training Procedure

SETCON is trained end-to-end on a mixture of image and video segmentation datasets. The training procedure is designed to jointly optimize the LVLM's language generation, the projection module, and the segmentation decoder, while keeping the image encoder and detector decoder frozen (they are inherited from the pretrained SAM 3).

Model components and backbone. The LVLM backbone is Qwen3-VL-8B-Instruct, a large vision-language model with 8 billion parameters. The segmentation model is SAM 3, which provides an image encoder, a detector decoder, a multimodal decoder, and (for video) a memory-based tracker. During training, the multimodal decoder is trainable while the image encoder and detector decoder remain frozen. This means SAM 3's visual feature extraction and base detection capability are preserved; only the high-level reasoning-to-mask interface is fine-tuned.

LoRA adapters for the LVLM. To efficiently fine-tune the LVLM without updating all 8 billion parameters, the paper uses Low-Rank Adaptation (LoRA) with the following configuration:

• Rank: 128
• Alpha: 256 (the scaling factor for the LoRA updates relative to the pretrained weights)
• Dropout: 0.05
• LoRA adapters are attached to all linear layers of the LVLM's language model.

This means that the LVLM's attention query/key/value projections, feed-forward layers, and output projection all receive low-rank trainable updates, while the base weights remain frozen. The total number of trainable parameters is substantially smaller than full fine-tuning (the exact count is not reported, but with rank 128 across all linear layers of an 8B model, it is likely in the range of tens to low hundreds of millions), making training more memory-efficient and reducing catastrophic forgetting of the pretrained capabilities.

Trainable components in full. Besides the LoRA adapters, the following are jointly optimized:

• The token embeddings of the LVLM — these are trained so that the <ref> and </ref> delimiter tokens learn meaningful representations.
• The language-modeling head of the LVLM — the final linear layer that maps hidden states to vocabulary logits is trained so the model learns to generate concept-bearing responses in the correct format.
• The projection module — the learned mapping from LVLM hidden states to the segmentation decoder's feature space.
• The multimodal decoder of SAM 3 — the component that takes visual features and projected concept conditions to produce mask predictions.

Loss functions. The paper states that SETCON is trained with two losses applied jointly:

1. An auto-regressive language-modeling loss on the response R. This is the standard next-token-prediction cross-entropy loss that trains the LVLM to generate concept-bearing text in the correct format. It ensures the model learns what concepts to name and how to structure them hierarchically.
2. A DETR-style set-prediction loss on the mask predictions of each concept group. This is the same loss function used in DETR and its segmentation variants (Mask2Former, etc.) and consists of a bipartite matching step followed by a combination of classification and mask losses. It ensures the decoded masks correspond to the correct ground-truth targets. This loss is applied to both the primary mask set M_0 and each sub-category mask set M_i.

Handling no-target samples. Some queries legitimately have no valid target in the image (e.g., "Which animals are in this picture of an empty room?"). To teach the LVLM to handle such cases, the paper uses a training strategy where, for no-target samples:

• In 50% of the cases, the ground-truth concept is replaced with the phrase "no target." This teaches the LVLM that sometimes the correct answer is to state explicitly that nothing matches.
• In the remaining 50%, the original concepts are kept. This teaches the segmentation decoder that it may receive concept conditions even when there are no matches and should learn to output empty mask sets (abstention).

This mixed strategy prevents the model from learning a degenerate solution where it always predicts "no target" (from the 50% replacement) or always hallucinates masks (from the 50% keeping). The paper describes this as "encouraging the LVLM and segmentation model D to learn abstention from complementary signals."

Video training. For video clips, the paper uses a specific sampling strategy: "we randomly sample sparse frames per clip and share a single concept response across them, as the referred semantics remain consistent over time." This means:

• For a video clip of, say, 200 frames, the training procedure samples a small number of frames (the exact number is not specified in the main paper — likely 2–4 frames, consistent with common practice in video segmentation training to fit within GPU memory).
• The LVLM is invoked once for the clip and generates a single concept-bearing response. This response is used as the semantic condition for all sampled frames.
• The set-prediction loss is computed per-frame, encouraging the decoder to produce consistent masks for the same concept across different frames.

This training design mirrors the inference-time design: the LVLM processes the clip once, concepts are shared, and per-frame decoding produces masks that the tracker then associates over time. By training on sparse frames, the model learns to produce concept embeddings that generalize across the visual variations of the same object in motion, which is critical for the tracker to maintain identity consistency.

Optimization hyperparameters. The full optimization configuration is:

• Optimizer: AdamW
• Learning rate: 4 × 10^{-5}
• Learning rate schedule: cosine annealing (decaying from the initial rate to near zero over the course of training)
• Batch size: 64
• Number of epochs: 1
• Hardware: 8 NVIDIA H200 GPUs

The single-epoch training is notable. Given the dataset size (236,396 samples across both image and video, as described in the annotation section), one epoch with batch size 64 processes roughly 3,700 iterations. This is a relatively short training run, made feasible by the fact that most of the model (the SAM 3 encoder and detector, the LVLM base weights) is frozen, and the LoRA adapters are parameter-efficient. The short training also reduces the risk of overfitting to the fine-tuning data and helps preserve the LVLM's pretrained open-world knowledge.

Training data composition. The training set combines:

• Image datasets: RefCOCO, RefCOCO+, RefCOCOg (standard single-target referring), gRefCOCO (multi-target referring), ReasonSeg (reasoning segmentation), and MUSE (multi-target open-ended reasoning segmentation).
• Video datasets: MeViS (motion expression video segmentation), Ref-DAVIS, Ref-YouTube-VOS, and ReVOS (reasoning video object segmentation).

The referring datasets (RefCOCO/+/g, gRefCOCO) are used with their original natural-language queries, since those already specify target categories explicitly. The reasoning segmentation datasets (ReasonSeg and MUSE) are augmented with hierarchical semantic annotations via the two-stage pipeline described in Section 3.4.5. This creates a training corpus where every sample has both the original query and the hierarchical concepts, enabling the model to learn the mapping from open-ended queries to structured concept output.

3.4.4 Inference Procedure

At inference time, SETCON follows a specific orchestration that produces the final labeled mask set. The procedure differs slightly between image and video inputs, but the core concept-to-mask decoding is shared.

Image inference. For a single image, the pipeline is:

1. LVLM generates the response. The LVLM takes the image and the query and generates a textual response R containing the set-level concept C_0 (delimited by <ref>...</ref>) and the sub-category concepts {C_1, ..., C_N} (each also delimited by <ref>...</ref>). The number of sub-categories N is "freely determined by the LVLM rather than fixed in advance" — the model decides how many subsets are needed based on the query and image content.

2. Concept extraction and projection. The hidden states of the tokens in each concept span are extracted and projected into the decoder feature space, yielding \tilde{H}_0 for the set-level concept and {\tilde{H}_i} for the sub-category concepts.

3. Mask decoding. The segmentation model D separately decodes:

   • \tilde{H}_0 to produce the primary mask set M_0. This is a set of binary masks covering all pixels that satisfy the query.
   • Each fused sub-category condition to produce sub-category mask sets {M_i}. Each M_i is a set of binary masks for the i-th sub-category.

   The paper does not specify the exact mechanism by which D produces multiple masks from a single condition embedding \tilde{H}_0. In standard DETR-style architectures, the decoder outputs a fixed number of mask predictions (e.g., 100 object queries), each with an associated confidence score. Masks with confidence below a threshold are discarded. The paper mentions a threshold of 0.7 for the multi-object benchmarks: "we discard predicted masks whose confidence is below 0.7 before aggregation" (Appendix C). This suggests that D produces many candidate masks, and low-confidence ones are filtered out.

4. Mask alignment and labeling. The union of all sub-category masks \cup_i M_i is computed. This union is then aligned with the primary mask set M_0 via mask-level Hungarian matching. Hungarian matching is an algorithm for optimal bipartite matching that finds a one-to-one correspondence between two sets while minimizing a cost function. Here, the cost between a sub-category mask m_i \in \cup_i M_i and a primary mask m_0 \in M_0 is based on their overlap (typically Intersection-over-Union). The algorithm finds the pairing that maximizes overall overlap.

   After matching:

   • Each primary mask that is matched to a sub-category mask inherits that sub-category's concept phrase as its semantic label. For example, a primary mask matched to a mask from the "baseball bat" sub-category gets labeled "baseball bat."
   • Sub-category masks not matched to any primary mask are discarded (they are likely false positives from the sub-category decoder).
   • Primary masks not matched to any sub-category mask remain in the output but without a sub-category label (they are part of the target set but were not assigned to a specific sub-type).

5. Answer aggregation by benchmark. The final output format depends on which benchmark is being evaluated:

   • For gRefCOCO: All retained masks are merged into a single foreground mask, and the evaluation metric is gIoU and cIoU.
   • For MUSE: The retained mask set is aligned with the ground truth via Hungarian matching, and evaluation reports gIoU, cIoU (segmentation quality), and F1@0.5 (set-level detection performance, where a mask is considered a correct detection if its IoU with a ground-truth mask exceeds 0.5).
   • For RefCOCO/+/g and ReasonSeg: The highest-confidence prediction is selected as the final mask (single-target output), and cIoU is reported.

Video inference. For a video clip, the pipeline leverages the detect-and-track paradigm of SAM 3:

1. LVLM invoked once per clip. The video clip is processed by the LVLM, which generates a single concept-bearing response R. The paper describes the concepts as "motion-aware" because the LVLM sees motion information (multiple frames or a clip-level representation) and can generate concepts that reference dynamic properties (e.g., "the bicycle that is not moving" vs. "the one that is in motion" — an example from Figure 7).

2. Concepts are broadcast to all frames. The projected concept embeddings \tilde{H}_0 and {\tilde{H}_i} are computed once and shared across every frame in the clip. They serve as "persistent semantic anchors" — fixed reference points that do not change as the video progresses.

3. Per-frame detection. For each frame, the segmentation model D performs per-frame mask decoding using the shared concept embeddings. This produces per-frame mask sets, but because the semantic conditions are identical across frames, the masks for the same concept tend to be visually similar (same object categories, similar appearance).

4. Memory-based mask propagation. The per-frame masks are fed to SAM 3's off-the-shelf memory-based tracker, which propagates masks temporally. The tracker maintains a memory bank of past frame features and uses attention mechanisms to associate current-frame masks with past-frame masks of the same identity. This tracker operates "without any video-specific modification" — SETCON does not modify SAM 3's tracking mechanism; it only provides better semantic anchors for the tracker to work with.

5. Output aggregation. For video benchmarks (ReVOS, ReasonVOS, Ref-YTVOS, Ref-DAVIS, MeViS, Ref-SeCVOS), the union of all tracked masks in each frame is taken as the final foreground sequence, and the standard J&F metric (the average of Jaccard Index, measuring region similarity, and F-measure, measuring contour accuracy) is reported.

Why the video design matters. The key advantage of this detect-and-track formulation over per-frame independent prediction is identity consistency. In the [SEG] token paradigm applied to video, each frame would independently generate [SEG] tokens and decode masks. Since the [SEG] hidden states encode spatial position more than semantics (as shown in Figure 2b), the same object at different spatial positions across frames would produce different [SEG] representations, making it difficult for the tracker to associate them. By contrast, SETCON's concept "yellow headband" produces the same semantic condition regardless of where the headband is in the frame. The tracker receives a stable, semantically meaningful reference that remains constant even as the object moves, occludes, or changes appearance slightly. This explains the large gains on challenging video benchmarks like MeViS (+10.9 J&F) and Ref-SeCVOS (+12.4 J&F), where "stable semantic anchors are most beneficial for long-horizon temporal association."

3.4.5 Hierarchical Semantic Annotation Pipeline

Training the concept interface requires linguistic supervision at both the set and sub-category levels — specifically, for each training sample, the model needs to know: (1) what is the overall set-level concept that defines the target scope, and (2) what are the fine-grained sub-category phrases for each subset of targets. Existing reasoning segmentation datasets (ReasonSeg, MUSE) do not provide this. Their annotations are instance-level and closed-vocabulary: each target mask is tagged with a category from a fixed list (e.g., "person," "car," "chair"). They do not provide free-form sub-category descriptions, and they have no notion of a set-level concept that groups multiple targets under a shared semantic scope.

The paper constructs the missing supervision through a two-stage annotation pipeline that uses a stronger LVLM (Qwen3-VL-235B-A22B, a 235B-parameter model with 22B active parameters via mixture-of-experts) as the annotator, followed by rule-based filtering and manual spot-checking. The pipeline takes as input: the image, the original query, the per-target instance masks, and the source closed-vocabulary labels. It produces: (i) a free-form sub-category phrase for each target, and (ii) a global set-level concept summarizing the target scope.

Stage 1: Diverse Sub-category Labeling. For each individual target, the annotator LVLM is given two views:

• The original image.
• The same image overlaid with a colored mask highlighting the specific target — this spatial cue tells the annotator exactly which object to describe.

The annotator is also conditioned on the source closed-vocabulary label (e.g., "light," "bicycle"). This conditioning is critical and deliberately designed: it "suppresses semantic drift" — the annotator is not allowed to silently invent a new category, only to refine the existing label. The instruction is to produce a natural noun phrase of 1–6 words that may include visual attributes (color, material, shape) when these are clearly visible in the image.

Example transformations:

• Source label "light" → refined phrase "traffic light" (if the object is a traffic signal, not a lamp)
• Source label "bicycle" → refined phrase "red bicycle" (if the bicycle is red)
• Source label "bottle" → refined phrase "wine bottle" (if contextually it's clearly a wine bottle)

The sampling temperature is low (0.2) to "favor faithfulness over diversity." This means the annotator is conservative — it makes small, justified refinements rather than creative, potentially incorrect descriptions. The output is a free-form phrase that is more specific and contextually accurate than the source label but remains grounded in the ground-truth category.

Stage 2: Set-level Concept Synthesis. After Stage 1 produces per-target sub-category phrases, Stage 2 creates the global set-level concept. The annotator LVLM is given:

• The original image.
• The original query (e.g., "If you wanted to make a refreshing summer vegetable salad, which objects in the image would be the best to pick first?").
• The per-target sub-category phrases collected from Stage 1 (e.g., "green cucumber," "ripe, red tomato," "purple onion").

The instruction is to "summarize the overall scope of the referred targets into a single set-level concept, i.e., an 8–15 word free-form description of the scenario, behavior, or purpose that the targets collectively express." The output is constrained to lie within [5, 20] words:

• Outputs exceeding 20 words are truncated (hard cutoff).
• Outputs below 5 words are replaced by a "query-conditioned fallback template" — the paper doesn't specify the template, but it likely involves using the query itself or a generic description derived from it.

The sampling temperature is higher (0.7) to "encourage linguistic diversity." Here, diversity is desirable because the set-level concept is a creative summarization, not a factual labeling task. Two different annotator runs on the same input might produce different but equally valid phrasings ("the essentials for a refreshing summer salad" vs. "key salad ingredients for a summer meal"), and this diversity helps the trained model generalize.

Quality Control. The paper describes a multi-step post-processing pipeline that explicitly targets failure modes observed in practice. The quality control has three stages:

1. Label sanity check. Sub-category phrases are scanned for degeneracies through two complementary methods:

   • A rule-based pass catches "unambiguous degeneracies": system-level error tokens (e.g., OMPI, MPI_Init — MPI library initialization messages), residual Markdown markup (e.g., **, _), URLs, leading exclamation marks, and outputs shorter than 2 characters.
   • The remaining unique entries are aggregated into a global vocabulary and reviewed by a "text-only LLM judge" (likely a simpler language model without vision) that flags: non-English characters, gibberish, code snippets, full sentences (rather than phrases), apologetic or meta-commentary phrases (e.g., "I'm sorry, I cannot..."), and disallowed punctuation.
   • Flagged entries are collected into a global blacklist.

2. Targeted re-generation. For each sample whose sub-category phrases include a blacklisted entry:

   • The visual annotator LVLM is re-invoked with the same image and mask overlay to regenerate a replacement phrase.
   • This is preferred over "wholesale reverting them to the source label" because it preserves the linguistic richness of Stage 1 for the parts of the sample that were correctly annotated. Only the corrupted phrase is regenerated.

3. Within-sample de-duplication. A common failure mode is that two distinct sub-category groups under the same query get independently relabeled to the same phrase — e.g., two instances of "coffee cup" that are visually and functionally distinct should ideally have distinguishable descriptions. For each detected duplicate:

   • The annotator LVLM is asked to decide, given the image and the two highlighted mask groups, whether to:
     • Merge them into a single sub-category (if the source dataset over-split a coherent semantic group, or if the two groups truly share the same fine-grained semantics).
     • Split them into two distinguishable ones (if the two groups are visually or functionally distinct but happened to share the same closed-vocabulary source label).
   • If merged, the two mask groups are consolidated under one shared label.
   • If split, the duplicate labels are replaced with "distinctly worded alternatives."
   • A second de-duplication pass runs afterwards to catch any residual collisions from the resolution process.

Additionally, the paper conducts "multiple rounds of manual spot-checking on random subsets to verify annotation faithfulness." Systematic failure modes observed during these spot-checks prompted iterative refinement of the prompts and filters used in the automated pipeline.

Dataset statistics. The final corpus resulting from this pipeline:

• 236,396 samples total (spanning both the image reasoning datasets — ReasonSeg and MUSE — that were augmented).
• 784,809 concept phrases in total across all samples.
• Average of 2.32 sub-categories per sample, meaning most queries refer to more than one distinct type of object.
• Over 80% of samples involve more than one sub-category, confirming that the dataset provides the set-level supervision that prior corpora lack.
• Reaching up to 16 sub-categories in the long tail — for complex queries that refer to many distinct types.
• Average concept phrase length: 3.5 words, confirming that the output behaves as "natural-language expressions rather than closed-vocabulary tokens."

The distribution plots in Figure 8 show the per-sample sub-category count (long-tailed, mode at 2–3) and the per-concept word count (mode at 3–4 words). This dataset is the foundation that enables the concept-based training: without it, there would be no supervision signal for the LVLM to learn to generate set-level and sub-category concepts, and no ground truth for the mask decoder to learn the correspondence between concept embeddings and specific subsets of masks.

Why a two-stage pipeline? The paper could have attempted a single-stage approach where the LVLM generates both the sub-category labels and the set-level concept simultaneously. The two-stage design is deliberate for two reasons:

1. Anchoring prevents semantic drift. Stage 1 conditions on the source label, which provides a hard constraint that the refined phrase must match the ground-truth mask's category. If the LVLM were asked to generate sub-category labels from scratch without this anchor, it might invent new categories that don't correspond to any ground-truth mask, creating a misalignment between the linguistic supervision and the mask supervision.

2. Decomposition improves quality. Generating the set-level concept (a creative, summarizing task) is different from refining per-target labels (a factual, constrained task). Splitting them allows different temperatures (low for Stage 1, moderate for Stage 2) and different prompt designs optimized for each sub-task. A single-stage approach would need to balance these competing requirements, likely producing lower-quality output on one or both dimensions.

3.4.6 DETR-Style Set-Prediction Loss

The paper states that the mask predictions are trained with a "DETR-style set-prediction loss" but does not give the full mathematical specification in the main paper. However, based on the DETR framework (Carion et al., 2020) and its segmentation extensions, and the description in the paper, the following is the standard form of this loss.

The set-prediction loss operates on a set of predicted masks and a set of ground-truth masks, where the model produces a fixed number K of mask predictions (e.g., K = 100), each with an associated confidence score (or class probability), and the ground truth has a variable number G of masks (where G may be smaller than K). The loss consists of two steps:

Step 1: Bipartite matching. A one-to-one correspondence is established between the K predictions and the G ground-truth masks (plus K - G "no object" dummy matches). The matching is found by the Hungarian algorithm that minimizes a matching cost:

$C_{\text{match}}(i, j) = -\mathds{1}_{\{c_i \neq \emptyset\}} \cdot p_i(c_j) + \mathds{1}_{\{c_i \neq \emptyset\}} \cdot \mathcal{L}_{\text{mask}}(m_i, m_j^{\text{gt}})$

where i indexes a ground-truth mask (or the "no object" class), j indexes a prediction, p_i(c_j) is the predicted probability that prediction j belongs to class c_i, m_i and m_j^{\text{gt}} are the predicted and ground-truth masks, and \mathcal{L}_{\text{mask}} is a mask loss (typically a combination of Dice loss and binary cross-entropy).

What it computes: This cost is computed for every possible pairing of ground-truth targets and predictions. A low cost means the prediction is confident for the correct class and its mask matches the ground truth well. The Hungarian algorithm finds the globally optimal assignment that minimizes the sum of costs, yielding a one-to-one mapping where each ground-truth target is matched to exactly one prediction (or to a "no object" slot if no good prediction exists). Predictions not matched to any ground-truth target are assigned to the "no object" class.

Why this form: The bipartite matching step is essential because the model produces an unordered set of masks, and during training, there is no pre-specified correspondence between the n-th prediction and the n-th ground-truth target. Without matching, the loss would be computed based on arbitrary ordering, creating an unstable training signal where the model chases a moving target. Bipartite matching finds the best possible assignment under the current model state, providing a consistent optimization target. The use of class probability in the matching cost (not just mask quality) ensures that the model learns to predict the correct sub-category label for each mask, not just to segment any region that looks object-like.

Step 2: Loss computation on matched pairs. Once the optimal matching \sigma is found (where \sigma(i) is the index of the prediction matched to ground-truth target i), the loss is computed as:

$\mathcal{L} = \sum_{i=1}^{G} \left[ -\log p_{\sigma(i)}(c_i) + \mathds{1}_{\{c_i \neq \emptyset\}} \cdot \mathcal{L}_{\text{mask}}(m_{\sigma(i)}, m_i^{\text{gt}}) \right]$

where the first term is the classification loss (negative log-likelihood of the correct class for the assigned prediction) and the second term is the mask loss for the matched prediction. For predictions matched to "no object" (c_i = \emptyset), only the classification loss applies (they are trained to predict the "no object" class with high confidence), which discourages the model from predicting masks where there are none.

What it computes: For each ground-truth target, the model incurs a penalty if the matched prediction has low confidence for the correct class (classification loss) or if the predicted mask is a poor match for the ground-truth mask (mask loss). The mask loss \mathcal{L}_{\text{mask}} is typically:

$\mathcal{L}_{\text{mask}}(m, m^{\text{gt}}) = \lambda_{\text{dice}} \cdot \mathcal{L}_{\text{Dice}}(m, m^{\text{gt}}) + \lambda_{\text{BCE}} \cdot \mathcal{L}_{\text{BCE}}(m, m^{\text{gt}})$

where the Dice loss penalizes poor overlap (1 minus the Dice coefficient, which ranges from 0 for no overlap to 1 for perfect overlap) and the binary cross-entropy loss penalizes pixel-wise classification errors. The Dice loss is particularly important for segmentation because it is insensitive to class imbalance — in a typical image, most pixels are background, and BCE alone would be dominated by easy negative examples.

Why this form: The set-prediction loss enforces the set-level properties that are central to SETCON's design:

• Mutual exclusivity is enforced indirectly: if two predictions both claim the same ground-truth mask, the Hungarian matching will assign one to it, and the other will be assigned to "no object" (or to a different target). The one assigned to "no object" will be penalized for predicting a mask where there shouldn't be one, encouraging it to reduce its confidence. Over training, this pressure causes the model to produce non-overlapping masks for distinct targets.

• Completeness is enforced by the bipartite matching penalizing unmatched ground-truth targets: if the model produces too few masks to cover all ground-truth targets, the unmatched targets cannot be assigned to any prediction, which means they cannot contribute to the loss (they are effectively ignored, which is suboptimal because the model misses them). In DETR, this is handled by having the model produce a fixed number of predictions (typically more than the maximum expected number of targets), and predictions unmatched to any ground truth are assigned to "no object." The model's incentive to cover all targets comes from the fact that each ground-truth target can be matched to at most one prediction, and the best-matching predictions are selected. To minimize total loss, the model must produce enough high-quality predictions to cover all targets.

• Explicit cardinality prediction is avoided. Instead of learning to predict the exact number of targets (which is an error-prone counting task for autoregressive models), the model always produces a fixed large set of candidate masks, and the confidence threshold at inference time (0.7 for multi-object benchmarks) removes low-confidence predictions. This decouples cardinality estimation from mask quality: the model can over-produce candidate masks and let the confidence filter prune them.

In SETCON, this loss is applied separately to the primary mask set M_0 (matched against the union of all ground-truth masks) and to each sub-category mask set M_i (matched against the ground-truth masks of that specific sub-category). This joint application ensures that the set-level concept learns to produce masks covering all targets, while each sub-category concept learns to produce masks for its specific subset within that coverage.

4. Key Insights and Innovations

Innovation 1: The [SEG] Token Interface Encodes Spatial Position More Than Semantic Category — A Diagnostic Finding That Motivates the Entire Reformulation

The paper's most intellectually distinctive contribution is not the SETCON architecture itself, but the diagnostic analysis in Section 3.1 that reveals why the prevailing paradigm fails under open-ended conditions. Prior work on LVLM-based segmentation — from LISA through Sa2VA, GSVA, PixelLM, and their video extensions — universally adopted the [SEG] token as the interface between language reasoning and pixel prediction. This design was inherited, not questioned. The field's implicit assumption was that the [SEG] token's hidden state would learn to encode whatever information was necessary for mask decoding — identity, appearance, spatial location, relational context — compressed into a single dense vector. There was no empirical investigation of what these vectors actually represented.

The paper's t-SNE analysis in Figure 2(b) constitutes a conceptual intervention. By coloring the projected [SEG] embeddings by semantic category and spatial position and observing that clusters align "more clearly with position than with category," the authors make visible a representational collapse that had been invisible — and consequential — throughout the prior literature. A [SEG] token for a "car" on the left side of the image is representationally closer to a [SEG] token for a "chair" on the left side than to a [SEG] token for another "car" on the right. This is not merely a curiosity; it is an architectural diagnosis. It means the interface that is supposed to bridge semantics and pixels has, in practice, settled into a spatial coordinate encoder that happens to work for single-target queries (where the language description already disambiguates identity and the token just needs to localize) but becomes fundamentally inadequate when the model must distinguish multiple semantically distinct targets — as in the open-ended multi-target settings of MUSE and gRefCOCO.

This finding is significant beyond its role as motivation for SETCON. It provides an explanatory framework for why prior multi-target extensions of the LISA paradigm (PixelLM predicting multiple [SEG] tokens, GSVA predicting [SEG] sequences) showed limited scaling: they were operating in a representational space that was structurally misaligned with the semantic demands of the task. The paper's accompanying finding in Figure 2(a) — that performance degrades sharply with target count — is the behavioral correlate of this representational diagnosis. Together, they form a coherent failure analysis: the model's internal representations cannot discriminate fine-grained semantic categories for multiple targets because they are organized around spatial proximity rather than semantic similarity, and this manifests as increasingly poor segmentation as the number of distinct targets grows.

This is an incremental but fundamental contribution. The individual techniques (t-SNE visualization, stratified performance analysis) are standard, but the inference drawn — that the interface itself, not the training data or loss function, is the bottleneck — reframes the problem from "how do we make this architecture work better?" to "this architecture has a structural limitation that we must replace, not patch." That reframing is what licenses the paradigm shift to explicit concept prediction, and it would remain valid as a diagnostic insight even for researchers who might pursue different solutions to the identified problem.

The evidence anchoring this claim is Figure 2, supported by the quantitative degradation in Figure 2(a). The finding's strength lies in its specificity: it names not just that the paradigm fails, but how it fails at the representational level, and that specificity provides clear design guidance (the replacement must encode semantics, not just location).

---

Innovation 2: Replacing Implicit Token Interfaces with Explicit Language-Grounded Concepts from the Model's Own Vocabulary

The architectural contribution of SETCON — generating natural-language concept phrases rather than [SEG] tokens — could be read as a straightforward substitution: replace one special token with a sequence of regular tokens. But the innovation is deeper, and it operates at the level of representation reuse versus representation relearning.

In the [SEG] token paradigm, the LVLM's contribution to mask decoding is an opaque vector from a token that was newly initialized — its embeddings and the pathways for producing its hidden state were trained from scratch during fine-tuning. The LVLM's pretrained semantic knowledge — the fact that "cucumber" and "tomato" are nearby in embedding space, that "salad ingredients" is a valid superordinate category, that "catcher's mitt" is semantically distinct from "baseball bat" but both belong to "baseball equipment" — is effectively discarded at the interface. The model must relearn all these relationships from the fine-tuning data, compressed into a single vector, competing with the easier-to-learn spatial signal that dominates the representation.

SETCON's concept interface changes this by reading semantic information directly from the LVLM's own vocabulary. When the model generates the phrase "ripe, red tomatoes," the token embeddings for "ripe," "red," and "tomatoes" already occupy positions in the LVLM's pretrained semantic space that encode their meanings, relationships, and distinctions. The projection module maps these into the decoder's feature space, but the structure of the representational space — the distances and directions that encode semantic similarity — is inherited from pretraining, not learned anew. This is a form of transfer learning at the interface: the decoder receives conditions that already encode rich semantic distinctions, and it need only learn to map those conditions to visual features, not to construct the semantic distinctions themselves from scratch.

This contribution connects to a broader insight about multimodal architectures: when a model needs to connect two modalities (language and vision, here), placing the interface at a point where one modality already has rich structure (the LVLM's semantic space) is more efficient than placing it at a point where both modalities must learn their structure jointly (newly initialized tokens). The [SEG] token approach does the latter; SETCON does the former. The t-SNE evidence in Figure 4(a) provides direct support: concept-based representations "form clearer concept-level clusters" than token-based representations, confirming that the pretrained semantic geometry indeed transfers through the projection and organizes the decoder's input space.

This is a fundamental architectural reframing rather than an incremental improvement. It changes what kind of information the interface carries (from implicit, opaque, spatially dominated vectors to explicit, interpretable, semantically structured sequences) and where that information comes from (from relearning during fine-tuning to reuse of pretrained knowledge). The significance extends beyond referring segmentation: any LVLM-based system that needs to connect language reasoning to structured visual output (grounded conversation, visual question answering with spatial referents, embodied instruction following) faces the same interface design question, and SETCON's concept-based approach provides a template for making that interface semantically explicit.

The quantitative evidence for this innovation's impact is distributed across Tables 1, 2, and 5, but the most targeted evidence is the ablation in Table 3: moving from the implicit-token baseline to set-level prediction (still without explicit concept conditioning) provides a substantial gain (+7.8 gIoU on MUSE), and adding explicit concept conditioning provides a further gain (+1.8 gIoU). This separates the contribution of set-level modeling from the contribution of explicit semantic grounding, showing that both matter independently.

---

Innovation 3: Hierarchical Semantic Decomposition as an Architectural Mechanism for Enforcing Set-Level Properties

The third conceptual contribution is the hierarchical organization of concepts — a set-level concept that defines the target scope, decomposed into sub-category concepts that partition it. At first glance, this appears to be an annotation strategy (labeling data with two levels of description). But in SETCON, it functions as an architectural mechanism for addressing the set-level properties — completeness and mutual exclusivity — that the per-token paradigm neglects.

The key insight is that a flat list of sub-category concepts provides no signal about boundary: which objects are in the target set versus out of it. Each sub-category concept in isolation tells the decoder "segment things that look like X," but nothing tells it "segment only things that belong to the set {X, Y, Z} and exclude everything else." In the per-token paradigm, this boundary is left entirely to the language model's autoregressive generation: the model generates tokens for targets it remembers to include, and if it forgets one, there is no recovery mechanism.

The hierarchical decomposition introduces an explicit scope condition — the set-level concept \tilde{H}_0 — that is shared across all sub-category predictions. Because \tilde{H}_0 is trained to produce masks covering the union of all targets, it encodes the full extent of what should be included. The sub-category concepts then operate within this scope: each \tilde{H}_i is fused with \tilde{H}_0, conditioning fine-grained predictions on the shared boundary. This creates an architectural separation of concerns: the set-level concept handles the "what belongs?" question (scope determination), and the sub-category concepts handle the "which subset is this?" question (discrimination within scope).

This is fundamentally different from prior approaches to multi-target segmentation. Methods like PixelLM or GSVA that predict multiple [SEG] tokens treat each token as an independent prediction, with no shared conditioning that defines the target set's boundary. The DETR-style set-prediction loss provides some pressure toward completeness (unmatched ground-truth targets incur implicit cost), but this is a weak signal delivered through the loss function, not an architectural constraint. SETCON's hierarchical fusion makes the boundary an explicit, shared condition that every sub-category decoder must respect — a stronger form of constraint that operates through the forward pass rather than just the backward gradient.

The evidence for this innovation's impact is Figure 4(b): the hierarchical variant consistently outperforms the flat variant on MUSE, with the margin widening as the number of referred categories increases. This is exactly the pattern one would expect if the hierarchical mechanism provides a shared scope that becomes more valuable as the target set grows more complex. For single-category queries (leftmost point in Figure 4b), the flat and hierarchical variants perform similarly, because the scope is trivially defined by the single category. For five-category queries (rightmost point), the gap is substantial, because without the shared scope, the model struggles to maintain coherent coverage across all subsets.

This contribution is fundamental in its architectural reasoning (it introduces a new organizing principle — hierarchical scope conditioning — for connecting language to masks) but incremental in its technical realization (the fusion mechanism is a standard operation, and the hierarchical annotation pipeline is model-assisted). Its significance lies in identifying what kind of structure the interface needs (hierarchical, scope-defining) to handle complex sets, not in the novelty of any single component.

---

Innovation 4: Unified Image-and-Video Framework Through Stable Semantic Anchors

SETCON's video extension is not simply an application of the image method to video frames. It embodies a conceptual insight about what makes video referring segmentation different from image referring segmentation, and why the concept interface is particularly well-suited to the video domain.

In the [SEG] token paradigm applied to video, each frame independently generates tokens and decodes masks. The [SEG] hidden states are frame-specific: they encode the spatial position of the target in that frame, which changes as the object moves. The tracker must then associate masks across frames based on visual similarity and motion cues, but the semantic reference point — "this is the yellow headband" — is not stable; it is re-derived at each frame from a fresh [SEG] token whose representation is contaminated by the changing spatial signal. This creates a drifting semantic anchor problem: the very thing that should stay constant (the identity of the target) is re-computed from noisy, spatially dominated representations at every timestep.

SETCON's concept interface solves this by generating the concept once per clip and broadcasting it to all frames. The concept phrase "yellow headband" produces the same projected embedding regardless of frame, providing a stable semantic reference that does not drift with object motion. The tracker can then rely on visual continuity plus this stable semantic reference for identity association. This is what the paper means by "persistent semantic anchors" — anchors that persist not just across space (multiple instances in one image) but across time (the same instance across multiple frames).

This is more than an efficiency optimization (invoking the LVLM once rather than per-frame). It is a structural alignment of the representation with the task's temporal invariance. The identity of a referred target — "the yellow headband" — does not change from frame 1 to frame 200, even as its spatial position, scale, occlusion state, and appearance all change. The concept interface makes this invariance explicit in the model's architecture: the semantic condition is constant across time, and only the visual features vary. The [SEG] token interface, by contrast, violates this invariance: the semantic condition is recomputed per frame and varies spuriously with spatial position.

The quantitative evidence for this innovation is concentrated in Table 2, with gains of +10.9 J&F on MeViS and +12.4 J&F on Ref-SeCVOS — the two benchmarks that are explicitly designed to be challenging for long-horizon temporal association. The paper explicitly ties these gains to the stable anchor property: "stable semantic anchors are most beneficial for long-horizon temporal association." The magnitude of these gains — substantially larger than those on standard referring video benchmarks like Ref-DAVIS (+4.9 J&F) — supports this interpretation: the benefit scales with the temporal difficulty of the benchmark.

This contribution is fundamental in its reconceptualization of the video referring segmentation problem (from per-frame semantic grounding to clip-level semantic anchoring) but incremental in its implementation (it leverages SAM 3's existing detect-and-track pipeline without modification). Its significance extends beyond SETCON: any video understanding system that needs to track semantically specified targets across time faces the same semantic anchor stability problem, and the concept-broadcasting strategy provides a general template.

5. Experimental Analysis

Evaluation Methodology

• Datasets. The paper evaluates on thirteen benchmarks spanning image and video domains. For images: RefCOCO (val, testA, testB splits), RefCOCO+ (val, testA, testB), RefCOCOg (val, test), gRefCOCO (val, testA, testB), ReasonSeg (val, test), and MUSE (val, test). For video: ReVOS, ReasonVOS, Ref-YouTube-VOS, Ref-DAVIS, MeViS v1, MeViS v2, and Ref-SeCVOS. The training set combines RefCOCO/+/g, gRefCOCO, ReasonSeg, and MUSE for images, plus MeViS, Ref-DAVIS, Ref-YouTube-VOS, and ReVOS for videos. Each benchmark captures a different aspect of the referring segmentation problem: RefCOCO/+/g focus on single-target standard referring, ReasonSeg on reasoning-intensive single-target queries, gRefCOCO and MUSE on multi-target referring (with MUSE being explicitly open-ended and cross-category), and the video benchmarks add temporal consistency requirements with MeViS and Ref-SeCVOS being particularly challenging for long-horizon tracking.

• Base model. The LVLM backbone is Qwen3-VL-8B-Instruct, an 8-billion-parameter vision-language model from the Qwen3 family, chosen for its strong visual grounding and instruction-following capabilities. The segmentation model is SAM 3, which provides a pretrained image encoder, detector decoder, multimodal decoder, and (for video) a memory-based tracker. The paper argues that Qwen3-VL-8B represents the capabilities of contemporary LVLMs, and SAM 3 provides state-of-the-art segmentation and tracking primitives. This combination lets the paper isolate the contribution of the concept interface — both components are strong off-the-shelf models, and any performance improvement over prior work that uses similar or weaker backbones can be attributed to the architectural reformulation rather than to using a fundamentally more powerful base model.

• Metrics. Per-benchmark evaluation follows standard protocols. For single-target benchmarks (RefCOCO/+/g, ReasonSeg), the metric is cIoU (cumulative Intersection-over-Union) — the highest-confidence prediction is taken as the final mask and its IoU with the ground truth is computed. For multi-target image benchmarks: gRefCOCO reports gIoU (generalized IoU, measuring per-image foreground mask quality after merging all predicted masks) and cIoU; MUSE additionally reports F1@0.5 (set-level detection: a predicted mask counts as a correct detection if its IoU with any ground-truth mask exceeds 0.5, and F1 is the harmonic mean of precision and recall across all predictions in the image). For video benchmarks, the universal metric is J&F — the average of the Jaccard index (region similarity, measuring mask overlap) and the F-measure (contour accuracy, measuring boundary quality). The paper reports standard test-set numbers from each benchmark's evaluation protocol.

• Baselines. The paper compares against a comprehensive set of prior work spanning different methodological families. Classical referring segmentation models: LISA (7B and Llama2-13B variants), GSVA, PixelLM (7B and Llama2-13B), EVF-SAM, PSALM, HyperSeg. LVLM-based multi-target models: SAM4MLLM, Text4Seg, MLLMSeg, DETRIS, RICE, X-SAM. Reasoning-enhanced methods: VisionReasoner, Seg-Zero, RSVP. Promptable foundation models: SAM 3 Agent (the authors reproduce this baseline by extending SAM 3 to the multi-target setting). Video-specific methods: LBDT, ReferFormer, VLT+TC, HTML, OnlineRefer, LMPM, SOC, SgMg, TrackGPT, VideoLISA, VISA, DsHmp, DMVS, VideoGLaMM, ViLLa, SAMWISE, GLUS, Sa2VA, InstructSeg, VRS-HQ, SDAM. This coverage spans the evolution from early fusion-based methods through LVLM-token-based methods to the most recent reasoning-integrated approaches, ensuring SETCON is compared against the strongest available baselines in each category.

• Generation budget / compute accounting. For image inference, the decoder produces a fixed set of candidate masks (inherited from DETR-style architecture) and predictions with confidence below 0.7 are discarded for multi-object benchmarks. Mask set cardinality is not constrained by the LVLM's autoregressive generation — the number of sub-categories N is freely determined by the LVLM, and the number of instance masks per sub-category is handled by the decoder's overproduction-and-filtering mechanism. For video, the LVLM is invoked exactly once per clip regardless of clip length, and per-frame detection plus memory-based tracking handles temporal propagation — there is no per-frame LVLM inference cost. Training was done on 8 NVIDIA H200 GPUs for one epoch with batch size 64, using LoRA adapters (rank 128, α=256) on all linear layers of the LVLM's language model, AdamW optimizer with learning rate 4 × 10⁻⁵, and cosine annealing schedule. This parameter-efficient fine-tuning keeps the total trainable parameter count modest despite the 8B-parameter backbone.

• Cross-validation / statistical protocol. For the benchmark evaluations in Tables 1, 2, and 5, the paper reports standard test-set numbers following each benchmark's official evaluation protocol. For the ablation studies on MUSE (Tables 3, 4; Figure 4), the results are reported on the MUSE test set. For the annotation quality scoring in Table 4, the paper uses Qwen3-VL-8B-Instruct as an LVLM-based judge to assess the quality of the resulting semantic labels (reported as "LLM Score"). No explicit cross-validation is described for the main benchmark results — the model is trained on the union of all training sets and evaluated on the standard test splits. The pilot study (Figure 2a) uses a Sa2VA baseline fine-tuned on the MUSE training set and evaluated on the MUSE test set, stratified by target count; the t-SNE visualization (Figures 2b and 4a) uses a deduplicated, class-balanced subset of RefCOCO for the token-based baseline and MUSE for the concept-based analysis.

Main Quantitative Results

Multi-Object Image Referring Segmentation (gRefCOCO and MUSE)

Table 1 reports results on the two primary multi-target image benchmarks. SETCON achieves the best score on every column across all splits of gRefCOCO and MUSE. On gRefCOCO val, SETCON attains 78.4 gIoU, surpassing the previous best (MLLMSeg at 75.1) by +3.3 gIoU — a substantial margin on a benchmark where recent methods had been clustering in the 71–75 range. The improvement is consistent across the test splits: +1.5 gIoU on testA (78.5 vs. 77.0 for MLLMSeg) and +3.4 gIoU on testB (73.1 vs. 69.7 for MLLMSeg). On MUSE val, the gains are substantially larger: SETCON reaches 56.9 gIoU, compared to the previous best of 44.8 gIoU from PixelLM-Llama2-13B — an improvement of +12.1 gIoU. On MUSE test, the corresponding gain is +10.9 gIoU (53.2 vs. 42.3). For F1@0.5, SETCON achieves 71.2 on MUSE val and 69.3 on MUSE test, compared to the best previous F1@0.5 of 54.5 from VisionReasoner on val and 49.2 on test — improvements of +16.7 and +20.1 F1 points, respectively.

The paper highlights that these margins widen as the number of referred targets increases (Figure 2a). On MUSE, stratified by the number of ground-truth targets per query, the Sa2VA-based baseline (representing the [SEG] token paradigm) degrades sharply — from approximately 74% F1@0.5 for single-target queries to approximately 52% for five-target queries — while SETCON remains comparatively stable, maintaining higher F1@0.5 across all target counts. The paper does not give exact stratified numbers for SETCON in Figure 2a, but the visual trend is clear: the performance gap between SETCON and the baseline grows with target count, corroborating the claim that the concept interface is disproportionately beneficial for complex, multi-target scenarios.

Single-Object Referring and Reasoning Segmentation (RefCOCO/+/g and ReasonSeg)

Table 5 demonstrates that the set-level formulation does not regress on conventional single-target settings. Across the nine splits of RefCOCO/+/g (three benchmarks × three splits each), SETCON achieves competitive or best performance on most splits: 83.7 cIoU on RefCOCO val (vs. 85.1 for X-SAM, the best), 79.4 cIoU on RefCOCO+ val (vs. 79.4 for RICE, tied for best), and 80.9 cIoU on RefCOCOg val (second to X-SAM's 83.8). On RefCOCOg test, SETCON achieves 80.0 cIoU (vs. 83.9 for X-SAM). The pattern is that SETCON performs near the top but slightly below the absolute best on RefCOCO/+/g — these benchmarks are dominated by single-instance, closed-vocabulary queries where the [SEG] token paradigm is not bottlenecked by the semantic representation issues that motivate SETCON's design. The fact that SETCON remains competitive here (within 1–3 points of the best, and achieving the best overall average of 78.9 cIoU across all nine splits) is a robustness check: the concept interface does not sacrifice single-target performance.

On ReasonSeg, which tests reasoning-intensive single-target queries ("reasoning segmentation" rather than standard referring), SETCON achieves state-of-the-art results: 70.6 cIoU on val and 70.5 on test, substantially ahead of the previous best (VisionReasoner at 66.3 val and 63.6 test). This is a +4.3 and +6.9 cIoU improvement, respectively. ReasonSeg queries are more open-ended and context-dependent than RefCOCO queries, and the explicit concept prediction appears to provide benefits even for single-target scenarios when the target identity requires compositional reasoning rather than simple category lookup.

The paper reports the best overall average across all single-target and multi-target image benchmarks, positioning SETCON as a unified framework that handles the full spectrum from single-instance to complex open-ended sets without requiring task-specific design changes.

Video Referring Segmentation (Seven Benchmarks)

Table 2 reports results on referring video object segmentation. SETCON achieves state-of-the-art performance on all seven benchmarks, with particularly large gains on the most challenging settings. On MeViS v1, SETCON reaches 62.4 J&F, surpassing the previous best (Sa2VA at 51.5) by +10.9 J&F. On MeViS v2 (a more challenging version with longer clips and more complex motion expressions), SETCON achieves 60.1 J&F vs. 47.4 for Sa2VA — a +12.7 J&F improvement. On Ref-SeCVOS (designed for semantically complex video understanding with progressive concept construction), SETCON reaches 72.2 J&F vs. 59.5 for VISA — a +12.7 J&F improvement. On the standard referring video benchmarks, gains are more modest but consistent: +9.1 J&F on ReVOS (69.5 vs. 60.4 for Sa2VA), +11.9 J&F on ReasonVOS (68.0 vs. 56.1 for Sa2VA), +5.5 J&F on Ref-YTVOS (78.8 vs. 73.3 for Sa2VA), and +4.9 J&F on Ref-DAVIS (80.8 vs. 75.9 for Sa2VA).

The gradient of improvement — largest on MeViS and Ref-SeCVOS, moderate on ReVOS and ReasonVOS, smallest but still positive on Ref-YTVOS and Ref-DAVIS — aligns with the paper's hypothesis about stable semantic anchors. MeViS and Ref-SeCVOS feature long clips with complex motion patterns and semantically demanding queries, where the benefit of clip-level, motion-aware concept generation (rather than per-frame [SEG] token prediction) is maximized. Ref-DAVIS and Ref-YTVOS, while still requiring temporal consistency, have shorter clips and simpler motion, where the [SEG] token paradigm's spatial encoding is less detrimental. This gradient is consistent with the mechanistic explanation: the concept interface's advantage scales with the temporal complexity of the task.

Pilot Study Results (Motivating the Reformulation)

Figure 2 provides the quantitative and qualitative diagnostic evidence that motivates the entire architectural reformulation. On MUSE, the Sa2VA-based baseline extended to multi-target prediction (by sequentially generating multiple [SEG] tokens) shows F1@0.5 dropping from approximately 74% for single-target queries to approximately 52% for five-target queries (Figure 2a). Exact numbers for each count are read from the bar chart and are approximate; the paper does not tabulate them. SETCON's corresponding curve is shown alongside, remaining higher and flatter, though exact stratified numbers for SETCON are not reported separately. The key comparison is the relative stability: the [SEG] token paradigm degrades sharply with count, while SETCON degrades less, suggesting that the concept interface's structural properties (set-level scope, explicit semantics) mitigate the scaling problem.

The t-SNE visualization in Figure 2b shows that for the Sa2VA baseline's [SEG] tokens, "clusters align more clearly with position than with category." This is a qualitative visual claim supported by the side-by-side t-SNE plots colored by semantic category (scattered, overlapping) and by 2D spatial position (clearer cluster structure). The quantitative basis for this claim is the t-SNE projection itself, which is a dimensionality reduction technique that preserves local neighborhood structure; the observation that spatial-position coloring reveals structure while category coloring does not implies that the high-dimensional [SEG] representations are organized by spatial proximity. Figure 4a provides the counterpart for SETCON's concept-based representations: concept embeddings "form tighter, more category-aligned clusters" compared to the token-only variant, corroborating that explicit semantic anchoring reshapes the feature space toward semantic structure.

Ablation Studies and Robustness Checks

Set-level prediction module (Table 3): Starting from a baseline that follows the prevailing implicit-token paradigm (the "✗, ✗" row — no set-level prediction, no concept conditioning), the paper isolates two contributions. Adding set-level prediction yields a substantial improvement: gIoU rises from 43.6 to 51.4 (+7.8 gIoU), and F1@0.5 rises from 59.2 to 67.8 (+8.6 points). Further adding explicit concept conditioning provides an additional gain: gIoU from 51.4 to 53.2 (+1.8 gIoU), and F1@0.5 from 67.8 to 69.3 (+1.5 points). This decomposition separates the benefit of modeling referred targets as a set (the set-level prediction) from the benefit of grounding targets in explicit semantic concepts (the concept conditioning). Both contribute positively, with set-level modeling providing the larger share of the improvement on MUSE.

Hierarchical semantic decomposition (Figure 4b): When comparing SETCON with hierarchical decomposition against a flat variant (no set-level concept, each sub-category decoded independently), the paper stratifies by the number of referred categories on MUSE. The hierarchical variant consistently outperforms the flat variant, and the gap widens as the number of categories increases. For 1-category queries, both variants perform similarly (approximately 79% vs. 74% F1@0.5 — estimated from the bar chart). For 5-category queries, the hierarchical variant maintains substantially higher F1@0.5 (approximately 67% vs. 52% for the flat variant, estimated from Figure 4b). This pattern demonstrates that the shared set-level concept becomes increasingly valuable as the target set grows more complex, providing a semantic scope that helps the model maintain coherent coverage across multiple subsets.

Annotation quality: diverse labeling (Table 4): When replacing the original rigid closed-vocabulary labels with diverse natural-language descriptions (Diverse Labeling: ✓, Hierarchical Semantic: ✗), the gIoU is essentially unchanged (50.0 vs. 50.3), but the LLM Score (a quality metric from an LVLM judge) improves from 5.99 to 6.92. This indicates that richer descriptions provide more informative semantic cues without compromising mask prediction accuracy — the expressiveness gain comes without a localization cost.

Annotation quality: hierarchical semantic structure (Table 4): Adding hierarchical semantic annotation on top of diverse labeling (both ✓) improves gIoU from 50.0 to 53.2 (+3.2 gIoU) and LLM Score from 6.92 to 6.94. The gIoU improvement indicates that organizing enriched labels into a coarse-to-fine structure helps the model better exploit the semantic supervision for mask prediction; the LLM Score improvement (small but positive) suggests the resulting concept descriptions are judged as higher quality by the LVLM judge.

Concept interface representation quality (Figure 4a): The t-SNE comparison between token-based and concept-based representations, both colored by semantic category, shows that concept-based representations "form tighter, more category-aligned clusters" than the token-only variant. This is a qualitative robustness check that the explicit concept conditioning reshapes the decoder's input space toward semantic structure, consistent with the design motivation. No quantitative clustering metric (e.g., silhouette score, Davies-Bouldin index) is reported.

Handling of no-target samples (Section 3.3, training details): The paper describes but does not ablate the mixed strategy for no-target samples (50% replacement with "no target," 50% keeping original concepts). The absence of an ablation comparing this strategy against alternatives (always replacing, always keeping, omitting no-target samples) leaves open whether this specific ratio is important or whether any reasonable handling of no-target samples would suffice. The description suggests this was a practical design choice to prevent the model from learning degenerate behaviors (always predicting "no target" or always hallucinating masks), but without ablation, its quantitative contribution is unmeasured.

Video training strategy (Section 3.3, training details): The paper uses sparse frame sampling during video training, sharing a single concept response across sampled frames. The number of sampled frames per clip is not specified, and no ablation compares this against alternatives (dense frame sampling, per-frame concept generation, different numbers of sampled frames). The video inference strategy (LVLM once per clip, concepts broadcast to all frames) is also not ablated against alternatives (per-frame LVLM invocation). The large video gains in Table 2 are attributed to the concept interface broadly, but the specific contribution of the clip-level concept sharing versus other aspects of the video pipeline (SAM 3's tracker, the detect-and-track formulation) is not isolated.

LoRA configuration (Section 4.1): The paper uses LoRA with rank 128 and α=256 on all linear layers. No sweep over LoRA ranks or comparison against full fine-tuning is reported. Given that the LVLM's language generation is central to the concept interface, the extent to which the LoRA parameterization limits concept quality (versus full fine-tuning) is unknown. The single-epoch training schedule makes full fine-tuning impractical (training instability on 8B parameters with one epoch), but a rank ablation would have illuminated whether the concept prediction quality is bottlenecked by the adapter capacity.

Confidence threshold for multi-object benchmarks (Appendix C): The paper uses a threshold of 0.7 for discarding low-confidence masks on gRefCOCO and MUSE. No threshold sweep or sensitivity analysis is reported, leaving open whether the reported gains are sensitive to this specific threshold choice.

Projection module architecture (Section 3.2): The paper states that hidden states of concept tokens are "projected into the decoder feature space as semantic conditions" but does not specify the projection module's architecture (e.g., linear layer, MLP, cross-attention). No ablation comparing projection architectures is reported, and the fusion operation between set-level and sub-category concept embeddings is also underspecified. These are architectural choices that could affect how well the LVLM's semantic space aligns with the decoder's feature space, but their contributions are not isolated.

Negative results: SAM 3 Agent baseline (Table 1): The paper reproduces SAM 3 Agent on gRefCOCO and MUSE (marked with †) and reports substantially lower performance than SETCON: 59.2 gIoU on gRefCOCO val vs. 78.4 for SETCON (−19.2 gIoU), and 27.8 gIoU on MUSE val vs. 56.9 (−29.1 gIoU). SAM 3 Agent uses SAM 3's concept-based prompting but in an interactive, per-sample manner without the hierarchical decomposition or the LVLM-driven concept generation. This large gap demonstrates that the concept interface alone (SAM 3's promptable design) is insufficient without the hierarchical organization and the LVLM's query-conditioned concept prediction — the gains come from the full framework, not just from using SAM 3 as the segmentation backbone.

Critical Assessment

Claim: SETCON achieves state-of-the-art results on multi-object and video benchmarks, with margins that grow as the number of referred targets increases.

What the experiments demonstrate: Tables 1 and 2 compellingly show that SETCON outperforms all prior methods on gRefCOCO (+3.3 gIoU val), MUSE (+12.1 gIoU val), and all seven video benchmarks (with gains up to +12.7 J&F). The gradient of improvement — largest on the most challenging benchmarks (MUSE, MeViS, Ref-SeCVOS) — is consistent with the mechanistic motivation. Figure 2a demonstrates that the Sa2VA baseline degrades with target count while SETCON remains more stable, supporting the claim about scaling behavior.

Caveats and untested conditions:

• The comparison against SAM 3 Agent (reproduced by the authors) uses an interactive prompting paradigm that is not directly comparable to SETCON's end-to-end inference. It demonstrates that SAM 3's concept interface needs the hierarchical decomposition and LVLM-driven prediction, but it does not isolate how much of the gain comes from the LVLM's reasoning capability versus the concept representation format.
• The stratified analysis in Figure 2a compares SETCON against a single baseline (Sa2VA), not against all methods in Table 1. It is possible that other multi-target methods (PixelLM-13B, VisionReasoner) also degrade with target count but less sharply than Sa2VA, and the claim about SETCON's scaling advantage relative to all prior methods is not directly tested.
• The MUSE test set is 500 questions (from the original paper's release), split into difficulty or count strata that are not specified in SETCON beyond the count-based analysis. The sample size per stratum may be small (e.g., five-target queries are likely rare), and confidence intervals on the stratified F1@0.5 numbers are not reported.

Claim: The explicit concept interface provides interpretable semantic anchors that encode category information better than [SEG] tokens.

What the experiments demonstrate: Figures 2b and 4a provide qualitative t-SNE evidence that concept-based representations cluster more cleanly by semantic category than [SEG] token representations. The ablation in Table 3 shows that adding explicit concept conditioning provides a small but positive gain (+1.8 gIoU) over set-level prediction alone. The qualitative examples in Figure 5 show that SETCON produces masks with interpretable concept labels that match the query semantics.

Caveats and untested conditions:

• The t-SNE evidence is qualitative and lacks quantitative clustering metrics (silhouette score, normalized mutual information with category labels, etc.). The claim that concept embeddings "form clearer concept-level clusters" is based on visual inspection of 2D projections, which can be misleading (t-SNE exaggerates local structure and the perplexity parameter can dramatically change cluster appearance).
• The +1.8 gIoU gain from concept conditioning in Table 3 is modest relative to the +7.8 gIoU gain from set-level prediction, and the paper does not separate whether this gain comes from the interpretability of concepts, the distributed representation across multiple tokens, the reuse of pretrained embeddings, or some combination. The specific mechanism by which concepts help is not isolated.
• The claim of "interpretability" is demonstrated through qualitative examples (Figure 5) showing concept phrases alongside masks, but there is no user study or quantitative measure of interpretability (e.g., whether human annotators can better understand or correct model errors using the concept labels versus [SEG] tokens).

Claim: Hierarchical semantic decomposition improves performance on high-cardinality queries by providing a shared set-level scope.

What the experiments demonstrate: Figure 4b shows that the hierarchical variant outperforms the flat variant on MUSE, with the gap widening from near-zero for 1-category queries to substantial for 5-category queries. This is exactly the pattern predicted by the architectural motivation: the shared scope matters more when there are more subsets to coordinate.

Caveats and untested conditions:

• The exact F1@0.5 values in Figure 4b are not tabulated — they must be estimated from a bar chart, making precise comparison difficult. The paper reports the trend but not the specific numbers per category count.
• The flat variant in this ablation still uses the concept interface (sub-category concepts decoded independently) but omits the set-level concept. This means the ablation isolates the contribution of the set-level concept specifically, not the concept interface as a whole. It tells us that hierarchy helps, but not whether hierarchy without explicit concepts (e.g., a hierarchical [SEG] token scheme) would also help.
• The number of samples per category-count stratum on MUSE is not reported. If 5-category queries are very rare (the annotation statistics in Figure 8a show a long tail, and MUSE's original distribution may differ), the stratified comparison may be based on very few samples, making the widening gap potentially noisy.

Claim: In video, stable semantic anchors from clip-level concept generation enable better long-horizon tracking than per-frame [SEG] token prediction.

What the experiments demonstrate: Table 2 shows that SETCON's largest gains are on MeViS (+10.9 J&F on v1, +12.7 on v2) and Ref-SeCVOS (+12.7 J&F), the benchmarks designed to be challenging for long-horizon temporal association. Gains on shorter-clip benchmarks (Ref-DAVIS +4.9, Ref-YTVOS +5.5) are smaller but positive.

Caveats and untested conditions:

• This pattern is consistent with the stable anchor hypothesis but does not isolate the mechanism. SETCON's video pipeline differs from prior work in multiple ways: it uses SAM 3's tracker (which Sa2VA also uses, though Sa2VA's tracker operates on [SEG] token-derived masks), it invokes the LVLM once per clip rather than per-frame (which is also done by some prior methods like Sa2VA and GLUS that process clips rather than individual frames), and it uses hierarchical concepts rather than flat [SEG] tokens. The paper attributes the gains to "stable semantic anchors," but the specific contribution of concept stability versus other design differences (e.g., the tracker's quality, the detection architecture, the training data mix) is not isolated.
• No ablation compares clip-level concept generation against per-frame concept generation using SETCON's own architecture. Such an ablation would directly test whether the stability of the concept embedding (constant across frames) matters, or whether the gains come from other aspects of the video pipeline.
• The paper does not report per-clip LVLM inference cost. For long clips (MeViS clips can span hundreds of frames), a single LVLM invocation is efficient, but the LVLM must process sufficient visual information to generate motion-aware concepts. The paper does not specify how many frames the LVLM sees when generating the clip-level concept, what the frame sampling strategy is, or whether motion information is explicitly provided or implicitly inferred.

Claim: The hierarchical semantic annotation pipeline provides the set-level supervision that enables concept-based training.

What the experiments demonstrate: Table 4 shows that hierarchical semantic annotations improve gIoU by +3.2 over diverse labeling alone (53.2 vs. 50.0) and that diverse labeling improves LLM Score over rigid labels (6.92 vs. 5.99) without hurting gIoU. The dataset statistics (236,396 samples, 784,809 concept phrases, 2.32 average sub-categories) are reported.

Caveats and untested conditions:

• The annotations are generated by a larger, stronger LVLM (Qwen3-VL-235B-A22B) that is substantially more capable than the 8B model being trained. The paper does not investigate whether the annotation quality depends on the annotator model's scale — could a smaller annotator produce similar-quality labels? This matters for reproducibility: the annotation pipeline is expensive (235B-parameter MoE model run on hundreds of thousands of samples) and may not be replicable without access to similarly scaled models.
• The quality control pipeline includes "multiple rounds of manual spot-checking" whose scale and criteria are not quantified. The paper does not report inter-annotator agreement, human evaluation of annotation quality, or the rejection rate at each filtering stage. The final annotation quality is therefore difficult to assess independently.
• The LLM Score in Table 4 is computed by the same model family (Qwen3-VL-8B-Instruct) that is used as the SETCON backbone, raising a potential circularity concern: the model is evaluating annotations that were designed to be learnable by models of its own architecture. An external evaluation (human judgment, a different model family) would strengthen the annotation quality claim.

Missing Experiments That Would Have Strengthened the Paper

• Ablation of the concept phrase length or specificity. The annotation pipeline produces concept phrases averaging 3.5 words. Would longer or more detailed phrases help or hurt? What if concepts are constrained to single-word labels? This would illuminate whether the expressiveness of free-form language is actually used by the model or whether simpler labels suffice.

• Comparison against a stronger [SEG] token baseline. The pilot study uses Sa2VA, but the paper does not attempt to improve the [SEG] token paradigm by, for instance, adding positional encoding that distinguishes different [SEG] tokens, using a per-target semantic embedding derived from the surrounding linguistic context, or adding an explicit set-level loss to the [SEG] token baseline. Such improvements might close the gap with SETCON and would help isolate whether the concept interface is genuinely necessary or whether the [SEG] token paradigm can be salvaged with better design.

• Cross-model generalization. All experiments use Qwen3-VL-8B-Instruct as the LVLM backbone. Would the concept interface provide similar gains with a different LVLM family (e.g., LLaVA, GPT-4V, Gemini)? The paper's claims about reusing pretrained semantic geometry are model-agnostic in principle, but the specific embedding space quality depends on the LVLM's pretraining.

• Failure case quantification. Figure 6 shows qualitative failure cases but the paper does not report the frequency or distribution of failure modes (target disambiguation errors, concept granularity mismatches, hallucinated concepts). A systematic error analysis — e.g., categorizing all errors on MUSE by failure type — would reveal whether the remaining errors are addressable within the concept paradigm or represent fundamental limitations.

• Inference latency and memory analysis. The paper reports generation budget in broad strokes (one LVLM invocation per clip for video, overproduction-and-filtering for masks) but does not provide wall-clock timing, GPU memory usage, or FLOPs comparisons against baselines. For practical deployment, the cost of the LVLM generating concept-bearing text (versus a few [SEG] tokens) matters, as does the overhead of the projection module and the Hungarian matching step.

Overall Assessment

The experimental results provide strong support for SETCON's core technical claim: reformulating the referring segmentation interface from implicit [SEG] tokens to explicit, hierarchical, language-grounded concepts yields substantial performance improvements on multi-target, cross-category, and long-horizon video benchmarks. The gains are large in absolute terms, consistent across thirteen benchmarks, and show the expected difficulty-dependent gradient (largest where the [SEG] token paradigm should be weakest).

However, the experiments demonstrate that the full SETCON framework works more than they demonstrate why each component matters to the degree claimed. The ablation studies are limited to a few key axes (set-level prediction vs. not, concept conditioning vs. not, hierarchical vs. flat annotation, diverse vs. rigid labels) and leave many intermediate design choices unexamined. The t-SNE evidence, while visually suggestive, is qualitative. The annotation pipeline's quality is self-assessed rather than externally validated. The video gains, while striking, are not decomposed into the contributions of concept stability versus other pipeline improvements.

The strongest evidence is the gradient of improvement across benchmarks — the fact that SETCON gains +12.1 gIoU on MUSE but only modest gains (or slightly trails) on RefCOCO is exactly what the mechanistic story predicts. If SETCON were merely a better-tuned baseline, the gains would be more uniform. The gradient is a signature that the concept interface specifically addresses the representation bottleneck that matters most for complex, multi-target, cross-category queries, which is the paper's central argument.

6. Limitations and Trade-offs

Difficulty Estimation Cost Is Unaccounted For, Potentially Exceeding the Inference Budget Being Optimized

The assumption or constraint. The paper's compute-optimal allocation framework requires estimating a prompt's difficulty before deciding how to spend the inference compute budget. The method for doing so — generating 2048 samples per question and averaging their correctness (oracle) or PRM scores (predicted) — is extremely expensive. The authors acknowledge this explicitly in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

The consequence. In a realistic deployment, the total cost would be difficulty estimation plus strategy execution. Generating 2048 samples dwarfs the largest test-time budgets studied (256–512 generations). The paper's headline efficiency claims — e.g., achieving equivalent accuracy with 4× fewer generations than best-of-N (Figures 4, 8) — are computed after difficulty is known, without amortizing the cost of learning it. If the difficulty estimation cost were included, the break-even point against best-of-N would shift substantially, potentially eliminating the reported advantage at all but the highest inference-to-pretraining ratios where a one-time estimation cost can be amortized over many queries of the same difficulty distribution.

What evidence exists in the paper. The paper reports compute-optimal scaling curves with both oracle and predicted difficulty bins (Figures 4, 8), showing that predicted bins (using the PRM's own scores, not ground-truth correctness) perform nearly as well as oracle bins. This demonstrates that the approach works without ground-truth labels, but it does not address the sampling cost. The 2048-sample requirement is stated in Section 3.2; no experiment varies the number of samples used for difficulty estimation to find a cost-quality tradeoff, and no experiment includes the difficulty estimation cost in the compute budget reported on the x-axes of Figures 3–9.

Mitigation status. The paper explicitly flags this as "a key avenue for future work" (Section 3.2), suggesting that a model could be trained to predict difficulty directly from the question text, eliminating the sampling step entirely. No such model is developed or evaluated. Until this gap is closed, the 4× efficiency figure should be understood as an upper bound on achievable efficiency — what a compute-optimal policy would achieve if difficulty were known at negligible cost — rather than a realized deployment gain.

---

The Framework Provides No Path for Problems Outside the Base Model's Capability Range

The assumption or constraint. All test-time compute strategies studied in this paper — search against a verifier and iterative revisions — operate by finding or refining correct solutions that already exist at some non-trivial rate in the base model's output distribution. The authors state this boundary condition explicitly in the Section 7 takeaway box:

"test-time compute can not compensate for fundamental capability gaps that larger pretraining would address"

The consequence. On the hardest problems (difficulty bin 5, where the base model's pass@1 is near zero), no amount of test-time compute helps. In Figure 3 (right), bin 5 accuracy hovers at 1–3% for all search methods and all budgets up to 256 generations. In Figure 7 (right), bin 5 shows roughly 2–3% accuracy regardless of the sequential-to-parallel revision ratio. In the FLOPs-matched comparison (Figure 9), the bin 5 scaling curve is essentially flat near 0–5%, and the 14× larger pretrained model substantially outperforms compute-optimal test-time compute with the smaller model on these problems (by −52.9% relative disadvantage for PRM search at R ≫ 1, per the bar chart in Figure 1). This means the framework offers zero leverage for genuinely novel reasoning that exceeds the base model's training distribution — it amplifies existing capability but cannot create it.

What evidence exists in the paper. The difficulty-bin analyses (Figures 3 right, 7 right, 9) consistently show flat or near-zero performance on bin 5 across all methods and budgets. The FLOPs-matched comparison quantitatively demonstrates that pretraining is strictly preferable to test-time compute for these problems.

Mitigation status. The paper is transparent about this limitation, explicitly stating in the Section 7 discussion that "pretraining remains the only viable path for genuinely novel or out-of-distribution reasoning." No mitigation is proposed within the framework — the limitation is fundamental to the approach of optimizing over the existing proposal distribution rather than expanding it.

---

Single Benchmark, Single Model Family — Generalization to Other Domains, Modalities, and Architectures Is Unverified

The assumption or constraint. All experiments use the MATH benchmark (500 test questions) with PaLM 2-S* as the base model. The authors state they "believe this model is representative of the capabilities of many contemporary LLMs" (Section 4), but this claim is not validated across model families, scales, or task domains beyond competition-level math.

The consequence. Several aspects of the findings could be model-specific or domain-specific in ways that practitioners relying on this paper would need to anticipate:

• PRM quality and over-optimization behavior. The PRM is trained via Monte Carlo rollouts from PaLM 2-S*'s output distribution. A model with different calibration properties — particularly one that is more or less confident in its incorrect answers — would produce different rollout-derived training labels, potentially changing the PRM's reliability and the budget at which over-optimization sets in. The difficulty thresholds at which beam search degrades (bins 1–2 in Figure 3 right) could shift substantially for a different base model.
• Revision model trainability. The ability to learn effective revision from incorrect in-context examples depends on the base model's in-context learning capabilities. The paper's edit-distance-based training data construction requires that the base model produces incorrect answers that are "close" to correct ones in edit distance — a property that may not hold for models with different error patterns (e.g., models that produce nonsensical output when wrong rather than structurally similar but flawed reasoning).
• Task-domain transfer. The MATH benchmark tests symbolic mathematical reasoning. It is unclear whether the difficulty-dependent scaling patterns generalize to code generation (where correctness can be verified by unit tests), factual QA (where errors stem from knowledge gaps rather than reasoning failures), or open-ended generation (where correctness is ambiguous). The paper's finding that search helps on medium problems but over-optimizes on easy ones may be specific to tasks where verifier training via Monte Carlo rollouts produces well-calibrated scores — a property that degrades on tasks with sparser or noisier correctness signals.

The PRM is trained on PaLM 2-S* outputs using Monte Carlo rollouts. The paper explicitly notes that the PRM800k dataset (GPT-4-labeled) was "largely ineffective" for their PaLM 2 models (Section 5.1), confirming that verifier quality is model-specific. This implies that practitioners using a different base model would need to train their own PRM from scratch using the Monte Carlo rollout procedure — a computationally expensive requirement not reflected in the paper's efficiency claims.

What evidence exists in the paper. No cross-model or cross-domain experiments are reported. The paper acknowledges the single-model limitation implicitly in Section 4 by calling PaLM 2-S* "representative," but provides no evidence for this representativeness.

Mitigation status. The paper does not address this limitation. Section 8 proposes extending the analysis to other domains and modalities as future work but does not test it. Practitioners should treat the specific difficulty thresholds, optimal budget allocations, and revision training recipe as potentially model- and task-specific until replication studies confirm otherwise.

---

Verifier Over-Optimization Is a Hard Ceiling That the Compute-Optimal Policy Mitigates But Does Not Solve

The assumption or constraint. All search-based test-time compute methods rely on the process reward model (PRM) to score intermediate solution steps and guide search. The paper demonstrates that the PRM is imperfect and gameable — aggressive search optimization finds solutions that score highly under the PRM but are actually incorrect.

The consequence. This creates a fundamental ceiling on how far test-time compute can be scaled, even on problems within the base model's capability range. The evidence is concrete:

• Beam search degrades on easy problems at high budgets (Figure 3, right, bins 1–2): the PRM assigns high scores to incorrect solutions that happen to look locally reasonable, and beam search exploits these spurious signals.
• Lookahead search — the strongest optimizer — paradoxically performs worst overall (Figure 3, left): adding lookahead steps makes the optimization more aggressive, which amplifies the PRM's errors rather than compensating for them.
• Qualitative examples show degenerate outputs (Appendix M, Figures 29 etc.): search finds solutions with repetitive low-information steps or overly short 1–2 step solutions that score highly under the PRM because they avoid making any obviously wrong intermediate claims, even though the final answer is wrong.

The compute-optimal policy routes around this problem by assigning easy questions to best-of-N (weaker optimization) and medium questions to beam search (stronger optimization where the PRM has more room to provide genuine guidance). But on medium-difficulty problems where beam search is deployed, over-optimization still limits the scaling ceiling — the beam search curves in Figure 3 flatten and sometimes decline well before the maximum budget. This means the framework is fundamentally bounded by verifier quality, and further scaling of test-time compute requires better verifiers, not better search algorithms or allocation strategies.

What evidence exists in the paper. Figure 3 (right) shows beam search performance dropping on easy problems as budget increases. Figure 3 (left) shows lookahead search underperforming simpler methods at the same budget. Appendix M provides qualitative examples of degenerate outputs. The paper explicitly identifies verifier over-optimization in Section 5.3:

"The degradation at high budgets is attributed to over-optimization of the PRM — search finds solutions that score highly under the PRM but are actually incorrect."

Mitigation status. The compute-optimal policy is a mitigation strategy (using weaker optimization where the verifier is unreliable) but not a solution to the underlying problem. The paper does not explore verifier improvements — adversarial training, ensemble methods, KL-constrained search — that might push the over-optimization threshold higher. Section 8 flags this as future work but provides no experimental results toward it. For practitioners, this means that the test-time compute scaling ceiling is determined by verifier quality, and investment in better verifiers (not more sophisticated search) is the bottleneck for further improvement.

---

The FLOPs-Matched Comparison Uses a Suboptimal Pretraining Baseline, Potentially Overstating the Advantage of Test-Time Compute

The assumption or constraint. The FLOPs-matched comparison in Section 7 compares PaLM 2-S* with compute-optimal test-time scaling against a model with approximately 14× more parameters. This larger model is trained by scaling only model parameters while holding training data fixed, following the LLaMA paradigm (Touvron et al., 2023). The authors explicitly acknowledge (Section 7) that this departs from compute-optimal pretraining (Hoffmann et al., 2022), where both data and parameters are scaled equally:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

Additionally, the 14× larger model uses only greedy decoding — no majority voting, no best-of-N, no search, no revision. It receives zero test-time compute augmentation, while the smaller model receives a carefully optimized allocation of hundreds of generations.

The consequence. Both design choices make the pretraining baseline weaker than it needs to be. A Chinchilla-optimal model trained with 14× more total FLOPs (scaling both parameters and data) would likely outperform a parameter-only-scaled model, particularly on knowledge-intensive or rare-pattern problems. Giving the larger model even a modest test-time compute budget — say, best-of-8 or a short revision chain — would create a stronger baseline. The reported advantages of test-time compute over pretraining (e.g., +27.8% relative improvement on easy questions at R ≪ 1 with revisions, per the bar chart in Figure 1) may shrink or reverse against a more rigorously optimized pretraining baseline.

What evidence exists in the paper. The experimental details for the FLOPs-matched comparison are described in Section 7. The parameter-only scaling and greedy decoding choices are explicitly stated. No ablation tests the sensitivity of the findings to these choices — e.g., by giving the larger model a test-time compute budget or by comparing against a model trained with the Chinchilla scaling recipe.

Mitigation status. The paper acknowledges the parameter-only scaling choice and frames the compute-optimal pretraining comparison as future work (Section 7). The greedy decoding choice for the larger model is not discussed as a limitation, but it is a significant asymmetry in the comparison: one model receives extensive test-time optimization and the other receives none. A fairer FLOPs-matched comparison would allocate some fraction of the total FLOPs budget to test-time compute for both models, or would use the best available inference strategy (not just greedy) for the larger model. The current comparison is best understood as an upper bound on the advantage of test-time compute over a naively deployed larger model, not over a well-optimized larger model deployment.

---

The Revision Model Suffers from a Systematic 38% Correct-to-Incorrect Reversion Rate with Only Heuristic Mitigation

The assumption or constraint. The revision model is fine-tuned exclusively on trajectories where all in-context answers are incorrect followed by a correct target. This means the model never sees examples where the current answer is already correct and should be preserved.

The consequence. At inference time, when the model generates a chain of sequential revisions, it occasionally produces a correct answer at an intermediate step and then "revises" it to an incorrect answer in a subsequent step. The paper reports (Section 6.1):

"approximately 38% of correct answers get converted back to incorrect ones"

This is a direct consequence of the training data construction: the model has learned that its job is to produce a different answer from the previous one (since all training examples involve an incorrect-then-correct transition), and it has no training signal for "recognize that the current answer is already correct and output it unchanged." The effect is that longer revision chains do not monotonically improve — correct answers can be lost as the chain progresses, and the model must rely on post-hoc selection mechanisms (majority voting or verifier-based selection across the chain) to recover them.

What evidence exists in the paper. The 38% figure is reported in Section 6.1. Figure 6 (left) shows that pass@1 at each revision step improves gradually from ~18% to ~25% over 64 steps, but the trajectory is noisy and does not increase monotonically — this is consistent with correct answers occasionally being lost. The paper's mitigation (within-chain selection) is described in Section 6.1 and used in all revision experiments.

Mitigation status. The paper mitigates this with majority voting or verifier-based selection across the entire revision chain, picking the best answer from any step rather than always taking the last revision. This is a heuristic patch, not a structural solution. A more principled fix — such as including "no change needed" training examples where the target answer equals the last in-context answer, or training the model with an explicit termination criterion — is not explored. The ReST^EM experiment (Appendix K, Figure 16) suggests that revision training is fragile: an attempt to optimize the revision model with on-policy RL-style training caused performance to degrade substantially, indicating that the 38% reversion rate may be sensitive to training methodology in ways that are not fully understood. For practitioners, this means revision chains require a post-hoc selection step (adding complexity) and cannot be relied upon to monotonically improve with length — the optimal chain length is not simply "as long as the budget allows."

7. Implications and Future Directions

How This Work Changes the Landscape

SETCON's primary impact is to reframe the interface design problem for LVLM-based dense visual grounding. The paper does not propose a better vision encoder, a stronger language model, or a more sophisticated tracker — it changes how language reasoning connects to pixel prediction. This is a fundamental architectural reframing rather than an incremental component improvement, because it alters the nature of the information that flows across the critical boundary in the system.

Before this work, the field had converged on an implicit assumption: the LVLM-to-mask interface should be a learned, non-linguistic token (the [SEG] token paradigm, from LISA through Sa2VA, GSVA, PixelLM, and their video extensions). This assumption was so deeply embedded that prior work on improving multi-target segmentation (PixelLM predicting multiple [SEG] tokens, GSVA generating [SEG] sequences) operated entirely within it — the question was how many tokens to generate and in what order, not whether tokens were the right interface at all.

SETCON's diagnostic finding in Figure 2b — that [SEG] token representations cluster by spatial position more than by semantic category — provides the first empirical evidence that this interface is structurally misaligned with the semantic demands of open-ended, multi-target, cross-category queries. This is not a tuning problem (fixable with more data or better losses) but a representational one: the interface compresses semantic information into vectors organized around the wrong organizing principle. The paper's reframing — replace implicit tokens with explicit, language-grounded concepts from the LVLM's own vocabulary — is a direct response to this diagnosis. It says: the LVLM already knows how to name what it sees; stop forcing it to compress that knowledge into a non-linguistic bottleneck, and instead let it speak directly to the decoder.

The visual evidence in Figure 7b (demonstrating the interface's benefit across multi-instance, cross-category, and open-ended scenarios) and the gradient of improvement across benchmarks in Tables 1 and 2 (largest gains on the most semantically complex benchmarks — +12.1 gIoU on MUSE, +12.7 J&F on Ref-SeCVOS — and more modest gains on single-target benchmarks) constitute the behavioral signature of this reframing. The approach scales with the semantic complexity of the query, exactly where the prior paradigm scales poorly.

This work also changes the attractiveness of research directions in grounded visual understanding:

• More attractive: Research on using LVLMs' pretrained semantic knowledge explicitly (through vocabulary-level interfaces) rather than implicitly (through context conditioning). The paper's demonstration that concept embeddings inherit useful semantic structure from pretraining — and that this structure transfers through a simple projection — opens the door to richer interfaces where the LVLM generates structured descriptions (attributes, relations, functional roles) that the decoder uses as structured conditions, not just flat category labels.
• More attractive: Research on hierarchical scope-conditioning for set prediction. The paper's hierarchical decomposition — a set-level concept fused with sub-category concepts — is a specific architectural pattern for enforcing completeness and mutual exclusivity through forward-pass conditioning rather than loss-function pressure. This pattern is general: any task requiring a model to produce a coherent set of outputs (grounded scene graphs, multi-object tracking, panoptic segmentation from language queries) could benefit from an explicit scope condition that defines the set's boundary.
• Less attractive: Incremental improvements within the [SEG] token paradigm for multi-target settings. The paper's diagnostic analysis suggests that the paradigm's representational bottleneck is fundamental — tokens encoding spatial position cannot discriminate semantic categories at scale, regardless of how many tokens are generated or how they are ordered. Research effort invested in making [SEG] token sequences longer or more cleverly arranged is unlikely to close the gap with explicit concept interfaces on complex queries, because it operates in a representational space that is structurally limited for the task.

The paper also reconciles a latent tension in the LVLM-based segmentation literature. Prior work showed that LVLMs could segment objects from language queries (LISA, Sa2VA) but struggled with multi-target settings (PixelLM, GSVA showed limited gains). The field had no clear explanation for why the same architecture that worked well for single targets degraded for multiple targets. SETCON's diagnostic provides that explanation: the interface encodes space, not semantics, and when multiple semantically distinct targets must be segmented, spatial encoding is insufficient. This reframes the narrative from "multi-target segmentation is hard" to "the implicit token interface is the wrong tool for multi-target segmentation" — a more actionable diagnosis that directly motivates the concept-based replacement.

Follow-Up Research This Work Enables

Characterizing the representational properties of concept embeddings versus [SEG] token embeddings with quantitative clustering metrics. The paper's t-SNE evidence (Figures 2b, 4a) is qualitative. A follow-up study could compute normalized mutual information (NMI) between embedding clusters and ground-truth semantic category labels for both [SEG] tokens and concept embeddings across a range of LVLM backbones (Qwen3-VL at multiple scales, LLaVA, InternVL) and task domains (MATH-style reasoning, open-vocabulary detection, fine-grained classification). The key measurement would be: how much category information is preserved in the projected embeddings, and how does this vary with backbone scale and pretraining corpus? If concept embeddings consistently show higher NMI than [SEG] tokens, this validates the paper's central representational claim quantitatively and provides guidance for selecting backbones. If some [SEG] token variants (e.g., with positional encoding distinguishing different instances) can match concept embedding NMI, this would refine the boundary between "the [SEG] paradigm is fundamentally limited" and "the [SEG] paradigm needs better token differentiation."

Stress-testing the concept interface on queries with intentionally ambiguous or overlapping target boundaries. SETCON's failure cases (Figure 6) show that open-ended queries can introduce ambiguity in target disambiguation and concept granularity. A targeted stress test could construct a benchmark of 500-1000 carefully designed ambiguous queries — where a single query could reasonably refer to multiple different target sets (e.g., "the items for a salad" in an image containing both a salad and a fruit bowl), or where the granularity of sub-category decomposition is underspecified (e.g., "the animals in the scene" — should "animals" be one concept or split into "mammals," "birds," "reptiles"?). Evaluating SETCON against a [SEG] token baseline on this benchmark would reveal whether the concept interface provides better handling of ambiguity (because concepts make assumptions explicit and auditable) or worse (because the model commits to a specific decomposition that may not match the annotator's intent). This would clarify whether the concept interface's interpretability translates to better ambiguity management or merely makes errors more visible.

Ablating the hierarchical decomposition against alternative set-level coordination mechanisms. The paper's hierarchical fusion (set-level concept embedding fused with each sub-category embedding) is one specific mechanism for enforcing set-level coherence, but there are alternatives: (a) a set-level loss that explicitly penalizes masks that overlap more than a threshold (mutual exclusivity) and ground-truth targets that are not covered (completeness), without any hierarchical architecture; (b) a sequential decoding scheme where earlier sub-category masks modulate later ones through attention masking; (c) a contrastive objective that pushes sub-category embeddings apart in the projected space. A controlled comparison on MUSE, holding all other components constant (same LVLM, same SAM 3 decoder, same training data), would isolate whether the hierarchical fusion architecture is necessary or whether the benefits of hierarchy come primarily from the training signal (the set-level concept provides a better target for the DETR-style loss) rather than from the forward-pass conditioning. If a flat concept list with a well-designed set-level loss achieves similar performance on high-cardinality queries (matching Figure 4b's trend without hierarchy), this would suggest that the annotation structure matters more than the architectural fusion — a practically important finding because flat architectures are simpler to implement and train.

Investigating whether the concept interface enables zero-shot or few-shot transfer to new visual domains without retraining the projection module. The paper trains the projection module and LoRA adapters on a specific mix of referring and reasoning segmentation datasets. An open question is whether the concept interface, by reusing the LVLM's pretrained semantic space, transfers better to out-of-distribution visual domains than the [SEG] token interface. A concrete experiment: train SETCON on the standard mix (RefCOCO/+/g, gRefCOCO, ReasonSeg, MUSE) and evaluate zero-shot on a held-out domain — e.g., medical images with referring queries about anatomical structures, or satellite imagery with queries about geographic features. Compare against a [SEG] token baseline trained on the same data. The hypothesis is that concept embeddings ("liver," "tumor," "runway," "river") carry semantic information that transfers across visual domains (a "runway" is a runway whether it appears in a satellite image or an airport photo), while [SEG] token embeddings are more tightly coupled to the visual statistics of the training domains. If SETCON shows substantially less degradation on the zero-shot domain, this would demonstrate that the concept interface provides not just better in-distribution multi-target performance but also better visual domain generalization, which is a practically valuable property for deployment in applications where the visual distribution shifts.

Scaling the hierarchical annotation pipeline with explicit human-in-the-loop validation and inter-annotator agreement metrics. The paper's annotation pipeline uses a large LVLM (Qwen3-VL-235B-A22B) with rule-based filtering and "multiple rounds of manual spot-checking," but the scale, criteria, and reliability of the human validation are not quantified. A follow-up annotation-quality study could sample 1000 annotated samples from the pipeline, have 3-5 human annotators independently judge (a) whether each sub-category phrase accurately describes the masked target, (b) whether the set-level concept correctly summarizes the scope, and (c) whether any targets are incorrectly included or excluded. Report Fleiss' kappa for inter-annotator agreement, precision/recall of the sub-category labels against human judgments, and the fraction of samples requiring correction. This would provide the first external validation of the annotation quality and establish reliability benchmarks for model-assisted annotation pipelines in dense visual grounding. If human agreement is high (>0.8 kappa), this validates the pipeline as a scalable alternative to fully manual annotation; if low, it identifies specific failure modes (e.g., attribute hallucination, granularity mismatches) that need targeted filtering or prompt redesign.

Practical Applications and Downstream Use Cases

Interactive image and video editing with open-ended natural-language target selection. In tools like Photoshop's "select subject" or video editing software's "mask tracking," users currently specify targets either by clicking on them individually or by selecting from a predefined set of semantic categories (person, sky, building). SETCON's concept interface enables a different interaction model: the user types "select all the ingredients I need to make guacamole" or "track the player who is about to receive the pass," and the system generates explicit sub-category labels (e.g., "ripe avocado," "lime," "red onion," "cilantro") and produces complete, non-overlapping mask sets across the image or video. The +12.1 gIoU improvement on MUSE and +12.7 J&F on Ref-SeCVOS directly translate to reduced manual correction effort — users would need to add or remove fewer masks compared to a [SEG] token-based system that misses targets (incomplete) or duplicates them. The interpretable concept labels also provide an audit trail: users can see why the system segmented what it did and quickly correct concept-level errors (e.g., "that's a jalapeño, not a green bell pepper — relabel it"). This application is gated primarily by inference latency (the LVLM generates a concept response before mask decoding begins), which SETCON does not benchmark, but the single-LVLM-invocation-per-clip design for video suggests that for editing workflows where the user issues one complex query per scene (rather than many simple queries per frame), the latency overhead is amortized.

Embodied AI and robotic manipulation with open-ended task specification. In warehouse automation, service robotics, or domestic assistive robots, commands like "clear all the fragile items from this shelf" or "set the table for four people" require segmenting heterogeneous target sets across multiple categories, often with implicit functional constraints (fragility, table-setting conventions). SETCON's hierarchical decomposition — set-level concept defining the scope ("all fragile items"), sub-category concepts partitioning it ("wine glasses," "ceramic plates," "glass bottles") — provides a structured representation that a robot planner can use directly: the set-level concept determines which objects to interact with, and the sub-category concepts determine how to group them for sequential manipulation (all wine glasses together, then all plates, etc.). The +12.1 gIoU improvement on MUSE (which specifically tests open-ended, cross-category referring in cluttered scenes) matters here because missing a fragile item (incomplete set) or double-counting it (duplicate mask) could cause the robot to overlook a hazard or waste time re-grasping. The explicit concept labels also enable failure recovery: if the robot's grasp fails on a "ceramic plate," the system can recognize that "ceramic plate" objects require a different grip strategy than "plastic cup" objects, without needing a pre-programmed category-to-strategy mapping. The main practical barrier is the 0.7 confidence threshold for mask filtering (Appendix C) — in safety-critical manipulation, false negatives (missed fragile items) are worse than false positives (segmented but not picked), and the threshold would need task-specific tuning.

Large-scale video annotation and dataset creation for temporal visual reasoning. Creating ground-truth annotations for referring video object segmentation (RVOS) benchmarks is labor-intensive, requiring annotators to draw masks on hundreds of frames per clip for each referred target. SETCON's video pipeline — LVLM invoked once per clip, concepts broadcast as semantic anchors, SAM 3 tracker handling propagation — could serve as a pre-annotation engine: generate mask tracks automatically, then have human annotators verify and correct rather than create from scratch. The +10.9 J&F on MeViS and +12.7 on Ref-SeCVOS suggest that the automatic masks already approach or exceed the quality of prior fully supervised methods. A concrete deployment would measure annotation time reduction: how many human-hours does SETCON save versus manual annotation or versus a [SEG] token-based pre-annotation system, while maintaining or exceeding the inter-annotator agreement of the original benchmark? The key metric is the "correction rate" — the fraction of automatically generated masks that require human adjustment — and SETCON's high F1@0.5 on MUSE (71.2 val, up from the previous best of 54.5 from VisionReasoner) suggests this rate would be substantially lower than with prior methods, directly reducing annotation cost.

When to Prefer This Method

The paper positions SETCON as a replacement for the [SEG] token paradigm specifically in settings where the query refers to multiple, semantically distinct targets spanning cross-category groups. The evidence supports a clear decision rule:

• Prefer SETCON's concept interface when: (1) the expected query distribution includes open-ended, multi-target, cross-category referring (as in MUSE, gRefCOCO, or in-the-wild interactive applications like image editing and embodied AI); (2) the LVLM backbone has strong open-vocabulary semantic knowledge that the concept interface can exploit (the paper demonstrates this with Qwen3-VL-8B, but the principle should generalize to any LVLM with rich pretrained semantic representations); (3) interpretability and auditability of the model's target selection matters for the application (the concept labels make the model's assumptions explicit); and (4) the deployment can tolerate the latency of autoregressive concept generation before mask decoding (not benchmarked in the paper, but implied by the architecture).

• Prefer the [SEG] token paradigm when: (1) the query distribution is dominated by single-target, closed-vocabulary referring (as in RefCOCO/+/g), where Table 5 shows SETCON is competitive but slightly behind the best [SEG] token-based methods (X-SAM, HyperSeg) — the concept interface's advantages do not materialize on these queries, and the simpler [SEG] token interface may be more parameter-efficient and faster; (2) inference latency is critical and the cost of generating a multi-token concept phrase cannot be amortized over multiple queries (e.g., real-time video applications running at 30+ FPS); (3) the LVLM backbone has limited open-vocabulary semantic knowledge (e.g., smaller models trained primarily on vision-language alignment rather than broad text corpora), where the concept phrases may be lower quality and the interface's advantage diminishes; or (4) the application does not require set-level coherence properties (completeness, mutual exclusivity) because queries always refer to a single pre-specified category.

The paper does not articulate a "prefer A over B" matrix for the hierarchical decomposition versus flat concept lists, but the evidence in Figure 4b supports a more granular rule: use hierarchical decomposition when the number of referred sub-categories exceeds 2, where the shared set-level scope demonstrably improves performance, and flat concept lists may suffice for simpler cases where the scope is trivially defined by the single sub-category. This is a practical implementation choice within the SETCON framework, not a choice between SETCON and alternative paradigms.