Towards Pixel-Level VLM Perception via Simple Points Prediction

ArXiv: 2601.19228

🎯 Pitch

Standard MLLMs can be trained to output object masks as mere sequences of point coordinates, without any specialized decoders, achieving segmentation accuracy that rivals complex decoder-based systems. Surprisingly, this entirely decoder-free approach unlocks a strong, latent capacity for fine-grained spatial perception directly within the model's language space.

1. Executive Summary

This paper introduces SimpleSeg, a minimalist decoder-free approach that reframes segmentation as straightforward sequential point prediction — the model outputs a sequence of textual coordinates tracing object boundaries entirely within its language space — and demonstrates that standard MLLM architectures possess a strong, inherent capacity for pixel-level perception that can be unlocked without specialized modules. The authors validate their method on the refCOCO referring expression segmentation benchmarks using Qwen2.5-VL-7B and Kimi-VL backbones, training through a two-stage SFT→RL pipeline where Reinforcement Learning with an IoU-based reward refines point sequences to match ground-truth contours (e.g., RL lifts gIoU by ~9.7 points on refCOCO over SFT alone). SimpleSeg achieves a 73.6–74.8 average cIoU on referring segmentation, comparable to or surpassing complex decoder-based systems while maintaining a purely language-space interface, establishing that precise spatial understanding can emerge from simple point prediction without architectural compromise — though the hardest shape classes (sharp corners, thin structures) remain challenging under aggressive sparsification.

2. Context and Motivation

The Core Problem: MLLMs Are Image-Level Thinkers, Not Pixel-Level Perceivers

The paper addresses a fundamental gap in how Multimodal Large Language Models (MLLMs) see. Modern MLLMs like GPT-4o, Gemini, LLaVA, and Qwen-VL have achieved remarkable fluency in open-ended vision-language tasks — they can caption images, answer visual questions, and even reason about spatial relationships at a high level. But this understanding is overwhelmingly image-level (Section 1). When asked "where is the cat?", an MLLM can tell you "on the couch" and maybe produce a bounding box. When asked "describe the shape of the cat's left ear", it lacks the vocabulary and mechanism to trace that boundary precisely. The model understands that the cat exists, but it cannot delineate its exact contours.

This distinction between semantic competence (knowing what things are) and spatial precision (knowing exactly where they extend) is the paper's central concern. The authors frame it bluntly in the introduction:

"today's MLLMs remain largely image-level in their perception, struggling to precisely localize and delineate fine structures — from object boundaries to thin parts — that are essential for genuine spatial understanding."

This is more than a benchmarking gap. It represents a ceiling on the class of tasks MLLMs can perform natively. Without pixel-level perception, an MLLM cannot participate meaningfully in:

Controllable image editing (Section 1): If you want to say "make the background behind the person brighter, but only up to the edge of their silhouette," you need a mask. A bounding box includes background pixels and clips the person's shape.
Vision-based tool use (C. Wang et al., 2025): A robot arm instructed to "grasp the handle" needs to know the handle's exact extent, not its approximate rectangular region.
GUI-grounded agents (Yuhang Liu et al., 2025; Yujia Qin et al., 2025): Clicking a precisely rendered button versus anywhere in its bounding box neighborhood is the difference between correct and incorrect action.

The paper argues that this limitation is partly historical and architectural rather than fundamental. As the authors note in Section 1:

"dense prediction tasks like segmentation have historically been overlooked as a foundational capability, as they often rely on specialized decoders or complex architectural designs not native to language-centric models."

In other words, the community didn't try to make MLLMs segment at pixel level because it assumed they couldn't — that segmentation required architectural additions. The paper's core bet is that this assumption is wrong, and that the capability is latent within standard architectures, waiting to be unlocked with the right training methodology and output format.

Why Existing Approaches Don't Solve This

Prior work on pixel-level MLLM perception falls into two camps, each with distinct tradeoffs. The paper is careful to position itself relative to both.

The Hybrid-Architecture Camp: Effective but Architecture-Confining

The dominant approach has been to augment an MLLM backbone with specialized decoders — task-specific modules that convert the MLLM's internal representations into pixel-level outputs (Section 2). Examples discussed include:

LISA (Lai et al., 2024): Appends a SAM-style mask decoder to the MLLM.
PixelLM (Z. Ren et al., 2024): Uses a similar decoder design for pixel-level reasoning.
Groundhog (Y. Zhang et al., 2024): Grounds LLM outputs to holistic segmentation through external modules.
GSVA (Xia et al., 2024), LaSagnA (Cong Wei et al., 2024), OMG-LLaVA (T. Zhang et al., 2024), GLaMM (Rasheed et al., 2024): All follow this augment-with-decoder template.

These methods work — Table 1 shows Groundhog achieving 74.2 average cIoU on referring segmentation, and Text4Seg with SAM reaching 75.4. But the paper identifies several structural drawbacks:

Architectural coupling to specific tasks: The decoder is purpose-built for segmentation. If you want the model to also do detection, keypoint localization, or instance tracking, you need additional heads, parameter sets, and training procedures. The design does not gracefully extend to new spatial tasks.
Complicated end-to-end training: Extra parameters mean extra gradients, extra hyperparameters to tune, and extra opportunities for training instability. The training pipeline becomes fragmented — the MLLM backbone learns one set of objectives, the decoder learns another, and coordinating them is non-trivial.
Outputs exit the language space: The decoder produces a dense pixel mask, not text. This means the mask cannot be easily composed with language tokens, cannot be directly read or debugged by a human, and cannot participate in the chain-of-thought reasoning that makes LLMs powerful. As the paper puts it: the design "pushes outputs out of the language space, weakening interpretability and compositional reasoning."

The hybrid camp has prioritized performance at the cost of architectural integrity. SimpleSeg's position is that this tradeoff may be unnecessary.

The Unified-Interface Camp: Language-Aligned but Low-Fidelity

The alternative approach keeps everything in the language space — representing masks as text sequences that the MLLM can generate token by token. This preserves the unified architecture and the interpretability benefits, but prior attempts have struggled with the fundamental challenge of representing two-dimensional, continuous geometry in a one-dimensional, discrete token sequence.

Text4Seg (Lan et al., 2024) is the most directly comparable prior work. It serializes masks using an RLE (run-length encoding) representation, which produces a token sequence encoding an entire mask as text. The problems with this approach are twofold:

Dense token budgets: RLE encoding of a mask can be extremely long, consuming thousands of tokens for a single object. In Table 1, the decoder-free Text4Seg variant (without SAM) achieves only 71.4 average cIoU, trailing decoder-based methods by several points, and this is partly because the dense encoding is token-inefficient and difficult for the language model to generate accurately.
Compromised interpretability: An RLE string like 0:10, 1:5, 0:20, 1:8 is not human-readable in any meaningful sense. You cannot look at it and visualize the shape.

The decoder-free Text4Seg also still relies on SAM as a mask refiner for its best results (Table 1 shows Text4Seg w/ SAM reaching 75.4 cIoU, while the decoder-free variant reaches only 71.4), meaning it doesn't fully escape the hybrid paradigm.

VisionLLM (W. Wang et al., 2024) takes a different approach — emitting polygons directly. This is conceptually closer to SimpleSeg, but the paper identifies a critical limitation: VisionLLM restricts polygons to "a small number of vertices," limiting its ability to capture fine-grained boundaries. The authors don't specify the exact vertex count, but the implication is clear — the geometric fidelity ceiling is low because the representation isn't designed to scale with shape complexity.

UFO (Tang et al., 2025) represents a hybrid of these philosophies: it uses special mask tokens that are decoded through "hacking the intermediate feature for retrieval mechanisms" (Section 2). The paper characterizes this as still requiring architectural manipulation of the MLLM's internal features, even if it avoids an explicit external decoder. Performance is strong (73.3 average cIoU in Table 1), but the architecture is not "pure" MLLM.

GiT (Haiyang Wang et al., 2024) approaches from the opposite direction — using a Vision Transformer backbone with tokenized text — but this "relies on an architecture customized for visual tasks, limiting the generalization power compared to a true vision-language model" (Section 2). It's a vision model that can read text, not a language model that can see.

The Conceptual Gap: A Mismatch Between Architecture and Optimization

Beyond the taxonomy of prior methods, the paper identifies a deeper issue with how these models are trained for segmentation. Even in the unified-interface approaches, training relies on token-level supervision — the model is trained to predict exactly the coordinate tokens that appear in the ground-truth annotation. This is the standard next-token prediction loss that works well for language but has a fundamental mismatch with geometric outputs:

A mask can be represented by many different point sequences that produce the same rendered shape. Different starting points on the boundary, different sampling densities, different vertex distributions — all can represent the identical mask. Token-level cross-entropy loss forces the model to match one specific sequence, penalizing valid alternative sequences that would produce the same geometry. This is what the paper means when it says:

"we are not aiming to force the model to rigidly regress fixed ground-truth coordinates in the training data, as contour sequences are inherently flexible."

The prior unified approaches (Text4Seg, VisionLLM) are overfitting to annotation artifacts rather than learning the underlying geometric concept. This is a subtle but important insight: the training signal is misaligned with the evaluation metric (IoU), creating a credit assignment problem where the model gets penalized for producing a valid but superficially different boundary.

How This Paper Positions Itself

SimpleSeg enters this landscape with a specific, falsifiable hypothesis: that a standard MLLM architecture, without any modification, can achieve high-fidelity pixel-level segmentation if given a sufficiently flexible and learnable output representation, and if the training signal directly optimizes for geometric quality rather than token-level matching.

The paper's positioning is not to claim absolute state-of-the-art on every metric — it doesn't beat the best decoder-based methods on all splits (Groundhog and Text4Seg+SAM still lead on some refCOCO subsets in Table 1). Instead, the claim is demonstrative: it shows that the gap between decoder-based and decoder-free approaches can be largely closed through the combination of:

Point trajectory representation instead of dense encodings or fixed-vertex polygons — trading off vertex count against geometric fidelity in a controllable way through the sparsification tolerance parameter $\epsilon$ .
Unified query interface — a 4-tuple [text, point, box, mask] formulation that multiplies supervision sources by allowing any element to query any other element (text → mask, point → mask, bbox → mask, etc.), standardizing the training format across tasks.
Reinforcement learning on rendered masks — rather than token-level cross-entropy, using sequence-level IoU rewards that evaluate the geometry directly, allowing the model to discover valid alternative trajectories.

The paper frames its contribution as much as a finding as a method: the Takeaway 1 that "standard MLLM architectures have a strong, inherent, but previously latent, capacity for precise, pixel-level perception." The implication is that the research community's investment in architectural augmentation was partly misdirected — the capability was there all along, hidden by suboptimal training signals and output formats.

The positioning relative to prior work can be summarized across three axes:

Axis	Decoder-based (LISA, Groundhog, etc.)	Prior Unified (Text4Seg, VisionLLM)	SimpleSeg
Architecture	MLLM + external decoder	Pure MLLM	Pure MLLM
Output format	Dense mask tensor	Text (RLE) or fixed-vertex polygon	Variable-length point trajectory
Training signal	Token-level + decoder loss	Token-level cross-entropy	Token-level SFT → sequence-level RL with IoU reward
Geometric fidelity	High	Low-to-medium	Comparable to decoder-based
Interpretability	Low (tensor output)	Low (RLE) or medium (polygon)	High (human-readable coordinates)

The paper's claim to be "the first to successfully apply reinforcement learning to a decoder-free MLLM for segmentation" (Section 1) is specific and verifiable: prior RL work in the MLLM space (DeepSeek-R1, Kimi k1.5) focused on reasoning tasks like math and code, not on low-level visual perception. The extension of RL to geometric optimization — using a rendering function to evaluate mask quality — is genuinely novel in this context.

The Practical Stakes: Why "Good Enough" Decoder-Free Segmentation Matters

The paper is not just making an academic point about architectural purity. There are concrete practical reasons to prefer a decoder-free approach if the fidelity threshold can be met:

Pre-training integration: As the paper notes in its key benefits, a segmentation task that stays entirely within the language space "can be seamlessly and efficiently integrated as a new, core pre-training task for foundation models, similar to visual grounding." Adding a decoder to a pre-training pipeline is architecturally invasive; adding a new text output format is not. This matters for the large-scale training infrastructure that foundation model teams have already optimized for pure transformer architectures.
Multi-task unification: A model that can output points, boxes, and masks using the same token vocabulary can naturally handle mixed-task queries without task-specific routing. If a user asks "Show me the bounding box of the person and the mask of their face," a unified model can produce both in a single response. A decoder-based model would need to either run multiple decoders or multiplex a single decoder across tasks.
Tool integration: The human-readable coordinate sequences can be directly consumed by downstream tools (image editing APIs, robot control systems, rendering engines) without deserializing a dense tensor representation. This is the "versatile framework for deploying pixel-level perception" the paper envisions.
Debugging and trust: When a model produces coordinates that can be inspected, errors are diagnosable. If the mask cuts through an object, you can see which vertices went wrong. Dense mask outputs offer no such transparency.

The paper's core wager is that these practical benefits are worth pursuing, and that the fidelity gap between decoder-free and decoder-based approaches is not fundamental — it's an artifact of representation choice and training signal design. The experiments in Section 4 are designed to test whether this wager pays off.

3. Technical Approach

3.1 Reader Orientation

This paper develops SimpleSeg, an approach that teaches a standard Multimodal Large Language Model (MLLM) to perform pixel-level segmentation by generating a sequence of text coordinates — essentially, the model "traces" object boundaries by predicting points one at a time, entirely within its regular language vocabulary. The system solves the problem of giving MLLMs native fine-grained perception without architectural modification: rather than adding specialized decoder modules or dense mask encoding layers, SimpleSeg reframes segmentation as a standard sequence generation task and uses a two-stage supervised fine-tuning → reinforcement learning pipeline to teach the model to output polygons as human-readable coordinate lists.

3.2 Big-Picture Architecture (Diagram in Words)

The SimpleSeg system has six major components organized into two conceptual layers — a data construction layer and a training layer — connected through a unified representation format:

Data Construction Layer (offline, before training):

Data Annotation Pipeline — an automated pipeline that takes raw web images and produces instance-level segmentation labels with associated text descriptions. It chains together: Grounding-DINO for object detection and phrase grounding, SAM for mask extraction, the Suzuki-Abe contour tracing algorithm for converting masks to point sequences, and an off-the-shelf VLM for optional object description refinement.
Unified Query Interface — a schema that represents any perceptual target as a 4-tuple [text, point, box, mask] and allows any element to serve as either input query or output target. This generates training examples through Cartesian products (e.g., text → mask, point → mask, bbox → mask, text → bbox), multiplying the available supervision from a single annotation.
Point Sequence Representation — the core output format: a mask is represented as [[x₁, y₁], [x₂, y₂], ..., [x_V, y_V]], a variable-length list of normalized 2D coordinates tracing the object boundary in clockwise order, with a configurable sparsification tolerance ε controlling the tradeoff between vertex count and geometric fidelity.

Training Layer (during model optimization):

Base MLLM (Qwen2.5-VL-7B or Kimi-VL) — a standard, unmodified multimodal language model that serves as the "generator." It takes an image and a text prompt and autoregressively produces a text response containing coordinate sequences. No architectural components are added, removed, or modified.
Supervised Fine-Tuning (SFT) Stage — the model is trained on instruction-response pairs spanning the unified query interface, using standard next-token prediction loss. This stage teaches the model the output format, coordinate normalization convention, and basic grounding priors.
Reinforcement Learning (RL) Stage — using GSPO (Group Sequence Policy Optimization), the model generates candidate point sequences, the sequences are rendered back into binary masks, and the resulting masks are compared to ground-truth masks through an IoU-based reward function plus a format validity check. The RL signal optimizes the entire sequence for geometric quality rather than token-level matching.

Information flow at inference time: An image and a textual prompt (e.g., "Locate the polygon of the cat") enter the base MLLM → the model generates a text response containing coordinate sequences in the learned JSON-like grammar → the coordinates are parsed and rendered into a binary mask for downstream use.

3.3 Roadmap for the Deep Dive

The following detailed breakdown proceeds in an order that mirrors how the system is built, from data representation through training:

First, the output representation (Section 3.1 of the paper): how masks are encoded as point trajectories, the coordinate normalization scheme, the text grammar that constrains decoding, and why this specific representation was chosen over alternatives like RLE strings or fixed-vertex polygons.
Second, the unified query interface: the 4-tuple formulation and how it multiplies training signals, with concrete examples of query→response pairs.
Third, the data annotation pipeline (Figure 3): the chain of off-the-shelf models that produces training data at scale, including the mask-to-contour conversion algorithm and sparsification.
Fourth, the Supervised Fine-Tuning stage: what data is used, the training configuration, and what SFT accomplishes before RL begins.
Fifth, the Reinforcement Learning stage: the GSPO algorithm, the reward function components (IoU, distance, format), the thresholding logic, and — critically — why RL is used for this task instead of staying with token-level supervision.
Sixth, the training hyperparameters and infrastructure for both stages: optimizer choices, learning rates, batch sizes, and the models validated.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a methodology paper with an empirical demonstration, whose core idea is that segmentation in MLLMs should be treated as a sequence generation problem with geometric optimization, not an architectural augmentation problem. The two key technical decisions are: (1) representing masks as variable-length point trajectories rather than dense encodings or fixed-vertex shapes, and (2) using RL with rendered-mask evaluation rather than token-level cross-entropy for the final optimization stage.

Point Trajectory Representation: Masks as Text Sequences

The foundation of SimpleSeg is a deliberate choice about how to encode a segmentation mask in the language space. A mask is conceptually a binary image — a 2D array of pixels where each pixel is either inside or outside the object. The challenge is converting this rich spatial structure into a sequence of discrete tokens that an autoregressive language model can generate token by token, while preserving geometric fidelity and keeping the sequence length manageable.

The paper adopts a boundary-based representation: instead of encoding every pixel's state (as dense encodings like RLE do), encode only the object's contour — the 1D curve that separates inside from outside. The contour is sampled at discrete points, and those points form the token sequence.

Formal representation. A mask is serialized as:

$\langle \text{mask} \rangle : [[x_1, y_1], [x_2, y_2], \dots, [x_V, y_V]]$

where each $[x_i, y_i]$ is a normalized 2D coordinate with $x_i, y_i \in [0, 1]$ , and $V$ is the variable number of vertices. Coordinates are normalized to $[0, 1]$ by dividing pixel coordinates by the image width and height respectively, making the representation resolution-independent.

What this representation computes: Given a binary mask (a 2D array of 0s and 1s), the representation extracts the object's outer boundary as a closed polygon and samples it at $V$ vertices. The sequence of vertex coordinates, written as text tokens, becomes the target that the MLLM learns to generate. At inference time, the reverse process renders the predicted coordinates back into a binary mask by filling the polygon interior.

Why this form over alternatives:

Versus dense encodings like RLE (Text4Seg): RLE produces a token budget proportional to the mask's area — a large object at high resolution consumes thousands of tokens. Point trajectories produce a token budget proportional to the boundary's complexity — typically hundreds of tokens, not thousands. This is a more efficient use of the language model's context window and reduces the decoding burden. Additionally, coordinate sequences are human-readable and directly inspectable; RLE strings like 0:10, 1:5, 0:20 are not.
Versus fixed-vertex polygons (VisionLLM): Restricting polygons to a small fixed number of vertices (as VisionLLM does) forces the model to approximate curved or complex boundaries with straight-line segments, producing low-fidelity masks for intricate shapes. Variable-length trajectories allow the vertex count to scale with shape complexity — a simple rectangle might need 4 vertices, while a tree's silhouette might need 200. The sparsification tolerance $\epsilon$ (discussed below) provides a principled knob for controlling this tradeoff.
Versus decoder-based approaches (LISA, Groundhog, etc.): Point trajectories keep the output in the language token space, meaning no architectural changes are needed and the output can be composed with other text tokens in the same response. A decoder produces a separate tensor output that cannot be interleaved with language.

Three benefits the paper claims for this representation (Section 3.1):

Interpretability: "human-readable coordinates" — a human can inspect the coordinate list and understand the shape approximately, and individual vertex errors are diagnosable.
Compositionality: "same token space as text/points/boxes" — the coordinate tokens are just numbers and brackets, drawn from the same vocabulary as the rest of the model's output. This means a single response can interleave language descriptions, bounding boxes, points, and mask coordinates without switching output modalities.
Controllable token budget: The vertex count is "linear in the number of vertices rather than image resolution," and is controlled by the sparsification parameter $\epsilon$ . This gives the practitioner direct control over the accuracy-efficiency tradeoff.

Mask-to-contour conversion (the Suzuki-Abe algorithm). The paper uses the Suzuki-Abe border following algorithm (Suzuki et al., 1985), implemented in OpenCV, to extract polygonal contours from binary masks. This algorithm traces the boundary of a binary region and produces an ordered sequence of boundary pixels. The key properties enforced by the algorithm are:

The traversal follows a consistent clockwise order, which the paper identifies as critical for learnability (Figure 7 shows that random ordering produces chaotic, unusable predictions).
The contour is closed — the last vertex implicitly connects back to the first, forming a closed polygon.

Sparsification with tolerance $\epsilon$ . Raw boundary pixel chains from Suzuki-Abe can be extremely dense — a large object might produce thousands of vertices, each separated by only 1-2 pixels. This is token-inefficient and makes the sequence unnecessarily long for the language model to generate. The paper applies a tolerance-based sparsification: the Douglas-Peucker algorithm (or equivalent) approximates the contour with fewer vertices, where $\epsilon$ controls the maximum allowed deviation between the simplified polygon and the original contour. A smaller $\epsilon$ (e.g., 0.001) preserves more detail and produces more vertices; a larger $\epsilon$ (e.g., 0.01) aggressively simplifies and produces fewer vertices.

The relationship between $\epsilon$ and performance is studied in Figure 4 and Section 4.3:

At $\epsilon$ producing ~78 tokens on average, cIoU is only 35.6 — the polygon is too coarse and underfits the shape.
At $\epsilon$ producing ~221 tokens, cIoU peaks — this represents the sweet spot where geometric fidelity and model capacity are balanced.
At $\epsilon$ producing ~859 tokens, cIoU drops to 72.5 — the sequence is too long, and the language model struggles with long-horizon decoding, producing errors (the paper calls this "long-horizon decoding errors and length exposure").

The paper's Takeaway 4 formalizes this: "A sweet spot — between the model's capacity for sequential understanding and the contour's geometric fidelity is crucial for achieving effective geometric reasoning. And Reinforcement Learning automatically finds it." This is a significant claim: RL doesn't just improve vertex placement; it also adjusts the effective vertex count toward a more efficient operating point (as shown in Figure 6, where response length evolves during RL without explicit length penalties).

Unified Query Interface: The 4-Tuple Formulation

SimpleSeg does not train the model for a single task (e.g., "given a text description, output a mask"). Instead, it defines a unified interface that standardizes all perceptual localization tasks under one schema (Section 3.1):

$\text{target} = [\text{text}, \text{point}, \text{bbox}, \text{mask}]$

where each element is a modality of spatial information:

text: a natural language description (e.g., "the red car on the left")
point: a single $[x, y]$ coordinate (e.g., a center point or a user click)
bbox: a bounding box $[x_1, y_1, x_2, y_2]$ (top-left and bottom-right corners)
mask: a polygon $[[x_1, y_1], ..., [x_V, y_V]]$ (the point trajectory)

How the interface generates training data. Any element of the 4-tuple can serve as the query (input condition) and any element can serve as the target (output to generate). The Cartesian product of [text, point, bbox, mask] as inputs and [text, point, bbox, mask] as outputs yields 16 possible task types. The paper explicitly instantiates several:

(text → bbox): "What is the bounding box of the cat?" → [0.2, 0.3, 0.7, 0.8]
(point → mask): "Give the polygon of the object at [0.5, 0.6]" → [[0.4, 0.5], ...]
(text → mask): "Locate the polygon of the dog" → [[0.1, 0.2], ...]
(bbox → mask): "Show me the contour of the object in [0.1, 0.2, 0.5, 0.6]" → [[...], ...]
(text → point): "Point out the middle giraffe in the image" → [0.483, 0.396]
(point → bbox): "Describe the bounding box of the person at [0.45, 0.6]" → [0.404, 0.423, 0.590, 0.832]

Why this multiplication of tasks matters. A single annotated instance (an image with a mask and a text description) can generate multiple training examples by deriving the other elements automatically:

The point is derived as the centroid of the mask.
The bbox is derived as the min/max of the mask coordinates in x and y.
The mask is the original annotation.
The text is the original description (optionally refined by the VLM in the annotation pipeline).

This means one annotation produces supervision for multiple task formats, effectively multiplying the training data. The paper calls this a mechanism that "multiplies supervision sources by recombining weak labels." This is important because high-quality instance segmentation annotations with text descriptions are expensive — the unified interface extracts more learning signal from each one.

The text grammar for output constraint. To reduce the entropy of decoding (the language model could theoretically produce any text, including invalid coordinate formats), SimpleSeg constrains outputs with a "minimal JSON-like grammar" (Section 3.1):

$\underbrace{[[x, y], [x, y], \dots],}_{\text{polygon}} \underbrace{[x, y]}_{\text{point}}, \underbrace{[x_1, y_1, x_2, y_2]}_{\text{bbox}}.$

The model is trained to output coordinates in this specific bracket structure. At inference time, the grammar can be enforced through constrained decoding (only tokens that produce valid bracket/number sequences are allowed) or post-processing (parse the output and reject malformed sequences). This is a standard technique in structured LLM outputs and ensures that the generated text is always mechanically parseable into coordinates.

Data Annotation Pipeline

Since SimpleSeg requires instance-level segmentation labels (images with masks and associated text descriptions) for training, and manually annotating such data at scale is prohibitively expensive, the paper constructs an automatic data annotation pipeline (Section 3.1, Figure 3). This pipeline chains together several off-the-shelf models, each performing a specific sub-task, to produce labeled training data from raw web images.

Pipeline components and their responsibilities, in processing order:

Grounding-DINO (Shilong Liu et al., 2023): This open-vocabulary object detection model takes a raw image and produces phrase grounding and object detection outputs — bounding boxes for objects in the image, optionally associated with text phrases. It identifies what objects exist and where they are roughly located.
SAM (Segment Anything Model): For each detected object region (from Grounding-DINO's bounding boxes or point prompts), SAM produces a fine-grained binary segmentation mask. SAM converts the coarse bounding box location into a pixel-precise mask. This is the critical step where the annotation pipeline moves from approximate localization (boxes) to pixel-level delineation (masks).
Suzuki-Abe Contour Extraction (OpenCV): For each binary mask from SAM, the Suzuki-Abe border following algorithm extracts the outer boundary as an ordered sequence of pixel coordinates. The algorithm enforces clockwise traversal order. An optional sparsification step (using tolerance $\epsilon$ ) reduces the number of vertices to a practical token budget.
Off-the-shelf VLM (optional): For instances where the original text description from Grounding-DINO is coarse or missing, an additional VLM can generate "refined object description tagging" — more detailed or disambiguated natural language descriptions of the segmented object. This improves the quality of the text element in the 4-tuple for training.

What the pipeline produces for one image: A set of instance annotations, each containing:

An image (the original input)
A text description (from Grounding-DINO or the VLM refinement)
A point (centroid of the mask)
A bounding box (min/max of the mask or from Grounding-DINO directly)
A mask (from SAM, converted to a point trajectory)

Each instance can then be expanded into multiple query→response pairs through the unified query interface.

Data sources for the pipeline. The paper mentions using "large-scale open-source and web data, including LAION and Coyo" for pre-training data (Appendix B.1). LAION (Large-scale Artificial Intelligence Open Network) and Coyo are web-scale image-text paired datasets commonly used for vision-language pre-training. The pipeline processes these raw image-text pairs into instance-level segmentation annotations.

Pre-training vs. SFT/RL data distinction. The paper makes an important distinction in Appendix B.1:

Pre-training data: Large-scale web data (LAION, Coyo) processed through the annotation pipeline. This is massive but noisy — the automatic labels from Grounding-DINO and SAM contain errors, but the volume provides broad perceptual priors.
SFT and RL data: The RefCOCO series (refCOCO, refCOCO+, refCOCOg) plus refCLEF, exactly following Text4Seg's data processing protocol. These are human-annotated referring expression datasets with high-quality masks and text descriptions. The SFT dataset contains 800k samples; the RL prompt set contains 400k samples.

The paper explicitly notes (Appendix B.1) that the main results in Tables 1 and 2 use models trained only on the RefCOCO data (SFT + RL stages, no pre-training on web data), to ensure fair comparison with prior work. The pre-training experiments in Table 3 are presented as an ablation study and scaling analysis, not as part of the main benchmark results. Pre-training provides a significant boost (+4.6 gIoU on refCOCO for SFT, +13.0 for SFT+RL), but the model is competitive even without it.

Stage I: Supervised Fine-Tuning (SFT)

The first training stage is standard supervised fine-tuning — the MLLM is trained on instruction-response pairs using next-token prediction loss. The purpose of this stage is to cold-start structured generation: teach the model the output format, the coordinate normalization convention, the bracket grammar, and basic grounding relationships between text, points, boxes, and masks.

What SFT teaches, specifically:

Format compliance: The model learns that when asked for a polygon, it should output [[[x₁, y₁], [x₂, y₂], ...]] with proper bracket nesting and comma separation, not free-form text or an RLE string or a bounding box.
Coordinate normalization: The model learns that coordinates should be in $[0, 1]$ , not in pixel units or some other range. This is learned implicitly from the training data, where all targets use normalized coordinates.
Basic grounding: The model learns to associate text descriptions with spatial locations (text → mask/bbox/point), to localize objects from point clicks (point → mask/bbox), and to describe the extent of objects in bounding boxes (bbox → mask, text → bbox).
Clockwise ordering: The model learns to produce vertices in consistent clockwise traversal order, which the paper shows (Figure 7) is critical for producing valid, non-self-intersecting polygons.

Training data for SFT. As noted above, the SFT dataset consists of 800k instruction-response pairs derived from the RefCOCO series (refCOCO, refCOCO+, refCOCOg train splits) plus refCLEF, following Text4Seg's data processing protocol exactly (Appendix B.1). The prompts span the Cartesian product of the unified query interface.

Training configuration (Appendix B.2, Table 4):

Optimizer: Enhanced Muon (a variant of the Muon optimizer from Liu et al., 2025, which uses matrix-sign-based gradient transformations for efficient distributed training; "enhanced" likely refers to modifications for vision-language model training).
Maximum learning rate: $5 \times 10^{-5}$
Minimum learning rate: $2 \times 10^{-6}$ (with cosine decay schedule)
Weight decay: 0.1
Adam betas: $(0.9, 0.95)$ (these are typical for Muon, which builds on Adam-style momentum)
Gradient norm clip: 1.0
Learning rate schedule: Cosine decay from max to min
Warmup ratio: 0.03 (3% of training steps used for linear warmup from zero to max learning rate)
Numerical precision: FP16 (half-precision floating point)
Global batch size: 256 (distributed across 32 GPUs, implying 8 samples per GPU)
Training samples per epoch: 800k (the full SFT dataset)
Total epochs: 1 (single pass through the SFT data)

Why only one epoch? This is a standard practice in instruction tuning of large pre-trained models — the model already has strong general capabilities from pre-training; SFT is teaching a specific output format and task structure. Multiple epochs risk overfitting to the specific annotation patterns and degrading general capabilities. The paper doesn't explicitly state this rationale, but it's consistent with the broader instruction-tuning literature (e.g., LLaVA, Alpaca).

What SFT achieves empirically (Table 3). On the RefCOCO series validation sets, SFT alone (without pre-training and without RL) reaches:

refCOCO: 65.5 gIoU
refCOCO+: 60.8 gIoU
refCOCOg: 60.4 gIoU

These numbers establish that the point trajectory representation and unified query interface, trained with standard token-level supervision, already produce a functional segmentation model. The gIoU (generalized Intersection-over-Union) metric measures the overlap between the rendered polygon from predicted coordinates and the ground-truth mask, capturing both localization accuracy and shape fidelity. A score of 60-65 means the model produces recognizable but imprecise masks — it gets the approximate location and shape right but misses fine details.

The limitation of SFT for this task. Token-level cross-entropy loss penalizes the model for predicting tokens that differ from the ground-truth annotation. But for contours, many different token sequences produce the same rendered mask. The model might predict a perfectly valid polygon that achieves high IoU with the ground truth, but if its vertices differ from the annotated vertices (different starting point, slightly different sampling along the same boundary), the token-level loss penalizes it. Conversely, the model might predict tokens that exactly match the annotation but produce a slightly offset mask due to subtle variations in how coordinates are rendered. The loss is misaligned with the true objective: mask quality.

This misalignment motivates the second training stage.

Stage II: Reinforcement Learning with GSPO and IoU-Based Rewards

The second training stage switches from token-level supervision to sequence-level reinforcement learning, where the model generates complete coordinate sequences, the sequences are rendered into masks, and the masks are compared against ground truth using geometric metrics. This directly optimizes what we care about — mask quality — while allowing the model to discover alternative valid vertex sequences.

Algorithm: GSPO (Group Sequence Policy Optimization). The paper adopts GSPO (Zheng et al., 2025) as the RL algorithm. GSPO is a variant of policy optimization designed for language model fine-tuning. Key configuration parameters (Appendix B.2, Table 5):

Clip ratio: $[3 \times 10^{-4}, 4 \times 10^{-4}]$ — this is the PPO-style clipping range that constrains how much the policy can change per update. The range notation suggests it may be adaptive or layer-dependent. This value is notably small (standard PPO often uses 0.1–0.2), indicating conservative updates to preserve the SFT-learned behaviors.
KL coefficient (KL Alpha): 0.1 — a penalty weight on the KL divergence between the current policy and the reference (SFT) policy, preventing the model from diverging too far from its SFT initialization.
Responses per group: 8 — for each prompt, the model generates 8 candidate responses, and the GSPO algorithm uses the relative rewards within this group for optimization. This is the "group" in Group Sequence Policy Optimization.
Rollout temperature: 0.8 — controls the stochasticity of generation during RL exploration. A temperature of 0.8 (below 1.0) produces moderately focused sampling, balancing exploration of alternative vertex sequences with generation of mostly valid outputs.

Why GSPO specifically? The paper doesn't elaborate on the choice of GSPO over standard PPO or other RL algorithms (GRPO, DPO). GSPO is a relatively recent method, and the choice likely reflects the authors' infrastructure or prior success with it. The group-based relative reward structure is well-suited to the task because absolute IoU values vary significantly across object sizes and complexities, making relative comparisons within a group more stable.

The reward function. The RL training signal comes from a rule-based reward system with three components (Section 3.2):

1. Mask IoU Reward:

The primary reward is the Intersection-over-Union between the predicted mask (obtained by rendering the generated point sequence into a binary mask) and the ground-truth mask. Formally, if $P$ is the set of pixels inside the predicted polygon and $G$ is the set of pixels inside the ground-truth mask:

$R_{\text{IoU}} = \begin{cases} \frac{|P \cap G|}{|P \cup G|} \times 0.1 & \text{if } \frac{|P \cap G|}{|P \cup G|} \geq \tau \\ 0 & \text{otherwise} \end{cases}$

where $\tau$ is a threshold (value not explicitly stated in the paper) below which the reward is zero, and the raw IoU (in $[0, 1]$ ) is scaled by 0.1 to produce a reward in $[0.0, 0.1]$ .

What it computes: For each generated polygon, the system renders it into a binary mask (pixels inside the polygon = 1, outside = 0), computes the pixel-wise intersection and union with the ground-truth mask, and returns the ratio multiplied by 0.1. The threshold $\tau$ acts as a floor — predictions that miss the object entirely or cover the wrong region get zero reward, preventing the model from being reinforced for random guesses that accidentally achieve tiny overlap.

Why this form: The IoU directly measures what we care about — overlap quality — and is the standard evaluation metric for segmentation (cIoU in Tables 1 and 2). The scaling by 0.1 keeps the reward magnitude comparable to other reward components and prevents the RL optimization from becoming unstable due to large reward swings. The thresholding prevents reward hacking where the model might learn to produce large, imprecise polygons that cover everything (achieving moderate IoU through high recall but terrible precision) rather than precisely tracing object boundaries.

2. MSE Distance IoU Reward:

A supplementary reward based on the centroid distance between the predicted and ground-truth masks:

$R_{\text{distance}} = -\frac{1}{N} \sum_{i=1}^{N} (\hat{c}_i - c_i)^2$

where $N = 2$ (the x and y dimensions), $\hat{c}$ is the centroid of the predicted mask, and $c$ is the centroid of the ground-truth mask. The negative sign makes it a reward (higher is better — smaller distance = less negative reward). The distance is normalized by the image size.

What it computes: The Euclidean distance between the center of the predicted polygon and the center of the ground-truth mask, squared and negated. This rewards the model for placing the polygon in the right location, even if the boundary shape is imperfect.

Why this form: Centroid distance provides a coarse localization signal that is complementary to IoU. IoU can be zero if the polygon is slightly offset from the object (no overlap at all), providing no gradient signal. The centroid distance always provides a signal about which direction to move, helping the model correct gross localization errors that IoU alone cannot fix. The paper notes (Table 5) that adding this reward provides an average gain of ~0.2 gIoU — small but consistent.

3. Format Reward:

A binary reward that checks whether the model's output is parseable as valid polygon coordinates:

$R_{\text{format}} = \begin{cases} 1 & \text{if output can be parsed as valid coordinate sequence} \\ 0 & \text{otherwise} \end{cases}$

If the format is invalid, the total reward (including IoU and distance components) is set to zero, since an invalid output cannot be rendered into a mask for evaluation.

What it computes: A simple syntactic check — does the output have properly nested brackets, valid numbers in $[0, 1]$ , and no other tokens interfering with the coordinate structure?

Why this form: Without format enforcement, the RL optimization might discover "shortcuts" where the model produces text that the reward model interprets favorably (or crashes on) without actually producing valid coordinates. The binary format gate ensures that only structurally valid outputs contribute to training. This is a standard technique in RL for structured generation.

Combined reward. The total reward for a generated sequence is:

$R_{\text{total}} = \begin{cases} R_{\text{IoU}} + R_{\text{distance}} & \text{if } R_{\text{format}} = 1 \\ 0 & \text{if } R_{\text{format}} = 0 \end{cases}$

What was tested but rejected (Table 5). The paper experimented with adding a length penalty to the reward — penalizing sequences with too many tokens to encourage conciseness. This decreased performance substantially (e.g., on refCOCO, IoU + distance + length penalty achieves 66.7 gIoU vs. 77.1 with IoU + distance alone). The model appears to naturally find an appropriate token budget without explicit pressure, as shown in Figure 6.

RL training configuration (Appendix B.2, Table 5):

Optimizer: Enhanced Muon (same as SFT)
Constant learning rate: $2 \times 10^{-6}$ (no decay during RL)
Adam betas: $(0.9, 0.95)$
Gradient norm clip: 1.0
Numerical precision: FP16
Global batch size: 256
Samples per epoch: 400k (the RL prompt set)
Total epochs: 2

Why a constant, low learning rate for RL? RL fine-tuning on top of SFT requires careful stability management. The constant low learning rate ( $2 \times 10^{-6}$ , an order of magnitude below the SFT peak of $5 \times 10^{-5}$ ) combined with the tight clip ratio prevents the policy from diverging from the SFT initialization. The two-epoch setting allows the model to see each prompt twice, refining its outputs while the KL penalty keeps it grounded.

Why RL for this task — the conceptual argument. The paper makes a deep point about why RL is not just "a way to improve performance" but is fundamentally more appropriate than token-level supervision for contour prediction (Section 3.2):

Many-to-one mapping: "Contours are inherently many-to-one w.r.t. masks; enforcing exact token matching is suboptimal." A single mask has infinitely many valid vertex sequences (different starting points, different sampling densities, equivalent but different vertex placements along smooth boundary sections). Token-level cross-entropy forces the model to match one specific sequence, which is an overconstrained problem. RL evaluates the rendered result and is agnostic to which token sequence produced it.
Sequence-level optimization: "Reinforcement learning well bridges the gap and evaluates the rendered mask, directly aligning optimization with the end metric." The end metric is mask IoU; RL optimizes for mask IoU directly, while SFT optimizes for token matching which is only a proxy.
Non-differentiable rendering: The conversion from coordinates to mask (point-in-polygon rendering) is not differentiable with respect to the coordinates in a way that permits backpropagation through the language model. RL circumvents this by treating the rendering as a black-box environment that produces a reward signal.
Improving geometric properties: RL "improves closure and thin-structure adherence that are difficult to teach via token-level losses alone." Token-level loss treats each coordinate independently; it has no mechanism to penalize non-closed polygons or self-intersections. The IoU reward naturally penalizes these because a non-closed or self-intersecting polygon produces a malformed mask with low overlap.

Why RL hasn't been used for MLLM segmentation before. The paper claims to be "the first to leverage RL in the realm of decoder-free MLLM segmentation." This is plausible because: (a) prior decoder-free methods (Text4Seg, VisionLLM) used only SFT; (b) RL in MLLMs has focused on reasoning tasks (math, code) where the reward is based on answer correctness, not geometric rendering; (c) the infrastructure to render text coordinates into masks and compute IoU rewards during RL training is non-trivial and requires careful engineering.

RL stage results (Table 3). The empirical impact is substantial:

On refCOCO (no pre-training): SFT achieves 65.5 gIoU; SFT+RL achieves 75.2 gIoU — a gain of +9.7.
On refCOCO+ (no pre-training): 60.8 → 70.6 (+9.8).
On refCOCOg (no pre-training): 60.4 → 70.9 (+10.5).
With pre-training, the gains are even larger: 70.1 → 78.5 on refCOCO (+13.0).

These gains are remarkably consistent across datasets (~10 gIoU points), suggesting that the RL stage is addressing a fundamental limitation of token-level supervision rather than overfitting to dataset-specific patterns.

The adaptive token budget phenomenon (Figure 6). During RL training, the model's average response length changes without any explicit length penalty. At high initial density ( $\epsilon = 0.001$ , producing many vertices), the length decreases moderately — the model learns to drop redundant vertices that don't contribute to IoU. At low initial density ( $\epsilon = 0.01$ , producing few vertices), the length increases slightly — the model learns to add vertices where needed to improve boundary fidelity. The paper calls this a "reasonable accuracy-efficiency balance" that RL discovers automatically, formalized in Takeaway 4.

Model Architectures and Infrastructure

SimpleSeg is validated on two open-source MLLM backbones, demonstrating architecture-agnosticism:

Qwen2.5-VL-7B (Bai et al., 2023): A 7-billion-parameter vision-language model from the Qwen family. It uses a standard ViT + LLM architecture with a vision encoder, a language model backbone, and a cross-modal connector (typically a linear projection or MLP that maps vision features into the language model's embedding space). The model supports multimodal inputs (image + text) and text outputs natively. No modifications are made — SimpleSeg uses it exactly as released.

Kimi-VL (Team, A. Du, B. Yin, et al., 2025): A Mixture-of-Experts (MoE) model with 2.8 billion activated parameters. In an MoE architecture, the total parameter count is larger than the activated count — only a subset of "expert" modules are used for any given token, making inference more compute-efficient than a dense model of equivalent total size. The exact total parameter count is not specified in the paper, but the 2.8B activated parameters make it roughly comparable to a dense 3B-scale model in inference cost. The paper's use of Kimi-VL demonstrates that SimpleSeg works on a sparsely-activated architecture, not just dense transformers.

Infrastructure. Both training stages use 32 NVIDIA GPUs (type not specified, but the FP16 precision and batch sizes suggest modern datacenter GPUs like A100s or H100s) with a global batch size of 256. The distributed training uses data parallelism (each GPU processes 8 samples per step, synchronizing gradients across all GPUs). The Enhanced Muon optimizer (Liu et al., 2025) is a recent development designed for scalable LLM training, using matrix-sign-based gradient transformations that reportedly improve training stability and convergence compared to AdamW for large-scale transformer training.

Summary of Design Choices and Their Justifications

Point trajectory over RLE/dense encodings: token-efficient (linear in boundary complexity, not area), human-readable, and composable with other text tokens. The variable-length format scales naturally with shape complexity.
Clockwise consistent ordering (enforced by Suzuki-Abe): reduces model entropy — the model doesn't need to learn an arbitrary traversal direction. Figure 7 shows that random ordering produces unusable outputs.
Unified 4-tuple query interface: multiplies supervision from each annotation, standardizes training format across tasks, and enables the model to handle diverse query types (text, point, bbox → mask) without task-specific routing.
Two-stage SFT→RL training: SFT cold-starts structured generation and teaches basic format/grounding; RL then optimizes for geometric quality directly, addressing the token-level supervision misalignment problem.
IoU-based reward with thresholding: directly optimizes the evaluation metric; thresholding prevents reward hacking through over-large polygons.
GSPO with tight clip ratio and KL penalty: conservative RL that preserves SFT-learned behaviors while refining geometry.
Constant low learning rate for RL: prevents catastrophic forgetting of the SFT-taught format and grounding skills during geometry optimization.
Sparsification tolerance $\epsilon$ : provides a controllable accuracy-efficiency knob; the paper finds an optimal operating point around 221 tokens average, and notes that RL can automatically adjust effective density toward this sweet spot.
No architectural modifications to the base MLLM: the entire system operates within the language token space, meaning SimpleSeg can be integrated into any MLLM without changing its architecture, training infrastructure, or inference pipeline — a practical advantage for adoption.

4. Key Insights and Innovations

Innovation 1: The Capability Is Latent, Not Missing — Standard MLLMs Can Perceive at Pixel Level Without Architectural Change

The paper's most fundamental contribution is not a new architecture, loss function, or training trick. It is a diagnostic finding that reframes the entire conversation about pixel-level perception in MLLMs: the dominant assumption that dense visual understanding requires specialized decoder modules is, the paper argues, wrong. What the field interpreted as a missing capability was actually a latent capability masked by suboptimal output representation and training signal.

This is a genuinely disruptive claim because it inverts the burden of proof. Before SimpleSeg, the default position — supported by the entire hybrid-architecture literature (LISA, PixelLM, Groundhog, GSVA, GLaMM, and a dozen other decoder-augmented systems) — was that if you want an MLLM to segment, you add a mask decoder. This wasn't seen as a compromise; it was seen as necessary. The language modeling head simply wasn't designed to produce dense spatial outputs, and the community's collective engineering effort went into bridging that architectural gap. SimpleSeg's core finding is that this effort was, at least partly, solving a problem that didn't exist — or rather, solving an architectural problem when the real bottleneck was representational and optimization-related.

The evidence for this claim is structural, not just quantitative. SimpleSeg uses exactly the released Qwen2.5-VL-7B and Kimi-VL models with zero architectural modifications — no added layers, no modified attention, no specialized token types, no feature-map hacking. The entire system operates by teaching the existing model to output text differently. And it achieves 73.6–74.8 average cIoU on referring segmentation (Table 1), placing it in the same performance tier as decoder-augmented systems like Groundhog (74.2), PixelLM (69.2), and GSVA (71.4). The decoder-free UFO (73.3) is competitive, but UFO still relies on "hacking the intermediate feature for retrieval mechanisms" (Section 2) — it's decoder-free in name but not in spirit. SimpleSeg is architecturally pure in a way no prior high-performing system has been.

What makes this finding intellectually significant beyond SimpleSeg as a method is its falsifiability and generality. The paper demonstrates the latent capability on two different architectures (dense 7B and MoE 2.8B-activated), suggesting it's not an artifact of a specific model design or scale. If the finding generalizes — and the authors explicitly call for others to test this — it implies that the entire research program of building specialized decoders for MLLM perception was addressing a non-fundamental limitation. The capability was there; the field just hadn't figured out how to ask the model to use it.

This is not an incremental improvement over prior decoder-based work; it's a reframing of what the problem is. The problem isn't "how do we add segmentation to MLLMs" — it's "how do we elicit the segmentation capability that MLLMs already possess." That shift in framing changes what future research looks like: instead of designing better decoders, the focus moves to representation design, training signal alignment, and optimization for geometric objectives — all within the language space.

Innovation 2: RL on Rendered Masks as a Solution to the Token-Level Supervision Misalignment Problem

The second major contribution is the identification and resolution of a fundamental misalignment in how segmentation models are trained. Prior decoder-free approaches (Text4Seg, VisionLLM) trained with standard next-token prediction loss — the same cross-entropy objective used for language modeling. The paper diagnoses why this is inappropriate for geometric outputs and introduces a solution that, while conceptually straightforward, had never been applied to this domain.

The misalignment is this: a segmentation mask has infinitely many valid textual representations. The same rendered shape can be described by contour sequences with different starting points, different vertex densities, and different vertex placements along smooth boundary sections. Token-level cross-entropy treats these alternative sequences as errors — the model gets penalized for producing [[0.1, 0.2], [0.15, 0.25], ...] when the annotation says [[0.12, 0.22], [0.14, 0.24], ...], even if both sequences trace the same boundary. The training signal is enforcing annotation-specific vertex choices rather than geometric fidelity. The paper calls this out explicitly:

"we are not aiming to force the model to rigidly regress fixed ground-truth coordinates in the training data, as contour sequences are inherently flexible."

This misalignment manifests concretely in the SFT-only results (Table 3): 60–65 gIoU means the model produces recognizable but imprecise masks. The token-level loss has taught format and rough localization, but it can't teach geometric precision because precision isn't what the loss measures.

The solution — reinforcement learning with an IoU-based reward on rendered masks — is elegant in its directness. Instead of asking "did you predict the right tokens?", it asks "did you produce a good mask?" The rendering step (converting coordinates to a binary mask) acts as a semantic bridge: it evaluates the consequence of the token sequence rather than the sequence itself, aligning the training signal with the evaluation metric. This is not a technique the paper borrows from prior MLLM segmentation work because no prior MLLM segmentation work used RL at all, as the paper claims:

"To our knowledge, this is the first work to successfully apply reinforcement learning to a decoder-free MLLM for segmentation."

The significance of this innovation extends beyond the 9.7–10.5 gIoU point gains in Table 3 (though those gains are substantial and remarkably consistent across datasets). It opens a new optimization paradigm for spatial outputs from language models. If RL on rendered outputs works for segmentation, it could work for keypoint localization (render keypoints and compute PCK), for lane detection (render lanes and compute IoU), for any task where the output is a geometric structure that can be rendered and evaluated. The paper demonstrates the concept on contours; the principle generalizes.

The observation that RL automatically adjusts the effective vertex count toward an efficient operating point (Figure 6, where response length changes during RL without any length penalty) is a bonus finding that reinforces the core insight: when you optimize for the right thing (mask quality), the model discovers efficient representations on its own. This is the paper's Takeaway 4 and it's a concrete instance of a broader principle — that sequence-level geometric rewards induce better representations than token-level matching — that hasn't been documented in the MLLM literature before.

Innovation 3: Difficulty-Aware Sparsification as a Mechanism Design Problem, Not a Hyperparameter

The paper's analysis of the sparsification tolerance $\epsilon$ (Section 4.3, Figure 4) reveals something deeper than a hyperparameter sweep. It shows that the relationship between vertex count and segmentation quality is unimodal with a clearly identifiable peak, and that this peak is determined by the interaction between the model's sequential generation capacity and the geometric fidelity required.

This is not the typical "tune the hyperparameter and pick the best value" finding. The paper frames it as an instance of a mechanism design problem: you must choose a representation that the model can actually generate (given its finite context window, its autoregressive error accumulation, and its training distribution over sequence lengths), not just one that is geometrically sufficient. The Douglas-Peucker algorithm can give you arbitrarily high geometric fidelity by reducing $\epsilon$ , but at some point the token sequences become too long for the MLLM to generate reliably — the paper calls this "long-horizon decoding errors and length exposure." At the other extreme, aggressive sparsification produces sequences the model can handle easily but that underfit the shape.

The finding that the optimal operating point is around 221 tokens (vs. 78 at the low end producing 35.6 cIoU and 859 at the high end producing 72.5) is less interesting than the diagnostic framework it implies. Before this work, the community's approach to representing spatial outputs as text was largely ad hoc — VisionLLM just picked a fixed small vertex count, Text4Seg used dense RLE without analyzing the token budget implications. SimpleSeg provides a principled way to think about this: the representation capacity must be matched to the model's generation capacity, and the match is empirically discoverable through the unimodal performance curve.

This connects to a broader insight about emergent precision in language models. The model has finite sequential reasoning capacity; you can ask it to produce 1000 tokens describing a shape, but error accumulation in autoregressive generation means that later tokens are less reliable than earlier ones. The sparsification tolerance is, in effect, a knob that trades off per-vertex precision against sequence-level reliability. Finding the sweet spot is not about the geometry — it's about the model's cognitive architecture. This is a genuinely cross-disciplinary insight at the intersection of computer vision (contour approximation theory) and language modeling (autoregressive decoding reliability), and it hasn't been articulated in prior work.

The reinforcement learning dynamic adds another layer: RL can shift the effective operating point. Figure 6 shows that when initialized with high density ( $\epsilon = 0.001$ ), RL decreases the average response length — the model learns to drop redundant vertices. When initialized with low density, RL increases it — the model learns where additional vertices improve the IoU reward. This means the sparsification tolerance isn't just a static data preprocessing choice; it's an initial condition for an optimization process that adapts the representation to the model's capabilities. That framing — representation as an equilibrium of a learning dynamic rather than a fixed encoding — is a conceptual contribution that distinguishes SimpleSeg's analysis from a simple hyperparameter ablation.

5. Experimental Analysis

Evaluation Methodology

Dataset. The primary evaluation uses the refCOCO series: refCOCO, refCOCO+ (Kazemzadeh et al., 2014), and refCOCOg (Mao et al., 2016), plus refCLEF for training. For referring expression segmentation (RES), the standard validation and test splits are used: refCOCO val/testA/testB, refCOCO+ val/testA/testB, and refCOCOg val/test. Training follows Text4Seg's protocol exactly, using the train splits of refCOCO, refCOCO+, refCOCOg, and refCLEF, yielding 800k SFT samples and 400k RL prompts (Appendix B.1). The paper also uses large-scale web data (LAION, Coyo) for pre-training experiments, though main benchmark results in Tables 1 and 2 use only RefCOCO training data to ensure fair comparison with prior work.
Base model(s). Two open-source MLLM architectures are validated: Qwen2.5-VL-7B (Bai et al., 2023), a 7-billion-parameter dense vision-language model, and Kimi-VL (Team, A. Du, B. Yin, et al., 2025), a Mixture-of-Experts model with 2.8 billion activated parameters. The paper states these were chosen to demonstrate architecture-agnosticism — that the latent pixel-level perception capability exists across both dense and sparsely-activated designs. No architectural modifications are made to either model; they are used exactly as released.
Metrics. Two primary metrics are used. For referring expression segmentation (RES), the metric is cumulative IoU (cIoU), the standard metric in this literature — the Intersection-over-Union between the predicted mask (rendered from the generated point sequence) and the ground-truth mask, averaged across all test instances. For referring expression comprehension (REC), the metric is Acc@0.5 — the fraction of instances where the IoU between the predicted bounding box (derived from the predicted mask via min/max operations on vertex coordinates) and the ground-truth bounding box exceeds 0.5. For the training stage ablation (Table 3), the paper uses gIoU (generalized IoU) on validation splits, which is a variant of IoU that accounts for non-overlapping predictions by incorporating the smallest enclosing box.
Baselines. The paper compares against two categories. Decoder-based models: LISA (X. Lai et al., 2024), PixelLM (Z. Ren et al., 2024), GSVA (Xia et al., 2024), NEXT-Chat (A. Zhang et al., 2023), AnyRef (He et al., 2024), LaSagnA (Cong Wei et al., 2024), Groundhog (Y. Zhang et al., 2024), and Text4Seg with SAM refiner (Lan et al., 2024). Decoder-free models: UFO (Tang et al., 2025) using LLaVA-1.5-7B backbone, and Text4Seg (Lan et al., 2024) using InternVL2-8B backbone without the SAM refiner. All baseline numbers are taken directly from the cited papers; no re-implementation or re-evaluation was performed.
Generation budget / compute accounting. The paper does not use a formal "generation budget" framework as seen in test-time compute scaling work. All models generate a single response per query (one point sequence). Compute is implicitly measured by the number of output tokens, which is controlled by the sparsification tolerance ε — the vertex count varies with shape complexity but averages around 221 tokens at the optimal operating point (Figure 4). No comparison across different generation budgets (e.g., best-of-N sampling) is performed for the segmentation task. The RL stage uses 8 response candidates per prompt for group-based optimization (GSPO), but at inference time only one response is generated.
Cross-validation / statistical protocol. The paper does not report cross-validation procedures for the main benchmark results (Tables 1 and 2) — these are standard test-set evaluations on established benchmarks with fixed splits. For the training stage ablation (Table 3), validation set results are reported for the refCOCO, refCOCO+, and refCOCOg val splits. No confidence intervals, error bars, or statistical significance tests are reported for any result, which is consistent with the conventions of the referring segmentation literature but represents a limitation for assessing result reliability given the test set sizes (refCOCO val: 1,500 instances; testA: 1,975; testB: 1,810; refCOCO+ val: 1,500; testA: 1,975; testB: 1,798; refCOCOg val: 1,300; test: 1,300 — exact counts from Kazemzadeh et al., 2014 and Mao et al., 2016).

Main Quantitative Results

Referring Expression Segmentation (Table 1)

The headline result is that SimpleSeg achieves 73.6 average cIoU with the Kimi-VL backbone and 71.9 with Qwen2.5-VL-7B (both without pre-training on web data), placing it competitively among both decoder-based and decoder-free methods. With pre-training on large-scale web data (LAION, Coyo), SimpleSeg reaches 74.8 (Kimi-VL) and 74.6 (Qwen2.5-VL-7B) average cIoU.

Comparison with decoder-based methods (Table 1, "Decoder-based Models" section): The strongest decoder-based competitors are Groundhog at 74.2 average cIoU and Text4Seg with SAM at 75.4. SimpleSeg with Kimi-VL (73.6 without pre-training, 74.8 with) falls between these two, outperforming Groundhog and trailing Text4Seg+SAM by approximately 0.6–1.8 points depending on the variant. On specific splits, SimpleSeg-Kimi-VL* (with pre-training) achieves 80.0 on refCOCO val and 80.6 on testA — these are competitive with Groundhog's 78.5/79.9 and Text4Seg+SAM's 79.2/81.7. On the harder refCOCO+ testB split, SimpleSeg-Kimi-VL* reaches 67.1, exceeding Text4Seg+SAM's 66.5.

A notable pattern: SimpleSeg performs relatively better on refCOCOg (val 72.8, test 74.7 with Kimi-VL*) compared to refCOCO/refCOCO+, where the gap to decoder-based methods is larger. This may reflect the nature of refCOCOg — its expressions are longer and more descriptive (averaging 8.4 words vs. 3.6 for refCOCO), requiring more language understanding relative to fine-grained localization, which plays to the MLLM's strengths.

Comparison with decoder-free methods (Table 1, "Decoder-free Models" section): SimpleSeg-Kimi-VL (73.6) slightly outperforms UFO (73.3) and substantially exceeds Text4Seg without SAM (71.4). The margin over Text4Seg (2.2 points) is meaningful given that both are pure decoder-free approaches using only text-space outputs; the difference is attributable to the point trajectory representation versus RLE encoding and the RL training stage. With pre-training, the gap widens further: SimpleSeg-Kimi-VL* at 74.8 vs. Text4Seg at 71.4 — a 3.4-point advantage.

The Qwen2.5-VL-7B variant (71.9 without pre-training) underperforms UFO (73.3) and SimpleSeg-Kimi-VL, suggesting that the MoE architecture's efficiency (2.8B activated parameters performing at 7B-dense-model levels) provides an advantage for this task, possibly because the sparsely-activated experts can specialize differently for geometry versus language aspects of the generation.

The * (pre-training) variants show inconsistent gains: Pre-training improves refCOCO val from 76.9 to 80.0 for Kimi-VL (+3.1 cIoU) but decreases testA from 78.9 to 80.6 (+1.7, but note this is an increase — the anomaly is on Qwen) and oddly decreases Qwen2.5-VL-7B's testA from 78.7 to 77.8 (-0.9). This irregular pattern — where pre-training helps most splits but marginally hurts refCOCO testA for Qwen — suggests the pre-training data distribution (web images from LAION/Coyo) may not perfectly align with the refCOCO test distribution for that specific split, or that the result reflects variance rather than a systematic effect.

Referring Expression Comprehension (Table 2)

SimpleSeg achieves 87.2 average Acc@0.5 with Kimi-VL and 85.2 with Qwen2.5-VL-7B. This places it near the top of the field: UFO achieves 87.5, Text4Seg+SAM achieves 87.1, and the best decoder-based methods (PixelLM at 86.0, Text4Seg+SAM at 87.1) are in a similar range.

The REC results validate an important claim: that SimpleSeg's mask outputs are geometrically accurate enough that simple min/max bounding box extraction produces competitive detection performance. This means the system is genuinely multi-task — a single model producing mask coordinates can derive bounding boxes (and points, via centroids) without separate training or task-specific heads.

The pattern across splits: SimpleSeg-Kimi-VL performs strongest on refCOCOg val (86.1) and test (86.5), consistent with the RES finding that refCOCOg's descriptive expressions favor the MLLM's language understanding. It is slightly weaker on refCOCO+ testB (80.2), where the expressions are shorter and objects are more visually similar, making precise spatial discrimination harder.

Training Stage Ablation (Table 3)

The most striking quantitative result in the paper is the RL contribution. Table 3 reports gIoU on validation splits under different training configurations, using Qwen2.5-VL-7B as the backbone:

SFT only (no pre-training): 65.5 (refCOCO), 60.8 (refCOCO+), 60.4 (refCOCOg) — a functional baseline establishing that the point trajectory format is learnable from token-level supervision alone.
SFT + RL (no pre-training): 75.2 (+9.7), 70.6 (+9.8), 70.9 (+10.5) — RL provides a consistent ~10 gIoU point improvement across all three datasets.
Pre-training only (no SFT, no RL): 25.3 (−45.7 from SFT-only), 18.7 (−46.4), 25.7 (−43.0) — pre-training without task-specific fine-tuning performs poorly due to distribution shift between pre-training prompts and RefCOCO-style questions (the pre-training uses different query formats from the evaluation protocol).
Pre-training + SFT: 70.1 (+4.6 over SFT-only), 65.0 (+4.2), 65.7 (+5.3) — pre-training provides a consistent ~4–5 gIoU point boost over SFT alone.
Pre-training + SFT + RL: 78.5 (+13.0 over SFT-only), 69.8 (+9.0), 71.7 (+11.3) — the full pipeline achieves the best results, with the pre-training gain compounding on top of the RL gain (though the refCOCO+ gain of +9.0 is slightly lower than without pre-training at +9.8, which may reflect variance or a ceiling effect).

The consistency of the RL gain (~10 points across all three datasets) is remarkable. It suggests the improvement is not dataset-specific but addresses a fundamental limitation of token-level supervision for geometric outputs — the misalignment between matching annotation coordinates and producing good masks.

Why pre-training + RL on refCOCO+ shows a smaller gain (69.8, +9.0 from SFT-only baseline of 65.0, but the no-pre-training RL gain was +9.8): The paper doesn't discuss this, but one interpretation is that pre-training already teaches some of what RL would otherwise need to optimize (better boundary adherence, closure), so the marginal gain from RL is slightly reduced. Alternatively, the 400k RL prompt set sampled from RefCOCO may interact differently with a pre-trained initialization, or it may simply be variance given the lack of confidence intervals.

Ablation Studies and Robustness Checks

Point density / Sparsification tolerance (ε): Figure 4 shows the relationship between average token length (controlled by ε) and RES performance (cIoU). At 78 tokens (coarse sparsification), performance is 35.6 cIoU — the polygon critically underfits shapes. At 221 tokens (moderate sparsification), performance peaks at 72.5 cIoU — the sweet spot balancing geometric fidelity and model generation capacity. At 859 tokens (minimal sparsification), performance degrades to 72.5 cIoU — the long sequence induces decoding errors. This is not a trivial "more points = better until diminishing returns" curve; it shows a genuine degradation at high vertex counts, confirming the paper's "long-horizon decoding errors and length exposure" hypothesis. The experiment was conducted in SFT-only mode (no RL), so it isolates the effect of representation density on token-level learning.

Reward design components (Table 5): Using the IoU reward alone achieves 76.9/71.9/72.9 gIoU on refCOCO/refCOCO+/refCOCOg validation sets. Adding the centroid distance reward yields 77.1/72.2/73.1 — a small but consistent gain of ~0.2 gIoU points across all splits. Adding a length penalty to encourage conciseness decreases performance to 66.7/62.4/62.0 — a substantial degradation of ~10 points. This is a noteworthy negative result: the length penalty interferes with the model's ability to allocate vertices where needed. The paper's interpretation — that RL automatically finds an appropriate token budget without explicit pressure (Figure 6) — is supported by the fact that adding explicit pressure hurts.

Clockwise vs. alternative point ordering (Figure 7): When the Suzuki-Abe algorithm's clockwise ordering is not enforced, the coordinate sequence "fails to form a valid polygon, and no segmentation mask can be derived." This is a binary result — the model cannot learn unorganized point clouds. Alternative orderings (the specific alternatives tested are not described) "confuse the model and yield chaotic or repeated points, decreasing the token efficiency." This ablation validates the importance of the consistent ordering constraint for learnability, though the qualitative presentation (visual examples in Figure 7, not quantitative metrics) limits precise assessment of the effect size.

Metrics trends during RL (Figure 6): During RL training, the model's response length (number of output tokens) changes dynamically without any length-related reward component. At high initial density (ε = 0.001, producing many vertices), length decreases moderately — the model drops redundant vertices. At low initial density (ε = 0.01, producing few vertices), length increases slightly — the model adds vertices at critical boundary regions. The reward increases and validation gIoU increases in both cases, demonstrating that RL adaptively discovers efficient vertex allocation. This is not a typical ablation (it studies dynamics rather than a controlled comparison), but it serves as evidence for the claim that RL automatically finds the sweet spot identified in Figure 4.

Pre-training contribution with and without SFT (Table 3): Pre-training without SFT performs catastrophically (25.3 gIoU vs. 65.5 for SFT-only), confirming that the pre-training data distribution (web images with automatic annotations) does not align with the RefCOCO evaluation format and cannot serve as a standalone training stage. Pre-training + SFT outperforms SFT-only by 4–5 gIoU points, confirming that the web-scale data provides useful perceptual priors even when task-specific fine-tuning is still required. Pre-training + SFT + RL outperforms SFT + RL by 3.3 (refCOCO), -0.8 (refCOCO+), and 0.8 (refCOCOg) gIoU points — the benefit is present but less consistent than the SFT → RL gain, and the negative value on refCOCO+ is unexplained.

Extended tasks (Appendix C, Figures 8–10): The paper provides qualitative results for tasks beyond text→mask, including point→mask, bbox→mask, text→point, text→bbox, and panoptic segmentation (multiple objects in one image). These are visual demonstrations without quantitative metrics, but they show the unified query interface functioning as designed. The model successfully segments objects from point clicks, bounding boxes, and text descriptions, and handles multiple objects in a scene. The paper also shows a failure case on objects with holes (Figure 12, a donut) — the model segments the outer boundary but cannot represent the hole, which requires either multiple contours or a different polygon representation (the Suzuki-Abe algorithm extracts only the outer boundary; hole representation with inner contours is not addressed).

Critical Assessment

Claim: "Standard MLLM architectures possess a strong, inherent capacity for fine-grained perception." The experiments support this claim for the tested architectures and task. SimpleSeg achieves competitive performance with decoder-based systems using unmodified Qwen2.5-VL-7B and Kimi-VL — two different architectures (dense and MoE) at different scales (7B total vs. 2.8B activated). This is genuine evidence for latent capability. However, the claim is demonstrated on exactly two model families from a single era of MLLM development. Whether this finding generalizes to LLaVA-style architectures (which use a different vision encoder and connector design), to much smaller models (1–3B parameters), or to much larger models (70B+) is untested. The claim could be qualified as: "For mid-scale MLLMs (~3–7B parameters) with modern vision encoders, pixel-level perception capability exists latently and can be unlocked without architectural modification." The gap between this qualified statement and the paper's stronger framing is a limitation of the experimental scope.

Claim: "Reinforcement Learning with IoU-based reward refines point sequences to accurately match ground-truth contours." The evidence in Table 3 is strong — RL provides a consistent ~10 gIoU point improvement across three datasets, and Figure 6 shows the optimization is dynamic and adaptive. However, the ablation of RL components is incomplete. Table 5 tests reward function variants (IoU, IoU+distance, IoU+distance+length) but does not ablate the RL algorithm itself. There is no comparison to alternative RL algorithms (PPO, GRPO, DPO), no comparison to alternative optimization approaches (e.g., iterated SFT on self-generated data, or direct preference optimization on rendered mask pairs), and no comparison to simply using a better SFT loss (e.g., a loss that incorporates IoU during SFT through differentiable rendering approximations). The paper's strong claim that RL is "a more reasonable and efficient optimization method for perception tasks" requires evidence that RL outperforms these alternatives, but that evidence is not provided. The demonstrated gain is over SFT only — a baseline that the paper itself argues is fundamentally misaligned for this task.

Claim: "Performance comparable to, and often surpasses, methods relying on complex, task-specific designs." The results in Table 1 support this with qualifications. SimpleSeg-Kimi-VL* (74.8) outperforms most decoder-based methods (Groundhog at 74.2, GSVA at 71.4, PixelLM at 69.2, LISA at 69.9) but is outperformed by Text4Seg+SAM (75.4). SimpleSeg-Kimi-VL without pre-training (73.6) is behind Groundhog, Text4Seg+SAM, and roughly ties UFO. "Often surpasses" is accurate if interpreted as "surpasses most decoder-based methods most of the time," but "comparable to" is the more precise characterization — the method is in the same performance tier, not clearly dominant. The claim's rhetorical force ("strikingly simple yet highly effective," "challenging the prevailing need for auxiliary components") implies that architectural augmentation was unnecessary, but the 1–2 point gap to the best decoder-based system (Text4Seg+SAM) suggests there is still a small but persistent advantage to having a specialized mask decoder. Whether this gap matters in practice depends on the application's fidelity requirements, but the paper's claims would be strengthened by a failure analysis showing where those 1–2 points of cIoU are lost (sharp corners? thin structures? occlusion boundaries?).

Missing experiment: combined SFT + RL with best-of-N or beam search at inference. The paper uses single-response generation at inference time. Given the RL stage's group-based optimization (8 responses per prompt), there is a natural extension: at test time, generate multiple candidate point sequences, render each to a mask, and select the one with the highest self-consistency or PRM-like score. This would increase inference compute but could close the remaining gap to decoder-based methods. The paper does not explore this, leaving open the question of whether decoder-free methods can match the best decoder-based methods at equal inference cost or only at equal training cost.

Missing experiment: direct comparison to VisionLLM's polygon approach. The paper positions itself relative to VisionLLM (Section 2) as using a superior representation (variable-length vs. fixed-vertex polygons), but there is no head-to-head comparison. VisionLLM does not report refCOCO results in the format used by Table 1, so direct comparison is impossible from the paper alone. Including a VisionLLM-style baseline (fixed small vertex count, no RL) within the SimpleSeg framework would strengthen the claim that the variable-length representation and RL optimization are the critical innovations.

Missing experiment: resolution sensitivity. The paper uses normalized [0, 1] coordinates, which in principle makes the representation resolution-independent. However, the underlying images in refCOCO have varying resolutions, and the MLLM's vision encoder operates at a fixed input resolution (typically 336×336 or 448×448 for Qwen2.5-VL, with possible tiling for high-res images). There is no analysis of how performance varies with image resolution or object size — small objects that occupy few pixels in the vision encoder's feature map may be fundamentally harder to segment via point prediction than large objects, but this is untested. This is particularly relevant because the paper's qualitative examples (Figures 2, 8–11) show mostly prominent, well-resolved objects.

On the pre-training benefit: The pre-training results (Table 3) show a meaningful but not transformative gain (+4–5 gIoU for SFT, +3.3 to -0.8 for SFT+RL). The paper frames pre-training as a scaling analysis rather than a key contribution, which is appropriate given the inconsistency of the benefit (the negative value on refCOCO+ for the full pipeline). However, this also means that the paper's "simplicity" claim — that the method can be "seamlessly and efficiently integrated as a new, core pre-training task for foundation models" — is not empirically validated. The pre-training experiment shows it can be done, but not that the pre-training integration is particularly efficient or that it provides benefits beyond what SFT on task-specific data already achieves.

On the single-epoch SFT design: SFT uses exactly one epoch through 800k samples. This is standard for instruction tuning, but it means the model sees each RefCOCO training example exactly once. Whether additional epochs would improve SFT performance (potentially reducing the reliance on RL) is not tested. The paper's implicit claim is that SFT saturates after one epoch and RL is necessary for further improvement, but the evidence for saturation is not provided.

On the format reward interaction: The format reward in RL is binary — valid outputs get the full geometric reward, invalid outputs get zero. This means the RL optimization receives no gradient signal about how to fix invalid outputs; it only learns that invalid formats are bad. There is no analysis of the rate of format errors during RL training (do they decrease? increase? how does the model learn to avoid them?). If format errors are rare after SFT (as one would expect from a well-trained SFT model), the format reward is doing very little. The paper does not report format error rates.

On the centroid distance reward contribution: The +0.2 gIoU gain from adding the distance reward (Table 5) is very small and may fall within the variance of the experiment (no confidence intervals are reported). The paper's interpretation that it helps correct gross localization errors makes theoretical sense, but the empirical support is weak. A stronger test would be to evaluate the distance reward's impact specifically on instances where the SFT-only model produces large centroid errors — the hypothesis predicts a larger gain on those instances, which is not tested.

On generalization beyond RefCOCO: All quantitative results are on the RefCOCO series. The paper's title claims "Towards Pixel-Level VLM Perception," and the qualitative results (Figures 2, 8–11) show generalization to anime, charts, infographics, and synthetic shapes, but there are no quantitative metrics for these domains. The failure cases (donut holes in Figure 12, texture confusion in zebras in Figure 12) are shown qualitatively but not systematically categorized or quantified. The claim of "strong generalization" is supported anecdotally but not statistically.

Summary of experimental strengths: (1) Consistent RL gains across three datasets provide strong evidence for the optimization approach. (2) Two-architecture validation (dense + MoE) partially addresses model-specificity concerns. (3) The sparsification analysis (Figure 4) is thorough and produces a non-trivial finding about the accuracy-efficiency tradeoff. (4) The reward ablation (Table 5) includes a revealing negative result (length penalty hurts). (5) The training stage ablation (Table 3) cleanly isolates the contribution of each component.

Summary of experimental limitations: (1) Single benchmark family (RefCOCO) with no quantitative results on other segmentation tasks (panoptic, instance, semantic, video). (2) No confidence intervals or statistical tests reported. (3) No ablation of the RL algorithm choice — only GSPO is tested. (4) No direct comparison to a fixed-vertex polygon baseline within the same framework. (5) The pre-training benefit is inconsistent and its scalability is not demonstrated. (6) No test-time compute scaling analysis (best-of-N, majority voting, verifier-based selection). (7) No resolution or object-size sensitivity analysis. (8) The format error rate during RL and its interaction with optimization is unanalyzed.

6. Limitations and Trade-offs

Limitation 1: Long Sequences Remain a Bottleneck for Complex Shapes, and the Method Fundamentally Cannot Represent Holes

The paper is transparent about this limitation in Appendix A, stating:

"While SimpleSeg eliminates task decoders, long sequences remain a bottleneck for high-resolution, highly-curved objects. Errors tend to cluster at sharp corners and thin structures under aggressive sparsification."

This is not merely an implementation inconvenience — it reflects a structural tension in the point trajectory representation. The Suzuki-Abe algorithm extracts a single outer boundary. For objects with holes (a donut, a button, a chain-link fence), a single boundary polygon cannot represent the interior void. The failure case in Figure 12 (the donut) illustrates this clearly: the model segments the outer boundary but has no mechanism to represent the hole. The rendered mask fills the entire circular region, producing a false-positive region in the center.

The consequence is twofold. First, hole topology is excluded from the representational capacity — a practitioner segmenting donuts, wheels, letters with counters (A, B, D, O, P, Q, R), or any object with interior voids cannot rely on SimpleSeg as currently formulated. Extending to multiple contours per instance (outer boundary + inner boundaries for each hole) would require both a representation change (how to encode multiple polygons in a single text response) and a rendering change (even-odd fill rule), neither of which is addressed. Second, even for hole-free objects, the sequence length for highly curved or finely detailed boundaries scales with boundary complexity, and Figure 4 shows that the language model's autoregressive decoding reliability degrades at long sequence lengths — performance peaks at ~221 tokens and drops at ~859 tokens. For applications requiring sub-pixel precision on intricate shapes (medical imaging, satellite imagery, industrial inspection), the token budget needed for geometric fidelity may exceed what the MLLM can generate reliably.

The paper provides partial evidence for this limitation: Figure 4 quantifies the accuracy-sequence-length tradeoff, and Figure 12 shows qualitative failure on a donut. But the limitation is not systematically measured — there is no breakdown of performance by shape complexity (number of vertices in the ground-truth annotation, boundary curvature, presence of thin structures), no quantitative analysis of corner accuracy or thin-structure preservation, and no statistics on what fraction of real-world segmentation tasks involve hole topologies.

Mitigation status: The paper acknowledges the limitation in Appendix A as future diagnostic work, suggesting "boundary F-score, vertex-wise Chamfer distance, and token-per-mask analyses across object scales." It does not propose a solution for hole representation or long-sequence reliability. For the sequence length issue, Figure 6 shows that RL partially mitigates the problem by adaptively adjusting vertex count, but this only shifts the operating point within the existing representational constraints — it does not solve the fundamental tradeoff between fidelity and token budget for very complex shapes.

Limitation 2: Difficulty Estimation for Adaptive Sparsification Is Not Operationalized; the Optimal Vertex Count Is Found by Exhaustive Sweep, Not Learned

Figure 4 identifies a clear optimal operating point (~221 tokens average) for the sparsification tolerance ε, and the paper frames this as a "sweet spot — between the model's capacity for sequential understanding and the contour's geometric fidelity" (Takeaway 4, Section 4.3). However, this sweet spot was found by training and evaluating separate models at different ε values — a post-hoc analysis that is not available to a practitioner deploying on a new domain or a different model architecture.

The paper does not provide a mechanism for predicting or learning the optimal ε for a given model, dataset, or object class. A practitioner deploying SimpleSeg on their own domain (medical images with smooth organ boundaries, aerial imagery with rectilinear buildings, natural images with mixed complexity) faces a chicken-and-egg problem: they need to train the model to determine what ε works best, but choosing ε requires knowing how the trained model will perform. The paper's analysis (Figure 4) is descriptive, not prescriptive.

The consequence is that the representational design — which the paper frames as a key contribution of the method — is not algorithmically determined but empirically swept. This undermines the simplicity narrative: SimpleSeg is "simple" only after the practitioner has performed an expensive hyperparameter search over the sparsification tolerance, training and evaluating multiple models at different vertex densities. The paper does not report the computational cost of this sweep (at minimum, 3+ full SFT training runs on 800k samples with 32 GPUs each, per Figure 4's data points), but it is substantial.

A secondary issue: the optimal ε is likely model-dependent and data-dependent. A larger MLLM with better long-range sequential modeling might benefit from higher vertex counts; a smaller model might prefer coarser sparsification. A dataset with mostly rectilinear objects (buildings, furniture) might need fewer vertices than one with biological shapes (leaves, cells). The paper's single-point finding (221 tokens for Qwen2.5-VL-7B on RefCOCO) does not generalize to other settings, and the paper provides no guidance for how ε should scale with model capacity, image resolution, or shape complexity.

The paper partially acknowledges this by showing that RL can adaptively adjust effective vertex count (Figure 6: RL decreases length at high density, increases length at low density), but this adaptation happens within a fixed ε-determined initialization — RL shifts the operating point along the curve but cannot discover the optimal region of the curve if the initial ε is far from optimal. The experiment in Figure 6 tests only two ε values (0.001 and 0.01); it does not demonstrate that RL starting from any ε converges to the same optimal vertex count.

Mitigation status: The paper does not address this as a limitation. Takeaway 4 presents the sweet spot as a finding rather than an unresolved design challenge, and the paper offers no method for automatic ε selection, no learned sparsification policy, and no analysis of how ε optimality transfers across models or domains.

Limitation 3: Single Benchmark Evaluation on Referring Expression Segmentation Only; No Quantitative Evidence for the Claimed Task Generality

The paper makes strong claims about task generality in Section 1:

"By framing segmentation as a text-generation problem, our approach is inherently flexible. The model can be easily adapted to a wide range of vision-language tasks that require precise spatial localization."

and in Section 3.1: "we extend the perception task via a unified query interface beyond just the query of the reference phrase." The qualitative results in Appendix C (Figures 8–11) show examples of point→mask, bbox→mask, text→point, text→bbox, panoptic segmentation, and multi-part segmentation. These are visually impressive demonstrations.

However, all quantitative results (Tables 1, 2, 3, 5; Figures 4, 6) are exclusively on the RefCOCO referring expression segmentation and comprehension benchmarks. There is no quantitative evaluation of:

Point-conditioned segmentation (point→mask): How often does the model correctly segment the object at a given point? How sensitive is it to point placement (on-center vs. near-boundary)?
Bbox-conditioned segmentation (bbox→mask): When given a loose or tight bounding box, does the model segment the contained object or the box region itself?
Panoptic segmentation: What is the panoptic quality (PQ) on standard benchmarks like COCO Panoptic? The qualitative examples show the model can handle multi-object scenes, but no metrics are reported.
Multi-part segmentation: Figure 11 shows the model segmenting parts of objects and stuff classes (sky, floor, road), but there are no part-level or stuff-level metrics.
Class-agnostic segmentation: The model is trained on RefCOCO's referring expressions, which always target specific object instances. Can it handle open-vocabulary segmentation, where the input is a category name rather than a referring expression?

The consequence is that the "task generality" claim is supported anecdotally but not empirically. A practitioner evaluating SimpleSeg for a non-RefCOCO use case (e.g., interactive segmentation from user clicks, automatic instance segmentation for a robotics pipeline, panoptic segmentation for autonomous driving) has no quantitative basis for estimating performance. The paper's qualitative examples are selected successes; the failure rate and failure modes for these extended tasks are unknown.

The gap between the paper's framing and its evaluation is significant: the introduction and methodology sections emphasize the unified 4-tuple interface as a core innovation that "multiplies supervision sources" and enables diverse tasks, but the experimental section evaluates only the text→mask (RES) and mask→bbox (REC) pathways. The other 14 possible task types in the Cartesian product of [text, point, bbox, mask] receive either qualitative examples or no evaluation at all.

This limitation is compounded by the lack of comparison to task-specific baselines for the extended tasks. The point→mask task could be compared against SAM (the Segment Anything Model) on standard point-prompt benchmarks. The text→point and text→bbox tasks could be compared against specialized referring expression comprehension and visual grounding models. Without these comparisons, the paper's claim that SimpleSeg provides "versatile perception capacity" (Section 4.3, final paragraph) is unvalidated — we don't know whether the unified model's performance on these auxiliary tasks is competitive with specialized systems, or whether the unified training provides benefits (positive transfer) or costs (negative interference) relative to single-task training.

Mitigation status: The paper does not acknowledge this as a limitation. The extended task results are presented as a strength ("This significantly demonstrates the generality of our framework," Section 4.3) without noting the absence of quantitative metrics. The Appendix C qualitative examples are provided as evidence of generality, but the paper does not frame the lack of quantitative evaluation as a limitation or suggest benchmarks for future work to validate the task generality claim.

Limitation 4: RL Relies on Ground-Truth Masks for Reward Computation; No Path to Self-Improvement Without Dense Annotations

The RL stage's reward function (Section 3.2) requires ground-truth binary masks to compute the IoU and centroid distance rewards. For each training instance, the system must render the model's generated point sequence into a mask and compare it pixel-by-pixel against a human-annotated (or SAM-generated) ground-truth mask. This means the RL training signal is fully supervised — it is reinforcement learning in the algorithmic sense (optimizing a sequence-level reward via policy gradients), but not in the self-improvement sense (using model-generated or automatically available feedback).

The consequence is that the RL stage cannot be applied to unlabeled data. The paper's data annotation pipeline (Section 3.1, Figure 3) uses SAM to generate pseudo-ground-truth masks from web images, but even these pseudo-labels require running a separate segmentation model. The RL stage does not use these automatically generated masks — it uses the RefCOCO human annotations (Appendix B.1: "the prompt set with 400k samples for Reinforcement Learning was also derived from the RefCOCO series"). This means the 10-point gIoU gain from RL (Table 3) is contingent on access to a dataset with instance-level mask annotations.

This is a practical limitation for several deployment scenarios:

Domain adaptation: If a practitioner wants to deploy SimpleSeg on a new domain (medical images, satellite imagery, industrial parts), they need dense mask annotations for that domain to run RL. The SFT stage alone (60–65 gIoU, Table 3) may be insufficient for the target application.
Scaling to large-scale pre-training: The paper envisions SimpleSeg as a pre-training task for foundation models (Section 1: "it can be seamlessly and efficiently integrated as a new, core pre-training task"). But the RL stage's dependence on ground-truth masks means that pre-training can only use SFT on automatically labeled data (which Table 3 shows is ineffective without task-specific SFT: pre-training-only achieves 25.3 gIoU). The RL benefit cannot be realized at web scale without a source of high-quality mask annotations.
Iterative self-improvement: The paper's Takeaway 4 notes that RL "automatically finds" the optimal vertex density, suggesting a self-correcting dynamic. But this dynamic requires external ground truth; the model cannot evaluate its own mask quality without a reference.

This limitation is particularly significant because the paper's core innovation is the RL stage — it is what distinguishes SimpleSeg from prior decoder-free approaches (Text4Seg, VisionLLM) that used only SFT, and it provides the majority of the performance gain (+10 gIoU in Table 3). If the RL benefit cannot be realized without dense annotations, the practical advantage over decoder-based methods (which also require mask annotations for training) narrows considerably. The "simplicity" argument then rests primarily on architectural purity (no decoder to implement and maintain) rather than on reduced annotation requirements or the ability to leverage unlabeled data.

Mitigation status: The paper does not address this limitation. The data annotation pipeline (Section 3.1) provides a mechanism for generating automatic masks at scale, but it is used only for pre-training SFT data, not for RL. The paper does not explore whether SAM-generated pseudo-masks could serve as RL rewards (which would create a self-improvement loop where the model learns to reproduce SAM's outputs), whether the SFT model's own predictions could be used for self-training, or whether the RL reward could be restructured to use weaker supervision (e.g., point clicks, bounding boxes, image-level labels). The dependence on dense mask annotations for the RL stage is an unstated constraint on the method's applicability.

Limitation 5: No Analysis of Inference Latency, Memory, or the Serial Generation Bottleneck

The paper measures computational cost exclusively in terms of training infrastructure (32 GPUs, batch size 256, optimizer choice) and output token count (the ~221 token average at the optimal operating point). There is no analysis of inference-time latency, GPU memory consumption, or throughput. For a method that generates segmentation masks autoregressively — one coordinate pair at a time, each conditioned on all previously generated tokens — this is a critical omission.

Autoregressive decoding for coordinate sequences introduces a serial bottleneck that does not exist in decoder-based segmentation systems. A decoder-based system like SAM or LISA produces the entire mask in one parallel forward pass through the decoder (a few convolutional or transformer layers). SimpleSeg must generate each of ~60 coordinate pairs (~120 tokens including brackets and commas) sequentially, with each token requiring a full forward pass through the entire MLLM (7B parameters for Qwen2.5-VL-7B, or 2.8B activated parameters for Kimi-VL). The latency is:

$T_{\text{SimpleSeg}} \approx V_{\text{tokens}} \times T_{\text{per-token}}$

where $T_{\text{per-token}}$ is the time for one autoregressive step through the full model (typically 10–50ms for a 7B model on a modern GPU, depending on batch size and KV-cache management). For 221 tokens, this is approximately 2–11 seconds per mask at batch size 1. A decoder-based method producing a mask in a single pass would take 10–50ms.

The consequence for deployment: SimpleSeg is inappropriate for real-time or high-throughput segmentation applications. Interactive segmentation (where a user clicks and expects instant mask feedback), video segmentation (processing 30+ frames per second), or large-batch offline processing (segmenting millions of images) would all be severely bottlenecked by the autoregressive generation latency. The paper's qualitative examples (Figure 2, 8–11) show impressive accuracy on diverse domains, but a practitioner in autonomous driving or robotic manipulation — where decisions must be made in milliseconds — cannot use this method regardless of its accuracy.

This latency problem is inherent to the decoder-free design choice. It is not an implementation inefficiency that can be optimized away — autoregressive generation is structurally sequential, and while techniques like speculative decoding or KV-cache optimization can reduce per-token latency, they cannot make 221 serial steps as fast as one parallel step. The paper's positioning of SimpleSeg as "simple" and "efficient" is accurate with respect to architectural complexity and training, but misleading with respect to inference efficiency.

The paper does report one data point relevant to this tradeoff: the token count analysis in Figure 4 shows that coarser sparsification (fewer vertices) reduces sequence length, which would reduce latency. But the paper does not frame this as a latency-accuracy tradeoff or provide latency measurements at different ε values. There is no comparison of wall-clock inference time between SimpleSeg and a decoder-based baseline on the same hardware.

An additional inference-time concern: the paper's RL stage uses 8 response candidates per prompt for group-based optimization (GSPO). If this multi-sample strategy were used at inference time (e.g., generate 8 candidate masks and select the best via self-consistency), latency would multiply by 8×. The paper does not explore test-time compute scaling for SimpleSeg, so the latency cost of potential accuracy improvements is unknown.

Mitigation status: The paper does not acknowledge this limitation. There is no discussion of inference efficiency, latency, throughput, or deployment constraints anywhere in the paper. The training infrastructure details (Appendix B.2) focus exclusively on training-time considerations (batch size, optimizer, learning rate). The omission is significant because it limits the paper's practical guidance — a practitioner reading the paper cannot assess whether SimpleSeg is suitable for their deployment scenario without independently benchmarking inference latency on their hardware, which requires implementing the method first.

Limitation 6: The 14× Larger Model Baseline and Broader Scaling Comparison Are Absent; the Latent Capability Claim Has Not Been Tested Across Model Scales

The paper's central claim — that standard MLLM architectures possess "a strong, inherent, but previously latent, capacity for precise, pixel-level perception" (Abstract, Takeaway 1) — is demonstrated on exactly two model variants: Qwen2.5-VL-7B (7B parameters, dense) and Kimi-VL (2.8B activated parameters, MoE). Both are mid-scale models from a similar era of MLLM development (2024–2025), both use Vision Transformer encoders with similar design philosophies, and both were developed by research labs with overlapping technical approaches.

The claim that the capacity is "inherent" to the architecture implies it should exist across different scales, different architectural families, and different training regimes. The paper does not test this:

Scale variation: Is the latent perception capability present in a 1B-parameter model? A 70B-parameter model? Does it scale with model size (larger models produce more precise masks), saturate, or exhibit emergent behavior at a critical scale? The paper's two test points (2.8B activated, 7B total) are too close in effective capacity to characterize a scaling relationship.
Architectural family variation: Qwen2.5-VL and Kimi-VL share design conventions common to Chinese LLM labs (Qwen-derived architectures, specific training data mixtures, specific vision encoder choices). Would the same approach work on LLaVA-NeXT (which uses a different vision encoder and connector), on InternVL2 (which uses a different training paradigm), or on proprietary models like GPT-4V or Gemini? The paper shows two models, but they are from overlapping research lineages.
Training regime variation: Both base models were pre-trained with substantial vision-language alignment data. Would a model pre-trained primarily on text with only minimal vision adaptation exhibit the same latent capability? The paper's pre-training ablation (Table 3) shows that pre-training on web data helps SFT (+4–5 gIoU), but this is additive — the model already achieves 60+ gIoU from SFT on RefCOCO alone, suggesting the base models' original pre-training already provided substantial perceptual capability. Whether this capability is a universal property of the transformer architecture or a consequence of the specific pre-training recipe used by Qwen and Kimi is untestable with the data provided.

The consequence is that the "latent capability" claim is overbroad relative to the evidence. A more accurate statement would be: "Two mid-scale MLLMs (Qwen2.5-VL-7B and Kimi-VL) from similar architectural lineages can achieve competitive pixel-level segmentation when trained with point trajectory outputs and RL-based geometric optimization." This qualified claim is still significant — it shows that decoder augmentation is not strictly necessary for these specific models — but it does not support the paper's stronger implication that all MLLMs possess this capability latently and that the field's investment in decoder architectures was unnecessary.

This limitation is particularly relevant because the paper's title and framing position it as a general finding about MLLM perception, not a method that works on two specific models. The Takeaway 1 language ("Standard MLLM Architectures have a strong, inherent, but previously latent, capacity") uses the present tense and universal quantifier, implying a property of the architecture class. The experiments support a much narrower claim.

The paper also does not compare to a larger model baseline to test whether scaling model size (with or without test-time compute) is more effective than the SFT→RL pipeline on a smaller model. This is a standard analysis in work that studies capability emergence (e.g., scaling laws papers test across multiple orders of magnitude of model size). Without it, we cannot distinguish between "the capability was latent in all standard architectures" and "the capability emerges at the ~3–7B scale but is absent in smaller models" or "the capability is present in models pre-trained with certain data mixtures but not others."

Mitigation status: The paper does not acknowledge this as a limitation. The two-model validation is presented as evidence of architecture-agnosticism rather than as a preliminary test that requires broader validation. The paper would be strengthened by (a) qualifying the "inherent capacity" claim to reflect the tested model scope, (b) calling for community validation on additional architectures and scales as future work, and (c) discussing what properties of the tested models might be necessary for the observed capability (vision encoder resolution? pre-training data scale? instruction tuning format?). None of this is present.

7. Implications and Future Directions

How This Work Changes the Landscape

SimpleSeg does not introduce a new architecture, a new loss function, or a new model scale. Its primary contribution is diagnostic, not algorithmic: it demonstrates that a standard, unmodified MLLM can achieve pixel-level segmentation at a quality level comparable to decoder-augmented systems. This finding, if it proves robust across architectures and domains, reframes what the research community should consider the bottleneck for fine-grained perception in vision-language models. The bottleneck is not the absence of a specialized mask decoder — it is the choice of output representation and the alignment of the training signal with geometric quality.

This is a reframing of the problem statement, not a paradigm shift. The dominant paradigm — "to get segmentation from an MLLM, add a decoder" — is challenged by a counterexample, not overturned. A single counterexample (or two, counting both Qwen2.5-VL-7B and Kimi-VL) is sufficient to falsify a universal claim ("decoders are necessary"), and SimpleSeg provides that falsification. But it does not yet establish a new positive claim ("decoders are unnecessary in general"), because the demonstrated approach has clear limitations (hole topology, long-sequence reliability, inference latency) that decoder-based methods handle differently. The paper's contribution is better characterized as opening a viable research direction that was previously considered closed — the field assumed decoder-free segmentation would be low-fidelity (as in Text4Seg's 71.4 cIoU without SAM), and SimpleSeg shows this assumption was wrong.

The paper also resolves a specific contradiction in the prior literature. Text4Seg and VisionLLM demonstrated that decoder-free segmentation was possible, but at a substantial fidelity cost relative to decoder-based methods (Text4Seg w/o SAM: 71.4 vs. Groundhog: 74.2, a ~3-point gap). This created a narrative that architectural augmentation was the price of quality. SimpleSeg narrows that gap to ~0.6–1.8 points (74.6–74.8 for SimpleSeg with pre-training vs. 75.4 for Text4Seg+SAM), showing that the fidelity gap was partly an artifact of suboptimal representation and training, not a fundamental limitation of language-space outputs. The reconciliation is specific: RLE encodings and token-level supervision are the culprits; point trajectories with RL-based geometric optimization recover most of the lost ground.

The work redirects research attention in several specific ways:

Away from decoder architecture design and toward output representation design. If decoder-free methods can approach decoder-based performance, the marginal return on improving mask decoder architectures (better upsampling, multi-scale feature fusion, transformer-based decoders) may be lower than previously assumed, while the return on better textual representations of geometry may be higher.
Toward RL for perceptual outputs from language models. The paper demonstrates that sequence-level geometric rewards (IoU on rendered masks) provide a ~10 gIoU point gain over token-level supervision (Table 3), a gain that is remarkably consistent across three datasets. This opens a new class of training objectives for spatial outputs — keypoint localization with PCK rewards, edge detection with boundary F-score rewards, instance tracking with MOTA rewards — that the MLLM community has not explored because RL for reasoning tasks (math, code) was the dominant focus.
Toward adaptive, learned sparsification. The paper's finding that the optimal vertex count is unimodal (Figure 4) and that RL automatically adjusts vertex count toward an efficient operating point (Figure 6) suggests that representation density should be learned, not fixed at data preparation time. This is a genuinely new problem formulation: rather than pre-processing all masks with a single ε, an MLLM could learn to allocate vertices dynamically based on shape complexity, object size, and contextual importance. This connects to broader work on adaptive computation in transformers.
Away from treating segmentation as a standalone task and toward treating it as one element in a unified spatial reasoning interface. The 4-tuple formulation [text, point, bbox, mask] with bidirectional querying means that segmentation is not a separate capability bolted onto an MLLM but a modality that can be interleaved with other spatial reasoning. A model that can go from text to mask and from mask to text can participate in compositional spatial reasoning (e.g., "segment the person, then describe the shape of their left hand in the mask") that modular decoder-based systems struggle with because the language model cannot directly inspect the decoder's output.

One important limitation on the landscape impact: the demonstrated capability is on two model families from a narrow era (Qwen2.5-VL-7B and Kimi-VL, both developed in late 2024/early 2025 with similar architectural philosophies). If the latent perception capability is a consequence of specific pre-training data mixtures or vision encoder properties shared by these models — rather than a universal property of MLLM architectures — then the reframing is narrower than the paper implies. The community needs negative results (architectures where the approach fails, scales where it degrades) to establish the boundary conditions of the finding. Without those, the paper's reframing is suggestive but not definitive.

Follow-Up Research This Work Enables

1. Stress-testing the "latent capability" claim across model scales and architectural families. The paper's central Takeaway 1 — that standard MLLM architectures possess inherent pixel-level perception capacity — is tested on two mid-scale models (2.8B activated, 7B total). A rigorous test requires applying the SimpleSeg pipeline (point trajectory representation, unified 4-tuple interface, SFT→RL training) to a broader range: a small model (1B parameters, e.g., Qwen2.5-VL-1B or SmolVLM), a large model (70B+, e.g., Qwen2.5-VL-72B or LLaVA-NeXT-72B), a model from a different architectural lineage (LLaVA-NeXT uses a different vision encoder and connector design from Qwen-VL), and a model with minimal vision-language pre-training (to test whether the capability is pre-training-dependent or architecture-inherent). The specific experiment: train SimpleSeg on each model using the identical RefCOCO data and SFT→RL protocol, measure cIoU, and characterize the scaling relationship. A strong result for the paper's claim: cIoU scales monotonically with model size, and all models reach at least 60+ cIoU with SFT alone (demonstrating that the capability is architecture-inherent and scales predictably). A weak result: performance collapses below some scale threshold, or fails entirely on models without extensive vision-language pre-training, suggesting the capability is contingent on specific training recipes rather than architectural universality. This experiment also addresses the missing "larger model baseline" limitation identified in Section 6 — it would show whether scaling model size provides gains comparable to or exceeding those from the SFT→RL pipeline.

2. Hole-aware polygon representation and multi-contour decoding. The current SimpleSeg representation cannot handle objects with holes (Figure 12: the donut failure), because the Suzuki-Abe algorithm extracts only the outer boundary and the text grammar has no convention for encoding multiple contours per instance. A natural extension: extend the text grammar to support multiple polygons per mask, using a format like [[outer_polygon], [inner_hole_1], [inner_hole_2]] where the rendering step applies an even-odd fill rule. The specific experiment: create a test set of objects with holes (donuts, rings, letters with counters from the COCO-Stuff or ADE20K datasets), measure SimpleSeg's current hole-rendering error rate (false positive regions inside holes), implement the multi-contour extension, retrain SFT+RL, and report the reduction in hole errors. The paper already identifies this as a limitation (Appendix A, Figure 12), making it a low-risk, high-clarity follow-up. A secondary question this enables: does RL naturally learn to distinguish outer boundaries from inner boundaries when both are represented in the same token format, or does it require explicit topology-aware rewards?

3. Learned, dynamic sparsification policies via meta-RL or difficulty-conditioned ε prediction. Figure 4 shows that the optimal vertex count is a unimodal function of ε, but the optimal ε is found by exhaustive sweep — a practitioner cannot select ε for a new domain without training multiple models. A follow-up would develop a learned sparsification policy that predicts the appropriate number of vertices for a given object instance based on its shape complexity, size, and the model's decoding reliability. The specific design: train a lightweight predictor (a small MLP or an MLLM prompt) that takes an image crop and an object description and outputs a target vertex count or ε value, trained to maximize the expected IoU of the resulting SFT-generated polygon. Alternatively, make sparsification part of the RL optimization by allowing the model to output a variable number of vertices and penalizing vertex count in the reward with an adaptive coefficient that targets a specified accuracy-efficiency tradeoff curve. The paper already shows that RL can adjust vertex count without explicit penalties (Figure 6), suggesting that a multi-objective RL formulation (maximize IoU - λ × token_count) could directly learn the accuracy-efficiency Pareto frontier. The experiment: train a model with this multi-objective reward across different λ values, plot the resulting accuracy-efficiency curve, and compare to the fixed-ε baselines from Figure 4. A strong result: the RL-learned representations achieve higher IoU at the same token budget than any fixed-ε baseline, because they allocate vertices non-uniformly (more vertices at sharp corners, fewer on straight edges) rather than using uniform Douglas-Peucker sparsification.

4. Test-time compute scaling for decoder-free segmentation: best-of-N, verifier-guided selection, and iterative refinement. The paper uses single-response generation at inference time, but the RL stage already generates 8 candidate responses per prompt for GSPO optimization. A natural extension: at test time, generate N candidate point sequences (N = 4, 8, 16, 32), render each to a mask, and select the final mask via self-consistency (majority voting on pixel-level mask agreement) or via an IoU prediction model trained to estimate mask quality from the generated sequence alone (analogous to a process reward model but for geometric outputs). The specific experiment: sweep N across powers of 2 on the RefCOCO validation sets, measure cIoU as a function of test-time compute, and determine whether best-of-N selection can close the remaining ~0.6–1.8 point gap to Text4Seg+SAM. A secondary experiment: use the RL-trained value function (from GSPO's critic) as a per-sequence quality estimator, compare its selection accuracy against ground-truth IoU ranking, and report whether value-guided selection outperforms random selection at fixed N. This connects SimpleSeg to the growing literature on test-time compute scaling for reasoning tasks (DeepSeek-R1, Kimi k1.5) and would establish whether decoder-free segmentation benefits from inference-time compute in the same way that reasoning tasks do. The paper's architecture-agnostic design makes this experiment straightforward to implement — no decoder means no architectural changes needed to support batched generation.

5. Cross-domain quantitative evaluation of the unified query interface on non-RefCOCO tasks. The paper claims task generality via the 4-tuple interface and shows qualitative examples of point→mask, bbox→mask, text→point, text→bbox, and panoptic segmentation (Appendix C, Figures 8–11), but provides quantitative metrics only for text→mask (RES) and mask→bbox (REC) on RefCOCO. A rigorous follow-up would evaluate all 16 task types in the Cartesian product of [text, point, bbox, mask] on standard benchmarks: point→mask on the SAM point-prompt benchmark (or COCO with simulated point clicks), bbox→mask on COCO instance segmentation (using ground-truth boxes as prompts), text→point on Visual Genome or RefCOCO referring expression comprehension (point-based localization), and multi-object panoptic segmentation on COCO Panoptic (using the text→mask pathway with category names as prompts). The specific experiment: take the SimpleSeg model trained on RefCOCO (no additional task-specific training), evaluate all pathway-benchmark pairs, and report both the absolute performance and the relative performance compared to task-specific baselines (SAM for point→mask, specialized REC models for text→point, Panoptic-DeepLab for panoptic). This would convert the paper's qualitative generality demonstrations into a quantitative task generality matrix, revealing which pathways benefit from the unified training (positive transfer) and which suffer from task interference. The paper's current "task generality" claim, supported only by selected qualitative examples, would either be substantiated or qualified by this experiment. A negative result (e.g., point→mask performance substantially below SAM) would not invalidate the paper's core contribution but would appropriately bound its generality claims.

6. Self-improvement without dense mask annotations: RL from SAM pseudo-labels or from self-consistency. The RL stage currently requires human-annotated RefCOCO masks to compute IoU rewards, limiting its applicability to domains without dense annotations (Limitation 4 in Section 6). A follow-up would investigate whether RL can use automatically generated reward signals: (a) SAM-generated pseudo-masks as ground truth (using the data annotation pipeline from Section 3.1 to generate rewards at scale), (b) self-consistency rewards (generate N candidate masks for the same input, compute pairwise IoU, and reward masks that agree with the consensus), or (c) cycle-consistency rewards (mask → text description via a captioning model → new mask → IoU between original and reconstructed mask). The specific experiment: take the RefCOCO-trained SFT model, run RL on LAION/Coyo web images using SAM pseudo-masks as rewards (no human annotations), and measure the resulting cIoU on RefCOCO test splits compared to both the SFT baseline and the human-annotation RL model. A strong result: SAM-pseudo-label RL recovers 70–80% of the human-annotation RL gain (+7–8 gIoU instead of +10), making the approach viable for domains without annotation budgets. A secondary experiment: compare SAM-pseudo-label RL to the pre-training + SFT approach from Table 3, testing whether RL on automatically labeled data provides more benefit than simply doing more SFT on the same data. This would address the most significant practical limitation of the current method and align with the paper's vision of SimpleSeg as a scalable pre-training task for foundation models.

Practical Applications and Downstream Use Cases

1. Interactive image editing with natural language and spatial precision. Current MLLM-based image editing tools (e.g., SeedEdit, InstructPix2Pix) typically operate on either the full image or bounding-box-defined regions. SimpleSeg enables a workflow where a user says "make the sky more dramatic, but only up to the building silhouettes" and the model produces both the mask (as point coordinates tracing the skyline) and the edited image, all within a single MLLM without switching between a language model and a separate segmenter. The practical benefit is reduced system complexity — one model handles the understanding, the localization, and the editing instruction — and improved precision over bounding boxes for irregular boundaries like skylines, hair, or tree canopies. The paper's ~74 cIoU on referring segmentation means the mask will capture the rough shape correctly but may miss fine details (individual leaves, wispy hair), so the application is best suited to edits where exact boundary precision is less critical than getting the right object (e.g., sky replacement, background blur, object recoloring). The human-readable coordinate format means the mask can be inspected and manually corrected by a user before committing to an edit, a debugging advantage over dense mask tensors.

2. GUI-grounded agents that can click precise UI elements. GUI agents (InfiGUIAgent, UI-TARS) need to locate and interact with on-screen elements like buttons, text fields, and icons. Bounding boxes are the current standard, but for densely packed UIs (data tables, icon grids, mobile layouts), bounding boxes overlap or cover adjacent elements, leading to misclicks. SimpleSeg's mask output enables the agent to delineate the exact clickable region of a button (excluding its shadow or decorative border) or to select a specific cell in a spreadsheet rather than the approximate rectangular region. The paper's qualitative results on "in-screen content" (Figure 2: anime, data charts, infographics) demonstrate that the model's pixel-level perception extends to rendered digital content, not just natural photographs. The practical integration: an agent pipeline where the MLLM receives a screenshot, identifies the target element via text description, outputs its mask as point coordinates, computes the mask centroid for the click location, and passes the click coordinates to the UI automation tool — all using text-space outputs that can be logged, debugged, and audited. The ~87% Acc@0.5 on REC (Table 2) means bounding-box-level localization is reliable; the mask provides additional precision for elements where the bounding box would be ambiguous.

3. Large-scale training data generation for specialist segmentation models. The data annotation pipeline described in Section 3.1 (Grounding-DINO → SAM → contour extraction → VLM captioning) is already a working automatic labeling system. SimpleSeg could replace or augment this pipeline by generating mask annotations directly from text descriptions, without running SAM as an intermediate step. The practical scenario: a team needs instance segmentation labels for a custom domain (e.g., retail product images, medical scans, satellite imagery) but has only image-level captions or object category lists. They prompt SimpleSeg with "segment all cars in this image" and obtain polygon annotations that can be used to train a faster, real-time segmentation model (e.g., YOLO-seg, Mask2Former) for deployment. The benefit is reducing or eliminating the need for SAM in the annotation loop, which matters when SAM fails on the target domain (e.g., medical images where SAM's general-purpose training doesn't transfer well). The paper's pre-training results (Table 3: +4–5 gIoU from web-scale pre-training) suggest that SimpleSeg already generalizes beyond RefCOCO's distribution, making this application plausible. The quality of generated annotations would need validation against human labels on the specific domain; the paper's ~74 cIoU provides a starting estimate for the expected annotation quality.

4. Accessible debugging and auditing of segmentation model outputs in high-stakes applications. In medical imaging, autonomous driving, or industrial inspection, model decisions need to be explainable and auditable. A decoder-based segmentation model produces a dense mask tensor — human-interpretable when visualized as an overlay, but not inspectable as data. SimpleSeg's human-readable coordinate sequences mean that an auditor can examine individual vertex placements ("the tumor boundary contour includes this pixel at [0.342, 0.567] — is that correct?") and identify systematic errors (consistent oversegmentation at sharp corners, vertex drift along straight edges). The specific benefit: integration into a human-in-the-loop review system where the model's mask is displayed as coordinates alongside the image, an expert can edit the coordinates directly (move, add, or delete vertices), and the model can be fine-tuned on the corrected coordinates. This is not feasible with dense mask tensors without specialized annotation tools. The paper's interpretability benefit (Section 1, key benefits: "explicit, human-readable coordinate sequences") directly enables this use case, though the paper does not demonstrate it. The practical constraint: for complex shapes with ~200 vertices, manual inspection of every coordinate is impractical, but a reviewer could sample and adjust critical vertices (e.g., the ones the model placed with low confidence, if a confidence measure were available).