Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and Grounding

ArXiv: 2601.10611

🎯 Pitch

An open 8B vision-language model, trained entirely without proprietary model distillation, not only matches state-of-the-art open video understanding but introduces pixel-level video pointing and tracking capabilities that beat Gemini 3 Pro—achieving 38.4 vs. 20.0 F1 on pointing—proving high-quality, human-collected spatio-temporal data can leapfrog closed-source distillation.

1. Executive Summary

Molmo2 introduces a family of fully open vision-language models that achieve state-of-the-art video understanding and grounding among open-source systems, trained entirely without distilling from proprietary VLMs. The core contribution is a suite of nine novel datasets—including dense video captions averaging 924 words per video, free-form video QA, open-vocabulary video pointing with 650k queries, and complex object tracking data—paired with a training recipe employing packing and message-tree encoding, bidirectional attention on vision tokens, and a token-weighting strategy to balance learning across diverse tasks. On video grounding, Molmo2-8B substantially outperforms existing open-weight models like Qwen3-VL (35.5 vs. 29.6 accuracy on video counting) and surpasses proprietary models like Gemini 3 Pro on certain tasks (38.4 vs. 20.0 F1 on video pointing; 56.2 vs. 41.1 J&F on video tracking), while on short-video understanding it matches or exceeds open-weight models on MVBench and MotionBench, establishing that open-data training can approach proprietary performance when grounded in large-scale, human-annotated spatio-temporal data rather than closed-model distillation.

2. Context and Motivation

The Core Problem: Open Video-Language Models Lack Grounding and Are Trapped in a Distillation Loop

The fundamental problem this paper addresses is a structural failure in the open-source ecosystem for video-language models: the strongest open-weight VLMs rely on synthetic data distilled from proprietary models, which creates a closed loop that prevents genuine independent progress. Even setting aside this distillation dependency, existing open models lack a critical capability — grounding, the ability to answer questions by pointing to specific locations in space and time rather than just producing text descriptions. This gap is not just academic; it directly blocks real-world applications.

The paper frames this problem through two intertwined deficiencies (Section 1):

Deficiency 1: The distillation trap. The current dominant paradigm for creating open video-language models involves generating training data by prompting proprietary VLMs—such as GPT-4 or LLaMA 3—with video frames and having them produce captions or QA pairs (as done in LLaVA-Video [184], ShareGPT4Video [19], and Eagle 2.5 [17]). This means the open model's capabilities are fundamentally bounded by what the proprietary model already knows. The paper is explicit about why this matters:

"the open-source community lacks the foundations needed to improve on the state-of-the-art video (and image) language models"

If all training data flows through a closed system whose training recipe, data, and failure modes are unknown, then researchers working with open models cannot identify whether limitations come from their architecture, their data, or the biases of the proprietary teacher. The field becomes a game of imitation rather than innovation.

Deficiency 2: Grounding is missing entirely. Even proprietary VLMs—despite their strong text-based understanding—do not offer robust grounding capabilities for video. The paper notes this explicitly:

"Although image grounding is now standard, video grounding is only supported in some proprietary systems, and even there in a limited form."

Grounding means the ability to produce spatial and temporal coordinates—answering "When did the cup fall off the table?" by returning a track of the cup across frames, or responding to "How many times does the robot grasp the red block?" by emitting points for each grasp event. Without grounding, models remain abstract reasoning engines that cannot connect language to the physical world in a way that robots, autonomous vehicles, or assistive technologies require. The paper positions this as the enabling capability for embodied AI, citing applications like "household and industrial robotics, assistive technologies, sports analytics, security and traffic monitoring, and autonomous driving" (Section 1).

Why This Problem Matters: The Real-World Stakes

The paper's motivation is not purely about benchmark leaderboard positions. It identifies concrete downstream applications that are blocked by the current state of open VLMs:

• Robotics and embodied AI. A robot that watches a demonstration video and needs to reproduce actions must understand not just what happened but where and when. Grounding—particularly spatio-temporal pointing and tracking—transforms passive video understanding into actionable spatial representation.
• Assistive technologies for the visually impaired. Describing a video is helpful; being able to answer "When did the person in the red jacket enter the room?" with a precise timestamp and location is transformative.
• Autonomous driving and surveillance. These domains require identifying and tracking multiple objects across extended time horizons with open-vocabulary queries—a capability that existing models, even proprietary ones, struggle with (the paper notes that Gemini 2.5 Pro, the strongest API model tested, "struggles to generate accurate object tracks" in Table 5).
• Video search and content moderation. Being able to search through millions of hours of video for specific events or objects—and point to exactly where they occur—requires grounding that text-only understanding cannot provide.

The paper further argues that the current situation creates an unhealthy research dynamic where progress depends on access to proprietary APIs. Since proprietary models can be deprecated, changed, or removed, research built on their outputs is fragile. True open science requires a foundation that does not depend on closed systems, which is why "all our data is constructed without distilling from proprietary models" is foregrounded as a key contribution (Section 1).

Where Prior Approaches Fall Short

The paper identifies specific limitations across three axes of prior work: data, capabilities, and model architectures.

On the data front: the distillation pipeline dominates. The paper provides a detailed account of how current video-language instruction data is created (Section 5):

1. Videos are segmented into clips.
2. A proprietary VLM (like GPT-4V) generates descriptive captions for each clip.
3. A powerful LLM synthesizes video-level captions and QA pairs from these clip captions.

This pipeline, used by LLaVA-Video [184], Eagle 2.5 [17], PerceptionLM [22], and ShareGPT4Video [19], has a critical flaw:

"a critical limitation of these approaches is their reliance on closed-source Video-Language Models (VLMs) for the initial clip captioning step. This introduces an inherent, often proprietary, bias into the generated data, as the underlying VLM’s training data and biases are inaccessible to the research community"

The paper points to a concrete data quality problem: LLaVA-Video's captions average 547 words, ShareGPT4Video's average 280 words—both far shorter than human-authored dense captions (the paper's Molmo2-Cap averages 924 words). This suggests that model-generated captions, even from strong proprietary systems, miss significant visual detail that human annotators capture.

Even the "fully open" alternatives depend on closed models. A few projects do release all training details and data—PerceptionLM [22] and LLaVA-Video [184] are cited—but "use biased data generated by proprietary VLMs." The paper draws a sharp distinction: being "open-weight" is not the same as being truly open if the training data inherits closed-model biases. This positions Molmo2 as filling a specific gap: the first state-of-the-art video-language model that is fully open in weights, data, and training code, with zero distillation from proprietary VLMs.

Grounding datasets are narrow in scope and vocabulary. The paper identifies a critical limitation in existing video grounding benchmarks and training datasets (Section 2):

"Existing video grounding datasets tend to be narrow in scope or vocabulary, which is insufficient for training general models that can respond to arbitrary user input."

For example, existing Referring Video Object Segmentation (Ref-VOS) datasets like Ref-DAVIS [66] typically involve single objects with simple referring expressions (e.g., "the person in red"). They lack the linguistic diversity needed to handle user queries like "Track all dancers that move from the left to right group" or "Point to the moment when the player celebrates scoring"—queries that combine complex reasoning, multi-object tracking, and action recognition. The paper notes a specific gap: "a lack of language prompts referring to multiple objects or diverse actions" in prior tracking datasets.

Tracking and pointing are handled by separate specialized models. The paper acknowledges the existence of specialized video object tracking models like VideoLISA [11], VideoGLaMM [121], Sa2VA [177], and SAM 3 [16]. However, these models are trained exclusively for segmentation or tracking; they "do not support other capabilities" like free-form QA or captioning. Conversely, general-purpose VLMs excel at broad understanding but cannot ground their outputs in space and time. The paper's motivation is to bridge this divide: a single model that can both describe a video in detail and point to exactly where and when things happen.

How This Paper Positions Itself

The paper's positioning strategy has several layers:

Layer 1: Extending a proven open-data philosophy to video. Molmo2 explicitly builds on the original Molmo [29], which established that state-of-the-art image understanding and grounding could be achieved without distilling from proprietary VLMs. Molmo2 extends this philosophy to the video domain, where the data challenges are substantially harder—temporal dynamics, object re-identification across frames, and much longer outputs. The paper frames this extension not as a trivial application but as requiring fundamentally new data collection pipelines (spoken descriptions for dense captions, human-and-LLM collaboration for QA, LLM-generated queries for pointing diversity).

Layer 2: Grounding as the missing differentiator. The paper positions grounding not as a niche capability but as the central missing piece in both open and proprietary VLMs. The executive summary's claim that Molmo2 "surpasses proprietary models like Gemini 3 Pro on some tasks (38.4 vs. 20.0 F1 on video pointing and 56.2 vs. 41.1 J&F on video tracking)" is strategically chosen: it shows that even the strongest closed models have a blind spot that open research can address. This inverts the typical narrative of open models merely catching up to proprietary ones—on grounding, the open model leads.

Layer 3: Data as the primary contribution, not architecture. The paper is unusual in that its core technical contributions are datasets, not novel model architectures. The architecture is described as following "the common design of combining a pre-trained LLM and a vision transformer" (Section 3.1). The innovations are in (a) the scale and diversity of human-annotated spatio-temporal data, (b) the training recipe innovations (packing, message trees, token weighting, bidirectional attention) that make training on this data efficient, and (c) the demonstration that this data mix produces capabilities that even large-scale proprietary distillation cannot match. This positions the work as a contribution to the data foundations of the field, arguing that the bottleneck is not model scale or proprietary intelligence but access to high-quality, human-annotated, diverse grounding data.

Layer 4: Fully open as a deliberate contrast. The paper draws explicit contrasts with the two categories of prior models (Section 5 and Tables 2, 3, 4):

• Proprietary API models (GPT-5, Gemini, Claude): strongest overall but closed.
• Open-weight models (Qwen3-VL, InternVL3.5, Eagle 2.5): release weights but "don't release their training recipes or don't release their training data" and most use proprietary VLM distillation.
• Open models (PerceptionLM, LLaVA-Video): release data and models but "use biased data generated by proprietary VLMs."

Molmo2 positions itself as a fourth category: open weights, open data with no distillation, and open code—the first video-language model to occupy this space at competitive performance levels. This categorization is strategic because it defines a new axis of evaluation: not just performance, but independence from closed systems.

The Specific Gap in Long-Video Understanding

The paper identifies a known weakness that Molmo2 is designed to address: open models perform poorly on long-duration content. Section 4 notes that Molmo2 "lags behind the best open-weight models" on long video benchmarks, and the paper attributes this to "a lack of open-source long (10+ minutes) training data and computational limitations." The SubtitleQA and CapQA datasets are explicitly designed to tackle videos longer than those in current benchmarks, and the long-context SFT stage (Section 3.2) directly addresses the sequence-length gap. However, the paper is honest that this remains a limitation—Molmo2 doesn't fully solve long-video understanding but provides the data infrastructure to make progress on it.

Connecting to the Broader Research Landscape

The paper situates Molmo2 within the broader trajectory of multimodal LLMs (Section 5), noting that the field has rapidly moved from image-only models [29, 70, 136] to video-capable systems. Most video LLMs "extend the CLIP-style image encoding and use image embedders to individually embed each frame in a video" [17, 22, 99], creating a linear growth in token count with video length. This has spawned research on efficient video encoding [129, 164, 170, 79, 149], which the paper engages with through its SlowFast experiments (Appendix E). The Molemo2 approach—using fixed-frame sampling with pooling rather than learned compression—is positioned as a practical middle ground that balances computational cost with representational capacity.

In summary, the paper's motivation is that the open-source video understanding community is stuck: it cannot surpass proprietary models because it depends on them for training data, and it cannot build the grounded, embodied AI applications of the future because it lacks spatio-temporal pointing and tracking. Molmo2 proposes to break this cycle by investing in large-scale human annotation of diverse video data—dense captions, free-form QA, open-vocabulary pointing, and complex tracking—creating a foundation that others can build on without closed-model dependency.

3. Technical Approach

3.1 Reader Orientation

This paper presents a family of vision-language models—Molmo2—that can watch videos (or look at single/multiple images) and produce both free-form text descriptions and spatially-temporally grounded outputs (points, tracks, and counts with timestamps and coordinates). The core problem it solves is that existing open-source VLMs either cannot ground their understanding in space and time at all, or they achieve this capability only by distilling from proprietary models with unknown biases. Molmo2's solution is to build a massive, diverse, human-annotated dataset pipeline specifically targeting grounding tasks (pointing, counting, tracking) alongside traditional video understanding (dense captioning, QA), and to train on this data mixture using a set of training innovations—packing, message trees, token weighting, and bidirectional vision attention—that make joint training on such heterogeneous tasks efficient and effective.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components arranged in a pipeline from raw data to trained model:

1. Video and Image Data Sources — a collection of Creative Commons and open-source videos/images drawn from YT-Temporal, YouTube keyword search, Koala-36M, InternVid, and various academic datasets. These are the raw visual inputs that will be annotated.
2. Data Annotation Pipelines — a set of human-in-the-loop and synthetic data generation processes that produce dense captions, free-form QA, video pointing queries, video tracking labels, multi-image QA, and multi-image pointing data. These pipelines use spoken descriptions, LLM-assisted rewriting, SAM 2 for segmentation mask generation, and human annotators for verification and refinement.
3. Vision Encoder (SigLIP 2 ViT) — a pre-trained vision transformer that processes individual frames (for video) or image crops (for images) into patch-level feature vectors. This encoder is frozen in architecture but fine-tuned during training.
4. Vision-Language Connector — a pooling and projection module that compresses ViT patch features into a smaller set of visual tokens (2×2 pooling for images, 3×3 pooling for video frames) using multi-headed attention, then projects them into the LLM's input space via an MLP. It also handles multi-crop tiling for high-resolution images and interleaves visual tokens with text timestamps and image indices.
5. Large Language Model (Qwen3 or OLMo 3) — a decoder-only transformer that receives the interleaved visual and text tokens and autoregressively generates text responses, including special HTML-like tags for point coordinates and track IDs. During training, it learns from the SFT mixture through next-token prediction with token-weighted loss.

Information flows as follows: raw videos/images enter the annotation pipelines to produce training data → during training, videos are sampled at 2 fps (or uniformly for long videos), each frame is encoded by the ViT, pooled by the connector, interleaved with text timestamps and input prompts, and fed to the LLM → the LLM generates outputs (captions, answers, point coordinates, tracks) → loss is computed only on the output tokens, with per-task weights to balance long-form and short-answer tasks.

3.3 Roadmap for the Deep Dive

• First, the model architecture (ViT, connector, LLM, cropping and frame sampling strategy) because all subsequent training and inference depend on how visual information is converted into tokens that the LLM can process.
• Second, the pointing and tracking representation format because this is the core technical interface that enables grounding—how coordinates, timestamps, object IDs, and counts are serialized as text tokens that the LLM can both learn to generate and use for downstream applications.
• Third, the three-stage training pipeline (pre-training, SFT, long-context SFT) because the ordering and mixing ratios between stages are critical to achieving stable performance across image, video, and grounding tasks simultaneously.
• Fourth, the data mixture and sampling strategy (the seven dataset groups, their sampling rates, within-group balancing, and the token weighting scheme) because the heterogeneity of the data—from 1-token multiple-choice answers to 4,000+ token video captions—requires careful balancing to avoid catastrophic interference.
• Fifth, the packing and message tree system because this is the efficiency innovation that makes training on such diverse data computationally feasible, achieving ~15× training efficiency over naive batching.
• Sixth, the training hyperparameters and implementation details (learning rates, warmup schedules, precision, parallelism strategies) because these practical choices interact with the data mixture to determine final model quality.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a data and training methodology paper whose core idea is that state-of-the-art video grounding can be achieved entirely through open data—without distilling from proprietary VLMs—by collecting large-scale, diverse, human-annotated datasets that specifically target spatio-temporal localization (pointing, counting, tracking) alongside traditional video understanding, and by developing training innovations that allow a single model to learn all these capabilities simultaneously without interference.

Architecture: ViT, Connector, and LLM

Molmo2 follows the standard multimodal LLM design pattern: a vision encoder converts pixels into feature vectors, a connector compresses and projects those features into the language model's embedding space, and an LLM generates text autoregressively. What distinguishes Molmo2 is the specific design choices for handling high-resolution images, variable-length videos, and the spatial-temporal coordinate format.

Vision Encoder: SigLIP 2 So400m/14 384px. All Molmo2 variants (4B, 7B, 8B) use the same frozen-architecture vision transformer: SigLIP 2 So400m/14 with 384×384 pixel input resolution. Its specifications are:

• Parameters: 380 million
• Hidden dimension: 1152
• MLP dimension: 4304
• Activation: GELU
• Attention heads: 16
• KV heads: 16
• Layers: 27
• Patch size: 14×14 pixels
• Dropout: 0.0

The ViT processes each 384×384 image crop independently, producing a grid of 27×27 = 729 patch embeddings (since 384 ÷ 14 ≈ 27.4, the image is divided into 27×27 patches with the edges adjusted). For video, each sampled frame is processed as a separate image, meaning the ViT does not see temporal information directly—temporal reasoning emerges from the LLM's ability to attend across frame tokens with timestamps.

Cropping Strategy for Images. High-resolution image understanding requires processing details that would be lost at a single 384×384 resolution. Molmo2 uses overlapping crop tiling:

• Training: A single downscaled square crop of the entire image PLUS up to K = 8 overlapping crops that tile the image at higher resolution. Images that cannot be fully covered by K crops are downscaled to fit.
• Inference: K = 24 crops are used to provide even finer detail.
• Unlike the original Molmo which padded crops with black to maintain aspect ratios, Molmo2 resizes crops to 378 pixels (changing their aspect ratio), following how SigLIP 2 was trained. If the number of image patches is not evenly divisible by the pooling size, the bottom and rightmost patches are pooled with fewer patches rather than being padded.

For multi-crop images, "column tokens" are inserted to indicate the image's aspect ratio, helping the model understand the spatial layout. Single-crop images are always square so they don't need column tokens.

Frame Sampling for Video. Video processing involves a careful tradeoff between temporal coverage and computational cost:

• Sampling rate: S = 2 frames per second. This is a deliberate choice that balances temporal resolution with token efficiency—2 fps captures most actions while keeping the total frame count manageable.
• Maximum frames: F = 128 during SFT training, increased to F = 384 during long-context post-training.
• Sampling strategy: If a video's duration × 2 fps ≤ F, all frames at 2 fps are included plus the last frame (since "most video players will display the last frame after the video finishes playing, and it therefore might have special importance to users"). If the video is too long, frames are sampled uniformly to hit exactly F frames, with the first and last frames always included.
• Implementation: Frames are extracted using torchcodec based on timestamps (not frame indices) to handle variable frame rate videos correctly.

Vision-Language Connector. The connector serves two functions: (1) pool the ViT's patch-level features into a smaller number of visual tokens to reduce the LLM's sequence length, and (2) project these pooled features into the LLM's embedding dimension.

• Feature source: The connector takes features from both the third-to-last and ninth-from-last ViT layers (not just the final layer), following Molmo [29]. This provides multi-scale visual features—deeper layers capture semantic information while shallower layers retain spatial detail important for precise grounding.
• Pooling via multi-headed attention:
  • For images: 2×2 patches are pooled into a single vector. A multi-headed attention layer takes the mean of the four patches as the query and the individual patch features as keys and values, producing a single pooled representation.
  • For video frames: a larger 3×3 patch window is used instead to reduce token count further, since video already generates many frames. The paper ablates this in Table 8b, finding that increasing to 4×4 pooling "slightly lowers QA performance but causes a significant drop in captioning quality"—the larger pooling discards fine-grained visual details needed for dense description.
• Heads and dimensions: The pooling attention uses 16 heads with input dimension 1152 (matching the ViT output).
• MLP projection: After pooling, a shared MLP projects features into the LLM's embedding dimension. The connector MLP uses the same intermediate dimension as its target LLM: 9,728 for the 4B model, 12,288 for the 8B model, and 10,0352 for the 7B (OLMo-based) model. The activation is SwiGLU. Crucially, the same connector parameters are shared for both image and video frame pooling.
• Total connector parameters: 57M for the 4B, 88M for the 8B, and 80M for the 7B.

LLM Integration. Visual tokens from the connector are inserted first, right after the BOS token, followed by any text input:

• For video: frames are interleaved with text timestamps written as seconds to one decimal point (e.g., "0.0", "0.5", "1.0").
• For multi-image: images are interleaved with "Image 1", "Image 2", etc., labels.
• For multi-crop images: column tokens indicate aspect ratio.
• Special tokens: Different start and end tokens distinguish video tokens, multi-crop image tokens, and single-crop image tokens, allowing the LLM to learn modality-specific processing.

Bidirectional Attention on Vision Tokens. A critical architectural choice: Molmo2 allows image tokens (even from different frames or images) to forward-attend to each other. In a standard causal LLM, each token can only attend to previous tokens in the sequence. Molmo2 relaxes this for vision tokens: a video frame token can attend to tokens from later frames (and vice versa), and tokens from different images can attend to each other. The text tokens remain causally masked. The paper finds this "can increase performance" (Table 8b)—removing bidirectional attention reduces QA performance from 64.8 to 64.4 and captioning F1 from 39.5 to 38.5. The intuition is that full bidirectional attention between visual tokens allows the model to build a holistic understanding of the visual scene before generating text, rather than processing frames sequentially.

LLM Choices. Three variants demonstrate generality:

• Molmo2-4B and Molmo2-8B: Based on Qwen3, with embedding dimension 2,560 (4B) or 4,096 (8B), 36 layers, 32 attention heads, 8 KV heads, and SwiGLU activation.
• Molmo2-O-7B: Based on OLMo 3, with 7.3B parameters, embedding dimension 4,096, 32 layers. This variant exists to demonstrate "what can be achieved with a fully-open language model" since OLMo 3 releases training data, while Qwen3 does not.

Pointing and Tracking Representation Format

A central technical contribution is how spatial and temporal coordinates are serialized as plain text that the LLM can both generate and parse. The format must satisfy several constraints: it must be token-efficient (since some tasks involve hundreds of points), unambiguous (distinguishing object identities across frames), human-readable (for interpretability), and consistent across pointing, counting, and tracking tasks.

The HTML-like <points> and <tracks> tags. Molmo2 uses custom XML-style tags to wrap coordinate sequences, with a compact attribute-based format for the data:

• <points> is used for static pointing and counting tasks (points at specific frames or images without continuous tracking).
• <tracks> is used for continuous object tracking across multiple video frames where the same object ID must be maintained.
• Both tags contain a coords attribute with a compact string encoding.

Coordinate string encoding. The coords attribute value is a space-separated sequence where each entry represents either a frame/image header or a point:

• Frame/image headers: a timestamp (for video, in seconds with one decimal, e.g., 0.0, 1.5) or an image index (for multi-image, starting from 1, e.g., 1, 2). These mark when/where the following points occur.
• Points within a header: each point is an object index (integer starting from 1, unique per distinct object) followed by x and y coordinates normalized to [0, 1000]. Points are sorted by time/image index, then by x, then y.

Object indices and their semantics. Object indices serve two critical functions:

1. Counting: The maximum object index across all points equals the total count of distinct objects. The model learns this correspondence—when asked "How many X are there?", it generates points and the final index in the sequence is the answer.
2. Tracking: The same object index repeated across multiple timestamps indicates that it is the same object being tracked through time. Different objects receive different indices.

For example, if a video has two waterfalls at frame 0.0 seconds and one at frame 3.5 seconds:

<points coords="0.0 1 555 169 2 649 154;3.5 3 758 175">waterfall</points>

This encodes: at timestamp 0.0, object 1 at (555, 169) and object 2 at (649, 154); at timestamp 3.5, object 3 at (758, 175). The maximum index is 3, so the answer is 3 waterfalls.

Why this format instead of JSON? The paper states explicitly: "We elect to use this format over a format like JSON since it dramatically reduces the number of tokens needed to represent points." JSON would require key-value pairs for each field (e.g., {"timestamp": 0.0, "object_id": 1, "x": 555, "y": 169}), each of which costs multiple tokens. The compact format uses only numeric tokens and minimal separators.

Auxiliary tracking outputs. During training for tracking, Molmo2 also learns auxiliary tasks:

• Temporal grounding: predicting only the first and last frames where objects appear, without generating intermediate points. This teaches the model about temporal extent.
• Single-point tracking: given an initial point and query, generating the track for just that object. This handles cases where only one object is specified.

Point sorting invariance. Points are always output sorted (by time, then x, then y), which makes the format deterministic—the model doesn't need to learn an arbitrary ordering. This also makes parsing and evaluation straightforward.

Three-Stage Training Pipeline

Molmo2 uses a staged training approach where each stage builds capabilities incrementally and the data mixture shifts to target specific skills.

Stage 1: Pre-training (image-only). The goal is to teach the model basic vision-language alignment and pointing before introducing the complexity of video:

• Data mixture: 60% image captioning (PixMo-Cap with length conditioning and transcript prediction), 30% image pointing (PixMo-Points, PixMo-Count, CoSyn-Point), 10% NLP (Tulu SFT data, filtered to remove non-English and code).
• Length: 32,000 steps at batch size 128, max sequence length 2,560 tokens.
• Why pointing in pre-training? The paper ablates this in Table 18: removing pointing pre-training and compensating by 2× sampling during SFT leads to small but consistent drops across all benchmarks, including non-pointing tasks. The hypothesis: "pointing pre-training simplifies the SFT stage for the model since it no longer needs to learn the basic pointing format and task, allowing for more focus on the non-pointing tasks." Essentially, learning to generate coordinates is a distinct skill that, if front-loaded, doesn't compete with other capabilities during fine-tuning.
• Special pre-training techniques: response-only dropout (0.1 on output tokens only, not input tokens), length conditioning for captions, and both caption and transcript prediction (following Molmo [29]).
• Learning rate scheduling: The connector gets a high learning rate (2e-4) with short warmup (200 steps) while the ViT and LLM use lower rates (6e-6 and 2e-4 respectively) with longer warmup (2,000 steps). This ensures the first steps mostly train the connector, establishing good vision-language alignment before the ViT or LLM change significantly. All rates use cosine decay to 10% of peak.

Stage 2: Joint Supervised Fine-Tuning (SFT). This is the main training stage where the model learns all video capabilities:

• Data mixture: The full Molmo2 mixture plus open-source datasets, organized into seven categories (Table 1): Image QA (22.7% sampling rate, 32 datasets), Video QA (18.2%, 32 datasets), Captions/Long QA (13.6%, 6 datasets), Video Pointing (13.6%, 7 datasets), Video Tracking (13.6%, 22 datasets), Image Pointing (9.1%, 4 datasets), NLP (9.1%, 1 dataset).
• Length: 30,000 steps at batch size 128, max sequence length 16,384 tokens.
• Regular dropout: 0.1 residual dropout (applied to all layers, not just output).
• Mixing within batches: Examples from different dataset categories are mixed within each batch so the model sees diverse tasks in each gradient step.

Stage 3: Long-Context SFT. After standard SFT, the model undergoes a short adaptation phase on longer sequences:

• Data: The same SFT mixture, but with sequence length increased to 36,864 tokens and max frames F increased from 128 to 384.
• Length: only 2,000 steps—the paper describes this as "a short final training stage since it adds significant overhead to the training."
• Context parallelism: The LLM is sharded across groups of 8 GPUs using Ulysses attention, where each GPU processes a portion of the attention heads. The paper chose Ulysses over other parallelism methods because its all-gather communication pattern offers "flexibility with the custom attention masks used by our packing and message tree system." Video frame processing by the ViT and connector is also distributed across the CP group to reduce memory.
• Effect: Table 11 shows long-context SFT "significantly improves model performance on long video QA benchmarks" (from 64.4 to 67.4 average) but "the video caption performance drops" (from 42.3 to 39.9 F1). Short video QA and image QA are essentially unchanged. The drop in captioning is likely because the model must now handle much longer sequences, distributing its capacity differently.

Why three stages instead of end-to-end? Staged training allows the model to first learn basic vision-language alignment and pointing format (Stage 1) before tackling the full complexity of video tasks (Stage 2). Long-context adaptation is kept separate (Stage 3) because the computational cost scales with sequence length—training at 36k tokens for all of SFT would be prohibitively expensive. The paper's approach is to train the model to be competent first, then briefly adapt it to longer contexts.

Data Mixture and Sampling Strategy

Managing the heterogeneity of training data—from single-token multiple choice answers to 4,000+ token dense video captions—is a central challenge. The paper's solution involves manual category-level sampling rates, within-category proportional sampling, and per-example token weighting.

Seven dataset categories and their sampling rates (Table 1, Figure 4):

• Image QA (22.7%): Multiple-choice and short-answer questions on images. Includes PixMo-Clocks, VQA v2, OK-VQA, A-OKVQA, DocVQA, ChartQA, TextVQA, AI2D, ScienceQA, CoSyn-chart/doc/table/diagram/math/music/chemical, NLVR2, TallyQA, FigureQA, PlotQA, DVQA, ST-VQA, InfographicVQA, and others. This large allocation maintains strong image understanding alongside video capabilities.
• Video QA (18.2%): Similar format but for videos. Includes Molmo2-CapQA, Molmo2-SubtitleQA, TVQA, TGIF, ActivityNet, NExT-QA, PerceptionTest, MotionBench, Ego4D, and many others. This category is downsampled because "video-benchmarks converge quickly"—the model saturates on existing video QA benchmarks with relatively little data.
• Captions/Long QA (13.6%): Dense captioning and long-form Q&A. Includes Molmo2-Cap, PixMo-Cap, PixMo-CapQA, PixMo-AskModelAnything, Molmo2-AskModelAnything, Molmo2-MultiImageQA. This provides the detailed visual description capability.
• Video Pointing (13.6%): Spatial-temporal pointing and counting. Includes Molmo2-VideoPoint (the largest component) and academic pointing datasets. This category is upsampled "since this task is slow to converge"—grounding requires more training examples than classification-style tasks.
• Video Tracking (13.6%): Continuous object tracking. Includes Molmo2-VideoTrack and 14 academic tracking datasets. Re-weighted "to emphasize tail concepts"—rare objects and complex queries get higher effective sampling rates.
• Image Pointing (9.1%): Static image grounding. Includes PixMo-Points (weighted to emphasize high counts), PixMo-Count, CoSyn-Point, and Molmo2-MultiImagePoint. Downsampled relative to video pointing because it was already seen during pre-training.
• NLP (9.1%): Text-only SFT data from Tulu to preserve language understanding capabilities.

Within-category sampling. Sampling is proportional to the square root of each dataset's size, with manual rebalancing. The square root rule means a dataset with 4× more examples gets sampled only 2× more often, preventing large synthetic datasets from dominating small human-annotated ones. Manual adjustments include "downsampling large synthetic datasets"—for example, Molmo2-CapQA (950k QA pairs) would overwhelm the mixture if sampled purely proportionally.

Training example count vs. annotation count (Table 13). The paper distinguishes between "annotations" (raw question-answer or caption pairs) and "training examples" (what actually gets passed to the model after formatting into message trees). For example, Molmo2-Cap has 104k videos with 280k annotations, but after formatting into message trees with multiple branches per video, the effective training examples are only 100k. This is because message tree encoding packs multiple annotations for the same visual input into a single sequence, reducing redundancy.

Token Weighting. This is a critical innovation for preventing long-output tasks from dominating the loss:

• Problem: "These long-output examples can easily become the large majority of loss tokens even if they are sampled rarely, which can cause degradation on short-answer or multiple-choice tasks." A 4,000-token video caption contributes 4,000× more to the loss than a single-token multiple-choice answer, even if both examples appear once per epoch.
• Fixed weights: Video captions get weight 0.1, pointing tasks get weight 0.2. These are the longest-output tasks, so hard-weighting prevents them from dominating.
• Adaptive weight heuristic: For all other tasks, the weight is $\sqrt[4]{n}$ where $n$ is the number of answer tokens. This is the fourth root, not the square root—it compresses the range aggressively. For a 1-token answer, the weight is $\sqrt[4]{1} = 1.0$; for a 100-token answer, $\sqrt[4]{100} \approx 3.16$. Without weighting, the 100-token answer would contribute literally 100× more to the loss. With weighting, it contributes only ~3.16× more, preserving the signal from short-answer tasks.

The paper ablates token weighting in Table 8b: removing it improves captioning F1 (from 39.5 to 40.0) but degrades QA (64.8 to 64.0), confirming the tradeoff—token weighting sacrifices some caption quality to maintain QA performance.

Gradient averaging correction. The paper mentions a subtle implementation detail about gradient computation:

"We always compute the per-device gradient by dividing the total loss on that device by the average number of loss tokens across all devices, not the number of loss tokens on that particular device."

Without this correction, examples with few output tokens receive effectively larger gradient updates per token, which would bias training toward short-answer tasks. Dividing by the global average ensures that the gradient magnitude is consistent regardless of whether a device happens to be processing mostly short or mostly long examples.

Packing and Message Tree Encoding

These two innovations are the efficiency backbone that makes training on heterogeneous data feasible. Without them, training batches would be dominated by padding tokens and redundant visual encoding.

Message Trees. A message tree encodes a single visual input with multiple annotations as a branching structure:

• The visual input (video frames or images) is the root message.
• Each annotation (a caption, a QA pair, a pointing instruction) becomes a branch extending from the root.
• Different branches share the same visual encoding (the root), avoiding redundant computation.
• Linear representation: The tree is linearized into a single sequence. A custom attention mask prevents branches from cross-attending to each other—Question 1 should not attend to Answer 2 or vice versa. This is illustrated in Figure 3: within a packed sequence of two examples, the attention mask blocks cross-attention between different examples and between distinct QA pairs within the same example.
• Statistics: "On average, examples in our data have 4 annotations, and packing is able to fit 3.8 examples into a 16384 token sequence during SFT, leading to 15× training efficiency."

Packing Algorithm. Packing combines multiple short examples into one long sequence to minimize padding:

• Pool-based approach: A pool of M = 48 preprocessed and tokenized examples is maintained. When the pool is full, a dynamic programming solver finds the optimal subset that maximizes $T + I \times w_i$ subject to $T \leq 16384$ and $I \leq 128$, where $T$ is the total number of text tokens, $I$ is the total number of image crops, and $w_i = 30$ is a hyperparameter that balances text and image packing.
• Why not greedy? Dynamic programming guarantees the pack is as token-efficient as possible given the pool. The paper notes that "increasing M quickly leads to diminishing returns" and larger pools don't meaningfully improve packing density.
• Quantized solver: The problem is simplified by rounding token counts to the nearest multiple of 32.
• Robustness: The algorithm is "usually robust to $w_i$," but if $w_i$ is too low, the pool fills with image-heavy examples (128 crops each) that can't pack with anything else, reducing efficiency.
• Implementation: The packing logic is added to PyTorch's DataLoader so each data worker runs the solver independently. The paper notes this "adds some unnecessary overhead when there are many data workers" but in practice doesn't slow training because "loading and extracting frames from videos remains, by far, the most costly part of data loading."
• During long-context SFT: The max increases to 36,864 tokens and 384 images.

Efficiency gain: The 15× figure means that for the same GPU memory and compute, 15 times more training examples are processed per step compared to naive batching with padding. This enables training on the full data mixture in reasonable wall-clock time.

Training Hyperparameters and Implementation

Optimizer: AdamW. All stages use the AdamW optimizer with $\beta_1 = 0.9$, $\beta_2 = 0.95$, $\epsilon = 1 \times 10^{-6}$, and no weight decay.

Learning rates (Table 12):

The pattern is deliberate: during pre-training, the connector gets 100× higher LR than the ViT so the first steps primarily train the connector. During SFT, all components use similar low rates (5e-6 to 1e-5) for stable fine-tuning. All rates decay via cosine schedule to 10% of peak.

Batch size and sequence length: 128 per batch for all stages, with sequence lengths of 2,560 (pre-training), 16,384 (SFT), and 36,864 (long-context SFT).

Precision: PyTorch Automatic Mixed Precision (AMP) with bfloat16 for most operations. Layer normalization and Rotary Position Embedding (RoPE) are computed in full precision for numerical stability.

Parallelism: Fully Sharded Data Parallel (FSDP) 2 for distributed training. During long-context SFT, Ulysses context parallelism splits the LLM across groups of 8 GPUs.

Compilation: torch.compile is used with static shapes for both the LLM and ViT to maximize throughput. The paper is explicit: "shapes in the LLM and ViT are static so the model can be statically compiled, which we find essential for maximizing throughput."

Attention implementation: The paper uses PyTorch's Scaled Dot Product Attention (SDPA) rather than FlashAttention because FlashAttention "does not support custom attention masks" needed for message trees and packing.

Training stability: "We find training to be stable, without loss spikes or NaNs."

Truncation: "We truncate examples that are longer than the max sequence length. This occurs in < 0.1% of cases, usually due to videos with both subtitles and a large number of annotations."

Training time (Table 14):

All training uses Nvidia H100 GPUs. Notably, "during SFT, a high portion of the computation is from the ViT because, for videos, 9 patches in the ViT are processed for each visual token in the LLM"—the 3×3 spatial pooling means 9 ViT patches compress to 1 LLM token, but those 9 patches all need to be encoded first. This means increasing LLM size has a reduced effect on total training time.

Prompting and Style Tags

To handle the diverse output formats across datasets:

• Human-written questions: Used directly from PixMo-AskModelAnything, PixMo-CapQA, and Molmo2-AskModelAnything.
• Captioning, multiple-choice, and grounding: Template-based prompts generate varied ways to request the same output format, preventing the model from overfitting to specific phrasings.
• Academic datasets with unnatural answer styles: Many academic benchmarks have "idiosyncratic quirks due to how the data was collected"—terse answers, specific formatting requirements. For these, Molmo2 uses "style tags" like short_video_answer: prepended to the prompt. These tags signal that the model should adopt the dataset's expected answer style only when the tag is present, maintaining natural behavior for user-facing interactions.

During inference, greedy decoding is used for most benchmarks. For human evaluation and video captioning, the paper uses top_p = 0.95, temperature = 0.7, and frequency_penalty = 0.1, which "produces more natural results when generating long outputs."

4. Key Insights and Innovations

Innovation 1: Grounding as an Independent Axis of Model Capability, Not a Downstream Afterthought

The most intellectually distinctive move in this paper is the reframing of grounding—spatio-temporal pointing, counting, and tracking—from a specialized add-on capability into a first-class dimension of visual intelligence that is trainable alongside general understanding in a single model. Before Molmo2, the dominant architecture in the field was to treat video understanding and video grounding as separate problems requiring separate models: general-purpose VLMs like Qwen3-VL, InternVL3.5, or GPT-5 handled captioning and QA, while specialized segmentation models like VideoLISA [11], VideoGLaMM [121], Sa2VA [177], and SAM 3 [16] handled pixel-level localization. Even when a single model attempted both (e.g., Gemini 2.5 Pro), grounding was an afterthought—the paper shows these models "fail to generate accurate points" (Tables 3-5) and must be prompted to output bounding boxes whose centers are then extracted for evaluation.

Molmo2's conceptual breakthrough is demonstrating that grounding is not a separate task but a representation format. By encoding spatial and temporal coordinates as plain-text tokens in an HTML-like markup (<points coords="0.0 1 555 169">waterfall</points>), the paper makes pointing, counting, and tracking indistinguishable from any other text generation task at the architectural level. This means the same LLM that generates a dense video caption can generate point coordinates—no separate segmentation head, no auxiliary loss, no post-processing pipeline. The implications are deeper than architectural convenience:

• Grounding and language understanding reinforce each other. The paper's ablation results support this: adding pointing data to pre-training improves non-pointing benchmarks (Table 18), and training on video QA data improves captioning and vice versa (Tables 8a, 8c). The model learns that "pointing to the waterfalls" and "describing the waterfalls" are complementary ways of expressing the same visual understanding, and the shared representation benefits both.
• Grounding enables counting through identity, not enumeration. Rather than learning counting as a separate skill, Molmo2 learns that the maximum object index in a pointing sequence equals the count. This elegantly ties counting accuracy to the model's ability to identify distinct object instances—a grounded form of numeracy that does not rely on memorizing number patterns from text.
• Grounding scales with data diversity, not model size. The paper demonstrates that even a relatively small model (4B parameters) achieves state-of-the-art video pointing and tracking by training on 520k human-annotated pointing instances and 800k tracking examples. This is a fundamentally different scaling story than the "bigger model, more pre-training" paradigm that dominates language-only capabilities. On grounding tasks, data specificity—not parameter count—is the binding constraint.

What makes this innovative rather than merely engineering is that it challenges the implicit assumption that visual localization requires specialized architectures with explicit spatial reasoning modules. The paper shows that a generic autoregressive language model, given sufficient diverse grounding data with a sensible text-based coordinate format, learns spatial reasoning as an emergent property of next-token prediction. This is an existence proof with significant implications for the design space of future multimodal models.

The evidence is compelling: Molmo2-8B achieves 38.4 F1 on video pointing (Table 3) compared to Gemini 3 Pro's 20.0 F1—nearly 2× better—and 56.2 J&F on video tracking (Table 5) compared to Gemini 3 Pro's 44.6. These are not incremental improvements; they represent a qualitative leap in a capability that proprietary models, despite massive compute and parameter budgets, have not developed. The paper's counterfactual is clear: if grounding were merely a matter of scale, Gemini would dominate. The fact that it doesn't implies that targeted, diverse, human-annotated grounding data is the bottleneck, not model capacity.

Innovation 2: Breaking the Distillation Dependency—Proof That Open-Data Training Can Match or Exceed Proprietary-Distilled Models

This is both a methodological contribution and a strategic intervention in how the field develops open models. The dominant assumption in video-language model training—codified by LLaVA-Video [184], Eagle 2.5 [17], ShareGPT4Video [19], and PerceptionLM [22]—has been that generating high-quality video training data requires a strong proprietary VLM (like GPT-4V or Gemini) to produce initial clip captions, which an LLM then synthesizes into video-level descriptions and QA pairs. This creates a closed loop: to build an open model, you must first have access to a closed model, and your open model's quality is fundamentally bounded by the closed model you distilled from.

Molmo2 demonstrates that this dependency is not necessary. By investing in large-scale human annotation pipelines—spoken video descriptions for dense captions, human-and-LLM collaboration for free-form QA, human point-clicking for spatio-temporal localization—the paper produces training data that is, in key dimensions, better than what proprietary distillation produces. The evidence is in the caption length statistics: Molmo2-Cap averages 924 words per video, compared to 547 in LLaVA-Video-178K (GPT-generated), 280 in ShareGPT4-Video, and 89-100 in earlier human-annotated datasets like Video Localized Narratives. This is not just a "longer is better" claim—the paper's captioning evaluation uses an LLM-as-judge to measure factual precision and recall against human reference captions, and the performance in Table 2 shows Molmo2-8B achieving 43.2 caption F1, far ahead of PerceptionLM-8B (10.9) and LLaVA-Video-7B (19.9). The open-data captions are both longer and more accurate.

The intellectual significance goes beyond Molmo2's specific numbers. This paper provides the first large-scale evidence that proprietary model distillation is not on the Pareto frontier—there exist data collection strategies that produce better training signals without the biases, ceiling effects, and reproducibility problems of closed-model dependency. This is a strategic reframing: the field should view proprietary distillation not as a necessary evil but as a local optimum that the community is stuck in due to insufficient investment in open annotation infrastructure.

The paper is careful not to claim that human annotation is always superior—Molmo2 still uses closed text-only LLMs for data generation and rewriting, and the synthetic CapQA and SubtitleQA datasets (1.3M examples) use an LLM to generate questions from captions. The key distinction is that the visual understanding component—the captions, the pointing coordinates, the tracking labels—comes from humans or from models trained on human data, not from proprietary VLMs. This means the visual knowledge in Molmo2 is bootstrapped from human perception, creating a foundation that can be iteratively improved without dependency on any particular company's API.

The practical implication is significant: Molmo2 shows that a fully open pipeline can achieve competitive or superior performance not just against other open models, but against proprietary APIs on specific grounded capabilities (Tables 3-5: outperforming Gemini 3 Pro on video pointing and tracking). This inverts the typical narrative of open models "catching up" and instead suggests that on capabilities requiring fine-grained spatio-temporal reasoning, open data might actually have a structural advantage—human annotators can be explicitly instructed to capture exactly the diversity and complexity needed, while proprietary models' training data and biases remain unknown and uncontrollable.

Innovation 3: Message Trees and Token Weighting as a Diagnostic Framework for Multi-Task Interference

While message-tree encoding and token weighting appear at first glance to be training efficiency techniques (and they are effective as such—15× throughput improvement), their deeper intellectual contribution is that they make visible and address a fundamental problem in multimodal training: the interference between tasks with radically different output lengths and formats. Before this paper, the standard approach to multi-task VLM training was to naively mix datasets and accept that some tasks would dominate the loss due to their output token count. The consequences—degradation on short-answer tasks, instability on pointing tasks with variable-length outputs—were observed but not systematically diagnosed.

Molmo2's solution is not one technique but a coherent diagnostic and mitigation framework with two components:

1. Message trees reveal the cost of redundant visual encoding. The key insight in message trees is that different annotations for the same visual input share a visual encoding. When the model generates a 4,000-token caption for a video and then separately generates a 1-token multiple-choice answer for the same video, the visual features are computed once—but in a naive training setup, they're recomputed for each annotation. Message trees make this sharing explicit in the attention mask (Figure 3), and the efficiency gains (3.8 examples packed into 16k tokens) quantify how much wasted computation the naive approach incurs. More subtly, message trees prevent the model from learning spurious correlations between different annotations—the attention mask ensures that the caption branch cannot attend to the QA branch and vice versa, which would be a form of training data leakage if annotations were simply concatenated.

2. Token weighting as a principled solution to output-length imbalance. The paper's token-weighting scheme is more sophisticated than it first appears. Using $\sqrt[4]{n}$ (the fourth root of output token count) rather than a linear or square-root weighting reflects an empirical finding about how loss contributions scale with sequence length. The ablation in Table 8b confirms the tradeoff: removing token weighting improves captioning F1 (40.0 vs. 39.5) but degrades QA performance (64.0 vs. 64.8). This is not just "weight long outputs less"—it's a specific claim about the elasticity of different tasks to loss contribution. Captioning, as a dense-output task, benefits from more loss signal per example; short-answer QA saturates quickly and needs protection from being drowned out. The fourth-root heuristic encodes this asymmetry: long outputs still contribute more to the loss than short ones, but the ratio is compressed aggressively enough to preserve the short-answer signal.

The fundamental advance here is not the specific weighting formula but the recognition that multi-task VLM training requires explicit per-task loss modulation, and that without it, the Pareto frontier of joint task performance is unattainable. Prior work either ignored this (training on uniformly mixed data and accepting whichever tasks dominated) or addressed it through ad-hoc dataset balancing (adjusting sampling rates). Molmo2 shows that sampling rates alone are insufficient—a single 4,000-token caption contributes as much to the loss as 4,000 single-token answers, regardless of how they're sampled. Token weighting and sampling rates are complementary controls operating at different granularities: sampling rates determine which tasks the model sees, while token weighting determines how much each example of that task contributes once sampled.

This framework is generalizable: any multi-task VLM training setup with heterogeneous output lengths can adopt the principle of explicit loss modulation, even if the specific $\sqrt[4]{n}$ formula needs recalibration for different model scales or data mixtures. The paper has given the field a language for discussing this problem—"token weighting" as distinct from "data sampling"—that was previously absent.

Innovation 4: Verifying That Video Grounding Bottlenecks on Data Diversity and Task Formulation, Not Model Architecture

This is a negative result with strong positive implications: the paper systematically demonstrates that the primary bottleneck in video pointing, counting, and tracking is not model capacity, not vision encoder quality, not even the choice of LLM—it is the diversity and formulation of the training data itself. This is a fundamental claim about where the field should invest effort.

The evidence comes from multiple converging sources:

• Specialized models don't dominate. Tables 4-5 show that specialized segmentation models like VideoLISA, VideoGLaMM, Sa2VA, and SAM 3—models designed specifically for tracking and segmentation—are outperformed or matched by Molmo2, a general-purpose VLM, on tracking benchmarks. Sa2VA-8B, a specialized model, achieves 46.9 J&F on the Molmo2-Track benchmark versus Molmo2-8B's 56.2. This is counterintuitive: a general model with no segmentation-specific architecture beats models whose sole purpose is segmentation. The explanation must be in the data.
• Data source ablation shows clear composition effects. Table 9b demonstrates that Molmo2-VideoPoint and AcademicVideoPoint are complementary: the model trained on both achieves 35.0 F1 on Molmo2-VP, versus 34.3 with Molmo2 data alone and 9.0 with academic data alone. The academic data contributes coverage of rare objects and scenarios, while the Molmo2 data contributes linguistic diversity and real-world query complexity. Neither alone is sufficient.
• Upsampling high-count examples improves both counting and pointing (Table 9c). This is a direct causal demonstration that data distribution shapes capability: by upsampling examples with 10+ points, the model's accuracy on medium-high count ranges improves (Burst-VC close accuracy drops slightly from 62.4 to 61.5 with upsampling, but Molmo2-VC accuracy jumps from 32.1 to 34.5 and Molmo2-VP F1 from 28.1 to 31.8). The model learns to handle density from seeing density.
• Counting via pointing is dramatically better than direct counting (Table 9a). The model trained to "point then count" achieves 61.5 BVC accuracy versus 61.3 for direct counting (a small gain), but 34.5 MVC accuracy versus 28.1 (a 23% relative improvement). This is a task formulation insight: counting is not a distinct skill but a byproduct of the model's ability to individuate and point to distinct instances. Formulating counting as a pointing task—where the answer is implicit in the maximum object index—changes what the model learns.

The deepest implication is about the nature of video grounding as a capability. Unlike language understanding, which benefits from massive text pre-training, or image recognition, which benefits from massive image pre-training, video grounding appears to have no equivalent "free" pre-training signal. The skills required—individuating objects across frames, re-identifying them after occlusion, understanding complex referring expressions in a temporal context—are not naturally present in internet-scale image-text or video-text data. They must be explicitly taught through carefully constructed training examples that cover the specific linguistic and visual combinations the model will encounter.

This finding redirects research investment: improving video grounding is primarily a data collection and task design problem, not an architecture problem. A better vision encoder or a larger LLM will not help if the training data doesn't teach the model to track objects through occlusion or to count 50+ instances in a single frame. The paper's contribution is making this bottleneck visible and providing a concrete recipe—diverse human pointing annotations, upsampling of high-count and complex-query examples, and the pointing-as-counting task formulation—for addressing it.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper evaluates Molmo2 across a broad spectrum: 15 video understanding benchmarks (Table 2) spanning short-video (NextQA, PerceptionTest, MVBench, Tomato, MotionBench, TempCompass) and long-video (Video-MME, Video-MME-Sub, LongVideoBench, MLVU, LVBench, VideoEvalPro, EgoSchema) tasks; custom evaluation sets for dense video captioning (Molmo2-CapTest, 693 Creative Commons-licensed videos with 4+ human captions each), video counting (Molmo2-VideoCount, 533 examples with up to 60 points per query), and video pointing (Molmo2-VideoPointVal, 181 examples with SAM 2-validated ground-truth masks); 5 referring video object segmentation benchmarks for tracking (BURST, MeViS, Ref-YT-VOS, Ref-Davis, ReasonVOS) plus a custom Molmo2-Track benchmark (1,386 clips across 5 video domains: animals, person, sports, dancers, misc); 14 image/multi-image benchmarks (Table 6) including AI2D, ChartQA, DocVQA, InfoQA, TextVQA, VQA v2.0, RealWorldQA, MMMU, MathVista, CountBench, PixMoCount, MuirBench, MMIU, Blink; and Point-Bench for image pointing evaluation (Table 7). Additionally, a large-scale human preference study collected 450 open-ended questions plus 51 captioning queries, yielding over 105K pairwise ratings from annotators, from which Elo scores were computed using the Bradley-Terry model.

• Base models. Three Molmo2 variants are trained: a 4B and 8B model based on Qwen3 LLMs, and a 7B model based on OLMo 3 (Molmo2-O-7B), with all variants sharing the same SigLIP 2 So400m/14 384px vision encoder. The Qwen3 variants use embedding dimensions 2,560 (4B) and 4,096 (8B) with 36 layers, 32 attention heads, 8 KV heads, while the OLMo variant has 7.3B parameters with embedding dimension 4,096 and 32 layers, chosen specifically to demonstrate what can be achieved with a fully-open language model. During inference, 384 frames are sampled at 2 fps with greedy decoding for benchmarks; for human evaluations and captioning, sampling uses top_p=0.95, temperature=0.7, and frequency_penalty=0.1.

• Metrics. Video understanding benchmarks use standard dataset-specific accuracy metrics; video captioning is evaluated via an F1 score derived from LLM-as-judge precision and recall of atomic statements against human reference captions (harmonic mean of precision and recall); video counting uses exact accuracy and "close accuracy" (correct if |pred − gt| ≤ ∆ where ∆ = 1 + ⌊0.05 × gt⌋, rewarding approximate correctness on high-count examples); video pointing uses F1, recall, and precision computed by checking whether predicted points fall within SAM 2-generated ground-truth segmentation masks within a 3-second window around annotated timestamps; video tracking uses J&F (Jaccard and boundary F-measure for segmentation quality) and point F1 (1 fps, correct if point falls inside ground-truth mask), with HOTA (√(DetA × AssA)) adapted for point-based tracking by defining binary similarity (point is in mask or not); image benchmarks use standard accuracy; image pointing uses Point-Bench metrics; human evaluation uses bootstrapped Elo ratings with 1,000 rounds.

• Baselines. The paper compares against three tiers: (1) proprietary API models—GPT-5, GPT-5 mini, Gemini 3 Pro, Gemini 2.5 Pro, Gemini 2.5 Flash, Claude Sonnet 4.5; (2) open-weight models—InternVL3.5-4B and -8B, Qwen3-VL-4B and -8B, Keye-VL-1.5-8B, GLM-4.1V-9B, MiniCPM-V-4.5-8B, Eagle2.5-8B; (3) fully-open models—PerceptionLM-3B and -8B, LLaVA-Video-7B, VideoChat-Flash-7B; plus specialized open models for tracking—VideoLISA, VideoGLaMM, Sa2VA-8B, Sa2VA-Qwen3-VL-4B, SAM 3, and VideoMolmo-7B. For pointing and tracking evaluations, API models are prompted to output bounding boxes (since they "fail to generate accurate points"), with box centers extracted as proxy points and SAM 2 used to convert boxes to segmentation masks.

• Generation budget / compute accounting. Since Molmo2 is a model family evaluated at inference time rather than a method that allocates variable compute, there is no per-question generation budget in the sense of best-of-N or beam search. All evaluations use a single forward pass with fixed settings (384 frames, greedy or sampling-based decoding). The test-time scaling experiments in Appendix E explore varying the number of frames (up to 512) and SlowFast encoding to keep vision tokens fixed at ~10.6k while increasing temporal coverage, with the main compute metric being maximum vision tokens processed by the LLM.

• Cross-validation / statistical protocol. Difficulty estimation and strategy selection are not the focus here (unlike the reference example's compute-optimal framework). However, the human evaluation uses 1,000 bootstrap rounds to estimate Elo confidence intervals, and video captioning uses multiple human reference captions per video with LLM-as-judge matching. Primary benchmark results are single-run evaluation on fixed test sets, with validation set results used for ablations (e.g., EgoSchema, PerceptionTest, MLVU validation sets for video; InfoQA, DocQA, ChartQA, VQA v2, AI2D validation sets for images).

Main Quantitative Results

Video Understanding, Captioning, and Counting (Table 2)

Headline numbers. Molmo2-8B achieves the highest average among fully-open models (where "average" represents the mean across all video QA, captioning, and counting benchmarks shown in Table 2). On short-video benchmarks specifically, Molmo2-8B achieves 69.9 average (NextQA, PerceptionTest, MVBench, Tomato, MotionBench, TempCompass), outperforming all open-weight models except Qwen3-VL-8B (65.3) and approaching proprietary models like GPT-5 (73.1). On video captioning, Molmo2-8B reaches 43.2 F1—the highest among all models except GPT-5 mini (56.6), substantially ahead of the next fully-open model (PerceptionLM-8B at 10.9) and all other open-weight models (Qwen3-VL-8B at 26.7, Eagle2.5-8B at 22.8). On video counting (Molmo2-VideoCount close accuracy), Molmo2-8B achieves 35.5, second only to GPT-5 (35.8) and Gemini 3 Pro (37.1) among all models and the best among open models by a large margin (next best: Eagle2.5-8B at 28.9).

Key comparisons at equal or similar scale. At the 8B scale, Molmo2-8B (69.9 short QA avg, 43.2 caption F1, 35.5 count accuracy) versus Qwen3-VL-8B (65.3, 26.7, 29.6): Molmo2 leads on all three, with particularly dramatic gaps on captioning (16.5 points) and counting (5.9 points). Versus InternVL3.5-8B (63.0 short QA avg, 7.8 caption F1, 26.1 count accuracy): Molmo2 leads by 6.9, 35.4, and 9.4 points respectively. At the 4B scale, Molmo2-4B (69.3 short QA avg, 39.9 caption F1, 34.3 count accuracy) versus Qwen3-VL-4B (63.7, 25.2, 25.3): again large leads across all three. Even against the 9B GLM-4.1V-9B, Molmo2-8B leads on captioning (43.2 vs. 18.4) and counting (35.5 vs. 26.6) while trailing slightly on short QA (69.9 vs. 64.2—but note Molmo2's average is 5.7 points higher).

Long video understanding gap. Molmo2-8B's long-video QA average is 64.1, behind Eagle2.5-8B (65.2) and the proprietary models (Gemini 2.5 Pro at 80.4). The paper is explicit about this limitation: "Molmo2 outperforms previous fully-open models but lags behind the best open-weight models. We believe this is due to a lack of open-source long (10+ minutes) training data and computational limitations that made it challenging to run extensive ultra-long context training." The long-context SFT ablation (Table 11) confirms partial mitigation: long-context training improves long QA from 64.4 to 67.4 but degrades captioning from 42.3 to 39.9.

Human preference evaluation (Table 15, Figure 5). Molmo2-8B achieves an Elo score of 1057, ranking 5th overall behind only Gemini 2.5 Pro (1096), Gemini 2.5 Flash (1084), Gemini 3 Pro (1082), and GPT-5 mini (1076), but ahead of GPT-5 (1031) and Claude Sonnet 4.5 (1008). Among open models, Molmo2-8B ranks 1st (ahead of Qwen3-VL-8B at 1054 and Qwen3-VL-4B at 1048). Task-specific breakdowns reveal that Molmo2 excels on QA (1059 Elo, ranking 5th among all models and 1st among open models) but underperforms on captioning (1049 Elo, ranking 8th overall, behind Qwen3-VL-8B at 1105 and GLM-4.1V-9B at 1013). The paper attributes the captioning gap to "repetitive or non-sensical content at the end of the caption, which we believe is due to text-repetition issues when generating extremely long output." Pairwise win rates (Figure 6) show Molmo2-8B beats Qwen3-VL-8B with a 53% win rate and Molmo2-4B beats Qwen3-VL-4B with 51%.

Image and Multi-Image Understanding (Table 6)

Headline numbers. Molmo2-8B achieves 81.7 average on 11 image benchmarks, the highest among fully-open models and competitive with the best open-weight models (Qwen3-VL-8B: 81.2, InternVL3.5-8B: 78.2, Eagle2.5-8B: 81.2). On individual benchmarks: VQA v2.0 (87.0) and RealWorldQA (77.6) are state-of-the-art among all open models; counting benchmarks (CountBench: 93.7, PixMoCount: 88.5) are second only to Eagle2.5-8B (90.2, 90.2) among open models; OCR-heavy benchmarks (DocVQA: 93.2, InfoQA: 80.1) are competitive but slightly behind the best (GLM-4.1V-9B: 93.3, 80.3). On multi-image benchmarks, Molmo2-8B averages 56.4, behind GLM-4.1V-9B (67.4) and some open-weight models, with the paper noting: "GLM-4.1V-9B is notably ahead of all other models." The Molmo1 family (72B) achieves 81.2 image average overall—Molmo2-8B at 81.7 shows improvement despite being 9× smaller.

Reasoning benchmarks as a weakness. Molmo2-8B scores 53.0 on MMMU and 58.9 on MathVista, substantially behind GPT-5 (81.8, 82.7), Gemini 3 Pro (85.2, 89.1), and even some open-weight models (InternVL3.5-8B: 73.4, 78.4). The paper acknowledges this directly: "Molmo2 is behind on open-weight reasoning benchmarks (MathVista, MMMU), possibly due to the lack of multi-modal reasoning training data." This is a genuine limitation—the data mixture heavily emphasizes perception and grounding over mathematical or scientific reasoning.

Video Grounding: Pointing, Counting, and Tracking

Video counting results (Table 3, Tables 16-17). On BURST-VideoCount close accuracy, Molmo2-4B achieves 76.1 and Molmo2-8B achieves 75.0—the highest among all models including proprietary APIs, narrowly ahead of GPT-5 (73.7) and GPT-5 mini (73.0). Note that BURST close accuracy is reported separately from exact accuracy (61.5 and 60.8 for Molmo2, 43.1 and 44.0 for GPT-5 and Gemini 3 Pro). On Molmo2-VideoCount close accuracy, Molmo2-8B (35.5) is second to Gemini 3 Pro (37.1) and GPT-5 (35.8), with the next open-weight model (Qwen3-VL-8B) at 29.6. Category breakdowns (Table 16) reveal Molmo2-8B leads on Action/Event counting (50.0 vs. Gemini 3 Pro's 58.6 but ahead of GPT-5's 46.6) and performs competitively on Object and Animal counting. Critically, by count range (Table 17), Molmo2-8B achieves 12.3-7.0% accuracy on 25-60 range examples, versus Qwen3-VL-8B's 0.0%—showing that other models completely fail on high-count scenarios.

Video pointing results (Table 3). On Molmo2-VideoPointVal F1, Molmo2-4B achieves 39.9 and Molmo2-8B achieves 38.4—nearly 2× better than the strongest proprietary model (Gemini 3 Pro at 20.0 F1) and dramatically ahead of all open-weight models (Qwen3-VL-8B: 1.5, Qwen3-VL-4B: 0.0). The paper notes that for baseline models, "we carefully tune the prompts and try both point and bounding-box formats for our baseline models; however, we were unable to find a formulation that achieved very strong performance." This is perhaps the strongest single demonstration of Molmo2's unique grounded capabilities.

Video tracking results on academic benchmarks (Table 4). Across five benchmarks, Molmo2-8B achieves: MeViS valid-u F1 75.9, Ref-YT-VOS valid F1 78.7, Ref-Davis valid F1 81.3, ReasonVOS test F1 70.8—all substantially ahead of both proprietary API models (Gemini 2.5 Pro: 41.2, 44.5, 62.7, 50.2 respectively) and specialized segmentation models (Sa2VA-8B: 46.9 J&F on MeViS, 68.1 on Ref-Davis). On HOTA (tracking association accuracy), Molmo2-8B achieves 72.6 on MeViS, 77.3 on Ref-YT-VOS, and 68.6 on ReasonVOS—metrics that only Molmo2 can report because "baseline models do not output stable track IDs but only counts." The most directly comparable model, VideoMolmo-7B (which also outputs point tracks), scores 59.4 F1 on MeViS and 75.4 on Ref-Davis—Molmo2-8B leads by 16.5 and 5.9 points respectively.

Video tracking on Molmo2-Track benchmark (Table 5). Across five video domains, Molmo2-8B achieves 57.1 overall F1 and 57.5 overall HOTA, compared to Gemini 3 Pro (32.2 F1, 29.1 HOTA) and the strongest specialized model Sa2VA-8B (46.7 J&F, but no F1 or HOTA reported since it outputs masks, not points with IDs). By domain: Animals (82.0 F1), Person (47.9), Sports (53.3), Dancers (63.9), Misc (31.5). The paper observes an interesting pattern: specialized models like Molmo + SAM 2 achieve high J&F (54.2) but low F1 (14.0) on the overall benchmark, suggesting they "generate large, coarse masks covering entire people rather than precisely localizing individual objects"—inflating segmentation overlap while failing the precise pointing task.

Image Pointing (Table 7)

Headline numbers. On Point-Bench, Molmo2-8B achieves an average of 68.7, the highest score on the leaderboard (as of 12/15/25), surpassing dedicated pointing models like Poivre-7B (67.5) and general models like the original Molmo-72B (63.8). By category: Affordance (84.8), Spatial Reasoning (71.3), Steerability (71.5), Counting (44.5), Average (71.4). The counting subcategory (44.5) is notably stronger than Gemini-2.5-Pro (41.0) and Qwen2.5-VL-72B (46.5), but still substantially below human performance (86.3). The paper attributes the improvement over Molmo to "the improved vision encoder, pointing pre-training, and token-weighting."

Long-Context SFT Ablation (Table 11)

Comparing Molmo2-4B before and after long-context post-training: short video QA is essentially unchanged (69.6 → 69.4), long video QA improves significantly (64.4 → 67.4), video captioning degrades (42.3 → 39.9), and image QA is unchanged (80.5 → 80.6). The tradeoff is explicit—better long-video understanding at the cost of captioning detail—and the paper provides no solution for recovering the lost captioning quality.

Ablation Studies and Robustness Checks

Caption Specialization: Joint training with other video data improves video caption performance (Table 8a). A model trained only on video data achieves 64.8 QA average and 39.5 caption F1, while a model trained on Molmo2-Cap alone achieves only 35.8 caption F1. The presence of QA data during training transfers positively to captioning, suggesting that learning to answer detailed questions about videos improves the model's ability to generate detailed descriptions.

Bidirectional Attention: Removing bidirectional attention on vision tokens degrades performance (Table 8b). A model without bidirectional attention (i.e., all tokens causally masked) achieves 64.4 QA average (down from 64.8) and 38.5 caption F1 (down from 39.5). The improvement from bidirectional attention is modest but consistent across both metrics, validating the architectural choice.

Token Weighting: Removing token weighting improves captioning but degrades QA (Table 8b). A model without token weighting achieves 64.0 QA average (down from 64.8) but 40.0 caption F1 (up from 39.5). This is the expected tradeoff—without explicit loss weighting, long caption outputs dominate the loss and push the model toward better captioning at the expense of short-answer tasks. The paper chooses to retain token weighting to preserve QA performance.

Time Tokens: Removing frame timestamps degrades both QA and captioning (Table 8b). A model without interleaved text timestamps achieves 64.5 QA average (down from 64.8) and 37.4 caption F1 (down from 39.5), confirming that temporal information is important for both understanding tasks, with a larger impact on captioning (which requires temporal narrative structure).

Video Pool Size: Larger pooling degrades captioning quality (Table 8b). Increasing the video pool size from 3×3 to 4×4 reduces caption F1 from 39.5 to 37.0 while QA drops only slightly (64.8 to 64.3). The paper's interpretation: "the video benchmarks are relatively high-level and do not require understanding small details, so decreasing the pooling size is not very harmful. This illustrates the importance of tracking the captioning metric in addition to the other benchmarks, which requires a much more fine-grained understanding of the video."

Video SFT Data Composition: Molmo2-Cap and Molmo2-QA are complementary (Table 8c). Starting from academic datasets only (62.9 QA avg, 5.0 caption F1): adding QA data raises QA to 64.5 and captioning to 17.2; adding captions raises QA to 65.3 and captioning to 38.4; adding both achieves 64.8 QA and 39.5 captioning. The near-identical performance of "+ Cap" and "+ Cap/QA" on QA (65.3 vs. 64.8) suggests that caption data alone transfers well to QA, while "+ QA" alone transfers modestly to captioning (17.2)—the captioning capability primarily requires caption training data.

Caption Data Format: Merged video+frame captions are critical (Table 8d). Using only human video transcripts (V) yields 22.1 caption F1. Adding frame-level captions via merging (VF) dramatically improves to 35.5 F1. Further adding transcript (V) or frame captions (F) separately does not improve—35.6, 35.7, and 35.8 F1 for VF+F, VF+V, and VF+V+F respectively. The key insight: frame-level detail is essential for dense captioning, but supplying it as a separate task output does not help; it must be integrated into the target caption itself.

Counting Strategy: Pointing then counting is substantially better than direct counting (Table 9a). A model trained to directly predict counts achieves 61.3 BURST-VC accuracy and 28.1 Molmo2-VideoCount accuracy. Switching to "point then count" (where the model outputs points and the count is implicit in the maximum object index) achieves 61.5 BVC and 34.5 MVC—a 23% relative improvement on the more challenging Molmo2-VideoCount. The pointing-based approach particularly helps on complex scenarios with many objects.

Pointing Data Sources: Molmo2-VideoPoint and AcademicVideoPoint are complementary (Table 9b). Training with both sources achieves 61.5 BVC, 34.5 MVC, and 35.0 MVP F1. With only Molmo2-VP: 60.0, 34.3, 35.0 (slight drops). With only Academic-VP: 61.6, 9.0, 9.0—catastrophic drops on MVC and MVP, showing that academic data alone provides insufficient linguistic diversity and real-world query complexity for general video pointing, though it maintains BURST counting accuracy.

Upsampling High-Count Examples: Upsampling improves counting and pointing on complex cases (Table 9c). With medium-high count upsampling: 61.5 BVC, 34.5 MVC, 31.8 MVP. Without upsampling: 62.4 BVC (better), 32.1 MVC (worse), 28.1 MVP (worse). The small degradation on BURST (which skews toward lower counts) is offset by meaningful gains on the more challenging Molmo2 benchmarks.

Adding Pointing to Tracking Training: Pointing data improves tracking (Table 10a). A tracking-only model achieves 64.9 J&F, 70.0 F1, 68.4 HOTA. Adding pointing tasks during training raises these to 65.7, 71.1, 69.4—modest but consistent improvements, showing transfer from discrete pointing to continuous tracking.

Tracking Data Sources: Progressive improvement from academic to Molmo2 data (Table 10b). Starting from academic VOS only: 64.3 J&F, 68.8 F1, 66.7 HOTA. Adding bounding-box tracks raises F1 to 69.3 (J&F drops slightly to 63.9). Adding Molmo2-VideoTrack: 64.9 J&F, 70.0 F1, 68.4 HOTA—the best combination, with Molmo2 data contributing unique linguistic and temporal complexity.

Tracking Sub-tasks: Temporal grounding helps, single-point tracking may interfere (Table 10c). On academic VOS only: a pure tracking model achieves 64.2 J&F, 68.4 F1, 66.2 HOTA. Adding temporal grounding (predicting first/last frames) raises to 64.8, 69.4, 67.2. Adding single-point tracking drops to 64.3, 68.8, 66.7—suggesting the single-point task may pull the model away from multi-object tracking capability.

Pre-training Ablation: Pointing pre-training provides small but consistent benefits (Table 18). With pointing pre-training: 66.8 video QA, 31.8 caption F1, 80.9 image QA, 73.0 image pointing. Without pointing pre-training (compensated by 2× pointing sampling during SFT): 65.9, 31.3, 80.1, 71.8. All benchmarks degrade slightly, even those unrelated to pointing, supporting the hypothesis that "pointing pre-training simplifies the SFT stage."

Long-Context SFT: Improves long-video QA at the cost of captioning (Table 11, see Main Results).

Test-Time Scaling with SlowFast (Table 20, Figure 7). Increasing max frames from 128 to 224 improves long-video average from 64.6 to 65.6 but increases vision tokens from 10.6k to 18.6k. Using query-based SlowFast selection (where frames are assigned to slow or fast pathways based on cosine similarity to the text query) keeps vision tokens at 10.7k while achieving 65.7 long-video average—matching the performance of 224 frames with 43% fewer tokens. SlowFast trained during SFT does not outperform inference-only query selection, suggesting the model can generalize to different pooling sizes without explicit SlowFast training.

Critical Assessment

Claim from the executive summary: "Molmo2 are state-of-the-art among open-source models" on video understanding. This claim is supported with qualifications. On short-video benchmarks, Molmo2-8B does lead the fully-open category (Table 2: 69.9 average vs. PerceptionLM-8B's 68.5 and LLaVA-Video-7B's 59.4) and is competitive with open-weight models (Qwen3-VL-8B: 65.3, Eagle2.5-8B: 67.0). On long-video benchmarks, it trails open-weight models (64.1 vs. Eagle2.5-8B's 65.2). The paper itself makes this distinction explicit ("outperforms previous fully-open models but lags behind the best open-weight models" on long videos), so the claim is carefully scoped. However, a reader might reasonably interpret "state-of-the-art among open-source models" as claiming to beat all open models on all video axes, which is not true for long videos.

Claim: "Molmo2 significantly outperforms existing open-weight models like Qwen3-VL (35.5 vs 29.6 accuracy on video counting)." Strongly supported by Table 2 (Molmo2-VideoCount close accuracy) and Tables 16-17 (category and count-range breakdowns). The gap is particularly stark on high-count examples (25-60 range: Molmo2-8B achieves 7.0-12.3% accuracy vs. Qwen3-VL-8B's 0.0%). This is not a marginal improvement but a qualitative capability difference—other models simply break down on complex counting. The evaluation is fair: models are evaluated in their native output format (Molmo2 with points, baselines with integers), and the "close accuracy" metric prevents penalizing near-correct answers. However, Molmo2-VideoCount is a custom benchmark (533 examples), and the paper does not report how representative it is of real-world counting difficulty distribution. The possibility that the benchmark was constructed in a way that implicitly favors Molmo2's training distribution (since both come from the same annotation pipeline) should be considered—though the BURST-VC results, an independent benchmark, also show Molmo2 leading (76.1 close accuracy, highest overall), which mitigates this concern.

Claim: "Molmo2 surpasses proprietary models like Gemini 3 Pro on some tasks (38.4 vs 20.0 F1 on video pointing and 56.2 vs 41.1 J&F on video tracking)." Strongly supported for pointing (Table 3: 38.4 vs. 20.0 F1 on Molmo2-VideoPointVal) and tracking (Table 5: 56.2 vs. 44.6 J&F on Molmo2-Track). The pointing gap is genuinely large (nearly 2×) and the paper transparently describes the prompt engineering effort for baselines: "we carefully tune the prompts and try both point and bounding-box formats for our baseline models; however, we were unable to find a formulation that achieved very strong performance." However, three caveats apply:

1. Custom benchmark construction. Molmo2-VideoPointVal (181 examples) is a subset of Molmo2-VideoPoint filtered by SAM 2 mask quality. Since the evaluation protocol (point must fall within SAM 2 mask) implicitly assumes the SAM 2 mask is accurate, any systematic differences in mask quality between query types could affect results. The paper manually filters for incorrect masks, but the filtering criteria and inter-annotator agreement are not reported.

2. Format mismatch for baselines. API models are evaluated using bounding boxes (extracting centers as proxy points) because they "fail to generate accurate points." This creates an inherent disadvantage: bounding box centers may not correspond to the ground-truth segmentation mask centroid, especially for non-convex objects. However, Molmo2's training explicitly optimizes for point-in-mask accuracy, making the evaluation protocol favorable to its output format.

3. Tracking benchmark composition. Molmo2-Track (Table 5) spans 5 domains (animals, person, sports, dancers, misc) with 1,386 clips, which is substantial. But the paper does not analyze whether performance differences between models are consistent across domains—the overall average could be driven by a few domains where Molmo2 excels while being weaker on others where baselines have different relative strengths. The per-domain breakdown in Table 5 partially addresses this: Molmo2 leads on Animal F1 (82.0 vs. Gemini 3 Pro's 62.3), Dancers (63.9 vs. 44.3), and Person (47.9 vs. 30.7), but the margin varies substantially.

A structural concern across all grounding evaluations: The paper builds its own benchmarks (Molmo2-VideoCount, Molmo2-VideoPointVal, Molmo2-Track) using the same annotation pipeline and crowdworker pool as the training data. Even with careful validation set separation, this introduces a potential distribution match advantage: Molmo2's training data and evaluation data share annotator biases, interface conventions, and linguistic patterns that may not generalize to truly out-of-distribution grounding queries. The academic benchmarks (BURST, MeViS, Ref-Davis, ReasonVOS, Ref-YT-VOS) are fully independent and do show strong Molmo2 performance, but the largest gaps between Molmo2 and baselines appear on the custom benchmarks—the F1 gap on Molmo2-VP (38.4 vs. 20.0) is much larger than the academic Ref-Davis gap (81.3 F1 for Molmo2 vs. 62.7 for Gemini 2.5 Pro, though the metrics differ). This pattern is consistent with a distribution match effect, though not conclusive.

Missing experiments that would strengthen the work:

• No comparison against Molmo1 with video capability added post-hoc. The paper evaluates Molmo + SAM 2 (Table 5, bottom rows) as a tracking baseline, but this is a crude pipeline (Molmo points → SAM 2 propagates) rather than an end-to-end trained model. A comparison against a Molmo1 model fine-tuned on the same video data would isolate whether the gains come from the new data or from architectural improvements (bidirectional attention, token weighting, etc.). The fact that the Molmo2-8B image average (81.7) is only slightly ahead of Molmo-72B (81.2) despite being 9× smaller suggests the data and training recipe improvements are significant, making this ablation more important.

• No difficulty-stratified video grounding analysis. The paper does not break down pointing/counting/tracking performance by query complexity, object count, occlusion level, or video length. Such a breakdown would reveal whether Molmo2's advantage is concentrated in "easy" queries (few objects, short videos) or extends to truly challenging cases. Table 17 provides count-range breakdowns, but no analogous breakdown exists for linguistic complexity or visual difficulty.

• No cross-model contamination analysis for the human evaluation. The human preference study uses videos and questions collected by the same annotation pipeline as training data. While the videos themselves are presumably held out, the linguistic style and question types may favor Molmo2. A control experiment using independently sourced videos and questions (e.g., from existing benchmarks not in the training mix) would strengthen the human evaluation claims.

• No scaling analysis of data quantity vs. model size for grounding. A key claim of the paper is that grounding bottlenecks on data diversity rather than model capacity. An experiment that varies the amount of grounding training data (e.g., 10%, 25%, 50%, 100% of the full Molmo2 grounding datasets) for both 4B and 8B models would directly test this. If the 4B model with full data matches or exceeds the 8B model with partial data, the data bottleneck claim is supported; if the 8B model consistently outperforms regardless of data quantity, the claim is weaker.

Robustness concerns with specific results:

• Human evaluation captioning Elo scores (Table 15) show Molmo2-8B at 1049, ranking 8th overall. This is a notable weakness—GPT-5, Gemini, and even Qwen3-VL-8B (1105!) substantially outperform Molmo2 on captioning in human preference. The paper attributes this to text repetition, but the gap to Qwen3-VL-8B (56 Elo points) is large enough to suggest that Molmo2's captioning, despite high automated F1 scores (43.2), does not meet human quality expectations. The automated caption F1 metric measures factual completeness and accuracy, while human evaluation also captures fluency, coherence, and naturalness—Molmo2 appears to trade off the latter for the former.

• The long-context SFT tradeoff (Table 11) is genuine and significant: captioning drops from 42.3 to 39.9 F1. This is not a small fluctuation—it's a 5.6% relative decline. The paper does not explore mitigation strategies (e.g., mixing long and short sequences during long-context training, or using a higher captioning sampling rate). The tradeoff suggests that the model's representation is being stretched by the longer context, and captioning is the capability that suffers.

• Multi-image performance (Table 6, bottom rows) is a clear weakness. Molmo2-8B's average of 56.4 on multi-image benchmarks is substantially behind GLM-4.1V-9B (67.4) and barely ahead of some 4B models. The paper's multi-image QA dataset (Molmo2-MultiImageQA, 72k pairs) and synthetic multi-image data are relatively small compared to the image and video QA components. The gap suggests that multi-image reasoning requires different data than what Molmo2 emphasizes—possibly more interleaved text-and-image documents or structured comparison tasks.

Summary of claim strength and boundaries:

The paper's strongest claims—that Molmo2 achieves state-of-the-art video grounding by investing in large-scale, diverse human-annotated pointing and tracking data—are well-supported by the independent academic benchmarks (BURST, MeViS, Ref-Davis, ReasonVOS) and the large margins on custom benchmarks. The claim that this is achievable without proprietary VLM distillation is a fact about the training pipeline, not an empirical hypothesis requiring testing. The claim that Molmo2 is competitive with open-weight models on general video understanding is supported for short videos but qualified for long videos and reasoning-heavy tasks. The claim that Molmo2 matches or exceeds proprietary models on grounding is supported but should be interpreted as "on the specific grounding capabilities we targeted and evaluated," not as a general claim about video understanding—Molmo2 does not match proprietary models on most video QA benchmarks, captioning human preference, or reasoning tasks. The most well-supported specific finding is that pointing-based counting dramatically outperforms direct integer counting (Table 9a: 34.5 vs. 28.1 MVC accuracy), which is a clear, replicable insight about task formulation that does not depend on model scale or benchmark construction.

The most significant unexamined question is whether Molmo2's grounding capabilities transfer to tasks that require acting on grounded understanding—e.g., robotic manipulation guided by pointing coordinates, or video editing guided by tracking outputs. The paper evaluates static metrics (point accuracy, segmentation overlap, counting accuracy) but does not test whether the generated points are practically useful for downstream applications, which would be the ultimate validation of grounding as a general capability rather than a benchmark-specific skill.

6. Limitations and Trade-offs

6.1 The Long-Video Understanding Gap Remains Substantial, and Addressing It Hurts Caption Quality

Molmo2 achieves competitive short-video understanding (69.9 average on short QA benchmarks, Table 2) but "lags behind the best open-weight models" on long-video tasks (64.1 long QA average vs. Eagle2.5-8B's 65.2 and Gemini 2.5 Pro's 80.4). The paper attributes this to "a lack of open-source long (10+ minutes) training data and computational limitations that made it challenging to run extensive ultra-long context training" (Section 5, Discussion of Table 2).

The consequence is a genuine trade-off between temporal coverage and output quality. The long-context SFT ablation (Table 11) reveals that increasing the sequence length to 36,864 tokens and max frames to 384 improves long-video QA from 64.4 to 67.4 but degrades video captioning F1 from 42.3 to 39.9—a 5.6% relative decline. Short-video QA and image QA are unaffected. This is not a minor fluctuation; it indicates that forcing the model to process longer sequences redistributes its representational capacity away from the fine-grained visual detail needed for dense captioning. The paper does not explore whether this trade-off can be mitigated through data mixing strategies (e.g., interleaving short and long sequences during long-context training, or up-weighting captioning examples during this stage). A practitioner deploying Molmo2 for applications requiring both long-video comprehension and detailed description faces an inherent tension: you can have one or the other, but not both at maximum quality, without further architectural or training advances.

Mitigation status: The paper acknowledges this as a limitation driven by data availability and compute constraints, and the long-context SFT stage partially addresses it (improving long QA by 3 points). However, the captioning degradation is presented as an observed fact without proposed remediation, making it a live trade-off for any deployment.

6.2 Video Grounding Degrades on Long Videos and High-Frequency Objects, and the Training Data Does Not Cover Extended Durations

The paper's grounding capabilities—pointing, counting, and tracking—are trained on videos with a maximum of 128 frames at 2 fps during SFT, and 384 frames during long-context training, with all grounding annotations "always at 2 fps" (Appendix H). This imposes a hard upper bound on the video duration for which grounding is reliable: approximately 64 seconds in SFT and 192 seconds after long-context training. The paper states this explicitly:

"Grounding has limited support for long (3 minutes+) videos because our grounding training is limited to that length."

The consequence is that grounding performance likely deteriorates—perhaps catastrophically—on videos exceeding the training length. The paper notes specific observed failure modes that become more common on longer videos or high-frequency objects: "degenerate outputs, such as a long line of points on one frame or the same point for every frame" (Appendix H). These failures suggest the model has not learned robust temporal generalization for grounding, instead relying on the statistical regularities of the training distribution (videos under ~3 minutes, objects appearing at most a few dozen times). For applications like surveillance footage analysis, sports video review, or any domain with extended continuous video, Molmo2's grounding capabilities are untested and likely unreliable. Even within the supported duration range, the paper observes that grounding "is less consistent than image grounding," with no model exceeding 40% accuracy on video counting or pointing benchmarks compared to 70-90% for image grounding. The temporal dimension introduces re-identification challenges ("understanding whether two objects in two different frames are the same object or not") that the current training data does not fully resolve.

Mitigation status: The paper acknowledges this as a limitation (Appendix H) and suggests that "customizing how frames are sampled in these cases to ensure that all grounding annotations are selected" could help, but provides no implementation or evaluation. The lower frame rate needed for longer videos (<2 fps) would misalign with the 2 fps annotations, creating a fundamental tension between temporal coverage and annotation fidelity that the paper does not resolve.

6.3 The Custom Grounding Benchmarks are Constructed Using the Same Annotation Pipeline as Training Data, Creating a Potential Distribution-Match Advantage

Molmo2's most impressive grounding results—outperforming Gemini 3 Pro by nearly 2× on video pointing (38.4 vs. 20.0 F1, Table 3) and achieving large margins on Molmo2-Track (56.2 vs. 44.6 J&F, Table 5)—are evaluated on benchmarks constructed using the same annotation pipeline, crowdworker pool, and linguistic conventions as the training data. The Molmo2-VideoPointVal benchmark consists of 181 examples manually filtered from Molmo2-VideoPoint annotations with SAM 2 mask validation. The Molmo2-VideoCount benchmark (533 examples) is drawn from the same annotation effort. The Molmo2-Track benchmark (1,386 clips) was collected "using the same pipeline" as the Molmo2-VideoTrack training data, with annotators writing text queries for the same video sources.

The consequence is that the reported performance gaps on custom benchmarks may overstate Molmo2's advantage on truly out-of-distribution grounding queries. The model has been extensively trained on examples that share annotator biases, interface conventions, vocabulary preferences, and query formulation patterns with the evaluation data. This does not invalidate the results—the model genuinely learned to perform the task on held-out examples from the same distribution—but it means the benchmarks measure in-distribution generalization, not generalization to independently collected grounding queries. A user asking "Point to the moment the athlete celebrates" on a video from a completely different source with different filming conventions, object appearances, and action types may encounter lower performance than the benchmark numbers suggest. The academic benchmarks (BURST, MeViS, Ref-Davis, ReasonVOS, Ref-YT-VOS) are fully independent from the Molmo2 annotation pipeline, and the paper does show strong performance there (e.g., MeViS valid-u F1 of 75.9, Table 4). However, the margins on academic benchmarks are typically smaller than on custom benchmarks—compare Molmo2-8B's lead on Ref-Davis F1 (81.3 vs. Gemini 2.5 Pro's 62.7, a 1.3× gap) to the Molmo2-VP gap (38.4 vs. 20.0, a 1.9× gap). This pattern is consistent with a partial distribution-match effect, though far from conclusive.

Mitigation status: The paper does not discuss this as a limitation. The academic benchmark results provide independent validation that Molmo2's grounding advantage is genuine, but the magnitude of that advantage is most precisely characterized by the custom benchmarks, which are the least independent. Future work should evaluate Molmo2 on independently collected, out-of-distribution grounding benchmarks to establish the true generalization gap.

6.4 The Revision Model (Molmo2) Has No Mechanism for Self-Correction or Iterative Refinement; Point Tracking Drift is a Known but Unquantified Failure Mode

Unlike the reference paper's approach of training dedicated revision models that iteratively improve outputs, Molmo2 produces grounding outputs—points and tracks—in a single forward pass. For tracking specifically, the paper notes:

"Molmo2's generated tracks will sometimes change the location of its output point on the target object" (Appendix H)

This is point tracking drift: the model's predicted point for the same tracked object shifts from frame to frame, even when the object is stationary or moving smoothly. The paper attributes this to the training data generation pipeline, which "does not always ensure that the point is consistently placed within the target object for every frame."

The consequence is that downstream applications requiring stable, pixel-precise tracking—robotic grasping, video editing with precise masks, scientific motion analysis—may find Molmo2's outputs unreliable without post-processing. The paper's evaluation metrics (J&F, point F1, HOTA) partially capture this: HOTA specifically penalizes inconsistent object IDs and track fragmentation, and Molmo2 achieves strong HOTA scores (e.g., 72.6 on MeViS, Table 4). However, HOTA measures association accuracy (whether the correct object ID is maintained), not spatial precision of individual points across frames. The drifting point problem would manifest as reduced mask quality when Molmo2's points are fed to SAM 2 for segmentation, but since the paper reports J&F (segmentation quality) using SAM 2's outputs, and SAM 2 has its own temporal smoothing, the raw point drift may be partially hidden by the evaluation pipeline. A user who needs precise point coordinates for a single object across 100 frames—without SAM 2 post-processing—may encounter qualitatively worse performance than the benchmark numbers indicate.

Mitigation status: The paper acknowledges the point tracking drift issue (Appendix H) and suggests "future improvements in generating points from bounding box or segment mask data could mitigate this issue," but does not provide quantitative characterization of drift magnitude, frame-to-frame displacement distributions, or the conditions under which drift is most severe. This is a known failure mode with no evaluated solution.

6.5 Token Weighting is an Empirical Heuristic with Task-Specific Trade-offs, Not a General Solution to Multi-Task Loss Imbalance

The token-weighting scheme—fixed weights of 0.1 for video captions and 0.2 for pointing, plus $\sqrt[4]{n}$ for other tasks—is presented as a practical solution to the problem that "long-output examples can easily become the large majority of loss tokens even if they are sampled rarely" (Section 3.2). The ablation in Table 8b shows the trade-off: removing token weighting improves captioning F1 (from 39.5 to 40.0) but degrades QA performance (from 64.8 to 64.0). The paper chooses to retain weighting to preserve QA performance.

The consequence is that the optimal weighting scheme is task-dependent and likely dataset-dependent, making it non-trivial to adapt to new data mixtures or task priorities. A practitioner who wants to prioritize captioning quality over QA would be better off removing token weighting, as the ablation shows. A practitioner adding a new task with extremely dense outputs (e.g., 10,000-token video transcripts) would need to determine a new weight through trial and error. The $\sqrt[4]{n}$ formula is not derived from theory or a scaling law; it is an empirical choice that happens to work for this specific data mixture at this model scale. The paper does not ablate different weighting schemes (e.g., square root, no weighting, task-specific learned weights) beyond the binary on/off comparison, so the sensitivity of final performance to the specific weighting values is unknown. The interaction between token weighting and the sampling rate balancing (Table 1, Figure 4) is also unexplored—these are two separate knobs that both affect per-task loss contribution, and the paper provides no guidance on how to jointly tune them.

Mitigation status: The paper does not frame this as a limitation; token weighting is presented as a successful training innovation. However, the ablation in Table 8b shows it is a deliberate trade-off, not a Pareto improvement. The lack of sensitivity analysis or theoretical justification makes it difficult to adopt the specific weighting values in different training contexts without extensive re-ablation.

6.6 The Model is Evaluated on a Single Model Family (Qwen3/OLMo 3) with One Vision Encoder (SigLIP 2), and the Data Pipeline Uses Closed Text-Only LLMs, Limiting Claims of Full Openness and Generalizability

All Molmo2 variants use the same SigLIP 2 So400m/14 384px vision encoder. While the paper demonstrates the approach with two LLM backbones (Qwen3 and OLMo 3), the vision encoder is fixed. SigLIP 2, as the paper notes, "utilize[s] a closed-data SigLIP 2 image encoder" (Appendix H): "We chose to use SigLIP 2 because there are currently no competitive open-data encoders." Additionally, the data generation pipeline "use[s] closed text-only LLMs for data generation, as is common practice" (Appendix H)—specifically, Claude Sonnet 4.5 for initial answer generation in Molmo2-AskModelAnything, and an unnamed LLM for rewriting spoken captions, generating CapQA/SubtitleQA questions, and canonicalizing multi-image pointing labels.

The consequence is that Molmo2 is not fully reproducible from scratch without access to proprietary components. While the paper's core claim—no distillation from proprietary VLMs—is true and significant, the reliance on closed text-only LLMs and a closed-data vision encoder means the training pipeline is not fully independent of commercial APIs. If Claude Sonnet 4.5 became unavailable or its behavior changed, the AskModelAnything dataset could not be reproduced exactly. More subtly, the quality of the LLM-generated questions in CapQA and SubtitleQA (1.3M examples) depends on the specific LLM used, and a weaker open LLM might produce lower-quality synthetic data, reducing the effectiveness of these dataset components. The fixed vision encoder also limits claims about the generality of the approach: it is unknown whether the same data mixture and training recipe would produce comparable results with a different vision encoder (e.g., a fully open ViT trained on open data, if one becomes available), or whether the SigLIP 2 encoder's specific properties are load-bearing for the grounding capabilities.

Mitigation status: The paper is transparent about these dependencies (Appendix H): "We call upon the open-source community to explore such alternatives in future work" regarding the vision encoder, and notes that "future open LLMs will become sufficiently proficient to be used in place of closed ones to reproduce this dataset in a fully open manner." The paper draws a clear distinction between VLM distillation (which it avoids) and text-only LLM use (which it considers acceptable as a temporary dependency). However, a practitioner seeking to replicate Molmo2's data pipeline exactly will need access to these closed systems, and the paper provides no characterization of how sensitive the final model quality is to the specific LLM used for data generation. A controlled experiment substituting an open LLM for the closed one at each data generation step—and measuring the impact on downstream model performance—would quantify this dependency but is not performed.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not introduce a new architecture or a fundamentally new learning algorithm. Its contribution is something more unusual in the current multimodal LLM literature: a demonstration that a deliberate, large-scale investment in diverse human-annotated spatio-temporal data can produce capabilities that proprietary model distillation cannot match, and that these capabilities—grounded pointing, counting, and tracking—can be integrated into a general-purpose VLM without architectural specialization. This is less a paradigm shift than a strategic reframing of where the bottleneck lies for video understanding, with several specific consequences for the field.

Reframing video grounding from an architecture problem to a data problem. Before Molmo2, the dominant assumption—reflected in the proliferation of specialized segmentation and tracking models like VideoLISA, VideoGLaMM, Sa2VA, and SAM 3—was that precise video localization required dedicated architectures: separate mask decoders, temporal propagation modules, or explicit correspondence heads. Molmo2 demonstrates that a standard autoregressive language model, with no segmentation-specific components, can outperform these specialized models on their own benchmarks (Table 4: Molmo2-8B achieves 75.9 F1 on MeViS vs. Sa2VA-8B's 46.9 J&F; Table 5: 56.2 J&F on Molmo2-Track vs. Sa2VA-8B's 46.7). The mechanism is not architectural magic—it is that the model learns to individuate, re-identify, and localize objects by being trained on 520k human-annotated pointing instances and 800k tracking examples that cover the specific linguistic and visual diversity users actually query. This finding redirects research investment: improving video grounding is primarily a data collection and task formulation problem. A better vision encoder or a larger LLM will not teach a model to track objects through occlusion if the training data never shows it examples of occlusion with tracking supervision. The paper's ablation showing that upsampling high-count pointing examples directly improves high-count accuracy (Table 9c: MVC accuracy rises from 32.1 to 34.5 with upsampling) is a clean causal demonstration that data distribution shapes capability in a way that no architecture change alone could achieve.

Making grounding a first-class VLM capability rather than a post-processing step. The paper's most influential technical decision—representing points as plain-text tokens in an HTML-like format—means that grounding is no longer a separate pipeline (detect objects → track → answer query) but an integrated output modality of the same model that generates captions and answers questions. This collapses a stack of specialized models into a single system, which has practical consequences: a robot that needs to both describe a scene and point to objects within it can now use one model with one set of weights. The human preference evaluation (Table 15) shows that this integration does not come at the cost of general understanding—Molmo2-8B ranks 5th overall in Elo score (1057), ahead of GPT-5 (1031) and Claude Sonnet 4.5 (1008). The field now has an existence proof that grounding and general video understanding are complementary, not competing, capabilities in a unified architecture. This makes it harder to justify shipping VLMs without grounding—the capability gap between models that can point and those that cannot is large enough (38.4 vs. 1.5 F1 on video pointing, Table 3) that it will increasingly be seen as a basic requirement rather than a premium feature.

Reconciling why prior video QA and captioning data pipelines (distillation from GPT-4V) produce models that cannot ground. The paper provides an implicit explanation for a puzzling observation: video models trained on GPT-4V-distilled captions and QA pairs (LLaVA-Video, Eagle 2.5, PerceptionLM) achieve strong general understanding scores but near-zero pointing and tracking performance. The reason is that proprietary VLMs, even the strongest ones, do not produce spatial or temporal coordinates in their training data—or if they do, they do so unreliably (the paper shows Gemini 2.5 Pro achieves only 13.0 F1 on pointing, Table 3). Distilling from these models propagates their blindness to precise localization. The open-source community's reliance on proprietary distillation was not just a matter of convenience or cost; it was actively blocking progress on grounded capabilities because the teacher models lacked those capabilities themselves. Molmo2 breaks this circular dependency by bootstrapping from human perception directly—annotators click on objects in video frames, and the model learns to associate language with spatio-temporal coordinates from those clicks. This reframes the field's data strategy: for capabilities that proprietary models do not possess, distillation is not just suboptimal but actively impossible; only direct human annotation or self-supervised methods that do not depend on an existing VLM can create the training signal.

The training efficiency innovations (packing, message trees, token weighting) generalize beyond this paper. While packing and message trees are presented as engineering contributions, they address a structural problem in multimodal training: how to combine tasks with radically different output lengths (1 token to 4,000+ tokens), visual token counts (single image to 384-frame videos), and annotation density (one QA pair per video to 60+ pointing annotations per video) without having any single task dominate the loss or the memory footprint. The 15× training efficiency gain from packing and the token-weighting ablation (Table 8b: removing weighting improves captioning at the cost of QA) provide concrete empirical evidence that these are not just conveniences but necessary components of a stable multi-task training recipe. Future multimodal models trained on similarly heterogeneous data mixtures will need to adopt equivalent mechanisms, and the paper provides a tested baseline: pool-based dynamic programming packing with quantized token counts, message-tree attention masking to prevent cross-branch contamination, and $\sqrt[4]{n}$ token weighting as a reasonable starting heuristic. The finding that training without SlowFast but using query-based frame selection at inference time matches training-with-SlowFast (Appendix E: 65.7 vs. 65.5 long video average) is a specific, surprising result that suggests the model generalizes to different pooling sizes without explicit training—a useful practical insight for any deployment that needs to handle variable-length videos.

What becomes less attractive as a research direction. The paper's results make it harder to argue that specialized segmentation or tracking architectures are necessary for video grounding in VLMs. Sa2VA-8B, a purpose-built model that marries SAM 2 with LLaVA, is outperformed by Molmo2-8B on tracking (Table 4: 75.9 vs. 46.9 F1 on MeViS; Table 5: 56.2 vs. 46.7 J&F on Molmo2-Track). This does not mean specialized models have no role—they may be more computationally efficient, or better at extremely high-resolution mask prediction—but it shifts the burden of proof: a specialized architecture must now demonstrate gains over a general VLM trained on sufficient grounding data, not just over other specialized models. Similarly, the finding that Gemini 2.5 Pro, despite its massive scale, achieves only 13.0 F1 on video pointing (Table 3) undermines the assumption that scaling model size or pre-training data will naturally produce grounding capabilities. Grounding appears to require explicit supervision of the specific output format (point coordinates with temporal indexing) and the specific cognitive skill (individuating and re-identifying objects across frames), which even the largest proprietary models do not develop through scale alone.

Follow-Up Research This Work Enables

Quantifying the generalization limits of video grounding: construct an out-of-distribution grounding benchmark with independently sourced videos and queries. The paper's custom benchmarks (Molmo2-VideoPointVal, Molmo2-VideoCount, Molmo2-Track) are drawn from the same annotation pipeline as the training data, creating a potential distribution-match advantage. A follow-up study should collect video grounding queries on entirely new video sources—different platforms, different filming conventions, different annotator demographics—and evaluate Molmo2 against baselines to establish the true generalization gap. The hypothesis to test: does Molmo2's advantage shrink on truly out-of-distribution queries, and if so, by how much? The academic benchmarks (BURST, MeViS, Ref-Davis) provide partial evidence of generalization, but they skew toward simple referring expressions and short videos. A benchmark of 500+ queries on long (3-10 minute), user-generated videos with complex, natural-language grounding requests (e.g., "Track the person who enters the frame from the right, interacts with the whiteboard, and then exits left") would stress-test whether Molmo2's grounding capabilities are a robust skill or a distribution-specific learned behavior. Such a benchmark would also serve the broader community by providing a standardized evaluation for video grounding that does not overlap with any existing training pipeline.

Combining Molmo2-style grounding data with RL-based self-improvement to close the loop on pointing and tracking quality. The paper's pointing and tracking training is purely supervised—the model learns to imitate human annotations. But the outputs are known to have failure modes: degenerate repeating points, tracking drift, and false positives on high-frequency objects (Appendix H). A natural extension is to treat the pointing/tracking task as a sequential decision problem where the model generates point coordinates autoregressively and receives a reward based on whether the point falls within the ground-truth object mask, whether object IDs are consistent, and whether all instances are found. The reward signal can come from SAM 2 (for mask validation on unlabeled video) or from a learned verifier trained on the human annotations. This would allow the model to improve beyond the ceiling of human annotation quality—annotators sometimes miss objects or place points imprecisely, and RL could learn to correct these errors. The paper's failure to improve the revision model with ReST$^{EM}$ (Appendix K: "additional sequential revisions substantially hurt performance") suggests that naive RL is not sufficient, but the pointing task—with its clean, mask-based verifiability—is a more natural fit for RL than open-ended caption refinement. A concrete experiment: pre-train on Molmo2-VideoPoint, then fine-tune with PPO where the reward is the F1 score against SAM 2-generated masks on unlabeled video, and measure whether pointing accuracy improves on held-out benchmarks.

Training a difficulty-aware pointing policy that allocates more inference compute to complex queries. The paper observes that "video grounding is less consistent than image grounding" (Appendix H) and that high-count queries (25-60 objects) particularly challenge all models (Table 17: Qwen3-VL-8B achieves 0% accuracy in this range, Molmo2-8B achieves 7-12%). This suggests a test-time compute allocation problem: for queries likely to involve many objects, the model should be allowed to process more frames, use higher resolution, or generate multiple pointing hypotheses that are then merged. A follow-up could train a lightweight difficulty predictor (from the query text and a few initial frames) that estimates the expected object count and query complexity, then uses that prediction to allocate a variable frame budget—e.g., 32 frames for simple queries, 384 frames for complex ones. The paper's SlowFast experiments (Appendix E) provide a starting point: query-based frame selection already improves long-video understanding without increasing token count (65.7 long QA average with query-based SlowFast vs. 64.6 with 128 frames). Extending this to pointing—where the query directly describes what to point at—could yield similar efficiency gains. A concrete experiment: on Molmo2-VideoCount, compare fixed-384-frame pointing against an adaptive scheme that starts with 64 frames, estimates the count from initial points, and requests more frames if the count is high or the model is uncertain, measuring whether adaptive allocation matches fixed-allocation accuracy with fewer total frames processed.

Verifying that Molmo2's pointing coordinates are practically useful for downstream robotics and video editing tasks, not just mask-overlap metrics. The paper evaluates pointing via F1 against ground-truth masks—a reasonable proxy metric, but one that does not measure whether the points are spatially precise enough or temporally stable enough for applications that consume them. A robotics experiment would feed Molmo2's point predictions (for a query like "Point to the handle of the drawer") to a manipulation policy and measure task success rate—does the robot successfully grasp the indicated object? A video editing experiment would use Molmo2's tracking outputs (object tracks with consistent IDs) to apply effects (blur, color shift) to tracked objects and measure whether the effect stays on-target through occlusions and rapid motion, as judged by human raters. If Molmo2's point drift problem (Appendix H) causes visible jitter or ID swaps in the edited output, the benchmark F1 scores would be revealed as overestimates of practical utility. This type of study would calibrate the field's metrics: what F1 threshold corresponds to "usable for robotics" vs. "usable for casual search" vs. "not production-ready." The paper's strong academic benchmark results (75.9 F1 on MeViS, 56.2 J&F on Molmo2-Track) set high expectations, but the translation to task success is unmeasured.

Characterizing the data scaling laws for video grounding: how much pointing and tracking data is enough? The paper trains on 520k pointing instances and 800k tracking examples, but provides no evidence about whether this quantity is near-optimal, insufficient, or already past the point of diminishing returns. A scaling study that trains Molmo2 variants on 10%, 25%, 50%, 75%, and 100% of the grounding data, evaluating on both in-distribution and academic benchmarks, would reveal the shape of the data scaling curve. The key questions: does tracking performance continue to improve log-linearly with data quantity (suggesting more annotation investment would yield gains), or does it plateau (suggesting the current data is sufficient and further gains require different data, e.g., more diverse queries or longer videos)? Does pointing data transfer to tracking (as Table 10a suggests) with a specific efficiency ratio—i.e., N pointing examples provide the same tracking improvement as M tracking examples? These scaling curves would guide future data collection investments, telling the community whether to prioritize breadth (more video sources) or depth (more annotations per video) or complexity (more multi-object, long-duration queries). The paper's own ablation (Table 10b) showing progressive improvement from academic VOS → bounding box tracks → Molmo2 data already hints at a positive scaling relationship, but the functional form is unknown.

Stress-testing the pointing-as-counting formulation against edge cases where individuation is ambiguous. The paper's elegant insight—that the maximum object index in a pointing sequence equals the count—works when objects are clearly distinct and annotators agree on what constitutes a separate instance. But what happens when individuation is ambiguous? A video of a flock of birds wheeling in the sky, or a crowd of people where individuals merge and separate, or a pile of identical objects where boundaries are unclear—in these cases, human annotators may disagree on the count, and different annotators might mark different numbers of points for the same video. A follow-up could collect multiple independent annotations on such ambiguous-count videos and measure the inter-annotator agreement (e.g., Krippendorff's alpha) as a function of object type and density. Then measure whether Molmo2's counting predictions fall within the range of human annotator counts, and whether the model's uncertainty (perhaps measured via multiple samples with temperature > 0) correlates with annotator disagreement. This study would establish the ceiling of the pointing-as-counting approach: when humans cannot agree on a count, a model trained to imitate humans should not be expected to produce a singular correct answer, and evaluation metrics should account for this inherent ambiguity. The paper's "close accuracy" metric (∆ = 1 + ⌊0.05 × gt⌋) is a step in this direction but is a fixed formula, not a data-driven estimate of acceptable tolerance.

Practical Applications and Downstream Use Cases

Video search with spatio-temporal localization for large media archives. A media company with millions of hours of archived footage needs to answer queries like "Find all shots where a person in a red jacket walks past the fountain." Current video search systems rely on metadata or whole-video embeddings and cannot localize the specific timestamp and region within the frame. Molmo2's pointing capability—trained on exactly this type of query (Table 16: Action/Event category in Molmo2-VideoPoint)—enables a search pipeline where the model processes videos at 2 fps, generates point coordinates for matching events, and returns ranked clips with bounding timestamps and spatial regions. The paper's BURST-VC close accuracy of 76.1 (Table 3) on counting benchmarks suggests reasonable reliability for this application, though the pointing F1 of 39.9 on Molmo2-VP indicates that false positives and missed instances remain significant. The practical deployment would run Molmo2 as an offline indexing process (tolerating the computational cost of 384-frame processing per video) and surface candidate clips for human verification, reducing the search space from hours of footage to a few candidate moments.

Assistive technology for visually impaired users: real-time video description with spatial referencing. A visually impaired user points their phone camera at a scene—a busy kitchen, a street intersection, a shelf of products—and asks "Where is the red mug?" or "Which bus is the number 42?" Existing video description models can answer "The red mug is on the counter" but cannot indicate where on the counter, requiring the user to search manually. Molmo2's pointing capability enables a different interaction: the model outputs "The red mug is here" with a spatial coordinate that the phone can translate into haptic feedback (vibration intensity/direction), audio spatialization ("left side of the counter, near the toaster"), or screen-reader-accessible coordinates. The paper's image pointing results (Point-Bench average 68.7, Table 7, with Affordance at 84.8) suggest that spatial precision is strongest for clearly defined objects in typical viewpoints—exactly the setting for assistive queries. The key deployment challenge is latency: real-time use requires processing at (ideally) 5-10 fps rather than the paper's 2 fps annotation rate, and the SlowFast results (Appendix E: matching 224-frame performance with 43% fewer tokens using query-based selection) provide a path to reducing per-frame computation for video streams.

Data annotation acceleration: using Molmo2 to pre-annotate video tracking datasets for human refinement. The paper's annotation pipeline for Molmo2-VideoTrack required human annotators to write text queries describing object tracks, a process that produced 29k queries across 8k video clips (Table 21). This is expensive and slow. Molmo2 itself, once trained, could serve as a pre-annotation tool: given a new video, the model proposes candidate object tracks (using its tracking capability) and generates draft text queries describing those tracks. Human annotators then verify, correct, and refine these drafts rather than creating annotations from scratch. The paper's strong tracking performance on diverse domains (Table 5: 82.0 F1 on Animals, 63.9 on Dancers, 47.9 on Person) suggests that the model is reliable enough for common object categories to provide useful starting points, with human effort focused on correcting edge cases (occlusions, rare objects, complex referring expressions). The economics are compelling: if Molmo2's pre-annotations are correct 60% of the time (roughly its F1 scores on person and sports domains) and partially correct on another 20%, human annotation time could be reduced by 50% or more compared to annotation from scratch, while the final quality (after human correction) matches fully manual annotation. This creates a virtuous cycle where Molmo2 accelerates the creation of more training data, which can then be used to train improved versions of the model.

Fine-grained video content moderation with explainable localization. Content moderation platforms need to identify specific moments and regions within videos that violate policies—a weapon appearing at a specific timestamp, a dangerous activity in a specific region of the frame. Molmo2's pointing capability enables a moderation pipeline where the model not only flags a video as containing policy-violating content but outputs the exact timestamps and spatial coordinates where the violation occurs. This provides explainability (moderators can jump directly to the indicated moment and verify) and reduces the review burden (rather than watching the full video, moderators check the pointed-to segments). The paper's anomaly pointing training (10k videos of generative AI artifacts, Appendix F) suggests the model can learn to localize specific defect types, and extending this to a broader set of violation categories is a matter of collecting appropriate training data. The key metric for deployment would be recall at high precision: the system must rarely miss violations (high recall, since missed violations have legal/regulatory consequences) while maintaining a manageable false positive rate (precision high enough that human reviewers are not overwhelmed). The paper's pointing F1 of 39.9 on open-vocabulary queries suggests that current precision-recall tradeoffs may not yet meet production requirements without human-in-the-loop filtering, but the localization capability itself—absent in text-only moderation systems—is a qualitative improvement over existing approaches.