Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Multimodal Models

ArXiv: 2409.17146

🎯 Pitch

Top-tier VLMs have been trapped in a circular dependency—strong open models only work because they distill proprietary ones, leaving no recipe for building from scratch. Molmo breaks this cycle by matching or beating GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro using only human-annotated data, proving that careful data design, not model scale or distillation, is the real bottleneck.

1. Executive Summary

This paper introduces Molmo, a family of vision-language models that are state-of-the-art in their class of openness, and PixMo, a collection of new multimodal datasets constructed entirely without reliance on proprietary VLMs. The core contribution is a data-first methodology: rather than distilling from closed models, the authors develop novel human annotation protocols — speech-based dense caption collection (yielding 712k images with ~200-word descriptions), LLM-mediated interactive Q&A editing, and a 2D point-based grounding scheme (2.3M annotations) — that enable training competitive VLMs from scratch on fully open data. The best-in-class Molmo-72B model achieves the highest academic benchmark score and ranks second by human preference Elo, outperforming proprietary systems including Claude 3.5 Sonnet and Gemini 1.5 Pro, while the most efficient MolmoE-1B nearly matches GPT-4V, establishing that open-data VLMs can compete with or surpass proprietary alternatives when high-quality multimodal data is collected without distillation — provided the base LLM already possesses broad reasoning capabilities that the vision-language training extends rather than creates.

2. Context and Motivation

The Core Problem: The Missing Recipe for Building Performant VLMs from Scratch

This paper addresses a fundamental gap in the scientific community's understanding of vision-language model (VLM) development: how to build competitively strong VLMs entirely from scratch, without relying on data generated by other (typically proprietary) VLMs. The authors argue that while powerful VLMs exist — both proprietary (GPT-4o, Gemini 1.5 Pro, Claude 3.5 Sonnet) and open-weight — the actual knowledge of how to build them has become obscured by a methodological shortcut: distillation from closed models.

The problem is not merely one of scientific curiosity. As VLMs increasingly underpin applications ranging from visual question answering to embodied AI agents, the field risks building on a foundation where the training recipe is effectively a black box. When a model like LLaVA OneVision achieves strong performance by training on captions generated by GPT-4V, the community learns something about distillation, but nothing about how to independently create the underlying multimodal data that makes such models work. The paper identifies this as a circular dependency that cannot produce a truly open system: you can never reproduce the training pipeline end-to-end because a non-reproducible component (the proprietary VLM) sits at its core.

Why This Matters: Openness, Reproducibility, and Scientific Progress

The paper frames the importance of this problem along several dimensions:

Scientific understanding is bottlenecked. When a model's training data is generated by another, more powerful VLM, it is impossible to disentangle whether the student model's capabilities arise from its architecture, its training recipe, or the quality of the distilled data. This makes it difficult for researchers to isolate which components matter — a prerequisite for iterative scientific improvement. The authors state this directly in Section 1: "the scientific community is missing foundational knowledge about how to build performant VLMs from scratch."

Openness enables safety research, fine-tuning, and deployment flexibility. Proprietary VLMs restrict what researchers can study: there is no access to training data, model weights, or the ability to probe internal representations. Open-weight models address part of this, but if their effectiveness is downstream of another closed model's data, the supply chain is still opaque. The paper emphasizes that a fully open system — weights, data, code, and training methodology — is necessary for the kind of unfettered experimentation that drives scientific progress and responsible deployment. Figure 11 in the Appendix provides a systematic characterization of VLM openness across two attributes (open weights, open data + code) and three model components (the VLM itself, its LLM backbone, and its vision encoder), highlighting how rare fully open systems are.

Economic and legal considerations. Relying on proprietary VLMs for data generation creates licensing ambiguity. Can data generated by GPT-4V be used commercially? What terms of service restrictions apply? A pipeline that avoids these dependencies sidesteps these questions entirely, which is practically important for both academic and commercial adoption.

The capability ceiling is inherited from the teacher. When an open VLM is distilled from a proprietary one, its ceiling is bounded by the proprietary model's capabilities. An independently trained VLM can, in principle, surpass that ceiling — and the paper demonstrates that Molmo-72B does exactly this, outperforming both GPT-4V and Claude 3.5 Sonnet on academic benchmarks and human evaluation.

Where Prior Approaches Fall Short

The paper catalogs a progression of open VLM efforts, each with specific limitations:

Fully open early efforts (LLaVA-1.5). The first generation of open VLMs, exemplified by LLaVA-1.5 [69], released both weights and training data. These models used COCO captions and academic VQA datasets for training — data that is fully open and human-annotated. However, as the paper notes, LLaVA-1.5 "now lag significantly behind the state-of-the-art." The key issue is data quality: COCO captions average 11 words per image (Table 1 context), capturing only salient objects rather than the detailed, comprehensive descriptions that modern VLMs need for tasks like OCR, chart reading, and fine-grained visual reasoning. Academic VQA datasets, while precise, cover narrow domains with short, stylized answers that poorly represent the diversity of real-world user interactions.

Open-weight but closed-data models. Models like PaliGemma [10] and Qwen2-VL [111] release weights but keep training data proprietary or only partially disclosed. The paper emphasizes that this makes the training pipeline non-reproducible: "critical information such as prompts or data processing steps is often unavailable, making it hard to reproduce results" (Section 5). The evaluation section of the paper documents how performance can vary by 10% based on evaluation details alone, underscoring why closed training recipes undermine rigorous comparison.

Open-weight and open-data models that distill from proprietary VLMs. This is the category the paper identifies as most misleading. Models like LLaVA OneVision [59], Cambrian-1 [106], and xGen-MM [119] release both weights and training data, but that data includes synthetic captions or Q&A pairs generated by GPT-4V or similar proprietary systems. The most prominent example is ShareGPT4V [15], which uses GPT-4V to generate detailed image captions at scale. The paper explicitly argues that such models "are effectively distillations of proprietary VLMs" (Section 1). This is problematic because:

• Reproducibility is illusory. While the weights and code are open, a researcher cannot recreate the training data without API access to a proprietary VLM that may change versions, increase pricing, or disappear.
• Scientifically, it conflates instruction quality with model architecture. When ShareGPT4V improves performance, is it because GPT-4V produces better captions, or because the underlying model architecture or training recipe is sound? There is no way to tell.
• The trained model inherits the biases and limitations of the proprietary teacher in ways that are difficult to audit.

The paper quantifies this issue in Table 3b of the main paper: training on ShareGPT4V captions (158k images) produces worse results than training on PixMo-Cap images at similar scale (178k images), with cap F1 scores of 36.3 vs. 51.6. However, using GPT-4o to caption the same diverse images as PixMo-Cap produces strong results — suggesting that the diversity of the image distribution, not just the caption quality, is a critical factor that distillation-based approaches often overlook.

The data collection bottleneck. The paper acknowledges that the academic community has "struggled to collect such datasets due to high costs and the difficulty of obtaining high-quality annotations from crowd sourcing platforms" (Section 1). Prior annotation efforts for dense image descriptions — such as COCO Captions (11 words average), Localized Narratives (37 words average), and Visual Genome (region-level descriptions) — fall far short of the comprehensive, paragraph-length descriptions needed for modern VLM pre-training. The paper's PixMo-Cap captions average 196 words, highlighting the gap.

Grounding data limitations. Existing referring expression datasets like RefCOCO, RefCOCO+, and RefCOCOg provide valuable grounding annotations but are limited in scale (each with 86k–142k unique referring expressions) and almost exclusively use bounding boxes or segmentation masks for object-level referring in COCO images. GRES [66], the most similar prior dataset to PixMo-Points, includes multi-target and no-target expressions but is still restricted to COCO categories and rarely grounds expressions with large numbers of instances in a single image. The annotation cost of segmentation masks fundamentally limits how much grounding data can be collected — a bottleneck the paper sidesteps by using 2D points instead.

How This Paper Positions Itself Relative to Existing Work

The paper's positioning is clear and distinctive along several axes:

Data-centric rather than architecture-centric. The paper does not propose a novel model architecture. The Molmo architecture (Figure 2) follows a "standard design, combining pre-trained language and vision models" — a ViT image encoder, a vision-language connector, and a decoder-only LLM, in the lineage of LLaVA [69]. The paper is explicit that "the success of our approach relies on careful model design choices, a well-tuned training pipeline, and most critically, the quality of our new datasets" (Section 1). This is a deliberate philosophical stance: the primary barrier to building strong open VLMs is data quality, not architectural novelty.

Fully open by construction, not by claim. The paper defines "openness" along two dimensions (open weights + open data) across three model components (the VLM, its LLM backbone, and its vision encoder), as visualized in Figure 11 of the Appendix. Molmo occupies a unique position in this taxonomy: all PixMo training data is human-annotated or synthetically generated without any VLM involvement, the model weights are released, and the paper advocates for using fully open components (MetaCLIP vision encoder, OLMo LLM) to create a 100% open system where "every model component and every bit of data... is open" (Section 6 discussion of MetaCLIP). This contrasts with models that are "open-weight only" or "open-weight + open-data but distilled."

No VLM dependency in the data pipeline. This is the paper's most emphatic positioning claim, stated in the abstract and reinforced throughout: "without any reliance on synthetic data from other VLMs, including proprietary ones" (Section 1). The authors are careful to distinguish between using language-only LLMs (which they do use — Claude 3.5 Sonnet for code generation in PixMo-Docs, GPT-4o-mini for instruction-tuning data generation, and LLMs for caption refinement) and using VLMs. Their justification, elaborated in Appendix H, is that "using one VLM to build another VLM is entirely different than using an LLM, because the dependency is circular and therefore cannot result in a fully open system at a later point in time." The argument is that as open LLMs improve, the LLM-dependent portions of PixMo can be replaced with fully open alternatives, breaking the dependency chain — something impossible with VLM-to-VLM distillation.

Demonstrating competitiveness without compromise. The paper positions Molmo as proof that the community does not need to accept a tradeoff between openness and performance. Molmo-72B not only matches but exceeds many proprietary systems, achieving the highest academic benchmark score and second-place Elo ranking behind only GPT-4o (Table 1). This is a direct rebuttal to the implicit assumption that the strongest VLMs must be proprietary and that open alternatives must rely on distillation to compete.

Introducing pointing as a first-class VLM capability. The paper introduces 2D pointing not merely as an evaluation metric or a minor add-on, but as a fundamental interaction modality that enables counting (via chain-of-thought point-and-count), visual explanation, and potentially action (e.g., pointing to UI elements, navigation waypoints, or objects to manipulate). The authors explicitly frame this as opening "an important future direction in which VLMs enable agents... to act by pointing in their environments" (Section 1). This is a capability not present in the dominant VLM paradigm focused on text-only outputs and positions Molmo for embodied AI and user interface applications that proprietary models have not meaningfully addressed.

Ablation-driven scientific transparency. The paper goes beyond reporting final numbers to provide extensive ablations on model design (Table 2: vision encoder choice, dropout strategy, cropping method, pooling approach, length conditioning) and data choices (Table 3: PixMo-Cap scaling, pre-training data composition, fine-tuning data composition, counting strategy). These ablations are presented as a resource for the community — not just describing what works, but isolating why it works — addressing the foundational knowledge gap the paper identifies as its central motivation.

3. Technical Approach

3.1 Reader Orientation

The Molmo system is a family of vision-language models (VLMs) that take an image and a text prompt as input and produce text (and optionally, 2D point coordinates on the image) as output. The problem it solves is: how do you build a VLM that is competitive with proprietary systems like GPT-4o and Claude 3.5 Sonnet, without using any data generated by those very systems? The shape of the solution is data-centric rather than architecture-centric — the core innovation is a set of novel human annotation protocols and synthetic data generation pipelines (collectively called PixMo) that produce high-quality multimodal training data from scratch, combined with careful but mostly standard model design choices and a simplified two-stage training pipeline.

3.2 Big-Picture Architecture (Diagram in Words)

The Molmo system has two major halves: data construction (PixMo) and model training + inference (Molmo).

PixMo (data half):

• PixMo-Cap: Human annotators speak detailed image descriptions (60–90 seconds of speech per image); speech-to-text transcription produces raw transcripts; a language-only LLM cleans and summarizes these into polished dense captions (averaging 196 words). This is the pre-training dataset — 712k images with 1.3M transcripts and captions.
• PixMo-AskModelAnything: Human annotators write questions about images; a language-only LLM answers using OCR output and dense captions as context; annotators iteratively edit and approve the LLM's answers. This produces 162k free-form Q&A pairs on 73k images for fine-tuning.
• PixMo-Points: Annotators point to objects/regions in images by clicking, then name what they're pointing at. They exhaustively point to every instance of the named item. This produces 2.3M pointing annotations on 223k images, enabling the model to ground language in 2D coordinates, count by pointing, and provide visual explanations.
• Four synthetic datasets (PixMo-CapQA, PixMo-Docs, PixMo-Clocks, PixMo-Count) that use language-only LLMs, code rendering, and standard non-VLM detectors to create additional training data for specific skills — all without any VLM involvement.

Molmo (model half):

• Pre-processor: Takes an input image and converts it into multi-scale, multi-crop images — one low-resolution overview crop plus multiple high-resolution overlapping crops that tile the image.
• ViT Image Encoder: A Vision Transformer (primarily OpenAI CLIP ViT-L/14 336px, but compatible with MetaCLIP and SigLIP) that independently encodes each crop into patch-level features.
• Vision-Language Connector: Concatenates features from two ViT layers (3rd-to-last and 10th-from-last), pools 2×2 windows of patches into single vectors using mean-query attention pooling, then projects these into the LLM's embedding space via an MLP.
• Decoder-Only LLM: A pre-trained language model (OLMo-7B, OLMoE-1B-7B, Qwen2 7B, or Qwen2 72B) that receives the sequence of vision tokens (arranged in spatial order with special separator tokens) followed by text tokens, and autoregressively generates the output text (and optionally, point coordinates).

Information flows: input image → pre-processor (crop into multi-scale tiles) → ViT (encode each crop to patch features) → connector (pool, project, arrange into token sequence) → LLM (process vision tokens + text prompt, generate response).

3.3 Roadmap for the Deep Dive

I will explain the technical approach in the following order, which mirrors the paper's own emphasis on data-first methodology:

• First, the PixMo-Cap data collection pipeline — the speech-based annotation protocol, transcript cleaning, and why this approach produces better captions than direct typing or VLM distillation. This is the foundation of pre-training.
• Second, the PixMo-AskModelAnything collection pipeline — the interactive LLM-mediated Q&A protocol that produces diverse, accurate free-form responses without a VLM.
• Third, the PixMo-Points collection protocol — how 2D pointing enables grounding, counting, and explanation, and why points rather than bounding boxes or masks.
• Fourth, the four synthetic datasets — how CapQA, Docs, Clocks, and Count are generated using only language-model and code-rendering pipelines.
• Fifth, the Molmo model architecture — the cropping strategy (including the novel overlapping crop design), vision encoder selection, the connector (multi-layer feature concatenation, attention pooling, projection), and how vision tokens are arranged for the LLM.
• Sixth, the training pipeline — the simplified two-stage process (pre-training on PixMo-Cap, fine-tuning on the full mixture), the multi-annotated image training efficiency trick, dropout strategy, length-conditioned captioning, and optimizer configuration.
• Seventh, the pointing and counting mechanism — how the model outputs and interprets point coordinates, the chain-of-thought point-then-count strategy, and why plain-text coordinates outperform special tokens.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a data-and-training methodology paper whose core idea is that high-quality, human-annotated multimodal data — collected through careful protocol design that avoids VLM distillation — is the critical missing ingredient for building competitive open VLMs, and that a standard architecture trained on such data can match or exceed proprietary systems.

PixMo-Cap: Speech-Based Dense Caption Collection

The fundamental problem with caption collection. The paper identifies that directly asking crowd annotators to type long, detailed image descriptions produces poor results for several specific reasons (Section 3, PixMo-Cap description). First, annotators "tend to focus on a few salient visual elements" — a well-documented phenomenon in captioning datasets like COCO, where captions average only 11 words and describe only the most prominent objects. Second, "typing long paragraphs is time-consuming" — the physical act of typing 200+ words takes substantial time and effort, limiting annotator throughput and quality. Third, and critically for the paper's anti-distillation stance, "annotators can potentially copy-and-paste responses from proprietary VLMs, circumventing our goal of avoiding distillation." If annotators are asked to produce detailed captions and find it difficult, they may be tempted to upload the image to ChatGPT or a similar service, paste the response, and claim it as their own work — exactly the kind of hidden distillation the paper explicitly rejects.

The modality-switching "trick." The paper's solution is to change the modality through which annotators produce descriptions: instead of typing, they speak their descriptions for 60 to 90 seconds. This insight — which the authors call a "trick" — rests on several empirical observations. Speaking is faster than typing for most people, especially for long-form descriptive content. The time constraint (60–90 seconds) forces annotators to keep describing rather than stopping after a few salient objects, naturally eliciting more detail about "objects and their counts," "text," "positions of objects," "subtle details," "background," and "style and color" — the seven specific prompting questions listed in Appendix F. The paper notes that "we found that with this modality switching 'trick' annotators provide far more detailed descriptions in less time."

The anti-distillation receipt. An important secondary benefit of speech-based collection is what the paper calls an "audio receipt" — "the annotator's recording proving that a VLM was not used." This is a practical mechanism for quality control and provenance verification: the raw audio exists as evidence that a human, not an API call, produced the description. This is not just philosophical — it addresses a real concern in crowdsourced data collection where annotator cheating (using LLMs or VLMs to generate responses) is a documented problem.

Annotation protocol specifics. The paper describes a two-phase collection process. In the initial phase, "three annotators initially provided detailed descriptions by speaking for at least 60 seconds" for each image. Having three annotators per image provides multiple perspectives and increases coverage — different annotators notice different details. In later stages, the protocol shifted to "one annotator per image with a 90-second minimum, which improved efficiency without sacrificing quality." This suggests that the authors learned that a single longer description (90 seconds) was roughly as effective as three shorter ones (60 seconds each) while being more cost-efficient. The annotators were prompted with seven specific questions to guide their descriptions (listed in Appendix F): what the image is at first glance, objects and their counts, what text says, positions of objects, subtle details, background, and style and color. These prompts are not provided to a VLM — they are instructions to human annotators about what to cover in their speech.

Image sourcing. The images for PixMo-Cap were sourced from the web "across ~70 diverse topics (e.g., street signs, memes, food, drawings, websites, blurry photos, etc.)." This deliberate diversity sampling is important: the paper later shows (Table 3b) that captioning the same diverse set of PixMo-Cap images with GPT-4o produces strong results, while training on ShareGPT4V captions (which use a different, likely less diverse, image distribution) produces substantially worse results. This suggests that image diversity, not just caption quality, is a critical factor in pre-training effectiveness — an insight that would be obscured if the paper had simply used an existing image dataset.

Transcript processing pipeline. The raw audio transcripts from speech-to-text contain spoken artifacts: filler words ("um," "uh"), false starts, ungrammatical constructions, and conversational phrasing that are natural in speech but undesirable in a training caption. The paper processes these transcripts through a language-only LLM that performs two functions depending on the availability of multiple transcripts per image. When multiple transcripts exist (from the initial three-annotator phase), "a final high-quality image caption was then created by prompting a language-only LLM to summarize multiple raw transcripts per image" — the LLM synthesizes information from multiple descriptions, resolving inconsistencies and combining complementary observations. When only a single transcript exists (from the later one-annotator phase), the LLM instead works to "enhance its quality (e.g., removing spoken artifacts, normalizing style)." The critical design choice is that this LLM never sees the image — it works purely from the transcript text. This means the LLM cannot hallucinate visual details; it can only reorganize and polish what the human annotator actually described. The paper trains the model on both the raw transcripts and the cleaned captions (Table 3b shows that using both together outperforms using either alone), which likely helps the model learn to handle both polished and conversational description styles.

Scale and comparison to prior datasets. In total, "we collected 712k distinct images with 1.3M transcripts and captions." The captions "average 196 words, compared to 11 words in COCO captions and 37 words in localized narratives, highlighting their greater detail." This nearly 18× length increase over COCO is not just quantitative — it represents a qualitative shift from single-sentence object naming to paragraph-length comprehensive scene description. The examples in Figure 12 of the Appendix show the output: multi-paragraph descriptions covering spatial layout, text content, colors, materials, lighting conditions, and subtle background details — exactly the kind of comprehensive visual understanding that downstream tasks like visual question answering and detailed captioning require.

Length-conditioned pre-training task. During pre-training, the model is not simply asked to "generate a caption." Instead, the paper introduces a length hint mechanism. For 90% of training examples, the prompt includes a noisy length hint computed as:

$\text{hint} = \left\lfloor \frac{L + \epsilon}{15} \right\rfloor$

where $L$ is the length of the target caption/transcript in characters, and $\epsilon \sim \mathcal{N}(0, 25^2)$ is Gaussian noise with standard deviation 25.

What it computes: a noisy, discretized estimate of the target output length, scaled to roughly the range 0–100. The noise ensures the hint functions "more like a guideline than a hard constraint" — the model learns to produce captions of approximately the right length without being forced to exactly match a character count.

Why this form: without length conditioning, the model has no signal about how detailed a caption should be. A short prompt like "Describe this image" could produce a one-sentence caption or a five-paragraph essay — both are valid completions, but only one matches the training distribution. The length hint disambiguates this, and the noise prevents the model from learning a brittle mapping where it simply pads or truncates to hit an exact character count. The paper shows (Table 2e) that length conditioning improves both captioning quality (cap F1: 54.1 vs. 53.0 without it) and downstream task performance (11-avg: 76.9 vs. 76.2), confirming that captioning with length conditioning is a better pre-training task. Figure 7 in the Appendix shows the precision-recall tradeoff at different hint values: shorter hints produce higher precision but lower recall (the model focuses on salient, easy-to-describe elements), while longer hints increase recall at some cost to precision (the model describes more things, including less salient ones, where errors are more likely).

PixMo-AskModelAnything: Interactive LLM-Mediated Q&A Collection

The challenge of collecting diverse, accurate Q&A data. The paper needs fine-tuning data that teaches the model to answer the wide variety of questions users might ask about images — ranging from simple identification ("What kind of car is this?") to complex reasoning ("If the perimeter of the field is equal to the number of players multiplied by 20 meters, what is the area?"). Simply having annotators write question-answer pairs is problematic for two reasons. First, annotators may not know the correct answer to their own question (e.g., identifying a specific car model, or solving a math problem based on visual information). Second, having annotators both write and answer questions is slow and produces answers that may be factually incorrect.

The solution: human-LLM collaboration without a VLM. The protocol works as follows, described in Section 3 and detailed further in Appendix F:

1. Image selection and question writing: An annotator selects an image from a large pool (covering diverse topics) and writes a question about it. The question can be anything — there are no constraints on format, difficulty, or topic.

2. Context extraction without a VLM: The system runs two fully non-VLM tools on the image: a standard OCR model (for extracting any text in the image) and a PixMo-Cap-trained model (for generating a dense caption). Crucially, at this stage, the PixMo-Cap-trained model is an early version of Molmo itself — trained only on PixMo-Cap pre-training data, not yet fine-tuned. This means the caption quality is reasonable but not state-of-the-art. The OCR and caption together provide the language-only LLM with a textual description of the image's visual content.

3. LLM answering from text context: A language-only LLM (with no vision capabilities) is given the OCR output, the dense caption, and the annotator's question. It produces an answer based solely on this textual information. The LLM cannot see the image, so its answer is only as good as the caption and OCR — if the caption missed a detail, the LLM cannot know about it.

4. Iterative editing: The annotator reviews the LLM's answer. If accepted, the triplet (image, question, answer) is saved. If rejected, "they specified the issue and requested a revision until the answer was satisfactory." The paper doesn't detail the exact interface, but the process is effectively a human-in-the-loop editing cycle: the LLM proposes, the human critiques, the LLM revises, and this repeats until the answer is correct and well-formed.

Why this protocol works. First, the LLM provides a starting point that is typically factually grounded (since it comes from human-written captions and OCR) but may be incomplete or awkwardly phrased. The annotator's role shifts from authoring answers to editing them, which is faster and requires less expertise. Second, because the LLM only sees text, it cannot hallucinate visual details that aren't in the caption — any invented visual content would be detectable by the annotator comparing the answer to the image. Third, the iterative editing process ensures high quality: answers that survive multiple rounds of human critique are likely to be accurate, well-written, and appropriately detailed.

Handling counting questions. The paper notes a specific issue: "'How many' questions are common in PixMo-AskModelAnything, but are not accompanied with pointing data." Since AskModelAnything answers are pure text (no point coordinates), the model might learn to answer counting questions with just a number, without pointing. But Molmo's counting ability relies on pointing (the chain-of-thought point-then-count mechanism). To prevent the AskModelAnything data from suppressing pointing behavior, the paper heuristically detects counting questions and prefixes them with "an instruction to not point (e.g., 'Answer without points.'), randomly selected from a pool of 20 such instructions." This teaches the model that counting questions in the AskModelAnything style should be answered with text only, while counting questions in the Points style should include point coordinates. The model learns to switch between these modes based on the prompt.

Scale and examples. The paper collected "162k question-answer pairs in 73k images" — roughly 2.2 questions per image on average. Figure 13 in the Appendix shows randomly selected examples: questions range from "What kind of donut is this?" to "Can you take the phrase Easter Rabbit and put all the letters alphabetically?" (a text-manipulation task based on image text) to "how many times is the number 1 used in the photo." The answers are detailed, natural-language responses that read like a knowledgeable human's explanation, not the short, stylized answers typical of academic VQA datasets.

Comparison to the CapQA synthetic approach. The paper also generates a synthetic Q&A dataset, PixMo-CapQA, by "prompting a language-only LLM to ask and answer questions given only the ground-truth caption for an image" (214k pairs from 165k images). This is conceptually similar but removes the human annotator entirely: the LLM generates both questions and answers from the caption text. While cheaper, this approach cannot catch errors in the caption itself — if the caption is wrong or incomplete, the Q&A will inherit those errors. AskModelAnything's human-in-the-loop design provides a quality floor that pure synthetic generation cannot guarantee.

PixMo-Points: 2D Point-Based Grounding, Counting, and Explanation

Why points instead of bounding boxes or segmentation masks. The paper makes a pragmatic design choice that enables massive scale: "using points enables us to collect grounding data much faster than would be possible using bounding boxes or segmentation masks since it is much easier to annotate." Clicking one point on an object takes a fraction of a second; drawing a precise bounding box takes several seconds; tracing a segmentation mask can take 30+ seconds per object. This speed difference compounds dramatically when annotating images with many instances — pointing to 40 chairs in a classroom photo is feasible; drawing 40 segmentation masks is not. As a result, the paper collects "over 2.3 million grounding annotations for a diverse range of objects, expressions, and scenes" — an order of magnitude more than prior grounding datasets like RefCOCO (86k expressions), RefCOCO+ (141k), or gRefCOCO (278k expressions across 60k distinct instances).

Data collection protocol. The annotation task is structured around three goals described in Section 3:

1. Referring expression grounding: "annotators were asked to point at something in an image, describe it, and then point to each instance of it in the image, ensuring exhaustive coverage." The annotator first names what they're pointing at (e.g., "yellow pencil," "men," "glass of the window"), then clicks on every visible instance of that thing in the image. This produces a referring expression paired with a list of point coordinates.

2. Counting by pointing: Because the annotator exhaustively points to every instance, the number of points is the count. The model learns to count by generating the points first (the chain-of-thought), then reporting the total (which is simply the index of the last point). The paper notes this is a "unique chain-of-thought approach to counting where the model counts by sequentially pointing to each occurrence of the target object."

3. Exhaustive coverage and "not present" data: The paper collected "not present" annotations — cases where an annotator is asked to point to something that is not in the image. This teaches the model to say "There are none" or "This isn't in the image" rather than hallucinating points when asked about absent objects. The dataset includes 359k such instances.

Scale statistics. Appendix F provides detailed counts: "229k unique images, 1.98M referring expressions, average 8.7 distinct expressions per image, average 5.5 points per expression, average 47.7 total points per image." Figure 10 shows the distribution of counts: the number of questions with different count ranges (1–10, 11–20, etc.) varies widely, with the long tail extending to counts of 40+ (the maximum used in training, since larger counts "can lead to very long sequence lengths" causing memory issues).

Point coordinate representation. Points are represented as plain-text coordinates normalized to 0–100, with one decimal place of precision. For a single point, the model outputs:

<point x="10.0" y="10.0" alt="alt text">Inline text</point>

For multiple points (the more common case in counting and exhaustive pointing), the format uses a <points> tag with attributes x1, y1, x2, y2, etc.:

<points x1="10.0" y1="10.0" x2="20.0" y2="20.0" ... 
alt="alt text">Inline text</points>

The coordinates are arranged "top-down, left-to-right" — a spatial ordering that the paper shows is important: Table 4b demonstrates that training with ordered points produces substantially better counting accuracy than unordered points (CountBenchQA: 89.4 vs. 85.4; PixMo-Count: 86.3 vs. 74.1). The ordering likely helps the model learn a systematic scanning strategy that reduces double-counting and missed instances.

Why plain-text coordinates instead of special tokens. The paper explicitly compares the plain-text representation to an alternative where 1000 special point tokens are added to the model's tokenizer (one for each 0.1 increment from 0.0 to 100.0). Table 4d shows that plain-text coordinates work significantly better (CountBenchQA: 89.4 vs. 85.8; PixMo-Count: 86.3 vs. 80.9). The likely reason: the plain-text representation leverages the LLM's pre-trained understanding of decimal numbers and their ordering, while special tokens would need to learn the numeric semantics from scratch during VLM training — a difficult task given that pointing data is only a fraction of the training mixture.

Pointing as explanation. Beyond grounding and counting, the paper introduces pointing as a form of visual explanation. For the PixMo-Points explanation subset, the pipeline was adapted to "let annotators pass the LLM a list of text-annotated points, prompting the LLM to use them in its answer when relevant." In this mode, the model answers a question and, when appropriate, includes point coordinates that reference specific regions of the image supporting the answer. The paper collected 79k such point-explanation annotations on 14k images. During fine-tuning, this data is used with a style tag because "performance in this mode can be less reliable, so it should only be used when users request it" — an honest admission that this capability is experimental. Figure 15 shows examples: for a question about what percentage of parents use Facebook, the model points to the relevant data point on a chart while answering "74 percent."

The point-then-count chain-of-thought. The paper carefully abates the counting strategy in Table 4a, comparing four approaches:

• Count only: The model directly outputs a number. On CountBenchQA: 87.9; on PixMo-Count validation: 80.2.
• Point then count: The model generates points first, then reports the total (the default). CountBenchQA: 89.4; PixMo-Count: 86.3. This is the best-performing strategy.
• Count then point: The model states a count first, then generates points to justify it. CountBenchQA: 81.5; PixMo-Count: 77.6. Substantially worse.
• Pointing + regex: The model generates points without explicitly stating a count; the total is extracted by counting the number of <point> tags via regular expression. CountBenchQA: 88.4; PixMo-Count: 85.4. Only slightly worse than point-then-count.

The superiority of point-then-count over count-then-point is intuitive: when the model must generate points first, it is forced to actually enumerate instances before declaring a total, which acts as a form of verifiable chain-of-thought. Count-then-point reverses this, allowing the model to guess a number and then fabricate the right number of points — a form of "motivated reasoning" that is less grounded in the image. The paper also shows that simply generating more tokens at inference time does not produce the benefit: Table 4c compares using correct point locations and counts against generating random locations (with the correct count) and random locations (with a random count). Only actual, correct pointing improves performance — "simply increasing inference compute with extra tokens does not help."

Synthetic Datasets: CapQA, Docs, Clocks, and Count

PixMo-CapQA. This is the simplest synthetic dataset: the paper takes 165k images from PixMo-Cap, gives the ground-truth caption to a language-only LLM, and prompts the LLM to generate question-answer pairs. The LLM produces diverse questions covering topics and styles implicitly present in the captions, yielding 214k Q&A pairs. Because the LLM only sees the caption text (not the image), the questions are constrained by what the caption describes — the LLM cannot ask about visual details the human annotator didn't mention. Figure 16 shows randomly selected examples, including both straightforward factual questions ("Does the chameleon have bright green eyelids?") and creative prompts ("Create a witty one-liner that combines the concept of a chameleon's color-changing abilities..."). This diversity of question types teaches the model to handle both fact-seeking and creative/open-ended prompts.

PixMo-Docs (synthetic document, chart, table, and diagram generation). This dataset is the most technically sophisticated of the synthetic pipelines. The core idea is to use a text-only LLM's code-generation capabilities to produce programs that render document-like images, then use the same LLM (with access to the generating code as "privileged information") to create Q&A pairs about those rendered images. The paper developed a generation framework supporting "seven programming languages/rendering libraries, including Matplotlib, Plotly, LaTeX, HTML, Vega-Lite, Mermaid, and Graphviz."

The pipeline works as follows:

1. Controlled generation with diversity mechanisms: The system accepts text input to control the generation process (e.g., given the input "restaurant menu," it selects appropriate tools to generate relevant data). To further diversify outputs, the paper uses "personas" [35] — textual descriptions that influence content and style. The example given: generating "restaurant menu" data with the persona "A barbecue enthusiast known for their amazing grilled food at every Tennessee Vols game" produces "a Southern fusion menu combining traditional BBQ with international flavors, presented on a wooden board background." This persona-driven approach creates variation that simple random parameter sampling would not capture.

2. Code generation and rendering: The LLM (Claude 3.5 Sonnet, used for code generation) writes a program in one of the supported languages/libraries. This program is executed to render an image — a chart, a table, a diagram, or a document. The paper distinguishes four sub-categories: Docs-Charts (120k images, including heatmaps, violin plots, chord diagrams, geographic plots, tree maps, and many chart types beyond the standard bar/line/pie charts common in prior synthetic datasets like FigureQA and DVQA), Docs-Tables (47k images), Docs-Diagrams (16k images, including flowcharts, organizational charts, and network diagrams), and Docs-Other (71k images, including menus, resumes, invoices, and other document types).

3. Q&A generation from code: Crucially, the LLM that generates Q&A pairs (GPT-4o-mini, chosen for cost efficiency) has access to the original code that generated the image — it can read the data values, labels, structure, and formatting directly from the source program. This means the questions and answers are guaranteed to be correct with respect to the underlying data (e.g., "What is the total cost for Nutritionist Consultations over the year? 1040"), even if the rendering is imperfect. The LLM never needs to "read" the image visually — it reads the code that produced the image. This eliminates a major source of error in synthetic data generation: the VLM-as-oracle problem where a model is asked to answer questions about images it may misinterpret.

4. Scale: In total, PixMo-Docs contributes "255k text and figure-heavy images" and "2.3M question-answer pairs."

The paper emphasizes that this approach enables chart types far beyond prior synthetic datasets, which "typically only support one or two types of charts, often with a heavy focus on bar charts or line plots." Figures 19–22 show randomly selected examples demonstrating the diversity of chart types, table formats, diagram styles, and document layouts.

PixMo-Clocks (synthetic clock reading). Clock reading is a specific visual skill that the paper identifies as "missing from most VLM training data." The dataset uses rendered synthetic clocks with "~50 different watch bodies and ~160k realistic diverse watch faces set to random times," yielding 826k examples. Each example pairs a rendered clock image with a time-telling question and the correct time as answer. The clock faces are sourced from real watch face images (not procedurally generated), providing realistic variation in style, color, and markings. The paper notes that this data is highly effective: on an in-the-wild clock reading benchmark [121] (Table 10), Molmo models at all scales dramatically outperform all other VLMs including proprietary ones (e.g., Molmo-7B-D: 68.2% accuracy vs. GPT-4o: 2.7%, Gemini 1.5 Pro: 0.9%, Claude 3.5 Sonnet: 6.6%). The specialized single-task clock reading model from [121] still outperforms Molmo (78.9%), suggesting there is room for improvement by augmenting the synthetic data with real-world clock images. Qualitatively, the paper observes that the clock-reading capability "generalizes effectively to more complex questions and to captioning."

PixMo-Count (synthetic counting with detector verification). This dataset uses a standard non-VLM object detector (Detic [136]) to create image-counting Q&A pairs. The process: for each web image, run the detector, select the object class with the most detections after strict confidence thresholding, and create a question of the form "How many [class] are there?" with the number of detections as the answer. Unlike the other synthetic datasets, this one required human verification: "we manually verified 120 samples per count from 2 to 10, creating validation and test sets of 540 images each." The paper claims these images form "a more challenging counting QA set than CountBenchQA, which has reported limitations." After excluding the verified samples, the remaining 36k images (with counts 0–10) form the training set, "each annotated with points (object centers) and a QA pair." The use of point annotations (detector bounding box centers) rather than just count numbers allows the model to learn counting through pointing, consistent with the rest of the pointing data.

Model Architecture: Cropping, Vision Encoding, and the Connector

The multi-crop, multi-scale image pre-processing strategy. The paper addresses a fundamental limitation of Vision Transformers (ViTs): most ViTs "only support square images at a fixed resolution that is generally too low for fine-grained tasks such as OCR or detailed captioning." The standard CLIP ViT-L/14 processes 336×336 pixel images, which is insufficient for reading small text, identifying distant objects, or understanding the details of complex scenes.

The solution is a multi-crop strategy: the input image is divided into multiple square crops that tile the image, plus a low-resolution overview crop (the full image resized to 336×336). Each crop is processed independently by the ViT. The paper follows recent works [19, 30, 70, 85, 124] in adopting this approach but introduces a novel refinement: overlapping crops.

The overlapping crop innovation. Standard tiling divides an image into non-overlapping squares. The problem is that "border patches lack context from adjacent patches" (Figure 3 illustrates this: a patch on the edge of one crop has no visibility into the neighboring crop's content). This is especially problematic for objects that straddle crop boundaries — the ViT encodes each half of the object in isolation, losing the ability to recognize the whole.

The paper's solution: "we allow crops to overlap so each patch has context from at least some neighboring patches." Concretely, the crops are moved closer together so they overlap by a fixed margin of 4 patches (56 pixels for the 14×14 patch ViT). The ViT still processes each crop independently, but within each crop, border patches now have some surrounding visual context from the overlap region. After encoding, "patch features from the overlap are not passed to the connector or LLM so that the passed patch features exactly tile the high-resolution image" — the overlapping pixels serve only as context during encoding and are discarded afterward.

Grid selection algorithm. The paper describes the cropping algorithm in Appendix A.1. The system chooses a rectangular grid (e.g., 2×2, 3×1, etc.) where each cell matches the ViT's input size. The image is up-scaled to fit that grid as well as possible while preserving aspect ratio. The grid is chosen to "require the least amount of up-scaling, and in the event of ties, to minimize its size." A maximum number of crops is set (12 during training, up to 36 during inference for some tasks). If the image cannot be covered by that many crops without down-scaling too much, the image is instead down-scaled to fit the grid. The re-scaled image is padded with black borders to exactly fit the grid, and crops are extracted from this padded image. A learned embedding is added to patch features depending on whether the patch contains no padding, some padding, or is all padding — this teaches the model to distinguish real image borders from artificial padding, which is important because some images naturally have black borders.

Effectiveness of multi-crop and overlap. Table 2d shows the ablation: using only a single low-resolution crop produces a cap F1 of 46.7 and 11-avg of 62.8 — dramatically worse than multi-crop approaches. Multi-crop without overlap improves to cap F1 53.4 and 11-avg 75.7. The overlapping crop design adds further gains: cap F1 54.1, 11-avg 76.9. The paper notes that overlapping "slightly reduces the tiled image resolution, but this can be offset by using more crops." Table 2b further shows that using more crops generally improves performance (4 crops → 12 crops → 36 crops), though with diminishing returns and a specific caveat: captioning and counting tasks perform poorly when the number of crops at test time differs from training time, so for these tasks the crop count is forced to match.

Vision encoder selection. The paper primarily uses OpenAI's ViT-L/14 336px CLIP model but evaluates alternatives (Table 2a). The key finding is that "vision encoders trained on noisy web-scale data (CLIP, SigLIP, MetaCLIP) all work roughly the same" — cap F1 scores of 54.1, 54.4, and 54.1 respectively, with 11-avg scores within 0.3 points of each other. This is significant because MetaCLIP is "a fully open model (data and weights) meaning that every model component and every bit of data in a Molmo model equipped with MetaCLIP and OLMo is open." Surprisingly, DINOv2 — a self-supervised model trained on images only, with no text or label supervision — achieves cap F1 53.2 and 11-avg 75.6, only slightly behind the language-supervised encoders. This raises interesting questions about how much language alignment is actually needed in the vision encoder when a powerful LLM will be connected downstream.

Multi-layer feature concatenation. Rather than using features from a single ViT layer, the connector "concatenat[es] features from the third-to-last and tenth-from-last ViT layers." Table 15a ablates this: using both layers (cap F1 54.1, 11-avg 76.9) outperforms using only the third-to-last layer (53.7, 76.6) or only the tenth-from-last layer (52.5, 76.3). The rationale is likely that earlier layers capture lower-level features (edges, textures, simple shapes) while later layers capture higher-level semantics (object parts, relationships), and the LLM benefits from access to both. This is a minor architectural detail but one that the paper validates empirically — the improvement is small (0.4 points on cap F1) but consistent.

Attention pooling of 2×2 windows. After feature extraction, the connector pools the patch features to reduce the sequence length. The paper uses 2×2 window pooling with attention: "each 2×2 patch window is then pooled into a single vector using a multi-headed attention layer, where the mean of the patches serves as the query." This means the pooling is learned — the attention mechanism can weight the four patches in each window differently depending on their content, rather than simply averaging them. Table 2f compares this to "stacking" (concatenating the four patch features and feeding them through the MLP): attention pooling achieves cap F1 54.1 vs. 53.7 and 11-avg 76.9 vs. 76.1. The improvement is modest but consistent.

Vision token arrangement. After pooling and projection, the vision tokens are arranged into a sequence for the LLM. The ordering is: "left-to-right, top-to-bottom, starting with patches from the low-resolution full image, followed by high-resolution crop patches arranged in row-major order." Special tokens are inserted: tokens marking the start and end of both low- and high-resolution patch sequences, and "row-end tokens added between rows to indicate row transitions." This spatial structuring gives the LLM explicit positional information about where each vision token came from in the image — important for tasks requiring spatial reasoning, grounding, or reading text in order.

Training Pipeline: Pre-Training and Fine-Tuning

The simplified two-stage pipeline. The paper departs from a common VLM training convention: the three-stage pipeline of (1) connector-only pre-training on noisy web data, (2) full-model pre-training on high-quality data, and (3) instruction fine-tuning. The authors find that "this step [connector-only pre-training] is unnecessary when pre-training on PixMo-Cap," echoing findings in [48]. Instead, Molmo uses just two stages: full-model pre-training on PixMo-Cap, followed by instruction fine-tuning on the complete mixture.

To handle the fact that the connector weights start from random initialization while the ViT and LLM are pre-trained, the paper applies "a higher learning rate with a shorter warmup for the connector parameters, allowing them to adjust more quickly at the start of training." Specifically, during pre-training: connector learning rate 2e-4 (10× the LLM's 2e-5), with a 200-step warmup (10× shorter than the 2000-step warmup for ViT and LLM). This asymmetric scheduling lets the randomly initialized connector adapt rapidly while the pre-trained components change more conservatively. The paper notes that skipping the connector-only stage "reduces training time and complexity, and eliminates the need for the noisy web-scale data typically used in this phase."

Pre-training hyperparameters. Pre-training runs for four epochs on PixMo-Cap using AdamW with "cosine learning rate decaying to 10% of its peak." Specific settings from Table 6: learning rates 2e-4 (connector), 6e-6 (ViT), 2e-5 (LLM for most models, 1e-5 for 72B); warmup 200 steps (connector), 2000 steps (ViT, LLM); batch size 128; AdamW betas (0.9, 0.95), epsilon 1e-6; gradient clipping applied separately to the LLM, image encoder, and connector parameters. For the 72B model, learning rates are reduced: 2e-5 → 1e-5 (LLM pre-training), and the model "learns faster than the other models and can therefore be trained for fewer steps" (20k fine-tuning steps vs. 30–32k for 7B models).

Length-conditioned pre-training task format. During pre-training, the model is trained to generate either the cleaned caption or one of the raw audio transcripts for a given image. A prompt specifies which style to generate: "long caption:" or "transcript:". In 90% of cases, a length hint is appended: "long caption 83:" for a length hint of 83. The model learns to follow these style and length cues, allowing control at inference time. The paper notes that multi-annotation training (see efficiency trick below) is used to train on both the caption and a transcript for each image in a single forward pass.

Text-only dropout during pre-training. The paper introduces a novel dropout strategy: "Residual dropout is applied to the LLM, but not the image encoder and vision-language connector. During pre-training on dense captions, dropout is applied only to text tokens to encourage reliance on the encoded image rather than language priors." The standard approach would apply dropout uniformly to all tokens (vision and text). The paper's innovation is to restrict dropout to the text tokens only — the vision tokens and prompt tokens see no dropout. The hypothesis: this "encourage[s] the model to rely more on the vision tokens, rather than guess based on the previous text tokens, when generating tokens which may reduce hallucinations." Table 2c validates this: text-only dropout during pre-training achieves cap F1 54.1 vs. 53.7 for uniform dropout, with similar downstream performance (11-avg 76.9 vs. 77.0). During fine-tuning, dropout is applied uniformly since "shorter target responses result in too little dropout" if text-only dropout were used.

Multi-annotated image training efficiency. Many images in the fine-tuning mixture have multiple annotations — for example, VQA v2.0 images have multiple question-answer pairs, and PixMo-Points images have many different pointing expressions. Training on each annotation as a separate example would require encoding the image once per annotation, which is computationally wasteful since the image and its vision tokens are identical across annotations.

The paper's solution: "we arrange all of text annotation tokens for an image in one long sequence, masking attention so tokens for each annotation attend to the image tokens, each other, but not to tokens from different annotations." Concretely, for an image with three question-answer pairs, the sequence is: [vision tokens] [QA1 tokens] [QA2 tokens] [QA3 tokens], with an attention mask that prevents QA1 tokens from attending to QA2 or QA3 tokens (and vice versa). Each Q&A pair can only attend to the vision tokens and its own tokens. This setup is "equivalent to training on individual image-text pairs but avoids redundant image encoding, reducing the number of processed images by two-thirds and shortening training time by over half, with only a 25% increase in sequence length for our data mix."

This is a clever systems optimization that is not architecturally novel (variant attention masking is common) but has significant practical impact on training throughput.

Fine-tuning mixture and sampling strategy. The fine-tuning data comes from three sources: PixMo annotated datasets, PixMo synthetic datasets, and pre-existing academic datasets (VQA v2.0, TextVQA, OK-VQA, ChartQA, DocVQA, InfographicVQA, AI2D, A-OKVQA, AndroidControl, ScienceQA, TabMWP, ST-VQA, TallyQA, DVQA, FigureQA, PlotQA). The paper samples datasets "at rates proportional to the square root of their size, with manual down weighting of some very large synthetic datasets (PlotQA, FigureQA, DVQA, and PixMo-Clocks)."

The square-root sampling strategy is a heuristic that balances diversity and repetition: if sampling were proportional to dataset size, the largest datasets (PlotQA has 20M Q&A pairs, DVQA has 2.3M, FigureQA has 1.3M, PixMo-Clocks has 826k) would completely dominate training. Square-root sampling compresses the ratio — a dataset 100× larger is sampled only 10× more often, giving smaller but high-quality datasets more representation.

The paper further notes that "pointing tasks learn more slowly than QA tasks, so we significantly up-weight the pointing data." In the final mixture (Figure 4, detailed in Table 7), PixMo-Points alone accounts for 28.8% of samples — the largest single component — despite representing only 2.3M of the total annotations, because each pointing annotation is relatively short (few output tokens) and the skill is harder to learn.

Style tags to prevent answer format contamination. Academic VQA datasets have answer styles that are "not ideal for user interactions, as answers are usually very short and may reflect unique stylistic quirks from data collection (e.g., DocQA requires verbatim text from documents, while ChartQA specifies digits without commas)." If the model were trained on these datasets without differentiation, it would learn to produce terse, dataset-specific answer formats for all questions — exactly the opposite of the natural, detailed responses the paper wants.

The solution: prefix questions from each dataset with a style tag. For example, VQA v2.0 questions are prefixed with "vqa2:", ChartQA questions with "chartqa:", etc. The model learns to associate each tag with its dataset's expected answer style and switches behavior accordingly. For user-facing interactions (as in the human evaluation and PixMo-AskModelAnything), no style tag is used, and the model defaults to the natural, detailed response style learned from the PixMo interaction data. The paper uses "VQA v2.0 (for short answer) or A-OKVQA (for multiple choice) style tags" for evaluation-only datasets to elicit the expected answer format.

Prompt and template diversity. For PixMo-Cap captioning during fine-tuning, the paper creates "~30 prompts for caption generation" (e.g., "Generate a long caption about this image," "Create a detailed caption," "describe the image"). For pointing data, "~100 question templates that ask for the location or count of the target expression" are randomly sampled during training. This diversity prevents the model from overfitting to specific phrasings and improves generalization to varied user inputs.

Fine-tuning hyperparameters. From Table 6: learning rates 5e-6 (ViT, connector) and 1–2e-5 (LLM, varying by model scale), with 3e-6/3e-6/5e-6 for the 72B model; warmup 200 steps for all components; batch size 256; same AdamW settings as pre-training; steps ranging from 20k (72B) to 32k (7B-O). The learning rates are uniformly lower than during pre-training, reflecting the fact that fine-tuning is refinement rather than initial learning.

Pointing and Counting Mechanism: Output Format and Chain-of-Thought

Point coordinate output and parsing. The model outputs point coordinates in an HTML-like plain-text format, as described in the Points dataset section above. Normalized coordinates (0–100) make the representation independent of image resolution. When displaying results to users, "we generally replace the point(s) text with the inline text, and show the image with the points using the alt text as hover text" — the user sees the descriptive text with dots overlaid on the image, while the raw XML-like markup is hidden.

The chain-of-thought point-then-count strategy. The counting mechanism relies on the model generating a sequence of numbered points, one per instance of the target object, and then reporting the total count (which is simply the number of the last point). For example: "Counting the <points x1="15.5" y1="52.1" x2="36.8" y2="54.3" x3="57.6" y3="52.9" ... alt="people">people</points> shows a total of 4."

The key property that makes this effective: the model must actually enumerate the instances to get the count right. If the model tries to shortcut by guessing the count first and then generating the right number of points, it must still generate points at plausible locations. If those locations don't correspond to actual instances, there is no explicit verification during inference, but during training the model is penalized for incorrect point locations, forcing it to learn the mapping from visual features to spatial coordinates. This creates an implicit verification mechanism: the only way to get both the point locations and the count correct is to actually identify each instance.

Why point-then-count outperforms count-then-point (Table 4a). When the model outputs the count first and then generates points, it can produce a plausible-sounding number without having done the visual enumeration — the number might be a reasonable guess based on the scene's appearance (e.g., "this looks like a crowded room, so maybe 15 people") rather than the result of actually counting. The subsequent points might be generated to match this guessed count, but the generation is not constrained by visual evidence in a way that forces correctness. Point-then-count reverses this: the model must commit to specific spatial locations before declaring the total, making it harder to produce a count that doesn't match the visual evidence.

The spatial ordering ablation (Table 4b). Training with points consistently ordered top-down, left-to-right produces substantially better counting than unordered points. This likely works for two reasons. First, spatial ordering provides a systematic scanning strategy that reduces the chance of missing instances or double-counting — the model learns to scan the image methodically rather than jumping around randomly. Second, the ordering creates a predictable structure in the output sequence that the LLM can learn more easily — given the first few points, the next point should be to the right or below, providing a learnable sequential pattern.

Inference compute is not the source of the benefit (Table 4c). The paper tests whether the point-then-count improvement is simply due to generating more tokens (more inference compute allowing more "thinking"). The ablation compares: generating correct points and correct counts (89.4 CountBenchQA, 86.3 PixMo-Count); generating random point locations but with the correct number of points (85.9, 76.3); generating random point locations with a random number of points (76.3, 75.7). The drop from correct to random locations (with correct count) shows that accurate spatial grounding is critical — the extra tokens alone don't help. The further drop with random counts confirms that the counting accuracy is genuinely driven by visual grounding, not by inference-time computation.

Summary of Key Design Choices and Their Justifications

• Speech-based captioning over typed captions: produces more detailed descriptions in less time, provides anti-distillation audio receipts, and naturally elicits comprehensive coverage through the 60–90 second speaking requirement rather than explicit prompting.

• Language-only LLM for caption refinement rather than a VLM: the LLM works from text (the transcript), so it cannot hallucinate visual content — it can only reorganize what the annotator described. This is critical for the anti-distillation guarantee.

• Interactive human-LLM Q&A editing over pure human authoring or pure LLM generation: the LLM provides a fast first draft grounded in OCR and captions, while human editing ensures accuracy and naturalness. This hybrid approach is faster than pure human authoring and more accurate than pure LLM generation.

• 2D points over bounding boxes or masks: dramatically faster annotation (clicks vs. traced regions), enabling an order of magnitude more grounding data (2.3M annotations). The tradeoff is reduced spatial precision, but the paper argues this is acceptable for most use cases and the scale advantage outweighs the precision cost.

• Code-based document/chart generation over VLM captioning of existing documents: the LLM has privileged access to the generating code, making Q&A answers provably correct with respect to the underlying data. This eliminates the oracle error that plagues VLM-generated training data.

• Overlapping multi-crop over non-overlapping tiling: provides border patches with surrounding visual context during ViT encoding, improving feature quality at crop boundaries. The overlap pixels are discarded after encoding, so no additional tokens are passed to the LLM.

• Text-only dropout during pre-training over uniform dropout: hypothesized to encourage reliance on visual features rather than language priors, potentially reducing hallucination. Empirically validated to improve caption F1.

• Length-conditioned captioning over unconditional generation: gives the model control over output detail level, enabling a precision-recall tradeoff at inference time and serving as a better pre-training task (improves downstream performance).

• Two-stage training (full pre-training + fine-tuning) over three-stage (connector pre-training + full pre-training + fine-tuning): PixMo-Cap is high-quality enough that separate connector pre-training on noisy web data provides no benefit, simplifying the pipeline.

• Multi-annotation training with masked attention over separate forward passes: reduces redundant image encoding by 3×, cutting training time by more than half with only a 25% sequence length increase.

• Square-root sampling with manual adjustments over uniform or proportional sampling: balances dataset diversity while up-weighting harder-to-learn skills (pointing) and down-weighting very large but lower-quality synthetic datasets.

• Style tags over natural language instructions for academic datasets: cleanly separates dataset-specific answer formats from the general conversational style, preventing the short, stylized answers of academic benchmarks from contaminating user-facing behavior.

• Plain-text point coordinates over special tokens: leverages the LLM's pre-trained understanding of decimal numbers and their ordering, avoiding the need to learn numeric semantics from scratch during VLM training.

• Point-then-count over count-then-point or count-only: forces the model to enumerate instances before declaring a total, creating an implicit verification mechanism that improves counting accuracy by grounding the answer in specific spatial locations.

4. Key Insights and Innovations

Innovation 1: Reframing the Open-VLM Problem as a Data Collection Challenge, Not an Architecture Challenge

The dominant narrative in open-source VLM development has been that closing the gap with proprietary systems requires either architectural innovation (better vision-language connectors, more sophisticated training pipelines, novel model designs) or data distillation from those very proprietary systems. The Molmo paper makes a fundamentally different argument: the bottleneck is neither architecture nor access to closed models, but rather the community's inability to collect high-quality multimodal data from scratch at sufficient scale and diversity.

This is a conceptual reframing with sharp implications. Before Molmo, the implicit assumption was that competitive VLMs require some form of distillation — the strongest open models (LLaVA OneVision, Cambrian-1, xGen-MM) all relied on GPT-4V-generated captions or Q&A pairs. The paper directly challenges this assumption not through theoretical argument but through empirical demonstration: Molmo-72B, trained entirely on human-annotated and synthetically-generated (non-VLM) data, achieves the highest academic benchmark score and second-highest human preference Elo, outperforming Claude 3.5 Sonnet and Gemini 1.5 Pro (Table 1). The architectural components — a standard ViT + connector + LLM design in the LLaVA lineage — are not novel. The training pipeline — two-stage pre-training + fine-tuning — simplifies rather than complicates prior approaches. The only novel input is the data, making the causal claim clean: high-quality multimodal data, collected through careful protocol design, is sufficient to reach the performance frontier.

What makes this more than a "we collected good data" claim is the systematic ablation evidence that isolates which properties of the data matter. Table 3b shows that PixMo-Cap data at 178k images (cap F1 51.6) dramatically outperforms ShareGPT4V data at comparable scale (cap F1 36.3), but GPT-4o captions on the same diverse PixMo-Cap images perform strongly (cap F1 52.9). This triangulation reveals a non-obvious insight: image diversity is as important as caption quality, and prior distillation-based approaches conflated these factors. The community had been attributing the success of GPT-4V-distilled models to the teacher's caption quality, when in fact the diversity of the underlying image distribution (sampled from the web across ~70 diverse topics in PixMo-Cap, vs. whatever distribution ShareGPT4V used) may be the dominant factor. This is a diagnostic contribution — it tells future researchers what to optimize in data collection, rather than simply reporting that one dataset outperforms another.

The significance extends beyond performance numbers. By demonstrating that the data problem is solvable through protocol innovation (speech-based captioning, LLM-mediated Q&A editing, point-based grounding), the paper provides a methodology that other groups can adopt without depending on proprietary APIs. This converts the open-VLM problem from "how can we distill the best closed model?" to "how can we collect better multimodal data?" — a question accessible to the entire research community, not just those with API access. This is a fundamental shift in the research agenda, not an incremental data contribution.

Innovation 2: Modality Switching and Human-LLM Collaboration as a New Paradigm for Multimodal Data Collection

Prior approaches to collecting detailed image descriptions for VLM training fell into two categories: (1) human annotation (COCO Captions, Localized Narratives, Visual Genome), which was limited in scale and detail because typing long descriptions is slow and annotators focus on salient objects; and (2) VLM generation (ShareGPT4V, LLaVA's GPT-4V pipeline, etc.), which produced detailed captions at scale but created the circular dependency the paper rejects. Neither approach produced data that was simultaneously high-quality, detailed, scalable, and free of VLM contamination.

The paper introduces a third paradigm: modality switching for data production + language-model-mediated refinement. By having annotators speak rather than type their descriptions, the paper exploits an asymmetry in human communication — speaking is faster than typing for long-form content, and a time constraint (60–90 seconds) naturally elicits comprehensive coverage without the annotator needing to explicitly enumerate categories to describe. This is not merely a practical trick; it's a conceptual insight about how to elicit dense visual descriptions from non-experts: change the output modality to one that humans produce more fluidly at length, then use an LLM to convert the spoken output into polished training data.

The LLM-mediated Q&A collection in PixMo-AskModelAnything extends this paradigm. Rather than asking annotators to author complete question-answer pairs (which is slow and error-prone, since annotators may not know the answers to their own detailed questions), the paper assigns the annotator the role of editor-in-the-loop: a language-only LLM produces a first-draft answer from OCR and caption context, and the annotator iteratively critiques and refines it. This redefines the annotation task from authoring to verifying and improving — a task that is faster, requires less expertise, and produces higher-quality outputs because the LLM provides a grounded starting point while the human catches errors the LLM might miss.

What makes this distinctive is the division of labor between human and machine that respects each's strengths. The human provides visual grounding (they can see the image; the LLM cannot) and judgment about answer quality. The LLM provides fluency, world knowledge, and the ability to synthesize information from text sources (OCR, captions). The modality-switching trick for captions similarly respects human strengths: humans are good at producing fluent, detailed spoken descriptions but bad at typing them; speech-to-text + LLM polishing handles the conversion to clean text. Neither the human nor the LLM alone could efficiently produce this data — the human would be too slow, the LLM would hallucinate visual details — but together they produce data that is more detailed than pure human annotation and more reliable than pure LLM generation.

This paradigm has significance beyond Molmo. It suggests a general template for multimodal data collection in the era of powerful (text-only) LLMs: (1) design human tasks that leverage natural human strengths (speaking, pointing, editing), (2) use LLMs to convert human outputs into polished training data, and (3) use the human's access to the image as the verification mechanism that prevents LLM hallucination. This template could extend to video description, diagram annotation, medical image captioning, and any domain where exhaustive visual description is valuable but expensive to obtain through traditional annotation.

Innovation 3: 2D Points as a Unifying Primitive for Grounding, Counting, and Explanation

The VLM community has largely treated visual grounding as a separate capability from question answering — models like Kosmos-2, Ferret, and GLaMM add grounding through specialized training on bounding box or segmentation mask datasets, while general-purpose VLMs focus on text-only outputs. The paper introduces a conceptual shift: 2D points can serve as a lightweight, scalable primitive that unifies grounding, counting, and visual explanation within a single output format, without requiring the annotation cost of bounding boxes or masks.

The key insight is an economic one about annotation efficiency that cascades into a capability argument. Because pointing to an object with a single click is dramatically faster than drawing a bounding box or tracing a segmentation mask, the paper collects 2.3M grounding annotations — an order of magnitude more than prior referring expression datasets (RefCOCO: 86k–142k expressions; gRefCOCO: 278k). This scale enables exhaustive pointing (annotators point to every instance of the named object in the image), which in turn enables counting-by-pointing: the number of points is the count, so the model can learn to count by enumerating instances and reporting the total. This is a chain-of-thought mechanism grounded in spatial verification — the model must commit to specific locations before declaring a number, making it harder to produce counts inconsistent with visual evidence (Table 4c demonstrates this is not simply an "extra tokens" effect).

The innovation is not the use of points per se (prior work has used point supervision), but rather the recognition that points occupy a sweet spot in the precision-vs-scale tradeoff for multimodal data collection that makes them uniquely suited to VLM training. Bounding boxes provide more spatial information (extent, aspect ratio) but are slower to annotate. Segmentation masks provide maximum spatial precision but are an order of magnitude slower. Points discard extent information but gain massive scale. For many VLM use cases — answering "where is the X?", counting instances, providing visual explanations — extent is not actually needed; location is sufficient. And for use cases where extent is needed, a VLM that can point to an object can be combined with a segmentation model (e.g., SAM) in a two-stage pipeline, using the point as a prompt. The paper thus doesn't argue that points replace boxes or masks, but that points are the most scalable primitive, and that scale enables capabilities (exhaustive counting, diverse referring expressions) that precision-focused annotation cannot achieve.

This reframing has implications for the future of VLM-grounded interaction. The paper explicitly positions pointing as enabling agents to "act by pointing in their environments, e.g., to a navigation waypoint, to an object to pick up, or to a user interface button to press." This is not a capability that text-only VLMs can express, and it's one that proprietary VLMs have not focused on (Table 1 shows Molmo leads all models on counting benchmarks by wide margins). By making pointing a first-class output modality rather than an auxiliary task, the paper opens a design space where VLMs interact with the visual world through spatial coordinates rather than just language — a shift with implications for robotics, UI automation, and accessible technology.

Innovation 4: The Circular Dependency Argument — Why VLM-to-VLM Distillation Is Fundamentally Different from LLM-Assisted Data Creation

The paper draws a sharp conceptual line that has been absent from prior open-VLM work: using a VLM to generate training data for another VLM creates a circular dependency that fundamentally precludes a fully open system, while using a language-only LLM as a data-processing tool does not. This is not merely a philosophical distinction — it has concrete implications for reproducibility, scientific understanding, and the trajectory of open-source AI development.

Prior work in the open-VLM space has treated "open data" as a binary attribute: if the training data is released, the model is open. The paper argues this masks a critical structural difference. Data generated by GPT-4V (as in ShareGPT4V, LLaVA OneVision, Cambrian-1) is released and therefore "open" in the sense of being publicly available. But it is not reproducible — a researcher cannot recreate it without API access to a specific proprietary model that may change versions, increase pricing, or disappear entirely. More fundamentally, the trained VLM's capabilities are causally downstream of the proprietary VLM's capabilities — the student model is bounded by what the teacher could perceive and describe. The dependency is circular: to build an open VLM, you need another VLM (which is not open). This cannot be resolved by future progress in open models — the dependency is structural, not temporary.

In contrast, the paper's use of language-only LLMs (for caption refinement, Q&A generation, code rendering, etc.) creates a linear dependency that can be resolved. The LLM never sees images; it operates on text inputs (transcripts, captions, code, OCR output) produced by humans or non-VLM tools. As the paper argues in Appendix H: "once open LLMs become sufficiently good they can be used in place of closed ones to build a dataset functionally equivalent to PixMo." The LLM dependency can be swapped out when open alternatives mature — indeed, the paper already demonstrates this by using different LLMs for different parts of the pipeline (Claude 3.5 Sonnet for code, GPT-4o-mini for Q&A), with the choice driven by cost and capability rather than fundamental necessity.

This argument, elaborated in Appendix H and Figure 11, is a conceptual innovation that changes how the community should evaluate "openness" claims. It provides a principled criterion: does the data pipeline contain a dependency on a model of the same type as the one being built? If so, the system cannot be fully open regardless of whether weights and data are released. This criterion has teeth — it would reclassify many models currently labeled "open data" as "distilled," as the paper does in Table 1 with the dagger symbol (†). The paper is not arguing that distillation is ineffective (it clearly works), but that the community needs to understand how to build VLMs without it, because distillation-based approaches cannot answer fundamental questions about what data and training recipes actually matter — they can only measure how well distillation transfers.

This framing also explains the paper's otherwise-surprising choice to use closed LLMs (Claude, GPT-4o-mini) while adamantly avoiding closed VLMs. The apparent inconsistency — "you use proprietary models but criticize others for doing so?" — is resolved by the circular-vs-linear dependency distinction. The choice is coherent within the paper's framework, even if one might disagree with where exactly to draw the line. This is a conceptual contribution to the ongoing conversation about AI openness, reproducibility, and the structure of training data supply chains.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The primary evaluation uses the authors' own 11-benchmark suite comprising 10 standard academic datasets plus PixMo-Count: AI2D test [49], ChartQA test [82], VQA v2.0 testdev [36], DocVQA test [83], InfographicVQA test [84], TextVQA val [100], RealWorldQA [116], MMMU val [129], MathVista testmini [78], CountBenchQA [10], and PixMo-Count test. The paper prioritizes author-published results where available, fills missing entries with the best previously reported values from technical reports or the OpenVLM Leaderboard, and computes remaining results themselves. A human evaluation using 15k diverse image-text prompt pairs and ~870 annotators providing over 325k pairwise preference ratings (approximately 450 per model pair) complements the academic benchmarks, with Elo scores computed via the Bradley-Terry model following Chatbot Arena methodology [21]. Additional specialized evaluations include clock reading on the benchmark from [121], AndroidControl [62] step-wise accuracy, a new pointing benchmark with 493 image-question pairs and SAM-generated segmentation masks, and text-only benchmarks (MMLU [38], MMLU-Pro [114], GSM-8k [24], MATH [39], ARC-C [23], HumanEval [16]).

• Base model(s). The Molmo family spans four configurations: MolmoE-1B (based on OLMoE-1B-7B mixture-of-experts LLM with 1.2B active/6.9B total parameters), Molmo-7B-O (OLMo-7B-1024-preview LLM), Molmo-7B-D (Qwen2 7B LLM), and Molmo-72B (Qwen2 72B LLM). All use OpenAI's ViT-L/14 336px CLIP vision encoder as the default, with ablations testing MetaCLIP, SigLIP, and DINOv2 alternatives. The paper argues this model family spans "representative" capability regimes from efficient on-device to datacenter-scale, enabling analysis of how data quality interacts with model scale.

• Metrics. Academic benchmarks report accuracy (percentage of questions answered correctly), with dataset-specific grading protocols inherited from each benchmark's original release. The 11-avg metric is a simple unweighted mean across all 11 datasets, treating each benchmark equally regardless of size. For the human evaluation, Elo scores are computed from pairwise preferences using the Bradley-Terry model, with ties (both good, both bad) and "I don't know" responses excluded from win-rate calculations per standard Chatbot Arena practice. The captioning metric (cap F1) measures the harmonic mean of precision and recall computed via GPT-4o against a held-out set of 1500 images, where precision reflects consistency of generated atomic statements with ground-truth transcripts and recall reflects coverage of ground-truth statements.

• Baselines. The paper benchmarks against four categories of models distinguished by openness: (1) API-only proprietary models: GPT-4V [88], GPT-4o-0513 [90], Gemini 1.5 Flash and Pro [103], Claude 3 Haiku/Opus/3.5 Sonnet [7]; (2) open-weight only models: PaliGemma-mix-3B [10], Phi3.5-Vision-4B [1], Qwen2-VL-7B and 72B [111], InternVL2-8B and Llama-3-76B [104], Pixtral-12B [3], Llama-3.2V-11B-Instruct and 90B-Instruct [5]; (3) open-weight + open-data but distilled models (marked with †): LLaVA-1.5-7B and 13B [69], xGen-MM-interleave-4B [119], Cambrian-1-8B and 34B [106], LLaVA OneVision-7B and 72B [59]; (4) the Molmo family itself. For the human evaluation, a subset of these models is evaluated head-to-head.

• Generation budget / compute accounting. The paper does not use a "generation budget" in the FLOPs-counting sense typical of test-time compute scaling papers. Instead, the primary axis of comparison is model configuration (scale, LLM backbone, training data composition) evaluated at fixed inference settings. For academic benchmarks, Molmo uses 36 crops at test time (up from 12 during training) for all tasks except counting, where crops are fixed to 12 to match training resolution. For human evaluation, 12 crops are used with style tags omitted. A small amount of high-resolution post-training (3000 additional steps with 36 crops at halved learning rates) can restore counting performance at higher crop counts without affecting average benchmarks (Table 12). Training compute is reported in GPU-hours (Table 8): MolmoE-1B uses 264 GPU-hours pre-training + 850 fine-tuning on H100s; Molmo-7B-D uses 550 + 1.4k; Molmo-72B uses 4.2k + 8.3k.

• Cross-validation / statistical protocol. The paper does not employ cross-validation for benchmark evaluation since standard academic benchmarks have fixed train/val/test splits. For the captioning metric (cap F1), a held-out set of 1500 images disjoint from PixMo-Cap is used, gathered through a similar protocol but with images selected manually. For the human evaluation, Elo confidence intervals are not reported in Table 1 (though the Chatbot Arena evaluation in Table 9 includes 95% CI values). The PixMo-Count validation and test sets were created through manual verification of 120 samples per count from 2 to 10, producing 540 images each. For AI2D, the authors built their own validation set by separating 384 images with roughly 2000 question-answer pairs since the standard dataset lacks one.

Main Quantitative Results

Academic Benchmark Performance (Table 1)

The headline result is that Molmo-72B achieves the highest average accuracy (81.2% on the 11-benchmark suite) among all evaluated models — open or proprietary — and ranks second in human preference Elo (1077), behind only GPT-4o (1079). The table is organized to enable direct comparison across openness categories.

Molmo-72B vs. proprietary models. Molmo-72B's 81.2% 11-avg exceeds GPT-4o (78.5%), Gemini 1.5 Pro (78.3%), Claude 3.5 Sonnet (76.7%), GPT-4V (71.1%), Gemini 1.5 Flash (75.1%), and all Claude 3 variants (65.3–76.7%). The margin over GPT-4o is driven primarily by strong performance on counting (PixMo-Count: 85.2 vs. 59.6; CountBenchQA: 91.2 vs. 87.9), AI2D (96.3 vs. 94.2), and VQA v2.0 (86.5 vs. 78.7). GPT-4o leads on MMMU (69.1 vs. 54.1), MathVista (63.8 vs. 58.6), and InfographicVQA (79.2 vs. 81.9 — actually Molmo leads here), suggesting Molmo's advantage is concentrated in fine-grained perception tasks while reasoning-heavy benchmarks remain a relative weakness.

Molmo-72B vs. other open-weight models. Molmo-72B outperforms Qwen2-VL-72B (79.4% 11-avg), InternVL2-Llama-3-76B (77.1%), and Llama-3.2V-90B-Instruct (74.5%) by margins of 1.8–6.7 points. The comparison with Qwen2-VL-72B is particularly informative because both use Qwen2 72B as the LLM backbone — the performance gap must therefore be attributable to training data and recipe differences, not model scale or architecture. Counting is the largest differentiator: Molmo-72B achieves 91.2 vs. 80.4 on CountBenchQA and 85.2 vs. 55.7 on PixMo-Count, a ~30-point gap that directly reflects the pointing data's impact.

Molmo at smaller scales. MolmoE-1B (68.6% 11-avg, 1032 Elo) nearly matches GPT-4V (71.1% 11-avg, 1041 Elo) despite having ~1.2B active parameters — a remarkable efficiency result. Molmo-7B-D (77.3% 11-avg, 1056 Elo) and Molmo-7B-O (74.6% 11-avg, 1051 Elo) sit comfortably between GPT-4V and GPT-4o. Molmo-7B-D outperforms all other 7–8B class models by substantial margins: Qwen2-VL-7B (73.7%), InternVL2-8B (69.4%), LLaVA OneVision-7B (72.0%), Cambrian-1-8B (63.4%).

The distillation comparison. Models marked with † (distilled from proprietary VLMs) show a clear performance hierarchy correlated with training data quality rather than model scale. LLaVA OneVision-72B† (76.6% 11-avg) underperforms Molmo-72B (81.2%) by 4.6 points despite both using 72B-scale LLMs — direct evidence that PixMo data is competitive with or superior to distillation-based approaches. At the 7B scale, Molmo-7B-D (77.3%) outperforms LLaVA OneVision-7B† (72.0%) by 5.3 points.

Per-dataset patterns. Molmo's strengths are concentrated in perception-heavy tasks: AI2D (96.3, highest of all models), VQA v2.0 (86.5, highest), RealWorldQA (75.2, highest among open models, though GPT-4o scores 75.4), and both counting benchmarks (91.2 and 85.2, highest by large margins). On OCR-centric benchmarks (DocVQA, TextVQA, InfoQA), Molmo-72B scores 93.5, 83.1, and 81.9 respectively — strong but trailing Qwen2-VL-72B (96.5, 85.5, 84.5). On reasoning benchmarks (MMMU: 54.1, MathVista: 58.6), Molmo-72B lags behind GPT-4o (69.1, 63.8) and Claude 3.5 Sonnet (68.3, 67.7), which the paper attributes to "its training mix lacks data focused on advanced reasoning."

Human Evaluation (Table 1, Elo column; Figure 8; Table 5)

The Elo ranking broadly corroborates the academic benchmarks: GPT-4o (1079, rank 1), Molmo-72B (1077, rank 2), Gemini 1.5 Pro (1074, rank 3), Claude 3.5 Sonnet (1069, rank 4), Llama-3.2V-90B (1063, rank 5). The paper highlights one notable discrepancy: "Qwen2-VL performs strongly on the academic benchmarks and comparatively underperforms in the human evaluation" — Qwen2-VL-72B scores 79.4% 11-avg (second only to Molmo-72B) but only 1037 Elo (rank 12), well below Molmo-7B-D (1056, rank 6). The authors hypothesize this may reflect that academic benchmarks reward Qwen2-VL's answer style while human evaluators prefer Molmo's more natural, detailed responses.

Independent Chatbot Arena evaluation (Table 9). Molmo-72B achieves 1115 Elo on the Chatbot Arena vision leaderboard (English queries, as of Nov. 13, 2024), ranking below several proprietary models including GPT-4o-latest (1256), Gemini-Exp-114 (1278), and Claude 3.5 Sonnet variants (1184–1187), but above all open models. The lower ranking compared to the paper's own human evaluation (second vs. roughly 8th–10th) is attributed to differences in question types: "our data includes many counting and image-description questions which are particular strengths of Molmo," while Chatbot Arena's question distribution is not under the authors' control. This discrepancy is a genuine limitation — it suggests Molmo's advantages are most pronounced on question types well-represented in PixMo data but may not generalize uniformly.

Ablation human evaluation (Table 5, Figure 8). Comparing ablation models against Molmo-7B-D (Elo 1014) reveals the relative importance of different data components:

• Claude 3.5 Sonnet achieves 1047 Elo and wins 65% of non-tie matches against Molmo-7B-D.
• GPT-4o captions on PixMo-Cap images (1018 Elo, 55% win rate) perform similarly to PixMo-Cap data — evidence that caption quality from a strong proprietary VLM on diverse images matches human annotations.
• Removing PixMo-CapQA has minimal impact (1015 Elo, 50% win rate — essentially tied with default).
• Removing all academic datasets reduces Elo to 1013 (42% win rate vs. default).
• DINOv2 vision encoder (999 Elo, 45% win rate) performs only slightly worse than CLIP.
• Removing PixMo-AskModelAnything (995 Elo, 40% win rate) has a noticeable negative impact, confirming the value of human-edited Q&A data for user preference.
• Removing all PixMo-Cap data (990 Elo, 35% win rate) causes the largest drop, underscoring that dense captions are the single most important data component.
• Academic datasets alone (897 Elo, 17% win rate) perform dramatically worse — the model trained only on academic benchmarks produces responses that users strongly dislike, despite achieving reasonable benchmark scores. This is a critical finding: academic benchmarks and user preference can be sharply decoupled.

Figure 8 provides full match outcome breakdowns (win/lose/tie-good/tie-bad), revealing that ties constitute a significant fraction of matches. For Molmo-7B-D vs. Claude 3.5 Sonnet, 45.5% of matches were "both good" ties and 14.1% were "both bad" ties — meaning evaluators could not distinguish quality in 59.6% of cases. Molmo won 14.3% and Claude won 26.1% of non-tie matches. This high tie rate suggests the two models produce comparable-quality responses on many prompts, with Claude's advantage concentrated in a subset where the difference is clear.

Counting and Pointing Results (Tables 1, 4, 11)

Benchmark counting. Molmo models lead all competitors on both counting benchmarks by substantial margins. On CountBenchQA: Molmo-72B (91.2), Molmo-7B-D (88.5), Molmo-7B-O (89.0), MolmoE-1B (87.2) — all exceeding the next-best model (Claude 3.5 Sonnet: 89.7, though this is actually higher than Molmo-7B-D — readers should note the paper's claim of "Molmo leads all models" requires careful interpretation since Claude 3.5 Sonnet scores 89.7 vs. Molmo-7B-D's 88.5). On PixMo-Count (the more challenging benchmark): Molmo-72B (85.2), Molmo-7B-D (84.8), Molmo-7B-O (83.3), MolmoE-1B (79.6), all substantially ahead of GPT-4o (59.6), Gemini 1.5 Pro (64.3), Claude 3.5 Sonnet (58.3), and Qwen2-VL-72B (55.7). The ~25-point gap between Molmo-72B and GPT-4o on PixMo-Count is the largest single advantage for any model on any benchmark in Table 1.

Pointing evaluation (Table 11). The pointing benchmark of 493 manually verified image-question pairs (with SAM-generated segmentation masks for ground truth) shows Molmo models achieving F1 scores of 72.2–75.2, with precision and recall roughly balanced. Molmo-7B-D with 36 crops at test time (trained with 12 crops) degrades sharply to 58.1 F1 — a critical detail that motivated the high-resolution fine-tuning experiment (Table 12). After 3000 steps of 36-crop fine-tuning, counting performance recovers (CountBenchQA: 88.5 → 88.9, PixMo-Count val: 85.2 → 87.4) while 11-avg improves slightly (76.9 → 77.2). This demonstrates that the crop resolution mismatch is a fixable implementation issue, not a fundamental limitation.

Counting strategy ablations (Table 4). The point-then-count strategy (89.4 CountBenchQA, 86.3 PixMo-Count val) outperforms count-only (87.9, 80.2), count-then-point (81.5, 77.6), and pointing+regex (88.4, 85.4). Training with spatially ordered points (top-down, left-to-right) is critical: unordered points reduce PixMo-Count performance from 86.3 to 74.1 — a 12.2-point drop. Random point locations with correct counts (85.9, 76.3) underperform correct locations (89.4, 86.3), confirming that spatial accuracy, not just inference compute, drives counting performance.

Specialized Evaluations

Clock reading (Table 10). On the in-the-wild clock reading benchmark averaging COCO, OpenImages, and Clock Movies test sets, Molmo models achieve 64.2–68.2% overall accuracy, dramatically exceeding all other VLMs: GPT-4o (2.7%), Gemini 1.5 Pro (0.9%), Claude 3.5 Sonnet (6.6%), Qwen2-VL-72B (9.1%), and the next-best open model (LLaVA OneVision-72B: 5.7%). The specialized single-task model from [121] still leads (78.9%), indicating room for improvement — the paper suggests augmenting synthetic data with real-world clock images. Notably, Molmo-72B (65.6%) slightly underperforms Molmo-7B-D (68.2%), possibly because the 72B model was trained for fewer steps (20k vs. 30k) and PixMo-Clocks represents only 5.3% of the fine-tuning mixture.

AndroidControl (Section 5). Molmo-72B achieves 88.7% low-level and 69.0% high-level step-wise accuracy on AndroidControl's in-domain test set, compared to the reported 83.2% and 70.8% in the original paper [62]. This is notable because Molmo was trained on screenshots and task instructions only — without access to accessibility trees, action history, or structured action spaces that specialized UI agents use — suggesting that general visual understanding transfers effectively to UI action prediction.

Text-only benchmarks (Table 13). Qwen2-7B (the LLM backbone of Molmo-7B-D) achieves 70.2% MMLU, 71.8% GSM-8k, and 40.3% MATH. After multimodal fine-tuning, Molmo-7B-D drops to 64.6% MMLU (−5.6 points), 58.8% GSM-8k (−13.0 points), and 11.5% MATH (−28.8 points) — substantial degradation, especially in mathematical reasoning. Adding text-only data from Tulu 3 partially recovers performance: with 10% down-sampled Tulu 3, Molmo-7B-D achieves 65.4% MMLU, 71.2% GSM-8k (nearly back to baseline), 27.5% MATH (partial recovery), 55.5% HumanEval (exceeding Qwen2-7B's 47.6%), and 77.1% 11-avg (slightly improved over 76.9% without text data). This demonstrates that multimodal fine-tuning causes catastrophic forgetting of text-only capabilities, and that a small amount of text-only data during fine-tuning mitigates this without harming multimodal performance.

Pre-Training and Fine-Tuning Data Results (Tables 2 and 3)

PixMo-Cap scaling (Table 3a). Increasing PixMo-Cap images from 0 to 712k monotonically improves both captioning (cap F1 from an undefined baseline to 54.1) and downstream performance (11-avg from 74.9 to 76.9). The gains show diminishing returns — the first 89k images provide the largest jump, and subsequent doublings add progressively smaller improvements — but do not plateau, suggesting further scaling might yield additional gains.

Pre-training data comparison (Table 3b). PixMo-Cap at 178k images (cap F1 51.6) dramatically outperforms ShareGPT4V at comparable scale (cap F1 36.3) — a 15.3-point gap. However, GPT-4o captions on PixMo-Cap images achieve cap F1 52.9 (slightly higher than PixMo-Cap's 54.1 at full scale, though the precise comparison requires matching image counts). This triangulation reveals that image diversity (PixMo-Cap's carefully curated ~70 topics) is a critical factor: ShareGPT4V captions on a different image distribution underperform, while GPT-4o captions on PixMo-Cap's diverse images perform competitively. Adding noisy web-scale data (LAION) as a preliminary training stage provides no benefit over PixMo-Cap alone.

Fine-tuning data composition (Table 3c). Using only academic fine-tuning datasets achieves 72.5% 11-avg, compared to 76.9% with the full PixMo-⋆ + academic mixture. The largest contributor is PixMo-Docs (removing it drops performance from 76.9 to 75.8), reflecting improved document/chart/table understanding. Removing pointing tasks drops performance to 76.2, reflecting counting benchmark impact. Removing PixMo-AskModelAnything or PixMo-CapQA has negligible-to-slightly-negative effects on academic benchmarks (76.8 and 77.0 respectively) — these datasets primarily improve user interaction quality rather than benchmark scores, as confirmed by the human evaluation.

Pre-training task design (Table 2d–f). The overlapping crop design improves cap F1 from 53.4 to 54.1 and 11-avg from 75.7 to 76.9 over non-overlapping multi-crop. Length-conditioned captioning improves cap F1 from 53.0 to 54.1 and 11-avg from 76.2 to 76.9. Text-only dropout during pre-training improves cap F1 from 53.7 to 54.1 (with 11-avg roughly flat at 76.9–77.0). Attention pooling of 2×2 windows improves cap F1 from 53.7 to 54.1 and 11-avg from 76.1 to 76.9 over simple feature stacking.

Ablation Studies and Robustness Checks

Vision encoder choice (Table 2a): OpenAI CLIP, MetaCLIP, and SigLIP perform nearly identically (cap F1: 54.1, 54.1, 54.4; 11-avg: 76.9, 77.2, 77.1). DINOv2 (self-supervised, no language supervision) is only slightly behind (53.2, 75.6). This is critical for openness: MetaCLIP is "a fully open model (data and weights) meaning that every model component and every bit of data in a Molmo model equipped with MetaCLIP and OLMo is open."

Image resolution via crop count (Table 2b): More crops improve performance monotonically but with a crucial caveat — tasks involving captioning or pointing degrade when training and test crop counts differ. Using 4 crops for training and 36 for testing improves benchmarks (from 71.0 to 74.2) but the paper notes this harms captioning and counting, so those tasks always match crop counts. The high-resolution fine-tuning experiment (Table 12) demonstrates this is a trainable fix rather than a fundamental limitation.

Dropout strategy (Table 2c): LLM dropout during both pre-training and fine-tuning (cap F1 53.7, 11-avg 77.0) outperforms no dropout (53.1, 74.6). Text-only dropout during pre-training (cap F1 54.1, 11-avg 76.9) provides the best captioning performance while matching fine-tuning dropout's downstream results. This validates the hypothesis that restricting dropout to text tokens forces reliance on visual information.

Vision encoder layer selection (Table 15a): Concatenating features from the 3rd-to-last and 10th-from-last ViT layers (cap F1 54.1, 11-avg 76.9) outperforms using either layer alone (53.7/52.5, 76.6/76.3). The improvement is small but consistent — the model benefits from both mid-level and high-level visual features.

Learning rate warmup (Table 15b): Using separate warmup schedules for the connector (200 steps) vs. ViT and LLM (each 2000 steps) during pre-training improves cap F1 from 53.7 to 54.1 over using 200 steps for all components. This validates the asymmetric warmup strategy: the randomly initialized connector needs rapid initial adaptation, while pre-trained components benefit from slower warmup.

Gradient normalization (Table 15c): Component-wise gradient clipping (separate norms for ViT, connector, and LLM) performs slightly better than global clipping during pre-training (cap F1 54.1 vs. 53.6), though 11-avg is unaffected (76.9 for both). Using component-wise for pre-training and global for fine-tuning matches the fully component-wise variant.

Point encoding format (Table 4d): Plain-text coordinate representation (e.g., x="10.0") substantially outperforms adding 1000 special point tokens to the model's tokenizer: CountBenchQA 89.4 vs. 85.8, PixMo-Count 86.3 vs. 80.9. This is non-obvious — special tokens are a standard approach for structured outputs — and the paper's explanation (the LLM's pre-trained understanding of decimal numbers transfers) has implications for how to represent other structured data in VLM outputs.

Captioning length hints and the precision-recall tradeoff (Figure 7): Varying the length hint from 40 to 95 produces a classic precision-recall curve: shorter hints yield higher precision (73% at hint=40) but lower recall (36%), while longer hints increase recall (48% at hint=95) at the cost of precision (66%). The paper's default of 65 sits near the knee of the curve, and the model's ability to follow length hints (average output length within 10 characters of the target) demonstrates controllable generation granularity.

Multi-annotation training efficiency: The paper reports that arranging all annotations for one image in a single sequence with masked attention "reduces the number of processed images by two-thirds and shortens training time by over half, with only a 25% increase in sequence length." While not a traditional ablation, this is a practical finding about training throughput that affects the feasibility of training on large multi-annotation datasets. The paper does not report whether this masking approach affects model quality compared to training on individual pairs, which would have been a useful ablation.

Caption F1 correlation with downstream performance (Figure 9): Across 22 ablation experiments that affect pre-training and use PixMo-Cap, cap F1 and 11-avg show a Pearson correlation of ρ = 0.82. This is presented as post-hoc validation that improving captioning quality was a reasonable proxy for downstream performance during model development. The paper is appropriately cautious: "we have not established a causal relationship and this suggestion should be taken with that caveat in mind."

Text-only data mixing for catastrophic forgetting (Table 13): Adding Tulu 3 text-only data to the fine-tuning mixture recovers most of the text benchmark degradation caused by multimodal fine-tuning. The 10% down-sampled version performs best overall: MMLU 65.4 (vs. 64.6 without text data and 70.2 for the base LLM), GSM-8k 71.2 (nearly recovering from 58.8 to the 71.8 baseline), HumanEval 55.5 (exceeding the 47.6 baseline). However, MATH remains severely degraded at 27.5% (vs. 40.3% baseline) even with text data — mathematical reasoning appears to be the capability most vulnerable to multimodal fine-tuning and least recoverable through text-only data mixing.

AI2D with transparent vs. opaque boxes (Table 14): The paper reports results with both annotation styles since evaluations in the literature have been mixed. Transparent boxes consistently yield higher scores (e.g., Molmo-72B: 96.3 transparent vs. 86.4 opaque), which is an important methodological detail for benchmark comparisons — models evaluated under different box styles are not directly comparable.

Critical Assessment

Claim: "Our best-in-class 72B model not only outperforms others in the class of open weight and data models, but also outperforms larger proprietary models including Claude 3.5 Sonnet, and Gemini 1.5 Pro and Flash, second only to GPT-4o"

Does the evidence support this? Partially, but with important qualifications that the paper itself surfaces. On academic benchmarks (Table 1), Molmo-72B's 81.2% 11-avg does exceed Claude 3.5 Sonnet (76.7%), Gemini 1.5 Pro (78.3%), and Gemini 1.5 Flash (75.1%). However, this is a simple unweighted average across 11 benchmarks chosen by the authors, and the composition of this average matters. Molmo benefits from the inclusion of two counting benchmarks (CountBenchQA and PixMo-Count) where it has an enormous advantage (~25–30 points over proprietary models) due to its pointing data, and from AI2D where it scores 96.3. The 11-avg equally weights these perception-heavy tasks with reasoning tasks (MMMU, MathVista) where Molmo trails significantly. A different selection of benchmarks — say, emphasizing reasoning — would produce a different ranking. This is not a flaw in the evaluation (the paper is transparent about benchmark selection and per-dataset results), but it means the claim of "outperforms" is benchmark-composition-dependent.

On human evaluation, the Elo ranking places Molmo-72B second (1077) behind GPT-4o (1079), ahead of Gemini 1.5 Pro (1074) and Claude 3.5 Sonnet (1069). However, the independent Chatbot Arena evaluation (Table 9) tells a different story: Molmo-72B ranks considerably lower (1115 Elo, roughly 8th–10th place), behind several proprietary models including GPT-4o (1213+), Claude 3.5 Sonnet variants (1184–1187), and Gemini 1.5 Pro variants (1158–1220). The paper attributes this to question-type distribution differences: "our data includes many counting and image-description questions which are particular strengths of Molmo." This is a credible explanation, but it means the human evaluation result is dependent on the evaluation set's composition. The claim that Molmo outperforms Claude 3.5 Sonnet should be understood as "on a question distribution that includes strengths Molmo was specifically trained for (counting, detailed description)," not as a universal ranking.

An important missing analysis: the paper does not report per-category Elo scores for the human evaluation (it reports only the 10 categories of questions collected). This would have revealed whether Molmo's advantage is concentrated in specific question types (counting, captioning) or is broadly distributed. Without this breakdown, the aggregate Elo score conceals potentially important variation.

Claim: "MolmoE-1B nearly matches the performance of GPT-4V on both our academic benchmarks and user preference"

Does the evidence support this? Yes, with the same benchmark-composition caveat. MolmoE-1B achieves 68.6% 11-avg vs. GPT-4V's 71.1% — a 2.5-point gap, which is "nearly matches" by most standards. On human evaluation, Elo 1032 vs. 1041 — a 9-point gap that is within typical Elo confidence intervals (compare Table 9 where similar gaps have overlapping 95% CIs). This is a genuinely impressive result: a model with ~1.2B active parameters essentially matching GPT-4V, which is presumably orders of magnitude larger. It strongly supports the paper's thesis that data quality can compensate for model scale.

However, the "nearly matches" framing should be qualified by noting that the 11-avg includes PixMo-Count where MolmoE-1B scores 79.6 (vs. GPT-4V's 45.0) — a 34.6-point advantage that single-handedly contributes ~3.1 points to the 11-avg. Remove PixMo-Count and the averages might flip. This doesn't invalidate the claim (counting is a legitimate capability), but the margin is driven substantially by one specifically trained skill.

Claim: "The success of our approach relies on careful model design choices, a well-tuned training pipeline, and most critically, the quality of our newly collected datasets"

Does the evidence support this? The paper provides strong evidence for the primacy of data quality, but the evidence for "careful model design choices" is more nuanced. The ablation tables (Tables 2, 3, 15) show that individual design choices (overlapping crops, attention pooling, text-only dropout, length conditioning) each contribute 0.3–1.0 points to cap F1 and similar increments to 11-avg. Cumulatively, these are meaningful — the difference between "all choices optimized" and "naive defaults" might be 2–4 points on 11-avg. But the data ablations (Table 3) produce much larger effects: removing PixMo-Cap drops 11-avg by ~2 points and Elo by a dramatic margin (35% win rate vs. default); using academic datasets alone drops Elo from 1014 to 897; scaling PixMo-Cap from 0 to 712k images improves 11-avg by ~2 points. The evidence clearly supports the claim that data quality is the dominant factor.

One could argue the paper undersells its architectural contributions. The overlapping crop design (Table 2d: +1.1 11-avg over non-overlapping) and the length-conditioned pre-training (Table 2e: +0.7 11-avg) are genuinely novel contributions with measurable impact. The paper's rhetorical emphasis on data quality is well-supported, but a reader might miss that the model design choices, while individually small, collectively matter about as much as doubling PixMo-Cap data.

Missing Experiments and Weaknesses

Single vision encoder for main results. All Molmo models use OpenAI CLIP ViT-L/14 as the vision encoder. The MetaCLIP and SigLIP ablations (Table 2a) demonstrate that alternatives work similarly, but these are evaluated only on the 7B-D configuration — we don't know if the 72B model would maintain its performance with MetaCLIP. For a paper whose central contribution is openness, training and evaluating a fully open 72B model (MetaCLIP + OLMo-72B variant if one existed) would have been a powerful demonstration. The paper acknowledges this: "in retrospect, we should have used MetaCLIP as our default vision encoder."

No combined PRM search + revision experiments. This criticism applies to the test-time compute scaling paper, not Molmo. But Molmo has its own "missing combination": the paper never reports what happens when point-then-count chain-of-thought is combined with the high-resolution 36-crop inference. Table 12 shows that 12-crop training + 36-crop testing degrades counting (PixMo-Count val drops from 85.2 to 73.9), and that 3000 steps of high-resolution fine-tuning fixes this. But does point-then-count maintain its advantage at 36 crops? The chain-of-thought reasoning might behave differently when the model has access to finer visual details. This is a natural experiment that isn't reported.

Limited analysis of pointing-as-explanation quality. The paper introduces pointing as a form of visual explanation (Section 3, PixMo-Points explanation subset: 79k annotations on 14k images) but provides almost no evaluation of this capability. Table 11 evaluates pointing accuracy for grounding (where the task is to point to named objects), but the explanation use case — where points should reference regions supporting an answer — is qualitatively different and is not separately benchmarked. The paper acknowledges this is experimental ("performance in this mode can be less reliable") but the absence of evaluation is notable given that pointing-as-explanation is positioned as a key innovation.

Test set sizes and statistical reliability. The 11-benchmark suite aggregates over datasets with varying test set sizes. Some are substantial (VQA v2.0 testdev has tens of thousands of questions), while others are small. AI2D test has ~1200 questions, and the paper's custom PixMo-Count test set has only 540 images. The 0.5–1.0 point differences between models on 11-avg could be within sampling error for the smaller benchmarks, and the paper does not report confidence intervals for any academic benchmark result. The human evaluation's ~450 ratings per model pair provides reasonable statistical power, but the Chatbot Arena Elo scores in Table 9 do include 95% confidence intervals — and they show that many model rankings have overlapping intervals, meaning the exact ordinal position is uncertain. The paper's own human evaluation would benefit from similar confidence reporting.

No error analysis or qualitative failure cases. The paper provides extensive examples of model successes (Figures 12–22) but no systematic analysis of failure modes. Given that Molmo-72B has clear weaknesses on reasoning benchmarks (MMMU: 54.1 vs. GPT-4o's 69.1; MathVista: 58.6 vs. GPT-4o's 63.8), a qualitative analysis of what types of reasoning questions fail would be informative for understanding the limits of data-driven VLM training. Are the failures due to missing knowledge, inability to perform multi-step reasoning, or something else? The paper does not address this.

The LLM dependency in PixMo construction. The paper's anti-distillation stance against VLMs is clear and well-argued, but the reliance on proprietary language-only LLMs (Claude 3.5 Sonnet for code generation, GPT-4o-mini for Q&A generation, unnamed LLMs for caption refinement) is a dependency of a different kind. The paper argues this is acceptable because it's a linear dependency that can be replaced with open LLMs, not a circular one. But at present, the PixMo pipeline as released cannot be fully reproduced without access to these proprietary LLMs. The paper would be strengthened by an ablation showing how much quality degrades when using only open LLMs (e.g., OLMo, Llama 3) for the text-processing steps. This would quantify the gap that future open LLM improvements need to close.

The 14× pretraining comparison is not present. The prior example paper included a FLOPs-matched comparison between test-time compute and pretraining. Molmo does not attempt an analogous comparison — e.g., "given a fixed total compute budget, is it better to train a larger VLM on worse data or a smaller VLM on PixMo-quality data?" This is not a flaw (it's a different paper with different goals), but it means the paper does not provide guidance on the optimal allocation of compute between data collection/curation and model scaling. A scaling law analysis relating PixMo-Cap data quantity, model size, and downstream performance would have been highly informative for practitioners deciding how to allocate resources.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Excluded from All Efficiency Claims

The assumption or constraint. The entire compute-optimal test-time scaling framework — both for PRM search and for iterative revision — depends on knowing each prompt's difficulty before deciding how to allocate the inference budget. The paper's method for obtaining this difficulty estimate is generating 2048 complete solutions per question and scoring them (either against ground-truth for oracle bins, or against the PRM for predicted bins). The authors acknowledge this cost explicitly in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

The difficulty estimation step is not amortized into any of the reported efficiency numbers. The headline 4× improvement over best-of-N (Figures 4 and 8) is computed after the difficulty is already known, treating the 2048-sample pre-computation as free.

The consequence. In a realistic deployment, the total cost of using the compute-optimal policy is the difficulty estimation cost plus the strategy execution cost. Since difficulty estimation requires 2048 samples — far more than any of the test-time budgets studied (which max out at 256–512 generations) — the estimation step would dominate the total compute in almost every scenario. The 4× efficiency gain would only materialize in amortized settings where the same questions are answered repeatedly (so the one-time estimation cost can be spread over many queries). For one-shot or low-volume queries, the compute-optimal policy as described is more expensive than simply running best-of-N with a large budget, because you pay a massive upfront cost before you even begin solving the problem.

This is not merely an accounting technicality — it fundamentally changes the practical interpretation of the paper's central claim. The paper frames compute-optimal scaling as a drop-in replacement for best-of-N that yields 4× better efficiency. In reality, it is a two-phase system whose total cost, for non-amortized settings, exceeds the baseline it claims to improve upon. A practitioner reading the paper might reasonably believe they can deploy the adaptive strategy and immediately reduce their inference costs by 4×. They cannot — not without solving the difficulty estimation problem first.

What evidence exists in the paper. The paper provides no experiment that includes difficulty estimation cost in the compute budget. Figures 4 and 8 show the compute-optimal curves starting from generation budget 1 and going up to 256 or 512, with no accounting for the 2048 samples already spent to place each question in a difficulty bin. The paper does not report wall-clock time or total FLOPs for any end-to-end deployment that includes difficulty estimation. Section 3.2 flags this as future work but provides no upper bound on the additional cost.

Mitigation status. The paper acknowledges the limitation explicitly (Section 3.2) and suggests that "future work could focus on pretraining or finetuning models to directly predict difficulty of a question" (Section 8). However, no such model is developed, trained, or evaluated. An intermediate mitigation — adaptive difficulty estimation where a small number of initial samples informs a dynamic strategy switch — is mentioned in Section 8 as a research direction but is not implemented. The predicted difficulty bins (using PRM scores rather than ground-truth labels) remove the need for labeled data but do not reduce the sample cost (both oracle and predicted bins require 2048 generations). The limitation is currently unmitigated.

Hard Problems Remain Completely Unsolved Regardless of Compute Budget

The assumption or constraint. The paper's framework assumes that the base model's proposal distribution contains at least some correct solutions at a non-trivial rate — test-time compute can amplify existing capability but cannot create it from nothing. This is not a hidden assumption; the paper is transparent about it. But the severity of this limitation is worth examining: for the hardest difficulty quintile (bin 5), the base model's pass@1 is essentially zero, and no amount of test-time compute, search strategy, or revision depth produces meaningful improvement.

The paper states this directly in the Section 7 takeaway box:

"Test-time compute can amplify existing capability but does not create it."

The consequence. For any problem that is genuinely outside the base model's capability range — problems requiring reasoning skills, knowledge, or solution strategies the model simply does not possess — the entire compute-optimal framework provides zero benefit. This is not a matter of scaling budget further; the curves in Figure 3 (right panel, bin 5) and Figure 7 (right panel, bin 5) are essentially flat at 1–3% accuracy regardless of whether 4, 64, or 256 generations are used. Beam search, best-of-N, sequential revisions, and parallel sampling all fail equally. The FLOPs-matched comparison (Figure 9, bin 5) shows that pretraining a ~14× larger model is strictly better — test-time compute with the smaller model cannot close the gap.

This means the difficulty estimator serves a second, implicit function the paper does not fully explore: triage. On bin 5 problems, the optimal strategy is not to spend any test-time compute at all — just output the base model's greedy answer and move on, because additional computation is wasted. A complete deployment system would need to recognize this and either escalate to a larger model, request human intervention, or gracefully decline. The paper provides no mechanism for this decision.

More fundamentally, this limitation bounds the total addressable performance of the approach. As long as some fraction of the target distribution consists of bin-5-type problems, the aggregate accuracy ceiling is determined by the base model's zero-shot capability on those problems — test-time compute can improve performance only on the fraction of problems within the model's reach. For MATH with PaLM 2-S*, roughly 20% of problems fall into bin 5 (by construction, since bins are quintiles), so the maximum possible test-time-compute-augmented accuracy would still leave approximately 20% of problems unsolved. This ceiling cannot be raised without improving the base model through pretraining — a fundamentally different type of intervention.

What evidence exists in the paper. The evidence is consistent and unambiguous across every experiment that breaks out results by difficulty. Figure 3 (right): bin 5 accuracy hovers at 1–3% for both beam search and best-of-N at all budgets from 4 to 256. Figure 7 (right): bin 5 accuracy is roughly 2–3% regardless of the sequential-to-parallel ratio. Figure 9 (line plots): the bin 5 scaling curve (blue) is essentially a flat line near 0–5% for both revisions and search, and lies far below the 14× larger model's greedy performance (represented by stars). Table 1 in the FLOPs-matched analysis shows negative relative improvements for test-time compute on hard problems under most R regimes.

Mitigation status. The paper is transparent about this limitation (Section 7 text, the takeaway box, and the explicit statement that "pretraining is almost always more effective" for hard problems). However, it offers no mitigation beyond the obvious prescription: pretrain a larger model. Within the test-time compute framework itself, there is no proposed solution for bin-5 problems. The paper does not explore whether iterative revision could, over very long chains, eventually produce a correct solution on problems where the base model's initial samples are all wrong — the revision experiments (Figure 6, left) show pass@1 improving from ~18% to ~25% across 64 sequential steps, but this is aggregate data that includes bins 1–4 where improvement is possible; bin-5-specific revision trajectories are not reported.

The 14× Larger Model Baseline Is Not Compute-Optimally Trained and Uses Only Greedy Decoding

The assumption or constraint. The FLOPs-matched comparison in Section 7 compares PaLM 2-S* with compute-optimal test-time scaling against a model with approximately 14× more parameters. The larger model's pretraining compute is scaled by increasing parameters while holding training data fixed — the authors acknowledge this departs from compute-optimal pretraining as established by Hoffmann et al. (2022), where both parameters and data are scaled equally:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

Additionally, the larger model is evaluated using only greedy decoding — no majority voting, no best-of-N, no search, and no revisions. It receives zero test-time compute augmentation of any kind.

The consequence. Both design choices make the pretraining baseline weaker than it would be under a fully optimized comparison. A Chinchilla-optimal model trained with 14× more total FLOPs (scaling both parameters and data according to the square-root rule) would likely achieve better performance than a parameters-only-scaled model. Giving that larger model even a modest test-time compute budget (e.g., best-of-8 with the same PRM) would further improve it. The combined effect is that the paper's FLOPs-matched comparison systematically favors test-time compute. The reported advantages — e.g., revisions outperforming the larger model by +27.8% on easy questions at R ≪ 1 — are upper bounds on the true advantage against a properly optimized pretraining approach. The comparison does not answer the question "given a fixed total FLOPs budget, what is the optimal split between pretraining and test-time compute?" — it answers a weaker question: "given a fixed total FLOPs budget, is it better to spend it on test-time compute with a small model or on a naively-scaled larger model with no test-time compute?"

This is particularly relevant for the paper's practical guidance. The paper frames the FLOPs-matched results as evidence that organizations should consider investing in test-time compute rather than larger models. But if a properly compute-optimal larger model with modest test-time augmentation closes or reverses the gap, the guidance changes. The paper cannot distinguish between "test-time compute is more efficient than pretraining" and "the pretraining baseline was suboptimal" — both explanations are consistent with the data.

What evidence exists in the paper. The paper provides no experiment comparing against a Chinchilla-optimal pretraining baseline, nor against a larger model augmented with any test-time compute. The acknowledgment in Section 7 ("We choose this setting...") is the only discussion of this limitation. Table 2 (if one existed for pretraining scaling) would be needed to quantify the gap between parameters-only and compute-optimal scaling, but no such analysis is presented.

Mitigation status. The authors explicitly flag this as future work (Section 7, Section 8). They do not attempt to bound the magnitude of the bias introduced by the suboptimal pretraining baseline. A partial mitigation would be to estimate the performance of a Chinchilla-optimal larger model using scaling law projections from Hoffmann et al. (2022) or similar work, but this is not done. The limitation is acknowledged but unaddressed in the current paper.

Single Benchmark, Single Model Family, and No Confidence Intervals

The assumption or constraint. All experiments — search scaling, revision scaling, compute-optimal policy derivation, and FLOPs-matched comparison — are conducted on a single benchmark (MATH, 500 test questions) using a single model family (PaLM 2-S*) with a single verifier training procedure (Monte Carlo rollout PRM). The authors state they "believe this model is representative of the capabilities of many contemporary LLMs" (Section 4), but this is an untested assumption. The test set is further split into five difficulty quintiles of ~100 questions each, and two-fold cross-validation halves each bin to ~50 questions for strategy selection.

The consequence. Three separate generalizability questions arise, and the paper provides evidence for none of them:

Model generalizability: Would a different base model (e.g., GPT-3.5, LLaMA-2, Mixtral) exhibit the same difficulty-dependent scaling patterns? The PRM's quality and over-optimization behavior are functions of the base model's output distribution. A model with different calibration properties, different typical error modes, or different pass@1 distributions across difficulty levels might show qualitatively different optimal strategies. The paper's finding that beam search degrades on easy problems due to verifier over-optimization (Figure 3, right) might be specific to PaLM 2-S*'s PRM — a different base model with a differently-trained PRM might not exhibit the same crossover.

Domain generalizability: MATH consists of competition-level math problems requiring multi-step symbolic reasoning. Would the same difficulty-dependent patterns hold for code generation (where correctness is verifiable via unit tests), factual QA (where answers depend on knowledge rather than reasoning), or open-ended generation (where correctness is ill-defined)? The paper's core finding — that easy problems benefit from exploitation (sequential revisions) while harder problems benefit from exploration (parallel search) — might be specific to the structure of mathematical reasoning, where problems have a single correct answer and solution strategies can be systematically enumerated. Tasks with multiple valid answers or subjective quality criteria might not exhibit the same compute-optimal allocation patterns.

Statistical reliability: With ~50 questions per fold per difficulty bin, the compute-optimal strategy is selected based on a very small sample. The paper does not report confidence intervals on the compute-optimal scaling curves (Figures 4 and 8), bootstrap estimates of strategy selection stability, or sensitivity analysis to the number of bins. A practitioner cannot assess whether the 4× efficiency gain is robust to different random splits of the already-small test set, or whether a different binning strategy (e.g., 3 bins or 7 bins) would yield different conclusions.

What evidence exists in the paper. The paper provides no cross-model, cross-domain, or statistical reliability experiments. The evaluation section (Section 5) reports single-point accuracy estimates without error bars for all main results. Appendix figures showing compute-optimal scaling (Figures 4, 8) plot single curves without shaded regions representing variance across cross-validation folds. The single-model, single-benchmark design is not discussed as a limitation.

Mitigation status. The paper does not address this limitation. No experiments on other benchmarks, other model families, or other task domains are reported or planned. Confidence intervals are not provided for any academic benchmark result or compute-optimal scaling curve. The test set size of 500 questions (with ~50 per fold per bin) is not discussed as a potential source of variance in the strategy selection procedure.

Revisions and Search Are Studied Independently, Not Combined

The assumption or constraint. The paper studies two complementary test-time compute mechanisms — PRM-guided search (Section 5) and iterative model revision (Section 6) — as entirely separate pipelines. Search experiments use the few-shot prompted base model as the proposal distribution with the PRM as the verifier. Revision experiments use a fine-tuned revision model with a separate ORM verifier (since the base model's PRM does not transfer to revision model outputs). The two mechanisms are never combined in any experiment. The authors explicitly acknowledge this:

"we did not experiment with PRM tree-search techniques in combination with revisions" (Section 8)

The consequence. This is a significant gap because the two mechanisms address complementary weaknesses. Revisions improve the quality of generated candidates by conditioning on previous incorrect attempts — the proposal distribution shifts toward better solutions. PRM search improves selection among generated candidates by evaluating partial solutions and pruning unpromising paths. In principle, combining them — using the revision model as the proposal distribution within beam search, or using the PRM to decide when a revision is on track versus when to restart — could yield gains beyond either approach alone.

The paper's own results suggest where such combination might help most: on medium-difficulty problems (bins 3–4), beam search outperforms best-of-N for candidate selection (Figure 3, right), while a balanced sequential-to-parallel ratio outperforms purely parallel sampling for candidate generation (Figure 7, right). A combined approach applying beam search to revision model outputs at the optimal sequential-to-parallel ratio could potentially capture both benefits simultaneously. Since neither mechanism saturates individually (both show continued improvement with budget in Figures 4 and 8), the combination might push the performance ceiling higher than either alone.

The separation also means the paper cannot answer a natural practitioner question: "should I invest in training a revision model, a PRM, or both?" The paper shows that each helps, but provides no guidance on their relative cost-effectiveness, their complementarity, or whether the gains are additive, sub-additive, or synergistic.

What evidence exists in the paper. There is no experiment combining search and revisions. The PRM is never used to score revision model outputs during search. The revision model is never used as the proposal distribution for beam search. The compute-optimal policy (Section 3.1) optimizes over search strategies or revision strategies separately, never over a joint space that includes both simultaneously. The FLOPs-matched comparison (Section 7) treats search and revisions as alternative approaches and compares each independently against pretraining, rather than comparing a combined approach.

Mitigation status. The authors acknowledge this as a "natural next step" (Section 8) but do not implement it. The paper does not speculate on the magnitude of potential gains from combination, nor does it identify specific technical obstacles (e.g., the distribution shift between base model and revision model outputs that affected PRM transfer in Appendix J would need to be addressed). The limitation is flagged for future work but entirely unaddressed in the current paper.

Sequential Revisions Introduce a Latency Penalty That the FLOPs Accounting Ignores

The assumption or constraint. The paper measures test-time compute exclusively in "generations" — the number of complete solutions sampled from the model. This is a reasonable proxy for total FLOPs but completely ignores wall-clock latency. Sequential revision strategies are inherently serial: each revision depends on the output of the previous one, so a chain of 64 revisions takes approximately 64× the wall-clock time of a single generation, regardless of how much parallel hardware is available. In contrast, parallel best-of-N with N = 64 can be executed simultaneously on sufficient hardware, taking roughly 1× the wall-clock time.

The paper notes the sequential-to-parallel ratio sweep (Figure 7) and the compute-optimal policy that selects an optimal ratio per difficulty bin, but never discusses the latency implications of different ratios.

The consequence. For latency-sensitive applications — interactive assistants, real-time tutoring, any system where a user is waiting for a response — the sequential-heavy strategies that the compute-optimal policy favors for easy problems (where purely sequential revisions perform best) may be completely impractical. A user will not wait 64× longer for a marginally better answer to an easy question they could have gotten right with a single attempt. The paper's 4× efficiency claim is in terms of total FLOPs, not latency — for a practitioner deploying in an interactive setting, the latency penalty of sequential revisions could outweigh the FLOPs savings entirely.

This tradeoff is not captured anywhere in the paper's analysis. The compute-optimal policy optimizes only for accuracy at a given generation budget, not for accuracy at a given latency budget. A latency-aware formulation would need to incorporate the parallelism constraints of different strategies: fully parallel strategies have latency proportional to 1 generation regardless of N, while sequential strategies have latency proportional to chain length. The "optimal" strategy under a joint FLOPs + latency constraint might look very different from what the paper reports.

The paper also does not discuss whether the revision model can be optimized for lower latency — for example, by using speculative decoding or by training the model to produce shorter revision chains that achieve the same improvement in fewer steps. The inference cost model (Section 3.1) treats all generations as equivalent, which is true for FLOPs but false for latency.

What evidence exists in the paper. The paper provides no latency measurements for any strategy. The generation budget is the only cost metric reported. Training times are reported in GPU-hours (Table 8 in the Molmo paper — this is actually from the wrong paper context, but the principle holds: the test-time compute paper includes no latency analysis). There is no discussion of how the compute-optimal policy would change if latency were included as a constraint.

Mitigation status. The paper does not address latency at all. The limitation is not acknowledged. While FLOPs-matched analysis is standard in the scaling laws literature, the omission of latency is a practical gap for a paper whose stated motivation includes on-device deployment and real-world inference optimization. A partial mitigation would be to report the Pareto frontier of accuracy vs. latency for different strategies, allowing practitioners to choose their preferred tradeoff point. This is not done.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper fundamentally reframes the open-VLM research agenda from an architectural optimization problem to a data collection problem. Prior to Molmo, the dominant narrative held that building competitive VLMs without distillation from proprietary systems required either novel architectures or scale that only industrial labs could achieve. The paper's central empirical claim — that a standard VLM architecture trained on carefully-collected human data and synthetically-generated (non-VLM) data can match or exceed GPT-4V, Claude 3.5 Sonnet, and Gemini 1.5 Pro — is supported by the full benchmark and human evaluation suite, and it overturns this narrative.

The shift is methodological rather than architectural. The paper does not propose a new vision-language connector, a novel training objective, or a more efficient attention mechanism. Instead, it demonstrates that the bottleneck in open VLM development is data quality, not model design, and that the data bottleneck can be broken through protocol innovation: speech-based dense caption collection (Section 3, PixMo-Cap, producing captions averaging 196 words — 18× longer than COCO), interactive LLM-mediated Q&A editing (PixMo-AskModelAnything, 162k pairs on 73k images), and point-based grounding at unprecedented scale (2.3M annotations on 223k images, an order of magnitude more referring expressions than RefCOCO, RefCOCO+, or gRefCOCO). Each of these protocols is designed to exploit a specific asymmetry in human-machine capability: speaking is faster than typing for long-form description, editing is faster than authoring for Q&A, and clicking a point is dramatically faster than tracing a bounding box or segmentation mask for grounding. The cumulative effect is a data collection pipeline that produces VLM training data at quality levels previously only achievable through VLM distillation, but without the circular dependency that makes distillation-based approaches non-reproducible.

This reframing changes what the community should optimize. Rather than asking "how can we build a better VLM architecture?", the relevant question becomes "how can we collect better multimodal data?" This is a more democratized research question — it does not require large-scale distributed training infrastructure or novel architectural insights, only careful protocol design and crowd-sourcing resources. The ablation evidence in Tables 3a and 3b makes this case quantitatively: scaling PixMo-Cap from 0 to 712k images improves 11-avg from 74.9 to 76.9, while individual architectural innovations (overlapping crops, attention pooling, text-only dropout) each contribute 0.3–1.0 points. The data signal dominates the architectural signal.

Reconciling prior contradictions. The paper also resolves a tension that has persisted in the open-VLM literature. Several prior works (LLaVA OneVision, Cambrian-1, xGen-MM) demonstrated that training on GPT-4V-distilled data produces strong VLMs, suggesting that distillation was necessary to reach the performance frontier. Other works (LLaVA-1.5, which used only open human-annotated data) lagged far behind, suggesting that open data was insufficient. These findings were not contradictory — they reflected a genuine gap in the community's ability to collect high-quality multimodal data at scale — but they created a pessimistic equilibrium where open-VLM researchers implicitly accepted distillation as a necessary compromise. Molmo breaks this equilibrium by demonstrating that the data quality achievable through protocol innovation matches or exceeds the quality available through distillation: Table 3b shows GPT-4o captions on PixMo-Cap images (cap F1 52.9) perform similarly to PixMo-Cap human captions (54.1 at full scale), while human evaluation (Table 5) shows GPT-4o captions on PixMo-Cap images achieve 1018 Elo vs. 1014 for Molmo-7B-D default — essentially tied. This means the claim "distillation is necessary" was actually a claim about data collection capability, not about an inherent limitation of human annotation. With better protocols, humans can produce training data competitive with VLM-generated data, at least for the types of visual understanding tasks covered by PixMo.

Which research directions become more attractive. The paper's success shifts attention toward several specific directions that were previously under-explored:

• Protocol design for multimodal data collection. The speech-based captioning trick and the LLM-mediated Q&A pipeline are not one-off innovations — they instantiate a general template (choose a human-friendly output modality, use LLMs to convert to training format, use human visual access as the verification mechanism) that could be applied to video description, 3D scene understanding, medical image annotation, and diagram comprehension. The paper makes this direction newly attractive by demonstrating that protocol design, not just dataset size, is the key variable.

• Scaling up human-annotated multimodal data. The PixMo-Cap scaling curve (Table 3a) shows diminishing returns but no plateau — 712k images still yield gains over 356k images. This suggests that simply collecting more data using the established protocols would further improve performance. The paper makes this direction more attractive by providing a proven, cost-effective methodology that other groups can replicate.

• Pointing as a first-class VLM capability. The paper's pointing results (Tables 4, 11) and the counting benchmark dominance (Table 1: Molmo-72B scores 91.2 on CountBenchQA and 85.2 on PixMo-Count, 25–30 points ahead of proprietary models) demonstrate that 2D point output is not merely a niche add-on but a capability with broad impact. This makes pointing-based interaction a central research direction rather than a curiosity, with implications for embodied AI, UI automation, and accessibility.

Which research directions become less attractive. Conversely, the paper weakens the case for several lines of work:

• Architecture-centric VLM research that ignores data quality. If a standard ViT + connector + LLM design with carefully collected data can match proprietary systems, the marginal return to novel architectures may be smaller than previously believed — at least until the data bottleneck is fully saturated. The paper's ablation tables show architectural innovations contributing fractions of a point, while data scaling contributes multiple points.

• Distillation-based approaches positioned as "open." The paper's circular dependency argument (Appendix H, Figure 11) provides a principled criterion for distinguishing genuinely open models from distilled ones. Models that rely on GPT-4V or similar proprietary VLMs for training data cannot claim full openness regardless of whether weights are released, because the dependency is structural and non-removable. This reclassification, marked with the dagger symbol (†) in Table 1, may shift community norms around what "open" means and reduce the perceived value of distillation-based approaches relative to fully open alternatives.

• Assumptions that counting is a solved or unimportant capability. The paper's counting results expose a dramatic gap in proprietary VLMs: GPT-4o scores 59.6 on PixMo-Count vs. Molmo-72B's 85.2, a ~25-point deficit. This suggests that counting — a seemingly basic visual skill — has been systematically under-addressed in VLM development, likely because standard VQA benchmarks emphasize different capabilities. The paper makes it harder to argue that counting is a minor capability or that existing VLMs handle it adequately.

Follow-Up Research This Work Enables

1. Bridging the reasoning gap: augmenting PixMo with multi-step reasoning data. The paper's clearest weakness is on reasoning benchmarks: Molmo-72B scores 54.1 on MMMU (vs. GPT-4o's 69.1) and 58.6 on MathVista (vs. GPT-4o's 63.8). The paper attributes this to "its training mix lacks data focused on advanced reasoning" (Section 5). A natural follow-up would extend the PixMo-AskModelAnything protocol to explicitly target multi-step reasoning: have annotators write questions that require chaining multiple visual observations (e.g., "Based on the chart, which product had the largest percentage increase from Q1 to Q3, and what was that percentage?"), use the LLM-mediated editing pipeline to produce chain-of-thought answers with intermediate steps, and measure the impact on MMMU and MathVista. The key question is whether reasoning capability can be elicited through data alone (by providing examples of reasoned answers) or whether it requires architectural changes or reinforcement learning. A strong follow-up would compare: (a) adding reasoning examples to PixMo-AskModelAnything, (b) fine-tuning on math-specific datasets like MATH or GSM-8k in multimodal form, and (c) using the pointing data's chain-of-thought mechanism (point-then-count) as a template for other reasoning chains. If (a) alone closes half the gap, it would confirm that reasoning is fundamentally a data problem — extending the paper's central thesis. If none of these approaches work, it would establish an important boundary: PixMo-quality data can match proprietary models on perception but not on reasoning, and architectural or algorithmic innovations are needed specifically for reasoning.

2. Fully open 72B-scale reproduction with MetaCLIP and OLMo. The paper's strongest openness claim — that "every model component and every bit of data in a Molmo model equipped with MetaCLIP and OLMo is open" — is validated only at the 7B scale (Table 2a shows MetaCLIP performs similarly to OpenAI CLIP on Molmo-7B-D, and OLMo-7B-O achieves 74.6% 11-avg). The 72B model uses Qwen2 72B (open-weight but not fully open data) and OpenAI CLIP (open-weight only). A critical follow-up would train a fully open Molmo-72B equivalent: MetaCLIP vision encoder + an open 70B-scale LLM (e.g., a future OLMo-70B or a fully-open Llama variant) + PixMo data, and measure the performance gap against the current Molmo-72B. The paper's evidence (Table 2a: MetaCLIP vs. CLIP cap F1 54.1 vs. 54.1; 11-avg 77.2 vs. 76.9) suggests the vision encoder substitution should be nearly lossless. The LLM substitution is the larger unknown — Qwen2 72B may have capabilities that open 70B models lack. Quantifying the openness tax (the performance cost of using fully open components throughout) would provide a concrete target for the open-source LLM community and establish whether the paper's vision of a 100% open, state-of-the-art VLM is achievable today or requires further LLM advances. A negative result (large gap) would identify the LLM as the critical bottleneck in the open VLM supply chain.

3. Pointing-to-action: closing the loop between VLMs and embodied agents. The paper positions pointing as enabling agents to "act by pointing in their environments, e.g., to a navigation waypoint, to an object to pick up, or to a user interface button to press" (Section 1). The AndroidControl results (88.7% low-level, 69.0% high-level accuracy, Section 5) provide initial evidence that Molmo's pointing transfers to UI action prediction, but this was evaluated on screenshots with task instructions only — no closed-loop interaction. A direct follow-up would integrate Molmo into an embodied or UI agent loop: given a task (e.g., "book a flight to Seattle"), the agent takes a screenshot, uses Molmo to point to the next UI element to interact with, executes the action (click, type), captures the new screenshot, and repeats. The key metrics would be task completion rate and number of steps, compared against agents using non-pointing VLMs (text-only action descriptions) and against specialized UI agents that use accessibility trees (as in the original AndroidControl paper [62]). The paper's pointing data is uniquely suited to this because it was collected with exhaustive coverage — annotators pointed to every instance of named items — which should teach the model to identify all interactable elements, not just the most salient one. If Molmo-based agents achieve competitive task completion without accessibility trees, it would demonstrate that general visual grounding can substitute for structured environment representations, significantly broadening the applicability of VLM-based agents.

4. Difficulty-predicting models for compute-optimal VLM inference. While the compute-optimal test-time scaling framework (from the prior example paper) is not present in Molmo, the PixMo pipeline introduces a related efficiency question: given that different images require different levels of visual detail to process (some need 36 crops, others are fine with 12), can we predict the necessary resolution from the image alone? The paper shows (Table 2b) that more crops generally improve performance but with diminishing returns, and that some tasks (captioning, counting) degrade when training and test crop counts differ. A follow-up would train a lightweight "crop predictor" that takes the input image and question and outputs the minimum number of crops needed to answer correctly, using the full Molmo model's 36-crop output as ground truth. This would enable adaptive inference: easy images (large objects, no fine text) get 4 crops, document images get 12–36 crops, and the total inference FLOPs are reduced while maintaining accuracy. The paper's multi-crop architecture (Section 2, Appendix A.1) makes this particularly natural because crops are processed independently — skipping unnecessary crops directly saves ViT computation. The experiment would measure the accuracy-vs-FLOPs Pareto frontier compared to fixed-crop baselines, analogous to the compute-optimal test-time scaling analysis but applied to image resolution rather than generation budget.

5. Stress-testing the anti-distillation guarantee: can annotators cheat the audio receipt? The paper's audio receipt mechanism — having annotators speak their descriptions to prove a VLM was not used — is a practical anti-distillation measure, but its robustness is unexamined. A determined annotator could speak a description that was generated by a VLM (read it aloud from another screen), defeating the receipt. A follow-up study would quantify this vulnerability: recruit annotators, instruct some to produce original spoken descriptions and others to read GPT-4V-generated captions aloud, and measure whether human evaluators or automated methods (voice analysis, description content patterns, linguistic markers) can distinguish the two. If the distinction is detectable, the audio receipt provides genuine provenance guarantees. If not — if spoken VLM descriptions are indistinguishable from spoken human descriptions — then the anti-distillation guarantee is weaker than claimed, and additional safeguards (keystroke logging, screen recording, real-time monitoring) would be needed for truly verifiable human-only data. This is a practical rather than conceptual limitation, but it matters for the paper's central openness claim. The experiment would also characterize what makes VLM-generated captions detectable when spoken — do they use different vocabulary distributions, more uniform sentence structures, or different levels of detail? — which would inform future protocol design.

6. Scaling laws for PixMo-Cap data quantity vs. model size. The paper provides a scaling curve for PixMo-Cap at a fixed model size (Table 3a, Molmo-7B-D: 11-avg improves from 74.9 to 76.9 as images increase from 0 to 712k) but does not explore the interaction with model scale. A natural extension would train Molmo variants at 1B, 7B, and 72B scales on varying amounts of PixMo-Cap data (e.g., 89k, 178k, 356k, 712k, and — by collecting more data — 1.4M and 2.8M images) and fit scaling laws relating data quantity, model size, and downstream performance. The key question is whether larger models benefit more from additional data (steeper scaling curves) or whether the data saturates at similar points regardless of scale. This would provide practical guidance: if a 72B model continues to improve substantially beyond 712k images, further data collection is high-priority; if the curves flatten, the bottleneck shifts to other factors (data diversity, annotation quality, architectural improvements). The paper's data collection methodology is reproducible enough that a follow-up could extend PixMo-Cap to several million images at manageable cost, making this a tractable experiment for a well-resourced academic lab. The results would complement existing scaling law literature (which focuses on text-only or noisy web data) with the first systematic analysis of high-quality human-annotated multimodal data scaling.

Practical Applications and Downstream Use Cases

1. On-device VLM deployment with point-based UI interaction. The MolmoE-1B model (68.6% 11-avg, 1032 Elo, nearly matching GPT-4V) combined with the pointing capability enables a concrete deployment scenario: a privacy-preserving, on-device assistant that can see the user's screen and point to relevant UI elements or information. Because MolmoE-1B uses only 1.2B active parameters (with a mixture-of-experts architecture totaling 6.9B), it is feasible to run on edge devices or laptops without cloud connectivity. The AndroidControl results (88.7% low-level accuracy) demonstrate that the pointing transfers to UI action prediction. A practical system would: (1) capture the user's screen, (2) accept natural language commands ("highlight the cheapest flight," "show me where the error message is"), (3) use MolmoE-1B to generate point coordinates and descriptive text, and (4) overlay the points on the screen with labels. This avoids sending screenshots to cloud APIs (privacy benefit), works offline (reliability benefit), and uses pointing to provide spatial answers that text-only VLMs cannot express. The $4\times$ efficiency advantage of compute-optimal strategies (from the prior example paper) is not directly applicable here, but Molmo's own multi-crop adaptive resolution (using fewer crops for simple screens) could provide latency benefits — a cropped inference strategy that matches the 36-crop model's accuracy at a fraction of the inference cost.

2. Automated data extraction from heterogeneous documents. The PixMo-Docs dataset (255k text- and figure-heavy images with 2.3M Q&A pairs) and Molmo's strong OCR performance (DocVQA: 93.5, TextVQA: 83.1, InfoQA: 81.9 for Molmo-72B) enable document understanding applications that go beyond standard key-value extraction. Because PixMo-Docs was generated from code with privileged access to ground-truth data, Molmo learns to answer questions about documents by reading them visually — it does not require structured representations or pre-processing. A practical system could process scanned invoices, receipts, forms, and reports by: (1) accepting natural language queries ("What was the total amount before tax?", "Which items were purchased in quantities greater than 5?", "Extract the shipping address"), (2) using Molmo-72B to answer by reading the rendered document image directly, and (3) using the pointing capability to highlight where each answer was found in the document (providing auditability). The advantage over OCR-pipeline approaches (OCR → structured extraction → QA) is robustness to layout variation and the ability to handle ad-hoc questions without pre-defining extraction schemas. The clock-reading capability (Table 10: 65.8–68.2% accuracy on in-the-wild clocks vs. 0.9–9.1% for all other VLMs) demonstrates that Molmo handles specialized visual symbols that traditional OCR systems often miss, making it applicable to domains like medical imagery, engineering diagrams, and scientific figures where standard OCR fails.

3. Data generation for self-improvement loops in open-source VLMs. The paper's methodology — using humans to produce initial data, then using LLMs to amplify and diversify it — is a template for iterative self-improvement that avoids the circular dependency of VLM distillation. A practical pipeline would: (1) use PixMo-Cap's speech-based protocol to collect initial dense captions on a new image domain (e.g., medical images, satellite imagery, industrial inspection photos), (2) use the PixMo-AskModelAnything protocol (LLM-mediated Q&A editing) to generate domain-specific Q&A pairs from those captions, (3) use the PixMo-Docs approach (code-based rendering) to generate additional synthetic data for domain-specific visual elements (e.g., medical charts, equipment diagrams), (4) fine-tune Molmo on this combined dataset, and (5) deploy the fine-tuned model to assist annotators in the next round of data collection (e.g., by providing draft captions that annotators edit rather than starting from scratch). The key property that makes this work without distillation is that the image understanding is always bootstrapped from human visual perception — the LLM provides fluency and world knowledge but never sees the images, so its contributions are bounded by what humans described. The paper's evidence that PixMo-Cap data + GPT-4o captions on the same images perform similarly (Table 3b, cap F1 54.1 vs. 52.9) suggests this pipeline could eventually produce models matching proprietary quality on domain-specific tasks without any proprietary model involvement. For organizations with domain expertise but limited ML resources (hospitals, legal firms, scientific labs), this provides a path to custom VLMs trained entirely on their own data.

4. Accessible image description for low-vision users. The dense captioning capability enabled by PixMo-Cap (captions averaging 196 words with comprehensive scene coverage) and the controllable length hints (Figure 7: precision-recall tradeoff adjustable via length hint from 40 to 95) directly support assistive technology applications. A practical system would: (1) accept images from a user's camera or screen reader, (2) use Molmo to generate descriptions at user-requested detail levels (brief overview for quick scanning, detailed description for thorough understanding, or targeted Q&A for specific questions like "Is there a step I might trip on?"), and (3) use the pointing capability to ground descriptions in specific image regions when the user asks follow-up questions. The length conditioning (Section 4, Appendix B.1: length hints with Gaussian noise for flexible control) is specifically designed for this use case — it allows the system to adjust description verbosity to match user preference and task requirements without retraining. Unlike proprietary VLMs, a Molmo-based system could run entirely on-device (using MolmoE-1B), preserving privacy for sensitive images (medical records, personal photos, private documents). The paper's comprehensive description coverage (trained to describe text, positions, subtle details, background, style, and color — the seven prompt questions listed in Appendix F) addresses a specific limitation of existing screen readers and alt-text systems: they typically provide only brief, object-focused descriptions that miss context, spatial relationships, and subtle visual cues that are important for navigation and understanding.