ARM-Thinker: Reinforcing Multimodal Generative Reward Models with Agentic Tool Use and Visual Reasoning

ArXiv: 2512.05111

🎯 Pitch

Current reward models just guess—they never actually check the evidence. ARM-Thinker trains a 7B model to autonomously crop images, retrieve document pages, and verify constraints before judging, boosting accuracy by 16.2% and even beating GPT-4o.

1. Executive Summary

This paper introduces ARM-Thinker, an agentic multimodal reward model that autonomously invokes external tools—image cropping and zoom-in for fine-grained visual inspection, document page retrieval for cross-page evidence gathering, and instruction-following check tools for constraint verification—to ground judgments in verifiable evidence through an explicit think–act–verify loop, replacing static single-pass reward scoring. Evaluated on the newly introduced ARMBench-VL benchmark (1,499 questions spanning fine-grained perception, long-document QA, and instruction following) and built on Qwen2.5-VL-7B, ARM-Thinker achieves a +16.2% average improvement across reward modeling benchmarks, +9.6% on tool-use tasks, and outperforms GPT-4o on reward modeling accuracy, establishing that agentic tool invocation significantly enhances both judgment accuracy and interpretability of reward models when verification requires evidence grounded in retrieved or localized multimodal content.

2. Context and Motivation

The Core Problem: Existing Reward Models Cannot Verify Their Own Judgments

The fundamental challenge this paper addresses is that current multimodal reward models operate as passive, single-pass scorers that lack any mechanism to gather evidence before rendering a judgment. When an LLM generates an answer to a complex multimodal question—say, identifying a zip code buried in page 5 of a 14-page PDF, or determining whether a tiny visual detail in a high-resolution photograph matches a textual claim—a reward model must decide whether that answer is correct. Current reward models attempt this by looking at the question, the image, and the candidate response all at once, then emitting a score or preference. They never retrieve the specific document page where the evidence lives. They never zoom in on the relevant image region. They never run a word-count check to verify a claimed constraint.

This matters because reward models are the gatekeepers of alignment. They provide the training signal in RLHF (Reinforcement Learning from Human Feedback) that steers vision-language models toward helpful, accurate, and honest behavior. If the reward model reliably rewards fluent but factually incorrect answers—because it hallucinates evidence it never actually checked—the entire alignment pipeline fails upstream. The paper frames this failure mode sharply: reward models today suffer from systematic hallucination of rationales, position and length biases that favor certain response styles over correctness, and a complete inability to actively retrieve or verify cited content. They produce judgments grounded in nothing but the model's internal representations of a single forward pass.

This gap is especially acute for three categories of tasks that the paper identifies as increasingly common in production VLMs (Section 1):

• Multi-step evidence-grounded reasoning (e.g., long-document QA): correctness depends on sequentially retrieving, localizing, and cross-referencing visual and textual cues across multiple pages—not single-shot matching. A reward model that cannot turn to page 7 to check a claimed figure cannot reliably judge whether that claim is true.

• Fine-grained perception under tool-mediated transforms: when a model claims "the small text in the bottom-left corner says X," verifying that claim may require cropping and enlarging a tiny image region. A reward model without crop/zoom tools must guess based on a downsampled global view.

• Instruction-following verification: when a task requires "write between 120 and 150 words" or "include exactly three keywords," verifying compliance requires programmatic checks (word counting, substring matching) that are trivial for tools but error-prone for language models operating purely from text generation.

The paper's central motivating observation is that judgments should be conditional on accessible evidence, not just surface fluency (Section 1). Current reward models violate this principle by design—they have no "act" step, only "score."

Why This Problem Is Important

The practical significance is immediate. Reward models directly govern what behavior gets reinforced in the RLHF post-training pipeline that virtually all deployed VLMs undergo. As the paper notes (Section 2), "reward models are critical for aligning vision-language systems with human preferences." When these reward models are unreliable on the exact types of tasks—fine-grained visual understanding, long-document reasoning, constraint-following—that increasingly define VLM capabilities, the ceiling on model alignment is the verifier's reliability, not the model's potential.

There is also a theoretical significance. The paper frames multimodal judgment as fundamentally a planning problem (Section 1): the judge must decide when to think, which tool to call, how to parameterize it, and how to integrate intermediate results into a coherent causal chain. This reframes reward modeling from a regression/classification task into a sequential decision-making task—one that requires explicit credit assignment for tool-use decisions. This conceptual shift connects reward modeling to the broader agentic AI literature (ReAct, tool-augmented LMs) in a way that opens new research directions for verifier design.

The paper also highlights a timing argument: as VLMs are deployed on increasingly complex tasks (multi-page document understanding, high-resolution image analysis, multi-step instruction following), the gap between what models can potentially do and what reward models can reliably verify widens. "Tasks are becoming more cross-modal, open-ended, and fine-grained," the authors note in Section 1, and "evaluating correctness now depends on semantic understanding and grounding in evidence, rather than on brittle string matching against scarce or ambiguous ground truth." Closing this verification gap is urgent.

Where Existing Approaches Fall Short

The paper identifies three existing paradigms and explains why each is insufficient for complex multimodal reward modeling.

Rule-based verifiers. These are the simplest approach: check if the model's output contains a specific string, matches a regex, or falls within a numeric range. The paper cites DeepSeek-R1 (Guo et al., 2025) and Tülu 3 (Lambert et al., 2024) as examples. Rule-based verifiers work for math problems with unique numeric answers or constrained-format tasks, but they break down catastrophically on the multimodal tasks this paper targets. They are "brittle to paraphrase" (an answer expressing the same fact in different words is scored as wrong), "incapable of partial credit" (a nearly-correct answer gets zero), and "impractical when ground truth is subjective" (open-ended visual description has no single correct string). The paper does not dismiss rule-based verifiers entirely—they have their place—but argues they cannot be the primary verification mechanism for rich multimodal outputs.

Non-agentic generative reward models. This is the dominant paradigm and the paper's primary foil. Models like UnifiedReward-7B (Wang et al., 2025c), InternLM-XComposer2.5-Reward (Zang et al., 2025), and various proprietary systems take a (question, image, response) triple and output a score or pairwise preference in a single forward pass. The paper identifies several systematic failure modes:

• Hallucinated rationales (Li et al., 2023): the reward model generates a plausible-sounding explanation for its judgment that cites visual details without actually verifying them. The model may confidently claim "the zip code is 89104, as shown in the document" when that zip code appears nowhere in the image.

• Position and length biases (Dubois et al., 2024): reward models systematically prefer responses that appear first, last, or are longer, independent of factual correctness. This is a well-documented failure mode in LLM-as-judge evaluations.

• No retrieval or verification capability: the reward model has no mechanism to fetch specific evidence (a document page, a cropped image region) before judging. It must rely entirely on what is visible in the provided inputs and what its internal representations can infer.

• Misalignment between training and inference: most reward models are trained via pointwise scoring (how good is this response?) or pairwise preference (is A better than B?) using static datasets. At inference time, they face judgment problems that would benefit from interactive evidence gathering, but they have never been trained to decide when additional evidence would help or how to obtain it.

The paper emphasizes that non-agentic reward models "optimize broad coverage via pointwise scoring or pairwise preferences rather than evidence-grounded reasoning: they lack a think–act–verify loop, provide no credit assignment for tool decisions, and misalign training with inference behavior" (Section 1). This is a precise characterization: the training objective (predict a score or preference) does not incentivize the behaviors (retrieve, localize, verify) that would make judgments reliable.

Specialized visual reasoning models with tool use. A separate line of work has developed models that use tools—specifically image crop and zoom-in operations—for fine-grained visual perception. The paper cites DeepEyes (Zheng et al., 2025), Pixel Reasoner (Su et al., 2025), and Mini-o3 (Lai et al., 2025) as examples of this "think-with-images" paradigm. These models are not reward models—they solve visual QA tasks directly rather than judging others' answers—but they demonstrate that tool-mediated visual inspection dramatically improves accuracy on fine-grained perception benchmarks like V* Bench and HRBench.

However, the paper identifies two critical gaps in this line of work that prevent its direct application to reward modeling. First, these models are trained on curated tool-use demonstrations specific to a narrow set of operations (zoom, crop) and tasks (spatial reasoning, object localization). They do not generalize to the broader set of tools (document retrieval, instruction checking) that reward modeling requires. Second, and more fundamentally, these models are not designed to judge: they answer questions, not evaluate the quality of others' answers. The cognitive skill of verification—weighing evidence, comparing claims, assigning partial credit—is distinct from the skill of answering, and prior tool-use models have not been trained or evaluated for it.

Agentic reward modeling in the text-only domain. One recent work, Peng et al. (2025), proposed "Agentic Reward Modeling" for pure language tasks, integrating verifiable correctness signals (e.g., code execution, mathematical proof checking) into reward model training. The paper cites this as conceptual precedent but notes that the multimodal extension is fundamentally harder: visual evidence (document pages, image regions) requires spatial grounding, multiple modalities (text + image tool outputs), and a different set of tool interfaces than text-only verifiers.

How This Paper Positions Itself

ARM-Thinker is explicitly positioned as bridging the gap between passive reward scoring and active, tool-mediated verification for multimodal tasks (Section 1). The paper does not claim to invent tool use, agent loops, or reward models individually—all of these exist in prior work. Rather, it claims to be the first to unify them into a single system where the reward model can autonomously decide which tools to invoke, when to invoke them, and how to integrate the returned evidence into an interpretable judgment.

The positioning has several key aspects:

Extending the think-with-images paradigm to reward modeling. The paper frames its agent loop as "an extension of the think-with-images paradigm" (Section 3.1): the image crop and zoom-in tools realize iterative visual reasoning, but now in service of judging others' outputs rather than answering questions directly. This is a conceptually clean extension—the same tool capabilities, applied to a different task—but it requires the model to learn the judge's cognitive moves (comparing claims, weighing evidence, assigning credit) rather than the answerer's moves (finding, describing, computing).

Reframing judgment as a planning problem. The paper explicitly characterizes agentic evaluation as requiring the model to "decide when to think, which tool to call, how to parameterize it, and how to integrate intermediate results into a coherent, causal chain without hallucinations" (Section 1). This reframing connects reward modeling to the agentic AI literature and provides a framework for credit assignment: the model should be rewarded not just for correct final answers but for correct intermediate tool-use decisions.

Agentic capability as an emergent property of reward-based training. A distinctive aspect of the paper's positioning is that ARM-Thinker acquires tool-calling abilities "emergently through reward-based optimization, without explicit tool-use supervision in its training data" (Section 5.3). During GRPO training, the model learns to autonomously decide whether, when, and how many times to invoke tools, rather than following fixed tool-use patterns learned from curated demonstrations. This contrasts with models like DeepEyes and Pixel Reasoner, which are explicitly trained on tool-use trajectories. The paper argues this makes the approach more general: appropriately designed reward signals can induce systematic tool-use strategies without requiring expensive curated demonstrations for every new tool.

Scalability through synthetic data generation. Recognizing that high-quality agentic reward data is scarce and expensive to produce with human annotators, the paper introduces a scalable data-generation pipeline that constructs discriminative preference pairs anchored by verifiable checks (Section 1). A cold-start pipeline bootstraps such pairs via counterfactuals and perturbations (e.g., taking a ground-truth answer and generating a semantically similar but factually incorrect variant using GPT-4o-mini), followed by filtering for evidence validity. This makes the approach practical at scale, addressing the label-efficiency bottleneck that often limits reward model development.

Complementarity, not replacement. The paper does not claim that ARM-Thinker makes non-agentic reward models obsolete. It positions the agentic approach as specifically beneficial for tasks where verification requires evidence beyond what is available in a single forward pass—fine-grained perception, long-document QA, multi-constraint instruction following. For simpler judgment tasks where the correct answer is obvious from surface-level features, a non-agentic reward model may be sufficient and more computationally efficient. The paper does not explore this efficiency-accuracy tradeoff systematically, but the framing implies that agentic and non-agentic reward models may coexist in a practical deployment, with the agentic model reserved for complex cases.

A new benchmark to drive progress. The introduction of ARMBench-VL is itself a positioning move. Existing reward model benchmarks (VL-RewardBench, RewardBench-2, MME-RW) assess judgment accuracy on static QA pairs. They cannot evaluate whether a reward model correctly decides to retrieve a document page, zoom into a region, or run an instruction check—because they don't provide tools. By releasing the first reward benchmark that includes a toolkit and requires evidence-gathering, the paper aims to establish a new evaluation standard that incentivizes the development of agentic reward models rather than further optimization of passive scorers on existing benchmarks.

3. Technical Approach

3.1 Reader Orientation

ARM-Thinker is a reward model that, instead of passively scoring answers in a single forward pass, actively uses external tools—cropping images to inspect fine details, retrieving specific pages from long documents, and running programmatic checks on text—to gather verifiable evidence before rendering a judgment. It solves the problem that existing reward models hallucinate their justifications and cannot verify claims requiring evidence beyond what is visible in the provided inputs, by structuring judgment as an iterative think–act–verify loop where the model plans which evidence it needs, invokes tools to obtain that evidence, and only then produces a grounded score with an interpretable rationale.

3.2 Big-Picture Architecture (Diagram in Words)

ARM-Thinker has five major components, arranged in a closed-loop architecture where information flows cyclically:

1. Agent Loop Engine — the central execution framework that orchestrates the think–act–observe cycle. It maintains an indexed memory map of texts (candidate responses being judged) and images (original inputs and tool-produced crops), receives observations from executed tools, and feeds accumulated context back to the model for the next reasoning step.

2. Base Vision-Language Model (Qwen2.5-VL-7B) — the pretrained backbone that generates reasoning thoughts, decides which tool to call (including the choice to finalize with an answer), and parameterizes each tool call with specific arguments. All tool-use behavior and judgment logic is acquired through multi-stage training on top of this backbone.

3. Multimodal Toolkit — three families of tools accessible through a unified function-calling interface: (a) image crop and zoom-in tools for fine-grained visual inspection of spatial regions, (b) document retrieval tools (semantic query-based page search and index-based page access) for navigating multi-page documents, and (c) instruction-following check tools (19 textual validators) for verifying linguistic constraints like word count, keyword presence, and formatting.

4. Training Pipeline — a three-stage process: (1) supervised fine-tuning on high-quality trajectories combining general reward data and tool-specific cold-start data, (2) GRPO Stage 1 that encourages the model to explore tool usage through a tool-call exploration reward, and (3) GRPO Stage 2 that refines for accuracy using a hierarchically structured reward balancing correctness, format compliance, and tool efficacy.

5. ARMBench-VL Evaluation Harness — a static benchmark of 1,499 questions across three task categories, each providing the model with a toolkit and requiring evidence-grounded judgment. Not part of the system itself, but the evaluation framework that defines success.

Information flows through these components as follows: a judgment query (question, image, candidate responses) enters the agent loop → the base model generates a thought (planning what evidence is needed) → the model emits either a tool call (specifying which tool and its parameters) or a final answer → if a tool is called, the toolkit executes the corresponding function and returns text and/or images → the observation updates the agent's memory map → the cycle repeats, with each observation informing subsequent thoughts → when the model determines sufficient evidence has been gathered, it emits a final answer with a judgment and evidence-grounded rationale.

3.3 Roadmap for the Deep Dive

• First, the agent loop architecture and its formal specification (Equation 1): the think–act–observe cycle, the trajectory representation, and the indexed memory map that makes the loop stateful across tool calls. This is the runtime engine that every other component plugs into.

• Second, the three families of multimodal tools—their interfaces, internal implementations, and the design choices that make them composable within a unified function-calling schema. Understanding the tools is prerequisite to understanding what the model can do with the agent loop.

• Third, the data generation pipeline for constructing training data: how preference pairs are built from existing datasets, how cold-start trajectories are generated and filtered, and why difficulty filtration matters. This explains where the supervision signal comes from before we discuss how it is used.

• Fourth, the three-stage training procedure—SFT + cold start, GRPO Stage 1 (tool encouragement), GRPO Stage 2 (accuracy refinement)—with the specific reward function designs (Equations 4 and 5) and the rationale for separating tool-use learning from accuracy optimization.

• Fifth, how all of this is evaluated via ARMBench-VL: what makes this benchmark different from existing reward benchmarks, how it was constructed, and what the three task categories measure. This closes the loop by connecting the system's design to how its success is measured.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and training methodology paper whose core idea is that reward models should actively gather evidence through tool use before rendering judgments, and that this agentic behavior can be acquired through multi-stage reinforcement learning with carefully designed reward functions that separately incentivize tool exploration and judgment accuracy.

3.4.1 The Agent Loop: Think–Act–Observe Architecture

ARM-Thinker's runtime behavior is governed by an iterative think–act–observe loop, following the ReAct paradigm (Yao et al., 2022) and inheriting the structured tool-calling format from WebWatcher (Geng et al., 2025). The model does not produce a score in a single forward pass. Instead, it engages in a multi-turn process where each turn consists of three phases, and the model can continue for as many turns as needed until it decides to finalize.

Phase 1: Thought. The model generates an intermediate reasoning or planning step, enclosed in the special tokens thinking... response. This thought is internal—it is not executed as a tool call and does not produce an observation. It serves to plan the next action, interpret previous observations, or deliberate about what evidence is still needed. For example, a thought might read: "The two responses disagree about the zip code—response 1 says 89104, response 2 says 89706. I need to find the page in the document that contains the mailing address to verify which is correct."

Phase 2: Action. After thinking, the model chooses one of two possible actions. It can invoke an external tool by emitting a structured tool call wrapped in <tool_call>...</tool_call> tags, specifying the tool name and its arguments in a JSON-like format. Alternatively, it can terminate the reasoning process by emitting a final answer wrapped in <answer>...</answer> tags. Only one tool call is allowed per turn—the model cannot batch multiple tool invocations into a single action. This constraint forces sequential reasoning: each observation must be processed before the next tool is called.

Phase 3: Observation. When a tool is invoked, the environment (the tool execution harness) runs the corresponding function and returns the result to the model, wrapped in <tool_response>...</tool_response> tags. The observation can contain both text (e.g., "Retrieved pages: [7, 5, 10, 14, 13]") and images (e.g., a stitched concatenation of the retrieved document pages). Critically, the observation is appended to the model's context and is available for all subsequent thought steps. This means the model can reference evidence it retrieved three turns ago when making its final judgment.

The process continues until the model emits a final answer. At that point, the trajectory is complete, containing the full reasoning trace with all tool calls, observations, and the final judgment.

Formally, a trajectory of length $L$ is defined as:

$\tau = \{(\theta_0, t_0, o_0), (\theta_1, t_1, o_1), \ldots, (\theta_L, t_L, o_L)\}$

where $\theta_i$ is the internal thought generated at step $i$, $t_i$ is the tool invocation chosen at step $i$ (drawn from the toolkit $\mathcal{T}$), and $o_i$ is the observation returned by executing $t_i$. The trajectory terminates when the agent emits a Finish action, producing a final reasoning trace $\theta^*$ and answer $a^*$.

What this notation captures: For each turn in the conversation, we record what the model was thinking before it acted, which tool it chose to call (with what parameters), and what the environment returned in response. The sequence of these triples forms a complete, auditable record of how the model arrived at its judgment—every piece of evidence it gathered, in what order, and how it used each observation to inform subsequent reasoning.

Why this structured format: The explicit separation of thought, tool call, and observation enables credit assignment during training. The training process can inspect the trajectory and determine not just whether the final answer was correct, but whether intermediate tool calls were appropriate (was the right tool called with the right parameters?), whether observations were correctly interpreted (did the model draw the right conclusion from the evidence it retrieved?), and whether the reasoning chain is coherent (do thoughts follow logically from observations?). A flat "input → output" format would hide all of this intermediate structure, making it impossible to reward or penalize specific tool-use decisions.

Indexed Memory Map. During multi-turn reasoning, ARM-Thinker maintains a lightweight memory structure that stores original and intermediate multimodal artifacts. The memory has two maps:

• Texts map: stores candidate responses being compared, keyed by logical identifiers like "resp 1" and "resp 2". This allows the model to reference specific responses in its thoughts and tool calls without repeatedly including their full text.

• Images map: stores image paths accessible during reasoning, keyed by identifiers like "img 0" (the original image) or observation indices for tool-produced crops. When the image zoom-in tool produces a cropped region, it is added to this map so that subsequent tool calls (e.g., a second zoom-in on the already-cropped region) can reference it.

The memory map provides a lightweight yet structured mechanism for the model to track what evidence it has already gathered and what is still needed, without requiring the full conversation history to be re-parsed for referencing specific artifacts. The paper shows this structure in Figure 2(a) as a sidebar alongside the main agent loop.

Design choice: why not allow multiple simultaneous tool calls? The one-tool-per-turn constraint is not arbitrary. It enforces a sequential reasoning pattern where each observation must be processed before the next tool is chosen. If the model could batch calls, it might retrieve five document pages and then examine them all at once—but this would defeat the purpose of the think–act–observe loop, which is to let observations change the plan. By forcing sequential turns, the architecture ensures that the model can pivot: if the first retrieved page doesn't contain the needed information, the model can formulate a refined query for the next retrieval, or switch to a different tool entirely. This is the "planning" aspect the paper emphasizes.

3.4.2 Multimodal Tools: Unified Interface, Three Families

ARM-Thinker integrates three categories of multimodal tools, all inheriting from a common baseTool interface and exposing a unified OpenAI-style function-calling schema. Each tool implements a standard create–execute–release lifecycle and returns a ToolResponse object that can contain both textual feedback and images. The unified interface means the model learns a single tool-calling format and can compose tools across categories within a single trajectory—for example, retrieving a document page, zooming into a specific region on that page, and then running an instruction check on the extracted text.

Image Crop and Zoom-in Tool

This tool enables fine-grained inspection of visual regions by cropping a sub-region from an existing image. The model specifies two parameters:

• image_key: a logical identifier for which image to crop from, resolved against the images map (e.g., "original image" for the input image, or an observation reference for a previously cropped region).

• bbox_2d: a bounding box expressed in normalized integer coordinates within [0, 1000] along each axis. The normalization to a fixed range (0 to 1000) rather than pixel coordinates makes it easier for the language model to reason about relative locations—"the bottom-left quarter" translates naturally to x1=0, y1=500, x2=500, y2=1000—while still allowing precise cropping.

At execution time, the tool validates the bounding box: it checks that all four values are numeric, each in [0, 1000], and that x1 < x2 and y1 < y2. Normalized coordinates are converted to absolute pixel coordinates based on the underlying image dimensions. A helper function clamps the box to image boundaries, enforces reasonable aspect ratios, and ensures the cropped region is not too small—specifically, both width and height of the final crop must be at least MIN_QWEN_DIMENSION pixels (set to 28). If the original box is too small, it is automatically expanded around its center while respecting image boundaries. Very small crops are optionally upsampled by a factor of 2 using bicubic interpolation to improve readability.

Design choice: normalized coordinates over absolute pixels. The [0, 1000] normalized range is a deliberate interface design that makes the tool callable by a language model. Language models are notoriously bad at precise pixel-level reasoning—they can reason about "the left half" or "the bottom-right corner" much more reliably than "pixels 847 to 1243." The normalization abstracts away image resolution, so the same thought process works regardless of whether the original image is 512×512 or 4096×4096.

Design choice: minimum crop size enforcement and automatic expansion. The minimum dimension requirement (28 pixels) prevents degenerate crops that would be too small to contain useful information. The automatic expansion means the model doesn't need to precisely guess optimal bounding boxes—it can specify an approximate region, and the tool will ensure the result is usable. This is a robustness measure that reduces the precision burden on the language model.

Why this is composable with the agent loop. The zoom-in tool returns the cropped image as a new observation, which is added to the images map with a unique observation identifier (e.g., "observation 2"). The model can then reference this cropped region in subsequent thoughts or make further zoom-in calls on it. This enables iterative refinement: the model might zoom into a document page to locate a table, then zoom further into a specific cell of that table to read a value.

Document Retrieval Tools

Two complementary tools support long-document question answering over pre-rendered page images stored with the naming pattern {filename}_{page}.{ext}.

DocPageSearchTool (semantic retrieval). This tool takes a filename (document identifier) and a natural-language query as input. Internally, it uses a CLIP-ViT-B/32 encoder (loaded via SentenceTransformer from a local HuggingFace cache in offline mode) to produce 512-dimensional embeddings for document pages. These embeddings are stored and queried via a ChromaDB persistent client. At execution time, the tool queries the Chroma collection with the provided query string, filtering to the specified document (where={"source": filename}), and retrieves up to k results (default k=5). Returned page indices are deduplicated while preserving order, and each page is resolved to an image path with multiple extension attempts. The retrieved page images are horizontally concatenated into a single stitched image. Each page is resized to a fixed maximum long side (RAG_IMAGE_MAX_SIDE, default 1120 pixels) while preserving aspect ratio. To avoid out-of-memory errors, the tool enforces a hard cap on total pixel count (MAX_CONCAT_PIXELS); if the stitched image exceeds this budget, it is downsampled isotropically. The tool returns the stitched image along with a textual description summarizing which pages were retrieved.

DocPageByIndexTool (deterministic access). This tool takes a filename and an image_idx (page number) and directly returns the corresponding page image. It validates the index, resolves the page to a file path (trying multiple extensions), and fails with a clear error if the image cannot be found or the index is out of range. The page is loaded and resized using the same long-side constraint as the search tool.

Design choice: two complementary tools rather than a single unified retrieval interface. The paper's design reflects how a human would navigate a document: first, use semantic search to find which pages are likely relevant; then, use deterministic page access to inspect specific pages in detail. The model can use DocPageSearchTool with a broad query like "zip code of the mailing address" to narrow down candidate pages, then call DocPageByIndexTool to examine each promising page individually. This separation also provides a fallback: if semantic retrieval fails (e.g., the embedding model doesn't capture the right similarity), the model can iterate through pages by index.

Implementation detail: lazy initialization with asyncio.Lock. The retriever manager (encoder + Chroma collection) is initialized exactly once using an asyncio.Lock to avoid race conditions during concurrent tool calls. This is an engineering consideration that becomes important when multiple trajectories are being executed in parallel during training rollouts.

Design choice: CLIP-ViT-B/32 as the retrieval encoder. The paper uses a relatively lightweight encoder (CLIP-ViT-B/32, producing 512-dimensional embeddings) rather than a larger vision encoder. This choice prioritizes retrieval speed and memory efficiency over embedding quality, which is appropriate because the retrieval step is meant to narrow candidates, not make final judgments—the model itself will inspect retrieved pages visually to verify content.

Instruction-Following Check Tools

The third family consists of 19 textual validators that verify compliance with linguistic and structural constraints. All inherit from a shared BaseInstructionFollowingTool class, which automatically constructs the function schema from a declarative parameters list and manages a per-instance response store. The response store exposes a texts map mapping logical keys (e.g., "text 0") to full string outputs. A helper function resolves these keys and provides informative errors when resolution fails.

Each concrete tool implements an asynchronous execute_logic method that returns a boolean, wrapped into a textual ToolResponse of the form "Check result: True/False". The tools fall into several categories:

• Length and segmentation constraints: ParagraphNumberInRangeTool, SentenceNumberInRangeTool, WordCountInRangeTool, and per-paragraph variants check whether total or per-paragraph counts of paragraphs, sentences, or words fall within specified bounds. Sentences are segmented using NLTK's sentence tokenizer; paragraphs are defined via blank-line separation. For poetry-like formatting, the per-paragraph sentence checker automatically switches to a line-based heuristic.

• Lexical and formatting constraints: NotContainSubstringTool, EachSentenceBeginsWithTool, ResponseBeginsWithTool, ResponseEndsWithTool, NoArabicNumberTool, and others enforce the presence or absence of certain substrings, required prefixes/suffixes, or the prohibition of standalone Arabic numerals. All matching is case-insensitive after light normalization that strips punctuation and ellipses from boundaries.

• Keyword coverage: EachKeywordMentionedInRangeTool and TotalKeywordsMentionedInRangeTool check how often specific keywords appear, supporting both per-keyword bounds and global bounds over total mention count. A specialized matcher handles hashtags and special characters robustly.

• Numeric precision: PercentagePrecisionTool and NumberPrecisionTool verify that all percentage expressions or decimal numbers in the response have exactly a specified number of digits after the decimal point.

Why these are tools rather than built-in checks. A language model could attempt to verify word counts or keyword presence through pure text generation, but it would be unreliable—LLMs are known to miscount words, miss substring occurrences, and hallucinate compliance checks. By implementing these as deterministic, programmatic tools, the verification is guaranteed correct. The model's job is reduced to selecting which tool to call with which parameters, not to performing the check itself. This is a crucial design principle: offload mechanical verification to tools, keep the model focused on reasoning about which verifications matter and how to interpret the results.

Why 19 tools rather than a single parameterized constraint checker. The paper implements each constraint type as a separate tool with its own explicit function schema. This design makes each tool's purpose and parameters obvious to the model during tool selection—the model sees a clear function signature like word_count_in_range(text_key, min_words, max_words) rather than a generic check_constraint(constraint_type, ...). The explicit naming functions as documentation for the model, reducing ambiguity in tool selection.

Unified Tool Interface

All tools—image, document, and text—share the same calling convention. The model emits:

<tool_call>
{"name": "image_zoom_in", "arguments": {"image_key": "original image", "bbox_2d": [0, 500, 500, 1000]}}
</tool_call>

And receives:

<tool_response>
{"status": "success", "observation": "observation 2", "image": "<path_to_crop>"}
[The cropped image is now available as observation 2. Continue your reasoning...]
</tool_response>

This uniformity means the model doesn't need to learn different interaction protocols for different tool families. The function-calling schema is the same; only the tool names and parameter schemas differ. During training, this means a single reward structure can cover all tool-use decisions regardless of which tool was called.

3.4.3 Data Generation Pipeline: Building Training Data for Agentic Reward Modeling

Before training ARM-Thinker, the paper must construct training data that teaches the model both (1) how to judge responses and (2) how to use tools during judgment. This data does not exist off-the-shelf—existing reward modeling datasets provide preference pairs without tool interactions, and tool-use datasets provide task-solving trajectories without judgment. The paper's data pipeline bridges this gap in three steps.

Step 1: Preference Data Construction

The foundation is preference-based reward data that tells the model which responses are better than others. The paper combines two sources:

General multimodal QA reward supervision from LLaVA-Critic (Xiong et al., 2025), which provides human-aligned preference annotations for general visual question answering. This data covers broad image understanding and multimodal QA but contains no agentic interaction patterns—the judgments are simple pairwise preferences without tool use. It serves to bootstrap the model's basic reward capability before tool-specific training.

Agentic task data from three task-specific sources, each aligned with one tool family:

• DeepEyes (Zheng et al., 2025) for image crop and zoom-in tasks. DeepEyes provides high-resolution images with questions requiring fine-grained visual inspection, where the correct answer depends on details only visible at high zoom levels.

• MM-IFEngine (Ding et al., 2025) for instruction-following checking tasks. This dataset provides multimodal instructions with multiple constraints (e.g., "write a story about this image between 120 and 150 words, including the keyword 'serene'") and model responses that may or may not satisfy those constraints.

• MP-DocVQA (Tito et al., 2023) for document retrieval tasks. This dataset contains multi-page document visual question answering, where questions target specific pages within documents presented as full-page screenshots.

The sampling statistics (Appendix Section A.2) show approximately 40k samples from LLaVA-Critic, 4k from DeepEyes, 1k from MM-IFEngine, and 1k from MP-DocVQA for the SFT stage. The GRPO stage samples 20k from DeepEyes and 4k from MP-DocVQA (instruction-following data is excluded from GRPO because the paper found that once the model learns tool-use through other tasks, it generalizes naturally to instruction checking without explicit GRPO training on that data).

Negative response generation. Since most task-specific datasets contain only (question, image, ground-truth response) triplets, the paper must construct negative responses to form preference pairs. It uses GPT-4o-mini to generate "semantically related but flawed responses, introducing controlled negative samples with diverse error types." For each question–image pair $(q, I)$ with ground-truth response $r^+$, a negative response $r^-$ is generated, producing preference pairs:

$\mathcal{D}_{\text{pair}} = \{(q, I, r^+, r^-)\}$

where $r^+ \succ r^-$ denotes that the positive response better captures ground-truth details and factual correctness.

Deduplication by similarity. After generation, overly similar response pairs are removed to maintain sufficient diversity between $r^+$ and $r^-$. If the positive and negative responses are nearly identical (differing by only a word or two), the preference signal is too weak to be useful for training. The paper filters to ensure each pair provides "a clear and informative contrast."

Step 2: Cold-Start Trajectory Generation

Once preference pairs exist, the paper must produce reasoning trajectories—full think–act–observe sequences—that demonstrate how to judge these pairs using tools. The base model (Qwen2.5-VL-7B) cannot generate these trajectories from scratch because "VLMs typically exhibit limited zero-shot competence in executing novel tool invocations" (Section 3.3.1). The cold-start process bootstraps this capability.

Difficulty filtration. Before generating trajectories, the paper applies a difficulty filtration step: it removes "trivial samples on which the base model achieves 100% accuracy in five sampling rollouts." The rationale is that training on problems the model already solves correctly without tools provides no learning signal for tool use—the model can judge correctly without any evidence gathering, so adding tools would be superfluous. This filtration ensures subsequent training "focuses on more informative and challenging instances."

Trajectory generation with stronger models. The remaining data are then used for inference within the agent loop, where "stronger LVLMs" (larger vision-language models, likely larger Qwen variants, though the paper does not specify exact models) generate multimodal chain-of-thought trajectories augmented with explicit tool invocations. These stronger models, having better zero-shot tool-use capability, produce plausible trajectories showing how to use tools to verify judgments.

Three-dimensional trajectory filtering. Generated trajectories are filtered along three axes:

1. Format: the trajectory must conform to the structured think–act–observe format with proper XML tag usage. Malformed trajectories (missing closing tags, incorrect nesting, garbled tool calls) are discarded.

2. Accuracy: the final judgment must be correct—the model must correctly identify which response is better. A trajectory with perfect tool usage but a wrong final answer is worse than useless for training a reward model.

3. Behavior: the trajectory must demonstrate successful tool calls—the model must actually invoke tools (not just think about invoking them) and the tools must return valid results. Trajectories where the model tries to call a tool but provides invalid parameters, or where tool execution fails, are discarded.

The final filtered set constitutes "high-quality multimodal CoT data" used for both SFT and cold-start iterations, "progressively improving the model's reasoning depth and tool-use proficiency."

Design choice: why use stronger models for cold-start generation rather than human annotation. Human annotation of tool-use trajectories would be prohibitively expensive—each trajectory might involve 5-10 tool calls, each requiring the annotator to decide which tool to call, with what parameters, and how to interpret the observation. The paper's approach leverages the observation that sufficiently large VLMs have some zero-shot tool-use capability, even if imperfect. The three-dimensional filtering then keeps only the trajectories where this zero-shot capability produced a correct, well-formed result, yielding high-quality data without human labor.

Step 3: Iterative Refinement

The cold-start process is not one-shot. Trajectories generated by stronger models are used to train the 7B model via SFT, which improves its tool-use capability. The improved 7B model can then generate additional trajectories, which are filtered and fed back into training. This iterative bootstrapping progressively improves the model's tool-use proficiency without requiring ever-larger models for generation.

3.4.4 Multi-Stage Training: SFT + Two-Stage GRPO

The training pipeline has three stages, each with a distinct objective and data composition. Figure 2(b) in the paper diagrams this pipeline.

Stage 0: Supervised Fine-Tuning and Cold Start

The base model (Qwen2.5-VL-7B) is fine-tuned on the high-quality multimodal trajectory data produced by the data generation pipeline. The training data has two components:

• General reward capability data from LLaVA-Critic: these trajectories teach the model how to judge response quality for general multimodal QA, enhancing "the model's fundamental reward capability for general image understanding and multimodal question answering." These trajectories may or may not involve tool use—they primarily teach the judgment skill itself.

• Agentic cold-start data containing explicit tool interactions: these trajectories teach the model the structured reasoning format and correct tool-use behaviors. They serve to "initialize the model with structured reasoning and correct tool-use behaviors, as VLMs typically exhibit limited zero-shot competence in executing novel tool invocations."

The SFT stage uses the standard language modeling objective—next-token prediction on the trajectory tokens, with loss computed only on the assistant's outputs (thoughts, tool calls, and final answers), not on the observation tokens (which come from the environment, not the model).

After SFT, the model has two capabilities it lacked before: it can follow the think–act–observe format, and it has some (imperfect) ability to call tools appropriately. But it has not been optimized for the specific reward modeling objective—it knows how to use tools but not when or why. The two GRPO stages address this gap.

Stage 1: Tool-Call Encouragement (GRPO)

The first GRPO stage addresses the exploration problem: the model may know how to call tools but may default to judging without tools because tool use requires more effort (more tokens, more turns). The stage-1 reward function $R_{\text{tool}}$ is designed to actively encourage tool usage:

$R_{\text{tool}} = R_f + R_{\text{try}} \cdot \mathbb{1}_{\text{tool\_calls} > 0}$

where $R_f$ is a format reward that enforces the correct output format following the think–act–observe style, $R_{\text{try}}$ is a positive signal assigned when the model makes any reasonable attempt to call a tool, and $\mathbb{1}_{\text{tool\_calls} > 0}$ is an indicator that equals 1 if the trajectory contains at least one tool invocation and 0 otherwise.

What this computes: the model receives a base reward for producing correctly formatted output, plus a bonus if it attempted to use at least one tool. The bonus is binary—any tool use counts, regardless of which tool, whether it was the right tool, or whether it contributed to a correct answer.

Why this form: in early training, the model has not yet learned which tool-use actions lead to correct judgments. A reward that penalized incorrect tool use (e.g., calling the wrong tool for the situation) would discourage exploration entirely—the model would learn that tool use is risky and default to tool-free judgment. The binary encouragement reward instead makes tool use always beneficial in expectation, creating a gradient toward exploration. Once the model is consistently attempting tool use, Stage 2 can refine which tool uses are good.

Training procedure: Following GRPO (Group Relative Policy Optimization; Shao et al., 2024), for each multimodal query–image pair $(q, I)$, the model generates a group of $n$ trajectories (the paper does not specify $n$ explicitly, but GRPO typically uses 4-16 trajectories per query):

$G = \{(\tau_i, a_i)\}_{i=1}^n$

Each trajectory $\tau_i$ is a complete think–act–observe sequence, and $a_i$ is the final answer. The GRPO algorithm computes advantages by comparing the rewards of trajectories within the same group, then updates the policy to increase the probability of trajectories with above-average reward and decrease the probability of trajectories with below-average reward. This group-relative formulation reduces variance compared to using an absolute reward threshold.

Stage 2: Accuracy Refinement (GRPO)

After the model has learned to invoke tools consistently, the reward function shifts focus toward factual correctness and verifiable tool efficacy. The stage-2 reward $R_{\text{acc}}$ is hierarchically structured based on whether the tool use contributed to a correct answer:

$R_{\text{acc}} = \begin{cases} R_f + R_{\text{try}}, & \text{if } R_a = 0 \text{ and tool\_calls} > 0; \\ R_f + R_a, & \text{if } R_a > 0 \text{ and succ\_tool\_calls} = 0; \\ R_f + R_a + R_{\text{succ}}, & \text{if } R_a > 0 \text{ and succ\_tool\_calls} > 0. \end{cases}$

where $R_f$ is the format reward (same as Stage 1), $R_{\text{try}}$ is the tool exploration bonus (same as Stage 1), $R_a$ evaluates the factual correctness of the final answer, $R_{\text{succ}}$ assigns additional credit when tool usage directly contributes to a correct prediction, and succ_tool_calls counts the number of tool calls that were both correctly parameterized and produced observations that the model correctly used in its reasoning.

What this computes: The reward has three tiers:

1. Tool use without correctness: if the model called tools but got the final answer wrong ($R_a = 0$), it receives only the format reward and the exploration bonus. No accuracy component, no success bonus. This rewards the attempt at tool use but provides no credit for incorrect judgments.

2. Correct judgment without successful tool use: if the model got the answer right but either didn't use tools or used them unsuccessfully ($R_a > 0$ but succ_tool_calls = 0), it receives the format reward and the accuracy reward, but no tool success bonus. This rewards getting the right answer, acknowledging that tool use isn't always necessary.

3. Correct judgment with successful tool use: if the model got the answer right and its tool calls were successful ($R_a > 0$ and succ_tool_calls > 0), it receives all three components: format, accuracy, and tool success. This is the highest reward tier, incentivizing the model to use tools when they contribute to correctness.

Why this hierarchical structure: The three-tier design prevents several failure modes that a flat reward would encourage. If tool use were always rewarded regardless of correctness (tier 1 only), the model would learn to call tools gratuitously without caring about judgment accuracy—the over-use failure mode. If correctness were the only reward (tier 2 only), the model would avoid tool use whenever possible (since tool use costs tokens and increases the chance of format errors)—the under-use failure mode. If tool success were rewarded even when the final answer is wrong, the model would learn to make technically correct tool calls that don't actually help with the judgment.

The hierarchical structure instead creates a clear incentive gradient: tool use that leads to correct judgments is best (tier 3), correct judgments without tools are acceptable (tier 2), and tool use without correctness gets minimal reward (tier 1). This shapes the model toward instrumental tool use—tools are a means to accurate judgment, not an end in themselves.

Definition of successful tool calls. The paper defines succ_tool_calls as tool invocations that "directly contribute to a correct prediction." This requires determining whether an observation from a tool call was used in subsequent reasoning that led to the correct answer. The paper does not detail the exact implementation of this credit assignment—whether it uses heuristic matching (e.g., does the final answer text reference content from the observation?) or a learned verifier—but the conceptual definition is clear: a tool call is successful if removing it would likely change the judgment.

Design choice: why separate tool encouragement (Stage 1) from accuracy refinement (Stage 2) into two distinct training phases. This is perhaps the most important training design decision. If the model were trained with the Stage 2 reward from the beginning, the exploration problem would be severe: the model would initially lack tool-use competence, so most tool calls would be unsuccessful, so $R_a$ and $R_{\text{succ}}$ would both be zero for most trajectories, and the model would receive only the format reward. This provides no gradient toward tool use. The model would converge to a tool-free policy (always answer without tools, getting at least $R_a$ when it happens to be right) and never discover that tools can improve accuracy.

Stage 1 solves this by temporarily removing the accuracy requirement: any tool use is rewarded, so the model explores the tool-use space broadly, discovering which tools exist, what parameters they take, and what observations they return. Once this exploration has produced a policy that uses tools frequently, Stage 2 tightens the reward to demand that tool use actually improves correctness. The separation mirrors the exploration-then-exploitation pattern from reinforcement learning.

GRPO implementation details. The paper uses GRPO (Group Relative Policy Optimization; Shao et al., 2024), an algorithm that optimizes a policy by comparing rewards within groups of trajectories generated from the same prompt. For each training step, the model generates multiple trajectories for each query, computes rewards using the stage-appropriate reward function, normalizes advantages within each group, and updates the policy. The paper does not specify exact GRPO hyperparameters (learning rate, KL penalty coefficient, number of trajectories per group) in the main text, though these details may appear in the appendix (which is not fully reproduced in the provided content).

What the Model Learns During GRPO

The ablation study in Section 5.5 (Figure 4) provides insight into what each reward component teaches. Three variants are compared:

• Only Acc & Fmt Reward: optimizes task accuracy and response format without any tool-awareness. This model's tool-call rate stabilizes around 0.7 calls per trajectory, and its accuracy plateaus early at 77.5%. The low tool-call rate confirms that without explicit tool encouragement, the model under-uses tools.

• Fixed Tool Reward: adds a constant bonus whenever the model invokes a tool. This model's tool-call rate climbs unchecked to approximately 1.15 calls per trajectory, yet its accuracy reaches only 78.5%. The high call rate but mediocre accuracy confirms that simply rewarding all tool use leads to over-use—the model calls tools for the bonus, not because they help.

• ARM-Thinker Reward (the proposed two-stage design): the tool-call rate rises to approximately 1.12 but then stabilizes and even contracts slightly after step 54, while accuracy reaches the highest final value (the paper doesn't give the exact number in the ablation text, but the figure shows it above the other two variants). The stabilization of the tool-call curve is key evidence that the model has learned an optimal policy—it invokes tools based on contextual utility rather than merely chasing a fixed bonus.

This ablation validates the core design thesis: adaptive, context-dependent reward shaping successfully balances accuracy maximization with appropriate tool utilization, avoiding both under-use and over-use.

3.4.5 Evaluation Framework: ARMBench-VL

ARMBench-VL is introduced not as part of the model but as the evaluation framework that defines success for agentic reward modeling. Understanding its construction is essential for interpreting all experimental results.

Benchmark Motivation

Existing reward benchmarks (VL-RewardBench, RewardBench-2, MME-RW) assess judgment accuracy on static QA pairs where the model directly outputs a score or preference without tools. They cannot evaluate whether a model correctly decides to retrieve a document page, zoom into a region, or run an instruction check. As the paper states, "their question types are simple and the tasks are narrow, assessing only basic perception and reasoning" (Section 4.1).

ARMBench-VL is explicitly designed to be "the first multimodal reward benchmark that requires tool use" (Section 4.1). It provides a toolkit (the same image crop, document retrieval, and instruction-following tools available during training) and requires the model to decide which tools to invoke during evaluation. The benchmark measures both whether the final judgment is correct and (implicitly) whether the model used tools appropriately to arrive at that judgment.

Three Task Categories

The benchmark comprises 1,499 questions across three tasks:

1. Fine-Grained Perception (550 questions). Questions focus on local details in high-resolution images—identifying small objects, reading tiny text, or verifying claims about specific image regions. The model has access to image crop and zoom-in tools and must decide when the global view is insufficient and a zoomed inspection is needed. This task includes 295 general pairwise comparison items, 163 2-way comparisons, and 92 4-way comparisons (where the model must select the best among four candidate responses).

2. Multimodal Long Document QA (460 questions). Questions target a specific page within a multi-page document presented as full-page screenshots. The model must use the document page retrieval tools to locate the relevant page. This task includes 173 single-response judging items (is this answer correct?) and 287 2-way pairwise comparisons.

3. Multimodal Instruction Following (489 questions). Questions impose multiple constraints (e.g., output format, required keywords, word count ranges). The model must analyze which tools from the instruction-following tool pool are appropriate to verify whether the response satisfies all stated constraints. All 489 items are single-response judging—the model determines whether a given response satisfies the constraints or not.

Benchmark Construction Process

The benchmark is built from multiple existing datasets through a process of selection, expansion, and reconstruction:

Source datasets: Fine-grained perception draws from V*Bench (Wu & Xie, 2024) and VisualProbe. Long document QA draws from MMlongbench-doc. Instruction following draws from MM-IFEval (Ding et al., 2025).

Difficulty enhancement. For fine-grained perception and long document QA, the paper first removes questions with overly simple answers (e.g., yes/no questions). Then, Qwen3-VL-235B-A22B-Thinking—a much larger model—is used to (a) expand the original answers into more detailed responses while preserving the original information, and (b) generate matched incorrect responses with similar style and length. The use of a 235B model for generation ensures the incorrect responses are plausible and the contrasts are subtle, creating challenging discrimination tasks.

Question rewriting for diversity. Some questions are rewritten into descriptive questions (e.g., "What color is the hat?" becomes "Describe the man on the roof in detail, focusing on his hat"). The corresponding responses are revised so that correct answers describe the relevant local region while mentioning the correct fact, and incorrect answers introduce plausible but wrong details. This creates a more naturalistic evaluation where the model must compare descriptions rather than match single-word answers.

Multi-way comparisons. To raise difficulty, the paper increases the number of incorrect answers for some items, constructing both 2-way (one correct, one incorrect) and 4-way (one correct, three incorrect) comparison settings. The 4-way setting is particularly challenging because the model must evaluate and discard multiple plausible-sounding incorrect responses.

Quality control. During construction, the generating model (Qwen3-VL-235B-A22B-Thinking) is explicitly instructed "not to mention correctness-related cues in its responses." Any items where the correct answer is mistakenly included in the negative responses are filtered out. This ensures that the benchmark cannot be solved by surface-level text matching—the model must actually verify the claimed facts against the visual evidence.

For instruction following, after filtering model outputs from Qwen2.5-VL-7B (the base model), the paper keeps a subset of responses that may be correct or incorrect, and the reward model must directly judge whether each response satisfies the stated constraints. The full construction procedures and prompts are provided in Appendix Section F (reproduced in the supplementary material).

Benchmark Properties

Table 1 in the paper compares ARMBench-VL with existing reward benchmarks:

The key differentiator is the "Tools" column: ARMBench-VL is the first reward benchmark that provides a toolkit and allows models to freely select tools to evaluate responses. The paper argues this enables "a more precise assessment of the model's reward-modeling capability" because judgment accuracy on complex tasks cannot be separated from the model's ability to gather the evidence needed for accurate judgment.

Evaluation Protocol

For evaluating agentic models like ARM-Thinker, each task is presented with a fixed chain-of-thought prompt (Figure 7 in the appendix) that instructs the model to:

1. Output reasoning inside thinking... response tags.
2. Either provide a final answer within <answer>...</answer> or call a tool within <tool_call>...</tool_call>.
3. Call tools multiple times across turns as needed for judgment or verification, but only one tool per turn.
4. Handle tool call failures by retrying or providing a final answer.
5. Once no more tool calls are needed, provide the final judgment within <answer>...</answer>.

For image-based tasks, the prompt specifies that "the given image is 'original image'" (which maps to img 0 in the memory map). For document-based tasks, it provides the document ID and page indexing convention: "page indices in the combined image start from 1 at the top-left corner and increase horizontally from left to right, then continue to the next row from top to bottom."

The evaluation metrics are straightforward: accuracy (is the final judgment correct?) for each task category and overall. There is no separate metric for tool-use quality—the model is evaluated on end-to-end judgment accuracy, and appropriate tool use is instrumental to achieving high accuracy on this benchmark, not a separately scored objective.

Design choice: why include single-response judging alongside pairwise comparison. Pairwise comparison (is A better than B?) is the dominant paradigm in reward model evaluation because it aligns with how reward models are used in RLHF (the Bradley-Terry preference model). However, single-response judging (is this response good or bad?) is also important: it maps onto absolute quality assessment, which is needed for filtering, rejection sampling, and pointwise reward assignment. ARMBench-VL includes both formats, making it a more comprehensive evaluation of reward modeling capability than benchmarks that exclusively use pairwise comparison.

Why the benchmark construction uses a 235B model for data generation. The paper leverages the observation that larger models are more capable at generating plausible but incorrect responses—subtle enough to fool a lazy judge but distinguishable with careful evidence gathering. Using a 235B model for generation creates a high-quality adversarial test set that would be difficult and expensive to produce with human annotators. The quality control steps (filtering out items where incorrect responses accidentally contain the correct answer, instructing the generator not to reveal correctness cues) ensure this adversarial generation doesn't introduce artifacts that could be exploited by superficial judging strategies.

4. Key Insights and Innovations

Innovation 1: Reframing Reward Modeling as an Evidence-Gathering Planning Problem

The paper's most fundamental conceptual move is redefining what a reward model is. Prior work—whether rule-based verifiers (Guo et al., 2025), non-agentic generative reward models (Wang et al., 2025c; Zang et al., 2025), or even text-only agentic reward models (Peng et al., 2025)—conceptualizes judgment as a mapping from (question, image, response) to (score or preference). The reward model is a function approximator. Its quality is measured by how well it approximates human preferences on static evaluation sets.

ARM-Thinker rejects this functional framing. The paper argues that complex multimodal judgment is fundamentally a sequential planning problem: the judge must decide when to think (deliberation needed), which tool to call (what evidence is missing), how to parameterize it (which document page, which image region, which constraint), and how to integrate intermediate results into a causal chain that supports the final judgment. This reframes the reward model's job from predicting correctness to constructing a verifiable chain of evidence. The output is not just a score—it is a fully auditable trajectory showing what evidence was gathered, in what order, and how it supports the conclusion.

This is not a minor extension. It shifts the theoretical underpinning of reward modeling from supervised learning (learn the mapping from inputs to human-preferred judgments) to sequential decision-making under uncertainty (learn a policy for gathering evidence that maximizes the probability of correct judgment). The training objective changes accordingly: instead of minimizing cross-entropy on static preference pairs, the model must learn a policy $\pi(\text{action} \mid \text{context})$ over tool calls and final answers, optimized through reinforcement learning with rewards that credit both correct judgments and instrumental tool use.

Why this matters beyond the specific system: once judgment is framed as planning, the entire toolbox of sequential decision-making research—exploration strategies, credit assignment, meta-reasoning about when to stop gathering evidence—becomes applicable to reward model design. The paper's two-stage GRPO training (Stage 1 for exploration, Stage 2 for accuracy refinement) is a first instantiation of this perspective, but the framing opens many directions the paper does not explore: learned termination policies, uncertainty-aware tool selection, information-theoretic objectives for evidence gathering. The conceptual shift from "judgment as prediction" to "judgment as evidence construction" is the paper's deepest intellectual contribution, and it is a fundamental reframing rather than an incremental improvement.

Evidence for the importance of this framing appears in the ablation study (Figure 4): when the reward function treats tool use as instrumental to correctness (ARM-Thinker reward) rather than as intrinsically valuable (Fixed Tool Reward) or irrelevant (Only Acc & Fmt Reward), the model learns a disciplined tool-use policy—the tool-call curve stabilizes around 1.12 calls per trajectory and even contracts slightly after step 54, indicating the model has learned to call tools when they help and skip them when they don't. This emergent discrimination between useful and gratuitous tool use would be impossible under a functional framing that doesn't model the decision process.

Innovation 2: Tool-Use Capability as an Emergent Property of Reward Design, Not Curated Demonstrations

A striking aspect of ARM-Thinker is how it acquires tool-use behavior. The dominant paradigm for tool-augmented models—exemplified by DeepEyes (Zheng et al., 2025), Pixel Reasoner (Su et al., 2025), and Mini-o3 (Lai et al., 2025)—is to train on curated demonstrations of correct tool-use trajectories. These models are shown examples of "here is when you should zoom in," "here is how you parameterize the crop," "here is how you interpret the observation," and they learn by imitation. This works but is expensive (each new tool requires new demonstrations), brittle (the model learns tool-use patterns specific to the training distribution), and inherently limited (the model can only reproduce tool-use strategies that appeared in the training data).

ARM-Thinker takes a fundamentally different approach. The model is never explicitly shown when or how to use tools. During GRPO training, it receives only outcome-based rewards: format compliance, tool-call attempts (Stage 1), answer correctness, and tool success (Stage 2). The model must discover, through reinforcement learning, that calling doc_page_search with an appropriate query makes it more likely to find the evidence page, which makes it more likely to judge correctly, which yields higher reward. The tool-use strategy is induced by the reward structure, not imitated from demonstrations.

The paper calls this "emergent" tool-calling ability, which is a controversial term but accurately describes the phenomenon: the model is not taught tool use; it discovers that tool use is instrumental to achieving its objective. The cold-start SFT stage provides some tool-use trajectories (generated by larger models and filtered), but the GRPO stages—where the model generates its own trajectories and is rewarded based on outcomes—are where the sophisticated tool-use policy actually develops. The cold-start data teaches the model how to format tool calls (the syntax); GRPO teaches it when to call which tool (the strategy).

Why this is intellectually distinctive: it suggests a scalability argument that the paper implies but does not fully articulate. If tool-use strategies can be induced through reward design rather than hand-crafted demonstrations, then adding new tools to the reward model's toolkit requires only defining the tool's interface and ensuring the reward function can detect whether tool calls contributed to correct judgments. No new demonstrations, no manual trajectory annotation. The 19 instruction-following check tools in ARM-Thinker—which were excluded from GRPO training data entirely, yet the model learned to use them—provide preliminary evidence for this generalization claim (Section 5.3: "once the model is trained with our framework, its tool-use capability generalizes naturally to such tasks without requiring explicit inclusion of this data").

The paper's positioning of this insight against the existing tool-use paradigm (Section 5.3: "Unlike specialized visual reasoning models trained directly on tool-use demonstrations... ARM-Thinker-7B acquires tool-calling abilities emergently through reward-based optimization, without explicit tool-use supervision") is deliberate. It argues that demonstration-based approaches scale poorly with the number and diversity of tools, while reward-based approaches scale naturally—each new tool just expands the action space, and the same reward structure (correctness + tool success bonus) shapes the policy toward instrumental use of all tools.

The evidence appears in Table 3: ARM-Thinker-7B achieves 76.5% average accuracy on think-with-images benchmarks, matching or exceeding Mini-o3 (76.1%) and outperforming DeepEyes (72.5%) and Pixel Reasoner (72.9%)—all models trained on curated tool-use demonstrations. This is a significant negative result for the demonstration-based paradigm: an emergent tool-use policy, trained only with outcome rewards, matches or exceeds models that were explicitly taught tool use. It doesn't prove that demonstrations are unnecessary, but it demonstrates that they are not required for strong tool-use performance, which has implications for how future systems should allocate their data collection budget.

Innovation 3: The Hierarchical Two-Stage Reward Structure as a Solution to the Exploration-Exploitation Tradeoff in Tool-Augmented RL

The paper's two-stage GRPO design—Stage 1 encourages any tool use, Stage 2 rewards only instrumentally useful tool use—is not just an implementation detail. It is a principled solution to a fundamental tension in training tool-augmented agents: exploration of the tool space requires trying actions whose immediate outcomes are unknown, but the agent initially lacks the competence to use tools effectively, so early tool use is likely to fail and produce negative reward signals. If the reward function penalizes incorrect tool use from the start, the agent learns that tool use is risky and converges to a tool-free policy. If the reward function always rewards tool use regardless of outcome, the agent learns to call tools gratuitously—the over-use failure mode documented in Figure 4.

The paper's innovation is recognizing that this is not a tuning problem (find the right weight for a tool-use bonus) but a sequencing problem: the objective function must change over the course of training. Early in training, when the agent has no tool-use skill, the reward function should encourage exploration without penalizing failure. Later, when the agent can use tools competently, the reward function should demand that tool use actually improves outcomes. This sequencing turns an impossible simultaneous optimization (maximize both exploration and accuracy) into two feasible sequential optimizations (first maximize exploration, then maximize accuracy conditioned on exploration).

Why this is intellectually distinctive: the standard approach to exploration in RL is to add an entropy bonus or curiosity-driven intrinsic reward that encourages the agent to visit novel states alongside the task reward. These methods optimize a single composite objective throughout training. ARM-Thinker's two-stage design instead recognizes that the appropriate balance between exploration and exploitation changes qualitatively as competence develops. Stage 1 isn't just giving a higher weight to exploration—it's temporarily ignoring accuracy entirely for tool-use decisions ($R_a$ and $R_{\text{succ}}$ don't appear in Equation 4). This is a sharper intervention than standard exploration bonuses and reflects a conceptual insight: you cannot reward tool accuracy until the agent has enough tool competence to occasionally succeed, because all tool attempts would receive zero accuracy reward and the signal would be purely noise.

The ablation in Figure 4 empirically validates that no single reward function can solve this—both the "Only Acc & Fmt" (under-use) and "Fixed Tool Reward" (over-use) variants underperform the two-stage design. This is clean evidence that the sequencing, not just the reward components, is load-bearing. The two-stage design is not simply the sum of Stage 1 and Stage 2 reward components applied simultaneously; it's the temporal separation that matters.

This innovation has implications beyond ARM-Thinker. Any system that needs an agent to learn a complex behavioral repertoire (tool use, information seeking, multi-step planning) through RL faces the same exploration-competence chicken-and-egg problem. The paper's solution—stage the objective from "try the behavior" to "do the behavior well"—is a generalizable training strategy that could apply to RL fine-tuning of language agents more broadly, not just reward models.

Innovation 4: ARMBench-VL as a Diagnostic Instrument That Makes Tool Use a First-Class Evaluation Dimension

The paper's introduction of ARMBench-VL is not merely "we built a new benchmark because existing ones are insufficient." The benchmark embodies a specific diagnostic philosophy: you cannot evaluate a judge's accuracy on complex cases separately from evaluating its evidence-gathering process, because access to evidence is what makes accurate judgment possible. Existing reward benchmarks (VL-RewardBench, RewardBench-2, MME-RW) evaluate judgment accuracy on static inputs, where the model sees the question, image, and response all at once and must score them. These benchmarks conflate two distinct capabilities: (a) can the model perceive the relevant information in the given inputs, and (b) can the model reason about that information to produce a correct judgment?

ARMBench-VL disentangles these by making perception active. The model is not given all the information it needs—it must decide what information is missing, retrieve it through tool calls, and then judge. A model that fails on ARMBench-VL but succeeds on VL-RewardBench reveals that its judgment accuracy on static benchmarks was relying on surface-level heuristics or lucky co-occurrence of visual cues, not on genuine evidence-grounded reasoning. Conversely, a model that succeeds on ARMBench-VL demonstrates not just judgment accuracy but the meta-cognitive ability to know what evidence is needed and how to obtain it.

The diagnostic value of this design appears in the results. Table 2 shows that Qwen3-VL-8B scores 66.0% on VL-RewardBench (no tools required) but only 50.6% on ARMBench-VL (tools required). The 15.4-point gap reveals a capability that VL-RewardBench cannot measure: the model's judgment degrades substantially when it must actively gather evidence rather than passively receive it. ARM-Thinker-7B, by contrast, shows a much smaller gap (67.8% on VL-RewardBench, 64.6% on ARMBench-VL). The benchmark thus identifies where judgment capability breaks down—not in reasoning about given information, but in the process of acquiring that information.

Why this is more than a benchmark contribution: ARMBench-VL defines a new axis of evaluation for reward models that will shape what the field optimizes toward. Before this benchmark, a reward model developer could achieve state-of-the-art by improving static judgment accuracy on existing benchmarks—through better training data, longer reasoning chains, or larger models. ARMBench-VL makes that optimization path insufficient: to score well, a reward model must now demonstrate tool-use competence. This shifts the research agenda from "improve judgment accuracy" to "improve the evidence-gathering process that enables accurate judgment." It's an instance of benchmarking as a steering mechanism for the field's research priorities.

The inclusion of 4-way comparison items (92 in the fine-grained perception task) further sharpens this diagnostic function. A model that can distinguish a correct answer from one incorrect distractor (2-way) but fails on 4-way must contend with multiple plausible-sounding incorrect responses. This probes the depth of the model's evidence gathering: with more candidates, surface-level cues (length, style, keyword matching) become less reliable, and genuine evidence-grounded verification becomes more necessary. The graded difficulty within the benchmark (2-way vs. 4-way, single-response vs. pairwise) allows future work to precisely characterize under what conditions a reward model's evidence gathering is sufficient.

5. Experimental Analysis

Evaluation Methodology

• Datasets. The paper evaluates across three benchmark categories. For reward modeling: VL-RewardBench (multimodal inputs, 1,250 cases, 2-way pairwise), RewardBench-2 (text-only, 1,865 cases, 4-way pairwise + Tie), and ARMBench-VL (the proposed benchmark, 1,499 questions across three task categories with tool use). For tool-assisted visual reasoning ("think-with-images"): V* Bench (Wu & Xie, 2024), HRBench-4K and HRBench-8K (Wang et al., 2025b), and MME-RealWorld (Zhang et al., 2024b). For general visual reasoning: MMMU (Yue et al., 2024), MathVista (Lu et al., 2023), MathVision (Wang et al., 2024b), MathVerse (Zhang et al., 2024a), WeMath (Qiao et al., 2025), and LogicVista (Xiao et al., 2024). No explicit train/val/test splits are mentioned for these benchmarks—they are used exclusively for evaluation.

• Base model(s). The backbone is Qwen2.5-VL-7B (Bai et al., 2025), a 7-billion-parameter vision-language model. The paper states this model is built upon for ARM-Thinker-7B. For the FLOPs-matched comparison in Section 7, a larger model with approximately 14× more parameters is mentioned, though details are deferred to that section. The choice of a 7B scale is motivated by representativeness—it sits in a regime where the model has non-trivial multimodal reasoning capability but far from saturation, leaving room for test-time strategies to make a measurable difference.

• Metrics. The primary metric throughout is accuracy—the fraction of judgment questions for which the model produces the correct answer or preference. For pairwise comparison tasks, this means correctly identifying which response is better (or identifying a tie where applicable). For single-response judging (instruction following in ARMBench-VL), this means correctly determining whether the response satisfies the stated constraints (True/False). No secondary metrics (e.g., precision/recall, calibration, tool-call efficiency) are systematically reported, though the ablation study does report tool-call frequency as a diagnostic.

• Baselines. The paper compares against a diverse set spanning general-purpose LVLMs, specialized reward models, and visual tool-use models. General-purpose LVLMs: InternVL3-8B (Zhu et al., 2025), InternVL3.5-8B (Wang et al., 2025a), Qwen3-VL-8B (Bai et al., 2025), GPT-4o (Hurst et al., 2024), and the base model Qwen2.5-VL-7B (Bai et al., 2025). Specialized reward model: UnifiedReward-7B (Wang et al., 2025c), a recent multimodal reward model evaluated on the same reward benchmarks. Visual tool-use models: DeepEyes (Zheng et al., 2025), Pixel Reasoner (Su et al., 2025), and Mini-o3 (Lai et al., 2025), which are explicitly trained with tool-use supervision and serve as strong references on the think-with-images benchmarks. For general reasoning, Gemma-3-27B (Team et al., 2025) provides a larger-model upper-bound reference. The paper notes that results for some baselines (DeepEyes, Pixel Reasoner, Mini-o3, GPT-4o on tool-use benchmarks) are copied from Lai et al. (2025).

• Generation budget / compute accounting. The paper does not use a unified "generation budget" or FLOPs metric for fair comparison across models. Each model is evaluated under its standard inference protocol: non-agentic models produce judgments in a single forward pass, while ARM-Thinker and other agentic-capable models can invoke tools iteratively. The cost of tool calls (additional tokens, image processing, retrieval computation) is not accounted for in any efficiency metric. The paper acknowledges this implicitly when discussing the difficulty estimation cost in Section 3.2, but the reported accuracy numbers do not incorporate any compute-normalization. This means comparisons between agentic and non-agentic models on accuracy alone are not compute-matched—the agentic model likely uses substantially more inference compute per judgment.

• Cross-validation / statistical protocol. The paper does not describe confidence intervals, statistical significance tests, or cross-validation for the main benchmark evaluations. Accuracy numbers are reported as point estimates. For strategy selection on ARMBench-VL, there is no cross-validation because ARMBench-VL is a fixed benchmark with no training split—models are evaluated directly on the 1,499 test questions. The two-fold cross-validation described in Section 3.2 applies to the compute-optimal strategy selection for search and revision experiments, not to the ARM-Thinker evaluations. For ARM-Thinker's own GRPO training, the paper does not describe a held-out validation set or specify how training steps were selected to avoid overfitting.

Main Quantitative Results

Reward Modeling Benchmarks (Table 2)

ARM-Thinker-7B achieves 67.8% accuracy on VL-RewardBench (multi-modal), 59.6% on RewardBench-2 (text-only), and 64.6% on ARMBench-VL (agentic, the proposed benchmark), yielding an average of 64.0% across the three benchmarks. This represents a +16.2 percentage point improvement over the Qwen2.5-VL-7B base model's average of 47.8% (a relative improvement of approximately 34%).

On VL-RewardBench, the gain over the base model is +17.7 points (50.1% → 67.8%), with the largest component-level improvement coming from the "General" subcategory (+18.0 points, from 37.7% to 55.7%), while "Hallucination" detection improves by +23.3 points (48.7% → 72.0%) and "Reasoning" evaluation sees a smaller +4.4 point gain (60.4% → 64.8%). On RewardBench-2 (text-only), the gain is +12.5 points (47.1% → 59.6%). On ARMBench-VL, the gain is +18.5 points (46.1% → 64.6%), with balanced improvements across Fine-Grained perception (+22.2 points, from 45.4% to 67.6%), Instruction Following (+28.4 points, from 45.4% to 73.8%—note the paper shows 45.4% as the baseline for IF, though Table 2 lists the baseline overall as 46.1%, suggesting per-task baseline numbers are slightly different from the aggregate), and Document Understanding (+11.3 points, from 41.1% to 52.4%).

Comparative analysis across models. ARM-Thinker-7B outperforms GPT-4o on VL-RewardBench (67.8% vs. 65.8%) and is roughly comparable on ARMBench-VL (64.6% vs. 63.3%), despite being a 7B open-source model versus a proprietary system. On RewardBench-2, GPT-4o retains a lead (65.5% vs. 59.6%), suggesting the text-only reward modeling capability of the larger proprietary model is not fully matched. Among open-source models, ARM-Thinker-7B substantially outperforms UnifiedReward-7B (average 64.0% vs. 52.8%), InternVL3.5-8B (53.4%), and Qwen3-VL-8B (58.5%). The comparison with UnifiedReward-7B is particularly informative: UnifiedReward achieves 66.1% on VL-RewardBench (comparable to ARM-Thinker's 67.8%) but only 45.1% on RewardBench-2 and 47.4% on ARMBench-VL, showing a sharp drop in transfer from multimodal to text-only and agentic reward tasks. ARM-Thinker's more balanced profile (67.8%, 59.6%, 64.6%) suggests the agentic training produces reward capabilities that transfer more robustly across task formats.

What the ARMBench-VL subcategory scores reveal. The Instruction Following task shows the largest absolute gain (+28.4 points from baseline to 73.8%), which is notable because instruction-following data was explicitly excluded from GRPO training (Appendix A.2: "We do not include Multimodal Instruction Following task–related tool-use data here"). The paper attributes this to generalization: "once the model is trained with our framework, its tool-use capability generalizes naturally to such tasks." The Document Understanding task shows the smallest gain (+11.3 points to 52.4%), which is also the lowest absolute score among the three ARMBench-VL subcategories. This is consistent with the paper's characterization of document retrieval as a harder problem requiring multi-step sequential reasoning, and suggests that tool-use generalization is not uniform across tool families—the model learns to use image crop tools more effectively than document retrieval tools.

Tool-Use (Think-with-Images) Benchmarks (Table 3)

ARM-Thinker-7B achieves an average of 76.5% across the four think-with-images benchmarks, a +9.6 point improvement over the Qwen2.5-VL-7B baseline's 66.9%. Per-benchmark: V* Bench improves from 75.4% to 86.4% (+11.0 points), HRBench-4K from 69.1% to 80.1% (+11.0 points), HRBench-8K from 64.6% to 73.7% (+9.1 points), and MME-RealWorld from 58.5% to 65.8% (+7.3 points). The pattern shows larger gains on benchmarks where the baseline was weaker (V* Bench, HRBench-4K) and smaller gains where the baseline was stronger or the task harder (MME-RealWorld).

Comparison with specialized tool-use models. ARM-Thinker-7B (76.5%) matches Mini-o3 (76.1%) and outperforms DeepEyes (72.5%), Pixel Reasoner (72.9%), and GPT-4o (57.7%). This is the paper's strongest evidence for the claim that tool-use capability can be acquired through reward-based optimization without curated demonstrations—Mini-o3, DeepEyes, and Pixel Reasoner were all explicitly trained on tool-use trajectories. ARM-Thinker's competitive performance against these models, despite never being shown a demonstration of correct tool use during GRPO training, supports the paper's argument that appropriately structured reward signals can induce systematic tool-use strategies.

The gap between ARM-Thinker and the next-best open-source general-purpose model (Qwen3-VL-8B at 73.1%) is 3.4 points, which is smaller than the gap between ARM-Thinker and its own base model (9.6 points). This suggests that part of ARM-Thinker's gain comes from the base model's general capability improvements (Qwen3-VL-8B being a newer generation), but the majority comes from the agentic training.

Resolution-dependent pattern. The improvement on HRBench-4K (+11.0 points) is larger than on HRBench-8K (+9.1 points). The paper doesn't directly comment on this, but it is consistent with the hypothesis that tool-assisted zoom-in is more beneficial when the base model's native resolution is less adequate—at 4K, fine details are at the edge of the base model's perception, so zoom-in provides large gains; at 8K, the base model can already see more detail natively, so the marginal benefit of zoom-in is smaller.

General Reasoning Benchmarks (Table 4)

ARM-Thinker-7B achieves an average of 49.0% across six reasoning benchmarks, a +4.2 point improvement over the Qwen2.5-VL-7B baseline's 44.8%. The gains are concentrated on two benchmarks: WeMath improves from 35.2% to 46.1% (+10.9 points) and LogicVista improves from 44.1% to 52.8% (+8.7 points). The other four benchmarks show marginal improvements: MMMU (+2.2 points, 55.0% → 57.2%), MathVista (+2.4 points, 67.8% → 70.2%), MathVision (+0.5 points, 25.4% → 25.9%), and MathVerse (+0.5 points, 41.1% → 41.6%).

Interpretation of the uneven gains. The large improvements on WeMath and LogicVista, coupled with negligible improvements on MathVision and MathVerse, suggest that the benefit of agentic reward training for general reasoning is task-specific. The paper attributes these gains to the fact that "judging response quality requires careful logical analysis and error detection," which transfers to the reasoning skills tested by WeMath and LogicVista. However, the near-zero gains on MathVision and MathVerse (both math reasoning benchmarks) suggest this transfer is not universal—math reasoning may rely on capabilities (symbolic manipulation, theorem application) that agentic judgment training does not develop.

Comparison with the larger Gemma-3-27B. ARM-Thinker-7B (49.0%) outperforms Gemma-3-27B (47.3%) on average, driven primarily by the WeMath and LogicVista advantages. On MMMU and MathVista, Gemma-3-27B leads (64.9% vs. 57.2% and 59.8% vs. 70.2% respectively—note: MathVista shows ARM-Thinker ahead, so Gemma-3-27B leads only on MMMU). This mixed comparison against a ~4× larger model mirrors the pattern seen in the prior sections' compute-optimal scaling analysis: test-time compute strategies (here, agentic training) can compensate for model scale on some task types but not others.

Ablation Studies and Robustness Checks

Tool Use vs. No Tool Use (Table 5). The paper compares both Qwen2.5-VL-7B and ARM-Thinker-7B with tool calling disabled ("default") and enabled ("w/ tool") across ARMBench-VL, V* Bench, HRBench-4K, and HRBench-8K. Three findings stand out. First, the base model (Qwen2.5-VL-7B) does not benefit from tool access—its performance degrades when tools are enabled (ARMBench-VL: 46.1% → 44.3%, V* Bench: 75.4% → 50.3%, HRBench-4K: 69.1% → 60.1%, HRBench-8K: 64.6% → 51.8%). The paper attributes this to the base model lacking "training signals that associate tool use with improved performance" and rarely invoking tools without explicit supervision. Second, ARM-Thinker maintains strong performance even without tools (59.2% on ARMBench-VL vs. the baseline's 46.1%, suggesting substantial capability gains beyond tool use alone), and achieves consistent additional gains when tools are enabled (59.2% → 64.6% on ARMBench-VL, +5.4 points). Third, the tool-enabled gains for ARM-Thinker are moderate in absolute terms (+3.2 to +5.4 points) compared to the gap between ARM-Thinker (no tools) and the baseline (no tools), which is +13.1 points on ARMBench-VL. This suggests that a large fraction of ARM-Thinker's improvement comes from better judgment capability acquired during training, not solely from tool use at inference time.

Reward Function Design (Figure 4). Comparing three reward designs during GRPO training reveals distinct failure modes. The Only Acc & Fmt Reward variant (no tool awareness) achieves the lowest tool-call rate (~0.7 calls/trajectory) and plateaus at 77.5% accuracy, confirming that without explicit tool encouragement, the model under-uses tools. The Fixed Tool Reward variant (constant bonus for any tool call) drives tool-call rate to ~1.15 calls/trajectory but achieves only 78.5% accuracy, confirming that indiscriminate tool rewards cause over-use without accuracy gains. The ARM-Thinker Reward (the proposed two-stage adaptive design) achieves the highest accuracy (above 78.5%, exact value not specified) with a tool-call curve that stabilizes at ~1.12 and contracts slightly after step 54, indicating learned discrimination between useful and gratuitous tool use. The contraction after step 54 is particularly informative: it suggests the model is learning to reduce tool use when it doesn't help, rather than monotonically increasing tool calls. The paper does not report statistical significance for any of these comparisons.

Omission of instruction-following data from GRPO training (Appendix A.2). The paper explicitly notes that Multimodal Instruction Following task data is excluded from GRPO training ("their core challenge lies in selecting the appropriate tool rather than executing complex tool-use logic"), yet ARM-Thinker achieves 73.8% on the Instruction Following subcategory of ARMBench-VL (+28.4 points over baseline). This is presented as evidence for generalization, but the paper does not provide an ablation showing what performance would have been with instruction-following data included in GRPO. The generalization claim is supported by the result but not tested through controlled exclusion.

Revision model / ReST^EM results. The paper mentions in Appendix K that a ReST^EM-trained revision model showed degraded performance (sequential revisions "substantially hurt" accuracy), but this ablation relates to a separate line of experiments (the revision model described in Section 6 of the original paper's framework) and does not appear to be part of the ARM-Thinker evaluation. The provided paper content does not include this appendix in full, so the exact numbers cannot be verified.

Missing ablations. Several ablations that would strengthen the paper are absent. There is no ablation on the number of GRPO stages (would a single stage with a blended reward work as well as two separate stages?). There is no ablation on the cold-start data quantity (how many trajectories are needed for the model to learn the tool-calling format?). There is no ablation on the difficulty filtration threshold (what happens if harder or easier samples are included?). There is no ablation varying which tool families are included during training (does training with image tools transfer to document tools, as the instruction-following result suggests, or are they independent?). There is no ablation on the backbone model (would the same training pipeline work with Qwen2.5-VL-3B or Qwen2.5-VL-72B?).

Critical Assessment

The experimental results provide substantial evidence for the paper's headline claim—that equipping a reward model with agentic tool-use capability improves judgment accuracy—but the strength of support varies across the specific claims, and several important boundaries of the contribution remain unexamined.

Claim 1: ARM-Thinker achieves +16.2% average improvement on reward modeling benchmarks.

The numbers in Table 2 directly support this: 64.0% average vs. 47.8% baseline, a difference of 16.2 percentage points. However, this average masks substantial variance across benchmarks—+17.7 points on VL-RewardBench, +12.5 points on RewardBench-2, +18.5 points on ARMBench-VL. The averaging is reasonable but the constituent benchmarks measure different capabilities (multimodal judgment, text-only judgment, agentic judgment), and the improvement is notably smaller on text-only tasks. This is expected (ARM-Thinker's agentic training is multimodal), but it means the "average" should be interpreted as summarizing heterogeneous gains rather than representing a uniform capability improvement.

More critically, the comparison between ARM-Thinker and baseline models is not compute-matched. ARM-Thinker uses a multi-turn agent loop with potentially multiple tool calls per judgment, while baselines like UnifiedReward-7B and Qwen3-VL-8B produce judgments in a single forward pass. The exact compute disparity is not quantified—the paper doesn't report the average number of tool calls, tokens generated, or wall-clock time per judgment for ARM-Thinker versus baselines. The +16.2% accuracy gain should therefore be understood as coming at an unknown but potentially substantial inference cost premium. This is not a flaw in the experimental design so much as an incomplete characterization: the paper demonstrates that accuracy can be improved through agentic tool use, but does not quantify the cost of that improvement.

Claim 2: ARM-Thinker outperforms GPT-4o on reward modeling accuracy.

This claim holds on VL-RewardBench (67.8% vs. 65.8%) and ARMBench-VL (64.6% vs. 63.3%), but not on RewardBench-2 (59.6% vs. 65.5%). The paper's abstract states that ARM-Thinker "achieves performance competitive with, and in some cases superior to, proprietary models like GPT-4o," which is an accurate characterization of these mixed results. The "superior to" phrasing applies to two out of three benchmarks. The text-only result (GPT-4o leads by 5.9 points) is important context that the abstract's framing downplays—ARM-Thinker's advantage is specifically in multimodal and agentic settings, and the larger proprietary model retains an edge in pure text judgment.

A significant caveat: the GPT-4o results are reported for a specific evaluation protocol (likely the standard non-agentic format, since GPT-4o is not described as having been configured for the ARM-Thinker agent loop). If GPT-4o were evaluated in an agentic setup with the same tool access, it might perform differently. The paper does not report GPT-4o with tool access, so the comparison is between ARM-Thinker (with tools) and GPT-4o (without tools, presumably). This makes the outperformance claim simultaneously impressive (a 7B model beating a frontier proprietary system) and methodologically asymmetric (fair comparison would give both models the same tool access).

Claim 3: ARM-Thinker matches or exceeds specialized tool-use models (Mini-o3, DeepEyes, Pixel Reasoner) despite not using curated tool-use demonstrations.

Table 3 supports this: ARM-Thinker (76.5%) edges out Mini-o3 (76.1%) and clearly outperforms DeepEyes (72.5%) and Pixel Reasoner (72.9%). The claim that ARM-Thinker achieves this "without explicit tool-use supervision in its training data" is accurate with respect to the GRPO stages—the model receives only outcome-based rewards during reinforcement learning. However, the cold-start SFT stage does include tool-use trajectories generated by larger models, which constitutes a form of implicit supervision on tool-use behavior. The distinction the paper draws is between learning tool-use strategies from demonstrations (what DeepEyes et al. do) versus learning tool-use format and basic mechanics from SFT data and then discovering effective strategies through RL. This is a meaningful distinction, but the paper slightly overstates the "without explicit tool-use supervision" claim—the model does see tool-use examples during SFT, even if the strategic decisions (when to call which tool, how many times) are learned through RL.

The results are reported as point estimates without confidence intervals, and the differences between ARM-Thinker and Mini-o3 (0.4 points) are small relative to what is likely substantial benchmark variance with test sets of a few hundred to a few thousand questions. The paper does not report whether this difference is statistically significant.

Claim 4: Agentic reward training transfers to improved general reasoning (WeMath: +10.9%, LogicVista: +8.7%).

Table 4 supports these specific numbers, but the pattern of results raises questions about the mechanism. If agentic reward training improves general reasoning through "careful logical analysis and error detection," why do MathVision (+0.5%) and MathVerse (+0.5%) show essentially no improvement? Both are math reasoning benchmarks that should benefit from careful logical analysis. The paper offers no explanation for this differential transfer. Without additional experiments (e.g., analyzing error types on MathVision vs. WeMath, or examining whether ARM-Thinker's tool-use patterns during evaluation differ across benchmarks), the transfer claim remains correlational rather than causal. It is possible that the gains on WeMath and LogicVista reflect something other than improved reasoning—for instance, the training data distribution may overlap more with these benchmarks, or the specific reasoning patterns required by WeMath are more similar to the evidence-evaluation patterns learned during reward model training.

Claim 5: The two-stage reward design avoids both under-use and over-use of tools.

The ablation in Figure 4 provides clean evidence for this claim: the "Only Acc & Fmt" variant under-uses tools, the "Fixed Tool Reward" variant over-uses tools, and the ARM-Thinker reward achieves both higher accuracy and a stabilizing tool-call rate. This is the most internally valid result in the paper because it compares variants within the same training framework, holding all else equal. However, the ablation is reported only for a single training run (presumably—the paper doesn't mention multiple seeds), and the accuracy difference between the ARM-Thinker reward and the Fixed Tool Reward appears small (both around 78-79% at the final step, with ARM-Thinker slightly ahead). The more interesting result is the tool-call curve dynamics (stabilization and contraction for ARM-Thinker, continued increase for Fixed Tool Reward), but the paper doesn't quantify the statistical reliability of this pattern. If the tool-call contraction after step 54 is noise rather than a genuine policy shift, the interpretation that the model has learned an optimal policy is weakened.

Weaknesses and missing evidence.

Single benchmark suite for agentic evaluation. ARMBench-VL is the paper's own benchmark, and while the construction process is well-documented, the evaluation of ARM-Thinker on ARMBench-VL is not an independent test—the model was trained on data sourced from the same task families (DeepEyes, MP-DocVQA, MM-IFEngine) that ARMBench-VL draws from. The paper states that GRPO data excludes instruction-following data, and that the benchmark was reconstructed with question rewriting and response regeneration, but the underlying task distributions overlap. An evaluation on a fully independent agentic reward benchmark (which does not exist at the time of this work, to be fair) would provide a stronger test of generalization.

No latency or cost accounting. As noted above, the paper reports accuracy without any compute-normalization. For a system whose core innovation is more computation at inference time (through iterative tool calls), this omission is significant. A practical deployment would need to know whether the +16.2% accuracy gain requires 2×, 10×, or 100× more inference compute. The paper provides no data to answer this question.

Single backbone model (Qwen2.5-VL-7B). All ARM-Thinker results are on a single model family and scale. The paper claims the framework is "backbone-agnostic" (Section 1), but this claim is not tested. Would the same training pipeline work with InternVL3, LLaVA, or a text-only LLM? The paper provides no evidence.

No human evaluation. All results are against benchmark ground truth, which—for ARMBench-VL—was itself generated by a large model (Qwen3-VL-235B-A22B-Thinking). The paper does not validate that ARMBench-VL's ground-truth judgments align with human preferences. If the generating model has systematic biases in its judgment of what constitutes a better response, ARM-Thinker may be learning to reproduce those biases rather than genuine human-aligned judgment.

The difficulty estimation cost problem from Section 3.2. The paper's prior sections identified that estimating question difficulty requires generating 2048 samples per question—a cost that makes the compute-optimal framework impractical without cheaper difficulty estimation. ARM-Thinker inherits a related problem: deciding when to call tools and how many to call is a difficulty-estimation problem in microcosm. The paper does not analyze how much of ARM-Thinker's inference budget is spent on "exploratory" tool calls that don't contribute to the final judgment, or whether ARM-Thinker could be made more efficient by learning a termination policy.

What would strengthen the paper.

• An evaluation of ARM-Thinker with tool access disabled on the same benchmarks, isolating the contribution of tool use at inference time from the contribution of improved base capabilities from multi-stage training. Table 5 partially provides this, but only on a subset of benchmarks.

• A compute-matched comparison where non-agentic baselines are given equivalent inference budgets (e.g., best-of-N sampling, longer chain-of-thought) to match ARM-Thinker's multi-turn agent loop cost.

• Evaluation on a held-out agentic reward benchmark not constructed from the same source datasets as the training data.

• Direct measurement of the correlation between ARM-Thinker's judgments and human judgments on a sample of ARMBench-VL items, to validate that the benchmark's ground truth (generated by Qwen3-VL-235B) aligns with human preferences.

• Ablations varying the backbone model to test the claimed backbone-agnostic property.

• Statistical significance testing or confidence intervals for the main benchmark results, particularly for close comparisons like ARM-Thinker vs. GPT-4o or ARM-Thinker vs. Mini-o3.

Overall assessment. The experiments demonstrate that ARM-Thinker achieves substantial accuracy gains over its base model and competitive or superior performance relative to existing open-source reward models and specialized tool-use models. The evidence is strongest for the claim that agentic tool use improves multimodal judgment accuracy. It is moderate for the claim that these gains transfer to general reasoning (uneven transfer). It is suggestive but incomplete for the claim that tool-use strategies emerge from reward design without demonstrations (cold-start SFT provides implicit supervision). The evidence is weak for claims about computational practicality (no cost accounting) and backbone-agnosticism (single model tested). The paper's framing of tool use as emergent and the two-stage reward design as a general solution to exploration-competence tradeoffs is conceptually compelling, but the experiments do not test these ideas beyond the specific instantiation of ARM-Thinker.

6. Limitations and Trade-offs

6.1 Inference Cost Is Unquantified, Making Practical Utility Difficult to Assess

The assumption or constraint. ARM-Thinker improves judgment accuracy by enabling multi-turn tool-augmented reasoning—the model iteratively calls tools to retrieve document pages, crop image regions, and run programmatic checks before producing a final judgment. The paper reports all accuracy results without accounting for the computational cost of these additional inference turns. There is no measurement of the average number of tool calls per judgment, the total tokens generated, the wall-clock latency, or any FLOPs-normalized comparison against baselines that produce judgments in a single forward pass.

The paper acknowledges this implicitly in the context of difficulty estimation (Section 3.2), noting that "our experiments do not account for this cost largely for simplicity," but this same omission applies to the ARM-Thinker evaluation itself. The ablation in Table 5 shows ARM-Thinker without tools achieves 59.2% on ARMBench-VL and with tools achieves 64.6%—a +5.4 point gain—but does not report how many additional tool calls (and therefore how much additional compute) that +5.4 points costs.

The consequence. A practitioner considering deployment cannot determine whether ARM-Thinker's accuracy gains are worth the inference overhead. If ARM-Thinker averages 3 tool calls per judgment, each requiring a full forward pass through the 7B model plus tool execution (CLIP embedding computation, ChromaDB queries, image resizing and concatenation), the per-judgment cost could be 4× or more than a single-pass baseline like Qwen3-VL-8B. The paper demonstrates that accuracy can be improved through agentic tool use, but provides no data to assess whether the improvement is cost-effective.

This omission is particularly significant because ARM-Thinker is positioned as a reward model—a component used to provide training signals in RLHF pipelines. In that context, slower reward model inference directly translates to slower RL training loops. A reward model that is 16% more accurate but 4× slower per judgment may or may not be a net win for end-to-end policy improvement, depending on whether the improved reward signal accelerates policy convergence enough to offset the per-step slowdown. The paper provides no evidence to evaluate this tradeoff.

What evidence exists in the paper. None. No latency measurements, no token counts, no FLOPs comparisons, and no end-to-end RLHF experiments using ARM-Thinker as the reward model appear in the paper. The tool-call frequency curves in Figure 4 (right panel) show ARM-Thinker stabilizes at ~1.12 tool calls per trajectory during GRPO training, but this is from the training process, not from the final evaluation protocol, and the paper does not report per-benchmark tool-call statistics.

Mitigation status. Not addressed. The paper does not discuss inference cost as a limitation, nor does it suggest efficiency improvements (e.g., tool-call caching, early termination policies, distillation of agentic judgments into a single-pass model) as future work.

6.2 Single Backbone Model and Training Pipeline Limit Generality Claims

The assumption or constraint. ARM-Thinker is built exclusively on Qwen2.5-VL-7B. The cold-start trajectory generation uses "stronger LVLMs" (presumably larger Qwen variants), the SFT data is generated from Qwen3-VL-235B-A22B-Thinking, and the GRPO training uses Qwen2.5-VL-7B as the policy model. Every component of the pipeline—base model architecture, tool-use capability emergence, reward function design, data generation—is validated on a single model family at a single scale (7B parameters).

The paper claims the framework is "backbone-agnostic and modality-agnostic, enabling seamless extension to new tools and tasks" (Section 1), but this claim is entirely untested. No experiments demonstrate that the same training pipeline works with InternVL, LLaVA, or a text-only LLM backbone. No experiments show that the GRPO reward design generalizes to 3B or 72B parameter variants.

The consequence. A practitioner who wants to deploy ARM-Thinker on a different model family (e.g., for compatibility with their existing infrastructure, or to leverage a model with better domain-specific pretraining) has no evidence that the reported gains will transfer. The tool-use emergence claim—that reward-based optimization induces systematic tool-use strategies without curated demonstrations—may depend on specific properties of Qwen2.5-VL-7B's pretraining (e.g., its instruction-following capability, its zero-shot tool-calling aptitude after SFT). A model with weaker instruction-following or different visual encoding architecture might fail to learn effective tool use from the same reward structure.

Furthermore, the cold-start data generation depends on a 235B model (Qwen3-VL-235B-A22B-Thinking) to produce high-quality trajectories. If a practitioner wanted to apply ARM-Thinker to a non-Qwen backbone, they would need either (a) a comparably large model from the same family to generate cold-start trajectories, or (b) evidence that smaller models or alternative data sources can serve the same role. The paper provides neither.

What evidence exists in the paper. All results in Tables 2–5 are on ARM-Thinker-7B, built on Qwen2.5-VL-7B. The baselines include other model families (InternVL3, Qwen3-VL, GPT-4o) evaluated as off-the-shelf reward models, but no variant of ARM-Thinker is trained on these backbones. The paper includes no ablation varying the backbone model.

Mitigation status. Not addressed. The "backbone-agnostic" claim appears in Section 1 as an asserted property of the framework but is not validated experimentally. No future work is suggested to test this claim.

6.3 Training Data and Evaluation Benchmark Share Task Distributions, Limiting Independence of Assessment

The assumption or constraint. ARM-Thinker's training data for tool-use (SFT cold-start and GRPO) is drawn from DeepEyes (image crop/zoom-in), MP-DocVQA (document retrieval), and MM-IFEngine (instruction following). ARMBench-VL is constructed from V*Bench/VisualProbe (fine-grained perception), MMlongbench-doc (long document QA), and MM-IFEval (instruction following). While the paper states that the benchmark was "reconstructed" with question rewriting, response regeneration using Qwen3-VL-235B, and filtering, the underlying task distributions—high-resolution images requiring fine-grained inspection, multi-page documents requiring page retrieval, multimodal instructions with constraints—are the same families from which training data was sourced.

The paper notes that MM-IFEngine data was excluded from GRPO training (Appendix A.2) and that the instruction-following performance on ARMBench-VL demonstrates generalization: "once the model is trained with our framework, its tool-use capability generalizes naturally to such tasks without requiring explicit inclusion of this data." However, instruction following constituted only 489 of 1,499 ARMBench-VL questions. The other 1,010 questions (fine-grained perception and document QA) are drawn from task families that were included in training, and the paper does not provide a held-out evaluation set from a fully independent distribution.

The consequence. ARM-Thinker's performance on ARMBench-VL may overestimate its capability on genuinely novel agentic reward tasks. The model has been trained on data that teaches it to (a) recognize when fine-grained image inspection is needed, (b) formulate retrieval queries for document pages, and (c) select appropriate instruction-checking tools. ARMBench-VL tests these exact skills on held-out instances but not held-out task types. A practitioner deploying ARM-Thinker to judge outputs on a new type of tool-augmented task—say, verifying claims that require querying a database or executing code—has no evidence that the model's tool-use strategy generalizes beyond the three tool families seen during training.

The instruction-following generalization result (73.8% on ARMBench-VL IF subcategory, +28.4 points over baseline, despite exclusion from GRPO) provides some evidence for cross-task transfer within the same tool family (instruction-checking tools). But this is a weak test of generalization: the model saw instruction-following tools during SFT cold-start, and the tool interfaces are structurally identical to what was used in training for other tasks. It would be more surprising if the model failed to use tools it had already learned to call.

What evidence exists in the paper. Section 4.2 and Appendix Section F document the benchmark construction process and its relationship to training data sources. Tables 2 and 5 report ARMBench-VL results. No evaluation on an independently developed agentic reward benchmark is provided.

Mitigation status. Partially addressed through the instruction-following data exclusion, which provides a within-benchmark test of generalization. However, this only partially mitigates the concern—it shows transfer to unseen instances, not to unseen task families or tool types. The paper does not suggest evaluation on independent benchmarks as future work.

6.4 No Validation That Benchmark Ground Truth Aligns with Human Judgment

The assumption or constraint. ARMBench-VL's ground-truth labels (which response is correct, which response is better) are constructed algorithmically, not through human annotation. For fine-grained perception and document QA tasks, the correct answer is derived from the original dataset's ground truth, and incorrect responses are generated by Qwen3-VL-235B-A22B-Thinking. The paper states that the generating model is instructed "not to mention correctness-related cues in its responses" and that generated pairs are filtered "to remove any items where the correct answer is mistakenly included in the negative responses" (Section 4.2).

However, these quality control steps only ensure that negative responses don't accidentally contain the correct answer text. They do not validate that the negative responses represent plausible errors that a human would consider clearly worse than the correct response, or that the benchmark's ranking of response quality aligns with human preferences. The 235B model might generate negative responses that are factually wrong but stylistically indistinguishable from correct ones—making the discrimination task artificially hard in ways that don't reflect real-world reward modeling difficulty. Alternatively, it might generate negative responses with subtle giveaway cues (e.g., hedging language, less confident phrasing) that a model could exploit without genuine evidence-grounded reasoning.

The consequence. ARM-Thinker's measured accuracy on ARMBench-VL may not reflect its ability to produce human-aligned judgments. If the benchmark's ground truth diverges systematically from human preferences—for instance, if the 235B model's "incorrect" responses are actually reasonable alternative answers that a human might accept—then training and evaluating against this ground truth could produce a reward model that is well-optimized for the benchmark but poorly aligned with human judgment in deployment.

This concern is not hypothetical. The RLHF literature has extensively documented that reward models can over-optimize against proxy metrics that diverge from human preferences. ARM-Thinker is trained to maximize accuracy on automatically constructed preference pairs and evaluated on automatically constructed benchmark labels—the entire supervision chain is model-generated. Without a human validation study, there is no evidence that improvements on ARMBench-VL translate to improvements in human-rated judgment quality.

What evidence exists in the paper. Section 4.2 describes the benchmark construction and quality control process. No human evaluation of ARMBench-VL labels is reported. No correlation between ARM-Thinker's judgments and human judgments on any subset of ARMBench-VL items is measured.

Mitigation status. Not addressed. The paper does not discuss the lack of human validation as a limitation, nor does it suggest human evaluation studies as future work. Given that ARM-Thinker is positioned as a reward model for aligning VLMs with human preferences (Section 1: "Reward models are critical for aligning vision-language systems with human preferences"), the absence of any human judgment validation is a significant gap.

6.5 The Cold-Start Pipeline Depends on a 235B Model, Making the Approach Impractical for Many Practitioners

The assumption or constraint. The data generation pipeline (Section 3.2) uses Qwen3-VL-235B-A22B-Thinking—a 235-billion-parameter model—for two critical functions: generating incorrect but plausible responses for preference pair construction, and generating cold-start tool-use trajectories that teach the 7B model the think–act–observe format. The paper also notes that "stronger LVLMs" are used for trajectory generation in the agent loop inference step, without specifying exact models.

The entire training pipeline therefore assumes access to a model roughly 33× larger than the target model being trained. This is not a minor implementation detail—the cold-start trajectories are the bridge between the base model's lack of tool-use capability and the GRPO stages that refine that capability. Without high-quality cold-start data showing correct tool-calling formats, the base model (which "typically exhibits limited zero-shot competence in executing novel tool invocations," Section 3.3.1) cannot begin the exploration process that GRPO would shape.

The consequence. A practitioner who wants to train an ARM-Thinker-style reward model but lacks access to a 235B vision-language model (due to hardware constraints, API costs, or model availability) cannot replicate the pipeline as described. The paper provides no evidence that smaller models could serve as cold-start trajectory generators, or that alternative bootstrapping approaches (e.g., human-written tool-use examples, synthetic data from weaker models with more filtering, or entirely RL-based exploration from scratch) could replace the 235B model's role.

This limitation is particularly acute for the document retrieval and image zoom-in tools, which require the trajectory generator to understand spatial reasoning and multi-page navigation well enough to produce coherent chains of tool calls. A weaker generator might produce trajectories that pass the format filter but contain nonsensical tool-use patterns (e.g., zooming into irrelevant regions, retrieving unrelated pages), which would teach the student model to mimic the form of tool use without the substance.

What evidence exists in the paper. Section 3.2 and Appendix A.2 describe the data generation pipeline and the models used. The paper does not ablate the cold-start model size or compare trajectories generated by different model scales. The iterative refinement process ("progressively improving the model's reasoning depth and tool-use proficiency" through multiple rounds of generation and filtering) is mentioned but not detailed—it is unclear whether the 235B model is needed for all rounds or only the first.

Mitigation status. Minimally addressed. The paper's claim that tool-use strategies are "emergent" from GRPO training (Section 5.3) implies that the cold-start data is primarily for format learning rather than strategy learning, but this is not tested. The paper does not suggest lower-cost alternatives to the cold-start pipeline as future work, though the general direction of reducing data requirements is implied by the scalability framing in Section 1.

7. Implications and Future Directions

How This Work Changes the Landscape

ARM-Thinker changes the reward modeling landscape by establishing that judgment quality on complex multimodal tasks cannot be separated from the evidence-gathering process that enables it. This is not an incremental improvement—it is a reframing of what a reward model fundamentally is. Prior work conceptualized reward models as function approximators mapping (question, image, response) to (score). ARM-Thinker redefines them as sequential decision-making agents that construct verifiable chains of evidence through tool use. The output is no longer merely a score; it is a fully auditable trajectory showing what evidence was gathered, in what order, and how it supports the conclusion.

The magnitude of this shift becomes clear when examining what the paper makes newly possible. Before ARM-Thinker, the field had no way to verify whether a reward model's judgment on a long-document QA question was grounded in actual evidence or hallucinated. A judge that confidently declares "the zip code is 89104, as shown on page 5" could be entirely wrong, and no existing benchmark could detect this because no existing benchmark provided tools for evidence retrieval. ARM-Thinker makes this verification gap visible and measurable—ARMBench-VL quantifies it as the 18.5-point accuracy gap between the base model (46.1%) and ARM-Thinker (64.6%) when tools are available and needed. This diagnostic capability alone shifts the field's standards: reward models must now demonstrate not just judgment accuracy but the meta-cognitive ability to know what evidence is missing and how to obtain it.

The paper also resolves a tension that has been brewing beneath the surface of the RLHF literature. On one hand, reward model training emphasizes coverage—building models that generalize across diverse prompts and response styles. On the other hand, the most reliable forms of verification are programmatic and tool-based (unit tests for code, symbolic evaluation for math, constraint checkers for formatting). Prior work treated these as separate paradigms: learned reward models for open-ended quality, rule-based verifiers for objective correctness. ARM-Thinker unifies them. The model learns to invoke rule-based verifiers (the 19 instruction-following check tools) as instruments within a learned reasoning policy. The unification is bidirectional: the tools provide deterministic reliability where it matters (word counts, substring matching), while the learned policy provides the judgment to know when those checks are relevant and how to weigh their results against other evidence. The 73.8% instruction-following accuracy on ARMBench-VL—achieved without explicit GRPO training on instruction-following data—demonstrates that this unification is not merely architecturally possible but functionally effective.

The finding that tool-use strategies can be induced through reward design rather than curated demonstrations (Section 5.3, Table 3) has implications for tool-augmented AI broadly, beyond reward modeling. The dominant paradigm for tool use—exemplified by DeepEyes, Pixel Reasoner, and Mini-o3—has been demonstration-based: collect examples of correct tool-use trajectories, train by imitation. This approach faces a scalability ceiling: each new tool, each new task, each new domain requires new curated demonstrations. ARM-Thinker's result—matching Mini-o3 (76.5% vs. 76.1%) without explicit tool-use demonstrations during GRPO—suggests an alternative path. If the same outcome-reward structure (format + correctness + tool success) generalizes to new tool families, then adding a new tool to a system requires only defining its interface and ensuring the reward function can detect when it contributes to correct outcomes. No manual trajectory annotation, no curated examples. The 19 instruction-following check tools—which were added to the toolkit without GRPO training data and yet were used effectively at test time—provide preliminary evidence for this scalability claim.

The paper also reorients research priorities for the RLHF community. Before ARM-Thinker, improving reward models meant improving their static judgment accuracy—better training data, longer reasoning chains, larger models. ARM-Thinker demonstrates that the bottleneck on complex tasks is not judgment reasoning but evidence access. A reward model with perfect reasoning but no way to retrieve the relevant document page will hallucinate an answer; a reward model with mediocre reasoning but reliable evidence-gathering tools can reach the correct judgment by inspecting what it retrieves. This shifts the optimization target from "improve the judge's reasoning" to "improve the judge's evidence-gathering process." It makes verifier robustness—ensuring that tool outputs are correctly interpreted and not over-optimized against—as important as verifier accuracy.

Research directions that become more attractive after this work:

• Tool-augmented evaluation more broadly: if reward models benefit from tools, so might other evaluators (safety classifiers, factuality checkers, automated red-teaming).
• Learned termination and tool-selection policies: the paper shows that tool-call frequency stabilizes and contracts with appropriate reward shaping (Figure 4), suggesting that meta-reasoning about when to stop gathering evidence is learnable and important.
• Distillation of agentic judgments: if agentic reward models produce better judgments, can those judgments be distilled into a faster single-pass model for deployment?
• Multi-agent evaluation: if a single reward model with tools improves accuracy, what about multiple specialized evaluators with different toolkits, coordinated through debate or consensus?

Research directions that become less attractive (at least for complex multimodal tasks):

• Further optimization of single-pass reward models on static benchmarks: the 18.5-point gap on ARMBench-VL between the base model and ARM-Thinker, which narrows substantially for non-agentic models, suggests diminishing returns from improving static judgment accuracy without addressing evidence access.
• Pure demonstration-based tool-use training: ARM-Thinker's competitive performance against demonstration-trained models, achieved through reward-based optimization, suggests that collecting expensive tool-use demonstrations may not be the most efficient path to tool-use competence.

Follow-Up Research This Work Enables

Directly measuring the inference cost of agentic reward modeling. The paper reports accuracy without any compute normalization. A critical follow-up would instrument ARM-Thinker during ARMBench-VL evaluation to record: average number of tool calls per judgment, distribution of tool calls by task type (do document QA questions require more retrieval calls than fine-grained perception questions require zoom calls?), total tokens generated per judgment, and wall-clock latency. These measurements would produce a cost-per-accuracy-gain curve that practitioners need for deployment decisions. The natural comparison points are: (a) ARM-Thinker with tools vs. without tools (Table 5 already shows this, but without cost data), (b) ARM-Thinker vs. the base model with equivalent inference budget (e.g., giving Qwen2.5-VL-7B best-of-4 or best-of-8 sampling with majority vote, spending the compute that ARM-Thinker spends on tool calls on parallel sampling instead), and (c) ARM-Thinker vs. GPT-4o with the same tool access, which would reveal whether the open-source agentic model's advantage persists when both models operate under identical inference protocols. A strong study would also measure how tool-call frequency varies with question difficulty—does ARM-Thinker learn to spend more evidence-gathering effort on harder questions, analogous to the compute-optimal scaling described in the prior paper's framework?

Training ARM-Thinker variants on different backbone models to test the claimed backbone-agnostic property. The paper asserts that ARM-Thinker's framework is "backbone-agnostic and modality-agnostic" (Section 1), but validates this on exactly one model (Qwen2.5-VL-7B). A direct test would replicate the entire training pipeline—cold-start trajectory generation, SFT, two-stage GRPO—on at least two additional backbones: one from a different model family (e.g., InternVL3-8B) and one at a different scale (e.g., Qwen2.5-VL-3B or Qwen2.5-VL-72B). The key question is whether the tool-use emergence observed with Qwen2.5-VL-7B depends on specific properties of that model's pretraining (instruction-following aptitude, spatial reasoning capability, zero-shot tool-calling after SFT) or whether it is a general consequence of the reward structure. A negative result—failure to learn effective tool use on a different backbone—would be highly informative: it would reveal that the emergent tool-use claim is backbone-dependent and that the cold-start data or reward function design needs adaptation for different model families. A positive result would substantially strengthen the paper's generality claims and make ARM-Thinker a drop-in recipe for any VLM.

Evaluating ARM-Thinker as an RLHF reward model in an end-to-end policy improvement loop. The paper evaluates ARM-Thinker as a standalone judge but never uses it to provide training signal for policy optimization. The canonical evaluation for a reward model is not its accuracy on a judgment benchmark but whether using it in RLHF produces a better policy than using an alternative reward model. A strong follow-up would take a base VLM policy (e.g., Qwen2.5-VL-7B, or a smaller model to make training tractable), run RLHF using ARM-Thinker as the reward model, and compare the resulting policy against one trained with (a) a non-agentic reward model (UnifiedReward-7B), (b) a rule-based verifier alone, and (c) a combination. The key metric is downstream task performance of the optimized policy, not the reward model's standalone accuracy. This experiment would address the most important open question the paper leaves: does the +16.2% improvement in reward model accuracy translate to faster or better policy learning, or does the increased inference cost of agentic reward evaluation bottleneck the RL loop enough to offset the accuracy gains? It would also reveal whether ARM-Thinker's tool-use behavior remains stable under distribution shift—the policy's outputs during RLHF will drift away from the static evaluation distributions, and the reward model must remain calibrated.

Stress-testing the generalization of tool-use to entirely novel tool families. ARM-Thinker learns to use three tool families (image crop, document retrieval, instruction checking). The instruction-following result—strong performance despite exclusion from GRPO training—provides within-framework evidence for generalization, but all three tool families are structurally similar (they take parameters, return text and/or images, follow the same calling convention). A genuine stress test would introduce at least one fundamentally different tool at evaluation time, never seen during any training stage (including SFT cold-start), and measure whether ARM-Thinker can learn to use it from a few-shot description alone. Candidates include: a code execution tool (the model can run Python to verify calculations), a web search tool (the model can query the internet for factual claims), or a database query tool (the model can execute SQL to check structured data). The experiment would provide the tool's function schema at inference time (as ARMBench-VL already does) and measure whether ARM-Thinker can incorporate the new tool into its evidence-gathering strategy without any training on that tool. This tests whether the agentic reasoning capability is genuinely a meta-skill (knowing how to use tools in general) or is specific to the tool interfaces seen during training. A negative result—ARM-Thinker fails to use a new tool effectively—would bound the generality claim and suggest that some minimal exposure during training (even if not GRPO-optimized) is necessary.

Validating ARMBench-VL ground truth against human judgments. The paper's entire evaluation chain—training data construction, benchmark label generation, and final accuracy measurement—uses model-generated supervision without human validation. A methodologically important follow-up would sample 200-300 ARMBench-VL items (stratified across the three task categories and difficulty levels), collect human judgments from multiple annotators (at least 3 per item to measure inter-annotator agreement), and compute the correlation between ARMBench-VL's ground-truth labels and human majority-vote judgments. The key numbers to report: (a) raw agreement rate between benchmark labels and human judgments, (b) Cohen's kappa or similar chance-corrected metric, (c) agreement broken down by task category (is the 235B model better at generating correct labels for fine-grained perception than for document QA?), and (d) whether ARM-Thinker's accuracy against benchmark ground truth correlates with its accuracy against human judgments (does the model that wins on ARMBench-VL also win in human evaluation?). This study is essential for calibrating how much to trust ARMBench-VL as an evaluation instrument. If human agreement is high (e.g., >85%), the benchmark is validated as a low-cost substitute for human evaluation. If agreement is low or systematically biased, the field needs to know this before ARMBench-VL becomes a standard evaluation.

Ablating the cold-start pipeline to determine the minimum viable bootstrapping. The paper's data pipeline depends on a 235B model for generating cold-start trajectories and negative responses. This creates a practical barrier: a researcher without access to a 235B VLM cannot replicate the pipeline. A systematic ablation would vary the cold-start trajectory generator across model scales (e.g., 7B, 32B, 72B, 235B) and measure the downstream ARM-Thinker performance after identical SFT and GRPO training on each. The critical question: is there a threshold model size below which cold-start trajectories are too low-quality to bootstrap tool-use learning, or does the three-dimensional filtering (format, accuracy, behavior) compensate for weaker generators by simply keeping fewer but still adequate trajectories? A related ablation would test whether the GRPO stages can succeed with no cold-start data at all—starting from the base model with only format-reward shaping—which would test the strong form of the emergent tool-use claim. Even if performance is lower, partial emergence without cold-start would be a striking result. This ablation would provide the recipe that the community actually needs to replicate and extend ARM-Thinker.

Practical Applications and Downstream Use Cases

RLHF training loops for vision-language models on complex multimodal tasks. The most direct application is replacing single-pass reward models with ARM-Thinker in the RLHF pipeline for VLMs. The paper's results show that ARM-Thinker achieves +18.5 points higher accuracy than its base model on ARMBench-VL (46.1% → 64.6%), with the largest gains on tasks requiring fine-grained evidence gathering (fine-grained perception: +22.2 points; instruction following: +28.4 points). In an RLHF setting, this means the reward signal guiding policy optimization would be substantially more reliable specifically on the types of tasks where single-pass reward models are weakest—long-document QA, high-resolution image understanding, multi-constraint instruction following. A VLM RLHF-trained with ARM-Thinker could plausibly learn to produce outputs that are more factually grounded and constraint-compliant because the reward model providing the training signal can actually verify those properties rather than hallucinating them.

Automated evaluation and filtering in VLM data generation pipelines. Many self-improvement and data-generation pipelines (STaR, ReST^EM, rejection sampling fine-tuning) use a reward model or verifier to filter model-generated outputs—keeping good ones for training, discarding bad ones. The quality of the filter directly bounds the quality of the resulting training data. ARM-Thinker's 64.6% accuracy on ARMBench-VL (vs. 46.1% for the base model) means it would correctly identify a substantially higher fraction of good outputs in a filtering scenario. On instruction-following tasks specifically, where the base model achieves 45.4% and ARM-Thinker achieves 73.8%, the improvement in filtering precision would be dramatic: imagine generating 100 candidate responses to a constrained instruction, roughly 50 of which actually satisfy the constraint. The base model's filter would correctly identify ~23 of those 50 (plus some false positives); ARM-Thinker's filter would correctly identify ~37, enabling roughly 60% more high-quality training data from the same generation budget. For organizations running large-scale data generation pipelines, this efficiency gain translates directly to reduced API costs or compute expenditure.

Deployment as a guardrail or output verifier in production VLM systems. Beyond training, ARM-Thinker could serve as a runtime verification layer that checks VLM outputs before they reach users. In high-stakes applications—medical image analysis where a VLM describes findings, legal document review where a VLM extracts clause language, financial report generation where a VLM produces numerical summaries—the cost of an incorrect output is high. ARM-Thinker's agentic verification provides an auditable evidence chain: rather than trusting that the VLM's output is correct, the system could run ARM-Thinker to independently verify the key factual claims in the output by retrieving the relevant document pages, zooming into the relevant image regions, and checking any stated constraints. The interpretability benefit—the model produces a full trajectory showing what evidence it checked—enables human oversight: a human reviewer can inspect the verification trace and confirm that the evidence actually supports the conclusion, without needing to re-derive the entire answer from scratch. The paper's case study in Figure 1(a) illustrates exactly this pattern: ARM-Thinker shows its work ("on page 5, the second image, the mailto address is listed as Carson City, NV 89706"), making the judgment transparent and contestable in a way that a single-pass model's output ("Answer 1 is better") is not.