PixelCraft: A Multi-Agent System for High-Fidelity Visual Reasoning on Structured Images

ArXiv: 2509.25185

🎯 Pitch

MLLMs can get chart reading wrong by misreading a single bar height—PixelCraft fixes this by giving models a dynamic visual scratchpad and pixel-precise tools. A fine-tuned grounding model drives classical CV operators to extract values accurately, while an agent debate loop with image memory lets the system backtrack and explore alternative reasoning paths when critics flag errors.

1. Executive Summary

This paper proposes PixelCraft, a multi-agent system for high-fidelity visual reasoning on structured images such as charts and geometric diagrams. Evaluated on the CharXiv, ChartQAPro, EvoChart, and Geometry3K benchmarks with GPT-4o, GPT-4.1-mini, and Claude 3.7 Sonnet as backbone MLLMs, PixelCraft integrates query-aware agent selection, role-driven agent discussion, and iterative self-correction with a planner-managed image memory — a "cognitive whiteboard" that selectively recalls prior visual states rather than streaming all historical images — alongside high-fidelity visual tool agents powered by a fine-tuned grounding model (Qwen2.5-VL-3B, trained on a curated 53k-sample synthetic corpus) whose pixel-level coordinates drive classical CV operators (e.g., subfigure cropping, legend-based masking). Across all three chart reasoning benchmarks and all tested backbones, PixelCraft consistently outperforms direct-answer, chain-of-thought, multi-agent debate, Reconcile, and the tool-augmented Refocus baselines — achieving gains of up to +9.5 percentage points (GPT-4.1-mini on CharXiv) and setting a new state of the art, while establishing that this non-linear, discussion-centric workflow delivers flexible high-fidelity reasoning only when grounded in precise visual localization and complementary critic verification.

2. Context and Motivation

The Core Problem: Structured Images Demand Precision That MLLMs Lack

The central challenge this paper tackles is deceptively specific but practically pervasive: current multimodal large language models (MLLMs) cannot reliably reason over structured images — charts, geometric diagrams, scientific plots — because they lack both the perceptual precision to extract fine-grained visual information and the cognitive flexibility to explore non-linear reasoning paths.

This distinction between natural and structured images is the paper's foundational framing (Section 1). Natural images — photographs of scenes, objects, people — are characterized by textures, local visual patterns, and semantic features like object categories and spatial relationships. Vision models have been optimized for decades to handle these, and modern MLLMs inherit this capability. Structured images are fundamentally different. They encode symbolic and structural elements: data point coordinates, bar heights relative to axis ticks, line connections between geometric vertices, numerical annotations, legend-to-color mappings. Interpreting these requires precise symbolic abstraction — reading a specific value from a specific subplot — rather than approximate pattern recognition. The paper is explicit about the stakes:

"While coarse visual features may suffice for understanding natural images, even subtle differences in structured images (e.g., slightly misreading the height of a single bar) can dramatically alter the interpretation and downstream reasoning."

This is not a marginal failure mode. In a multi-step reasoning chain — common in chart analysis (compare subfigure A to subfigure B, compute the difference, round to one decimal place) — an early perceptual slip cascades into an erroneous conclusion. The error is not in the reasoning but in the evidence on which the reasoning operates. The paper's Figure 1 illustrates exactly this: the base MLLM (using chain-of-thought) attempts to read values from a multi-panel chart and produces three different answers (0.3, 0.2, 0.1) depending on how it misinterprets the visual data, while PixelCraft arrives at the correct answer (0.1) through precise subfigure cropping and legend-based data masking.

Why This Matters: Real-World Deployment and Theoretical Significance

The importance of this problem extends in several directions:

Practical deployment of AI systems. Charts and structured diagrams are ubiquitous in professional and scientific communication. Financial analysts read candlestick charts, biomedical researchers interpret dose-response curves, engineers consult circuit diagrams, and data scientists explore multi-panel visualization grids. An MLLM that cannot reliably read a chart is effectively barred from these high-value applications. The benchmarks used in the paper reflect this real-world complexity: CharXiv includes realistic charts from arXiv papers with dense multi-panel layouts; ChartQAPro features human-authored questions requiring deep compositional reasoning; EvoChart covers real-world chart types with varied visual styles. Performance on these benchmarks is not merely an academic metric — it correlates with usefulness in document understanding, automated report generation, and scientific literature analysis.

The gap between capability and reliability. Modern MLLMs like GPT-4o and Claude 3.7 Sonnet appear to understand charts — they can describe trends, identify chart types, and answer simple questions. But as Table 1 shows, even the strongest backbone models (Claude 3.7 Sonnet) achieve only 67.1% accuracy on CharXiv's reasoning subset under direct-answer prompting. This means that on one-third of questions, the model produces wrong answers despite having the underlying reasoning capability. The bottleneck is not logical reasoning per se — it is the perceptual pipeline that feeds evidence into that reasoning. PixelCraft's contribution is showing that externalizing this pipeline into verifiable, high-fidelity tool operations closes a significant portion of this gap (73.9% for Claude 3.7 Sonnet + PixelCraft on CharXiv, a +6.8 percentage point gain).

A theoretical question about reasoning architectures. Beyond practical concerns, the paper engages a deeper question: what is the right architecture for visual reasoning? Is it an end-to-end neural model that internalizes all operations? Or a hybrid system where neural models handle high-level semantics while classical algorithms handle precise geometric operations? The paper's design — an MLLM acting as "smart eye" for semantic grounding, classical CV operators acting as "robotic hands" for precise edits — is an argument for the latter. This connects to broader debates about neurosymbolic architectures and the division of labor between learned and engineered components in AI systems.

Where Prior Approaches Fall Short

The paper identifies three distinct lines of prior work, each with specific limitations that motivate PixelCraft's design.

Text-Only Reasoning Loses Spatial Precision

The earliest and most straightforward approach to improving chart understanding is to enhance the reasoning component: fine-tune models on chain-of-thought (CoT) trajectories specialized for chart analysis (Masry et al., 2024b; Huang et al., 2025; Xu et al., 2024), or more recently, apply reinforcement learning to chart reasoning (Chen et al., 2025). These methods keep the visual input fixed — a single chart image — and focus entirely on generating better textual reasoning paths.

The limitation is structural, not methodological. As the paper states:

"Relying solely on textual reasoning often leads to the loss of fine-grained spatial and structural information, making it difficult to capture patterns such as subtle visual differences or geometric constraints."

The problem is that the vision encoder's representation of the chart is compressed and lossy. When the model needs to compare the height of a specific bar in subfigure (row 2, col 1) to another bar in subfigure (row 3, col 2) across a dense 4×4 grid, relying on the encoder's holistic embedding of the entire image introduces localization noise. The model must "read" values from its internal representation rather than from explicit pixel-level operations, and this introduces errors that no amount of reasoning training can correct — the evidence itself is degraded.

Visual Chain-of-Thought Is Too Rigid and Low-Fidelity

A more recent line of work attempts to externalize reasoning into intermediate visual cues — generating annotated images, cropped regions, or highlighted elements that support step-by-step visual reasoning (Meng et al., 2023; Hu et al., 2024; Fu et al., 2025; Kumar et al., 2024). This is the "visual CoT" paradigm, and it addresses the evidence-quality problem by producing explicit intermediate images.

However, the paper identifies two critical limitations in existing visual CoT methods:

Low-fidelity image processing. Methods like Refocus (Fu et al., 2025) rely on detecting specific visual primitives — lines, contours, bounding boxes — using heuristic or learned detectors. The paper explicitly notes that Refocus's "reliance on these predefined structures limits its applicability to a narrow range of chart types." More critically, the grounding accuracy is poor: the base Qwen2.5-VL-3B model achieves an overall IoU of only 0.10 on structured chart elements, and even GPT-4.1-mini achieves only 0.31 (Table 6). When a tool crops the wrong subfigure or masks the wrong data series, the downstream reasoning operates on corrupted evidence — often worse than no tool use at all. This explains why Refocus underperforms the CoT baseline on several benchmarks (Table 1): GPT-4o + Refocus scores 47.2 on CharXiv vs. 51.1 for GPT-4o + CoT; on ChartQAPro, it scores 46.30 vs. 56.52. Bad tools are worse than no tools.

Linear, rigid reasoning patterns. Even when tools produce correct intermediate images, existing visual CoT methods chain them linearly — each image derives solely from its immediate predecessor. The paper articulates this critique sharply:

"This linear approach imposes a cognitive rigidity, forcing the model into a monologue of one-way reasoning. Sophisticated visual analysis, however, is inherently a non-linear process of hypothesis testing, formulating a premise, exploring its implications visually, and, upon encountering contradictions, backtracking to revise earlier assumptions."

In practice, this means that if a visual CoT system crops subfigure A, then subfigure B, then compares them — and the comparison reveals that subfigure A was cropped incorrectly — there is no mechanism to backtrack and re-crop subfigure A. The error is locked in. This is not a hypothetical concern; the paper's planner-managed image memory is specifically designed to enable revisiting earlier visual states, which linear approaches cannot do.

Moreover, the linear approach feeds all historical images into an ever-expanding context window, which the paper notes causes performance degradation due to documented multi-image input issues in MLLMs (Tian et al., 2025; Zhang et al., 2025c). The image memory avoids this by selective recall — the planner retrieves only relevant images rather than streaming the entire history.

Multi-Agent Methods Without Visual Specialization Are Insufficient

The paper also evaluates multi-agent collaboration methods (Debate, Reconcile) that have shown strong results on text-only reasoning tasks. These methods have multiple agents independently generate solutions and then converge through discussion or voting.

On chart reasoning, they provide "little benefit" (Section 5.1). The Debate method with GPT-4.1-mini on ChartQAPro scores 57.75 — nearly identical to the direct-answer baseline (57.85) and below CoT (62.21). Reconcile performs similarly. The paper's diagnosis is straightforward:

"Multiple agents without specialized visual tools struggle to perform visual reasoning on charts."

The failure mode is that all agents share the same perceptual bottleneck — the base MLLM's imperfect vision. More discussion among agents cannot correct a systematic error in reading a chart value. The agents debate from the same flawed evidence, so they converge on agreement rather than correctness. This finding is a negative result that motivates the paper's central design principle: visual tools must provide high-fidelity evidence before multi-agent collaboration can add value.

How This Paper Positions Itself

PixelCraft is positioned at the intersection of three research directions — visual tool use, multi-agent collaboration, and structured image understanding — but with a specific synthesis that differentiates it from each.

Relative to visual CoT methods (Refocus, Visual Sketchpad, OpenThinkImg): PixelCraft shares the goal of producing intermediate visual evidence but diverges in two fundamental ways. First, it achieves high-fidelity processing through a fine-tuned grounding model plus classical CV operators, rather than relying on off-the-shelf MLLM vision or heuristic detectors. The grounding model achieves 0.93 overall IoU (Table 6) vs. 0.26 for the base model — a 3.6× improvement that makes tools reliable rather than error-prone. Second, it replaces the linear reasoning pattern with a planner-managed image memory that enables "explicit branching and backtracking" (Section 1). This is not an incremental improvement to visual CoT — it is an architectural shift from a chain to a graph of visual states.

Relative to multi-agent debate methods (Debate, Reconcile): PixelCraft uses agents not for debating solutions but for specialized role execution (planning, tool operation, reasoning, criticizing). The agents have asymmetric capabilities — the planner orchestrates, tool agents edit images, the reasoner interprets them, critics verify them — creating a workflow where each step produces auditable intermediate artifacts. This is closer to the role-playing paradigm (Wu et al., 2023; Shao et al., 2023) than to debate, but with the crucial addition that tool agents execute visual operations rather than generating text. The paper's empirical results (Table 4) show that adding tool agents provides the largest single performance gain over CoT, confirming that specialized visual capability — not multi-agent structure per se — is the key driver.

Relative to general tool-use frameworks: Unlike MM-REACT or ViperGPT, which generate code or API calls for general visual tasks, PixelCraft's tools are specifically designed for structured image processing (subfigure cropping, legend-based masking, auxiliary line drawing) and are implemented with a hybrid approach: an MLLM for semantic grounding (understanding "the subfigure at row 2, column 1") and classical CV for precise execution (pixel-level coordinate operations). The paper explicitly notes that purely LLM-generated tools were "often ineffective" (Section 3.1) and required manual refinement — acknowledging a limitation but also arguing that the hybrid approach is necessary for the fidelity required by structured images.

The paper's overarching thesis is that structured image reasoning requires both high-fidelity visual tools and flexible reasoning orchestration, and that neither alone suffices. The ablation study (Table 4) supports this: tool agents alone (+CoT) provide a 1.2 percentage point gain on CharXiv; adding the image memory and planner coordination (enabling flexible reasoning) provides an additional 3.1 points; critics add another 1.6 points. The gains are cumulative and synergistic — each component addresses a distinct failure mode (perceptual errors, rigid reasoning, error propagation) that the others cannot.

3. Technical Approach

3.1 Reader Orientation

PixelCraft is a multi-agent system where a central "planner" MLLM orchestrates a team of specialized agents — visual tool operators, a reasoner, and critics — to answer complex questions about structured images like charts and geometric diagrams. It solves the problem that MLLMs make perceptual errors (misreading chart values, confusing subfigures) that cascade into wrong answers, by externalizing visual operations into verifiable, high-fidelity tool calls while replacing rigid linear reasoning with a flexible workflow where the planner can revisit earlier visual states, explore alternative branches, and self-correct based on critic feedback.

3.2 Big-Picture Architecture (Diagram in Words)

The system has six agent roles orchestrated through a three-stage workflow:

1. Dispatcher — analyzes the question and activates only the relevant visual tool agents, avoiding unnecessary computation.
2. Planner — the central conductor that decomposes the question into sub-tasks, sequences agent calls, manages an image memory of all intermediate visual outputs, and selects which historical image to use at each step.
3. Visual Tool Agents — specialized operators (subfigure cropping, legend-based masking, region magnification, auxiliary lines, geometric line construction) that use a fine-tuned grounding model to map textual descriptions to pixel coordinates, then execute precise edits via classical CV algorithms.
4. Reasoner — interprets the processed images and performs logical analysis on the visual evidence.
5. Visual Critic — operates in-loop, verifying that tool outputs satisfy their assigned goals and that processed images contain sufficient information to answer the subquery before the reasoner acts on them.
6. Planning Critic — performs post-hoc review of the entire reasoning trace, identifying flawed tool choices or reasoning steps, and provides feedback for a second re-answering attempt.

Information flows as follows: (1) Question → Dispatcher selects relevant tools; (2) Planner decomposes question, activates agents, retrieves images from memory; (3) Tool agents process images, Visual Critic validates outputs; (4) Reasoner analyzes validated images; (5) Planner iterates steps 2-4 until answer is generated; (6) Planning Critic reviews trace, triggers re-answering if errors found.

3.3 Roadmap for the Deep Dive

• First, the grounding model — how it's trained, what data it uses, and why pixel-level accuracy is the foundation that makes all tool agents reliable. This is the prerequisite that everything else depends on.
• Second, the visual tool agents — their individual implementations, how they interface with the grounding model, and what specific image operations they perform. Understanding each tool's capabilities is essential for understanding the planner's decision space.
• Third, the image memory — the data structure and protocol that enables non-linear reasoning. This is the architectural innovation that differentiates PixelCraft from linear visual CoT.
• Fourth, the three-stage workflow — agent selection, role-driven discussion, and iterative refinement — including how agents communicate, how the visual critic gates information flow, and how the planning critic triggers re-answering.
• Fifth, the overall design rationale — why the paper chose a hybrid MLLM+CV approach over end-to-end neural methods or purely code-generated tools.

3.4 Detailed, Sentence-Based Technical Breakdown

This is a systems paper whose core idea is that structured image reasoning requires both high-fidelity visual tools (achieved through a fine-tuned grounding model + classical CV) and flexible, non-linear reasoning orchestration (achieved through a planner-managed image memory with critic-guided self-correction), and that combining these produces cumulative gains beyond either approach alone.

Grounding Model: The Foundation for High-Fidelity Tool Execution

Problem statement. Every visual tool agent in PixelCraft needs to convert a textual description (e.g., "the subfigure at row 2, column 1" or "the legend item labeled 'Joint-CNN'") into precise pixel coordinates within the image. Off-the-shelf MLLMs perform this grounding poorly on structured images: the base Qwen2.5-VL-3B achieves an overall IoU of only 0.10 across subplot regions, legend regions, text labels, and axis ticks (Table 6). Even GPT-4.1-mini achieves only 0.31. At this accuracy level, tool agents frequently operate on the wrong region, producing corrupted evidence that is worse than no tool use at all.

Why existing approaches fail. The paper identifies that existing chart tool methods like Refocus "rely on detecting specific visual primitives (e.g., lines, squares)" which "limits its applicability to a narrow range of chart types." The problem is not just accuracy but generality — a line-detection-based approach works for charts with clear ruled lines but fails on gradient-shaded heatmaps or scatter plots with irregular marker distributions. The paper's approach instead trains a general grounding capability: given any textual description of a visual element and the image, output a bounding box.

Training data construction: programmatically synthesized charts. The paper builds a synthetic data pipeline to generate 53,000 chart images with ground-truth bounding box annotations (Section 4). The pipeline operates in two stages:

Stage 1: Single-panel charts. GPT-4o generates structured JSON objects specifying textual elements (titles, legends, axis labels, data series). These JSON specifications are fed into Matplotlib code templates that are "programmatically augmented and rewritten by GPT-4o to enhance visual and structural diversity." Critically, the rendering process is instrumented: as Matplotlib draws each element, the pipeline records its precise pixel coordinates. This yields <image, textual label, bounding box> triples at scale without human annotation.

Stage 2: Multi-panel compositions. Single-panel charts are randomly sampled, duplicated, and arranged into composite grid layouts containing between 2 and 16 subfigures with "randomized spatial margins." This creates the complex multi-panel images common in the CharXiv benchmark (arXiv figures often contain dense subfigure grids). The final dataset contains 43,000 annotations from single-panel charts and 10,000 from multi-panel compositions.

Training data construction: geometric diagrams. To add geometric reasoning capability, the paper augments the dataset with 2,000 samples from the Inter-GPS benchmark (Lu et al., 2021). These diagrams contain annotated geometric elements — points, lines, vertices — with accompanying textual labels. The paper "extracts the position coordinates of geometric points along with their corresponding textual labels" from the source annotations, creating point-level grounding data for geometric tools.

Training procedure. The task is formulated as autoregressive sequence prediction:

$P(Y|I, P) = \prod_{t=1}^{T} P(y_t | y_{<t}, I, P)$

where $I$ is the input image, $P$ is a textual prompt (e.g., "Please locate the pixel coordinates of the subfigure at row 2, column 2"), and $Y = (y_1, ..., y_T)$ is the output sequence jointly encoding the textual answer and bounding box coordinates. The bounding boxes are represented as absolute pixel coordinates in the format <|box start|>[x1, y1, x2, y2]<|box end|>, which aligns with Qwen2.5-VL's native grounding format.

What it computes: the model processes the image through its vision encoder, attends to the textual prompt through cross-attention, and generates tokens autoregressively. When the model encounters a grounding request, it predicts four coordinate values representing the top-left and bottom-right corners of the bounding box. The training objective is standard next-token prediction with cross-entropy loss on all tokens, including the coordinate tokens.

Why this form: predicting coordinates as text tokens rather than regressing them through a separate detection head leverages the full multimodal understanding of the pretrained MLLM. The model must understand "row 2, column 2" semantically — it needs to recognize the grid layout, count rows and columns, and identify subfigure boundaries — before outputting coordinates. A dedicated detection head would treat this as a pure pattern-matching task, losing the semantic reasoning. The text-token formulation also allows the model to output "Not found" when elements are absent — a graceful failure mode that coordinate regression cannot express.

Training hyperparameters. The paper reports (Appendix B.1): full-parameter supervised fine-tuning on Qwen2.5-VL-3B-Instruct, a single epoch on four NVIDIA A100 GPUs, learning rate $1 \times 10^{-5}$. The model has 3 billion parameters, making it compact enough to run as a sub-component within the larger agent system without dominating the compute budget.

Results. The fine-tuned model achieves dramatic improvements (Table 6): overall IoU rises from 0.10 (base 3B model) to 0.93. On subplot regions specifically, accuracy reaches 0.99 IoU — near-perfect subfigure localization. On axis ticks — the most fine-grained elements — PCK@0.01 (the fraction of predicted keypoints within 1% of the image's larger dimension from ground truth) reaches 0.93. The model also substantially outperforms much larger models: Qwen2.5-VL-7B scores 0.26 overall, GPT-4.1-mini scores 0.31, while the fine-tuned 3B model scores 0.93. This demonstrates that targeted fine-tuning on in-domain structured image data is more effective than scaling model size — the capability is data-driven rather than scale-driven.

Visual Tool Agents: Precise Pixel-Level Image Operations

The tool agents bridge the gap between semantic understanding (knowing which region to operate on) and precise execution (performing the operation at the correct pixel coordinates). Each tool follows a unified pattern: (1) receive a textual specification from the planner, (2) call the grounding model to obtain pixel coordinates, (3) execute a classical CV operation using those coordinates, (4) return the processed image to the planner for storage in image memory.

Tool generation and curation process. The paper initially attempted to have GPT-4.1-mini automatically generate tool code by sampling 500 questions from ArxivQA and prompting the model to produce Python functions for visual operations. This yielded 468 tool candidates. The tools were embedded using Qwen3-Embedding-8B (Zhang et al., 2025b) and clustered to identify functional categories.

The clustering revealed four dominant categories (Table 5, Fig. 6a): Subfigure Cropping (148 tools), Adding Auxiliary Lines (128 tools), Masking Data with Legend (101 tools), and Region Magnification (64 tools). Only 27 tools fell outside these categories — most of which attempted more complex operations like "find peaks in data" or "draw spectral type guides."

Critically, the paper reports that purely LLM-generated tools "were often ineffective, primarily due to the lack of precise grounding coordinates, or were simply invalid, suffering from incorrect code execution and producing erroneous visual outputs" (Section 3.1). The second experiment — constraining the model from generating known grounding-based tools — produced more complex tools (e.g., detecting all data points in a scatter plot to compute their mean, locating the intersection of two plotted lines), but these "consistently exhibited low accuracy, even after being subjected to the same GPT-o3 rewriting process" (Appendix A, Fig. 6b).

The final toolkit was curated by: selecting the most promising tools from recurring clusters, having GPT-o3 automatically rewrite them for robustness, then manually tuning them for "correctness and practical usability." The paper explicitly acknowledges this manual refinement as a limitation (Appendix D) — future work on fully automated tool generation is needed.

Chart tool: Subfigure Cropping. This is the most frequently used tool (351 calls on CharXiv, Fig. 4b), reflecting the prevalence of multi-panel chart figures in the benchmarks.

Inputs: an image containing multiple subfigures, and a textual description (e.g., "the subfigure at row 2, column 2").

Procedure: (1) The grounding model localizes the bounding box of the target subfigure based on the textual description. (2) If a legend is spatially separate from the subfigure (common in scientific plots where legends are placed to the side or below), the grounding model also localizes the legend region. (3) Both regions are cropped and programmatically composed into a single coherent output image — the subfigure and its legend placed together.

Why this matters: many chart questions ask about specific subfigures within dense multi-panel layouts. Without cropping, the MLLM must attend to the correct subfigure within a complex grid, which introduces localization noise and cross-subfigure interference. Cropping isolates the relevant visual evidence, eliminating this noise source.

Chart tool: Masking Data with Legend. The second most-used tool (38 calls on CharXiv), this addresses questions that require focusing on a specific data series while ignoring others.

Inputs: an image, and a legend item description (e.g., "the line representing Group A").

Procedure: (1) The grounding model localizes the specified legend item within the chart. (2) The tool extracts the dominant color from the localized legend icon (the colored marker or line sample next to the text label). (3) This color is used to create a binary mask: pixels matching this color in the plot area are retained; all other data series pixels are suppressed. The output is an image showing only the queried data series against the chart background.

Design choice: color-based masking rather than semantic segmentation. The paper uses a classical CV approach — dominant color extraction and binary masking — rather than training a segmentation model for data series. This is simpler, more interpretable, and avoids the distribution shift issues that would arise from training a segmentation model on synthetic charts and deploying on real charts. The tradeoff is that it assumes each data series has a distinct color (true for most scientific charts but not all), and that the legend icon accurately represents the plotted color (also generally true). For charts violating these assumptions, the tool would fail — but the visual critic provides a safety net by detecting such failures (see Visual Critic section below).

Chart tool: Region Magnification. This tool zooms into a specified region of a chart to reveal local details.

Inputs: an image, and start/end tick values on the x and y axes defining the region of interest (e.g., "magnify the region between x=10 and x=50, y=0 to y=0.5").

Procedure: (1) The grounding model identifies the pixel coordinates corresponding to the specified axis tick marks. (2) The tool crops the image to the bounding box defined by these four corner coordinates. (3) The cropped region is resized (upsampled) to a standard resolution, magnifying local details such as individual data point positions, small oscillations, or closely-spaced markers.

Why tick-based rather than coordinate-based: the planner specifies regions in data-space terms (tick values), not pixel-space terms, because the planner reasons about data semantics (e.g., "I need to see the behavior between x=10 and x=50"). The grounding model translates this semantic specification into pixel coordinates by finding the visual tick marks on the axes. This is more robust than asking the MLLM to directly estimate pixel coordinates for data ranges — a task that requires implicit understanding of the mapping between data coordinates and pixel space, which MLLMs perform unreliably.

Chart tool: Adding Auxiliary Lines. This tool draws horizontal or vertical reference lines at specified axis values, supporting threshold comparison and value reading tasks.

Inputs: an image, an axis specification (x or y), and a tick value (e.g., "draw a horizontal line at y=30").

Procedure: (1) The grounding model identifies the pixel coordinate of the specified tick mark on the indicated axis. (2) The tool renders a straight horizontal or vertical line at this pixel position across the full plot area. The line serves as a visual reference, making it easy for the reasoner to compare data points against this threshold.

Why this helps: when questions ask "at what x does the curve cross y=0.5?", drawing the y=0.5 line makes the intersection point visually explicit — the reasoner no longer needs to mentally interpolate between grid lines. This reduces reading errors on dense charts with fine grid spacing.

Geometric tools: Point Connection, Perpendicular Line Construction, Parallel Line Construction. These three tools address geometric diagram reasoning and share a common implementation pipeline.

Common pipeline: (1) The grounding model detects the key points specified in the query — line endpoints, vertices, intersection points — and returns their pixel coordinates. (2) The tool programmatically renders the desired geometric construction based on these coordinates, using standard computational geometry: drawing a straight line segment between two points, constructing a perpendicular by computing the normal vector and offsetting, constructing a parallel by copying the direction vector and offsetting.

Why classical geometry rather than learned: the geometric relationships (perpendicularity, parallelism, point connection) have deterministic mathematical definitions. Implementing them through vector operations is exact and verifiable; implementing them through a learned model would add unnecessary uncertainty. The hybrid approach — MLLM for interpreting which points to connect, CV for executing the connection — allocates uncertainty to the component that handles ambiguity (semantic interpretation) and determinism to the component that handles precision (geometric construction).

Tool usage patterns. The paper provides frequency statistics (Fig. 4): on CharXiv, Subfigure Cropping dominates (351 calls), reflecting the multi-panel nature of arXiv charts. Legend Masking (38 calls), Region Magnification (35 calls), and Adding Auxiliary Lines (3 calls) are less frequent but show high accuracy gains when used — Legend Masking improves accuracy by 18.4 percentage points over the base CoT method (Fig. 4a). On Geometry3K, Point Connection dominates (56 calls) with Perpendicular Line (7 calls) and Parallel Line (6 calls) used less frequently. The imbalance is described as "query-driven and image-driven" — the tool selection naturally follows from the types of questions in each benchmark rather than from arbitrary system preferences.

Image Memory: Enabling Non-Linear Visual Reasoning

The image memory is the architectural mechanism that differentiates PixelCraft from linear visual CoT approaches. Rather than feeding all historical images into an ever-expanding context window (the linear approach), the planner maintains an indexed store of all intermediate visual outputs and selectively retrieves images for each reasoning step.

Data structure. The image memory is a collection of <image, description> pairs. Every time a tool agent produces a processed image (e.g., a cropped subfigure, a masked chart), the planner stores it along with its associated goal description (e.g., "subfigure at row 2, column 1 with its legend"). The reasoner's textual outputs are similarly stored, allowing the planner to associate analytical conclusions with specific visual states.

Selective recall protocol. At each reasoning step, the planner consults the image memory and selects the most relevant image(s) for the current sub-task. The paper does not specify the exact retrieval mechanism — whether it's keyword matching against descriptions, index-based lookup, or implicit selection through the planner's reasoning — but the key property is that only the selected images are passed to the active agent, not the entire history. This is described as:

"Instead of indiscriminately feeding all historical images into an ever-expanding context, the planner stores intermediate clues and selectively recalls them for subsequent steps" (Section 1).

Enables branching. Because all intermediate states are preserved in memory rather than overwritten by a linear chain, the planner can revisit an earlier visual state and explore a different path. The example implied by the paper (though not fully instantiated in a concrete figure): if the planner crops subfigure A, then subfigure B, then the reasoner compares them and finds an inconsistency, the planner can return to the memory entry for subfigure A, re-crop it with different parameters, and re-run the comparison — without losing the subfigure B analysis.

Enables backtracking. When the planning critic identifies an error (e.g., "the wrong subfigure was cropped" or "the legend masking used the wrong color"), the planner can retrieve the original image from memory (before the erroneous operation) and restart from that point, rather than restarting the entire chain from scratch.

Reduces long-context overhead. The paper explicitly cites recent work showing "MLLM performance degrades on multi-image inputs" (Tian et al., 2025; Zhang et al., 2025c) as a motivation for selective recall. The image memory avoids the linear CoT problem where the context window fills with a growing sequence of images, degrading the model's attention quality. By retrieving only relevant images, the context remains compact regardless of how many total visual operations were performed.

Contrast with linear visual CoT. In a typical visual CoT approach (e.g., Refocus, Visual Sketchpad), each step produces an image that becomes input to the next step: image₁ → operation → image₂ → operation → image₃ → ... → answer. The model at step N receives all previous images (1 through N-1) in its context. If step 3 was erroneous, steps 4 through N are all corrupted, and there is no mechanism to fix step 3 without restarting from scratch. In PixelCraft, the planner at step N can choose to use image₂, or image₁, or the original input image — it is not forced to build on the immediate predecessor.

The Three-Stage Workflow

The system's execution unfolds across three sequentially invoked stages, with the planner acting as the central coordinator throughout.

Stage I: Query-Aware Agent Selection (Dispatcher). The workflow begins with the dispatcher analyzing the incoming question. Its responsibility is to determine which tool agents are potentially relevant and activate only that subset, "thereby optimizing both computational efficiency and the relevance of the subsequent process" (Section 3.2). The dispatcher does not plan the execution order — that is the planner's job — but it narrows the tool set, preventing the planner from considering irrelevant tools that could introduce confusion or unnecessary computation.

The paper does not detail the dispatcher's implementation beyond this conceptual description, but the prompt template (Appendix B.3) shows that it receives the question, the chart image, and the full tool descriptions, and outputs a list of selected tool names (or an empty list if no tools are relevant). The selection is based on matching question requirements (counting, comparison, extraction, calculation) to tool capabilities, considering tool constraints and the chart structure.

Stage II: Role-Driven Agent Discussion (Planner + Tool Agents + Reasoner + Visual Critic). This is the core execution stage where the actual visual reasoning occurs.

Step 1: Planner decomposes and sequences. The planner receives the original question, the original image, and the dispatcher's selected tool list. It decomposes the question into sub-tasks and begins a step-by-step execution loop. At each step N, the planner outputs a THOUGHT (analysis of current progress), an ACTION (which agent to call with what parameters), and expects an OBSERVATION (the agent's response).

Step 2: Planner selects image and activates agent. Before each agent call, the planner selects the appropriate image from the image memory. The selection is driven by the sub-task: for cropping a subfigure, the planner would select the original full image; for comparing two cropped subfigures, the planner would select both cropped images; for re-analyzing a masked version, the planner would select the masked image. The planner then specifies a concrete goal for the selected agent — for tool agents, a precise instruction like "crop the subfigure at row 1, column 1"; for the reasoner, a subquery like "what is the HOVz value at Rho=0 in this subfigure?"

Step 3: Agent execution and visual critic gating. The activated agent produces its output:

• Tool agents return a processed image and a status message.
• Reasoner returns textual analysis (observed values, comparisons, logical deductions).

After a tool agent produces an image, the visual critic performs two checks before the image proceeds to the reasoner:

Goal satisfaction check: The visual critic evaluates "whether the visual output successfully fulfills the goal assigned by the planner" (Section 3.2). For example, if the planner asked for "subfigure at row 2, column 1" and the tool accidentally cropped row 1, column 2, the visual critic should detect the mismatch.

Answerability check: When the planner sends a processed image and a subquery to the reasoner, the visual critic first assesses "the image's answerability" — whether the image contains sufficient information to answer the subquery. For instance, if the subquery asks about a specific data point's value but the cropped image doesn't include the axis tick labels, the visual critic returns false, preventing the reasoner from hallucinating a value from incomplete evidence.

When either check fails, "the error alert will be returned to the planner for subsequent reasoning, ensuring the reasoning's robustness" (Section 3.2). The planner can then re-attempt the tool operation with adjusted parameters, or try an alternative tool.

Step 4: Planner iterates. Steps 2-3 repeat until the planner determines that sufficient evidence has been gathered to answer the original question. The planner then outputs a FINAL ANSWER with a TERMINATE action.

The paper's prompt template (Appendix B.3) enforces a strict format for each iteration:

THOUGHT N: [Analysis]
ACTION N: tool_name(key=value)
OBSERVATION N: [Result]

The response history (all previous THOUGHT-ACTION-OBSERVATION triples) is included in the planner's context at each step, providing a traceable reasoning audit trail.

Stage III: Iterative Refinement and Self-Correction (Planning Critic). After the planner produces a final answer, the planning critic performs a holistic review of the entire reasoning trace.

What the planning critic evaluates: The prompt template (Appendix B.3) specifies the critic's analysis dimensions: (1) correctness of the final answer based on the image evidence; (2) correctness of intermediate extracted information (were tool outputs appropriate?); (3) whether any tools failed or were incorrectly applied; (4) whether the tool list should be adjusted (tools added or removed).

Output format: The planning critic outputs two key elements:

• ADJUSTMENT: True or False — whether re-answering is needed.
• If True, a set of suggestions including: which reasoning steps were flawed, which tools to remove or add, and how the subqueries or tool usage could be improved.

When ADJUSTMENT: True, the system re-executes: the updated tool list and critic suggestions are provided as additional input, and the planner re-answers the query with this guidance. The paper reports (Fig. 5) that most identified errors are true positives (39 out of 43 identified queries were truly erroneous in the first round), and the identification rate drops to near zero after 2 rounds, suggesting that two rounds of re-answering are sufficient for most queries. The refinement process improves overall accuracy: from 67.2% (base, before refinement) to 68.1% after refinement (Fig. 5, right panel).

Design Rationale: Why This Multi-Agent Architecture?

The paper's architectural choices reflect a set of explicit design principles that can be inferred from the ablation study (Table 4) and the comparison with baselines (Table 3):

Principle 1: Externalize visual operations into verifiable tools. The largest single performance gain comes from adding Tool Agents to the base CoT workflow (Table 4: CharXiv improves from 63.8 to 65.0, ChartQAPro from 62.21 to 63.66). This validates the paper's central claim that perceptual errors — not reasoning errors — are the primary bottleneck. By externalizing visual operations into tools that produce explicit, auditable intermediate images, the system makes visual evidence subject to critic verification rather than buried in the MLLM's internal representations.

Principle 2: MLLM for semantics, CV for precision. The hybrid design — grounding model provides coordinates, CV operators execute edits — is justified by negative results. Purely LLM-generated tools were "often ineffective" with "incorrect code execution" (Section 3.1). Purely CV approaches (like Refocus's primitive detection) are limited to specific chart types. The division of labor — semantic understanding in the MLLM, precise pixel manipulation in CV — leverages the strengths of each paradigm while containing their weaknesses.

Principle 3: Role specialization over debate. The Debate and Reconcile baselines (Table 1) show minimal or no improvement over CoT on chart reasoning, confirming that "multiple agents without specialized visual tools struggle to perform visual reasoning on charts." PixelCraft's agents have asymmetric, complementary capabilities rather than symmetric, competing ones. The planner doesn't debate the reasoner — it assigns it tasks. The critics don't propose alternative answers — they verify specific properties of the process.

Principle 4: Non-linear reasoning via image memory. The comparison with simplified Visual CoT (Table 3) shows that the image memory and flexible agent workflow provide gains beyond what the visual tools alone achieve: 68.1 vs. 65.0 on CharXiv, 65.56 vs. 61.04 on ChartQAPro. The paper attributes this to the ability to "selectively recall prior visual states" and "explore alternative reasoning branches" — capabilities that linear visual CoT fundamentally cannot support.

Principle 5: Two-layer error correction. The visual critic (in-loop) and planning critic (post-hoc) form complementary verification layers. The visual critic catches immediate tool failures before they corrupt downstream reasoning — a form of online error prevention. The planning critic catches higher-level reasoning errors — a form of offline error correction. The ablation (Table 4) shows both contribute positively: visual critic adds 1.1 percentage points (64.89 vs. 63.66 on ChartQAPro), planning critic adds another 0.6-0.7 points (65.56 vs. 64.89 on ChartQAPro). Their effects are additive, confirming they address distinct failure modes.

Principle 6: Deliberately simple reasoning prompts. The paper explicitly states that the reasoner uses "a brief generic instruction" rather than "various sophisticated prompting strategies" specifically "to demonstrate the inherent power of our high-fidelity tool agents and the overall framework, rather than attributing performance gains to complex prompt engineering" (Section 3.1, Reasoner description). This methodological choice strengthens the paper's claims: the gains come from the architecture, not from prompt tuning.

4. Key Insights and Innovations

Innovation 1: High-Fidelity Is in the Grounding, Not in the Model Size

The paper's most important conceptual move is diagnostic rather than architectural: it identifies that the bottleneck in structured image reasoning is not reasoning quality or tool design, but pixel-level grounding accuracy — and that this bottleneck is invisible if you only look at end-task accuracy rather than intermediate localization metrics.

Prior work in visual tool use for chart understanding treated grounding as an implicit capability of whatever backbone MLLM was being used. Refocus (Fu et al., 2025) called the same GPT-4o API that answers questions to also localize chart elements. Visual Sketchpad (Hu et al., 2024) relied on the base model's vision for sketching operations. The implicit assumption was: if a model can describe a chart correctly, it can also localize elements within it correctly. The paper demolishes this assumption with Table 6: GPT-4.1-mini achieves only 0.31 overall IoU on structured chart elements despite being a strong reasoning model (58.6% direct-answer accuracy on CharXiv). The capabilities are not coupled — a model can understand what a chart shows while being unable to point precisely to where its elements are.

This is a fundamental reframing of the problem. Rather than asking "how do we build better visual reasoning systems?", the paper asks "what component is actually failing, and can we fix just that component without changing everything else?" The answer — the grounding model — is 3 billion parameters, fine-tuned on 53k synthetic examples, and achieves 0.93 IoU. It outperforms models 2-3× its size (Qwen2.5-VL-7B at 0.26, GPT-4.1-mini at 0.31) by a factor of 3× or more. The implication is not "small models are better" but rather "targeted training on in-domain structured data dominates general-purpose scale for this specific capability." This is a miniature version of the Chinchilla scaling insight applied to visual grounding: data quality and domain alignment matter more than parameter count.

The downstream impact is validated in Figure 3a: plugging the same PixelCraft agent framework with different grounding backends produces substantially different accuracy (GPT-4.1-mini grounding yields ~64%, fine-tuned Qwen yields ~68% on CharXiv). The ~4 percentage point gap is entirely due to grounding accuracy — same tools, same planner, same reasoner, same critics. This isolates grounding quality as a first-class variable in visual reasoning system design, not an implementation detail to be absorbed into the backbone model's general capabilities.

The paper thus establishes a new diagnostic principle: evaluate intermediate localization accuracy when building visual tool systems, not just end-task performance. Bad grounding produces tools that operate on wrong regions, feeding corrupted evidence into reasoning — a failure mode that task-level metrics obscure because the error manifests as a "reasoning error" (wrong answer) when it is actually a "perception error" (wrong pixels). This diagnostic move is significant beyond chart understanding: any system where visual tools operate on specified regions should separately validate localization accuracy rather than assuming it tracks end-task capability.

Innovation 2: Image Memory as an Architectural Primitive for Non-Linear Visual Reasoning

The second conceptual contribution is the introduction of an image memory — a planner-managed store of intermediate visual states with selective recall — as an architectural mechanism that enables branching, backtracking, and non-linear reasoning trajectories in visual reasoning systems.

Prior work in visual chain-of-thought (Meng et al., 2023; Hu et al., 2024; Fu et al., 2025; Su et al., 2025) adopted what the paper calls a "linear, chain-like paradigm" where each intermediate image derives solely from its predecessor and all previous images are fed into an ever-expanding context window. This is technically simple — you append images to the conversation history and let the model's attention mechanism sort out relevance — but it imposes two constraints that the paper argues are fatal for complex structured image tasks.

First, cognitive rigidity: the model is forced into "a monologue of one-way reasoning" (Section 1). Once an image is produced and consumed, there is no mechanism to return to it, revise it, or explore an alternative path from it. If a subfigure was cropped incorrectly in step 2, steps 3 through N all build on corrupted evidence, and the only recovery mechanism is to restart the entire chain. The paper frames this as fundamentally incompatible with how sophisticated visual analysis actually works: "formulating a premise, exploring its implications visually, and, upon encountering contradictions, backtracking to revise earlier assumptions."

Second, long-context degradation: feeding all historical images into the context degrades model performance, a problem documented in recent work on multi-image MLLM inputs (Tian et al., 2025; Zhang et al., 2025c). The paper doesn't just cite this problem — it provides a controlled comparison in Table 3: a simplified Visual CoT baseline using the exact same tools and prompts but with linear history concatenation scores 65.0 on CharXiv and 61.04 on ChartQAPro, while the full PixelCraft framework with image memory scores 68.1 and 65.56 respectively. The ~3-4 percentage point gap is attributable to the image memory and the flexible agent orchestration it enables — same visual evidence, different organization of access to it.

What makes this an innovation rather than an engineering optimization is that it elevates image access patterns to a first-class design dimension in visual reasoning architectures. The image memory is not a cache or a performance optimization — it is a reasoning primitive. The planner's ability to "adaptively recall any historical image" (Section 3.1) changes what reasoning strategies are possible. It enables the planner to:

• Compare non-adjacent states: retrieve subfigure A from step 2 and subfigure B from step 7 and compare them, even though they weren't produced sequentially.
• Branch from an earlier state: if step 5's analysis suggests an alternative interpretation, the planner can return to the visual state from step 3 and pursue a different tool sequence from there, without losing the work done in steps 4-5.
• Verify retroactively: when the planning critic identifies a problem, the planner can retrieve the exact image that caused the problem and re-process it, rather than re-processing everything.

This is the visual reasoning analog of a concept from symbolic AI planning: maintaining a search tree of states with the ability to backtrack, rather than committing to a linear trajectory. But unlike symbolic planners, PixelCraft's "states" are pixel-level images generated by neural tool agents, and the "search" is guided by an MLLM planner making semantic decisions about which visual state to revisit. The innovation is the hybridization of a symbolic reasoning pattern (state-space search with backtracking) with neural visual processing — a fundamental architectural contribution that transfers a concept from one paradigm (planning) to another (visual reasoning) by inventing the appropriate primitive (image memory with selective recall).

Innovation 3: Critic-Gated Information Flow as Error Prevention and Error Correction

The paper introduces a two-layer critic architecture — the visual critic (in-loop, pre-reasoning) and the planning critic (post-hoc, pre-re-answering) — that together form a comprehensive error management system. What makes this distinctive is not the idea of having critics (self-critique is well-established in LLM research), but the specific positioning and function of each critic within the information flow.

The standard pattern in self-critique for reasoning (Madaan et al., 2023; Shinn et al., 2023; Chen et al., 2024) is to generate a complete output, then critique it, then regenerate. This is post-hoc correction: the error propagates fully before detection. The paper's visual critic operates differently: it sits between tool execution and reasoner consumption, gating whether a processed image is allowed to reach the reasoner. The paper specifies two distinct gating criteria:

Goal satisfaction: "whether the visual output successfully fulfills the goal assigned by the planner." This is an operational check — did the tool do what it was asked to do? If the planner requested "crop subfigure at row 2, column 1" and the tool produced an image of row 1, column 2, the visual critic should catch this before the reasoner extracts data from the wrong subfigure.

Answerability: "whether the image contains sufficient information to answer the subquery." This is a sufficiency check — even if the tool operation was correct, is the resulting image adequate for the reasoner's task? If the subquery asks for a specific data point's value but the cropped image lacks axis tick labels, the reasoner would need to hallucinate or guess, so the visual critic blocks the flow.

This gating function is novel because it prevents error propagation rather than correcting errors after propagation. In a linear visual CoT system, if a cropped subfigure is wrong, the reasoner will still produce an analysis — a wrong analysis, built on wrong evidence — and that analysis will feed into subsequent steps. The error multiplies. The visual critic interrupts this cascade at its source: bad evidence is detected, an alert is sent to the planner, and the planner re-attempts the tool operation or tries an alternative. The paper's ablation (Table 4) quantifies the value: adding the visual critic improves ChartQAPro accuracy from 64.43 to 64.89 (after controlling for tool agents and dispatcher), a modest but positive gain that the paper frames as preventing specific failure cases rather than boosting average performance.

The planning critic, in contrast, operates in the conventional post-hoc mode — reviewing the entire trace after an answer is generated — but with a specific diagnostic focus: tool usage correctness rather than answer correctness. The prompt template (Appendix B.3) asks the critic to evaluate whether "some tools failed or were incorrectly applied during the reasoning process" and to output an updated tool list with additions or removals. This is more specific than general answer verification — the critic is not just saying "this answer looks wrong" but identifying which tool choices or subquery decompositions led to the error. This specificity makes the critic's feedback actionable for the planner's re-answering attempt.

The paper's empirical characterization of critic effectiveness (Figure 5) is unusually detailed for a system paper: it reports both true positive and false positive identification rates across refinement rounds, showing that most flagged errors are genuine (39 TP vs. 3 FP in round 1) and that re-answering improves accuracy (from 67.2% to 68.1% on CharXiv). This transparency about critic reliability — acknowledging 3 false positives — is itself a methodological contribution, demonstrating what level of critic accuracy is achievable with the described architecture.

Together, the two critics form what the paper might call a defense-in-depth strategy for visual reasoning: the visual critic prevents errors from entering the reasoning chain (online prevention), while the planning critic catches errors that slip through and enables correction (offline remediation). This dual-layer approach to error management, with the in-loop critic specifically architected as a gating function rather than a post-hoc evaluator, is a conceptual contribution that generalizes beyond the specific tools and tasks in the paper.

Innovation 4: A Negative Result That Redefines the Tool Generation Problem

The paper's appendix (Appendix A) contains a finding that, while understated, has significant implications for the field: purely LLM-generated visual tools are unreliable for high-fidelity image processing, and automated tool generation for visual domains is fundamentally harder than for text or code domains.

The experiment is clean: prompt GPT-4.1-mini with 500 chart questions from ArxivQA, ask it to generate Python tool code, cluster the 468 resulting tools by embedding similarity. The first round, where the model is unconstrained, produces tools that cluster into four recurring categories — all of which require precise grounding coordinates. These tools, the paper reports, "were often ineffective, primarily due to the lack of precise grounding coordinates, or were simply invalid, suffering from incorrect code execution and producing erroneous visual outputs."

The second round, where the model is explicitly constrained from generating grounding-based tools, pushes it toward more complex operations: "detecting all data points in a scatter plot to compute their mean" or "locating the intersection point of two plotted lines." These tools "consistently exhibited low accuracy, even after being subjected to the same GPT-o3 rewriting process" (Appendix A, Figure 6b).

The paper's response to this finding — manually curating and refining the generated tools, then acknowledging this as a limitation (Appendix D) — is less interesting than the finding itself. What makes this a conceptual contribution is what it says about the nature of visual tool generation as a problem class.

There is a growing literature on automated tool generation for LLMs (Zhang et al., 2025a; Cai et al., 2024; Qin et al., 2024), where LLMs generate Python functions or API calls that extend their capabilities. This paradigm works well for text-processing tools, mathematical operations, and code execution — domains where correctness is deterministically verifiable. Visual tools are different: correctness depends on pixel-level precision, visual fidelity, and alignment between generated code and the actual pixel content of images. An LLM generating a cropping function cannot see whether its crop coordinates are correct — it can only output code, and the correctness is only knowable by executing the code against the specific image.

This is a more general insight: tool generation is fundamentally limited when the tools operate on modalities that the generator cannot directly perceive. An LLM can generate a Python function to extract data from a CSV file because it can reason about the textual structure. An LLM cannot generate a reliable Python function to crop a specific subfigure from a chart because it cannot see the chart's layout. The paper's negative result — that even with GPT-o3 rewriting, complex visual tools remain unreliable — suggests that automated tool generation for visual domains requires either (1) a generator that can visually perceive its inputs, or (2) a different paradigm where tools are generated as compositions of verified, pre-built primitives rather than de novo code.

This connects to the paper's broader defense of the hybrid MLLM+CV approach. The unreliability of generated tools is not a temporary limitation of current LLMs — it reflects a structural mismatch between the modality of generation (text/code) and the modality of operation (pixel-level visual processing). The paper's solution — manual curation of a small set of high-fidelity tools backed by a separately trained grounding model — is arguably the correct approach given current technology, not a shortcut awaiting automation. The negative result thus serves as both a justification for the paper's specific design choices and a caution for future work on end-to-end automated visual tool systems.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper evaluates on three chart reasoning benchmarks — CharXiv (Wang et al., 2024), ChartQAPro (Masry et al., 2025), and EvoChart (Huang et al., 2025) — plus one geometric reasoning benchmark, Geometry3K (Lu et al., 2021). For CharXiv, the authors use the "reasoning-focused question set" only (not the descriptive subset). For ChartQAPro and EvoChart, the full test sets are used. For Geometry3K, the paper filters 128 complex test samples that "require the intermediate visual clues to answer" from the full test set, specifically targeting problems requiring auxiliary line construction. The exact sizes of the CharXiv, ChartQAPro, and EvoChart test splits are not explicitly stated in the paper — this is a notable omission for reproducibility. The Geometry3K auxiliary-line subset is 128 questions.

• Base model(s). The paper integrates PixelCraft with three backbone MLLMs: GPT-4o-20240806, GPT-4.1-mini-20250414, and Claude-3.7-sonnet. These serve as the "brain" for the dispatcher, planner, reasoner, and critics. The grounding model — a separate component — is Qwen2.5-VL-3B-Instruct (Bai et al., 2025), fine-tuned on the authors' synthetic corpus (Section 4). The choice of three diverse, commercially prominent backbones (one OpenAI, one Anthropic, spanning both large and small variants) is a deliberate robustness check: if PixelCraft's gains are consistent across backbone architectures and scales, the system's value is not tied to any specific model's quirks. The paper explicitly states that the reasoner uses "a brief generic instruction" rather than sophisticated prompting "to demonstrate the inherent power of our high-fidelity tool agents and the overall framework, rather than attributing performance gains to complex prompt engineering" (Section 3.1).

• Metrics. The primary metric is accuracy — the fraction of questions for which the final answer matches the ground truth. For chart benchmarks (CharXiv, ChartQAPro, EvoChart), evaluation follows the CharXiv protocol: an LLM-as-a-judge approach using GPT-4.1-mini-20250414 assesses correctness. The paper explicitly notes a methodological choice: "to ensure a fair comparison across all methods, particularly given the long outputs generated by agentic and CoT approaches, we omit the 5% margin typically used for ChartQAPro and apply this unified evaluation standard consistently" (Section 5.1). This means that for ChartQAPro, answers must exactly match ground truth rather than falling within a 5% tolerance band — a stricter standard that affects all methods equally but may depress absolute scores relative to papers using the tolerance. For Geometry3K, standard exact-match grading applies. Grounding quality is evaluated separately using IoU (Intersection over Union) for bounding box predictions on subplot regions, legend regions, and text labels, and PCK@0.01 (Percentage of Correct Keypoints within 1% of the image's larger dimension) for axis tick localization (Table 6).

• Baselines. The paper compares against five baselines spanning different reasoning paradigms:

  1. Direct answer — the backbone MLLM is prompted to answer the question directly without intermediate reasoning.
  2. Chain of Thought (CoT) (Wei et al., 2022) — the backbone MLLM generates step-by-step textual reasoning before producing the final answer.
  3. Debate (Du et al., 2023) — a multi-agent method where two reasoners independently generate solutions and converge on a final answer through discussion. The paper sets up Debate with two reasoners "corresponding to the two agents (planner and reasoner) in our work" (Section 5.1).
  4. Reconcile (Chen et al., 2024) — a round-table conference approach where diverse LLMs discuss to reach consensus. Configured similarly with two reasoners.
  5. Refocus (Fu et al., 2025) — the representative chart-tool approach that couples an LLM with chart-related visual tools (contour detection, line detection, etc.). This is the only baseline that uses explicit visual tools, making it the most direct comparator for PixelCraft's tool-based approach.

  For geometric reasoning (Table 2), Refocus and Visual Sketchpad are omitted because their tools "do not support geometric reasoning tasks or require the source code to perform image processing, which is commonly unavailable" (Section 5.2). A simplified Visual CoT baseline is introduced in Table 3 for architectural ablation — it uses the same tools and prompts as PixelCraft but employs "a single planner to handle both task planning and visual reasoning monolithically" with all historical images fed linearly into context.

• Generation budget / compute accounting. The paper does not explicitly measure compute in FLOPs or tokens. Instead, cost is reported as average response time in seconds (Table 7), measured on 100 sampled questions from CharXiv and 100 from ChartQAPro using GPT-4.1-mini as the backbone. This is a latency-based measurement rather than a FLOPs-based measurement. CoT averages 3.75s on CharXiv and 3.44s on ChartQAPro; PixelCraft averages 16.45s and 11.22s respectively — roughly a 3-5× increase in wall-clock time. This latency increase is an important efficiency tradeoff that the paper acknowledges but does not deeply analyze. The paper does not account for the one-time cost of fine-tuning the grounding model (a single epoch on four A100 GPUs, Appendix B.1) in any runtime comparison, nor does it amortize the cost of generating the 53k synthetic training samples across benchmark evaluations.

• Cross-validation / statistical protocol. The paper does not employ cross-validation or report confidence intervals. The 500 ArxivQA questions used for tool generation (Appendix A) are distinct from the test benchmarks. The grounding model is evaluated on "a test set of 500 samples" (Appendix C.3) annotated using the same synthetic pipeline as training data, ensuring distributional consistency but not representing out-of-distribution generalization to real chart styles. The planning critic's error identification accuracy is reported with raw true positive and false positive counts (Fig. 5: 39 TP, 3 FP in round 1), without confidence intervals. The paper does not report multiple runs with different random seeds, standard deviations, or statistical significance tests for any experiment — all results are single-run point estimates. This is a significant methodological limitation: with test sets of unreported size (CharXiv, ChartQAPro, EvoChart) and only 128 geometric questions, even modest variance could affect relative rankings, particularly given that the performance gaps between some baselines are small (e.g., Debate vs. Direct Answer on ChartQAPro with GPT-4.1-mini: 57.75 vs. 57.85 in Table 1).

Main Quantitative Results

Chart Reasoning: Aggregate Performance Across Benchmarks and Backbones

Table 1 reports accuracy on CharXiv, ChartQAPro, and EvoChart for all methods across three backbone models. The headline result: PixelCraft consistently and substantially outperforms all baselines across every benchmark-backbone combination. The gains range from +5.6 to +9.5 percentage points over the strongest baseline per configuration.

GPT-4o backbone (Table 1, top row block). PixelCraft achieves 55.2 on CharXiv, 58.83 on ChartQAPro, and 70.24 on EvoChart. The strongest baseline varies by benchmark: CoT on CharXiv (51.1), CoT on ChartQAPro (56.52), CoT on EvoChart (68.64). The absolute improvements over these strongest baselines are +4.1, +2.31, and +1.60 respectively — modest but consistent. The performance of Refocus is notably poor: 47.2 on CharXiv (below Direct Answer's 49.6), 46.30 on ChartQAPro (below Direct Answer's 52.51), and 50.88 on EvoChart (far below Direct Answer's 62.64). This negative result for the only competing tool-based method validates the paper's claim that low-fidelity tools are actively harmful. Multi-agent Debate (50.7, 49.38, 65.52) and Reconcile (52.4, 54.36, 65.20) provide minimal uplift over CoT, with Debate actually underperforming CoT on ChartQAPro (49.38 vs. 56.52).

GPT-4.1-mini backbone (Table 1, middle row block). This is where PixelCraft shows its largest absolute gains. PixelCraft scores 68.1 on CharXiv, 65.56 on ChartQAPro, and 79.44 on EvoChart. The strongest baselines are Reconcile on CharXiv (63.5), CoT on ChartQAPro (62.21), and CoT on EvoChart (76.64). The improvements over these baselines are +4.6, +3.35, and +2.80. The Δ row reports gains over the next-best method: +9.5 on CharXiv, +7.71 on ChartQAPro, +8.16 on EvoChart — but note these are computed against all baselines including the weakest ones, not against the strongest baseline. The +9.5 on CharXiv compares against Refocus (60.7), not against the strongest baseline Reconcile (63.5). The actual gap over the strongest baseline is +4.6.

Claude-3.7-sonnet backbone (Table 1, bottom row block). PixelCraft scores 73.9 on CharXiv, 69.82 on ChartQAPro, and 80.48 on EvoChart. The strongest baselines are Reconcile on CharXiv (68.5), Debate on ChartQAPro (66.58), and Reconcile on EvoChart (78.56). Improvements over strongest baselines: +5.4, +3.24, +1.92. The Δ row reports +6.8, +6.99, +6.32 over the weakest baseline per benchmark. This reporting convention — comparing against the weakest baseline rather than the strongest — slightly overstates the practical gain. The more relevant comparison against the strongest baseline shows PixelCraft providing +3.24 to +5.4 points on Claude-3.7-sonnet, which is substantial but less dramatic than the Δ row suggests.

A cross-backbone pattern emerges: PixelCraft's relative gains are largest with GPT-4.1-mini and smallest with GPT-4o, particularly on EvoChart where the GPT-4o + CoT baseline already achieves 68.64 and PixelCraft adds only 1.60 points. This suggests that PixelCraft provides diminishing returns as the backbone model's intrinsic chart reasoning capability improves — the strongest backbone (Claude-3.7-sonnet + CoT: 77.92 on EvoChart) still gains from PixelCraft (80.48, +2.56) but the margin narrows. This is consistent with the paper's implicit theory: PixelCraft compensates for perceptual limitations, and stronger backbone models have fewer severe perceptual errors to correct.

Cross-benchmark pattern. The baseline hierarchy is consistent across backbones: CoT provides a clear boost over Direct Answer (3-6 points typically), validating that explicit reasoning helps. The multi-agent methods (Debate, Reconcile) show inconsistent and generally small improvements, confirming the paper's diagnosis that "multiple agents without specialized visual tools struggle to perform visual reasoning on charts" (Section 5.1). Refocus is the only baseline that sometimes underperforms Direct Answer — a critical finding because it shows that bad tools actively harm performance, making tool reliability (PixelCraft's grounding-based approach) not just beneficial but essential.

Geometric Reasoning: Auxiliary Line Subset of Geometry3K

Table 2 reports accuracy on the 128-question auxiliary-line subset of Geometry3K. The results show a clear but more compressed hierarchy than the chart benchmarks.

GPT-4o backbone. PixelCraft achieves 26.56, tied with Reconcile and above Debate (25.00) and CoT (17.19). Direct Answer scores 21.09 — notably higher than CoT (17.19), an inversion not seen in chart benchmarks. This suggests that for GPT-4o on geometric reasoning, unstructured chain-of-thought may introduce errors that direct answering avoids, possibly due to hallucinated geometric relationships in the reasoning chain.

GPT-4.1-mini backbone. PixelCraft achieves 34.38, above Reconcile (32.81), Debate (29.69), CoT (26.56), and Direct Answer (24.22). The progression is monotonic: each added capability (CoT → multi-agent → PixelCraft) improves accuracy. The gain over the strongest baseline (Reconcile) is +1.57 points.

Claude-3.7-sonnet backbone. PixelCraft achieves 33.59, above Reconcile (32.03), Debate (28.13), CoT (28.13), and Direct Answer (30.47). Again, Direct Answer (30.47) outperforms CoT (28.13) — the same inversion seen with GPT-4o, suggesting that unstructured reasoning on geometric problems can be counterproductive for some models. PixelCraft's gain over the strongest baseline (Reconcile at 32.03) is +1.56.

The overall picture on geometric reasoning is more modest than on chart reasoning: the gains are in the 1-2 percentage point range over the strongest baseline (Reconcile), compared to 3-5 point gains on chart benchmarks. This may reflect the smaller test set (128 questions) reducing statistical power, the different nature of geometric reasoning (which may depend less on precise value extraction and more on spatial reasoning that MLLMs handle better natively), or the smaller toolkit (only 3 geometric tools vs. 4 chart tools). The paper does not analyze this discrepancy.

Architectural Ablation: Visual CoT vs. PixelCraft Framework

Table 3 isolates the contribution of PixelCraft's agent architecture (image memory, role specialization, critic gating) from the contribution of its visual tools. The Visual CoT baseline uses the exact same tools and prompts as PixelCraft but with a simplified architecture: a single planner handles both task decomposition and reasoning, and all historical images are fed linearly into the context in chain-like fashion. The backbone model is GPT-4.1-mini.

On CharXiv, Visual CoT scores 65.0, while PixelCraft scores 68.1 — a +3.1 point advantage attributable to the agent architecture. On ChartQAPro, Visual CoT scores 61.04, while PixelCraft scores 65.56 — a +4.52 point advantage.

These results demonstrate that the visual tools alone are not sufficient — how they are orchestrated matters substantially. The paper attributes the gap to image memory enabling selective recall (avoiding long-context degradation and enabling backtracking) and to the visual critic preventing error propagation. This is one of the paper's strongest internal validity checks: by holding the tools constant and varying only the reasoning architecture, it cleanly isolates the contribution of the proposed workflow innovations.

Component Ablation: Sequential Addition of PixelCraft Components

Table 4 reports a systematic ablation on CharXiv and ChartQAPro using GPT-4.1-mini, adding PixelCraft components incrementally to a CoT baseline.

CoT baseline (no components): 63.8 on CharXiv, 62.21 on ChartQAPro.

+ Tool Agents (TA): 65.0 on CharXiv (+1.2), 63.66 on ChartQAPro (+1.45). The paper notes this is "the largest average performance gain relative to the no-component CoT baseline" (Section 5.3). This validates the paper's central claim that tool-based evidence improvement addresses the primary bottleneck.

+ Tool Agents + Dispatcher (TA + Disp): 65.9 on CharXiv (+0.9 over TA-only), 64.43 on ChartQAPro (+0.77). The dispatcher's contribution is filtering irrelevant tools, "leading to more accurate tool usage" (Section 5.3).

+ Tool Agents + Visual Critic (TA + VC, no Dispatcher): 66.0 on CharXiv (+1.0 over TA-only), 63.96 on ChartQAPro (+0.30). The visual critic provides a small but positive gain, "filtering invalid processed images and avoiding erroneous answers."

+ Tool Agents + Dispatcher + Visual Critic (TA + Disp + VC): 67.5 on CharXiv, 64.89 on ChartQAPro. This three-component combination shows synergistic gains — the total improvement over TA-only (+2.5 on CharXiv, +1.23 on ChartQAPro) exceeds the sum of Dispatcher-only (+0.9, +0.77) and VC-only (+1.0, +0.30) gains for CharXiv (2.5 > 1.9) and approximately matches for ChartQAPro (1.23 ≈ 1.07).

+ All components including Planning Critic (TA + Disp + VC + PC): 68.1 on CharXiv (+0.6 over TA+Disp+VC), 65.56 on ChartQAPro (+0.67). The planning critic's post-hoc refinement provides an additional small but consistent gain.

The cumulative improvement from CoT to full PixelCraft is +4.3 points on CharXiv and +3.35 points on ChartQAPro. The paper presents this as evidence that "each component contributes positively and cumulatively to the agent's overall reasoning capabilities" (Section 5.3).

Tool-Specific Analysis: Usage Frequency and Performance Gain

Figure 4 decomposes the contribution of individual visual tools using GPT-4.1-mini on CharXiv (charts) and Geometry3K (geometry).

Chart tools (Figure 4a-b). The four chart tools show dramatically different usage frequencies, reflecting the nature of questions in CharXiv:

• Subfigure Cropping: 351 calls (the dominant tool). Accuracy when this tool is used: 63.8 (base CoT without tools) → gain not explicitly stated as a delta, but the bar chart in Figure 4a shows the tool-specific accuracy is higher than the CoT baseline.
• Masking Data with Legend: 38 calls. Accuracy improves by approximately +18.4 percentage points over the base CoT method when this tool is invoked — the largest per-tool gain. This makes intuitive sense: questions requiring legend-based masking often involve isolating a specific data series from a cluttered multi-series chart, a task that is exceptionally difficult for MLLMs without explicit color-based masking.
• Region Magnification: 35 calls. Accuracy gain is shown as positive but smaller than Legend Masking.
• Adding Auxiliary Lines: only 3 calls. Despite the low frequency, the bar chart shows positive accuracy when used.

Geometric tools (Figure 4c-d). Point Connection dominates (56 calls), with Perpendicular Line (7 calls) and Parallel Line (6 calls) used much less frequently. The point connection tool achieves approximately 35.7% accuracy on its associated questions, while Perpendicular Line construction achieves approximately 57.1% — notably higher, though the small sample size (7 calls) makes this estimate unreliable.

The paper emphasizes that "all of the visual tools improve the performance over the base CoT method without visual tools, with a significant improvement on some tools" (Section 5.3). The imbalanced usage is characterized as "query-driven and image-driven" — the tools are invoked when the question and chart structure demand them, not based on arbitrary system preferences. The dominance of Subfigure Cropping reflects the multi-panel nature of CharXiv charts (arXiv figures frequently contain 4-16 subplots); the dominance of Point Connection reflects the need to visualize vertex relationships in geometry problems.

Self-Correction Analysis

Figure 5 characterizes the planning critic's error identification and refinement behavior on CharXiv with GPT-4.1-mini.

Error identification accuracy (Figure 5, left panel). Across three refinement rounds, the planning critic identifies queries as erroneous:

• Round 1: 39 true positives, 3 false positives (identification rate of 42 out of an unstated total)
• Round 2: approximately 3 true positives, approximately 1 false positive (dramatic drop)
• Round 3: approximately 1 true positive, 0 false positives

The rapid decline in identifications across rounds indicates that errors are largely resolved after one round of re-answering. The false positive rate is low (3/42 ≈ 7% in round 1), suggesting the critic is conservative — it rarely flags correct answers as wrong.

Refinement accuracy (Figure 5, right panel). The overall accuracy after refinement:

• Base (before any refinement): approximately 67.2%
• After round 1 refinement: approximately 67.6%
• After round 2: approximately 67.9%
• After round 3: approximately 68.1%

The gains from refinement are modest (+0.9 points total over three rounds) but monotonic. The paper frames this as evidence that "PixelCraft refines some incorrect answers, resulting in higher accuracy" (Section 5.3), though the magnitude of improvement is small relative to the gains from adding tool agents or the dispatcher.

Ablation Studies and Robustness Checks

• Grounding model comparison (Figure 3a). The paper tests PixelCraft with three different grounding backends — the fine-tuned Qwen2.5-VL-3B (0.93 IoU), the base Qwen2.5-VL-7B (0.26 IoU), and GPT-4.1-mini (0.31 IoU) — while holding all other system components constant. On CharXiv with GPT-4.1-mini backbone, the fine-tuned grounding model achieves approximately 68.1% accuracy, compared to approximately 64.1% for the base Qwen2.5-VL-7B and approximately 65.0% for GPT-4.1-mini grounding. The ~3-4 percentage point gap between the best and second-best grounding methods demonstrates that grounding quality directly and substantially impacts downstream reasoning — it is not merely a nice-to-have but a performance-critical component. A qualitative example (Figure 3b) shows the fine-tuned model correctly localizing a queried subfigure while both the base Qwen model and Refocus fail, explaining the quantitative gap.

• Grounding accuracy by element type (Table 6). The fine-tuned model's performance varies across chart element types: Subplot regions achieve 0.99 IoU (near-perfect), Legend regions achieve 0.89 IoU, Text labels (titles, axis labels) achieve 0.90 IoU, and Axis ticks achieve 0.93 PCK@0.01. The base Qwen2.5-VL-3B scores are 0.27, 0.04, 0.05, and 0.04 respectively — a 25× improvement on legend regions and a 23× improvement on axis ticks. The larger Qwen2.5-VL-7B (0.52, 0.17, 0.21, 0.15) and GPT-4.1-mini (0.78, 0.18, 0.11, 0.17) both show severely degraded performance on legend regions and axis ticks compared to subplot regions, suggesting that these element types are disproportionately difficult for general-purpose vision models and that the fine-tuning dataset's coverage of these element types is particularly valuable.

• Architecture ablation: Visual CoT vs. full PixelCraft (Table 3). As discussed above, this cleanly isolates the workflow architecture's contribution at +3.1 points on CharXiv and +4.52 on ChartQAPro. This is the strongest evidence that the image memory and agent orchestration — not just the tools — drive performance.

• Component-wise cumulative ablation (Table 4). Discussed in detail above. Each component (Tool Agents, Dispatcher, Visual Critic, Planning Critic) shows positive additive contribution, with Tool Agents providing the single largest gain and the Planning Critic providing the smallest.

• Self-correction round analysis (Figure 5). The planning critic identifies genuine errors at a high true-positive rate (39/42 in round 1) and the re-answering process monotonically improves accuracy (67.2% → 68.1% over three rounds). This validates that the critic is identifying real problems rather than generating noise. However, the absolute gain from three refinement rounds (+0.9 points) is small relative to the gains from tool agents (+1.2-1.45 points in a single pass), suggesting that error prevention (via visual critic gating) is more impactful than error correction (via planning critic re-answering) in this system.

• Tool usage frequency and per-tool performance gain (Figure 4). All seven visual tools (four chart, three geometric) show positive accuracy gains over the no-tool CoT baseline. The gains vary substantially by tool: Legend Masking provides the largest relative improvement (+18.4 points on CharXiv), while more frequently used tools like Subfigure Cropping show smaller per-use gains (which is expected — cropping is used for a broader range of question types, many of which are answerable without cropping, while legend masking is invoked specifically for questions that are very difficult without it). No tool shows negative gain — there are no tools that actively harm performance, unlike Refocus where tool use sometimes reduces accuracy below the Direct Answer baseline.

• Geometry3K auxiliary-line subset construction. The paper constructs this 128-question subset by filtering Geometry3K's test set for problems requiring intermediate visual clues (auxiliary lines). This is a non-standard split — the paper does not report how the filtering was done, whether it was manual or automatic, or whether the filter criteria were validated. This makes replication difficult and introduces potential selection bias. The subset may not be representative of Geometry3K as a whole, and the small size (128 questions) means per-model accuracy differences of 1-2 points (as seen in Table 2) may not be statistically meaningful.

• Backbone model robustness. The consistent gains across GPT-4o, GPT-4.1-mini, and Claude-3.7-sonnet (Table 1) are the paper's primary robustness check — they demonstrate that PixelCraft's benefits are not tied to a specific backbone architecture, training procedure, or scale. However, all three backbones are proprietary commercial models with unknown training data and architectures, so the paper cannot analyze why PixelCraft works across them beyond observing that it does. Testing on open-weight models would allow deeper analysis of the interaction between backbone capability and PixelCraft's added value.

• Refocus negative result. The paper reports that Refocus underperforms CoT on several configurations (GPT-4o on CharXiv: 47.2 vs. 51.1; GPT-4o on ChartQAPro: 46.30 vs. 56.52). This is treated as evidence that low-fidelity tools are harmful. However, the paper does not diagnose why Refocus fails — is it the specific tools, the grounding accuracy, the linear reasoning pattern, or the lack of critic verification? The paper attributes Refocus's failure to its tool quality, but this is an inference rather than a directly tested claim.

Critical Assessment

The paper makes several central claims, each of which receives varying degrees of experimental support.

Claim 1: PixelCraft significantly improves structured image reasoning across diverse benchmarks and backbones.

This claim is well-supported for chart reasoning. Table 1 shows consistent improvements over all baselines for all three backbone models on all three chart benchmarks. The gains range from +1.60 (GPT-4o on EvoChart, over CoT) to +5.4 (Claude-3.7-sonnet on CharXiv, over Reconcile). The effect is robust across benchmark difficulty (CharXiv's reasoning subset, ChartQAPro's compositional questions, EvoChart's real-world charts) and backbone model (three different providers, different scales).

However, the geometric reasoning results (Table 2) are substantially weaker. The gains over the strongest baseline (Reconcile) are +1.56-1.57 points on the 128-question auxiliary-line subset — small enough that statistical significance is questionable given the test set size. The paper does not provide a breakdown of why geometric gains are weaker. Possible explanations: (1) the geometric tool set (3 tools) is smaller and less diverse than the chart tool set (4 tools), limiting the system's ability to address different geometric reasoning patterns; (2) geometric reasoning may depend more on spatial transformations and angle relationships that MLLMs handle better than precise value extraction, leaving less room for improvement; (3) the 128-question subset may be too small to reliably measure small accuracy differences. The paper's geometric results should be viewed as suggestive but not conclusive — they demonstrate that PixelCraft can handle geometric reasoning, not that it provides large gains.

Claim 2: High-fidelity grounding is the foundation that makes visual tool use beneficial rather than harmful.

The evidence for this claim is strong and multi-faceted. Table 6 establishes that off-the-shelf grounding is poor (IoU 0.10-0.31) and that targeted fine-tuning dramatically improves it (IoU 0.93). Figure 3a shows that grounding quality directly impacts downstream accuracy within the same PixelCraft framework (+3-4 points for the fine-tuned model vs. base model grounding). The Refocus negative result (Table 1) provides external validation: a competing tool-based system with poor grounding underperforms even tool-free baselines. The per-tool accuracy gains in Figure 4 show that when tools operate correctly (backed by good grounding), they provide positive value — there is no evidence of a tool that helps in principle but is undermined by grounding in practice.

A missing experiment: the paper does not test PixelCraft with an intentionally degraded grounding model to establish the causal threshold at which grounding accuracy becomes insufficient. Would 0.50 IoU be enough? 0.70? The paper shows that ~0.30 (base models) is insufficient and ~0.93 is sufficient, but doesn't map the continuous relationship between grounding accuracy and downstream task accuracy. This limits the generalizability of the finding — other practitioners cannot determine what grounding accuracy they need to target.

Claim 3: The image memory and non-linear workflow provide gains beyond what linear visual CoT achieves with the same tools.

Table 3 provides clean evidence: +3.1 points on CharXiv and +4.52 on ChartQAPro when comparing the full PixelCraft workflow against a simplified Visual CoT baseline using identical tools. This is methodologically strong — the comparison holds tools constant and varies only the reasoning architecture.

However, the Visual CoT baseline may not represent the strongest possible linear architecture. The paper describes it as using "a single planner to handle both task planning and visual reasoning monolithically" — but it's unclear whether the baseline retains other PixelCraft components like the visual critic or dispatcher. If the baseline strips all architectural innovations (not just the image memory), the gap conflates multiple factors. A cleaner decomposition would test: (1) linear history with full agent roles vs. (2) linear history with monolithic planner vs. (3) image memory with monolithic planner vs. (4) image memory with full agent roles. This would separate the contribution of role specialization from the contribution of image memory.

Additionally, the paper does not directly demonstrate branching or backtracking behavior in its case studies. The image memory is described as enabling these capabilities, and the quantitative gains in Table 3 suggest the architecture provides value, but the paper does not show concrete examples where the planner revisits an earlier image and pursues a different reasoning path. The self-correction example (Figure 8) shows the planning critic identifying a missing tool and the planner re-answering with the tool added — but this is a restart from the beginning with additional guidance, not a mid-trajectory backtrack to an earlier visual state. Whether the image memory truly enables branching and backtracking, or whether it simply enables cleaner context management (avoiding multi-image degradation), is not definitively established by the presented evidence.

Claim 4: The two-layer critic architecture provides complementary error prevention and correction.

The ablation (Table 4) supports complementary contributions: Visual Critic alone (TA+VC) adds +1.0 on CharXiv and +0.30 on ChartQAPro over TA-only; Planning Critic (full system vs. TA+Disp+VC) adds +0.6 and +0.67 respectively. The gains are additive and consistent across benchmarks.

However, both critic contributions are small in absolute terms (0.3-1.0 points). The paper's framing of the visual critic as providing critical error prevention is plausible but not directly validated — the paper does not report how often the visual critic blocks an image, what fraction of those blocks prevent actual downstream errors, or what would happen without the visual critic (the ablation shows the aggregate effect, not the mechanism). The self-correction analysis (Figure 5) provides stronger evidence for the planning critic: 39 true positives vs. 3 false positives in round 1, with monotonic accuracy improvement across rounds. But the absolute gain from three rounds is only +0.9 points, raising the question of whether the planning critic's computational cost (re-running the entire reasoning pipeline) is justified by the modest improvement.

A missing experiment: what happens if the planning critic simply re-runs the same pipeline without any modifications? Is the gain from the critic's diagnostic feedback, or simply from having a second independent reasoning attempt? A baseline where the planner re-answers without critic feedback would isolate the value of the critic's specific suggestions.

Claim 5: PixelCraft's gains are attributable to the architecture and tools, not to prompt engineering.

The paper explicitly states that the reasoner uses "a brief generic instruction" (Section 3.1). The prompt templates in Appendix B.3 are disclosed, supporting transparency. However, the planner's prompt is highly structured and detailed — it includes explicit workflow instructions, formatting requirements, tool descriptions, and image pool management. This prompt engineering is itself a form of optimization. The paper's claim is better characterized as "gains are not attributable to reasoning-specific prompt optimization" rather than "no prompt engineering was involved." The distinction matters: the planner prompt's structure likely contributes to the system's performance, and the paper does not ablate prompt complexity (e.g., testing a minimal planner prompt vs. the full structured prompt).

Additional weaknesses and missing experiments:

• Test set sizes are not reported for chart benchmarks. The paper states it uses the "reasoning-focused question set" for CharXiv and the "full test sets" for ChartQAPro and EvoChart, but never specifies how many questions these contain. Without test set sizes, readers cannot assess statistical reliability or compare absolute accuracy numbers to other papers that may use different splits.

• No multiple-run statistics. All results are single-run point estimates. Without standard deviations or confidence intervals, it is impossible to determine whether a +1.5 point difference (e.g., PixelCraft vs. Reconcile on Geometry3K with GPT-4.1-mini) is statistically significant or within noise.

• No direct comparison to other chart-tool systems beyond Refocus. Visual Sketchpad (Hu et al., 2024) and OpenThinkImg (Su et al., 2025) are cited as related work but not evaluated as baselines. The paper argues that their tools are "specialized" and don't support the same range of chart types, but this claim is not empirically validated — a comparison on the subset of charts where their tools work would strengthen the paper's claim of generality.

• The grounding model is evaluated only on synthetic data. The test set of 500 samples used for Table 6 is "annotated using the same pipeline as our synthetic training data" (Appendix C.3). This means the grounding accuracy numbers (0.93 IoU) measure in-distribution performance on programmatically generated charts. The paper does not evaluate grounding accuracy on real charts (e.g., CharXiv's arXiv figures), leaving open the question of whether the synthetic-to-real transfer is as strong as implied. The downstream benchmark performance provides indirect evidence (good grounding would be necessary for the strong results), but direct grounding evaluation on real charts would substantially strengthen the claim.

• The geometry benchmark construction is opaque. The 128-question auxiliary-line subset is created by filtering Geometry3K's test set, but the filtering criteria are not described, and the representativeness of the subset is not validated. The small size (128 questions) limits the reliability of the reported accuracy differences.

• Latency tradeoff is reported but not analyzed. Table 7 shows PixelCraft taking 3-5× longer than CoT, but the paper does not discuss whether this latency is acceptable for deployment scenarios, how it scales with the number of agents and tool calls, or whether the accuracy gains justify the latency cost at the margin.

• No failure analysis. The paper does not report what types of questions PixelCraft still gets wrong, what the most common failure modes are, or whether the system has systematic biases (e.g., performing well on bar charts but poorly on scatter plots, or well on single-variable comparisons but poorly on multi-variable interactions). A failure taxonomy would guide future improvements and help practitioners understand when to deploy PixelCraft vs. alternative approaches.

Summary of evidential strength. The paper's strongest claims — that PixelCraft improves chart reasoning across diverse settings, that grounding quality is critical, and that the agent architecture adds value beyond tools alone — are well-supported by the presented experiments. The claims about geometric reasoning, non-linear branching behavior, and critic mechanisms are suggestive but less thoroughly validated. The paper's value is primarily as a demonstration that a hybrid MLLM+CV approach with careful tool orchestration can substantially improve structured image reasoning, with the specific architectural innovations (image memory, critic gating) showing additive value. The experiments do not establish that PixelCraft's particular configuration is optimal, nor do they characterize the boundary conditions where it succeeds vs. fails — but they convincingly establish that the approach is directionally correct and practically effective for the tested benchmarks.

6. Limitations and Trade-offs

6.1 Grounding Accuracy Validated Only on Synthetic Data, Not Real Charts

The assumption or constraint. The paper's grounding model — the foundation that makes all visual tools reliable — is fine-tuned and evaluated exclusively on programmatically synthesized charts. The 53,000 training samples are generated via a Matplotlib-based pipeline where GPT-4o diversifies code templates (Section 4). The 500-sample test set used for the IoU evaluation in Table 6 is "annotated using the same pipeline as our synthetic training data" (Appendix C.3). The paper never evaluates grounding accuracy on real charts — CharXiv's arXiv figures, ChartQAPro's diverse chart types, or EvoChart's real-world visual styles.

The consequence. The headline grounding accuracy of 0.93 overall IoU (Table 6) measures in-distribution performance on programmatically generated charts with controlled visual styles, known rendering parameters, and clean layouts. Real charts introduce distribution shift along multiple axes: anti-aliased vs. crisp fonts, non-standard color palettes, variable DPI and compression artifacts, hand-drawn annotation elements, and irregular subfigure arrangements that don't follow the grid patterns common in synthetic layouts. If grounding accuracy degrades significantly on real charts — even to, say, 0.60 IoU — the downstream consequences are severe: tool agents would crop the wrong regions, mask incorrect data series, or draw auxiliary lines at wrong positions, producing corrupted evidence that cascades into wrong answers. The paper's own evidence (Figure 3a) shows that grounding quality directly controls downstream accuracy: switching from fine-tuned grounding (~68% CharXiv accuracy) to base model grounding (~64%) costs ~4 percentage points. A further degradation on real charts would eat into PixelCraft's reported gains.

What evidence exists in the paper. The paper provides no direct grounding evaluation on real charts. The downstream benchmark results (Table 1) provide indirect evidence — the system achieves strong accuracy on CharXiv, ChartQAPro, and EvoChart, which implies the grounding works well enough on real charts to support correct answers. However, this is a joint test: good grounding + good reasoning = correct answer; bad grounding + compensatory reasoning might also coincidentally produce a correct answer. The paper cannot disentangle whether the grounding model maintains its 0.93 IoU on real charts or whether the strong downstream results reflect robustness to somewhat degraded grounding (e.g., 0.70 IoU may still be sufficient when the visual critic catches major errors). The illustrative grounding example in Figure 3b shows the fine-tuned model correctly localizing a subfigure on what appears to be a real chart, but this is a single qualitative example, not a quantitative evaluation.

Mitigation status. The paper does not explicitly acknowledge this as a limitation. The downstream benchmarks serve as an implicit validation — if grounding were severely degraded on real charts, the system would not achieve the reported accuracy gains. However, this is an argument from outcomes rather than a direct measurement. Future work should evaluate grounding accuracy on a hand-annotated subset of real charts from the target benchmarks to quantify the synthetic-to-real transfer gap. The paper's suggestion in Appendix D that "future work on fully automated tool generation and verification" could help does not directly address the grounding evaluation gap.

6.2 Difficulty Estimation and Compute Cost Are Unaccounted for in Headline Comparisons

The assumption or constraint. PixelCraft introduces substantial computational overhead relative to the baselines it outperforms. Table 7 reports average response times: on CharXiv, PixelCraft takes 16.45 seconds vs. 3.75 seconds for CoT with GPT-4.1-mini — a ~4.4× increase. On ChartQAPro, it takes 11.22 seconds vs. 3.44 seconds — a ~3.3× increase. This latency reflects the multi-agent workflow: the planner makes multiple reasoning steps, tool agents call the grounding model and execute CV operations, the visual critic evaluates intermediate images, and the planning critic reviews the full trace. The paper reports accuracy gains without adjusting for this compute differential — the comparison is between CoT at 3.75s and PixelCraft at 16.45s, not between compute-matched alternatives.

Beyond per-query latency, there are unaccounted fixed costs: fine-tuning the grounding model (one epoch on four A100 GPUs, Appendix B.1), generating the 53,000 synthetic training samples via GPT-4o, manually curating the tool set (which required human expert refinement after automated generation failed, Appendix A), and the computational cost of the dispatcher, visual critic, and planning critic all calling the backbone MLLM (GPT-4.1-mini or equivalent) with their own prompts.

The consequence. A practitioner deciding whether to deploy PixelCraft needs to evaluate a cost-benefit tradeoff: is a +4.3 percentage point gain on CharXiv (Table 4, CoT to full PixelCraft with GPT-4.1-mini) worth a ~4× increase in latency and the associated API costs? The answer depends on the application. For batch processing where throughput matters (e.g., processing thousands of charts overnight), 4× latency translates directly to 4× cost, and a less expensive approach — such as running the base CoT with a larger backbone model — might achieve similar accuracy at lower total cost. The paper's FLOPs-matched comparison absence (unlike the compute-optimal test-time scaling paper's Section 7) means there is no principled answer to "how much accuracy gain do I get per unit of additional compute?"

More subtly, PixelCraft's performance is bounded by the backbone MLLM's capability in its critic roles. If the visual critic or planning critic uses the same backbone as the reasoner, their errors are correlated — a model that misreads a chart value will also fail to detect that misreading during critic review. The paper's Figure 5 shows the planning critic has a non-zero false positive rate (3 out of 42 flagged queries in round 1), and the accuracy gains from three refinement rounds are only +0.9 points — suggesting that the critic's ability to improve answers is limited by the same perceptual bottleneck that causes the initial errors. Spending more compute on self-correction yields diminishing returns.

What evidence exists in the paper. Table 7 provides the latency numbers, and the paper is transparent that PixelCraft is slower. The ablation (Table 4) quantifies the per-component gains: Tool Agents add +1.2 points and introduce the tool execution latency; the Dispatcher adds +0.9 points; the Visual Critic adds +1.0 points; the Planning Critic adds +0.6 points and requires a full re-answering pass. A compute-adjusted analysis would ask: is the 0.6-point gain from the Planning Critic (which requires re-running the entire pipeline) worth doubling the total latency? The paper does not answer this.

Mitigation status. The paper reports the latency numbers in Table 7 but does not discuss the cost-benefit tradeoff. The paper frames the overhead as acceptable: "While slower than CoT, this trade-off is offset by the substantial accuracy gains discussed in the main text" (Appendix C.4). This is an assertion, not an analysis — "substantial accuracy gains" are not compared against alternative ways to spend the same compute budget (e.g., running majority voting with a larger backbone, or scaling the number of parallel samples). The paper identifies "developing more robust agent communication protocols or lightweight, specialized models for planning and criticism" as future work (Appendix D), which would partially address the cost issue by reducing per-step overhead.

6.3 Linear Visual CoT Baseline May Underrepresent Achievable Linear Performance

The assumption or constraint. Table 3 compares PixelCraft against a "simplified Visual Chain-of-Thought (CoT) baseline, which simply includes all the historical images for reasoning" using "a single planner to handle both task planning and visual reasoning monolithically." This baseline strips out PixelCraft's architectural innovations — the image memory, the role specialization (separate planner/reasoner/critics), and the dispatcher — while retaining the same visual tools.

The consequence. The ~3-4.5 point gap attributed to PixelCraft's architecture (Table 3: 68.1 vs. 65.0 on CharXiv, 65.56 vs. 61.04 on ChartQAPro) may overstate the contribution of the image memory and agent orchestration specifically, because the baseline removes multiple architectural components simultaneously. The paper cannot distinguish how much of the gap comes from: (1) the image memory enabling selective recall rather than linear concatenation; (2) role specialization (separate planner and reasoner) improving per-step performance; (3) the visual critic preventing error propagation (which the simplified baseline lacks); or (4) the planning critic enabling correction (also absent). A practitioner wanting to implement the minimal viable architecture would need to know which of these components drives the gain — but the paper's ablation in Table 4 adds components to a CoT baseline (without tools), not to the Visual CoT baseline, so the decomposition is incomplete.

Furthermore, the simplified Visual CoT baseline may not represent the strongest possible linear approach. A linear visual CoT system could include role specialization (separate planner and reasoner in sequence), a form of image selection (dropping earlier images when context length exceeds a threshold), or lightweight in-loop verification (checking tool outputs before passing to the reasoner) — all while maintaining a linear history structure. The paper does not test whether a well-optimized linear baseline would close the gap with PixelCraft's non-linear approach.

What evidence exists in the paper. Table 3 provides only the aggregate comparison between the simplified baseline and full PixelCraft. Table 4's ablation starts from a CoT baseline without tools, so the contribution of the image memory relative to a tool-equipped linear baseline is not directly measured. The paper attributes the gap to image memory enabling "flexible visual reasoning by allowing image selection and image recall for alternative branches of reasoning" (Section 5.3), but this attribution is an interpretation of the aggregate number, not a decomposition.

Mitigation status. The paper does not acknowledge this as a limitation. The Visual CoT baseline is presented as a valid architectural comparison, and the gap is attributed to the claimed innovations. A more rigorous decomposition would require: (1) a tool-equipped baseline with separate planner/reasoner but linear history; (2) a tool-equipped baseline with image selection but without branching/backtracking; and (3) the full PixelCraft. Each pairwise comparison would isolate one architectural feature. This decomposition is proposed as future work implicitly — the existing ablations already follow an additive logic — but the specific comparison that isolates image memory is absent.

6.4 The System Cannot Solve Hard Problems Where the Base MLLM Lacks Fundamental Capability

The assumption or constraint. PixelCraft's tools operate by presenting visual evidence to the reasoner in a more accessible form — cropping subfigures to reduce visual clutter, masking irrelevant data to highlight target series, adding reference lines to make comparisons explicit. The tools do not change the reasoner's underlying capability to interpret that evidence. If a question requires reasoning that exceeds the backbone MLLM's ability — complex multi-step logical deduction, integration of information across many subfigures, or domain expertise the model lacks — PixelCraft's tools provide no remedy. The paper's geometric reasoning results (Table 2) hint at this ceiling: the absolute accuracies remain low (33.59% for the strongest configuration, Claude-3.7-sonnet + PixelCraft on the auxiliary-line Geometry3K subset), and the gains over baselines are modest (+1.56 over Reconcile). On the 128-question subset, a full two-thirds of problems remain unsolved even with PixelCraft's best configuration.

The consequence. PixelCraft's value proposition is bounded: it helps when the model can reason correctly but is defeated by perceptual noise (misreading values, confusing subfigures, failing to isolate relevant data). It does not help when the model fundamentally cannot solve the problem regardless of how cleanly the evidence is presented. The paper does not characterize what fraction of benchmark errors fall into each category. Without this analysis, a practitioner cannot predict whether PixelCraft will help on their specific task distribution — if their problems are inherently too hard for the backbone model, the system's overhead yields no benefit.

This limitation also interacts with backbone model choice. Table 1 shows that PixelCraft's absolute gains are smaller with stronger backbone models on benchmarks where those backbones already perform well: Claude-3.7-sonnet on EvoChart with CoT achieves 77.92; PixelCraft adds +2.56 points to reach 80.48. On the same benchmark, GPT-4.1-mini + CoT achieves 76.64; PixelCraft adds +2.80 points. The gains are roughly constant in absolute terms, meaning the relative improvement shrinks as the baseline rises — there is less room for improvement as the backbone approaches the benchmark's ceiling. For problems near or at the backbone's capability ceiling, PixelCraft's tools provide marginal or zero benefit.

What evidence exists in the paper. The geometric reasoning results (Table 2) provide the clearest evidence: even with PixelCraft, accuracy on the auxiliary-line Geometry3K subset is 33.59% for the strongest configuration. The paper does not analyze what types of geometric problems remain unsolved, or whether the failures are due to perceptual errors (which tools should fix), reasoning errors (which tools cannot fix), or fundamental gaps in the backbone's geometric knowledge. The self-correction analysis (Figure 5) shows diminishing returns from re-answering — after three rounds, accuracy plateaus at 68.1% on CharXiv, suggesting the remaining ~32% of errors are not addressable by the current system's mechanisms. The paper does not categorize these residual errors.

Mitigation status. The paper does not explicitly discuss this capability bound. The focus is on the positive results — the gains over baselines — rather than on characterizing the problems that remain unsolved. This is a standard limitation of systems papers that evaluate on benchmarks without failure analysis, but it is particularly consequential for PixelCraft because the system's value proposition (paying a 4× latency cost for accuracy gains) depends on knowing when those gains will materialize. The paper's Appendix D acknowledges that "PixelCraft requires strong backbone MLLMs for task decomposition and tool orchestration" and that "weaker ones may constrain the framework's effectiveness with incorrect planning and tool calling," but this addresses the lower bound (backbone too weak for the planner role) rather than the upper bound (backbone insufficient for the reasoning task regardless of tools).

6.5 The Tool Set Is Manually Curated and Not Dynamically Extensible

The assumption or constraint. PixelCraft's visual tools are a fixed set of seven operations: four for chart reasoning (subfigure cropping, legend-based masking, region magnification, adding auxiliary lines) and three for geometric reasoning (point connection, perpendicular line construction, parallel line construction). These tools were produced through a semi-automated process: GPT-4.1-mini generated 468 tool candidates from 500 ArxivQA questions, the candidates were clustered, the most promising were selected from recurring categories, GPT-o3 rewrote them, and human experts performed final manual tuning for "correctness and practical usability" (Section 3.1, Appendix A). The paper explicitly reports that purely automated tool generation failed — constrained-generation tools that attempted novel operations (computing means of scatter plot points, detecting line intersections) "consistently exhibited low accuracy" even after GPT-o3 rewriting (Appendix A, Figure 6b).

The consequence. The tool set is a closed set, tuned for the chart and geometry benchmarks evaluated in the paper. If a new chart type or question type requires an operation not in the tool set — for example, extracting data from a ternary plot, reading values from a polar chart, or interpreting a Sankey diagram — PixelCraft has no mechanism to generate or acquire the necessary tool dynamically. The system would fall back to the reasoner's native visual capability, which Table 1 shows is substantially weaker than tool-augmented reasoning (e.g., GPT-4.1-mini CoT on CharXiv: 63.8 vs. PixelCraft: 68.1).

More critically, the tool set's coverage of the evaluation benchmarks may be artificially high because the tool generation process was seeded from ArxivQA questions, which share the multi-panel scientific chart domain with CharXiv. The paper does not report whether the 468 auto-generated tools included candidates that would handle chart types not present in the benchmarks, or whether the clustering process implicitly filtered to operations that match the benchmark distribution. This creates a risk of benchmark-adjacent tool curation — the tools work well on CharXiv and ChartQAPro because the questions that motivated tool design came from a similar distribution. A practitioner processing a different chart corpus (financial charts, medical imaging, engineering schematics) cannot assume the existing tool set will suffice, and the paper provides no guidance on how much manual effort is required to extend it.

The manual curation requirement also limits scalability and reproducibility. The paper acknowledges (Appendix D) that "purely LLM-generated tools were often ineffective" and that "future work on fully automated tool generation and verification is thus critical to improve adaptability and reduce this curation overhead." Until such automation exists, each new domain would require the same semi-automated-then-manual pipeline, with uncertain transfer of the approach's success.

What evidence exists in the paper. Appendix A provides the detailed narrative of tool generation and the negative result on constrained generation. Figure 4 shows tool usage frequency — the four chart tools cover the CharXiv questions, but the frequency distribution (Subfigure Cropping: 351 calls, Legend Masking: 38, Region Magnification: 35, Adding Auxiliary Lines: 3) reveals that one tool dominates and two are rarely used. This raises the question of whether the tool set is well-matched to the benchmark or whether a larger, more diverse tool set would yield different results. The paper does not report the frequency with which questions require operations not covered by any tool — a metric that would quantify the tool set's coverage gap.

Mitigation status. The paper is transparent about the manual curation in Appendix A and explicitly identifies fully automated tool generation as future work in Appendix D. However, the paper does not provide a protocol for tool set extension, an analysis of what fraction of benchmark errors are attributable to missing tool capabilities, or a characterization of the tool set's coverage relative to the full space of chart and geometric operations. The limitation is acknowledged but not quantified, leaving practitioners uncertain about how much effort would be required to adapt PixelCraft to new domains.

6.6 Evaluation Is Limited to a Narrow Family of Benchmarks and Proprietary Backbones

The assumption or constraint. All chart reasoning experiments use three benchmarks — CharXiv, ChartQAPro, and EvoChart — that share a common domain: scientific and statistical charts with quantitative axes, legends, and multi-panel layouts. The geometric reasoning benchmark (Geometry3K auxiliary-line subset, 128 questions) is the only non-chart evaluation. All experiments use proprietary backbone MLLMs: GPT-4o, GPT-4.1-mini, and Claude-3.7-sonnet. The paper does not evaluate on: (1) non-scientific chart types (flowcharts, organizational charts, Gantt charts, UML diagrams); (2) natural image reasoning tasks (which would test whether the image memory and critic architecture generalize beyond structured images); (3) open-weight models (which would enable reproducibility and deeper analysis of model-specific effects); or (4) non-English or multilingual chart benchmarks.

The consequence. A practitioner working with chart types outside the scientific/statistical domain — technical diagrams, process flows, network topologies, architectural plans — has no empirical evidence that PixelCraft's tool set and architecture transfer. The tools were designed for and evaluated on charts with numerical axes and data series; they may be inapplicable or actively harmful on chart types that lack these elements (e.g., a flowchart has no axes, legends, or subfigures to crop or mask).

The exclusive use of proprietary backbones limits reproducibility and analysis. The paper cannot report model size, training data, or architecture for GPT-4o or Claude-3.7-sonnet, so the relationship between backbone capability and PixelCraft's added value cannot be characterized parametrically. The paper observes that gains vary across backbones (larger for GPT-4.1-mini, smaller for Claude-3.7-sonnet on some benchmarks) but cannot explain why — is it a function of model size, training data composition, instruction tuning quality, or something else? Open-weight models would allow controlled experiments varying model scale while holding architecture constant, directly testing the paper's implicit claim that PixelCraft compensates for perceptual limitations independent of reasoning quality.

The narrow benchmark scope also raises questions about the generality of the image memory's claimed benefits. Table 3 shows the image memory architecture provides +3.1-4.5 points on chart benchmarks where subfigure cropping and multi-step comparisons are common. It is unclear whether these gains would transfer to tasks with different reasoning patterns — single-image questions, tasks requiring spatial rather than quantitative reasoning, or tasks where the primary challenge is visual search rather than value extraction. The paper's geometric results (Table 2), while limited in statistical power, show smaller gains (+1.56 over Reconcile), suggesting that the architecture's benefits may be domain-specific.

What evidence exists in the paper. Tables 1 and 2 cover four benchmarks total, three of which are chart-specific. The paper does not claim generality beyond structured images and explicitly scopes its contributions to "chart and geometric understanding" (Section 3). The consistent results across three proprietary backbones are presented as a robustness check (Section 5.1), but the paper acknowledges that the backbone models are "representative of the capabilities of many contemporary LLMs" without evidence for this representativeness claim.

Mitigation status. The paper does not explicitly identify the benchmark or backbone scope as a limitation. The stated scope — charts and geometric diagrams — is clear and the experiments align with it. However, the paper's language in Section 1 ("setting a new standard for structured image reasoning") and Section 6 ("establishing a new standard for reliable and sophisticated multimodal reasoning") implies broader generality than the empirical coverage supports. The paper's future work suggestion of "evolving PixelCraft into a more general and adaptable framework for structured image reasoning" (Appendix D) implicitly acknowledges the current scope limitation, but the paper does not specify which additional domains would need to be tested to validate generality.

7. Implications and Future Directions

How This Work Changes the Landscape

PixelCraft does not introduce a new model architecture, a new training paradigm, or a new theoretical framework. It is a systems contribution — a carefully engineered multi-agent architecture that combines existing components (MLLMs, classical CV, multi-agent orchestration) into a configuration that substantially outperforms both monolithic reasoning and prior tool-augmented approaches on structured image benchmarks. The paper's impact on the field comes from three conceptual reframings, each supported by specific empirical evidence.

Reframing 1: The bottleneck in structured image reasoning is perceptual, not cognitive — and it is fixable with targeted engineering rather than scale.

Prior to this work, the dominant responses to poor chart understanding performance were to scale up model size (train larger MLLMs), improve reasoning capabilities (fine-tune on CoT data, apply RL), or add more agents to debate answers. The paper's results — particularly the failure of Debate and Reconcile to improve over CoT (Table 1) and the substantial gains from simply adding high-fidelity tool agents (Table 4: +1.2-1.45 points from tool agents alone) — demonstrate that reasoning quality is not the binding constraint for current MLLMs on structured images. The models can reason; they cannot see precisely enough to populate their reasoning with correct evidence.

This shifts the research agenda for structured image understanding away from reasoning enhancement and toward perception enhancement. The paper makes this shift actionable by quantifying grounding accuracy directly (Table 6: 0.10 IoU for the base Qwen2.5-VL-3B, 0.93 after fine-tuning) and showing that grounding quality directly controls downstream accuracy (Figure 3a: a 3-4 point gap between grounding backends within the same PixelCraft framework). This creates a new diagnostic standard: papers proposing improvements to chart or diagram understanding should report intermediate localization accuracy on structured elements (subfigures, legend items, axis ticks), not just end-task accuracy. A claim of improved chart understanding that does not verify whether the model is actually looking at the right pixels is incomplete.

The magnitude of this shift is incremental but discipline-changing. It does not overturn the deep learning paradigm — it refocuses it on a neglected component. The paper's finding that a 3B parameter model fine-tuned on 53k synthetic examples achieves 3× better grounding accuracy than models 2-8× its size (Qwen2.5-VL-7B: 0.26 IoU, GPT-4.1-mini: 0.31 IoU) is a miniature version of the data-quality-over-scale insight that has reshaped NLP and vision, applied to a previously unmeasured sub-capability. It suggests that the field's default assumption — that general visual capability scales with model size and transfers to all visual tasks — is false for the specific skill of precise element localization in structured images.

Reframing 2: Visual reasoning benefits from architectural separation between perception and reasoning, with verified intermediate states.

The paper's comparison between Visual CoT (linear, monolithic) and PixelCraft (non-linear, role-separated) in Table 3 quantifies the value of architectural separation: +3.1 points on CharXiv and +4.52 on ChartQAPro when using the exact same tools but with a different orchestration architecture. This is not a marginal optimization — it is a 5-7% relative improvement from changing how the system accesses and verifies visual evidence, holding the evidence content constant.

This finding pushes against the end-to-end training philosophy that has dominated multimodal AI. It suggests that for tasks requiring precise, multi-step visual operations, externalizing intermediate states as explicit images managed by a separate planner is more effective than training a single model to handle everything internally. The image memory — a simple data structure (storing processed images with descriptions and enabling selective recall) — functions as a cognitive prosthesis for the MLLM, compensating for its inability to maintain precise visual representations across long reasoning chains.

The paper does not fully resolve the end-to-end vs. hybrid debate — it does not train an end-to-end model with comparable intermediate supervision and compare against PixelCraft — but it provides strong circumstantial evidence that the hybrid approach is effective for the tested benchmarks. This increases the attractiveness of neurosymbolic architectures for visual reasoning: learned components for semantic understanding, engineered components for precise geometric operations, and a learned orchestrator managing the interaction. The paper's tool generation negative result (Appendix A: purely LLM-generated visual tools are unreliable) further supports this direction by showing that learned components cannot yet handle the precision side of the operation.

Reframing 3: Tool-augmented visual reasoning has a reliability threshold — below it, tools hurt; above it, tools help substantially.

The paper's most striking negative result is not its own failure case but the performance of Refocus, the only competing tool-based baseline. Table 1 shows Refocus with GPT-4o scoring 47.2 on CharXiv (below Direct Answer's 49.6) and 46.30 on ChartQAPro (far below CoT's 56.52). The paper attributes this to "low-fidelity image processing" — tools that operate on incorrectly localized regions produce corrupted evidence that is worse than no tool use at all.

This establishes a reliability threshold principle for visual tool systems: tool accuracy must exceed the baseline model's native perceptual accuracy for tools to provide net benefit. When tools are noisy, they introduce new error modes (wrong cropping, wrong masking) that compound with the model's existing reasoning errors, producing worse outcomes than letting the model attempt the task with its native vision. This principle, while intuitive, was not previously demonstrated with controlled evidence on structured image benchmarks. The paper's grounding model achieves 0.93 IoU, which is sufficiently above the threshold to produce consistent gains; the base model's 0.10-0.31 IoU is below it, explaining Refocus's failures.

This reframing has implications for how the field evaluates tool-augmented systems. A paper proposing a new visual tool should not only demonstrate that the tool improves performance when working correctly but should also report the tool's reliability rate (how often it produces the intended output vs. an erroneous one) and analyze whether unreliable tool outputs are caught by downstream verification (e.g., a visual critic) or propagate into errors. The paper's visual critic — which gates tool outputs before they reach the reasoner — is an architectural response to this principle, and its positive contribution in the ablation (Table 4: +1.0 on CharXiv) validates the approach.

Reconciling prior contradictions. Before PixelCraft, there was an apparent contradiction in the literature: some works claimed visual tools improve chart understanding (Hu et al., 2024; Su et al., 2025), while Refocus showed tools sometimes underperform baselines. The paper resolves this by showing that tool quality, not tool presence, determines the outcome. The grounding accuracy metrics in Table 6 explain why: tools backed by precise localization (0.93 IoU) help; tools backed by imprecise localization (0.10-0.31 IoU) hurt. The contradiction was an artifact of evaluating systems with different grounding quality under the same "tool-augmented" label.

Research directions that become more attractive:

• Training specialized grounding models for specific domains (medical images, engineering diagrams, financial charts) using synthetic data pipelines similar to the paper's — the 53k-sample pipeline yielding 0.93 IoU demonstrates feasibility at modest scale.
• Architectures with explicit visual state management (image memory as a first-class component) rather than linear context concatenation — the +3.1-4.5 point gain in Table 3 is large enough to justify architectural investment.
• Two-layer error correction with in-loop gating (visual critic) and post-hoc review (planning critic) — the additive gains in Table 4 show these address distinct failure modes.
• Measuring grounding accuracy as a standard diagnostic when building visual tool systems — the paper provides a replicable methodology (synthetic data generation + IoU/PCK metrics).

Research directions that become less attractive:

• Purely textual multi-agent debate for visual reasoning — Table 1 shows Debate and Reconcile provide minimal benefit on chart benchmarks, confirming that symmetric agent architectures cannot compensate for shared perceptual limitations.
• End-to-end training of chart understanding without explicit intermediate visual states — the paper does not disprove this approach, but the large gains from externalizing visual operations shift the burden of proof onto end-to-end methods to demonstrate they can match the precision of explicit tool operations.
• Automated tool generation without verification — the negative result in Appendix A (LLM-generated tools are unreliable for visual operations) suggests this direction is premature and that manual curation or strong automated verification is currently necessary.

Follow-Up Research This Work Enables

Synthetic-to-real grounding gap quantification. The paper's grounding model is trained and evaluated exclusively on programmatically generated charts (Section 4, Appendix C.3). The 0.93 overall IoU measures in-distribution performance. A critical follow-up would manually annotate a subset of 100-200 real charts from CharXiv or ChartQAPro with bounding boxes for subfigures, legends, and axis ticks, then evaluate the fine-tuned grounding model on these real charts. The key measurement: the IoU drop from synthetic (0.93) to real. If the drop is small (e.g., to 0.85+), the synthetic data pipeline is validated as a general grounding solution. If the drop is large (e.g., to 0.50-0.60), then the downstream accuracy gains in Table 1 suggest the system is robust to moderate grounding degradation — or that the visual critic is compensating more heavily than the paper reports. This experiment would characterize the system's real-world robustness and inform whether the synthetic data pipeline needs augmentation with real annotated samples.

Compute-normalized comparison against scaled baselines. PixelCraft's latency is 3-5× higher than CoT (Table 7: 16.45s vs. 3.75s on CharXiv). A rigorous follow-up would compare PixelCraft against baselines that are allocated equivalent compute. The most natural comparison: GPT-4.1-mini + PixelCraft (16.45s, 68.1% on CharXiv) vs. Claude-3.7-sonnet + CoT with self-consistency at N=4 or N=8 (matching the 16.45s budget through parallel sampling). Does PixelCraft's architectural sophistication outperform simply using a stronger model with majority voting? A second comparison: PixelCraft with the planning critic disabled (saving the re-answering cost) vs. PixelCraft running at N=2 self-consistency (two independent PixelCraft runs with majority voting, matching the same total latency). This would determine whether the planning critic's 0.6-point gain (Table 4) is more cost-effective than simply running the system twice. The experiment would directly address the paper's unexamined cost-benefit tradeoff.

Failure taxonomy and capability-bound characterization. The paper reports accuracy improvements without analyzing the 32% of CharXiv questions that PixelCraft still gets wrong (100 - 68.1 = 31.9%, from Table 4). A systematic failure analysis on a random sample of 100-200 errors would categorize them into: (1) grounding failures (tool operated on wrong region but visual critic didn't catch it); (2) tool inadequacy (question requires an operation not in the tool set); (3) reasoning errors (evidence was correct but the reasoner drew wrong conclusions); (4) critic failures (planning critic missed an error or incorrectly flagged a correct answer — the paper reports 3 false positives in Figure 5, but what were their consequences?); and (5) fundamentally unsolvable (question exceeds backbone model capability regardless of evidence quality). The distribution across categories would tell practitioners what to expect when deploying PixelCraft on their own tasks, guide prioritization of future improvements (e.g., if 40% of errors are grounding failures, invest in the grounding model; if 40% are missing tools, expand the tool set), and establish the performance ceiling for the current backbone model.

Domain transfer to non-scientific chart types. The paper's charts are scientific and statistical — line plots, bar charts, scatter plots — from arXiv papers and curated benchmarks. A direct test of generality would evaluate PixelCraft on a benchmark of organizational charts, flowcharts, process diagrams, or network topologies — chart types that lack numerical axes, legends, and data series but require spatial reasoning about connectivity, hierarchy, and sequence. The existing chart tools (subfigure cropping, legend masking, axis-based region magnification, auxiliary reference lines) would be largely inapplicable. The experiment would measure: (a) what baseline accuracy does the backbone MLLM achieve on these chart types? (b) does the planner attempt to use tools, and if so, do the tools fail (producing corrupted evidence) or does the dispatcher correctly deactivate them? (c) does the image memory architecture provide any benefit when the tool set is irrelevant (i.e., is the architectural contribution separable from the tool contribution)? If the architecture alone (without domain-matched tools) provides no benefit, it would confirm that PixelCraft's value is tool-dependent and domain-specific. If it provides a small benefit (e.g., through better context management), it would suggest a more general architectural contribution.

Open-weight backbone implementation for controlled scaling analysis. All experiments use proprietary models with unknown sizes and training recipes, preventing parametric analysis of how backbone capability interacts with PixelCraft's added value. Reimplementing PixelCraft with open-weight models of varying sizes (e.g., Qwen2.5-VL at 3B, 7B, 72B; or LLaVA variants at multiple scales) would enable controlled experiments: does the absolute gain from PixelCraft decrease as backbone model size increases (as the paper's cross-backbone results weakly suggest, with Claude-3.7-sonnet showing smaller relative gains than GPT-4.1-mini)? Is there a crossing point where the backbone model becomes sufficiently capable that PixelCraft's overhead is no longer justified? Does the optimal configuration (which tools to include, how many refinement rounds) depend on backbone scale? This would transform PixelCraft from a demonstration on three proprietary models into a characterized scaling relationship, enabling practitioners to predict whether PixelCraft will help for their specific backbone model size and capability level.

Dynamic tool generation with execution-grounded verification. The paper's negative result — that purely LLM-generated visual tools are unreliable (Appendix A) — is not a dead end but a problem specification. A strong follow-up would develop a system where an LLM generates tool code that is then executed on a validation set of synthetic charts with known ground truth, and only tools that achieve above-threshold accuracy (e.g., >0.90 IoU on the grounding step, >95% correctness on the operation) are admitted to the toolkit. This closes the loop that the paper left open: the LLM generates candidate tools, an automated verifier tests them against ground-truth annotations (available from the synthetic data pipeline), and human curation is replaced by quantitative filtering. The experiment would measure: (a) what fraction of generated tools pass the verification threshold? (b) does the verified tool set achieve comparable accuracy to the manually curated set on downstream benchmarks? (c) does this pipeline successfully generate tools for chart types not in the original training distribution (e.g., radar charts, polar plots, ternary diagrams)? Success would directly address the paper's self-identified limitation that "purely LLM-generated tools were often ineffective" and move toward the "fully automated tool generation" the paper calls for.

Practical Applications and Downstream Use Cases

Scientific literature analysis at scale. PixelCraft is directly applicable to automated extraction and reasoning over figures in scientific papers — exactly the domain of CharXiv, whose charts come from arXiv papers. A research organization or pharmaceutical company processing thousands of papers could deploy PixelCraft to answer compositional queries about published results: "compare the dose-response curve in Figure 3B of paper X with Figure 4A of paper Y" or "what is the IC50 value reported in the bottom-right panel of Figure 2?" The 68.1% accuracy on CharXiv with GPT-4.1-mini (Table 4) provides a realistic performance estimate. The grounding model's 0.99 IoU on subplot regions (Table 6) means that isolating the correct subfigure — a prerequisite for any answer — is near-perfect. The 16.45s average latency (Table 7) is acceptable for batch processing overnight but would need reduction (through the lightweight planning models suggested in Appendix D) for interactive use.

Automated report generation from business intelligence dashboards. Enterprise dashboards often present complex multi-panel visualizations: revenue by region in one panel, customer churn by cohort in another, with shared legends and axis scales. PixelCraft could automate the generation of narrative summaries: given a dashboard image, crop relevant subfigures, mask data series by legend, extract values, and compose a text summary. The Legend Masking tool's +18.4 percentage point gain over the no-tool baseline (Figure 4a) directly addresses the common failure mode where an MLLM confuses which data series corresponds to which legend item in a dense multi-series chart. The image memory enables generating section-by-section narratives where earlier observations inform later ones (e.g., "Revenue in Q3 declined, which we can see reflected in the customer churn panel...").

Geometry problem-solving in educational technology. PixelCraft's geometric reasoning tools provide a concrete capability for automated geometry tutoring systems. Given a geometry problem diagram, the system can construct auxiliary lines (point connections, perpendiculars, parallels) to make geometric relationships visually explicit, then reason about the resulting configuration. The 34.38% accuracy on the Geometry3K auxiliary-line subset with GPT-4.1-mini (Table 2) is modest but above all baselines, and the tool usage analysis (Figure 4c-d) shows that Point Connection — the most fundamental geometric operation — is used in 56 of 69 tool calls. An educational application could use PixelCraft not for fully autonomous problem-solving but for generating hints: "Try connecting points A and C — what do you notice about the resulting triangle?" The fact that geometric tools produce explicit visual outputs means that hints are accompanied by annotated diagrams, making the system's suggestions interpretable to students.

Financial document processing with audit trails. Financial analysts and auditors process documents containing complex charts — portfolio performance over time, risk factor decomposition, scenario analysis with multiple subplots. PixelCraft's architecture provides a key property beyond accuracy: auditability. The planner produces a traceable sequence of THOUGHT-ACTION-OBSERVATION steps (Appendix B.3); each tool execution produces an explicit intermediate image stored in memory; the visual critic's gating decisions are logged; and the planning critic provides explicit post-hoc feedback. If an analyst questions a system-generated conclusion ("why does it say the Sharpe ratio declined by 0.3?"), the full reasoning trace — including the cropped subfigure, the masked data series, and the reference line at the threshold value — is available for inspection. This is a significant advantage over end-to-end MLLM reasoning, where the evidence for a conclusion is distributed across the model's activations and cannot be retrieved for human review. The 3-5× latency increase over CoT (Table 7) is acceptable in audit contexts where accuracy and traceability matter more than speed. </output>