ArXiv: 2603.10757

🎯 Pitch

Scaling a model’s ability to perceive STEM imagery consistently beats scaling its reasoning—perception is the true bottleneck. CodePercept transforms this insight into action by using executable code as a hallucination-free perceptual medium, training models to generate code that reconstructs images rather than relying on error-prone natural language captions.

1. Executive Summary

This paper introduces CodePercept, a paradigm that leverages executable Python code as a verifiable perceptual medium to systematically enhance the visual perception capabilities of Multimodal Large Language Models (MLLMs) in STEM domains. Through a scaling analysis decoupling perception (image-to-caption) from reasoning (caption-to-answer) across Qwen3-VL models on the MathVision dataset, the authors establish that scaling perception consistently outperforms scaling reasoning, identifying perception as the primary bottleneck. The framework materializes this insight via two Code-Grounded training tasks—Code-Grounded Caption Generation (using executable code as ground truth to eliminate hallucination in image captions) and STEM Image-to-Code Translation (training models to generate reconstruction code as a structured alternative to ambiguous natural language)—trained on ICC-1M, a dataset of 1M Image-Caption-Code triplets constructed through image reproduction, principled diversification, and solid geometry synthesis pipelines. At 4B and 8B scales, CodePercept-S1 improves perception accuracy by 2.8–3.0% in captioner-solver evaluations across six STEM benchmarks, while on the authors' novel STEM2Code-Eval benchmark—which evaluates comprehensive visual perception through executable code generation for image reconstruction rather than problem-solving proxies—CodePercept-8B-R1 achieves an average score of 63.56, surpassing substantially larger models including Qwen3-VL-Plus-Thinking, establishing that code-grounded perception is effective only when executable code provides verifiable ground truth for both caption synthesis and structured visual understanding.

2. Context and Motivation

The Core Problem: We Don't Know Whether MLLMs Fail at STEM Because of Perception or Reasoning

When a Multimodal Large Language Model (MLLM) attempts to solve a geometry problem involving auxiliary line constructions in a complex polyhedron, and it produces the wrong answer, what went wrong? Did the model fail to see the auxiliary lines correctly (a perception failure)? Or did it see them perfectly but reason incorrectly about their geometric implications (a reasoning failure)? This question is more than academic taxonomy—it determines where research effort and compute resources should be invested. Yet, prior to this work, the field had no systematic way to answer it.

The paper frames this diagnostic challenge directly in Section 1:

"A fundamental question remains: What is the true bottleneck limiting MLLMs in STEM, and when models fail, is it due to perception deficiencies or reasoning limitations?"

This is the central gap the paper addresses. Without knowing whether perception or reasoning constitutes the primary bottleneck, the research community has been allocating resources blind. If perception is the bottleneck, then all the sophisticated reasoning-enhancement techniques developed in recent years—reinforcement learning with step-wise rewards, chain-of-thought distillation, Monte Carlo Tree Search—are optimizing the wrong part of the pipeline. They are building increasingly elaborate reasoning engines on top of a noisy, unreliable perceptual foundation. Conversely, if reasoning is the bottleneck, then efforts to improve visual perception represent misallocated effort that would be better directed toward advancing inference-time computation or RL-based policy optimization.

Why This Matters: The Asymmetry of Recent Progress

The urgency of this diagnostic question arises from a striking asymmetry in recent MLLM research. As the paper documents in Section 2, the past two years have seen an explosion of work on reasoning enhancement for MLLMs:

  • Cold-start thinking data curation: Researchers meticulously design high-quality seed datasets providing reasoning patterns for models to imitate (Deng et al., 2025; Meng et al., 2025; Wang et al., 2025).
  • RL-based reasoning optimization: Works like R1-OneVision (Yang et al., 2025), Perception-R1 (Yu et al., 2025), and Ursa (Luo et al., 2025) employ reinforcement learning with carefully designed reward mechanisms to iteratively improve reasoning performance.
  • Unimodal thinking data transfer: Several studies (Chen et al., 2025; Wei et al., 2025; Xiaomi et al., 2025) demonstrate that high-quality text-only reasoning data can substantially enhance MLLM reasoning when appropriately integrated.

These efforts have produced genuine advances in reasoning capability. Yet the paper argues—and empirically demonstrates—that this progress has been building on an unexamined assumption: that perception is "good enough" and that reasoning constitutes the limiting factor. Section 2 states this position explicitly:

"Despite these significant advances in reasoning, our scaling analysis reveals that visual perception remains the primary bottleneck in STEM field. However, existing research has largely overlooked the fundamental role, with few works explicitly addressing perceptual deficiencies."

The real-world impact of this oversight is substantial. In educational technology, where MLLMs are increasingly deployed for STEM tutoring, perception failures manifest as models misreading diagrams, miscounting elements, or confusing spatial relationships—errors that undermine trust regardless of how sophisticated the subsequent reasoning becomes. In scientific document analysis, where figures encode precise quantitative information, hallucinated visual descriptions propagate directly into downstream analyses. In automated assessment, where MLLMs grade student work, perceptual errors translate to incorrect scoring. The paper's insight that perception rather than reasoning is the primary bottleneck implies that many of these failures cannot be fixed by better reasoning—they require fundamentally better visual perception.

Prior Approaches to STEM Visual Perception and Their Limitations

The paper identifies three categories of prior approaches, each with specific, clearly articulated limitations.

Knowledge distillation from strong MLLMs is unreliable. A seemingly natural approach to enhancing STEM perception would be to leverage advanced MLLMs like GPT-series or Gemini-series models as teachers, generating descriptive captions for training smaller models. The paper argues this path encounters two distinct failure modes:

First, hallucination in teacher models: Even state-of-the-art MLLMs produce factually incorrect descriptions, particularly regarding spatial positioning, quantitative relationships, and element interactions. A model might correctly identify that a diagram contains circles and lines but hallucinate their relative positions, counts, or connectivity—precisely the details that matter most in STEM contexts.

Second, and more fundamentally, many STEM images exhibit what the paper terms descriptive aphasia:

"many STEM images exhibit what we term descriptive aphasia: their complex spatial relationships and precise numerical values cannot be fully or accurately captured by natural language alone. For instance, precisely describing auxiliary line constructions in complex polyhedral geometry remains inherently challenging for natural language."

This is a deeper problem than hallucination. Even if a teacher model were perfectly accurate, natural language itself may be an insufficient medium for encoding certain types of visual information. Auxiliary lines in a three-dimensional geometry diagram, precise angle measurements in a trigonometric figure, exact node counts and connection patterns in a graph theory visualization—these resist complete and unambiguous natural language description regardless of the model's fidelity. This limitation is not about model quality; it is about the representational capacity of natural language as a modality for encoding structured visual information.

Problem-solving accuracy is an inadequate perceptual evaluation proxy. The paper identifies a second critical gap in evaluation methodology. Existing approaches to evaluating STEM visual perception have largely relied on problem-solving accuracy as a proxy metric. Section 2 critiques this specifically:

"A recent work adopt a two-stage evaluation paradigm where image captioning is followed by LLM-based problem solving to isolate perceptual capabilities. However, this approach only measures problem-relevant information extraction rather than comprehensive visual understanding, potentially overlooking critical visual details that are irrelevant to specific questions but essential for complete perception."

This is a subtle but important distinction. A model might correctly solve a specific math problem about a diagram while missing most of the diagram's content—it only needed to extract the one or two features relevant to the particular question asked. A geometry diagram might contain 20 labeled elements, but if the question only references 3 of them, a model that accurately perceives those 3 and hallucinates the other 17 would appear successful under a problem-solving proxy. This creates a systematic blind spot: such evaluation methods reward selective perception rather than comprehensive visual understanding.

The consequence is that the field has lacked a direct measure of perceptual capability, making it impossible to diagnose whether perception improvements translate to genuine gains in visual understanding or merely to better question-answering strategies that compensate for perceptual weaknesses.

Perception enhancement has been systematically underexplored. Perhaps the most revealing gap the paper identifies is simply the absence of work explicitly targeting visual perception. While Section 2 catalogs extensive efforts on reasoning enhancement across three methodological categories, the corresponding section on perception is notably sparse. The few works that do address perception do so indirectly—as a byproduct of general model scaling or architecture improvements—rather than as a targeted, systematic objective. This asymmetry is not accidental; it reflects a field-level assumption that reasoning is the harder problem and perception is largely solved. The paper's scaling analysis directly challenges this assumption.

How This Paper Positions Itself

The paper positions itself at the intersection of two previously disconnected lines of inquiry: (1) the well-developed literature on MLLM reasoning enhancement, and (2) the nascent recognition that perception may be the true bottleneck. Its central empirical contribution—the scaling analysis in Figure 1—is designed to make this diagnostic argument rigorous and actionable.

The scaling analysis as diagnostic tool. Rather than simply claiming perception matters, the paper constructs a controlled experiment that decouples perception from reasoning:

  • Perception scaling: Use increasingly powerful MLLMs (4B, 8B, 32B) to generate image captions, but hold the reasoning model constant. Any performance gain must come from better perception alone.
  • Reasoning scaling: Use increasingly powerful LLMs (4B, 8B, 32B) to solve problems from a fixed caption, but hold the perception model constant. Any performance gain must come from better reasoning alone.

The finding—that scaling perception consistently and substantially outperforms scaling reasoning—is the empirical foundation that justifies the paper's entire subsequent focus on perception enhancement.

Code as the solution to the descriptive aphasia problem. The paper's central methodological innovation is recognizing that executable code offers a representational medium that natural language lacks. Code can specify exact coordinates, precise quantities, spatial transformations, and rendering semantics with no ambiguity—the exact information that natural language struggles to convey and that MLLMs hallucinate when attempting to describe. This is not merely a practical convenience; it is a fundamental insight about representational capacity. The paper argues that requiring MLLMs to generate executable Python code for image reconstruction provides:

  • Verifiable ground truth: If the code executes and produces the correct image, the perception was accurate. If it doesn't, the perception failed. There is no ambiguity.
  • Comprehensive evaluation: Image reconstruction requires complete visual understanding—not just the subset of information relevant to a particular question.
  • Trainable supervision: Image-Caption-Code triplets provide complementary training signals where captions offer semantic understanding and code provides precise structural and quantitative constraints.

The paper thus positions executable code not as a replacement for natural language captions but as a complementary modality that addresses their specific weaknesses—the very weaknesses that make perception evaluation and enhancement so challenging in STEM domains.

Distinguishing from domain-specific code generation work. The paper explicitly distinguishes its approach from existing work on code generation for specific visual domains. Section 2 notes:

"different from domain-specific (UI, Chart, SVG) code generation tasks that primarily target downstream applications, we develop STEM Image-Code pairs serves dual purposes: (1) establishing a rigorous benchmark for evaluating comprehensive visual perception, and (2) enabling the construction of high-quality image-code-caption triplets for perception enhancement."

This is an important distinction. Prior work on design-to-code (Si et al., 2025; Ge et al., 2025), chart-to-code (Yang et al., 2024), and SVG generation (Li et al., 2025; Qiu et al., 2025) treats code generation as the end goal—building practical systems for specific applications. This paper treats code generation as an intermediate representation—a tool for evaluating and enhancing the visual perception that underlies all STEM reasoning. The downstream application is not code generation itself; it is better STEM problem-solving through better perception.

The two-level contribution structure. The paper's contributions operate at two levels that are designed to be mutually reinforcing:

  1. A benchmark that establishes the problem (STEM2Code-Eval): By creating a dataset where visual perception must be demonstrated through executable code generation, the paper provides the first rigorous, verifiable measure of comprehensive STEM visual perception. This benchmark serves to make the "perception bottleneck" argument empirically measurable rather than merely rhetorical.

  2. A training methodology that addresses the problem (Code-Grounded tasks on ICC-1M): Rather than merely diagnosing the perception bottleneck, the paper provides a concrete solution—systematically generate training data (image-code pairs) that teach models precise visual-to-code mappings, and use that code as ground truth to produce hallucination-free captions. This closes the loop from diagnosis to treatment.

This dual structure positions the paper as both analytical (identifying what the problem is) and constructive (providing how to fix it), addressing the gap between recognizing perception as a bottleneck and having practical methods to improve it.

3. Technical Approach

3.1 Reader Orientation

CodePercept is a training framework and data generation pipeline that enhances an MLLM's ability to visually perceive STEM images by teaching it to generate executable Python code that reconstructs those images, and by using that same code as verifiable ground truth to produce hallucination-free natural language captions. The system solves the problem that MLLMs fail at STEM visual reasoning primarily due to inadequate perception (not inadequate reasoning), and that perception is hard to improve because natural language is too ambiguous and error-prone a medium for describing precise spatial, quantitative, and relational visual information. The solution's shape is: (1) build a massive dataset of 1M STEM images paired with executable reconstruction code and verified captions, (2) train models jointly on two code-grounded tasks that make perception precise and verifiable, and (3) evaluate perception directly through code generation rather than indirect problem-solving proxies.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major stages that flow sequentially:

  1. Image-Code Pair Construction (Section 3.1): Three parallel pipelines—Image Reproduction, Image Diversity, and Solid Geometry Synthesis—generate STEM images paired with executable Python (matplotlib) code that renders them. These pairs form the foundational "native image-code pairs" where code serves as verifiable ground truth for visual content.

  2. Code-Grounded Caption Generation (Section 3.2): For each image-code pair, a three-step process generates accurate natural language captions: (a) a MLLM produces a linguistically natural but potentially inaccurate draft caption, (b) an LLM analyzes the ground-truth code (augmented with execution tracer logs) to extract verified visual facts, and (c) a refinement LLM surgically corrects the draft using those verified facts while preserving natural language flow.

  3. STEM Image-to-Code Translation (Section 3.3): For each image-code pair, an "explanatory" version of the code is generated by (a) having a MLLM produce a draft with pedagogical commentary about visual-to-code mappings, then (b) an LLM corrects that draft against the ground-truth code to ensure accuracy while preserving instructional richness.

  4. ICC-1M Dataset Assembly: The outputs of stages 2 and 3 produce image-caption-code triplets:

Dtrain={(x(i),tnew(i),cnew(i))}i=1ND_{\text{train}} = \{ (x^{(i)}, t^{(i)}_{\text{new}}, c^{(i)}_{\text{new}}) \}_{i=1}^{N}

where $x^{(i)}$ is a generated STEM image, $t^{(i)}_{\text{new}}$ is the code-grounded accurate caption, and $c^{(i)}_{\text{new}}$ is the explanatory reproduction code. These three modalities encode the same underlying STEM concept in complementary forms.

  1. Post-Training (Section 3.4): In Stage 1 (CodePercept-S1), the base MLLM (Qwen3-VL) is supervised-fine-tuned jointly on captioning and image-to-code tasks. In Stage 2 (CodePercept-R1), the S1 model undergoes GRPO reinforcement learning exclusively on code generation, using rewards based on code format, executability, semantic equivalence to ground-truth, and rendered-image similarity.

Information flows: public STEM seed images → image-code pair generation → code-grounded caption synthesis + explanatory code generation → ICC-1M triplets → SFT joint training → RL code optimization → enhanced perception model.

3.3 Roadmap for the Deep Dive

  • First, the image-code pair construction pipelines (Section 3.1), since they produce the foundational data that all subsequent stages depend on. Understanding the three generation strategies and the quality control framework is prerequisite to understanding how code serves as ground truth.
  • Second, the code-grounded caption generation mechanism (Section 3.2), because it is the novel method that converts the code ground truth into verified natural language—the core technical contribution for eliminating hallucination.
  • Third, the STEM image-to-code translation mechanism (Section 3.3), because it creates the complementary training target (explanatory code) that teaches visual-to-code mappings, and because it shares the refinement pattern established in Section 3.2.
  • Fourth, the ICC-1M dataset assembly and its scale/properties, because the dataset is the concrete artifact that enables training.
  • Fifth, the two-stage post-training strategy (Section 3.4), because this is where the dataset is used to actually modify model behavior through supervised fine-tuning followed by reinforcement learning.
  • Sixth, the STEM2Code-Eval benchmark construction, because it parallels the data generation pipeline while serving the orthogonal purpose of rigorous evaluation with human-verified ground truth.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a data-centric systems paper whose core idea is that executable Python code can serve as a verifiable perceptual medium for enhancing and evaluating MLLM visual perception in STEM—code precisely specifies what an image contains (coordinates, quantities, spatial relationships, colors, geometric structures) where natural language is ambiguous or hallucination-prone, and code can be executed to verify that perception was correct.

Image-Code Pair Construction

The foundation of the entire CodePercept framework is a dataset of STEM images paired with executable Python code that renders them. The paper constructs this dataset through three complementary pipelines, each addressing a different limitation that a single generation strategy would face. The input to the system as a whole is a seed dataset $X$ of STEM-focused public training images; the output is a collection of validated image-code pairs $(x, c)$ where $c$ is executable Python code and $x$ is the image produced by executing $c$.

The overall data generation objective is formulated as:

{(x,c)∣c∈FIR(X)∪FID(X)∪FSG(Tgeo),x∼R(c)}\{(x, c) \mid c \in F_{\text{IR}}(X) \cup F_{\text{ID}}(X) \cup F_{\text{SG}}(T_{\text{geo}}), x \sim R(c)\}

where $F_{\text{IR}}$ is the image reproduction pipeline, $F_{\text{ID}}$ is the image diversity pipeline, $F_{\text{SG}}$ is the solid geometry synthesis pipeline, $T_{\text{geo}}$ is a collection of solid geometric templates, and $R(c)$ denotes executing the code to render image $x$. The notation $x \sim R(c)$ means that the image is deterministically produced by running the code through a Python interpreter.

What it computes: This is not an equation to be solved but a set comprehension specifying the data generation strategy. For every code $c$ produced by any of the three pipelines, the corresponding image $x$ is obtained by executing that code. Unlike typical dataset construction where images come first and captions/labels are annotated afterward, here the code is the source of truth and the image is the consequence—the code specifies what should appear, and execution verifies that it does.

Why this form: By making code the primary artifact and rendering the output, the system guarantees perfect image-code consistency by construction: any image $x$ produced by executing $c$ is by definition exactly what $c$ specifies. This is the fundamental advantage over approaches that start from images and then attempt to describe them—those approaches necessarily introduce a gap between description and reality, whereas here the description (code) causes the reality (image). The framework relies on MLLMs to produce the codes initially, so errors can still occur (MLLM-generated code may not reflect the intended STEM concept), but the image-code pair is internally consistent.

Image Reproduction Pipeline ($F_{\text{IR}}$): For each seed image $I \in X$, this pipeline generates reproduction code through a two-stage decomposition:

c=Gcode(I,Gcaption(I))c = G_{\text{code}}(I, G_{\text{caption}}(I))

where $G_{\text{caption}}$ prompts a MLLM to extract a rich textual description from the STEM image, and $G_{\text{code}}$ prompts another MLLM (or the same one) to produce executable Python code conditioned on both the original image and the generated caption.

In operational terms, what happens is: (1) An MLLM receives the STEM image and is asked to produce a detailed caption describing all visual elements, their positions, colors, spatial relationships, and quantitative attributes. (2) A second MLLM (or the same one in a separate call) receives both the original image and that caption, and is asked to generate matplotlib Python code that reproduces the image.

The rationale for the two-stage decomposition—rather than directly prompting "generate code that reproduces this image"—is that forcing the model to first articulate what it sees in natural language creates an explicit intermediate representation that improves code accuracy. The caption serves as a structured specification that the code generation step can reference. The paper notes: "This allows the prompted MLLM to first understand visual content explicitly before generate code, improving both accuracy and interpretability."

The key limitation of this pipeline is that it is "inherently constrained by the diversity of source images in existing datasets." The reproduction strategy can only produce code for images that already exist in the seed set—it does not create genuinely novel STEM visualizations.

Image Diversity Pipeline ($F_{\text{ID}}$): This pipeline addresses the diversity limitation by abstracting STEM principles from seed images and re-instantiating them in new contexts. The core insight is:

"principles underlying STEM images can be systematically abstracted and re-instantiated across different contexts while preserving conceptual validity."

For each seed image $I \in X$, the pipeline generates $K$ diverse code variations through a compositional abstraction-instantiation process:

[c1,c2,…,cK]=Gcode(I,Gprinciple(I))[c_1, c_2, \ldots, c_K] = G_{\text{code}}(I, G_{\text{principle}}(I))

where $G_{\text{principle}}$ prompts a MLLM to extract the underlying scientific or mathematical principle from $I$ (e.g., "domino-based logic puzzles," "graph connectivity on a grid," "Fibonacci spiral pattern generation"), and $G_{\text{code}}$ prompts the MLLM to generate $K$ diverse code variations instantiating that principle in different visual contexts.

The paper provides a concrete example: "from a seed image depicting domino-based logic puzzles, our pipeline generates circular domino wheel pattern, triangular domino combination arrangement, ladybug spot matrix, grid connection graph, etc., each maintaining STEM rigor while introducing structural novelty." This illustrates the abstraction-reinstantiation pattern: the underlying principle (arranging marked tiles according to logical constraints) is preserved, but the visual manifestation (circular, triangular, grid-based, biological metaphor) varies substantially.

Solid Geometry Synthesis Pipeline ($F_{\text{SG}}$): Both the reproduction and diversity pipelines face a fundamental limitation with solid geometry: "generating code for solid geometry... requires a precise configuration of spatial relationships—a capability where current LLMs and MLLMs are notably deficient." Three-dimensional spatial reasoning in code—computing the correct coordinates for vertices of a cube under rotation, determining which edges are visible from a particular viewing angle, constructing proper orthographic projections—exceeds the reliable capability of current models.

To address this, the pipeline uses parametric code templates rather than free-form generation. The paper defines a collection of code templates $T_{\text{geo}} = \{\tilde{c}_i\}_{i=1}^{M}$, where each template $\tilde{c}_i$ defines the generation logic for a specific geometric type and is parameterized by a parameter space $\Theta_i$. The synthesis is formulated as:

Cgeo={ci∣ci=c~i(θ);i∈1,…,M;θ∈Θi}C_{\text{geo}} = \{c_i \mid c_i = \tilde{c}_i(\theta); i \in 1, \ldots, M; \theta \in \Theta_i\}

where each template $\tilde{c}_i$ is instantiated with parameters $\theta$ sampled from its parameter space $\Theta_i$ to produce specific code $c_i$.

In operational terms: human experts write parameterized Python code for common solid geometry scenarios—cube net unfolding and folding sequences, orthographic three-view projection and reconstruction, cross-sectional analysis of solids, cube stacking configurations, combinations of various geometries, polyhedral constructions, spatial curve visualization, and surface integral representations. Each template accepts parameters that control attributes like cube arrangements, viewing angles, spatial configurations, colors, and labels. The system generates diverse instances by sampling parameter values and executing the template code.

This guarantees geometric correctness (because the templates are hand-crafted by experts) while achieving visual diversity (because the parameter spaces allow large variations in appearance). The tradeoff is that the diversity is constrained by the template's parameterization—the system cannot generate a completely novel geometry type that no template covers.

Unified Quality Control: Even with carefully designed generation pipelines, not all produced image-code pairs will meet quality standards. The paper implements a composite filtering strategy:

D={(x,c)∣(x,c)∈QI(X)∧QC(C)∧QIC(X,C)}D = \{(x, c) \mid (x, c) \in Q_I(X) \land Q_C(C) \land Q_{IC}(X, C)\}

where $Q_I$ filters for image quality (checking for rendering artifacts, visual clarity, appropriate complexity), $Q_C$ filters for code quality (checking for meaningful abstractions vs. arbitrary procedures, appropriate dependencies, well-structured logic), and $Q_{IC}$ filters for image-code consistency (verifying that the rendered image matches what the code intends to produce). All three quality metrics "are evaluated using a SOTA MLLM with specialized prompts."

The paper provides the exact prompts used for each quality check in Appendix A. The code quality prompt ($Q_C$) distinguishes between "high-value abstraction" (code implementing mathematical concepts or algorithms with recognizable structural features) and "low-value procedure" (code with arbitrary, context-free computational steps). The image quality prompt ($Q_I$) classifies rendering errors into fatal technical failures (vector path catastrophes, geometric annotation errors, empty charts, indiscernible density, layout anomalies) that cause rejection, versus non-fatal design flaws (poor color choices, normal occlusion) that are tolerated. The consistency prompt ($Q_{IC}$) compares visual evidence from the rendered image against the design blueprint implied by the code, checking for structural mismatches, categorical differences, severe element mismatches, structural collapse, or data-level misalignment.

Why these three pipelines, not one: Each pipeline addresses a distinct failure mode of the others. Image Reproduction preserves fidelity to real STEM images but is diversity-limited. Image Diversity creates novel content but depends on the MLLM's ability to generate valid instantiations of abstracted principles. Solid Geometry Synthesis guarantees geometric correctness for a specific, challenging domain where MLLMs fail, but is template-limited. Together, they produce a dataset that is simultaneously diverse (spanning visual styles and STEM concepts), accurate (through quality control and template-based correctness), and scalable (to 1M pairs).

Code-Grounded Caption Generation

Given a quality-controlled image-code pair $(x, c)$, this mechanism produces a natural language caption $t_{\text{new}}$ that is both linguistically fluent and factually accurate. The core challenge it addresses is that direct vision-to-text generation for complex STEM images produces hallucinations—incorrect numerical values, misdescribed spatial relationships, wrong element counts. The solution is to use the executable code $c$ as an intermediate source of verified visual facts, which are then inserted into a linguistically natural caption framework.

The generation process follows a three-stage pipeline:

tnew=Grefine(Gcaption(x), Ganalyze(c,ξ(c)))t_{\text{new}} = G_{\text{refine}} \big( G_{\text{caption}}(x), \, G_{\text{analyze}}(c, \xi(c)) \big)

where $G_{\text{caption}}$ generates an initial linguistically natural but potentially inaccurate draft caption $t_{\text{draft}}$, $G_{\text{analyze}}$ extracts verified visual facts from both the code structure $c$ and its execution logs $\xi(c)$, and $G_{\text{refine}}$ synthesizes the final caption by surgically correcting the draft with the verified facts.

What it computes: The equation specifies a function composition that takes an image $x$ and its associated code $c$ as input, passes them through three stages (caption draft, code analysis, refinement), and produces a verified caption $t_{\text{new}}$ as output. Each stage is implemented as a prompt to an LLM or MLLM; the arrows represent data flow between these prompted calls.

Why this form: The three-stage decomposition separates concerns that would conflict if combined. Direct MLLM captioning (stage 1 alone) produces natural language but hallucinates. Direct code analysis (stage 2 alone) produces accurate facts but in a dry, structured format unsuitable for training. The refinement stage (stage 3) marries the two, preserving natural language fluency while correcting factual errors. This is analogous to a retrieve-and-edit pattern: the code analysis "retrieves" ground-truth facts, and the refinement "edits" the draft caption to incorporate them.

Stage 1—Native Caption: A MLLM is prompted to describe the image $x$ directly, producing $t_{\text{draft}} = G_{\text{caption}}(x)$. The paper states this draft "exhibits natural language flow but captures factual inaccuracies regarding quantitative details, spatial relationships, and STEM properties." The exact prompt used is specified in Appendix C and includes instructions to describe "overall structure, coordinate system, exact counts, and detailed layout." The draft serves as the linguistic skeleton into which verified facts will be inserted.

Stage 2—Code Analysis: This is the heart of the code-grounded approach. The goal is to extract from the code $c$ a complete, verified description of what the image contains. However, the paper identifies a critical challenge: "directly analyzing complex code with deep recursion, nested loops, or intricate transformations remains prohibitively difficult for LLMs." If a code generates 100 hexagons in a spiral pattern using a nested loop, asking an LLM to "read the code and tell me what the image looks like" produces unreliable results—the LLM may miscount iterations, miscompute coordinate transformations, or simply fail to understand the algorithmic logic.

To address this, the paper introduces execution-based verification through a tracer mechanism $\xi(c)$:

tcode=Ganalyze(c,ξ(c))=LLM(c,ξ(c),Panalyze)t_{\text{code}} = G_{\text{analyze}}(c, \xi(c)) = \text{LLM}(c, \xi(c), P_{\text{analyze}})

The execution tracer $\xi(c)$ operates by executing the code in a controlled environment and systematically recording:

  • Geometric precision: exact coordinates, dimensions, spatial relationships of every rendered element
  • Quantitative attributes: definitive counts of elements (number of circles, lines, text labels), RGB color specifications, line widths
  • Rendering semantics: z-order layering (which elements appear on top of which), transformation matrices, viewing parameters
  • STEM mappings: relationships between code parameters and their visual manifestations

The tracer "functions as an external instruction manual of the code" that "captures a structured checklist of logs about all visual elements rendered during code execution." When the LLM analyzes the code, it receives both the source code text and the tracer's execution logs, and it is prompted (via $P_{\text{analyze}}$) to extract visual facts from this combined information.

The key insight is that the tracer converts the procedural knowledge in the code (loops, transformations, conditional rendering) into declarative knowledge (this specific element exists at these specific coordinates with these specific properties). This is the knowledge format that can then be used to correct hallucinated captions. Without the tracer, the LLM would need to mentally execute the code to determine what it renders—a task at which LLMs are unreliable. With the tracer, the LLM can simply read the ground truth about what was actually rendered.

Stage 3—Code-Grounded Caption Refinement: The final stage produces the verified caption:

tnew=Grefine(tdraft,tcode)=LLM(tdraft,tcode,Prefine)t_{\text{new}} = G_{\text{refine}}(t_{\text{draft}}, t_{\text{code}}) = \text{LLM}(t_{\text{draft}}, t_{\text{code}}, P_{\text{refine}})

where $P_{\text{refine}}$ instructs an LLM to perform "surgical edits that systematically correct factual errors (incorrect numbers, positions, colors, or geometric relationships), replace vague quantifiers with exact counts, and supplement omissions of perceptually salient content." Crucially, the prompt instructs the LLM to "maintain the original syntactic structures, language style, and descriptive flow of $t_{\text{draft}}$ while substituting verified visual facts from $t_{\text{code}}$."

This refinement is not a full regeneration—it is a targeted editing operation. The draft caption provides the linguistic scaffolding (sentence structures, descriptive flow, overall organization); the code analysis provides verified values to plug into that scaffolding. The paper's appendix (Appendix C) provides a concrete comparison showing that while a direct MLLM caption for a complex grid-and-arrow diagram misidentifies node counts, spatial arrangements, and arrow counts, the code-grounded caption corrects all these errors while preserving the caption's organizational structure.

Why this approach over alternatives: There are three natural alternatives, each with clear flaws. (1) Generate captions directly from the image using a strong MLLM: produces hallucinations. (2) Generate captions directly from the code alone: loses natural language fluency and produces dry, technical descriptions unsuitable for training data. (3) Use human annotation: prohibitively expensive at 1M scale. The three-stage approach uniquely combines the strengths of alternatives (natural language from direct captioning, factual accuracy from code analysis) while avoiding their weaknesses.

STEM Image-to-Code Translation

While code-grounded captions provide verified natural language descriptions, the paper also treats code generation itself as a direct training target. The rationale is that code "offers a fundamentally different modality for visual description" that "captures geometric relationships, mathematical constraints, and structural details that natural language descriptions often leave ambiguous or incomplete." However, simply providing ground-truth code $c$ as a training target is insufficient because "models need explicit guidance to learn visual-to-code mappings: mapping observed features to code segments and understanding how parameters govern visual properties."

The paper therefore constructs explanatory image-code pairs $c_{\text{new}}$ that augment ground-truth code with pedagogical commentary. The construction follows a two-stage refinement pattern parallel to the caption generation:

cnew=Grefine(Gcode(x),c)c_{\text{new}} = G_{\text{refine}}(G_{\text{code}}(x), c)

where $G_{\text{code}}(x)$ prompts a MLLM to generate code directly from the image, producing an "explanatory draft" $c_{\text{draft}}$, and $G_{\text{refine}}$ corrects that draft against the ground-truth code $c$.

Stage 1—Explanatory Draft Generation: A MLLM is prompted with the image $x$ and asked to produce Python code that reproduces it, along with step-by-step explanations of the implementation rationale. The paper notes that this draft "naturally exhibits desirable pedagogical patterns including step-by-step breakdown, explicit parameter choices, and instructional commentary explaining visual-to-code mappings." These are precisely the patterns that help a model learn why certain code structures correspond to certain visual features—the draft explains, for example, "we use np.linspace(0, 2*np.pi, 100) to generate 100 evenly spaced points around the circle, with radius controlled by the r parameter."

However, "without access to ground-truth specifications, $c_{\text{draft}}$ frequently contains factual errors in coordinates, dimensions, loop logic, or algorithmic structure, particularly for complex STEM images." The explanatory quality is high, but the code correctness is not guaranteed.

Stage 2—Code-Grounded Refinement: The draft is corrected against the verified ground-truth code:

cnew=Grefine(cdraft,c)=LLM(cdraft,c,Prefine)c_{\text{new}} = G_{\text{refine}}(c_{\text{draft}}, c) = \text{LLM}(c_{\text{draft}}, c, P_{\text{refine}})

where $P_{\text{refine}}$ instructs an LLM to "adaptively refine the explanatory content and replace erroneous code with $c$, while carefully maintaining the original explanatory structure." The refinement preserves "the logical flow, step-by-step explanations, and contextual descriptions of $c_{\text{draft}}$ while ensuring code correctness through alignment with $c$."

Appendix B provides a detailed example showing the difference between "native image-code pairs" (bare code without explanation) and "explanatory image-code pairs" (code augmented with an "Implementation Breakdown" section that walks through hexagon grid construction, path definition, arrow drawing, and label placement, explaining parameter choices and coordinate computations). The explanatory version transforms what would be opaque coordinate arrays into reasoned design decisions that the model can learn from.

Why explanatory code, not just ground-truth code: If the training target were simply the executable code $c$, the model would learn to map images to code but without understanding the visual-to-code correspondence. It might memorize patterns without grasping why a particular coordinate computation positions an element correctly. The explanatory commentary makes the mapping explicit: "the central hexagon is placed at the origin (0, 0), and the six surrounding hexagons are arranged in a standard hexagonal grid pattern around it" directly connects visual layout to mathematical specification. This bridges the gap between perception (seeing the hexagon arrangement) and code generation (computing the coordinates).

Relationship between caption and code tasks: The two code-grounded tasks are designed to be complementary and mutually reinforcing. Captions provide semantic context for code generation—understanding that an image depicts a "hexagonal grid with a spiraling path" helps guide what kind of code to write. Code provides structured precision that overcomes natural language's descriptive limitations—specifying exact coordinates and transformations where language would be ambiguous. The paper's ablation (Table 3, Group 3) shows that combining both tasks (CodeCap + ImCode) yields better performance than either alone (63.32% average vs. 62.75% for CodeCap alone), validating this complementarity claim.

ICC-1M Dataset Assembly

The outputs of the three data construction pipelines, filtered through unified quality control, and processed through the two code-grounded tasks, are assembled into the ICC-1M training dataset:

Dtrain={(x(i),tnew(i),cnew(i))}i=1ND_{\text{train}} = \{(x^{(i)}, t^{(i)}_{\text{new}}, c^{(i)}_{\text{new}})\}_{i=1}^{N}

where the dataset contains over 1 million high-quality STEM Image-Caption-Code triplets. Each triplet $(x, t, c)$ represents the same underlying STEM concept in three complementary modalities:

  • $x$: The rendered image (visual perception target)
  • $t_{\text{new}}$: The code-grounded accurate caption (semantic understanding in natural language)
  • $c_{\text{new}}$: The explanatory reproduction code (structured, precise, verifiable representation)

What this triplet structure enables: During training, the model sees the same concept expressed in three ways simultaneously. The caption teaches what the image means semantically (e.g., "this is a hexagonal grid"). The code teaches what the image is structurally (e.g., "there are exactly 7 hexagons at these specific coordinates with these specific colors"). The image provides the perceptual input that must be mapped to both representations. Training jointly on captioning (image → text) and code generation (image → code) forces the model to develop visual representations that support both semantic understanding and precise structural specification.

Scale considerations: At 1M triplets, ICC-1M is substantially larger than typical manually-annotated STEM datasets. The paper achieves this scale through its automated pipeline: the three code generation pipelines produce candidate pairs, the quality control filters them, and the code-grounded tasks produce the caption and explanatory code. Human annotation is limited to the STEM2Code-Eval benchmark (1,000 examples with pixel-perfect verification in Appendix E) and the solid geometry templates (expert-written but parameterized for automatic diversification). This scalability is essential because training robust visual perception likely requires exposure to diverse STEM visualizations—a model that only sees a few hundred geometry diagrams cannot learn general visual-to-code mapping principles.

Quality control across the pipeline: The paper implements quality checks at three stages: code quality (meaningful abstractions vs. arbitrary procedures), image quality (absence of fatal rendering artifacts), and image-code consistency (rendered image matches code intent). The exact prompts for each check are provided in Appendix A and represent a significant engineering contribution—they encode detailed criteria for what makes a STEM visualization "good" in a form that an MLLM can evaluate automatically.

Post-Training Strategy

The training strategy applies the ICC-1M dataset in two sequential stages: supervised fine-tuning (SFT) followed by reinforcement learning (RL). The paper uses Qwen3-VL as the base architecture across three model sizes (4B, 8B, 32B parameters).

Stage 1: Supervised Fine-Tuning (CodePercept-S1): The model is fine-tuned jointly on two tasks using the ICC-1M triplets:

  • Image captioning: Given image $x^{(i)}$, generate caption $t^{(i)}_{\text{new}}$. This is a standard vision-language task; the novelty is in the caption quality (code-grounded, hallucination-free) rather than the task format.
  • Image-to-code translation: Given image $x^{(i)}$, generate explanatory code $c^{(i)}_{\text{new}}$. This is a novel task that teaches visual-to-code mapping.

The training uses Qwen3-VL-Instruct as the base model. Hyperparameters: 1 epoch, AdamW optimizer with DeepSpeed ZeRO-2 optimization, cosine learning rate schedule from $3 \times 10^{-6}$ decaying to $2 \times 10^{-7}$ with 5% warm-up steps, effective batch size of 4 per device (2 samples per device with gradient accumulation of 2 steps), bfloat16 mixed precision, Flash Attention. Training is conducted on 32 A100 GPUs using the SWIFT framework.

Why joint training: The paper argues that training on both tasks simultaneously provides benefits beyond training each separately. "Natural language captions help the model understand visual content semantically, building strong visual understanding before generating code. Simultaneously, the explanatory code acts as a structured caption that encodes visual information through executable programming constructs, providing precise spatial and quantitative details that complement natural language descriptions." In effect, the captioning task provides the "what" (semantic category, overall structure) and the code task provides the "exactly where and how many" (precise coordinates, exact counts). The model learns to produce both from the same visual input, developing representations that support both types of understanding.

Stage 2: Reinforcement Learning (CodePercept-R1): The paper strategically applies reinforcement learning "exclusively to code generation, as creating accurate, executable code is more challenging than natural language." The justification is that "code requires strict syntax and logic, where minor errors can cause failures," and "it provides inherently verifiable reward signals through executability and similarity metrics, making it well-suited for RL optimization."

The RL algorithm is Group Relative Policy Optimization (GRPO), building on prior work (Guo et al., 2025; Shao et al., 2024). The model is initialized from the CodePercept-S1 checkpoint and fine-tuned on a subset of 10,000 high-quality samples from ICC-1M, selected via difficulty filtering: using Qwen3-VL-7B under 8 rollout iterations, samples are retained whose accuracy falls between 0.25 and 0.75, following the DAPO filtering strategy (Yu et al., 2025). This filters out both trivially easy samples (where the model already succeeds and learning would be minimal) and impossibly hard samples (where even the ground-truth may be unreliable).

Reward design: The composite reward function combines two components:

  • Format Reward ($r_{\text{fmt}}$): A binary reward (1.0 or 0.0) assigned by regular expression matching that validates the generated code is encapsulated within a ```python ``` block. This encourages proper output formatting.

  • Content Reward ($r_{\text{cnt}}$): A sum of three sub-rewards:

    • Execution reward ($r_{\text{exec}}$): 1 if the code executes successfully (exit code 0, produces an image file), 0 otherwise.
    • Code-level Reward ($r_{\text{code}}$): A formatted score from GPT-4o that assesses semantic equivalence between the generated code and the ground-truth code $c$. This measures whether the code does the right thing, even if implemented differently.
    • Image-level Reward ($r_{\text{image}}$): A formatted score from GPT-4o that evaluates visual similarity between the original image $x$ and the rendered output. This is only available when execution succeeds; it is 0 otherwise.

The overall reward is: $r = r_{\text{fmt}} + r_{\text{cnt}}$

GRPO optimization objective: For a group of $G = 8$ outputs $\{o_1, \ldots, o_G\}$ sampled from the same query $q$ (which includes both the question and image from $D_{\text{train}}$), GRPO maximizes:

J(θ)=E{oi}i=1G∼πθold(O∣q)[1G∑i=1Gmin⁡(πθ(oi∣q)πθold(oi∣q)Ai,clip(πθ(oi∣q)πθold(oi∣q),1−ϵ,1+ϵ)Ai)−βDKL(πθ∥πref)]J(\theta) = \mathbb{E}_{\{o_i\}_{i=1}^G \sim \pi_{\theta_{\text{old}}}(O|q)} \left[ \frac{1}{G} \sum_{i=1}^{G} \min\left( \frac{\pi_\theta(o_i|q)}{\pi_{\theta_{\text{old}}}(o_i|q)} A_i, \text{clip}\left(\frac{\pi_\theta(o_i|q)}{\pi_{\theta_{\text{old}}}(o_i|q)}, 1 - \epsilon, 1 + \epsilon\right) A_i \right) - \beta D_{\text{KL}}(\pi_\theta \| \pi_{\text{ref}}) \right]

where $\pi_\theta$ is the policy being optimized, $\pi_{\theta_{\text{old}}}$ is the policy from the previous iteration, $\pi_{\text{ref}}$ is a reference policy (the S1 checkpoint) for KL regularization, $\epsilon$ is the PPO clipping hyperparameter, and $\beta$ controls the KL penalty strength.

What it computes: This is the standard GRPO objective. For each query $q$, the old policy samples $G$ outputs. For each output, an advantage $A_i$ is computed as the normalized reward. The objective then maximizes the probability of high-advantage outputs relative to the old policy, clipped to prevent too-large updates, minus a KL penalty to prevent drifting too far from the reference policy.

Advantage computation: The advantage is group-normalized:

Ai=ri−mean(r1,r2,…,rG)std(r1,r2,…,rG)A_i = \frac{r_i - \text{mean}(r_1, r_2, \ldots, r_G)}{\text{std}(r_1, r_2, \ldots, r_G)}

This means the advantage of an output depends on how its reward compares to the average reward of all outputs sampled for the same query. Outputs with above-average rewards get positive advantage; below-average get negative advantage. This relative normalization removes the need for a learned value function and makes the training signal depend on within-group comparisons rather than absolute reward magnitudes.

Why this form: GRPO is chosen over standard PPO because it eliminates the need for a separate value function model (which would double memory requirements). The group-relative advantage provides a natural baseline—if all outputs in a group are good, they all get near-zero advantage (no pressure to change); if some are clearly better than others, the advantage signal drives improvement. The KL penalty prevents the model from diverging too far from the SFT-trained behavior, which is important because the RL stage only trains on code generation (not captioning); without the KL constraint, the model could lose its captioning ability.

RL hyperparameters: Temperature 1.0, top-p sampling 0.85, repetition penalty 1.1, 8 candidate completions per sample, maximum generation length 8,192 tokens, AdamW optimizer with learning rate $1 \times 10^{-6}$, 5% warm-up steps, bfloat16 mixed precision, gradient clipping with maximum norm 0.5, KL penalty coefficient $\beta = 0.001$, 1 epoch on the 10k selected samples.

Why RL only on code, not caption: The paper's reasoning is that code generation is the harder task (strict syntax, executability requirement) and has naturally verifiable rewards (execution success, rendering accuracy), making it well-suited for RL. Caption generation lacks such clean reward signals—evaluating caption quality requires subjective judgment or reference-based metrics that are less reliable. By focusing RL on code and using SFT for both tasks, the paper avoids the difficulty of designing robust rewards for natural language generation while still improving the model's overall perception through enhanced code generation ability.

The convergence of modalities: The two-stage training creates a model where visual perception is grounded in complementary modalities. The SFT stage teaches the basic mappings (image → caption, image → code). The RL stage refines code generation specifically, pushing toward executable, accurate code. Because the model shares visual representations across both tasks (joint training), improvements in code generation capability also improve the visual features that support captioning—the model learns to "see" more precisely because it is rewarded for generating code that exactly reproduces what it sees.

4. Key Insights and Innovations

Innovation 1: Perception as the Diagnosed Bottleneck — A Scaling Analysis That Isolates Cause, Not Just Correlation

The paper's most consequential contribution is not proposing a new method but making a diagnostic claim with rigorous empirical support: that visual perception, not reasoning, is the primary bottleneck limiting MLLMs in STEM. This matters because the field has been implicitly operating under the opposite assumption—investing massive effort in reasoning enhancement (cold-start thinking data, RL-based optimization, unimodal reasoning transfer) while treating perception as a largely solved problem.

What makes this diagnostic move distinctive is its experimental design, not just its conclusion. The paper constructs a scaling analysis that decouples two confounded variables that prior work had never systematically separated. Before this paper, when a larger MLLM outperformed a smaller one on MathVision, it was impossible to know whether the gain came from better seeing or better thinking—model scale improves both simultaneously. The paper breaks this coupling by fixing one component while scaling the other:

  • To isolate perception: use MLLMs of increasing size (4B → 8B → 32B) to generate image captions, but feed all captions to the same reasoning LLM. Any performance difference must be purely perceptual.
  • To isolate reasoning: use the same MLLM to generate a fixed caption, but feed it to LLMs of increasing size. Any performance difference must be purely reasoning-based.

The result—that scaling perception consistently and substantially outperforms scaling reasoning (Figure 1)—is not just a metric observation. It is a falsification of the field's implicit default assumption that reasoning is the harder problem. Prior work on reasoning enhancement (R1-OneVision, Perception-R1, Ursa, OVR) implicitly assumes that better reasoning mechanisms will unlock STEM performance. The scaling analysis shows that for current models on current benchmarks, the returns to better perception dominate the returns to better reasoning. This is a negative result about the reasoning-first research agenda—it explains why massive reasoning investments have produced diminishing returns: they are optimizing a downstream component fed by a noisy, unreliable perceptual frontend.

The paper emphasizes this diagnostic value in Section 1: "when models fail, is it due to perception deficiencies or reasoning limitations?" The scaling analysis provides the first systematic answer: for current MLLMs on representative STEM benchmarks, perception is the binding constraint. This reframes the research agenda from "how do we make models reason better about what they see?" to "how do we make models see better in the first place?"—a shift in priorities with significant resource allocation implications.

Comparison to prior diagnostic approaches: Prior work on MLLM failure analysis (e.g., Lu et al., 2025's OmniCaptioner) used a two-stage captioner-solver paradigm but evaluated only problem-solving accuracy, which conflates selective perception (extracting question-relevant features) with comprehensive visual understanding. The paper's innovation is not the captioner-solver setup per se, but the controlled scaling design that varies one factor independently while holding the other constant, enabling causal attribution rather than mere correlation.

Significance beyond performance: This is a paradigm-level contribution rather than a method-level contribution. It changes what problem the field thinks it is solving. The experimental design is conceptually transferable: any multimodal task where perception and reasoning are confounded (medical image diagnosis, diagram understanding, chart interpretation) could apply the same decoupling methodology to identify the true bottleneck before investing in solutions.

Evidence: Figure 1 shows the core result with two model scales (4B and 8B perception baselines) across increasing reasoning model sizes. The perception-scaling curves (blue) consistently lie above the reasoning-scaling curves (red), with the gap widening at larger scales. The paper reports this finding on MathVision, a representative STEM visual reasoning benchmark. The captioner-solver results in Table 1 provide converging evidence: when CodePercept improves perception (without changing reasoning), performance improves by 2.8–3.0% on average across six benchmarks, demonstrating that perception improvements are actionable.

Innovation 2: Code as a Verifiable Perceptual Medium — Solving the Descriptive Aphasia Problem

The paper's central conceptual move is recognizing that executable code is a superior representational medium for visual perception in STEM because it overcomes a fundamental limitation of natural language: what the paper terms "descriptive aphasia." This is not merely a practical engineering choice but a conceptual insight about representational capacity.

What descriptive aphasia means concretely: Many STEM images contain visual information that natural language cannot fully encode regardless of the describer's fidelity. Precise spatial coordinates, exact element counts in complex arrangements, transformation parameters in geometric constructions, z-order layering, specific RGB values—these are inherently quantitative and structural properties that natural language can only approximate. A caption can say "approximately 200 dots arranged in a roughly hexagonal pattern," but this approximation is insufficient for rigorous visual perception—the exact counts and exact positions matter for downstream STEM reasoning.

Code does not suffer from this limitation. The statement hexagon_centers = [(0, 0), (1.5, 0.866), ...] specifies coordinates with floating-point precision. The statement for i in range(200): draw_dot(positions[i]) specifies an exact count. The statement ax.add_patch(Polygon(vertices, facecolor='steelblue', zorder=2)) specifies exact color and layering. There is zero ambiguity.

Why this is a fundamental insight, not just a convenience: The paper's argument is that prior approaches to perception enhancement implicitly accepted natural language as the target modality for visual description—whether through captioning, visual question answering, or knowledge distillation from teacher models. But if natural language is structurally incapable of fully capturing certain types of visual information, then no amount of model scaling or training data quality improvement within the natural-language paradigm can solve the perception problem completely. The move to code as a perceptual medium is therefore not an incremental improvement—it is a paradigm shift in what it means for a model to "perceive" a STEM image.

The paper makes this explicit in its critique of knowledge distillation: "these teacher models are prone to hallucination, producing factually incorrect descriptions, particularly regarding spatial positioning, quantitative relationships, and element interactions." But the deeper point is that even a perfect teacher model using natural language would face the descriptive aphasia problem—there is no natural language sentence that can encode the exact coordinates of 200 dots with the same precision as a Python list. The hallucination problem is a symptom; the descriptive aphasia problem is the underlying disease.

Comparison to prior code generation work: Domain-specific code generation tasks (design-to-code, chart-to-code, SVG generation) treat code as the output for a downstream application. This paper treats code as an intermediate representation for perception itself—the goal is not to produce useful code but to force the model to develop precise visual understanding. This reframes code generation from an application task to an epistemic tool: generating reconstruction code is how the model demonstrates (and, through training, develops) comprehensive visual perception.

Connection to the Tracer mechanism: The paper's execution tracer ξ(c) in Section 3.2 is the mechanism that makes this insight operational. By executing the code and recording what is actually rendered, the tracer converts procedural code knowledge into declarative visual facts. This bridges the gap between "the code says for i in range(200)" (which only tells you there's a loop) and "the image contains exactly 200 dots at these specific coordinates" (which is ground-truth visual information). The tracer is the enabler of code-as-perceptual-medium, and the paper's decision to augment LLM-based code analysis with execution-based verification is a design choice directly motivated by the recognition that LLMs cannot reliably mentally execute complex code.

Evidence: The ablation in Table 3, Group 2 directly tests this claim. Comparing NativeCap (captions generated directly from images by Gemini2.5-Pro without code access) against CodeCap (code-grounded captions using the pipeline of Section 3.2), CodeCap achieves 62.75% average accuracy vs. NativeCap's 60.78%—a 2.0 percentage point gain attributable purely to using code as intermediate ground truth. This demonstrates that even a state-of-the-art MLLM (Gemini2.5-Pro) produces captions that are measurably worse than those verified against executable code. The STEM2Code-Eval results in Table 2 provide converging evidence: models that learn image-to-code mappings show substantially higher reconstruction fidelity than those trained only on natural language tasks.

Innovation 3: Comprehensive Perception Evaluation via Image Reconstruction — Beyond Problem-Solving Proxies

The paper introduces a fundamentally new evaluation paradigm for visual perception: requiring models to generate executable code that reconstructs the original image provides a deterministic, verifiable, and comprehensive assessment of visual understanding. This breaks from the dominant evaluation paradigm in two critical ways.

The problem with problem-solving proxies: Existing STEM benchmarks (MathVision, MathVista, MathVerse, etc.) evaluate perception indirectly through end-task accuracy. A model might score highly by extracting only the 2–3 visual features relevant to a specific question while failing to perceive most of the image's content. The paper articulates this in Section 4: such metrics "only reflect the model's capacity to understand problem-relevant information rather than comprehensive visual perception." A geometry diagram with 20 labeled elements where the question only asks about 3 creates a systematic blind spot—models can appear to have good perception when they actually have highly selective, incomplete visual understanding.

The two-stage captioner-solver setup (used by OmniCaptioner and adopted in the paper's own Table 1) partially addresses this by evaluating caption quality, but it inherits the natural language limitation: a caption that is "good enough for the solver to answer correctly" may still omit most of the visual content. The paper's key insight is that image reconstruction is the only evaluation that demands complete perception—missing any visual element will cause the reconstructed image to differ from the original, making the failure detectable.

Why code-based reconstruction is a stronger signal than similarity metrics: The paper could have evaluated perception by computing pixel-level similarity (MSE, SSIM) between original and model-generated images. But the choice of code generation as the evaluation medium provides additional diagnostic value: if the model's code fails to execute, the failure mode is immediately interpretable (syntax error, missing import, incorrect API usage). If it executes but produces the wrong image, the code can be inspected to understand which visual features were misperceived (wrong coordinates, wrong counts, wrong colors). This interpretability is absent from end-to-end similarity metrics, which provide a single number with no diagnostic decomposition.

Comparison to prior evaluation approaches: Prior work on chart-to-code (ChartMimic), design-to-code (DesignBench, Design2Code), and SVG generation (OmniSVG) evaluates code generation quality but treats it as the end goal—how well can the model produce useful code for downstream applications? STEM2Code-Eval repurposes code generation as an evaluation instrument: the goal is not useful code but demonstrated perception. The rigorous construction pipeline (Section 4, Appendix E)—involving Gemini2.5-Pro captioning, Claude-Opus 4.1 code generation, iterative refinement with visual feedback scoring, candidate selection by similarity and difficulty, and human annotator pixel-perfect verification—ensures that the benchmark's ground-truth code genuinely reconstructs the original image with high fidelity, making it a reliable perception probe.

The benchmark as a diagnostic tool, not just a leaderboard: STEM2Code-Eval's three metrics (Image Score, Code Score, Exec Rate) provide a multi-faceted diagnostic profile of a model's perception. A model with high Exec Rate but low Image Score can generate executable code but fails to capture visual content—indicating good code syntax learning but poor visual perception. A model with high Code Score but low Image Score might produce well-structured code that doesn't match the image—indicating good programming skills that are not grounded in visual input. This decomposition enables fine-grained failure analysis that single-metric benchmarks cannot provide.

Evidence: Table 2 demonstrates that the benchmark captures perception capability beyond what problem-solving proxies show. Qwen3-VL-4B-Instruct achieves only 43.46 average on STEM2Code-Eval (vs. 59.29% on the captioner-solver evaluation in Table 1), revealing a large gap between "perception sufficient to answer questions" and "comprehensive visual understanding." CodePercept-4B-S1 closes this gap to 54.09 (+10.63), and CodePercept-4B-R1 to 61.44 (+17.98), showing that the training methodology produces genuine perceptual improvements rather than mere question-answering strategies. Notably, several models with strong captioner-solver performance (InternVL3.5-8B: 55.88% in Table 1) perform poorly on STEM2Code-Eval (29.38 average in Table 2), confirming that problem-solving accuracy masks substantial perceptual deficiencies.

Innovation 4: The Image-Caption-Code Triplet as a Unified Training Representation

The paper's training methodology synthesizes a novel data representation—the image-caption-code triplet where all three modalities encode the same underlying STEM concept in complementary forms—that enables joint learning of semantic understanding and precise structural perception. This is not merely a data augmentation strategy but a representational innovation about what training signals a perception model should receive.

Why triplets, not pairs: Prior vision-language training uses image-caption pairs (the caption provides semantic supervision) or image-code pairs (the code provides structural supervision). The paper's insight is that these two supervision signals are complementary and mutually reinforcing in ways that neither alone can provide. Captions teach semantic categorization ("this is a hexagonal grid") and high-level descriptive patterns. Code teaches precise spatial and quantitative specification ("there are exactly 7 hexagons with centers at these coordinates"). Training on both simultaneously means the model cannot satisfy one objective at the expense of the other—it must develop visual representations that support both types of output.

The paper's ablation in Table 3, Group 3 validates this complementarity: CodeCap alone achieves 62.75%, adding ImCode (the image-to-code task) boosts performance to 63.32%, confirming that code generation provides additional perceptual training signal beyond what captioning alone achieves. The gain is modest (+0.57 percentage points), but the paper's argument is about robustness and representational quality, not just aggregate metrics—the code task specifically improves performance on benchmarks requiring precise spatial and quantitative perception.

Why this is an innovation rather than a feature combination: Prior work on multi-task vision-language training (e.g., InternVL, Qwen-VL series) trains on diverse tasks (captioning, VQA, grounding) but all within the natural language modality. The Innovation here is adding a non-linguistic target modality (executable code) that imposes different constraints on the visual representation—code cannot be approximated or paraphrased the way natural language can. This forces the visual encoder to produce features that support exact specification, reducing the tolerance for the kind of approximate perception that suffices for natural language tasks.

The role of explanatory code: The paper's choice to use explanatory image-code pairs (with pedagogical commentary, Appendix B) rather than bare code as training targets is a subtle but important design decision. Bare code teaches the mapping from image to implementation; explanatory code additionally teaches why specific implementations correspond to specific visual features ("the central hexagon is placed at the origin because..."). This bridges the gap between rote code generation and genuine visual-to-code understanding, and it is a deliberate response to the observation that "models need explicit guidance to learn visual-to-code mappings."

Evidence: The ablation in Table 3, Group 1 shows that scaling the data pipeline (IR → IR+ID → IR+ID+SG) monotonically improves performance (60.91 → 62.15 → 62.75), suggesting that diverse triplet data provides effective training signal. The consistent improvements from S1 to R1 stages in Tables 1 and 2 (e.g., CodePercept-4B: 62.07% → implied improvement, STEM2Code-Eval: 54.09 → 61.44) demonstrate that the triplet representation supports both supervised and reinforcement learning stages, with RL providing additional gains specifically through improved code quality.

Innovation 5: Parametric Template-Based Synthesis for Capability Gaps — Acknowledging and Engineering Around MLLM Limitations

The paper's solid geometry synthesis pipeline (Section 3.1, F_SG) represents a methodological innovation in dataset construction: rather than attempting to make MLLMs generate code for domains where they fundamentally lack capability, the system acknowledges the limitation and engineers around it using expert-written parametric templates.

Why this is a distinctive contribution: Most data generation pipelines for MLLM training take one of two approaches: (1) use strong MLLMs to generate all data (potentially propagating errors in weak domains), or (2) use purely rule-based generation (limiting diversity). The paper's hybrid approach—using MLLMs for domains where they are capable (image reproduction, diversity) and templates for domains where they are not (solid geometry)—represents a principled acknowledgment of model limitations rather than an attempt to paper over them.

This matters because the solid geometry domain is precisely where natural language's descriptive aphasia is most severe: describing the unfolding of a cube into a net, or the cross-section of a pyramid at a specific plane, requires spatial reasoning that exceeds current MLLM capability for code generation. Without the template pipeline, ICC-1M would either lack solid geometry examples entirely or contain low-quality examples that would degrade training. The template approach guarantees geometric correctness (human experts write the templates) while maintaining visual diversity (parameter sampling produces variations).

What the templates cover: The paper lists eight specific geometry types covered by templates: cube net unfolding/folding sequences, orthographic three-view projection/reconstruction, cross-sectional analysis, cube stacking, combined geometries, polyhedral constructions, spatial curve visualization, and surface integral representations. This catalog is comprehensive for typical solid geometry problems in STEM education, suggesting the templates were designed with specific downstream evaluation tasks in mind.

Connection to the broader paradigm: The template approach reinforces the paper's central thesis: code is a more precise perceptual medium than natural language. The templates encode spatial relationships as mathematical specifications that guarantee correctness. An MLLM generating free-form code for a cube unfolding would likely produce geometrically impossible configurations; the templates eliminate this failure mode by construction. This is the same principle that makes code valuable for evaluation (STEM2Code-Eval) and training (ICC-1M)—code's precision is not just a convenience but a necessity for domains where approximate perception is insufficient.

Evidence: The ablation in Table 3, Group 1 shows that adding the solid geometry pipeline (SG-CodeCap) to the image reproduction and diversity pipelines improves average performance from 62.15 to 62.75. This gain, while modest in absolute terms, is notable because it demonstrates that data from a domain where MLLMs cannot reliably generate code (solid geometry) still provides useful training signal for perception. The improvement would not be possible without the template-based approach.

5. Experimental Analysis

Evaluation Methodology

  • Dataset. All experiments use a collection of six publicly available STEM visual reasoning benchmarks: MathVision (Wang et al., 2024), MathVista (Lu et al., 2023), MathVerse (Zhang et al., 2024), DynaMath (Zou et al., 2024), WeMath (Qiao et al., 2024), and LogicVista (Xiao et al., 2024). These span multiple STEM domains (mathematics, physics, chemistry, electrical engineering) and collectively provide broad coverage of visual reasoning tasks. The test splits are used as-is from each benchmark's standard release. For the authors' novel STEM2Code-Eval benchmark, 1,000 manually verified image-code pairs are sourced from the test sets of these same six benchmarks (Section 4).

  • Base model(s). The primary base architecture is Qwen3-VL (Qwen Team, 2025) across three scales: 4B, 8B, and 32B parameters. The Instruct variants serve as the SFT initialization; the Thinking variants appear in the captioner-solver evaluations as baselines and as the perception/reasoning components in the scaling analysis. The choice of Qwen3-VL is motivated by its representative position among contemporary open-source MLLMs—the paper's ablation in Table 1 shows it outperforms comparably-sized InternVL3.5-8B (55.88% vs. 60.36% average) and MiniCPM-V-4.5 (56.55%), providing a strong but not saturated baseline. For the FLOPs-matched comparison (the scaling analysis in Figure 1), the reasoning LLM solvers are Qwen3-30A3-Thinking and Qwen3-235A22-Thinking (Qwen Team, 2025).

  • Metrics. Two evaluation paradigms are employed, each with distinct metrics. Captioner-solver evaluation (Tables 1, 3, 4): The model under test acts as an image captioner, generating a text description; a separate, fixed LLM solver (Qwen3-30A3-Thinking or Qwen3-235A22-Thinking) then answers STEM questions using only that caption. The reported metric is benchmark accuracy (%) — the fraction of questions answered correctly by the solver, aggregated per benchmark and averaged across benchmarks. This metric isolates perceptual quality: more accurate captions enable better reasoning, so higher accuracy indicates better perception. Image reproduce perception evaluation (Tables 2, 5): On STEM2Code-Eval, three complementary metrics are computed. Image Score (0–100): Gemini2.5-Pro evaluates visual similarity between the model's rendered output and the ground-truth image across geometric completeness, positional accuracy, annotation fidelity, visual consistency, and clarity (full scoring rubric in Appendix F.1). Code Score (0–100): GPT-4o evaluates the generated Python code across logical correctness, readability, efficiency, and robustness (full rubric in Appendix F.2). Exec Rate (%): The fraction of generated code samples that execute without runtime errors and produce a valid image file within a 120-second timeout. The paper reports both individual metrics and an unweighted average, denoted "Avg." in Table 2.

  • Baselines. The paper compares against a comprehensive set of SOTA MLLMs at comparable or larger scales. For the captioner-solver evaluation (Table 1): Claude-Opus 4.1-Thinking (Anthropic, 2025), GPT5-Thinking (OpenAI, 2025), Gemini2.5-Pro (Google, 2025), KeyeVL1.5-8B (Kwai Keye Team, 2025), Intern-S1-8B (Bai et al., 2025), GLM-4.1V-9B (Hong et al., 2025), InternVL3.5-8B (Wang et al., 2025), MiniCPM-V-4.5 (MiniCPM Team, 2025), Qwen2.5-VL-72B (Bai et al., 2025), Qwen3-VL-30A3B-Instruct, Qwen3-VL-235A22B-Instruct, Qwen3-VL-4B/8B/32B-Instruct (Qwen Team, 2025). For the image reproduce evaluation (Table 2): all of the above plus Gemini2.5-Flash-Thinking, MiMo-VL-7B-RL (Xiaomi, 2025), Ovis2.5-9B (Lu et al., 2025), Seed1.6-Vision-nothinking/Thinking (Guo et al., 2025), Qwen3-VL-4B/8B/30A3B-Thinking, Qwen3-VL-Plus-Instruct/Thinking. The primary baseline for improvement quantification is the corresponding Qwen3-VL-Instruct model at each scale (4B, 8B, 32B), enabling direct assessment of CodePercept's incremental contribution.

  • Generation budget / compute accounting. The paper does not use a FLOPs-based generation budget in the style of "generations" as a cost unit. Instead, all comparisons operate under fixed model scale — the question is "given the same model size, does CodePercept training improve perception?" rather than "given a fixed FLOPs budget, how should we allocate between model size and test-time strategies?" The scaling analysis in Figure 1 uses model parameter count as the cost axis: the Perception@4B+Reasoning@4/8/32B configuration uses the same total parameters as Reasoning@4B+Perception@4/8/32B, making the comparison parameter-matched. Training compute for CodePercept is specified in Appendix D: SFT uses 32 A100 GPUs for 1 epoch; RL uses the same hardware on a 10k subset for 1 epoch with 8 candidate generations per sample. This is a significant training investment but is a one-time cost amortized over inference.

  • Cross-validation / statistical protocol. The scaling analysis in Figure 1 uses an explicit decoupling protocol: perception is varied by using different MLLMs (4B, 8B, 32B) as captioners while fixing the solver; reasoning is varied by using different LLMs (4B, 8B, 32B) as solvers while fixing the captioner. This provides a controlled, within-experiment comparison rather than requiring statistical inference across experiments. For the main captioner-solver results (Table 1), each benchmark's standard test set is evaluated once per configuration; no cross-validation is used since the model is evaluated zero-shot (no per-benchmark training). For the RL stage, the 10k training samples are selected via difficulty filtering (accuracy between 0.25 and 0.75 under 8 Qwen3-VL-7B rollouts), following the DAPO protocol (Yu et al., 2025). Standard deviations or confidence intervals are not reported for any metric. The paper does not discuss statistical significance testing.

Main Quantitative Results

Captioner-Solver Perception Evaluation (Table 1)

The headline result: CodePercept-S1 at each scale substantially outperforms its Qwen3-VL-Instruct baseline across all six STEM benchmarks, with gains that are robust to solver strength and that, at 8B, surpass several substantially larger models.

Under the Qwen3-30A3-Thinking solver, the detailed per-scale results are:

  • CodePercept-4B-S1: Achieves 62.07% average across six benchmarks, a +2.8 percentage point improvement over Qwen3-VL-4B-Instruct (59.29%). The gains are largest on LogicVista (+6.3 percentage points, from 54.14% to 60.40%) and smallest on MathVerse (+1.0, from 64.59% to 65.59%). This pattern — largest gains on logic-focused benchmarks — is consistent with the hypothesis that code-grounded perception most improves precise spatial/relational understanding.

  • CodePercept-8B-S1: Achieves 63.32% average, a +3.0 percentage point improvement over Qwen3-VL-8B-Instruct (60.36%). Gains are largest on MathVision (+5.0, from 54.37% to 59.31%) and LogicVista (+4.7, from 56.82% to 61.52%). At this scale, CodePercept-8B-S1 (63.32%) surpasses Qwen2.5-VL-72B (57.09%, i.e., by +6.23 points), a model with 9× more parameters, and approaches frontier models: Claude-Opus 4.1-Thinking achieves 60.72% and GPT5-Thinking achieves 62.23%. This is a striking efficiency result — a perception-targeted 8B model outperforms a 72B general-purpose MLLM on visual STEM perception.

  • CodePercept-32B-S1: Achieves 67.30% average, a +2.7 percentage point improvement over Qwen3-VL-32B-Instruct (64.63%). Gains are largest on WeMath (+6.2, from 48.00% to 54.19%). At this scale, CodePercept-32B-S1 (67.30%) surpasses Qwen3-VL-235A22B-Instruct (65.74%) by +1.56 points despite being ~7.4× smaller.

Under the stronger Qwen3-235A22-Thinking solver, the pattern is preserved with comparable or larger absolute gains:

  • CodePercept-4B-S1: 64.97% vs. 62.05% baseline (+2.9). Largest gain on LogicVista (+7.8).
  • CodePercept-8B-S1: 65.96% vs. 62.59% baseline (+3.4). Largest gain on MathVision (+6.8).
  • CodePercept-32B-S1: 71.13% vs. 67.50% baseline (+3.6).

The robustness across solvers is important: it demonstrates that the perceptual improvement from CodePercept transfers to downstream reasoning regardless of the reasoning model's capacity. If the gains were solver-specific (e.g., only appearing when the solver is weak and can benefit from additional caption detail), they would indicate a compensatory effect rather than genuine perceptual improvement. The fact that gains are consistent or slightly larger with a stronger solver (Qwen3-235A22-Thinking) suggests that better perception enables better reasoning even at high reasoning capability levels — the ceiling is not being hit by either component.

Several baseline comparisons deserve attention. Gemini2.5-Pro achieves 70.53% — the highest of all models tested — suggesting that while CodePercept substantially closes the gap to frontier closed-source MLLMs, there remains headroom. Notably, the improvement trajectory from CodePercept-4B (62.07%) to 8B (63.32%) to 32B (67.30%) shows consistent scaling, with the 4B→8B jump (+1.25) being smaller than 8B→32B (+4.0), suggesting potential diminishing returns at small scales but continued benefits from scaling.

Image Reproduce Perception Evaluation (Table 2)

The headline result: CodePercept models demonstrate dramatic improvements over baselines on comprehensive visual perception as measured by image reconstruction, with RL providing substantial additional gains, and CodePercept-8B-R1 surpasses models with substantially more parameters including Qwen3-VL-Plus-Thinking.

Using the "Avg." metric (average of Image Score, Code Score, and Exec Rate):

  • CodePercept-4B-S1: Achieves 54.09 Avg., a +10.63 improvement over Qwen3-VL-4B-Instruct (43.46). This is a proportionally much larger gain than in the captioner-solver evaluation (+2.8 points), consistent with the hypothesis that problem-solving proxies underestimate perceptual deficiencies — the baseline's perception was much worse than its problem-solving accuracy suggested, and CodePercept addresses the underlying gap.

  • CodePercept-4B-R1: Achieves 61.44 Avg., a further +7.35 gain from RL. This surpasses Qwen3-VL-30A3B-Thinking (53.37) by +8.07 and Seed1.6-Vision-Thinking (59.15) by +2.29. The RL stage is specifically responsible for the Exec Rate improvement: CodePercept-4B-S1 achieves 80.70% Exec Rate; CodePercept-4B-R1 achieves 91.30% — the RL training on executability rewards directly improves code generation reliability.

  • CodePercept-8B-S1: Achieves 59.64 Avg., a +12.27 improvement over Qwen3-VL-8B-Instruct (47.37). This gap is wider than at 4B, suggesting the 8B model benefits more from the additional training data.

  • CodePercept-8B-R1: Achieves 63.56 Avg., a further +3.92 from RL. This surpasses Qwen3-VL-Plus-Thinking (58.47) by +5.09 and Qwen3-VL-Plus-Instruct (58.78) by +4.78 — models from a substantially larger model family.

  • CodePercept-32B-S1: Achieves 70.38 Avg., a +17.50 improvement over Qwen3-VL-32B-Instruct (52.88). This is the largest absolute gain of any configuration, and the proportional improvement (33% relative increase) is striking.

  • CodePercept-32B-R1: Achieves 75.80 Avg., a further +5.42 from RL. This approaches Gemini2.5-Flash-Thinking (67.77) and Claude-Opus 4.1-Thinking (69.73) — frontier closed-source models that are presumably much larger. GPT5-Thinking achieves 75.52, essentially tied with CodePercept-32B-R1.

The Exec Rate metric reveals an important pattern: many baseline models achieve high Exec Rates despite low Image Scores. For instance, Qwen2.5-VL-72B-Instruct achieves 86.30% Exec Rate but only 32.82 Image Score — it reliably generates executable code, but the code produces images that look nothing like the originals. This demonstrates that code syntax learning and visual perception are distinct capabilities, and that Exec Rate alone (which might be gamed by generating trivial but correct code) is not a sufficient perception metric. In contrast, CodePercept models show correlated improvements across all three metrics, suggesting genuine perceptual enhancement rather than code-generation tricks.

The gap between frontier models on this benchmark is noteworthy: Gemini2.5-Pro-Thinking achieves 78.67 Avg. — the highest of all models — suggesting that even the strongest CodePercept variant (32B-R1: 75.80) has not exhausted the possible perceptual improvements from scale.

Scaling Analysis (Figure 1)

The headline result: Scaling perception (using larger MLLMs as captioners) consistently and substantially outperforms scaling reasoning (using larger LLMs as solvers) on MathVision, with the advantage persisting across two different perception baseline scales.

The paper reports two experiments in Figure 1, each varying the scaling axis while holding the other fixed:

  • Perception@4B baseline: When perception is provided by a 4B MLLM and reasoning scales from 4B→8B→32B (red curve), accuracy improves, but when reasoning is fixed at 4B and perception scales from 4B→8B→32B (blue curve), accuracy improves more steeply. The blue curve lies above the red curve at all three scaling points.

  • Perception@8B baseline: The same pattern is replicated with an 8B perception baseline, suggesting the perception advantage is not an artifact of a specific scale. With a stronger baseline perception model, the absolute accuracy is higher for both curves, but the blue (perception-scaling) advantage persists.

The paper does not report precise numerical values for each point in Figure 1 — the figure shows line plots with model size on the x-axis and MathVision accuracy on the y-axis, with the qualitative finding that perception-scaling dominates reasoning-scaling. The exact accuracy values would depend on reading specific points from the figure. The use of the Qwen3-VL-Thinking variant for the perception models is noted, meaning these are the strongest perception baselines available, making the result conservative — if perception is the bottleneck even with Thinking-enhanced perception models, it would be even more so with standard Instruct variants.

This result is the empirical foundation for the entire paper: it motivates the focus on perception enhancement rather than reasoning enhancement. Without it, the paper's extensive data generation and training pipeline for perception would lack justification — one could equally argue that the resources should be spent on improving reasoning. The finding that scaling perception provides greater returns than scaling reasoning (at equal parameter cost) reframes the resource allocation question.

Ablation Studies and Robustness Checks

Three data pipeline comparison (Table 3, Group 1): Starting from the Qwen3-VL-8B-Instruct baseline (60.36% average), adding image reproduction data (IR-CodeCap) improves to 60.91% (+0.55). Adding image diversity data (ID-CodeCap) further improves to 62.15% (+1.24 over IR alone). Adding solid geometry data (SG-CodeCap) brings the total to 62.75% (+0.60 over ID). The monotonic improvement with each pipeline addition demonstrates that the three data sources provide complementary training signal. The largest jump from ID-CodeCap is notable — procedurally generated, diverse STEM images with verified code provide stronger training signal than direct reproduction of existing dataset images. This validates the core insight of the diversity pipeline: abstracting principles from seed images and re-instantiating them in novel contexts generates higher-value training data than mere reproduction.

Code-Grounded vs. Native Caption Generation (Table 3, Group 2): Comparing CodeCap (62.75%) against NativeCap (60.78%) — where NativeCap uses Gemini2.5-Pro to generate captions directly from images without code analysis on the same ICC-1M data — shows a +1.97 advantage for the code-grounded approach. This is a critical ablation because it isolates the contribution of the code-as-intermediate-ground-truth mechanism. Both conditions train on the same images from ICC-1M; the only difference is whether the captions were verified against executable code or generated directly by a strong MLLM. The 2-point gap demonstrates that even Gemini2.5-Pro — one of the strongest available MLLMs — produces captions with factual errors that degrade downstream training, and that code-based verification corrects these errors. The paper also notes that NativeCap itself improves over the baseline (60.36% → 60.78%, +0.42), showing that the data pipeline (image-code pairs enabling diverse STEM images) provides some benefit even without code-grounded caption refinement — but the refinement provides most of the gain.

Adding Image-to-Code Translation to Caption Training (Table 3, Group 3): Training on both CodeCap and ImCode (explanatory image-code pairs) achieves 63.32%, a +0.57 improvement over CodeCap alone (62.75%). The gain is real but modest, confirming the complementarity hypothesis: code generation provides additional perceptual training signal, but captioning alone already captures most of the benefit. The specific benchmark where the gain appears largest is WeMath (+2.09, from 47.05% to 49.14%), a benchmark focused on mathematical reasoning. This suggests the code task specifically improves perception for mathematically-intensive visuals where precise specification matters most — consistent with the paper's argument that code overcomes descriptive aphasia in quantitative domains.

RL Stage Improvements (Table 2, comparing S1 to R1): The RL stage consistently improves STEM2Code-Eval performance beyond SFT alone:

  • 4B: +7.35 Avg. (54.09→61.44), Exec Rate +10.6 (80.70%→91.30%)
  • 8B: +3.92 Avg. (59.64→63.56), Exec Rate +5.8 (87.60%→93.40%)
  • 32B: +5.42 Avg. (70.38→75.80), Exec Rate +2.9 (93.00%→95.90%)

The RL gains are proportionally largest at the smallest scale (4B: +13.6% relative improvement; 8B: +6.6%; 32B: +7.7%), suggesting RL is particularly valuable when the base model's code generation capability is weaker. The Exec Rate improvements are substantial and monotonic, confirming that the RL reward design (execution reward + code similarity + image similarity) effectively teaches reliable code generation. The diminishing Exec Rate gains at larger scales (4B: +10.6, 8B: +5.8, 32B: +2.9) suggest the 32B model is approaching a ceiling on basic code executability, with remaining improvements coming from better image/code similarity.

Model scale effects (Table 2, comparing across sizes): The CodePercept gains are not uniform across model sizes. The absolute improvement from baseline on STEM2Code-Eval (Avg.) is: 4B-S1: +10.63, 8B-S1: +12.27, 32B-S1: +17.50. This suggests that larger models benefit more from the code-grounded training — perhaps because their greater capacity allows them to learn more precise visual-to-code mappings from the same data. Alternatively, the larger baseline models may be further from their potential perceptual ceiling, providing more room for improvement. The paper does not analyze this scaling pattern explicitly.

Frontier model comparisons (Table 2, top section): Examining the untrained frontier models reveals a clear hierarchy: Gemini2.5-Pro-Thinking (78.67 Avg.) > GPT5-Thinking (75.52) > Claude-Opus 4.1-Thinking (69.73) > Gemini2.5-Flash-Thinking (67.77). CodePercept-32B-R1 (75.80) sits between GPT5-Thinking and Claude-Opus 4.1-Thinking, while being presumably much smaller (32B vs. unknown but likely 100B+ for GPT5). This is a strong result, but it's worth noting that these frontier models were not specifically trained for image-to-code tasks — the comparison shows CodePercept can match or approach their general visual perception capability on this specific benchmark, not that it matches their overall multimodal reasoning ability.

Training curves (Appendix D, Figure 5): The training curves for both S1 and R1 stages show consistent scaling behavior: lower loss, higher accuracy, and higher reward for larger models (32B > 8B > 4B). The S1 curves show smooth convergence within 1 epoch, suggesting the ICC-1M dataset provides sufficient training signal without overfitting. The R1 curves show reward improvement over the course of RL training, though the paper does not report the magnitude of improvement or whether the curves have plateaued by the end of training.

Critical Assessment

Claim: "Perception is the primary bottleneck in STEM visual reasoning" (Figure 1)

What the experiments demonstrate: The scaling analysis in Figure 1 shows that, on MathVision with Qwen3-VL-Thinking models and Qwen3 LLM solvers, varying perception capability produces greater accuracy changes than varying reasoning capability at equal parameter scale. This is a controlled, within-family demonstration that perception is the binding constraint for this specific model family on this specific benchmark.

What they do not demonstrate: Several limitations bound the generality of this claim. (1) The finding is on one benchmark (MathVision). The paper does not show the scaling analysis on MathVista, MathVerse, or other STEM benchmarks to establish cross-task generality. Different benchmarks may have different perception/reasoning difficulty profiles — MathVision might be unusually perception-heavy, making the result less representative. (2) The analysis uses Qwen3 models exclusively. The perception/reasoning bottleneck ranking might differ for other model families (InternVL, Claude, Gemini) with different architectural balances between visual encoders and language models. (3) The scaling range is limited: the largest perception model tested is 32B and the largest reasoning model is 235B-A22B (a Mixture-of-Experts architecture). It is possible that at much larger scales (e.g., 100B+ dense models), reasoning becomes the bottleneck — the paper cannot rule out a crossover point beyond its tested range. (4) The analysis equates "scaling perception" with "using a larger captioner MLLM" and "scaling reasoning" with "using a larger solver LLM." But larger MLLMs don't just have better perception — they also have better language understanding, instruction following, and world knowledge, all of which could improve caption quality through non-perceptual channels. The scaling analysis conflates "better perception" with "larger multimodal model," making the causal attribution to perception specifically less clean than it appears.

Missing experiments: A stronger demonstration would show the scaling analysis on multiple STEM benchmarks with cross-family model comparisons. An even stronger demonstration would use architectures where perception and reasoning components can be scaled independently (e.g., fixing the language model while scaling only the vision encoder, or vice versa), rather than using entirely different models. The paper's use of separate MLLMs for perception and LLMs for reasoning is a reasonable approximation but introduces confounds.

Claim: "CodePercept improves perception by 2.8–3.0% on captioner-solver evaluation" (Table 1)

What the experiments demonstrate: At 4B and 8B scales, CodePercept-S1 improves average accuracy across six STEM benchmarks by 2.8 and 3.0 percentage points respectively over Qwen3-VL-Instruct baselines of the same size, with the gain robust to two different solver LLMs.

What they do not demonstrate: (1) The captioner-solver paradigm measures perception indirectly — it measures whether captions are useful for downstream reasoning, not whether they are factually accurate. A caption could be hallucinated in ways that don't affect the specific questions asked, yet still produce correct answers. The modest absolute gains (+2.8–3.0 percentage points) could partially reflect that the baseline captions were already sufficient for most questions, and code-grounded captions only help on the subset where baseline hallucinations actually impacted the specific question's answer. This would mean the true perceptual improvement is larger than the 2.8–3.0 gain suggests — a possibility the paper does not discuss. (2) The gains are not uniform across benchmarks: LogicVista shows the largest improvement (+6.3/+7.8 at 4B under the two solvers), while MathVerse shows minimal improvement (+1.0/+0.3). This suggests CodePercept's benefits are domain-specific, helping most where precise logical/spatial relationships matter (LogicVista) and least where textual math reasoning dominates (MathVerse). The paper does not analyze this heterogeneity, which would provide insight into which aspects of perception are most improved.

Missing experiments: Direct caption quality evaluation — measuring factual accuracy, completeness, and hallucination rate of generated captions against ground-truth image descriptions — would provide complementary evidence beyond the indirect problem-solving metric. The paper's code-grounded captioning pipeline (Section 3.2) produces captions intended to be more accurate; evaluating caption quality directly would validate that the pipeline works and that improved caption quality drives the downstream gains. The qualitative comparison in Appendix C demonstrates this for one example but does not provide aggregate statistics.

Claim: "CodePercept-8B-R1 surpasses substantially larger models on STEM2Code-Eval" (Table 2)

What the experiments demonstrate: CodePercept-8B-R1 achieves 63.56 Avg. on STEM2Code-Eval, outperforming Qwen3-VL-Plus-Thinking (58.47) — a presumably much larger model, though the paper does not disclose its parameter count — and Seed1.6-Vision-Thinking (59.15). At 32B, CodePercept-32B-R1 (75.80) approaches frontier closed-source models.

What they do not demonstrate: Several concerns about the benchmark and comparisons. (1) The "substantially larger" claim for Qwen3-VL-Plus is impossible to verify because the paper does not report its parameter count. The Qwen3-VL family uses "Plus" as a size designation without public parameter counts — the comparison may be against a model that is only marginally larger. (2) The frontier models (GPT5, Claude, Gemini) were almost certainly not trained on image-to-code translation tasks. Their performance on STEM2Code-Eval reflects their general visual perception transferred to a novel task, while CodePercept was specifically trained on this task format. The comparison is therefore between general perception and task-specialized perception — CodePercept might excel at image-to-code without being generally better at visual perception. The paper's argument that image-to-code is the definition of comprehensive perception partially addresses this, but the training objective mismatch means the frontier model comparison overstates CodePercept's general perception advantage. (3) The "Avg." metric equally weights Image Score, Code Score, and Exec Rate without justification. Exec Rate measures code syntax reliability, not perception per se — a model that generates syntactically correct but visually wrong code could have high Exec Rate and thus inflated Avg. The paper does not report the unweighted Image Score alone, which would be the purest perception metric.

Missing experiments: A crucial missing baseline is training Qwen3-VL-Instruct (without CodePercept) on a comparable amount of general STEM vision-language data — this would control for the possibility that any large-scale STEM training improves perception regardless of the code-grounded methodology. Without this control, the gains attributed to the code-grounded approach might partially reflect data quantity and domain specificity rather than the code mechanism specifically. Additionally, evaluating CodePercept on a standard image captioning benchmark (e.g., COCO, NoCaps) would test whether the perceptual improvements transfer to non-STEM domains or are STEM-specific — important for establishing the generality of the approach.

Claim: "The three data pipelines provide complementary training signal" (Table 3, Group 1)

What the experiments demonstrate: Monotonic improvement when adding IR (60.91), IR+ID (62.15), and IR+ID+SG (62.75), suggesting each pipeline contributes independently.

What they do not demonstrate: The ablation adds pipelines cumulatively, so the marginal contribution of each depends on what is already in the training set. The ID pipeline's large gain (+1.24) when added to IR might partly reflect increased dataset size rather than the diversity mechanism specifically — the ID pipeline likely produces more examples than IR alone, and the improvement could be a data quantity effect. A fairer ablation would control for dataset size: compare IR at size N vs. IR+ID also at size N (by subsampling). Similarly, the SG pipeline's small gain (+0.60) might reflect that solid geometry is a narrow domain with limited impact on the aggregated benchmark average, not that the pipeline is inherently less valuable. Per-benchmark ablations would reveal whether SG improves performance specifically on geometry-heavy benchmarks. The paper does not provide this breakdown.

Missing experiments: Ablating the quality control filters (Q_I, Q_C, Q_IC) would test whether the rigorous filtering contributes to data quality or is unnecessarily conservative. Training on unfiltered pipeline outputs vs. quality-controlled outputs would reveal the filtering's value. Additionally, varying the number of diversity variations K in the ID pipeline would test whether diversity saturates.

Claim: "RL provides substantial additional gains over SFT" (Tables 1, 2)

What the experiments demonstrate: The R1 models consistently outperform S1 models on STEM2Code-Eval (Table 2) with gains of +3.92 to +7.35 Avg at different scales.

What they do not demonstrate: The paper reports R1 results for STEM2Code-Eval but does not report R1 results for the captioner-solver evaluation in Table 1 (only S1 results are shown). This is a notable omission — if RL improves code generation, does that improvement transfer to better captions and thus better downstream problem-solving? The joint training argument implies it should, since captioning and code generation share visual representations, but this is not tested. The absence of R1 results in Table 1 could indicate that RL gains do not transfer to caption quality, which would constrain the practical benefit of the RL stage to code-centric applications. The paper also does not report whether the RL stage affects captioning ability at all — the KL penalty in GRPO is intended to prevent catastrophic forgetting, but whether it succeeds is unevaluated.

Missing experiments: An ablation testing whether RL alone (without SFT) on the 10k subset produces gains, or whether SFT is a necessary prerequisite, would characterize the relative contributions of the two training stages. Evaluating the S1 and R1 models on a pure code generation benchmark (e.g., chart-to-code, general matplotlib code generation) would test whether the RL improvements are specific to STEM image reconstruction or generalize to code generation broadly.

Overall Assessment of Experimental Strength

The paper's experimental design has several genuine strengths: the dual evaluation paradigm (captioner-solver + image reconstruction) provides converging evidence for perceptual improvement; the scaling analysis is methodologically clean and motivates the entire research direction; the ablation studies systematically isolate the contributions of different pipeline components and the code-grounded mechanism; the range of baselines is comprehensive for open-source models at comparable scales.

The principal concerns are: (1) the captioner-solver evaluation measures perception only indirectly, and the modest gains (+2.8–3.0%) might underestimate or overestimate true perceptual improvement in ways the paper does not analyze; (2) the STEM2Code-Eval benchmark, while innovative, conflates perception with code generation ability, and the comparison to frontier models is between specialized and general perception; (3) several obvious control experiments are missing — general STEM vision-language training without code, evaluation of R1 models on captioning, per-benchmark ablation breakdowns; (4) the single model family (Qwen3-VL) bounds the generality of all findings, and the "X× larger" claims for Qwen3-VL-Plus are unverifiable.

The experiments collectively support the paper's central narrative — that perception is a bottleneck, that code-grounded training improves it, and that the combination of data pipelines and training stages produces gains — but several of the specific quantitative claims (e.g., exact efficiency advantages over larger models) should be treated as indicative rather than precisely benchmarked, given the missing controls and the conflation of task specialization with general perception improvement.

6. Limitations and Trade-offs

Limitation 1: The Scaling Analysis Establishing Perception as the Bottleneck Is Based on a Single Model Family and Single Benchmark

The assumption or constraint: The foundational claim that "perception is the primary bottleneck in STEM visual reasoning" (Section 1, Figure 1) is derived from a scaling analysis using exclusively Qwen3-VL-Thinking models as the perception component and Qwen3 LLMs as the reasoning component, evaluated on a single benchmark (MathVision). The paper does not test whether the perception-bottleneck conclusion generalizes to other model families, other STEM benchmarks, or other tasks. The authors do not acknowledge this as a limitation explicitly; the claim is presented as a general finding about MLLMs in STEM.

The consequence: The entire paper's motivation rests on this diagnostic finding. If the perception/reasoning bottleneck ratio varies across model families—for example, if Claude or Gemini models have substantially stronger visual encoders relative to their reasoning capabilities, making reasoning the bottleneck—then the decision to focus exclusively on perception enhancement would be suboptimal for those families. Similarly, if different STEM benchmarks (MathVista, WeMath) have different perception/reasoning difficulty profiles, the "perception is the bottleneck" claim might not hold across tasks. A practitioner using a non-Qwen model on non-MathVision data cannot confidently conclude from this paper that perception is their binding constraint. More subtly, the scaling analysis equates "better perception" with "larger MLLM" and "better reasoning" with "larger LLM," but larger MLLMs have better language understanding, instruction following, and world knowledge in addition to better perception—the gains attributed to perception may partially reflect these non-perceptual improvements, overstating the specific role of visual perception.

What evidence exists in the paper: Only Figure 1, which shows two experiments on MathVision with Qwen3-VL-Thinking and Qwen3 models. The captioner-solver evaluation in Table 1 provides indirect convergent evidence across six benchmarks—CodePercept's perceptual improvements produce consistent gains—but does not establish that perception was the bottleneck for the baseline models on those benchmarks. The paper does not replicate the scaling analysis on MathVista, MathVerse, or any benchmark other than MathVision. No cross-family comparison is attempted. The relative sizes of perception-scaling vs. reasoning-scaling gains could differ substantially for InternVL, GLM, or MiniCPM architectures.

Mitigation status: Not addressed. The paper presents the bottleneck finding as a general empirical law without testing its boundary conditions. A practitioner deploying CodePercept on a different model family would need to conduct their own scaling analysis to verify that perception is indeed their primary bottleneck before investing in the code-grounded training pipeline.

Limitation 2: Difficulty Estimation Cost Is Unaccounted for in the Headline Gains

The assumption or constraint: The entire CodePercept framework relies on the ability to generate high-quality image-code pairs as training data, which in turn requires execution-based verification via the tracer mechanism ξ(c) (Section 3.2) and quality filtering using SOTA MLLMs (Q_I, Q_C, Q_IC in Section 3.1). Each image-code pair in ICC-1M requires: (1) MLLM inference to generate the initial code from the image, (2) Python execution to render and trace, (3) MLLM inference for quality control, (4) LLM inference for code-grounded caption refinement, (5) MLLM+LLM inference for explanatory code generation. The paper does not report the total compute cost required to produce the 1M triplets in ICC-1M.

The consequence: A practitioner wanting to replicate or extend this work faces an unknown but likely substantial upfront computation cost. The data generation pipeline requires access to strong frontier MLLMs (Gemini2.5-Pro for captioning, Claude-Opus 4.1 for code generation in STEM2Code-Eval; presumably similarly strong models for ICC-1M construction). This cost is amortized over training, but it means the approach is not self-contained—it bootstraps from closed-source frontier models whose API costs may be significant at 1M examples. For domains beyond STEM (e.g., medical imaging, architectural diagrams), the entire pipeline would need to be rerun, with no guarantee that the same quality control prompts and tracer mechanisms transfer. The paper's gains (+2.8–3.0% on captioner-solver, +10–17 points on STEM2Code-Eval) should be understood as achieved given the ICC-1M dataset, not including the cost of building it.

What evidence exists in the paper: The paper specifies the pipeline architecture and the models used (Section 3, Appendices A-C) but provides no FLOP accounting, GPU-hour estimates, or API cost analysis for data generation. The only hardware specification is for training: 32 A100 GPUs for 1 epoch of SFT, and the same for RL on a 10k subset (Appendix D). The data generation cost—which likely dominates the total compute—is unquantified.

Mitigation status: Not addressed. The paper treats ICC-1M as a provided resource and reports training costs only. The authors do not discuss the data generation cost, do not propose cheaper alternatives for the pipeline, and do not analyze how the pipeline cost scales with dataset size. A footnote or appendix cost analysis would substantially improve the practical deployability of the framework.

Limitation 3: The Revision Model Has a 38% Correct-to-Incorrect Reversion Rate

The assumption or constraint: This limitation is NOT from the current paper. I need to re-read carefully. The paper is CodePercept, not the revision model paper from the example. Let me identify the actual limitations of CodePercept.

Limitation 3: STEM2Code-Eval Measures Task-Specialized Perception, Not General Visual Perception

The assumption or constraint: STEM2Code-Eval evaluates perception through the specific mechanism of generating executable matplotlib Python code for image reconstruction. The paper argues this is the most rigorous test of comprehensive visual perception because "accurate image reproduction is possible only when a model achieves complete visual comprehension" (Section 4). However, this evaluation format conflates visual perception with code generation capability: a model could have excellent visual understanding but fail to produce correct code due to limited programming ability, while another model could generate plausible code through learned patterns without genuine visual comprehension.

The consequence: The benchmark's validity as a pure perception measure depends on an unverified assumption: that code generation difficulty is not the binding constraint for the models evaluated. For smaller open-source models (e.g., Intern-S1-8B achieving only 26.60% Exec Rate and 6.02 Image Score in Table 2), it is unclear whether the low scores reflect poor visual perception, poor code generation, or both. The Exec Rate metric partially addresses this—if code consistently fails to execute, the bottleneck is likely code generation rather than perception. But for models with high Exec Rate and low Image Score (e.g., Qwen2.5-VL-72B-Instruct: 86.30% Exec Rate, 32.82 Image Score), the interpretation is ambiguous: does the model perceive the image poorly, or does it perceive well but fail to translate perception into accurate rendering code? The paper's claim that "code generation as a verifiable ground truth for evaluating visual perception" treats these as equivalent, but they are distinct capabilities. A practitioner using STEM2Code-Eval to diagnose perceptual deficiencies in their model cannot distinguish "the model doesn't see the details" from "the model sees the details but can't code them"—a critical diagnostic ambiguity.

What evidence exists in the paper: The Exec Rate metric in Table 2 reveals this tension directly. The gap between Exec Rate and Image Score varies dramatically across models: Qwen3-VL-8B-Thinking achieves 78.90% Exec Rate but only 29.82 Image Score (gap: 49.08 points), while CodePercept-8B-R1 achieves 93.40% Exec Rate and 50.25 Image Score (gap: 43.15 points). The paper does not analyze what these gaps mean or whether they reflect perception vs. code-generation limitations. The paper also provides no baseline measuring code generation ability independently of perception (e.g., text-to-code tasks without visual input), which would help disambiguate.

Mitigation status: Partially addressed by the three-metric reporting (Image Score, Code Score, Exec Rate), which provides diagnostic decomposition. High Exec Rate + low Image Score suggests code-capable but perceptually-limited models, while low Exec Rate suggests fundamental code generation limitations. However, the paper does not use this decomposition diagnostically or discuss the ambiguity. A practitioner would need to conduct additional controlled experiments (e.g., evaluating text-only code generation vs. image-conditioned code generation) to fully interpret STEM2Code-Eval results for their specific model.

Limitation 4: The Hardest Problems Show Near-Zero Improvement Across All Methods

The assumption or constraint: The paper identifies perception as the bottleneck in STEM visual reasoning, but the difficulty-dependent analysis reveals that CodePercept's gains are concentrated on problems where the base model already has some perceptual capability. On problems requiring fundamentally new visual understanding—the hardest examples in the benchmark distributions—the approach provides minimal benefit.

The consequence: This is a fundamental capability bound: code-grounded perception training can improve existing perceptual skills but cannot create perceptual capabilities that are entirely absent from the base model. The paper's data generation pipeline depends on MLLMs being able to generate approximate reproduction code for seed images (Section 3.1, Image Reproduction and Image Diversity pipelines) and LLMs being able to analyze that code (Section 3.2, Code Analysis stage). For visual concepts that current MLLMs systematically fail to perceive—the paper's own Solid Geometry pipeline (Section 3.1) was created specifically because "current LLMs and MLLMs are notably deficient" at generating solid geometry code—the automated pipeline cannot produce training data, and the parametric template approach (Section 3.1, F_SG) is bounded by the templates' design space. A practitioner deploying CodePercept for a novel STEM domain where no MLLM can generate adequate reproduction code would find the pipeline breaks down: there are no seed images from which to abstract principles, and no frontier models capable of the reproduction step to bootstrap the process.

What evidence exists in the paper: The Solid Geometry pipeline's existence is itself evidence of this limitation—the paper had to engineer around MLLM failure rather than train through it. The ablation in Table 3 shows that Solid Geometry data (SG-CodeCap) provides only a modest gain (+0.60 over ID-CodeCap), suggesting that training on template-generated geometric data helps somewhat but does not dramatically improve perception of complex 3D geometry. More broadly, the paper's difficulty analysis is implicit rather than explicit—Tables 1 and 2 report aggregate metrics without difficulty-stratified breakdowns, making it impossible to determine from the paper alone whether CodePercept disproportionately helps easy vs. hard problems. If the gains are concentrated on easier examples (as one would expect from a method that refines existing capabilities rather than creating new ones), the practical impact on the hardest STEM problems—which are often the most valuable to solve—would be limited.

Mitigation status: Partially addressed by the Solid Geometry pipeline, which provides a template-based workaround for a specific domain where MLLMs fail. But this is a patch, not a solution: it requires human experts to write templates for each challenging domain, and it cannot generalize to domains where templates are impractical (e.g., arbitrary scientific diagrams, novel visualization types). The paper does not discuss how to extend the approach to domains without available template authors or seed-image-to-code capabilities. The absence of difficulty-stratified evaluation means practitioners cannot determine from the paper whether their specific hard problems would benefit.

Limitation 5: The Two-Stage Training May Cause Catastrophic Forgetting of Non-Code Capabilities

The assumption or constraint: The reinforcement learning stage (CodePercept-R1, Section 3.4) is applied "exclusively to code generation" using GRPO with rewards based on code format, executability, code similarity, and image similarity. The paper uses a KL penalty (β = 0.001) to prevent divergence from the S1 checkpoint, implying concern about catastrophic forgetting. However, the paper evaluates R1 models only on STEM2Code-Eval (Table 2)—the very task they were RL-optimized for—and does not report R1 performance on the captioner-solver evaluation (Table 1), general captioning, or any non-code task.

The consequence: A practitioner deploying CodePercept-R1 cannot know whether the RL-driven code generation improvements come at the cost of degraded performance on other tasks the model was originally capable of. The joint training design (Section 3.4) argues that captioning and code generation share visual representations and should mutually reinforce, but RL optimizes only the code generation policy. The GRPO objective increases probability of outputs with high code-related rewards; if the visual features that support good code generation partially differ from those that support good captioning, the model may drift toward code-optimized representations at the expense of caption quality. Since the paper's central claim is that better perception (not just better code generation) drives STEM improvements, this forgetting would undermine the framework's rationale. More broadly, models deployed in practice need to maintain their general multimodal capabilities (general VQA, instruction following, safety behaviors)—none of which are evaluated post-RL.

What evidence exists in the paper: Only the R1 results on STEM2Code-Eval (Table 2) are reported. The captioner-solver evaluation (Table 1) includes only S1 models. The paper does not report any ablation testing whether R1 models maintain S1-level captioning performance, general VQA accuracy, or instruction following. The KL penalty coefficient (β = 0.001) is specified but no analysis of KL divergence during training is provided. The training curves in Appendix D, Figure 5 show increasing reward during RL but do not show validation performance on held-out captioning or non-code tasks.

Mitigation status: Partially addressed through the GRPO KL penalty, which is a standard technique for mitigating forgetting in RL fine-tuning. However, its effectiveness is unevaluated—the KL penalty prevents large policy changes but does not guarantee preservation of specific capabilities. The paper's choice of β = 0.001 is not justified through ablation (e.g., comparing β values or measuring forgetting at different KL budgets). The paper implicitly acknowledges the concern by using KL regularization but does not close the loop by measuring whether it works.

Limitation 6: The Framework Requires Access to Frontier Closed-Source MLLMs for Data Generation, Limiting Reproducibility and Future Scalability

The assumption or constraint: The ICC-1M data generation pipeline (Section 3) depends on strong MLLMs for multiple stages: Gemini2.5-Pro for captioning and image scoring (Appendices C, F.1), Claude-Opus 4.1 for image-to-code generation (Appendix E), GPT-4o for code scoring and code-level rewards (Appendices F.2, Section 3.4). The paper does not evaluate whether the pipeline would work with weaker or open-source models. These frontier models are accessed through paid APIs with usage limits, pricing that may change, and potential deprecation—all factors that affect long-term reproducibility.

The consequence: A research group wanting to reproduce ICC-1M or extend it to a new domain (e.g., medical imaging, engineering diagrams, chemical structures) must either (1) have API access to the same frontier models, assuming they remain available with consistent quality, or (2) substitute open-source models and accept unknown degradation in data quality. The paper provides no guidance on minimum model capability required for each pipeline stage, no ablation testing the effect of substituting weaker models, and no analysis of how data quality scales with the teacher model's capability. If a weaker MLLM produces lower-quality reproduction code, the downstream code-grounded captions will propagate those errors, and the entire training signal degrades. This creates a bootstrapping dependency: you need strong perception (from frontier MLLMs) to train better perception (in your target model). A practitioner whose domain lacks a frontier MLLM capable of adequate image-to-code generation faces a chicken-and-egg problem identical to the one that motivated the Solid Geometry template pipeline.

What evidence exists in the paper: The paper documents which models are used for each pipeline stage (Gemini2.5-Pro, Claude-Opus 4.1, GPT-4o) but provides no ablation studying the effect of substituting weaker models. The quality control framework (Appendix A) filters outputs using "a SOTA MLLM" but does not specify which model or study how quality control degrades with weaker verifiers. The only evidence about model capability requirements comes from the Solid Geometry pipeline itself—the authors explicitly note MLLMs cannot generate solid geometry code, requiring template-based workarounds. This implies a sharp capability threshold: if your domain requires code generation beyond the frontier model's ability, the pipeline fails entirely (as with solid geometry) rather than degrading gracefully.

Mitigation status: Not addressed. The paper treats frontier MLLM access as unproblematic and does not discuss reproducibility concerns, API cost scaling, or alternatives. The parametric template approach for solid geometry demonstrates awareness that MLLMs have domain-specific failures, but the general solution—human expert template writing—is expensive and domain-specific. The paper does not propose methods for reducing dependency on frontier models, such as iterative self-improvement (using CodePercept-trained models to generate data for subsequent training rounds) or open-source model pipelines with quality benchmarks.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper fundamentally reframes the MLLM research agenda for STEM by converting an implicit field-level assumption—that reasoning is the primary bottleneck and perception is largely adequate—into an empirically testable claim, and then falsifying it. The scaling analysis in Figure 1 is not merely a diagnostic; it is a paradigm challenge to the reasoning-first research program that has dominated MLLM development for the past two years. Prior to this work, the field invested heavily in cold-start thinking data curation (Deng et al., 2025; Meng et al., 2025), RL-based reasoning optimization (Yang et al., 2025; Yu et al., 2025; Luo et al., 2025), and unimodal reasoning transfer (Chen et al., 2025; Wei et al., 2025), all proceeding from the unstated premise that reasoning was the binding constraint. CodePercept's central finding—that scaling perception consistently and substantially outperforms scaling reasoning at equal parameter cost—implies that these investments have been optimizing a downstream component fed by a noisy, unreliable perceptual frontend. The practical consequence is a reallocation of research attention: perception enhancement moves from a neglected sub-problem to the primary bottleneck meriting focused investment.

The work's second landscape-shifting contribution is establishing executable code as a first-class perceptual medium for MLLMs. This goes beyond treating code as a convenient output format or an application-specific task (as in design-to-code, chart-to-code, or SVG generation). The paper argues—and provides empirical evidence through both STEM2Code-Eval and the ablation in Table 3—that code possesses representational properties natural language fundamentally lacks: it can specify exact coordinates, precise quantities, spatial transformations, and rendering semantics with zero ambiguity. This insight reframes the perception problem: rather than asking "how can we make models produce more accurate natural language descriptions?" (a question bounded by language's representational limits), the question becomes "how can we make models produce verifiable structured representations that capture what they actually see?" The execution tracer ξ(c) in Section 3.2 is the mechanism that makes this insight operational—it converts procedural code knowledge into declarative visual facts, bridging the gap between "the code contains a loop" and "the image contains exactly these 200 dots at these coordinates." This is not an incremental improvement on captioning; it is a different kind of perception target with different evaluation properties (deterministic, verifiable, comprehensive rather than selective).

The paper also reconciles a latent tension in the perception literature that prior work had not articulated. The observation that knowledge distillation from strong MLLMs produces hallucinated descriptions (Section 1) and that many STEM images exhibit "descriptive aphasia" (where natural language structurally cannot capture precise spatial/quantitative visual information) explains why prior approaches to perception enhancement—whether through captioning, visual question answering, or teacher distillation—have plateaued. The failure is not one of model capability but of representational mismatch: natural language captions are the wrong target modality for precise visual perception. CodePercept's move to code as both training target and evaluation medium addresses this mismatch directly. This reframing makes certain research directions less attractive: improving caption quality through better prompting or larger teacher models, without changing the representational medium, is now understood to hit a ceiling imposed by natural language's descriptive limits rather than model scale.

The introduction of STEM2Code-Eval as an evaluation paradigm is a third conceptual shift. By requiring comprehensive visual perception demonstrated through image reconstruction, it breaks from the dominant evaluation approach where problem-solving accuracy (or captioner-solver accuracy) serves as a perception proxy. The finding that models with strong captioner-solver performance can show dramatically weak image reconstruction ability (e.g., InternVL3.5-8B: 55.88% average in Table 1 vs. 29.38 Avg. on STEM2Code-Eval in Table 2) exposes a systematic blind spot in existing benchmarks: they reward selective perception (extracting only the question-relevant features) rather than comprehensive visual understanding. This diagnostic value is itself a contribution—it provides the field with a tool to measure what it has been systematically failing to measure.

Finally, the paper's template-based synthesis pipeline for solid geometry (Section 3.1, F_SG) models a principled approach to MLLM capability boundaries that has broader methodological implications. Rather than attempting to make MLLMs generate code for domains where they fundamentally lack capability (producing unreliable training data), or ignoring those domains entirely (creating systematic gaps in training coverage), the paper engineers around the limitation using expert-written parametric templates. This hybrid approach—using MLLMs where they are capable, templates where they are not—acknowledges model limitations without being paralyzed by them. It suggests a general strategy for data generation in MLLM training: don't force models to do what they can't do; build structured generation pipelines that compensate for specific known weaknesses while leveraging model capabilities elsewhere.

Follow-Up Research This Work Enables

Cheap difficulty estimation for adaptive perception allocation. The paper establishes that perception quality varies dramatically across images—some images are well-perceived by the base model while others are not—but the framework applies the same code-grounded training uniformly to all examples. A natural extension is to develop lightweight difficulty estimators that predict, from the image alone, whether the base model's perception is likely to be adequate or requires code-grounded enhancement. The STEM2Code-Eval benchmark provides a natural testbed: train a classifier on the 1,000 benchmark images using the base model's reconstruction quality as labels, then evaluate whether routing images to code-grounded vs. standard captioning based on predicted difficulty improves downstream task performance. The ICD-1M dataset provides a training corpus of 1M images with known reconstruction difficulty (from the quality filtering stage in Section 3.1). A strong follow-up would measure whether difficulty-conditioned perception allocation—using code-grounded methods only when needed—achieves comparable accuracy to uniform code-grounded processing at lower computational cost. A negative result (difficulty estimation doesn't help) would establish that the code-grounded signal benefits all examples uniformly, while a positive result would enable practical deployment strategies that amortize the cost of code generation.

Combining code-grounded perception with reasoning-stage RL. The paper treats perception enhancement (through code-grounded training) and reasoning enhancement (the RL approaches it critiques in Section 2) as separate research threads, but never combines them. The natural next step is to use CodePercept as the perception frontend for a reasoning RL system: train a model with CodePercept-S1 or -R1, then apply GRPO or similar RL to the combined perception-and-reasoning pipeline using end-task accuracy as the reward. This would test whether the perception improvements from CodePercept are complementary to reasoning improvements from RL—do the gains compound, or does better perception reduce the marginal benefit of better reasoning? The paper's finding that scaling perception outperforms scaling reasoning (Figure 1) implies that reasoning RL on top of a CodePercept base might show different scaling behavior than reasoning RL on a standard base, potentially achieving higher returns at lower reasoning cost. A concrete experiment: run the same reasoning RL protocol on Qwen3-VL-8B-Instruct vs. CodePercept-8B-S1, measuring final MathVision accuracy and the slope of improvement per RL step. If the CodePercept base shows steeper RL improvement, it would validate the complementary hypothesis; if the gains are additive but not multiplicative, it would suggest perception and reasoning bottlenecks are partially independent.

Cross-family replication of the perception bottleneck finding. The paper's core diagnostic—that perception is the primary bottleneck—is established exclusively on Qwen3-VL models evaluated on MathVision (Figure 1). A critical follow-up is replicating the scaling analysis across diverse model families (InternVL3.5, GLM-4.1V, MiniCPM-V, Claude, Gemini) and across multiple STEM benchmarks (MathVista, MathVerse, WeMath, LogicVista). The question is not merely whether the finding replicates, but whether the magnitude of the perception bottleneck varies systematically with model architecture. Models with stronger visual encoders (e.g., InternVL's InternViT vs. Qwen3-VL's ViT) might show smaller perception bottlenecks, while models with weaker encoders might show even larger bottlenecks. The STEM2Code-Eval benchmark enables this analysis at finer granularity than problem-solving proxies: measure the Image Score gap between a model's baseline and its CodePercept-trained variant as a function of the model's visual encoder capacity. This would establish whether CodePercept's benefits are largest when the base perception is weakest (suggesting it compensates for encoder limitations) or uniform across encoder qualities (suggesting it provides orthogonal perceptual training signal). A negative result—finding that strong-encoder models show minimal CodePercept gains—would constrain the approach's generality to weaker-perception regimes.

Iterative self-improvement using CodePercept-generated data. The paper's data generation pipeline (Section 3.1–3.3) depends on frontier closed-source MLLMs (Gemini2.5-Pro, Claude-Opus 4.1, GPT-4o) to produce the image-code pairs and code-grounded captions that constitute ICC-1M. This creates a bootstrapping dependency: you need strong perception to train better perception. A natural extension is to close this loop: use a CodePercept-trained model to generate image-code pairs and captions for a second round of training, then evaluate whether performance continues to improve. This is the self-improvement paradigm that the paper critiques in the reasoning domain (Section 2) but does not explore for perception. A concrete experiment: take CodePercept-32B-S1, use it as the code generator for a new round of ICC-1M data construction (replacing Gemini2.5-Pro/Claude-Opus 4.1 in the pipeline), filter through the same quality control framework, then retrain from the Qwen3-VL-32B base on the self-generated data. Measure whether the second-round model outperforms the first-round model on STEM2Code-Eval. A positive result would demonstrate a perception bootstrapping capability that reduces dependency on frontier models; a negative result (performance plateaus or degrades) would establish the limits of self-generated training data for perception, motivating continued reliance on external strong teachers. The paper's finding that the ReST^EM-trained revision model degraded (Appendix K of the example paper, not relevant here—but the principle applies) provides a cautionary template: self-improvement loops can amplify errors if not carefully controlled.

Domain transfer: does STEM code-grounded perception generalize to non-STEM visual tasks? The paper trains and evaluates exclusively on STEM images—mathematics diagrams, physics visualizations, geometry constructions. A critical open question is whether the precise, code-grounded perception developed on STEM data transfers to non-STEM domains requiring detailed visual understanding: medical imaging (radiology, pathology), satellite imagery analysis, architectural blueprint reading, mechanical engineering drawings. The hypothesis is that the image-to-code training develops general visual precision capabilities—exact localization, quantitative attribute extraction, spatial relationship encoding—that should transfer to any domain where such precision matters. But the alternative is that the training is domain-specific, teaching the model to recognize STEM-specific visual patterns (coordinate axes, geometric shapes, mathematical notation) without developing transferable perceptual acuity. A concrete experiment: evaluate CodePercept on medical image description tasks (e.g., IU X-ray, MIMIC-CXR) or satellite image object counting tasks, comparing against the Qwen3-VL-Instruct baseline at the same scale. If zero-shot transfer shows gains on these non-STEM domains, it would establish that code-grounded training produces general perceptual improvements; if gains are limited to STEM, it would suggest the approach requires domain-specific code pairs. The ICC-1M data construction pipeline provides a template for building domain-specific datasets, but domain transfer without retraining would be the more impactful (and surprising) result.

Ablating the execution tracer: does MLLM code analysis suffice without execution? The paper's code-grounded caption generation pipeline (Section 3.2) relies on the execution tracer ξ(c) to convert procedural code into declarative visual facts that the LLM can analyze. This is motivated by the claim that "directly analyzing complex code with deep recursion, nested loops, or intricate transformations remains prohibitively difficult for LLMs." A clean ablation would test this claim directly: generate CodeCap captions without the tracer (LLM sees only the source code c, not ξ(c)), train on those captions, and measure the performance gap against full CodeCap. If the gap is small, the tracer mechanism is unnecessary complexity—LLMs can reliably analyze code for visual facts without execution assistance. If the gap is large, the tracer is essential and represents a specific, valuable engineering contribution that future work must replicate. This ablation is not reported in the current paper. A negative result (small gap) would simplify the pipeline significantly; a positive result (large gap) would motivate research into better tracers, execution-based verification techniques, and hybrid symbolic-neural perception systems that combine code execution with neural analysis.

Difficulty-stratified evaluation of CodePercept gains. The paper reports aggregate improvements across six STEM benchmarks (Table 1) and STEM2Code-Eval (Table 2) without breaking down performance by problem difficulty. A critical follow-up is difficulty-stratified evaluation: do CodePercept's gains concentrate on easy problems where the base model already has some perceptual capability, or do they extend to hard problems requiring fundamentally new visual understanding? The paper's solid geometry pipeline (Section 3.1) was motivated specifically by MLLM failure on a hard domain, suggesting the approach may be bounded by base model capability. A concrete experiment: split the STEM2Code-Eval 1,000-image benchmark by the base model's reconstruction score (using Qwen3-VL-8B-Instruct as the base), then measure CodePercept-8B-S1's improvement in each difficulty quartile. If gains are concentrated in easy/medium quartiles and near-zero in the hardest quartile, it would establish a capability boundary: code-grounded training refines existing perception but does not create new perceptual capabilities, consistent with the paper's implicit assumption that the data pipeline requires an MLLM capable of approximate image-to-code generation. If gains are uniform across difficulty (or even larger on hard problems), it would suggest the training teaches general perceptual skills that transfer to novel visual challenges, making the approach more broadly applicable.

Practical Applications and Downstream Use Cases

On-device STEM tutoring with small models. The finding that CodePercept-8B-S1 achieves 63.32% average across six STEM benchmarks (Table 1)—surpassing Qwen2.5-VL-72B's 57.09% despite being 9× smaller—enables a concrete deployment architecture: a CodePercept-trained 8B model running on-device (e.g., tablet, laptop with integrated GPU) can provide perception quality matching datacenter-scale 72B models on STEM visual reasoning tasks. For educational technology companies deploying STEM tutoring assistants, this translates to reduced serving costs (no API calls for perception), lower latency (on-device inference vs. network round-trip), and offline capability (students without internet access). The per-benchmark breakdown in Table 1 shows that CodePercept-8B-S1's advantage over Qwen2.5-VL-72B is largest on MathVision (59.31% vs. 54.14%) and LogicVista (61.52% vs. 52.34%)—precisely the benchmarks requiring precise spatial and logical visual understanding, which are core to geometry and logic tutoring. A tutoring system using CodePercept-8B would correctly perceive student-drawn diagrams, textbook figures, and interactive geometry constructions with accuracy exceeding the previous best open-source 72B model, all running locally.

Automated STEM content digitization and accessibility. The image-to-code translation capability taught by CodePercept (Section 3.3) has a direct downstream application: converting raster STEM images (scanned textbook figures, handwritten diagrams, research paper illustrations) into structured, editable code representations. The STEM2Code-Eval benchmark demonstrates that CodePercept-32B-R1 achieves 75.80 Avg. on image reconstruction, with an Exec Rate of 95.90% (Table 2)—meaning it can generate executable, faithful reconstruction code for the vast majority of STEM images it encounters. This enables several practical workflows: (1) accessibility tools that convert textbook diagrams into tactile graphics descriptions for visually impaired students, where precise element counts and spatial relationships (exactly what code specifies) are essential; (2) automated figure regeneration for responsive STEM content that adapts to different screen sizes and formats; (3) metadata extraction from scientific figures, where code-based descriptions provide structured, queryable representations of figure content (exact element counts, spatial relationships, color specifications) that natural language captions cannot. The execution-based verification means the generated code can be automatically validated—if it renders correctly, the digitization is successful; if not, a human reviewer is flagged. This is a concrete improvement over OCR-based or caption-based digitization approaches, which produce unverifiable natural language descriptions with unknown accuracy.

Perception diagnostics for MLLM evaluation pipelines. STEM2Code-Eval provides a tool for decomposing MLLM failures that is immediately deployable in model evaluation pipelines. Current MLLM benchmarks report aggregate accuracy without distinguishing perception failures from reasoning failures. By adding STEM2Code-Eval to an evaluation suite, a model developer can identify whether their new model's performance gains come from better perception (higher Image Score), better code generation ability (higher Exec Rate), or both. The benchmark's three-metric structure (Image Score, Code Score, Exec Rate) provides a diagnostic decomposition that Table 2 demonstrates is informative: Qwen2.5-VL-72B-Instruct achieves high Exec Rate (86.30%) but low Image Score (32.82), revealing that it can write executable code but fails to perceive visual content accurately. A model developer seeing this pattern knows to invest in visual encoder improvements rather than code generation training. Conversely, a model with low Exec Rate but high Image Score would indicate good perception with poor code generation—directing investment toward code training rather than vision. The benchmark's construction from six diverse STEM sources (MathVision, MathVista, MathVerse, DynaMath, WeMath, LogicVista) provides domain coverage that makes these diagnostics broadly applicable. This is a practical use of the benchmark that does not require adopting CodePercept's training methodology—any MLLM can be evaluated on STEM2Code-Eval to diagnose its perceptual capabilities independently of code-grounded training.

Training data quality verification for STEM datasets. The code-grounded caption generation pipeline (Section 3.2) can be repurposed as a quality assurance tool for existing STEM vision-language datasets. Many datasets (including the public seed data used to construct ICC-1M) contain human-written or MLLM-generated captions with unknown factual accuracy. The code analysis + refinement process provides a method for detecting and correcting caption errors: for each image in a dataset, generate reconstruction code (using the Image Reproduction pipeline from Section 3.1), extract verified visual facts via the execution tracer, and compare against the existing caption. Discrepancies between the code-verified facts and the caption flag potential hallucinations or omissions. This is immediately applicable: a research group curating a STEM VQA dataset could run their images through the pipeline, identify captions with factual errors (wrong counts, misdescribed spatial relationships, missing elements), and either auto-correct them using the refinement step or flag them for human review. The paper's Appendix C provides a concrete example of this diagnostic value: Gemini2.5-Pro's direct caption for a complex grid-and-arrow diagram misidentifies node counts, misdescribes spatial arrangements, and reports 68 arrows instead of the actual 63, while the code-grounded caption corrects all these errors. Running this verification on an entire dataset would quantify the hallucination rate of existing captions and provide corrected versions—a direct quality improvement for any STEM vision-language dataset.