RealChart2Code: Advancing Chart-to-Code Generation with Real Data and Multi-Task Evaluation

ArXiv: 2603.25804

🎯 Pitch

Models nearing 100% on existing chart-to-code benchmarks collapse to an average of 8.2/12 when faced with real-world data and multi-panel layouts. The new RealChart2Code benchmark reveals a 2x capability gap between proprietary and open-weight VLMs, with iterative refinement failures showing models break previously correct code while fixing errors.

---

1. Executive Summary

This paper introduces RealChart2Code, a new large-scale benchmark of over 2,800 instances grounded in authentic datasets from Kaggle, designed to systematically evaluate Vision-Language Models (VLMs) on chart-to-code generation across three tasks of escalating complexity: Chart Replication (reverse-engineering a visualization from an image alone), Chart Reproduction (generating plots from the image plus large-scale raw CSV data), and Chart Refinement (iteratively debugging flawed code through a multi-turn conversation with user feedback). Evaluating 14 leading VLMs—five proprietary and nine open-weight—the paper finds that models performing near-saturation on existing simpler benchmarks (e.g., Gemini-3-Pro-Preview averaging ~96% on ChartMimic) collapse dramatically on RealChart2Code, with the best proprietary model (Claude-4.5-Opus) achieving an average score of only 8.2 out of a possible 12 across the three tasks, while the top open-weight model (Qwen3-VL-235B) reaches just 3.6, exposing a capability gap of more than 2× between model tiers. The benchmark establishes that strong performance on simple, single-panel synthetic benchmarks does not transfer to complex, multi-panel visualizations derived from real-world data—proprietary models suffer predominantly from data mapping and layout reasoning failures, while open-weight models are plagued by syntax and execution errors—with iterative refinement revealing a "Regressive Editing" failure mode where models fix one error only by introducing new ones in previously correct code.

2. Context and Motivation

The Core Problem: Chart-to-Code Benchmarks Have Stagnated

The fundamental problem this paper addresses is that existing benchmarks for evaluating Vision-Language Models on chart-to-code generation have become too easy and too narrow. While the broader field of code generation has seen rapid advancement—with models tackling increasingly complex software engineering tasks (Jimenez et al., 2023), algorithmic problem-solving (Chen et al., 2021a), and even front-end design (Xu et al., 2025)—the specific sub-domain of chart-to-code generation remains bottlenecked by evaluation suites that test only the most basic capabilities.

This matters because chart-to-code generation is not a toy problem. The paper articulates a compelling real-world use case in Section 1: users frequently encounter compelling visualizations—in presentations, reports, or published dashboards—where the underlying source code is unavailable, yet they need to edit, extend, or adapt the chart for their own purposes. Reverse-engineering the visualization code from a static image is the natural workflow, and as VLMs become more integrated into data science tooling, their ability to perform this task reliably moves from "nice to have" to "critical infrastructure."

However, the benchmarks available to the research community before this paper suffered from three interrelated deficiencies that made them poor proxies for real-world capability:

1. They test on synthetic or simplified data rather than authentic datasets. Prior benchmarks like Plot2Code (Wu et al., 2024) and ChartMimic (Yang et al., 2025) construct their evaluation instances using synthetic data or by replicating figures from scientific papers. While this provides a controlled testbed, it sidesteps the messy reality of real data: irregular sampling intervals, missing values, heterogenous column types, and massive scale. A model that can perfectly plot a clean 50-row table may completely fail when faced with a 10-million-row CSV with nested categories and datetime parsing requirements.

2. They focus on simple, single-panel charts. As the paper explicitly states in Section 2.2, existing benchmarks "predominantly feature simple, single-panel plots, which are insufficient for evaluating an LLM's ability to handle complex layouts and high information density." Real-world data visualization rarely produces isolated bar charts; it produces dashboards with multiple coordinated subplots, dual-axis overlays, inset zoom regions, and hierarchical grid layouts (as illustrated in Figures 1, 8, 9, 10, and 11). A benchmark that never tests whether a model can orchestrate a 2×3 GridSpec with shared axes and a colorbar spanning two rows is not testing anything close to real-world competence.

3. They lack evaluation of iterative refinement. The paper points out a crucial omission: "they lack metrics for evaluating a model's ability to refine code in multi-turn conversation" (Section 1). Coding in practice is rarely one-shot; it involves debugging, responding to feedback, and incrementally improving an initial draft. Existing chart-to-code benchmarks treat the task as a single-turn generation problem, providing no signal about how models perform in the conversational workflows that modern AI-assisted development environments are designed around.

The Evidence of Saturation

The paper's own experiments on existing benchmarks demonstrate that this is not a hypothetical concern—it is an empirical reality. In Table 3, the top proprietary models reach near-saturation on both ChartMimic and Plot2Code:

• Gemini-3-Pro-Preview achieves 96.0% on ChartMimic and 96.8% on Plot2Code (scores normalized to 0–100 in Appendix E.1, Figure 7).
• Claude-4.5-Opus achieves 92.3% on ChartMimic and 95.1% on Plot2Code.
• Even the weaker proprietary models (GPT-5.1, Gemini-2.5-Flash) score in the 70–90% range.

Open-weight models also perform competitively on these benchmarks: Qwen3-VL-235B reaches 80.4% on ChartMimic and 81.8% on Plot2Code, while Intern-VL-3.5-241B hits 76.9% and 78.4%, respectively. A model scoring 80%+ on a benchmark designed to measure chart-to-code capability appears competent. But the paper's entire argument is that these numbers are misleading precisely because the benchmarks are too easy.

When the same models are evaluated on RealChart2Code (Table 2), the collapse is stark:

• Gemini-3-Pro-Preview drops from ~96% to an average task score of 8.1 out of 12 (~50.6% when linearly normalized to 0–100% in Appendix E.1), a roughly 2× performance degradation.
• Qwen3-VL-235B drops from ~85% to 3.6 out of 12 (~22.5% normalized), a roughly 3.8× degradation.
• DeepSeek-VL-7B drops from ~38% (average across existing benchmarks) to 0.7 out of 12 (~4.4% normalized), an 8.6× degradation.

The paper visualizes this in Figure 7 (Appendix E.1) by plotting existing-benchmark scores against RealChart2Code scores. All models fall substantially below the diagonal (y = x), with the distance from the diagonal being what the paper terms the "Complexity Gap." This gap is not noise—it is systematic evidence that existing benchmarks measure a fundamentally different, and much easier, capability than RealChart2Code does.

What Makes RealChart2Code Distinctly Harder

The paper identifies four dimensions that distinguish RealChart2Code from prior work, summarized in Table 1:

❶ Real data. While Plot2Code and ChartMimic use either pre-existing chart-code pairs from the internet (risk of data leakage) or synthetic data designed to mirror scientific figures, RealChart2Code is grounded in authentic datasets from Kaggle. The curation pipeline (Section 3.3, Appendix B.2) starts with over 8,000 candidate datasets containing more than 100,000 files and 30 billion rows. After rigorous two-phase filtering (automated screening by community metrics, then expert manual review), 1,036 high-quality datasets are selected, yielding 3,271 raw data files with approximately 860 million rows total. This massive scale means models cannot rely on pattern-matching to a handful of rows—they must implement correct data loading, filtering, aggregation, and transformation logic on datasets whose structure they have never seen.

The Chart Reproduction task makes this demand explicit: the model receives the raw CSV alongside the reference image and must write code that loads the correct columns, applies the correct transformations, and maps the resulting data to the correct visual elements. As detailed in Appendix C.1.2, the "Data Alignment" metric verifies this programmatically rather than visually—it checks computational equivalence of the data pipeline, not just whether the chart looks similar. This is a fundamentally harder evaluation than visual-only comparison because it catches errors that produce visually plausible but numerically incorrect results (e.g., using df['A'] instead of df['B'], or .sum() instead of .mean()).

❷ Complex chart structures. The paper's taxonomy (Appendix A.1) spans 7 high-level visualization intents (correlation, deviation, ranking, distribution, composition, change, groups) and 50 distinct chart types, from common types (line charts, scatter plots, bar charts) to specialized ones (parallel coordinates, Sankey diagrams, dendrograms, alluvial diagrams). Critically, the benchmark intentionally features composite and multi-panel layouts—a single task might require a 2×3 grid where one cell is a scatter plot with marginal distributions, another is a dual-axis line-and-bar chart, and a third contains a heatmap with a shared colorbar spanning multiple cells. The case studies in Appendix F vividly illustrate where current models break: Figures 8-11 show failures in GridSpec layout management, hierarchical composition of nested sub-layouts, global canvas scaling, and correct semantic decomposition of composite structures like joint plots.

❸ Interactive refinement via multi-turn conversation. The Chart Refinement task (864 instances) simulates a realistic debugging workflow: the model receives flawed code and an incorrect chart rendering, along with a natural language instruction to fix a specific issue (e.g., "the axis range is wrong," "the legend overlaps the plot," "the wrong column is mapped to the y-axis"). The model must diagnose the error, apply a targeted fix, and output corrected code. This tests capabilities that single-turn benchmarks never probe: whether the model can maintain global consistency while making local edits, whether it understands the causal relationship between code changes and visual changes, and whether it can follow precise user instructions without hallucinating unrelated modifications.

❹ Multi-task evaluation. By testing the same underlying visualization scenario across three task variants (replicate from image only, reproduce from image + data, refine from flawed code), the benchmark enables analysis of which specific sub-capability is failing. A model might perform well on Chart Replication (strong visual-to-code translation) but poorly on Chart Reproduction (can't handle large data), or vice versa. This disaggregation is essential for guiding model development, as opposed to a single aggregate score that conflates multiple failure modes.

How Existing Approaches Fall Short

The paper identifies specific limitations in prior benchmarks along multiple axes:

Plot2Code (Wu et al., 2024): This benchmark evaluates chart-to-code generation from scientific plots, but is limited to relatively simple chart types and single-panel layouts. The paper's evaluation in Table 3 shows that models can nearly saturate its metrics (97–100% pass rates for top proprietary models), indicating that its difficulty ceiling is too low to differentiate among the latest generation of VLMs.

ChartMimic (Yang et al., 2025): This benchmark tests cross-modal reasoning via chart-to-code generation but similarly focuses on replicating individual charts from papers. The paper's evaluation shows similar saturation effects, with top models scoring in the 90s. Neither benchmark incorporates real-world data at scale, multi-panel composite layouts, or interactive refinement.

Implicit data leakage concern: The paper notes that prior benchmarks often rely on "pre-existing chart-code pairs from the internet" (Section 1). While not elaborated extensively, this raises the concern that models may have memorized chart-code correspondences from their training data rather than genuinely learning to perform visual-to-code translation. RealChart2Code mitigates this by curating entirely new visualization tasks from Kaggle datasets that are unlikely to appear in standard training corpora in chart-code-pair form.

Text-to-Visualization (Text2Vis) vs. Chart-to-Code: The paper carefully distinguishes its focus from related but distinct tasks (Section 2.2). Text2Vis benchmarks (e.g., NVBench 2.0, Luo et al., 2025) test generation from natural language descriptions, while Chart-to-Code requires visual perception and reverse-engineering. The latter demands that models "translate pixel-level visual information (layout, styling, color mapping) into executable matplotlib code with high fidelity" (Appendix B.1), a capability that Text2Vis benchmarks do not assess.

How This Paper Positions Itself

The paper frames its contribution not as proposing a new model or training method, but as building the evaluation infrastructure that the field needs to make genuine progress. This is a benchmark paper in the classic sense: its primary argument is that by providing a more difficult, more realistic, and more diagnostic test set than currently exists, it will (a) reveal limitations that were previously invisible under saturated benchmarks, (b) provide fine-grained diagnostic signals about where models fail and why, and (c) guide future research toward the sub-capabilities that most need improvement.

The positioning is explicitly critical of the field's current evaluation standards while being constructive about what should replace them. The paper does not claim that existing benchmarks are useless—they served their purpose when models were less capable—but rather that "with the rapid advancement of LLMs, such benchmarks are no longer sufficient for evaluating a model's ability to handle chart-to-code tasks involving complex, real-world data and intricate plot structures" (Section 1). The four key distinctions enumerated in Table 1 (real data, complex structures, interactive refinement, multi-task design) are presented as the minimum viable upgrade needed to keep pace with model capability growth.

Importantly, this paper is not proposing agentic architectures, fine-tuning recipes, or novel prompting strategies to solve the chart-to-code task. It is demonstrating, through rigorous evaluation of 14 models across 2,896 instances, that even state-of-the-art VLMs fall far short when tested on realistic visualization tasks, and that their failure modes are systematic and analyzable. The implicit argument is that the research community should be alarmed by these results and should direct effort toward closing the Complexity Gap, not toward further optimizing performance on benchmarks that have already been saturated.

3. Technical Approach

This is primarily a benchmark construction and evaluation paper whose core idea is that existing chart-to-code benchmarks are too simple to differentiate modern VLMs, so the authors construct a more demanding benchmark grounded in real-world data and multi-turn interaction that exposes systematic capability gaps across model tiers.

3.1 Reader Orientation

The paper constructs the RealChart2Code benchmark—a carefully curated dataset of 2,896 chart-to-code instances paired with an automated multi-agent evaluation framework and an execution sandbox—and then uses it to measure 14 Vision-Language Models. The core problem is that prior benchmarks are saturated: top models score above 90%, leaving no room to see which capabilities are actually missing. The solution takes the shape of a three-task framework where each task probes a different sub-capability (visual perception, data integration, iterative debugging), using complexity and real-world grounding to create headroom for diagnostic measurement.

3.2 Big-Picture Architecture

The system has four major components:

1. Data Curation Pipeline — a four-stage process that selects real-world datasets, designs complex visualization tasks, implements ground-truth code, and injects errors for refinement tasks. Its output is the benchmark instances themselves: images, raw data files, prompts, and reference code.
2. Execution Sandbox — a Docker-containerized Python 3.13 environment with fixed library versions where model-generated code is executed under timeout and resource limits. It produces either execution errors (auto-scored as zero) or rendered chart images for evaluation.
3. Multi-Agent Evaluation Framework — a panel of LLM judges that vote on generated charts across eight visual accuracy criteria and three qualitative design criteria. It consumes generated chart images and reference images, and produces score vectors per instance.
4. 14 Evaluated VLMs — the models under test (5 proprietary, 9 open-weight), queried via standardized API calls with greedy decoding. They consume prompts (image + text) and produce executable Python code.

Information flows as follows: a curated benchmark instance (image, optional raw data, optional flawed code + instruction) is fed to a VLM via its API → the VLM generates Python code → code executes in the sandbox → if execution fails, score = 0; if execution succeeds, the rendered chart is passed to the multi-agent judge → the judge compares the generated chart to the reference across 8 + 3 criteria → final scores are aggregated and analyzed.

3.3 Roadmap for the Deep Dive

• First, the data curation pipeline (Section 3.3 of the paper), since the benchmark's quality and difficulty are entirely determined by how instances are created—understanding the curation explains why models fail.
• Second, the three task definitions and their prompts (Section 3.1), because the task structure determines what each VLM is asked to do and what information it receives.
• Third, the evaluation metrics (Section 3.4, Appendix C.1), including both the functional pass/fail check and the multi-agent scoring rubric across eleven dimensions, since these define what "good performance" means.
• Fourth, the execution sandbox and inference setup (Section 4.1, Appendix A.2), since reproducibility and safety constraints shape what code can actually run.
• Fifth, the 14 models under test (Section 4.1, Appendix C.2), to understand what is being compared and why the selection matters.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a benchmark construction and evaluation paper whose core idea is that chart-to-code evaluation must move beyond simple single-panel figures on synthetic data to complex multi-panel layouts grounded in authentic large-scale datasets, and must include multi-turn refinement evaluation, to meaningfully differentiate model capabilities at current performance levels.

---

Data Curation: The Four-Stage Pipeline

The entire benchmark rests on the quality of its instances. The construction follows four sequential stages (Section 3.3, Appendix B), with quality control interlocks at each transition.

Stage 1: Data Collection and Filtering

The paper starts with Kaggle as the source because it provides open-access datasets spanning diverse real-world domains with community quality signals. The raw pool consists of over 8,000 candidate datasets containing more than 100,000 files and approximately 30 billion data rows. This is intentionally massive because the authors need statistical power to filter down to a high-quality subset.

The first phase of filtering is automated large-scale screening based on community engagement metrics. The paper states the criteria explicitly: datasets are "ranked and filtered according to vote counts, download frequency, and official usability ratings" (Appendix B.2). The rationale is that these metrics proxy for dataset quality—highly-voted, frequently-downloaded datasets with good usability scores are likely to be well-documented, correctly formatted, and substantively interesting, while datasets with poor community signals are often incomplete or trivial.

The second phase is expert curation performed by the paper's team. From the pre-screened pool, human reviewers manually select datasets suitable for complex visualization tasks using two criteria: data richness (do the datasets contain sufficient columns—numeric, categorical, temporal—to support multi-dimensional plotting?) and domain diversity (do the datasets cover different thematic areas to avoid overfitting to a single visualization style?). The manual review ensures that automated metrics do not miss subtle quality issues—for example, a dataset might have high download counts because it is used in a popular tutorial, not because it is well-suited for complex visualizations.

The final curated collection contains 1,036 high-quality datasets, comprising 3,271 raw data files with approximately 860 million rows total. The paper emphasizes in Section 3.2 that these datasets cover eight high-level domains (Finance, Industry, Health, Research, Society, Media, Technology, Environment) and 35 fine-grained sub-topics, with the diversity verified through CLIP embedding visualization and t-SNE projection (Figure 3a, 3b)—the embeddings are widely dispersed, confirming that the datasets span heterogeneous visual styles and data distributions.

A critical design choice: the paper uses only open-source Kaggle datasets for "strictly scientific research" (Appendix B.2), explicitly deferring to original licenses for other uses. This is standard practice but imposes a practical constraint—all benchmark data is publicly available, which means future models might train on it. The paper does not discuss contamination mitigation beyond noting that prior benchmarks relying on internet chart-code pairs carry leakage risk.

Stage 2: Visualization Task Design

Using the 1,036 curated datasets, the authors design 1,016 unique visualization scenarios. The term "scenario" here means a specific, contextually meaningful visualization that answers an analytical question about the dataset—not an arbitrary or decorative plot. For example, given a dataset about global temperature anomalies, a scenario might be "show the warming trend by continent with uncertainty bands, using a faceted layout."

Each of these 1,016 visualizations serves double duty: it becomes both a Chart Replication instance (model receives only the rendered image) and a Chart Reproduction instance (model receives the image plus the raw data). This yields 2,032 instances across the two single-turn tasks (1,016 for each, since the same visualization is tested under two different information conditions).

The design process is guided by a taxonomy of 7 high-level visualization intents and 50 distinct chart types (Appendix B.3, A.1). The seven intents are deliberate categories chosen to span the analytical purposes that real-world visualizations serve:

• Correlation — scatter plots, bubble charts, pair plots, heatmaps showing relationships between variables
• Deviation — diverging bar charts, area charts highlighting variations from a baseline
• Ranking — ordered bar charts, lollipop charts, slope charts displaying comparative rankings
• Distribution — histograms, box plots, violin plots, density plots analyzing data spread
• Composition — stacked bar charts, treemaps, pie charts showing part-to-whole relationships
• Change — line charts, time series decompositions, area charts visualizing trends over time
• Groups — cluster plots, parallel coordinates revealing cluster structures within data

The 50 chart types enumerated in Table 5 (Appendix A.1) span a wide frequency distribution. Common types appear more frequently—line charts (845 instances), scatter plots (782), bar charts (723)—reflecting their prevalence in real-world usage. Specialized types appear less frequently—polar charts (12), stripplots (9), ternary plots (18)—but their inclusion is crucial because a benchmark that never tests dendrograms or Sankey diagrams cannot claim to measure general chart-to-code capability. The paper explicitly states that each unique visualization task is counted only once in Table 5, preventing inflated counts from the multi-task structure.

The crucial design requirement: all visualizations are intentionally complex, featuring "composite charts or intricate multi-panel layouts" (Section 3.2). A single instance might combine multiple chart types and multiple subplots—for example, a dashboard with a 2×2 grid where one cell is a scatter plot with marginal histograms, another is a dual-axis time series with bar overlays, a third is a grouped bar chart, and the fourth is a heatmap. This compositional complexity is what creates the "Complexity Gap" that the paper's results document: models that can handle isolated bar charts collapse when they must orchestrate heterogeneous chart types across a coordinated layout.

The paper does not detail the exact process by which humans designed these 1,016 scenarios—whether it was a systematic enumeration over the taxonomy or a more organic design process—but it does specify that each visualization was designed to be "contextually relevant to its source dataset, ensuring practical, real-world meaning" (Section 3.3). This constraint prevents the degenerate case where a visualization is arbitrarily complex but meaningless—the complexity must arise from genuine analytical needs.

Stage 3: Ground-Truth Code Implementation

For each of the 1,016 visualizations, the paper's "in-house team of five expert Python developers manually implemented the ground-truth code using Matplotlib and its associated libraries" (Section 3.3). This is a deliberate and expensive choice: the authors explicitly reject using model-generated code as ground truth, even if it might save labor. The concern is that model-generated code would bake in systematic errors or stylistic biases that would then contaminate evaluation—a model that happens to share the same biases as the ground-truth generator would score artificially high.

The implementation strictly uses matplotlib as the core plotting library, with standard data processing libraries (pandas, numpy). The paper provides a complete list of the sandbox pre-installed libraries in Appendix A.2: matplotlib, seaborn, plotly, squarify, scikit-learn, statsmodels, and others, all running on Python 3.13. This bounds the solution space: generated code must use only these libraries to execute successfully. The limitation to matplotlib is acknowledged as a scope constraint in Section 7 ("Limitations"), where the authors argue that "the granular and imperative nature of Matplotlib code effectively reflects a model's fundamental visualization understanding and logical reasoning capabilities, serving as a robust proxy for general plotting skills."

The ground-truth code must satisfy three properties (Appendix B.4):

1. Executable — runs error-free in the sandbox with no warnings or exceptions
2. Reproducible — accurately regenerates the target visualization from the provided data
3. Idiomatic — follows standard coding practices so the benchmark evaluates generation of maintainable code, not just code that happens to produce the right pixels

The quality control for ground-truth code (Appendix B.6) uses a cross-validation peer review workflow:

• Execution sanity check: all ground-truth scripts are run in the standardized sandbox. Failures, warnings, or timeouts trigger automatic flagging.
• Visual fidelity review: a secondary expert who did not author the code inspects each rendered chart against a three-point checklist: data accuracy (does the chart correctly represent the underlying raw data?), visual clarity (are labels, legends, and layouts legible and overlap-free?), and idiomatic coding (does the implementation follow standard matplotlib best practices?).
• Adjudication: flagged code is returned to the original author for correction. Disagreements are escalated to a senior lead for final decision to ensure consistency.

This three-stage peer review ensures that the ground truth is not merely one possible implementation but a high-quality reference that correctly, clearly, and idiomatically achieves the visualization goal.

Stage 4: Error Injection for Chart Refinement

The Chart Refinement task requires starting points that are deliberately flawed in specific, identifiable ways. The paper manually introduces errors into a subset of the ground-truth implementations to create 864 distinct refinement tasks (Section 3.3, Appendix B.5). This is not randomized corruption—the errors are designed to simulate real-world debugging scenarios and are categorized into three types:

• Visual/Stylistic Errors — overlapping elements, incorrect color schemes, missing legends, or illegible labels. These require the model to notice that something looks wrong and identify the code responsible.
• Data Mapping Errors — incorrect column assignments to axes, wrong aggregation methods, or incorrect data filtering. These are more subtle because the chart may look structurally correct but represent the wrong data relationships.
• Chart Type Errors — using a suboptimal or incorrect chart type for the given data (e.g., a bar chart for continuous time-series data). These test whether the model understands chart semantics, not just syntax.

For each refinement instance, the model receives a triplet of: (1) the flawed code, (2) the incorrectly rendered chart image, and (3) a natural language instruction specifying what to fix. The task is multi-turn: the model must produce corrected code that fixes the specified error without introducing new ones.

The quality control for refinement tasks (Appendix B.6) uses a triple-verification strategy:

• Triplet validation: each (flawed code/image, user instruction, corrected ground truth) triplet is reviewed by independent annotators.
• Logic and consistency check: annotators verify two conditions. First, error visibility—the injected error must be clearly distinguishable in the rendered image (an error that is invisible to human inspection cannot be fairly expected of a VLM to detect). Second, solvability—the user instruction must provide sufficient information for a model to deduce the correct fix without ambiguity (an instruction like "make it better" is too vague; "the legend overlaps the rightmost bar, move it to the upper left" is sufficiently specific).
• Majority vote and consensus: a task is accepted only if it receives a unanimous "Pass" vote from all reviewers. Partial agreement triggers a consensus discussion to polish the prompt. Tasks that fail to reach consensus are discarded to prevent ambiguity from adding noise to the evaluation.

This rigorous verification ensures that Chart Refinement failures reflect genuine model limitations rather than poorly specified tasks.

---

Task Definitions and Prompt Structure

The paper defines the chart-to-code problem formally as a conditional generation task (Section 3.1):

$C = \mathcal{F}(V, P)$

where $C$ is the generated executable code snippet, $V$ is the source chart image, $P$ is the accompanying textual prompt, and $\mathcal{F}$ is the VLM. $C$ must render a visualization that accurately reproduces the visual and structural elements of $V$ while adhering to any requirements in $P$.

What it computes: the VLM takes a multimodal input (image + text) and outputs a string containing Python code. The code's correctness is determined not by static analysis but by actually executing it and comparing the resulting rendered image to the reference.

Why this form: the conditional generation formulation makes explicit that the model must ground its code generation in visual perception—it is not free-generation from a text prompt, nor is it selecting from a predefined set of chart templates. This distinguishes chart-to-code from the related Text2Vis task described in Section 2.2, where only a natural language description is provided.

The three tasks instantiate this general formulation with different information conditions:

Task 1: Chart Replication. The model receives only the rendered chart image and a prompt that asks it to reproduce the visualization. The prompt does not provide the underlying data, axis labels, color codes, or layout specifications—the model must infer all of these from the pixels alone. This tests pure visual-to-code translation ability without any data management burden. The prompt template is provided in Appendix D (the paper's appendices are not fully reproduced in the main text, but the template is described as containing the image and instructions to generate Python code that replicates the chart).

Task 2: Chart Reproduction. The model receives the chart image, the raw CSV data file(s), and a prompt with metadata about the data structure. Crucially, the model must now write code that (a) loads the correct file, (b) selects the correct columns, (c) applies the correct transformations (filtering, aggregation, type conversion), and (d) maps the resulting data to the correct visual elements. The Data Alignment metric (Appendix C.1.2) verifies this programmatically by checking computational equivalence of the data pipeline, not just visual similarity of the output. This is the most demanding task because it combines visual perception with data engineering.

Task 3: Chart Refinement. This is a multi-turn conversational task. Turn 1: the model receives a flawed chart image and its corresponding flawed code, along with a user instruction describing what to fix. The model must produce corrected code. No additional turns are explicitly detailed in the paper's main evaluation, but the task definition states it "requires the model to correct a chart with predefined errors through a multi-turn dialogue" (Section 3.1). The paper's analysis in Section 5.2 identifies a failure mode called "Regressive Editing" where the model successfully applies the requested fix but inadvertently introduces new errors in previously correct parts of the code—this is inherently a multi-turn observation that could not be detected in single-shot evaluation.

All tasks use a standardized prompt format compliant with the ChatML structure (OpenAI, 2022). The evaluation uses greedy decoding (temperature = 0) across all models, with a maximum output token limit of 32,768 for models that support it. Models with lower context limits use their own maximum. Results are averaged over three independent runs (Section 4.1), providing some protection against stochastic variation even with greedy decoding (which can still exhibit non-determinism due to floating-point operations or infrastructure-level randomness).

---

Evaluation Metrics: Pass Rate and Multi-Agent Voting

The evaluation framework operates in two stages: functional correctness filtering, then detailed visual assessment. This two-stage design ensures that models cannot score points for code that looks plausible but cannot actually execute.

Stage 1: Pass Rate (Execution Filter)

The Pass Rate is defined as "the percentage of generated code snippets that execute successfully in our sandbox environment without errors" (Section 3.4). Submissions that fail execution—whether due to syntax errors, import errors, runtime exceptions, or timeout—are automatically assigned a score of zero across all metrics.

The paper emphasizes in Section 5.2 that Pass Rate is a major differentiator between model tiers: open-weight models frequently fail at this stage due to hallucinated library imports or invalid API calls (e.g., Qwen3-VL-235B hallucinates plt.style.use('seaborn-v0_11'), which does not exist), while proprietary models largely pass execution but fail downstream on visual accuracy. This means the Pass Rate is both a metric and a diagnostic: a low Pass Rate immediately identifies models with fundamental code correctness issues rather than subtle visual reasoning failures.

The execution environment is a Docker container running Python 3.13 with a fixed set of pre-installed libraries (Appendix A.2). Each execution runs without network access and is subject to a 120-second timeout. The server has 128 CPU cores and 1,024 GB of RAM, which is relevant mainly for ensuring that large-data tasks (where files can have millions of rows) can actually execute—the timeout, not the hardware, is the practical bottleneck.

Stage 2: Multi-Agent Voting on Visual Accuracy

For all code that passes execution, the rendered chart is evaluated by a multi-agent judging panel using a voting system (Section 3.4). Each chart is assessed on a 3-point scale—0 (fail), 1 (partial match), or 2 (exact match)—across eight visual accuracy criteria and three qualitative design criteria.

The eight visual accuracy criteria (detailed in Appendix C.1.1) are:

1. Chart Type Consistency — whether the generated chart uses the same chart types as the reference for all visualizations, including primary chart types and any overlaid secondary elements (dual-axis charts, combination charts). Score 2 requires exact matches across all subplots; Score 1 allows primary types to match but secondary elements to differ; Score 0 means any primary chart type is wrong.

2. Spatial Layout Consistency — whether the subplot grid dimensions, relative positioning, and element placement match the reference. Score 2 requires exact grid structure and subplot positioning within ±2% of reference dimensions; Score 1 allows minor positioning deviations (5–15%); Score 0 means wrong grid dimensions or fundamentally altered spatial relationships.

3. Text Element Consistency — whether all textual content (titles, subtitles, annotations, axis labels) matches in wording, placement, and hierarchy, excluding axis tick values. Score 2 requires exact match; Score 1 allows minor discrepancies in secondary elements (1–2 missing annotations, slightly reworded footnotes); Score 0 means missing or altered critical text.

4. Axis Configuration Consistency — whether axis properties (variable names, units, scale types, ranges, tick intervals) and legend specifications match. Score 2 requires perfect match; Score 1 allows minor deviations (±10–20% range extension, different but equivalent unit abbreviations); Score 0 means incorrect scale type or wrong variable assignments.

5. Color Scheme Consistency — whether color mappings for all data series, categories, and visual elements match. Score 2 requires identical mappings with exact hex/RGB matches or perceptually indistinguishable equivalents; Score 1 allows minor shade variations within 15% saturation; Score 0 means fundamentally different color palette or incorrect categorical assignments.

6. Style and Format Consistency — whether stylistic attributes (grid styles, marker shapes, line patterns, font families, background colors, border styles) match. Score 2 requires all attributes to match; Score 1 allows minor deviations (grid style slightly different, marker shapes similar but not exact); Score 0 means multiple style mismatches.

7. Component Completeness — whether all visual components from the reference are present, including data series, markers, annotations, grid elements, legends, statistical overlays (trendlines, confidence intervals), and decorative features. Score 2 requires all components; Score 1 allows missing 1–3 minor elements; Score 0 means missing critical components.

8. Data Pattern Consistency — whether data patterns (point positions, trend shapes, distribution profiles, statistical markers) visually replicate the reference. Score 2 requires >95% positional accuracy; Score 1 allows core trends to be recognizable with 5–15% positional variance; Score 0 means significantly divergent patterns.

For the Chart Reproduction task specifically, the Data Pattern Consistency metric is replaced by Data Alignment, which performs code-level verification rather than visual comparison. The scoring rubric (Appendix C.1.2) specifies three sub-checks:

• Data Source Matching: both codes must load data from the same file(s), and column selections must access the same fields, even if using different syntax (df['col'] is equivalent to df.col is equivalent to df.loc[:, 'col']).
• Data Transformation Equivalence: for each subplot, the final data arrays passed to plotting functions must be computationally equivalent. The rubric provides explicit equivalence rules: different groupby syntax producing the same result is accepted; different filtering methods producing the same subset are accepted; reordered operations that do not affect output are accepted. Conversely, the rubric rejects different columns, different aggregation types, different filter thresholds, or missing/extra transformations.
• Visual Element Data Mapping: for each subplot, corresponding visual elements must use equivalent data arrays.

Score 2 requires "all data transformations are computationally equivalent, and all visual elements map to identical data arrays." Score 1 allows "core data transformations correct with minor acceptable variations." Score 0 means "data transformations produce different results."

Why this programmatic approach matters: visual comparison alone might give a passing score to a chart that looks similar but represents the wrong data—for example, a model might correctly render a bar chart structure but map the wrong column to the y-axis, producing bars that look like a reasonable distribution but are numerically incorrect. The Data Alignment check catches errors that visual-only metrics miss, which is precisely the type of error the paper finds prevalent in proprietary models (Section 5.2: "proprietary model failures are predominantly Data Mapping Errors").

The three qualitative design criteria, scored on the same 0/1/2 scale, are:

• Visual Clarity — whether the chart is legible, well-spaced, and free of clutter
• Compositional Balance — whether visual weight is appropriately distributed across the canvas
• Typographic Quality — whether fonts, text sizes, and label placements are appropriate and readable

The paper does not provide detailed rubrics for these three criteria at the level of the eight visual accuracy metrics; they are mentioned in Section 3.4 and appear in Figure 4's radial charts but are described more briefly.

The multi-agent judging mechanism: the paper states that a "multi-agent judging panel that uses a voting system" evaluates each chart (Section 3.4), but does not specify the exact number of judges or the voting aggregation method in the main text. However, Section 5.1 provides reliability metrics: Fleiss' κ is computed across the entire benchmark for inter-agent agreement (Table 4), with an average κ of 0.82. Cohen's κ between the agent panel and human experts is computed on 600 tasks sampled from Claude-4.5-Sonnet results, yielding an average of 0.83. These reliability scores indicate that the judges are not a single LLM call but multiple agents whose votes are aggregated—the Fleiss' κ specifically measures agreement among three or more raters, confirming at least three agents are involved.

The aggregated score: the paper reports a single "Score" per task (Table 2), which is the sum of per-metric scores. With eight visual accuracy metrics each scored 0/1/2, the maximum possible score per instance is 16. The paper does not explicitly state whether the three qualitative metrics are included in this total—Figure 4 shows them as separate dimensions in the radial charts—but Table 2's maximum values (e.g., Claude-4.5-Opus scoring 7.8 on Chart Replication out of a maximum implied by the scale) suggest a 0–12 range. This is consistent with averaging the eight visual metrics (max 16) or some other normalization; the paper does not fully specify the aggregation formula. The confusion is a minor documentation gap.

The reliability analysis (Table 4, Figure 5) validates that the automated judge is not merely arbitrary. The inter-agent Fleiss' κ ranges from 0.781 (Completeness) to 0.998 (Layout), indicating high internal consistency—the judges agree with each other strongly. The agent-human Cohen's κ ranges from 0.72 (Color) to 0.99 (Layout), with an average of 0.83, indicating that the automated evaluation aligns with what human experts would judge. Figure 5 shows score distributions with 95% confidence intervals for Claude-4.5-Sonnet and Gemini-3.0-Pro-Preview, demonstrating that the judge's scores are both discriminatory (the distributions are distinct between models) and precise (the confidence intervals are narrow).

Execution Sandbox and Inference Setup

The execution environment (Appendix A.2) is designed to provide determinism, isolation, and safety. The Docker container runs Python 3.13 with a fixed set of pre-installed libraries: pandas, numpy, matplotlib, seaborn, plotly, squarify, scikit-learn, statsmodels. Network access is disabled, preventing models from downloading additional libraries or accessing external APIs during execution. Each code execution is subject to a 120-second timeout. The hardware is specified as "a server with 128 CPU cores and 1024 GB of RAM," ensuring that even large-data tasks (the benchmark contains files with approximately 860 million rows total) can complete within the timeout.

All model inference uses the standard OpenAI API format with ChatML compliance (Section 4.1). Greedy decoding (temperature = 0) is used across all models to maximize reproducibility and eliminate the stochastic variation that would complicate failure analysis. The maximum output token limit is set to 32,768; models with lower native limits use their own maximum. Proprietary models are queried via official APIs, while open-weight models are served using the SGLang framework. Each reported result is the average of three independent runs.

Models Under Test

The paper evaluates 14 VLMs (Section 4.1, Appendix C.2, Table 6) chosen to span the capability spectrum. The selection includes 5 proprietary models and 9 open-weight models with parameter sizes ranging from 7B to 241B.

Proprietary models:

• Claude-4.5-Sonnet and Claude-4.5-Opus (Anthropic, released September and November 2025)
• GPT-5.1 (OpenAI, released November 2025)
• Gemini-2.5-Flash and Gemini-3-Pro-Preview (Google, released June and November 2025)

Open-weight models:

• Qwen3-VL-30B and Qwen3-VL-235B (October 2025)
• DeepSeek-VL-7B (March 2024)
• Intern-VL-3.5-30B and Intern-VL-3.5-241B (August 2025)
• GLM-4.1V-9B and GLM-4.5V-106B (July 2025)
• MiMo-VL-7B-RL (August 2025)
• ChartCoder (January 2025)

The inclusion of ChartCoder is notable: it is the only model in the evaluation specifically designed for chart-to-code generation (Zhao et al., 2025). Its performance—scoring 3.2 average across the three tasks (Table 2), below even some general-purpose open-weight models at larger scales—demonstrates that task-specific fine-tuning on simpler benchmarks does not transfer to the complexity of RealChart2Code.

The paper does not describe different prompt engineering strategies per model, suggesting a uniform prompt was used. This is a deliberate choice for fair comparison—customizing prompts per model would optimize for benchmark performance rather than measuring out-of-the-box capability—but it means the results should be interpreted as evaluating default model behavior, not what each model could achieve with optimal prompting.

The Formal Task Definition

The paper's core formulation from Section 3.1:

$C = \mathcal{F}(V, P)$

where $C$ is the generated code, $V$ is the chart image, $P$ is the text prompt, and $\mathcal{F}$ is the VLM.

What it computes: a Python source code string that, when executed, renders a visualization reproducing $V$. The function $\mathcal{F}$ is a black-box neural model; the paper makes no assumptions about its internal architecture beyond it being multimodal (accepting image and text input).

Why this form: this is the simplest formulation that captures the chart-to-code task. It does not decompose the problem into sub-steps (e.g., first extract chart structure, then generate code), nor does it allow the model to interact with the execution environment. The paper deliberately tests end-to-end capability rather than component capability because real-world usage would involve a single model call, not a pipeline.

The three task variants differ in what $P$ contains:

• Chart Replication: $P$ is a fixed prompt with instructions to replicate the chart from the image only (no data provided)
• Chart Reproduction: $P$ includes the raw data files and metadata in addition to replication instructions
• Chart Refinement: $P$ includes the flawed code, the flawed rendered image, and the user correction instruction; the model's output $C$ is the corrected code

The uniform greedy decoding (temperature = 0) and standardized API format ensure that performance differences across models reflect genuine capability gaps rather than inference-time stochasticity or interface artifacts.

4. Key Insights and Innovations

Innovation 1: The "Complexity Gap" as a Diagnostic Concept

The paper's most intellectually distinctive contribution is not the benchmark itself but the diagnostic concept of the Complexity Gap—the systematic, measurable, and often enormous performance collapse that occurs when models transition from simple synthetic benchmarks to complex real-world tasks, even when the task category (chart-to-code) remains nominally the same. This is a far stronger claim than "our benchmark is harder." It is a claim about benchmark validity: existing evaluations were not simply measuring a weaker version of the same capability, but were measuring a fundamentally different, and much easier, capability whose mastery does not transfer to realistic settings.

Before this work, the field's implicit assumption—evident in the proliferation of chart-to-code benchmarks achieving 90%+ scores from top models (Plot2Code, ChartMimic)—was that strong performance on simple chart replication indicated general chart-to-code competence. The dominant evaluation paradigm was to test on the simplest instances that could still be considered charts and to report aggregate accuracy, implicitly treating the task as a unidimensional difficulty spectrum where scaling up model capability would monotonically improve performance. The paper's results shatter this assumption with empirical force: models that saturate existing benchmarks at 95%+ collapse to 50% or below on RealChart2Code, with the magnitude of collapse varying systematically across model tiers (Figure 7, Appendix E.1: proprietary models drop ~2×, open-weight models drop ~4×, older models drop ~8×).

The significance of the Complexity Gap as a concept extends beyond this specific benchmark. It provides a framework for evaluating benchmark adequacy that is portable to other domains: if a benchmark cannot produce a substantial Complexity Gap relative to simpler predecessors, it is not testing capabilities that go beyond what prior benchmarks already covered, and therefore adds little diagnostic value. The paper uses this implicitly to justify its own existence—the fact that the gap is large and consistent validates that RealChart2Code measures genuine capability dimensions absent from prior work—but the conceptual move is generalizable. Any new benchmark in any domain can now be evaluated by asking: "Does it produce a Complexity Gap, and what does that gap reveal about which sub-capabilities are missing?"

Critically, the Complexity Gap is not merely a gap in performance level (which would be expected—harder tasks should produce lower scores). It is a gap that is disproportionately severe for certain model tiers and certain sub-capabilities, enabling fine-grained diagnosis. The paper demonstrates this in Section 5.2: open-weight models collapse primarily because they cannot produce syntactically valid code (low Pass Rate), while proprietary models collapse primarily because they cannot correctly map data to visual elements despite producing syntactically flawless code. These are qualitatively different failure modes that a uniform performance drop would not reveal. The Complexity Gap thus functions as a lens for capability decomposition: by measuring the gap across model tiers and across task variants, the benchmark teases apart which capabilities are bottlenecking which models.

Innovation 2: Disaggregated Task Structure as a Capability Probe

The paper's three-task design—testing the same underlying visualization scenario under different information conditions (image only, image + raw data, flawed code + correction instruction)—is not merely a way to generate more instances. It is a diagnostic framework for isolating which specific sub-capability is failing in which model. This represents a conceptual shift from prior benchmarks, which treated chart-to-code as a monolithic capability measured by a single aggregate score.

Prior benchmarks (Plot2Code, ChartMimic) evaluated models on a single task variant. If a model scored poorly, the reason could be anything: weak visual perception, poor code generation, inability to handle data, or fragile spatial reasoning. The single score conflated all failure modes into one number, providing no signal about where to direct improvement efforts. This paper's disaggregation into three tasks—each probing a distinct capability axis—transforms evaluation from a pass/fail gate into a diagnostic instrument.

The diagnostic logic works as follows: Chart Replication tests pure visual-to-code translation (perception + code generation), with no data management burden. Chart Reproduction adds the data engineering dimension—the model must now load, filter, transform, and map large-scale real datasets correctly—so the performance gap between Replication and Reproduction isolates the data handling capability. Chart Refinement adds iterative debugging and instruction following—the model must maintain global consistency while making targeted local edits—so the gap between single-turn and multi-turn performance isolates the refinement capability.

This design enables specific, actionable findings that a monolithic benchmark could never produce. For example, the paper's observation that proprietary models perform relatively well on Replication but degrade on Reproduction (Table 2: Claude-4.5-Opus drops from 87.7 to 86.1, a modest decline, but Gemini-3-Pro-Preview drops from 82.1 to 68.7, a much steeper drop) localizes the failure to data handling specifically, not to visual perception generally. Similarly, the finding that Chart Refinement reveals "Regressive Editing"—where models fix one error only by introducing new ones—isolates a multi-turn consistency problem that single-turn evaluation completely masks. Without the disaggregated task structure, these would be invisible.

The conceptual contribution here is the design principle that evaluation tasks should be structured to create counterfactual contrasts: by varying one information condition while holding the visualization scenario constant, the benchmark makes the effect of that condition directly measurable. This is an application of controlled experimentation thinking to benchmark design, and it is a principle that future benchmarks in any multi-capability domain (code generation, reasoning, planning) could adopt.

Innovation 3: Programmatic Data Alignment Verification for a Visual Task

The decision to evaluate Chart Reproduction using programmatic data alignment checking rather than visual comparison is a subtle but profound innovation. In a benchmark whose primary output is visual, the natural, intuitive approach—and the one taken by prior work—is to compare rendered images: does the generated chart look like the reference? The paper argues that this approach systematically misses an entire class of errors: charts that appear visually correct but represent the wrong data, because the model used the wrong column, the wrong aggregation, or the wrong filtering logic.

The Data Alignment metric (Appendix C.1.2) performs code-level verification: it checks whether the data pipeline in the generated code produces computationally equivalent results to the reference implementation, even when using different syntax, different intermediate variable names, or differently ordered operations. This is conceptually novel because it recognizes that visual plausibility is not the same as correctness in data visualization. A bar chart that "looks like" the reference because it has the right number of bars and roughly the right proportions could still be catastrophically wrong if it represents total revenue instead of average revenue, or US data instead of global data. Visual-only metrics would give this a passing score; programmatic alignment catches it.

The significance of this innovation extends beyond this benchmark. It embodies a counterintuitive design principle: in a benchmark that appears to be evaluating visual generation capability, the most diagnostic evaluation dimension may be non-visual. The chart image is the output, but the correctness resides in the data transformation logic that produces it, and visual inspection is an insufficient proxy for that logic. This principle has implications for evaluating code generation in other visually-grounded domains (web design, UI generation, scientific figure reproduction) where outputs can "look right" while being wrong in ways that matter.

The empirical evidence for why this matters comes from the error analysis in Section 5.2: proprietary models, which pass execution at high rates and produce visually structured charts, predominantly fail on Data Mapping Errors. A benchmark that evaluated only visual similarity would report these models as highly competent; RealChart2Code's programmatic alignment reveals that they are not. The metric is what makes the Complexity Gap so large for proprietary models on Chart Reproduction, and it is what distinguishes RealChart2Code from benchmarks that treat chart generation as a purely visual task.

Innovation 4: Systematic Error Taxonomy Across Model Tiers

The paper's error analysis (Section 5.2, Figure 6) produces a structured taxonomy of failure modes that cleanly separates proprietary from open-weight models, revealing that the two tiers fail for fundamentally different reasons. This is an empirical finding with conceptual implications: it demonstrates that current evaluation practice—ranking models by a single score—is not merely imprecise but actively misleading, because models occupying adjacent positions on a leaderboard may have non-overlapping capability profiles.

Before this work, benchmark papers typically reported per-model accuracy and perhaps some qualitative failure cases, but did not systematically categorize errors across model tiers. The implicit assumption was that all models fail in broadly similar ways, just at different frequencies. This paper's error categorization into four types (Syntax and Execution Errors, Layout and Structural Failures, Data Mapping Errors, and Instruction Neglect) and its quantification of their distribution across models (Figure 6) reveals a stark division: open-weight models suffer heavily from Syntax and Execution Errors (hallucinated imports, invalid API calls) and Layout Failures (spatial reasoning breakdowns in multi-panel grids), while proprietary models have nearly zero syntax errors but struggle with Data Mapping (wrong columns, wrong aggregations) and Instruction Neglect in multi-turn settings.

The significance of this finding is that it reframes the capability gap between proprietary and open-weight models. The gap is not quantitative (open-weight models do everything worse to the same degree) but qualitative (they fail at entirely different stages of the pipeline). This means that strategies for improving open-weight models should not mimic whatever proprietary models are doing—they should focus on syntax correctness and spatial reasoning. Conversely, improving proprietary models requires better data mapping verification and multi-turn consistency, not better code generation. The error taxonomy provides a roadmap for targeted improvement that a single accuracy number could never offer.

The identification of "Regressive Editing" in the Chart Refinement task (Section 5.2) is a specific instance of this taxonomic approach yielding an otherwise-invisible insight. The phenomenon—models fix a specified error only to introduce a new error in previously correct code—is a multi-turn consistency failure that single-turn evaluation cannot detect. By creating a task specifically designed to expose it, and then analyzing failures through a systematic taxonomy, the paper identifies a failure mode that is likely widespread in conversational AI systems but rarely measured. This is a new diagnostic category that future work can explicitly target, evaluate, and attempt to mitigate.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The primary evaluation is on RealChart2Code itself, comprising 2,896 instances across three tasks: 1,016 for Chart Replication, 1,016 for Chart Reproduction, and 864 for Chart Refinement (Section 3.3). The benchmark is constructed from 1,036 high-quality Kaggle datasets spanning eight domains, with all instances featuring complex multi-panel or composite chart layouts. For cross-benchmark comparison, the paper also evaluates on Plot2Code (Wu et al., 2024) and ChartMimic (Yang et al., 2025), though the specific sizes of those benchmarks are not stated in the provided paper content beyond their being "existing state-of-the-art benchmarks" (Appendix E.1). The RealChart2Code test set uses the entire curated collection; there is no train/test split for the evaluation itself, as this is a benchmark for zero-shot/few-shot evaluation rather than fine-tuning, though a 600-task subset is sampled from Claude-4.5-Sonnet results for the human correlation study (Section 5.1).

• Base models. The paper evaluates 14 Vision-Language Models: 5 proprietary (Claude-4.5-Sonnet, Claude-4.5-Opus, Gemini-2.5-Flash, Gemini-3-Pro-Preview, GPT-5.1) and 9 open-weight (DeepSeek-VL-7B, Intern-VL-3.5-241B, Intern-VL-3.5-30B, Qwen3-VL-235B, Qwen3-VL-30B, GLM-4.5V-106B, GLM-4.1V-9B, MiMo-VL-7B-RL, and ChartCoder). The selection spans parameter sizes from 7B to 241B and includes both general-purpose VLMs and ChartCoder, a model specifically fine-tuned for chart-to-code generation (Zhao et al., 2025). The models were chosen to represent "the capability spectrum" (Section 4.1) across both proprietary and open-weight tiers, with model versions and release dates detailed in Appendix C.2 Table 6.

• Metrics. The evaluation uses a two-stage framework (Section 3.4). First, Pass Rate is computed as the percentage of generated code snippets that execute successfully in the sandbox without errors; failures automatically receive zero scores across all subsequent metrics. Second, for successfully executing code, a multi-agent voting panel evaluates rendered charts on a 0/1/2 scale across eight visual accuracy criteria (Chart Type Consistency, Spatial Layout Consistency, Text Element Consistency, Axis Configuration Consistency, Color Scheme Consistency, Style and Format Consistency, Component Completeness, and Data Pattern Consistency) and three qualitative design criteria (Visual Clarity, Compositional Balance, Typographic Quality). For Chart Reproduction specifically, Data Pattern Consistency is replaced by Data Alignment, which performs programmatic code-level verification checking computational equivalence of data transformations rather than visual comparison (Appendix C.1.2). The per-task Score reported in Table 2 is the sum of per-metric scores; the paper does not fully specify the aggregation formula for the qualitative metrics' inclusion, though results in Table 2 suggest a 0–12 maximum range. The AVG score is the simple average of scores across the three tasks.

• Baselines. The paper does not propose a new method, so there are no method baselines in the conventional sense. Instead, it compares model performance across three benchmarks: RealChart2Code (the proposed benchmark), Plot2Code (Wu et al., 2024), and ChartMimic (Yang et al., 2025). The two prior benchmarks serve as baselines that establish the pre-existing evaluation standard against which RealChart2Code's increased difficulty is measured. Within RealChart2Code, the three tasks themselves provide internal baselines: Chart Replication is the simplest condition and serves as a baseline for Chart Reproduction (which adds data handling) and Chart Refinement (which adds multi-turn debugging). Majority voting across the multi-agent judging panel functions as the evaluation baseline rather than a model baseline.

• Generation budget / compute accounting. The paper does not report generation budgets in the sense of sampling multiple candidates or conducting search. All inference uses greedy decoding (temperature = 0) with a single generation per instance, with results averaged over three independent runs to account for any infrastructure-level non-determinism (Section 4.1). The maximum output token limit is set to 32,768; models with lower native limits use their own maximum. The execution environment imposes a 120-second timeout per code execution, with each instance running in an isolated Docker container without network access. The server hardware (128 CPU cores, 1,024 GB RAM) is specified to ensure large-data tasks can complete within the timeout. There is no FLOPs accounting or compute-matching between models, as the comparison is purely capability-based rather than efficiency-based.

• Cross-validation / statistical protocol. The primary statistical validation is the reliability analysis in Section 5.1. Inter-agent agreement is assessed using Fleiss' κ computed across the entire RealChart2Code benchmark (all 2,896 instances), yielding an average of 0.82 across the eight visual accuracy metrics (Table 4). Agent-human agreement is assessed using Cohen's κ on 600 tasks sampled from Claude-4.5-Sonnet results, yielding an average of 0.83 (Table 4). Score distributions with 95% confidence intervals are reported for Claude-4.5-Sonnet and Gemini-3.0-Pro-Preview across all three tasks (Figure 5). For cross-benchmark analysis, all scores are normalized to a 0–100% scale for comparability (Appendix E.1, Figure 7). The paper reports that results are "an average of three independent runs" (Section 4.1) for the main results in Tables 2 and 3, though standard deviations or confidence intervals on these model-level scores are not reported except through the radial charts in Figure 4 and the distribution plots in Figure 5. No cross-validation is used for model selection, as no model training occurs.

Main Quantitative Results

RealChart2Code Benchmark Performance (Table 2)

The headline finding from Table 2 is that even the best model achieves substantially sub-saturation performance, with the top proprietary model (Claude-4.5-Opus) scoring 8.2 out of a maximum ~12 AVG score, and the top open-weight model (Qwen3-VL-235B) scoring only 3.6. The performance hierarchy within and across tiers is as follows:

Proprietary models (AVG score, Table 2):

• Claude-4.5-Opus: 8.2 (87.7 Chart Replication, 86.1 Chart Reproduction, 91.2 Chart Refinement)
• Gemini-3-Pro-Preview: 8.1 (82.1 Replication, 68.7 Reproduction, 83.6 Refinement)
• Claude-4.5-Sonnet: 7.0 (85.8 Replication, 73.7 Reproduction, 89.0 Refinement)
• GPT-5.1: 5.4 (71.2 Replication, 64.8 Reproduction, 73.0 Refinement)
• Gemini-2.5-Flash: 5.3 (63.2 Replication, 57.6 Reproduction, 69.0 Refinement)

The relative ordering shifts noticeably across tasks. Claude-4.5-Opus leads on Chart Reproduction (86.1) and Chart Refinement (91.2), but Gemini-3-Pro-Preview achieves the highest Chart Replication score at 9.0 (which maps to a higher Pass Rate than Claude-4.5-Opus's 7.8, despite Claude's higher raw Score of 87.7 — note that the relationship between Pass Rate and Score is not linear because Score incorporates multiple sub-metrics). GPT-5.1 substantially underperforms the other high-end proprietary models across all three tasks, with its AVG of 5.4 placing it closer to the open-weight tier than to the proprietary leaders.

Open-weight models (AVG score, Table 2):

• Qwen3-VL-235B: 3.6 (49.2 Replication, 38.2 Reproduction, 65.1 Refinement)
• Intern-VL-3.5-241B: 3.4 (54.3 Replication, 49.2 Reproduction, 60.2 Refinement)
• ChartCoder: 3.2 (48.0 Replication, 40.5 Reproduction, 54.7 Refinement)
• GLM-4.5V-106B: 2.3 (38.4 Replication, 39.1 Reproduction, 44.3 Refinement)
• Qwen3-VL-30B: 1.3 (17.3 Replication, 19.0 Reproduction, 45.2 Refinement)
• Intern-VL-3.5-30B: 1.2 (20.3 Replication, 15.3 Reproduction, 41.6 Refinement)
• GLM-4.1V-9B: 1.1 (12.8 Replication, 16.2 Reproduction, 39.9 Refinement)
• MiMo-VL-7B-RL: 0.8 (11.0 Replication, 10.7 Reproduction, 34.8 Refinement)
• DeepSeek-VL-7B: 0.7 (9.7 Replication, 5.9 Reproduction, 31.4 Refinement)

Several patterns are notable. First, the gap between the best proprietary model (8.2) and the best open-weight model (3.6) is a factor of ~2.3×, far larger than on existing benchmarks. Second, open-weight models show a consistent pattern of performing better on Chart Refinement than on the other two tasks—Qwen3-VL-30B jumps from 17.3 (Replication) to 45.2 (Refinement), and even DeepSeek-VL-7B reaches 31.4 on Refinement versus single digits on Replication and Reproduction. This is non-obvious and the paper does not fully explain it, but a plausible factor is that Refinement provides the model with working code as a starting point, reducing the demand for from-scratch code generation.

Third, model scale matters significantly within the open-weight tier: the 235B and 241B models substantially outperform their 30B and 7B counterparts, but the relationship is not purely monotonic—ChartCoder (a specialized model) outperforms the general-purpose GLM-4.5V-106B despite being presumably smaller (its exact parameter count is not stated in the provided paper content, but it scores between the 241B and 106B general models).

The Pass Rate column in Table 2 reveals the starkest tier division. Proprietary models achieve Pass Rates of 5.2–9.0% on Chart Replication (these appear to be raw percentages); open-weight models range from 0.4–3.5%. For Chart Reproduction, the proprietary range is 4.8–7.4%, open-weight is 0.3–2.5%. This confirms the error analysis in Section 5.2: open-weight models frequently fail at the execution stage due to syntax errors and hallucinated imports, while proprietary models pass execution but fail on visual accuracy.

Cross-Benchmark Comparison: The Complexity Gap (Table 3 vs. Table 2)

Table 3 presents model performance on the prior benchmarks ChartMimic and Plot2Code, using three evaluation types: Direct Mimic, Customized Mimic, and (for Plot2Code) Pass Rate and Score, plus a Text Rating for Direct Mimic. The saturation evidence is clear:

Top proprietary models on ChartMimic and Plot2Code (Table 3):

• Claude-4.5-Opus: 98.5 Pass / 92.3 Score (ChartMimic), 100.0 Pass / 95.1 Score (Plot2Code)
• Gemini-3-Pro-Preview: 97.3 Pass / 96.0 Score (ChartMimic), 100.0 Pass / 96.8 Score (Plot2Code)
• Claude-4.5-Sonnet: 100.0 Pass / 90.1 Score (ChartMimic), 99.5 Pass / 91.5 Score (Plot2Code)

Top open-weight models on ChartMimic and Plot2Code (Table 3):

• Qwen3-VL-235B: 91.5 Pass / 80.4 Score (ChartMimic), 92.7 Pass / 81.8 Score (Plot2Code)
• Intern-VL-3.5-241B: 90.5 Pass / 76.9 Score (ChartMimic), 90.7 Pass / 78.4 Score (Plot2Code)
• ChartCoder: 90.9 Pass / 75.3 Score (ChartMimic), 92.1 Pass / 76.7 Score (Plot2Code)

The cross-benchmark comparison (Appendix E.1, Figure 7) normalizes all scores to 0–100% and plots existing-benchmark averages against RealChart2Code averages. The results are:

"Gemini-3-Pro-Preview and Claude-4.5-Opus achieve near-perfect scores on existing tasks, averaging 96.0% and 91.2% respectively. However, their performance practically halves on RealChart2Code, dropping to 50.6% and 51.3%."

"Qwen3-VL-235B perform[s] admirably on standard benchmarks (averaging 84.7%), [but fails] to generalize to the rigor of real-world data science, scoring only 22.5% on RealChart2Code."

The Complexity Gap—the vertical distance from the y=x diagonal in Figure 7—is approximately 45 percentage points for proprietary models and 62 percentage points for the best open-weight models. This gap validates the paper's central claim that existing benchmarks are saturated and non-diagnostic for top-tier models.

Per-Metric Performance Breakdown (Figure 4)

Figure 4 provides radial charts breaking down scores across the eight visual accuracy metrics, Pass Rate, and three quality metrics for top models across all three tasks. The paper's text references these charts qualitatively rather than extracting specific numeric values, but the patterns described include:

• Claude-4.5-Opus and Gemini-3-Pro-Preview show relatively balanced performance across all visual accuracy metrics on Chart Refinement, with high scores on Text, Layout, and Type consistency.
• Chart Reproduction shows degraded performance on the Data Alignment metric (replacing Data Pattern Consistency for this task) relative to the visual accuracy metrics, consistent with the finding that proprietary models struggle with correct data mapping.
• Open-weight models show notably low scores on Layout and Component Completeness, reflecting their spatial reasoning limitations documented in the error analysis.

Error Analysis Results (Section 5.2, Figure 6)

Figure 6 presents the distribution of error types across models, though the paper describes the categories and patterns rather than extracting precise percentages for every model. The four error categories and their distribution across model tiers are:

Syntax and Execution Errors: Predominantly affect open-weight models. The paper states that these models "frequently hallucinate non-existent libraries or invoke invalid functions, leading to immediate code execution failures" (Section 5.2). A specific example is Qwen3-VL-235B's "recurring tendency to generate plausible-looking but functionally invalid API calls, affecting approximately 20% of the generated solutions," specifically plt.style.use('seaborn-v0_11'), which does not exist (Appendix F, Error Case 5). Proprietary models exhibit "robust coding capabilities with minimal syntax errors" (Section 5.2).

Layout and Structural Failures: Also predominantly affect open-weight models, manifesting as "overlapping subplots or incorrect grid definitions" (Section 5.2). The case studies in Appendix F provide concrete examples: Figure 8 shows a model failing to use matplotlib.gridspec for layout management, resulting in overcrowded subplots; Figure 9 shows a model unable to compose nested sub-layouts (scatter matrix, joint plot) within a larger dashboard grid; Figure 10 shows a model failing to balance figure size with element scaling, producing a thumbnail-sized plot in a large canvas; Figure 11 shows a model decomposing joint plots (scatter with marginal histograms) into disconnected independent subplots rather than recognizing them as integrated semantic units.

Data Mapping Errors: Predominantly affect proprietary models. These errors involve "the visual structure [being] correct, but specific data series [being] mapped to the wrong axes or visual attributes [not matching] the prompt requirements" (Section 5.2). This is consistent with the observation that proprietary models pass execution at high rates but underperform on Chart Reproduction relative to Chart Replication—the Data Alignment metric (Appendix C.1.2) catches these errors programmatically.

Instruction Neglect: The paper identifies this as a failure mode in the Chart Refinement task, where models fail to follow user correction instructions precisely, or introduce new errors while fixing old ones (the "Regressive Editing" phenomenon). The paper states: "When users request a specific modification, models often successfully apply the fix but inadvertently introduce new errors in previously correct parts of the code" (Section 5.2). This affects both proprietary and open-weight models but is more relevant to proprietary models since open-weight models often fail at earlier stages (execution or layout).

Human Evaluation and Metric Validation (Section 5.1, Table 4, Figure 5)

Table 4 reports Fleiss' κ (inter-agent agreement) and Cohen's κ (agent-human agreement) across the eight visual accuracy metrics. The inter-agent Fleiss' κ ranges from 0.781 (Component Completeness) to 0.998 (Spatial Layout), with an average of 0.82. The agent-human Cohen's κ (computed on 600 tasks from Claude-4.5-Sonnet) ranges from 0.72 (Color Scheme) to 0.99 (Spatial Layout), with an average of 0.83. These are strong agreement scores—conventionally, κ > 0.8 is considered "almost perfect" agreement, and κ between 0.6–0.8 is "substantial." The lower scores on Color (0.72 agent-human) and Completeness (0.74 agent-human) suggest these dimensions are more subjective, which is expected for aesthetic judgments like color nuance or what constitutes a "minor" missing component.

Figure 5 shows score distributions with 95% confidence intervals for Claude-4.5-Sonnet and Gemini-3.0-Pro-Preview across all three tasks. The paper states these show "distinct distributions combined with narrow intervals confirm that the judge provides a discriminatory and precise assessment of visualization quality" (Section 5.1).

Ablation Studies and Robustness Checks

This paper is a benchmark construction and evaluation paper, not a method paper, so it does not contain ablations in the conventional sense of removing or varying components of a proposed method. However, it does contain several robustness checks and cross-validation analyses that serve analogous purposes.

Oracle vs. automated evaluation: agent-human agreement (Section 5.1, Table 4). The multi-agent judge's alignment with human experts is validated on 600 sampled tasks, with an average Cohen's κ of 0.83 across the eight visual accuracy metrics. This confirms that the automated evaluation framework—which is what makes the benchmark scalable—produces judgments consistent with what human experts would assign, with the caveat that Color and Completeness show only "substantial" rather than "almost perfect" agreement (κ of 0.72 and 0.74, respectively). This is a critical validation because if the automated judge were poorly correlated with human judgment, the entire benchmark's validity would be undermined.

Inter-agent agreement across the full benchmark (Section 5.1, Table 4). Fleiss' κ computed across all 2,896 instances yields an average of 0.82, with all individual metrics exceeding 0.78. This demonstrates that the multi-agent voting system produces internally consistent scores—the variance attributable to which specific agents evaluate a given instance is low. The highest agreement is on Spatial Layout (0.998), which is the most geometrically objective dimension, and the lowest is on Component Completeness (0.781), which requires more subjective judgment about what constitutes a "minor" versus "critical" missing component.

Cross-benchmark performance normalization (Appendix E.1, Figure 7). To ensure that cross-benchmark comparisons are not distorted by different metric scales, the paper normalizes all scores to 0–100%. The resulting Complexity Gap shown in Figure 7—where all models fall substantially below the y=x diagonal—is robust to this normalization because the gap is not a function of scale choice but of genuine performance differences. The specific normalized scores (Gemini-3-Pro-Preview: ~96% existing → ~50.6% RealChart2Code; Qwen3-VL-235B: ~84.7% → ~22.5%) are derived from averaging normalized scores across ChartMimic and Plot2Code for the x-axis and across all three RealChart2Code tasks for the y-axis.

Three-run averaging for score stability (Section 4.1). All reported results are averages of three independent runs with greedy decoding. While greedy decoding should be largely deterministic, floating-point non-determinism in GPU operations or infrastructure-level randomness can introduce variation. The paper states results are averaged but does not report variance across runs—this is a minor documentation gap, though the narrow confidence intervals in Figure 5 suggest low run-to-run variance at least for those specific comparisons.

Sandbox reproducibility (Appendix A.2). The execution environment is fully containerized with fixed library versions on Python 3.13, with no network access and a 120-second timeout. This eliminates the possibility that model performance differences are attributable to environment variation—any model that can generate correct code under these constraints would generate correct code in any replication of the environment.

Task design verification via peer review and consensus (Appendix B.6). The ground-truth code, visualization tasks, and refinement error triplets all undergo multi-stage human verification with explicit pass/fail criteria and consensus-based adjudication. The refinement tasks specifically require unanimous "Pass" votes from independent annotators on both error visibility (can the error be seen in the rendered image?) and solvability (is the correction instruction unambiguous?). Tasks failing to reach consensus are discarded. This quality control ensures that benchmark failures reflect model limitations rather than poorly constructed tasks. However, the paper does not report how many tasks were discarded during this process, which would provide a quantitative measure of quality control stringency.

Critical Assessment

The paper makes one central claim: that RealChart2Code reveals a substantial and systematic capability gap in chart-to-code generation that prior benchmarks (Plot2Code, ChartMimic) completely miss, and that this gap is qualitatively different across model tiers. The experiments provide strong evidence for this claim, but there are important qualifications and gaps.

The Complexity Gap is convincingly demonstrated. The cross-benchmark comparison (Table 2 vs. Table 3, Figure 7) provides clear evidence that the same models performing at 90%+ on prior benchmarks collapse to 50% or below on RealChart2Code. This is not a subtle effect—it is a dramatic, consistent, and tier-differentiated performance drop. The evidence is strengthened by the fact that all 14 models show the gap (no exceptions or reversals), and the magnitude scales sensibly with model quality (smaller for newer/larger models, larger for older/smaller models). The claim that "strong performance on simpler benchmarks does not guarantee success on RealChart2Code" (Section 4.2, finding ❷) is well-supported.

But the complexity gap is a claim about benchmark validity, not about models. The experiments demonstrate that RealChart2Code is harder than prior benchmarks—this is unsurprising given the paper's explicit design goal of increased difficulty. What is less well-supported is the implicit claim that performance on RealChart2Code generalizes to real-world chart-to-code tasks more faithfully than performance on prior benchmarks. The benchmark is more realistic in its use of real data, complex layouts, and multi-turn refinement, but it is still a specific benchmark with specific design choices (Matplotlib only, specific chart types, specific data domains). Whether models that perform well on RealChart2Code would also perform well on genuinely novel real-world tasks (with unfamiliar libraries, chart types, or data structures) is not tested—and cannot be tested without an external validation set drawn from a different distribution, which the paper does not provide. This is a limitation shared by nearly all benchmarks, but it means the Complexity Gap should be interpreted as "RealChart2Code measures capabilities that existing benchmarks miss" rather than "RealChart2Code perfectly captures real-world capability."

The open-weight vs. proprietary divide is well-documented but the causal factors are underspecified. The paper's error analysis (Section 5.2, Figure 6) clearly shows that open-weight models fail primarily at syntax/execution and layout, while proprietary models fail at data mapping and multi-turn consistency. This is a genuine and important empirical finding. However, the paper does not attempt to tease apart why this divide exists. Is it model scale (proprietary models are larger, though their exact sizes are unknown)? Training data composition (proprietary models may have seen more code)? Architectural differences? Post-training (RLHF, instruction tuning)? The error taxonomy is descriptive but not explanatory—it tells us where models fail, not why they fail there. This is not a failure of the paper (attribution is a separate research question), but a reader should not interpret the error analysis as identifying root causes.

The ChartCoder comparison is a missed opportunity for deeper analysis. ChartCoder (Zhao et al., 2025) is a model specifically fine-tuned for chart-to-code generation, yet it underperforms general-purpose models at similar or larger scales (e.g., Qwen3-VL-235B). This is a striking finding that the paper largely mentions in passing. A deeper analysis of why task-specific fine-tuning on simpler benchmarks fails to transfer would have been highly informative. Does fine-tuning cause overfitting to simple chart types? Does it degrade the model's ability to handle real data? Does it create syntax-level regressions? The paper's scope does not include fine-tuning analysis, but the presence of ChartCoder in the evaluation creates an expectation of analysis that is not fully met.

The three-task structure enables diagnosis but is underutilized in the quantitative analysis. The paper's conceptual framework (Section 3.1) argues that the three tasks isolate different sub-capabilities: Replication tests visual-to-code translation, Reproduction tests data integration, and Refinement tests iterative debugging. However, the quantitative analysis of this diagnostic power is limited. The paper reports per-task scores (Table 2) and notes cases where models show different relative performance across tasks (e.g., Gemini-3-Pro-Preview leads Replication but falls behind Claude-4.5-Opus on Reproduction and Refinement), but it does not systematically analyze the Reproduction-minus-Replication gap or the Refinement-minus-Replication gap as diagnostic signals. For example, a model with a large Reproduction-minus-Replication gap (like Gemini-3-Pro-Preview: 82.1 → 68.7, a decline of 13.4 points in raw Score) has a specific weakness in data handling relative to visual perception, while a model with a small gap (like Claude-4.5-Opus: 87.7 → 86.1, a decline of only 1.6 points) has more balanced capabilities. Quantifying and reporting these gap scores systematically across all 14 models would have strengthened the diagnostic argument.

The human evaluation validates the judge but not the benchmark's coverage. The 600-task human correlation study (Section 5.1) shows strong agreement between the automated judge and human experts (average κ = 0.83). This validates that the judge measures what it claims to measure—visual fidelity to a reference. However, it does not validate that the benchmark's task distribution is representative of real-world chart-to-code tasks, or that the specific difficulty factors (multi-panel layouts, real data, refinement) are the most important ones. Human evaluation of benchmark coverage—asking domain experts whether the tasks represent realistic visualization challenges—is absent. This is typical for benchmark papers (coverage validation is expensive and subjective), but it means the benchmark's claim to evaluate "real-world" capability rests on face validity (the tasks come from real Kaggle datasets) rather than empirical validation that performance on the benchmark predicts performance in deployed settings.

The 20% hallucination rate for Qwen3-VL-235B (Appendix F, Error Case 5) is reported qualitatively but not quantitatively analyzed across all models. The paper states that Qwen3-VL-235B exhibits plt.style.use('seaborn-v0_11') hallucinations "affecting approximately 20% of the generated solutions," but does not report hallucination rates for other models or other specific hallucination patterns. A systematic analysis of library hallucination frequency across models would have been very informative, as it would quantify the extent to which execution failures (the dominant open-weight failure mode) are attributable to hallucinated versus genuinely incorrect syntax.

The three-run averaging has unclear necessity with greedy decoding. The paper averages results over three independent runs, but with temperature = 0 (greedy decoding), the outputs should be deterministic barring infrastructure-level non-determinism. The paper does not report variance across runs, so it is unclear whether the averaging is providing meaningful robustness or is merely conventional. If run-to-run variance is near zero (as the narrow confidence intervals in Figure 5 suggest), the averaging is unnecessary; if there is meaningful variance, it should be reported and explained.

Missing experiments that would have strengthened the paper:

• Difficulty bin analysis. Given that the paper argues complexity is the key differentiator, breaking down performance by chart complexity (number of subplots, number of chart types, data size, layout complexity) would have been highly informative. Does the performance gap grow monotonically with complexity, or are there thresholds where models suddenly fail? The paper's error cases hint at threshold effects (e.g., models handle individual subplots but fail at coordinated layouts), but no quantitative analysis across difficulty levels is provided.

• Prompt sensitivity analysis. All models are evaluated with the same prompt format. How sensitive are the results to prompt wording, including the amount of instruction detail, the specification of output format, or the inclusion of examples? The paper argues that greedy decoding with fixed prompts measures "out-of-the-box capability," but a few-shot or chain-of-thought baseline would contextualize how much of the performance gap is due to suboptimal prompting rather than fundamental capability limitations.

• Pass@k analysis. With greedy decoding, only a single attempt is evaluated per instance. A pass@k metric (e.g., pass@10 or pass@100 with non-zero temperature) would reveal whether models have latent capability that single-shot evaluation misses. This is particularly relevant for open-weight models struggling with execution—if they sometimes generate correct code but are inconsistent, pass@k would capture that latent capability, while the current single-shot evaluation treats inconsistency as outright failure.

Overall assessment: The experiments strongly support the paper's central claim that existing benchmarks are saturated and that RealChart2Code reveals a substantial, systematic, and tier-differentiated capability gap. The evidence is internally consistent across 14 models and three task variants, with the error analysis providing additional qualitative depth. The primary limitations are not in what the experiments show but in the scope of what they do not test: generalizability to truly out-of-distribution real-world tasks, root causes of the open-weight vs. proprietary divide, and the diagnostic potential of the three-task structure (which is stated as a design principle but not fully exploited in quantitative analysis). These limitations are consistent with the paper's scope as a benchmark introduction rather than a comprehensive capability analysis, but they bound what can be concluded from the reported results.

6. Limitations and Trade-offs

Matplotlib-Only Scope Restricts Generalization to Broader Visualization Ecosystems

The paper explicitly confines all ground-truth implementations and model-generated code to Matplotlib and its associated Python libraries (pandas, numpy, seaborn, plotly, squarify, scikit-learn, statsmodels). Section 7 acknowledges this directly:

"our code implementations are currently confined to Matplotlib. While this scope is specific, the granular and imperative nature of Matplotlib code effectively reflects a model's fundamental visualization understanding and logical reasoning capabilities, serving as a robust proxy for general plotting skills."

The consequence of this scope restriction is that the benchmark cannot detect whether models have learned Matplotlib-specific patterns rather than generalizable visualization reasoning. A model might excel at Matplotlib API calls while being incapable of generating equivalent visualizations in ggplot2 (R), Vega-Lite, D3.js, or declarative plotting libraries. The paper's argument that Matplotlib serves as a "robust proxy" for general plotting skill is an untested assumption—there is no cross-library evaluation to verify that Matplotlib performance correlates with performance in other visualization frameworks.

More subtly, the choice of Matplotlib constrains what kinds of charts are evaluated. Matplotlib's imperative API requires explicit specification of every visual element, which tests a model's ability to decompose a visual scene into low-level drawing commands. Declarative visualization libraries (e.g., Vega-Lite, Altair) operate at a higher level of abstraction, where the model specifies data mappings and the library handles layout and styling automatically. A benchmark built on declarative libraries would test different capabilities (data-to-visual mapping specification) than one built on imperative libraries (pixel-level layout control). The paper's results therefore measure capability on imperative chart generation, not chart generation in general.

The evidence in the paper is entirely within the Matplotlib ecosystem—there are no experiments with alternative libraries, no cross-library comparisons, and no analysis of whether the error patterns observed (layout failures, data mapping errors) are Matplotlib-specific or library-agnostic. The mitigation status is that the paper acknowledges the limitation in Section 7 and makes the "robust proxy" argument, but provides no empirical support for it. The paper suggests future work on "more fine-grained visual evaluation" (Section 7) but does not specifically propose extending to other visualization libraries.

Difficulty Estimation Cost Is Unaccounted For — The Benchmark Itself Does Not Measure Difficulty, Preventing Compute-Adaptive Deployment

Unlike the reference paper's compute-optimal framework—which explicitly estimates prompt difficulty and then adaptively allocates inference compute—RealChart2Code provides no difficulty estimation mechanism at all. The benchmark is designed to be uniformly hard (all instances feature complex multi-panel layouts and authentic large-scale data), but it does not stratify instances by difficulty level, does not report performance by difficulty bin, and does not provide any mechanism for a deployed system to estimate how much compute a given chart-to-code task requires.

The consequence is that a practitioner using this benchmark to guide model selection has no signal about whether the observed performance generalizes across difficulty regimes. A model that achieves a score of 3.6 (Qwen3-VL-235B) might perform adequately on simpler subsets of real-world tasks while collapsing on the hardest ones, or it might perform uniformly poorly—the benchmark's aggregate score cannot distinguish these scenarios. Conversely, a model that scores well on average might still fail catastrophically on certain chart types or data scales, but the benchmark provides no systematic breakdown by these factors. The paper's error analysis (Section 5.2) categorizes failure modes but does not correlate failure frequency with instance-level difficulty features (number of subplots, data size in rows, number of distinct chart types in a composite visualization, presence of specific layout patterns like GridSpec or joint plots).

The evidence for this limitation is primarily the absence of what would be expected. The paper reports aggregate scores per task per model (Table 2), per-metric breakdowns for top models (Figure 4), and error type distributions (Figure 6), but never breaks down performance by chart complexity, data scale, or number of subplots. The case studies in Appendix F (Figures 8–11) qualitatively suggest that specific layout patterns (GridSpec, joint plots, nested sub-layouts) are failure triggers, but there is no quantitative analysis of how failure rates vary with these features. The one partial exception is the data length distribution in Figure 3(c), which shows the range of data sizes in the benchmark (median and mean token counts), but this distribution is never linked to model performance—we do not know whether models perform worse on longer data files.

The mitigation status is that the paper does not acknowledge this as a limitation. The focus is on establishing that a complexity gap exists between RealChart2Code and prior benchmarks, not on characterizing the difficulty spectrum within RealChart2Code itself. The taxonomy of chart types (Table 5, Appendix A.1) and visualization intents (Appendix B.3) provides the scaffolding for a difficulty-stratified analysis, but that analysis is not performed. A future version of the benchmark could tag instances with difficulty features (subplot count, chart type count, data row count, layout complexity score) and report performance stratified by these features, which would substantially increase the benchmark's diagnostic utility.

The Automated Multi-Agent Judge Shows Weaker Agreement on Subjective Dimensions, Limiting Fine-Grained Quality Assessment

The paper's evaluation framework relies on a multi-agent LLM judging panel to score generated visualizations on an 11-dimensional rubric. The reliability analysis in Table 4 reveals that while inter-agent and agent-human agreement is strong overall (average Fleiss' κ = 0.82, average Cohen's κ = 0.83), agreement on specific dimensions is substantially weaker. The lowest-performing dimensions are:

• Color Scheme Consistency: Agent-human κ = 0.72 (the lowest agent-human agreement across all eight visual accuracy metrics)
• Component Completeness: Fleiss' κ = 0.781, agent-human κ = 0.74
• Style and Format Consistency: Fleiss' κ = 0.829
• Data Pattern Consistency: Fleiss' κ = 0.813

The consequence is that scores on these dimensions contain more evaluation noise than scores on high-agreement dimensions like Spatial Layout (agent-human κ = 0.99) or Chart Type Consistency (κ = 0.91). For Color in particular, a Cohen's κ of 0.72 falls in the "substantial agreement" range (0.61–0.80) rather than the "almost perfect" range (>0.80), meaning that inter-rater disagreement on color scoring could meaningfully affect a model's rank ordering. If Model A and Model B differ primarily on color quality, the judge's reliability on that dimension is insufficient to confidently distinguish them.

The paper acknowledges this limitation in Section 7:

"although our MLLM-based judges show strong correlation with human experts, they may still fail to detect subtle visual artifacts, such as minor element overlaps or precise color nuances. These areas offer promising directions for future research in more fine-grained visual evaluation."

However, this framing understates the issue. The problem is not just that judges "may still fail to detect subtle visual artifacts"—it is that the metrics on which they show weakest agreement (Color, Completeness, Style) are precisely the dimensions that differentiate "good enough" from "professional quality" visualizations, which is arguably what practitioners care about most once basic structural correctness is achieved. A benchmark that can reliably assess layout correctness but is noisy on aesthetic quality may encourage models that optimize for structural fidelity at the expense of visual polish.

The mitigation approach is partial: the reliability analysis identifies which dimensions are noisiest, but the paper does not propose alternative evaluation strategies for those dimensions (e.g., programmatic color distance metrics, pixel-level overlap detection, or human-in-the-loop evaluation for borderline cases). The multi-agent voting system partially mitigates noise through aggregation, but the Fleiss' κ on Color (0.879 inter-agent) indicates that even multiple agents agree only moderately strongly with each other, suggesting the noise is not purely random but reflects genuine ambiguity in the scoring rubric for that dimension.

Proprietary Model Evaluations Are Opaque and Non-Reproducible

The paper evaluates five proprietary models (Claude-4.5-Sonnet, Claude-4.5-Opus, GPT-5.1, Gemini-2.5-Flash, Gemini-3-Pro-Preview) via their official APIs, using the specific model versions listed in Table 6 (Appendix C.2). The consequence is that these results are fundamentally non-reproducible: the exact model versions evaluated may be deprecated, updated, or retired by their providers, meaning that a researcher attempting to reproduce the evaluation in six months may be querying different models (even if the model name remains the same, as providers frequently update model weights without changing the public identifier). This is a well-known problem in LLM evaluation, but it is particularly acute for a benchmark paper whose primary claim is that RealChart2Code provides a stable, discriminative evaluation framework—if the proprietary baselines are ephemeral, the benchmark's ability to track progress over time is compromised.

The evidence is in Table 6, which lists specific version strings (e.g., claude-4-5-sonnet-20250929, gpt-5-1-2025-11-13, gemini-3-pro-preview). Some of these are preview versions (Gemini-3-Pro-Preview) that are explicitly non-final and subject to change. The paper reports results as averages of three independent runs, but does not report the dates when API calls were made, making it impossible to verify whether all runs used the same model version or whether silent updates occurred between runs.

The paper does not acknowledge this as a limitation. The mitigation would be to release the complete API responses (or at minimum the generated code for all instances) so that future researchers can evaluate new models against the same generated outputs without needing to re-query deprecated APIs, but the paper does not indicate whether such artifacts are included in the released benchmark. The dataset release at the provided HuggingFace URL may include model outputs, but the paper text does not state this explicitly.

Small Test Set for Human Correlation Study Limits Confidence in Metric Validity

The human evaluation validating the multi-agent judge is conducted on only 600 tasks sampled from Claude-4.5-Sonnet results (Section 5.1). With 14 models evaluated across up to three tasks each, Claude-4.5-Sonnet represents a single model's output distribution, and 600 instances—while a reasonable absolute number—cover only a fraction of the 2,896 total instances and only one of 14 model output distributions.

The consequence is that the strong agent-human agreement (average Cohen's κ = 0.83) is validated only for the specific distribution of outputs produced by a single, relatively high-performing proprietary model. We do not know whether the automated judge maintains calibration on the outputs of weaker models that produce more severely flawed charts—a judge trained or prompted to distinguish subtle differences between good charts may behave unpredictably when presented with catastrophically bad charts that lie far outside the distribution of its training or few-shot examples. For example, if an open-weight model generates a chart where the layout is completely broken (overlapping subplots, missing axes, gargantuan whitespace), the judge's rubric designed for fine-grained comparison to a reference may produce noisy or miscalibrated scores because the generated chart is so far from the reference that the 0/1/2 scale becomes ambiguous at the low end.

The paper provides partial mitigation through the inter-agent Fleiss' κ, which is computed across the entire benchmark (all 2,896 instances, all 14 models) and shows strong agreement overall (average 0.82). This suggests that the agents agree with each other consistently even on low-quality outputs. However, inter-agent agreement does not guarantee alignment with human judgment on those low-quality outputs—multiple agents could share the same systematic bias. The human correlation study, which would detect such bias, is limited to Claude-4.5-Sonnet outputs, leaving open the possibility that agent-human agreement degrades for outputs from weaker models. The paper does not discuss this coverage limitation.

The Benchmark Cannot Distinguish Capability from Prompt Sensitivity

All 14 models are evaluated with the same prompt format (standardized ChatML structure, greedy decoding, fixed system prompt content described in Appendix D). The paper's rationale is that this provides a fair, controlled comparison of "out-of-the-box capability" (Section 4.1). However, this design choice conflates two distinct sources of performance variation: genuine differences in chart-to-code capability versus differences in prompt sensitivity. A model that scores poorly on RealChart2Code with the standard prompt might perform substantially better with a different prompt format—more detailed instructions, chain-of-thought prompting, few-shot examples, or explicit decomposition of the task into sub-steps (e.g., "first describe the chart structure, then generate the code").

The consequence is that the benchmark's model rankings may reflect prompt compatibility rather than underlying capability. This is particularly relevant for open-weight models, which are often sensitive to prompt formatting differences (e.g., chat template style, system prompt presence, instruction phrasing) in ways that proprietary models—which undergo extensive post-training to handle diverse prompts—may not be. A model like Qwen3-VL-235B might underperform on RealChart2Code partially because the standardized prompt is suboptimal for its specific chat template or instruction-following behavior, not because it fundamentally lacks chart-to-code capability. The paper's finding that open-weight models "frequently hallucinate non-existent libraries or invoke invalid functions" (Section 5.2) is consistent with a prompt sensitivity explanation: a prompt that more explicitly instructs the model to use only standard Matplotlib functions might reduce such hallucinations.

The paper provides no prompt sensitivity analysis—no comparison of alternative prompt formats, no few-shot baselines, no chain-of-thought ablation. The one partial exception is the Chart Refinement task, which uses a different prompt structure (flawed code + user instruction), but this is a task-level variation, not a prompt-sensitivity measurement within a task. The mitigation status is that the paper does not acknowledge prompt sensitivity as a limitation; the uniform prompting is presented as a feature (fair comparison) rather than a confound (potential under-measurement of some models' capabilities). A standard mitigation in benchmark papers is to report few-shot baselines alongside zero-shot to bound the prompt sensitivity effect, but this is not done here.

7. Implications and Future Directions

How This Work Changes the Landscape

RealChart2Code does not propose a new model, training method, or architectural innovation. Its contribution is infrastructural: it provides the field with an evaluation instrument that reveals capability gaps invisible under saturated benchmarks. The significance of this contribution depends on the field's willingness to confront the uncomfortable finding that 90%+ scores on existing chart-to-code benchmarks are misleading—not because the benchmarks are wrong, but because they measure a fundamentally easier capability than what real-world chart generation demands.

The paper shifts the conversation around chart-to-code evaluation in three specific ways:

First, it establishes that chart complexity—specifically multi-panel layouts, real data scale, and iterative refinement—is the dimension on which current models break, not chart type vocabulary or basic plotting syntax. Prior benchmarks treated chart-to-code as a monolithic capability where progress meant higher scores on a fixed test set. This paper demonstrates that the capability is multi-dimensional: models that have mastered single-panel chart replication (scores of 90%+ on ChartMimic and Plot2Code) lack entirely different skills—global layout orchestration, data pipeline correctness, and multi-turn code consistency—that only become visible at higher complexity. This reframes the research problem from "make chart-to-code scores go up" to "identify and close specific sub-capability gaps," which is a more productive framing because it directs effort toward the actual bottlenecks rather than toward whatever is easiest to optimize.

The evidence for this multi-dimensionality is in the disaggregated error analysis (Section 5.2, Figure 6): proprietary models fail primarily on Data Mapping (wrong columns mapped to correct visual structures), while open-weight models fail primarily on Syntax and Execution (hallucinated imports, invalid API calls) and Layout (spatial reasoning breakdowns in multi-panel grids). These are qualitatively different failure modes that single-score benchmarks conflate. The paper provides the diagnostic framework—eight visual accuracy metrics, programmatic data alignment checking, and a multi-turn refinement task—that makes sub-capability diagnosis possible for the first time in this domain.

Second, it introduces the Complexity Gap as a criterion for benchmark adequacy. The paper's finding that all models fall substantially below the y=x diagonal in Figure 7 (existing benchmark scores vs. RealChart2Code scores) is not merely evidence that RealChart2Code is harder—it is evidence that prior benchmarks failed to measure capabilities that matter. The Complexity Gap is now a portable concept: any new chart-to-code benchmark can be evaluated by whether it produces a substantial gap relative to RealChart2Code, and if not, it is probably not testing capabilities beyond what RealChart2Code already covers. This creates a natural progression where benchmarks build on each other's difficulty, rather than proliferating independently with no clear difficulty hierarchy.

However, this landscape shift comes with an important scope constraint: the Complexity Gap is demonstrated entirely within the Matplotlib ecosystem. The paper's argument (Section 7) that Matplotlib's "granular and imperative nature" serves as a "robust proxy for general plotting skills" is plausible but unverified. If a future benchmark built on a declarative visualization framework (e.g., Vega-Lite) shows that models have different relative strengths—better at data-to-visual mapping specification, worse at pixel-level layout control—then the landscape shifts again, and the current Complexity Gap is understood as specific to imperative plotting rather than general visualization capability. The paper's contribution is thus a domain-specific advancement whose generalizability to other visualization paradigms remains an open question.

Third, it makes multi-turn refinement evaluation a first-class concern rather than an afterthought. Prior chart-to-code and code generation benchmarks overwhelmingly evaluate single-turn generation. The paper's Chart Refinement task—864 instances where models must debug flawed code based on natural language feedback—demonstrates that single-turn capability and multi-turn consistency are distinct dimensions. The finding of "Regressive Editing" (Section 5.2)—models fix the specified error only to break previously correct code—is a failure mode that single-turn evaluation completely masks, and it is arguably the most practically important finding in the paper because it directly impacts the user experience of AI-assisted visualization tools. The paper does not solve Regressive Editing, but it provides the first benchmark where the phenomenon can be systematically measured rather than anecdotally observed.

The paper also validates a methodological principle: that programmatic evaluation of data correctness (the Data Alignment metric in Appendix C.1.2) can catch errors that visual comparison misses, and that this matters because the most common proprietary model failures are precisely the errors (wrong column mapping, wrong aggregation) that produce visually plausible but numerically incorrect charts. This principle is portable to any benchmark where generated code produces visual outputs—web design, UI generation, scientific figure reproduction—and suggests that benchmarks relying solely on visual comparison are systematically overestimating model capability.

Follow-Up Research This Work Enables

Difficulty-stratified evaluation of chart-to-code capability. The paper's aggregate scores per task (Table 2) and error type distribution (Figure 6) are informative but coarse. A natural follow-up is to tag each of the 2,896 instances with difficulty features—number of subplots, number of distinct chart types in the composite, data row count, presence of specific layout patterns (GridSpec, joint plots, dual-axis overlays, hierarchical grids)—and then analyze how each model's failure rate varies with each feature. The paper already has the data to do this (the 1,016 visualizations are manually designed with known chart types and known layout structures), but it does not report performance stratified by difficulty features.

A strong follow-up would produce a difficulty profile for each model: at what subplot count does Qwen3-VL-235B's Layout score drop below a threshold? At what data size does Gemini-3-Pro-Preview's Data Alignment score degrade? Does the Regressive Editing rate in Chart Refinement increase with code length? These profiles would transform RealChart2Code from a single-number benchmark into a diagnostic instrument that tells a practitioner exactly which tasks their model can handle and where it will fail. They would also reveal whether the performance gap between proprietary and open-weight models is uniform across difficulty or concentrated in specific difficulty regimes, which would guide targeted improvement efforts.

Cross-library generalization of chart-to-code capability. The paper is entirely Matplotlib-based. A natural stress test is to construct a parallel benchmark using a declarative visualization library (e.g., Altair, Vega-Lite, or ggplot2 in R) with the same chart scenarios—same datasets, same visualization designs, but implemented in a fundamentally different API paradigm. The research question is: do models that perform well on Matplotlib chart-to-code also perform well on declarative chart-to-code, or is the capability library-specific? If capability transfers, it supports the paper's claim that Matplotlib tests "fundamental visualization understanding." If it does not, it reveals that the paper's findings are about imperative plotting specifically, and the field needs separate evaluations for each visualization paradigm.

A well-designed experiment would evaluate the same 14 models (or their successors) on a declarative counterpart to RealChart2Code and compute the correlation between Matplotlib and declarative scores. A strong negative result—models with high Matplotlib scores but low declarative scores—would be highly informative, as it would indicate that current VLMs are learning library-specific API patterns rather than generalizable visualization reasoning. This would redirect research toward training methods that encourage abstraction across visualization frameworks rather than memorization of specific APIs.

Prompt engineering ablation across RealChart2Code tasks. The paper's uniform prompting strategy (standard ChatML, greedy decoding, no few-shot examples) is a deliberate choice for fair comparison, but it leaves open the question of how much of the observed performance gap is due to suboptimal prompting rather than fundamental capability limitations. A systematic ablation would test: (1) few-shot prompting with 1-3 example chart-to-code pairs, (2) chain-of-thought prompting that first asks the model to describe the chart structure before generating code, (3) explicit library constraint prompts (e.g., "Use only standard Matplotlib functions; do not import any libraries not listed here"), and (4) multi-step decomposition (first generate the data loading code, verify it, then generate the plotting code).

The specific hypothesis to test is whether the open-weight models' high syntax error rate—particularly the 20% hallucination rate for Qwen3-VL-235B on plt.style.use('seaborn-v0_11') documented in Appendix F—is a genuine code generation deficiency or a prompt sensitivity issue. If providing explicit library constraints in the prompt reduces syntax errors to near zero for open-weight models, then the current benchmark scores underestimate their capability and the Complexity Gap is partially a measurement artifact. If explicit constraints do not help, then the syntax errors reflect fundamental training data contamination or code generation weaknesses that prompt engineering cannot fix. Either result is informative, but the experiment is necessary to calibrate how much of the reported gap is "real."

Regressive Editing measurement and mitigation. The paper identifies Regressive Editing as a failure mode in Chart Refinement—models fix the specified error but introduce new errors in previously correct code—but does not quantify its frequency or severity. A systematic follow-up would: (1) define an operational metric for Regressive Editing (e.g., for each refinement instance, count how many previously correct sub-metrics degrade after the edit), (2) compute Regressive Editing rates for all 14 models across the 864 refinement instances, (3) correlate Regressive Editing rates with model characteristics (size, architecture, training recency), and (4) test whether specific mitigation strategies reduce the rate.

Candidate mitigations include: self-verification (ask the model to check its own corrected code against the reference before outputting the final version), diff-based editing (output only the changed lines rather than regenerating the entire file), and explicit instructions to "preserve all other functionality unchanged." Measuring whether these strategies reduce Regressive Editing would provide actionable guidance for building AI-assisted visualization debugging tools. The negative result—that even state-of-the-art models cannot reliably avoid Regressive Editing regardless of prompting strategy—would be equally valuable, as it would indicate that this is a fundamental limitation of current architectures requiring architectural innovation rather than prompt engineering.

ChartCoder analysis: why does task-specific fine-tuning fail to transfer? ChartCoder (Zhao et al., 2025) is a model specifically fine-tuned for chart-to-code generation, yet it underperforms general-purpose models at similar scales on RealChart2Code (scoring 3.2 AVG vs. Qwen3-VL-235B's 3.6). This is the most striking negative result in the paper that receives the least analysis. A dedicated follow-up would compare ChartCoder's behavior to general-purpose models on a fine-grained level: Does its fine-tuning improve syntax correctness at the cost of layout reasoning? Does it overfit to the simple chart types prevalent in its training data, causing it to fail more catastrophically on the complex layouts that distinguish RealChart2Code? Does its training on synthetic or simplified data create a distribution shift that degrades performance on authentic messy data?

Answering these questions would clarify whether task-specific fine-tuning as currently practiced is counterproductive for complex chart-to-code tasks—and if so, what data characteristics (chart complexity, data realism, layout diversity) are necessary for fine-tuning to transfer to RealChart2Code-level difficulty. This is immediately actionable for teams building specialized chart-to-code models.

Temporal benchmarking: track capability evolution on RealChart2Code. The paper evaluates 14 models with release dates ranging from March 2024 (DeepSeek-VL-7B) to November 2025 (Claude-4.5-Opus, Gemini-3-Pro-Preview, GPT-5.1). A natural extension is to maintain RealChart2Code as a living benchmark, periodically re-evaluating new model releases (both proprietary and open-weight) and tracking how the Complexity Gap evolves over time. Does the gap shrink monotonically as models improve, or does it show step-function improvements when specific architectural innovations (e.g., longer context windows, native multimodal fusion, reinforcement learning from visual feedback) are introduced? A temporal analysis would reveal which innovations actually address the sub-capability gaps that RealChart2Code measures, versus which improve performance on saturated benchmarks without moving the needle on realistic chart generation. This would serve as a progress metric for the field rather than for individual models.

Practical Applications and Downstream Use Cases

Model selection for data science and analytics platforms. The paper's cross-benchmark comparison (Tables 2 and 3) provides concrete guidance for platforms integrating AI-assisted visualization. A platform that needs to handle simple user requests ("plot this column against that column") can use any model scoring above 80% on ChartMimic or Plot2Code—which includes most open-weight models—and achieve high reliability. However, a platform that promises to reverse-engineer complex multi-panel dashboards from screenshots needs the capabilities that only RealChart2Code measures. The paper's results indicate that such a platform should deploy a proprietary model (Claude-4.5-Opus or Gemini-3-Pro-Preview, with AVG scores of 8.2 and 8.1) and should expect ~50% normalized accuracy on complex tasks (per Appendix E.1, Figure 7). This number—~50% on real-world-complexity charts—is the honest capability ceiling that product teams should plan around, not the 90%+ numbers that prior benchmarks would suggest.

Error-aware deployment: routing by expected failure mode. The paper's error analysis (Figure 6) enables a deployment architecture where tasks are routed to different models based on predicted failure mode. Open-weight models, which suffer heavily from syntax errors and layout failures, could be deployed with a syntax-checking pre-processor that catches hallucinated imports and invalid API calls before code execution, potentially recovering a substantial fraction of their otherwise-zero scores. Proprietary models, which pass execution reliably but suffer from Data Mapping errors, could be paired with a programmatic data alignment verifier (similar to the paper's own evaluation metric in Appendix C.1.2) that flags cases where the wrong column was used, enabling automatic correction or human review. The paper's multi-metric scoring framework (Figure 4) provides the diagnostic signal needed to implement such a routing system, because it tells the platform not just how often a model fails, but in which dimension it fails.

Training data curation for visualization-specialized models. The paper's finding that ChartCoder—a model specifically fine-tuned for chart-to-code—underperforms general-purpose models (AVG of 3.2 vs. 3.6 for Qwen3-VL-235B, Table 2) suggests that current fine-tuning datasets lack the complexity and real-world grounding needed for transfer. RealChart2Code provides a blueprint for what a fine-tuning dataset should contain: multi-panel composite layouts, large-scale authentic data with messy real-world structure, and multi-turn refinement trajectories. A team building a specialized chart-to-code model could use the paper's data curation pipeline (Section 3.3, Appendix B) as a template for generating training data—the same four stages (collect real datasets, design complex visualizations, implement ground-truth code, inject errors for refinement examples) but scaled up with LLM assistance for the code implementation stage (using the human-authored ground truth as few-shot examples for generation). The benchmark's rubric (Appendix C.1) could serve as automated quality filtering for generated training instances.

Benchmark-driven capability tracking for regulatory or procurement purposes. As VLMs become integrated into tools where visualization correctness has consequences—medical data dashboards, financial reporting, scientific publication figures—organizations may need to verify that their chosen model meets a minimum capability standard. RealChart2Code, with its multi-dimensional scoring and validated alignment with human judgment (average Cohen's κ = 0.83, Table 4), provides an auditable evaluation standard. An organization could require that any deployed model achieve a minimum score on specific dimensions (e.g., score ≥ 1 on Data Alignment for Chart Reproduction, indicating correct data mapping) before being approved for use in contexts where data misrepresentation would be costly. The benchmark's release on HuggingFace and the documented evaluation protocol in Appendix A.2 make such procurement evaluations reproducible, assuming access to the same proprietary model versions.