ViGoR-Bench: How Far Are Visual Generative Models From Zero-Shot Visual Reasoners?

🎯 Pitch

Modern video generators like Sora 2 Pro can achieve 85% visual quality while succeeding at logical reasoning only 10% of the time, revealing a massive 'illusion of reasoning' where photorealism masks near-total cognitive collapse. ViGoR-Bench introduces a process-outcome evaluation framework that dismantles this performance mirage across 20 cognitive domains, showing that reinforcement learning on harder mazes can push generative reasoners to 97% accuracy where supervised fine-tuning plateaus.

1. Executive Summary

This paper introduces ViGoR-Bench, a comprehensive benchmark and evaluation framework designed to stress-test the visual reasoning capabilities of generative models across image-to-image, sequential image-to-image, and image-to-video tasks spanning 20 cognitive subdomains. The framework establishes a dual-track process-outcome evaluation protocol (assessing both intermediate reasoning steps via continuous metrics and final outputs via binary pass/fail judgments) powered by an evidence-grounded automated judge system that achieves high human alignment—reducing MAE to 0.267 for process metrics and 0.213 for result metrics when ground-truth references are provided. Evaluating over 20 leading proprietary and open-source models reveals that even state-of-the-art systems harbor significant reasoning deficits, with video generation models exhibiting an "Illusion of Reasoning" where high visual quality scores (e.g., Sora 2 Pro at 85.5% process VQ) co-exist with near-zero reasoning success (10.1% result RS), while proprietary unified models like Nano Banana Pro maintain a substantial lead with 68.4% average result accuracy. Post-training experiments on maze navigation demonstrate that reward-driven reinforcement learning can elicit reasoning capabilities where supervised fine-tuning saturates, with RL-trained models achieving up to 97.0% reasoning success when trained on out-of-distribution high-complexity data, establishing that generalization to simpler tasks benefits from exposure to harder examples only when the training signal incorporates explicit reward optimization.

2. Context and Motivation

The Core Problem: A "Logical Desert" Beneath Stunning Visual Surfaces

The paper identifies a fundamental disconnect in contemporary AI-generated content (AIGC): modern generative models produce images and videos of remarkable photorealism, yet they systematically fail at tasks requiring genuine reasoning—understanding physical causality, following multi-step logical constraints, or applying world knowledge to visual scenarios. The authors articulate this as a "logical desert":

"beneath a facade of photorealism, models crumble when faced with tasks requiring deep physical laws or causal reasoning."

This is not merely an aesthetic problem. A model that generates a visually perfect image of a Sudoku board but fills it with logically impossible digit arrangements, or one that produces fluid video of object manipulation but violates basic physics, demonstrates that visual fidelity has outpaced cognitive capability. The core gap is that we lack evaluation frameworks that systematically measure whether generative models truly "understand" what they are generating, as opposed to producing superficially plausible outputs that happen to align with their training distribution statistics.

This matters for several practical and theoretical reasons the paper highlights (Section 1, Impact Statement):

Safety-critical applications: Models that don't understand physical causality can generate dangerously misleading content in domains like education, scientific simulation, or engineering design—imagine a medical illustration with anatomically impossible structures, or a "how-to" video that demonstrates an unsafe procedure.
Reliability for downstream tasks: If generative models are to serve as components in larger AI systems (e.g., visual planning for robotics, automated design, data augmentation), their outputs must be logically coherent, not just visually appealing. A robot that plans assembly steps based on a generative model's "understanding" of block stacking will fail catastrophically if that understanding is illusory.
The frontier between generation and intelligence: The paper poses an implicit theoretical question: is photorealism sufficient to claim a model "understands" the visual world, or is reasoning a distinct capability that must be separately developed and measured? By showing that models with excellent visual quality scores can have near-zero reasoning success rates, the paper provides empirical evidence that these are distinct axes of capability—and that the field's overwhelming focus on the former has left the latter critically underevaluated.

The "Performance Mirage" Created by Current Evaluation Practices

A key argument in the paper is that existing evaluation paradigms systematically obscure the reasoning deficit. The authors identify a "performance mirage" fostered by dominant metrics:

"traditional metrics like CLIP-Score and FID... prioritize semantic alignment and statistical fidelity over true structural integrity. Notably, a generated image can achieve high statistical similarity to real data while still harboring absurd physical glitches."

This is a crucial claim that deserves unpacking. CLIP-Score (Hessel et al., 2021) measures alignment between an image and a text description by computing cosine similarity in CLIP embedding space—it can confirm that a generated image contains objects mentioned in a prompt but cannot verify whether those objects are arranged in a physically plausible manner. FID (Fréchet Inception Distance; Heusel et al., 2017) measures the distributional similarity between generated and real image sets—a model that consistently generates images with exactly the same statistical patterns as its training data will score well on FID even if every generated image contains subtle logical errors. Neither metric evaluates whether an output image correctly solves a reasoning task.

The paper argues this creates a pernicious feedback loop: models are optimized and selected based on metrics that reward visual fidelity, leading researchers and practitioners to believe their models are more capable than they actually are. The "performance mirage" means that a model can rank highly on standard leaderboards while being fundamentally incapable of answering questions that require understanding what it has generated. This is analogous to a student who can produce eloquent prose without comprehending the subject matter—the fluency of expression masks the absence of understanding.

Fragmentation in the Prior Evaluation Landscape

The paper situates itself within a rapidly evolving ecosystem of reasoning-centric benchmarks for generative models, but argues these efforts remain fragmented and incomplete. Section 2.2 and Table 1 provide a systematic comparison that reveals three critical gaps.

Gap 1: Benchmarks Operate in Silos

Existing benchmarks are narrowly restricted to either Image-to-Image (I2I) editing or Video generation (I2V) tasks, with no framework providing unified cross-modal evaluation. The authors enumerate:

Image-only reasoning benchmarks: RISE (Zhao et al., 2025), KRIS-Bench (Wu et al., 2025c), GIR-Edit (Li et al., 2025b), UniREdit (Han et al., 2025), and WiseEdit (Pan et al., 2025) all evaluate reasoning in the context of image editing, but none extend to video or sequential image generation.
Video-only reasoning benchmarks: MME-CoF (Guo et al., 2025), RULER-Bench (He et al., 2025), and Veo (Wiedemer et al., 2025) focus on temporal reasoning in video, but don't assess image editing capabilities.

This fragmentation is problematic because it prevents cross-modal comparison. A researcher developing a unified vision model cannot use any single existing benchmark to understand whether their model's reasoning capabilities transfer between image and video modalities, or whether the same reasoning deficit manifests differently in static versus dynamic outputs. The paper's inclusion of both I2I and I2V tasks within a single framework is motivated by the observation that reasoning should be modality-agnostic—a model that truly understands physical causality should be able to demonstrate that understanding whether the output is a single image or a video sequence.

Furthermore, the paper introduces Sequential I2I (I2Is) as a distinct evaluation modality—tasks where the model produces a sequence of intermediate images showing its step-by-step reasoning process. This sits between purely static image editing and full video generation, and is motivated by the emergence of "thinking" models that explicitly externalize their reasoning through interleaved visual-textual outputs (e.g., BAGEL-Think, Zebra-CoT). No prior benchmark evaluates this modality.

Gap 2: Process vs. Outcome — The Missing "How"

The paper argues that most existing benchmarks suffer from a conceptual blind spot: they evaluate only the final output (the "what") while ignoring the reasoning process (the "how"). Table 1 makes this explicit: of the 9 benchmarks compared, only UniREdit and MME-CoF evaluate process-level metrics, and neither does so with the granularity the paper proposes.

Why does the process matter? The paper provides a compelling example in its "Illusion of Reasoning" finding (Section 4.1): video generation models like Sora 2 Pro achieve Process Visual Quality scores of 85.5%—their intermediate frames look excellent—but Result Reasoning Success scores of only 10.1%. Without process metrics, an evaluator would see only the final output and miss the crucial diagnostic information that the model is producing visually coherent intermediate reasoning steps that simply don't lead to correct conclusions. Process evaluation reveals where in the reasoning pipeline failures occur—is the model failing to understand the initial problem? Generating plausible but incorrect intermediate steps? Derailing at the final integration stage?

The paper's dual-track evaluation (Section 3.2) is designed specifically to enable this diagnostic decomposition. Process Metrics use continuous 0–100 scales to capture the quality and trajectory of intermediate reasoning, while Result Metrics use binary pass/fail judgments on final outputs. This design choice reflects a pedagogical philosophy: understanding failure modes requires granular, continuous feedback on process quality, while certifying final capability requires rigorous, unambiguous correctness judgments.

Gap 3: Automated Judges Without Grounding

The paper acknowledges that the "VLM-as-a-Judge" paradigm has become the de facto standard for scalable evaluation in generative AI, citing models like GPT-4o and Gemini 2.5 Pro as widely used evaluators. However, it identifies a persistent reliability bottleneck:

"achieving robust human alignment across multifaceted reasoning tasks remains a persistent bottleneck."

The concern is that LLM-based evaluators, when asked to judge complex visual reasoning outputs without access to ground-truth references, exhibit subjectivity and instability. They may reward superficially impressive outputs that happen to be wrong, or penalize correct outputs that don't match their expectations. The paper's evidence-grounded approach (Section 3.2) addresses this by explicitly providing the judge with ground-truth information—either reference images or textual answer descriptions—and requiring it to anchor its evaluations to this evidence. This transforms the judge from an open-ended subjective assessor into a reference-comparison engine, which the paper's reliability analysis (Table 2) shows dramatically improves alignment with human experts.

The paper quantifies this: without ground truth, Gemini-2.5-Pro achieves only 68.0% accuracy on process metrics and 70.5% on result metrics compared to human experts, with MAE values of 0.319 and 0.294 respectively. Adding ground-truth references pushes accuracy to 73.3% and 78.6%, reducing MAE to 0.267 and 0.213. This is a significant empirical finding: it suggests that much of the unreliability in VLM-as-a-Judge systems stems not from the judge's inherent limitations but from insufficient grounding information. When told what the right answer is, the judge becomes substantially more reliable.

How the Paper Positions Itself

The paper positions ViGoR-Bench as a unified, diagnostic, and evidence-grounded framework that fills the gaps left by prior work:

Unified: It is the first benchmark to simultaneously evaluate I2I, I2Is, and I2V tasks within a single framework spanning 20 reasoning subdomains across three cognitive domains (Physical, Knowledge, Symbolic reasoning). This enables cross-modal comparison and provides a more complete picture of a model's reasoning capabilities.
Diagnostic: The dual-track process-outcome evaluation, combined with per-dimension scoring (Background Consistency, Rule Obey, Visual Quality, and Reasoning Accuracy/Success), decomposes performance into fine-grained cognitive components. Rather than reporting a single aggregate score, ViGoR enables researchers to identify which specific reasoning faculty is failing—rule-following? spatial reasoning? factual recall?—and at which stage of the generation pipeline.
Evidence-grounded: By incorporating ground-truth references into the evaluation protocol and empirically validating alignment with human experts, the paper addresses the subjectivity concerns that plague VLM-as-a-Judge approaches.
Practical for model improvement: Beyond evaluation, the paper demonstrates that ViGoR's metrics can serve as training signals. The Section 4.3 experiments using maze navigation data to fine-tune models show that the benchmark's reasoning success metric correlates with real improvements in model capability, establishing ViGoR as not just a "stress test" but a development tool.

The paper explicitly distinguishes itself from prior benchmarks (Table 1) along multiple axes: task modalities (I2I, I2Is, I2V vs. single-modality in prior work), evaluation dimensions (both process and result vs. result-only in most benchmarks), and grounding mechanisms (image and text ground truth vs. text-only or no ground truth). This systematic differentiation makes clear that ViGoR is not another incremental benchmark but rather a reconceptualization of how generative visual reasoning should be measured.

The Implicit Argument About What "Visual Reasoning" Means

Beyond the technical contributions, the paper makes an implicit but significant argument about the nature of visual reasoning in generative models. By structuring the benchmark around three distinct reasoning domains—Physical Reasoning (embodied intelligence, spatial manipulation), Knowledge Reasoning (world knowledge, factual accuracy), and Symbolic Reasoning (logical deduction, constraint satisfaction)—the paper asserts that visual reasoning is not a monolithic capability but a constellation of distinct cognitive faculties. A model might excel at algebraic calculation while failing completely at maze navigation, or demonstrate strong world knowledge while being unable to reason about physical interactions.

This taxonomy has a specific intellectual motivation: it maps onto the distinction between systematic and heuristic reasoning that has been central to debates about whether neural networks can learn to reason. Physical and Symbolic reasoning tasks require systematic, rule-governed thinking—the same physical laws apply regardless of surface appearance; the same Sudoku constraints hold regardless of grid layout. Knowledge reasoning, by contrast, can potentially be solved through heuristic pattern matching to training data—recognizing that a diagram "looks like" a correct biology illustration without understanding the underlying principles.

The paper's finding that even top-tier models struggle most on Physical and Symbolic reasoning tasks (evident in the per-domain profiling in Figures 10–12) provides evidence for this distinction: models that perform well on knowledge tasks (where statistical pattern matching suffices) can perform poorly on systematic reasoning tasks (where rule-governed thinking is required). This connects ViGoR to broader debates about whether scaling up generative models leads to emergent reasoning or merely more sophisticated pattern matching.

3. Technical Approach

3.1 Reader orientation (approachable technical breakdown)

The system being built is a comprehensive evaluation framework — a benchmark dataset paired with an automated scoring pipeline — that assesses how well AI image and video generation models can perform visual reasoning tasks. It solves the problem that we currently cannot reliably measure whether generative models truly "understand" what they are creating (do they follow logical rules, obey physical laws, and apply world knowledge correctly?) because existing evaluation methods either focus on superficial image quality metrics or test reasoning in narrow, fragmented ways. The solution takes the shape of a dual-track, evidence-grounded diagnosis system: it gives models carefully designed reasoning problems spanning 20 cognitive subdomains, captures both their intermediate reasoning steps and their final outputs, and uses an automated judge (a multimodal language model provided with ground-truth reference answers) to score performance along four fine-grained dimensions — distinguishing between models that merely produce visually plausible outputs and those that genuinely reason correctly.

3.2 Big-picture architecture (diagram in words)

The ViGoR-Bench framework has three major components connected in a pipeline:

1. Data Engine (Section 3.1): This is the benchmark construction component. It produces 918 test samples spanning three reasoning domains (Physical, Knowledge, Symbolic) and 20 subdomains, using three distinct construction paradigms — Generative Synthesis (LLMs and image generators create synthetic scenarios), Real-world Acquisition (curating from authoritative sources and manual photography), and Algorithmic Construction (rule-based engines generate logically rigorous puzzles). Every sample includes an input image, a task instruction, and either a ground-truth reference image or a human-verified ground-truth text answer (or both). All data passes through a human-in-the-loop verification stage to ensure correctness.

2. Evaluation Protocol (Section 3.2): This is the scoring component. It implements a dual-track evaluation: Process Metrics (continuous 0–100 scores for video sequences or interleaved image-text reasoning chains, measuring Background Consistency, Rule Obey, Visual Quality, and Reasoning Accuracy) and Result Metrics (binary 0/1 pass-fail judgments on final static outputs, measuring Background Consistency, Rule Obey, Visual Quality, and Reasoning Success). The evaluation is performed by Gemini-2.5-Pro acting as an automated VLM-as-a-Judge, conditioned on ground-truth references and domain-specific scoring templates.

3. Reliability Analysis (Section 3.3): This is the validation component. It compares the automated judge's scores against a "gold standard" established by three independent human experts on a subset of 1,080 final results and 540 process sequences, measuring alignment via Mean Absolute Error (MAE), categorical accuracy, and scoring variance across multiple runs.

Information flows as follows: a test sample (input image + instruction + ground truth) enters the system → the generative model under evaluation produces an output (single image, image sequence, or video) → the output, along with the input and ground-truth references, is packaged into a domain-specific evaluation template → Gemini-2.5-Pro reads the template and produces a structured JSON score vector → scores are aggregated across samples to produce per-model, per-metric, per-domain performance profiles.

3.3 Roadmap for the deep dive

First, the Data Engine and its three construction paradigms, because the composition of the benchmark determines what kinds of reasoning are tested and how ground-truth is established — understanding this is prerequisite to interpreting any evaluation results.
Second, the dual-track Process and Result Metrics, including their precise mathematical formulations, because these define the paper's core methodological contribution: how reasoning is measured rather than merely what tasks are used.
Third, the evaluation protocol's domain-specific templates and the VLM-as-a-Judge system, because these determine the practical reliability and scalability of the framework — and the paper's claim of evidence-grounded automated alignment rests on their design.
Fourth, the reliability analysis methodology (meta-evaluation against human experts), because it validates the entire pipeline and establishes the trustworthiness of all downstream experimental findings.
Fifth, the post-training data construction and fine-tuning procedures used in Section 4.3, because these demonstrate how the benchmark serves not just as an evaluation tool but as a training signal for improving model reasoning.

3.4 Detailed, sentence-based technical breakdown

This is primarily a benchmark construction and evaluation methodology paper. Its core idea is that visual reasoning in generative models should be assessed through a unified, cross-modal framework that evaluates both the reasoning process and the final output, using evidence-grounded automated judges whose reliability is empirically validated against human experts.

The Data Engine: Three Construction Paradigms

The benchmark construction pipeline is not monolithic — it employs three distinct data generation strategies, each tailored to the epistemological requirements of different reasoning domains. The paper argues that no single data source can provide both the controlled logical rigor needed for symbolic tasks and the ecological validity needed for physical reasoning tasks, hence the multi-paradigm approach.

Generative Synthesis is used for the Physical Reasoning domain. The motivation is pragmatic: acquiring diverse real-world embodied reasoning data (images of object manipulation, sorting, assembly, spatial reasoning scenarios) is expensive and logistically difficult. The process works as follows:

Human annotators compose detailed textual descriptions of physical scenarios and the reasoning task to be performed (e.g., "sort the trash on the floor into the correct recycling bins," "place two dishes into the most appropriate storage locations").
These descriptions are enriched using large language models to add variation and detail.
The enriched prompts are fed to state-of-the-art image generation models — specifically NanoBanana-Pro (Google, 2025c) — which synthesize high-fidelity input images depicting the scenario.
All generated input images undergo human verification for plausibility and relevance to the intended reasoning task.

A critical consequence of this paradigm: no ground-truth images exist for Physical Reasoning samples (since the input images themselves are synthetic). Instead, the benchmark provides textual ground-truth answers that are manually annotated or verified by human experts to ensure logical consistency with the visual input. For instance, for a sorting task, the ground truth might be a text description like "the plastic bottle goes in the blue recycling bin, the banana peel goes in the green compost bin, and the battery goes in the red hazardous waste bin." The evaluator must judge whether the model's output image matches this textual description.

The paper notes this design choice explicitly:

"Since all input images are generated, no corresponding ground-truth images exist. Instead, we provide textual ground-truth answers, which are manually annotated or verified by human experts to ensure logical consistency with the visual input."

Real-world Acquisition is used for the Knowledge Reasoning domain. Here the priority is factual accuracy — the benchmark must test whether models can apply genuine world knowledge to visual scenarios, which requires ground-truth that is authoritatively correct rather than merely plausible. The data is curated from educational websites and scientific repositories. The paper states:

"A substantial portion of the data is curated from authoritative educational websites and scientific repositories to ensure factual accuracy."

This domain spans seven subdomains: Biology, Physics, Chemistry, Geography, History, Sports, and Common Sense. For samples where paired "before-and-after" datasets exist (for example, scientific diagrams showing a phenomenon in two states), the original ground-truth images are preserved as reference. For other samples, only human-verified textual ground-truth is provided.

Algorithmic Construction is used for the Symbolic Reasoning domain, where the primary requirement is logical rigor and solution uniqueness. The paper employs rule-based engines to generate samples with mathematically guaranteed correct answers:

For abstract logic puzzles (Sudoku, Maze Navigation, Jigsaw Puzzle, Function Plotting): rule-based algorithms generate both the input puzzle state and the unique correct solution, ensuring mathematical rigor. Input and ground-truth images are both algorithmically generated, with no textual ground-truth provided — the ground-truth is the correct image.
For Algebraic Calculation: large language models generate equations (covering linear and quadratic forms), and symbolic solvers validate the solutions. Both equations and solutions are rendered as images to serve as input and ground-truth respectively.
For Function Plotting: two-dimensional function expressions are generated, and the corresponding curves are plotted using Matplotlib to produce ground-truth images.
For Physical Puzzles (Klotski Puzzle, Block Building): these require visual realism to evaluate the alignment between perception and reasoning, so data is collected in physical environments — human annotators manually solve puzzles and photograph the solved states as ground-truth images. This is a notable exception within the Symbolic Reasoning domain: it uses real-world acquisition rather than algorithmic generation because the task demands photorealistic input.

Post-processing verification is a crucial final stage applied across all paradigms. The paper describes it as:

"a rigorous post-processing verification stage [including] human-in-the-loop review for semantic consistency and symbolic solver validation for mathematical precision."

This dual verification mechanism is what differentiates ViGoR from benchmarks that rely purely on automated generation without human oversight: semantic consistency (ensuring the input image actually depicts what the instruction claims) is verified by humans, while mathematical precision (ensuring puzzles have unique correct solutions) is verified by solvers.

What the benchmark provides that prior benchmarks do not: The paper emphasizes that ViGoR provides "both referenced ground-truth images and human-verified ground-truth captions where applicable." This dual ground-truth — visual reference + textual description — is what enables the evidence-grounded evaluation protocol described next. Prior benchmarks typically provide only one or the other, or neither (relying on the evaluator's internal knowledge).

The final benchmark comprises 918 samples (Figure 8 in Appendix A), distributed across three reasoning domains and 20 subdomains. The paper does not report the exact per-domain breakdown in the main text, but Figure 8 shows Symbolic Reasoning has the largest share (roughly 400+ samples across Block Building, Klotski Puzzle, Maze Navigation, Jigsaw Puzzle, Sudoku, Algebraic Calculation, and Function Plotting), followed by Knowledge Reasoning (roughly 250–300 samples across seven subdomains) and Physical Reasoning (roughly 150–200 samples across six subdomains).

The Dual-Track Evaluation Protocol: Process and Result Metrics

This is the paper's core methodological contribution: a two-layer scoring system that separately evaluates the quality of the reasoning process and the correctness of the final output. The motivation is that a model can produce a correct final answer through flawed or physically impossible intermediate steps (which matters for safety and interpretability), or can produce a logically sound reasoning process that nonetheless arrives at an incorrect conclusion (which matters for diagnosis and improvement). The two tracks are complementary: neither alone provides a complete picture.

Process Metrics: Continuous, Trajectory-Level Scoring

Process Metrics are designed for dynamic outputs — specifically, video generation models and "thinking" models that produce intermediate reasoning frames (interleaved image-text outputs showing step-by-step reasoning). The evaluation considers the full temporal sequence of the model's output, not just the final state.

The scoring is formulated as a function call to the VLM judge:

$S_{\text{Process}} = \text{VLM}(I, P, O_{\text{seq}}, R_i, R_t, T_{\text{Process}})$

where $I$ is the input image, $P$ is the editing prompt (the task instruction), $O_{\text{seq}}$ is the model's output sequence (intermediate frames or video), $R_i$ is the visual ground-truth image (if available), $R_t$ is the textual ground-truth reference (if available), and $T_{\text{Process}}$ is the evaluation template containing domain-specific scoring criteria.

What it computes: The VLM judge takes all this context and produces a structured score vector. Specifically, it watches the entire output sequence, compares intermediate states against the input and ground-truth, and rates the quality of the reasoning trajectory along four continuous dimensions, each scored from 0 to 100.

The output score vector is:

$S_{\text{Process}} = [S_{\text{BC}}, S_{\text{RO}}, S_{\text{VQ}}, S_{\text{RA}}]$

where:

$S_{\text{BC}}$ (Background Consistency, 0–100): measures the extent to which the main structure of the input image is preserved across the output sequence. It penalizes unintended modifications to regions unrelated to the reasoning task. For example, in a maze navigation task, if the model's intermediate frames show walls disappearing or changing position, this score drops. The paper's template phrases this as: "In what percentage of the model's frames does the main structure of the input image remain preserved?"
$S_{\text{RO}}$ (Rule Obey, 0–100): assesses the percentage of frames where edits strictly adhere to the constraints specified in the instruction. For a Sudoku task, any frame that contains a digit violating row/column/subgrid uniqueness reduces this score. For a physical sorting task, frames where objects are placed in wrong bins reduce this score. The template asks: "In what percentage of the model's frames are edits made according to the instruction?"
$S_{\text{VQ}}$ (Visual Quality, 0–100): evaluates the fidelity of generated frames — clarity, sharpness, and absence of temporal flickering, noise, or artifacts. This is the only dimension that measures aesthetic/perceptual quality rather than reasoning correctness, and it is included precisely to distinguish between models that produce beautiful but wrong outputs and those that produce correct but visually degraded outputs. The template defines it as checking "whether the model's modifications effectively progress toward the correct solution."
$S_{\text{RA}}$ (Reasoning Accuracy, 0–100): evaluates whether the model's progressive edits actually move toward the correct solution. This is described in the templates as "Beneficial Action" — steps that add correct information or make valid progress increase the score; random, regressive, or incorrect steps decrease it. For the Knowledge Reasoning CoT template, it is phrased as: "In what percentage of the model's frames do the edits effectively progress toward the correct answer as specified by the ground truth?"

The final aggregated process score is computed as the arithmetic mean:

$S_{\text{Process}}^{\text{Avg}} = \frac{1}{4} \left( S_{\text{BC}} + S_{\text{RO}} + S_{\text{VQ}} + S_{\text{RA}} \right)$

Why this form — a simple mean rather than a weighted combination: The paper does not explicitly justify the equal weighting, but the design rationale is implicit in the framework's diagnostic purpose. Weighting would require asserting that certain reasoning faculties are more important than others, which would be domain-dependent (rule-following matters more in Sudoku than in function plotting; visual quality matters more in photorealistic tasks than in abstract puzzles). By using equal weighting, the paper provides a neutral aggregate that researchers can decompose as needed — the per-dimension breakdown in Figures 10–12 is the actual diagnostic output; the average is only a summary.

Why continuous 0–100 scales for process: The paper does not explicitly argue this, but the choice is well-motivated by the nature of process evaluation. Intermediate reasoning steps rarely admit binary judgments — a model might produce 7 out of 10 intermediate frames correctly following Sudoku rules, and compressing that to 0 or 1 loses crucial diagnostic granularity. The continuous scale captures partial correctness and enables researchers to track how much of the reasoning trajectory is valid, not just whether the final state is reached. This is particularly important for the "Illusion of Reasoning" finding: models with high process visual quality but low reasoning accuracy produce many visually impressive but logically vacuous frames, and the continuous scale quantifies exactly how many.

Result Metrics: Binary, Output-Level Scoring

Result Metrics target the final output only — either the static image produced by image editing models or the final frame of a reasoning sequence. Unlike Process Metrics, which ask "how good was the journey?", Result Metrics ask "was the destination correct?"

The scoring function is analogous:

$S_{\text{Result}} = \text{VLM}(I, P, O_{\text{final}}, R_i, R_t, T_{\text{Result}})$

where $O_{\text{final}}$ is the final generated image (rather than the full sequence $O_{\text{seq}}$ in Process Metrics).

The output vector is:

$S_{\text{Result}} = [S_{\text{BC}}, S_{\text{RO}}, S_{\text{VQ}}, S_{\text{RS}}]$

Note the fourth dimension changes from $S_{\text{RA}}$ (Reasoning Accuracy, a continuous trajectory measure) to $S_{\text{RS}}$ (Reasoning Success, a binary final-outcome measure). The first three dimensions — Background Consistency, Rule Obey, Visual Quality — are conceptually parallel to their Process counterparts but use binary {0, 1} scoring rather than continuous 0–100:

$S_{\text{BC}} \in \{0, 1\}$ (Background Consistency): a binary check on whether the output image retains the structural integrity of the input, with irrelevant areas untouched. The template asks: "Does the model output retain the main structure of the input image?"
$S_{\text{RO}} \in \{0, 1\}$ (Rule Obey): determines if the result complies with explicit instructions and essential reasoning constraints. The Maze Navigation template, for instance, checks whether "the red path strictly follows maze rules by staying within open corridors and never crossing or overlapping maze walls."
$S_{\text{VQ}} \in \{0, 1\}$ (Visual Quality): verifies whether the final output maintains high realism and is free from degradation, distortions, or physical implausibility.
$S_{\text{RS}} \in \{0, 1\}$ (Reasoning Success): the critical measure of task completion — does the final state match the reference answer $R_t$ or the reference image $R_i$ ? The paper describes this as evaluating "whether the final state matches the reference answer or the reference image, signifying a correct solution to the reasoning problem."

The aggregated result score is similarly a mean:

$S_{\text{Result}}^{\text{Avg}} = \frac{1}{4} \left( S_{\text{BC}} + S_{\text{RO}} + S_{\text{VQ}} + S_{\text{RS}} \right)$

Why binary for result metrics: This design choice reflects a fundamental distinction between process and outcome evaluation. For the final answer, the field needs to know definitively whether the model solved the problem — partial credit is inappropriate for safety-critical applications where an approximately correct surgery diagram or a nearly-valid physics simulation is still dangerous. The paper's evaluation principles in Appendix C (Table 5–22) are explicit about this: "Assign '1' only if the criterion is fully satisfied with no errors. Partial correctness counts as 0." This is a deliberate philosophical stance: result-level evaluation should be unforgiving because downstream applications that depend on generated outputs being correct cannot tolerate "mostly correct" answers.

Why the same dimensions with different scoring regimes: The parallel structure (BC, RO, VQ in both tracks) enables direct comparison between process quality and output quality. The paper's finding that video models have high Process VQ (Sora 2 Pro: 85.5%) but low Result RS (10.1%) is only interpretable because the dimensions are aligned — we can see that visual quality is maintained throughout the process but reasoning fails at the final integration stage. If the two tracks used different dimensions, such cross-track comparison would be impossible.

Domain-Specific Evaluation Templates: The Template System

The evaluation protocol is not a single monolithic prompt but rather a family of domain-specific templates (reproduced in Appendix C, Tables 5–22 of the paper). Each reasoning subdomain has its own binary template (for Result Metrics) and its own CoT template (for Process Metrics). These templates share a common structure but differ in their evaluation criteria, reflecting the distinct reasoning requirements of each task.

Template structure. Every template follows a consistent pattern of five sections:

Role definition: establishes the judge's persona (e.g., "You are an expert visual reasoning evaluator for step-by-step Sudoku-solving processes").
Task description: provides domain-specific context about what constitutes success (e.g., for Maze Navigation: "The goal of this task is to navigate a 2D maze starting from the green letter 'S' and ending at the red letter 'E'").
Input information structure: specifies the ordering and meaning of images provided to the judge — critically, it identifies which images are model outputs (to be evaluated) versus ground-truth references (for comparison only, not for evaluation). Every template includes the directive: "CRITICAL: Evaluate ONLY the model's output frames, NOT the input or ground truth reference image."
Reference information: injects the instance-specific instruction (the task prompt given to the generative model) and the ground-truth answer (either text or image).
Evaluation dimensions: provides detailed, domain-specific scoring rubrics for each of the four metrics (Background Consistency, Rule Obey, Visual Quality, and Reasoning Accuracy/Success or Beneficial Action). These rubrics operationalize abstract evaluation concepts into concrete, observable criteria.
Output format: mandates a strict JSON structure with named fields and an explanation object.

How templates vary across domains. The evaluation criteria for the same dimension differ substantially depending on the reasoning domain, which is a crucial design feature. Consider Rule Obey:

For Sudoku (Table 10): Rule Obey checks "whether each row, column, and subgrid contains all unique digits without repetition" — a formal logical constraint.
For Jigsaw Puzzle (Table 12): Rule Obey checks whether "each tile is placed once, without overlap or missing regions, and edges align correctly" — a spatial assembly constraint.
For Physical Reasoning (Table 8): Rule Obey checks "sorting criteria (e.g., by color, size, type), placement rules (e.g., specific locations, correct containers), selection criteria (e.g., choosing appropriate tools or items), safety constraints, operational procedures" — a heterogeneous set of embodied task constraints.
For Knowledge Reasoning (Table 5): Rule Obey checks "whether the image is edited according to the instruction" — a simpler instruction-following criterion.

This domain-specificity is what makes ViGoR a genuine diagnostic benchmark rather than a generic quality assessment. A model that scores poorly on Rule Obey in Sudoku but well in Jigsaw Puzzle has a specific deficit in logical constraint satisfaction rather than a general instruction-following problem. The template system enables this granular diagnosis.

The evidence-grounded mechanism. The most critical design element in the templates is the explicit injection of ground-truth references. Every template includes a "Reference Information" section that provides either {gt prompt} (textual ground truth) or identifies a ground-truth image in the input sequence. The VLM judge is instructed to use this reference for comparison:

"Ground Truth: The third image shows the correct completed Sudoku solution for comparison."

The templates also include an evaluation principle that operationalizes evidence-grounding: "Base your reasoning on visual comparison between input, GT, and model output." This transforms the judge from an open-ended assessor (which might hallucinate criteria or apply inconsistent standards) into a comparison engine that need only determine whether the model's output matches the provided reference.

Why external planners are noted for CoT models. In the main results (Table 3), models marked with † (GPT-image-1†, Nano Banana†, Nano Banana Pro†) are flagged as lacking native interleaved generation capabilities. The paper explains that for these models, "we employed external planners (GPT-5 / Gemini-2.5-Pro) to decompose tasks into sequential generation steps." This is an important methodological detail: the evaluation templates for Process Metrics expect a sequence of intermediate frames showing step-by-step reasoning, but some models cannot natively produce such sequences. The paper's solution — using a separate LLM to generate a textual plan, then feeding each step as a separate generation request — creates an apples-to-oranges comparison between models with native interleaved generation (BAGEL-Think, Zebra-CoT, Uni-CoT) and those with externally scaffolded CoT. The paper acknowledges this distinction but does not control for it, which is a potential confound in the CoT vs. non-CoT comparison.

The VLM-as-a-Judge System: Gemini-2.5-Pro as Automated Evaluator

The paper uses Gemini-2.5-Pro (Comanici et al., 2025) as the automated judge for all evaluations, citing its "advanced multimodal understanding capabilities." The choice is motivated by practical considerations — the benchmark needs a judge that can process image sequences, understand domain-specific scoring rubrics, and produce structured JSON outputs at scale. However, the paper's contribution is not the choice of judge but rather the evidence-grounded evaluation protocol that makes the judge reliable.

What the judge receives as input. For each evaluation instance, the judge receives:

The input image (the original problem, e.g., an empty maze or an unsolved Sudoku).
The model output (either a single final image for Result Metrics, or a sequence of frames/video for Process Metrics).
The ground-truth reference (either a reference image showing the correct answer, or a textual description of the correct answer, or both).
The task instruction (the prompt that was given to the generative model).
The evaluation template (domain-specific scoring rubric with explicit criteria for each dimension).

The judge's task is to compare the model output against the ground-truth reference and the instruction, then produce a structured score.

What the judge outputs. For Process Metrics, the output is a JSON object with four integer fields (Background Consistency, Rule Obey, Visual Quality, and Beneficial Action/Reasoning Accuracy, each 0–100) plus an explanation object containing textual justifications for each score. For Result Metrics, the output is four binary fields (0 or 1) plus explanations.

Why structured explanations are included. The templates require the judge to produce not just scores but also natural-language justifications for each dimension. This serves a dual purpose: (1) it enables qualitative inspection of evaluation quality — researchers can read the explanations to verify that the judge's reasoning aligns with the rubric; (2) it forces the judge to ground its scores in observable evidence from the images, reducing the likelihood of hallucinated or inconsistent judgments. The template specifies: "Be concise but precise in explanations" and "Base your reasoning on visual comparison between input, GT, and model output."

The scoring regime distinction. For Process Metrics, the templates emphasize that "Percentages should be integers (e.g., 85 for 85%)" and that the judge should "Evaluate the full temporal sequence of the MODEL'S OUTPUT — consider overall trends, not single-frame anomalies." This encourages the judge to produce a holistic assessment rather than scoring each frame independently and averaging. For Result Metrics, the judgment is holistic by design since there is only one output to evaluate.

Reliability Analysis: Validating Automated Judgments Against Human Experts

This section (3.3 of the paper) is a meta-evaluation: it evaluates the evaluator. The question is whether Gemini-2.5-Pro's automated judgments align sufficiently with human expert judgments to serve as a reliable proxy for large-scale evaluation.

Experimental setup for the meta-evaluation. The paper constructs a "tiny split" from the benchmark generation results:

Result outputs: 1,080 final result outputs from various models, yielding 4,320 metric evaluation instances (each output is scored on 4 binary dimensions).
Process outputs: 540 process sequences, yielding 2,160 metric evaluation instances (each sequence scored on 4 continuous dimensions, but the paper reports 1,064 instances — suggesting some sequences may have been excluded due to format issues, or the counting methodology differs).

Three human experts independently score these instances using the same input information and evaluation templates as the VLM judge. The average of the three human scores serves as the "gold standard." Simultaneously, Gemini-2.5-Pro evaluates the same set over three independent runs under two conditions: with ground-truth references provided (w/ GT) and without ground-truth references (w/o GT). The three-run design is crucial for measuring stability — it reveals whether the judge produces consistent scores across repeated evaluations of the same instance.

Metrics for meta-evaluation. The paper assesses reliability along three dimensions:

MAE (Mean Absolute Error): measures the distributional distance between the VLM's average score (across three runs) and the human expert average. For continuous Process Metrics scored 0–100, MAE is computed directly on the raw scores. For binary Result Metrics, the paper reports MAE on the binary values, meaning the error is either 0 (agreement) or 1 (disagreement) per instance, averaged across all instances. The paper reports MAE values in Table 2: for Process Metrics with GT, MAE = 0.267; for Result Metrics with GT, MAE = 0.213.
Accuracy (categorical agreement): this requires special treatment because Process Metrics use continuous 0–100 scales while Result Metrics use binary 0/1. For Process Metrics, the paper discretizes the continuous scores into three intervals to enable categorical comparison — Bad [0, 33], Moderate [34, 67], and Good [68, 100] — and measures whether the VLM and human average fall into the same interval. For Result Metrics, accuracy is simply the fraction of binary judgments that match. The reported accuracies with GT are 73.3% for Process and 78.6% for Result.
Variance: measures the consistency of the VLM judge across its three independent runs. Lower variance indicates that the judge produces stable, reproducible scores rather than fluctuating between evaluations. The paper reports that the VLM's variance is "comparable to, and in some cases lower than, the inter-annotator variance of human experts," which is a strong validity claim — it suggests the automated judge is not just approximately aligned with humans but is actually more consistent than individual human annotators (even if slightly less aligned with the human consensus).

Key findings from the meta-evaluation (Table 2). The paper draws three conclusions:

High human alignment: With GT references provided, Gemini-2.5-Pro achieves substantially better alignment than without. The MAE drops from 0.319 to 0.267 for Process and from 0.294 to 0.213 for Result, while accuracy rises from 68.0% to 73.3% (Process) and from 70.5% to 78.6% (Result). This empirically validates the paper's central methodological claim: evidence-grounding (providing references) is essential for reliable automated evaluation.
Criticality of ground truth: The gap between w/ and w/o GT is substantial and consistent across all metrics, confirming that the judge's unreliability stems largely from insufficient grounding information rather than inherent incapability. When the judge knows the correct answer, it becomes significantly more reliable.
Stability: The variance of the VLM judge is competitive with human inter-annotator variance. The paper reports human expert variance as 0.051 for Process and 0.011 for Result (these are presumably variance in the continuous scores and binary judgments respectively), while Gemini-2.5-Pro with GT achieves variance of 0.034 (Process) and 0.029 (Result). The automated judge is slightly less variable than humans on Process Metrics (0.034 vs. 0.051) but somewhat more variable on Result Metrics (0.029 vs. 0.011).

What the meta-evaluation does NOT validate. The paper does not report several analyses that would strengthen the reliability claims: (1) per-domain breakdown of alignment — it is possible that the judge aligns well with humans on Knowledge Reasoning but poorly on Symbolic Reasoning, which would affect the trustworthiness of domain-specific findings; (2) inter-annotator agreement among the three human experts (e.g., Fleiss' kappa or Krippendorff's alpha), which would establish the difficulty of the evaluation task itself; (3) analysis of systematic biases — does the judge consistently over-score or under-score certain model families, or certain types of errors? These omissions mean the reliability results should be interpreted as average-case validation rather than comprehensive calibration.

Section 4.3 of the paper uses ViGoR-Bench not just for evaluation but as a training signal source, constructing synthetic maze navigation datasets to fine-tune models and demonstrate that the benchmark's metrics correlate with real capability improvements. This section describes the data construction methodology that enables those experiments.

Training data generation. The paper constructs three distinct training sets, each containing 10,000 synthetic Maze Navigation samples with specific grid dimensions: $4 \times 4$ , $6 \times 6$ , and $8 \times 8$ . Crucially, the evaluation (on ViGoR-Bench) covers grid dimensions from $2 \times 2$ to $7 \times 7$ , meaning the $8 \times 8$ training set is Out-Of-Distribution (OOD) relative to the test set — it contains harder problems than any the model will encounter during evaluation. This is an intentional design choice to test whether training on harder data improves generalization to easier data.

The paper uses Qwen-Image-Edit (versions 2509 and 2511) as base models. These are open-source image editing models, and the maze navigation task requires them to take an input image of an unsolved maze (with green "S" start and red "E" end markers) and produce an output image with a red path connecting S to E without crossing walls.

Training procedure. The paper applies a two-stage training pipeline:

Supervised Fine-Tuning (SFT): The base model is fine-tuned on pairs of (input maze image, correct solution image) from the synthetic training set. This teaches the model what a correct maze solution looks like, but through standard next-token prediction loss — the model learns to imitate correct outputs without explicit reward for correctness.
Reinforcement Learning (RL): Using the SFT model as initialization, an additional phase of RL training is applied with the GRPO (Group Relative Policy Optimization) algorithm (Liu et al., 2025a). The paper does not specify the exact reward function, but given the maze navigation task, the reward is presumably based on whether the generated path correctly connects S to E without wall violations. The paper reports this in the context of demonstrating that "Reward-driven RL demonstrates superior potential in advancing visual reasoning capabilities where SFT exhibits saturation" (Section 4.3, key finding 3).

Why this setup tests specific hypotheses. The three training splits ( $4 \times 4$ , $6 \times 6$ , $8 \times 8$ ) enable testing of two hypotheses simultaneously:

Does RL improve over SFT? By comparing SFT-only and SFT+RL variants on the same base model and training data, the paper isolates the effect of reward optimization. The finding that RL significantly boosts performance (e.g., Qwen-Image-Edit-2511-RL trained on $8 \times 8$ achieves 97.0% Reasoning Success vs. 39.0% for SFT on the same data, per Table 4) provides evidence that explicit reward signals are necessary for reasoning tasks where imitation learning plateaus.
Does OOD training improve generalization? By comparing models trained on $4 \times 4$ vs. $8 \times 8$ data (both evaluated on the same $2 \times 2$ to $7 \times 7$ test set), the paper tests whether exposure to harder problems transfers to better performance on easier problems. The finding that the $8 \times 8$ -trained model achieves the best overall performance (99.0 Avg for Qwen-Image-Edit-2511-RL, Table 4) supports the claim that "training on high-complexity data forces the model to learn the underlying reasoning rules rather than merely overfitting to surface patterns."

A subtle caveat on the training-evaluation relationship. The paper evaluates post-trained models on the maze navigation sub-domain of ViGoR-Bench, which was constructed independently of the synthetic training data. However, the paper does not specify whether the ViGoR-Bench maze navigation samples were generated using the same algorithmic engine as the training data, which would affect the OOD claim. If both training and test mazes come from the same distribution of maze structures (differing only in grid size), then the $8 \times 8$ training is OOD in terms of problem complexity but potentially in-distribution in terms of maze topology — a less stringent test of generalization than if the test mazes were generated by a different algorithm.

The post-training experiments serve a dual purpose in the paper: they validate that ViGoR-Bench metrics are sensitive to real capability improvements (the benchmark's scores correlate with training progress), and they demonstrate that the benchmark can guide model development (researchers can use per-dimension scores to identify weaknesses and target them with training interventions). This positions ViGoR not merely as a "stress test" but as a development tool — a departure from most benchmarks that are designed purely for evaluation.

4. Key Insights and Innovations

Innovation 1: Difficulty-Conditioned Compute-Optimal Test-Time Scaling

The phrase "compute-optimal" is, by this point in the paper, well-understood from Section 3: it refers to the meta-strategy of adaptively allocating test-time compute — search algorithm choice, beam width, sequential-to-parallel ratio — based on estimated prompt difficulty. What makes this contribution intellectually distinctive is not the optimization itself (the field optimizes everything) but the paper's demonstration that test-time compute scaling exhibits qualitatively different, sometimes inverted, behavior across difficulty levels, and that ignoring this heterogeneity leaves 4× efficiency gains on the table.

Prior to this work, the dominant assumption — implicit in best-of-N sampling (Cobbe et al., 2021), self-consistency (Wang et al., 2022), and even the tree-of-thought literature (Yao et al., 2023) — was that test-time compute is a uniform knob: turn it up, and performance improves monotonically, with the only question being how much improvement per unit of compute. This paper falsifies that assumption. The evidence is in Figure 3 (right): on easy questions (difficulty bins 1–2), beam search degrades performance with increasing budget — the PRM over-optimization phenomenon pushes accuracy down as compute increases. On medium questions (bins 3–4), beam search consistently outperforms best-of-N. On hard questions (bin 5), nothing helps at all. These are not differences of degree but of kind: the same strategy produces opposite effects depending on difficulty.

This is a fundamental reframing, not an incremental optimization. It transforms test-time compute from a unidimensional resource (more = better) to a conditional resource whose optimal deployment depends on problem characteristics. The parallel to Chinchilla scaling laws (Hoffmann et al., 2022) — which showed that optimal pretraining compute allocation between model size and data depends on total budget — is explicit and intentional. But the mechanism is entirely different: Chinchilla optimizes over continuous variables (parameters, tokens); this paper optimizes over a discrete space of strategy hyperparameters conditioned on a difficulty estimate. The intellectual contribution is the concept of inference-time scaling laws with difficulty as the key conditioning variable, not any particular optimization algorithm.

The finding that predicted (non-oracle) difficulty bins perform nearly identically to oracle bins (curves overlap in Figures 4 and 8) is the detail that makes this innovation practical rather than merely analytical. If difficulty estimation required ground-truth labels, the approach would be circular — you'd need to know the answer to decide how to find the answer. The PRM's own score distribution serving as a sufficient proxy means the framework is deployable. This closes a loop that could have remained open.

The $4\times$ efficiency figure itself is not the contribution — it's the empirical manifestation of the deeper insight that uniform test-time compute allocation is deeply suboptimal in ways that are predictable from difficulty. This changes how researchers should think about inference budgets: not as "how much can we afford to spend," but as "on which problems should we spend more, and on which should we spend less, and with which mechanism?"

Innovation 2: Verifier Over-Optimization as a First-Class Phenomenon in Test-Time Scaling

Reward hacking and over-optimization are well-documented in the RLHF literature (Gao et al., 2022; Casper et al., 2023). What this paper contributes — and what distinguishes it from prior work — is the empirical demonstration that the same phenomenon governs test-time search scaling and is the primary bottleneck preventing unbounded improvements from additional inference compute. This was not obvious a priori. One might have expected that search against a learned verifier would improve monotonically with budget (more search → better solutions found), saturating only when the search space is exhausted. Instead, the paper shows that search degrades past a verifier-reliability threshold.

The evidence is scattered across multiple figures and forms a coherent pattern. Beam search degrades easy-problem performance at high budgets (Figure 3, right). Lookahead search — which is the most powerful optimizer in the comparison, simulating multiple steps forward to get better step-level value estimates — paradoxically performs worst overall (Figure 3, left). This is the signature of over-optimization: a stronger optimizer exploits the verifier's imperfections more aggressively, producing solutions that score highly under the PRM but are actually incorrect. The qualitative examples in Appendix M (degenerate outputs with repetitive low-information steps, overly short solutions) provide mechanistic evidence for how the exploitation manifests.

Why is this a distinct innovation rather than just another instance of a known phenomenon? Because it shifts the research narrative around test-time compute. Prior to this work, the natural response to limited test-time compute gains would be to develop more sophisticated search algorithms — better tree search, smarter pruning, more efficient exploration. This paper's finding implies that such efforts are largely futile until the verifier is improved: better search optimizers will only exploit the verifier's flaws more efficiently. The bottleneck is verifier robustness, not search algorithm sophistication. This redirects research attention from search algorithm design (a well-studied area) to verifier training and calibration (a relatively underexplored area for generative reasoning).

The compute-optimal policy can be reinterpreted through this lens: it is not just an efficiency mechanism but a mitigation strategy for verifier over-optimization. By routing easy problems away from aggressive search (using best-of-N instead of beam search), it keeps the optimization pressure below the verifier's reliability threshold on problems where the verifier is most vulnerable (because easy problems have high base-model accuracy, meaning the PRM's training data is rich, but also meaning that optimization can amplify any residual errors). On medium problems, where the PRM's guidance genuinely helps, beam search is deployed because the signal-to-noise ratio of the verifier's scores is high enough to withstand optimization.

The paper doesn't solve over-optimization — it documents it and works around it. But the documentation itself is the contribution: it establishes verifier quality as the limiting reagent in test-time compute scaling, which changes the priorities for future research.

Innovation 3: The "Illusion of Reasoning" as a Diagnostic Concept

This is a finding that is easy to overlook because it appears in a single sentence in Section 4.1, but it represents a genuinely novel diagnostic concept that the paper's dual-track evaluation framework makes visible for the first time. The phrase itself — "Illusion of Reasoning" — captures a phenomenon that no prior benchmark could detect: video generation models producing outputs with high process-level visual quality but near-zero result-level reasoning success.

The data in Table 3 makes this stark: Sora 2 Pro achieves 85.5% Process Visual Quality (its intermediate frames look excellent) and 70.5% Process Background Consistency, but only 10.1% Result Reasoning Success. Kling 1.6 achieves 77.0% Process VQ but 1.6% Result RS. Veo 3 achieves 85.3% Process VQ but 8.4% Result RS. These are not marginal discrepancies — they are chasms. The models are producing visually coherent, temporally smooth sequences that look like they're reasoning through a problem, but the reasoning is systematically wrong.

Why is this a conceptual innovation rather than just an unsurprising finding? Because it falsifies the implicit assumption that visual coherence implies logical coherence. Prior evaluation frameworks that assess only final outputs (or only aggregate quality scores) treat visual quality and reasoning correctness as correlated — a model that produces good-looking outputs is presumably "doing well." ViGoR-Bench's dual-track design explicitly separates these dimensions and reveals that they are largely decorrelated in current video generation models. The models have learned to simulate the appearance of reasoning (producing sequences of plausible intermediate states) without learning the substance of reasoning (producing sequences that actually satisfy logical constraints).

This has implications beyond evaluation. For the field of video generation, it suggests that architectural advances that improve temporal consistency and visual quality are not automatically advancing reasoning capability — these may be learned from fundamentally different signals in the training data (low-level pixel prediction vs. high-level logical constraint satisfaction). For the broader AI community, it provides empirical evidence that scaling up generative models on internet-scale data may produce increasingly convincing simulations of reasoning without producing the underlying capability — a finding that resonates with the distinction between system 1 and system 2 cognition.

The diagnostic value is in the separation. Before ViGoR-Bench, a researcher observing Sora 2 Pro's output might be impressed by its smooth motion and conclude it "reasons well about object manipulation." The dual-track metrics reveal that the smooth motion is real but the reasoning is not — the model is an excellent animator and a terrible logician. This decomposition enables researchers to target the right problem (logical reasoning) rather than being misled by proxy metrics (visual quality) into thinking no problem exists.

Innovation 4: OOD Complexity Training as a Generalization Strategy for Visual Reasoning

Section 4.3 reports a finding that is counterintuitive on its surface: training a maze navigation model on $8 \times 8$ grids (harder than any evaluation grid) produces better performance on $2 \times 2$ to $7 \times 7$ test grids than training directly on those in-distribution grid sizes. This is not obviously true — one might expect that training on the hardest examples causes the model to overfit to high-complexity patterns that don't transfer to simpler cases. The data in Table 4 contradicts that expectation: the Qwen-Image-Edit-2511-RL model trained on $8 \times 8$ achieves 97.0% Reasoning Success and 99.0% Average, substantially outperforming models trained on $4 \times 4$ (80.8%) and $6 \times 6$ (93.0%).

What makes this an innovation rather than a standard generalization result is the interaction with the training paradigm. The paper demonstrates that this OOD benefit is contingent on reinforcement learning: SFT alone on $8 \times 8$ data yields only 39.0% Reasoning Success (Qwen-Image-Edit-2511-SFT), while adding RL pushes it to 97.0%. The SFT curve plateaus during training (Figure 6, left), while RL "successfully elevates the model's performance to a new level" (per the paper). This suggests that the mechanism is not simply "more data" or "harder data" but rather RL-driven discovery of generalizable reasoning rules that SFT alone cannot extract from the same examples.

The paper's interpretation is that "training on high-complexity data forces the model to learn the underlying reasoning rules rather than merely overfitting to surface patterns." This is a specific hypothesis about why OOD training works: harder problems admit fewer shortcut solutions. In a $4 \times 4$ maze, a model might learn superficial heuristics (always go right, then down) that happen to work on many $4 \times 4$ configurations but fail on larger grids. In an $8 \times 8$ maze, such heuristics are much less likely to succeed, forcing the model (under RL pressure to find paths that actually reach the goal) to learn algorithms that work regardless of grid size. SFT, which only imitates correct outputs, may not provide sufficient pressure to abandon heuristics that work on $8 \times 8$ grids some of the time — the model can simply memorize the training solutions rather than learning to solve mazes.

This is a significant finding for the field of visual reasoning because it challenges the default assumption of training on in-distribution data. If the result generalizes beyond maze navigation to other structured reasoning tasks (which the paper does not test, but the framework enables), it suggests a general principle: for systematic reasoning tasks, training on harder-than-evaluation problems with RL optimization may be more effective than training on the evaluation distribution directly. This has practical implications for how to construct training curricula for visual reasoning models.

The innovation is not the RL algorithm itself (GRPO is from Liu et al., 2025a) but the empirical demonstration of a specific interaction effect — OOD complexity × RL — that produces a capability jump SFT alone cannot achieve. This positions ViGoR-Bench not merely as an evaluation tool but as a platform for discovering training principles, which is a distinctive framing for a benchmark paper.

Innovation 5: Evidence-Grounded Automated Evaluation as a Validated Paradigm

The "VLM-as-a-Judge" paradigm (using large multimodal models to evaluate generative outputs) has become standard practice, but its reliability has been largely assumed rather than systematically validated. The paper's meta-evaluation (Table 2) makes a specific, empirically grounded argument about when and why automated judges are trustworthy: they are reliable when provided with ground-truth references, and substantially less reliable without them.

The numbers are precise and the design is careful. With GT references, Gemini-2.5-Pro achieves 73.3% accuracy on Process Metrics and 78.6% on Result Metrics, with MAE values of 0.267 and 0.213. Without GT, accuracy drops to 68.0% and 70.5%, with MAE rising to 0.319 and 0.294. The inclusion of three independent human annotators as a gold standard, and the reporting of variance (both human inter-annotator and VLM inter-run), elevates this from an anecdotal calibration to a systematic validation.

What distinguishes this from prior work on VLM-as-a-Judge is the conceptual reframing of the judge's role. Rather than treating the judge as an autonomous evaluator that independently assesses quality, the paper repositions it as a reference-comparison engine — its primary function is to determine whether a model output matches a provided ground truth, not to generate quality judgments from first principles. This is a subtle but important shift. It acknowledges the fundamental limitation that current VLMs are not reliable enough to serve as autonomous judges for complex reasoning tasks, but that they are reliable enough to serve as comparison engines when anchored to correct answers. The evidence-grounding mechanism (explicitly injecting GT into the evaluation template) operationalizes this reframing.

The practical significance is substantial. For benchmark creators, this provides a validated recipe: include ground-truth references in your evaluation pipeline, validate alignment against human experts, and report reliability metrics alongside model performance. For the broader field, it establishes a standard of evidence for automated evaluation claims — a benchmark that uses VLM-as-a-Judge but does not report human-alignment metrics on its specific task distribution is making an unvalidated claim. The paper's own reliability analysis is specific to ViGoR-Bench's task distribution and templates; it does not claim that Gemini-2.5-Pro is a universally reliable judge. This specificity is a feature, not a bug — it models how evaluation validation should work.

The finding that variance of the automated judge is comparable to human inter-annotator variance is the clinching detail. It means the automated pipeline is not just approximately aligned with humans but is actually more consistent than individual human annotators on Process Metrics (0.034 vs. 0.051 variance), though somewhat less consistent on Result Metrics (0.029 vs. 0.011). This suggests a regime where automated evaluation can complement or even replace human evaluation for certain dimensions, provided the grounding is sufficient — a claim the paper supports with evidence rather than assertion.

5. Experimental Analysis

Evaluation Methodology

Dataset. The evaluation uses ViGoR-Bench itself, the benchmark introduced in this paper. It comprises 918 samples spanning three primary reasoning domains (Physical Reasoning, Knowledge Reasoning, Symbolic Reasoning) and 20 subdomains, as detailed in Section 3.1 and Figure 8 in Appendix A. The benchmark covers three task modalities: Image-to-Image (I2I) editing, Sequential Image-to-Image (I2Is), and Image-to-Video (I2V) generation. Each sample provides an input image, a task instruction, and either a ground-truth reference image, a human-verified textual ground-truth answer, or both. The 918 samples are used exclusively for zero-shot evaluation; no training split is created from them.
Base model(s). The paper evaluates over 20 leading generative models, categorized into four distinct groups: Image Editing Models (FLUX.1-Kontext-dev, FLUX.2-dev, Qwen-Image-Edit-2509, Qwen-Image-Edit-2511, LongCat-Image-Edit, Step1X-Edit, HiDream-E1.1, ICEdit), Unified Models without Chain-of-Thought (Bagel, OmniGen2, UniWorld-V1, UniPic2-M-9B, Ovis-U1-3B, DiMOO, Seedream 4.0, GPT-image-1, Nano Banana, Nano Banana Pro), Unified Models with Chain-of-Thought (Bagel-Think, Zebra-CoT, Uni-CoT, GPT-image-1†, Nano Banana†, Nano Banana Pro†), and Video Generation Models (Wan 2.2, Kling 1.6, Seedance 1.0 Pro, Veo 3, Sora 2 Pro). The selection spans both open-source and proprietary systems across diverse architectures (diffusion-based, autoregressive, hybrid) and scales (from 3B to large proprietary models), chosen to represent the contemporary landscape of generative visual AI. For the post-training experiments in Section 4.3, Qwen-Image-Edit versions 2509 and 2511 serve as base models for fine-tuning, selected as representative open-source image editing models amenable to training.
Metrics. The paper employs a dual-track evaluation protocol (fully described in Section 3.2) comprising Process Metrics and Result Metrics, with all scores normalized to a 100-point scale for consistency. Process Metrics (for video generation models and "thinking" models with intermediate reasoning frames) produce four continuous 0–100 scores—Background Consistency (S_BC), Rule Obey (S_RO), Visual Quality (S_VQ), and Reasoning Accuracy (S_RA, also termed "Beneficial Action" in the templates)—with a final aggregated Process Average computed as the arithmetic mean of the four components. Result Metrics (for static image outputs or final frames) produce four binary {0, 1} scores—Background Consistency (S_BC), Rule Obey (S_RO), Visual Quality (S_VQ), and Reasoning Success (S_RS)—with a Result Average similarly computed as the mean of the four binary values. All evaluation is performed by Gemini-2.5-Pro as an automated VLM-as-a-Judge, conditioned on domain-specific evaluation templates and ground-truth references.
Baselines. The paper does not report comparisons against external benchmark scores or baseline methods in the traditional sense—the evaluation is a zero-shot assessment of each model's absolute reasoning capabilities on ViGoR-Bench tasks. The relevant comparison framework is cross-model ranking: models within each category (Edit w/o CoT, Unified w/o CoT, Unified w/ CoT, Video Gen) are compared against each other on identical tasks, and the best performance in each category is bolded in Table 3. The "best overall" performance across all models is marked in bold across categories. For the post-training experiments (Table 4), the baselines are the untrained base models (Qwen-Image-Edit-2509 and 2511) and the top-performing proprietary models (Nano Banana, GPT-Image-1, Nano Banana Pro) evaluated on the same maze navigation sub-domain.
Generation budget / compute accounting. The paper does not measure or control for generation budget (number of inference steps, FLOPs, or wall-clock time) in the main evaluation. Each model is evaluated using its default inference parameters as recommended by official implementations, in a zero-shot setting. This means generation costs are not normalized across models—a model using more inference compute gets no penalty. This is a deliberate choice for a benchmark paper assessing capability ceilings rather than efficiency, but it means the comparisons reflect what each model can achieve with its standard configuration, not what it can achieve under a fixed compute budget. For the post-training experiments, training compute is reported in terms of dataset size (10K synthetic samples per training split) and training stages (SFT followed by RL), but no FLOPs accounting is provided.
Cross-validation / statistical protocol. The paper does not report cross-validation or statistical significance testing for the main benchmark results. Model scores are reported as point estimates without confidence intervals, standard deviations, or error bars. For the reliability meta-evaluation (Section 3.3), the automated judge is run over three independent runs to measure scoring variance, and alignment with human experts is assessed via MAE and categorical accuracy on a "tiny split" (1,080 result outputs yielding 4,320 metric instances; 540 process sequences yielding 1,064 metric instances). Three human experts independently score the same instances, and their average serves as the gold standard. The post-training experiments in Section 4.3 similarly report point estimates without variance measures.

Main Quantitative Results

Overall Model Performance Across Reasoning Domains

Table 3 presents the comprehensive quantitative results for all evaluated models across both Process and Result Metrics. The headline finding is the substantial performance gap between proprietary and open-source models, with proprietary unified models maintaining a decisive lead. Nano Banana Pro (Unified w/o CoT) achieves the best overall Result Average of 68.4% (with component scores of BC: 70.2%, RO: 62.0%, VQ: 95.1%, RS: 46.4%), substantially outperforming the best open-source unified model (Seedream 4.0 at 41.5% Result Average). Among image editing models, the best open-source performer is DiMOO with a Result Average of 34.0% (BC: 73.3%, RO: 13.2%, VQ: 48.0%, RS: 1.4%), while the best proprietary editing model is Nano Banana Pro at 68.4%.

The performance hierarchy within categories reveals consistent patterns. For Image Editing Models, FLUX.1-Kontext-dev leads with 38.9% Result Average, followed by Step1X-Edit (34.6%), DiMOO (34.0%), and Qwen-Image-Edit-2509 (32.2%). The dropoff to HiDream-E1.1 (1.7%) and ICEdit (19.1%) is severe. For Unified Models w/o CoT, Nano Banana Pro (68.4%) leads, followed by Nano Banana (47.7%), GPT-image-1 (46.9%), and Seedream 4.0 (41.5%), with open-source Bagel at 23.1%. For Video Generation Models, Process Metrics are available: Sora 2 Pro leads with Process Average 57.4% (BC: 70.5%, RO: 38.8%, VQ: 85.5%, RA: 34.8%), followed by Veo 3 (55.8%) and Kling 1.6 (43.2%), but all video models show dramatically lower Result Averages—Sora 2 Pro at 17.8%, Kling 1.6 at 30.0%, Veo 3 at 20.5%.

A critical cross-category finding (discussed in Section 4.1 as "Illusion of Reasoning") is the stark discrepancy between Process Visual Quality and Result Reasoning Success for video generation models. Sora 2 Pro achieves Process VQ of 85.5% but Result RS of only 10.1%; Kling 1.6 achieves Process VQ of 77.0% but Result RS of 1.6%; Veo 3 achieves Process VQ of 85.3% but Result RS of 8.4%. The paper states this explicitly: "current video models excel at simulating fluid motion and maintaining visual coherence but struggle to internalize the underlying logical constraints required for rigorous reasoning tasks."

Chain-of-Thought: Interpretability Without Guaranteed Accuracy

Models with explicit Chain-of-Thought prompting (Unified w/ CoT in Table 3) demonstrate a complex relationship between process quality and final accuracy. Nano Banana Pro† (with external CoT planning) achieves a Process Average of 72.0% (BC: 86.0%, RO: 58.6%, VQ: 90.9%, RA: 52.0%)—the highest process scores of any model category—but its Result Average drops to 61.2% (RS: 40.2%). This means that while the CoT-enhanced model produces the highest-quality intermediate reasoning frames, it still fails to produce correct final answers on roughly 60% of tasks. The paper summarizes: "Task decomposition aids in clarifying the reasoning trajectory; however, it does not necessarily compensate for the base model's execution limitations—effectively, a model may 'think' correctly but fail to 'draw' accurately."

The gap between Process Average and Result Average for CoT models is consistently large: GPT-image-1† shows 54.2% Process vs. 31.3% Result; Nano Banana† shows 63.3% vs. 50.5%; Nano Banana Pro† shows 72.0% vs. 61.2%. This pattern does not hold for open-source CoT models: Bagel-Think achieves only 14.6% Process Average and 9.5% Result Average—both poor—suggesting that the benefit of explicit CoT reasoning is contingent on a sufficiently strong base model. The paper notes that "the elongation of the inference chain introduces the risk of error accumulation, where minor execution deviations in early steps cascade into compounded failures in the final output."

Comparing CoT vs. non-CoT variants of the same base model family reveals mixed effects. Nano Banana Pro w/o CoT achieves 68.4% Result Average; with CoT (external planner), it drops to 61.2%. GPT-image-1 drops from 46.9% (w/o CoT) to 31.3% (w/ CoT). However, Nano Banana improves from 47.7% to 50.5%—the only case where CoT improves final accuracy. The paper does not analyze this heterogeneity, but it suggests that CoT's effect is model-specific and not universally beneficial.

Problem Complexity: Monotonic Decline with a Sudoku Anomaly

Figure 5 investigates how model performance scales with problem dimensionality across three symbolic reasoning tasks: Sudoku, Jigsaw Puzzle, and Maze Navigation. For Maze Navigation and Jigsaw Puzzle, the paper reports "a sharp, monotonic decline in Reasoning Success as the grid size expands." This is the expected pattern—larger grids present exponentially larger search spaces and require more complex spatial reasoning. However, Sudoku presents an intriguing inverted-U pattern: "performance peaks at intermediate dimensions but drops at both extremes." The paper hypothesizes this stems from "training data distribution biases, where standard grid sizes are over-represented compared to non-standard variants."

The exact data points are not tabulated in the main text but are visualized in Figure 5. Four models are compared: Nano Banana, Nano Banana Pro, GPT-image-1, and Seedream 4.0. The figure shows Reasoning Success on the y-axis (0.0 to 0.8 scale, suggesting proportions rather than percentages) against grid size on the x-axis (2x2 through 8x8 for Sudoku, 2x2 through 8x8 for Jigsaw, 2x2 through 7x7 for Maze). Nano Banana Pro consistently achieves the highest performance across all tasks and grid sizes, but even it degrades substantially—on Jigsaw Puzzle, its Reasoning Success drops from approximately 0.75 at 2x2 to near 0.0 at 7x7.

Post-Training: Eliciting Reasoning via SFT and RL

Table 4 presents results for Qwen-Image-Edit models fine-tuned on synthetic maze navigation data, evaluated on the ViGoR-Bench maze navigation sub-domain (covering grid dimensions 2x2 to 7x7). The headline result is that RL-trained models dramatically outperform both SFT-only variants and proprietary baselines. Specifically, Qwen-Image-Edit-2511-RL trained on 8x8 data achieves 97.0% Reasoning Success and 99.0% Result Average, compared to Nano Banana Pro's 11.0% RS and 63.0% Avg on the same task. The untrained Qwen-Image-Edit-2511 achieves only 2.0% RS (36.3% Avg).

The effect of training data complexity follows a clear gradient. For the best-performing model variant (Qwen-Image-Edit-2511-RL):

Trained on 4x4: 59.0% RS, 80.8% Avg
Trained on 6x6: 81.0% RS, 93.0% Avg
Trained on 8x8: 97.0% RS, 99.0% Avg

The pattern is monotonic: harder training data produces better generalization to the evaluation distribution. The paper explicitly notes that "the model trained on the strictly Out-Of-Distribution (OOD) and higher-complexity 8x8 data yields the best overall performance" and interprets this as evidence that "training on high-complexity data forces the model to learn the underlying reasoning rules rather than merely overfitting to surface patterns."

The SFT-to-RL gap is substantial and consistent across both model versions (2509 and 2511) and all training data sizes. For Qwen-Image-Edit-2509 trained on 4x4 data: SFT achieves 27.0% RS (56.3% Avg), while RL achieves 60.0% RS (81.3% Avg)—a 33 percentage point gain in RS. For Qwen-Image-Edit-2511 trained on 8x8 data, the gap is even larger: SFT achieves 39.0% RS (61.0% Avg), while RL achieves 97.0% RS (99.0% Avg)—a 58 percentage point gain. Figure 6 visualizes this as training curves, showing that "the performance of the Supervised Fine-Tuning (SFT) model, as measured by validation metrics, reaches a plateau during the course of training" while RL "successfully elevates the model's performance to a new level."

The Background Consistency scores for RL-trained models are uniformly 100.0% across all configurations, indicating perfect preservation of maze structure—a prerequisite for valid path drawing that even top proprietary models fail to achieve (Nano Banana Pro achieves 93.0% BC on the same task). The Rule Obey scores similarly approach ceiling: 99.0% for the best RL model versus 50.0% for Nano Banana Pro.

Fine-Grained Diagnostic Profiling

Figures 10, 11, and 12 (in Appendix B) provide per-subdomain and per-dimension performance breakdowns for five representative models (Nano Banana Pro w/ CoT, Nano Banana w/ CoT, GPT-image-1 w/ CoT, Veo 3, and Sora 2 Pro) plus two additional models for Process Metrics (Nano Banana Pro, FLUX.2-dev) and one additional for Result Metrics (Nano Banana w/ CoT repeated, Nano Banana Pro w/ CoT). The figures are organized as radar-like heatmaps showing performance across subdomains for each of the four Process and four Result dimensions.

Symbolic Reasoning (Figure 10): Leading models demonstrate "consistently strong performance in Algebraic Calculation and Block Building," but "substantially lower performance is observed for combinatorial and structural tasks such as Jigsaw Puzzle, Function Plotting, and Maze Navigation." Across Process Metrics, Background Consistency and Visual Quality remain high across subdomains, while Rule Obey shows "pronounced drops" for tasks requiring strict symbolic constraints. Result-level evaluations "further amplify this gap," with Reasoning Accuracy and Reasoning Success exhibiting "significant degradation on puzzle-oriented tasks." The paper notes that this reveals that "visually plausible intermediate states do not reliably translate into correct symbolic reasoning outcomes."

Physical Reasoning (Figure 11): Models exhibit "consistently strong performance in Visual Quality and Background Consistency across most categories," but "pronounced performance gaps emerge in Rule Obey and Reasoning Accuracy, particularly for tasks involving Measurement & Verification, Object Assembly, and Situational Decision Making." This highlights "persistent challenges in instruction-following and multi-step embodied reasoning." The discrepancy between process and result metrics again indicates that visually plausible intermediate states do not guarantee correct outcomes.

Knowledge Reasoning (Figure 12): Across seven subdomains, models demonstrate "consistently strong performance in Background Consistency and Visual Quality." However, Rule Obey "exhibits noticeably lower and more variable performance, particularly in knowledge domains that require precise factual grounding and temporal or causal reasoning, such as History, Geography, and Sports." Result-level evaluations "reveal a clear performance degradation compared to process metrics," with "Reasoning Accuracy and Reasoning Success remaining limited across multiple knowledge sub-tasks, highlighting persistent challenges in translating visually plausible outputs into correct knowledge-grounded reasoning outcomes."

Ablation Studies and Robustness Checks

Reliability of automated evaluation with vs. without ground-truth references (Table 2): The paper ablates the presence of ground-truth references in the VLM-as-a-Judge pipeline, comparing Gemini-2.5-Pro's alignment with human experts under w/ GT and w/o GT conditions. Including ground-truth references improves Process Metric accuracy from 68.0% to 73.3% and reduces MAE from 0.319 to 0.267. For Result Metrics, accuracy improves from 70.5% to 78.6% and MAE drops from 0.294 to 0.213. The finding that evidence-grounding is critical for reliable automated judging validates the paper's central evaluation design decision. Notably, variance also decreases with GT inclusion (from 0.039 to 0.034 for Process, from 0.034 to 0.029 for Result), indicating that GT anchoring not only improves alignment but also stabilizes the judge across repeated evaluations.

Human expert baseline as gold standard (Table 2): The paper implicitly ablates the evaluation paradigm by establishing human expert performance as a ceiling. Human experts achieve perfect self-consistency (MAE = 0.000, Acc = 1.000 by construction since the gold standard is their own average) with inter-annotator variance of 0.051 for Process and 0.011 for Result. The automated judge with GT achieves variance of 0.034 (Process) and 0.029 (Result)—lower than humans on Process Metrics but higher on Result Metrics. This suggests the automated judge is more consistent than individual humans for continuous trajectory assessments but somewhat less consistent for binary final-output judgments.

SFT vs. RL training comparison (Table 4, Figure 6): The post-training experiments include a direct comparison of Supervised Fine-Tuning alone versus SFT followed by Reinforcement Learning, across two model versions and three training data complexities. This functions as an ablation of the training paradigm. Across all six comparisons (2 model versions x 3 data sizes), RL consistently and substantially outperforms SFT. The SFT performance plateau observed in Figure 6 (left) contrasted with the RL lift in Figure 6 (right) indicates that imitation learning alone saturates while reward optimization continues to improve capability. The paper does not report an RL-only baseline (without SFT initialization), which would strengthen the ablation by showing whether the SFT pretraining is necessary or whether RL from the base model would suffice.

Training data complexity as a generalization factor (Table 4): Comparing models trained on 4x4, 6x6, and 8x8 maze data (all evaluated on the same 2x2 to 7x7 test distribution) ablates the effect of training distribution complexity on generalization. The monotonic improvement with increasing training difficulty—most pronounced for RL-trained models but also visible for SFT—supports the claim that harder training data improves transfer to easier evaluation tasks. However, there is no control condition training on the full distribution of evaluation grid sizes (2x2 through 7x7 mixed), which would establish whether OOD high-complexity training actually outperforms matched-distribution training, or whether it simply outperforms narrow-distribution training on easier splits.

Model version comparison in post-training (Table 4): Qwen-Image-Edit-2509 and 2511 are evaluated as separate base models, with 2511 consistently outperforming 2509 after equivalent training. The untrained 2511 achieves 36.3% Avg vs. 8.0% for 2509 on maze navigation, indicating a meaningful capability difference between versions. This serves as a robustness check showing that the post-training benefits (particularly the SFT-to-RL gap and the OOD generalization effect) replicate across model versions, though the absolute performance levels differ.

Multiple runs for judge stability (Section 3.3): The paper evaluates the VLM judge over three independent runs to measure scoring stability, with variance reported in Table 2. This is an ablation of the evaluation protocol's reproducibility. The finding that Gemini-2.5-Pro's variance (0.034 Process, 0.029 Result) is competitive with human inter-annotator variance (0.051 Process, 0.011 Result) suggests that the automated pipeline produces acceptably consistent scores. The paper does not report whether individual model rankings remain stable across the three runs, which would be a more practically relevant measure of evaluation reproducibility than raw score variance.

Negative result: CoT does not guarantee improved final accuracy (Table 3): The comparison between Unified w/o CoT and Unified w/ CoT models reveals that explicit Chain-of-Thought prompting can degrade final accuracy (Nano Banana Pro drops from 68.4% to 61.2% Result Average; GPT-image-1 drops from 46.9% to 31.3%) despite improving process interpretability. This is a notable negative finding: it demonstrates that making reasoning explicit does not automatically make it correct, and that the same CoT approach that improves performance in text-only LLM reasoning does not transfer straightforwardly to visual generation tasks.

Negative result: Video models' reasoning failure despite high visual quality (Table 3): The "Illusion of Reasoning" finding is itself a negative result—video generation models that achieve excellent Process Visual Quality scores (Sora 2 Pro: 85.5%, Veo 3: 85.3%) produce near-zero Reasoning Success (Sora 2 Pro: 10.1%, Kling 1.6: 1.6%). This is not an ablation in the traditional sense but reveals that temporal coherence and visual fidelity, which are the primary optimization targets in video generation research, do not entail logical reasoning capability. The paper does not ablate whether this gap closes with model scale or architectural changes.

Critical Assessment

Does the evidence support the claim that ViGoR-Bench is a "critical stress test" revealing "significant reasoning deficits"?

Yes, but with an important qualification about what "reasoning deficit" means. The evidence in Table 3 and Figures 10–12 clearly demonstrates that state-of-the-art generative models fail on the specific tasks ViGoR-Bench presents. Nano Banana Pro, the top performer, achieves only 46.4% Reasoning Success on Result Metrics—meaning it fails to correctly solve more than half of the benchmark's reasoning problems. Open-source models fare far worse. The per-dimension breakdowns show that even when models maintain visual quality and background consistency, they systematically fail at Rule Obey and Reasoning Success, particularly on symbolic and physical reasoning tasks.

However, the paper does not establish whether these failures reflect fundamental reasoning incapabilities or task formatting and instruction-following failures that are specific to the benchmark's design. For instance, a model might understand how to solve a Sudoku conceptually but fail because it cannot produce the specific visual output format the evaluation template expects (drawing digits in the correct cells with the correct appearance). The paper's Result Metrics are unforgiving binary judgments—"Assign '1' only if the criterion is fully satisfied with no errors"—which means even a model that demonstrates partial reasoning competence but makes a single visual execution error scores 0 on multiple dimensions. The gap between Process Reasoning Accuracy (continuous, trajectory-level) and Result Reasoning Success (binary, final-outcome) is partially attributable to this scoring strictness. A model with 52.0% Process Reasoning Accuracy (Nano Banana Pro†, Table 3) might be demonstrating substantial reasoning competence throughout its process that gets masked by a strict final-outcome binary when the last step contains a minor execution error. The paper does not analyze whether the Process-to-Result gap reflects reasoning failures or execution failures, which is a significant interpretive limitation.

Additionally, the paper does not establish that ViGoR-Bench tasks are valid proxies for real-world reasoning demands. The benchmark deliberately tests edge cases (complex symbolic puzzles, specific physical manipulation scenarios, specialized knowledge tasks), but it is unclear whether poor performance on these tasks predicts poor performance on practically important reasoning tasks that generative models are actually deployed for (e.g., photo editing with spatial constraints, instructional diagram generation, data visualization). The claim of a "logical desert" is supported within the oasis of ViGoR-Bench, but the paper does not demonstrate that this desert extends to the broader landscape of generative model applications.

Does the evidence support the claim that video models exhibit an "Illusion of Reasoning"?

Strongly supported, subject to the same caveat about distinguishing reasoning vs. execution failures. The data in Table 3 is unambiguous: Sora 2 Pro achieves 85.5% Process VQ and 1.6% Result RS for Kling 1.6; 85.3% Process VQ and 8.4% Result RS for Veo 3. The gap is enormous and consistent. The Process Metrics capture the model's intermediate frames looking visually coherent and temporally smooth; the Result Metrics capture the final output being systematically wrong. The label "Illusion of Reasoning" is justified: these models produce outputs that look like they are reasoning through a problem while actually failing to solve it.

What the paper does not establish is whether this is specific to video models or whether image editing models exhibit the same phenomenon. The comparison is not directly possible because Process Metrics are only reported for video generation models and CoT-enabled unified models—not for standard image editing models, which produce only final outputs and thus have no process to evaluate. It is possible that image editing models have equally poor reasoning but we simply cannot observe their "process" to detect a discrepancy. If image editing models were evaluated on Process Metrics (by, for instance, generating intermediate editing steps), they might show the same illusion. The paper's framing treats the Illusion of Reasoning as distinctive to video models, but the experimental design only makes it visible in video models—not necessarily distinctive to them.

Does the evidence support the claim that CoT "does not guarantee improvement in final accuracy"?

Yes, with an important methodological caveat. Table 3 shows that three of four model pairs with both CoT and non-CoT variants exhibit worse Result Averages with CoT (Nano Banana Pro: 68.4% → 61.2%; GPT-image-1: 46.9% → 31.3%; Bagel: 23.1% → 9.5% for Bagel-Think). Only Nano Banana shows improvement (47.7% → 50.5%). The evidence supports the narrower claim that "explicit CoT does not guarantee improved final accuracy" and, in fact, often degrades it.

However, the comparison between CoT and non-CoT variants is confounded by the external planner design. Models marked with † (GPT-image-1†, Nano Banana†, Nano Banana Pro†) use external LLM planners to decompose tasks into sequential steps, which introduces two potential confounds: (1) the task decomposition itself may introduce errors if the external planner misunderstands the visual input or produces suboptimal step plans, and (2) the multi-step generation process may accumulate errors across steps in ways that a single-pass generation does not. The performance drop from non-CoT to CoT for these models may reflect the quality of the external planner rather than any inherent limitation of CoT for visual reasoning. The paper acknowledges this design choice but does not control for it, making the CoT vs. non-CoT comparison less informative than it appears.

For open-source CoT models with native interleaved generation (Bagel-Think, Zebra-CoT, Uni-CoT), performance is uniformly poor (Result Averages of 9.5%, 23.1%, and 18.0% respectively), but there are no non-CoT variants of these models to compare against. It is impossible to determine whether their poor performance reflects the CoT approach, the base model's limited capability, or both.

Does the evidence support the claim that RL "demonstrates superior potential" over SFT for advancing visual reasoning?

This claim from Section 4.3 is supported by the maze navigation experiments (Table 4) but with significant limitations on generalizability. The finding is clear for this specific task, model family, and RL algorithm: across all six SFT-vs-RL comparisons (2 model versions × 3 training data sizes), RL substantially outperforms SFT, with gains ranging from 21 to 58 percentage points in Reasoning Success. The SFT plateau (Figure 6) and RL lift are consistent.

However, the paper tests only one reasoning task (maze navigation), one model family (Qwen-Image-Edit), one RL algorithm (GRPO), and one dataset construction methodology (synthetic algorithmic generation). The claim that RL "demonstrates superior potential" is stated in general terms but supported by a single case study. We do not know whether RL would outperform SFT on Knowledge Reasoning tasks (where the reasoning is factual rather than procedural), on Physical Reasoning tasks (where the constraints are embodied rather than symbolic), or on other model architectures. The paper also does not ablate the RL algorithm—would PPO or simpler reward-weighted regression produce similar gains? The GRPO reference (Liu et al., 2025a) is cited but no implementation details are provided to assess whether the result is algorithm-specific.

Additionally, the RL reward function is not specified in the paper. The maze navigation task has a natural reward signal (path correctness), but it is unclear whether the reward is sparse (binary success/failure at the final step) or dense (incremental progress toward the goal). This matters because sparse-reward RL is notoriously difficult, and the paper's strong results might depend on reward shaping that would not be available for other benchmark tasks without similarly clean success signals. The paper's claim about RL's superior potential rests on an experiment where the reward is perfectly aligned with the evaluation metric—a best-case scenario that may not generalize.

Does the evidence support the claim that OOD high-complexity training improves generalization?

Supported for maze navigation specifically, but the generalization claim is stated more broadly than the evidence warrants. Table 4 shows a clear monotonic improvement in generalization performance as training data complexity increases from 4x4 to 8x8, with the 8x8-trained model achieving the best test-set performance despite the test set containing only smaller grids (2x2 to 7x7). This is a genuine OOD generalization result.

However, several limitations qualify the strength of this evidence:

The paper does not compare against a baseline trained on the full evaluation distribution (mixed 2x2 through 7x7 data). It is possible that training on the exact test distribution would outperform training on harder OOD data. The claim that OOD training is better than in-distribution training is not directly tested.
The result is demonstrated for a single task (maze navigation) with a simple complexity dimension (grid size). On other tasks—say, Jigsaw Puzzle where complexity also scales with grid size, or Sudoku where the paper found an inverted-U pattern—the OOD-to-ID transfer might not hold. The paper does not test whether increasing jigsaw puzzle training complexity to 8x8 improves 2x2 through 7x7 generalization.
The mechanism proposed ("training on high-complexity data forces the model to learn the underlying reasoning rules") is a post-hoc interpretation, not an experimentally validated mechanism. The paper provides no evidence that the RL-trained model has learned generalizable maze-solving algorithms as opposed to more robust heuristics that happen to transfer to smaller grids. Probing experiments (e.g., testing on maze topologies radically different from the training distribution) would be needed to validate the "underlying reasoning rules" claim.

Missing experiments that would strengthen the paper

Per-model statistical variance: All benchmark results in Table 3 are point estimates without error bars or confidence intervals. Given the benchmark's 918 samples, some subdomains have relatively few samples (Figure 8 shows substantial variation in per-domain counts). Without variance estimates, it is impossible to determine whether the differences between similarly-performing models (e.g., FLUX.1-Kontext-dev at 38.9% vs. Step1X-Edit at 34.6% vs. DiMOO at 34.0% Result Average) are statistically significant or noise. Running each model multiple times with different random seeds and reporting means with standard deviations would address this.

Human performance baseline: The paper positions ViGoR-Bench as measuring reasoning capabilities, but never reports how humans perform on these tasks. A human baseline would contextualize the model scores—is Nano Banana Pro's 46.4% Reasoning Success close to human performance (suggesting models are approaching human-level reasoning) or far below it (suggesting a large capability gap)? The paper has human experts available (three of them scored the meta-evaluation subset), but does not ask them to solve the benchmark tasks themselves.

Ablation of evaluation template design choices: The paper's evaluation templates include domain-specific scoring criteria, ground-truth references, and structured output formats. An ablation removing ground-truth references (already done in the reliability analysis, Table 2) shows substantial degradation. But other template design choices are not ablated: Would simpler templates (without per-dimension textual explanations) produce different scores? Would a different VLM judge (GPT-4o, Claude) produce substantially different rankings? The finding that Gemini-2.5-Pro is reliable for ViGoR-Bench under the specific template design is validated, but the generalizability to other judges or template styles is untested.

Correlation between ViGoR-Bench scores and downstream task performance: The paper demonstrates that post-training on maze navigation data improves ViGoR-Bench maze navigation scores (Table 4), establishing that the benchmark is sensitive to training. But it does not demonstrate that improvements on ViGoR-Bench predict improvements on any external task of practical importance. This is a standard validity concern for benchmarks: does the benchmark measure something that matters outside the benchmark? For ViGoR to serve as a development tool (as the paper claims), researchers need evidence that optimizing for ViGoR scores leads to better models on real-world visual reasoning tasks.

Scaling analysis: The paper evaluates over 20 models of varying sizes but does not systematically analyze how performance scales with model size, training compute, or architecture family. Are reasoning capabilities emergent at certain scales (as has been observed for text LLMs), or do they improve smoothly with model capacity? The current evaluation provides a snapshot of model rankings at a single point in time but does not provide the scaling insights that would help researchers predict future progress or allocate resources. A Chinchilla-style scaling analysis—evaluating a single model family at multiple scales—would be more informative than evaluating many heterogeneous models at their default scales.

Summary of evidential support

The paper's central contribution—a comprehensive, cross-modal, dual-track benchmark with evidence-grounded automated evaluation—is well-supported by the design and validation presented. The benchmark exists (918 samples across 20 subdomains), the evaluation protocol is specified in replicable detail (complete templates in Appendix C), and the automated judge's reliability is validated against human experts on a subset of the benchmark (Table 2). The per-model diagnostic profiling (Figures 10–12) demonstrates the granularity that the dual-track approach enables.

The paper's specific empirical claims about model capabilities are supported with appropriate caveats: proprietary models lead open-source ones (Table 3), CoT does not guarantee accuracy improvement (Table 3, within the external-planner confound), video models exhibit an Illusion of Reasoning (Table 3), and problem complexity degrades performance (Figure 5). The post-training findings—RL outperforms SFT; OOD high-complexity training improves generalization—are supported for the specific maze navigation case study but are not demonstrated to generalize beyond it.

The most significant limitations are: (1) the absence of per-model statistical variance estimates, making ranking reliability uncertain; (2) the lack of a human performance baseline, making absolute capability assessment impossible; (3) the untested generalizability of the maze navigation training findings to other reasoning domains; and (4) the interpretive ambiguity between reasoning failures and visual execution failures in the benchmark's unforgiving binary Result Metrics. The paper's framing as revealing a "logical desert" is evocative but would be strengthened by evidence that ViGoR-Bench failures predict failures on practical visual reasoning tasks outside the benchmark—the desert is mapped, but whether it extends beyond the map's borders is unverified.

6. Limitations and Trade-offs

The Cost of Difficulty Estimation Is Not Accounted for in the Reported Efficiency Gains

The assumption or constraint. The compute-optimal test-time scaling framework depends entirely on knowing each prompt's difficulty before allocating the inference budget. The paper's method for estimating difficulty — generating 2048 samples per question and averaging either ground-truth correctness (oracle bins) or PRM final-answer scores (predicted bins) — is extraordinarily expensive. The authors acknowledge this explicitly in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

The consequence. The headline efficiency claim — "more than 4× better efficiency over a standard best-of-N baseline" — is computed by comparing only the problem-solving compute (generations used to produce the final answer) between compute-optimal and best-of-N strategies, while ignoring the cost of determining which strategy to use. Since the difficulty estimation procedure requires 2048 generations per question, it consumes more compute than the largest test-time budgets studied (which go up to 256–512 generations for problem-solving). In a realistic deployment, the total cost would be 2048 + strategy_execution generations, making the true efficiency relative to best-of-N far lower than 4× — potentially negative, since the total compute exceeds best-of-N's budget in many configurations. The paper's own difficulty estimation cost is never added to any budget comparison.

Furthermore, this limitation is not passive but structural. The compute-optimal policy is a conditional lookup table: for each difficulty bin and budget level, the best strategy is pre-computed offline using validation-set performance. At deployment, this lookup is cheap, but populating the lookup table requires evaluating every candidate strategy at every budget level on every difficulty bin — a cost that scales with the number of strategies × budget levels × questions × 2048 samples per difficulty estimate. The paper amortizes this over the benchmark's 500 test questions, but for a new model or a new problem distribution, this entire process must be repeated.

What evidence exists in the paper. The 2048-sample difficulty estimation procedure is described in Section 3.2 with no cost accounting. Figures 4 and 8 plot compute-optimal scaling curves with x-axis labeled in "generations," which refers only to problem-solving generations. The difficulty estimation cost is mentioned once and never factored into any efficiency calculation. There is no experiment measuring total compute (difficulty estimation + strategy execution) versus best-of-N, and no sensitivity analysis showing how the efficiency claim degrades if difficulty estimation uses fewer samples.

Mitigation status. The authors flag this as a limitation and suggest future work on "pretraining or finetuning models to directly predict difficulty of a question" (Section 8). This is a sensible direction — a lightweight classifier that takes only the question text and predicts difficulty could reduce the estimation cost to a single forward pass — but no such model is developed or evaluated. The paper also floats the idea of adaptive difficulty estimation where the assessment cost is amortized into the problem-solving process itself (Section 3.2), but this is presented as speculation. As of the paper's current results, the 4× efficiency claim should be understood as an upper bound achievable only when difficulty is known for free, which is not the case in any deployment scenario the paper describes.

The $14\times$ Larger Model Baseline Is Weakened by Non-Compute-Optimal Pretraining and Greedy Decoding

The assumption or constraint. The FLOPs-matched comparison in Section 7 compares the base model (PaLM 2-S*) with compute-optimal test-time scaling against a model with approximately $14\times$ more parameters. The paper acknowledges two choices that make this baseline weaker than it could be:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

The larger model scales only parameters while holding training data fixed (following the LLaMA paradigm of Touvron et al., 2023), rather than scaling both parameters and data following Chinchilla-optimal pretraining (Hoffmann et al., 2022). Additionally, the larger model uses only greedy decoding — no majority voting, no best-of-N sampling, no verifier-guided selection.

The consequence. The reported advantages of test-time compute over pretraining compute — for example, +27.8% relative improvement on easy questions at low inference-to-pretraining ratio $R$ (Section 7, Figure 9) — are evaluated against a suboptimal baseline. A Chinchilla-optimal model trained with $14\times$ more total FLOPs (scaling both parameters and data) would likely outperform a parameter-only-scaled model, since it would be trained on more data with a better parameter-data balance. This could substantially narrow or even reverse the reported advantage of test-time compute on some difficulty levels and $R$ regimes.

Worse, the larger model is denied access to any test-time compute strategies at all, despite these being the very techniques the paper advocates. The comparison is effectively: "small model with carefully optimized inference strategy versus large model with zero inference optimization." A fairer comparison would give the larger model a test-time compute budget proportional to its inference cost, or at least allow it best-of-N sampling with a budget equivalent to the smaller model's total compute. The paper's reported crossover points — where test-time compute with the smaller model outperforms the larger model — would shift rightward (requiring more inference budget) or disappear entirely against a properly optimized pretraining baseline.

What evidence exists in the paper. Section 7 explicitly states both design choices: the parameter-only scaling and the greedy decoding for the larger model. Figure 9 plots the larger model's performance as a single star at each $R$ value, and the compute-optimal scaling curves for the smaller model are compared against these single points. There is no ablation where the larger model receives any test-time compute budget, and no comparison against a model trained with Chinchilla-optimal scaling. The paper does not report the effect of giving the larger model even modest test-time compute (e.g., best-of-8 or best-of-16), which would directly test the robustness of the claimed advantage.

Mitigation status. The paper is transparent about the parameter-only scaling choice and frames it as leaving the compute-optimal pretraining comparison to future work. The greedy decoding choice for the larger model is less transparently acknowledged — it is mentioned that the larger model uses greedy decoding, but the paper does not discuss how this choice affects the fairness of the comparison. Neither limitation is tested with ablations. The paper's conclusion that "test-time compute can substitute for pretraining" is appropriately hedged with difficulty-dependent conditions in Section 7, but the specific numerical claims about the magnitude of the substitution (+27.8%, +19.1%, etc.) should be interpreted as evaluated against this specific baseline configuration and may not hold against stronger pretraining baselines.

Hard Problems Remain Fundamentally Unsolved, Establishing a Hard Ceiling on the Approach

The assumption or constraint. The compute-optimal framework assumes that the base model's proposal distribution contains at least some correct solutions for the given prompt — that pass@1 is non-trivially above zero. When this assumption fails, no amount of test-time compute — search, revisions, or any combination — can help, because there are no correct solutions in the distribution to find or refine. The paper explicitly documents this boundary in its difficulty bin analysis.

The consequence. On the hardest difficulty quintile (bin 5), all methods — PRM search, iterative revisions, and their compute-optimal combinations — show near-zero improvement regardless of compute budget. In Figure 3 (right), bin 5 accuracy hovers at 1–3% for all search methods at all budget levels up to 256 generations. In Figure 7 (right), bin 5 shows roughly 2–3% accuracy irrespective of the sequential-to-parallel ratio at 128 generations. In the FLOPs-matched comparison (Figure 9), the bin 5 scaling line is essentially flat near 0–5% across all three $R$ regimes, well below the larger model's performance. The paper states this clearly:

"On the hardest questions (bin 5), no method makes meaningful progress — the base model simply lacks the capability to produce correct solutions regardless of how the budget is allocated."

This is not a gradual degradation but a hard ceiling: test-time compute can amplify existing capability but cannot create it from nothing. If the base model's pass@1 is near zero on a problem class, the approach offers no path forward. For such problems, pretraining (or a fundamentally different model architecture) remains the only viable option. This limits the framework's applicability to problems within the base model's rough capability range — a boundary that is itself unknown at deployment without the kind of expensive difficulty estimation the paper's 2048-sample procedure represents.

What evidence exists in the paper. The bin 5 failure is documented consistently across all experimental sections: search (Figure 3, right; Section 5.3), revisions (Figure 7, right; Section 6.2), compute-optimal search (Figure 4, where bin 5 contributes negligibly to the overall performance), compute-optimal revisions (Figure 8, same), and FLOPs-matched comparison (Figure 9, where bin 5 lines are flat and below the larger model's stars). The paper's takeaway box in Section 7 explicitly states this boundary. No experiment tests whether a different base model architecture, a different training objective, or a different test-time technique could break through this ceiling.

Mitigation status. The paper is frank about this limitation and does not claim to have solved it. The boundary is presented as an empirical finding rather than a failure of the method. However, the paper does not offer a diagnostic for determining, at deployment time, whether a given problem falls beyond this capability ceiling — the 2048-sample difficulty estimation procedure identifies which problems are in bin 5, but the cost of that identification is itself prohibitive. A practical system that could quickly recognize "this problem is unsolvable by me, escalate to a larger model or flag for human review" would make the hardness boundary actionable, but such a system is not developed. The paper's suggestion that the largest model's relative advantage on hard problems supports pretraining over test-time compute (Section 7) is a policy recommendation rather than a technical solution.

The Revision Model Suffers from a 38% Correct-to-Incorrect Reversion Rate with No Principled Solution

The assumption or constraint. The revision model is trained exclusively on sequences where all in-context answers are incorrect, followed by a correct target (Section 6.1). This training data construction — sampling 0–4 incorrect answers then one correct answer, with the last incorrect answer selected for minimum edit distance to the correct answer — teaches the model to revise incorrect answers toward correctness. However, it provides no training signal for what to do when the current answer is already correct. At inference time, when the revision chain happens to produce a correct answer at some step, the model has never seen this situation during training and has no learned behavior for it.

The consequence. The model exhibits a systematic failure mode: approximately 38% of correct answers produced during a revision chain get "revised" back to incorrect answers in the subsequent step (Section 6.1). This means that even as the model's per-step pass@1 improves throughout the chain (Figure 6, left, showing improvement from ~18.2% at step 1 to ~24–25% by steps 15–20), the chain as a whole does not monotonically improve — it oscillates between correct and incorrect states. A user taking only the final output of a long revision chain would often get a wrong answer even though a correct answer appeared earlier in the chain.

The paper mitigates this by applying within-chain selection: a verifier or majority voting evaluates all answers in the chain and picks the best one, rather than always taking the last revision. This is effective but introduces a new dependency: the within-chain selection mechanism must itself be reliable, and in the revision setting the base-model PRM does not transfer well to revision model outputs due to distribution shift (Appendix J, Figure 15a). The paper trains a separate ORM specifically for revision outputs, but this adds engineering complexity and training cost.

What evidence exists in the paper. The 38% reversion rate is reported in Section 6.1. The revision chain accuracy (Figure 6, left) shows non-monotonic behavior consistent with the reversion problem — performance improves but oscillates. The within-chain selection mitigation is described and its necessity is implied by the reversion rate, but the paper does not report what the final-step-only accuracy of a revision chain would be without within-chain selection — the ablation that would quantify exactly how much the reversion problem damages end-to-end performance.

Mitigation status. The paper treats within-chain selection as a sufficient mitigation, and the sequential revision results with verifier-based selection (Figure 6, right) demonstrate that it works well enough to make revisions outperform parallel sampling. However, this is a patch, not a solution. The model itself has not learned to recognize correctness or to stop revising when an answer is adequate. A more principled approach — such as training the revision model on trajectories that include correct answers in context, or adding a "stop revising" token to the training data — is not explored. The ReST $^{EM}$ experiment (Appendix K, Figure 16) provides negative evidence that naïve RL-style optimization of the revision model makes things worse, suggesting the reversion problem is not trivially fixable with more training of the same kind. The paper does not discuss architectural or data-generation changes that could eliminate the reversion behavior at its source, leaving this as an open problem.

Sequential Revisions Incur Latency Penalties That Make Them Impractical for Latency-Sensitive Applications

The assumption or constraint. The paper measures test-time compute in "generations" — the total number of complete solutions sampled — which is a reasonable proxy for total FLOPs or throughput. However, this metric ignores wall-clock latency, which depends on the dependency structure of the computation. Sequential operations (each revision depends on the previous one) incur latency proportional to the chain length, while parallel operations (independent samples) can be executed simultaneously given sufficient hardware.

The consequence. The compute-optimal policies discovered in Sections 5 and 6 heavily favor sequential revisions on easy problems and balanced sequential-parallel hybrids on medium problems (Figure 7, right). At a fixed budget of 128 generations, the optimal ratio for difficulty bins 2–3 involves substantial sequential depth — meaning the computation takes roughly sequential_depth × single_generation_latency wall-clock time, even if enough hardware is available to parallelize the independent chains. In contrast, a purely parallel best-of-N strategy with 128 independent samples takes only single_generation_latency wall-clock time (assuming 128× parallelism). The compute-optimal strategy might therefore be 4× more efficient in total FLOPs but 10–100× slower in wall-clock time, depending on the sequential depth.

This tradeoff matters for any latency-sensitive deployment: interactive assistants (where users wait for responses), real-time systems, or any application where the cost of delay is high. The paper's framework provides no way to incorporate latency constraints into the optimization — the compute-optimal objective (Equation 1) optimizes only for accuracy at a given generation budget, not for accuracy at a given latency budget. A practitioner who needs answers within 5 seconds cannot use a strategy that takes 50 seconds regardless of its FLOP efficiency.

What evidence exists in the paper. None. The paper never discusses latency, wall-clock time, or the dependency structure of computation. All budgets and efficiency claims are in terms of "generations" or FLOPs. The sequential-to-parallel ratio analysis (Figure 7) reports accuracy as a function of generation budget and ratio, with no corresponding latency axis. There is no experiment measuring end-to-end wall-clock time for any strategy, and no discussion of how hardware parallelism assumptions affect the feasibility of different allocation strategies.

Mitigation status. Not addressed. The paper treats compute as a unidimensional resource measured in generations, implicitly assuming that all generations are equivalent in cost. In practice, sequential generations are far more expensive in latency terms, and the paper's recommended strategies (heavily sequential for easy problems) may be completely impractical for many deployment scenarios. Future work that incorporates a latency dimension into the compute-optimal framework — or that develops strategies that achieve similar accuracy with lower sequential depth — would be necessary to make the approach deployable in latency-constrained settings. The paper does not flag this as a limitation or suggest it as future work, making it a blind spot in the otherwise careful efficiency analysis.

7. Implications and Future Directions

How This Work Changes the Landscape

ViGoR-Bench is best understood not as a paradigm shift but as a diagnostic reorientation — it changes what the field measures rather than how models are built. The paper's core contribution is methodological: a framework that decomposes "visual reasoning" from a monolithic, vaguely-defined aspiration into a set of specific, measurable cognitive dimensions, and then validates that decomposition against human judgment. This matters because the field has been optimizing generative models against metrics (FID, CLIP-Score, aesthetic scores) that are decorrelated from reasoning correctness — the paper's "Illusion of Reasoning" finding is the empirical smoking gun. Sora 2 Pro can score 85.5% on Process Visual Quality while scoring 10.1% on Result Reasoning Success. That is not a small gap; it is evidence that the metrics we have been using are measuring the wrong thing.

The landscape shift this produces is primarily evaluative rather than architectural. Before ViGoR-Bench, a researcher developing a new video generation model could report strong FID and CLIP scores, show a few cherry-picked examples of plausible object interactions, and claim their model "understands physics." After ViGoR-Bench, that claim becomes testable — and the paper shows it is likely false for current models. The benchmark makes previously invisible failures visible, which changes the burden of proof: a model that cannot demonstrate above-chance Reasoning Success on ViGoR-Bench's systematic reasoning tasks should not be described as possessing visual reasoning capability, regardless of its visual fidelity. This is a standards-raising contribution analogous to what GLUE and SuperGLUE did for natural language understanding — not a new technique, but a new accountability mechanism that forces the field to confront the gap between surface fluency and underlying competence.

The paper also resolves a latent tension in the literature that is less explicit than the conflicts the example summary paper addressed (where Huang et al. said self-correction fails while Madaan et al. said it helps), but is equally real. The generative AI community has been operating under two competing narratives: (1) that scaling up models on internet-scale data produces increasingly capable and general systems (the "scaling hypothesis"), and (2) that generative models lack genuine understanding and merely perform sophisticated pattern matching (the "stochastic parrot" critique). ViGoR-Bench provides a framework for adjudicating between these narratives on specific, measurable dimensions. The finding that even the best proprietary models fail on more than half of the benchmark's reasoning tasks, and that CoT prompting can degrade accuracy (Nano Banana Pro dropping from 68.4% to 61.2% with CoT), provides evidence for the stochastic parrot position on reasoning tasks. But the finding that RL post-training on maze navigation can push a relatively weak open-source model to 97.0% Reasoning Success — dramatically surpassing all proprietary models on that subdomain — provides evidence that these capabilities are learnable with the right training signal. The tension is not resolved in favor of either extreme position; rather, ViGoR-Bench provides the measurement infrastructure that makes the question empirically tractable rather than merely philosophical.

A less obvious but important landscape change is the paper's reframing of the VLM-as-a-Judge paradigm. The dominant approach in generative model evaluation has been to treat large multimodal models as autonomous judges — present them with a model output and an instruction, ask them to score quality, and trust the result. The paper's meta-evaluation (Table 2) demonstrates that this approach is substantially unreliable without ground-truth anchoring: Gemini-2.5-Pro's accuracy drops from 78.6% to 70.5% on Result Metrics when ground-truth references are removed, and MAE increases from 0.213 to 0.294. The paper's alternative — treating the VLM as a reference-comparison engine rather than an autonomous judge — is a methodological shift that should influence how future benchmarks are designed. If this standard is adopted, the field will move away from "VLM evaluates your model's output quality" benchmarks toward "VLM determines whether your model's output matches a verified ground truth" benchmarks. The latter is empirically more reliable and conceptually more defensible: it acknowledges the VLM's limitations while leveraging its genuine capability for visual comparison.

Finally, the paper's post-training experiments in Section 4.3 hint at a direction that could change the research priorities of the generative AI field. The finding that RL on OOD high-complexity data produces capability jumps that SFT cannot achieve — and that the resulting model can surpass proprietary systems on specific reasoning tasks — suggests that training methodology may matter as much as model scale for visual reasoning. If this result generalizes beyond maze navigation, it implies that the current focus on scaling up pretraining (bigger models, more data) may be missing a more tractable path: taking existing models and training them on carefully constructed reasoning data with reward optimization. The paper does not make this claim explicitly, and the evidence is limited to a single case study, but the 97.0% Reasoning Success achieved by a fine-tuned Qwen-Image-Edit model versus 11.0% for the best proprietary model is a provocative data point that should redirect attention toward post-training as a reasoning-enhancement strategy.

Follow-Up Research This Work Enables

Replicating the OOD complexity training effect across reasoning domains and model families. The maze navigation result — that training on 8×8 grids with RL produces better generalization than training on in-distribution sizes — is the paper's most actionable finding, but it is demonstrated for exactly one task, one model family, and one RL algorithm. A direct replication study would test this on the other ViGoR-Bench symbolic reasoning tasks (Sudoku, Jigsaw Puzzle, Klotski Puzzle, Function Plotting) using Qwen-Image-Edit and at least one other open-source image editing model (e.g., FLUX.1-Kontext-dev) as the base. The key measurement is whether the OOD complexity gradient (4×4 to 8×8 training data improving 2×2–7×7 test performance) replicates across tasks that have different underlying reasoning structures — Sudoku requires constraint satisfaction, jigsaw puzzles require spatial assembly, function plotting requires mathematical computation. If the effect replicates only for path-finding tasks (maze, possibly Klotski) and not for constraint-satisfaction or mathematical tasks, that would refine our understanding of which reasoning skills benefit from OOD training and suggest that the mechanism is related to learning algorithmic procedures rather than declarative knowledge. If it fails to replicate with a different base model architecture (e.g., a non-Qwen diffusion model), that would indicate the effect is architecture-specific and not a general principle.

Determining whether ViGoR-Bench scores predict practical visual reasoning capability. The benchmark claims to measure "visual reasoning," but does not demonstrate that performance on ViGoR-Bench correlates with performance on any external task of practical importance. A validation study would correlate ViGoR-Bench scores (particularly the per-dimension breakdowns for Rule Obey and Reasoning Success) against human judgments of model outputs in realistic deployment scenarios: instructional diagram generation (does the model produce anatomically correct biology diagrams?), photo editing with spatial constraints (does the model respect real-world object relationships when editing photographs?), and data visualization (does the model produce mathematically accurate charts?). This would use a subset of ViGoR-Bench tasks that have natural real-world analogs — Knowledge Reasoning tasks like biology diagram completion and geography map labeling, Physical Reasoning tasks like object sorting and spatial arrangement — and compare model rankings on ViGoR-Bench against expert human ratings of the same models' outputs on the real-world tasks. If the rankings correlate strongly (Spearman's ρ > 0.8), ViGoR-Bench is validated as a development tool. If they do not, the benchmark measures a capability that does not transfer to practice, and the field needs to understand why — perhaps the benchmark's deliberately adversarial edge cases do not represent typical deployment difficulties.

Ablation of the evaluation template design to identify the active ingredients in evidence-grounded judging. The paper's meta-evaluation (Table 2) shows that providing ground-truth references improves VLM judge reliability, but it does not isolate which aspect of the template design matters most. A systematic ablation would test variations of the evaluation templates (provided in Appendix C, Tables 5–22) on the same "tiny split" used in the reliability analysis, measuring human alignment under each variant: (a) template with full GT (image + text), (b) template with text GT only, (c) template with image GT only, (d) template with no GT but detailed scoring rubrics, (e) template with no GT and minimal rubrics, (f) template with GT but no explanation requirement (JSON scores only). This would reveal whether the reliability benefit comes primarily from the visual ground-truth image, the textual ground-truth description, or the structured rubric itself. If text GT alone provides most of the benefit (for tasks where it is available), that has practical implications: benchmark creators could economize on expensive ground-truth image collection. If the explanation requirement is crucial for reliability (because it forces the judge to ground its scores in observable evidence), that validates the paper's template design and provides guidance for future benchmark designers. If different dimensions benefit differently from different grounding mechanisms — say, Background Consistency benefits most from visual GT while Rule Obey benefits most from textual rubrics — that would enable dimension-specific template optimization.

Testing whether the "Illusion of Reasoning" is specific to video models or universal across generative architectures. The paper's finding that video generation models exhibit a large gap between Process Visual Quality (Sora 2 Pro: 85.5%) and Result Reasoning Success (10.1%) is presented as distinctive to video models, but the comparison is confounded by the fact that only video models and CoT-enabled models have Process Metrics reported. An experiment that forces all model categories through the same Process evaluation pipeline — for instance, by having image editing models produce a sequence of intermediate editing steps (even if they don't natively support this, using the same external-planner decomposition the paper uses for CoT models marked with †) — would test whether image editing models exhibit the same illusion. If FLUX.1-Kontext-dev or DiMOO, when forced to output step-by-step editing sequences, also show high Process VQ but low Result RS, then the illusion is a general property of current generative models, not something distinctive about video architectures. The implication would be that temporal coherence training (which video models receive and image models generally do not) produces the appearance of reasoning capability without the substance, but the underlying capability deficit is universal. If, conversely, image editing models show a much smaller Process VQ-to-Result RS gap when evaluated on process sequences, that would suggest that video training specifically teaches models to simulate reasoning dynamics without learning the underlying logic — a finding with implications for how video generation models should be trained (perhaps by incorporating explicit reasoning supervision rather than relying on next-frame prediction alone).

Developing and validating a cheap difficulty estimator for adaptive test-time compute allocation. The paper's compute-optimal framework (discussed extensively in Sections 3–4) is bottlenecked by the cost of difficulty estimation — 2048 samples per prompt is completely impractical for deployment. A practical follow-up would train a lightweight difficulty classifier that takes only the question text and input image as input and predicts the difficulty quintile. The training data already exists: the paper has already computed oracle difficulty bins for all test-set questions using the 2048-sample PRM-based procedure. A classifier (a small vision-language model, or even a CLIP-based similarity scorer comparing the prompt embedding to embeddings of known-easy and known-hard prompts) could be trained on the 500 test questions (with cross-validation to avoid contamination) to predict the bin. The evaluation would measure: (a) classification accuracy (how often does the predicted bin match the PRM-based bin?), (b) downstream impact (does using predicted bins for compute-optimal allocation achieve comparable efficiency gains to oracle bins, as measured by the gap between the two curves in Figure 4 and Figure 8?), and (c) cost amortization (at what number of queries does the cost of building the classifier plus its per-query cost become lower than the cost of online difficulty estimation?).

RL reward function analysis for visual reasoning tasks beyond maze navigation. The paper's Section 4.3 uses maze navigation because it has a natural reward signal (path correctness), but most ViGoR-Bench tasks do not have similarly clean, programmatic reward functions. A systematic study would design reward functions for the other symbolic reasoning tasks — Sudoku (reward = fraction of correctly filled cells that satisfy constraints), jigsaw puzzle (reward = image similarity to ground truth), algebraic calculation (reward = symbolic equality of answer), function plotting (reward = mean squared error against ground-truth curve) — and test whether RL with these rewards produces similar SFT-to-RL capability jumps as observed for maze navigation. The critical measurement is whether the RL gain depends on having a perfect reward signal (as maze navigation does — a path either reaches the goal or it doesn't) or whether approximate, learned, or heuristic rewards also produce the jump. If only tasks with programmatic ground-truth rewards benefit from RL, that limits the practical applicability of RL for visual reasoning to tasks where such rewards are available. If learned reward models (e.g., using a ViGoR-Bench-trained verifier as the reward signal) also work, that opens the door to applying RL to the full benchmark, potentially producing models that show reasoning improvements across all 20 subdomains simultaneously.

Practical Applications and Downstream Use Cases

Filtering and selecting generative models for reasoning-intensive deployment scenarios. Organizations building applications that require generative models to produce logically coherent visual outputs — automated diagram generation for textbooks, instructional illustration for technical documentation, visual data reporting for business intelligence — can use ViGoR-Bench as a model selection tool today. The benchmark's per-dimension scores (particularly Rule Obey and Reasoning Success) directly measure the capabilities these applications need, unlike standard metrics like FID or CLIP-Score which measure visual similarity to a reference distribution. A publisher choosing between Nano Banana Pro (62.0% Rule Obey, 46.4% Reasoning Success) and GPT-image-1 (29.3% Rule Obey, 13.4% Reasoning Success) for generating textbook diagrams would have a clear, quantified basis for their decision: Nano Banana Pro is more than 3× more likely to follow the specified rules and more than 3× more likely to produce a factually correct diagram, despite both models producing visually impressive outputs. The benchmark also enables subdomain-specific selection — a team building a maze-based educational game would see from Table 4 and the post-training results that even a fine-tuned open-source model (Qwen-Image-Edit-2511-RL at 97.0% RS on maze navigation) dramatically outperforms the best proprietary option (Nano Banana Pro at 11.0% RS on the same subdomain), making the case for task-specific fine-tuning over off-the-shelf API usage.

Guiding post-training investments for model improvement. The paper's Section 4.3 provides a concrete, validated recipe for improving visual reasoning in generative models: construct synthetic task-specific training data, apply SFT, then apply RL with a task-aligned reward signal, using OOD high-complexity examples. For an organization with access to a fine-tunable image generation model (Qwen-Image-Edit, FLUX.1-Kontext-dev, or similar), this is immediately actionable. The cost structure is clear from the paper: 10K synthetic training examples (generated algorithmically, so marginal cost near zero), SFT training until validation plateau, then RL training using GRPO until convergence. The expected gain for maze-navigation-like tasks is dramatic — from single-digit Reasoning Success to near-ceiling — though the paper does not establish how many subdomains this recipe transfers to. A team could start by replicating the maze navigation result on their own infrastructure, then extend to the other symbolic reasoning subdomains one at a time, using ViGoR-Bench to measure progress. The benchmark's subdomain structure (20 distinct tasks) makes this incremental approach feasible: you don't need to improve on everything at once; you can target specific weaknesses identified by the diagnostic profiling.

Automated quality assurance for generative model outputs in production pipelines. The evidence-grounded VLM-as-a-Judge pipeline validated in Section 3.3 can be deployed as an automated quality assurance system for production generative models. When a model generates an output for a user query, the same Gemini-2.5-Pro-based evaluation that scores the benchmark can score the production output against expected ground truth (if available) or against task-specific constraints. The reliability analysis (Table 2) provides the confidence bounds: with ground-truth references, the judge achieves 73.3% accuracy on Process Metrics and 78.6% on Result Metrics compared to human experts. This is not high enough for fully autonomous QA without human oversight, but it is high enough for a triage system: flag outputs with low Reasoning Success scores for human review, automatically accept outputs with high scores, and route borderline cases to a queue for spot-checking. For a deployment processing thousands of generation requests per day, even 78.6% accurate automated triage would dramatically reduce the human review burden compared to reviewing all outputs manually. The key practical consideration is that the ground-truth references must be available — this works for tasks where the correct answer can be pre-computed or retrieved (textbook diagrams with known correct answers, maze puzzles with algorithmically generated solutions, mathematical plots where the function is known) but not for open-ended creative generation where no ground truth exists.

When to Prefer This Method

The paper itself does not position ViGoR-Bench against named alternative benchmarks in a prescriptive tradeoff framework — it is presented as a comprehensive evaluation framework that subsumes more narrowly-scoped prior benchmarks (RISE, KRIS-Bench, GIR-Edit, etc.) rather than competing with them on a specific axis. Table 1 makes this explicit by showing that ViGoR-Bench is the only benchmark that covers I2I, I2Is, and I2V tasks simultaneously, the only one that provides both process and result evaluation with full image and text ground truth, and one of only two (with UniREdit) that evaluates process-level metrics for image editing. There is no scenario the paper describes where a researcher would prefer a more narrowly-scoped benchmark over ViGoR-Bench for evaluating visual reasoning — the paper argues that existing benchmarks are incomplete rather than making a different tradeoff.

The closer the paper comes to articulating a preference tradeoff is in the tension between evaluation depth and evaluation cost. ViGoR-Bench requires ground-truth references (images or verified text), domain-specific scoring templates, and a capable VLM judge (Gemini-2.5-Pro) — this is substantially more infrastructure and per-evaluation compute than reference-free metrics like CLIP-Score or aesthetic predictors. For a researcher who only needs to quickly rank model checkpoints during training on a perceptual quality metric, ViGoR-Bench is overkill. For a researcher who needs to determine whether their model genuinely reasons about visual content, ViGoR-Bench is currently the most comprehensive option, and the paper's reliability analysis provides the confidence bounds that alternative VLM-as-a-Judge benchmarks do not.

The paper also implies a preference for evidence-grounded evaluation over open-ended VLM judging, based on the Table 2 results showing that ground-truth anchoring substantially improves alignment with human experts. This is not presented as a conditional tradeoff but as a design principle: whenever ground truth is available, it should be included in the evaluation template. When ground truth is not available (e.g., for genuinely open-ended creative tasks), the paper's framework does not apply, and the reliability of automated evaluation degrades — this is not a choice between methods but a boundary on where ViGoR-Bench's methodology is valid. The benchmark deliberately constrains itself to tasks with verifiable correct answers, which is both its strength (enabling rigorous evaluation) and its limitation (excluding open-ended generation).