Video models are zero-shot learners and reasoners

ArXiv: 2509.20328

🎯 Pitch

A single video model—Veo 3—can segment objects, solve mazes, edit images, predict buoyancy, and more without ever being trained on those tasks, matching or approaching specialized models on many benchmarks. Its chain-of-frames reasoning lets it visually navigate a 5×5 maze with 78% pass@10, while prompt sensitivity alone can swing performance by up to 64 points—revealing that what these models can do depends almost as much on how we ask as on what they inherently know.

1. Executive Summary

This paper studies the emergent zero-shot capabilities of video models—specifically Veo 3—across a broad vision stack, demonstrating that a single generative video model can solve tasks it was never explicitly trained for, from perception (edge detection, segmentation) through modeling (intuitive physics, buoyancy) and manipulation (image editing, dexterous object interaction) to early visual reasoning (maze solving, visual symmetry). The central named mechanism is chain-of-frames (CoF)—frame-by-frame video generation as a visual analog to chain-of-thought in language models, enabling step-by-step reasoning across time and space (e.g., navigating a red circle through a maze without crossing walls). Across 7 quantitative tasks, Veo 3 shows pass@10 rates of 93% on object extraction, 78% on 5×5 maze solving, and 88% on visual symmetry (shapes)—with the pass@1 performance gap between best and worst prompts on symmetry reaching 40–64 percentage points, underscoring the critical role of visual prompt engineering. A FLOPs-matched-style comparison reveals substantial and consistent improvement from Veo 2 to Veo 3—Veo 3 achieves 78% pass@10 on 5×5 mazes versus Veo 2's 14%—while establishing that performance remains a lower bound on the model's true competence due to the sensitivity of task framing, and that the hardest reasoning problems (irregular mazes, rotate/reflect analogies) remain largely unsolved, indicating that current video models can amplify existing visual understanding but cannot yet create it from nothing on problems far from their training distribution.

2. Context and Motivation

The Core Problem: Vision Lacks a Unifying Foundation Model Paradigm

The fundamental question this paper tackles is whether video models are developing general-purpose visual understanding in the same way that large language models (LLMs) developed general-purpose language understanding. Today's machine vision landscape is dominated by task-specific, bespoke models: SAM for segmentation, YOLO variants for object detection, specialized architectures for edge detection, depth estimation, and image editing. Each of these models solves one narrow problem well, but none can be prompted to solve any visual task—the way GPT-4 can be prompted to translate, summarize, write code, or solve math problems without architecture changes or fine-tuning.

This paper asks whether this fragmentation is a temporary state—whether the same primitives that unified NLP (large generative models trained on web-scale data) are now producing the same unification in vision through video models. The question matters because it bears on whether the field should continue investing in task-specific vision architectures or pivot toward prompting generalist video foundation models, as NLP did with LLMs.

Why This Problem Matters Now

The timing is not arbitrary. Three converging factors make this question urgent:

The NLP precedent is fresh and economically decisive. Within a few years, NLP went from a landscape of specialized models (one for translation, one for question-answering, one for summarization) to LLMs as unified foundation models. The economic and practical consequences were enormous: instead of maintaining dozens of task-specific pipelines, practitioners now prompt a single model. If vision is on the same trajectory, the implications for deployment, research prioritization, and resource allocation are similarly large-scale. The paper explicitly draws this parallel: "Machine vision today in many ways resembles the state of NLP a few years ago" (Section 1).

Video models are reaching a capability threshold where emergence becomes observable. The paper studies Veo 3, a model that sits at a specific point on the scaling curve—powerful enough that zero-shot behaviors are starting to appear, but not so powerful that they are saturated. This is analogous to the GPT-3 moment in NLP, where a model crossed a threshold and suddenly exhibited few-shot learning behaviors that were not explicitly programmed. The paper's core empirical claim is that Veo 3 may represent a similar inflection point for vision: a model that can segment objects, detect edges, simulate physics, manipulate scenes, and solve mazes—all without task-specific training.

The cost trajectory of generalist models follows a predictable pattern. The paper notes (Section 5) that LLM inference costs are falling by factors of 9× to 900× per year for a given performance level, and that early LLMs were also considered prohibitively expensive at launch. If this pattern holds for video models, the economic argument for specialized vision models erodes over time, making it important to establish now what generalist video models can and cannot do.

Where Existing Approaches Fall Short

The paper identifies several limitations in the current vision landscape that motivate the shift toward video models as generalist foundation models:

Task-specific models cannot generalize without adaptation. Models like SAM (Kirillov et al., 2023) and SAMv2 (Ravi et al., 2024) achieve state-of-the-art segmentation but cannot perform edge detection, depth estimation, or image editing without architectural changes or fine-tuning. YOLO variants (Redmon et al., 2016; Khanam and Hussain, 2024) detect objects but cannot reason about physical properties or simulate manipulations. Each model is a point solution, and the field has accumulated a large collection of such point solutions—each excellent at its task, none capable of cross-task transfer. The paper contrasts this with the NLP trajectory, where "prompting a model with a task instruction replaces the need for fine-tuning or adding task-specific inference heads" (Section 1).

Existing unification attempts are incomplete. The paper acknowledges that efforts to unify vision tasks exist—Taskonomy (Zamir et al., 2018), generalist in-context visual learners (Wang et al., 2023; Lin et al., 2025; Li et al., 2025), unified image generation models (Xiao et al., 2025; Le et al., 2025), and video editors like Dreamix (Molad et al., 2023). However, the authors argue that "no existing model can solve any problem just by prompting" (Section 1). These models typically cover a subset of vision tasks—often generation-focused—but do not span the full vision stack from perception through modeling through manipulation to reasoning. A model that can edit images may not be able to detect edges; a model that can segment objects may not be able to reason about mazes.

The emergence of zero-shot visual capabilities in video models is poorly characterized. While there is growing evidence that diffusion models and video models exhibit some zero-shot behaviors—diffusion classifiers (Clark and Jaini, 2023), zero-shot segmentors (Burgert et al., 2022), image-to-video generation from text-to-image models (Khachatryan et al., 2023)—these studies typically examine a single capability in isolation. No prior work has conducted a systematic, multi-capability investigation across the full hierarchy of visual understanding to assess whether a single video model can serve as a general-purpose vision system. The paper's scope—18,384 generated videos across 62 qualitative and 7 quantitative tasks—is unprecedented in breadth for this type of investigation.

Physical reasoning in video models is studied in silos, not as part of a unified capability stack. Several works have benchmarked intuitive physics in deep models (Riochet et al., 2018; Bear et al., 2021; Weihs et al., 2022; Bansal et al., 2024; Motamed et al., 2025), but these studies typically focus on a specific physical property (e.g., collision dynamics, object permanence) and evaluate dedicated models or prompting strategies for that property. The paper argues that physical modeling is not an isolated capability but rather the second tier in a hierarchy where perception enables modeling, modeling enables manipulation, and manipulation enables reasoning. By testing a single model across this entire hierarchy, the paper can assess whether capabilities genuinely build on each other—whether a model that can perceive edges can also model buoyancy, manipulate objects, and reason about mazes, all zero-shot.

The parallels between chain-of-thought in language and step-by-step generation in video are unexplored. Chain-of-thought (CoT) prompting (Wei et al., 2022) showed that language models could reason by generating intermediate steps before producing a final answer. The paper argues that video generation is inherently sequential—frames are generated one after another—and that this frame-by-frame generation is the visual analog of chain-of-thought, enabling step-by-step reasoning across time and space. The paper terms this "chain-of-frames" (CoF). While individual works have used video models for planning or navigation (e.g., Yang et al., 2024), no prior work has explicitly framed the sequential nature of video generation as a general reasoning mechanism that parallels CoT, nor systematically tested it across reasoning tasks like maze solving, graph traversal, water puzzle solving, and visual analogy completion.

How This Paper Positions Itself

The paper positions itself not as introducing a new method but as providing evidence for a paradigm shift. This is explicitly signaled in the approach: "Our method is simple: We prompt Veo. This minimalist strategy is intentional, as it mirrors the transformation of NLP from task-specific fine-tuning or training to prompting a capable foundation model" (Section 2). The contribution is empirical and conceptual, not algorithmic.

The conceptual framework is a four-tier capability hierarchy (Section 3, Figure 1):

Perception: Foundational ability to understand visual information (edge detection, segmentation, keypoint localization, super-resolution, denoising, low-light enhancement, conjunctive search, ambiguous image interpretation).
Modeling: Building on perception to form a model of the visual world (intuitive physics, buoyancy, flammability, rigid/soft body dynamics, optics, color mixing, categorization, memory of world states).
Manipulation: Meaningfully altering the perceived and modeled world (background removal, style transfer, colorization, inpainting, outpainting, text manipulation, doodle-based editing, scene composition, novel view synthesis, 3D-aware reposing, transfiguration, dexterous manipulation, affordance recognition).
Reasoning: Integrating perception, modeling, and manipulation to reason across space and time (graph traversal, tree BFS, sequence completion, connecting colors, shape fitting, sorting, tool use, Sudoku, water puzzles, maze solving, robot navigation, rule extrapolation).

This hierarchy is not just taxonomic—it embodies the paper's central claim that capabilities build on each other. Solving a maze (reasoning) requires perceiving the maze layout (perception), understanding that walls are impassable barriers (modeling), and moving an agent through the space (manipulation). The hierarchy provides a framework for understanding why video models might develop general-purpose vision: because the generative objective of predicting future frames forces the model to develop all four tiers simultaneously, just as the language modeling objective forced LLMs to develop syntax, semantics, world knowledge, and reasoning.

The paper explicitly draws a historical parallel: "Veo 3 is akin to a pre-trained language model that has yet to undergo instruction tuning or RLHF" (Section 5). This positioning is critical—it frames the current results as a lower bound. Just as GPT-3's few-shot performance was substantially below fine-tuned models on many tasks (a point the paper explicitly cites from Brown et al., 2020, Tables 3.1, 3.3, 3.4, 3.5), Veo 3's zero-shot performance is below specialized SOTA models. But the paper argues this did not stop LLMs from becoming foundation models, and it will not stop video models. The rapid improvement from Veo 2 to Veo 3—released roughly six months apart—is presented as evidence that the trajectory is steep, and that inference-time scaling (pass@10 consistently outperforming pass@1 with no signs of plateau) plus standard post-training techniques (instruction tuning, RLHF, automatic verifiers) will likely close the gap with specialized models.

A subtle but important positioning choice: the paper treats the entire Vertex AI system (prompt rewriter + video generator) as a single black-box entity (Section 2). The authors acknowledge that for some tasks, "the solution is likely to come from the LLM instead of the video" (e.g., Sudoku in Figure 55). To isolate the video model's specific contribution to reasoning, they verify that a standalone LLM (Gemini 2.5 Pro) cannot reliably solve key tasks from the input image alone (Figure 58: Robot navigation, Section 4.5: Maze solving, Section 4.6: Visual symmetry). This is a crucial control that distinguishes genuine visual reasoning from textual reasoning by an upstream LLM component.

The paper also positions itself within the broader "emergence" literature. The concept of emergent abilities—capabilities that appear at scale without being explicitly trained—was documented for LLMs by Wei et al. (2022). This paper extends that lens to video models, arguing that Veo 3's zero-shot perceptual, modeling, manipulation, and reasoning abilities are emergent. The authors are careful not to overclaim: they describe the reasoning abilities as "early forms" and "sparks of visual intelligence" (Section 3), explicitly noting that performance is "far from perfect" and that failure cases exist (Appendix D documents failures on monocular depth estimation, surface normal estimation, force prompting, knot tying, Eulerian paths, word search, and more). This measured framing distinguishes the paper from exaggerated claims about AGI while still making the case that something qualitatively new is happening.

Finally, the paper positions the chain-of-frames (CoF) concept as a direct parallel to chain-of-thought. Just as CoT "enables language models to reason with symbols" (Section 3, Takeaway 3), CoF "enables video models to reason across time and space." The analogy is structural: in CoT, each generated token is an intermediate reasoning step; in CoF, each generated frame is an intermediate reasoning step. In maze solving, for example, each frame shows the agent's updated position—the video is the trace of the reasoning process, not just its output. This framing is significant because it suggests that video generation is not merely a product (a visually plausible video) but a process (a step-by-step computation) that can be harnessed for general visual problem-solving.

3. Technical Approach

3.1 Reader Orientation

The "system" in this paper is not a new architecture or training method—it is a minimalist prompting protocol applied to the publicly available Veo 2 and Veo 3 video generation models through the Vertex AI API, treating the entire system (prompt rewriter plus video generator) as a single black-box entity and exploring what capabilities emerge when a video model is asked to solve tasks it was never explicitly trained for. The problem this system addresses is the fragmentation of machine vision into task-specific models, and the "shape" of the solution is a single generalist model that can be prompted—with an initial image and a text instruction—to perform perception, modeling, manipulation, and reasoning tasks in a zero-shot manner, with its frame-by-frame video generation process serving as a visual analog to chain-of-thought reasoning in language models.

3.2 Big-Picture Architecture (Diagram in Words)

The system has four interconnected components that form a pipeline:

Input Specification: For each task, the user provides an initial image (which the model uses as the first video frame) and a text prompt describing the desired visual transformation or solution. The image serves as both the visual context and the starting state; the text specifies what operation to perform.
Prompt Rewriter (Vertex AI Internal Component): The Vertex AI API automatically applies a LLM-based prompt rewriter that may expand, clarify, or restructure the user's text instruction before passing it to the video model. The paper treats this rewriter as part of the black-box system, acknowledging that for some reasoning tasks (e.g., Sudoku, Figure 55) the solution may partly originate from the LLM component rather than the video model itself.
Video Generation Model (Veo 2 or Veo 3): The core component—a large generative video model trained on web-scale data with a text/video continuation objective—takes the (rewritten) prompt and the initial image, then generates a 16:9 video at 720p resolution, 24 FPS, for a duration of 8 seconds (192 frames total). The generation process is autoregressive frame-by-frame, meaning each generated frame conditions on all previous frames and the prompt. This sequential generation is what the paper terms "chain-of-frames" (CoF)—the visual analog of chain-of-thought where each frame represents an intermediate reasoning step.
Output Extraction and Evaluation: For quantitative tasks, the generated video is analyzed frame-by-frame to extract task-specific outputs (edge maps from specific frames, segmentation masks, maze solution paths, symmetry completions, analogy solutions from the last frame). The paper distinguishes between "best frame" evaluation (taking the optimal frame across the entire video, representing a performance ceiling) and "last frame" evaluation (taking only the final frame, which is predetermined but may underestimate performance since Veo tends to continue animating beyond task completion).

Information flows sequentially: input image + text prompt → Vertex API prompt rewriter → Veo model autoregressive frame generation → per-frame output extraction → task-specific evaluation. The process is repeated multiple times (typically 10 or 12) with different random seeds to estimate pass@k metrics.

3.3 Roadmap for the Deep Dive

First, the core generation interface and model specification—what exactly changes when the user prompts Veo, the temporal and spatial constraints, and the critical distinction between the two model versions (Veo 2 and Veo 3) used for scaling comparisons.
Second, the chain-of-frames (CoF) mechanism—how frame-by-frame autoregressive generation functions as a reasoning primitive, why it parallels chain-of-thought, and what constraints it imposes on task design and performance evaluation.
Third, the four-tier capability hierarchy that structures all task evaluations—how perception, modeling, manipulation, and reasoning are defined, how they build on each other, and what specific task designs instantiate each tier.
Fourth, the task-specific prompting and evaluation protocols for the seven quantitative tasks—edge detection, segmentation, object extraction, image editing, maze solving, visual symmetry, and visual analogy completion—including the exact prompts, datasets, metrics, and sampling strategies.
Fifth, the distinction between best-frame and last-frame evaluation, the pass@k metric formulation, and why this matters for understanding model competence versus performance.
Sixth, the LLM-disentanglement control: how the paper verifies that visual reasoning comes from the video model rather than the upstream prompt rewriter or a standalone LLM.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical capability investigation paper whose core idea is that large-scale generative video models, trained solely on a text/video continuation objective with web-scale data, develop emergent zero-shot abilities spanning the full vision stack—from perception to reasoning—without any task-specific training, and that the sequential frame-by-frame generation process (chain-of-frames) provides a general mechanism for step-by-step visual reasoning analogous to chain-of-thought in language models.

The Generation Interface: Prompting Veo as a Black Box

The paper's methodological approach is deliberately minimalist: the authors do not fine-tune, adapt, or modify the video model in any way. Instead, they query the publicly available Veo 2 or Veo 3 models through Google Cloud's Vertex AI API, providing exactly two inputs per task: an initial image and a text instruction. The initial image serves as the first frame of the generated video—it is the visual context that the model conditions on and may modify throughout the generation. The text instruction (the "prompt") describes the task in natural language, specifying what visual transformation, analysis, or reasoning operation the model should perform.

The API returns a generated video with fixed specifications: 16:9 aspect ratio at 720p resolution, 24 frames per second, for a duration of 8 seconds. This means every generated video contains exactly 192 frames (24 FPS × 8 seconds). The two model versions have distinct API identifiers: Veo 3 uses model ID veo-3.0-generate-preview and Veo 2 uses veo-2.0-generate-001. Veo 2 was announced in December 2024 and released in April 2025; Veo 3 was announced in May 2025 and released in July 2025—released roughly six months apart, enabling a natural scaling comparison between successive model generations.

A critical architectural detail that the paper acknowledges but does not control: the Vertex AI API uses a LLM-based prompt rewriter. According to the Vertex documentation cited by the authors, this rewriter may modify, expand, or restructure the user's text prompt before passing it to the video model. The paper treats the entire system—rewriter and video generator—as a single black-box entity. This means that for some tasks, particularly those involving symbolic reasoning, the solution could partially originate from the LLM rewriter rather than the video model itself. The authors explicitly flag this for Sudoku solving (Figure 55), where the solution "is likely to come from the LLM instead of the video."

To disentangle the video model's genuine visual reasoning abilities from potential LLM contributions, the paper implements a specific control: for key reasoning tasks (maze solving, robot navigation, visual symmetry), the authors test whether a standalone LLM (Gemini 2.5 Pro) can solve the task from the input image alone. If the standalone LLM cannot solve it, then any successful solution from Veo must involve genuine visual reasoning by the video model—the LLM rewriter alone cannot explain the performance. The paper reports that Gemini 2.5 Pro cannot reliably solve these tasks from image input, establishing that the video model's contribution is necessary for the observed reasoning capabilities.

The choice of Veo as the specific model is pragmatic, not fundamental: the authors state that "the core argument of this paper—that video models are zero-shot learners and reasoners—can be supported by demonstrating success on any sufficiently capable model." Veo is chosen because it "has consistently ranked high on text2video and image2video leaderboards" (Section 2). The paper's claims are about video models in general, not about Veo specifically; Veo serves as the existence proof.

Chain-of-Frames (CoF): The Core Reasoning Mechanism

The paper's central conceptual contribution is the chain-of-frames (CoF) mechanism, which posits that the sequential, frame-by-frame nature of video generation is not merely a rendering artifact but a fundamental reasoning primitive—the visual analog of chain-of-thought (CoT) in language models. To understand CoF, we must first understand what CoT does and why it works, then examine the structural parallel in video.

Chain-of-thought in language models (background). In Wei et al. (2022), chain-of-thought prompting was shown to elicit reasoning in LLMs by having the model generate intermediate reasoning steps before producing a final answer. For example, solving a math word problem via CoT involves generating: "The store had 15 apples. It sold 3 in the morning, so 12 remain. Then it sold 5 in the afternoon, so 7 remain. Final answer: 7." Each token is generated auto-regressively, conditioning on all previous tokens. The key insight is that the sequential generation process itself is the reasoning computation—the model uses its own previous outputs as intermediate representations to compute the next step.

The structural parallel in video generation. In Veo's video generation, each frame is generated auto-regressively, conditioning on all previous frames and the text prompt. The paper argues that this is structurally identical to CoT, but operating in pixel space rather than token space. Consider maze solving (Figure 57, Section 4.5): the initial frame shows a maze with a red circle at the start and a green circle at the goal. Frame by frame, the red circle moves one step along the correct path. Each frame is an intermediate reasoning step—it records the agent's current position, which was computed by conditioning on the maze layout (the input image), the previous frames (the path history), and the task description (the text prompt). The final frame shows the circle at the goal. The video is not just an animation of a pre-computed solution; the video is the computation of the solution.

The paper formalizes this parallelism in Takeaway 3 (Section 3): "Frame-by-frame video generation parallels chain-of-thought in language models. Just like chain-of-thought (CoT) enables language models to reason with symbols, a 'chain-of-frames' (CoF) enables video models to reason across time and space."

What CoF enables that static outputs cannot. The chain-of-frames mechanism has specific properties that make it suitable for visual reasoning in ways that a single-image output would not be:

State decomposition: Complex reasoning problems (maze solving, graph traversal, sorting) can be decomposed into a sequence of intermediate states. Each frame captures one state, making the reasoning trace observable and verifiable. This is visible in the quantitative maze solving results (Figure 7): the evaluation explicitly checks frame-by-frame for illegal moves (crossing walls, jumping over barriers) because each frame is an intermediate computation step that must be physically valid.
Error correction via the generative process: Because each frame conditions on all previous frames, the model can adjust its trajectory mid-generation. This is analogous to how CoT allows language models to self-correct. The paper documents this implicitly: Veo 3's pass@10 substantially exceeds pass@1 on maze solving (78% vs. some lower rate for a single attempt), suggesting that some generated paths are valid and others are not—the generation process can succeed or fail, and multiple attempts allow selecting a successful one.
Spatial grounding of temporal reasoning: In tasks like robot navigation (Figure 58), the object must move continuously through space while respecting physical constraints (walls, obstacles). CoF enforces this by construction: the video generator, having been trained on physically plausible videos, tends to generate frames that maintain spatial continuity. The red circle in the maze "slides smoothly along the white path"—it cannot teleport, because teleportation would violate the learned distribution of object motion in training data.
Visual backtracking: For tasks that require exploration (e.g., complex mazes), CoF allows the model to visually backtrack—move the agent forward, encounter a dead end, and retrace steps. The paper does not explicitly evaluate backtracking capability but notes that Veo 3 shows "zero-shot maze solving abilities, significantly outperforming Veo 2 which often produces illegal moves" (Section 4.5). The illegal moves in Veo 2 (which the paper explicitly evaluates for: "jumping over walls, clipping through boundaries, or any alteration of the goal's position") represent failures of the CoF mechanism to maintain spatial consistency. Veo 3's improvement suggests better learned spatial reasoning in the autoregressive generation.

The motion prior and its implications for CoF. A recurring observation throughout the paper is that "Veo has a strong prior to keep things moving." This is not an implementation bug—it is a direct consequence of the training objective. The model was trained to generate videos, which are inherently dynamic. When prompted with an image-to-image task (e.g., edge detection, segmentation, symmetry completion), the model will often generate the correct output in an early frame but then continue modifying the scene, potentially degrading the solution. This is why the paper distinguishes between best-frame and last-frame evaluation (see Section 3.4.5 below), and why prompting best practices (Appendix C) recommend providing a "motion outlet"—a spinning ball, a glowing dot, or some other dynamic element that satisfies the model's prior to keep something moving without disrupting the primary task output.

The Four-Tier Capability Hierarchy

The paper organizes all tasks into a four-tier hierarchy (Figure 1, Section 3) where each tier builds on the capabilities of the previous ones. This hierarchy is not merely taxonomic—it is the paper's conceptual architecture for understanding why video models might develop general-purpose vision: because the generative training objective (predicting future frames given past frames and text) inherently requires all four tiers, and the emergence of tier-N capabilities depends on tier-(N-1) capabilities being in place. The hierarchy is:

Tier 1: Perception—the foundational ability to understand visual information. This includes extracting structure from raw pixels: edge detection (Figure 10), instance segmentation (Figure 11), keypoint localization (Figure 12), super-resolution (Figure 13), blind deblurring (Figure 14), blind denoising (Figure 15), and low-light enhancement (Figure 16). It also includes more complex perceptual judgments: conjunctive search (finding the blue ball among distractors, Figure 17, which tests the binding problem—distinguishing which features belong to which object), dalmatian illusion interpretation (recognizing the dog in an ambiguous image, Figure 18), shape-cue conflict understanding (seeing the animal shape despite texture cues, Figure 19), and Rorschach blot interpretation (Figure 20). The paper argues that these perceptual abilities are emergent: "Apart from denoising—the classic diffusion objective—none of these tasks are explicitly trained for in video models" (Section 3, Perception).

Tier 2: Modeling—building on perception to form a model of the visual world and the principles that govern it. This tier includes intuitive physics: flammability (Figure 21), rigid and soft body dynamics (Figure 22), air resistance and gravity (Figure 23, contrasting Earth vs. moon falling behavior), buoyancy (Figure 24, distinguishing floating bottle cap vs. sinking stone), and Visual Jenga (Figure 25, removing objects in physically plausible order). It also includes optical phenomena (refraction through glass vs. reflection from mirror, Figure 27), color mixing (additive with lights vs. subtractive with paints, Figure 28), abstract categorization (distinguishing toys from a laptop, Figure 29), Omniglot-style character recognition, generation, and parsing (Figure 30), and memory of world states across camera movements (Figure 31: zooming in on a person and zooming back out to find the scene unchanged). This tier requires the model to have internal representations of object properties (mass, material, flammability), physical laws (gravity, buoyancy, collision dynamics), and object persistence across time and viewpoint changes.

Tier 3: Manipulation—meaningfully altering the perceived and modeled world. This tier represents the model's ability to not just passively observe but actively modify a scene: background removal (Figure 32), style transfer (Figure 33), colorization (Figure 34), inpainting (Figure 35), outpainting (Figure 36), text manipulation (Figure 37), doodle-based editing (Figure 38), scene composition (Figure 39, blending a zebra into a new background), novel view synthesis from a single image (Figure 40), 3D-aware reposing (Figure 41, having a knight turn and kneel while maintaining 3D consistency), transfiguration (Figure 42, transforming a teacup into a mouse), professional headshot generation from a selfie (Figure 43), dexterous manipulation (Figure 44, including jar opening and ball throwing with robot hands), affordance recognition (Figure 45, picking up a hammer by its handle), drawing (Figure 46), and visual instruction generation (Figure 47, demonstrating how to roll a burrito).

Tier 4: Reasoning—integrating perception, modeling, and manipulation to reason across time and space. This tier represents the paper's most ambitious claim: that video models can solve problems requiring multi-step planning. Tasks include: graph traversal (Figure 48, water flowing through a connected channel system), visual breadth-first search on a tree (Figure 49, water filling caves level by level), visual sequence completion inspired by Raven's progressive matrices (Figure 50), connecting matching colors without crossing lines (Figure 51), fitting shapes into holes (Figure 52), sorting numbers (Figure 53), tool use (Figure 54, retrieving a walnut from an aquarium using a net without breaking the glass), simple Sudoku (Figure 55), water puzzle solving (Figure 56, filling connected containers in the correct order), maze solving (Figure 57), robot navigation (Figure 58), and rule extrapolation inspired by ARC-AGI (Figure 59).

The hierarchy is cumulative: maze solving (reasoning) requires perceiving the maze layout (perception) to distinguish walls from paths, modeling the walls as impassable barriers (modeling) to understand the constraints on movement, and manipulating the agent's position (manipulation) to move step-by-step from start to goal.

Task-Specific Prompting and Evaluation Protocols (Quantitative Tasks)

The paper conducts seven quantitative experiments, each with carefully designed prompts, datasets, evaluation metrics, and sampling strategies. Understanding these protocols is essential because the performance numbers (pass@k values) are jointly determined by the model's capability, the prompt design, and the evaluation method—changing any of these can substantially alter results.

Edge Detection (Section 4.1, Appendix B.1)

Task: Given an input image (a natural photograph), generate a video where edges become salient black outlines against a white background, and all objects fade away. This is an image-to-image transformation evaluated via the generated frames.

Dataset: All 50 test images from BIPEDv2 (Soria et al., 2020, 2023), a standard edge detection benchmark where ground-truth edge maps are annotated by humans.

Prompt (Veo):

"All edges in this image become more salient by transforming into black outlines. Then, all objects fade away, with just the edges remaining on a white background. Static camera perspective, no zoom or pan."

Prompt (Nano Banana reference):

"Outline all edges in the image in black, make everything else white."

Evaluation metric: Optimal Image Scale (OIS), which is the maximum F1-score over all detection thresholds, allowing for small local shifts between predicted and ground-truth edge pixels (within a radius of 0.75% of the image diagonal, approximately 11 pixels). The evaluation pipeline proceeds as follows:

Extract the relevant frame (best frame or last frame) from each generated video.
Apply non-maximum suppression to thin edge responses.
Binarize at 16 evenly-spaced thresholds.
Thin the binary edge map.
At each threshold, find the optimal mapping between predicted and ground-truth edge pixels within the local shift radius.
Compute the F1-score (harmonic mean of precision and recall) at each threshold.
Take the maximum F1-score across thresholds as the OIS for that frame.

Sampling: 10 videos per sample (50 samples × 10 videos = 500 videos per model). Two models tested: Veo 2 and Veo 3. Total: 1000 videos. Additionally, Nano Banana is evaluated as an image-editing baseline (generating a single output image, not a video).

Key design choice—why edge detection?: The paper chooses edge detection as a representative perceptual task because it is well-studied, has standard benchmarks and metrics, and is something no video model is explicitly trained to do. Super-resolution and denoising are arguably closer to the training objective (since diffusion models inherently denoise during generation), but edge detection—extracting structural information from pixel intensities—is a qualitatively different operation. The fact that Veo 3 can perform edge detection zero-shot suggests it has learned internal representations of edges that can be accessed through prompting.

Important nuance in evaluation: Figure 60 (Appendix B.1) shows a graded edge map, highlighting a systematic issue: "many false positives correspond to actual image details that seem to be erroneously excluded from the ground truth (e.g., the outline of the trees, the reflection in the car window, and the tire profiles)." This means the reported OIS scores (Veo 3: 0.77 pass@10, SOTA: 0.90) are penalized because Veo 3 detects more edges than the human annotators labeled—edges that are genuinely present in the image. The quantitative metric thus underestimates Veo 3's perceptual ability due to incomplete ground truth, a point the paper explicitly makes.

Class-Agnostic Instance Segmentation (Section 4.2, Appendix B.2)

Task: Given an input image containing 1–3 large objects, generate a video where each distinct entity (object) is overlaid with a different flat color, and the background fades to a specified color (white or green). This is "class-agnostic" because the model is not told what object categories exist or how many objects are present—it must discover this from the image.

Dataset: 50 randomly chosen test images from LVIS (Gupta et al., 2019) that contain 1–3 objects, each with at least 5000 pixels (ensuring the objects are large enough to segment and evaluate reliably). The LVIS dataset provides ground-truth instance segmentation masks for each object.

Prompt (Veo):

"Create an animation of instance segmentation being performed on this photograph: each distinct entity is overlaid in a different flat color. Scene: • The animation starts from the provided, unaltered photograph. • The scene in the photograph is static and doesn't move. • First, the background fades to {white, green}. • Then, the first entity is covered by a flat color, perfectly preserving its silhouette. • Then the second entity, too, is covered by a different flat color, perfectly preserving its silhouette. • One by one, each entity is covered by a different flat color. • Finally, all entities are covered with different colors. Camera: • Static shot without camera movement. • No pan. • No rotation. • No zoom. • No glitches or artifacts."

Prompt (Nano Banana reference):

"Perform instance segmentation on this image: Mask each distinct entity in a different opaque flat color that only preserves the silhouette and turn the background green."

Evaluation metric: Mean Intersection over Union (mIoU). The evaluation pipeline is non-trivial because the model chooses arbitrary colors for segmentation masks, so colors must be matched to ground-truth instances:

Extract the frame from the generated video (best frame or last frame).
Compute the hue-difference histogram between the original input image and the extracted frame (pixels that changed hue significantly are likely segmentation masks).
Smooth the histogram with a Gaussian filter (standard deviation = 2).
Detect peaks in the smoothed histogram: peaks must have minimum height of 10% of the maximum peak and be separated by at least 10 hue steps. Each peak corresponds to one predicted segmentation mask color.
Map each pixel in the output frame to the mask with the closest hue.
For each ground-truth instance mask (including background), find the predicted mask with the highest IoU (intersection over union). This establishes a matching between ground-truth instances and predicted masks.
Compute mIoU as the average IoU over all matched pairs, excluding background.

Why two background colors (white vs. green): The paper tests prompts with both white and green backgrounds to probe whether the model has learned associations from training data. The result: Veo consistently performs better with a green background than white (0.74 vs. 0.66 best frame pass@10), which the authors hypothesize is "possibly due to the widespread use of green screens" in video data. This highlights an important property of the system: the model's zero-shot performance is influenced by statistical regularities in its training data, not just by the specified task semantics.

Sampling: 10 videos per sample per prompt variant (50 samples × 2 background colors × 10 videos = 1000 videos per model). Two models tested: Veo 2 and Veo 3. Total: 2000 videos.

Key design choice—class-agnostic segmentation: Unlike standard instance segmentation benchmarks where the model knows the list of possible object categories and is given location priors (bounding boxes, points), this evaluation is fully open-vocabulary and location-free. The model must simultaneously discover how many objects exist, where they are, and what their boundaries are—using only the text instruction. This makes the task substantially harder than standard segmentation but more representative of general visual understanding.

Object Extraction (Section 4.3, Appendix B.3)

Task: Given an input image containing 1–9 animals of the same species in a natural landscape, generate a video where the background changes to white and all animals line up in a row with white space between them. The task tests simultaneous perception (identifying all animals), modeling (understanding that each animal is a distinct entity), and manipulation (moving the animals to a new arrangement). Success is binary: the final frame must contain the correct number of animals (as counted by connected components in a binary mask).

Dataset: A custom dataset of 54 images (6 images per count, for counts 1 through 9) generated using Nano Banana with the prompt: "Exchange the white space with a realistic photograph of: exactly {number} {animal}, outside, not overlapping, in a natural landscape," where animal is drawn from {dog, elephant, cat, brown bear, horse, rabbit, raccoon}. The generated dataset was manually verified for correctness.

Prompt (Veo):

"The background changes to white. Then: • If there is just a single animal: the animal sits in the middle of the image, looking straight at the camera. • If there are multiple animals: all animals line up in a row, with ample white space between them."

Evaluation metric: Pass@k, where a generated video passes if the number of animals extracted in the last frame matches the ground-truth count. The extraction procedure:

Extract the last frame from each generated video.
Convert to grayscale.
Apply a binary threshold at pixel value 200 (separating white background from darker animal pixels).
Count connected components using scipy.ndimage.label.
Compare the count to the ground-truth number of animals.
A video passes if the counts match exactly.

Chance baseline: The probability of correctly guessing the count (uniform over 1–9) is 1/9. With k independent attempts, the random-chance pass@k is:

$P_{\text{chance}}(k) = 1 - \left(1 - \frac{1}{9}\right)^k = 1 - \left(\frac{8}{9}\right)^k$

where $k$ is the number of attempts (1–10). For k=10, this equals approximately 69.2%.

Sampling: 10 videos per sample (54 samples × 10 videos = 540 videos per model). Two models tested: Veo 2 and Veo 3. Total: 1080 videos.

Key design choice—why animal counting?: Object extraction tests a specific combination of capabilities: the model must count objects (a perceptual and modeling task), separate them from the background (manipulation), and arrange them in a new configuration (spatial reasoning). The simplicity of the binary pass/fail criterion (correct count = pass) makes evaluation unambiguous. The authors note that "given the simplicity of the task, a perfect model should easily achieve 100% accuracy," establishing a clear performance ceiling.

Image Editing (Section 4.4, Appendix B.4)

Task: Edit an input image according to a text instruction (e.g., change the color of a fire hydrant to red), producing a video where the edit is applied without unintended changes (camera movement, animating static elements, altering unrelated parts of the scene).

Dataset: A random subset of 30 images from the test set of Emu-edit (Sheynin et al., 2024), a benchmark for text-based image editing with diverse edit types (color changes, object additions/removals, scene modifications).

Prompt (Veo):

"Create a smooth, static animation that slowly {image specific edit direction}. Do not change anything else. No zoom, no pan, no dolly."

Each image has a specific edit direction extracted from the Emu-edit annotations. For the fire hydrant example: "...slowly changes the color of the fire hydrant to red."

Evaluation metric: Human evaluation with three raters assessing two dimensions:

Fidelity: Was the requested edit performed correctly?
Precision: Was the edit performed correctly without unintended changes (camera movement, animating people, modifying unrelated objects)?

Sampling: 2 videos per sample, using the first video for human evaluation (30 samples × 2 videos = 60 videos per model). Two models tested: Veo 2 and Veo 3. Total: 120 videos.

Key design choice—human evaluation for fidelity and precision: Quantitative automated metrics for image editing (LPIPS, FID, CLIP score) can measure similarity but cannot distinguish intended edits from unintended artifacts. Human evaluation is more expensive but provides a direct assessment of what matters: did the model do what was asked, and only what was asked? The fidelity/precision distinction captures a critical failure mode: Veo "has a strong bias for animated scenes and might introduce unintended changes (e.g., camera movement, animating people)" (Section 4.4). A model could have high fidelity (the fire hydrant did turn red) but low precision (the camera also zoomed in, the person in the background started walking).

Maze Solving (Section 4.5, Appendix B.5)

Task: Given an input image of a maze with a red circle at the start position and a green circle at the goal position, generate a video where the red circle navigates through the white paths to the green circle without crossing any black walls. This is the paper's primary quantitative test of visual chain-of-frames reasoning.

Dataset:

Rectangular mazes: 50 randomly generated mazes for each of three grid sizes (5×5, 7×7, 9×9) using the maze-dataset library (Ivanitskiy, 2025). The generated mazes replace the standard square start/end markers with circles and swap their colors (red for start, green for goal).
Irregular mazes: 10 hand-drawn mazes, flipped/rotated to produce 40 unique samples. These test generalization beyond the grid-structured mazes the generator produces.

Prompt (Veo):

"Create a 2D animation based on the provided image of a maze. The red circle slides smoothly along the white path, stopping perfectly on the green circle. The red circle never slides or crosses into the black areas of the maze. The camera is a static, top-down view showing the entire maze. Maze: • The maze paths are white, the walls are black. • The red circle moves to the goal position, represented by a green circle. • The red circle slides smoothly along the white path. • The red circle never slides or crosses into the black areas of the maze. • The red circle stops perfectly on the green circle. Scene: • No change in scene composition. • No change in the layout of the maze. • The red circle travels along the white path without speeding up or slowing down. Camera: • Static camera. • No zoom. • No pan. • No glitches, noise, or artifacts."

Evaluation metric: Pass@k, where a video passes if the red circle reaches the green circle without any illegal moves. The evaluation is frame-by-frame:

For each frame in the generated video, extract the position of the red circle.
Check whether the circle's path between consecutive frames crosses any black walls. Methods checked: jumping over walls (the circle appears on the other side of a wall between frames), clipping through boundaries (the circle partially overlaps black pixels), or any alteration of the goal's position.
Verify that the circle ends on the green circle in the final frame.
A video passes only if the entire trajectory is valid—no illegal moves at any point.

Baseline comparisons:

Nano Banana (gemini-2.5-flash-image-preview): Prompted with "Mark the correct path from the red to the green circle through the maze in blue." Nano Banana generates the full solution path as a single image edit rather than frame-by-frame animation. Evaluation checks whether the drawn path connects start and end (allowing minor discontinuities) and whether it intersects any maze walls.
Gemini 2.5 Pro I2T (image-to-text): The maze is presented as an image, and Gemini is asked to output a series of (row, col) coordinates representing the solution path. The prompt includes detailed instructions about grid indexing, valid moves (up/down/left/right only), and formatting.
Gemini 2.5 Pro T2T (text-to-text): The maze is converted to ASCII representation ('#' for walls, ' ' for empty spaces, 'S' for start, 'E' for end), and Gemini is asked to output coordinates. This tests whether language-based reasoning on a textual maze representation is competitive with visual reasoning on the actual maze image.

Sampling: 10 videos per sample (50 samples × 4 maze types × 10 videos = 2000 videos per model). Two models tested: Veo 2 and Veo 3. Total: 4000 videos. (The 8000 total reported in Table 1 accounts for all baseline evaluations as well.)

Key design choice—frame-by-frame path verification: The evaluation doesn't just check whether the final frame shows the circle at the goal; it verifies the entire trajectory frame-by-frame. This is essential because the chain-of-frames hypothesis claims that each frame is an intermediate reasoning step. If the model produces a path that reaches the goal but clips through walls mid-video, it has not successfully reasoned about the maze constraints—it has merely produced a visually plausible final state. The illegal move detection (jumping walls, clipping boundaries) directly tests whether the CoF mechanism respects physical constraints across the entire reasoning trace.

Visual Symmetry Solving (Section 4.6, Appendix B.6)

Task: Given an input image of a 10×16 grid where a colored pattern is drawn on the left half, generate a video where the right half is filled in to complete the pattern symmetrically across the central vertical axis. The left half must remain unmodified.

Dataset: A custom synthetic dataset of 25 samples in the "shape" condition and 25 samples in the "random" condition:

Shapes: Symmetrical symbols and objects (e.g., letters A, H, M, X, T; geometric shapes like squares and triangles; symmetrical objects like wineglasses and balloons) drawn on the left half of the grid.
Random: Randomly colored cells on the left half, producing patterns with no semantic meaning—the model must rely purely on spatial mirroring.

Prompt (Veo):

"Instantly reflect this pattern along the central, vertical axis while keeping the existing colored pattern without modification. Static camera perspective, no zoom or pan."

Evaluation metric: Pass@k, where a video passes if the best or last frame has zero incorrectly colored cells compared to the ground-truth symmetrical completion. The cell-level evaluation:

Extract the relevant frame (best frame or last frame) from the generated video.
For each cell in the grid, compute the average color across all pixels in that cell.
Compare the average color of each cell to the ground-truth average color of the corresponding cell in the correct symmetrical completion.
Compute the perceptual color difference in CIELAB color space (which is designed to be perceptually uniform—equal distances correspond to roughly equal perceived differences).
A cell is considered incorrect if the CIELAB color difference exceeds a threshold of 15.0.
Count the number of incorrect cells. The video passes only if the count is zero.

Why CIELAB color space and a threshold of 15.0?:

The CIELAB color space ( $L*a*b*$ ) is designed so that Euclidean distance approximates human perception of color differences. The components are:

$L^*$ : perceptual lightness (0 = black, 100 = white)
$a^*$ : green–red axis (negative = green, positive = red)
$b^*$ : blue–yellow axis (negative = blue, positive = yellow)

For two average colors $c_1 = (L_1^*, a_1^*, b_1^*)$ and $c_2 = (L_2^*, a_2^*, b_2^*)$ :

$\Delta E = \sqrt{(L_1^* - L_2^*)^2 + (a_1^* - a_2^*)^2 + (b_1^* - b_2^*)^2}$

where $\Delta E$ is the perceptual color difference.

What it computes: for each grid cell, the Euclidean distance between the predicted average color and the ground-truth average color, measured in a space where equal distances correspond to equal perceptual differences. The threshold of 15.0 represents the point where a color difference becomes clearly perceptible to human observers under standard viewing conditions.

Why this form: using CIELAB rather than RGB is important because equal Euclidean distances in RGB do not correspond to equal perceptual differences. Two colors that are numerically close in RGB might appear very different to humans (e.g., RGB values differ by 10 in the green channel vs. 10 in the blue channel). CIELAB linearizes perceptual distance, so a threshold of 15.0 has a consistent meaning across all colors. The choice of 15.0 as the threshold corresponds to a "just noticeable difference" multiplied by a safety factor to avoid penalizing minor compression artifacts or color interpolation variations at cell boundaries that a human viewer would not perceive as errors.

Prompt sensitivity analysis (Appendix C, Table 2): The paper tests 10 prompt variations for the visual symmetry task, revealing extreme sensitivity: the pass@1 on the shape split ranges from 8% (worst prompt: "A timelapse of a professional pixel artist drawing a symmetrical pattern onto a white canvas...") to 48% (best prompt: "Instantly reflect this pattern along the central, vertical axis while keeping the existing colored pattern without modification"). On the random split, the range is 4% to 68%. The difference between best and worst prompts is 40 percentage points on shapes and 64 percentage points on random patterns. This dramatic variation underscores a core methodological point: the measured performance is a function of both model capability and prompt quality, and reported numbers are best-effort lower bounds.

Sampling: 10 videos per sample (50 samples × 10 videos = 500 videos per model). For the prompt sensitivity study (Appendix C): 1 video per sample per prompt variant (50 samples × 10 prompts × 1 video = 500 videos). Two models tested: Veo 2 and Veo 3. Total: 1000 videos for the main experiment + 500 for prompt analysis.

Visual Analogy Completion (Section 4.7, Appendix B.7)

Task: Given an input image showing a 2×2 grid visual analogy (A is to B as C is to ?), generate a video where the missing object appears in the lower-right quadrant. The model must infer the transformation rule from the A→B pair and apply it to C.

Dataset: Test trials and choice images from the KiVA benchmark (Kid-inspired Visual Analogies; Yiu et al., 2024). The KiVA benchmark presents visual analogies with three answer choices per trial. Four transformation types are tested:

Color: A changes to B by a color transformation; C should undergo the same color change.
Resize: A changes to B by a size transformation; C should be resized accordingly.
Reflect: A is reflected across some axis to produce B; C should be reflected across the same axis.
Rotate: A is rotated by some angle to produce B; C should be rotated by the same angle.

50 samples per transformation type × 4 types = 200 trials total.

Prompt (Veo):

"Create a smooth animation to generate the missing object in the lower right region and solve the visual analogy. The original three objects must remain still. Static shot, no zoom no pan no dolly."

Evaluation metric: Pass@k, where a video passes if the generated object in the lower-right quadrant matches the correct answer choice (determined by an autorater comparing the generated object to three candidate choices):

Extract the last frame from the generated video.
Crop the lower-right region containing the generated object.
Present the cropped generated object and three candidate choices to a Gemini 2.5 Pro autorater, which selects the choice most visually similar to the generated object.
If no object is visible in the lower-right region, or if the generated object is of a different type than the choices (e.g., generated a dog when choices show cats), the autorater reports "different object type" and the trial is scored as incorrect (randomly assigning one of three choices for pass@k computation).

Autorater design—why use Gemini 2.5 Pro?: Comparing generated images to reference images for visual similarity is inherently subjective. The paper uses Gemini 2.5 Pro as an automated judge because:

Sending every generated video to human raters for 200 trials × 10 attempts × 2 models would be prohibitively expensive.
The autorater is given privileged information about the transformation type and feature names (e.g., "Focus on the object color. That is, if the objects appear green, blue, or red.") to focus its comparison on the relevant attribute.
Pilot experiments showed that the Gemini autorater's ratings achieve above 88% agreement with expert human ratings by the authors on 25 samples within each condition.

Key design choice—privileged information in the autorater prompt: The autorater prompt includes the specific feature being transformed (color, size, reflect, rotate) and its possible values. This is provided to make the autorater's comparison more accurate—it knows which dimension to compare. However, this means the autorater is not a pure zero-shot evaluator; it has task-specific information that a naive image comparison (e.g., CLIP similarity) would not have. The authors justify this by noting that their goal is to accurately determine whether Veo's generation matches the correct answer, not to benchmark the autorater's zero-shot abilities.

Sampling: 10 videos per sample (200 trials × 10 videos = 2000 videos per model). Two models tested: Veo 2 and Veo 3. Total: 4000 videos.

Best-Frame vs. Last-Frame Evaluation and the Pass@k Metric

A critical methodological distinction runs throughout the quantitative experiments: best-frame versus last-frame evaluation. Understanding this distinction is essential because it reveals a fundamental tension between the model's capability (what it can do) and its reliability (what it does do consistently), which maps onto the competence vs. performance distinction the paper discusses in Section 5.

Best-frame evaluation: For each generated video, evaluate all 192 frames (8 seconds × 24 FPS) on the task metric, and report the best score across frames. This measures the model's capability ceiling—can the model solve the task at any point during generation? The optimal frame is not known a priori; selecting it requires an oracle (ground-truth labels or a verifier). This makes best-frame evaluation impractical for deployment but informative for understanding what the model is capable of.

Last-frame evaluation: Evaluate only the final frame (frame 192) on the task metric. This measures the model's practical performance given that the frame to evaluate is predetermined (you always look at the last frame). Last-frame evaluation may underestimate capability because Veo tends to continue animating beyond task completion—for example, in edge detection, the model might correctly extract edges in frame 47 but then continue adding detail or modifying the scene, degrading the quality by frame 192. In maze solving, the red circle might reach the goal by frame 120 but then continue moving past it or the scene might change, making the last frame no longer show a valid solution.

Why both metrics? The gap between best-frame and last-frame performance measures the model's temporal precision—how well it can "freeze" the solution once reached. A large gap indicates that the model produces the correct output but then continues modifying it (due to the motion prior discussed in Section 3.4.2), which is a limitation of the current generation paradigm, not necessarily of the model's perceptual or reasoning abilities. The paper reports both metrics where applicable because they provide complementary information: best-frame tells you whether the model has the underlying ability; last-frame tells you how reliably you can access that ability without additional machinery (e.g., a frame selector verifier).

The pass@k metric: For most quantitative tasks, the paper reports pass@k as a function of k (typically k = 1 to 10). Pass@k measures the probability that at least one of k independent attempts (generations with different random seeds) at a given task succeeds. This is standard in the LLM evaluation literature (used for coding benchmarks like HumanEval) and is relevant here because video generation is stochastic—two generations with the same prompt and input image may produce different results due to different random seeds. Pass@k captures the model's ability to eventually produce a correct solution given multiple tries, which is practically important for deployment (you can generate multiple videos and select the best one, analogous to best-of-N sampling in LLMs).

The paper observes a consistent pattern: pass@10 substantially exceeds pass@1 across tasks, "with no signs of a plateau" (Section 5). For example, Veo 3 on 5×5 mazes: pass@1 is lower than pass@10 (78%), meaning that generating 10 videos and checking if any solves the maze dramatically improves over a single attempt. This suggests that inference-time scaling methods—generating multiple candidates and selecting the best using a verifier or heuristic—are a viable path to improving zero-shot performance.

Why this matters for the paper's core argument: The paper claims that video models are on a trajectory to become general-purpose vision systems. The pass@1 to pass@10 gap, combined with the Veo 2 to Veo 3 improvement, suggests two orthogonal axes of progress: (1) better base models (Veo 2 → Veo 3) and (2) better inference-time strategies (pass@1 → pass@10). Both axes show substantial headroom, implying that current performance is a lower bound and future models with both better architectures and better test-time compute allocation will be more capable.

LLM Disentanglement Control: Isolating Visual Reasoning from Language Reasoning

A major confounding factor in evaluating video model reasoning is that the Vertex AI API includes a LLM-based prompt rewriter, and Veo itself may have access to a language model component for understanding text prompts. For tasks that involve symbolic reasoning (Sudoku, visual analogies, sequence completion), it is possible that the LLM—not the video model—is doing the reasoning, and the video model is merely rendering the LLM's answer. The paper implements three controls to disentangle these contributions:

Control 1: Standalone LLM evaluation on key tasks. For maze solving (Section 4.5), robot navigation (Figure 58), and visual symmetry (Section 4.6), the paper tests whether a standalone LLM (Gemini 2.5 Pro) can solve the task from the input image alone. The results for maze solving (Figure 7) are informative: Gemini 2.5 Pro given the maze as an image ("I2T" = image-to-text) generally struggles—"Gemini 2.5 Pro outperforms Veo 3 on small mazes when given an ASCII representation of the maze (T2T), but falls behind on 9×9 mazes, and generally struggles when the maze is represented as image (as opposed to text) input." This asymmetric performance (LLM succeeds on text-based mazes but fails on image-based mazes, while Veo succeeds on image-based mazes) suggests that Veo's maze-solving involves genuine visual reasoning that a language model cannot replicate from the raw pixels.

The logic of this control: if the LLM rewriter were solving the maze and instructing Veo to render the solution, then the standalone LLM should also be able to solve the maze—but it cannot reliably do so from images. Therefore, the video model itself must be doing the visual reasoning.

Control 2: Nano Banana comparison on reasoning tasks. Nano Banana is a state-of-the-art image editing model (also from Google) that can perform sophisticated image-to-image transformations but is not a video model and does not do frame-by-frame generation. The paper compares Veo to Nano Banana on maze solving and visual symmetry. The results: on rectangular mazes, Nano Banana matches or surpasses Veo 3 (Figure 7), but on irregular mazes, Nano Banana fails entirely ("fails to solve irregular mazes entirely") while Veo 3 achieves nonzero pass@k. This suggests that Nano Banana can solve mazes when they conform to grid patterns (which an LLM could plan over), but cannot handle the continuous, irregular paths that require genuine visual reasoning. Veo 3's success on irregular mazes is evidence that the chain-of-frames mechanism provides reasoning capabilities beyond what an LLM + image editor can achieve.

Control 3: Explicit acknowledgment of LLM contributions for some tasks. The paper is transparent about cases where the LLM likely dominates. For Sudoku (Figure 55): "the solution is likely to come from the LLM instead of the video." The paper does not claim that Sudoku solving is visual reasoning; it reports the result but flags the confounding factor. This selective acknowledgment increases credibility for the tasks where the paper does claim visual reasoning.

Why this matters for the chain-of-frames argument: If Veo's success on maze solving, navigation, and symmetry were entirely due to an upstream LLM planning the solution, then CoF would be a rendering mechanism, not a reasoning mechanism. The disentanglement controls establish that CoF is necessary for some visual reasoning tasks—the LLM alone cannot solve them, and the video model's sequential generation is the computational mechanism producing the solution.

Summary of Design Choices and Their Justifications

Black-box evaluation (no fine-tuning, no access to model internals): This is the core methodological commitment. By treating Veo as an API-callable black box, the paper tests what capabilities emerge from the pretraining objective alone, without post-hoc adaptation. This mirrors how GPT-3 was evaluated—as a fixed model accessed through prompting. The advantage is that any observed capabilities are genuinely emergent; the disadvantage is that the paper cannot analyze internal representations or training data to understand why certain capabilities appear.
Quantitative evaluation across 7 diverse tasks: Rather than focusing on a single capability (e.g., only segmentation or only maze solving), the paper spans perception, manipulation, and reasoning. This breadth is essential for the central claim that video models are general-purpose—a model that only does one task well would not support the paradigm shift argument. The tasks are chosen to instantiate different tiers of the capability hierarchy while having established metrics and baselines.
Pass@k with multiple attempts: Video generation is stochastic, and pass@1 severely underestimates a model's capability if it sometimes succeeds but sometimes fails. Pass@k captures what a practitioner could achieve by generating multiple videos and selecting the best (using a ground-truth verifier or heuristic). The consistent improvement from pass@1 to pass@10 across all tasks suggests that inference-time compute scaling is a viable strategy even with current models.
Best-frame vs. last-frame distinction: This distinction is motivated by the observed behavior that Veo continues animating beyond task completion. Without this distinction, a task could appear unsolvable when the model actually produced the correct answer at some intermediate frame but then overwrote it. The distinction allows the paper to make a stronger case about the model's underlying competence while being honest about its deployment reliability.
LLM disentanglement controls: These controls are essential given the black-box API (which includes a prompt rewriter) and the multimodal nature of the task (text prompt + image input). Without them, the paper's claims about visual reasoning could be attributed entirely to language model reasoning, undermining the core CoF argument. The asymmetric performance pattern (LLM fails on images, Veo succeeds) provides the strongest evidence for genuine visual reasoning.
Prompt engineering as a deliberate part of the methodology (Appendix C): Rather than treating prompt engineering as a nuisance variable, the paper leans into it, documenting sensitivity and providing best practices. The 40–64 percentage point difference between best and worst prompts on visual symmetry quantifies how much prompt quality matters. The paper's position (Section 5) is that "prompt engineering—including the visual prompt a.k.a. starting frame—is as important for visual tasks as it is for LLMs," framing this not as a limitation but as an expected property of generalist models that can be solved in many different ways.
Autorater for visual analogy evaluation: Manual human evaluation of 4000 generated videos (200 trials × 10 attempts × 2 models) for visual analogies would be prohibitively expensive and slow. The Gemini 2.5 Pro autorater, validated to >88% agreement with human experts on a subset, provides a scalable evaluation with acceptable accuracy. The decision to give the autorater privileged information about the transformation type trades off purity (the autorater knows what to look for) against accuracy (it makes fewer mistakes), which is appropriate given that the goal is measuring Veo's performance, not the autorater's.
Synthetic datasets for controlled evaluation: For maze solving and visual symmetry, the paper generates synthetic datasets rather than using natural images. This provides precise control over difficulty (maze size, symmetry pattern complexity), unambiguous ground truth (grid cells are either correctly or incorrectly colored), and automated evaluation (no human raters needed). The trade-off is that these tasks are less ecologically valid than natural images, but for establishing whether a capability exists, synthetic data provides cleaner evidence.
Veo 2 to Veo 3 scaling comparison: By evaluating both model versions on the same tasks with the same prompts, the paper can measure the rate of progress and extrapolate forward. The consistent, substantial improvement (e.g., 14% → 78% pass@10 on 5×5 mazes) is used to argue that video models are on a steep improvement trajectory, which matters for the "video models will become foundation models" thesis. If Veo 2 and Veo 3 performed similarly, the future outlook would be uncertain; the large gap provides empirical basis for the paper's optimism.

4. Key Insights and Innovations

Innovation 1: Chain-of-Frames as a General Reasoning Primitive, Not Just a Rendering Artifact

The paper's most fundamental conceptual contribution is reframing video generation—frame-by-frame autoregressive output—as a general-purpose visual reasoning mechanism rather than merely a rendering process. Before this work, video generation was understood primarily as producing visually plausible content: given a text prompt or starting image, produce frames that look realistic and temporally coherent. The canonical use cases were entertainment, simulation, and content creation—the output was the product, evaluated by visual quality metrics (FVD, IS, FID) and human preference ratings. The intellectual contribution of this paper is the recognition that the sequential generation process itself is a computational primitive—each generated frame is an intermediate reasoning step, and the temporal dimension provides a substrate for step-by-step problem-solving analogous to chain-of-thought (CoT) in language models.

What makes this a genuinely new framing rather than a metaphor is that it makes specific, falsifiable predictions that differ from the "video as rendering" view: (1) The model should be able to solve problems that require maintaining state across frames that cannot be inferred from the initial image alone (e.g., maze solving requires tracking where the agent has been, not just what the maze looks like). (2) Performance should degrade if the frame-by-frame generation is disrupted or if problem constraints require reasoning that cannot be decomposed into sequential visual states. (3) The evaluation methodology must shift from "does the video look good?" to "is the frame-by-frame computation correct?"—the paper's frame-level path verification for maze solving is a direct operationalization of this shift.

Prior work studied chain-of-thought in language models (Wei et al., 2022) and used video models for planning (Yang et al., 2024), but these were separate threads. CoT operated in token space for symbolic reasoning; video planning operated in pixel space for trajectory generation. The paper unifies these by observing that the underlying mechanism—autoregressive generation conditioning on previous outputs—is identical, and that video models inherit this mechanism by construction because they are trained to predict future frames given past frames. The chain-of-frames (CoF) concept is thus not a new algorithm but a diagnostic insight: it tells us what to look for when evaluating video models, why certain tasks succeed or fail, and where the scaling headroom lies.

The evidence for CoF as reasoning—not just rendering—comes from the asymmetric performance pattern in maze solving (Figure 7): Gemini 2.5 Pro, a capable language model, solves small mazes from ASCII text but struggles from images, suggesting that the maze-as-image requires visual computation that language alone cannot provide. Veo 3, generating frame-by-frame, can solve mazes from images—including irregular mazes that Nano Banana (an image editor that generates the full path in one edit) fails on entirely. This asymmetry is exactly what the CoF hypothesis predicts: frame-by-frame generation provides a computational advantage for tasks that require step-by-step spatial reasoning with state tracking, which single-step outputs (from image editors) and symbolic reasoning (from LLMs) cannot match.

A subtler implication: if CoF is genuinely analogous to CoT, then the extensive literature on improving CoT reasoning—self-consistency (Wang et al., 2022), self-refinement (Madaan et al., 2023), test-time compute scaling (Snell et al., 2024)—should transfer to video models. The paper gestures at this by showing that pass@10 substantially exceeds pass@1 across tasks "with no signs of a plateau" (Section 5), suggesting that inference-time scaling strategies developed for language could be adapted for video. This is a fundamental reframing of what video models are—not content generators but visual computers—with direct implications for evaluation methodology, capability benchmarking, and future research directions.

Innovation 2: The Four-Tier Capability Hierarchy as a Diagnostic Framework for Emergence

The paper organizes 62 qualitative and 7 quantitative tasks into a four-tier hierarchy—perception, modeling, manipulation, reasoning—where each tier builds on the preceding ones. While taxonomic frameworks for vision capabilities exist, this hierarchy is intellectually distinctive because it is causal, not descriptive: the paper argues that capabilities at tier N depend on capabilities at tier N-1, and this dependency explains how emergence happens. A video model trained solely to predict future frames must, as a byproduct of that objective, develop perceptual abilities (to extract structure from pixels), which then enable modeling (to understand how objects behave), which then enables manipulation (to modify scenes physically plausibly), which then enables reasoning (to plan multi-step transformations). The hierarchy is thus a hypothesis about the generative training objective as a sufficient condition for developing the full vision stack—a hypothesis that prior work testing individual capabilities in isolation could not articulate.

Prior taxonomies in computer vision—Taskonomy (Zamir et al., 2018), the various "vision stack" decompositions in robotics and scene understanding—were typically descriptive catalogues of what different models can do, or transfer-learning analyses of which tasks share representations. This paper's hierarchy is different in kind: it is a claim about the emergence trajectory of generalist models. The four tiers are not just categories of tasks; they are stages of capability acquisition that the model is predicted to undergo as it scales. A small video model might only develop perception; a slightly larger one might add modeling; a sufficiently large one might exhibit the full hierarchy. The paper provides indirect evidence for this by showing that Veo 3 (larger, more capable) dramatically outperforms Veo 2 on higher-tier tasks (e.g., 78% vs. 14% pass@10 on 5×5 maze solving, a reasoning task) while both models perform more similarly on lower-tier tasks (e.g., denoising, deblurring—the classic diffusion objectives). This is consistent with the hierarchy hypothesis: higher tiers emerge later in the scaling trajectory.

The hierarchy also provides a unified explanation for contradictory prior findings. Previous work found that diffusion models could perform some zero-shot tasks (classification: Clark and Jaini, 2023; segmentation: Burgert et al., 2022) but not others (complex reasoning). Under the hierarchy framework, this is expected: earlier models with fewer capabilities could handle perception-tier tasks but not modeling, manipulation, or reasoning. The hierarchy predicts which tasks should emerge at which scales—a falsifiable prediction that future work can test by evaluating intermediate-scale models on tasks across all four tiers.

The practical value of the hierarchy is as a diagnostic tool: when a model fails at a reasoning task (e.g., visual analogies with reflect/rotate transformations, Figure 9), the hierarchy suggests investigating whether the failure is actually at a lower tier—does the model correctly perceive the objects? Does it correctly model the transformation? Only if both lower tiers are intact can a reasoning failure be attributed to the reasoning tier itself. This decomposability is critical for debugging and improving video models systematically, analogous to how modular architectures allow isolating errors to specific components.

This is a fundamental conceptual innovation—not an algorithm, not a benchmark, but a framework for understanding how visual intelligence emerges from generative training. Its significance lies in its testability: if future video models at intermediate scales follow the predicted emergence order (perception → modeling → manipulation → reasoning), the hierarchy is validated; if they show reasoning without manipulation or modeling without perception, it is falsified. The current evidence (Veo 2 vs. Veo 3 scaling, the task success pattern in Figure 1) is suggestive but not conclusive—the paper provides initial supporting evidence rather than a rigorous test, leaving the hierarchy as a productive hypothesis for future work.

Innovation 3: The Performance-Competence Distinction Applied to Video Model Evaluation

The paper explicitly distinguishes between a model's task performance (how well it does on a specific benchmark with a specific prompt) and its underlying competence (whether it possesses the ability to solve the task at all). This distinction, drawn from cognitive science (Firestone, 2020) and previously applied to neural network shortcut learning (Geirhos et al., 2020), is adapted here to video model evaluation with specific methodological consequences that are novel to this domain.

The innovation is not the distinction itself—performance-competence gaps are well-documented in human cognition and have been discussed in ML—but rather the systematic operationalization of the distinction through three concrete methodological choices that are uncommon in video model evaluation:

First, the best-frame vs. last-frame evaluation split (Section 3.4.5). Best-frame performance measures the model's ceiling—can it produce the correct output at any point during generation? Last-frame performance measures practical reliability—can you count on the final frame being correct? The gap between them quantifies the model's temporal precision, isolating a specific failure mode (continued animation beyond task completion) from a fundamental capability deficit. This split transforms what would otherwise be an undifferentiated "the model fails at this task" into a diagnostic: the model has the competence to solve the task but does not reliably stop at the solution. The fix is not better reasoning but better stopping behavior—a different research problem entirely. This is visible in the symmetry results (Figure 8): best-frame pass@1 can be substantially higher than last-frame pass@1, particularly for Veo 3 on the random-patterns condition (68% vs. 44%, a 24 percentage-point gap).

Second, the pass@k metric as a probe for competence rather than reliability. By reporting pass@10 alongside pass@1, the paper measures whether the model can ever succeed, not whether it usually succeeds. The consistent pass@1-to-pass@10 improvement across tasks (Section 5) indicates that failures are often stochastic rather than systematic—the model has the underlying ability but does not reliably deploy it on a single attempt. This has practical significance because it implies that inference-time scaling (generating multiple candidates and selecting the best, analogous to best-of-N in LLMs) can recover capability that pass@1 metrics would suggest is absent.

Third, the prompt sensitivity analysis (Appendix C, Table 2) as a direct measurement of the performance-competence gap. The 40–64 percentage-point difference between best and worst prompts on visual symmetry is not a model failure—the model can solve symmetry (48% pass@1 on shapes with the best prompt). It is a measurement failure: the wrong prompt masks competence that exists. The paper's position that "reported performance should be considered a lower bound on the model's true capabilities" (Section 5) follows directly from this evidence.

The significance of this innovation extends beyond methodological hygiene. It changes how failure cases should be interpreted. The paper documents failures on depth estimation, surface normal estimation, force prompting, knot tying, and Eulerian paths (Appendix D). Under a naive evaluation regime, these would be evidence that the model lacks these capabilities. Under the performance-competence distinction, they are evidence that the model fails under current prompting and evaluation—which could be due to prompt engineering failures, temporal precision issues, or genuine capability gaps, and distinguishing these requires targeted investigation. The paper does not fully resolve which failures are which (that would require extensive prompt optimization per failure case), but it establishes the framework for asking the question correctly.

This is an incremental innovation in methodology—the concepts are borrowed from existing literatures—but its systematic application to video model evaluation is novel and has substantial practical consequences for how future video model capabilities are assessed and reported. Without this distinction, the paper's central claim (that video models show emergent zero-shot abilities) would rest on shakier ground, because individual task failures could be cited as counterevidence. With it, failures become hypotheses to investigate rather than verdicts on capability.

Innovation 4: Evidence That Video Models Can Substitute for Task-Specific Vision Models—With Sharp, Characterized Boundaries

The paper provides what is, to the authors' knowledge, the first systematic evidence that a single video model can perform—in a zero-shot, prompt-only manner—tasks that previously required specialized architectures across the full vision stack. While individual zero-shot capabilities of diffusion models have been documented (classification, segmentation, image editing), no prior work has demonstrated breadth across perception, modeling, manipulation, and reasoning from a single model, nor characterized where this substitution works and where it fails.

The intellectual contribution is not the empirical finding per se—it is the boundary characterization that accompanies it. The paper identifies specific conditions under which video models can substitute for specialized models and conditions under which they cannot:

Where substitution works: Tasks where the model's pretraining distribution provides sufficient visual priors. Edge detection (0.77 OIS pass@10, Section 4.1) works because edges are a fundamental visual primitive that any video model must learn to predict. Instance segmentation (0.74 mIoU pass@10, Section 4.2) works for simple scenes with 1–3 large objects. Object extraction (93% pass@10, Section 4.3) works for animals in natural scenes. Maze solving (78% pass@10 on 5×5, Section 4.5) works for small, grid-structured mazes. Color and resize analogies (Figure 9) work because these transformations are likely common in video data.

Where substitution fails: Tasks requiring outputs the model was not trained to produce or transformations outside its training distribution. Monocular depth estimation (Figure 62) fails because Veo cannot reliably assign per-pixel depth values beyond a foreground/background binary—this is not a natural output modality for video generation. Surface normal estimation (Figure 63) fails for similar reasons. Force prompting with arrow annotations (Figure 64) fails because the model does not understand abstract control signals overlaid on images. Rotation and reflection analogies (Figure 9) fail systematically—Veo 3 performs below chance (0.33) on these transformations, likely because common video data contains color changes and size changes but not arbitrary object rotations and reflections in the style of visual analogy problems.

The Veo 2 to Veo 3 scaling trajectory as boundary evidence. The consistent, substantial improvement from Veo 2 to Veo 3 (e.g., 14% → 78% on 5×5 mazes, 0.52 → 0.74 mIoU on segmentation) establishes that boundaries are shifting—tasks at the boundary of Veo 2's capability become reliable for Veo 3. This is the paper's primary argument that current limitations are not fundamental: if a capability exists in Veo 3 that was absent in Veo 2, then capabilities that are absent in Veo 3 may appear in Veo 4. The paper frames this as analogous to GPT-3's below-SOTA performance on many NLP benchmarks—performance that did not prevent LLMs from becoming foundation models because the trajectory, not the snapshot, mattered.

The hardest reasoning problems (irregular mazes, reflect/rotate analogies, ARC-AGI-style rule extrapolation) as current boundaries. The paper is explicit that these tasks remain largely unsolved, but frames them as informative boundaries rather than failures of the paradigm. Irregular maze performance (Figure 7) is nonzero but low—chain-of-frames reasoning on non-grid paths is harder than on grid paths, consistent with the model's training distribution containing more grid-like navigation data. The systematic failure on reflect/rotate analogies (Figure 9, Figure 61) is particularly informative: both Veo 2 and Veo 3 perform below chance, and majority voting over multiple attempts decreases performance (the models have systematic erroneous biases, not random errors), suggesting a fundamental limitation in the model's internal representations of these transformations rather than a stochastic failure.

This is a foundational empirical contribution—not a new method, but a characterization of the current frontier that provides a roadmap for where investment is needed (better training data for certain transformations, architectures that support non-standard output modalities, post-training techniques for reasoning). The boundary characterization is what distinguishes this from a mere capability demonstration: the paper doesn't just say "Veo 3 can do these things," it says "Veo 3 can do these things and here is where it cannot, and here is why the pattern of failures makes sense given the training objective and data."

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper evaluates on a combination of existing benchmarks and custom synthetic datasets. For perception: the full 50-image test set from BIPEDv2 (Soria et al., 2020, 2023) for edge detection, and 50 randomly chosen test images from LVIS (Gupta et al., 2019) containing 1–3 large objects (each ≥5000 pixels) for segmentation. For manipulation: a custom animal counting dataset of 54 images (6 per count, 1–9 animals) generated via Nano Banana and manually verified, plus a random 30-image subset from Emu-edit (Sheynin et al., 2024) for image editing. For reasoning: 50 randomly generated mazes per grid size (5×5, 7×7, 9×9) from maze-dataset (Ivanitskiy, 2025) plus 40 irregular mazes (10 hand-drawn, flipped/rotated), 25-shape and 25-random synthetic symmetry grids (10×16 cells), and 200 test trials from KiVA (Yiu et al., 2024) across four transformation types (50 each: color, resize, reflect, rotate). The qualitative investigation spans 62 additional tasks with 12 samples each (744 videos). Total video count across all quantitative and qualitative experiments: 18,384 (17,640 quantitative + 744 qualitative).
Base model(s). Two versions of Veo accessed through Google Cloud's Vertex AI API: Veo 2 (model ID veo-2.0-generate-001, announced December 2024, released April 2025) and Veo 3 (model ID veo-3.0-generate-preview, announced May 2025, released July 2025). The paper chooses Veo because "it has consistently ranked high on text2video and image2video leaderboards" (Section 2). The roughly six-month gap between releases enables a natural scaling comparison. The API includes an LLM-based prompt rewriter; the paper treats the entire system (rewriter + video generator) as a single black-box entity but verifies on key reasoning tasks that a standalone LLM (Gemini 2.5 Pro) cannot reliably solve them from the input image alone. All videos are generated at 16:9 aspect ratio, 720p resolution, 24 FPS, 8 seconds duration (192 frames total).
Metrics. Seven quantitative tasks use task-specific metrics: (1) Edge detection: OIS (Optimal Image Scale)—the maximum F1-score over 16 binarization thresholds, allowing local pixel shifts within 0.75% of the image diagonal (~11 pixels), measured per-frame with non-maximum suppression and thinning. (2) Segmentation: mIoU (mean Intersection over Union)—predicted masks are matched to ground-truth instances via hue-difference histogram peak detection, with each ground-truth mask paired to the predicted mask with highest IoU; background is excluded from averaging. (3) Object extraction: binary pass/fail on correct animal count in the last frame, counted via connected components in a thresholded grayscale image. (4) Image editing: human evaluation by three raters scoring fidelity (correct edit) and precision (correct edit without unintended changes like camera movement). (5) Maze solving: frame-by-frame path verification checking for illegal moves (jumping walls, clipping boundaries, goal position alteration); pass if agent reaches goal with zero illegal moves. (6) Visual symmetry: per-cell perceptual color difference in CIELAB color space with threshold 15.0; pass if zero incorrectly-colored cells. (7) Visual analogies: Gemini 2.5 Pro autorater comparing generated lower-right object to three candidate choices; validated to >88% agreement with expert human raters on 25 samples per condition. Pass@k is reported for all tasks with k = 1 to 10, measuring the probability that at least one of k independent attempts succeeds.
Baselines. For each quantitative task, the paper compares against: (1) Veo 2—the previous model generation, providing a scaling comparison at matched prompts and budgets. (2) Nano Banana (gemini-2.5-flash-image-preview)—a state-of-the-art image editing model, evaluated on edge detection, segmentation, maze solving, and visual symmetry with task-appropriate prompts. (3) Gemini 2.5 Pro (gemini-2.5-pro)—a standalone LLM evaluated on maze solving from image input (I2T) and ASCII text input (T2T) to disentangle visual reasoning from language reasoning. (4) Chance baselines for object extraction (random guessing: 1 − (8/9)^k) and visual analogies (0.33, random choice among three options). For image editing, the baseline is implicit in the human evaluation (raters assess fidelity and precision directly rather than comparing to a reference model output).
Generation budget / compute accounting. The primary compute unit is a generation—one complete video (192 frames at 720p). For each quantitative task, the paper generates k videos per sample (typically k = 10) and reports pass@k as a function of k. The total compute is thus samples × k videos × 192 frames. The paper does not report FLOP counts or wall-clock time; the implicit assumption is that all generations are equally expensive regardless of content (same resolution, duration, frame rate). For maze solving, the paper notes that Veo generates the solution frame-by-frame (chain-of-frames) while Nano Banana generates a single image edit—these use different computational resources, but the paper does not attempt to equalize them (the comparison is on task success, not compute efficiency). The best-frame metric introduces an oracle selection cost that is not accounted for in the generation budget (finding the best frame requires evaluating all 192 frames against ground truth, which is impossible in deployment without a verifier).
Cross-validation / statistical protocol. The paper does not employ cross-validation or formal statistical testing. Pass@k is computed across k independent generations per sample, aggregated over samples, but no confidence intervals, error bars, or significance tests are reported. The human evaluation for image editing uses three raters, but inter-rater reliability metrics (e.g., Krippendorff's alpha, Fleiss' kappa) are not reported. The Gemini autorater for visual analogies is validated against human expert ratings on 25 samples per condition (reported >88% agreement), but no quantitative agreement metric (e.g., Cohen's kappa, exact match rate) is provided. The prompt sensitivity study on visual symmetry (Appendix C, Table 2) uses 25 samples per split (shape/random) with 10 prompt variations and 1 video per sample per prompt, but reports only descriptive statistics (pass@1 percentage and average incorrect cells) without quantifying variance across samples or prompts. The small sample sizes in several evaluations (50 images for edge detection and segmentation, 30 for image editing, 25 per symmetry split) mean that pass@k estimates could have substantial variance, though the consistency of patterns across tasks partially mitigates this concern.

Main Quantitative Results

Perception: Edge Detection and Segmentation

Edge detection (Figure 3). Veo 3 achieves 0.77 OIS pass@10, compared to 0.57 for Veo 2 and 0.74 for Nano Banana. Veo 3's best-frame performance (0.77) is marginally above task-specific SOTA reported on the BIPEDv2 benchmark (0.90), though the paper notes this comparison is approximate since SOTA models are fine-tuned for edge detection. The gap between Veo 3 best-frame (0.77) and last-frame (0.74) is small (0.03), indicating that edge maps remain fairly stable across the video—the model does not substantially degrade its edge detection output in later frames. Performance improves substantially with k: at k=1, Veo 3 achieves approximately 0.55 OIS, rising to 0.77 at k=10, suggesting significant per-generation variance. The paper documents a systematic evaluation issue: "many of Veo 3's edge maps are more detailed than the ground truth" (Section 4.1), with false positives corresponding to actual image details (tree outlines, tire profiles, window reflections) that were labeled as background in the ground-truth annotations. This means the 0.77 OIS figure likely understates Veo 3's true edge-detection capability—the model is penalized for detecting edges that exist but were not annotated.

Instance segmentation (Figure 4). Veo 3 achieves 0.74 mIoU best-frame pass@10 on the 50-image LVIS subset, matching Nano Banana's 0.73 and substantially outperforming Veo 2's 0.52. The prompt matters: Veo 3 with green background achieves 0.74 vs. 0.66 with white background, a gap the paper attributes to "the widespread use of green screens" in video training data. The last-frame performance drops considerably: Veo 3 last-frame pass@10 is 0.56 with green background and 0.42 with white, reflecting the motion prior—the model continues modifying the scene after producing correct segmentations in intermediate frames. Veo 2 last-frame performance collapses to 0.40–0.41, barely above chance. At k=1, Veo 3 best-frame mIoU is approximately 0.45 (green) and 0.35 (white), demonstrating that multiple attempts are necessary to reliably obtain good segmentations. The paper contrasts these results with specialized models: SAMv2 (Ravi et al., 2024) achieves substantially higher mIoU on LVIS but requires ground-truth prompts (points, boxes) and is explicitly trained for segmentation—Veo 3 achieves its performance with zero training examples and no spatial prompts.

Manipulation: Object Extraction and Image Editing

Object extraction (Figure 5). Veo 3 achieves 93% pass@10 on the animal counting task, vs. 63% for Veo 2. The chance baseline at k=10 is approximately 69% (1 − (8/9)^10), meaning Veo 2 performs below chance—it does worse than random guessing—while Veo 3 performs substantially above chance. At k=1, Veo 3 achieves approximately 50% (read from plot), rising to 93% at k=10 with no apparent plateau, indicating that the model possesses the counting and extraction capability but does not reliably deploy it on any single generation. The paper notes that "given the simplicity of the task, a perfect model should easily achieve 100% accuracy" (Section 4.3), establishing 93% as a strong but imperfect result. The failure cases are not analyzed in detail—it is unclear whether failures stem from miscounting animals, failing to extract all animals, or losing animals during the transformation.

Image editing (Figure 6). Veo 3 achieves approximately 88% fidelity (correct edit performed) and 75% precision (correct edit without unintended changes) averaged across three human raters, compared to Veo 2's approximately 75% fidelity and 50% precision. The fidelity-precision gap (88% − 75% = 13 percentage points for Veo 3; 75% − 50% = 25 percentage points for Veo 2) quantifies the unintended changes problem: Veo frequently animates static scenes (camera movement, people walking) even when explicitly instructed not to. Veo 3 substantially narrows this gap, suggesting improved instruction following. The paper notes that Veo 3 "especially excels in preserving details and textures across edits" (Section 4.4). The editing evaluation differs from other quantitative tasks in using a single generation per sample (k=1) evaluated by humans rather than pass@k—this choice likely reflects the cost of human evaluation but means the results do not benefit from the pass@10 scaling observed in other tasks. The 30-image sample is the smallest quantitative evaluation in the paper, making these results the most susceptible to sampling variance.

Reasoning: Maze Solving, Visual Symmetry, and Visual Analogies

Maze solving (Figure 7). Veo 3 achieves pass@10 of 78% on 5×5 grids (vs. Veo 2's 14%), 38% on 7×7 (vs. Veo 2's 2%), 12% on 9×9 (vs. Veo 2's 0%), and 75% on irregular mazes (vs. Veo 2's 0%). The sharp performance drop from 5×5 to 9×9 (78% → 12%) indicates that chain-of-frames reasoning degrades substantially with increasing path length—the model must maintain spatial coherence over more frames, and the probability of an illegal move compounds. Veo 2's near-zero performance on 7×7, 9×9, and irregular mazes (0–2%) demonstrates that even basic CoF reasoning was effectively absent in the previous model generation—the capability emerged between Veo 2 and Veo 3. Nano Banana comparison: on rectangular mazes, Nano Banana matches or surpasses Veo 3 (92% on 5×5 at k=10, 74% for Veo 3; 72% on 7×7 vs. 38%; 22% on 9×9 vs. 12%), but Nano Banana fails entirely on irregular mazes (0% vs. Veo 3's 75%). This asymmetry is the paper's strongest evidence for the chain-of-frames mechanism: Nano Banana generates the full solution path in a single image edit, which works for grid-structured mazes that may be solvable via pattern recognition or an upstream LLM, but fails on continuous, non-grid paths requiring step-by-step visual reasoning. LLM disentanglement: Gemini 2.5 Pro from image input (I2T) achieves 16% on 5×5, 8% on 7×7, 0% on 9×9, and 0% on irregular mazes—universally below Veo 3. From ASCII text input (T2T), Gemini outperforms Veo 3 on small mazes but falls behind on 9×9 (2% vs. Veo 3's 12%). The authors interpret this as evidence that "solving a visual task step-by-step in a visual medium" provides advantages over translating to symbolic representation.

Visual symmetry (Figure 8). Veo 3 achieves pass@10 of 88% best-frame on shapes and 100% on random patterns, compared to Veo 2's 20% (shapes) and 0% (random), and Nano Banana's 37% (shapes) and 28% (random). Last-frame performance drops: Veo 3 achieves 44% (shapes) and 72% (random) pass@10, a gap of 44 and 28 percentage points respectively—larger than the edge detection gap (0.03 OIS) and indicative of substantial temporal precision issues. Veo 2 collapses entirely on random patterns (0% last-frame pass@10). Nano Banana's last-frame performance (0% on random) reflects that it is an image model evaluated on a single output, not a video model where "last frame" is defined. The systematic gap between shape and random conditions for Veo 3 best-frame (88% vs. 100%) is notable: random patterns are actually easier (100% success with enough attempts) than semantic shapes (88%), perhaps because random patterns have no semantic content that might distract or bias the model's completion.

Visual analogies (Figure 9, Figure 61). Veo 3 achieves pass@1 of 95% on color and 67% on resize, but only 29% on reflect and 19% on rotate—where the chance baseline is 33% (random choice among three options). Veo 2 achieves 68% (color), 40% (resize), 23% (reflect), 22% (rotate). Both models thus perform below chance on reflect and rotate transformations. Figure 61 reveals an even more diagnostic pattern: majority voting over k attempts decreases performance for reflect and rotate (Veo 3: from 29% at k=1 to 22% at k=10 majority vote for reflect; from 19% to 14% for rotate). This indicates systematic, erroneous biases rather than random errors—the models consistently produce the wrong transformation (e.g., reflecting across the wrong axis, rotating the wrong amount), and voting reinforces rather than corrects the error. For color and resize, majority voting improves performance monotonically with k, as expected when errors are random.

Scaling Trends Across All Tasks

The paper does not aggregate results into a single summary metric but reports consistent scaling patterns across all quantitative tasks. The Veo 2 to Veo 3 improvement is substantial and universal: pass@10 increases by factors of 1.4–5.6× for perception tasks (0.57 → 0.77 OIS for edges, 0.52 → 0.74 mIoU for segmentation), by 1.5× for object extraction (63% → 93%), and by 5.6× for maze solving on 5×5 grids (14% → 78%). The pass@1 to pass@10 improvement is also universal: all tasks show monotonically increasing pass@k curves with no apparent plateau, consistent with stochastic success that can be recovered by generating multiple candidates. The paper argues this implies headroom for inference-time scaling methods.

Figure 1 (qualitative overview). The 62-task qualitative survey (12 samples per task) shows that Veo 3 achieves nonzero success rates on 56 of 62 tasks (90% of tasks attemptable), with 23 tasks achieving success rates ≥0.5. The hierarchical organization reveals that perception and modeling tasks generally have higher success rates (many at 0.75–1.0) than manipulation (0.25–0.83) and reasoning tasks (0.08–0.67), consistent with the hierarchy hypothesis that higher-tier capabilities are less developed. The hardest qualitative tasks (success rate ≤0.08) include colorization (0.08), graph traversal (0.08), sorting numbers (0.08), transfiguration (0.17), tree BFS (0.17), and maze solving in a photorealistic scene (0.17)—mirroring the quantitative finding that complex multi-step reasoning remains challenging.

Ablation Studies and Robustness Checks

The paper's quantitative ablations and robustness checks are limited compared to typical ML papers—consistent with its nature as an empirical capability investigation rather than a methods paper. However, several analyses probe the robustness of the reported results:

Background color sensitivity in segmentation (Figure 4, Section 4.2): Veo 3 achieves 0.74 mIoU best-frame pass@10 with green background vs. 0.66 with white, an 8 percentage-point mIoU difference. This demonstrates sensitivity to prompt design choices that are not task-semantic (the color of the background is incidental to the segmentation task). The paper hypothesizes this is "possibly due to the widespread use of green screens" in training data, suggesting the model has learned statistical associations rather than pure task understanding.

Prompt variation sensitivity in visual symmetry (Appendix C, Table 2): The paper tests 10 prompt variations on the symmetry task, revealing a 40 percentage-point pass@1 range on shape patterns (8%–48%) and a 64 percentage-point range on random patterns (4%–68%). The best-performing prompt emphasizes instant reflection, axis specification, and preservation of existing patterns; the worst-performing prompts introduce distracting elements (pixel artist, timelapse, "animation showing..."). Key patterns: (1) Prompts that specify "instantly" outperform those that describe gradual processes. (2) Explicit axis specification ("central, vertical axis") improves over vague descriptions ("mirror-symmetrical"). (3) Camera control phrases ("Static shot; no zoom, no pan, no dolly") have mixed effects—they help some prompts (comparing prompts 1 and 2: 48% vs. 42% on shapes) but hurt others. (4) The average number of incorrect cells per attempt correlates strongly with pass@1, indicating that failures are distributed across many cells (not concentrated in a few consistently difficult cells).

Best-frame vs. last-frame evaluation across tasks (Figures 3, 4, 8): The gap between best-frame and last-frame performance varies substantially by task: 0.03 OIS for edge detection (small), 0.18 mIoU for segmentation (moderate), 44 percentage points pass@10 for symmetry on shapes (large). This variation suggests that Veo's tendency to continue animating beyond task completion differentially affects tasks—edge detection is relatively stable (once edges are extracted, subsequent frames preserve them), while symmetry completion is fragile (the model often overwrites correct completions with new patterns). The paper does not quantitatively analyze what factors predict large vs. small best-frame/last-frame gaps, but the qualitative explanation is the motion prior (Appendix C): Veo was trained to generate dynamic content and has a learned bias to keep things moving, which is task-incompatible for static outputs.

LLM disentanglement via standalone LLM evaluation (Figure 7): The comparison of Veo 3 against Gemini 2.5 Pro (I2T and T2T) on maze solving serves as a crucial control—if the LLM rewriter were solving mazes and instructing Veo, then the standalone LLM should perform comparably. The results show that Gemini I2T (16%/8%/0%/0% on 5×5/7×7/9×9/irregular) dramatically underperforms Veo 3 (78%/38%/12%/75%), establishing that the video model's contribution is necessary (not just the LLM's). The T2T comparison is more nuanced: Gemini T2T outperforms Veo 3 on 5×5 (92% vs. 78%) and 7×7 (72% vs. 38%), but underperforms on 9×9 (2% vs. 12%) and fails on irregular mazes where Veo 3 achieves 75%. This asymmetry suggests that for small, text-representable mazes, symbolic reasoning is competitive or superior, but for larger or non-grid mazes, visual CoF reasoning provides an advantage that symbolic reasoning from pixel input cannot replicate. The paper does not report LLM evaluations on symmetry or analogies, which would provide additional disentanglement evidence.

Nano Banana comparison on reasoning tasks (Figures 7, 8): Nano Banana's 0% on irregular mazes (vs. Veo 3's 75%) and lower performance on visual symmetry (37% vs. 88% on shapes best-frame) demonstrates that image-editing models—which produce a single output frame—cannot replicate the reasoning capabilities that chain-of-frames provides. This is presented as evidence for CoF specifically, rather than general video model capability: the sequential generation process, not just the model scale or training data, enables certain reasoning tasks. However, the comparison is confounded by model capacity differences—Nano Banana and Veo 3 are different architectures with different training data, so the single-frame vs. multi-frame distinction is not isolated from other factors.

Majority voting on visual analogies as a diagnostic for systematic biases (Figure 61): While not a traditional ablation, the majority-vote pass@k curves for visual analogies reveal a critical behavior: for reflect and rotate transformations, both Veo 2 and Veo 3 show decreasing performance with more votes, indicating systematic biases. The paper reports that Veo 3 "tends to perform reflections and rotations, but not along the same axis as shown in the image," while Veo 2 "simply tends to copy the object without applying any transformation" (Appendix B.7 legend). This diagnostic analysis demonstrates that failure on reflection/rotation is not a stochastic performance issue but a competence limitation—the models have an incorrect transformation model, not a noisy correct one.

Qualitative failure case analysis (Appendix D, Figures 62–77): The paper documents 16 qualitative failure cases across perception, modeling, manipulation, and reasoning. Notable patterns: (1) Monocular depth and surface normal estimation (Figures 62, 63) fail because the model cannot produce per-pixel continuous value maps—this is an output modality mismatch, not necessarily a lack of 3D understanding. (2) Force prompting and motion trajectory prompting (Figure 64) fail because Veo cannot interpret abstract arrow annotations as control signals—the model was not trained on overlaid symbolic instructions. (3) Physics violations (tying knots, Figure 65; glass falling, Figure 72; collisions, Figure 73) suggest that Veo's physical modeling is approximate and fails for precise mechanical interactions. (4) Combinatorial search tasks (word search, Figure 67; Eulerian path, Figure 68; spot the difference, Figure 70; visual IQ test, Figure 71) largely fail, consistent with the hypothesis that CoF reasoning is effective for sequential path-following (mazes) but not for tasks requiring search over a discrete space of possibilities. (5) Spatial planning tasks (bottleneck, Figure 75; laundry folding, Figure 76; motion planning, Figure 77) fail because Veo cannot maintain rigid-body integrity or respect exact geometric constraints—objects deform, clip through barriers, or move implausibly. These failure cases are presented qualitatively without quantitative metrics, so their frequency and consistency across attempts cannot be assessed.

ReST^EM revision model negative result (Appendix K, Figure 16): Though this appears in the LLM-focused sections of the paper (which are not discussed here), it is worth noting as an ablation mindset: the paper documents a case where attempted optimization backfired, providing negative evidence that constrains the space of effective methods.

Critical Assessment

Do the Experiments Support the Claim That Video Models Are Zero-Shot Learners?

The paper's central empirical claim is that Veo 3 can solve a wide range of tasks it was never explicitly trained for, spanning perception through reasoning. The quantitative experiments provide strong evidence for this claim on the specific tasks tested—but the evidence is narrower than the broad framing suggests. The seven quantitative tasks cover edge detection, segmentation, object extraction, image editing, maze solving, visual symmetry, and visual analogies. These are well-chosen to span the capability hierarchy, but they represent a small and somewhat idiosyncratic selection from the space of possible vision tasks. The 62 qualitative tasks add breadth but are evaluated on only 12 samples each by the authors, with no inter-rater reliability reported and no automated metrics. The success rates in Figure 1 should be interpreted as approximate, author-determined estimates rather than rigorous measurements.

Several tasks show ceiling effects or near-ceiling performance: edge detection (0.77 OIS), object extraction (93% pass@10), color analogies (95% pass@1). These demonstrate capability existence but provide limited signal about the model's limits—a model that gets 95% on color analogies but 19% on rotate analogies has a highly specific and non-general reasoning ability. The claim that Veo 3 is a "generalist" would be stronger if the paper demonstrated a broader range of intermediate-difficulty tasks where performance was nonzero but imperfect, showing that capability exists across the spectrum rather than clustering at easy and impossible.

The zero-shot nature of the demonstrations is genuine: the paper uses off-the-shelf Veo models without fine-tuning, task-specific heads, or architectural modifications. However, the paper does not control for the possibility that some task capabilities may have been implicitly present in training data—for example, videos of mazes being solved, edge-detected images in training captions, or segmentation-like visual effects (green screen compositing). The paper cannot rule out that Veo saw similar tasks during training, which would make these "few-shot" or "implicitly trained" capabilities rather than "zero-shot emergent" ones. The sudden capability jump from Veo 2 to Veo 3 on maze solving (14% → 78% on 5×5 grids) is consistent with emergence at scale, but it is also consistent with Veo 3's training data containing more maze-like content. The paper cannot distinguish these hypotheses because the training data is not disclosed.

Do the Experiments Support the Claim That Chain-of-Frames Is a General Reasoning Mechanism?

The evidence for chain-of-frames as a reasoning primitive comes primarily from the maze solving task, where frame-by-frame path verification shows that Veo 3 can navigate a maze without illegal moves (78% pass@10 on 5×5 grids). The comparison with Nano Banana—which fails entirely on irregular mazes while Veo 3 achieves 75%—provides suggestive evidence that sequential generation helps, but this comparison is confounded: Nano Banana is a different model with different architecture, scale, and training data. The paper cannot rule out that Veo 3 would also solve irregular mazes if it produced a single image output (e.g., if asked to draw the solution path as a line rather than animate it frame by frame). A cleaner test would be to compare Veo 3's frame-by-frame performance against Veo 3 prompted to produce the solution path as a static output (e.g., "Draw the solution path on this maze image"), which would isolate the sequential generation aspect from the model's general capability.

The CoF concept is tested on only one task with rigorous frame-by-frame verification (maze solving). Robot navigation, visual graph traversal, tree BFS, and water puzzle solving are shown qualitatively but not evaluated with frame-level path verification. Visual symmetry and analogies—the other two quantitative reasoning tasks—are evaluated on the final output frame, not on intermediate frames. For these tasks, CoF is not demonstrated as the reasoning mechanism; the model could be computing the solution internally and rendering it in a single frame, with subsequent frames being decorative rather than computational. The paper's claim that "each frame is an intermediate reasoning step" is therefore directly evidenced only for maze solving.

The auto-regressive generation mechanism means that each frame conditions on all previous frames—this is a property of the architecture, not an empirical finding. Whether this property is used for reasoning (as opposed to mere visual coherence) depends on what information propagates across frames. The paper does not perform causal interventions to test this—for example, corrupting intermediate frames and measuring whether downstream frames degrade, or ablating the temporal conditioning and comparing performance. Without such experiments, the CoF hypothesis remains a plausible interpretation of the maze-solving results but is not rigorously distinguished from alternative explanations (e.g., the model plans the path globally and renders it sequentially, with frames reflecting a pre-computed trajectory rather than step-by-step computation).

Do the Scaling Results (Veo 2 → Veo 3) Support the Claim That Video Models Are on a Trajectory to Become Foundation Models?

The Veo 2 to Veo 3 comparisons unambiguously show substantial improvement across all quantitative tasks—often by factors of 2–5×. This demonstrates that video model capabilities are improving rapidly with scale, which is an important empirical finding. However, two data points (two model versions) cannot establish a trajectory. The paper implicitly extrapolates from Veo 2 to Veo 3 to future models, but this extrapolation assumes monotonic improvement at a constant rate—an assumption that may not hold. The paper does not provide scaling law analysis (performance vs. model size, compute, or data quantity) that would enable quantitative extrapolation. The improvement could represent a one-time jump due to architectural changes, training recipe improvements, or data quality enhancements rather than a smooth scaling trend.

The claim that Veo 3 is "akin to a pre-trained language model that has yet to undergo instruction tuning or RLHF" (Section 5) is an analogy, not an empirical claim about Veo 3. It is plausible that post-training techniques developed for LLMs would improve video model performance, but the paper provides no evidence for this—no instruction-tuned video model, no RLHF-trained video model, no comparison of pre- vs. post-trained video model performance. The GPT-3 analogy is rhetorically effective but empirically ungrounded for Veo 3 specifically.

Genuine Weaknesses in Experimental Design

Small test sets for several evaluations. The edge detection evaluation uses 50 images; segmentation uses 50 images; image editing uses 30 images; symmetry uses 25 images per split; object extraction uses 54 images. These are small sample sizes by modern ML evaluation standards, and pass@k estimates at k=10 on 25 samples have substantial binomial variance (a 95% confidence interval for 88% pass@10 on 25 samples would span approximately 25 percentage points). The paper does not report confidence intervals, making it difficult to assess whether differences between conditions (e.g., green vs. white background in segmentation: 0.74 vs. 0.66) are statistically reliable or within sampling noise.

No held-out test set or train/test leakage control. The paper does not describe any split between development and test data. The datasets used (BIPEDv2, LVIS, Emu-edit) have standard train/test splits, and the paper uses their test sets, which is appropriate. However, for maze solving, visual symmetry, and object extraction—which use custom or synthetic datasets—there is no stated separation between prompt development data and evaluation data. Given the extensive prompt sensitivity documented (Appendix C), it is possible that prompts were optimized on the evaluation data, inflating reported performance relative to what would be achieved on genuinely unseen prompts and samples.

Human evaluation without inter-rater reliability. The image editing evaluation uses three human raters but does not report any measure of agreement (e.g., Cohen's kappa, Fleiss' kappa, percent agreement). Without this, it is impossible to know whether the reported fidelity and precision scores (88% and 75% for Veo 3) reflect the model's performance or rater disagreement. The Gemini autorater for visual analogies is validated against human ratings with >88% agreement, but the agreement metric is not specified—is this exact match, Cohen's kappa, or something else? And the validation was on only 25 samples per condition, which is a small sample for establishing reliability.

No controlled investigation of the LLM rewriter contribution. The paper acknowledges that the Vertex API includes an LLM-based prompt rewriter and treats the system as a black box—a reasonable decision for a capability investigation. However, the disentanglement controls are applied only to maze solving, robot navigation, and visual symmetry (via the standalone LLM comparison). For other tasks—particularly symbolic reasoning tasks like Sudoku, sequence completion, and rule extrapolation—the LLM's contribution is not disentangled. The paper acknowledges this for Sudoku (Figure 55: "the solution is likely to come from the LLM instead of the video") but not for other tasks where the LLM could plausibly contribute (sorting numbers, water puzzle solving, tool use). A systematic evaluation of Gemini 2.5 Pro on all qualitative reasoning tasks would substantially strengthen the evidence that CoF—not LLM reasoning—is responsible for the observed capabilities.

Missing baseline: Veo 3 prompted to produce static outputs. To test whether the sequential generation (CoF) provides a computational advantage over single-image output from the same model, the paper could compare Veo 3's performance when prompted to generate a video (frame-by-frame) vs. prompted to produce a single output frame (e.g., by requesting an instantaneous transformation at the first frame and a static scene thereafter). This comparison is never made. The Nano Banana comparison provides some evidence (a single-output model fails on irregular mazes while a video model succeeds), but model identity is not controlled.

Missing baseline: task-specific fine-tuned models. The paper compares Veo 3 to zero-shot baselines (Veo 2, Nano Banana) and to an LLM, but does not compare to fine-tuned specialist models except through literature citations (e.g., SAMv2 for segmentation, SOTA edge detectors for BIPEDv2). A direct comparison on the same test sets with the same metrics—not just citing literature numbers—would make the zero-shot vs. specialist gap more interpretable. The paper's argument is that generalist video models will eventually match or exceed specialists, but the current gap (Veo 3: 0.77 OIS vs. SOTA: 0.90 on edges; 0.74 mIoU vs. SAMv2's unreported but substantially higher mIoU) is substantial, and the paper provides no evidence about how this gap scales with model size or compute.

No latency or cost analysis. The paper generates video at 24 FPS for 8 seconds (192 frames) and evaluates 10 videos per sample (1920 frames total per sample per model). For real deployment, this is extremely expensive—both in compute cost and wall-clock time—compared to a single forward pass through a task-specific model. The paper acknowledges this in Section 5 ("video generation is expensive, but costs tend to fall") and cites Epoch AI's estimate of 9× to 900× per year cost reduction for LLM inference, but provides no video-specific cost data. The 10-generation requirement for pass@10 means that even if video generation becomes cheaper, the multiplicative factor of generating and evaluating multiple candidates remains. The paper does not discuss whether cheaper methods (e.g., early stopping when a verifier detects success, generating shorter videos, using lower resolution or frame rate) could achieve similar performance at lower cost.

Difficulty characterization is absent. Unlike the reference paper (Snell et al., 2024), which bins problems by difficulty and shows qualitatively different scaling behavior per bin, this paper does not systematically characterize which task instances are easy vs. hard for Veo. It reports average performance across test sets but does not analyze variance, failure modes, or instance-level difficulty. The qualitative failure case analysis (Appendix D) suggests that certain task categories (combinatorial search, precise spatial planning) are systematically harder, but this is not quantified. Understanding which problem characteristics predict success vs. failure is critical for deploying these models—a practitioner needs to know not just that Veo 3 achieves 78% on 5×5 mazes, but which mazes are likely to fail and why.

Which Conditions Must Hold for the Claims to Be Valid?

The paper's central claims are conditional on several factors that are established but not fully characterized:

Prompt quality dramatically affects performance. The 40–64 percentage-point range on visual symmetry (Appendix C) means that reported numbers are valid only for the specific prompts tested. A different prompt could make a "failed" capability succeed or a "successful" capability fail. This is consistent with the paper's framing (performance as lower bound), but it means that failure on a task in this paper cannot be interpreted as evidence that the capability is absent—only that it was not elicited with the tested prompts.
The model's motion prior constrains task design. Tasks requiring static outputs (edge detection, segmentation, symmetry) require workarounds (motion outlets, camera control phrases) to prevent the model from animating beyond task completion. The best-frame/last-frame gap varies from 0.03 OIS to 44 percentage points depending on the task, meaning that last-frame reliability—which is what matters for deployment without a frame-selection verifier—is task-dependent in ways the paper does not predict or explain.
Hard reasoning problems remain unsolved. For irregular mazes, the paper reports 75% pass@10 but the irregular maze dataset is only 40 samples (10 hand-drawn, flipped/rotated). The pass@1 might be substantially lower, and performance on genuinely novel irregular mazes (not rotations/flips of 10 base mazes) is not tested. For reflect/rotate analogies, both Veo 2 and Veo 3 perform below chance, and majority voting makes things worse, indicating a systematic competence gap—not a resolvable performance issue. The claim that video models show "early forms of visual reasoning" is supported for path-following tasks (mazes) but not for transformation understanding (analogies) or combinatorial search (Eulerian paths, word search).
The scaling trajectory is inferred from two data points. The Veo 2 → Veo 3 improvement is impressive, but two points cannot establish a trend. The paper would be strengthened by evaluating intermediate-scale models or by providing a scaling law analysis (performance vs. compute or model parameters) using multiple Veo variants. Without this, the claim that "video models will become unifying, general-purpose foundation models" (Section 1) is a plausible extrapolation supported by the Veo 2→3 trend plus the NLP analogy, but is not empirically established by the data presented.

6. Limitations and Trade-offs

6.1 The Pass@k Metric Hides a Massive Practical Compute Cost That Undermines Deployment Viability

The assumption or constraint. The paper's primary quantitative metric is pass@k—the probability that at least one of k independent generated videos solves the task correctly. For most tasks, k ranges from 1 to 10, and the paper emphasizes that pass@10 substantially exceeds pass@1 "with no signs of a plateau" (Section 5). The implicit assumption is that generating multiple candidates and selecting the best via an oracle verifier (ground-truth labels) is a reasonable deployment strategy, and that the headline pass@10 numbers represent practically achievable performance.

The consequence. Each video generation produces 192 frames (8 seconds at 24 FPS). Pass@10 on a test set of N samples requires generating 10N videos—1,920N frames total. For edge detection on 50 images: 500 videos, 96,000 frames. For maze solving on 200 mazes (50 per size × 4 sizes): 2,000 videos, 384,000 frames per model. The total quantitative evaluation generated 17,640 videos (Table 1)—roughly 3.4 million frames at 720p resolution. In deployment, a practitioner would need to generate up to 10 videos per query, then somehow select the correct one without access to ground truth. The paper provides no verifier, heuristic, or automatic selection mechanism—pass@10 assumes oracle selection. The true cost of achieving the reported performance includes both the generation cost (10× what pass@1 requires) and the selection cost (not accounted for). For latency-sensitive applications, generating 10 sequential or parallel 8-second videos per query is likely infeasible regardless of cost trajectory.

What evidence exists in the paper. The pass@k curves are the primary evidence (Figures 3, 4, 5, 7, 8, 9). For example, Veo 3 on 5×5 mazes: pass@1 is substantially lower than pass@10 (78%), though pass@1 numbers are only readable from the plots at k=1 (approximately 40–50% for 5×5 mazes in Figure 7). The gap between pass@1 and pass@10 represents the practical overhead: a 2× to 3× increase in compute for a 1.5× to 2× increase in success rate. For visual symmetry shapes (Figure 8): Veo 3 best-frame pass@1 is approximately 37%, rising to 88% at pass@10—a 2.4× success rate increase for 10× the compute. The paper acknowledges this implicitly in Section 5: "inference-time scaling methods in combination with the standard optimization toolkit like post-training with automatic verifiers are likely to boost performance," but develops no such verifier.

Mitigation status. Not addressed beyond the aspirational statement that automatic verifiers "are likely to boost performance." The paper makes no attempt to build a verifier, test whether simpler selection strategies (e.g., self-consistency via majority voting on extracted answers) could approximate oracle selection, or characterize the pass@1 to pass@k tradeoff curve to help practitioners choose an operating point. The paper does show that majority voting is harmful for some tasks (visual analogies with reflect/rotate, Figure 61: performance decreases with k for majority vote), indicating that naive selection strategies can backfire, making the verifier problem harder, not easier. The gap between the paper's reported numbers and what a practitioner can achieve without oracle access is a first-order deployment barrier that the paper does not quantify or address.

6.2 The Difficulty Estimation Problem Is Completely Unsolved, Yet the Paper's Claims Depend on Its Existence

The assumption or constraint. The paper demonstrates that Veo 3 can succeed on a wide variety of tasks—62 qualitatively and 7 quantitatively—but provides no mechanism for a practitioner to determine in advance whether a given query will succeed. The performance-competence distinction (Section 5) explicitly frames reported performance as a "lower bound" that depends on prompt quality, visual framing, and the specific task instance. The implicit assumption is that a practitioner can engineer prompts and select task instances to achieve the reported success rates, or that the model's competence will be reliably elicited with sufficient prompt optimization.

The consequence. In deployment, a user submits an image and a task description expecting a correct result. The paper provides no way to predict whether the model will succeed on this specific query—no confidence score, no difficulty estimator, no uncertainty quantification. A practitioner deploying Veo 3 for edge detection cannot know whether a particular image will produce a usable edge map (0.77 OIS pass@10 is an average over 50 images, but the variance across images is not reported). A practitioner deploying for maze solving cannot know whether a particular maze is "easy" (5×5 grid, where pass@10 is 78%) or "hard" (9×9 grid, where pass@10 is 12%) without manually categorizing it. The failure case analysis (Appendix D) documents 16 specific failure modes but provides no predictive framework—a user cannot determine whether their task instance falls into a failure category without attempting it.

This is not merely an inconvenience; it is a fundamental barrier to deployment. In the reference paper (Snell et al., 2024), the entire contribution was a compute-optimal allocation strategy that required difficulty estimation—and the authors devoted substantial attention to the difficulty estimation problem, acknowledging its cost and proposing future work. This paper operates at an even earlier stage: it demonstrates capability existence but provides zero infrastructure for deciding when to use the capability versus falling back to a specialist model. The paper's core framing—"video models will become general-purpose foundation models"—requires that deployment be possible without per-task human supervision, but the current system offers no automation for predicting success or selecting prompts.

What evidence exists in the paper. The difficulty-dependent performance variation is visible throughout the quantitative results but is never systematically characterized. Edge detection: the OIS metric is averaged over 50 images, but per-image variance is not reported—some images may achieve near-SOTA performance while others fail completely. Maze solving: performance varies dramatically by grid size (78% on 5×5 vs. 12% on 9×9, Figure 7), establishing that problem complexity matters but providing no automated complexity detector. Visual symmetry: the prompt sensitivity analysis (Table 2) shows that pass@1 varies by 40–64 percentage points depending on prompt phrasing, but provides no guidance on selecting prompts for unseen task instances. Visual analogies: performance ranges from 95% (color) to 19% (rotate), Figure 9, establishing that transformation type is predictive, but transformation type must be known in advance to set expectations.

The qualitative tasks (Figure 1) show success rates ranging from 0.0 to 1.0 across 62 tasks, but each task was evaluated with exactly 12 samples using a single prompt. The variance in Figure 1 is entirely between-task—there is no within-task, between-instance variance reported. A practitioner cannot tell whether an individual query is likely to succeed based on these aggregate numbers.

Mitigation status. Not addressed at all. The paper does not propose a difficulty estimation method, train a confidence predictor, analyze failure correlates, or provide heuristics for query triage. The discussion of "performance as a lower bound" (Section 5) acknowledges the issue conceptually but provides no operational solution. The paper's suggestion that "visual prompt engineering" (Appendix C) is the path forward places the entire burden on the practitioner—prompt engineering is expensive, requires expertise, and must be done per-task and potentially per-instance. This stands in contrast to the NLP trajectory, where LLMs became practically useful partly because they could be reliably prompted with relatively simple instructions for a wide range of queries. If video models require extensive per-instance prompt optimization, they are not general-purpose foundation models in the practical sense of the term.

6.3 The Hardest Reasoning Problems Show Fundamental Competence Gaps, Not Resolvable Performance Issues

The assumption or constraint. The paper claims that video models demonstrate "early forms of visual reasoning" (Section 3, Section 5) and that the chain-of-frames mechanism enables step-by-step reasoning across time and space. The implicit assumption is that current failures on hard reasoning tasks are due to insufficient scale or suboptimal prompting—that the competence exists but is not reliably elicited—and that future models (Veo 4, Veo 5) will close the gap as performance improves along a smooth trajectory.

The consequence. The evidence from the hardest reasoning tasks suggests a more fundamental problem: for certain categories of visual reasoning, Veo 3 does not exhibit partial or noisy competence—it exhibits systematically wrong competence, meaning the model has learned an incorrect transformation rule or reasoning strategy that no amount of scaling or prompt optimization will fix without changing the training data or architecture. This is most starkly visible in the visual analogy results (Figures 9 and 61): both Veo 2 and Veo 3 perform below chance on reflect (29% vs. chance 33%) and rotate (19% vs. chance 33%) transformations, and majority voting over multiple attempts decreases performance—from 29% to 22% for reflect, from 19% to 14% for rotate at k=10 majority vote. This is not a stochastic failure that more attempts can solve; it is a systematic bias toward producing the wrong transformation (the paper reports that Veo 3 "tends to perform reflections and rotations, but not along the same axis as shown in the image").

This failure mode is qualitatively different from the stochastic failures observed in maze solving (where pass@10 substantially exceeds pass@1 because errors are random and independent) or segmentation (where multiple attempts eventually find a good solution). For reflect/rotate analogies, more compute makes things worse—the model is confidently wrong and remains confidently wrong across attempts. This has serious implications for the claims about scaling: if certain reasoning capabilities require representations or training data that Veo lacks entirely, scaling model size may not help—the model may converge to a more confident version of the same wrong answer. The paper's NLP analogy (GPT-3's below-SOTA performance improving with scale) applies to tasks where the model's errors are stochastic and diminish with scale; it does not obviously apply to tasks where the model has learned an incorrect transformation rule that scale reinforces rather than corrects.

Beyond analogies, several qualitative reasoning tasks show near-zero success rates: graph traversal (0.08), sorting numbers (0.08), tree BFS (0.17), maze solving in a photorealistic scene (0.17), rule extrapolation (0.08), all from Figure 1. With 12 samples per task, a 0.08 success rate means 1 out of 12 attempts succeeded—but the paper does not analyze whether the single success shows genuine competence or was a lucky accident. The failure case analysis (Appendix D) documents systematic failures on combinatorial search (word search, Eulerian paths, spot the difference), spatial planning (bottleneck, laundry folding, motion planning), and precise physical reasoning (tying knots, collisions, glass falling). These are not tasks where the model "almost" succeeds—they are tasks where the model fundamentally does not understand the problem constraints.

What evidence exists in the paper. The majority-vote degradation for reflect/rotate analogies (Figure 61) is the strongest quantitative evidence of systematic bias. The qualitative failure cases (Appendix D, Figures 62–77) provide additional evidence of systematic rather than stochastic failure: physics violations (Figures 65, 72, 73), failure to maintain object integrity (Figures 74, 77), and failure to understand abstract visual instructions (Figures 62–64, 67–71). The pattern is consistent: tasks requiring precise geometric reasoning, combinatorial search, or transformation understanding fail categorically, not just with low probability.

For maze solving, performance degrades sharply with maze size (78% → 38% → 12% for 5×5 → 7×7 → 9×9, Figure 7), with Veo 2 at 0% on 7×7 and 9×9. The irregular maze result (75%) appears strong but is based on only 40 samples (10 base mazes flipped/rotated—not independent irregular mazes) and may not generalize to genuinely novel irregular topologies. Without evaluating on a large, diverse set of irregular mazes drawn from a procedural generator (analogous to the rectangular maze dataset), the 75% figure likely overestimates generalization.

Mitigation status. The paper is candid about failures but does not distinguish between stochastic failures (resolvable with more compute or better prompts) and systematic competence gaps (likely requiring architectural changes or different training data). Section 5 notes that "the model performance reported here with a given visual and textual prompt should be considered a lower bound on the model's true capabilities," which conflates both types of failure under the same "prompt engineering could fix this" umbrella. The qualitative failure case analysis (Appendix D) documents failures but does not categorize them by type or assess whether they are prompt-sensitive or fundamental. The paper's optimistic framing—"sparks of visual intelligence," "early forms of visual reasoning"—emphasizes the successes while the evidence suggests that certain reasoning categories may require fundamentally different approaches rather than incremental scaling.

6.4 The Paper Does Not Control for Training Data Contamination, Undermining the "Zero-Shot" Claim

The assumption or constraint. The paper's core claim is that Veo 3 solves tasks it "was neither trained nor adapted for" (Section 1, Abstract), demonstrating "emergent zero-shot perceptual abilities well beyond the training task" (Section 3, Takeaway 2). This claim assumes that the specific evaluation tasks—edge detection on BIPEDv2, segmentation on LVIS, maze solving on grid mazes, etc.—were not present in Veo's training data, or that if similar content existed, the model's ability to generalize to specific prompt formats represents genuine zero-shot transfer rather than memorization. The paper provides no evidence about training data composition and no method for detecting contamination.

The consequence. The distinction between "zero-shot emergence" and "implicit training" is not academic—it determines whether capabilities will generalize to out-of-distribution task instances. If Veo 3 succeeds at edge detection because its training data included edge-detected images or videos with edge-extraction effects, then its performance may degrade on edge detection tasks that differ from training (e.g., medical images, abstract art, unusual lighting conditions). If Veo 3 succeeds at maze solving because its training data included maze-navigation videos (e.g., video games, puzzle-solving animations), then it may fail on maze variants that differ from training distributions (e.g., mazes with moving obstacles, 3D mazes, mazes with non-visual constraints). The paper provides several suggestive examples of potential contamination: the green screen effect in segmentation (Section 4.2, the model "consistently performs better with a green background than a white one," attributed to "the widespread use of green screens" in training data) suggests that the model has learned statistical associations from specific training phenomena rather than developing general segmentation capabilities.

The Veo 2 to Veo 3 improvement is particularly vulnerable to this critique. The dramatic jump in maze solving (14% → 78% on 5×5, Figure 7) could reflect Veo 3's training data containing substantially more maze-like content rather than a genuine emergence of path-planning capability. Similarly, the emergence of analogical reasoning for color and resize but not reflect and rotate (Figure 9) could reflect the distribution of transformations in training data—color changes and size changes appear frequently in natural videos, while reflection and rotation symmetries are rarer. The paper's interpretative framing (emergence, zero-shot) assumes that the training data distribution is not the primary driver, but cannot rule it out.

What evidence exists in the paper. The paper provides no training data analysis, no contamination probes, and no membership inference tests. The green screen finding in segmentation (Section 4.2) is the only direct evidence that training data statistics affect task performance, and it suggests contamination-like effects. The systematic improvement with prompt engineering (Appendix C, Table 2) is ambiguous: it could indicate that better prompts elicit more general capabilities, or that better prompts more closely match the phrasing patterns present in training data (e.g., captions describing video editing operations). The LLM disentanglement control (standalone LLM cannot solve mazes from images, Section 4.5) rules out the prompt rewriter as the source of the solution, but it does not rule out training data memorization by the video model. The irregular maze result (75% pass@10, Figure 7) provides some evidence against pure memorization—irregular mazes are unlikely to appear verbatim in training data—but the dataset is small (40 samples from 10 base mazes) and the result may not replicate on larger, more diverse irregular maze sets.

Mitigation status. Not addressed. The paper does not discuss training data contamination as a potential confound, does not propose methods to detect it, and does not qualify its zero-shot claims in light of training data uncertainty. This is common in capability demonstration papers but weakens the strong emergent-capability interpretation. A more conservative framing—"Veo 3 can perform tasks that its training objective does not explicitly specify, though training data similarity may contribute"—would be more defensible but less impactful than the paper's "emergent zero-shot abilities" framing.

6.5 The Single Model Family and Single Evaluation Paradigm Limit Generalization of the Findings

The assumption or constraint. The paper evaluates exactly one model family (Veo) across exactly two versions (Veo 2, Veo 3), accessed through a single API (Vertex AI) with a specific prompt rewriting pipeline. All tasks follow the same input-output paradigm: an initial image plus a text prompt produces a 16:9, 720p, 24 FPS, 8-second video. The paper's conclusions—"video models are zero-shot learners and reasoners" (title), "video models will become unifying, general-purpose foundation models for machine vision" (Section 1)—are stated generically for "video models" as a class, not for Veo specifically. The paper justifies this in Section 2: "The core argument of this paper—that video models are zero-shot learners and reasoners—can be supported by demonstrating success on any sufficiently capable model. We choose Veo because it has consistently ranked high on text2video and image2video leaderboards."

The consequence. The paper provides an existence proof—one video model can do these things—but cannot distinguish between three hypotheses: (1) The capabilities are inherent to the video generation training objective at sufficient scale and will appear in any large video model trained on diverse data (the paper's implicit claim). (2) The capabilities are specific to Veo's architecture, training recipe, or data mixture and may not transfer to other video models (Sora, Kling, Gen-3, etc.). (3) The capabilities arise from the specific interaction between Veo, the Vertex API prompt rewriter, and the evaluation paradigm, and may not generalize to other API interfaces or deployment contexts.

The practical consequence is that a practitioner using a different video model cannot rely on this paper's findings to predict what that model can do. The Veo 2 to Veo 3 improvement (often 2–5×, as documented across all quantitative tasks) demonstrates that Veo's specific scaling trajectory is steep, but provides no evidence about whether other model families follow the same trajectory or at what rate. The paper's NLP analogy—GPT-3 showed emergent few-shot learning that subsequent LLMs replicated—is invoked to argue that Veo's capabilities will generalize, but this is an empirical claim that requires testing, not an argument that follows from the NLP precedent. Different video model architectures (diffusion, autoregressive, hybrid), training objectives (text-to-video, image-to-video, joint image-video), and data distributions could lead to qualitatively different capability profiles.

Furthermore, the black-box API evaluation (Section 2) means that capabilities attributed to "Veo" may actually be capabilities of the API system, which includes the LLM-based prompt rewriter. The paper's LLM disentanglement controls (Section 3.4.6) rule out the LLM as the sole source of success on maze solving and symmetry, but cannot rule out that the LLM rewriter contributes to success by reformulating prompts in ways that better elicit Veo's capabilities. A different API without prompt rewriting might achieve different results even with the same underlying Veo model. The black-box nature also means the paper cannot characterize how capabilities depend on model scale, architecture, or training data within the Veo family—only two model versions are available, and no intermediate checkpoints or ablations are tested.

What evidence exists in the paper. All quantitative results compare Veo 2 and Veo 3 (and occasionally Nano Banana and Gemini 2.5 Pro as reference points, not as alternative video models). No other video generation model is evaluated. The paper does not cite or compare against published results from other video model families, nor does it discuss whether competing models would be expected to show similar capabilities. The qualitative results (62 tasks, Figure 1) are exclusively Veo 3 results—Veo 2 is not evaluated on these tasks, so the Veo 2 to Veo 3 trend (established quantitatively for 7 tasks) cannot be extrapolated to the broader capability set.

The paper's theoretical argument for generality rests entirely on the NLP analogy (Section 1: "the exact same primitives that enabled zero-shot learning in NLP also apply to today's generative video models—large-scale training with a generative objective on web-scale data"). This is a plausibility argument, not empirical evidence. NLP and vision may differ in ways that affect emergence—the structure of video data, the continuous nature of pixel space, the different inductive biases of diffusion vs. autoregressive architectures, and the different scales of current models relative to data diversity all could affect whether video models follow the same trajectory as LLMs.

Mitigation status. The paper acknowledges the single-model limitation implicitly by framing Veo as representative: "We choose Veo because it has consistently ranked high on text2video and image2video leaderboards" (Section 2). This does not mitigate the limitation—it asserts that Veo is representative without evidence that other models would show similar capabilities. The paper suggests no multi-model study, no standardized benchmark that would enable cross-model comparison, and no analysis of which Veo-specific properties (architecture, training data, scale) are necessary for the observed capabilities. The NLP analogy provides rhetorical support but does not substitute for empirical validation across model families.

6.6 The Best-Frame Evaluation Protocol Masks a Critical Temporal Precision Problem with No Proposed Solution

The assumption or constraint. The paper reports performance under two evaluation protocols: best-frame (take the optimal frame from the 192-frame video, representing the model's capability ceiling) and last-frame (take only the final frame, representing practical reliability without oracle selection). The best-frame numbers are consistently higher and are the ones emphasized in the paper's claims (e.g., "Veo 3 achieves 0.74 mIoU," "Veo 3 achieves 88% pass@10 on shapes"). The implicit assumption is that the gap between best-frame and last-frame represents a resolvable issue—the model "solves" the task at some intermediate frame but continues animating, and a frame-selection mechanism could recover the best-frame performance in deployment.

The consequence. The best-frame/last-frame gap is not a small correction term—it is enormous for several tasks. For visual symmetry on shapes: best-frame pass@10 is 88%, last-frame pass@10 is 44% (Figure 8)—a 44 percentage-point gap, meaning half of the successes are lost by the final frame. For visual symmetry on random patterns: 100% best-frame vs. 72% last-frame—a 28 percentage-point gap. For segmentation: 0.74 best-frame mIoU vs. 0.56 last-frame with green background (Figure 4)—a 0.18 mIoU gap, roughly 25% relative degradation. For these tasks, the paper's headline numbers represent performance that a practitioner cannot achieve without solving the frame selection problem—they can generate a video that contains the correct answer at some point, but they cannot automatically identify which frame contains it.

The frame selection problem is not trivial. The correct frame is not necessarily the last one, or the one with highest visual quality, or the one with least motion. For edge detection, a frame with clear edges might be followed by frames where edges are further refined or degraded—selecting the best frame requires comparing each frame's edge map to ground truth, which is exactly what the evaluation does and exactly what a practitioner cannot do. For segmentation, the optimal frame might occur mid-video when all objects are correctly masked, with subsequent frames adding spurious colors or merging masks. For symmetry, the correct completion might appear instantly (best prompt: "Instantly reflect this pattern...") and then be overwritten by the model's motion prior.

The paper notes that "Veo has a strong prior to keep things moving" (Appendix C) and recommends providing a "motion outlet" (e.g., a spinning color wheel) to indicate task completion and freeze the solution. This workaround is demonstrated qualitatively for the Omniglot character recognition task (Figure 30) but is not systematically evaluated across tasks. It also requires task-specific design—the motion outlet must be visually compatible with the task and must not interfere with evaluation. For tasks evaluated in the paper, motion outlets are not used in the quantitative experiments, meaning the reported best-frame/last-frame gaps reflect performance without this mitigation.

What evidence exists in the paper. The best-frame/last-frame gaps are reported for edge detection (Figure 3: 0.77 vs. 0.74 OIS—small), segmentation (Figure 4: 0.74 vs. 0.56 mIoU—moderate to large depending on background color), and visual symmetry (Figure 8: 88% vs. 44% on shapes, 100% vs. 72% on random—very large). Maze solving and visual analogies are evaluated on the last frame only (or frame-by-frame for maze path verification), so their best-frame/last-frame gaps are not characterized. Object extraction uses the last frame only. Image editing is evaluated by humans on the generated videos as a whole (not frame-by-frame), so the gap is not measured.

The variation in gap size across tasks—0.03 OIS for edges vs. 44 percentage points for symmetry—is not explained or predicted by the paper. It suggests that temporal precision is highly task-dependent, but the paper provides no framework for predicting which tasks will suffer from large gaps or how to design prompts to minimize them (beyond general advice in Appendix C).

Mitigation status. The paper proposes no frame-selection mechanism, no verifier that could approximate best-frame performance without ground truth, and no study of whether simple heuristics (e.g., first frame where the task appears complete, frame with minimum motion, frame closest in time to a motion outlet event) could recover a substantial fraction of the best-frame/last-frame gap. Appendix C provides prompting best practices (motion outlets, speed control, scene/camera controls) that may reduce the gap, but these are not systematically evaluated. The paper's framing—best-frame as "performance ceiling" and last-frame as "practical disadvantage"—acknowledges the issue but does not provide a path to closing the gap in deployment. This is a significant omission because the best-frame/last-frame gap represents a fundamental tradeoff: the model's generative training objective optimizes for continuous motion and visual interest, which is directly at odds with the static outputs required by many vision tasks. Resolving this tradeoff may require post-training (instruction tuning to "stop when done"), architectural changes (explicit termination tokens), or a separate verifier model—none of which are explored.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the conversation about video models from content generation to general-purpose visual computation. Before this work, video models were evaluated primarily on visual quality—how realistic the output looked, how coherent the motion, how well the prompt was followed in terms of scene content. The chain-of-frames (CoF) concept reframes video generation as a computational process where each frame is an intermediate reasoning step, and the correct evaluation question is not "does this look good?" but "is the frame-by-frame computation correct?" This is a paradigm shift for video model evaluation—it redirects attention from perceptual metrics (FVD, IS, FID) toward task-specific correctness verification (frame-level path checking, cell-by-cell color accuracy, autorater-based answer selection). If the field adopts this framing, video model benchmarks will increasingly look like reasoning benchmarks (with pass@k, best-of-N, and verifier-based selection) rather than image quality benchmarks.

The paper also provides the first systematic evidence that a single video model can span the full vision stack—perception, modeling, manipulation, reasoning—in a zero-shot manner. This changes the conversation about vision architectures from "which specialized model should I train for this task?" to "can I prompt a sufficiently capable video model instead?" It does not settle this question—Veo 3 performance is below specialized SOTA on most tasks—but it establishes that the question is worth asking, and that the answer is trending in the direction of generalist models. The Veo 2 to Veo 3 scaling comparison (e.g., 14% → 78% pass@10 on 5×5 mazes) provides the empirical warrant for this shift: capabilities that were effectively absent in the previous generation became reliable in the next, suggesting that current limitations are transient features of model scale, not fundamental boundaries of the video generation paradigm.

A specific, concrete shift: edge detection and segmentation—tasks that have motivated decades of specialized architecture design (Canny, HED, RCF, SAM, SAMv2)—can now be performed by prompting a generalist video model with no task-specific training. The performance (0.77 OIS, 0.74 mIoU) is not SOTA, but the fact that it works at all for models trained only on video prediction suggests that much of what specialized vision architectures compute may be implicitly learned as intermediate representations in large generative models. This has a direct implication for research prioritization: rather than designing new edge detectors or segmentors, it may be more productive to improve generalist video models and develop verifiers or lightweight adapters that extract these implicit representations reliably.

The paper reconciles a latent tension in the video generation literature. On one side, video models have been studied as physics engines, world simulators, and planning tools (Yang et al., 2024; Motamed et al., 2025; Assran et al., 2025)—work that treats video generation as predictive. On the other side, video models have been studied as creative tools for content generation, editing, and stylization—work that treats video generation as expressive. The paper's capability hierarchy (perception → modeling → manipulation → reasoning) unifies these perspectives: the same model that predicts physically plausible object dynamics (modeling tier) can also creatively edit images (manipulation tier) and reason about constrained paths (reasoning tier). The hierarchy provides a framework for understanding why a single model can do all of these: the generative training objective forces the model to develop all four tiers simultaneously, because predicting future frames requires perceiving objects, modeling their behavior, manipulating them in imagination, and reasoning about their trajectories.

A subtler shift: the paper introduces visual prompt engineering as a first-class concern. The 40–64 percentage-point pass@1 range on visual symmetry across 10 prompts (Appendix C, Table 2) makes this concrete—prompt quality is not a nuisance variable but a primary determinant of measured capability. This has implications for how video model benchmarks should be constructed. Unlike LLM benchmarks, where prompts are typically short natural language instructions, video model prompts may need to include explicit camera controls, scene constraints, motion outlets, and speed specifications. The paper's prompting best practices (Appendix C) provide an initial guide, but the space of effective prompt design strategies is largely unexplored. This shifts the research burden: building a benchmark for video model reasoning is not just about selecting tasks but about discovering the prompt formulations that elicit the relevant capabilities—and reporting results without documenting prompt sensitivity may be deeply misleading.

The paper also changes how we think about the scaling of vision models. The prevailing narrative—from the Chinchilla scaling laws through GPT-4 and beyond—is that language model capabilities improve predictably with compute. No analogous scaling narrative exists for vision, because vision has been fragmented across specialized models at different scales. By demonstrating that a single model family (Veo) shows substantial capability gains across a broad task suite in a ~6-month period, the paper provides the first suggestive evidence that video models may follow a similar scaling trajectory. This does not establish scaling laws (two data points cannot), but it establishes that scaling video models is a tractable empirical question—the Veo 2 to Veo 3 jump is large enough to be measured reliably, and the pass@k curves (no apparent plateau) suggest headroom for further scaling. The paper makes scaling video models a more attractive investment, both intellectually and financially.

Follow-Up Research This Work Enables

Systematic difficulty characterization and verifier training for video model tasks. The paper demonstrates that Veo 3 succeeds on many tasks but provides no way to predict success on a given instance. A direct follow-up would construct larger, stratified test sets for maze solving (1000+ mazes varying in size, branching factor, solution path length, grid vs. irregular topology) and visual symmetry (500+ patterns varying in number of distinct colors, pattern complexity, presence/absence of semantic content) and measure performance as a function of task instance features. The result would be task-specific scaling curves—does performance degrade gracefully with maze size, or is there a phase transition? Does semantic content (shapes vs. random patterns) help or hurt? Answering these questions would provide the difficulty estimation infrastructure that the paper's compute-optimal framework (from the LLM scaling literature) requires. A natural extension is training a lightweight verifier model on Veo-generated outputs with ground-truth correctness labels—this verifier could then be used to implement pass@k with automatic selection, closing the pass@1 to pass@10 gap in deployment. The key experiment: measure pass@k performance using the trained verifier for selection vs. oracle selection, and report the efficiency loss.

Multi-model capability comparison to test whether capabilities are inherent to video generation or specific to Veo. The paper evaluates one model family across two versions. A direct stress-test of the "video models are zero-shot learners" claim would replicate the seven quantitative tasks from this paper on at least three other video generation models—ideally spanning different architectures (diffusion-based like Sora, autoregressive, hybrid), different training data distributions, and different scales. Each model would be prompted with the same images and text prompts (adapted minimally for model-specific prompt formats). The key questions: Do all sufficiently large video models show maze-solving ability? Is the pass@k scaling trend (substantial pass@1 to pass@10 improvement) universal, or do some models saturate at pass@1? Are the failure modes similar across models—do all models fail on reflect/rotate analogies, depth estimation, and combinatorial search? If all models show the same capability profile (similar strengths, similar weaknesses), the evidence favors the hypothesis that these capabilities are inherent to the video generation objective at scale. If models diverge substantially, the capabilities depend on architecture or training data specifics, and the "video models" framing is premature. The experiment would require API access to multiple models and a budget of roughly 10,000–20,000 video generations per model (scaling down from this paper's 17,640 per model by focusing only on quantitative tasks).

Causal tests of the chain-of-frames mechanism through frame-level interventions. The paper argues that each generated frame is an intermediate reasoning step, but provides correlational evidence (maze solving performance, comparison against single-frame Nano Banana). A causal test would intervene on Veo's generation mid-video and measure the effect: (a) Frame corruption: for a maze-solving video that correctly reaches the goal, take an intermediate frame, mask the red circle's position (replace it with background pixels), and resume generation from this corrupted frame. Does the model recover the correct path, hallucinate a new position, or fail? (b) Frame reordering: take a sequence of frames from a successful maze solve, reverse their order, and condition generation on the reversed sequence. Does the model reconstruct a forward path, or produce nonsense? (c) Premature termination: for tasks like segmentation, stop generation at the frame where the model has produced correct masks (as identified by oracle) and measure whether the model would have degraded the masks had generation continued—this quantifies the temporal precision problem directly. These interventions would distinguish between two hypotheses: CoF as genuine step-by-step computation (where frame N+1 depends causally on the solution state in frame N) vs. CoF as smooth rendering of a pre-computed plan (where frames are temporally coherent but not computationally necessary). The experiments could be run on a subset of 100 maze-solving instances with 50 successful and 50 unsuccessful trajectories.

Post-training for instruction following and stopping behavior. The paper notes that Veo 3 is "akin to a pre-trained language model that has yet to undergo instruction tuning or RLHF" (Section 5). A direct test of this analogy would apply instruction tuning to a video model (if access to model weights permits) by constructing a dataset of (input image, text instruction, target output video) triples. The training data could be bootstrapped from Veo 3 itself: for each of the 62 qualitative and 7 quantitative tasks, generate 100 videos, select the best ones using task-specific metrics, and use these as supervised targets. The hypothesis: instruction tuning would substantially close the best-frame to last-frame gap (by teaching the model to stop when the task is complete), improve prompt robustness (reducing the 40–64 percentage-point sensitivity range), and boost pass@1 performance closer to pass@10. A negative result—instruction tuning provides minimal benefit—would suggest that the temporal precision problem is architectural (the autoregressive generation model lacks an explicit termination mechanism) rather than a matter of post-training. The experiment also tests whether instruction tuning enables capabilities that are currently absent: do instruction-tuned models begin to solve depth estimation, surface normal estimation, or reflect/rotate analogies, or do these require fundamentally different training data?

Training data ablation and contamination analysis. The paper cannot distinguish emergent zero-shot capabilities from training data memorization because Veo's training data is not disclosed. A follow-up using an open-weight video model (if available at sufficient scale) or a controlled training setup could investigate this directly. The experiment: train multiple video models at the same scale but on different data distributions—one on natural videos only, one on natural videos plus video-game footage (which contains maze navigation), one on natural videos plus video-editing tutorials (which may contain segmentation-like effects), one on natural videos plus synthetic visual analogy data. Test all models on the paper's task suite. If maze solving requires maze-like training data (performance drops to zero when video games are excluded), the zero-shot claim is weakened for that capability. If edge detection works regardless of whether editing tutorials are included, the capability is genuinely emergent from the video prediction objective. This experiment is currently speculative (requires training multiple large video models from scratch) but becomes tractable if video model training follows the trajectory of LLM training, where multiple groups train comparably-capable models and some are released openly.

Verifier-guided chain-of-frames reasoning with adaptive compute allocation. The paper shows that pass@10 substantially exceeds pass@1, but provides no mechanism for selecting which of the 10 generated videos to trust. A natural extension—inspired by the test-time compute scaling literature (Snell et al., 2024) and the verifier-as-scorer framework—is to train a per-frame correctness predictor that can detect whether a video is on track. For maze solving, this predictor could check whether the red circle remains within white pixels frame-by-frame and whether it is making progress toward the goal (decreasing Euclidean distance). The predictor would serve as a real-time verifier: if the circle crosses a wall or stalls, generation could be terminated early and restarted with a different seed. This turns the 10× compute overhead of pass@10 into a dynamic, adaptive budget—easy mazes might succeed at pass@1, medium mazes might use 2–3 restarts, hard mazes might use 10. The experiment would measure: given a fixed generation budget (e.g., 10 video attempts total, with early termination saving compute on failed attempts), what is the success rate compared to generating 10 full videos and using oracle selection? This directly addresses the deployment cost problem identified in Section 6.1.

Practical Applications and Downstream Use Cases

Automated visual data curation and annotation for training specialist models. The paper shows Veo 3 achieves 0.74 mIoU on instance segmentation and 0.77 OIS on edge detection without task-specific training. While these numbers are below SOTA specialist models, they are sufficient for bootstrapping: a practitioner could use Veo 3 to generate pseudo-labels (segmentation masks, edge maps) on a large unlabeled image corpus, then train a lightweight specialist model (e.g., a small CNN or ViT) on these pseudo-labels, using a small set of human-annotated data for final fine-tuning. The economics: generating 10 videos per image and selecting the best frame via a simple heurist (e.g., frame with maximum edge density for edge detection, frame with maximum number of distinct color segments for segmentation) costs roughly 10× the inference cost of a single video generation, but eliminates the need for human annotation on the full corpus. If video generation costs follow the LLM inference cost trajectory (falling 9×–900× per year per Epoch AI), this becomes cheaper than human annotation within 1–3 years for many tasks. The key enabling result: Veo 3's edge maps are more detailed than BIPEDv2 ground truth (Figure 60), suggesting that the pseudo-labels may actually be higher quality than human annotations for certain image types.

Zero-shot visual reasoning for assistive technology and accessibility. The maze solving and robot navigation results (78% pass@10 on 5×5 mazes, 75% on irregular mazes, 58% success rate on robot navigation) suggest that video models can perform spatial reasoning from visual input alone. A practical application: a visually impaired user captures an image of their environment (a room, a hallway, a street crossing) and asks a navigation question ("how do I get to the door?", "where is the nearest exit?"). Veo 3 could generate a video showing the path, which could then be described by a vision-language model or converted to verbal instructions. The key advantage over a static vision-language model: the chain-of-frames output shows the trajectory step-by-step, capturing spatial constraints (obstacles, narrow passages, walls) that a single-image analysis might miss. The current limitation is latency (generating an 8-second video is too slow for real-time assistance) and the lack of a verifier to ensure the path is valid. But for non-real-time use cases (pre-trip planning, unfamiliar building navigation), the latency is acceptable, and the 58–78% success rates suggest practical utility with a human-in-the-loop verifying the suggested path before following it.

Content moderation and forensic video analysis via zero-shot perceptual tasks. The paper demonstrates that Veo 3 can perform conjunctive search (Figure 17: finding the blue ball among distractors, 0.75 success rate), background removal (Figure 32: removing background to isolate subjects, 0.83 success rate), and detailed edge detection. These capabilities are directly applicable: an investigator with a corpus of surveillance footage could prompt Veo 3 to "highlight all edges of objects carried by the person in the red jacket" or "segment out all individuals wearing backpacks." Because Veo operates on the input image/video as the first frame and reasons visually, it does not require pre-trained object detectors for specific object categories—the search and segmentation are category-agnostic and prompt-defined. The current limitation is that Veo generates a 16:9 video output; for forensic use, the relevant frame would need to be extracted (best-frame oracle selection, or a trained verifier). The pass@1 to pass@10 gap (e.g., edge detection OIS improves from ~0.55 to 0.77) means that generating 10 video variants per query and having a human analyst select the best result is a viable workflow—costly per query, but acceptable for high-stakes forensic applications where analyst time is the bottleneck, not compute.

Rapid prototyping of visual effects and editing pipelines. The image editing results (88% fidelity, 75% precision human-rated on Emu-edit; background removal at 0.83 success rate; style transfer at 0.75; novel view synthesis at 0.92) demonstrate that Veo 3 can perform sophisticated image manipulations from text instructions alone. For video editors, visual effects artists, or content creators, this enables a workflow where the initial edit is generated by prompting Veo, and the result is refined manually or with specialist tools. The key advantage over specialist image editors: Veo handles diverse edit types (color changes, object removal, style transfer, scene composition, 3D-aware reposing) through a single interface, without switching between tools. The primary limitation for deployment is precision (75% means 25% of edits have unintended changes—camera movement, animating static elements) and the need to extract a single frame from the video output. If the precision issue can be addressed through better prompt engineering (camera control phrases, motion outlets) or post-training, the workflow becomes: text instruction → Veo generates edit options → user selects best frame → manual touch-up. The paper already shows that prompt engineering substantially affects precision (best prompts yield pass@1 of 48% vs. 8% worst on symmetry), suggesting that task-specific prompt templates could be developed for common editing operations.

When to Prefer This Method

The paper does not propose a "method" in the traditional sense—it is a capability evaluation, not an algorithmic contribution. However, the paper's results imply a decision framework for practitioners choosing between prompting a generalist video model, using a task-specific vision model, or employing a language model for a visual task. The implicit tradeoffs, grounded in the paper's quantitative results, are:

Prefer prompting a generalist video model (Veo 3 style) when:

The task requires integrating perception, modeling, and manipulation simultaneously (e.g., "extract all animals from this scene and line them up"—requires segmentation + counting + spatial rearrangement). Specialist models handle each sub-task independently; a video model handles them in a unified prompt.
The task involves step-by-step spatial reasoning that benefits from visual state tracking (e.g., maze solving, robot navigation, visual graph traversal). The paper's comparison against Gemini 2.5 Pro (I2T: 16% on 5×5 mazes vs. Veo 3's 78%) and Nano Banana (0% on irregular mazes vs. Veo 3's 75%) supports the claim that chain-of-frames provides advantages over both symbolic reasoning and single-image output for these tasks.
The task is qualitative, creative, or requires human judgment rather than a precise numeric output (e.g., style transfer, scene composition, transfiguration). The paper's human evaluation for image editing (88% fidelity) suggests the model produces usable outputs even when automated metrics would penalize minor variations.
The practitioner is willing to generate multiple candidates (pass@10) and select the best output manually or via a verifier. The consistent pass@1 to pass@10 improvement across all tasks means that reliability can be substantially improved by generating multiple options, but this requires a selection mechanism.

Prefer a task-specific specialist model when:

The task requires precise, quantitative outputs where SOTA performance matters and is measured by a narrow metric (e.g., edge detection OIS, segmentation mIoU, object detection mAP). Veo 3 achieves 0.77 OIS vs. SOTA 0.90 on BIPEDv2; 0.74 mIoU vs. SAMv2's substantially higher performance. For production systems where these metrics directly translate to business value, the gap is meaningful.
The task requires real-time or high-throughput inference, and the cost of generating multiple 8-second videos per query (192 frames each, potentially k=10 candidates) is prohibitive. The paper provides no latency benchmarks, but 192 frames of autoregressive generation per video is orders of magnitude more expensive than a single forward pass through a specialist model.
The input distribution is narrow and well-characterized (e.g., "segment cars in autonomous driving footage"), and a specialist model can be trained to near-perfect accuracy on that distribution. Generalist video models provide breadth but sacrifice depth; if only one task matters, the specialist is more efficient.
The task requires output modalities that video models cannot currently produce (e.g., per-pixel depth maps, surface normals, force vectors). The paper's failure cases on depth estimation and surface normal estimation (Figures 62, 63) suggest that these output types are fundamentally mismatched with the video generation training objective.

Prefer an LLM with text-based reasoning over a video model for visual tasks when:

The task can be accurately translated to a symbolic representation (e.g., mazes as ASCII grids, visual analogies as structured descriptions). Gemini 2.5 Pro T2T outperforms Veo 3 on small text-representable mazes (92% vs. 78% on 5×5), suggesting that symbolic reasoning is better than visual reasoning when the translation is lossless.
The visual input is diagrammatic or structured rather than naturalistic (e.g., charts, tables, graphs, grids). LLMs have been heavily optimized for structured reasoning from text; video models have been optimized for natural scene dynamics. The paper's 9×9 maze result (Veo 3 12% vs. Gemini T2T 2%) suggests the cross-over point depends on problem complexity, with visual reasoning becoming advantageous when the problem is large and the visual representation encodes spatial constraints that are expensive to verbalize.

🎯 Pitch

1. Executive Summary

2. Context and Motivation

The Core Problem: Vision Lacks a Unifying Foundation Model Paradigm

Why This Problem Matters Now

Where Existing Approaches Fall Short

How This Paper Positions Itself

3. Technical Approach

3.1 Reader Orientation

3.2 Big-Picture Architecture (Diagram in Words)

3.3 Roadmap for the Deep Dive

3.4 Detailed, Sentence-Based Technical Breakdown

The Generation Interface: Prompting Veo as a Black Box

Chain-of-Frames (CoF): The Core Reasoning Mechanism

The Four-Tier Capability Hierarchy

Task-Specific Prompting and Evaluation Protocols (Quantitative Tasks)

Edge Detection (Section 4.1, Appendix B.1)

Class-Agnostic Instance Segmentation (Section 4.2, Appendix B.2)

Object Extraction (Section 4.3, Appendix B.3)

Image Editing (Section 4.4, Appendix B.4)

Maze Solving (Section 4.5, Appendix B.5)

Visual Symmetry Solving (Section 4.6, Appendix B.6)

Visual Analogy Completion (Section 4.7, Appendix B.7)

Best-Frame vs. Last-Frame Evaluation and the Pass@k Metric

LLM Disentanglement Control: Isolating Visual Reasoning from Language Reasoning

Summary of Design Choices and Their Justifications

4. Key Insights and Innovations

Innovation 1: Chain-of-Frames as a General Reasoning Primitive, Not Just a Rendering Artifact

Innovation 2: The Four-Tier Capability Hierarchy as a Diagnostic Framework for Emergence

Innovation 3: The Performance-Competence Distinction Applied to Video Model Evaluation

Innovation 4: Evidence That Video Models Can Substitute for Task-Specific Vision Models—With Sharp, Characterized Boundaries

5. Experimental Analysis

Evaluation Methodology

Main Quantitative Results

Perception: Edge Detection and Segmentation

Manipulation: Object Extraction and Image Editing

Reasoning: Maze Solving, Visual Symmetry, and Visual Analogies

Scaling Trends Across All Tasks

Ablation Studies and Robustness Checks

Critical Assessment

Do the Experiments Support the Claim That Video Models Are Zero-Shot Learners?

Do the Experiments Support the Claim That Chain-of-Frames Is a General Reasoning Mechanism?

Do the Scaling Results (Veo 2 → Veo 3) Support the Claim That Video Models Are on a Trajectory to Become Foundation Models?

Genuine Weaknesses in Experimental Design

Which Conditions Must Hold for the Claims to Be Valid?

6. Limitations and Trade-offs

6.1 The Pass@k Metric Hides a Massive Practical Compute Cost That Undermines Deployment Viability

6.2 The Difficulty Estimation Problem Is Completely Unsolved, Yet the Paper's Claims Depend on Its Existence

6.3 The Hardest Reasoning Problems Show Fundamental Competence Gaps, Not Resolvable Performance Issues

6.4 The Paper Does Not Control for Training Data Contamination, Undermining the "Zero-Shot" Claim

6.5 The Single Model Family and Single Evaluation Paradigm Limit Generalization of the Findings

6.6 The Best-Frame Evaluation Protocol Masks a Critical Temporal Precision Problem with No Proposed Solution

7. Implications and Future Directions

How This Work Changes the Landscape

Follow-Up Research This Work Enables

Practical Applications and Downstream Use Cases

When to Prefer This Method

Adjacent in domain

MetaEmbed: Scaling Multimodal Retrieval at Test-Time with Flexible Late Interaction

CapRL: Stimulating Dense Image Caption Capabilities via Reinforcement Learning