VideoPhy: Evaluating Physical Commonsense for Video Generation

ArXiv: 2406.03520

🎯 Pitch

Even the best text-to-video models fail to respect basic physics over 60% of the time—pouring milk doesn't raise the liquid level, rigid wood deforms, and sand shifts without force. VIDEOPHY systematically exposes this gap across 688 real-world interaction prompts, showing that scaling model size provides only marginal improvement and that physical commonsense remains a critical, unsolved challenge for video generation.

1. Executive Summary

This paper introduces VIDEOPHY, a benchmark designed to assess whether text-to-video generative models produce outputs that adhere to physical commonsense for real-world activities (e.g., marbles rolling down a slanted surface). Evaluating twelve diverse models—including open models like CogVideoX and closed models like Lumiere and Dream Machine—on 688 human-verified captions spanning solid-solid, solid-fluid, and fluid-fluid interactions, the work uses human evaluation to jointly measure semantic adherence (whether the video depicts the described action) and physical commonsense (whether the depicted dynamics follow intuitive physics). The best-performing model, CogVideoX-5B, generates videos that satisfy both criteria for only 39.6% of prompts, while all other models fall below 20%, establishing that current video generative models severely lack physical commonsense capability and that scaling model capacity provides only partial mitigation—performance degrades particularly sharply on prompts involving solid-solid interactions and on captions judged as harder to simulate physically.

2. Context and Motivation

The Core Problem: We Don't Know Whether Video Generators Understand Physics

Text-to-video (T2V) generative models have advanced rapidly, with models like Sora, CogVideoX, and Lumiere now capable of producing photo-realistic videos from natural language prompts. These models are trained on enormous corpora of internet video data, learning to generate complex scenes, realistic motions, and detailed object renderings. Given this capability, there is growing excitement—and active research investment—in the idea that such models could serve as general-purpose simulators of the physical world. For instance, researchers have proposed using text-guided video generation to train agents that can act, plan, and solve goals in real environments, treating generated videos as surrogates for real-world experience.

However, the paper identifies a fundamental gap: we have no systematic way to know whether these generated videos actually respect the laws of physics. A model might produce a visually compelling video of water pouring into a glass, but does the water level rise correctly? Does a rolling ball slow down from friction? Does a rigid object maintain its shape under force? These questions—which humans answer effortlessly through what cognitive scientists call intuitive physics, the understanding of object interactions developed through everyday experience without formal physics education—are entirely unaddressed by existing evaluation frameworks.

This gap is not merely academic. If T2V models are to be used as world simulators for robotics, planning, or scientific reasoning, their physical fidelity must be understood. Deploying a model that generates physically impossible dynamics (a brick floating, water flowing upward, a rigid spoon bending) in a downstream application that assumes physical realism could lead to catastrophic failures. The paper argues that before we can build systems that rely on video generators as physics engines, we first need to answer a foundational question: how much physical commonsense do these models actually possess?

Why Existing Evaluation Approaches Fall Short

The paper identifies several reasons why existing evaluation methods cannot answer this question:

1. Physical simulation is not ground truth—and is extremely hard to use as a reference. One might naively think that comparing generated videos against physics simulations would solve the evaluation problem. The paper explains why this is impractically difficult. First, accurately generating 3D geometries from single-view images or videos—a prerequisite for physical simulation—remains an unsolved problem in computer vision. Second, physics simulations require precise tuning of material parameters (stiffness, friction coefficients, viscosity) based on expert knowledge from graphics researchers. These parameters are unknown for arbitrary generated scenes. Third, converting simulation results back into realistic images requires accurate lighting and rendering parameters that are also unknown. Most fundamentally, the paper makes the philosophical point that physical simulations are not equivalent to ground truth—they are numerical approximations to differential equations proposed by researchers to model real-world dynamics. A simulation is itself a model, not an oracle.

Recent work has begun exploring how to tune simulation parameters from generated videos or integrate physics-based simulations into generation pipelines, but these approaches are circular: they depend on the physical plausibility of the generated videos themselves, which is precisely the property the paper aims to measure.

2. Standard video generation metrics don't capture physical commonsense. The traditional metric for evaluating video generation quality is Fréchet Video Distance (FVD), which measures distributional similarity between real and generated videos in feature space. FVD has several well-documented limitations that make it unsuitable for assessing physical commonsense: it requires reference videos (unavailable for novel scenes), it is biased toward superficial visual quality rather than dynamical correctness, and it fails to detect unrealistic motions. Similarly, CLIPScore measures semantic similarity between video frames and conditioning text in a shared embedding space, but semantic similarity says nothing about whether the depicted dynamics obey conservation of mass, Newton's laws, or material constitutive relationships.

3. Existing comprehensive benchmarks don't target physics. Benchmarks like VBench and EvalCrafter have recently emerged to evaluate video generation quality along multiple dimensions—motion smoothness, background consistency, object coherence, temporal flickering, and so forth. These represent important progress in moving beyond single-number metrics. However, the paper notes that they do not specifically address adherence to physical laws. A video can score perfectly on motion smoothness and temporal consistency while showing a wooden log deforming like rubber or water flowing upward. Physical commonsense is a conceptually distinct dimension that existing benchmarks do not operationalize.

4. No dataset exists for the specific purpose of evaluating physical commonsense in generated video. The paper identifies a gap in the evaluation infrastructure itself. To systematically measure physical commonsense, one needs a carefully constructed set of prompts that probe specific types of physical interactions, coupled with a reliable evaluation protocol. No such dataset existed prior to this work. The closest precedent in related domains is PIQA (Physical Interaction Question Answering), which evaluates physical commonsense reasoning in language models through multiple-choice questions, but this assesses reasoning about text, not the physical fidelity of generated visual content.

Conflicting Signals from Research and Practice

The paper is motivated by a tension between two observations. On one hand, T2V models clearly encode some physical regularities—they can generate videos of pouring, rolling, bouncing, and splashing that look broadly plausible. This has encouraged researchers to treat them as nascent world simulators. On the other hand, careful inspection reveals persistent failures: objects deform when they shouldn't, masses appear or disappear, motion violates momentum conservation, fluids flow in physically impossible ways. The paper's Figure 2 illustrates three such violations from real model outputs: the milk level not rising when milk is poured (conservation of mass), sand shifting without applied force (Newton's first law), and rigid wood deforming under no load (solid constitutive law violation).

These failures are not random noise—they reflect systematic limitations in what current models learn from video data. The paper hypothesizes that models may capture coarse appearance patterns (what pouring "looks like") without learning the underlying constraints (mass conservation, contact mechanics, material properties) that govern real physical dynamics. However, without a controlled benchmark that systematically varies interaction types and difficulty, it is impossible to characterize these limitations precisely or track progress as models improve.

How This Paper Positions Itself

The paper positions VIDEOPHY as filling three specific gaps:

First, it provides the dataset infrastructure for physical commonsense evaluation. Rather than relying on existing video data or simulation comparisons, the paper constructs a controlled set of 688 prompts through a multi-stage pipeline: LLM generation of candidate captions depicting interactions between different states of matter, human verification for quality and relevance, and expert annotation of perceived simulation complexity. The prompts are designed to probe specific physical interaction types (solid-solid, solid-fluid, fluid-fluid) with systematic variation in the materials involved and the complexity of the dynamics.

Second, it establishes a human evaluation protocol that directly measures both semantic and physical fidelity. The paper argues that since physical commonsense is inherently a human judgment (it is about intuitive physics, not exact equations), human evaluation is the appropriate gold standard. The protocol uses binary judgments (0/1) for semantic adherence and physical commonsense separately, which the paper argues is simpler and more reliable than asking annotators to list specific violations. Crucially, semantic adherence and physical commonsense are treated as independent metrics—a video can correctly depict the described action (high semantic adherence) while violating physics (low physical commonsense), and the paper explicitly finds that optimizing for one does not guarantee the other (e.g., Dream Machine achieves 61.9% semantic adherence but only 21.8% physical commonsense).

Third, it develops an automatic evaluator, VIDEOCON-PHYSICS, to make the benchmark practically usable at scale. Human evaluation, while reliable, is expensive and slow. Recognizing that model developers need rapid, scalable evaluation, the paper fine-tunes an existing video-language model (VIDEOCON) on the human annotations collected during benchmark construction. This auto-evaluator is shown to generalize to unseen prompts and unseen generative models, and its rankings correlate well with human rankings, making it a practical tool for the community.

The paper explicitly frames VIDEOPHY as complementary to existing benchmarks like VBench and EvalCrafter. While those benchmarks assess visual quality dimensions (smoothness, consistency, aesthetic appeal), VIDEOPHY assesses physical commonsense—a conceptually distinct axis that the paper argues is essential for any application where generated videos are used as surrogates for real-world dynamics. The finding that a model can score well on semantic adherence while failing physical commonsense underscores that physical commonsense is not automatically captured by existing metrics and requires dedicated evaluation.

The Choice of Intuitive Physics as the Evaluation Framework

A key conceptual decision in the paper is to evaluate intuitive physics rather than formal physics. The benchmark does not require videos to match precise differential equations or simulation outputs. Instead, it asks human annotators (who have studied high-school physics) to judge whether the depicted dynamics follow their intuitive understanding of how objects behave in the real world—the same understanding that lets a person predict that a ball will roll downhill or that poured water will fill a glass.

The paper justifies this choice on practical grounds: precise physical evaluation would require simulation comparisons and ground-truth 3D geometry that are unavailable for arbitrary generated scenes. But there is also a deeper rationale. The paper is ultimately interested in whether T2V models can serve as general-purpose world simulators for applications like robotics and planning, where the requirement is not exact numerical accuracy but broadly physically plausible dynamics. A model that generates approximately correct physics—where objects fall down, liquids fill containers, rigid objects maintain shape—is useful even if it doesn't solve Navier-Stokes equations precisely. Human judgment of intuitive physics directly operationalizes this "good enough for practical purposes" standard.

This choice also connects VIDEOPHY to a broader research tradition in cognitive science and AI that studies intuitive physics. Humans develop rich expectations about physical dynamics through experience, and these expectations can be probed through carefully designed stimuli. By adapting this paradigm to generated video evaluation, the paper bridges the gap between cognitive benchmarks and generative model assessment.

3. Technical Approach

3.1 Reader Orientation

The paper builds a benchmarking system that evaluates whether text-to-video (T2V) generative models produce physically plausible outputs. The core problem is that existing video evaluation metrics (FVD, CLIPScore, VBench) assess visual quality and semantic alignment but cannot tell us whether a generated video obeys physical laws — does the water level rise when liquid is poured? does a rigid object maintain its shape under force? The solution is a three-component system: a curated dataset of 688 prompts designed to probe specific physical interactions, a human evaluation protocol that separately measures semantic adherence and physical commonsense through binary judgments, and a fine-tuned automatic evaluator (VIDEOCON-PHYSICS) that learns to replicate these human judgments, enabling scalable evaluation of future models without expensive human annotation.

3.2 Big-Picture Architecture (Diagram in Words)

The system has three major components that operate sequentially — first, a dataset construction pipeline creates the evaluation prompts; second, a video generation and human evaluation pipeline produces and scores videos; third, an automatic evaluator is trained to replicate human judgments.

Component 1: Dataset Construction Pipeline. Takes as input the goal of covering diverse physical interactions and produces 688 human-verified captions with metadata. Major sub-components: (a) GPT-4 prompted to generate candidate captions for solid-solid, solid-fluid, and fluid-fluid interactions; (b) human verification to filter out low-quality or irrelevant captions; (c) expert graphics researchers annotating each caption with a difficulty label (easy/hard) based on perceived simulation complexity. Output: a set of 688 captions with category labels (interaction type) and difficulty labels.

Component 2: Video Generation and Human Evaluation. Takes the 688 captions and twelve T2V models (both open and closed) as input. For each (caption, model) pair, generates one or more videos. Human annotators (14 AMT workers with high-school physics background) then view each (caption, video) pair and provide binary judgments (0/1) for two independent metrics: semantic adherence (does the video depict the described action?) and physical commonsense (do the depicted dynamics follow intuitive physics?). The test set uses three annotators per video with majority voting; the training set uses one annotator per video. Output: a dataset of 36,500+ human annotations mapping (caption, model, video) to (SA, PC) scores.

Component 3: Automatic Evaluator (VIDEOCON-PHYSICS). Takes the human annotations from the training set as supervised data and fine-tunes VIDEOCON, a 7B-parameter video-language model originally trained for semantic adherence evaluation on real videos. The fine-tuned model takes a video and a multimodal query template as input and outputs Yes/No for semantic adherence or physical commonsense. Output: a model that can automatically score new generated videos, shown to generalize to unseen prompts and unseen generative models.

Information flows through these components in a fixed order: dataset construction produces prompts → video generation consumes prompts → human evaluation scores the generated videos → the scored videos train the automatic evaluator → the automatic evaluator scores future videos without human involvement.

3.3 Roadmap for the Deep Dive

• First, the dataset construction pipeline — how prompts are generated, verified, and annotated for difficulty — because the entire benchmark depends on having high-quality, physically meaningful prompts that systematically vary interaction types and complexity.
• Second, the human evaluation protocol — the metric definitions, annotation procedure, and inter-annotator agreement statistics — because this defines what "physical commonsense" means operationally and establishes the ground truth against which models and automatic evaluators are measured.
• Third, the video generation setup — which models are tested, how inference is performed, and how the data is split into train/test — because these choices determine which models can be evaluated and how the automatic evaluator is trained.
• Fourth, the automatic evaluator VIDEOCON-PHYSICS — the model architecture, fine-tuning procedure, and query templates — because this is the practical tool that enables scalable evaluation.
• Fifth, the design choices and their justifications — why binary judgments over Likert scales, why intuitive physics over formal physics, why VIDEOCON as the base model — because these choices reflect deliberate tradeoffs between reliability, cost, and scalability.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a benchmark construction and evaluation paper whose core idea is that physical commonsense in generated video can be measured through carefully constructed prompts probing specific physical interactions, combined with human judgments of intuitive physics, and that these judgments can be distilled into an automatic evaluator for scalable use.

Dataset Construction: LLM-Generated Captions (Stage 1)

The dataset construction begins with querying GPT-4 to generate candidate captions depicting real-world physical dynamics. The key design decision is to structure the query around states of matter — specifically, the interactions between solids and fluids, which the paper argues encompass the majority of real-world physical dynamics that would appear in video data.

The paper uses three interaction categories:

• Solid-solid interactions: involving two solid materials, with actions that require contact forces or friction forces (e.g., "A brick presses down on a metal can," "A snowball falls to the ground and splits apart").
• Solid-fluid interactions: involving one solid and one fluid material, again with contact-based dynamics (e.g., "An iron rod falls into the water," "A metal spoon stirs the honey in a cup").
• Fluid-fluid interactions: involving two fluid materials, with actions that require mixing, layering, or contact forces between fluids (e.g., "The wind creating ripples across the surface of the lake," "Oil falls into a transparent cup of water").

The paper requests GPT-4 to generate 500 candidate captions each for solid-solid and solid-fluid interactions, and 200 candidate captions for fluid-fluid interactions — a total of 1,200 initial candidates. The asymmetry (500 vs. 200) reflects the greater diversity of solid materials and their constitutive behaviors compared to fluids.

The GPT-4 prompts (reproduced in Appendix D, Figures 6-8 of the paper) contain detailed constraints designed to ensure the captions are appropriate for video generation evaluation. Specifically, the prompts instruct GPT-4 to:

• Focus on common everyday materials (avoiding rare or seldom-used materials) because the benchmark aims to test general physical understanding, not exotic material science.
• Exclude actions that don't involve clear physical interaction between materials — examples given include "celebrating," "arguing," or "laughing" — because these social actions would not probe physical dynamics.
• Avoid static scenes (e.g., "Lid covers pot to retain heat," "Stack of paper sits on the desk") because a static image cannot reveal whether the model understands dynamics over time.
• Avoid participle phrases that add descriptive flourishes (e.g., "sweetening it," "a creamy swirl," "fizzing energetically") because these complicate the caption without adding physical content.
• For solid-solid interactions, explicitly exclude actions requiring penetration forces (e.g., "A diamond scratching glass," "A key scratches the surface of a wooden table") because penetration involves fracture mechanics that are both extremely hard to simulate and often invisible in video.
• For solid-fluid interactions, exclude actions involving state changes (solid to fluid or vice versa, like melting or freezing) and invisible processes (dissolving, corroding, boiling) because these cannot be assessed visually in a short video.
• For fluid-fluid interactions, similarly exclude invisible processes (dissolving, chemical reactions) and state changes.

The requirement to format each caption as a JSON-like structure (with fields for 'action', the materials involved, and the 'caption' string) ensures structured metadata that enables fine-grained analysis later. Each generated caption is limited to 7-10 words, keeping the descriptions concise.

This LLM-based generation approach allows the paper to rapidly produce a large, diverse set of candidate captions covering a systematic taxonomy of physical interactions. The alternative — manually writing 688 captions — would be prohibitively expensive and might miss important categories. The explicit negative examples in the prompts ("Do Not Generate Such Captions") are a practical quality-control mechanism, leveraging GPT-4's instruction-following capability to pre-filter obviously unsuitable candidates before human verification.

Dataset Construction: Human Verification (Stage 2)

The 1,200 GPT-4-generated captions are then filtered through human verification performed by the authors themselves. This step addresses the fact that LLM-generated captions, even with careful prompting, may contain errors: captions that are unclear, excessively complex, incorrectly categorized, or that violate the stated constraints.

The verification applies three criteria:

1. Clarity and understandability: the caption must be unambiguous about what action is occurring and what materials are involved.
2. Appropriate complexity: the caption should not describe overly varied objects or excessively intricate dynamics that would be unreasonably difficult for current T2V models to even attempt.
3. Correct interaction category: the caption must accurately reflect the intended material states — for instance, if the caption claims to be a solid-fluid interaction, both a solid and a fluid must be mentioned.

After verification, the final dataset contains 688 captions distributed as: 289 captions for solid-solid interactions, 291 for solid-fluid interactions, and 108 for fluid-fluid interactions. The fluid-fluid category has fewer captions because there are inherently fewer distinct fluid-fluid interaction types in everyday experience compared to solid-solid or solid-fluid interactions.

The paper explicitly acknowledges that 688 captions is a moderate size, but argues that data quality is more important than quantity for evaluating foundation models. The paper draws parallels to other high-quality but small evaluation datasets used in the vision-language community: Winoground (400 examples), Visit-Bench (500 examples), LLaVA-Bench (90 examples), and Vibe-Eval (269 examples). These benchmarks are widely used precisely because their careful curation ensures that performance differences reflect genuine capability differences rather than noise.

The average caption length is 8.5 words, confirming that most captions are straightforward and do not rely on complex linguistic phrasing. This is important because it isolates physical commonsense evaluation from language understanding — a model with poor text comprehension might fail on long captions for reasons unrelated to physics. The dataset includes 138 unique actions grounded in the captions, indicating substantial diversity in the types of dynamics being probed.

Dataset Construction: Difficulty Annotation (Stage 3)

To enable fine-grained analysis of where models succeed and fail, each caption is annotated with a perceived difficulty label: easy (0) or hard (1). The annotation is performed by two experienced graphics researchers (senior Ph.D. students in physics-based simulation) who independently classify each caption based on their perception of how complex it would be to simulate the described objects and motions using state-of-the-art physics engines.

The difficulty is evaluated within each interaction category (solid-solid, solid-fluid, fluid-fluid), meaning an "easy" solid-solid caption and an "easy" fluid-fluid caption are not necessarily comparably difficult — the difficulty label reflects relative complexity within the category, not absolute difficulty across categories.

The paper identifies two primary factors that influence perceived simulation difficulty:

• Material model complexity: different materials require different constitutive models (mathematical descriptions of how the material responds to forces). For example, rigid bodies (which maintain their shape) are modeled with simple ordinary differential equations — they are "easy." Deformable bodies (which change shape under external forces) are governed by more complex partial differential equations (PDEs) involving strain and stress tensors — they are "hard." Elastic materials, plastic materials, fracturing materials, granular materials (sand), and non-Newtonian fluids each introduce additional model complexity. The paper lists examples of hard materials: deformable bodies, thin shells, metal with plasticity, fracture mechanics, cream, and sand.

• Numerical difficulty in solving the governing equations: even when the constitutive model is known, solving the equations numerically can be challenging. Higher-velocity motions introduce higher-order terms in the PDEs that require finer temporal resolution (smaller time steps) to resolve accurately. Faster-moving materials are therefore harder to simulate than slower-moving ones. Contact and collision handling between different materials adds further numerical complexity — resolving exactly when and where two objects touch, and computing the resulting forces, requires sophisticated algorithms (the paper references incremental potential contact and codimensional methods from the graphics literature).

The two annotators initially classify independently, then discuss disagreements (which occurred for less than 5% of instances) to reach unanimous judgment. The final distribution is 366 easy captions and 322 hard captions, providing roughly balanced coverage.

This difficulty annotation serves two purposes. First, it allows the paper to test whether T2V model performance correlates with simulation difficulty — if models are learning physical regularities from data, they might be expected to struggle more on physically more complex scenarios, and the paper indeed finds this pattern (Table 6). Second, it provides a bridge between the computer graphics community (which has formal understanding of simulation difficulty) and the generative modeling community (which needs evaluation metrics), grounding the benchmark in established physical modeling knowledge.

Human Evaluation: Metric Definitions

The paper defines two independent binary metrics for evaluating generated videos:

Semantic Adherence (SA) $\in \{0, 1\}$. This metric assesses whether the text caption is semantically grounded in the video frames — specifically, whether the actions, events, entities, and their relationships described in the caption are perceived to be correctly depicted. A video receives $\text{SA} = 1$ if the annotator judges that the caption is faithfully rendered in the video. The paper gives the example: for the caption "water pouring into the glass," $\text{SA} = 1$ if water is indeed flowing into a glass in the generated video.

Physical Commonsense (PC) $\in \{0, 1\}$. This metric assesses whether the depicted actions and object states follow intuitive physics — the physical laws that humans acquire through real-world experience without formal education. A video receives $\text{PC} = 1$ if the annotator judges that the generated movements and interactions align with how objects actually behave in the physical world. The paper gives the example: for the water pouring caption, $\text{PC} = 1$ requires that the water level in the glass increases as water flows in (conservation of mass), in addition to the water flowing downward (gravity).

The paper explicitly emphasizes that physical commonsense is entirely grounded in the video and independent of the conditioning caption. A video could show physically plausible dynamics that don't match the caption (SA = 0, PC = 1), or it could show caption-matching content with physically impossible dynamics (SA = 1, PC = 0). The joint metric of interest, reported as the primary benchmark score, is $\text{SA} = 1$ AND $\text{PC} = 1$ — the fraction of prompts for which the generated video is both faithful to the caption and physically plausible.

The choice of binary (0/1) feedback rather than Likert scales or dense annotations is justified on several grounds. The paper acknowledges that dense feedback provides more detailed information but argues it is harder to acquire and is often miscalibrated across annotators. Binary judgments are simpler, more consistent, and have precedent in text-to-image evaluation. The paper's experiments demonstrate that binary feedback "effectively highlights differences in the model's quality across various object interactions and levels of task complexity," validating this design choice.

For static scenes (where the T2V model generates essentially a still image rather than a video with motion), the paper instructs annotators to judge the physical plausibility of the static scene. For example, "a static scene of a folded brick does not follow physical commonsense" because a brick, as a rigid body, should not be folded. If static scenes are noisy (unwanted grainy or speckled patterns), annotators are instructed to treat this as poor physical commonsense.

The annotators are not asked to list specific physics law violations. The paper argues that doing so would make annotations more time-consuming and expensive, and would require advanced physics education. Instead, the task is designed to be performable by workers who have studied high-school physics and have everyday physical experience — for instance, they know that "water flows down from a tap" and "the shape of a wood log will not change while floating on water." This design choice makes annotation scalable but limits the granularity of failure mode analysis; the paper compensates with a separate qualitative failure mode analysis conducted by the authors (Section 5.2).

Human Evaluation: Annotation Procedure

The human evaluation uses 14 qualified Amazon Mechanical Turk (AMT) workers who have studied high-school physics. Qualification involves passing a test designed by the authors. The workers are provided with detailed task descriptions and have access to a shared Slack channel for clarifications.

In the annotation task, each worker is presented with a caption and the corresponding generated video, with no information about which generative model produced the video. The worker provides two independent binary scores: SA (0 or 1) and PC (0 or 1). The paper states that annotators "were instructed to treat semantic adherence and physical commonsense as independent metrics" and were shown "several solved examples by the authors before starting the main annotation task" to calibrate their judgments.

For the test set (used for benchmarking), each (caption, video) pair is annotated by three different annotators, and the final score is determined by majority voting. The paper reports inter-annotator agreement of 75% for semantic adherence and 70% for physical commonsense. The lower agreement for physical commonsense is attributed to greater subjectivity — different annotators may have different tolerance thresholds for what constitutes a physical commonsense violation in imperfect videos. The paper notes that "as generative models improve, human annotations will align more closely," implying that current disagreement partly reflects borderline cases where violations are subtle.

For the training set (used to train VIDEOCON-PHYSICS), each video is annotated by only one annotator to reduce costs. The paper explicitly states this is a financial constraint.

The workers were compensated at $18 per hour**, and the total annotation cost was approximately **$3,500, covering 24,500 annotations for benchmarking (test set: 344 prompts × 12 models × 1 video × 3 annotators × 2 metrics = 24,768 annotations, approximately) and 12,000 annotations for training (training set: 344 prompts × 9 models × 2 videos × 1 annotator × 2 metrics = 12,384 annotations, approximately). This cost is modest relative to many human evaluation efforts in AI, made possible by the binary, intuitive-physics-based annotation protocol.

Video Generation Setup: Models and Inference

The paper evaluates twelve T2V generative models, spanning both open-source and closed/proprietary systems, to ensure broad coverage of the current model landscape.

Open models (7 total):

• CogVideoX-5B and CogVideoX-2B: two variants of a recently released open-source model using a MMDiT-like architecture (Multi-Modal Diffusion Transformer). The 5B and 2B denote parameter counts.
• VideoCrafter2: a latent diffusion model that improves quality by training on high-quality image-text data.
• LaVIE: a cascaded video latent diffusion model with specialized training data.
• OpenSora: an open-source replication effort of the closed Sora model, using diffusion transformers.
• SVD-T2I2V (Stable Video Diffusion): a latent diffusion model. Since only the image-to-video (I2V) version is publicly available, the paper uses SD-XL-Base-1.0 to first generate a conditioning image from the caption, then feeds that image to SVD for video generation — hence "T2I2V" (text-to-image-to-video).
• ZeroScope: a latent diffusion model adapted from text-to-image models.

Closed models (5 total):

• Pika: a commercial model with undisclosed architecture.
• Dream Machine (Luma AI): a recently released commercial model.
• Lumiere-T2V and Lumiere-T2I2V (Google): a space-time diffusion model. T2V generates directly from text; T2I2V generates from an image produced by a text-to-image model (similar to SVD-T2I2V).
• Gen-2 (Runway): a closed latent video diffusion model.

The paper notes that several prominent models (Sora, Kling AI, Genmo) could not be included due to lack of API access.

Inference configurations for each model are listed in Appendix Table 9. Key details include:

• CogVideoX: resolution 480×720, 25 frames, guidance scale 7.5, 50 sampling steps, DDPM scheduler.
• ZeroScope: resolution 320×576, 32 frames, guidance scale 9, 50 steps, DPMSolverMultiStep scheduler.
• VideoCrafter2: resolution 320×512, 32 frames, guidance scale 12, 50 steps, DDIM scheduler.
• LaVIE: resolution 320×512, 32 frames, guidance scale 7.5, 50 steps, DDPM scheduler.
• OpenSora: resolution 240×426, 32 frames, guidance scale 7, 100 steps, IDDPM scheduler.
• SVD-T2I2V: resolution 1024×576, 25 frames, guidance scale (1, 3), 25 steps, EulerDiscrete scheduler.
• Lumiere-T2V: resolution 1024×1024, 80 frames, guidance scale 8, 256 steps.
• Lumiere-T2I2V: resolution 1024×1024, 80 frames, guidance scale 6, 256 steps.
• Gen-2: resolution 720×1280, 32 frames, guidance scale 8.5, 100 steps.
• Dream Machine: resolution 1280×720, 24 frames.
• Pika: resolution 640×1088, 72 frames, guidance scale 12.

For benchmarking, one video per test prompt per model is generated, producing 344 × 12 = 4,128 videos for the test set. For the training set, two videos per training prompt per model are generated for nine models (excluding the recently released CogVideoX and Dream Machine, which were not available when training data was collected), producing 344 × 9 × 2 = 6,192 videos. The total number of generated videos in the dataset is reported as 11,330, which implies some additional generation beyond these figures (possibly including the full set of prompts for all models, since the 688 prompts include both train and test).

Dataset Splits and Training Setup for VIDEOCON-PHYSICS

The 688 prompts are split equally into train and test sets (344 prompts each), with stratification to ensure similar distributions of interaction categories (solid-solid, solid-fluid, fluid-fluid) and complexity levels (easy, hard) across splits.

The training set for VIDEOCON-PHYSICS consists of paired (video, human annotation) data:

• Videos from nine models: VideoCrafter2, ZeroScope, LaVIE, OpenSora, SVD-T2I2V, Gen-2, Lumiere-T2V, Lumiere-T2I2V, and Pika. CogVideoX and Dream Machine are excluded because they were released after training data collection began.
• Two videos per training prompt per model (344 × 9 × 2 = 6,192 videos).
• One human annotation per video for each of SA and PC (12,384 annotations total, though the paper states 12,000 in Section 4, likely due to some failed generations or filtering).

The test set for benchmarking consists of:

• Videos from all twelve models (including CogVideoX and Dream Machine).
• One video per test prompt per model (344 × 12 = 4,128 videos).
• Three human annotations per video with majority voting for each of SA and PC.

This split enables two key evaluations: (1) whether VIDEOCON-PHYSICS generalizes to unseen prompts (by training on the training split prompts and evaluating on the test split prompts, using videos from models seen during training), and (2) whether VIDEOCON-PHYSICS generalizes to unseen generative models (by training on a subset of models and evaluating on held-out models, described in Section 6).

VIDEOCON-PHYSICS: Architecture and Multimodal Query Templates

VIDEOCON-PHYSICS is built by fine-tuning VIDEOCON, an existing open-source video-language model with 7 billion parameters, originally trained on real videos for robust semantic adherence evaluation. VIDEOCON is chosen because it was designed specifically for the task of judging whether a video matches a text description — closely related to the VIDEOPHY evaluation task — but it operates on real videos and cannot judge physical commonsense out of the box.

The core interface for VIDEOCON-PHYSICS is a multimodal query template $T_t(x)$ that takes a video $V$ and optionally a caption $C$, and produces a text prompt instructing the model what to judge. There are two task-specific templates:

For semantic adherence:

T_SA(V, C) = "[V] Does this video entail the description [T]?"

where $V$ is the video frames and $T$ (denoted $C$ in the formal notation) is the conditioning caption.

For physical commonsense:

T_PC(V) = "[V] Does this video follow physical laws?"

Note that the physical commonsense template does not include the caption — the paper explicitly states that "physical commonsense is independent of the conditioning caption," meaning the video frames alone should contain all information needed to judge physical plausibility.

The model processes the concatenation of video frames (sampled as 32 frames, resized to 224×224 pixels, created by dividing the video into 32 segments and sampling the middle frame from each segment) and the text template, and produces a token distribution over its vocabulary. The score for each task is computed as:

$s_\theta(T_t(x)) = \frac{p_\theta(\text{Yes} \mid T_t(x))}{p_\theta(\text{Yes} \mid T_t(x)) + p_\theta(\text{No} \mid T_t(x))}$

where $p_\theta(\text{Yes} \mid T_t(x))$ is the model's predicted probability of the token "Yes" conditioned on the multimodal template, and $p_\theta(\text{No} \mid T_t(x))$ is the predicted probability of "No."

What this computes: a normalized confidence score between 0 and 1 indicating how strongly the model judges the video to satisfy the criterion (semantic adherence or physical commonsense). The score is the ratio of the "Yes" probability to the sum of "Yes" and "No" probabilities, which normalizes out the effect of other tokens in the vocabulary. When $s_\theta$ is close to 1, the model is confident that the answer is Yes; when close to 0, confident that the answer is No.

Why this form: the paper notes that VIDEOCON, as a large video multimodal model, predicts a token distribution over the entire vocabulary, so $p_\theta(\text{Yes}) + p_\theta(\text{No}) \neq 1$ — there is probability mass on other tokens (e.g., punctuation, other words). Normalizing by the sum of Yes and No probabilities gives a score that is invariant to the total probability mass assigned to the binary answer tokens, focusing only on the relative confidence between the two options. The alternative — using $p_\theta(\text{Yes})$ directly — would conflate task certainty with model calibration (a model might assign low probability to both Yes and No because it's uncertain about the format, not because the video is borderline).

For evaluation, the paper binarizes these scores: a video is classified as SA = 1 if the SA score exceeds a threshold (implicitly 0.5, based on the ROC-AUC evaluation), and similarly for PC. The effectiveness of the auto-evaluator is primarily measured by ROC-AUC between the continuous scores and the binary human judgments — this measures how well the model's confidence separates positive from negative examples without requiring a specific threshold.

VIDEOCON-PHYSICS: Fine-Tuning Procedure

The fine-tuning uses Low-Rank Adaptation (LoRA) applied to all linear layers in the attention blocks, including QKVO (query, key, value, output projections), gate, up, and down projection matrices. LoRA is used because it is parameter-efficient — it trains only low-rank update matrices while keeping the pre-trained weights frozen, making fine-tuning feasible with limited compute. The hyperparameters are:

• LoRA rank $r = 32$ and scaling factor $\alpha = 32$: these control the capacity of the low-rank adaptation. With rank 32, each weight matrix $W \in \mathbb{R}^{d \times d}$ is augmented with $W + \frac{\alpha}{r} BA$ where $B \in \mathbb{R}^{d \times r}$ and $A \in \mathbb{R}^{r \times d}$, adding $2dr = 64d$ trainable parameters per matrix — a small fraction of the original $d^2$ parameters.
• Dropout = 0.05: applied to the LoRA layers for regularization.
• Training for 5 epochs with the Adam optimizer.
• Learning rate schedule: peak learning rate of $1 \times 10^{-4}$ with a linear warmup of 50 steps followed by linear decay.
• Batch size of 32 using 2 A6000 GPUs.

The model is fine-tuned to maximize the log-likelihood of the correct Yes/No token conditioned on the multimodal template, for both semantic adherence and physical commonsense tasks jointly (multi-task training). The paper notes that "finetuning separate classifiers for semantic adherence and physical commonsense did not provide any additional benefits over a single classifier trained in a multi-task manner" (Appendix J footnote), which suggests that the two tasks share relevant visual understanding capabilities — knowing what objects are present and how they move is useful for both judging whether the caption is depicted and whether the motion is physically plausible.

The video input uses 32 frames, created by dividing the video into 32 temporal segments and sampling the middle frame from each segment. This uniform sampling strategy ensures that the entire temporal extent of the video is covered regardless of the original video length, which varies across models (from 24 to 80 frames). The frames are resized to 224×224 pixels by the image processor.

Design Choices and Their Justifications

Binary judgments over Likert scales. The paper argues that binary (0/1) feedback is simpler, more reliable, and more consistent across annotators than multi-point rating scales. The concern is that Likert scales (e.g., 1-5 ratings for physical plausibility) introduce calibration issues — different annotators may use different parts of the scale differently, making inter-annotator agreement harder to achieve and harder to interpret. Binary judgments force a clear decision and have been widely used in related benchmarks (e.g., text-to-image alignment evaluation). The 70% inter-annotator agreement on physical commonsense, while lower than the 75% on semantic adherence, is reasonable for a subjective judgment and supports the viability of binary annotation.

Intuitive physics over formal physics. The paper deliberately frames the evaluation around intuitive physics — the understanding that humans develop through everyday experience — rather than requiring compliance with specific differential equations. This choice is pragmatic: formal physical evaluation would require simulation ground truth that is unavailable for arbitrary generated scenes. But it is also philosophically aligned with the paper's goal: to assess whether T2V models can serve as general-purpose simulators for practical applications (robotics, planning), where approximate physical plausibility (objects fall down, liquids fill containers, rigid objects maintain shape) is sufficient, and exact numerical accuracy is unnecessary. The high-school physics qualification for annotators ensures a baseline level of physical knowledge without requiring graduate-level physics expertise.

VIDEOCON as the base model for the auto-evaluator. The paper chooses VIDEOCON because it is (1) open-source (enabling fine-tuning and community use), (2) specifically designed for video-text alignment evaluation (matching the semantic adherence task), (3) reasonably sized at 7B parameters (making LoRA fine-tuning feasible on consumer GPUs), and (4) shown to be robust on real videos (providing a strong initialization). The paper considered but rejected using GPT-4Vision or Gemini-1.5-Pro-Vision as the auto-evaluator. GPT-4Vision's judgments were "close to random" (ROC-AUC of 53% for both SA and PC, against a random baseline of 50%), likely because GPT-4V processes only sampled frames (8 frames uniformly sampled from the video) and lacks native video understanding. Gemini-1.5-Pro-Vision, which can process entire videos, performed better on semantic adherence (73% ROC-AUC) but remained near-random on physical commonsense (58% ROC-AUC), highlighting that even state-of-the-art multimodal models lack the capability to judge physical commonsense out of the box. Fine-tuning a dedicated model on in-domain annotations is therefore necessary.

Difficulty annotation by graphics experts over crowd workers. The paper uses two experienced graphics researchers (senior Ph.D. students in physics-based simulation) to annotate perceived simulation difficulty rather than crowd workers. This choice reflects the specialized nature of the judgment: determining how complex it would be to simulate a given scenario requires deep knowledge of constitutive models, numerical methods, and contact mechanics that typical crowd workers lack. The less-than-5% disagreement rate between the two experts suggests high reliability of these judgments, though it also means the difficulty labels are based on a very small pool of annotators and may reflect specific biases of the graphics simulation community.

Exclusion of CogVideoX and Dream Machine from training data. The paper explicitly notes that these recently released models could not be included in the VIDEOCON-PHYSICS training set. This is actually turned into a strength: it allows the paper to evaluate whether the auto-evaluator generalizes to unseen model distributions (Section 6, Table 5). The experiment training VIDEOCON-PHYSICS on a restricted set of six models and evaluating on the held-out models (including CogVideoX and Dream Machine) shows good generalization (15-point improvement over the zero-shot VIDEOCON baseline for both SA and PC), establishing that the evaluator is not merely memorizing model-specific artifacts.

One video per prompt for benchmarking, two videos for training. The paper generates only one video per test prompt per model for benchmarking because human evaluation with three annotators per video is expensive. For the training set, generating two videos per prompt doubles the training data at relatively low computational cost compared to human annotation cost, providing more examples for VIDEOCON-PHYSICS to learn from. This asymmetry (one video for evaluation, two for training) is a practical budget optimization.

Prompt formatting with structured metadata. The GPT-4 prompts require captions to be output in a structured format with separate fields for 'action', the materials involved, and the 'caption' string. This design enables the fine-grained analysis in Section 5 (breakdowns by interaction category) and the diversity analysis in Appendix G (verb-noun frequency visualizations). Without this structured metadata, extracting interaction categories and material types would require post-hoc annotation or unreliable keyword matching.

No ground-truth simulation comparison. Perhaps the most consequential design choice is the explicit decision not to compare generated videos against physics simulations. The paper justifies this on multiple grounds: lack of methods to reconstruct 3D geometry from single-view generated video, the difficulty of tuning simulation parameters without expert knowledge, the unknown lighting and rendering parameters, and the philosophical point that simulations are themselves models, not ground truth. However, this choice also means that VIDEOPHY cannot distinguish between a video that is physically correct in its fine-grained dynamics and a video that merely "looks right" to a human observer — the two should correlate, but they are not equivalent. This is an inherent limitation of the intuitive physics evaluation framework, which the paper acknowledges implicitly by framing the benchmark as measuring "physical commonsense" (human judgment) rather than "physical accuracy" (simulation fidelity).

4. Key Insights and Innovations

Innovation 1: Physical Commonsense as an Independent, Measurably Distinct Axis of Video Generation Quality

The paper's most fundamental conceptual contribution is establishing that physical commonsense is a distinct dimension of video generation quality that is not captured by existing metrics — and, critically, that optimizing for standard metrics like semantic adherence does not guarantee physical plausibility. This is not an incremental addition to existing evaluation frameworks; it is a diagnostic reframing that identifies a blind spot in how the field assesses progress.

Before VIDEOPHY, the dominant evaluation paradigm for T2V models operated on two axes: visual quality (FVD, VBench's motion smoothness and temporal consistency scores) and semantic alignment (CLIPScore, text-video retrieval accuracy). These metrics implicitly assume that a video that looks good and matches its caption is a good video. The paper demolishes this assumption with a single, stark empirical finding: Dream Machine achieves 61.9% semantic adherence but only 21.8% physical commonsense (Table 3). In plain terms, this model correctly depicts the described action nearly two-thirds of the time, yet produces physically impossible dynamics in four out of five videos. A model could ace existing benchmarks while producing videos where water flows upward, rigid objects deform like rubber, and mass spontaneously appears or disappears.

This finding is significant beyond its specific numbers because it exposes a structural blind spot in the field's evaluation philosophy. Visual quality metrics assess whether video frames look like real video frames (distributional similarity in feature space). Semantic adherence metrics assess whether the depicted content matches the text. Neither asks whether the depicted dynamics could actually happen in the physical world. The paper demonstrates that these are answerably different questions — and that the answer to the third question is alarming.

The paper's resolution is not just to add another metric to the pile but to create an evaluation framework that disentangles physical fidelity from visual and semantic fidelity. By separately annotating semantic adherence and physical commonsense, and reporting the joint metric (SA = 1, PC = 1), the benchmark makes it possible to diagnose whether a model's failures stem from not understanding what the prompt asks for (low SA) or not understanding how objects behave (low PC) — two very different failure modes with different implications for model development. This diagnostic granularity is absent from prior benchmarks like VBench and EvalCrafter, which aggregate multiple quality dimensions into composite scores that obscure the specific nature of failures.

The conceptual move here parallels what happened in natural language processing when researchers realized that language models could be fluent but factually wrong — fluency and factuality are distinct axes requiring distinct evaluation. VIDEOPHY makes the analogous move for video generation: visual quality and physical commonsense are distinct axes, and we have been optimizing for the former while largely ignoring the latter.

Innovation 2: Difficulty-Conditioned Analysis as a Diagnostic for Understanding Model Limitations

The paper introduces difficulty annotation based on physics simulation complexity as a lens for analyzing where and why T2V models fail. This is not merely a stratification variable — it is a conceptual bridge between the computer graphics community's formal understanding of physical simulation and the generative modeling community's need to characterize model capabilities.

Prior video generation benchmarks typically treat all test prompts as equally difficult, reporting aggregate performance that averages over heterogeneous scenarios. VIDEOPHY's difficulty labels (easy/hard, annotated by graphics researchers based on material model complexity and numerical solution difficulty) enable a fundamentally more informative analysis: performance drops systematically with physical complexity (Table 6). For example, CogVideoX-5B's SA drops from 63.8% on easy captions to 62.5% on hard captions (a modest decline), while its PC drops from 55.3% to 50.3%. This pattern — where physical commonsense degrades more sharply with difficulty than semantic adherence — holds across most models and suggests that current T2V architectures learn surface-level appearance patterns (what an action "looks like") more readily than the underlying physical constraints that govern how materials actually behave under forces.

The significance of this finding extends beyond the specific numbers. It provides evidence for a specific hypothesis about what T2V models are learning from internet video data: they may be learning to generate videos that are visually reminiscent of training examples showing particular actions, without learning the causal structure — conservation laws, material properties, contact mechanics — that produces those appearances in the real world. A model that has only learned correlations between pixels and text descriptions of "pouring" will produce videos where liquid appears to move from one container to another, but will not consistently respect the constraint that total liquid volume is conserved. The difficulty-conditioned analysis exposes this pattern because harder prompts typically involve more complex material models (deformable bodies, granular materials, non-Newtonian fluids) where appearance-based generation is more likely to deviate from physically correct dynamics.

This conceptual contribution is methodologically important for the field: it demonstrates that not all evaluation prompts are equally informative, and that systematic variation in prompt properties (here, physical simulation complexity) can surface model limitations that aggregate metrics conceal. Future benchmark design in video generation — and potentially in other generative domains — should incorporate similar difficulty stratification grounded in domain expertise about what makes certain generations harder than others.

Innovation 3: The Viability of Intuitive Physics as an Evaluation Standard for Generated Video

The paper makes a non-obvious methodological argument that has implications for how the field evaluates physical fidelity: intuitive physics — human judgment of physical plausibility based on everyday experience — is a viable and appropriate evaluation standard for generated video, and may be preferable to formal simulation-based evaluation in the near term.

This position cuts against a natural impulse: if we want to know whether generated videos obey physics, shouldn't we compare them against physics simulations? The paper argues convincingly that this impulse is not just practically infeasible (due to the unsolved problems of 3D reconstruction, material parameter estimation, and inverse rendering from single-view generated video) but conceptually misguided, because physics simulations are themselves models — numerical approximations to differential equations that researchers have proposed to describe real-world dynamics. A simulation is not ground truth; it is one particular computational model of physical reality, with its own assumptions, approximations, and failure modes. Using simulation as an evaluation oracle would conflate "agrees with this particular simulator" with "is physically correct," potentially penalizing physically plausible dynamics that happen to deviate from the simulator's approximations.

The paper's alternative — human judgment of intuitive physics — is grounded in a different epistemology: physical commonsense is ultimately about whether dynamics match human expectations developed through a lifetime of real-world experience. This is the same standard we apply when we watch a movie with CGI effects and judge whether they "look real." The paper operationalizes this through annotators with high-school physics education, binary judgments, and explicit instructions to treat physical commonsense independently of semantic adherence.

The 70% inter-annotator agreement on physical commonsense (vs. 75% on semantic adherence) is informative here. It shows that the judgment is somewhat more subjective than semantic adherence — different people have different tolerance thresholds for subtle violations — but not so subjective as to be unreliable. The paper's argument that agreement will improve as models produce fewer borderline cases is plausible and aligns with experience in other annotation domains.

This methodological contribution matters because it provides a practical path forward for physical evaluation without requiring solved problems in 3D reconstruction and simulation. If the field had to wait for perfect simulation-based evaluation, physical commonsense assessment might remain infeasible for years. The intuitive physics standard makes evaluation possible now, while acknowledging its limitations (it cannot detect violations too subtle for human perception, and it conflates approximate plausibility with exact correctness). For the practical goal of determining whether T2V models can serve as approximate world simulators for robotics and planning — where getting the broad strokes of dynamics right matters more than exact numerical accuracy — this standard is arguably more appropriate than simulation fidelity anyway.

Innovation 4: Distillation of Human Physical Judgments into an Automatic Evaluator That Generalizes Across Models

The paper demonstrates that human judgments of physical commonsense in generated video can be distilled into an automatic evaluator that generalizes to unseen prompts and, more importantly, to videos from unseen generative models. This is a practical contribution with conceptual significance: it shows that physical commonsense evaluation, which might seem inherently subjective and resistant to automation, can be learned from a modest amount of annotated data (12,000 annotations) and deployed at scale.

The conceptual move here is treating physical commonsense evaluation as a supervised learning problem rather than a rule-based or simulation-based check. VIDEOCON-PHYSICS does not encode any explicit physics knowledge — no conservation laws, no material models, no contact mechanics. It learns to predict human judgments from (video, label) pairs, presumably by learning visual patterns that correlate with human assessments of physical plausibility. The fact that this works — achieving ROC-AUC of 82% for semantic adherence and 73% for physical commonsense, substantially outperforming zero-shot GPT-4Vision (53% on both tasks) and Gemini-1.5-Pro-Vision (73% SA, 58% PC) — is evidence that there are learnable visual signatures of physical implausibility that generalize across models and prompts.

The generalization to unseen generative models (Table 5: 79% SA and 72% PC when trained on six models and evaluated on held-out models) is particularly important. It suggests that VIDEOCON-PHYSICS is not merely memorizing model-specific artifacts (e.g., "OpenSora videos always look blurry, so blurry = bad physics") but learning something more fundamental about what physical plausibility looks like. This is a non-trivial finding: different T2V models have very different visual signatures (resolution, frame rate, color palette, artifact patterns), and a naive learned evaluator might easily overfit to these superficial features rather than learning to assess dynamics.

The automatic evaluator also enables a type of analysis that would be prohibitive with human evaluation: the paper uses VIDEOCON-PHYSICS to construct an automatic leaderboard (Appendix M) that tracks the human leaderboard's rankings. The correlation between automatic and human rankings (CogVideoX-5B > VideoCrafter2 > LaVIE > CogVideoX-2B > SVD-T2I2V > ZeroScope > OpenSora for open models) validates the evaluator's practical utility and suggests it can be used for rapid model development — developers can evaluate candidate models on VIDEOPHY in minutes rather than waiting days for human annotations and paying thousands of dollars.

This contribution shifts the bottleneck in physical commonsense evaluation from annotation cost to model training cost. Once VIDEOCON-PHYSICS is trained, evaluating a new model on VIDEOPHY requires only inference compute, which is cheap and fast. This is the same dynamic that made benchmarks like ImageNet transformative: the availability of an automatic evaluation protocol (top-1 accuracy against human labels) enabled rapid iteration that human-in-the-loop evaluation would have made impossible. VIDEOCON-PHYSICS aims to play the same role for physical commonsense evaluation in video generation.

5. Experimental Analysis

Evaluation Methodology

• Dataset. All experiments use the VIDEOPHY dataset, comprising 688 human-verified captions covering solid-solid (289), solid-fluid (291), and fluid-fluid (108) interactions, with each caption annotated for difficulty (366 easy, 322 hard) by two experienced graphics researchers. The dataset is split equally into train and test sets (344 prompts each), stratified to preserve the distribution of interaction categories and difficulty levels across splits. Videos are generated from these captions using twelve T2V models, producing 11,330 total videos.

• Base model(s). The evaluation tests twelve text-to-video generative models spanning both open-source and closed/proprietary systems: CogVideoX-5B, CogVideoX-2B, VideoCrafter2, LaVIE, OpenSora, SVD-T2I2V, ZeroScope (open); and Pika, Dream Machine, Lumiere-T2V, Lumiere-T2I2V, Gen-2 (closed). These models were chosen to represent diverse architectures (diffusion-based, diffusion-transformer, cascaded), training datasets, and inference strategies, covering the contemporary T2V landscape as of mid-2024. Notable omissions include Sora, Kling AI, and Genmo due to lack of API access.

• Metrics. The primary evaluation metrics are semantic adherence (SA) and physical commonsense (PC), each measured as a binary (0/1) human judgment. SA assesses whether the text caption is semantically grounded in the generated video frames — specifically, whether the described actions, events, entities, and their relationships are perceived to be correctly depicted. PC assesses whether the depicted actions and object states follow intuitive physics — the physical laws humans acquire through real-world experience, such as water flowing downward and rigid objects maintaining shape. The joint metric of interest is the fraction of prompts for which both SA = 1 and PC = 1, reported as a percentage. Inter-annotator agreement is 75% for SA and 70% for PC across three annotators per test video, with majority voting used for final scores. For the automatic evaluator VIDEOCON-PHYSICS, effectiveness is measured by ROC-AUC between the model's continuous confidence scores and binary human judgments.

• Baselines. For the human evaluation benchmarking, no separate baselines are compared — all twelve models are ranked against each other on the same captions using the same human evaluation protocol. For the automatic evaluator experiments, the following baselines are compared: Random (50% ROC-AUC by construction), GPT-4-Vision (zero-shot, using 8 uniformly sampled frames and the caption, instructed to output 0/1 for SA and PC), Gemini-1.5-Pro-Vision (zero-shot, processing the entire generated video natively, with identical instruction prompts), and VIDEOCON (the un-fine-tuned 7B video-language model from Bansal et al., 2023, evaluated zero-shot using the same multimodal query templates as VIDEOCON-PHYSICS).

• Generation budget / compute accounting. For the benchmark, one video per test prompt per model is generated, producing 344 × 12 = 4,128 videos for testing. For the training set, two videos per training prompt are generated for nine models (CogVideoX and Dream Machine excluded due to late release), producing approximately 6,192 videos for training. Inference configurations vary by model, with resolutions ranging from 240×426 (OpenSora) to 1280×720 (Dream Machine), frame counts from 24 to 80, and sampling steps from 25 to 256. No unified compute budget is imposed across models — each model is run with its standard inference settings, meaning the comparison reflects models as they would be deployed in practice, not a FLOPs-matched comparison. The human annotation cost totals approximately $3,500, covering 24,500 benchmark annotations (test set) and 12,000 training annotations, with workers compensated at$18 per hour.

• Cross-validation / statistical protocol. For statistical comparison between model pairs, the paper applies a paired t-test at 95% confidence interval, finding that the difference between CogVideoX-5B and all other video models is statistically significant (p < 0.0001). For the automatic evaluator, model performance is assessed by ROC-AUC on the test set, and generalization to unseen models is tested by training VIDEOCON-PHYSICS on a restricted set of six models (VideoCrafter2, ZeroScope, LaVIE, OpenSora, SVD-T2I2V, Gen-2) and evaluating on held-out models from the remaining testbed.

Main Quantitative Results

Human Evaluation: Overall Model Performance

Headline result: CogVideoX-5B achieves the best joint performance (SA = 1, PC = 1) at 39.6%, while all other models fall below 20% (Table 3). The second-best open model, VideoCrafter2, reaches only 19.0%, and the best closed model, Pika, reaches 19.7%. This establishes a substantial performance gap between the top model and all competitors, and more importantly, demonstrates that even the best current T2V model generates physically plausible, caption-faithful videos for fewer than two in five prompts.

The performance hierarchy for the joint metric reveals clear stratification. Among open models: CogVideoX-5B (39.6%) dominates, followed by VideoCrafter2 (19.0%), CogVideoX-2B (18.6%), LaVIE (15.7%), SVD-T2I2V (11.9%), ZeroScope (11.9%), and OpenSora (4.9%). Among closed models: Pika leads at 19.7%, followed by Dream Machine (13.6%), Lumiere-T2I2V (12.5%), Lumiere-T2V (9.0%), and Gen-2 (7.6%).

A critical finding is the decoupling between semantic adherence and physical commonsense. Several models achieve moderate to high SA while scoring poorly on PC, revealing that these are distinct capabilities that do not automatically co-occur. Dream Machine exemplifies this pattern most starkly: it achieves the second-highest SA score overall at 61.9% (behind only CogVideoX-5B's 63.3%), yet its PC score is only 21.8% — the lowest among all models. In other words, Dream Machine correctly depicts the described action in nearly two-thirds of videos, but in four out of five of those videos, the dynamics violate intuitive physics. CogVideoX-5B, by contrast, achieves both high SA (63.3%) and high PC (53%), suggesting that its training recipe — which includes detailed captions and filtering of videos with low motion and poor quality — enables learning of both semantic correspondence and physical regularities.

The gap between CogVideoX-5B and CogVideoX-2B (39.6% vs. 18.6% on the joint metric, 53% vs. 34.1% on PC) demonstrates a scaling effect: doubling the parameter count more than doubles the physical commonsense performance. This suggests that larger model capacity enables better capture of the underlying physical constraints of internet-scale video data, and that further scaling may continue to improve physical commonsense — though the 39.6% ceiling even for the largest model indicates that scaling alone is unlikely to solve the problem.

Human Evaluation: Fine-Grained Analysis by Interaction Category

Headline result: All models perform substantially worse on solid-solid interactions than on solid-fluid or fluid-fluid interactions (Table 3, "Solid-Solid" vs. "Solid-Fluid" vs. "Fluid-Fluid" columns). For the joint metric (SA = 1, PC = 1), CogVideoX-5B achieves 24.4% on solid-solid, 53.1% on solid-fluid, and 43.6% on fluid-fluid. This pattern — solid-solid being the hardest category — holds consistently across nearly all models.

The physical commonsense score alone tells an even sharper story. CogVideoX-5B's PC drops from 59.3% on solid-fluid and 61.8% on fluid-fluid to only 43.3% on solid-solid. Other models show even larger gaps: VideoCrafter2 achieves 41.8% PC on solid-fluid but only 23.8% on solid-solid; CogVideoX-2B achieves 34.9% on solid-fluid vs. 28.1% on solid-solid. The semantic adherence scores tell a similar story — models are better at depicting fluid-involving actions than solid-solid actions — but the physical commonsense degradation is consistently sharper.

This finding has significant implications: current T2V models appear to learn fluid dynamics more readily than solid mechanics from video data. The paper's difficulty annotation framework helps explain why — solid materials exhibit much more diverse constitutive behaviors (rigid, elastic, plastic, granular, fracturing) than common fluids, and the visual signatures of solid deformation are often subtle compared to the dramatic visible changes in fluid flow. A model that learns only surface-level appearance patterns may capture the broad strokes of fluid motion but fail to maintain rigid body constraints, leading to the characteristic failure mode of solids deforming when they shouldn't.

Notable exceptions to the general trend: Pika achieves its highest joint performance on fluid-fluid interactions (44.0%), significantly outperforming its solid-solid (13.6%) and solid-fluid (16.3%) scores. This model-level variation suggests that different training data distributions and architectures may specialize in different physical regimes — Pika may have been trained on proportionally more fluid-heavy video data, or its architecture may be better suited to capturing continuous deformations than discrete contact mechanics.

Human Evaluation: Fine-Grained Analysis by Complexity

Headline result: Performance degrades systematically from easy to hard captions across both semantic adherence and physical commonsense for all models (Table 6). For CogVideoX-5B, SA drops modestly from 63.8% (easy) to 62.5% (hard), while PC drops more noticeably from 55.3% (easy) to 50.3% (hard). The pattern of sharper PC degradation than SA degradation is consistent across models: CogVideoX-2B goes from 38.3% to 29.0% PC (a 9.3-point drop) while SA drops from 51.1% to 42.6% (an 8.5-point drop), and the trend intensifies for weaker models — OpenSora's PC barely changes (25.4% to 21.3%) because it's already near floor performance.

Dream Machine presents an interesting case: its SA drops from 65.2% (easy) to 57.8% (hard) — a 7.4-point decline — while its PC drops from 29.4% to 12.5%, a catastrophic 16.9-point decline on a metric that was already poor. This suggests that Dream Machine's physical commonsense failures are not uniformly distributed but are disproportionately concentrated on physically complex scenarios, exactly where realistic simulation would be hardest. On easy prompts, Dream Machine gets physics right about 29% of the time; on hard prompts, only 12.5% — barely above OpenSora's baseline.

The correlation between simulation difficulty and model performance supports the paper's hypothesis that T2V models are learning appearance-based generation rather than physics-based simulation. If models had internalized physical laws, they might be expected to struggle with mathematically harder dynamics (since the underlying PDEs are more complex), but the performance gap would likely be smaller because the model would be applying the same physical principles to both easy and hard cases. Instead, the large performance gap between easy and hard captions — especially for PC — is consistent with models that have learned coarse visual templates for common actions (easy cases) but lack the causal understanding to generalize to more complex dynamics. The hard captions involve materials like deformable bodies, granular matter, and non-Newtonian fluids where surface-level appearance varies substantially with specific conditions, making template-based generation unreliable.

Automatic Evaluator: VIDEOCON-PHYSICS Performance

Headline result: VIDEOCON-PHYSICS achieves ROC-AUC of 82% on semantic adherence and 73% on physical commonsense when evaluated against human judgments on test-set videos from models seen during training (Table 4). This substantially outperforms all baselines: zero-shot VIDEOCON (65% SA, 54% PC), GPT-4-Vision (53% on both, essentially random), and Gemini-1.5-Pro-Vision (73% SA, 58% PC).

The 17-point improvement over zero-shot VIDEOCON on SA (65% → 82%) and 19-point improvement on PC (54% → 73%) demonstrate that fine-tuning on in-domain generated video data and human annotations is essential — the distribution shift from real videos to generated videos, combined with the specific physical commonsense judgment task, is too large for zero-shot transfer. The 15-point gap between VIDEOCON-PHYSICS and Gemini-1.5-Pro-Vision on PC (73% vs. 58%) is particularly revealing. Gemini can process entire videos natively (unlike GPT-4V's 8-frame sampling) and achieves reasonable SA (73%), yet its PC performance barely exceeds random. This suggests that judging physical commonsense is not simply a harder version of video understanding — it requires capabilities that even state-of-the-art multimodal foundation models lack out of the box, and that can be acquired through task-specific fine-tuning on human annotations.

The consistent pattern across all evaluators — SA scores are higher than PC scores — mirrors the human annotation finding (70% vs. 75% inter-annotator agreement) and suggests that physical commonsense is genuinely a harder judgment task than semantic adherence, both for humans and for learned models.

Generalization to unseen models: When VIDEOCON-PHYSICS is trained on only six models (VideoCrafter2, ZeroScope, LaVIE, OpenSora, SVD-T2I2V, Gen-2) and evaluated on the held-out models (including CogVideoX and Dream Machine, which have noticeably different visual characteristics), it achieves ROC-AUC of 79% on SA and 72% on PC (Table 5). This represents a 15-point improvement over zero-shot VIDEOCON on both metrics (64% → 79% SA, 57% → 72% PC). The minimal degradation from the full-training setting (82% → 79% SA, 73% → 72% PC) indicates that VIDEOCON-PHYSICS is learning features of physical plausibility that generalize across model architectures and visual styles, rather than overfitting to model-specific artifacts.

This generalization result is crucial for practical deployment: it means new T2V models can be evaluated on VIDEOPHY using the existing VIDEOCON-PHYSICS checkpoint without requiring additional human annotations for each new model. The 72% PC ROC-AUC on unseen models, while somewhat lower than the 73% on seen models, is still sufficient to provide meaningful automatic evaluation — and is dramatically better than the 58% achieved by the much larger Gemini-1.5-Pro-Vision model.

Automatic leaderboard correlation: The paper constructs separate automatic and human leaderboards by ranking models according to their scores (Table 10, Appendix M). For open models, the automatic leaderboard ranking (CogVideoX-5B > VideoCrafter2 > LaVIE > CogVideoX-2B > SVD-T2I2V > ZeroScope > OpenSora) matches the human leaderboard ranking (CogVideoX-5B > VideoCrafter2 > CogVideoX-2B > LaVIE > SVD-T2I2V > ZeroScope > OpenSora) for all but one position: CogVideoX-2B and LaVIE swap positions between the two leaderboards. For closed models, Pika's automatic ranking is notably lower than its human ranking — the only significant discrepancy, attributed by the paper to limited training data for VIDEOCON-PHYSICS and suggesting that acquiring more annotations could improve alignment. Overall, the strong rank correlation validates VIDEOCON-PHYSICS as a practical tool for model development, enabling rapid screening of candidate models before committing to expensive human evaluation.

Ablation Studies and Robustness Checks

• Multi-task vs. separate classifiers for SA and PC: The paper reports (in a footnote, Appendix J) that "finetuning separate classifiers for semantic adherence and physical commonsense did not provide any additional benefits over a single classifier (VIDEOCON-PHYSICS) trained in a multi-task manner." This is a non-obvious finding — one might expect specialized classifiers to perform better on their respective tasks — but the result suggests that the visual features needed for judging whether a caption is depicted (object recognition, action recognition, spatial relationships) substantially overlap with those needed for judging physical plausibility (material property recognition, motion analysis, contact detection). Joint training likely enables positive transfer between the two tasks.

• Choice of base model for automatic evaluation: The paper's comparison of GPT-4-Vision (53% ROC-AUC, both tasks), Gemini-1.5-Pro-Vision (73% SA, 58% PC), and zero-shot VIDEOCON (65% SA, 54% PC) in Table 4 serves as an implicit ablation of model architecture and scale for zero-shot physical commonsense evaluation. The near-random performance of GPT-4V — despite its strong performance on many vision-language benchmarks — is a striking negative result that the paper attributes to its multi-image reasoning approach (8 sampled frames rather than full video) being insufficient for detecting physical violations that unfold over time. Gemini's ability to process entire videos improves SA substantially but barely helps PC, indicating that even native video understanding with a very large model is not enough — physical commonsense requires either task-specific training or fundamentally different architectural capabilities.

• Exclusion of CogVideoX and Dream Machine from training data: This is less an ablation and more a robustness check, but it serves the function of testing whether VIDEOCON-PHYSICS overfits to specific model distributions. The results in Table 5 demonstrate strong generalization, with only marginal degradation (3 points SA, 1 point PC) compared to full training. This also validates the decision not to include these models in training data — the paper turns a practical constraint (late model release) into a clean generalization experiment.

• Fine-tuning video models with VIDEOPHY data (negative result): The paper attempts to fine-tune Lumiere-T2I2V using training-set videos that achieved SA = 1 and PC = 1 (approximately 1,000 training pairs). Post-fine-tuning, SA drops from 46% to 36.5% while PC remains unchanged at ~25% (Table 12, Appendix P). This is a significant negative result — simply training on "good" videos does not improve the model's physical commonsense, and may actively harm semantic adherence. The paper attributes this to several factors: insufficient training samples, optimization difficulties from training on a mix of on-policy and off-policy videos (since the training data comes from multiple generative models with different visual characteristics), and vanilla fine-tuning being poorly suited for learning physical constraints. This negative result is valuable in its own right — it demonstrates that improving physical commonsense in T2V models is not simply a matter of filtering training data for physically plausible examples, and that post-training approaches may require more sophisticated algorithms (reinforcement learning, preference optimization, physics-informed losses) that were beyond the paper's scope.

• Correlation with video quality and motion (Appendix O): As a supplementary analysis, the paper computes Pearson correlation between SA/PC scores and two auxiliary metrics: video quality (LAION aesthetic classifier score) and video motion (RAFT optical flow magnitude). The key findings: aesthetics correlates positively with both SA (0.5) and PC (0.3), while motion correlates negatively with PC (-0.8) and weakly with SA (-0.1) (Table 11). The strong negative PC-motion correlation is particularly informative: models make more physical commonsense errors when generating videos with larger motions. This is consistent with the difficulty analysis (harder prompts involve higher velocities and more complex dynamics) and suggests that the physical commonsense failures are not random noise but are systematically related to the complexity of the depicted motion. The positive but moderate aesthetics-PC correlation (0.3) confirms that visual quality and physical commonsense are related but distinct — a video can look aesthetically pleasing while being physically nonsensical.

Critical Assessment

Claim 1: "Existing models severely lack the ability to generate videos adhering to physical commonsense." This claim is strongly supported. The empirical evidence is unambiguous: the best model achieves 39.6% on the joint metric, and all others fall below 20% (Table 3). Even when considering PC alone (ignoring semantic adherence), CogVideoX-5B reaches only 53% — meaning that even the best model produces physically implausible dynamics in nearly half of all generated videos. The difficulty breakdown (Table 6) and interaction category breakdown (Table 3) further confirm that these failures are systematic rather than concentrated in particularly challenging edge cases — even on "easy" captions, CogVideoX-5B's PC is only 55.3%. The qualitative analysis (Figures 15-26) provides concrete, interpretable examples of specific physics law violations (conservation of mass, Newton's laws, solid constitutive laws, fluid constitutive laws, non-physical penetration), ruling out the possibility that the low scores reflect annotation noise rather than genuine model failures.

However, the strength of this claim is contingent on accepting the human evaluation protocol as a valid measure of physical commonsense. The 70% inter-annotator agreement on PC, while reasonable, indicates non-trivial subjectivity — different annotators disagree on 30% of judgments. If the true rate of physical violations is near the decision boundary (videos with subtle rather than obvious violations), the binary judgment protocol may systematically misclassify borderline cases. The paper's argument that agreement will improve with better models is plausible but untested — it's also possible that as models improve, disagreements will concentrate on increasingly subtle judgments that remain hard to resolve. This does not undermine the qualitative conclusion (models are bad at physics) but adds uncertainty to the precise numerical scores.

Additionally, the paper evaluates only one video per prompt per model for benchmarking. Physical commonsense failures in generated video can be stochastic — some samples from a model may be physically correct while others from the same model on the same prompt are not. Using only one sample per (prompt, model) pair means the reported scores reflect a mix of systematic model limitations and sampling variance. Generating multiple videos per prompt and reporting pass@k metrics (as is done in MATH benchmark evaluation) would provide a more robust assessment of model capability versus stochastic failure.

Claim 2: "Physical commonsense is independent of semantic adherence — models can achieve high SA while scoring poorly on PC." This claim is well-supported by the Dream Machine result (61.9% SA, 21.8% PC — Table 3). The evidence is particularly compelling because it's an extreme case that cleanly demonstrates the decoupling. Other models show more correlated SA-PC performance (CogVideoX-5B: 63.3% SA, 53% PC; OpenSora: 18.0% SA, 23.5% PC), but the Dream Machine case proves that high SA does not guarantee high PC, establishing these as genuinely distinct evaluation dimensions.

A potential weakness: the paper treats SA and PC as independent metrics in the annotation protocol (annotators are instructed to judge them independently), but does not investigate whether annotators can truly separate these judgments in practice. When a video shows a physically impossible action (e.g., water flowing upward), does the annotator mark SA = 0 because the video doesn't match the caption (which says "water pouring," implying downward flow), or PC = 0 because the physics is wrong? The paper's framing suggests SA should be 1 in this case (the video depicts pouring, just with impossible dynamics), but in practice, a human annotator may struggle to disentangle "does this video show the action X?" from "does this video show the action X as it would actually occur?" The paper does not report any analysis of cases where SA = 1 but PC = 0 versus cases where both are 0, which would help characterize whether the metrics are genuinely independent in annotator behavior.

Claim 3: "The difficulty annotation based on simulation complexity is predictive of model performance." Supported by Table 6, which shows systematic degradation from easy to hard captions for all models, with the effect being more pronounced for PC than SA. The paper's claim is modest — it does not assert that simulation difficulty is the only factor determining model performance, only that it is predictive. The data supports this: there is a clear, consistent gap between easy and hard subsets for every model evaluated.

However, the difficulty annotation itself is based on only two annotators (both graphics Ph.D. students from the same lab), which raises questions about the generalizability and replicability of the difficulty labels. The paper reports less than 5% disagreement between the two annotators, which speaks to their calibration with each other but does not guarantee that different graphics researchers — or researchers with different simulation expertise — would produce the same easy/hard split. A more robust approach would use a larger pool of annotators and report inter-annotator agreement metrics (e.g., Cohen's kappa) with confidence intervals. Additionally, the difficulty labels are binary (easy/hard), which is a coarse discretization of what is presumably a continuous spectrum of simulation complexity. A continuous difficulty score might reveal more nuanced relationships between simulation complexity and model performance.

The causal interpretation — that harder-to-simulate dynamics cause worse model performance — is plausible but not the only explanation for the observed correlation. It is possible that "hard" captions correlate with other factors that degrade model performance: they might involve less common objects or actions (and thus be underrepresented in training data), they might involve more complex visual scenes with more objects, or they might describe higher-velocity motions that are inherently harder for any frame-by-frame generation process regardless of physical understanding. The paper does not control for these potential confounds, so the claim that simulation difficulty specifically drives the performance gap — as opposed to correlated factors like training data frequency or visual complexity — remains suggestive rather than demonstrated.

Claim 4: "VIDEOCON-PHYSICS can serve as a reliable automatic evaluator for physical commonsense in generated video." Supported with qualifications. The ROC-AUC of 73% on PC (Table 4) and 72% on unseen models (Table 5) is substantially above random and substantially above all baselines, demonstrating that the model has learned non-trivial physical commonsense evaluation capability. The automatic leaderboard (Table 10) also broadly tracks the human leaderboard, suggesting practical utility for model comparison.

The qualification concerns the absolute level of agreement. An ROC-AUC of 73% means VIDEOCON-PHYSICS correctly ranks a randomly chosen positive (PC = 1) video above a randomly chosen negative (PC = 0) video 73% of the time — a useful signal but far from perfect. In practice, the model will misclassify a substantial fraction of videos, potentially more than it correctly classifies when the decision threshold is set optimally. For model development, this may be acceptable (developers care about relative ranking, not absolute accuracy), but for applications requiring reliable pass/fail decisions on individual videos, 73% ROC-AUC may be insufficient.

Furthermore, VIDEOCON-PHYSICS's training data includes videos from only six to nine models, all from the same era (early-to-mid 2024). As new T2V models are released with new architectures, training distributions, and visual signatures, the domain shift may be larger than what the unseen-model generalization experiment (Table 5) captures — that experiment still tests on models from the same general time period. Continuous re-training on videos from new models (and corresponding new human annotations) may be necessary to maintain evaluator reliability, undermining the "cheap and scalable" value proposition.

Missing experiments that would strengthen the paper:

• Multi-sample evaluation: Generating multiple videos per prompt per model and reporting pass@k or expected correctness over samples would distinguish systematic model failures from stochastic failures, and would produce more reliable per-model scores given the relatively small prompt count (344 test prompts). The current single-video-per-prompt protocol means a model's score on a particular prompt can flip from 0 to 1 based on one lucky or unlucky generation.

• Temporal analysis of physical violations: The paper does not investigate when physical violations occur in generated videos — do they happen early (initial conditions are wrong), late (dynamics drift over time), or uniformly? This analysis would be diagnostic for understanding whether the root cause is poor initialization (the model generates a physically implausible starting state) versus poor temporal modeling (the model cannot maintain physical constraints over time).

• Training data ablation for models: The paper speculates that CogVideoX-5B's superior performance is due to its data curation pipeline (detailed captions, filtering low-motion/low-quality videos), but there is no experiment directly testing this hypothesis — for instance, no comparison of the same architecture trained with and without data filtering. This limits the paper's ability to offer actionable guidance to model developers beyond "use better data."

• Human evaluation of physical commonsense on real videos: A control experiment evaluating human annotators on real-world videos of the same actions described in the prompts would establish a human performance ceiling. If humans achieve, say, 95% PC on real videos, the gap between 95% and current model performance quantifies the headroom. If humans achieve only 80% (because some scenario descriptions are ambiguous), this would recalibrate expectations for what T2V models can achieve. Without this ceiling, 39.6% is hard to interpret in absolute terms.

• Breakdown of PC failures by violation type: The qualitative analysis identifies six common failure modes (conservation of mass, Newton's first law, Newton's second law, solid constitutive law, fluid constitutive law, non-physical penetration), but there is no quantitative breakdown of how frequently each failure type occurs. Such a breakdown would be enormously informative for model developers — knowing that 40% of failures are solid constitutive law violations versus 10% are fluid constitutive law violations would focus research efforts. The paper's choice not to ask annotators to classify violation types is understandable for cost reasons, but a follow-up study with expert annotators classifying failures on a subset of videos would add substantial value.

Genuine weaknesses in the experimental design:

• Single-dataset evaluation with moderate prompt count: The test set contains 344 prompts, and when stratified by interaction category and difficulty, some cells become quite small. For example, fluid-fluid hard captions likely contain fewer than 50 prompts (given 108 total fluid-fluid captions, split roughly 50-50 easy-hard, then split in half for train-test). Performance estimates in these small cells have high variance, and the paper does not report confidence intervals.

• No account of generation stochasticity: As noted above, a single video per prompt makes the scores noisy and potentially unfair to models with high variance in their generation quality. A model that sometimes produces perfect physics and sometimes catastrophic failures would alternate between scores of 1 and 0 on the same prompt, while a model that consistently produces mediocre-but-not-terrible physics might achieve a stable score — the single-sample protocol cannot distinguish these cases.

• The 688 prompts, while carefully curated, cover a limited slice of physical phenomena. Rotational dynamics, collisions involving more than two objects, articulated mechanisms, and many other everyday physical phenomena are absent or underrepresented. The benchmark provides excellent coverage of the categories it targets (solid-solid, solid-fluid, fluid-fluid interactions with contact forces) but cannot claim to comprehensively assess a model's physical commonsense across all domains.

• Annotation quality control relies on qualification tests and instructions but does not include attention checks or calibration items interspersed with real evaluation items. Without such checks, annotator fatigue or drift over the course of thousands of judgments could introduce systematic error that is not captured by inter-annotator agreement computed over the full dataset.

6. Limitations and Trade-offs

The Benchmark Covers Only a Narrow Slice of Physical Phenomena

The assumption or constraint. VIDEOPHY evaluates physical commonsense exclusively through captions depicting solid-solid, solid-fluid, and fluid-fluid interactions involving contact forces, generated by GPT-4 and filtered for everyday scenarios with concise descriptions (average 8.5 words). The paper explicitly acknowledges this scope limitation in Appendix A:

"In the future, it will be pertinent to expand the physical commonsense understanding to more branches of physics, including projective geometry."

The 688 prompts, while carefully curated, systematically exclude entire categories of physical phenomena: rotational dynamics (spinning tops, gyroscopes), articulated mechanisms (hinges, levers, pulleys), collisions involving more than two objects, gravitational orbital mechanics, electromagnetic effects, thermodynamic processes (expansion/contraction with temperature), and fluid phenomena beyond simple contact interactions (turbulence, vortex shedding, capillary action). The paper also explicitly excludes penetration-based actions (e.g., scratching, cutting, puncturing), stating in its GPT-4 prompts: "Do not focus on the actions that require penetration forces." This means fracture mechanics, a central topic in solid mechanics, is entirely absent from the benchmark.

The consequence. A model that achieves 100% on VIDEOPHY could still be catastrophically wrong on large classes of physical phenomena that the benchmark does not test. The headline claim—that current models "severely lack the ability to generate videos adhering to physical commonsense"—is proven for the specific interaction types tested, but the benchmark cannot reveal whether models are equally poor on untested physical domains, or whether different physical domains exhibit different difficulty patterns. A practitioner deploying a T2V model for a physics-sensitive application cannot infer from VIDEOPHY performance alone whether the model will succeed or fail on phenomena outside the benchmark's coverage—the evaluation provides no signal for rotational dynamics, fracture, thermodynamics, or electromagnetism. Furthermore, the solid-solid/f solid-fluid/fluid-fluid taxonomy, while well-motivated, is not exhaustive of contact mechanics: granular flows (sand, powder), viscoelastic materials, and phase transitions (melting, freezing, boiling) are underrepresented or absent, yet these are common in real-world physical scenarios.

What evidence exists in the paper. The diversity analysis in Appendix G (Figures 12-14) and the main text's verb-noun frequency diagram (Figure 3) reveal that the 138 unique actions are concentrated in a relatively small set of verb categories (pouring, stirring, splashing, rolling, floating, etc.). This confirms that the benchmark provides deep coverage of certain interaction types but limited breadth across the full physical phenomenon space. The exclusion of penetration forces is documented in the GPT-4 prompts (Appendix D, Figures 6-8) and is not revisited as a limitation in the main evaluation. The paper does not measure or discuss coverage of physical law types beyond the qualitative failure mode analysis (Section 5.2), which identifies six common violation categories but does not quantify how many prompts probe each violation type.

Mitigation status. Not addressed beyond the brief acknowledgment in Appendix A. The paper does not provide a roadmap for expanding coverage, nor does it characterize the gap between what VIDEOPHY tests and the full space of physical commonsense. The focus on quality over quantity is well-justified for establishing initial feasibility, but the limited scope means VIDEOPHY should be understood as evaluating a specific subset of physical commonsense rather than physical commonsense in general. The paper's title and framing ("Evaluating Physical Commonsense for Video Generation") implies broader coverage than the benchmark actually provides.

Difficulty Estimation Requires Prohibitively Expensive Expert Annotation and Does Not Scale

The assumption or constraint. The paper introduces difficulty labels (easy/hard) for each caption, annotated by "two experienced graphics researchers (senior Ph.D. students in physics-based simulation)" who independently classify each caption based on perceived complexity in rendering objects or synthesizing motions using state-of-the-art physics engines. The criteria involve material model complexity (e.g., rigid body vs. deformable body PDEs) and numerical difficulty in solving governing equations (e.g., higher velocities requiring finer temporal resolution). This annotation protocol is fundamentally not scalable: it requires rare expertise in physics-based simulation, and even with expert annotators, the process demands detailed discussion to resolve disagreements (which occurred for less than 5% of instances).

The consequence. The difficulty-conditioned analysis in Section 5—which reveals that models degrade on harder captions (Table 6) and that solid-solid interactions are hardest (Table 3)—is a key diagnostic contribution of the paper. However, this analysis cannot be replicated or extended without equivalent expert annotation, which most research groups evaluating T2V models do not have access to. A benchmark maintainer seeking to expand VIDEOPHY with new prompts (e.g., covering rotational dynamics or fracture) would need to hire graphics researchers to annotate difficulty for those new prompts—a costly and slow process. More critically, the difficulty labels are based on the annotators' perception of simulation complexity, not on any objective measurement (e.g., actual simulation failure rates, computational cost of running simulations, or quantitative metrics of PDE stiffness). The less-than-5% disagreement rate between two annotators from the same lab does not guarantee that other graphics researchers—or researchers with different simulation expertise—would produce identical labels, and the paper provides no measure of external validity for the difficulty annotations. A group specializing in fluid simulation might classify fluid-fluid captions differently than generalists, and a group working primarily with rigid body simulation might have different thresholds for solid-solid difficulty.

What evidence exists in the paper. The difficulty annotation process is described in Section 2, Stage 3, and the results in Table 6 show that easy/hard labels do stratify model performance—harder prompts yield lower SA and PC scores. However, the paper provides no inter-annotator agreement metric beyond the less-than-5% disagreement statement, no confidence intervals on difficulty labels, and no validation of the difficulty annotations against any external criterion (e.g., correlation with actual simulation computation time, or with independent difficulty judgments from a different group of graphics experts). The binary (easy/hard) discretization is acknowledged as coarse—within each bin, there is presumably substantial variation in actual simulation difficulty, but the paper does not explore whether a continuous difficulty score would be more predictive of model performance.

Mitigation status. The paper does not attempt to automate or scale difficulty annotation. The difficulty labels are used as a static annotation layer on the existing 688 prompts, and no method is proposed for extending difficulty labeling to new prompts. This limits VIDEOPHY's growth potential—the benchmark can expand in prompt count but would require re-incurring the expert annotation cost to maintain the difficulty-conditioned analysis capability that is one of the paper's key innovations. The paper does not discuss this limitation explicitly.

Human Evaluation Protocol Introduces Subjectivity That Limits Score Interpretability and Cross-Study Comparability

The assumption or constraint. The paper uses binary (0/1) human judgments of semantic adherence and physical commonsense, performed by 14 AMT workers with high-school physics education, with majority voting over three annotators per test video. The paper explicitly reports inter-annotator agreement of 75% for SA and 70% for PC (Section 4), acknowledging that "human annotators find the task of judging physical commonsense more subjective than semantic adherence." The paper attributes annotation variation to "differing tolerance for commonsense violations in imperfect videos" and notes that "as generative models improve, human annotations will align more closely."

The consequence. At 70% agreement, approximately 30% of physical commonsense judgments are not consensus decisions but reflect annotator-specific thresholds for what constitutes a violation. For videos near the boundary between "physically plausible enough" and "physically wrong," the binary judgment protocol forces a decision that may differ across annotators, across annotation batches, or across studies using different annotator pools. This introduces several practical problems for benchmarking. First, absolute scores on VIDEOPHY are not directly comparable across studies that use different annotator pools—a model scoring 39.6% with the paper's annotators might score 35% or 45% with a different set of equally qualified annotators who have slightly different tolerance thresholds. Second, small performance differences between models may be dominated by annotation noise rather than genuine capability differences. The paper's paired t-test finding that CogVideoX-5B significantly outperforms other models (p < 0.0001) is robust because the gap is large (39.6% vs. <20%), but as models improve and gaps narrow, distinguishing a genuinely better model from annotation variance will become increasingly difficult. Third, the binary protocol discards information about violation severity—a video with a subtle violation (a rigid object deforms very slightly) and a video with a catastrophic violation (a rigid object deforms like liquid) both receive PC = 0, even though they represent qualitatively different failure modes with different implications for downstream applications.

What evidence exists in the paper. The 70% inter-annotator agreement (Section 4) is the key evidence. The paper also reports that annotator variation arises from "differing tolerance for commonsense violations," which implies that the boundary between PC = 0 and PC = 1 is not a natural category boundary but a zone of ambiguity. The paper does not analyze whether disagreements are concentrated on particular types of prompts, particular difficulty levels, or particular models, which would help characterize the sources of subjectivity. There is no calibration study comparing the AMT annotators' judgments to those of domain experts (e.g., the graphics researchers who annotated difficulty, or physics educators), which would help establish whether the high-school physics qualification produces systematically different judgments than more expert evaluation.

Mitigation status. The paper does not attempt to reduce annotation subjectivity beyond using majority voting and providing solved examples for calibration. The binary protocol is defended as "simpler and more reliable" than dense feedback (Section 3.1), but the 70% agreement rate—while reasonable for a subjective task—indicates that the simplicity of binary judgment does not eliminate subjectivity; it merely forces it into a binary decision. The paper's expectation that agreement will improve as models get better is plausible (fewer borderline cases), but this means VIDEOPHY's reliability as a benchmark is contingent on models being sufficiently bad—a paradoxical property for an evaluation framework. The paper does not propose alternative annotation protocols (e.g., Likert scales with calibrated anchors, pairwise comparison, or expert consensus panels) that might reduce subjectivity or provide richer signal about violation severity.

The Automatic Evaluator (VIDEOCON-PHYSICS) Has Moderate Absolute Accuracy and Faces an Unbounded Domain Shift Problem

The assumption or constraint. VIDEOCON-PHYSICS achieves ROC-AUC of 82% on SA and 73% on PC when evaluated on test-set videos from models seen during training (Table 4), and 79% on SA and 72% on PC when evaluated on videos from unseen models (Table 5). The paper frames these results as demonstrating that VIDEOCON-PHYSICS "can judge semantic adherence and physical commonsense as new T2V generative models are released" (Section 6). This framing assumes that future T2V models will produce videos within the distribution that VIDEOCON-PHYSICS can reliably evaluate.

The consequence. An ROC-AUC of 73% on PC means the model correctly ranks a randomly chosen physically plausible video above a randomly chosen physically implausible video 73% of the time—useful for relative ranking but insufficient for reliable per-video classification. In practice, depending on the decision threshold and class balance, VIDEOCON-PHYSICS will misclassify a substantial fraction of videos. For model development workflows that use the auto-evaluator to filter training data or to flag problematic generations (as the paper proposes in Appendix N), these misclassifications propagate: physically implausible videos labeled as plausible will contaminate filtered datasets, and plausible videos labeled as implausible will be unnecessarily discarded. More critically, the unseen-model generalization experiment (Table 5) tests on models from the same era (early-to-mid 2024) as the training models. As T2V technology advances—with new architectures (e.g., diffusion-transformer hybrids, autoregressive video models), new training paradigms (e.g., physics-informed losses, reinforcement learning from human feedback), and new visual characteristics (higher resolution, longer duration, more complex scenes)—the distribution shift between training data and evaluation targets will grow. VIDEOCON-PHYSICS's performance on future models is unbounded in the worst case: a model that generates videos with novel artifact patterns or motion characteristics unseen in the training set could cause the auto-evaluator's accuracy to degrade toward random, and the paper provides no mechanism for detecting when this degradation occurs.

What evidence exists in the paper. Table 5 provides the key evidence: there is a small but measurable drop from full-training to unseen-model evaluation (82% → 79% SA, 73% → 72% PC), indicating that domain shift is already present even for models released shortly after the training data was collected. The paper also reports that Pika achieves a relatively low score on the automatic leaderboard compared to its human ranking (Appendix M), noting this as "a limitation that can be improved by acquiring more data for VIDEOCON-PHYSICS." This discrepancy between automatic and human rankings for one model is the paper's own evidence that VIDEOCON-PHYSICS is not perfectly reliable even within the current model set. The paper does not report performance stratified by model—we cannot see whether the auto-evaluator is consistently good across all training models or whether it performs well on some models and poorly on others.

Mitigation status. The paper partially addresses this by demonstrating generalization to unseen models (Table 5) and by constructing the automatic leaderboard (Table 10) as a validation of practical utility. However, the paper does not propose a mechanism for continual adaptation—there is no plan for re-training VIDEOCON-PHYSICS on videos from new models, no method for detecting when the evaluator's accuracy degrades, and no analysis of how much new training data would be needed to maintain performance as the model landscape evolves. The paper's vision of VIDEOCON-PHYSICS enabling "cheap and scalable evaluation" (Section 3.3) assumes that the one-time training cost produces an evaluator with enduring reliability, but the evidence only supports reliability for models released in the same time window as the training data. The human annotation cost that VIDEOCON-PHYSICS is meant to avoid would need to be periodically re-incurred to keep the evaluator calibrated to new model distributions, partially undermining its cost-saving value proposition.

The Benchmark Cannot Distinguish Between Fundamental Physical Understanding and Surface-Level Appearance Matching

The assumption or constraint. VIDEOPHY evaluates physical commonsense through human judgment of intuitive physics—annotators assess whether "generated movements and interactions align with intuitive physics that humans acquire with their experience in the real-world" (Section 3.1). The paper explicitly rejects simulation-based evaluation, arguing that "physical simulations are not equivalent to ground truth" and that simulation-based comparison is infeasible due to unsolved problems in 3D reconstruction, material parameter estimation, and inverse rendering (Section 1). This framing means VIDEOPHY measures whether videos look right to humans, not whether they are physically correct in a mechanistic sense.

The consequence. A model could achieve high VIDEOPHY scores by learning to generate videos that are visually convincing to human observers—capturing the broad appearance of correct dynamics—without internalizing any causal physical understanding. A video of water pouring might show the water level roughly increasing without precisely conserving volume; a video of a ball bouncing might show approximately correct trajectories without solving Newton's equations. Humans, especially with the brief viewing times and binary judgment protocol used in the evaluation, may not detect these subtle violations. Conversely, a video that is physically precise but visually atypical (e.g., correct fluid dynamics rendered in an unusual style, or correct rigid body motion with atypical textures) might be judged as physically implausible because it doesn't match human expectations of what that scenario "should look like." The intuitive physics standard thus conflates physical correctness with perceptual typicality—the benchmark measures whether generated videos align with human visual expectations of physical scenarios, which is correlated with but not identical to whether the underlying dynamics obey physical laws. For applications where approximate visual plausibility is sufficient (e.g., entertainment, artistic content creation), this conflation is acceptable. For applications where causal physical understanding is required (e.g., using generated video as training data for robotic control, or as a forward model for planning), the benchmark provides no guarantee—a model that scores 100% on VIDEOPHY might still fail to produce dynamics that would be physically correct enough to transfer to the real world, because the evaluation only verifies that the video passes a human plausibility check, not that it obeys conservation laws, material constitutive relationships, or contact mechanics.

What evidence exists in the paper. The paper does not directly measure or analyze the gap between intuitive physics judgments and objective physical correctness because the latter is explicitly out of scope. However, the qualitative analysis (Section 5.2) provides indirect evidence: the identified failure modes (conservation of mass violation, Newton's law violations, constitutive law violations) are detected through human visual inspection, meaning they represent violations that are obvious enough for humans to notice. Subtler violations—a ball that bounces with a slightly wrong coefficient of restitution, a fluid that flows with slightly incorrect viscosity, a rigid body that stress-deforms at a level below human perceptual threshold—would not be detected by the current evaluation protocol even though they represent genuine physical errors. The paper's Table 11, showing a -0.8 correlation between video motion magnitude and physical commonsense scores, suggests that violations are most detectable in high-motion scenarios—physical errors in low-motion scenarios may systematically escape detection.

Mitigation status. Not addressed. The paper frames the intuitive physics approach as a deliberate methodological choice justified by the infeasibility of simulation-based evaluation and the philosophical argument that simulations are themselves models, not ground truth (Section 1). However, the paper does not discuss the consequent limitation—that the benchmark is measuring perceptual plausibility rather than physical accuracy—nor does it characterize the expected gap between these two concepts. There is no experiment comparing human physical commonsense judgments on generated videos against any objective physical correctness measure (even a partial one, such as checking whether the center of mass of a rigid object follows a parabolic trajectory under gravity) that would help quantify how much physical error is required before humans notice. For practitioners who need causal physical understanding rather than approximate visual plausibility, the paper provides no guidance on how to interpret VIDEOPHY scores or what level of score is sufficient for their use case.

Single-Video-Per-Prompt Evaluation Protocol Produces Noisy Scores and Cannot Characterize Model Reliability

The assumption or constraint. For benchmarking, the paper generates one video per test prompt per model (Section 4: "we generate one video per test prompt for each T2V generative model in our testbed"). The joint metric (SA = 1, PC = 1) is then computed as the fraction of the 344 test prompts for which that single generated video satisfies both criteria. This protocol implicitly treats a model's performance as deterministic—the single generated video is taken as representative of what the model would produce for that prompt.

The consequence. T2V generation is stochastic due to random noise initialization in diffusion models and random sampling in autoregressive models. A model might generate a physically plausible video for a given prompt 70% of the time and a physically implausible video 30% of the time, but with only one sample, the prompt is scored as either 0 or 1 based on that single draw. This introduces substantial binomial variance into the per-model scores: with 344 test prompts and an underlying success probability p (the model's true physical commonsense rate for a given prompt averaged over generation stochasticity), the observed score has a standard error of approximately sqrt(p(1-p)/344). For a model with true p = 0.4 (similar to CogVideoX-5B), the standard error is approximately 2.6 percentage points due to prompt sampling alone, and this is compounded by generation stochasticity. Two models whose true physical commonsense rates differ by less than ~5-7 percentage points may not be reliably distinguished by the current protocol.

More importantly, the single-sample protocol cannot characterize model reliability. A model that consistently produces mediocre but passable physics on every prompt and a model that alternates between perfect physics and catastrophic failures could achieve the same aggregate score, but they have fundamentally different reliability profiles for downstream applications. The first model is predictable—you know each generation will be roughly acceptable. The second model is unpredictable—some generations are excellent and some are useless, and you don't know which until you inspect. The current protocol cannot distinguish these cases, yet practitioners deploying T2V models need to know not just the average quality but the variance and worst-case behavior.

What evidence exists in the paper. The paper generates two videos per prompt for the training set (used to train VIDEOCON-PHYSICS) but only one video per prompt for the test set (used for benchmarking). This asymmetry is acknowledged as a practical cost constraint (human annotation with three annotators per video is expensive), but the implications for score reliability are not discussed. The paper does not report any measure of generation variance—no standard deviation of scores across multiple generations for the same prompt, no pass@k analysis (what fraction of prompts have at least one physically correct generation among k attempts), no analysis of how model rankings would change if evaluated on multiple samples. The statistical significance testing uses a paired t-test (Section 5.1: "the difference between CogVideoX-5B and other video models is statistically significant (p<0.0001)"), but this only accounts for variation across prompts, not variation across generations of the same prompt, and the very small p-values largely reflect the large gap between CogVideoX-5B and all other models rather than high statistical power for close comparisons.

Mitigation status. Not addressed. The paper does not discuss generation stochasticity as a confounding factor, does not propose multi-sample evaluation protocols for future benchmark iterations, and does not report generation variance statistics even for a subset of prompts where collecting multiple generations would be computationally cheap (generation is much cheaper than human annotation). A simple and affordable mitigation—generate multiple videos per prompt for a small subset of prompts, compute per-prompt pass rates, and report the variance—would have substantially strengthened the reliability characterization without significant additional human annotation cost, since annotating multiple videos for the same prompt from the same model is annotation work that can be parallelized.

7. Implications and Future Directions

How This Work Changes the Landscape

VIDEOPHY does not introduce a new model, training recipe, or architecture. It introduces something arguably more consequential: a lens for seeing a problem the field has been systematically blind to. Before this paper, the dominant evaluation framework for text-to-video generation operated on two axes — visual quality (FVD, VBench's motion smoothness and temporal consistency scores) and semantic alignment (CLIPScore, caption-following accuracy). These metrics implicitly assume that a video which looks good and matches its caption is a good video. VIDEOPHY demolishes this assumption with a single, stark finding: Dream Machine achieves 61.9% semantic adherence but only 21.8% physical commonsense (Table 3). The model correctly depicts the described action nearly two-thirds of the time yet produces physically impossible dynamics in four out of five videos. A model can ace existing benchmarks while producing videos where water flows upward, rigid objects deform like rubber, and mass spontaneously appears or disappears.

This is not an incremental refinement. It is a diagnostic reframing of what "video generation quality" means. The paper establishes physical commonsense as a distinct, independently measurable axis of evaluation that is not captured by — and not correlated with — existing metrics. The conceptual move parallels what happened in NLP when the community realized that language models could be fluent but factually wrong: fluency and factuality are distinct axes requiring distinct evaluation. VIDEOPHY makes the analogous move for video generation: visual quality and physical commonsense are distinct axes, and the field has been optimizing exclusively for the former.

Several specific shifts follow from this reframing:

First, the burden of proof shifts for claims about "world simulators." When Sora's technical report and subsequent research claimed that video generators could serve as general-purpose simulators of the physical world, the field lacked any systematic way to test this claim. VIDEOPHY provides that test — and the results are sobering. The best current model succeeds on only 39.6% of prompts when both semantic adherence and physical commonsense are required. This does not mean video generators cannot eventually become world simulators, but it means the field can no longer assert this capability without evidence. VIDEOPHY operationalizes the question and establishes a baseline from which progress can be measured.

Second, the paper reconciles a tension between anecdotal evidence and systematic evaluation. Anyone who has used T2V models extensively has noticed physical failures — objects deforming, liquids behaving strangely, motion violating basic mechanics. But these failures were previously dismissed as edge cases or model-specific artifacts, without a framework for characterizing their frequency, distribution, or severity. VIDEOPHY transforms these anecdotes into quantitative evidence: physical commonsense failures are not edge cases, they are the norm for most models, and they are systematically related to the type of physical interaction (solid-solid is hardest) and the complexity of the depicted dynamics (hard prompts degrade more sharply). The paper thus converts scattered observations into a coherent diagnostic picture with clear boundary conditions.

Third, VIDEOPHY redirects research attention toward a neglected axis of model capability. Before VIDEOPHY, a researcher improving T2V models would focus on visual quality (higher resolution, fewer artifacts, better temporal consistency) or semantic adherence (better caption following). These are the dimensions that existing benchmarks reward. VIDEOPHY makes physical commonsense a legible target for improvement — model developers now have a specific benchmark against which they can optimize, and a specific gap (39.6% → 100%) they know they need to close. This shifts the optimization landscape: techniques that improve visual quality at the expense of physical fidelity will now be detectable and penalizable, while techniques that improve physical understanding — even at some cost to visual quality — may be valued in contexts where physical accuracy matters.

Fourth, the paper's finding that CogVideoX-5B dramatically outperforms CogVideoX-2B on physical commonsense (39.6% vs. 18.6% on the joint metric, 53% vs. 34.1% on PC alone) establishes a scaling relationship that was previously only speculated about. The more-than-doubling of performance with roughly 2.5× the parameters suggests that physical commonsense — like language understanding, reasoning, and other capabilities — benefits from scale. This provides empirical support for the hypothesis that larger models, trained on more video data, will continue to improve on physical commonsense. However, the 39.6% ceiling even at 5B parameters also suggests that scaling alone is insufficient — architectural innovations, training data improvements, or explicit physics-informed training objectives will likely be necessary to approach human-level physical commonsense.

Fifth, the VIDEOCON-PHYSICS component shifts the bottleneck in physical evaluation from annotation cost to model training cost. Prior to this work, evaluating physical commonsense required expensive, slow human annotation (the paper spent $3,500 on 36,500+ annotations, and scaling to more models or more prompts would cost proportionally more). By distilling human judgments into an automatic evaluator that achieves 73% ROC-AUC on PC and generalizes to unseen models (72% ROC-AUC, Table 5), the paper makes rapid, scalable evaluation feasible. This is the same dynamic that transformed ImageNet from a research dataset into an engine of progress — the availability of automatic evaluation enabled rapid iteration cycles that were impossible with human-in-the-loop protocols.

What becomes less attractive after this work: The naive approach of treating T2V models as off-the-shelf physics simulators. The paper demonstrates conclusively that current models cannot be relied upon to produce physically correct dynamics, even for simple everyday scenarios. Research directions that assume physical fidelity as a given (e.g., using generated video as training data for robotic control without verification) must now account for the high failure rates documented in VIDEOPHY. Similarly, evaluation protocols that report only visual quality and semantic alignment metrics without measuring physical commonsense are now demonstrably incomplete — a model that achieves state-of-the-art on VBench or EvalCrafter may still be physically incoherent, and reviewers and practitioners should demand physical commonsense evaluation as a standard component of model assessment.

What becomes more attractive: Research into the training data, architectures, and training objectives that specifically improve physical commonsense — a direction that was previously difficult to pursue because there was no way to measure progress. The paper identifies several promising hypotheses: CogVideoX's data curation pipeline (detailed captions, filtering of low-motion and low-quality videos) may partially explain its superior PC performance; the difficulty-conditioned analysis (Table 6) suggests that improving performance on hard prompts (those requiring complex constitutive models or high-velocity dynamics) is the highest-leverage target; and the negative result in Appendix P (fine-tuning on physically correct videos does not improve PC) suggests that simple data filtering is insufficient and more sophisticated training interventions (physics-informed losses, reinforcement learning from physical feedback, synthetic data from physics simulators) may be necessary.

Follow-Up Research This Work Enables

1. Causal dissection of why CogVideoX-5B outperforms other models. CogVideoX-5B achieves 39.6% on the joint metric while the next-best open model (VideoCrafter2) achieves only 19.0% (Table 3). The paper attributes this to CogVideoX's "high-quality data curation including inclusion of detailed captions, and filtering videos with less motion and poor quality," but this is speculation — there is no experiment isolating the effect of data filtering, caption quality, or architecture on physical commonsense. A strong follow-up study would systematically ablate CogVideoX's training pipeline: train a CogVideoX-2B (same architecture, smaller) on the same curated data to separate model capacity from data quality; train a CogVideoX-5B on unfiltered data to measure the contribution of data curation; and evaluate all variants on VIDEOPHY to determine which factor(s) actually cause the PC improvement. If data filtering accounts for most of the gap, the implication is that all T2V models could improve physical commonsense through better data curation without architectural changes. If architecture or scale dominates, the implication is that physical commonsense requires fundamentally different model designs or much larger models.

2. Physics-informed training objectives for T2V models. The negative result in Appendix P — fine-tuning Lumiere-T2I2V on videos that achieved SA = 1 and PC = 1 causes SA to drop from 46% to 36.5% while PC remains unchanged — demonstrates that simple supervised fine-tuning on "good" examples does not improve physical commonsense. This implies that physical understanding is not easily acquired from positive examples alone and may require explicit physics-informed training signals. Concrete follow-up directions include: (a) training with physics-based auxiliary losses (e.g., predicting optical flow and penalizing flow fields that violate conservation of mass or rigid body constraints), (b) synthetic data augmentation using physics simulators (generate videos of the same captions from a simulator and train the model to match the simulated dynamics), (c) contrastive or preference-based training where the model receives negative feedback on physically implausible outputs (using VIDEOCON-PHYSICS as the reward model in a reinforcement learning or direct preference optimization loop), and (d) multi-task training where the model jointly learns to generate videos and to predict physical properties (material type, contact forces, trajectories) as auxiliary tasks. VIDEOPHY provides the evaluation infrastructure to measure whether any of these interventions actually improve physical commonsense — previously, researchers had no way to quantify progress on this axis.

3. Temporal diagnostic of physical failure modes in generated video. VIDEOPHY evaluates physical commonsense as a binary judgment on the entire video, but does not investigate when physical violations occur or how they evolve over time. A diagnostic follow-up would annotate the temporal location and type of physical violations in a subset of VIDEOPHY videos: do violations occur early (the initial state is physically impossible, e.g., an object floating), late (the dynamics drift from physical correctness over time, e.g., a bouncing ball gradually loses the correct trajectory), or throughout? This temporal analysis is enabled by VIDEOPHY because the benchmark provides a set of prompts and model outputs where physical violations are known to occur. The finding would have direct implications for model architecture: early violations suggest problems with initial state generation or conditioning; late violations suggest problems with temporal consistency or long-range physical constraint satisfaction. A model that is physically correct for the first 0.5 seconds but drifts thereafter requires different fixes (e.g., recurrent physical consistency modules) than a model that starts wrong and stays wrong (e.g., better physical priors in the generation process). This analysis could be performed using the existing VIDEOPHY videos and would not require new data collection — only a new annotation layer on existing videos.

4. Quantitative failure mode taxonomy and model-specific error fingerprints. The paper's qualitative analysis (Section 5.2) identifies six common failure modes: conservation of mass violation, Newton's first law violation, Newton's second law violation, solid constitutive law violation, fluid constitutive law violation, and non-physical penetration. However, there is no quantitative breakdown — we do not know whether different models exhibit different failure mode distributions, whether certain failure modes are easier to fix than others, or whether failure mode prevalence varies with difficulty or interaction type. A follow-up study would hire expert annotators (physics students or graphics researchers) to classify the specific failure mode(s) present in each VIDEOPHY video that received PC = 0, producing per-model error fingerprints. If CogVideoX-5B primarily fails on solid constitutive law violations while Dream Machine primarily fails on conservation of mass and Newton's law violations, developers would know exactly where to focus improvement efforts. This study would also reveal whether certain failure modes are universal across models (suggesting fundamental limitations of current architectures) or model-specific (suggesting differences in training data or inductive biases). The existing VIDEOPHY videos and human annotations provide the substrate for this analysis; the only additional cost is expert annotation time.

5. Multimodal foundation models as zero-shot physical commonsense evaluators — diagnosing the failure. The paper's finding that Gemini-1.5-Pro-Vision — a state-of-the-art multimodal model that processes entire videos — achieves only 58% ROC-AUC on physical commonsense (barely above random, Table 4) is a striking negative result that demands explanation. Is Gemini failing because it cannot perceive the relevant visual details (limited spatial or temporal resolution), because it cannot reason causally about physical dynamics from visual input, or because it has not been trained to map visual observations to physical law judgments? A diagnostic follow-up would systematically vary the input to Gemini: provide ground-truth physics violation labels as text alongside the video (does Gemini correctly identify that "the water level doesn't rise" means a violation?); provide still frames that unambiguously show violations (does Gemini need the temporal dimension, or does it fail even on static images of physically impossible states?); provide synthetic videos from physics simulators where ground-truth correctness is known exactly (does Gemini's near-random performance reflect genuine inability to judge physics, or is it an artifact of the generated video distribution?). This study would clarify whether VIDEOCON-PHYSICS's superior performance (73% ROC-AUC) stems from task-specific fine-tuning on in-domain data (suggesting that any large video model could be adapted to physical commonsense evaluation with sufficient annotated data) or from architectural advantages of VIDEOCON's design (suggesting that physical commonsense evaluation requires specific architectural choices). The answer has implications for whether the field should invest in better general-purpose multimodal models or in specialized physical evaluation models.

6. Extension of VIDEOPHY to non-contact and invisible physical phenomena. VIDEOPHY's prompts are restricted to contact-based interactions between solids and fluids, explicitly excluding penetration forces, state changes, dissolving, chemical reactions, and other "invisible" processes. This was a pragmatic choice for initial benchmark construction, but it leaves large swaths of physical commonsense unevaluated. A natural follow-up is VIDEOPHY-2.0 covering: (a) rotational and articulated dynamics — spinning tops, pendulums, levers, pulleys, gears; (b) non-contact forces — magnetism, electrostatic attraction/repulsion, gravitational orbital motion; (c) phase transitions — melting ice, boiling water, condensation; (d) penetration and fracture — cutting, scratching, breaking, shattering; (e) thermodynamic and fluid phenomena beyond simple contact — convection currents, vortex shedding, capillary action, diffusion. Each new category would require new prompt engineering (likely using the same GPT-4-based pipeline with modified constraints), new difficulty annotation (requiring graphics or physics expertise for the relevant phenomena), and new human evaluation (because the current annotator pool was only trained on contact-based dynamics). The expansion would test whether the patterns observed in the original VIDEOPHY — solid-solid interactions being hardest, performance degrading with simulation complexity, CogVideoX dominating — generalize across physical phenomenon types, or whether different models have different strengths in different physical regimes. This would transform VIDEOPHY from a single-category benchmark into a comprehensive physical commonsense evaluation suite.

Practical Applications and Downstream Use Cases

1. Model selection and quality assurance for video generation APIs and products. Companies building products on top of T2V models (video generation APIs, creative tools, advertising content generators) currently have no way to evaluate physical commonsense when choosing between model providers. Two models may have similar visual quality and similar pricing, but one may produce physically coherent videos 40% of the time (CogVideoX-5B level) while the other produces physically coherent videos only 5% of the time (OpenSora level). If the downstream application involves content where physical plausibility matters (product demonstrations, educational content, synthetic training data), the choice of model has enormous practical consequences. VIDEOPHY — particularly the VIDEOCON-PHYSICS automatic evaluator, which costs only inference compute to run — enables systematic model comparison on this dimension before committing to a provider. A company could evaluate all candidate models on VIDEOPHY in minutes, produce a physical commonsense score alongside visual quality and cost metrics, and make an informed procurement decision. The 4× gap between CogVideoX-5B (39.6%) and the next-best open model (VideoCrafter2 at 19.0%) on the joint metric demonstrates that model choice alone can massively impact physical fidelity, and VIDEOPHY makes this choice evidence-based rather than vibes-based.

2. Data filtering for training improved video generation models. The paper demonstrates that fine-tuning on physically correct videos does not improve physical commonsense (Appendix P, Table 12), but that does not mean physical commonsense evaluation is irrelevant to training. A more sophisticated training pipeline could use VIDEOCON-PHYSICS as a data quality filter during pretraining data curation: score all videos in the training corpus for physical commonsense (using the auto-evaluator on video-caption pairs), and either downweight, filter out, or flag videos with low PC scores. This is directly analogous to how LAION aesthetic scores are used to filter image datasets for text-to-image models — remove low-aesthetic-quality images to improve generation quality. The paper's finding that CogVideoX-5B — which used "filtering videos with less motion and poor quality" in its training pipeline — achieves dramatically better PC than models without such filtering suggests that data curation is a promising lever. VIDEOCON-PHYSICS makes this approach feasible at scale: scoring millions of videos for physical commonsense using human annotators would be prohibitive (at $18/hour and ~30 seconds per judgment,$3,500 covered only 36,500 annotations), but automated scoring with 73% ROC-AUC can process entire video corpora for the cost of inference compute. Even with imperfect accuracy, filtering out the worst 50% of videos by PC score could substantially shift the training distribution toward physically plausible content.

3. Synthetic data generation for robotics and embodied AI. A major motivation for T2V models is generating synthetic training data for robots and embodied agents — if a model can produce diverse, physically accurate videos of manipulation tasks, navigation scenarios, or object interactions, these videos could substitute for expensive real-world data collection. VIDEOPHY provides the first systematic evidence of whether current models are adequate for this use case. The answer is sobering: even the best model (CogVideoX-5B) produces physically plausible videos for only 39.6% of prompts when both caption following and physical commonsense are required. For solid-solid interactions — precisely the type of manipulation tasks most relevant to robotics — the figure drops to 24.4%. A robotics lab considering using T2V-generated videos as training data should, at minimum, evaluate their candidate model on VIDEOPHY's solid-solid subset to estimate the fraction of generated data that will be physically meaningless, and budget accordingly (e.g., if only 24.4% of generated videos are useful, generate 4× more videos than needed and filter). More constructively, VIDEOPHY can be used to track progress: as new models are released, their VIDEOPHY scores on solid-solid interactions provide a direct measure of whether the field is approaching the reliability threshold needed for synthetic data generation. A model achieving >80% on VIDEOPHY solid-solid interactions would be a qualitatively different tool for robotics than current models achieving <25%.

4. Benchmarking as a regulatory or certification tool for high-stakes video generation applications. As T2V models are deployed in contexts where physical fidelity has real-world consequences — generating training data for autonomous vehicles, creating educational science content, simulating emergency scenarios for first responder training — the lack of physical commonsense becomes a safety issue rather than merely a quality issue. A generated video used to train a self-driving car's pedestrian prediction module that shows pedestrians sliding frictionlessly or deforming on impact could cause the trained system to make dangerously wrong predictions. VIDEOPHY, while not designed for regulatory use, provides a template for how physical commonsense evaluation could be operationalized in certification workflows. A regulator could require T2V models used in safety-critical applications to achieve a minimum VIDEOPHY score (e.g., >70% on the joint metric) with independent evaluation, and could use the per-category and per-difficulty breakdowns to ensure the model performs adequately on the specific physical interaction types relevant to the application. The VIDEOCON-PHYSICS auto-evaluator, while not accurate enough for certification decisions (73% ROC-AUC leaves too much uncertainty), demonstrates that automated physical commonsense evaluation is feasible and could be improved with more training data and better base models — pointing toward a future where physical fidelity certification is routine and automated.

When to Prefer This Method

This paper introduces an evaluation framework, not a generative method, so the standard "prefer A over B" decision matrix does not apply in the usual sense. However, the paper does articulate a clear evaluation philosophy tradeoff between intuitive physics-based human evaluation (VIDEOPHY's approach) and simulation-based evaluation (the alternative the paper explicitly rejects). The decision rule is:

• Prefer VIDEOPHY-style intuitive physics human evaluation when: (1) The goal is to assess whether generated videos meet human expectations of physical plausibility for practical applications (content creation, training data filtering, model comparison), where approximate correctness — "does this look physically right to a person?" — is sufficient. (2) Accurate 3D reconstruction, material parameter estimation, and inverse rendering from generated video are infeasible (which they currently are, for arbitrary single-view generated video). (3) The standard of correctness is alignment with human physical intuition developed through real-world experience, rather than precise agreement with a specific physics simulator's numerical solutions. (4) Rapid, scalable evaluation is needed, enabled by the VIDEOCON-PHYSICS auto-evaluator after an initial investment in human annotation.

• Prefer simulation-based evaluation when: (1) The application requires precise physical accuracy — exact trajectories, forces, material deformations — where subtle errors that humans might miss could cause downstream failures (e.g., using generated video as a forward model for robotic control in high-precision tasks). (2) The scenario involves physical phenomena where human intuition is known to be unreliable (e.g., unintuitive fluid dynamics, complex multi-body orbital mechanics, phenomena at scales humans don't experience). (3) Ground-truth 3D geometry, material parameters, lighting, and rendering are available — which currently limits this approach to synthetic or carefully controlled real scenes, not arbitrary generated video. (4) The evaluation is being used for safety certification or regulatory compliance where human perceptual judgment is considered too subjective.

The paper's key empirical finding — that Dream Machine achieves 61.9% semantic adherence but only 21.8% physical commonsense (Table 3) — demonstrates exactly why the distinction matters. A simulation-blind evaluation that only measures semantic adherence would rank Dream Machine as a strong model (second-highest SA), while VIDEOPHY's joint evaluation correctly identifies it as physically unreliable. The choice of evaluation framework determines which models appear successful, and VIDEOPHY argues — with evidence — that physical commonsense must be measured separately and explicitly.