Everything in Its Place: Benchmarking Spatial Intelligence of Text-to-Image Models

ArXiv: 2601.20354

🎯 Pitch

Even top text-to-image models like those from OpenAI score under 30% on basic spatial reasoning tasks—like telling left from right or handling occlusion—while acing object recognition above 70%. This new benchmark probes that gap with 1,230 dense prompts and shows that fixing it requires better data, not bigger models.

1. Executive Summary

This paper introduces SpatialGenEval, a benchmark that systematically evaluates the spatial intelligence of text-to-image models using 1,230 long, information-dense prompts that each integrate all 10 spatial sub-domains — from object position and layout to occlusion and causality — paired with 12,300 omni-dimensional multiple-choice questions. The study benchmarks 23 state-of-the-art models and reveals spatial reasoning (relative comparison, occlusion, proximity) as the primary performance bottleneck, with even top models scoring below 30% on these sub-domains while exceeding 70% on basic object and attribute generation. To demonstrate utility beyond evaluation, the paper also constructs SpatialT2I, a 15,400 text-image pair dataset with rewritten prompts that preserve information density while improving image consistency, yielding consistent fine-tuning gains of +4.2%, +5.7%, and +4.4% on Stable Diffusion-XL, UniWorld-V1, and OmniGen2 respectively — establishing that a data-centric paradigm can improve spatial intelligence in text-to-image models, though only within the bounds of capabilities the base model already possesses.

2. Context and Motivation

The Core Problem: Text-to-Image Models Can Generate Objects, but They Can't Place Them Correctly

Modern text-to-image (T2I) models have achieved remarkable success at rendering what a scene contains. Given a prompt like "a red apple on a wooden table," they reliably produce an apple, a table, the correct colors, and plausible materials. This is what the paper calls Spatial Foundation—the ability to generate semantically correct objects and bind attributes to them correctly.

The problem arises when prompts shift from "what" to "where," "how," and "why." Consider a more demanding instruction: "A green book lies on the left side of a desk, with a blue pen touching its right edge. The book is three times wider than the pen. A coffee mug, closest to the viewer, partially obscures the book's lower-right corner." This prompt requires an integrated understanding of absolute positioning ("left side"), relative proximity ("touching its right edge"), quantitative comparison ("three times wider than"), 3D depth and occlusion ("partially obscures"), and physical interaction—all simultaneously.

As Figure 1 demonstrates, even state-of-the-art models from OpenAI, Qwen, and ByteDance fail persistently on such prompts. They misplace objects, orient them incorrectly, disregard numerical comparisons, or fail to render causal interactions. These are not cosmetic flaws in image quality or aesthetic appeal. They represent a fundamental capability gap: the absence of spatial intelligence in a medium where spatial coherence is definitional.

The paper defines spatial intelligence through a hierarchical framework that builds from basic perception to higher-order reasoning. At the base, spatial foundation covers object presence and attribute binding—the "what" that current models handle decently. Spatial perception moves to geometric arrangements: absolute and relative positions, object orientations, and multi-object layouts. Spatial reasoning demands quantitative comparisons, fine-grained proximity judgments, and implicit 3D occlusion understanding. At the peak, spatial interaction requires depicting dynamic motion and causal cause-effect relationships. This hierarchy is not arbitrary; it draws from cognitive science research on how spatial cognition develops in humans, progressing from basic object identification through relational inference to dynamic causal understanding.

The implication is stark: current models inhabit flat, 2D semantic space where objects exist as independent labels floating on a canvas. They lack the internal world model needed to reason about depth, physics, proportion, and causality simultaneously. This is precisely what prevents T2I systems from being used as reliable design tools, architectural visualization aids, or embodied AI training data generators—all applications where spatial correctness is non-negotiable.

Why This Matters: Beyond Academic Benchmarking to Real-World Deployment

The significance of this gap extends far beyond academic interest. Several practical stakes make spatial intelligence a critical frontier for the field:

Creative and professional tools cannot rely on spatially unreliable models. Architects and interior designers need precise control over object placement, scale relationships, and occlusion to use AI as a visualization partner. A model that misplaces furniture relative to architectural features, or scales objects incorrectly in perspective, produces outputs that are useless—or worse, misleading—for professional workflows. Similarly, game designers and film pre-visualization artists require coherent spatial layouts that respect physical and cinematic constraints.

Embodied AI systems depend on spatially coherent training data. A growing research direction uses T2I models to generate training scenarios for robots and autonomous agents. If these generated images show physically impossible arrangements—objects floating in space, inconsistent occlusion, violated causal relationships—the downstream AI learns a distorted model of reality. The paper connects to this explicitly by grounding its spatial intelligence definition in cognitive science work on embodied intelligence, where understanding space is inseparable from planning action within it.

The gap exposes a fundamental architectural limitation, not a training data problem. Prior work in T2I evaluation has largely focused on improving semantic composition: can the model generate the specified objects with the correct attributes? This has driven progress in text encoder quality (the paper notes that models with stronger LLM-based encoders like T5 consistently outperform CLIP-only architectures) and in scaling model parameters. But the paper's benchmark reveals that these improvements disproportionately benefit basic object-attribute generation while leaving higher-order spatial reasoning largely untouched. This suggests the bottleneck is not simply a matter of more data or larger models, but a deeper architectural question about how spatial relationships are represented and composed during the generation process.

Without principled spatial evaluation, progress appears deceptively strong. Existing benchmarks that focus on object presence and simple binary relations create a misleading picture of model capability. A model that correctly generates "a cat" and "a cushion" can score well on a detection-based benchmark while completely failing to render the cat sitting on the cushion with correct occlusion and scale. This means the field lacks the diagnostic tools to identify where models actually fail, making it impossible to direct research effort toward the genuine bottlenecks. The paper is essentially arguing that we have been grading models on multiple-choice vocabulary tests while ignoring their inability to write coherent paragraphs.

Where Existing Benchmarks Fall Short: Short Prompts, Sparse Information, Coarse Evaluation

The paper identifies a specific, systematic weakness in the current evaluation landscape. Table 1 provides a structured comparison with five major benchmarks, and the deficiencies cluster into three categories:

1. Short or information-sparse prompts fail to probe real-world complexity.

Benchmarks like T2I-CompBench and GenEval use prompts that specify one or two spatial relationships at most—typically a simple binary relation like "a ball to the left of a cup." Even benchmarks that claim to test more complex instruction-following, like TIIF-Bench, still structure their prompts around isolated spatial constraints rather than densely interleaved ones.

This design choice has a critical consequence: it tests whether a model can handle any single spatial constraint but reveals nothing about whether it can handle multiple constraints simultaneously. Real-world prompts—whether from a professional designer or a casual user—rarely specify just one spatial relationship. They describe scenes holistically, with spatial constraints that interact with and constrain each other. A model that can place "A left of B" and separately "B above C" may fail entirely when asked to do both in the same scene because the constraints interact geometrically and the model must resolve their intersections.

The paper's solution is to design prompts that are long (approximately 60 words) and information-dense, with each prompt integrating all 10 spatial sub-domains simultaneously. This forces the model to handle the combinatorial complexity of real spatial reasoning rather than cherry-picking isolated successes.

2. Single-dimension evaluation metrics create blind spots.

Previous benchmarks typically evaluate prompts along a single axis at a time. A benchmark might test "spatial relationships" but treat relative position, orientation, and occlusion as undifferentiated components of a single score. This means a model that excels at absolute positioning but fails at occlusion can appear competitive with a model showing balanced performance across both.

The paper addresses this through omni-dimensional evaluation: each prompt is paired with exactly 10 multiple-choice questions, one for each spatial sub-domain. This produces a fine-grained performance profile across the full spatial intelligence hierarchy. A model's score is no longer a single number masking severe imbalances; it becomes a 10-dimensional vector that reveals precisely where spatial reasoning breaks down.

3. Coarse-grained metrics force binary judgments that miss nuanced failures.

Existing benchmarks predominantly use detection-based scoring (does the object exist in the correct region?) or binary yes/no questions (is the cat left of the dog?). Detection-based methods require expensive annotation pipelines and can miss subtle spatial violations that a human would immediately notice. Binary questions provide limited diagnostic granularity—a model might get "mostly right" spatial relationships but receive a zero score because the exact threshold wasn't met.

The paper's multiple-choice format with a "None" option creates a richer signal. Each question presents four plausible spatial configurations plus the option to reject all choices. This means an evaluator can detect partial understanding (when a model selects a spatially plausible but incorrect arrangement) versus complete failure (when it selects "None" or a nonsensical option). The 5-round majority voting mechanism further stabilizes this evaluation, requiring the MLLM judge to reach the same conclusion consistently rather than depending on a single potentially noisy inference.

4. No systematic coverage across the full spatial intelligence hierarchy.

Even benchmarks that test spatial relationships typically focus on narrow subsets—often just 2D positioning or simple proximity. The paper's literature review reveals that no prior benchmark systematically covers the full progression from basic object perception through dynamic causal interaction. This means the field has been evaluating spatial capability without a theoretical framework for what spatial capability entails, making it impossible to assess relative strengths and weaknesses across the hierarchy.

The 10 sub-domains in SpatialGenEval form a deliberate progression. The paper motivates each one by its role in real-world spatial cognition, arguing that mastering position without occlusion, or motion without causality, represents an incomplete form of spatial intelligence. This framework is what allows the benchmark not just to rank models but to diagnose them—identifying whether a model's spatial failures stem from basic perceptual limitations, reasoning deficits, or interaction failures.

How This Paper Positions Itself: Evaluation as Discovery and Improvement

The paper positions itself at the intersection of two research directions that are usually pursued independently: benchmarking for evaluation and dataset construction for improvement. Most benchmark papers stop at measurement—they categorize what models can and cannot do and leave it to subsequent work to address the identified gaps. Most dataset papers focus on training data quality and scaling without connecting their choices to a principled evaluation framework.

SpatialGenEval bridges this divide through its companion dataset, SpatialT2I. The same principles that make the benchmark diagnostic—information-dense prompts covering all 10 sub-domains, carefully curated to be logically sound and lexically fair—also make for an effective fine-tuning dataset. The paper demonstrates this through concrete gains: fine-tuning on SpatialT2I improves spatial performance consistently across three architecturally different T2I models (a diffusion model, a unified model, and an autoregressive model), with the largest gains appearing in the spatial reasoning sub-domains where models struggle most.

This dual contribution changes the paper's role in the research landscape. It is not merely identifying a problem; it is providing an initial recipe for addressing it. The fine-tuning results do not close the gap—models still fail badly on occlusion and comparison even after SpatialT2I training—but they validate that the information-dense, spatially-structured data format is the right intervention target. This suggests that the bottleneck is at least partly data-driven: models have not seen enough examples where spatial constraints are densely interleaved in coherent, real-world-like prompts. Scaling up SpatialT2I-style data generation is therefore a plausible path toward closing the spatial intelligence gap, even if it cannot fully solve the hardest problems that likely require architectural innovations.

The paper also positions itself relative to the broader evolution of T2I evaluation. It traces a trajectory from basic composition benchmarks (T2I-CompBench, GenEval) through instruction-following benchmarks (TIIF-Bench, DPG-Bench) to emerging specialized evaluations for physics (PhyBench), world knowledge (Wise), and reasoning (R2I-Bench). SpatialGenEval sits at the convergence of these trends: it is simultaneously about composition (can you place objects correctly?), instruction following (can you process a complex prompt?), and reasoning (can you infer implicit spatial relationships?). The paper is arguing that spatial intelligence is not one specialized capability among many but rather the integrative challenge that ties together composition, instruction following, and physical reasoning into a coherent whole.

Finally, the paper distances itself from a common pitfall in benchmark design: the assumption that the evaluator must be as capable as the generator. By explicitly designing the benchmark so that evaluation questions are simpler verification tasks (direct visual question answering) while the generation task is the complex synthesis challenge, the paper makes a practical engineering choice that is theoretically justified. The MLLM evaluator does not need to plan spatial layouts or reason about occlusion; it merely needs to identify whether the generated image satisfies specific, visually checkable constraints. This design choice is validated by high human-MLLM alignment scores, particularly on the simpler spatial sub-domains where evaluation is most critical.

In summary, this paper addresses a recognized but poorly measured problem: the systematic failure of T2I models to handle complex, multi-dimensional spatial constraints in a coherently integrated way. It argues this matters for real-world deployment, provides the first benchmark with sufficient diagnostic granularity to pinpoint the specific failures, and demonstrates that the same principles underlying the benchmark can guide practical improvement through targeted fine-tuning data. The central thesis is that spatial intelligence is not an emergent property that larger models will simply acquire—it requires deliberate evaluation, targeted data, and potentially architectural innovation—and the paper supplies the evaluation framework and initial data recipe to pursue this agenda.

3. Technical Approach

3.1 Reader Orientation

This paper constructs two tightly coupled artifacts: a diagnostic benchmark (SpatialGenEval) that measures how well text-to-image models handle complex, multi-dimensional spatial constraints, and a fine-tuning dataset (SpatialT2I) built on the same principles that demonstrably improves spatial performance when used for supervised training. The core problem is that existing benchmarks use short, information-sparse prompts that test isolated spatial relationships (e.g., "a ball left of a cup"), which means they capture whether a model can handle any single constraint but reveal nothing about whether it can handle multiple interacting constraints simultaneously—the actual demand in real-world usage. The solution takes the form of a hierarchical evaluation framework that systematically probes 10 spatial sub-domains per prompt using multiple-choice visual question answering, combined with a data generation pipeline that maps these same dense prompts to curated text-image pairs for supervised fine-tuning.

3.2 Big-Picture Architecture (Diagram in Words)

The overall system has five major components, with two distinct pipelines sharing a common generation philosophy:

Spatial Intelligence Taxonomy — a hierarchical decomposition defining 4 domains (spatial foundation, perception, reasoning, interaction) and 10 corresponding sub-domains. This taxonomy drives all downstream prompt generation, question design, and evaluation. It is not a post-hoc categorization but the generative scaffold for benchmark construction.
Prompt Generation Engine (Gemini 2.5 Pro) — takes a real-world scene and the 10 sub-domain definitions as input, then synthesizes a single fluent prompt (~60 words) that seamlessly integrates all 10 spatial constraints. This is the "information-dense" design principle: one prompt must simultaneously specify object identities, attributes, absolute positions, relative arrangements, orientations, layout patterns, quantitative comparisons, proximity relations, occlusion structure, motion dynamics, and causal interactions.
Question-Answer Generation Engine (Gemini 2.5 Pro) — takes the synthesized prompt and the 10 sub-domain definitions, then generates exactly 10 multiple-choice questions (one per sub-domain), each with one ground-truth answer and three plausible distractors. A fifth "E: None" option is programmatically appended to every question to allow refusal when no option matches the generated image.
Human-in-the-Loop Refinement Pipeline — five expert annotators review every prompt and every QA pair with a detailed guidebook, correcting stylistic awkwardness, logical impossibilities, ambiguous vocabulary, and answer leakage (questions that reveal their answer through phrasing). This stage is the quality gate that transforms MLLM-generated drafts into a reliable benchmark.
Evaluation and Training Pipelines — the benchmark pipeline: 23 T2I models generate images from each prompt, then Qwen2.5-VL-72B (or GPT-4o) evaluates each image against its 10 QA pairs using 5-round majority voting, producing a per-model accuracy profile across all 10 sub-domains. The training pipeline (SpatialT2I): a separate set of 1,230 prompts is generated following the same principles; 14 top-performing T2I models generate images; Gemini 2.5 Pro rewrites each prompt to match the actual image content while preserving information density; the resulting 15,400 text-image pairs are used to fine-tune base T2I models.

Information flows as follows: the spatial taxonomy constrains every step → a real-world scene is selected → Gemini 2.5 Pro generates a dense prompt and 10 QA pairs in a single pass → human annotators refine both → T2I models generate images → MLLM judges evaluate images against QA pairs → per-model performance profiles are aggregated. The SpatialT2I branch follows a parallel flow but substitutes "rewrite prompts to match generated images" for evaluation, producing training data rather than benchmark scores.

3.3 Roadmap for the Deep Dive

First, the spatial intelligence taxonomy and the 10 sub-domains — these are the conceptual armature on which everything else hangs, and understanding the precise definitions of each sub-domain is prerequisite to understanding the benchmark’s diagnostic power.
Second, the information-dense prompt generation pipeline — how Gemini 2.5 Pro transforms a scene + taxonomy into an integrated prompt, and why the "all 10 constraints in one prompt" design is the key innovation over prior benchmarks.
Third, the omni-dimensional QA generation pipeline and the MLLM-as-judge evaluation protocol — how the questions are constructed to prevent answer leakage, the role of the "E: None" refusal option, and the 5-round majority voting mechanism that stabilizes automated evaluation.
Fourth, the human-in-the-loop refinement process — the specific failure modes that human annotators catch (disjointed phrasing, logical impossibilities, answer leakage, obscure vocabulary) and why automated generation alone would produce an unreliable benchmark.
Fifth, the MLLM evaluator design choices — why Qwen2.5-VL-72B is selected as the primary evaluator, how the evaluation prompt is structured to prevent the judge from using external knowledge, and the human alignment study that validates automated evaluation.
Sixth, the SpatialT2I fine-tuning dataset construction — how it extends the benchmark principles into a training data pipeline, the prompt rewriting strategy that corrects text-image misalignment while preserving spatial density, and the model selection criteria for which generated images enter the dataset.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an evaluation and dataset paper whose core idea is that spatial intelligence in T2I models must be diagnosed at fine granularity across a hierarchical taxonomy of spatial capabilities, and that the same information-dense prompt structure that enables diagnosis also serves as effective fine-tuning data for improving those capabilities.

The Spatial Intelligence Taxonomy: 4 Domains, 10 Sub-Domains

The benchmark's entire structure rests on a hierarchical decomposition of spatial intelligence into four domains of increasing cognitive complexity, with 10 corresponding sub-domains that serve as independently evaluable dimensions. The paper motivates this taxonomy by drawing on cognitive science literature on spatial cognition development and recent computer vision frameworks for spatial understanding, but the specific decomposition is a novel synthesis designed to cover the full progression from basic object identification through dynamic causal reasoning.

Spatial Foundation (S1/S2): The "What" of a Scene

This domain evaluates whether the model can generate semantically correct objects and bind attributes to them correctly. It is the prerequisite layer — if a model fails here, it will fail everywhere else.

S1: Object Category — tests compositional completeness: can the model generate all mentioned objects without omission or hallucination? The evaluation question is not merely "Is there an X?" but probes whether the set of objects present matches the full specified set. The paper notes a critical design choice: questions deliberately test comprehensive scene awareness rather than single-object presence (e.g., "What else exists in the scene besides A, B, and C?" rather than "Is there an A?"), because the latter would trivially pass if the model generates the object but miss omission failures.
S2: Object Attribution — tests attribute binding: can the model assign attributes (color, shape, material, texture) to their correct referent objects without "attribute leakage" (where a color specified for object A erroneously appears on object B)? The evaluation questions query for multi-attribute combinations (e.g., "What is the color and material of the vase?") rather than single, isolated properties, because isolated attribute checks miss binding errors where the correct attributes are generated but on the wrong objects.

Spatial Perception (S3/S4/S5): The "Where" and "How Arranged"

This domain moves from object identity to geometric and relational arrangement on the 2D canvas. It tests the accurate translation of spatial language into visual form.

S3: Spatial Position — tests localization using both absolute terms (e.g., "top-left," "in the bottom half," "centered") and relative terms (e.g., "to the right of the book," "to his left side," "beneath the chandelier"). The evaluation questions require determining an object's position from another object's viewpoint, not just a global third-person perspective. This is a deliberate escalation: a model might correctly place an object in the global left half of the image while failing to place it "to the left of the sofa" because the sofa's position varies.
S4: Spatial Orientation — tests rotational alignment: can the model generate objects with specified facing directions (e.g., "facing left," "upside down," "pointing toward the horizon") rather than defaulting to canonical poses (typically front-facing)? The paper explicitly identifies orientation as a common challenge, noting that models often overfit to front-view renderings, which reveals a lack of viewpoint-aware object representation.
S5: Spatial Layout — tests understanding of multi-object collective arrangements beyond pairwise relations. This includes linear sequences ("in a line from left to right," "arranged alphabetically"), circular formations ("in a circle around the central fountain"), grid patterns, and other specified group structures. This sub-dimension is crucial for testing comprehension of group-level spatial patterns, which cannot be reduced to aggregating pairwise relations: "A, B, and C form a circle" requires simultaneous constraints on all three relative positions.

Spatial Reasoning (S6/S7/S8): The "How They Relate" — Higher-Order Inference

This domain moves beyond direct perception to assess the model's ability to understand and render abstract, implicit, and 3D-aware spatial relationships. These are not directly visible from object positions alone; they require inferential computation.

S6: Spatial Comparison — tests relative quantitative attribute reasoning. The model must generate objects adhering to comparative statements about properties such as size ("three times taller than," "half the width of"), quantity ("more trees than benches"), or length ("the shadow is longer than the object"). The evaluation questions demand quantitative distinctions (e.g., "How many times taller is X than Y?") rather than simple "more/less" dichotomies, forcing the model to encode and render precise magnitude relationships.
S7: Spatial Proximity — tests fine-grained physical distance control. The model must render precise spatial interactions like "touching" (zero gap), "closest to" (distance ordering), "far from" (large gap), or "just inches away from" (small but non-zero gap). This sub-dimension is critical because it assesses whether the model controls object boundaries and depicts intimate spatial relationships, which are often overlooked in favor of simple co-occurrence (the objects exist, but their distance relationships are random).
S8: Spatial Occlusion — tests implicit 3D scene structure and depth understanding. The model must generate scenes where one object partially or fully obscures another (e.g., "the vase is partially obscuring the book," "the sofa hides the lower half of the table"). Success requires a sophisticated world model that reasons about viewpoint, relative depth ordering, and object layering. This is the sub-domain that most clearly distinguishes flat 2D composition from genuine 3D-aware scene understanding.

Spatial Interaction (S9/S10): The "Why Things Happen" — Dynamic Events

This is the most advanced domain, evaluating whether the model can depict dynamic events and physical causality. It moves beyond static scene composition into rudimentary physics and temporal reasoning.

S9: Spatial Motion Interaction — tests the generation of objects in dynamic states or mid-action sequences. The model must capture a specific temporal moment: "a dog jumping over a log," "a mid-flight ball," "leaves blowing in the wind." This tests the ability to convey movement through pose, trajectory, and contextual cues rather than relying on static or canonical placements. The difficulty lies in synthesizing a single frame that unambiguously communicates a dynamic process.
S10: Spatial Causal Interaction — tests explicit cause-effect relationships between objects or environments. Examples include "a rock hitting water and causing ripples," "a hammer striking a nail into wood," "wind bending the trees." Success implies the model can reason about functional physical relationships and translate latent dynamics into visually consistent, logically plausible images. This is distinct from motion interaction: a jumping dog is motion without necessarily showing what caused the jump; causal interaction requires the effect to be visibly attributable to a specific cause depicted in the scene.

Why This Particular Taxonomy?

The paper's taxonomy is not arbitrary. It follows a deliberate progression sequence that mirrors the development of spatial cognition: perceiving objects → perceiving their arrangements → inferring relationships between them → understanding dynamic events and their causes. This progression is motivated by references to cognitive science (Malanchini et al., 2020; Gupta et al., 2021; Ruan et al., 2025) and recent computer vision benchmarks for MLLM spatial intelligence (Yang et al., 2025a; Stogiannidis et al., 2025; Gong et al., 2025). The key design insight is that each sub-domain builds conceptually on the ones before it: you cannot test occlusion without first testing object generation; you cannot test causal interaction without first testing motion perception. This dependency structure means the benchmark can isolate where in the hierarchy a model's performance collapses, providing diagnostic precision that single-score benchmarks cannot offer.

Information-Dense Prompt Generation (SpatialGenEval Construction Stage 1)

The prompt generation pipeline is the central mechanism that distinguishes SpatialGenEval from prior benchmarks. Rather than generating many prompts each targeting one or two spatial dimensions, it generates a moderate number of prompts (1,230) where each prompt integrates all 10 sub-domains simultaneously. This is the "information-dense" design principle.

Automated Generation via Gemini 2.5 Pro

The generation process has two inputs fed to Gemini 2.5 Pro:

A specific scene selected from a curated set of 25 real-world scenes spanning five categories: Nature (Forest, Mountain, Desert, Beach, Underwater, Cityspace, Village — 28.5% of prompts), Outdoor (Airport, Railway, Park, Zoo, Shopping Mall, Art Gallery, Cafe, Library — 32.5%), Indoor (Kitchen, Classroom, Living Room, Office — 16.3%), Human (Sports, Human Activities, Portraits — 12.2%), and Design (Cartoon, Advertisement, Story — 10.6%).
The definitions of all 10 spatial sub-domains, each with its specific evaluation focus.

The MLLM is instructed to "seamlessly integrate all 10 spatial constraints into a single, fluent, and logically sound prompt" based on the given scene. The target length is approximately 60 words, chosen as a deliberate compromise. On the one hand, shorter prompts (10-20 words) cannot accommodate 10 distinct spatial constraints without becoming unnatural laundry lists. On the other hand, prompts much longer than 60 words exceed the 77-token context window of standard CLIP text encoders, which would unfairly penalize models that rely on CLIP without the adaptation mechanisms (pooling, chunking) that some architectures use. The 60-word target balances information density against encoder compatibility.

The paper's Appendix A.4 includes the full meta-instruction, which specifies that the prompt must:

Cover all 10 spatial sub-domains without omission.
Form a coherent, natural-sounding scene description rather than a checklist.
Use common, unambiguous vocabulary.
Avoid logical contradictions (e.g., cyclical layouts where A is left of B, B is left of C, and C is left of A).
Target approximately 60 words.

The model generates 50 distinct prompts per scene in a single batch (50 prompts × 25 scenes = 1,250 initial candidates, later refined to 1,230 after human filtering), requiring strong creative ability to avoid repetition within the same scene.

Why Gemini 2.5 Pro?

The paper reports a controlled comparison (Appendix A.9.1) where three MLLMs (Gemini 2.5 Pro, GPT-4o, Qwen2.5-VL-72B) were each instructed to generate prompts. All three demonstrated "strong instruction following ability," but Gemini 2.5 Pro "outperformed the others in terms of creative ability." This was measured both by (a) consistent human annotator agreement across five reviewers and (b) quantitative sentence similarity scores: average cosine similarity between prompts within the same scene was Gemini 2.5 Pro (0.4938) < GPT-4o (0.5125) < Qwen2.5-VL-72B (0.5548), measured using Sentence-BERT embeddings. Lower similarity means less repetition, which is essential for generating a diverse benchmark where no two prompts describe the same arrangement.

This is a non-trivial consideration. If the prompt generator produces template-like outputs ("The [object A] is [relation] to the [object B]. The [object C] is [relation] to the [object D]."), the benchmark would systematically underestimate model capability because every prompt would share the same linguistic template, making the test more about parsing a specific syntactic pattern than about general spatial reasoning. Gemini 2.5 Pro's lower similarity scores indicate greater linguistic diversity in how spatial constraints are expressed.

Every MLLM-generated prompt undergoes human review by five expert annotators following a detailed guidebook (Appendix A.2). The refinement targets four specific failure modes:

Stylistic awkwardness and disjointed phrasing. MLLMs sometimes produce choppy descriptions with repetitive structures: "There is a robot. It is rusty. It is standing. It is near a table." Human annotators combine such fragments into natural prose: "A rusty robot stands near the table." This is important not merely for aesthetics but because unnatural phrasing can confuse T2I models' text encoders, artificially depressing scores on prompts that are linguistically obscure.
Logical impossibilities. MLLMs can generate spatially contradictory constraints, particularly when multiple relative position statements interact. For example, a prompt might specify "A is left of B, B is left of C, C is left of A," which is geometrically impossible in a linear arrangement. Human annotators identify and correct these contradictions. The paper does not report the frequency of such errors, but their existence underscores why fully automated prompt generation would corrupt the benchmark's validity.
Obscure or ambiguous vocabulary. The benchmark's goal is to test spatial reasoning, not lexical knowledge. Human annotators replace rare or ambiguous words with common synonyms: "vermilion" → "bright red," "bustling" → "crowded," "tranquil" → "calm," "futuristic" → removed or replaced. The guiding principle is that if a word is likely to be out-of-vocabulary for standard CLIP text encoders, or if its meaning depends on cultural context rather than visual properties, it biases the test against models with smaller vocabularies.
Fairness and classification checks. Each prompt is verified to ensure it covers all 10 spatial sub-domains without omission and that the scene description remains recognizable as belonging to its intended scene category.

The total human effort is reported as "over 168 person-hours" across the prompt and QA refinement stages combined. All five annotators first complete a calibration phase using the guidebook to maintain high inter-annotator agreement, though the paper does not report a quantitative inter-annotator agreement score.

The refinement process yields the final set of 1,230 high-quality prompts. Each prompt is a self-contained, fluent English sentence or short paragraph that a human would recognize as a natural scene description, but which has been engineered to contain exactly 10 distinct, independently evaluable spatial constraints.

Omni-Dimensional QA Generation and Evaluation Protocol (SpatialGenEval Construction Stage 2)

For each of the 1,230 prompts, the benchmark provides exactly 10 multiple-choice questions paired with an automated evaluation protocol that produces per-sub-domain accuracy scores. This is the "omni-dimensional evaluation" principle.

Automated QA Generation

The same Gemini 2.5 Pro model generates the QA pairs. Its input for each prompt includes three components:

The refined prompt text.
The definitions of all 10 spatial sub-domains.
A set of example question-answer pairs demonstrating the expected format and difficulty level.

The MLLM generates all 10 QA pairs in a single pass. Each pair consists of:

A question explicitly targeting one spatial sub-domain.
One ground-truth answer drawn from the prompt.
Three plausible but incorrect options (distractors) designed to test detailed understanding rather than surface-level pattern matching.

The full meta-instruction is in Appendix A.4. The model is instructed to ensure that each question maps to exactly one sub-domain and that the distractors are sufficiently plausible that a model relying on shallow visual heuristics (rather than genuine spatial understanding) would select them.

Table 9 in Appendix A.1 provides representative question examples for each sub-domain, illustrating the deliberate escalation in question difficulty. For S2 (Attribute), instead of "What is the color of the cup?", the question asks "What color is the smaller of the two cups, and what color is the larger?"—embedding an attribute query within a comparative frame. For S3 (Position), instead of "Where is the cat?", the question asks "From the perspective of the armchair, where is the bookshelf located?"—demanding relational rather than absolute spatial reasoning. For S6 (Comparison), instead of "Which is larger?", the question asks "Approximately how many times wider is the table than the chair?"—requiring quantitative, not merely ordinal, comparison.

Anti-Leakage Design: Human Validation of Questions

A critical threat to benchmark validity is answer leakage: a question whose phrasing reveals its answer without requiring the generated image. For example, "What is the layout of the leaves that are arranged in a circle?" telegraphs "circle" in the question stem. An evaluator MLLM could answer correctly by parsing the question text, never looking at the image.

Human annotators are explicitly tasked with identifying and eliminating leaked answers. The process (detailed in Appendix A.2) requires them to:

Read each question and determine whether the ground-truth answer can be inferred from the question text alone.
If so, rewrite the question to remove the leaked information. The example given: "What is the layout of the leaves that are arranged in a circle?" is revised to "What is the layout of the leaves in the image?"—preserving the spatial query while removing the answer from the prompt.

This is a labor-intensive quality control step, but without it, the benchmark would measure MLLM reading comprehension rather than T2I spatial performance. The paper also notes (Section 2.1, Principle 3) that the text-to-image prompt itself is not provided to the evaluator MLLM, preventing the judge from answering by matching question text against the generation prompt.

The "E: None" Refusal Option

After human validation, a fifth option "E: None" is programmatically appended to every question. This addresses a specific failure mode: when a T2I model generates an image that contains none of the four suggested options (A-D), a forced-choice evaluator would be compelled to select an incorrect answer, producing random noise in accuracy scores. The "None" option allows the evaluator to refuse when the generated image is faithful to none of the provided alternatives.

This is formally stricter than a standard 4-choice format. For a question to be scored correct, the evaluator must select the ground-truth option from A-D. Selecting "E: None" is always incorrect (since the question was designed so that the ground truth is among A-D if the image correctly follows the prompt). However, selecting "E: None" when the image genuinely does not match any option provides a more informative signal than random guessing—it indicates a generation failure rather than an evaluation failure.

The paper's Table 12 in Appendix A.9.2 validates that the MLLM evaluator does use "E: None" appropriately: when evaluated on questions without any image input (text-only evaluation), the overall accuracy drops to 16.9%, below the random-guessing baseline of 19.8% (which would be 20% for uniform 5-choice, but the paper reports 19.8% as the empirical random baseline, possibly due to the specific distribution of correct answers across A-E). This below-chance performance indicates that the evaluator often selects "E: None" when lacking visual evidence, rather than guessing among A-D.

MLLM-as-Judge Evaluation Protocol

The evaluation is formulated as a zero-shot, multiple-choice visual question answering task. The evaluator MLLM receives:

A generated image (from one of the 23 T2I models).
Its 10 corresponding multiple-choice questions (one at a time).

The evaluator's instruction (detailed in Appendix A.4, Stage 3) includes an explicit prohibition:

"DO NOT use any external resources like world knowledge."

This discourages the MLLM from answering based on what "should" be in the scene (e.g., "kitchens usually have sinks") rather than what is actually visible in the generated image. The evaluator must ground its answer solely in the image content.

Five-Round Majority Voting

To enhance stability and reduce randomness in the MLLM's judgments, the paper implements a 5-round voting mechanism inspired by self-consistency (Wang et al., 2022). For each image-question pair, the evaluator is queried 5 times, and a response is considered correct only if the MLLM selects the ground-truth answer in at least 4 out of 5 rounds.

This is a conservative threshold. Requiring 4/5 consensus means that even moderately confident model outputs (e.g., 3/5) are scored as incorrect. The paper argues this reduces the impact of "random inference noise" typical in MLLM responses, where the same model may produce different answers to identical queries due to stochastic decoding.

The final score for each T2I model is reported as accuracy on each of the 10 spatial sub-domains: the percentage of questions within that sub-domain where the MLLM achieved at least 4/5 agreement with the ground truth.

Primary Evaluator: Qwen2.5-VL-72B

The paper selects Qwen2.5-VL-72B as the primary evaluator for two stated reasons:

Reproducibility: it is an open-source model with publicly available weights, meaning the entire evaluation pipeline can be replicated without dependence on commercial API access. This is a deliberate choice in a landscape where many benchmark papers rely exclusively on GPT-4V, creating a reproducibility bottleneck when API pricing, availability, or model versions change.
State-of-the-art capability: the paper cites Qwen2.5-VL-72B's strong multimodal understanding performance, aligning with independent evaluations.

For cross-validation, the paper also conducts evaluation using GPT-4o (a closed-source model), and Table 4 (Section 3.2) reports that "both judges produce similar model rankings and numbers." The main leaderboard (Table 2) uses Qwen2.5-VL-72B, and the full GPT-4o leaderboard is provided in Appendix A.3 (Table 10).

Human Alignment Study

To validate that the MLLM evaluators produce judgments consistent with human assessment, the paper conducts a human alignment study (reported in Section 3.2 and Table 5). The methodology:

200 images are randomly sampled from Qwen-Image outputs (8 from each of 25 scenes).
Five human annotators work independently, selecting the best option for each of the 10 questions per image, based solely on the image content.
Annotators do not have access to the original text-to-image prompt (preventing leakage).
Alignment is measured using balanced accuracy (Brodersen et al., 2010), which accounts for class imbalance by averaging per-class recall, following the methodology of Li et al. (2025).

Table 5 reports balanced accuracy for three evaluators: Qwen2.5-VL-72B, GPT-4o, and Gemini-2.5-Pro. The key findings:

All three MLLMs align well with human judgment overall.
Gemini-2.5-Pro performs best across the board.
Alignment correlates with sub-domain difficulty: higher on simpler sub-domains (Spatial Foundation/Perception/Interaction) and lower on Spatial Reasoning sub-domains. Despite this, "the alignment score still nears 80%" even on the hardest sub-domains, validating MLLM effectiveness as evaluators.

This is a crucial finding because it addresses a natural concern: if the evaluator MLLM itself struggles with spatial reasoning, then benchmark scores reflect evaluator weakness rather than generator weakness. The high human-MLLM alignment on even the challenging sub-domains suggests that the evaluation questions are sufficiently simple relative to MLLM capabilities, even though the generation task that produced the images being evaluated is complex. This decoupling—complex generation, simple verification—is a deliberate design choice (explicitly argued in Appendix A.9.3).

Evaluation Infrastructure

The paper reports the evaluation infrastructure for transparency and reproducibility:

Hardware: 8× H20 GPUs.
Serving framework: vLLM (Kwon et al., 2023) for efficient LLM inference.
Throughput: approximately 1.8 seconds per image for evaluation, completing all 1,230 generated images in about 40 minutes.

This is a practical consideration: evaluating 23 models × 1,230 prompts × 5 rounds × 10 questions creates a large computational burden, and the vLLM deployment is designed to make the full evaluation feasible without requiring prohibitive compute resources.

SpatialT2I: From Evaluation Benchmark to Fine-Tuning Dataset

The paper's second major contribution is the SpatialT2I dataset, which adapts the SpatialGenEval construction principles into a supervised fine-tuning pipeline. This demonstrates that the benchmark is not merely diagnostic but also prescriptive—the same information-dense, spatially-structured data format that enables evaluation also drives improvement when used for training.

Stage 1: Generating a Fresh Set of Prompts

To prevent data contamination (where models are evaluated on prompts they were fine-tuned on), SpatialT2I is constructed from a completely separate set of 1,230 prompts generated via the identical procedure described in Section 2.3. These prompts have no overlap with the evaluation benchmark. The generation follows the same principles: select a scene from the 25 categories, feed it into Gemini 2.5 Pro with the 10 sub-domain definitions, and obtain an information-dense prompt covering all spatial constraints. The same human-in-the-loop refinement process (though presumably less intensive, since these prompts are inputs to T2I models rather than evaluation targets) is applied.

Stage 2: Curating Generated Images from Top-Performing Models

The 1,230 SpatialT2I prompts are sent to 14 top-performing open-source T2I models—specifically, models that achieved average scores above 50% on the SpatialGenEval benchmark (as reported in Table 2 and Appendix A.3). This is a deliberate filtering decision: including images from models that score below 50% would inject predominantly failed spatial renderings into the training data, teaching the fine-tuned model to replicate spatial errors.

Each of the 14 models generates images from each prompt. The "Design" scene category (130 prompts, 10.6% of the total) is excluded from the dataset due to "low image quality"—likely meaning that even top models produce unacceptably poor renderings in abstract/artistic categories like Cartoon and Advertisement, which would degrade training data quality.

This yields: $(1230 - 130) \times 14 = 15,400$ text-image pairs across 22 scenes.

The paper reports that this pipeline used 14 models, but the specific identities of all 14 are not exhaustively listed in the main text. The selection is implied by Table 2 and Table 10: models scoring above 50% on either evaluator. The context suggests candidates include Qwen-Image, FLUX.1-krea, FLUX.1-dev, SANA 1.5, SD-3-M, SD-3.5-L, PixArt-sigma, Bagel, Infinity, NextStep-1, OmniGen2, UniWorld-V1, Janus-Pro, and Show-o—roughly 14 open-source models meeting the threshold, though the exact list should be inferred from the leaderboard.

Stage 3: Prompt Rewriting for Text-Image Alignment

This is the most critical stage that distinguishes SpatialT2I from naive data collection. When a T2I model generates an image from a prompt with 10 spatial constraints, the image typically satisfies some constraints and violates others. Using the original prompt as the ground-truth caption for this image would teach the fine-tuned model that the generated spatial arrangement is correct—reinforcing the very errors the benchmark reveals.

The paper's solution is prompt rewriting: a powerful MLLM (GPT-4o) takes each generated image and its original prompt, then produces a mildly rewritten version that accurately describes what is actually in the image while preserving information density and all spatial dimensions.

The rewriting instruction (Appendix A.5) specifies that the rewritten prompt must:

Accurately describe the generated image (ensuring text-image consistency).
Preserve the original prompt's information density (not simplify or remove spatial constraints).
Maintain all 10 dimensions of spatial intelligence.

The paper's Table 11 (Appendix A.7) provides a validation experiment: for four models (SD-3.5-L, OmniGen2, UniWorld-V1, Qwen-Image), the prompts are rewritten without changing the model weights (i.e., the rewritten prompts are fed to the same models at inference time, not used for training). The results show consistent improvements: SD-3.5-L +2.3% overall, OmniGen2 +2.1%, UniWorld-V1 +1.7%, Qwen-Image +1.1%. These gains are concentrated in explicit spatial relationship sub-domains (Position +4.7% for SD-3.5-L, Comparison +4.5% for OmniGen2) and minimal for implicit visual reasoning (Occlusion, Orientation). This validates that rewriting clarifies textual ambiguity but cannot compensate for the generator's core 3D reasoning deficits—an important boundary on what data-centric approaches can achieve.

Stage 4: Supervised Fine-Tuning

The 15,400 text-image pairs are used to fine-tune three architecturally distinct base models:

Stable Diffusion-XL (diffusion-based, 3.5B parameters).
UniWorld-V1 (unified multimodal, 12B parameters).
OmniGen2 (autoregressive, 4B parameters).

The paper states that fine-tuning uses "their official settings," meaning the training recipes recommended by each model's authors for standard supervised fine-tuning—hyperparameters are not independently re-tuned for SpatialT2I. This is a deliberate choice to isolate the effect of the data rather than conflating it with hyperparameter optimization.

The quantitative results (Table 6 in Section 4) show consistent gains across all three models:

SD-XL: 41.2% → 45.4% (+4.2 percentage points).
UniWorld-V1: 54.2% → 59.9% (+5.7 percentage points).
OmniGen2: 56.4% → 60.8% (+4.4 percentage points).

The gains are not uniform across sub-domains. For SD-XL, the largest absolute improvements appear in Position (+6.6%), Comparison (+3.9%), and Occlusion (+4.8%). For UniWorld-V1, the gains concentrate in Position (+8.4%), Layout (+10.4%), and Comparison (+8.5%). For OmniGen2, the pattern is similar: Position (+6.2%), Comparison (+9.3%), Occlusion (+6.9%). This distribution aligns with the paper's diagnosis that spatial reasoning (S6, S7, S8) is the primary bottleneck—these are precisely the dimensions that SpatialT2I fine-tuning most improves.

Critically, even after fine-tuning, scores on the hardest sub-domains remain low: Occlusion improves from ~23% to ~32% across the three models; Comparison improves from ~19-26% to ~23-35%. These are meaningful relative gains but leave substantial room for improvement. The paper does not claim to have "solved" spatial reasoning; it claims to have demonstrated that data-centric improvement is possible and that information-dense prompts are the right intervention target.

Ablation: Data Quality Matters More Than Any Single Source

The paper conducts an ablation study (Figure 7, Section 4) to test whether the performance gains are driven by the quantity of fine-tuning data or the quality of the models that generated it. Three subsets of SpatialT2I are selected based on the performance of the generating model on SpatialGenEval:

Low-performing subset: images generated by UniPic-v2 (overall score 54.3%).
Medium-performing subset: images generated by Bagel (overall score 57.0%).
High-performing subset: images generated by Qwen-Image (overall score 60.6%).

Each subset contains 1,100 text-image pairs.

The results (Figure 7, top) show that fine-tuning on any subset yields performance gains, but higher-scoring model subsets contribute more significantly. This makes intuitive sense: higher-performing models produce images that more faithfully follow the spatial constraints in the prompts, so the rewritten captions for those images contain fewer spatial errors. Fine-tuning on data from better models means the model sees more examples where the text-image correspondence is genuinely correct rather than incorrectly rewritten.

The paper also reports a data scaling trend (Figure 7, bottom): performance consistently improves when training data is increased from 0% to 100% by progressively adding higher-scoring subsets. This suggests that further scaling of SpatialT2I-style data—generating more prompts, using more (and better) models for image generation, and expanding to additional scenes—would yield additional gains. The trend does not appear to plateau at the current dataset size, though the authors acknowledge this is tentative without larger-scale experiments.

Design Rationale Summary

The SpatialT2I construction pipeline reflects several deliberate design choices that are worth making explicit:

No overlap with evaluation data. Using a fresh prompt set prevents inflated fine-tuning scores from memorization rather than generalization.
Model-quality filtering. Using only models scoring above 50% ensures the training images are at least partially correct, preventing the model from learning to reproduce systematic spatial errors. The ablation confirms this matters quantitatively.
Prompt rewriting rather than raw captioning. Directly captioning the generated image (without reference to the original prompt) would produce accurate but simplified captions that lack the information density of the original. Rewriting preserves the dense spatial structure while correcting misalignments.
Cross-architecture validation. Testing on diffusion, autoregressive, and unified models demonstrates that the data-centric approach is architecture-agnostic—spatial intelligence deficits are a data problem, not solely an architecture problem.
Scene diversity concreteness. The 25 real-world scenes are not arbitrary; they are chosen to span indoor/outdoor, natural/human-made, physical/abstract domains, ensuring the dataset captures spatial reasoning in diverse contexts.

Summary of Design Choices and Their Justifications

All 10 constraints per prompt rather than one per prompt: enables testing of simultaneous constraint handling, which is the real-world demand and where models demonstrably fail.
Approximately 60-word target: balances information density against 77-token CLIP encoder context windows, avoiding unfair penalty for CLIP-based architectures.
Gemini 2.5 Pro for generation: selected over GPT-4o and Qwen2.5-VL-72B based on superior creative diversity (lower within-scene prompt similarity), confirmed by both human judgment and quantitative embedding similarity.
Five-round 4/5 majority voting: conservative threshold that requires stable evaluator judgments, reducing noise from stochastic MLLM decoding.
"E: None" as the fifth option: allows evaluator refusal when the generated image is unfaithful to all suggested options, preventing forced-choice noise.
Human-in-the-loop for both prompts and QA pairs: catches logical impossibilities, answer leakage, obscure vocabulary, and stylistic awkwardness that would corrupt benchmark validity if left to automated generation alone.
Qwen2.5-VL-72B as primary evaluator: open-source for reproducibility, validated against both GPT-4o and human annotators to ensure judgment quality.
Separate prompt set for SpatialT2I: prevents training-evaluation contamination.
Model-quality filtering for SpatialT2I training data: ensures training images are at least partially spatially correct, confirmed by ablation showing higher-quality sources yield larger gains.
Prompt rewriting, not raw captioning: preserves the information-dense spatial structure while correcting text-image misalignment.
Cross-architecture fine-tuning validation: demonstrates that spatial intelligence deficits are addressable through data, not exclusively through architecture changes.

4. Key Insights and Innovations

Innovation 1: A Hierarchical Taxonomy That Transforms Spatial Intelligence from an Implicit Intuition into a Diagnosable, Decomposable Capability

The paper's most fundamental intellectual move is not the benchmark itself but the conceptual armature it builds for thinking about spatial intelligence. Prior to SpatialGenEval, the field operated with a vague, undifferentiated notion of "spatial understanding" in T2I models. A model could be described as "bad at spatial relationships" or "sometimes confused about layout," but there was no shared vocabulary for distinguishing whether a failure stemmed from misperceiving an object's position, misreasoning about a comparative relationship, or failing to render occlusion—let alone a systematic framework linking these failures to a cognitive hierarchy.

The paper constructs this vocabulary through a four-domain, ten-sub-domain taxonomy that imposes a developmental structure on spatial intelligence: Spatial Foundation (object identity and attribute binding) → Spatial Perception (geometric arrangement) → Spatial Reasoning (relational inference, quantitative comparison, 3D depth) → Spatial Interaction (dynamic motion and causality). This progression is not merely a classification scheme; it embodies a causal theory about what spatial intelligence requires. The dependency structure is inherent: a model cannot meaningfully be evaluated on causal interaction (S10) if it fails to generate the relevant objects (S1) and perceive their positions (S3). The taxonomy makes this dependency explicit, transforming spatial intelligence from a monolithic judgment into a debuggable capability profile.

Why this is more than taxonomy engineering: The paper grounds each level in cognitive science literature—spatial cognition research by Malanchini et al. (2020), embodied intelligence frameworks (Gupta et al., 2021; Ruan et al., 2025), and recent computer vision benchmarks for MLLM spatial understanding (Yang et al., 2025a; Stogiannidis et al., 2025). But its contribution is not to import cognitive science; it is to translate cognitive theory into an operational evaluation framework where each sub-domain maps to a specific, automatically evaluable question type. The innovation is the bridge: cognitive decomposition on one side, automated multiple-choice evaluation on the other, with the taxonomy serving as the load-bearing connection.

The evidence for this framework's diagnostic power appears throughout the results. The leaderboard (Table 2) does not just rank models; it reveals where in the hierarchy each model's performance collapses. Qwen-Image scores 77.2% on Object Attribute (S2)—basic attribute binding works—but 28.6% on Spatial Comparison (S6) and 30.8% on Spatial Occlusion (S8). The taxonomy makes these failures legible: the model understands what objects are but cannot reason about how they relate quantitatively or how they layer in depth. This diagnostic precision is impossible without the taxonomy, and it is what transforms the paper from a benchmark report into a conceptual contribution.

Equally important is what the taxonomy reveals through its structure: a non-monotonic difficulty gradient. The failure analysis (Figure 5) shows that Relational Reasoning errors exceed 35% for all models and all scene types, dominating over Basic Composition errors (which are typically below 10%). This is not a smooth degradation of performance across sub-domains; it is a qualitative collapse at the transition from perception (S3-S5) to reasoning (S6-S8). The taxonomy surfaces this as a phase transition—a point in the hierarchy where fundamentally different cognitive machinery is required and current architectures lack it. This is a finding that would be invisible in aggregate scores and that no prior benchmark could diagnose because none systematically decomposed spatial intelligence along a cognitive progression.

Innovation 2: The Information-Dense, Multi-Constraint Prompt as the Evaluative and Training Primitive

The paper's second conceptual contribution is the elevation of the information-dense prompt—a single, fluent scene description integrating 10 simultaneous spatial constraints—as the atomic unit of both evaluation and training. This is a deliberate departure from the dominant paradigm in T2I benchmarking, where prompts target one or two spatial relationships in isolation (GenEval, T2I-CompBench, DPG-Bench all follow this pattern; see Table 1 for explicit coverage comparisons). Prior benchmarks essentially test: "Can the model handle spatial constraint C in isolation?" The paper's key insight is that this is the wrong question. The right question is: "Can the model handle constraints C1 through C10 simultaneously, where they interact and constrain each other?"

This reframing matters because it exposes a combinatorial failure mode that single-constraint benchmarks systematically hide. Consider a model that correctly places "A left of B" in 80% of cases and "A touching B" in 80% of cases. A single-constraint benchmark reports 80% on each, implying strong spatial performance. But when these constraints are combined—"A is left of B and touching B"—the model must satisfy both simultaneously, and the probability of joint success may be far below 80%. The information-dense prompt forces this joint evaluation, testing whether the model maintains internal consistency across constraints that geometrically interact.

The paper's evidence that this matters is in the gap between high Spatial Foundation scores (Qwen-Image: 77.2% on S2) and low Spatial Reasoning scores (Qwen-Image: 28.6% on S6). The model is not failing because it cannot understand any single constraint; it is failing because it cannot maintain consistency across the full set. The constraints in an information-dense prompt are not independent add-ons—comparing two objects' sizes interacts with their relative positions, which interacts with occlusion if one partially hides the other, which interacts with proximity if they must be adjacent. The dense prompt reveals whether the model possesses an integrated spatial world model or merely a bag of independently activated spatial features.

This insight also produces a methodological innovation beyond evaluation: the information-dense prompt as a training data primitive. The SpatialT2I dataset is constructed not by collecting diverse captions but by insisting that every training pair instantiates all 10 spatial sub-domains simultaneously. The fine-tuning gains (+4.2% to +5.7% across architectures) provide evidence that the capacity for joint constraint handling is partially trainable—the model can learn to maintain spatial consistency across dimensions if it sees enough diverse examples of such consistency. But the fact that post-fine-tuning scores remain low on the hardest sub-domains (Comparison and Occlusion still below 35% for most models) reveals a deeper truth: some aspects of joint spatial reasoning resist data-centric approaches, likely requiring architectural mechanisms for explicit spatial computation.

The paper's ~60-word target is itself a non-trivial design choice that encodes a theoretical position. By deliberately balancing against the 77-token CLIP context window, the paper makes a methodological claim: an evaluation benchmark must be fair to current architectures while probing their limits, not constructed at an arbitrary prompt length that artificially advantages or disadvantages specific model families. This is a practical design decision that reflects a deeper methodological commitment—evaluation should diagnose, not artificially create failure modes through mismatched prompt formats.

Innovation 3: Decoupling Generation Complexity from Evaluation Complexity to Enable Reliable Automated Benchmarking

This paper solves a subtle but fundamental infrastructure problem that has limited the scale and reliability of T2I evaluation: how do you reliably evaluate complex spatial outputs when the evaluator (an MLLM) itself has imperfect spatial understanding? Prior approaches have relied on detection models (limited to object presence and coarse bounding boxes), human evaluation (expensive, slow, not reproducible), or MLLM judgment with unknown reliability. The paper's solution is a complexity decoupling strategy: make the generation task maximally complex (10 simultaneous spatial constraints) while making the evaluation task a series of simple, visually checkable multiple-choice questions that are well within current MLLM capabilities.

The key insight is that generation and evaluation do not need to operate at the same cognitive level. A human can reliably check whether "the vase is partially obscuring the book" is true in a generated image even if they could not compose the scene from scratch. The paper argues and empirically validates that the same is true for MLLMs: Qwen2.5-VL-72B can verify spatial relationships it could not generate. The human alignment study (Table 5, Section 3.2) provides the numerical backbone: balanced accuracy above 80% on even the hardest sub-domains, with the scores dropping slightly but not catastrophically on Spatial Reasoning sub-domains where MLLMs themselves are least capable.

This is a methodological innovation with implications beyond this paper. It demonstrates that automated evaluation of complex generative outputs is tractable when the evaluation task is structured to be simpler than the generation task. The specific mechanisms that make this work—multiple-choice format with distractors, "E: None" refusal option, 5-round majority voting, explicit prohibition on external knowledge—are individually incremental, but their systematic combination into a validated evaluation protocol is not. The paper doesn't just claim MLLM evaluation works; it provides a replicable recipe and validates it against human judgment.

The cross-evaluator consistency (Table 4: Qwen2.5-VL-72B and GPT-4o produce "similar model rankings and numbers") provides further evidence that the evaluation protocol, not any specific MLLM's quirks, determines reliability. When two independently trained MLLMs with different architectures and training data agree on model rankings, the signal is the evaluation structure, not the evaluator.

This innovation also carries a cautionary finding embedded in the human alignment results: alignment correlates with sub-domain difficulty. Evaluators are most reliable on Spatial Foundation questions and least reliable on Spatial Reasoning questions. This means the benchmark's most diagnostically interesting measurements—on Comparison, Proximity, and Occlusion where models fail most dramatically—are also the measurements with the highest inherent evaluator noise. The paper acknowledges this implicitly through the alignment scores but does not fully explore the implications: a 2-3% performance difference on Spatial Occlusion between two models may be within evaluator noise margins, even if the protocol is the best available. This boundary on reliability is itself a useful finding for anyone building on the benchmark.

Innovation 4: Establishing Verifiable Boundaries on What Data-Centric Approaches Can and Cannot Fix for Spatial Intelligence

The fine-tuning experiment using SpatialT2I is not merely a demonstration that "more data helps." It is a structured experiment that reveals where the spatial intelligence bottleneck changes from a data problem to an architecture problem. The paper could have simply reported aggregate gains of +4.2% to +5.7% and claimed success. Instead, it drills into per-sub-domain improvements and finds a distinctive pattern that amounts to a fundamental finding about the nature of spatial reasoning deficits.

The gains from SpatialT2I fine-tuning are concentrated in sub-domains that benefit from clarified textual descriptions: Spatial Position (+6.2 to +8.4 percentage points) and Spatial Layout (+5.0 to +10.4 points). These are domains where the mapping from language to spatial arrangement is direct and where more training examples of correct language-to-arrangement mappings plausibly improve performance. The rewritten prompts in SpatialT2I that fix text-image misalignments teach the model precisely this mapping.

In contrast, the gains on domains requiring implicit 3D understanding are smaller: Occlusion improves from 22.4% to 27.2% on SD-XL, from 26.8% to 31.6% on UniWorld-V1, and from 27.3% to 34.2% on OmniGen2. The absolute levels remain well below 35% even after fine-tuning. Comparison improves comparably (19.1% → 23.0% for SD-XL; 26.1% → 34.6% for UniWorld-V1; 26.0% → 35.3% for OmniGen2), but again remains a primary bottleneck. This asymmetry is informative: it suggests that explicit spatial relationships (position, layout) have a strong mapping to text conditioning signals that training data can strengthen, while implicit spatial relationships (occlusion, quantitative comparison) require a form of geometric reasoning that supervised fine-tuning on text-image pairs cannot sufficiently instill.

The prompt rewriting ablation (Table 11, Appendix A.7) independently corroborates this boundary. When prompts are rewritten to clarify spatial language without changing model weights, the gains again concentrate on explicit relationships: +4.7% in Position for SD-3.5-L, +4.5% in Comparison for OmniGen2. Occlusion shows negligible improvement from rewriting alone (+0.2% for SD-3.5-L, +0.4% for OmniGen2). This pattern—rewriting helps where language disambiguation helps, fails where geometric reasoning is required—establishes that the occlusion problem is not a text understanding problem. Models understand that occlusion is requested; they cannot render it correctly because they lack the internal spatial representation to reason about depth ordering and viewpoint-dependent occlusion boundaries.

The most incisive version of this finding is the data scaling trend (Figure 7, bottom). Performance improves as more and better training data is added, and does not appear to plateau at the current dataset size (15,400 pairs). This suggests that data-centric approaches have not yet hit diminishing returns for spatial intelligence—scaling SpatialT2I by an order of magnitude could yield substantially larger gains. But the differential improvement across sub-domains implies that even unlimited training data will not close the occlusion and comparison gaps if the architecture lacks mechanisms for 3D-consistent spatial composition.

This constitutes a negative result of theoretical significance: the paper demonstrates what cannot be achieved through data alone, and in doing so, identifies the precise capabilities that require architectural innovation. This is more valuable than a simple "more data helps" finding because it redirects research effort from scaling existing approaches toward developing new ones for the hardest sub-domains.

5. Experimental Analysis

Evaluation Methodology

Dataset. The SpatialGenEval benchmark consists of 1,230 information-dense prompts spanning 25 real-world scenes across five categories (Nature 28.5%, Outdoor 32.5%, Indoor 16.3%, Human 12.2%, Design 10.6%). Each prompt is paired with 10 multiple-choice question-answer pairs targeting distinct spatial sub-domains, yielding 12,300 total QA pairs. The evaluation uses all 1,230 prompts without a separate train/validation/test split since the benchmark is used for zero-shot inference assessment, not model training. Scene selection is detailed in Appendix A.1, with full statistics in Table 7.
Base models. The paper evaluates 23 text-to-image models spanning four architectural categories: (1) Diffusion models — SD-1.5, PixArt-alpha, SD-XL, Playground-v2.5, PixArt-sigma, SD-3-M, SD-3.5-L, SANA 1.5, FLUX.1-dev, FLUX.1-krea, and Qwen-Image; (2) Autoregressive models — OmniGen2, NextStep-1, Infinity; (3) Unified multimodal models — Janus-Pro, Show-o, UniWorld-V1, UniPic-v2, Bagel; (4) Closed-source models — DALL-E-3, GPT-Image-1, Nano Banana, Seed Dream 4.0. Model sizes range from 0.6B (PixArt-alpha) to 20B (Qwen-Image). The selection is motivated by covering diverse architectures, scales (from small academic models to large production systems), and accessibility levels (open-source checkpoints vs. commercial APIs), ensuring the benchmark captures the full landscape of contemporary T2I capability. Table 8 in Appendix A.1 provides release dates, resolutions, and source URLs for all models.
Metrics. The primary metric is sub-domain accuracy (%), computed as the percentage of questions within each of the 10 spatial sub-domains where the MLLM evaluator selects the ground-truth answer in at least 4 out of 5 evaluation rounds. The overall score for a model is the average accuracy across all 10 sub-domains and all 1,230 prompts. The paper reports per-sub-domain accuracy in the leaderboard (Table 2) and per-domain aggregate trends in the error analysis (Figure 5). The 5-round majority voting mechanism (based on self-consistency, Wang et al. 2022) requires ≥4/5 consensus for a response to count as correct; 3/5 or lower is scored as incorrect even if the model occasionally selects the right answer.
Baselines. The paper does not evaluate task-specific baselines (e.g., spatial-aware T2I variants) against the benchmark. Rather, the benchmark itself serves as a baseline comparison framework where the primary reference point is random selection accuracy (reported as 19.8% overall in Table 2, row "Random"), computed empirically by measuring the accuracy from uniformly random selection among the 5 answer options. Within the benchmark results, majority voting is used only as a evaluation protocol mechanism (not as a T2I generation strategy). The paper compares model rankings on SpatialGenEval against meta-rankings on four existing benchmarks — GenEval (Ghosh et al., 2023), DPG-Bench (Hu et al., 2024a), Wise (Niu et al., 2025), and TIIF-Bench (Wei et al., 2025) — to establish convergent validity, as reported in Table 3.
Generation budget / compute accounting. The evaluation budget is measured in number of prompts (1,230) and questions per prompt (10), totaling 12,300 image-question pairs per model. Each T2I model generates one image per prompt at its default resolution (varies by model: SD-1.5 at 1024 × 1024, Janus-Pro at 384 × 384, Qwen-Image at 1328 × 1328; see Table 8 for all resolutions). The paper does not standardize image resolution across models, meaning resolution differences could theoretically affect evaluator accuracy (higher resolution may reveal finer spatial detail). Generation uses each model's default inference settings; no test-time compute scaling, beam search, or iterative refinement is applied, as the benchmark targets zero-shot spatial capability assessment. The evaluation pipeline runs on 8× H20 GPUs using vLLM, processing approximately 1.8 seconds per image and completing all 1,230 images in roughly 40 minutes (Section 3.1).
Cross-validation / statistical protocol. The paper does not employ cross-validation for model evaluation since no model training or strategy selection occurs on the benchmark prompts. For the SpatialT2I fine-tuning experiments, the training and evaluation prompt sets are completely disjoint (Section 4, Stage 1: "SpatialT2I is constructed separately and has no overlap with our evaluation benchmark"), preventing data contamination. The 5-round majority voting serves as a statistical stabilization mechanism, reducing the impact of stochastic MLLM decoding on per-question accuracy. The human alignment study (Table 5, Section 3.2) uses balanced accuracy (Brodersen et al., 2010) to account for class imbalance in the human annotation process, with five independent annotators evaluating 200 randomly sampled images (8 per scene, 25 scenes).

Main Quantitative Results

Overall Leaderboard and Model Rankings

The full leaderboard (Table 2) evaluates 23 models using Qwen2.5-VL-72B as the primary evaluator, with a parallel evaluation using GPT-4o reported in Appendix A.3 (Table 10). The headline finding is that no model exceeds the 65% threshold on overall spatial intelligence, and the highest score (Seed Dream 4.0 at 62.7%) is only marginally above what the paper calls the "passing threshold" of 60%.

The top five models by overall score are:

Seed Dream 4.0: 62.7% (closed-source)
Nano Banana: 61.7% (closed-source)
Qwen-Image: 60.6% (open-source)
GPT-Image-1: 60.5% (closed-source)
FLUX.1-krea: 58.5% (open-source)

The gap between the best open-source model (Qwen-Image, 60.6%) and the best closed-source model (Seed Dream 4.0, 62.7%) is approximately 2.1 percentage points, which the paper interprets as evidence that "open-source models are catching up to closed-source ones." However, the gap widens on higher-order sub-domains: for Spatial Occlusion (S8), Seed Dream 4.0 scores 33.8% versus Qwen-Image's 30.8% (a 3.0 point gap); for Spatial Comparison (S6), Seed Dream 4.0 scores 32.1% versus 28.6% (a 3.5 point gap).

The random selection baseline is 19.8% overall. The oldest and simplest model evaluated, SD-1.5 (2021, 0.86B parameters), scores 28.5%, only ~9 points above random, establishing a lower bound on spatial intelligence for basic diffusion architectures. The progression from SD-1.5 (28.5%) through SD-XL (41.2%) to SD-3-M (54.6%) and SD-3.5-L (54.0%) demonstrates steady improvement as model scale and architecture evolve, though the jump from SD-3-M to SD-3.5-L shows minimal gain (+0.6 points overall, despite scaling from 2B to 8B parameters), suggesting diminishing returns from scale alone.

The Performance Gap Between Spatial Foundation and Spatial Reasoning

The most striking pattern across all models is the imbalanced performance between basic object generation and higher-order spatial reasoning, visible in the per-column breakdown of Table 2.

For Spatial Foundation (S1, S2), top models perform strongly:

Qwen-Image: Object Category 61.0%, Object Attribute 77.2%
Bagel: Object Category 55.3%, Object Attribute 73.7%
FLUX.1-krea: Object Category 58.0%, Object Attribute 75.4%

These scores represent the "what" capability—models can generate specified objects and bind attributes correctly with reliability exceeding 70% for attribute binding.

For Spatial Perception (S3-S5), performance remains moderate-to-strong:

S3 (Position): Qwen-Image 55.6%, FLUX.1-dev 50.0%, Seed Dream 4.0 57.2%
S4 (Orientation): Qwen-Image 56.7%, FLUX.1-dev 55.5%, Seed Dream 4.0 58.9%
S5 (Layout): Qwen-Image 69.7%, FLUX.1-dev 66.7%, Seed Dream 4.0 70.1%

Layout (S5) scores are notably higher than Position (S3) and Orientation (S4) for most models. This may reflect that layout patterns (linear sequences, circular arrangements) are easier to specify and evaluate than fine-grained absolute or relative positioning, though the paper does not analyze this discrepancy explicitly.

For Spatial Reasoning (S6-S8), performance collapses across all models:

S6 (Comparison): Qwen-Image 28.6%, FLUX.1-krea 28.3%, Seed Dream 4.0 32.1%
S7 (Proximity): Qwen-Image 67.7%, FLUX.1-krea 66.7%, Seed Dream 4.0 68.3%
S8 (Occlusion): Qwen-Image 30.8%, FLUX.1-krea 28.0%, Seed Dream 4.0 33.8%

The critical finding is that Comparison (S6) and Occlusion (S8) scores are only ~8-13 points above the random baseline (20.3% and 19.6% for those sub-domains respectively), indicating that models perform only marginally better than random guessing on quantitative spatial comparison and depth-ordering tasks. Proximity (S7) is an exception within Spatial Reasoning, scoring substantially higher (60-70% range), likely because proximity questions ask about coarser distinctions ("closest to," "far from") that are easier to render and evaluate than precise quantitative comparisons or occlusion boundaries.

For Spatial Interaction (S9, S10), performance is relatively strong:

S9 (Motion): Qwen-Image 78.1%, FLUX.1-krea 76.0%, Seed Dream 4.0 83.0%
S10 (Causal): Qwen-Image 80.2%, FLUX.1-krea 78.8%, Seed Dream 4.0 83.8%

These scores are among the highest across all sub-domains, which is initially counterintuitive given the paper's framing of Spatial Interaction as the "most advanced domain." The paper does not directly explain this paradox. A plausible interpretation is that motion and causal interaction questions are designed to be verifiable through simpler visual heuristics (e.g., "is the dog in a jumping pose?" rather than "is the jump trajectory physically accurate?"), making them easier to evaluate correctly. The human alignment study (Table 5) supports this: alignment is highest on Spatial Interaction sub-domains, suggesting the evaluation questions are easier for MLLMs to answer. Alternatively, T2I models may genuinely handle motion and causality better than quantitative reasoning because training data contains abundant examples of dynamic scenes with recognizable motion patterns, while precise size comparisons and occlusion relationships are rarer and more demanding.

Failure Case Analysis Across Scenes and Skill Levels

Figure 5 provides a detailed breakdown of error types across five scene categories (Nature, Outdoor, Indoor, Human, Design) based on all T2I models evaluated. The analysis classifies errors into four skill levels: Basic Composition (S1-S2), Visual Perception (S3-S5), Relational Reasoning (S6-S8), and Motion Interaction (S9-S10).

The key finding is that Relational Reasoning is the dominant error category across all scene types, with error rates consistently exceeding 35%. For Nature scenes, errors peak at ~28.5% for Visual Perception (the highest perception error rate across scenes) and remain elevated for Relational Reasoning. For Outdoor scenes, Relational Reasoning errors dominate with similar magnitude. In contrast, Basic Composition errors are uniformly low (typically below 10%), and Motion Interaction errors remain below 18% across all scene types.

This distribution confirms that the primary barrier is not a linear progression where each skill level is equally challenging, but rather a qualitative jump in difficulty at the transition from perception to reasoning. The paper interprets this as evidence that "the principal barrier to achieving advanced spatial intelligence is not a linear progression of skills, but a critical weakness in processing relational logic." This is supported by the Model-specific examples in Figure 5 (right panel), which show concrete failure cases from FLUX.1-dev and OmniGen2 where relational reasoning constraints are violated (objects incorrectly positioned relative to each other, comparative size relationships wrong) while basic objects are correctly generated.

Text Encoder Capability as a Key Determinant of Spatial Performance

The paper identifies a consistent pattern linking text encoder quality to spatial intelligence scores, supporting the claim that "a deeper understanding of complex, information-dense prompts is critical to achieve high-fidelity spatial generation."

Models using standard CLIP text encoders perform worst:

SD-1.5 (CLIP): 28.5% overall
PixArt-alpha (CLIP): 38.2% overall

Models enhancing CLIP with T5 encoders show substantial improvement:

SD-3-M (T5): 54.6% overall
SD-3.5-L (T5): 54.0% overall
FLUX.1-dev (T5): 56.5% overall
FLUX.1-krea (T5): 58.5% overall

Models leveraging full LLM-based encoders achieve the highest scores:

Qwen-Image (LLM encoder): 60.6% overall

The pattern is not perfectly monotonic—FLUX.1-krea (T5, 12B) outperforms SD-3.5-L (T5, 8B) despite using similar encoder architecture, suggesting that model capacity and training data also play significant roles—but the encoder quality gradient is the strongest single architectural predictor of overall performance.

This finding has direct implications for the benchmark's validity: the 60-word prompt target was deliberately chosen to be compatible with 77-token CLIP context windows, preventing encoder limitations from unfairly penalizing CLIP-based architectures. Despite this accommodation, CLIP-only models still underperform substantially, confirming that the deficits are not merely textual truncation artifacts but reflect genuine limitations in encoding and composing complex spatial language into visual representations.

Model Scale and Architecture as Two Complementary Pathways

The paper identifies two concurrent trends driving spatial intelligence improvements.

Model scale within diffusion architectures: Within the Stable Diffusion family, performance generally correlates with parameter count:

SD-1.5 (0.86B): 28.5%
SD-XL (3.5B): 41.2%
SD-3-M (2B): 54.6%
SD-3.5-L (8B): 54.0%
Qwen-Image (20B): 60.6%

The jump from SD-1.5 to SD-XL is large (+12.7 points), but further scaling from SD-3-M (2B) to SD-3.5-L (8B) produces negligible gain (+0.6 points), and the leap to Qwen-Image (20B) recovers meaningful improvement (+6.0 points over SD-3.5-L). This non-monotonic pattern suggests scale alone does not guarantee spatial intelligence; the Qwen-Image advantage likely reflects its LLM-based encoder and improved training recipe rather than pure parameter count.

Unified architectures achieve better parameter efficiency: The 7B Bagel model (57.0% overall) outperforms the 12B FLUX.1-dev (56.5%) and approaches the 12B FLUX.1-krea (58.5%), despite having roughly half the parameters. This supports the paper's claim that "unified models demonstrate greater parameter efficiency by integrating understanding and generative abilities." Similarly, the 7B Janus-Pro (50.6%) outperforms the 3.5B SD-XL (41.2%) despite a smaller parameter advantage ratio, suggesting that unified training on both understanding and generation tasks provides inductive biases beneficial for spatial reasoning.

However, the evidence is correlational rather than causal: unified models are trained on different data mixtures with different objectives compared to pure diffusion models, so the parameter efficiency advantage could stem from data scale, training duration, or other confounds rather than architectural unification per se.

Correlation with Existing Benchmarks

Table 3 reports meta-rankings of five models across four established benchmarks alongside SpatialGenEval. The five models compared are SD-XL, PixArt-sigma, SD-3-M, FLUX.1-dev, and Qwen-Image. The meta-ranking (average rank across benchmarks) shows strong agreement with SpatialGenEval rankings: Qwen-Image consistently ranks highest, followed by FLUX.1-dev, with SD-XL and PixArt-sigma ranking lower across all benchmarks.

The paper interprets this alignment as validation that "SpatialGenEval [is] a reliable indicator of a model's overall generative capability." This is a reasonable but limited claim. Strong correlation with existing benchmarks suggests SpatialGenEval is measuring a capability dimension that is correlated with general T2I quality, but it does not demonstrate that SpatialGenEval captures distinctive spatial intelligence information beyond what existing benchmarks measure. A benchmark that perfectly correlated with existing benchmarks would be redundant; the value of SpatialGenEval lies in capturing capabilities that existing benchmarks miss. The paper does not provide a residual analysis showing what SpatialGenEval reveals that the meta-ranking from four benchmarks does not.

Judge-Dependency Analysis

Table 4 compares overall model rankings and absolute scores between Qwen2.5-VL-72B (open-source) and GPT-4o (closed-source) as evaluators for five representative models. The paper states that "both judges produce similar model rankings and numbers," validating "the robustness of our benchmark and the selected evaluator."

Examining the specific numbers would require access to Table 4's contents, which the main text references but does not reproduce numerically. The full GPT-4o leaderboard in Table 10 (Appendix A.3) provides per-model scores that can be compared to the Qwen2.5-VL-72B leaderboard in Table 2. Several notable discrepancies exist:

SD-1.5: 28.5% (Qwen) vs. 24.6% (GPT-4o) — a 3.9 point difference
Janus-Pro: 50.6% (Qwen) vs. 48.0% (GPT-4o) — a 2.6 point difference
FLUX.1-krea: 58.5% (Qwen) vs. 58.0% (GPT-4o) — a 0.5 point difference
Qwen-Image: 60.6% (Qwen) vs. 60.8% (GPT-4o) — a 0.2 point difference
Seed Dream 4.0: 62.7% (Qwen) vs. 62.1% (GPT-4o) — a 0.6 point difference

The absolute score differences are generally small (within 1-4 points) and the ranking order is largely preserved, supporting the paper's robustness claim. However, the larger discrepancy for SD-1.5 (3.9 points) suggests evaluator agreement may degrade for models that produce lower-quality or less photorealistic images, where spatial relationships are harder to discern. The paper does not analyze this interaction between model quality and evaluator agreement.

Human Alignment Study

Table 5 reports balanced accuracy for three MLLM evaluators (Qwen2.5-VL-72B, GPT-4o, Gemini-2.5-Pro) against human judgments across the four spatial domains and overall.

The key numbers:

Overall balanced accuracy: Qwen2.5-VL-72B achieves the reported alignment, with Gemini-2.5-Pro performing best overall.
By domain trend: Alignment is highest on Spatial Interaction (motion and causal questions) and lowest on Spatial Reasoning (comparison, proximity, occlusion). This domain-level variation is expected: Spatial Reasoning questions require the evaluator to make fine-grained judgments about relative sizes, exact distances, and occlusion boundaries, which are inherently more ambiguous and subjective than checking whether an object is present or whether an action is depicted.

The paper interprets this as validation that "current leading MLLMs are suitable and capable in evaluating the questions in our SpatialGenEval benchmark." The alignment scores, while high overall, reveal a systematic limitation: the benchmark's measurements on Spatial Reasoning sub-domains—the very sub-domains where the paper's central finding about bottlenecks is located—are subject to the highest evaluator noise. A model achieving 32% on Spatial Comparison versus 28% may not represent a statistically reliable difference if evaluator alignment with humans in that sub-domain is around 80% (meaning evaluator judgments diverge from humans on ~20% of questions).

Ablation Studies and Robustness Checks

Prompt rewriting as an alternative improvement strategy (Table 11, Appendix A.7): The paper tests whether rewriting prompts to make spatial dimensions more explicit and unambiguous—without any model fine-tuning—can improve spatial intelligence scores. Four models are evaluated with original vs. rewritten prompts: SD-3.5-L (+2.3% overall gain), OmniGen2 (+2.1%), UniWorld-V1 (+1.7%), and Qwen-Image (+1.1%). The gains are concentrated in explicit spatial relationship sub-domains: Position improves by +4.7% for SD-3.5-L and +4.2% for OmniGen2; Comparison improves by +4.5% for OmniGen2 and +3.0% for UniWorld-V1. However, implicit visual reasoning sub-domains show minimal or no improvement: Occlusion improves by only +0.2% for SD-3.5-L and +0.1% for UniWorld-V1. This establishes a clear boundary: prompt disambiguation helps where the bottleneck is language understanding (position, layout) but does not help where the bottleneck is visual reasoning (occlusion, orientation). The Qwen-Image gains are smallest (+1.1%), likely because its LLM-based encoder already handles complex language well, leaving less room for improvement from clarification.

Evaluator without image input (Table 12, Appendix A.9.2): To test whether the MLLM evaluator relies on visual evidence rather than answering from textual cues or prior knowledge, the paper evaluates questions without providing any image. Overall accuracy drops to 16.9%, below the random-guessing baseline of 19.8%. This below-random performance indicates that the evaluator frequently selects "E: None" when lacking visual evidence, rather than guessing among A-D options. Sub-domain breakdown reveals that some biases exist (e.g., Spatial Interaction scores 28.1% without images versus 20.0% random baseline), suggesting the evaluator may possess some prior knowledge about common motion and causal patterns that weakly influences its selections. However, the effect is small and does not approach the 60-80% range seen when images are provided, validating that the evaluation is predominantly visually grounded.

Data quality ablation in SpatialT2I fine-tuning (Figure 7, top): Three subsets of the SpatialT2I dataset are selected based on the generating model's SpatialGenEval score: low-performing (UniPic-v2, 54.3%), medium-performing (Bagel, 57.0%), and high-performing (Qwen-Image, 60.6%). Each subset contains 1,100 text-image pairs (roughly the scale of the full evaluation benchmark). Fine-tuning SD-XL and OmniGen2 on each subset reveals that all subsets produce performance gains over the base model, but higher-scoring subsets contribute more significantly. This confirms that training data quality—measured by the spatial fidelity of the images from which rewritten prompts are derived—directly impacts fine-tuning effectiveness.

Data scaling trend (Figure 7, bottom): Performance consistently improves as training data is increased from 0% (base model) to 100% (full SpatialT2I), with training subsets added in order of increasing quality. The trend does not plateau at the current dataset size of 15,400 pairs, suggesting that further scaling would yield additional gains. However, the paper does not fit a scaling law or extrapolate expected performance at larger dataset sizes.

Cross-architecture fine-tuning validation (Table 6): Fine-tuning SpatialT2I on three architecturally distinct models—SD-XL (diffusion, 3.5B), UniWorld-V1 (unified, 12B), and OmniGen2 (autoregressive, 4B)—produces consistent overall gains of +4.2%, +5.7%, and +4.4% respectively. The gains are not uniform across sub-domains: Comparison improves by +3.9 to +9.3 points across the three models, while Occlusion improves by +4.8 to +6.9 points, and Causal interaction shows mixed results (+1.2% for SD-XL, +6.1% for UniWorld-V1, +2.2% for OmniGen2). The fact that improvements generalize across architectures supports the paper's claim that spatial intelligence deficits are at least partially data-driven rather than purely architectural. However, the residual gap—even fine-tuned models score below 35% on Comparison and Occlusion—indicates a ceiling that data-centric approaches have not yet breached.

Scene category exclusion from SpatialT2I: The "Design" scene category (130 prompts, including Cartoon, Advertisement, and Story sub-scenes) is excluded from the SpatialT2I dataset "due to low image quality." The paper does not further define "low image quality" or report metrics on how Design-scene images differ from those in other categories. This exclusion reduces the training data by approximately 10.6% and removes an entire category of abstract/non-photorealistic spatial reasoning from the fine-tuning corpus, meaning the fine-tuned models may not have improved spatial intelligence in abstract scenes. The paper does not evaluate fine-tuned models separately on held-out Design prompts to test this.

Critical Assessment

Do the experiments support the claim that spatial reasoning is the primary bottleneck?

Yes, strongly. The per-sub-domain leaderboard (Table 2) provides unambiguous evidence: for every model evaluated, scores on Spatial Comparison (S6, range 12.8-32.1%) and Spatial Occlusion (S8, range 15.6-33.8%) are dramatically lower than scores on Spatial Foundation (S1-S2, range 51.1-61.0% for top models) and Spatial Interaction (S9-S10, range 72.6-83.8% for top models). The gap is consistent across architectures, scales, and accessibility levels—it is the single most robust finding in the paper.

What supports this claim beyond the raw numbers is the error distribution analysis (Figure 5), which shows that Relational Reasoning errors dominate across all scene types while Basic Composition errors are rare. This pattern would not emerge if the bottleneck were merely a general difficulty gradient; instead, it indicates a specific capability deficit at the reasoning level of the spatial hierarchy.

A nuance the paper does not fully address: the benchmark may make Spatial Reasoning sub-domains intrinsically harder to evaluate accurately, not just harder for T2I models to generate. The human alignment study (Table 5) shows lower alignment scores on Spatial Reasoning, meaning evaluator noise is highest precisely where model scores are lowest. This opens the possibility that some of the measured gap between foundation and reasoning performance reflects evaluator difficulty rather than generative capability. However, the magnitude of the gap (40+ percentage points for top models) is far too large to be explained by evaluator noise alone, especially given that alignment scores remain near 80% even on the hardest sub-domains.

Do the experiments support the claim that the open-source / closed-source gap is narrowing?

Partially. The top open-source model (Qwen-Image, 60.6%) is within 2.1 points of the top closed-source model (Seed Dream 4.0, 62.7%) on overall score. This is a genuinely small gap relative to the performance range (28.5% to 62.7%), and it supports the narrowing claim. However, the gap widens on the most challenging sub-domains: Seed Dream 4.0 outscores Qwen-Image by 3.5 points on Comparison and 3.0 points on Occlusion. These are precisely the dimensions where the paper argues improvement is most needed, and the closed-source advantage is most pronounced on them.

The limitation is that the paper compares only four closed-source models against 19 open-source ones, making the "narrowing" claim depend heavily on where those specific closed-source models fall in their respective companies' capability distributions. If a new closed-source model were released that substantially surpassed Seed Dream 4.0, the gap could instantly widen. The narrowing is a snapshot, not a trend, since the paper evaluates each model at a single point in time.

Do the experiments support the claim that text encoder capability determines spatial intelligence?

The evidence is strongly suggestive but correlational. Models with T5 or LLM encoders consistently outperform CLIP-only models: the 16 models scoring above 50% overall all use either T5 or custom LLM encoders, while CLIP-only models (SD-1.5, PixArt-alpha) score below 40%. This is a robust association.

However, the paper does not provide a controlled ablation where the same model architecture is trained with different encoders while holding all else constant. The models differ in architecture, training data, training duration, resolution, and numerous other factors in addition to their text encoders. The claim that "text encoder capability emerges as a key determinant" is best interpreted as identifying a strong correlate, not establishing a causal mechanism. The prompt rewriting experiment (Table 11) provides partial causal evidence: when prompts are clarified (reducing the burden on the text encoder), all models improve, with larger gains for models that initially struggle (SD-3.5-L +2.3% vs. Qwen-Image +1.1%). This pattern is consistent with the text encoder being a bottleneck, but the effect sizes are modest (+1-2 points overall).

Do the experiments support the claim that data-centric fine-tuning improves spatial intelligence, with a boundary on what data alone can fix?

Yes, and this is the paper's most nuanced and well-supported finding from the fine-tuning experiments. The consistent gains across three architectures (Table 6) establish that SpatialT2I fine-tuning works—spatial intelligence is partially improvable through targeted training data without architectural changes. The differential gains across sub-domains establish the boundary: Position (+6.2 to +8.4 points) and Layout (+5.0 to +10.4 points) improve substantially, while Occlusion (+4.8 to +6.9 points) and Comparison (+3.9 to +9.3 points) improve but remain at low absolute levels (below 35%). This asymmetry is replicated in the prompt-rewriting experiment (Table 11), where rewriting helps explicit relationships but not implicit visual reasoning, independently confirming the boundary.

The data scaling trend (Figure 7, bottom) showing no plateau at 15,400 pairs suggests there is room for further data-driven improvement, but the differential rates of improvement across sub-domains imply that even large-scale data scaling will likely leave Occlusion and Comparison as stubborn bottlenecks unless architectural innovations address the underlying geometric reasoning deficit.

What experiments would have strengthened the paper?

Per-prompt, not just per-sub-domain, analysis. The benchmark reports per-sub-domain accuracy aggregating across all 1,230 prompts. It does not report how many prompts achieve "perfect" spatial intelligence (correct on all 10 sub-domains simultaneously). This metric would directly test the joint-constraint-handling capability that the information-dense prompt design is meant to probe and would likely reveal a dramatically lower score than per-sub-domain averages suggest.
Difficulty-controlled comparison with existing benchmarks. The paper compares model rankings on SpatialGenEval against meta-rankings from four other benchmarks (Table 3), but does not show whether SpatialGenEval captures variance in spatial intelligence that existing benchmarks miss. A regression analysis showing that SpatialGenEval scores predict human judgments of spatial quality after controlling for existing benchmark scores would strengthen the claim that SpatialGenEval measures something distinctive.
Controlled encoder ablation. To support the text encoder claim causally, an experiment where the same T2I model (e.g., SD-3-M) is evaluated with different text encoders (CLIP vs. T5 vs. LLM) while holding the generation backbone constant would isolate the encoder's contribution. The current cross-model comparison confounds encoder type with model architecture, training data, and scale.
Fine-tuning on non-spatial dense data as a control. The SpatialT2I fine-tuning gains could partially reflect general benefits of additional training data rather than the specific spatial density of the prompts. A control fine-tuning experiment using a dataset of comparable size but with short, non-spatial prompts (e.g., standard image captioning data) would isolate the contribution of information density to the observed gains.
Inter-annotator agreement metrics for human refinement and human alignment. The paper reports that five annotators "complete a calibration phase using a detailed guidebook" but does not report Cohen's κ or similar inter-annotator agreement statistics for either the prompt/QA refinement process or the human alignment study. This makes it difficult to assess the reliability of the human judgments that anchor both the benchmark quality and the evaluator validation.
Confidence intervals or significance tests. The paper reports point estimates for all accuracy scores without confidence intervals, statistical significance indicators, or error bars in figures. Given that the test set is 1,230 prompts split into difficulty bins that are themselves small (e.g., the human alignment study uses only 200 images), the sampling variability of the reported accuracies could be non-trivial. A 2-3 point difference between two models may not be statistically reliable, especially on sub-domains with fewer questions or lower evaluator alignment.
Resolution-controlled evaluation. Models in Table 8 generate images at different resolutions (from 384×384 for Janus-Pro to 1328×1328 for Qwen-Image). The paper does not control for whether resolution differences affect MLLM evaluator accuracy—higher-resolution images may make spatial relationships easier to discern, conferring an unfair advantage on high-resolution models. An experiment evaluating a subset of models at standardized resolutions would rule out resolution as a confound.

6. Limitations and Trade-offs

Capability Ceiling: Hardest Spatial Reasoning Sub-Domains Remain Near-Random Despite Fine-Tuning

The paper's central diagnostic finding is that Spatial Reasoning—particularly Comparison (S6) and Occlusion (S8)—is the primary bottleneck across all models. The fine-tuning experiments with SpatialT2I demonstrate that these sub-domains improve (+3.9 to +9.3 points for Comparison, +4.8 to +6.9 points for Occlusion; Table 6 in Section 4), but the absolute post-fine-tuning scores remain extremely low. The fine-tuned OmniGen2 achieves 35.3% on Comparison and 34.2% on Occlusion—roughly 15 percentage points above the random baseline of ~20% (Table 2, "Random" row). A practitioner deploying a fine-tuned model for an application requiring reliable quantitative comparison or depth ordering would encounter failures on roughly two-thirds of prompts.

This is not merely a matter of "more data needed." The paper's own evidence suggests a qualitative ceiling that data-centric approaches have not breached. The prompt rewriting experiment (Table 11, Appendix A.7) shows that clarifying spatial language improves explicit relationships (Position +4.7% for SD-3.5-L) but leaves Occlusion essentially unchanged (+0.2% for SD-3.5-L). This asymmetry implies that the bottleneck for occlusions and comparisons is not text understanding but a fundamental lack of 3D-aware spatial composition in the generation process—something supervised fine-tuning on 2D image-text pairs cannot fully supply.

The paper is transparent about this limitation. Section 4 explicitly states that the fine-tuned models achieve scores in the ~35% range on the hardest sub-domains, and the data scaling trend (Figure 7, bottom) shows improvement without plateau, leaving open the possibility that much larger datasets might push further. However, the paper does not provide evidence that continued scaling would close the gap rather than asymptote at a sub-50% level. This leaves a critical open question for adopters: for applications where occlusion or quantitative comparison are essential (architectural visualization, technical illustration, embodied AI training data), current methods—including SpatialT2I fine-tuning—do not provide acceptable reliability.

The paper partially mitigates this by identifying the boundary explicitly, which is valuable as a negative result: it directs future research toward architectural innovations rather than purely data-centric approaches for these sub-domains. But the mitigation is diagnostic, not operational—there is no solution offered for the two hardest sub-domains.

Practical Overhead of Benchmark Construction Is Not Accounted for in Assessment of Scalability

The SpatialGenEval benchmark relies on a human-in-the-loop pipeline that the paper itself describes as "labor-intensive" (Appendix A.10). The construction required "over 168 person-hours" from five expert annotators (Appendix A.2), plus the computational cost of generating 1,230 prompts and 12,300 QA pairs via Gemini 2.5 Pro. Each prompt undergoes review for stylistic awkwardness, logical impossibility, obscure vocabulary, and answer leakage—all judgment-intensive checks that resist full automation.

This matters for two reasons. First, it limits benchmark expansion to new domains, scenes, or spatial sub-domains. The paper acknowledges this in Appendix A.10: "the process is labor-intensive and presents challenges for scaling the benchmark to an even larger size or to new domains." If a practitioner wanted to adapt SpatialGenEval to evaluate spatial intelligence in, say, medical imaging or industrial design, they would need to replicate the full semi-automated pipeline with comparable human oversight, at comparable cost.

Second, it limits SpatialT2I-style dataset scaling, which the paper identifies as a promising direction. The data scaling trend (Figure 7, bottom) shows no plateau at 15,400 pairs, suggesting that order-of-magnitude scaling could yield further gains. But the SpatialT2I construction pipeline inherits the same human-in-the-loop dependency for prompt generation and QA validation, plus the additional cost of generating images from 14 models and rewriting prompts via GPT-4o. The paper provides no estimate of the marginal cost per training pair, making it impossible for practitioners to assess the feasibility of scaling to, say, 150,000 or 1.5 million pairs.

The paper does not account for these construction costs in any headline metric. The reported +4.2% to +5.7% fine-tuning gains (Table 6) are measured in accuracy improvement, not in accuracy-per-dollar or accuracy-per-annotator-hour. A deployment decision would need to weigh these gains against the cost of curating additional SpatialT2I-style data versus alternative approaches like architectural improvements or larger-scale pretraining.

Generalization Is Tested on a Single Benchmark Family with a Single Evaluator Paradigm

All results in this paper are derived from the MATH-equivalent of spatial intelligence evaluation: a single benchmark (SpatialGenEval) with 1,230 prompts across 25 scenes, evaluated by a specific family of MLLM judges (Qwen2.5-VL-72B primarily, GPT-4o as secondary validation). The paper's claims about spatial reasoning being the primary bottleneck, about open-source models catching up to closed-source ones, and about the effectiveness of SpatialT2I fine-tuning are all conditioned on this specific evaluation framework.

The paper provides several internal validation checks: cross-evaluator consistency between Qwen2.5-VL-72B and GPT-4o (Table 4), correlation with four existing benchmarks (Table 3), and human alignment studies (Table 5). These checks establish that SpatialGenEval is internally coherent and correlates with external measures. They do not establish that the paper's specific findings—the sub-domain difficulty ordering, the relative model rankings, the magnitude of fine-tuning gains—generalize to other evaluation frameworks.

Consider a concrete threat: the paper's finding that Spatial Interaction (Motion and Causal, S9-S10) scores are relatively high (70-83% for top models; Table 2) while Spatial Reasoning (Comparison and Occlusion, S6-S8) scores are low (below 35%). This difficulty ordering is central to the paper's diagnostic narrative. But it is potentially an artifact of how the evaluation questions are designed. If Motion and Causal questions are answerable through simpler visual heuristics (e.g., "is the dog in a jumping pose?" vs. "is the jump trajectory physically plausible?"), while Comparison questions demand precise quantitative judgment that the MLLM evaluator also finds challenging (as suggested by lower human alignment on Spatial Reasoning in Table 5), then the measured difficulty gap partially reflects evaluator capability rather than generator capability. The paper's human alignment study provides partial reassurance, but with only 200 images from a single model (Qwen-Image), the sample is too small to rule out evaluator-specific difficulty artifacts across the full model range.

The paper does not evaluate whether a differently constructed spatial intelligence benchmark—with different scene distributions, question formats, or evaluator paradigms—would produce the same sub-domain difficulty ordering or relative model rankings. This is a standard limitation of single-benchmark papers, but it is amplified here because the benchmark itself is a novel construct without an established track record of predictive validity for downstream spatial intelligence tasks.

The "E: None" Refusal Option Creates an Asymmetric Scoring Penalty That May Distort Model Comparisons

The paper includes "E: None" as a fifth option in every multiple-choice question to allow the MLLM evaluator to refuse when none of the four substantive options match the generated image. This is motivated by a principled concern: preventing forced-choice guessing when the T2I model produces an image that is faithful to none of the suggested answers (Section 2.1, Principle 4).

However, this design creates an asymmetric scoring dynamic that is not fully analyzed. Selecting "E: None" is always scored as incorrect, since the ground-truth answer is among options A-D by construction. But the likelihood of the evaluator selecting "E: None" depends on two factors: (a) whether the T2I model generated an image that matches one of the four substantive options, and (b) whether the MLLM evaluator recognizes that match. The second factor introduces evaluator capability as a confound.

Consider two scenarios for the same generated image containing a spatial error. An evaluator with strong spatial understanding might recognize that option B is the closest match (despite imperfection) and select it. A weaker evaluator might see that none of A-D perfectly describe the image and select "E: None." Both receive a score of 0, but for different reasons—one is an evaluator failure, the other is a genuine "none of the above" judgment. This asymmetry means that a model evaluated by a more "lenient" judge (one that is willing to select the best available option even when it is imperfect) could receive a higher score than the same model evaluated by a stricter judge, even if both judges are equally accurate when the image is correct.

The paper's human alignment study (Table 5) and cross-evaluator comparison (Table 4) suggest this effect is small for top-performing MLLMs, but they do not directly analyze it. The ablation where questions are evaluated without images (Table 12, Appendix A.9.2) shows that the evaluator selects "E: None" frequently when lacking visual evidence (accuracy drops to 16.9%, below random), confirming that "E: None" is used. But this ablation does not reveal whether two different MLLM evaluators would make systematically different "E: None" decisions on borderline images, potentially shifting model rankings.

This limitation is most consequential when comparing models with small absolute score differences. The paper reports overall scores with one-decimal precision (e.g., 60.6% vs. 60.5% for Qwen-Image vs. GPT-Image-1; Table 2). A systematic difference in "E: None" usage between evaluators of 1-2 percentage points could reverse such close rankings. The paper partially mitigates this by reporting cross-evaluator results (Table 10), which show qualitatively similar rankings, but does not report "E: None" selection rates by evaluator or by sub-domain, which would illuminate the magnitude of this effect.

The Fine-Tuning Gains Are Not Decomposed by Scene Category, Leaving Open Which Spatial Contexts Benefit

The SpatialT2I fine-tuning results (Table 6, Section 4) report per-sub-domain accuracy improvements aggregated across all 22 scene categories used in training. The paper notes that the "Design" category (Cartoon, Advertisement, Story) was excluded from SpatialT2I due to "low image quality" (Section 4, Stage 2), but does not report whether the remaining scene categories benefit uniformly from fine-tuning or whether gains are concentrated in specific visual contexts.

This matters because the paper's claim that "a data-centric paradigm [can] achieve spatial intelligence" (Abstract) implies domain-general improvement. If the gains are concentrated in, say, Indoor scenes (where spatial constraints are rigid and predictable: kitchens, offices, living rooms with canonical object layouts) while Outdoor or Nature scenes show minimal improvement (where spatial arrangements are more variable and less stereotyped), then the claim of general spatial intelligence improvement would be overstated. The per-sub-domain aggregation masks this heterogeneity.

The paper provides scene-level analysis only for the error distribution across all models (Figure 5, left panel), not for the fine-tuning gains specifically. A practitioner deciding whether to invest in SpatialT2I-style data for a specific application domain (e.g., architectural visualization, which maps primarily to Indoor scenes, or landscape generation, which maps to Nature scenes) cannot determine from the current results whether their domain would benefit proportionally.

The ablation study (Figure 7) provides indirect evidence relevant to this limitation: it shows that fine-tuning gains increase with the quality of the generating model (higher-scoring subsets contribute more). If certain scene categories are systematically more difficult for the generating models, the rewritten prompts for those categories may be of lower quality, yielding smaller fine-tuning gains. The paper does not test this interaction.

The Benchmark Does Not Resolve Whether Spatial Reasoning Failures Are Failures of Comprehension or Failures of Rendering

A fundamental ambiguity runs through the paper's diagnostic framework: when a model scores poorly on a Spatial Reasoning sub-domain, the failure could occur at the comprehension stage (the model fails to encode or reason about the specified spatial constraint from the text prompt) or at the rendering stage (the model understands the constraint but cannot translate it into pixel-level spatial arrangement). These two failure modes suggest different interventions—better text encoders for comprehension, better geometric priors for rendering—but the benchmark cannot distinguish between them.

The paper provides suggestive evidence that both mechanisms matter. The text encoder gradient (CLIP-only models underperform substantially; Section 3.2, finding "Text encoder capability emerges as a key determinant") points toward comprehension. The prompt rewriting experiment (Table 11), where clarifying language produces modest but real gains (+1.1% to +2.3% overall), also supports a comprehension component. But the finding that rewriting fails to improve Occlusion (+0.2% for SD-3.5-L) while fine-tuning on SpatialT2I images does (+4.8% for SD-XL) points toward a rendering component that requires visual training data to address.

This ambiguity is not entirely resolvable without access to model internals—the benchmark can only observe the final output, not the intermediate reasoning. But it limits the benchmark's diagnostic precision. A model scoring 28% on Spatial Comparison could be failing because it never encoded the "three times taller" constraint, or because it encoded it but could not render proportionate object sizes, or some mixture of both. The paper's decision to evaluate exclusively on final generated images makes this decomposition impossible.

The paper acknowledges this implicitly through its two-pronged intervention strategy: improved prompts (Table 11) target comprehension, while SpatialT2I fine-tuning (Table 6) targets the full comprehension-to-rendering pipeline. That both produce gains suggests both bottlenecks are real, but the absence of a decomposition analysis means practitioners cannot prioritize which intervention to invest in for a specific model or application.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not introduce a new model architecture or training algorithm. It introduces something more foundational: a diagnostic instrument for measuring a capability that the field has been intuitively aware of but unable to quantify. The conceptual shift is from treating spatial intelligence as an undifferentiated, qualitative property of T2I models ("this model is good at spatial relationships") to treating it as a decomposable, measurable, hierarchically structured capability with specific failure modes at specific levels of the hierarchy.

The magnitude of this shift is best understood through the paper's own framing: current benchmarks evaluate whether models can handle any spatial constraint in isolation; SpatialGenEval evaluates whether they can handle all constraints simultaneously. This is not an incremental refinement of existing evaluation methodology. It is a different kind of measurement targeting a different construct—joint constraint satisfaction under information density rather than independent constraint satisfaction under sparsity. The existence of this measurement instrument changes the research landscape in several specific ways:

It transforms "spatial intelligence" from a complaint into a research target. Before SpatialGenEval, a researcher who wanted to improve spatial reasoning in their T2I model had no standardized way to measure whether they were succeeding, beyond qualitative examples or proxy metrics like detection accuracy. The benchmark provides a 10-dimensional performance profile that a researcher can track across training runs, architectural variations, and data interventions, with per-sub-domain granularity that reveals which spatial capabilities are improving and which are not. This is the same transformation that benchmarks like GLUE and SuperGLUE performed for natural language understanding—converting an ill-defined capability into a set of measurable targets that structure research effort.

It reframes the relationship between text understanding and visual composition in T2I models. The finding that models with T5 or LLM-based text encoders systematically outperform CLIP-only models (Table 2: SD-1.5 at 28.5% vs. FLUX.1-krea at 58.5%) establishes that prompt comprehension is a necessary condition for spatial intelligence. But the finding that even the strongest LLM-encoder model (Qwen-Image, 60.6%) scores below 35% on Comparison and Occlusion—and that prompt rewriting cannot close this gap (Table 11: Occlusion +0.2% from rewriting)—establishes that comprehension is not sufficient. Spatial intelligence requires visual compositional capabilities beyond what current architectures provide, even when text understanding is strong. This reframing redirects research from the predominantly text-encoder-focused improvement narrative toward a more balanced recognition that spatial reasoning is a joint text-visual competence where the visual pathway is the current bottleneck.

It establishes verifiable boundaries on data-centric improvement, which is itself a significant negative result. The literature on T2I improvement has largely operated on the assumption that more and better training data—possibly combined with stronger text encoders and larger models—will progressively resolve capability gaps. The SpatialT2I fine-tuning experiment (Table 6) confirms that this approach works for explicit spatial relationships: Position, Layout, and Proximity all improve by 5-10 percentage points. But it also demonstrates that the same approach plateaus at roughly 35% on Occlusion and Comparison, even after training on 15,400 curated, information-dense, spatially-verified text-image pairs. This is a boundary result: it identifies the specific capabilities that resist data-centric scaling, and in doing so, it implies where architectural innovation—explicit 3D geometric priors, depth-aware rendering, structured spatial representations—must supplement raw data scale. This redirects effort away from "collect more spatial data" as a universal solution and toward targeted architectural research for the hardest sub-domains.

It reconciles a latent tension in the T2I evaluation literature. Prior benchmarks created a fragmented picture: some emphasized composition (GenEval, T2I-CompBench) and found steady progress, while others tested instruction following (TIIF-Bench) or world knowledge (Wise) and found persistent gaps. SpatialGenEval's hierarchical taxonomy shows that these results are not contradictory—they are measuring different levels of the same capability stack. The steady progress on Spatial Foundation (object identity and attribute binding) is real, and the persistent gaps on Spatial Reasoning (comparative, proximity, occlusion) are equally real. The taxonomy reveals that the field has been making progress on the lower floors of the spatial intelligence building while leaving the upper floors largely unconstructed. This reconciliation is valuable because it prevents the optimistic narrative ("scores keep improving") and the pessimistic narrative ("models still can't reason spatially") from talking past each other; they are both right about different parts of the hierarchy.

It makes MLLM-based evaluation of generative models methodologically credible at scale. The paper's validation suite—cross-evaluator consistency between Qwen2.5-VL-72B and GPT-4o (Table 4), human alignment studies with balanced accuracy near 80% even on hard sub-domains (Table 5), and below-random accuracy without images confirming visual grounding (Table 12)—establishes a replicable protocol for using MLLMs as judges for spatial generation tasks. This matters beyond this paper because it reduces the barrier to constructing new generative benchmarks: future work can adopt the same protocol (multiple-choice questions with distractors, "E: None" refusal, 5-round majority voting, explicit knowledge prohibition) with confidence that the resulting measurements align with human judgment. This is a methodological contribution that outlasts the specific benchmark.

Follow-Up Research This Work Enables

Testing whether the spatial reasoning bottleneck is a comprehension failure or a rendering failure through controlled architectural interventions. The paper's diagnostic framework cannot distinguish between a model that fails to understand the "three times taller" constraint from the prompt text and a model that understands the constraint but cannot render proportional object sizes in the generated image. Both failure modes produce the same low Comparison score. A controlled experiment would take a single T2I architecture and evaluate it under two conditions: (1) standard text-to-image generation from information-dense prompts, and (2) a "layout-conditioned" variant where the model receives an explicit spatial layout specification (bounding boxes, depth maps, or scene graphs) alongside the text prompt, removing the need to infer spatial relationships from text alone. If Comparison and Occlusion scores improve substantially in the layout-conditioned condition, the bottleneck is primarily text-to-spatial comprehension. If they remain low, the bottleneck is primarily visual rendering. The paper's encoder gradient analysis and prompt rewriting results suggest both mechanisms operate, but a controlled decomposition would quantify their relative contributions and direct investment toward improved text encoders, geometric rendering modules, or both.

Evaluating whether the SpatialT2I data scaling trend continues beyond 15,400 pairs or asymptotes at a sub-50% ceiling for Occlusion and Comparison. The data scaling curve in Figure 7 (bottom) shows monotonic improvement without plateau at the current dataset size. The paper's own interpretation is that "further scaling is a promising direction" (Section 4, ablation discussion). A direct follow-up would scale SpatialT2I by an order of magnitude—generating ~150,000 text-image pairs using the same pipeline but with a larger prompt set, potentially automated QA generation with lighter human oversight, and a wider set of generating models. The key measurement would be whether Occlusion and Comparison scores on fine-tuned models approach the 50-60% range (competitive with Spatial Foundation performance) or asymptote at 35-40% despite 10× more data. An asymptote at low levels would provide strong evidence for the architectural-bottleneck hypothesis and would motivate a shift from data-centric to architecture-centric research for these sub-domains.

Extending the information-dense, omni-dimensional evaluation paradigm to temporal consistency in text-to-video generation. The paper briefly mentions text-to-video as a future direction (Appendix A.11), but the extension is conceptually natural and technically specific. Beyond static spatial relationships, video generation must maintain spatial consistency across time: an object that is "to the left of the sofa" in frame 1 must remain in a spatially coherent position through camera motion or object movement in subsequent frames. The SpatialGenEval framework can be adapted by generating information-dense prompts that include temporal spatial constraints (e.g., "the red ball rolls from the left side of the table to the right side, passing behind the blue vase") and constructing omni-dimensional QA pairs that probe spatial relationships at multiple timestamps. The primary challenge is automated evaluation: a single frame can be checked with an MLLM as in the current benchmark, but multi-frame spatial consistency requires temporal reasoning that current MLLMs handle imperfectly. A strong follow-up would develop a temporal extension of the evaluation protocol, validate it against human judgments of video spatial consistency, and benchmark current text-to-video models (Sora, Kling, Gen-3, etc.) against the extended taxonomy.

Using SpatialGenEval scores as a reward signal for reinforcement learning-based fine-tuning of T2I models. The paper currently demonstrates supervised fine-tuning on SpatialT2I as the improvement mechanism (Section 4), but the benchmark's 10-dimensional accuracy profile could serve as a more direct optimization target. Recent work on RL for T2I models (Flow-GRPO, Pref-GRPO) uses MLLM feedback as a reward signal; SpatialGenEval's per-sub-domain accuracy provides a structured reward that could guide the model toward balanced spatial improvement rather than over-optimizing for the sub-domains where reward is easiest to achieve. The key experimental design would be: train a T2I model using RL with SpatialGenEval accuracy (or a learned proxy thereof) as the reward, compare against the SFT baseline from this paper, and measure whether RL produces larger gains on the hardest sub-domains (Comparison, Occlusion) where SFT plateaus. The paper notes this direction in Appendix A.11 under "post-training strategies," but the specific connection between the benchmark's diagnostic structure and RL's ability to target specific capability gaps is not developed.

Stress-testing the benchmark's diagnostic claims by evaluating on models with explicit 3D/geometric priors. If the paper's central claim is correct—that current T2I models lack the internal spatial representations needed for occlusion, comparison, and 3D reasoning—then models that have been augmented with explicit depth estimation, 3D awareness, or geometric priors should show disproportionate gains on Spatial Reasoning sub-domains relative to Spatial Foundation. Candidate models include T2I architectures that incorporate monocular depth estimation during generation, view-conditioned diffusion models trained on multi-view data, or models that use explicit scene graphs as intermediate representations before rendering. The diagnostic prediction is testable: these models should close the gap between Spatial Foundation (~60-77%) and Spatial Reasoning (~28-35%) scores that the paper identifies as the universal bottleneck. A failure to close this gap—if even 3D-aware models score below 50% on Occlusion—would suggest the bottleneck is even more fundamental than the paper's current diagnosis, perhaps involving issues of object-centric representation or physical commonsense that neither text encoders nor geometric priors fully address.

Constructing a difficulty-predictive model that estimates per-prompt spatial difficulty before generation, enabling adaptive resource allocation for T2I models. The paper's benchmark reveals enormous variance in model performance across prompts (some prompts are solved correctly by all models; others fail across the board), but the benchmark does not provide a way to predict which prompts will be hard a priori. A follow-up could train a lightweight text classifier on the SpatialGenEval prompts, using per-model average accuracy as difficulty labels, to learn which linguistic and spatial features predict generation difficulty. If such a classifier achieves reasonable accuracy, it enables a deployment strategy analogous to difficulty-aware inference-time compute allocation in LLMs: easy prompts are generated with a small, fast model; hard prompts are routed to a larger model or allocated additional generation budget (more samples, iterative refinement). The paper's per-sub-domain accuracy breakdown provides the training signal; the challenge is extracting predictive features from prompt text alone without access to generated images.

Practical Applications and Downstream Use Cases

Pre-deployment quality assurance for T2I models in spatially-sensitive applications. An organization building a T2I-powered design tool for interior decoration, architectural visualization, or product placement would need to guarantee that generated images respect spatial constraints (furniture correctly positioned relative to walls, products correctly scaled relative to each other, occlusions rendered realistically). Running SpatialGenEval on candidate models before deployment provides a 10-dimensional capability profile that reveals which spatial failure modes are likely in production. If the application requires reliable occlusion rendering, the benchmark tells the organization that even the best model scores only 34% on that sub-domain—a red flag that may require human-in-the-loop verification of generated outputs. This is not hypothetical: the benchmark's finding that Comparison and Occlusion scores remain below 35% even after fine-tuning (Table 6) is directly actionable as a risk assessment for any deployment where these capabilities are load-bearing.

Data filtering and curation for embodied AI training pipelines. Research that uses T2I models to generate synthetic training data for robots or autonomous agents (as discussed in the paper's Broader Impact, Appendix A.12) needs images where spatial relationships are physically correct—otherwise the downstream AI learns a distorted world model. SpatialGenEval provides a screening mechanism: candidate T2I models can be evaluated on prompts representative of the target embodied domain, and models with low Spatial Reasoning scores can be filtered out or their outputs can be subjected to additional verification (e.g., running a depth estimator on generated images to check occlusion consistency). The benchmark's per-sub-domain granularity is especially useful here because it distinguishes between models that fail at basic object presence (which would contaminate any training pipeline) and models that fail only at higher-order reasoning (which may be acceptable if the training task does not require precise occlusion or comparison). For example, a training pipeline focused on object recognition might accept a model scoring 70% on Spatial Foundation even if its Spatial Reasoning is poor, while a pipeline for manipulation planning would require strong scores on Proximity and Occlusion.

Guided data collection for fine-tuning spatial capabilities in specific domains. The paper's ablation (Figure 7) showing that higher-quality training data produces larger fine-tuning gains provides a recipe for domain-specific improvement. A team building a T2I system for a particular vertical—say, generating product photos for e-commerce where objects must be correctly positioned, scaled relative to each other, and partially occluded by props—can replicate the SpatialT2I construction pipeline using their domain-specific scenes and prompts. The key finding they can exploit is that the quality of the generating model matters: using the best available T2I model to generate initial images, then rewriting prompts to correct text-image misalignment, produces the most effective fine-tuning data. The benchmark's leaderboard (Table 2) tells them which models are strongest on their target sub-domains, letting them select generating models that maximize training data quality for their specific spatial requirements. The consistent +4-6% gains across architectures (Table 6) suggest this recipe transfers across domains, though the paper does not test domain transfer directly.

Benchmark-driven model selection for multi-model T2I pipelines. The paper's finding that even the best model (Seed Dream 4.0, 62.7%) has significant sub-domain imbalances—it scores 83.8% on Causal Interaction but 33.8% on Occlusion—suggests that no single model is uniformly best across all spatial capabilities. A deployment that can afford to query multiple models could use SpatialGenEval's per-sub-domain leaderboard as a routing table: when a user prompt heavily emphasizes position and layout, route to a model strong on Spatial Perception; when occlusion or comparison is critical, route to the model that scores highest on those specific sub-domains even if its overall score is lower. This is speculative given the current paper—the benchmark evaluates models on uniform prompts, not prompt-adaptive routing—but the per-sub-domain breakdown makes such routing technically specifiable in a way that single-score benchmarks do not.

When to Prefer This Benchmark Over Alternatives

The paper does not explicitly position SpatialGenEval against named alternative benchmarks in a "choose A over B" framework. However, it does articulate the conditions under which its benchmark provides diagnostic value that existing benchmarks miss (Section 1, paragraphs on "Where Existing Benchmarks Fall Short"), and the evaluation results themselves imply natural selection criteria. The following guidance is derived from the paper's explicit claims and findings:

Prefer SpatialGenEval over detection-based benchmarks (GenEval, T2I-CompBench) when the goal is to diagnose why a model fails on complex real-world prompts, not merely whether it generates specified objects. The paper's evidence that models scoring above 70% on Spatial Foundation can score below 30% on Spatial Reasoning (Table 2: Qwen-Image at 77.2% on S2 vs. 28.6% on S6) means detection-based benchmarks give a falsely optimistic picture of spatial capability by measuring only the lowest levels of the hierarchy. For applications where multi-constraint spatial prompts are common (design tools, professional visualization, scene composition), SpatialGenEval is the appropriate diagnostic.
Prefer SpatialGenEval over instruction-following benchmarks (TIIF-Bench, DPG-Bench) when the goal is fine-grained capability decomposition rather than aggregate instruction-following ability. The omni-dimensional evaluation with per-sub-domain accuracy (Table 2, columns S1-S10) reveals capability profiles that aggregate scores mask. For a research team iterating on model architecture or training data, knowing that their intervention improved Layout by +8 points but left Occlusion unchanged is more actionable than knowing that "overall instruction following improved by 3%."
Prefer existing benchmarks over SpatialGenEval when rapid, low-cost evaluation is the priority. SpatialGenEval's evaluation pipeline—1,230 prompts × 10 questions × 5 rounds × MLLM inference—is computationally intensive relative to detection-based methods that can be parallelized efficiently. The paper reports 40 minutes per model on 8× H20 GPUs (Section 3.1), which is reasonable for a thorough evaluation but expensive for frequent iteration during model development. Detection-based benchmarks like GenEval may be more suitable for fast development cycles, with SpatialGenEval reserved for milestone evaluations that require spatial diagnostic depth.

🎯 Pitch

1. Executive Summary

2. Context and Motivation

The Core Problem: Text-to-Image Models Can Generate Objects, but They Can't Place Them Correctly

Why This Matters: Beyond Academic Benchmarking to Real-World Deployment

Where Existing Benchmarks Fall Short: Short Prompts, Sparse Information, Coarse Evaluation

How This Paper Positions Itself: Evaluation as Discovery and Improvement

3. Technical Approach

3.1 Reader Orientation

3.2 Big-Picture Architecture (Diagram in Words)

3.3 Roadmap for the Deep Dive

3.4 Detailed, Sentence-Based Technical Breakdown

The Spatial Intelligence Taxonomy: 4 Domains, 10 Sub-Domains

Spatial Foundation (S1/S2): The "What" of a Scene

Spatial Perception (S3/S4/S5): The "Where" and "How Arranged"

Spatial Reasoning (S6/S7/S8): The "How They Relate" — Higher-Order Inference

Spatial Interaction (S9/S10): The "Why Things Happen" — Dynamic Events

Why This Particular Taxonomy?

Information-Dense Prompt Generation (SpatialGenEval Construction Stage 1)

Automated Generation via Gemini 2.5 Pro

Why Gemini 2.5 Pro?

Human-in-the-Loop Prompt Refinement

Omni-Dimensional QA Generation and Evaluation Protocol (SpatialGenEval Construction Stage 2)

Automated QA Generation

Anti-Leakage Design: Human Validation of Questions

The "E: None" Refusal Option

MLLM-as-Judge Evaluation Protocol

Five-Round Majority Voting

Primary Evaluator: Qwen2.5-VL-72B

Human Alignment Study

Evaluation Infrastructure

SpatialT2I: From Evaluation Benchmark to Fine-Tuning Dataset

Stage 1: Generating a Fresh Set of Prompts

Stage 2: Curating Generated Images from Top-Performing Models

Stage 3: Prompt Rewriting for Text-Image Alignment

Stage 4: Supervised Fine-Tuning

Ablation: Data Quality Matters More Than Any Single Source

Design Rationale Summary

Summary of Design Choices and Their Justifications

4. Key Insights and Innovations

Innovation 1: A Hierarchical Taxonomy That Transforms Spatial Intelligence from an Implicit Intuition into a Diagnosable, Decomposable Capability

Innovation 2: The Information-Dense, Multi-Constraint Prompt as the Evaluative and Training Primitive

Innovation 3: Decoupling Generation Complexity from Evaluation Complexity to Enable Reliable Automated Benchmarking

Innovation 4: Establishing Verifiable Boundaries on What Data-Centric Approaches Can and Cannot Fix for Spatial Intelligence

5. Experimental Analysis

Evaluation Methodology

Main Quantitative Results

Overall Leaderboard and Model Rankings

The Performance Gap Between Spatial Foundation and Spatial Reasoning

Failure Case Analysis Across Scenes and Skill Levels

Text Encoder Capability as a Key Determinant of Spatial Performance

Model Scale and Architecture as Two Complementary Pathways

Correlation with Existing Benchmarks

Judge-Dependency Analysis

Human Alignment Study

Ablation Studies and Robustness Checks

Critical Assessment

Do the experiments support the claim that spatial reasoning is the primary bottleneck?

Do the experiments support the claim that the open-source / closed-source gap is narrowing?

Do the experiments support the claim that text encoder capability determines spatial intelligence?

Do the experiments support the claim that data-centric fine-tuning improves spatial intelligence, with a boundary on what data alone can fix?

What experiments would have strengthened the paper?

6. Limitations and Trade-offs

Capability Ceiling: Hardest Spatial Reasoning Sub-Domains Remain Near-Random Despite Fine-Tuning

Practical Overhead of Benchmark Construction Is Not Accounted for in Assessment of Scalability

Generalization Is Tested on a Single Benchmark Family with a Single Evaluator Paradigm

The "E: None" Refusal Option Creates an Asymmetric Scoring Penalty That May Distort Model Comparisons

The Fine-Tuning Gains Are Not Decomposed by Scene Category, Leaving Open Which Spatial Contexts Benefit

The Benchmark Does Not Resolve Whether Spatial Reasoning Failures Are Failures of Comprehension or Failures of Rendering

7. Implications and Future Directions

How This Work Changes the Landscape

Follow-Up Research This Work Enables

Practical Applications and Downstream Use Cases

When to Prefer This Benchmark Over Alternatives

Adjacent in domain

EvasionBench: A Large-Scale Benchmark for Detecting Managerial Evasion in Earnings Call Q&A

Retrieval-Infused Reasoning Sandbox: A Benchmark for Decoupling Retrieval and Reasoning Capabilities