Proactive Agents for Multi-Turn Text-to-Image Generation Under Uncertainty

🎯 Pitch

Turning a vague prompt into the right image usually forces users into frustrating trial-and-error, but giving a T2I agent the ability to ask clarifying questions when it’s uncertain cuts that effort dramatically—achieving over 2× higher alignment scores in just a few turns. Over 90% of users in human studies found the agent's proactive clarifications helpful, and a majority even preferred its output to their own manually refined results.

1. Executive Summary

This paper proposes proactive text-to-image (T2I) agents that engage users in multi-turn clarification dialogues to resolve prompt underspecification, using a novel belief graph — a symbolic representation of the agent's uncertainty about entities, attributes, and relations in the intended image — to drive question selection and present editable uncertainty to users. Experiments on three datasets (ImageInWords, COCO, and a new artist-focused DesignBench) with Gemini 1.5-powered agents and Imagen 3 generation demonstrate that three agent variants (a heuristic-score agent, a belief-prompted agent, and a principle-prompted agent) achieve at least 2× higher VQAScore than single-turn T2I baselines within 5 interaction turns, with the principle-prompted agent (Ag3) achieving the strongest results by relying solely on conversation history and question-asking principles rather than explicit belief graph access. Human studies confirm that over 90% of regular T2I users find proactive clarifications helpful and over 80% prefer agent-generated images over single-turn outputs, establishing that interactive uncertainty reduction substantially improves image-prompt alignment when initial prompts leave room for disambiguation — though the approach remains bounded by the prompt-following fidelity of the underlying frozen T2I model.

2. Context and Motivation

The Fundamental Problem: Prompt Underspecification Breaks Text-to-Image Alignment

The core challenge this paper tackles is deceptively simple: when a user gives a T2I model a brief prompt like "A breakfast plate" or "A rabbit near a cat," the model must resolve enormous ambiguity about what the user actually wants. A prompt is inherently a low-information signal relative to the high-information output (a detailed image with specific entities, attributes, spatial relationships, lighting, style, and background). Any model generating an image from an underspecified prompt must make implicit assumptions — about cuisine type, rabbit breed, spatial proximity, artistic style — and these assumptions frequently diverge from the user's mental image.

This is not a theoretical edge case. The paper identifies it as a pervasive practical frustration (Section 1, Section 7.4). In a human survey of 143 regular T2I users (Table 4, Appendix G), the authors document that:

83% experience occasional, frequent, or very frequent frustration due to needing many prompt iterations
70% report frustrations from prompt misinterpretation
71% face inconsistent generations where re-running the same prompt produces different (and differently wrong) outputs
60% are frustrated by incorrect model assumptions about unspecified details

The misalignment problem compounds because it creates a trial-and-error cycle: users see an unsatisfactory image, guess at what details to add to their prompt, regenerate, and repeat — often without knowing which details the model actually needs. Worse, different cultural backgrounds interpret identical prompts in radically different ways (the paper's "breakfast plate" example in Section 1), meaning the model's default assumptions may systematically exclude certain cultural representations, propagating bias and potentially causing offense (Basu et al., 2023; Bianchi et al., 2023; Kannen et al., 2024).

The paper frames this as a communication problem at the interface between human intention and machine generation, situating it within the broader AI challenge of "moving from low-information (prompts) to high-information (images)" (Section 1). This framing is significant because it shifts attention from improving the generative model itself — the dominant research direction — to improving the interaction protocol through which user intent flows to the model.

The Gap: Existing Systems Are Passive, Not Proactive

Prior to this work, the standard T2I workflow was fundamentally one-directional: user writes a prompt, model generates an image, user inspects the output. If the image is unsatisfactory, the user must diagnose what went wrong and how to fix it through prompt revision — a process that demands both prompt engineering skill and the ability to mentally decompose an image into model-understandable attributes. This places the entire burden of disambiguation on the user.

The paper identifies several categories of prior work that address pieces of this problem but fall short of a genuine interactive solution (Section 2):

Prompt expansion (Betker et al., 2023; Garg et al., 2024; Datta et al., 2024) is the closest practical technique: automatically enrich a short prompt with additional details before feeding it to the T2I model. ImageInWords (Garg et al., 2024) demonstrates that hyper-detailed captions significantly improve generation quality, and Datta et al. (2024) show that prompt expansion increases user satisfaction. However, these approaches are non-interactive: they expand prompts based on the model's assumptions about what details matter, not based on the specific user's actual intent. An expanded prompt that adds "American-style breakfast with bacon and eggs" when the user wanted a Faroese breakfast still produces misalignment — it just produces it more confidently. The paper explicitly notes that "evaluating our method as a prompt expansion tool is outside of our scope" (Section 2), distinguishing its interactive, user-driven approach from automatic expansion.

Ambiguity resolution (Mehrabi et al., 2022) has been explored for cases where prompts have multiple distinct meanings (e.g., "elephant" could mean the animal or a political symbol). The paper distinguishes this from its own focus: the problem is not ambiguity (a prompt having multiple valid interpretations) but underspecification (a prompt lacking sufficient information to fully describe the intended image). The prompt "a rabbit near a cat" is not ambiguous — it clearly describes both animals — but it is severely underspecified: it says nothing about breed, color, pose, background, or spatial arrangement.

Multi-turn T2I systems like DialogGen (Huang et al., 2024) and DSG-GAN (Sun and Guo, 2023) have begun to explore interactive generation. However, the paper positions these as addressing different problems (browsing or refining generated images, or responding to user-initiated corrections) rather than the proactive, uncertainty-driven clarification that the authors propose. The critical distinction is that prior interactive systems are reactive — they respond to user feedback — while the proposed agents are proactive — they anticipate what information they need and ask for it before generating.

Clarification question research in NLP and HCI has studied when and how models should ask users for clarification. However, the paper argues these approaches have not been systematically applied to the T2I domain with its specific characteristics: visual-spatial reasoning, the need to communicate about images before they exist, and the availability of well-defined metrics for evaluating alignment.

The Deeper Intellectual Gap: No Representation of Uncertainty About User Intent

Perhaps the most fundamental gap the paper identifies is conceptual rather than methodological: existing T2I systems have no explicit representation of what they don't know about the user's intent. When a standard model receives the prompt "A rabbit near a cat," it immediately commits to specific rabbits, specific cats, and specific spatial relationships — but has no mechanism to surface that it is uncertain about any of these choices. The model's internal uncertainty about what the user wants is invisible to both the model and the user.

The paper positions this gap within a rich intellectual tradition in AI. Classical AI research on belief states (Kaelbling et al., 1998; Nilsson, 1986) and symbolic world representations (McCarthy and Hayes, 1969; Minsky, 1974; Fikes and Nilsson, 1971) developed formal frameworks for representing an agent's uncertain knowledge about the world as probability distributions over states. In these frameworks, an agent explicitly tracks what it knows, what it doesn't know, and how confident it is about each piece of knowledge. However, these classical approaches relied on rigid, hand-engineered predicates (e.g., in the blocks world, is_block(a), on(a, b)) that don't scale to the open-ended visual world of T2I generation.

The paper sees a natural but unexplored connection: a T2I agent, like a classical AI planning agent, should maintain an explicit belief state about the world it needs to create, tracking its uncertainty about each entity, attribute, and relation. This belief state could then drive information-gathering actions (clarification questions) and be surfaced to the user for direct inspection and editing. The Davidsonian Scene Graph (DSG) formalism (Cho et al., 2023; Davidson, 1965, 1967a,b) — which decomposes image descriptions into atomic propositions — provides a bridge between classical symbolic representations and modern LLM-based text processing, suggesting a path toward scalable belief representation.

How This Paper Positions Itself

The paper constructs a deliberate three-part thesis that directly addresses the gaps identified above:

First, it proposes that the missing piece in T2I interaction is proactivity — the agent should actively seek information from the user rather than passively accept underspecified prompts. This is not merely an interface preference but a principled approach grounded in the active learning and Bayesian experimental design literatures (Cohn et al., 1996; Houlsby et al., 2011; Chaloner and Verdinelli, 1995): the agent should ask questions that maximize expected information gain about the user's intent.

Second, it introduces the belief graph as the representational bridge that makes proactive questioning tractable. Rather than hand-engineering predicates for every scenario, the belief graph uses LLMs to adaptively generate structured representations of entities, attributes, and relations from arbitrary image descriptions, enriched with probability distributions and importance scores that enable principled question selection (Section 4). This marries the structure of classical belief states with the flexibility of modern LLMs.

Third, it develops and evaluates a spectrum of agent designs — from a fully explicit, heuristic-score-driven agent (Ag1) that explicitly computes scores over the belief graph, to a principle-prompted agent (Ag3) that embeds the question-asking principles (no redundancy, uncertainty reduction, relevance, easy-to-answer) directly into LLM prompts without explicit belief graph access — to understand how much of the benefit comes from the explicit belief representation versus from the interaction principles themselves (Section 6).

The paper explicitly frames itself as a prototype and framework rather than a production system. Section 1 states the goal as demonstrating that "simple prototypes for such agents" can achieve substantial gains, and Section 5 frames the design principles as "high-level principles and design that guide our agent how to behave and interact with users." The modular architecture — frozen T2I model, replaceable LLM for belief parsing, pluggable question selection strategies — is intentionally designed to allow individual components to be swapped as technology improves.

Why This Problem Matters Beyond T2I

While the paper uses T2I generation as its testbed, it positions the work as addressing a broader challenge in AI: effective multi-turn communication between AI agents and human users with vague or incomplete instructions (Section 1, first paragraph). The T2I domain is selected because it offers three desirable properties for studying this problem: (1) a clear gap between low-information inputs and high-information outputs, making the underspecification problem acute; (2) well-defined evaluation metrics (VQAScore, DSG, DINO similarity) that enable automatic comparison of generated outputs to ground truth; and (3) a bounded scope that makes comprehensive evaluation feasible.

The paper also connects to broader concerns about AI safety and inclusivity. Section 8 and the Impact Statement argue that models that can ask clarifying questions rather than making implicit assumptions are safer — they are less likely to generate content that a specific user finds offensive or harmful because they learn the user's boundaries through interaction. They are also more inclusive — rather than defaulting to a Western-centric or majority-culture interpretation of prompts, they can elicit the cultural context that determines what "breakfast plate" means to a particular user in a particular cultural setting. This reframes clarification from a convenience feature to a responsibility mechanism.

What's at Stake

The paper's human study results (Section 7.4) quantify the practical urgency: 55% of regular T2I users report frequent or very frequent frustration due to prompt iteration, and clarification features are rated as "likely" or "very likely" to help by 91% of users. The human evaluation of generated images (Section 7.3) shows that agent-generated images are preferred over single-turn T2I outputs in 80%+ of cases for both content and aesthetics. These are not marginal improvements at the edge of statistical significance — they represent a fundamental gap between what users want and what current systems deliver.

The practical stakes are amplified by the growing role of T2I in creative workflows, design, and content creation. As these systems move from novelty to tool, the cost of prompt iteration — in time, frustration, and creative disruption — becomes a real productivity barrier. The paper's approach offers a path toward T2I systems that act more like collaborative design partners, actively eliciting requirements rather than passively accepting incomplete specifications.

3. Technical Approach

3.1 Reader Orientation

This paper designs and evaluates proactive T2I agents — software systems that sit between a user's initial image prompt and a frozen text-to-image model, engaging the user in a multi-turn clarification dialogue before generating images. The system solves the problem that underspecified prompts force T2I models to make implicit (often wrong) assumptions about what the user wants; instead of guessing, the agent explicitly tracks its uncertainty about entities, attributes, and relations in a structured belief graph, converts the most uncertain and important unknowns into clarification questions, and only generates images after the user has resolved the critical ambiguities.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five interconnected components that operate in a loop:

Belief Graph Constructor — Parses the current merged prompt (accumulated conversation history) using LLM-based in-context learning parsers to produce a structured representation of entities (explicit, implicit, background), their attributes with value-probability distributions, and their spatial/functional relations, each annotated with both a probability (reflecting uncertainty about presence or value) and an importance score (reflecting the element's impact on the final image).
Action Selector — Applies a question-asking strategy (heuristic scoring, belief-prompted LLM, or principle-prompted LLM) that scores candidate questions by combining uncertainty (entropy over value distributions) and importance, then selects the single question that maximally reduces the agent's overall uncertainty about the user's intent.
Question Verbalizer — Converts the selected question target (e.g., "color attribute of rabbit entity") into a natural-language question that follows the Easy-to-Answer principle — typically by offering multiple-choice options from the most probable candidates — and presents it to the user.
Belief Transition Engine — Takes the user's answer, merges it into the conversation history via LLM summarization (converting Q&A pairs to declarative sentences and combining with the prior merged prompt), then re-runs the full belief parsing pipeline on the updated prompt to produce a refreshed belief graph reflecting reduced uncertainty about the clarified elements.
Frozen T2I Model (Imagen 3) — At any point, the merged prompt can be fed as a detailed image description to an off-the-shelf T2I model that is never fine-tuned or modified; the agent's job is solely to produce better prompts for this model.

Information flows in a loop: initial prompt → belief graph → question selection → user answer → belief transition (updated prompt + updated belief graph) → repeat until the budget of interaction turns is exhausted or the agent's uncertainty is sufficiently low → final merged prompt → T2I generation.

3.3 Roadmap for the Deep Dive

First, the formal agent framework (⟨B, A, O, τ, π⟩) and the optimization objective — this establishes the mathematical lens through which all design choices are made and defines what "optimal question-asking" means.
Second, the belief graph data structure in full detail — how entities, attributes, and relations are defined, how probabilities and importance scores are assigned, and why this particular decomposition was chosen over alternatives like raw text uncertainty or end-to-end learned representations.
Third, the belief parsing pipeline — the three specialized LLM-based parsers (entity, attribute, relation) that construct belief graphs from arbitrary text prompts, including the exact prompt instructions that govern what each parser outputs and how probabilities are calibrated.
Fourth, the action selection strategies for the three agent variants — the heuristic scoring function (Ag1), the belief-prompted LLM strategy (Ag2), and the principle-prompted LLM strategy (Ag3) — explaining how each computes which question to ask next and what trade-offs each design embodies.
Fifth, the belief transition mechanism — how user answers are merged into the conversation history, how the belief graph is refreshed, and the specific post-processing logic (redundancy elimination, information retention) that prevents the agent from asking the same question twice.
Sixth, the simulated user for automatic evaluation — how an LLM-based user simulator with access to ground-truth detailed captions enables scalable multi-turn evaluation without human subjects.

3.4 Detailed, Sentence-Based Technical Breakdown

This is a systems and empirical evaluation paper whose core idea is that making an agent's uncertainty about user intent explicit and actionable — through a structured belief graph and principled question-asking strategies — enables multi-turn clarification that substantially improves image-prompt alignment compared to single-turn generation, even when the agent is built entirely from frozen, off-the-shelf models with no fine-tuning.

Formal Agent Framework and Optimization Objective

The paper formalizes the interactive T2I agent as a five-tuple ⟨B, A, O, τ, π⟩ (Appendix B), where:

S is the state space of all possible user intents — for T2I, each state s ∈ S is a specific image the user wants to generate, though S is never explicitly enumerated since user intents are latent and only revealed in hindsight.
B is the belief state space — the set of all possible representations the agent can have about what the user wants. In the implementation, each b ∈ B has two complementary representations: (i) a merged prompt, which is a natural-language summary of the entire conversation history, and (ii) a belief graph, which is a structured symbolic representation parsed from that merged prompt containing entities, attributes, relations, probabilities, and importance scores.
A is the action space — the set of all possible actions the agent can take. In this paper, actions are primarily clarification questions (though the framework also allows for direct belief graph presentation as an action, and potentially image generation as an action). The action selection strategy π maps from a belief state to an action: π: B → A.
O is the observation space — the set of all possible user responses the agent can receive. Observations include answers to clarification questions and direct edits to the belief graph through the agent interface.
τ: B × A × O → B is the belief transition function — given the current belief, the action taken, and the observation received, this function produces an updated belief. In the implementation, τ works by merging the new Q&A pair into the conversation history, generating an updated merged prompt, and re-parsing the belief graph from that updated prompt.
π: B → A is the action selection strategy — the policy that decides which question to ask given the current uncertainty. This is where the three agent variants (Ag1, Ag2, Ag3) differ.

The paper also defines a score function f: B × S → ℝ that evaluates the alignment between the agent's belief and the true user intent at any turn. This function is available only in hindsight (once the ground truth image is known) and is used for evaluation, not for the agent's internal decision-making. The agent's objective is to maximize f with as few interaction turns as possible — a trade-off between image quality and user effort.

The key insight embedded in this formalism is that the agent doesn't need to know S or have access to f during operation. Instead, it works entirely through its belief b, which encodes its current best guess about the user's intent as a probability distribution over possible image configurations. The action selection strategy π converts belief-space uncertainty into concrete information-gathering actions, and the transition function τ shows how each new piece of information incrementally reduces that uncertainty.

The Belief Graph: Structure and Rationale

The belief graph is the central representation that makes proactive questioning tractable. Rather than using hardcoded symbolic predicates like classical AI belief states (Fikes and Nilsson, 1971), the belief graph uses LLMs to adaptively generate structured representations from arbitrary image descriptions (Section 4).

Nodes (Entities). Each node in the belief graph is an entity that may appear in the image. Entities are categorized into three types:

Explicit entities: items directly mentioned in the user's prompt (e.g., "rabbit", "cat" in "A rabbit near a cat"). These typically have probabilities of appearing close to 1.0, since the prompt confirms their presence.
Implicit entities: items not mentioned in the prompt but likely to appear given the scene context (e.g., "fork" for a breakfast scene, "collar" for a pet image). These have probabilities between 0 and 1, reflecting the agent's uncertainty about whether they belong in the user's intended image and whether they should be generated. This is a critical design choice: by surfacing implicit entities, the agent can elicit information about scene elements the user wants but didn't think to specify.
Background entities: contextual elements that shape the image's overall character but aren't foreground objects — image style (photorealistic, cartoon, 3D render), time of day, weather, location, mood/atmosphere, camera angle. These are listed as entities (not just attributes) because they fundamentally determine what kind of image is generated, even though they don't correspond to objects in the scene.

Each entity carries metadata: a name (unique identifier), a description (short textual description), an entity_type (explicit, implicit, or background — no other values are allowed, enforced by the parser prompt), a probability_of_appearing (a float between 0 and 1), and an importance_to_ask_score (a float between 0 and 1).

The probability_of_appearing follows specific assignment rules encoded in the entity parser prompt (Appendix D.5):

If the prompt says an entity definitely does not exist, assign 0.0 probability.
If the prompt indicates an entity definitely exists, assign 1.0 probability.
If the prompt is silent about an entity's existence, assign a value between 0 and 1, higher if the entity is more likely given the scene context.

The importance_to_ask_score measures how much asking about this entity would reduce overall uncertainty about the user's intent. The parser prompt specifies that scores should be increased for entities that are primary subjects, entities that strongly influence the layout or portrayal of other entities, and implicit entities whose presence "can significantly impact the image." Scores should be decreased for entities "that are already well specified in the prompt."

Attributes. Each entity has a list of attributes — properties that describe the entity's appearance or character. For example, the "rabbit" entity might have attributes for color, breed, age, expression, and size. Each attribute has:

A name (string identifier).
An importance_to_ask_score (float 0–1): the parser prompt instructs the LLM to assign higher scores to attributes that are the primary characteristics of important entities and attributes that could strongly influence the generation of other attributes. For example, "breed of a dog would impact other attributes like color, size, etc. So the breed attribute should have a higher importance score than color, size, etc."
A candidates list: a mapping from possible values to their estimated probabilities. For example, {Dutch: 0.20, Mini Lop: 0.15, Netherland Dwarf: 0.15, ...} for rabbit breed. The probabilities must sum to 1.0. If the user's prompt already specifies a value (e.g., "white rabbit"), that candidate receives probability 1.0 and all others receive 0.0, and the importance score is reduced since there's no uncertainty to resolve.

The candidates list is what enables the agent to produce multiple-choice questions: it can surface the 3–5 most probable values as options, allowing users to select rather than type responses (implementing the Easy-to-Answer principle).

Relations. Relations are a special case of attributes that connect two entities rather than describing one. Each relation has:

A name (in the format entity1-entity2).
A description (textual description of the relationship).
A spatial_relation map: possible spatial relationships between the two entities mapped to probabilities (e.g., {above: 0.8, below: 0.0, "left of": 0.1, "right of": 0.1}). Standard values include "above", "below", "in front of", "behind", "left of", "right of", "part of", "under", "overlap", and "in the center of".
An importance_to_ask_score: higher if the two entities are very important, the relation between them is very unclear, and the relation significantly impacts the image layout.
name_entity_1 and name_entity_2 (the two entities involved).
is_bidirectional (boolean): whether the relation is symmetric.

The relation representation is what allows the agent to ask spatial questions like "Are the rabbit and cat close to each other?" or "Is the fork in the center of the table?" — questions that go beyond entity presence/attributes to capture the compositional structure of the scene.

What makes belief graphs novel compared to scene graphs. The paper explicitly positions belief graphs against Davidsonian Scene Graphs (DSG) from Cho et al. (2023), which decompose an image description into atomic propositions for evaluation. The key differences are:

Uncertainty: DSGs represent what is in an image description; belief graphs represent what might be in the image, with explicit probability distributions over entity presence, attribute values, and relation types. This makes belief graphs suitable for driving information-gathering rather than just evaluation.
Importance scores: DSGs have no notion of which propositions matter more for image quality; belief graphs explicitly encode importance to prioritize questions that reduce consequential uncertainty.
Adaptive generation: DSG parsers are typically rule-based; belief graphs are generated by LLMs with in-context learning, making them applicable to arbitrary prompts without pre-defined predicate vocabularies.
Implicit entities: DSGs capture only what's mentioned; belief graphs anticipate unmentioned entities the user might want, enabling the agent to proactively surface possibilities the user hadn't considered.

The relationship between belief graph probabilities and the T2I generation process. The paper is careful to note that belief graph probabilities do not correspond to the T2I model's internal distribution over outputs given an underspecified prompt. Instead, they represent the agent's uncertainty about what should be in the image from the user's perspective — essentially, the agent's estimate of what the user would want if they fully specified the scene. This distinction is important because it means the belief graph is trying to model user intent, not model behavior, which justifies the use of LLMs (which have world knowledge about typical scenes and cultural associations) rather than requiring access to T2I model internals.

Belief Parsing Pipeline: From Text to Structured Uncertainty

The belief parsing pipeline converts a natural-language prompt (or merged conversation summary) into a complete belief graph through three sequential LLM-based parsers, all implemented via in-context learning (ICL) with Gemini 1.5 Pro (Appendix D.1). Each parser receives a detailed prompt instruction with multiple input-output examples and outputs structured JSON.

Entity Parser (Appendix D.5). The entity parser takes a single input — the user_prompt (which may be the initial prompt or a merged prompt accumulating interaction history) — and outputs a JSON list of entity objects. The prompt instruction (reproduced verbatim in the appendix) guides the LLM to identify three categories of entities:

Explicit Entities: "List all clearly stated entities within the prompt (people, objects, animals, locations, etc.)."
Implicit Entities: "Identify potential entities that are implied or strongly suggested by the prompt, even if not explicitly mentioned."
Background Entities: "Deduce relevant background elements which could impact the image generation from the prompt or context, including: Weather, Location, Time of Day, Mood or Atmosphere."

For each entity, the LLM must produce a JSON object with name, importance_to_ask_score, description, entity_type, and probability_of_appearing. The prompt instruction includes detailed rules for assigning these values — for example, "If the prompt says an entity does not exist, assign a 0.0 probability. Because the entity does not exist, you should also assign 0 to importance_to_ask_score of this entity" and "If the prompt does not say anything about the existence of the entity, assign a probability between 0 and 1. This probability is higher if the entity is more likely to appear in the image given the context specified by the prompt."

The instruction also embeds a bias toward asking about implicit entities: "significantly increase the score for implicit entities that are likely to appear in the image and their appearance can significantly impact the image." Conversely, it deprioritizes fully specified elements: "significantly decrease the score for entities that are already well specified in the prompt."

The prompt includes several few-shot examples that demonstrate the expected behavior. For instance, given the prompt "generate an image of a lionhead rabbit running on grass with sun shining. There is no trees in the background," the example output includes:

"rabbit" with probability_of_appearing: 1.0 and importance_to_ask_score: 0.5 (explicit, but breed/color unspecified so moderate importance)
"tree" with probability_of_appearing: 0 and importance_to_ask_score: 0 (explicitly negated, no need to ask)
"camera angle" with probability_of_appearing: 1.0 and importance_to_ask_score: 0.8 (background entity, always present but unspecified, high importance because it significantly impacts the image)
"image style" with probability_of_appearing: 1.0 and importance_to_ask_score: 1.0 (background entity, maximum importance because "the style of the image" is a fundamental determinant of output)

This construction reveals the design philosophy: entities that are always present but never specified get the highest importance scores because they represent large sources of unresolved uncertainty that strongly affect the final image.

Attribute Parser (Appendix D.6). For each entity produced by the entity parser, the attribute parser takes three inputs — the user_prompt, the entity (specifically entity.name), and a list of other_existing_entities (to prevent listing other entities as attributes of the current entity) — and outputs a JSON list of attribute objects. Each attribute object includes name, importance_to_ask_score, and candidates (a dictionary mapping candidate values to probabilities).

The attribute parser prompt includes critical instructions about candidate generation: "Make sure to generate at least 5 or more possible values. These should be realistic for the given entity." The probabilities over candidates must sum to 1.0, and the LLM is instructed to "infer the probability based on the prompt. For example, for a dog with breed Samoyed, the color attribute has a very high probability of white" — demonstrating that the parser should use world knowledge to set priors over attribute values.

If the user prompt already specifies an attribute value, the parser outputs only that single candidate with probability 1.0 and is supposed to (though does not always, as seen in Ag1's error cases) assign a lower importance score. The instruction also prevents the parser from including "other entities in the attributes" — e.g., when parsing attributes of "rabbit," it should not output "grass" as an attribute.

The importance score assignment follows rules similar to those for entities: increase for attributes that are "primary attributes of an important entity," "significantly increase the score for attributes that could strongly influence the generation or portrayal of OTHER attributes in the scene," and decrease for "attributes that are already well specified in the prompt."

Relation Parser (Appendix D.7). The relation parser takes the user_prompt and a list of entity_names (all entity names from the entity parser) and outputs a JSON list of relation objects. Critically, the parser is instructed to "Ignore entity pairs without relations" — it only outputs relations when there is a meaningful connection between two entities.

Each relation object includes name (formatted as "entity1-entity2"), description, spatial_relation (a map from relation types to probabilities summing to 1.0), importance_to_ask_score, name_entity_1, name_entity_2, and is_bidirectional.

The spatial_relation map uses standard relation types: "above", "below", "in front of", "behind", "left of", "right of" — these provide the vocabulary for spatial questions. If the prompt already specifies a spatial relation (e.g., "rabbit sitting on grass"), the corresponding relation type gets probability 1.0 and all others get 0.0, and the importance score should be low (0.1 in the example).

The importance score for relations is driven by a different set of considerations: "Assign a higher score if the two entities are very important, the relation between them is very unclear, and the relation is very important for the layout of the image." This means that a relation between two implicit background entities would get a low importance score even if highly uncertain, while a relation between the two main subjects would get a high score if unspecified.

LLM configuration. All three parsers use Gemini 1.5 Pro (Gemini Team Google, 2024) with the default temperature and a 32K context length (Section 7.1.1). The choice of Gemini 1.5 is pragmatic: its long context window enables the inclusion of extensive few-shot examples and conversation histories. The default temperature is used (not temperature 0), meaning the parser outputs are stochastic — the same prompt may produce slightly different entity lists or probability distributions across runs. The paper does not report experiments on parsing consistency or its impact on downstream question quality.

Why this pipeline design? The three-parser decomposition (entities → attributes → relations) is chosen over a single end-to-end parser for several reasons:

Modularity: each parser can be improved, replaced, or fine-tuned independently as better LLMs become available or as domain-specific training data is collected. The paper's Section 8 explicitly frames this as a benefit of the overall modular design.
Structured output control: by breaking the task into separate parsing steps with focused prompts, each prompt instruction can be more specific about its expected output format and decision rules, reducing the likelihood of format errors compared to asking a single LLM to output a complete nested belief graph in one call.
Sequential dependency: the attribute parser needs to know which entities exist before it can generate attributes for each; the relation parser needs the full entity list to identify entity pairs — the sequential design enforces this information flow explicitly rather than relying on a single LLM to internally maintain consistency.
Interpretability: intermediate parser outputs can be inspected and debugged separately, which is valuable for a prototype system where understanding failure modes is as important as achieving high performance.

The primary cost of this design is parsing latency and LLM API calls — constructing a full belief graph requires at minimum one entity parser call, plus one attribute parser call per entity, plus one relation parser call, which for a scene with 10 entities means roughly 12 LLM API calls per belief graph construction. The paper does not report the end-to-end latency of this pipeline, which would be important for real-time interactive applications.

Action Selection: The Three Agent Question-Asking Strategies

The action selection strategy π is where the three agent prototypes diverge. All three agents share the same belief graph construction pipeline and the same belief transition mechanism; they differ solely in how they decide which question to ask next.

Ag1: Heuristic Score Agent (MHIS strategy). Ag1 selects questions by computing an explicit heuristic score over every element in the belief graph and choosing the highest-scoring element (Section 6.1, Appendix D.2). The heuristic scoring function (Equation 1 in the paper) for an attribute question about entity e with attribute a that has candidate list c is:

$\max_{e,a,c,r} \left( \text{IS}(e) \cdot \text{IS}(a) \cdot P(e) \cdot \text{Ent}(c), \quad \text{IS}(r) \cdot P(r) \cdot \text{Ent}(c) \right)$

where IS(e) is the importance-to-ask score of the entity (a float in [0,1] from the entity parser), IS(a) is the importance-to-ask score of the attribute (a float in [0,1] from the attribute parser), P(e) is the probability of the entity appearing (a float in [0,1] from the entity parser), Ent(c) is the entropy of the candidate probability distribution (computed as H(c) = -Σ p_i log p_i from the attribute parser's candidate list), IS(r) is the importance-to-ask score of a relation, and P(r) is related to the probability of the relation.

What this equation computes: for every entity-attribute pair and every relation in the belief graph, compute the product of (a) how important the entity is, (b) how important this specific attribute is, (c) how likely the entity is to appear (you don't want to ask about entities that probably don't exist), and (d) how uncertain you are about the attribute's value (entropy of the candidate distribution, which is maximal at 0.5/0.5 and minimal at 1.0/0.0). Take the maximum over all such products. The element with the highest product becomes the target of the next question.

Why this form: the multiplicative combination means that all four criteria must be non-zero for a question to be selected. An attribute of a very important entity that is completely certain (entropy = 0) gets score 0 — correctly, because there's no information to gain. An attribute of an unimportant implicit entity that is highly uncertain gets low score because IS(e) and IS(a) are small — correctly, because the uncertainty doesn't matter much for the final image. The entropy term pushes the agent toward attributes where the candidate distribution is flat (high uncertainty) over those where one candidate dominates. The parallel treatment of attributes and relations ensures the agent can ask spatial questions when relations are both important and uncertain.

Once the highest-scoring element is identified, the question is verbalized by a separate LLM call (Appendix D.12). This verbalization prompt takes the entity name, attribute name, candidate list, and entity type, and produces a multiple-choice question. For example, "What color of the rabbit do you have in mind? a. black, b. white, c. brown. d. unknown. If none of these options, what color of the rabbit do you have in mind?" The prompt provides multiple examples covering explicit entities (multiple-choice question with options from candidates), implicit entities (yes/no question about whether the entity should be included), and relations (question about spatial arrangement with direction options).

The advantage of Ag1 is that its question selection is interpretable and auditable: you can inspect the heuristic scores and see exactly why a particular question was chosen. The disadvantages include: (1) the additive structure assumes independence between elements when in reality asking about one attribute might partially resolve another (e.g., asking about rabbit breed might partially resolve color), (2) the importance scores and probabilities are themselves LLM estimates and may be miscalibrated, (3) the hardcoded score function cannot adapt to conversation patterns or learn from the effectiveness of past questions, and (4) as shown in the results (Figure 5), this approach can generate redundant questions — asking whether "cake" exists when it was mentioned in the initial prompt — because the LLM parser might assign the "cake" entity a probability less than 1.0 (due to parsing error) and a high importance score, causing the heuristic to select it despite the redundancy.

Ag2: Belief-Prompted Agent (AICQ_B strategy). Ag2 abandons the explicit scoring function in favor of directly prompting an LLM to generate the next question using the belief graph, merged prompt, and conversation history as context (Section 6.2, Appendix D.11). The prompt instruction (AICQ_B) provides the LLM with:

The user prompt (initial description).
A structured text dump of the entire belief graph, including for each entity its name, description, importance score, probability of appearing, and a list of all attributes with their candidate distributions and importance scores. For example: "Attribute Name: color, Importance to ask Score: 0.9, Candidates: [white: 0.5, black: 0.5] means the color is either white or black, each with 0.5 probability. If you ask about attributes, you should ask about the attribute with the highest uncertainty. Your uncertainty can be judged by the probabilities."
The full conversation history enclosed in <dialogue> tags, with explicit instructions to NOT ask any question that has been asked before.
Meta-instructions: "Based on the user prompt and the belief of the agent, please provide a question to ask about the image. The question should be as concise and direct as possible. The question should aim to obtain the most information about the style, entities, attributes, spatial layout and other contents of the image. Remember to ask for information that are critical to knowing the critical details of the image that is important to the user."
A final instruction preserving some flexibility: "Your question does not have to be entirely decided by the belief. You can construct any question that make yourself more confident about what the image is."

The LLM must output its question within <question> and </question> markers and is instructed to "Think step by step and reason about your uncertainty of the image to generate" before producing the question.

The critical difference from Ag1: Ag2 delegates both question selection (which topic to ask about) and question formulation (how to phrase it) to the LLM, which has access to the full conversation history and belief graph simultaneously. This allows the LLM to avoid redundancy by checking whether a topic has already been discussed (the prompt says "DO NOT ask any question that has been asked in the dialogue above"), to ask questions that span multiple belief graph elements (e.g., "What color and breed is the rabbit?" rather than two separate questions), and to produce questions that are not limited to multiple-choice format.

The advantage is that Ag2 can leverage the LLM's natural language understanding to generate more coherent, contextually appropriate questions that better satisfy the Redundancy and Relevance principles. The disadvantage is that it relies on the LLM correctly interpreting the importance scores and probability distributions in the belief graph — the paper notes that "Ag2 excessively relies on importance scores in beliefs to ask questions, and if the importance scores are not estimated properly, the quality of the questions decreases" (Section 7.2). Additionally, because the LLM must digest the entire belief graph in context, this approach is limited by context window size for scenes with very many entities.

Ag3: Principle-Prompted Agent (AICQ_base strategy). Ag3 represents the most radical design: it does not use the belief graph at all for question generation (Section 6.3, Appendix D.10). Instead, it prompts an LLM with only the conversation history and the four question-asking principles from Section 5.2: (i) No Redundancy, (ii) Uncertainty Reduction, (iii) Relevance, and (iv) Easy-to-Answer.

The prompt for the first question (when there is no conversation history) is: "Based on the original prompt please provide a question to ask about the image. The question should be as concise and direct as possible. The question should aim to learn more about the attributes and contents of the image, the objects, the spatial layout, and the style."

For subsequent questions, the prompt becomes: "Based on the chat history please provide a new question to ask about the image. The chat history is as follows and is enclosed in <chat_history> and </chat_history> markers. The question should be as concise and direct as possible. The question should aim to learn more about the attributes and contents of the image, the objects, the spatial layout, and the style."

What Ag3 explicitly does NOT have access to: the belief graph, importance scores, probability distributions, entity lists, or any structured uncertainty representation. It relies entirely on the LLM's implicit understanding of what constitutes "important" information to ask about, inferred from the prompt and conversation.

Why this design is included: Ag3 serves as an ablation that tests whether the explicit belief graph representation actually matters for question quality, or whether a sufficiently capable LLM can implicitly perform the same information-gathering function from conversation history alone. The surprising result (Table 1, Figure 3) — that Ag3 outperforms both Ag1 and Ag2 across all datasets and most metrics — is one of the paper's most interesting findings, suggesting that current LLMs may be better at intuiting which questions matter than explicit belief graph computations can capture.

The paper hypothesizes that Ag2's underperformance relative to Ag3 may stem from Ag2's excessive reliance on the importance scores encoded in the belief graph: if those scores are poorly calibrated (e.g., overestimating the importance of certain attributes while underestimating others), the LLM following them will ask suboptimal questions, whereas Ag3's LLM can learn from the conversation flow which topics are actually productive.

Belief Transition: Updating Uncertainty After User Feedback

After the user answers a clarification question, the agent must update its belief state to reflect the new information. The belief transition function τ operates in two stages (Appendix D.4, D.8, D.9):

Stage 1: Summarization and merging. Two sequential LLM calls handle the integration of new information:

Verbalization (Appendix D.8): The Q&A pair from the current turn is converted into a single declarative sentence. The prompt instruction is: "Turn the question and action into a single declarative sentence that describes the answer - do not phrase it as a question. Example output: the firetruck in the image is red." This step ensures that user answers are integrated as factual assertions rather than dialog fragments.
Prompt merging (Appendix D.9): The existing merged prompt (containing all prior interaction history) is combined with the new declarative sentence into an updated comprehensive prompt. The merge instruction is: "You are writing a prompt for a text-to-image model based on user feedback. The original prompt is {prompt}. The user has provided some additional information: {additional_info}. Please write a new prompt for the text-to-image model. The new prompt should be a meaningful sentence or a paragraph that combines the original prompt and the additional information. Do not add any new information that is not mentioned in the prompt or the additional information. Make sure the information in the original prompt is not changed. Make sure the additional information is included in the new prompt."

This merging step is crucial: it produces a natural-language prompt that can be directly fed to the frozen T2I model (Imagen 3) at any point, and it serves as the input for the next round of belief parsing. The merged prompt gradually accumulates all specified details — starting from "A rabbit near a cat" and growing to "A photorealistic image of a Dutch rabbit sitting in front of a black cat on a wooden floor in a sunlit room, with the rabbit facing left and the cat facing right."

Stage 2: Belief graph re-generation. The updated merged prompt is fed back through the full belief parsing pipeline (entity parser → attribute parser → relation parser). This produces a fresh belief graph reflecting the agent's updated uncertainty: attributes that were clarified by the user now have a single candidate with probability 1.0 and (ideally) reduced importance scores; entities that the user confirmed as present or absent have updated probabilities; implicit entities that the user didn't mention in their answer retain their uncertainty.

Ag1-specific post-processing. Ag1 applies additional post-processing to the re-generated belief graph (Appendix D.4) to enforce memory and prevent the heuristic scorer from repeatedly selecting the same questions:

Redundancy elimination: For any attribute or relation that has been addressed in the conversation history, the corresponding user response is set as the sole candidate with probability 1.0, and its importance score is forced to 0. This ensures the heuristic scoring function from Equation (1) will never select an already-answered question (since the product with importance score 0 is 0).
Information retention: If an attribute or relation from the conversation history is absent in the updated belief graph (e.g., because the re-parsed graph didn't include it), it is explicitly added back. This prevents information loss when the LLM-based belief parser fails to capture something that was previously known.

These post-processing steps are a direct response to the LLM parser reliability problem: because the entity, attribute, and relation parsers use stochastic LLM calls (default temperature) and may produce inconsistent outputs across re-parsing of similar prompts, the agent needs explicit mechanisms to maintain conversation coherence. Without redundancy elimination, Ag1 was observed to ask about "cake" even after the user had discussed the cake (Figure 5) — the parser might assign the newly mentioned "cake" entity a probability of less than 1.0 (perhaps 0.95), and the heuristic would compute a non-zero score for attributes of cake, triggering an unnecessary question.

Ag2 and Ag3 do not use this post-processing because they rely on the LLM's in-context processing of the conversation history (for Ag3) or the conversation history plus belief graph (for Ag2) to avoid redundancy. The prompt instructions for both agents explicitly say "DO NOT ask any question that has been asked in the dialogue above" (Appendix D.11). The paper's experimental results (Figure 4, showing Ag2 and Ag3 having fewer issues with redundancy than Ag1) suggest that the LLM-based redundancy avoidance is more effective than the hardcoded post-processing, though neither approach is perfect.

Simulated User for Automatic Evaluation

A key contribution of the paper is the automatic evaluation framework that enables scalable experimentation without requiring human subjects for every experiment (Section 7.1, Appendix D.3, Algorithm 2). The framework simulates user-agent conversations through LLM self-play, where one LLM instance acts as the agent (asking questions) and another acts as the user (answering them).

Ground truth construction. For each evaluation example, the framework requires:

A goal image I — the image the user intends to generate (ground truth).
A ground truth caption c — a detailed description of I including "spatial layout, background elements, and style." This is either a human annotation (ImageInWords long captions, some COCO ground truth expansions) or an LLM-generated detailed caption (Gemini 1.5 Pro prompted with the goal image for COCO and DesignBench).
An initial prompt p0 — a short, underspecified description of I containing only "the primary elements of the scene." This is either the shortest of human COCO captions, an auto-summarized version of the ImageInWords long caption, or an LLM-generated short description for DesignBench.
A ground truth belief graph b_gt — constructed by running the same belief parsing pipeline on the ground truth caption c.

The deliberate construction of p0 as "intentionally less detailed than c to necessitate multi-turn refinement" is what creates the underspecification problem that the agent must resolve.

Simulated user operation. The simulated user maintains the ground truth caption c and ground truth belief graph b_gt as its internal representation of the intended image. When the agent asks a question, the simulated user:

Takes the ground truth caption, the conversation history, and the simulated user's internal belief graph as input.
Prompts an LLM (using the same AICQ_B prompt format from Appendix D.11) to generate an answer based on the ground truth information.
The LLM is instructed to provide "concise answers to agent questions" — it does not volunteer additional information beyond what is asked, mimicking a real user who answers only the question posed without elaborating on unrelated details.

The simulated user is essentially the inverse of the agent: while the agent tries to ask the most informative question, the simulated user provides answers that are faithful to the ground truth image but limited to the scope of the question asked.

Self-play loop (Algorithm 2). The evaluation proceeds as:

Initialize the agent with p0.
For each turn up to max_turns (set to 15 in experiments):
- The agent selects and verbalizes an action (question).
- The simulated user answers the question using the ground truth caption.
- The agent transitions its belief state using the Q&A pair.
- The agent's merged prompt is recorded as pf.
After the loop (or at any intermediate turn), the agent generates an image from its current merged prompt using Imagen 3.
Metrics are computed comparing the generated image and merged prompt to the ground truth image I and caption c.

Why this evaluation design matters. The self-play framework solves two problems simultaneously:

Scalability: it enables evaluation across hundreds of images (250 from ImageInWords, 250 from COCO, 30 from DesignBench) with 15 interaction turns each, which would be prohibitively expensive and slow with human subjects.
Reproducibility: the simulated user provides deterministic (or at least controlled) answers conditioned on the ground truth, enabling fair comparison across agent variants without the confounding factor of different human response styles.
Grounded evaluation: by anchoring the simulated user's answers to a specific goal image, the framework provides a clear correctness criterion that human evaluation (where different raters might have different aesthetic preferences) cannot.

The primary limitation of the simulated user is that it may not capture the full range of real human answering behavior — for instance, real users might provide additional unsolicited details, might misinterpret questions, might change their mind about what they want during the conversation, or might decline to answer questions they find irrelevant. The human studies in Section 7.3 partially validate the automatic results, but the gap between simulated and real user behavior remains underexplored.

Evaluation Metrics: Quantifying Alignment Between Agent Output and Ground Truth

The paper uses five automatic metrics computed at each turn of the interaction (Section 7.1.3), spanning text-text similarity, image-image similarity, text-image similarity, and belief graph alignment:

Text-Text Similarity (T2T) — Embedding-based. The ground truth caption c and the agent's final merged prompt are each embedded using the Gemini 1.5 Pro embeddings API, and cosine similarity between the two embedding vectors is computed. This metric captures semantic similarity at the caption level — how closely the accumulated conversation summary matches the detailed ground truth description of the image. Values range from -1 to 1, with higher being better. The metric is reported as T2T (EmbedSim) in Table 1.

Text-Text Similarity (T2T) — DSG-based. The Davidsonian Scene Graph (DSG) metric (Cho et al., 2024) is adapted to compare the ground truth caption and agent's merged prompt by parsing both into scene graph representations and computing structural similarity. DSG decomposes text into atomic propositions about entities, attributes, and relations, making it a more fine-grained complement to embedding-based similarity. This metric is reported as T2T (DSG) in Table 1.

Image-Image Similarity (I2I). Cosine similarity between DINOv2 (Oquab et al., 2024) feature embeddings of the ground truth image and the generated image. DINOv2 embeddings capture high-level visual semantics and are commonly used for comparing image content in generative model evaluation. This metric is reported as I2I (DINO) in Table 1. Values range from -1 to 1, with typical values for good matches in the 0.5–0.7 range across the three datasets.

Text-Image Similarity (T2I) — VQAScore. The ground truth caption is compared to the generated image using VQAScore (Lin et al., 2024), which converts the caption into a yes/no visual question ("Does this image show {caption}?") and uses Gemini 1.5 Pro as the underlying multimodal LLM to answer. The score represents the model's confidence that the image matches the description. This metric is the paper's primary measure of image-prompt alignment and is reported as T2I (VQAScore) in Table 1.

Negative Log Likelihood (NLL) of Ground Truth State Under Agent Belief. This metric quantifies how well the agent's belief graph captures the ground truth. The ground truth belief graph b_gt is converted to a "state with no uncertainty" — all probabilities are set to 1.0 or 0.0 based on what actually appears in the ground truth caption. Then, assuming independence of all entities, attributes, and relations, the log probability of the ground truth state under the agent's belief distribution is computed:

$\text{NLL} = -\sum_{e \in \text{entities}} \log P(e \text{ present?} \mid b) - \sum_{a \in \text{attributes}} \log P(a = v_{gt} \mid b) - \sum_{r \in \text{relations}} \log P(r \text{ type?} \mid b)$

where P(e present? | b) is the agent's estimated probability that entity e is present in the image (taken from the entity's probability_of_appearing), P(a = v_gt | b) is the agent's probability assigned to the ground-truth value of attribute a (taken from the candidate distribution), and P(r type? | b) is the agent's probability for the ground-truth relation type. Lower NLL means the agent's belief graph assigns higher probability to the correct configuration.

The paper acknowledges an approximation error in NLL: "This approximation does not account for potential similarities in the names of entities or attributes. This could lead to approximation errors if, for example, the model confuses 'Persian cat' with 'Siamese cat' due to their similar names." The independence assumption also ignores correlations — for instance, knowing the rabbit's breed changes the likely color distribution — so the NLL may not be a perfectly calibrated measure of belief quality. However, as a relative metric for comparing agents, it tracks whether uncertainty is being reduced in the right direction.

Human evaluation metrics (Section 7.3). In addition to automatic metrics, the paper reports human ratings on:

Content alignment: the percentage of cases where the generated image was ranked as "closest to the ground truth image in terms of content" compared to other agents' outputs.
Aesthetics/style alignment: similar ranking but for "style and aesthetics."
Dialog-image correspondence: how well the final generated image matches the "original prompt and the dialogue" (rated on a 1–5 scale from "Very Close" to "Very Different").

The human evaluation setup uses unlabeled images presented in random order with 3 human raters per example, totaling approximately 550 image-dialog pairs per agent rated (Section 7.3).

Summary of Design Choices and Their Justifications

Belief graph over raw text uncertainty: the belief graph decomposes uncertainty into structured elements (entities, attributes, relations) with explicit probabilities, enabling principled question selection through entropy-weighted importance scoring. Raw text uncertainty ("I'm not sure what the user wants") doesn't decompose — you can't compute which specific question would maximally reduce it.
LLM-based parsers over hardcoded predicate vocabularies: classical AI belief states require pre-defined predicate sets (e.g., is_block, on_table) that don't scale to open-ended visual scenes. LLMs with in-context learning can generate entity lists, attribute candidates, and relation types for arbitrary prompts without manual vocabulary engineering.
Multiplicative heuristic score (Ag1) over additive: the multiplicative form IS(e) · IS(a) · P(e) · Ent(c) ensures that if any factor is zero (entity doesn't exist, attribute already specified, importance is zero), the entire score is zero, preventing wasted questions. An additive form could select questions about low-importance but highly uncertain attributes.
Belief-prompted (Ag2) and principle-prompted (Ag3) LLM strategies over purely heuristic (Ag1): LLMs can leverage conversation history to avoid redundancy and can generate open-ended questions that capture more nuanced information than multiple-choice templates. The paper's results show Ag3 dominates, suggesting that the LLM's implicit understanding of image composition and user intent outweighs the benefit of explicit belief graph access, at least with current LLM capabilities.
Frozen T2I model (Imagen 3) rather than fine-tuned model: the modular architecture treats the T2I model as a black-box tool, meaning any improvement in T2I technology automatically improves the agent's final output quality without retraining. This also makes the system evaluable on the specific contribution of the agent's prompts, separate from the T2I model's generation quality.
Simulated user self-play for evaluation over human-only evaluation: enables exhaustive multi-turn testing across hundreds of examples that would be cost- and time-prohibitive with human subjects, while providing reproducible ground truth for comparing agent variants. Human studies validate the automatic results but are reserved for final confirmation rather than iterative development.
15-turn maximum interaction budget: provides sufficient headroom to observe when performance plateaus or degrades (which Figure 3 shows happens around 10 turns for some metrics), while being short enough to simulate across 550+ examples without prohibitive computational cost.
Default temperature for all LLM calls: using non-zero temperature means belief graph parsing and question generation are stochastic, which more realistically reflects production behavior but also introduces variance that the paper does not quantify (no confidence intervals or multiple-run averages are reported for parsing consistency).

4. Key Insights and Innovations

Innovation 1: Explicit Uncertainty Representation as the Missing Interface Between User Intent and Image Generation

The paper's most fundamental conceptual contribution is the argument that the absence of an explicit uncertainty representation is what makes T2I systems frustrating to use, and that making this uncertainty visible — both to the agent (for question selection) and to the user (for direct editing) — transforms the interaction from a guessing game into a collaborative specification process.

This is a framing innovation rather than a technical one. Prior work on T2I interaction operated under an implicit model where the problem was insufficient user skill (users don't know how to write good prompts) or insufficient model capability (models can't follow complex prompts). The dominant responses were either to teach users prompt engineering or to build better prompt-following models. The paper's alternative diagnosis is that the real problem is information asymmetry: the user knows what they want but can't efficiently transmit it because the model provides no signal about which missing information would be most valuable. The user is effectively debugging a black box without error messages.

The belief graph operationalizes this diagnosis into a concrete interface. What makes this distinctive from prior work on model interpretability (which also surfaces model internals) is that the belief graph does not represent the T2I model's internal state or output distribution. The paper is explicit about this (Section 2): "the belief graph does not correspond to the distribution over outputs of the T2I model itself conditioned on the underspecified prompt. Instead, the belief graph is designed to align with the distribution over image-prompts generated by the agent." This is a crucial distinction. The belief graph models the user's intent, not the model's behavior, which means it can encode cultural knowledge, common-sense expectations about scenes, and aesthetic norms that a T2I model's internal representations may not capture. It also means the belief graph can be generated entirely from language (via LLM prompting) without access to the T2I model's weights or activations.

The significance beyond performance is that this framing opens up a new design space for human-AI interaction: rather than trying to make models that guess correctly the first time, build systems that explicitly track what they don't know and collaborate with users to resolve it. The belief graph is one instantiation of this principle, but the principle is broader — it suggests that uncertainty visualization and uncertainty-driven dialog are under-explored tools for AI alignment in general, not just for T2I. The paper's human study results lend credibility to this argument: 88% of users found entity graphs "could help" / "likely" / "very likely" to help their workflow (Table 2), and 86% said the same for relationship graphs, even though they were presented as static interface mockups rather than fully functional tools.

Evidence anchor: Table 2 (perceived helpfulness of belief graph features from 143 human raters) and the contrast in Section 2 between the belief graph's intent-modeling purpose and the T2I model's output distribution.

Innovation 2: The Counterintuitive Finding That Explicit Belief Access Is Not Necessary — and May Be Harmful — for LLM-Powered Question Generation

Perhaps the paper's most intellectually provocative result is that the principle-prompted agent (Ag3), which has no access to the belief graph and relies solely on conversation history plus high-level question-asking principles, substantially outperforms both the heuristic-scoring agent (Ag1) and the belief-prompted agent (Ag2) across nearly all metrics and datasets (Table 1, Figure 3). This is not an incremental finding — it challenges the paper's own premise that explicit structured uncertainty is the key to effective clarification.

The field's default assumption, explicitly baked into the paper's design, was that explicit uncertainty representation would enable better question selection: compute importance-weighted entropy over entities/attributes/relations, identify the most valuable information to acquire, and ask about it. This is the logic behind Ag1's heuristic scoring function (Equation 1) and Ag2's belief-prompted question generation. It mirrors standard approaches in active learning and Bayesian experimental design, where an explicit uncertainty model drives the selection of information-gathering actions (Cohn et al., 1996; Houlsby et al., 2011; Chaloner and Verdinelli, 1995).

Ag3 violates this assumption entirely. It does not compute importance scores, does not measure entropy over candidate distributions, and does not have access to a structured representation of what entities, attributes, or relations the agent is uncertain about. It prompts an LLM with only the conversation history and a brief instruction to "learn more about the attributes and contents of the image, the objects, the spatial layout, and the style" (Appendix D.10). The LLM must implicitly infer what's important to ask about, what's already been covered, and what remains uncertain — all from the raw dialog text.

Why does Ag3 work better? The paper offers a hypothesis grounded in the limitations of explicit belief representation (Section 7.2): "When inspecting the reasoning steps of Ag2, we found that Ag2 excessively relies on importance scores in beliefs to ask questions, and if the importance scores are not estimated properly, the quality of the questions decreases." In other words, the belief graph encodes a miscalibrated model of what matters. The importance scores are themselves LLM estimates (generated by the entity and attribute parsers) and may be wrong in systematic ways — overestimating the importance of certain entity attributes while underestimating others, or failing to capture emergent importance that depends on the specific conversation trajectory rather than the scene in isolation. When Ag2 feeds these noisy importance scores to the question-generating LLM, it constrains the LLM toward asking about things that the belief graph thinks are important, even when a more contextually intelligent question would target something else.

Ag3, by contrast, gives the LLM maximum flexibility to use its full conversational understanding — including the ability to recognize what the user has emphasized, what details seem to be causing confusion, and what aspects of the scene remain conspicuously underspecified after several rounds of dialog. This flexibility appears to outweigh the structural guidance that the belief graph provides, at least with current LLM capabilities (Gemini 1.5 Pro). The finding suggests a threshold effect: explicit uncertainty models help when the underlying question generator is weak (e.g., a simple heuristic scorer like Ag1), but they become a constraint — a form of premature specification — when the question generator is a powerful LLM that can infer uncertainty more holistically from conversational context.

This negative result has significant implications for future research. It suggests that the primary bottleneck in proactive clarification systems may not be better uncertainty modeling, but rather better conversational reasoning — the ability to track what information has been exchanged, identify implicit gaps, and formulate questions that efficiently fill those gaps. It also raises questions about when and why explicit symbolic representations are beneficial versus constraining in LLM-based systems, a debate that extends well beyond T2I to planning, reasoning, and tool use.

Evidence anchor: Table 1 — Ag3 achieves the highest scores across all datasets and metrics (e.g., VQAScore of 0.955 on DesignBench vs. 0.855 for Ag2 and 0.685 for Ag1). Figure 3 (NLL panel) — Ag3's NLL drops dramatically faster than Ag2's, indicating more efficient uncertainty reduction despite (or because of) not having explicit access to the belief graph. Figure 5 (qualitative dialog) — Ag2 asks reasonable but narrow questions ("What color would you like the cake to be?"), while Ag3 asks broader, more open-ended questions that elicit richer information ("What types of berries are on the cake and how are they arranged?").

Innovation 3: Quantitative Evidence That Multi-Turn Clarification Achieves a Qualitative Leap in Alignment — and That the Gains Materialize Within Practically Viable Interaction Budgets

While the idea that clarification questions improve alignment is intuitive, the paper provides the first systematic, multi-metric, multi-dataset quantification of just how large the gains are and how quickly they accrue. This transforms what could have been a vague design intuition into a claim with practical deployment implications.

The headline finding — at least 2× higher VQAScore than single-turn T2I within 5 turns (Section 1) — understates the magnitude of the improvement when examined across metrics and datasets. On DesignBench (Table 1), Ag3 achieves a VQAScore of 0.955 compared to 0.353 for the single-turn baseline — a 2.7× improvement. The DSG text-text similarity metric jumps from 0.607 to 0.949, indicating that the agent's accumulated prompt captures the compositional structure of the ground truth description almost perfectly after 15 turns. The NLL metric drops from 320.9 to 257.4 on DesignBench, quantifying the direct reduction in belief-state uncertainty.

What's distinctive about the paper's approach to quantifying these gains is the turn-by-turn analysis (Figure 3), which reveals non-trivial dynamics that aggregate numbers obscure. The T2T (embedding similarity) and T2I (VQAScore) metrics plateau or slightly decrease after approximately 10 turns, while I2I (DINO similarity) continues to improve. This divergence suggests an important subtlety: after a certain point, additional questions add detail that improves visual similarity to the ground truth image (captured by DINO) without substantially changing the overall semantic similarity of the prompt text (captured by T2T). The NLL metric continues to drop across all turns, showing that the agent's internal uncertainty keeps decreasing even when the downstream metrics saturate. This pattern has practical implications: there's a point of diminishing returns for text-level alignment around 10 turns, but image-level alignment continues to benefit from further clarification for at least 15 turns.

The practical viability of these interaction budgets matters. The paper shows that most of the gain is achieved within 5 turns (roughly the steepest part of the curves in Figure 3), and that human raters evaluate agent-generated questions as having issues in only 14% or fewer cases (Figure 4, left). This suggests that a real user could achieve substantially better image alignment with approximately 5 back-and-forth exchanges — a burden that human study participants rated as acceptable (91% said clarifications would be helpful to their workflow, Table 2). The 15-turn maximum used in experiments should thus be understood as an upper bound for exploring asymptotic behavior, not as a recommended interaction length.

The significance beyond raw performance is that these results establish an empirical baseline for what's achievable with multi-turn T2I clarification using frozen, off-the-shelf models (Gemini 1.5 Pro for dialog, Imagen 3 for generation). Future systems that fine-tune models on multi-turn image-text dialog data (Section 8, Future Work) can be evaluated against this baseline to determine whether the additional training cost yields proportional gains. The paper has essentially established that zero-shot LLM prompting is already good enough to deliver substantial user value, which changes the calculus for whether the multi-turn clarification problem requires specialized training or can be solved with general-purpose models.

Evidence anchor: Table 1 (absolute and relative metrics across all three datasets), Figure 3 (turn-by-turn trajectories showing plateau behavior), Figure 4 (human ratings of question quality and dialog-image correspondence), Table 2 (91% of users find clarifications helpful).

Innovation 4: DesignBench as a Purpose-Built Benchmark That Exposes the Gap Between Standard Captioning Datasets and the Needs of Creative T2I Users

The paper introduces DesignBench, a carefully curated benchmark of 30 scenes "specifically designed" to evaluate T2I agents in the context of "artists and designers" — users for whom aesthetic detail, stylistic control, and compositional precision are paramount rather than incidental (Section 7.1.2). This is a resource contribution with conceptual implications: it identifies a systematic blind spot in standard evaluation datasets and proposes a concrete remedy.

Standard T2I evaluation datasets like COCO Captions (Lin et al., 2014) and even the more detailed ImageInWords (Garg et al., 2024) are predominantly photorealistic and descriptive: they capture scenes as they appear in photographs, with captions that describe what's visually present. This biases evaluation toward literal object recognition and scene composition, implicitly treating the T2I task as "reproduce a photograph from a description." But a substantial fraction of T2I users — particularly professional artists, designers, and creative practitioners — are not trying to reproduce photographs. They're trying to create imagined scenes with specific artistic styles, visual metaphors, compositional tension, and aesthetic effects that go beyond literal object description.

DesignBench addresses this gap by including "a mix of human generated cartoon graphics, photorealistic yet improbable scenes, and artistic photographic images" (Section 7.1.2). The inclusion of "improbable scenes" — images that could not be photographs because they depict physically impossible or stylized compositions — is particularly important because it tests whether the agent can elicit and represent creative intent (what the user imagines) rather than merely descriptive accuracy (what a camera would capture). The difference between "a rabbit near a cat" (a photographable scene) and "a rabbit playing chess with a cat in a surrealist painting style" (a creative scene) is not just more entities and attributes — it's a fundamentally different kind of specification that involves artistic conventions, cultural references, and compositional choices.

The benchmark's construction methodology also represents a practical advance: each scene follows a standardized (I, p0, c, b_gt) format where the initial prompt p0 is intentionally less detailed than the ground truth caption c, creating a controlled underspecification gap that all agents must resolve. The paper validates the quality of its ground truth captions by showing that T2I VQA similarity between c and I has a mean of 0.999 and a median of 1.0, with a standard deviation of 4.5e-07 (Section 7.1.2) — effectively perfect alignment. This means that if an agent could somehow produce the ground truth caption through its questioning, the T2I model would (with near-certainty) generate the target image, making the benchmark a clean test of the interaction, not of the generation model's capabilities.

The significance beyond the resource itself is that DesignBench highlights a broader evaluation principle: benchmarks for interactive AI systems should be designed around user populations and use cases, not just around available data. COCO exists because human annotators were asked to describe photographs, not because those descriptions represent the full range of what T2I users want. By explicitly targeting the needs of artists and designers as a user group, DesignBench models a user-centered approach to benchmark construction that could be replicated for other AI interaction domains (medical image generation for radiologists, architectural visualization for designers, educational illustration for teachers).

Evidence anchor: Section 7.1.2 (dataset description, composition, and validation), Table 1 (DesignBench results column showing consistent patterns across agents, with Ag3 achieving 0.955 VQAScore and 0.949 DSG), and the dataset release itself on HuggingFace.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper evaluates on three datasets: COCO Captions (validation split, 250 images, where the shortest human caption per image serves as the underspecified initial prompt p0 and an expanded Gemini 1.5 Pro caption serves as the ground truth c), ImageInWords (Garg et al., 2024; a diverse set of realistic and cartoon images with dense human-annotated captions — the long human annotation is used as ground truth c and an auto-summarized version serves as p0), and DesignBench (a newly curated benchmark of 30 scenes with human-generated cartoon graphics, photorealistic improbable scenes, and artistic photographic images, where Gemini 1.5 Pro generates both p0 and c from the goal image). For all three, each data point is a tuple (I, p0, c, b_gt) where I is the target image, p0 is the intentionally underspecified starting prompt, c is a detailed ground truth caption, and b_gt is the ground truth belief graph parsed from c.
Base model(s). All agents use Gemini 1.5 Pro (Gemini Team Google, 2024) with default temperature and a 32K context length for belief parsing, question generation, and user simulation. Image generation uses Imagen 3 (Baldridge et al., 2024) as a frozen, off-the-shelf T2I model called via the Vertex API. Neither the LLM nor the T2I model is fine-tuned — the entire system operates through in-context learning and prompting. The choice of Gemini 1.5 Pro is motivated by its long-context capabilities (enabling multi-turn conversation history and extensive few-shot parser prompts) and its recency at the time of development.
Metrics. Five automatic metrics are computed at each turn of the 15-turn interaction: (1) T2T (EmbedSim) — cosine similarity between Gemini 1.5 Pro embeddings of the ground truth caption c and the agent's merged prompt, (2) T2T (DSG) — Davidsonian Scene Graph similarity (Cho et al., 2024) between c and the merged prompt, parsing both into atomic propositions, (3) I2I (DINO) — cosine similarity between DINOv2 (Oquab et al., 2024) embeddings of the ground truth image I and the generated image, (4) T2I (VQAScore) — VQAScore (Lin et al., 2024) using Gemini 1.5 Pro as the VQA model to assess how well the generated image matches the ground truth caption c, and (5) NLL — approximate negative log likelihood of the ground truth belief graph b_gt under the agent's current belief graph b, computed by assuming independence of entities, attributes, and relations and summing their log probabilities (lower is better). For human evaluation, raters provide (6) Human Eval (Content) and (7) Human Eval (Aesthetics) — the percentage of cases where each agent's generated image was ranked as closest to the ground truth image in terms of content or aesthetics/style, respectively.
Baselines. The primary baseline is T2I — a standard single-turn T2I model that generates an image directly from the initial prompt p0 without asking any questions, no updates to the prompt. This represents the default user experience the paper aims to improve upon. No other multi-turn interactive baselines (e.g., DialogGen from Huang et al., 2024, or prompt expansion from Datta et al., 2024) are systematically compared in the quantitative experiments — the paper evaluates variants of its own agent design against the single-turn baseline and against each other.
Generation budget / compute accounting. Interaction is measured in turns of the agent-user conversation loop (Algorithm 2). All agents are run for a maximum of 15 turns. At each turn, the agent asks one question and receives one answer — the question and answer are merged into the accumulated prompt, and a new image can be generated at any point using Imagen 3 from the current merged prompt. There is no token-level or FLOPs-level accounting for the cost of different question-asking strategies, nor is there reporting of wall-clock latency for the belief parsing pipeline (which requires multiple sequential LLM calls for entity parsing, per-entity attribute parsing, and relation parsing). The "compute" cost of interaction is implicitly measured in user effort (number of questions the user must answer), not in model inference cost.
Cross-validation / statistical protocol. The paper does not employ cross-validation for strategy selection or hyperparameter tuning. The three agent variants are fixed prototypes with pre-defined configurations, not tuned per dataset. For human evaluation, approximately 550 image-dialog pairs per agent are rated by 3 human raters each, with generated images presented in random unlabeled order. The automatic evaluation runs on the full test sets (250 COCO images, 250 ImageInWords images, 30 DesignBench images) with a single run per configuration — no confidence intervals, error bars, or multiple random seeds are reported for the automatic metrics. The simulated user is deterministic in the sense that answers are grounded in the ground truth caption, though the LLM-based answer generation itself may have stochastic variation (default temperature).

Main Quantitative Results

Aggregate Agent Performance Across All Three Datasets (Table 1)

The headline quantitative result is that all three multi-turn agents substantially outperform the single-turn T2I baseline across all automatic metrics and all three datasets. The magnitude of improvement varies by dataset and metric, but the direction is uniform and large. On DesignBench — the dataset most representative of the paper's target use case for artists and designers — the best agent (Ag3) achieves:

T2T (EmbedSim): 0.943 vs. 0.874 for single-turn T2I (an improvement from already-high to near-perfect text similarity).
T2T (DSG): 0.949 vs. 0.607 — a dramatic 56% relative improvement, indicating that the agent's accumulated prompt captures the compositional scene graph structure far more faithfully than the initial underspecified prompt.
I2I (DINO): 0.692 vs. 0.544 — a 27% improvement in visual similarity to the ground truth image.
T2I (VQAScore): 0.955 vs. 0.353 — a 2.7× improvement, the largest relative gain across any metric-dataset pair. The single-turn T2I generates images that align with the ground truth caption only about one-third of the time according to VQAScore, while Ag3's generated images align nearly perfectly (0.955 on a 0–1 scale).
NLL: 257.4 vs. 320.9 — a substantial reduction in belief-state uncertainty.

The pattern holds on ImageInWords (Ag3 VQAScore: 0.662 vs. 0.371 for T2I baseline — 1.8× improvement) and COCO Captions (Ag3 VQAScore: 0.775 vs. 0.298 — 2.6× improvement). The agents' T2T scores in Table 1 (all above 0.92 for embedding similarity, all above 0.75 for DSG) "ablate the T2I model, showing that they have already reached high alignment on the caption level" (Section 8), meaning the remaining gap to perfect image alignment is attributable to the T2I model's prompt-following fidelity rather than to the agent's inability to specify what should be generated.

Table 1 (Human Eval columns) shows that human raters overwhelmingly prefer agent-generated images over single-turn T2I outputs. On DesignBench, Ag3's images are ranked as closest to the ground truth in terms of content in 63.5% of cases, compared to 3.2% for the single-turn baseline — a 20:1 preference ratio. For aesthetics/style, Ag3 is preferred in 36.5% of cases vs. 12.7% for the baseline. Even the weakest agent (Ag1) is preferred over single-turn T2I by large margins on content (17.5% vs. 3.2% on DesignBench).

Agent Ranking: Ag3 Consistently Outperforms Ag1 and Ag2 (Table 1, Figure 3)

Across all three datasets and all automatic metrics, the agent ranking is Ag3 > Ag2 ≥ Ag1. This ordering is consistent and typically monotonic. On DesignBench:

VQAScore: Ag3 (0.955) > Ag2 (0.855) > Ag1 (0.685) > T2I baseline (0.353). The gap between Ag3 and Ag2 is 0.100 — substantial on a 0–1 scale, suggesting that the principle-prompted strategy not only outperforms the belief-prompted strategy but does so by a meaningful margin.
DSG: Ag3 (0.949) > Ag2 (0.918) > Ag1 (0.829) > T2I baseline (0.607). The same monotonic ordering with Ag1 already providing large gains over the baseline, and Ag3 providing further gains.
NLL: Ag3 (257.4) < Ag2 (271.3) < Ag1 (295.2) < T2I baseline (320.9). Lower NLL is better, and Ag3 achieves the lowest (best) NLL, indicating its belief state most accurately captures the ground truth configuration.

On ImageInWords, the ordering is similar but the gaps are smaller: Ag3 VQAScore 0.662 vs. Ag2 0.570 vs. Ag1 0.506. On COCO Captions, Ag3's VQAScore advantage over Ag2 is roughly 0.112 (0.775 vs. 0.663), comparable to the DesignBench gap.

The human evaluation (Table 1, Human Eval columns) partially confirms but partially complicates this ranking. On DesignBench content ranking, Ag3 dominates (63.5% ranked closest), but on COCO Captions, Ag3's content advantage (51.1%) over Ag2 (25.3%) is more modest than the automatic metrics would suggest, and on ImageInWords, Ag3 (36.5%) and Ag2 (27.7%) are closer. For aesthetics/style, the human ranking is less consistent: Ag3 is preferred on DesignBench (36.5%) and ImageInWords (24.4%), but Ag2 is preferred on COCO Captions (31.6% vs. Ag3's 29.7%).

This discrepancy between automatic and human rankings on style metrics warrants attention. The automatic metrics (VQAScore, DSG, DINO) measure alignment to the ground truth image, which is a specific target. Human raters, when evaluating style/aesthetics, may have preferences that diverge from the ground truth — they may prefer a generated image that is aesthetically nicer but stylistically different from the target. The paper does not explore this distinction, but it suggests that while agents are better at hitting the target, they may not always produce images humans find more aesthetically pleasing in absolute terms.

Turn-by-Turn Dynamics: Rapid Initial Gains, Plateauing Text Metrics, Continuing Visual Improvement (Figure 3)

Figure 3 plots T2T (embedding similarity), I2I (DINO), T2I (VQAScore), and NLL as a function of turn number (0–15) for all agents on the ImageInWords dataset (similar patterns appear on other datasets per Figure 11 for DSG). The key observations:

Most of the gain occurs in the first 5–10 turns. For T2T embedding similarity (Figure 3a), all agents rise sharply from the baseline (~0.88) to above 0.94 by turn 3–5, with Ag3 reaching its plateau earliest (turn 5) and Ag1 taking slightly longer (~turn 8). For T2I VQAScore (Figure 3c), Ag3 rises from 0.37 to 0.64 by turn 6, and further gains to turn 15 are modest but positive. For I2I DINO (Figure 3b), the trajectory continues rising through turn 15 for all agents, with no clear plateau — suggesting that while text-level similarity saturates, visual similarity continues to benefit from additional detail.

The T2T and T2I metrics plateau or slightly decrease after approximately 10 turns (Figure 3a, 3c, explicitly noted in Section 7.2: "the T2T and the T2I VQA similarity scores seem to plateau or decrease after about 10 interactions, while the I2I scores continue to increase"). This divergence is important: after roughly 10 questions, the agent's accumulated prompt is textually very similar to the ground truth caption, so further questions add marginal text-level similarity. But the additional details — even if semantically small — continue to improve the visual fidelity of the generated image (I2I). The NLL metric (Figure 3d) drops most rapidly for Ag3 and continues decreasing through turn 15, confirming that the agent's internal uncertainty keeps being reduced even as downstream metrics saturate.

Ag3's NLL trajectory is notably steeper than Ag2's and Ag1's. At turn 0, all agents start with similar NLL (~450–500). By turn 5, Ag3's NLL has dropped to approximately 440, while Ag2 and Ag1 remain around 450–460. By turn 15, Ag3 reaches ~429, Ag2 ~444, and Ag1 ~450. This gap in uncertainty reduction efficiency — despite Ag3 having no explicit access to the belief graph — is the quantitative evidence for the paper's finding that the principle-prompted strategy reduces uncertainty more effectively than the belief-informed strategies.

The single-turn T2I baseline (horizontal blue line in all panels) is static — since it asks no questions, there is no turn-by-turn improvement. All multi-turn agents exceed this baseline by turn 2–3 across all metrics.

DSG Scene Graph Similarity Confirms Compositional Gains (Figure 11)

Figure 11 (Appendix) plots T2T DSG similarity per turn for all three datasets. The DSG metric captures whether the agent's prompt correctly specifies the atomic propositions (entity presence, attribute values, spatial relations) that make up the scene description. On DesignBench, all agents start at ~0.6 (the baseline DSG score of the initial underspecified prompt) and rise to: Ag3 ~0.95, Ag2 ~0.92, Ag1 ~0.83 by turn 15. The gap between Ag3 and Ag2 on DSG (~0.03 on DesignBench) is smaller than the gap on VQAScore (~0.10), suggesting that the text-level compositional quality of Ag2's prompts is closer to Ag3's than their image-level alignment would indicate — Ag3's advantage may partly stem from asking questions that elicit details which, while structurally similar in DSG terms, are more consequential for image generation fidelity.

On COCO Captions and ImageInWords, the DSG trajectories show similar patterns with Ag3 achieving the highest final scores, Ag2 second, and Ag1 third, though the absolute DSG values are lower on COCO (Ag3 reaching ~0.90) than on ImageInWords (~0.91) or DesignBench (~0.95), reflecting the relative complexity and descriptive adequacy of the ground truth captions in each dataset.

Human Evaluation of Question Quality: Ag1's Questions Are Simpler but Less Informative; Ag2 and Ag3 Ask Richer but Longer Questions (Figure 4)

Figure 4 (left) shows the percentage of questions marked by human raters as having specific issues. Approximately 8,000 questions per agent were rated. The key findings:

Ag1 has the highest rate of "no information gain" questions (~14%), consistent with the paper's observation (Section 7.2) that Ag1's templated question creation "sometimes leads to questions that gather minimal information in return." This includes asking about entity existence when it was already mentioned in the prompt (Figure 5 example: "Is the entity cake present in the image?") due to LLM parsing errors assigning cake less than 100% probability and high importance.
Ag2 and Ag3 have higher rates of "too long/complex" questions (~9% and ~13%, respectively), consistent with the paper's description (Section 6.3) that Ag3 "tends to ask more open-ended questions" and "more complex questions." Example from Figure 5: Ag3 asks "What types of berries are on the cake and how are they arranged?" — a multi-part question that elicits richer information but is more demanding to answer.
All agents have very low rates of "unclear and ambiguous," "irrelevant to prompt," and "unimportant attributes" questions (all below ~5%), indicating that all three question-asking strategies generally stay on-topic and produce comprehensible questions.

Figure 4 (right) shows human ratings of how well the final generated image corresponds to the user prompt and dialogue. More than 96% of all image-dialog pairs across all agents are rated as "Very Close" or "Fairly Close" with some differences. This high rating validates that the T2I model (Imagen 3) is faithfully rendering the accumulated prompts produced by the agents — the bottleneck is not the T2I model's prompt-following ability but rather the completeness of the specification that the agent elicits.

User Survey: High Reported Frustration with Current T2I, Strong Optimism About Proposed Features (Tables 2, 3, 4, 5)

The human survey of 143 regular T2I users (Section 7.4, Appendix G) provides the motivational context for the quantitative results:

83% experience occasional-to-very-frequent frustration from needing many prompt iterations, with 55% reporting frequent or very frequent frustration specifically due to the iteration frequency necessary (Table 4). This is the most prevalent frustration, ahead of misinterpretation (70%), inconsistent generations (71%), and incorrect assumptions (60%).
91% of users rated clarifications as "could help" / "likely to help" / "very likely to help" their workflow (Table 2). Entity graphs received 88% positive ratings and relationship graphs 86%. Clarifications were expected to deliver value "very soon / immediately" by 57.7% of users (Table 5) — the highest urgency rating of the three features.
Participant demographics confirm regular usage patterns (Table 3): 39.9% use T2I at least once a week, 30.8% many times a week, 9.1% many times a day — the respondents are genuinely experienced T2I users for whom prompt iteration is a real productivity concern, not hypothetical users speculating about a technology they rarely use.

These survey results are anecdotal evidence about user attitudes, not experimental evidence about system performance. The paper appropriately treats them as motivation and validation rather than as a controlled experiment — they establish that the problem the agents solve is a genuine pain point and that the proposed solution is perceived as valuable by its target audience.

Ablation Studies and Robustness Checks

The paper's ablation studies are design comparisons between agent variants rather than systematic component ablations. Each agent variant implicitly ablates a different combination of design choices. Here is the structured breakdown:

Ablation of explicit belief graph access for question generation (Ag3 vs. Ag2 vs. Ag1). Ag3 uses only conversation history and question-asking principles (no belief graph); Ag2 uses conversation history plus explicitly provided belief graph; Ag1 uses only the belief graph via a hardcoded heuristic scoring function (no conversation history access). The finding — that Ag3 substantially outperforms Ag2 across all datasets and metrics (Table 1, Figure 3) — is the paper's most significant ablation result. It suggests that, at least with Gemini 1.5 Pro, explicit access to the structured belief graph with importance scores and probability distributions constrains rather than enhances question quality, likely because the importance scores are miscalibrated (Section 7.2: "if the importance scores are not estimated properly, the quality of the questions decreases"). This is a negative result against the paper's own hypothesis that explicit uncertainty representation is the key to effective clarification questioning.

Ablation of conversation history access for question generation (Ag1 vs. Ag2/Ag3). Ag1 uses only the current belief graph for question selection via Equation (1), without access to the full conversation history. Its post-processing attempts to compensate (redundancy elimination by zeroing importance scores of addressed attributes), but human raters still rate ~14% of its questions as providing "no information gain" (Figure 4, left) and qualitative examples (Figure 5) show redundant questions about already-confirmed entities. This ablation demonstrates that conversation history is necessary for the Redundancy principle — hardcoded rules are insufficient to prevent the agent from asking about topics that have already been discussed, especially when LLM parsing errors introduce false uncertainty.

Ablation of multiple-choice vs. open-ended question format (Ag1 vs. Ag2/Ag3). Ag1 generates exclusively multiple-choice questions (via the verbalization prompt in Appendix D.12). Ag2 and Ag3 generate questions that can be open-ended (Ag2: "What color would you like the cake to be, and should it have any frosting?"; Ag3: "What types of berries are on the cake and how are they arranged?" from Figure 5). The human evaluation shows that Ag1 has fewer "too long/complex" questions (~4% vs. ~9% for Ag2 and ~13% for Ag3, Figure 4 left), but higher "no information gain" rates (~14% vs. ~4% for both Ag2 and Ag3). This suggests a trade-off: multiple-choice questions are easier to answer (satisfying Easy-to-Answer) but gather less information per question; open-ended questions gather richer information but at higher cognitive cost. The paper does not systematically vary this parameter within a single agent design, so the causal effect of question format cannot be isolated from the concurrent differences in question selection strategy.

Ablation of T2I model errors through seed-based filtering (Appendix E.2). The paper demonstrates that agent-user QA pairs can be used to improve T2I fidelity by filtering generated images across multiple random seeds. On DesignBench: (1) generate 10 images per ground truth prompt with Imagen 3 at different seeds; (2) average DINO (I2I) against ground truth = 0.7637; (3) convert the first 5 Q&A pairs from Ag2 into binary VQA questions; (4) score all 10 images with VQA and select the best; (5) new average DINO = 0.7838. The improvement is Δ = +0.02. This is a small (+2.6% relative) but positive effect, demonstrating that agent dialog can serve as an auxiliary quality filter. The ablation confirms that T2I model stochasticity is a real bottleneck (as the paper acknowledges in Section 8) and that the agent's information-gathering produces artifacts (Q&A pairs) that can partially mitigate this bottleneck, though the benefit is modest.

Dataset-level ablation (DesignBench vs. COCO vs. ImageInWords). While not presented as a formal ablation, the consistent agent ranking (Ag3 > Ag2 > Ag1) across three datasets with different characteristics — COCO has simple photographic scenes with short human captions, ImageInWords has diverse realistic and cartoon scenes with dense human captions, DesignBench has artistic and improbable scenes with LLM-generated captions — serves as a robustness check on the qualitative findings. The agent design is not overfit to a particular type of imagery or captioning style. However, the absolute metrics differ across datasets: agents perform best on DesignBench (Ag3 VQAScore 0.955), then COCO (0.775), then ImageInWords (0.662). This ordering reflects differences in ground truth caption quality and specificity (DesignBench captions are LLM-generated to be near-perfect descriptions, ImageInWords captions are human-written and may have stylistic variation, COCO ground truth captions are LLM-expanded from short human captions and may be less precise) rather than differences in agent effectiveness.

The paper does not perform several ablations that would clarify causal mechanisms. There is no ablation of the importance scores (e.g., setting all importance scores to 1.0 to test whether entropy alone is sufficient for Ag1), no ablation of the three-entity-type classification (explicit / implicit / background), no ablation comparing the sequential parser pipeline to an end-to-end single-LLM belief graph generator, no ablation of the number of few-shot examples in parser prompts, and no systematic ablation of the question-asking principles individually (e.g., removing "Easy-to-Answer" from Ag3's prompt to see whether question complexity increases and answer quality changes). These are all reasonable future experiments but are outside the scope of a prototype evaluation paper.

Critical Assessment

Claim 1 (from the paper's stated contributions): "The proposed T2I agents actively ask clarification questions when uncertain, and present their uncertainty about user intent as an understandable and editable belief graph."

This claim is borne out by the system design (Sections 4–6, Appendices D.5–D.12) and the qualitative examples (Figures 1, 5), but limited evidence is provided about the editability and user-facing utility of the belief graph. The automatic evaluation never tests the graph editing functionality — the simulated user does not edit the graph. The human study on the interface (Section 7.4, Appendix G) showed users static mockups and asked about expected usefulness, not about actual usage. The paper does not report a controlled experiment where users generate images with and without access to the editable belief graph and their outcomes are compared. The claim that the belief graph is "understandable and editable" is thus supported by qualitative design description and user attitude surveys, but not by experimental evidence of its causal impact on task performance.

Claim 2: "Multi-turn agents achieve at least 2 times higher VQAScore than traditional single-turn T2I generation within just 5 turns of interaction."

This claim is supported. On DesignBench, Ag3 achieves 0.955 VQAScore at turn 15 vs. 0.353 for the single-turn baseline — a 2.7× improvement. Even at turn 5, Figure 3c shows VQAScore for Ag3 on ImageInWords has risen to approximately 0.55–0.60, compared to the baseline of 0.371 — roughly 1.5–1.6×, approaching the 2× claim. On COCO (not plotted turn-by-turn but available at turn 15 in Table 1), the improvement is 2.6× (0.775 vs. 0.298). The "within 5 turns" qualification is more assertive than what the printed plots support for some datasets — the 2× threshold is likely crossed between turns 5 and 10 on ImageInWords, though the VQAScore curve is steepest in turns 0–5.

An important nuance: the single-turn baseline uses the initial underspecified prompt p0, not the ground truth caption c. So the 2.7× improvement compares "agent after 15 questions" against "no questions at all." What is not measured is "user manually iterates their prompt 15 times without agent guidance" — a more realistic baseline for experienced users. A user who knows what details to add could, in principle, achieve similar alignment through manual prompt engineering. The 2×+ improvement is thus an upper bound on the agent's benefit relative to a completely passive system, not relative to a skilled human prompter.

Claim 3: "At least 90% of human subjects found these agents and their belief graphs helpful for their T2I workflow."

Supported, with a narrower interpretation needed. The 90%+ figure comes from Table 2: 91.0% of subjects rated clarifications as "could help" / "likely" / "very likely" to help; 88.2% for entity graphs; 86.1% for relationship graphs. These are attitude measures from a survey showing static mockups — they measure expected helpfulness, not experienced helpfulness after using the system. The human evaluation of actual generated images (Table 1, Human Eval columns) provides behavioral evidence: agent-generated images are preferred over single-turn T2I outputs in 80%+ of cases for content on DesignBench (63.5% for Ag3 specifically) and in over 80% of cases in aggregate across datasets. But the preference rates for individual agents vary — Ag3 is preferred in 63.5% of DesignBench content rankings, Ag2 in 15.9%, Ag1 in 17.5%. The 90% survey enthusiasm should be understood as user appetite for the concept, while the 60–80% image preference rates represent actual performance advantage over the single-turn alternative.

Claim 4: "The belief graph enables the agent to express uncertainty about entities, attributes, and relations."

Supported by design. The belief parsing pipeline demonstrably produces structured representations with probability distributions (Figures 1, 10, Appendix D). The NLL metric in Table 1 quantifies uncertainty reduction over turns. However, the causal role of the belief graph in the most effective agent (Ag3) is zero — Ag3 does not use the belief graph for question generation. This means the belief graph, while conceptually central to the paper's framework, is not the mechanism that produces the best results. The paper does not resolve this tension: if the best agent ignores the belief graph, what is the value proposition of the belief graph beyond interpretability (which itself was tested only via survey attitudes, not usage)?

Methodological weaknesses:

Single LLM family, single T2I model. All experiments use Gemini 1.5 Pro and Imagen 3. The paper argues the agent design is modular and model-agnostic (Section 8: "seamless integration of improved off-the-shelf T2I models") but provides no evidence that the findings generalize to other LLMs (GPT-4, Claude) or T2I models (DALL-E, Stable Diffusion, Midjourney). The finding that Ag3 outperforms Ag2 may be specific to Gemini 1.5 Pro's particular strengths at open-ended reasoning versus structured data interpretation — a different LLM might show the opposite pattern.
Small DesignBench test set (30 scenes). The paper's flagship benchmark for its target use case (artists and designers) has only 30 examples. Human evaluation on DesignBench used approximately 21 image-dialog pairs (footnote to Figure 21). This limits the statistical reliability of the DesignBench-specific claims, particularly the very high VQAScore of 0.955 — on a sample of 30, a few lucky generations could substantially inflate the average.
No error bars or confidence intervals. All automatic metrics in Table 1 report means and standard deviations, but these are standard deviations across images, not estimates of experimental variability from multiple runs. The stochasticity of LLM parsing and question generation (default temperature) means that re-running the same agent on the same prompt could produce different questions and different metric trajectories. Without multiple-run statistics, the reported differences between agents — particularly the Ag2 vs. Ag3 gap — cannot be assessed for statistical significance.
No comparison to human-in-the-loop prompt engineering. A skilled human who iteratively refines their prompt by inspecting generated images and adding details could potentially achieve the same alignment as the agent with a comparable number of turns. The paper does not include a "human prompt engineer" baseline, making it impossible to separate the value of multi-turn interaction from the value of agent-guided multi-turn interaction.
The simulated user may be unrealistically cooperative. The simulated user (Appendix D.3) always provides answers that are "concise" and "faithful to the ground truth." Real users might provide vague answers, misinterpret questions, change their mind, refuse to answer, or give contradictory information across turns. The automatic evaluation results should be interpreted as measuring agent performance under ideal user cooperation, which is a necessary first evaluation but not a demonstration of real-world usability.
Latency and cost are not reported. The belief parsing pipeline requires at minimum 1 entity parser call, plus 1 attribute parser call per entity, plus 1 relation parser call, plus 1 question generation call, plus 2 calls for belief transition (verbalization and prompt merging). For a scene with 10 entities, that's roughly 15 LLM API calls per turn, each potentially involving long context windows and chain-of-thought reasoning. The paper does not report the wall-clock time per turn or the API cost per conversation, which are critical for assessing practical deployability.

Missing experiments that would strengthen the paper:

A within-agent ablation of the belief graph's causal role. Keep the agent design fixed (e.g., Ag2) and systematically degrade or remove the belief graph information: (a) full belief graph with probabilities and importance scores, (b) belief graph with only entity/attribute/relation names and no probabilities, (c) no belief graph. This would isolate whether it's the structure or the scores that matter for question quality.
A head-to-head comparison with prompt expansion (Datta et al., 2024) or ImageInWords-style detailed captioning. How much of the agent's gain comes from simply having more text in the prompt versus having the right text informed by user feedback? An ablation where the agent expands the prompt using an LLM without asking questions (automatic expansion) would separate the value of interactivity from the value of detail.
Multiple LLM backends. Replicating the Ag2 vs. Ag3 comparison with at least one other LLM family (e.g., GPT-4o) to test the robustness of the finding that explicit belief access is outperformed by principle-based prompting.
Evaluation of belief graph editing. A user study where participants actually use the editable belief graph interface to refine images, with measured outcomes (time to satisfactory image, number of interactions, satisfaction scores), would provide the missing evidence for the claim that belief graphs are "understandable and editable" in practice.

Despite these limitations, the paper's core empirical contribution is solid and well-triangulated: three different agent designs, three different datasets (with different caption styles and image types), five complementary automatic metrics, and three human evaluation tasks all point in the same direction — multi-turn clarification substantially improves image-prompt alignment, and the gains are large enough (2×+ on VQAScore, 80%+ human preference) to be practically meaningful even accounting for methodological imprecision. The finding that Ag3, which abandons the belief graph for question generation, performs best is genuinely surprising and informative, even if its generalizability across LLMs remains untested.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Unaccounted For, Making the Headline Efficiency Gains an Upper Bound

The assumption or constraint. The agent must construct a belief graph before it can ask any clarification questions. This requires the full parsing pipeline: entity parser (1 LLM call), attribute parser (1 call per entity), and relation parser (1 call per entity pair). For a scene with 10 entities, this is roughly 12 sequential LLM API calls using Gemini 1.5 Pro with long-context prompts and chain-of-thought reasoning. The paper never reports the wall-clock latency, token cost, or FLOPs-equivalent of this pipeline. Section 5's formal framework defines the agent's objective as maximizing alignment "with as few turns of interaction with the user as possible" (Appendix B), but turns are measured only as user-facing questions, not as total system cost. The difficulty estimation — parsing the belief graph to determine what the agent is uncertain about — is completely excluded from the cost accounting.

The consequence. In a real deployment, the agent's first response to a user prompt would not be a question but a multi-second (or longer) delay while the belief parsing pipeline executes. For a complex scene with many entities, the parsing cost could exceed the cost of generating the image itself — and it is incurred before the user sees any value. The reported efficiency gains (2× VQAScore improvement within 5 turns) are computed without amortizing the upfront parsing cost. If belief parsing takes the equivalent of 2–3 turns of user waiting time, the effective interaction budget is 7–8 turns of total latency for 5 visible questions. This matters especially for the paper's target use case of rapid creative iteration (artists and designers), where responsiveness is critical to maintaining creative flow.

What evidence exists in the paper. The paper provides no measurement of belief parsing latency, no token counts for parser prompts, and no cost accounting for the LLM API calls. The parser prompts themselves are reproduced in full in Appendices D.5–D.7 and are substantial — the entity parser prompt alone contains multiple few-shot examples and covers approximately 100 lines of instruction text. The paper's modular architecture description (Section 8) frames the ability to "seamlessly integrate improved off-the-shelf T2I models" as a benefit, but it does not discuss whether improved LLMs would reduce parsing latency or simply produce better belief graphs. The simulated user evaluation (Algorithm 2) elides parsing time entirely: each turn is modeled as "select action → verbalize → answer → transition," with no accounting for the computational cost of the transition step.

Mitigation status. The paper does not address this limitation, propose mitigation strategies, or acknowledge it as a constraint. Future work on "fine-tuning the model on multi-turn trajectories" (Section 8) could potentially reduce parsing overhead by replacing the multi-step ICL parsing pipeline with a single end-to-end fine-tuned model, but this is speculative and not evaluated.

The Best-Performing Agent Ignores the Belief Graph, Undermining Its Claimed Centrality

The assumption or constraint. The paper's core conceptual contribution is the belief graph — a structured representation of the agent's uncertainty about user intent, with explicit entities, attributes, relations, probabilities, and importance scores. Sections 1, 4, and 5 motivate the belief graph as the mechanism that "allows the agent to express uncertainty," "enables proactive questioning," and "provides an understandable and editable interface for users." Three agent variants are designed around progressively less reliance on the belief graph: Ag1 computes explicit heuristic scores over it, Ag2 prompts an LLM with it, and Ag3 ignores it entirely.

The consequence. The experimental results show that Ag3, which has zero access to the belief graph for question generation, substantially and consistently outperforms both belief-graph-informed agents across all datasets and nearly all metrics (Table 1, Figure 3). On DesignBench, Ag3 achieves VQAScore 0.955 vs. Ag2's 0.855 and Ag1's 0.685. On ImageInWords, Ag3 reaches 0.662 vs. Ag2's 0.570. On DSG text similarity (DesignBench), Ag3 scores 0.949 vs. Ag2's 0.918. The belief graph is not merely unnecessary for the most effective questioning strategy — the paper's own evidence suggests it may be actively harmful when used to guide an LLM's question generation. The authors hypothesize (Section 7.2) that "Ag2 excessively relies on importance scores in beliefs to ask questions, and if the importance scores are not estimated properly, the quality of the questions decreases," meaning the belief graph encodes miscalibrated importance estimates that constrain the LLM toward suboptimal questions.

This creates an unresolved tension in the paper's narrative. The belief graph is presented as the key innovation (it appears in the paper's title via "Under Uncertainty," in Figure 1 as the central interface element, and in Section 4 as the primary design contribution). Yet the prototype that achieves the best results — and which human raters most prefer (63.5% content ranking on DesignBench vs. 15.9% for Ag2; Table 1) — operates as if the belief graph does not exist. The paper never reconciles this: if the belief graph is not causally responsible for the performance gains, what is its role? The human survey (Section 7.4) shows users express positive attitudes toward belief graph features (88% find entity graphs helpful), but this measures anticipated helpfulness from static mockups — not actual usage value in a functioning system.

What evidence exists in the paper. The finding is robust across all three datasets in Table 1, across all five automatic metrics, and across the turn-by-turn trajectories in Figure 3. The paper acknowledges the finding (Section 7.2: "Question prompts with question-asking principles show advantage over those with beliefs") but treats it as an interesting empirical observation rather than as a challenge to the paper's architectural premise. The qualitative dialog examples in Figure 5 illustrate the difference concretely: Ag2 asks narrow, belief-graph-guided questions ("What type of berries would you like on the cake?"), while Ag3 asks broader questions that elicit more information per turn ("What types of berries are on the cake and how are they arranged?").

Mitigation status. The paper does not attempt to fix the belief graph's underperformance — for instance, by improving importance score calibration, by combining Ag2's belief access with Ag3's principle prompting, or by ablating whether the issue is specifically the importance scores (vs. the probability estimates or the structural representation itself). The suggested future work on "fine-tuning models on multi-turn trajectories" (Section 8) would operate on raw dialog, not on belief graphs, implying the belief graph's role may diminish further in future iterations.

Generalization Is Limited to a Single LLM Family, Single T2I Model, and a Specific Prompt-Underspecification Regime

The assumption or constraint. All experiments use Gemini 1.5 Pro for belief parsing, question generation, answer simulation, and metric computation, and Imagen 3 for image generation (Section 7.1.1). The paper positions its agents as modular and model-agnostic (Section 8: "This allows for seamless integration of improved off-the-shelf T2I models as they become available, boosting system performance without requiring further adaptation"), but provides no evidence that the findings generalize. The three datasets — COCO Captions, ImageInWords, and DesignBench — all evaluate the same type of task: starting from an intentionally underspecified short prompt and recovering a specific, detailed ground truth image. There is no evaluation of scenarios where the user genuinely does not know what they want and is exploring possibilities, where the ground truth image does not exist (creative ideation), or where user preferences change mid-conversation.

The consequence. Several findings may be model-specific in ways that practitioners cannot assess from the evidence provided:

The Ag3 > Ag2 > Ag1 ranking depends on Gemini 1.5 Pro's relative strength at open-ended conversational reasoning vs. structured data interpretation. A different LLM family (GPT-4o, Claude) with different inductive biases might show the opposite ranking — where explicit belief graph access helps more than it constrains. The paper's hypothesis that Ag2 over-relies on miscalibrated importance scores (Section 7.2) is a claim about Gemini 1.5 Pro's parsing behavior, not a universal property of LLM-belief graph interaction.
The absolute performance numbers (VQAScore 0.955 on DesignBench, 0.775 on COCO) are specific to Imagen 3's prompt-following fidelity. A weaker T2I model would show lower absolute scores (since the agent's prompts would not be faithfully rendered), while a stronger model could potentially close the gap between agents (if all prompts produce near-perfect images, the agent's questioning quality matters less).
The 2×+ VQAScore improvement is measured against a single-turn baseline that uses the initial underspecified prompt p0. This p0 is deliberately constructed to be information-poor — the shortest COCO caption, an auto-summarized ImageInWords description. In real usage, many users provide moderately detailed prompts that are not as severely underspecified as p0. The improvement over a "typical user prompt" baseline is unknown and smaller than the improvement over p0.

What evidence exists in the paper. The paper provides no cross-model experiments — no ablation using a different LLM backend for belief parsing or question generation, no comparison using a different T2I model for final image generation. The three datasets provide some diversity in image type (photographs, cartoons, artistic compositions) and caption style (human-written, LLM-generated), but they all share the same evaluation protocol: start from underspecified p0, target specific ground truth I. The paper does note (Section 7.1.2) that existing image-caption datasets "primarily focus on simple scenes or focus on very specific categories," and creates DesignBench to partially address this, but DesignBench adds only 30 scenes and maintains the same task structure.

Mitigation status. The paper acknowledges the single-model limitation implicitly through its modular architecture claims (Section 8), but does not present cross-model validation, ablation across LLMs, or experiments with different T2I backends. The release of DesignBench and the agent code is a step toward enabling others to perform these validations, but the paper itself provides no generalizability evidence beyond its specific model stack.

The Simulated User Evaluation Overestimates Real-World Effectiveness by Assuming Ideal User Cooperation

The assumption or constraint. The automatic evaluation framework (Section 7.1, Algorithm 2, Appendix D.3) uses an LLM-based simulated user that operates with access to the ground truth caption c and ground truth belief graph b_gt. The simulated user is instructed to provide "concise answers to agent questions" — it always answers the specific question asked, it never volunteers additional unsolicited information, it never misinterprets a question, it never changes its mind about what it wants, it never declines to answer, and it never provides vague or contradictory responses. This is an ideal-user assumption: the simulated user is perfectly cooperative, has perfect knowledge of its own intent, and communicates that knowledge without noise or ambiguity.

The consequence. The automatic evaluation results measure agent performance under best-case user behavior — what the agent can achieve when the user knows exactly what they want and answers every question accurately and concisely. Real users exhibit several behaviors that would degrade performance:

Vague answers: "What color is the cat?" → "I don't know, something dark" — which the belief transition function (merging Q&A pairs into the accumulated prompt) would embed as a vague specification, producing an image that may still not match the user's actual (uncommunicated) preference.
Question misinterpretation: users may misunderstand what the agent is asking about, providing answers that resolve a different uncertainty than the one the agent intended to target.
Evolving preferences: users may discover what they want through the interaction process — seeing a generated image sparks a new idea — which the ground-truth-anchored simulated user cannot model.
Answer fatigue: the human survey (Section 7.4) identifies that 55% of users already experience "frequent or very frequent frustration" from prompt iterations. Adding agent-initiated questions may increase rather than decrease frustration if users perceive the questions as tedious or obvious. The paper does not measure how many questions a real user would tolerate before abandoning the interaction.

What evidence exists in the paper. The paper acknowledges this limitation only indirectly. Section 2 distinguishes the simulated user as a tool for scalable evaluation, and the human studies (Section 7.3) are presented as complementary validation rather than as a replacement for the automatic evaluation. The human raters' evaluation of question quality (Figure 4, left) provides partial evidence about real-user experience: 13–14% of Ag1's questions are rated as providing "no information gain," and 9–13% of Ag2/Ag3 questions are rated as "too long/complex." These are signs that even the best agent asks some questions that real users would find unhelpful or burdensome, but the simulated user dutifully answers them anyway. The gap between simulated-user performance and real-user performance is not quantified or estimated.

Mitigation status. The paper uses human studies (Section 7.3, 7.4) to partially validate the automatic results, but these studies are observational (raters evaluating pre-generated dialogs and images) rather than interventional (users actively conversing with agents and generating images). No experiment measures how real users actually interact with the agent in an unconstrained setting — how many turns they engage for, whether they find the questions helpful or annoying, whether the generated images satisfy them. The paper's future work section (Section 8) calls for "collecting data such as gold-standard trajectories or annotations on the quality of trajectories of human-agent conversations," acknowledging this gap, but the current evaluation provides no lower-bound estimate on performance degradation under realistic user behavior.

The Approach Cannot Help When the Base T2I Model Fundamentally Lacks the Capability to Render the User's Intent

The assumption or constraint. The agent architecture treats the T2I model (Imagen 3) as a frozen, black-box tool — the agent's sole mechanism for improving output quality is to produce better text prompts for this model (Section 5, Section 8). The agent never fine-tunes the T2I model, never intervenes in its generation process, and never provides image-level feedback (e.g., "make the rabbit larger," "move the cat to the left"). The agent operates entirely in the space of text prompt specification. If the user's intended image contains elements that Imagen 3 cannot faithfully render from text descriptions — due to limitations in the model's training data, architecture, or prompt-following capabilities — no amount of clarification questioning will bridge that gap.

The consequence. The paper's own analysis (Section 8) identifies this bottleneck: "T2I prompt-image alignment errors can limit the effectiveness of our proposed agents." The experiment in Appendix E.2 quantifies one aspect of this: even with perfect ground truth captions, Imagen 3 generates images with an average DINO (I2I) similarity of only 0.7637 across 10 seeds, and filtering using agent QA pairs improves this to only 0.7838 — a +0.02 gain. This means roughly 22% of the visual similarity gap is attributable to the T2I model failing to render what the prompt specifies, a ceiling that the agent's questioning cannot break through.

More importantly, there is an entire class of user intents that are intrinsically difficult to specify textually regardless of how much clarification occurs. Fine-grained spatial composition ("the rabbit's left ear should overlap the cat's tail by exactly 15%"), precise color gradients, complex lighting interactions, and artistic effects that lack standard linguistic descriptions all fall into this category. A user who wants these effects may answer every clarification question perfectly and still receive an image that misses their intent — not because the agent failed to elicit their specification, but because the specification itself is inexpressible in the text modality that the agent uses to communicate with the T2I model.

What evidence exists in the paper. Appendix E.2 provides the key quantitative evidence: Imagen 3's prompt-following fidelity imposes a measurable ceiling on image quality even with ideal prompts. The T2T metrics in Table 1 provide indirect corroboration: the agents achieve embedding similarity of 0.94+ and DSG similarity of 0.90+ (Ag3 on COCO) at the text level, yet the image-level metrics (I2I DINO, T2I VQAScore) are substantially lower (0.658 I2I, 0.775 VQAScore for Ag3 on COCO), indicating a gap between what the prompt specifies and what the image renders.

Mitigation status. The paper acknowledges the limitation (Section 8: "a limitation of the proposed pipeline is that its overall capability is constrained by the prompt-following abilities of the employed text-to-image model") and proposes two mitigation strategies: (1) using agent QA pairs to filter across multiple generation seeds (Appendix E.2, yielding a small +0.02 improvement), and (2) the modular architecture allowing future T2I model upgrades to automatically improve performance. The paper also suggests (Section 8, Future Work) "generating images directly from belief graphs and fine-tuning VLMs on image-text interleaved multi-turn trajectory data" as a path toward bypassing the text bottleneck entirely. However, the current system offers no mechanism for users to provide image-level feedback (e.g., selecting regions to refine, providing reference images, or using visual widgets to adjust composition), which means that once the text specification ceiling is hit, the interaction cannot progress further regardless of how many additional turns the user invests.

Human Evaluation Scope Does Not Validate Real-Time Interactive Usage

The assumption or constraint. All human evaluations in the paper are retrospective judgments of pre-generated outputs, not measurements of live user-agent interaction. Human raters in Task 1 (Figure 19) evaluate individual questions from pre-recorded dialogs. In Task 2 (Figure 20), they rate how well a final generated image corresponds to a pre-existing dialog and prompt. In Task 3 (Figure 21), they rank pre-generated images from different agents against a ground truth image. The user survey (Section 7.4) presents static interface mockups and asks about expected helpfulness. No experiment places a human user in front of a functioning agent, asks them to generate an image they genuinely want, and measures outcomes.

The consequence. Several critical real-world variables are unmeasured:

Engagement and abandonment rates. How many turns do users actually engage for before either being satisfied or giving up? The 15-turn maximum in automatic evaluation may dramatically overestimate real user patience — the survey finding that 55% of users already experience "frequent or very frequent" frustration with manual prompt iteration (Table 4) suggests low tolerance for extended back-and-forth, even if the agent (rather than the user) is driving it.
Question utility in context. Human raters evaluating isolated questions (Figure 4, left) can judge whether a question is clear and relevant, but cannot assess whether asking that question at that specific point in the conversation was productive for the user's goal. A question that is clear, relevant, and informative in isolation might still be annoying if the user feels it's obvious, repetitive, or unnecessary given what they've already communicated.
The value of the belief graph as an interface. The survey reports that 88% of users think entity graphs would be helpful (Table 2), but this measures anticipated value of a static mockup. Whether users would actually interact with an editable belief graph during a creative session — and whether that interaction would improve their outcomes or their satisfaction — is unknown.
Learning effects and user adaptation. In real extended usage, users might learn to write more detailed initial prompts (knowing the agent will ask follow-ups), or learn to anticipate the agent's question patterns and provide preemptive details in their answers. Conversely, users might become frustrated if the agent repeatedly asks questions they consider obvious, reducing trust in the system. None of these dynamics can be captured by static evaluation.

What evidence exists in the paper. Section 7.3 describes the human evaluation tasks and sample sizes (~550 image-dialog pairs per agent, 3 raters each; ~8,000 questions per agent). Section 7.4 describes the survey of 143 T2I users. All of these provide useful data about user attitudes and about the relative quality of agent outputs, but they do not constitute a user study of interactive system usage. The paper does not claim otherwise — it describes the human studies as providing "more insights that complement the quantitative results from our automatic evaluations" (Section 7.3) — but the distinction matters because the paper's ultimate claim is about improving the interactive experience of T2I generation, and the evaluation does not directly measure that experience.

Mitigation status. The paper acknowledges the need for richer interaction data in its Future Work section: "collecting data such as gold-standard trajectories or annotations on the quality of trajectories of human-agent conversations" (Section 8). This is an explicit recognition that the current evaluation is incomplete. However, the paper's strong claims about user benefit — "over 90% of human subjects found these agents and their belief graphs helpful for their T2I workflow" — are based on survey attitudes, not on demonstrated improvement in actual T2I workflows. The distinction between "users expect this would help" and "users who tried it were more successful" is significant for a practitioner deciding whether to invest in building such a system.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the conversation around human-AI interaction for generative models from passive prompt acceptance to proactive uncertainty resolution. Before this work, the dominant paradigm for T2I interaction was: user writes prompt, model generates image, user iterates. The entire burden of disambiguation fell on the user, who had to guess which missing details were responsible for unsatisfactory outputs. The paper establishes a competing paradigm: the model should tell the user what it doesn't know and ask about it before generating. This is not an incremental improvement to prompt engineering — it's a reframing of who bears the cognitive cost of underspecification.

The magnitude of this shift is best understood as a new interaction primitive rather than a new model architecture. The paper does not improve the T2I model itself (Imagen 3 remains frozen), nor does it fine-tune the LLM that drives questioning. It demonstrates that substantial gains — 2×+ VQAScore improvement, 80%+ human preference over single-turn generation — can be achieved purely by inserting a structured information-gathering loop between the user and an off-the-shelf generator. This has a disruptive implication: it suggests that interaction protocol design may be a higher-leverage research investment than incremental model scaling for improving user-perceived output quality in underspecified generation tasks. The paper's human survey (Section 7.4) corroborates this: 91% of regular T2I users believe clarification features would help their workflow, and 55% report frequent or very frequent frustration from prompt iteration — a pain point that no amount of model size increase addresses, since larger models still make implicit assumptions when prompts are underspecified.

The paper also provides a reconciliation framework for a fragmented research landscape. Prior work on multi-turn T2I (DialogGen, DSG-GAN), prompt expansion (ImageInWords, Datta et al.), ambiguity resolution (Mehrabi et al.), and clarification dialog in NLP had each addressed pieces of the underspecification problem, but without a unifying representation of what information the model lacks. The belief graph — whether or not it proves to be the optimal implementation — establishes a common vocabulary for talking about underspecification: entities (explicit, implicit, background), attributes with probability distributions and importance scores, and relations with spatial types. This decomposition makes it possible to compare different question-asking strategies on equal footing (as the paper does with Ag1/Ag2/Ag3), to measure uncertainty reduction quantitatively (via NLL), and to design interfaces that make model uncertainty inspectable and editable. Even if future systems abandon the specific belief graph representation, they will likely retain the conceptual decomposition into what might appear (entities), what properties it might have (attributes), and how elements relate (relations) — the three axes along which user intent can be underspecified.

Perhaps the most significant and surprising implication is the limited causal role of explicit uncertainty representations when the question-generator is a sufficiently capable LLM. The paper's headline agent ranking — Ag3 (no belief graph) > Ag2 (belief-prompted) > Ag1 (heuristic over belief graph) — flips the natural assumption that more structured uncertainty information produces better questions. The authors' diagnosis — that miscalibrated importance scores in the belief graph constrain the LLM toward suboptimal questions (Section 7.2) — is a specific instance of a deeper phenomenon: explicit symbolic representations can become straitjackets when the underlying LLM has richer implicit knowledge than the representation captures. The belief graph assigns importance scores based on the entity/attribute parser's static analysis of the prompt; the LLM in Ag3 infers importance dynamically from the full conversation trajectory and its world knowledge about what details matter in images. The finding that dynamic inference outperforms static scoring has implications beyond T2I — it suggests that for LLM-powered agents, the value of explicit uncertainty tracking may depend on whether the LLM's implicit uncertainty assessment (via prompting) is better or worse calibrated than the explicit representation, a threshold that likely varies by LLM capability, domain, and representation granularity.

This finding redirects research attention away from better uncertainty representations and toward better conversational reasoning. Before this paper, a natural research program would have been: improve the belief graph (better parsers, better importance score calibration, richer relation types, learned priors over entity co-occurrence). After this paper, the more promising direction appears to be: improve the LLM's ability to track what information has been exchanged, identify gaps in the specification, and formulate questions that efficiently fill those gaps — potentially by fine-tuning on multi-turn image-text dialog trajectories rather than by engineering better symbolic structures. The paper's Future Work section (Section 8) explicitly calls for this: "approaches to fine-tune the model on multi-turn trajectories of images and text, which can potentially improve the performance of the agent." This is a concrete redirection of effort — from representation engineering to dialog data collection and fine-tuning.

The paper also establishes that multi-turn clarification is not just theoretically appealing but quantitatively impactful within viable interaction budgets. The 2×+ VQAScore improvement within 5 turns and the turn-by-turn trajectories showing most gains accruing in the first 5–10 turns (Figure 3) provide an empirical anchor that other researchers can cite when justifying work on interactive clarification. Before this paper, the field had intuitions that asking questions should help; after it, there is a specific, replicable magnitude of benefit that can be used to evaluate whether new approaches represent meaningful improvements. The 80%+ human preference rate for agent-generated images over single-turn outputs (Table 1) further anchors the practical significance — this is not a marginal statistical improvement but a qualitative difference in output quality that users reliably notice and prefer.

Finally, the paper's modular architecture — frozen T2I model, frozen LLM for dialog, ICL-based belief parsing, pluggable question selection strategies — establishes a baseline approachability for the problem. A research group without the resources to train large generative models can still contribute to this area by proposing new question-asking strategies, new belief representations, or new evaluation metrics, and comparing against the paper's released code and DesignBench benchmark. This lowers the barrier to entry and should accelerate progress compared to a field where only organizations with access to T2I model training pipelines can participate.

Follow-Up Research This Work Enables

Fine-tuning VLMs on multi-turn image-text clarification trajectories. The paper's most direct call for future work (Section 8) is to collect "gold-standard trajectories or annotations on the quality of trajectories of human-agent conversations" and fine-tune models on these trajectories. The specific gap this addresses: the current system uses ICL prompting with frozen Gemini 1.5 Pro for every component (belief parsing, question generation, answer simulation, prompt merging), which is computationally expensive (12+ LLM calls per turn), slow (latency unmeasured but likely multiple seconds), and brittle (stochastic parsing causes inconsistencies that Ag1's post-processing attempts to patch). A fine-tuned model could potentially collapse the multi-step parsing pipeline into a single forward pass, dramatically reducing latency and cost. A strong follow-up would: (1) collect human-agent dialog trajectories where users genuinely try to generate specific images, recording all questions, answers, intermediate belief states, intermediate generated images, and final satisfaction ratings; (2) fine-tune a VLM (e.g., a Gemini variant or open-weight alternative) on these trajectories with a loss that encourages both accurate belief state tracking (predicting the ground truth entities/attributes/relations from dialog history) and effective question generation (predicting which question a human expert would ask next); (3) compare the fine-tuned model against the ICL-prompted Ag3 baseline on the same automatic evaluation protocol used in this paper, measuring both quality (VQAScore, DSG, DINO) and efficiency (wall-clock time, API cost). The key measurement is whether fine-tuning can close the remaining gap between Ag3's performance and the ground truth ceiling (the T2T metrics in Table 1 show agents have near-perfect text alignment, but I2I and VQAScore remain below 1.0), or whether the ceiling is dominated by the frozen T2I model's prompt-following limitations (Appendix E.2 suggests ~22% of the I2I gap is T2I-limited).

Cross-LLM and cross-T2I-model replication of the Ag3 > Ag2 finding. The paper's most surprising result — that the principle-prompted agent without belief graph access outperforms the belief-prompted agent — is established only for Gemini 1.5 Pro + Imagen 3. The specific question this raises: is this result a general property of capable LLMs (suggesting explicit uncertainty representations are net-negative for LLM-driven question generation), or is it specific to Gemini 1.5 Pro's relative strengths at open-ended reasoning vs. structured data interpretation? A strong follow-up would replicate the Ag2 vs. Ag3 comparison using at least two additional LLM families (GPT-4o, Claude 3.5 Sonnet) and at least one additional T2I model (DALL-E 3, Stable Diffusion 3), keeping all other components identical (same belief parsing pipeline, same datasets). The critical measurement is whether the Ag3 > Ag2 ranking holds across LLM families and whether the magnitude of the gap correlates with any measurable LLM property (e.g., performance on structured data extraction benchmarks). A reversal — where Ag2 outperforms Ag3 with a different LLM — would not invalidate the paper's thesis but would establish that the optimal question-asking strategy is LLM-dependent, which has direct practical implications for system builders choosing an LLM backend. Additionally, measuring the effect of T2I model quality on the agent ranking would reveal whether better prompt-following models reduce the performance gap between agents (by rendering all prompts more faithfully) or amplify it (by making the quality of the specification more consequential).

Within-agent ablation of belief graph components to isolate causal factors. The paper compares three agent designs that differ along multiple dimensions simultaneously: Ag1 (heuristic scoring, multiple-choice only, belief graph for scoring, no conversation history), Ag2 (LLM-driven question generation, open-ended questions, belief graph in prompt, conversation history in prompt), Ag3 (LLM-driven question generation, open-ended questions, no belief graph, conversation history in prompt). The Ag1-vs-Ag2 gap conflates the question generation mechanism (heuristic vs. LLM), the question format (multiple-choice vs. open-ended), and the use of conversation history (none vs. full). This makes it impossible to attribute performance differences to specific design choices. A strong follow-up would systematically ablate within a single agent architecture: (a) Ag2 with importance scores set to uniform (1.0 for all elements) — testing whether miscalibrated importance scores are the bottleneck or whether any explicit scoring constrains the LLM; (b) Ag2 with only entity/attribute/relation names (no probabilities, no importance scores) — testing whether the structural decomposition alone helps; (c) Ag2 restricted to multiple-choice question format — isolating the effect of question format from question selection strategy; (d) Ag3 augmented with explicit belief graph information in its prompt — testing whether combining Ag3's principle-prompting with Ag2's belief access yields gains over either alone. The key finding to watch for is whether there exists any configuration where adding belief graph information to an otherwise well-performing agent (Ag3) improves performance. If no such configuration exists, the field should deprioritize symbolic uncertainty representations for LLM-based dialog agents; if a configuration does exist, it would identify the specific conditions (e.g., belief graph without importance scores, or belief graph with conversation-derived rather than parser-derived scores) under which explicit uncertainty helps.

Evaluating real-time human-agent interaction with outcome and experience metrics. The paper's human evaluation (Section 7.3) is retrospective: raters judge pre-generated dialogs and images, but never interact with a live agent. The gap between simulated-user performance (where the user always provides accurate, concise answers) and real-user performance is completely unmeasured. A strong follow-up would conduct a controlled user study where participants are assigned a specific goal image (from DesignBench or a similar benchmark) and asked to reproduce it using either (a) a standard single-turn T2I interface where they iteratively refine their own prompts, (b) an Ag3-powered agent that asks clarification questions, or (c) an Ag3-powered agent with the additional belief graph editing interface shown in Appendix F. Key outcome metrics: time to satisfactory image (user declares done), number of interaction turns, final I2I similarity to the goal image (using DINO), and post-task satisfaction ratings. Key process metrics: how often users decline to answer agent questions, how often they provide answers that contradict ground truth (intentional or not), how often they use the belief graph editing interface vs. relying on text Q&A, and whether usage patterns change over multiple tasks (learning effects). This study would provide the first measurement of whether the simulated-user results translate to real-user settings and would quantify the magnitude of the sim-to-real gap — essential information for anyone considering deploying such a system.

Extending the approach to domains beyond T2I, focusing on domains where underspecification is the primary bottleneck. The paper positions T2I as a "testbed for the communication problem" (Section 1), and the formal agent framework (⟨B, A, O, τ, π⟩) in Appendix B is domain-agnostic. But no evidence is provided that the approach transfers. A strong follow-up would apply the same belief-graph-plus-clarification-dialog architecture to a different underspecified generation task and measure whether similar gains materialize. Candidate domains where underspecification is acute: (a) code generation from natural language — prompts like "write a function to process user data" are severely underspecified (what data format? what processing? error handling? performance constraints?), and clarification questions could elicit specifications before code is written; (b) music generation from text — prompts like "a happy song" underspecify genre, tempo, instrumentation, key, and structure; (c) 3D scene generation — prompts like "a living room" underspecify layout, furniture style, lighting, and camera angle. For each domain, the follow-up would need to define: what replaces the belief graph structure (e.g., for code: functions, parameters, types, error conditions), what replaces the T2I model (a code generation model, a music generation model), what replaces the evaluation metrics (functional correctness for code, human preference for music), and what replaces the simulated user (a unit-test oracle for code, a music-theory rule checker for music). The key question is whether the 2×+ alignment improvement generalizes or whether T2I's specific properties — the ease of defining visual entities/attributes/relations and the availability of VQAScore as an automatic metric — make it uniquely suited to this approach.

Training a difficulty predictor to reduce upfront parsing cost. The paper identifies but does not address the computational cost of initial belief graph construction (Section 3.2 of the main paper, flagged in the Limitations section of this response). The entity/attribute/relation parsing pipeline requires 12+ sequential LLM calls before the first question can be asked. A strong follow-up would train a lightweight classifier — potentially a small fine-tuned language model or even a distilled version of the entity parser — that takes only the initial prompt p0 as input and directly outputs the top-k most uncertain and important belief graph elements (entities to confirm, attributes to specify, relations to clarify), bypassing the full parsing pipeline for the initial question. This classifier could be trained on (prompt, belief graph) pairs generated by the full pipeline across thousands of prompts, with a loss that encourages the classifier's importance-weighted uncertainty scores to match those of the full parser. The evaluation would measure: (a) correlation between the classifier's question ranking and the full pipeline's question ranking (do they select the same first question?), (b) end-to-end VQAScore after N turns when the first question is generated by the classifier vs. the full pipeline (does the latency saving cost any quality?), and (c) wall-clock time to first question. If a classifier can achieve 90%+ agreement with the full pipeline's first-question selection at 1% of the computational cost, it makes the agent practical for real-time interactive use.

Practical Applications and Downstream Use Cases

Creative professional tools for artists and designers. The paper's DesignBench benchmark directly targets this use case: professional creatives who use T2I as part of their workflow and for whom aesthetic specificity, stylistic control, and compositional precision are not optional but essential. The current workflow for these users — iteratively refining prompts while inspecting generated outputs — is inefficient because the model provides no signal about which missing details are causing misalignment. A deployed Ag3-style agent integrated into a creative tool (e.g., a Figma plugin, an Adobe Firefly interface, a Midjourney bot) would: (1) accept an initial concept description ("a surrealist book cover with a clock melting over a tree"), (2) ask 3–5 targeted clarification questions about style, composition, color palette, and key visual metaphors, (3) generate an initial set of images, and (4) allow the user to either refine via additional Q&A or directly edit the belief graph to adjust specific elements. The paper's results suggest this could reduce the number of manual prompt iterations from ~15 (the typical frustration level reported by 55% of users in Table 4) to ~5 agent-guided turns, with the generated images being preferred by humans in 80%+ of cases (Table 1, Human Eval). The specific metrics that justify this use case: DesignBench VQAScore of 0.955 for Ag3 vs. 0.353 for single-turn, and 63.5% human preference for Ag3 content alignment on DesignBench. The modular architecture means the T2I backend can be swapped as better models become available without rebuilding the clarification system.

Accessibility layer for non-expert T2I users. The paper's human survey (Table 4) reveals that prompt iteration is the most prevalent frustration among T2I users (83% experience it, 55% frequently or very frequently). Non-expert users — those who use T2I casually for social media, personal projects, or exploration — are disproportionately affected because they lack the prompt engineering skills to diagnose why their outputs don't match their intent. A deployed proactive agent serves as an accessibility layer that translates vague natural-language desires ("a cozy reading nook") into the detailed specifications that T2I models need, without requiring the user to learn prompt engineering. The agent's multiple-choice question format (Ag1) or open-ended conversational questions (Ag3) meet users at their level of expertise, and the belief graph visualization (even if static) provides transparency about what the agent is assuming. The paper's finding that 91% of users believe clarifications would help their workflow (Table 2) and that the questions are rated as clear and relevant in 86%+ of cases (Figure 4, left) supports this use case. The specific benefit: a non-expert user who currently needs 10–15 frustrating iterations to get an acceptable image could achieve better results (2×+ VQAScore improvement) in 5 agent-guided turns, with the agent bearing the cognitive load of identifying which details matter.

Data generation and filtering for T2I model evaluation and training. The paper's Appendix E.2 demonstrates that agent-user QA pairs can serve as an auxiliary quality filter for generated images: converting the first 5 Q&A pairs from an agent dialog into binary VQA questions and using VQAScore to select the best image from N seeds improves I2I similarity by +0.02 (from 0.7637 to 0.7838). This has practical implications for organizations that generate large volumes of images and need automated quality filtering. A deployed pipeline could: (1) use an agent to generate clarifying Q&A pairs for each prompt, (2) generate N images per prompt with different random seeds, (3) score each image against the Q&A-derived VQA questions, and (4) retain only images above a quality threshold. This is computationally cheaper than human review and more targeted than generic aesthetic scoring models (e.g., PickScore, ImageReward) because the VQA questions are specific to the user's stated intent for that particular image. The paper's modest +0.02 improvement likely understates the practical value because it was measured on perfectly specified ground truth captions — on real underspecified prompts where the T2I model makes diverse implicit assumptions across seeds, the Q&A-based filtering would have more variance to resolve and should yield larger gains.

Cultural adaptation and bias mitigation in deployed T2I systems. The paper explicitly frames clarification as a safety and inclusivity mechanism (Section 8, Impact Statement): "learning more about the user through clarification questions can mitigate risks, make models safer, and pave the way to more inclusive generative AI systems." The mechanism is direct: rather than the model defaulting to a culturally specific interpretation of an underspecified prompt (e.g., Western breakfast for "breakfast plate"), the agent asks "Which type of cuisine should be pictured?" and offers culturally diverse options. This transforms the model from an assumption-imposer (which may systematically exclude minority cultural representations) to an information-elicitor (which surfaces the user's actual cultural context). The practical deployment scenario: a global T2I service (e.g., Google's ImageFX, OpenAI's DALL-E) integrates a proactive agent that, upon receiving potentially culturally underspecified prompts, asks 1–2 clarification questions about cuisine type, clothing style, architectural context, or other culturally variable elements before generating. The benefit is twofold: (a) users from non-dominant cultures get images that reflect their actual intent rather than the model's default assumptions, reducing the well-documented problem of cultural homogenization in T2I outputs (Basu et al., 2023; Kannen et al., 2024), and (b) the service reduces the risk of generating culturally offensive content by checking the user's preferences rather than guessing. The paper's finding that importance scores can guide the agent toward asking about consequential attributes (cuisine type of breakfast vs. wood type of table) provides the mechanism for making these clarification questions targeted rather than overwhelming — the agent asks about the 1–2 most culturally impactful uncertainties, not about every possible attribute.

When to Prefer This Method

The paper does not explicitly position its proactive clarification approach against named alternative methods for addressing T2I underspecification (e.g., prompt expansion, user-driven iterative refinement, fine-tuned personalization models). It presents the agents as a novel paradigm and compares only against a passive single-turn baseline. The decision rules below are therefore derived from the paper's implicit positioning and experimental boundary conditions rather than from an explicit trade-off analysis articulated by the authors.

However, the paper's results do establish clear conditions under which the approach is most and least effective, which can be translated into practical guidance:

When the initial prompt is underspecified but the user has a concrete intended image they can describe when asked. This is the regime the automatic evaluation simulates: a ground truth image exists, the user can answer clarification questions accurately, and the bottleneck is the information flow from user to model. The paper's 2×+ VQAScore gains and 80%+ human preference apply here. If the user genuinely does not know what they want and is using T2I for exploration or inspiration, clarification questions may feel constraining rather than helpful — the paper provides no evidence either way.
When interaction latency of a few seconds per question is acceptable. The paper's unmeasured but likely substantial belief parsing latency (12+ LLM calls before the first question) means the agent is not suitable for sub-second real-time creative flow. The target use case is deliberate image creation where users are willing to invest 1–2 minutes in specification to avoid 10+ minutes of frustrating trial-and-error.
When the T2I model is capable of rendering the user's intent given a sufficiently detailed prompt. The paper's own analysis (Appendix E.2, Section 8) shows that Imagen 3's prompt-following fidelity imposes a ceiling: even with perfect prompts, DINO I2I similarity is 0.76, not 1.0. If the user's desired image requires capabilities the T2I model fundamentally lacks (e.g., precise spatial composition beyond what text can specify, fine-grained control over lighting, accurate text rendering), clarification questions will not help — the bottleneck shifts from specification quality to generation capability. Users in this regime would be better served by tools that provide direct visual manipulation (inpainting, ControlNet, image-to-image editing) rather than text-only clarification.
When the user population has diverse cultural backgrounds, aesthetic preferences, or specific requirements that generic T2I assumptions would fail to capture. The paper's strongest normative argument for proactive clarification is that models should not silently impose assumptions on users. This applies particularly in deployed systems serving global, heterogeneous user bases where a one-size-fits-all interpretation of underspecified prompts would systematically disadvantage minority users.
When the computational cost of the clarification loop is amortizable across multiple generations for the same or similar prompts. The upfront belief parsing cost is fixed per conversation. If a user asks one question and generates one image, the overhead-to-value ratio is unfavorable. If a user engages in a 10-turn conversation exploring variations on a theme, generating multiple images with different stylistic specifications elicited during the dialog, the parsing cost is amortized across all outputs.