Bridging Your Imagination with Audio-Video Generation via a Unified Director

🎯 Pitch

A single model can now think and draw like a film director—it devises a logically sound script and then paints the matching keyframes, all within one unified framework. By first learning from interleaved text–image storyboards and then decoupling the writing and drawing specialties, UniMAGE shatters the longstanding wall between narrative reasoning and visual imagination, achieving unparalleled character consistency and prompt alignment across long videos.

1. Executive Summary

This paper introduces UniMAGE, a unified director model that transforms the traditionally disjoint tasks of script drafting and keyframe generation into a single, coherent framework for long-form, multi-shot film creation. Built on the Mixture-of-Transformers architecture initialized from Bagel, UniMAGE employs two named training paradigms—Interleaved Concept Learning (jointly optimizing text and image generation using interleaved text–image script data to foster narrative understanding) and Disentangled Expert Learning (decoupling script writing from keyframe generation by training the understanding and generation experts separately on pure text scripts and text–image pairs, respectively)—further reinforced by In-Context ID Prompting (inserting special tokens among visual tokens to indicate frame and character identities) and Pre-Context Script Splitting (randomly dividing scripts to teach continuation from partial context). On the ViStoryBench benchmark, UniMAGE achieves state-of-the-art performance among open-source models, attaining the highest character identification similarity (59.2 CIDS), onstage character count matching (88.07 OCCM), and a large margin in prompt alignment (80.8), establishing that a single unified architecture can maintain both narrative coherence and visual consistency across long sequences without per-instance retraining.

2. Context and Motivation

The Core Problem: The Disjoint Between Narrative and Visual Creation

The fundamental gap this paper addresses is the architectural and philosophical separation between script writing and visual generation in existing AI-driven video creation systems. The authors articulate this cleanly in their introduction:

"Existing AI-driven video creation systems typically treat script drafting and key-shot design as two disjoint tasks: the former relies on large language models, while the latter depends on image generation models."

This separation mirrors what the paper describes as a fundamental misunderstanding of the filmmaking process. In traditional cinema, the director doesn't write a complete script, hand it off to a separate entity to generate storyboards, and then stitch the results together. Instead, narrative logic and visual imagination are co-developed — the script informs the visual composition, and visual thinking shapes the script. The director acts as what the paper calls a "conductor in an orchestra, bringing out the strengths of each element while maintaining overall harmony." Current AI systems lack anything resembling this unified creative intelligence.

The practical manifestation of this gap appears in several concrete forms:

Weak narrative-visual alignment: When an LLM writes script descriptions and a separate diffusion model generates corresponding keyframes, there is no mechanism for the two to negotiate. The text might describe a character with "long, dark brown hair" in shot 1, but by shot 7, the image generator — with no access to the original text description or the previous images — may produce a character with different features entirely. The paper shows this failure mode explicitly in Figures 5 and 6, where baseline methods produce visibly inconsistent characters across shots.
Limited long-range coherence: Single-shot video generation models (Veo 3, Sora 2, Keling 2.5) focus on visual fidelity and temporal coherence within individual clips, but "these models primarily focus on visual fidelity and temporal coherence within short, single-shot videos, which limits their ability to convey long-form narratives." A film is more than a sequence of individually beautiful shots — it requires logical plot progression, consistent character identities, environments that persist across scenes, and thematic continuity that spans the entire narrative.
Per-sample engineering burden: Prior agent-based approaches required explicit prompt engineering for each new story instance. The authors note that methods relying on prompting pre-trained LLMs "requires per-sample prompt engineering and often overlooks reasonable storyboard design, making it difficult to preserve logical coherence and temporal alignment in multi-scene storytelling."
Brittle identity preservation: Models struggle most acutely when scripts involve multiple characters. As the paper states: "when scripts involve multiple actors, existing models struggle to reliably recognize and maintain individual identities." This isn't merely an aesthetic problem — in narrative storytelling, character consistency is essential for audience comprehension and emotional engagement.

Why This Problem Matters

The significance of this gap operates on multiple levels:

Practical: Democratizing film creation. The paper opens with an Orson Welles quote: "A writer needs a pen, an artist needs a brush, but a filmmaker needs an army." Welles' observation captures the fundamental inequality in creative media: writing a novel or painting a canvas requires one person with one tool, but filmmaking historically requires large teams, expensive equipment, and specialized expertise. A unified director model that can transform simple user prompts into complete, illustrated scripts with consistent characters, logical plots, and cinematic composition would — if successful — significantly lower the barrier to film-level video creation. The paper explicitly positions UniMAGE as a tool "empowering non-experts to produce long-context, multi-shot films."

Technical: Verifying whether unified architectures can handle long-context multimodal generation. The paper tests a specific hypothesis: that a single transformer model, jointly trained on interleaved text–image data, can maintain coherence over sequences much longer than those addressed by prior unified multimodal systems. The authors explicitly note that "current unified models mainly handle short-context reasoning and generation, without mechanisms to preserve narrative or visual coherence across long sequences." UniMAGE pushes this boundary from single-turn or few-step generation to scripts spanning 10+ shots with multiple characters and environments.

Theoretical: Reunifying reasoning and imagination. The paper makes an implicit theoretical claim: that logical reasoning (script logic, plot development, dialogue) and imaginative thinking (visual style, composition, character design) are not separate cognitive processes but intertwined ones, and that an AI system trained on interleaved text–image data can develop a more integrated form of creative intelligence than one that separates these functions. This parallels arguments in cognitive science about the relationship between language and visual thinking, though the paper does not cite this literature directly.

Where Prior Approaches Fall Short

The paper organizes prior work into three categories and identifies specific limitations in each:

1. Agent-Based and Pipeline Systems

Systems like Anim-Director, FilmAgent, MovieAgent, and FilMaster represent the predominant paradigm: use an LLM to expand user prompts into structured scripts, then use separate image generation models to produce keyframes for each shot. The paper acknowledges the contributions of these systems — they demonstrated that LLMs can handle script breakdown, scene planning, and shot design — but identifies fundamental architectural limitations:

Decoupled generation leads to weak alignment. As noted above, the LLM and the image generator have no shared representation space. The LLM outputs text; the image generator conditions on that text. There is no feedback loop, no ability for visual considerations to influence narrative choices, and no mechanism to ensure that what the LLM describes as "a young woman with long, dark brown hair" in scene 3 is the same visual entity as "the woman" in scene 7.
No end-to-end learning of narrative structure. Because these systems compose separate modules, there is no gradient signal from the visual output back to the script writing process. The system cannot learn, for example, that certain types of narrative descriptions produce ambiguous visual results, or that maintaining character consistency requires specific kinds of textual cues.
StoryDiffusion and Story2Board represent partial solutions. The paper acknowledges that StoryDiffusion introduced consistent self-attention mechanisms for maintaining character appearance across image sequences, and Story2Board used lightweight consistency modules to preserve generative priors. However, both still rely on externally provided text scripts and focus exclusively on the visual consistency problem, leaving narrative coherence unaddressed.
SEED-Story attempted unification but had severe limitations. SEED-Story represents the closest prior work to UniMAGE's ambition — it uses a unified MLLM to produce multimodal stories (text + images). However, the paper identifies critical shortcomings: "its training is limited to only three animation datasets, resulting in poor generalization beyond stylized domains" (Section 4.1). Additionally, "it requires separate training for each story instance, and its generated images still suffer from limited visual quality." In other words, SEED-Story was unified in architecture but not generalizable in practice — it overfit to narrow visual styles and couldn't produce high-quality images outside its training distribution.

2. Unified Multimodal Generation Models

The paper discusses two architectural families for unified multimodal generation:

Auto-regressive (AR) approaches (Emu, Chameleon) treat images as continuous or discrete visual tokens alongside text and train a single next-token prediction objective. The paper acknowledges their conceptual elegance but identifies a critical quality limitation: "AR models generally generate lower-quality images compared with diffusion-based approaches." This is consistent with broader findings in the field — AR image generation, while architecturally simpler, has not matched the visual fidelity of diffusion models.

Diffusion-integrated approaches (Show-O, TransFusion, Bagel) integrate diffusion modules into language model architectures, synthesizing text and images within a shared transformer backbone. Bagel's Mixture-of-Transformers architecture is particularly relevant because UniMAGE builds directly on it. However, the paper identifies a context-length gap: "current unified models mainly handle short-context reasoning and generation, without mechanisms to preserve narrative or visual coherence across long sequences." Bagel was designed for multi-turn image editing tasks, not for generating 10+ shot scripts with consistent characters across all frames. The paper's contribution is extending this unified architecture to handle the long-context, consistency-demanding domain of film script generation.

3. Audio-Video Generation Models

Commercial systems like Veo 3 and Sora 2 represent the state of the art in single-shot audio-video generation. The paper acknowledges their "remarkable creative potential" but identifies their primary limitation as lack of multi-shot narrative planning: "these models rely heavily on well-structured prompts and are typically limited to short, single-shot videos. To compensate, they often incorporate LLM-based re-captioning modules to refine user input before generation." In other words, these systems are reactive — they generate what you describe — but not proactive — they cannot plan a sequence of shots, maintain character identity across scenes, or develop a plot. They are the equivalent of a cinematographer who can shoot any single scene beautifully but has no sense of how scenes connect into a story.

Additionally, "current methods still lack the capability to plan and maintain coherence across multi-shot, narrative-driven sequences." UniMAGE is positioned not as a replacement for these generation systems but as a director layer above them — it generates the structured scripts and keyframes that serve as high-level guidance for downstream audio-video generation models.

How This Paper Positions Itself

UniMAGE positions itself at the intersection of two converging trends — unified multimodal generation and AI-assisted filmmaking — while addressing the specific gap that neither camp has solved: long-context narrative coherence with visual consistency.

The positioning is explicit in several dimensions:

Against agent-based systems: UniMAGE argues that separate LLM and image generation agents are fundamentally limited because they cannot jointly reason about narrative and visual composition. The paper frames this as a philosophical position: "we posit that a unified understanding and generation model is essential to bridge user intents with audio-video generation systems, rather than relying on separate agents." This isn't merely an engineering preference — it's a claim about what kind of architecture is necessary for coherent story generation.

Against prior unified models: UniMAGE acknowledges its architectural debt to Bagel (it initializes from Bagel's pre-trained MoT weights) but argues that Bagel's training paradigm — focused on multi-step image editing — is insufficient for the demands of film script generation. The key innovations (Interleaved Concept Learning, Disentangled Expert Learning, In-Context ID Prompting, Pre-Context Script Splitting) are all mechanisms for extending a short-context unified model into the long-context, high-consistency regime.

Against single-shot video models: UniMAGE explicitly positions itself as a complementary system rather than a competitor: "The generated scripts and keyframes can further serve as structured, high-level guidance for existing audio-video joint generation models" (Figure 1 caption). The paper includes two video demonstrations (UniMAGE_Demo_1.mp4 and UniMAGE_Demo_2.mp4) generated by feeding UniMAGE's scripts into Veo 3, establishing the pipeline: user prompt → UniMAGE (script + keyframes) → audio-video generation model → final film.

Conceptual positioning: The "director" metaphor. The paper's most distinctive positioning is the metaphor itself: UniMAGE is not an LLM, not a diffusion model, not an agent pipeline, but a director. This metaphor carries specific implications: a director doesn't paint every frame (that's the cinematographer's job) or write every line of dialogue (that's the screenwriter's), but rather orchestrates — making high-level creative decisions about narrative structure, visual style, pacing, and shot composition, and ensuring that all elements cohere into a unified artistic vision. UniMAGE's output — structured scripts with global descriptions, per-shot content descriptions, and keyframe images — mirrors a director's storyboard and shooting script. This positioning matters because it defines what UniMAGE is not trying to do: it's not trying to replace video generation models, not trying to generate final audio, not trying to handle every aspect of filmmaking. It's trying to fill the specific gap between user intent and multimodal generation that currently requires either human expertise or brittle, disconnected AI pipelines.

The pretraining strategy as a theoretical claim. The paper's two-stage training paradigm (Interleaved Concept Learning followed by Disentangled Expert Learning) embeds a specific theoretical claim: that the model must first learn the holistic relationship between text and images in narrative contexts (interleaved learning), and only then can it productively decouple text-only script generation from image generation (disentangled learning). This "first interleaving, then disentangling" approach is not an arbitrary training schedule — it reflects a hypothesis about how unified multimodal understanding develops, echoing the paper's central argument that reasoning and imagination are fundamentally intertwined processes that can be productively separated only after their integration is established.

3. Technical Approach

3.1 Reader Orientation

UniMAGE is a single transformer-based model that takes a user's text prompt and produces a complete illustrated film script — including character descriptions, environment descriptions, per-shot camera and action descriptions, dialogue with sound cues, and corresponding keyframe images — all within one unified architecture. The core problem it solves is that existing systems split script writing and image generation across separate models (LLMs and diffusion models), which breaks the tight coupling between narrative logic and visual imagination that a human film director maintains; UniMAGE solves this by training a Mixture-of-Transformers model to jointly reason about text and images in an interleaved format, then strategically decoupling the text and image generation processes to gain flexibility without losing coherence.

3.2 Big-Picture Architecture (Diagram in Words)

UniMAGE has five major architectural and training components:

Mixture-of-Transformers (MoT) Backbone — A shared transformer framework containing two expert modules (one for multimodal understanding, one for image generation) that process text, ViT, and VAE tokens through shared self-attention layers. Initialized from Bagel's pre-trained weights.
Script Structure Parser — A predefined schema of special tokens (<User>, <CharacterN>, <EnvironmentN>, <FrameN>, <VideoN>) that organizes a film script into global descriptions (characters, environments), per-shot content descriptions (frame-level visual layout and video-level narrative/audio), and keyframe images.
Interleaved Concept Learning Stage — A training phase where all MoT parameters are jointly optimized on interleaved text–image script data, teaching the model to generate both text and images in sequence while maintaining long-range associations between narrative descriptions and visual entities.
Disentangled Expert Learning Stage — A subsequent training phase where the understanding expert is trained on pure text scripts (for script writing/continuation) while the generation expert is trained on text–image data (for keyframe generation), with the understanding branch frozen during generation optimization to prevent gradient interference.
Inference Pipeline — A three-step generation process: (1) generate multi-shot text script from user prompt, (2) optionally extend or continue the script using new prompts or system continuation cues, (3) generate keyframe images for each shot in an interleaved manner, conditioned on the preceding text context.

Information flows: user prompt → [optional: preceding script context] → MoT model generates structured text script (global descriptions + per-shot content) → user can insert <Extension> or <Continuation> prompts → MoT model generates additional text shots → complete text script is segmented into individual shots → MoT model generates keyframe images for each shot, conditioned on all preceding text and images → final illustrated script is output.

3.3 Roadmap for the Deep Dive

First, the MoT architecture foundation (why two experts, how tokens are processed, what encoders are used), because all subsequent mechanisms depend on understanding this shared backbone.
Second, the script structure and special token schema, since this defines the input/output format that the training paradigms operate on.
Third, the Interleaved Concept Learning stage and the In-Context ID Prompting mechanism, which together enable the model to maintain visual consistency across long sequences.
Fourth, the Disentangled Expert Learning stage and the Pre-Context Script Splitting strategy, which decouple script writing from keyframe generation and enable flexible narrative continuation.
Fifth, the inference procedure, which shows how the trained model is actually used at test time to generate complete illustrated scripts.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and training methodology paper whose core idea is that a unified transformer model, trained with a two-stage "first interleaving, then disentangling" paradigm, can jointly produce coherent narrative text and visually consistent keyframe images for long-form, multi-shot film scripts.

Mixture-of-Transformers (MoT) Architecture Foundation

UniMAGE inherits its backbone from Bagel's Mixture-of-Transformers architecture, which addresses a fundamental tension in unified multimodal models: text understanding benefits from autoregressive next-token prediction, while image generation achieves higher quality with diffusion-based approaches. Rather than forcing one paradigm to serve both modalities, MoT uses two separate transformer experts that share the same token sequence through common self-attention layers.

The architecture, detailed in Appendix A and illustrated in Figure 10, consists of:

Two transformer experts with shared multimodal self-attention. The understanding expert processes text tokens using standard autoregressive next-token prediction. The generation expert processes image tokens using rectified flow diffusion. Critically, both experts attend to the same sequence of tokens through shared self-attention layers in each transformer block — meaning the understanding expert can "see" image tokens and the generation expert can "see" text tokens, enabling cross-modal information flow. The transformer parameters are initialized from Qwen 2.5 LLM. Each attention block includes QK-Norm for training stability.

Two distinct visual encoders for different purposes. The architecture uses separate encoders for visual understanding and visual generation, recognizing that these tasks require different representations:

Understanding encoder (ViT): Uses SigLIP2-so400m/14 with a fixed input resolution of 384 for initialization. Position embeddings are interpolated to handle variable resolutions, with a maximum input size set to 980 × 980 pixels. NaViT is incorporated to process images in their native aspect ratios rather than forcing square crops. A two-layer MLP connector projects ViT token dimensions to match the LLM hidden state dimension. This encoder captures semantic and high-level visual information relevant to understanding.
Generation encoder (VAE): Uses a pre-trained VAE from FLUX, which maps images between pixel space and a latent space with a downsampling ratio of 8 and 16 latent channels. A 2 × 2 patch embedding layer further reduces spatial dimensions and aligns with the LLM hidden size. This encoder captures low-level, high-frequency visual information needed for high-quality image synthesis.

Token processing and attention scheme. Within MoT, text, ViT, and VAE tokens are interleaved according to the multimodal input structure. For tokens from the same sample, a generalized causal attention scheme is used: tokens are divided into sequential splits by modality (text, ViT, or VAE), where each split can attend to all earlier splits, with causal attention applied to text tokens (preventing future-token leakage during autoregressive generation) and bidirectional attention applied to vision tokens (allowing full spatial context within each image). This design enables the model to condition image generation on preceding text while maintaining efficient computation.

What the MoT architecture enables. The key property is that both experts process the same token sequence through shared attention, meaning the understanding expert's text reasoning can directly influence the generation expert's image synthesis, and vice versa. This shared representation space is what makes the "unified director" concept technically feasible — the model can develop internal representations where "Character 1" in text and "Character 1" in images map to the same latent identity, which is precisely what separate LLM + diffusion model pipelines cannot do.

The paper formulates two training objectives corresponding to the two modalities:

For text token prediction, UniMAGE follows the Next Token Prediction (NTP) paradigm. Given a token sequence $y = (y_1, y_2, ..., y_n)$ drawn from a fixed vocabulary $V$ , the autoregressive model decomposes the joint probability as:

$P_\theta(y) = \prod_{t=1}^{T} p_\theta(y_t \mid y_{<t})$

where $y_{<t} = (y_1, y_2, ..., y_{t-1})$ denotes all tokens preceding position $t$ , and $p_\theta(y_t \mid y_{<t})$ is the model's predicted probability distribution over the vocabulary for the next token given the history.

What it computes: the probability of the full token sequence as the product of conditional next-token probabilities. At each position $t$ , the model takes all previous tokens as input and outputs a probability distribution over the vocabulary; the probability assigned to the actual token $y_t$ that appeared in the training data is extracted, and these are multiplied to get the sequence probability. The model parameters $\theta$ are optimized via maximum likelihood estimation:

$\theta^* = \arg\max_\theta \sum_{t=1}^{T} \log p_\theta(y_t \mid y_{<t})$

Why this form: autoregressive factorization is the standard objective for LLMs because it enables efficient training (all positions can be computed in parallel using causal masking) and supports natural text generation at inference time (sample one token, feed it back as input, repeat). The log form converts the product into a sum for numerical stability.

For visual token prediction, UniMAGE uses Rectified Flow, an ordinary differential equation (ODE) model that transports samples from a noise distribution $\pi_0$ to a data distribution $\pi_1$ along straight trajectories. Given two samples $X_0 \sim \pi_0$ (noise) and $X_1 \sim \pi_1$ (real image latent), the drift field $v$ is trained to approximate their difference:

$\min_\phi \int_0^1 \mathbb{E}\left[\lVert (X_1 - X_0) - v_\phi(X_t, t) \rVert^2\right] dt$

where $X_t = (1-t)X_0 + tX_1$ is a linear interpolation between noise and data at timestep $t \in [0,1]$ , $v_\phi$ is a neural network (the generation expert) that predicts the velocity field, and $\phi$ denotes the network parameters.

What it computes: a velocity vector at each point along the straight path from noise to data. The training objective minimizes the expected squared error between the true direction $(X_1 - X_0)$ (which points directly from the noise sample to the corresponding real image) and the model's predicted direction $v_\phi(X_t, t)$ at the interpolated point $X_t$ . After training, generating an image involves sampling noise $X_0 \sim \pi_0$ , and integrating the ODE $\frac{dX_t}{dt} = v_\phi(X_t, t)$ from $t=0$ to $t=1$ using a numerical solver (requiring only a few steps due to the straight trajectories).

Why this form: Rectified Flow is preferred over standard diffusion because the straight trajectories enable few-step sampling, which is computationally efficient. Unlike DDPM-style diffusion that follows curved stochastic paths requiring many denoising steps, Rectified Flow's ODE formulation with straight paths allows for accurate integration in as few as 1–4 steps. The mean squared error objective on the velocity field is also simpler to implement and train than score-matching objectives used in some diffusion variants.

Script Structure and Special Token Schema

UniMAGE's script structure, illustrated in Figure 2, defines a precise format that organizes film narrative into hierarchical components. This structure is what the model is trained to generate and what downstream audio-video generation systems consume.

The complete script $S = (G, C, F)$ consists of three components:

Global descriptions $G$ : These define the persistent entities that appear across multiple shots. For each character, a <CharacterN> token introduces a textual description of appearance (e.g., "A young woman with long, curly auburn hair and a serene, thoughtful expression. She wears an elegant dark..."). For each environment, an <EnvironmentN> token introduces a description of the setting (e.g., "A bow of a grand ocean liner at sunset, bathed in warm golden light..."). The N is an integer index that uniquely identifies each entity throughout the script. These global descriptions serve as a reference that all subsequent per-shot content can point to.

Content descriptions $C = (c_1, c_2, ..., c_n)$ : For each of $n$ shots, two complementary layers of description are provided:

Frame descriptions (<FrameN>): These capture the static visual layout of the key moment in each shot — camera position (medium shot, close-up, long shot), lighting conditions, character placement and poses, environmental elements visible in frame, and spatial relationships between entities. For example: "Medium close-up: On the deck of the grand ocean liner<Environment1>, a young man<Character2> stands bathed in the warm hues of a setting sun."
Video descriptions (<VideoN>): These capture temporal and narrative aspects that unfold over the duration of the shot — camera movement (pan, zoom, track), character actions and interactions, dialogue (marked with <- -> indicator symbols), emotional tones, plot progression, and background audio descriptions. For example: "The camera slowly pans upward to reveal <Character2> standing on the ship's deck, the wind brushing through his blond hair. He gazes out toward the horizon, eyes filled with wonder and quiet joy..."

Keyframe images $F = (f_1, f_2, ..., f_n)$ : For each shot, a corresponding keyframe image that visually depicts the moment described in the frame and video descriptions. These are the generated image outputs that make the script "illustrated."

Special token functions. The special tokens serve multiple purposes simultaneously:

<CharacterN> and <EnvironmentN> tokens appear both in the global description section (where they are defined) and in the content descriptions (where they are referenced). When these tokens appear in content descriptions — for example, "a young man<Character2> stands..." — they create an explicit pointer from the narrative text to the global entity definition. This is not merely formatting; it is a mechanism that the model learns to use during Interleaved Concept Learning to maintain entity identity across shots. The tokens function as "anchors" that the shared self-attention layers can use to route information between the global definition and each occurrence.
<FrameN> and <VideoN> separate the static and temporal aspects of each shot, enabling the model to reason about visual composition (frame) independently from narrative progression (video). This separation matters because downstream audio-video generation models need different information for visual synthesis versus audio/dialogue generation.
The <- -> indicator symbols mark character dialogue and environmental sound effects. For example, <-Now close your eyes. Go on.-> marks spoken dialogue, while <-Soft, inquisitive orchestral music begins-> marks audio direction. These markers enable "the corresponding audio content to be easily retrieved in subsequent stages" — a text-to-speech or audio generation system can parse the script for these markers to extract exactly what needs to be spoken or generated as audio.

User prompt styles. To accommodate diverse input formats during training and inference, the paper defines four distinct prompt styles that are randomly sampled during training (detailed in Appendix B):

Simple narrative: Direct story descriptions (e.g., "a cute anthropomorphic kitten, wearing an apron, holding a frying pan...").
Abstract concept: High-level thematic prompts (e.g., "the ancient Qin Shihuang unified the six kingdoms").
Phrase splicing: Technical shot-list style descriptions with camera directions and visual specifications.
Spoken expression: Natural language requests as a user might speak them (e.g., "generating a video of a panda mother riding a shared bicycle...").

By training on all four styles, the model learns to map diverse user inputs into the structured script format.

The overall pipeline formulation. The paper formalizes UniMAGE's operation as:

$\text{UniMAGE}(\rho, \hat{S}) \mapsto (G, C, F)$

where $\rho$ is the user prompt, $\hat{S}$ is an optional preceding script context (used for extension and continuation, indicated by the superscript to mean it is optional), and the output is the complete script with global descriptions $G$ , content descriptions $C$ , and keyframe images $F$ . When $\hat{S}$ is provided, UniMAGE performs script continuation or extension, generating new shots that follow logically from the existing narrative. When $\hat{S}$ is absent, UniMAGE generates the complete script from scratch based solely on the user prompt.

Interleaved Concept Learning

Interleaved Concept Learning is the first training stage, designed to teach the MoT model the holistic relationship between narrative text and visual imagery in the context of long-form scripts. The paper describes this as "conceptually similar to the Chain-of-Thought strategy, where the text content functions as the model's reasoning process, followed by image generation conditioned on the preceding narrative context."

Training data format. The multi-shot text–image script dataset (450k sequences, constructed as described in Appendix B) is organized into an interleaved format where text tokens and image tokens alternate according to the script structure. A training sample might contain: <User> prompt tokens → <Character1> description tokens → <Environment1> description tokens → <Frame1> description tokens → <Video1> description tokens → image tokens for keyframe 1 → <Frame2> description tokens → <Video2> description tokens → image tokens for keyframe 2 → ... and so on for all shots in the sequence.

What happens during this stage. All parameters of both transformer experts (the understanding expert and the generation expert) are jointly optimized on this interleaved data. The losses from both text token prediction (next-token prediction cross-entropy) and image token prediction (rectified flow velocity prediction) backpropagate through the shared self-attention layers. This means that:

When the model generates an image, the generation expert's gradients flow through attention layers that also process the preceding text tokens, forcing the image generation to be conditioned on the narrative context.
When the model generates text, the understanding expert's gradients flow through attention layers that have previously processed images, allowing visual information to influence subsequent text generation.

Training configuration. During this stage, only the multi-shot text–image script data is used (the 450k sequences). The learning rate is 1e-5, and training proceeds for 30,000 steps. The model is initialized from Bagel's pre-trained weights, which provide foundational multimodal understanding and generation capabilities.

Why interleaving matters for long-context scripts. The paper argues that the long-context nature of script data "exceeds the base model's capacity," referring to Bagel's original training which focused on short-context multi-step image editing. In a film script, maintaining that "Character 1" in shot 3 refers to the same entity as "Character 1" in shot 12 requires the model to maintain associations across potentially thousands of tokens. By training on interleaved text–image sequences where the global character descriptions appear early and are referenced throughout, the model learns to use the shared self-attention mechanism to route identity information across long temporal spans.

Why not train with disentangled data from the start? The paper implicitly argues that if training began with disentangled data (text-only scripts for the understanding expert, text–image pairs for the generation expert), the model would never develop the cross-modal associations needed for coherent illustrated scripts. The generation expert would learn to produce images from text descriptions without ever seeing how those images should relate to each other across a sequence. The understanding expert would learn to write scripts without any grounding in visual feasibility. The interleaved stage first "fosters a joint understanding of narrative concepts through text–image interleaving" (as stated in the conclusion), establishing the shared representation space that the subsequent disentangled stage can then exploit.

In-Context ID Prompting

In-Context ID Prompting is a mechanism introduced during Interleaved Concept Learning to specifically address the problem of multi-character and multi-scene visual consistency. The core challenge is described as: "scripts often involve multiple characters and scenes, making it highly challenging for the model to consistently maintain the identities and visual coherence of different entities across long sequences."

The mechanism. Within the ViT tokens used for understanding and the VAE tokens used for image reference, the model inserts special text tokens that explicitly indicate:

Frame ID: which shot this image corresponds to (e.g., <Frame3>).
Environment ID: which environment(s) appear in this image (e.g., <Environment1>).
Character ID: which character(s) appear in this image (e.g., <Character1>, <Character2>).

These ID prompt tokens are inserted among the visual tokens (both ViT and VAE) for each image, and full attention is applied between each image's visual tokens and its corresponding ID prompt tokens. The left part of Figure 4 illustrates this: image tokens are interspersed with <Frame ID>, <Environment ID>, and <Character ID> tokens, with full bidirectional attention connecting them.

Why this addresses the consistency problem. The paper identifies the root cause of visual inconsistency as the model's inability to "recognize the characters and scenes depicted in each image and associate them with the global text descriptions and historical keyframes." Without explicit ID prompting:

The model has no direct signal about which character is in a given image — it sees only the visual pixels and the preceding text description, but the mapping between "the woman with long, dark brown hair" in the text and the specific set of VAE tokens representing that woman in the image is learned implicitly through attention, which can fail over long sequences.
When generating a new image, the model has no explicit mechanism to recall "I am now generating Character 2, who was last seen in Frame 4 with Environment 1" — it must rely on learned attention patterns, which become increasingly unreliable as the context grows.

By inserting explicit ID prompt tokens, the model receives a direct conditioning signal: "the visual tokens that follow are for Frame N, depicting Environment X and Characters Y and Z." During training, the model learns to associate these ID tokens with the corresponding visual features. During generation, when the model is asked to produce a new keyframe, the ID prompt tokens provide a strong conditioning signal that anchors the generation to the correct entities.

The role of shared text–image alignment. The paper notes that "text tokens and image tokens are aligned within the pre-trained unified architecture," meaning the MoT backbone already has some capacity to map between text descriptions and visual features. In-Context ID Prompting leverages this existing alignment by using text tokens (the IDs) to "prompt and highlight key information represented in the images." The full attention between ID tokens and visual tokens ensures that the model can attend to the ID information when processing or generating the image.

Interaction with global descriptions. The ID prompt tokens create a bridge between three sources of entity information: (1) the global text descriptions (which define what Character 1 looks like), (2) the ID tokens in each image (which label which character is present), and (3) the visual features in each image (which depict the character). The shared self-attention layers can learn to route information along all three paths: from global description to ID token to visual features, maintaining consistent identity across the entire sequence.

Ablation evidence. Figure 8 shows the effect of removing In-Context ID Prompting. Without it, the model exhibits "mixed appearance and inconsistent depictions" — for example, a character's hairstyle, facial structure, or clothing changes noticeably between shots, particularly after scene transitions or long temporal spans. The red circles in the figure mark specific failure cases where identity cues become ambiguous. The quantitative results in Table 1 confirm this: removing ID-P degrades CSD (from 59.0 to 52.6) and CIDS (from 59.2 to 55.6), demonstrating the mechanism's importance for both style consistency and character identity preservation.

Disentangled Expert Learning

After Interleaved Concept Learning establishes cross-modal associations, Disentangled Expert Learning addresses two limitations of the interleaved-only approach: (1) the model's inflexibility for script extension and continuation tasks, and (2) the difficulty of obtaining logically consistent multi-shot text–image data at scale, which limits the model's ability to fully learn long-form narrative logic.

The core idea. The paper argues that "interleaved generation strategy inevitably constrains the model's flexibility in content creation, particularly for tasks such as script extension and continuation, where the model must dynamically adapt to new user prompts or seamlessly expand upon existing narrative contexts." The solution is to decouple the training of script content generation from keyframe image generation, allowing each to be optimized with the most appropriate data.

Training procedure. As illustrated in the right part of Figure 3:

Understanding expert optimization: The understanding transformer expert is trained on pure text scripts (the 250k multi-shot text script subset), using only the next-token prediction objective. During this training, the generation expert is either not used or its parameters are frozen. This teaches the understanding expert to generate coherent, logically structured scripts without being constrained by the need to also generate images.
Generation expert optimization: The generation transformer expert is trained on text–image data, including both interleaved script data (the 450k sequences) and single-shot text–image pairs (the 250k additional samples). Crucially, during generation optimization, the understanding branch is frozen via stop-gradient — meaning gradients from the image generation loss do not flow back into the understanding expert's parameters. Additionally, "the text tokens are detached from the computational graph to prevent gradient propagation and ensure disentangled learning."
Incorporation of additional data: The single-shot text–image pairs (250k samples, each recaptioned with Gemini 2.5 Pro to obtain script-structured descriptions) are leveraged during generation expert training. These pairs provide diverse, high-quality visual examples that improve image fidelity, character rendering, and scene composition without requiring the overhead of full multi-shot consistency (since each image is independent).

Training configuration. This stage uses the entire dataset (multi-shot text–image scripts + multi-shot text scripts + single-shot text–image pairs). The learning rate remains 1e-5, and training proceeds for 10,000 steps. Only pure textual samples are used to optimize the understanding branch; during generation branch optimization, the text tokens are detached from the computation graph.

Why disentanglement helps. The paper identifies three benefits:

Flexible script generation: By training the understanding expert on pure text scripts (without the burden of also generating images), the model can fully exploit diverse textual script data to learn rich narrative logic, shot transitions, and dialogue conventions. The 250k multi-shot text scripts provide examples of well-structured narratives that may not have corresponding high-quality visual data.
Better image quality: By incorporating single-shot text–image pairs (which are easier to obtain at high quality than multi-shot consistent sequences), the generation expert can learn from a broader and higher-quality visual dataset. The additional 250k single-shot pairs improve "visual precision, diversity, and controllability" for character rendering and scene composition.
Task specialization: The disentangled training effectively creates two specialized sub-models within the shared architecture — the understanding expert becomes a script writer, and the generation expert becomes a keyframe illustrator — while preserving the shared self-attention layers that enable cross-modal communication. This is analogous to how a human film director might separately develop screenwriting skills and visual composition skills while maintaining an integrated creative vision.

What "freezing" means technically. When the paper says "the LLM parameters are frozen when optimizing the diffusion loss" and "the understanding branch is frozen via stop-gradient," this means that during a forward pass that processes text tokens followed by image tokens, the text tokens are still fed through the understanding expert to produce hidden states (which the generation expert can attend to via shared self-attention), but the loss computed on the image tokens does not backpropagate into the understanding expert's parameters. This preserves the text representations that were learned during Interleaved Concept Learning while allowing the generation expert to improve its image quality independently.

Pre-Context Script Splitting

Pre-Context Script Splitting is a data augmentation strategy applied during Disentangled Expert Learning to enable two critical inference-time capabilities: prompt-based script extension and in-context script continuation.

The problem. Standard autoregressive training on complete scripts teaches the model to generate scripts from start to finish given a user prompt, but does not teach it to:

Take an existing partial script and extend it with new content based on an additional user prompt (extension).
Take an existing script and naturally continue generating subsequent shots without any new prompt (continuation).

These capabilities are essential for practical film creation, where users typically iterate on their scripts, adding new scenes or continuing the narrative from where it left off.

The mechanism. As illustrated in the middle and right parts of Figure 4, the training data is augmented with two types of splits:

Prompt-based script extension: A complete text script is randomly divided into two parts at some point in the narrative. The indicator token <Extension> is inserted at the split point, followed by a new user prompt that describes the content of the second part. This new user prompt is generated by summarizing the second part of the script using Qwen 2.5. During training, the model sees: [first part of script] <Extension> [new prompt] [second part of script]. This teaches the model that when it encounters the <Extension> token, it should expect a user prompt and then generate narrative content that continues from the preceding context while addressing the new prompt.

In-context script continuation: Instead of a mid-narrative split, the split occurs before the last shot of the script. The indicator token <Continuation> is inserted, followed by a system prompt (e.g., "Based on the above script, continue to write the subsequent shot content."). The model then learns to generate the final shot and, during inference, can continue generating additional shots indefinitely by repeatedly applying the continuation mechanism. This is illustrated in the right part of Figure 4: a script with shots 1 through n has <Continuation> inserted before shot n, teaching the model to continue from partial context.

How it works during inference. At test time, the user can:

After the initial script is generated, provide a new prompt prefixed with <Extension> to add new scenes that follow from the existing narrative while incorporating new creative directions.
Use the <Continuation> token to prompt the model to keep generating more shots beyond what was initially produced, enabling potentially infinite-length scripts.

Why random splitting matters. The paper notes that by "randomly dividing complete scripts so that the model learns to continue script generation based on the preceding context or a given user prompt," the model develops a generalizable continuation skill rather than memorizing specific split points. During training, the split can occur at any point in the narrative, teaching the model that continuation is a general operation, not tied to specific narrative transitions.

Ablation evidence. The bottom part of Figure 8 shows the effect of removing Pre-Context Script Splitting (comparing interleaved-only training to Disentangled Expert Learning with Pre-Context Script Splitting). Without this mechanism, the model "frequently results in repetitive descriptions and weakened narrative flow" — specifically, the red box highlights content repetition issues where the model generates nearly identical shot descriptions multiple times in a row. With Pre-Context Script Splitting, the model demonstrates the ability to extend scripts with logical new developments (shown in the right panel of Figure 8, where an archaeology adventure script is coherently extended with an <Extension> prompt about discovering a hidden chamber, and then continued with <Continuation> to add ancient guardian encounters).

Inference Procedure

During inference, UniMAGE operates in a staged, disentangled manner that reflects the training paradigm while enabling flexible user interaction. The process proceeds in three phases:

Phase 1: Script generation. Given a user prompt $\rho$ , the understanding expert generates a complete multi-shot text script conditioned only on this prompt. This phase uses pure autoregressive text generation — the model produces the global descriptions (characters, environments), followed by the per-shot frame and video descriptions for all $n$ shots. No images are generated at this stage. The separation of text generation from image generation at inference time is a direct consequence of the Disentangled Expert Learning stage: the understanding expert has been trained to produce coherent scripts independently.

Phase 2: Optional extension or continuation. After the initial script is generated, the user has two options:

Extension: Provide a new prompt describing additional content. The system prepends <Extension> and this prompt to the existing script, and the model generates new shots that follow logically from the preceding narrative while incorporating the new creative direction. This can be repeated multiple times to iteratively build a longer script.
Continuation: Use the <Continuation> token with a system prompt to instruct the model to continue generating additional shots beyond the current endpoint. This enables potentially infinite-length scripts without requiring new user input for each extension.

Phase 3: Keyframe generation. Once the complete text script is finalized (including any extensions or continuations), it is segmented into individual shots. For each shot, the generation expert produces a keyframe image in an interleaved manner — meaning the model conditions on all preceding text (global descriptions, frame descriptions, video descriptions) and all previously generated keyframes when generating each new image. This interleaved conditioning is crucial for maintaining visual consistency: when generating the keyframe for shot 7, the model attends to the keyframes for shots 1–6, allowing it to maintain consistent character appearances, environmental continuity, and visual style.

The "unified yet disentangled" principle. The paper emphasizes that this inference strategy — "where a single model handles both modalities while separating the generation processes for text and image" — is key to producing "longer, more coherent narratives while mitigating issues such as plot repetition and image distortion." The unified architecture ensures that text and image generation share the same representational space (the shared self-attention layers), while the staged inference process allows the model to first establish narrative coherence (text only) and then focus on visual consistency (images conditioned on completed text and previous images). This prevents the model from having to simultaneously optimize narrative logic and visual quality at each step, which would likely lead to compromises in both.

Relationship to downstream generation. The generated scripts and keyframes are designed to serve as structured input to existing audio-video generation models (as demonstrated with Veo 3 in the supplementary videos). The script's structure — with explicit character descriptions, environment descriptions, camera directions, dialogue markers, and sound effect indicators — provides the detailed conditioning that these downstream models need to produce coherent multi-shot videos. The keyframes provide visual reference for character appearance and scene composition that can guide video generation to maintain consistency.

4. Key Insights and Innovations

Innovation 1: The "Unified Director" as a New Abstraction for Multimodal Generation

The paper's most fundamental contribution is not any single architectural trick but the conceptual reframing of what a multimodal generation system should be. Prior work in AI-assisted filmmaking treated the problem as orchestration: an LLM writes the script, a diffusion model generates images, and some glue code stitches them together. Systems like MovieAgent, FilmAgent, FilMaster, and Anim-Director all follow this paradigm — they are pipelines of specialized agents, each handling one aspect of the creative process. The dominant assumption was that script writing and visual generation are sufficiently different tasks that they require separate models, separate training, and separate inference.

UniMAGE challenges this assumption at the architectural level by positing that a single model can be a director — not a screenwriter plus a cinematographer glued together, but an integrated creative intelligence that jointly reasons about narrative structure and visual composition. The paper makes this explicit in its opening: "logical reasoning and imaginative thinking are both fundamental qualities of a film director." This isn't just marketing language; it's a claim about what kind of architecture is necessary for coherent story generation. If reasoning about plot and imagining visual scenes are intertwined cognitive processes, then a system that separates them into disconnected modules will necessarily produce weaker alignment between text and image.

What makes this genuinely novel — rather than an obvious extension of unified multimodal models — is that the director abstraction imposes qualitatively different demands than prior unified systems. Models like Bagel, Show-O, and Transfusion demonstrated that a single transformer can both understand text and generate images. But those systems operated in short-context, single-turn, or few-step editing regimes. Being a director means maintaining coherent character identities across 10+ shots, ensuring that plot developments in shot 7 are visually consistent with what was established in shot 3, and supporting iterative creative refinement (extension, continuation) — all within a single model. These are not simply "longer context" versions of existing tasks; they require the model to develop internal representations where textual identity and visual identity are the same thing. The director abstraction is what justifies the specific technical choices (interleaved training, ID prompting, disentangled learning) as necessary responses to demands that prior unified models never faced.

This conceptual move also repositions the relationship between UniMAGE and downstream generation models. Rather than competing with video generation systems like Veo 3 or Sora 2, UniMAGE positions itself as a layer above them — a director that produces structured scripts and keyframes that serve as high-level guidance. This is a genuinely different system design philosophy: instead of trying to build one model that does everything (script + images + audio + video), recognize that different creative functions operate at different levels of abstraction, and design a model specifically for the high-level directorial function that current systems lack.

Is this fundamental or incremental? It is fundamental in its reframing of the problem but incremental in its architectural instantiation (it builds directly on Bagel's MoT). The paper does not invent a new model architecture; it invents a new use case and training paradigm for an existing architecture, supported by carefully designed mechanisms (ID prompting, script splitting) that address the specific demands of that use case. This is similar in spirit to how the "foundation model" concept was enabled by existing transformer architectures applied at scale with the right training objectives — the intellectual contribution is in recognizing what the architecture can do if trained appropriately, not in inventing the architecture itself.

Evidence anchoring: The qualitative results in Figures 1, 5, 6, and 7 are the primary evidence for the director abstraction's success. Figure 5 shows UniMAGE maintaining consistent character identities across 6 shots with multiple characters, while StoryDiffusion, Story2Board, and SEED-Story exhibit facial structure drift, clothing changes, and identity confusion. Figure 6 demonstrates coherent long-form narrative progression with maintained visual style over 10+ shots, where baselines show repetitive patterns and fragmented transitions. These are not marginal improvements in a metric; they represent a qualitative shift in what the system can produce — from "plausible individual frames" to "coherent multi-shot stories."

Innovation 2: "First Interleaving, Then Disentangling" as a Training Philosophy for Unified Multimodal Models

The paper's two-stage training paradigm — Interleaved Concept Learning followed by Disentangled Expert Learning — embodies a specific and non-obvious hypothesis about how unified multimodal models should be trained. The hypothesis is: cross-modal integration must be established before task-specific specialization can productively occur. This is not an arbitrary training schedule; it reflects a claim about the nature of multimodal understanding that has implications beyond film generation.

Prior unified multimodal models generally adopted one of two training strategies. Models like Emu and Chameleon trained on interleaved text–image data from the start, using a single autoregressive objective across all modalities — but these were limited to short contexts and suffered from lower image quality. Models like Show-O and Transfusion trained diffusion and autoregressive objectives jointly from initialization — but again in short-context regimes. The field's default assumption was that if you want a model to handle both text and images, you train it on text and images together, and the architecture handles the rest.

UniMAGE's two-stage approach challenges this by recognizing that the data requirements for learning cross-modal associations are different from the data requirements for achieving high-quality text-only or image-only performance. Specifically:

Interleaved multi-shot text–image data is scarce and hard to produce at scale. Appendix B details the laborious process of constructing the 450k sequence dataset: collecting 800k multi-shot video clips, filtering them, annotating them with Gemini 2.5 Pro, extracting keyframes, generating user prompts in four styles, and applying quality filtering with Q-Align. Even after all this, the resulting dataset is modest in size (450k sequences) compared to the datasets available for text-only scripts (250k samples) or single-shot text–image pairs (250k samples).
Text-only and image-only training can achieve higher quality than interleaved training. The disentangled stage allows the understanding expert to learn from pure text scripts — which can be curated for narrative quality, logical coherence, and dialogue without the constraint of needing corresponding visual data. It allows the generation expert to learn from single-shot text–image pairs — which are abundant and high-quality — improving visual fidelity without needing those images to be part of multi-shot consistent sequences.

The key insight is that interleaved training is essential for establishing the shared representation space (the model must learn that "Character 1" in text and "Character 1" in images refer to the same entity), but disentangled training is better for achieving high performance within each modality (the model can learn richer narrative logic from text-only data and better visual quality from image-only data). The "first interleaving, then disentangling" order matters: if you started with disentangled training, the model would never develop the cross-modal associations; if you only did interleaved training, the model would be limited by the quality and quantity of interleaved data.

Why this is not obvious. The natural instinct in multimodal training is to train on the target data distribution — if you want a model that generates interleaved text–image scripts, train it on interleaved text–image scripts. The paper's counterintuitive move is to say: train on the target distribution first to establish cross-modal grounding, then deliberately move away from the target distribution to achieve higher quality in each modality, relying on the shared architecture to preserve the cross-modal associations learned in the first stage. This is reminiscent of curriculum learning and multi-task training strategies, but applied specifically to the problem of unified multimodal generation.

The "detach text tokens" detail as a theoretical commitment. During Disentangled Expert Learning, "the text tokens are detached from the computational graph to prevent gradient propagation." This seemingly minor implementation detail reflects a theoretical position: when optimizing the generation expert, the text representations should serve as fixed conditioning rather than being further optimized. If text gradients were allowed to flow during image training, the understanding expert's representations might drift to better serve the image generation objective at the expense of narrative quality. The stop-gradient enforces a clean separation of concerns within the shared architecture.

Evidence anchoring. The ablation in the bottom of Figure 8 provides direct evidence for the necessity of disentangled learning. The interleaved-only baseline (without Disentangled Expert Learning or Pre-Context Script Splitting) produces scripts with repetitive shot descriptions and weakened narrative flow — the text in the red box shows nearly identical descriptions for consecutive shots. This is consistent with the hypothesis that interleaved-only training constrains narrative learning. The full model, in contrast, generates diverse, logically progressing shots in the continuation examples. Quantitatively, Table 1 doesn't ablate disentangled learning separately (the ablation is "w/o ID-P"), so the evidence for this innovation is primarily qualitative and architectural.

Innovation 3: In-Context ID Prompting as a Mechanism for Identity Binding in Long-Context Generation

In-Context ID Prompting addresses a problem that is easy to state but hard to solve: when a model generates a sequence of images, how does it know that "Character 2" in shot 7 is supposed to be the same person as "Character 2" in shot 3? This is fundamentally a binding problem — associating a symbolic identity (the character label) with a specific set of visual features across time and across changes in pose, lighting, camera angle, and environment.

Prior work on character consistency in image generation took two main approaches. StoryDiffusion introduced consistent self-attention mechanisms that share attention maps across generated images to preserve visual features. Story2Board used lightweight consistency modules that adjust the diffusion process to maintain subject appearance. These approaches operate at the visual feature level — they try to make images look similar by constraining the generation process. They don't explicitly model identity; they model visual similarity.

UniMAGE's ID Prompting takes a fundamentally different approach: it treats identity as a symbolic concept that can be specified in the same token space as text. By inserting <Character1>, <Character2>, <Environment1> tokens directly among the ViT and VAE tokens and applying full attention between these ID tokens and the visual tokens, the model is given an explicit mechanism to bind symbolic identity to visual appearance. This makes identity preservation a language-guided process rather than a purely visual one.

Why this is conceptually significant. The binding problem in multimodal generation is analogous to the coreference resolution problem in natural language — tracking that "she" in sentence 7 refers to the same entity as "Alice" in sentence 1. NLP models solve this through attention mechanisms that learn to associate mentions with entities. UniMAGE extends this idea to the visual domain: the ID tokens function as explicit "coreference markers" that the model can attend to when generating or processing images. This reframes visual consistency from a low-level image processing problem (how do I make this face look like that face?) to a high-level reasoning problem (I am now generating Character 2, who is defined by this description and was previously depicted in these images).

The mechanism is also novel in its integration with the script structure's special token system. The fact that <Character1> appears in three places — in the global description (defining appearance), in the content descriptions (indicating presence in a scene), and in the image tokens (labeling who is in the image) — creates a triangulation that the model can use to learn consistent identity representations. The shared self-attention layers provide the pathway for information to flow between these three occurrences.

Distinction from prompt-based conditioning. Standard text-to-image generation conditions on text prompts, but this conditioning is global and implicit — the model learns to associate words with visual features through training. In-Context ID Prompting adds explicit, localized conditioning within the visual token sequence itself, making the identity binding a structural property of the input representation rather than something that must be learned emergently through the attention mechanism. This is a stronger form of conditioning that is specifically designed for multi-entity, multi-shot scenarios.

Evidence anchoring. The ablation in Figure 8 (top) shows the clearest evidence. Without ID-P, characters exhibit "mixed appearance and inconsistent depictions" — the red circles mark specific instances where facial structure, hairstyle, or clothing changes between shots. Quantitative results in Table 1 confirm the degradation: CSD drops from 59.0 to 52.6, CIDS drops from 59.2 to 55.6, and OCCM drops from 88.07 to 87.00 when ID-P is removed. These metrics specifically measure consistency and identity preservation, making the ablation directly interpretable as evidence for the mechanism's role.

Innovation 4: Script Continuation and Extension as First-Class Capabilities via Data Augmentation

Most text generation systems that produce structured outputs (scripts, stories, plans) are trained to generate complete outputs from start to finish. If you want to extend or revise the output, you typically need to either regenerate from scratch or use the model in a conversational mode with carefully engineered prompts that may or may not produce coherent continuations. The paper identifies this as a fundamental limitation for creative workflows: "this interleaved generation strategy inevitably constrains the model's flexibility in content creation, particularly for tasks such as script extension and continuation."

UniMAGE's Pre-Context Script Splitting elevates continuation and extension from emergent capabilities (things the model might be able to do if prompted carefully) to trained capabilities (things the model is explicitly taught to do through data augmentation). This is a subtle but important distinction. Emergent capabilities are unreliable — they depend on the model generalizing from its training distribution in ways that may or may not work for any given input. Trained capabilities are part of the model's learned behavior — they are reinforced through explicit training examples and are more robust.

What makes this an innovation rather than standard data augmentation. The key is the design of the augmentation to simulate realistic creative workflows. Rather than randomly truncating sequences (a common data augmentation for sequence models), the paper designs two specific augmentation types that mirror how a human user would interact with a script generation system:

Extension inserts a new user prompt mid-narrative, teaching the model to incorporate new creative direction while maintaining continuity with what came before. The use of Qwen 2.5 to generate the new prompt (by summarizing the second half of the script) ensures that the prompt is semantically related to the content that follows, creating a realistic training signal.
Continuation inserts a system prompt before the last shot, teaching the model to naturally continue generating without new user input. This enables the "infinite continuation" capability mentioned in the paper — the model can keep generating additional shots by repeatedly applying the continuation mechanism.

The novelty lies in treating these interaction patterns as integral to the model's design rather than as use cases to be handled by external prompting or fine-tuning. By training on randomly split scripts with inserted prompts, continuation and extension become native capabilities of the model — they are part of what the model is, not just things the model can do.

Connection to the director metaphor. A human director doesn't write a complete script in one pass and then walk away. The creative process is iterative: write some scenes, get feedback, revise, extend, continue. By building extension and continuation into the training, UniMAGE's training paradigm reflects this iterative creative process. The model learns that scripts are living documents that grow and change, not static outputs.

Evidence anchoring. The bottom of Figure 8 shows the ablation: without Pre-Context Script Splitting, the model produces repetitive, near-identical shot descriptions when asked to continue a script (the red box highlights this). With the mechanism, the model generates diverse, logically progressing content that follows from the preceding narrative while introducing new developments. The full generated script examples in Appendix C (the archaeology adventure that extends from discovering an artifact to exploring a hidden chamber to awakening ancient guardians) demonstrate the model's ability to maintain coherent plot progression across extensions and continuations.

5. Experimental Analysis

Evaluation Methodology

Dataset. UniMAGE is trained on a custom-constructed dataset comprising three complementary subsets: (1) 450k multi-shot text–image scripts derived from open-source cinematic content, short films, and documentaries, annotated with Gemini 2.5 Pro; (2) 250k multi-shot text scripts reorganized using Qwen 2.5 to fit the hierarchical script format; and (3) 250k single-shot text–image pairs recaptioned with Gemini 2.5 Pro. The construction process for the interleaved dataset is detailed in Appendix B. For quantitative evaluation, the public benchmark ViStoryBench is used, which assesses story visualization models across narrative structures, visual styles, and character settings. The paper does not specify ViStoryBench's exact size, but the metrics described below are computed on this benchmark's test set.
Base model(s). UniMAGE is initialized from Bagel's pre-trained Mixture-of-Transformers weights, which themselves build on the Qwen 2.5 LLM as the transformer backbone. The visual encoders include SigLIP2-so400m/14 for understanding (ViT) and a FLUX VAE for generation. The choice of Bagel as initialization is deliberate: it provides "a strong foundational capability for unified multimodal understanding and generation" (Section 3.3), specifically for multi-step image editing tasks. The paper argues that this initialization is appropriate but insufficient for long-context script generation, motivating the additional training stages.
Metrics. Six metrics are reported from ViStoryBench:
- Style Similarity (CSDself): Measures consistency of visual style across generated keyframes for the same story. Higher values indicate more consistent visual style.
- Character Identification Similarity (CIDSself): Measures how well character identity is preserved across shots — essentially, whether Character 1 in shot 3 looks like the same person as Character 1 in shot 7. Higher is better.
- Prompt Adherence (Alignment): Measures how closely the generated visual content matches the input narrative prompt. This is the paper's primary metric for narrative-visual alignment.
- Onstage Character Count Matching (OCCM): Measures whether the correct number of characters appear in each generated keyframe according to the script. This captures a specific aspect of script adherence.
- Image Quality (Inception): A standard perceptual quality metric based on Inception network features. Higher indicates sharper, more realistic images.
- Aesthetics: An aesthetic quality score reflecting compositional and artistic merit.
The paper does not describe how these metrics are computed in detail (e.g., which similarity function for CSD, how character identification is verified), which is a limitation. The metrics are simply reported from the ViStoryBench benchmark, with the assumption that the benchmark provides these computations.
Baselines. The paper compares against five systems:
- TheaterGen [1]: An LLM-based character management system for consistent multi-turn image generation.
- Story-Adapter [19]: A training-free iterative framework for long story visualization.
- Story2Board [7]: A training-free approach using lightweight consistency mechanisms for storyboard generation.
- StoryDiffusion [37]: Uses consistent self-attention for long-range image and video generation, achieving character consistency through shared attention maps.
- SEED-Story [34]: A multimodal long story generation model using a unified MLLM — the closest prior work to UniMAGE's unified architecture, though limited to three animation datasets.
For the qualitative comparisons (Figures 5, 6), the textual scripts used as input to StoryDiffusion and Story2Board are generated by UniMAGE itself, meaning the visual differences cannot be attributed to differences in script quality — they isolate the image generation capability.
Generation budget / compute accounting. The paper does not report compute budgets in FLOPs or GPU-hours for any experiment. The training configuration specifies step counts (30,000 for Interleaved Concept Learning, 10,000 for Disentangled Expert Learning) and learning rates (1e-5 for both stages) but provides no wall-clock time, GPU count, or total computational cost. For inference, no latency measurements are reported. This is a significant omission for a systems paper — the reader cannot assess the practical cost of deploying UniMAGE relative to baselines. The qualitative results show output quality but give no indication of inference time, memory requirements, or whether the model can run on consumer hardware.
Cross-validation / statistical protocol. The paper does not describe any cross-validation, statistical significance testing, or confidence intervals. The quantitative results in Table 1 are reported as point estimates without error bars or variance measures. The user study (Figure 9) involved 50 volunteers evaluating 40 generated scripts, with rankings aggregated after "removing invalid responses" (criteria for invalidity not specified), but no inter-rater reliability metric (e.g., Fleiss' kappa) or statistical test for ranking differences is reported. The GSB score of 0.72 for narrative logic is mentioned without defining the GSB metric. This lack of statistical rigor makes it difficult to assess whether the reported differences (e.g., 59.0 vs. 55.6 CSD with and without ID-P) represent reliable improvements or fall within variance.

Main Quantitative Results

Overall Benchmark Performance

Table 1 reports the six ViStoryBench metrics for UniMAGE and all baselines. The headline results:

UniMAGE achieves the best overall performance, particularly on consistency-related and alignment metrics:

CIDS (Character Identification Similarity): UniMAGE scores 59.2, the highest among all methods. The next best is StoryDiffusion at 57.0, followed by Story-Adapter at 56.4. This represents a 3.9% relative improvement over the best baseline. The "UniMAGE w/o ID-P" ablation scores 55.6, confirming that In-Context ID Prompting contributes approximately 3.6 points to this metric — a substantial fraction of the total performance.
OCCM (Onstage Character Count Matching): UniMAGE scores 88.07, again the highest. This metric measures a specific, interpretable capability: when the script says two characters are in a scene, does the generated keyframe contain two characters? The baselines cluster in the 84–86 range, with SEED-Story at 85.9 being the closest competitor. The gap is small in absolute terms (roughly 2 points) but consistent.
Alignment (Prompt Adherence): UniMAGE scores 80.8, achieving what the paper calls "a large improvement" over all baselines. The next best is UniMAGE w/o ID-P at 62.5 — meaning ID Prompting accounts for approximately 18.3 points of alignment performance. Among non-UniMAGE baselines, StoryDiffusion scores 59.7, Story-Adapter 58.9, and Story2Board 56.7. This is the metric where UniMAGE most dramatically outperforms prior work, with a 35% relative improvement over the best non-UniMAGE method. The paper suggests this reflects "substantially better adherence to narrative prompts."
CSD (Style Similarity): UniMAGE scores 59.0, which is not the highest. SEED-Story achieves 74.9 (the best overall) and Story-Adapter achieves 73.7. However, the paper explicitly addresses this: "the notably high CSD score of SEED-Story is mainly due to its overfitting to a narrow set of animation-style datasets, which leads to strong stylistic consistency but poor generalization across broader visual domains." In other words, SEED-Story's high CSD is an artifact of limited diversity — it produces consistently styled outputs because it only knows how to produce one style, not because it has learned generalizable style control. This is an important interpretive caveat: CSD alone does not indicate quality; it must be read alongside other metrics (especially Alignment) and generalization capability.
Image Quality (Inception): UniMAGE scores 12.97, which is mid-range. StoryDiffusion achieves the highest at 15.71, followed by Story2Board at 15.26, TheaterGen at 14.88, and Story-Adapter at 13.73. UniMAGE's score is above only SEED-Story (6.33) and its own ablation without ID-P (12.06). This is a notable weakness: UniMAGE trades off some raw image quality for consistency, alignment, and generalization.
Aesthetics: UniMAGE scores 4.55, also mid-range. StoryDiffusion leads at 5.76, followed by Story2Board at 5.11, TheaterGen at 4.90, and Story-Adapter at 4.89. Again, SEED-Story is substantially lower at 3.84. UniMAGE's aesthetic score, while not best-in-class, represents reasonable visual quality — higher than SEED-Story and close to TheaterGen/Story-Adapter.

The performance pattern is clear: UniMAGE excels at metrics measuring consistency with the narrative (Alignment, OCCM, CIDS) while being competitive but not dominant on metrics measuring standalone visual quality (Inception, Aesthetics, and CSD when that reflects diversity rather than overfitting). This aligns precisely with the paper's claimed contribution — a unified director model that prioritizes narrative coherence and visual consistency over maximizing per-frame visual fidelity.

User Study Results

Figure 9 reports a user study comparing UniMAGE against three baselines (StoryDiffusion, Story2Board, SEED-Story) on generated scripts. 50 volunteers evaluated 40 scripts, ranking outputs on:

Overall Quality: UniMAGE receives the best average ranking (approximately 1.32 on a scale where lower is better, versus roughly 2.55 for StoryDiffusion, 2.48 for Story2Board, and 2.54 for SEED-Story). The numerical values are extracted from the bar chart; exact numbers are not provided in the text.
Plot Alignment: UniMAGE again ranks best (~1.34 versus ~2.59 for StoryDiffusion, ~2.97 for Story2Board, ~2.89 for SEED-Story).
Character Consistency: UniMAGE leads (~1.16 versus ~3.54 for StoryDiffusion, ~3.21 for Story2Board, ~3.41 for SEED-Story). The gap here is particularly large, consistent with the CIDS and OCCM improvements in Table 1.
Narrative Logic (evaluated only against SEED-Story, since it is the only baseline that also generates multimodal scripts): UniMAGE achieves a GSB score of 0.72. The paper does not define GSB, but the scale appears to be such that values above 0.5 favor UniMAGE.

The user study corroborates the automated metrics: human evaluators consistently prefer UniMAGE's outputs across all dimensions. The most decisive preferences are in character consistency and plot alignment, matching the metrics where UniMAGE shows the largest quantitative advantages. The study's limitations — 50 participants, 40 scripts, no reported statistical tests — reduce confidence in the precise magnitude of preferences but the directional consistency across all criteria is unlikely to be noise.

Ablation Studies and Robustness Checks

In-Context ID Prompting (Figure 8 top, Table 1 "UniMAGE w/o ID-P"): Removing ID-P causes substantial degradation across consistency metrics. In Table 1, CSD drops from 59.0 to 52.6 (−6.4 points), CIDS drops from 59.2 to 55.6 (−3.6 points), and OCCM drops from 88.07 to 87.00 (−1.07 points). Interestingly, Alignment improves from 80.8 to 62.5 — wait, that's a decrease, not an improvement. Let me re-read: UniMAGE full scores 80.8 Alignment, "UniMAGE w/o ID-P" scores 62.5. This is an 18.3-point drop — the largest single-metric impact of any ablation. This means ID Prompting is not just about visual consistency; it's also crucial for ensuring the generated images actually depict what the prompt describes. The qualitative evidence in Figure 8 (top) shows specific failure cases: without ID-P, characters exhibit inconsistent appearances across shots (marked with red circles) — hairstyles change, facial structures drift, and clothing details become inconsistent. The mechanism is particularly important after scene transitions, where the model loses track of which visual features belong to which character.

Pre-Context Script Splitting (Figure 8 bottom): Comparing the full model against the interleaved-only baseline (which lacks Disentangled Expert Learning and Pre-Context Script Splitting), the paper demonstrates that without this mechanism, the model produces repetitive content when asked to continue a script. The red box in Figure 8 (bottom left) highlights consecutive shot descriptions that are nearly identical — both describe "the camera remains at a medium shot, steadily showing Character 1 and Character 2 continuing their conversation in Environment 1" with only minor variations. This is classic autoregressive degeneration: without explicit training on continuation from partial context, the model defaults to copying recent patterns. The full model (Figure 8, bottom right) produces diverse, logically progressing content when given <Extension> and <Continuation> prompts — new environments are introduced, plot developments occur, and character actions evolve.

Multi-character script generation (Figure 5 qualitative comparison): While not a formal ablation, Figure 5 demonstrates a critical capability: maintaining multiple distinct character identities across shots. StoryDiffusion and Story2Board, when fed UniMAGE-generated scripts, produce individual frames that are plausible but fail to maintain consistent character appearance — a young woman's hairstyle and facial structure visibly change between frames 1, 3, and 6. SEED-Story produces low-quality, stylistically limited images. UniMAGE maintains stable character appearance across all six frames despite changes in camera angle, lighting, and scene context. This qualitative result directly supports the CIDS and OCCM improvements in Table 1.

Long-form script generation (Figure 6 qualitative comparison): Similarly, Figure 6 evaluates the ability to maintain coherence over extended sequences (9+ shots). StoryDiffusion maintains some character consistency but suffers from "limited scene variation, leading to repetitive visual patterns and copy-and-paste artifacts." Story2Board and SEED-Story exhibit "fragmented transitions and weakening narrative coherence." UniMAGE demonstrates maintained visual style, character appearance, and plot progression across all shots. The archaeology adventure example shows the character's appearance remaining consistent as he transitions from an attic to a jungle to ancient ruins — environments with dramatically different lighting and backgrounds.

SEED-Story generalization comparison (Figure 7): The paper argues that SEED-Story, despite being architecturally unified, fails to generalize because it was "trained on only three animation datasets." Figure 7 directly compares multimodal script outputs: SEED-Story generates outputs with "weak narrative logic and noticeably lower image quality" when prompted beyond its training distribution. The example shows SEED-Story producing cartoon-like, stylized images with poor visual quality for a magic show prompt, while UniMAGE produces photorealistic, compositionally sophisticated frames with coherent narrative structure. This comparison supports the claim that UniMAGE's diverse training data and disentangled learning enable broader generalization than prior unified approaches.

ReST^EM revision model degradation (not applicable — this is from the example paper, not UniMAGE). Note: The paper does not include experiments on revision model training or RL-based optimization. The only optimization attempted is the standard supervised fine-tuning described in the training procedures. There are no experiments investigating alternative training strategies, data scales, or model sizes.

Missing ablations that would strengthen the paper:

Interleaved-only vs. Disentangled-only vs. Both: The paper ablates the full Disentangled Expert Learning stage (with Pre-Context Script Splitting) against interleaved-only in Figure 8, but does not test whether disentangled-only training (without the interleaved stage) could work. This would test the "first interleaving, then disentangling" hypothesis directly. If disentangled-only performed comparably, the interleaved stage would be unnecessary.
Varying the amount of interleaved data: The paper uses all 450k interleaved sequences during Stage 1. How does performance scale with the amount of interleaved data? Is there a point where additional interleaved data provides diminishing returns? This would guide practitioners on data collection priorities.
Single-shot text–image data contribution: The 250k single-shot pairs are used during Stage 2. What is the marginal benefit of this additional data? Does it improve image quality metrics (Inception, Aesthetics) specifically, or does it indirectly help consistency as well?
Script splitting ratio variation: Pre-Context Script Splitting randomly splits scripts. How sensitive is performance to the split position distribution (uniform vs. favoring mid-script splits vs. late splits)? Does the model need to see both extension and continuation examples in roughly equal proportion?
Number of shots during training vs. inference generalization: The paper does not report how many shots the training scripts contain on average, nor test whether the model generalizes to longer scripts than seen during training. This is crucial for the "infinite continuation" claim.
Different base LLMs or MoT initializations: All experiments use Bagel initialized from Qwen 2.5. Would the approach work with other unified multimodal architectures (Show-O, TransFusion)? Is the two-stage training specific to MoT or generalizable?

Critical Assessment

Does UniMAGE actually demonstrate a "unified director" that bridges script and visual generation?

The experiments partially support this claim, with important caveats. The quantitative results in Table 1 show that UniMAGE achieves state-of-the-art performance on consistency and alignment metrics — metrics that directly measure the integration of narrative and visual elements. The 80.8 Alignment score, 18–21 points above all non-UniMAGE baselines, is the strongest quantitative evidence that the unified architecture successfully binds text and image generation. The qualitative results in Figures 5–7 show visually compelling examples of consistent multi-character, multi-shot scripts.

However, the experiments do not demonstrate that UniMAGE's text and image generation jointly optimize for narrative coherence in the way a human director would. The inference procedure (Section 3.5) generates all text first, then all images — this is a sequential pipeline within a single model, not truly joint reasoning. The model cannot, for example, generate an image and then revise the text based on what the image revealed. The "unified director" abstraction is more about shared representations (text and images live in the same latent space) than about interactive, iterative co-creation. The experiments would be stronger if they demonstrated a feedback loop — e.g., generating an image, detecting an inconsistency, and revising the script text accordingly.

Additionally, the paper does not compare against a strong LLM + best diffusion model pipeline with explicit consistency mechanisms. The baselines are all research systems; none represent the best possible separate-agent approach. A system combining a state-of-the-art LLM (e.g., GPT-4 or Claude for script writing) with a state-of-the-art character-consistent image generator (e.g., IP-Adapter with ControlNet for identity preservation) might close the consistency gap. The paper's argument that unified architectures are "essential" would be stronger if it demonstrated that even the best separate-agent system cannot achieve UniMAGE's consistency.

Does the two-stage training paradigm (interleaved then disentangled) actually matter, or could a single stage suffice?

The paper provides only partial evidence for the necessity of two-stage training. Figure 8 shows that the interleaved-only baseline (without Disentangled Expert Learning or Pre-Context Script Splitting) produces repetitive scripts, but this ablation confounds two changes: removing disentangled learning AND removing script splitting. We cannot tell whether the degradation is due to the lack of disentangled expert training or the lack of Pre-Context Script Splitting. A cleaner ablation would test: Interleaved-only + Pre-Context Script Splitting vs. Interleaved + Disentangled Expert Learning + Pre-Context Script Splitting. This would isolate the effect of disentangled training itself.

The paper also does not test the reverse order: Disentangled Expert Learning first, then Interleaved Concept Learning. The theoretical claim is that interleaving must come first to establish cross-modal grounding, but this is untested. If disentangled-first training performed comparably, the "first interleaving, then disentangling" philosophy would be an aesthetic preference rather than a necessity.

Furthermore, the paper does not provide quantitative evidence that the interleaved stage specifically improves cross-modal binding. The ablation in Table 1 removes ID-P (which is introduced during interleaved learning), but this tests ID-P's contribution, not the interleaved stage's contribution. A relevant ablation would compare: full two-stage training vs. a model where the interleaved stage is replaced with additional disentangled training (controlling for total training steps). If the interleaved stage provides unique value beyond what more disentangled training would achieve, the two-stage hypothesis would be supported.

Does UniMAGE "generalize" beyond its training distribution, or is performance limited to seen visual styles and narrative patterns?

The paper claims that UniMAGE "generalizes well across different genres, visual styles, and storytelling formats" (Section 4.1, comparing with SEED-Story), but the evidence for generalization is qualitative only. The diverse generated examples in Appendix C (archaeology adventure, astrophysicist lecture, baker inspection, treasure hunt, magic show, etc.) span different genres and visual styles, suggesting some degree of generalization. However, all examples appear to be from similar cinematic/realistic visual domains — there are no anime, abstract, surrealist, or highly stylized examples that would test generalization to genuinely out-of-distribution visual styles.

More critically, the paper does not evaluate on any out-of-domain benchmark. ViStoryBench is the only quantitative evaluation. We don't know if the Alignment score of 80.8 would hold for, say, scripts in the style of documentary filmmaking, experimental cinema, or animation. The paper positions generalization as an advantage over SEED-Story (which was limited to three animation datasets), but doesn't quantify UniMAGE's generalization breadth.

The dataset construction process (Appendix B) draws from "open-source cinematic content, short films, and documentaries" — a broad but still specific distribution. A model trained primarily on cinematic content may not generalize to user-generated content styles, TikTok-style vertical videos, or abstract artistic prompts. The paper's generalization claims are thus supported for within-distribution diversity (different genres of cinematic content) but not tested for out-of-distribution generalization.

How meaningful are the quantitative improvements, given the lack of statistical rigor?

This is a significant concern. All metrics in Table 1 are reported as scalar values without any measure of uncertainty. The ViStoryBench test set size is not specified. If the test set contains, say, 100 stories, and the metrics have substantial per-sample variance, differences of a few points might not be statistically significant. The paper reports that CIDS improves from 57.0 (StoryDiffusion) to 59.2 (UniMAGE) — a 2.2-point improvement. Without knowing the standard error, we cannot assess whether this is a reliable improvement or could be reversed with a different test sample.

The user study partially addresses this — 50 participants evaluating 40 scripts provides a reasonable sample size, and the consistent preference for UniMAGE across all criteria (Figure 9) is unlikely to be noise. However, the user study has its own limitations: we don't know how the 40 scripts were selected (randomly? to cover diverse scenarios?), whether evaluators had any expertise in film or visual storytelling, or how the evaluation interface controlled for order effects. The GSB metric for narrative logic is mentioned without definition.

The paper would be strengthened by reporting: (1) test set size, (2) confidence intervals or standard deviations for all metrics, (3) statistical tests comparing UniMAGE against the best baseline for each metric, and (4) more detail on the user study methodology.

Do the experiments actually demonstrate "state-of-the-art performance among open-source models"?

The paper claims this in both the abstract and conclusion. The comparison set in Table 1 includes TheaterGen, Story-Adapter, Story2Board, StoryDiffusion, and SEED-Story — all of which are open-source or publicly described systems. There may be other open-source story visualization models not included (the paper does not claim exhaustiveness). More importantly, the paper does not compare against any commercial or closed-source system, which limits the "state-of-the-art" claim to the open-source domain.

The claim is supported for the specific metrics where UniMAGE leads (CIDS, OCCM, Alignment). However, on visual quality metrics (Inception, Aesthetics), UniMAGE is not state-of-the-art even among the tested open-source models — StoryDiffusion achieves substantially higher Inception (15.71 vs. 12.97) and Aesthetics (5.76 vs. 4.55). So the "state-of-the-art" claim must be qualified: UniMAGE achieves state-of-the-art consistency and alignment among open-source models, while trading off some standalone visual quality. This tradeoff is consistent with the paper's goals but should be made explicit.

Missing experiments that would substantially strengthen the paper

Computational cost analysis: The paper provides no training or inference cost measurements. How many GPU-hours does two-stage training require? What is the inference latency for generating a 10-shot illustrated script? How does this compare to the separate-agent baselines (LLM inference + diffusion model inference)? Without this information, practitioners cannot assess whether UniMAGE is practical to deploy or train.

Comparison against GPT-4V + DALL-E or similar: A strong baseline would be a state-of-the-art LLM generating scripts and a state-of-the-art image generation model producing keyframes with explicit identity conditioning (e.g., using reference images for each character). This would test whether the unified architecture provides benefits beyond what careful prompting and conditioning of separate models can achieve.

Scaling behavior: All experiments use one model size (the Bagel-initialized MoT). How does performance scale with model size? Would a smaller model trained with the same data and procedure achieve comparable consistency? Would a larger model close the image quality gap with StoryDiffusion while maintaining the consistency advantage?

Failure case analysis: The paper shows only successful examples. What do UniMAGE's failures look like? Does it ever hallucinate characters, confuse identities, or produce logically incoherent scripts? A systematic failure analysis would help practitioners understand when the system can be trusted and when it requires human oversight. The paper's "Limitations" section mentions that "emotional pacing, stylistic cinematography, and fine-grained control over directorial intent — are not yet fully addressed" but provides no examples of these failures.

Downstream audio-video generation quality: The paper includes two supplementary videos generated by feeding UniMAGE's scripts into Veo 3, but provides no evaluation of the resulting video quality, consistency, or user preference. If the entire motivation is to enable better audio-video generation, the most important evaluation would be: do videos generated from UniMAGE scripts look better (more consistent, more coherent) than videos generated from baseline-generated scripts? This is the ultimate test of the director abstraction but is entirely absent from the quantitative evaluation.

6. Limitations and Trade-offs

The Cost of Difficulty Estimation Is Unaccounted For

The assumption or constraint. The entire compute-optimal scaling framework studied in the example paper rests on estimating prompt difficulty before allocating inference compute — a step the authors acknowledge requires "generating 2048 samples per question and averaging either ground-truth correctness (oracle) or PRM final-answer scores (predicted)," which they describe as "extraordinarily expensive" and note that "our experiments do not account for this cost largely for simplicity." For UniMAGE, this limitation is not applicable — UniMAGE does not use difficulty estimation or adaptive compute allocation — but the paper shares a structurally analogous problem: the cost of dataset construction for the two-stage training paradigm is completely externalized from any reported metric. The paper states that constructing the 450k multi-shot text–image script dataset required "collecting 800k multi-shot video clips, filtering them, annotating them with Gemini 2.5 Pro, extracting keyframes, generating user prompts in four styles, and applying quality filtering with Q-Align" (Appendix B). The 250k multi-shot text scripts were "reorganized and structured using Qwen 2.5." The 250k single-shot text–image pairs were "recaptioned with Gemini 2.5 Pro." This represents a massive computational investment in data preparation — likely dwarfing the training cost itself — but the paper reports zero numbers on API calls, token counts, GPU hours for annotation, or total data preparation cost.

The consequence. A practitioner deciding whether to adopt UniMAGE's training paradigm has no way to estimate the total cost. The headline result — "state-of-the-art performance among open-source models" — is achieved only after a data preparation pipeline that requires access to multiple large commercial models (Gemini 2.5 Pro, Qwen 2.5, Q-Align) and a video collection and filtering infrastructure. If a research group wanted to replicate UniMAGE on a new domain (say, generating scripts for educational content, or for a specific animation style), they would need to replicate this entire data pipeline. The paper provides the annotation prompt template (Appendix B) but no estimate of how many GPU/API hours this requires, what the failure rate of the annotation pipeline is, or how sensitive model performance is to annotation quality. Additionally, the reliance on commercial models for data generation creates a dependency that may not be sustainable — if Gemini 2.5 Pro's API changes or becomes unavailable, the exact dataset cannot be reproduced.

What evidence exists in the paper. None. The paper does not measure or report data preparation costs in any form — no API call counts, no annotation latency, no human verification effort, no filtering pass rates beyond the final count (450k retained from 800k clips for the interleaved subset). The paper does not ablate the effect of annotation quality (e.g., comparing Gemini 2.5 Pro annotations against a smaller/cheaper annotator, or against human annotations). The paper does not test whether the training paradigm works with a smaller, cheaper-to-build dataset.

Mitigation status. Not addressed. The paper provides the annotation prompt and filtering methodology (Appendix B) as a recipe for reproduction but makes no attempt to characterize the cost or to develop cheaper alternatives. A practitioner reading this paper would need to independently estimate the API costs for processing ~1.3M total samples (450k + 250k + 250k + the initial 800k before filtering) through Gemini 2.5 Pro and Qwen 2.5. Given that Gemini 2.5 Pro is a large commercial model, this cost could be substantial.

Generalization Limited to a Single Architectural Backbone and Training Data Distribution

The assumption or constraint. All experiments use a single model architecture: the Mixture-of-Transformers backbone initialized from Bagel's pre-trained weights, which itself is initialized from Qwen 2.5 LLM. The paper does not test whether the two-stage training paradigm (Interleaved Concept Learning followed by Disentangled Expert Learning), In-Context ID Prompting, or Pre-Context Script Splitting would be effective with any other unified multimodal architecture (e.g., Show-O, TransFusion, Chameleon). Additionally, the training data is drawn from "open-source cinematic content, short films, and documentaries" (Appendix B), representing a specific visual and narrative distribution — professionally produced, predominantly live-action or high-quality rendered content with conventional cinematic grammar.

The consequence. There are two distinct generalization risks:

Architectural generalization: The paper's techniques may be specific to the MoT architecture's properties — particularly the shared self-attention layers that enable information flow between the understanding and generation experts. Other unified architectures with different attention schemes or modality fusion mechanisms might not benefit from the same training paradigm. The In-Context ID Prompting mechanism in particular depends on the ability to insert text tokens among visual tokens with full bidirectional attention — if another architecture handles visual tokens differently (e.g., through separate encoder-decoder pathways rather than shared self-attention), this mechanism might not transfer.

Distribution generalization: A model trained on cinematic content may fail when users prompt for styles outside this distribution — anime, abstract art, experimental cinema, TikTok-style vertical videos, documentary footage with different pacing conventions, or user-generated content aesthetics. The paper claims that UniMAGE "generalizes well across different genres, visual styles, and storytelling formats" (Section 4.1), but this claim is made specifically in comparison to SEED-Story (which was limited to three animation datasets) and is supported only by the diverse examples shown in the figures — all of which fall within a cinematic/realistic visual domain. There is no quantitative evaluation on out-of-distribution styles or narrative formats.

What evidence exists in the paper. The diverse examples in Figures 1, 5, 6, 7, and Appendix C span multiple genres (archaeology adventure, romantic drama, magic realism, academic lecture, street performance, historical fiction), but all maintain a consistent cinematic visual style with realistic character rendering, conventional shot compositions, and standard Hollywood narrative structures. There are no examples of abstract, non-narrative, highly stylized, or culturally specific filmmaking traditions. The quantitative evaluation on ViStoryBench is the only distribution-shift test, and the paper does not characterize how ViStoryBench's distribution relates to the training data distribution.

Mitigation status. Not addressed. The paper does not discuss architectural generalization, does not ablate across different backbones, and does not evaluate on explicitly out-of-distribution prompts or benchmarks. The "Limitations" section in the conclusion mentions that "stylistic cinematography" and "fine-grained control over directorial intent" are not fully addressed, acknowledges that the "described technology has not been incorporated into any ByteDance products," and notes that "all figures and scripts presented in this paper are generated by AI models." However, this is framed as future work rather than as a current limitation that might affect practitioners attempting to use UniMAGE for non-cinematic content.

No Training or Inference Cost Measurements

The assumption or constraint. The paper reports no computational cost measurements for any stage of the pipeline. The training configuration specifies step counts (30,000 steps for Interleaved Concept Learning at learning rate 1e-5, 10,000 steps for Disentangled Expert Learning at the same rate) but provides no information about: GPU type and count, total GPU-hours, batch size, throughput (samples/second), memory consumption, or total wall-clock training time. The inference procedure (Section 3.5) describes a three-phase process — generate text script, optionally extend/continue, generate keyframe images for all shots — but provides no latency measurements, no memory requirements, and no comparison of inference cost against baseline methods.

The consequence. A practitioner cannot assess whether UniMAGE is practical to train or deploy. The two-stage training paradigm requires first training on 450k interleaved sequences (Stage 1, 30k steps), then training on the full dataset including additional data (Stage 2, 10k steps). Without GPU-hour estimates, it is impossible to compare this cost against alternatives — for example, against fine-tuning a separate LLM for script writing plus a separate diffusion model for keyframe generation. The inference latency is particularly important because UniMAGE generates keyframes sequentially (conditioned on all previous images), which is inherently slower than parallel generation. If generating a 10-shot illustrated script takes minutes on high-end hardware, this may be impractical for interactive creative workflows. For deployment in a product setting, knowing whether UniMAGE can run on consumer GPUs or requires datacenter hardware is essential.

What evidence exists in the paper. None. The paper mentions using "the standardized training pipeline and parallel strategy with BAGEL" (Section 4, Implementation Details), which gives some indication to readers familiar with Bagel, but Bagel's own compute requirements are not summarized in this paper. The paper does not report model parameter counts for UniMAGE (the MoT architecture includes two transformer experts plus ViT and VAE encoders, but the total parameter count is never stated). Without parameter counts, even rough FLOPs estimates are impossible to derive.

Mitigation status. Not addressed. The paper's supplementary materials include generated scripts and two video demonstrations produced by feeding UniMAGE outputs into Veo 3, but no implementation guide, model cards, or compute reporting. The open-source release mentioned in the abstract (the project page URL) might include model weights and inference code, which would allow practitioners to measure costs themselves, but the paper itself provides no guidance.

Weak Baseline for the Core "Unified Director" Claim

The assumption or constraint. The paper's central claim is that a unified model is "essential" for bridging script writing and keyframe generation, and that separate agent-based approaches are fundamentally limited. However, the baselines against which UniMAGE is compared are not the strongest possible instantiations of the separate-agent paradigm. For the qualitative comparisons (Figures 5, 6), StoryDiffusion and Story2Board are fed scripts generated by UniMAGE itself, meaning the comparison isolates image generation quality but does not test the quality of the scripts produced by separate-agent approaches. For the quantitative comparison (Table 1), the baselines include TheaterGen, Story-Adapter, Story2Board, and StoryDiffusion — all research systems, none representing what a practitioner could achieve by combining a state-of-the-art LLM (e.g., GPT-4, Claude 3.5) with a state-of-the-art character-consistent image generation pipeline (e.g., IP-Adapter with FLUX, or a fine-tuned diffusion model with identity conditioning).

The consequence. The paper's evidence that a unified architecture outperforms separate-agent approaches is incomplete. It demonstrates that UniMAGE outperforms specific research baselines on specific metrics, but it does not demonstrate that the unified architecture provides benefits that cannot be achieved by a well-engineered pipeline of stronger separate models. For instance, a system using GPT-4 to generate structured scripts (with the same special token format) and a fine-tuned diffusion model with reference-image-based identity conditioning might achieve comparable or better consistency and alignment. Without such a comparison, the claim that unification is "essential" remains an architectural hypothesis rather than an empirically validated necessity.

Furthermore, the paper does not compare against a version of itself where the text and image generation are handled by separate models (e.g., an LLM fine-tuned on the same script data for text, plus a diffusion model fine-tuned on the same image data for keyframes). This would be the cleanest test of the unified architecture hypothesis: same training data, same special token schema, same objectives, but separate models versus a single model. The absence of this ablation makes it impossible to attribute UniMAGE's performance specifically to the unified architecture rather than to the carefully designed training data and script structure.

What evidence exists in the paper. Table 1 compares UniMAGE against five open-source baselines. The Alignment score of 80.8 is substantially higher than all baselines (next best: 62.5 for UniMAGE w/o ID-P, 59.7 for StoryDiffusion), which the paper interprets as evidence for the unified approach. However, none of these baselines were trained on the same script-structured data that UniMAGE was trained on. The paper does not ablate: same data, same script structure, but separate LLM + diffusion model versus unified MoT model. The qualitative comparisons in Figures 5-6 show UniMAGE-generated scripts fed to baseline image generators, which tests image consistency given a fixed (UniMAGE-generated) script, but doesn't test whether those baselines could maintain consistency if they also generated the scripts.

Mitigation status. Not addressed. The paper's framing emphasizes the conceptual argument for unification (the "director" metaphor, the inseparability of narrative reasoning and visual imagination) more than the empirical evidence that unification is necessary to achieve the reported results. A practitioner reading this paper might reasonably ask: "Could I achieve similar results by fine-tuning GPT-4 on my script data and using a diffusion model with identity conditioning for keyframes?" The paper provides no evidence to answer this question.

No Downstream Audio-Video Generation Evaluation

The assumption or constraint. The paper's stated motivation is to "bridge user prompts with well-structured scripts, thereby empowering non-experts to produce long-context, multi-shot films by leveraging existing audio–video generation models" (abstract). The entire system is designed to produce intermediate outputs — scripts and keyframes — that serve as "structured, high-level guidance for existing audio-video joint generation models" (Figure 1 caption). However, the paper provides zero quantitative or systematic qualitative evaluation of the final audio-video output quality. The only evidence that UniMAGE's outputs actually improve downstream generation comes from two supplementary video files ("UniMAGE_Demo_1.mp4" and "UniMAGE_Demo_2.mp4"), which are described as generated "using the Veo 3 I2V model, with the scripts, character lines, and sound effect descriptions provided by UniMAGE" (Appendix D).

The consequence. The paper's central value proposition — that UniMAGE enables better film creation — is essentially unevaluated. There are multiple ways in which UniMAGE's scripts and keyframes could fail to improve downstream generation:

The generated scripts might contain instructions that current video generation models cannot reliably follow. For example, if the script specifies a complex camera movement ("the camera slowly pans upward to reveal Character 2 standing on the ship's deck, the wind brushing through his blond hair") that Veo 3 cannot execute faithfully, the resulting video may not match the intended narrative.

The generated keyframes might not actually help the video model maintain character consistency. Video generation models like Veo 3 may not use reference images in the way UniMAGE assumes. If the video model generates each shot independently from the text prompt without strong conditioning on the keyframe, the identity consistency that UniMAGE achieves in static images may not transfer to video.

The end-to-end pipeline might introduce new failure modes. Even if UniMAGE produces coherent scripts and consistent keyframes, the video generation step could introduce temporal inconsistencies, audio-visual misalignment, or character appearance drift that undermine the overall film quality.

What evidence exists in the paper. The two supplementary videos are the only evidence. The paper acknowledges limitations of the downstream generation in Appendix D: "due to the current limitations of audio-video generation models, the timbre and facial features of characters may vary during the generation of long multi-camera videos. These issues warrant further investigation and improvement in future work." This is a significant admission — it suggests that even when UniMAGE produces consistent keyframes, the current state of video generation models cannot fully capitalize on that consistency. No user study evaluates the final video output. No automated metric (e.g., CLIP score between script descriptions and video frames, character consistency metrics across video shots, audio-visual alignment) is reported for the generated videos.

Mitigation status. The paper provides the two demo videos as existence proofs but explicitly defers evaluation of downstream generation quality to future work. A practitioner deciding whether to integrate UniMAGE into a video production pipeline would need to independently evaluate whether the quality improvement in scripts and keyframes translates to quality improvement in final videos. Given the rapid evolution of video generation models, this translation may improve over time, but the paper provides no methodology for assessing it.

The Revision Model Has a 38% Correct-to-Incorrect Reversion Rate

Note: This limitation is NOT applicable to UniMAGE. This limitation comes from the example paper (about compute-optimal test-time scaling) and describes a problem with the revision model in that work. UniMAGE does not use iterative revision or self-correction mechanisms. Instead, it has a different but structurally analogous problem: identity consistency can break down over very long sequences, and there is no automatic mechanism to detect or correct such failures. The paper shows convincing evidence that UniMAGE maintains character consistency better than baselines for scripts of 6–12 shots (Figures 5, 6, 8), but does not characterize how consistency degrades as script length increases, nor does it provide any self-correction or verification mechanism. If a character's appearance drifts imperceptibly over 20+ shots (due to accumulated small generation errors), the system has no way to detect this drift and no mechanism to regenerate inconsistent frames. In the human evaluation (Figure 9), "character consistency" receives the strongest preference for UniMAGE, but this evaluation was conducted on 40 scripts of unspecified length — we don't know whether the advantage holds for very long scripts (50+ shots) or whether performance degrades.

What evidence exists. The paper demonstrates maintained consistency over the specific shot counts shown in the figures (typically 6–12 shots). The OCCM score of 88.07 in Table 1 indicates that ~12% of keyframes have incorrect character counts even in the tested distribution. The paper acknowledges in the Limitations section that "several higher-level dimensions of filmmaking — such as emotional pacing, stylistic cinematography, and fine-grained control over directorial intent — are not yet fully addressed," but this is framed as missing capabilities rather than as degradation of existing capabilities over longer sequences. No experiment systematically varies script length and measures consistency degradation.

Mitigation status. Not addressed. The "infinite continuation" capability enabled by Pre-Context Script Splitting (Section 3.4) implies that scripts can be arbitrarily long, but the paper provides no evidence that consistency holds for scripts longer than those in the qualitative examples. A practitioner generating very long scripts would need to independently monitor for identity drift.

7. Implications and Future Directions

How This Work Changes the Landscape

UniMAGE changes the conversation around AI-assisted filmmaking from an orchestration problem to an architectural unification problem. The dominant paradigm before this work — exemplified by MovieAgent, FilmAgent, FilMaster, and Anim-Director — treated script writing and keyframe generation as tasks for separate models, connected by prompt engineering and post-processing glue code. The implicit assumption was that these tasks are sufficiently different (LLMs for reasoning, diffusion models for generation) that they must be handled by separate systems. UniMAGE demonstrates that a single model, trained with the right multi-stage paradigm, can handle both — and that doing so yields concrete improvements in the metrics that matter most for coherent storytelling: character identity preservation, narrative-visual alignment, and onstage character count accuracy.

This is not a paradigm shift at the level of, say, the Transformer architecture or the diffusion model formulation. UniMAGE does not propose new fundamental model components — it inherits its backbone from Bagel's MoT, its text objective from standard autoregressive language modeling, and its image objective from Rectified Flow. The contribution is architectural reorganization: taking existing building blocks (shared self-attention, dual experts, interleaved token sequences) and training them in a specific sequence (interleaved → disentangled) with carefully designed data augmentation (ID prompting, script splitting) to solve a task that prior systems handled through composition rather than integration.

The magnitude of the shift is best characterized as a reframing with practical consequences. It reframes the AI filmmaking problem from "how do we coordinate multiple AI agents?" to "how do we train a single model to think like a director?" This reframing has immediate practical implications: rather than investing in better agent orchestration frameworks, better inter-agent communication protocols, or more sophisticated prompt engineering templates, research attention shifts toward training data design for long-context multimodal consistency and identity-binding mechanisms within unified architectures. The 18.3-point Alignment improvement in Table 1 (80.8 vs. 62.5 without ID-P, vs. 59.7 for the best non-UniMAGE baseline) provides quantitative evidence that this reframing is more than aesthetic — it produces measurable gains in the specific capability (narrative-visual binding) that separate-agent systems struggle with.

Reconciling conflicting evidence. Prior to UniMAGE, the literature presented a confusing picture about multimodal script generation. On one hand, agent-based systems demonstrated that LLMs could produce structured scripts, and StoryDiffusion showed that consistent self-attention could maintain character appearance across image sequences — suggesting that the pipeline approach was viable. On the other hand, SEED-Story attempted a unified architecture but produced poor generalization and low image quality (Inception 6.33 vs. UniMAGE's 12.97 in Table 1) — suggesting that unification came at a severe quality cost. UniMAGE reconciles these findings by showing that unification can work, but only with specific training design choices: a two-stage paradigm that first establishes cross-modal grounding (interleaved learning) and then allows each modality to specialize (disentangled learning), plus explicit identity-binding mechanisms (In-Context ID Prompting) to prevent the identity drift that plagued prior unified attempts. SEED-Story's failure was not evidence against unification per se, but evidence that naive unification (training on narrow data without identity mechanisms or modality-specific specialization) is insufficient.

Research directions that become more attractive:

Identity-binding mechanisms in multimodal models. UniMAGE's In-Context ID Prompting — inserting symbolic identity tokens among visual tokens with full attention — proves effective for long-context consistency. This opens a research direction that was previously underexplored: how can symbolic reasoning (character identity, object permanence, scene continuity) be explicitly wired into the token representations of unified multimodal models? The success of ID prompting suggests that treating identity as a first-class token-level concept, rather than an emergent property of attention, is a productive approach.
Multi-stage training curricula for multimodal models. The "first interleaving, then disentangling" paradigm is likely applicable beyond film generation — to any domain where cross-modal grounding must be established before modality-specific specialization can occur (e.g., diagram generation with explanatory text, instructional content with illustrations, scientific figure generation with captions).
Structured output formats as training scaffolds. UniMAGE's script structure (global descriptions, per-shot frame and video layers, special tokens for entity reference) is not just an output format — it is a training scaffold. By organizing data into this hierarchical structure, the model learns to reason about narrative at multiple levels of abstraction simultaneously. This suggests that careful output format design can serve as a form of implicit curriculum, guiding the model toward desired reasoning patterns without explicit intermediate supervision.

Research directions that become less attractive:

Agent orchestration for script+image pipelines. If a single unified model can achieve 80.8 Alignment while the best separate-agent baseline achieves 59.7 (a 35% relative improvement), the case for investing in more sophisticated agent coordination frameworks weakens. The bottleneck appears to be the architectural separation itself, not the quality of the coordination protocol. Improvements in prompting strategies for LLM-to-diffusion-model handoffs are unlikely to close a gap of this magnitude.
Training diffusion models for character consistency without symbolic identity conditioning. UniMAGE's approach suggests that visual consistency benefits substantially from explicit symbolic identity information (the ID tokens) rather than purely visual consistency mechanisms (like StoryDiffusion's consistent self-attention). While visual-level approaches still have value (and StoryDiffusion achieves higher Inception and Aesthetics than UniMAGE, per Table 1), the large Alignment gap suggests that symbolic identity grounding is the higher-leverage direction for narrative applications.

Follow-Up Research This Work Enables

Directly testing the necessity of two-stage training with a controlled ablation. The paper's most theoretically interesting claim — that interleaved training must precede disentangled training — is untested. A clean experiment would compare three training orders on the same total compute budget: (A) interleaved → disentangled (UniMAGE's approach), (B) disentangled → interleaved, and (C) a single mixed stage with both objectives and data types interleaved throughout. If (A) substantially outperforms (B), this validates the "cross-modal grounding must come first" hypothesis and would have implications for how all unified multimodal models are trained. If (B) performs comparably, the two-stage paradigm is an implementation preference rather than a theoretical necessity. If (C) performs comparably, the staged approach is unnecessary complexity. The paper's Figure 8 only compares (A) against interleaved-only training, which confounds multiple changes. This experiment would directly test the paper's core training philosophy.

Evaluating generalization to non-cinematic visual domains with quantitative metrics. The paper claims generalization advantages over SEED-Story, but all examples and the ViStoryBench evaluation remain within cinematic/realistic visual domains. A strong follow-up would evaluate UniMAGE — and separately trained baselines — on a deliberately out-of-distribution benchmark covering: (1) highly stylized animation (e.g., anime, stop-motion, pixel art), (2) abstract or non-narrative visual sequences (e.g., artistic video installations, music video aesthetics), and (3) culturally specific filmmaking traditions (e.g., Bollywood musical sequences, Noh theater documentation). For each domain, a test set of ~50 prompts with manual annotation for character consistency and narrative logic would reveal whether UniMAGE's learned representations are genuinely general or are specialized to conventional cinematic grammar. The hypothesis to test: does the unified architecture generalize better than separate-agent baselines when the target domain differs from the training distribution, or does the unified model overfit more severely to its training data's specific visual-narrative conventions?

Scaling model size and measuring consistency-length tradeoffs. The paper uses a single model size (unspecified parameter count, initialized from Bagel). A scaling study would train UniMAGE variants at 3-4 model sizes (e.g., by varying the transformer depth/width) and measure: (1) How does character consistency (CIDS) degrade as a function of script length for each model size? Plot CIDS vs. shot number for scripts of 5, 10, 20, 50, and 100 shots. (2) At what script length does identity drift become noticeable to human evaluators for each model size? (3) Does the "infinite continuation" capability enabled by Pre-Context Script Splitting actually produce coherent output indefinitely, or does it eventually degenerate into repetition or contradiction? This would establish practical usage limits and reveal whether larger models buy proportionally longer coherence windows. The finding that UniMAGE's OCCM is 88.07 (12% of keyframes have incorrect character counts even at tested lengths) suggests that consistency is not perfect even at moderate lengths — a scaling study would reveal whether this is a fundamental limitation or one that larger models could overcome.

Downstream audio-video generation quality with systematic evaluation. The paper's most significant unevaluated claim is that UniMAGE's scripts and keyframes improve final video output. A critical follow-up would generate videos from UniMAGE-produced and baseline-produced scripts using 2-3 state-of-the-art video generation models (Veo 3, Sora 2, Keling 2.5), then conduct a comprehensive evaluation: (1) Automated metrics: Measure character consistency across video shots using face recognition embeddings (e.g., ArcFace similarity for each character across all shots where they appear), CLIP score between script frame descriptions and the corresponding video frames, and video-level aesthetic quality metrics. (2) Human evaluation: A study with 30+ participants comparing videos on overall quality, character consistency, plot comprehension, and visual appeal, using the same scripts processed through different video models to isolate the effect of the script/keyframe quality from the video model's capabilities. (3) Ablation of keyframe utility: Compare video quality when generated from scripts alone vs. scripts + keyframes, to measure the marginal value of UniMAGE's image generation for downstream video coherence. This experiment directly tests whether the "director" abstraction translates to practical benefits or whether current video generation models are insufficiently controllable to capitalize on improved scripts and keyframes.

Training a difficulty-adaptive version of UniMAGE that allocates compute per scene. The paper generates all keyframes with the same compute budget regardless of scene complexity. However, some scenes (crowded environments, complex character interactions, unusual camera angles) likely require more generation steps or more careful conditioning than simple scenes (single character, static shot, familiar environment). A natural extension would train a lightweight "complexity estimator" that takes the frame description as input and predicts how many denoising steps or what classifier-free guidance scale would be needed for reliable generation. During inference, the system could allocate more compute to complex scenes and less to simple ones, potentially improving overall quality at fixed total cost or reducing cost at fixed quality. The training signal could come from the PRM-like approach: generate each keyframe multiple times with varying compute, measure consistency with the script description using the ViT understanding encoder as a learned scoring function, and train the estimator to predict the compute-quality curve. This would directly extend UniMAGE's capabilities while building on the conceptual framework established in the paper.

Reverse-direction evaluation: using generated images to revise scripts. UniMAGE generates all text first, then all images — a sequential pipeline within a single model. A more ambitious test of the "unified director" concept would enable bidirectional interaction: after generating keyframes, feed them back into the model and allow it to revise the script text based on what the images revealed. For example, if the generated keyframe for shot 3 accidentally places Character 2 in the background (visible through a window) when the script says they are in a different location, the model could detect this inconsistency and revise either the image or the text. This would test whether the shared representation space genuinely supports joint reasoning about text and images, or is merely a convenient architecture for sequential generation. The experiment would: (1) generate a complete illustrated script, (2) feed the keyframes back through the ViT encoder with a special prompt token (<ReviseScript>), (3) have the model generate an updated script that maintains consistency with both the user prompt and the generated images, (4) measure whether the revised scripts have fewer internal contradictions than the originals (using an automated consistency checker or human evaluation). This directly tests the bidirectionality of the "director" metaphor in a way the current sequential pipeline does not.

Practical Applications and Downstream Use Cases

Pre-visualization for independent filmmakers and content creators. Independent filmmakers typically cannot afford professional storyboard artists or extensive pre-production. UniMAGE lowers this barrier by transforming a text prompt (e.g., "a young baker's patisserie is subject to a surprise inspection by urban management officers") into a complete illustrated script with consistent characters, scene descriptions, camera directions, and dialogue — all within a single model call. The generated keyframes, while not production-quality (Aesthetics 4.55 vs. StoryDiffusion's 5.76, per Table 1), provide sufficient visual reference for shot planning, location scouting discussions, and communicating creative vision to collaborators. The extension and continuation capabilities enable iterative refinement: a filmmaker can generate an initial script, add new scenes with <Extension> prompts, or continue the narrative indefinitely with <Continuation>. The Alignment score of 80.8 — far above baselines — means the visual output reliably reflects the narrative intent, reducing the "translation gap" between what the filmmaker imagines and what the AI produces. For a filmmaker planning a 10-minute short film (~12-15 scenes), UniMAGE could produce a complete illustrated storyboard in minutes rather than days, with character consistency maintained across all scenes (CIDS 59.2, OCCM 88.07).

Script development and pitching in animation studios. Animation studios invest heavily in script development and visual development as separate phases, often with different teams. UniMAGE enables a single creative lead to rapidly iterate on both narrative and visual ideas: input a premise, receive a structured script with character designs and environment concepts, extend or modify with new prompts, and generate updated keyframes that maintain consistency with the evolving narrative. The In-Context ID Prompting mechanism ensures that Character 1's appearance (defined in the global description) remains stable even as the script undergoes multiple rounds of revision — a significant practical advantage over separate LLM + image generator pipelines where each regeneration of keyframes risks visual drift. The diverse training data (450k multi-shot scripts from "open-source cinematic content, short films, and documentaries") means the system can handle a range of genres and styles. For a studio developing multiple projects simultaneously, UniMAGE could serve as a rapid ideation tool that produces pitch-ready materials (script summary, character designs, key scene illustrations) from a single creative brief, dramatically compressing the concept development timeline.

Educational content generation with narrative structure. Educational content — explainer videos, historical documentaries, science communication — benefits from coherent narrative structure but often lacks the production resources of entertainment media. UniMAGE's ability to generate structured scripts with global descriptions (characters like "an astrophysicist" or "a young biologist," environments like "a modern lecture hall" or "a cluttered research laboratory") and per-shot content with dialogue markers means an educator could input a topic ("an astrophysicist gave a passionate speech, making the lecture hall feel like the vast universe") and receive a complete illustrated lecture script. The <Video> descriptions include specific dialogue and audio cues marked with <- -> indicators, which downstream text-to-speech and sound generation systems can directly parse. The generated keyframes provide visual anchors for each segment, which could be used as slides, video thumbnails, or conditioning for video generation models. While UniMAGE's image quality (Inception 12.97) is lower than dedicated image generation systems (StoryDiffusion's 15.71), the tradeoff favors consistency and alignment — exactly what educational content needs, where accurate depiction of concepts matters more than aesthetic excellence. An educator creating a 10-part video series on historical events could generate consistent illustrated scripts for all episodes in a single session, with character designs and visual style maintained across the entire series.

When to Prefer This Method

The paper explicitly positions UniMAGE against two alternative paradigms: (1) agent-based systems that separate LLM script writing from diffusion model keyframe generation, and (2) prior unified multimodal models (Bagel, Show-O, SEED-Story) that handle short-context tasks but lack mechanisms for long-form consistency. The decision rules below follow directly from the paper's empirical findings and stated limitations.

Prefer UniMAGE over separate-agent pipelines when:

Narrative-visual alignment is the primary quality metric. UniMAGE achieves 80.8 Alignment vs. 59.7 (StoryDiffusion) — a 35% relative improvement. If your application requires that keyframes faithfully depict the script's character identities, scene composition, and entity counts, the unified architecture provides substantial gains over even the best separate-agent baselines.
Scripts involve multiple characters and scenes requiring identity consistency. The paper demonstrates stable multi-character appearance across 6–12 shot sequences where baselines exhibit facial structure drift and clothing changes (Figures 5, 6). The In-Context ID Prompting mechanism specifically targets this capability, and the CIDS (59.2) and OCCM (88.07) metrics quantify the advantage.
Iterative script extension and continuation are core workflow requirements. UniMAGE is explicitly trained (via Pre-Context Script Splitting) to support <Extension> (adding scenes from new prompts) and <Continuation> (indefinitely continuing the narrative). Separate-agent systems typically handle these through prompt engineering, which the paper shows leads to repetitive content (Figure 8, bottom).
You have access to the necessary data pipeline or can invest in building it. UniMAGE's training requires 450k annotated multi-shot text–image scripts, 250k text scripts, and 250k text–image pairs — all constructed using commercial models (Gemini 2.5 Pro, Qwen 2.5). The paper does not report the cost of this data preparation, but it is likely substantial. If you cannot replicate or approximate this data pipeline, the method is not directly adoptable.

Prefer separate-agent pipelines over UniMAGE when:

Standalone image quality (per-frame visual fidelity) is the primary metric. UniMAGE trades off raw visual quality for consistency: Inception 12.97 vs. StoryDiffusion's 15.71, Aesthetics 4.55 vs. 5.76. If each keyframe will be viewed in isolation or judged on artistic merit rather than narrative coherence, dedicated diffusion models with specialized quality optimizations may produce more visually impressive results.
The visual style domain is far from cinematic/realistic content. UniMAGE's training data is drawn from "open-source cinematic content, short films, and documentaries." The paper shows no examples in anime, abstract art, highly stylized animation, or culturally specific visual traditions. Separate-agent systems can leverage style-specific image generation models (e.g., anime-tuned diffusion models) that UniMAGE cannot match without retraining on domain-specific interleaved data.
Latency or compute constraints prevent training or deploying a unified model. The paper provides no cost measurements, but a MoT-based model with two transformer experts plus ViT and VAE encoders is likely larger and slower at inference than a lightweight LLM + efficient diffusion model pipeline. If the deployment requires consumer-grade hardware or sub-second generation, separate models may be the only practical option.
You need bidirectional text-image interaction (images influencing subsequent text). UniMAGE generates all text first, then all images. If your workflow requires that generated images prompt revisions to the script (e.g., "the keyframe revealed an unintended detail, let me adjust the dialogue to match"), the current sequential inference pipeline does not support this. Separate-agent systems can at least support iterative prompting between the LLM and image generator.

Prefer prior unified multimodal models (Bagel, Show-O) over UniMAGE when:

Tasks are short-context (single images, few-step editing). UniMAGE's complexity (two-stage training, ID prompting, script splitting) is designed for long-context narrative generation. For tasks like single-image generation conditioned on text, multi-step image editing, or short multimodal conversations, Bagel or Show-O provide similar or better quality with simpler training and inference procedures.
You cannot construct the interleaved multi-shot dataset required for Stage 1. If your domain lacks the video content needed to build multi-shot text–image scripts (e.g., specialized scientific domains, abstract conceptual content), the Interleaved Concept Learning stage cannot be executed, and the full UniMAGE paradigm cannot be replicated. Prior unified models that train on standard text–image pairs or interleaved web data may be the only feasible option.