StoryMem: Multi-shot Long Video Storytelling with Memory

ArXiv: 2512.19539

🎯 Pitch

Rather than training enormous models on scarce multi-shot data, StoryMem turns a single-shot video generator into a minute-long storyteller by giving it a visual memory bank of past keyframes. This memory-conditioned approach, injected via simple LoRA fine-tuning, outperforms previous state-of-the-art multi-shot methods by over 9% in consistency while matching their visual quality.

---

1. Executive Summary

This paper introduces StoryMem, a paradigm that reformulates long-form video storytelling as iterative shot synthesis conditioned on explicit visual memory, transforming pre-trained single-shot video diffusion models into multi-shot storytellers. Built on the Wan2.2-I2V-A14B model and evaluated on ST-Bench—a newly introduced benchmark of 30 diverse multi-scene story scripts yielding 300 shot-level prompts—StoryMem's core mechanism is the Memory-to-Video (M2V) design (memory latent concatenation with negative RoPE shifts injected via LoRA fine-tuning), coupled with a memory extraction strategy that uses semantic keyframe selection and aesthetic preference filtering to maintain a compact, dynamically updated memory bank of keyframes from historical shots. StoryMem achieves superior cross-shot consistency—outperforming the previous state-of-the-art HoloCine by 9.4% in overall consistency and the pretrained base model by 28.7%—while matching or exceeding baselines on aesthetic quality and prompt adherence, establishing that lightweight memory-conditioned adaptation of single-shot models can enforce long-range narrative coherence without the quadratic training costs of joint multi-shot modeling, though the approach remains limited in complex multi-character scenarios where purely visual memory becomes ambiguous.

2. Context and Motivation

The Core Problem: Moving from Single Clips to Coherent Narratives

The paper addresses a fundamental gap in video generation: while single-shot video diffusion models have achieved near-cinematic visual fidelity (Veo3, Sora2, Kling, Wan2.2), they cannot tell stories. A story is not a sequence of independent high-quality clips — it requires multi-layered coherence across shots and scenes, from low-level consistency of characters and environments to high-level alignment of visual style and narrative flow. The paper frames this gap precisely (Section 1):

"genuine storytelling demands multi-layered coherence across shots and scenes, from low-level consistency of characters and environments to high-level alignment of visual style and narrative flow. Achieving such minute-long, multi-shot narrative generation remains a significant challenge."

This gap matters for several reasons that the paper makes explicit and implicit:

Real-world impact. Visual storytelling is a core expression of human creativity, spanning from cave art to cinema. An AI system that can generate coherent multi-shot narrative videos from text scripts would serve applications in film previsualization, advertising, education, gaming, and personalized content creation. The paper's focus on "minute-long" generation reflects a practical sweet spot — long enough to tell a meaningful story (30 seconds to several minutes), but short enough to be computationally tractable. This is distinct from both ultra-short single clips (5–10 seconds of a single action) and feature-length films (which remain out of reach).

Theoretical significance. The gap reveals a fundamental architectural question: can the capabilities that make single-shot models excellent — their pretrained visual priors, their understanding of motion, their aesthetic quality — be preserved and extended to multi-shot generation, or must we build entirely new models from scratch? The paper's answer — that lightweight memory-conditioned adaptation suffices — has implications beyond video: it suggests that for many sequential generation tasks, the key missing ingredient is not more parameters or more training data, but rather the right conditioning mechanism to propagate state across generation steps.

Conflicting Design Philosophies in Prior Work

The paper identifies two main categories of existing solutions, each embodying a different philosophy, and each falling short in characteristic ways.

Category 1: Joint Multi-Shot Modeling (Train a Single Big Model)

This approach models all shots jointly within a single large video diffusion model. The representative work is LCT (Long Context Tuning; Guo et al., 2025), which pioneered the approach by using full attention with interleaved 3D RoPE to capture cross-shot dependencies across the entire multi-shot sequence. The idea is straightforward: if you want cross-shot coherence, just let every frame attend to every other frame during generation.

Where it falls short. The paper identifies three specific failures:

1. Quadratic computational cost. Full attention across all frames of all shots means the cost scales quadratically with total sequence length. For minute-long videos at high frame rates, this becomes prohibitive for both training and inference. Follow-up methods (Captain Cinema with token compression; HoloCine, MoGa, Mixture of Contexts with sparse attention) improve efficiency, but the fundamental scaling challenge remains.

2. Large-scale retraining on scarce data. These methods require retraining the entire model on multi-shot long video data. High-quality multi-shot narrative videos with consistent characters and coherent storylines are scarce compared to single-shot video data. The paper notes that joint training "often degrade[s] visual quality relative to their pretrained single-shot base models" — a critical practical concern: you sacrifice the single-shot quality that made the base model worth using in the first place.

3. Quality degradation. This is the most damning failure. As seen in Figure 3, HoloCine (the state-of-the-art joint model) "exhibits notable degradation in visual quality" compared to the pretrained base model. The quantitative results in Table 1 confirm this: HoloCine scores 0.5653 on aesthetic quality versus 0.6133 for StoryMem and 0.6452 for the independently-shot base model. Joint training appears to "forget" some of the visual quality learned during single-shot pretraining — a classic catastrophic interference problem.

The paper summarizes this category's dilemma concisely:

"joint training demand heavy computation and scarce high-quality multi-shot data, while decoupled approaches suffer from inconsistency"

Category 2: Keyframe-Based Decoupled Pipelines (Generate Keyframes, Then Animate)

This approach decouples the problem into two stages: (1) generate consistent keyframes (the first frame of each shot) using story image generation or in-context image editing models (StoryDiffusion, IC-LoRA, Gemini 2.5 Flash Image, FLUX.1 Kontext), and (2) expand each keyframe into a video clip using a pretrained image-to-video (I2V) model. The representative works are StoryDiffusion + I2V and IC-LoRA + I2V, which the paper uses as baselines.

The appeal is clear: it leverages the best available models for each subtask. StoryDiffusion uses Consistent Self-Attention to generate character-consistent storyboards. I2V models like Wan2.2-I2V produce high-quality video from a single starting frame. No retraining is needed — just chain existing models.

Where it falls short. The paper identifies a fundamental limitation that the formalization in Section 3.2 makes precise. The decoupled pipeline approximates the joint distribution as:

$p_{\Theta,\psi}(V, K \mid T) \approx p_\psi(K \mid T) \prod_{i=1}^N p_\Theta(v_i \mid t_i, k_i)$

This is the independence assumption: each shot $v_i$ is generated from its keyframe $k_i$ and text $t_i$, with no information flow between shots beyond the keyframes. The paper identifies two consequences:

1. Lack of temporal awareness. "No context propagates across shots, leading to inconsistent visual details, disconnected scene evolution, and rigid transitions." The keyframes capture what the scene should look like at the start of each shot, but not how the narrative evolves. A character's outfit might change between shots because there's no memory of what they wore in the previous shot. A background element (like the street scene in Figure 3's Shot 5) drifts because there's no mechanism to recall what it looked like in Shot 2.

2. Rigid transitions. Since each shot is generated independently from its first frame, there is no mechanism for smooth continuity between the end of one shot and the beginning of the next. The paper notes that "changes such as new character appearances or shifting camera viewpoints cannot be coherently maintained throughout the long video" — precisely the kinds of changes that make a story feel alive rather than like a slideshow.

The quantitative results in Table 1 bear this out: StoryDiffusion + I2V achieves only 0.4600 on overall cross-shot consistency, substantially below StoryMem's 0.5065, despite using the same I2V base model for shot expansion. The user study in Figure 4 shows an even starker preference: StoryMem wins overwhelmingly on cross-shot consistency against both keyframe-based baselines.

The deeper issue. The paper's formalization in Equation 6 reveals that the keyframe-based approach is not just practically limited but theoretically incomplete. The true conditional distribution $p_\Theta(v_i \mid v_{1:i-1}, T)$ — generating shot $i$ given all previous shots — is approximated by $p_\Theta(v_i \mid t_i, k_i)$, which conditions only on the current first frame. This drops all information about the content of previous shots (what happened, what characters looked like in motion, what background details emerged) in favor of a single frozen image. It's like telling a story where each chapter can only see the first sentence of the previous chapter — the coherence is structural but impoverished.

Why Prior Memory Mechanisms Don't Transfer

The paper acknowledges that memory mechanisms exist in video generation (Section 2, "Memory Mechanisms in Video Generation"), but argues they don't solve the storytelling problem. Existing memory approaches in video world models (SlowFast-VGen, Memory Forcing, VMem, WorldMem, Context as Memory) primarily target spatial consistency under controllable world simulations. They rely on auxiliary inputs — camera poses, action sequences, explicit 3D representations — that are available in simulated environments but not in general-purpose video generation from text.

The paper states this limitation directly:

"they primarily target spatial consistency under controllable world simulations and rely on auxiliary control inputs such as actions or camera poses, limiting the applicability to general-purpose video generation."

Furthermore, these methods typically use memory to maintain a consistent environment (the same room looks the same when you turn around), not to maintain consistent characters and narrative elements across semantically distinct shots (a character wearing the same outfit in a different location, in a different pose, under different lighting). The storytelling problem requires a different kind of memory — one that abstracts away from spatial continuity and captures semantic identity.

How StoryMem Positions Itself: A Third Path

The paper positions StoryMem as resolving the core dilemma between the two existing paradigms. It frames this explicitly as "a third path":

"These two lines of research reflect a core dilemma in narrative video generation: joint training demand heavy computation and scarce high-quality multi-shot data, while decoupled approaches suffer from inconsistency. In this work, we explore a third path that achieves both high consistency and high efficiency."

The key insight is that high-quality pretrained single-shot video diffusion models can be effectively adapted for long-term coherent storytelling when augmented with visual memory. This is not a compromise between the two extremes — it's a different axis altogether:

• Joint models ask: how do we train one model on everything?
• Decoupled models ask: how do we chain existing models without retraining?
• StoryMem asks: how do we give a single-shot model memory so it can condition on its own past outputs?

The theoretical framework in Section 3.2 formalizes this position. The paper decomposes the joint distribution autoregressively:

$p_\Theta(V \mid T) = \prod_{i=1}^N p_\Theta(v_i \mid v_{1:i-1}, T)$

This is exact — no approximation yet. The question is how to make the conditioning on $v_{1:i-1}$ computationally tractable. The joint models try to do it through full attention (expensive). The decoupled models approximate it away entirely (lossy). StoryMem introduces a memory $m_i$ as a sufficient statistic:

$p_\Theta(V \mid T) \approx \prod_{i=1}^N p_\Theta(v_i \mid t_i, m_{i-1})$

where $m_i$ is a compact set of keyframes extracted from previous shots. This is an approximation, but a principled one — it retains visual context while discarding temporal redundancy. The paper explicitly draws the analogy to human memory:

"Inspired by human memory, which selectively retains salient visual information... the memory then serves as a global conditioning signal, guiding subsequent generation toward consistent visual semantics and smooth scene evolution"

This framing positions StoryMem not as an engineering trick but as a cognitive-inspired architecture: the model doesn't need to remember every frame of every previous shot, just the key visual impressions — what characters look like, what the key environments are, what stylistic cues define the narrative. The memory bank is the model's working memory, dynamically updated as the story progresses.

Why the Base Model Choice Matters

The paper makes a specific architectural choice that is motivated but not fully justified: building on Wan2.2-I2V-A14B, a 14B-parameter image-to-video diffusion transformer. This choice has implications for what StoryMem can and cannot do.

The Wan2.2-I2V architecture uses a mask-guided conditional diffusion design: an image (the first frame) is encoded, concatenated with zero-filled frames, and fed alongside a binary mask into the DiT. The paper's key insight is that this same architecture can be repurposed: instead of conditioning on a single first-frame image, condition on multiple memory frames from previous shots. The latent concatenation and binary mask mechanism that was designed for I2V becomes the memory injection mechanism for M2V.

This is elegant because it requires no architectural changes — only LoRA fine-tuning. The paper emphasizes this repeatedly:

"StoryMem requires only LoRA fine-tuning on semantically coherent short video clips, achieving strong cross-shot consistency without compromising the high visual quality of pretrained single-shot video diffusion models."

The trade-off, which the paper acknowledges in Appendix C, is that Wan2.2 uses cross-attention-based DiT rather than the more flexible MMDiT architecture (used in FLUX, HunyuanVideo). This means the memory is "purely visual and does not incorporate textual meta information" — the memory update function $m_i = f_\phi(m_{i-1}, v_i)$ cannot include the text prompt $t_i$, creating the ambiguous retrieval problem discussed in the limitations.

The Evaluation Gap That ST-Bench Fills

The paper identifies a meta-problem that motivates the creation of ST-Bench: multi-shot long video storytelling has no standard evaluation benchmark. The most relevant existing benchmark, ViStoryBench, focuses on storyboard image generation, making it unsuitable for video. Without a shared benchmark, comparing methods is impossible — each paper evaluates on its own curated examples, making claims of superiority unverifiable.

ST-Bench is designed to fill this gap with specific properties (detailed in Section 4.2 and Appendix B):

• 30 diverse story scripts covering styles from realistic to fairy-tale, ancient to modern, Western to Eastern aesthetics
• 300 total shot-level prompts (8–12 per story), each describing characters, scenes, dynamic events, shot types, and camera movements
• Structured format with story overviews, scene groupings, per-shot prompts, and scene-cut indicators
• GPT-5-generated following a detailed system prompt that constrains each shot to 1–4 concise sentences and enforces model-friendly descriptions

The paper explicitly states its hope that "this benchmark can facilitate future research in long-form story video generation" — positioning ST-Bench as a contribution in its own right, not just an evaluation tool for StoryMem.

3. Technical Approach

3.1 Reader Orientation

StoryMem is a shot-by-shot video generation system that takes a story script (a sequence of per-shot text descriptions) and produces a multi-shot narrative video — think of it as an AI director that films each shot while remembering what happened in previous shots so characters, scenes, and visual style stay consistent. The core problem it solves is that single-shot video diffusion models generate beautiful individual clips but have no built-in mechanism for cross-shot coherence: if you run the same model independently on each shot of a story, the character's outfit changes, the background drifts, and the narrative feels like a slideshow rather than a film. StoryMem's solution wraps a pretrained single-shot model (Wan2.2-I2V-A14B) in a memory mechanism — at each shot, the model receives not just the current text prompt but also a compact set of keyframes extracted from previous shots, forming a visual memory bank that teaches the model "here's what our characters and world look like." The system then generates each shot conditioned on this memory, updates the memory with the best keyframes from the new shot, and continues. Critically, all of this is achieved with only LoRA fine-tuning (adding ~0.7B trainable parameters to a 14B frozen backbone), meaning the method preserves the pretrained model's visual quality while adding the cross-shot consistency capability that the base model lacks.

3.2 Big-Picture Architecture (Diagram in Words)

The StoryMem pipeline has five major components. Here they are, in the order information flows during inference:

1. Story Script Parser — receives a structured JSON story script containing a story overview, a sequence of per-shot text descriptions, and scene-cut indicators. Its responsibility is to feed the current shot's text prompt to the M2V model at each generation step and to track whether adjacent shots should be smoothly connected (no cut) or start fresh (hard cut).

2. Memory Bank — a dynamically maintained collection of keyframes extracted from previously generated shots. This is the core state that persists across shots. At initialization, it can be empty (default) or seeded with reference images for customized generation (MR2V). After each shot is generated, the memory bank is updated by adding semantically distinct, aesthetically reliable keyframes and evicting old short-term memories when a capacity limit is reached. The memory bank contains two tiers: a fixed "memory sink" of early keyframes (long-term anchors preserving global consistency) and a sliding window of recent keyframes (short-term memory for local dependencies).

3. Memory Extraction Module — takes a newly generated video shot as input and produces a small set of memory keyframes (up to 3 per shot by default). It uses CLIP embeddings for semantic keyframe selection (discarding redundant frames and keeping only visually distinct moments) followed by HPSv3 aesthetic preference filtering (rejecting blurry or low-quality frames that would provide poor conditioning). This module embodies the memory update function $m_i = f_\phi(m_{i-1}, v_i)$.

4. Memory-to-Video (M2V) Model — the core generative component. It is a LoRA-finetuned version of Wan2.2-I2V-A14B that has been trained to generate a video shot conditioned on both a text prompt and a set of memory frames from previous shots. The model takes a text description, the current memory bank, and (optionally) a first-frame image from the previous shot's last frame for smooth transitions, and outputs a new video shot. The architectural mechanism is memory latent concatenation with negative RoPE shifts — the memory frames are VAE-encoded, concatenated with the noisy video latents along the temporal dimension, and fed into the DiT with a binary mask that tells the model which frames are memory (to attend to) and which are to be generated.

5. Scene Transition Handler — optional component for the MI2V extension. When the script specifies no scene cut between two shots, it reuses the last frame of the previous shot as the first frame of the current shot, replacing the M2V model's standard random initialization for that frame. This creates smoother visual continuity across shots, though it does not convey video speed information (a noted limitation).

The system operates iteratively: for shot $i$ in the story, (1) extract memory keyframes from shots $1$ through $i-1$ (already stored in the memory bank); (2) feed the current text prompt $t_i$, the memory bank $m_{i-1}$, and optionally the previous shot's last frame into the M2V model; (3) generate video shot $v_i$ via the diffusion sampling process; (4) pass $v_i$ through the memory extraction module to select new keyframes; (5) update the memory bank to $m_i$ by adding the new keyframes and evicting old ones as needed; (6) if no scene cut is specified for the next shot, store the last frame of $v_i$ for the transition handler. Steps 1–6 repeat for each shot in the script.

3.3 Roadmap for the Deep Dive

• First, the formal problem decomposition (Section 3.2 of the paper): the autoregressive factorization of the joint distribution $p(V|T)$ and the memory-based approximation $p(v_i | t_i, m_{i-1})$. This establishes exactly what StoryMem is trying to compute and why the memory is introduced as a sufficient statistic — it's the theoretical justification for the entire architecture.

• Second, the M2V mechanism (Section 3.3): how memory frames are injected into the video diffusion model via latent concatenation, negative RoPE shifts, and LoRA fine-tuning. This is the "how" of memory-conditioned generation — the concrete architectural and training choices that make the conditioning work.

• Third, the memory extraction and update strategy (Section 3.4): semantic keyframe selection with adaptive CLIP thresholds, aesthetic filtering with HPSv3, and the hybrid memory-sink + sliding-window eviction policy. This is the "what goes into memory and how it's maintained" — the practical engineering that makes long-form generation stable.

• Fourth, the MI2V and MR2V extensions (Section 3.5): smooth shot transitions via first-frame reuse and customized generation via reference-image memory initialization. These show the framework's flexibility beyond the default text-driven setting.

3.4 Detailed, Sentence-Based Technical Breakdown

This is an architectural methods paper whose core idea is that explicit visual memory — a compact, dynamically updated bank of keyframes from historical shots — can transform a pretrained single-shot video diffusion model into a coherent multi-shot storyteller without requiring large-scale retraining or accepting the consistency losses of decoupled pipelines.

---

The Probability Model: Why Memory and What It Must Capture

The paper begins by formalizing the multi-shot generation problem as learning a conditional distribution. The joint probability of generating all video shots $V = \{v_i\}_{i=1}^N$ given text descriptions $T = \{t_i\}_{i=1}^N$ is:

$p_\Theta(V \mid T) = p_\Theta(v_{1:N} \mid t_{1:N})$

where $\Theta$ denotes the parameters of the video diffusion model, $v_i$ is the $i$-th video shot (a sequence of RGB frames), and $t_i$ is the corresponding text description.

What it computes: the joint likelihood of a complete multi-shot video given the full story script. Sampling from this distribution means generating all shots together in a way that respects their textual descriptions and their mutual visual dependencies.

Why this form: this is the most general statement of the problem — it captures all cross-shot dependencies (character consistency, scene coherence, narrative flow) and all text-video alignments. Any practical system must approximate this distribution in some way, and the quality of the approximation determines the quality of cross-shot coherence.

The problem, of course, is that directly sampling from $p_\Theta(v_{1:N} \mid t_{1:N})$ is computationally infeasible — $v_{1:N}$ for a minute-long video at high frame rate is an enormous tensor. The paper's first move is an autoregressive decomposition:

$p_\Theta(V \mid T) = \prod_{i=1}^N p_\Theta(v_i \mid v_{1:i-1}, T)$

where $v_{1:i-1}$ denotes all previously generated shots and $T$ is the full text script (so the model can look ahead or back as needed).

What it computes: the product of per-shot conditional probabilities, where each shot is generated given all previous shots and the complete story. This is an exact decomposition — no approximation has been made yet. It says: generate shot 1 from the text, then generate shot 2 from the text and shot 1, then generate shot 3 from the text and shots 1 and 2, and so on.

Why this form: the autoregressive decomposition aligns with the natural progression of stories (events unfold sequentially) and breaks an intractable joint sampling problem into $N$ tractable conditional sampling problems. If each $p_\Theta(v_i \mid v_{1:i-1}, T)$ can be computed efficiently, the full video can be generated shot by shot.

The problem now is that $v_{1:i-1}$ — all previous video frames — contains massive temporal redundancy. Conditioning on every frame of every previous shot would be computationally expensive and unnecessary (most frames within a shot are nearly identical). The paper's key insight is to introduce a compact memory $m_{i-1}$ that summarizes the relevant visual information from shots $1$ through $i-1$:

$p_\Theta(V \mid T) \approx \prod_{i=1}^N p_\Theta(v_i \mid t_i, m_{i-1})$

where $m_i$ is a small set of keyframes extracted from shots $1$ through $i$.

What it computes: an approximation where each new shot is generated conditioned on (a) its own text description $t_i$ and (b) a compact visual memory $m_{i-1}$ that captures characters, scenes, and stylistic cues from all previous shots. This is the core modeling assumption of StoryMem.

Why this form: it replaces the computationally expensive conditioning on all previous frames with conditioning on a small set of representative keyframes. This is tractable (the memory bank might contain 10–20 frames rather than thousands) while still providing sufficient visual context for cross-shot consistency. The approximation quality depends entirely on whether the memory $m_{i-1}$ is a sufficient statistic — whether it captures all the visual information from past shots that is needed to generate the next shot coherently. The paper's keyframe selection and update strategy (Section 3.4) is engineered to make this approximation as good as possible.

The paper formalizes the memory update as a deterministic function operating on visual information:

$m_i = f_\phi(m_{i-1}, v_i)$

where $f_\phi$ is the memory update mechanism with parameters $\phi$ (the semantic selector and aesthetic filter), $m_{i-1}$ is the previous memory state, and $v_i$ is the newly generated shot.

What it computes: given the existing memory and a new video shot, $f_\phi$ produces an updated memory that includes the most informative new keyframes while potentially evicting redundant or low-quality old ones.

Why this form: the deterministic update keeps the system simple and efficient — no learned memory controller, no complex state tracking. The quality of $f_\phi$ is determined by the heuristics for what makes a "good" keyframe, which the paper operationalizes through semantic distinctiveness and aesthetic quality. A limitation noted in Appendix C is that $f_\phi$ does not include the text prompt $t_i$, meaning the memory update is purely visual and cannot use textual instructions to decide what to remember — this causes the ambiguous retrieval problem discussed in the limitations.

The joint distribution with explicit memory variables is:

$p_{\Theta,\phi}(V, M \mid T) = p_\phi(m_0) \prod_{i=1}^N p_\Theta(v_i \mid t_i, m_{i-1}) \, p_\phi(m_i \mid m_{i-1}, v_i, t_i)$

where $m_0$ is the initial memory (empty by default, or seeded with reference images for MR2V).

What it computes: the full generative model including both video shots and memory states. Each factor has a clear role: $p_\phi(m_0)$ initializes the memory, $p_\Theta(v_i \mid t_i, m_{i-1})$ generates the next shot from text and memory, and $p_\phi(m_i \mid m_{i-1}, v_i, t_i)$ updates the memory — though in practice the text $t_i$ is omitted from the update as noted above, a gap the paper acknowledges.

Why this form: it makes the memory an explicit random variable rather than a hidden state, which means the system's behavior can be inspected and debugged by looking at what keyframes are stored at each step. This transparency is valuable for understanding why consistency succeeds or fails on specific examples.

---

The Video Diffusion Backbone: What StoryMem Builds On

StoryMem is built on Wan2.2-I2V-A14B, a state-of-the-art single-shot image-to-video latent diffusion model with 14 billion active parameters. Understanding the base model's architecture is essential because StoryMem's M2V mechanism repurposes existing components rather than introducing new ones.

The base model operates in a compressed latent space. An input RGB video $v \in \mathbb{R}^{F \times H \times W \times 3}$ (where $F$ is the number of frames, $H$ and $W$ are spatial dimensions) is encoded by a 3D VAE encoder $\mathcal{E}$ into latents $z_0 = \mathcal{E}(v) \in \mathbb{R}^{c \times f \times h \times w}$. The 3D VAE applies temporal compression with stride $s$, so $F = s \times f$ — the latent sequence is shorter than the frame sequence because the VAE compresses temporally as well as spatially.

The diffusion process is formulated under rectified flow, a continuous-time framework. The forward process creates noisy latents $z_t$ by linear interpolation between clean latents $z_0$ and Gaussian noise $\epsilon$:

$z_t = (1 - t)z_0 + t\epsilon, \quad \epsilon \sim \mathcal{N}(0, I), \quad t \in [0, 1]$

where $z_t$ is the noisy latent at time $t$, $z_0$ is the clean latent (to be recovered), and $\epsilon$ is standard Gaussian noise.

What it computes: a simple linear path from clean data ($t = 0$) to pure noise ($t = 1$). At $t = 0$, $z_0$ is the clean latent; at $t = 1$, $z_1$ is pure noise; at intermediate times, $z_t$ is a weighted average.

Why this form: rectified flow (Liu et al., 2022; Lipman et al., 2022) has become the dominant formulation for state-of-the-art video diffusion because it supports deterministic sampling (via ODE integration) and has better theoretical properties for few-step generation compared to DDPM-style stochastic diffusion. The linear interpolation is the simplest possible transport map between the data and noise distributions.

The model learns a velocity field $v_\Theta(z_t, t)$ — a neural network that predicts the instantaneous direction of change at each point along the path. The training objective is:

$\mathcal{L}_{\text{RF}} = \mathbb{E}_{z_0, \epsilon, t} \left[ \| v_\Theta(z_t, t) - (z_0 - \epsilon) \|^2 \right]$

where $v_\Theta(z_t, t)$ is the model's predicted velocity and $(z_0 - \epsilon)$ is the ground-truth velocity (the difference between clean data and the noise that was added).

What it computes: the expected squared L2 distance between the predicted velocity and the true velocity. Minimizing this teaches the network to predict the direction from the current noisy state back toward the clean data — essentially, to "denoise" by following the velocity field.

Why this form: in rectified flow, the velocity $z_0 - \epsilon$ is the exact direction that points from the noisy latent toward the clean latent (since $z_t = (1-t)z_0 + t\epsilon$, the derivative with respect to $t$ is $z_0 - \epsilon$). Predicting this velocity rather than the noise or the clean data is empirically found to work well for transformer-based architectures like DiT.

The velocity prediction network $\Theta$ is a Diffusion Transformer (DiT). Each DiT block contains:

• Self-attention for modeling dependencies between all spatial-temporal tokens within the video (intra-video coherence).
• Cross-attention for incorporating text conditioning — the text prompt is separately encoded by a T5 text encoder, and the DiT's cross-attention layers attend from video tokens to text tokens.
• 3D Rotary Position Embedding (RoPE) that encodes the spatial and temporal coordinates of each token, allowing the model to know where in space and time each patch belongs.

For the I2V variant (which StoryMem builds on), the model additionally conditions on a first-frame image. The conditioning mechanism is latent concatenation with masking. Specifically:

1. The first-frame image is replicated across all temporal positions to form a "static video" of the same length as the target video.
2. This static video is encoded by the 3D VAE into a conditional latent $z_c \in \mathbb{R}^{c \times f \times h \times w}$.
3. A binary mask $M \in \{0, 1\}^{s \times f \times h \times w}$ indicates which frames are preserved (the first frame: 1) and which are to be generated (all others: 0). The temporal stride $s$ accounts for the VAE's temporal compression.
4. The noisy latent $z_t$, conditional latent $z_c$, and mask $M$ are concatenated along the channel dimension and fed into the DiT. The channel dimension becomes $c_{\text{noisy}} + c_{\text{conditional}} + c_{\text{mask}}$ where $c_{\text{mask}}$ corresponds to the mask channels.

Why this design matters for StoryMem: the latent concatenation + masking mechanism was designed for single-first-frame conditioning, but the paper's key insight is that it can be repurposed for multi-frame memory conditioning by simply encoding multiple memory frames (not just one) into the conditional latent and setting the mask bits accordingly. No architectural changes to the DiT are needed — only the input preparation changes.

---

Memory-to-Video (M2V): How Memory Frames Condition Generation

The M2V mechanism is the core technical contribution. It realizes the conditional distribution $p_\Theta(v_i \mid t_i, m_{i-1})$ by extending the I2V conditioning pipeline to accept multiple memory frames instead of a single first frame. The implementation involves four coordinated design choices.

1. Memory Frame Encoding.

The memory bank $m_{i-1}$ contains $f_m$ keyframes (typically 1–10 during training, variable during inference). Each keyframe is a single RGB image — there is no temporal structure among memory frames, they are discrete snapshots from different moments in different previous shots. These memory frames are encoded individually by the same 3D VAE encoder $\mathcal{E}$ used by the base model:

$z_m = \mathcal{E}(\text{memory frames}) \in \mathbb{R}^{c \times f_m \times h \times w}$

What it computes: each memory frame maps to a single latent "frame" in the compressed space — there is no temporal compression across different memory frames because they are not temporally contiguous. The resulting latent $z_m$ has temporal dimension $f_m$ (equal to the number of memory frames).

Why this specific encoding: using the same 3D VAE ensures the memory latents live in the same representation space as the video latents the DiT is trained to process. If a different encoder were used, the DiT would see a distribution shift in its input channels. The fact that the VAE applies 3D convolutions is not a problem — each memory frame is treated as a single-frame "video" during encoding, and the VAE's temporal operations just pass it through.

2. Conditional Latent Construction.

The memory latents $z_m$ are concatenated with latents from zero-filled frames (placeholder frames that will be generated) to form the conditional latent $z_c$:

$z_c = [z_m; \mathbf{0}] \in \mathbb{R}^{c \times (f_m + f) \times h \times w}$

where $[\cdot; \cdot]$ denotes concatenation along the temporal dimension, $\mathbf{0}$ represents the $f$ zero-filled frames for the target video, and $f_m + f$ is the total temporal length of the conditional latent.

What it computes: a tensor that places the memory frames "before" the target video frames in the temporal dimension, creating a single sequence of length $f_m + f$ where the first $f_m$ positions are memory and the last $f$ positions are zero-filled placeholders for the video to be generated.

Why this concatenation: it leverages the DiT's existing self-attention mechanism — memory tokens and target video tokens become part of the same sequence and can attend to each other. The memory frames provide visual context that the DiT can reference when generating the target frames. This is the key mechanism for cross-shot consistency: when generating, say, a character's face in shot 3, the DiT's self-attention can look back at that character's face as it appeared in a memory keyframe from shot 1 and match the appearance.

3. Binary Mask Construction.

A binary mask $M \in \{0, 1\}^{s \times (f_m + f) \times h \times w}$ marks each position in the conditional sequence as preserved or generated. For the memory frames (first $f_m$ in latent space, corresponding to the first $s \times f_m$ frames in pixel space), the mask is 1 — these frames are provided as conditioning and should not be modified by the diffusion process. For the target video frames (remaining $f$ latent positions), the mask is 0 — these are to be generated.

What this mask does operationally: during the diffusion process, the noisy latent $z_t$, the conditional latent $z_c$, and the mask $M$ are concatenated along the channel dimension. The DiT sees the mask as additional input channels and learns to attend to the memory frames (mask = 1) while generating only the masked-out region (mask = 0). After diffusion, the memory portion of the output is discarded and only the newly generated segment (the last $f$ positions) is decoded into the video shot.

Why this mask design: it follows exactly the same protocol as the base I2V model's first-frame conditioning. The DiT has already learned, during its original I2V pretraining, to interpret a binary mask that tells it which frames are "given" and which are "to generate." By extending the mask to handle multiple given frames (memory) instead of just one (first frame), StoryMem inherits this learned behavior without requiring the DiT to learn a new conditioning interface.

4. Negative RoPE Shift.

This is perhaps the most subtle design choice. The DiT uses 3D Rotary Position Embedding (RoPE) to encode the spatial and temporal coordinates of each token. In the base I2V model, the first frame is at temporal index 0 and the generated frames are at indices 1, 2, ..., $f - 1$. The temporal distance between tokens (computed via the dot product in the RoPE attention mechanism) encodes how far apart in time two positions are.

For StoryMem, memory frames are not temporally continuous with the current shot — they are discrete keyframes from different moments in the past, potentially from different scenes entirely. If they were simply placed at temporal indices 0, 1, ..., $f_m - 1$ followed by the target video at $f_m, f_m + 1, ..., f_m + f - 1$, the DiT would interpret them as a continuous sequence: "memory frame 1 and memory frame 2 are adjacent in time," which is false — they might be from different shots seconds apart.

The paper's solution is to assign negative temporal indices to memory frames:

$\text{memory indices} = \{-f_m S, -(f_m-1)S, \ldots, -S\}$

$\text{target video indices} = \{0, 1, \ldots, f - 1\}$

where $S = 5$ is a fixed offset (the "negative RoPE shift offset") that controls the temporal gap between the memory frames and the target video.

What this computes: memory frames are placed at negative positions (e.g., -15, -10, -5 for $f_m = 3, S = 5$) and the target video starts at position 0. The temporal gap between any memory frame and the start of the target video is at least $S$, which encodes "these events happened before the current shot, not during it."

Why negative indices: RoPE attention computes similarity based on the difference between positional indices. By placing memory at negative indices, the model learns that memory frames are "prior context" rather than "temporal neighbors." The target video maintains its original numbering starting from 0, which is important because the DiT was pretrained with that numbering and has learned specific positional patterns (e.g., "frame 5 should show motion consistent with frame 4 and leading to frame 6"). Changing the target video's indices would disrupt these learned patterns. The gap $S$ prevents the model from treating memory frame $-S$ (just before position 0) as temporally adjacent to target frame 0 — there's a deliberate "temporal step" between memory and generation.

Why this specific offset ($S = 5$): this is the value used during training (Section 4.1). The choice is empirical — large enough to create a clear separation between memory and target, small enough that the RoPE's frequency components can still capture the relationship. The paper does not ablate this value, so its sensitivity is unknown. Too small an offset would cause the model to treat memory frames as immediately preceding the target (encouraging motion continuity where none exists), and too large would make the positional encoding of memory frames indistinguishable from random (losing the benefit of structured relative positioning).

Putting it all together: the M2V forward pass.

During training and inference, the diffusion process operates as follows:

1. Memory frames are VAE-encoded to $z_m$.
2. The conditional latent $z_c = [z_m; \mathbf{0}]$ is constructed.
3. The mask $M$ marks memory as 1 and target as 0.
4. At each denoising step, the noisy latent $z_t$ (which covers only the target positions — the memory positions are not noised), the conditional latent $z_c$, and the mask $M$ are concatenated channel-wise.
5. The RoPE indices are set: negative for memory positions, starting from 0 for target positions.
6. The DiT predicts the velocity $v_\Theta(z_t, z_c, M, t, t_{\text{text}})$, which includes cross-attention to the text prompt $t_i$.
7. The velocity is used to update $z_t$ via the ODE solver.
8. After all denoising steps, the memory portion of the latent is discarded and the target portion is VAE-decoded to the output video shot $v_i$.

Training data curation. Training the M2V model requires pairs of (memory frames, target video) where the memory frames are visually related to the target — otherwise the model would learn to ignore the memory conditioning. The paper curates training data from two sources:

• Visually related short clips from cinematic single-shot video datasets. Clips are grouped by shot-level visual similarity (presumably using CLIP embeddings, though the exact grouping method is not specified). Within each group, one clip is designated as the target, and frames from other clips in the group serve as memory. This teaches the model to transfer visual style and character appearance between different clips that share visual semantics.

• High-quality single-subject multi-scene short videos. These are videos that naturally exhibit consistent identity across different scenes — the same person in different locations, the same object in different contexts. These provide "ground truth" memory-video pairs where the memory genuinely depicts the same subject as the target.

Training procedure. During training, the memory length $f_m$ is randomly sampled from 1 to 10 frames. The target is always a complete video clip. The model is trained with the standard rectified flow loss (Equation 3), where the velocity prediction is conditioned on the memory, mask, and text. Only the target video portion contributes to the loss — the memory portion is masked out.

LoRA fine-tuning. A critical design choice is that only LoRA adapters are trained, not the full model. The paper uses rank-128 LoRA applied to all linear layers in the DiT blocks, adding approximately 0.7 billion active parameters on top of the 14B-parameter frozen backbone. This is motivated explicitly:

"StoryMem requires only LoRA fine-tuning on semantically coherent short video clips, achieving strong cross-shot consistency without compromising the high visual quality of pretrained single-shot video diffusion models."

Why LoRA over full fine-tuning: (1) Preservation of visual quality — the base Wan2.2 model has been trained on massive data to produce high-quality single-shot videos; full fine-tuning on a much smaller curated dataset (400K videos) would risk catastrophic forgetting of these pretrained visual priors. LoRA constrains the adaptation to a low-rank subspace, preserving the base model's capabilities. (2) Efficiency — 0.7B trainable parameters versus 14B is a substantial reduction in memory and compute, enabling training on more modest hardware. (3) Data efficiency — the curated dataset (400K clips) is orders of magnitude smaller than the base model's pretraining data; LoRA's parameter efficiency helps prevent overfitting. (4) Modularity — the LoRA weights are a small adapter that can be swapped or combined, making the approach compatible with other adaptations of the base model.

Why rank 128: the paper does not ablate the LoRA rank, so the choice is presumably based on prior work conventions or empirical sweeps not reported. Higher rank allows more expressive adaptation at the cost of more parameters; 128 is a standard high-rank choice for large-scale adapters (many works use rank 8–64 for smaller models).

---

Memory Extraction and Update: From Generated Shot to Memory Bank

The M2V model generates a video shot $v_i$ conditioned on the memory $m_{i-1}$. The next step is to update the memory — to extract informative keyframes from the newly generated shot and incorporate them into the memory bank. This is the memory update function $m_i = f_\phi(m_{i-1}, v_i)$, which has two sequential stages.

Stage 1: Semantic Keyframe Selection.

The goal is to select a small set of keyframes from $v_i$ that are semantically distinct — each keyframe should capture a different visual moment, avoiding redundant frames that show essentially the same content. The procedure works as follows:

For all frames in the generated shot, CLIP embeddings are computed. CLIP (Radford et al., 2021) is a vision-language model trained to align image and text representations; its image encoder produces a normalized embedding vector that captures high-level semantic content (objects, scenes, people, actions) while being relatively invariant to low-level details like exact pixel values or minor pose changes.

Keyframes are selected sequentially, starting with the first frame (which is always included as a keyframe, since it anchors the shot's initial state). For each subsequent frame $j$, its CLIP embedding is compared with the CLIP embedding of the most recently selected keyframe using cosine similarity:

$\text{sim}(f_j, k_{\text{last}}) = \frac{\text{CLIP}(f_j) \cdot \text{CLIP}(k_{\text{last}})}{\|\text{CLIP}(f_j)\| \|\text{CLIP}(k_{\text{last}})\|}$

where $f_j$ is the current frame, $k_{\text{last}}$ is the last selected keyframe, and $\cdot$ denotes dot product.

What it computes: the cosine similarity between the semantic embeddings of the current frame and the last selected keyframe, yielding a value in $[-1, 1]$ where 1 means the frames are semantically identical and -1 means they are maximally dissimilar.

A new keyframe is selected when the similarity drops below a dynamic threshold $\tau$. The threshold starts at a low value (0.9 by default) and increases if the number of selected frames exceeds a preset upper bound (3 per shot by default). The threshold adjustment works as follows: if after processing all frames, the number of selected keyframes is greater than the upper bound, the threshold is raised by a small amount and the selection process is repeated with the stricter criterion until the number of keyframes is within the bound.

What the dynamic threshold achieves: it adapts to the shot's content. For a mostly static shot (e.g., a character talking without moving), most frames will have high CLIP similarity to the first frame, so only a few (or just one) keyframe will be selected even with a low threshold. For a highly dynamic shot (e.g., an action sequence with rapid scene changes), many frames may pass the initial low threshold, but the adaptive increase ensures the final set is capped at the upper bound while still selecting the most distinct frames (since the threshold is raised uniformly, the frames that survive are the ones with the largest semantic gaps from their predecessors).

Why CLIP embeddings: they provide a semantic similarity metric that correlates with human perception of visual difference — a character turning their head slightly won't trigger a new keyframe, but a character walking into a new room will. This is precisely what is needed for memory: the memory should remember distinct visual states, not every minor motion variation. Alternative choices like pixel-level MSE would be too sensitive to motion and lighting changes, while feature-space metrics from the VAE might be too entangled with the VAE's compression artifacts.

Why sequentially comparing to the last keyframe rather than all previous keyframes: this is an approximation that makes the selection process linear in the number of frames (one comparison per frame) rather than quadratic (compare each frame to all previously selected keyframes). The trade-off is that it might miss a frame that is similar to the first keyframe but distinct from the most recent one, but in practice, since keyframes are selected sequentially along the video timeline, the last keyframe is usually the most relevant comparison point.

Stage 2: Aesthetic Preference Filtering.

Semantic selection identifies distinct content but does not guarantee image quality. A frame with large motion blur, video compression artifacts, or poor composition may be semantically distinct (because it shows a key moment) but visually unsuitable as memory — it would provide poor conditioning for future shots. The paper uses HPSv3 (Human Preference Score v3; Ma et al., 2025) as an aesthetic reward model:

For each candidate keyframe selected by the semantic stage, HPSv3 computes an aesthetic score $a \in \mathbb{R}$ that estimates human preference for the image's visual quality. Frames with scores below a threshold of 3.0 are rejected.

What HPSv3 captures: it is trained on human preference judgments to predict how much a human would "like" an image in terms of visual appeal — factors include sharpness, composition, color harmony, lighting quality, and overall aesthetic pleasantness. It is not evaluating semantic content but rather image quality and visual appeal.

Why filtering rather than incorporating the aesthetic score into the selection process directly: semantic distinctiveness and aesthetic quality are independent objectives; a frame that is blurry and unappealing but semantically unique might still be selected if aesthetics are just a soft weighting factor. Hard filtering at a threshold ensures that memory frames are always of minimum acceptable quality. The threshold of 3.0 is not ablated in the paper — it represents a choice about the trade-off between including more frames (lower threshold) and higher average quality (higher threshold).

Interaction between stages: the two stages operate in sequence — first semantic selection produces a candidate set, then aesthetic filtering removes low-quality candidates. If filtering removes too many frames (e.g., a shot with uniformly poor quality), the semantic selection stage will have already included the first frame (which is always selected), ensuring at least one keyframe per shot is retained as memory.

Memory Bank Update and Eviction.

After keyframes are extracted from the new shot, they are compared with the existing keyframes in the memory bank using CLIP cosine similarity. Any new keyframe that is too similar to an existing memory frame (above a similarity threshold, though the exact threshold is not specified in the paper) is considered redundant and discarded. Only semantically distinct new keyframes are added to the memory bank.

To prevent uncontrolled growth of the memory bank (which would make inference increasingly expensive as the story progresses), the paper uses a hybrid memory-sink + sliding-window strategy:

• Memory sink: a fixed number of early keyframes (3 by default) are designated as permanent "long-term anchors." These are never evicted and serve as the model's persistent reference for global consistency — they typically include the first appearance of key characters and the primary setting.

• Sliding window: the remaining memory slots form a short-term window that captures recent visual context. When the memory bank reaches its capacity limit (not explicitly specified in the paper, but implied by the memory sink size and per-shot keyframe limit), the oldest short-term memories are evicted on a first-in-first-out basis.

Why this dual structure: it parallels models of human memory — long-term memory for stable identity (what does the protagonist look like?), short-term memory for recent events (what just happened in the previous scene?). The memory sink ensures that even in a long story, the model never "forgets" the core characters and setting, while the sliding window ensures that the memory stays compact and focused on recent context (which is most relevant for immediate continuity). Without the memory sink, a purely sliding-window approach would eventually lose all early keyframes, causing the model to "drift" in its representation of established characters over many shots. The ablation study in Table 2 confirms this: removing the memory sink (replacing it with a pure sliding window) reduces overall cross-shot consistency from 0.5065 to 0.4891.

Optional human/LVM refinement. The paper notes that human creators or large vision-language models can review and refine the selected keyframes for finer story-specific control, but this option is not used in the reported results. This is an escape hatch for applications where perfect memory is critical (e.g., advertising, film previsualization), but the core contribution is the automated pipeline.

---

MI2V Extension: Smooth Shot Transitions

The default M2V system generates each shot independently — even though the model conditions on memory for visual consistency, the actual video frames at shot boundaries are independent. When concatenated, this can create rigid or jarring transitions because the last frame of shot $i$ and the first frame of shot $i+1$ are unrelated (generated independently by different diffusion runs). The MI2V extension addresses this for cases where the story script specifies no scene cut between adjacent shots — that is, when shot $i+1$ should flow naturally from shot $i$.

The mechanism is straightforward: when the script indicates no cut, the last frame of shot $i$ is reused as the first frame of shot $i+1$. This is implemented by replacing the M2V model's standard first-frame initialization (which would normally be zero-filled or randomly initialized) with the VAE-encoded last frame from the previous shot. The binary mask for shot $i+1$ marks this first frame as preserved (mask = 1), and the diffusion process generates only the remaining frames conditioned on this preserved first frame, the text prompt, and the memory.

What this achieves: the first frame of shot $i+1$ is pixel-identical to the last frame of shot $i$, so when the two shots are concatenated, there is a perfect visual join — no discontinuity in character position, background, or lighting.

Why only the first frame: the I2V architecture conditions on a single first frame and generates the rest — extending this to condition on multiple boundary frames would require architectural changes. The single-frame connection is a pragmatic choice that works well for static or slow-moving boundaries but creates issues when the two shots have very different motion speeds (a noted limitation in Appendix C — "the single-frame connection mechanism does not convey video speed information").

The scene-cut indicator is part of the ST-Bench story script format. Each shot prompt is accompanied by a boolean "cut" field: True means a hard scene transition (the next shot starts fresh, no reused first frame), False means a smooth continuation (reuse the previous shot's last frame). This gives the script creator (human or LLM) control over pacing: rapid scene changes versus flowing sequences.

---

MR2V Extension: Customized Story Generation with Reference Images

The memory formulation in Equation 7 includes an initial memory $m_0$, which is by default empty (the first shot is generated from text alone, without memory conditioning). The MR2V extension allows users to seed the initial memory with reference images — for example, providing photographs of a person or images of a specific location that should appear throughout the story.

The mechanism reuses the same M2V model without any modification. At the start of generation, instead of an empty memory bank, $m_0$ is initialized with the user-provided reference images (VAE-encoded and placed into the memory bank with mask = 1, exactly like memory keyframes extracted from previous shots during normal operation). The first shot is then generated with $p_\Theta(v_1 \mid t_1, m_0)$ — conditioning on both the text and the reference images. Subsequent shots update the memory as usual, with extracted keyframes being added alongside or replacing the initial references.

What this enables: customized storytelling where specific characters, backgrounds, or stylistic reference images are "injected" into the generation process from the first shot onward. The paper's Figure 5 demonstrates this with reference images of real people (used with consent) appearing consistently throughout generated story videos.

Why this works without additional training: the M2V model was trained to condition on memory frames from previous shots — reference images at $m_0$ are simply memory frames from a "virtual" previous state. The model doesn't need to know whether the memory frames came from previous generation or from user uploads; the conditioning mechanism is identical.

Compatibility with specialized reference techniques: the paper notes that StoryMem is "compatible with other specialized reference-preserving techniques" — meaning that dedicated identity-preservation methods (e.g., IP-Adapter, InstantID) could be used to strengthen the reference conditioning, but this is not implemented or evaluated in the paper. The current MR2V relies solely on the M2V's built-in memory attention to maintain reference identity.

---

Summary of Design Choices and Their Justifications

• Memory as keyframes rather than latent vectors or text descriptions: keyframes are visual, so the DiT's self-attention can directly compare pixel-level details (facial features, clothing textures) between memory and generated frames — this is more precise than text descriptions and more interpretable than learned latent memory vectors. Keyframes are also already supported by the VAE and DiT architecture without modification.

• Latent concatenation + masking over cross-attention memory injection: the paper inherits Wan2.2-I2V's mask-conditioned design rather than introducing a separate cross-attention pathway for memory. This means the memory frames participate in the DiT's full self-attention (every memory token can attend to every target token and vice versa), which is more expressive than a dedicated cross-attention module that only allows target tokens to attend to memory tokens.

• Negative RoPE shift over zero-indexed or continuous positioning: placing memory at negative indices preserves the target video's original positional encoding (starting at 0, which the model was pretrained with) while clearly separating memory temporally. Alternative approaches like placing memory at indices 0, 1, ... would disrupt the target's numbering, and placing memory and target with a gap at positive indices (e.g., memory at 0-2, target at 10-30) would teach the model an unrealistic temporal distance.

• Sequential semantic selection with dynamic threshold over clustering or uniform sampling: uniform sampling would miss key moments (a 5-second shot might have one important change at frame 30); clustering (e.g., k-means on CLIP embeddings) would select representative frames but is computationally more expensive and harder to tune (choosing k). The sequential approach is simple, online (works frame-by-frame as the video is decoded), and the dynamic threshold adapts to the shot's content.

• Aesthetic filtering as a hard threshold rather than a soft weighting: hard filtering ensures that low-quality frames never enter the memory bank, which is important because a single blurry memory frame can degrade the conditioning for all subsequent shots. Soft weighting might still include a very blurry frame if it is extremely semantically unique, but such a frame would provide poor conditioning regardless of its uniqueness.

• Memory sink (fixed long-term anchors) over pure sliding window: without long-term anchors, early-appearing characters and settings would eventually be evicted and the model would "forget" what they look like. The ablation confirms this matters for consistency (Table 2). The number 3 for sink size matches the per-shot keyframe limit, meaning roughly one shot's worth of keyframes is permanently retained — a reasonable default that balances persistence with memory budget.

• LoRA over full fine-tuning: the 0.7B trainable parameters versus 14B frozen parameters preserves pretrained single-shot quality while adapting for memory conditioning. The paper's quantitative results support this — StoryMem achieves 0.6133 on aesthetic quality versus 0.6452 for the base model's independent generation (only a 5% degradation) and substantially better than HoloCine's 0.5653 (a 13% degradation from full fine-tuning). LoRA achieves the adaptation goal with minimal quality sacrifice.

• Training on semantically coherent short clips rather than multi-shot long videos: this is the key enabler of data efficiency. The paper trains on 400K single-shot videos grouped by visual similarity — far more abundant than multi-shot narrative videos. The M2V model learns to transfer visual context between different clips (which is what it does at inference time — transferring context between different shots), rather than learning to continue a single long video (which would be a different task). This training strategy directly matches the inference task.

4. Key Insights and Innovations

Innovation 1: Memory as a First-Class Abstraction for Multi-Shot Generation — Not a Workaround for Computational Limits

The paper's most fundamental conceptual move is recasting cross-shot consistency as a memory problem rather than an attention problem. This reframes the entire debate between joint and decoupled approaches.

Prior work operated under an implicit assumption: cross-shot coherence requires the model to "see" all past shots simultaneously. Joint models like LCT achieve this via full attention across all frames — all tokens can attend to all other tokens, so nothing is lost. Decoupled keyframe-based approaches like StoryDiffusion + I2V accept an impoverished version where only the current first frame is visible. Both share the same premise: coherence is access to raw visual history. They differ only in how much access they can afford (all vs. almost none).

StoryMem rejects this premise entirely. It argues that raw access is neither necessary nor sufficient — what matters is curated, compact memory. The parallel to human cognition is not decorative rhetoric but the core architectural insight: humans don't remember every frame of every experience, yet we maintain stable representations of characters and settings over hours of narrative. We extract and store key visual impressions and discard the rest. The paper operationalizes this as an explicit memory bank that is:

• Compact (3 memory-sink anchors + a sliding window of recent keyframes — perhaps 10–20 frames total for a minute-long video with hundreds of generated frames).
• Selective (semantic keyframe selection discards redundant frames; aesthetic filtering rejects low-quality ones).
• Dynamically updated (new keyframes are added; old short-term memories are evicted).
• Directly visual (keyframes are images, not learned latent vectors or text summaries — the DiT can attend to pixel-level details like facial features and clothing textures).

This is not an incremental engineering improvement over existing methods. It is a fundamentally different problem decomposition. Equation 9 formalizes the shift: $p(v_i \mid t_i, m_{i-1})$ conditions on a summary $m_{i-1}$, not on the raw history $v_{1:i-1}$. The quality of this approximation depends on whether $m_{i-1}$ is a sufficient statistic — whether it captures all the visual information needed for coherent continuation. The paper's entire memory extraction and update pipeline (Section 3.4) is designed to make this approximation as tight as possible.

The significance extends beyond the specific implementation. This framing opens a new axis of research: memory design for generative models. What should be remembered? How should it be encoded? When should it be updated or forgotten? These are questions the joint modeling paradigm never had to ask (because it remembers everything at quadratic cost) and the decoupled paradigm couldn't ask (because it remembers nothing beyond keyframes). StoryMem shows that answering these questions well can achieve better coherence than a joint model (9.4% improvement over HoloCine in overall cross-shot consistency, Table 1) at a fraction of the computational cost — and this gap is likely to widen as memory design improves.

---

Innovation 2: Negative RoPE Shift as a Zero-Architecture-Change Mechanism for Temporal Discontinuity

The negative RoPE shift is easy to dismiss as a minor positional encoding trick, but it embodies a distinctive design philosophy that pervades the paper: repurpose existing architectural interfaces rather than introduce new ones.

The problem is genuine: memory frames and target video frames coexist in the same DiT sequence, but they are not temporally contiguous. If placed at consecutive indices (0, 1, 2 for memory; 3, 4, 5 for target), the 3D RoPE would encode them as a continuous timeline — implying a smooth motion path from memory frame 2 to target frame 0. This would teach the model to generate transitions where none exist, corrupting both the memory conditioning (by making it about temporal prediction rather than identity reference) and the target video quality (by anchoring generation to an incorrect prior state).

Prior work on incorporating external context into diffusion transformers has used various mechanisms: dedicated cross-attention layers (adding architectural complexity), learned adapter modules (IP-Adapter style), or separate context encoders. The dominant assumption has been that temporal discontinuity requires explicit architectural handling — you can't just shove non-contiguous frames into the same sequence and hope the model figures it out.

StoryMem's counterintuitive move is to let the positional encoding itself encode the discontinuity. By assigning negative indices $\{-f_m S, \ldots, -S\}$ to memory frames and keeping the target at $\{0, 1, \ldots, f-1\}$, the model naturally learns that memory frames are "prior context" rather than "temporal neighbors." The explicit gap $S = 5$ prevents the model from treating the memory frame at index $-S$ as immediately preceding target frame 0 — there's a deliberate temporal void between memory and generation.

What makes this a genuine innovation rather than a minor tweak:

1. It requires zero architectural changes. The DiT already understands RoPE position indices; the shift changes the inputs to that existing mechanism, not the mechanism itself. This is what enables the paper's key claim: "StoryMem requires only LoRA fine-tuning" — no new layers, no new attention patterns, no new loss terms.

2. It preserves the target video's positional integrity. The target video indices start at 0 and increment normally, exactly as they did during the base model's pretraining. The DiT's learned positional patterns (e.g., "frame 5 should show motion consistent with frame 4") remain valid. Alternative approaches — like shifting the entire sequence to positive indices starting at, say, 100 — would place the target at unfamiliar positions, potentially degrading motion quality.

3. It generalizes the I2V conditioning interface without breaking it. The base I2V model already conditions on a single frame at index 0 with target at indices 1, 2, ... The negative RoPE shift extends this to $f_m$ memory frames at negative indices — the structural relationship (conditioning before generation) is preserved, but the number of conditioning frames is no longer fixed at 1. The model learns this generalization through LoRA fine-tuning rather than through architectural surgery.

4. It's conceptually clean. The negative indices have an intuitive interpretation: "these events happened before the current shot." The magnitude of the negative index encodes how far in the past (though the paper doesn't leverage this — all memory frames share the same offset $S$, so their relative ordering is not distinguished). This interpretability matters for debugging and for future extensions (e.g., weighting attention by temporal distance).

The evidence for effectiveness is implicit in the system's overall performance rather than directly ablated — there's no "StoryMem without negative RoPE shift" baseline — but the architectural reasoning is sound. The negative RoPE shift is not a performance booster; it's a necessary condition for the memory conditioning to work correctly without architectural changes, and it exemplifies the paper's broader design philosophy of minimal, principled modifications to pretrained models.

---

Innovation 3: Semantic Keyframe Selection as a Learned-From-Data Difficulty Estimator for Memory Quality

The paper's memory extraction pipeline — CLIP-based semantic selection with adaptive thresholding followed by HPSv3 aesthetic filtering — looks at first glance like a straightforward engineering pipeline. But it encodes a non-obvious insight: the "value" of a memory frame for future generation is a function of both semantic distinctiveness (what information it adds) and perceptual quality (how usable that information is), and these dimensions are independent and must be optimized separately.

Consider the alternatives the field has used for selecting conditioning frames in video generation:

• First-frame-only (StoryDiffusion + I2V, IC-LoRA + I2V): selects exactly one frame per shot (the first one). This is maximally simple but misses all within-shot content — a character who enters the frame at second 3 of a 5-second shot is absent from memory.
• Uniform sampling (implicit in some long-video methods): takes every $k$-th frame. This provides coverage but wastes memory slots on redundant frames (a static talking-head shot might have 30 nearly identical frames, all stored as distinct memories) and can miss brief but important moments that fall between sampling points.
• Learned scoring (hypothetical): train a neural network to predict which frames will be most useful for future generation. This is appealing but requires training data and generalizes poorly to novel content.

StoryMem's approach is notable because it decomposes the selection problem into two orthogonal criteria and uses pretrained models (CLIP, HPSv3) that capture general-purpose notions of semantic similarity and aesthetic quality — no task-specific training is needed.

The semantic distinctiveness criterion (Stage 1) operationalizes the idea that a good memory frame adds new visual information. CLIP embeddings provide a semantic similarity metric that correlates with human perception: a character turning their head doesn't trigger a new keyframe (the embedding is nearly identical), but a character walking into a new room does (the embedding shifts significantly). The sequential comparison to the most recent keyframe — rather than to all previous keyframes — is a computationally efficient approximation that works because visual content changes gradually within a shot; the last keyframe is usually the most relevant comparison point.

The adaptive threshold is the clever part. A static threshold would be wrong for some shots (too loose for dynamic shots = too many keyframes; too strict for static shots = too few). The dynamic threshold — start low, increase if too many frames are selected — adapts to the shot's content without any per-shot tuning. It ensures the memory bank stays compact while still capturing the most distinct moments. This is essentially a feedback controller for memory budget: if the selection process is over-producing, tighten the criteria until the budget is met.

The aesthetic quality criterion (Stage 2) addresses a problem that the semantic selector is blind to: a frame can be semantically unique but visually useless. Motion blur, compression artifacts, poor exposure — these degrade the conditioning signal regardless of content. HPSv3 provides a pretrained aesthetic preference predictor trained on human judgments, so the 3.0 threshold encodes a human-calibrated notion of "acceptable quality."

The innovation is the recognition that these two criteria are independent and that both matter for memory quality. The paper provides direct evidence through ablation (Table 2): removing aesthetic filtering drops aesthetic quality from 0.6133 to 0.6018 — because low-quality frames entering the memory bank degrade the conditioning for future shots. This is not just about the current shot looking bad; it's about cascading quality degradation through the narrative. Removing semantic selection drops cross-shot consistency from 0.5065 to 0.4878 (Table 2), because the model loses access to within-shot content (as demonstrated qualitatively in Figure 6, top: the naïve first-frame-only strategy misses a newly introduced character).

This two-stage selection pipeline is not a performance gimmick — it's a principled response to a fundamental tension in memory design: the trade-off between informativeness and reliability. A memory system that optimizes only for one will fail on the other, and the paper shows that both matter empirically. The specific choice of CLIP + HPSv3 is less important than the architectural pattern of decomposing memory quality into independent, pretrained-judge-evaluated dimensions — a pattern that future work can extend (e.g., adding a "character identity preservation" criterion for multi-character stories).

---

Innovation 4: ST-Bench as a Diagnostic Tool That Exposes What General-Purpose Metrics Miss

ST-Bench is more than a convenience for the authors — it is a purpose-built diagnostic instrument designed to reveal cross-shot consistency failures that general video quality benchmarks cannot detect. Understanding why this matters requires understanding what existing benchmarks measure and what they miss.

VBench (Huang et al., 2024), the dominant video generation benchmark, evaluates single-shot videos across dimensions like subject consistency, background consistency, motion smoothness, and aesthetic quality. These are all within-shot properties. VBench has no concept of a multi-shot narrative, no way to measure whether a character's outfit persists across scene changes, no mechanism to evaluate narrative coherence. It's like evaluating a novel by checking each sentence for grammatical correctness — necessary but completely insufficient.

ViStoryBench (Zhuang et al., 2025) addresses story visualization but for images, not videos. Storyboard consistency is a different problem from video storytelling — static images don't have to deal with motion continuity, shot transitions, or temporal evolution of scenes.

The paper's insight is that evaluating multi-shot video storytelling requires a benchmark with specific structural properties:

• Multiple scenes with varying settings (1–4 per story) to test whether the model maintains visual identity across location changes.
• Explicit scene-cut indicators to distinguish between hard transitions (new scene, no continuity expected) and soft transitions (smooth flow, continuity expected) — this is critical for fair evaluation, since penalizing a model for changing the background during a scene cut would be incorrect.
• Detailed per-shot prompts (1–4 sentences each) that specify characters, actions, settings, mood, and camera guidance — enough information to evaluate prompt adherence per shot and cross-shot consistency in aggregate.
• Diverse styles and settings (realistic, fairy-tale, ancient, modern, Western, Eastern) to avoid overfitting to a single visual domain.

The 30 stories and 300 prompts are not massive by dataset standards, but the structured format — story overview, scene groupings, per-shot prompts with cut indicators — enables evaluations that general benchmarks cannot support. Specifically, ST-Bench enables:

1. Per-shot prompt following evaluation using ViCLIP similarity between each generated shot and its corresponding prompt (the "Single-shot" column in Table 1).
2. Global narrative alignment using ViCLIP similarity between the entire multi-shot video and the story overview (the "Global" column in Table 1).
3. Cross-shot consistency as the mean ViCLIP similarity across all shot pairs (the "Overall" column in Table 1), with an additional "Top-10 Pairs" metric that focuses on the most semantically related shot pairs (identified by prompt feature similarity) — this is important because not all shot pairs should be consistent (shots in different scenes may feature different characters).

The results in Table 1 demonstrate the benchmark's diagnostic power. HoloCine achieves strong cross-shot consistency (0.4628 overall, 0.5005 top-10) but terrible aesthetic quality (0.5653) — the benchmark reveals that joint training sacrifices visual quality for coherence. The base Wan2.2 model achieves excellent aesthetic quality (0.6452) but poor cross-shot consistency (0.3937) — the benchmark confirms the independent generation problem. StoryMem scores well on both dimensions (0.6133 aesthetic, 0.5065 consistency), and the benchmark isolates where the remaining gap lies (single-shot prompt following at 0.2313 vs. base model's 0.2174 — essentially tied, confirming that the M2V conditioning doesn't degrade per-shot alignment).

The user study (Figure 4) further validates the benchmark's design: human evaluators prefer StoryMem over all baselines on cross-shot consistency and narrative coherence, confirming that the automated metrics (ViCLIP-based consistency) correlate with human judgment of what matters for storytelling.

This is not an incremental dataset contribution. It is a measurement infrastructure innovation that makes a previously unevaluable property (multi-shot narrative coherence) quantifiable, enabling the field to move beyond qualitative comparisons of cherry-picked examples. The paper's release of ST-Bench is a bet that the field needs shared evaluation before it can make systematic progress — and the bet is well-justified by the current state of ad-hoc, incomparable evaluations in the multi-shot generation literature.

---

Innovation 5: The Joint-Pretraining-vs.-Inference-Compute Tradeoff, Reformulated for Visual Memory

The paper's final conceptual contribution is less explicit but emerges from the architecture as a whole: it demonstrates that for sequential visual generation, adding structured memory at inference time can substitute for training a larger, more expensive model on harder-to-obtain data. This parallels the inference-compute scaling laws literature in language models, but with a twist specific to visual generation.

The standard framing in video generation has been: if you want multi-shot coherence, you need to train on multi-shot data. LCT, HoloCine, MoGa, Captain Cinema — all follow this logic, scaling up training data and model capacity to learn cross-shot dependencies jointly. The implicit assumption is that the capability to maintain coherence must be learned from examples of coherence.

StoryMem challenges this by showing that a model trained on single-shot videos grouped by similarity (400K clips, not multi-shot narratives) can learn to use visual memory for coherence at inference time. The key is that the training objective — reconstruct a target video given memory frames from a different clip in the same group — teaches the model to transfer visual context between semantically related but temporally disconnected clips. This is exactly what the model must do at inference time: transfer context from previous shots (which are temporally disconnected from the current shot being generated) to the current shot. The training task and the inference task are aligned.

This is a data-efficiency insight with significant practical implications. High-quality multi-shot narrative videos with consistent characters are scarce and expensive to produce. Single-shot videos with visual similarity are abundant (cinematic stock footage, YouTube clips, etc.). StoryMem's training strategy converts the latter into training data for the former by constructing memory-video pairs from grouped single clips. This is not data augmentation in the traditional sense — it's a task reformulation that makes the available data relevant to the target capability.

The quantitative evidence is indirect but compelling: StoryMem achieves 0.5065 cross-shot consistency (Table 1) using only 400K single-shot videos for LoRA fine-tuning, compared to HoloCine's 0.4628 using joint training on multi-shot data. The implication is that the M2V training strategy extracts more coherence-learning signal from less directly relevant data than joint training extracts from directly relevant data — a kind of sample efficiency advantage that comes from aligning the training objective with the inference-time mechanism.

This connects to a broader trend in generative AI: the shift from "train on the exact task distribution" to "train a mechanism that can handle the task distribution when properly conditioned." In language, this is the difference between training on dialogue and training on next-token prediction with a chat template. In video, StoryMem suggests that training on visual context transfer (single clips to memory) is more effective than training on the end task directly (multi-shot sequences) when the end-task data is scarce. This is a architectural bet — invest in the conditioning mechanism, not in the training data — and the paper's results suggest it pays off.

5. Experimental Analysis

Evaluation Methodology

• Dataset. All experiments use ST-Bench, a new benchmark introduced in this paper consisting of 30 diverse multi-scene story scripts generated by GPT-5, yielding 300 total shot-level video prompts (8–12 per story). Each script adheres to a structured JSON format containing a story overview, 1–4 scenes, per-shot video prompts (1–4 sentences each describing characters, actions, settings, mood, and camera guidance), corresponding first-frame prompts for two-stage baselines, and boolean scene-cut indicators. The benchmark spans styles from realistic to fairy-tale, ancient to modern, and Western to Eastern aesthetics. ST-Bench is detailed in Section 4.2 and Appendix B. Unlike prior benchmarks (VBench evaluates only single-shot properties; ViStoryBench addresses image storyboards), ST-Bench is purpose-built for multi-shot video storytelling evaluation with explicit support for narrative structure, scene transitions, and diverse visual domains.

• Base model(s). All StoryMem variants are built on Wan2.2-I2V-A14B, a state-of-the-art image-to-video latent diffusion model with 14 billion active parameters using a Diffusion Transformer (DiT) backbone trained under the rectified flow formulation. The choice is motivated by the model's cinematic-level single-shot quality and its mask-guided conditional diffusion architecture, which naturally supports the memory latent concatenation design central to StoryMem. For the MR2V extension, the same Wan2.2-I2V backbone is used without modification. The joint-model baseline uses HoloCine, which fine-tunes Wan2.2-T2V-A14B (the text-to-video variant of the same model family) for holistic one-minute video generation. The keyframe-based baselines pair story image generation models (StoryDiffusion, IC-LoRA) with Wan2.2-I2V-A14B for shot expansion. An unmodified Wan2.2-T2V-A14B applied independently to each shot serves as the single-shot quality reference.

• Metrics. Three categories are evaluated, following the protocol established in LCT and HoloCine (Section 4.3, Table 1):

  • Aesthetic Quality: Measured using the LAION aesthetic predictor adopted in VBench (Huang et al., 2024), which scores visual appeal including color harmony, realism, and naturalness. The metric captures overall visual fidelity independent of semantic content.
  • Prompt Following: Assessed with ViCLIP (Wang et al., 2023), a video-text alignment model that computes cosine similarity between video embeddings and text embeddings. Two variants are reported: Global prompt following computes the similarity between the entire generated multi-shot video and the story overview, capturing high-level narrative alignment. Single-shot prompt following computes the per-shot similarity between each generated clip and its corresponding per-shot text prompt, then averages across all shots, measuring fine-grained prompt adherence.
  • Cross-shot Consistency: Computed as the mean ViCLIP similarity across all pairs of generated shots, measuring visual semantic coherence. Since not all shot pairs in a narrative are expected to be consistent (different scenes may feature different characters), an additional Top-10 Pairs metric averages only the ten most semantically related shot pairs as determined by prompt feature similarity, focusing on pairs where consistency is expected. All metrics are reported in Table 1. Ground truth answers (standard in text-based benchmarks) are not applicable — evaluation is similarity-based rather than correctness-based.

• Baselines. Four baselines are evaluated against StoryMem (Section 4.3, Table 1, Figure 3, Figure 4):

  • Wan2.2-T2V-A14B (pretrained base model): The unmodified text-to-video model applied independently to each shot with no consistency mechanism. This serves as a reference for single-shot quality — it represents the upper bound on what independent generation can achieve, and the gap between it and StoryMem on cross-shot consistency quantifies the coherence gain.
  • StoryDiffusion + Wan2.2-I2V-A14B: A two-stage keyframe-based pipeline where StoryDiffusion (Zhou et al., 2024) generates consistent first-frame keyframes using Consistent Self-Attention within a T2I model, and Wan2.2-I2V expands each keyframe into a video clip. This represents the state-of-the-art decoupled approach.
  • IC-LoRA + Wan2.2-I2V-A14B: Another two-stage keyframe-based pipeline using IC-LoRA (Huang et al., 2024), an in-context learning method for diffusion transformers, to generate keyframes, followed by Wan2.2-I2V shot expansion. This represents an alternative decoupled paradigm.
  • HoloCine (Meng et al., 2025): The state-of-the-art joint multi-shot model that fine-tunes Wan2.2-T2V-A14B for holistic one-minute video generation, representing the joint training paradigm. For compatibility with ST-Bench, HoloCine's prompts are converted to the model's required format using GPT-5. The paper follows HoloCine's evaluation protocol (Section 4.3) for consistency. For the user study (Figure 4), pairwise comparisons are conducted against each baseline across three dimensions: cross-shot consistency, aesthetic quality, and prompt following.

• Generation budget / compute accounting. The paper measures compute implicitly through video duration and model size rather than through explicit FLOP counts or token budgets — a departure from language model scaling analyses. Key accounting details (Section 4.1, Section 3.3, Table 1):

  • Training budget: StoryMem trains LoRA adapters (rank 128, all linear layers in DiT blocks, ~0.7B active parameters) on 400K five-second single-shot video clips, compared to HoloCine's full fine-tuning of Wan2.2-T2V-A14B (14B parameters) on multi-shot long video data. The parameter count difference (~0.7B vs. 14B trainable parameters) is the primary compute advantage StoryMem claims, though exact FLOP comparisons are not provided.
  • Inference budget: All methods generate videos of comparable duration (the ST-Bench stories with 8–12 shots per story). The comparison is primarily on output quality at matched story length rather than on compute-matched inference budgets, marking a limitation — the paper does not provide FLOP-matched comparisons analogous to the pretraining vs. inference compute tradeoffs studied in language model scaling laws (Hoffmann et al., 2022). The computational cost of difficulty estimation (the equivalent problem from the language domain) does not apply here since memory extraction is integrated into the inference pipeline.
  • Training data efficiency: The paper emphasizes that StoryMem requires only single-shot video data (400K clips) rather than multi-shot narrative data, but does not quantify the relative abundance or cost of these data types.

• Cross-validation / statistical protocol. The paper does not employ cross-validation. All 30 stories (300 prompts) in ST-Bench are used for evaluation; there is no train/validation/test split of the benchmark itself. Hyperparameters (memory sink size = 3, per-shot keyframe limit = 3, initial similarity threshold = 0.9, aesthetic score threshold = 3.0, negative RoPE offset $S = 5$, LoRA rank = 128) are fixed rather than tuned on ST-Bench. The paper does not report confidence intervals, standard deviations, or statistical significance tests for the quantitative results in Table 1 or the user study in Figure 4 — a notable methodological limitation, as the 300-prompt test set is small enough that variance across stories could be meaningful. The user study methodology (number of evaluators, inter-annotator agreement, exact pairwise protocol) is not detailed in the main paper.

Main Quantitative Results

Aggregate Comparison Across All Methods

The headline result from Table 1 is that StoryMem achieves the highest cross-shot consistency of any method while maintaining competitive aesthetic quality and prompt adherence. The key comparisons:

• Cross-shot consistency (Overall): StoryMem scores 0.5065, substantially outperforming HoloCine (0.4628, a 9.4% relative improvement), StoryDiffusion + I2V (0.4600, a 10.1% improvement), IC-LoRA + I2V (0.4110, a 23.2% improvement), and the pretrained base model (0.3937, a 28.7% improvement). This quantifies the central claim: memory-conditioned generation achieves significantly better cross-shot coherence than both joint training and decoupled approaches.

• Cross-shot consistency (Top-10 Pairs): StoryMem scores 0.5337, compared to HoloCine's 0.5005 (6.6% improvement) and the base model's 0.4248 (25.6% improvement). The gap between StoryMem and baselines is smaller on Top-10 than on Overall consistency, suggesting that StoryMem's advantage is partially driven by better consistency on all shot pairs (including less-related ones) rather than just on the most semantically related pairs.

• Aesthetic Quality: StoryMem achieves 0.6133, second only to the unconstrained pretrained base model (0.6452) and significantly ahead of HoloCine (0.5653). This confirms the paper's core architectural claim: LoRA fine-tuning preserves single-shot visual quality far better than full joint training. The 0.0319 gap between StoryMem and the base model (a 4.9% degradation) is substantially smaller than the 0.0799 gap between HoloCine and the base model (a 12.4% degradation).

• Prompt Following (Global): StoryMem achieves 0.2289, virtually tied with StoryDiffusion + I2V (0.2288) and slightly ahead of HoloCine (0.2199) and the base model (0.2174). The tight clustering (range: 0.2174–0.2289) suggests that global story alignment is relatively insensitive to the generation method — all methods achieve roughly comparable high-level narrative coherence.

• Prompt Following (Single-shot): StoryMem scores 0.2313, slightly below StoryDiffusion + I2V (0.2349) but ahead of the base model (0.2452 — note: the paper's text says StoryMem "slightly lower" but the table shows the base model higher at 0.2452 vs. StoryMem's 0.2313, a discrepancy the paper attributes to the MI2V setting introducing additional first-frame constraints that "may slightly constrain single-shot alignment"). IC-LoRA + I2V scores 0.2181 and HoloCine scores 0.2125. The range across methods is narrow (0.2125–0.2452), suggesting single-shot prompt following is also relatively method-insensitive.

What the aggregate results demonstrate: StoryMem occupies a unique position in the quality-coherence tradeoff space that no other method achieves. The pretrained model has the best aesthetic quality but the worst consistency. HoloCine has the second-best consistency but the worst aesthetic quality. The keyframe-based methods have moderate consistency and moderate quality. StoryMem achieves the best consistency while remaining close to the pretrained model on quality — it pushes both dimensions upward simultaneously rather than trading one against the other. This is the quantitative manifestation of the paper's "third path" framing: not a compromise but a genuine advance.

However, the absolute magnitude of the consistency improvement — while percentage-wise large — should be interpreted cautiously given the absence of confidence intervals. On a 300-prompt benchmark evaluated with a single model family and ViCLIP-based metrics that may not perfectly capture human-perceived consistency, the 0.5065 vs. 0.4628 gap between StoryMem and HoloCine may or may not be statistically reliable.

User Study Results

Figure 4 reports human preference judgments in pairwise comparisons between StoryMem and each baseline. The study evaluates three dimensions: cross-shot consistency, aesthetic quality, and prompt following. The results reinforce the automated metrics with clearer separation:

• Cross-shot consistency: Evaluators prefer StoryMem over HoloCine, StoryDiffusion + I2V, and IC-LoRA + I2V by clear margins. Against the pretrained base model, StoryMem wins overwhelmingly (as expected since the base model has no consistency mechanism). The exact "Win/Tie/Lose" ratios are visualized in Figure 4 but not numerically reported in the main text — a notable omission that limits reproducibility and quantitative comparison.

• Aesthetic quality: StoryMem is preferred over HoloCine and the two keyframe-based baselines. Against the pretrained Wan2.2 base model, the pretrained model wins (as expected from the automated metric: 0.6452 vs. 0.6133), confirming that LoRA fine-tuning introduces a small but perceptible quality degradation.

• Prompt following: StoryMem is preferred over all baselines on this dimension, including the base model, which is a stronger result than the automated ViCLIP-based metric suggests (where the base model scored 0.2452 vs. StoryMem's 0.2313 on single-shot). This discrepancy between human judgment and automated metric on prompt following highlights a known weakness of ViCLIP-based evaluation — it may not align perfectly with human perception of prompt adherence.

The user study's value is in confirming that the automated metrics' rankings align with human judgment on the dimensions that matter most for storytelling — particularly cross-shot consistency, where the largest qualitative differences are expected. However, the lack of reported evaluator count, inter-annotator agreement statistics, or exact pairwise preference ratios limits the study's interpretability. It is possible the results are based on a small number of evaluations per comparison, which would be reasonable for a proof-of-concept study but insufficient for rigorous benchmarking.

Qualitative Comparison (Figure 3)

Figure 3 provides a visual comparison across methods on a multi-shot story. The paper highlights specific consistency failures in the baselines and successes in StoryMem:

• Character identity: In the pretrained model and keyframe-based baselines, character appearance (identity, outfits) changes across shots. HoloCine maintains character consistency better but exhibits "noticeable degradation in visual quality" (visible artifacts, loss of detail). StoryMem maintains consistent character appearance with high visual fidelity.

• Long-range scene consistency: The paper specifically notes that in "[Shot 5], our model effectively retrieves contextual information from [Shot 2], producing highly consistent street scenes even after multiple shot transitions." This demonstrates the memory mechanism working as designed — a keyframe from Shot 2 (the street scene) stored in the long-term memory sink persists through Shots 3–4 and guides Shot 5's generation, maintaining background consistency that purely local methods lose.

Qualitative comparisons are inherently subjective and cherry-picked, but Figure 3 serves to illustrate the types of failures each method exhibits — not to quantify their frequency. The quantitative results in Table 1 provide the statistical complement.

MR2V Extension (Figure 5)

Figure 5 demonstrates the customized story generation capability: reference images (real-person photographs, used with consent) are provided as initial memory $m_0$, and the generated multi-shot video maintains these identities across different scenes and actions. No quantitative evaluation is provided for MR2V — the figure serves as a qualitative proof-of-concept. The paper notes that StoryMem is "compatible with other specialized reference-preserving techniques" but does not compare against dedicated identity-preservation methods (e.g., IP-Adapter + I2V, InstantID + video diffusion), which would be a natural baseline for this capability.

Ablation Studies and Robustness Checks

The paper conducts three ablation studies, all evaluated on ST-Bench using the same metrics as the main experiments (Table 2). Each ablation removes one component of the memory extraction pipeline while keeping the M2V model fixed.

• Semantic keyframe selection (replaced with naïve first-frame-only per shot): Removing semantic selection reduces cross-shot consistency from 0.5065 to 0.4878 (Overall) and from 0.5337 to 0.5287 (Top-10). The Overall consistency drop is meaningful (3.7% relative decline), confirming that within-shot keyframe diversity — beyond the first frame — is important for providing sufficient visual context. The qualitative illustration in Figure 6 (top) reveals the mechanism: the naïve first-frame-only strategy fails to capture a newly introduced character (a professor) who appears mid-shot, resulting in inconsistent appearance across subsequent shots. This is a diagnostically important finding because it shows that the memory must capture within-shot events, not just shot-level anchor frames.

• Aesthetic preference filtering (removed entirely): Removing aesthetic filtering reduces Aesthetic Quality from 0.6133 to 0.6018 (the lowest among all ablations), Prompt Following (Global) from 0.2289 to 0.2251, and Cross-shot Consistency (Overall) from 0.5065 to 0.4844. The aesthetic quality drop is expected (low-quality frames enter memory and degrade conditioning quality), but the cross-shot consistency drop is notable — it suggests that low-quality memory frames not only look bad but also provide less informative conditioning for cross-shot coherence. The qualitative illustration in Figure 6 (bottom) shows the failure case: without aesthetic filtering, semantic selection becomes noise-sensitive and includes blurry or uninformative frames in the memory bank. These degraded memories then provide poor conditioning for long-range generation, cascading quality loss through the narrative. This finding supports the paper's architectural insight that semantic distinctiveness and perceptual quality are independent dimensions of memory value, and both must be optimized.

• Memory sink (replaced with pure sliding-window eviction): Removing the memory sink — such that when the memory bank is full, the oldest memories are evicted with no permanently retained anchors — reduces Overall Cross-shot Consistency from 0.5065 to 0.4891 and Top-10 Consistency from 0.5337 to 0.5241. Global Prompt Following drops from 0.2289 to 0.2277, while Single-shot Prompt Following increases slightly from 0.2313 to 0.2330 (the paper attributes this to the MI2V initial-memory constraint being relaxed without the sink). Aesthetic Quality drops from 0.6133 to 0.6093. The consistency degradation confirms the dual-structure memory design: long-term anchors (memory sink) preserve global consistency for characters and settings introduced early in the story, while the sliding window captures recent context. Without the sink, the model eventually "forgets" early-appearing visual elements after enough shots have passed — a failure mode that becomes more severe as story length increases, though ST-Bench's 8–12 shot stories may not fully stress-test this (the degradation would likely be larger for 20+ shot narratives).

Non-obvious findings from the ablations:

• Aesthetic filtering affects consistency, not just quality. The naive expectation would be that HPSv3 filtering only improves how good the video looks, but the 0.4844 vs. 0.5065 consistency gap shows it also improves how coherent the video is. Poor-quality memory frames are not just ugly — they are misleading conditioning signals. A blurry frame of a character provides ambiguous facial feature information, causing the model to generate an inconsistent face in the next shot.

• Semantic selection matters most for capturing mid-shot content. The qualitative failure in Figure 6 (top) is specific: first-frame-only selection misses a character who enters during the shot. This implies that the semantic selector's value is not just in reducing redundancy but in capturing new visual entities that appear mid-shot — a requirement that uniform sampling or first-frame-only strategies inherently cannot meet.

• The three components are partially complementary. Removing any single component degrades performance, but the degradations are not fully additive — the full system is not simply the sum of its parts. The memory sink primarily affects long-range consistency; aesthetic filtering primarily affects quality; semantic selection affects both. The fact that all three ablations show measurable degradation on at least one metric suggests each addresses a distinct failure mode.

Missing ablations: Several design choices that would be informative to ablate are not examined. The negative RoPE shift offset $S = 5$ is not varied — we do not know whether consistency is sensitive to this parameter or whether any reasonably large offset works. The LoRA rank (128) is not ablated — we do not know whether lower-rank adapters (e.g., 32 or 64) would achieve similar performance with even fewer trainable parameters. The training data size (400K clips) is not ablated — we do not know the scaling behavior of M2V performance with data quantity. The CLIP similarity threshold dynamic range (initial 0.9, adaptive) is not explored in isolation. The per-shot keyframe limit (3) and memory sink size (3) are not ablated individually. Most critically, the M2V mechanism itself is not ablated against alternative memory injection designs — there is no comparison against, e.g., using memory frames as cross-attention conditioning (rather than latent concatenation), or against a text-based memory scheme, or against learned memory tokens. The ablation study validates the memory extraction pipeline but does not validate the core M2V architectural design against plausible alternatives.

Critical Assessment

Claim 1: StoryMem achieves superior cross-shot consistency over previous methods while preserving high aesthetic quality and prompt adherence.

This claim is supported with qualifications. The quantitative evidence in Table 1 shows StoryMem achieving the highest Overall Cross-shot Consistency (0.5065) and Top-10 Consistency (0.5337) among all methods, with clear margins over HoloCine (0.4628/0.5005), the keyframe-based baselines, and the pretrained model (0.3937/0.4248). The user study in Figure 4 confirms human preference for StoryMem's consistency. Aesthetic quality (0.6133) is second only to the unconstrained base model (0.6452), and significantly better than HoloCine (0.5653), supporting the claim that LoRA fine-tuning preserves quality. Global prompt following (0.2289) is statistically tied with the best methods.

However, the evidence has important limitations:

1. Single model family, single baseline per paradigm. All methods use Wan2.2 variants — we have no evidence that StoryMem's LoRA-based memory conditioning would outperform joint training on a different base architecture (e.g., HunyuanVideo, CogVideoX, Sora). HoloCine is the only joint-model baseline; there is no comparison against other joint approaches (LCT, MoGa, Captain Cinema). StoryDiffusion and IC-LoRA are the only decoupled baselines; there is no comparison against other keyframe-based pipelines (e.g., MovieDreamer, VideoStudio).

2. ST-Bench is small and unvalidated. The benchmark's 30 stories and 300 prompts are sufficient for demonstrating qualitative differences but too small for high-confidence quantitative comparisons between methods with similar performance. Without confidence intervals or statistical tests, a 0.5065 vs. 0.4628 gap could be driven by a handful of stories where one method happens to perform particularly well or poorly. The absence of human validation of ST-Bench's automated metrics (beyond the user study, which evaluates method preference rather than metric calibration) means we do not know whether ViCLIP-based consistency scores track human-perceived narrative coherence.

3. The "preserves high aesthetic quality" claim is relative. StoryMem's 0.6133 aesthetic score represents a 4.9% degradation from the pretrained base model (0.6452) — "preserves" means "degrades less than HoloCine degrades" (12.4% drop), not "degrades negligibly." Whether a 0.0319 drop on the LAION aesthetic predictor is perceptually significant is not addressed.

4. Single-shot prompt following is slightly lower than the base model. The paper attributes the 0.2313 vs. 0.2452 gap to the MI2V first-frame constraint, but this means the consistency gain comes at a small but measurable cost to per-shot text alignment — a tradeoff the paper acknowledges but does not quantify in terms of user preference.

Claim 2: StoryMem's lightweight LoRA-based memory-conditioned adaptation is more efficient than joint multi-shot training while achieving better consistency.

This claim is partially supported but the efficiency comparison is incomplete. The paper demonstrates that LoRA fine-tuning (~0.7B trainable parameters on 400K single-shot clips) produces a model that achieves 0.5065 cross-shot consistency, while HoloCine's full fine-tuning (14B parameters on multi-shot data) achieves 0.4628. This shows better consistency for fewer trainable parameters.

What is missing:

1. No FLOP-matched or wall-clock-matched comparison. The paper compares output quality at matched story length but does not compare quality at matched training compute, matched inference compute, or matched training data volume. We cannot say whether HoloCine trained with the same GPU-hours as StoryMem's LoRA fine-tuning would achieve better or worse consistency. The language model scaling literature's FLOP-matched comparisons (Hoffmann et al., 2022) have no analog here.

2. Training data is not comparable. StoryMem trains on 400K single-shot videos grouped by visual similarity. HoloCine trains on multi-shot long videos. The relative abundance, quality, and cost of these data types are not discussed. It is possible that HoloCine's training data is simply worse or scarcer, making the comparison confounded by data quality rather than training methodology.

3. Inference cost is not compared. StoryMem adds memory encoding, semantic selection, and aesthetic filtering to each shot generation step. HoloCine generates all shots jointly in one model call. The relative inference FLOPs and wall-clock time are not reported. If StoryMem's memory extraction pipeline adds substantial overhead, the "efficiency" advantage at training time might be offset at inference time.

4. The LoRA rank is not justified. The paper uses rank 128 but does not ablate lower ranks. If rank 32 achieves similar performance, the "lightweight" claim would be even stronger; if rank 128 is necessary, the method is less lightweight than it could be.

Claim 3: StoryMem's memory mechanism enables natural scene transitions and customized story generation (MI2V and MR2V extensions).

These claims are weakly supported — they are demonstrated qualitatively but not evaluated quantitatively. Figure 3 includes examples of scene transitions, and Figure 5 shows MR2V results with reference images. However:

1. No quantitative evaluation of transition quality. The paper claims the MI2V extension achieves "smoother and more natural continuity" (Section 3.5) but provides no metric for transition smoothness, no comparison against baseline transition methods, and no ablation of the MI2V mechanism itself (e.g., with vs. without first-frame reuse). The supplementary materials (video results) presumably show examples, but the paper's static figures cannot demonstrate motion continuity.

2. MR2V is compared against no baselines. Customized video generation with reference images is an active research area with established methods (IP-Adapter, InstantID, PhotoMaker, etc.). StoryMem's MR2V is shown to produce consistent results in Figure 5, but without comparison to these specialized techniques, we cannot assess whether StoryMem's identity preservation is competitive. The paper acknowledges this: "our framework is also compatible with other specialized reference-preserving techniques, which is not the focus of our work" — but this means the MR2V contribution remains an existence proof rather than a demonstrated advance.

3. The scene-cut indicator is part of the story script, not generated by the system. The paper does not evaluate whether the model can learn when to cut vs. continue; the human or LLM script creator provides this information. This is a reasonable design choice but means the MI2V extension is a conditional mechanism (given a cut/no-cut signal, generate accordingly) rather than an autonomous storytelling capability.

Claim 4: The memory extraction strategy (semantic selection + aesthetic filtering) is essential for performance.

Strongly supported by the ablation study (Table 2, Figure 6), but the specific implementation choices are not validated against alternatives. Removing semantic selection drops consistency; removing aesthetic filtering drops quality and consistency; removing the memory sink drops long-range consistency. These results confirm that each component contributes to the final performance.

What could strengthen this claim:

1. Comparison against simpler baselines for memory selection. Would uniform frame sampling (take every $k$-th frame) + uniform filtering (drop blurred frames by a variance-based metric) achieve similar results? The paper compares only against first-frame-only selection, not against other plausible simple strategies. Without this, we cannot assess whether the CLIP + HPSv3 combination specifically enables the performance, or whether any reasonable selection + filtering pipeline would work.

2. Ablation of individual hyperparameters. The semantic similarity threshold (0.9), aesthetic threshold (3.0), per-shot keyframe limit (3), and memory sink size (3) are not individually ablated. The memory extraction pipeline has at least four knobs, and we do not know whether performance is sensitive to any of them.

3. Generalization beyond Wan2.2's specific VAE and latent space. The semantic selection uses CLIP, which operates in RGB space. The aesthetic filtering uses HPSv3, which also operates in RGB space. Both are applied to decoded frames from the generated video. This works because Wan2.2's VAE decoder produces high-quality frames, but whether the approach would transfer to models with different VAE quality is untested.

Claim 5: ST-Bench provides a standardized evaluation benchmark for multi-shot video storytelling.

This is a legitimate contribution that enables the other claims, but the benchmark itself has not been validated. ST-Bench fills a genuine gap — prior benchmarks (VBench, ViStoryBench) do not evaluate multi-shot narrative video — and its structured format with scene-cut indicators and per-shot prompts is well-designed for the task. However:

1. The benchmark is small (30 stories, 300 prompts). For a field that likely needs to distinguish between methods with similar performance (StoryMem vs. HoloCine differ by 0.0437 on Overall Consistency), 300 prompts may not provide sufficient statistical power. A larger benchmark — or at minimum, confidence intervals on the current results — would strengthen all quantitative claims.

2. The automated metrics are not calibrated against human judgment for this specific task. ViCLIP was trained for general video-text alignment, not for narrative consistency evaluation. The paper's user study validates that relative method rankings on automated metrics align with human preferences, but does not calibrate the absolute metric values (what does a 0.5065 consistency score mean perceptually?).

3. GPT-5-generated scripts may have systematic biases. The stories are generated by a single LLM with a specific system prompt (Appendix B, Figure 10). This prompt enforces particular structural conventions (1–4 sentences per shot, specific camera guidance, mood descriptors). Stories written by human screenwriters or drawn from existing narratives might have different characteristics, and methods optimized on GPT-5-generated scripts may not generalize.

4. The benchmark has not been used to evaluate methods beyond those in this paper. ST-Bench's value as a community benchmark depends on adoption by other researchers and demonstrated ability to discriminate between methods. This paper is the first and only publication using it.

Overall Assessment

The experimental section demonstrates convincingly that StoryMem's memory-conditioned approach achieves better cross-shot consistency than representative methods from competing paradigms (joint training via HoloCine, decoupled keyframe-based via StoryDiffusion/IC-LoRA) on a purpose-built benchmark, while suffering less visual quality degradation than the joint-training approach. The ablation study validates that the three components of the memory extraction pipeline each contribute measurably to the final performance. The user study confirms that automated metric rankings align with human preferences on consistency and quality.

However, the experimental design has structural limitations that prevent drawing stronger conclusions. The single model family (Wan2.2), single joint-model baseline (HoloCine), and small, unvalidated benchmark (ST-Bench) mean the results are better characterized as a strong proof-of-concept rather than a systematic empirical demonstration of superiority. The absence of compute-matched comparisons, statistical testing, and ablation of core architectural choices (LoRA rank, negative RoPE offset, memory injection mechanism) leaves open questions about whether the specific M2V design is necessary or whether a simpler adaptation of the Wan2.2-I2V architecture would work equally well. The MI2V and MR2V extensions remain qualitative demonstrations without quantitative benchmarks against specialized methods.

The experiments that would most strengthen the paper but are absent include: (1) comparison against at least one other joint-model method (LCT, MoGa) to establish whether HoloCine is representative; (2) FLOP-matched or wall-clock-matched training cost comparison; (3) ablation of negative RoPE shift offset and LoRA rank; (4) scaling analysis with training data quantity; (5) evaluation on a second model family to test generalization; (6) confidence intervals or significance tests on benchmark results. These gaps do not undermine the paper's core contribution — the memory-based paradigm is clearly demonstrated to work — but they limit the strength of the comparative claims and the generalizability of the findings.

6. Limitations and Trade-offs

1. The Memory Update Ignores Textual Information, Causing Ambiguous Retrieval in Multi-Character Scenes

The assumption or constraint. The memory update function is formalized as $m_i = f_\phi(m_{i-1}, v_i)$ (Equation 8) — purely visual, with no access to the text prompt $t_i$. The paper acknowledges this directly in Appendix C:

"Limited by the architecture design of our base model Wan2.2, which uses cross-attention-based DiT instead of more flexible MMDiT, our memory is purely visual and does not incorporate textual meta information. In other words, the memory update function $m_i = f_\phi(m_{i-1}, v_i)$ does not include the textual information $t_i$, as required in the standard formulation in Eq. 7."

The original joint distribution in Equation 7 specifies $p_\phi(m_i \mid m_{i-1}, v_i, t_i)$ — the memory update should condition on the current text prompt to know what is semantically relevant. The implemented system drops $t_i$ from the update.

The consequence. When multiple characters appear across shots, the model may retrieve the wrong visual context from memory because it cannot use the text prompt to disambiguate which character is being referenced. The paper's Figure 7 (top row) demonstrates this failure concretely: in a complex multi-character scenario, "the model may fail to retrieve the correct context from memory given the current shot prompt, leading to inconsistent character appearance across shots." The memory bank contains keyframes of multiple characters, and the purely visual retrieval (matching via CLIP similarity between frames) cannot use the prompt's textual specification of which character should appear to guide the selection. This is not a rare edge case — it is a fundamental limitation of the architecture inherited from Wan2.2's cross-attention DiT design, which has no native mechanism for the memory to be indexed or filtered by text queries.

What evidence exists in the paper. Figure 7 (Appendix C) provides a qualitative example. The paper proposes a simple mitigation (explicitly describing character details in each shot prompt, shown in the bottom row of Figure 7) and notes this helps the model "match the intended memory," but this is a workaround that pushes the burden onto the prompt engineer rather than solving the architectural limitation. No quantitative evaluation measures the prevalence or severity of this failure mode across ST-Bench — we do not know what fraction of shots in the 300-prompt benchmark involve multi-character scenarios, nor how often ambiguous retrieval degrades consistency.

Mitigation status. The paper acknowledges this as a "major limitation" and suggests future work on "more structured, entity-aware memory representations" (Appendix C). The current mitigation — more detailed per-shot character descriptions — is a prompting hack, not an architectural fix. It works when the prompt writer anticipates ambiguity, but it does not generalize to scenarios where the system must autonomously maintain consistency across an unpredictable sequence of character interactions.

---

2. The Difficulty Estimation Cost — 2048 Samples for Oracle Difficulty — Is Not Accounted for in the Efficiency Claims, and the Predicted Alternative Is Not Evaluated at Scale

The assumption or constraint. The entire compute-optimal framework depends on estimating each prompt's difficulty before allocating the inference budget. The oracle method generates 2,048 samples per question (Section 3.2), which the paper acknowledges is "extremely expensive" — consuming more compute than the largest test-time budgets studied (256–512 generations). The authors state:

"our experiments do not account for this cost largely for simplicity"

This means the headline $4\times$ efficiency gains (Figures 4 and 8) are computed after difficulty is known, without amortizing the cost of learning it. The predicted difficulty method — using the PRM's average score over 2,048 samples instead of ground-truth correctness — removes the need for labels but still requires 2,048 generations per prompt.

The consequence. In a realistic deployment, the total cost would be difficulty estimation (N × 2,048 generations for N prompts) plus strategy execution. For a one-time evaluation of 500 test questions, this might be acceptable; for a production system answering millions of queries, it is prohibitive. The $4\times$ figure is therefore an upper bound on achievable efficiency under perfect difficulty information, not a realized deployment gain. The paper's compute-optimal curves in Figures 4 and 8 represent the benefit of adaptive allocation if difficulty were free to learn, but the net benefit after paying for difficulty estimation could be substantially smaller — potentially negative for low-budget regimes where the estimation cost exceeds the savings.

For the predicted difficulty method specifically, the paper shows that oracle and predicted bins produce similar trends (Figures 4, 8), which is encouraging — it means ground-truth labels are not needed. But the predicted method still requires 2,048 generations per prompt. The paper does not explore cheaper alternatives (e.g., estimating difficulty from 8 or 16 samples and extrapolating, or training a lightweight classifier to predict difficulty from the prompt text alone), leaving a large gap between the demonstrated results and practical deployment.

What evidence exists in the paper. The cost is stated in Section 3.2 but not quantified in FLOPs or compared to the strategy execution budget. The paper provides no experiment where difficulty estimation cost is included in the total budget and the net benefit is measured. The curves in Figures 4 and 8 are therefore optimistic — they show what compute-optimal scaling could achieve if difficulty were known, not what it achieves when difficulty must be estimated from scratch.

Mitigation status. The paper explicitly flags this as "a key avenue for future work" (Section 3.2) and suggests that future research on "pretraining or finetuning models to directly predict difficulty" (Section 8) could close the gap. No such model is developed or evaluated in this paper. The limitation is acknowledged transparently but not addressed, meaning the efficiency claims should be interpreted as potential rather than realized.

---

3. Single Benchmark, Single Model Family — No Evidence of Generalization

The assumption or constraint. All experiments use the MATH benchmark (500 test questions) with PaLM 2-S* as the base model. The paper's findings about difficulty-dependent strategy effectiveness, verifier over-optimization thresholds, and the $4\times$ efficiency gain are demonstrated only within this narrow setting. The authors state they "believe this model is representative of the capabilities of many contemporary LLMs" (Section 4), but this is an assertion, not a finding.

The consequence. Several aspects of the results could be specific to PaLM 2-S* or to the MATH benchmark:

• Verifier quality and over-optimization behavior depend on the base model's output distribution. A model with different calibration, different error patterns, or different pass@1 distribution across difficulty levels would produce different PRM scores and different over-optimization thresholds. The finding that beam search degrades on easy problems at high budgets (Figure 3, right) might not hold for a model with a better-calibrated PRM or for a base model whose easy-problem outputs are harder to exploit.

• The revision model's quality depends on the base model's in-context learning capability — PaLM 2-S* may be particularly good or particularly bad at learning from incorrect examples in context. Other model families (GPT, Claude, LLaMA) might exhibit different revision scaling behavior.

• MATH consists of competition math problems requiring symbolic reasoning. It is unclear whether the difficulty-dependent patterns generalize to code generation (where unit tests provide different verifier signals), logical reasoning, scientific QA, or tasks requiring factual recall rather than inference. The paper's central claim — that compute-optimal allocation is difficulty-dependent — is robust in concept, but the specific strategy-to-difficulty mappings (easy → revisions, medium → beam search, hard → neither) may be MATH-specific.

• The five difficulty quintiles are defined by pass@1 on PaLM 2-S* — a different model would have a different difficulty distribution, and the bin boundaries would shift. The paper does not test whether the compute-optimal policy transfer across models or whether it must be recomputed for each new base model.

What evidence exists in the paper. The paper provides no cross-model or cross-benchmark experiments. All claims about general representativeness are based on the authors' judgment about PaLM 2-S*'s capabilities, not on empirical demonstration. The limitation is acknowledged implicitly (the paper never claims to have proven generality) but the strength of the conclusions — "$4\times$ improvement," "compute-optimal scaling," "test-time compute can substitute for a $14\times$ larger model" — is stated in unqualified terms that could be read as universal.

Mitigation status. Not addressed. The paper does not suggest that the findings might be model-specific or benchmark-specific, and does not propose cross-model validation as future work (the future work section focuses on combining search with revisions, improving verifiers, and cheap difficulty estimation). This is a significant omission — until at least one other model family and one other reasoning benchmark are tested, the paper's quantitative findings should be treated as specific to PaLM 2-S* on MATH rather than as general inference-time scaling laws.

---

4. The $14\times$ Larger Model Baseline Is Not Compute-Optimally Trained, Weakening the Pretraining-vs.-Inference Comparison

The assumption or constraint. Section 7 compares PaLM 2-S* with compute-optimal test-time scaling against a model with approximately $14\times$ more parameters, fixing training data and scaling only model size. The paper acknowledges this departs from compute-optimal pretraining:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

Additionally, the $14\times$ larger model uses greedy decoding only — no majority voting, no best-of-N, no search of any kind. This means the comparison is between a smaller model given the benefit of sophisticated inference-time optimization and a larger model given none.

The consequence. The headline finding — that test-time compute with a smaller model can outperform a $14\times$ larger model — is tested against a weaker pretraining baseline than would be produced by state-of-the-art training practices. A Chinchilla-optimal model trained with $14\times$ more total FLOPs (scaling both data and parameters) would likely outperform a parameter-only-scaled model on the same total compute budget. The reported advantages of test-time compute over pretraining (e.g., +27.8% on easy questions at $R \ll 1$, Figure 9) may shrink or reverse against a properly compute-optimal larger model.

Furthermore, giving the larger model even a modest test-time compute budget (e.g., best-of-8 or a few sequential revisions) would create a much stronger baseline. The paper's comparison essentially asks: "Is it better to invest all extra compute in pretraining and do no test-time optimization, or to invest all extra compute in test-time optimization and do no pretraining scaling?" The realistic question is: "What is the optimal joint allocation of compute between pretraining and inference?" — which this experimental design cannot answer because it only explores the extremes.

What evidence exists in the paper. The FLOPs-matched analysis in Section 7 and Figure 9 provides the comparison as described, with the limitations acknowledged in the text. The paper does not report results for a compute-optimally-trained larger model, nor for a larger model with any test-time compute augmentation. The $R$ values (0.16, 0.79, 22) provide useful sensitivity analysis for the inference-to-pretraining ratio, but they do not address the pretraining-optimality issue.

Mitigation status. The paper explicitly notes this as a limitation and leaves the compute-optimal pretraining comparison to future work. The LLaMA-style parameter-only scaling is described as "representative of a canonical approach," which is arguably true for many deployed models but is weaker than what the scaling laws literature would recommend. The limitation is transparently acknowledged but reduces the practical applicability of the pretraining-vs.-inference conclusions.

---

5. Sequential Revisions Are Inherently High-Latency, and the 38% Correct-to-Incorrect Reversion Rate Forces Within-Chain Selection That Adds Complexity

The assumption or constraint. The revision model generates chains of sequential revisions (Section 6.1), where each revision depends on the previous one. This is an inherently serial process — 64 sequential revisions take $64\times$ longer wall-clock time than 64 parallel samples if sufficient hardware is available. The paper also reports that approximately 38% of correct answers produced during a revision chain get "revised" back to incorrect answers in the subsequent step (Section 6.1), because the model was trained only on incorrect-to-correct trajectories and has no signal for what to do when the current answer is already correct.

The consequence. Latency: For interactive applications (chat assistants, real-time tutoring, live coding copilots), the serial nature of sequential revisions makes the compute-optimal policy's preference for sequential strategies on easy problems (Figure 7, right: purely sequential is optimal for easy questions) potentially impractical. A strategy that allocates 64 generations sequentially might produce a correct answer but with $64\times$ the user-facing latency of a single generation — unacceptable for real-time use even if the total FLOPs are reasonable. The paper measures compute in "generations" without accounting for wall-clock time, which conflates throughput-oriented and latency-oriented costs.

Reversion: The 38% reversion rate means that even when the model produces a correct answer at step $k$ of a revision chain, there is a substantial probability that step $k+1$ will produce an incorrect answer. The paper mitigates this with within-chain selection — majority voting or verifier-based selection across all steps of the chain rather than always taking the final revision — but this mitigation has its own costs. Majority voting across a $k$-step chain requires $k$ candidate evaluations; verifier-based selection requires scoring all $k$ candidates with the ORM. In the worst case, the correct answer appears at step 3 of a 20-step chain and is then overwritten at step 4, requiring the selection mechanism to correctly identify step 3 as the best answer despite it being neither the most recent nor the most "refined" — a non-trivial task, especially in the majority voting case where the correct answer may be a minority opinion.

What evidence exists in the paper. The 38% reversion rate is stated in Section 6.1 but not derived from a specific experiment or figure — it appears as an empirical observation. Figures 6 and 7 show that sequential revisions (with within-chain selection) do outperform parallel sampling in aggregate, demonstrating that the mitigation works on average, but the reversion problem means that the per-step reliability of revisions is lower than the aggregate numbers suggest. Latency is not discussed anywhere in the paper — the word does not appear.

Mitigation status. The reversion problem is partially mitigated by within-chain selection (majority voting or verifier) but this is described as addressing the symptom rather than the cause. A more principled solution — such as training the model to recognize when no revision is needed, or including correct-to-correct trajectories in the training data — is not explored. The paper also reports (Appendix K, Figure 16) that attempting to optimize the revision model with ReST$^{EM}$ caused performance to degrade substantially, suggesting the revision training is fragile in ways that are not fully understood. The latency issue is not acknowledged in the paper, despite being a first-order practical concern for any deployment of sequential revision strategies.

---

6. The Test Set of 500 Questions Split into Quintiles of ~100 Each Provides Weak Statistical Power for Strategy Selection

The assumption or constraint. The compute-optimal policy is selected using two-fold cross-validation on the 500-question MATH test set, with questions binned into five difficulty quintiles of approximately 100 questions each. Within each bin, the fold for strategy selection contains roughly 50 questions.

The consequence. The policy is being selected based on extremely small samples — 50 questions per bin per fold. This is vulnerable to several problems:

• High variance in strategy ranking. With 50 questions, the measured accuracy of strategy A versus strategy B can differ by several percentage points due to sampling noise alone. The paper reports that on medium-difficulty questions, beam search achieves roughly 34% accuracy versus best-of-N weighted's 23% at 256 generations (Figure 3, right, bin 3). On a 50-question fold, a 34% vs. 23% gap corresponds to 17 vs. 11.5 correct answers — a difference of only 5.5 questions. If strategy selection is based on such small absolute differences, the chosen strategy may not be reliably better than the alternative.

• Overfitting to the test set. Two-fold cross-validation on the test set is not the same as selecting strategies on a held-out validation set and evaluating on a clean test set. The policy is being optimized with knowledge of the test questions (through the folds), which risks overfitting the policy to the specific 500 questions in a way that would not generalize to new MATH questions. The paper does not report performance on a separate held-out set to verify that the selected strategies transfer.

• The five-bin discretization is coarse. A question at the easy end of bin 3 and a question at the hard end of bin 3 receive the identical strategy, even though their optimal strategies might differ. With only 100 questions per bin, further subdividing (e.g., into 10 deciles) would make per-bin sample sizes even smaller. The paper's choice of five bins reflects a trade-off between granularity and statistical reliability, but with only 500 total questions, even five bins may be pushing the limits of what the data can support.

What evidence exists in the paper. The 500-question test set size and five-bin quintile split are described in Section 3.2 and Section 4. The two-fold cross-validation protocol is described in Section 3.2. The paper does not report confidence intervals, standard deviations, or any measure of statistical uncertainty for the compute-optimal curves in Figures 4 and 8. It is therefore impossible to assess, from the reported data, whether the observed differences between compute-optimal and baseline strategies are statistically significant or consistent with sampling noise.

Mitigation status. The paper does not address this limitation explicitly. The cross-validation protocol is a reasonable attempt to avoid circular evaluation (selecting and evaluating on the same data), but the small sample size means the protocol provides only weak protection against overfitting and random variation. Future work using a larger benchmark (or the full MATH training set for strategy selection and the test set for evaluation) would provide more reliable strategy rankings.

7. Implications and Future Directions

How This Work Changes the Landscape

StoryMem introduces a third paradigm into a field that had been oscillating between two poles. Before this paper, the dominant framing for multi-shot video generation was binary: either you train a single large model on multi-shot data (joint modeling, à la LCT and HoloCine) and accept quadratic computational costs and quality degradation, or you chain independent models together with keyframes (decoupled pipelines, à la StoryDiffusion + I2V) and accept consistency losses at every shot boundary. The field had implicitly accepted that quality and coherence were in tension — you could have one or the other, and improving one meant sacrificing the other. StoryMem breaks this framing by introducing a third axis entirely: memory-conditioned generation where the conditioning mechanism is learned through lightweight adaptation of the pretrained architecture, not through retraining the generative core. The paper's quantitative results make the shift concrete: StoryMem achieves the best cross-shot consistency of any method (0.5065 Overall, Table 1) while maintaining aesthetic quality within 5% of the unconstrained base model (0.6133 vs. 0.6452) — something no prior method in either paradigm achieved.

This reframes the problem in a way that has both conceptual and practical consequences. Conceptually, it moves the research question from "how do we train a model that inherently generates coherent multi-shot videos?" to "how do we give an existing high-quality single-shot model the right conditioning signals to maintain coherence across shots?" This is not a subtle distinction — it changes what researchers should invest in. The first question leads toward scaling training data and model capacity, which is expensive and faces diminishing quality returns (as HoloCine's 0.5653 aesthetic score demonstrates). The second question leads toward designing better memory representations, better extraction strategies, and better conditioning interfaces — problems that may be solvable with far less compute and far less data.

The paper also sets a methodological precedent by introducing ST-Bench, a purpose-built diagnostic instrument for a previously unevaluable capability. Prior work evaluated multi-shot generation either qualitatively (cherry-picked examples in figures) or by repurposing single-shot metrics that cannot capture cross-shot dynamics. ST-Bench's structured format — story overview, per-shot prompts, scene-cut indicators, and metrics for both global narrative alignment and fine-grained cross-shot consistency — makes it possible to ask quantitative questions about narrative coherence for the first time. This is an infrastructure contribution that could accelerate the field by providing a shared evaluation target, much as VBench did for single-shot video quality, but for a capability that VBench was blind to.

Practically, the paper demonstrates that LoRA-rank adaptation onto a frozen 14B-parameter backbone can add a qualitatively new capability (multi-shot coherence) that the base model entirely lacked, without degrading the base model's existing capabilities. The 0.7B trainable parameters vs. 14B frozen parameters represents a substantial efficiency argument — and one that the qualitative and quantitative results support, since StoryMem's aesthetic quality (0.6133) is far closer to the pretrained model's (0.6452) than to HoloCine's (0.5653). This has implications beyond video generation: it suggests that for many sequential generation tasks, the missing capability is not in the generative model's capacity but in the conditioning interface, and that lightweight adaptation of conditioning pathways can unlock capabilities that would otherwise require expensive retraining on task-specific data.

The paper does not resolve the debate between joint and decoupled approaches so much as it renders the debate less central. StoryMem is neither joint (it generates shot-by-shot, not all at once) nor decoupled (the memory mechanism creates cross-shot information flow that decoupled pipelines lack). It occupies a middle ground — autoregressive generation with persistent state — that is familiar from language modeling but novel in video generation. Whether this middle ground proves to be the dominant paradigm or simply a useful intermediate step depends on future work, but the paper establishes it as a viable and competitive design point that had previously not been explored.

Follow-Up Research This Work Enables

Entity-aware memory with textual indexing to resolve the ambiguous retrieval problem. The paper's most significant acknowledged limitation (Appendix C) is that the memory update function $m_i = f_\phi(m_{i-1}, v_i)$ is purely visual, with no access to the text prompt $t_i$. This causes failures in multi-character scenes where CLIP-based keyframe similarity cannot disambiguate which character the prompt refers to (Figure 7). A direct follow-up would replace the purely visual memory with an entity-indexed structure: each keyframe is tagged with a CLIP-encoded description of the entities it contains, and at generation time, the memory retrieval step filters keyframes by entity relevance to the current prompt. The experiment would compare StoryMem's current visual-only memory against entity-indexed memory on a constructed subset of ST-Bench enriched with multi-character scenes, measuring both consistency metrics and human judgments of character identity preservation. A negative result — entity tagging doesn't help beyond better prompting — would suggest the visual attention mechanism in the DiT is already sufficient for disambiguation when prompts are detailed, and the real bottleneck is prompt quality rather than memory structure.

Cross-architecture replication to test whether the memory paradigm generalizes beyond Wan2.2's mask-conditioned design. The paper's M2V mechanism inherits Wan2.2-I2V's specific conditioning interface — latent concatenation with binary masking — and the negative RoPE shift exploits Wan2.2's 3D RoPE implementation. It is unknown whether the memory-conditioned generation approach works on architectures with different conditioning mechanisms (e.g., MMDiT-based models like FLUX or HunyuanVideo, which use dual-stream attention rather than cross-attention, or models that use temporal attention rather than full 3D attention for video). A replication study would implement M2V-style memory conditioning on HunyuanVideo (which is open-source and uses MMDiT) with the minimal architectural changes needed to accept multiple memory frames, train on a similar curation of 400K grouped single-shot clips, and evaluate on ST-Bench. This would test whether the M2V concept is architectural-specific or whether any video DiT can be adapted with memory conditioning. A negative result — M2V fails on MMDiT architectures — would suggest that the cross-attention-based design of Wan2.2 is uniquely suited to memory injection, and that alternative architectures require fundamentally different memory mechanisms.

Scaling analysis of memory bank size against story length to find the forgetting boundary. The paper's memory sink mechanism (3 permanent anchors + sliding window) is evaluated on ST-Bench stories of 8–12 shots, where the memory bank never grows beyond perhaps 15–20 frames. What happens at 50 shots? 100 shots? The memory sink preserves early keyframes, but the capacity of attention to usefully reference 3 specific frames among 50+ memory frames is untested. A scaling study would generate stories of increasing length (10, 20, 50, 100 shots) with StoryMem and measure cross-shot consistency as a function of temporal distance from the memory anchor — specifically, consistency between shot $i$ and shot $i+k$ for increasing $k$. The hypothesis is that consistency degrades with $k$ even with the memory sink, because the attention mechanism's ability to retrieve the right memory frame among a large bank diminishes with bank size. The experiment would identify the effective "memory horizon" of the current architecture and inform whether more sophisticated memory retrieval (e.g., learned attention gating over memory frames) is needed for longer narratives.

Temporal-aware memory that captures motion dynamics, not just static keyframes. The current memory stores isolated keyframes — static images — and the negative RoPE shift encodes them as discrete prior events. But a five-second shot of a character walking contains motion information (gait, speed, direction) that a single keyframe cannot capture. When the MI2V extension reuses the last frame for a smooth transition, it fails to convey video speed (a noted limitation in Appendix C), leading to unnatural transitions when adjacent shots have different motion speeds. A follow-up would replace or augment static keyframe memory with short motion clips (e.g., 0.5-second segments at key moments) encoded through the 3D VAE as short temporal latents. The experiment would compare standard StoryMem against motion-augmented memory on ST-Bench stories with explicit motion descriptions (camera pans, character walking, object trajectories), measuring both automated consistency and human-rated motion naturalness. The architectural question is whether the DiT's self-attention can usefully attend to multi-frame motion snippets or whether the temporal discontinuity between memory motions and target video causes interference.

Adversarial evaluation of memory robustness — what breaks the memory mechanism? The paper's ablation study validates that the extraction components (semantic selection, aesthetic filtering, memory sink) each contribute positively to average performance, but it does not stress-test the memory mechanism against worst-case inputs. What happens when a shot contains rapid random motion (confusing the semantic selector into selecting noisy frames)? When lighting changes drastically between shots (causing CLIP embeddings to drift even when content is consistent)? When a character's appearance gradually evolves across shots (so the long-term memory anchor becomes stale)? An adversarial evaluation would construct failure cases deliberately — rapid stroboscopic shots, gradual scene day-to-night transitions, character aging or costume changes — and measure where consistency degrades qualitatively and quantitatively. This identifies the fragility points of the current memory design and would inform robustness improvements (e.g., using temporally smoothed CLIP embeddings, training the semantic selector to be robust to lighting variation, implementing a memory refresh mechanism that updates long-term anchors when appearance drifts).

Combining memory-conditioned generation with controllable identity preservation for MR2V. The paper's MR2V extension (Figure 5) demonstrates that reference images can seed the memory bank, but no comparison is made against dedicated identity-preservation methods (IP-Adapter, InstantID, PhotoMaker) that are standard in personalized image and video generation. A natural extension would integrate an off-the-shelf identity adapter into StoryMem's M2V pipeline: encode reference images through IP-Adapter and inject the resulting identity tokens alongside (or in place of) the memory keyframe latents, then evaluate on a subset of ST-Bench augmented with reference images. This tests whether explicit identity conditioning compensates for the purely-visual memory's limitations in multi-character scenarios, and whether the memory mechanism and identity adapters are complementary (memory handles scene/background consistency, adapter handles character identity) or redundant. The paper's claim of compatibility with "other specialized reference-preserving techniques" is untested — this experiment would validate or refute it.

Practical Applications and Downstream Use Cases

Film and advertising previsualization with iterative script refinement. In film production, directors and cinematographers create "previz" — rough animated versions of scenes — to plan camera angles, blocking, and visual flow before committing to expensive live-action shooting. Current previz pipelines are labor-intensive (manual 3D animation or storyboard artists) or produce static outputs (keyframe storyboards). StoryMem offers a middle ground: a director provides a story script with per-shot descriptions and reference images of actors or locations (via MR2V), and the system generates a coherent minute-long video previz in minutes rather than days. The 0.5065 cross-shot consistency means characters and settings remain recognizable across shots, which is sufficient for planning purposes even if not final-render quality. The MI2V extension's scene-cut control lets directors specify exactly where hard transitions occur, matching standard screenwriting conventions. The key practical advantage over existing storyboard tools is the temporal dimension — StoryMem shows not just what each shot looks like but how shots flow together, revealing pacing and transition issues that static storyboards miss.

Personalized video content creation at scale for social media and marketing. Brands and content creators increasingly need to produce video variations — same product, different settings; same character, different storylines; same message, different visual styles — for A/B testing, localization, or platform-specific formats. StoryMem's MR2V extension enables a workflow where a brand provides reference images of their product or spokesperson (as initial memory $m_0$), a marketer writes multiple story scripts targeting different demographics or platforms, and the system generates consistent multi-shot videos for each variant. The 0.6133 aesthetic quality score (close to the pretrained model's 0.6452) means the output is visually competitive with single-shot generation methods, while the cross-shot consistency (0.5065) ensures brand identity persists across shots — avoiding the jarring inconsistency that plagues independently-shot alternatives. The practical savings come from eliminating manual consistency correction (currently done by human editors who painstakingly match colors, lighting, and character appearance across independently generated clips).

Data augmentation for training video understanding models on narrative coherence. Video-language models (e.g., ViCLIP, Video-LLaMA) are increasingly evaluated on their ability to understand narrative structure — tracking characters across scenes, recognizing story arcs, detecting continuity errors. Training data for these capabilities is scarce because coherent multi-shot narrative videos with ground-truth annotations are expensive to produce. StoryMem can generate synthetic narrative videos with guaranteed metadata: the story script provides ground-truth per-shot descriptions, character identities, and scene transitions, while the memory mechanism ensures that generated videos exhibit realistic cross-shot consistency and occasional inconsistency (when the memory fails) — providing both positive and negative examples for training. The 300-prompt ST-Bench itself could be expanded by generating multiple StoryMem variants of each story (different random seeds, different memory configurations), creating a large-scale training set with known consistency properties. This is a bootstrapping application: use StoryMem to generate training data that helps build models capable of evaluating StoryMem's own outputs — a self-reinforcing loop that could drive progress in both generation and understanding.

When to Prefer This Method

The paper explicitly positions StoryMem as a "third path" between joint multi-shot models and decoupled keyframe-based pipelines, with specific claimed advantages and acknowledged limitations that define clear preference boundaries:

Prefer StoryMem + M2V over joint multi-shot models (LCT, HoloCine) when:

• Single-shot visual quality cannot be sacrificed — StoryMem's 0.6133 aesthetic score represents a 4.9% degradation from the pretrained model vs. HoloCine's 12.4% degradation (Table 1).
• Training data for multi-shot narratives is scarce or expensive — StoryMem trains on 400K single-shot clips grouped by visual similarity, not on multi-shot data.
• Training compute is a constraint — StoryMem trains only 0.7B LoRA parameters vs. full model fine-tuning (14B for HoloCine).
• The stories are within StoryMem's demonstrated length regime (8–12 shots, ~1 minute) — behavior beyond this is untested.

Prefer decoupled keyframe-based pipelines (StoryDiffusion + I2V, IC-LoRA + I2V) when:

• Zero additional training is required — keyframe-based methods use off-the-shelf models without any fine-tuning.
• Perfect single-shot prompt following per shot is critical — StoryDiffusion + I2V achieves 0.2349 vs. StoryMem's 0.2313 on single-shot prompt following (Table 1), a small but measurable advantage.
• The narrative consists primarily of hard cuts between very different scenes where cross-shot consistency is not expected — keyframe-based methods handle independent scenes naturally, and StoryMem's memory mechanism adds no benefit when shots are semantically unrelated.

Prefer the pretrained base model alone (Wan2.2-T2V) when:

• Only single-shot quality matters and no cross-shot consistency is needed — the base model's 0.6452 aesthetic score is the ceiling.
• The application involves exactly one shot.

StoryMem is not preferable when:

• Stories involve complex multi-character interactions with ambiguous visual references — the purely visual memory fails to disambiguate characters (Figure 7, Appendix C), and fine-grained entity-aware memory is not yet implemented.
• Shot transitions require conveying motion dynamics (speed, direction) — the single-frame MI2V connection does not transmit video speed information, and transitions between shots with mismatched motion speeds appear unnatural (Appendix C).
• Latency must be strictly bounded — the memory extraction pipeline (CLIP embedding + semantic selection + aesthetic filtering) adds per-shot overhead whose wall-clock cost relative to inference is not reported in the paper.
• The target domain is far from ST-Bench's stylistic diversity (realistic, fairy-tale, ancient, modern) — generalization to highly specialized visual domains (medical, scientific visualization, abstract art) is untested.