NarraScore: Bridging Visual Narrative and Musical Dynamics via Hierarchical Affective Control

🎯 Pitch

A frozen vision-language model can act as a continuous affective sensor for long-form video, distilling visual streams into emotion trajectories that steer a music generator without architectural cloning. This lightweight approach achieves state-of-the-art narrative alignment in soundtracks, with the performance gap over other methods widening on longer sequences, while adding negligible computational cost.

1. Executive Summary

NarraScore introduces a hierarchical framework for synthesizing narratively coherent soundtracks for long-form videos, using emotion as a high-density compression of narrative logic to bridge the semantic gap that current methods fail to address. The system repurposes a frozen Video-Language Model (VideoLlama-3) as a continuous affective sensor via a Lightweight Latent Affective Decoder, distilling visual streams into dense Valence-Arousal trajectories, and then employs a Dual-Branch Injection Strategy to reconcile macro-level stylistic stability (a Global Semantic Anchor conditioning the model on genre and atmosphere) with micro-level narrative tension (a Token-Level Affective Adapter that injects distilled affective cues as additive bias into shallow decoder layers). Experiments on long-form video datasets with the MusicGen-Small backbone demonstrate state-of-the-art emotional dynamic consistency and narrative alignment — with the performance gap widening on extended sequences — while introducing negligible parameter overhead, establishing that frozen VLMs can serve as autonomous narrative-to-music bridges even with limited labeled affective data and that a lightweight adapter suffices to steer a frozen acoustic backbone without architectural cloning.

2. Context and Motivation

The Core Problem: Long-Form Video Soundtracks Require Narrative Understanding, Not Just Visual Matching

The fundamental problem NarraScore tackles is that generating a coherent background music track for a long-form video (think: a 10-minute film scene, a documentary segment, a narrative commercial) is qualitatively different from generating music for a short clip (30 seconds or less). The paper argues that this distinction is not merely one of duration — it reflects a fundamental shift in what the soundtrack must accomplish.

For a short clip, a static mood often suffices. A 15-second action shot can be paired with energetic orchestral stabs, and if the intensity is right and the beats roughly align with the cuts, the result feels passable. But for a long-form narrative, the music must evolve. Tension rises gradually across dialogue scenes, releases at a climax, shifts into melancholy during a denouement. The soundtrack is not accompaniment — it is a narrative engine that actively shapes how viewers interpret visual events. The paper cites evidence that music functions as more than background: it influences visual attention, modulates affective responses, and shapes the perceived meaning of ambiguous scenes (Millet et al., 2021; Ma, 2022; Dasovich-Wilson et al., 2022).

This matters for several practical and research reasons the paper articulates (Section 1):

The generative video landscape is shifting toward long-form creation. With advances in video generation models (OpenAI's Sora, Google DeepMind's Veo 3, and others cited in the introduction), the ability to produce long-form visual content is rapidly democratizing. But the corresponding soundtrack capability has not kept pace — creating a bottleneck where high-quality visuals are paired with rudimentary, repetitive, or manually stitched audio.
Professional scoring remains labor-intensive and expensive. Human composers understand narrative arcs intuitively — they know when to introduce a motif, when to build tension, when to resolve. Automating this requires machines to develop something analogous: an understanding of story progression, not just frame-by-frame visual similarity.
Current automated methods fail in ways that are perceptually salient to viewers. The paper's user studies (Tables 2 and 3) show that existing approaches produce soundtracks rated as "generic," "weakly coupled to the visual storyline," or suffering from "narrative drift" and "inconsistent motifs" in long-form contexts. These are not subtle academic deficiencies — they are immediately noticeable to human listeners.

The Three Bottlenecks: Scalability, Coherence, and Semantic Blindness

The paper identifies three specific impediments that stall current video-to-music generation when extrapolated from short clips to long-form narratives (Section 1):

Computational scalability. Dense frame-level attention mechanisms — the dominant paradigm in prior work — incur quadratic memory costs with sequence length. For minute-level videos, this becomes prohibitive. But the paper identifies a deeper issue beyond raw cost: attention dilution. When every frame attends to every other frame across a long sequence, critical narrative cues (a character's subtle expression change, a quiet moment before action) are drowned out by visual redundancy. The signal-to-noise ratio collapses. This is not solved by simply throwing more compute at the problem; it requires structural changes to how visual information is aggregated.

Temporal coherence and style drift. Standard autoregressive models (MusicGen, Music Transformer, etc.) lack global semantic anchors when applied to video conditioning. Without an explicit mechanism to maintain a consistent musical identity across time, the generated soundtrack fragments — the instrumentation shifts, the tempo wanders, the harmonic language drifts. The paper frames this as a structural problem: these models are designed for short-horizon next-token prediction, not for maintaining a coherent musical persona over hundreds of tokens. Prior work that attempts long-form generation through sliding windows without global conditioning often produces audible "seams" at window boundaries.

Semantic blindness to narrative logic. This is the paper's most critical diagnosis and the one it most directly addresses. Current methods rely on surface-level visual representations — features extracted by video encoders like SlowFast, VideoMAE, or ViT-based backbones trained primarily for object recognition and action classification. These features encode what is in the frame (a person, a car, a building) but not what it means in the context of a story. A close-up of a face is just a face — the encoder doesn't know if the character is learning devastating news or receiving a marriage proposal. The editing rhythm (fast cuts vs. long takes), the lighting changes, the camera movement — all carry narrative information that surface-level features fail to capture.

The paper terms this semantic blindness and identifies it as the root cause of the "monotonous background ambience" that previous long-form methods produce. They are acoustically continuous but narratively inert — the music plays on without responding to the story.

Prior Approaches and Their Specific Shortcomings

The paper organizes prior work into three categories (Section 2), each with clearly articulated failure modes at the long-form scale:

Frame-level dense attention methods (CMT, Video2Music, MuVi, VMAS, GVMGen, VeM). These approaches — which represent the dominant paradigm for short-clip generation — project dense video frames through adapters into the conditioning space of a pre-trained audio backbone like MusicGen. They work reasonably well for clips under 30 seconds because the frame-level correspondence is sufficient when the mood is static and the computational cost is manageable. But when applied to minute-long sequences, two failures compound: (a) the quadratic memory cost becomes computationally infeasible, and (b) more fundamentally, the lack of any hierarchical temporal abstraction means the model cannot distinguish between moment-to-moment visual variation (which should drive local dynamics) and scene-level narrative shifts (which should drive structural musical changes). Everything is flattened into a single stream of frame-level features.

Long-form specific architectures (VidMuse, JenBridge). The paper acknowledges these as partial advances that address specific symptoms without treating the underlying disease. VidMuse proposes specialized adapters for long-term and short-term temporal features with sliding-window inference — this solves the computational scalability problem but does not address semantic blindness. JenBridge adopts a divide-and-conquer approach: segment the video, score each segment independently, stitch via transitions — this achieves acoustic continuity but, as the paper argues, "overlooks the fundamental semantic shift in long-form content." The paper's critique is sharp: "By treating long videos merely as extended sequences, current solutions fail to capture these dynamic arcs, yielding monotonous background ambience rather than responsive, narrative-aligned soundtracks."

Emotion-driven methods (Video2Music, EMSYNC, M2UGen, FilmComposer, JenBridge, VeM, MTCV2M). This line of work recognizes that emotion is the bridge between visual storytelling and musical expression. But the paper identifies critical limitations in how prior methods operationalize this insight:

Discrete classification is too coarse. Video2Music uses CLIP for frame-level emotion classification into discrete categories. EMSYNC maps discrete categorical predictions to continuous Valence-Arousal values via a psychology-driven mapping mechanism. These approaches inherit the accuracy limitations of CLIP-based classifiers on nuanced emotional content and, more importantly, produce categorical outputs that cannot represent the continuous, fluid nature of emotional evolution in long narratives. A scene doesn't jump from "happy" to "sad" — it transitions through gradations that discrete bins miss.
LLM-based captioning avoids continuous control. Methods like M2UGen, FilmComposer, and JenBridge leverage Large Language Models to analyze visual content and generate emotion captions or descriptive text. This captures richer semantic understanding than CLIP classifiers, but the paper notes a crucial limitation: these methods "largely circumvent the use of continuous emotion curves." They produce a single global label or a text description that controls the overall mood but lacks the temporal granularity — the frame-by-frame or second-by-second evolution — needed for dynamic soundtrack generation. The paper attributes this avoidance to the "scarcity of continuous affective data and the prohibitive cost of fine-grained annotation."
Extrinsic guidance is unscalable. VeM and MTCV2M attempt to incorporate fine-grained temporal control, but they rely on externally provided guidance signals — essentially requiring manual specification of emotional trajectories. The paper argues this renders them "unscalable for autonomous, large-scale applications." If a human must draw the emotion curve, the automation benefit is largely lost.

The deeper problem with off-the-shelf emotion recognizers. Section 2.3 provides a nuanced critique of existing emotion recognition models that is essential for understanding NarraScore's design choices. Mainstream affective recognition benchmarks (AFEW-VA, Aff-Wild2, VEATIC) are predominantly face-centric — they focus on decoding the expressed emotion of actors via facial dynamics. But video soundtrack generation requires interpreting induced affect — the emotional response the video is designed to elicit in viewers, which depends on cinematography, editing, music itself, narrative context, and cultural conventions, not just facial expressions. A horror film scene might show a character smiling (expressed emotion: happy) while the audience feels dread (induced affect: fear). A model trained on facial affect alone would produce completely wrong musical accompaniment.

The paper acknowledges that LIRIS-ACCEDE attempts to align with viewer-perceived emotion, but "the limited scale across these benchmarks precludes the training of robust, data-hungry models." Moreover, existing state-of-the-art methods "often lack the high-level semantic reasoning required to capture nuanced narrative shifts, resulting in noisy and temporally inconsistent predictions in complex, non-facial scenes." This is a key justification for why NarraScore does not simply plug in an existing emotion recognizer — doing so would introduce "significant error propagation."

How NarraScore Positions Itself

NarraScore's positioning is defined by three strategic design decisions that directly address the failures enumerated above:

First, it treats emotion as a high-density compression of narrative logic, not a supplementary signal. The paper's central insight is that Valence-Arousal trajectories — continuous, time-varying, two-dimensional curves — can serve as a sufficient statistic for the narrative information that matters for music generation. This is a specific, falsifiable claim: if you can accurately extract the emotional arc from a video, you can generate a coherent, narratively aligned soundtrack. The entire architecture is built around this hypothesis — the VLM is probed to produce these trajectories, and the acoustic decoder is conditioned on them. This contrasts with prior emotion-driven work that either (a) used discrete emotions, losing temporal continuity, or (b) used global labels, losing temporal structure. NarraScore's continuous VA curves preserve both the what (valence: positive/negative; arousal: calm/excited) and the when (second-by-second evolution) of narrative emotion.

Second, it repurposes frozen VLMs as affective sensors rather than training task-specific recognizers. This is the paper's most architecturally innovative claim. Instead of training a dedicated emotion recognition model — which would require large-scale continuous affect annotations that don't exist for narrative video — NarraScore probes the internal representations of a frozen VideoLlama-3 model. The VLM was not trained for emotion recognition; it was trained for general video understanding, question answering, and instruction following. The paper's hypothesis is that the rich semantic priors learned during this broad pretraining include implicit representations of narrative tension, mood, and emotional progression — they're just not explicitly surfaced. By training only a lightweight probing head (an MLP with average pooling, Equation 3) on a limited amount of continuous affect data, NarraScore can extract these implicit representations into explicit VA trajectories without fine-tuning the massive backbone.

This is significant methodologically because it sidesteps the data bottleneck. The paper explicitly acknowledges that "the annotative burden of continuous emotion labeling has led to a persistent scarcity of large-scale, high-quality datasets" (Section 2.3). By leveraging a frozen VLM, NarraScore needs only enough labeled data to train a small probe — not enough to train an entire emotion recognition system from scratch. The frozen VLM provides the semantic understanding; the probe just learns to read it out in the right format.

The paper also makes a specific engineering argument for using a VLM rather than a simpler visual encoder: the VLM's training on interleaved video-text data and its instruction-following capabilities mean it has developed reasoning pathways that connect visual sequences to abstract concepts. The "Instruction-Driven Semantic Steering" component (Section 3.3) exploits this by prepending a system instruction that explicitly primes the model for narrative analysis — "suppress the activation of low-level object enumeration patterns while activating high-level narrative reasoning pathways." This is only possible because the VLM has these pathways to activate; a standard video encoder trained on action recognition would not.

Third, it decomposes video-to-music generation into orthogonal global and local control streams. The Dual-Branch Injection strategy (Section 3.5) is NarraScore's answer to the coherence-vs-dynamism tension. The Global Semantic Anchor handles the macro-level: it produces a text description of genre, instrumentation, emotional atmosphere, and rhythmic pacing that conditions the acoustic decoder through standard cross-attention. This ensures that the entire soundtrack maintains a consistent musical identity — you don't get orchestral swells in the first minute and synth-pop in the third. The Token-Level Affective Adapter handles the micro-level: it injects the continuous VA trajectory as additive bias into the shallow layers of the decoder, modulating local tension and dynamics without disrupting the global style.

The paper positions this as architecturally minimalist — a deliberate contrast to the "heavy architectural cloning" of prior adapter-based approaches. The global branch uses the decoder's existing cross-attention mechanism (no new parameters), and the local branch adds only a small temporal super-resolution adapter (dilated convolutions to smooth the sparse VA signal into dense audio-rate control) plus zero-initialized additive bias terms in the shallow layers. This keeps the parameter overhead negligible and, the paper argues, "preserves the generative priors of the frozen backbone" — a practical concern because fine-tuning all of MusicGen's parameters on limited video-music pairs would risk catastrophic forgetting of the high-quality audio synthesis capabilities learned during pretraining.

The Specific Gap This Paper Fills

Reading across Sections 1 and 2, the specific unfilled gap that NarraScore occupies can be stated precisely: no prior method autonomously extracts continuous, temporally dense affective trajectories from raw video using the reasoning capabilities of frozen VLMs, and then uses those trajectories for token-level control of a frozen acoustic decoder in a parameter-efficient, long-form-capable architecture. Every element of this description matters:

Autonomously — no manual guidance, no external classifiers, no human-drawn curves. The system ingests pixels and outputs music.
Continuous, temporally dense — not discrete categories, not global labels, but a second-by-second VA curve that captures the fluid evolution of narrative emotion.
Using reasoning capabilities of frozen VLMs — not surface visual features, not CLIP embeddings, but the deep semantic understanding that emerges from instruction-tuned video-language pretraining.
Token-level control — not just a global conditioning signal, but frame-aligned modulation of the decoder's hidden states.
Parameter-efficient — the VLM and acoustic backbone are frozen; only a small probe, a temporal adapter, and gating scalars are trained.
Long-form-capable — the sliding window inference with global semantic anchoring handles sequences beyond the training clip length without coherence collapse.

Each piece of this specification addresses a documented failure mode of prior work, and the combination is what the paper claims as novel.

3. Technical Approach

3.1 Reader Orientation

NarraScore is a system that watches a long-form video and automatically generates a continuous background music track that responds to the story's emotional arc — music that gets tense when the scene gets tense, resolves when the narrative resolves, and maintains a consistent musical style throughout, all without any human specifying what the music should do. The system solves the problem that current video-to-music methods treat videos as sequences of visual frames (recognizing objects and actions) rather than as narratives (understanding rising tension, emotional subtext, dramatic pacing) by introducing a hierarchical architecture where a frozen Vision-Language Model serves as a narrative sensor that extracts continuous emotion curves, and a frozen music generator is steered by those curves through a lightweight adapter that adds narrative tension information directly into the music generation process without retraining the entire audio model.

3.2 Big-Picture Architecture (Diagram in Words)

NarraScore processes a video through four major stages, with information flowing from raw pixels to emotional understanding to structured control signals and finally to audio waveforms:

Unified Visual-Narrative Backbone (VideoLlama-3, frozen): The full-length video is sampled at 1 frame per second and fed into a frozen Vision-Language Model alongside text timestamps and a system instruction that primes the model for narrative analysis. This backbone is not trained — it provides all the semantic understanding.
Narrative-Aware Affective Reasoning (lightweight probe, trainable): A small MLP head reads the VLM's internal representations at each timestep and projects them onto the Valence-Arousal plane, producing a continuous two-dimensional curve $\mathcal{E}_{local}$ that captures how the narrative emotion evolves second-by-second. Simultaneously, the VLM's language generation capability produces a structured text description $\mathcal{S}_{global}$ of the overall musical style (genre, instrumentation, atmosphere, pacing).
Hierarchical Acoustic Synthesis (MusicGen-Small, frozen + trainable adapter): The global style description $\mathcal{S}_{global}$ conditions the music generator through its standard cross-attention mechanism (ensuring consistent musical identity). The local emotion curve $\mathcal{E}_{local}$ is temporally upsampled to match the audio rate, then injected as additive bias into the shallow transformer layers of the music generator (modulating moment-to-moment tension and dynamics).
Sliding-Window Long-Form Inference: For videos longer than the model's context window, the system processes overlapping segments while carrying forward the global style anchor and the final audio tokens from previous segments as a prompt prefix, ensuring seamless musical continuity across the entire duration.

3.3 Roadmap for the Deep Dive

First, the problem formulation (Section 3.1): The formal definition of inputs, outputs, and the dual constraints of global coherence and local alignment — this establishes what "success" means mathematically.
Second, the overall architecture (Section 3.2): The cascaded perception-synthesis pipeline and the autoregressive formulation in Equation 1, which sets up the decomposition into global and local conditioning that everything else depends on.
Third, the Narrative-Aware Affective Reasoning module (Section 3.3): The most novel component — how a frozen VLM is probed to produce continuous Valence-Arousal trajectories. This includes the temporal alignment strategy, the instruction-driven semantic steering, the latent affective probing mechanism (Equation 3), and the hybrid loss function (Equation 4). Understanding this is essential because the quality of the emotion curves determines everything downstream.
Fourth, the Holistic Musical Conceptualization module (Section 3.4): How the VLM produces the global semantic anchor — the structured text description that controls overall musical style. This is conceptually simpler but structurally essential for preventing style drift.
Fifth, the Hierarchical Acoustic Synthesis module (Section 3.5): How the global and local control signals are injected into the frozen MusicGen backbone. This includes the temporal super-resolution adapter (Equation 5), the token-level residual modulation (Equation 6), and the training objective (Equation 7). This is where the architecture's minimalism is demonstrated.
Sixth, the scalable long-form inference strategy (Section 3.6): The overlapping sliding-window approach and how global reasoning, continuous affective extraction, and autoregressive continuation are coordinated across segments.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems architecture paper whose core novel idea is that a frozen Vision-Language Model, when probed with a lightweight learned head, can extract continuous emotion trajectories from raw video that are sufficient to drive a frozen music generator toward narratively coherent output, and that this can be done by injecting those trajectories as additive bias into shallow decoder layers — a parameter-efficient alternative to full fine-tuning or architectural cloning.

Problem Formulation

The paper formally defines the video-to-music generation task in terms that make the long-form challenge explicit (Section 3.1). The input is a video sequence $\mathcal{V} = \{v_1, \dots, v_{T_v}\}$ consisting of $T_v$ frames. The target output is a discrete acoustic sequence $\mathcal{A} \in \{1, \dots, N\}^{T_a \times K}$ , which is the tokenized representation produced by a neural audio codec — specifically EnCodec, which uses $K$ residual codebooks (multiple parallel streams of discrete tokens representing different aspects of the audio signal at different fidelity levels) and a vocabulary size of $N$ . Here $T_a$ denotes the length of the acoustic token sequence, which is inherently dense (many tokens per second of audio), while the visual sequence $T_v$ is deliberately kept sparse (the paper uses 1 frame per second, or 1 Hz) to respect the memory capacity limits of the Vision-Language Model when processing minute-level inputs.

The critical design constraint is stated as a dual requirement: global coherence (the music must maintain a unified stylistic identity across the entire video — no genre switching, no abrupt instrumentation changes, no tempo wandering) and local alignment (the music must respond to frame-level narrative dynamics — tension must rise and fall in synchrony with the visual story). These two requirements are in tension because mechanisms that enforce global consistency (like fixed conditioning vectors) tend to suppress local variation, while mechanisms that enable local responsiveness (like frame-level attention) tend to fragment long-term structure. NarraScore's architecture is explicitly designed to reconcile this tension by separating the two control pathways.

The paper notes that the acoustic sequence is "inherently dense" while the visual sequence is "kept sparse" — this resolution discrepancy is a structural challenge that the Temporal Super-Resolution Adapter (Section 3.5) is specifically designed to address. At a 1 Hz frame rate with EnCodec operating at 50 Hz, there are 50 acoustic tokens for every visual observation, creating a severe underspecification problem that naive interpolation would not solve.

The Autoregressive Decomposition and Hierarchical Conditioning

The generative process is formulated as an autoregressive sequence prediction conditioned on hierarchical cues (Equation 1):

$p(\mathcal{A} \mid \mathcal{V}) = \prod_{t=1}^{T_a} p(a_t \mid a_{<t}, \mathcal{S}_{global}, \mathcal{E}_{local})$

where $\mathcal{S}_{global}$ is the global semantic anchor (a text description capturing genre, instrumentation, atmosphere, and pacing) and $\mathcal{E}_{local}$ is the local affective trajectory (a sequence of Valence-Arousal pairs at each timestep).

What it computes: the probability of the entire acoustic token sequence $\mathcal{A}$ given the video $\mathcal{V}$ is factorized as the product of conditional probabilities of each acoustic token $a_t$ given all previous acoustic tokens $a_{<t}$ and the two hierarchical conditioning signals $\mathcal{S}_{global}$ and $\mathcal{E}_{local}$ . At each generation step, the model considers its entire history of generated audio tokens plus the global style description plus the current position in the emotion curve to predict the next audio token.

Why this form: this decomposition separates the generation problem into two orthogonal conditioning axes — what the paper calls "macro-scale atmospheric modeling" (via $\mathcal{S}_{global}$ ) and "micro-scale tension tracking" (via $\mathcal{E}_{local}$ ). The standard autoregressive factorization $\prod p(a_t \mid a_{<t})$ is standard for audio language models (MusicGen, AudioLM, etc.), but the key addition is the explicit decoupling of global and local conditioning signals. Without this decoupling — if, for example, all conditioning information were collapsed into a single vector per frame — the model would have no structural incentive to maintain a consistent musical identity across time, because each frame's conditioning would be treated independently. The global anchor provides a stationary signal that the autoregressive process can learn to respect even as the local emotion signal varies.

This factorization also reflects the paper's core architectural claim: that the video-to-music problem can be decomposed into two sub-problems that can be solved by separate mechanisms. The global anchor $\mathcal{S}_{global}$ is produced by the VLM's language generation capability (a form of cross-modal reasoning that maps visual scenes to musical concepts), while the local trajectory $\mathcal{E}_{local}$ is produced by probing the VLM's internal representations (a form of representation reading that maps visual sequences to continuous affect). These two extraction mechanisms operate on the same VLM but produce outputs at different levels of abstraction and temporal granularity.

Narrative-Aware Affective Reasoning: The VLM as Continuous Affective Sensor

This is the paper's most technically innovative module. The challenge is to extract a continuous, temporally dense Valence-Arousal trajectory from raw video frames using only a limited amount of continuous affect training data. The solution has four sub-components: temporal alignment, semantic steering, latent probing, and the optimization objective.

Semantically-Anchored Temporal Alignment

The first problem is how to present a long video sequence to a Vision-Language Model that was not designed for dense temporal regression. The paper's solution is "Semantically-Anchored Temporal Alignment": instead of using learned temporal position embeddings (which would require training and might not generalize to unseen sequence lengths), the system discretizes the video stream into a uniform 1 Hz sequence (one frame per second) and interleaves each frame with a discrete semantic clock — a text token formatted as "Time: t s" that explicitly tells the model where in the temporal sequence each frame belongs.

The video is thus reconstructed as a linear causal sequence that conforms to the VLM's native "interleaved" schema (the format the model was trained on: alternating text instructions, visual tokens, and text descriptions). The visual tokens $V_t$ retain their spatial fidelity via the VLM's intrinsic positional encoding mechanism — the VLM's Vision Transformer front-end already handles spatial position within each frame, so the temporal dimension is the only one that needs explicit structuring.

This design choice has specific advantages that the paper implies but doesn't fully articulate: (1) using text-based temporal markers leverages the VLM's strong text understanding capabilities to establish temporal relationships — the model "knows" that "Time: 30 s" comes after "Time: 29 s" because it understands natural language temporal ordering, not because it learned temporal position embeddings; (2) the 1 Hz sampling rate balances narrative granularity against the VLM's context window limitations — one frame per second captures narrative-relevant visual changes (facial expressions, scene transitions, action beats) while keeping the total token count manageable for minute-long sequences; (3) the format is compatible with the VLM's instruction-following interface, enabling the semantic steering mechanism described next.

Instruction-Driven Semantic Steering

The second problem is that VLMs are trained as generalist visual reasoners — they can describe objects, answer questions about scenes, and follow instructions — but they are not specifically trained for continuous emotion regression. The paper's solution is to repurpose the VLM's instruction-following interface to "steer" its internal representations toward affective analysis.

The mechanism is a Semantic Primer: a system instruction $\mathcal{T}_{inst}$ that is prepended to the aligned video sequence. The full input representation $X$ is constructed as: $X = [\mathcal{T}_{inst}, \tau_1, V_1, \tau_2, V_2, \dots, \tau_T, V_T]$

where $\mathcal{T}_{inst}$ is the system instruction (a carefully crafted text prompt that primes the model for narrative emotion analysis), $\tau_t$ is the temporal marker for frame $t$ (the text "Time: t s"), and $V_t$ is the visual tokens for frame $t$ .

The paper makes a specific functional claim about what this instruction does: it "modulates the self-attention mechanism to explicitly suppress the activation of low-level object enumeration patterns while activating high-level narrative reasoning pathways." In operational terms, the instruction text influences the attention patterns throughout the VLM's transformer layers by providing context that biases the model to attend to narrative-relevant visual features (character expressions, lighting changes, camera movement, scene composition) rather than object-identity features (what objects are present, their spatial relationships).

This is only possible because the VLM has both types of processing pathways available from its pretraining. A model trained solely on object recognition (like a standard ViT) would not have "high-level narrative reasoning pathways" to activate. The instruction-following capability of modern VLMs — their ability to modulate their behavior based on text prompts — is what makes this semantic steering possible. The paper does not provide the exact text of $\mathcal{T}_{inst}$ , but from context it likely includes instructions to focus on emotional tone, narrative tension, mood evolution, and atmospheric qualities while ignoring object-level descriptions.

Latent Affective Probing

The third component is the mechanism that actually extracts the continuous Valence-Arousal values from the VLM's internal representations. This is a probing approach: rather than training the VLM to output emotion values (which would require fine-tuning the entire model and risk catastrophic forgetting of its general capabilities), NarraScore trains only a lightweight readout head that maps the VLM's frozen hidden states to the Valence-Arousal plane.

Let $Z_t \subset H^{(L)}$ denote the set of contextualized hidden states corresponding to the visual tokens of the $t$ -th frame, extracted from the final layer $L$ of the VLM. These are the internal representations produced by the transformer after processing the entire sequence up to that point — they encode not just the content of frame $t$ but its relationship to all previous frames, the instruction context, and the temporal position. The probing head operates in three steps:

Spatial Average Pooling: The multiple visual tokens for a single frame (a ViT typically produces one token per image patch, so there are multiple tokens per frame) are aggregated into a single holistic frame-level representation by computing the mean across all visual tokens: $\frac{1}{M} \sum_{z \in Z_t} z$ , where $M$ is the number of visual tokens per frame. This collapses the spatial dimension, discarding information about where in the frame emotional cues are located to focus on the global emotional content.
MLP Projection: The pooled representation is transformed by a Multi-Layer Perceptron (MLP) — a small feedforward neural network — that maps from the VLM's hidden dimension (likely 4096 or similar for VideoLlama-3) to a 2-dimensional output representing Valence and Arousal.
Clipping: The output is clipped to $[-1, 1]$ using the $\operatorname{Clip}_{[-1,1]}$ operator, constraining the Valence and Arousal values to the standard psychological range where -1 represents negative valence / low arousal and +1 represents positive valence / high arousal (Russell's circumplex model of affect, cited in the paper). This is formalized in Equation 3:

$e_t = \operatorname{Clip}_{[-1,1]}\left(\operatorname{MLP}\left(\frac{1}{M}\sum_{z \in Z_t} z\right)\right)$

where $e_t \in \mathbb{R}^2$ is the predicted Valence-Arousal vector for frame $t$ .

What it computes: for each frame, the probe reads the VLM's internal representation of that frame (after spatial pooling to collapse the patch-level information), passes it through a small neural network, and produces a 2D point on the Valence-Arousal circle bounded between -1 and +1. This point represents the system's estimate of the emotional content of the video at that moment — how positive/negative the narrative feels (valence) and how calm/excited it feels (arousal). When computed for every frame, these points form a continuous trajectory $\mathcal{E}_{local} = \{e_1, e_2, \dots, e_T\}$ .

Why this form: the probing approach is the paper's key strategy for circumventing the data bottleneck. The VLM is frozen — all $H^{(L)}$ representations come from a model that was never trained on emotion regression. The MLP probe has only a few thousand parameters (two layers with GELU activation, projecting from the VLM's hidden dimension to a 2D output). This means it can be trained on the limited continuous affect data available (the paper uses approximately 884 minutes of annotated video) without overfitting. The spatial average pooling discards spatial information that is largely irrelevant for narrative emotion (whether an emotional expression is in the top-left or bottom-right of the frame doesn't matter) while preserving the global emotional content. The clipping to $[-1, 1]$ enforces the psychological convention of the Valence-Arousal circumplex and prevents the model from producing degenerate out-of-range values that would confuse the downstream music generator.

A subtle but important point: the probe reads from the final layer of the VLM ( $H^{(L)}$ ), meaning it has access to representations that have been contextualized by the full sequence up to that point, including the system instruction and all previous frames. This is what allows the probe to capture narrative emotion rather than just visual emotion — frame 30's representation is influenced by what happened in frames 1–29, so the predicted valence at a character's smile can be modulated by whether the preceding context suggests the smile is genuine or menacing.

Optimization Objective for Affective Probing

The probe is trained using a hybrid loss function that combines L2 and L1 norms (Equation 4):

$\mathcal{L}_{emo} = \frac{1}{T}\sum_{t=1}^{T}\left(|e_t - \hat{e}_t|_{2}^{2} + \lambda|e_t - \hat{e}_t|_{1}\right)$

where $e_t \in \mathbb{R}^2$ is the predicted Valence-Arousal vector, $\hat{e}_t \in \mathbb{R}^2$ is the ground-truth Valence-Arousal vector from the continuous affect dataset, $T$ is the number of frames, and $\lambda$ is a balancing coefficient that controls the relative weight of the L1 and L2 terms.

What it computes: for each frame in the training video, the loss measures the discrepancy between the probe's predicted emotion and the human-annotated ground-truth emotion. The first term $|e_t - \hat{e}_t|_{2}^{2}$ is the squared Euclidean distance (L2 norm squared) — this penalizes large errors heavily and provides smooth gradients that aid convergence. The second term $\lambda|e_t - \hat{e}_t|_{1}$ is the absolute distance (L1 norm) — this is more robust to outliers in the annotation data because it does not square the error. The total loss for a video is the average over all frames.

Why this form: the hybrid L1+L2 objective is chosen for a specific calibration reason. The L2 (Mean Squared Error) component provides strong gradients when predictions are far from targets, driving fast initial convergence. However, MSE is sensitive to outliers — if a few frames have noisy or inconsistent human annotations (which is common in continuous affect labeling due to annotator fatigue or ambiguity), MSE would disproportionately penalize the model for those frames. The L1 (Mean Absolute Error) component provides robustness against such outliers because it scales linearly with error rather than quadratically. The balancing coefficient $\lambda$ allows the practitioner to trade off between these two properties. The paper does not report the specific value of $\lambda$ used, but the formulation acknowledges that continuous affect annotation is inherently noisy and that a pure MSE objective would be poorly calibrated for this setting.

Holistic Musical Conceptualization: The Global Semantic Anchor

While the affective reasoning module produces the temporally dense $\mathcal{E}_{local}$ trajectory, a separate pathway in the VLM produces the global semantic anchor $\mathcal{S}_{global}$ — a structured text description that controls the overarching musical style. This module operates at the macro level and is conceptually simpler than the affective probe, but structurally essential for preventing the style drift that plagues prior long-form methods.

The paper frames this as a "cross-modal sensory translation" task: the VLM must translate visual scene content into auditory imagery descriptions. The key design principle is modality decoupling: the system explicitly suppresses references to specific visual objects and cinematography, forcing the model to describe only the implied auditory experience. For example, rather than producing "a dark forest with a character walking slowly" (visual description), the system should produce "a brooding orchestral piece with low strings and sparse percussion, evoking mystery and isolation" (auditory description aligned with music generation).

To achieve this, the paper employs what it calls a structured instruction paradigm. Rather than allowing free-form text generation (which would produce variable and potentially musically irrelevant descriptions), the VLM is constrained to synthesize a description that covers four specific musical dimensions:

Genre and stylistic context: the overarching musical genre (orchestral, electronic, ambient, jazz, etc.) and the stylistic tradition the music should evoke.
Instrumentation and timbral texture: the specific instruments or sound sources and their textural qualities (e.g., "solo piano with light string accompaniment," "distorted electric guitar with heavy drums").
Emotional atmosphere: the primary emotional quality of the music (e.g., "melancholic," "triumphant," "tense").
Rhythmic pacing: the tempo and rhythmic character (e.g., "slow and spacious," "driving and urgent").

The paper argues that this schema "significantly reduces output variance and mitigates the ambiguity often associated with open-ended captioning." By projecting the visual content onto this predefined semantic subspace, the VLM's output becomes a reliable control signal for the downstream music generator rather than a variable natural language description that might or might not contain musically relevant information.

An important practical detail: the system uses the keyframe extraction capability of the VLM to compress the entire long-form video into a sparse sequence of representative frames before performing this global reasoning. This allows the VLM to consider the entire narrative arc within its limited context window — it sees the beginning, middle, and end of the story in compressed form — and produces a single global description that captures the overarching mood rather than being biased toward any particular segment. This global description $\mathcal{S}_{global}$ is then used as a stationary condition throughout the entire music generation process, ensuring that the musical identity remains consistent even as the local emotion signal varies.

The global semantic anchor is integrated into the acoustic decoder through the standard cross-attention mechanism that MusicGen already uses for text conditioning (described in the Hierarchical Acoustic Synthesis section below). This means no new architectural components are needed for the global pathway — it leverages the existing text-to-music capabilities of the pretrained backbone.

Hierarchical Acoustic Synthesis: Dual-Branch Injection into MusicGen

This module translates the two extracted narrative priors ( $\mathcal{S}_{global}$ and $\mathcal{E}_{local}$ ) into a coherent acoustic waveform. The core design challenge is to inject these signals into a frozen MusicGen-Small backbone without disrupting its pretrained generative distribution or requiring expensive fine-tuning of its 300M+ parameters. The solution has four sub-components: explicit semantic bridging (global), dense affective projection (local temporal upsampling), token-level control injection (local residual modulation), and the training objective.

Explicit Semantic Bridging (Global Pathway)

The global semantic anchor $\mathcal{S}_{global}$ — the structured text description covering genre, instrumentation, atmosphere, and pacing — is integrated through MusicGen's pre-trained cross-attention mechanism. This is the standard conditioning pathway that MusicGen was designed for: the text description is tokenized and encoded, then the acoustic decoder's transformer layers attend to the text tokens via cross-attention at each generation step.

The paper emphasizes that this requires "no invasive architectural modifications" — it uses the backbone's existing capability. This is a deliberate design choice: MusicGen was pretrained on large-scale text-to-music data, so its cross-attention mechanism already knows how to map text descriptions of genre, instrumentation, and mood into musical characteristics. By feeding the VLM's narrative-derived description through this pathway, NarraScore leverages the backbone's existing text-to-music alignment without any additional training for this branch.

The global anchor establishes the timbral and stylistic foundation — the choice of instruments, the harmonic language, the overall texture — and remains constant throughout the generation. This is what prevents the style drift that occurs in methods without global conditioning: even as the local emotion signal modulates dynamics and tension, the fundamental musical identity (e.g., "orchestral, strings and brass, melancholic atmosphere, slow tempo") stays fixed.

Dense Affective Projection (Local Pathway Resolution Matching)

A structural challenge specific to the local pathway is the significant resolution discrepancy between the sparse visual emotion cues (1 Hz, one VA pair per second) and the dense acoustic tokens (EnCodec operates at 50 Hz, producing 50 audio tokens per second of music). Directly injecting the sparse signal would produce stepped, unnatural transitions in the musical output — the music would jump between emotional states at 1-second intervals rather than evolving smoothly.

The solution is a Temporal Super-Resolution Adapter $\mathcal{F}_{\phi}$ that transforms the sparse affective trajectory into a dense control signal aligned with the audio rate. The process has two stages (Equation 5):

$C_{local} = \mathcal{F}_{\phi}(\operatorname{Interp}(\mathcal{E}_{local}))$

where $\mathcal{E}_{local} \in \mathbb{R}^{T_v \times 2}$ is the sparse Valence-Arousal trajectory (one 2D point per video frame), $\operatorname{Interp}$ is linear interpolation that upsamples from the video frame rate to the acoustic token rate (from $T_v$ points to $T_a$ points), and $\mathcal{F}_{\phi}$ is a learnable temporal smoothing network that transforms the 2-dimensional Valence-Arousal manifold into a $D$ -dimensional latent space compatible with the acoustic decoder's hidden dimension.

What it computes: first, linear interpolation fills in the gaps between the 1-second-spaced VA points to produce a continuous curve at the audio rate — this gives a smooth baseline evolution. Second, $\mathcal{F}_{\phi}$ applies a stack of dilated temporal convolutions (a 1D convolutional network with exponentially increasing dilation factors, following the Temporal Convolutional Network architecture from Bai et al., 2018) that serves two purposes: (1) smoothing local jitter and artifacts from the interpolation, and (2) expanding the temporal receptive field so that each point in $C_{local}$ is influenced by a wider temporal context. The result $C_{local} \in \mathbb{R}^{T_a \times D}$ is a dense control signal with the same temporal resolution as the acoustic tokens and a dimensionality matching the decoder's hidden states.

Why this form: the two-stage design separates the problem into a non-learned resolution-matching step (interpolation) and a learned smoothing/mapping step (dilated convolutions). Linear interpolation handles the "obvious" part of the resolution mismatch — making sure there's a value at every time step — while the dilated convolutions handle the musically sophisticated part: ensuring that emotional transitions are fluid rather than mechanical, and that the control signal captures longer-range emotional trends (a gradual tension buildup over 10 seconds) rather than just second-by-second fluctuations. The use of dilated convolutions (rather than, say, a simple low-pass filter) is significant because it provides a large receptive field with few parameters — dilation factors of 1, 2, 4, 8, ... can cover a wide temporal window without the quadratic cost of dense attention or the vanishing gradient issues of deep standard convolutions.

The projection from 2D (Valence-Arousal) to $D$ dimensions (the decoder's hidden dimension, likely 1024 for MusicGen-Small) is necessary because the acoustic decoder's hidden states live in a much higher-dimensional space than the VA plane. The 2D VA signal doesn't have enough capacity to directly modulate the full richness of the decoder's representations — projecting to $D$ dimensions gives the adapter the representational capacity to learn how different aspects of emotion map to different aspects of musical expression (e.g., arousal might map primarily to rhythmic density and dynamic range, while valence maps to harmonic mode and timbral brightness).

Token-Level Control Injection (Residual Modulation in Shallow Layers)

Once the dense affective features $C_{local}$ are produced, they must be injected into the frozen acoustic decoder in a way that modulates the generation without disrupting the backbone's pretrained distribution. The paper's solution is a residual modulation strategy with specific design choices about where and how the injection occurs.

The key design decision is to restrict injection exclusively to the shallow transformer blocks of the decoder. Let $h_t^{(l)}$ denote the hidden state at time step $t$ and layer $l$ of the decoder (after the self-attention and feedforward operations of that layer). The injection is applied as an additive bias (Equation 6):

$h_t^{(l)^{\prime}} = h_t^{(l)} + \gamma \cdot C_{local, t}, \quad \forall l \in \{1, \dots, L_{shallow}\}$

where $h_t^{(l)^{\prime}}$ is the modulated hidden state, $C_{local, t} \in \mathbb{R}^D$ is the dense control signal at time $t$ , $\gamma$ is a learnable scalar gating parameter, and $L_{shallow}$ is the number of shallow layers that receive the injection (the exact number is not specified, but the paper indicates it is a subset of the total decoder layers).

What it computes: at each time step in each of the shallow decoder layers, the system takes the hidden state produced by the frozen transformer and adds a small, learned perturbation $\gamma \cdot C_{local, t}$ . This perturbation encodes the narrative emotion at that moment — a point in the projected VA space. The scalar $\gamma$ gates the magnitude of the perturbation and is initialized to zero, meaning at the start of training the injection has no effect and the model behaves exactly like the original MusicGen. As training progresses, $\gamma$ learns how strongly to weight the affective signal.

Why this form and the shallow-layer restriction: this design embodies three specific hypotheses about how music generation transformers work:

Shallow layers control high-level structure; deep layers refine acoustic detail. The paper explicitly cites "prior studies regarding the hierarchical distribution of information in music generation models" (Lan et al., 2024, Musicongen) as motivation for restricting injection to shallow layers. The hypothesis is that early transformer layers in autoregressive music models primarily determine the high-level trajectory — the overall shape, dynamics, and tension arc — while deeper layers focus on optimizing acoustic fidelity, harmonic voice-leading, and fine-grained timbral consistency. By injecting the narrative emotion signal only into the shallow layers, NarraScore aligns the generation trajectory with the intended emotional arc without interfering with the deep layers' ability to produce high-quality audio. If injection were applied to all layers, the affective signal might over-constrain the generation and produce artifacts or degraded audio quality.
Additive bias is less invasive than architectural cloning. Many prior controllable generation methods (including ControlNet for images, which the paper cites as Zhang et al., 2023b) create a copy of the backbone's architecture and train it to produce control signals that are added to the frozen backbone's outputs. This "architectural cloning" approach preserves the backbone but at the cost of doubling the parameter count during training. NarraScore's additive bias approach — injecting a single learned vector per timestep — introduces far fewer parameters: only the temporal super-resolution adapter $\mathcal{F}_{\phi}$ , the gating scalars $\gamma$ (one per injected layer), and the zero-initialized bias. The paper claims "negligible parameter overhead," which is credible given that $\mathcal{F}_{\phi}$ is a small stack of 1D convolutions and the projection from 2D to $D$ dimensions requires only a linear layer.
Zero-initialization preserves the pretrained distribution. The gating scalar $\gamma$ is initialized to zero so that at the start of training, $h_t^{(l)^{\prime}} = h_t^{(l)}$ — the injection contributes nothing and the model produces exactly the same outputs as the original MusicGen. This is a standard technique in adapter-based fine-tuning (similar to LoRA's zero-initialized B matrix) that ensures the model doesn't suffer from initial random perturbations that could destabilize training. As $\gamma$ grows during training, the affective signal gradually shapes the generation, allowing the model to smoothly interpolate between its original unconditional distribution and the narratively-conditioned distribution.

The paper's claim that this design "balances the trade-off between control precision and audio quality" can be understood in terms of this layer-specific injection: the affective signal provides enough guidance in the shallow layers to steer the overall emotional trajectory (control precision), while the deep layers remain free to optimize for acoustic quality using the backbone's full pretrained capabilities (audio quality).

Optimization Objective for Acoustic Synthesis

The acoustic synthesis module is trained using the standard autoregressive language modeling objective — Cross-Entropy loss on the next-token prediction task — but with only the adapter parameters being optimized (Equation 7):

$\mathcal{L}_{gen} = -\frac{1}{T_a}\sum_{t=1}^{T_a} \log p(a_t \mid a_{<t}, \mathcal{S}_{global}, C_{local})$

where $a_t$ is the ground-truth acoustic token at position $t$ , $a_{<t}$ are all previous tokens in the sequence, $\mathcal{S}_{global}$ conditions the model via cross-attention, and $C_{local}$ conditions the model via the residual injection described above.

What it computes: the standard negative log-likelihood objective for autoregressive sequence modeling. At each time step, the model produces a probability distribution over the $N$ -word vocabulary of the audio codec. The loss measures how much probability mass the model assigned to the correct token — the model is penalized for placing low probability on the ground-truth token. The loss is averaged over all $T_a$ time steps in the sequence.

Why this form: the critical design choice here is not the loss function (which is standard) but what is frozen versus trained. The paper explicitly states: "Keeping all parameters of the pre-trained autoregressive decoder frozen, we strictly optimize the adapter parameters $\phi$ and the gating scalars $\gamma$ ." This means that during training, backpropagation flows through the frozen decoder to compute gradients for the injected $\gamma \cdot C_{local, t}$ terms, but the decoder's own weights are never updated.

This has several practical advantages: (1) it prevents catastrophic forgetting of MusicGen's pretrained audio synthesis capabilities — the model never "unlearns" how to produce high-quality music because those weights never change; (2) it dramatically reduces training cost, since only the small adapter (likely a few hundred thousand parameters) needs optimizer states and gradient computation; (3) it means the adapter can be trained on relatively little paired video-music data (the paper uses approximately 1351 minutes of music data with emotion annotations) without overfitting, because the model's core generative capacity comes from the frozen backbone trained on much larger music datasets.

The inference process follows naturally: the trained adapter is used to guide the frozen backbone in generating discrete acoustic tokens autoregressively, and the final token sequence is decoded into a continuous waveform using the EnCodec decoder. The EnCodec decoder is a separately trained neural vocoder that converts the discrete token representation back into raw audio at the original sampling rate — this is a standard component in token-based audio generation and is not modified by NarraScore.

Scalable Long-Form Inference: The Sliding-Window Strategy

The training setup processes 30-second clips, but the system must generate coherent soundtracks for videos of arbitrary length — potentially several minutes or more. The paper's solution is an overlapping sliding-window inference strategy that coordinates the global and local control pathways across segment boundaries.

Global Semantic Reasoning Across the Full Video

Before any music is generated, the system performs a single global analysis pass over the entire video. The VLM's keyframe extraction capability compresses the long visual sequence into a sparse set of representative frames — not every frame, but frames that capture the major visual beats of the narrative. This compressed representation fits within the VLM's context window and enables holistic reasoning about the entire story arc. The output is a single $\mathcal{S}_{global}$ description that captures the overarching musical identity.

This global anchor is maintained as a stationary condition throughout all subsequent processing. Every sliding window, every generated segment, uses the same $\mathcal{S}_{global}$ as its text conditioning. This is what ensures that the music maintains a consistent genre, instrumentation, and atmosphere from the first second to the last — the global style is locked in before any local generation begins.

Continuous Affective Reasoning with Overlapping Windows

In contrast to the global analysis (which uses temporal compression), the extraction of local affective cues requires high temporal density — narrative tension evolves second-by-second and cannot be captured from sparse keyframes. The paper "explicitly eschews temporal compression strategies" for this pathway.

Instead, the video is processed in sequential windows that share a defined overlap (the overlap size is not specified in the paper, but the training data uses a 15-second overlap for 30-second clips, suggesting a 50% overlap ratio). Each window is independently processed through the VLM with the affective probe to produce a Valence-Arousal trajectory for that segment. The overlap region ensures that the prediction for the current window is contextually aligned with the preceding frames — the VLM sees the tail of the previous segment as context for the beginning of the current segment, preventing the emotion curve from having discontinuities at window boundaries.

The paper claims this "yields a continuous emotional curve despite the segmented processing." The mechanism is effectively an autoregressive continuation in emotion space: the VLM's internal state for a frame near the beginning of window $k$ is influenced by frames from the end of window $k-1$ (because they're included in the overlap), so the predicted VA values transition smoothly across the boundary.

Autoregressive Acoustic Continuation

The acoustic synthesis similarly operates within the sliding-window framework, with a specific mechanism for ensuring musical coherence at segment boundaries.

For each window, the system generates a segment of acoustic tokens conditioned on the stationary $\mathcal{S}_{global}$ (for global style) and the window's portion of the continuous $C_{local}$ signal (for local dynamics). To ensure seamless transitions between segments, the system uses the final sequence of acoustic tokens generated in the preceding window as the prompt prefix for the current window.

This is a standard technique in autoregressive generation: rather than starting each window's generation from scratch (which would produce independent musical fragments with no relationship to each other), the model is "prompted" with the ending of the previous segment. The autoregressive process naturally continues the musical phrase from where it left off — the rhythm continues, the harmony progresses logically, the melodic line extends. The paper explicitly claims this "guarantees seamless rhythmic continuity and acoustic causality throughout the long-form video."

The combination of three mechanisms — a stationary global anchor, overlapping emotional extraction, and token-level acoustic prompting — addresses the three failure modes of previous long-form approaches:

Style drift is prevented by the stationary $\mathcal{S}_{global}$ .
Emotional discontinuity at segment boundaries is prevented by the overlap in affective reasoning.
Acoustic seams at segment boundaries are prevented by the token-level prompt prefix.

The paper does not provide details on how window boundaries align with natural musical phrase boundaries or whether the overlap region in the audio synthesis is handled through crossfading (common in audio stitching) or purely through the autoregressive continuation. This is a notable gap — in practice, autoregressive continuation alone might not guarantee seamless transitions if the model's generated ending doesn't naturally lead to the next segment's beginning, and some form of overlap-add or crossfade might be needed in implementation.

Implementation Architecture Details

The paper specifies the concrete trainable components in Section 4.1 (Implementation Details), though these are part of the experimental setup rather than the methodology per se. The system integrates two trainable modules:

The Projector serves as the semantic interface between the VLM and the acoustic decoder for the global pathway. It accepts visual features $F_v \in \mathbb{R}^{T \times D_v}$ (extracted from the VLM, where $D_v$ is the visual embedding dimension of the VLM) and uses a two-layer MLP with GELU activation to map them to the acoustic feature space $\mathbb{R}^{T \times D_a}$ , where $D_a$ is the acoustic decoder's hidden dimension. A dropout rate of 0.1 is applied during this projection for regularization. This projector is trained first for 150 epochs to align static semantic features across modalities.

The Temporal Adapter models long-term affective dependencies for the local pathway. It uses a dilated convolution layer on the sequence dimension (expanding the temporal receptive field while maintaining the channel dimension $D_a$ ) followed by LeakyReLU activation and a linear layer to produce the final condition embeddings $C \in \mathbb{R}^{T \times D_a}$ . This adapter is fine-tuned for 50 epochs after the Projector is trained.

The two-stage training strategy (Projector first for 150 epochs, then Adapter for 50 epochs) is designed for stability: the Projector learns the static cross-modal mapping (what visual features correspond to what acoustic features in a time-averaged sense) before the Adapter introduces temporal dynamics. This prevents the temporal modeling from interfering with the initial alignment learning.

Summary of Key Design Choices and Their Justifications

Frozen VLM with lightweight probe over training a dedicated emotion recognizer: sidesteps the data bottleneck for continuous affect annotation by leveraging the VLM's pretrained semantic understanding; the probe needs only enough data to learn the readout mapping, not enough to learn visual understanding from scratch.
Semantically-Anchored Temporal Alignment over learned temporal embeddings: leverages the VLM's text understanding for temporal reasoning, conforms to the model's native interleaved format, and avoids training temporal parameters that might not generalize to unseen sequence lengths.
Instruction-Driven Semantic Steering over architectural modifications: repurposes the VLM's existing instruction-following capability rather than adding external control modules; modulates attention patterns through natural language rather than through hard architectural constraints.
Spatial Average Pooling over attention-based aggregation: discards spatial information that is irrelevant for global narrative emotion while being computationally trivial; the VLM's individual patch tokens encode fine-grained spatial relationships that are important for object recognition but noise for holistic affect prediction.
Hybrid L1+L2 loss over pure MSE for affective probing: provides robustness against the inherent noise in continuous affect annotations while maintaining the convergence benefits of squared-error gradients.
Structured four-dimension schema over free-form captioning for global anchor: reduces output variance and ensures the text description is musically actionable (containing genre, instrumentation, atmosphere, and pacing) rather than being a general visual description that might lack musical relevance.
Stationary global anchor across sliding windows: prevents style drift by locking in the musical identity before any generation begins, rather than re-computing style per segment (which would produce inconsistent identities).
Temporal Super-Resolution Adapter with dilated convolutions over simple interpolation: transforms sparse emotion cues into fluid musical evolution by smoothing jitter and expanding temporal context; dilated convolutions provide a large receptive field with few parameters.
Token-level additive bias in shallow layers only over full-layer injection or architectural cloning: modulates high-level structure (dynamics, tension) without interfering with deep-layer acoustic refinement, introduces negligible parameter overhead, and preserves the backbone's pretrained distribution through zero-initialization of gating scalars.
Autoregressive prompting across windows over independent segment generation: ensures rhythmic continuity and acoustic causality by using the previous segment's ending as the current segment's beginning, rather than generating disconnected fragments.
Two-stage training (Projector then Adapter) over joint training: decouples the static cross-modal alignment from the temporal dynamics modeling, preventing the temporal adapter from interfering with the initial semantic mapping.

4. Key Insights and Innovations

Innovation 1: Emotion as a High-Density Compression of Narrative Logic

The paper's most fundamental intellectual move is reframing what the video-to-music generation problem actually is. The dominant assumption in prior work — spanning from early rule-based MIDI mapping through contemporary frame-level attention methods — is that the task is fundamentally about visual-to-audio feature translation: you extract visual features, map them to acoustic features, and generate music that matches. This framing treats the problem as perceptual correspondence. A fast-moving scene gets fast music. A dark scene gets low-register music. A face gets... well, the model guesses, usually badly.

NarraScore argues this framing is wrong at its foundation. The real problem is not perceptual correspondence but narrative interpretation. What matters for professional scoring is not what objects appear in the frame but what those objects mean in context — a smile during a wedding toast versus a smile during a villain's monologue demand completely different music, yet any perceptual system sees the same face. The paper's core diagnostic insight is that emotion is a sufficient statistic for the narrative information that matters for music generation. If you can accurately extract the Valence-Arousal trajectory of a scene — how positive/negative it feels and how calm/excited it feels, moment by moment — you have compressed the relevant narrative logic into a form that directly maps to musical dynamics.

This is not incremental. Prior emotion-driven approaches (Video2Music, EMSYNC, M2UGen) treated emotion as a feature to add — a discrete classification label or a text description that supplements visual features. NarraScore treats emotion as the primary control signal, with visual features serving only as raw material for extracting that signal. The shift from "extract visual features and also try to guess the emotion" to "use visual processing solely to extract emotion, then generate music from emotion" is a fundamental re-conceptualization, analogous to the shift in representation learning from manual feature engineering to learned end-to-end representations. The paper's ablation (Table 4) showing that the Narrative-Aware Affective Reasoning (NAR) module improves performance even when added to state-of-the-art VLMs confirms this is not just a different implementation of the same idea — it is a different idea about what the input representation should be.

The paper validates this claim through its strongest empirical result: the widening performance gap on long-form videos (Tables 2 and 3). Short clips can often be adequately scored by perceptual matching because the mood is static and the visual surface correlates with the desired music. But as duration increases, the correlation between perceptual features and narrative emotion breaks down — action scenes give way to dialogue, tension builds without visual change, subtext emerges without surface indicators. The fact that baselines degrade on long-form content while NarraScore maintains quality is direct evidence that the emotion-as-compression framing captures something that perceptual feature matching does not.

Innovation 2: Frozen VLMs as General-Purpose Narrative Sensors

The paper's second conceptual contribution is demonstrating that frozen Vision-Language Models — models trained for general video understanding, question answering, and instruction following — contain actionable narrative understanding that can be extracted through lightweight probing without any fine-tuning of the backbone. This is not obvious. VLMs are not trained for emotion recognition; their training data consists of video-caption pairs, question-answer data, and instruction-following examples. There is no explicit "emotion prediction" objective in VideoLlama-3's pretraining. The paper's claim is that narrative reasoning emerges as a byproduct of general video-language understanding, and that this implicit capability can be surfaced through a simple probing approach.

This framing is significant because it reconceptualizes what VLMs are. The standard view of VLMs in the video-to-music literature is as feature extractors or caption generators — you use them to describe what's in the video, and that description conditions the music generator (the Caption2Music baseline embodies this assumption). But NarraScore uses the VLM as something closer to a cognitive model — a system whose internal representations encode not just visual content but narrative interpretation, emotional subtext, and dramatic tension. The probing approach (training only a small MLP head on the frozen VLM's hidden states) is the methodological realization of this view: you don't ask the VLM to output emotions (which it wasn't trained to do and might do poorly), you read emotions from its internal processing.

The evidence for this claim comes from the affective probe's performance: a simple MLP trained on only ~884 minutes of annotated video data (a tiny dataset by modern standards) achieves sufficient prediction quality to drive state-of-the-art music generation. This would be impossible if the VLM were providing only surface-level visual features — the probe would need much more data to learn the mapping from raw pixels to narrative emotion. The fact that it works with such limited supervision implies that the VLM is providing representations that are already organized along affective dimensions, and the probe merely learns to project them onto the standard Valence-Arousal coordinate system.

The paper's ablation (Table 4) provides an intriguing nuance: when using lightweight VLMs, the Holistic Musical Conceptualization (HMC) module is necessary for good performance, but when using advanced VLMs (specifically Gemini 2.5, cited in the ablation), HMC appears to constrain rather than enhance performance. This suggests a capability threshold: below a certain level of visual reasoning sophistication, explicit high-level guidance helps; above that threshold, the VLM already possesses sufficient cross-modal reasoning to directly infer appropriate musical characteristics. This is a diagnostic finding with implications beyond this paper — it suggests that as VLMs improve, the optimal architecture for downstream tasks shifts from explicit conditioning to implicit probing.

Innovation 3: Dual-Branch Dissociation of Global Style and Local Tension

The paper's third conceptual contribution is the explicit architectural decomposition of the video-to-music conditioning problem into orthogonal global and local control pathways — and, more importantly, the claim that these pathways should be implemented through different conditioning mechanisms at different architectural depths because they serve fundamentally different functions.

Prior work tends to collapse all conditioning into a single mechanism: dense frame-level features are projected into a conditioning vector that feeds into the music generator through cross-attention or concatenation. This treats all visual information as homogeneous — a frame from a car chase and a frame from a quiet conversation are the same kind of thing, just with different values. NarraScore argues this homogeneity is the root cause of both style drift and narrative insensitivity. When every frame contributes equally to the conditioning signal, the generator has no structural incentive to maintain a consistent musical identity (because the conditioning changes with every frame) and no way to distinguish between moment-to-moment visual variation (which should drive local dynamics) and scene-level narrative shifts (which should drive structural musical changes).

The solution — separate global semantic anchoring (via text description through cross-attention at all decoder layers) and local affective modulation (via additive bias in shallow layers only) — is more than an engineering convenience. It encodes a specific hypothesis about how music generation transformers process hierarchical information: shallow layers determine high-level trajectory (dynamics, tension shape, structural form), while deep layers refine acoustic detail (timbre, harmonic voice-leading, local texture). By injecting the local affective signal only into shallow layers, NarraScore claims it can modulate the emotional arc without disrupting the deep layers' ability to produce high-quality audio. The paper cites prior work on hierarchical information distribution in music transformers (Musicongen by Lan et al., 2024) as partial motivation, but the application of this insight to the video-conditioning problem — specifically, the claim that global and local signals belong at different depths — is novel.

The evidence for this hypothesis is somewhat indirect: the paper reports qualitatively superior spectrograms (Figure 4) showing "clear fundamental frequencies alongside vertical onset markers" versus the "smooth horizontal energy bands" of Caption2Music or the "fragmented energy distributions" of GVMGen. The subjective evaluation (Tables 2 and 3) shows the largest advantage in Emotional Dynamic Consistency and Long-term Coherence — precisely the dimensions that the dual-branch design targets. However, the paper does not report an explicit ablation comparing shallow-only injection to full-depth injection, which would be the clearest test of the hierarchy hypothesis. This remains a plausible but not definitively validated architectural claim.

Innovation 4: Parameter-Efficient Narrative Control as an Alternative to Architectural Cloning

The paper's fourth contribution is methodological: demonstrating that fine-grained, temporally dense control of a frozen audio generation model can be achieved through additive bias injection with minimal trainable parameters, rather than the architectural cloning paradigm that dominates controllable generation.

The dominant approach in controllable diffusion models — exemplified by ControlNet (Zhang et al., 2023b, which NarraScore cites) — is to create a trainable copy of the backbone's architecture, run the control signal through it, and add the outputs to the frozen backbone's activations. This preserves the backbone's capabilities but essentially doubles the parameter count and computational cost during training. NarraScore's alternative is far more minimalist: train only a small temporal super-resolution adapter (dilated convolutions on the 2D Valence-Arousal signal) plus zero-initialized gating scalars for the shallow decoder layers. The control signal itself — a projected version of the VA trajectory — is added directly as bias without passing through a cloned architecture.

The significance of this contribution is not the specific parameter count (the paper does not report exact numbers, claiming only "negligible parameter overhead") but the demonstration that continuous affective control does not require complex conditioning architectures. The paper's implicit argument is that if the control signal is semantically aligned with the generation task — if emotion is indeed a high-density compression of the narrative information that matters for music — then even a simple additive injection suffices to steer the generator. The prior paradigm of heavy architectural cloning was, in this view, compensating for poor alignment between control signals and generation targets: you needed a complex adapter because you were trying to map surface-level visual features (object identities, motion vectors) into musical properties, which requires learning a complex, non-linear transformation. By switching to an emotion-based control signal that is intrinsically aligned with musical dynamics, NarraScore eliminates the need for that complex transformation.

The paper's training setup supports this interpretation: keeping the entire MusicGen-Small backbone frozen (no fine-tuning of any decoder parameters) while training only the external adapter means the model cannot learn new mappings from visual features to audio — it can only learn to let the affective signal modulate its existing generation capabilities. The fact that this minimal setup achieves state-of-the-art results (Tables 1–3) suggests the control signal itself — the continuous VA trajectory — carries sufficient information, and the frozen backbone already knows how to translate emotional dynamics into musical dynamics without additional training.

A practical implication that the paper does not fully articulate: this parameter-efficient approach makes NarraScore compatible with API-based models where backbone weights are inaccessible or too expensive to fine-tune. If a music generation service exposes hidden states for injection (or if the additive bias can be applied through a limited API), NarraScore could operate without ever accessing the backbone weights — only the lightweight adapter would need to be trained and deployed. This contrasts with ControlNet-style approaches that require full access to the backbone architecture.

5. Experimental Analysis

Evaluation Methodology

Datasets. The paper uses two publicly available benchmark datasets, one for video-to-emotion prediction and one for emotion-to-music generation. For video-to-emotion, NarraScore employs a continuous movie dataset with frame-level Valence-Arousal annotations capturing viewer-induced emotion (the emotional responses elicited in audiences during film watching), as opposed to datasets that annotate actors' expressed facial emotions. This alignment with induced rather than expressed affect is essential because background music must match the narrative atmosphere perceived by viewers, not individual characters' facial expressions. The effective dataset size after preprocessing is approximately 884 minutes. For emotion-to-music generation, the paper utilizes a music emotion dataset annotated for dynamic affective content, comprising excerpts and full songs with dense per-second valence-arousal labels. After preprocessing — which includes vocal suppression via Demucs (source separation to isolate instrumental background), segmentation into 30-second clips with 15-second overlap, and filtering of clips where silence exceeds 40% — the effective size is approximately 1351 minutes. The paper does not name either dataset explicitly, which is a notable gap in reproducibility. While the continuous movie dataset is described in terms that suggest it is designed for induced emotion prediction in narrative video (distinguishing it from face-centric benchmarks like AFEW-VA and Aff-Wild2), the absence of explicit dataset names, sizes in number of examples, and train/val/test split ratios makes it impossible to assess whether the evaluation covers sufficient diversity or to replicate the exact data configuration.
Base Models. NarraScore integrates two frozen pretrained models. The visual-narrative backbone is VideoLlama-3 (Zhang et al., 2025), a frontier multimodal foundation model for image and video understanding with instruction-following capabilities. The acoustic decoder is MusicGen-Small (Copet et al., 2023), a 300M-parameter autoregressive transformer-based music generation model that operates on discrete audio tokens from the EnCodec neural audio codec. The paper argues that MusicGen-Small was chosen because it is "representative" and because its text-conditioning cross-attention mechanism provides a natural pathway for the global semantic anchor. However, no experiments are conducted with MusicGen-Medium or MusicGen-Large, which would test whether the adapter-based approach scales with backbone capacity. For the ablation study (Table 4), the paper additionally tests lightweight VLMs and Gemini 2.5 (Comanici et al., 2025) as alternative visual backbones to assess the framework's adaptability across different reasoning capabilities, though VideoLlama-3 remains the primary configuration.
Metrics. The paper employs multiple complementary evaluation dimensions. For objective evaluation (Table 1), the metrics include: Fréchet Audio Distance (FAD) to measure distributional discrepancy between generated and ground-truth audio in a learned acoustic feature space; Fréchet Distance (FD) computed on audio features following the standard GAN evaluation protocol; Kullback-Leibler Divergence (KLD) to assess distribution matching; Density and Coverage metrics to quantify the diversity and fidelity of generated samples; and the ImageBind score (Girdhar et al., 2023) to measure cross-modal semantic consistency between video and generated audio. The paper acknowledges that ImageBind "may not fully capture the temporal progression and the intricate narrative flow inherent in video content" — an important caveat given that temporal dynamics are the paper's central concern. For subjective evaluation (Tables 2 and 3), a user study with 10 participants assessed five dimensions on a rating scale: Emotional Dynamic Consistency (EDC) — how well the music's emotional evolution matches the video's narrative arc; Global Style Matching (GSM) — how well the overall musical genre and atmosphere fit the video; Long-term Coherence (LTC) — whether the music maintains consistent identity without fragmentation; Music Quality (MQ) — the standalone acoustic quality of the generated audio; and Overall Preference (OP) — a holistic quality judgment. The user study size (n=10) is small, which limits statistical power and means individual participant variance could substantially influence the reported rankings, though this sample size is not uncommon in generative music evaluation where expert listeners are scarce.
Baselines. Five representative approaches are benchmarked (Section 4.1, Tables 1-3). M2UGen (Liu et al., 2023) and Video2Music (Kang et al., 2024) serve as foundational multimodal frameworks that represent established benchmarks in the field. VidMuse (Tian et al., 2025) and GVMGen (Zuo et al., 2025) represent the latest state-of-the-art in synchronized video-music synthesis, with VidMuse specifically designed for long-form generation via sliding-window inference. Caption2Music is a custom two-stage pipeline baseline constructed by the authors: VideoLlama3-2B generates detailed visual captions from the video, and these captions condition MusicGen for audio synthesis. This baseline is important because it directly tests the paper's claim that the hierarchical affective injection strategy outperforms a naive combination of state-of-the-art vision and audio models — if Caption2Music performed comparably, it would undermine the necessity of the continuous VA trajectory and dual-branch design. Notably absent from the baselines are: (1) a version of NarraScore using ground-truth VA annotations (to establish an upper bound on what perfect emotion extraction could achieve); (2) a variant where the VLM is fine-tuned rather than frozen (to test whether the probing approach actually preserves valuable pretrained knowledge); (3) any method that uses a dedicated emotion recognition model paired with MusicGen (to test whether the VLM probing is necessary or whether a simpler emotion estimator would suffice). The absence of the first two baselines is particularly significant because they would help disentangle the contributions of the VLM-as-sensor insight from the dual-branch injection architecture.
Generation Budget and Compute Accounting. The paper does not define a unified generation budget in the way typical inference-scaling papers do. There are no measurements of FLOPs, inference time, or number of generation steps — the standard metrics for comparing computational efficiency across methods. The paper claims that NarraScore introduces "negligible parameter overhead" and that the lightweight adapter design makes it computationally efficient compared to architectural cloning, but no quantitative evidence supports this claim. No table compares training time, inference latency, or memory usage across baselines. This is a significant omission for a paper that explicitly positions its minimalist design as an advantage over "heavy architectural cloning." The only efficiency-related specification is that the temporal super-resolution adapter uses dilated convolutions rather than dense attention, and that injection is restricted to shallow layers, but the actual computational savings are never measured. For the sliding-window long-form inference, the paper does not report the effective throughput (seconds of video processed per second of compute), the latency for generating a minute of audio, or how these scale with video length — all of which would be relevant for assessing the practical deployability the paper claims to enable.
Cross-Validation and Statistical Protocol. The paper reports no cross-validation, no error bars, no confidence intervals, and no statistical significance tests for any of its quantitative results (Tables 1-4). The subjective evaluation with 10 participants reports only mean scores without standard deviations, making it impossible to assess whether the reported differences between methods are statistically meaningful or within the noise of inter-rater variability. The user study protocol — how participants were recruited, whether they had musical training, what reference videos were used, whether the evaluation was blind to model identity, and how many video-music pairs each participant rated — is not described. For an evaluation that claims state-of-the-art performance, the absence of any statistical rigor weakens the empirical contribution substantially. The ablation study (Table 4) reports performance changes when components are removed, but without variance estimates, one cannot distinguish genuine effects from random fluctuation — especially given that the test set size is not reported (the paper specifies that the video dataset is ~884 minutes and the music dataset is ~1351 minutes, but the held-out evaluation split is never defined).

Main Quantitative Results

The paper's quantitative evaluation is organized around three tables and one figure, with different but overlapping baselines across the tables. I organize the results by the evaluation dimension rather than by table, since Tables 1-3 cover distinct aspects (objective metrics, subjective long-form, subjective short/mid-length) of the same comparisons.

Objective Generation Quality (Table 1)

Table 1 reports five computational metrics (FAD, FD, KLD, Density, Coverage) and one semantic alignment metric (ImageBind score) across NarraScore and four baselines. M2UGen is excluded from the table without explanation.

The headline result is that NarraScore achieves the best FAD (lower is better, indicating the generated audio distribution is closest to the ground-truth distribution in the Fréchet Audio Distance feature space) and the best ImageBind score (higher is better, indicating stronger video-audio semantic alignment). The paper presents these as evidence that NarraScore produces higher-fidelity audio that better matches the visual content.

However, several patterns in Table 1 warrant scrutiny. First, the performance differences on FD and KLD are small — in some cases, baselines that perform poorly on FAD and ImageBind actually achieve better FD or KLD scores, suggesting the metrics may be capturing different aspects of quality that don't consistently align. The paper does not discuss these inconsistencies. Second, Density and Coverage metrics show mixed results across methods, with no clear winner — NarraScore is not consistently best on these diversity metrics, which might indicate that the strong affective control actually reduces output diversity (the model learns to produce a narrower but more "correct" distribution). Third, and most critically, Table 1 evaluates short-form generation (30-second clips, matching the training segment length), which is precisely the regime where the paper acknowledges that baseline methods are competitive. The paper's core claim is that its advantages emerge and widen on long-form content, but Table 1 cannot demonstrate this because the objective metrics are only computed on short segments. The FAD/FD metrics require comparing distribution statistics against a reference set, and it's unclear whether that reference set contains long-form audio or only short clips — if the latter, the metrics fundamentally cannot assess the narrative coherence properties that the paper claims as its primary contribution.

A specific concern: the ImageBind score measures static semantic alignment — essentially, whether the generated audio embedding is close to the video embedding in ImageBind's joint space. As the paper itself notes in Section 4.2, ImageBind "may not fully capture the temporal progression and the intricate narrative flow inherent in video content." Yet this is the only cross-modal metric reported, and it is computed on short segments where temporal progression is minimal. The paper's central innovation — continuous emotion-driven temporal dynamics — is evaluated by a metric that the paper acknowledges cannot capture temporal dynamics. This disconnect between the paper's claims and its quantitative evaluation is a significant empirical gap.

Subjective Evaluation on Long-Form Videos (Table 2)

Table 2 reports mean opinion scores from the 10-participant user study on five dimensions (EDC, GSM, LTC, MQ, OP) for NarraScore and four baselines (Caption2Music, GVMGen, VidMuse, Video2Music). M2UGen is again excluded. The paper does not report the length of the videos used in this evaluation beyond describing them as "long-form," nor does it specify how many videos were evaluated or whether the same videos were used across all methods.

NarraScore achieves the highest score on all five dimensions, with the largest margin on Emotional Dynamic Consistency (EDC) — the dimension most directly related to the paper's core contribution of narrative emotion tracking. The paper interprets this as evidence that the continuous VA trajectory and dual-branch injection successfully capture and express evolving affective content. The second-largest margin is on Long-term Coherence (LTC), which the paper attributes to the global semantic anchor preventing the style drift that baselines suffer from in extended sequences.

The baseline analysis reveals patterns consistent with the paper's critique of prior work. Caption2Music — the two-stage pipeline that represents the naive VLM+MusicGen combination — performs worst on visual-audio correspondence dimensions (EDC, GSM), which the paper attributes to an "information bottleneck" where text captions filter out the temporal and dynamic cues needed for precise audiovisual alignment. GVMGen and VidMuse, despite being the most recent state-of-the-art methods designed for long-form generation, show particular weakness on Long-term Coherence, with the paper reporting that user feedback indicates these methods "frequently fail to sustain a stable musical trajectory, manifesting as narrative drift, inconsistent motifs, or disjointed segmentation." Video2Music receives relatively high Music Quality scores but low narrative alignment scores, which the paper characterizes as an "acoustic fidelity versus narrative alignment" tradeoff — the MIDI-based approach produces clean audio but "generic" music "weakly coupled to the visual storyline."

The paper claims that the performance gap between NarraScore and baselines widens in the long-form evaluation compared to the short/mid-length evaluation (Table 3). This is the central empirical claim of the paper: that the advantage of hierarchical affective control grows with sequence duration because the baseline methods' narrative blindness becomes increasingly costly as the story evolves. However, without statistical significance tests or standard deviations, the magnitude of this widening cannot be assessed — it could be a genuine effect or an artifact of the small participant sample.

Subjective Evaluation on Short- and Mid-Length Videos (Table 3)

Table 3 reports the same five subjective dimensions on short- and mid-length videos, with VidMuse, GVMGen, and Video2Music as baselines (Caption2Music is excluded). The paper notes that "baselines remain competitive in short- and mid-length scenarios due to limited temporal horizons," consistent with the argument that perceptual feature matching suffices for static moods.

NarraScore still achieves the highest Overall Preference (OP) in the short/mid-length evaluation, but the margins are narrower than in the long-form evaluation (Table 2). The paper does not report exact margin differences — the comparison between Tables 2 and 3 is qualitative rather than quantitative, with the claim that "a widening margin in the long-form evaluation" validates the approach's handling of extended temporal dependencies. This claim would be substantially strengthened by a direct statistical comparison showing the interaction between method and video length.

A notable pattern: Video2Music performs disproportionately well on short videos relative to long videos, which the paper attributes to MIDI-based methods being adequate for simple static-mood scenarios but failing when narrative complexity increases. GVMGen shows a similar degradation pattern, consistent with the paper's argument that dense frame-level attention (GVMGen's core mechanism) dilutes over long sequences.

Qualitative Spectrogram Analysis (Figure 4)

Figure 4 presents Mel-spectrogram visualizations comparing NarraScore's generated audio against four baselines (Caption2Music, GVMGen, Video2Music, VidMuse) alongside the corresponding Valence-Arousal emotion curves. This is the only qualitative evidence in the paper.

The paper provides a detailed spectral analysis of each method's failure modes. Caption2Music produces "excessively smooth horizontal energy bands with a notable absence of vertical transients" — interpreted as spectral stagnation indicating lack of rhythmic pulses and dynamic fluctuations. GVMGen exhibits "significant discontinuities in its spectral manifold characterized by fragmented energy distributions and abrupt temporal breaks" — interpreted as failure to maintain long-term acoustic coherence. Video2Music produces an "overly monotonic and mechanical" spectrum with "absence of structured harmonic variations and distinct beat markers" — interpreted as narrative flatness where the music doesn't respond to visual climaxes. VidMuse, despite producing content with "high spectral density," is "plagued by a high concentration of stochastic noise in the high-frequency regions above 8000Hz" — interpreted as reliance on literal object-level associations rather than narrative synthesis.

NarraScore's spectrogram is characterized as demonstrating "superior spectral hierarchy that balances harmonic stability with rhythmic precision," with "clear fundamental frequencies alongside vertical onset markers" indicating successful "synchronization of discrete rhythmic events with the narrative arc." The paper does not specify which video was used for this qualitative analysis or whether the spectrograms are representative or cherry-picked. Without multiple examples or a systematic qualitative evaluation protocol, Figure 4 provides illustration rather than evidence — it shows what NarraScore's output looks like but doesn't establish that this appearance is typical or that the differences from baselines are consistent across videos. The emotion curves shown below the spectrograms appear to be the extracted VA trajectories that condition the generation, but the paper doesn't specify whether these are predicted or ground-truth curves, nor does it quantitatively measure how well the generated music's acoustic features (spectral centroid, RMS energy, tempo variations) correlate with the conditioning VA trajectory — which would be a more rigorous test of narrative alignment than visual spectrogram inspection.

Ablation Study (Table 4)

Table 4 reports ablation results across three axes: the presence/absence of the Narrative-Aware Affective Reasoning (NAR) module; the presence/absence of the Holistic Musical Conceptualization (HMC) module; and the injection ratio into the acoustic decoder (percentage of shallow layers receiving the affective bias). Additionally, the ablation tests different VLM backbones: VideoLlama3-2B, VideoLlama3-7B, and Gemini 2.5. Results are reported on FAD and ImageBind score only.

The primary findings are:

NAR module contribution: Adding NAR consistently improves both FAD and ImageBind scores across all backbone configurations, including the strongest VLM (Gemini 2.5). This validates the paper's claim that explicit affective reasoning is beneficial even when the underlying VLM is highly capable — the probing approach extracts signal that the VLM's own generative outputs don't surface. The paper frames this as evidence that NAR is "model-agnostic" and complements rather than replaces the VLM's inherent reasoning.

HMC module and VLM capability interaction: This is the most interesting and non-obvious result in the ablation. When using lightweight VLMs (VideoLlama3-2B), removing HMC causes noticeable performance degradation — the model produces "literal descriptions of visual scenes without establishing meaningful connections to musical elements." However, when using the powerful Gemini 2.5 backbone, the pattern reverses: HMC constrains rather than enhances performance. The paper interprets this as a capability threshold effect: advanced VLMs "possess inherent cross-modal understanding abilities that allow them to directly infer appropriate musical characteristics from visual content without explicit high-level guidance." In other words, Gemini 2.5 doesn't need to be told to think about genre, instrumentation, atmosphere, and pacing — it does this implicitly. The structured four-dimension schema that helps weaker models becomes a restrictive bottleneck for stronger ones. This finding has broader implications for how VLMs should be integrated into generative pipelines: the optimal interface depends on the VLM's sophistication, and one-size-fits-all conditioning strategies may be suboptimal.

Injection ratio optimality: The paper sweeps different proportions of shallow decoder layers receiving the affective bias and reports that 75% injection ratio (the default in NarraScore) represents "an optimal balance." Both reducing and increasing this ratio degrades performance — too little injection fails to provide adequate semantic alignment (the narrative emotion signal doesn't reach enough layers to shape the generation), while too much injection "overwhelms the acoustic modeling capacity" of the decoder. This is the closest the paper comes to validating the shallow-layer injection hypothesis (that early layers control high-level structure while deep layers refine acoustic detail), but the ablation is limited: it varies the proportion of shallow layers receiving injection but never tests injection into deep layers (which would be the clearest test of the hierarchy hypothesis). If injection into deep layers also degraded performance, that would support the claim that shallow layers are special; the current ablation only shows that there's an optimal shallow-injection ratio, not that shallow injection is better than deep injection.

Critical Assessment

The paper's central empirical claims, as articulated in the executive summary and throughout the body, are: (1) NarraScore achieves state-of-the-art emotional dynamic consistency and narrative alignment; (2) the performance gap versus baselines widens on long-form sequences, validating the hierarchical affective control approach; (3) a frozen VLM can serve as an autonomous narrative sensor, extracting continuous emotion trajectories that drive music generation; and (4) a lightweight adapter with additive bias injection suffices for precise control, introducing negligible parameter overhead. The experiments provide partial support for these claims, but each has significant empirical gaps that warrant careful qualification.

Claim 1: State-of-the-art narrative alignment. The subjective evaluation (Tables 2 and 3) shows NarraScore receiving the highest mean scores on Emotional Dynamic Consistency and Overall Preference. This is positive evidence but is substantially weakened by the small participant sample (n=10), the absence of any statistical testing or variance reporting, and the lack of detail about the evaluation protocol. We cannot determine whether the reported differences — which appear modest in some dimensions — would replicate with a larger sample or different evaluators. Moreover, the study compares against only 4 baselines (M2UGen is excluded from the subjective evaluation entirely), and important comparison points are missing: a version of NarraScore with ground-truth VA annotations (which would measure how much the VLM probing limits performance), and a version with the VLM fine-tuned rather than frozen (which would test whether the probing approach is genuinely necessary or just convenient). Without these comparisons, "state-of-the-art" cannot be distinguished from "better than the specific baselines tested."

Claim 2: Performance gap widens on long-form videos. This is the paper's most distinctive empirical claim — that the hierarchical affective control approach is not just better in general but specifically better where prior methods fail most, and that the advantage grows with video length. The evidence is the comparison between Table 2 (long-form) and Table 3 (short/mid-length), where NarraScore's margins appear larger in the long-form evaluation. However, this comparison is entirely qualitative — the paper reports no statistical test for an interaction between method and video length, doesn't specify whether the same videos were used in both evaluations, and doesn't quantify the margin difference. A rigorous test would require the same videos evaluated at both short and long durations, with the same participants, using a within-subjects design analyzed via repeated-measures ANOVA or a similar framework. The current evidence is suggestive but not demonstrative.

Claim 3: Frozen VLM as autonomous narrative sensor. The primary evidence is that the affective probe, trained on a limited dataset (~884 minutes), produces emotion trajectories that — when injected into the decoder — yield subjectively better narrative alignment than baselines. The ablation shows NAR improves performance across VLM backbones. However, the paper never directly evaluates the quality of the emotion prediction itself. There is no reported correlation between predicted and ground-truth VA values on a held-out test set, no comparison of the probe's accuracy against existing emotion recognition baselines, and no analysis of where the probe fails (e.g., are prediction errors concentrated in certain emotional regions, certain video genres, or certain narrative structures?). This makes it impossible to assess whether the VLM is genuinely extracting narrative understanding or whether the probe is picking up on simpler visual correlates that happen to correlate with emotion in the training data. The paper's argument that frozen VLMs encode narrative reasoning is plausible but empirically underdetermined by the reported experiments.

Claim 4: Lightweight adapter with negligible overhead. This claim is asserted but never quantitatively substantiated. The paper provides no parameter counts comparing NarraScore's trainable components to ControlNet-style architectural cloning, no FLOPs measurements, no inference latency comparisons, and no memory usage profiling. The "negligible parameter overhead" claim is particularly suspect given that the temporal super-resolution adapter, while described as lightweight, involves dilated convolutions, a projection from 2D to the decoder's hidden dimension, and per-layer gating scalars — and no absolute numbers are provided. For a paper that positions architectural minimalism as a key contribution, the absence of any efficiency measurements is a significant gap.

Missing experiments that weaken the conclusions. Several experiments that are standard in the controllable generation literature and would substantially strengthen the paper are absent:

No correlation analysis between predicted VA and acoustic features of the generated music. The paper's central mechanism is that the VA trajectory controls the music's dynamic evolution. One would expect, for example, that generated music during high-arousal segments has higher RMS energy, faster tempo, and greater spectral flux than during low-arousal segments. Demonstrating these correlations — and showing they're stronger for NarraScore than for baselines — would provide direct evidence for the claimed control mechanism. The qualitative spectrograms (Figure 4) gesture in this direction but are insufficient.
No evaluation with ground-truth VA annotations. Using human-annotated VA curves (from the video-to-emotion dataset's test split) as the conditioning signal would establish an upper bound on the affective control pathway. If NarraScore with ground-truth VA performs substantially better than with predicted VA, that would quantify the cost of imperfect emotion extraction; if the difference is small, that would validate the VLM probing approach. Neither comparison is made.
No comparison with a dedicated emotion recognition model. The paper argues that off-the-shelf emotion recognizers are unsuitable because they're face-centric and lack narrative reasoning (Section 2.3). This argument would be strengthened by actually comparing against one — showing that, empirically, an existing state-of-the-art video emotion recognition model produces VA trajectories that lead to worse music generation than the VLM probing approach. Without this comparison, the critique of prior emotion recognition methods remains theoretical.
No evaluation on out-of-domain videos. All experiments use the same movie dataset (or at least the same domain) for training and evaluation. The paper's claim that NarraScore learns narrative understanding — not just dataset-specific visual-emotion correlations — would be tested by evaluating on videos from different genres (documentaries, animation, user-generated content, sports) where the visual features differ but the narrative logic is similar. No such generalization test is reported.
No ablation on the sliding-window overlap ratio for long-form inference. The paper describes the overlapping window strategy for long-form generation but provides no sensitivity analysis — does performance degrade if overlap is reduced? Is there a minimum overlap needed for seamless transitions? How does the generated soundtrack quality vary with window size? These practical parameters are left unexplored.
No analysis of the temporal super-resolution adapter's contribution in isolation. The adapter's dilated convolutions are supposed to smooth the VA trajectory and expand the temporal receptive field. An ablation comparing the full adapter against simple linear interpolation (without the learned convolutions) would quantify how much the learned temporal smoothing contributes versus the VA signal itself.

Conditional validity of the claims. The paper's results are conditioned on several factors that limit their generalizability:

Dataset specificity. Both training datasets are unnamed, and the video dataset is described only as a "continuous movie dataset designed for induced emotion prediction." Movie soundtracks have specific conventions — orchestral scoring, leitmotif structures, dramatic pacing — that may not transfer to other video domains. The paper's approach may work well for cinematic content and poorly for vlogs, tutorials, or news footage where the relationship between visuals and appropriate music is different.
MusicGen-Small only. All experiments use the smallest MusicGen variant (300M parameters). The paper's parameter-efficient design is partly motivated by the cost of fine-tuning larger backbones, but if the approach doesn't scale to MusicGen-Large (1.5B parameters), its practical value is limited — state-of-the-art audio quality typically requires larger models.
Single VLM family. The primary experiments use VideoLlama-3, and the ablation tests only two additional backbones (VideoLlama3-2B and Gemini 2.5). The approach's dependence on VLM quality — particularly the finding that HMC interacts with VLM capability — means results may not generalize to other VLM architectures with different training distributions or instruction-following behaviors.
Short training segments. Both datasets are segmented into 30-second clips for training. The model never sees or learns from continuous long-form audio during training — the long-form capability comes entirely from the sliding-window inference strategy. This means the model has no explicit training signal for cross-segment coherence, and the autoregressive continuation mechanism at inference time is relied upon to generalize beyond its training context. Whether this works reliably for videos of dramatically different lengths (1 minute vs. 10 minutes vs. 1 hour) is untested.

The evaluation's strongest elements. Despite these limitations, certain aspects of the evaluation are well-conceived. The Caption2Music baseline is a particularly strong control — it explicitly tests whether the paper's architectural innovations (continuous VA trajectories, dual-branch injection) outperform a straightforward application of the same underlying models (VideoLlama + MusicGen). The poor performance of Caption2Music, especially on narrative alignment dimensions, provides genuine evidence that something beyond caption-based conditioning is needed. The ablation finding that HMC helps weak VLMs but constrains strong VLMs is a non-obvious result with implications beyond this paper, and it suggests the authors are engaging seriously with their architecture's behavior rather than simply reporting favorable configurations. The qualitative spectrogram analysis, while insufficient as standalone evidence, provides a concrete, interpretable visualization of the failure modes the paper claims to address — the spectral stagnation of Caption2Music, the fragmentation of GVMGen, and the high-frequency noise of VidMuse are visually apparent and align with the paper's theoretical critiques.

Overall, the experimental evaluation demonstrates that NarraScore outperforms the tested baselines on subjective metrics in a human study with a small sample, that the NAR module contributes positively across VLM backbones, and that there exists an optimal injection ratio for the affective bias. Whether these results constitute "state-of-the-art" performance, whether the advantages genuinely stem from narrative understanding rather than dataset-specific correlations, and whether the approach offers computational efficiency advantages over alternatives remain open questions that the reported experiments do not resolve.

6. Limitations and Trade-offs

Unresolved Error Propagation in the Cascaded Architecture

The constraint. NarraScore is fundamentally a cascaded pipeline: the VLM produces emotion trajectories, and the acoustic decoder generates music conditioned on those trajectories. The paper explicitly acknowledges in Section 6 (Limitations) that "the cascaded design risks error propagation from upstream affective reasoning." Errors in the VA prediction — a falsely high arousal estimate during a quiet scene, a valence misreading that confuses menace for warmth — propagate directly into the generated music with no mechanism for downstream correction. The acoustic decoder has no access to raw video; it only sees the processed VA curve and the global text description. If the VLM probe produces a noisy or semantically wrong emotion trajectory, the music will faithfully follow that wrong trajectory.

The consequence. This creates a brittle failure mode: the system's output quality is bounded by the worst-case performance of the affective probe, not its average performance. A 30-second segment where the VLM misreads the narrative tone (perhaps due to ambiguous visual cues, cultural specificity of emotional expression, or out-of-distribution cinematography) will produce music that is not just suboptimal but actively counterproductive — tense music during a calm scene, cheerful music during a tragic revelation. The paper provides no mechanism for detecting or recovering from such errors at generation time. This is particularly concerning for long-form content, where the probability of encountering at least one such failure increases with video length, potentially ruining an otherwise coherent soundtrack with a jarring mismatch.

What evidence exists. The paper reports no direct evaluation of the affective probe's prediction quality. There is no reported correlation between predicted and ground-truth VA values on a held-out test set, no confusion matrix or error distribution analysis, and no qualitative examples showing where the probe succeeds versus fails. Section 5 already noted this absence. The subjective evaluation (Tables 2 and 3) shows NarraScore scoring highest on Emotional Dynamic Consistency, but this is an aggregate measure — it doesn't reveal whether the system fails catastrophically on specific segments or degrades gracefully. The spectrogram analysis (Figure 4) shows a single presumably successful example. The paper's ablation (Table 4) demonstrates that the NAR module improves aggregate metrics when present versus absent, but doesn't characterize its failure modes when present.

Mitigation status. The paper acknowledges the limitation explicitly in Section 6 but proposes only future work: "end-to-end joint optimization to mitigate these dependencies." No architectural mechanism for error detection or recovery is built into the current system. The hierarchical design — where the global anchor is fixed and the local trajectory is injected as bias — means there is no feedback loop where the generated audio's characteristics could signal that the emotional conditioning might be wrong and trigger re-extraction or fallback behavior. The system operates open-loop from pixels to music.

Difficulty Estimation Cost Is Unaccounted For and Potentially Dominant

The constraint. The paper's entire approach depends on running every video frame through a frozen but computationally expensive Vision-Language Model (VideoLlama-3) to extract the continuous VA trajectory and global semantic description. This extraction happens before any music is generated and must process the full video (for global reasoning via keyframe extraction, Section 3.6) plus every frame at 1 Hz (for local affective reasoning). The paper makes no attempt to measure, report, or account for this pre-processing cost in any of its efficiency claims. There is no comparison of total wall-clock time, FLOPs, or memory usage between NarraScore and any baseline.

The consequence. The paper's central architectural claim — that the lightweight adapter introduces "negligible parameter overhead" and that the minimalist design is computationally efficient — is fundamentally incomplete. The VLM forward pass likely dominates the total computational budget. VideoLlama-3, even in its smaller configurations, is a multi-billion-parameter model performing full transformer inference over hundreds of frames. For a 5-minute video at 1 Hz, this means 300 forward passes through a large VLM, plus the cost of keyframe extraction and holistic reasoning over the compressed sequence. This cost may exceed the music generation cost itself — MusicGen-Small is only 300M parameters and generates tokens autoregressively, while VideoLlama-3 is orders of magnitude larger and processes all frames. The "parameter-efficient" adapter design saves training parameters but does nothing to reduce the dominant inference-time cost.

In practical deployment terms, this means NarraScore may be slower and more expensive than baselines that use lightweight visual encoders (like the CLIP or ViT features used by Video2Music, GVMGen, and VidMuse), even if those baselines require more adapter parameters. The paper's silence on inference latency makes it impossible for a practitioner to assess whether the narrative alignment improvements justify the computational cost. If generating a 5-minute soundtrack requires 10 minutes of VLM preprocessing before audio generation begins, the system may be unusable for interactive or time-sensitive applications regardless of output quality.

What evidence exists. None. The paper provides no FLOPs counts, no latency measurements, no GPU memory profiling, and no throughput numbers for any component. Section 4.1 (Implementation Details) specifies the architecture but reports no computational metrics. The paper claims "negligible computational overhead" in the abstract and "negligible parameter overhead" in Section 3.5, but these claims are entirely unsubstantiated. The comparison tables (Tables 1-3) report only quality metrics with no efficiency dimension. No baseline comparison includes inference time.

Mitigation status. None in the current paper. The paper mentions in Section 6 that future work will "explore knowledge distillation to reduce the computational latency of the visual backbone," acknowledging the problem implicitly. But the current system provides no mechanism for trading off VLM computation against generation quality — no smaller VLM variant is tested as the primary configuration, no frame-rate subsampling below 1 Hz is evaluated, and no caching or incremental processing strategy is proposed for the sliding-window inference.

Single-Domain Evaluation with No Generalisation Evidence

The constraint. All experiments in the paper use a single unnamed "continuous movie dataset designed for induced emotion prediction" for video content and a single unnamed "music emotion dataset" for audio generation training. The paper provides no evaluation on videos from other domains, genres, or production contexts. The test set is drawn from the same distribution as the training data — same dataset, same cinematic conventions, same visual language, same musical scoring traditions. The paper's claims about narrative understanding and emotional reasoning are not tested against videos that differ in visual style, cultural context, editing conventions, or intended audience.

The consequence. The paper's central argument — that NarraScore captures deep narrative logic rather than surface-level visual-emotion correlations — cannot be distinguished from the simpler explanation that it learns dataset-specific statistical associations. Movie datasets have strong correlations between visual features and music: dark lighting correlates with minor keys, fast cutting correlates with high tempo, close-ups of faces correlate with certain emotional registers. These correlations are reliable within the cinematic domain but may not reflect genuine narrative understanding. A model exploiting these correlations would perform well on in-distribution test data while failing on content where the same visual features carry different narrative meanings — animation where color palettes don't follow live-action conventions, documentary footage where editing rhythms serve informational rather than emotional purposes, user-generated content where cinematography is amateur and unpredictable, or films from different cultural traditions with different emotional expression norms.

This domain specificity is particularly concerning given the paper's specific claims about induced versus expressed affect (Section 2.3). The paper argues that movie datasets capture viewer-perceived emotion rather than actor-expressed emotion, making them appropriate for soundtrack generation. But if the model has only seen one movie dataset, it cannot have learned the general mapping from cinematic conventions to viewer affect — it has learned the specific mapping present in that particular dataset's selection of films, which may reflect Western Hollywood conventions, specific genres, or particular annotator populations.

What evidence exists. The paper describes its video dataset only as a "continuous movie dataset designed for induced emotion prediction" (Section 4.1) without naming it, specifying its size in terms of number of distinct videos, describing its genre distribution, or reporting its country/language of origin. No zero-shot or few-shot evaluation on out-of-domain videos is conducted. The paper reports no analysis of performance variation across video genres, emotional categories, or narrative structures within the test set — such an analysis would at least characterize the domain's internal diversity. The generalization question is simply not addressed.

Mitigation status. None. The paper does not acknowledge domain specificity as a limitation, does not test on multiple datasets, and does not frame the results as domain-conditional. The claims in the abstract and introduction are phrased in general terms ("long-form video soundtrack generation," "fully autonomous paradigm") without qualification about the cinematic domain.

The Revision / Correct-to-Incorrect Regression Problem

The assumption. The paper assumes that the extracted Valence-Arousal trajectory, once upsampled and injected into the decoder, will produce appropriate musical dynamics — that the mapping from VA space to musical features (tempo, harmonic tension, instrumentation density, dynamic range) is monotonic and well-behaved. More specifically, the system assumes that when the VLM probe detects rising tension (increasing arousal), injecting that signal as additive bias into shallow decoder layers will produce music with appropriately increasing intensity.

The consequence. There is a fundamental risk that the additive bias mechanism can produce musically incoherent output when the VA trajectory contains rapid fluctuations, noise, or physiologically implausible transitions. The Valence-Arousal circumplex model assumes smooth emotional trajectories in human experience — emotions don't jump instantaneously from "ecstatic" to "terrified" without transition. But the VLM probe, operating on 1 Hz frame samples without explicit temporal smoothness constraints beyond what the training data provides, may produce trajectories with unrealistic discontinuities — especially at scene cuts, during rapid editing, or when visual cues are ambiguous. The temporal super-resolution adapter ( $\mathcal{F}_{\phi}$ , Equation 5) applies dilated convolutions for smoothing, but this is a learned module trained on the same limited dataset; it may not generalize to out-of-distribution trajectory shapes.

When an implausible VA trajectory is injected, the decoder — which was pretrained on real music with natural dynamic evolution — may produce output that is technically responsive to the control signal but musically nonsensical: sudden unmotivated crescendos, abrupt instrumentation changes, tempo fluctuations that don't follow musical phrasing conventions. This is analogous to the "correct-to-incorrect reversion" problem documented in the revision model literature, where sequentially conditioned models can undo their own correct outputs because they've never seen correct examples in context. Here, the decoder has never been trained on trajectories with non-musical dynamics, so it may faithfully follow a bad trajectory into bad music.

What evidence exists. The paper provides no analysis of the extracted VA trajectories' properties — no measurement of temporal smoothness, no detection of physiologically implausible jumps, no comparison of trajectory statistics against natural emotion dynamics. The qualitative spectrogram (Figure 4) shows one apparently successful case. The subjective evaluation's Emotional Dynamic Consistency metric captures human perception of whether the music's emotional evolution matches the video — high scores suggest the trajectories are reasonable on average, but this doesn't rule out occasional catastrophic failures on specific segments. No per-segment or per-video analysis is reported that would reveal outlier failures.

Mitigation status. Partial. The temporal super-resolution adapter ( $\mathcal{F}_{\phi}$ ) includes dilated convolutions that the paper claims "smooth local jitter" (Section 3.5). However, this smoothing is learned rather than explicitly constrained — there is no regularization term enforcing trajectory smoothness in the affective probe's training objective (Equation 4, which uses only L2 and L1 regression losses without temporal consistency terms). The clipping of VA values to [-1, 1] prevents extreme outlier predictions but doesn't prevent rapid oscillations within that range. The overlap in sliding-window inference (Section 3.6) ensures continuity at window boundaries but doesn't constrain within-window trajectory quality.

Hard Problems: When the VLM Has No Narrative Signal to Extract

The assumption. The entire NarraScore framework rests on the assumption that the video content contains extractable narrative emotion — that there is a coherent Valence-Arousal trajectory to be found in the VLM's representations. This assumption holds for professionally produced cinematic content (movies, TV shows, commercials) where directors intentionally craft emotional arcs through cinematography, editing, performance, and music. But it breaks down for large categories of video content where narrative emotion is absent, ambiguous, or uncorrelated with visual features.

The consequence. The system has no graceful degradation mode for videos that lack clear narrative emotional structure. Consider these failure cases:

Atmospheric or ambient video (landscape footage, slow TV, meditation content): there may be no intended emotional trajectory — the video is designed to be neutral or steadily calm. The VLM probe, forced to produce a continuous VA output, may hallucinate emotional fluctuations from random visual variation (clouds passing, waves moving), causing the generated music to impose a false narrative onto content that was meant to be affectively neutral.
Instructional or informational video (tutorials, lectures, news reports): the emotional content is carried by speech and facial expression, not by visual narrative. The 1 Hz frame sampling may capture slides changing or presenters gesturing, which carry little emotional information. The VLM may have no meaningful signal to extract, producing a flat or random VA trajectory that generates inappropriate music.
Abstract or non-narrative video (music visualizations, art films, experimental video): the relationship between visuals and intended affect may be intentionally unconventional or non-existent. The VLM trained on narrative content may impose conventional emotional readings onto unconventional content.
Silent or text-heavy video (presentations with text overlays, surveillance footage): the visual stream contains essentially no emotional information at the frame level. The VLM probe has nothing to work with.

In all these cases, NarraScore's architecture provides no fallback — the system will always produce a VA trajectory (even if meaningless) and generate music that follows it. The user has no way to signal "this video doesn't need narrative scoring" or "use a static mood." The paper's framing assumes all video scoring is narrative scoring, which is true for the movie dataset but false for video in general.

What evidence exists. None. The paper evaluates only on a movie dataset where narrative emotion is present by construction. No evaluation on non-narrative video is conducted, and no analysis characterizes what the VLM probe outputs when fed content without clear emotional arcs. The paper doesn't discuss what "chance-level" or "null" VA extraction would look like or how the downstream music generator would behave when conditioned on such trajectories.

Mitigation status. None. The paper does not acknowledge this as a limitation. The architecture provides no mechanism for detecting when narrative emotional content is absent, for falling back to a static conditioning mode, or for estimating the confidence of the affective predictions. The system is designed for exactly one type of input (narrative video) and is evaluated only on that type. The "fully autonomous paradigm" claimed in the abstract implicitly assumes all videos have narratives to align with.

Latency-Liveness Tradeoff: Sequential Dependency Prevents Real-Time Generation

The constraint. NarraScore's architecture imposes a strict sequential dependency: the entire video must be processed through the VLM to extract the global semantic anchor (via holistic reasoning over keyframes) and the full affective trajectory before any music generation can begin. The paper explicitly states in Section 3.6 that "prior to sequential processing, we derive the global semantic anchor" through keyframe extraction that requires seeing the entire video. Even if the sliding-window approach processes windows sequentially, the global conditioning — which is essential for preventing style drift — depends on whole-video analysis.

The consequence. This makes NarraScore fundamentally unsuitable for any application requiring online or low-latency generation. Live video streams, real-time game audio, interactive installations, or any scenario where music must respond to video as it arrives cannot use this system. The global semantic anchor, which the paper identifies as critical for preventing style drift (and which the ablation suggests is necessary for weaker VLMs), requires knowing how the story ends before the music begins. A film composer watching a rough cut can score it because they see the whole film; they cannot score a live broadcast in real-time with the same narrative coherence.

Even in offline batch processing, the latency may be practically problematic. For a 10-minute video, the VLM preprocessing (300 frames × VLM forward pass + keyframe reasoning) must complete before the first second of audio is generated. If this preprocessing takes longer than the video duration itself, the total processing time exceeds real-time by a factor of two or more — acceptable for post-production but unusable for preview or iterative editing workflows where quick turnaround matters.

The sliding-window inference for music generation (Section 3.6) is also inherently sequential: each window's generation depends on the previous window's final tokens as a prompt prefix. This cannot be parallelized across windows, further constraining throughput.

What evidence exists. The paper provides no latency measurements, no throughput analysis, and no discussion of the online/offline distinction. The inference strategy is described in Section 3.6 as an overlapping sliding-window approach, which is inherently sequential. The architecture diagrams (Figures 2 and 3) show the VLM processing happening before the acoustic decoder, confirming the sequential dependency. The paper does not analyze whether the global anchor could be approximated incrementally (updating as more video is seen) or whether a lower-latency variant sacrificing some global coherence might be viable for online use.

Mitigation status. None. The paper does not acknowledge the offline-only nature of the system, does not discuss latency constraints, and does not propose any mechanism for incremental or streaming operation. The term "real-time" never appears in the paper. This is a fundamental architectural constraint that limits the deployability of the approach to post-production and batch processing scenarios — a qualification that should accompany any claim of practical applicability.

7. Implications and Future Directions

How This Work Changes the Landscape

NarraScore does not introduce a fundamentally new generative architecture — MusicGen remains MusicGen, and VideoLlama remains VideoLlama. What it introduces is a reframing of what the video-to-music conditioning signal should be, and this reframing has downstream consequences that may prove more durable than any particular architectural choice.

The dominant paradigm in video-to-music generation, from CMT through VidMuse, treats video as a sequence of perceptual features to be projected into the acoustic domain. Under this view, progress means better visual encoders, more sophisticated cross-modal adapters, and architectures that can handle longer sequences. NarraScore argues — through its architecture and through its strongest empirical result, the widening long-form performance gap in Tables 2 and 3 — that this view is fundamentally misaligned with the actual requirements of professional scoring. The relevant information in a video is not what objects appear in each frame but how the narrative feels as it unfolds. The paper's demonstration that a 2D Valence-Arousal trajectory — just two numbers per second — can drive state-of-the-art musical narrative alignment when injected as additive bias challenges the assumption that dense, high-dimensional visual features are necessary.

This is a diagnostic reframing rather than a paradigm shift. The paper's lasting contribution is not "use VLM probes for everything" but rather the specific claim that many of the difficulties in long-form video-to-music generation are actually difficulties in narrative understanding, not difficulties in cross-modal mapping. If this claim is correct, it redirects research attention: the bottleneck is not the adapter architecture (the paper shows a minimal additive bias suffices) but the quality of narrative extraction (where the paper provides only indirect evidence that the VLM probe works). Future work that takes this diagnostic seriously should invest heavily in measuring and improving narrative understanding capabilities, treating them as first-class objects of study rather than as a preprocessing step to be assumed.

The paper also changes the landscape by providing concrete reconciliation of conflicting findings in the emotion-driven generation literature. Prior work split into two camps: those using discrete emotion classification (Video2Music, EMSYNC) because continuous annotation is too expensive, and those using LLM-based captioning (M2UGen, FilmComposer, JenBridge) because classification is too coarse. The discrete-classification camp captured some temporal structure but at coarse granularity. The captioning camp captured richer semantics but lost temporal structure entirely. NarraScore's position — that frozen VLMs can produce continuous, temporally dense emotion trajectories if probed rather than prompted — synthesizes these approaches: it captures the semantic richness of captioning-based methods (the VLM understands narrative subtext) with the temporal density of classification-based methods (a prediction at every frame), without requiring the expensive continuous annotation that made fine-grained temporal control previously infeasible at scale. This reconciliation explains why both prior camps produced unsatisfying long-form results — each was missing a different half of the solution — and provides a template for how VLMs can bridge the gap between semantic understanding and temporal control in other multimodal generation tasks.

The ablation result showing that the Holistic Musical Conceptualization (HMC) module helps lightweight VLMs but constrains Gemini 2.5 (Table 4) has implications beyond this paper. It suggests a capability-dependent interface design principle: the optimal way to extract structured control signals from VLMs depends on the VLM's sophistication. Below a certain reasoning threshold, you need explicit schemas and constraints to get usable outputs; above that threshold, the constraints become bottlenecks. This is not specific to music generation — it applies to any pipeline that uses VLMs as components. Current practice typically uses the same prompting strategy regardless of the underlying model's capability (everyone writes the same kind of system prompt for GPT-4 as for smaller models). NarraScore's ablation suggests this is suboptimal — that interface design should be adapted to model capability, and that as VLMs improve, architectures should shift from explicit conditioning structures toward more implicit probing. This finding is methodologically significant and, unlike the paper's more specific architectural claims, is backed by a clean ablation result.

The paper also makes architectural cloning less attractive as a default strategy for controllable generation, though this contribution is more suggestive than definitive given the missing efficiency measurements. The demonstration that additive bias injection in shallow layers — a simpler mechanism than the encoder-copy approach of ControlNet — suffices for temporally dense control of a frozen backbone suggests that complex adapter architectures may be compensating for poor alignment between control signals and generation targets rather than being inherently necessary. If corroborated by the missing FLOPs/latency comparisons, this would redirect adapter research from designing more expressive conditioning architectures toward designing better-aligned control signals.

Follow-Up Research This Work Enables

Direct measurement of VLM narrative understanding through probing benchmark creation. The paper asserts that frozen VLMs encode narrative emotion in their hidden states, but provides no direct evidence — no correlation between predicted and ground-truth VA, no analysis of where predictions fail. A natural follow-up constructs a benchmark specifically designed to test this claim: take annotated continuous affect datasets (LIRIS-ACCEDE, the unnamed movie dataset, and ideally a newly collected multi-genre dataset covering documentaries, animation, and user-generated content), probe multiple VLM architectures (VideoLlama, Gemini, LLaVA, GPT-4V if accessible) using the paper's MLP head approach, and report per-frame VA prediction accuracy as a function of VLM size, training data, and video characteristics. The key metric is not just aggregate correlation but conditional error analysis: do prediction failures concentrate at scene boundaries? In ambiguous emotional contexts? In specific genres? For non-Western content? This would transform the paper's plausible-but-unverified claim into an empirically characterized capability. A strong negative result — finding that VLM probes achieve high accuracy only on cinematic content with conventional emotional signaling — would bound the approach's applicability and redirect effort toward domain-specific fine-tuning or hybrid architectures.

Comparison of VLM probing against best-in-class dedicated emotion recognition under controlled conditions. The paper critiques off-the-shelf emotion recognizers as face-centric and lacking narrative reasoning (Section 2.3) but never empirically compares against them. A rigorous head-to-head evaluation would train NarraScore's full pipeline with three alternative emotion extraction methods: (a) the VLM probing approach, (b) a state-of-the-art video emotion recognition model (e.g., a modern temporal CNN or transformer trained on Aff-Wild2 and fine-tuned on the target movie dataset), and (c) ground-truth human annotations as an upper bound. All three would feed the same MusicGen adapter pipeline, isolating the emotion extraction quality as the independent variable. The evaluation would report both VA prediction accuracy on held-out annotations and downstream music generation quality (subjective EDC scores). This experiment answers a specific, falsifiable question: does the VLM's semantic reasoning actually produce better emotion trajectories for music generation than a dedicated recognizer, or is the VLM simply a convenient way to avoid training a task-specific model? If the dedicated recognizer performs comparably or better, the paper's central narrative — that VLMs are uniquely suited to narrative emotion extraction — collapses, even if the adapter architecture remains useful.

End-to-end fine-tuning of the VLM for narrative affect regression and its effect on downstream generation. The paper freezes the VLM and trains only an MLP probe, arguing that this preserves the VLM's general reasoning capabilities. But the probe can only read out representations the VLM already produces — it cannot teach the VLM to attend to narrative cues it currently ignores. A natural extension fine-tunes the VLM itself (or a low-rank adapter like LoRA applied to the VLM) on the continuous affect regression objective, then compares the resulting VA trajectories and generated music against the frozen-probe baseline. The hypothesis is that task-specific fine-tuning will improve VA prediction accuracy (especially on out-of-domain content where the VLM's general representations may not align with narrative affect) but may degrade the VLM's general reasoning capabilities that the Holistic Musical Conceptualization module depends on. The experiment would measure both VA prediction accuracy and music generation quality as a function of fine-tuning degree, identifying a potential accuracy-versus-generality tradeoff. A finding that even modest VLM fine-tuning substantially improves narrative alignment on held-out genres would challenge the paper's commitment to frozen backbones and suggest that the probing approach leaves significant performance on the table.

Stress-testing the approach on deliberately non-narrative and cross-cultural video content. The paper's evaluation is confined to a single (unnamed) movie dataset from what is likely a Western cinematic tradition. Two stress tests would characterize the approach's boundaries. First, evaluate on systematically varied non-narrative content: landscape/ambient footage (where the correct musical response is often static or cyclical, not narratively evolving), instructional video (where speech carries emotional content and 1 Hz frame sampling may miss it), and abstract/experimental video (where intended affect may be intentionally ambiguous or unconventional). The key measurement is not just mean quality scores but the variance and failure characteristics — does the system impose false narratives on content that shouldn't have them? Second, evaluate on films from different cultural traditions (Bollywood, Nollywood, Japanese anime, Korean drama) where emotional signaling conventions, editing rhythms, and cinematographic language differ from Hollywood norms. The paper's VLM (VideoLlama-3) was trained predominantly on Western video content, and its narrative understanding may not transfer. A finding of significant cross-cultural performance degradation would bound the "autonomous" claim and motivate culture-specific fine-tuning or explicit cultural context conditioning.

Quantitative validation of the affective control mechanism through acoustic feature correlation analysis. The paper's central mechanistic claim is that the injected VA trajectory controls the music's dynamic evolution, but it never demonstrates this quantitatively — the spectrogram analysis (Figure 4) is qualitative and cherry-picked. A rigorous validation would compute per-second acoustic features of the generated music (spectral centroid, RMS energy, spectral flux, tempo estimation, harmonic tension via dissonance measures, mode detection for major/minor) and correlate them with the conditioning VA trajectory. Specific predictions: arousal should positively correlate with RMS energy, spectral flux, and estimated tempo; valence should positively correlate with major mode probability and negatively with harmonic dissonance. These correlations should be computed within each generated track across time (measuring responsiveness), then averaged across the test set. The same analysis should be run for baseline methods to demonstrate that NarraScore's correlations are stronger (showing that the affective control mechanism is actually operating, not just present in the architecture). A null result — finding no significant within-track correlation between VA and acoustic features despite subjectively better ratings — would suggest the subjective improvement comes from the global semantic anchor or some other pathway, not from the token-level affective modulation the paper claims as its core innovation.

Latency-optimized variants for interactive and streaming applications. The paper's architecture is fundamentally offline: global semantic anchoring requires seeing the entire video, and sliding-window inference is sequential. A practical extension would develop and evaluate variants that relax these constraints for latency-sensitive use cases. One direction: replace whole-video keyframe extraction with incremental global style estimation — update the semantic anchor periodically as more video is observed, starting with a prior based on genre metadata (if available) or the first N seconds. Measure how quickly the estimated anchor converges to the full-video anchor and how much music quality degrades when using partial-video anchors. A second direction: parallelize the sliding-window music generation by using the previous window's planned ending as the prompt prefix for the next window (accepting some boundary discontinuity risk) rather than waiting for the previous window to fully generate. Measure the tradeoff between generation latency and boundary coherence. The goal is to characterize the Pareto frontier between latency and quality, making NarraScore's approach applicable to scenarios (preview rendering, iterative editing) where full offline processing is unacceptable.

Practical Applications and Downstream Use Cases

Automated post-production scoring for independent and low-budget film. This is the most direct application and the one the paper's architecture is explicitly designed for. Independent filmmakers, documentary producers, and content creators producing long-form narrative video typically cannot afford professional composers, yet their work demands exactly the kind of evolving, narratively responsive soundtrack that NarraScore targets. The system's offline processing model (full video analysis before generation) is compatible with post-production workflows where latency is not a constraint. The ~884-minute video dataset used for training, while not enormous, covers a meaningful range of cinematic content. The practical value proposition is straightforward: take a rough cut of a 20-minute short film, run it through NarraScore, and receive a continuous soundtrack with consistent musical identity and dynamic emotional responsiveness — output that the paper's subjective evaluation suggests is preferred over existing automated methods. The key deployment consideration is whether the VLM's narrative understanding generalizes beyond the training dataset's cinematic conventions — the cross-cultural stress test proposed above would directly inform which films this can serve.

Emotion-guided music editing and re-scoring tools. A less obvious but practically valuable application is interactive music editing where the user adjusts the extracted VA trajectory — not the music directly — and the system regenerates accordingly. A filmmaker could load a scene, view the automatically extracted emotion curve, and manually adjust it: "make this section more tense," "delay the emotional peak by 3 seconds," "this character's smile should read as menacing, not warm." Because NarraScore separates emotion extraction from music generation, the VA trajectory becomes an editable intermediate representation that is semantically meaningful to non-musicians (unlike MIDI or audio waveforms) but sufficiently expressive to control the musical output. The practical workflow would involve: load video → auto-extract VA curve → user adjusts curve (drag control points, adjust intensity) → regenerate music with modified conditioning → iterate. This puts narrative-aware scoring within reach of editors who understand story but not music theory, bridging a gap that currently requires either accepting automated output as-is or hiring a composer for revisions.

Data augmentation for music emotion recognition and generation research. The paper's pipeline can be run in reverse for research purposes: given a large corpus of videos without affect annotations (feature films, YouTube content, archival footage), use the VLM probe to generate pseudo-ground-truth VA trajectories, then use these trajectories to train or augment music generation models. The paper demonstrates that even with only ~884 minutes of annotated video, the probe produces trajectories sufficient for state-of-the-art generation — suggesting the pseudo-labels may be good enough for self-training or distillation. A concrete research application: train a lightweight video-to-VA model (a small CNN or transformer, much cheaper than the full VLM) using the VLM probe's outputs as training targets, then deploy this lightweight model in latency-sensitive settings where the full VLM is impractical. This is essentially the "knowledge distillation to reduce the computational latency of the visual backbone" that the paper mentions as future work (Section 6), but framed as an immediate practical application enabled by the current system's demonstrated capability.

When to Prefer This Method

The paper does not provide explicit decision criteria comparing NarraScore against named alternatives in different deployment contexts — there is no "use NarraScore for X, use VidMuse for Y" guidance, and the missing efficiency measurements make cost-based comparisons impossible. The paper positions NarraScore as a general improvement over existing approaches within the domain of long-form video soundtrack generation, supported by higher subjective scores across all tested dimensions (Tables 2 and 3) and all video lengths. Without the paper articulating specific tradeoffs or conditions under which alternatives would be preferable, a forced decision matrix would invent distinctions the paper does not support. The practical considerations discussed in the applications and limitations sections — the offline processing requirement, the unmeasured VLM computational cost, the single-domain training — imply rather than state deployment constraints, and converting these implications into explicit "prefer A when..." rules would overclaim beyond what the experimental evidence can justify.