ArXiv: 2511.07299

🎯 Pitch

VADER reveals that explicitly modeling how objects interact across time—not just what they look like—is key to explaining surveillance anomalies. By training a compact encoder to distinguish normal from anomalous interaction transitions, the system generates causally grounded descriptions that beat prior methods by substantial margins, even when the upstream anomaly detector makes mistakes.

1. Executive Summary

This paper introduces VADER, an LLM-driven framework for Video Anomaly unDErstanding that integrates keyframe-level object relation features with visual cues to produce causally grounded descriptions and reasoning about anomalous events in surveillance videos. The system couples a Context-AwarE Sampling (CAES) strategy — which adaptively selects pre-event, on-event, and post-event keyframes based on anomaly score gradients to capture causal narrative structure — with a COntrastive Relation Encoder (CORE) — a lightweight MLP trained with triplet loss to encode dynamic object-interaction transitions into compact relational tokens — and feeds both visual and relational representations into a frozen MLLM fine-tuned with LoRA. Across three benchmarks (CUVA, HIVAU-70k, HAWK), VADER achieves state-of-the-art results on anomaly description and question-answering tasks, including a 7.38-point MMEval improvement on CUVA's causation task and BLEU-1 gains of 0.054 for description and 0.165 for QA on HAWK over prior methods, establishing that modeling temporally resolved object interactions substantially improves causal anomaly comprehension even when upstream detection modules introduce error propagation and the framework remains biased toward high-motion events.

2. Context and Motivation

The Core Problem: Video Anomaly Detection Sees, But Doesn't Understand

The fundamental gap this paper addresses is the disconnect between detecting that something abnormal is happening in a video and explaining what happened, why it happened, and how the objects involved interacted to produce the anomaly. Traditional video anomaly detection (VAD) — the dominant paradigm in surveillance, traffic monitoring, and public safety applications — focuses on answering a binary question: is this frame or clip anomalous? Methods output temporal localization boundaries [13, 28, 29, 40, 46, 53, 54, 71] or per-frame anomaly scores [14, 35, 38, 72, 73] but produce no natural language explanation of the event.

This is a critical practical limitation because anomaly detection in real-world deployments is almost never the end goal. A surveillance system that flags "anomaly at timestamp 14:23:05" and a system that reports "a person is forcibly opening the rear door of a parked vehicle while a lookout stands at the street corner" place fundamentally different cognitive demands on the human operator. The former requires the operator to watch the footage, diagnose the situation, and infer intent — exactly the kind of high-latency, error-prone human judgment that automated surveillance is meant to reduce. The latter provides actionable intelligence directly.

Why This Matters: From Detection to Understanding

The shift from detection to understanding — what the field now calls Video Anomaly Understanding (VAU) — matters for several reasons the paper either states explicitly or implies through its experimental design:

Decision-making under time pressure. In security operations centers, traffic management systems, and emergency response coordination, human operators monitor dozens of video feeds simultaneously. A system that provides natural language descriptions ("fight breaks out between two individuals near the platform edge") enables faster triage and prioritization than one that merely highlights an anomalous time window, because language carries semantic content that anomaly scores do not — it tells the operator what category of event to expect before they look.

Causal reasoning for prevention. Understanding an anomaly means more than describing visual content; it means identifying the causal chain that produced the event. The paper's CUVA benchmark [11] explicitly tests for this: it evaluates whether models can answer what happened, why it happened, and how it unfolded. Detecting that a car collision occurred (VAD) versus understanding that the collision happened because a driver ran a red light (VAU) represent vastly different analytical capabilities. The latter enables root-cause analysis that can inform interventions — redesigning the intersection, adjusting signal timing, deploying enforcement — while the former only documents the incident.

Auditability and trust. Anomaly detection systems based on learned representations often function as black boxes. When a system flags an event as anomalous, there is no built-in mechanism to explain on what basis. A natural language description generated by the system serves as a form of self-explanation that human supervisors can verify against the video evidence, increasing trust and enabling graceful failure modes (e.g., "the system described a person running, but the video shows a person walking quickly — the description is partially wrong, so discount its severity assessment").

Bridging to the multimodal era. The emergence of powerful Multimodal Large Language Models (MLLMs) [3, 16, 21, 30, 31, 36, 61] has created a technological opportunity that didn't exist even a few years ago: the ability to jointly reason over visual content and natural language in a single model. This means VAU is not just a desirable capability — it is now technically feasible in a way that it wasn't when video understanding was built from separate detection, tracking, and captioning pipelines. The paper positions itself squarely in this emerging paradigm, leveraging MLLMs as the reasoning backbone rather than as post-hoc captioners bolted onto a classical VAD system.

Where Prior Approaches Fall Short

The paper identifies three specific categories of prior work, each with distinct limitations that VADER is designed to address:

Category 1: Detection-Only Methods (Traditional VAD)

Classical and deep learning-based VAD methods — whether unsupervised [13, 23, 37, 40], weakly supervised [35, 46, 72, 73], or reconstruction-based [14, 27, 48, 71] — share a common limitation: they produce scalar anomaly scores or bounding boxes but zero semantic interpretation. The paper groups these under the umbrella of "focusing solely on detecting and localizing anomalies" (Section 1, paragraph 1). The gap is not that these methods perform detection poorly — many achieve strong localization results on UCF-Crime and XD-Violence — but rather that their output format is fundamentally insufficient for the downstream tasks the paper cares about: description, explanation, and causal reasoning.

A deeper limitation the paper doesn't dwell on but which is implicit in its design: detection-only methods treat anomalies as deviations from a learned normality manifold, which means they are fundamentally appearance-based rather than relation-based. A person running against a crowd and a person running in an empty street may both be anomalous in different contexts, but they produce similar low-level motion signatures. Without modeling the relationship between the runner and other objects (the crowd, the street, nearby vehicles), the detector cannot distinguish between these scenarios. The paper's emphasis on relation modeling is a direct response to this weakness.

Category 2: MLLM-Based VAU Methods (With Structural Gaps)

The paper identifies several recent works that apply MLLMs to anomaly understanding, positioning them as the most direct baselines:

Holmes-VAU [67] uses an anomaly-focused temporal sampling strategy (ATS) to select keyframes for an MLLM, producing clip-level and event-level descriptions. The paper's critique is indirect but clear: Holmes-VAU focuses on which frames to select but does not model how objects within those frames interact. The sampling strategy is temporally aware but the per-frame analysis remains appearance-based.

CUVA [11] explicitly targets causation — asking why an anomaly occurred — but does so by prompting general-purpose MLLMs with video frames and task-specific instructions. The paper's critique is that this approach "often neglect[s] deeper causal relationships and dynamic object interactions" (Section 2.2). The model may describe what it sees in each frame, but without an explicit relational representation, it cannot easily track how a person's action at time t−1t-1 caused another person's reaction at time tt.

HAWK [47] introduces interaction-aware modeling, which is the closest prior work to VADER's stated goals. However, the paper distinguishes VADER's approach by emphasizing temporally resolved dynamics — HAWK models interactions, but VADER models how interactions change over time via the relational volatility curve and the CORE encoder. The distinction the paper draws (implicitly) is between static interaction detection ("person A is interacting with object B") and dynamic interaction trajectory encoding ("the interaction between person A and object B shifts from cooperative to aggressive between frames tt and t+1t+1").

VAD-LLaMA [34] and VERA [60] adapt MLLMs for explainable detection but remain detection-oriented at their core — they explain that an anomaly exists and provide a basic description, but don't structure the explanation around causal chains or relational dynamics.

The paper's unifying critique of all these prior MLLM-based approaches is stated in Section 1:

"these approaches often neglect deeper causal relationships and dynamic object interactions, which are critical for understanding unusual behaviors"

This is not a claim that prior work is wrong or ineffective — the quantitative comparisons in Section 4 show that VADER substantially outperforms these baselines, but the baselines themselves are non-trivial systems — but rather that prior work treats object relationships as either absent from the model's reasoning (relying solely on visual features) or as static properties of individual frames rather than as dynamic trajectories that encode the temporal structure of causal events.

Category 3: Scene Graph Methods for Video Understanding

The paper connects to the scene graph literature (Section 2.3), which uses structured representations — objects as nodes, relationships as edges — to capture interactions. Works like EGTR [17] extract relational information from transformer-based object detectors, and DecoAD [8] integrates scene and action features via knowledge graphs for anomaly detection. However, the paper identifies a key limitation: these approaches typically use pre-trained scene graph models that output discrete relationship labels ("holding," "riding," "standing on") drawn from fixed vocabularies (e.g., Visual Genome [18]).

These generic labels fail for anomaly understanding in two ways. First, anomaly-specific relationships are often not in the vocabulary — "attacking" or "fleeing from" may not be pre-defined relationship categories. Second, discrete labels lose the continuous relational information present in the intermediate features before the classification head. VADER's innovation is to extract the raw relational tensor before classification and encode its temporal dynamics into continuous tokens, avoiding the vocabulary bottleneck entirely.

A Deeper Underlying Gap: The Causality-Visibility Asymmetry

There is a subtle but important motivation the paper does not state explicitly but which is embedded in its architecture: many anomalous events have causal structures that span time intervals longer than the visually salient event itself. A physical altercation (the "event") may last 10 seconds, but the pre-event context — two individuals arguing, one shoving the other — spans the preceding 30 seconds, and the post-event context — one person fleeing, bystanders reacting — spans the following 20 seconds. A system that only sees the 10-second fight clip can describe the punches but cannot explain the escalation that caused them.

This is why the paper invests heavily in the CAES sampling strategy with its explicit pre-event, on-event, and post-event segmentation. It's not just about picking good frames — it's about providing the MLLM with the temporal narrative structure that causal reasoning requires. The anomaly score gradient (the slope of the score curve at the boundaries of the anomalous interval) serves as a signal for where the causal context begins and ends: the rising slope marks the escalation (pre-event), the peak marks the event, and the falling slope marks the resolution (post-event). This design choice reflects a specific view of what "causal understanding" means operationally: it means access to the temporal neighborhood of the event, not just the event itself.

How VADER Positions Itself

VADER is not positioned as a better anomaly detector or a better MLLM fine-tuning recipe — it is positioned as a framework for integrating three previously separate capabilities:

  1. Temporally adaptive sampling (CAES) that identifies not just anomalous frames but the causal context around them, using anomaly score dynamics rather than fixed windows.
  2. Dynamic relational encoding (CORE) that converts raw object relationship tensors into compact, temporally-aware tokens through weakly supervised contrastive learning on relational volatility peaks.
  3. End-to-end MLLM integration that fuses visual tokens, relational tokens, and task instructions so the LLM can reason over both appearance and interaction dynamics simultaneously.

The paper's contribution is the combination, not any single component in isolation. The claim is that prior work addressed at most two of these three aspects: Holmes-VAU has adaptive sampling but no relational modeling, HAWK has interaction modeling but (the paper implies) weaker temporal dynamics capture, and CUVA has causal task design but relies on general-purpose visual reasoning without structured relation representations. VADER's headline improvements — 7.38 MMEval points on CUVA causation, BLEU-1 gains of 0.054 (description) and 0.165 (QA) on HAWK — are attributed to this integration.

The paper also implicitly positions itself in a particular sub-regime of the VAU space: it is object-centric, meaning it models anomalies as arising from unusual configurations or trajectories of distinct objects and their pairwise interactions. This is a deliberate scope choice — the paper acknowledges (Section 5, Limitations) that this design limits its ability to handle scene-level anomalies (smoke, lighting changes) and group-level behaviors (crowd panics) that don't decompose neatly into pairwise object interactions. The choice reflects a tradeoff: object-centric modeling enables rich relational representations but at the cost of coverage. The paper does not claim universality; it claims superiority within the object-centric paradigm.

Finally, VADER is positioned as practical and retrainable, not as a one-off model. The anomaly scorer is pre-trained once on an aggregated dataset and applied across benchmarks, avoiding per-dataset retraining (Section 3.1.1). The CORE encoder is a lightweight MLP, and the MLLM is frozen with only projectors and LoRA adapters fine-tuned. This design philosophy — modular, frozen backbones, lightweight trainable components — makes VADER adaptable to new datasets and scenarios without requiring full retraining of large models, which the paper implicitly argues is a practical advantage over end-to-end MLLM fine-tuning approaches.

3. Technical Approach

3.1 Reader Orientation

VADER is a pipeline that takes a surveillance video as input and produces detailed, causally-grounded natural language descriptions and answers to questions about anomalous events — it is not just a detector that says "something weird happened here," but a system that explains what happened, why it happened, and how objects interacted to produce the anomaly. The problem it solves is the gap between low-level anomaly detection (per-frame scores, bounding boxes) and high-level semantic understanding (descriptions, causal explanations, interactive QA), and the shape of the solution is a modular framework that (1) adaptively selects a small set of narrative-relevant keyframes, (2) extracts how object relationships change over time from those frames, (3) encodes those dynamic interactions into compact tokens, and (4) feeds everything — visual features, relational tokens, and task instructions — into a frozen multimodal LLM fine-tuned with lightweight adapters.

3.2 Big-Picture Architecture (Diagram in Words)

The VADER pipeline has five major components connected in sequence, with a side branch for relational processing:

  1. Anomaly Scorer — a CLIP-based module that assigns a per-frame anomaly score to every frame in the input video. Its output is a sequence $S$ of scalar scores indicating how abnormal each frame is, plus predicted anomaly class labels that provide semantic hints.

  2. Context-AwarE Sampling (CAES) — takes the anomaly score sequence $S$ and produces a set of selected keyframes $K$ (typically 64 frames) by detecting anomalous intervals, expanding them into pre-event and post-event context segments based on score gradients, and sampling uniformly from each segment.

  3. Visual Encoder — a frozen vision encoder (from the backbone MLLM, NVILA) that converts each selected keyframe into visual tokens — compact feature representations that the LLM can process.

  4. Relation Branch (parallel side path):

    • Relation Feature Extractor — a pre-trained scene graph generation model (EGTR [17]) that takes each keyframe and outputs raw relational tensors: for each pair of detected objects, the intermediate features computed before the relationship classification head, capturing continuous relational dynamics rather than discrete labels.
    • Temporal Association — tracks object identities across frames using appearance and spatial overlap, then computes a relational volatility curve: for each adjacent frame pair, the maximum L2 norm of the difference between relational features for all co-tracked object pairs.
    • COntrastive Relation Encoder (CORE) — a lightweight MLP trained with triplet loss that takes relational-change pairs (before/after vectors from volatility peaks) and maps them to compact relation tokens that encode salient interaction transitions.

  5. Multimodal LLM (NVILA) — a frozen backbone that receives visual tokens from the encoder, relation tokens from CORE (both projected through learned adapters to match the LLM's input dimension), and text tokens from the task instruction. Only the projectors and LoRA adapters are fine-tuned; the LLM and vision encoder remain frozen. The LLM autoregressively generates the output description, answer, or causal explanation.

Information flow: Video → Anomaly Scorer → Score sequence $S$ → CAES → Keyframes $K$ → [Visual Encoder → Visual Tokens] + [Relation Extractor → Raw Relations → Temporal Association → Volatility Curve → CORE → Relation Tokens] → Concatenated Tokens → Frozen LLM with LoRA → Text Output.

3.3 Roadmap for the Deep Dive

  • First, the Anomaly Scorer (Section 3.1.1): how per-frame anomaly scores are computed using a CLIP-based normality prototype and semantic direction projection, because these scores drive all downstream frame selection.
  • Second, the CAES sampling strategy (Section 3.1.2): how the anomaly score sequence is converted into a fixed-size set of narrative keyframes using adaptive thresholding, gradient-based context expansion, and multi-segment uniform sampling, because this determines what visual information the LLM sees.
  • Third, the Relation Feature Extractor (Section 3.2.1): how object-level relational tensors are extracted from keyframes and how temporal association produces the relational volatility curve, because this is the raw signal that CORE distills.
  • Fourth, the CORE encoder (Section 3.2.2): how the volatility curve is mined for positive and negative relational-change pairs, how the triplet loss and semi-hard negative mining train the encoder, and what the output relation tokens represent, because this is VADER's core technical novelty.
  • Fifth, LLM integration and fine-tuning (Section 3.3): how visual tokens, relational tokens, and text instructions are concatenated and fed to the frozen LLM, what adapters are trained, and what training configuration is used, because this is how everything comes together at inference.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems integration paper whose core idea is that video anomaly understanding requires modeling temporally dynamic object relationships — not just static visual appearances or discrete interaction labels — and that these relational dynamics can be encoded into compact tokens via weakly supervised contrastive learning on a per-video relational volatility signal, then fed to a frozen MLLM to produce causally coherent descriptions.

Anomaly Scorer: Per-Frame Abnormality Estimation

What it is and why it exists. Before VADER can select which frames to analyze or extract relations from, it needs a signal indicating where in the video anomalous events occur and what type of anomaly each frame likely contains. The Anomaly Scorer provides this signal: it takes every frame of the input video and assigns a scalar anomaly score $S_i$ between 0 and 1, along with a predicted anomaly class. This module is pre-trained once on an aggregated dataset spanning diverse video scenarios, so it does not require per-dataset retraining — a deliberate design choice to keep the overall framework dataset-agnostic.

The scorer is inspired by the CLIP-based framework of Zanella et al. [64], which the authors adapt and simplify. The key insight is that CLIP's joint image-text embedding space encodes semantic information about normality and abnormality that can be surfaced without training a specialized anomaly classifier for each target dataset.

Step 1: Normality prototype construction. The first step is to define what "normal" looks like in CLIP's feature space. Given a training set of normal (non-anomalous) video frames $\mathcal{I}_\text{norm}$, the authors pass each frame through the CLIP image encoder $E_I$ and average the resulting feature vectors:

m=1∣Inorm∣∑x∈InormEI(x)\mathbf{m} = \frac{1}{|\mathcal{I}_\text{norm}|} \sum_{\mathbf{x} \in \mathcal{I}_\text{norm}} E_I(\mathbf{x})

where $\mathbf{m}$ is the normality prototype — a single vector in CLIP's embedding space representing the centroid of all normal frame features, $\mathcal{I}_\text{norm}$ is the set of all normal training frames, $\mathbf{x}$ is an individual frame, and $E_I(\mathbf{x})$ is the CLIP image encoder's output feature vector for that frame.

What it computes: the empirical mean of CLIP features over all normal frames. The result is a single vector $\mathbf{m}$ that sits at the center of mass of "normal behavior" in CLIP's latent space.

Why this form: by centering the feature space at the normality prototype, the authors create a coordinate system where distance from the origin naturally encodes abnormality — normal frames cluster near zero, while anomalous frames produce features farther from the origin. This is computationally cheaper than learning a separate normality model per dataset and leverages CLIP's pre-trained semantic understanding of what makes scenes typical versus unusual. An alternative would be to train a binary classifier, but that requires anomalous training examples and would lock the scorer to a specific dataset's anomaly distribution.

Step 2: Feature re-centering and semantic projection. After computing the normality prototype, every frame feature $E_I(\mathbf{x})$ — whether from normal or anomalous data — is re-centered by subtracting $\mathbf{m}$:

x~=EI(x)−m\tilde{\mathbf{x}} = E_I(\mathbf{x}) - \mathbf{m}

where $\tilde{\mathbf{x}}$ is the re-centered feature vector. This shifts the origin to the normality centroid, so $\tilde{\mathbf{x}} \approx \mathbf{0}$ for normal frames and $\tilde{\mathbf{x}}$ points in some semantic direction for anomalous frames.

The re-centered features are then projected onto class-specific semantic directions $\mathbf{d}_c$. These directions are obtained from CLIP's text encoder: for each anomaly class $c$ (e.g., "fighting," "car accident," "burglary"), a text prompt describing that class is encoded, and the resulting text feature vector serves as the semantic direction $\mathbf{d}_c$ for that class. The projection $\tilde{\mathbf{x}}^\top \mathbf{d}_c$ measures how well the frame's visual content aligns with the textual description of anomaly class $c$.

These projection scores are passed through a softmax to produce a conditional class distribution:

pc∣A(Ii)=exp⁡(x~i⊤dc/τ)∑c′exp⁡(x~i⊤dc′/τ)p_{c|A}(I_i) = \frac{\exp(\tilde{\mathbf{x}}_i^\top \mathbf{d}_c / \tau)}{\sum_{c'} \exp(\tilde{\mathbf{x}}_i^\top \mathbf{d}_{c'} / \tau)}

where $p_{c|A}(I_i)$ is the probability that frame $I_i$ belongs to anomaly class $c$, conditioned on the frame being anomalous, $\tilde{\mathbf{x}}_i$ is the re-centered feature for frame $I_i$, $\mathbf{d}_c$ is the text-encoded semantic direction for class $c$, and $\tau$ is a temperature parameter controlling the sharpness of the distribution.

What it computes: given that an anomaly is present, which specific type of anomaly is most consistent with the visual content? The softmax converts raw projection scores into a normalized probability distribution over anomaly classes. This provides semantic cues — if the model predicts "fighting" with high probability, the downstream LLM receives this information and can incorporate it into its reasoning.

Why this form: the conditional probability formulation $p_{c|A}$ separates the type of anomaly from the presence of an anomaly. This decomposition means the class predictor can be trained to discriminate between anomaly types without needing to model the base rate of normality — a simpler learning problem than joint classification of "normal vs. fighting vs. accident vs. ...". The alternative of training a flat classifier over all classes (including "normal") would require the model to simultaneously learn a normality decision boundary and between-class anomaly boundaries, which is harder with limited anomalous data.

Step 3: Per-frame anomaly probability. The unconditional probability that frame $I_i$ contains an anomaly, denoted $p_A(I_i)$, is estimated by a separate temporal module. The paper does not specify this module's architecture in detail — it states that it "captures short- and long-term dependencies across frame sequences" — but its role is clear: it takes a sequence of frame features and outputs a scalar probability that each frame deviates from normality, without committing to a specific anomaly class. This temporal modeling is essential because single-frame appearance alone is often ambiguous — a person running might be anomalous in a library but normal on a sports field, and the temporal context (preceding and following frames) disambiguates these cases.

Step 4: Joint probability and final anomaly score. The per-frame anomaly score is computed as the maximum over classes of the joint probability:

Si=max⁡c(pA(Ii)⋅pc∣A(Ii))S_i = \max_c \big(p_A(I_i) \cdot p_{c|A}(I_i)\big)

where $S_i$ is the anomaly score for frame $i$, $p_A(I_i)$ is the unconditional anomaly probability from the temporal module, and $p_{c|A}(I_i)$ is the conditional class probability from the semantic projection.

What it computes: the probability that frame $I_i$ is anomalous and belongs to the most likely anomaly class. The max operation selects the class that maximizes this joint probability, which serves as the frame's overall anomaly score. The predicted class $c^* = \arg\max_c p_A(I_i) \cdot p_{c|A}(I_i)$ is also recorded for downstream use by the LLM.

Why this form: the max over joint probabilities is a natural way to combine the anomaly presence signal with the anomaly type signal. If a frame has high $p_A$ (clearly anomalous) and high $p_{c|A}$ for some class (clearly a specific type), the joint probability is high and the score reflects strong confidence. If $p_A$ is high but $p_{c|A}$ is diffuse across many classes (the frame is anomalous but of an ambiguous type), the maximum joint probability is lower, reflecting uncertainty. The authors state that the predicted anomaly classes "provide semantic cues for downstream reasoning by LLMs" — the class label $c^*$ is converted to text and included in the prompt or metadata fed to the LLM, giving it a prior on what type of anomaly to expect before it processes the visual frames.

Context-AwarE Sampling (CAES): Narrative-Driven Keyframe Selection

What it is and why it exists. The Anomaly Scorer produces a score for every frame in the video, but modern MLLMs have limited context windows — feeding all frames would be computationally infeasible and would dilute the model's attention across irrelevant content. CAES addresses this by selecting a fixed-size set of keyframes (typically 64) that together tell the complete story of each anomalous event: what led up to it, what happened during it, and what followed. The key insight is that anomaly scores alone are insufficient for frame selection because they only indicate abnormality, not narrative structure — a frame with a high score might show the climax of an event, but excluding the preceding frames removes the causal context needed for understanding why the event occurred.

Formally, CAES is a function that maps the anomaly score sequence to a set of keyframes:

K=CAES(S)K = \text{CAES}(S)

where $K$ is the set of selected keyframe indices and $S = \{S_1, S_2, \ldots, S_T\}$ is the sequence of per-frame anomaly scores.

Step 1: Anomalous interval detection. CAES first identifies all temporal intervals in the video where anomalous activity occurs. For each video independently, the authors compute an adaptive threshold: the 97th percentile of that video's anomaly scores. Any continuous segment where the score exceeds this threshold is marked as an "on-event" (anomalous) interval. Using a percentile-based threshold rather than a fixed absolute value makes the detection adaptive to each video's score distribution — a video where all scores are low will have a lower absolute threshold than a video with generally elevated scores, preventing false negatives in the former case and false positives in the latter.

Step 2: Context expansion via gradient thresholds. For each detected anomalous interval, CAES expands it into three segments — pre-event (causal lead-up), on-event (the anomaly itself), and post-event (aftermath) — by examining the slope (gradient) of the anomaly score curve at the interval boundaries.

The authors compute the anomaly score gradient over a 5-frame window around each boundary. Two gradient thresholds are defined:

  • Rise threshold: the 95th percentile of the positive score slopes in the video. When the anomaly score is increasing faster than this threshold (moving leftward from the interval start), the system identifies the "rising slope" region that marks the pre-event escalation. The pre-event segment is defined as extending from the point where the slope first exceeds the rise threshold backward to the point where it drops below a second threshold (the "calm" level).
  • Calm threshold: the 85th percentile of the score slopes. When the anomaly score decreases below this threshold (moving rightward from the interval end), the system identifies the "falling slope" region that marks the post-event resolution. The post-event segment extends from the interval end forward to the point where the slope falls below the calm threshold.

The maximum context window on either side is capped at 30 frames to prevent unbounded expansion into normal content.

Why gradient-based expansion matters: using score values alone — e.g., "include all frames with score above X" — would miss the narrative structure. The key causal information is often in frames where the score is changing (ramping up or ramping down), not just where it's high. A fight starts with escalating tension before any punches are thrown — the score might be moderate but rising fast, and the gradient captures this escalation dynamic. Similarly, the aftermath of a car crash involves the score decreasing as the scene stabilizes, and including these frames helps the LLM understand consequences and resolutions. The gradient thresholds turn the continuous score curve into a three-act narrative segmentation: escalation (pre-event), climax (on-event), denouement (post-event).

Step 3: Uniform sampling from each segment. Once the pre-event, on-event, and post-event segments are defined for each anomalous interval, CAES samples keyframes uniformly from each segment. The paper specifies the sampling budget: 4 frames from pre-event context, 8 frames from the on-event interval, and 4 frames from post-event context per anomalous event. This 8:4:4 ratio reflects a design choice that the event itself deserves twice as many frames as its surrounding context — the climax carries more visual information than the lead-up or aftermath, but the context frames are still essential for causal reasoning.

Step 4: Budget management and multi-event handling. If a video contains multiple anomalous events, the total sampled frames may exceed the target budget (64 frames). In this case, frames with the highest anomaly scores across all events are prioritized — the system greedily selects the 64 frames with the largest $S_i$ values, ensuring that if budget is tight, the most clearly anomalous content is preserved.

Conversely, if fewer than 64 frames are selected from anomalous events (e.g., a video with one short anomaly), the remaining slots are filled by sampling uniformly from background segments — portions of the video not included in any detected anomalous interval. This ensures the LLM always receives exactly 64 frames (maintaining a consistent input format) and sees some normal context that helps it distinguish anomaly from baseline behavior.

Why 64 frames? The choice is primarily dictated by the context window of the backbone MLLM (NVILA). The paper does not ablate this number explicitly, but the 64-frame budget represents a balance: enough frames to cover multi-event videos with context, but small enough to fit within the LLM's input capacity while leaving room for relational tokens and text instructions.

Step 5: Dynamic window strategy (ablated in Table 7). The paper compares three strategies for defining context segment boundaries: (i) a fixed window that takes consecutive frames immediately adjacent to the anomalous interval, (ii) exponential interval sampling that spaces context frames at increasing intervals to capture multi-scale context, and (iii) the dynamic window based on gradient thresholds described above. The dynamic window achieves the best overall performance (BLEU 0.668, ROUGE 0.274, Consistency 0.343 in Table 7), which the authors attribute to its adaptive, event-specific context coverage — different anomalies have different escalation and resolution timescales, and the gradient-based method automatically adapts to the temporal dynamics of each specific event rather than applying a one-size-fits-all window.

Relation Feature Extractor: Raw Relational Signal Extraction

What it is and why it exists. CAES selects which frames to analyze, but VADER's core contribution is understanding how objects interact within those frames and how those interactions change over time. The Relation Feature Extractor provides the raw signal for this analysis. It takes each keyframe and, for every pair of detected objects, extracts a relational feature vector that encodes the nature of their interaction — not as a discrete label like "holding" or "standing on," but as a continuous vector in a learned relational embedding space.

The extractor uses a pre-trained DETR-based scene graph generation model, EGTR [17]. EGTR is a transformer-based object detector that has been extended to predict relationships between detected objects. During inference, EGTR performs three tasks: (1) detect objects with bounding boxes and class labels, (2) extract appearance embeddings for each object, and (3) compute pairwise relational features between all object pairs.

The critical design choice is where in EGTR the features are extracted: the authors take the intermediate features computed for each object pair before the relationship classification head. The relationship classification head is the final layer that maps these intermediate features to a probability distribution over a fixed vocabulary of relationship labels (e.g., "holding," "riding," "standing on," "wearing" — the standard Visual Genome [18] predicate set). By extracting features before this classification, VADER obtains:

  • Continuous relational information that is not quantized into a discrete vocabulary. Many anomaly-relevant relationships — "attacking," "fleeing from," "breaking into" — are not in standard scene graph vocabularies, but their relational signatures are present in the continuous feature space.
  • Full-dimensional feature vectors that can encode subtle gradations of interaction (e.g., the difference between "gently pushing" and "violently shoving" as a continuous shift in the relational embedding).
  • Features that can be compared across frames via vector distances, enabling the temporal volatility analysis described next.

For a frame with $N$ detected objects, the relational tensor has dimensions $N \times N \times d_r$, where $d_r$ is the dimensionality of the per-pair relational feature vector. Each entry $\mathbf{r}_{ij}$ represents the interaction from object $i$ to object $j$ (the relation is directed — "person A pushes person B" is different from "person B pushes person A").

Object association across frames. To track how relationships evolve over time, VADER needs to know which object in frame $t$ corresponds to which object in frame $t-1$. The system performs temporal object association using two cues:

  • Appearance similarity: the cosine similarity between the appearance embeddings (extracted by EGTR along with the relational features) of objects in consecutive frames, weighted at 0.8 in the matching score.
  • Spatial overlap: the Intersection-over-Union (IoU) of the bounding boxes in consecutive frames, weighted at 0.2.

The matching between the two sets of objects (objects in frame $t$ and objects in frame $t-1$) is solved by the Hungarian algorithm [19], which finds the optimal one-to-one assignment that maximizes total similarity. A maximum track age of 15 frames is enforced — if an object disappears and is not matched for more than 15 frames, its track is terminated, and a new object ID is assigned if it reappears later.

Relational volatility curve. Once objects are tracked across frames, the system computes a single scalar for each pair of consecutive frames that quantifies how much the object relationships changed:

Volatility(t)=max⁡(i,j)∥rij(t)−rij(t−1)∥2\text{Volatility}(t) = \max_{(i, j)} \left\| \mathbf{r}_{ij}(t) - \mathbf{r}_{ij}(t-1) \right\|_2

where $\text{Volatility}(t)$ is the relational volatility at the boundary between frames $t-1$ and $t$, $\mathbf{r}_{ij}(t)$ is the relational feature vector from object $i$ to object $j$ in frame $t$, $\mathbf{r}_{ij}(t-1)$ is the corresponding feature vector in the previous frame, $\|\cdot\|_2$ is the Euclidean (L2) norm, and the max is taken over all object pairs $(i, j)$ that are co-tracked (present in both frames with consistent identities).

What it computes: for every adjacent frame pair, the largest change in any pairwise object relationship. A high volatility value at time $t$ means that at least one pair of objects experienced a dramatic shift in how they are interacting between frames $t-1$ and $t$ — for example, the relationship between person A and person B transitions from "standing apart" to "physical contact" instantaneously. A low volatility value means all relationships are stable or changing gradually.

Why max rather than mean: the max operator ensures that a single dramatic interaction change — one person suddenly attacking another in an otherwise calm scene — is not averaged out by many stable relationships (bystanders standing still, vehicles parked). The anomaly is typically driven by a specific interaction change, not by global scene change, and the max captures this targeted signal. An alternative like the mean would dilute the signal from the anomalous interaction with noise from irrelevant stable relationships.

Why L2 distance on the raw relational features: the L2 distance in the continuous relational embedding space captures semantic shifts in the relationship that might not correspond to a change in the discrete relationship label. For instance, two people talking at a normal distance versus two people talking face-to-face with aggressive postures might both be classified as "talking to" by the relationship classifier, but their relational feature vectors will differ significantly due to changes in relative pose, proximity, and attention direction encoded in the intermediate features. The L2 distance surfaces these semantically meaningful but label-invisible shifts.

Gaussian smoothing. The raw volatility curve can be noisy due to object detection failures, tracking errors, or minor frame-to-frame jitter. To make peak detection robust, the volatility sequence is smoothed with a Gaussian filter with standard deviation $\sigma = 2.0$. This value was selected through ablation (Table 9 in the supplementary material): $\sigma = 2.0$ with a top-5% peak threshold achieved 72.12% AUC, outperforming both $\sigma = 1.0$ (70.18%) and $\sigma = 3.0$ (69.25%). The authors attribute this to $\sigma = 2.0$ providing improved noise filtering without excessive smoothing that would obscure salient events.

COntrastive Relation Encoder (CORE): From Raw Volatility to Relational Tokens

What it is and why it exists. The relation feature extractor produces raw relational feature vectors and a volatility curve, but these are not directly consumable by an LLM: the raw features are high-dimensional, per-frame, and per-object-pair, making them too voluminous and unstructured for the LLM's input; and the volatility curve alone is a 1D signal that says when relationships changed but not how or between which objects. CORE addresses this by learning to map salient relational transitions — the before/after feature pairs from volatility peaks — into a small set of compact, fixed-dimensional relation tokens that summarize the key interaction dynamics of the video and can be concatenated with visual tokens as input to the LLM.

CORE is a lightweight two-layer MLP trained with a triplet margin loss in a weakly supervised manner — "weakly supervised" because it uses only video-level anomaly labels (is this video anomalous or normal?) to mine positive and negative relational-change pairs, without frame-level or pair-level annotations of which interactions are anomalous.

Step 1: Mining positive and negative relational-change pairs. Given a smoothed volatility curve for a video, CORE selects training samples as follows:

  • Positive samples (anomalous relational changes): for videos labeled as anomalous, the top $k\%$ of smoothed volatility peaks are identified. For each selected peak at time $t$, the relational-change pair $[\mathbf{r}_{ij}(t-1); \mathbf{r}_{ij}(t)]$ is extracted, where $(i, j)$ is the object pair that produced the maximum L2 difference at that timestep (the argmax pair from the volatility computation). This pair encodes "the relationship just before the dramatic change" and "the relationship just after it," both as raw EGTR feature vectors. The concatenation $[\mathbf{r}_{ij}(t-1); \mathbf{r}_{ij}(t)]$ is the input to CORE. The top-$k\%$ threshold is set to $k = 5\%$ based on the ablation in Table 9, which showed 5% yields the best AUC (72.12%) compared to 3% (70.85%) and 7% (71.64%). The authors argue that 3% is too selective (too few positives for effective training) while 7% includes noisy, less informative peaks.

  • Negative samples: two types of negatives are collected: (i) volatility valleys from anomalous videos — relational-change pairs from timesteps with the lowest smoothed volatility values, representing stable, non-anomalous interaction periods; (ii) all samples from normal videos — relational-change pairs extracted uniformly from videos with no anomalous events, representing background interaction dynamics. This dual-negative strategy ensures the encoder learns to distinguish anomalous interaction changes from both stable periods within anomalous videos and all interactions in completely normal videos.

Why top-k peaks rather than all peaks: not every sharp change in object relationships is anomalous. A person who was sitting standing up produces a relational volatility spike (the "sitting on chair" relationship abruptly ends), but this is normal behavior. By restricting to the top 5% of peaks in anomalous videos, the weak supervision assumption is that the most dramatic relational changes in videos known to contain anomalies are more likely to reflect the anomalous event than are moderate changes. This is a form of multiple-instance learning: the video-level label ("anomalous") provides weak supervision for which relational changes to treat as positive, without requiring frame-level or pair-level annotations.

Step 2: Triplet margin loss training. CORE is a function $f_\theta: \mathbb{R}^{2d_r} \to \mathbb{R}^{d_{\text{token}}}$ that maps a concatenated before/after relational pair to a compact relation token vector of dimensionality $d_{\text{token}}$. The training objective is:

Ltriplet=max⁡(0,d(fa,fp)−d(fa,fn)+α)\mathcal{L}_{\text{triplet}} = \max\left(0, d(f_a, f_p) - d(f_a, f_n) + \alpha\right)

where $f_a = f_\theta(\mathbf{x}_a)$ is the CORE encoding of an anchor sample (a positive relational-change pair — one from a volatility peak in an anomalous video), $f_p = f_\theta(\mathbf{x}_p)$ is the CORE encoding of another positive sample from the same or a different anomalous video, $f_n = f_\theta(\mathbf{x}_n)$ is the CORE encoding of a negative sample (from a volatility valley or normal video), $d(\cdot, \cdot)$ is the L2 distance, and $\alpha = 0.5$ is the margin.

What it computes: the triplet loss tries to make the encoding of an anomalous relational change (anchor) closer to the encoding of another anomalous relational change (positive) than to the encoding of a normal relational change (negative) by at least the margin $\alpha$. If the distance $d(f_a, f_p)$ plus the margin is already smaller than $d(f_a, f_n)$, the loss is zero — the triplet is already correctly ordered. Otherwise, the loss is the amount by which the ordering constraint is violated, and gradient updates push $f_a$ and $f_p$ closer together while pushing $f_a$ and $f_n$ apart.

Why triplet loss over contrastive or classification: a classification loss (e.g., binary cross-entropy to predict anomalous vs. normal) would learn a decision boundary but wouldn't structure the embedding space — relation tokens for different types of anomalous interactions would not necessarily cluster by interaction type, and the tokens might not preserve distance-based similarity structure. Triplet loss directly optimizes for a metric space where distance encodes semantic similarity: anomalous relational changes are close to each other, normal relational changes are far from anomalous ones, and the relative distances reflect degrees of relational anomaly. This is well-suited for downstream LLM processing because the LLM's attention mechanism can exploit distance-based similarity patterns in the token space.

Step 3: Semi-hard negative mining. To improve training efficiency and discriminative power, the authors employ semi-hard negative mining [43]. Rather than using any random negative sample, the system selects negatives that satisfy:

d(fa,fp)<d(fa,fn)<d(fa,fp)+αd(f_a, f_p) < d(f_a, f_n) < d(f_a, f_p) + \alpha

These are negatives that are farther from the anchor than the positive is, but not far enough to satisfy the margin — they lie in the "grey zone" where the model is almost correct but not quite. Semi-hard negatives provide the most informative gradient signal because they represent difficult cases that the model needs to learn to separate from positives. Easy negatives (already far beyond the margin) would produce zero loss and no learning; hard negatives (closer to the anchor than the positive) can destabilize training by creating contradictory gradient signals. A pool size of 30 negatives is maintained for semi-hard selection.

Training configuration. The CORE encoder is trained with Adam optimizer, learning rate $1 \times 10^{-4}$, with a StepLR scheduler for 50 epochs. The two-layer MLP architecture is chosen for simplicity and speed — the relational feature extraction and triplet training together need to be efficient because they run as part of the video processing pipeline, and a lightweight encoder avoids becoming a computational bottleneck.

Step 4: Relation token generation at inference. At inference time, for each video, the system identifies the top 5% of volatility peaks and extracts the corresponding relational-change pairs $[\mathbf{r}_{ij}(t-1); \mathbf{r}_{ij}(t)]$. Each pair is passed through the trained CORE encoder to produce one relation token. The set of all relation tokens for the video is then projected to match the LLM's input dimension and concatenated with the visual tokens and text tokens.

What the relation tokens represent: each token is a compact summary of one salient interaction transition in the video — it encodes not just the fact that relationships changed, but the nature of the change as captured by the pre/post EGTR relational feature vectors. In the learned embedding space, similar types of interaction changes (e.g., different instances of physical aggression transitions) should map to nearby tokens, while qualitatively different changes (aggression vs. object abandonment) should map to distant tokens. The LLM, when it attends to these tokens alongside the visual frames, can associate specific visual content with relational transitions — for example, it can link the relation token for frame pair $(t-1, t)$ with the visual frames at those timestamps to ground the abstract relational change in concrete visual evidence.

Ablation comparison (Table 8). The paper compares relation tokens against two alternatives:

  • Relational visual cues: using only the static visual features and locations of detected objects, without pairwise relational features. This achieved BLEU 0.670 and Consistency 0.326, underperforming relation tokens (BLEU 0.718, Consistency 0.357) because static object properties miss interaction dynamics.
  • Scene graph text: converting the discrete relationship labels predicted by EGTR into text descriptions (e.g., "person A holding person B") and feeding these as additional text tokens. This achieved BLEU 0.686 and Consistency 0.335. The performance is better than visual cues alone but still below relation tokens, which the authors attribute to the vocabulary bottleneck — discrete labels from Visual Genome [18] are not tailored for nuanced, context-specific anomaly interactions.

LLM Integration and Fine-Tuning

What it is and why it exists. The final stage of VADER takes all extracted representations — visual tokens from keyframes, relation tokens from CORE, and text tokens from the task instruction — concatenates them into a single sequence, and feeds them to a frozen multimodal LLM that autoregressively generates the output text. The LLM is treated as a reasoning engine: it is not trained to detect anomalies or extract relations (those are handled by upstream modules), but rather to synthesize the provided visual and relational evidence into coherent, causally grounded natural language.

Backbone model. The authors use NVILA [20] as the MLLM backbone. NVILA is a recent efficient visual language model that supports video inputs. The specific configuration is not detailed in the main paper but is referenced in the implementation details (Appendix G). All parameters of the NVILA backbone — including the vision encoder and the language model decoder — are kept frozen during training.

Input construction. For each video, three types of tokens are concatenated:

  1. Visual tokens: each of the 64 keyframes selected by CAES is passed through NVILA's frozen vision encoder, producing a set of visual feature tokens. These tokens capture the appearance and spatial layout of each frame.

  2. Relation tokens: the set of relation tokens produced by CORE from the top 5% of volatility peaks. The number of relation tokens varies per video depending on the number of detected anomalous events and the length of the video, but each token has a fixed dimensionality $d_{\text{token}}$.

  3. Text tokens: the task instruction (e.g., "Give a compact description of the anomalous segment in the video.") and any additional metadata (e.g., the predicted anomaly class from the Anomaly Scorer) are tokenized by the LLM's text tokenizer.

These three token sequences are concatenated into a single input sequence: [visual tokens | relation tokens | text tokens]. The precise ordering — whether relational tokens precede or follow visual tokens, or are interleaved — is not specified in the paper, but the standard approach for MLLM input construction places visual tokens before text tokens, with additional modalities inserted between or after.

Projector layers. The visual tokens and relation tokens come from different encoders and have different dimensionalities than the LLM's expected input dimension. Two separate projector layers — typically small MLPs or linear projections — map the visual tokens and relation tokens to the LLM's hidden dimension. During fine-tuning, both projectors are trainable while the encoders that produce the raw tokens (vision encoder, CORE) and the LLM decoder remain frozen.

LoRA fine-tuning. To adapt the frozen LLM to the anomaly understanding domain without full fine-tuning (which would be prohibitively expensive), the authors apply Low-Rank Adaptation (LoRA) [15]. LoRA inserts trainable low-rank matrices into the attention layers of the LLM, parameterized as $W' = W + AB$ where $W$ is the frozen pre-trained weight matrix, $A \in \mathbb{R}^{d \times r}$ and $B \in \mathbb{R}^{r \times d}$ are the trainable low-rank factors, and $r \ll d$ is the rank. Only $A$ and $B$ receive gradient updates; the base weights $W$ remain frozen. This allows the model to learn domain-specific adaptation (how to interpret relational tokens, how to generate anomaly-focused language) while preserving the general video understanding capabilities learned during pre-training and keeping training costs manageable.

The training configuration is:

  • Learning rate: $2 \times 10^{-5}$ with a cosine schedule and a warm-up ratio of 0.03.
  • Trainable parameters: multimodal projectors and LoRA adapters only; all other parameters frozen.

Training data and objective. The LLM is fine-tuned on each benchmark's training data with the standard autoregressive language modeling objective — the model is trained to predict the next token of the ground-truth description, answer, or explanation given the concatenated input tokens. The loss is standard cross-entropy between predicted token probabilities and ground-truth tokens, computed only over the output text tokens (not over the input visual or relational tokens).

Inference. At inference time, the same pipeline runs end-to-end: video → Anomaly Scorer → CAES → visual tokens + relation tokens → LLM → autoregressive text generation. The LLM generates tokens one at a time, conditioned on all input tokens and previously generated output tokens, until an end-of-sequence token is produced or a maximum length is reached.

Why freeze and adapt rather than full fine-tuning: full fine-tuning of a multimodal LLM on relatively small anomaly understanding datasets (the HAWK training set, for instance, is limited compared to the pre-training data of NVILA) risks catastrophic forgetting — the model would overfit to the specific phrasing and scenarios in the training data and lose the broad visual reasoning capabilities from pre-training. By freezing the backbone and only training lightweight adapters, VADER preserves the model's general video understanding while adding the specific capability to attend to relational tokens and generate anomaly-focused language. This also makes the framework practically retrainable: adapting to a new anomaly dataset requires only re-training the projectors and LoRA adapters, not the full model.

The role of the predicted anomaly class. Recall that the Anomaly Scorer outputs both a per-frame anomaly score and a predicted anomaly class $c^*$. The paper states that these classes "provide semantic cues for downstream reasoning by LLMs." At inference time, the predicted class is likely converted to text (e.g., "The anomaly scorer predicts this event may involve fighting") and included in the text tokens fed to the LLM, giving it a prior hypothesis about the event type. The LLM can then use the visual and relational evidence to confirm, refine, or override this prior — if the visual frames clearly show a car accident but the anomaly scorer predicted "fighting," the LLM can use its stronger visual reasoning to correct the upstream error, making the system robust to scorer mistakes.

Summary of Design Choices and Their Justifications

  • CLIP-based anomaly scorer pre-trained on aggregated data: avoids per-dataset retraining and leverages CLIP's semantic understanding; the normality prototype $\mathbf{m}$ centers the feature space so anomaly scores are geometrically meaningful as distances from normal.
  • Gradient-based context expansion in CAES rather than fixed windows or score thresholds: captures the narrative structure of events (escalation, climax, resolution) by tracking how anomaly scores change over time, not just their absolute values; adapts to the temporal dynamics of each specific event.
  • Raw relational tensors from before the classification head in EGTR: bypasses the discrete vocabulary bottleneck of scene graph labels, providing continuous features that encode subtle gradations of interaction and support meaningful L2 distance comparisons across frames.
  • Max operator in the volatility computation: captures the single most dramatic interaction change per frame pair, preventing the signal from being diluted by many stable background relationships.
  • Top 5% peak mining with video-level weak supervision: a multiple-instance learning assumption that the most dramatic relational changes in anomalous videos are the anomalous interactions; avoids requiring expensive frame-level or pair-level annotations.
  • Triplet loss with semi-hard negative mining: learns a metric space where distance encodes the degree of relational anomaly, providing structured token representations that LLM attention mechanisms can exploit.
  • Frozen LLM with LoRA and projector fine-tuning: prevents catastrophic forgetting on small domain-specific datasets; enables practical retraining and adaptation to new anomaly scenarios.
  • Dynamic window over fixed or exponential interval context sampling: adapts context boundaries to each event's specific escalation and resolution profile, achieving the strongest overall metrics in ablation (Table 7).

4. Key Insights and Innovations

Innovation 1: Reframing Anomaly Understanding as a Relational Dynamics Problem

The central intellectual move of this paper is a reframing: video anomaly understanding is not fundamentally about detecting unusual appearances, but about detecting unusual transitions in the relationships between objects. This shifts the foundational assumption underlying decades of video anomaly detection research. The dominant paradigm — from classical handcrafted feature methods [23, 37, 42] through deep reconstruction-based approaches [14, 27, 48, 71] to modern weakly supervised methods [35, 46, 72, 73] — treats anomalies as statistical deviations from learned normality manifolds in appearance or motion feature space. The anomalousness is a property of pixels and their temporal derivatives in a frame or clip. Whether the system uses autoencoder reconstruction error, one-class SVM boundaries, or multiple-instance learning, the fundamental representational unit is the frame or clip embedding, not the structured interaction between objects.

VADER breaks from this tradition by declaring that what makes an event anomalous is relational: a person running is not anomalous in isolation; a person running against a crowd or toward a victim or away from a security checkpoint is anomalous, and the anomaly is constituted by the relationship between the runner and other entities in the scene. The anomaly lives in the edges of a dynamically evolving scene graph, not in the node features.

This is not the first paper to use scene graphs for video understanding — the paper acknowledges prior work like EGTR [17] for graph extraction and DecoAD [8] for knowledge-graph-based detection. But those approaches used scene graphs as an additional input modality to supplement appearance features — they extracted discrete relationship labels ("holding," "riding") and fed them alongside visual features to a detector. The relationships were treated as static properties of frames. What VADER reframes is that the relevant anomaly signal is the temporal derivative of the relationship embeddings, not the relationship labels themselves. A fight between two people is not anomalous because the frame contains a "fighting" relationship label; it is anomalous because the relationship transitions from "standing apart" to "physical contact" over a short time window. The anomaly is the change, not the state.

This reframing has downstream consequences that the paper demonstrates empirically: by encoding relational transitions rather than relational states, VADER outperforms prior MLLM-based VAU methods on causal reasoning tasks (the +7.38 MMEval improvement on CUVA's "Causes" task in Table 2) because causal reasoning inherently requires tracking how interactions evolve, not just what objects are present. The paper's ablation in Table 8 provides direct evidence: replacing relation tokens (which encode dynamic transitions) with scene graph text (which encodes static relational labels) drops BLEU from 0.718 to 0.686 and Consistency from 0.357 to 0.335, confirming that the temporal dynamics — not just the presence of relational information — drive the gains.

The significance of this reframing extends beyond VADER's architecture. It suggests that the broader field's difficulty in scaling anomaly detection to complex, context-dependent scenarios (where the same action is anomalous in one context and normal in another) stems from a representational mismatch: appearance-based features capture what things look like, but context-dependent anomaly requires capturing how things relate. If this reframing generalizes beyond VADER's specific benchmarks, it implies a research direction where anomaly detection architectures are built around relation-tracking modules (scene graph extractors, interaction encoders, temporal graph neural networks) rather than around ever-larger spatiotemporal feature extractors.

This is a fundamental conceptual reframing, not an incremental improvement. It changes what the field measures and optimizes for, from "does this look normal?" to "did the relationships between objects change in a typical way?"

Innovation 2: Weak Supervision via Relational Volatility as a Natural Anomaly Signal

The paper's second distinctive contribution is methodological: the recognition that pairwise relational volatility — the per-frame-pair maximum L2 distance between before/after relational feature vectors — serves as an effective, training-free anomaly signal for mining supervision data, without requiring any frame-level or interaction-level annotations. This is a form of weak supervision from structural priors that has no direct precedent in the video anomaly understanding literature.

To appreciate why this is novel, consider the standard approaches to obtaining supervision for video anomaly models. Supervised methods require per-frame labels of normal vs. anomalous, which are expensive to collect at scale and suffer from class imbalance (most frames are normal). Weakly supervised methods [35, 72, 73] use video-level labels only, applying multiple-instance learning to identify which frames within a labeled-anomalous video contain the anomaly — but these methods still optimize appearance-based detectors and still require model training to produce the frame-level signal. Unsupervised methods [13, 23, 14] require no labels at all but learn only what's normal and flag deviations, without any signal about what type of anomaly or which interaction caused it.

VADER introduces a third path: the relational volatility curve is computed without any learning from the pre-trained EGTR [17] relational features and the pre-trained CLIP-based anomaly scorer. It requires no anomaly-specific training data — only the frozen, off-the-shelf scene graph model and the normality prototype. The volatility peaks emerge from the geometric property that anomalous events involve sudden, large-magnitude changes in object-object interaction patterns, which is a structural assumption about the world, not a learned pattern from labeled data. This assumption — that relational change magnitude correlates with event significance — is simple but powerful. It transforms the unsupervised EGTR features into a noisy but informative anomaly signal without training a single parameter on anomaly data.

The practical consequence is that CORE can be trained with triplet loss using only video-level labels ("this video contains an anomaly") and the automatically-mined volatility peaks as positive samples. This bypasses the chicken-and-egg problem that afflicts many weakly supervised approaches: to train a good relation encoder, you need good examples of anomalous relational changes; to identify anomalous relational changes, you need a good relation encoder. By using the raw L2 distance in the pre-trained EGTR feature space as a bootstrapping signal, VADER sidesteps this circularity — the EGTR features are already semantically structured from their pre-training on Visual Genome, and the L2 distance in that space correlates (imperfectly but usefully) with interaction change magnitude.

The ablation in Table 9 (supplementary) validates the non-obvious parameter choices this approach requires. The Gaussian smoothing sigma of 2.0 and the top-5% peak threshold were selected through systematic sweep: sigma = 2.0 achieved 72.12% AUC vs. 70.18% for sigma = 1.0 and 69.25% for sigma = 3.0, and top-5% outperformed both top-3% (70.85%) and top-7% (71.64%). These specific values reflect a balance between noise suppression and signal preservation that is dataset- and model-dependent — the insight is not the values themselves but the demonstration that there exists a meaningful optimum, confirming that the volatility peaks carry genuine anomaly signal rather than being purely noisy.

This is best characterized as a methodological innovation — a new way to extract weak supervision from pre-trained representations — rather than a theoretical advance. The underlying idea (that temporal change in a structured representation can serve as a task-relevant signal) has precedents in other domains (e.g., optical flow as a motion signal from pixel-space differences), but its application to relational embeddings for anomaly understanding is novel and practically impactful because it provides a path to training interaction-aware models without the prohibitive cost of annotating pairwise object interactions in video.

Innovation 3: Narrative-Structured Temporal Sampling via Anomaly Score Gradients

The paper's third contribution is the insight that the temporal structure of anomalous events — escalation, climax, resolution — can be automatically segmented using the gradient (slope) of per-frame anomaly scores, and that this segmentation enables more causally coherent downstream reasoning than score-based or fixed-window selection. This is a diagnostic reframing of a problem that prior work had approached as a data reduction problem (how to subsample frames to fit the LLM's context window) rather than a narrative reconstruction problem (how to select frames that tell the causal story of the event).

Prior work on frame selection for video-language models generally falls into two categories: uniform sampling (take frames at regular intervals, losing temporal structure) and score-based selection (take frames with the highest anomaly scores, losing causal context). Holmes-VAU [67] introduced adaptive temporal sampling (ATS) based on anomaly scores using density-aware selection, which is the closest prior work to CAES and is included as a baseline in Table 6. ATS selects frames based on where anomaly scores are high and diverse, which improves over uniform sampling but still treats anomaly scores as the primary signal — it selects frames that are themselves anomalous rather than frames that explain how an anomaly developed.

CAES makes a subtle but critical conceptual shift: the frames most valuable for causal reasoning are not necessarily the frames with the highest anomaly scores — they are the frames where anomaly scores are changing fastest. The rising slope (pre-event) captures the escalation that explains why the event occurred, even though these frames may have moderate or low absolute scores. The falling slope (post-event) captures the resolution and aftermath, even though the scores are decreasing. The event segment itself (high scores, stable) captures the climax. This three-part structure — escalation, climax, resolution — mirrors the narrative structure that humans use when describing and reasoning about events, and it maps cleanly onto causal reasoning tasks: pre-event frames support "why" questions, event frames support "what" questions, and post-event frames support "what happened next" questions.

The ablation in Table 6 provides evidence for this claim. CAES (BLEU 0.668, Consistency 0.343) outperforms Top-K sampling (BLEU 0.663, Consistency 0.327) — where Top-K selects frames with the highest absolute anomaly scores — on every metric. This is notable because Top-K gets to see exactly the same anomalous content as CAES; the only difference is that CAES also includes lower-score pre- and post-event context frames. The fact that including these less anomalous frames improves performance confirms that causal context, not just anomaly saliency, drives understanding. Table 7 further refines this by showing that the dynamic window strategy (using gradient thresholds to determine context boundaries) outperforms fixed-window and exponential-interval alternatives on GPT-guided Consistency (0.343 vs. 0.343 vs. 0.342, with larger BLEU gains in the main table for exponential interval but Consistency parity for fixed window) — the dynamic window adapts to each event's specific temporal profile rather than applying a uniform context size.

This is an architectural design insight rather than a theoretical breakthrough. The idea that events have pre/during/post structure is not new — it's implicit in how humans narrate events and is formalized in narrative theory. What's new is the demonstration that anomaly score gradients provide a sufficient signal to automatically recover this structure, and that providing this structure to an LLM measurably improves causal reasoning. The practical implication is that future VAU systems should treat frame selection as a narrative reconstruction problem, not a saliency maximization problem, and that anomaly score dynamics (not just values) contain the relevant signal for narrative boundary detection.

Innovation 4: Demonstrating That Relational Tokens Enable MLLM Causal Reasoning Without Full Model Retraining

The paper's fourth contribution is more subtle but practically significant: it provides empirical evidence that a frozen MLLM can learn to interpret compact, learned relational tokens and integrate them with visual features to perform causally-grounded video understanding, without requiring the MLLM to be fine-tuned end-to-end or to process raw scene graph structures. This is a finding about representational interoperability — whether a general-purpose vision-language model, pre-trained on diverse internet data, can make productive use of a separately trained, domain-specific relational encoder's output when the two are connected only through lightweight projection layers and LoRA adapters.

Prior approaches to incorporating structured knowledge into MLLMs have generally fallen into two camps: (1) convert the structured knowledge to natural language text and append it to the prompt (e.g., scene graph text descriptions, as in Table 8), or (2) fine-tune the entire MLLM end-to-end with additional structured inputs so the model's internal representations adapt to the new modality. Approach (1) is simple but loses the continuous, fine-grained information in the structured representation — the ablation in Table 8 shows scene graph text achieves only BLEU 0.686 vs. relation tokens at 0.718, a substantial gap. Approach (2) is powerful but computationally expensive and risks catastrophic forgetting.

VADER demonstrates a third path: the MLLM backbone is kept entirely frozen; only the projector (mapping relation tokens to the LLM's hidden dimension) and LoRA adapters (low-rank matrices in attention layers) are trained. The relation tokens from CORE are not converted to text — they remain in a continuous embedding space learned by the triplet loss — and the LLM must learn to attend to them as an additional "sense" alongside visual tokens. The fact that this works — that a frozen NVILA backbone can learn to extract causal information from CORE's tokens through just a projection layer and low-rank attention adaptation — is not obvious a priori. It suggests that the relational information encoded by CORE is sufficiently well-structured (clustered by interaction type, ordered by anomaly degree) that the LLM's pre-trained attention mechanisms can exploit it without fundamental representational retraining.

The ablation in Table 5 provides evidence for this interoperability claim. Removing CORE entirely (keeping only visual tokens) drops BLEU from 0.718 to 0.668 and GPT-guided Reasonability from 0.428 to 0.419. These are not catastrophic drops — the frozen LLM with visual-only input still performs reasonably — but the gains from adding relational tokens are consistent across all metrics, confirming that the LLM extracts additional signal from the relational modality beyond what visual features alone provide. If the LLM were simply ignoring the relational tokens (e.g., because they come from a different distribution than its pre-training data), adding them would not improve performance, and might even degrade it through attention dilution. The consistent improvement indicates genuine cross-modal integration.

This is an empirical finding with practical significance rather than a theoretical advance. It establishes a design pattern — frozen backbone + lightweight domain-specific encoder + learned projector + LoRA — that can be replicated for other domain-specific video understanding tasks (traffic anomaly analysis, industrial inspection, medical procedure monitoring) where training a full end-to-end MLLM on limited domain data is infeasible. The pattern separates concerns cleanly: domain experts design the relational encoder (CORE) to capture task-specific structure, while the general-purpose MLLM provides broad visual and linguistic reasoning capabilities, and only the interface between them needs to be learned from domain data. The paper demonstrates this pattern works for video anomaly understanding; whether it generalizes to other structured reasoning domains is an open question that the paper implicitly invites others to investigate.

5. Experimental Analysis

Evaluation Methodology

  • Dataset. VADER is evaluated on three recent and challenging VAU benchmarks: (1) CUVA [11] — targets causation understanding with detailed annotations explaining what happened, why, and how for each anomalous event; (2) HIVAU-70k [67] — a large-scale dataset comprising videos from UCF-Crime [55] and XD-Violence [46], providing hierarchical annotations at clip, event, and video levels for both perception and reasoning tasks; and (3) HAWK [47] — focuses on open-world video anomaly understanding, collecting videos from seven diverse datasets [2, 7, 27, 33, 46, 52, 59], supporting both event description and question answering tasks. The paper follows each benchmark's official train/test splits without modification.

  • Base model(s). The MLLM backbone is NVILA [20], a recent efficient visual language model that supports video inputs. All experiments use the same NVILA architecture with frozen backbone weights; only the multimodal projectors and LoRA adapters are fine-tuned. The Anomaly Scorer uses a CLIP-based framework [64] pre-trained once on an aggregated dataset spanning diverse video scenarios, not retrained per benchmark. The Relation Feature Extractor uses EGTR [17], a pre-trained DETR-based scene graph generation model, also frozen at inference time.

  • Metrics. Evaluation follows each benchmark's official protocols, with metrics grouped into three categories: (1) Lexical-level metrics — BLEU [39], ROUGE [25], METEOR [4], and CIDEr [49] — measure n-gram overlap, precision, recall, and synonym matching with reference texts; (2) Semantic-level metrics — BLEURT [44] and MoverScore [70] — use pretrained language models to capture meaning and contextual similarity beyond exact word matches; (3) Judge-based evaluative metrics — GPT score [1] and MMEval [11] — directly use LLMs or VLMs as evaluators to judge plausibility, informativeness, and consistency of generated explanations with video content, simulating human judgment in open-ended settings. All metrics are reported such that higher scores indicate better performance. Table 1 summarizes which metrics are used for each benchmark.

  • Baselines. For CUVA, baselines include: mPLUG-owl [61], Video-LLAMA [66], PandaGPT [45], Otter [20], Video-ChatGPT [36], and CUVA* [11] (prompt or adapter tuning). For HIVAU-70k, baselines include: Video-ChatGPT [36], Video-LLAMA [66], Video-LLAVA [24], LLAVA-Next-Video [69], QwenVL2 [51], InternVL2 [9], NVILA [20], and Holmes-VAU‡ [67] (finetuned with same backbone size as VADER). For HAWK, baselines include: Video-ChatGPT [36], VideoChat [22], Video-LLAMA [66], LLAMA-Adapter [68], Video-LLAVA [24], NVILA [20], and HAWK† [47] (finetuned on HAWK). Models marked with † are finetuned on the respective benchmark; those marked with ‡ are finetuned and use the same backbone size as the primary comparison target; those marked with * employ prompt or adapter tuning; all others are evaluated without domain-specific finetuning.

  • Generation budget / compute accounting. For all experiments, the number of keyframes fed to the LLM is fixed at 64 frames (determined by the context window of the NVILA backbone, with the CAES strategy used to select which 64 frames). The paper does not report generation length budgets or beam search parameters for the LLM, suggesting standard greedy or temperature-based autoregressive decoding. All methods being compared can generate text outputs of unrestricted length. For computational efficiency comparison (Appendix E, Table 10), inference time is measured in total minutes and frames per second (fps) on the HAWK benchmark, with NVILA achieving 33.63 fps vs. VADER's 19.22 fps.

  • Cross-validation / statistical protocol. No cross-validation or statistical testing is reported. Results appear to be single-run evaluations on the standard test splits of each benchmark. The ablation studies on CORE hyperparameters (Table 9, Appendix B) report AUC on a held-out test set for the binary classification task of distinguishing anomalous from normal relational changes, but the main VAU results do not report confidence intervals, standard deviations, or significance tests. For the HIVAU-70k benchmark, two VADER variants are reported (‡ finetuned with same backbone size as Holmes-VAU, and † finetuned without that constraint), but this represents different training configurations rather than statistical validation.

Main Quantitative Results

CUVA Benchmark: Causation Understanding

The CUVA benchmark evaluates three causally-structured tasks: Description (what happened), Causes (why it happened), and Effect (what resulted). VADER's headline result is a +7.38 point improvement in MMEval score on the Causes task over the best prior method (CUVA*, 66.30 vs. 58.92), while maintaining competitive performance on Description and Effect tasks (Table 2).

Examining the full results table reveals several patterns:

  • MMEval (judge-based metric): VADER achieves 78.89 on Description (second to Video-ChatGPT's 78.55, by a margin of 0.34), 66.30 on Causes (best, +7.38 over CUVA*'s 58.92), and 63.26 on Effect (second to PandaGPT's 69.45). The Causes improvement is the standout result — this task directly tests causal reasoning about why an anomaly occurred, which is precisely where relational dynamics modeling should provide the largest benefit.

  • UniEval (semantic-level): VADER achieves 85.00 on Description, 79.37 on Causes, and 82.06 on Effect — all substantially above prior methods. The next best UniEval scores are 70.82 (Video-ChatGPT, Description), 70.77 (Video-ChatGPT, Causes), and 54.35 (Video-ChatGPT, Effect). The gap on Effect (+27.71 over Video-ChatGPT) is particularly large, suggesting that modeling how interactions evolve helps capture consequence chains that surface-level description methods miss.

  • BLEURT (semantic similarity): VADER's BLEURT scores (32.34, 37.48, 37.60) are lower than some baselines (e.g., Video-ChatGPT at 46.83, 49.52, 37.24). The paper explicitly addresses this: "This is due to BLEURT's focus on surface-level wording similarity." BLEURT measures similarity to reference descriptions; if VADER generates factually correct descriptions with different phrasing than the reference, BLEURT penalizes it. The strong MMEval and UniEval scores (which measure factual correctness and coherence rather than lexical overlap) support this interpretation — VADER is generating correct content that happens to be phrased differently from the reference.

A critical nuance: PandaGPT achieves the highest MMEval on Effect (69.45 vs. VADER's 63.26). The paper does not discuss this specific comparison. PandaGPT is evaluated without domain-specific finetuning on CUVA (no † or ∗ marking), meaning it uses its general pre-training. The fact that a non-finetuned model beats VADER on this subtask suggests that predicting event effects may rely more on general world knowledge (which large pre-trained models possess) than on relational dynamics extracted from the specific video. VADER's strength is in grounding its reasoning in observed interactions (Causes, where it leads), while Effect prediction may draw more on common-sense priors about how events typically resolve.

HIVAU-70k Benchmark: Hierarchical Anomaly Understanding

HIVAU-70k evaluates at three granularities: clip level (C), event level (E), and video level (V). VADER achieves state-of-the-art results across all metrics and all evaluation levels (Table 3). Two VADER variants are reported: VADER‡ (finetuned, same backbone size as Holmes-VAU) and VADER† (finetuned, possibly with a larger or different configuration).

Key comparisons at the most competitive setting (VADER† vs. Holmes-VAU‡):

MetricLevelHolmes-VAU‡VADER†Improvement
BLEUClip0.9131.266+0.353
BLEUEvent0.8041.246+0.442
BLEUVideo0.5661.268+0.702
CIDErClip0.4671.040+0.573
CIDErEvent1.5191.763+0.244
CIDErVideo1.4371.812+0.375
METEORClip0.1900.247+0.057
METEOREvent0.1650.216+0.051
METEORVideo0.1210.164+0.043
ROUGEClip0.3290.429+0.100
ROUGEEvent0.3700.463+0.093
ROUGEVideo0.3550.446+0.091

The video-level improvements are the most substantial (BLEU +0.702, CIDEr +0.375), which aligns with VADER's design: video-level tasks require reasoning over longer temporal spans where the pre-event/event/post-event narrative structure and relation dynamics become increasingly important. At the clip level, where context is limited, the improvement is smaller (BLEU +0.353) — suggesting CAES and CORE provide proportionally greater benefit as the reasoning task demands broader temporal integration.

An important baseline: NVILA (the frozen backbone without any VADER components) achieves BLEU scores of 0.610 (C), 0.340 (E), 0.283 (V). VADER† improves these to 1.266, 1.246, and 1.268 respectively — roughly doubling or more the BLEU scores. This confirms that the gains come from VADER's added components (CAES, CORE, fine-tuning), not from the backbone architecture alone.

The paper notes that "performance gains are particularly significant at the event and video levels, highlighting VADER's ability to reason over complex, temporally extended anomalous events." However, the hierarchical structure of HIVAU-70k means that clip-level predictions inform event-level predictions, which inform video-level predictions. The paper does not clarify whether VADER processes each level independently (generating separate outputs for clip, event, and video queries) or whether there is information sharing across levels. If processing is independent, the video-level gains genuinely reflect better long-term temporal reasoning; if there is cross-level dependency, the gains may partly reflect error propagation improvements at lower levels.

HAWK Benchmark: Description Generation and Question Answering

HAWK evaluates two main tasks: Anomaly Video Description Generation (producing natural language descriptions of anomalous events) and Anomaly Video Question-Answering (answering questions about anomalous events). Each task is evaluated with both text-level metrics (BLEU-1 through BLEU-4) and GPT-guided metrics (Reasonability, Detail, Consistency). VADER achieves leading results across both tasks (Table 4), with particularly strong performance on question answering.

For Description Generation, VADER vs. HAWK† (the previous best finetuned model):

MetricHAWK†VADER†Improvement
BLEU-10.2700.324+0.054
BLEU-20.1390.196+0.057
BLEU-30.0740.127+0.053
BLEU-40.0430.071+0.028
Reasonability0.2830.428+0.145
Detail0.3200.442+0.122
Consistency0.2180.357+0.139

The GPT-guided metric improvements (+0.122 to +0.145) are substantially larger in relative terms than the BLEU improvements (+0.028 to +0.057), indicating that VADER's descriptions are not just lexically closer to references but are judged by the LLM evaluator as more reasonable, detailed, and consistent — exactly what causal relation modeling should provide.

For Question Answering, VADER vs. HAWK†:

MetricHAWK†VADER†Improvement
BLEU-10.3190.484+0.165
BLEU-20.1790.311+0.132
BLEU-30.1120.210+0.098
BLEU-40.0730.150+0.077
Reasonability0.8400.828−0.012
Detail0.7940.825+0.031
Consistency0.7530.794+0.041

The QA improvements are larger than the description improvements (+0.165 BLEU-1 vs. +0.054 BLEU-1), suggesting that the relational tokens are particularly valuable for answering specific questions about anomalous events — questions that often probe causal structure explicitly. Interestingly, the Reasonability score for QA is slightly lower for VADER (0.828) than HAWK (0.840), a decline of −0.012. The paper does not comment on this, but it's a negligible difference. The Consistency improvement (+0.041) for QA is consistent with VADER's emphasis on tracking how interactions evolve — answering follow-up questions consistently requires maintaining a coherent event model across multiple answers, which relation tokens help enforce.

A notable baseline: Video-LLAMA [66] and VideoChat [22], evaluated without domain finetuning, achieve BLEU-1 scores of 0.062 and 0.053 for description (vs. VADER's 0.324), and 0.156 and 0.261 for QA (vs. VADER's 0.484). These ~5× gaps confirm that domain-specific finetuning (even just LoRA on a frozen backbone) is essential for VAU — general-purpose video-language models, despite their pre-training scale, do not naturally produce anomaly-focused language without adaptation.

Ablation Studies and Robustness Checks

All ablation studies are conducted on the HAWK description task (Tables 5–8, with additional results in Appendix B and E).

Impact of key components (Table 5): The full VADER model achieves BLEU 0.718 and GPT-guided Consistency 0.357. Removing CORE (the relation reasoning branch) reduces BLEU to 0.668 (−0.050) and Consistency to 0.343 (−0.014), confirming the importance of modeling object interactions but showing the drop is modest — visual features alone already capture substantial anomaly information, and CORE provides a smaller but consistent additive benefit. Removing CAES (disabling context-aware sampling, falling back to what is presumably uniform sampling since that is the baseline in Table 6) reduces BLEU to 0.594 (−0.124) and Consistency to 0.331 (−0.026), a more substantial drop suggesting that how frames are selected matters more than whether relational features are included. Removing fine-tuning entirely (frozen NVILA evaluated zero-shot) drops BLEU to 0.209 (−0.509), with an anomalous Detail spike to 0.527 (vs. 0.442 for full VADER). The paper explains: "the GPT-guided detail score is anomalously high, likely because the model generates lengthy but less focused responses due to the lack of domain adaptation." This is an important negative result — it shows that frozen MLLMs produce verbose but inaccurate descriptions, and fine-tuning is needed to calibrate the model toward concise, accurate anomaly language.

Keyframe selection strategy (Table 6): Four strategies are compared:

  • Uniform: standard uniform sampling, BLEU 0.594, Consistency 0.331
  • Top-K: selects frames with highest anomaly scores, BLEU 0.663, Consistency 0.327
  • ATS [67]: Holmes-VAU's adaptive temporal sampling, BLEU 0.641, Consistency 0.335
  • CAES (Ours): BLEU 0.668, Consistency 0.343

CAES outperforms all alternatives on every metric except BLEU vs. Top-K (0.668 vs. 0.663, a small 0.005 margin). The more informative comparisons are on the GPT-guided metrics: CAES achieves Reasonability 0.419 (vs. Top-K 0.381), Detail 0.477 (vs. Top-K 0.411), and Consistency 0.343 (vs. Top-K 0.327). These larger gaps on judge-based metrics suggest that the pre/post-event context frames included by CAES (and missed by Top-K's focus on peak anomaly frames) specifically improve the perceived coherence and detail of generated descriptions, even when lexical overlap with references is similar. ATS achieves only marginal gains over Top-K (0.641 vs. 0.663 BLEU in the opposite direction), which the paper attributes to "limited context diversity in complex scenarios." This is a non-obvious finding: Holmes-VAU's density-aware sampling, designed specifically for anomaly understanding, underperforms Top-K on BLEU in VADER's setting, suggesting that the sampling strategy interacts with other architectural choices (the LLM backbone, the presence of relation tokens) and is not universally superior.

Context sampling window strategy (Table 7): Three methods for defining pre/post-event context boundaries within CAES:

  • Fixed window: consecutive frames adjacent to the anomaly, BLEU 0.647, Consistency 0.343
  • Exponential interval: increasing spacing between context frames, BLEU 0.665, Consistency 0.342
  • Dynamic window (Ours): gradient-based adaptive boundaries, BLEU 0.668, Consistency 0.343

The differences are small. Dynamic window achieves the best BLEU (0.668) and ties fixed window for Consistency (0.343). Exponential interval achieves slightly higher ROUGE (0.278 vs. 0.274 for dynamic and 0.273 for fixed), but lower on GPT-guided Reasonability (0.392) and Detail (0.421). The practical takeaway is that context boundary selection has a measurable but modest impact — the larger gains come from whether context is included at all (Table 6) rather than how context boundaries are defined.

Relation representation modality (Table 8): Three encodings of relational information:

  • Relational visual cue: static object visual features and locations, BLEU 0.670, Consistency 0.326
  • Scene graph text: discrete relationship labels converted to text, BLEU 0.686, Consistency 0.335
  • Relation token (Ours): CORE-encoded continuous tokens, BLEU 0.718, Consistency 0.357

Relation tokens outperform both alternatives on every metric. The gap vs. scene graph text (BLEU +0.032, Consistency +0.022) is larger than the gap vs. relational visual cue (BLEU +0.048, Consistency +0.031). This pattern supports the paper's claim that the vocabulary bottleneck of discrete scene graph labels (restricted to Visual Genome predicates, which lack anomaly-specific relationships like "attacking") limits their utility, while the continuous CORE embeddings capture interaction nuances that the discrete vocabulary misses. The fact that scene graph text outperforms purely visual cues (BLEU 0.686 vs. 0.670) confirms that some relational information (even coarse, discrete labels) is better than none, but continuous tokens are substantially better than discrete text descriptions.

CORE hyperparameter sensitivity (Table 9, Appendix B): The Gaussian smoothing sigma (σ) and top-k percentile threshold for peak mining were swept over {1.0, 2.0, 3.0} and {3%, 5%, 7%} respectively, with AUC on a held-out relational-change classification test set as the metric. The optimal configuration is σ = 2.0, top-k = 5%, achieving 72.12% AUC. Using σ = 2.0, reducing k to 3% drops AUC to 70.85% (too few positives), increasing k to 7% drops AUC to 71.64% (too many noisy positives). With top-k = 5%, reducing σ to 1.0 drops AUC to 70.18% (insufficient smoothing, noisy peaks), increasing σ to 3.0 drops AUC to 69.25% (excessive smoothing, peaks blurred out). The sensitivity is moderate — the best configuration outperforms the worst (σ = 3.0, k = 3% at 68.17%) by 3.95 AUC points, indicating that while optimal parameters matter, the CORE training procedure is not brittle to small hyperparameter changes.

Computational efficiency (Table 10, Appendix E): VADER processes HAWK benchmark videos at 19.22 fps (87.25 minutes total), compared to 33.63 fps (49.85 minutes) for the NVILA backbone alone. The additional modules (CAES, relation extraction, CORE encoding) add approximately 75% overhead. The paper frames this as "still operating at a practical speed for real-world applications," which is a qualitative assessment — 19 fps is below the 25-30 fps typical of surveillance video, meaning VADER cannot process video at real-time rates on the hardware used. However, for offline batch analysis (forensic review of flagged incidents), 19 fps is more than adequate.

Critical Assessment

Claim 1: VADER achieves state-of-the-art results on video anomaly description, explanation, and reasoning tasks.

What the experiments demonstrate: VADER outperforms all compared baselines on nearly all metrics across three benchmarks — CUVA, HIVAU-70k, and HAWK. The evidence is comprehensive in coverage (three benchmarks, multiple tasks per benchmark, multiple metric types) and consistent in direction (VADER leads on the majority of metrics). On CUVA, it achieves best or second-best MMEval scores on all three causation tasks (Table 2). On HIVAU-70k, it leads all metrics at all three granularity levels (Table 3). On HAWK, it leads all text-level BLEU metrics and most GPT-guided metrics for both description and QA (Table 4).

What the experiments do not address: The "state-of-the-art" claim depends on which baselines are included. The paper compares against recent MLLM-based VAU methods but does not compare against several potentially relevant alternatives: (a) fine-tuned versions of the strongest general-purpose video MLLMs (e.g., GPT-4V with domain-specific prompting, Gemini Pro with video input) — the commercial models are absent; (b) ensemble methods combining multiple MLLMs with voting or verifier-based selection; (c) methods that use larger backbone LLMs than NVILA. Since NVILA is the base model, and VADER adds components on top, the fair comparison is VADER vs. NVILA with equivalent fine-tuning but without CAES/CORE. This comparison is partially covered (Table 5, "w/o Fine-Tuning" as a zero-shot baseline, and NVILA in Tables 3 and 4 as an unfinetuned baseline), but there is no "NVILA + LoRA finetuning + uniform sampling" baseline that would isolate the contribution of CAES and CORE from the contribution of domain adaptation. The "w/o CAES" ablation in Table 5 may serve this role, but it is not explicitly benchmarked against prior SOTA methods in Tables 2-4.

Conditional assessment: The claim holds within the specific comparison set the paper has chosen — VADER outperforms prior specialized VAU systems (Holmes-VAU, HAWK, CUVA) and general MLLMs evaluated zero-shot. Whether VADER would outperform a strongly finetuned NVILA baseline without VADER's specific components, or fine-tuned versions of larger commercial models, remains untested. The claim is supported for the academic VAU baselines but should be understood as "SOTA among published open-source VAU-specific methods" rather than "SOTA among all possible MLLM configurations."

Claim 2: Modeling temporally resolved object interactions (via CORE) substantially improves causal anomaly comprehension.

What the experiments demonstrate: Table 8 (ablation on relation representation) provides direct evidence: relation tokens (CORE) achieve BLEU 0.718 vs. 0.670 for visual-only relational cues and 0.686 for scene graph text, with corresponding GPT-guided Consistency of 0.357 vs. 0.326 and 0.335. This is a controlled comparison isolating the representation of relational information. Table 5 shows that removing CORE entirely drops BLEU from 0.718 to 0.668 and Consistency from 0.357 to 0.343. On CUVA (Table 2), the largest gain is on the Causes task (+7.38 MMEval points), which directly tests causal reasoning about why anomalies occurred — precisely where dynamic interaction modeling should help.

What the experiments do not address: The ablation removes CORE entirely (Table 5) or replaces it with alternatives (Table 8), but does not ablate the temporal aspect of CORE — i.e., there is no comparison with "relation tokens from CORE trained on static per-frame relations rather than temporal change pairs." Without this, we cannot distinguish whether the gains come from: (a) having any relational information beyond visual features, (b) having continuous relational tokens rather than discrete labels, or (c) having temporally dynamic relational tokens that encode interaction transitions. The Table 8 comparison vs. scene graph text addresses (b) — continuous tokens beat discrete labels — but the dynamic vs. static question is untested. The paper's claim is specifically about "temporally resolved" interactions, but the ablation evidence supports a slightly weaker claim: "relational information, particularly when encoded continuously, improves causal comprehension." The temporal dynamics contribution is plausible given the volatility-based mining and before/after pair encoding, but is not cleanly isolated.

Conditional assessment: The claim is supported with qualifications. The relational tokens clearly improve over no relational information and over discrete relational labels. The specific contribution of temporal dynamics (vs. static relational encoding) is not directly ablated, though the architecture design (volatility peaks, before/after pairs) strongly suggests it matters. A future experiment comparing CORE trained on before/after pairs vs. CORE trained on single-frame relational tensors would resolve this.

Claim 3: The CAES sampling strategy, which captures pre-event, on-event, and post-event context, is superior to alternative frame selection methods.

What the experiments demonstrate: Table 6 shows CAES outperforms Uniform, Top-K, and ATS on all GPT-guided metrics (Reasonability, Detail, Consistency), with BLEU of 0.668 (vs. 0.594 Uniform, 0.663 Top-K, 0.641 ATS). Table 7 shows the Dynamic Window variant within CAES achieves the best overall combination of metrics. The gains on GPT-guided metrics (especially Detail: 0.477 for CAES vs. 0.411 for Top-K) are larger than the gains on text-level metrics, suggesting that CAES's narrative structure primarily improves perceived description quality (detail, coherence) rather than exact lexical match with references.

What the experiments do not address: The frame budget is fixed at 64 in all experiments. There is no ablation of the frame budget itself — would the advantage of CAES over uniform sampling shrink or grow with more frames (e.g., 128 or 256)? If the LLM's context window allowed more frames, uniform sampling might catch enough context frames by chance to close the gap. Additionally, there is no ablation of the 8:4:4 sampling ratio (event:pre:post) — would a different allocation, perhaps 6:5:5 for tasks requiring more causal context, improve performance further? The ablation in Table 7 compares window strategies but keeps the per-segment frame counts fixed.

What is genuinely demonstrated: CAES is better than the compared alternatives (Uniform, Top-K, ATS) at a 64-frame budget on the HAWK description task. More than that is inference, not direct evidence.

Claim 4: Weakly supervised contrastive learning on relational volatility peaks provides effective training signal for the CORE encoder.

What the experiments demonstrate: Table 9 (Appendix B) shows that the CORE training procedure (triplet loss with mined positives/negatives) achieves 72.12% AUC at optimal hyperparameters on the relational-change classification test set. This confirms that the volatility-based mining produces a training signal that learns a non-trivial separation between anomalous and normal relational changes. The downstream effect is shown in Table 8: CORE relation tokens outperform alternatives that don't use this training procedure.

What the experiments do not address: The AUC metric in Table 9 measures how well CORE distinguishes mined anomaly peaks from mined valleys/normal videos on a test set — but this is a binary classification evaluation of the CORE embedding space, not a direct measure of how those embeddings contribute to downstream VAU performance. There is no experiment showing how downstream HAWK performance varies with CORE AUC — i.e., if we trained CORE to higher AUC (e.g., 80%), would HAWK metrics improve proportionally, or is there a saturation point? Additionally, the "weak supervision" claim relies on video-level labels being sufficient for the peak mining to identify genuinely anomalous relational changes. There is no analysis of false positive peaks (relational volatility spikes in anomalous videos that are not related to the anomaly — e.g., someone in the background of a fight video standing up from a bench) and how they affect training. If a substantial fraction of the top-5% peaks are false positives, the CORE encoder would be learning to encode spurious relational changes as anomalous, potentially degrading downstream performance. The paper provides no false positive rate analysis.

Claim 5: The overall VADER framework advances the frontier of explainable video anomaly analysis.

What the experiments demonstrate: Across three benchmarks with diverse tasks (description, causation, QA, hierarchical understanding), VADER produces strong quantitative results. The qualitative examples (Figures 4, 5, 6, 9) show VADER generating detailed, contextually grounded descriptions that avoid the hallucinations or generic outputs of baseline MLLMs.

What the experiments do not address: "Explainability" as a concept is not directly measured by any metric in the paper. BLEU and ROUGE measure lexical similarity to references; MMEval and GPT Score measure judged quality, but neither quantifies explainability specifically — whether the generated text correctly identifies causal factors, provides reasoning chains, or supports counterfactual analysis. The CUVA benchmark comes closest with its Causes and Effect tasks, but even these are evaluated through the same metrics (MMEval, BLEURT) as descriptions. There is no human evaluation, no error analysis of what types of causal errors VADER makes (e.g., does it correctly identify causes but fail on effects? does it confuse correlation with causation?), and no comparison with human-written explanations. The qualitative examples are cherry-picked — the paper does not report the rate of hallucination, factual error, or causal misattribution across the full test set, making it impossible to assess how reliably explainable VADER's outputs are.

Overall Strengths of the Experimental Design

  • Multi-benchmark evaluation across three distinct benchmarks with complementary task designs (causation in CUVA, hierarchical understanding in HIVAU-70k, description and QA in HAWK) provides strong evidence that VADER generalizes across evaluation protocols rather than overfitting to a single benchmark's quirks.
  • Multiple metric categories — lexical, semantic, and judge-based — provide a more comprehensive picture than BLEU alone. The pattern where VADER excels on judge-based metrics but sometimes lags on lexical metrics (e.g., BLEURT on CUVA) is correctly identified and explained.
  • Controlled ablations (Tables 5-9) systematically isolate the contribution of each major component, with the CAES and relation representation ablations being particularly informative about why VADER works.
  • Negative results reported — the ReST^EM-like finding that removing fine-tuning produces verbose but unfocused outputs (Table 5), and the observation that ATS underperforms Top-K on BLEU (Table 6) — add credibility.

Overall Weaknesses and Missing Experiments

  • No confidence intervals or significance tests. All results are single numbers without error bars. With test sets of unknown size (the paper does not report per-benchmark test set sizes), it's impossible to assess whether a 0.005 BLEU difference (e.g., CAES vs. Top-K in Table 6) is statistically meaningful or noise.
  • No comparison with strongly finetuned baselines. The paper compares against zero-shot MLLMs and prior specialized VAU methods, but does not compare against "NVILA + LoRA finetuning + uniform sampling" as a baseline that would isolate the contribution of CAES/CORE from the contribution of domain adaptation alone. This is the most obvious missing experiment.
  • No human evaluation. For a paper advancing "explainable" video analysis, the absence of human judgment studies is a significant gap. Automated metrics (even GPT-based ones) have known biases and may not correlate with human assessments of explanation quality, causal accuracy, or trustworthiness.
  • No analysis of failure modes by anomaly type. The paper acknowledges in Section 5 (Limitations) that VADER is biased toward high-motion events and object-centric reasoning, but does not provide a quantitative breakdown of performance by anomaly type. Which anomalies does VADER handle well, and which does it systematically fail on? This would be more informative than aggregate metrics.
  • No ablation of frame budget. The 64-frame budget is fixed and not justified beyond "dictated by the context window of the backbone MLLM." Without ablation at different budgets (16, 32, 128), we cannot assess whether CAES's advantage stems from better frame selection or from hitting a particular frame density that uniform sampling misses.
  • Single MLLM backbone. All experiments use NVILA. The paper does not test whether VADER's components (particularly CORE) transfer to other MLLM backbones (e.g., LLaVA, Video-LLaMA, Qwen-VL). Backbone-specific interactions with the relational token representation are unexplored.
  • No temporal dynamics ablation for CORE. The central claim that dynamic (temporally resolved) interactions matter is not tested against a static relational encoding baseline trained identically to CORE but on single-frame rather than before/after pairs.
  • Difficulty estimation cost not included in efficiency comparison. The computational overhead of CAES (gradient computation, peak detection, context expansion) and CORE (volatility curve computation, inference through the encoder) adds approximately 75% to inference time (Table 10). This cost is reported but not factored into any efficiency-adjusted performance comparison — there is no "accuracy per FLOP" or "accuracy per second" metric that would penalize VADER for its additional computation. A strongly finetuned NVILA baseline processing twice as many frames in the same wall-clock time might close some or all of the performance gap.

6. Limitations and Trade-offs

1. The Cost of Difficulty Estimation (Keyframe Selection Overhead) Is Not Accounted For in Headline Performance

The assumption or constraint. The entire VADER pipeline depends on CAES to select narrative-structured keyframes, which in turn depends on per-frame anomaly scores from the Anomaly Scorer and gradient-based context expansion. The Anomaly Scorer, while pre-trained once, must run inference on every frame of every test video — computing CLIP image embeddings, re-centering with the normality prototype, projecting onto class-specific semantic directions, and aggregating with the temporal module's anomaly probability. The relation extraction path adds further overhead: EGTR must detect objects and compute pairwise relational tensors for every CAES-selected frame, object association must be solved via the Hungarian algorithm across all frame pairs, and the full volatility curve must be computed and smoothed. The paper acknowledges this implicitly by reporting inference speed in Appendix E (Table 10) — 19.22 fps for VADER vs. 33.63 fps for the NVILA backbone alone — but does not include this overhead in any efficiency-adjusted accuracy comparison. The paper does not claim CAES is cheap; it claims it improves accuracy, and the cost is reported as a separate measurement.

The consequence. The headline results (Tables 2–4) compare VADER's accuracy against baselines that use simpler frame selection (uniform sampling, Top-K, or no selection at all) without adjusting for the fact that VADER does substantially more computation before the LLM ever sees a frame. A practitioner comparing VADER against, say, NVILA + LoRA fine-tuning with double the frame budget (128 uniform-sampled frames, which would take roughly the same wall-clock time as VADER's 64-frame CAES + relation extraction pipeline) would not know from the paper's tables whether VADER's gains come from smarter frame selection or from spending more total computation. The 75% inference overhead (Table 10) is large enough that this is not a minor accounting detail — it is a genuine question about whether the method's improvements survive a compute-matched comparison.

What evidence exists in the paper. Table 10 (Appendix E) reports the raw inference time and fps. The ablations in Table 6 compare CAES against Uniform, Top-K, and ATS, but all at the same 64-frame budget — the comparison controls for the number of frames fed to the LLM but does not control for the cost of selecting those frames. There is no experiment that gives uniform sampling (or any baseline) additional frames to match VADER's total computation budget, which is the relevant practical comparison. The CAES ablation in Table 5 removes CAES and shows a BLEU drop from 0.718 to 0.594, but the "w/o CAES" setting presumably falls back to uniform sampling at 64 frames — it does not test whether 128 uniform frames (with the same total cost) would recover the gap.

Mitigation status. Not addressed. The paper does not propose a cheaper difficulty estimation method, nor does it factor the overhead into any compute-matched comparison. Appendix D proposes "Module Robustness Enhancement" as a future direction, but this is framed as improving upstream module accuracy, not reducing their cost. The computational efficiency section (Appendix E) reports the numbers transparently but does not integrate them into the performance analysis. A compute-matched experiment or an ablation varying frame budget under a fixed total-time constraint would substantially strengthen the practical case for VADER.

2. The Framework Has an Inherent Bias Toward High-Motion, Object-Centric Anomalies and Systematically Fails on Subtle or Scene-Level Events

The assumption or constraint. VADER's core signal — the relational volatility curve — is defined as the maximum L2 distance between before/after pairwise object relationship vectors in adjacent frames (Equation 4). This means the signal is proportional to the speed and magnitude of change in object-object interactions. The paper explicitly acknowledges this in Section 5:

"The reliance on relational volatility as the core signal introduces a bias toward anomalies with strong motion, making the framework less sensitive to subtle or low-motion anomalies."

Furthermore, the relation extraction pipeline is object-centric by design — it requires EGTR to detect and pair objects, computes pairwise relational features, and tracks object identities across frames. The paper acknowledges:

"VADER's current design is object-centric, which limits its ability to capture scene-level or group-level anomalies such as environmental changes or collective crowd behaviors."

The consequence. VADER will systematically underperform on (or entirely miss) several important classes of real-world anomalies:

  • Low-motion anomalies: unattended bags left on a platform, a person loitering in a restricted area without sudden movements, gradual environmental changes (smoke accumulation, water leakage). These events produce minimal relational volatility — the bag's relationship to its surroundings changes only when it is abandoned, but the change is a single frame transition (owner present → owner absent) that may produce only a modest volatility peak, easily lost in Gaussian smoothing or outranked by unrelated high-motion background events.
  • Scene-level anomalies: lighting failures, camera tampering, weather events affecting visibility (fog, heavy rain). These have no object-level relational signal at all — there are no objects to track or no interactions to measure — and VADER's EGTR extractor may not even detect objects in such conditions.
  • Collective crowd behaviors: a crowd suddenly panicking and dispersing, or a group converging on a point of interest. These involve many simultaneous pairwise interactions that the max operator in the volatility computation reduces to a single pair's change, losing the emergent group-level signal.

What evidence exists in the paper. The failure cases in Appendix C (Figures 7 and 8) explicitly demonstrate these limitations. Figure 7 shows VADER focusing on a physical confrontation while ignoring the initial theft that triggered it (missing the causal chain because the theft was low-motion), and identifying a car collision but missing the underlying cause of a vehicle running a traffic signal (because the signal violation was visually subtle compared to the crash). Figure 8 shows VADER describing individual pedestrians entering and exiting a subway but failing to notice that a crowd gathered to watch a street performance — the group-level anomaly is invisible to the pairwise object interaction model.

The paper provides no quantitative breakdown of performance by anomaly type, making it impossible to assess what fraction of test-set anomalies fall into these failure categories or how severely VADER's performance degrades on them. The aggregate metrics in Tables 2–4 include both high-motion and low-motion anomalies, so the reported numbers represent an average that obscures the systematic failures.

Mitigation status. The paper proposes mitigation strategies in Appendix D (D.2 and D.3): object state modeling (tracking transitions like "carried → stationary" for abandoned objects), global scene context modeling (encoding entire frames to capture environmental anomalies), and group-level reasoning (identifying and modeling clusters of objects). None of these are implemented or evaluated. They are listed as future work directions. The limitation remains unaddressed in the current system, and a practitioner deploying VADER would need to accept the risk of missed detection on these anomaly types or invest in developing the proposed extensions independently.

3. Generalization Claims Rest on a Single MLLM Backbone and Three Benchmarks Drawn from Overlapping Surveillance Domains

The assumption or constraint. All experiments use NVILA as the sole MLLM backbone. All three evaluation benchmarks — CUVA, HIVAU-70k, and HAWK — are built from surveillance videos (UCF-Crime, XD-Violence, and various public surveillance datasets) depicting street crime, traffic accidents, physical altercations, and similar real-world security scenarios. The method's components (particularly CORE's relational tokens) are designed to interface with NVILA's specific architecture: the token dimensionalities, the projector design, and the LoRA adaptation strategy are all tuned to one backbone. The paper states it uses NVILA because it is a "representative" efficient visual language model (Appendix G), but provides no evidence that VADER's improvements would transfer to other backbones (LLaVA, Video-LLaMA, Qwen-VL, GPT-4V) or to non-surveillance anomaly domains (industrial inspection, medical procedure monitoring, autonomous driving anomaly detection).

The consequence. Three distinct but related concerns:

  • Backbone dependence: CORE produces relation tokens in a learned embedding space shaped by triplet loss training. Whether a different frozen MLLM could learn to productively attend to these tokens through the same projector + LoRA adaptation scheme is unknown. The representational interoperability finding — that a frozen MLLM can extract signal from a separately trained domain-specific encoder — may be specific to NVILA's pre-training data, architecture, or attention patterns. A backbone with different inductive biases (e.g., one pre-trained on different video distributions) might ignore or misinterpret the relational tokens.

  • Domain narrowness: All three benchmarks draw from the same broad domain (public-space surveillance with human-centric anomalies). The paper's strong results do not provide evidence that VADER would work for manufacturing defect detection, surgical procedure error identification, autonomous vehicle anomaly analysis, or sports foul detection — all domains where "anomalous object interactions" have very different signatures than street crime. The EGTR scene graph extractor was pre-trained on Visual Genome, which is heavily biased toward everyday object interactions; its relational features may not transfer to domains with specialized objects and interaction types.

  • Benchmark overlap: The three benchmarks share video sources (e.g., UCF-Crime and XD-Violence appear in HIVAU-70k; HAWK draws from seven datasets that partially overlap with sources of the other benchmarks). The paper does not analyze whether VADER's advantage is consistent across non-overlapping subsets or concentrated in videos from shared training sources.

What evidence exists in the paper. The paper reports no backbone ablation — there is no experiment replacing NVILA with another MLLM while keeping CAES and CORE fixed. There is no domain transfer experiment where VADER is trained on one benchmark's data and evaluated on another. There is no analysis of per-video-source performance breakdown. The paper implicitly acknowledges the domain scope by its benchmark selection (all surveillance-oriented) but does not discuss the limitation or claim broader applicability.

Mitigation status. Not addressed. The paper does not claim the method generalizes beyond the tested benchmarks — it makes domain-specific claims about "video anomaly understanding" in the surveillance context — but the framing (particularly the abstract's "advancing the frontier of explainable video anomaly analysis") could be read as implying broader applicability. A practitioner in a non-surveillance domain would need to assume substantial risk when adopting VADER without backbone and domain transfer experiments. Appendix D's proposed extensions (global scene modeling, group reasoning) focus on expanding coverage within the surveillance domain rather than testing transfer to new domains.

4. The Temporal Dynamics Claim for CORE Is Not Directly Validated Against a Static Relational Baseline

The assumption or constraint. VADER's core technical claim is that temporally dynamic relational modeling — encoding before/after interaction transitions rather than static per-frame relations — drives its performance gains. CORE is trained on relational-change pairs $[\mathbf{r}_{ij}(t-1); \mathbf{r}_{ij}(t)]$ extracted from volatility peaks, and generates tokens that summarize transitions in object interactions. The paper's motivation (Section 1) explicitly distinguishes this from prior work that "neglect[s] deeper causal relationships and dynamic object interactions."

The consequence. Without a direct ablation comparing CORE trained on before/after pairs against an identical encoder trained on single-frame relational tensors (i.e., encoding what the relationship is rather than how it changed), we cannot determine whether the gains attributed to "temporal dynamics" actually come from: (a) the dynamic information itself, (b) simply having any relational features beyond visual appearance, or (c) the continuous, vocabulary-free encoding (vs. discrete scene graph labels). The paper's Table 8 compares relation tokens against scene graph text and relational visual cues, but neither alternative uses static continuous relational encoding — scene graph text encodes static discrete labels, and relational visual cues encode object appearance rather than relationships. A fair test would be: train an encoder identical to CORE (same architecture, same triplet loss) on single-frame EGTR relational tensors from CAES-selected frames, without using before/after pairs, and compare the resulting tokens to CORE's dynamic tokens.

What evidence exists in the paper. The ablation in Table 8 shows that continuous relation tokens outperform discrete scene graph text (BLEU 0.718 vs. 0.686) and visual-only cues (0.718 vs. 0.670). This establishes that continuous relational encoding beats the alternatives tested, but does not isolate the contribution of temporal dynamics specifically. The volatility curve computation and before/after pair mining are central to CORE's training, but their unique contribution (beyond providing a principled way to select training pairs for a continuous relational encoder) is not measured. The qualitative example in Figure 9 shows CORE helping the model "distinguish directional actions," which is suggestive but not a systematic validation.

Mitigation status. Not addressed. The paper does not discuss this as a limitation, and the missing ablation is non-obvious — most readers would likely accept that since CORE uses temporal dynamics and outperforms non-temporal alternatives, the temporal dynamics must be responsible. But that inference conflates "CORE's training procedure uses before/after pairs" with "CORE's advantage comes from encoding transitions rather than states." A single well-designed ablation (static CORE trained on per-frame EGTR tensors) would resolve this ambiguity and substantially strengthen the paper's headline claim about dynamic relational modeling.

5. No Human Evaluation or Failure Mode Quantification for an "Explainable" System

The assumption or constraint. The paper positions VADER as advancing "explainable video anomaly analysis" (Section 1) and "causally grounded video anomaly understanding" (Section 3). The evaluation relies entirely on automated metrics: lexical overlap (BLEU, ROUGE, METEOR, CIDEr), semantic similarity (BLEURT, MoverScore), and LLM-as-judge metrics (GPT Score, MMEval). No human evaluators assess the quality, factual accuracy, causal correctness, or trustworthiness of VADER's outputs. The paper does not report the rate at which VADER generates factually incorrect descriptions (hallucinations), makes causal attribution errors, or produces outputs that a human supervisor would find misleading.

The consequence. This is a critical gap for a paper whose contribution is improved understanding and explainability:

  • Automated metrics can be gamed or mislead: The paper itself notes that BLEURT penalizes VADER for generating factually correct descriptions with different phrasing than the reference (CUVA results, Table 2). Conversely, the GPT-guided metrics (which give VADER its strongest results) are known to exhibit biases — they may favor fluent, detailed outputs over concise, accurate ones, and the paper's own ablation in Table 5 shows that removing fine-tuning causes the Detail score to increase to 0.527 (vs. 0.442 for full VADER) because the zero-shot model generates "lengthy but less focused responses." If a zero-shot model scores higher on Detail than the fine-tuned, carefully-designed system, the metric is not measuring what practitioners care about.

  • Causal reasoning quality is not directly measured: The CUVA benchmark has separate tasks for Description, Causes, and Effect, but evaluates all three with the same metrics (MMEval, BLEURT, etc.). There is no metric that specifically checks whether identified causes are actually causal (i.e., would the effect still occur if the identified cause were removed?) versus merely temporally correlated or plausibly associated. VADER's +7.38 MMEval improvement on CUVA Causes (Table 2) could reflect better language modeling around causal language rather than genuinely improved causal attribution.

  • Trustworthiness requires failure mode awareness: A security operator using VADER in a real deployment needs to know: how often does VADER describe an event that didn't happen? How often does it correctly identify that something anomalous occurred but misidentify the type? How often does it miss anomalies entirely? None of these error rates are reported.

What evidence exists in the paper. The failure cases in Appendix C (Figures 7 and 8) are the only direct window into VADER's errors, but they are qualitative examples selected by the authors — there is no quantitative error taxonomy, no confusion matrix across anomaly types, and no systematic sampling of failures. The quantitative results show VADER outperforming baselines on aggregate metrics, which tells us it is better on average but not how it fails when it does fail.

Mitigation status. Not addressed. The paper does not discuss the absence of human evaluation as a limitation. The proposed future work in Appendix D focuses on architectural extensions (global scene modeling, group reasoning) rather than on validation of the explainability claims. A practitioner cannot determine from the paper whether VADER's outputs are reliable enough for a high-stakes deployment (e.g., security operations), and the failure case figures (while useful) do not substitute for systematic error characterization.

6. Weakly Supervised Peak Mining Assumes the Largest Relational Changes Are the Most Anomalous — With No False Positive Analysis

The assumption or constraint. CORE's positive training samples are mined by selecting the top 5% of smoothed volatility peaks from videos with video-level anomaly labels. This embeds a specific assumption: that in an anomalous video, the largest-magnitude relational changes are the anomalous ones, or at least that they are sufficiently correlated with the anomaly to provide useful training signal. The paper does not validate this assumption empirically — there is no measurement of what fraction of top-5% peaks in anomalous videos actually correspond to the annotated anomalous event versus being false positives from unrelated activity (someone in the background standing up, a door opening, a vehicle passing normally).

The consequence. If a substantial fraction of the top-5% peaks in anomalous videos are false positives — large relational changes that are not related to the annotated anomaly — then CORE is being trained to encode those spurious transitions as "anomalous" in its embedding space. This would have two effects:

  • Reduced discriminative power: the triplet loss would push CORE to cluster together true anomalous transitions and false positive transitions, making it harder for the downstream LLM to distinguish genuine anomaly signals from background activity. The CORE embedding space would represent "large relational change" rather than "anomalous relational change."

  • Deployment fragility: in videos with significant background activity (busy street scenes, crowded environments), the volatility peak miner might systematically select false positives while missing the true (but visually subtle) anomaly, causing CORE to produce tokens that highlight irrelevant interactions and distract the LLM from the actual event.

What evidence exists in the paper. The only evidence bearing on this is the AUC metric in Table 9 (Appendix B), which measures CORE's ability to distinguish mined peaks from valleys on a held-out test set. An AUC of 72.12% indicates non-trivial but far-from-perfect separation. However, this AUC is computed against the very same mining procedure used to generate the labels — peaks are "positive" and valleys are "negative" — so it measures consistency of the embedding with the mining heuristic, not accuracy against ground-truth anomalous interaction labels. If the mining heuristic itself has a 30% false positive rate, CORE could achieve high AUC by faithfully encoding those false positives as "anomalous," and the downstream impact would be invisible in the AUC metric.

Mitigation status. Not addressed. The paper does not discuss this assumption or its potential violation. There is no experiment correlating CORE's encoded features with human annotations of which interactions are actually anomalous. Appendix B (Table 9) sweeps the top-k percentile threshold and Gaussian smoothing sigma, but these sweeps optimize for the mining heuristic's self-consistency, not for alignment with ground-truth anomalous interactions. A practitioner deploying VADER in an environment with different background activity characteristics (e.g., denser crowds, more frequent non-anomalous interactions) would have no guidance on whether the top-5% threshold remains appropriate or whether false positive rate increases would degrade performance.

7. Implications and Future Directions

How This Work Changes the Landscape

VADER represents a reframing rather than a paradigm shift — it does not invent fundamentally new architectural primitives (scene graph extraction, contrastive learning, MLLM fine-tuning all predate it), but it reorganizes these primitives around a new central claim: causal anomaly understanding requires modeling the temporal dynamics of object interactions, not just the appearance of frames or the static presence of relationships. This reframing has several downstream consequences for how the field approaches video anomaly understanding.

From detection-centric to interaction-centric reasoning. The dominant trajectory in video anomaly research has been detection → temporal localization → open-vocabulary detection → description. Each step added capability but preserved the underlying representational assumption that anomalies are properties of frames or clips — deviations from a learned normality distribution in appearance or motion feature space. VADER challenges this by demonstrating that explicitly representing how object-object interactions change over time improves performance on tasks that require causal reasoning (the +7.38 MMEval improvement on CUVA Causes in Table 2), while contributing less to tasks that primarily require scene description (the +0.34 MMEval improvement on CUVA Description). This pattern — large gains on causation, moderate gains on description — suggests that relational dynamics are not just another feature to add, but are specifically what enable models to answer "why" questions about events. This reshapes the problem formulation: a good VAU system is not one that describes frames well, but one that tracks interaction trajectories and identifies anomalous transitions.

Reconciling the scene graph promise with the vocabulary bottleneck. Scene graphs have been proposed for video understanding for years, but their impact on anomaly tasks has been limited by the discrete vocabulary of pre-trained scene graph models — Visual Genome predicates like "holding," "riding," and "standing on" do not capture the interaction types relevant to surveillance anomalies ("attacking," "fleeing from," "breaking into"). VADER's key design choice — extracting relational features before the classification head — bypasses this bottleneck and demonstrates that the intermediate continuous representations carry substantially more anomaly-relevant information than the discrete labels (Table 8: relation tokens BLEU 0.718 vs. scene graph text BLEU 0.686, with larger gaps on semantic metrics). This finding redirects scene graph research for anomaly applications away from expanding predicate vocabularies and toward learning continuous relational embeddings that can be compared and tracked over time.

The narrative structure of events as a computational primitive. CAES's gradient-based segmentation of videos into pre-event, on-event, and post-event phases treats narrative structure not as a post-hoc interpretative frame applied to model outputs, but as an input structuring mechanism that directly controls which frames the LLM sees. The ablation results (Table 6: CAES outperforms Top-K on all GPT-guided metrics, e.g., Reasonability 0.419 vs. 0.381, Detail 0.477 vs. 0.411) provide quantitative evidence that narrative-aware frame selection improves downstream reasoning quality beyond what anomaly saliency alone achieves. This suggests that future video-language systems — not just for anomaly understanding, but for any task requiring causal reasoning about events — should treat temporal segmentation into narrative phases as a first-class preprocessing step rather than assuming uniform or saliency-based sampling is sufficient.

The frozen-backbone + domain-encoder design pattern. VADER provides a concrete, validated instance of a design pattern with broader implications: keep the general-purpose MLLM frozen, train a lightweight domain-specific encoder (CORE) on a task-relevant signal (relational volatility), and connect them through learnable projectors and LoRA adapters. The fact that this works — that a frozen NVILA can extract additional signal from CORE's tokens (Table 5: removing CORE drops BLEU 0.718 → 0.668, Consistency 0.357 → 0.343) — establishes that domain-specific structured representations can be integrated with general-purpose multimodal models without full end-to-end retraining. This pattern lowers the barrier to entry for applying MLLMs to specialized video understanding tasks: domain experts can focus on designing the right relational encoder for their task, then plug it into a frozen off-the-shelf MLLM with minimal fine-tuning.

Which research directions become more attractive and which become less so.

  • More attractive: continuous relational representation learning for video (extending beyond scene graph vocabularies), weakly supervised mining of interaction transitions as anomaly signals (the volatility-based approach), narrative-aware temporal segmentation as a preprocessing step for video-language models, and the frozen-backbone + domain-encoder design pattern for other specialized video domains (medical procedure monitoring, industrial inspection, sports analytics).
  • Less attractive: expanding discrete scene graph predicate vocabularies as a path to better anomaly understanding (the vocabulary bottleneck is structural, not scale-limited), end-to-end fine-tuning of large MLLMs on small anomaly datasets (VADER demonstrates that lightweight adaptation of a frozen backbone achieves strong results, and the "w/o Fine-Tuning" ablation in Table 5 shows that zero-shot MLLMs produce verbose but inaccurate outputs — suggesting that domain adaptation matters more than backbone scale), and frame selection strategies that optimize for anomaly saliency rather than narrative structure (Top-K underperforms CAES on causal reasoning metrics despite having access to the same anomalous frames).

Follow-Up Research This Work Enables

1. Compute-matched comparison: does CAES + CORE beat uniform sampling with an equivalent total compute budget? VADER's inference is ~75% slower than the NVILA backbone alone (19.22 vs. 33.63 fps, Table 10). The ablation in Table 6 compares CAES against uniform sampling at the same frame budget (64 frames), but a deployment-relevant comparison would match total compute time: give the uniform-sampling baseline 112 frames (64 × 1.75) within the same wall-clock time as VADER's 64-frame CAES pipeline, fine-tune the same NVILA + LoRA configuration on those 112 uniformly-sampled frames, and compare all HAWK metrics. If uniformly sampling more frames recovers most or all of VADER's gains, then CAES's advantage is primarily about frame budget efficiency rather than narrative structure — a practical but less fundamental finding. If CAES still leads despite the frame budget disadvantage, the narrative structure claim is substantially strengthened. This experiment would also quantify the cost-effectiveness tradeoff that practitioners actually face.

2. Static vs. dynamic CORE: does the temporal dynamics claim hold under direct ablation? Train two identical CORE encoders: one on relational-change pairs $[\mathbf{r}_{ij}(t-1); \mathbf{r}_{ij}(t)]$ from volatility peaks (as in VADER), and one on single-frame relational tensors $\mathbf{r}_{ij}(t)$ from the same CAES-selected frames, using the same triplet loss architecture but with positives/negatives defined by per-frame anomaly scores rather than volatility peaks. Compare the resulting relation tokens on the full HAWK evaluation. If the dynamic encoder significantly outperforms the static encoder, the paper's central claim — that encoding interaction transitions specifically drives the gains — is validated. If the static encoder performs comparably, then VADER's advantage comes from having any continuous relational encoding (vs. discrete scene graph labels or visual-only cues) rather than from modeling temporal dynamics per se, which would refocus future work on continuous relational representation learning rather than on transition detection.

3. False positive analysis of volatility peak mining: what fraction of top-5% peaks correspond to the annotated anomalous event? For a subset of HAWK or CUVA videos that have temporal annotations of anomalous event boundaries, manually label whether each top-5% volatility peak falls within the annotated anomalous interval, in the pre-event context, in the post-event context, or in an unrelated segment. This would produce a precision curve for the mining heuristic. If precision is high (e.g., >80%), the weak supervision assumption is validated and the approach can be recommended with confidence. If precision is low (e.g., <50%), CORE is being trained substantially on spurious relational changes, which would explain the modest AUC (72.12% in Table 9) and suggest that improving the mining heuristic — perhaps by incorporating anomaly class predictions from the Anomaly Scorer to filter peaks by semantic relevance — could yield large downstream gains. This experiment also enables a breakdown of VADER's failure modes by mining precision: do videos with low peak precision produce worse descriptions?

4. Cross-backbone transfer: do CORE's relation tokens transfer to a different frozen MLLM without retraining? Fix a trained VADER pipeline (CAES, CORE, and projectors trained with NVILA as the backbone). Replace NVILA with a different video-capable MLLM (e.g., Video-LLaVA, LLaVA-Next-Video, Qwen2-VL), keeping the new backbone frozen. Train only the projector from CORE's output dimension to the new backbone's hidden dimension and LoRA adapters on the new backbone, using the same HAWK training data. If performance approaches VADER-NVILA levels, the relational tokens are backbone-agnostic — they encode interaction structure that any capable MLLM can exploit — which would make CORE a plug-and-play module for the broader video understanding ecosystem. If performance degrades substantially, the token representations are specific to NVILA's pre-training distribution or attention patterns, and backbone-specific training would be needed for each deployment.

5. Difficulty-stratified evaluation: which anomaly types does VADER handle well, and which does it systematically fail on? The paper acknowledges biases toward high-motion and object-centric anomalies (Section 5, Appendix C) but provides no quantitative breakdown. A follow-up should annotate the HAWK or CUVA test set with anomaly type categories — physical altercations, traffic accidents, theft/burglary, abandoned objects, crowd behaviors, environmental anomalies, loitering/suspicious inactivity — and report per-category metrics (BLEU, GPT-guided Consistency, and a new human-judged factual accuracy score). Categories with low relational volatility (abandoned objects, loitering) should show substantially lower performance if the acknowledged bias is real. Categories requiring group-level reasoning (crowd behaviors) should also underperform. This breakdown would produce a deployability map: practitioners could consult it to determine whether VADER is appropriate for their specific surveillance context (e.g., a train station with frequent abandoned luggage vs. a stadium with crowd management priorities).

6. End-to-end fine-tuning of the upstream modules: can the Anomaly Scorer and Relation Feature Extractor be jointly optimized with the LLM objective? VADER keeps the Anomaly Scorer and EGTR frozen, which makes errors in detection or association propagate to the final output (Limitation 1). A follow-up would unfreeze the Anomaly Scorer's final layers and EGTR's relational feature head, and fine-tune them jointly with the projectors and LoRA adapters using the same autoregressive language modeling loss on HAWK description data. This would test whether upstream modules can adapt their representations to better serve the downstream reasoning task — e.g., the Anomaly Scorer might learn to produce anomaly scores that better predict narrative boundaries, and EGTR might learn relational features that are more discriminative for anomaly transitions. The risk is catastrophic forgetting in the upstream modules (they were pre-trained on different objectives), so a careful comparison with frozen baselines and intermediate-task evaluation (does anomaly detection AUC degrade after joint fine-tuning?) would be essential. If successful, this would transform VADER from a modular pipeline into an end-to-end trainable system, potentially closing the performance gap on anomaly types where frozen upstream modules currently fail.

Practical Applications and Downstream Use Cases

1. Forensic video review in security operations centers. Security personnel reviewing hours of surveillance footage after an incident needs to quickly understand what happened across multiple camera feeds. VADER's ability to generate causally-structured descriptions — including what led up to an event, what occurred, and what followed — directly supports this workflow. The 64-frame narrative representation means a human reviewer can read a VADER-generated summary (covering pre-event, event, and post-event context) and then watch only the specific keyframes if they need visual confirmation, rather than scanning the full video. The +0.145 improvement in GPT-guided Reasonability on HAWK descriptions (Table 4: 0.428 vs. 0.283 for HAWK) translates, in practical terms, to descriptions that a human operator is more likely to find coherent and actionable on first reading without needing to double-check the footage. The 19 fps inference speed (Table 10) is adequate for offline forensic review, where videos are processed in batches after the fact rather than in real time.

2. Automated incident report generation for traffic management systems. Traffic monitoring centers need structured reports of accidents, near-misses, and traffic violations for insurance claims, safety analysis, and infrastructure planning. VADER's CUVA results — particularly the +7.38 MMEval improvement on the Causes task (Table 2: 66.30 vs. 58.92) and the +27.71 UniEval improvement on the Effect task (82.06 vs. 54.35 for Video-ChatGPT) — suggest the system can generate reports that not only describe a collision but explain contributing factors (a vehicle running a red light) and consequences (traffic backup, secondary collisions). This transforms the output from a detection alert ("collision at intersection 14 at 08:23:15") to an actionable report that can be routed to insurance adjusters, traffic engineers, or law enforcement with minimal human rewriting. The limitation to high-motion events (Section 5) means the system would be most reliable for collisions and near-misses, while more subtle traffic anomalies (illegal parking, wrong-way driving at low speed) would require the proposed object state modeling extensions.

3. Training data generation for specialized surveillance AI systems. Deploying anomaly detection or understanding systems at a new site typically requires annotated data from that site's camera angles, lighting conditions, and typical activity patterns — which is expensive to collect. VADER could be deployed as a pseudo-labeler: run it on raw surveillance footage from the new site, generate descriptions and causal explanations, and use these as training targets for a smaller, site-specific model (e.g., a lightweight video captioner fine-tuned for that camera). VADER's modular design — frozen backbone, lightweight trainable components — makes this practical: the system can be run once to generate a training corpus, then the smaller model is trained offline, and VADER itself doesn't need to run in production. The key question for this use case is VADER's false positive rate on generated descriptions, which the paper does not quantify — deploying this would require a human audit step to filter clearly incorrect pseudo-labels before training the site-specific model.

4. Priority triage for real-time surveillance monitoring. In a control room with dozens of camera feeds and limited human attention, VADER's descriptions enable semantic triage beyond simple anomaly score thresholds. Instead of a binary alert ("anomaly on camera 7"), the system can surface a one-line description ("physical altercation between two individuals near the platform edge on camera 7") that lets the operator assess severity without switching their gaze. Cameras showing pre-event escalation (CAES's rising slope detection) could be flagged as "developing situations" for proactive monitoring. The 19 fps speed (Table 10) is below real-time for a single high-frame-rate feed, but in a multi-camera setup where each camera is processed at lower temporal resolution (e.g., 1-2 fps for anomaly screening, sufficient for most surveillance scenarios), VADER could keep up with multiple streams. The main deployment risk is VADER's acknowledged bias toward high-motion events — a developing situation with subtle cues (someone loitering near a restricted entrance) might not trigger sufficient relational volatility to be selected for description, causing missed alerts for exactly the kind of precursor behavior that proactive monitoring aims to catch.

When to Prefer This Method

The paper does not explicitly frame VADER against named alternative deployment strategies (e.g., larger MLLM with simpler frame selection vs. VADER's specialized pipeline) in a way that articulates clear decision boundaries. The ablation tables demonstrate that each component matters, but the paper does not provide a compute-matched comparison or a difficulty-stratified breakdown that would enable a practitioner to decide, for a specific deployment scenario, whether VADER's overhead is justified over a simpler baseline. Without such analysis from the paper itself, a "Prefer VADER when..." decision rule would be based on the reviewer's inference rather than on claims the paper makes and supports. The limitations in Section 5 do identify conditions where VADER's current design is weaker (low-motion anomalies, scene-level events, group behaviors), but these are stated as limitations to address in future work rather than as part of a comparative tradeoff analysis against named alternatives.