Proact-VL: A Proactive VideoLLM for Real-Time AI Companions

ArXiv: 2603.03447

🎯 Pitch

Proact-VL achieves a 15% F1 improvement over the best prior real-time video model by learning when to speak—not just what to say—through a lightweight gating mechanism that cuts latency by 70%. This enables AI companions to interject naturally in live streams, matching human commentary timing while maintaining strong video understanding.

1. Executive Summary

This paper introduces Proact-VL, a framework that shapes multimodal language models into proactive, real-time interactive agents for AI companionship, instantiated through gaming commentary and guidance scenarios and evaluated on the newly constructed Live Gaming Benchmark. The framework combines three key mechanisms: a chunk-wise input-output schema for continuous streaming (processing video in one-second intervals), a lightweight proactive response mechanism that autonomously decides when to speak (a gated MLP head operating on a special <|FLAG|> token hidden state), and a multi-tier training loss with transition-smoothed classification and stability regularization for coherent speaking behavior. Proact-VL achieves superior response timing and text quality across solo commentary, co-commentary, and guidance settings, attaining the best overall F1 of 64.87 and substantially outperforming prior real-time baselines (e.g., LiveCC-7B-Instruct at 56.05 F1) while maintaining robust video understanding capabilities, establishing that effective real-time AI companionship requires jointly optimizing both what to say and when to speak rather than treating these as independent problems.

2. Context and Motivation

The Core Problem: Real-Time AI Companions Require More Than Just Good Text Generation

The fundamental challenge this paper addresses sits at the intersection of three requirements that have historically been studied in isolation: low-latency streaming video understanding, autonomous decisions about when to speak, and controlled generation of appropriately-timed short utterances. A human-like AI companion — whether a game commentator, a live-stream co-host, or a real-time guide — must do all three simultaneously. It must watch a continuous video feed, decide which moments warrant a response, and produce commentary that fits within tight temporal windows (approximately one second), all without falling behind the live stream.

This is not simply a harder version of video question-answering. In offline video understanding, a model receives the entire video, processes it holistically, and generates a complete answer. There is no time pressure, no need to decide when to interject, and no constraint that the output must be delivered in real-time bursts. The shift to real-time, proactive interaction transforms the problem fundamentally: the model must operate causally (only seeing past frames), decide autonomously whether the current second merits speech or silence, and generate content that is immediately consumable — brief enough to be spoken in roughly one second while remaining contextually coherent with the ongoing stream.

The paper frames this through two concrete gaming applications — commentator and guide — which the authors argue are "selected for their suitability for automatic evaluation" (Section 1). Gaming provides a natural testbed: professional commentary already exhibits the desired behavior (humans speaking in short bursts at appropriate moments), guide scenarios have clear success criteria (did the player follow the advice and succeed?), and the domain offers rich, publicly available training data from platforms like YouTube. But the problem generalizes far beyond gaming — any application where an AI must watch a live feed and interact naturally (security monitoring, surgical assistance, live tutoring, accessibility tools for the visually impaired) faces the same core tension between when to speak, what to say, and how to say it within real-time constraints.

Three Simultaneous Challenges, Not One

The paper explicitly identifies three challenges that must be solved together (Section 1):

1. Achieving low-latency inference under continuous streaming inputs. The model cannot batch-process an entire video; it must ingest frames as they arrive and produce outputs without accumulating lag. This requires efficient KV-cache management and a processing cadence matched to real-time delivery (one second in this work).

2. Autonomously deciding when to respond. Unlike conventional VLMs that only speak when prompted, a companion must determine for itself which moments merit commentary. Constant talking disrupts the user experience, while excessive silence undermines the sense of companionship. There's no external signal — the model must learn this from observing human commentary patterns.

3. Controlling both quality and quantity of generated content to meet real-time constraints. Even when the model correctly decides to speak, it must produce content that fits within a ~1-second delivery window. This is fundamentally different from generating a complete, multi-sentence answer: the model must chunk its thoughts into clip-level utterances, maintaining coherence across multiple speaking turns while each individual utterance remains self-contained enough to be understood immediately.

These three challenges are deeply intertwined. A model with perfect timing but poor content quality is useless. A model that generates beautiful commentary but with 5-second latency is unusable for live streams. A model that produces perfect real-time text but never stops talking is annoying. The paper's central thesis is that these must be optimized jointly — prior work's failure to do so explains the gap between existing systems and human-like interaction.

Prior Approaches and Their Systematic Shortcomings

The paper organizes prior work on real-time video understanding into two threads, each solving part of the problem but failing on the full picture (Section 2.2).

Thread 1: Proactive Models That Decide When to Speak — But Speak Too Much When They Do

Proactive models — including VideoLLM-online (Chen et al., 2024b), MMDuet (Wang et al., 2024b), LiveStar (Yang et al., 2025), and others (Fu et al., 2025; Liao et al., 2025; Qian et al., 2025; Ding et al., 2025; Wang et al., 2025a) — tackle the "when to speak" problem by learning a policy or lightweight network that decides whether the current video context warrants a response. This is a genuine advance over purely reactive models: it means the system can initiate interaction rather than waiting for a user prompt.

However, the paper identifies a critical flaw: "once triggered, the model generates a complete, relatively long answer" (Section 1). The proactive decision is binary and coarse — speak now or don't — but once the decision is made, the model defaults to generating a full response as if answering an offline question. In the context of live commentary, this produces responses that are too lengthy and too high-latency. Imagine a sports commentator who, upon deciding a goal is worth mentioning, launches into a 30-second analysis while the game continues — by the time they finish, two more goals may have been scored.

The paper quantifies this failure mode in Table 2. Proactive baselines like MMDuet achieve abysmal F1 scores on timing metrics: 0.05 in Solo Commentary, 0.16 in Co-Commentary, 0.32 in Guidance (overall F1 of 0.18). Their TimeDiff scores are extremely high — MMDuet records 27.85 seconds in Solo Commentary, meaning its triggered responses are, on average, nearly half a minute away from where human commentators actually spoke. These models learned that they should speak at certain moments, but not how to speak in the brief, clip-level bursts that real-time commentary demands.

The paper summarizes this limitation pointedly: "triggered responses tend to be lengthy and high-latency, which is unsuitable for video commentary" (Section 2.2).

Thread 2: Real-Time Models That Emphasize Low Latency — But Can't Stop Talking

The second line of work — represented by LiveCC (Chen et al., 2025) and StreamingVLM (Xu et al., 2025c) — optimizes for streaming video understanding with low-latency response generation. LiveCC scales up streaming-style supervision to produce sentence-level outputs at a one-second cadence. StreamingVLM optimizes attention and caching mechanisms to support effectively unbounded video understanding.

These models solve the latency problem: they can ingest video chunks and produce text quickly enough for real-time use. But they "often provide limited control over when the model should speak" (Section 2.2). Because they are trained to produce output at every time step (each second gets a caption), they lack an explicit silence mechanism. The result is excessive talking — the model narrates continuously, which is appropriate for dense video captioning tasks but inappropriate for companionship, where strategic silence is as important as timely speech.

The paper's metrics capture this failure mode as well. In Table 2, LiveCC-7B-Base achieves reasonable but not outstanding timing: F1 of 47.05 in Solo and 43.25 in Co-Commentary. These are far better than the proactive models but fall short of human alignment. StreamingVLM does better on F1 (56.92 Solo) but shows the complementary weakness: without explicit proactivity training, its speaking decisions are driven by the captioning objective rather than by learning when human commentators actually choose to speak.

The Fundamental Tension: No Prior System Does Both Well

The paper's diagnosis is that these two threads represent a tradeoff that shouldn't exist. Proactive models solve the if (speak or not) but fail on the how (brief, clip-level output). Real-time models solve the how (low-latency generation) but fail on the if (they lack explicit speaking control). The paper states this tension directly:

"Existing methods struggle to balance proactivity timing with content quality in complex, real-world scenarios." (Section 1)

This is not merely an implementation detail that could be fixed by combining existing components. The two objectives interact: the decision to speak depends on whether the model can actually produce useful content in the available time window, and the content generation depends on whether the model has correctly identified a moment worth speaking about. A system that first decides to speak and then generates a long answer fails on latency; a system that generates text at every time step and then filters post-hoc wastes computation on unwanted output and struggles with coherence. The paper argues these must be trained jointly with objectives that explicitly reward both good timing and good content.

The Data Gap: No Benchmark for Proactive, Real-Time Commentary

Beyond the methodological gap, the paper identifies a critical resource gap: no existing dataset provides the right kind of supervision for training and evaluating proactive, real-time commentary systems.

Standard video QA datasets (MVBench, Ego4D, etc.) are designed for offline evaluation — given a video and a question, produce an answer. They don't provide per-second speaking labels, don't capture the temporal dynamics of when commentary should occur, and don't include the multi-speaker coordination patterns of co-commentary.

The authors needed data with three specific properties:

1. Per-second speech/silence labels aligned with human commentary patterns — so the model can learn when to speak from real human behavior, not from a heuristic.
2. Short, timestamped utterances — so the model can learn to produce content appropriate for ~1-second delivery, rather than multi-sentence monologues.
3. Multi-speaker scenarios — so the model can learn turn-taking in co-commentary, where speaking at the wrong time means interrupting a human collaborator.

No public dataset satisfied these requirements for the gaming domain. Hence the construction of the Live Gaming Dataset (561 hours of commentary across 12 games, processed through a custom pipeline involving ASR, speaker identification, paralinguistic labeling, and domain-specific polishing, as detailed in Section 3.2).

How This Paper Positions Itself

Proact-VL situates itself as bridging the proactive/real-time divide — a model that is simultaneously proactive (decides when to speak) and real-time (generates short, low-latency utterances). The key architectural insight is the decide-then-generate pipeline: at each one-second chunk, the model first computes a speaking probability via a lightweight response head, and only if that probability exceeds a threshold does it generate a brief commentary clip. This decouples the binary speak/silence decision from the content generation process, allowing each to be optimized with its own objective.

Critically, the paper does not claim to have invented chunk-wise processing or proactive triggering. Both ideas existed in prior work. The contribution is in how these components are integrated and trained:

• The response decision uses a dedicated <|FLAG|> token and MLP head rather than relying on token-generation probabilities (which are sensitive to decoding hyperparameters and lead to unstable thresholding, as argued in Appendix A.1).
• The training objective combines causal language modeling with a specialized response loss that includes transition-aware weighting (emphasizing rare speak↔silence transitions over state persistence) and stability regularization (suppressing jitter and calibrating the overall speaking rate to match human baselines). This joint optimization ensures the model learns both what to say and when to say it from the same signal, avoiding the coordination failures of pipelined approaches.
• The infinite inference mechanism (dual-cache with reverse-RoPE correction, Appendix A.2) enables stable operation over arbitrarily long streams without positional encoding drift — a practical necessity for real deployment that prior streaming models handled differently or not at all.

The paper's position is thus: proactivity and real-time generation are not separate problems to be solved by separate systems and combined post-hoc; they are two facets of a single interactive behavior that must be learned jointly from data that captures both dimensions simultaneously. The Live Gaming Dataset provides that data, and Proact-VL's architecture and training strategy provide the framework for learning from it. The experimental results — particularly the stark contrast between Proact-VL's F1 scores (63.25 Solo, 77.44 Co-Commentary, 53.91 Guidance) and the near-zero F1 scores of prior proactive models (MMDuet: 0.05 Solo) — validate that this joint approach resolves the tradeoff that prior systems faced.

3. Technical Approach

3.1 Reader Orientation

Proact-VL is a fine-tuned video large language model that watches a live video stream and acts as a real-time commentator or guide, deciding for itself when to speak and generating short, time-appropriate utterances. The system solves the joint problem of low-latency streaming perception, autonomous speak/silence decisions, and controlled-length generation by structuring the model as a chunk-wise processor with a dedicated lightweight trigger mechanism, trained end-to-end with objectives that simultaneously reward speaking at the right times and producing good content.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components arranged in a continuous loop:

1. Streaming Video Input — the raw video feed, discretized into fixed one-second chunks. Each chunk becomes a visual token sequence (frames sampled at 2 FPS, encoded by a vision encoder, with a per-chunk pixel budget controlled by resolution constraints).

2. Chunk-Wise Input Schema — at each time step t, the system assembles a structured input triplet: the current video chunk $V_t$, an optional user query $Q_t$, and environmental context $B_t$ (which includes summaries of previous commentary from the model itself and any co-commentators). These are serialized into a ChatML-style message format with special delimiter tokens.

3. Persistent Transformer with KV Cache — the core language model (initialized from LiveCC-7B-Base or a Qwen-series backbone) processes the input autoregressively, maintaining a key-value cache $\mathcal{K}_{t-1}$ from all previous time steps. This enables incremental processing: the model only encodes the new chunk, attending to cached representations of all prior context.

4. Proactive Response Mechanism — after processing the current chunk's input, the model extracts the hidden state at a special <|FLAG|> token and passes it through a lightweight gated MLP head with sigmoid activation to produce a speaking probability $p_t$. If $p_t$ exceeds a learned threshold $\tau$, the model generates a short commentary utterance; otherwise, it outputs a silence placeholder. This "decide-then-generate" structure decouples the binary speak/silence decision from content generation.

5. Infinite Inference Engine — when the cumulative context exceeds the model's maximum context length, a dual-cache sliding-window mechanism evicts the oldest streaming tokens while preserving the system prompt, then applies a reverse-RoPE correction to realign positional encodings across the eviction boundary, enabling unbounded streaming without positional drift.

Information flows cyclically: video frame arrives → encoded into visual tokens → assembled with history and query into a ChatML message → processed by transformer (using cached prior context) → <|FLAG|> hidden state extracted → response head computes $p_t$ → if above threshold, model generates short utterance $U_t$ → $U_t$ appended to context stream for time step $t+1$ → loop continues indefinitely.

3.3 Roadmap for the Deep Dive

• First, the chunk-wise input schema — how continuous video is discretized into time steps, how the input triplet $(V_t, Q_t, B_t)$ is structured, and how the ChatML template organizes these signals for the model. This is the foundation that enables all downstream processing.

• Second, the proactive response mechanism — the <|FLAG|> token, the gated MLP head, the thresholding logic, and the design rationale for separating the speak/silence decision from token generation. This is the core architectural innovation.

• Third, the training strategy — the four-part loss function combining causal language modeling with transition-smoothed classification and stability regularization. Understanding why each term exists and how they interact is essential to grasping why the system works.

• Fourth, the infinite inference mechanism — the dual-cache architecture, the eviction strategy, and the reverse-RoPE correction for positional continuity. This is what makes the system deployable for arbitrarily long streams.

3.4 Detailed, Sentence-Based Technical Breakdown

This is a systems-building paper whose core idea is that real-time proactive interaction requires jointly optimizing the when to speak decision and the what to say generation, implemented through a chunk-wise streaming architecture with a dedicated response head and a multi-objective training loss that explicitly penalizes timing errors and speaking-rate misalignment.

Chunk-Wise Input Schema: Discretizing Continuous Video into Processable Time Steps

The first design challenge is converting a continuous, unbounded video stream into discrete units that a transformer can process causally. A transformer has finite context length; a live stream has no predetermined end. The solution is a fixed-cadence chunking scheme where time is divided into one-second intervals, with the model producing a decision and optionally an utterance at each interval boundary.

Input triplet construction. At each time step $t$, the model receives three sources of information assembled into a single structured input:

• $V_t$: the visual content occurring during the current one-second window. Video is decoded at 2 FPS, so a one-second chunk contains 2 frames. Each frame is encoded by the vision encoder into visual tokens. The per-frame pixel budget is governed by a resolution policy: MIN_PIXELS = 128 × 28 × 28 and MAX_PIXELS = 540 × 28 × 28, with an additional aggregate cap MAX_VIDEO_PIXELS = 36 × 540 × 28 × 28. The per-frame budget is set to $\min(\text{MAX\_PIXELS}, \text{MAX\_VIDEO\_PIXELS} / T)$ where $T$ is the chunk duration in seconds. This adaptive budgeting prevents individual high-resolution frames from exhausting the context window.

• $Q_t$: an optional user query that provides immediate interaction context. In the solo commentary setting, $Q_t$ may be empty. In the guidance setting, $Q_t$ contains the user's question or stated goal. The query is delimited by special tokens <|query_start|> and <|query_end|>.

• $B_t$: environmental context, primarily containing summaries of previous commentary. In solo commentary, this is the model's own prior utterances. In co-commentary, it includes other assistants' last-second commentary, enabling turn-taking coordination. The context is delimited by <|history_start|> and <|history_end|>.

ChatML serialization. These three signals are serialized into a ChatML-style message format (illustrated in Figure 6, Appendix A.1) with the following fixed ordering:

User:
<|history_start|> [environment context] <|history_end|>
<|vision_bos|> [video chunk visual tokens] <|vision_eos|>
<|query_start|> [user query] <|query_end|>
<|FLAG|>

The ordering is intentional and motivated by three considerations described in Appendix A.1:

1. History before video: the environmental context (last-second commentary from co-commentators) is not directly observable from the current video chunk. Placing it first allows the model to incorporate this contextual signal early in processing, improving continuity and co-commentator coordination.

2. Video before query: the video chunk represents the primary grounding signal. Placing it before the query ensures visual information is available when the model attends to the user's request.

3. Query before <|FLAG|>: the user query is semantically closest to the assistant's response. Placing it immediately before the decision token makes the query easier to attend to near generation time, strengthening user-intent conditioning.

Causal auto-regressive processing with persistent cache. The model processes this input sequentially within a single forward pass, but the key architectural insight is incremental processing across time steps. The formal operation is:

$(U_t, \mathcal{K}_t) = f_\theta(V_t, Q_t, B_t; \mathcal{K}_{t-1})$

where $f_\theta$ is the transformer with parameters $\theta$, $U_t$ is the generated utterance at step $t$, $\mathcal{K}_{t-1}$ is the key-value cache from all prior time steps, and $\mathcal{K}_t$ is the updated cache after including the new tokens.

What this computes: at each one-second boundary, the model takes the new video chunk, query, and context, plus the compressed representation of all prior history stored in the KV cache, and produces two outputs — the utterance text (if the response mechanism triggers) and the updated cache for the next step. The KV cache prevents re-encoding the entire history from scratch at every second, which is what makes real-time operation computationally feasible.

Why this form: the critical property is that $U_t$ becomes part of the input for $t+1$. The model's own generated utterances are automatically appended to the context stream, creating a continuous dialogue history without external management. This is what enables multi-second commentary sequences where later utterances reference earlier ones. Alternative designs that treated each second independently would lose this coherence — the model would have no memory of what it just said.

A subtle design decision: why maintain user-assistant alternation during silence? Appendix A.1 explains that even when the model stays silent, the ChatML structure preserves the <user><assistant><user><assistant> alternation by inserting a silence placeholder (e.g., "...") into the assistant turn. If silent timesteps omitted the assistant turn entirely, the conversation structure would deviate from the base model's pretraining distribution, potentially destabilizing generation. This "fill-with-silence" strategy preserves structural consistency while allowing the response head to independently control whether actual content is generated.

Proactive Response Mechanism: Deciding When to Speak Without Generating Text

The second design challenge is the speak/silence decision. Prior proactive models embedded this decision in the token generation process itself — the model would either generate a special silence token or begin generating commentary. The paper identifies two problems with this approach (Appendix A.1):

1. Decoding sensitivity: the probability of generating a <|SILENCE|> token varies substantially with decoding hyperparameters (temperature, top-p), making threshold calibration unreliable. A small change in temperature could transform a mostly-silent model into one that chatters continuously.

2. Training instability: treating silence as a frequent assistant-side token creates a highly imbalanced training pattern — many samples become Assistant: <|SILENCE|> — which can destabilize optimization and lead the model to collapse into always-silence or always-speaking modes.

The solution: a dedicated response head. Proact-VL decouples the binary decision from token generation entirely. The mechanism works as follows:

1. Insert a special token: a semantic-free marker <|FLAG|> is appended at the end of every user message (after the query). This token carries no inherent meaning — it serves purely as an anchor point for extracting a decision-relevant hidden state. Crucially, the authors intentionally avoid reusing any existing token as the decision anchor, because common tokens carry semantic content that could introduce unintended priors. A token like "?" or "respond" would bias the model toward speaking whenever that token's semantics are relevant, independent of whether the current moment actually warrants commentary.

2. Extract the hidden state: after the transformer processes the complete input for time step $t$, the hidden state $\mathbf{h}_t$ corresponding to the <|FLAG|> token position is extracted. This vector is a function of all preceding context — the video chunk, the user query, the environmental history, and all prior conversation stored in the KV cache — compressed through the transformer's layers into a single representation. Specifically, the paper uses the hidden state from the penultimate transformer layer (Appendix C.3), not the final layer, which may capture more task-relevant features before the last-layer transformations.

3. Compute speaking probability: a minimal gated MLP head maps this hidden state to a scalar probability:

$p_t = \sigma(\text{MLP}(\mathbf{h}_t))$

where $\sigma$ is the sigmoid function, $\text{MLP}$ is a small multi-layer perceptron (the paper describes it as a "minimal gated MLP head," implying the presence of gating mechanisms but not detailing the exact architecture), and $\mathbf{h}_t$ is the hidden state at the <|FLAG|> token from the penultimate layer.

What this computes: given all available context at time $t$, the probability that a human commentator would speak at this moment. The MLP head learns to map from the high-dimensional transformer hidden state (which encodes complex visual, textual, and temporal features) to a single scalar between 0 and 1.

Why this form: a scalar probability with a fixed threshold is far more controllable than a token-generation probability. The threshold $\tau$ can be adjusted at deployment time to trade off speaking frequency against precision, without retraining the model. The sigmoid ensures the output is well-calibrated as a probability, and the gated MLP provides sufficient capacity to learn complex decision boundaries without adding substantial computational overhead — the head is "lightweight," meaning it adds negligible latency to the per-chunk processing time.

4. Binary decision via thresholding: the probability is compared against a fixed threshold $\tau$:

$a_t = \mathbb{I}[p_t \geq \tau]$

where $a_t = 1$ triggers utterance generation and $a_t = 0$ maintains silence. The paper uses $\tau = 0.3$ for main results (Sections 5.2–5.4) and $\tau = 0.5$ for analyses and ablations (Appendix C.3). The threshold sensitivity analysis in Appendix I.1 (Figure 8) shows that thresholds in the $0.3$–$0.5$ range provide a stable balance: lower thresholds increase F1 but reduce commentary consistency (CC), while higher thresholds improve CC at the cost of coverage.

5. Triggered generation: if $a_t = 1$, the model appends the Assistant: prefix and autoregressively generates a short commentary clip. The generation is constrained to fit within the real-time delivery window (roughly one second of speech). The generated utterance $U_t$ then becomes part of the context for time step $t+1$.

6. Silence handling: if $a_t = 0$, the model appends a fixed silence placeholder (e.g., "...") into the assistant turn. This maintains the ChatML structure while explicitly representing the decision not to speak.

Why this architecture over alternatives: Appendix A.1 provides a detailed justification contrasting this design with token-based silence prediction. The key advantages are:

• Threshold calibration independence: the decision threshold operates on a dedicated probability $p_t$ that is independent of text generation hyperparameters. If the same decision were embedded in token probabilities, adjusting temperature for generation quality would simultaneously alter speaking behavior, making the system impossible to tune independently for content and timing.

• Training stability: by decoupling the binary decision from the language modeling objective, the response head can be trained with a specialized loss (described in Section 4.3) without interfering with the main causal language modeling loss. The imbalanced silence/speak ratio becomes a classification problem with explicit weighting rather than a token prediction problem that distorts the language model's output distribution.

• Deployment flexibility: $\tau$ provides a single, interpretable knob for controlling speaking frequency at inference time. Operators can tune this based on application requirements — a lower threshold for high-engagement scenarios, a higher threshold for background commentary — without retraining or modifying the text generation pipeline.

A subtle implementation detail (Appendix C.3): the response head receives the hidden state from the penultimate transformer layer, not the final layer. This choice likely reflects the observation that the final layer's representations are optimized for next-token prediction, while intermediate layers may retain more general features useful for the binary classification task. The paper does not ablate this choice, but it is consistent with the broader literature on probing classifiers, which often find that intermediate layers contain more task-generalizable representations than the final layer.

Training Strategy: A Four-Part Loss for Joint Optimization of Content and Timing

The training strategy is where Proact-VL's integration of proactivity and real-time generation is realized. The model is trained with a composite loss function that simultaneously optimizes what to say (via causal language modeling) and when to speak (via a specialized response loss with transition weighting and stability regularization).

The overall objective is:

$\mathcal{L} = \mathcal{L}_{\text{main}} + \alpha \mathcal{L}_{\text{resp}}$

where $\mathcal{L}_{\text{main}}$ is the standard causal language modeling loss, $\mathcal{L}_{\text{resp}}$ is the response behavior loss, and $\alpha = 0.2$ is a scalar weight balancing the two objectives. The paper sets this weight empirically; Appendix C.3 notes that all hyperparameters are kept identical across backbone models for fair comparison.

Primary Language Modeling Loss: Learning What to Say

$\mathcal{L}_{\text{main}}$ is the standard autoregressive next-token prediction loss applied only to assistant-generated tokens (not system prompts, user messages, or silence placeholders). Specifically:

"For the main causal language modeling loss $\mathcal{L}_{\text{main}}$, we compute loss only on tokens generated by the active assistant, up to <|im_end|>, and mask out all tokens corresponding to silent assistants." (Appendix C.3)

This masking ensures that the model learns to generate good commentary when it speaks, without being penalized for producing silence tokens or system messages. The base model (LiveCC-7B-Base) already has strong language generation capabilities from its pretraining; $\mathcal{L}_{\text{main}}$ fine-tunes these capabilities for the specific domain of gaming commentary, conditioned on the ChatML template structure.

Response Loss: Learning When to Speak

The response loss $\mathcal{L}_{\text{resp}}$ is the key contribution — it is what enables the model to learn speaking behavior from human commentary patterns. The loss has three components that work together to produce stable, human-aligned speaking decisions.

Component 1: Transition-Smoothed Classification Loss

Given ground-truth speaking labels $y_t \in \{0, 1\}$ for each second (derived from human commentary patterns in the training data), a naive approach would be to treat each second as an independent binary classification problem and optimize per-step cross-entropy. The paper identifies a critical flaw in this approach:

"The per-second response state should not be treated as independent points; it is a sequence learning problem. The dominant imbalance is not simply the number of silence vs. response seconds, but the imbalance between state transitions and state persistence." (Section 4.3)

In human commentary, most seconds are persistent — if a commentator was speaking at $t-1$, they are likely still speaking at $t$; if they were silent, they likely remain silent. The rare events are transitions — the moments when a commentator starts speaking or falls silent. These transitions are the most informative events for learning proactivity (they define the boundaries of response-worthy moments), but they are drowned out by the overwhelming number of persistence seconds in a standard loss function.

The solution is transition-aware weighting:

$w_t = \begin{cases} \gamma & \text{if } y_t \neq y_{t-1} \\ 1 & \text{otherwise} \end{cases}$

where $\gamma = 5$ (set because the ratio of transition steps to persistence steps in the training set is approximately $1:5$). Transition steps receive $5\times$ the weight of persistence steps in the loss computation.

The weighted binary cross-entropy loss is then:

$\mathcal{L}_{\text{cls}} = \frac{1}{\sum_t w_t} \sum_t w_t \left(-y_t \log p_t - (1 - y_t) \log(1 - p_t)\right)$

where $y_t$ is the ground-truth speak/silence label at second $t$, $p_t$ is the model's predicted speaking probability from the response head, and $w_t$ is the transition-aware weight.

What this computes: a weighted average of per-second binary cross-entropy between the model's predicted speaking probability and the human ground truth, where the weight is 5 for seconds where the human's speaking state changed from the previous second (transition) and 1 for seconds where it stayed the same (persistence).

Why this form: without transition weighting, the loss would be dominated by persistence seconds — the model could achieve low loss simply by learning to maintain its current state, without ever learning when to change state. This would produce a model that either stays silent forever or talks continuously once triggered, which is exactly the failure mode of prior proactive systems. The $\gamma = 5$ weighting forces the model to pay disproportionate attention to the rare but critical transition events, learning the visual, textual, and temporal cues that signal the beginning or end of a response-worthy segment. The normalization by $\sum_t w_t$ ensures the loss magnitude remains comparable regardless of the sequence's transition density.

Component 2: Stability Regularization — Local Temporal Consistency

A well-calibrated classifier with transition weighting could still produce jittery behavior — rapid oscillations between speaking and silence within a single response-worthy segment. Human commentators don't stutter their speaking decisions; once they start commenting on an event, they continue for several seconds before falling silent. The first regularizer enforces this local smoothness:

$\mathcal{L}_{\text{reg, local}} = \mathbb{E}\left[(p_t - p_{t-1})^2 \mid y_t = y_{t-1}\right]$

where the expectation is taken over all time steps where the ground-truth state is persistent (i.e., the human commentator's state didn't change).

What this computes: the average squared difference between consecutive predicted probabilities, but only during segments where the ground truth indicates persistence. If the human was speaking at both $t-1$ and $t$, the model's predicted probability should not jump dramatically between these two seconds.

Why this form: this is a smoothness penalty, not an accuracy penalty. It doesn't force $p_t$ to equal any particular value — it only penalizes large changes in $p_t$ when the ground truth says the state should be stable. This allows the model to maintain whatever probability level is appropriate for the current segment (high during speech, low during silence) while suppressing the rapid oscillations that would produce unnatural, stutter-like commentary behavior. The conditioning on $y_t = y_{t-1}$ is crucial: during actual transitions, the model should change its probability rapidly, and this term correctly does not penalize those changes.

Component 3: Stability Regularization — Global Speaking Rate Calibration

Even with smooth transitions, the model could learn to speak much more or much less frequently than human commentators. The second regularizer enforces a global speaking rate constraint:

$\mathcal{L}_{\text{reg, global}} = \left(\mathbb{E}[p_t] - \mathbb{E}[y_t]\right)^2$

where $\mathbb{E}[p_t]$ is the model's average predicted speaking probability across all time steps, and $\mathbb{E}[y_t]$ is the average ground-truth speaking rate (treated as a constant computed from the training data).

What this computes: the squared difference between the model's average speaking probability and the human average speaking rate. If human commentators speak during 40% of seconds, this term penalizes the model for producing an average $p_t$ that deviates from 0.4.

Why this form: this provides a global "thermostat" on speaking behavior. Without it, the model could drift toward always-silent or always-speaking modes, especially under the transition-weighted loss which emphasizes decision boundaries but doesn't constrain the overall ratio. The squared error form penalizes both over-speaking (annoying constant chatter) and under-speaking (excessive silence that fails to provide companionship). Treating $\mathbb{E}[y_t]$ as a constant (not a batch-level estimate) means the target is stable across training, preventing the model from chasing a moving target as its own behavior changes.

Combined response loss. The three components are summed:

$\mathcal{L}_{\text{resp}} = \mathcal{L}_{\text{cls}} + \mathcal{L}_{\text{reg}}$

where $\mathcal{L}_{\text{reg}} = \mathcal{L}_{\text{reg, local}} + \mathcal{L}_{\text{reg, global}}$.

Why these three components together: the classification loss $\mathcal{L}_{\text{cls}}$ provides the primary supervision signal — it teaches the model which moments warrant speech. But alone, it produces jittery, poorly calibrated behavior because the per-second classification problem is too sparse and the transition-weighted variant, while fixing one issue, introduces sensitivity to decision boundaries without constraining what happens between boundaries. The local smoothness term suppresses jitter without interfering with the model's ability to learn genuine transitions. The global rate term prevents the model from drifting into degenerate regimes (always silent or always speaking) that the local objectives don't penalize. Together, they produce the smooth, human-aligned score curves visible in Figure 5.

Training hyperparameters (Appendix C.3). The model is fine-tuned with learning rate $1 \times 10^{-5}$ using a cosine scheduler, batch size 64, and 2,000 training steps (approximately 200 H100 GPU-hours). Gradient clipping is applied with max_grad_norm = 1.0. The system prompt is composed dynamically during training: a game-specific role, one randomly sampled persona (from the mined persona profiles), and one randomly sampled task template (from six prepared templates covering Solo Commentary, Co-Commentary, and Guidance). This randomization improves robustness to system-prompt variations.

Infinite Inference: Unbounded Streaming Without Positional Drift

A practical deployment challenge is that video streams are unbounded, while transformers have a fixed maximum context length (approximately 32K tokens for the Qwen-VL series used as backbones). Naively, as the stream continues, new tokens are assigned monotonically increasing position IDs. Over long streams, these position IDs can drift far beyond the range seen during training, weakening the model's ability to follow instructions and eventually destabilizing generation.

The paper's solution is a dual-cache sliding-window mechanism with reverse-RoPE correction (Appendix A.2 and A.3).

Dual-cache architecture. The KV cache is split into two parts:

• System cache: a persistent cache containing the initial system prompt (role, persona, task instructions). This never gets evicted because it defines the model's behavior and identity.

• Streaming cache: a dynamic cache containing interleaved user messages (video chunks, queries, history) and assistant messages (generated utterances or silence placeholders). This accumulates over time and is subject to eviction.

Eviction strategy. When the combined length of system cache, streaming cache, and current input tokens approaches the model's maximum context capacity, the oldest 20% of the streaming cache is removed. Specifically:

"We implement an eviction strategy that removes the oldest 20% of the streaming cache while preserving recent interactions. This selective eviction balances context retention with memory constraints, ensuring the model maintains awareness of recent dialogue while operating within computational limits." (Appendix A.2)

The 20% figure represents a balance: evicting too little means frequent evictions with high overhead; evicting too much loses recent context that might be relevant for understanding the current moment.

The positional discontinuity problem. When tokens are evicted from the middle of the sequence (the earliest streaming tokens, while the system cache and more recent streaming tokens remain), a gap opens in the position IDs. If position ID 10,000 is evicted but position ID 10,001 remains, the model sees a discontinuity — the effective sequence length shrinks but the position IDs don't reflect this. Even worse, if the remaining tokens keep their original position IDs, newly arriving tokens get assigned IDs in the 30,000+ range, far beyond what the model saw during training.

Reverse-RoPE correction. The solution exploits an algebraic property of Rotary Positional Embeddings (RoPE). RoPE encodes position $p$ by applying a 2D rotation matrix to each pair of channels in the key and query vectors:

$\mathcal{R}(p) = \text{diag}\left(R(p\omega_0), R(p\omega_1), \dots, R(p\omega_{d/2-1})\right)$

where $R(\theta) = \begin{pmatrix} \cos\theta & -\sin\theta \\ \sin\theta & \cos\theta \end{pmatrix}$ is the standard 2D rotation matrix, and $\omega_m$ are fixed frequencies.

The critical property (derived in Appendix A.3, Equation 3) is that these rotations are additive: applying rotation for position $a$ and then rotation for position $b$ is equivalent to applying rotation for position $a+b$. Formally, $R(a)R(b) = R(a+b)$. A corollary is that rotation can be reversed: $R(-\Delta)R(p) = R(p - \Delta)$.

This means that if a key was RoPE-encoded at position $p$, we can retroactively shift it to behave as if it were encoded at position $p - \Delta$ by applying a reverse rotation $R(-\Delta)$ to the already-computed key vector:

$\hat{k}_p^{\text{rope}} = \mathcal{R}(-\Delta) k_p^{\text{rope}} = \mathcal{R}(-\Delta)\mathcal{R}(p) k_p = \mathcal{R}(p - \Delta) k_p$

What this computes: given a cached key vector that was encoded with RoPE at position $p$, applying the reverse rotation matrix for shift $\Delta$ produces a key vector that is mathematically identical to having encoded it at position $p - \Delta$ from scratch. This operation is exact (not an approximation) due to the additive property of rotation matrices, and it operates directly on the cached vectors without re-running the transformer on past tokens — it's a simple linear algebra operation on the KV cache.

Why this form: the alternative would be to re-encode the entire remaining cache from scratch with new position IDs, which would require a full forward pass over potentially tens of thousands of tokens — completely infeasible for real-time operation. The reverse-RoPE trick reduces the eviction overhead to a single matrix multiplication per head per evicted layer, making it essentially free computationally.

Applying the correction (Appendix A.3). Suppose the cached keys have position IDs $\{p_0, \dots, p_I\}$, where $\{p_0, \dots, p_{i-1}\}$ belong to the system prompt and $\{p_i, \dots, p_I\}$ are from streaming tokens. When the first $j$ streaming tokens are evicted, the remaining cache covers $\{p_j, \dots, p_I\}$. The shift is set to $\Delta = p_j - p_i$, and the reverse rotation is applied to all remaining cached keys, giving them effective position IDs $p' = p - p_j + p_i$ — that is, they now start from position $i$ (immediately after the system prompt). For subsequent decoding, new query positions are assigned in this re-based coordinate system, ensuring consistency between queries and cached keys.

This mechanism enables Proact-VL to operate continuously over arbitrarily long streams (the paper demonstrates stability over 50-minute videos in Section 5.4, Tables 4 and 5) without positional encoding drift and without the computational overhead of re-encoding historical context. The streaming evaluation results confirm that text quality remains consistent over time, while response quality shows only mild degradation before stabilizing — evidence that the infinite inference mechanism works as designed.

System Prompt and Data Construction: The Foundation for Training

While not part of the model architecture per se, the system prompt composition and training data construction are critical to understanding what Proact-VL learns and why it generalizes.

Dynamic system prompt assembly. During training, each sample's system prompt is constructed by concatenating three components (Appendix C.3):

1. Role: a game-specific commentator identity (e.g., "You are a live commentator for a Cyberpunk 2077 game."). One fixed role template exists per game.

2. Persona: a mined profile capturing stylistic patterns, communicative preferences, and interaction strategies extracted from real commentary data (Section 3.2.3). Personas are characterized across three dimensions: tone (analytical, humorous, or hype-driven), vocabulary (domain-specific terminology and colloquialisms), and rhythm and pacing (preference for exposition versus concise reactions). One persona is randomly sampled per training sample.

3. Task: a template specifying the downstream setting. Six templates are prepared covering Solo Commentary, Co-Commentary, and Guidance. One template is randomly sampled.

This randomization during training is not cosmetic — it serves as data augmentation for system prompt robustness. By exposing the model to diverse persona-task combinations during training, the model learns to extract the task-relevant instruction from the prompt rather than overfitting to a specific phrasing, improving generalization to deployment scenarios where the prompt may differ from the training templates.

Training data sources (Appendix F, Table 17). The model is trained on a mixture of three data sources:

• Gaming data: 128,000 samples from 10 games in the Live Gaming Dataset, covering Solo Commentary, Co-Commentary, and Guidance settings.
• Ego4D data: egocentric video narration data providing general guidance-style supervision, which the paper finds transfers to gaming guidance and improves egocentric commentary performance (Ego4D CC drops 22.39% when removed).
• Live-SFT data: streaming supervision from LiveCC datasets, providing additional real-time commentary examples that improve Livesports commentary (CC drops 7.69 when removed).

The ablation in Appendix F shows that each source contributes to its corresponding domain, and the full mixture enables cross-domain transfer — notably, the full model achieves the strongest Ego4D results, which the authors attribute to "guidance-style supervision also present in the Gaming data, providing transferable signals that further improve egocentric narration."

Why these three sources: gaming data alone would produce a model specialized for game commentary but potentially brittle on general video streams. Ego4D provides a broader distribution of everyday activities, improving generalization. Live-SFT provides additional streaming-specific supervision that reinforces the chunk-wise processing behavior beyond what the gaming data alone provides. The combination produces a model that excels at gaming commentary while maintaining strong performance on out-of-domain video (demonstrated on Black Myth: Wukong in Section 5.3).

Summary of Design Choices and Their Justifications

• One-second chunk cadence over variable-length segmentation: provides a fixed, predictable processing rhythm that matches the real-time delivery constraint. Variable-length chunks would complicate KV cache management and make latency guarantees harder to provide.

• ChatML with history-video-query ordering over alternatives: places context-establishing information (history) before grounding information (video) before task-specifying information (query), which the authors argue follows an information-flow logic that improves attention patterns (Appendix A.1).

• Dedicated <|FLAG|> token + MLP response head over token-based silence prediction: decouples the speaking decision from text generation, enabling independent threshold calibration, stable training under extreme class imbalance, and deployment-time control over speaking frequency without model modification.

• Transition-weighted classification loss over standard per-step cross-entropy: addresses the fundamental imbalance between rare but critical state transitions and abundant but uninformative persistence steps, forcing the model to learn when to change state rather than just what state to maintain.

• Dual regularization (local smoothness + global rate) over classification loss alone: the local term suppresses jitter within persistent segments, producing human-like smooth score curves; the global term prevents drift into degenerate always-silent or always-speaking regimes. Together they complement the transition-weighted loss by constraining behavior between transitions.

• Penultimate layer hidden state for response head over final layer: likely provides more task-generalizable features, consistent with the broader probing literature showing that intermediate layers often contain richer representations than the final task-optimized layer.

• Dynamic prompt assembly with random persona-task sampling over fixed prompts: acts as data augmentation for instruction following, forcing the model to attend to prompt content rather than memorizing prompt-specific behaviors.

• Dual-cache with reverse-RoPE over re-encoding or naive truncation: enables exact position re-basing with minimal computational overhead (a single matrix multiplication per head), avoiding both the cost of full re-encoding and the positional drift of naive truncation.

• Multi-source training data (Gaming + Ego4D + Live-SFT) over gaming-only: provides cross-domain transfer that improves generalization to unseen games and general video commentary, while the gaming data's guidance-style supervision strengthens egocentric narration.

4. Key Insights and Innovations

Innovation 1: Proactivity and Real-Time Generation Are Two Halves of the Same Learning Problem, Not Two Separate Systems

The paper's most fundamental conceptual move is reframing the "proactive" and "real-time" axes of video interaction not as independent capabilities to be built by separate models and combined post-hoc, but as two facets of a single interactive behavior that must be learned jointly from data that captures both dimensions simultaneously. Before Proact-VL, the field had effectively bifurcated: one line of work built models that decided when to speak but generated long, high-latency responses when triggered (VideoLLM-online, MMDuet, LiveStar, and others cited in Section 2.2), while another line built models that generated low-latency output at every time step but lacked explicit control over speaking frequency (LiveCC, StreamingVLM). Each approach solved half the problem while failing catastrophically on the other half — a fact the paper quantifies starkly in Table 2, where proactive baselines achieve F1 scores as low as 0.05 (MMDuet, Solo) while real-time baselines like LiveCC-7B-Base reach 47.05 F1, far better but still far from human alignment.

The insight is that this bifurcation is artificial and harmful. The decision to speak depends on whether the model can actually produce useful content within the available time window — a model that knows it can only generate 5-second monologues should learn different speaking thresholds than one capable of 1-second bursts. Conversely, the content-generation behavior should depend on whether the moment was correctly identified as response-worthy; generating text at every second and then filtering post-hoc wastes computation and breaks coherence. By training the speak/silence classifier and the text generator jointly — with the response head receiving the same transformer hidden state that feeds into text generation, and with the composite loss $\mathcal{L} = \mathcal{L}_{\text{main}} + \alpha\mathcal{L}_{\text{resp}}$ backpropagating through shared parameters — Proact-VL ensures that the two capabilities co-adapt. The dramatic F1 gap between Proact-VL (63.25 Solo, 77.44 Co-Commentary) and the best prior real-time model (LiveCC-7B-Instruct at 62.05 Solo, 61.26 Co-Commentary, from Table 2) is not from any single architectural trick but from this joint optimization principle.

This reframing is fundamental rather than incremental because it changes what problem the field should be trying to solve. The paper argues implicitly that prior work's framing — "build a proactive trigger, then plug it into a text generator" or "build a real-time captioner, then decide when to display captions" — is a category error. The interactive behavior is not decomposable into independent perception-decision-generation stages in the way a pipeline architecture would suggest. The paper's evidence for this is the failure of exactly such pipeline approaches (the proactive baselines in Table 2) and the success of the joint training approach, but also the ablation in Table 6 showing that removing either the classification loss or the stability regularizer from the joint objective degrades both timing and content quality — the objectives are complementary in a way that pipelined training would miss.

Innovation 2: The Dominant Learning Challenge in Proactive Behavior Is Not Silence-vs-Speech Imbalance, but Transition-vs-Persistence Imbalance

This is a diagnostic insight about why prior approaches to learning speaking behavior failed, and it drives the specific design of the loss function. The natural first approach to learning when to speak is to treat each second as an independent binary classification problem: given the video and context, predict whether a human commentator would speak at this second. Standard binary cross-entropy would be the default loss. But the paper identifies that the real learning challenge is not the imbalance between silent seconds and speaking seconds (which could be handled by class weighting), but something subtler: the overwhelming dominance of state persistence over state transitions.

In human commentary, if you were speaking at second $t-1$, you are very likely still speaking at second $t$; if you were silent, you likely remain silent. The rare events — the seconds where the state changes — are the moments the model most needs to learn from, because they define the boundaries of response-worthy segments. But in a standard per-second loss, these transition events constitute only about one-sixth of the data (the paper reports a ~1:5 transition-to-persistence ratio in the training set) and are drowned out by the abundant persistence seconds. A model trained with uniform per-second cross-entropy can achieve low loss simply by learning to maintain its current state, without ever learning the visual and contextual cues that signal when to change state. This produces the degenerate behavior visible in the prior proactive baselines: models that either stay silent forever or, once triggered, talk continuously — because they learned state persistence but not state transitions.

The transition-aware weighting ($w_t = 5$ for transition seconds, $w_t = 1$ for persistence seconds, based on the observed $1:5$ ratio in the data) is the direct response to this diagnosis. But the deeper conceptual contribution is the diagnosis itself — the recognition that proactive behavior learning is fundamentally a change-point detection problem embedded in a sequence, not a frame-level classification problem. Prior work that treated it as the latter (implicitly or explicitly) was solving the wrong problem. The paper's evidence for this diagnosis comes from the ablation in Table 6: removing the transition-aware component (by removing $\mathcal{L}_{\text{cls}}$ entirely) causes F1 to drop dramatically (by 49.05 points) and TimeDiff to spike (by 15.09 seconds), confirming that the transition-weighted objective is doing qualitatively different work than a standard classification loss would. This is a diagnostic innovation that should change how future work approaches proactive behavior learning across domains — not just gaming commentary, but any setting where an agent must learn when to act based on temporal cues in a persistent-state environment.

Innovation 3: Speaking Rate Control Needs Both Local Smoothness and Global Calibration — and They Address Distinct Failure Modes

The two-pronged stability regularizer ($\mathcal{L}_{\text{reg}} = \mathcal{L}_{\text{reg, local}} + \mathcal{L}_{\text{reg, global}}$) might appear at first glance to be two variants of the same idea — "make the speaking behavior stable." But the paper demonstrates that they address qualitatively different failure modes that are both present in prior systems but have different causes and require different solutions.

Local smoothness ($\mathbb{E}[(p_t - p_{t-1})^2 \mid y_t = y_{t-1}]$) addresses jitter: rapid oscillation between speak and silence within what should be a continuous segment. This failure mode arises from the per-second independence assumption in the classification loss — even with transition weighting, the model receives no penalty for flipping its decision from second to second as long as each individual decision is correct on average. The consequence (visible in the threshold analysis of Figure 5, where the $\tau = 0.1$ curve shows highly oscillatory scores) is stutter-like commentary that sounds unnatural to human listeners. The local smoothness term fixes this by penalizing probability changes (not absolute values) during persistent segments.

Global rate calibration ($(\mathbb{E}[p_t] - \mathbb{E}[y_t])^2$) addresses drift: the model gradually learning to speak much more or much less than human commentators, even if its individual decisions are well-timed. This failure mode arises because the transition-weighted classification loss emphasizes decision boundaries but provides no constraint on the overall ratio of speech to silence. A model could learn to identify transitions accurately but with a systematically biased threshold, producing either a hyperactive commentator that speaks 80% of the time or a laconic one that speaks 10% of the time. The global rate term fixes this by penalizing any deviation from the human average speaking rate.

The reason this distinction matters beyond Proact-VL's specific implementation is that it identifies two independent axes of speaking-behavior quality that future systems should evaluate and optimize separately. Current metrics in the proactive interaction literature tend to conflate these: a PAUC score might reward good timing but be insensitive to over-triggering; an F1 score captures both but doesn't distinguish whether low F1 is due to jitter or rate miscalibration. The paper's decomposition suggests a more granular evaluation framework and a corresponding training strategy where local and global constraints are explicitly separated. The ablation in Table 6 confirms their independence: removing the regularizer (i.e., using only $\mathcal{L}_{\text{cls}}$) degrades both timing and content quality, but the qualitative analysis in Figure 5 shows that different threshold settings produce characteristically different failure patterns (jitter vs. over-triggering) that correspond to which constraint is violated. This is a conceptual reframing with practical implications for how proactive systems are trained and evaluated.

Innovation 4: The Live Gaming Dataset as a New Resource Class — Streaming Commentary with Per-Second Speech Labels

While dataset contributions are often treated as secondary in papers that also introduce methods, the Live Gaming Dataset addresses a genuine resource gap that has shaped what kinds of proactive systems could be built. Prior to this work, no publicly available dataset provided the three properties needed to train a joint proactive-real-time commentary system: (1) per-second speech/silence labels aligned with human behavior (not heuristically derived), (2) short, timestamped utterances suitable for clip-level generation training, and (3) multi-speaker scenarios with turn-taking dynamics. Standard video QA datasets (MVBench, Ego4D) are designed for offline, single-response evaluation. Streaming datasets from prior work (LiveCC, StreamingVLM) provide video-caption pairs at high temporal density but lack explicit speak/silence annotations derived from human commentary decisions — the captioning is continuous, so the model never learns when not to speak.

The dataset's construction pipeline (Section 3.2, Figure 3) is itself a contribution: it demonstrates how to transform raw gaming footage with noisy ASR transcripts into structured training data suitable for proactive interaction learning. Key steps include speaker identification to isolate primary commentary from background noise, paralinguistic labeling to preserve prosodic cues (pauses, laughter, phonetic elongations) that are lost in plain text, and domain-specific polishing using a strong LLM (DeepSeek-V3.2-Exp) to correct ASR errors in game-specific terminology. The persona enrichment step (Section 3.2.3) — extracting structured commentator profiles with tone, vocabulary, and pacing dimensions — is a clever way to inject stylistic diversity into training without requiring human annotation of style attributes.

The significance of this dataset goes beyond enabling Proact-VL's training. It establishes a new resource category for the field: streaming interaction datasets with human-derived speaking decisions. As the paper demonstrates by training Proact-VL variants on different backbone models (Qwen2-VL, Qwen2.5-VL, Qwen3-VL — Appendix C.4, Table 9), the framework transfers across architectures, suggesting the dataset captures generalizable interaction patterns rather than model-specific artifacts. The cross-domain generalization results (Black Myth: Wukong in Section 5.3, Ego4D in Appendix C.5, Table 10) further indicate that the dataset's supervision signal — learning when and how to speak from human examples — transfers beyond the specific games in the training set. For a field where data scarcity has been a bottleneck for proactive system development, this is a foundational resource contribution that enables a research program beyond what any single method paper could achieve.

5. Experimental Analysis

Evaluation Methodology

• Dataset. All main experiments use the Live Gaming Dataset, constructed by the authors. The training set covers 10 games (128,000 samples from sources including Cyberpunk 2077, StarCraft II, Baldur’s Gate 3, Elden Ring, Tears of the Kingdom, Yu-Gi-Oh, League of Legends, CSGO, Street Fighter 6, and Minecraft, plus LiveCC and Ego4D general streaming data). The primary test set, the Live Gaming Benchmark, contains 2,640 in-domain clips plus 240 out-of-domain clips (Black Myth: Wukong) and 134 general-scenario clips (Ego4D Goal-Step), for a total of 3,014 clips organized into three subsets: Solo Commentary, Co-Commentary, and Guidance. A separate streaming test set (Live Gaming Benchmark-Streaming) evaluates long-horizon stability using 10 full-length videos (30 minutes to 2 hours). A video-wise 80%/10%/10% train/test/reserve split is used within each game (Appendix B.2).

• Base model(s). The primary backbone is LiveCC-7B-Base (Chen et al., 2025), a 7B-parameter model pretrained for streaming video understanding at a one-second cadence. The authors argue this provides a strong initialization because it already produces sentence-level outputs at the target temporal granularity, meaning the fine-tuning focuses on adding proactivity control rather than learning streaming perception from scratch. Additional experiments use Qwen2-VL, Qwen2.5-VL, and Qwen3-VL backbones (Appendix C.4, Table 9) to demonstrate cross-architecture transferability.

• Metrics. Evaluation is split into two complementary categories. Text quality uses three metrics: (1) Closed Captions (CC), a win-rate metric comparing outputs against Gemini 2.5 Pro via an LLM judge (GPT-5.1); (2) LiveU, which scores second-level streaming usability as the mean of three sub-scores — Time (speaking at the right moments), Rate (appropriate pacing), and TextU (immediate clarity as live speech); (3) FinalQ, which concatenates all clip outputs within a segment (timestamps removed, silent seconds ignored) and evaluates the consolidated script quality as the mean of Fidelity, Continuity, and Substance. Proactivity timing uses three metrics: (1) TimeDiff, measuring temporal alignment between predicted and ground-truth response onsets (lower is better, with a penalty term for predictions outside an expanded tolerance window); (2) PAUC, a cumulative trajectory metric assessing proactive quality integrated over the video timeline (evaluated with GPT-5.1 as judge, with initial score 0 and ω = 0.5); (3) F1, computed by treating the full video timeline as a binary positive/negative labeling (ground-truth intervals are positive, all other timestamps are negative) and comparing the model's response timeline against it. All LLM-judge metrics use GPT-5.1. Robustness to judge model choice and stochasticity is evaluated in Appendix D.

• Baselines. Three categories are compared. Offline commercial models: GPT-4o (gpt-4o_2024-11-20) and Gemini 2.5 Pro, evaluated by providing full videos and requesting complete commentary, serving as upper bounds on text quality given unlimited context. Proactive models: VideoLLM-online (Chen et al., 2024a), MMDuet (Wang et al., 2024b), and LiveStar (Yang et al., 2025) — these decide when to respond via learned policies but, once triggered, generate complete responses without clip-level length control. Real-time models: LiveCC-7B-Base, LiveCC-7B-Instruct (Chen et al., 2025), and StreamingVLM (Xu et al., 2025c) — these emphasize low-latency streaming generation but lack explicit speak/silence control mechanisms.

• Generation budget / compute accounting. The comparison is not FLOPs-matched; rather, all models operate under the same streaming constraint: one-second chunk processing with a response threshold of τ = 0.3 for main results (τ = 0.5 for ablations). Efficiency is measured in wall-clock latency per chunk (Table 7): the system is profiled with varying window sizes (2048–32768 tokens) and per-frame token budgets (196–364 tokens/frame), reporting peak GPU memory, cache update time, generation time, and end-to-end per-chunk latency. The estimate is that with 364 tokens/frame and a 0.3-second generation budget, the system handles 10–15 FPS video streams.

• Cross-validation / statistical protocol. No formal cross-validation is reported for the main benchmark results. The training set uses a fixed 80%/10%/10% video-wise split per game. LLM-judge robustness is evaluated by repeating judging five times (Appendix D, Tables 14–15) with reported means, standard deviations, and 95% confidence intervals, showing small variability (win rates stable within ~1–2% across runs). A user study (Appendix E) with 15 evaluators performing 18 pairwise comparisons each (3 evaluators per pair) provides human validation of automated metrics, with Proact-VL significantly outperforming all three baseline categories.

Main Quantitative Results

Live Gaming Commentary: Text and Response Quality

The core results appear in Tables 1 and 2, evaluating on the in-domain subset of the Live Gaming Benchmark across three interaction settings: Solo Commentary, Co-Commentary, and Guidance.

Text quality (Table 1). Proact-VL achieves the best overall CC (49.23) and LiveU (6.52), substantially ahead of the strongest real-time baseline LiveCC-7B-Instruct (CC: 28.35, LiveU: 4.90) and the strongest offline model GPT-4o (CC: 39.42, LiveU: 4.62). In Solo Commentary, Proact-VL reaches CC of 53.62 and LiveU of 6.89, compared to LiveCC-7B-Base at 41.17/4.85 and GPT-4o at 21.54/4.56. In Co-Commentary, it scores CC of 51.46 and LiveU of 5.15 versus LiveCC-7B-Base at 45.89/3.78. The Co-Commentary FinalQ shows Proact-VL at 3.59, only marginally ahead of LiveCC-7B-Instruct at 3.28 and Gemini 2.5 Pro at 3.59, indicating that co-commentary script coherence remains challenging across all models. In Guidance, Proact-VL leads on LiveU (7.52) but trails GPT-4o on FinalQ (6.02 vs. 6.66), suggesting offline models maintain an advantage in consolidated instructional quality even as Proact-VL delivers better per-second guidance.

Prior proactive models (VideoLLM-online, MMDuet, LiveStar) lag severely on text quality: VideoLLM-online achieves CC of only 13.78 overall, and MMDuet reaches 20.08, reflecting the paper's diagnosis that these models generate overly long responses unsuited for clip-level evaluation. StreamingVLM, despite its real-time optimization, scores CC of 14.89 overall, below even some proactive baselines, indicating that low latency alone does not guarantee commentary quality.

Response quality (Table 2). Proact-VL achieves the best overall F1 of 64.87, compared to 56.05 for LiveCC-7B-Instruct, 54.88 for GPT-4o, and 49.23 for Gemini 2.5 Pro. The most dramatic gains appear in Co-Commentary, where Proact-VL's F1 of 77.44 surpasses LiveCC-7B-Instruct (61.26) by 16 points and the proactive baseline MMDuet (0.16) by a factor of nearly 500×. This is the paper's strongest single result: in the multi-speaker coordination setting where turn-taking matters most, Proact-VL's joint proactivity-real-time design radically outperforms prior systems. In Guidance, Proact-VL reaches F1 of 53.91 vs. 44.83 for LiveCC-7B-Instruct, a smaller but still substantial margin. TimeDiff scores confirm temporal precision: Proact-VL achieves 1.20 seconds in Solo (vs. 1.04 for LiveCC-7B-Instruct and 1.16 for GPT-4o) and 0.71 in Co-Commentary — the lowest of any model, substantially below LiveCC-7B-Instruct at 2.01.

The PAUC metric shows a more nuanced picture: Proact-VL's PAUC of 20.36 in Solo is below GPT-4o's 25.14, and its Guidance PAUC of 26.92 is below GPT-4o's 42.26. The paper acknowledges this (Section 5.2): "PAUC is slightly below GPT-4o, leaving headroom for sharper separation of response-worthy moments." PAUC rewards models that respond within ground-truth intervals, and GPT-4o's advantage likely reflects its ability to produce higher-quality content when it does speak within those windows, even if its overall F1 (reflecting both precision and recall across the full timeline) is lower.

Common and General Commentary: Out-of-Domain Generalization

Table 3 (main text) and Table 10 (Appendix C.5) evaluate on Ego4D Goal-Step (general egocentric video) and Black Myth: Wukong (unseen game). On Ego4D, Proact-VL achieves CC of 63.43, LiveU of 7.21, and FinalQ of 5.42 — the strongest across all metrics. The real-time baseline LiveCC-7B-Base scores CC of 12.69 on the same data, a gap of over 50 points, indicating that the gaming-trained proactivity transfers well to general video narration. On Black Myth: Wukong, Proact-VL scores CC of 55.21 and LiveU of 6.22, competitive with LiveCC-7B-Base at 56.46 CC (the highest on this metric for this game) and ahead on LiveU (6.22 vs. 4.55). TimeDiff is 0.90 for Proact-VL vs. 0.87 for LiveCC-7B-Instruct, essentially tied for best. F1 is 60.06 vs. 59.88 for LiveCC-7B-Instruct, again nearly identical. The paper interprets this as showing "strong out-of-domain generalization to a previously unseen game" (Section 5.3), though the near-tie with LiveCC-7B-Instruct on Black Myth suggests the generalization benefit is larger for egocentric video than for unseen-game commentary specifically.

Live Gaming Streaming: Long-Horizon Stability

Tables 4 and 5 evaluate on the Live Gaming Benchmark-Streaming (10 full-length videos, 30 minutes to 2 hours) with metrics reported at 10-minute intervals from 10 to 50 minutes, plus overall. The text quality metric SC (Streaming Commentary) is defined as the predicted win rate of Proact-VL against StreamingVLM. Proact-VL's SC remains stable across all time horizons, with no significant degradation from 10 to 50 minutes (specific numerical values not quoted in the main text but the paper states "text quality remains consistent as the inference horizon increases"). Response quality shows "only a mild degradation and then tends to stabilize," again without specific numbers in the main text, but the conclusion is "robustness of Proact-VL for long-form streaming commentary."

The significance of this result is that it validates the infinite inference mechanism (dual-cache with reverse-RoPE, Appendix A.2–A.3). Without this mechanism, positional encoding drift would cause degradation over long streams; the stability evidence suggests the reverse-RoPE correction successfully prevents this failure mode.

Inference Efficiency

Table 7 profiles latency under varying configurations. With a streaming window of 32768 tokens and 364 tokens per frame, peak GPU memory is 39.6 GB, cache update takes 0.30 seconds, forward generation takes 0.26 seconds, and end-to-end chunk latency is 0.81 seconds. The per-token generation time remains essentially constant (~0.011–0.012 seconds/token) across window sizes (2048–32768) and frame budgets (196–364 tokens), indicating stable decoding efficiency. The paper estimates the system can handle 10–15 FPS streams with a 0.3-second generation budget and 364 tokens/frame.

A notable observation: the cache update time grows from 0.02 seconds at 2048 tokens to 0.32 seconds at 32768 tokens (with 256 tokens/frame), a 16× increase, while generation time remains flat. This means the preprocessing/cache stage, not text generation, becomes the bottleneck at large context windows — a practical engineering consideration for deployment that the paper documents but does not deeply explore.

Ablation Studies and Robustness Checks

Training loss components (Table 6): Removing $\mathcal{L}_{\text{reg}}$ causes the most severe degradation: F1 drops by 49.05 points and TimeDiff increases by 15.09 seconds, while CC also drops. Removing $\mathcal{L}_{\text{cls}}$ degrades all metrics but less dramatically. The full loss ($\mathcal{L}_{\text{cls}} + \mathcal{L}_{\text{reg}}$) achieves the best F1, TimeDiff, and CC, confirming that both components are complementary and jointly necessary for robust response behavior.

Training data sources (Appendix F, Table 17): Removing Gaming data drops Gaming CC by 13.08%. Removing Ego4D drops Ego4D CC by 22.39%. Removing Live-SFT drops Livesports CC by 7.69. The full mixture achieves the best overall performance, with the model trained on all three sources reaching the strongest Ego4D results despite Ego4D being only one component — the paper attributes this to transfer from guidance-style supervision in the Gaming data.

Response threshold (Appendix I.1, Figure 8): Increasing τ consistently reduces F1 across all three settings (Solo, Co-Commentary, Guidance), as expected since higher thresholds mean fewer triggered responses. Conversely, CC generally improves with higher thresholds: Co-Commentary CC increases monotonically (peaking at τ = 0.9), Solo CC peaks around τ = 0.6, and Guidance CC peaks around τ = 0.5. This quantifies the precision-coverage tradeoff: conservative thresholds produce higher-quality but sparser commentary. The paper selects τ = 0.3 for main results to favor coverage, and τ = 0.5 for ablations.

Context window size (Appendix I.2, Figure 9): CC generally improves with larger windows, saturating around 16384–24576 tokens across all settings. F1 remains relatively stable across window sizes, suggesting the response timing is robust to context length. Solo CC rises from 50.58 at 2048 tokens to 55.08 at 24576 tokens. Co-Commentary CC peaks at 56.98 at 24576 tokens. Guidance CC peaks at 43.02 at 16384 tokens.

Base model backbone (Appendix C.4, Table 9): Proact-VL fine-tuned from Qwen-series backbones (Qwen2-VL, Qwen2.5-VL, Qwen3-VL) achieves "significantly better commentary quality and proactive responses than base model," demonstrating the framework transfers across architectures.

LLM judge robustness (Appendix D, Tables 11–15): Replacing GPT-5.1 with an alternative judge (GPT-4o) preserves the relative ranking: Proact-VL achieves the best LiveU and FinalQ across all three settings under both judges. Five repeated runs of GPT-5.1 judging on Proact-VL show small standard deviations (specific values in Table 15), with 95% confidence intervals confirming stability.

Prompt injection effect on baselines (Appendix G, Table 18): Adding system prompts (task/persona instructions) and user-template context (history/query) to LiveCC-Base substantially alters behavior: F1 in Solo drops from 47.05 to 16.40 with full prompt injection, demonstrating that the base model is sensitive to prompt formatting in ways that affect speaking behavior. LiveCC-Instruct is more robust to prompt changes. This ablation contextualizes Proact-VL's gains: part of the improvement over baselines comes from careful prompt design, but the joint training with response objectives is what enables stable behavior under these prompts.

Offline video understanding preservation (Appendix H, Table 19): Proact-VL fine-tuned from Qwen3-VL achieves 64.65% overall accuracy on MVBench, compared to 66.30% for the un-fine-tuned baseline, a drop of only 1.65 percentage points. Gains appear on action-centric tasks (Action Sequence: +5.0, Object Interaction: +5.0) while some motion/attribute categories regress (Moving Attribute: −12.0, Moving Count: −8.0). This demonstrates that domain-specific fine-tuning for gaming commentary largely preserves general video understanding capability.

Game-wise breakdown (Appendix J.1, Figure 10): Radar plots show Proact-VL achieves higher LiveU and FinalQ scores on nearly all individual games compared to baselines, indicating the gains are consistent across diverse gameplay styles and not driven by a subset of titles. Specific per-game numerical values are visualized but not tabulated.

Critical Assessment

Does Proact-VL genuinely outperform baselines on both text quality and timing?

The evidence is strong for the in-domain Live Gaming Benchmark. Tables 1 and 2 show Proact-VL leading on overall CC (49.23 vs. 39.42 for GPT-4o and 28.35 for LiveCC-7B-Instruct), LiveU (6.52 vs. 4.90 for LiveCC-7B-Instruct), and F1 (64.87 vs. 56.05 for LiveCC-7B-Instruct and 54.88 for GPT-4o). The Co-Commentary F1 gap (77.44 vs. 61.26) is particularly large and meaningful — it directly measures the multi-speaker coordination that the paper identifies as a failing of prior systems.

However, the baselines raise questions. GPT-4o and Gemini 2.5 Pro are evaluated as offline models — they receive the full video and generate complete commentary, without any streaming or proactivity constraints. Their poor F1 scores (54.88 and 49.23) partly reflect that they were not designed for this task, making the comparison somewhat artificial. A fairer baseline would give these models the same chunk-wise constraints Proact-VL operates under, potentially with the same prompt template (though they lack the dedicated response head). The paper does not report results for GPT-4o or Gemini prompted in a streaming format with explicit speak/silence instructions, which would test whether Proact-VL's architecture provides benefits beyond what strong commercial models can achieve with appropriate prompting.

The proactive baselines (VideoLLM-online, MMDuet, LiveStar) perform so poorly (F1 scores of 6.54, 0.18, and 0.20 respectively) that they barely constitute baselines at all. Their near-zero F1 scores reflect a fundamental mismatch with the evaluation: these models were designed for sparse, complete responses to salient events, not for dense per-second commentary. The paper's framing ("proactive models fail on real-time commentary") is accurate, but the measured failure is so extreme that it's unclear whether the comparison is informative beyond confirming that prior proactive models were not designed for this use case.

Does the joint training approach cause the improvement, or is it the data?

The paper ablates the loss function components (Table 6) but does not ablate the training data against a baseline trained with a simpler objective on the same data. The LiveCC-7B-Base and LiveCC-7B-Instruct baselines are trained on different data (the original LiveCC training set). To establish that Proact-VL's architecture and loss function (rather than the Live Gaming Dataset itself) drive the gains, one would need to see: (1) LiveCC-7B-Base fine-tuned on the Live Gaming Dataset with only the standard language modeling loss (no response head, no $\mathcal{L}_{\text{resp}}$), and (2) Proact-VL trained on the original LiveCC data. Neither ablation is reported. The data ablation in Appendix F only varies which subsets of Proact-VL's training data are used under the full Proact-VL training recipe — it doesn't ablate the recipe against a simpler baseline on the same data. This leaves open the possibility that the Live Gaming Dataset's scale and quality explain a substantial portion of the gains, and that a simpler streaming model fine-tuned on this data (without the response head architecture) might close much of the gap.

Does Proact-VL generalize to unseen scenarios?

The out-of-domain results are mixed. On Ego4D, Proact-VL shows strong generalization (CC: 63.43 vs. 12.69 for LiveCC-7B-Base). On Black Myth: Wukong, Proact-VL is essentially tied with LiveCC-7B-Instruct on F1 (60.06 vs. 59.88) and slightly behind LiveCC-7B-Base on CC (55.21 vs. 56.46). The Ego4D result is the stronger evidence of transfer; the Black Myth result suggests the generalization advantage narrows when moving to an unseen game rather than a different video domain entirely. The paper tests only one unseen game, which limits confidence in the breadth of generalization. Testing on 3–5 held-out games covering different genres (not just action RPG like Black Myth: Wukong) would provide a more convincing case.

Are the metrics capturing what matters?

The paper uses LLM-as-a-judge for all text quality metrics (CC, LiveU sub-scores, FinalQ sub-scores) and PAUC. The judge is GPT-5.1, a strong model, and Appendix D shows the rankings are stable under judge substitution and repeated runs. However, the paper does not report human correlation for these automated metrics beyond the user study (Appendix E), which uses pairwise preference rather than direct metric validation. It's possible that the LLM judge favors certain commentary styles (e.g., shorter utterances, simpler language) that align with Proact-VL's outputs but might not be universally preferred by human listeners. The user study shows Proact-VL wins against baselines in pairwise comparisons, but pairwise preference doesn't directly validate the absolute scale of CC, LiveU, or FinalQ scores.

Streaming stability claims need more evidence.

The long-horizon streaming evaluation (Tables 4 and 5) reports that Proact-VL's "text quality remains consistent" and response quality shows "only a mild degradation and then tends to stabilize." However, the main text does not report the actual numerical SC values at each time point, making it impossible to assess what "mild degradation" means quantitatively. The comparison is only against StreamingVLM (the SC metric is defined as win rate vs. StreamingVLM). A comparison against LiveCC under the same streaming conditions, or absolute quality metrics (not just relative win rates), would strengthen the claim that the infinite inference mechanism prevents degradation. Additionally, the streaming test covers only 10 videos — one per game — which is a small sample for evaluating stability claims.

Missing experiments.

Several experiments would have strengthened the paper's claims:

• Proactive baselines with clip-level output: The paper argues that prior proactive models fail because they generate long responses when triggered. A natural experiment would be to take a proactive baseline (e.g., MMDuet) and constrain its output length to match Proact-VL's one-second clip format, testing whether the F1 gap closes when output length is equalized. This would distinguish whether the gap is architectural (response head vs. token-based decision) or simply a consequence of output length mismatch with the evaluation metric.

• Ablation of the <|FLAG|> token placement: The paper places <|FLAG|> after the query and before the assistant prefix. Would placing it at a different position (e.g., before the video chunk, or immediately after the history) change performance? This would test the claim that the specific ordering matters for attention patterns.

• Ablation of penultimate vs. final layer for the response head: The paper uses the penultimate layer's hidden state for the response head without ablating this choice. Given the probing literature's mixed findings on optimal layer selection, this choice could meaningfully affect response quality.

• Scaling analysis: The paper uses a 7B model throughout. Does the joint training approach provide larger or smaller gains at different model scales? This is particularly relevant for the claim that the architecture matters independently of the base model, since only one scale is tested.

• Non-gaming domains: All training uses gaming or egocentric video. Would the approach transfer to entirely different streaming domains (e.g., sports commentary, security monitoring, surgical assistance) without domain-specific training data? This would test the paper's implicit claim that the method (joint proactivity-real-time training) generalizes rather than the data being domain-specific.

Bottom line. The experiments convincingly demonstrate that Proact-VL outperforms existing real-time and proactive models on the Live Gaming Benchmark, particularly in multi-speaker coordination (Co-Commentary F1: 77.44). The contribution of the joint training approach relative to data quality, and the generalization breadth beyond gaming, remain partially open questions. The paper's strongest empirical result is the Co-Commentary F1 — this is a genuinely hard problem where prior systems effectively score zero, and Proact-VL achieves performance approaching practical usability. The weakest link is the limited characterization of long-horizon streaming stability and the absence of experiments that would isolate the architectural contribution from the data contribution.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is the Elephant in the Room

The assumption or constraint. Proact-VL's proactive response head is trained on per-second human speaking labels derived from the Live Gaming Dataset. However, the model does not receive explicit difficulty information about video content — it must infer whether a moment warrants commentary purely from its learned representations. This creates an implicit assumption: that the training data provides sufficient coverage of diverse gaming scenarios and speaking patterns that the model's internal representations generalize to deployment conditions without requiring per-scenario calibration.

The paper acknowledges that the model can still produce content "only weakly correlated with the on-screen evidence, i.e., the text may be plausible but not tightly grounded in the visual stream" (Appendix L). It further notes that practical deployment would involve "high-resolution, high-frame-rate videos (e.g., HD/Blue-ray quality and 120+ FPS)" while the current setting processes only 2 FPS at reduced resolution — a 60× temporal undersampling gap that the authors explicitly flag as a limitation (Appendix L).

The consequence. The consequence is that Proact-VL's speaking decisions and content quality degrade in precisely the situations that matter most for real-world deployment: fast-paced action sequences, rare but critical game events, and visually complex scenes. At 2 FPS, the model misses everything that happens between frames — a 0.5-second window where a game-winning move occurs, a critical UI change, or a rapidly developing situation that a human commentator would catch because they watch at 30–60 FPS with full visual acuity. The failure cases in Appendix K.4 (Figures 15 and 16) illustrate this: hallucinated numerical values from misreading small HUD text (Figure 15), and repetitive filler output ("Oh, no!") when the interface is too cluttered for reliable visual grounding at the model's resolution (Figure 16). These are not rare edge cases — they are systemic consequences of the resolution and frame-rate constraints under which Proact-VL operates.

Furthermore, the model has no mechanism for adaptive difficulty: it cannot recognize when it is confused and signal uncertainty, escalate to a higher-capacity system, or adjust its speaking behavior based on confidence. A human commentator who cannot see clearly would say less or qualify their statements; Proact-VL has no such meta-cognitive capability and will produce content at its trained speaking rate regardless of whether it actually understands what is happening on screen.

What evidence exists in the paper. The paper provides direct evidence of this limitation in the failure cases (Appendix K.4, Figures 15–16), the inference efficiency table (Table 7, which confirms the 2 FPS processing rate and pixel budget constraints), and the explicit acknowledgment in Appendix L that "sparse frames... can miss critical transient cues and yield temporally discontinuous observations." The MVBench evaluation (Appendix H, Table 19) shows a 1.65-point overall accuracy drop after fine-tuning, with specific regressions on motion-heavy tasks (Moving Attribute: −12.0, Moving Count: −8.0, Moving Direction: −6.0) — precisely the categories most sensitive to frame rate. No experiment tests Proact-VL at higher frame rates or resolutions, so the performance ceiling under improved visual fidelity is unknown.

Mitigation status. The paper does not mitigate this limitation. Appendix L identifies it as future work: "Future work should explore more efficient streaming video encoders and memory mechanisms to scale to higher FPS and higher resolution under real-time latency and compute budgets." The current Proact-VL architecture has no built-in pathway for incorporating higher frame rates without increasing per-chunk latency, which would break the real-time guarantee. The efficiency analysis in Table 7 shows that the cache update time already grows with window size and token count; adding more frames per second would directly increase this cost, pushing end-to-end latency above the one-second chunk deadline.

The Joint Training Claim Is Underspecified — Data Quality vs. Architecture Cannot Be Disentangled from Reported Experiments

The assumption or constraint. The paper's central thesis is that joint optimization of speaking decisions and content generation — through the composite loss $\mathcal{L} = \mathcal{L}_{\text{main}} + \alpha\mathcal{L}_{\text{resp}}$ and the shared transformer backbone — is what enables Proact-VL to outperform prior systems that separate proactivity and real-time generation. This claim implicitly assumes that the architectural integration (response head, joint loss) is the primary driver of improvement, rather than the scale, quality, or domain-specificity of the Live Gaming Dataset itself.

The consequence. If the Live Gaming Dataset — with its 561 hours of curated commentary, per-second speech labels, persona enrichment, and domain-specific polishing — is the primary source of Proact-VL's gains, then the paper's methodological contribution is substantially weaker than claimed. A practitioner could potentially achieve comparable results by fine-tuning a simpler streaming model (e.g., LiveCC-7B-Base) on the same dataset with a standard language modeling objective, without the response head architecture or specialized loss terms. The paper provides no experiment that would rule this out, because all Proact-VL ablations (Table 6, Appendix F) are conducted within the Proact-VL framework — they vary loss components or training data subsets while keeping the response head and chunk-wise architecture fixed. There is no baseline where LiveCC-7B-Base is fine-tuned on the Live Gaming Dataset with only $\mathcal{L}_{\text{main}}$ (no response head, no speaking labels) and evaluated on the same proactivity metrics. Without this, the contribution of the architecture versus the data remains unquantified.

This matters because training data construction represents a substantial engineering investment — the multi-stage pipeline (ASR, speaker identification, paralinguistic labeling, domain polishing, persona extraction) described in Section 3.2 may be the primary transferable contribution, with the architectural innovations being secondary or even unnecessary when strong base models are available.

What evidence exists in the paper. The paper provides several pieces of indirect evidence but no direct test of the data-vs-architecture question:

• The data ablation (Appendix F, Table 17) shows that removing Gaming, Ego4D, or Live-SFT data from Proact-VL's training mixture degrades performance, but this only demonstrates that the data is necessary given the Proact-VL architecture — it does not test whether a simpler architecture would benefit equally from the same data.
• The prompt injection experiment (Appendix G, Table 18) shows that adding system and user prompts to LiveCC-Base dramatically alters its speaking behavior (F1 in Solo drops from 47.05 to 16.40), but this demonstrates prompt sensitivity in the baseline, not that Proact-VL's architecture is necessary to achieve stable behavior under prompts.
• The base model comparison (Appendix C.4, Table 9) shows Proact-VL training improves Qwen-series backbones, but these are all trained with the full Proact-VL recipe — there is no Qwen-7B fine-tuned on the Live Gaming Dataset with a standard language modeling loss for comparison.

The absence of a "LiveCC-Base + Live Gaming Dataset + standard LM loss" baseline is the critical missing experiment.

Mitigation status. The paper does not acknowledge this as a limitation. The ablation in Table 6 ("Effectiveness of training loss") removes loss components from Proact-VL but cannot address whether the loss components are beneficial conditional on having the response head architecture or whether the response head architecture is beneficial conditional on having the specialized loss. These two questions are confounded in the current experimental design. Future work would need to train a model with the Live Gaming Dataset but without the response head (treating speaking decisions as a standard sequence generation problem, potentially with a special speak/silence token) to disentangle data quality from architectural contribution.

The Response Threshold Is a Deployment-Time Parameter That Controls a Fundamental Tradeoff the Paper Characterizes but Does Not Resolve

The assumption or constraint. Proact-VL's speaking behavior is controlled by a single scalar threshold $\tau$, applied uniformly across all video content and all time steps. The response head produces a probability $p_t$, and the model speaks whenever $p_t \geq \tau$. The paper uses $\tau = 0.3$ for main results and $\tau = 0.5$ for ablations, but acknowledges through its threshold sensitivity analysis (Appendix I.1, Figure 8) that this parameter controls a systematic tradeoff: lower thresholds increase coverage (higher F1) but reduce commentary quality (lower CC), while higher thresholds improve quality at the cost of triggering fewer responses.

The consequence. There is no single "correct" threshold. The optimal $\tau$ depends on the application's priorities — a high-engagement companion should speak more (lower $\tau$), while a background commentator should speak less (higher $\tau$) — and possibly on the content itself (fast-paced action sequences may warrant different thresholds than slow exploration segments). The paper provides no mechanism for content-adaptive thresholding, no guidance on how a practitioner should select $\tau$ for a new domain or game, and no evaluation of how sensitive the user experience is to threshold choice. A deployment where the operator guesses $\tau$ poorly could produce a system that either chatters annoyingly or remains awkwardly silent, even though the underlying model is well-trained.

More subtly, the threshold sensitivity curves in Figure 8 reveal that the tradeoff is asymmetric across interaction modes. In Co-Commentary, CC improves monotonically with higher thresholds, peaking at $\tau = 0.9$, while F1 drops from ~80 to near zero over the same range. In Solo Commentary, CC peaks at $\tau = 0.6$ and then declines. In Guidance, the optimal CC threshold is ~0.5. This means a single threshold setting cannot be simultaneously optimal across all three modes — a system that switches between Solo, Co-Commentary, and Guidance (which the paper presents as capabilities of the same model) would need mode-dependent thresholds, but the paper does not provide or evaluate such a mechanism.

What evidence exists in the paper. Figure 8 (Appendix I.1) provides detailed threshold-vs-metric curves for all three interaction modes, clearly showing the tradeoff. The main results (Tables 1–2) use $\tau = 0.3$ by default, while ablations use $\tau = 0.5$ — the paper transparently reports this discrepancy (Appendix C.3) but does not discuss its implications for fair comparison. The response score curve visualization in Figure 5 shows how different thresholds produce qualitatively different speaking patterns (near-continuous triggering at $\tau = 0.1$, all-silence at $\tau = 1.0$, alternating at $\tau = 0.5$), confirming that the threshold is not merely a minor calibration detail but a parameter that fundamentally shapes the interaction experience.

Mitigation status. The paper treats the threshold as a deployment-time hyperparameter to be chosen empirically, but provides no principled method for selection beyond sweeping and inspecting metric curves. There is no learned threshold calibration (e.g., optimizing $\tau$ on a validation set for a target F1-CC tradeoff), no content-adaptive thresholding mechanism, and no study of how robust the optimal threshold is across games or video genres. The threshold sensitivity analysis is descriptive (showing what happens at different values) rather than prescriptive (telling a practitioner how to choose). This is a partially mitigated limitation — the paper provides the data needed to make an informed choice — but the absence of automation or guidance means the burden of threshold selection falls entirely on the deployer.

The Generalization Evidence Is Narrow: One Unseen Game and One General Video Dataset, with Mixed Results

The assumption or constraint. Proact-VL is trained on 10 games spanning multiple genres (RPG, FPS, MOBA, fighting, sandbox, card game). The paper evaluates generalization on two held-out settings: Black Myth: Wukong (an unseen action RPG) and Ego4D Goal-Step (egocentric procedural activity video, a completely different domain). The implicit claim is that strong performance on these two settings demonstrates broad generalization capability.

The consequence. The generalization evidence is thin and the results are mixed. On Black Myth: Wukong (Table 10, Appendix C.5), Proact-VL achieves F1 of 60.06, essentially tied with LiveCC-7B-Instruct at 59.88, and CC of 55.21, slightly behind LiveCC-7B-Base at 56.46. On Ego4D, the generalization is stronger — Proact-VL achieves CC of 63.43 vs. 12.69 for LiveCC-7B-Base — but this is a domain shift from gaming to egocentric video, not from one game to another, and the LiveCC baselines were not trained on egocentric data at all, making the comparison asymmetric.

The key uncertainty is whether Proact-VL's advantage over baselines persists when generalizing to games specifically, as opposed to generalizing to completely different video genres. The Black Myth result suggests the advantage narrows substantially (essentially tied with the best real-time baseline), while the Ego4D result suggests a large advantage that may be partly due to the baselines' lack of egocentric training. With only one unseen game tested, and that game being an action RPG (a genre represented in the training set by Elden Ring and Cyberpunk 2077), the breadth of game-to-game generalization is poorly characterized. Would Proact-VL maintain its advantage on a puzzle game, a racing game, a strategy game with an entirely different visual interface? There is no evidence either way.

Additionally, all evaluation is on English-language content from professional broadcasts and expert influencer channels. The data pipeline was optimized for English (Section 3.1: "we focused on English-language videos"). There is no evaluation on non-English commentary, user-generated content with lower production quality, or streams with heavy background noise and overlapping speakers — conditions that would stress the ASR-derived training labels and the model's ability to ground commentary in degraded visual and auditory input.

What evidence exists in the paper. Section 5.3 and Table 10 (Appendix C.5) provide the generalization results. The game-wise analysis (Figure 10, Appendix J.1) shows consistent improvement across the 10 training games, but this is in-domain evaluation — it demonstrates that the method works across diverse games that were seen during training, not that it transfers to unseen games. The data ablation (Appendix F, Table 17) shows that removing Gaming data drops Gaming CC by 13.08%, and removing Ego4D data drops Ego4D performance by 22.39%, but these are measurements of within-domain data importance, not cross-domain transfer.

Mitigation status. The paper does not frame the narrow generalization evaluation as a limitation, but it implicitly acknowledges the issue by noting the mixed Black Myth result and attributing the Ego4D success to "transferable signals" from guidance-style supervision in the gaming data (Appendix F). No future work is explicitly proposed for broader generalization testing. A more convincing evaluation would include 3–5 held-out games covering genres not represented in training, systematic comparison against baselines trained on the same in-domain data, and evaluation on non-English or lower-quality content to test robustness to the assumptions embedded in the data processing pipeline.

Visual Grounding Remains Fragile, and the Paper Provides No Systematic Measurement of Hallucination Rates

The assumption or constraint. Proact-VL generates commentary based on visual input at 2 FPS with constrained resolution (MIN_PIXELS = 128 × 28 × 28, MAX_PIXELS = 540 × 28 × 28). The model inherits its base vision encoder (from LiveCC-7B-Base or Qwen-series backbones) and receives no specialized training for fine-grained visual recognition, OCR on small HUD elements, or entity identification beyond what the base model provides. The paper's evaluation metrics (CC, LiveU, FinalQ) measure commentary quality holistically through LLM judges or win rates against a reference model, but do not directly measure factual accuracy, visual grounding precision, or hallucination rates.

The consequence. The model can produce fluent, stylistically appropriate commentary that is factually wrong, as demonstrated in the first failure case (Figure 15, Appendix K.4): Proact-VL comments on a "2K lead" in a League of Legends match when the actual gold difference shown on screen is 0.1K. This is a 20× exaggeration derived from misreading small HUD text. The paper notes this explicitly: "accurate commentary in competitive games often requires both reliable OCR on small HUD text and lightweight numerical reasoning (e.g., subtraction and magnitude judgment), which Proact-VL does not robustly support yet" (Appendix K.4).

The holistic metrics (LiveU, FinalQ) may not penalize such errors heavily if the surrounding commentary is fluent and stylistically appropriate, because LLM judges evaluate overall quality rather than fact-checking specific claims against video evidence. A commentator who says "huge 2K gold lead" sounds more engaging than one who says "0.1K gold difference," even though the latter is correct. The paper provides no measurement of how frequently the model makes such errors, whether error rates vary by game or scene complexity, or whether users notice or care about factually incorrect commentary — the user study (Appendix E) uses overall preference judgments that could mask factual inaccuracies beneath stylistic quality.

The second failure case (Figure 16) shows a complementary problem: when the visual interface is too cluttered for reliable grounding, the model degenerates into repetitive filler ("Oh, no!") rather than recognizing its uncertainty and either falling silent or hedging. This is a direct consequence of the architecture's lack of confidence calibration for visual grounding — the response head decides when to speak based on learned patterns, but has no mechanism for assessing whether the current visual input is actually interpretable enough to support meaningful commentary.

What evidence exists in the paper. The failure cases (Appendix K.4) provide qualitative examples but no quantitative frequency analysis. The MVBench evaluation (Appendix H, Table 19) shows specific regressions on tasks requiring fine-grained visual recognition: Object Existence (−7.5), Moving Attribute (−12.0), Moving Count (−8.0), Counterfactual Inference (−6.0). These drops suggest that domain-specific fine-tuning on gaming data may partially overwrite the base model's general visual recognition capabilities, trading off broad visual understanding for gaming-specific commentary style — a tradeoff the paper does not explicitly characterize. No experiment systematically measures hallucination rates in the generated commentary.

Mitigation status. The paper acknowledges this limitation in Appendix L: "commentary language is often inherently open-ended and associative... our current model can still produce content that is only weakly correlated with the on-screen evidence, i.e., the text may be plausible but not tightly grounded in the visual stream. Improving fine-grained visual grounding and reducing hallucinatory or generic narration remain important directions." It further notes that "accurately identifying in-game characters, roles, and entities is still challenging" and suggests "retrieval-augmented grounding (e.g., linking to an up-to-date game knowledge base)" as a potential solution. However, no mitigation is implemented in the current system, and no metric for measuring grounding accuracy is proposed. Future work would need both better evaluation methodology (factual accuracy metrics, not just holistic quality scores) and architectural improvements (higher resolution, higher frame rate, retrieval augmentation) to address this gap.

7. Implications and Future Directions

How This Work Changes the Landscape

Proact-VL changes the landscape by demonstrating that proactivity and real-time generation are not two independent problems to optimize separately, but two facets of a single interactive behavior that must be learned jointly from data that captures both dimensions simultaneously. Before this work, the field had effectively bifurcated into two communities with incompatible assumptions: one built sparse-trigger, long-response proactive systems (VideoLLM-online, MMDuet, LiveStar) and another built dense-output, low-latency real-time captioners (LiveCC, StreamingVLM). Practitioners choosing between these approaches faced an implicit tradeoff — you could have good timing or good content, but not both. Proact-VL resolves this tradeoff by showing that a model trained with a joint objective (causal language modeling for content, transition-weighted classification with stability regularization for speaking behavior, operating on shared transformer representations) achieves substantially better performance on both dimensions than either approach alone, with the most dramatic gains in the hardest setting: multi-speaker Co-Commentary, where Proact-VL achieves F1 of 77.44 compared to 61.26 for the best real-time baseline and 0.16 for the best proactive baseline (Table 2).

This is best understood as a reframing with practical consequences rather than a paradigm shift. The individual components — chunk-wise processing, proactive triggering, streaming KV-cache management — all existed in prior work. What changes is the conceptual organization: the paper argues that the "decide-when-to-speak" and "generate-what-to-say" functions must co-adapt during training through shared representations, and that treating them as a pipeline (first decide, then generate) is a category error that systematically produces either untimely responses or inappropriate content lengths. The ablation in Table 6 provides evidence for this claim: removing the response loss degrades both timing and content quality (CC drops alongside F1 and TimeDiff), confirming that the two objectives are complementary in a way that pipelined training would miss.

The paper also makes a diagnostic contribution that should influence how future proactive systems are designed and evaluated: the recognition that the dominant learning challenge in proactive behavior is not silence-vs-speech class imbalance (which can be handled by standard weighting), but transition-vs-persistence imbalance — the overwhelming dominance of state-continuation seconds over state-change seconds. The transition-weighted loss (γ = 5 for transition steps, γ = 1 for persistence steps, reflecting the observed ~1:5 ratio in training data) and the dual stability regularizer (local smoothness for persistent segments, global speaking rate calibration) are direct responses to this diagnosis. Prior work that treated per-second speaking decisions as independent classification problems was implicitly solving the wrong problem — learning state maintenance rather than state change detection. This diagnosis is transferable: any domain where an agent must learn when to act from temporal data with persistent states (dialogue turn-taking, anomaly detection in monitoring, adaptive sampling in robotics) faces the same imbalance and would benefit from similar transition-aware weighting.

The paper also establishes the Live Gaming Dataset as a new resource category: streaming interaction data with human-derived per-second speaking labels, short timestamped utterances, and multi-speaker coordination patterns. Prior datasets for video-language tasks (MVBench, Ego4D) treated videos as offline units to be described or queried; streaming datasets (LiveCC) provided dense video-caption pairs but lacked explicit speak/silence annotations from human behavior. The Live Gaming Dataset fills this gap for the gaming domain, and its construction pipeline (multi-stage ASR, speaker identification, paralinguistic labeling, domain-specific polishing, persona extraction) provides a template for building similar datasets in other domains (sports commentary, live tutoring, security monitoring). The cross-backbone transfer results (Appendix C.4, Table 9, showing Proact-VL improves Qwen-series models) and the cross-domain generalization to Ego4D (CC of 63.43 vs. 12.69 for LiveCC-7B-Base, Table 10) suggest the dataset captures generalizable interaction patterns rather than model-specific or game-specific artifacts, making it a foundational resource for a research program beyond what any single method paper could achieve.

Finally, the paper provides a realistic benchmark for the difficulty of real-time AI companionship through its baseline comparisons. The near-total failure of prior proactive models (MMDuet F1 of 0.05 in Solo Commentary) on a task that Proact-VL handles with F1 of 63.25 demonstrates that the gap between "deciding when to speak" and "deciding when to speak briefly and appropriately" is far larger than the literature had recognized. The strong performance of offline commercial models on text quality (GPT-4o FinalQ of 4.80 overall, comparable to Proact-VL's 5.03 in Table 1) but their limited timing performance (GPT-4o F1 of 54.88 vs. Proact-VL's 64.87 in Table 2) confirms that even the most capable general-purpose models, when applied naively to streaming interaction, produce reasonable content but miss the temporal dynamics that make interaction feel natural. This establishes a concrete performance ceiling that future work must surpass: building systems that match or exceed offline model content quality while achieving Proact-VL-level temporal alignment.

Follow-Up Research This Work Enables

Disentangling data contribution from architectural contribution through controlled fine-tuning experiments. The most important open question is whether Proact-VL's gains come from the Live Gaming Dataset's scale and quality, or from the joint training architecture (response head + composite loss). A direct experiment would train LiveCC-7B-Base on the Live Gaming Dataset with a standard language modeling objective — treating speaking decisions as a special token prediction problem within the existing autoregressive framework — and evaluate on the same proactivity metrics. If this simpler baseline achieves comparable F1 and CC, then the dataset is the primary contribution and the architectural innovations are secondary. If it underperforms substantially, the joint training approach is validated as necessary. A stronger variant would train three models on the same data: (1) standard LM fine-tuning, (2) LM fine-tuning with a separate lightweight classifier head trained post-hoc on frozen representations, and (3) Proact-VL's joint training. If (2) approaches (3), then the benefit comes from having a dedicated decision head rather than from joint optimization; if only (3) works, then the shared gradient signal during training is essential. This experiment would resolve the central uncertainty in the paper's contribution claim and guide whether future work should invest in better datasets, better architectures, or both.

Testing Proact-VL at higher frame rates and resolutions under real-time latency budgets. The paper explicitly identifies the 2 FPS, reduced-resolution processing as a limitation (Appendix L), and the inference efficiency analysis (Table 7) provides a latency budget baseline: at 364 tokens/frame and 0.3 seconds generation time, the system handles 10–15 FPS. A natural extension would push frame rate to 10 FPS (5× the current sampling rate) while measuring three outcomes: (1) improvement on motion-heavy MVBench categories where Proact-VL currently regresses (Moving Attribute: −12.0, Moving Count: −8.0 from Table 19), which would directly test whether the regressions are due to temporal undersampling; (2) reduction in hallucinated numerical claims from HUD misreading (the League of Legends "2K lead" failure case in Figure 15); and (3) any degradation in real-time latency that would require architectural optimization (e.g., more efficient vision encoders, token pruning for redundant frames, or adaptive frame-rate selection based on scene motion). This experiment would establish the performance ceiling under improved visual fidelity and determine whether the current architecture can scale to practical frame rates, or whether fundamentally different efficiency mechanisms are needed.

Adaptive thresholding that varies the speak/silence decision by content difficulty or interaction mode. Figure 8 shows that the optimal threshold τ differs across Solo Commentary (CC peaks at τ = 0.6), Co-Commentary (CC improves monotonically to τ = 0.9), and Guidance (CC peaks at τ = 0.5). The response score curves in Figure 5 show that different thresholds produce qualitatively different speaking patterns — near-continuous triggering at τ = 0.1, all-silence at τ = 1.0, alternating at τ = 0.5. A natural extension would train a lightweight threshold predictor that takes the current context (visual features, recent speaking history, interaction mode) and outputs a per-chunk threshold, trained to optimize a composite reward balancing F1 and CC. The evaluation would compare fixed-threshold Proact-VL against adaptive-threshold Proact-VL across all three interaction modes simultaneously, measuring whether adaptive thresholding can achieve the best of all worlds — high coverage in Solo, conservative turn-taking in Co-Commentary, and balanced guidance pacing — without manual per-mode tuning. A negative result (adaptive thresholding provides no benefit over the best fixed threshold) would be equally informative, suggesting that the mode-dependent optimal thresholds in Figure 8 reflect evaluation artifact rather than genuine content-dependent tradeoffs.

Combining Proact-VL's speak/silence learning with retrieval-augmented grounding for entity and event accuracy. The paper identifies two distinct failure modes: hallucinated numerical claims from poor visual grounding (Figure 15) and repetitive filler when the visual interface is too cluttered for reliable interpretation (Figure 16). A natural follow-up would augment Proact-VL with a retrieval module that queries an up-to-date game knowledge base (character stats, item properties, map information) at each time step, providing grounding information that the model can reference in its commentary — analogous to retrieval-augmented generation in text-only LLMs. The evaluation would measure hallucination rate on a manually annotated subset of commentary (factual accuracy of specific claims about game state, character identities, numerical values), comparing Proact-VL with and without retrieval augmentation, and against a baseline that uses the knowledge base directly to filter or correct generated commentary post-hoc. This would test whether the hallucination problem is primarily a visual grounding failure (which better vision encoders might fix) or a knowledge grounding failure (which retrieval augmentation would address).

Replicating the joint training approach on non-gaming streaming domains with per-second human behavior labels. The paper demonstrates strong out-of-domain generalization to Ego4D (egocentric procedural video) but mixed results on Black Myth: Wukong (an unseen game where Proact-VL essentially ties LiveCC-7B-Instruct). To determine whether the method (joint proactivity-real-time training) or the data (gaming-specific commentary patterns) drives generalization, a follow-up would apply the Proact-VL training recipe to an entirely different streaming domain where human speaking behavior can be labeled at per-second granularity. Candidate domains: (1) sports commentary (basketball, soccer — existing broadcast data with professional commentators, similar structure to gaming), (2) surgical training videos with instructor narration (where speaking correlates with critical procedural steps), (3) security monitoring with operator commentary (where speaking correlates with anomaly detection). For each domain, the experiment would construct a dataset following the Live Gaming Dataset pipeline template, train Proact-VL from scratch on only that domain's data, and measure whether the transition-weighted loss and stability regularization produce the same benefits observed in gaming. This would test the generality of the paper's central diagnostic — that transition-vs-persistence imbalance is the dominant learning challenge in proactive behavior, independent of domain.

Stress-testing infinite inference stability over sessions lasting multiple hours, not minutes. The current streaming evaluation (Section 5.4, Tables 4 and 5) tests videos up to 50 minutes (one 30-minute, eight 1-hour, and one 2-hour video, per Appendix B.2). The reverse-RoPE mechanism should, in principle, support unbounded streaming, but the empirical evidence covers only ~1 hour of sustained operation. A stress test would run Proact-VL for 8–24 hour continuous streams, measuring commentary quality at hourly intervals, and monitoring for three potential failure modes: (1) KV-cache memory growth that eventually exceeds GPU memory despite the sliding window (since the system cache is never evicted, and the streaming cache retains 80% of tokens after each eviction), (2) gradual degradation in the reverse-RoPE correction that accumulates rounding errors over many eviction cycles, and (3) diversity collapse in commentary content (the model repeating phrases or patterns as the effective context, though bounded in length, becomes stale relative to the very long history). This would establish whether the infinite inference mechanism is genuinely robust for production deployment (e.g., 24/7 live-stream companionship) or whether periodic cache resets or more aggressive context management are needed.

Practical Applications and Downstream Use Cases

Automated esports and gaming live-stream commentary at scale. The most direct application is deploying Proact-VL as a real-time commentator for gaming live streams, particularly for amateur or niche content that lacks professional commentary. The Live Gaming Dataset covers 12 games, and the game-wise analysis (Figure 10) shows consistent strong performance across diverse titles. For a streaming platform hosting thousands of concurrent gaming streams, most of which have no commentary at all, Proact-VL could provide baseline narration that makes content more accessible (for visually impaired viewers, or viewers multitasking who cannot watch the screen continuously) and more engaging (viewers stay longer on streams with commentary). The inference efficiency analysis (Table 7) indicates the system handles 10–15 FPS video with ~0.8 seconds end-to-end latency per chunk on a single H100-class GPU, making it feasible for server-side deployment at scale, though per-stream GPU costs would need to be weighed against the engagement benefit. The threshold parameter τ provides a straightforward knob for platform operators to tune commentary density — higher τ for background ambiance, lower τ for high-engagement featured streams.

Real-time coaching and guidance in complex interactive applications. The Guidance results (Table 1: LiveU of 7.52, FinalQ of 6.02; Table 2: F1 of 53.91) demonstrate Proact-VL's capability to provide timely, action-oriented instruction during gameplay. This generalizes naturally beyond gaming to any application where a user performs a complex task while watching a screen and would benefit from real-time guidance: software tutorials (the model watches the user's screen and provides step-by-step instructions at appropriate moments), medical procedure training (watching a surgical video feed and offering coaching-style prompts), or industrial equipment operation (monitoring a control interface and guiding the operator through complex sequences). The key advantage over existing tutorial systems is proactivity — the model decides when to offer guidance based on what it observes, rather than following a predetermined script or waiting for the user to ask. The failure cases (Figures 15–16) highlight the current limitation — guidance accuracy depends on reliable visual grounding, so safety-critical applications (medical, industrial) would require the grounding improvements identified in Appendix L before deployment.

Accessibility tools for live content through real-time audio description. Proact-VL's Solo Commentary capability (LiveU of 6.89, F1 of 63.25) could be repurposed as a real-time audio description system for live video content, providing spoken narration of visual events for visually impaired users. Unlike existing audio description systems that work on pre-recorded content (where descriptions can be scripted and inserted during natural pauses), a live-stream audio description system faces exactly the challenge Proact-VL addresses: deciding when to interject descriptions without disrupting the native audio, and producing descriptions that fit within the available temporal gaps. Proact-VL's lightweight response head and one-second generation cadence are well-suited to this task — the model could monitor the live audio stream for natural pauses and insert brief visual descriptions during those gaps. The Co-Commentary results (F1 of 77.44) are particularly relevant, as they demonstrate the model's ability to coordinate its speaking with an existing audio stream (the co-commentator's speech), which is essentially the same capability needed for non-disruptive audio description. The main barrier to this application is the current 2 FPS vision processing — audio description benefits from higher temporal resolution to catch rapid visual events — making the frame-rate scaling experiments described above a prerequisite for deployment.

When to Prefer This Method

The paper positions Proact-VL against two families of alternatives: proactive models (which decide when to speak but generate long, high-latency responses) and real-time models (which generate low-latency output but lack explicit speak/silence control). The experimental results in Tables 1 and 2 provide clear evidence for when Proact-VL should be preferred:

• Prefer Proact-VL over proactive baselines (MMDuet, VideoLLM-online, LiveStar) when the interaction requires clip-level, real-time delivery rather than sparse complete responses. The proactive baselines achieve F1 scores near zero (0.05–0.20 overall in Table 2) because they were designed for sparse, event-driven responses — once triggered, they generate long answers unsuited for per-second evaluation. Any application where the companion must speak in short, timely bursts rather than occasional monologues (live commentary, real-time guidance, turn-taking dialogue) should use Proact-VL or a similarly joint-trained architecture. The Co-Commentary results are dispositive here: in multi-speaker settings where turn-taking coordination matters, the proactive baselines fail entirely.

• Prefer Proact-VL over real-time baselines (LiveCC, StreamingVLM) when the application requires explicit control over speaking frequency and timing, not just low-latency generation. The real-time baselines achieve reasonable F1 (LiveCC-7B-Instruct: 56.05 overall) but lack the dedicated response mechanism that enables deployment-time threshold tuning (Figure 8) and explicit stability regularization. If the deployment requires the operator to adjust commentary density (e.g., different τ for different games, user preferences, or interaction modes), or if the domain has a specific target speaking rate that should match human baselines, Proact-VL's response head and global rate calibration provide control that the real-time baselines cannot offer. The gap is largest in Co-Commentary (F1 of 77.44 vs. 61.26 for LiveCC-7B-Instruct), making Proact-VL the clear choice for multi-speaker scenarios.

• Prefer offline commercial models (GPT-4o, Gemini 2.5 Pro) only when text quality is paramount and latency/timing constraints are absent. GPT-4o achieves FinalQ of 6.66 in Guidance (Table 1), exceeding Proact-VL's 6.02, suggesting offline models may produce better consolidated instructional scripts when given full video context. However, they lack streaming capability entirely, making them unsuitable for any real-time application. For offline video description or post-hoc commentary generation, where timing is irrelevant and quality is the sole objective, strong commercial models remain competitive with or superior to Proact-VL. The decision rule should be: if the output will be consumed in real-time alongside the video, use Proact-VL; if the output will be consumed asynchronously or the video is pre-recorded, benchmark Proact-VL against offline models on the specific quality metrics that matter for the application, since the offline advantage is task-dependent (present in Guidance FinalQ, absent in Solo LiveU).