TOPReward: Token Probabilities as Hidden Zero-Shot Rewards for Robotics

ArXiv: 2602.19313

🎯 Pitch

Simply asking a video vision-language model to output task progress as text yields near-zero accuracy on open-source models—but reading its internal token probabilities reveals a latent progress estimator that scores 0.947 correlation across 130+ real-world tasks, completely without training. This zero-shot reward signal not only catches successes near-perfectly but also enables real policy improvement, turning a frozen VLM into a plug-and-play robot training signal.

1. Executive Summary

This paper introduces TOPReward, a zero-shot progress estimator that extracts robotic task-completion signals directly from the token logits of pretrained video Vision-Language Models, bypassing the numerical generation bottleneck that causes prior text-output methods to fail on open-source VLMs. Evaluating across 130+ real-world manipulation tasks on the newly introduced ManiRewardBench and 39 Open X-Embodiment datasets spanning Franka, YAM, and SO-100/101 platforms, TOPReward with Qwen3-VL-8B achieves a mean Value-Order Correlation of 0.947—dramatically outperforming the state-of-the-art GVL baseline, which collapses to near-zero correlation on the same open-source model—while additionally enabling success detection (0.654 ROC-AUC vs. GVL's chance-level 0.519) and consistent real-world policy improvement through advantage-weighted behavior cloning, establishing that robust progress estimation is an emergent capability already latent in pretrained video VLMs, accessible only when one queries internal token probabilities rather than relying on autoregressive text generation.

2. Context and Motivation

The Core Problem: Scaling Robotic Reinforcement Learning Requires Dense Reward Signals That Don't Exist

The fundamental challenge this paper addresses is that real-world robotic reinforcement learning (RL) is starved for reward signals, and existing approaches to providing those signals either require expensive per-task engineering or fail to work reliably on the open-source models that make robotic research reproducible and accessible. This gap creates a bottleneck: Vision-Language-Action (VLA) models have seen rapid progress in pretraining—models like OpenVLA, $\pi_0$, and MolmoAct now demonstrate strong language-conditioned manipulation across diverse embodiments—but actually deploying them reliably in real-world settings demands fine-tuning with RL, and RL in turn needs a reward function.

The paper frames this as a reward bottleneck for VLA (Section 2): without dense, informative feedback at every step of a trajectory, RL algorithms struggle with the extreme sample inefficiency that comes from sparse success/failure signals. In simulation, this is manageable because you can run millions of trials at low cost and often have access to ground-truth state information for reward computation. In the real world, where each trial takes seconds and robot hardware wears down, the cost of sparse-reward RL is prohibitive. The paper cites contexts where prior work attempted real-world RL for robot policies—RL-100 training diffusion-based visuomotor policies directly on physical robots with human-provided success signals, and $\pi^*_{0.6}$ improving $\pi_0$ through real-world RL with human-annotated episode outcomes. Both cases demonstrate the problem: they remain reliant on manual reward specification, highlighting the core gap the paper targets.

Why This Matters: The Stakes of Generalizable Reward Models

This problem has both practical urgency and theoretical significance. On the practical side, the paper's opening section catalogs an emerging consensus in the research community that generalizable process reward models—models that provide fine-grained, step-by-step feedback rather than coarse binary success signals—are essential for making RL tractable in real-world robotics. Without them, each new task, each new robot platform, and each new environment requires a custom reward function. Hand-crafted rewards are fragile under distribution shift, difficult to scale across diverse tasks, and represent a form of domain-specific engineering that undermines the promise of generalist robot policies.

The theoretical stake is equally significant: progress estimation functions as a temporal value function—a function that monotonically increases as a task nears completion and can be interpreted as a measure of "how much of the task has been accomplished so far." As the paper notes (Section 1), this connects directly to the principles of Universal Value Function Approximators, which provide a theoretical framework for general-purpose value estimation across tasks and states. The challenge is that while a well-trained value function is theoretically sufficient for RL, training one that generalizes across tasks requires precisely the kind of large-scale, cross-embodiment world knowledge that VLMs acquire during pretraining.

Prior Approaches and Where They Fall Short

The paper situates existing work along three axes, each with characteristic limitations:

Learned reward models that require task-specific data. A substantial body of prior work trains reward models on curated datasets of robot trajectories. RoboDopamine, for instance, trains a reward model on 3,400+ hours of manipulation data with step-aware multi-view perception, but requires task-specific demonstrations for adaptation to each new task. RoboReward fine-tunes a VLM on a large dataset of robot trajectories with human-provided success labels and progress scores, yet reports clear gaps across different embodiments and camera views, indicating that generalization is not guaranteed. In the paper's words (Section 1):

"these approaches rely on non-scalable constraints: RoboDopamine requires additional demonstrations when adapting to each new task, and RoboReward reports clear gaps across different embodiments and views, indicating limited generalization guarantees."

Other approaches in this category include embedding-based methods like VIP, LIV, and R3M, which learn visual representations that capture progress toward a goal but require task-specific fine-tuning and offer limited language grounding. VQA-style methods like SuccessVQA reframe reward as binary classification—asking a VLM whether the task succeeded—but produce signals too coarse for dense reward shaping, returning only success/failure rather than the continuous progress signal that RL algorithms need.

The common thread across all these approaches is dependence on training or domain-specific resources: they require either collecting extensive robot data, obtaining human annotations, or fine-tuning models for specific embodiments. As the paper frames it, this is fundamentally a scalability problem—each new deployment setting demands new investment.

Training-free value estimation with VLMs that only works on proprietary models. A more directly relevant line of work asks whether VLMs can estimate task progress zero-shot, without any additional training. The current state-of-the-art, Generative Value Learning, poses progress prediction as a temporal ordering problem: given a batch of shuffled trajectory frames, the VLM is prompted to assign per-frame progress scores from 0 to 1, exploiting its semantic grounding to rank frames by task completion. This approach works well on proprietary VLMs like Gemini and GPT-4, achieving strong performance on tasks like dataset filtering and advantage-weighted regression without task-specific reward engineering.

The critical finding that motivates TOPReward, however, is that GVL collapses on open-source alternatives. The OpenGVL benchmark evaluates this paradigm across diverse tasks and model families, and the paper quotes its finding directly: open-source VLMs "fall substantially short of their proprietary counterparts on temporal progress prediction." On Molmo2-8B, GVL achieves a VOC of −0.016—essentially random or anti-correlated predictions. On Qwen3-VL-8B, the VOC reaches only 0.194, far below the useful threshold. This creates a two-tier landscape where progress estimation is viable only if you have access to proprietary API-based models, which raises reproducibility concerns, imposes cost and latency constraints, and prevents on-device deployment.

The paper's central hypothesis is that this failure stems not from a lack of visual understanding in open-source VLMs, but from a representation bottleneck in text generation—specifically, what the paper calls "the models' inconsistent instruction-following and their notorious bias in representing numerical tokens" (Section 1). VLMs like Qwen3-VL and Molmo2 have strong video understanding capabilities developed during pretraining, but when asked to output precise numerical scores (e.g., "progress = 0.73"), they must navigate two simultaneous challenges: (1) following a complex formatting instruction that requires structured output, and (2) producing well-calibrated numerical values in text form. Prior work in NLP has established that LLMs are poorly calibrated when asked to produce precise numerical outputs—their internal representations of quantities are more reliable than their ability to verbalize them. The paper cites cognitive dissonance research showing that language model outputs can disagree with internal representations of truthfulness, and work demonstrating that models "mostly know what they know" through their internal activations even when their generated text is unreliable.

How This Paper Positions Itself

The paper's positioning is built on a single, sharp insight: rather than asking a VLM to generate a progress value, one can instead probe what the model already knows through its internal representations. This is not wholly unprecedented—the paper acknowledges "a growing body of work in NLP" showing that logits, hidden states, and embeddings track certainty and factual accuracy more reliably than generated text, and cites prior robotics work that has begun leveraging such representations for reward definition, "bypassing the instabilities of text generation." But TOPReward takes this principle further than prior work by constructing a probabilistically grounded temporal value function directly from token logits.

The key technical insight is a reformulation of the problem. Instead of prompting the VLM with "what is the progress value from 0 to 1?" (which requires numerical generation), TOPReward poses a binary completion query: "Does this trajectory complete the task? True or False." The probability the model assigns to the token "True" becomes the reward signal. This formulation has several properties that make it well-suited to the problem:

• It sidesteps numerical generation entirely, eliminating the calibration bottleneck that causes GVL to fail on open-source models.
• It is zero-shot, requiring no fine-tuning, no domain-specific data, and no task-specific adaptation.
• It naturally produces a signal that increases with task completion, because as more frames of a successful trajectory are observed, the visual evidence makes the completion statement more plausible—the probability of "True" monotonically rises.
• It is compatible with any VLM that exposes token logits, including open-source models like Qwen3-VL-8B and Molmo2, enabling reproducible research without proprietary API dependencies.

The paper is careful to frame this not as a claim that open-source VLMs are better than proprietary ones, but rather that the text-generation interface is the wrong way to access their capabilities. The paper's results—0.947 VOC on Qwen3-VL-8B versus GVL's near-zero performance—support this narrative: the progress estimation capability was there all along, but prior methods couldn't see it because they were asking the model to do something it's not good at (generate calibrated numbers) rather than something it is good at (evaluate the probability of a statement being true).

The Evaluation Gap: Why ManiRewardBench Was Needed

A secondary motivation that becomes clear in Section 4 is that existing benchmarks for reward modeling in robotics are limited in scope and "do not fully stress-test reward models on real-world manipulation trajectories." The paper introduces ManiRewardBench to fill this gap, designing it explicitly to evaluate three dimensions that matter for practical deployment: progress sensitivity (whether the reward model produces accurate, monotonic progress estimates), completion detection (whether it can distinguish successful from failed trajectories), and cross-embodiment robustness (whether it works across different robot platforms). The benchmark spans 130 unique manipulation tasks across four robot platforms (Franka, SO-100/101, single-arm YAM, bimanual YAM) with subtask-level temporal annotations, making it possible to probe fine-grained progress understanding rather than just coarse success/failure. The inclusion of both successful and failed trajectories—23 tasks with 156 episodes mixing both outcomes—enables evaluation of success detection, which the paper demonstrates is a blind spot of VOC-based methods that the benchmark is designed to expose.

Summary of the Motivation Architecture

The paper's motivation can be understood as a chain of dependencies, each motivating the next step:

1. Real-world robot RL needs dense reward signals → hand-crafted rewards don't scale across tasks and embodiments.
2. Learned reward models exist but require task-specific data → they reintroduce the scaling problem they were meant to solve.
3. Training-free VLM methods exist but only work on proprietary models → this creates a reproducibility and accessibility gap (GVL on Qwen3-VL: 0.194 VOC).
4. The failure of open-source VLMs on GVL is not a vision problem but a text-generation problem → VLMs can't produce well-calibrated numerical outputs, but their internal token probabilities are reliable.
5. Therefore: extract progress from token logits, not generated text → formulate as binary completion probability rather than numerical value prediction.
6. Evaluate rigorously on a benchmark that tests the dimensions that matter → ManiRewardBench provides the needed stress-test that existing benchmarks lack.

This structure positions TOPReward not as an incremental improvement over GVL but as a paradigm shift in how progress estimation interfaces with VLMs—moving from a text-generation interface (which is fragile and model-dependent) to a logit-probing interface (which is robust and leverages the model's pretrained capabilities directly). The paper's title captures this framing precisely: token probabilities are "hidden" zero-shot rewards—they're already there, but you have to know where to look.

3. Technical Approach

3.1 Reader Orientation (Approachable Technical Breakdown)

TOPReward is a zero-shot reward calculator that watches a robot's video and estimates how much of the task has been completed — without any training, fine-tuning, or task-specific adaptation. It solves the problem that open-source vision-language models can see task progress in videos but can't reliably say what percentage of the task is complete; TOPReward sidesteps this by asking a yes/no question ("Is the task complete?") and using the model's internal confidence in "yes" as the progress signal, converting a brittle text-generation problem into a robust probability-extraction problem.

3.2 Big-Picture Architecture (Diagram in Words)

The TOPReward system consists of four components connected in a feedforward pipeline:

1. Input Video and Instruction — a robot trajectory video (sequence of frames) and a natural language task description (e.g., "fold the towel") enter the system together.
2. Video Prefix Sampler — the full trajectory is sliced into $K$ uniformly spaced prefixes (the first $t_1$ frames, first $t_2$ frames, ... up to the full trajectory), each representing a "snapshot" of partial task execution.
3. VLM Token-Probability Extractor — for each video prefix, the system constructs a prompt asking whether the observed frames complete the instruction, feeds the prompt through a pretrained video VLM, and reads off the raw logit probability assigned to the token True (the model's internal confidence that the answer is "yes, the task is complete").
4. Normalization and Reward Output — the raw log probabilities across all prefixes are min-max normalized into the $[0, 1]$ range to produce a clean progress curve, and optionally converted into per-step dense rewards via an exponential scaling of progress increments.

Information flows linearly: full video → prefix slices → VLM probability queries → normalized progress scores → (optional) per-step rewards. No component involves training; the VLM backbone is used frozen with pretrained weights.

3.3 Roadmap for the Deep Dive

• First, the core reward definition (Equation 1) — how log probabilities become a progress signal, why the True token, and what the prompt format looks like. This is the mathematical heart of the method.
• Second, the progress estimation pipeline — how prefix sampling, min-max normalization, and dense reward conversion work together to produce a temporal progress curve from a set of scalar probability queries.
• Third, the design choices and their justifications — why no chat template, why True over other affirmative tokens, why the binary completion query over alternative formulations. These choices distinguish TOPReward from GVL and explain why it works where GVL fails.
• Fourth, the ManiRewardBench benchmark construction — what makes this benchmark different from existing evaluation resources, how the subtask annotations work, and why it was necessary for rigorous evaluation. While the benchmark isn't part of TOPReward's internal mechanism, understanding its design is essential for interpreting the experimental results.

3.4 Detailed, Sentence-Based Technical Breakdown

TOPReward is fundamentally an inference-time method that repurposes the internal probability distributions of pretrained video VLMs as temporal value functions. The core idea is deceptively simple: instead of asking a VLM to generate a calibrated numerical score (which open-source VLMs do poorly), ask it to evaluate the truth of a statement about task completion and read off its internal confidence. This section walks through every component of this process, the mathematical formalism, and the design decisions that make it work.

The Core Reward Definition: Token Probabilities as Progress

The problem formulation. The paper formalizes the progress estimation task as follows: given a natural language instruction $x$ and a video trajectory $\tau_{1:T} = (I_1, \dots, I_T)$ where $I_t$ are frames in chronological order, produce a scalar progress signal $p_t \in [0,1]$ for each prefix $\tau_{1:t}$ (the first $t$ frames) that increases as the task is completed. The prefix notation $\tau_{1:t}$ captures a key insight: progress estimation necessarily operates on partial trajectories, and a good progress function should produce monotonically increasing values as more of the task is observed.

This framing is directly aligned with Universal Value Function Approximators: the progress value $p_t$ is a temporal value function that maps from (state, instruction) pairs to completion probability, where "state" here is not the robot's proprioceptive state but the visual history of the trajectory up to time $t$.

The key insight: binary query instead of numerical generation. GVL's approach is to prompt the VLM with shuffled frames and ask it to output a number between 0 and 1 for each frame — a text-generation task that requires both instruction-following (to produce correctly formatted output) and numerical calibration (to map visual evidence to precise numeric values). TOPReward's key insight is that these are the wrong capabilities to demand from the model. Instead, the paper asks a simpler question that maps directly onto the VLM's pretraining objective of next-token prediction: "Is the statement true?"

The prompt template in the paper (Section 3.1) is:

<|video|> The above video shows a robot manipulation trajectory that
completes the following task: {INSTRUCTION}. Decide whether the above
statement is True or not. The answer is: {a}

where $\{a\}$ is the set of possible answer tokens (True or False). The VLM is never asked to generate an answer autoregressively. Instead, the system directly reads the logit value assigned to the token True at the position where the answer would be generated.

The mathematical definition of reward from logits. Let $p_\theta$ be a VLM defining a next-token distribution with pretrained weights $\theta$ — meaning that for any context (video + text), $p_\theta(\text{token} \mid \text{context})$ returns a probability in $(0,1)$. The paper denotes the video-conditioned textual context (the prompt plus the video prefix) as $c(\tau_{1:t}, u)$ where $u$ is the prompt text. Then the raw reward for a trajectory prefix is:

$r_t = \log p_\theta(a \mid c(\tau_{1:t}, u))$

where $r_t$ is the raw reward at prefix length $t$, $p_\theta(a \mid c(\tau_{1:t}, u))$ is the VLM's predicted probability of the answer token $a$ (specifically, the token True) conditioned on the video prefix $\tau_{1:t}$ and the prompt text $u$, and $\log$ is the natural logarithm.

What it computes: the system takes the first $t$ frames of a robot trajectory, concatenates them with a prompt that claims "this trajectory completes [instruction]," feeds the combined input through a frozen video VLM, and reads the raw log-probability the model assigns to the word True being the next token. This produces a single scalar $r_t \in (-\infty, 0]$ at each prefix length. When $t = T$ (the full trajectory), the system evaluates whether the complete video matches the instruction; when $t$ is small, it evaluates whether the partial video provides enough evidence to conclude the task is done.

Why this form: three deliberate design choices are embedded in this equation:

1. Using $\log$ probability rather than raw probability. Log probabilities spread out values near zero (where raw probabilities of True and False might be close to 0.5) into a more discriminative range — the difference between $p = 0.51$ and $p = 0.99$ is enormous in log space ($\log 0.51 \approx -0.67$ vs. $\log 0.99 \approx -0.01$) but nearly invisible in raw probability space. Since progress estimation requires distinguishing subtle differences in the model's confidence as visual evidence accumulates, log space provides better dynamic range. The paper doesn't explicitly state this rationale, but it is the standard motivation for operating in log-probability space in classification and language modeling.

2. Using the generation probability of True rather than internal hidden states. The paper could have extracted progress signals from the VLM's hidden representations (the activations inside the transformer layers before the output projection), which some prior NLP work has used for truthfulness probing. The logit-based approach is simpler (no need to train a probe) and more interpretable (the probability of True has a direct semantic meaning: "the model thinks the statement is true with confidence $e^{r_t}$"). The logit also enjoys the VLM's full video-conditioning, since the probability of True is computed from the same contextualized representation that would be used for generation.

3. Asking a binary question about the instruction rather than directly prompting for progress. This is the most important design choice and the one that separates TOPReward from GVL. By asking "does this complete the task?" rather than "what percentage is complete?", the method eliminates the need for the VLM to (a) map visual evidence onto a numerical scale, (b) produce a well-calibrated floating-point value, and (c) follow a complex output format. The VLM only needs to do what it was pretrained to do: evaluate the plausibility of a statement given evidence.

Why the token True specifically. The paper selects True (rather than other affirmative tokens like Yes or Correct) for a specific empirical reason documented in Appendix B. The authors evaluated the separation between successful and failed trajectories in terms of mean final-step token probability across different candidate tokens. The token True shows "the largest absolute difference in mean token probability across episodes" (Appendix B, Figure 8). The paper hypothesizes this is because True is a single token in the evaluated model vocabularies (meaning it doesn't require multiple tokenization steps that could introduce variance) and because boolean truth evaluation aligns well with the kind of judgment VLMs learn during pretraining when trained on data involving factual claims and verification.

Why not aggregate over the entire instruction? Appendix A describes an alternative formulation the authors tried and rejected: instead of asking a binary completion question, they tested a formulation where the reward is the log-probability of generating the entire instruction text given the video:

$r_t = \sum_i \log p_\theta(\text{inst}_i \mid c(\tau_{1:t}, u, \text{inst}_{<i}))$

where $\text{inst}_i$ is the $i$-th token of the instruction and $u$ is the prompt between video and instruction. This alternative was "less effective." The paper's hypothesis (Appendix A) is that the model assigns high probability to entities in the instruction whenever they appear in the video, regardless of task completion: if the instruction is "peel the apple" and the video shows an apple in frame 1, the model assigns high probability to the token "apple" even though no progress has been made. In other words, the instruction-conditioned formulation is distracted by object appearance, while the binary completion formulation focuses the model's evaluation on the specific claim that the task is finished.

Building the Temporal Progress Curve

The raw reward $r_t$ is a single scalar at one prefix length. To obtain a full progress curve, the system needs to evaluate $r_t$ at multiple points along the trajectory and convert the resulting sequence into a normalized, well-behaved signal.

Prefix sampling. The paper evaluates Equation 1 on a set of $K$ uniformly spaced prefix lengths $\{t_k\}_{k=1}^K$ where $1 = t_1 < t_2 < \dots < t_K = T$. Uniform spacing means the time between consecutive evaluation points is roughly constant — if the trajectory has 100 frames and $K = 20$, the system evaluates at frames 5, 10, 15, ..., 100. The paper describes this as $K$ model forwards, each producing a raw reward $r_{t_k}$: the VLM processes the video prefix $\tau_{1:t_k}$ through its video encoder, conditions on the prompt text through the language model backbone, and outputs the log probability of True.

The choice of uniform spacing is a practical default; the paper doesn't explore adaptive or task-dependent sampling strategies. The paper also doesn't specify an exact value for $K$, but the qualitative results in Figure 4 show smooth curves with roughly 20–30 evaluation points per trajectory, suggesting $K$ in the range of 20–30 for practical use.

Min-max normalization to $[0, 1]$. The raw log probabilities $r_{t_k}$ live in $(-\infty, 0]$ — they're always negative (or zero for $p=1$) and their absolute magnitude depends on the specific VLM backbone, the task, and the length of the prefix. This means raw $r_{t_k}$ values can't be compared across different trajectories or interpreted directly as "percentage complete." The paper applies per-episode min-max normalization:

$s_{t_k} = \frac{r_{t_k} - \min_j r_{t_j}}{\max_j r_{t_j} - \min_j r_{t_j} + \varepsilon}$

where $s_{t_k} \in [0, 1]$ is the normalized progress score at prefix $t_k$, $\min_j r_{t_j}$ is the minimum raw reward across all $K$ evaluated prefixes in this trajectory, $\max_j r_{t_j}$ is the maximum raw reward across all prefixes, and $\varepsilon$ is a small constant for numerical stability (preventing division by zero when all rewards are identical).

What it computes: within a single trajectory, this transformation (1) subtracts the minimum reward so the lowest-confidence point maps to 0, (2) divides by the range so the highest-confidence point maps to 1, and (3) scales all intermediate points linearly between these extremes. If the trajectory is successful (the model becomes increasingly confident), the earliest frames will typically have the lowest $r_t$ and map near 0, while the final frames will typically have the highest $r_t$ and map near 1.

Why this form: the normalization is per-episode rather than global, which is both a practical necessity and a deliberate design choice. It is practically necessary because different tasks and models produce log probabilities in different ranges — there's no universal "completion threshold" across all tasks that would make global normalization meaningful. It is a deliberate choice because it ensures the progress signal is relative to each individual trajectory's evidence range: the normalized score tells you "where in the confidence range of this specific execution are we right now?" rather than "what absolute probability of completion does this trajectory have?" The paper acknowledges this as a limitation in Section 6: "min-max normalization is performed per-episode, which prevents direct comparison of absolute progress values across different trajectories without additional calibration." However, for within-trajectory progress estimation (which is what RL algorithms need for dense reward shaping), per-episode normalization is exactly the right inductive bias — what matters is whether the agent is making forward progress relative to its starting point, not whether its absolute completion score matches some global standard.

Converting progress to per-step dense rewards. When TOPReward is deployed for policy learning (as in Section 5.3's advantage-weighted behavior cloning), the normalized progress curve needs to be converted into per-step rewards that an RL algorithm can use. The paper defines the per-step reward as the progress increment:

$\Delta_{t_k} = \text{clip}(\tau \cdot \exp(s_{t_k} - s_{t_{k-1}}), \min=0, \max=\delta_{\max})$

where $\Delta_{t_k}$ is the reward assigned to the step between prefixes $t_{k-1}$ and $t_k$, $s_{t_k} - s_{t_{k-1}}$ is the change in normalized progress between consecutive evaluation points, $\exp(\cdot)$ is the exponential function (making the reward sensitive to even small progress when near completion), $\tau$ is a temperature scaling factor controlling how aggressively good actions are distinguished from bad ones, and $\delta_{\max}$ is a ceiling that caps individual step rewards.

What it computes: the formula measures how much progress each action chunk contributes, exponentiates this difference to amplify the reward for actions that make meaningful progress (especially late in the trajectory when even small percentage gains represent important finishing steps), scales by $\tau$ to control the magnitude, and clips at $\delta_{\max}$ to prevent any single step from dominating the learning signal. The $\min=0$ in the clip ensures that actions that don't advance progress (or that regress) receive zero reward rather than negative reward. For the experiments in Section 5.3, the paper uses $\tau = 2.0$ and $\delta_{\max} = 2.0$.

Why this form: the exponential of the progress increment has a specific property that flat increments don't: it is multiplicative in probability space. If progress jumps from 0.2 to 0.4 (a 0.2 absolute increase, or 2× relative), and then from 0.8 to 1.0 (also a 0.2 absolute increase), both receive the same reward under a linear scheme. But the final step is arguably more valuable — it's the difference between "almost done" and "done." The exponential formulation naturally gives higher rewards to progress increments at higher completion levels, encoding the intuition that closing the last gap matters more than making early progress (which might be easier and noisier). The clip at $\delta_{\max}$ prevents this from creating extremely large outlier rewards for actions that happen to cross an evaluation boundary with a large jump.

The Prompt Design and Why Chat Templates Are Avoided

A seemingly minor but empirically critical design choice is the prompt format. The paper explicitly states (Section 3.1): "We do not use any chat template in our experiments." This is not an arbitrary decision — it is a deliberate choice supported by an ablation study in Section 5.4.

What is a chat template? Modern instruction-tuned VLMs are trained to expect inputs in a specific format: typically, a system message, followed by user-assistant turns, with special tokens marking the boundaries. For example, a chat template for Qwen3-VL might look like <|im_start|>user\n<video>Describe this.<|im_end|>\n<|im_start|>assistant\n. This template changes the distribution of inputs the model sees — the model was fine-tuned to follow instructions in this format, and its token probabilities are calibrated differently in chat mode versus raw completion mode.

The ablation result. When the paper wraps the TOPReward prompt in a chat template and re-evaluates, performance drops substantially. For Qwen3-VL-8B, the VOC score falls by nearly 50% relative to the no-template version. For Molmo2-8B, the drop is about 20%. The paper summarizes this in Table 5 and discusses it as the key explanation for why TOPReward underperforms GVL on Gemini: "the Gemini API enforces a chat template on our prompt," which the paper cannot disable.

Why this happens. The paper's hypothesis (Section 5.4) connects this to the pretraining objective: "progress estimation is better aligned with the pretraining objective of next-token prediction." When a VLM is used without a chat template, it treats the input as a straightforward completion problem — "Given this video and this text, what token comes next?" — which is exactly the task it was pretrained on with maximum likelihood. When a chat template is applied, the model shifts into "assistant mode," where it expects to generate full responses, often with hedging, conversational tone, and formatting conventions that don't map cleanly onto the binary True/False probability being extracted. The chat template introduces a distribution shift between the pretraining distribution (where the model learned its video understanding) and the evaluation distribution (where we're asking it to apply that understanding), and this shift degrades the reliability of the token-probability signal.

A practical implication. This finding has a concrete consequence for users of the method: TOPReward works best with base (pretrained) VLMs, not instruction-tuned chat models, or at minimum requires API access that allows disabling the chat wrapper. For open-source models like Qwen3-VL-8B and Molmo2, this is straightforward — the user controls the exact input format. For proprietary APIs like Gemini, it may not be possible, which partially explains the performance gap on Gemini noted in Tables 1 and 2. The paper's recommendation is implicit but clear: if deploying TOPReward, use the raw pretrained model interface, not the chat interface.

The ManiRewardBench Benchmark Design

While ManiRewardBench is an evaluation resource rather than a component of TOPReward itself, understanding its design is essential because the paper's experimental claims depend on it. The benchmark was built to address specific shortcomings in existing evaluation resources that the paper identifies: existing benchmarks "remain limited in scope and do not fully stress-test reward models on real-world manipulation trajectories" (Section 4).

Scope and diversity. The benchmark contains 130 unique manipulation tasks spanning four robot platforms: Franka Emika arms, SO-100/101, single-arm YAM, and bimanual YAM. The task diversity is explicitly designed to test cross-embodiment generalization — a reward model that works on a Franka arm but fails on a bimanual YAM has limited practical value, since real-world deployment environments vary. The tasks cover a deliberately broad range of manipulation categories documented in Appendix D: multi-step reasoning tasks (e.g., "Push the puzzles to spell word GO" requiring spatial reasoning and sequential object manipulation, "Build a pyramid" requiring four distinct subtasks), fine manipulation tasks (e.g., "Rotate the banana by 90 degrees," "Make the screw point to the glue" requiring precise orientation alignment), deformable object handling (e.g., "Fold the towel" with non-rigid geometry), and abstract/symbolic tasks (e.g., "Press enter and then space key" requiring keyboard interaction, "Set table" requiring understanding of cultural conventions).

Stage-aware annotation. This is the feature that distinguishes ManiRewardBench from simpler benchmarks that only label success/failure. For each task in the benchmark, episodes are "manually labeled and segmented into a sequence of predefined subtasks" (Section 4). Each subtask has a start_second (the wall-clock time when the subtask begins) and an end_second (when it ends). Subtasks are "non-overlapping and strictly ordered in time, with each subtask beginning immediately after the previous one ends" (Appendix D.1). For a task like "Clean the table" (illustrated in Appendix D.1, Figure 17), the subtasks might be: (a) Grab the can (0.0s–3.9s), (b) Place the can in the plate (4.0s–6.4s), (c) Grab the spoon (6.5s–9.5s), (d) Place the spoon in the plate (9.6s–11.4s).

This annotation enables two forms of evaluation that a simple success/failure label does not. First, it allows computing a stage-aware ground-truth progress curve: at time $t$, the ground-truth progress is the fraction of subtasks that have been completed by time $t$ (a step function that increases at each subtask boundary). This provides a reference signal against which the reward model's continuous predictions can be compared — the evaluation in Figure 4 directly overlays TOPReward's predicted progress against these stage-aware ground-truth curves, visually demonstrating that TOPReward tracks the step-function structure with reasonable fidelity.

Second, it enables the partial success metric used in the real-world experiments (Table 4). Rather than scoring each trial as binary success/failure, the paper scores each trial as the fraction of predefined subtasks completed, summed over 10 trials to give a maximum score of 10. This metric is more informative than binary success because it can distinguish between a policy that consistently completes 2 of 4 subtasks (partial competence) and one that occasionally completes all 4 but usually fails early (high variance). For reinforcement learning where reward shaping matters, the partial success metric better reflects the quality of the learned behavior.

Failure trajectories. A crucial design decision is the inclusion of both successful and failed trajectories — 23 tasks with 156 episodes mixing both outcomes. This is important for evaluating success detection (Section 5.2), which requires the benchmark to contain trajectories that do not complete the task (so the detector can be tested on its ability to reject them). If a benchmark only contains successful demonstrations, a trivial method that always outputs high progress near the end would score well on VOC (since the ordering would be correct) but would be useless as a success detector (since it would report high completion for failed trajectories too). The paper explicitly uses this aspect of the benchmark to demonstrate that VOC-based evaluation has a blind spot: trajectories that plateau early can still achieve high VOC because predictions remain well-ordered, even though the task is incomplete.

Access control. The paper notes that "to ensure evaluation integrity and prevent data leakage, the underlying dataset will remain restricted" with access "available exclusively through a controlled evaluation protocol" (Section 4). This is a standard practice for benchmarks where the authors want to prevent models from being trained directly on the evaluation data, but it does limit independent verification of the reported results until such access is granted.

Summary of Design Choices and Their Justifications

The TOPReward approach can be understood as a series of deliberate choices, each with a specific justification rooted in either empirical observation or principled reasoning about VLM behavior:

• Binary completion query over numerical progress prediction: eliminates the VLM's need to produce calibrated numerical outputs (the bottleneck that causes GVL to fail on open-source models) and maps the task onto the model's pretraining objective of evaluating statement plausibility.
• Token logit extraction over text generation: bypasses autoregressive decoding entirely, avoiding the compounding errors and calibration drift that occur when models generate multi-token structured outputs.
• True as the completion token: empirically, True shows the largest separation between successful and failed trajectories in token-probability space across multiple VLM backbones (Appendix B, Figure 8), making it the most discriminative single-token signal.
• Log probability over raw probability: provides better dynamic range for distinguishing confidence levels, especially near the decision boundary where raw probabilities cluster around 0.5.
• Per-episode min-max normalization over global normalization: ensures the progress signal is relative to each trajectory's evidence range, which is the right inductive bias for within-trajectory reward shaping, even though it precludes direct cross-trajectory comparison of absolute completion levels.
• No chat template over chat-template formatting: preserves alignment with the pretraining distribution where the model's video understanding was learned, avoiding the distribution shift that degrades token-probability reliability in instruction-tuned chat modes (confirmed by ablation in Section 5.4, Table 5).
• Exponential progress increment over linear increment: amplifies rewards for progress made near task completion, encoding the intuition that closing the final gap is more valuable than making early progress, and provides a natural multiplicative scaling in probability space.

Each of these choices would be defensible a priori, but the paper's contribution is demonstrating that their combination unlocks progress estimation capabilities that were latent in open-source VLMs but invisible to prior methods that demanded text-based numerical outputs. The architecture is intentionally minimal — no additional training, no auxiliary networks, no task-specific adaptation — because the paper's thesis is that robust progress estimation is an emergent property of the VLM's pretrained video understanding, accessible only when the interface is designed to query that understanding in the model's native probability space rather than forcing it through a text bottleneck.

4. Key Insights and Innovations

Innovation 1: The Text-Generation Bottleneck as a Diagnostic Category—And the Logit-Extraction Escape Hatch

The paper's most fundamental intellectual move is not proposing a new algorithm, but renaming the problem that open-source VLMs face on progress estimation. Prior work—GVL and the OpenGVL benchmark—had documented that open-source VLMs fail at temporal progress prediction, but interpreted this as evidence that these models lack the capability: contemporary studies claimed that "open-source VLMs are not yet 'robotics-ready' for progress estimation." The field's working assumption was that better progress estimation would require better models.

TOPReward challenges this diagnosis by separating capability from interface. The paper's hypothesis is that the failure of open-source VLMs on GVL "stems not from a lack of temporal understanding, but from the representation bottleneck of textual output—specifically, the models' inconsistent instruction-following and their notorious bias in representing numerical tokens." This reframes the problem from model quality to measurement protocol: open-source VLMs can see progress in video, but they cannot reliably say what they see in numerical form. The progress estimation capability is present but invisible to methods that demand calibrated numerical generation.

The evidence for this reframing is stark and comes from the paper's head-to-head comparison in Tables 1 and 2. On Qwen3-VL-8B, GVL—which requires the VLM to output structured numerical scores—achieves a VOC of 0.194 on Open X-Embodiment and ranges from 0.164 to 0.544 across ManiRewardBench datasets. TOPReward—which asks the same model a yes/no question and reads its internal token probability—achieves 0.857 and 0.942–0.954 respectively. The jump from near-zero correlation to near-perfect correlation on the same model backbone is diagnostic: the visual understanding was there all along, but the text-generation interface was the bottleneck, not the model's competence. The VOC metric itself demonstrates this pointedly—a value near 0 means the model's predictions are essentially unrelated to the true temporal ordering, which would be the expected outcome if the model truly could not perceive task progress. The fact that probing logits on the same frames yields VOC > 0.94 is strong evidence that the deficit was in the output modality, not the visual processing.

This insight has implications beyond progress estimation. The "text-generation bottleneck" is a diagnostic category that can travel to other domains where VLMs or LLMs are asked to produce calibrated numerical outputs: reward modeling, confidence estimation, quantitative reasoning, or any task where correctness hinges on numerical precision rather than semantic plausibility. The paper's finding that chat templates degrade logit-based progress estimation by 20–50% (Table 5) reinforces this framing: the same model is differently calibrated depending on whether it is in "raw completion" mode versus "assistant" mode, meaning that the measurement interface itself determines what capabilities are observable.

This is a fundamental conceptual shift rather than an incremental improvement. The paper is not proposing a better way to prompt VLMs for numerical output—it is arguing that numerical output is the wrong paradigm entirely for accessing certain kinds of model knowledge, and that token-probability extraction is a more principled interface for capabilities that were acquired during pretraining but are obscured by instruction-tuning and autoregressive decoding.

Innovation 2: The Hidden Reward Hypothesis—Progress Estimation as an Emergent Property of Video Pretraining

The paper advances a substantive claim about what video VLMs learn during pretraining: that the ability to estimate task progress is an emergent property of large-scale video-language pretraining, not something that must be explicitly taught through fine-tuning or reward-specific training data. The paper's title encodes this claim directly—token probabilities are "hidden zero-shot rewards," already present in the model but invisible to conventional prompting.

This is a stronger claim than simply demonstrating that a particular method works. It asserts something about the nature of the representations that VLMs develop: that pretraining on internet-scale video-text data (which includes instructional videos, how-to content, and demonstrations) causes the model to internalize a notion of task completion and progress that can be queried through the probability of a completion statement being true. The model has not been trained on robot manipulation data with progress annotations, yet its token probabilities for "True" systematically increase as visual evidence of task completion accumulates across diverse robot platforms and task types.

The evidence for emergence comes from the zero-shot nature of the results. TOPReward requires no fine-tuning, no task-specific adaptation, and no training data beyond the VLM's existing pretrained weights. Yet on Qwen3-VL-8B, it achieves consistent VOC > 0.94 across all four robot platforms in ManiRewardBench (Franka: 0.949, Bimanual YAM: 0.954, Single-arm YAM: 0.942, SO-100: 0.944 from Table 2)—platforms with different kinematics, camera viewpoints, and task distributions. The cross-embodiment consistency suggests the progress signal is not an artifact of specific visual features that happen to correlate with time, but rather reflects a genuine semantic understanding of "is this task closer to being done?" that transfers across embodiments.

This stands in contrast to the dominant paradigm in learned reward modeling, where models like RoboDopamine require 3,400+ hours of manipulation-specific training data and RoboReward requires human-provided progress scores. Those approaches construct progress estimation capability through domain-specific supervision; TOPReward's results suggest that this capability may have been latent in the VLM all along, accessible if queried correctly. The paper does not claim that pretrained VLMs replace specialized reward models—it claims something more specific: that pretrained VLMs already possess a form of progress estimation that is good enough for reward shaping, success detection, and policy improvement, and that future improvements in video VLMs "should directly translate to better progress estimation" (Section 6).

This reframes the relationship between pretraining and reward modeling. Rather than viewing reward modeling as a separate capability that must be built from scratch or fine-tuned, the paper suggests it may be partially bootstrapped from the world knowledge that already exists in pretrained models, with domain-specific fine-tuning serving to refine and calibrate rather than to create the capability from nothing.

Innovation 3: Exposing VOC's Blind Spot—Why Monotonicity Isn't Enough for Reward Functions

The paper makes a diagnostic contribution that extends beyond its own method: it identifies and empirically demonstrates a fundamental limitation of the Value-Order Correlation (VOC) metric as the primary evaluation criterion for progress estimators, and shows that this limitation has practical consequences for downstream applications like success detection.

VOC measures rank correlation between the chronological order of frames and the order of predicted values. A trajectory whose predictions monotonically increase from 0 to 0.3—plateauing well before task completion—can still achieve high VOC because the ordering is correct, even though the absolute completion level is wrong. The paper formalizes this with a controlled illustration in Section 5.2: synthetic trajectories plateauing at 80%, 50%, and 30% completion all achieve VOC ≥ 0.85, demonstrating that "VOC may not distinguish a well-ordered but incomplete (early-plateau) trajectory from a complete trajectory" (Figure 5).

The empirical consequence of this blind spot is documented on the failure trajectory split of ManiRewardBench. Failed trajectories often exhibit a pattern where the robot makes progress early (approaching the object, grasping it) and then fails (drops the object, misses the placement) without undoing the early progress. Since VOC measures only ordering, these trajectories can achieve high VOC scores that are indistinguishable from successful trajectories—the paper reports that "mean VOC with our method is virtually identical for failed and successful trajectories (0.946 vs. 0.943)" (Section 5.2).

The practical implication is that a progress estimator optimized for VOC alone may be useless—or worse, misleading—for success detection, which requires distinguishing trajectories that reach true completion from those that progress partially and stall. The paper demonstrates that GVL on open-source models collapses precisely on this axis: GVL achieves ROC-AUC of 0.519 on Qwen3-VL-8B for success detection—essentially random—while TOPReward achieves 0.654 using the same model backbone. This is because TOPReward directly measures the probability of instruction completion (which is low for failed trajectories regardless of early progress), rather than the temporal ordering of frames.

This diagnostic insight has implications for benchmark design beyond this paper. The field's reliance on VOC as a progress estimation metric—used by GVL, RoboReward, LIV, and others—may be systematically overestimating the quality of progress signals for downstream RL applications where absolute completion detection matters. The paper's ManiRewardBench was explicitly designed to address this: by including failure trajectories and subtask-level annotations, it enables evaluation of both ordering accuracy (VOC) and completion detection (ROC-AUC), providing a more complete picture of reward model quality. This is not a methodological innovation of TOPReward itself, but rather a benchmark design insight that should influence how future progress estimation methods are evaluated.

Innovation 4: Latent Capability Extraction as an Alternative to Fine-Tuning for Domain Transfer

While the paper's primary framing is about progress estimation for robotics, a broader methodological principle emerges from its results: latent capability extraction through token-probability probing can substitute for domain-specific fine-tuning when the target capability is semantically adjacent to the model's pretraining distribution, even when text-generation interfaces suggest the capability is absent.

The evidence for this principle is the dramatic gap between TOPReward's and GVL's performance on open-source models. GVL represents the text-generation paradigm: it fine-tunes the VLM's output format expectations through prompt engineering, demanding structured numerical scores. When this fails (VOC of −0.016 on Molmo2), the natural engineering response would be to fine-tune the model—train it on robot progress data so it learns to produce calibrated scores. This is the approach taken by RoboReward and RoboDopamine. TOPReward demonstrates an alternative path: keep the model frozen, change the query format from numerical generation to binary logit extraction, and the capability that appeared absent suddenly becomes reliable.

This is not merely a "better prompting" finding. It is evidence that the interface between a model's representations and its outputs can be a more significant bottleneck than the quality of the representations themselves. The paper cites cognitive dissonance research in NLP showing that "language model outputs disagree with internal representations of truthfulness," and work demonstrating that models "mostly know what they know" through activations even when their generated text is unreliable. TOPReward operationalizes this principle for a robotics domain, showing that progress estimation is another capability where internal representations are more trustworthy than generated text.

The ablation on chat templates (Table 5) reinforces this principle: the same model's token probabilities are substantially more reliable when queried in raw pretraining format than in instruction-tuned chat format, because the chat template shifts the model into a generation mode where probabilities are calibrated for conversational fluency rather than factual accuracy. This suggests a more general design principle for leveraging pretrained models: query capabilities in the modality and format closest to the model's pretraining objective, rather than in the format that is most convenient for downstream use.

This principle is broadly applicable to other domains where VLMs or LLMs are pressed into service as zero-shot evaluators, judges, or reward models. The paper's contribution is not inventing token-probability extraction—NLP work had done this for truthfulness and calibration—but demonstrating that it unlocks a capability (progress estimation) that the field had concluded was absent from open-source VLMs, and providing the benchmark infrastructure to quantify exactly how large the gap between text-generation and logit-extraction can be (roughly 0.75 VOC points on Qwen3-VL based on the ManiRewardBench results in Table 2). The magnitude of this gap—from non-functional to near-perfect on the same model—is what elevates this from a technique to a conceptual finding about model evaluation.

5. Experimental Analysis

Evaluation Methodology

• Dataset. TOPReward is evaluated on two primary sources: (1) the Open X-Embodiment (OXE) dataset, specifically the LeRobot collection, from which 39 datasets are selected with 20 randomly sampled episodes each (780 episodes total), spanning diverse tasks, camera configurations, and robot platforms; and (2) ManiRewardBench, the authors' newly introduced benchmark containing 130 unique manipulation tasks across four robot platforms (Franka, SO-100/101, single-arm YAM, bimanual YAM) with 113 tasks and 497 episodes for progress estimation evaluation and an additional 23 tasks with 156 episodes mixing successful and failed trajectories for success detection. The OXE evaluation tests breadth across academic datasets; ManiRewardBench tests depth on real-world, stage-annotated trajectories with known ground-truth subtask boundaries.

• Base model(s). Three video-language models are evaluated: Qwen3-VL-8B and Molmo2-8B as representative open-source models with strong video understanding capabilities (enabling reproducible research without proprietary API dependencies), and Gemini-2.5-Pro as the proprietary baseline (chosen specifically because it provides access to logit distributions required by TOPReward's method). The selection of open-source models is central to the paper's thesis — that open-source VLMs possess progress estimation capability but prior methods couldn't access it — while Gemini provides a ceiling reference from a state-of-the-art proprietary system.

• Metrics. Two primary metrics are used:

  • Value-Order Correlation (VOC): Spearman's rank correlation between the chronological order of input video frames and the predicted progress values. Formally, VOC = rank-correlation(argsort(s_{t_1}, ..., s_{t_K}), (t_1, ..., t_K)), ranging from −1 (perfectly reversed order) to 1 (perfect alignment). This measures whether the progress estimator correctly orders frames temporally, and is the standard metric used by prior work (GVL, RoboReward, LIV).
  • ROC-AUC for success detection: Binary classification performance distinguishing successful from failed trajectories, computed on the failure split of ManiRewardBench (23 tasks, 156 episodes). For TOPReward, success detection uses the average log probability over the last 3 sampled frames; for GVL, it uses the VOC score. This metric captures a capability that VOC does not: recognizing absolute completion rather than just relative ordering.
  • For the real-world behavior cloning experiments, partial success score is used: the fraction of predefined subtasks completed per trial, summed over 10 trials (maximum score 10). This is more informative than binary success because it distinguishes partial competence from high-variance performance.

• Baselines. The primary baseline is GVL (Generative Value Learning) (Ma et al., 2024), the state-of-the-art training-free progress estimator that "poses progress prediction as a temporal ordering problem: given a batch of shuffled trajectory frames, the VLM is prompted to assign per-frame progress scores, exploiting its semantic grounding to rank frames by task completion" (Section 2). GVL is evaluated in its zero-shot configuration on the same model backbones for direct comparison. For the real-world behavior cloning experiments, the baseline is standard Behavior Cloning (BC) that minimizes the unweighted flow matching loss on the same demonstration dataset, without advantage weighting from TOPReward.

• Generation budget / compute accounting. TOPReward's compute cost scales with K, the number of uniformly spaced prefix lengths evaluated per trajectory. Each prefix evaluation requires one VLM forward pass (processing the video prefix through the vision encoder and the prompt through the language model to extract the True token logit). The paper does not specify K explicitly but Figure 4 shows smooth curves with approximately 20–30 evaluation points per trajectory. This cost is directly comparable to GVL's approach of scoring all frames in a shuffled batch, though TOPReward avoids the computational overhead of autoregressive text generation (which GVL requires to produce its structured numerical output). No specific FLOP budget or wall-clock time comparison between methods is reported.

• Cross-validation / statistical protocol. No k-fold cross-validation or statistical significance testing is reported. For the OXE evaluation, results are reported as mean VOC over 39 datasets with 20 episodes each, with error bars in Figure 3 showing standard deviation across datasets within each evaluation set. For ManiRewardBench, results are reported as mean VOC over 113 tasks and 497 episodes, though no per-task variance or confidence intervals are provided. The paper does not report train/validation/test splits for any component, as TOPReward requires no training — all evaluations are zero-shot on the evaluation data. The restricted-access protocol for ManiRewardBench is intended to prevent data leakage rather than to support statistical validation procedures.

Main Quantitative Results

Large-Scale Zero-Shot Progress Estimation on Open X-Embodiment

The headline result on Open X-Embodiment (Table 1, Figure 3) is that TOPReward with Qwen3-VL-8B achieves a mean VOC of 0.857, dramatically outperforming GVL's 0.194 on the same model — a difference of 0.663 in rank correlation. This is the central empirical demonstration of the paper's thesis: the capability is present in the open-source model, but GVL's text-generation interface cannot access it.

Breaking this down by model backbone (Table 1):

• Qwen3-VL-8B: TOPReward achieves 0.857 VOC vs. GVL's 0.194 — a +0.663 improvement. This is the strongest result and the paper's primary evidentiary claim.
• Molmo2-8B: TOPReward achieves 0.417 vs. GVL's −0.016. While TOPReward's absolute performance is lower on Molmo2 than Qwen3-VL, the relative improvement (+0.433) remains substantial, and more importantly, GVL's negative VOC on Molmo2 indicates anti-correlated predictions — the model cannot even consistently order frames correctly via text generation, while TOPReward extracts a moderately useful signal from the same backbone.
• Gemini-2.5-Pro: GVL achieves 0.541, outperforming TOPReward's 0.433. This reversal — where the text-generation method wins on the proprietary model — is attributed to the Gemini API enforcing a chat template, which the ablation in Section 5.4 demonstrates degrades TOPReward's performance. The paper treats this as a limitation of API access rather than a fundamental weakness of the logit-extraction approach: "the Gemini API forces a chat template, which negatively affects our method."

The per-dataset breakdown (Appendix C, Table 6) shows that TOPReward's advantage over GVL on open-source models is consistent rather than driven by outliers. The per-episode VOC distributions (Appendix C, Figure 9) visually confirm that TOPReward's scores are concentrated near high VOC values on both ManiRewardBench and OXE for Qwen3-VL, while GVL's scores on the same model are broadly distributed with many near-zero or negative values. The ΔVOC distribution (Figure 10) shows per-dataset improvement, with positive values dominating for open-source models.

Large-Scale Zero-Shot Progress Estimation on ManiRewardBench

The ManiRewardBench results (Table 2, Figure 3) demonstrate remarkably consistent performance across diverse robot platforms. On Qwen3-VL-8B, TOPReward achieves VOC scores of 0.949 (Franka), 0.954 (Bimanual YAM), 0.942 (Single-arm YAM), and 0.944 (SO-100) — a tight range spanning only 0.012 across four platforms with different kinematics, camera viewpoints, and task distributions. The overall mean across all four datasets and 113 tasks is 0.947.

The comparison with GVL on the same benchmark (Table 2) reveals the stark performance gap on open-source models:

• On Qwen3-VL, GVL achieves per-dataset VOC scores of 0.164–0.544 — substantially below TOPReward and with much higher variance across platforms.
• On Molmo2-8B, GVL achieves near-zero or negative VOC across all four datasets, while TOPReward achieves 0.427.
• On Gemini, GVL achieves 0.400–0.700 across datasets, outperforming TOPReward's 0.340–0.483, consistent with the chat template effect.

The qualitative progress traces in Figure 4 provide visual evidence that TOPReward produces "smooth, monotonically increasing progress signals that closely track stage-aware ground-truth task completion" across diverse manipulation tasks including stacking, pouring, and deformable object handling. The traces show TOPReward's predictions (orange) rising in alignment with the dashed ground-truth step function computed from annotated subtask boundaries, with progress plateaus corresponding to intermediate subtask completions. Overlaid Gemini-GVL traces (blue, when available) exhibit "noisier predictions with frequent non-monotonic fluctuations."

Success Detection on ManiRewardBench

The success detection experiments (Table 3) test whether progress estimators can distinguish successful from failed trajectories on the 23-task failure split of ManiRewardBench (156 episodes mixing both outcomes). The headline finding is that GVL on open-source models fails at success detection, while TOPReward extracts a usable signal.

ROC-AUC results by model:

• Qwen3-VL-8B: TOPReward achieves 0.654 vs. GVL's 0.519. GVL's 0.519 is essentially chance-level (0.5 is random), meaning GVL's VOC-based success detection provides no discriminative power on this open-source backbone. TOPReward's 0.654 is a +0.135 improvement — modest in absolute terms (well below the 0.8+ range typically considered strong) but qualitatively different because it crosses the threshold from non-functional to marginally useful.
• Gemini-2.5-Pro: Both methods perform comparably and substantially better than on Qwen3-VL: TOPReward achieves 0.826 and GVL achieves 0.823. This near-parity on Gemini supports the paper's narrative: on a proprietary model where GVL already works well for progress estimation, success detection performs similarly under both approaches.

The paper explicitly connects these results to the VOC blind spot identified in Section 5.2 (Figure 5): "GVL does not rely on the maximum output progress as an indicator of successful trajectories. Rather, they argue that failed trajectories can be difficult for the model to reshuffle, leading to low VOC scores." But on open-source models, GVL's reshuffling is already poor (VOC near zero or negative), so failure trajectories are not substantially harder to reshuffle than successful ones — eliminating the signal that GVL relies on for success detection. TOPReward sidesteps this by directly measuring completion probability rather than ordering difficulty.

Real-World Advantage-Weighted Behavior Cloning

The real-world experiments (Table 4, Figures 6, 7) deploy TOPReward on six single-arm SO-100 manipulation tasks to test whether the zero-shot progress signal improves policy learning in practice. The setup: a base policy $\pi_0$ pretrained on 200 hours of publicly available SO-100 data is fine-tuned on 50 demonstrations per task (potentially noisy and suboptimal) using either standard Behavior Cloning (BC) or Advantage-Weighted Regression (AWR) with TOPReward-computed advantage weights.

Table 4 reports partial success scores out of 10 (fraction of subtasks completed, summed over 10 trials):

Key observations from Table 4:

• TOP-AWR consistently outperforms BC across all six tasks. The improvement is not marginal — on "Place doll in box," AWR achieves 8.50 vs. BC's 3.00 (a 2.83× improvement in partial success score); on "Pick up cow," AWR achieves perfect 10/10 vs. BC's 8.00.
• The gains are distributed unevenly across tasks. Tasks where BC already performs well (Pick up cow: 8.00, Pick up blue block: 7.25) see modest improvements to 9–10. Tasks where BC struggles (Place doll in box: 3.00, Put eggplant in basket: 3.25) see substantially larger relative gains, suggesting TOPReward's progress signal is most valuable when the demonstration data contains high variance or suboptimal actions that BC naively imitates.
• The AWR hyperparameters are fixed across all tasks ($\tau = 2.0$, $\delta_{\max} = 2.0$), meaning no per-task tuning was performed — the method works out of the box.

Figure 7 provides a qualitative rollout comparison on "Place doll in box," showing the pretrained policy and BC both failing (the robot drops the doll or misses the box) while TOP-AWR succeeds consistently (frames show the doll being placed correctly in the box). This is presented as a representative example rather than a systematic qualitative analysis across all six tasks.

Ablation Studies and Robustness Checks

Chat template effect on TOPReward VOC: Adding a chat template to the TOPReward prompt substantially degrades performance across both open-source models (Table 5). On Qwen3-VL-8B, VOC drops from 0.857 to a lower value (the paper states "nearly 50%" degradation). On Molmo2-8B, the drop is approximately 20%. This ablation directly supports the paper's hypothesis that progress estimation is better aligned with the pretraining objective of next-token prediction rather than instruction-tuned chat formats, and explains the Gemini performance reversal (since the Gemini API enforces a chat template that cannot be disabled). The implication is that TOPReward's effectiveness depends on querying the model through its pretraining interface, not its instruction-tuned interface.

True token selection: Appendix B (Figure 8) empirically justifies the choice of True over alternative affirmative tokens. The figure shows the top 10 tokens by absolute difference in mean final-step token probability between successful and failed trajectories. True exhibits the largest separation — meaning it provides the strongest discriminative signal for distinguishing completion from non-completion at the final timestep. This ablation validates that the token choice is not arbitrary but is based on empirical signal-to-noise ratio in the token probability space. The paper notes that True is also a single token in the evaluated vocabularies (avoiding multi-token variance) and hypothesizes that "boolean truth evaluation aligns well with the kind of judgment VLMs learn during pretraining."

Alternative reward formulation (instruction-conditioned log-probability): Appendix A describes and rejects an alternative: computing reward as the log-probability of generating the entire instruction text given the video, rather than the binary True/False completion query. This alternative was "less effective." The paper hypothesizes that the model assigns high probability to instruction tokens whenever corresponding entities appear in the video, regardless of task completion: if the instruction is "peel the apple" and an apple appears in frame 1, the model assigns high probability to the token "apple" even though no peeling progress has occurred. The binary completion formulation focuses evaluation on the claim that the task is finished, preventing this object-appearance distraction. This ablation demonstrates that the specific prompt design — binary completion judgment rather than instruction reconstruction — is critical to the method's effectiveness.

Model backbone sensitivity: The VOC results in Tables 1 and 2 serve as an implicit ablation over model backbones. TOPReward's performance varies substantially: Qwen3-VL-8B achieves VOC ≈ 0.95 on ManiRewardBench, Molmo2-8B achieves ≈ 0.43, and Gemini achieves ≈ 0.43 (with chat template limitation). This variation demonstrates that TOPReward's effectiveness is bounded by the video understanding capabilities of the underlying VLM — the method amplifies video understanding into progress estimation but cannot compensate for poor video understanding. The paper acknowledges this explicitly as a limitation (Section 6), noting that "future improvements in video VLMs should directly translate to better progress estimation."

Cross-platform robustness (implicit in ManiRewardBench results): The near-identical VOC scores across four robot platforms on Qwen3-VL (0.942–0.954 from Table 2) serve as evidence of cross-embodiment generalization, though this is not explicitly framed as an ablation. The consistency across Franka, SO-100, single-arm YAM, and bimanual YAM — platforms with substantially different kinematics and camera viewpoints — suggests the progress signal reflects a semantic understanding of task completion rather than platform-specific visual features. No controlled ablation isolating camera viewpoint, robot morphology, or task distribution is reported.

Important missing ablations: Several experiments that would strengthen the paper are absent. There is no ablation over the number of prefix samples K — the paper does not vary K to show whether progress estimation degrades gracefully with fewer evaluations or plateaus above some threshold. There is no ablation over the normalization scheme — the paper uses per-episode min-max normalization without comparing against alternatives (e.g., softmax normalization, calibration against a fixed reference point, or no normalization). There is no ablation over the exponential temperature $\tau$ or max reward $\delta_{\max}$ in the per-step reward conversion (Equation 3) — the values 2.0 are used without sensitivity analysis. There is no comparison against a simple baseline like using frame-to-frame visual similarity as a proxy for progress (optical flow magnitude, CLIP embedding distance), which would help contextualize whether the VLM is genuinely doing task-level reasoning or simply tracking visual change.

Critical Assessment

Claim: TOPReward with Qwen3-VL-8B achieves 0.947 mean VOC on ManiRewardBench, dramatically outperforming GVL.

What the experiments demonstrate: The reported numbers (Table 2) are unambiguous — TOPReward achieves 0.942–0.954 VOC across ManiRewardBench datasets on Qwen3-VL, compared to GVL's 0.164–0.544 on the same model. The per-episode distributions (Figure 9) and per-dataset breakdowns (Table 6) show this is not an artifact of averaging but a consistent pattern. The qualitative traces in Figure 4 visually corroborate that the progress signals are well-behaved.

What warrants caution: The paper does not report per-task variance or confidence intervals on VOC scores. With 113 tasks contributing to the mean, it is possible that a subset of tasks dominates the average while others perform substantially worse. The qualitative traces (Figure 4) show a small number of examples — these may be cherry-picked for visual clarity. The dataset-level histogram of ΔVOC (Figure 10) provides some distributional evidence but is aggregated at the dataset level rather than the task level, which could mask within-dataset variance. Additionally, ManiRewardBench is the authors' own benchmark — while access is restricted to prevent data leakage, the absence of independent evaluation means the results cannot be independently verified at time of publication.

A subtler concern: VOC measures temporal ordering, not absolute progress calibration. The paper acknowledges that per-episode min-max normalization prevents direct comparison of absolute progress across trajectories. This means a trajectory that fails at 30% completion could still achieve high VOC if TOPReward's predictions monotonically increase to 1.0 (since normalization maps the endpoint to 1.0 regardless of true completion). The paper addresses this by separately evaluating success detection (ROC-AUC), but the reported 0.947 VOC should not be interpreted as "the model knows exactly when tasks are complete" — it means the model correctly orders frames temporally, which is a weaker signal.

Claim: GVL collapses to near-zero correlation on open-source models, while TOPReward does not.

What the experiments demonstrate: This is the paper's strongest empirical result. On Molmo2-8B, GVL achieves VOC of −0.016 on OXE and negative to near-zero values on ManiRewardBench (Table 2, with GVL Molmo2 values reported but difficult to read precisely). On Qwen3-VL, GVL achieves 0.194 on OXE. TOPReward achieves 0.417 and 0.857 respectively. The gap is not marginal — it is the difference between a non-functional signal and a useful one.

What warrants caution: The paper's explanation — that the text-generation bottleneck, not visual understanding, causes GVL's failure — is supported by the logit-extraction results but not directly tested. A direct test would be: evaluate the same VLM on pure video understanding tasks (action recognition, temporal localization) where text generation is not the bottleneck, and show that the VLM performs well on those tasks. The paper does not conduct such a test. The alternative hypothesis — that GVL's prompt format or shuffle-based paradigm is flawed for reasons other than numerical generation — is not ruled out. A head-to-head comparison against other text-generation-based progress estimators (besides GVL) would strengthen the claim.

Additionally, the GVL results on Molmo2 (VOC of −0.016) are concerning because negative VOC suggests predictions are systematically anti-correlated with time — the model is not just noisy but wrong in a consistent direction. This is a more severe failure mode than random noise (which would give VOC ≈ 0) and may indicate a bug in GVL's prompt format for that specific model rather than a fundamental limitation. The paper does not investigate whether GVL could be improved through prompt engineering on Molmo2 before concluding that text generation is the bottleneck.

Claim: TOPReward serves as a reliable success detector, overcoming VOC's blind spot on failure trajectories.

What the experiments demonstrate: On Qwen3-VL, TOPReward achieves 0.654 ROC-AUC compared to GVL's 0.519 (Table 3), demonstrating that TOPReward extracts a completion signal from the same model where GVL's ordering-based detection fails. On Gemini, both methods perform comparably (~0.82–0.83), showing the VOC blind spot is most consequential when the underlying VLM already struggles with calibrated progress estimation.

What warrants caution: The absolute ROC-AUC of 0.654 on Qwen3-VL is modest. In binary classification terms, an AUC of 0.65 means the detector is better than random but far from reliable — at a typical operating point, you would expect substantial false positive and false negative rates. The paper does not report precision, recall, or F1 score at any specific threshold, making it difficult to assess whether 0.654 ROC-AUC translates to practically useful success detection. For comparison, the Gemini results (~0.82) are in a qualitatively different regime — they represent a genuinely useful detector, while the Qwen3-VL results represent a signal that is above chance but may not be sufficient for autonomous decision-making (e.g., automatically terminating a trajectory when success is detected).

The success detection evaluation is also limited to 23 tasks with 156 episodes — a relatively small sample for binary classification evaluation, making the ROC-AUC estimates potentially noisy. Confidence intervals are not reported.

Claim: TOPReward enables consistent real-world policy improvement through advantage-weighted behavior cloning.

What the experiments demonstrate: Across six tasks, TOP-AWR outperforms BC on partial success score (Table 4), with improvements ranging from +1.0 (Pick up blue block: 7.25 → 9.00) to +5.5 (Place doll in box: 3.00 → 8.50). On one task (Pick up cow), AWR achieves perfect performance (10/10) where BC was already strong (8/10).

What warrants caution: Several limitations weaken this claim. First, the experiment involves only 10 trials per task — this is extremely low for drawing conclusions about policy robustness in real-world manipulation where trial-to-trial variance can be high due to object placement, lighting, and robot calibration drift. Statistical significance is not reported, and with 10 trials, even a 2–3 trial difference could arise from noise. Second, there is no ablation isolating whether the advantage weighting specifically (versus simply fine-tuning with the same data and a different loss function) is responsible for the improvement — a comparison against a baseline that fine-tunes with equal weighting but uses the same number of training steps would help disentangle the effect of additional training from the effect of TOPReward's advantage weights. Third, the base policy is pretrained on 200 hours of SO-100 data, and the fine-tuning uses 50 demonstrations per task — it is unclear whether the improvement comes from fine-tuning on task-specific data (which BC also receives) or specifically from the advantage weighting. An experiment where BC is given equal fine-tuning data but not weighted by TOPReward would be a cleaner comparison.

The claim of "consistent" improvement is technically accurate (AWR wins on all 6 tasks) but the small sample size and lack of error bars make "consistent" a qualitative rather than statistical characterization. A single additional trial could flip the outcome on tasks where the margin is small (Put zucchini on plate: 5.75 → 6.40, margin of 0.65 partial success units).

Missing Experiments That Would Strengthen the Paper

Sensitivity to prefix sampling density (K): The paper does not vary the number of timestamp evaluations. Understanding how performance degrades with fewer samples would be important for practical deployment, since each prefix evaluation requires VLM inference.

Comparison against simple visual baselines: Demonstrating that TOPReward's progress signal correlates with task completion beyond what frame-to-frame visual similarity (e.g., CLIP embedding distance between current frame and final frame) would achieve would strengthen the claim that the VLM is doing task-level reasoning rather than generic visual change detection.

Ablation on the normalization scheme: Per-episode min-max normalization is a substantive design choice that affects the shape of the progress curve. Comparing against alternatives (global calibration using a reference set of complete trajectories, softmax over prefix scores, or using raw unnormalized probabilities as a "confidence" signal) would help readers understand whether the normalization is doing heavy lifting or just cleaning up an already-good signal.

Scale analysis: The paper evaluates on 8B-parameter models (Qwen3-VL-8B, Molmo2-8B). Evaluating on smaller variants (e.g., Qwen3-VL-2B if available) would test whether progress estimation capability emerges at a certain scale or is present even in smaller models. This would directly inform the "emergent capability" claim.

Direct probing of visual understanding: To support the claim that GVL's failure is due to text generation rather than visual understanding, the paper could evaluate the same VLM backbones on a pure video understanding task (e.g., temporal action localization, phase classification) where text generation is not the bottleneck, and show high performance. The absence of this experiment leaves open the possibility that open-source VLMs genuinely have weaker video understanding than proprietary models, and TOPReward's improvement over GVL comes partly from a better interface and partly from being evaluated on different (perhaps easier) aspects of progress estimation.

Summary of Claim-Experiment Alignment

The paper's central structural claim — that token-probability extraction from pretrained VLMs recovers progress estimation capabilities that text-generation interfaces obscure — is well-supported by the head-to-head VOC comparisons on open-source models. The secondary claims about success detection and real-world policy improvement are directionally supported but weakened by modest absolute performance (0.654 ROC-AUC), small sample sizes (10 trials per task in real-world experiments), and missing ablations that would isolate the contribution of TOPReward's specific design choices from the effects of additional fine-tuning or task-specific data. The paper succeeds in demonstrating a large and previously undocumented gap between text-generation and logit-extraction performance on progress estimation for open-source VLMs, which is a genuine empirical finding with implications for how the field evaluates VLM capabilities in robotics. The real-world deployment results and success detection results are better interpreted as promising proof-of-concept demonstrations rather than rigorous validation that TOPReward is ready for production use.

6. Limitations and Trade-offs

1. Per-Episode Normalization Prevents Cross-Trajectory Progress Comparison

The assumption or constraint. TOPReward applies min-max normalization within each individual trajectory to map raw log probabilities into the [0, 1] range (Equation 2). This means the normalized progress score $s_{t_k}$ is defined relative to that trajectory's own range of confidence values, not against any absolute standard of task completion. The paper explicitly acknowledges this in Section 6:

"The min-max normalization in Equation 2 is performed per-episode, which prevents direct comparison of absolute progress values across different trajectories without additional calibration."

This is not merely an implementation detail — it is a fundamental constraint on what the reward signal means. The progress value 0.8 in trajectory A and 0.8 in trajectory B may correspond to completely different levels of true task completion, because each is normalized against its own minimum and maximum confidence. A failed trajectory where the model's confidence rises from −5 to −1 (normalized range: 0 → 1) would produce progress values indistinguishable from a successful trajectory where confidence rises from −3 to −0.1 — yet the absolute probability of completion at the endpoint differs by orders of magnitude.

The consequence. This normalization scheme creates a deployment ambiguity for any application that requires comparing progress across trajectories. In reinforcement learning, an agent typically needs to know not just "am I making forward progress relative to my starting point?" but "am I close to completing the task?" — the latter requires some notion of absolute progress that per-episode normalization cannot provide. A policy that learns from per-episode normalized rewards may develop behaviors that optimize the shape of the progress curve (making it look monotonic) without improving the absolute completion probability.

The paper's own success detection experiments (Section 5.2) demonstrate the consequence of this gap: because per-episode normalization erases absolute completion information, success detection requires a separate mechanism — using the average log probability over the last 3 frames without normalization. The fact that the paper must switch to a different signal for success detection reveals that the normalized progress curve is insufficient for that purpose. For a practitioner deploying TOPReward in an RL pipeline, this means the reward signal used for dense shaping (Equation 3) is decoupled from the signal used for episode termination or success evaluation — the agent receives rewards that indicate forward progress but cannot directly infer from those rewards whether the task is actually complete.

Additionally, per-episode normalization is inherently online-unfriendly. To normalize a prefix's reward, the system needs the minimum and maximum log probability across the entire trajectory (including future frames not yet observed). In an online RL setting where the agent is actively executing actions and receiving rewards step by step, these min and max values are unknown until the episode ends. The paper does not discuss how to adapt the normalization for online deployment — one would need either a running estimate of min/max (which would be unstable early in the episode), a fixed calibration set of complete trajectories (which reintroduces domain dependence), or a different normalization scheme entirely (which may not have the same monotonicity properties).

What evidence exists in the paper. The success detection results in Table 3 provide indirect evidence of this limitation. On Qwen3-VL-8B, the normalized progress signal achieves 0.947 VOC (nearly perfect temporal ordering), but the unnormalized success detection achieves only 0.654 ROC-AUC (Section 5.2). The gap between these two numbers — near-perfect ordering but only modest completion detection — is precisely what the normalization limitation would predict: per-episode normalization makes temporal ordering look excellent by construction (since any monotonically increasing signal normalizes to a clean 0→1 curve), but erases the absolute completion information needed to distinguish successful from failed endpoints. The paper's own analysis of the VOC failure mode (Figure 5) shows that trajectories plateauing at 30% completion can achieve VOC ≥ 0.85 — the same mechanism that creates this blind spot in VOC evaluation also means that normalized progress curves can look excellent while hiding the fact that the trajectory never truly completes.

Mitigation status. The paper acknowledges this limitation explicitly in Section 6 but does not attempt to mitigate it within the current method. The suggestion that "the probability of the entire trajectory enables quality comparison across episodes" (Section 6) is a partial mitigation — it recovers cross-trajectory comparison by using the unnormalized endpoint probability — but this requires a different signal than the one used for within-trajectory reward shaping, creating a two-signal architecture that the paper does not fully develop. The paper does not propose a normalization scheme that preserves absolute progress information, nor does it explore calibration against a reference set of complete trajectories, nor does it address the online deployment challenge. This remains an open problem for future work.

2. Success Detection Performance on Open-Source Models Is Modest and Below Production Thresholds

The assumption or constraint. The paper claims TOPReward "naturally supports success detection—where VOC-based methods degrade to chance-level performance on open-source models" (Section 6). This claim is true directionally — TOPReward's 0.654 ROC-AUC on Qwen3-VL is statistically better than GVL's 0.519 — but the absolute performance level merits scrutiny. An ROC-AUC of 0.654 means the detector ranks a randomly chosen successful trajectory higher than a randomly chosen failed trajectory only about 65% of the time. This is a weak classifier by most operational standards.

The consequence. For a practitioner considering TOPReward for autonomous success detection — for example, automatically terminating episodes when the robot has completed the task, or filtering demonstration datasets to retain only successful trajectories — an ROC-AUC of 0.654 is not sufficient for reliable deployment. In a typical binary classification operating regime, this level of discriminative power implies that at a threshold that captures, say, 80% of true successes (recall = 0.8), the false positive rate (failed trajectories incorrectly labeled as successful) would be substantial — possibly 40–60%.

The concrete failure mode: if TOPReward were deployed to filter a dataset of robot demonstrations by automatically detecting successes, it would either (a) retain many failed trajectories (if the threshold is set leniently to include most successes), contaminating the training data with failures, or (b) discard many successful trajectories (if the threshold is set strictly to exclude failures), wasting valuable demonstration data. In either case, the downstream policy trained on the filtered data would suffer.

The paper's real-world behavior cloning experiments (Section 5.3) use TOPReward for advantage weighting, not for binary success detection, so this limitation does not directly affect those results. But for applications that require hard success/failure decisions — which the paper explicitly claims TOPReward supports — the modest absolute performance is a significant practical constraint.

What evidence exists in the paper. Table 3 reports the ROC-AUC values: 0.654 for Qwen3-VL-8B, 0.677 for Molmo2-8B, and 0.826 for Gemini-2.5-Pro. The paper does not report precision, recall, F1, or any operating-point-specific metrics. The success detection evaluation is conducted on only 23 tasks with 156 episodes (Section 4) — a relatively small sample for binary classification, meaning the ROC-AUC estimates may have wide confidence intervals that the paper does not compute. For comparison, the Gemini results (0.826) are in a qualitatively different performance regime: an AUC of 0.82 with 156 episodes represents a detector that could plausibly be deployed with reasonable precision-recall tradeoffs, while 0.65 does not. The paper does not discuss this performance gap between open-source and proprietary backbones as a limitation of the method, though it is implicitly present in the numbers.

Mitigation status. The paper does not address the modest absolute detection performance on open-source models. Section 6 notes that "future improvements in video VLMs should directly translate to better progress estimation," which would also improve success detection, but this shifts responsibility to the model provider rather than offering a methodological solution within TOPReward. The paper does not explore whether calibration techniques (temperature scaling, Platt scaling) could improve the discriminative power of the unnormalized endpoint probability, nor whether ensembling multiple token probabilities (beyond just True) could provide a stronger completion signal. The success detection results serve primarily as evidence that TOPReward has a completion signal that GVL lacks (since GVL achieves chance-level performance), but the paper does not develop this signal into a practically reliable detector.

3. The Method Is Not Demonstrated in Online Reinforcement Learning — Only Offline Weighting

The assumption or constraint. The paper's abstract and introduction frame TOPReward as enabling RL: "developing generalizable process reward models is essential for providing the fine-grained feedback necessary to bridge this gap" (Abstract), and the paper motivates the reward bottleneck by discussing real-world RL efforts like RL-100 and $\pi^*_{0.6}$ (Section 2). However, all experimental validation of TOPReward in a policy learning context (Section 5.3) uses offline advantage-weighted behavior cloning — a form of weighted imitation learning on pre-collected demonstration data — rather than online RL where the agent interacts with the environment, collects new data, and receives TOPReward rewards for novel states.

This is a significant scope gap: behavior cloning with advantage weights is fundamentally different from online RL. In the offline setting, TOPReward computes rewards once on a fixed dataset of trajectories, and the policy is trained to imitate those trajectories with higher weight on high-progress segments. In online RL, TOPReward would need to compute rewards on a streaming basis for trajectories the agent generates during training — trajectories that may be substantially different from the demonstration distribution (since the agent starts from a random or pretrained policy and explores), and that may contain novel failure modes, recovery behaviors, or partially successful attempts that were never present in the demonstration data. TOPReward's progress estimates for these out-of-distribution trajectories are untested.

The consequence. A practitioner who reads the paper and decides to use TOPReward as a reward function for online RL would be operating in an unevaluated regime. Several specific failure modes are plausible but unmeasured:

• Exploration trajectories that make partial progress then fail in novel ways. TOPReward's progress estimates may not generalize to failure modes not represented in the evaluation data. If a novel failure causes TOPReward to assign spuriously high progress (because the visual pattern superficially resembles a successful trajectory at an intermediate stage), the RL agent would receive positive reinforcement for failing behavior.
• Recovery behaviors. An RL agent might learn to recover from mistakes — e.g., dropping an object and picking it up again. TOPReward's progress curve would need to decrease when progress is undone and increase again during recovery. The paper only evaluates on monotonic progress (successful trajectories or failed trajectories that plateau), not on trajectories with non-monotonic ground-truth progress. If TOPReward fails to detect progress regression (because the VLM has never been trained to recognize that an object has been dropped and needs to be picked up again), the agent would receive undifferentiated rewards during recovery, making it hard to learn corrective actions.
• The online normalization problem discussed in Limitation 1 becomes acute in online RL, where the full trajectory is not available at each step to compute per-episode min and max values for normalization.

What evidence exists in the paper. None. The paper contains no online RL experiments. The offline AWR experiments (Table 4) demonstrate that TOPReward-computed advantage weights improve behavior cloning on fixed demonstration datasets, but this only validates TOPReward as a data weighting mechanism for offline learning, not as a reward function for online RL. The distinction between these roles — weighting pre-collected expert data versus providing exploration-guided rewards for online interaction — is not discussed in the paper.

Mitigation status. The paper does not acknowledge this as a limitation. Section 5.3 describes the AWR experiments as demonstrating "the effectiveness of TOPReward in real-world robot learning," which is accurate but imprecise: the demonstrated effectiveness is in offline learning from demonstrations, not in online RL. The paper's broader framing around "bridging the gap" for real-world RL and "providing the fine-grained feedback necessary" for RL agents to learn implies an applicability to online RL that the experiments do not validate. A transparent discussion of what needs to be tested before TOPReward can be used for online RL — robustness to out-of-distribution trajectories, behavior under non-monotonic progress, online normalization strategies — would substantially strengthen the limitations section.

4. Chat Template Sensitivity Creates a Model Access Dependency That Undermines Open-Source Reproducibility Claims

The assumption or constraint. The paper's central narrative is that TOPReward unlocks progress estimation on open-source VLMs, enabling reproducible research without proprietary API dependencies. However, the ablation in Section 5.4 (Table 5) reveals that TOPReward's effectiveness is highly sensitive to whether a chat template is applied — and on some models, the user may not be able to control this. Adding a chat template degrades Qwen3-VL-8B VOC by "nearly 50%" and Molmo2-8B by approximately 20% (Section 5.4). The paper attributes Gemini's underperformance (where TOPReward achieves 0.433 on OXE vs. GVL's 0.541) to exactly this issue:

"the Gemini API forces a chat template, which negatively affects our method as detailed in Section 5.4" (Table 2 footnote).

The consequence. This creates an access dependency that partially undermines the paper's open-source narrative. While open-source models like Qwen3-VL and Molmo2 technically allow users to bypass chat templates (since they control the inference code), this is not always straightforward. Many open-source model distributions come with recommended chat templates baked into their inference pipelines (e.g., in HuggingFace tokenizer.apply_chat_template()), and the default behavior of popular serving frameworks may apply them unless explicitly disabled. A practitioner who naively loads Qwen3-VL through a standard inference pipeline and sends TOPReward prompts may unknowingly trigger the chat template and get substantially degraded performance — a 50% reduction in VOC would turn a near-perfect progress estimator into one that is only marginally better than GVL.

More importantly, this sensitivity means TOPReward's performance is not purely a function of the VLM's underlying video understanding capabilities — it is a function of the interaction between those capabilities and the particular inference-mode distribution the model is evaluated under. The paper's claim that progress estimation is an "emergent capability already present in pretrained video VLMs" (Abstract) is qualified by the fact that the capability only emerges when the model is queried in a specific mode (raw pretraining format) and may be substantially diminished under equally valid query modes (chat format). This makes TOPReward less of a "zero-shot" method in practice than the framing suggests — successful deployment requires specific knowledge about how to interface with the model that may not be obvious from standard documentation.

What evidence exists in the paper. Table 5 directly reports the VOC degradation with chat templates. The paper also discusses this in the context of the Gemini results: "We hypothesize that this is because the Gemini API enforces a chat template on our prompt" (Section 5.4). However, the paper does not systematically test chat template effects across a range of models or inference configurations — only two open-source models and one proprietary model are evaluated, and the proprietary model's no-template performance cannot be measured (since the API does not allow it).

Mitigation status. The paper partially mitigates this by explicitly stating "We do not use any chat template in our experiments" (Section 3.1) and providing the ablation in Section 5.4, so practitioners who read carefully will know to disable chat templates. However, the paper does not provide guidance on how to detect whether a given VLM deployment applies a chat template, nor does it characterize the performance of TOPReward under partial template application (e.g., system prompt only vs. full conversation format). The limitation is acknowledged but not solved — a user deploying TOPReward on a new VLM backbone or through a new API would need to independently test whether their inference configuration introduces a template and how much it degrades performance.

5. The Method Requires a Video VLM with Logit Access — A Narrow and Potentially Brittle Model Dependency

The assumption or constraint. TOPReward's core mechanism — extracting the probability of the token True from the VLM's output distribution — requires a model that (a) accepts video input, (b) exposes raw token logits (not just generated text), and (c) has a vocabulary where True is a single token. These requirements, while met by the three evaluated model backbones (Qwen3-VL-8B, Molmo2-8B, Gemini-2.5-Pro), exclude large classes of models and deployment configurations:

• Image-only VLMs cannot be used, since TOPReward requires temporal reasoning across video frames to assess whether a task has been completed. This excludes many widely-used open-source VLMs that only process single images (e.g., LLaVA variants, early Molmo versions).
• API-based models that don't expose logits are incompatible. Most commercial LLM/VLM APIs (OpenAI, Anthropic, many Google model endpoints) return only generated text tokens with their associated probabilities, not the full logit distribution over the vocabulary. Even Gemini-2.5-Pro access required for this paper is specifically noted as providing "access to logit distributions required by our method" (Section 5) — a capability that is not universally available even within Google's model ecosystem.
• Models where True is tokenized into multiple subword units would need adaptation. The paper selects True partly because "it is a single token in our evaluated vocabularies" (Appendix B). For models with different tokenizers (e.g., those that split "True" into "Tr" + "ue"), the method would need to be modified to aggregate probabilities across the token sequence, which introduces additional complexity and potential noise.
• Future models trained with different tokenizers or vocabularies might not even have a clean "True" token. While most English-language models include it, this is a contingent fact about current tokenizer design, not a guaranteed property.

The consequence. TOPReward's applicability is bounded by a specific set of model capabilities that may not be available in many practical deployment scenarios. A robotics practitioner who wants to use TOPReward as a reward function for their VLA policy cannot simply grab any video VLM — they need one that exposes logits, processes video natively, and has the right tokenization properties. If their preferred model (e.g., for latency, cost, or hardware compatibility reasons) doesn't meet these criteria, TOPReward is not applicable.

This also creates a future-proofing concern: the paper's dramatic results on Qwen3-VL-8B rely on that specific model's combination of strong video understanding and logit accessibility. If future open-source VLM releases shift toward chat-optimized interfaces (which often return generated text but not raw logits) or adopt different tokenization schemes, TOPReward's gains may not transfer. The paper presents TOPReward as unlocking a capability that is "hidden" in pretrained VLMs, but this capability is only accessible through a specific, narrow interface that may not be preserved as the model ecosystem evolves.

What evidence exists in the paper. The paper evaluates only three models, all of which meet TOPReward's requirements. There is no evaluation on models that fail these requirements (e.g., image-only VLMs, logit-inaccessible APIs, or models where True is multi-token) to characterize the performance drop. The paper does not discuss model compatibility as a methodological constraint, and the Appendix B justification for True partly addresses the single-token concern but does not explore what would happen if True were not available.

Mitigation status. The paper does not address this as a limitation. The selection of "three video-language models" (Section 5) is presented as evaluating on representative backbones, not as revealing a narrow set of compatible architectures. The paper does not discuss alternatives for models that don't expose logits (e.g., using hidden state probes instead of output logits, or approximating token probabilities from generated text likelihoods), nor does it provide guidance on minimum requirements for a VLM to be TOPReward-compatible. A practitioner encountering this limitation would need to independently determine whether their available models meet the requirements and, if not, whether approximate alternatives exist.

6. No Evaluation of Sensitivity to Trajectory Quality, Camera Viewpoint, or Temporal Resolution — Generalization Boundaries Are Uncharacterized

The assumption or constraint. The paper evaluates TOPReward on two datasets: Open X-Embodiment (39 academic datasets) and ManiRewardBench (in-house real-world data across four platforms). While this is a reasonable breadth, several deployment-relevant dimensions of generalization are not tested:

• Trajectory quality variation. The evaluation uses expert demonstrations (successful trajectories from ManiRewardBench) or curated datasets (OXE). There is no systematic evaluation of TOPReward's robustness to suboptimal, jerky, or hesitant trajectories — the kind of data an RL agent would generate during early training. Even the failure trajectories in Section 5.2 are pre-collected human failures, not agent-generated exploration trajectories. The paper does not test whether TOPReward's progress estimates remain well-behaved when the robot moves inefficiently, pauses, backtracks, or executes actions at non-demonstration speeds.
• Camera viewpoint shifts. While ManiRewardBench spans four robot platforms (which implicitly vary camera positions), there is no controlled experiment varying camera angle, distance, or occlusion on the same task. The paper cannot distinguish whether TOPReward's cross-platform consistency (0.942–0.954 VOC on Qwen3-VL) reflects genuine viewpoint invariance or simply that the four platforms' default camera setups happen to be within the model's training distribution.
• Temporal resolution and video length. The paper uses uniformly spaced prefix samples at a fixed number of evaluation points K, but does not vary frame rate, video duration, or sampling density. A trajectory captured at 30 fps vs. 10 fps, or lasting 5 seconds vs. 60 seconds, may produce substantially different progress estimates — especially since video VLMs have finite context windows and may behave differently on temporally sparse vs. dense sequences. The paper provides no guidance on minimum frame rate, maximum trajectory length, or recommended sampling density.

The consequence. These untested dimensions mean a practitioner cannot predict how TOPReward will perform when deployed in conditions that differ from the evaluation setup. Specific failure modes that are plausible but unmeasured:

• In early RL training, the agent produces trajectories that are substantially more chaotic than expert demonstrations — rapid oscillations, repeated failed grasp attempts, pauses while the policy "thinks." TOPReward's progress estimates might become noisy, non-monotonic, or spuriously high (if the VLM misinterprets rapid motion as task progress) when applied to such data. This would directly undermine TOPReward's utility as an RL reward function, since the very training regime that needs the reward most (early, noisy exploration) is where the reward might be least reliable.
• A camera repositioned during deployment — e.g., to accommodate a new workspace layout — might produce viewpoints outside the VLM's effective range, causing progress estimates to degrade even on tasks where the robot's behavior is unchanged. Without viewpoint sensitivity characterization, a practitioner cannot assess this risk.
• A task that takes 2 minutes rather than the typical 10–30 seconds in the evaluation data might exceed the VLM's effective temporal context window, causing the model to "forget" early progress cues and produce underestimates of completion.

What evidence exists in the paper. Very limited. The per-dataset VOC breakdown on OXE (Table 6) shows some variation — TOPReward's Qwen3-VL VOC ranges from roughly 0.6 to 1.0 across different OXE datasets — which hints at dataset-dependent performance, but the paper does not analyze why certain datasets perform worse. Camera angle, trajectory quality, video length, and task complexity are all confounded in the dataset identity, so the variation cannot be attributed to any specific factor. The cross-platform consistency on ManiRewardBench (Table 2, Qwen3-VL range: 0.942–0.954) is evidence of some robustness to embodiment, but this is across platforms intentionally designed for similar tabletop manipulation tasks, not across radical viewpoint or quality shifts.

Mitigation status. The paper does not address these generalization dimensions. The limitations section (Section 6) mentions that "our method inherits the visual perception limitations of the underlying VLM: tasks requiring fine-grained spatial reasoning (e.g., precise alignment or small object manipulation) may receive noisy progress estimates when the model cannot visually distinguish intermediate states," which touches on task complexity but not on trajectory quality, viewpoint, or temporal resolution. No systematic stress-testing or sensitivity analysis is reported for any of these dimensions. A practitioner needs to assume that TOPReward transfers to their specific deployment conditions (camera setup, trajectory characteristics, video specifications) without evidence from the paper that such transfer will succeed.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper triggers a fundamental diagnostic shift in how the field evaluates what vision-language models understand about robotic task execution. Prior to TOPReward, the dominant narrative — reinforced by OpenGVL and related benchmarks — held that open-source VLMs simply lacked the temporal reasoning capability needed for progress estimation, and that bridging this gap would require either (a) access to proprietary models with stronger video understanding, or (b) extensive domain-specific fine-tuning on robot trajectory data with human progress annotations. TOPReward dismantles this narrative with a single, sharp empirical result: on Qwen3-VL-8B, VOC jumps from 0.194 (GVL, text generation) to 0.857–0.947 (TOPReward, logit extraction) across OXE and ManiRewardBench. The capability was latent in the model all along — it was the text-generation interface, not visual understanding, that was failing.

The magnitude of this finding constitutes more than an incremental refinement of prompt engineering. It is a paradigm reframing within the specific subfield of VLM-based reward modeling: the unit of analysis shifts from "which model has better temporal understanding?" to "which interface correctly queries the understanding that already exists?" This reframing has a direct corollary that should influence experimental design across the field: whenever a VLM underperforms on a task requiring calibrated output (numerical scores, structured predictions, quantitative reasoning), the default investigative step should be to test whether logit-probability extraction recovers the capability. The paper provides a template for this investigation: formulate the task as a binary or multi-choice query, extract token probabilities directly, and compare against text-generation performance on the same model backbone. The gap between these two numbers — which the paper demonstrates can be enormous (0.75 VOC points on ManiRewardBench) — reveals how much capability is being lost to the text bottleneck.

The paper also reconciles a tension in the literature that was previously interpreted as a model-quality hierarchy. Prior work documented that proprietary VLMs (Gemini, GPT-4 variants) substantially outperform open-source VLMs on temporal progress prediction. This was interpreted through a "better models" lens: proprietary VLMs have superior video understanding acquired through larger-scale pretraining, and open-source models simply aren't competitive. TOPReward's results suggest a more nuanced picture: proprietary VLMs may genuinely have stronger video understanding, but the interface gap between text generation and logit extraction is smaller for proprietary models (perhaps because their instruction-tuning better aligns text output with internal representations) and larger for open-source models (where instruction-tuning may actively degrade calibration on tasks like progress estimation). The chat template ablation (Table 5) provides mechanistic evidence: the same model queried in raw pretraining format versus instruction-tuned chat format shows a 20–50% VOC degradation, suggesting that interface distribution shift — not raw capability — explains much of the observed gap.

This finding redirects research attention along two axes. First, it makes logit-probing a more attractive research direction for any task where VLMs are deployed as zero-shot evaluators, judges, or reward models. The NLP literature had already demonstrated that internal representations track truthfulness and calibration more reliably than generated text; TOPReward extends this principle to video understanding for robotics, providing a concrete methodological template. Second, it makes text-generation-based progress estimation less attractive as a primary approach — at least for open-source models. The paper's evidence that GVL achieves negative VOC on Molmo2-8B and chance-level success detection on Qwen3-VL suggests that further prompt engineering within the text-generation paradigm is unlikely to bridge the gap; the bottleneck is structural, not a matter of finding better prompts. Future progress estimation work should default to logit-based interfaces unless there is a specific reason to prefer text generation (e.g., API access that doesn't expose logits, or tasks requiring open-ended reasoning that can't be reduced to binary queries).

More broadly, the paper contributes a new benchmark infrastructure (ManiRewardBench) that changes what the field can evaluate. Prior benchmarks for reward modeling in robotics focused on ordering accuracy (VOC) but lacked the failure trajectories and subtask annotations needed to evaluate absolute completion detection and fine-grained progress sensitivity. ManiRewardBench's inclusion of 156 mixed-outcome episodes across 23 tasks, with per-subtask temporal annotations, enables the success detection evaluation that exposed GVL's blind spot on open-source models (ROC-AUC of 0.519 vs. TOPReward's 0.654). The paper's demonstration that VOC-optimized progress estimators can achieve near-perfect ordering while failing at completion detection (Figure 5, and the near-identical VOC for failed vs. successful trajectories on ManiRewardBench) should shift benchmark design in the field toward multi-metric evaluation that separates ordering accuracy from absolute progress calibration.

Follow-Up Research This Work Enables

Characterizing the logit-to-generation gap across VLMs and tasks. The paper demonstrates a massive gap between logit-extraction and text-generation performance for progress estimation on Qwen3-VL and Molmo2, but this gap is measured on only one task (temporal progress prediction) and only two open-source models. A natural follow-up would systematically measure the gap across a range of video understanding tasks — action recognition, temporal localization, phase classification, anomaly detection — and across a range of open-source VLM backbones (Qwen3-VL at multiple scales, Molmo variants, LLaVA-NeXT-Video, InternVideo2). For each task and model, compare (a) text-generation performance (prompt the VLM to output a structured answer), (b) logit-extraction performance on a binary or multi-choice formulation, and (c) a fine-tuned specialist model as an upper bound. The resulting matrix would reveal whether the gap is specific to progress estimation (which involves numerical output) or generalizes to any task requiring structured text generation. If the gap is large and consistent across tasks, it would motivate a field-wide shift toward logit-based evaluation interfaces for VLM benchmarking; if it's narrow for most tasks but large for progress estimation specifically, it would point to a specific interaction between numerical calibration and video understanding that merits deeper investigation in VLM training.

Online reinforcement learning with TOPReward rewards on real robots. The paper demonstrates TOPReward in offline advantage-weighted behavior cloning (Table 4), but the framing throughout the introduction and related work emphasizes the reward bottleneck for online RL. The critical follow-up is a direct test: deploy a VLA policy (e.g., OpenVLA or $\pi_0$) on a real robot for a set of 5–10 manipulation tasks, use TOPReward as the dense reward function during online fine-tuning, and compare against (a) sparse human-provided success signals, (b) a fine-tuned reward model like RoboReward, and (c) GVL-based rewards (where applicable). Key measurements: sample efficiency (episodes to reach 80% success rate), final success rate, and reward shaping quality (how well the learned policy's behavior aligns with the intended subtask sequence). This experiment would directly address the paper's unvalidated claim that TOPReward bridges the reward gap for real-world RL. Particular attention should be paid to failure modes that the offline evaluation cannot capture: TOPReward's behavior on agent-generated exploration trajectories (which may be more chaotic than expert demonstrations), its sensitivity to non-monotonic progress (recovery behaviors where the agent drops and re-grasps an object), and the online normalization problem (how to compute per-episode min-max normalization when the full trajectory is not yet observed). A negative result — TOPReward rewards fail to improve online RL while succeeding at offline weighting — would reveal that the method's utility is bounded to data curation and offline learning, not the broader RL setting the paper motivates.

Calibrating TOPReward for cross-trajectory progress comparison. The per-episode min-max normalization (Equation 2) is the paper's acknowledged primary limitation: it prevents comparison of progress values across different trajectories and requires a separate unnormalized signal for success detection. A targeted follow-up would develop and evaluate calibration methods that preserve within-trajectory monotonicity while enabling meaningful cross-trajectory comparison. Candidate approaches: (1) reference-set calibration — collect a small set of complete trajectories per task, use their endpoint log probabilities to define a "true completion" reference point, and normalize all trajectories against this reference rather than per-episode; (2) learned calibration layer — train a lightweight model (e.g., a 2-layer MLP) that maps raw log probabilities to calibrated progress scores, using the ManiRewardBench subtask annotations as supervision, then test whether this generalizes to held-out tasks and platforms; (3) softmax over prefix scores — replace min-max normalization with a softmax that can be computed causally (no future frames needed) and provides a natural probabilistic interpretation. Evaluation would use the failure split of ManiRewardBench: measure whether calibrated progress scores at trajectory endpoints cleanly separate successes from failures (ROC-AUC) while preserving within-trajectory VOC. Success would mean a single signal serves both progress estimation and success detection, eliminating the two-signal architecture that the current method requires.

Scaling analysis: does progress estimation emerge at a specific model scale? The paper evaluates on 8B-parameter models (Qwen3-VL-8B, Molmo2-8B) and one proprietary model (Gemini-2.5-Pro, scale undisclosed). The claim that progress estimation is an "emergent capability already present in pretrained video VLMs" (Abstract) would be substantially strengthened by evaluating TOPReward across model scales — for Qwen3-VL, this might mean testing on the 2B, 8B, and 32B variants if available, or on comparable models at different parameter counts. Key question: is there a threshold scale below which logit-extraction fails entirely (VOC near zero or negative, similar to GVL on small models), or does performance improve smoothly with scale? If the capability emerges suddenly at a specific scale, it would support an "emergent capability" interpretation and have direct practical implications for minimum model size requirements. If performance improves gradually, it would suggest progress estimation is a continuous function of video understanding quality rather than a discrete capability that switches on. Additionally, testing on non-video models (image-only VLMs evaluated on single frames, or on frame pairs without temporal context) would reveal how much temporal reasoning — versus static scene understanding — contributes to TOPReward's performance. This ablation would clarify whether TOPReward is extracting a temporal progress signal or primarily recognizing object-goal spatial relationships from individual frames.

Combining TOPReward with fine-tuned reward models for domain-specific calibration. The paper positions TOPReward as an alternative to fine-tuned reward models like RoboReward and RoboDopamine, but a natural synthesis is to use TOPReward's logit-based progress signal as a pretraining or initialization for fine-tuned reward models. Specifically: take a base VLM (e.g., Qwen3-VL-8B), extract TOPReward progress estimates on a large unlabeled robot dataset, use these zero-shot estimates as pseudo-labels to fine-tune the VLM's own progress prediction head (or full weights), and evaluate whether the fine-tuned model outperforms both the zero-shot TOPReward and a reward model trained from scratch on human-labeled data. The hypothesis: TOPReward provides a strong initialization that captures general progress estimation capability learned during internet-scale pretraining, while fine-tuning adapts this capability to the specific visual distribution, task types, and calibration requirements of a target deployment domain. Evaluation would use a held-out domain (e.g., fine-tune on Franka data, evaluate on YAM data) to test whether the combination generalizes better than either component alone. A negative result — fine-tuning degrades the zero-shot TOPReward signal rather than improving it — would be equally informative, suggesting that the pretrained progress estimation capability is fragile and easily overwritten by narrow-domain supervision.

Stress-testing TOPReward under distribution shift: adversarial camera views, suboptimal trajectories, and novel failure modes. The paper's evaluation uses curated datasets with standard camera setups and (mostly) expert demonstrations. A systematic stress-test would characterize the method's brittleness boundary: at what point does TOPReward's progress signal degrade to unusability? Dimensions to vary: (1) camera perturbation — systematically rotate, translate, and occlude the camera view on a fixed set of tasks, measuring VOC decay as a function of angular deviation from the training distribution; (2) trajectory quality — collect demonstrations at varying quality levels (expert, novice, random exploration, adversarial jitter), measure whether TOPReward's progress estimates distinguish high-quality from low-quality trajectories or collapse to noise on non-expert data; (3) novel failure modes — script specific failure types (grasp failure, placement failure, collision with environment) and evaluate whether TOPReward's probability of True decreases appropriately for each failure, or whether certain failures are "invisible" to the VLM; (4) temporal perturbation — vary video frame rate (1 fps to 30 fps), trajectory duration (5 seconds to 2 minutes), and prefix sampling density to establish minimum requirements for reliable progress estimation. The output would be a deployment envelope — a characterization of the conditions under which TOPReward is reliable — rather than a single point estimate of average performance, which is what the paper currently provides.

Practical Applications and Downstream Use Cases

Automatic demonstration filtering and curation for robot learning datasets. Robot demonstration datasets collected in the wild often contain mixed-quality trajectories: some are clean expert demonstrations that complete the task, others are partial attempts, failures, or mislabeled data. TOPReward's success detection capability, even at a modest 0.654 ROC-AUC on Qwen3-VL, can be used to automatically filter or rank trajectories during dataset construction. A practical pipeline: for a collection of candidate demonstrations, compute TOPReward's average endpoint log probability for each trajectory, retain trajectories above a lenient threshold (prioritizing recall — don't discard good data), and train a policy on the filtered set. The paper's AWR results (Table 4) demonstrate that even with only 50 potentially noisy demonstrations per task, TOPReward-weighted training improves behavior cloning across all six tasks. Scaling this to larger datasets — e.g., filtering thousands of crowd-sourced demonstrations for VLA pretraining — could substantially reduce the human effort required for data curation. The key advantage over manual filtering is that TOPReward provides a consistent, automated quality signal that can be applied at scale without per-task human annotation.

Reward shaping for sim-to-real transfer of robot policies. A common robotics workflow trains policies in simulation with privileged reward access, then deploys them in the real world where those rewards are unavailable. TOPReward can serve as a real-world reward proxy during sim-to-real fine-tuning: train a policy in simulation with ground-truth rewards, deploy it on a real robot, continue fine-tuning using TOPReward as the reward signal, and measure whether real-world performance improves. The paper's cross-platform consistency on ManiRewardBench (0.942–0.954 VOC across four robot platforms on Qwen3-VL) suggests the reward signal should transfer across embodiments, making it suitable for scenarios where the simulation uses a different robot morphology than the real hardware. The practical benefit is eliminating the need for hand-crafted real-world reward functions per task — a single TOPReward deployment, querying a frozen Qwen3-VL-8B, could provide dense rewards for any task within the VLM's visual understanding capabilities. The main practical constraint is the online normalization challenge (Limitation 1), which would need to be addressed for streaming reward computation.

Autonomous success detection for long-horizon task execution. In long-horizon manipulation tasks where a robot must complete a sequence of subtasks (e.g., "clean the table" involving grasping a can, placing it, grasping a spoon, placing it), autonomous detection of subtask completion enables error recovery and progress monitoring without human supervision. TOPReward's normalized progress curve (Figure 4) shows visible plateaus and accelerations that align with subtask boundaries — the qualitative traces in the paper demonstrate that progress increases sharply during active manipulation phases and flattens during transitions. A deployed system could monitor the derivative of the progress curve: a sustained near-zero derivative after a period of increase signals subtask completion, and a failure to reach the expected final progress level after the expected number of subtasks signals task failure. The paper's demonstration that this works across embodiments (Franka, YAM, SO-100) means the monitoring system could be deployed once and used across a fleet of different robots. The practical constraint is the detection latency: TOPReward requires evaluating video prefixes through a full VLM forward pass, which may take several seconds per evaluation point depending on hardware — potentially too slow for real-time intervention during fast manipulation.

When to Prefer This Method

The paper positions TOPReward against GVL (text-generation-based progress estimation) and against fine-tuned reward models (RoboReward, RoboDopamine). The following decision framework emerges directly from the paper's results and acknowledged limitations:

Prefer TOPReward over text-generation methods (GVL) when:

• You are using an open-source video VLM (Qwen3-VL, Molmo2, or similar) as the backbone. The paper shows GVL achieves near-zero or negative VOC on these models, while TOPReward achieves 0.417–0.947 depending on backbone quality.
• You need within-trajectory progress estimation for reward shaping or data weighting, where per-episode min-max normalization is acceptable. The paper validates this in offline AWR (Table 4).
• You have API or model access that exposes raw token logits and allows disabling chat templates. The ablation in Table 5 shows a 20–50% VOC degradation when chat templates are applied.

Prefer TOPReward over fine-tuned reward models (RoboReward, RoboDopamine) when:

• You lack task-specific training data with progress annotations or success labels. TOPReward is zero-shot; fine-tuned models require thousands of labeled trajectories.
• You need cross-embodiment generalization without per-platform adaptation. TOPReward achieves 0.942–0.954 VOC across four robot platforms on Qwen3-VL without any platform-specific tuning.
• You are in a research or prototyping phase where rapid iteration across tasks is more valuable than maximizing absolute reward quality. TOPReward can be deployed immediately on any new task within the VLM's visual understanding range.

Prefer fine-tuned reward models over TOPReward when:

• You need high-reliability success detection (ROC-AUC > 0.8) on open-source models. TOPReward achieves only 0.654 on Qwen3-VL (Table 3), and fine-tuned models trained on domain-specific success/failure data can likely exceed this.
• You are deploying in an online RL setting where per-episode min-max normalization is infeasible. The paper does not validate TOPReward for online RL, and the normalization challenge is unresolved.
• Your task involves fine-grained spatial reasoning (precise alignment, small object manipulation) that the underlying VLM cannot visually distinguish. The paper acknowledges this limitation explicitly in Section 6 and provides no evidence that TOPReward works in these regimes.

Prefer proprietary VLMs with text-generation methods (GVL on Gemini, GPT-4 variants) when:

• You have API access to a proprietary VLM that performs well on GVL (Gemini achieves 0.541 VOC on OXE with GVL), and your application can tolerate API latency, cost, and the reproducibility constraints of closed models. The paper shows that on Gemini, GVL slightly outperforms TOPReward due to the chat template limitation, and success detection is comparable (0.823 vs. 0.826 ROC-AUC, Table 3).
• You do not need to deploy on-device or in low-latency settings where API calls are impractical. TOPReward's open-source compatibility matters most when proprietary API access is unavailable or undesirable.

The paper does not claim TOPReward universally dominates these alternatives; rather, it establishes that TOPReward fills a specific gap — zero-shot progress estimation on open-source VLMs — that prior methods left empty. The decision to use TOPReward should be driven by whether your deployment constraints (open-source requirement, no training data, cross-embodiment need) match the gap the paper fills.