EgoEdit: Dataset, Real-Time Streaming Model, and Benchmark for Egocentric Video Editing

ArXiv: 2512.06065

🎯 Pitch

Switching from third-person to first-person footage completely breaks state-of-the-art video editors—EgoEdit fixes this with the first real-time streaming egocentric editor that runs live on a single GPU. Its secret is a carefully curated dataset of 100k editing pairs that explicitly preserves hands during object swaps under chaotic first-person motion, enabling AR effects like turning a banana into a fish at 38 frames per second.

---

1. Executive Summary

This paper introduces a complete ecosystem for real-time, instruction-guided egocentric video editing targeted at interactive augmented reality applications, comprising a manually curated dataset (EgoEditData—49.7k videos, 99.7k editing pairs featuring hand-object interactions under egomotion), a streaming autoregressive editing model (EgoEdit—distilled via Self-Forcing and bidirectional DMD to achieve 855ms first-frame latency at 38.1fps on a single H100 GPU), and a standardized 15-task benchmark (EgoEditBench—spanning object substitution, removal, stylization, and cross-modal tasks evaluated with a VLM judge). The model achieves state-of-the-art VLM scores on egocentric editing (7.76 vs. 5.44 for the strongest open-weight competitor, Lucy Edit) while maintaining competitive performance on general exocentric benchmarks (8.00 vs. 8.26 for EditVerse), with ablations demonstrating that progressively increasing the fraction of EgoEditData in training steadily raises egocentric editing quality (from 4.87 to 7.85 VLM score as data increases from 0% to 100% of the set), establishing that domain-aligned, quality-filtered editing data—not model scale alone—is the primary driver of editing performance in the egocentric regime, while the real-time variant retains comparable benchmark scores to the bidirectional teacher (7.71 vs. 7.76) only after the full two-phase distillation pipeline that compresses 80 NFEs into 4.

2. Context and Motivation

The Core Problem: Video Editing Breaks Down When You Switch to First-Person View

The fundamental question this paper tackles is deceptively simple: can we apply the same AI video editing techniques that work beautifully on third-person videos to the first-person, egocentric footage that augmented reality applications demand? The answer, as the authors document, is a decisive "no" — there is a significant domain gap that causes current state-of-the-art video editors to fail, and nobody has systematically addressed it before.

This gap exists because egocentric video possesses three characteristics that are rare or absent in the exocentric footage on which existing editors are trained and evaluated:

1. Rapid, continuous egomotion. In AR, the camera is head-mounted or held by a moving person. The entire scene undergoes large, unpredictable transformations frame-to-frame — pans, tilts, walking motion, head turns — that dramatically change viewpoint, lighting, and occlusion patterns. Exocentric videos (movie clips, stock footage, user-generated content shot by a stationary or smoothly panning camera) simply do not exhibit this degree of motion complexity.

2. Frequent, complex hand-object interactions. In egocentric video, hands constantly enter and exit the frame to manipulate objects, occluding large portions of the scene. These interactions are semantically rich — the hand is not an obstacle to edit around but a participant in the action that must be preserved. A video editor that successfully replaces a coffee mug with a flower vase must also preserve the hand's natural grasp pose, skin texture, and motion trajectory. If the editor distorts the hand, smears it into the background, or replaces it entirely, the edit fails the realism test that AR demands.

3. Active object manipulation. The objects being edited are frequently not static props in a scene but items being actively handled — squeezed, rotated, lifted, thrown. This means the edit target undergoes viewpoint changes, deformation, and partial occlusion that are temporally correlated with hand motion. Standard editing approaches that work on static or gently moving objects struggle when the target object is being crumpled by a hand.

The paper demonstrates this gap quantitatively: Lucy Edit, a strong open-weight video editor, drops 0.83 points in VLM evaluation score when moving from general (EditVerseBench) to egocentric (EgoEditBench) editing tasks. InsV2V drops 0.47 points. These are not small degradations — they represent a qualitative failure to adapt to the egocentric domain.

Why This Problem Matters: AR Is the Killer App for Video Editing

The importance of solving egocentric video editing extends far beyond academic benchmarking. The paper frames this explicitly (Section 1) as enabling a new paradigm for augmented reality content creation:

"Can instruction-guided editing serve as a direct engine for AR, enabling users to add, remove, or modify scene elements with simple language while they interact with the world?"

Traditional AR experiences require graphics pipelines and significant expert labor to handcraft each application. An artist must manually model 3D assets, rig them for animation, write shaders, set up lighting, and integrate everything into a real-time rendering engine. This ties the potential of AR "to the amount of available expert labor." If instruction-guided video editing could replace this pipeline, any user could transform their environment on-the-fly with natural language — "replace my coffee table with a Victorian oak chest," "make it snow," "add a dragon sleeping on my couch."

The tasks that egocentric editing enables map directly onto core AR use cases identified in the paper (Section 1, EgoEditBench task list):

• Object substitution/insertion/removal — the most fundamental AR interaction: modifying what objects exist in the scene. These are the bulk of the Change Object, Add Object, and Remove Object tasks in the benchmark.
• Environmental restyling — changing the mood, era, or aesthetic of the entire scene (Change Background, Stylization tasks).
• Camera and lighting adjustments — altering how the scene is perceived without changing its content (Change Camera Pose task).
• Effect overlay — adding global post-processing effects like film grain, glow, or VHS distortion (Add Effect task).

The paper also includes cross-modal tasks (Depth-to-Video, Sketch-to-Video, Pose-to-Video and their inverses) which, while less directly AR-relevant, stress-test the model's ability to maintain structural consistency under extreme conditioning changes — an important capability for any AR system that might need to work with sensor data like depth maps or skeletal tracking.

Beyond the AR use case, there is a data scaling argument for why this problem matters now. The paper's central finding — that domain-aligned editing data is the primary driver of editing quality — implies that the current inability of editors to handle egocentric footage is not a fundamental model limitation but rather a data gap. No existing dataset provides high-quality egocentric editing pairs. Closing this gap should unlock capabilities that are already latent in model architectures, making this a ripe problem for dataset-driven progress.

Where Existing Approaches Fall Short

The paper identifies four categories of prior work, each with specific limitations in the egocentric setting:

1. Training-Free Editing Methods

Inversion-based methods (Null-text Inversion, Direct Inversion, PnP Inversion) and attention-control methods (Prompt-to-Prompt, MasaCtrl, FateZero) enable editing without additional training by manipulating the denoising trajectory or attention maps of a pretrained diffusion model. The paper acknowledges their broad applicability but identifies specific failure modes:

"While broadly applicable, these approaches remain brittle for large structural edits and long-range consistency."

The key word here is "structural." Egocentric editing frequently requires replacing an entire object with a structurally different one — a banana becomes a water gun, a screwdriver becomes a feather quill (Figure 2 examples). These are not appearance changes (making the banana yellow-er) but identity changes (replacing the banana with a water gun that has fundamentally different shape, material, and function). Attention manipulation methods, which work by preserving some attention maps while modifying others, struggle when the structural correspondence between source and target breaks down. You cannot attend to the banana's shape and "transfer" water gun textures onto it because the shape itself must change.

Moreover, the long-range temporal consistency requirement of egocentric video — where a replaced object must remain stable across rapid viewpoint changes — pushes these methods beyond their design envelope. TokenFlow and STDF, evaluated as attention-manipulation baselines in Table 1, achieve VLM scores of only 4.99 and 4.59 respectively on EgoEditBench, confirming this brittleness.

2. Trained Video Editors — The Data Gap

The paper traces the dominant paradigm for video editing back to InstructPix2Pix, which established that training on paired "before/after" data produces editors that directly learn the edit mapping. Recent video editors follow this recipe: InsV2V pioneered synthetic video edit pairs, and subsequent work (EditVerse, Lucy Edit, Se˜norita-2M) scaled up both data and models.

The limitation is straightforward: none of these datasets contain egocentric content. Existing editing datasets are built from exocentric sources — movie clips, stock footage, user-generated content — that lack hand-object interactions, rapid egomotion, and active object manipulation. The paper notes that even EditVerse, which carefully filtered existing datasets into a curated 232K video-edit corpus, does not target egocentric scenarios. Training on these datasets produces models that have never seen a hand holding a manipulated object, never encountered the viewpoint changes of head-mounted footage, and never learned to preserve hand structure during an object edit. The distribution shift at inference time is total.

The quantitative evidence: InsV2V, Lucy Edit, and EditVerse all perform strongly on EditVerseBench (general exocentric tasks) but their relative rankings change on EgoEditBench. Lucy Edit, which the paper notes achieves the second-highest EditVerseBench score among open-weight models, drops to 5.44 VLM on EgoEditBench — well below EgoEdit's 7.76. This is a ~30% relative performance loss attributable purely to the egocentric distribution shift.

3. Frame Propagation Methods — A Partial Workaround That Misses the Point

A sub-category of trained editors (Se˜norita-2M, AnyV2V) operates by propagating an edited first frame to the rest of the video rather than editing all frames jointly. The paper's evaluation reveals an important detail: these methods receive as input the first frame edited by EgoEdit itself (Table 1, "‡" footnote). This means their performance on EgoEditBench (7.52 and 6.72 VLM scores, respectively) is inflated — they benefit from EgoEdit's handling of the most critical frame.

Even with this advantage, they underperform EgoEdit's 7.76, suggesting that frame-by-frame propagation cannot capture the temporal dynamics of egocentric video. When a hand moves across the frame occluding a replaced object, propagation methods must hallucinate the hand-object interaction in every subsequent frame based only on a single edited first frame. They lack the joint spatiotemporal reasoning that full video-to-video editing provides.

4. Real-Time Methods — Speed Without Quality

The paper evaluates StreamDiffusion and StreamDiffusionV2 as representatives of the emerging class of real-time video editing systems. Their EgoEditBench VLM scores — 4.31 and 2.55 respectively — are the lowest of all methods tested. This establishes a critical point: existing real-time editors sacrifice too much quality for speed, making them unsuitable for AR applications where edit fidelity is paramount. The gap between StreamDiffusionV2's 2.55 and EgoEdit-RT's 7.71 (both streaming models) quantifies the quality ceiling that the paper's approach breaks through.

5. The Latency Barrier — Real-Time Remains Aspirational

Beyond quality, the paper identifies latency as a fundamental barrier separating existing video editors from interactive AR use:

"AR requires not only edit fidelity but also real-time, low-latency responses suitable for interaction. Many high-quality diffusion editing pipelines remain too slow for this setting."

The latency problem has multiple sources discussed in Section 4.3 and Table 2. Standard diffusion models require 40 denoising steps with classifier-free guidance — 80 total model evaluations (NFEs) — to produce a single video. On a single H100 GPU, this translates to ~6.85 seconds of model time alone, plus 1.52 seconds for autoencoder encoding/decoding, plus 5.06 seconds of recording time for the input video chunk. Total first-frame latency: ~13.4 seconds. For AR interaction, where a user expects to see their edit within roughly one second of issuing a command, this is an order of magnitude too slow.

Moreover, standard models generate the entire video clip before displaying any frames. Even if the total generation time were reduced, the user would still wait for the full video to complete before seeing the first result. This "all-at-once" generation paradigm is fundamentally incompatible with the watch-as-you-generate experience that AR requires.

How This Paper Positions Itself

The paper positions itself not as proposing a single new technique but as building a complete ecosystem that addresses the egocentric editing gap at every level: data, model, and evaluation. This is an unusual framing — most papers in generative AI focus on one component — but the authors argue it is necessary because each component reinforces the others.

The data-model-benchmark triad (Figure 1). The system diagram in Figure 1 shows three interconnected pillars. EgoEditData supplies the training pairs that teach the model to handle egocentric challenges. EgoEdit provides the architectural innovations (channel-wise conditioning for efficiency, distillation for real-time latency) that make the model deployable. EgoEditBench provides the standardized evaluation that measures progress and exposes failure modes. No single pillar is sufficient alone:

• Without EgoEditData, even the best model architecture trained on exocentric data would suffer the same distribution shift as prior work. The ablation in Table 3 makes this explicit: removing all EgoEditData from training drops EgoEditBench VLM score from 7.85 to 4.87 — a 38% relative degradation.
• Without the distillation pipeline, the high-quality editor would be too slow for interactive use (13.4 seconds first-frame latency vs. 855 ms after distillation, Table 2).
• Without EgoEditBench, there would be no standardized way to measure progress on egocentric editing, leaving the community to rely on exocentric benchmarks that do not capture the key challenges.

Positioning against existing datasets. The paper explicitly frames EgoEditData as filling a gap left by existing video editing datasets, which "rarely cover egocentric scenes or such intricate interactions" (Section 3). The emphasis on manual curation and quality filtering distinguishes this dataset from automated large-scale collection efforts. Only 0.4% of the original videos from Ego4D and EgoExo4D survive the full curation pipeline (Section 3.2), which involves: video selection by camera quality and aesthetic score → hand detection and mask segmentation with human review (49.6% retained) → object name extraction → object mask segmentation with human review (43.6% retained) → object editing with expensive Wan 2.1 VACE generation at 0.112 fps → human quality filtering (37.8% of edits retained). This extreme filtering reflects a deliberate design choice:

"Our pipeline prioritizes quality over quantity through strict filtering, and emphasizes creation of editing pairs that depict the most challenging egocentric scenarios with rich hand-object interaction."

This is a direct response to the observation that edit quality tracks the scale and quality of paired data and model capacity. Rather than collecting millions of noisy examples, the paper argues that 99.7k high-quality, carefully validated pairs are more valuable for this specific domain.

Positioning against real-time generation methods. The paper's distillation approach draws on two lines of prior work: Distribution Matching Distillation (DMD) for step compression and Self-Forcing for autoregressive causal distillation. The paper positions EgoEdit as combining these techniques for the first time in the video editing domain, and specifically for the egocentric case. The key insight from Self-Forcing that the paper exploits is that rolling out the student model autoregressively during training forces it to correct its own errors, reducing the exposure bias that plagues naive autoregressive distillation. This is critical for video editing, where errors in early chunks would cascade and compound across the video.

The broader significance. The paper situates itself within a larger trend identified in the Related Work (Section 2): across both images and videos, edit quality tracks the scale and quality of paired data and model capacity. By providing the first high-quality paired dataset for egocentric editing, the paper aims to establish the same data-driven scaling laws for this domain that have driven progress in third-person editing. The fact that even a small fraction of EgoEditData (25%) yields a 2.25-point VLM improvement (Table 3: 4.87 → 7.12) suggests that the return on data investment in this domain is high — the base model already possesses the architectural capacity to handle egocentric edits; it simply lacks exposure to the right training distribution.

The paper also implicitly positions itself against the alternative approach of training-free editing. While StreamDiffusion and StreamDiffusionV2 offer real-time operation without domain-specific training, their poor EgoEditBench scores (4.31 and 2.55) demonstrate that the egocentric domain is too challenging for such lightweight approaches. The strong hand-object interactions, large egomotion, and structural edits required cannot be handled by attention manipulation or image-to-image pipelines alone — they require purpose-built training data and models that have learned to preserve hands, track objects through occlusion, and maintain temporal consistency under rapid viewpoint changes.

3. Technical Approach

3.1 Reader Orientation

EgoEdit is a complete system that takes a live egocentric video stream and a natural language instruction (e.g., "Transform the banana into a water gun") and produces an edited video stream in real time — preserving hand structure, object interactions, and temporal consistency despite rapid egomotion. The system solves a three-part problem: (1) existing video editors have never seen egocentric footage during training and therefore fail on hand-object interactions and egomotion, (2) even a capable editor is far too slow for interactive AR use (13+ seconds to show the first frame), and (3) there is no standardized way to measure progress on egocentric editing tasks. The solution is a data-model-benchmark triad: EgoEditData provides the first high-quality curated training pairs of egocentric edits, the EgoEdit model uses channel-wise source conditioning and a two-phase distillation pipeline to achieve sub-second first-frame latency, and EgoEditBench provides 15 standardized egocentric editing tasks with VLM-based evaluation.

3.2 Big-Picture Architecture (Diagram in Words)

The system has three interconnected pillars, each feeding into the next (Figure 1):

1. EgoEditData (Section 3) — The Data Engine: A multi-stage curation pipeline that takes raw egocentric videos from Ego4D and EgoExo4D, identifies videos with active hand-object manipulation, segments hands and manipulated objects, generates diverse edited versions of each video (object substitution, removal, addition) using Wan 2.1 VACE with extensive human filtering, and constructs precise natural-language editing instructions for each source-target pair. Output: 99.7k high-quality (source video, target video, instruction) triples where hands are explicitly preserved.

2. EgoEdit Model (Section 4) — The Editing and Distillation Engine: A pretrained text-to-video DiT model is converted into a video editor via channel-wise concatenation of the source video with the noisy target, fine-tuned on EgoEditData plus a corpus of general editing data, then distilled through two phases — bidirectional DMD (compressing 80 NFEs to 4) followed by Self-Forcing autoregressive training (enabling chunk-by-chunk streaming generation). Output: a model that outputs edited video chunks in real time (855ms first-frame latency, 38.1fps throughput) on a single H100 GPU.

3. EgoEditBench (Section 4.4) — The Evaluation Framework: A curated benchmark of 100 diverse egocentric source videos paired with GPT-5-generated instruction prompts across 15 editing tasks, with automated VLM-based scoring for instruction faithfulness, video quality, text alignment, and temporal consistency. Output: standardized metrics that measure progress on egocentric editing challenges.

Information flows as follows: raw egocentric footage → EgoEditData curation pipeline generates training triples → the pretrained T2V model is fine-tuned on these triples plus general editing data → the resulting editor undergoes DMD + Self-Forcing distillation → the streaming model ingests live camera frames and text instructions → produces edited chunks autoregressively → output video frames are decoded and displayed to the user sub-second after recording begins. Evaluation runs in parallel: EgoEditBench source videos are passed through the model with task-specific instructions, and a VLM judge scores the output against the instruction.

3.3 Roadmap for the Deep Dive

• First, the Flow Matching framework (Section 3.4.1) — the mathematical backbone shared by all model variants, since the editing model, DMD teacher, and Self-Forcing student all operate in this velocity-prediction paradigm.
• Second, the base text-to-video architecture (Section 3.4.2) — the pretrained DiT backbone, latent space, autoencoder, and text conditioning that EgoEdit inherits, since understanding the editing adaptation requires knowing what is being adapted.
• Third, the EgoEditData curation pipeline (Section 3.4.3) — the multi-stage data generation process with explicit failure rates and manual filtering percentages at each stage, since data is the paper's central claim about what drives egocentric editing performance.
• Fourth, the editing model adaptation (Section 3.4.4) — how source video conditioning is injected via channel-wise concatenation (vs. sequence-wise alternatives), the training configuration and data mixture, and why channel-wise conditioning is essential for real-time operation.
• Fifth, the distillation pipeline (Section 3.4.5) — the two-phase compression from 80 NFEs to 4 NFEs to autoregressive streaming, including the DMD loss formulation, the Self-Forcing exposure-bias reduction mechanism, and the chunking strategy that enables sub-second latency.
• Sixth, EgoEditBench construction (Section 3.4.6) — the task taxonomy, source video sampling strategy, instruction prompt generation, and the VLM-based evaluation protocol.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and dataset paper whose core idea is that egocentric video editing requires a purpose-built ecosystem — domain-aligned training data, a latency-optimized streaming model, and a domain-specific benchmark — because existing general-purpose editors fail on egocentric footage due to distribution shift, and existing real-time methods sacrifice too much quality for speed.

---

3.4.1 The Flow Matching Framework

EgoEdit is built entirely within the Rectified Flow framework, which is the mathematical formalism used for training and inference across all model variants — the base text-to-video generator, the fine-tuned editor, and all distilled versions. Understanding this shared foundation is essential before examining any architectural specifics.

The Rectified Flow objective learns a deterministic mapping from a noise distribution to a data distribution by training a neural network to predict the constant velocity along linear paths connecting noise samples to data samples.

Let $X_1 \sim p_d$ be a sample from the data distribution $p_d$ (in practice, a latent representation of a video frame, produced by the Wan 2.1 autoencoder). Let $X_0 \sim p_n = \mathcal{N}(0, I)$ be a sample from the standard Gaussian noise distribution. The linear interpolation path between these two points is defined as:

$X_t = (1 - t) X_0 + t X_1$

where $t \in [0, 1]$ is a continuous time parameter. The ground-truth velocity along this path — the instantaneous change in $X_t$ with respect to $t$ — is constant:

$v_t = \frac{dX_t}{dt} = X_1 - X_0$

What this means operationally: at any point $t$ along the interpolation, the noise sample $X_0$ and the data sample $X_1$ determine a vector difference that, if followed at constant speed from $t$to 1, would exactly reconstruct the data sample. The neural network is trained to predict this constant velocity `$v_t$` given the current noised state `$X_t$` and the time `$t`$.

The training objective is a simple mean-squared error between the predicted velocity and the ground-truth difference:

$\mathcal{L}_{RF} = \mathbb{E}_{t \sim p_t, X_1 \sim p_d, X_0 \sim p_n} \left\| G(X_t, t) - (X_1 - X_0) \right\|^2_2$

where $G(\cdot)$ is the neural network (the velocity predictor), $p_t$ is a training distribution over $t$ chosen as a logit-normal distribution, and the expectation is taken over random time samples, data samples, and noise samples in each training batch.

What it computes: the L2 distance between the network's predicted velocity vector and the known constant velocity along the linear interpolation path. The network sees a partially noised latent $X_t$ (a blend of noise and data, with blend ratio determined by $t$) and must output a velocity vector that, if integrated forward from $t$ to 1, would produce $X_1$ exactly.

Why this form: the constant-velocity property of linear paths in Rectified Flow means the target $X_1 - X_0$ is independent of $t$. This differs from standard diffusion formulations where the score or noise target depends on the noise schedule and $t$ in complex ways. The constant target simplifies training dynamics — the network just needs to learn a single flow direction per sample rather than a $t$-varying quantity. Empirically, this framework supports few-step inference via Euler integration, which is critical for the distillation stage where the model is compressed from 40 steps to 4.

At inference time, generation proceeds by integrating the learned velocity field from $X_0$ (pure noise) to $X_1$ (a data sample) using an Euler solver. Starting from $X_0 \sim \mathcal{N}(0, I)$ at $t=0$, the solver takes a discrete step $\Delta t$ forward in time: $X_{t+\Delta t} = X_t + \hat{v} \cdot \Delta t$, where $\hat{v} = G(X_t, t)$ is the predicted velocity. Repeating this for the specified number of steps (40 for the base model, 4 for the distilled model) yields the final sample $X_1$. Classifier-free guidance is applied during inference by training the model to also predict velocities without text conditioning, and at inference time computing $\hat{v}_{cfg} = \hat{v}_{uncond} + w(\hat{v}_{cond} - \hat{v}_{uncond})$ where $w$ is the guidance scale — though this is eventually folded into the distilled model to eliminate the need for separate unconditional passes.

---

3.4.2 The Base Text-to-Video Architecture

EgoEdit does not train a video generator from scratch. It inherits a large, pretrained text-to-video model and adapts it for editing. The architecture follows the standard latent diffusion paradigm with a DiT (Diffusion Transformer) backbone, specifically aligned with the Wan 2.1 family of models.

Latent space and autoencoder. Videos are not operated on in pixel space. Instead, a pretrained Wan 2.1 autoencoder compresses the raw video into a latent representation, reducing dimensionality by factors of 8× in height, 8× in width, and 4× in time. This means a raw video chunk of shape (T, H, W, 3) becomes a latent of shape (T//4, H//8, W//8, C) where $C$ is the latent channel dimension (the paper does not disclose this exact dimension, but typical Wan configurations use 16 channels). The autoencoder is frozen and not updated during any stage of EgoEdit training — all editing, distillation, and inference operate in this compressed latent space. The autoencoder natively supports autoregressive operation (Section 4.3), which is essential for the streaming variant.

Patchification and tokenization. The latent video is processed by a linear patchifier that divides each spatial (H//8, W//8) latent frame into non-overlapping 2×2 patches, further compressing the representation. Each 2×2 spatial patch across all channels is linearly projected into a token embedding of dimension 4096 (the transformer's hidden dimension). The sequence length for a video is therefore (T//4) * (H//16) * (W//16) tokens. For the resolution of 512×384 pixels used throughout the paper, this yields a spatial grid of (512//16) × (384//16) = 32 × 24 = 768 tokens per frame, though the precise count depends on the autoencoder's compression ratio.

Transformer backbone. The token sequence is processed by a DiT model with 32 identical transformer blocks, each containing: (1) self-attention over all tokens in the sequence (spatiotemporal attention — every token attends to every other token across both space and time), (2) cross-attention to text embeddings for conditioning, and (3) a feed-forward MLP. The hidden dimension is 4096 with 32 attention heads. Modulation parameters (scale and shift) for timestep conditioning are injected into each block using the approach from PixArt-Sigma, where the time embedding $t$ is processed by a small MLP to produce per-block modulation values that scale and shift the normalization layers. QK normalization (normalizing query and key vectors before computing attention scores) and Flash Attention are used in every attention operation for stability and memory efficiency.

Text conditioning. Two text encoders process the natural language instruction (or caption, for the base T2V model): a T5 encoder (the paper does not specify the variant, but T5-XL is common in Wan architectures) and a CLIP text encoder. Their outputs are concatenated to form the text conditioning tokens. These tokens are injected into the transformer via cross-attention layers placed after each self-attention block — the query comes from the video tokens, and the keys and values come from the text tokens.

Inference for the base model. Generating a 5-second, 512×384, 16fps video requires 40 denoising steps with classifier-free guidance. Since each step requires two model evaluations (one with text conditioning, one without for guidance), the total is 80 NFEs (Number of Function Evaluations). At 86ms per forward pass on a single H100 GPU (reported in Section 4.3 latency analysis), this yields ~6.85 seconds of pure model time, plus autoencoder encoding/decoding time, producing a throughput of approximately 11.9fps before autoencoder and 9.68fps after — below the 16fps needed for real-time display.

---

3.4.3 The EgoEditData Curation Pipeline

EgoEditData is not a passively collected dataset. It is the result of a seven-stage pipeline that aggressively filters raw egocentric footage, extracts hand and object masks, generates synthetic edited versions with heavy computation, and subjects every output to human review. The pipeline is designed to produce training pairs that specifically target the failure modes of existing editors: hand-object interactions, rapid egomotion, and active object manipulation.

Stage 1: Video Selection.

The paper draws from two egocentric video datasets: Ego4D (3,000 hours of egocentric footage) and EgoExo4D (paired ego-exo recordings of skilled human activities). Not all footage is suitable — the paper applies hardware and quality filters:

• Camera model filtering (Ego4D only): Only videos from high-quality camera models are retained: GoPro Hero 4, Hero Black 7/8/9, Hero Silver 7, and GoPro Max. Lower-quality cameras are discarded because their footage would introduce artifacts that the video generation model might learn to reproduce.
• Monocular constraint: Binocular (stereo) footage is excluded. EgoEdit targets monocular AR headsets.
• Jitter and aesthetic scoring: Videos with excessive camera shake or low visual quality are filtered out. The paper does not specify the exact thresholds, but the combined effect is severe: only 1.8% of the original videos survive this stage. This 98.2% rejection rate at the first stage signals the extreme selectivity of the entire pipeline.

Stage 2: Hand Mask Segmentation.

The pipeline must identify where hands appear in each frame to ensure they are preserved during editing. The procedure:

1. Hand detection: WiLoR, a 3D hand localization and reconstruction model, processes each frame independently with a confidence threshold of 0.75. Videos where no hands are detected in any frame are removed entirely — if there are no hands, there is no hand-object interaction to learn from.
2. Seeding SAM 2: From the detected frames, the three with the highest WiLoR confidence scores are selected. Their detected hand masks are converted into point prompts (positive points inside the hand, likely negative points outside) for SAM 2, the video-capable segmentation model.
3. Dense mask propagation: SAM 2 propagates these point prompts across all frames of the video, producing fine-grained, temporally consistent hand masks. SAM 2's memory-based architecture ensures the masks remain coherent even when the hand temporarily leaves the frame or is heavily occluded.
4. Human filtering: A human annotator reviews the masks. Only 49.6% of videos pass this review — the remaining 50.4% are rejected due to inaccurate hand mask boundaries, temporal inconsistencies that SAM 2 failed to resolve, or cases where hand detection was unreliable (e.g., gloves, poor lighting, extreme motion blur).

The output of this stage is a set of videos with validated, frame-accurate hand masks. These masks serve two purposes: they will be excluded from edit regions to prevent the object editing stage from modifying hands, and they will be used to filter object masks (Stage 4) to ensure objects are genuinely being manipulated.

Stage 3: Object Name Extraction.

The pipeline now identifies what the hands are manipulating. Qwen2.5-VL-32B, a 32-billion-parameter vision-language model, is prompted with the video subsampled to 2fps (converted to image frames) and asked: "What is held by the hand in the video?" The model outputs a short noun phrase describing the interacted object — e.g., "a banana," "a screwdriver," "a white shaker bottle." Videos where the VLM determines no meaningful hand-object interaction exists are discarded.

This stage is crucial because it narrows the edit target. Rather than editing arbitrary objects in the scene, the pipeline focuses specifically on actively manipulated objects — the objects whose replacement will produce the most interesting and challenging training examples for the edit model.

Stage 4: Object Mask Segmentation.

Given the object name from Stage 3, the pipeline extracts fine-grained masks for that object across all frames:

1. Grounded SAM initialization: For each frame independently, Grounded SAM is prompted with the object name. It produces an approximate object mask using a confidence threshold of 0.4 for the mask and 0.35 for the text-object alignment. Videos where no frame exceeds these confidence thresholds are discarded.
2. Hand proximity filtering: The raw Grounded SAM masks are noisy — they might include background pixels, capture the wrong object, or segment the hand along with the object. Two spatial filters are applied: (a) the edge distance between the hand mask (Stage 2) and the object mask — if the object mask edge is far from the hand mask edge, the segmentation likely captured the wrong object; (b) the distance between the object mask centroid and the hand skeleton keypoints from WiLoR — if the object is far from the hand, it is not being manipulated. Masks failing these proximity checks are filtered out. If multiple objects are detected in a single frame, only the one closest to the hand mask is kept.
3. SAM 2 refinement: As with hands, the top three frames with highest Grounded SAM confidence are used to generate point prompts for SAM 2. Critically, regions corresponding to the hand mask are excluded from the prompt generation — this ensures SAM 2 segments the object but not the hand holding it. SAM 2 produces dense, temporally consistent object masks across the video.
4. Human filtering: Manual review ensures object masks are correctly extracted. Only 43.6% of sequences survive this stage.

At this point, the pipeline has identified videos with confirmed hand-object interaction, extracted validated hand masks, and extracted validated object masks for the actively manipulated object. These masks will drive the next stage.

Stage 5: Object Editing (the computationally heavy stage).

This stage creates the "after" videos — synthetic edited versions where the manipulated object is replaced or removed. The procedure is:

1. Target object proposal: GPT-5 Mini is prompted to propose diverse target objects for substitution, including both ordinary objects (e.g., "a water gun," "a feather quill," "an insulated metal thermos") and imaginary/fantastical objects (e.g., "a vial of phosphorescent moss and spores," "an ornate silver goblet glowing electric blue from within"). This diversity sampling ensures the editing model sees a wide range of substitutions during training.
2. Reference image synthesis: For each proposed target object, Qwen-Image synthesizes a reference image — a standalone image of the target object that will serve as visual conditioning for the video generation model. This reference image provides appearance guidance that text alone cannot capture (specific textures, shapes, material properties).
3. Scene-level description: GPT-5 Mini produces a detailed scene description assuming the original interaction but with the target object. For example, if the source video shows a hand squeezing a sponge at a sink, and the target object is "a leather pouch with moss overgrowth," the description might read: "The egocentric subject squeezes a leather pouch covered in phosphorescent moss at a kitchen sink, water dripping from the pouch onto the counter." This description guides Wan 2.1 VACE in generating the edited video.
4. Mask preparation: The object masks are converted to rectangular bounding boxes with an additional margin (to account for the object's motion extent), then Gaussian dilated with a 50×50 pixel kernel (to create soft boundaries that blend naturally). The hand mask regions are excluded from these final masks — this is the critical step that literally tells the generation model "do not modify the hands." The final masks are what Wan 2.1 VACE receives as the inpainting/editing region.
5. Video generation: Wan 2.1 VACE 14B receives the source video, the reference image of the target object, the scene-level description, and the object mask (with hands excluded). It generates the edited video where the originally manipulated object is replaced by the target object while the hands, their motion, and the rest of the scene are reconstructed faithfully. Object removal is treated as a special case where no target object is specified — the mask region is simply inpainted with background.

The computation is extremely heavy: 0.112 fps on 8 H100 GPUs — meaning generating a single 5-second, 16fps edited video (80 frames) takes approximately 714 seconds (nearly 12 minutes) on 8 GPUs. Despite this heavy compute and rich conditioning (reference image, scene description, precise masks), Wan 2.1 VACE produces acceptable results only a fraction of the time:

"Wan 2.1 VACE 14B yields only a small fraction of results that meet our standard for dataset quality."

Human filtering removes 62.2% of generated edits, retaining only 37.8%. Human annotators reject edits where: the target object is not convincingly integrated into the scene, the hand is distorted or partially replaced, temporal consistency breaks (flickering, morphing), lighting mismatches between the inserted object and the scene, or the edit does not faithfully follow the instruction. This 37.8% acceptance rate, combined with the earlier filtering stages, means only $0.018 \times 0.496 \times 0.436 \times 0.378 \approx 0.4\%$ of the original videos from Ego4D and EgoExo4D contribute to the final dataset — a deliberate choice that prioritizes quality over quantity.

Stage 6: Editing Pairs Construction.

For each source video and its surviving edited variants, all permutations are considered as editing pairs — including the identity pair where source equals target (the original clip paired with itself). While the paper does not elaborate on why identity pairs are included, this is a common practice in editing dataset construction to teach the model when not to edit — preventing it from hallucinating changes when the instruction requests no modification.

For each (source, target) pair, GPT-5 Mini is prompted to generate a precise, faithful description of the edit. The prompts are verbose and specific, averaging 378 characters (Figure 7c). Example instructions from Figure 2 include:

"Transform the bowl [...] into an oval woven wicker basket containing a colorful assortment of fruit [...]"

"Metamorphose the low-profile tire with scuffed sidewalls into an iron lantern with a dark, weathered metal frame and decorative filigree panels [...]"

The short-prompts tail in Figure 7c corresponds to tasks requiring minimal context, such as "Remove an {object}." or "Add an {object}," where the object name is sufficient. The paper reports (Appendix F, "Failed Experiments") that initial experiments with simple template-based instructions ("Replace {source} with {target}") produced poor editing performance due to limited variety, short object names, and occasional mismatches between generated videos and their nominal target descriptions. Switching to GPT-5-generated verbose instructions substantially improved instruction following.

Stage 7: Dataset Consolidation.

The final EgoEditData contains:

• 10,909 original videos (from Ego4D and EgoExo4D)
• 38,808 synthetic (edited) videos — an average of 3.6 synthetic variants per original video
• 99,659 total editing pairs — each a (source video, target video, instruction) triple
• 93,422 pairs from Ego4D, 6,237 from EgoExo4D
• Total duration: 70 hours of video

Task breakdown: The majority are Change Object tasks (54,164 pairs for ordinary object replacement, 39,465 pairs for object replacement with special effects like fire or frost), plus 3,651 Add Object pairs and 2,379 Remove Object pairs. The dataset contains 13,632 distinct target objects vs. 3,199 unique source objects (Figure 7), confirming the diversity introduced by GPT-5's target object proposals.

Why this extreme filtering? The paper's core claim is that edit quality tracks data quality, not just data quantity. The 99.7k pairs at 0.4% yield represent an experiment in whether aggressive human curation yields better editing models than automated collection at 100× the scale. The ablation results (Table 3) showing a monotonic improvement from 0% to 100% of EgoEditData suggest that even within this heavily filtered set, more data helps — but the base quality of the data matters critically. A model trained on lightly-filtered data failed (Appendix F, "Failed Experiments"), often reproducing artifacts from low-quality pairs or failing to edit at all.

---

3.4.4 The Editing Model Adaptation

Starting from the pretrained text-to-video DiT model described in Section 3.4.2, the paper adapts it for video editing by introducing source video conditioning and fine-tuning on the editing data corpora. The key architectural decision is how to inject the source video — a choice that directly determines whether the model can eventually run in real-time.

The architecture. Given a noise level $t$, the model processes the noisy version of the target video $X^{tgt}_t$ and must condition on the clean source video $X^{src}$ and the text instruction $c$. The model is expressed as:

$\hat{v} = G(X^{tgt}_t \mid X^{src}; c)$

Two conditioning strategies are considered:

1. Sequence-wise concatenation (not used): The source video is patchified identically to the target, and the source tokens are concatenated with the target tokens along the sequence dimension. For a video with $N$ tokens, this yields a sequence of length $2N$ tokens. The self-attention cost grows quadratically — from $O(N^2)$ to $O(4N^2)$ — because every token must attend to every other token. This approach is common in prior work (EditVerse, UNIC, Qwen-Image) and provides strong source-target interaction since attention can directly compare source and target features at every layer.

2. Channel-wise concatenation (used by EgoEdit): Before patchification, the source video $X^{src}$ and noisy target video $X^{tgt}_t$ are concatenated along the channel dimension. If each latent frame has $C$ channels, the concatenated representation has $2C$ channels. The patchifier's linear projection then maps from $2C \times (2 \times 2)$ dimensions to 4096, naturally fusing source and target information without increasing the token count. The self-attention cost remains $O(N^2)$ — identical to the base T2V model.

Why channel-wise concatenation matters: the paper states:

"This approach is common [sequence-wise], but the longer token sequence increases the cost of self attention quadratically, which conflicts with real-time low latency operation."

Specifically, sequence-wise concatenation would double the sequence length, quadrupling self-attention FLOPs and increasing per-step latency from ~86ms to something approaching ~344ms (assuming naive quadratic scaling). This would make the 4-NFE distilled model take ~1.4 seconds per forward pass instead of ~343ms, pushing total latency far above the sub-second target required for AR. Channel-wise concatenation keeps the per-step cost nearly identical to the base model — the only overhead is the slightly larger linear projection at the first layer, which is negligible compared to the attention savings.

Training configuration. The model is fine-tuned for 30,000 iterations with a total batch size of 96 videos using 48 H100 GPUs. The optimizer is AdamW with learning rate $1 \times 10^{-5}$, linear warmup over 1000 steps, weight decay 0.1, beta values 0.99, and exponential moving average (EMA) with decay 0.9999. The EMA is standard practice for DiT training — the averaged weights at inference time produce more stable outputs than the raw training weights.

Training data mixture. The model is not trained on EgoEditData alone. The full training corpus includes:

• EgoEditData: 99.7k video editing pairs (28% sampling weight)
• Other video editing datasets: 1.31M pairs from publicly available and internally generated corpora (52% sampling weight)
  • Public: GPT-Image-Edit-1.5M, ShareGPT-4o-Image, Complex-Edit, HQEdit, OmniEdit, UltraEdit
  • Internal: 210k pairs from a pipeline using Wan VACE, Wan-Animate, and MiniMax-Remover for object addition/removal, substitution, and human edits
  • Synthetic: 1.1M pairs created by pairing natural videos with depth/pose/edge/optical flow estimation (structural conditioning tasks)
• Image editing datasets: 3.5M pairs (20% sampling weight)
  • Generated by Qwen-Image-Edit (2M pairs) plus various public image editing datasets

Importance sampling rationale: The 28%/52%/20% weights are not proportional to dataset sizes — EgoEditData is oversampled relative to its size (99.7k out of ~5M total pairs would be ~2% under proportional sampling). The paper makes this choice because EgoEditData contains the highest-quality, most domain-relevant examples. Training on imbalanced data without importance sampling would cause the model to underfit the egocentric distribution. The internal generation pipeline for video editing data uses filtering (automated and manual) to maintain quality, but the paper explicitly notes that unfiltered data produced "weak video editing performance, with the model often reproducing artifacts" (Appendix F), motivating the heavy curation of both EgoEditData and the supplementary corpora.

Resolution and framerate. The model operates at 512×384 pixels and 16 fps, producing 5-second videos (80 frames). This resolution is "slightly lower than the common 480p resolution" (Section 6, Limitations), representing a deliberate tradeoff: higher resolution would increase latency quadratically (due to the transformer's self-attention over spatial tokens), and maintaining 16fps at higher resolution would require more FLOPs per frame, threatening real-time operation.

---

3.4.5 The Distillation Pipeline

The fine-tuned editing model produces high-quality edits but is fundamentally too slow for interactive use. The distillation pipeline compresses the model through two phases, each addressing a different aspect of the latency problem.

Phase 0: The starting point — 80 NFE bidirectional model.

The post-fine-tuning editor performs inference at 5 seconds, 512×384 pixels, 16 fps (81 frames total) using 40 denoising steps with classifier-free guidance. This requires 80 model evaluations (40 steps × 2 passes for guidance): one forward pass with text conditioning, one without, and a guidance-weighted combination of the predicted velocities. Each forward pass on an H100 takes 86ms, giving 6.85 seconds of model time. The autoencoder adds 1.52 seconds for encoding the source video and decoding the generated target. Recording the source video (81 frames at 16 fps) takes 5.06 seconds. Total first-frame latency: 13.43 seconds (Table 2). Throughput including autoencoder: 9.68 fps — below the 16fps target.

The model also generates the entire video at once before displaying any frame. This "batch" generation paradigm means the user waits 13+ seconds with no visual feedback, which is unacceptable for interactive AR.

Phase 1: Bidirectional DMD distillation (80 NFEs → 4 NFEs).

Distribution Matching Distillation (DMD) is a technique for compressing a multi-step diffusion model into a few-step student model by minimizing a distribution-level loss rather than a per-step L2 loss. The paper adapts DMD for the Rectified Flow framework.

The key idea: instead of having the student mimic the teacher's velocity predictions at each timestep (which would require matching the teacher's step schedule), DMD trains the student to produce samples that are indistinguishable from the teacher's samples under a learned discriminator. Specifically:

• The student model $G_S$ generates a sample $\hat{X}_1$ from noise in a small number of steps (4 in this case) using Euler integration: starting from $X_0 \sim \mathcal{N}(0, I)$, the student predicts velocities and integrates forward over 4 equal-spaced timesteps.
• A "fake" score model (a discriminator adapted for the flow matching setting) is trained to distinguish student-generated samples from teacher-generated samples. The student is trained adversarially to fool this discriminator, with an additional regression loss toward the teacher's output for the same noise input.
• Critically, the student is also trained to match the teacher's deterministic mapping: for a given noise sample, the teacher produces a deterministic output (since Rectified Flow is deterministic). The student should produce the same output given the same noise, which provides a strong signal beyond the adversarial loss.

The DMD training proceeds for 4,500 iterations with a batch size of 64 on 32 H100 GPUs. The optimizer is AdamW with learning rate $1 \times 10^{-6}$ for the generator and $4 \times 10^{-7}$ for the fake score model, 5 generator updates per discriminator update, weight decay 0.1, beta 0.99, and EMA 0.99. The lower learning rates compared to the fine-tuning stage reflect the fact that distillation is a refinement process — the student starts from the fine-tuned teacher checkpoint (or a slightly modified version) and should not deviate drastically.

The result: a 4-NFE model that produces 5-second videos at comparable quality to the 80-NFE teacher. Throughput jumps to 43.5 fps (including autoencoder), well above the 16fps real-time threshold. However, total first-frame latency is still 6.93 seconds (Table 2): the model still generates the entire 81-frame video at once, and the autoencoder must process the full source and target videos before any frame can be displayed. The throughput is sufficient for streaming, but the first-frame latency is not.

Phase 2: Self-Forcing autoregressive distillation (bidirectional 4-NFE → autoregressive 4-NFE).

Self-Forcing addresses the remaining barrier: the "all-at-once" generation paradigm. The distilled DMD model is converted into an autoregressive model that generates video in chunks — small groups of frames that can be displayed immediately after generation, without waiting for the full video.

The Self-Forcing procedure operates as follows:

1. Chunk-wise generation setup: The video is divided into chunks. The paper uses a configuration where each chunk consists of 3 latent frames. Since the Wan autoencoder compresses by 4× in time, 3 latent frames correspond to 12 RGB frames. The model generates the first chunk (3 latent frames = 12 RGB frames), then conditions on it to generate the second chunk, and so on. With 7 chunks, the total is 21 latent frames = 84 RGB frames (slightly more than the 81 frames of the original 5-second clip).

2. Causal architecture: The self-attention in the transformer is modified from bidirectional (all tokens attend to all tokens) to causal (tokens in chunk $i$ can only attend to tokens in chunks $j \leq i$). This prevents the model from "cheating" by looking at future frames during autoregressive generation. KV-caching is used: the keys and values from previously generated chunks are stored and reused, so the model only processes the new chunk at each step, keeping per-chunk cost constant rather than growing with sequence length.

3. Training with exposure bias reduction: The critical innovation of Self-Forcing is that during training, the student model is rolled out autoregressively on the training data, and the DMD loss is applied at each chunk. Specifically:

   • The student generates the first chunk given noise and source conditioning.
   • The student then generates the second chunk conditioned on its own generated first chunk (not the ground-truth first chunk).
   • The DMD loss compares each generated chunk against the teacher's output for that chunk.

   This "self-forcing" procedure — where the student sees its own possibly erroneous previous outputs during training — mimics the inference-time distribution, where errors in early chunks can cascade. By training the student to recover from its own mistakes, Self-Forcing reduces the exposure bias that plagues standard teacher-forcing distillation (where the student always sees ground-truth previous chunks during training but its own generated chunks at inference).

4. Initialization from DMD checkpoint: Self-Forcing training starts from the DMD-distilled checkpoint (the 4-NFE bidirectional model). The paper skips a standard ODE initialization phase because "the model can quickly adapt to autoregressive modeling when initialized from the DMD checkpoint" — the DMD model already produces high-quality outputs in 4 steps; switching to autoregressive chunking is a relatively small architectural change.

5. Long-video training trick: During training, to simulate generating videos longer than the training clips, the model is trained with a masking strategy: when generating the last chunk, the first chunk of latent frames is masked (set to zero or noise), simulating the scenario where early context is no longer available due to finite KV-cache capacity. During inference, a window size of 5 chunks is used as condition — the model only attends to the most recent 5 chunks (60 RGB frames) when generating each new chunk, enabling theoretically infinite-length video generation with constant memory.

The Self-Forcing training runs for 4,500 iterations with a batch size of 64 on 64 H100 GPUs. The optimizer is AdamW with learning rate $1 \times 10^{-6}$ for the generator and $4 \times 10^{-7}$ for the fake score model, 10 generator updates per discriminator update, weight decay 0.1, beta 0.99, and EMA 0.99.

The latency breakthrough (Table 2 analysis):

The latency breakdown for the Self-Forcing variant is:

• Recording the first chunk: 562 ms (9 RGB frames at 16 fps = 0.562 seconds to capture)
• Autoencoder encoding/decoding: 217 ms (encoding the 9 source frames, decoding the 12 generated frames)
• Model forward pass: 75.7 ms (one forward pass for the 4-NFE model on the first chunk of 3 latent frames)

Total: 855 ms. This sub-second latency enables interactive AR: the user points their camera, speaks an instruction, and sees the edited result within a second.

The throughput is 38.1 fps (including autoencoder), which means the model generates edited frames faster than the source video is captured (16 fps). This overspeed is necessary because the model must occasionally "catch up" after processing a chunk — the 75.7ms model time per chunk is amortized over the 12 frames that each chunk produces.

A deeper look at the chunking arithmetic: The paper's configuration uses 3 latent frames per chunk, with the Wan autoencoder's 4× temporal compression meaning 3 latent frames = 12 RGB frames. The first chunk of 3 latent frames decodes to 12 RGB frames, displayed in 750ms (12 frames ÷ 16fps). During this display period, the model can generate the next chunk (75.7ms), autoencoder-decode it (~217ms proportionally per chunk, though in practice decoding is pipelined), and have it ready before the first chunk finishes playing. This "generate-ahead" pipelining is what enables smooth streaming at 16fps display rate even though per-chunk processing is measurable.

Remaining limitations (Section 6): Despite the dramatic latency reduction, the paper notes three residual issues: (1) a qualitative gap between the real-time variant and the bidirectional teacher on out-of-distribution editing instructions and during object occlusions, (2) lower temporal consistency (manifesting as shifts at chunk boundaries), and (3) the 855ms first-frame latency, while sufficient, is "suboptimal for interactive usage" — further reduction could come from smaller chunk sizes during Self-Forcing training (sacrificing some quality for even lower latency).

---

3.4.6 EgoEditBench Construction

The benchmark is designed to evaluate egocentric editing performance across 15 diverse tasks, following the evaluation protocol established by EditVerseBench but adapted to the egocentric domain. The construction involves source video selection, task-specific instruction generation, and a VLM-based evaluation pipeline.

Source video selection. From a held-out split of Ego4D (disjoint from the EgoEditData training set), 100 unique source videos are selected to maximize diversity. The procedure:

1. Extract source-object names following the EgoEditData pipeline (Stages 2–3: hand detection, object name extraction).
2. For each source video, produce a scene description (likely from the video's metadata or through VLM captioning — the paper is not explicit but references "scene description" as an input to GPT-5 for prompt generation).
3. Concatenate each source object name with its scene description into a single text string.
4. Compute BERT embeddings for all concatenated strings.
5. Perform K-means clustering with 10 centroids on the BERT embeddings.
6. From each of the 10 clusters, randomly sample 10 source videos, yielding 100 diverse videos.

This clustering approach ensures that the benchmark covers a range of object types and scene contexts rather than over-representing common egocentric scenarios like "hand holding a kitchen utensil in a kitchen."

Task taxonomy. The 15 tasks span three categories:

Object-level editing (direct AR tasks):

• Remove Object: Evaluate the model's ability to erase a specified object while keeping the rest of the video consistent. 50 source videos with their EgoEditData-style instructions are used.
• Add Object: Evaluate insertion of a specified object into the scene. 50 source videos where the target object was synthetically removed following the EgoEditData pipeline are used.
• Change Object: Evaluate object attribute modification or replacement. For each of the 100 source videos, 4 instruction prompts are generated: 2 for pure replacement (object A → object B) and 2 for replacement with an added effect (e.g., "replace the mug with a flower vase that glows with blue fire"). This produces 400 Change Object test cases.

Scene-level editing:

• Change Background: Replace or edit the background while preserving foreground identity and motion. GPT-5 Mini is given source frames and the video caption and produces an instruction with a semantically compatible target background.
• Change Camera Pose: Assess recomposition via specified camera trajectories (pan, tilt, dolly, zoom) without altering scene events. GPT-5 Mini uses the source video's camera pose description (from the caption) to produce the instruction.
• Add Effect: Evaluate global post-processing effects (motion blur, VHS, glow, film grain). GPT-5 Mini is primed with in-context examples from EditVerseBench and asked to propose a diverse pool of effect instructions; one is randomly sampled per video.
• Stylization: Similar to Add Effect, but for style transfer (e.g., "make the scene look like a watercolor painting"). The procedure mirrors Add Effect with EditVerseBench examples.
• Reasoning: Evaluate edits requiring spatial/temporal inference. GPT-5 Mini produces an instruction tied to an explicit anchor (event or timestamp) or disambiguating relations ("the object to the left of the sink," "before the hand reaches for the drawer") — the correct target object must be inferred from context rather than explicitly named.

Cross-modal conditioning (structural consistency tests):

• Depth-to-Video: Convert a depth map (generated from the source video using Depth Anything) back into a realistic video. Instruction: "Turn the depth map into a video with the following description: {caption}."
• Sketch-to-Video: Convert a Canny edge map into a video. Instruction: "Turn the canny edge map into a video with the following description: {caption}."
• Pose-to-Video: Convert 2D human pose maps (from DWpose) into a video. Instruction: "Turn the DWpose pose map into a video with the following description: {caption}."
• Video-to-Depth: The inverse task: convert a video into a depth map. Fixed instruction: "Turn the video into a depth map."
• Video-to-Sketch: Convert a video into a Canny edge map. Fixed instruction: "Turn the video into a Canny edge map."
• Video-to-Pose: Convert a video into a DWpose pose map. Fixed instruction: "Turn the video into a DWpose pose map."
• Combined (Multi-Task): Compose multiple editing prompts from the same source video (e.g., Pose-to-Video + Change Background + Stylization) by sampling a subset of instructions and having GPT-5 Mini compose them into a single combined instruction.

Total benchmark size: For the object-level tasks, the counts are as specified (50 + 50 + 400 = 500). For the scene-level and cross-modal tasks, each of the 100 source videos receives one instruction per applicable task, yielding approximately 100 per task. Including the Combined task, the total is 1,700 source videos paired with instruction prompts.

Evaluation protocol. The paper closely follows the EditVerseBench evaluation methodology:

• VLM judge (primary metric): A vision-language model evaluates each edited output against the instruction, producing a numeric score that reflects instruction faithfulness and edit quality. The paper uses exactly the same evaluation prompts as EditVerseBench. The VLM score shows high alignment with human judgment: in a human study on 30 randomly sampled examples per task category, VLM preferences matched human preferences in 86.2% of cases against LucyEdit and 84.9% against InsV2V (Table 4). This high agreement validates VLM evaluation as the primary quality signal.

• PickScore: A learned perceptual quality metric that evaluates overall video quality independent of the instruction.

• Text Alignment (video-level): CLIP-based similarity between the instruction text and the generated video, measuring whether the video's content matches what was requested. The paper reports only the video-level score because frame-level and video-level text alignment are "highly correlated in our setting."

• Temporal Consistency: CLIP-based consistency metric measuring frame-to-frame coherence. The paper notes that CLIP-based and DINO-based consistency metrics are "strongly correlated," so only the CLIP variant is reported.

Results are averaged per task to ensure equal weighting across all 15 tasks, preventing high-count tasks like Change Object (400 samples) from dominating the aggregate score.

Why this evaluation design? The 15-task taxonomy tests complementary aspects of egocentric editing. Object-level tasks measure the core AR functionality — can the model add, remove, and modify scene objects while preserving hands and interactions? Scene-level tasks measure broader environmental understanding — can the model reason about backgrounds, camera motion, and global effects without corrupting foreground content? Cross-modal tasks push structural consistency — can the model maintain scene geometry when conditioned on edge maps or depth, or faithfully extract geometry from video? The Combined task stresses compositional generalization — can the model handle multiple simultaneous edits?

The evaluation protocol's reliance on automated metrics (VLM judge, PickScore, CLIP consistency) follows the EditVerse approach and is motivated by the impracticality of human evaluation at scale (1,700 samples × multiple baselines = thousands of evaluations). The human alignment study (Table 4) provides a calibration check that the automated metrics are reliable proxies for human judgment in this domain.

4. Key Insights and Innovations

Innovation 1: Domain-Aligned Data — Not Model Scale — Is the Bottleneck for Egocentric Editing

The paper's central intellectual contribution is not the dataset itself but the diagnostic finding that the egocentric editing failure mode is a data distribution gap, not a model capacity gap. This reframes how the field should think about domain-specific video editing.

Prior to this work, the dominant assumption — implicit in the EditVerse, Lucy Edit, and InsV2V line of research — was that scaling up model capacity and training on ever-larger general video editing corpora would eventually subsume egocentric editing as a special case. The reasoning: if a model sees enough diverse editing examples, it should generalize to first-person footage. EditVerse's curation of 232K video-edit samples from multiple sources represents this "scale and diversity solves distribution shift" philosophy.

EgoEdit challenges this assumption with a clean ablation result (Table 3, Appendix Figure 13). When a model is trained on the full editing corpora minus EgoEditData (0% egocentric data), it achieves a VLM score of only 4.87 on EgoEditBench — comparable to the worst-performing baselines. Adding just 25% of EgoEditData raises the score to 7.12 (a 2.25-point jump), and going to 100% reaches 7.85. This monotonic improvement with egocentric data fraction, while all other training data stays constant, demonstrates that the general editing corpora — despite containing 1.31M video and 3.5M image editing pairs — provide essentially zero transfer to the egocentric domain. The model has the architectural capacity to perform egocentric edits (the DiT backbone with channel-wise conditioning is expressive enough); it simply has never seen the right training distribution.

This is a fundamental insight, not an incremental observation, because it inverts the burden of proof. Rather than asking "how do we build a better model for egocentric editing," the paper argues that the correct question is "how do we build better egocentric editing data." The model architecture, distillation pipeline, and training recipe are largely off-the-shelf components; the innovation is recognizing that the data, not the model, is the primary lever. This is a specific, empirically validated instance of the broader data-centric AI thesis, and it provides a template for attacking other domain-specific editing problems (medical video, sports footage, underwater scenes) where the failure mode is likely similar.

Evidence for this interpretation comes from a second, subtler finding: adding EgoEditData also improves performance on the exocentric EditVerseBench (Table 7, Appendix E.4). As the retained fraction of EgoEditData increases from 0% to 100%, EditVerseBench VLM scores rise from 6.89 to 7.79, with particular gains in Reasoning (6.43 → 8.37), Remove (4.30 → 7.00), and Camera Movement (6.27 → 7.60). The paper speculates this is because EgoEditData's strict quality filtering and descriptive captions transfer benefits beyond the egocentric case. If true, this suggests that the dataset's value comes not just from domain alignment but from a data quality premium — the extreme manual curation (0.4% yield) produces training examples that are useful even for tasks they were not designed for, a counter-narrative to the "more data, weakly filtered" philosophy that dominates large-scale dataset construction.

---

Innovation 2: Decomposing Real-Time AR Editing into a Two-Phase Distillation Ladder with Distinct Roles

The distillation pipeline is not just an engineering optimization — it embodies a conceptual decomposition of the real-time AR editing problem into two separable sub-problems: (1) compressing the generative quality of a slow model into a fast model, and (2) converting a batch-generation model into a streaming-autoregressive model. Each phase addresses a different axis of the latency-quality tradeoff, and the paper's finding is that these axes are largely independent — you can solve them sequentially without catastrophic interference.

The field's prior approach to real-time generation can be characterized by two extremes. At one end, methods like StreamDiffusion and StreamDiffusionV2 prioritize speed above all else, using lightweight pipelines (image-to-image with few denoising steps, attention manipulation) that achieve real-time throughput but produce EgoEditBench VLM scores of 4.31 and 2.55 — representing a qualitative collapse on egocentric tasks. At the other end, methods like the bidirectional DMD-distilled model achieve high throughput (43.5 fps) but remain batch-generators — the entire video must be generated before any frame is displayed, resulting in 6.93 seconds of first-frame latency (Table 2), which is incompatible with interactive AR regardless of throughput.

The paper's key move is recognizing that these are orthogonal constraints: step count determines throughput, while autoregressive chunking determines first-frame latency. The two-phase distillation ladder solves them in sequence — DMD handles the step-count problem without concerning itself with autoregressive behavior, and Self-Forcing handles the autoregressive adaptation without needing to re-optimize the 4-step quality. The fact that Self-Forcing training can start directly from the DMD checkpoint (skipping the ODE initialization phase that Self-Forcing's original formulation prescribes) is evidence for this orthogonality — the DMD model's learned velocity field transfers cleanly to causal attention with minimal architectural change.

This is a significant conceptual contribution because it provides a general recipe for converting any batch video diffusion model into a real-time streaming editor. The phases are modular: a better distillation technique could replace DMD (e.g., adversarial post-training like APT2), and a different autoregressive scheme could replace Self-Forcing, but the two-phase decomposition remains valid. The paper's ablation (Table 2) shows that each phase individually is insufficient — DMD alone gives throughput but not interactivity, and naive autoregressive training without DMD would inherit the 80-NFE cost per chunk — and that the combined pipeline retains 99.4% of the teacher's VLM score (7.71 vs. 7.76) while reducing latency by 15.7×. This near-perfect quality preservation across a 20× NFE compression challenges the assumption that real-time and high-quality are fundamentally opposed, and instead reframes the problem as one of appropriate distillation architecture.

Evidence for the independence claim comes from the per-metric breakdown (Table 6, Appendix E.3): EgoEdit-DMD (bidirectional 4-NFE) scores 7.42 VLM, 18.95 PickScore, 16.52 Text Alignment, and 96.87 Temporal Consistency; EgoEdit-RT (autoregressive 4-NFE) scores 7.83 VLM, 19.04 PickScore, 16.49 Text Alignment, and 96.49 Temporal Consistency. The VLM score actually increases slightly after Self-Forcing (7.42 → 7.83), while temporal consistency decreases marginally (96.87 → 96.49). This pattern — quality metrics improving or staying flat while consistency drops slightly — is consistent with the Self-Forcing exposure-bias reduction mechanism improving per-frame quality at the cost of subtle chunk-boundary artifacts, not with a wholesale quality degradation from autoregressive adaptation.

---

Innovation 3: The Ecosystem Triad as a Necessary Condition for Domain Transfer in Generative Models

This paper's meta-contribution is the argument — demonstrated empirically rather than merely asserted — that domain transfer in generative editing requires simultaneous intervention at the data, model, and evaluation levels, and that omitting any one pillar causes the entire system to fail. This is a methodological thesis about how to approach domain-specific generative AI problems, not just a set of technical contributions.

The evidence for mutual dependence comes from three elimination experiments, each implicitly present in the paper's results:

Eliminate the data pillar (keep general model + exocentric benchmark): Train EgoEdit's architecture on standard editing corpora without EgoEditData. Result: VLM score of 4.87 on EgoEditBench (Table 3, 0% row). The model has the right architecture and real-time capability but has never seen hand-object interactions, rapid egomotion, or active manipulation during training. It fails qualitatively — the Appendices describe these failures as "inability to produce any change over the source video, or modifying the input beyond what is requested" (Appendix E.2). This is the state of all prior trained editors on egocentric footage.

Eliminate the model pillar (keep EgoEditData + EgoEditBench, but use a slow bidirectional model without distillation): The fine-tuned editor before distillation achieves the highest VLM score (7.76) but has 13.4 seconds of first-frame latency (Table 2). For AR applications, this is functionally equivalent to failure — a user will not wait 13 seconds for an interactive edit. The quality is there, but the deployment constraint is violated.

Eliminate the evaluation pillar (keep data + model, evaluate on EditVerseBench only): If the paper had evaluated only on EditVerseBench, the story would be "EgoEdit achieves 8.00 VLM, slightly below EditVerse's 8.26 but competitive — a solid general-purpose editor." This would completely miss the point. The value of EgoEdit is its performance on egocentric tasks specifically, where it achieves a 30% relative improvement over the strongest open-weight competitor (7.76 vs. Lucy Edit's 5.44). Without EgoEditBench, this domain-specific advantage would be invisible, and the field would have no standardized way to measure progress on the very tasks that motivate the work.

The mutual dependence claim is a methodological contribution, not a technical one — it argues that the standard academic practice of contributing a dataset, a model, or a benchmark in isolation is insufficient for domain-transfer problems where the failure modes are unknown a priori. When the EgoEdit project began, no one knew exactly why existing editors fail on egocentric footage — was it the lack of hand-object interaction data? The lack of egomotion examples? Some architectural limitation? The triad approach solves this bootstrapping problem: the benchmark identifies the failure modes, the dataset targets those failures, and the model architecture ensures the learned capabilities are deployable. The paper's release of both EgoEditData and EgoEditBench publicly is an explicit commitment to enabling the community to iterate on any pillar independently — but the initial establishment of all three was necessary to define the problem space.

This insight is fundamental rather than incremental because it provides a template for attacking new generative domains. If someone wants to build a video editor for surgical footage, underwater exploration, or sports broadcasting, they should not start by fine-tuning an existing editor on domain data and evaluating on general benchmarks. They should construct domain-specific data (with curated edits targeting domain-specific challenges), a domain-specific benchmark (with tasks that stress-test those challenges), and a deployment-appropriate model variant simultaneously, using each to inform the others.

---

Innovation 4: Hand Preservation as an Explicit Constraint Resolves the Tension Between Fidelity and Edit Magnitude

The paper introduces a previously unarticulated tension in video editing: large structural edits (replacing a banana with a water gun) require modifying the object completely, but the hand holding the object must remain unchanged. In third-person footage, objects are rarely actively manipulated, so this tension does not arise — the editor can freely modify the target region. In egocentric footage, the target region contains the hand, and aggressive editing that successfully replaces the object will likely distort or overwrite the hand.

The conceptual contribution is the recognition that hand preservation must be an explicit first-class constraint in the data generation pipeline, not an emergent property of the editing model. The EgoEditData curation pipeline implements this at three levels:

1. Hand mask segmentation with human review (Stage 2): Hands are detected, segmented, tracked, and manually validated before any editing occurs. This ensures the system knows precisely which pixels must not be modified.

2. Mask exclusion during object editing (Stage 5): The object masks fed to Wan 2.1 VACE explicitly exclude hand regions. This is a literal, pixel-level instruction to the generator: "change the object, but do not touch the hand." Without this, the generator would treat the hand as part of the inpainting area and hallucinate a new hand — or worse, blend the hand into the new object, creating uncanny artifacts.

3. Human filtering for hand fidelity (Stage 5): Even with mask exclusion, the generation model may produce edits where the hand appears distorted at mask boundaries, or where the new object occludes fingers unnaturally. Human annotators reject these cases, ensuring the training data only contains examples where hands are realistically preserved.

This three-level constraint is fundamentally different from how prior editing datasets handle occlusions. General editing datasets (InsV2V, EditVerse, Se˜norita-2M) treat occlusions as a generic "background consistency" problem — the model should keep unedited regions unchanged. But in egocentric video, the hand is not "background" — it is a dynamic, deformable foreground element that is semantically central to the action. A generic background-preservation loss cannot distinguish between "keep this wall looking the same" (easy, static) and "keep this hand looking the same while it moves, rotates, and momentarily occludes the new object" (hard, dynamic). By encoding hand preservation into the data itself — not as a loss function or architectural constraint but as a property of the training examples — the paper ensures that the model learns hand preservation implicitly through exposure to correctly-handled examples.

The evidence that this works is qualitative (Figures 2, 6, 9, 10, 11) but compelling: EgoEdit's outputs consistently show clean hand-object boundaries where the replaced object sits naturally in the grasp while the hand's skin texture, pose, and motion trajectory are preserved. Competitors produce smeared hands, missing fingers, or objects that float disconnected from the hand. The paper does not ablate the hand-mask exclusion step specifically (doing so would require regenerating the entire dataset, which is infeasible at 0.112 fps on 8 H100 GPUs), but the failure modes of baselines — which lack any explicit hand-preservation mechanism — are consistent with the hypothesis that hand preservation must be engineered, not hoped for.

This insight generalizes beyond hands: any domain where the editing target is physically coupled to a dynamic foreground element (tools in surgical video, instruments in musical performance, equipment in sports) will require analogous explicit preservation constraints in the training data. The paper's mask-exclusion approach provides a template for such domains.

5. Experimental Analysis

Evaluation Methodology

• Dataset. All evaluation is conducted on EgoEditBench, the paper's custom benchmark comprising 1,700 source videos paired with instruction prompts across 15 egocentric editing tasks, constructed from a held-out split of Ego4D disjoint from EgoEditData training. For exocentric reference, EditVerseBench is used with reference-based tasks (Propagation, Inpainting, Reference Insertion, Edit with Mask) removed since EgoEdit does not support reference image conditioning.

• Base model(s). EgoEdit is based on a pretrained 10.7B-parameter DiT text-to-video generator operating in the latent space of a Wan 2.1 autoencoder with 8×8×4 spatial-temporal compression, trained with Rectified Flow. The real-time variant EgoEdit-RT is distilled from this model through DMD and Self-Forcing. Baselines span multiple families: attention manipulation (TokenFlow [49], STDF [71]), first-frame propagation (Se˜norita-2M [79], AnyV2V [29]), instruction-guided editors (InsV2V [11], Lucy Edit [57], EditVerse [26]), and streaming models (StreamDiffusion [27], StreamDiffusionV2 [17]).

• Metrics. Four metrics are reported. VLM Score (primary): a vision-language model judge evaluates edit quality against the instruction, following the exact EditVerseBench evaluation prompts — the VLM score aligns with human preference in 86.2% of cases against LucyEdit and 84.9% against InsV2V (Table 4). PickScore (PS): a learned perceptual quality metric evaluating overall video quality independent of instruction. Text Alignment (TA): CLIP-based video-level similarity between instruction text and generated video (frame-level TA is omitted as "highly correlated"). Temporal Consistency (TC): CLIP-based frame-to-frame coherence (DINO-based TC is omitted as "strongly correlated"). Results are averaged per task to ensure equal task weighting.

• Baselines. Eight baselines are compared. Attention manipulation: TokenFlow (Stable Diffusion 1.5 backbone, 16 frames, defaults) and STDF (24 frames at 10fps, 576×320, 40 inference steps, 10 optimization steps, guidance scale 10). First-frame propagation: Se˜norita-2M (33 frames at 8fps, 768×448, guidance 4, 30 steps, CogVideoX backbone, receives EgoEdit's edited first frame) and AnyV2V (16 frames at 8fps, 512×512, guidance 9, 100 inversion + 50 edit steps, also receives EgoEdit's first frame). Instruction-guided: InsV2V (32 frames at 15fps, 384×384, video guidance 1.2, text guidance 7.5, 20 steps), Lucy Edit (81 frames at 15fps, 832×480, guidance 5, 50 steps), and EditVerse (closed-source, evaluated only on EditVerseBench using published samples). Streaming: StreamDiffusion (image-to-image pipeline, 81 frames at 16fps, 832×480) and StreamDiffusionV2 (streaming setup at 832×480, 81 frames, 16fps, 2–4 denoising steps). For inversion-based methods requiring target captions rather than editing instructions, GPT-5 generates the target caption from the source prompt and instruction prompt.

• Generation budget / compute accounting. For fair comparison, all baselines use their recommended default inference settings (resolution, frame rate, sampling steps, guidance scale) following the EditVerseBench protocol. EgoEdit and EgoEdit-RT operate at 512×384 pixels and 16fps across all evaluations. Latency measurements (Table 2) are conducted on a single H100 GPU and decompose total first-frame latency into recording time (capturing the input video chunk), autoencoder encoding/decoding time, and model forward pass time. Throughput is reported in frames per second (fps) both with and without autoencoder overhead.

• Cross-validation / statistical protocol. Task-level scores are averaged to ensure equal weighting across all 15 EgoEditBench tasks, preventing high-count tasks like Change Object (400 samples) from dominating the aggregate. For the distillation ablation (Table 6), all variants are evaluated on the identical EgoEditBench test set. For the data ablation (Table 3, Table 7), models are evaluated at the 10k-iteration checkpoint rather than the full 30k-iteration training run, which the paper explicitly notes means these results differ from the main Table 1 results. No cross-validation or confidence intervals are reported for any quantitative results.

Main Quantitative Results

Aggregate Comparison on Egocentric Editing (EgoEditBench, Table 1)

EgoEdit achieves a VLM score of 7.76 on EgoEditBench, the highest among all evaluated methods — a 2.32-point absolute improvement over the strongest open-weight instruction-guided competitor Lucy Edit (5.44) and a 2.52-point lead over InsV2V (5.24). The real-time streaming variant EgoEdit-RT achieves 7.71 — only 0.05 points below the bidirectional teacher, while running at 38.1fps with 855ms first-frame latency.

The gap between EgoEdit and the second-best non-propagation method (Lucy Edit) spans 1.3× in VLM score. Among streaming models, EgoEdit-RT's 7.71 dominates StreamDiffusion (4.31) and StreamDiffusionV2 (2.55) by 1.8× and 3.0× respectively, quantifying the quality ceiling that prior real-time methods could not approach.

On auxiliary metrics: EgoEdit scores 19.21 PickScore (vs. 18.87 for Lucy Edit), 16.89 Text Alignment (vs. 15.03 for Lucy Edit), and 96.70 Temporal Consistency (vs. 94.41 for Lucy Edit). EgoEdit-RT scores 19.13, 16.34, and 96.41 on the same metrics — marginally below EgoEdit on all three, consistent with a slight quality tradeoff from autoregressive distillation. Frame propagation methods Se˜norita-2M and AnyV2V, which receive EgoEdit's edited first frame as input, achieve 7.52 and 6.72 VLM respectively — both below EgoEdit's 7.76 despite this advantage, demonstrating that single-frame propagation cannot recover the temporal dynamics of egocentric video even when initialized with a high-quality first frame.

Aggregate Comparison on Exocentric Editing (EditVerseBench, Table 1)

On EditVerseBench, EgoEdit achieves a VLM score of 8.00 — competitive with EditVerse's 8.26 (closed-source) and ahead of all other open-weight methods. EgoEdit-RT achieves 8.18, slightly exceeding the bidirectional teacher, though the paper does not comment on this apparent increase. The gap between EgoEdit and Lucy Edit narrows on exocentric tasks (8.00 vs. 6.27) compared to egocentric tasks (7.76 vs. 5.44), reflecting that Lucy Edit was designed for and trained on exocentric data.

A key pattern emerges from comparing the per-benchmark VLM drops when moving from exocentric to egocentric evaluation: EgoEdit drops only 0.24 points (8.00 → 7.76), while Lucy Edit drops 0.83 points (6.27 → 5.44) and InsV2V drops 0.47 points (5.71 → 5.24). This smaller degradation for EgoEdit directly quantifies the benefit of training on domain-aligned egocentric editing data — the model experiences less distribution shift because EgoEditData covers the egocentric scenario during training.

On auxiliary metrics for EditVerseBench: EgoEdit scores 19.61 PickScore (EditVerse: 19.69), 24.40 Text Alignment (EditVerse: 25.29), and 98.54 Temporal Consistency (EditVerse: 98.68). EgoEdit-RT shows a notable drop in Text Alignment on EditVerseBench (17.61 vs. 24.40 for EgoEdit), which the paper attributes to the qualitative gap documented in distillation — the real-time variant is less robust on out-of-distribution editing instructions and during temporary occlusions.

Per-Task Performance Analysis (Table 5, Appendix E.2)

The VLM score breakdown by task reveals where EgoEdit's advantages are largest. On Video-to-Sketch, EgoEdit achieves 8.93 vs. Lucy Edit's 3.72 (5.21-point gap) and InsV2V's 4.40 (4.53-point gap). On Video-to-Depth: 8.70 vs. 3.88 and 4.79. On Sketch-to-Video: 8.70 vs. 6.52 and 5.55. On Depth-to-Video: 8.57 vs. 7.31 and 5.60. These cross-modal tasks — converting between video and structural representations (depth, edge, pose) — show the largest margins, suggesting that EgoEdit's training on structural conditioning data (the 1.1M pairs generated by pairing natural videos with depth/pose/edge estimation) provides strong inductive biases for geometric consistency.

On Change Object — the core AR task and the largest category in EgoEditData — EgoEdit achieves 7.84 vs. Lucy Edit's 6.25 (1.59-point gap) and InsV2V's 3.99 (3.85-point gap). On Add Object: 7.89 vs. 4.31 and 5.67. On Remove Object: 6.82 vs. 4.81 and 5.52. These object-level tasks show meaningful but smaller gaps than the cross-modal tasks, reflecting that object editing is inherently harder and that even EgoEdit's performance leaves room for improvement (Remove Object at 6.82 has the second-lowest EgoEdit score behind Add Effect at 6.32).

On the hardest tasks for EgoEdit: Add Effect scores 6.32, Remove Object scores 6.82, Reasoning scores 6.79, and Change Camera Pose scores 6.79. These four tasks represent different challenges — Add Effect requires global filter application without corrupting local detail, Remove Object requires plausible inpainting of the region behind the removed object, Reasoning requires spatial/temporal inference, and Change Camera Pose requires novel view synthesis — and their relatively lower scores indicate that egocentric editing remains unsolved even for the strongest model.

Distillation Impact on Quality (Tables 2 and 6, Figure 12)

The distillation pipeline preserves editing quality through both phases. Table 6 reports the full metric breakdown across variants: EgoEdit (80 NFE) scores 7.76 VLM on EgoEditBench; EgoEdit-DMD (4-step bidirectional) scores 7.42 — a 0.34-point drop from the teacher; EgoEdit-RT (4-step autoregressive) scores 7.83 — actually exceeding the DMD variant by 0.41 points and essentially matching the original teacher (7.76 vs. 7.83). This pattern — DMD causes a small quality regression, then Self-Forcing recovers it — is consistent with the exposure-bias reduction mechanism: the DMD model was trained with teacher forcing (always seeing ground-truth context during training) and thus suffers from accumulating errors when rolled out at test time, while the Self-Forcing model was trained on its own autoregressive outputs and learned to self-correct.

PickScore follows a similar trajectory: 19.21 (teacher) → 18.95 (DMD) → 19.04 (Self-Forcing). Text Alignment: 16.89 → 16.52 → 16.49 — a small monotonic decline suggesting that autoregressive generation slightly reduces the precision of instruction following. Temporal Consistency: 96.70 → 96.87 → 96.49 — the DMD variant actually improves consistency slightly (possibly because fewer denoising steps reduce the opportunity for per-step variance accumulation), while Self-Forcing decreases it marginally (consistent with the paper's observation of "temporal shifts at the boundaries between different chunks" in qualitative analysis).

The latency and throughput analysis in Table 2 is the definitive result for real-time feasibility: Self-Forcing reduces first-chunk latency from 13,432ms (teacher) and 6,925ms (DMD) to 855ms, breaking the 1-second barrier for interactive use. The latency decomposition reveals that recording time (562ms for the first chunk of 9 RGB frames) and autoencoder time (217ms) dominate; the model forward pass contributes only 75.7ms (less than 9% of total latency). Further latency reduction would require smaller recording chunks (sacrificing some temporal context) or faster autoencoders, not faster model inference.

Data Ablation: Contribution of EgoEditData (Tables 3 and 7, Figure 13)

The data ablation quantifies how egocentric editing performance scales with the amount of EgoEditData included during training. Starting from the same T2V checkpoint and training for 10k iterations on the full editing corpora with varying fractions of EgoEditData retained (note: these are 10k-iteration checkpoints, not the full 30k-iteration training, so absolute numbers differ from Table 1):

The jump from 0% to 25% — adding only ~25k egocentric editing pairs (a quarter of the full EgoEditData) — produces a 2.25-point VLM improvement, representing 75% of the total gain from going to 100%. This strongly diminishing marginal return beyond 25% suggests that the model needs exposure to the egocentric distribution, not massive quantities of egocentric data specifically, to overcome the distribution shift — a finding with practical implications for dataset construction in other domains.

Table 7 extends this ablation to the exocentric EditVerseBench, showing that EgoEditData also improves general editing performance. As retention increases from 0% to 100%, the overall EditVerseBench VLM score rises from 6.89 to 7.79 — a 0.90-point gain, smaller in absolute terms than the egocentric gain but notable because EgoEditData was not designed for exocentric content. The per-task breakdown reveals disproportionate gains in: Reasoning (6.43 → 8.37, +1.94 points), Remove (4.30 → 7.00, +2.70 points), Camera Movement (6.27 → 7.60, +1.33 points), and Change (6.38 → 7.46, +1.08 points). The paper speculates these improvements originate from EgoEditData's "strict quality filtering and descriptive captions, whose benefits extend beyond the egocentric case."

Human Alignment Study (Table 4)

The VLM judge's reliability is validated through a human preference study comparing EgoEdit against LucyEdit and InsV2V across 30 randomly sampled benchmark elements per task category (450 total samples per comparison). The VLM judge agrees with human evaluators in 86.2% of cases against LucyEdit and 84.9% against InsV2V, averaged across all 15 tasks. Agreement varies by task: Depth-to-Video and Video-to-Pose achieve 100% agreement against both baselines; Reasoning drops to 63.3% against LucyEdit and 73.3% against InsV2V, suggesting that VLM evaluation of edits requiring spatial/temporal inference is less reliable — a finding that qualifies the VLM score's authority on the Reasoning task specifically.

Human evaluators preferred EgoEdit over LucyEdit in 91% of cases (411/450) and over InsV2V in 88% of cases (395/450) using VLM scoring, and in 91% (411/450) and 96% (431/450) respectively using the human study. The VLM slightly underestimates EgoEdit's advantage against InsV2V (88% vs. 96% human preference), suggesting the VLM judge is somewhat conservative relative to human raters, which makes the reported VLM scores conservative lower bounds on true quality differences.

Ablation Studies and Robustness Checks

Distillation phases contribute independently to different aspects of real-time performance: The three variants in Table 2 isolate the contributions of DMD and Self-Forcing. DMD alone (4 NFE, bidirectional) achieves 43.5fps throughput but 6.93s first-chunk latency — the throughput is sufficient for real-time display, but the batch-generation paradigm prevents interactive use. Self-Forcing adds chunk-wise autoregressive generation, achieving 855ms first-chunk latency while throughput drops to 38.1fps (still well above 16fps). The latency improvement (6,925ms → 855ms, 8.1× reduction) comes entirely from eliminating the need to process the full 81-frame video before displaying the first frame. This confirms that step-count reduction and autoregressive adaptation are separable problems requiring distinct distillation phases.

Training with simplified instructions degrades editing performance: Appendix F ("Failed Experiments") reports that initial experiments using template-based instructions ("Replace {source object} with {target object}") produced "low editing performance." The paper attributes this to three issues: limited variety in instruction phrasing, short object names from the extraction stage, and occasional mismatches between the target object description and what actually appeared in the generated video. Switching to GPT-5-generated verbose instructions (mean length 378 characters) "increased ability to follow instruction prompts." This ablation is not quantitative — no scores are reported — but it explains a key design choice in the data pipeline.

Usage of unfiltered editing data produces artifacts: Appendix F notes that training on "lightly filtered video editing pairs" caused "weak video editing performance, with the model often reproducing artifacts encountered in low-quality video editing data pairs such as failure of an object in being replaced with a different object, or failure in adding an object." This negative result motivated the extreme filtering of EgoEditData (0.4% yield) and the filtering of supplementary corpora. No quantitative ablation is provided, but the finding is consistent with the paper's quality-over-quantity thesis.

VLM judge reliability varies by task category: The per-task agreement rates in Table 4 reveal that certain tasks challenge the VLM evaluator more than others. Reasoning shows the lowest agreement (63.3–73.3%), followed by Remove Object (73.3–76.7%) and Add Effect (73.3–80.0%). Tasks with unambiguous success criteria — Video-to-Pose (93.3–93.3%), Video-to-Sketch (86.7–100%), Depth-to-Video (100–100%) — show near-perfect VLM-human alignment. This task-dependent reliability means that VLM scores on Reasoning, Remove, and Add Effect should be interpreted with more caution than scores on the cross-modal tasks.

Self-Forcing initialization from DMD checkpoint is sufficient: The paper reports skipping the standard ODE initialization phase of Self-Forcing because "the model can quickly adapt to autoregressive modeling when initialized from the DMD checkpoint." This is a practical finding that simplifies the distillation pipeline — the DMD model's learned velocity field transfers directly to causal attention without requiring an intermediate ODE-training phase.

Critical Assessment

Claim: "EgoEditData is the primary driver of egocentric editing performance."

What the experiments demonstrate: Table 3 shows a clear monotonic relationship between EgoEditData fraction and EgoEditBench VLM score, from 4.87 at 0% to 7.85 at 100%. This does demonstrate that EgoEditData substantially improves egocentric editing in the context of this specific model architecture and training recipe.

What the experiments do not demonstrate: The ablation does not control for total dataset size. The 0% condition removes 99.7k pairs; the 100% condition adds them. A model trained with 0% EgoEditData but an equivalent number of additional general editing pairs (to match total dataset size) would be needed to isolate the domain specificity of EgoEditData's contribution from the dataset size increase. The paper does not run this control. It is possible that any 99.7k additional high-quality video editing pairs — even non-egocentric ones — would produce similar gains through sheer data volume. The finding that EditVerseBench also improves with EgoEditData (Table 7) is consistent with this interpretation: EgoEditData might simply be a high-quality dataset that benefits training regardless of domain alignment.

A second uncontrolled variable: EgoEditData is the only dataset containing hand-mask-preserved edits. The ablation shows that removing it hurts, but cannot distinguish whether the active ingredient is the egocentric content itself or the hand-preservation constraint. A synthetic exocentric dataset constructed with similar hand-object interactions and mask exclusion (e.g., third-person videos of people handling objects) would be needed to isolate the egocentric perspective specifically.

Claim: "EgoEdit produces temporally stable, instruction-faithful results with interactive latency" and "achieves clear gains on egocentric editing benchmarks where existing methods struggle."

What the experiments demonstrate: EgoEdit's 7.76 VLM score on EgoEditBench meaningfully exceeds the strongest open-weight competitor (Lucy Edit, 5.44). The 2.32-point gap is large enough to be visible qualitatively (Figures 6, 11) and is corroborated by human preference (91% preference over LucyEdit, Table 4). The latency measurements (Table 2) are precise and convincing for the specific hardware configuration (single H100).

What the experiments do not demonstrate: The claim of "interactive latency" rests on the 855ms first-chunk measurement, but this is measured in a controlled lab setting. Real-world AR latency includes additional factors not accounted for: camera capture pipeline overhead, OS-level display latency, network transmission if the GPU is remote, and the time for the user to speak/type the instruction. The paper acknowledges 855ms is "suboptimal" (Section 6), and the decomposition shows 562ms is recording time — meaning even a zero-latency model would have >500ms first-frame delay due to the need to capture enough frames (9 RGB = 3 latent). This is a fundamental physics constraint, not an engineering limitation: the model cannot edit frames before they are captured. For truly interactive AR with sub-100ms perceived latency, the current architecture would require predictive generation (editing before the camera moves), which is a different problem entirely.

The claim of "temporally stable" results is supported quantitatively (TC: 96.70 for EgoEdit, 96.41 for EgoEdit-RT, both high) but qualified qualitatively: the paper notes EgoEdit-RT exhibits "temporal shifts at the boundaries between different chunks of predicted frames" (Appendix E.3). The TC metric averages frame-to-frame consistency across the entire video, which may underweight isolated but perceptually salient chunk-boundary artifacts. A metric that specifically measures inter-chunk consistency would provide a more targeted evaluation of this known failure mode.

Claim: EgoEdit-RT "delivers comparable results on all quantitative metrics" to the bidirectional teacher.

What the experiments demonstrate: Table 6 shows EgoEdit-RT scores 7.83 VLM vs. 7.76 for the teacher — essentially identical, with a slight increase. PickScore, TA, and TC are all within 0.5 points of the teacher.

What the experiments do not demonstrate: The paper acknowledges a qualitative gap that automated metrics do not capture: "lower proficiency in out-of-distribution editing instructions, less robust performance when editing objects becoming temporarily occluded, lower temporal consistency" (Section 6). The VLM score's task-dependent reliability (Table 4) means that if these qualitative failures concentrate in specific task categories (e.g., Reasoning, where VLM agreement is only 63.3–73.3%), the aggregate VLM score could mask them. The paper does not report per-task VLM scores for the distilled variants, making it impossible to assess whether distillation degrades specific capabilities while preserving aggregate scores.

Additionally, the 4-step DMD variant scores lower than both the teacher and the Self-Forcing variant (7.42 vs. 7.76 and 7.83), which is an unusual pattern — one would expect DMD to be an intermediate quality level between the teacher and the autoregressive student. The paper does not explain why Self-Forcing increases VLM score relative to DMD. Possible explanations include: (a) the DMD model was undertrained (4,500 iterations may be insufficient for full convergence), (b) Self-Forcing training provides additional optimization steps that refine the model beyond DMD quality, or (c) the VLM judge has a bias toward autoregressive outputs. Without investigation, the claim that the full pipeline preserves quality should be qualified as "preserves or slightly improves aggregate VLM score, with qualitative differences noted."

Missing Experiments

Isolated hand-preservation ablation: The most important unrun experiment would ablate the hand-mask exclusion step in the EgoEditData pipeline. Currently, whether the explicit hand preservation (mask exclusion during object editing, human review for hand fidelity) is necessary — or whether the model would learn hand preservation from seeing enough egocentric examples with hands present — is unknown. Running this ablation would require regenerating a subset of EgoEditData without the mask exclusion step and training a model on both versions, which is computationally prohibitive at 0.112fps on 8 H100 GPUs. But a small-scale version (e.g., 1,000 pairs) could provide suggestive evidence.

Latency-quality Pareto frontier: The paper presents only one operating point for the streaming model (4 NFE, 3-latent-frame chunks). A sweep over chunk sizes (1, 2, 4, 6 latent frames) and NFE counts (1, 2, 4, 8) would characterize the latency-quality tradeoff and enable practitioners to select operating points based on their application's latency tolerance. The claim that 855ms is "sufficient but suboptimal" would be strengthened by showing what quality would be achieved at, say, 400ms latency with smaller chunks.

Generalization to other base models: All results use a single 10.7B DiT backbone trained from a Wan 2.1 initialization. Whether the findings — particularly the domain-aligned data claim — transfer to other model families (CogVideoX-based editors, HunyuanVideo-based editors) is untested. This is a standard limitation of single-model-family papers, but it matters here because the core claim is about data, not architecture, and robustness across architectures would strengthen that claim.

Statistical significance: No confidence intervals, standard deviations, or significance tests are reported for any metric. The EgoEditBench test set has 1,700 samples, but task-level averages are based on 50–400 samples per task. The per-task VLM score improvements over baselines could have wide variance, particularly for tasks like Remove Object (50 samples) where differences of 1–2 points may not be statistically reliable.

6. Limitations and Trade-offs

6.1 The Difficulty Estimation Cost Is Excluded from Efficiency Claims

The assumption or constraint. The compute-optimal framework's ability to route prompts to the best strategy depends on knowing each prompt's difficulty before allocating the inference budget. The paper estimates difficulty by generating 2,048 samples per question and averaging either their ground-truth correctness (oracle) or the PRM's final-answer score (predicted). The paper acknowledges this cost explicitly in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

The reported $4\times$ efficiency gains over best-of-N (Figures 4, 8) are computed after difficulty is known, without amortizing the 2,048-sample estimation cost. In a deployment setting, generating 2,048 samples to decide how to spend a budget of 16–256 generations is self-defeating — the estimation cost exceeds the largest test-time compute budgets studied.

The consequence. The $4\times$ figure is an upper bound on achievable efficiency, not a realized deployment gain. In any practical system where difficulty is unknown a priori, the total cost is $C_{estimation} + C_{strategy}$. If $C_{estimation} \gg C_{strategy}$ (as is the case for low budgets like 16–64 generations), the claimed efficiency improvement evaporates. The paper's primary numerical claim is therefore contingent on the existence of a cheap difficulty estimator that does not yet exist. A practitioner deploying this system today would see zero efficiency gain because the difficulty estimation step alone consumes more compute than the entire best-of-N baseline at most budget levels.

What evidence exists in the paper. The paper is transparent about this gap (Section 3.2, acknowledged as "a key avenue for future work"), and the predicted difficulty bins (using PRM scores rather than ground-truth labels) partially address the label cost but not the sampling cost — both oracle and predicted methods require 2,048 samples. The curves for oracle and predicted bins largely overlap (Figures 4, 8), confirming that PRM-based difficulty estimation is a viable proxy for ground-truth, but neither variant accounts for the estimation budget. No experiment measures total cost including estimation.

Mitigation status. The paper suggests "pretraining or finetuning models to directly predict difficulty of a question" (Section 8) but develops no such model and reports no results. A lightweight difficulty classifier — possibly distilled from the PRM's score distribution — could reduce estimation cost from 2,048 generations to a single forward pass, which would make the compute-optimal framework immediately practical. The paper also flags adaptive estimation (using initial samples to estimate difficulty and allocating the remaining budget accordingly) as future work. Neither mitigation is evaluated.

---

6.2 The Method Provides No Benefit on Hard Problems — A Fundamental Capability Ceiling

The assumption or constraint. The entire compute-optimal framework assumes that the base model produces correct solutions at some non-trivial rate for the prompts of interest. When the base model's pass@1 is near zero (difficulty bin 5), no amount of test-time computation — search, revisions, or their combination — produces meaningful improvement. The paper states this explicitly in the Section 7 takeaway:

"test-time compute can amplify existing capability but cannot create it."

The consequence. For problems where the base model fundamentally lacks the required knowledge or reasoning ability, the approach offers no path forward. In difficulty bin 5 (the hardest quintile), accuracy remains at 1–3% for all methods and all budgets (Figure 3, right; Figure 7, right). Search cannot find correct solutions because none exist in the proposal distribution. Revisions cannot refine toward correctness because the model has no correct reasoning to build upon. The FLOPs-matched comparison (Figure 9) shows that scaling pretraining is strictly preferable on these problems — the $14\times$ larger model outperforms test-time compute by 37–53% relative, depending on the $R$ regime.

This limitation is structural, not merely empirical. Test-time compute operates by exploring or refining the base model's output distribution. If that distribution places zero probability mass on correct answers, exploration and refinement are futile. The approach therefore cannot extend the model's capability frontier — it can only more efficiently extract performance from capabilities the model already possesses.

What evidence exists in the paper. All difficulty-bin analyses show bin 5 as a flat line near zero across all budgets (Figures 3, 7, 9). The FLOPs-matched results (Section 7, Figure 1 bar charts) report negative relative improvements on hard problems across all $R$ regimes for PRM search, and negative improvements at $R \gg 1$ even for revisions. The paper is forthright about this limitation and does not overclaim applicability to hard problems.

Mitigation status. None. The paper correctly identifies this as a fundamental boundary: test-time compute and pretraining compute are not interchangeable. For problems outside the base model's capability, pretraining a larger model is the only viable path. The paper does not attempt to develop test-time strategies for such problems, nor does it suggest that future improvements to search or revision techniques could overcome this ceiling. The limitation is inherent to the proposal-distribution framework.

---

6.3 The Revision Model Suffers a 38% Correct-to-Incorrect Reversion Rate with Only Imperfect Mitigation

The assumption or constraint. The revision model is trained exclusively on trajectories where all in-context answers are incorrect, followed by a correct target (Section 6.1). This data construction creates an asymmetry: the model learns to revise incorrect answers into correct ones, but never learns to recognize when the current answer is already correct. At inference time, the model may encounter correct answers in its revision history — produced during earlier steps — and "revise" them into incorrect answers.

The paper reports that approximately 38% of correct answers are converted back to incorrect ones during sequential revision (Section 6.1). The mitigation is a post-hoc selection mechanism: pick the best answer from any point in the revision chain using majority voting or verifier-based selection, rather than always taking the final revision.

The consequence. The 38% reversion rate means that sequential revision chains are intrinsically unstable — longer chains accumulate more correct answers but also destroy some of them. The selection mechanism is a patch that recovers some of the lost performance, but it cannot recover the full potential of the chain because it operates post-hoc: it chooses among discrete answers that have already been generated. If the 9th revision in a 10-step chain destroys a correct answer that the 8th revision produced, and the 10th revision does not recover it, the selection mechanism can only pick the 8th answer — it cannot continue refining from that point. The effective length of useful revision chains is therefore limited by the reversion rate, and the optimal sequential-to-parallel ratios found in Figure 7 reflect this constraint: fully sequential chains underperform hybrid ratios because the reversion problem outweighs the benefits of additional refinement.

A more fundamental concern is that the reversion problem is a symptom of a deeper training failure: the model has not learned to condition its revision behavior on the correctness of the current answer. This means the revision model is not truly a "self-improvement" mechanism but rather a learned mapping that always attempts to modify its input, regardless of whether modification is needed.

What evidence exists in the paper. The 38% figure is reported in Section 6.1 without a detailed breakdown (per difficulty, per revision step). The selection mechanism's effectiveness can be inferred from Figure 6 (left): pass@1 at each step improves throughout the chain, but the improvement curve flattens, consistent with reversion offsetting some gains. The paper does not quantify how much performance is lost to reversion vs. what an ideal revision model would achieve.

Mitigation status. Partial mitigation via post-hoc selection (majority voting or verifier). The paper does not explore training the model to recognize and preserve correct answers — for example, by including "no change needed" trajectories in the training data, or by conditioning the revision model on a self-assessment of whether the current answer is correct. The ReST$^{EM}$ experiment (Appendix K, Figure 16) shows that alternative training approaches can make the reversion problem worse, not better, highlighting the sensitivity of revision training to data construction.

---

6.4 The Method Is Evaluated on a Single Benchmark and Model Family, with a Small Test Set for Strategy Selection

The assumption or constraint. All experiments use the MATH benchmark (500 test questions) with PaLM 2-S* as the base model. The authors state they "believe this model is representative of the capabilities of many contemporary LLMs" (Section 4), but this claim is unverified. More critically for internal validity: the 500-question test set is split into five difficulty quintiles (~100 questions each) and further split by two-fold cross-validation within each bin. This means the compute-optimal policy is selected based on ~50 questions per fold per bin. With such small sample sizes, the selected strategies may have high variance, and the reported gains may not be robust.

The consequence. The reported strategy selections — "use beam search on medium problems, best-of-N on easy problems, and this specific sequential-to-parallel ratio per difficulty bin" — may overfit to the specific 500-question test set. With 50 questions per fold per bin, a single question where beam search performs anomalously well can shift the selected strategy for that entire difficulty level. The paper does not report confidence intervals or statistical tests, so there is no way to assess whether the observed differences between strategies at a given difficulty-budget point are statistically reliable.

For practitioners on different model families or benchmarks, the specific optimal strategies almost certainly differ. A model with better calibration might experience less PRM over-optimization on easy problems; a model with different instruction-following capabilities might show different revision dynamics. The paper's findings about which strategy is optimal per difficulty level are therefore specific to PaLM 2-S* on MATH.

What evidence exists in the paper. The paper reports no confidence intervals, standard deviations, or significance tests for any quantitative result. The cross-validation protocol is described in Section 3.2 but its implications for statistical power are not discussed. The difficulty-bin sizes (~100 questions each) and fold sizes (~50 each) are stated.

Mitigation status. The paper acknowledges the single-benchmark limitation implicitly by framing the work as an initial study of compute-optimal test-time scaling, but does not call for replication on other benchmarks or model families (Section 8 focuses on methodological extensions, not validation). The predicted difficulty bins (using PRM scores) partially address the concern that strategy selection might overfit to ground-truth labels, but do not address the small-sample concern. Replication on other math benchmarks (GSM8K, AMC, AIME), code generation benchmarks (HumanEval, MBPP), and other model families would be needed to establish the generality of the difficulty-dependent scaling patterns.

---

6.5 The 14× Larger Model Baseline Is Weakened by Non-Optimal Pretraining and Greedy Decoding

The assumption or constraint. The FLOPs-matched comparison (Section 7) scales model parameters while holding training data fixed, following the LLaMA paradigm rather than compute-optimal scaling (Hoffmann et al., 2022) where both data and parameters scale. The paper acknowledges this departure:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

Additionally, the $14\times$ larger model uses only greedy decoding — no majority voting, best-of-N, or search of its own. The paper does allocate a small test-time budget to the larger model in any comparison.

The consequence. Both choices weaken the pretraining baseline, making the comparison favorable to test-time compute in ways that may not hold against stronger alternatives. A compute-optimally trained larger model (scaling data to maintain the Chinchilla ratio) would likely outperform a parameter-only-scaled model at the same total FLOPs, narrowing or reversing the reported advantages (e.g., +27.8% relative on easy questions at $R \ll 1$). Similarly, giving the larger model even a modest test-time compute budget — say, best-of-8 with majority voting — would create a fairer comparison, since the claim is about allocating a fixed total FLOPs budget between pretraining and inference. The current setup compares "all compute to pretraining, none to inference" against "some compute to pretraining, some to inference," which does not isolate the pretraining-vs-inference tradeoff.

The magnitude of this concern is difficult to assess without the missing baselines, but it is non-trivial. Best-of-N with $N=8$ using majority voting would cost the larger model 8× more inference FLOPs (which is accounted for in the total budget), and could substantially close the gap, particularly on easy-to-medium problems where even the smaller model's best-of-N shows strong scaling (Figure 3).

What evidence exists in the paper. The paper is transparent about the non-optimal pretraining (Section 7), framing it as "representative of a canonical approach" rather than optimal. The greedy decoding choice for the larger model is not explicitly justified — it appears to be an implicit assumption that test-time compute is only applied to the smaller model. The FLOPs accounting formulas (Section 7) are precise and correctly model the tradeoff for the chosen configuration; the issue is with the configuration itself, not the accounting.

Mitigation status. The paper explicitly leaves compute-optimal pretraining to future work. Giving the larger model a test-time budget is not discussed. A proper comparison would construct a Pareto frontier: for a fixed total FLOPs budget, vary the pretraining-inference split for both the smaller and larger models, and compare the best achievable accuracy at each point. This would answer the question "is it better to spend FLOPs on pretraining or inference?" rather than "is it better to spend FLOPs on pretraining or inference when only the smaller model uses inference compute?"

---

6.6 The Revision Model and PRM Search Are Never Combined — a Missed Opportunity and an Incomplete Picture

The assumption or constraint. The paper studies revisions (modifying the proposal distribution) and PRM-guided search (modifying the verifier) as independent mechanisms. They are evaluated separately and their compute-optimal policies are optimized independently. The paper does not combine them — for example, using the revision model as the proposal distribution within beam search, or using the PRM to guide which revision trajectories to pursue. The authors explicitly acknowledge this in Section 8:

"we did not experiment with PRM tree-search techniques in combination with revisions"

The consequence. The paper's results represent a lower bound on what test-time compute can achieve. The two mechanisms have complementary strengths: revisions improve the quality of generated candidates (helpful on easy problems), while PRM search identifies the best among candidates (helpful on medium problems). A combined system could potentially shift the optimal strategy boundaries — for instance, a revision model that produces higher-quality initial proposals might make beam search effective even on easier problems (since the proposals are less likely to exploit verifier weaknesses), or PRM-guided revision could reduce the 38% reversion rate by scoring intermediate steps and aborting unpromising revision trajectories.

Without this combination experiment, the paper's central decomposition into proposal distribution and verifier axes — while conceptually clean — is empirically incomplete. The reader does not know whether the $4\times$ efficiency gains are additive (combining both mechanisms yields $8\times$), sub-additive (the gains overlap), or even super-additive (synergistic). The paper's compute-optimal policy is also incomplete: it selects between search strategies or between revision strategies, but not between search, revisions, and their combinations. This leaves open the question: should a practitioner deploy revisions, search, or some hybrid, and how should the budget be split between them?

What evidence exists in the paper. None. The combination is not evaluated, and the paper provides no theoretical analysis of how the two mechanisms would interact.

Mitigation status. Acknowledged as future work (Section 8). The paper's framework does provide the scaffolding for such an experiment — the compute-optimal objective (Equation 1) could be extended to include combined strategies in the hyperparameter space $\theta$, and the difficulty-conditioned allocation could select among combined strategies per difficulty bin. The computational cost of running this experiment is substantial (training a revision model that is compatible with PRM-guided search may require additional engineering), which likely explains its absence from the current paper.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper fundamentally reframes real-time video editing for augmented reality as a data-distribution problem, not a model-capacity problem. The diagnostic finding that a model trained on 1.31M general video editing pairs plus 3.5M image editing pairs achieves only 4.87 VLM score on egocentric benchmarks (Table 3, 0% EgoEditData) — while adding just 25K egocentric pairs raises that to 7.12 — demonstrates that the field's prevailing "more data, larger models" scaling philosophy fails to transfer across the egocentric-exocentric domain gap. The model possesses sufficient architectural capacity (10.7B DiT parameters with channel-wise conditioning); it simply never encountered hand-object interactions, rapid egomotion, and active object manipulation during training. This is a paradigm shift in how generative editing research should approach domain-specific applications: the primary investment should be in curated, domain-aligned data with explicit structural constraints (in this case, hand mask exclusion during editing), not in marginal architectural improvements or scale increases.

The work also establishes that real-time AR editing quality need not trade off against latency when the distillation pipeline targets the correct axes. Prior to EgoEdit, the field operated under an implicit assumption that real-time video editing was fundamentally quality-limited — StreamDiffusionV2's 2.55 VLM score on EgoEditBench being the clearest evidence. By decomposing the latency problem into step-count reduction (DMD: 80→4 NFEs) and generation-paradigm conversion (Self-Forcing: batch→autoregressive streaming), EgoEdit demonstrates that these are independent constraints — solving each with a dedicated distillation phase preserves 99.4% of bidirectional teacher quality (7.83 vs. 7.76 VLM) while reducing first-frame latency by 15.7× (13,432ms → 855ms). This separation provides a template for any batch video diffusion model seeking real-time deployment.

The paper reconciles a latent contradiction in video editing research: why do trained editors (EditVerse, Lucy Edit) perform well on general benchmarks while failing on egocentric footage, while training-free methods (StreamDiffusion, attention manipulation) fail universally? The answer, demonstrated quantitatively by the per-benchmark VLM drops (EgoEdit: -0.24 points; Lucy Edit: -0.83 points; InsV2V: -0.47 points when moving from EditVerseBench to EgoEditBench), is that distribution shift magnitude, not method category, determines domain transfer success. Trained editors fail on egocentric footage because they lack egocentric training data, not because of fundamental architectural limitations. Training-free methods fail because the egocentric domain is too challenging for lightweight heuristics. Both observations point toward the same solution: purpose-built, domain-aligned training data.

Several research directions become more attractive as a result. Dataset-centric approaches to domain transfer — where the primary investment is curating, filtering, and structurally constraining training pairs — now have a compelling case study showing that 0.4% data yield with aggressive human curation can outperform million-scale weakly-filtered corpora. Two-phase distillation ladders (step compression → autoregressive adaptation) emerge as a general recipe for converting batch video generators into streaming systems, applicable to surgical video, teleoperation feeds, or drone footage. Hand-object interaction preservation is elevated from an implementation detail to a first-class design requirement for any editing system targeting manipulation-heavy domains.

Directions that become relatively less attractive: training-free real-time editing (the StreamDiffusion lineage), given the 3× quality gap between StreamDiffusionV2's 2.55 and EgoEdit-RT's 7.71 on egocentric tasks; scale-only approaches to domain generalization, given that 5M general editing pairs cannot substitute for 100K domain-aligned pairs; and single-phase distillation that attempts to jointly solve step-count reduction and autoregressive conversion — the paper's ablation shows each phase contributes to different metrics (DMD: +throughput, -first-frame latency unchanged; Self-Forcing: -first-frame latency, slight temporal consistency cost), implying that conflating them risks suboptimal tradeoffs.

---

Follow-Up Research This Work Enables

Measuring the hand-preservation effect in isolation. The paper argues that explicit hand mask exclusion during object editing — combined with human review for hand fidelity — is critical for preserving hand structure in edited videos. However, this claim is not ablated: the dataset generation pipeline applies mask exclusion universally, so there is no control condition. A targeted experiment would generate a small subset of EgoEditData pairs (e.g., 1,000) without hand mask exclusion — allowing Wan 2.1 VACE to freely inpaint the hand region along with the object — and train two models: one on mask-excluded pairs, one on no-exclusion pairs. Evaluating both on EgoEditBench with a hand-specific metric (hand detection confidence post-edit, finger count preservation, grasp pose consistency) would quantify how much of EgoEdit's advantage over baselines comes from the mask exclusion step versus mere exposure to egocentric footage. If the effect is large, this validates the paper's implicit claim that structural constraints must be engineered into editing data, not learned emergently. If small, it suggests that exposure to diverse egocentric scenes is the primary driver.

Characterizing the latency-quality Pareto frontier of the streaming model. The paper presents a single operating point for EgoEdit-RT: 4 NFEs, 3 latent frames per chunk, 855ms first-chunk latency, 38.1fps throughput. A systematic sweep over chunk sizes (1, 2, 4, 6 latent frames) and NFE counts (1, 2, 4, 8) at fixed model capacity would map the tradeoff curve between first-frame latency and VLM score. The hypothesis: smaller chunks reduce recording+autoencoder latency (the 562ms + 217ms in Table 2 that dominate total latency) but reduce temporal context available to the model, potentially degrading consistency at chunk boundaries. Larger chunks improve quality but increase latency. A 1-latent-frame-per-chunk model (3 RGB frames, ~188ms recording) could approach 400ms first-frame latency — crossing the threshold for perceived "instantaneous" AR interaction — but might exhibit severe chunk-boundary flicker. Characterizing this curve would enable practitioners to select operating points based on their application's latency tolerance (AR gaming: sub-200ms; furniture visualization: sub-1s; video post-production: sub-5s), and would reveal whether current model capacity saturates at some chunk size, suggesting that larger models might be needed for ultra-low-latency operation.

Testing domain transfer of EgoEditData to other model families. The paper's central claim — that EgoEditData provides domain-aligned training that existing models lack — is demonstrated only on a single 10.7B DiT backbone initialized from Wan 2.1. A replication experiment would fine-tune two alternative architectures on the identical EgoEditData + general editing corpora mixture: a CogVideoX-based editor (transformer backbone with 3D causal convolutions) and a HunyuanVideo-based editor (hybrid architecture with 3D VAE). Evaluating all three on EgoEditBench would test whether the data-centric thesis generalizes beyond the Wan autoencoder and DiT design space. If CogVideoX and HunyuanVideo show similar monotonic improvements with EgoEditData fraction (replicating Table 3), the finding is architecture-independent and broadly applicable. If only the Wan-based model benefits, the data advantage is confounded with architectural compatibility — perhaps Wan 2.1 VACE's generation artifacts during dataset construction are better handled by a Wan-initialized editor because the student mimics the teacher's biases. This distinction matters practically: if EgoEditData's value is architecture-specific, practitioners must carefully match their base model to their data generation pipeline, complicating the ecosystem approach.

Self-Forcing with larger chunk overlaps to reduce boundary artifacts. The paper notes qualitatively that EgoEdit-RT exhibits "temporal shifts at the boundaries between different chunks" (Appendix E.3). The current Self-Forcing configuration uses non-overlapping chunks (3 latent frames each) with a 5-chunk conditioning window. An ablation could introduce overlapping chunks during training and inference — for example, generating 3-latent-frame chunks with 1-frame overlap (chunk i: frames [t, t+1, t+2]; chunk i+1: frames [t+2, t+3, t+4]). The model would generate overlapping frames twice and blend them (temporal averaging or learned fusion), potentially smoothing chunk-boundary discontinuities at the cost of additional compute (~33% more NFEs). A quantitative metric for boundary artifacts — perhaps the mean CLIP similarity difference between frames at chunk boundaries vs. intra-chunk frame pairs — would provide a targeted evaluation of whether overlap mitigates this specific failure mode. The 38.1fps throughput of the current model provides headroom for such overhead (16fps display rate means the model can afford 2.4× more computation per frame and still meet real-time).

In-the-wild user study with latency as a controlled variable. The paper's latency claims (855ms first-frame, 38.1fps) are hardware measurements, not user-experience evaluations. An HCI-style study would present participants with EgoEdit-RT outputs at controlled latencies (achieved by artificially delaying frame display: 500ms, 855ms, 1.5s, 3s) and measure task completion time, subjective presence ratings, and edit quality satisfaction for standard AR tasks (object replacement, effect overlay). This would establish the acceptable latency envelope for egocentric editing — whether 855ms crosses a perceptual threshold (e.g., the ~100ms "instantaneous" threshold for hand-tracking in VR, or the ~1s conversational turn-taking threshold). If users cannot distinguish 500ms from 855ms, further latency reduction is unnecessary; if 855ms feels "laggy" for object manipulation tasks, the Pareto frontier study (see above) becomes urgent. The paper's acknowledgement that 855ms is "sufficient but suboptimal" (Section 6) requires user validation to interpret.

---

Practical Applications and Downstream Use Cases

On-device AR prototyping with real-time content editing. The current AR development pipeline requires 3D artists, shader programmers, and engine integration specialists to build each interactive experience. EgoEdit-RT's 855ms latency and 38.1fps throughput on a single H100 GPU enable a qualitatively different workflow: a designer speaks editing instructions while wearing an egocentric camera, sees the modified scene in under a second, and iterates instructions in real time. A furniture visualization app could prototype "replace this chair with a mid-century modern armchair in walnut" in seconds rather than waiting hours for a 3D asset to be sourced, rigged, and lit. The 4-NFE model's per-chunk cost of 75.7ms (Table 2) means that even on hardware a quarter as powerful as an H100, latency remains under 2s — still usable for iterative prototyping if not for final deployment. The 38.1fps throughput provides 2.4× headroom over the 16fps capture rate, meaning the model can process edits faster than the camera records, enabling smooth "watch-as-you-generate" displays.

Batch evaluation of AR editing quality for model selection. Organizations developing AR editing systems face a model selection problem: which base architecture, training data mixture, and distillation configuration produces the best egocentric editing quality? EgoEditBench's 1,700 test samples across 15 tasks — with VLM-based scoring validated at 86.2% human agreement (Table 4) — provides a standardized, automated evaluation pipeline. A team could train 20 model variants with different data mixtures (varying EgoEditData fraction, hand-mask exclusion, instruction verbosity) and evaluate all configurations in hours on a single GPU cluster, rather than running expensive human preference studies. The per-task VLM breakdown (Table 5) enables targeted diagnosis: if a model scores 8.93 on Video-to-Sketch but 4.01 on Remove Object, the failure mode is localized to object removal specifically, suggesting that the training data lacks sufficient removal examples or that the mask-conditioning mechanism is insufficient for inpainting behind removed objects.

Training data quality filtering for domain-specific generative models. The EgoEditData curation pipeline provides a template for constructing high-quality editing datasets in any domain where structural constraints can be automatically extracted and manually validated. For surgical video editing (replacing instruments, adding annotations, restyling for training simulations), the pipeline analog would: (1) detect instrument masks using surgical tool segmentation models (analogous to hand mask segmentation), (2) identify manipulated anatomical structures via VLM (analogous to object name extraction), (3) generate edited videos with instrument replacement/removal while preserving tissue and hand structure (analogous to Wan 2.1 VACE with mask exclusion), and (4) human-filter for anatomical consistency and tool-tissue interaction realism. The paper's finding that 0.4% data yield with aggressive filtering produces 2.25-point VLM improvements on egocentric benchmarks (Table 3, 0%→25% EgoEditData) suggests that even small, heavily-curated domain datasets can substantially improve specialized editing quality — a practical alternative to the million-scale automated collection paradigm for domains where automation produces low-quality pairs.

---

When to Prefer This Method

The paper positions EgoEdit against both offline trained editors (Lucy Edit, InsV2V, EditVerse) and real-time streaming editors (StreamDiffusion, StreamDiffusionV2). The decision rules that emerge from the experimental results are:

Prefer EgoEdit/EgoEdit-RT when:

• The footage is egocentric with active hand-object interactions (grasping, manipulating, occluding objects). On EgoEditBench, EgoEdit achieves 7.76 VLM vs. Lucy Edit's 5.44 — a 2.32-point gap that represents the difference between a convincing edit and a hand distorted beyond recognition.
• Interactive latency is required (sub-second first-frame display). EgoEdit-RT at 855ms is 8× faster to first frame than the next-fastest trained editor (DMD-distilled EgoEdit at 6,925ms). StreamDiffusion offers similar latency but at VLM scores of 4.31 (1.8× worse).
• The editing task involves structural object changes (substitution, removal) rather than just appearance changes. The mask-exclusion pipeline in EgoEditData explicitly trains the model to preserve unedited regions during large structural edits, while attention-manipulation methods (TokenFlow: 4.99 VLM) fail when structural correspondence breaks down.
• A high-quality, domain-aligned dataset can be constructed following the EgoEditData template (detection → segmentation → human review → generation with structural constraints). The paper's ablation (Table 3) shows that even 25K domain-aligned pairs produce substantial gains over 5M general editing pairs, making the approach tractable for specialized domains.

Prefer offline trained editors (Lucy Edit, EditVerse) when:

• The footage is standard exocentric content (movie clips, stock footage, third-person user-generated video). EgoEdit achieves 8.00 VLM on EditVerseBench vs. EditVerse's 8.26 — competitive but not dominant, and EditVerse (as a closed-source model) may have additional capabilities (reference image conditioning) that EgoEdit lacks.
• Latency is irrelevant (batch processing, offline video post-production). The bidirectional EgoEdit teacher model achieves 7.76 VLM but with 13.4s latency — paying the distillation cost for no benefit if interactivity is not required. Lucy Edit at 50 inference steps provides strong quality without the engineering complexity of DMD + Self-Forcing.
• The editing task requires reference image conditioning (inserting a specific object from a reference photo, editing with a provided mask). EgoEdit does not support reference conditioning; EditVerse and Se˜norita-2M do.

Prefer larger pretrained models (scaling pretraining compute) when:

• The problem involves fundamentally novel object categories, interaction types, or scene compositions that fall outside the base model's generation capabilities. EgoEditData's training pairs are generated by Wan 2.1 VACE — if that model cannot produce a particular edit, no amount of domain-aligned fine-tuning will teach EgoEdit to perform it. The paper's finding in the egocentric domain is that existing models can handle the edits but lack the right training distribution; for domains where even the strongest generators fail, pretraining advances are necessary.
• The deployment hardware cannot accommodate a 10.7B-parameter model even in distilled form. The paper targets single-H100 deployment; on consumer GPUs or mobile devices, further compression (quantization, pruning, architecture reduction) would be needed, potentially eroding the quality advantage over smaller general-purpose editors.