Optical flow models trained on high-quality data often degrade severely when confronted with real-world corruptions such as blur, noise, and compression artifacts. To overcome this limitation, we formulate Degradation-Aware Optical Flow, a new task targeting accurate dense correspondence estimation from real-world corrupted videos. Our key insight is that the intermediate representations of image restoration diffusion models are inherently corruption-aware but lack temporal awareness. To address this limitation, we lift the model to attend across adjacent frames via full spatio-temporal attention, and empirically demonstrate that the resulting features exhibit zero-shot correspondence capabilities. Based on this finding, we present DA-Flow, a hybrid architecture that fuses these diffusion features with convolutional features within an iterative refinement framework. DA-Flow substantially outperforms existing optical flow methods under severe degradation across multiple benchmarks.
Score note
›Score rationale
While the paper uses diffusion models for a visual task, its primary focus is on optical flow estimation and robust feature extraction under degradation, which does not overlap with the user's focus on LLM agentic reasoning, alignment, or efficient transformer architectures.
Restoration diffusion features, when given spatio-temporal attention across video frames, contain zero-shot correspondence information even under severe noise and blur. DA-Flow fuses these degradation-aware features with a standard flow backbone to estimate accurate motion from inputs where existing methods completely fail, achieving dramatic error reductions on corrupted benchmarks.
1. Executive Summary
This paper introduces Degradation-Aware Optical Flow, a new task targeting accurate dense correspondence estimation from real-world corrupted videos by leveraging intermediate features from image restoration diffusion models. The core insight is that a pretrained DiT-based image restoration model, when lifted to the video domain via full spatio-temporal attention across adjacent frames, produces features that exhibit zero-shot geometric correspondence capabilities even under severe corruption. The authors propose DA-Flow, a hybrid architecture that fuses these diffusion features with convolutional features from a RAFT-style encoder within an iterative refinement framework, and demonstrate that it substantially outperforms existing optical flow methods—reducing End-Point Error (EPE) by a clear margin on Sintel and Spring while achieving best outlier rates on TartanAir—establishing that degradation-aware diffusion priors enable reliable motion estimation from inputs whose visual evidence is severely corrupted, though the approach requires multiple denoising steps at inference, making it slower than conventional flow networks.
2. Context and Motivation
The Core Problem: Optical Flow Breaks Down Under Real-World Degradations
The fundamental question this paper tackles is deceptively simple: can we accurately estimate optical flow from severely corrupted video frames? Modern optical flow methods — exemplified by RAFT, SEA-RAFT, and FlowSeek — achieve remarkable accuracy on clean, high-quality inputs. However, the paper argues that this success is brittle. When confronted with the kinds of degradation that pervade real-world video — motion blur, sensor noise, compression artifacts, low resolution — these methods degrade severely. As the authors put it in Section 1:
"Real-world videos are rarely clean; motion blur, sensor noise, compression artifacts, and low resolution frequently co-exist, severely degrading visual quality. Despite the prevalence of such degradations, how optical flow models behave under such degradations remains largely unexplored."
This is not merely an academic observation about distribution shift. The paper makes a stronger claim: under severe degradation, correspondence estimation becomes inherently ambiguous rather than simply difficult. When fine textures are destroyed and motion boundaries are attenuated by blur, there simply isn't enough visual evidence to match pixels reliably using conventional feature representations. The problem is fundamentally ill-posed — the degradation destroys the very information that traditional optical flow pipelines depend on.
The practical stakes are high because optical flow is not an end in itself — it serves as a foundational component in numerous downstream video tasks, including video generation, scene reconstruction, frame interpolation, and temporal consistency enforcement in video restoration. When flow estimates are corrupted by input degradation, these downstream applications compound the errors. The paper's Appendix 0.C.4 concretely demonstrates this chain of consequences: existing video restoration methods that use off-the-shelf optical flow for frame alignment suffer degraded temporal consistency precisely because the flow model receives degraded frames as input and produces inaccurate correspondences.
A Gap Between Robustness Studies and Practical Solutions
The paper positions itself in response to a specific gap in the literature. RobustSpring (Schmalfuss et al., 2025) provided what the authors call "the first comprehensive study on the robustness of dense matching models," benchmarking how existing flow and stereo methods generalize from clean synthetic training data to a wide spectrum of real-world degradations. That study established that there is a problem — optical flow models are not robust — but it left unanswered a more ambitious question:
"is it truly impossible to accurately estimate optical flow from corrupted inputs?"
The paper reframes this from a robustness problem (where the goal is to maintain performance under distribution shift) to an accuracy problem (where the goal is to actually solve the harder corrupted-input task). This is a critical distinction. Robustness approaches — the authors mention "simply augmenting clean training data with synthetic corruptions" — attempt to make models insensitive to degradation, but they argue this is insufficient for severe corruption because the underlying task has changed: you are no longer matching clean pixels to clean pixels; you are trying to recover motion information that may be entirely absent from the observable pixel values.
The Missing Ingredient: Degradation-Aware Representations
The paper identifies the core bottleneck as the feature encoder. As Equation 1 in Section 3.1 formalizes, modern optical flow follows a three-stage pipeline: feature encoding (ℰ), correlation (𝒞), and iterative update (𝒰). The authors argue:
"the feature encoder ℰ is most directly affected by input degradation, as corrupted pixels lead to unreliable features that propagate errors into all downstream stages."
This is a precise diagnosis. The correlation operator — which constructs a cost volume from pairwise feature similarities — can only be as good as the features it receives. If the encoder produces noisy, textureless representations because the input is blurry or compressed, the cost volume will lack the discriminative structure needed to pinpoint correspondences. No amount of iterative refinement downstream can recover from this upstream failure.
What is needed, therefore, are representations that are both degradation-aware and structurally rich for dense matching. The features must encode information about what the degradation has done to the image — recovering clean structure from corrupted observations — while preserving the spatial detail needed for precise boundary localization. This dual requirement is what drives the paper's design choices.
Why Diffusion Models — and Why Restoration Diffusion Models Specifically
The paper builds on a growing body of evidence that diffusion models encode rich geometric information in their intermediate representations. DIFT (Tang et al., 2023) showed that correspondence can emerge from image diffusion features without any task-specific training. DiffTrack (Nam et al., 2025) extended this to video, demonstrating that query-key similarities in selected layers of video diffusion transformers encode temporal correspondences across frames. DINOv2 (Oquab et al., 2023) provided strong semantic features, and Tale (Zhang et al., 2023) showed that simply fusing diffusion and DINOv2 features yields robust dense correspondences.
However, the paper identifies a specific limitation in this prior work: these diffusion models were trained for image generation, not image restoration. A generation model learns to produce realistic images from noise, which implicitly requires learning about object structure, geometry, and semantics — but it does not explicitly learn to recover information from degraded observations. The paper's key insight is that image restoration diffusion models — which are trained to map degraded inputs to clean outputs — offer a more suitable foundation:
"Image restoration is likewise a highly underdetermined inverse problem, and models trained for this task must learn to recover clean structures from degraded inputs. As a result, their intermediate features naturally encode degradation patterns while preserving underlying scene geometry."
This is a non-obvious but elegant argument. An image restoration model, by the nature of its training objective (Equation 6: minimizing velocity prediction error from degraded-conditioned noisy latents), must develop internal representations that "understand" how degradation corrupts structure and how to recover it. These representations are, by construction, degradation-aware. The paper's contribution is recognizing that this degradation awareness is precisely what optical flow needs to handle corrupted inputs — and that these features also happen to encode geometric correspondence, thanks to the structural learning that diffusion models perform.
The Temporal Awareness Gap
But there is a catch. Image restoration diffusion models operate on single frames. They provide:
"strong per-frame restoration capability but lack any mechanism for temporal modeling."
Optical flow requires comparing features across pairs of frames. A per-frame feature extractor, however powerful, has no way to reason about temporal relationships — it processes frame k and frame k+1 independently, with no awareness of how pixels move between them. The features might be degradation-aware, but they are spatially static — they encode structure within a frame, not correspondence between frames.
This creates a specific architectural challenge that the paper must solve: how to inject temporal reasoning into an image restoration diffusion model without destroying its degradation-aware priors.
Why Not Simply Use a Video Restoration Diffusion Model?
A seemingly natural solution would be to use a video restoration diffusion model — since these models handle both degradation and temporal dynamics jointly. Models like STAR (Xie et al., 2025), Dove (Chen et al., 2025), and FlashVSR (Zhuang et al., 2025) are designed for exactly this dual purpose. However, the paper identifies a fundamental architectural mismatch:
"such models often encode a stack of degraded frames into a temporally compressed latent representation through 3D convolutions or temporal attention. This produces a shared latent tensor where the temporal axis is entangled early in the encoding pipeline."
The problem is structural, not qualitative. Video restoration models compress multiple frames into a single shared latent space early in processing. This is perfectly sensible for restoration — temporal smoothness and global consistency are desirable properties for output video quality. But for dense correspondence estimation, it is catastrophic:
"Optical flow requires comparing spatial features extracted independently from each frame to establish pixel-level correspondences. When degraded frames are jointly encoded into a shared spatio-temporal latent space, their per-frame spatial structure is no longer preserved as separable entities, making the representation ill-suited for explicit pairwise feature matching."
In other words, a video restoration model's compressed latent destroys the per-frame spatial independence that optical flow correlation operators depend on. You cannot construct a meaningful cost volume between two frames when their features have been fused into a shared, entangled representation. Appendix 0.B.3 empirically confirms this: extracting features from FlashVSR and attempting zero-shot correspondence produces "significantly inferior" EPE compared to both the baseline and the lifted image model — directly validating the architectural argument.
The Paper's Position: Lift, Don't Replace
The paper's approach is to lift rather than replace. Instead of abandoning the strong degradation-aware priors of image restoration models, the paper injects temporal reasoning in a controlled way:
Start from a pretrained DiT-based image restoration model (DiT4SR) that preserves per-frame spatial latents independently — avoiding temporal entanglement.
Modify the attention mechanism to operate across frames (full spatio-temporal attention), enabling the model to learn inter-frame correspondences during finetuning.
Keep the per-frame spatial structure intact — each frame's tokens remain separable entities — so that frame-frame feature comparisons remain valid for optical flow.
This design preserves what the paper calls the "strong degradation-aware priors from image restoration pretraining" while adding the temporal reasoning needed for correspondence estimation. It also avoids the computational overhead of video diffusion architectures, which the paper argues are substantially more expensive.
The approach is positioned as filling a specific gap between two lines of work: image diffusion models (which have geometric features but no temporal awareness) and video diffusion models (which have temporal modeling but entangle per-frame structure). By lifting an image restoration model with cross-frame attention, the paper claims to get the best of both: degradation-aware geometric features with explicit inter-frame correspondence capability.
How the Paper Fits Into the Broader Landscape
The paper explicitly builds on several prior lines of work:
Optical flow architectures (RAFT, SEA-RAFT, FlowSeek): DA-Flow retains the correlation and iterative refinement stages from these models, replacing only the feature encoder with the hybrid diffusion-CNN encoder. The paper is not proposing a new flow architecture but rather a new way to produce features for existing architectures.
Diffusion features for correspondence (DIFT, DiffTrack, Tale): The paper extends this line by showing that restoration-pretrained features are specifically suited for degraded-input correspondence, and that lifting to video via spatio-temporal attention makes these features usable for optical flow.
Image restoration diffusion (DiffBIR, SeeSR, DreamClear, DiT4SR): The paper repurposes restoration models as feature extractors for a different task, which is analogous to how prior work repurposed generation diffusion models as correspondence feature extractors.
Where the paper differs from prior work is in the combination of these elements: using a lifted image restoration diffusion model as a degradation-aware feature encoder within a conventional optical flow pipeline, trained with pseudo ground-truth on a large-scale video dataset. None of the prior work addresses degraded-input optical flow estimation specifically, and none combines restoration diffusion features with CNN features in a hybrid optical flow architecture.
3. Technical Approach
3.1 Reader Orientation
DA-Flow is a system that estimates dense optical flow (per-pixel motion) from severely degraded video frames by augmenting a standard optical flow network with features extracted from a video-lifted image restoration diffusion model. The system solves the problem that conventional optical flow feature encoders fail catastrophically on corrupted inputs — blur, noise, compression artifacts destroy the texture information needed for pixel matching — by providing a hybrid feature encoder that combines degradation-aware diffusion features (which "understand" how corruption affects image structure) with fine-grained convolutional features (which provide precise spatial localization), enabling reliable correspondence estimation even when the input's visual evidence is severely compromised.
3.2 Big-Picture Architecture
DA-Flow's architecture has five major components organized as a pipeline:
Lifted Diffusion Model ($\mathcal{D}_\phi$) — a pretrained DiT-based image restoration model (DiT4SR) that has been extended with cross-frame spatio-temporal attention and finetuned on video data. It takes pairs of degraded frames as input and produces multi-layer intermediate features that encode degradation-aware geometric structure.
DPT-Based Feature Upsampling Heads — three separate learnable upsampling networks (query head, key head, context head) that aggregate features from selected diffusion layers and map them from the coarse latent resolution (1/16 of input) to a higher spatial resolution (1/8 of input) suitable for cost volume construction.
RAF-T-Style CNN Feature Encoder ($\mathcal{E}$) — a standard convolutional encoder (image encoder $\mathcal{E}_{\text{img}}$ plus context encoder $\mathcal{E}_{\text{ctx}}$) that processes the degraded frames directly, producing features that capture fine-grained local spatial details which the globally-aggregated diffusion features lack.
Correlation Operator ($\mathcal{C}$) — the standard RAFT all-pairs correlation layer that constructs a 4D cost volume from the hybrid (concatenated diffusion + CNN) feature maps of two consecutive frames.
Iterative Update Operator ($\mathcal{U}$) — the standard RAFT recurrent GRU-based refinement module that takes the cost volume and hybrid context features (concatenated diffusion + CNN context) to iteratively produce refined flow estimates over $M = 12$ update steps.
Information flows as follows: two consecutive degraded frames enter the system → the lifted diffusion model performs 10 denoising steps, producing query features from frame $k$ and key features from frame $k+1$ at selected layers → DPT heads upsample these features to 1/8 resolution → CNN encoder independently extracts image features from both frames and context features from the reference frame → diffusion and CNN features are concatenated channel-wise to form hybrid representations → the correlation operator builds a cost volume from the hybrid frame features → the update operator iteratively refines a flow estimate by querying the cost volume, conditioned on the hybrid context features → the final flow field after 12 iterations is the output.
3.3 Roadmap for the Deep Dive
First, the formal problem formulation (Equation 7), which defines Degradation-Aware Optical Flow and identifies the feature encoder as the critical bottleneck — this establishes what needs solving and where to intervene.
Second, the lifting procedure for transforming an image restoration diffusion model into a video-capable model (Equations 8–10), explaining how full spatio-temporal attention is injected and why this avoids the temporal latent collapse of video diffusion architectures — this is the foundation that makes all downstream feature extraction possible.
Third, the diffusion feature analysis protocol and results (Equation 12, Figure 3), which empirically identifies which layers and feature types encode the strongest geometric correspondence — this provides the evidence base for selecting features in DA-Flow.
Fourth, the DA-Flow architecture in detail (Equations 13–17), covering the DPT upsampling heads, the hybrid fusion of diffusion and CNN features, and how these components integrate with the existing RAFT correlation and update operators — this is the complete system.
Fifth, the training procedure and loss function (Equation 18), including the two-stage training, pseudo ground-truth generation, and the degradation pipeline — this explains how the system is optimized given the absence of real degraded-video flow labels.
Sixth, the key hyperparameters, dataset statistics, and implementation details that govern the training and inference of the system.
3.4 Detailed, Sentence-Based Technical Breakdown
This is primarily a systems architecture paper whose core idea is that intermediate features from a video-lifted image restoration diffusion model, when combined with standard convolutional features, provide degradation-aware representations that enable accurate optical flow estimation from severely corrupted inputs — something that neither diffusion features alone nor CNN features alone can achieve.
Problem Formulation: Degradation-Aware Optical Flow
The paper defines Degradation-Aware Optical Flow as the task of estimating dense per-pixel motion from corrupted video frames, formalized in Equation 7.
Let $\mathbf{V}_{\text{LQ}}$ and $\mathbf{V}_{\text{HQ}}$ denote a low-quality (LQ) video and its corresponding high-quality (HQ) video respectively, each represented as a sequence of $N$ RGB frames $\{\mathbf{I}^i\}_{i=1}^{N}$ with $\mathbf{I}^i \in \mathbb{R}^{3 \times H \times W}$. For a consecutive frame pair indexed by $k$ and $k+1$, where $k \in \{1, \dots, N-1\}$, the goal is to learn a model $\mathcal{M}$ that estimates flow from the degraded inputs:
fk→k+1=M(ILQk,ILQk+1)≈fk→k+1∗
where $\widehat{\mathbf{f}}_{k \to k+1}$ is the predicted 2D flow field, $\mathbf{I}_{\text{LQ}}^{k}$ and $\mathbf{I}_{\text{LQ}}^{k+1}$ are the degraded input frames, and $\mathbf{f}_{k \to k+1}^{*}$ is the ground-truth flow (which, for real degraded videos, does not exist — hence the paper's use of pseudo ground-truth from HQ frames).
What it computes: given two corrupted video frames, the model $\mathcal{M}$ outputs a dense displacement field $\widehat{\mathbf{f}} \in \mathbb{R}^{H \times W \times 2}$ where each pixel $(x, y)$ in frame $k$ is assigned a 2D vector $(\Delta x, \Delta y)$ indicating its estimated location in frame $k+1$. The goal is for this field to approximate the true motion that would be observed between the clean versions of the frames.
Why this form: the notation makes explicit that the model takes degraded frames as input — this is what distinguishes the task from standard optical flow on clean inputs. The paper argues that the feature encoder $\mathcal{E}$ within $\mathcal{M}$ is the component most vulnerable to input degradation (as formulated in Equation 1: $\mathcal{M} = \mathcal{U} \circ \mathcal{C} \circ \mathcal{E}$), because corrupted pixels lead to unreliable features that propagate errors through the correlation and update stages. The paper's solution therefore focuses entirely on building a better $\mathcal{E}$ while keeping $\mathcal{C}$ and $\mathcal{U}$ unchanged — a design choice that isolates the contribution of the new feature encoder and ensures backward compatibility with existing optical flow architectures.
Lifting an Image Restoration Diffusion Model to Video
The base model: MM-DiT and DiT4SR. The paper builds on DiT4SR, a DiT-based (Diffusion Transformer) image restoration model that uses the MM-DiT (Multi-Modal Diffusion Transformer) architecture from Esser et al. (2024). DiT4SR operates on individual frames: given a low-quality frame $\mathbf{I}_{\text{LQ}}^k$ and its high-quality counterpart $\mathbf{I}_{\text{HQ}}^k$, both are encoded into latent space via a pretrained VAE:
zLQk=Enc(ILQk),zHQk=Enc(IHQk)
where $\mathbf{z}_{\text{LQ}}^k$ is the degraded latent (used only as conditioning), $\mathbf{z}_{\text{HQ}}^k$ is the clean latent (the target of the diffusion process), and both have spatial resolution reduced by a factor of 16 from the original image.
During training, a noisy latent $\mathbf{z}_t^k$ is constructed by linearly interpolating between Gaussian noise $\boldsymbol{\epsilon} \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$ and the clean target according to a continuous noise level $t \in [0, 1]$:
ztk=(1−t)ϵ+tzHQk
What it computes: this is a rectified flow formulation where $t = 0$ corresponds to pure noise and $t = 1$ corresponds to the clean latent. The noisy latent $\mathbf{z}_t^k$ is a convex combination that provides a straight path from noise to data. The model is trained to predict the velocity field along this path — the derivative of $\mathbf{z}_t^k$ with respect to $t$:
dtdztk=zHQk−ϵ
This velocity is the ground-truth target for the denoising network $\mathcal{D}$, which predicts $\mathbf{v}_t^k$ conditioned on the degraded latent $\mathbf{z}_{\text{LQ}}^k$:
vtk=D(ztk,t∣zLQk)
The training objective is the flow-matching loss:
Ldiff=Ek,t,ϵ[vtk−(zHQk−ϵ)22]
where the expectation is over frame indices $k$, noise levels $t$, and noise samples $\boldsymbol{\epsilon}$.
Why this form: rectified flow provides a simpler, more stable training objective than standard diffusion (which predicts noise rather than velocity). The straight-line interpolation path between noise and data means the model learns a direct transport map rather than navigating a curved probability flow, which is empirically more tractable. The conditioning on $\mathbf{z}_{\text{LQ}}^k$ is what makes the model restoration-specific — the model must learn to denoise in a way that is informed by the degraded observation, which implicitly forces its internal representations to become degradation-aware.
The limitation: no temporal reasoning. In the image-level MM-DiT architecture, each frame is processed independently. If we have a batch of $B$ video clips, each with $F$ frames, the standard approach is to fold the temporal dimension into the batch axis, yielding $BF$ separate sequences of $T$ patchified tokens with channel dimension $C$. The MM-Attention mechanism processes three modality-specific token sequences per frame: $\mathbf{F}_{\text{HQ}} \in \mathbb{R}^{(BF) \times T \times C}$ (the current denoising state), $\mathbf{F}_{\text{LQ}} \in \mathbb{R}^{(BF) \times T \times C}$ (conditioning tokens from the degraded input), and $\mathbf{F}_{\text{Text}} \in \mathbb{R}^{(BF) \times T \times C}$ (text tokens encoding semantic priors). Within each block, modality-specific projections produce queries, keys, and values:
which are concatenated along the token dimension to form $\mathbf{Q}, \mathbf{K}, \mathbf{V} \in \mathbb{R}^{(BF) \times 3T \times C}$ for joint attention. However, because the temporal dimension remains folded into the batch axis, attention operates independently per frame — tokens from frame $k$ never attend to tokens from frame $k+1$.
The solution: full spatio-temporal attention. To enable inter-frame reasoning, the paper reshapes each modality stream. Instead of $\mathbf{F}_m \in \mathbb{R}^{(BF) \times T \times C}$ (treating frames as batch items), the features are reshaped to $\tilde{\mathbf{F}}_m \in \mathbb{R}^{B \times (FT) \times C}$, concatenating all spatial tokens across the $F$ frames into a single sequence per video. Modality-specific projections and concatenation then yield spatio-temporal queries, keys, and values:
Q~,K~,V~∈RB×(3FT)×C
and full spatio-temporal MM-Attention is computed as:
MM-Attn=softmax(CQ~K~⊤)V~
where each token now attends to all spatial tokens across all frames and all modalities.
What it computes: for each token (whether from the HQ stream, LQ stream, or text stream, and from any frame), the attention mechanism computes a weighted sum of all value vectors — weights determined by the dot-product similarity between the query vector of the current token and the key vectors of every token across all frames. The softmax normalization ensures the weights sum to 1, and the $\sqrt{C}$ scaling prevents the dot products from growing too large as the channel dimension increases. The output is a context-aware representation that incorporates information from all spatial locations across all frames simultaneously.
Why this form: the key design choice is that per-frame spatial structure is preserved — tokens from each frame remain separable entities in the $FT$-length sequence, with the frame index explicitly encoded by position in the concatenated sequence. This means after attention, features from frame $k$ and frame $k+1$ can still be extracted independently for downstream pairwise comparison (cost volume construction). This is in contrast to video diffusion models, which compress multiple frames into a shared latent via 3D convolutions or early temporal fusion, making per-frame features inseparable. The paper argues this preservation of per-frame spatial identity is "structurally aligned with dense correspondence estimation" because optical flow requires comparing features extracted independently from each frame.
Training the lifted model. With full spatio-temporal attention applied to all layers, the lifted diffusion model $\mathcal{D}_\phi$ is finetuned on the YouHQ training dataset (38,576 videos, average resolution 1080×1920, 32 frames per clip) for 20K steps with a batch size of 32 on 4 NVIDIA H100 GPUs and a learning rate of $5 \times 10^{-5}$. The degradation pipeline from RealBasicVSR (which applies the Real-ESRGAN degradation model at the frame level followed by video compression over the sequence) is used to synthesize low-quality frames. During this lifting training, text prompts generated by the captioner provided with the original DiT4SR model are used as conditions, enabling the model to effectively learn cross-frame correspondences. After finetuning, the model is frozen and serves as the feature encoder for DA-Flow.
Diffusion Feature Analysis: Identifying Which Layers Encode Correspondence
After lifting the diffusion model, a critical question remains: which intermediate representations should be extracted for optical flow estimation? The model has many layers, and different layers may encode different types of information. The paper conducts a systematic zero-shot analysis to identify which layers and feature types are most suitable for geometric correspondence.
Feature extraction protocol. For a consecutive frame pair $(k, k+1)$, the paper extracts query features from frame $k$ and key features from frame $k+1$ from the HQ diffusion branch of the full spatio-temporal attention layers:
Q~HQk,K~HQk+1∈RB×T×C
where $T = h \times w$ is the number of spatial tokens at the latent resolution (input resolution divided by 16) and $C$ is the channel dimension. The choice of query and key features is motivated by DiffTrack (Nam et al., 2025), which showed that query-key similarities in video diffusion attention layers encode temporal correspondences. The intuition is that attention mechanisms are inherently about matching — query vectors "look for" relevant key vectors — so the query and key projections naturally encode pairwise spatial relationships suitable for correspondence.
Zero-shot flow estimation. To assess geometric correspondence quality without any task-specific training, the paper performs direct flow estimation. For a single frame pair, the extracted features are reshaped to $\mathbb{R}^{h \times w \times C}$ (recovering spatial structure from the token sequence). A cost volume $\mathbf{C} \in \mathbb{R}^{h \times w \times h \times w}$ is constructed by computing pairwise dot-product similarity:
C(i,j)=Q~HQk(i)⋅K~HQk+1(j)
where $i$ indexes a spatial location in frame $k$ and $j$ indexes a spatial location in frame $k+1$. The dot product produces a scalar similarity score for every pair of spatial locations.
What it computes: for each pixel $i$ in the query frame $k$ and each pixel $j$ in the key frame $k+1$, the dot product evaluates how well the query vector at $i$ matches the key vector at $j$. High similarity indicates that the two locations likely correspond to the same physical point. The resulting 4D tensor $\mathbf{C}$ encodes all possible pairwise similarities between all spatial locations in the two frames.
Why dot-product similarity: dot products are the natural similarity measure for the query-key space because attention weights are computed via scaled dot products. The query and key projections are learned specifically to make dot products meaningful — a high dot product between $\tilde{\mathbf{Q}}_{\text{HQ}}^{k}(i)$ and $\tilde{\mathbf{K}}_{\text{HQ}}^{k+1}(j)$ means the model "believes" location $i$ in frame $k$ corresponds to location $j$ in frame $k+1$. Using dot products for cost volume construction therefore directly leverages the correspondence information already encoded in the attention projections, without any additional learning.
A flow field is then obtained via softargmax (a differentiable approximation of argmax that computes the expected displacement as a weighted average of candidate displacements, with weights given by the softmax of the similarity scores) and upsampled to the original image resolution $H \times W$. The predicted flow is compared against pseudo ground-truth (obtained by applying a pretrained SEA-RAFT model to the corresponding HQ frame pair) using End-Point Error (EPE), which measures the Euclidean distance between predicted and ground-truth flow vectors at each pixel.
Evaluation setup. The analysis is conducted on the YouHQ40 validation split (a subset of YouHQ), using LQ–HQ video pairs where the LQ frames are generated with the same RealBasicVSR degradation pipeline used during lifting training. Crucially, unlike prior work (DIFT, DiffTrack) that injects input images into the generation branch at a specific noise level $t$, DA-Flow's features are extracted during the iterative denoising process at multiple timesteps, and the paper analyzes correspondence quality across denoising timesteps rather than at a single predetermined noise level.
Results: which layers work best? Figure 3(a) shows the timestep-averaged EPE for the top-10 layers of two configurations: Baseline (full spatio-temporal attention applied to the pretrained model but without finetuning) and Lifting (finetuned on YouHQ with full spatio-temporal attention). The lifted model achieves consistently lower EPE across all layer ranks. The top-4 layers selected for DA-Flow are $\{3, 13, 16, 17\}$ (numbered by layer index in the transformer), chosen based on their strong zero-shot correspondence performance.
Figure 3(b) examines EPE across denoising steps for these top-4 layers. The Baseline features exhibit high sensitivity to the extraction timestep — their correspondence quality varies dramatically depending on which denoising step they are extracted from. The Lifting features, in contrast, remain stable across the entire denoising trajectory. This stability is important for a practical system because it means the exact choice of extraction timestep is not critical — the features are consistently reliable.
Why query and key features rather than alternatives? Appendix 0.B.2 compares query/key features against post-AdaNorm features (which DITF showed are effective for semantic correspondence). In the zero-shot analysis (Figure 8(a)), query and key features achieve lower EPE than post-AdaNorm features at the baseline level, and this advantage persists after lifting, particularly in the top-3 layers. When the two feature types are used to train the full flow network (Figure 8(c)), query and key features lead to better EPE across all denoising steps. The paper conjectures that "this advantage stems from the attention mechanism, which inherently encodes pairwise spatial relationships in the query and key projections, making them better suited for geometric correspondence."
Why not video restoration diffusion features? Appendix 0.B.3 extracts query and key features from FlashVSR (a video restoration diffusion model) and applies the same zero-shot analysis. The results (Figure 9) show significantly worse EPE than both the Baseline and Lifting configurations. This empirically validates the paper's architectural argument: video restoration models compress multiple frames into a shared latent, making per-frame features inseparable and destroying the spatial independence needed for pairwise correspondence. The paper concludes that "features from image diffusion models are more suitable for degradation-aware optical flow than those from video diffusion models."
DA-Flow Architecture: Hybrid Feature Encoding
Building on the lifted diffusion model and the feature analysis, DA-Flow integrates the selected diffusion features into the RAFT optical flow pipeline. The overall architecture is expressed compactly as:
Mθ=U∘C∘(Up(Dϕ),E)
where $\mathcal{D}_\phi$ is the frozen lifted diffusion model (feature extractor), $\text{Up}$ denotes the learnable upsampling stage, $\mathcal{E}$ is the standard RAFT CNN encoder, $\mathcal{C}$ is the correlation operator, and $\mathcal{U}$ is the iterative update operator. The parameters $\theta$ encompass the upsampling heads, the CNN encoder, and the update operator — everything except $\mathcal{D}_\phi$ which remains frozen.
What the equation means operationally: given two degraded frames, the system follows this sequence: (1) the frozen diffusion model $\mathcal{D}_\phi$ runs 10 denoising steps and produces query/key features from selected layers; (2) the upsampling heads map these coarse features to higher spatial resolution; (3) the CNN encoder $\mathcal{E}$ independently processes both frames; (4) diffusion and CNN features are concatenated channel-wise; (5) the correlation operator $\mathcal{C}$ builds a cost volume from the hybrid features; (6) the update operator $\mathcal{U}$ iteratively refines the flow estimate.
Why keep $\mathcal{C}$ and $\mathcal{U}$ unchanged: this is a deliberate design choice to isolate the contribution of the new feature encoder. By keeping the correlation and update stages identical to RAFT, any improvement in flow accuracy can be unambiguously attributed to the degradation-aware features rather than to architectural changes downstream. This is standard practice in feature-centric papers — it provides a clean ablation baseline.
Feature Upsampling via DPT Heads
The diffusion features produced by $\mathcal{D}_\phi$ lie on a coarse spatial grid at 1/16 of the input resolution. This is too coarse for accurate flow estimation, which requires fine-grained spatial details for precise boundary localization. The upsampling stage $\text{Up}$ addresses this resolution gap by aggregating features from the top-$L$ diffusion layers and mapping them to 1/8 resolution via DPT (Dense Prediction Transformer) heads.
The paper selects $L = 4$ layers (specifically layers $\{3, 13, 16, 17\}$) based on the zero-shot analysis in Section 4.3. Three separate DPT heads are employed, each specialized for a different role:
\mathcal{F}_{\text{Q}}^{k,\uparrow} &= \text{DPT}_{\text{Q}}\!\left(\{\tilde{\mathbf{Q}}_{\text{HQ}}^{k,l}\}_{l=1}^{L}\right) \\
\mathcal{F}_{\text{K}}^{k+1,\uparrow} &= \text{DPT}_{\text{K}}\!\left(\{\tilde{\mathbf{K}}_{\text{HQ}}^{k+1,l}\}_{l=1}^{L}\right) \\
\mathcal{F}_{\text{ctx}}^{k,\uparrow} &= \text{DPT}_{\text{ctx}}\!\left(\{\tilde{\mathbf{Q}}_{\text{HQ}}^{k,l}\}_{l=1}^{L}\right)
\end{aligned}$$
where `$l$` indexes the selected diffusion layers, `$\tilde{\mathbf{Q}}_{\text{HQ}}^{k,l}$` is the query feature from frame `$k$` at layer `$l$`, and `$\tilde{\mathbf{K}}_{\text{HQ}}^{k+1,l}$` is the key feature from frame `$k+1$` at layer `$l$`. All upsampled features share a common spatial resolution of `$H/8 \times W/8$`, with query and key features having channel dimension `$c$` (for cost volume construction) and context features having channel dimension `$c'$` (for conditioning the update operator).
**What each head does:**
- **Query head (`$\text{DPT}_{\text{Q}}$`):** takes multi-layer query features from frame `$k$` and produces a single upsampled feature map `$\mathcal{F}_{\text{Q}}^{k,\uparrow}$` that represents frame `$k$` for cost volume matching. The query features encode "what to look for" — each spatial location's representation of itself.
- **Key head (`$\text{DPT}_{\text{K}}$`):** takes multi-layer key features from frame `$k+1$` and produces a single upsampled feature map `$\mathcal{F}_{\text{K}}^{k+1,\uparrow}$` that represents frame `$k+1$` for matching. The key features encode "what to match against" — each spatial location's searchable identity.
- **Context head (`$\text{DPT}_{\text{ctx}}$`):** takes multi-layer query features from the reference frame `$k$` (same input as the query head) but produces context features `$\mathcal{F}_{\text{ctx}}^{k,\uparrow}$` for conditioning the update operator, not for cost volume construction. Context features provide per-pixel information about the reference frame that helps the update operator make better refinement decisions.
**Why three separate heads:** the query and key features encode distinct representations — queries and keys in attention mechanisms are asymmetric (queries "ask," keys "answer"). Using separate heads preserves this asymmetry rather than collapsing it into a single representation. The context head is separate because it produces a different output dimension (`$c'$`) and serves a different purpose (conditioning rather than matching). The paper's use of query features for the context head (rather than key features) is because context features describe the reference frame at each pixel, and query vectors naturally encode "what is here" information about their spatial location.
**DPT architecture details.** The DPT heads follow the VGGT codebase implementation. When fusing multi-scale features across layers, a feature pyramid is constructed with resolution scales of `$(1, 1, 2, 2)$`, meaning: at the finest scale (scale 1), two pyramid levels operate at the input resolution (1/16 in latent space, upsampled toward 1/8); at the coarser scale (scale 2), two pyramid levels operate at half resolution. This multi-scale aggregation allows the upsampling to incorporate both local detail (from fine scales) and global context (from coarse scales) from the diffusion features.
##### Hybrid Feature Encoding: Fusing Diffusion and CNN Features
While the upsampled diffusion features provide strong degradation-aware representations, they are inherently globally aggregated — the attention mechanism blends information across all spatial locations, which can blur fine-grained local detail needed for precise boundary localization. To compensate, DA-Flow incorporates the conventional RAFT CNN feature encoder `$\mathcal{E}$`, which preserves local spatial details through its convolutional architecture.
Following RAFT, `$\mathcal{E}$` consists of two sub-encoders:
$$\begin{aligned}
\mathcal{F}_{\text{img}}^{k}, \mathcal{F}_{\text{img}}^{k+1} &= \mathcal{E}_{\text{img}}(\mathbf{I}_{\text{LQ}}^{k}), \mathcal{E}_{\text{img}}(\mathbf{I}_{\text{LQ}}^{k+1}) \\
\mathcal{F}_{\text{ctx}}^{k} &= \mathcal{E}_{\text{ctx}}(\mathbf{I}_{\text{LQ}}^{k})
\end{aligned}$$
where `$\mathcal{E}_{\text{img}}$` is applied to both frames equally (weight-shared) to produce dense feature maps for cost volume construction, and `$\mathcal{E}_{\text{ctx}}$` is applied only to the reference frame `$k$` to produce context features for the update operator. All encoder features have spatial resolution `$H/8 \times W/8$`, matching the upsampled diffusion features.
The hybrid representations are formed by channel-wise concatenation:
$$\begin{aligned}
\mathcal{F}^{k} &= \text{Concat}(\mathcal{F}_{\text{img}}^{k}, \mathcal{F}_{\text{Q}}^{k,\uparrow}) \\
\mathcal{F}^{k+1} &= \text{Concat}(\mathcal{F}_{\text{img}}^{k+1}, \mathcal{F}_{\text{K}}^{k+1,\uparrow}) \\
\mathcal{F}_{\text{h-ctx}}^{k} &= \text{Concat}(\mathcal{F}_{\text{ctx}}^{k}, \mathcal{F}_{\text{ctx}}^{k,\uparrow})
\end{aligned}$$
**What this computes:** for each spatial location, the feature vector is extended by appending the diffusion feature channels to the CNN feature channels. If the CNN features have `$D_{\text{cnn}}$` channels and the diffusion features have `$c$` channels, the resulting hybrid feature has `$D_{\text{cnn}} + c$` channels. The hybrid features `$\mathcal{F}^{k}$` and `$\mathcal{F}^{k+1}$` are used to construct the correlation volume via `$\mathcal{C}$` (the standard RAFT all-pairs dot-product correlation), while `$\mathcal{F}_{\text{h-ctx}}^{k}$` provides spatial conditioning for the iterative update operator `$\mathcal{U}$`.
**Why concatenation rather than addition or attention-based fusion:** concatenation is the simplest fusion mechanism — it preserves all information from both sources without introducing additional learned parameters or potential interference. The correlation operator can then learn (implicitly, through how it weights different feature channels in the dot products) which aspects of the diffusion features versus the CNN features are most informative for matching. Addition would force the two feature types to share the same channel space and scale, potentially drowning out the more abstract diffusion features. Attention-based fusion would introduce additional complexity and parameters with unclear benefits.
**Ablation evidence for the hybrid design.** Table 5 in Appendix 0.C.3 validates the hybrid approach. Row (d) — the full DA-Flow with DPT upsampling and CNN encoder — achieves the best performance. Removing the CNN encoder (row (b)) substantially degrades EPE, confirming that "the RAFT encoder provides complementary fine-grained spatial information that the diffusion features alone lack." Using bilinear upsampling instead of DPT (row (c)) also hurts performance, though the gap is smaller. Notably, DPT upsampling without the CNN encoder (row (b) vs. (a)) does not consistently improve over bilinear — it is only when DPT operates on the richer hybrid features (with CNN information) that its multi-scale aggregation yields clear gains. This interaction effect suggests that DPT benefits from having sufficiently detailed features to aggregate across scales.
##### Iterative Refinement via the Update Operator
The correlation volume and hybrid context features are processed through the standard RAFT update operator `$\mathcal{U}$`, which produces a sequence of refined flow estimates:
$$\{\mathbf{f}_{k \to k+1}^{(i)}\}_{i=0}^{M}$$
where `$\mathbf{f}^{(i)} \in \mathbb{R}^{H \times W \times 2}$` is the flow estimate after `$i$` refinement iterations and `$M$` denotes the total number of iterations. The final flow field `$\mathbf{f}_{k \to k+1}^{(M)}$` is taken from the last iteration.
**What happens inside `$\mathcal{U}$`:** the update operator is a recurrent GRU-based module. At each iteration `$i$`, it takes: (1) the current flow estimate `$\mathbf{f}^{(i-1)}$`, (2) the correlation features obtained by looking up the cost volume at locations indicated by the current flow (warping-based correlation lookup), and (3) the hybrid context features `$\mathcal{F}_{\text{h-ctx}}^{k}$`. The GRU produces a residual flow update `$\Delta \mathbf{f}^{(i)}$` which is added to the current estimate: `$\mathbf{f}^{(i)} = \mathbf{f}^{(i-1)} + \Delta \mathbf{f}^{(i)}$`. This iterative process allows the model to progressively refine coarse initial estimates into precise final flow fields.
**Why the hybrid context features matter:** the context features `$\mathcal{F}_{\text{h-ctx}}^{k}$` provide per-pixel information about the reference frame that helps the update operator make informed refinement decisions. By including both CNN context (which captures local texture, edges, and high-frequency detail) and diffusion context (which captures degradation-aware structural priors), the update operator has rich conditioning information about what the reference frame "looks like" — information that helps it assess whether a particular flow update is geometrically plausible.
**The number of iterations `$M = 12$`:** this matches the setting used for all baseline models in the evaluation, ensuring fair comparison. The RAFT pretrained weights (raft-things.pth from FlyingThings3D) provide the initialization for the update operator and CNN encoder, though these are then finetuned during the second training stage.
---
#### Training Procedure and Loss Function
**The fundamental training challenge.** Ground-truth optical flow for real-world degraded videos does not exist — you cannot instrument the real world with motion capture to obtain per-pixel flow labels for naturally degraded footage. The paper solves this with a **pseudo ground-truth strategy**: apply a high-quality pretrained flow model to clean (HQ) versions of the videos, and use these estimates as training targets for the model that sees degraded (LQ) versions of the same frames.
**Two-stage training.** DA-Flow is trained in two sequential stages, reflecting the architecture's separation into a frozen feature extractor and a trainable flow pipeline:
- **Stage 1 (Lifting):** The diffusion model `$\mathcal{D}_\phi$` is finetuned on YouHQ with the flow-matching objective `$\mathcal{L}_{\text{diff}}$` for 20K steps with batch size 32 on 4 NVIDIA H100 GPUs, learning rate `$5 \times 10^{-5}$`. This stage uses clips of `$F = 3$` consecutive frames (since optical flow needs frame pairs, three frames provide two adjacent pairs). The degradation pipeline from RealBasicVSR (applying Real-ESRGAN degradation at the frame level followed by video compression) generates the LQ inputs. Text prompts from the DiT4SR captioner are used as conditioning.
- **Stage 2 (Flow training):** The diffusion model `$\mathcal{D}_\phi$` is frozen, and the flow estimation pipeline `$\mathcal{M}_\theta$` (DPT heads, CNN encoder, update operator) is trained with the flow loss `$\mathcal{L}_{\text{flow}}$` for 20K steps with the same batch size and learning rate. During this stage, null text prompts are used (since the degraded inputs make reliable captioning difficult).
**Pseudo ground-truth generation.** For both the feature analysis (Section 4.3) and training DA-Flow (Stage 2), pseudo ground-truth flow is obtained by applying SEA-RAFT to HQ frame pairs. For the feature analysis, SEA-RAFT uses the spring-M configuration (4 recurrent iterations) with Tartan-C-T-TSKH-spring540x960-M.pth weights. For training DA-Flow, the same configuration is used but with 20 recurrent iterations to obtain higher-quality pseudo labels — the paper invests more computation in pseudo-label quality for training than for analysis, since training quality directly depends on label quality.
**The flow loss.** DA-Flow is optimized using the standard multi-scale flow loss that applies an exponentially decaying weight to earlier iterations:
$$\mathcal{L}_{\text{flow}} = \sum_{i=1}^{M} \gamma^{M-i} \left\|\mathbf{f}_{k \to k+1}^{(i)} - \mathbf{f}^{*}_{k \to k+1}\right\|_1$$
where `$M = 12$` is the number of refinement iterations, `$\gamma$` is a weight decay factor (less than 1, so `$\gamma^{M-i}$` gives higher weight to later iterations — e.g., `$i = M$` gets weight `$\gamma^0 = 1$`, while `$i = 1$` gets weight `$\gamma^{M-1}$`), `$\mathbf{f}^{(i)}$` is the flow estimate after iteration `$i$`, and `$\mathbf{f}^{*}$` is the pseudo ground-truth flow. The `$\ell_1$` norm is used for robustness to outliers compared to `$\ell_2$`.
**What it computes:** the loss penalizes flow errors at every refinement iteration, with exponentially increasing weight on later iterations. Early iterations (when the flow estimate is coarse) are penalized less because they are expected to be less accurate; later iterations (when the estimate should be refined) receive higher weight.
**Why this form:** multi-scale losses are standard in iterative flow estimation because they provide training signal throughout the refinement process, preventing the model from ignoring early iterations entirely. The `$\ell_1$` loss is chosen over `$\ell_2$` because optical flow often has large displacements and outliers; `$\ell_1$` is less sensitive to these large errors, preventing them from dominating the gradient. The exponential weighting `$\gamma^{M-i}$` ensures the final estimate receives the most emphasis while still providing some gradient signal to intermediate iterations.
**Training data details.** The YouHQ dataset comprises 38,576 high-resolution videos (average 1080×1920, 32 frames per clip) spanning diverse scenarios: street views, human portraits, animals, and other categories. During training, input frames are randomly cropped to 512×512. The LQ versions are generated using the same degradation pipeline as RealBasicVSR (frame-level Real-ESRGAN degradation + video compression), ensuring consistency between training and evaluation.
**Inference details.** At inference time, DA-Flow performs 10 denoising steps. Input videos are processed in chunks of 3 frames (the lifted model was trained on `$F = 3$`), and the resulting features are used to estimate optical flow with 20 refinement iterations (more than the 12 used during training — a common practice in optical flow to squeeze out additional accuracy at inference). Null text prompts are used since the degraded inputs make reliable captioning difficult. The model was initialized from RAFT's raft-things.pth pretrained on FlyingThings3D, providing a strong initialization for the CNN encoder and update operator before finetuning on the YouHQ pseudo-labels.
## 4. Key Insights and Innovations
### Innovation 1: Degradation-Aware Optical Flow as a Distinct Task, Not a Robustness Problem
The paper's most fundamental intellectual move is reframing degraded-input optical flow from a **robustness problem** to an **accuracy problem** — and, in doing so, defining it as a new task with fundamentally different requirements than standard optical flow. This is not merely terminological; it changes what solutions are admissible.
Prior to this work, the dominant framing treated degradation as a distribution shift issue: models trained on clean synthetic data should be made robust to real-world corruptions through data augmentation, domain adaptation, or architectural regularization. RobustSpring (Schmalfuss et al., 2025) embodied this paradigm — it benchmarked how existing models "generalize from clean synthetic training data to a wide spectrum of real-world degradations," implicitly treating the goal as maintaining clean-input performance under corruption.
The paper argues this framing is insufficient at a conceptual level. Under severe degradation, the task is not "match pixels despite noise" — it is "recover motion information that may be entirely absent from the observable pixel values." When motion blur smears texture into uniform gray or compression artifacts remove fine detail, there is literally no visual evidence at certain locations to support matching. The problem becomes **inherently ambiguous** rather than merely difficult. The paper's exact language in Section 1 captures this distinction:
> "In such regimes, correspondence estimation is not merely a matter of distribution shift but becomes inherently ambiguous."
This reframing has immediate architectural consequences. Robustness approaches — data augmentation with synthetic corruptions, domain-invariant feature learning — attempt to make features insensitive to degradation. But the paper argues that for severe corruption, what is needed is the opposite: features that are **sensitive to degradation patterns** in order to reason about what information was likely lost and recover it. This is a fundamentally generative, rather than purely discriminative, requirement — you need a model that can "imagine" the clean structure behind the corrupted observation, which is precisely what restoration diffusion models are trained to do.
The task definition in Equation 7 formalizes this: the model takes `$\mathbf{I}_{\text{LQ}}^{k}, \mathbf{I}_{\text{LQ}}^{k+1}$` as input and must output flow approximating `$\mathbf{f}_{k \to k+1}^{*}$`, the true motion between the (unobserved) clean frames. This is a harder problem than standard optical flow because the input domain is impoverished, but it is also a different problem — one where success depends on having a model of how degradation corrupts structure, not on being invariant to it.
This contribution is **fundamental rather than incremental** because it opens a new problem space. Robustness is about preservation; degradation-aware estimation is about recovery. The implications extend beyond optical flow to any dense correspondence task (stereo matching, multi-view geometry, tracking) where inputs can be corrupted — each of these has an analogous "degradation-aware" variant that the robustness framing would miss.
The evidence for why this reframing matters is indirect but compelling: the paper's comparisons with finetuned RAFT (RAFT* in Table 4, Appendix 0.C.2) show that simply finetuning a conventional flow network on the same degraded training data — the natural robustness approach — produces substantially worse results than DA-Flow's degradation-aware features. The gap cannot be closed by more data or better augmentation; it requires a fundamentally different representational strategy.
---
### Innovation 2: Restoration Diffusion Models as Degradation-Aware Feature Extractors — Repurposing a Generative Prior for a Discriminative Task
The paper's second conceptual contribution is recognizing that **image restoration diffusion models**, by virtue of their training objective, produce intermediate features that are simultaneously degradation-aware and geometrically structured — and that these features can be repurposed for a discriminative task (optical flow) for which they were never trained.
This is a distinctive intellectual move because it inverts the typical relationship between generative and discriminative models. Normally, one designs a feature extractor specifically for the task at hand (e.g., a cost-volume-aware CNN for optical flow) and trains it discriminatively. The paper instead observes that a pretrained generative model, trained for an entirely different purpose (denoising degraded images), already encodes exactly the kind of information that degraded-input flow estimation needs: an understanding of how corruption affects image structure, combined with the geometric priors that diffusion models inherently learn.
The key insight is not that diffusion features are useful for correspondence — DIFT (Tang et al., 2023) and DiffTrack (Nam et al., 2025) already established that. The insight is that **restoration-pretrained features are qualitatively different from generation-pretrained features** for this specific purpose. A generation diffusion model (e.g., Stable Diffusion) learns to produce realistic images from noise; its features encode object structure, semantics, and geometry because those are necessary for realistic generation. But it has no explicit training signal about degradation — it never sees corrupted inputs during training, so its features have no reason to encode how blur or noise affects structure.
A restoration diffusion model (DiT4SR in this paper) is trained on a fundamentally different objective: given a degraded latent `$\mathbf{z}_{\text{LQ}}^k$` as conditioning, predict the velocity field from noise to the clean latent `$\mathbf{z}_{\text{HQ}}^k$`. To minimize the flow-matching loss (Equation 6), the model's internal representations must learn to:
1. Encode the pattern of degradation in the input (to know what needs to be restored).
2. Represent the clean structure that should exist at each location (to produce the correct velocity toward the clean target).
3. Preserve geometric relationships (to ensure the restored output is structurally consistent).
The paper's claim is that requirement (3) is an emergent property of the training objective — the model must learn geometry to restore structure correctly — and that this makes restoration features inherently more suitable for degraded-input correspondence than generation features, which learn geometry for a different reason.
This is a **fundamental** rather than incremental contribution because it establishes a new class of feature extractors: task-specific generative models repurposed as discriminative feature backbones. The paper demonstrates this for optical flow, but the principle generalizes: any task that benefits from understanding how corruption affects structure (stereo matching from degraded images, depth estimation from noisy inputs, segmentation from compressed video) could potentially leverage features from a restoration model trained for that degradation type.
The zero-shot correspondence analysis in Figure 3 provides the empirical anchor: the lifted restoration model achieves lower EPE than the untrained baseline across all layer ranks, and its features remain stable across denoising timesteps (Figure 3b) — stability that a generation-only model lacks because it has never learned to be robust to the noise level at which features are extracted. The comparison with FlashVSR features (Figure 9, Appendix 0.B.3) further supports the claim: video restoration features, despite also being degradation-aware, perform significantly worse because their architectural design (temporal latent compression) destroys the per-frame spatial structure needed for correspondence. This negative result sharpens the positive claim: it is specifically **image restoration** features, with their preserved per-frame spatial latents, that work — not just any restoration features.
---
### Innovation 3: Lifting Image Models to Video via Controlled Temporal Attention — Avoiding Latent Collapse
The paper's third conceptual contribution is an architectural insight about **how to inject temporal reasoning into a pretrained image model without destroying the properties that made it useful in the first place**. This is a design principle rather than a specific architecture, and it addresses a tension that arises whenever one tries to extend single-frame models to multi-frame tasks.
The naive approach when encountering a video task is to use a video model — in this case, a video restoration diffusion model like STAR, Dove, or FlashVSR that jointly handles degradation and temporal dynamics. The paper identifies why this fails for dense correspondence: video restoration models "encode a stack of degraded frames into a temporally compressed latent representation through 3D convolutions or temporal attention," producing "a shared latent tensor where the temporal axis is entangled early in the encoding pipeline" (Section 1). This is structurally misaligned with dense matching, which requires comparing spatially independent features from each frame.
The paper's alternative — what it calls "lifting" — is to start from a per-frame model that preserves spatial independence and add temporal reasoning in a controlled, late-stage manner: modify the attention mechanism to compute full spatio-temporal attention (Equations 9–10) while keeping per-frame token sequences as separable entities that can be extracted independently after attention. The crucial property is that **tokens from each frame remain individually addressable** — they attend to tokens from other frames, but they are not fused into a shared representation that destroys per-frame identity.
This is a **fundamental** design insight because it generalizes beyond this specific paper. Any task that requires pairwise comparison between per-frame features — optical flow, stereo matching, tracking, video correspondence — faces the same architectural choice: use a video model that fuses frames early (and lose per-frame spatial structure) or use a per-frame model with late fusion (and potentially miss important cross-frame context). The paper's lifting approach occupies a principled middle ground: cross-frame attention provides the temporal reasoning needed for motion understanding, while per-frame token separability preserves the spatial structure needed for pairwise matching.
The evidence for why this matters comes from both the zero-shot analysis and the architectural ablation. The comparison with FlashVSR (Figure 9) shows that video restoration features, despite being trained for a closely related task, are "significantly inferior" for correspondence because their compressed latent cannot be separated into per-frame features. Meanwhile, the comparison between Baseline (untrained full-attention) and Lifting (finetuned full-attention) in Figure 3 shows that the lifting finetuning dramatically improves correspondence quality — the temporal attention mechanism, when properly trained, does learn inter-frame correspondences. The combination of these results tells a clear story: temporal reasoning is necessary (Baseline performs worse), but it must be implemented without temporal latent collapse (FlashVSR performs worse than either image-based variant).
The paper is explicit about the tradeoff their design makes: by avoiding temporal latent compression, they sacrifice some of the efficiency and temporal consistency that video diffusion models achieve ("substantially more efficient than video diffusion architectures," Section 1), but they gain representations that are "intrinsically suited for dense correspondence estimation." This is a genuine tradeoff, not a free lunch — the paper chooses structural suitability for downstream matching over end-to-end video modeling elegance — and recognizing when this tradeoff is appropriate is itself part of the intellectual contribution.
---
### Innovation 4: The Hybrid CNN-Diffusion Feature Encoding — Complementary Strengths, Not Just Ensembling
The paper's fourth contribution is demonstrating that diffusion features and convolutional features provide **complementary types of information** for optical flow, and that simple channel-wise concatenation is sufficient to unlock this complementarity — you don't need complex fusion mechanisms.
This might seem like a minor architectural detail, but it encodes a substantive claim about representation learning. The paper argues that diffusion features and CNN features serve different roles that align with different stages of the flow estimation process:
- **Diffusion features** (from the lifted restoration model) provide **degradation-aware structural priors** — global, semantically-informed representations that can "see through" corruption to the underlying scene geometry. These are the features that make correspondence possible when pixel-level evidence is destroyed.
- **CNN features** (from the RAFT encoder) provide **fine-grained spatial localization** — local, high-frequency detail (edges, textures, boundaries) at the structures that survive degradation. These are the features that make correspondence precise, enabling sharp motion boundaries and accurate sub-pixel localization.
The innovation is not the observation that both are useful — that would be obvious. It is the claim that they are **complementary in a specific, task-aligned way** that maps onto the dual requirements of dense matching: robustness to corruption (need degradation-aware features) and spatial precision (need fine-grained features). Neither alone suffices; together they cover both requirements.
The ablation in Table 5 (Appendix 0.C.3) provides the key evidence. Removing the CNN encoder (row (b) vs. row (d)) substantially degrades EPE across all metrics, confirming that diffusion features alone lack the spatial precision needed for accurate flow. Conversely, the comparison with finetuned RAFT (Table 4) shows that CNN features alone, even when trained on the same degraded data, cannot match the hybrid model's performance — confirming that the degradation-aware priors from the diffusion model provide something that finetuning a CNN encoder cannot recover.
The interesting negative result in the ablation is the interaction between DPT upsampling and CNN features. DPT upsampling without the CNN encoder (row (b) vs. row (a)) does not consistently improve over bilinear interpolation — it is only when DPT operates on the richer hybrid features (with CNN information) that its multi-scale aggregation yields clear gains (row (d) vs. row (c)). This suggests that DPT's multi-scale feature fusion benefits from having access to both global (diffusion) and local (CNN) information across scales — a finding that the paper observes but does not deeply analyze, and which points toward a more nuanced understanding of how these feature types interact.
This contribution is **incremental in mechanism** (channel-wise concatenation is the simplest possible fusion) but **fundamental in implication**: it establishes that degradation-aware features and spatially-precise features are distinct representational resources that can be combined with minimal architectural complexity, and that future work on degraded-input vision tasks should consider hybrid encoders rather than trying to make a single encoder serve both purposes.
---
### Innovation 5: Pseudo Ground-Truth Training for an Unsupervisable Task — Empirical Validation of a Practical Strategy
The paper's final conceptual contribution is less glamorous than the others but equally important for practical adoption: it demonstrates that **pseudo ground-truth flow from clean video pairs is a viable training signal for learning to estimate flow from degraded inputs**, despite the obvious domain mismatch.
The problem is circular in an interesting way. To train a model that estimates flow from degraded frames, you need ground-truth flow for degraded videos — but degraded videos, by definition, are corrupted real-world footage for which instrumented ground-truth capture is impossible. You cannot simultaneously have real degradation and ground-truth motion. The paper's solution — apply a pretrained flow model to the clean version of the same video and use those estimates as training targets — introduces systematic bias (the pseudo-labels inherit the pretrained model's errors) and a domain gap (the pseudo-labels come from clean frames, not degraded ones). A skeptical reader might expect this to produce a model that simply learns to mimic the pretrained flow model rather than learning to handle degradation.
The experimental results show otherwise. DA-Flow substantially outperforms the very model (SEA-RAFT) used to generate its pseudo-labels when evaluated on degraded inputs (Table 1: SEA-RAFT achieves 20.86 EPE on Sintel vs. DA-Flow's 3.82 EPE). This means the model is not merely imitating its teacher — it is learning representations that generalize beyond the teacher's capabilities on the target (degraded) domain. The diffusion features provide enough degradation-aware information that the model can produce more accurate flow from degraded inputs than the teacher produces from clean inputs — a non-obvious and significant result.
This is an **incremental** contribution in that pseudo-labeling is a well-established technique, but it is **practically significant** because it removes what would otherwise be a fatal barrier to training: the impossibility of obtaining real ground-truth flow for degraded videos. The paper's success with this strategy establishes a template for other degradation-aware dense prediction tasks (stereo, depth, segmentation) where ground-truth labels for degraded inputs are similarly unavailable but clean-input labels or pseudo-labels exist.
A limitation the paper does not fully address: the pseudo-labels are generated by SEA-RAFT, and if SEA-RAFT has systematic failure modes on certain types of motion or scene geometry, those failure modes will propagate into DA-Flow's training signal. The paper's outperformance of SEA-RAFT on degraded inputs suggests the diffusion features help overcome some of this label noise, but the effect is not quantified or analyzed. This is a meaningful caveat for practitioners considering adopting the pseudo-labeling strategy for other tasks.
## 5. Experimental Analysis
### Evaluation Methodology
- **Dataset.** The paper evaluates on three optical flow benchmarks: **Sintel** (Butler et al., 2012), using the training set with both clean and final rendering passes, which provides dense ground-truth flow with varying levels of motion blur and atmospheric effects; **Spring** (Mehl et al., 2023), a large-scale synthetic dataset with highly detailed scenes and complex motion patterns; and **TartanAir** (Wang et al., 2020), constructed from the validation split following the setup of FlowSeek. For all three benchmarks, low-quality (LQ) inputs are generated by applying the same RealBasicVSR degradation pipeline (frame-level Real-ESRGAN degradation + video compression) used during training, ensuring consistency between training and evaluation distributions. The evaluation set sizes are not explicitly stated but are the standard benchmark splits (Sintel training set: 1,041 frames across 23 sequences; Spring: 5,000 test frames; TartanAir validation split size unspecified but standard for the benchmark).
- **Base model(s).** The core feature extractor is **DiT4SR** (Duan et al., 2025), a DiT-based image restoration diffusion model using the MM-DiT architecture (Esser et al., 2024), initialized from the publicly available `dit4sr_q` weights. The optical flow pipeline is built on **RAF-T** (Teed and Deng, 2020), initialized from `raft-things.pth` pretrained on FlyingThings3D. The paper argues DiT4SR is chosen because it is a restoration model (trained to map degraded to clean images), making its intermediate features inherently degradation-aware, and because its per-frame processing architecture avoids the temporal latent collapse of video restoration models. RAFT is chosen as the flow backbone because it is a well-established, strong baseline whose correlation and iterative refinement stages can be reused unchanged, isolating the contribution of the new feature encoder.
- **Metrics.** The primary metric is **End-Point Error (EPE)**, defined as the average Euclidean distance `$\|\mathbf{f}_{\text{pred}} - \mathbf{f}_{\text{gt}}\|_2$` over all pixels, measured in pixels. Additionally, the paper reports **outlier rates** — the percentage of pixels whose EPE exceeds thresholds of 1 pixel, 3 pixels, and 5 pixels. EPE captures average accuracy, while outlier rates measure the proportion of catastrophically wrong estimates, which is important for downstream applications where a few large errors can cause visible artifacts. For the video restoration application (Appendix 0.C.4), additional metrics include PSNR, SSIM, LPIPS (reference-based image quality), DOVER (a no-reference video quality metric), and warping error `$E_{\text{warp}}$` (measuring temporal consistency by comparing warped adjacent frames).
- **Baselines.** The paper compares against three published optical flow methods: **SEA-RAFT** (Wang et al., 2024), which improves RAFT's efficiency and robustness through a simplified update mechanism and mixed Laplace loss; **FlowSeek** (Poggi et al., 2025), which incorporates stronger priors and more efficient architectures; and implicitly **RAF-T** (Teed and Deng, 2020) as the architectural backbone. All baseline models use 12 iterative updates during evaluation, matching DA-Flow's training setting (though DA-Flow uses 20 iterations at inference — see below). Additionally, Appendix 0.C.2 compares against **RAFT\*** — a version of RAFT finetuned on the exact same YouHQ degraded training data with the same pseudo ground-truth labels, providing a controlled comparison that isolates the contribution of the diffusion features from the effects of domain-specific finetuning.
- **Generation budget / compute accounting.** The paper does not formalize a unified compute budget across methods because the comparison involves fundamentally different architectures (standard CNNs vs. diffusion models with iterative denoising). Instead, fairness is ensured through: (1) matching the number of iterative refinement steps (12 for all methods during evaluation comparison, though Table 1's exact evaluation setting is ambiguous — see critical assessment); and (2) for the ablation comparing Baseline* vs. DA-Flow (Table 2), matching the number of denoising steps. DA-Flow uses **10 denoising steps** during inference, which is a hyperparameter chosen based on the feature analysis showing stability across timesteps (Figure 3b). The computational cost of these 10 denoising steps is substantially higher than the single forward pass of conventional flow networks, a limitation the paper acknowledges (Appendix 0.E) but does not quantify in FLOPs or wall-clock time.
- **Cross-validation / statistical protocol.** No cross-validation or statistical significance testing is reported. The paper evaluates on the standard test splits of each benchmark and reports single-run numbers. The feature analysis in Section 4.3 uses the YouHQ40 validation split for layer selection, but this is a one-time analysis used to choose the `L = 4` layers for DA-Flow — there is no iterative tuning on the test benchmarks. The ablation studies (Tables 2, 4, 5) similarly report single-run results without error bars or multiple seeds. This is standard practice in the optical flow literature (the baseline papers RAFT, SEA-RAFT, and FlowSeek also report single-run results), but it means the statistical reliability of the reported improvements — particularly the smaller margins (e.g., DA-Flow's 1px outlier advantage on Spring: 8.58% vs. FlowSeek's 8.93%) — cannot be assessed.
---
### Main Quantitative Results
#### Comparison Against Existing Methods on Degraded Benchmarks
Table 1 presents the primary quantitative comparison across Sintel, Spring, and TartanAir. DA-Flow achieves the best performance across most metrics, though with an interesting exception on TartanAir EPE.
**Sintel results (Table 1, top section).** DA-Flow achieves 3.82 EPE, substantially outperforming FlowSeek (5.02 EPE) and SEA-RAFT (20.86 EPE) — a reduction of 1.20 EPE (24% relative improvement) over the strongest baseline. The outlier rates show a similar pattern: DA-Flow achieves 26.50% at 1px (vs. FlowSeek's 30.57%), 12.38% at 3px (vs. 13.60%), and 7.41% at 5px (vs. 9.57%). The gap between SEA-RAFT and the other methods is dramatic — 20.86 EPE vs. 5.02 and 3.82 — confirming the paper's claim that standard flow methods degrade severely under corruption. The improvement from FlowSeek to DA-Flow, while smaller in absolute terms, is meaningful because FlowSeek already represents a strong modern baseline that incorporates architectural improvements beyond RAFT.
**Spring results (Table 1, middle section).** DA-Flow achieves 0.85 EPE, compared to 0.95 for FlowSeek and 1.32 for SEA-RAFT — a 0.10 EPE improvement over the strongest baseline. Outlier rates are: 8.58% (1px) vs. FlowSeek's 8.93% and SEA-RAFT's 10.01%; 3.99% (3px) vs. 4.63% and 5.88%; and 2.32% (5px) vs. 2.97% and 4.35%. The improvements on Spring are smaller in magnitude than on Sintel, which the paper does not directly address — a possible explanation is that Spring's more complex scenes and larger displacements make the task harder overall, compressing the performance differences between methods. Alternatively, Spring's synthetic degradations (applied via the same pipeline) may affect Spring's rendering characteristics differently than Sintel's.
**TartanAir results (Table 1, bottom section) — a notable anomaly.** DA-Flow achieves higher EPE (5.99) than FlowSeek (4.92) — a reversal of the pattern on the other benchmarks. However, DA-Flow achieves superior outlier rates at all thresholds: 33.53% (1px) vs. FlowSeek's 36.52%, 18.48% (3px) vs. 20.51%, and 12.51% (5px) vs. 14.70%. The paper addresses this discrepancy directly in Section 5.2:
> "This discrepancy can be attributed to a small number of pixels with large displacement errors; these outlier pixels disproportionately inflate the average endpoint error. The consistently lower outlier rates suggest that DA-Flow produces more accurate estimates over the majority of pixels."
This explanation — that EPE is sensitive to a heavy-tailed error distribution while outlier rates capture typical performance — is plausible but not empirically demonstrated (e.g., no histogram of per-pixel errors is shown). A skeptical interpretation is that DA-Flow's performance on TartanAir is genuinely mixed: better on typical pixels but worse on challenging ones, which could reflect a different failure mode (e.g., diffusion features helping with local structure but occasionally producing large, structured errors in regions of extreme degradation). The paper's single-sentence explanation does not fully resolve this tension.
**The magnitude of improvement relative to the degradation gap.** The most striking result in Table 1 is not DA-Flow vs. FlowSeek, but the enormous gap between any of these methods on degraded inputs and their performance on clean inputs (which, while not directly reported in Table 1, is implied by the SEA-RAFT numbers — SEA-RAFT achieves well under 1.0 EPE on clean Sintel, vs. 20.86 on the degraded version). DA-Flow's 3.82 EPE on degraded Sintel represents a substantial recovery — bringing performance from essentially unusable (20.86 EPE) to reasonable (3.82 EPE). However, the paper does not report clean-input baselines for context, making it difficult to assess how much of the clean-to-degraded performance gap DA-Flow actually closes.
---
#### Timestep-Wise Performance Analysis
Table 3 (Appendix 0.C.1) breaks down DA-Flow's performance across individual denoising steps, from step 0 to step 9. This analysis serves two purposes: validating that performance is stable across timesteps (consistent with the zero-shot finding in Figure 3b), and showing that the final step is indeed the best (justifying using step 9 for the Table 1 numbers).
**Sintel (Table 3, top section).** EPE starts at 3.89 (step 0), fluctuates slightly (3.87–3.92 range through step 7), then reaches 3.82 at step 9. The variation across steps is small (~0.10 EPE range), confirming stability. Outlier rates follow a similar pattern: 1px starts at 26.73% (step 0) and ends at 26.50% (step 9), with minor fluctuations.
**Spring (Table 3, middle section).** EPE ranges from 0.85 to 0.87 across steps, with step 9 achieving 0.85. The narrow range reinforces the stability finding.
**TartanAir (Table 3, bottom section).** Shows the most interesting timestep pattern: EPE starts at 5.31 (step 0), rises to a peak of 6.16 (step 1), then gradually declines to 5.99 (step 9). The early-step peak suggests that very early denoising steps produce features that are actually worse for flow than later steps — consistent with the intuition that early denoising steps represent mostly noise, while later steps contain more structural information. The paper notes that "DA-Flow achieves superior performance at both the initial (step 0) and final (step 9) denoising steps" compared to prior methods at those specific timesteps, though the absolute numbers show that step 9 is still the best overall (justifying its use as the reported result).
The practical implication of this analysis is that the number of denoising steps could potentially be reduced without major accuracy loss — if step 0 achieves 3.89 EPE vs. step 9's 3.82 on Sintel, the additional 9 steps provide only a 0.07 EPE improvement for 10× the diffusion compute. The paper does not explore this tradeoff explicitly, but the data suggests it as a direction for efficiency improvements.
---
#### Comparison with Finetuned Baseline (RAFT*)
Table 4 (Appendix 0.C.2) provides a crucial controlled comparison that isolates the contribution of the diffusion features. RAFT* is RAFT finetuned on exactly the same YouHQ degraded training data with the same pseudo ground-truth labels and degradation pipeline as DA-Flow, using 12 refinement iterations. The only difference is the feature encoder: RAFT* uses the standard RAFT CNN encoder alone, while DA-Flow uses the hybrid CNN + diffusion encoder.
On Sintel (averaged over all denoising timesteps), DA-Flow achieves 3.89 EPE vs. RAFT*'s 4.88 EPE — a 20% relative improvement. At 1px: DA-Flow 26.85% vs. RAFT* 33.23%. At 3px: DA-Flow 13.18% vs. RAFT* 18.55%. At 5px: not explicitly reported but implied to follow the same trend.
This result is important because it rules out the alternative explanation that DA-Flow's improvements come merely from domain-specific finetuning on YouHQ (which both models receive). The diffusion features provide a genuine representational advantage beyond what finetuning a CNN encoder on the same data can achieve. This is one of the cleanest pieces of evidence for the paper's central claim that degradation-aware diffusion features are necessary — not just helpful — for handling severely corrupted inputs.
---
#### Ablation: Baseline* vs. DA-Flow (Lifted vs. Non-Lifted Features)
Table 2 (Section 5.4.1) compares DA-Flow against Baseline*, a variant that uses the same DA-Flow architecture but extracts features from the full-attention model **without finetuning** (the "Baseline" configuration from the zero-shot analysis in Section 4.3). Both models use their respective top-4 layers: `{3, 13, 16, 17}` for DA-Flow and `{10, 12, 16, 17}` for Baseline* (selected per-model based on zero-shot EPE). The comparison is reported across all denoising steps on Sintel and Spring.
**Sintel (Table 2, left).** DA-Flow consistently outperforms Baseline* across every denoising step. At step 0: DA-Flow 3.89 EPE vs. Baseline* 5.30 EPE (26.6% improvement). At step 9: 3.82 vs. 4.25 (10.1% improvement). The gap narrows at later steps (Baseline* improves from 5.30 to 4.25 across steps, while DA-Flow is relatively flat at ~3.82–3.92), but DA-Flow maintains a clear advantage throughout. The 1px outlier rate shows a similar pattern: DA-Flow 26.50% at step 9 vs. Baseline* 29.73%.
**Spring (Table 2, right).** The EPE comparison is closer: DA-Flow 0.85 vs. Baseline* 0.86 at step 9 — a marginal difference. At other steps, the two are essentially tied (0.85–0.87 for both). However, the 1px outlier rate shows a more consistent advantage for DA-Flow: 8.58% vs. 9.27% at step 9. The paper notes: "the two methods perform comparably in EPE, while DA-Flow maintains a consistent advantage in 1px outlier rate across all steps."
This ablation demonstrates that the lifting finetuning provides a real but not overwhelming benefit over simply using full-attention features from the pretrained model. The improvement is clear on Sintel (where the degradation may be more severe or the scenes more diverse) but marginal on Spring (where the task may be harder overall, compressing differences). This suggests that the pretrained restoration model's features already encode substantial geometric information even without temporal finetuning — the lifting provides refinement rather than enabling correspondence from scratch. This is consistent with the zero-shot analysis (Figure 3a), where the Baseline already showed non-trivial correspondence quality, and Lifting improved it by a meaningful but not transformative margin.
---
#### Ablation: Architectural Components (DPT Upsampling, CNN Encoder)
Table 5 (Appendix 0.C.3) systematically ablates two design choices: the upsampling strategy (bilinear interpolation vs. DPT-based aggregation) and the inclusion of the CNN encoder. Results are on Sintel, averaged over all denoising timesteps.
**Row (a): Bilinear upsampling, no CNN encoder.** EPE 5.52, 1px 34.40%, 3px 17.12%, 5px 10.99%.
**Row (b): DPT upsampling, no CNN encoder.** EPE 5.54, 1px 34.48%, 3px 17.05%, 5px 11.23%. Essentially tied with bilinear — DPT provides no benefit without CNN features. This is a non-obvious finding: the more sophisticated upsampling only helps when there is rich local detail to aggregate across scales.
**Row (c): Bilinear upsampling, with CNN encoder.** EPE 4.74, 1px 30.36%, 3px 14.56%, 5px 9.34%. Adding the CNN encoder improves EPE by 0.78 (14% improvement) and reduces 1px outliers by 4 percentage points. This confirms that CNN features provide complementary information even without sophisticated upsampling.
**Row (d): DPT upsampling, with CNN encoder (DA-Flow).** EPE 3.89, 1px 26.85%, 3px 13.18%, 5px 8.39%. Adding DPT to the CNN+diffusion combination improves EPE by 0.85 (18% improvement over row (c)) and reduces 1px outliers by 3.5 percentage points. The improvement from DPT is dramatically larger when CNN features are present (row (d) vs. (c)) than when they are absent (row (b) vs. (a)), confirming the interaction effect: DPT's multi-scale aggregation amplifies the benefit of having both global (diffusion) and local (CNN) information.
The total improvement from row (a) to row (d) — 1.63 EPE (29.5% reduction) — validates the hybrid architecture, with the CNN encoder contributing roughly half the gain and DPT contributing the other half (but only in the presence of CNN features). This ablation is one of the strongest pieces of evidence for the paper's claim that diffusion and CNN features are complementary: neither alone with sophisticated upsampling achieves good performance, but together they enable a substantial leap.
---
#### Feature Type Comparison: Query/Key vs. Post-AdaNorm
Figure 8 (Appendix 0.B.2) compares using query/key features (DA-Flow's choice) against post-AdaNorm features (the alternative used in DITF for semantic correspondence). The analysis has three parts:
**Figure 8(a) — Zero-shot layer ranking.** Query/key features achieve lower EPE than post-AdaNorm features for both Baseline and Lifting configurations, with the gap being largest in the top-3 layers. This justifies the choice of query/key features for zero-shot correspondence.
**Figure 8(b) — Top-4 layer EPE over denoising steps.** Query/key features show lower EPE overall, and post-AdaNorm features exhibit a "noticeable spike at the final denoising step" — their correspondence quality degrades specifically at the end of denoising, which is where DA-Flow extracts its features for flow estimation. This spike is not explained mechanistically but is a practical problem: if features degrade at the final step, using the final step (which has the most structural information overall) becomes less effective.
**Figure 8(c) — EPE after training the full flow network.** When both feature types are used to train the full DA-Flow pipeline on Sintel and evaluated across denoising steps, query/key features achieve consistently lower EPE. This is the most important comparison because it tests not just zero-shot correspondence quality but actual downstream task performance after training. The result confirms that the zero-shot advantage of query/key features translates to trained performance.
The paper conjectures that the advantage "stems from the attention mechanism, which inherently encodes pairwise spatial relationships in the query and key projections, making them better suited for geometric correspondence." This explanation is plausible — query and key vectors in attention are explicitly designed to compute pairwise affinities via dot products, so features taken from these projections naturally encode the kind of pairwise similarity structure that optical flow correlation operators exploit. Post-AdaNorm features, which normalize the attention output, may discard some of this pairwise structure.
---
#### Application: Video Restoration with DA-Flow Optical Flow
Table 6 (Appendix 0.C.4) demonstrates a downstream application: using DA-Flow's flow estimates to enforce temporal consistency in video restoration through a guidance mechanism (following MGLD's approach of warping the current restored frame toward the next frame and minimizing the L2 distance between them as a guidance loss). The evaluation is on YouHQ40 (the validation split used for feature analysis).
DA-Flow + lifted DiT4SR achieves: PSNR 27.28, SSIM 0.7813, LPIPS 0.2378, DOVER 78.42, and warping error `$E_{\text{warp}}^*$` 6.65. Compared to the baselines:
- **Frame-by-frame DiT4SR** (no temporal guidance): PSNR 27.24, SSIM 0.7764, LPIPS 0.2400, DOVER 77.66, `$E_{\text{warp}}^*$` 11.99.
- **MGLD with SEA-RAFT** (the original MGLD pipeline using SEA-RAFT for flow): PSNR 27.19, SSIM 0.7811, LPIPS 0.2380, DOVER 78.29, `$E_{\text{warp}}^*$` 6.98.
DA-Flow achieves the best results across all metrics, with the most dramatic improvement in `$E_{\text{warp}}^*$` (temporal consistency): 6.65 vs. 11.99 for frame-by-frame restoration (44% reduction) and 6.65 vs. 6.98 for MGLD with SEA-RAFT. This demonstrates that better optical flow directly translates to better temporal consistency in video restoration — a concrete downstream benefit.
The qualitative comparison in Figure 10 shows that DA-Flow-based guidance reduces flickering and maintains structural stability across frames, while the frame-by-frame baseline exhibits visible temporal inconsistency. This application result closes the loop on the paper's motivation: optical flow degradation has real downstream consequences, and DA-Flow's improved flow estimates measurably improve those downstream tasks.
---
### Ablation Studies and Robustness Checks
- **Lifting finetuning (Table 2):** Replacing the lifted (finetuned) diffusion features with features from the untrained full-attention model (Baseline*) yields consistently worse performance on Sintel (e.g., EPE 4.25 vs. 3.82 at step 9) and marginally worse on Spring (EPE tied at 0.85–0.86, but 1px outlier rate higher: 9.27% vs. 8.58%). The lifting finetuning provides a clear but moderate benefit, suggesting the pretrained restoration model already encodes substantial geometric information.
- **CNN encoder contribution (Table 5, rows (a) vs. (c) and (b) vs. (d)):** Removing the RAFT CNN encoder consistently degrades performance regardless of upsampling strategy. With bilinear upsampling: EPE increases from 4.74 to 5.52 (16% degradation). With DPT upsampling: EPE increases from 3.89 to 5.54 (42% degradation — the gap is larger because DPT amplifies the benefit of CNN features). This confirms the paper's claim that CNN features provide complementary fine-grained spatial information.
- **DPT upsampling contribution (Table 5, rows (c) vs. (d)):** Replacing DPT upsampling with bilinear interpolation (while keeping the CNN encoder) degrades EPE from 3.89 to 4.74 (22% relative increase). However, without the CNN encoder (rows (a) vs. (b)), DPT provides no benefit (5.52 vs. 5.54). This interaction effect is a non-obvious finding: DPT's multi-scale aggregation only helps when there is rich local detail (from CNN features) to aggregate across scales.
- **Feature type selection (Figure 8, Appendix 0.B.2):** Using post-AdaNorm features instead of query/key features degrades trained flow performance. Figure 8(c) shows consistently higher EPE across denoising steps for post-AdaNorm features after training. Additionally, post-AdaNorm features exhibit a spike in zero-shot EPE at the final denoising step (Figure 8b), which is precisely where DA-Flow extracts features.
- **Video restoration diffusion features (Figure 9, Appendix 0.B.3):** Extracting query/key features from FlashVSR (a video restoration model) yields "significantly inferior" zero-shot geometric correspondence compared to both the baseline and lifted image model. This negative result validates the paper's architectural argument that video diffusion models' temporal latent compression destroys per-frame spatial structure needed for correspondence. Without this ablation, the choice of an image-based rather than video-based restoration model would be an arbitrary design decision rather than a principled one.
- **Domain-specific finetuning as a confounding factor (Table 4, Appendix 0.C.2):** The comparison between DA-Flow and RAFT* (RAFT finetuned on the same YouHQ data) shows that the improvement is not merely from domain adaptation — the diffusion features provide a 1.0 EPE improvement beyond what finetuning a standard CNN encoder on the same degraded data achieves.
- **Timestep stability (Table 3, Appendix 0.C.1):** DA-Flow's performance is stable across denoising timesteps (e.g., Sintel EPE range: 3.82–3.92 across 10 steps). This robustness to the extraction timestep is a practical strength, but the paper does not ablate fewer timesteps to determine the minimum required — a natural efficiency ablation.
---
### Critical Assessment
#### Does DA-Flow genuinely enable accurate flow estimation from severely degraded inputs where existing methods fail?
The evidence is largely convincing but comes with important caveats about what "accurate" means in absolute terms.
**What the experiments demonstrate:** On degraded Sintel, DA-Flow reduces EPE from 20.86 (SEA-RAFT) and 5.02 (FlowSeek) to 3.82 — a dramatic improvement that brings performance from essentially unusable to a level that, while still well above clean-input performance (typically <1.0 EPE for modern methods on clean Sintel), represents a meaningful recovery. The qualitative results (Figures 4–6) visually confirm that DA-Flow produces flow fields that capture motion structure where baselines produce noise.
**What the experiments do not demonstrate:** The paper never reports clean-input performance of any method, making it impossible to assess how much of the clean-to-degraded gap DA-Flow closes. If clean-input RAFT achieves, say, 0.8 EPE on Sintel, then DA-Flow's 3.82 EPE on degraded inputs closes roughly 80% of the gap from SEA-RAFT (20.86 → 3.82 out of 20.86 → 0.8). But without the clean baseline, this fraction is unknown. The paper would be strengthened by including a row in Table 1 showing each method's clean-input performance alongside its degraded performance, or at minimum reporting the clean-input EPE of a reference method.
**The TartanAir anomaly complicates the narrative.** DA-Flow achieves higher EPE than FlowSeek on TartanAir (5.99 vs. 4.92) despite better outlier rates. The paper's explanation — that a few large errors inflate EPE while most pixels are better — is plausible but unsubstantiated. Without error histograms or per-pixel analysis, this remains a post-hoc rationalization. A more rigorous treatment would show the distribution of per-pixel errors or identify the specific conditions under which DA-Flow produces large errors on TartanAir. This anomaly matters because TartanAir is the most diverse benchmark of the three, and if DA-Flow's advantage is inconsistent across datasets, the claim of general degradation-aware flow capability is weakened.
#### Does the improvement come specifically from degradation-aware diffusion features, or from other confounds?
**The controlled comparison with RAFT\* (Table 4) strongly supports the claim.** Both DA-Flow and RAFT* are trained on the same data with the same pseudo-labels; the only difference is the feature encoder. DA-Flow's 1.0 EPE improvement (3.89 vs. 4.88) on Sintel cannot be attributed to domain adaptation, data quantity, or pseudo-label quality. This is the cleanest isolation of the diffusion features' contribution.
**However, the comparison with Baseline\* (Table 2) reveals that the lifting finetuning provides only a modest benefit over using the pretrained full-attention features without finetuning.** On Sintel, DA-Flow beats Baseline* by 0.43 EPE at step 9 (3.82 vs. 4.25); on Spring, the EPE difference is negligible (0.85 vs. 0.86). This suggests that the core benefit comes from the pretrained restoration model itself — its features already encode degradation-aware geometric structure — and the lifting primarily serves to add temporal reasoning for the specific case of inter-frame correspondence. The paper could make a stronger case for the necessity of lifting by showing that the lifted model's features enable better performance specifically on scenes with complex motion (where temporal reasoning matters) while performing similarly on static or near-static scenes.
**The feature type comparison (Figure 8) and layer selection analysis (Figure 3) are rigorous and well-motivated.** The paper systematically evaluates which features to use and provides evidence for each choice (query/key over post-AdaNorm, layers {3, 13, 16, 17} based on zero-shot EPE). This is stronger than many papers that make architectural choices without justification.
#### Are the baselines fair and comprehensive?
**The choice of baselines is reasonable but has gaps.** SEA-RAFT and FlowSeek represent strong, modern optical flow methods. However:
- The paper does not compare against any method specifically designed for robustness to degradation, such as methods using data augmentation with the same degradation types evaluated, or domain adaptation approaches. The RobustSpring benchmark (which the paper cites as motivation) includes several methods beyond SEA-RAFT and FlowSeek; none of these are evaluated.
- The paper does not compare against an approach that simply applies image restoration as a preprocessing step before optical flow — e.g., denoise/deblur with DiT4SR, then run FlowSeek on the restored output. This would test whether the benefit comes from the restoration model's features specifically, or from having access to restored image information in general. If a two-stage pipeline (restore → flow) achieved competitive performance, it would weaken the paper's claim that diffusion features need to be integrated into the flow pipeline itself.
**The inference-time settings are not fully matched across methods.** The paper states that "the number of iterative updates for all baseline models is set to 12, ensuring a fair comparison," but DA-Flow uses 20 refinement iterations at inference (Appendix 0.A). This asymmetry — DA-Flow gets 20 iterations while baselines get 12 — is not justified beyond noting it is "a common practice in optical flow to squeeze out additional accuracy at inference." A fairer comparison would either give baselines 20 iterations as well, or evaluate DA-Flow at 12 iterations. The paper does not report DA-Flow's performance at 12 iterations, so the magnitude of advantage from the additional 8 iterations is unknown. This is a genuine fairness concern.
**The RAFT\* baseline (Table 4) uses 12 iterations, matching the training setting, but the paper does not specify whether RAFT\* could also benefit from more iterations at inference.** If RAFT\* at 20 iterations closed some of the gap, the unique contribution of diffusion features would be smaller.
#### Are the evaluation datasets and metrics appropriate?
**The benchmark selection covers diverse scenarios.** Sintel (animated, synthetic), Spring (synthetic, complex), and TartanAir (realistic simulation, diverse environments) provide a reasonable test of generalization. However, all evaluations use the same degradation pipeline (RealBasicVSR / Real-ESRGAN) that was used during training. The paper does not evaluate on:
- Different degradation types not seen during training (e.g., different noise models, different compression standards, different blur kernels).
- Real degraded video (where the degradation distribution may differ from the synthetic pipeline).
- Varying severity levels — the paper applies one fixed degradation setting; it would be informative to see how performance degrades as corruption severity increases.
**The metrics are standard for optical flow but have known limitations.** EPE is sensitive to large errors (as the TartanAir anomaly demonstrates). Outlier rates provide complementary information but at arbitrary thresholds (1px, 3px, 5px). The paper would benefit from additional metrics like:
- Per-pixel error histograms to characterize the error distribution.
- Performance broken down by motion magnitude (small vs. large displacements), texture level (textured vs. textureless regions), and occlusion status.
- Runtime or FLOPs measurements to contextualize the accuracy-compute tradeoff.
**The test set sizes are not explicitly stated and may be small for some benchmarks.** The Sintel training set has 1,041 frames; Spring test set has 5,000; TartanAir validation split size is not specified but is likely on the order of thousands of frames. Without confidence intervals, small but consistent improvements (e.g., DA-Flow's 0.10 EPE improvement on Spring) could be within the noise of a small test set.
#### Are the ablation studies convincing?
**The architectural ablation (Table 5) is well-designed and reveals a non-obvious interaction.** The finding that DPT upsampling only helps in the presence of CNN features is genuine and informative — it suggests the two components are complementary in a specific way that would not be captured by independently evaluating them. This is exactly the kind of ablation that strengthens a paper.
**However, several important ablations are missing:**
- No ablation on the number of diffusion layers used (`L = 4`). What happens with `L = 2`, `L = 8`, or all layers? The zero-shot analysis (Figure 3a) provides some guidance, but performance after training could differ.
- No ablation on the number of frames used during lifting (`F = 3`). Would `F = 2` (just the pair being evaluated) be sufficient, or does the model benefit from seeing the preceding/following frame?
- No ablation on the number of denoising steps at inference. The timestep analysis (Table 3) shows that performance is relatively flat across steps; a 5-step or even 1-step variant might achieve nearly the same accuracy at substantially lower cost. The paper identifies this as future work (Appendix 0.E) but does not provide empirical evidence.
- No ablation on the freezing of the diffusion model during Stage 2. What happens if `$\mathcal{D}_\phi$` is finetuned jointly with the flow pipeline? The paper freezes it for practical reasons (the model is large), but jointly finetuning might improve performance or reveal that the features need task-specific adaptation.
#### Are the reported gains likely to be statistically reliable?
The paper reports single-run results without error bars, multiple seeds, or statistical tests. This is standard in optical flow (RAFT, SEA-RAFT, and FlowSeek also report single-run numbers), but it means that:
- The smallest reported improvements (e.g., 0.10 EPE on Spring, 0.35% 1px outlier improvement) may not be statistically distinguishable from noise given typical variance in optical flow training.
- The TartanAir EPE reversal (DA-Flow worse than FlowSeek by 1.07 EPE but better on outlier rates) could reflect noise rather than a genuine difference in error distribution characteristics.
- The timestep fluctuations in Table 3 (e.g., Sintel EPE varying from 3.82 to 3.92 across steps) provide a rough sense of variance, but this is within a single trained model — run-to-run variance from different random seeds could be larger.
The paper would be strengthened by reporting results averaged over multiple training runs or providing confidence intervals, particularly for the benchmarks where margins are small relative to typical optical flow training variance.
#### Summary: What has the paper actually demonstrated?
The paper has convincingly demonstrated that:
1. **Standard optical flow methods break catastrophically under severe synthetic degradation**, with SEA-RAFT achieving 20.86 EPE on degraded Sintel — a level that would be completely unusable in practice (strongly supported, multiple baselines).
2. **A hybrid encoder combining features from a restoration diffusion model with standard CNN features substantially recovers performance**, achieving 3.82 EPE on the same degraded Sintel data — a 5.5× improvement over SEA-RAFT and 1.2 EPE better than the strongest existing baseline (strongly supported, Tables 1, 4, 5).
3. **The improvement comes specifically from the diffusion features, not merely from domain-specific finetuning**, as shown by the RAFT\* comparison (strongly supported, Table 4).
4. **The diffusion features and CNN features provide complementary information**, with the full model outperforming either alone (strongly supported, Table 5 ablation).
5. **Better flow estimates improve downstream video restoration temporal consistency** (supported, Table 6, one dataset).
What the paper has **not** demonstrated:
1. **Generalization to real degraded video** — all evaluations use the same synthetic degradation pipeline as training. Performance on genuinely real-world degraded footage (with different noise characteristics, compression artifacts, and blur patterns) is unknown.
2. **Robustness to degradation types not seen during training** — the paper evaluates only on the RealBasicVSR degradation pipeline used during both lifting and flow training. There is no evidence that DA-Flow's advantages transfer to other degradation types.
3. **Whether a simpler two-stage approach (restore → flow) would work equally well** — the paper never compares against applying DiT4SR restoration followed by FlowSeek on the restored output. If this baseline performed competitively, it would undermine the claim that diffusion features need to be integrated into the flow pipeline.
4. **Statistical reliability of the smaller margins** — the 0.10 EPE improvement on Spring and the mixed TartanAir results could reflect variance rather than genuine superiority.
5. **Efficiency tradeoffs** — DA-Flow requires 10 denoising steps at inference, which is substantially more expensive than a single forward pass of conventional flow networks. The paper does not quantify this cost or evaluate whether fewer denoising steps could achieve similar accuracy.
## 6. Limitations and Trade-offs
### Synthetic Degradation Only — No Evidence of Generalization to Real-World Corrupted Video
**The assumption or constraint.** The paper trains and evaluates exclusively on synthetically degraded video generated by a single degradation pipeline — RealBasicVSR's frame-level Real-ESRGAN degradation followed by video compression. The paper never evaluates on genuinely real-world degraded footage. The authors are transparent about this scope, but the paper's title and framing ("real-world corrupted videos," "severely degraded inputs") create an expectation of real-world generalization that the experiments do not test.
**The consequence.** A practitioner considering DA-Flow for real-world deployment — e.g., processing consumer video from smartphones with unknown noise profiles, authentic motion blur, and varied compression — cannot know whether the reported improvements transfer. The synthetic pipeline covers a specific degradation distribution; real-world degradation is far more diverse and may include corruption patterns (mixed noise types, spatially varying blur, interlacing artifacts) that the model has never seen during training. There is a substantial risk that DA-Flow's advantage over conventional methods shrinks or disappears when the degradation differs from the training distribution, because the diffusion features are trained to be degradation-aware specifically for the degradation types seen during lifting and flow training — they may not generalize to unfamiliar corruption patterns.
**What evidence exists in the paper.** None. All three evaluation benchmarks (Sintel, Spring, TartanAir) use the same RealBasicVSR degradation pipeline applied to originally clean data (Section 5.1.3: "For all benchmarks, LQ inputs are generated using the same degradation pipeline employed during training to ensure consistency"). The video restoration application (Appendix 0.C.4) also evaluates on YouHQ40 with the identical degradation. The paper provides no experiment where DA-Flow is tested on real degraded video, on different degradation types not seen during training, or even on variants of the degradation pipeline with different severity levels or noise models.
**Mitigation status.** Not addressed. The paper does not discuss this limitation beyond its scope statements in Section 5.1.3 and 0.C.4. Appendix 0.E mentions runtime as the primary limitation but does not mention distributional generalization. This is the single largest gap between the paper's claims and its evidence — "real-world corrupted videos" in the abstract versus "synthetic degradations" in the evaluation — and it significantly limits the practical deployability claim.
---
### Inference Cost: 10 Sequential Denoising Steps Add Substantial Unquantified Overhead
**The assumption or constraint.** DA-Flow requires running the lifted diffusion model for 10 denoising steps at inference time (Section 5.1.1: "DA-Flow performs 10 denoising steps during inference"). Each step involves a full forward pass through the DiT-based MM-DiT architecture with full spatio-temporal attention across 3 frames. This cost is entirely absent from conventional optical flow networks, which require a single forward pass through a lightweight CNN encoder.
**The consequence.** The paper's headline accuracy improvements — for example, 3.82 EPE on degraded Sintel versus FlowSeek's 5.02 — come with an unknown but substantial increase in inference compute. A practitioner needs to know whether DA-Flow's 1.2 EPE improvement is worth potentially 10× or more the inference FLOPs and latency. This matters acutely for applications where optical flow is a preprocessing step in a larger pipeline (e.g., video restoration as in Appendix 0.C.4, or frame interpolation, or tracking), where the flow estimation cost adds to the total system cost. The paper's timestep analysis (Table 3) reveals that the improvement from 10 steps over 1 step is small — Sintel EPE is 3.89 at step 0 and 3.82 at step 9, a mere 0.07 EPE improvement for 10× the diffusion compute — hinting that the cost could be dramatically reduced, but the paper does not explore this efficiency tradeoff.
**What evidence exists in the paper.** The timestep-wise results in Table 3 (Appendix 0.C.1) implicitly demonstrate the diminishing returns: on Sintel, EPE ranges from 3.82 to 3.92 across 10 steps, with step 0 already at 3.89. On Spring, EPE ranges from 0.85 to 0.87 across steps. On TartanAir, there is actually a degradation at early steps (EPE 5.31 at step 0, rising to 6.16 at step 1) before recovering to 5.99 at step 9. This paper does not report FLOPs, parameter counts, wall-clock inference time, or GPU memory consumption for DA-Flow versus baselines. There is no ablation on the number of denoising steps (e.g., performance at 1, 3, 5 steps), which would directly quantify the accuracy-efficiency tradeoff.
**Mitigation status.** Acknowledged but not measured. Appendix 0.E states: "Our approach leverages features from an image restoration diffusion model via lifting, which inherently requires multiple denoising steps at inference time, resulting in slower runtime compared to conventional flow estimation networks." The paper suggests one-step distillation as future work but provides no empirical exploration of efficiency — not even a single FLOPs or latency measurement. This makes the limitation transparent but leaves the practitioner with no quantitative basis for deciding whether the accuracy gain justifies the compute cost in their specific deployment scenario.
---
### Pseudo Ground-Truth Training: Label Noise From Teacher Model and Domain Mismatch
**The assumption or constraint.** DA-Flow is trained using pseudo ground-truth flow generated by applying SEA-RAFT to high-quality (HQ) video frames, while the model receives low-quality (LQ) frames as input (Section 4.4.3). This introduces two sources of noise into the training signal: (1) SEA-RAFT's own estimation errors on clean frames become training labels — the model is taught to replicate a teacher that makes mistakes; and (2) the pseudo-labels are computed from clean frames while the model sees degraded frames, meaning the target flow field may not perfectly correspond to the motion that is actually recoverable from the degraded observations.
**The consequence.** The training objective `$\mathcal{L}_{\text{flow}}$` minimizes the difference between DA-Flow's predictions and SEA-RAFT's outputs on HQ frames, not the true ground-truth flow. If SEA-RAFT has systematic failure modes — e.g., on textureless regions, thin structures, or large displacements — those failures are baked into DA-Flow's training targets. The model may learn to reproduce the teacher's biases rather than learning degradation-robust correspondence per se. Furthermore, some motion that is recoverable from clean frames may be genuinely irrecoverable from severely degraded frames; forcing the model to predict that motion may encourage hallucination rather than principled uncertainty. The domain mismatch between pseudo-labels (from clean frames) and input domain (degraded frames) creates a fundamental tension that the `$\ell_1$` loss does not resolve.
**What evidence exists in the paper.** The paper demonstrates that DA-Flow substantially outperforms its teacher (SEA-RAFT) on degraded inputs — SEA-RAFT achieves 20.86 EPE on degraded Sintel vs. DA-Flow's 3.82 (Table 1) — showing that the model does not simply mimic the teacher's errors. This is a strong positive signal: the diffusion features provide enough degradation-aware information to produce more accurate flow from degraded inputs than the teacher produces from clean inputs. However, the paper does not analyze which pixels improve and which do not relative to the teacher, nor does it compare DA-Flow against true ground-truth on a benchmark where both clean and degraded ground-truth exist (Sintel and Spring have clean ground-truth — the paper could have evaluated DA-Flow's pseudo-label-trained model against the real ground-truth on clean inputs to measure the label bias, but does not).
**Mitigation status.** Partially addressed through architecture rather than through loss design or label cleaning. The diffusion features' degradation awareness allows the model to overcome some of the pseudo-label noise, as evidenced by outperforming SEA-RAFT. However, there is no explicit mechanism to account for label noise — no confidence weighting, no uncertainty estimation, no comparison of DA-Flow's pseudo-label-trained performance against what it could achieve with oracle ground-truth on synthetic benchmarks where such ground-truth exists. The paper acknowledges the pseudo-labeling approach is a necessity ("obtaining ground-truth optical flow for real-world degraded videos is infeasible") but does not systematically quantify its impact on the upper bound of achievable performance.
---
### Federated Evaluation Does Not Match Inference: DA-Flow Uses 20 Refinement Iterations While Baselines Use 12
**The assumption or constraint.** The paper states that "the number of iterative updates for all baseline models is set to 12, ensuring a fair comparison" (Section 5.1.3). However, DA-Flow uses 20 refinement iterations at inference time (Appendix 0.A: "the resulting features are used to estimate optical flow with 20 refinement iterations"). This is an 8-iteration advantage for DA-Flow that is not justified beyond noting it as "a common practice in optical flow to squeeze out additional accuracy at inference."
**The consequence.** The additional 8 refinement iterations give DA-Flow access to more computation at inference than the baselines, making the comparison in Table 1 not strictly fair. RAFT-based methods are known to improve with more iterations (the RAFT paper itself shows continued improvement beyond 12 iterations on clean benchmarks). If the baselines — particularly FlowSeek, which uses a similar iterative refinement architecture — were allowed 20 iterations, they would likely achieve better numbers, potentially reducing or eliminating DA-Flow's reported advantage on the closer benchmarks (Spring, where the margin is only 0.10 EPE, and TartanAir, where DA-Flow is actually worse on EPE). The paper never reports DA-Flow's performance at 12 iterations or baselines' performance at 20 iterations, so the magnitude of the unfairness cannot be quantified.
**What evidence exists in the paper.** None directly. The paper reports DA-Flow's inference setting in Appendix 0.A but never compares 12-iteration DA-Flow against 12-iteration baselines, nor does it justify why DA-Flow gets more iterations. The RAFT\* finetuning baseline (Table 4, Appendix 0.C.2) reports results "averaged over all denoising timesteps" and does not specify the number of refinement iterations used — it may also use only 12 (as per the standard setting), making it a partially controlled but still asymmetric comparison. The qualitative results (Figures 4–6) are generated by DA-Flow with 20 iterations compared against baselines with 12, meaning the visual improvements may reflect the iteration advantage at least as much as the feature advantage.
**Mitigation status.** Not addressed. The iteration discrepancy is buried in Appendix 0.A without discussion or justification. This is the most straightforward fairness issue in the paper's evaluation, and it undermines confidence in the reported margins — particularly on Spring and TartanAir where the gaps are small or reversed. A simple experiment evaluating all methods at 12 and 20 iterations on at least one benchmark would resolve this concern entirely, and the absence of such an experiment is a notable gap in an otherwise thorough evaluation.
---
### No Combination of Restoration-as-Preprocessing with Standard Flow — Missing an Obvious Baseline
**The assumption or constraint.** The paper's central architectural claim is that diffusion features must be integrated into the optical flow pipeline rather than used as a separate preprocessing step. However, the paper never evaluates a straightforward two-stage baseline: apply DiT4SR (the same restoration model used for feature extraction) to restore the degraded frames, then run a standard flow method (FlowSeek or SEA-RAFT) on the restored output. This baseline would use the same restoration model but in its intended role (image restoration) rather than as a feature extractor — and would test whether the benefit comes from having access to restored image information, or specifically from having degradation-aware features inside the flow pipeline.
**The consequence.** If the two-stage pipeline (restore then estimate flow) performed comparably to DA-Flow, it would simplify deployment substantially — practitioners could use the restoration model as a drop-in preprocessor without modifying the flow network architecture. This baseline would also disentangle two possible explanations for DA-Flow's improvement: (1) the diffusion features directly encode geometric correspondence that helps the correlation operator, or (2) having access to higher-quality image information (through the restoration model's conditioning path) is sufficient, and any method that sees restored information would similarly improve. Without this baseline, the paper cannot claim that feature integration is necessary — only that it works.
**What evidence exists in the paper.** The video restoration application (Appendix 0.C.4) provides partial but indirect evidence. Table 6 reports frame-by-frame DiT4SR restoration as a baseline, and DA-Flow-based temporal guidance improves warping error from 11.99 to 6.65. However, this is a different evaluation — it measures video restoration quality, not flow accuracy — and it does not compare flow accuracy of DA-Flow against FlowSeek applied to DiT4SR-restored frames. The paper never constructs the direct two-stage baseline for its primary task (optical flow estimation on degraded benchmarks).
**Mitigation status.** Not addressed. This is a missing experiment rather than an acknowledged limitation. The paper compares against methods that see only degraded inputs (SEA-RAFT, FlowSeek), and against RAFT finetuned on the same degraded data (RAFT\*), but never against methods that see restored inputs. Given that the paper already has a trained DiT4SR model and evaluates on datasets where restoration would be straightforward, this baseline would be easy to implement and would substantially strengthen (or potentially weaken) the case for DA-Flow's architectural integration over simpler alternatives.
---
### Frozen Diffusion Model — No Exploration of Joint Finetuning or Task-Specific Adaptation
**The assumption or constraint.** DA-Flow freezes the lifted diffusion model `$\mathcal{D}_\phi$` during the second training stage (flow training) and trains only the upsampling heads, CNN encoder, and update operator (Section 5.1.1: "In the second stage, `$\mathcal{D}_\phi$` is frozen, and the flow estimation pipeline `$\mathcal{M}_\theta$` is trained"). This means the diffusion features are extracted from a model optimized for restoration, not for optical flow, and cannot adapt to the specific requirements of the flow task.
**The consequence.** The frozen model may produce features that are suboptimal for flow estimation — for example, layers that are informative for restoration (e.g., texture synthesis) but irrelevant for correspondence may still contribute their features, adding noise to the upsampling heads. Conversely, the model cannot learn to emphasize geometric correspondence at the expense of restoration quality during feature extraction. Joint finetuning of `$\mathcal{D}_\phi$` with the flow loss `$\mathcal{L}_{\text{flow}}$` could potentially improve performance by adapting the feature representations to the downstream task, at the cost of additional training complexity and the risk of catastrophic forgetting of the degradation-aware priors. The paper provides no evidence about whether freezing was necessary or merely convenient, leaving open the question of whether an end-to-end trained version would perform better.
**What evidence exists in the paper.** None. There is no ablation comparing frozen versus finetuned `$\mathcal{D}_\phi$` during Stage 2. The feature analysis in Section 4.3 evaluates zero-shot correspondence quality of the frozen features, but this does not address whether task-specific finetuning would improve the features further. The paper's design choice to freeze the diffusion model is practical (the model is large and finetuning it jointly with the flow pipeline would increase memory requirements and training time), but is presented without empirical justification.
**Mitigation status.** Implicitly acknowledged through the design choice but not discussed as a limitation. The freezing is a practical engineering decision rather than a principled one, and while the results show that frozen features work well, the paper cannot rule out that jointly finetuned features would work substantially better — meaning the reported performance may be a lower bound on what is achievable with the same architecture. This limitation is less severe than the others because frozen features already work well, but it matters for practitioners considering whether to invest in the engineering effort of joint training for potentially higher accuracy.
## 7. Implications and Future Directions
### How This Work Changes the Landscape
This paper creates a new task category — **degradation-aware dense correspondence** — that sits between robustness research (which asks "how well does our clean-input model survive corruption?") and restoration research (which asks "can we make the video look good?"). Neither of those established communities has directly tackled the question this paper makes central: can we estimate accurate geometric relationships *from* severely corrupted inputs, not by restoring the inputs first and then applying standard methods, but by building representations that natively understand what degradation does to structure?
This is a **reframing with methodological novelty**, not a paradigm shift. The paradigm of feature extraction from pretrained diffusion models was already established by DIFT, DiffTrack, and Tale. The paradigm of optical flow via iterative refinement was already established by RAFT. What the paper contributes is a specific integration — restoration diffusion as a feature backbone for degraded-input flow — and, more importantly, a **demonstration that the integration works substantially better than existing alternatives** on a task that nobody was previously studying as a distinct problem.
The reframing has immediate consequences for how researchers think about two separate lines of work. First, for the optical flow community: the paper's catastrophic degradation results for SEA-RAFT (20.86 EPE on degraded Sintel vs. presumably <1.0 on clean) establish that standard flow methods are not just "somewhat less accurate" on corrupted inputs — they are **non-functional**. This shifts the conversation from "let's make flow models more robust" (gradual improvement) to "we need fundamentally different representations for corrupted inputs" (architectural rethink). Second, for the diffusion feature community: the paper's comparison between query/key features and post-AdaNorm features (Figure 8), and between lifted image restoration features and video restoration features (Figure 9), provides concrete guidance on *which* diffusion features to use for correspondence tasks — a question that prior work left largely to empirical trial-and-error. The finding that post-AdaNorm features degrade at the final denoising step (Figure 8b) is a specific, actionable warning for anyone building correspondence systems on diffusion backbones.
The paper also provides an implicit reconciliation of a tension in the video processing literature. Video restoration models (STAR, Dove, FlashVSR) are designed to produce temporally coherent output, which might seem ideal for optical flow — if frames are temporally consistent, should not matching be easier? The paper's architectural analysis (Section 1) and FlashVSR experiment (Figure 9) show why this intuition is wrong: temporal latent compression destroys the per-frame spatial independence that pairwise matching requires. This explains why video restoration models, despite their obvious relevance, have not been repurposed for correspondence tasks the way image diffusion models have been — and it provides a concrete design principle (preserve per-frame token separability) for future architectures that aim to serve both purposes.
The landscape change is modest in scope but precise in implication: **restoration diffusion features are a new class of representation for degraded-input vision tasks**, and the paper establishes this for optical flow specifically, with evidence that neither conventional CNN features nor alternative diffusion feature types achieve comparable performance on this task. The open question — how broadly this finding generalizes to other correspondence tasks and real-world degradation — determines whether this reframing remains a specialized contribution or becomes the starting point for a broader research program.
### Follow-Up Research This Work Enables
**Real-world degraded video evaluation with varying degradation types and severities.** The paper's most urgent gap: all evaluations use a single synthetic degradation pipeline (RealBasicVSR with Real-ESRGAN frame-level degradation + video compression). A strong follow-up would construct a benchmark of real degraded video — e.g., paired cell-phone footage of the same scene captured simultaneously with a high-quality camera, or naturally degraded archival footage with independently estimated flow ground-truth from structure-from-motion on corresponding clean footage — and evaluate DA-Flow against baselines on this data. The paper's qualitative results (Figures 4–6) suggest the approach should generalize, but the lack of real-video evaluation leaves the deployment claim unvalidated. A negative result (DA-Flow underperforming on real degradation) would reveal that the synthetic training pipeline overfits to specific corruption statistics, which would redirect effort toward domain randomization or unsupervised adaptation rather than architectural refinement.
**Two-stage baseline: restore-then-estimate-flow using the same DiT4SR model.** The paper never evaluates whether applying DiT4SR restoration as a preprocessing step, followed by FlowSeek or SEA-RAFT on the restored output, achieves comparable flow accuracy to DA-Flow's integrated approach. This is a one-experiment ablation that would strongly affect the paper's architectural claims. If the two-stage pipeline performs competitively — say, achieving 4.5 EPE on degraded Sintel vs. DA-Flow's 3.82 — it would suggest that restoration quality, not feature integration, is the primary driver of improvement, and that practitioners can deploy off-the-shelf restoration and flow models without modifying either. If it performs poorly (e.g., 10+ EPE), it would confirm that restoration artifacts introduced by the VAE decoder (which the diffusion features bypass, since they are extracted from latent space) or the specific feature-level degradation awareness that DA-Flow exploits are essential. The experiment requires no new training — DiT4SR and FlowSeek are both publicly available — and could be completed in days.
**Efficiency ablations: how few denoising steps are sufficient, and can distillation close the gap?** The timestep analysis (Table 3) reveals that DA-Flow's performance is remarkably flat across denoising steps: Sintel EPE is 3.89 at step 0 and 3.82 at step 9 — a 0.07 EPE improvement for 10× the diffusion compute. A natural ablation would sweep the number of denoising steps at inference (1, 2, 3, 5, 10) on all three benchmarks and measure both accuracy and wall-clock time. If 1-step performance (3.89 EPE) is acceptable relative to the full 10-step model (3.82 EPE), the practical deployment cost drops dramatically. Going further, the paper's Appendix 0.E explicitly suggests one-step distillation, which would involve training a student model to directly predict the diffusion features that the 10-step teacher produces — a standard technique in the diffusion literature (progressive distillation, consistency models) that has not been applied to feature extraction for correspondence. A strong follow-up would report: (a) accuracy vs. denoising steps for DA-Flow, (b) accuracy vs. latency for DA-Flow vs. baselines, and (c) accuracy of a distilled single-step feature extractor vs. the 10-step teacher.
**Joint finetuning of the diffusion backbone with the flow loss.** The paper freezes `$\mathcal{D}_\phi$` during Stage 2, training only the upsampling heads, CNN encoder, and update operator. This is a practical choice — the diffusion model is large, and joint training would increase memory requirements — but it leaves open the question of whether end-to-end finetuning with `$\mathcal{L}_{\text{flow}}$` would improve the features for the specific demands of optical flow (e.g., sharper motion boundaries, better handling of occlusions). A follow-up would finetune `$\mathcal{D}_\phi$` jointly with the flow pipeline, either from the lifting checkpoint (warm start) or from scratch (cold start), and compare against frozen DA-Flow on all three benchmarks. A positive result would suggest that restoration diffusion features can be task-adapted — opening the door to diffusion features as general-purpose backbones for multiple dense prediction tasks (flow, depth, segmentation) with task-specific finetuning. A negative result (joint finetuning hurts, perhaps due to catastrophic forgetting of degradation-aware priors) would suggest that restoration and correspondence require incompatible feature properties, which would be an important finding for the diffusion feature community.
**Generalization to other dense correspondence tasks: stereo matching and multi-view geometry from degraded images.** The paper's framework is task-agnostic in principle — any dense correspondence problem that suffers from input degradation could benefit from restoration diffusion features. Stereo matching is the most natural extension: it has the same input structure (pairs of images), the same output structure (dense displacement fields), and suffers from the same degradation vulnerabilities (textureless regions from noise/blur, edge smearing from compression). A follow-up would replace the optical flow pipeline (correlation + update operator for temporal frames) with a stereo matching pipeline (cost volume construction along epipolar lines + disparity refinement) and train/evaluate on degraded versions of standard stereo benchmarks (SceneFlow, KITTI, Middlebury). The key question is whether the degradation awareness learned by an image restoration model transfers across different geometric tasks — if DA-Stereo works, it establishes restoration diffusion features as a general-purpose representation for degraded-input geometric vision. If it fails, it reveals that the temporal finetuning (lifting) is specific to motion patterns and does not transfer to static multi-view geometry.
**Application to self-supervised or semi-supervised learning from unlabeled degraded video.** The pseudo ground-truth strategy (using SEA-RAFT on HQ frames to train DA-Flow on LQ frames) requires paired HQ-LQ video, which limits training data to synthetically degraded footage. A more ambitious direction would be to train DA-Flow in a self-supervised manner directly on real degraded video, using photometric consistency losses (warp frame `$k+1$` to frame `$k$` using the predicted flow and compare pixel intensities) that do not require ground-truth flow. The challenge is that photometric losses assume brightness constancy, which is violated under real degradation (noise, compression, lighting changes). However, the diffusion features themselves could provide a more robust matching signal — rather than comparing raw pixel intensities, compare diffusion features between warped frames, which are trained to be invariant to degradation. A successful self-supervised DA-Flow would eliminate the HQ-LQ data requirement entirely, enabling training on in-the-wild degraded video (YouTube, archival footage, medical imaging) and potentially closing the generalization gap discussed above.
### Practical Applications and Downstream Use Cases
**Video restoration with motion-compensated temporal processing.** Appendix 0.C.4 already demonstrates this: using DA-Flow's flow estimates for warping-based temporal consistency guidance improves video restoration warping error by 44% (from 11.99 to 6.65 `$E_{\text{warp}}^*$`) over frame-by-frame DiT4SR, and outperforms the same guidance mechanism using SEA-RAFT flow (6.65 vs. 6.98). The practical scenario is clear: any video restoration pipeline that currently uses off-the-shelf optical flow for temporal alignment (a common pattern — MGLD, BasicVSR++, and others) can improve temporal consistency by substituting DA-Flow for the flow estimator, particularly when the input video is heavily degraded (which is precisely when restoration is needed). The accuracy gain is modest in absolute terms (0.33 `$E_{\text{warp}}^*$` improvement over SEA-RAFT) but comes at the cost of additional inference compute — the practical decision hinges on the efficiency tradeoff that the paper does not quantify. For offline, quality-critical restoration (archival footage, film remastering), the accuracy improvement likely justifies the cost. For real-time applications (video calls, live streaming), it almost certainly does not.
**Autonomous driving and robotics perception under adverse conditions.** Optical flow is a fundamental input to numerous robotics pipelines: obstacle detection, ego-motion estimation, tracking, and scene flow for 3D reconstruction. These systems routinely encounter degraded imagery — rain, snow, lens dirt, motion blur, low-light sensor noise, compression artifacts from wireless transmission. The paper's demonstration that DA-Flow achieves 3.82 EPE on degraded Sintel versus 20.86 for SEA-RAFT (a 5.5× improvement) suggests that degradation-aware flow could substantially improve the reliability of downstream perception under adverse conditions where current methods fail entirely. The caveat is that Sintel's synthetic degradations may not match real adverse-weather degradation patterns; a domain-specific evaluation on degraded driving datasets (e.g., nuScenes with added weather corruption, or the SeeingThroughFog dataset) would be needed to validate the transfer. If the diffusion features generalize, DA-Flow could serve as a drop-in replacement for the flow estimator in existing autonomy stacks, improving robustness without architectural changes to downstream modules.
**Frame interpolation and video compression from low-quality source material.** Frame interpolation — synthesizing intermediate frames between existing ones — relies fundamentally on accurate optical flow for motion-compensated interpolation. When source material is degraded (compressed, noisy, low-resolution), flow errors produce visible interpolation artifacts (ghosting, warping, flickering). DA-Flow's improved flow accuracy on degraded inputs (e.g., 1px outlier rate of 26.50% vs. FlowSeek's 30.57% on Sintel) would directly reduce these artifacts. Video compression similarly benefits from accurate flow for motion-compensated prediction — if the encoder can better predict frame `$k+1$` from frame `$k$` using improved flow estimates, it can achieve higher compression ratios at the same quality. This is a more speculative application (the paper does not evaluate interpolation or compression), but the flow accuracy improvements on degraded benchmarks make it a natural extension for practitioners working on video processing pipelines that consume real-world, non-pristine footage.
Dependency tree
Papers this one builds on, walking backwards through the same domain and shared topics.
