Fast-FoundationStereo: Real-Time Zero-Shot Stereo Matching

🎯 Pitch

This paper demonstrates that strong zero-shot stereo generalization can run at real-time speeds—over 10× faster than prior foundation models with minimal accuracy loss. The key insight is a divide-and-conquer acceleration pipeline combining distillation, automated architecture search, and structured pruning, which together compress a computationally prohibitive model to 49ms per frame without sacrificing in-the-wild robustness.

1. Executive Summary

This paper proposes Fast-FoundationStereo, a family of stereo matching architectures that achieve, for the first time, strong zero-shot generalization at real-time frame rates. Building on the computationally prohibitive FoundationStereo model and evaluating across Middlebury, ETH3D, KITTI 2012/2015, and Booster datasets, the authors employ a divide-and-conquer acceleration strategy with three named components—knowledge distillation to compress the hybrid monocular-stereo feature backbone into a single efficient student (replacing DepthAnything V2 plus a side-tuning CNN with a single lightweight backbone), blockwise neural architecture search for automatically discovering optimal cost filtering designs under latency budgets (solving a combinatorial ILP over independently distilled block candidates to avoid O(n^N) search complexity), and structured pruning guided by a recurrent dependency graph to eliminate redundancy in the iterative ConvGRU refinement module—complemented by an automatic pseudo-labeling pipeline that curates 1.4M in-the-wild stereo pairs for supplemental training. The resulting model runs over 10× faster than FoundationStereo (496 ms reduced to 49 ms on an NVIDIA 3090) while closely matching its zero-shot accuracy (e.g., Middlebury-H BP-2 error of 4.80% vs. 2.49%), establishing a new state-of-the-art among real-time methods, with gains that transfer to strong generalization only when the distillation, search, and pruning strategies operate in concert on problems within the foundation model's capability range.

2. Context and Motivation

The Core Problem: Zero-Shot Generalization and Real-Time Inference Are Mutually Exclusive

The fundamental gap this paper addresses emerges from a fundamental tension in modern stereo matching: models that generalize well to unseen domains without per-dataset fine-tuning are far too slow for real-time applications, while models fast enough for real-time deployment break catastrophically when applied to data outside their training distribution. This split, which the paper frames as two distinct research paths (Section 1, paragraph 3), has created a field where practitioners must choose between robustness and speed—with no existing method delivering both simultaneously.

This matters for several concrete reasons the paper highlights:

Embodied agents operating in the wild. Robots, drones, and AR headsets encounter diverse visual environments (indoor offices, outdoor forests, driving scenes, cluttered warehouses) that cannot all be represented in any single training dataset. A stereo system that requires per-domain fine-tuning is fundamentally incompatible with these applications, because the agent cannot pause to collect ground-truth depth data and retrain its model every time it enters a new environment. Yet these same applications demand low latency—a drone navigating at 30 frames per second cannot wait 400–500 milliseconds per frame for depth estimation, which is the runtime of FoundationStereo and its contemporaries.
The difficulty of obtaining ground-truth depth at scale. The paper explicitly notes (Introduction) that "the difficulty of obtaining the required dense, high-quality ground-truth depth at scale has prevented such efficient methods from being used as an off-the-shelf solution." Unlike object classification or segmentation, where human annotation is relatively straightforward, obtaining metric-accurate depth for every pixel in a stereo pair requires specialized hardware (structured light scanners, LIDAR, or carefully calibrated multi-view setups). This makes it economically infeasible to train per-domain models for every deployment scenario, which is why zero-shot generalization—the ability to deploy a model on domains never seen during training—is not a luxury but a necessity for real-world systems.
The computational mismatch between foundation models and edge hardware. The rise of vision foundation models (VFMs) such as DepthAnything V2, DINOv2, and SAM has revolutionized zero-shot performance across computer vision tasks, including stereo matching. However, these models are architecturally heavyweight: they employ Vision Transformers (ViTs) with self-attention over large feature maps, hybrid backbone designs that run multiple feature extractors in parallel, and iterative refinement modules that process high-dimensional features through dozens of recurrent steps. Deploying such architectures on the embedded GPUs or mobile processors typical of robotics and AR platforms is simply infeasible without significant compression.

Conflicting Prior Evidence

The paper situates itself within a literature where excellence on one axis reliably comes at the expense of the other. This is not an accidental gap—it reflects fundamental architectural and training data constraints:

Generalizable models are uniformly slow. FoundationStereo (Wen et al., 2025) achieves state-of-the-art zero-shot generalization by combining three computationally intensive components: (1) a hybrid feature backbone that runs both DepthAnything V2 (a large ViT-based monocular depth model trained on internet-scale data) and a side-tuning CNN simultaneously, extracting features from both streams; (2) a cost filtering module that employs both a 3D hourglass network with Axial-Planar Convolution (APC) layers and a Disparity Transformer branch performing multi-head self-attention over the 4D cost volume—effectively running two expensive filtering pathways in parallel; and (3) a ConvGRU refinement module that iteratively updates the disparity prediction over many recurrent steps. The result: 496 ms per frame on an NVIDIA 3090 (Table 1). Other generalizable methods fare no better—StereoAnywhere at 427 ms, MonSter at 336 ms, DEFOM-Stereo at 371 ms. None approach the ~30 ms threshold typical of real-time systems.

Real-time models generalize poorly. On the other side, architectures designed for speed—IINet, LightStereo, RT-IGEV, BANet—achieve frame rates in the 25–50 ms range by relying on lightweight CNN backbones, 2D convolutions, and local iterative refinement. But their zero-shot performance collapses on unseen domains. Table 1 makes this strikingly clear: LightStereo-L achieves Middlebury-H BP-2 error of 37.49% (versus FoundationStereo's 2.49%), BANet-3D reaches 44.90%, and IINet hits 25.88%. These errors are so large as to be unusable for any application requiring accurate depth—a robot relying on such estimates would routinely misjudge obstacle distances by enormous margins. The paper notes (Section 1) that these methods are "usually not targeted for zero-shot generalization" and are "mainly trained on SceneFlow," a synthetic dataset that, despite its size, cannot capture the visual diversity of real-world environments. When the authors retrain these methods on larger combined datasets including their pseudo-labels (indicated by † in Table 1), performance improves—RT-IGEV drops from 16.95% to 8.69% BP-2 on Middlebury-H—but remains far below generalizable models, confirming that the limitation is architectural as much as it is data-driven.

Where Existing Approaches Fall Short

The paper identifies specific technical bottlenecks that prevent straightforward solutions:

Hybrid backbones are fast only if both streams are fast. FoundationStereo's use of DepthAnything V2 plus a side-tuning CNN is powerful because it combines monocular priors (what does the world look like in 3D, learned from millions of single images?) with stereo priors (how do left and right views relate, learned from binocular data?). But running a ViT-based monocular depth model alongside a CNN extractor means the feature extraction step alone consumes a large fraction of the total runtime. Naively replacing the backbone with a smaller CNN loses the monocular priors and degrades zero-shot performance—a tradeoff the paper quantifies in its ablation (Table 3), where training a lightweight backbone only on ImageNet without distillation from the hybrid teacher increases Middlebury-H BP-2 error from 2.20% to 2.87%.

Cost filtering is architecturally unexplored under latency constraints. The cost filtering module—the component that processes the 4D cost volume (height × width × disparity × features) to produce an initial disparity estimate—represents a second major bottleneck. FoundationStereo's dual-branch design (3D hourglass + Disparity Transformer) is extremely effective at capturing both local geometric structure and long-range context, but it is also extremely expensive. The paper notes (Section 3.2) that "direct pruning of the cost filtering modules yields severe performance degradation for only marginal speedup, since the channel dimension in the cost volume is already small (mostly under 100)." This is a critical insight: the cost volume channels are already a compressed representation, so there is little fat to trim via standard pruning. What is needed is not simply smaller versions of the same architecture, but fundamentally different architectural designs optimized specifically for speed while retaining filtering quality—and the space of possible designs (combinations of different layer types, kernel sizes, channel dimensions, and connectivity patterns) is combinatorially vast and largely unexplored for real-time stereo.

Iterative refinement contains hidden redundancy. The ConvGRU module that progressively refines the disparity map operates over multiple iterations (typically 8–32), with each iteration consuming the previous disparity, hidden state, and cost volume features to produce an updated disparity. The paper observes (Section 3.3, Figure 9) that aggressive pruning of this module—removing a large fraction of its parameters—causes severe accuracy degradation initially, but that retraining nearly completely recovers the lost accuracy. This implies massive redundancy: the refinement module is significantly over-parameterized for its actual task, likely because it was designed for maximum accuracy without consideration of inference cost. However, removing this redundancy is not as simple as reducing channel counts uniformly, because the recurrent dependencies within the ConvGRU create complex inter-layer relationships: pruning one layer changes the expected input dimensions of the next, and the hidden state recurrence means that input and output channels at the recurrence boundary must remain consistent.

Training data for real-world generalization is scarce and hard to create. Even if the architectural bottlenecks could be solved, training a model to generalize zero-shot requires exposure to diverse real-world data during training—not just synthetic datasets like SceneFlow. The paper recognizes that "real-world data offers greater diversity and realism than synthetic data" but "obtaining real stereo images with ground-truth metric depth annotation is notoriously difficult" (Section 3.4). Prior stereo datasets collected with depth sensors (KITTI, Middlebury, ETH3D) are too small and domain-limited to support training for generalizable models. The alternative—using foundation models themselves to generate pseudo-labels on internet-scale stereo data—raises the question of quality control: pseudo-labels are imperfect, and naively training on them can introduce systematic errors that harm rather than help generalization.

How This Paper Positions Itself

The paper does not propose a fundamentally new stereo matching architecture from scratch. Instead, it takes the strongest existing generalizable model (FoundationStereo) and systematically accelerates each of its three main components using orthogonal techniques, chosen based on the specific properties of each component:

Feature extraction is compressed via knowledge distillation rather than pruning or architectural redesign, because the value lies in the learned representation (the monocular and stereo priors), which can be transferred to a student regardless of architecture. This is an architectural choice justified by the fact that "model pruning would require us to keep the dual module, which is constrained by the computational bottleneck of its underlying ViT" (Section 3.1). By distilling, the authors can freely choose any efficient backbone (EdgeNeXt, MobileNetV2) and train it to mimic the teacher's output features.
Cost filtering is redesigned via neural architecture search rather than distillation, because distillation for this module "requires manually designing the cost-filtering module alternatives, which remain less explored than feature backbones" (Section 3.2). NAS automates the exploration of a design space that human engineers have not thoroughly mapped, potentially discovering non-intuitive architectures that balance speed and accuracy better than hand-designed alternatives.
Refinement is accelerated via structured pruning (specifically, channel pruning guided by a dependency graph) rather than redesign, because analysis revealed that the existing module contains substantial redundancy—parameters that contribute little to final accuracy. Pruning removes this redundancy while preserving the module's architecture and retraining recovers any lost performance.

This "divide-and-conquer" framing (Section 3, paragraph 1) is deliberate: each component is bottlenecked in different ways, and no single acceleration technique would work effectively across all three. Distillation would require a hand-designed student architecture for the cost filtering module, which is precisely what the authors argue is under-explored. Pruning the feature backbone would retain the expensive ViT runtime. Redesigning the refinement module from scratch would ignore the redundancy already present. The paper's contribution is therefore not any single technique—knowledge distillation, NAS, and pruning are all well-established—but rather the strategic decomposition that identifies which technique to apply where, and the specific adaptations needed for each technique to work in the stereo matching context.

The paper also positions its pseudo-labeling pipeline as a complement to these architectural accelerations, not a replacement for good data. The teacher model's output is used to generate training targets for the student, creating an output-space distillation effect that reinforces the feature-space distillation from Section 3.1. This connects the training data problem (lack of diverse real-world stereo labels) to the acceleration problem (compressing the teacher) through a common distillation framework: the student is trained to match the teacher's behavior both at intermediate feature levels (Section 3.1), at block-level outputs (Section 3.2), and at the final output level (Section 3.4), creating a multi-level transfer of the teacher's generalization ability to the efficient student.

A subtle but important positioning choice: the paper never claims to outperform FoundationStereo in accuracy. The goal is explicitly to "closely match its zero-shot accuracy" while running "over 10× faster" (Abstract, Section 1). This framing acknowledges that some accuracy loss is inevitable with compression, and positions the contribution as enabling deployment where FoundationStereo simply cannot go—real-time systems—rather than as an accuracy improvement over the state-of-the-art. This is an honest characterization that distinguishes the paper from work claiming to simultaneously improve both speed and accuracy via architectural innovation alone.

3. Technical Approach

This is primarily a systems acceleration paper whose core idea is that the computational bottlenecks of a foundation stereo model can be systematically removed by applying orthogonal compression techniques to each of its three main components, chosen based on the specific properties and redundancy patterns of each component rather than applying a single uniform strategy.

3.1 Reader Orientation

The paper builds a family of stereo matching networks that ingest a rectified left-right image pair and output a per-pixel disparity map—a dense prediction of how many pixels each point shifts between the two views—while running fast enough for real-time applications (≥20 FPS on commodity hardware) and generalizing to environments never seen during training. The system solves the problem of FoundationStereo being too slow for deployment by treating the teacher model as three independent acceleration sub-problems: feature extraction is compressed via knowledge distillation (the teacher's learned priors are transferred to a lightweight student), cost filtering is redesigned via blockwise architecture search (a combinatorial optimizer selects efficient layer configurations under a latency budget), and iterative refinement is reduced via structured pruning with guided retraining (redundant channels are removed from the ConvGRU while maintaining recurrent consistency constraints).

3.2 Big-Picture Architecture (Diagram in Words)

The system takes FoundationStereo as a frozen teacher and produces an accelerated student through three parallel acceleration pipelines, each targeting one component, followed by end-to-end fine-tuning:

Feature backbone distillation pipeline: The teacher's hybrid backbone (DepthAnything V2 + side-tuning CNN) processes single left/right images to produce a four-level feature pyramid. A student backbone (EdgeNeXt, MobileNetV2, or similar) is trained via MSE to mimic the teacher's pyramid outputs. The teacher is frozen throughout.
Cost filtering blockwise NAS pipeline: The teacher's cost filtering module (3D hourglass + Disparity Transformer) is partitioned into 8 sequential blocks. For each block, 200 candidate designs are trained independently via distillation to match the teacher block's output, given the previous teacher block's output as input. Each candidate's accuracy impact (Δm) and runtime impact (Δt) are measured by plugging it into the full teacher and evaluating end-to-end. An integer linear program (ILP) selects one candidate per block to minimize total accuracy degradation subject to a total runtime budget.
Refinement pruning pipeline: A recurrent dependency graph is constructed for the ConvGRU module to identify coupled channels (hidden state recurrence enforces equal input/output dimensions at the recurrence boundary; output heads for disparity and convex upsampling mask retain fixed dimensions). A Taylor-expansion importance score ranks all parameters globally. The least important fraction α are removed, then the pruned module is retrained end-to-end (rest of teacher frozen) with a combined L1 disparity loss and L2 feature distillation loss.
Pseudo-labeling data pipeline: The frozen teacher processes 1.4M in-the-wild stereo pairs from Stereo4D. For each pair, the teacher's disparity map is converted to a normal map via 3D unprojection. Independently, a monocular depth estimator (UniDepthV2) processes the left image and its depth is also converted to a normal map. Per-pixel cosine similarity between the two normal maps produces a geometric consistency mask. Pairs with insufficient agreement are discarded; sky regions (detected by open-vocabulary segmentation) are set to zero disparity. Surviving pairs join the training set.
End-to-end assembly and training: The distilled backbone, NAS-selected cost filtering blocks, and pruned refinement module are assembled into the final student. This student is trained end-to-end on the same mixed datasets as FoundationStereo augmented with the pseudo-labeled data. Different combinations of backbone and cost filtering candidates produce a family of models spanning an accuracy-speed Pareto frontier.

3.3 Roadmap for the Deep Dive

First, the feature backbone distillation—how the hybrid monocular-stereo teacher priors are compressed into a single lightweight student, the training objective, and why distillation was chosen over pruning.
Second, the cost filtering blockwise search—how the combinatorial NAS is set up to avoid exponential training cost, the candidate design space, the blockwise training and evaluation protocol, and the ILP formulation.
Third, the refinement pruning—how the recurrent dependency graph encodes ConvGRU constraints, how importance is scored, and how retraining recovers accuracy.
Fourth, the pseudo-labeling pipeline—how normal consistency provides a more robust quality filter than depth comparison, and how the pipeline processes internet-scale data.
Fifth, the end-to-end training and model family generation—how the separately accelerated components are integrated, fine-tuned, and how the various candidate combinations produce the speed-accuracy tradeoff shown in Figure 2.

3.4 Detailed, Sentence-Based Technical Breakdown

3.4.1 Feature Backbone Distillation: Compressing Hybrid Monocular-Stereo Priors

FoundationStereo's feature extractor is a two-headed design: the input left and right images (each of size $I_l, I_r \in \mathbb{R}^{H \times W \times 3}$ ) are processed by two parallel networks. A DepthAnything V2 model—a Vision Transformer pretrained on internet-scale monocular depth data—extracts features that encode rich monocular priors (what does depth look like in a single image? Which objects are typically closer or farther based on appearance and context?). Simultaneously, a side-tuning CNN branch processes the same images and learns to adapt these monocular features for the binocular stereo setup—specifically, learning representations sensitive to the differences between left and right views that the purely monocular ViT cannot capture. Together they produce a four-level feature pyramid $f_l^{(i)}, f_r^{(i)} \in \mathbb{R}^{C_i \times \frac{H}{i} \times \frac{W}{i}}$ at scales $i \in \{4, 8, 16, 32\}$ .

This dual-module design is a significant computational bottleneck because the ViT backbone processes high-resolution feature maps through multiple self-attention layers, each of which requires computing pairwise interactions across all spatial positions—a cost that scales quadratically with spatial resolution. Yet the monocular priors it provides are precisely what enables zero-shot generalization on unseen domains. The challenge is to preserve these priors while eliminating the ViT's inference cost.

Why distillation, not pruning. The paper explicitly considers and rejects pruning for this component (Section 3.1) for two reasons. First, pruning would require keeping the dual-module structure alive—the pruned network would still be a ViT (even if narrower or shallower) plus a side-tuning CNN, and ViT self-attention at the feature map resolutions used here remains expensive regardless of channel count. Second, any accuracy degradation from aggressive pruning would be difficult or impossible to recover via fine-tuning because the monocular priors were learned from internet-scale data. The paper notes that retraining the backbone "without retraining on internet-scale imagery" would likely fail to recover the lost knowledge. Knowledge distillation solves both problems: it is "agnostic to architecture" (Section 3.1), allowing the authors to freely choose any efficient backbone design (CNN, hybrid CNN-transformer, etc.), and it transfers the teacher's learned representation directly, without requiring the student to re-learn from the original data.

Training procedure. The teacher backbone (DepthAnything V2 + side-tuning CNN) is frozen. For each image in a training batch (both left and right images are included to "retain the statistical similarity" of stereo pairs), the teacher produces target feature pyramids $\bar{f}^{(i)}$ at each of the four scales. The student backbone—which can be any architecture (the paper reports results with EdgeNeXt and MobileNetV2 variants)—processes the same image and produces student feature pyramids at corresponding scales. A linear projection layer is inserted when the student's output channel dimension differs from the teacher's at a given scale.

The distillation loss is standard mean squared error:

$\mathcal{L}_{\text{backbone}} = \sum_{i} \left\| f^{(i)}_{\text{student}} - \bar{f}^{(i)}_{\text{teacher}} \right\|_2^2$

where $f^{(i)}_{\text{student}}$ is the student's feature map at scale $i$ (after optional linear projection for channel matching) and $\bar{f}^{(i)}_{\text{teacher}}$ is the frozen teacher's feature map at the same scale.

What it computes: for each scale in the four-level pyramid, the element-wise squared difference between the student's feature map and the teacher's feature map is computed and summed over all spatial positions and channels. The total loss sums these per-scale MSEs. Each scale captures different information—the $i=4$ scale has highest spatial resolution and fine detail; the $i=32$ scale has the largest receptive field and most semantic information. By supervising at all scales, the student is forced to learn both fine geometric detail and coarse semantic context.

Why this form: MSE is a standard feature distillation objective that encourages the student to produce numerically similar feature activations to the teacher at each spatial position. It is appropriate here because the teacher and student operate on the same input resolution at each scale, and the goal is to replicate the teacher's representation, not just its final output. An alternative like KL-divergence would require treating the feature maps as probability distributions, which is less natural for intermediate features that can have arbitrary activation statistics. Cosine similarity loss (which the paper ablates in Table 3) normalizes feature magnitudes and only aligns directions, losing information about activation strength that may be important for downstream cost volume construction.

Ablation results (Table 3). The paper compares three backbone training strategies: (1) no distillation—the student backbone is pretrained only on ImageNet classification, with no awareness of depth or stereo tasks; (2) cosine similarity loss; and (3) MSE loss. On Middlebury-H BP-2 error, no distillation yields 2.87%, cosine similarity yields 2.29%, and MSE yields 2.20%. The pattern is consistent across datasets: transfer from the hybrid teacher reduces error substantially, and MSE outperforms cosine similarity, suggesting that both feature direction and magnitude matter for the downstream stereo task.

Multiple student architectures. By training different student backbones (EdgeNeXt, MobileNetV2 at various widths) with the same distillation procedure, the paper obtains a Pareto frontier of backbone options trading accuracy for speed. This is the first degree of freedom in the model family: a heavier student backbone retains more of the teacher's spatial reasoning but adds runtime; a lighter backbone sacrifices some detail for speed.

3.4.2 Cost Filtering Blockwise Neural Architecture Search

The cost filtering module is the second major computational bottleneck. Its input is a 4D cost volume $\mathbf{V_C} \in \mathbb{R}^{C \times \frac{D}{4} \times \frac{H}{4} \times \frac{W}{4}}$ , where $C$ is the feature channel dimension (computed from concatenation and group-wise correlation of left and right features), $D$ is the maximum disparity being considered, and the spatial dimensions are at 1/4 resolution. The cost volume encodes, for each spatial location and each candidate disparity level, the matching cost between left and right features—essentially a 3D grid of "how likely is disparity $d$ at pixel $(x,y)$ ?" stacked across channels that capture different matching features.

FoundationStereo processes this volume through a dual-branch architecture. The 3D hourglass branch uses Axial-Planar Convolution (APC) layers that factor 3D convolutions (which would be $k \times k \times k$ over spatial dimensions and disparity) into separate spatial convolutions ( $k \times k \times 1$ ) and disparity convolutions ( $1 \times 1 \times k$ ). This factorization captures 3D context—the fact that neighboring disparities at neighboring pixels are correlated—without the cubic memory and computation cost of full 3D convolutions. The hourglass structure progressively downsamples then upsamples, creating a multi-scale processing pathway. In parallel, the Disparity Transformer branch tokenizes the cost volume (treating each position-disparity vector as a token) and applies multi-head self-attention, allowing each position to attend to all other positions and disparities simultaneously for long-range context.

The paper's key insight for accelerating this module is that the cost volume's channel dimension is already small—"mostly under 100" (Section 3.2). This means standard pruning (removing channels or layers) yields minimal speedup before catastrophic accuracy degradation, because there are few channels to prune and each carries significant information. What is needed is a fundamentally different architecture optimised for the latency regime, but the space of possible cost filtering architectures (combinations of 3D convolutions at various kernel sizes, APC layers with different spatial and disparity kernel dimensions, residual connections, transformer layers with varying heads and feed-forward dimensions, and volume excitation mechanisms) is combinatorially vast and under-explored.

The exponential search problem. If the cost filtering module is divided into $N$ sequential blocks $\Phi_t(\mathbf{V_C}) = B_N \circ \cdots \circ B_2 \circ B_1 (\mathbf{V_C})$ , and each block $i$ has $C_i$ candidate designs, the total number of possible full-module architectures is $C = C_1 \cdot C_2 \cdots C_N$ . The paper notes that with $N=8$ blocks and $C_i \approx 200$ candidates per block, $C \approx 200^8 \approx 10^{18}$ —an astronomically large search space. Training and evaluating even a tiny fraction of these full architectures end-to-end (as standard evolutionary NAS methods require) is computationally infeasible. The paper's solution is a blockwise decomposition that reduces training complexity from $O(n^N)$ to $O(n)$ , where $n$ is the number of per-block candidates.

Block partitioning. The cost filtering module is divided into blocks at natural architectural boundaries: within the 3D hourglass, blocks are separated at the transitions where the channel or spatial dimension of the feature volume changes (typically at the downsampling and upsampling operations of the hourglass). The entire Disparity Transformer branch is treated as a single block "consisting of a number of repeated multi-head self-attention transformer layers" (Section 3.2). This gives $N$ blocks, each of which receives a feature volume from the previous block and produces a feature volume for the next block.

Candidate design space. For each block, the paper defines five layer types that can be composed into candidate architectures:

3D convolution layers with varying channel dimensions—standard $k \times k \times k$ convolutions operating on the cost volume. Varying the output channel count controls the block's capacity and speed.
3D deconvolution (transposed convolution) layers that double the spatial dimensions of the cost volume—used in the upsampling path of the hourglass to recover spatial resolution.
APC layers with different respective kernel sizes for the spatial and disparity dimensions—for example, a $5 \times 5$ spatial kernel combined with a $3$ -element disparity kernel versus a $3 \times 3$ spatial kernel with a $5$ -element disparity kernel, trading spatial detail for disparity continuity.
Residually connected 3D convolution layers (ResNet-style) where the block computes $F(x) + x$ with a skip connection—these preserve gradient flow and allow the block to learn residual corrections to the identity mapping, which is useful when the optimal transformation is close to identity.
Feature guided volume excitation layers—a mechanism from prior work (Bangunharcana et al., 2021) that uses side information (likely pooled features from the cost volume or external context) to modulate the cost volume channels via excitation weights, similar to squeeze-and-excitation but adapted for 4D volumes.

For the Disparity Transformer block (treated as a single block in the sequence), the candidate space varies the feed-forward layer dimensions (the hidden dimension inside each transformer layer), the number of attention heads (controlling how many independent attention patterns are computed in parallel), and the number of transformer layers (controlling depth of self-attention processing).

A critical constraint for every candidate: (1) the block's runtime $t_B^s$ must be faster than its teacher counterpart $t_B^t$ (ensuring the search produces acceleration), and (2) the block's input and output channel dimensions must match the original block's (ensuring compatibility when blocks are assembled into the full module—adjacent blocks' channel dimensions must agree).

Blockwise distillation and evaluation protocol. Rather than training complete cost filtering modules, each candidate block $B_i^c$ is trained independently as a standalone network. The training objective is to mimic the teacher block's output:

$\mathcal{L}_{\text{block}} = \left\| B_i^c(f_{i-1}) - \bar{B}_i(f_{i-1}) \right\|_2^2$

where $f_{i-1}$ is the feature volume output from the previous teacher block (the input to this block position), $B_i^c$ is the candidate student block, and $\bar{B}_i$ is the frozen teacher block. For the final block that directly predicts the initial disparity map, the loss is smooth L1 against ground truth disparity rather than MSE against the teacher's features—because the final output is a disparity map, not an intermediate feature volume.

After distillation, each candidate is evaluated by plugging it into the full teacher model (replacing only the block at position $i$ while keeping all other blocks from the teacher) and running the complete model end-to-end on a separate validation dataset. Two quantities are measured:

$\Delta m_i^c$ : the relative change in error metric (e.g., BP-2 on the validation set) caused by replacing the teacher's block $i$ with candidate $c$ . A positive value means accuracy degraded; negative means improved.
$\Delta t_i^c$ : the change in runtime caused by the same replacement. A negative value means the block is faster than the teacher's version.

This protocol is crucial because it decouples block evaluation from block assembly: the impact of each candidate is measured in the context of the full teacher model, meaning the measured $\Delta m_i^c$ and $\Delta t_i^c$ values account for how the block interacts with neighboring teacher blocks. Training only $n$ blocks independently (rather than exploring $n^N$ combinations) makes the search tractable.

Combinatorial search as integer linear programming (ILP). With per-block candidate scores in hand, the problem of selecting one candidate per block to maximize accuracy under a total latency constraint becomes a combinatorial optimization. The paper formulates this as:

$\min_{\mathcal{E}} \sum_{i=1}^{N} (\Delta\mathbf{m}_i)^\top \mathbf{e}_i \quad \text{s.t.} \quad \sum_{i=1}^{N} (\Delta\mathbf{t}_i)^\top \mathbf{e}_i \leq \Delta\tau$

where $\Delta\mathbf{m}_i$ is the vector of error metric changes for all candidates at block $i$ (length $C_i$ ), $\Delta\mathbf{t}_i$ is the vector of runtime changes for all candidates at block $i$ , $\mathbf{e}_i \in \{0,1\}^{C_i}$ is a one-hot vector selecting exactly one candidate at block $i$ (only one entry is 1, the rest are 0), and $\Delta\tau$ is the total runtime budget relative to the teacher (how much faster the student cost filtering module must be than the teacher's version).

What it computes: the objective sums the error changes across all blocks for the selected candidates—the total accuracy degradation of the student module relative to the teacher. The constraint ensures that the total runtime reduction (sum of per-block runtime changes) meets or exceeds the budget $\Delta\tau$ . The ILP solver finds the combination of candidates (one per block) that minimizes total accuracy loss while satisfying the speed requirement.

Why this form: the ILP assumes that per-block accuracy and runtime impacts are additive and independent—that the effect of replacing block 3 with candidate A and block 5 with candidate B is approximately the sum of their individual effects. This is a strong assumption (interactions between blocks could mean certain combinations work better or worse than the sum of their parts), but the paper's validation experiment (Figure 8) shows that ILP-selected combinations consistently outperform random combinations under the same budget, suggesting the additivity assumption is reasonable in practice or that any interaction effects are swamped by the main effects of individual block quality. The ILP formulation is attractive because it can be solved exactly and efficiently with standard solvers (the paper uses PuLP; Mitchell et al., 2011), and by sweeping different values of $\Delta\tau$ , the paper obtains a Pareto frontier of cost filtering modules optimized for different speed regimes.

Search validation (Figure 8). The paper validates the blockwise search by comparing ILP-selected candidates against random assemblies under the same latency budget $\Delta\tau$ . As $\Delta\tau$ increases (more generous budget, slower but more accurate modules allowed), the selected candidate's accuracy improves monotonically—the search successfully finds better architectures when given more runtime headroom. Under all budgets, the selected candidate outperforms random assemblies, often by substantial margins. As $\Delta\tau$ decreases (tight latency constraints), some random assemblies show severe degradation, highlighting that network design choices become critical when the budget is tight—poor architectural choices under strict latency lead to unusable accuracy.

The refinement module takes the initial disparity map $d_0$ predicted from the filtered cost volume and iteratively improves it. The architecture is a Convolutional Gated Recurrent Unit (ConvGRU)—a recurrent network where the core computation involves gating mechanisms similar to a standard GRU but implemented with convolutions rather than fully-connected layers, preserving spatial structure. At each iteration $k$ , the ConvGRU consumes:

$d_{k-1}$ : the disparity map from the previous iteration (initialized with $d_0$ ),
$h_{k-1}$ : the hidden state from the previous iteration (initialized from a context network that processes the left image and cost volume features),
Indexed cost volume features: for each pixel, the cost volume is sampled along the disparity dimension at the current disparity estimate to extract local matching information,

and produces:

$d_k$ : the updated disparity map at iteration $k$ (computed as $d_{k-1} + \Delta d_k$ , a residual update),
$h_k$ : the updated hidden state, which is passed to the next iteration.

The recurrent structure creates a serial dependency chain: iteration $k$ cannot begin until iteration $k-1$ completes because $h_{k-1}$ is an input. The paper uses 8 refinement iterations by default (configurable; the number is a hyperparameter), and each iteration involves multiple convolutional layers within the ConvGRU's update gate, reset gate, and candidate hidden state computations.

Evidence of redundancy. The paper's analysis (Section 3.3) reveals that the refinement module is significantly over-parameterized. Figure 9 shows that pruning a large fraction of parameters (high $\alpha$ ) causes a dramatic accuracy drop when the pruned model is evaluated immediately after pruning, but that retraining with the combined loss (Equation 2) nearly completely recovers the lost accuracy. This pattern—pruning hurts, but retraining almost fully restores performance—is the hallmark of parameter redundancy: the module had more capacity than needed, and the retrained compact version learns essentially the same function with fewer parameters.

Building the recurrent dependency graph. Structured pruning removes entire channels or layers, not individual weights. The first step is to construct a dependency graph that identifies which layers' channel dimensions are coupled—changing one layer's output channels requires changing the next layer's input channels to match. For standard feed-forward networks, this graph can be automatically constructed by tracing the computation flow (as in Fang et al., 2023's DepGraph). For the ConvGRU refinement module, the paper introduces three additional constraints (Section 3.3, Figure 5):

Fixed output channels for prediction heads: The final layers that predict the disparity map update $\Delta d_k$ and the convex upsampling mask must retain fixed output channel dimensions because these correspond to specific semantic outputs (disparity is a 1-channel map; the upsampling mask has a fixed number of weights). These layers are not pruned—they are marked as fixed in the dependency graph.
Recurrent coupling: The layer that consumes the hidden state $h_{k-1}$ (as input to the ConvGRU) and the layer that produces the updated hidden state $h_k$ (as output from the ConvGRU) must have matching channel dimensions, because $h_k$ from one iteration becomes $h_{k-1}$ for the next. If pruning removes channels from the hidden state output, it must also remove the corresponding channels from the hidden state input. This creates an inter-dependent pruning constraint across the recurrent boundary.
Fixed input for motion encoder: The motion encoder that consumes indexed cost volume features must retain a fixed input channel dimension because those features come from the (non-pruned) cost volume and have a fixed structure. The encoder's first layer input channels cannot be pruned.

Importance scoring. With the dependency graph established, the paper scores each parameter's importance using a first-order Taylor expansion approximation (Molchanov et al., 2019). The idea: a parameter's importance is approximately how much the loss would change if that parameter were set to zero, which can be estimated as the absolute value of the parameter multiplied by its gradient:

$I(\theta) = |\theta \cdot \nabla_\theta \mathcal{L}|$

where $\theta$ is the parameter value and $\nabla_\theta \mathcal{L}$ is the gradient of the loss with respect to that parameter.

What it computes: the magnitude of the first-order term in the Taylor expansion of the loss around the current parameter value, when the parameter is zeroed out. Parameters with small $|\theta \cdot \nabla_\theta \mathcal{L}|$ can be removed with minimal impact on the loss (they are either very small already, or the loss is insensitive to them). Parameters with large $|\theta \cdot \nabla_\theta \mathcal{L}|$ are important and should be retained.

Why this form: computing exact importance (by actually removing each parameter and measuring the loss change) would be prohibitively expensive—requiring one forward pass per parameter. The first-order Taylor approximation requires only the parameter values and their gradients, both of which are already computed during standard backpropagation. The absolute value treats positive and negative contributions symmetrically. The paper notes that they explored "isomorphic pruning strategy" (Fang et al., 2024) but "observed slightly degraded performance," suggesting that the simpler Taylor importance metric is effective for this module.

To compute gradients for importance scoring, the paper runs the complete teacher model end-to-end with multiple refinement iterations on training data, accumulating gradients specifically for the refinement module parameters. The importance scores are then ranked globally across all prunable parameters (those not excluded by the dependency constraints), and the least important fraction $\alpha$ of parameters are pruned. The pruning ratio $\alpha \in (0, 1)$ controls the aggressiveness: $\alpha = 0.3$ removes 30% of parameters, $\alpha = 0.7$ removes 70%, and so on.

Retraining objective. After pruning, the refinement module alone is retrained end-to-end (the rest of the stereo network—backbone, cost filtering, context network—is frozen) to recover accuracy. The loss function combines disparity supervision and feature distillation:

$\mathcal{L} = \sum_{k=1}^{K} \gamma^{K-k} \left\| d_k - \overline{d} \right\|_1 + \lambda \sum_{i=1}^{L} \left\| x_i - \overline{x}_i \right\|_2^2$

where $K$ is the total number of refinement iterations, $d_k$ is the predicted disparity at iteration $k$ from the pruned student, $\overline{d}$ is the ground-truth disparity, $\gamma = 0.9$ is an exponential weighting factor (closer to 1 at later iterations, giving more weight to the final refined output), $L$ is the number of layers in the refinement module, $x_i$ and $\overline{x}_i$ are the per-layer latent features from the student and frozen teacher respectively, and $\lambda = 0.1$ balances the two loss terms.

What it computes: the first term is the sum over refinement iterations of the L1 distance between predicted and ground-truth disparity, weighted exponentially so later iterations (which should be more accurate) contribute more to the loss. The L1 loss is more robust to outliers than L2 and is standard for disparity regression. The second term is a distillation term: at each layer in the refinement module, the student's intermediate features are pulled toward the teacher's corresponding features via MSE. This ensures the pruned module not only produces accurate final outputs but also learns internal representations similar to the teacher—preventing the retraining from drifting to a different (potentially less generalizable) function.

Why this form: the combined loss addresses a key challenge of pruning: when a large fraction of parameters are removed, the reduced model may have enough capacity to fit the training data but settle into a different local minimum that generalizes worse. The distillation term ( $\lambda \sum \|x_i - \overline{x}_i\|_2^2$ ) acts as a regularizer that anchors the pruned module to the teacher's representation, preserving the generalization properties that were learned during the teacher's training on large-scale data. The L1 disparity term ensures the final output remains accurate. The initial disparity $d_0$ (before any refinement iteration) is excluded from supervision because it is produced by the cost filtering module, not the refinement module, and is unaffected by the pruning.

The exponential weighting $\gamma^{K-k}$ with $\gamma = 0.9$ means the loss at the final iteration ( $k = K$ ) receives weight $0.9^0 = 1.0$ , the penultimate iteration receives weight $0.9^1 = 0.9$ , and earlier iterations receive progressively smaller weights. This prioritizes the final output accuracy while still providing some supervision signal to intermediate iterations, encouraging the refinement process to improve monotonically.

3.4.4 Pseudo-Labeling Pipeline: Curating In-the-Wild Training Data

The three acceleration techniques above all rely on the teacher model's representations (distillation targets, block outputs, layer features) to transfer generalization capability to the student. But they operate on features, not on final predictions. The pseudo-labeling pipeline complements this by providing output-level supervision on diverse real-world data—effectively extending the distillation to the output space.

Why normal consistency, not depth comparison. The paper identifies a technical challenge in pseudo-labeling: directly comparing disparity or depth predictions for consistency is unreliable on in-the-wild images because depth ranges vary enormously across scenes (a close-up indoor shot might have depth 0.5–3 meters; an outdoor landscape might span 1–1000 meters). A "small" absolute error of 0.1 meters is negligible in the outdoor scene but enormous in the indoor one. Normal maps, which encode surface orientation (the direction a surface faces) rather than absolute distance, are scale-invariant: they depend only on relative depth changes, not absolute depth values.

The pipeline leverages this property: both the teacher's stereo disparity and the monocular depth estimator's output are converted to normal maps via 3D unprojection (using the camera parameters from Stereo4D) followed by Sobel operator gradient computation. The unprojection converts each pixel's depth and 2D image coordinates into a 3D point; neighboring 3D points define local surface patches whose normals are computed. The Sobel operator efficiently approximates gradients in the depth/disparity map, from which surface normals can be derived. Two predictions that agree on surface orientation but disagree on absolute depth (perhaps due to different scale calibration) would have high normal cosine similarity but low depth agreement—exactly the property the pipeline exploits.

Pipeline steps (Figure 6).

Teacher disparity prediction. For each rectified stereo pair from Stereo4D (a large-scale dataset of stereo videos), the frozen FoundationStereo teacher produces a disparity map for the left image.
Monocular depth prediction. Independently, the left image is fed to a monocular depth estimator (UniDepthV2; Piccinelli et al., 2025) to produce a depth map. This estimator has no access to the right image and relies on monocular cues only, providing an independent signal.
Normal map conversion. Both the disparity map (from the teacher) and the depth map (from the monocular estimator) are converted to normal maps via 3D unprojection using the known camera intrinsic parameters from Stereo4D, followed by gradient computation with a Sobel operator. The camera parameters ensure both normal maps are in the same coordinate frame.
Consistency computation. Per-pixel cosine similarity between the two normal maps:

$\text{sim}(p) = \frac{\mathbf{n}_{\text{stereo}}(p) \cdot \mathbf{n}_{\text{mono}}(p)}{\|\mathbf{n}_{\text{stereo}}(p)\| \|\mathbf{n}_{\text{mono}}(p)\|}$

where $\mathbf{n}_{\text{stereo}}(p)$ and $\mathbf{n}_{\text{mono}}(p)$ are the 3D normal vectors at pixel $p$ from the stereo and monocular estimates, respectively.

Thresholding and sky exclusion. Cosine similarity below a threshold (the paper does not specify the exact threshold value) indicates geometric inconsistency—the two depth sources disagree on surface orientation at that pixel, suggesting at least one is incorrect. Pixels below the threshold are excluded from the consistency mask. Additionally, sky regions are detected using open-vocabulary segmentation models (SAM; Ravi et al., 2024, combined with Open-Vocabulary segmentation; Xu et al., 2023) and excluded from both similarity computation and supervision, because sky has effectively infinite depth and is "underrepresented in common synthetic datasets used for training" (Section 3.4)—the models have no reliable prediction for sky, and including it would introduce noise.
Pair-level filtering and final label creation. If a stereo pair has insufficient agreement overall (too many pixels falling below the cosine similarity threshold), the entire pair is discarded. Surviving pairs have their teacher disparity maps designated as pseudo-labels, with sky regions explicitly set to zero disparity (infinite depth). The consistency mask can optionally be used to determine which pixels receive supervision during training—only pixels where both depth sources agree are used in the loss computation.
Temporal subsampling. Stereo4D contains videos, meaning consecutive frames are highly correlated. To avoid training on near-duplicate data, the pipeline subsamples temporally with a stride of 10 (keeping every 10th frame), yielding 1.4M suitable stereo pairs from the full Stereo4D dataset.

What this pipeline achieves. By requiring agreement between two independent depth estimation modalities (stereo matching and monocular depth), the consistency check filters out predictions where the teacher is likely incorrect—because it is improbable that two independent methods would produce the same wrong surface orientation, unless both are systematically biased in the same way (which the different architectures and training procedures of stereo and monocular models make unlikely). The 1.4M surviving pairs provide training data with far greater visual diversity than any existing real-world stereo dataset with ground truth (which typically number in the hundreds or low thousands of frames). This diversity is what enables the student model—trained in part on these pseudo-labels—to generalize to the diverse in-the-wild scenes shown in the qualitative results (Figures 1 and 7).

Integration with training. The pseudo-labeled data is included alongside the standard mixed synthetic datasets used to train FoundationStereo. The output-level supervision from pseudo-labels complements the feature-level distillation from Sections 3.1 and 3.2: the student learns to match the teacher's internal representations (via backbone and block distillation losses) and also learns to produce final disparity maps that match the teacher's predictions on diverse real data (via L1 disparity loss on pseudo-labeled pairs). The authors note (Section 3.4) that this "output-space distillation complements the feature-based distillation performed in previous steps."

3.4.5 End-to-End Assembly and Model Family Generation

The three accelerated components are not deployed independently—they must work together as a coherent stereo matching pipeline. The assembly process is:

Select a backbone variant from the family distilled in Section 3.4.1 (e.g., EdgeNeXt-S for higher accuracy, MobileNetV2-0.5× for higher speed).
Select a cost filtering configuration from the ILP-derived Pareto frontier in Section 3.4.2 (by choosing a specific $\Delta\tau$ value, which determines which candidate blocks are assembled). Different $\Delta\tau$ values yield different cost filtering modules optimized for different speed-accuracy points.
Select a pruning ratio $\alpha$ for the refinement module from the analysis in Section 3.4.3 (e.g., $\alpha = 0.5$ for a balanced tradeoff, $\alpha = 0.7$ for more aggressive speedup).

These three components are assembled into the complete Fast-FoundationStereo model and then trained end-to-end on the combined training data (original FoundationStereo mixed datasets plus the 1.4M pseudo-labeled pairs). The end-to-end training allows the components to adapt to each other—the backbone may adjust its features to better suit the pruned cost filtering design, and the refinement module may compensate for any remaining weaknesses.

The paper does not specify a single "best" configuration. Instead, by varying the backbone choice, the cost filtering $\Delta\tau$ , and the pruning ratio $\alpha$ , a family of models is generated spanning different points on the accuracy-speed Pareto frontier shown in Figure 2. The slowest variant (largest backbone, most permissive $\Delta\tau$ , lowest pruning $\alpha$ ) achieves the highest accuracy but runs at ~49 ms; the fastest variants (lightest backbone, most aggressive $\Delta\tau$ and $\alpha$ ) run at real-time speeds (≤30 ms) with some accuracy degradation. The family approach allows practitioners to select the model that best fits their specific latency requirements without needing to redesign or retrain.

The inference runtime profiled in Table 1 includes all three components operating together. The paper's default configuration for the main results uses 8 refinement iterations and a maximum disparity of 192. The paper notes that the disparity range is "not constrained" during evaluation—when a test image requires larger disparities than the model's configured maximum, the model can still produce predictions, though they may be less reliable for very large disparities. This is a practical detail: real-time systems typically fix the maximum disparity to a value appropriate for their expected depth range to control memory and computation.

Figure 10 shows the runtime decomposition between FoundationStereo and the slowest Fast-FoundationStereo variant. Each of the three components is accelerated by a large margin—feature extraction, cost filtering, and refinement all see substantial runtime reductions—and the total acceleration exceeds 10× (496 ms reduced to 49 ms, a 10.1× speedup). Deploying with TensorRT further reduces runtime (parenthetical values in Table 1: 49 ms reduced to 21 ms), but the paper emphasizes that even without TensorRT, the model achieves real-time performance at typical robotic perception resolutions.

4. Key Insights and Innovations

Innovation 1: Difficulty-Aware Divide-and-Conquer as an Acceleration Philosophy

The paper's most intellectually distinctive contribution is not any single compression technique but the strategic decomposition that treats a foundation model's three main components as independent acceleration sub-problems, each requiring a fundamentally different optimization approach based on its specific redundancy profile rather than a uniform compression strategy. This is a conceptual advance over the dominant paradigm in model acceleration, which typically applies a single technique—pruning, quantization, or distillation—uniformly across an entire network.

Prior work on vision foundation model acceleration exemplifies this uniform approach: SlimSAM (Chen et al., 2024) applies structured pruning to the entire SAM architecture; PTQ4SAM (Lv et al., 2024) quantizes all layers uniformly; EdgeSAM (Zhou et al., 2025) distills from a single teacher into a single student design. Even within stereo matching specifically, efficiency-oriented architectures (LightStereo, RT-IGEV, BANet) are designed from scratch as monolithic systems where all components are jointly optimized for speed under a single architectural philosophy—typically lightweight 2D convolutions throughout. The field's implicit assumption has been that what works for one component (e.g., pruning channels from a CNN backbone) works similarly for others (e.g., pruning channels from a cost volume processor).

The paper's evidence contradicts this assumption directly. For the backbone, pruning is rejected because "any deterioration in accuracy would be difficult to recover without retraining on internet-scale imagery" (Section 3.1)—the monocular priors come from pretraining data that cannot be replicated during fine-tuning, making distillation, which transfers representations without architectural constraints, the right choice. For cost filtering, pruning fails because "the channel dimension in the cost volume is already small (mostly under 100)" (Section 3.2)—there is no fat to trim, and what is needed is fundamentally different architecture discovered through automated search. For refinement, pruning succeeds precisely because the module is substantially over-parameterized (Figure 9 shows retraining recovers accuracy even after aggressive pruning), and the recurrent structure creates well-defined dependency constraints that structured pruning can target.

This is not merely an engineering choice—it is a diagnostic framework: before accelerating any component, first characterize its bottleneck type (representation bottleneck, architectural bottleneck, or parametric bottleneck), then select the technique matched to that diagnosis. The paper does not explicitly name this framework, but it is the organizing logic behind all three acceleration strategies and represents a transferable methodology for accelerating other multi-component foundation models. The generalizability of this insight is testable: one could apply the same diagnostic logic to, for example, accelerating a visual odometry pipeline (feature extraction → matching → pose optimization) or a multi-stage object detector (backbone → neck → detection head), asking at each stage whether the bottleneck is representation quality, architectural design, or parameter count. This elevates the contribution from a specific stereo acceleration recipe to a design principle for heterogeneous model compression.

The evidence for this claim is structural rather than quantitative—it lies in the fact that all three strategies are necessary and that no single strategy works across components. The paper's ablation implicitly validates this: backbone distillation (Table 3) and cost filtering NAS (Figure 8) and refinement pruning (Figure 9) are each evaluated independently, and each shows non-trivial contribution. But the more compelling evidence is the runtime decomposition in Figure 10, which shows that all three components remain significant contributors to total runtime even after acceleration—meaning no single component was the sole bottleneck. A uniform strategy (e.g., pruning everything) would have left at least one component suboptimally accelerated.

Innovation 2: Blockwise NAS with Independent Distillation as an Exponential Search Solution

The paper introduces a specific solution to a general problem in neural architecture search for large models: when the search space is combinatorially vast (here, ~10^18 possible configurations across 8 blocks), how do you train and evaluate candidates without exponential cost? The answer—train each block independently via distillation to predict its teacher block's output, then solve a combinatorial ILP—is not entirely novel in the NAS literature (Moons et al., 2021; Li et al., 2020 are cited), but its application to stereo cost filtering is significant because it solves a problem that standard NAS cannot touch at this scale.

To appreciate why this matters, contrast with the dominant NAS paradigm applied to dense prediction tasks like stereo matching. Hierarchical NAS (Cheng et al., 2020) and EASNet (Wang et al., 2022) perform architecture search for stereo networks, but they operate on the full network architecture rather than component-level blocks, and they use evolutionary search with end-to-end training of each candidate—a strategy that scales poorly with the number of components. For a cost filtering module with 8 blocks and 200 candidates per block, evolutionary search would require training thousands of complete cost filtering modules end-to-end, each involving the full 3D hourglass plus transformer processing, which is computationally prohibitive. The blockwise approach reduces this from O(n^N) to O(n × N): training 8 × 200 = 1600 small blocks, each independently, rather than exploring even a tiny fraction of the 200^8 combinations.

The intellectual novelty is in making the independence assumption work in practice for this specific domain. The ILP formulation (Equation 1) assumes that per-block accuracy impacts ( $\Delta\mathbf{m}_i$ ) are additive—that replacing block 3 hurts accuracy by some amount and replacing block 5 by another amount, and the combined effect is approximately the sum. This is an aggressive assumption because blocks in a cost filtering module are not independent: the hourglass architecture means that feature representations propagate forward and backward (through skip connections in the upsampling path), and the Disparity Transformer block interacts with the hourglass blocks through shared cost volume features. If interactions were strong, the ILP would produce combinations that look good on paper but perform poorly when assembled.

Yet Figure 8 demonstrates that the ILP-selected combinations consistently outperform random assemblies under the same latency budget. This is a validation of the independence assumption as a practical approximation for cost filtering architectures—even though blocks are not truly independent, their main effects dominate interaction effects enough that a first-order additive model finds good architectures. The failure mode is also informative: as $\Delta\tau$ decreases (tighter latency), random assemblies show severe degradation, implying that the search space contains many bad architectures and that independent evaluation successfully identifies which blocks are "robust" versus "brittle" under compression. A randomly chosen block that degrades severely when compressed may interact badly with other compressed blocks, but the blockwise training identifies such degradation before assembly.

This innovation has implications beyond stereo: any multi-stage processing pipeline where stages can be isolated and trained to match a teacher's intermediate outputs could use this approach. The key requirement is that the teacher's intermediate representations are stable and informative enough to serve as distillation targets—a condition that likely holds for many dense prediction architectures (optical flow, depth estimation, semantic segmentation) but may fail for architectures with highly entangled internal representations.

Innovation 3: Normal Consistency as a Domain-Agnostic Pseudo-Label Quality Metric

The paper's pseudo-labeling pipeline introduces a specific technical solution—using normal map consistency rather than depth comparison to filter pseudo-labels—that represents a conceptual advance in how to evaluate pseudo-label quality for 3D vision tasks. The innovation is not that pseudo-labeling is used (self-training with foundation models is widespread), but rather the choice of consistency metric and the reasoning behind it.

Prior work in pseudo-labeling for depth and stereo has predominantly used depth-space or disparity-space consistency checks. For example, multi-view consistency in depth estimation compares depth predictions from different viewpoints by reprojecting and checking for agreement in absolute depth values. The problem, which the paper identifies explicitly (Section 3.4), is that depth ranges vary dramatically across scenes: "A 'small' absolute error of 0.1 meters is negligible in the outdoor scene but enormous in the indoor one." This means a single absolute threshold for depth consistency cannot work across diverse in-the-wild data—it will be too permissive for close-range scenes (accepting poor pseudo-labels) or too strict for distant scenes (rejecting good ones).

Normal consistency solves this by being scale-invariant. Surface normals encode the direction a surface faces, which depends only on relative depth changes between neighboring pixels, not on absolute depth. Two depth predictions that agree on the shape of surfaces but disagree on absolute scale (a common failure mode for monocular depth estimators, which are scale-ambiguous) will produce similar normal maps but very different depth maps. By thresholding on normal cosine similarity rather than depth difference, the pipeline correctly accepts pseudo-labels where the stereo teacher and monocular estimator agree on geometry even if they disagree on absolute scale, while correctly rejecting pseudo-labels where they disagree on surface orientation (which indicates at least one prediction is fundamentally wrong).

The sky exclusion step is a second layer of this insight: sky is not just another region with uncertain depth—it is a region where the concept of depth is ill-defined (infinite distance), and forcing models to predict a finite depth for sky introduces systematic errors. By detecting sky via open-vocabulary segmentation and explicitly setting its disparity to zero (effectively infinite depth), the pipeline avoids contaminating training with a problem that neither the stereo teacher nor the monocular estimator can solve reliably. This is a semantic awareness that simple geometric consistency checks lack.

The significance of this innovation is that it provides a recipe for generating high-quality pseudo-labels on internet-scale data without per-dataset calibration. The normal consistency threshold (which the paper does not specify numerically, unfortunately) is the only hyperparameter, and it has a natural interpretation (minimum cosine similarity between surface orientation predictions) that transfers across scenes and depth ranges. This is a practical contribution with immediate applicability to other 3D tasks—depth estimation, multi-view stereo, structure from motion—where multiple independent depth signals can be compared via surface normals rather than absolute depth.

Table 4 provides the evidence: pseudo-labeling improves all methods tested, but the improvement is largest for methods previously trained only on synthetic data (LightStereo-L improves from 45.46% to 21.12% BP-1 on ETH3D; RT-IGEV improves from 11.52% to 8.69% BP-2 on Middlebury-H). This suggests the pipeline is not merely providing more data but is providing qualitatively different data that synthetic datasets cannot replicate—presumably the geometric complexity, texture diversity, and challenging materials (transparency, specularity) present in real-world scenes. Figure 6 bottom visualizes pseudo-labels that capture fine geometric detail (the folds of fabric, the contours of objects) that would be difficult to obtain from structured-light or LIDAR sensors at scale.

Innovation 4: Recurrent Dependency-Aware Pruning as a Redundancy Diagnostic

The paper's pruning strategy for the refinement module introduces a specific insight about how to identify redundancy in recurrent architectures: the dependency graph must explicitly model the coupling between input and output channels at recurrence boundaries. This is conceptually distinct from standard structured pruning for feed-forward networks because the hidden state recurrence creates a constraint that input and output channel dimensions must remain equal—pruning a channel from the output of the hidden state computation must also prune the corresponding channel from the input that receives the previous hidden state.

This seems like a minor technical detail, but it reveals a deeper point: the recurrent structure of the ConvGRU module is the source of its redundancy, not just a complication for pruning. Consider why the refinement module can be pruned so aggressively (Figure 9 shows substantial accuracy recovery even after removing a large fraction of parameters). In a feed-forward network, each layer transforms its input independently; pruning a channel removes a transformation pathway that may be important for some inputs. In a recurrent network, the hidden state is reused across iterations—it must encode information that is useful not just for the current iteration's update but for all future iterations. This encourages the network to learn redundant representations: if a piece of information might be needed in a future iteration, it is safer to encode it in multiple channels rather than risk losing it through the recurrence. The result is a hidden state that is substantially over-dimensioned for any single iteration's computation.

The pruning strategy exploits this by identifying which hidden state channels are actually used: the Taylor importance score (parameter × gradient) measures how much the loss would change if a parameter were zeroed, and parameters with consistently small importance across training examples are candidates for removal. The fact that retraining recovers performance (Figure 9) confirms that these parameters were indeed redundant—the pruned network can learn to encode the necessary information in fewer channels without loss of function.

This is significant because it provides a diagnostic tool: if retraining recovers accuracy after pruning, the module was over-parameterized; if retraining cannot recover accuracy, the module was at capacity and the pruning removed genuinely necessary computation. The paper does not frame it this way explicitly, but Figure 9 is effectively a capacity analysis experiment: by sweeping $\alpha$ from low (light pruning) to high (aggressive pruning) and measuring accuracy before and after retraining, one can identify the parameter regime where the module transitions from redundant (retraining recovers accuracy) to capacity-limited (retraining fails to recover). The optimal pruning ratio is the point just before this transition—removing maximum redundancy without cutting into functional capacity. The paper selects $\alpha$ values for its model family based on this analysis, though the specific transition point is not reported numerically.

This innovation complements the cost filtering NAS insight: both are about identifying what a module actually needs versus what it was given during initial design. The NAS approach searches for efficient architectures for a module that cannot be easily pruned (cost filtering); the pruning approach removes redundancy from a module that already has a well-designed architecture but too many parameters (refinement). Together, they demonstrate that acceleration requires understanding not just how to compress, but what to compress—an insight that transfers to any multi-component network where different components have different redundancy profiles.

5. Experimental Analysis

Evaluation Methodology

Datasets. Four standard public benchmarks are used for zero-shot evaluation, none of which are seen during training: Middlebury (indoor stereo pairs with high-quality structured-light ground-truth disparity), ETH3D (grayscale stereo pairs covering indoor and outdoor scenarios), KITTI 2012 and KITTI 2015 (real-world driving scenes with sparse LIDAR-derived ground-truth disparity). Additionally, Booster (Ramirez et al., 2023) is used to evaluate robustness to non-Lambertian surfaces such as translucent and specular materials (Section 4.2). The specific split from the Middlebury training set used for evaluation is referred to as "Middlebury-H" (Half resolution) and "Middlebury-Q" (Quarter resolution), with Middlebury-Q described as "similar to typical resolution for real-time robotic applications" (Section 4.3). All datasets are "unseen to all the evaluated methods" at test time (Section 4.3).
Base model(s). The teacher model is FoundationStereo (Wen et al., 2025), a state-of-the-art zero-shot generalizable stereo network that combines DepthAnything V2 monocular priors with a side-tuning CNN for feature extraction, a dual-branch cost filtering module (3D hourglass with APC layers + Disparity Transformer), and a ConvGRU-based iterative refinement module. The student family uses distilled backbones from the EdgeNeXt and MobileNetV2 families (Section 3.1), with specific configurations chosen to span different speed-accuracy tradeoffs as shown in Figure 2. The monocular depth estimator used in the pseudo-labeling pipeline is UniDepthV2 (Piccinelli et al., 2025), and the open-vocabulary segmentation model is based on SAM (Ravi et al., 2024) with open-vocabulary extensions (Xu et al., 2023).
Metrics. For Middlebury, ETH3D, and Booster, the primary metric is BP-X (Bad Pixel percentage at X pixels): "the percentage of pixels where the disparity error is larger than X pixels" (Section 4.2). Standard thresholds are BP-1 (error > 1 pixel), BP-2 (error > 2 pixels), and BP-3 (error > 3 pixels). For Booster, additional thresholds of BP-4, BP-6, and BP-8 are reported, along with End-Point Error (EPE) in pixels. For KITTI 2012 and 2015, the metric is D1: "the percentage of pixels whose disparity error is larger than 3 pixels and 5% of the ground-truth disparity" (Section 4.2), the standard KITTI benchmark metric. All metrics are evaluated on non-occluded regions only. The paper also reports runtime in milliseconds, profiled on the same NVIDIA 3090 GPU for all methods to ensure fair comparison (Section 4.3, Table 1).
Baselines. The paper compares against three categories of methods (Table 1, Table 2). Real-time methods include IINet (Li et al., 2024), LightStereo-L (Guo et al., 2025), RT-IGEV (IGEV++; Xu et al., 2025), BANet-2D and BANet-3D (Xu et al., 2025). These are primarily trained on SceneFlow only (when marked with ∗ in Table 1), with some additionally retrained on the paper's combined datasets including pseudo-labels (marked with †). Generalizable but slow methods include StereoAnywhere (Bartolomei et al., 2025), DEFOM-Stereo (Jiang et al., 2025), MonSter (Cheng et al., 2025), Zero-RAFT-Stereo (Wang et al., 2025), and FoundationStereo (Wen et al., 2025). For Booster (Table 2), additional slower baselines include RAFT-Stereo (Lipson et al., 2021), PSMNet (Chang et al., 2018), GMStereo, PCVNet (Zeng et al., 2023), DLNR (Zhao et al., 2023), Selective-IGEV, IGEV, and NMRF.
Generation budget / compute accounting. The paper measures compute via wall-clock runtime in milliseconds, profiled on a single NVIDIA 3090 GPU at Middlebury-Q resolution (Section 4.3). This is a practical choice: since none of the compared methods share a common architectural unit of computation (some use iterative refinement with variable iterations, others use single-pass architectures), counting FLOPs would be misleading due to differences in hardware utilization, memory access patterns, and parallelism. Runtime directly measures what matters for real-time deployment. Table 1 reports runtime for all methods, with the paper's own model achieving 49 ms (21 ms with TensorRT), compared to FoundationStereo at 496 ms. The maximum disparity for cost volume construction is fixed at 192 during inference (Section 4.1), with 8 refinement iterations used by default. On Middlebury-H (higher resolution), generalizable methods use a maximum disparity of 416 "for the best performance" (Section 4.3). For the blockwise NAS (Section 3.2), runtime is tracked per-block as $\Delta t_i^c$ and summed in the ILP constraint.
Cross-validation / statistical protocol. The blockwise NAS evaluation (Section 3.2) uses a separate validation dataset for measuring per-block error and runtime changes ( $\Delta m_i^c$ and $\Delta t_i^c$ ), though the paper does not specify the exact validation split. For the randomly assembled cost filtering baselines in Figure 8, 10 random candidate models are sampled per latency budget and trained end-to-end for comparison. The paper does not report confidence intervals or standard deviations for the main benchmark results in Tables 1 and 2—each number is a single evaluation on the full test set of each benchmark. For the pruning ratio analysis (Figure 9), accuracy and runtime are measured on Middlebury-Q as a function of $\alpha$ , though the paper does not specify whether this is a held-out subset of Middlebury or the full Middlebury-Q dataset.

Main Quantitative Results

Zero-Shot Generalization Against Real-Time Methods (Table 1)

The headline result from Table 1 is that Fast-FoundationStereo outperforms all existing real-time methods by a substantial margin across every benchmark and metric, while running at 49 ms—within the real-time envelope. On Middlebury-H, the most challenging resolution, the paper's model achieves BP-2 error of 4.80%, compared to the next-best real-time method RT-IGEV (trained on the same data including pseudo-labels, marked †) at 8.69%—a 45% relative reduction in error. The gap widens dramatically against methods trained only on SceneFlow: LightStereo-L∗ reaches 37.49% BP-2, BANet-3D∗ reaches 44.90%, and IINet∗ reaches 25.88%. These errors are so large that the disparity maps would be structurally unusable—for a 192-pixel disparity range, a BP-2 error of 37% means more than a third of all pixels are off by more than 2 pixels.

On ETH3D (BP-1 error), Fast-FoundationStereo achieves 1.22%, versus 5.12% for RT-IGEV† and 21.12% for LightStereo-L†—a 4.2× and 17.3× reduction, respectively. On KITTI 2015 (D1 error), the paper's model achieves 3.25%, compared to 4.00% for RT-IGEV, 4.40% for RT-IGEV†, and 6.00% for RT-IGEV∗. The improvement over RT-IGEV† is modest (0.75 percentage points) but consistent, and the paper's model runs at comparable speed (49 ms vs. 45 ms).

Comparing against slow generalizable methods, the paper's model is competitive or better than several. On Middlebury-H BP-2 error, Fast-FoundationStereo (4.80%) outperforms Zero-RAFT-Stereo (8.48%) by 43%, StereoAnywhere (9.67%) by 50%, and MonSter (9.33%) by 49%, while running 3.4× to 10.1× faster. Against FoundationStereo (2.49% BP-2), the accuracy gap is 2.31 percentage points—a modest degradation for a 10.1× speedup (496 ms vs. 49 ms). On KITTI 2015 D1, Fast-FoundationStereo (3.25%) outperforms FoundationStereo (2.80%) by 0.45 percentage points—a rare case where the accelerated student exceeds the teacher, possibly due to the regularization effect of distillation and pseudo-label training or to the specific test distribution favoring the student's architectural inductive biases.

Runtime comparison details. The slowest Fast-FoundationStereo variant (49 ms) is profiled at Middlebury-Q resolution. With TensorRT acceleration, this drops to 21 ms (shown as parenthetical in Table 1), corresponding to approximately 48 FPS—well above the 30 FPS threshold typically considered real-time. Fastest variants from Figure 2 run at approximately 15 ms (~67 FPS), though their specific accuracy numbers are not tabulated in Table 1. The paper's slowest variant is 5.8× faster than Zero-RAFT-Stereo (164 ms), 6.9× faster than MonSter (336 ms), and 10.1× faster than FoundationStereo (496 ms).

The qualitative results (Figures 1, 7) reinforce these numbers: on scenes with transparent glass doors, specular highlights on shiny surfaces, textureless walls, and complex illumination, Fast-FoundationStereo produces disparity maps that are visually nearly indistinguishable from FoundationStereo and MonSter, while real-time baselines (LightStereo, RT-IGEV) show gross errors—missing object boundaries, smoothing over fine detail, and producing large smooth regions of incorrect depth. Figure 1 specifically highlights cases where the paper's model "occasionally exceeds" MonSter, such as the shiny door in the top row and the paper towel bin in the bottom row, though the paper appropriately characterizes this as occasional rather than systematic.

Robustness to Non-Lambertian Surfaces (Table 2)

On the Booster-Q dataset, which specifically targets translucent and specular surfaces that violate the standard Lambertian assumption of stereo matching, Fast-FoundationStereo achieves BP-2 error of 6.61% and EPE of 1.54 pixels—superior to every real-time method and competitive with the best slow methods. RT-IGEV achieves 18.19% BP-2 (when trained on the same data, marked †), meaning the paper's model reduces error by 2.75×. The gap is even larger against the original RT-IGEV∗ (23.09% BP-2).

Against slow methods, Fast-FoundationStereo's 6.61% BP-2 is better than RAFT-Stereo (17.84%), PSMNet (34.47%), GMStereo (32.44%), and Selective-IGEV (18.52%), and comparable to IGEV (16.90%). FoundationStereo achieves 5.18% BP-2, a gap of 1.43 percentage points that is proportionally larger than on standard benchmarks, suggesting that non-Lambertian surfaces are particularly challenging for the compressed model. This makes sense: the teacher's Disparity Transformer provides long-range context that helps resolve ambiguous matches on specular surfaces where local appearance is misleading, and compressing this module likely reduces the model's ability to reason about such cases. StereoAnywhere achieves 9.01% BP-2, meaning the paper's model outperforms a generalizable method more than 8× slower (427 ms vs. 49 ms).

The EPE gap is similarly informative: FoundationStereo at 1.13 pixels vs. Fast-FoundationStereo at 1.54 pixels—a 0.41 pixel absolute increase that represents a 36% relative degradation but remains substantially below all real-time baselines (RT-IGEV† at 4.20 pixels, a 2.7× larger error).

Runtime Decomposition and Scaling Analysis (Figure 10, Figures 8-9)

Figure 10 decomposes the total runtime of FoundationStereo (496 ms) and the paper's slowest model (49 ms) into the three component-wise contributions. Feature extraction is accelerated from a dominant fraction of the total to a minority share; cost filtering sees a substantial absolute reduction; and refinement is reduced through pruning. The paper does not provide the absolute millisecond breakdown for either model (the bars in Figure 10 are unlabeled with exact values), but the visualization makes clear that all three components contribute meaningfully to the total speedup—no single component dominates the acceleration. This validates the divide-and-conquer strategy: if only one component were the bottleneck, accelerating it alone would suffice, but the roughly proportional reductions across all three components indicate that all three were bottlenecks in the original architecture.

Figure 8 validates the blockwise NAS approach by comparing ILP-selected cost filtering configurations against random assemblies under the same latency budget $\Delta\tau$ . Three findings emerge: (1) as $\Delta\tau$ increases (more runtime budget allowed), the selected candidate's accuracy improves monotonically, meaning the search successfully finds architectures that use additional runtime productively; (2) the selected candidate consistently outperforms random assemblies across all $\Delta\tau$ values; and (3) as $\Delta\tau$ decreases toward tight latency constraints, some random assemblies exhibit severe accuracy degradation, while the selected candidate degrades gracefully. This third point is critical for real-time deployment: it shows that architecture matters most when the budget is tightest, exactly the regime where real-time systems operate. A poorly chosen architecture under strict latency can produce unusable results, while a well-chosen one maintains reasonable accuracy.

Figure 9 shows the effect of pruning ratio $\alpha$ on accuracy (Middlebury-Q BP-2, left y-axis) and runtime per iteration (right y-axis). The key pattern is the gap between pre-retraining and post-retraining accuracy: at all pruning ratios, immediate post-pruning accuracy degrades substantially (the red bars drop lower as $\alpha$ increases), but retraining (blue bars) recovers nearly all the lost accuracy up to some threshold, after which degradation becomes irrecoverable. The paper does not specify the exact threshold, but the figure visually suggests that up to roughly 50–60% pruning ( $\alpha \approx 0.5\text{--}0.6$ ), retraining restores accuracy to near-baseline levels. Beyond this, even retrained accuracy begins to drop, indicating the module has been pruned into its capacity-limited regime. Runtime per iteration decreases roughly linearly with $\alpha$ , as expected from channel pruning.

Effect of Pseudo-Labeling (Table 4)

Table 4 quantifies the contribution of the pseudo-labeled in-the-wild data by comparing models trained with and without it (parenthetical values are without pseudo-labels). For Fast-FoundationStereo on Middlebury-H, pseudo-labeling reduces BP-2 error from 2.53% to 2.20%—a modest 0.33 percentage point improvement that nevertheless represents a 13% relative error reduction. The pattern is consistent across datasets: ETH3D BP-1 drops from 1.31% to 1.22%, KITTI 2015 D1 from 3.48% to 3.25%.

The more dramatic effects appear for the baselines. RT-IGEV's Middlebury-H BP-2 drops from 11.52% to 8.69% with pseudo-labels—a 24% relative improvement. LightStereo-L's ETH3D BP-1 drops from 45.46% to 21.12%—a 54% relative improvement, though the resulting error remains far above usable levels. This pattern—larger gains for methods starting from higher error—suggests the pseudo-labeled data primarily helps by exposing the model to visual phenomena (textureless surfaces, specularities, complex real-world geometry) that are absent from synthetic datasets like SceneFlow. Methods already exposed to diverse data through foundation model pretraining (Fast-FoundationStereo via distillation, FoundationStereo directly) benefit less because they already possess some generalization capability.

Ablation Studies and Robustness Checks

Feature backbone distillation loss (Table 3): Comparing no distillation (ImageNet pretraining only), cosine similarity loss, and MSE loss on the feature backbone reveals that MSE outperforms cosine similarity by a small but consistent margin. On Middlebury-H BP-2, MSE achieves 2.20%, cosine similarity achieves 2.29%, and no distillation achieves 2.87%. The 0.67 percentage point gap between no distillation and MSE represents a 23% relative reduction in error, confirming that the teacher's hybrid monocular-stereo priors provide generalization value beyond standard ImageNet features. The smaller gap between MSE and cosine similarity (0.09 percentage points on Middlebury-H) is consistent across datasets: ETH3D BP-1 shows 1.22% vs. 1.19% (a negligible 0.03 point difference), KITTI 2015 D1 shows 3.25% vs. 3.31%. The paper's conclusion that MSE is preferred is statistically weak—these differences are within what could be noise from a single evaluation on fixed test sets—but the ablation demonstrates that any form of feature distillation is substantially better than none.
Blockwise NAS vs. random search (Figure 8): This is the paper's most important validation experiment because it tests the core assumption of the blockwise NAS approach: that per-block independent evaluation ( $\Delta m_i^c$ , $\Delta t_i^c$ ) provides a good proxy for full-module performance. The ILP-selected architecture consistently outperforms random assemblies under the same latency budget, confirming the proxy's effectiveness. However, the paper does not report how close the ILP proxy ranking is to the true ranking of architectures—it demonstrates superiority over random baselines, but does not verify that the ILP-selected architecture is near-optimal among the 10^18 possible configurations. The 10 randomly sampled architectures per budget represent a tiny fraction of the search space, so the baseline, while reasonable, does not preclude the possibility that substantially better architectures exist that the ILP misses due to block interaction effects. An experiment comparing the ILP selection against a small number of end-to-end-trained full configurations (perhaps the top few from the ILP ranking) would provide stronger validation, but this is not reported.
Pruning ratio and retraining (Figure 9): The finding that retraining recovers accuracy up to high pruning ratios is the paper's primary evidence for redundancy in the refinement module. The pre-retraining degradation shows that the pruned parameters were not zero-valued (removing them immediately hurts), but the post-retraining recovery shows they were not uniquely necessary (the smaller network can relearn the function). This is a cleaner demonstration of redundancy than simply showing that pruning doesn't hurt—it distinguishes between "parameters that are small and removable without retraining" (standard magnitude pruning) and "parameters that are actively used but can be replaced by retraining a more compact representation" (structural redundancy). However, the paper does not report the number of refinement iterations used for Figure 9, nor the specific $\alpha$ value selected for the final model in Table 1. The claim of "large redundancy" is qualitative rather than quantitative without specifying the exact pruning ratio where accuracy begins to irrecoverably degrade.
Isomorphic pruning exploration (Section 3.3): The paper briefly mentions exploring "isomorphic pruning strategy (Fang et al., 2024) but observed slightly degraded performance." This is a negative result: a more sophisticated pruning method that enforces structural consistency across pruned layers performs worse than the simpler Taylor importance scoring with recurrent dependency constraints. The paper does not elaborate on why, but a plausible explanation is that isomorphic pruning's additional constraints prevent the removal of some redundant channels that happen to be isomorphic across layers, making it less aggressive at removing true redundancy. This result supports the paper's choice of a relatively simple importance metric, but the absence of detail (how much degradation? On which metrics?) makes it difficult to assess its significance.
Effect of pseudo-labels on different methods (Table 4): The ablation demonstrates that pseudo-labels improve all methods tested, but the magnitude of improvement varies substantially: 13% relative error reduction for Fast-FoundationStereo on Middlebury-H vs. 54% relative error reduction for LightStereo-L on ETH3D. This interaction—pseudo-labels help more for methods with worse baseline generalization—is consistent with the hypothesis that the pseudo-labels provide visual diversity absent from synthetic data, but it is not tested formally (e.g., by ablating the diversity vs. volume of pseudo-labeled data). An experiment that controls for dataset size (e.g., comparing equal-sized subsets of pseudo-labeled vs. synthetic data) would isolate the diversity effect from the data volume effect, but this is not performed.
Architecture search space components: The paper describes five layer types in the NAS search space (3D conv, 3D deconv, APC, residual 3D conv, feature guided volume excitation) plus transformer block variations, but does not ablate which of these layer types are most important for the final architectures. Do the ILP-selected configurations consistently favor certain layer types under tight latency budgets? Are APC layers essential, or do they get replaced by simpler 3D convolutions? This analysis would provide architectural insights beyond the search itself, but is not reported.
Refinement iteration count: The paper uses 8 refinement iterations "unless otherwise mentioned" (Section 4.1), but does not ablate this choice. Given that refinement is a serial bottleneck (Figure 10 shows it as a significant runtime component), and that the pruning analysis (Figure 9) shows redundancy in the per-iteration computation, understanding how accuracy scales with iteration count in the pruned module would clarify whether running more iterations with a heavily pruned module could recover some of the teacher's accuracy at lower total cost than running fewer iterations with a less pruned module. This ablation is absent.
Sky exclusion in pseudo-labeling: The paper mentions that sky regions are detected and set to zero disparity, and that consistency computation excludes sky, but does not ablate this step. Does sky exclusion matter, or would the normal consistency check naturally filter sky regions anyway? Given that sky regions are "underrepresented in common synthetic datasets" (Section 3.4), their treatment could significantly affect generalization to outdoor driving scenes (KITTI), but this is not isolated.

Critical Assessment

Claim 1: Fast-FoundationStereo achieves strong zero-shot generalization at real-time frame rate, establishing a new state-of-the-art among real-time methods by a substantial margin.

The experimental evidence in Tables 1 and 2 strongly supports this claim, but with important caveats about what "real-time" and "state-of-the-art" mean in context.

The accuracy margins over existing real-time methods are unequivocally large. On Middlebury-H BP-2, the 4.80% error vs. 8.69% for the next-best real-time method (RT-IGEV† trained on identical data) represents a 45% relative improvement—not a marginal gain. On ETH3D BP-1, the gap is even larger (1.22% vs. 5.12%, a 4.2× reduction). These are differences large enough to be qualitatively meaningful: a robot using Fast-FoundationStereo at BP-2 error of 4.80% has errors concentrated primarily near depth discontinuities and textureless regions that are inherently ambiguous, while at 8.69% or 37.49%, large contiguous regions of the scene are systematically wrong (as visible in Figure 7).

However, the runtime comparison is less straightforward than the 49 ms figure suggests. The 49 ms measurement is for the slowest Fast-FoundationStereo variant on an NVIDIA 3090 GPU—a desktop-class GPU with 24 GB of VRAM, not the embedded or mobile GPUs typical of real robotic and AR deployments. The paper claims real-time performance, but 49 ms corresponds to approximately 20 FPS, which is at the lower boundary of what is considered real-time (typically 20–30 FPS minimum for robot navigation, ≥30 FPS for AR). The TensorRT-accelerated 21 ms (~48 FPS) is comfortably real-time, but the paper acknowledges TensorRT as optional rather than part of the core contribution. The methods it compares against (RT-IGEV at 45 ms, LightStereo-L at 30 ms) are profiled on the same hardware, so the relative comparison is fair, but the absolute claim of "real-time" should be qualified by the hardware context: this is real-time on a high-end desktop GPU, not on embedded platforms. The paper does not report runtime on edge devices (Jetson, mobile GPU, etc.), which is the deployment target implied by the motivation of "embodied agents operating in in-the-wild environments" (Section 1).

The "substantial margin" claim is also sensitive to the choice of baseline training data. The paper reports RT-IGEV and LightStereo-L in two configurations: trained on SceneFlow only (∗, with catastrophically high errors) and trained on the same combined datasets as Fast-FoundationStereo (†, with dramatically improved but still inferior errors). The gap against the ∗ variants is partly a data effect (the paper's training recipe includes more diverse data), not purely an architectural effect. The gap against the † variants isolates the architectural contribution, and it remains substantial, but the paper could have made this distinction sharper by also training IINet and BANet on the combined data—their absence from the † group leaves open the possibility that those architectures would benefit even more from the additional data, potentially narrowing the gap. The paper's claim that it "outperforms other real-time models by a significant margin across the board" (Section 4.3) holds, but "across the board" should be understood as across the specific set of baselines selected and the specific training configurations evaluated.

Claim 2: The divide-and-conquer acceleration strategy systematically addresses computational bottlenecks without sacrificing robustness.

The evidence is mixed but generally supportive. The runtime decomposition in Figure 10 demonstrates that all three components are accelerated—this is a necessary condition for the claim that the strategy is systematic. If only one or two components were accelerated, the "divide-and-conquer" framing would be an overstatement; the fact that all three see substantial reductions validates that the paper correctly identified multiple independent bottlenecks.

The "without sacrificing robustness" claim is supported by the accuracy comparisons against generalizable methods in Tables 1 and 2. Fast-FoundationStereo's accuracy is substantially closer to FoundationStereo's than to any real-time method's, and it matches or exceeds several slow generalizable methods. On Middlebury-H BP-2, the degradation from FoundationStereo (2.49%) to Fast-FoundationStereo (4.80%) is a 2.31 percentage point absolute increase—not negligible, but dramatically smaller than the gap to real-time methods (RT-IGEV† at 8.69%). Robustness to non-Lambertian surfaces (Table 2) follows the same pattern: Fast-FoundationStereo at 6.61% BP-2 vs. FoundationStereo at 5.18%—a 1.43 point gap that is larger proportionally (28% relative degradation) but still places the model in the generalizable regime rather than the real-time regime.

However, the paper does not fully decompose how much of the retained robustness comes from each of the three acceleration strategies. Does backbone distillation contribute more to generalization than cost filtering NAS or refinement pruning? Is the pseudo-labeling data essential for the robustness claim, or would the accelerated architecture on its own (trained only on synthetic data) still generalize? The paper provides individual ablations for backbone distillation (Table 3) and pseudo-labeling (Table 4), but these ablations are applied to the full model, not to isolated components. The cost filtering NAS is validated against random architectures (Figure 8) but not against a "no search" baseline (e.g., a uniformly scaled-down version of the teacher's cost filtering module). The pruning strategy is validated by showing retraining recovery (Figure 9) but not by comparing the pruned-and-retrained module's generalization to a "from-scratch" compact refinement module trained without the teacher's guidance. These missing ablations mean the paper demonstrates that the complete strategy works, but not necessarily why each component is essential.

The absence of a combined ablation—varying all three strategies simultaneously to show that each contributes independently to the final result—is the primary evidence gap. The paper's strongest claim about the divide-and-conquer approach is that all three strategies are necessary and complementary, but without showing that removing any one strategy degrades performance beyond what the remaining two can compensate for, this remains a design narrative rather than an experimentally validated claim. The individual ablations are suggestive (Table 3 shows backbone distillation matters; Figure 8 shows architecture search matters; Figure 9 shows pruning matters), but they are performed in contexts where the other components are held at specific configurations, not systematically varied.

Claim 3: The blockwise NAS approach efficiently discovers optimal cost filtering designs under latency budgets.

This claim is partially supported by Figure 8, which shows ILP-selected architectures outperform random baselines. However, "optimal" is too strong a word for what the experiments demonstrate. The ILP finds architectures that are better than random samples from the same search space—a necessary but not sufficient condition for optimality. The true optimum among the 10^18 possible configurations is unknown and unknowable without exhaustive search. The claim should more precisely be that the blockwise NAS finds "good" or "competitive" architectures at dramatically reduced search cost.

The ILP formulation itself contains an important untested assumption: that per-block accuracy and runtime impacts are additive. The validation in Figure 8 provides circumstantial support (if interactions were strong and adversarial, the ILP would occasionally produce worse architectures than random, which it does not), but does not directly measure interaction effects. An experiment that takes the ILP-selected architecture and permutes the blocks (keeping the same set of candidates but reassigning them to different positions) would test whether block ordering matters beyond individual block quality—if permutation significantly changes accuracy, then the additivity assumption is violated and the ILP solution space is misspecified. This experiment is not reported.

The search space itself is also not ablated. The five layer types (3D conv, 3D deconv, APC, residual conv, volume excitation) represent a particular set of architectural primitives chosen by the authors. If the optimal real-time cost filtering module requires operations not in this space—for example, depthwise separable 3D convolutions, sparse 3D convolutions, or attention mechanisms different from the Disparity Transformer variant in the search space—the blockwise NAS cannot discover them. This is a reasonable limitation (all NAS is bounded by its search space), but it means the claim of discovering "optimal" designs should be qualified as "optimal within the defined search space."

Claim 4: The pseudo-labeling pipeline produces training data that consistently improves zero-shot generalization.

Table 4 cleanly supports this claim. Pseudo-labeling improves all tested methods consistently across all benchmarks, with particularly large gains for methods previously limited to synthetic data. The consistency check based on normal map similarity (rather than depth comparison) is the claimed mechanism, but this specific design choice is not ablated against alternatives—the paper does not compare against a baseline that uses depth-based consistency filtering or no filtering at all. The improvement from pseudo-labeling could be due to: (1) the normal consistency filter removing bad pseudo-labels (the claimed mechanism), (2) simply having more data (any pseudo-labels, even noisy ones, might help), or (3) the specific domain of Stereo4D providing visual diversity not present in the existing training mix. Without ablating the consistency filter's threshold or comparing against unfiltered pseudo-labeling, the specific contribution of normal consistency remains plausible but unconfirmed. The paper's main claim about pseudo-labeling—that it helps—is well-supported; the claim that normal consistency is the reason it helps is suggested but not experimentally isolated.

Missing experiments that would strengthen the paper:

Runtime on embedded hardware. The paper's motivation emphasizes "embodied agents operating in in-the-wild environments" (Section 1), implying deployment on robots, drones, or AR headsets with embedded GPUs. All runtime numbers are on an NVIDIA 3090. Benchmarks on Jetson Orin or comparable embedded platforms would directly address whether Fast-FoundationStereo achieves real-time on the target hardware.
Per-dataset training vs. zero-shot comparison. Fast-FoundationStereo is evaluated exclusively in zero-shot mode—it never sees data from the test datasets. But the paper's strong claim is about real-time methods "struggling to generalize due to their reliance upon per-domain fine-tuning" (Section 1). An experiment fine-tuning Fast-FoundationStereo on a target domain (e.g., KITTI) and comparing against fine-tuned real-time methods on that domain would test whether the foundation model distillation provides benefits beyond zero-shot—can the architecture also serve as a better initialization for domain-specific fine-tuning, or does the compression reduce adaptability?
Scaling of accuracy with runtime across the model family. Figure 2 shows a Pareto frontier of accuracy vs. speed, but the paper does not systematically analyze where on this frontier the various ablations place the model. Does making the backbone heavier while keeping cost filtering fixed provide diminishing returns? Is there an interaction where a heavier backbone requires a more capable cost filtering module to exploit its features? A grid of backbone × cost filtering × pruning configurations (rather than treating them as independent choices) would reveal whether the three acceleration axes are truly independent or whether certain combinations synergize.
Generalization to non-standard stereo geometries. All test datasets assume rectified stereo pairs with known calibration. Fast-FoundationStereo inherits FoundationStereo's architecture, which is designed for rectified stereo. The paper does not test generalization to non-rectified or uncalibrated settings, which would be necessary for truly "in-the-wild" deployment where perfect rectification cannot be assumed.
Failure case analysis. The paper's qualitative results (Figures 1, 7) show successful cases. Where does Fast-FoundationStereo fail relative to FoundationStereo? Are there systematic failure modes introduced by the acceleration—for example, thin structures that the teacher captures but the student misses due to reduced feature resolution, or large textureless regions where the pruned refinement module cannot recover from an incorrect initialization? A dedicated failure analysis (analogous to FoundationStereo's over-optimization analysis in the reference paper) would characterize the cost of acceleration more precisely than aggregate metrics.

Conditional assessment of the claims:

The claim that Fast-FoundationStereo achieves "strong zero-shot generalization at real-time frame rate" holds when "real-time" is interpreted as 20–48 FPS on a high-end desktop GPU, when "strong zero-shot generalization" means errors within 2–3× of the slowest generalizable methods and 2–17× better than existing real-time methods, and when the test domains are standard stereo benchmarks (indoor, outdoor, driving) with rectified inputs. The claim does not extend to embedded hardware, to domains requiring per-domain fine-tuning, or to non-rectified inputs—none of these were tested.

The claim that the divide-and-conquer strategy "systematically addresses computational bottlenecks" holds in the sense that all three components are accelerated and the final model is substantially faster. The stronger claim that the specific pairing of strategies to components (distillation for backbone, NAS for cost filtering, pruning for refinement) is optimal or even necessary is suggested by the component-specific reasoning in Section 3 but not experimentally validated against alternative pairings (e.g., pruning the backbone, distilling the cost filtering module).

The claim of "establishing a new state-of-the-art among real-time methods" is the most solidly supported: on every benchmark, every metric, Fast-FoundationStereo outperforms every real-time method tested, usually by large margins. The only caveat is the limited set of real-time baselines—the paper evaluates 5 real-time methods, which covers major recent architectures but not exhaustively all published real-time stereo networks. Given the magnitude of the margins (2–4× error reduction on most metrics), it is unlikely that untested real-time methods would close the gap, but the claim of "state-of-the-art" is only as comprehensive as the baseline selection.

6. Limitations and Trade-offs

6.1 All Results Are on a Single GPU; Embedded Deployment Is Unvalidated

The assumption or constraint. The entire motivation of this work centers on deploying foundation-model-quality stereo matching in latency-bound systems such as "embodied agents operating in in-the-wild environments" (Section 1). Every runtime measurement in the paper—49 ms for the slowest variant, 21 ms with TensorRT, and the comparisons in Table 1—is profiled on an NVIDIA 3090 GPU, a desktop-class card with 24 GB of VRAM and ampere-generation tensor cores. The paper never reports runtime on any embedded or mobile platform (e.g., NVIDIA Jetson Orin, Qualcomm Snapdragon, or mobile GPU), despite these being the deployment targets implied by the robotics and augmented reality use cases cited in the introduction (Lee et al., 2025; Kong et al., 2025).

The consequence. Practitioners deciding whether to deploy Fast-FoundationStereo on a robot or AR headset cannot estimate real-world performance from the reported numbers. The 3090 has approximately 35.6 TFLOPS of FP32 compute and a 936 GB/s memory bandwidth; a Jetson Orin AGX has roughly 5.3 TFLOPS and 204 GB/s—factors of ~7× and ~4.6× reduction, respectively. The 21 ms TensorRT result on a 3090 could translate to anywhere from 100–200 ms on an embedded platform, depending on whether the bottleneck is compute-bound or memory-bound, which the paper does not characterize. At 100–200 ms (5–10 FPS), the model would fail the real-time requirement for essentially all robotics and AR applications, invalidating the paper's central premise for the very use cases it motivates. Worse, the paper provides no profiling breakdown by operation type (convolution vs. attention vs. memory movement) that would enable practitioners to predict embedded performance from the 3090 numbers.

What evidence exists in the paper. The paper is transparent that all measurements use an NVIDIA 3090 (Section 4.3: "The inference runtimes for all methods are profiled over Middlebury-Q... on the same hardware with NVIDIA 3090 GPU"). The hardware is consistent across comparisons, so the relative speedup claims (e.g., 10.1× vs. FoundationStereo) are valid within that context. But there is zero embedded benchmarking, zero memory footprint reporting beyond runtime, and zero analysis of whether the model fits within the memory constraints of typical edge platforms. The TensorRT results (parenthetical in Table 1) demonstrate that GPU-specific optimizations help, but TensorRT is also available on Jetson platforms—whether the magnitude of improvement transfers is unknown.

Mitigation status. Not addressed. The paper does not acknowledge this as a limitation, does not suggest embedded benchmarking as future work, and does not qualify the "real-time" claim with a hardware specification. The phrase "real-time" is used without defining the latency threshold or the hardware class, leaving it ambiguous whether 49 ms on a 3090 qualifies. In the computer vision literature, "real-time" is typically understood as ≥20–30 FPS (33–50 ms), but the implicit assumption is that the hardware is appropriate for the deployment context—a 50 ms runtime on a $1,500 desktop GPU does not translate to real-time performance on a$ 500 embedded module running on battery power. This is the single largest gap between the paper's narrative ("bridging the gap to real-time deployment") and its experimental evidence.

6.2 The Pseudo-Labeling Pipeline Requires Two Additional Large Foundation Models at Inference Time for Training Data Curation

The assumption or constraint. The pseudo-labeling pipeline (Section 3.4) that produces the 1.4M in-the-wild training pairs depends on three separate foundation models operating simultaneously: FoundationStereo itself (the teacher, already the system being accelerated), UniDepthV2 (a monocular depth estimator; Piccinelli et al., 2025), and an open-vocabulary segmentation model based on SAM (Ravi et al., 2024) with extensions (Xu et al., 2023). The pipeline is described as an offline preprocessing step—the pseudo-labels are generated once, then used for training—but this masks an important assumption: generating those pseudo-labels requires inference from all three models on 1.4M stereo pairs from Stereo4D (which, after temporal subsampling at stride 10, implies processing ~14M raw frames, since 9 out of every 10 frames are discarded). The computational cost of this curation step—running FoundationStereo, UniDepthV2, and SAM on millions of images—is never quantified, and the availability of these specific models is implicitly assumed.

The consequence. Two distinct issues arise. First, the reproducibility cost: a practitioner wanting to replicate the full Fast-FoundationStereo training pipeline must first deploy three separate foundation models, each with its own GPU memory and runtime requirements, to generate training data. The compute spent on pseudo-label generation likely dwarfs the compute spent on student model training. If FoundationStereo runs at 496 ms per frame on a 3090, generating pseudo-labels for 14M frames would take approximately 1,900 GPU-hours for the teacher alone, plus additional time for UniDepthV2 and SAM—an enormous upfront cost that the paper does not disclose. Second, the method coupling: Fast-FoundationStereo's zero-shot generalization depends in part on training data generated by UniDepthV2 and SAM. If those models are unavailable (e.g., in a commercial deployment with licensing restrictions, or on a platform that cannot run them), or if they are superseded by better models whose output distributions differ, the pseudo-labeling pipeline cannot be replicated identically, and the resulting student model's performance may differ from reported numbers.

What evidence exists in the paper. Table 4 demonstrates that pseudo-labeling provides a consistent but modest improvement for Fast-FoundationStereo itself (Middlebury-H BP-2 drops from 2.53% to 2.20%, a 13% relative reduction). The gains are more dramatic for weaker baselines (RT-IGEV drops from 11.52% to 8.69%), suggesting the pseudo-labels primarily compensate for fundamental capability gaps rather than providing unique value to the accelerated model. The paper does not report the computational cost of the pseudo-labeling pipeline, the GPU-hours consumed, or an ablation showing whether similar gains could be achieved with simpler data augmentation or a smaller curated dataset. Section 3.4 describes the pipeline's steps but does not characterize its resource requirements.

Mitigation status. Partially mitigated by the fact that pseudo-labeling is an offline preprocessing step: the cost is incurred once, not at inference time. The paper also releases the pseudo-labels ("Our code, models and pseudo-labels will be released upon acceptance"), which eliminates the need for other researchers to re-run the pipeline. However, the release does not address the underlying limitation—the paper's method for generating training data depends on a constellation of large models, and if those models are unavailable or the domain shifts (e.g., to a new camera system with different intrinsics, where the normal consistency check requires recalibration), the pipeline is not self-contained. For practitioners who cannot use the released pseudo-labels (because they need to augment their own domain-specific data, for instance), replicating the pipeline requires deploying FoundationStereo, UniDepthV2, and SAM—a significant barrier.

6.3 The Difficulty Estimation Analogue Is Absent: No Per-Scene Adaptive Allocation Exists

The assumption or constraint. The paper produces a family of models spanning different speed-accuracy points on a Pareto frontier (Figure 2), and the user selects one model for deployment based on their latency budget. This implicitly assumes that the optimal accuracy-speed tradeoff is constant across all input scenes—every frame gets the same model, the same number of refinement iterations, and the same maximum disparity. The paper does not propose any mechanism for dynamically adjusting the compute budget per frame based on scene difficulty, despite scene difficulty varying enormously: a texture-rich indoor scene with clear edges may require far less refinement than a specular outdoor scene with transparent surfaces, as evidenced by the Booster results (Table 2) where errors spike on non-Lambertian surfaces.

The consequence. The user must choose between two suboptimal deployment strategies. Option A: deploy the fastest model that meets the worst-case accuracy requirement, meaning most frames (the easy ones) receive more compute than they need, wasting power and latency budget. Option B: deploy the most accurate model that meets the latency budget, meaning hard frames (specular, textureless, large depth range) produce unacceptably poor results while the model idles under capacity. Neither strategy is compute-optimal—a parallel to the difficulty-conditioned allocation problem from the reference paper on LLM test-time compute scaling, where uniform application of a budget-constrained strategy leaves substantial efficiency on the table. The paper's model family provides the Pareto frontier but provides no policy for navigating it dynamically, leaving the central insight of compute-optimal scaling entirely unexplored for stereo matching.

What evidence exists in the paper. The difficulty-dependent performance variation is clearly visible in the benchmarks, though the paper does not frame it as such. On Middlebury-H (indoor, structured light ground truth, mostly Lambertian), Fast-FoundationStereo achieves 4.80% BP-2. On Booster (non-Lambertian, specular, translucent), error rises to 6.61% BP-2—a 38% relative increase under the same model configuration. KITTI 2015 (outdoor driving, sparse LIDAR ground truth) shows D1 error of 3.25%, which is not directly comparable to Middlebury metrics but represents a different error distribution. The paper does not break down per-scene runtime or accuracy within a dataset, does not analyze whether refinement iterations could be reduced on easy scenes, and does not propose any difficulty estimator or adaptive budget allocation. The model family (Figure 2) is presented as a set of static configurations for the user to choose from, not as a space to navigate per-frame.

Mitigation status. Not addressed at all. The paper's focus is on producing a family of architectures that can meet different static latency targets, not on dynamic adaptation. This is consistent with the goals stated in the introduction and abstract, which frame the contribution as achieving zero-shot generalization at real-time speed for a given model configuration—not as optimal inference-time compute allocation. However, the absence of any discussion of per-scene adaptation is a notable gap given the wide variation in scene difficulty visible in the results and given that FoundationStereo itself (the teacher) uses a fixed number of refinement iterations and a fixed maximum disparity. A simple adaptation—reducing refinement iterations when the initial disparity confidence is high, or increasing the maximum disparity when the scene contains close objects—could potentially recover some of the accuracy lost to compression at minimal average latency cost, but this direction is unexplored.

6.4 The Blockwise NAS Independence Assumption Is Validated Only Weakly

The assumption or constraint. The blockwise NAS strategy (Section 3.2) reduces the search complexity from O(n^N) to O(n) by assuming that per-block accuracy and runtime impacts ( $\Delta m_i^c$ and $\Delta t_i^c$ ) are additive and that blocks can be optimized independently via distillation to match their teacher block's output without considering interactions between blocks. The ILP formulation (Equation 1) selects one candidate per block by summing their individual impacts, implicitly treating the cost filtering module as a linear system where the degradation from replacing block 3 with a lightweight candidate and block 5 with another lightweight candidate equals the sum of their individual degradations. The paper acknowledges this is a "surrogate objective" and a "proxy to the actual performance of a candidate model" (Section 4.4).

The consequence. If block interactions are substantial—for instance, if block 3's output representation changes in a way that block 5's lightweight candidate cannot effectively process, or if the hourglass skip connections create dependencies where the quality of early blocks affects the trainability of later blocks—then the ILP may select architectures that look good under the surrogate objective but underperform when assembled and trained end-to-end. The paper's validation (Figure 8) demonstrates that ILP-selected architectures outperform random assemblies, but this is a weak test: it shows the ILP is better than random, not that it is near-optimal. The 10 random architectures sampled per $\Delta\tau$ represent an infinitesimal fraction of the ~10^18 possible configurations, and the fact that the ILP beats them could simply mean that random architectures are mostly terrible—not that the ILP finds configurations close to the true Pareto frontier. A practitioner deploying the NAS in a different context (different teacher model, different search space) has no guarantee that the independence assumption transfers, because the paper provides no analysis of when it holds and when it fails.

What evidence exists in the paper. Figure 8 is the only validation experiment. No ablation tests the additivity assumption directly—for example, by comparing the ILP-predicted error for a configuration against its actual error after end-to-end training, and measuring whether prediction errors are systematic (certain block combinations consistently worse than predicted) or random. The paper does not report whether the ILP-selected blocks tend to come from certain block positions or certain candidate types, which would reveal whether the search is finding interpretable architectural patterns or merely exploiting quirks of the surrogate. The search space (five layer types, variable kernel sizes and channel dimensions) is described but not ablated—no experiment tests whether all five types appear in the ILP solutions or whether the search consistently favors certain operations, which would characterize the space and inform practitioners about which design dimensions matter.

Mitigation status. Partially mitigated by the end-to-end training step (Section 4.1): after the ILP selects block candidates, the assembled cost filtering module is trained together with the backbone and refinement module on the full training dataset. This end-to-end training can partially compensate for suboptimal block combinations by allowing adjacent blocks to adapt to each other's representations. However, end-to-end training is performed only for the final selected configuration, not for all candidates during search—the ILP still guides which architecture is chosen, and if the ILP ranking is misordered due to interaction effects, the final trained model may be substantially suboptimal even after adaptation. A more robust approach would use the ILP to generate a shortlist of promising candidates, train them all end-to-end, and select the best—but this would increase the search cost, partially defeating the purpose of the blockwise decomposition. The paper does not discuss this tradeoff.

6.5 Generalization to Non-Rectified or Uncalibrated Stereo Is Unexplored

The assumption or constraint. Fast-FoundationStereo inherits FoundationStereo's architecture, which operates on rectified stereo pairs with known camera intrinsics. Rectification warps the left and right images so that corresponding epipolar lines are horizontal and aligned, reducing the stereo correspondence problem from a 2D search to a 1D search along scanlines. This assumption holds for the evaluation datasets (Middlebury, ETH3D, KITTI, and Booster all provide rectified images), but it is violated in many real-world deployment scenarios: robots with roughly calibrated but not perfectly rectified stereo rigs, handheld devices where camera geometry flexes slightly, or any system where rectification error accumulates over time. The paper's "in-the-wild" claims (Abstract: "curate 1.4M in-the-wild stereo pairs"; Section 1: "embodied agents operating in in-the-wild environments") imply robustness to unconstrained capture conditions, but the architecture itself assumes rectified inputs.

The consequence. When deployed on imperfectly rectified stereo pairs, Fast-FoundationStereo's cost volume construction—which computes matching costs by correlating features along horizontal scanlines at each disparity level—will systematically fail for pixels where the true correspondence lies above or below the horizontal epipolar line. The Disparity Transformer provides some long-range context that might partially compensate for small rectification errors (by attending to nearby scanlines), but the 1D disparity search is hard-coded in the cost volume construction and cannot be circumvented by the learned components. The paper provides no characterization of how sensitive the model is to rectification error magnitude—does a 0.5-pixel vertical misalignment cause catastrophic failure, or graceful degradation? Without this information, a practitioner deploying on a real stereo rig with unknown rectification quality cannot estimate the model's reliability. This is especially concerning for the Booster dataset results (Table 2), where the model shows elevated error on non-Lambertian surfaces—some of that error may be due to rectification imperfections interacting with specular reflections, but the paper does not separate these effects.

What evidence exists in the paper. None. All evaluation datasets are rectified. The Stereo4D dataset used for pseudo-labeling is described as providing rectified stereo pairs, implying the pseudo-labeled data also assumes rectification. The paper does not test on any dataset with known rectification error, does not ablate sensitivity to vertical misalignment, and does not discuss rectification as a requirement or limitation. The term "rectified" appears only in the pseudo-labeling pipeline description (Section 3.4: "given a rectified stereo pair from Stereo4D"), not as an acknowledged constraint on the method's applicability.

Mitigation status. Not addressed. This is a limitation inherited from FoundationStereo and, more broadly, from the dominant paradigm in learning-based stereo matching (which overwhelmingly assumes rectified inputs). It is not a flaw specific to this paper, but it is a significant gap in the paper's narrative of in-the-wild deployment: "in-the-wild" should encompass imperfect calibration, yet the method provides no mechanism for handling it. Future work could address this by incorporating a rectification network into the pipeline, by extending the cost volume to handle small vertical disparities (a 2D or 2.5D search window), or by providing a calibration-quality detector that warns the system when rectification error exceeds a safe threshold. The paper does not suggest any of these directions.

6.6 The Training Data Mixture and Pseudo-Label Quality Are Not Controlled for Domain Leakage

The assumption or constraint. The paper trains Fast-FoundationStereo on "the same mixed datasets as FoundationStereo, as well as the pseudo-labeled real data" (Section 4.1). The composition of this mixed dataset is not specified—the reader is referred to FoundationStereo (Wen et al., 2025) for the exact training domains and their proportions. The pseudo-labels are generated from Stereo4D, a dataset of internet stereo videos whose domain distribution (indoor vs. outdoor, static vs. dynamic scenes, camera types, resolution, depth ranges) is not characterized in the paper. The models are evaluated in "zero-shot" mode on Middlebury, ETH3D, KITTI, and Booster—datasets that are described as "unseen to all the evaluated methods" (Section 4.3)—but whether Stereo4D or the FoundationStereo training mixture contains images visually similar to these evaluation domains is unknown.

The consequence. If Stereo4D or the FoundationStereo training mixture contains scenes that overlap in distribution with the evaluation datasets (e.g., indoor office scenes similar to Middlebury, outdoor driving scenes similar to KITTI), the claimed "zero-shot" generalization overstates the model's ability to generalize to truly novel domains. The improvement from pseudo-labeling shown in Table 4 could be partially explained by domain overlap rather than by the quality or diversity of the pseudo-labels per se. This matters for practitioners because the paper's central claim—that Fast-FoundationStereo works "off-the-shelf" without per-domain fine-tuning—depends on the evaluation datasets being genuinely out-of-distribution relative to training. If the training data inadvertently covers the test domains, a user deploying the model in a truly novel environment (e.g., underwater, aerial, medical imaging) may experience substantially worse performance than the benchmark numbers suggest.

What evidence exists in the paper. The paper provides no characterization of the training data distribution, the Stereo4D domain composition, or the similarity between training and evaluation domains. The temporal subsampling (stride 10) is described, but no statistics about Stereo4D's content are reported (e.g., indoor/outdoor ratio, scene type distribution, camera specifications). The datasets used for evaluation are standard benchmarks and are described as "unseen," but this only means they were not explicitly included in training—it does not guarantee distributional dissimilarity. The paper does not report nearest-neighbor analyses, FID scores, or any other distribution similarity metric between training and test domains.

Mitigation status. Not addressed. This is a common limitation in the stereo matching literature, where training data compositions are often treated as implementation details rather than as variables that affect the interpretation of zero-shot results. The paper's release of code, models, and pseudo-labels will partially mitigate this by enabling other researchers to inspect the training distribution, but the paper itself provides no analysis. A stronger evaluation would include results stratified by domain characteristics (indoor vs. outdoor, Lambertian vs. non-Lambertian, texture-rich vs. textureless) to reveal whether the model's generalization is uniform or domain-dependent, and an analysis of whether certain training domains disproportionately benefit certain evaluation domains. The Booster evaluation (Table 2) takes a step in this direction by isolating non-Lambertian surfaces, but no analogous stratification exists for the other benchmarks.

7. Implications and Future Directions

How This Work Changes the Landscape

Fast-FoundationStereo introduces a methodological shift in model acceleration, advancing the field from uniform compression strategies to component-aware heterogeneous acceleration guided by diagnostic analysis of each module's bottleneck type. This is not a paradigm shift in the sense of introducing a new model class—the paper explicitly builds on FoundationStereo and uses well-established techniques (knowledge distillation, NAS, pruning) that are individually standard. Rather, it is a reframing of the acceleration problem itself: instead of asking "how do we compress this model?", the paper asks "what is the specific nature of redundancy in each component, and which acceleration strategy matches that redundancy profile?" This diagnostic framework is the conceptual advance, and it is transferable to any multi-component dense prediction architecture.

The paper resolves a contradiction that has divided the stereo matching literature into two non-communicating research paths (Section 1). Generalizable models were assumed to require computationally intensive architectures—ViT backbones, dual-branch cost filtering, iterative attention-based refinement—as an inherent cost of zero-shot robustness. Real-time models were assumed to sacrifice generalization as an inherent cost of speed. By demonstrating that a 10× accelerated student can retain most of its teacher's generalization ability (Middlebury-H BP-2 of 4.80% vs. 2.49%), the paper shows that generalization is not intrinsically tied to expensive architectures; it is a property that can be distilled, searched, and pruned into efficient forms when the acceleration strategy respects each component's specific redundancy structure.

This has several immediate effects on research priorities:

The case for training monolithic efficient architectures from scratch weakens. Prior work on real-time stereo (LightStereo, RT-IGEV, BANet) treats efficiency as a ground-up design constraint, building architectures with lightweight convolutions throughout and training on whatever data is available. The paper's results show that starting from a powerful teacher and strategically compressing it yields dramatically better zero-shot generalization than designing for efficiency from scratch: Fast-FoundationStereo achieves 4.80% BP-2 on Middlebury-H vs. 23.76% for LightStereo-L, a 5× reduction in error, with comparable runtime. The implication is that future efficient architectures should be derived from foundation models through structured compression, not designed independently, at least for tasks where zero-shot generalization matters.
Pseudo-labeling via geometric consistency becomes a recognized complement to feature distillation. The paper's normal-consistency pipeline demonstrates that output-space supervision on real-world data (pseudo-labels) and feature-space supervision (backbone and block distillation) are complementary, with pseudo-labels providing the largest gains where feature distillation is weakest: on challenging materials and scene geometries that synthetic data cannot capture (Table 4 shows pseudo-labels reducing ETH3D BP-1 for LightStereo-L from 45.46% to 21.12%, a 54% improvement). This establishes a two-level distillation paradigm—transfer representations internally, transfer outputs externally—as a training recipe for compressed models that generalize.
Architecture search becomes viable for components with small channel dimensions. The paper's finding that cost filtering channels are "already small (mostly under 100)" (Section 3.2) and thus resist standard pruning is significant beyond stereo matching. Many vision architectures have bottleneck layers with compact representations—the neck of object detectors, the cost volume in optical flow, the feature matching layer in two-view geometry—where pruning provides minimal speedup. The blockwise NAS approach shows that architecture search, not pruning, is the right tool for these components, because it optimizes the structure of computation rather than merely removing parameters.
The difficulty-conditioned allocation problem is exposed as open territory for stereo matching. By showing that different scenes have different accuracy characteristics (Booster vs. Middlebury in Tables 1-2) and that the model family spans a speed-accuracy Pareto frontier (Figure 2), the paper implicitly demonstrates that no single model configuration is optimal for all inputs. This creates a natural research question—how to estimate scene difficulty and allocate compute adaptively—that the paper does not address but makes newly salient. The absence of any per-scene adaptation mechanism in the current work positions this as the logical next step, analogous to the compute-optimal test-time scaling problem in the LLM literature.

One research direction that becomes less attractive as a result of this work: developing hand-designed efficient stereo architectures that attempt to compete with foundation models on generalization through architectural innovation alone. The gap between hand-designed efficient methods and the compressed foundation model (4.80% vs. 37.49% BP-2 on Middlebury-H for LightStereo-L) is so large—nearly 8×—that incremental improvements to the hand-designed paradigm are unlikely to close it. The field's effort is better spent on improving compression techniques for foundation models, developing better pseudo-labeling pipelines, and understanding the fundamental accuracy ceiling of compressed models as a function of teacher quality.

Follow-Up Research This Work Enables

Dynamic per-scene compute allocation using confidence-based difficulty estimation. The paper produces a family of models with different speed-accuracy tradeoffs (Figure 2) but deploys them statically—every frame gets the same model, the same refinement iterations, and the same maximum disparity. A compelling follow-up would develop a lightweight scene difficulty estimator that runs in under 1–2 ms and predicts, from the initial cost volume or the first refinement iteration's disparity map, whether a scene is "easy" (few refinement iterations, low maximum disparity, lightweight backbone) or "hard" (full refinement, higher disparity range). The estimator could be trained on the pseudo-labeled Stereo4D data using the teacher's full-model error as ground-truth difficulty labels. The hypothesis to test: can adaptive allocation match the accuracy of the slowest Fast-FoundationStereo variant while running at the average speed of the fastest variant by spending compute only where needed? The key metric would be average runtime vs. accuracy on a mixed-difficulty benchmark combining Booster (hard) and Middlebury (moderate) scenes, compared against deploying a single fixed model on the entire benchmark. If adaptive allocation works, it would produce the stereo equivalent of the compute-optimal scaling curves from the LLM inference-time compute literature—a Pareto improvement over any static model configuration.

Diagnosing and improving the non-Lambertian accuracy gap through targeted distillation. Table 2 reveals that the accuracy gap between Fast-FoundationStereo and FoundationStereo is proportionally larger on non-Lambertian surfaces (Booster BP-2: 6.61% vs. 5.18%, a 28% relative increase) than on standard benchmarks (Middlebury-H BP-2: 4.80% vs. 2.49%, a 93% relative increase but from a much lower baseline in absolute terms—2.31 vs. 1.43 percentage points). This suggests the Disparity Transformer's long-range attention is disproportionately important for resolving specular and transparent surface matches, and compressing it via NAS degrades this specific capability. A targeted follow-up would ablate which components of the cost filtering search space (APC layers, transformer depth, number of attention heads) correlate with Booster performance, and then design a specialized distillation objective that emphasizes non-Lambertian surface regions during blockwise training—for example, by weighting the block distillation loss by a specularity mask derived from the teacher's attention maps or from the input image's gradient statistics. The experiment: compare two cost filtering modules with identical runtime but different configurations, one selected by the ILP under the standard uniform validation metric, and one selected under a Booster-weighted metric. Does the Booster-optimized module improve on specular scenes without degrading standard benchmark performance?

Stress-testing the blockwise NAS independence assumption with interaction-aware validation. The paper validates the blockwise NAS only by comparing ILP-selected configurations against random assemblies (Figure 8), which tests whether the surrogate is better than random but not whether it is near-optimal. A direct test of the independence assumption would take the ILP-selected architecture for a given latency budget and systematically permute its blocks—keeping the same set of 8 candidate blocks but reassigning them to different positions, then training each permutation end-to-end. If block interactions are negligible, all permutations should achieve similar accuracy; if interactions are substantial, some permutations will be significantly better or worse than others, and the ILP's position assignments (which assume additivity) are suboptimal. This experiment would not require new training infrastructure—the block candidates are already trained, and end-to-end assembly retraining follows the existing protocol. The result would either validate the independence assumption (strengthening the NAS methodology for broader adoption) or reveal interaction patterns that could be incorporated into a more sophisticated ILP formulation with pairwise terms. A negative result (strong interactions) would be equally valuable as a positive one, because it would characterize the limits of blockwise methods and guide practitioners on when to use them.

Extending the acceleration framework to teacher-student domain shift. Fast-FoundationStereo is evaluated exclusively on standard benchmarks (Middlebury, ETH3D, KITTI, Booster) but motivated by in-the-wild deployment. A critical open question is whether the distilled backbone and pruned refinement module generalize to domains that are genuinely out-of-distribution for both the teacher and the student—domains like underwater stereo, endoscopic imaging, or aerial photogrammetry, where visual appearance differs radically from the training data. The experiment: evaluate FoundationStereo (teacher) and several Fast-FoundationStereo configurations (lightweight, moderate, heavy) on a held-out domain that was not in the training mixture, measuring both absolute accuracy and the ratio of student-to-teacher error. Does the student preserve the same fraction of the teacher's accuracy on out-of-domain data as on in-domain data, or does compression amplify domain shift? If the student degrades more sharply, it would indicate that the distilled features capture domain-specific rather than truly generalizable representations—a finding that would motivate domain-robust distillation objectives. If the student and teacher degrade proportionally, it would validate that compression does not introduce domain brittleness, strengthening the case for deploying compressed foundation models in truly novel environments.

Combining Fast-FoundationStereo with self-supervised adaptation for long-term deployment. The paper's pseudo-labeling pipeline generates static training data from frozen teacher models. A natural extension is to close the loop: deploy Fast-FoundationStereo on a robot or AR device, use temporal consistency (disparity predictions should change smoothly across video frames) as an unsupervised quality signal, and fine-tune the student model online when temporal consistency degrades—indicating the model has encountered a domain where its predictions are unreliable. This would transform Fast-FoundationStereo from a static zero-shot model into a continuously adapting system that improves with deployment experience. The key technical challenge is designing an adaptation objective that improves performance on novel domains without catastrophically forgetting the generalization capability inherited from the teacher—possibly through elastic weight consolidation or a replay buffer of pseudo-labeled examples. The experiment: deploy on a long video sequence that transitions between indoor and outdoor environments, measure accuracy (against sparse LIDAR ground truth, if available) as a function of time with and without online adaptation, and track whether the adapted model maintains accuracy on previously seen environments.

Quantifying and reducing the pseudo-label generation cost through amortized difficulty estimation. The pseudo-labeling pipeline requires running three large models on 14M raw frames, representing an unquantified but likely enormous computational cost. A practical follow-up would characterize this cost precisely (GPU-hours, wall-clock time, energy consumption) and develop a filtering strategy that identifies which frames from Stereo4D are likely to produce useful pseudo-labels without running the full pipeline—for example, by training a lightweight classifier on image-level features (color histograms, gradient statistics, a small CNN) to predict the normal consistency score that the full pipeline would produce. Frames predicted to have low consistency could be discarded before running FoundationStereo and UniDepthV2, reducing total inference cost. The experiment: generate pseudo-labels for the full Stereo4D dataset, train the amortized filter on a subset, measure the correlation between predicted and actual consistency on a held-out subset, and compare the downstream training value (student accuracy) of pseudo-labels generated with and without pre-filtering. If a simple filter can discard 50% of frames while retaining 95% of the training value, the pipeline becomes significantly more practical for researchers without access to large GPU clusters.

Practical Applications and Downstream Use Cases

On-device depth estimation for consumer AR headsets. Current AR headsets (Meta Quest, Apple Vision Pro, Microsoft HoloLens) rely on active depth sensors (structured light, time-of-flight) or dedicated stereo matching ASICs to provide the low-latency depth maps needed for passthrough rendering, hand tracking, and spatial mapping. Fast-FoundationStereo at 21 ms with TensorRT (~48 FPS) on a desktop GPU suggests the architecture class could, with further optimization and quantization, run on the embedded GPUs inside these headsets, potentially eliminating the need for dedicated depth hardware. The zero-shot generalization is critical here: AR headsets operate in arbitrary indoor environments (living rooms, offices, factories) that cannot all be represented in a training dataset, and the model's demonstrated robustness to textureless walls and complex lighting (Figure 7) directly addresses failure modes of traditional real-time stereo. The benefit is both cost reduction (removing a depth sensor) and improved performance on challenging surfaces (transparent windows, specular objects) where active sensors fail. The missing piece is embedded benchmarking—a direct measurement on Snapdragon XR2 or Apple M-series hardware would determine whether the current architecture is viable or requires additional compression.

Vision-based obstacle avoidance for low-cost delivery robots. Last-mile delivery robots (Starship, Nuro, Amazon Scout) operate at low speeds (walking pace) in suburban environments and currently rely on LIDAR or ultrasonic sensors for obstacle detection, adding hundreds of dollars to the bill of materials. A stereo camera pair costs under $50 and provides dense depth information, but existing real-time stereo methods fail on the diverse visual conditions these robots encounter—direct sunlight, shadows, wet pavement, foliage. Fast-FoundationStereo's zero-shot performance on KITTI 2015 (D1 of 3.25%, comparable to FoundationStereo's 2.80%) and its demonstrated robustness to outdoor driving scenes in Figure 7 suggest it could replace LIDAR for close-range obstacle detection (under 20 meters, where depth errors of a few pixels correspond to acceptable distance errors). The 49 ms runtime on a 3090 would need to be reduced or the compute budget increased (a dedicated embedded GPU on the robot), but the accuracy numbers are already in the usable range: D1 of 3.25% means over 96% of pixels have correct disparity within the KITTI threshold, sufficient for detecting obstacles at safety-critical distances. The benefit is a 10–20× reduction in sensor cost with comparable depth quality in the near field.

Batch depth estimation for 3D scene reconstruction from internet stereo video. Platforms like YouTube and TikTok contain millions of stereo 3D videos (uploaded for VR viewing), which represent an enormous untapped source of 3D scene data for training embodied AI systems, generating synthetic environments, or building 3D maps. Processing this data with FoundationStereo at 496 ms/frame would require centuries of GPU-time for large-scale curation. Fast-FoundationStereo at 49 ms/frame (10× faster, with marginal accuracy loss) makes internet-scale stereo depth estimation economically feasible: processing 100 million frames drops from ~15,400 GPU-hours (FoundationStereo) to ~1,360 GPU-hours (Fast-FoundationStereo), a cost reduction from prohibitive to practical for a well-funded research lab. The pseudo-labeling pipeline's normal consistency check could be applied to filter out frames where even the accelerated model produces unreliable depth, ensuring quality of the curated dataset. The benefit is opening a new data source for 3D computer vision that was previously inaccessible due to the computational cost of generalizable stereo models.

When to Prefer This Method

The paper does not articulate an explicit tradeoff matrix against named alternatives that the user is expected to choose between at deployment time. The model family spanning the Pareto frontier in Figure 2 is the mechanism by which practitioners select a speed-accuracy point, but the paper frames this as an internal design choice (which of the Fast-FoundationStereo variants to deploy) rather than as a decision between Fast-FoundationStereo and a specific alternative method. The comparisons in Tables 1 and 2 are evaluative (showing that Fast-FoundationStereo outperforms existing real-time methods and competes with slow generalizable ones), not prescriptive. The paper does not provide guidance on when a practitioner should use Fast-FoundationStereo versus FoundationStereo, versus RT-IGEV, versus collecting domain-specific training data—it simply demonstrates that Fast-FoundationStereo occupies a previously empty region of the speed-accuracy space. A decision rule matrix is therefore not applicable without imposing a structure the paper does not provide.