Grounding World Simulation Models in a Real-World Metropolis

ArXiv: 2603.15583

🎯 Pitch

A video world model can now simulate kilometer-long traversals through actual city streets instead of imagined ones, but only after being explicitly trained to decouple permanent structures from ephemeral objects across mismatched photo references. Seoul World Model (SWM) achieves this by anchoring autoregressive generation to retrieved street-view snapshots and crucially introduces a Virtual Lookahead Sink that continuously injects future location cues, stabilizing long-horizon rendering without collapse.

---

1. Executive Summary

This paper introduces Seoul World Model (SWM), a city-scale world simulation model that grounds autoregressive video generation in the actual streets of Seoul through retrieval-augmented conditioning on nearby street-view images. SWM fine-tunes a pretrained video world model on 440k Seoul street-view images augmented with synthetic urban data from CARLA, addressing three challenges: temporal misalignment between static references and dynamic scenes via cross-temporal pairing (pairing references and target sequences captured at different timestamps to disentangle persistent structure from transient objects), training data sparsity via a synthetic dataset with diverse camera trajectories paired with a view interpolation pipeline using an intermittent freeze-frame strategy (synthesizing coherent video from sparse keyframes by repeating each keyframe to match the 3D VAE's temporal stride), and long-horizon error accumulation via a Virtual Lookahead Sink (dynamically inserting a retrieved future street-view frame as an attention anchor that re-grounds each generation chunk to upcoming locations). SWM outperforms six recent video world models on cross-city benchmarks in Busan and Ann Arbor, achieving a 28.43 FID on Busan-City-Bench compared to 49.63 for the best baseline, while maintaining stable generation over trajectories reaching hundreds of meters and supporting diverse camera movements and text-prompted scenario variations, establishing that real-world grounded world simulation is feasible only when models are explicitly trained to separate persistent geometric structure from transient dynamic content across temporally misaligned reference sources.

2. Context and Motivation

The Core Problem: World Models Imagine, but Don't Ground

The fundamental problem this paper tackles is that existing video world simulation models operate entirely within imagined environments. When a model like Cosmos [1] or Aether [63] generates a video of navigating city streets, every pixel beyond the initial frame—the geometry of unseen buildings, the layout of streets around the corner, the appearance of distant landmarks—is hallucinated by the model. This is an impressive technical achievement for open-ended generation, but it creates a critical gap: these models cannot be used to faithfully simulate a specific real city that physically exists.

This gap matters because many practical applications require not just visually plausible urban scenes but location-grounded simulation. The paper motivates this through several concrete use cases (Section 1):

• Urban planning visualization: A city planner wants to preview how a proposed streetscape change would look from the perspective of a pedestrian walking along a specific block. An imagined world model might generate generic "urban streets," but it cannot render the actual storefronts, building heights, and street configurations of, say, Jongno-gu in Seoul at coordinate 37.5718° N, 126.9770° E.
• Autonomous driving scenario generation: Testing self-driving perception systems requires diverse scenarios grounded in real-world road geometry. Synthetic simulators like CARLA [10] provide this, but they lack photorealism. A world model that could generate photorealistic driving video conditioned on real street geometry—while allowing variation in traffic, weather, and edge cases—would bridge the photorealism gap.
• Location-based exploration and entertainment: Users could navigate familiar city streets under novel conditions (a golden-hour sunset over their own neighborhood, a fictional event like Godzilla between skyscrapers they recognize) with faithful fidelity to the real location's appearance and layout.

The paper frames this distinction sharply in its abstract and introduction: "What if a world simulation model could render not an imagined environment but a city that actually exists?" This is not a marginal refinement—it's a fundamentally different design goal. An imagined world model succeeds if its output is visually plausible. A grounded world model succeeds only if its output is visually plausible and spatially faithful to a specific real location.

Why Prior Work Falls Short

The paper identifies four distinct lines of prior work, none of which addresses this goal:

1. Video world models (Section 2.2) generate imagined futures without external visual grounding. Models like Cosmos [1], Aether [63], DeepVerse [7], Yume1.5 [32], HY-World1.5 [20], FantasyWorld [8], and Lingbot [46] have achieved impressive capabilities: generating interactive visual observations conditioned on user actions across game environments [13, 45, 48], autonomous driving settings [23], and open-domain scenarios [20, 32, 46, 64]. Action representations range from discrete keyboard inputs [13, 45] to continuous camera trajectories [8, 23, 53, 59, 63, 64] and natural language [32]. Some incorporate persistent memory to maintain coherent world states over extended interactions [7, 25, 53, 59].

However, all of these models share a fundamental limitation: they operate end-to-end from a starting image, with no access to external observations about the real world. When the camera moves beyond what was visible in the first frame, the model must imagine everything from scratch. The paper states this crisply in Section 2.2: "existing world models operate entirely within imagined or synthetic environments, generating futures without grounding in external real-world observations. This becomes a key limitation when the simulated environment must stay faithful to a specific physical location." The quantitative results in Table 1 and Figure 8 demonstrate this concretely: when applied to the Busan and Ann Arbor benchmarks with retrieved street-view references provided as starting frames, all six baselines produce videos with substantially worse FID (49.63–141.24 on Busan), worse camera-following accuracy (RotErr 0.030–0.153), and worse structural fidelity (mPSNR 10.01–12.20) compared to SWM. Figure 8 shows qualitative examples where baselines drift in scene structure, produce blurred or collapsed output, or fail to maintain alignment with the target trajectory—precisely because they lack a mechanism to ground generation in real-world observations.

2. Geometry-aware video generation assumes static scenes from consistent inputs (Section 2.3). A separate line of work incorporates 3D geometric reasoning into video generation to improve spatial consistency. Novel view synthesis methods like Gen3C [34] and ViewCrafter [60] render point clouds from predicted depth to achieve geometric consistency for single-scene reconstruction. Recent world models integrate geometry through joint video-3D prediction [7, 8, 63], maintained 3D scene representations [24, 50], and spatial retrieval mechanisms that reuse previously generated context [7, 25, 36, 53, 59].

However, these approaches build geometric representations from either the model's own predictions or its generation history. They typically assume (1) that the scene is static or nearly static, and (2) that the multi-view images used as conditioning are temporally consistent with each other. This creates two mismatches for the real-world grounding setting. First, street-view images come from different capture timestamps—one image might show a parked car that is absent in another image of the same location taken months later. Static-scene geometry methods struggle with this temporal inconsistency because they assume all reference images depict a coherent static scene. The paper's comparison with Gen3C and VMem in Appendix C.5 (Table 4) confirms this: both models achieve competitive structural fidelity on static regions (Gen3C mLPIPS 0.524 vs. SWM 0.392, where lower is better, SWM wins; Gen3C mPSNR 14.16 vs. SWM 14.56), but produce significantly worse dynamic video quality (Gen3C FID 45.72 vs. SWM 28.43, FVD 416.94 vs. 301.76), with dynamic objects "frozen" in place rather than exhibiting natural motion.

3. Large-scale 3D reconstruction systems are static and lack generative capabilities (Section 1). Systems like Block-NeRF [44] and CityGaussian [29] model real cities at scale using neural radiance fields or 3D Gaussian splatting. These systems can render novel viewpoints of existing city scenes with high fidelity, but they are fundamentally interpolative: they reconstruct what was captured and cannot generate dynamic content, hypothetical scenarios, or text-conditioned variations. A Block-NeRF model of Seoul can render a street corner from a new angle, but it cannot simulate that corner being flooded, on fire, or populated with different vehicles and pedestrians. The paper positions SWM as bridging this gap: leveraging the generative capabilities of video world models while inheriting the spatial grounding that 3D reconstruction provides—but crucially, achieving this grounding through learned conditioning on retrieved references rather than explicit 3D reconstruction.

4. View interpolation for street-view data addresses temporal continuity but not free-form generation (Appendix D.2). A natural question is whether SWM is essentially doing "street-view interpolation"—taking sparse street-view images along a route and synthesizing smooth video between them. The paper explicitly distinguishes SWM from this task in Appendix D.2. Street-view interpolation takes a fixed sequence of images captured along a known trajectory and interpolates between them. The camera path is constrained to the original capture route, and dynamic objects visible in the input street-view images are interpolated as they appear.

SWM operates under fundamentally different constraints:

• The user specifies an arbitrary free-form camera trajectory, not necessarily one that follows a street-view capture route.
• The retrieved street-view references are captured at different timestamps, meaning their dynamic content (vehicles, pedestrians, signage) is mutually inconsistent across references.
• The model must generate a coherent dynamic scene—not interpolate between static snapshots—by learning to disentangle persistent structure from transient content.
• The user can provide text prompts that alter scene conditions (weather, time of day, hypothetical events like flooding or monster appearances), which street-view interpolation cannot support.

In other words, street-view interpolation takes temporally-consistent snapshots of what was and fills gaps. SWM takes temporally-inconsistent snapshots of what exists and generates what could be.

How This Paper Positions Itself

The paper frames its contribution as formalizing and solving a new task: real-world grounded video world simulation (the terminology is introduced in Section 1). This is not an incremental improvement to existing world models—it's a task that prior models were fundamentally not designed for, as evidenced by the consistent degradation of all baselines when applied to it.

The core insight that enables this task is that widely available street-view photographs provide a scalable source of location-specific visual references. Google Street View, NAVER Map, and similar services have already captured billions of geotagged, timestamped panoramic images covering urban areas worldwide. These images encode the persistent visual structure of real cities—building facades, road layouts, landmark appearances, typical vegetation—that a world model needs to ground its generation. The key technical challenge is how to use these references as conditioning, given that they are temporally misaligned with the dynamic scene being generated and are sparsely sampled.

SWM positions itself at the intersection of three research directions:

1. From video world models, it inherits the autoregressive chunk-by-chunk generation framework (specifically fine-tuning Cosmos-Predict2.5-2B [1]) and the ability to condition on camera trajectories and text prompts. But it extends this framework with retrieval-augmented conditioning that anchors each chunk to real-world observations.

2. From geometry-aware generation, it adopts the principle that known camera poses between references and targets enable geometric reasoning. But rather than requiring temporally consistent multi-view inputs or assuming static scenes, it introduces cross-temporal pairing to teach the model to separate persistent geometry from transient content, and it complements geometric warping (which provides spatial layout cues but loses fine detail) with semantic referencing (which preserves appearance detail through direct attention to reference images).

3. From 3D reconstruction of real cities, it adopts the idea that city-scale visual databases can ground spatial understanding. But rather than performing explicit reconstruction, it learns an implicit spatial representation through retrieval-augmented generation—the model learns to associate locations with their visual appearance through training, not through an explicit 3D representation.

The paper's position is that the primary bottleneck for real-world grounded simulation is not generative capability per se—Cosmos already generates high-quality videos—but rather the ability to disentangle persistent scene structure from transient dynamic content across temporally misaligned reference sources. This is why cross-temporal pairing is not merely a data augmentation trick but a fundamental design choice: without it, the model learns spurious correlations between references and targets (copying dynamic objects that happen to be co-temporal in training) and fails at inference when references come from arbitrary timestamps. The full architecture—geometric referencing for layout, semantic referencing for appearance, virtual lookahead sink for long-horizon stability, synthetic data for diverse trajectories, and view interpolation for temporal continuity—is designed to operationalize this disentanglement principle at scale.

The introduction concludes by stating the motivation in the most direct terms: "We hope this work encourages further exploration of world simulation that operates in the physical world beyond imagined environments." This positions the paper not as the final word on the topic, but as establishing the feasibility and the core technical challenges of a new research direction—one where generative models bridge the gap between imagination and physical reality.

3. Technical Approach

3.1 Reader Orientation

SWM is a video generation system that, given a starting geographic coordinate in Seoul, a desired camera trajectory, and a text prompt, produces a continuous video navigating through the real city as it actually looks—not as an AI imagines it. The problem it solves is that existing world models hallucinate everything beyond the first frame, so they cannot stay faithful to a specific physical location; SWM's solution is to continuously retrieve actual street-view photographs from a geo-indexed database and condition generation on them, but because those reference photos were taken at different times than the simulated moment, the model must learn to extract persistent geometric structure while ignoring transient content like cars and pedestrians that differ between the references and what should appear in the generated scene.

3.2 Big-Picture Architecture (Diagram in Words)

The system has seven major components, which interact in a pipeline:

1. Geo-indexed Street-View Database — 1.2M panoramic images of Seoul, each with GPS coordinates, capture timestamps, estimated depth maps, and 6-DoF camera poses. These are pre-processed once and stored for retrieval.

2. Street-View Retrieval Module — Given a target camera trajectory for the next generation chunk, searches the database for nearby street-view images using two-stage filtering: nearest-neighbor GPS search followed by depth-based reprojection filtering to ensure the retrieved view actually sees the target location.

3. Cross-Temporal Training Data Pipeline — During training, pairs target video sequences with reference images captured at different timestamps from the same location, forcing the model to learn that persistent structures (buildings, roads) are reliable while transient objects (cars, people) should be ignored.

4. View Interpolation Pipeline — Converts sparse street-view keyframes (captured every 5–20m) into temporally smooth training videos using an intermittent freeze-frame strategy that repeats each keyframe 4× to match the 3D VAE's temporal compression stride.

5. Synthetic Urban Dataset — 12.7K videos from the CARLA simulator spanning pedestrian, vehicle, and free-camera trajectories, each paired with synthetic street-view references rendered at 10m intervals. This compensates for the limited trajectory diversity in real street-view data.

6. Virtual Lookahead Sink — During autoregressive generation, retrieves a street-view image near the endpoint of each chunk and inserts it at a future temporal position in the latent sequence. This acts as a clean, error-free anchor that the model converges toward, preventing the accumulation of drift over long trajectories.

7. Autoregressive Video Diffusion Transformer (DiT) — The core generative model (fine-tuned from Cosmos-Predict2.5-2B), conditioned on noisy latents, history latents from the previous chunk, camera trajectory, text prompt, warped geometric references, and semantic reference tokens. It generates each chunk of 77 frames (TF) or 12 frames (SF) in a compressed latent space.

Information flows as follows: User specifies start coordinate + trajectory + text prompt → Retrieval module finds nearby street-view images → Geometric referencing warps the nearest reference into each target viewpoint → Semantic referencing injects encoded reference patches into the DiT's latent sequence → Virtual Lookahead Sink adds a future retrieval as an attention anchor → DiT generates the chunk autoregressively, conditioning on self-generated history from the previous chunk → Process repeats for the next chunk along the trajectory.

3.3 Roadmap for the Deep Dive

• First, the training data construction pipeline (street-view collection, cross-temporal pairing, view interpolation, synthetic data generation, and annotation), because the data strategy is the foundation that enables all subsequent mechanisms—without cross-temporal pairing, the model would learn the wrong correlations, and without synthetic data, trajectory diversity would be insufficient.
• Second, the street-view retrieval mechanism at inference time, since it is the entry point for grounding and determines which real-world observations the model receives as conditioning.
• Third, the geometric and semantic referencing pathways, which are the two complementary mechanisms by which retrieved references influence generation—geometric warping for spatial layout, semantic injection for appearance detail.
• Fourth, the Virtual Lookahead Sink, which addresses the compounding error problem specific to retrieval-augmented autoregressive generation over city-scale distances, and which represents a novel attention mechanism design.
• Fifth, the base model architecture and training configuration (DiT backbone, fine-tuning from Cosmos, Teacher Forcing vs. Self-Forcing variants, hyperparameters, and conditioning dropout), which ties together all the conditioning signals into a single generative model.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and methods paper whose core idea is that real-world grounded video world simulation is made possible by retrieval-augmented conditioning combined with explicit mechanisms for disentangling persistent scene structure from transient content across temporally misaligned reference sources. The paper does not propose a new generative architecture—it fine-tunes an existing DiT—but rather develops the data pipeline, retrieval strategy, conditioning pathways, and attention mechanism that collectively enable location-grounded generation.

---

Training Data Construction: The Street-View Dataset

The foundation of SWM is a training dataset that teaches the model how to use street-view images as conditioning for video generation. The raw material is 1.2M panoramic street-view images collected from NAVER Map covering the major urban areas of Seoul (Figure 11 shows the coverage area, extending approximately 44.8 km east-west and 31.0 km north-south, focusing on densely populated districts). Each image carries GPS coordinates and a capture timestamp as metadata. License plates and pedestrians are blurred for de-identification prior to use. After filtering and processing, 440K images are used for training (Section 3.1).

Cross-temporal pairing: the key data design choice. A training sequence consists of N consecutive street-view images along a route, which serve as the target sequence for supervision, paired with K spatially nearby panoramas that serve as reference conditioning (Section 3.1). Each panorama is rendered into a pinhole view: target sequences are rendered facing the forward driving direction with a random yaw rotation within ±90°, while references are rendered to match the viewing direction of the paired target frame.

The paper identifies a critical design decision in how references are selected: they must come from a different capture timestamp than the target sequence. This is called cross-temporal pairing.

Why this matters: if references and targets are co-captured (same vehicle pass, same timestamp), they share identical transient content—the same parked cars, the same pedestrians, the same temporary signage. During training, the model would observe a spurious correlation: the reference image contains a specific red car, and the target video also contains that red car, so the model learns to copy the car from the reference into the generated output. At inference time, however, references come from arbitrary past capture timestamps, and their transient content is inconsistent with the scene being generated. If the model learned to copy dynamic objects, it would produce abrupt appearances and disappearances of vehicles, pedestrians, and other transient elements as it moves between chunks conditioned on different references.

Cross-temporal pairing eliminates this by ensuring that during training, the transient content in references and targets is always different. The model has no choice but to learn that persistent structures (buildings, road layouts, permanent signage, vegetation patterns) are reliable conditioning signals while transient objects (vehicles, pedestrians, temporary signs) should be ignored. The paper's Figure 6 visualizes the resulting attention pattern: with cross-temporal pairing, the model's attention is distributed across the persistent scene structure in the reference image; without it, attention concentrates on dynamic objects like cars. The ablation in Table 2 confirms the quantitative impact: removing cross-temporal pairing causes the largest degradation across all metrics (FID increases from 28.43 to 44.74, mPSNR drops from 14.56 to 12.54, TransErr increases from 0.015 to 0.123), indicating that the model fails to ground generation in real geometry when it cannot separate persistent from transient content.

View interpolation: from sparse keyframes to smooth video. City-scale street-view databases provide panoramic images at sparse spatial intervals (typically 5–20 m between consecutive views along a route), not continuous video. Training a video generation model directly on such sparse sequences is problematic because pretrained video diffusion models (and their 3D VAEs) are designed for temporally smooth, continuous motion; abrupt jumps between distant viewpoints violate this assumption and would produce training signals inconsistent with the model's learned temporal priors.

SWM therefore develops a view interpolation pipeline that synthesizes T-frame videos from N sparse keyframes, where T ≫ N (Section 3.1). The pipeline uses a pretrained latent video generative model (Cosmos-Predict2.5-2B [1], the same model SWM later fine-tunes) to hallucinate smooth transitions between keyframes. The key technical challenge is how to inject the keyframe constraint into the diffusion process so that the generated video stays faithful to the known frames.

The paper first considers a straightforward approach: encode each keyframe's latent and concatenate it along the channel dimension at the corresponding temporal position in the noisy input, with zero-padding at non-keyframe positions. This requires widening the DiT's input projection from the standard 16+2=18 channels (16 latent channels + 2 optional conditioning channels) to 34 channels (to accommodate the additional keyframe latent channels). However, the authors observe that this approach "yields weak adherence to the keyframes, with generated frames deviating from the inputs" (Section 3.1). They attribute this to a mismatch with the 3D VAE's temporal compression: the VAE encoder compresses every 4 consecutive frames into a single latent representation, so an isolated single-frame keyframe does not form a valid 4-frame group. When the encoder processes a 4-frame window that contains only 1 real frame and 3 zero-padded frames, the resulting latent is a degraded representation of the keyframe's content.

To address this, SWM proposes the intermittent freeze-frame strategy. The procedure works as follows:

1. During training data construction: At each keyframe position in the pixel-space video, the keyframe is repeated 4 consecutive times. The resulting training video alternates between smooth motion segments (between keyframes, generated by the interpolation model) and brief freeze segments (at keyframe positions, where the same frame appears 4 times). Because each keyframe now forms a complete 4-frame group, the 3D VAE encodes it into exactly one clean latent that faithfully represents the keyframe's appearance.

2. During inference (when using the pipeline to generate training data for SWM): Each input keyframe is similarly repeated 4 times and encoded into a single clean latent via the 3D VAE encoder. This clean latent then replaces the corresponding position in the noisy input latent of the diffusion model at every denoising step, ensuring exact keyframe conditioning throughout the diffusion trajectory. Because the latent at that position is never denoised from noise—it is directly substituted with the clean encoded keyframe—the generated output is guaranteed to reproduce the keyframe exactly at that timestamp.

3. After generation and decoding: The three repeated frames per keyframe are discarded, recovering a video with the intended temporal length where keyframes appear once at their correct positions and the interpolated frames fill the gaps.

Appendix C.1 (Table 3) quantifies the improvement: on the Waymo test set, the intermittent freeze-frame strategy achieves PSNR 25.03 vs. 22.52 for channel concatenation, SSIM 0.703 vs. 0.628, and LPIPS 0.162 vs. 0.245. The interpolated videos serve as the target supervision for training SWM on street-view data. The paper acknowledges (Appendix D.1) that these interpolated videos "remain lower in quality than real captured video" and that incorporating real video data as it becomes available would further improve generation quality.

Temporal filtering and metadata challenges. The paper notes that street-view images are captured at equal distance intervals, not equal time intervals. When the capture vehicle slows down or stops (e.g., at traffic lights), consecutive street-view frames can span a large temporal gap. Although the authors filter sequences based on capture-time metadata, noisy metadata causes some temporally inconsistent sequences to pass the filter. When these sequences are converted into interpolated training video, dynamic objects may abruptly appear or disappear between frames. This artifact propagates into SWM, which "occasionally generates sudden appearance or disappearance of vehicles" (Appendix D.1, Figure 15 shows examples).

Depth and camera pose estimation. To enable the geometric referencing and semantic referencing pathways (Section 4.3), each street-view image needs a depth map and a 6-DoF camera pose in a globally consistent coordinate system. The paper estimates these using Depth Anything V3 (DA3) [27]. For each driving sequence, target pinhole keyframes and reference panorama images (rendered into eight directional views uniformly covering 360°) are jointly fed into DA3, which estimates scale-consistent depth maps and relative camera poses across all images in a single forward pass.

For longer sequences exceeding DA3's capacity, the sequence is partitioned into non-overlapping chunks processed independently. To recover real-world metric scale and align chunks into a shared coordinate system, each chunk's camera poses are aligned to real-world coordinates using GPS metadata. Specifically, a similarity transformation is estimated by matching the camera displacement from the first to the last frame of each chunk in the DA3 coordinate frame with the corresponding displacement derived from GPS coordinates. Metric depth is then obtained by scaling the affine-invariant DA3 depth with the estimated scale factor. Because all images in the database are processed through DA3 with GPS-based metric alignment, they share a globally consistent coordinate system—this is essential for retrieval at inference time, since the virtual lookahead sink and semantic references retrieved from different capture sessions must have compatible camera poses.

Text captioning. Each training video is captioned using Qwen2.5-VL-72B [2] with a structured prompt that instructs the model to produce both a long caption (up to 280 words) and a short caption (up to 30 words), covering urban scenery, dynamic actors, environmental conditions, specific events, and camera trajectory. During training, SWM randomly selects between the long and short caption variants. Additionally, a predefined camera-action sentence describing the trajectory direction (straight, left turn, right turn, stop) is prepended to the caption, derived from the camera pose sequence. This provides the model with explicit linguistic cues about the expected motion pattern, complementing the continuous camera trajectory conditioning.

Stylized video augmentation. The paper observes that text prompts describing events occurring within a scene (flooding, fire, monster appearance) generalize well to the fine-tuned model because these capabilities are "largely inherited from the pretrained world simulation model" [1]. However, prompts involving global style changes such as day-to-night transitions or weather variations tend to be less faithfully followed after fine-tuning on street-view data, likely because the street-view training distribution is dominated by daytime, clear-weather captures.

To address this, the authors construct a video stylization pipeline. Given a style-related text prompt, Qwen-Image-Edit [52] first edits the starting frame to reflect the target style, and TeleStyle [62] then propagates this style consistently across the entire video using a content-preserving style transfer approach. This pipeline augments a subset of the interpolated street-view videos with diverse style prompts, producing 10K additional stylized training videos that increase the diversity of lighting, weather, and atmospheric conditions in the training distribution.

---

Training Data Construction: The Synthetic Urban Dataset

Real street-view data has an inherent limitation: it is captured by vehicle-mounted cameras following road networks. This restricts both the types of camera trajectories (predominantly forward-facing driving along roads) and the camera motions (no pedestrian paths, no free-form exploration, no looking around at intersections). To complement this with diverse camera paths, SWM constructs a synthetic dataset using CARLA [10], an Unreal Engine-based urban driving simulator (Section 3.2).

Scale and composition. The synthetic dataset comprises 12.7K videos rendered from 6 urban maps (Town01 through Town06) spanning approximately 431,500 m² of city area. Three trajectory types are collected:

1. Pedestrian trajectories: First-person videos rendered from autonomous pedestrian agents, covering sidewalk movement, street crossing, and similar on-foot paths. These introduce camera motions (height, speed, viewpoint) absent from the vehicle-mounted street-view data.
2. Vehicle trajectories: Driving-perspective videos captured across diverse road types including highways, urban streets, and elevated roads. These cover lane changes, turns, and straight driving, providing complementary road configurations to the Seoul street-view data.
3. Free-camera trajectories: Random paths that freely navigate the scene while avoiding collisions with buildings, terrain, and other scene geometry. These are the most diverse, allowing camera motions that don't correspond to any real agent (e.g., flying through intersections, orbiting buildings).

Random vehicles and pedestrians are spawned before rendering to simulate realistic traffic. Depth maps and camera parameters are extracted per frame to support geometric and semantic referencing. Because CARLA natively operates in real-world metric scale, the rendered depth maps and camera poses are directly compatible with the GPS-aligned metric geometry used for the street-view data—no scale alignment is needed.

Street-view references in simulation. For each map, synthetic street-view reference images are rendered at regular intervals of 10 m along all roads, with 8 directional views per location (uniformly covering 360° horizontal field of view). Slight positional jitter is applied to the sampling interval, and the lane position is varied on multi-lane roads to reduce the sim-to-real gap by mimicking the natural variation in real street-view capture positions. In total, this yields 4K street-view positions and 32K reference frames across all maps.

Cross-temporal pairing in simulation. To apply the same cross-temporal pairing principle as the real data, target videos and street-view reference images are divided into multiple subsets, and each subset is rendered under distinct combinations of traffic patterns, lighting conditions, and weather. This ensures that even though the references and targets depict the same simulated locations, their transient content (which cars are where, what pedestrians are doing, the lighting state) differs. The model consistently encounters a temporal gap between references and targets across all data sources, reinforcing the disentanglement of persistent structure from transient content.

Ablation impact. Table 2 shows that removing synthetic data leads to a mixed effect: FID improves slightly (27.74 vs. 28.43 for the full model), but camera-following accuracy and 3D adherence degrade (RotErr 0.021 vs. 0.020, TransErr 0.020 vs. 0.015, mPSNR 13.52 vs. 14.56). The improved FID is expected: synthetic data has a domain gap from real photos, so including it slightly worsens distributional metrics. However, the degraded camera-following and structural fidelity indicate that the diverse trajectories provided by synthetic data are essential for the model to learn to handle camera motions beyond standard driving paths. The paper states that the model "no longer learns diverse trajectories during training" without synthetic data.

---

Training Data Mix and Additional Driving Video

In addition to the Seoul street-view data (processed through view interpolation) and the synthetic CARLA data, SWM incorporates the publicly available Waymo Open Dataset [43], which contains real-world driving videos with diverse traffic scenarios. This increases scenario diversity beyond what the static street-view captures can provide.

The three training datasets are mixed via ratio-based interleaved sampling (Appendix A.1): Waymo at 20%, Seoul street-view at 40%, and synthetic at 40%. This roughly follows the relative dataset sizes. Waymo samples never include street-view references—they serve as standard video prediction training data, teaching the model the basic dynamics of driving scenes without the added complexity of retrieval-augmented conditioning. For street-view and synthetic data, reference conditions are zeroed out with a probability of 20% during training (part of the conditioning dropout schedule described later). This mixed training strategy ensures the model can generate coherent videos both with and without reference conditioning, which is important for handling sparse retrieval scenarios where few references are available.

---

Street-View Retrieval at Inference Time

At inference time, given a user-specified starting location, camera trajectory, and text prompt, SWM must retrieve relevant street-view images from the pre-built geo-indexed database to condition each generation chunk. The retrieval database consists of the 1.2M collected panoramic images, each rendered into 8 equi-angular pinhole views (yielding approximately 9.6M pinhole images) with metric-scale depth maps and 6-DoF camera poses estimated via the DA3 pipeline described above.

Two-stage retrieval. For each generation chunk i with target camera trajectory C(i) = {c_t}_{t=0}^{T-1} (where c_t is the camera pose at frame t, and T is the chunk length in frames—77 for TF, 12 for SF), the retrieval proceeds in two stages (Section 4.1):

1. Nearest-neighbor GPS search: Candidate street-view locations along the target trajectory are identified by nearest-neighbor search in GPS coordinates. This narrows the search from millions of images to a small set of spatially relevant candidates.

2. Depth-based reprojection filtering: Among the candidates, only those whose projected pixels exceed a coverage threshold in the nearest target view are retained. This filtering works by taking each candidate reference image, lifting its pixels to 3D using its estimated depth map, and projecting those 3D points into the target camera viewpoint. If a sufficient fraction of the target view's field of view is covered by the projected points (indicating that the reference actually sees the same scene region), the reference is kept; otherwise, it is discarded. This prevents the model from being conditioned on references that are geographically nearby but face the wrong direction (e.g., a reference facing south when the target is looking north at the same intersection).

The retrieval yields up to K pinhole references X^(i)_ref = {x^(i)_{ref,k}}_{k=0}^{K-1}, each with its camera pose c^(i)_{ref,k} and depth estimate d^(i)_{ref,k}. Each reference is aligned to the viewing direction of the matched target viewpoint. During training, K=5 references are used; at inference, the number can be varied, with Appendix C.3 exploring the effect of reducing K down to 1.

Ensuring no information leakage. The evaluation benchmarks are constructed so that retrieved references exclude any street-view image belonging to the test sequence itself. This means the model never sees the ground-truth viewpoint during generation—it must reconstruct the scene entirely from nearby but non-identical reference views captured at different times. This is a stringent test of the model's ability to generalize from partial, temporally misaligned observations.

---

Geometric and Semantic Referencing

Once K references are retrieved, they condition the generation through two complementary pathways (Section 4.3). The key insight is that geometric warping and appearance injection serve different purposes and have complementary failure modes: warping provides dense spatial layout cues but loses fine appearance detail due to depth errors and occlusion; injecting original references preserves appearance detail (e.g., building facade textures, signage, lighting) but lacks explicit pixel-level spatial alignment with the target view. SWM uses both pathways simultaneously so that each compensates for the other's weaknesses.

Geometric referencing: depth-based reprojection. For each target frame x^(i)_t to be generated, the spatially nearest reference x^(i)_{ref,j} among the K retrieved references is identified. This reference is then reprojected into the target viewpoint via depth-based forward warping (splatting):

$x^{(i)}_{\mathrm{warp}, t} = \mathrm{Render}\bigl(\mathrm{Unproj}(x^{(i)}_{\mathrm{ref}, j},\; d^{(i)}_{\mathrm{ref}, j}),\; c^{(i)}_{\mathrm{ref},j \to t}\bigr)$

where x^(i)_{ref,j} is the reference image, d^(i)_{ref,j} is its estimated depth map, c^(i)_{ref,j→t} is the relative camera transformation from the reference viewpoint to the target viewpoint at frame t, Unproj(·) lifts the reference image pixels to 3D points in world space using the depth map, and Render(·) projects those 3D points into the target camera and renders them as a new image.

What this computes: given a reference image of a scene from viewpoint A with known depth, and a desired new viewpoint B, it synthesizes what the scene would look like from B by geometrically re-projecting every visible pixel. The output x^(i)_{warp,t} is a warped image at the same resolution as the target frame, where pixels that are visible in both views get their color from the reference, and pixels that are occluded or out of view in the reference are left blank.

Why this approach: forward splatting with known depth provides an explicit geometric cue about the spatial layout of the scene from the target viewpoint. Even when the warped image is noisy (due to depth estimation errors, missing regions from occlusion, or boundary artifacts), it conveys crucial structural information—where buildings are, how roads curve, what the relative scale of objects should be. Because the warped image is spatially aligned with the target frame (it's rendered at the target resolution from the target viewpoint), the model can use it as pixel-level layout guidance. Alternative approaches that don't provide pixel-aligned spatial cues (e.g., encoding the reference as a latent vector without warping) would lose this explicit geometric alignment.

Each target frame uses only its single nearest reference for warping, rather than fusing multiple references into one warped view. This avoids the noisy artifacts that arise when multiple images with slightly inconsistent depth estimates and color balances are simultaneously splatted into the same view—a practical engineering choice that prioritizes clean conditioning over completeness.

The warped video X^(i)_warp = {x^(i)_{warp,t}}_{t=0}^{T-1} (where T is the chunk length in frames) is encoded by the 3D VAE into a compressed latent representation, then channel-wise concatenated with the noisy target latent at the DiT input. This means the DiT's first layer receives 16 noisy latent channels (from the target) + an encoded warped video with the same spatial and temporal dimensions, with the warped conditioning providing a clean, geometric reference signal at every spatial position and every frame. During training, the warped video inputs are replaced with zeros with a probability of 20% (as part of the conditioning dropout schedule), ensuring the model can generate without geometric referencing when warped views are unavailable.

Semantic referencing: direct appearance injection. While geometric referencing provides spatial layout, it loses fine appearance detail—textures get blurred or distorted in the warping process, and depth errors cause misalignments. To preserve appearance, the original reference images are injected directly into the transformer's latent sequence.

Each reference image x^(i)_{ref,k} is encoded by the 3D VAE into a single latent z^(i)_{ref,k} (since each reference is a single frame, it is repeated 4 times before encoding to match the 4-frame temporal group expected by the VAE, then the resulting latent represents one frame's content). This latent is then patch-embedded (split into spatial patches and linearly projected, same as the target latents) and concatenated with the target latent tokens along the temporal axis.

The concatenated sequence fed to the DiT is:

$\mathbf{Z}^{(i)}_{\mathrm{seq}} = \Bigl[ \mathbf{Z}^{(i)}_{\mathrm{hist}};\; \mathbf{Z}^{(i)}_{\mathrm{target}};\; \mathbf{z}^{(i)}_{\mathrm{ref},0};\; ...;\; \mathbf{z}^{(i)}_{\mathrm{ref},K-1}\Bigr]$

where Z^(i)_hist are the H history latents from the previous chunk's output (providing temporal continuity), Z^(i)_target are the L noisy target latents to be denoised, and each z^(i)_{ref,k} is the encoded and embedded representation of the k-th reference image.

The reference tokens are assigned RoPE (Rotary Position Embedding) temporal positions at a large offset from the generation window:

$p^{(i)}_{\mathrm{ref},k} = H + L + G + k\Delta_{\mathrm{ref}}$

where H is the number of history latents, L is the number of target latents per chunk, G is a large temporal gap (G=50 in the TF configuration) that separates references from the generation window, and Δ_ref is the inter-reference spacing.

Why this form: placing the references at a large temporal offset (G=50) tells the model through positional encoding that these tokens are "far away" in time from the frames being generated. This is important because the reference images were captured at arbitrary past times—they are not temporally adjacent to the scene. By giving them distant RoPE positions, the model learns to treat them as contextual information rather than as temporally preceding frames that should smoothly transition into the generation. If references were placed at positions close to the target latents, the model's temporal attention patterns (which are shaped by RoPE to attend more strongly to nearby positions) might incorrectly attempt to maintain temporal continuity with the static reference content.

Camera pose encoding. Camera poses for all latents—target, reference, and virtual lookahead sink—are encoded via Plücker ray embeddings. For each pixel in each frame, a 6-channel Plücker ray map is constructed from the camera extrinsics (rotation and translation) and intrinsics (focal length, principal point), encoding the 3D ray direction and origin for every pixel. This ray map is fed through a shallow convolutional encoder that projects it into the latent space, and the resulting camera embeddings are added as residuals to both the main video tokens (target + history) and the reference tokens. This ensures that every token in the transformer sequence carries geometric information about the viewpoint from which it was captured or should be generated, enabling the attention mechanism to relate tokens based on both visual similarity and geometric compatibility.

Interaction between the two pathways. The geometric and semantic pathways are designed to be complementary. During generation, the DiT receives the warped video (providing dense, spatially aligned layout cues at every pixel) concatenated with the noisy target latent at the input level, and simultaneously attends to the original reference tokens (providing high-fidelity appearance information) through the self-attention layers. The model can use the warped input to establish the coarse spatial structure of the scene, then query the reference tokens via cross-attention-like mechanisms (since all tokens are in a unified sequence with RoPE positions, attention between target and reference tokens is just standard self-attention) to recover specific appearance details—the color and texture of a particular building facade, the exact signage on a storefront, the characteristic shape of a landmark.

The ablation in Table 2 quantifies the importance of each pathway. Removing geometric referencing (keeping only semantic) causes FID to increase from 28.43 to 33.01, RotErr to increase from 0.020 to 0.036, TransErr to increase from 0.015 to 0.051, and mPSNR to drop from 14.56 to 12.33. This indicates that without explicit spatial layout cues, the model struggles to maintain geometric alignment with the target trajectory and real scene structure. Removing semantic referencing (keeping only geometric) causes FID to increase from 28.43 to 30.27, mPSNR to drop from 14.56 to 14.08, and mLPIPS to increase from 0.392 to 0.442. The milder degradation in geometric metrics but larger drop in appearance quality (mLPIPS) confirms that semantic referencing is primarily responsible for preserving fine visual detail. Figure 9 qualitatively shows that without semantic referencing, the generated video loses appearance fidelity (building textures, signage clarity) compared to the full model.

How cross-temporal pairing enables these pathways. The geometric and semantic referencing mechanisms would be vulnerable to overfitting on transient content without cross-temporal pairing. If the model were trained with co-temporal references (where dynamic objects match between reference and target), it might learn to directly copy vehicle and pedestrian appearances from the reference tokens into the generated output. With cross-temporal pairing, this copying behavior is penalized during training because the dynamic content in references doesn't match the target, so the model learns to attend to what is consistent across timestamps: the persistent geometric structure and appearance of buildings, roads, and static urban elements. Figure 6 shows the attention pattern difference: with cross-temporal pairing, attention is distributed across the scene structure; without it, attention concentrates on dynamic objects like cars. This is why cross-temporal pairing is not just a data augmentation detail but a foundational requirement for the referencing pathways to work correctly.

---

Virtual Lookahead Sink

Autoregressive generation accumulates errors across chunks. Each chunk conditions on H history latents from the tail of the previous chunk's output. Any small error in the previous chunk—a slight misalignment in building placement, a subtle drift in camera motion—becomes part of the conditioning for the next chunk, which may amplify the error, which then feeds into the subsequent chunk, and so on. At the city scale, where the camera may travel hundreds of meters, this compounding drift causes progressive misalignment between the retrieved street-view references and the generated scene: the model's internal representation of where it is drifts away from the actual GPS coordinates, so the references retrieved for later chunks depict the correct location but no longer align with what the model is generating.

Prior work in long-horizon video generation [28, 40] mitigates this with an attention sink [54]: a fixed token (typically the first frame of the video) that persists throughout the entire generation, providing a stable global context that all chunks can attend to. In the standard formulation, the first frame's latent is included in every chunk's latent sequence with a fixed RoPE position (typically position 0), acting as an immutable anchor. This works well when the camera stays near the starting location—the first frame remains relevant—but becomes problematic as the camera moves farther away. In SWM's setting, after the camera travels several hundred meters, the first frame depicts a completely different location than the current generation chunk; it provides no useful spatial guidance and may even confuse the model by introducing visual information from an irrelevant location.

SWM proposes a Virtual Lookahead Sink tailored for retrieval-augmented long-horizon generation (Section 4.2). The core idea is to dynamically update the sink with a retrieved street-view image that is spatially near the endpoint of the current generation chunk, treating it as a virtual future destination that the model should converge toward.

Mechanism. For the i-th generation chunk with target trajectory C(i) = {c_t}_{t=0}^{T-1}, let the endpoint be c^(i)_{T-1}. SWM retrieves the nearest street-view image to this endpoint position from the database and encodes it into a single latent z^(i)_VL (using the same 4× frame repetition trick to match the VAE). This latent is appended to the token sequence:

$\mathbf{Z}^{(i)}_{\mathrm{seq}} = \Bigl[ \mathbf{Z}^{(i)}_{\mathrm{hist}};\; \mathbf{Z}^{(i)}_{\mathrm{target}};\; \mathbf{z}^{(i)}_{\mathrm{VL}} \Bigr]$

with corresponding RoPE temporal positions:

$\mathbf{p}^{(i)}_{\mathrm{seq}} = \Bigl[ \underbrace{1,\dots,H}_{\text{history}};\; \underbrace{H{+}1,\dots,H{+}L}_{\text{target}};\; \underbrace{H{+}L{+}\Delta_{\mathrm{VL}}}_{\text{sink}} \Bigr]$

where H is the number of history latents, L is the number of target latents in the current chunk, and Δ_VL is a temporal offset hyperparameter (set to 5 in both TF and SF configurations).

What this computes: the model receives a sequence of latents consisting of the previous chunk's output tail (H tokens at positions 1 through H), the noisy latents to be denoised for the current chunk (L tokens at positions H+1 through H+L), and a single clean latent from a retrieved future street-view image (1 token at position H+L+Δ_VL, which is 5 positions beyond the last generated frame). The large temporal gap (Δ_VL=5) tells the model via positional encoding that this sink token is a future destination, not an immediately adjacent frame.

Why this design: by placing a clean, error-free frame ahead of the chunk being generated (at a sufficient temporal gap), the model has a stable anchor to converge toward. The sink is not a reconstruction target—the model does not try to generate the exact frame at the sink position—but rather a guiding signal that constrains the generation to remain spatially faithful to the real location. Because the sink is retrieved from a spatially nearby location (near the chunk's endpoint), the grounding remains relevant to the region being generated. This contrasts with the standard first-frame attention sink, which becomes spatially irrelevant after the camera moves.

Crucially, the sink is refreshed for each chunk. For chunk i, the sink is retrieved near the endpoint of chunk i. For chunk i+1, the sink is retrieved near the endpoint of chunk i+1. This means the anchor continuously updates to track the camera's movement through the city, always providing guidance relevant to the current spatial neighborhood. The paper notes that "because the anchor is not a reconstruction target, it need not coincide with the exact future trajectory"—the retrieved image just needs to be near the endpoint location, not at the exact camera pose the trajectory will reach.

Training the model to use the lookahead sink. During training, a ground-truth future frame is sampled at a random temporal offset from the target chunk and used as the sink token. This exposes the model to varying lookahead distances (since the offset is randomized), so it learns how the anchor's proximity affects generation: a closer sink provides stronger constraint, a more distant sink provides weaker but still useful guidance. The training objective is the standard diffusion denoising loss on the target latents only; the sink token is provided as clean conditioning, and the model is not trained to predict it.

At inference, Δ_VL is fixed (to 5), and the ground-truth frame is replaced by a retrieved street-view image. Because the model was trained with random temporal offsets, it generalizes to this fixed-offset setting, and because it was trained with ground-truth frames (which are pixel-perfect), it learns to trust the sink as reliable conditioning and aligns its generation to match the sink's spatial structure.

Why not just use the first frame as the sink? The paper ablates four sink configurations (Table 2 and Figures 10, 14):

1. No sink: FID 33.06, mPSNR 14.16. Removing the sink entirely causes camera drift and structural degradation.
2. First-Frame (FF) sink: uses the first frame of the entire generation as a persistent attention sink. FID 32.71, mPSNR 14.25. Better than no sink, but as the camera moves far from the starting point, the first frame's visual content becomes irrelevant. Figure 10 shows sliding-window FID rises over time as the camera moves away.
3. First-Position (FP) sink: places a retrieved image at the first-frame position instead of the actual first frame. FID 32.41, mPSNR 14.35. Better than FF because the retrieved image is spatially relevant, but placing it at position 0 (very early in the sequence) means it functions as a "where we started" anchor rather than a "where we're going" destination.
4. Virtual Lookahead (VL) sink (the proposed method): dynamically retrieves near the endpoint and places at a future temporal position. FID 28.43, mPSNR 14.56. Best performance because the anchor is both spatially relevant and temporally positioned as a future destination, giving the strongest guidance for spatial convergence.

Figure 10 is particularly informative: it shows sliding-window FID (computed over a 200-frame window) as a function of generation time. The VL Sink achieves the lowest FID at all time points and shows the slowest rate of degradation over time, indicating that the dynamic, forward-looking anchor is more effective at preventing error accumulation than static anchors that remain fixed at the starting point.

Implementation details for the SF variant. Under Self-Forcing (SF) [18], the model generates with a causal attention mask where each token can attend only to tokens at earlier or equal sequential positions. The VL Sink and reference tokens are assigned RoPE positions beyond the current generation window, but all generated tokens must attend to them. This creates a tension: the sink needs to be visible to all tokens (so it must appear "earlier" in the attention mask) but positionally encoded as if it's in the future (so it's treated as a destination, not a predecessor).

SWM resolves this by separating RoPE temporal positions from token ordering. The VL Sink and reference tokens are prepended to the beginning of each chunk's token sequence (so they are visible to all generated tokens under the causal mask), but their RoPE positions correspond to their intended temporal locations: beyond the current generation window for the VL Sink and at a large temporal offset for references. Because RoPE encodes temporal information through positional embeddings rather than token ordering, the model interprets these tokens as being "in the future" (via their RoPE positions) while still being able to attend to them (because they appear before the generated tokens in the sequence). This separation of positional encoding from sequence order is a key implementation insight that enables the lookahead mechanism to work under causal attention constraints.

---

Base Model Architecture and Training Configuration

SWM is not trained from scratch. It fine-tunes Cosmos-Predict2.5-2B-I2W [1], a pretrained 2B-parameter Diffusion Transformer (DiT) designed for image-to-world video prediction. The choice of this base model leverages its existing capabilities for video generation conditioned on starting images, camera trajectories, and text prompts, reducing the amount of new learning required to add retrieval-augmented grounding.

Architecture details (Appendix A.1). Cosmos-Predict2.5-2B-I2W is a DiT [33] with 28 transformer blocks, 16 attention heads per block, and a hidden dimension of 2048. It operates in a 16-channel latent space produced by a 3D Variational Autoencoder (VAE) that applies 4× temporal compression and 8× spatial compression to pixel-space video. Given T frames of pixel-space video X = {x_t}_{t=0}^{T-1}, the 3D VAE encoder produces L = T/4 latents Z = {z_l}_{l=0}^{L-1}. The DiT then denoises these latents using a flow matching objective (the specific diffusion formulation inherited from Cosmos).

Input processing for SWM. The DiT backbone processes multiple conditioning signals that are integrated into the input latent representation and the token sequence:

1. Noisy target latents: The standard diffusion input—L latents corrupted with noise according to the current timestep, which the model learns to denoise.

2. Warped video conditioning: The geometric referencing output—encoded by the same 3D VAE and channel-wise concatenated with the noisy target latents at the input to the DiT's first layer. The patch embedding module for the main pathway is widened to accept 16 (target latent channels) + M (warped video channels) input channels instead of the standard 16.

3. History latents: H clean latents from the tail of the previous chunk's output (H=5 for TF, H=3 for SF). During TF training, these are ground-truth history latents; during SF training and TF inference, they are self-generated. These latents occupy the first H positions in the token sequence and are not denoised—they serve purely as conditioning for temporal continuity.

4. Reference tokens: Up to K encoded reference latents (each representing one street-view image), patch-embedded and concatenated at the end of the token sequence with large RoPE offsets. A dedicated reference embedder (initialized by copying the main embedder weights) processes these 16-channel latents with an additional 1-channel padding mask indicating which spatial regions are valid (since warped or partial references may have invalid regions).

5. Virtual Lookahead Sink token: One encoded latent from the retrieved future street-view image, embedded similarly to reference tokens using the same dedicated embedder, placed immediately after the target latents in the token sequence (for TF) or prepended to the beginning (for SF, to satisfy causal masking while maintaining future RoPE position).

6. Camera pose embeddings: Per-pixel 6-channel Plücker ray maps encoding the camera extrinsics and intrinsics for every frame (target, history, references, and sink). These ray maps are processed through a shallow convolutional encoder and added as residuals to both the main video tokens and the reference/sink tokens, ensuring every token carries geometric viewpoint information.

7. Text conditioning: T5 text embeddings from the Qwen2.5-VL-72B-generated captions (or the short camera-action sentence), injected into the DiT via cross-attention layers. Text captions are replaced with empty-string embeddings with a probability of 20% during training for classifier-free guidance.

Conditioning dropout schedule (Appendix A.1). To enable classifier-free guidance (CFG) at inference and ensure the model can handle missing conditions gracefully, SWM applies a structured dropout schedule during training:

• Text captions: replaced with empty-string T5 embeddings, probability 20%
• Reference conditions: zeroed out entirely, probability 20% for street-view and synthetic data; Waymo samples never include references (effectively 100% dropout for those samples)
• Warped video inputs: replaced with zeros, probability 20%

This means the model is trained with all combinations of conditioning presence/absence, so it learns to generate with or without text, references, and geometric warping. At inference, CFG can be applied by running the model with and without conditioning and extrapolating in the direction of the conditioned output.

Teacher Forcing (TF) configuration. The TF variant is trained using standard Teacher Forcing [51]: each chunk conditions on H=5 ground-truth history latents from the training data. During TF training, small Gaussian noise (μ=0, σ=0.1) is injected into the conditioning history frames with 50% probability. This noise injection reduces the gap between clean training inputs and the self-generated (and thus imperfect) inference history: by training the model to handle slightly noisy history latents, it becomes more robust to errors in self-generated history at inference time.

Under TF, each chunk consists of T=77 frames (encoded into L=19 latents by the 3D VAE), with K=5 references and G=50 temporal gap for semantic references. At inference, the history latents are replaced by self-generated output from the previous chunk, but the model processes each chunk independently (no KV cache is maintained across chunks). The generation is not truly autoregressive in the streaming sense—it is a chunk-wise rollout where each chunk is generated from scratch with the previous chunk's tail as fixed conditioning.

Training hyperparameters (Appendix A.1): AdamW optimizer [31] with learning rate 4.8×10⁻⁵, total batch size 48 across 24 NVIDIA H100 GPUs, trained for 10K iterations. All camera extrinsics are expressed in a unified Right-Down-Forward (RDF) coordinate system and defined relative to the first frame of each chunk.

Self-Forcing (SF) configuration. The SF variant follows the Self-Forcing paradigm [18], which bridges the train-test gap in autoregressive video diffusion by training the model to generate from its own previous outputs. The SF variant is initialized from the TF checkpoint using ODE initialization with 1K sequence pairs for 6K steps, followed by 10K iterations of fine-tuning.

Under SF, the model maintains a KV cache across chunks and generates with causal attention: each token can attend only to tokens at earlier or equal sequential positions. Chunks are shorter: T=12 frames (yielding L=3 latents), with H=3 history latents and K=1 reference per chunk. The smaller chunk size reduces the computational cost of maintaining the KV cache and makes generation faster: the SF variant achieves 15.2 fps on a single H100 GPU.

The VL Sink and reference tokens are prepended to the beginning of each chunk's token sequence (so they are visible to all generated tokens under the causal mask) while retaining their intended future/offset RoPE positions. This separation of token ordering from positional encoding, described in the VL Sink section above, is a key implementation detail that makes retrieval-augmented conditioning compatible with causal autoregressive generation.

Starting from arbitrary coordinates (Appendix A.1). By default, the user selects a starting coordinate that corresponds to an existing street-view location, and the corresponding pinhole image is used directly as the first frame. When the user specifies an arbitrary coordinate without an exact street-view match, SWM uses the nearest available street-view image as the first frame, generates a "buffer chunk" that navigates from that nearest location toward the user's target starting point, and then continues generation from the target coordinate onward, discarding the buffer chunk from the final output. This ensures the model can initialize generation from any coordinate within the coverage area, not just pre-existing street-view capture points.

---

Summary of Design Choices and Their Justifications

• Cross-temporal pairing over co-temporal pairing: Prevents the model from learning to copy dynamic objects from references, forcing it to rely on persistent scene structure. This is the single most important design choice—the ablation shows removing it causes the largest degradation across all metrics (FID +16.31, mPSNR −2.02, TransErr +0.108).
• Intermittent freeze-frame over channel concatenation for view interpolation: Matches the 3D VAE's 4-frame temporal compression stride, yielding exact keyframe conditioning and +2.51 PSNR improvement. The channel concatenation approach fails because the VAE expects 4-frame groups and degrades single-frame inputs.
• Two-stage retrieval (GPS + depth reprojection) over GPS-only: Filters out geographically nearby but visually irrelevant references (wrong facing direction, occluded views), ensuring conditioning images actually see the target scene.
• Dual geometric + semantic referencing over either alone: Geometric warping provides explicit pixel-aligned layout cues but loses appearance detail; semantic injection preserves appearance but without spatial alignment. Using both compensates for complementary failure modes.
• Virtual Lookahead Sink over static attention sink: Dynamically updates the anchor to track the camera through the city, maintaining spatial relevance. First-frame sinks become irrelevant after the camera moves; first-position sinks are backward-looking rather than forward-guiding.
• Synthetic data for trajectory diversity over real-only training: Real street-view data is predominantly vehicle-mounted forward-facing driving; synthetic data introduces pedestrian paths, free-camera exploration, and diverse road types, improving camera-following accuracy (RotErr improves from 0.021 to 0.020, TransErr from 0.020 to 0.015, mPSNR from 13.52 to 14.56 when synthetic data is included).
• Fine-tuning Cosmos rather than training from scratch: Leverages existing video generation, camera conditioning, and text-following capabilities learned from large-scale pretraining, allowing SWM to focus on learning the retrieval-augmented grounding behavior with a relatively modest dataset (440K real + 12.7K synthetic + 20% Waymo).
• RoPE-position-over-token-order trick for SF: Separates temporal semantics (where a token "is" in time, encoded via RoPE) from token ordering (where it appears in the sequence, which affects the attention mask), enabling future-positioned sink and reference tokens to be visible under causal attention constraints.

4. Key Insights and Innovations

Innovation 1: Real-World Grounded World Simulation as a Distinct Task Category

The paper's most fundamental contribution is not any specific method but the formulation of a new task category: real-world grounded video world simulation. Prior to SWM, the world modeling literature had bifurcated into two non-overlapping approaches: video world models that generate imagined, dynamic environments with no fidelity to any physical location [1, 7, 8, 13, 20, 32, 46, 63, 64], and 3D reconstruction systems that model real cities with high spatial accuracy but are fundamentally static and non-generative [29, 44]. These two research communities operated in parallel, pursuing different objectives with different evaluation criteria.

SWM's core intellectual move is to argue that this bifurcation is unnecessary and artificially limiting. The paper demonstrates that a generative model can be simultaneously dynamic (producing novel content, responding to text prompts, simulating hypothetical scenarios) and spatially faithful to a specific real city (rendering actual building facades, maintaining accurate road geometry, preserving landmark appearances). This is not a small extension of either prior approach—it requires resolving a tension that neither community had to confront: how to condition generation on real-world observations that are temporally inconsistent with the scene being generated.

What makes this formulation distinctive is that it identifies a genuine capability gap that was hiding in plain sight. Street-view databases containing billions of geotagged urban images exist worldwide, and video world models can generate impressive imagined environments—yet no prior work asked whether these could be combined. The paper's title question—"What if a world simulation model could render not an imagined environment but a city that actually exists?"—is not rhetorical; it identifies a design objective that prior work was fundamentally not designed to address. The fact that all six baseline world models degrade substantially when applied to this task (Table 1, with FID ranging from 49.63 to 141.24 on Busan-City-Bench compared to SWM's 28.43) is evidence that this is genuinely a different task, not merely a different evaluation of the same capability.

The significance of this formulation extends beyond the specific methods proposed. By defining the task, the paper establishes a new axis for evaluating world models—spatial fidelity to real locations—that complements existing criteria like visual quality and temporal coherence. It also opens a new set of research questions: what is the relationship between retrieval density and grounding quality? How does grounding degrade when references are sparse or outdated? Can these methods transfer across cities without retraining? The paper provides initial answers to some of these (Appendix C.3 on reference sparsity, the cross-city generalization results in Table 1), but the framing is more valuable than the specific answers.

Innovation 2: Cross-Temporal Pairing as a Principle for Disentangling Persistent Structure from Transient Content

The paper's most significant technical insight is that the central challenge in retrieval-augmented generation with real-world references is not the generative architecture or the retrieval mechanism—it is a data design problem: how to teach the model to extract what is useful from references (persistent spatial structure) while ignoring what is misleading (transient dynamic content that differs between reference capture time and generation time).

Prior work in geometry-aware video generation [34, 60] and spatial memory approaches [7, 24, 25, 50, 53] implicitly assumed that multi-view reference images depict temporally consistent scenes. Gen3C [34] expects all input views to capture the same static scene at the same instant. VMem [25] retrieves previously generated frames as references, which are temporally self-consistent by construction. These assumptions hold for controlled capture settings (multi-view studio setups, slowly-explored static environments) but break fundamentally when using real-world street-view databases, where images of the same location captured months apart differ in lighting, weather, parked vehicles, pedestrians, and temporary signage.

The paper's diagnosis is that this temporal inconsistency is not a nuisance to be filtered out—it is the defining property that a grounded world model must learn to handle. Cross-temporal pairing operationalizes this diagnosis by turning the inconsistency into a training signal rather than a training obstacle. By deliberately pairing references and targets from different timestamps during training, the model is forced to learn what remains invariant across temporal variation: the geometric layout of buildings, the structure of roads, the persistent appearance of static urban elements. Without this explicit pairing strategy, the model learns a spurious correlation (references contain specific cars → generated output should contain those cars) that generalizes catastrophically at inference when references come from arbitrary timestamps.

The evidence for the importance of this insight is the ablation result in Table 2: removing cross-temporal pairing causes the single largest degradation across all metrics compared to any other component removal (FID increases by +16.31, more than double the degradation from removing either referencing pathway or the attention sink). This is not an incremental improvement—it is the difference between a model that fundamentally works (generates coherent output grounded in real geometry) and one that fundamentally doesn't (attempts to copy transient objects from references, producing visual inconsistencies and structural drift). Appendix D.2 explains the mechanism: without cross-temporal pairing, the model learns to interpolate dynamic objects from co-temporal references, which "leads to abrupt appearance or disappearance of objects and visual discontinuities" at inference when reference timestamps don't match.

This insight has implications beyond SWM. Any system that conditions generation on temporally misaligned observations—satellite imagery for map generation, historical photos for time-travel simulation, security camera feeds for scene reconstruction—faces the same fundamental challenge. The principle of using temporal variation in training data to teach models to separate invariant structure from transient content is a general design pattern that the paper demonstrates at city scale.

Innovation 3: The Virtual Lookahead Sink as a Forward-Anchoring Attention Mechanism

Prior work in long-horizon autoregressive generation addressed error accumulation through backward-looking mechanisms: attention sinks [54] that maintain a fixed early frame throughout generation [28, 40], or memory banks that store and retrieve previously generated content [7, 25, 53]. These approaches are fundamentally retrospective—they stabilize generation by anchoring it to what has already been seen or generated. The insight underlying the Virtual Lookahead Sink is that for retrieval-augmented generation where ground-truth future observations are available (via the street-view database), it is more effective to anchor generation to what lies ahead rather than to what lies behind.

This is a conceptually distinctive move because it inverts the standard temporal structure of autoregressive generation. Autoregressive models are trained to predict the future from the past; the future is unknown and must be generated. The Virtual Lookahead Sink exploits the fact that in the grounded setting, a noisy but unbiased estimate of the future is available through retrieval—the street-view image near the chunk's endpoint shows the real appearance of the location the camera is approaching, even though it's captured at a different time with different dynamic content. By encoding this future observation as a clean latent at a future temporal position (Δ_VL=5 beyond the current chunk), the model receives a target to converge toward—a stabilizing gradient in the space of possible generations.

The comparison with alternative sink designs (Table 2, Figures 10, 14) demonstrates that the forward-looking property is what matters, not merely having an anchor. The First-Frame sink (backward-looking, static) and First-Position sink (backward-looking, updated) both underperform the Virtual Lookahead Sink, even though all three provide error-free conditioning. The key difference is temporal semantics: a sink at position 0 tells the model "stay similar to where you started," while a sink at position H+L+5 tells the model "converge toward this future location." For a moving camera traversing hundreds of meters, the latter is a more useful constraint.

The implementation insight that enables this under Self-Forcing—separating RoPE temporal positions from token ordering so that future-positioned tokens are visible under causal attention—is a practical contribution that generalizes beyond this specific architecture. Any autoregressive model that needs to condition on "future" information while maintaining causal generation constraints can use this trick, suggesting applications in navigation, planning, and any sequential generation task where partial future information is available.

Innovation 4: The Complementarity of Geometric and Appearance Conditioning Pathways

The paper contributes a diagnostic framework for understanding what different conditioning modalities provide, and validates this framework through ablation. Prior work in geometry-aware generation typically chose one approach: either explicit geometric warping (Gen3C [34], ViewCrafter [60]) or implicit appearance injection through attention (earlier video diffusion models that condition on reference images without warping). The implicit assumption was that these approaches are alternatives—that if you have good enough geometry, you don't need separate appearance conditioning, or vice versa.

SWM's ablation (Table 2) demonstrates that this assumption is incorrect: geometric and semantic referencing have complementary strengths and failure modes. Removing geometric referencing degrades camera-following accuracy (RotErr increases from 0.020 to 0.036) and structural fidelity (mPSNR drops from 14.56 to 12.33), but has relatively mild impact on distributional metrics (FID increases from 28.43 to 33.01). Removing semantic referencing has a milder impact on geometric metrics (mPSNR drops to 14.08) but degrades appearance quality (mLPIPS increases from 0.392 to 0.442). This pattern reveals a clean functional decomposition: geometric warping provides explicit spatial layout cues that guide the overall scene structure, while semantic injection provides fine appearance detail (building textures, signage, material properties) that geometric warping loses due to depth errors, occlusion, and resolution limitations.

The significance of this finding is that it provides a principled basis for combining conditioning modalities rather than selecting among them. Future systems can use this diagnostic pattern—ablate each conditioning pathway and measure which metrics degrade—to understand what information each modality contributes and whether they are redundant or complementary. The fact that the two pathways have different RoPE positions and different conditioning mechanisms (channel concatenation for warped video vs. token concatenation for references) further suggests that the architecture naturally separates "spatial layout" and "appearance" processing, which could inform future architectural designs for multi-modal world models.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The primary evaluation data is street-view imagery and driving video from three cities: Seoul (the training city), Busan (from NAVER Map street-view imagery), and Ann Arbor (from the MARS dataset [26]). The paper constructs two test-only benchmark datasets: Busan-City-Bench and Ann-Arbor-City-Bench, each containing 30 test sequences. Each sequence consists of 365 frames, corresponding to approximately 100 meters of travel. The Busan benchmark is constructed from sparsely captured street-view images and requires the view interpolation pipeline (Section 3.1) to synthesize temporally continuous ground-truth video for computing FVD. The Ann Arbor benchmark uses ground-truth video sequences directly from the MARS dataset. Critically, for both benchmarks, retrieved references exclude any street-view image belonging to the test sequence itself, ensuring the model cannot access the ground-truth viewpoint during generation.

• Base model. SWM fine-tunes Cosmos-Predict2.5-2B-I2W [1], a 2B-parameter Diffusion Transformer (DiT) pretrained for image-to-world video prediction. The paper argues this model is representative of contemporary video world model capabilities and sits in a useful pretraining regime: it already generates high-quality conditional video, so SWM can focus on learning the retrieval-augmented grounding behavior rather than basic video generation, which would require far more data and compute. Fine-tuning proceeds for 10K iterations on 24 H100 GPUs, mixing three data sources: Waymo (20%), Seoul street-view (40%), and synthetic CARLA data (40%). Two inference configurations are studied: Teacher Forcing (TF), which generates 77-frame chunks conditioned on H=5 history latents, and Self-Forcing (SF) [18], which generates 12-frame chunks with causal attention and a KV cache, achieving 15.2 fps on a single H100 GPU.

• Metrics. The paper evaluates generation quality across three dimensions: (1) visual and temporal fidelity: FID [14] computed between generated and ground-truth frames, FVD [47] computed between generated and ground-truth videos, and Image Quality from VBench [19]; (2) camera-following accuracy: Rotation Error (RotErr) measuring mean geodesic distance on SO(3) between predicted and ground-truth relative rotations, and Translation Error (TransErr) measuring mean ℓ₂ distance between scale-normalized relative translations; (3) 3D adherence to real locations: masked PSNR and LPIPS [61] computed only over static regions of the scene. For the static-region mask, SAM3 [5] is prompted with dynamic-object categories (pedestrians, vehicles) to segment moving objects in both generated and ground-truth frames; PSNR and LPIPS are then computed exclusively on the remaining static pixels. This masking is essential because generated dynamic content (traffic patterns, pedestrian behavior) is not expected to match the ground-truth—the metric isolates whether the model faithfully reproduces the persistent scene structure from the references.

• Baselines. The paper compares SWM against six recent video world models, each adapted to the benchmark by providing the subset of inputs that model natively supports: Aether [63] (camera conditioning via Plücker ray maps channel-concatenated with image latents), DeepVerse [7] (camera motion discretized into 27 action classes encoded as text embeddings), Yume1.5 [32] (camera control expressed through WASD-style keyboard commands in text), HY-World1.5 [20] (dual-path camera conditioning via PRoPE and discretized motion states), FantasyWorld [8] (camera input as per-pixel Plücker ray embeddings injected via Adaptive Layer Normalization), and Lingbot [46] (camera trajectories via Plücker ray maps with scale-and-shift modulation). All baselines receive the first frame of the target sequence as the starting image and generate autoregressively to cover the full 365-frame benchmark length. Additionally, Appendix C.5 (Table 4) compares against two static-scene video generation methods: GEN3C [34] (accepts multi-view reference images and generates video from camera trajectories, but assumes a static scene captured at a single instant) and VMem [25] (populates its geometric-aware memory with SWM's retrieved street-view images rather than self-generated frames).

• Generation budget / compute accounting. The paper does not formalize a FLOPs budget or compare at matched compute levels as in the reference example. Instead, comparisons are made at equal benchmark sequence lengths (365 frames) and equal chunk configurations within each model's autoregressive paradigm. The SF variant's throughput (15.2 fps on a single H100 GPU) is reported as a practical efficiency metric, but the quantitative results in Table 1 do not control for or report compute-equivalent comparisons. All baselines are run with their recommended inference settings, meaning differences in generation cost are not factored into metric comparisons.

• Cross-validation / statistical protocol. The paper does not report cross-validation or statistical significance testing for its main results. The Busan-City-Bench and Ann-Arbor-City-Bench are each 30 sequences long, treated as fixed test sets. The ablated models in Table 2 are evaluated on Busan-City-Bench only, without reporting variance across sequences. The difficulty of the task (new benchmark construction for a new task category) and the computational cost of generating 365-frame sequences from multiple models likely precluded more extensive statistical validation, but the absence of confidence intervals or standard deviations means the reported metric differences should be interpreted with appropriate caution, particularly for metrics computed over only 30 test sequences.

---

Main Quantitative Results

Comparison with Video World Models on Cross-City Benchmarks

Headline results (Table 1). SWM achieves the best performance across all metrics on both Busan-City-Bench and Ann-Arbor-City-Bench, with the TF variant generally outperforming the SF variant on accuracy metrics but trading off inference speed. On Busan-City-Bench, SWM (TF) achieves FID 28.43, FVD 301.76, RotErr 0.020, TransErr 0.015, mPSNR 14.56, and mLPIPS 0.392. The best baseline, HY-World1.5, achieves FID 49.63 (1.75× worse), FVD 544.04 (1.80× worse), RotErr 0.044 (2.20× worse), TransErr 0.079 (5.27× worse), mPSNR 11.87 (1.23× lower), and mLPIPS 0.588 (1.50× worse).

The performance gap is not uniform across metrics. Camera-following accuracy shows the largest relative gains: SWM's TransErr of 0.015 is 5.27× better than HY-World1.5's 0.079 and 6.87× better than Aether's 0.083. This is the metric most directly tied to the retrieval-augmented grounding mechanism—without real-world references, baseline models cannot maintain alignment with the target trajectory, and their generated camera motion drifts from the intended path. RotErr shows a similar pattern (SWM 0.020 vs. HY-World1.5 0.044, a 2.20× improvement), confirming that rotation fidelity also degrades without external grounding.

Structural fidelity metrics (mPSNR, mLPIPS) show the most modest improvements: SWM's mPSNR of 14.56 is 1.23× higher than HY-World1.5's 11.87, and mLPIPS of 0.392 is 1.50× lower than HY-World1.5's 0.588. This pattern makes sense given the metric design: PSNR and LPIPS on static regions measure how faithfully the model reproduces building facades, road geometry, and other persistent structures. Even the best baseline (HY-World1.5) has no mechanism to access ground-truth appearance of these structures, so it generates plausible but incorrect buildings; SWM's references provide explicit appearance and geometry cues that improve structural fidelity, but absolute reconstruction quality is limited by the quality of the retrieved references and the geometric warping process (depth errors, occlusion, resolution differences).

Distributional metrics (FID, FVD) show large but not extreme gaps. SWM's FID of 28.43 is better than all baselines, but the gap to HY-World1.5 (49.63) is smaller—proportionally—than the camera-following gap. This aligns with the intuition that FID measures how "realistic" generated frames look in distribution, not whether they match the specific real location. Baselines can generate plausible urban scenes (low-ish FID) that are geometrically wrong (high RotErr/TransErr, low mPSNR). SWM's advantage on FID likely comes from the higher-quality conditioning (references provide real texture and lighting cues) rather than from fundamentally better generative architecture.

Cross-city generalization. Both Busan and Ann Arbor are entirely absent from SWM's training data (trained only on Seoul). The fact that SWM outperforms baselines on these unseen cities demonstrates that the model has learned a generalizable skill of grounding generation in retrieved street-view references, not merely memorized the visual appearance of specific Seoul locations. The performance on Ann-Arbor-City-Bench is generally worse than on Busan-City-Bench across all models (e.g., SWM TF FID 56.61 vs. 28.43, mPSNR 15.18 vs. 14.56), which the paper does not analyze in detail. Possible explanations include: (1) domain gap between North American and Korean urban architecture, which affects all models but may disproportionately impact SWM if its pretrained visual features are biased toward Asian cities; (2) differences in street-view capture characteristics (camera height, field of view, lighting conditions); (3) the MARS dataset's ground-truth video having different dynamics than the Korean driving videos in SWM's training distribution. However, SWM's relative advantage over baselines is preserved: on Ann Arbor, SWM TF achieves FID 56.61 vs. HY-World1.5's 67.02, and mPSNR 15.18 vs. HY-World1.5's 14.26.

TF vs. SF comparison. The SF variant achieves slightly inferior accuracy metrics on Busan (FID 32.50 vs. 28.43 for TF, mPSNR 13.52 vs. 14.56) but superior speed (15.2 fps). On Ann Arbor, the pattern partially reverses: SF achieves better FID (43.97 vs. 56.61) but worse camera-following accuracy (RotErr 0.217 vs. 0.055). The paper does not analyze these per-city differences in the TF-SF gap, but they suggest that the SF variant may be more sensitive to domain shift (Ann Arbor's architectural style being more different from Seoul's training distribution than Busan's).

Qualitative comparison (Figure 8). The figure shows representative frames from SWM (TF and SF) against all six baselines on both benchmarks, alongside ground-truth frames. The baselines exhibit characteristic failure modes: blurred or washed-out textures (Aether, DeepVerse on Busan), structural collapse where generated content deviates completely from the ground-truth scene layout (DeepVerse on Ann Arbor), and loss of trajectory alignment where buildings and road structures bear no resemblance to the real location. SWM's outputs, while not pixel-perfect reconstructions, maintain recognizable structural correspondence with the ground-truth urban geometry—buildings appear in the correct locations, roads follow the correct curvature, and distant landmarks are recognizable. The text notes that baselines "often drift over long trajectories, leading to misalignment in both camera motion and scene structure and resulting in blurred videos, reduced motion, or complete collapse."

Qualitative Demonstration of Core Capabilities (Figure 7)

While not comparative, Figure 7 demonstrates three capabilities that the quantitative benchmarks do not separately evaluate:

Diverse scenarios (Figure 7a): SWM generates a 2.48km trajectory (substantially longer than the 100m benchmark sequences) with text prompts altering global scene conditions ("A massive tsunami rises in the distant skyline," "A warm late golden-hour sunset"). The generated frames show the tsunami scenario with large water masses consistent with the urban skyline geometry, and the golden-hour scenario with warm lighting that respects building occlusion patterns. This demonstrates that retrieval-augmented grounding does not eliminate text controllability—the model can simultaneously maintain spatial fidelity to real locations and respond to prompts that alter scene conditions. The paper attributes this to capabilities "largely inherited from the pretrained world simulation model" [1] and supported by the stylized video augmentation pipeline described in Section 3.1.

Diverse trajectories (Figure 7b): The figure shows frames from trajectories beyond standard vehicle-mounted driving paths—pedestrian-level viewpoints, turns at intersections, varied camera orientations. This validates the contribution of the synthetic CARLA dataset, which introduced pedestrian, vehicle, and free-camera trajectories absent from the real street-view data. Without synthetic data, SWM would have been trained predominantly on forward-facing driving trajectories and would likely struggle with non-driving camera motions.

Long-horizon generation (Figure 7c): The figure shows frames from a multi-kilometer trajectory without visible error accumulation or structural drift. The frames remain recognizable as the same urban environment, with consistent building geometry, road continuity, and stable lighting. This validates the Virtual Lookahead Sink's effectiveness at preventing the compounding drift that affects autoregressive generation without dynamic re-anchoring. The quantitative evidence for this claim is in Figure 10 and Table 5, where removing the VL Sink causes progressively worsening FID over extended trajectories.

Comparison with Static-Scene Video Generation Methods (Appendix C.5, Table 4)

This comparison isolates the contribution of dynamic generation capability relative to methods that can condition on multi-view references but cannot model scene dynamics. GEN3C [34] and VMem [25] both receive the same retrieved street-view images as SWM, but they operate under the assumption of a static scene—GEN3C expects all multi-view images to depict the same instant, and VMem stores and retrieves previously generated frames from its memory.

On Busan-City-Bench, the results are revealing: both static models struggle with dynamic quality metrics (GEN3C FID 45.72, FVD 416.94 vs. SWM FID 28.43, FVD 301.76), but GEN3C achieves competitive structural fidelity on static regions (mPSNR 14.16 vs. SWM 14.56, mLPIPS 0.524 vs. SWM 0.392—SWM wins but the gap is much smaller than on dynamic metrics). The paper reports that dynamic objects in the references "appear frozen in the generated videos" from these methods, since they treat temporally misaligned street-view references as if they depict a single coherent static scene. The frozen dynamic objects (vehicles stuck at intersections, pedestrians frozen mid-stride) degrade the visual quality metrics (FID, FVD) but have less impact on static-region PSNR/LPIPS, which explicitly mask out dynamic regions.

This comparison validates a central claim of the paper: the cross-temporal pairing strategy is necessary not just for faithfully reproducing persistent structure, but for enabling the model to generate plausible dynamics in the presence of temporally inconsistent references. Static-scene methods can partially reproduce the geometric layout (competitive mPSNR) but fail to produce natural motion because they have no mechanism for distinguishing what in the references should be treated as scene structure versus what should be treated as transient content.

Long-Horizon Stability (Appendix C.6, Table 5)

To evaluate the attention sink variants at scale, the paper constructs an extended version of Busan-City-Bench where each sequence is 1,460 frames (4× the standard 365-frame length, covering approximately 500 meters per sequence). This tests whether the sink design matters more or less as trajectories lengthen.

The results (Table 5) show that the performance gap between sink variants becomes more pronounced at longer horizons. Removing any attention sink increases FID from 25.13 (VL Sink) to 37.37 (no sink), a 48.7% degradation. The First-Frame sink achieves FID 30.85 (22.8% worse than VL Sink), and the First-Position sink achieves FID 28.57 (13.7% worse). The structural fidelity metrics show a similar pattern: VL Sink achieves mPSNR 13.70, while no sink drops to 12.94, FF sink drops to 13.08, and FP sink achieves 13.34. Camera-following accuracy shows a more nuanced pattern: the First-Position sink achieves the best RotErr (0.019 vs. 0.027 for VL Sink) and TransErr (0.021 vs. 0.029 for VL Sink). This suggests that placing the retrieved image as a backward-looking anchor (FP sink) may provide marginally better camera trajectory adherence than the forward-looking VL design, even though the VL design achieves better overall visual quality (FID, mPSNR, mLPIPS).

The paper does not analyze this camera-following vs. visual quality tradeoff, but a plausible interpretation is that the VL Sink's future-positioned anchor provides a stronger signal for "what the scene should look like" (improving FID and mPSNR) but may exert a slight "pull" that biases the generated camera motion toward the sink's viewpoint, slightly reducing trajectory fidelity (higher RotErr/TransErr). The FP sink, positioned at the start of the sequence, provides a stable reference point that helps maintain consistent camera motion but provides less spatial guidance about upcoming locations.

Figure 14 examines the same comparison under the SF configuration, showing qualitatively similar patterns: VL Sink achieves the lowest and most stable sliding-window FID over the extended trajectory, with no-sink degrading most rapidly and FF/FP sinks occupying intermediate positions.

Effect of Reference Density (Appendix C.3, Figure 13)

A practical concern for deployment: how does SWM perform when fewer street-view references are available per chunk? This tests robustness to sparse retrieval coverage, which varies across real-world locations.

Figure 13 sweeps the number of retrieved references K from 1 to 5 on Busan-City-Bench. The results are non-monotonic across metrics: as K decreases from 5 to 1, mPSNR drops monotonically (from approximately 14.59 at K=5 to approximately 14.35 at K=1), confirming that fewer references provide less coverage of the target scene and degrade structural fidelity. However, FID and FVD show no clear degradation—K=1 achieves the best FID (approximately 27, vs. 28–31 for higher K) and the second-best FVD (approximately 295, vs. 290–350). The paper interprets this as evidence that "the underlying video diffusion model retains its generative capability even with fewer reference constraints, producing visually plausible frames that score well on distributional metrics despite being less grounded to the specific location."

This interpretation suggests an important limitation of FID/FVD as evaluation metrics for this task: they capture whether frames "look like" the target distribution, not whether they faithfully reproduce the specific location. A model with fewer references might generate a generic but realistic-looking building where the real location has a distinctive landmark—this would score well on FID (the building looks plausible) but poorly on mPSNR (it doesn't match the ground-truth). The monotonic mPSNR trend with K confirms that reference conditioning primarily improves geometric and appearance grounding rather than overall visual realism, and that 5 references provide measurably better spatial fidelity than 1 reference.

---

Ablation Studies and Robustness Checks

All ablations in this section are evaluated on Busan-City-Bench and reported in Table 2 unless otherwise noted.

Cross-temporal pairing: Removing cross-temporal pairing causes the largest degradation across all metrics: FID increases from 28.43 to 44.74 (+57.4%), FVD increases from 301.76 to 487.87 (+61.7%), RotErr increases from 0.020 to 0.057 (+185%), TransErr increases from 0.015 to 0.123 (+720%), mPSNR drops from 14.56 to 12.54 (−13.9%), and mLPIPS increases from 0.392 to 0.519 (+32.4%). The catastrophic degradation in translation error (0.015 → 0.123, a more than 8× increase) confirms that without cross-temporal pairing, the model fails to maintain camera trajectory alignment with the target, likely because it attempts to copy dynamic objects from co-temporal references into the generated frames, producing visual inconsistencies that compound into trajectory drift. This ablation provides the strongest evidence for the paper's central claim that cross-temporal pairing is not a data augmentation detail but a foundational requirement.

Synthetic data: Removing the synthetic CARLA dataset produces a mixed effect: FID actually improves slightly (27.74 vs. 28.43), but camera-following accuracy degrades (RotErr 0.021 vs. 0.020, TransErr 0.020 vs. 0.015) and structural fidelity degrades (mPSNR 13.52 vs. 14.56, mLPIPS 0.427 vs. 0.392). The FID improvement is expected: synthetic data has a domain gap from real photographs, so including it slightly worsens distributional metrics by pulling the model's output distribution toward the synthetic domain. However, the degraded structural and camera metrics confirm that the diverse trajectories from synthetic data are essential for handling camera motions beyond standard driving paths. Without synthetic data, the model has been trained predominantly on forward-facing vehicle trajectories and struggles to generalize to arbitrary camera paths.

Real street-view data: Removing the real Seoul street-view data (trained only on Waymo and synthetic) yields FID 29.82, FVD 467.58, RotErr 0.059, TransErr 0.050, mPSNR 13.99, mLPIPS 0.411. Compared to the full model, this is worse across every metric, with particularly large degradation in translation error (0.050 vs. 0.015, 3.33× worse). This confirms that real street-view data provides essential grounding quality that synthetic data cannot fully substitute, despite the synthetic data's trajectory diversity benefits. The paper writes that real street-view data is essential because the model "consistently encounters a temporal gap between references and targets across all data sources," and synthetic data alone may not provide sufficient diversity of real urban appearance, lighting, and structural patterns to enable the model to generalize to real cities.

Geometric referencing (warped video conditioning): Removing geometric referencing (keeping only semantic referencing) causes FID to increase from 28.43 to 33.01, RotErr to increase from 0.020 to 0.036 (+80%), TransErr to increase from 0.015 to 0.051 (+240%), mPSNR to drop from 14.56 to 12.33 (−15.3%), and mLPIPS to increase from 0.392 to 0.525 (+33.9%). The disproportionate impact on camera-following metrics (especially TransErr) and structural fidelity (mPSNR) reveals that geometric warping provides explicit spatial layout cues that the model relies on to maintain geometric alignment with the target scene. Without these pixel-aligned layout cues (the warped video shows where buildings, roads, and structures should be in the target viewpoint), the model's internal spatial representation drifts, and the generated output no longer faithfully follows the real geometry, even though the semantic references still provide appearance information. Figure 9 qualitatively shows that without geometric referencing, the generated video loses spatial structure—buildings may appear in wrong positions or at wrong scales.

Semantic referencing (reference token injection): Removing semantic referencing (keeping only geometric referencing) causes FID to increase from 28.43 to 30.27, mPSNR to drop from 14.56 to 14.08, and mLPIPS to increase from 0.392 to 0.442 (+12.8%). The impact on camera-following metrics is smaller (RotErr 0.032 vs. 0.020, TransErr 0.022 vs. 0.015) compared to removing geometric referencing. The pattern—geometric ablation hurts camera metrics more, semantic ablation hurts appearance metrics more—confirms the functional decomposition the paper claims: geometric warping guides spatial layout, semantic injection provides appearance detail. Figure 9 qualitatively shows that without semantic referencing, the generated video loses appearance fidelity—building textures, signage clarity, and material details are degraded compared to the full model.

Attention sink design: The paper ablates four configurations: (1) full model with VL Sink, (2) no attention sink, (3) First-Frame (FF) attention sink using the first frame as a persistent anchor, and (4) First-Position (FP) sink placing a retrieved image at position 0. Results are reported in Table 2 and Figures 10 and 14:

• No sink: FID 33.06, mPSNR 14.16, RotErr 0.021, TransErr 0.016. Removing the sink entirely causes the worst visual quality (highest FID among sink variants) despite maintaining reasonable camera-following accuracy. This suggests that the sink primarily stabilizes visual appearance over long trajectories, rather than camera motion per se.
• FF sink: FID 32.71, mPSNR 14.25. Marginally better than no sink on structural metrics, but Figure 10 shows sliding-window FID rises steadily as the camera moves away from the starting position, confirming that a static first-frame anchor becomes progressively less useful.
• FP sink: FID 32.41, mPSNR 14.35. Better than FF sink because the retrieved image at position 0 is spatially relevant to the current location, but backward-looking (positioned as a "start" anchor rather than a "destination").
• VL Sink: FID 28.43, mPSNR 14.56. Best across all metrics, with the lowest and most stable sliding-window FID (Figure 10), confirming that a forward-looking, dynamically updated anchor provides the strongest stabilization for long-horizon generation.

The long-horizon versions of these ablations (Table 5, 1,460 frames) confirm that sink design matters more as trajectories lengthen: the gap between VL Sink (FID 25.13) and no sink (FID 37.37) widens from the standard-length benchmark, and the VL Sink achieves the best structural fidelity despite the FP sink achieving slightly better camera-following accuracy on this extended benchmark.

View interpolation strategy (Appendix C.1, Table 3): Comparing the proposed intermittent freeze-frame strategy against a channel concatenation baseline on the Waymo test set, the freeze-frame approach achieves PSNR 25.03 (vs. 22.52), SSIM 0.703 (vs. 0.628), and LPIPS 0.162 (vs. 0.245). This ablation validates the design choice motivated by the 3D VAE's temporal compression stride—the channel concatenation approach's weak keyframe adherence is quantitatively confirmed.

ReST^EM revision model (reference paper only, not applicable to SWM): This ablation is from the reference example's paper, not SWM. SWM does not use revision models or ReST^EM-style training. No equivalent ablation exists in SWM.

---

Critical Assessment

The experiments demonstrate that SWM outperforms existing world models on a new task that those models were not designed for, and that the proposed components (cross-temporal pairing, geometric/semantic referencing, VL Sink, synthetic data) each contribute measurably to performance. However, several aspects of the experimental design limit the strength of the conclusions that can be drawn.

For the central claim that retrieval-augmented conditioning enables real-world grounded world simulation: The evidence is strong but incomplete. SWM convincingly outperforms baselines on the constructed benchmarks (Table 1), and the qualitative results (Figures 7, 8) show spatially faithful generation over hundreds of meters. However, the benchmarks are small (30 sequences each, 365 frames per sequence), and no statistical significance testing is reported. With only 30 test trajectories per city, the reported metric differences could be influenced by a few outlier sequences where baselines happen to perform particularly poorly. The cross-city generalization results (Busan and Ann Arbor) provide evidence that the learned grounding skill transfers, but the performance gap between cities (SWM TF FID 28.43 on Busan vs. 56.61 on Ann Arbor) suggests substantial domain sensitivity that is not analyzed. The model was trained on Seoul data—a dense Asian metropolis with specific architectural styles, road configurations, and street-view capture characteristics. Its generalization to Busan (another Korean city, likely sharing many visual characteristics) is a weaker test than its generalization to Ann Arbor (a North American city with different architecture, road markings, vegetation, and vehicle types). The paper would be strengthened by analyzing which aspects of the domain gap cause the performance difference (architectural style? lighting? camera characteristics?) and whether targeted data augmentation could reduce it.

For the claim that cross-temporal pairing is the single most important design choice: The ablation (Table 2) strongly supports this: removing cross-temporal pairing causes the largest degradation across every metric, often by factors of 2–8×. However, the ablation only tests a binary comparison (with vs. without cross-temporal pairing). The paper does not explore whether the temporal gap matters—does pairing references from 1 hour apart work as well as 1 month apart? Does the effectiveness saturate at some temporal distance? Nor does it test whether the pairing strategy could be replaced by an explicit dynamic-object masking approach during training (e.g., segmenting and masking vehicles and pedestrians in references before conditioning). If a simpler approach achieved similar results, cross-temporal pairing would be a useful but not essential design choice. Without these comparisons, the claim that cross-temporal pairing is uniquely necessary is supported for the specific architecture tested but not established as a general principle.

For the claim that the Virtual Lookahead Sink outperforms alternative attention sink designs: The ablation (Table 2, Figures 10, 14) clearly shows VL Sink superiority, but the margin is modest on the standard-length benchmark (VL FID 28.43 vs. FP FID 32.41, a 12.3% relative improvement). The benefit becomes more pronounced on the extended 1,460-frame benchmark (VL FID 25.13 vs. FP FID 28.57, a 12.0% relative improvement), but even there, the gap in structural fidelity metrics (mPSNR 13.70 vs. 13.34) is small. Moreover, the FP sink achieves better camera-following accuracy on the extended benchmark (RotErr 0.019 vs. 0.027, TransErr 0.021 vs. 0.029), raising the question of whether the VL Sink's visual quality advantage comes at a cost in trajectory fidelity. The paper does not address this tradeoff or propose a combined approach.

A more concerning limitation is that the paper does not evaluate the VL Sink in isolation from the retrieval mechanism. The VL Sink's effectiveness depends on the quality of the retrieved street-view image near the chunk's endpoint—if the nearest available reference is far from the endpoint or faces the wrong direction, the sink may provide misleading guidance. This could be tested by artificially degrading the sink retrieval (adding positional noise, using wrong-facing references) but is not evaluated. In sparse-coverage areas where retrieval density is low, the VL Sink may actually harm performance by providing an anchor that doesn't match the target trajectory—a failure mode that the paper does not acknowledge or test.

For the claim that geometric and semantic referencing are complementary and both necessary: The ablation strongly supports complementarity: removing geometric referencing disproportionately degrades camera-following and structural metrics, while removing semantic referencing disproportionately degrades appearance metrics (mLPIPS). However, the magnitude of the degradation from removing geometric referencing (mPSNR drop from 14.56 to 12.33) is substantially larger than from removing semantic referencing (mPSNR drop to 14.08), suggesting that geometric warping is the more critical of the two components. This is not acknowledged in the paper's discussion, which treats them as roughly equally important. Furthermore, the ablation does not test whether geometric referencing alone with a higher-quality warping method (e.g., using learning-based warping like GenWarp [37] instead of forward splatting) could close the appearance gap, potentially making semantic referencing redundant. The paper cites GenWarp in the references but does not compare against or incorporate its approach.

For the claim that synthetic data improves trajectory diversity: The ablation is mixed: synthetic data improves structural and camera metrics but worsens FID. This is acknowledged but not deeply explored. The paper would be strengthened by testing whether the synthetic data benefit could be achieved through simpler data augmentation (e.g., applying geometric transformations to real street-view trajectories to simulate non-driving camera motions) rather than requiring a separate rendering pipeline. It would also be informative to see whether the benefit is primarily from pedestrian trajectories, free-camera trajectories, or vehicle trajectories—are all three synthetic trajectory types necessary, or is one category responsible for most of the improvement?

Missing experiments that would strengthen the paper:

1. Varying the temporal gap in cross-temporal pairing. How much temporal separation is needed? Could references from different times of the same day work, or is multi-day/multi-month separation required? This would illuminate the mechanism and guide practical data collection.

2. City-by-city breakdown of baseline failures. Table 1 reports average metrics across the 30 sequences per city, but which specific failure modes drive the metric gaps? A breakdown of failure types (camera drift, structural collapse, dynamic object artifacts, texture degradation) with per-category frequencies would characterize what baselines can and cannot do.

3. Ablating the retrieval mechanism. What happens if the GPS + depth reprojection filtering is replaced with random nearby retrieval? What if K=0 references are provided (pure generation from the base model on the new task)? This would isolate the contribution of retrieval quality from the contribution of the model's generative architecture.

4. Testing on Seoul test sequences. The paper evaluates only on cross-city benchmarks to test generalization, but this means we never see how SWM performs on its training city. A Seoul test set would reveal whether the cross-city performance drop is due to domain gap or due to fundamental limitations of the approach that apply even in-domain.

5. Computational cost comparison. The paper reports SF throughput (15.2 fps) but does not report or compare inference cost against baselines. A FLOPs-per-frame or GPU-seconds-per-sequence comparison would clarify the practical efficiency tradeoffs, particularly since the retrieval pipeline adds inference-time overhead (database query, depth reprojection, VAE encoding of references) that baselines do not incur.

6. Ablating the view interpolation quality. All street-view training data goes through the view interpolation pipeline. How does SWM's performance change if trained on the raw sparse keyframes instead of interpolated videos? This would test whether the interpolation quality (PSNR 25.03) is a bottleneck for the downstream task.

Summary of experimental strengths and weaknesses:

• Strengths: (1) The paper constructs a genuinely new evaluation setting (cross-city benchmarks with retrieval exclusion) that tests the core claim of real-world grounding. (2) The ablation study is comprehensive, covering all major components with clear quantitative impacts that align with mechanistic interpretations. (3) The comparison against six baselines plus two static-scene methods provides strong evidence that the task is distinct from existing model capabilities. (4) The qualitative results show capabilities (kilometer-scale trajectories, text-prompted scene variations, diverse camera motions) that the quantitative metrics cannot fully capture.

• Weaknesses: (1) Small benchmark size (30 sequences × 2 cities) without statistical testing, making metric differences hard to interpret as reliable rather than sampling- dependent. (2) No analysis of domain gap factors driving the Busan vs. Ann Arbor performance difference, leaving the cross-city generalization claim less substantiated than it appears. (3) Missing ablations on temporal gap magnitude, retrieval quality sensitivity, and Seoul in-domain performance. (4) The functional decomposition claim (geometric vs. semantic complementarity) is supported but the relative importance is asymmetric in ways the paper doesn't acknowledge. (5) No computational cost comparison, making practical deployment tradeoffs unclear. (6) The failure cases (Appendix D.1, Figure 15) are not systematically characterized—only one failure mode (abrupt vehicle appearance/disappearance) is shown, but other likely failure modes (structural drift in sparse-coverage areas, incorrect building appearance when references are outdated, failures under extreme text prompts) are not evaluated.

6. Limitations and Trade-offs

The Grounding Fidelity Is Fundamentally Bounded by the Quality and Density of the Street-View Database

The assumption or constraint. SWM's core capability — generating spatially faithful video of real locations — depends entirely on the availability of nearby street-view images at inference time. The retrieval database for Seoul contains 1.2M panoramas, but the paper explicitly acknowledges that "the quality of SWM's generation is closely tied to the quality of its training data" (Appendix D.1) and that "the density of available street-view images varies across locations" (Appendix C.3). The model was trained and evaluated in dense urban environments with thorough street-view coverage; there is no guarantee that the approach transfers to sparser areas (suburban, rural, or less-documented cities).

The consequence. In any real deployment, street-view coverage is non-uniform — some neighborhoods have images every 5 meters, others every 50 meters, and many regions worldwide have no street-view data at all. The paper's own analysis (Appendix C.3, Figure 13) shows that reducing the number of retrieved references from K=5 to K=1 causes mPSNR to drop monotonically (from approximately 14.59 to 14.35). This is a relatively modest degradation for a 5× reduction in references, but it only tests the scenario where references exist and are merely fewer. It does not test what happens when the nearest reference is far from the target trajectory — a scenario that is common in practice when the camera navigates away from roads (into parks, alleys, or pedestrian-only areas) or in cities with sparser coverage. The two-stage retrieval (GPS + depth reprojection) assumes references exist within a useful radius; if the nearest reference is 50 meters away and facing the wrong direction, the depth reprojection filter may reject it entirely, leaving the model with no conditioning for that chunk. What happens then is untested — the model may fall back to its imagined-world generation capabilities (since it was trained with 20% reference dropout), but whether this imagined generation stays faithful to the real location at that distance is unknown.

An additional consequence: the model's grounding fidelity reflects the timeliness of the database. If a building is demolished or a new structure is built between the street-view capture date and the inference date, SWM will generate the old appearance because the references show the outdated geometry. This makes SWM unsuitable for applications requiring up-to-date spatial fidelity without a continuously refreshed database. The paper acknowledges this only implicitly by noting that cross-temporal pairing teaches the model to rely on "persistent scene structure," but persistence is not permanence — urban environments change on timescales of months to years.

What evidence exists in the paper. Appendix C.3 (Figure 13) and the supporting text provide the only quantitative evidence on reference density effects, showing that mPSNR drops monotonically with decreasing K while FID/FVD do not degrade. The paper interprets the stable FID/FVD as evidence that "the underlying video diffusion model retains its generative capability even with fewer reference constraints," but this cuts both ways: stable FID means the video looks realistic, not that it faithfully matches the real location. The mPSNR drop (approximately −0.24 dB going from K=5 to K=1) quantifies the grounding degradation, but this is for a scenario where references still exist — just fewer. No experiment varies the spatial distance between references and the target trajectory, and no experiment tests complete reference unavailability. Figure 11 shows the Seoul coverage area, which spans ~44.8 km east-west and ~31.0 km north-south, but this map only shows where data exists, not its spatial density. The paper acknowledges in Appendix D.1 that "incorporating real video data as it becomes available would further improve generation quality," but does not frame this as the fundamental dependency that it is.

Mitigation status. The paper does not address this limitation except to acknowledge it as a direction for future data collection. The conditioning dropout schedule (20% reference dropout during training) provides a partial hedge — the model can generate without references — but the paper never evaluates the quality of un-grounded generation at inference time or characterizes how far the model can travel from the nearest reference before structural fidelity degrades below an acceptable threshold. A practitioner deploying SWM would need to characterize the coverage-density-to-fidelity relationship for their target region, but the paper provides no methodology or heuristics for doing so.

---

The Difficulty and Cost of Database Construction Are Not Accounted For

The assumption or constraint. SWM's inference-time retrieval depends on a pre-constructed database where every street-view panorama has been processed through Depth Anything V3 (DA3) [27] to estimate metric-scale depth maps and 6-DoF camera poses aligned to real-world GPS coordinates. This processing pipeline (Section 3.1 and Appendix A.2) requires running DA3 on all images — including partitioning long sequences into chunks, running DA3 independently per chunk, and solving a similarity transformation per chunk to align DA3's coordinate frame with GPS coordinates — then rendering each panorama into 8 equi-angular pinhole views, computing Plücker ray maps, and indexing everything for nearest-neighbor search.

The consequence. The paper reports that 440K street-view images were used for training, and the retrieval database contains 1.2M panoramas (yielding ~9.6M pinhole views after rendering 8 views per panorama). Processing 9.6M images through DA3 and the associated alignment pipeline is a substantial engineering effort and computational cost — one that is not quantified anywhere in the paper. For a practitioner wanting to deploy SWM in a new city, this database construction cost must be incurred before any frames can be generated. The paper frames SWM's inference as lightweight (15.2 fps for SF on a single H100, Appendix A.1), but this only accounts for the generation step, not the prerequisite of building the geo-indexed database. If a city planning department wanted to use SWM to simulate a neighborhood that is not yet in a commercial street-view database, they would need to first collect and process panoramic imagery, estimate depth and poses, and build the retrieval index — a multi-step pipeline that the paper does not characterize in terms of cost, time, or required expertise.

Additionally, the quality of the downstream generation depends on the quality of the DA3 depth and pose estimates, which themselves depend on the resolution and field of view of the input panoramas, the presence of texture for feature matching, and the accuracy of the GPS metadata used for metric alignment. The paper notes that GPS metadata "lacks sufficient accuracy" (Section 3.1) and requires the similarity-transformation-based alignment step. Errors in this alignment propagate to the geometric referencing pathway (Equation 2), where depth-based reprojection from the reference to the target viewpoint depends on accurate relative camera poses. If the alignment is off, the warped image x^(i)_{warp,t} will be geometrically misaligned with the target frame, providing incorrect spatial layout cues that could mislead the model. The paper does not evaluate sensitivity to depth or pose errors, nor does it report the accuracy of the DA3 estimates on the street-view data.

What evidence exists in the paper. The paper provides no quantitative characterization of the database construction cost or the accuracy of the estimated depths and poses. Section 3.1 and Appendix A.2 describe the pipeline in detail but do not report metrics (e.g., reprojection error, scale consistency across chunks, DA3 inference time per image). The existence of the pipeline is documented; its cost and accuracy are not. The cross-city generalization results (Table 1) implicitly demonstrate that the pipeline is reproducible — the authors built retrieval databases for Busan and Ann Arbor to run the benchmarks — but no details are provided on how much data was processed for these cities, what the coverage density was, or whether any city-specific tuning was required.

Mitigation status. Not addressed. The paper treats the geo-indexed database as a given resource, analogous to how a language model paper assumes a pretrained tokenizer. But unlike a tokenizer, which is trained once and widely shared, a city-specific street-view database must be constructed for each new deployment location, making the cost and complexity a first-order practical concern. The paper does not propose strategies for reducing this cost (e.g., using lighter-weight depth estimators, reusing poses across nearby panoramas, or amortizing database construction across multiple applications).

---

The Method Has Not Been Demonstrated to Scale Beyond a Single Training City with a Shared Architectural Style

The assumption or constraint. SWM is trained exclusively on data from Seoul — a dense East Asian metropolis with specific architectural styles (high-rise apartments, commercial signage in Hangul, specific road markings and traffic infrastructure) — plus synthetic CARLA data (which uses generic North-American-style urban maps) and the Waymo dataset (captured in North American cities). The paper evaluates cross-city generalization on Busan (another Korean city) and Ann Arbor (a North American city), but does not evaluate on Seoul itself.

The consequence. The cross-city results (Table 1) show that SWM outperforms baselines in both Busan and Ann Arbor, but the performance gap between the two cities is substantial: SWM TF achieves FID 28.43 on Busan vs. 56.61 on Ann Arbor, and mPSNR 14.56 vs. 15.18 (the two metrics move in opposite directions — Ann Arbor has better mPSNR but much worse FID). The paper does not analyze what causes this performance difference, but several hypotheses are plausible: (1) Busan shares architectural and visual characteristics with Seoul (Korean building styles, signage, road layouts), making the domain gap smaller; (2) the street-view imagery for Busan was captured by the same provider (NAVER Map) with the same camera hardware and processing pipeline, while Ann Arbor's MARS dataset [26] uses different capture equipment; (3) the Waymo data and CARLA synthetic data both use North American urban layouts, which may partially bridge the domain gap for Ann Arbor but confuse the model about which visual patterns to expect.

The critical unasked question is: how would SWM perform on a city that shares neither architectural style nor capture characteristics with Seoul — for example, a medieval European city with narrow cobblestone streets, a Middle Eastern city with distinctive architectural features, or a tropical city with different vegetation and building materials? The paper's cross-city generalization claim is supported for two specific cities but provides no evidence about the broader generalization envelope. A practitioner considering SWM for a city unlike Seoul, Busan, or Ann Arbor has no basis for estimating expected performance.

Furthermore, the absence of a Seoul test set means we cannot determine how much of SWM's performance advantage comes from the retrieval-augmented conditioning mechanism versus from training on more data from the target domain. It is possible that SWM's strong Busan performance reflects not a generalizable grounding skill but an easier generalization target (Korean city with Korean street-view capture characteristics), while its Ann Arbor performance reflects a harder but still partially bridged gap (via Waymo and CARLA training data). Without a Seoul baseline, the paper cannot distinguish "SWM learned to ground in real cities" from "SWM learned to ground in Seoul-like cities and partially generalizes elsewhere."

What evidence exists in the paper. Table 1 reports Busan and Ann Arbor results for all models. The Busan-vs-Ann-Arbor performance gap is visible but not analyzed. Section 3.1 and Appendix A.2 describe the Seoul data collection and processing in detail, and Appendix A.3 describes the CARLA synthetic data. The training data mix (Waymo 20%, Seoul 40%, synthetic 40%; Appendix A.1) is reported but its implications for generalization are not discussed. The paper does not report results on any held-out Seoul test set, making it impossible to compute the in-domain-to-out-of-domain performance ratio.

Mitigation status. The paper partially acknowledges this implicitly by evaluating on two cross-city benchmarks rather than only Seoul, which is a stronger test of generalization than most domain-specific generative model papers provide. However, the choice of cities (one culturally and visually similar, one different) and the lack of analysis of the performance gap means the generalization claim remains weakly characterized. Future work would need to evaluate on a systematically varied set of cities spanning different architectural styles, street layouts, capture hardware, and climate zones to map the generalization envelope. The paper's use of synthetic CARLA data, which depicts North American urban environments, suggests that mixing geographically diverse training data could improve generalization, but this is a hypothesis, not a tested claim.

---

Autoregressive Drift Is Reduced but Not Eliminated; Structural Fidelity Remains Far from Reconstruction Quality

The assumption or constraint. SWM's Virtual Lookahead Sink and retrieval-augmented conditioning are designed to reduce the error accumulation that plagues autoregressive generation over long trajectories. The paper demonstrates that these mechanisms improve stability relative to baselines (Table 2, Figures 10, 14) and show qualitatively stable generation over a 2.48 km trajectory (Figure 7a). However, the mechanism only partially solves the problem: it reduces the rate of drift, not the fact of drift.

The consequence. Even with the VL Sink, the structural fidelity metrics (mPSNR, mLPIPS) reported in Table 1 are far from what a pure reconstruction or novel-view-synthesis system would achieve. SWM's mPSNR of 14.56 on Busan and 15.18 on Ann Arbor means the generated frames are, on average, significantly different from the ground-truth frames even in the static regions that should be faithfully reproduced. For comparison, GEN3C [34] — a static-scene video generation method that makes the stronger assumption of temporally consistent multi-view inputs — achieves mPSNR 14.16 on the same benchmark (Appendix C.5, Table 4), indicating that SWM's structural fidelity is only marginally better than a method that is fundamentally disadvantaged (GEN3C assumes static, temporally consistent inputs, which street-view data violates). The mPSNR values are in the 14–15 dB range, which is low for reconstruction tasks (typical well-reconstructed scenes achieve 25–35 dB PSNR). This means the generated static scene structure is a recognizable approximation of the real location — buildings are in approximately the right places, roads follow approximately the right paths — but the exact geometry, texture, and fine detail are not faithfully reproduced.

The paper acknowledges this implicitly by framing SWM as a generative system rather than a reconstruction system: "SWM addresses is fundamentally different from street-view interpolation" (Appendix D.2), and the generated dynamic content is not expected to match the ground truth. But the low structural fidelity even on static regions means SWM is unsuitable for applications that require precise geometric accuracy — for example, measuring building heights for urban planning, identifying specific storefronts for location-based services, or verifying the presence or absence of particular infrastructure elements. The model generates a plausible version of the real city, not a faithful one.

Furthermore, Figure 10 shows that sliding-window FID degrades over time even with the VL Sink — the degradation is slower than without the sink, but it is not eliminated. Over the 1,200-frame trajectory shown in Figure 10, VL Sink FID rises from approximately 40 at frame 0 to approximately 55 at frame 1,200, a 37.5% increase. This means the generated video becomes progressively less realistic-looking as the trajectory length increases, even if the spatial anchoring prevents catastrophic structural collapse. For applications requiring very long trajectories (multiple kilometers), the visual quality at the distant end may be noticeably degraded relative to the start.

What evidence exists in the paper. Tables 1, 2, and 5 report mPSNR, mLPIPS, FID, and FVD at different trajectory lengths. Figure 10 shows temporal degradation of FID for different sink configurations. Table 4 compares against static-scene methods (GEN3C, VMem) on structural metrics. The mPSNR values consistently fall in the 13–15 dB range across all configurations and benchmarks — SWM TF achieves 14.56 on Busan and 15.18 on Ann Arbor (Table 1), the ablated variants range from 12.33 to 14.56 (Table 2), and the long-horizon benchmark yields 13.70 (Table 5). These values are notably low for a system that is claimed to ground generation in real-world observations, and the paper does not discuss what ceiling exists on achievable structural fidelity given the quality of the retrieved references, depth estimates, and geometric warping.

Mitigation status. Not addressed. The paper presents the structural fidelity metrics as evidence that SWM outperforms baselines (which it does — the best baseline achieves mPSNR 12.20 vs. SWM's 14.56 on Busan), but does not discuss the absolute magnitude of these metrics or what they imply about the practical usefulness of the generated output for applications requiring geometric accuracy. The paper does not propose strategies for improving structural fidelity, such as higher-resolution depth estimation, learned warping methods that are more robust to depth errors, or incorporating explicit 3D reconstruction as an intermediate representation.

---

The Revision Model and Search Mechanisms from Prior Test-Time Compute Literature Are Absent

The assumption or constraint. SWM generates video autoregressively in chunks, where each chunk conditions on the previous chunk's output and the retrieved street-view references. There is no mechanism for the model to self-correct, revise, or search over multiple generation candidates. If a chunk is generated with an error — a misaligned building, a distorted texture, an implausible dynamic object — that error becomes part of the history for all subsequent chunks and cannot be undone. This is the standard limitation of autoregressive generation without test-time search or revision.

The consequence. SWM's generation quality is bounded by the single-pass quality of each chunk. The paper does not explore whether sampling multiple candidates per chunk and selecting the best (via a verifier or consistency check) could improve structural fidelity, whether iterative refinement of chunks (generating, checking against retrieved references, and re-generating) could reduce errors, or whether beam search over possible trajectories could find better paths through the generation space. These techniques are standard in language model test-time compute scaling (as studied in the reference example paper) and have been applied to video generation in other contexts, but they are absent from SWM. The result is that the model has no way to recover from generation errors except by hoping the VL Sink and reference conditioning prevent them from occurring in the first place.

A specific failure mode: if the retrieval module returns a reference that is partially occluded or poorly aligned (e.g., a reference image where a truck blocks the building facade that should be visible in the target view), the geometric warping and semantic referencing will condition on degraded information for that chunk. The model has no mechanism to detect this and request a different reference or re-generate the chunk with different conditioning. The output for that chunk will be degraded, and the next chunk will inherit the degraded history. The paper mentions that dynamic objects in training data can cause artifacts (abrupt vehicle appearance/disappearance, Appendix D.1, Figure 15), but does not frame this as a problem that test-time verification could mitigate.

What evidence exists in the paper. None directly. The paper does not ablate or discuss test-time compute strategies. The SF variant generates at 15.2 fps with a single forward pass per chunk — this is fast because it does not perform any search or revision. Whether additional compute per chunk (e.g., generating 4 candidates and selecting the one with the best structural consistency with the reference, or generating a chunk, checking consistency, and re-generating if inconsistent) would yield meaningful improvements is an open question. The paper's computational efficiency reporting (15.2 fps, 24 H100 GPUs for training) suggests a design philosophy of single-pass generation, but this is a choice, not an inherent limitation — and the paper does not justify it.

Mitigation status. Not addressed. The paper does not discuss test-time compute allocation, candidate selection, or iterative refinement as possible extensions. The focus is entirely on the training-time mechanisms (cross-temporal pairing, data construction, conditioning pathways) and the inference-time retrieval + sink design. This is a reasonable scope for a first paper on a new task, but a practitioner wanting the highest possible grounding fidelity would likely explore test-time strategies, and the paper provides no guidance on whether they would help.

---

The Method Has Been Validated on Only One Model Architecture, One Base Model, and One Video Generation Framework

The assumption or constraint. SWM fine-tunes exactly one base model (Cosmos-Predict2.5-2B-I2W [1]), which is a specific 2B-parameter Diffusion Transformer with a specific 3D VAE, a specific pretraining dataset, and specific pretraining objectives. The paper does not experiment with other base model families (e.g., Stable Video Diffusion [3], Wan [49], CogVideoX [56]), other model scales, or other generative frameworks (e.g., autoregressive transformers, GAN-based generators).

The consequence. The paper cannot distinguish which of SWM's design choices are general principles that would transfer to other base models, and which are specific workarounds for quirks of the Cosmos base model. Several design choices are explicitly motivated by properties of the base architecture:

• The intermittent freeze-frame view interpolation strategy is designed to work around the 4-frame temporal compression stride of Cosmos's 3D VAE (Section 3.1). A base model with a different VAE (e.g., 1-frame or 8-frame temporal compression) would not need this strategy, or would need a different one.
• The specific RoPE position offsets (G=50 for semantic references, Δ_VL=5 for the VL Sink) are tuned for the Cosmos architecture's position embedding conventions. Different models with different positional encoding schemes (learned positions, ALiBi, no position encoding) would need different designs.
• The channel concatenation approach for geometric referencing requires the DiT's input projection to accept additional channels (from 18 to 34 or more, depending on configuration). Models with different input processing architectures might need different conditioning injection mechanisms.
• The training hyperparameters (learning rate 4.8×10⁻⁵, batch size 48, 10K iterations) are tuned for Cosmos-2B on 24 H100 GPUs. Other base models might need different schedules or might not converge with the same settings.

More fundamentally, the paper does not demonstrate that retrieval-augmented conditioning in general enables real-world grounded world simulation — only that it works when fine-tuning this specific base model on this specific data. If a practitioner has a preferred base model for their application (e.g., a model optimized for their target resolution, frame rate, or domain), the paper provides no evidence that SWM's techniques would transfer. The cross-city generalization results (Busan and Ann Arbor) demonstrate generalization across geographic locations, not across model architectures.

What evidence exists in the paper. The paper is transparent about using Cosmos-Predict2.5-2B-I2W (Section 5.1, Appendix A.1) and describes the base architecture in detail. However, it provides no ablations varying the base model, no discussion of how the design choices might change for different architectures, and no experiments with alternative VAE temporal compression ratios or conditioning injection methods. All results — quantitative (Tables 1, 2, 4, 5, 3) and qualitative (Figures 7, 8, 9, 16) — are from the single Cosmos-based fine-tuned model.

Mitigation status. Not addressed. The paper does not claim architecture-independence, but it also does not discuss the extent to which the proposed techniques are tied to the chosen base model. A practitioner reading the paper might reasonably ask: "Can I apply cross-temporal pairing and the VL Sink to my preferred video diffusion model, or do I need to use Cosmos?" The paper provides no evidence either way. Future work could test the transferability of the training data construction pipeline (cross-temporal pairing, view interpolation) to other base models, and the portability of the inference-time mechanisms (geometric/semantic referencing, VL Sink) to other DiT architectures with different conditioning interfaces.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper establishes that the boundary between world simulation and real-world reconstruction is not a fundamental architectural divide but a data design and conditioning problem. Prior to SWM, the research communities addressing "generate dynamic, interactive environments" (video world models) and "faithfully model real cities" (3D reconstruction) operated with different objectives, different evaluation criteria, and different assumptions about what information is available at inference time. SWM demonstrates that a generative model — without any explicit 3D reconstruction, without learned spatial representations maintained across generation, and without temporally consistent multi-view inputs — can produce video that is simultaneously dynamic (responding to text prompts, synthesizing novel scenarios) and spatially faithful to a specific physical location at city scale.

This is not a paradigm shift in the sense of introducing a fundamentally new generative architecture — SWM fine-tunes an existing DiT. The shift is conceptual: the paper's central finding is that the primary obstacle to grounding generation in real-world observations is not the lack of geometric reasoning capacity in generative models, but the temporal inconsistency between real-world reference sources and the scene being generated. Cross-temporal pairing is the mechanism that operationalizes this diagnosis, and the ablation (Table 2) showing it is the single most impactful design choice — causing the largest degradation when removed across every metric — provides the strongest evidence that this diagnosis is correct and that prior work's failure to address temporal inconsistency was the root cause of the gap between world simulation and real-world grounding.

The paper's reconciliation of apparently contradictory capabilities in prior work is worth noting. Video world models can generate dynamic content (moving cars, changing weather, physical interactions) but hallucinate all scene geometry beyond the first frame. 3D reconstruction systems are faithful to real locations but cannot generate dynamics. Geometry-aware generation methods (Gen3C, VMem) can condition on multi-view references but assume temporally consistent inputs and produce frozen dynamic objects (Table 4). SWM's architecture — cross-temporal pairing to separate persistent from transient content, dual geometric/semantic referencing to provide both spatial layout and appearance conditioning, and a dynamic lookahead sink to prevent drift — shows that a single model can simultaneously achieve dynamic generation, multi-view conditioning, and tolerance of temporal inconsistency. The three-way reconciliation suggests that these were never fundamentally competing objectives; they were optimization targets that prior systems failed to jointly satisfy because they lacked the right training data strategy.

Which research directions become more attractive:

• Retrieval-augmented generation for grounding in any geotagged visual database becomes a principled alternative to explicit 3D reconstruction for location-aware applications. The paper shows that the key bottleneck is not retrieval quality or model capacity but training the model to correctly interpret temporally inconsistent references — a data design challenge, not an architectural one.
• Cross-temporal and cross-modal pairing strategies become a general design pattern for any system that conditions generation on observations from a different time, sensor, or domain than the target output. Satellite-to-street-view, historical-to-present-day, or cross-seasonal generation all face the same fundamental challenge of separating invariant structure from transient or domain-specific content.
• Attention sinks as dynamic, information-seeking mechanisms rather than static persistence heuristics. The VL Sink shows that the sink should carry the most relevant spatial information for the current generation context, not a fixed early frame — and that when external observations are available, the sink should be grounded in those observations.
• Large-scale street-view databases as training resources for spatial understanding. The paper uses 440K Seoul street-view images; globally, billions of such images exist. The cross-city generalization results (Table 1) show the learned grounding skill transfers without retraining, suggesting that training on larger and more diverse street-view corpora could produce a broadly generalizable world-grounding capability.

Which directions become less attractive:

• Pure imagined-world generation without any grounding mechanism loses its claim to being a complete world simulation solution. If street-view databases are available and retrieval-augmented conditioning provides 4–5× better camera-following accuracy (TransErr 0.015 vs. 0.079–0.083 for baselines) and measurably better structural fidelity, imagined-only models are no longer sufficient for applications requiring location faithfulness — which includes most real-world deployment scenarios (urban planning, autonomous driving testing, location-based services).
• Geometry-aware generation methods that assume temporally consistent multi-view inputs are shown to be a dead end for real-world grounding unless they are extended with cross-temporal training strategies. GEN3C and VMem (Table 4) achieve competitive structural fidelity on static regions but fail catastrophically on dynamic content when references are temporally inconsistent — and real-world references are always temporally inconsistent.
• Training world models from scratch for location-specific applications becomes unnecessary when fine-tuning a pretrained model with retrieval-augmented conditioning on a modest dataset (440K images) achieves cross-city generalization. The paper's training cost (10K iterations on 24 H100s) is small relative to pretraining a video foundation model, suggesting the efficient path forward is to build retrieval-augmented adapters for existing models rather than training location-specific generators.

---

Follow-Up Research This Work Enables

Characterizing the generalization envelope across architectural, geographic, and temporal domains. The paper demonstrates generalization to two unseen cities (Busan and Ann Arbor) but observes a substantial performance gap between them (SWM TF FID 28.43 vs. 56.61). A systematic follow-up would construct a benchmark spanning cities selected to vary along specific axes: architectural style (East Asian, European, Middle Eastern, North American, South American), street layout (grid, radial, organic medieval), climate/vegetation (tropical, arid, temperate), and capture characteristics (different street-view providers, camera heights, image resolutions). For each city, the same retrieval database construction pipeline would be applied, and SWM would be evaluated zero-shot (no fine-tuning on target city data). The key measurements would be: (1) which axes of variation cause the largest performance degradation, (2) whether degradation is predominantly in structure (mPSNR, LPIPS) or visual quality (FID, FVD), and (3) whether adding even a small amount of target-city data (e.g., 1K images) closes the gap. This would transform the qualitative claim of "cross-city generalization" into a quantitative characterization of the generalization function, informing practitioners about expected performance in their target city and guiding data collection priorities.

Learned difficulty estimation and adaptive compute allocation for retrieval-augmented generation. The paper identifies that reference density varies across locations (Appendix C.3) and that structural fidelity degrades monotonically when fewer references are available per chunk (mPSNR dropping from ~14.59 at K=5 to ~14.35 at K=1, Figure 13), but the model currently treats every chunk identically — same K, same sink design, same generation budget. A natural extension, inspired by the compute-optimal scaling framework from the reference example paper, would be to train a lightweight difficulty predictor that estimates, from the retrieved references and the camera trajectory alone, how challenging a given chunk will be to generate faithfully. Chunks where the nearest reference is far from the target viewpoint or where depth reprojection coverage is low could receive additional compute: generating multiple candidates and selecting the most consistent via a structural fidelity check against the reference, or running additional denoising steps. The specific experiment: replace the fixed K=5 retrieval with an adaptive policy that allocates a total budget of B reference retrievals across the trajectory, using the difficulty predictor to assign more references to high-uncertainty chunks and fewer to low-uncertainty chunks. Compare against uniform K at the same total budget. This would test whether the monotonic mPSNR improvement with K can be leveraged more efficiently through allocation, and whether the 20% reference dropout training (which teaches the model to handle missing references) enables graceful degradation under budget constraints.

Disentangling the contribution of geometric warping quality from the contribution of the warping mechanism. The paper ablates geometric referencing entirely (Table 2), showing it is critical for camera-following accuracy and structural fidelity. But the warping method — depth-based forward splatting from monocular depth estimates — is known to produce artifacts (holes, edge fringing, misalignments) that limit conditioning quality. The paper does not test whether these artifacts are the bottleneck or whether a higher-quality warping method would close the remaining structural fidelity gap. A direct experiment would replace the forward-splatting warping with a learning-based approach (e.g., GenWarp [37], cited in the paper's references but not compared against) that uses a generative model to produce higher-fidelity warped views, then fine-tune SWM with this improved geometric conditioning and measure whether mPSNR and mLPIPS improve. The null result — if improved warping does not improve structural fidelity — would imply that the bottleneck is not warping quality but rather the fundamental limitation of single-reference geometric conditioning (a single reference cannot provide information about occluded or out-of-view regions). The positive result — if mPSNR jumps from 14.56 to, say, 17–18 dB — would identify warping quality as the primary lever for improving grounding fidelity and motivate investment in better warping methods.

Scaling laws for retrieval-augmented world model training. The paper trains on 440K real images + 12.7K synthetic videos, but gives no indication of whether performance is saturating at this data scale or whether more data (from Seoul or other cities) would yield continuing improvements. A scaling study would train SWM variants on logarithmically spaced subsets of the training data (e.g., 10K, 40K, 160K, and the full 440K Seoul images), evaluating each on Busan-City-Bench to measure how cross-city generalization improves with training data scale. The key question is whether the learned grounding skill follows a power-law scaling trend (suggesting investment in larger street-view datasets would be worthwhile) or saturates quickly (suggesting the current data scale is sufficient and other bottlenecks dominate). The synthetic data scaling could be studied separately: train variants with 1K, 4K, and the full 12.7K synthetic videos, measuring the trajectory diversity benefit (RotErr, TransErr) as a function of synthetic data scale. This would inform whether the 40% training mix ratio for synthetic data is near-optimal or whether more synthetic trajectories would further improve camera-following accuracy.

Combining SWM's retrieval-augmented conditioning with explicit 3D reconstruction as an intermediate representation. The paper positions SWM as an alternative to explicit 3D reconstruction — the model learns to ground generation through conditioning without building an explicit 3D model. But this choice means the model has no persistent spatial representation; it re-derives spatial understanding from retrieved references at each chunk. An alternative architecture would first run an efficient 3D reconstruction (e.g., Gaussian Splatting) on the retrieved street-view images to build a local radiance field, then render clean, multi-view consistent reference images from that field to condition the generative model. This would replace the noisy, single-reference geometric warping with high-quality rendered views, and would provide the semantic referencing pathway with multi-view consistent reference images. The experiment: for each chunk, reconstruct a local 3DGS model from the K=5 retrieved references (using known camera poses from the DA3 pipeline), render warped views and appearance reference views from this model, and feed them into SWM in place of the raw retrievals. Measure whether structural fidelity (mPSNR, mLPIPS) improves relative to the raw-retrieval baseline, and characterize the computational overhead of the reconstruction step. This hybrid would combine the generative flexibility of SWM with the geometric consistency of explicit reconstruction, potentially achieving the best of both worlds — but only if the reconstruction is fast enough to not dominate inference time.

Stress-testing the Virtual Lookahead Sink under adversarial retrieval conditions. The paper demonstrates the VL Sink's benefit when retrieval works well (nearby street-view images available near each chunk endpoint), but does not test what happens when the sink is misleading — when the nearest street-view image to the endpoint is captured from the wrong direction, is heavily occluded, or depicts the location under radically different conditions (night vs. day, construction vs. normal). A systematic stress test would construct a benchmark where sink retrieval is deliberately degraded: (1) spatial perturbation — retrieving the sink from increasing distances from the true endpoint (10m, 50m, 100m, 200m); (2) orientation perturbation — retrieving sinks facing 90°, 180°, or 270° away from the target viewing direction; (3) temporal perturbation — retrieving sinks from different seasons or times of day than the target scene. For each condition, measure how much the VL Sink's benefit degrades relative to the no-sink baseline. The key finding would be the tolerance envelope: at what spatial/orientation/temporal distance does the VL Sink transition from helpful anchor to harmful distraction? This would inform practical deployment — if the sink is robust to, say, 50m spatial error and 90° orientation error, it can be used confidently in moderate-coverage areas; if it requires near-perfect retrieval, its practical utility is limited to dense-coverage urban cores.

---

Practical Applications and Downstream Use Cases

Autonomous driving scenario generation with real-world road geometry. Current autonomous driving simulation pipelines rely on either (1) recorded real-world data, which is limited to what was captured and cannot generate novel scenarios, or (2) synthetic simulators like CARLA, which provide controllable scenarios but lack photorealism in appearance and suffer from a persistent sim-to-real domain gap. SWM offers a third option: given a real driving route (specified as GPS coordinates), it retrieves street-view imagery of the actual road and generates photorealistic video along that route while allowing variation in traffic patterns, weather, lighting, and edge-case events (via text prompts). The key advantage over pure synthetic simulation is that the generated video faithfully reflects the real road geometry, building placements, and visual appearance of the actual location — reducing the domain gap that degrades perception model performance when tested in simulation. The practical pipeline: take a route of interest from an HD map, query SWM to generate diverse scenarios along that route (clear day, heavy rain, nighttime, pedestrian crossing, construction zone), and use the generated videos to test perception and planning models. The paper's demonstration of diverse text-prompted scenarios (Figure 7a: tsunami, golden-hour sunset) and long-horizon stability over 2.48 km suggests this is feasible for real deployment routes, though the 15.2 fps SF throughput would need to scale for large-scale scenario generation (a 1 km route at 30 fps takes ~20 seconds to generate at 15.2 fps).

Urban planning visualization with photorealism and dynamism. Urban planners and architects currently use two categories of tools for visualizing proposed changes: (1) 3D rendering engines (Unreal Engine, Unity) that produce photorealistic stills and flythroughs but require labor-intensive 3D asset creation and cannot easily incorporate existing real-world appearance, and (2) street-view imagery or point clouds that show what currently exists but not what could be. SWM bridges this gap: a planner specifies a walking path through a neighborhood (as GPS coordinates) and a text description of proposed changes ("add a bike lane with green pavement," "replace street parking with extended sidewalk cafes," "plant cherry blossom trees along both sides"), and SWM generates a video of the walk that grounds the proposed changes in the actual existing building facades, road geometry, and spatial context. The benefit is not just photorealism — it's that the visualization is persuasive because it shows the proposal in the recognizable context of the real location, helping stakeholders and community members understand how changes would look and feel. The paper's structural fidelity metrics (mPSNR 14.56, mLPIPS 0.392 on Busan-City-Bench) indicate that, while the output is not a pixel-perfect reconstruction, it is sufficiently faithful that existing landmarks, building styles, and street layouts are recognizable — the threshold for "this looks like my street" rather than "this looks like a generic street." For applications where qualitative recognizability matters more than metric-level fidelity, this is sufficient. The limitation: the paper does not demonstrate localized edits (changing specific buildings or street elements while preserving everything else), only global scene-level text prompts. Extending SWM to support spatially localized editing would substantially increase its practical value for this use case.

Location-based entertainment and virtual tourism with user control. SWM enables an experience that neither pure game engines nor pure street-view browsing can provide: exploring a real city under novel, user-specified conditions. A tourism platform could let a user drop a pin anywhere in Seoul, specify a walking route, and explore that route under conditions that don't exist in any captured footage — autumn foliage at peak color along a route captured in summer, the city during a festival with decorations and crowds, or Seoul as it appeared in a historical period (if historical street-view references were available). The user's text prompt provides direct creative control, and the retrieval-augmented conditioning ensures the generated experience is anchored in the real city rather than an imagined approximation. The paper's cross-city generalization results (Table 1) suggest this could extend to any city with sufficient street-view coverage without requiring per-city retraining — the trained SWM model could be deployed as a service that takes a street-view database and a user query as input. The practical bottleneck is the database construction cost (DA3 depth/pose estimation for the target city's street-view images), which would need to be amortized across many user queries. For a tourism platform covering 50 major cities, the database construction is a one-time cost; the per-query cost is just retrieval + generation (15.2 fps on an H100), which at scale could serve many simultaneous users through batching and model serving infrastructure.

---

When to Prefer This Method

The paper does not articulate an explicit tradeoff against named, deployable alternatives in a way that yields a crisp decision rule. It compares against video world model baselines (Aether, DeepVerse, Yume1.5, HY-World1.5, FantasyWorld, Lingbot) that were not designed for real-world grounding — these are not alternatives that a practitioner would choose between for a location-grounded application; they are straw-man comparisons that demonstrate the task is new. It compares against static-scene methods (GEN3C, VMem) that are designed for a different setting (temporally consistent inputs). It does not compare against the most natural alternative pipeline: (1) reconstruct a 3D scene from street-view images using Gaussian Splatting or NeRF, (2) render novel viewpoints along the desired trajectory, and (3) apply a video stylization or dynamics-insertion model to add motion and text-prompted variations. This hybrid reconstruction-plus-generation pipeline would be the practical baseline that a practitioner would likely consider before adopting SWM, but it is not evaluated.

The paper also does not articulate tradeoffs between using SWM versus collecting more real video data (e.g., sending a capture vehicle along the desired trajectory). For one-off generation of a specific route, capturing real video may be cheaper and higher quality than building a street-view database, training a model, and generating synthetic video. SWM's value proposition is for scenarios where many different trajectories or text-prompted variations are needed for the same city — the fixed cost of database construction is amortized over many generations — but the paper does not quantify the break-even point (how many generations are needed before SWM is cheaper than real capture?). Without this analysis, and without a comparison against the reconstruction-plus-generation pipeline, a decision rule would be speculative rather than grounded in the paper's evidence. The appropriate characterization is that the paper establishes feasibility and relative performance against other world models, but does not provide the head-to-head comparisons against practical alternative workflows that would support a "prefer A when X, prefer B when Y" decision framework.