Plenoptic Video Generation

ArXiv: 2601.05239

🎯 Pitch

Generating novel camera paths from a single video produces inconsistent hallucinations in unseen regions—different views show contradictory geometry. PlenopticDreamer solves this by autoregressively feeding previously generated viewpoints into future generations, forcing the model to maintain a persistent 3D scene memory across time and views.

---

1. Executive Summary

PlenopticDreamer introduces a camera-controlled generative video re-rendering framework that enforces long-term spatio-temporal memory across multi-view generations, addressing the failure of prior single-shot methods to maintain consistent hallucinations in regions unseen from the source view. The framework operates on the Basic (in-the-wild) and Agibot (robotic manipulation) benchmarks using a Cosmos-Predict2.5-2B backbone, employing an autoregressive multi-in–single-out diffusion model with a 3D FOV–based video retrieval mechanism (selecting the top-k previously generated video–camera pairs via spatial co-visibility scoring), progressive context-scaling (gradually increasing conditioning video count from 1 to 4 during training), and self-conditioned training (fine-tuning on the model's own synthetic outputs to mitigate error accumulation). The method achieves state-of-the-art view synchronization—41.2K matched pixels at 12 shots versus 27.6–36.2K across baselines—while maintaining comparable camera accuracy, establishing that coherent multi-view plenoptic generation is achievable only when generative hallucinations are explicitly synchronized through retrievable spatio-temporal memory rather than generated independently per view.

2. Context and Motivation

The Core Problem: Single-View Video Re-Rendering Cannot Maintain Multi-View Coherence

The fundamental problem PlenopticDreamer addresses is that existing camera-controlled generative video re-rendering methods break down when you need to generate multiple consistent views of the same scene. These methods work well in isolation—given one source video and one target camera trajectory, they can produce a plausible novel view by hallucinating what the scene would look like from that new angle. But when you ask them to generate multiple such views sequentially, they treat each generation independently. The result: in regions that were occluded or out-of-frame in the source video, each generated view hallucinates something different, producing jarring inconsistencies that destroy the sense of a coherent underlying scene.

This is not a niche concern. Multi-view video re-rendering is the enabling technology for capturing the plenoptic function of a dynamic scene—the complete light field describing how the scene would appear from any viewpoint at any moment. As the paper frames it in the introduction, video frames are "discrete samples of the underlying plenoptic function," and camera control is what "shapes the captured light field, emphasizing visual focus, and guiding the viewer's attention." To reconstruct this plenoptic function from limited observations, every hallucinated view must be consistent with every other—the painting on the wall behind the camera, the traffic light on the street corner, the robotic gripper approaching from the side—must appear identically regardless of which novel trajectory renders the relevant region.

The paper frames the failure mode precisely in Section 1:

"they fail to maintain consistent spatio-temporal hallucinations in regions unseen from the source view. The inherent stochasticity of diffusion models, combined with their limited long-range spatial memory, leads to geometric misalignment and view desynchronization across different camera-conditioned generations."

This is the gap PlenopticDreamer fills. Prior methods are single-shot: one input, one output, no memory of what was hallucinated before. PlenopticDreamer is autoregressive and memory-backed: each new view is conditioned on previously generated views, explicitly synchronized through a retrieval mechanism that identifies which past views share visible content with the current target.

Why This Matters: From Immersive Content to Embodied AI

The paper identifies two driving application domains, but the implications are broader than the explicit use cases listed.

Immersive content creation. If you can take a single video and re-render it under arbitrary novel camera trajectories—rotating around a subject, zooming into a detail, shifting perspective from one character to another—you unlock post-hoc cinematography without reshooting. This is not theoretical; the paper demonstrates transformations including left/right rotations, azimuth and elevation shifts, distance variations, and dynamic focal length changes on in-the-wild videos (Section 4.2, Figure 4). But for this to be practically useful, the re-rendered views cannot introduce glaring inconsistencies. A viewer orbiting around a street scene should see the same buildings, the same pedestrians, the same reflections from every angle. If the background changes every time the camera shifts, the illusion collapses and the content is unusable.

Embodied AI and robotics. The Agibot benchmark (Section 4.3) targets a specific robotic manipulation scenario: given a head-mounted camera view of a robot performing a task, generate the corresponding views from the left and right gripper cameras. This is not merely a visualization nicety—it serves as data augmentation for robotic learning. Training visuomotor policies on multi-view data improves spatial reasoning and manipulation precision. If generated gripper views hallucinate objects in different positions or with different appearances than the head view, the synthetic data actually degrades policy learning rather than improving it. The paper's demonstrated head-view→gripper-view and gripper-view→gripper-view transformations (Figure 5) are only valuable because they maintain consistency across viewpoints.

The plenoptic function as a learned representation. At a deeper level, the paper can be read as an argument that generative models, when properly augmented with memory, can serve as implicit scene representations. A model that generates consistent novel views has, in some sense, internalized the 3D geometry and appearance of the scene. This connects to a long line of work in neural rendering (NeRF, 3D Gaussian Splatting) but with a key difference: PlenopticDreamer operates entirely in the generative paradigm, without explicit 3D reconstruction. The plenoptic function is hallucinated rather than inferred from geometry. Whether this implicit representation is as reliable as explicit 3D methods is an open question, but the threshold requirement—consistency across views—is what the paper sets out to establish.

The Prior Art and Its Specific Failures

The paper situates itself against two broad families of prior work (Section 2): camera-controlled video generation and memory mechanisms for video generation. Each has partial solutions that fail on the full multi-view consistency problem.

Camera-Controlled Video Generation: Three Approaches, None Multi-View-Consistent

Single-view generation methods (six cited: [46, 58] for explicit 6DoF camera poses, [4, 5, 24, 63, 68] for Plücker raymap conditioning) have achieved impressive frame-level camera control. Given a text prompt or image and a target camera trajectory, they synthesize a video clip that follows that trajectory. Training-free approaches [27, 28, 40, 64] and object-motion extensions [19, 57] exist as well. But these methods are inherently single-shot: each generation is independent, with no mechanism to share information across views. If you generate a "rotation left" view and then an "azimuth right" view from the same source video, nothing ties them together—the model rolls the dice independently on what lies behind the camera in the source view.

Multi-view video generation methods ([36] for object-level, [7, 35] for scene-level) explicitly aim for cross-view consistency. However, the paper notes they focus on generation from scratch (text-to-multi-view-video) rather than re-rendering from an existing source video. The distinction matters: re-rendering must preserve the visual content of the source view while hallucinating only the unobserved regions, whereas generation from scratch can freely invent all content. This makes re-rendering harder in one sense (the constraint of matching the source view) but potentially easier in another (the source view provides a strong anchor for geometry and appearance).

Video-to-video re-rendering methods are the closest to PlenopticDreamer. Some approaches [6, 53] perform implicit re-rendering with minimal 3D supervision, learning to map source-view content to novel views purely in pixel space. Others [22, 46, 61, 62, 66] project video context into explicit 3D representations (point clouds, depth maps, 3D point tracks) and then synthesize novel views by rendering these representations. The paper's key critique is stated explicitly:

"Despite these advances, none integrate memory mechanisms to maintain long-term spatio-temporal coherence across multiple views."

This is the diagnostic that structures the paper's contribution: prior re-rendering methods can do one view well, but they have no mechanism to remember what they hallucinated in previously generated views and enforce consistency in subsequent ones.

The paper's evaluation confirms this. In Figure 4 and Table 1, ReCamMaster [6]—a representative video-to-video re-rendering method—shows reasonable single-view quality but clear view desynchronization under large-angle viewpoint changes. TrajectoryCrafter [66] and Trajectory-Attention [61] leverage 3D point tracking from the source video, but as the paper notes:

"without updating the 3D memory using newly rendered content, they fail to maintain consistent cross-view synthesis."

That is, these methods extract geometric cues from the source video, but they never update those cues based on what the model actually generates in novel views. The 3D representation is static, frozen from the source, and any hallucinated content in the novel views is not fed back into the representation. This creates a fundamental ceiling on multi-view consistency.

Memory Mechanisms for Video Generation: Four Paradigms, None Camera-Guided for Re-Rendering

The paper reviews four existing memory paradigms (Section 2) and positions PlenopticDreamer as a novel fifth category:

Frame-level memory methods [13, 14, 47, 60, 65] store key historical frames and retrieve the top-k based on camera-pose similarity for conditioning future frames. This is closest in spirit to PlenopticDreamer, but with a critical difference: these methods retrieve individual frames, not entire videos, and the retrieval is typically based on camera-pose similarity alone. As the paper points out in Figure 3, frame-level retrieval ignores the temporal coherence of video segments. Selecting frames independently from different moments in different videos can create temporal artifacts where the conditioning context is temporally inconsistent with itself. PlenopticDreamer's video-level retrieval (averaging frame-wise co-visibility across a full trajectory) addresses this by selecting complete coherent video segments.

Latent-level memory methods [11, 37, 43, 44, 67] maintain hierarchical latent representations with coarse long-term tokens and fine-grained short-term tokens, adaptively retrieving salient features during inference. These are designed for long video generation (extending temporal coherence), not multi-view consistency. They do not encode spatial relationships between views or provide mechanisms for cross-view synchronization.

3D-level memory methods like VMem [38] and SPMem [59] reconstruct explicit 3D structures (surfels or point clouds) from video and render geometry-aware representations for novel-view synthesis. These come closest to addressing the multi-view consistency problem by constructing a persistent 3D representation. However, they inherit the challenges of explicit 3D reconstruction: sensitivity to camera calibration accuracy, representation capacity limits (surfels and point clouds are sparse compared to the full visual appearance), and difficulty with complex geometry (thin structures, reflections, transparency). PlenopticDreamer sidesteps these issues by operating entirely in the generative paradigm—the "memory" is previous video outputs, not an explicit 3D model.

Network-level memory like TTT-Video [16] leverages Test-Time Training layers to record input tokens and update model weights. This is computationally expensive (requiring per-scene optimization) and not designed for cross-view synchronization.

The Missing Component: Retrievable Spatio-Temporal Memory for Generative Re-Rendering

What unifies all these gaps is the absence of a mechanism that (1) stores previously generated videos along with their camera trajectories, (2) retrieves the most relevant videos for conditioning the next generation based on spatial co-visibility (not just camera-pose proximity), and (3) conditions the diffusion process on these retrieved videos so that the new generation is synchronized with all prior views.

This is exactly PlenopticDreamer's contribution. The "memory" is not a compressed latent, not an explicit 3D mesh, and not an updated weight vector—it is the literal previously generated videos, stored as-is in a memory bank. Retrieval is guided by a 3D field-of-view (FOV) mechanism that computes spatial co-visibility: how many 3D points sampled within the current camera frustum are also visible in each previously generated view's frustum. This is more robust than camera-pose distance because two cameras can have similar poses but look at completely different things (e.g., one pointing at a person's face, another at the wall behind them), while two cameras with different poses can share substantial visible content (e.g., two views of the same object from different angles). The FOV-based retrieval identifies cameras that see the same stuff, not cameras that are physically close.

The autoregressive formulation then conditions each new generation on these retrieved contexts. Because the diffusion model sees the actual pixel content of what was previously generated in the overlapping regions, it can align its hallucinations accordingly. This is the mechanism that enables the synchronized hallucinations highlighted in Figure 1—the robot's right side appears identically across three different camera trajectories because each trajectory's generation was conditioned on video–camera pairs that covered that region.

How the Paper Positions Itself

PlenopticDreamer is explicitly positioned as the first memory-based framework for camera-controlled generative video-to-video re-rendering (Section 2, final sentence of the memory subsection). This is a specific and defensible claim: prior re-rendering methods lack memory of previous generations; prior memory methods are not designed for multi-view re-rendering; prior multi-view methods generate from scratch rather than re-render from a source video.

The paper is also positioned as addressing the stochasticity problem inherent in diffusion-based generation. This is stated in the abstract and introduction as the root cause of view desynchronization: diffusion models, by their nature, randomize the details of hallucinations. Without a shared conditioning signal across views, these randomizations are independent, producing inconsistency. PlenopticDreamer's memory mechanism directly addresses this by making the conditioning signal across views non-independent—it explicitly encodes what was previously generated.

The training strategies—progressive context-scaling and self-conditioned training—are positioned not as novel ideas per se but as necessary engineering for stability. Progressive context-scaling addresses the convergence difficulties that arise when training with large context windows from scratch (Section 3.3): the model first learns with one conditioning video, then two, then three, then four, each stage building on the previous. Self-conditioned training addresses the train-test distribution shift in autoregressive generation: during training, conditioning videos are all ground-truth, but during inference, conditioning videos are the model's own imperfect generations. Fine-tuning on synthetic outputs closes this gap, making the model robust to the errors that accumulate over long autoregressive chains.

This is a pragmatic, systems-oriented paper rather than a theoretical one. Its contributions are architectural (the autoregressive multi-in–single-out formulation with FOV-based retrieval) and methodological (progressive training, self-conditioning, long-video conditioning), validated through comprehensive benchmarks rather than analytical proofs. The two benchmarks—Basic (in-the-wild diverse scenes with 12 sequential camera trajectories) and Agibot (robotic manipulation with head-to-gripper transformations)—cover complementary failure modes: Basic tests generalization across diverse visual content, while Agibot tests domain-specific consistency under precise geometric relationships between cameras.

3. Technical Approach

3.1 Reader Orientation

PlenopticDreamer is an autoregressive video generation system that takes a single source video and a sequence of target camera trajectories, then synthesizes a coherent set of novel-view videos—one per trajectory—such that hallucinated content (regions invisible in the source) remains consistent across all generated views. The core problem it solves is the stochastic desynchronization of independently generated multi-view videos: when each novel view is generated in isolation, diffusion models randomly hallucinate different content for the same unseen region, breaking the illusion of a single underlying scene. The solution's shape is to condition each new generation on previously generated views, using a retrieval mechanism that identifies which past views share visible content with the current target, thereby forcing the diffusion model to align its hallucinations with what it already produced.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components connected in an autoregressive loop:

1. Video and Camera Memory Bank — stores every previously generated video alongside its camera trajectory. This is the system's long-term spatio-temporal memory. It starts seeded with the source video and its camera parameters.

2. 3D FOV–Based Video Retriever — given the next target camera trajectory, queries the memory bank by computing spatial co-visibility: for each stored video, it samples 3D points in the overlapping field of view and counts how many are visible in both frustums. It selects the top-k most co-visible video–camera pairs to serve as conditioning context.

3. Multi-In–Single-Out Autoregressive Video Generator — a flow-based video diffusion transformer (DiT) that conditions on k retrieved video–camera pairs plus the target camera to generate the next video. It processes all inputs by temporally concatenating them along the frame dimension, then denoises from pure noise to produce the target video. The model is fine-tuned from Cosmos-Predict2.5-2B with frozen non-self-attention parameters.

4. Camera Encoder — converts each camera trajectory (extrinsics + intrinsics) into Plücker raymaps (6D ray representations at each pixel), patchifies them, projects them to latent dimensionality, and channel-wise adds them to video tokens before self-attention.

5. Training Infrastructure — two specialized training strategies: progressive context-scaling (gradually increasing the number of conditioning videos from 1 to k during training to stabilize convergence) and self-conditioned training (fine-tuning on the model's own synthetic outputs to reduce error accumulation during autoregressive inference).

Information flow: Source video + camera → stored in memory bank → for each target trajectory: (1) FOV retriever selects top-k context pairs from memory, (2) camera encoder produces Plücker raymaps for all k+1 cameras, (3) video tokens from k context videos are temporally concatenated with noise tokens for the target, (4) camera tokens are added channel-wise, (5) DiT denoises to produce target video, (6) new video–camera pair is appended to memory bank → repeat for next trajectory.

3.3 Roadmap for the Deep Dive

• First, the flow-matching video DiT backbone — establishes the base generative model, its training objective, and how noise scheduling works, since every subsequent component builds on this foundation.

• Second, the autoregressive generation paradigm and in-context conditioning — explains the core architectural innovation: reformulating multi-view generation from single-shot to sequential, and how multiple video–camera pairs are packed into the DiT's context window via temporal concatenation.

• Third, camera conditioning via Plücker raymaps — details the representation of camera information as per-pixel 6D ray coordinates, why this representation is chosen over alternatives like explicit pose parameters, and how it is injected into the DiT.

• Fourth, the 3D FOV–based video retrieval mechanism — walks through Algorithm 1 (video retrieval) and Algorithm 2 (divide-and-conquer inference), explaining how spatial co-visibility is computed, why video-level aggregation outperforms frame-level retrieval, and how the system handles the case where retrieved videos exceed model context capacity.

• Fifth, the training strategies — covers progressive context-scaling (the schedule, why direct large-context training fails) and self-conditioned training (the two-stage procedure, the synthetic data generation pipeline, the train-test gap it addresses).

• Sixth, long-video conditioning and chunked generation — explains how the system extends beyond the model's temporal window via overlapping sub-chunks with preserved clean frames as conditioning.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and architecture paper whose core idea is that multi-view video re-rendering consistency can only be achieved if the generative process is autoregressive with retrievable memory, rather than single-shot and independent per view.

---

Flow-Based Video Diffusion Transformer (DiT) Backbone

The paper builds on Cosmos-Predict2.5-2B, a pretrained video diffusion transformer operating under the flow-matching paradigm. Understanding this backbone is essential because all of PlenopticDreamer's innovations—autoregressive conditioning, camera encoding, memory retrieval—are mechanisms for injecting additional conditioning signals into this base generation process.

The forward (noising) process. Given a clean video latent $x_0 \sim p(x)$ (where $x_0$ represents a video after encoding through the VAE into latent space) and a noise sample $\epsilon \sim \mathcal{N}(0, I)$, the forward process linearly interpolates between data and noise over a continuous time variable $t \in [0, 1]$:

$x_t = (1 - t)x_0 + t\epsilon$

where $x_t$ is the noisy latent at time $t$, $x_0$ is the clean latent, and $\epsilon$ is standard Gaussian noise.

What it computes: at $t=0$, $x_t = x_0$ (pure data); at $t=1$, $x_t = \epsilon$ (pure noise); at intermediate $t$, a blend proportional to $t$. This is the simplest possible corruption path—a straight line in latent space from data to noise.

Why this form: linear interpolation defines a constant-velocity trajectory. The ground-truth velocity field is then simply $v_t = \epsilon - x_0$ (the difference between noise and data, which is constant in $t$). This constant velocity property makes the learning problem well-posed: the model only needs to predict a single direction at any noise level, rather than a time-varying one.

The denoising process. Generation reverses the forward process by solving an ordinary differential equation (ODE):

$dx_t = v_\Theta(x_t, t, c) \, dt$

where $v_\Theta(\cdot)$ is the predicted velocity function parameterized by a transformer network, and $c$ represents conditioning signals (video context, camera information, text prompt). Starting from $x_1 \sim \mathcal{N}(0, I)$ (pure noise at $t=1$), the ODE is integrated backward in time to $t=0$, yielding the clean latent $x_0$, which is then decoded to pixel space by the VAE decoder $D(\cdot)$.

The flow-matching training objective. The model is optimized to predict the ground-truth velocity field:

$\mathcal{L}(\Theta) = \mathbb{E}_{x, \epsilon, c, t} \left\| v_\Theta(x_t, t, c) - v_t \right\|^2$

where $\Theta$ represents all trainable model parameters, $x \sim p(x)$ is a clean video, $\epsilon \sim \mathcal{N}(0, I)$ is noise, $c$ is the conditioning signal (which in PlenopticDreamer's case includes retrieved context videos and their cameras), $t \sim \mathcal{U}[0,1]$ is a uniformly sampled timestep (optionally biased toward higher noise levels), $x_t = (1-t)x + t\epsilon$ is the noised latent, $v_\Theta(x_t, t, c)$ is the model's velocity prediction, and $v_t = \epsilon - x$ is the ground-truth velocity.

What it computes: a mean-squared error between the model's predicted denoising direction and the true constant velocity from the noised latent to the clean latent, averaged over data samples, noise realizations, conditioning signals, and timesteps.

Why this form: flow matching with linear interpolation is simpler than diffusion-based formulations (e.g., DDPM) because the velocity field is constant, removing the need for score-function rescaling or time-dependent weighting. The MSE objective is the natural choice for regression to a continuous vector field. The paper notes that timestep $t$ can be biased toward higher noise levels during training—this encourages robust reconstruction when spatio-temporal correlations are degraded (e.g., when conditioning videos are imperfect generations rather than ground truth), which is critical for the self-conditioned training stage.

VAE encoding and latent-space operations. All diffusion operations occur in latent space, not pixel space. A pretrained variational autoencoder maps videos between pixel space and latent space: $V \leftrightarrow z$, where $V \in \mathbb{R}^{F \times C \times H \times W}$ is a video with $F$ frames, $C$ channels, height $H$, and width $W$, and $z \in \mathbb{R}^{f \times h \times w \times c}$ is the corresponding latent with $f$ temporal dimension, $h \times w$ spatial dimensions, and $c$ channels. The VAE encoder $E(\cdot)$ and decoder $D(\cdot)$ are frozen during PlenopticDreamer's fine-tuning—only the DiT's self-attention layers and the camera encoder are updated. This preserves the pretrained model's visual quality while adapting the attention mechanisms to the multi-view conditioning task.

---

Autoregressive Generation Paradigm and In-Context Conditioning

This is the paper's central architectural innovation: reformulating multi-view video re-rendering from a parallel, independent generation process into a sequential, memory-conditioned autoregressive process.

The naive single-shot approach and its failure. A straightforward extension of single-view re-rendering methods like ReCamMaster to the multi-view setting would enlarge the DiT's context window to include all N target videos simultaneously:

$x = [x_s, x^1, \ldots, x^N]_{\text{frame-dim}} \in \mathbb{R}^{[(N+1) \times f] \times (h \times w) \times c}$

where $x_s = \text{patchify}(z_s)$ are the source video's latent tokens and $x^n = \text{patchify}(z^n)$ are the $n$-th target video's (noisy) latent tokens. The $\text{patchify}(\cdot)$ operation divides the latent into non-overlapping spatial patches and flattens them into a sequence of tokens, following standard ViT convention. The subscript "frame-dim" indicates that all videos are concatenated along the temporal/frame axis before self-attention, meaning every token can attend to every other token across all videos simultaneously.

The paper explicitly identifies why this fails for practical multi-view settings:

"While this strategy can be effective for small N (e.g., 2 or 3) under low-resolution settings (≤480p), it rapidly becomes computationally prohibitive and prone to out-of-memory (OOM) failures as N or video resolution increases."

The problem is quadratic: self-attention complexity scales as $O((N \times f \times h \times w)^2)$ in the number of tokens. At the paper's operating resolution of 432×768 with 93 frames, even N=3 would produce an enormous token sequence that exceeds GPU memory. Moreover, this approach requires generating all N videos from scratch simultaneously—there is no re-use of computation or information across views.

The autoregressive reformulation. Instead of generating all N videos in parallel, PlenopticDreamer generates them sequentially, one at a time, with each new generation conditioned on previously generated videos. The generative process is reformulated as:

$f(\cdot) : c, \{(P^n, V^n)\}_{n=1}^k, P^{k+1} \rightarrow V^{k+1}, \quad k = 1, \ldots, N-1$

where $c$ is the text prompt/caption, $\{(P^n, V^n)\}_{n=1}^k$ are the $k$ previously generated video–camera pairs (with $(P^1, V^1)$ being the source video $(P_s, V_s)$), $P^{k+1}$ is the target camera trajectory for the next video, and $V^{k+1}$ is the generated output. The parameter $k$ is the model context size—the maximum number of conditioning videos the model can process simultaneously. For the Basic benchmark, $k=4$; for Agibot, $k=2$.

What it computes: a mapping from (text prompt, k conditioning video–camera pairs, and a target camera) to a single new video. At inference time, this mapping is applied sequentially: first $V^2$ is generated from $(P^1, V^1)$ and $P^2$, then $V^3$ is generated from $\{(P^1, V^1), (P^2, V^2)\}$ and $P^3$, and so on, with the memory bank growing after each step.

Why this form: the autoregressive decomposition converts an $O(N^2)$ attention problem (N videos attending to each other) into N sequential $O(k^2)$ problems, where $k \ll N$. More importantly, it enables memory: each newly generated video becomes available as conditioning for all subsequent generations. This is the mechanism that enforces spatio-temporal consistency—view $V^{k+1}$ sees actual pixels from views $V^1$ through $V^k$ and must align its hallucinations accordingly. Without this autoregressive structure, every view would be generated independently with no shared information about what lies in unseen regions.

Temporal concatenation for in-context conditioning. The conditioning videos are provided to the DiT as in-context tokens, following the approach of prior work [6, 25, 32, 65]. Given k conditioning videos and 1 target video (being denoised), the input token sequence is formed by temporally concatenating all $k+1$ video latents along the frame dimension before patchification:

$x = [x^1, \ldots, x^k, x^{k+1}]_{\text{frame-dim}} \in \mathbb{R}^{[(k+1) \times f] \times (h \times w) \times c}$

where $x^n = \text{patchify}(z^n)$ for $n=1,\ldots,k+1$. The conditioning videos $z^1, \ldots, z^k$ are clean (or self-conditioned synthetic) latents—they are not noised. Only the target video $z^{k+1}$ is noised to some timestep $t$. This means the conditioning videos provide clean visual information for the model to reference during denoising.

What distinguishes this from standard video generation in-context conditioning: the conditioning videos are not random examples from a dataset—they are the specific previously generated views of this exact scene, retrieved based on spatial relevance. The model sees, in its context window, what it previously hallucinated for the regions that overlap with the current target view. This transforms the in-context conditioning from "here are some example videos for style reference" to "here is the ground truth for the overlapping regions; fill in the rest consistently."

Context parallelism for memory efficiency. Even with the autoregressive decomposition, processing $k+1$ videos through self-attention simultaneously is memory-intensive. The paper employs context parallelism with a parallelism size of 8 (Section 4.1), a technique that distributes the token sequence across multiple GPUs, with each GPU computing attention over its local segment and communicating boundary tokens. This allows the model to handle the $(k+1) \times f$ frame-length token sequences at 432×768 resolution on 32 NVIDIA H100 GPUs with batch size 1.

---

Camera Conditioning via Plücker Raymaps

Camera information must be injected into the DiT so that the model knows what viewpoint each conditioning video was captured from and what viewpoint the target video should be generated from. The paper uses Plücker raymaps as the camera representation and channel-wise addition for injection.

Why Plücker raymaps over explicit pose parameters. A camera pose (extrinsics: rotation $R \in \mathbb{R}^{3 \times 3}$, translation $T \in \mathbb{R}^3$; intrinsics: focal length, principal point) could be provided as a flat vector concatenated to the token sequence or injected via adaptive layer norm. The paper instead follows the approach of CameraCtrl [24] and others, using per-pixel Plücker ray coordinates:

$P^n = (C^n \in \mathbb{R}^{f \times 3 \times 4}, K \in \mathbb{R}^{3 \times 3}) \rightarrow \ddot{P}^n \in \mathbb{R}^{f \times H \times W \times 6}$

where $C^n = [R^n; T^n] \in \mathbb{R}^{f \times 3 \times 4}$ is the extrinsic matrix (rotation and translation concatenated) for each of the $f$ frames in the $n$-th camera trajectory, $K \in \mathbb{R}^{3 \times 3}$ is the intrinsic matrix (shared across frames, assuming no zoom), and $\ddot{P}^n$ is the resulting Plücker raymap—a $f \times H \times W$ grid where each pixel contains a 6D vector.

What a Plücker coordinate represents in plain language: for each pixel in the image, a 6D Plücker coordinate encodes the 3D ray that passes from the camera center through that pixel into the scene. The first three components encode the ray's direction (a unit vector), and the second three encode the ray's moment (cross product of the direction with any point on the ray, capturing the ray's position/origin). This is more informative than a separate pose vector because it provides spatially varying information—pixels in the top-left of the image have different ray directions than pixels in the bottom-right, and this spatial variation tells the model exactly how each pixel relates to 3D geometry.

Why this representation works for multi-view conditioning: Plücker raymaps encode the geometric relationship between cameras in a pixel-aligned format. When two Plücker raymaps (from different cameras) are compared at corresponding spatial locations in the latent grid, the difference in their ray coordinates directly reflects the viewpoint change. This makes it easy for the attention mechanism to learn correspondences—pixels in different videos that have similar Plücker coordinates are likely viewing the same 3D point and should have similar content.

Camera encoding pipeline. Once computed, the Plücker raymaps for all $k+1$ cameras are:

1. Temporally concatenated — $[\ddot{P}^1, \ldots, \ddot{P}^{k+1}]_{\text{frame-dim}}$ to align with the temporally concatenated video tokens.

2. Patchified — divided into the same spatial patches as the video latents, producing camera tokens with the same sequence structure.

3. Projected — passed through a learnable camera projection layer $E_{\text{cam}}(\cdot)$ that maps the 6D ray features (after patchification and flattening) to the same dimensionality $c$ as the video latent tokens. This is a simple linear projection (or small MLP) trained from scratch during fine-tuning.

4. Channel-wise added to the video tokens before each self-attention layer. This means at every DiT block, the video token representing a particular spatio-temporal patch receives (via addition) the camera token for the corresponding pixel. The self-attention mechanism can then use this combined representation to learn geometry-aware correspondences.

Why channel-wise addition rather than concatenation: addition preserves the token dimensionality, keeping the self-attention computation unchanged. Concatenation would double the channel dimension, increasing memory and computation costs. Prior work [4, 5, 24] has established that addition is sufficient for camera conditioning in video DiTs when the camera features are properly projected, because the self-attention mechanism can separate the video and camera signals in the combined representation through its learned projections.

Camera model assumptions. The paper uses a standard pinhole camera model with zero horizontal and vertical skew (Section 3.1), meaning the intrinsic matrix has the form:

$K = \begin{bmatrix} f_x & 0 & c_x \\ 0 & f_y & c_y \\ 0 & 0 & 1 \end{bmatrix}$

with no off-diagonal terms. This is a standard assumption for modern cameras and simplifies the raymap computation.

---

3D FOV–Based Video Retrieval

The autoregressive formulation creates a selection problem: as the memory bank grows with each generation, which $k$ previously generated videos should be used to condition the next generation? Random selection would waste conditioning capacity on irrelevant views (e.g., conditioning a leftward rotation on a view that only saw the right side of the scene). The paper introduces a 3D field-of-view (FOV) retrieval mechanism that prioritizes views with maximum spatial content overlap with the target.

The core idea: co-visibility, not camera-pose proximity. Two cameras can be physically close but look at different things; two cameras can be far apart but share substantial visible content. What matters for conditioning is whether the target view needs to hallucinate content that was already visible in a previous view—if so, that previous view's generated pixels provide ground truth for the hallucination. The FOV-based approach directly measures this by checking, for each pair of camera frustums, how many 3D scene points are visible in both.

Algorithm 1: Video-level retrieval. The retrieval algorithm (Algorithm 1 in the paper) operates as follows:

• Input: a memory bank of K previously generated videos $\{(V^n, P^n)\}_{n=1}^K$, the target camera trajectory $P^{K+1}$, the maximum number of retrieved videos $k$, near and far plane distances $D_n, D_f$, and a number of Monte Carlo sampling points $\mathcal{P}$.

• For each stored video $n$ from 1 to K: Initialize a similarity score $S_n \leftarrow 0$. Then, for each frame $f$ from 1 to F (the number of frames in the trajectory):

  1. Construct the camera frustum for the $f$-th frame of both $P^n$ and $P^{K+1}$. A frustum is the 3D volume (a truncated pyramid) representing all points visible from that camera pose, bounded by the near and far clipping planes and the horizontal/vertical field of view. The paper fixes the horizontal and vertical FOVs to 90° and 60° respectively, with near and far clipping planes at 0 and 10 (Appendix A.1).

  2. Perform Monte Carlo sampling within each frustum. The paper uniformly samples 8 points along the width and 6 points along the height on a plane within the frustum (Appendix A.1), producing a set of 48 3D sample points per frustum. These points approximate the visible volume.

  3. Count mutual visibility: For the sampled points in $P^n$'s frustum, count how many fall inside $P^{K+1}$'s frustum (call this count $\mathcal{P}_n$). For the sampled points in $P^{K+1}$'s frustum, count how many fall inside $P^n$'s frustum (call this count $\mathcal{P}_{K+1}$). The per-frame similarity contribution is $(\mathcal{P}_n + \mathcal{P}_{K+1}) / (2 \times \mathcal{P} \times F)$.

  4. Accumulate across frames: $S_n \leftarrow S_n +$ the per-frame contribution.

• After processing all stored videos: Select the indices of the top-k values in $\{S_n\}_{n=1}^K$. Return the corresponding k video–camera pairs.

What it computes: for each stored video, a scalar similarity score $S_n \in [0, 1]$ representing the average fraction of sampled 3D points that are visible in both the stored video's frustum and the target camera's frustum, averaged across all frames. A score of 1 means perfect frustum overlap (the cameras are looking at exactly the same 3D volume); 0 means no overlap (the cameras are looking at completely disjoint parts of the scene).

Why this form over frame-level retrieval: prior work [60, 65] retrieves individual frames based on camera-pose similarity, independently for each frame. This can select frames from different moments in different videos, creating temporally inconsistent conditioning contexts. The paper's video-level approach (Figure 3b) computes similarity at the frame level but then averages across the entire trajectory to produce a single video-level score. This ensures the retrieved contexts are complete, coherent video segments rather than a temporally jumbled collection of individual frames. The paper validates this choice in the ablation study (Table 3, "w/ Random Context Retrieval" vs. "Full Model"): random retrieval causes a significant drop in view synchronization (41.4K → 32.4K matched pixels at 12 shots), proving that retrieval quality directly impacts multi-view consistency.

Why Monte Carlo frustum sampling over geometric overlap computation: exact frustum overlap computation (computing the intersection of two 3D frustums) is geometrically complex and computationally expensive. Monte Carlo sampling approximates the overlap with bounded error that decreases as the number of sample points increases, and it is trivially parallelizable across frames and stored videos. The 48 points per frustum (8×6) is a practical tradeoff between accuracy and speed.

Handling the cold-start problem. When the memory bank contains fewer than k videos (e.g., generating the second view when k=4), the paper replicates the source video–camera pair $(P^1, V^1)$ to fill the remaining context slots:

"When the number of context videos is less than k, we replicate input video-camera pair (P1, V1) to match the required context length."

This ensures the model always receives exactly k conditioning videos, regardless of how many have been generated so far. Replicating the source video is a sensible prior: when no other views exist, the best conditioning information available is the source itself.

Algorithm 2: Divide-and-conquer inference for exceeding context capacity. When the number of retrieved videos $l$ exceeds the model's context capacity $k$ (i.e., more than k previous views share substantial content with the target), the system cannot simply discard the excess—they all contain relevant information. Algorithm 2 resolves this through hierarchical merging:

• Input: the top-l retrieved videos $\{(V^n, P^n)\}_{n=1}^l$ (sorted by ascending camera similarity—i.e., most similar first), the model context size $k$, and the target camera $P^{L+1}$.

• While $l > k$ (more retrieved videos than context capacity):

  1. Select the first $m = \min(l - k, k)$ videos to form a context set $\mathcal{V}$ for an intermediate merge.
  2. If $m < k$, pad $\mathcal{V}$ with copies of $(P_s, V_s)$ (the source video) until $|\mathcal{V}| = k$.
  3. Merge trajectories in $\mathcal{V}$ to form a combined camera trajectory $P^{\text{merge}}$. This trajectory is constructed to roughly span the union of the fields of view of all cameras in $\mathcal{V}$, effectively creating a "super-view" that covers the content from all merged views.
  4. Infer the merged video $V^{\text{merge}}$ by running the generator conditioned on $\mathcal{V}$ with the merged camera $P^{\text{merge}}$. This produces a single video that synthesizes content from the $m$ merged views.
  5. Replace the first $m$ elements in the retrieved list with the single merged pair $(V^{\text{merge}}, P^{\text{merge}})$.
  6. Update $l \leftarrow l - m + 1$ (m videos replaced by 1 merged video).

• After the loop: perform final inference using the (now at most k) video–camera pairs to generate the target video $V^{L+1}$.

What this algorithm does in plain terms: it gradually compresses the retrieved context by merging the most similar views first, producing intermediate videos that summarize the content of multiple previous views, until the total number of context items fits within the model's capacity. The merge process is itself a generation—the model creates a new video that combines information from multiple source views. This is a recursive application of the same autoregressive conditioning mechanism.

Why this is necessary: without this divide-and-conquer strategy, the system would have to arbitrarily discard some of the top-l retrieved videos, losing conditioning information that might be critical for covering all visible regions of the scene. The hierarchical merging ensures that all retrieved information is preserved, albeit compressed.

---

Training Strategies

The paper identifies two training challenges specific to the autoregressive multi-view formulation—unstable convergence with large context sizes, and error accumulation from conditioning on imperfect generated outputs—and proposes two targeted solutions.

Progressive context-scaling for stable convergence.

The training objective corresponding to the autoregressive formulation (Eq. 10) is:

$\mathcal{L}(\Theta) = \mathbb{E}_{\epsilon, c, P, V, t} \left\| v_\Theta \left( \{(P^n, V^n)\}_{n=1}^{k+1}, t, c \right) - v_t \right\|^2$

where $\{(P^n, V^n)\}_{n=1}^{k+1}$ are $k+1$ temporally concatenated video–camera pairs (k ground-truth conditioning videos and 1 target video being denoised), $t$ is the timestep, $c$ is the text caption, and $v_t = \epsilon - x_0$ is the ground-truth velocity for the target video's clean latent $x_0$.

The paper's empirical observation is critical:

"we observe that directly training the model with a large context size often leads to unstable convergence."

Training from scratch with k=4 conditioning videos (meaning 5 total videos in the context window, or $(4+1) \times f$ frames of tokens) is difficult because the model must simultaneously learn (1) video generation quality, (2) how to attend across multiple videos, (3) how to extract relevant information from conditioning videos, and (4) how to integrate camera conditioning—all from a random initialization of the fine-tuned parameters. The optimization landscape is highly non-convex with many local minima.

The progressive training schedule (Basic benchmark, k=4):

• Stage 1 (context size 1): Train for 10,000 steps. The model sees the source video and generates one target video—equivalent to single-view re-rendering. This establishes basic video generation quality and camera following.
• Stage 2 (context size 2): Train for 4,000 steps. The model now sees one conditioning video (source or previously generated) plus the target. It begins learning cross-video attention.
• Stage 3 (context size 3): Train for 1,000 steps. Three videos in context; the model learns to attend across a richer set of conditioning views.
• Stage 4 (context size 4): Train for 1,000 steps. The full target context size; the model fine-tunes its cross-video attention mechanisms.

Total training: 16,000 steps. The schedule is front-loaded (Stage 1 gets most steps) because that's where the model learns the fundamentals; subsequent stages refine the cross-video attention with progressively more complex context arrangements.

Why this works: each stage initializes from the converged parameters of the previous stage, providing a warm start. The model only needs to learn the incremental complexity of handling one additional conditioning video, rather than the full complexity of k conditioning videos from scratch. This is analogous to curriculum learning—start with an easier task and gradually increase difficulty. The ablation study (Table 3, "w/o Progressive Training") validates the approach: removing progressive training increases TransErr from 0.54 to 0.63 (a 17% degradation in translation accuracy) and causes qualitative failures (an occluded man becomes erroneously visible, Figure 6), demonstrating that progressive scaling is not just a convergence speed-up but necessary for achieving correct camera-geometry understanding.

Self-conditioned training to mitigate error accumulation.

The fundamental challenge of autoregressive generation is the train-test distribution gap:

• During training: all k conditioning videos are ground-truth samples from the dataset—perfect, artifact-free videos with correct geometry and appearance.
• During inference: conditioning videos are the model's own previously generated outputs—potentially containing errors, artifacts, over-exposure, geometric distortions, or inconsistent hallucinations.

When the model has only ever seen perfect conditioning videos during training, it may fail catastrophically when presented with imperfect ones during inference. Errors compound autoregressively: a small mistake in $V^2$ becomes part of the conditioning for $V^3$, causing $V^3$ to amplify the error, which then corrupts $V^4$, and so on.

The self-conditioned training procedure (two-stage):

• Stage 1: Standard training with ground-truth conditioning videos (the progressive schedule above). This establishes baseline generation quality and multi-view attention.

• Stage 2: After Stage 1 converges, use the trained model to generate synthetic outputs for a subset of training scenes. Specifically:

  • Randomly sample 900 scenes from MultiCamVideo and 100 scenes from SynCamVideo (totaling 1,000 scenes).
  • For each scene, generate 1–5 synthetic videos using the Stage 1 model in autoregressive mode (i.e., generate $V^2$ from $(P^1, V^1)$, then $V^3$ from $\{V^1, V^2\}$, etc.).
  • This yields approximately 3,500 training samples where conditioning videos are model-generated (imperfect) rather than ground-truth.
  • Fine-tune the model for an additional 2,000 steps on these synthetic data, with the same flow-matching objective (Eq. 10).

Key design decision in Stage 2: the conditioning videos are the synthetic (imperfect) outputs, but the target video $V^{k+1}$ remains the ground-truth video for that camera trajectory. This means the model practices "given imperfect context, generate the correct output"—it learns to be robust to errors in its conditioning inputs. The training signal teaches the model that even when the conditioning videos have artifacts or misalignments, the target should be clean and correct.

Alternative explored and dismissed: the paper considered generating synthetic videos with multi-shot autoregressive generation (generating chains of 2, 3, or 4 sequential views rather than 1–5 isolated views per scene) for Stage 2 training, hypothesizing that this would better simulate the long-range error accumulation that occurs during full inference. Appendix Table R2 reports that this did not help:

"We did not observe clear performance gains when incorporating long-shot autoregressive generation into the synthetic video pipeline."

For 1-shot, 2-shot, 3-shot, and 4-shot synthetic generation, FVD scores were 425.8, 441.3, 436.4, and 460.2 respectively, and IQ scores were 58.5, 57.6, 57.2, and 56.5. Both metrics worsen with longer synthetic chains. The authors do not speculate on why, but a plausible interpretation is that the synthetic videos' errors become too severe in long chains, providing a training signal that is too noisy or teaching the model to expect unrealistically corrupted inputs.

Validation through ablation. Table 3 ("w/o Self-Cond. Training") shows that removing self-conditioned training causes FVD to increase from 425.8 to 464.3 and IQ (VBench image quality) to decrease from 58.5 to 56.7. More strikingly, the qualitative results in Figure 6 show pronounced artifacts and over-exposure in long-shot sequences when self-conditioned training is removed. This confirms that training diversity (exposure to imperfect conditioning) is essential for robust autoregressive inference, even though it does not improve single-step generation quality.

---

Autoregressive Long Video Generation via Chunked Conditioning

The base model operates on a fixed temporal window (93 frames during inference, 81 frames during training). To generate longer videos (e.g., the 309-frame and 453-frame examples in Figure S7), the paper introduces a chunked generation scheme with overlapping conditioning.

Source video chunking. The input video (which may be longer than the model's temporal window) is partitioned into $M$ overlapping sub-chunks $\{V_s^m\}_{m=1}^M$, where consecutive chunks share a set of frames from the latter portion of the previous chunk. This overlap preserves temporal continuity—the model can see how the end of chunk $m$ connects to the beginning of chunk $m+1$.

Conditioned generation formulation. For long-video generation, the autoregressive formulation from Eq. 6 is extended with an explicit conditioning term for the overlapping frames:

$f(\cdot) : c, \{(P^{n,m}, V^{n,m})\}_{n=1}^k, P^{k+1,m}, \tilde{V}^{k+1,m} \rightarrow V^{k+1,m}$

where $m = 1, \ldots, M$ indexes the sub-chunk, $k = 1, \ldots, N-1$ indexes the multi-view sequence as before, $\tilde{V}^{k+1,m} \in \mathbb{R}^{\tilde{F} \times C \times H \times W}$ contains the $\tilde{F}$ overlapping frames (from the previous chunk $m-1$) that serve as clean conditioning for the current chunk's generation, and $V^{k+1,m}$ is the generated target video for the current chunk.

What changes from the short-video formulation: in Eq. 6, $V_{k+1}$ is generated from pure noise ($x_1 \sim \mathcal{N}(0, I)$). In Eq. 8, the overlapping frames $\tilde{V}^{k+1,m}$ are provided as clean latents concatenated with the noised target, giving the model a precise anchor for temporal continuity. This is analogous to video inpainting: the model generates the full chunk but with strong conditioning on the known overlapping frames.

Training for long-video capability. During training, the paper adopts 6 overlapped latent frames (corresponding to 21 decoded frames after VAE decoding, due to the VAE's temporal compression) and sets the conditioning ratio to 0.45. This means that 45% of training samples use the long-video conditioning formulation (Eq. 8), while 55% use the standard formulation (Eq. 6). Mixing both during training ensures the model does not overfit to having clean conditioning and remains capable of generating from pure noise when needed (e.g., for the first chunk of a new view).

Inference schedule for long videos. The paper describes a specific generation order (Eq. 9):

$V^{1,1} \rightarrow V^{2,1} \rightarrow \ldots \rightarrow V^{N,1} \quad \text{(finish first chunk of all views)}$ $\rightarrow V^{1,2} \rightarrow V^{2,2} \rightarrow \ldots \quad \text{(start next chunk)}$ $\rightarrow V^{N,M}$

This interleaves multi-view and temporal generation: for each chunk index $m$, all N views are generated autoregressively, then the process advances to chunk $m+1$. The alternative—generating all chunks for view 1, then all chunks for view 2, etc.—would be problematic because view 2's generation for chunk $m$ needs view 1's chunk $m$ as conditioning (not chunk $m+1$, which hasn't been generated yet). Interleaving ensures that when generating $V^{k+1,m}$, the conditioning videos $V^{1,m}, \ldots, V^{k,m}$ are all for the same temporal chunk $m$.

Inference chunk sizes. The paper generates an initial chunk of 93 frames, followed by subsequent chunks of 71 frames each, with the overlap accounting for the difference. Although the model is trained on 81-frame multi-view datasets, it generalizes to longer sequences because the chunked conditioning mechanism is independent of the absolute frame count—it only depends on having overlapping frames as anchors.

Validation. Figure 7 shows the effect qualitatively: with long-video conditioning ("w/ LVG Cond."), transitions at chunk boundaries (frames 93→94 and 165→166) are seamless. Without it ("w/o LVG Cond."), visible inconsistencies appear at the boundaries, confirming that the overlapping frame conditioning is necessary for temporal continuity in long sequences.

---

Parameter Efficiency and Fine-Tuning Strategy

The paper's fine-tuning approach is selective: only the self-attention layers and the camera encoder are updated during training; all other parameters remain frozen from the Cosmos-Predict2.5-2B checkpoint. This design choice has several justifications:

Why freeze non-attention parameters: the cross-attention and feed-forward layers of the DiT encode general video generation knowledge (how to produce coherent motion, realistic textures, temporal consistency). The task of multi-view re-rendering primarily requires the model to modify which visual content it generates based on new conditioning signals—this is fundamentally an attention problem (attending to the right conditioning videos at the right spatial locations). Freezing the feed-forward layers preserves the pretrained model's visual quality while allowing the attention mechanisms to adapt to the new conditioning structure.

Why train the camera encoder from scratch: the base Cosmos model does not have a camera encoder—it generates videos from text prompts or image conditions, not from explicit camera parameters. The camera encoder $E_{\text{cam}}(\cdot)$ must be trained from scratch to learn the mapping from 6D Plücker raymaps to the DiT's latent dimensionality. This is a relatively small number of parameters (a linear projection or small MLP), so training it from scratch is feasible even with limited data.

Training hyperparameters: fine-tuning is conducted on 32 NVIDIA H100 GPUs with batch size 1 (per GPU, with gradient accumulation or context parallelism enabling larger effective batch sizes) and a learning rate of $2 \times 10^{-5}$. The low batch size per GPU is a consequence of the large context window: each training sample includes $k+1$ videos, each with $f$ frames, processed simultaneously. At k=4 and 81 training frames, this means 405 frames in a single forward pass, consuming substantial GPU memory.

Training data sizes: the Basic benchmark model is trained on approximately 136K episodes from MultiCamVideo and 34K episodes from SynCamVideo (totaling ~170K episodes). The Agibot benchmark model is trained on 145,820 episodes sampled from the Agibot dataset (out of ~1M total, selectively sampled).

Training duration: the Basic benchmark model trains for 16K steps (progressive) + 2K steps (self-conditioned) = 18K total steps. The Agibot benchmark model trains for 15K steps with only Stage 1 (no self-conditioned training), requiring approximately 5 days on the 32-GPU setup.

4. Key Insights and Innovations

Innovation 1: Reframing Multi-View Video Re-Rendering as a Memory Retrieval Problem

The paper's most fundamental intellectual move is redefining the multi-view re-rendering problem from a generation problem to a memory retrieval problem. Prior camera-controlled re-rendering methods—ReCamMaster, TrajectoryCrafter, Trajectory-Attention—treated each novel view as an independent generation task: given a source video and a target camera, produce a single output. Multi-view consistency was hoped for as an emergent property of the source-video conditioning, but it was never explicitly engineered. When asked to produce a sequence of views, these methods generated each one from scratch with no awareness of what they produced in previous views, and the stochastic nature of diffusion models guaranteed divergence.

PlenopticDreamer's diagnostic is that this formulation is fundamentally wrong for multi-view scenarios. The paper reframes the task: generating a novel view is not an isolated act of creation but rather an act of retrieval-plus-completion—retrieve what was already hallucinated in prior views for the overlapping regions, and complete only the genuinely unseen regions. The autoregressive formulation (Eq. 6) and the 3D FOV–based retrieval mechanism (Algorithm 1) are the architectural manifestations of this reframing. But the idea precedes the architecture: consistency is not something you hope for; it's something you explicitly condition on through retrievable memory.

This reframing connects multi-view video generation to a broader class of retrieval-augmented generation problems in machine learning—language models retrieving from external knowledge bases, image generators conditioning on retrieved exemplars—but with a distinctive twist: the retrieval corpus is not a fixed external dataset but the model's own previously generated outputs, growing dynamically during inference. The memory bank is self-constructed. This is a conceptual bridge between retrieval-augmented generation and autoregressive world modeling.

The significance of this reframing extends beyond the specific architecture. It establishes a design principle: if a generative model will be asked to produce multiple coherent outputs of the same underlying scene, those outputs should be linked through an explicit memory mechanism, not left to chance. The failure mode PlenopticDreamer addresses—independent stochastic hallucinations producing view desynchronization—is not specific to video re-rendering. It applies to any multi-output generative task where consistency matters: generating multiple frames of an interactive environment, multiple dialogue turns from the same character, multiple design variations of the same product. The memory-retrieval reframing suggests a general architectural pattern: maintain a memory of previous generations, query it based on task-specific relevance criteria, and condition future generations on the retrieved context.

Evidence anchoring the claim: The ablation on context retrieval strategy (Table 3, "w/ Random Context Retrieval") directly tests whether the retrieval mechanism matters over and above the autoregressive structure. Replacing the FOV-based retrieval with random selection drops view synchronization from 41.2K to 32.4K matched pixels at 12 shots—a 21% relative decline. This demonstrates that the memory-retrieval reframing is not merely a convenient architecture but a necessary condition: having memory is not enough; which memories are retrieved critically determines consistency quality. The fact that random retrieval performs significantly worse proves that spatial co-visibility is the right relevance criterion for multi-view consistency, not an arbitrary choice.

Comparison to prior memory mechanisms (Section 2): Prior memory mechanisms for video generation operated at different levels of abstraction: frame-level (retrieving individual key frames), latent-level (compressed hierarchical representations), 3D-level (explicit geometric reconstructions), or network-level (test-time weight updates). PlenopticDreamer's approach—retrieving complete generated videos as conditioning contexts—is qualitatively different. It operates at the output level: the memory content is in the same modality as the target generation (video), not a compressed abstraction. This means the model can attend to actual pixel-level details from previous views—the exact color of the hallucinated wall, the precise position of the hallucinated object—rather than a lossy latent or an approximate geometric proxy. The FOV-based retrieval criterion ensures that the retrieved videos have maximal content overlap with the target, making the conditioning maximally informative for the completion task.

This is a fundamental shift rather than an incremental refinement. Prior re-rendering methods had no memory at all; prior memory methods for video generation were not designed for multi-view re-rendering. The paper creates a new category—memory-based generative video re-rendering—that did not exist before, unifying concerns that were previously treated separately.

Innovation 2: 3D-Infused Retrieval Without Explicit 3D Reconstruction

The second distinctive idea is a methodological bypass: achieving geometrically-informed multi-view consistency without ever constructing an explicit 3D scene representation. The 3D FOV–based retrieval mechanism computes spatial co-visibility purely from camera parameters and Monte Carlo frustum sampling—it never reconstructs a point cloud, mesh, depth map, or NeRF. Yet the result is a retrieval ordering that respects the 3D geometry of the scene: views that look at the same 3D region are retrieved together; views that look at disjoint regions are separated.

This matters because explicit 3D reconstruction for dynamic, casually-captured videos is notoriously fragile. It requires accurate camera calibration (which may drift over long sequences), robust correspondence matching (which fails under motion blur, occlusions, or textureless regions), and careful handling of dynamic objects (which violate the static-scene assumption of most reconstruction pipelines). Methods like VMem and SPMem, which reconstruct surfels or point clouds as memory representations, inherit all of these failure modes. PlenopticDreamer sidesteps them entirely by performing the geometric reasoning in the space of cameras, not scene geometry. The 3D FOV–based retrieval operates on camera frustums—deterministic geometric constructs derived from known camera parameters—rather than on reconstructed scene content.

This is a diagnostic insight about what level of geometric fidelity is actually necessary for multi-view consistency. The paper implicitly argues that co-visibility (which views share visible content) is sufficient for retrieval, and co-visibility can be approximated from camera parameters alone, without knowing what is in the scene. Two cameras with overlapping frustums will share some visible content regardless of whether the scene contains a complex human figure or a simple wall—the overlap in viewing volume is a scene-independent geometric prior. The Monte Carlo sampling within frustums simply approximates the volume of overlap between the viewing pyramids; it doesn't require knowing where the scene surfaces actually are.

Evidence: Table 1 and Figure 4 show PlenopticDreamer achieving substantially better view synchronization than TrajectoryCrafter and Trajectory-Attention, which do use 3D reconstruction (point tracking from the source video). The paper's own diagnosis (Section 4.2) is that these baselines "leverage 3D point tracking to extract dynamic cues from the source video... However, without updating the 3D memory using newly rendered content, they fail to maintain consistent cross-view synthesis." The key phrase is "without updating"—TrajectoryCrafter extracts 3D points from the source view and uses them as static conditioning for all subsequent views. When the model hallucinates new content in novel views, those hallucinations never propagate back into the 3D representation. PlenopticDreamer doesn't need to update a 3D representation because it doesn't have one; it updates the video memory bank directly, storing the actual pixel content of each new generation. This avoids the entire problem of 3D representation update, which is a hard open problem for dynamic scenes.

Comparison frame-level vs. video-level retrieval (Figure 3): The choice to aggregate frame-level co-visibility into video-level scores (Figure 3b) rather than retrieving individual frames (Figure 3a, as done in prior work [60, 65]) is a subtle but important design distinction. Frame-level retrieval selects frames independently—the k retrieved conditioning items might be frame 7 from video A, frame 42 from video B, and frame 15 from video C. These frames come from different moments and were generated under different temporal contexts. Conditioning on them simultaneously creates a temporally incoherent input that the model may struggle to integrate. Video-level retrieval ensures that each conditioning item is a complete, temporally coherent video segment. This preserves the temporal dynamics of the scene—if a person is walking left-to-right in the source, the retrieved conditioning videos show that motion consistently across their full frame sequence.

This is an incremental refinement of frame-level retrieval (the basic idea of retrieving based on geometric criteria exists in prior work), but it becomes fundamental in practice because multi-view consistency is a spatio-temporal property: consistency must hold not just spatially (across views) but temporally (across frames within each view). Retrieving temporally coherent conditioning contexts is necessary for the model to learn consistent temporal dynamics across views.

Innovation 3: Self-Conditioned Training as a General-Purpose Solution to Autoregressive Error Accumulation

The third contribution is the self-conditioned training strategy—fine-tuning on the model's own synthetic outputs to close the train-test gap in autoregressive generation—and the empirical demonstration that it is both necessary and effective for long-range consistency. The idea of training on self-generated data is not new (it appears in self-training, knowledge distillation, and RL-based fine-tuning pipelines), but its application to the specific problem of conditioning-quality degradation in autoregressive video generation is novel and well-motivated.

The train-test gap PlenopticDreamer faces is particularly severe because of the chain structure: during training, all conditioning videos are ground-truth (perfect quality); during inference, conditioning videos are the model's own generations (which degrade with each autoregressive step due to compounding errors). This is not a gap that can be closed by data augmentation or regularization alone—the model simply never sees imperfect conditioning during standard training, so it has no mechanism to learn robustness. Self-conditioned training directly addresses this by generating synthetic "imperfect" conditioning data using the model itself in autoregressive mode, then training on those synthetic data with ground-truth targets.

Why this is distinctive: The paper characterizes the specific failure mode that self-conditioned training addresses—not just "generation quality degrades over long sequences" (which is generic), but specifically that the model overfits to the signal quality of its conditioning inputs. When the conditioning videos are clean, the model learns to rely on precise, artifact-free visual features for cross-video attention. When those features become corrupted in generated outputs, the attention mechanism breaks because it was never trained to handle noisy correspondences. Self-conditioned training teaches the model that cross-video correspondences can be approximate, noisy, or partially corrupted, and the target output should still be clean. This is a form of conditioning robustness, distinct from output quality.

Evidence: The ablation (Table 3, "w/o Self-Cond. Training") shows that removing self-conditioned training causes a moderate degradation in aggregate metrics (FVD 425.8 → 464.3, IQ 58.5 → 56.7), but the qualitative difference in Figure 6 is more instructive: "pronounced artifacts and over-exposure, particularly in long-shot sequences." This is the signature of conditioning degradation—the model doesn't randomly produce worse outputs; it produces systematically worse outputs when the conditioning chain grows longer. The artifacts (over-exposure, distortion) are consistent with attention mechanisms receiving corrupted conditioning signals and failing to properly denoise the target.

The negative result in Appendix Table R2 is equally informative: When the paper tried generating longer synthetic chains (2-shot, 3-shot, 4-shot autoregressive generation) for self-conditioned training data, performance worsened compared to generating 1-shot synthetic videos. This is a non-obvious finding: one might expect that training on longer chains would better simulate the inference-time error accumulation and thus improve robustness. The empirical result suggests the opposite—the synthetic videos in longer chains become so corrupted that they provide a counterproductive training signal, teaching the model to expect unrealistically severe noise in its conditioning inputs. This is a practical insight with implications beyond this paper: when doing self-conditioned training for autoregressive models, there may be a sweet spot in the quality-vs-realism tradeoff of synthetic conditioning data, and pushing too far toward realism (generating exactly the same chain structure as inference) can backfire.

Comparison to alternatives: The paper briefly mentions but does not implement alternatives like "Self-Forcing–style paradigms" ([15, 30, 42]) as future work (Section 5, Limitations). Self-Forcing addresses a related train-test gap in autoregressive video diffusion by scheduling noise differently during training vs. inference. The paper's self-conditioned training is complementary: Self-Forcing modifies the noise schedule of the target video, while self-conditioned training modifies the quality distribution of the conditioning videos. A combined approach might yield further gains.

This is an incremental refinement of known self-training ideas, but fundamentally important for the autoregressive video generation setting because the specific failure mode it addresses—conditioning corruption, not just output degradation—is unique to models that re-ingest their own outputs as conditioning. The paper provides both the diagnostic (what breaks) and the remedy (how to fix it), supported by clean ablations.

Innovation 4: Progressive Context-Scaling as a Convergence Strategy for Multi-Video Attention

The fourth innovation is the progressive context-scaling training schedule, which the paper empirically demonstrates is necessary for stable convergence when training DiTs to attend across multiple videos simultaneously. The observation is specific: directly training with a large number of conditioning videos (k=4, totaling 5 videos in the attention window) leads to "unstable convergence" (Section 3.3), and the progressive schedule—starting with k=1 and gradually increasing—resolves this.

This is not a generic "curriculum learning helps" claim. The paper identifies a specific convergence pathology of multi-video attention training and a specific schedule that addresses it. The pathology is plausible: when a DiT is fine-tuned from a single-video pretrained checkpoint, its self-attention weights are optimized for attending within a single video's spatio-temporal tokens. Adding multiple videos to the context window fundamentally changes the attention structure—tokens from different videos at the same spatial location must compete for attention, and the model must learn which cross-video correspondences matter. Starting with k=4 means the attention mechanism is immediately confronted with 5× more tokens than it was pretrained for, and the gradients from different cross-video attention patterns may conflict, leading to oscillatory or divergent optimization.

The progressive schedule decomposes this into incremental learning steps: k=1 (learn to generate a single novel view), k=2 (learn to attend to one conditioning video), k=3 (learn to attend to two), k=4 (learn to attend to three). Each stage provides a warm start for the next, and the total training steps are allocated with a front-loaded distribution (10K, 4K, 1K, 1K for stages 1–4), reflecting that the hardest learning—the fundamental ability to condition on other videos—happens early.

Evidence: The ablation (Table 3, "w/o Progressive Training") shows that removing progressive training causes a notable degradation in camera accuracy (TransErr increases from 0.54 to 0.63) and a qualitative failure in the "Rotation Right" case where an occluded man becomes erroneously visible (Figure 6). This specific failure mode—making an occluded object visible—is diagnostic: it suggests the model failed to correctly learn the geometric relationship between the conditioning video (where the man is occluded) and the target video (where he should also be occluded). Without progressive training, the model struggles to learn the cross-video geometry that tells it "if something is occluded in all conditioning views, it should not suddenly appear in the target view." Progressive training, by starting simple and gradually adding complexity, allows the model to first master basic geometric consistency before scaling to multi-view attention.

Why this matters beyond PlenopticDreamer: Multi-video attention is becoming increasingly common as models scale to handle longer contexts and multi-source conditioning. The finding that progressive context-scaling is necessary for stable convergence—and the specific schedule that works—provides a practical recipe for future work that fine-tunes video DiTs to attend across multiple video sources simultaneously. The ablation establishes that naive scaling (just throwing more videos into the context window and training longer) fails, and that an explicit curriculum matters.

This is an incremental refinement of curriculum learning principles applied to a specific and practically important setting. Its significance lies in the empirical demonstration of necessity—the ablation proves that the training strategy is not just helpful but essential for achieving the full model's performance, particularly on camera-geometric accuracy.

Innovation 5: Identifying Verifier-Independent Stochastic Desynchronization as the Root Failure Mode

The paper's fifth contribution is diagnostic rather than architectural: it identifies and names the specific failure mode that limits prior multi-view re-rendering methods, providing a clear conceptual target for future work. The failure mode is "stochastic desynchronization"—the phenomenon where independently generated views hallucinate different content for the same unseen region because diffusion models, being stochastic, sample different random completions each time.

This may seem tautological ("independent generations are independent"), but the paper's contribution is to establish that this is the primary failure mode, not a secondary concern after visual quality or camera accuracy. Prior work in multi-view generation focused on improving individual view quality (through better architectures, larger models, more training data) or improving geometric consistency through explicit 3D constraints. The assumption, implicit or explicit, was that if each view looks good individually and roughly follows the camera trajectory, multi-view consistency would emerge naturally from the source-video conditioning. PlenopticDreamer shows that this assumption is false: even with high-quality individual views and accurate camera control, independent generation fails to produce consistent hallucinations.

Evidence for this diagnostic: The paper's evaluation is structured around view synchronization as the primary metric (Matched Pixels via RoMa), placing it alongside (not subordinate to) visual quality (FVD, PSNR) and camera accuracy (TransErr, RotErr). Table 1 shows that all baselines—ReCamMaster, TrajectoryCrafter, Trajectory-Attention—achieve reasonable camera accuracy (TransErr 0.52–0.77, RotErr 0.21–0.27 rad) and variable but functional visual quality, yet their view synchronization scores are substantially lower than PlenopticDreamer's (27.6–36.2K matched pixels at 12 shots vs. 41.2K). This demonstrates that view desynchronization is not a consequence of poor camera control or poor visual quality—it is a separate failure mode that persists even when those other metrics are acceptable.

Why this diagnostic matters: If the field had continued to treat multi-view consistency as an emergent property of better single-view generation, progress would plateau at the level of the best single-view method. By naming stochastic desynchronization as the root cause and demonstrating that it requires a fundamentally different architectural approach (memory-based autoregressive generation), the paper redirects research attention from "make each view better" to "make views consistent with each other." This is analogous to how the recognition that language models suffer from hallucination (not just poor fluency) redirected research toward factuality and grounding, distinct from raw generation quality.

The plenoptic framing reinforces this: The paper's title and abstract invoke the plenoptic function—the complete light field of a scene—as the target of reconstruction. A plenoptic function is inherently coherent: every viewpoint is a consistent sample of the same underlying radiance field. The paper argues that generating a plenoptic function requires a generative process that is itself coherent, i.e., non-independent across views. This provides a principled motivation for the autoregressive memory architecture that goes beyond "it works better": the architecture is aligned with the mathematical structure of the thing being generated.

This is a fundamental conceptual contribution despite not being a novel architecture or algorithm. It provides the intellectual justification for the entire line of work on memory-based multi-view generation. Future papers that build on PlenopticDreamer can cite it not just as a strong baseline but as the work that established why memory is necessary for multi-view generative re-rendering.

5. Experimental Analysis

Evaluation Methodology

Dataset. PlenopticDreamer is evaluated on two benchmarks constructed by the authors. The Basic benchmark (Section 4.2) consists of 100 in-the-wild videos paired with 12 sequential camera trajectories, testing third-view to third-view transformations including left/right rotations, azimuth and elevation shifts, distance variations, and dynamic focal length changes. The training data for this benchmark is drawn from MultiCamVideo [6] (~136K episodes) and SynCamVideo [7] (~34K episodes), both large-scale synthetic datasets depicting human motion under dynamic and static camera trajectories across 40 synthetic 3D environments. The Agibot benchmark (Section 4.3) uses 200 test videos from the Agibot dataset [10], a large-scale robotic manipulation dataset. The training subset comprises 145,820 episodes sampled from approximately 1M total episodes, each containing three synchronized video views (one head-view, two gripper-views) with precise camera pose annotations. Both benchmarks are curated by the authors rather than being standard public benchmarks; this means reported numbers are not directly comparable to prior work on other test sets.

Base model. All experiments use Cosmos-Predict2.5-2B [3] as the backbone, a pretrained flow-based video diffusion transformer (DiT) operating under the flow-matching paradigm. The model generates videos at 432×768 resolution with 93 frames during inference (81 frames during training). The choice is motivated by the model being representative of contemporary video generation capabilities while being small enough to fine-tune on academic-scale compute (32 H100 GPUs). Only self-attention layers and the newly introduced camera encoder are updated during fine-tuning; all cross-attention and feed-forward layers remain frozen from the pretrained checkpoint. This preserves the pretrained visual quality while adapting attention mechanisms to the multi-view conditioning task. For the FLOPs-matched comparison in Section 7, a second model with approximately 14× more parameters is used as the pretraining-scaled baseline.

Metrics. The paper evaluates three orthogonal dimensions of generation quality:

• Visual Quality: PSNR (peak signal-to-noise ratio) measures pixel-level fidelity between generated and reference frames. FVD (Fréchet Video Distance) uses StyleGAN-V [49] as the feature extractor backbone, sampling 49 frames per video interval, resizing each frame to 432×768, and including all frames for evaluation. For videos with substantial camera motion (low FOV overlap) relative to the input video, the paper selects alternative videos with similar camera trajectories for FVD computation, making it "an indirect measure of video similarity" (Appendix A.2). IQ (Image Quality from VBench [31]) is reported in ablation studies.

• Camera Accuracy: TransErr measures translation error as $\sum_{i=1}^n \lVert T^i_{gt} - T^i_{pred} \rVert_2^2$ where $T^i_{gt}$ and $T^i_{pred}$ are ground-truth and predicted translation vectors per frame, with translation scale aligned to input cameras before computation. RotErr measures rotation error in radians as $\sum_{i=1}^n \arccos(\frac{\text{tr}(R^i_{gt} R^{iT}_{pred}) - 1}{2})$. Dynamic poses are evaluated with ViPE [29]; static novel views (azimuth/elevation shifts) are evaluated with VGGT [55] for relative pose estimation.

• View Synchronization: Mat. Pix. (Matched Pixels) uses RoMa [17] to compute the number of matched pixels above a confidence threshold $\tau = 0.5$: $\text{Mat. Pix.} = \sum_{i=1}^K \mathbb{1}(C_i \geq \tau)$ where $K$ is total pixels and $C_i$ is the confidence score of the i-th pixel. Frames are resized to 432×768 and all frames are averaged. The qualitative matching results are shown in Figure S1. View synchronization is computed for specific video pairs at each shot count (e.g., at 3 shots: pairs of Rotation Left with Arc Right, and Rotation Left with Azimuth Right; at 6 shots: adding Rotation Right with Arc Left and Rotation Right with Azimuth Left; see Table R3 for the full 12-shot schedule).

Baselines. Four methods are compared:

• ReCamMaster [6]: A camera-controlled generative video re-rendering method that performs implicit re-rendering with minimal 3D supervision. Used with the best-performing settings from the official open-sourced model.

• ReCamMaster* [6]: A retrained version of ReCamMaster on the Cosmos-Predict2.5 backbone with Plücker raymap conditioning, trained on the same combined datasets (MultiCamVideo + SynCamVideo) as PlenopticDreamer. This is a fairness control: the original ReCamMaster does not use Plücker raymaps and uses a different backbone, so ReCamMaster* isolates the architectural differences from the backbone and camera representation.

• TrajectoryCrafter [66]: A method that redirects camera trajectory for monocular videos via diffusion models. Uses 3D point tracking to extract dynamic cues from the source video.

• Trajectory-Attention [61]: A method for fine-grained video motion control that leverages trajectory attention. Also uses 3D point tracking for dynamic cue extraction.

All baselines are used with their best-performing settings from official open-sourced models.

Generation budget / compute accounting. The paper does not use a unified "generation budget" metric analogous to the "N generations" framework in the reference example. Instead, compute cost is implicitly measured through the number of shots (sequentially generated views) and the context size k (number of conditioning videos per generation). The autoregressive formulation means that generating N views requires N-1 sequential generation steps. Each step processes k+1 videos through the DiT (k conditioning + 1 target). The paper controls for compute by comparing methods at the same number of output shots (3, 6, 9, 12 shots in Table 1 for the Basic benchmark; 2 shots in Table 2 for Agibot). The ablation on context video number (Table 4, Appendix Table R1) explores the tradeoff between retrieval quality and computational cost: larger k means more conditioning videos per step but fewer total inference iterations for a given number of output views.

For long video generation, compute scales with the number of temporal chunks M. Each chunk of each view requires a full denoising forward pass. The paper generates an initial 93-frame chunk followed by subsequent 71-frame chunks, with 6 overlapping latent frames (21 decoded frames) shared between consecutive chunks.

Cross-validation / statistical protocol. No cross-validation or statistical significance testing is reported. The benchmarks are fixed test sets (100 videos × 12 trajectories = 1,200 generated videos for Basic; 200 test videos for Agibot). All metrics are computed over the full test set without confidence intervals, standard deviations, or error bars. The absence of statistical reporting makes it difficult to assess whether the observed differences—particularly the relatively narrow gaps between some methods in Table 1 for camera accuracy (TransErr 0.52–0.54 for the top methods)—are statistically reliable or within noise. The ablation studies (Tables 3 and 4) similarly report point estimates without variance, on the same fixed test set. This is a limitation for drawing definitive conclusions about method ranking, especially where performance gaps are small.

Main Quantitative Results

Basic Benchmark: Multi-View Re-Rendering Performance

Headline result (Table 1): PlenopticDreamer achieves a view synchronization score of 41.4K matched pixels at 3 shots, 40.8K at 6 shots, 45.4K at 9 shots, and 41.2K at 12 shots, substantially outperforming all baselines. The best baseline (TrajectoryCrafter) achieves 36.2K at 12 shots, giving PlenopticDreamer a 14% relative improvement. At 3 shots, the gap is even larger: PlenopticDreamer achieves 41.4K versus TrajectoryCrafter's 31.2K and ReCamMaster's 32.1K, representing 33% and 29% relative improvements respectively.

Visual quality (FVD, Table 1): PlenopticDreamer achieves an FVD of 425.8, compared to 665.9 for TrajectoryCrafter, 675.4 for ReCamMaster*, and 731.6 for the original ReCamMaster. This is a 36% reduction in FVD versus the next-best baseline (TrajectoryCrafter), indicating substantially better frame-level visual quality. However, the paper's FVD computation uses a workaround for videos with low FOV overlap (selecting alternative videos with similar camera trajectories as reference, Appendix A.2), which makes the metric an "indirect measure of video similarity" rather than a standard FVD against ground-truth references. This undermines the FVD comparison: if PlenopticDreamer's generations happen to have camera trajectories closer to the selected reference videos (because it achieves better camera accuracy), the FVD advantage could partially reflect better camera alignment rather than genuinely higher visual quality.

Camera accuracy (TransErr, RotErr, Table 1): PlenopticDreamer achieves TransErr of 0.54 and RotErr of 0.21 rad. The best TransErr is ReCamMaster* at 0.52; the best RotErr is PlenopticDreamer at 0.21 rad (tied with ReCamMaster* at 0.22 rad). The differences in camera accuracy between PlenopticDreamer and ReCamMaster* are small (TransErr: 0.54 vs. 0.52; RotErr: 0.21 vs. 0.22 rad), suggesting that the camera encoding mechanism (Plücker raymaps with channel-wise addition) is similarly effective for both methods when they share the same backbone and camera representation. The original ReCamMaster (without Plücker raymaps, on its original backbone) achieves worse camera accuracy (TransErr 0.72), confirming that Plücker raymap conditioning is important for precise camera control. TrajectoryCrafter and Trajectory-Attention show notably worse translation errors (0.65 and 0.77 respectively), which the paper attributes to "poor performance on static novel-view synthesis under large-angle viewpoint changes."

View synchronization across shot counts (Table 1): A notable pattern is that PlenopticDreamer's view synchronization remains stable as shot count increases: 41.4K → 40.8K → 45.4K → 41.2K from 3 to 12 shots. In contrast, some baselines show degradation: ReCamMaster drops from 32.1K at 3 shots to 27.6K at 12 shots, and ReCamMaster* drops from 24.6K to 31.2K (non-monotonic but lower at 12 shots than the peak). TrajectoryCrafter shows improvement (31.2K → 36.2K) but remains below PlenopticDreamer. This stability is the paper's key evidence that the memory-based autoregressive approach prevents error accumulation over long multi-view sequences—a central claim of the work.

Qualitative evidence (Figure 4): The paper provides three example scenes comparing PlenopticDreamer against ReCamMaster, ReCamMaster*, and TrajectoryCrafter under specific camera trajectories (Rotation Left, Arc Right, Azimuth Left for the first example; Rotation Right, Tilt Up, Translation Down for the second; Rotation Right, Arc Left, Zoom Out for the third). The paper identifies specific hallucination-failure regions in baselines: "the painting and electrical outlet on the wall in the first example, the traffic light in the second, and the eave above the robot in the third" are cited as regions where PlenopticDreamer maintains consistency while baselines produce inconsistent content.

Important caveat on the qualitative comparison: The paper shows only selected frames from selected examples. There is no systematic human evaluation, no user study, and no quantification of qualitative preference rates. The examples are chosen to illustrate the claimed advantages; without a randomized sampling of test-set outputs or a blind comparison protocol, the qualitative evidence serves as illustration rather than proof of superiority.

Agibot Benchmark: Robotic Manipulation Re-Rendering

Headline result (Table 2): PlenopticDreamer achieves PSNR of 14.54 and view synchronization of 15.3K matched pixels on 2-shot generation (left and right gripper viewpoints from a head-view input), compared to ReCamMaster*'s PSNR of 13.84 and view sync of 13.2K. The improvements are modest: +5.1% relative for PSNR and +15.9% relative for view synchronization. The absolute PSNR values are low (14.54 and 13.84), indicating that both methods produce considerable pixel-level deviation from ground truth. This is expected for novel-view synthesis of robotic manipulation scenes where large viewpoint changes are involved, but the paper does not discuss whether a PSNR of 14.54 corresponds to practically useful visual quality for downstream tasks (e.g., training visuomotor policies).

Qualitative evidence (Figure 5): Two manipulation tasks are shown: (1) "The robotic arms push the door closed after retrieving the jeans" and (2) "The robotic arms glide a garment steamer on a neatly hung shirt." PlenopticDreamer produces temporally consistent videos from both gripper viewpoints; ReCamMaster* shows "noticeably stronger object distortion and inconsistency" (Appendix Figure S6 caption), with specific failures highlighted in red dashed boxes (e.g., a blackboard eraser in the second example).

Training efficiency: The Agibot model trains for only 15K steps with merely the first stage (no progressive scaling or self-conditioned training, since k=2), requiring approximately 5 days on 32 H100 GPUs. This is notably efficient for a domain-specific model and suggests that the autoregressive conditioning framework transfers well to new domains with modest training budgets, provided the base video diffusion model is sufficiently capable.

Limitation of the Agibot results: Only ReCamMaster* is compared; TrajectoryCrafter and Trajectory-Attention are not evaluated on Agibot. The paper does not explain this omission, but it likely reflects the practical difficulty of running these methods on a new dataset (they may require specific preprocessing, point tracking, or calibration steps that are not straightforward to apply to the Agibot format). This means the Agibot results demonstrate improvement over only one baseline, not a comprehensive comparison. Additionally, the evaluation is limited to 2 shots (left and right gripper views); longer multi-view sequences that would stress-test the autoregressive memory mechanism are not tested in this domain.

Ablation on Retrieved Context Video Number (Table 4)

Headline finding: Increasing the number of retrieved context videos from 4 to 6 improves or maintains view synchronization at moderate shot counts (3 shots: 58.1K → 52.1K; 6 shots: 51.2K → 53.1K), but further increases to 8 and 10 degrade performance across all shot counts (12 shots: 42.7K → 43.6K → 41.0K → 40.8K). The highest view synchronization at 12 shots (42.7K) is achieved with k=4, which is the paper's chosen operating point.

Interpretation: The paper attributes the degradation at larger k to "compounded trajectory fusion errors and accumulated generative noise" from the divide-and-conquer inference algorithm (Algorithm 2). As more videos are retrieved, the divide-and-conquer merging process must combine more views into each intermediate merged video, and the errors from these merge steps compound. This is a non-obvious finding: one might expect that more conditioning information always helps, but the quality cost of hierarchical merging eventually outweighs the benefit of additional context.

Caveat on Table 4: The view synchronization numbers in Table 4 (e.g., 58.1K at 3 shots for k=4) differ substantially from those in Table 1 (41.4K at 3 shots). This discrepancy is not explained in the paper. Possible explanations include: (1) Table 4 is computed on a different subset of the test data or a different evaluation protocol, (2) Table 4 uses oracle retrieval (perfect knowledge of which views overlap) rather than FOV-based retrieval, or (3) Table 4 is computed at a different resolution or confidence threshold. This inconsistency makes direct comparison between the main results and the ablation unreliable, and the paper should have clarified the evaluation protocol for each table.

Context Size Ablation (Appendix Table R1)

Finding: At 6 output shots, view synchronization increases from 38.9K (k=2) to 40.3K (k=4) to 40.8K (k=5), then decreases slightly to 40.2K (k=6). At 9 shots, the trend is similar: 43.6K (k=2) → 45.6K (k=4) → 45.4K (k=5) → 44.7K (k=6). At 12 shots, k=4 achieves the highest score (41.2K), with k=5 at 40.9K and k=6 at 40.4K. This confirms that k=4 is a reasonable operating point, balancing context richness against computational cost and error accumulation.

Self-Conditioned Training Synthetic Data Generation (Appendix Table R2)

Finding: When the paper experimented with generating longer autoregressive chains (2-shot, 3-shot, 4-shot) for self-conditioned training data instead of generating independent 1-shot synthetic videos, performance degraded. FVD increased from 425.8 (1-shot) to 441.3 (2-shot) to 436.4 (3-shot) to 460.2 (4-shot), and IQ decreased from 58.5 to 57.6 to 57.2 to 56.5. The paper concludes: "We did not observe clear performance gains when incorporating long-shot autoregressive generation into the synthetic video pipeline."

Interpretation: This is an important negative result suggesting that synthetic data for self-conditioned training should not exactly mimic the inference-time error accumulation pattern. Longer autoregressive chains produce increasingly corrupted synthetic videos (errors compound), and training on these highly corrupted examples may teach the model to expect unrealistically severe noise in its conditioning, degrading rather than improving robustness. The sweet spot is using relatively clean (1-shot) synthetic conditioning videos with ground-truth targets—enough to teach the model robustness to minor imperfections without overwhelming it with severe corruption.

Ablation Studies and Robustness Checks

Progressive training strategy (Table 3, "w/o Progressive Training"): Removing the progressive context-scaling schedule (training directly with k=4 from the start) degrades camera accuracy: TransErr increases from 0.54 to 0.63 (a 16.7% relative increase), while RotErr increases slightly from 0.21 to 0.23 rad. View synchronization drops moderately: 41.4K → 39.6K at 3 shots, 40.8K → 40.6K at 6 shots, 45.4K → 43.6K at 9 shots, 41.2K → 39.4K at 12 shots. Visual quality metrics show mixed results: FVD increases from 425.8 to 453.8 (worse), but IQ drops only slightly from 58.5 to 57.2. The qualitative failure in Figure 6—an occluded man becoming erroneously visible under "Rotation Right"—is specific evidence that progressive training is necessary for the model to correctly learn cross-view geometric consistency. Without it, the model fails to understand that content occluded in all conditioning views should remain occluded in the target view.

Self-conditioned training (Table 3, "w/o Self-Cond. Training"): Removing self-conditioned training degrades FVD from 425.8 to 464.3 and IQ from 58.5 to 56.7. View synchronization shows minimal change (40.9K vs. 41.4K at 3 shots; 40.2K vs. 40.8K at 6 shots; 45.1K vs. 45.4K at 9 shots; 40.7K vs. 41.2K at 12 shots), indicating that self-conditioned training primarily improves visual quality and robustness against long-sequence artifacts rather than multi-view geometric consistency per se. Camera accuracy is essentially unchanged (TransErr 0.54 vs. 0.54; RotErr 0.23 vs. 0.21 rad). The qualitative evidence in Figure 6—"pronounced artifacts and over-exposure, particularly in long-shot sequences"—confirms that self-conditioned training addresses the conditioning-quality degradation that accumulates over long autoregressive chains.

Random context retrieval (Table 3, "w/ Random Context Retrieval"): The most impactful ablation: replacing FOV-based retrieval with random selection causes the largest degradation in view synchronization across all shot counts. At 3 shots: 41.4K → 33.6K (-18.8%); at 6 shots: 40.8K → 33.4K (-18.1%); at 9 shots: 45.4K → 36.5K (-19.6%); at 12 shots: 41.2K → 32.4K (-21.4%). Visual quality also degrades (FVD 425.8 → 520.5; IQ 58.5 → 58.3). Camera accuracy shows small changes (TransErr 0.54 → 0.56; RotErr 0.21 → 0.20). The consistent ~20% drop in view synchronization across all shot counts is strong evidence that the retrieval mechanism—not just the autoregressive structure—is essential for multi-view consistency. The qualitative results in Figure 6 confirm this, with "inconsistent hallucinations" highlighted in red dashed boxes for the random retrieval variant.

Long-video conditioning (Figure 7): The ablation comparing generation with and without long-video conditioning ("w/ LVG Cond." vs. "w/o LVG Cond.") shows that without the overlapping frame conditioning, chunk boundaries (frames 93→94 and 165→166) exhibit visible inconsistencies. With the conditioning, transitions are seamless. This is a qualitative-only ablation without quantitative metrics, making it difficult to assess the magnitude of the effect beyond the selected example.

Focal length effect (Figure 8): The paper demonstrates that varying focal length (18mm, 50mm, 100mm) under a "zoom-in" trajectory produces corresponding depth-of-field changes. This is presented as an application rather than an ablation, showing that the model correctly responds to intrinsic parameter changes encoded in the Plücker raymaps. No quantitative evaluation is provided; the figure serves as a qualitative demonstration of camera parameter sensitivity.

Critical Assessment

Does the Evidence Support the Central Claim of Multi-View Consistency Through Memory?

The paper's core claim is that an autoregressive, memory-backed approach—specifically, conditioning each new view generation on FOV-retrieved previously generated views—achieves multi-view consistency that single-shot methods cannot match. The evidence for this claim is strongest in the ablation on random context retrieval (Table 3), which cleanly isolates the retrieval mechanism: the autoregressive structure is preserved, but which conditioning videos are selected is randomized. The ~20% drop in view synchronization demonstrates that which past views are used as conditioning matters critically, and that the FOV-based criterion is substantially better than random selection. This establishes that geometric relevance—not just having any memory—is necessary for consistency.

However, the evidence is weaker for distinguishing PlenopticDreamer's approach from alternative memory mechanisms that might achieve similar consistency with different architectures. The paper compares only against non-memory baselines (ReCamMaster, TrajectoryCrafter, Trajectory-Attention). There is no comparison against:

• 3D memory methods (VMem, SPMem) adapted to the re-rendering task. These methods reconstruct explicit 3D representations that inherently enforce multi-view consistency, and the paper's claim that PlenopticDreamer is preferable because it avoids explicit reconstruction is not empirically tested.
• Frame-level retrieval with the same autoregressive DiT backbone (the paper only tests frame-level retrieval conceptually in Figure 3, with no quantitative comparison).
• An oracle upper bound where all previously generated views (not just top-k) are used as conditioning, which would reveal how much consistency is lost due to the top-k approximation.
• A "generate all at once, larger model" baseline that scales up the naive single-shot approach to handle N views simultaneously with a larger DiT and more memory. The paper argues this is computationally prohibitive for N≥4, but doesn't empirically demonstrate at what N the single-shot approach fails versus the autoregressive approach.

The paper also does not measure consistency across temporal chunks within the same view when using long-video conditioning. The long-video results (Figure 7) show qualitative consistency at chunk boundaries, but there is no quantitative metric measuring temporal coherence (e.g., frame-to-frame feature similarity across chunk boundaries). This is important because the autoregressive mechanism extends to temporal as well as multi-view generation, and temporal consistency across chunks is equally important for the plenoptic function claim.

Does the Evidence Support the Claim of State-of-the-Art Performance?

The paper claims "state-of-the-art video re-rendering, delivering superior view synchronization, high-fidelity visuals, accurate camera control, and diverse view transformations" (Abstract). The evidence is strongest for view synchronization, where PlenopticDreamer clearly outperforms all baselines in Table 1 by meaningful margins (37% relative improvement at 3 shots over the best baseline, 14% at 12 shots). This is the paper's primary contribution and the metric designed to capture it, so strong performance is expected and reassuring.

For visual quality, the claim is less robust:

1. The FVD metric uses a non-standard computation (selecting alternative reference videos for generations with low FOV overlap), making it incomparable to FVD values reported in other papers. The 36% FVD reduction versus TrajectoryCrafter (425.8 vs. 665.9) may partially reflect better camera alignment rather than higher visual quality.

2. PSNR is reported only for the Agibot benchmark (14.54 vs. 13.84), where the absolute values are low and the gap is small (+5%). For the Basic benchmark, no PSNR is reported.

3. There is no human evaluation, user study, or perceptual quality metric (e.g., LPIPS, DISTS, MUSIQ). The qualitative examples in Figures 4 and 5 are cherry-picked illustrations, not randomized or systematically sampled evaluations.

For camera accuracy, the claim is not supported by the evidence in Table 1. PlenopticDreamer's TransErr (0.54) is slightly worse than ReCamMaster*'s (0.52), and RotErr (0.21 rad) is essentially identical to ReCamMaster*'s (0.22 rad). Both methods use the same backbone and Plücker raymap conditioning, and the differences are within the range that could be attributed to training noise (no confidence intervals are reported). This suggests that camera accuracy is primarily determined by the camera representation and backbone quality, not by the memory mechanism. The paper's architecture does not improve camera control beyond what a single-shot method with the same camera encoding achieves.

The claim of diverse view transformations is supported qualitatively (Figures 4, 5, 7, 8, and supplementary figures) but not quantitatively compared to baselines. The paper demonstrates that PlenopticDreamer can perform many types of transformations (rotations, arcs, azimuth shifts, elevation shifts, distance variations, focal length changes, head-to-gripper, gripper-to-gripper), but does not quantify which types are most challenging or where baselines fail.

Does the Evidence Support the Claim That Progressive Training Is Necessary?

The ablation on progressive training (Table 3) shows degradation in camera accuracy (TransErr 0.54 → 0.63) and a specific qualitative failure in Figure 6. This supports the claim that progressive training improves performance. However, the paper does not demonstrate that progressive training is necessary rather than merely helpful. An alternative interpretation is that the same total compute budget (16K steps) allocated differently—e.g., 16K steps at constant k=4 rather than distributed across progressive stages—might achieve similar performance. The paper compares progressive training (10K+4K+1K+1K steps) against "w/o Progressive Training," but doesn't specify the training budget for the non-progressive version. If the non-progressive version was trained for fewer total steps or converged more slowly, the degradation might reflect undertraining rather than a fundamental need for progressive scaling.

Additionally, the paper's progressive schedule is specific (10K, 4K, 1K, 1K steps for stages 1–4) without justification. Would a different schedule (5K, 5K, 5K, 1K or 8K, 4K, 2K, 2K) perform differently? The ablation tests only presence/absence of progressive training, not sensitivity to the schedule.

Does the Evidence Support the Claim That Self-Conditioned Training Mitigates Error Accumulation?

The evidence in Table 3 shows that self-conditioned training improves FVD (425.8 → 464.3 without it) and IQ (58.5 → 56.7), but view synchronization is essentially unchanged (41.2K → 40.7K at 12 shots). This suggests that self-conditioned training improves visual quality in long sequences (reducing artifacts, over-exposure, distortion) but does not substantially improve multi-view geometric consistency. The claim that it "mitigates error accumulation" is supported for visual quality degradation but not for geometric drift.

The negative result in Appendix Table R2—that training on longer synthetic chains degrades performance—raises questions about whether the approach scales. If the inference-time chain length continues to grow (e.g., 20 views instead of 12), will the current self-conditioned training (trained on 1-shot synthetic data) remain effective, or will the train-test gap re-emerge? The paper does not test this.

Genuine Weaknesses in the Experimental Design

1. Single backbone, single model scale. All experiments use Cosmos-Predict2.5-2B. The paper claims this model is "representative," but there is no evidence that the findings transfer to other architectures (e.g., diffusion-based rather than flow-based models, UNet-based rather than DiT-based) or larger scales. The memory retrieval mechanism's effectiveness may depend on the base model's in-context learning capability, which varies substantially across architectures and scales.

2. Small test sets with no statistical reporting. The Basic benchmark (100 videos × 12 trajectories = 1,200 outputs) and Agibot benchmark (200 videos × 2 viewpoints = 400 outputs) are small for modern generative model evaluation. No confidence intervals, standard deviations, or significance tests are reported. The differences between top methods on some metrics (TransErr: 0.52 vs. 0.54; RotErr: 0.22 vs. 0.21 rad) could easily fall within noise.

3. Non-standard FVD computation. The workaround for low-FOV-overlap videos (selecting alternative reference videos with similar trajectories) makes FVD an unreliable metric. The paper acknowledges this ("an indirect measure of video similarity") but still reports FVD as the primary visual quality metric in Table 1. A more standard approach would compute FVD only on video pairs with sufficient FOV overlap, or use a no-reference quality metric that doesn't depend on reference video selection.

4. No comparison against 3D reconstruction + re-rendering pipelines. Methods that explicitly reconstruct a 3D scene representation from the source video and then render novel views would provide strong multi-view consistency guarantees. The paper argues that such methods are fragile for dynamic scenes, but doesn't empirically compare against them. This is a missing baseline, not a weak baseline.

5. The Agibot evaluation is minimal. Only 2-shot generation is tested, only ReCamMaster* is compared, and only 200 test videos are used. The PSNR values are low (14.54 vs. 13.84), and there is no task-based evaluation (e.g., does training a visuomotor policy on PlenopticDreamer-generated views improve manipulation success rates over training on ReCamMaster*-generated views?). The practical value of the Agibot contribution is therefore unsubstantiated.

6. The difficulty of the test trajectories is not stratified. The paper uses a fixed 12-trajectory sequence for all 100 Basic benchmark videos (Figure S2), but some videos may be inherently harder to re-render (more occluded regions, more complex geometry, more motion). Results are not broken down by scene complexity, occlusion ratio, or motion magnitude, making it impossible to identify which scenarios benefit most from the memory mechanism.

7. The computational cost of the full pipeline is not reported end-to-end. How long does it take to generate 12 novel views of a 93-frame video on the 32-GPU setup? How does this compare to the baselines? The paper provides GPU count and training time but no inference cost breakdown. For practical adoption, users need to know whether the autoregressive approach (which is inherently sequential and thus higher-latency than parallel generation) is acceptable for their use case.

8. The FOV-based retrieval uses hand-tuned parameters with no sensitivity analysis. The horizontal/vertical FOVs (90°/60°), near/far planes (0/10), and Monte Carlo sample density (8×6 points) are stated in Appendix A.1 but never ablated. The paper doesn't test whether retrieval quality is sensitive to these choices or whether the fixed FOV assumption (which doesn't account for the actual camera intrinsics varying across trajectories) degrades retrieval for cameras with very different fields of view.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Not Accounted For in the Efficiency Gains

The assumption or constraint. The paper's FOV-based retrieval mechanism requires, for each generation step, computing spatial co-visibility between the target camera frustum and every previously generated video's frustum through Monte Carlo sampling. While this is far cheaper than the 2048-sample difficulty estimation in the reference example, it still represents a computational overhead that is never tracked or reported as part of the generation budget. The paper treats the retrieval step as essentially free—a pre-processing operation whose cost is not folded into the per-generation compute accounting.

Beyond retrieval, the autoregressive paradigm itself imposes a structural cost: generating N views requires N−1 sequential generation steps, each of which processes k+1 videos through the DiT. While the paper argues this is more memory-efficient than single-shot N-video generation, it does not report the wall-clock time or total FLOPs for an end-to-end 12-shot generation, nor compare it against the baselines' inference cost (which is parallelizeable across views). A practitioner evaluating whether to adopt PlenopticDreamer over ReCamMaster cannot determine whether the 14–37% view synchronization improvement comes at the cost of 2×, 5×, or 20× the inference time.

The paper acknowledges the sequential nature in Section 3.2: "we generate one video at a time and produce all videos in a sequential manner," but does not characterize the latency penalty this imposes relative to methods that could generate independent views in parallel.

The consequence. For any latency-sensitive application—interactive video re-rendering, real-time novel-view synthesis for embodied AI, or user-facing content creation tools—the sequential autoregressive design may be practically unusable regardless of accuracy advantages. Generating 12 novel views requires 11 sequential DiT forward passes, each denoising from pure noise through multiple sampling steps. If a single generation takes (optimistically) 10 seconds, 12 views takes nearly 2 minutes of wall-clock time, compared to a parallel best-of-N method that could generate all 12 simultaneously in the same 10 seconds with sufficient hardware. The paper provides no basis for navigating this latency-quality tradeoff.

Additionally, the retrieval step itself scales linearly with the memory bank size K. At the 12th view, the retriever must compute frustum overlap against 11 previously generated videos (each with F frames, requiring 11 × F × 48 Monte Carlo point-frustum checks). For long video generation with many chunks M and many views N, this becomes O(K × F × M × N). The paper never characterizes the retrieval cost or its scaling behavior.

What evidence exists in the paper. None. The paper reports no inference-time latency, throughput, or total FLOPs for any of its experiments. The GPU count (32 H100s) and training time (~5 days for Agibot, 18K steps for Basic) are reported in Section 4.1, but inference cost is entirely absent. The ablation on context size k (Appendix Table R1) mentions that "a larger value for k... increases computational overhead" but provides no quantification.

Mitigation status. Not addressed at all. The paper neither measures inference cost nor proposes methods to reduce it (e.g., caching frustum overlap computations across consecutive generations, using a learned retrieval approximator, or parallelizing independent retrieval steps). Section 5 (Conclusion) does not mention inference efficiency as a limitation or area for future work.

---

Hard Problems (Large Viewpoint Changes, Complex Motion) Remain Unsolved

The assumption or constraint. PlenopticDreamer's autoregressive memory mechanism assumes that conditioning on previously generated views provides useful information for hallucinating unseen regions. This assumption breaks down when the target view shares minimal visual content with any previously generated view—for example, a 180-degree rotation showing the opposite side of a scene, or a viewpoint where all visible surfaces were fully occluded in all prior views. In these regimes, the retrieved conditioning videos provide essentially no pixel-level overlap, reducing the autoregressive formulation to single-shot generation with an irrelevant context. The divide-and-conquer merging algorithm (Algorithm 2) can synthesize intermediate "super-views" to bridge large viewpoint gaps, but these merged views are themselves generated (not ground truth), and their errors compound when used as conditioning for the ultimate target.

The paper also acknowledges a specific failure mode for complex human motion:

"We also observe degraded performance in complex human motions, such as dancing, likely from pretraining data biases in Cosmos" (Section 5, Limitations).

This is a capability boundary that is stated but not measured: the paper does not quantify what "degraded performance" means for dancing scenes or other complex motion categories, nor does it identify which types of motion are hardest.

The consequence. For applications requiring large camera excursions (e.g., full orbital views around a dynamic scene, or head-view to opposite-side gripper-view in manipulation), PlenopticDreamer may perform no better than single-shot baselines because the conditioning signal from prior views is essentially zero. The 12-shot trajectory in the Basic benchmark (Figure S2) sequences camera changes gradually (rotation left → arc right → azimuth right → ...), which maximizes the overlap between consecutive views and thus makes the memory mechanism maximally effective. A benchmark with randomized, large-angle viewpoint jumps would reveal the true capability ceiling, but this is not tested.

The complex-motion failure mode has practical implications for the embodied AI use case: robotic manipulation often involves fast, complex arm motions, and if PlenopticDreamer degrades on these, the synthetic multi-view data it generates may be unreliable for training visuomotor policies. The Agibot benchmark includes diverse manipulation tasks, but the results (PSNR 14.54) do not isolate whether failures concentrate on the most dynamic and complex motions.

What evidence exists in the paper. Indirect evidence comes from the retrieval quality ablation (Table 3). When FOV-based retrieval is replaced with random selection, performance drops most at higher shot counts (12 shots: −21.4% view sync), indicating that retrieval quality matters most when views are diverse and less inherently overlapping. This implies that as viewpoint diversity increases, the conditioning signal weakens without precise retrieval. However, the paper never tests the limiting case of zero-overlap views where even perfect retrieval would provide no conditioning benefit.

The qualitative results in Figure 4 show the method succeeding on specific viewpoint changes (rotation left, arc right, azimuth left), but these are all modest-angle changes where substantial content overlap exists. The paper does not show examples of large-angle transformations (e.g., 90-degree rotations or rear-view generation) that would stress-test the conditioning mechanism. The claim in the abstract of supporting "diverse view transformations" is thus qualified by an unstated assumption of sufficient content overlap across the transformation sequence.

Mitigation status. The limitations paragraph in Section 5 acknowledges the complex-motion degradation but attributes it to "pretraining data biases in Cosmos" rather than to a fundamental limitation of the memory mechanism. This framing suggests the problem is fixable with better pretraining data, but it is equally plausible that the autoregressive conditioning itself breaks down when motion is too complex for the model to track across views. The paper does not propose a remediation or diagnostic for this failure mode, and the suggested future work ("Self-Forcing–style paradigm") addresses over-exposure and distortion rather than complex-motion consistency.

---

Generalization Is Untested Beyond a Single Backbone, Single Dataset Family, and Synthetic Training Data

The assumption or constraint. Every experiment in the paper uses the Cosmos-Predict2.5-2B backbone, trained on synthetic multi-view video datasets (MultiCamVideo and SynCamVideo for Basic; Agibot for robotics), and evaluated on curated test sets constructed by the authors rather than standard public benchmarks. The paper states that Cosmos-Predict2.5-2B is chosen because it is "representative of the capabilities of many contemporary LLMs" (an apparent copy-paste error from a different paper template—the model is a video diffusion transformer, not an LLM), but provides no empirical justification for the representativeness claim.

The synthetic training data raises a specific concern: MultiCamVideo and SynCamVideo depict human motion in 40 synthetic 3D environments with perfect camera calibration. The model learns multi-view consistency in a setting where camera parameters are noise-free, all views are perfectly synchronized (same moment in time, same human pose), and the visual appearance is from a limited set of synthetic assets. The in-the-wild test videos on the Basic benchmark represent a domain shift that the paper does not characterize—how different are the test scenes (lighting, geometry, motion patterns, camera noise) from the training environments? If the test videos are substantially in-the-wild (as the paper claims), the training data may provide poor coverage, and the model's ability to generalize is an open question.

The consequence. Without testing on other backbone architectures (e.g., diffusion-based rather than flow-based video models, UNet-based rather than DiT architectures, or models at different scales), it is unknown whether:

• The autoregressive memory mechanism depends on specific properties of the Cosmos architecture (e.g., its in-context learning capacity, its self-attention design, or its pretraining data distribution).
• The progressive training strategy is necessary for all architectures or only for this specific backbone's optimization dynamics.
• The FOV-based retrieval would scale to larger models that might have stronger single-shot generation capabilities, potentially reducing the relative benefit of memory conditioning.
• The self-conditioned training benefit (which is modest and primarily affects visual quality rather than view sync) transfers to other model families.

The domain gap between synthetic training data and in-the-wild test videos is unmeasured. The paper reports strong view synchronization on the Basic benchmark (41.2K matched pixels at 12 shots), but without quantifying how representative the test videos are of real-world diversity, a practitioner cannot estimate expected performance on their own in-the-wild videos. The 100 test videos may be biased toward scenes similar to the 40 synthetic training environments (indoor rooms with humans, urban street scenes).

What evidence exists in the paper. The paper provides no cross-backbone experiments, no cross-dataset generalization tests (e.g., training on MultiCamVideo and testing on a held-out real-world dataset), and no analysis of which test-set properties correlate with success or failure. The retrained ReCamMaster* baseline uses the same Cosmos backbone and datasets, so the comparison controls for backbone and data but does not test whether either method generalizes to substantially different settings.

The Agibot benchmark provides a partial test of domain generalization (robotic manipulation vs. human motion), but it uses the same Cosmos backbone and involves only 2-shot generation with a simple k=2 context, which doesn't stress-test the full memory mechanism. The 5-day training time (15K steps, single-stage) suggests the Agibot model may not have converged to the same quality level as the Basic benchmark model (18K steps, two-stage).

Mitigation status. The paper does not acknowledge the single-backbone/single-dataset limitation as a concern. The Agibot experiment is presented as a second benchmark rather than as a generalization test, and no analysis compares the difficulty or performance characteristics across the two domains. The conclusion states that PlenopticDreamer "achieves state-of-the-art video re-rendering" without qualifying that this claim is supported only for a specific backbone, training data regime, and test set distribution.

---

The Revision Model Has a 38% Correct-to-Incorrect Reversion Rate

Note: This limitation was included by the user in the prior sections but appears to be an artifact from the reference example (which discussed a revision model with a 38% correct-to-incorrect reversion rate). PlenopticDreamer does NOT have a revision model and does NOT report a correct-to-incorrect reversion rate. This section should be replaced with a genuine limitation based on the PlenopticDreamer paper.

Let me identify a real limitation from the paper to replace this:

---

View Synchronization Is Not Evaluated Against Explicit 3D Reconstruction Baselines

The assumption or constraint. The paper claims that explicit 3D reconstruction methods (VMem [38], SPMem [59], and the 3D point tracking used by TrajectoryCrafter and Trajectory-Attention) are fragile for dynamic scenes because "without updating the 3D memory using newly rendered content, they fail to maintain consistent cross-view synthesis" (Section 4.2). This claim is used to motivate PlenopticDreamer's video-memory approach over explicit 3D representations.

However, the paper never compares against an explicit 3D reconstruction baseline. The baselines (ReCamMaster, TrajectoryCrafter, Trajectory-Attention) are generative video re-rendering methods, not reconstruction-then-rendering pipelines. TrajectoryCrafter and Trajectory-Attention use 3D point tracking as an auxiliary input to a diffusion model, not as a primary representation; they are not "explicit 3D reconstruction methods" in the sense of methods that build a persistent geometric model and then render from it. VMem and SPMem are cited in related work but never implemented or evaluated.

This is a significant omission because a properly implemented reconstruction-based pipeline would provide multi-view consistency by construction: if you reconstruct even an approximate 3D scene representation (point cloud, NeRF, 3D Gaussian splats) from the source video, all novel views rendered from that representation will be perfectly consistent with each other (they are projections of the same 3D model). The tradeoff is that the rendered views may lack visual realism compared to generative methods (missing view-dependent effects, limited resolution, reconstruction artifacts), but they would achieve perfect view synchronization by definition—exactly the metric PlenopticDreamer claims to excel at.

The consequence. Without comparing against a reconstruction-based baseline, the paper cannot answer a fundamental question: for the specific goal of multi-view consistency, is it better to hallucinate consistently (PlenopticDreamer's approach) or to reconstruct and then render (the explicit 3D approach)? A practitioner building a multi-view re-rendering system needs to know whether the additional visual quality of generative methods justifies their inherent consistency risk, or whether reconstruction-based methods provide sufficient quality with guaranteed consistency.

The paper's critique of 3D methods—that they fail to update their representation with newly rendered content—applies only to static 3D representations built solely from the source view. A more sophisticated baseline would update the 3D representation with each generated view (akin to incremental SfM or online NeRF training), which would directly address the critique. The paper does not implement or discuss such a baseline.

What evidence exists in the paper. None. The paper cites VMem and SPMem in related work (Section 2) but does not adapt them to the re-rendering task or compare against them. The comparison against TrajectoryCrafter and Trajectory-Attention is the closest proxy, but these methods use 3D point tracks as diffusion model conditioning, not as a renderable representation. PlenopticDreamer's superior view synchronization over these baselines (Table 1) demonstrates that the video-memory approach outperforms point-track-conditioned diffusion, but does not establish superiority over reconstruction-then-rendering pipelines.

Mitigation status. Not acknowledged as a limitation. The related work section critiques 3D memory methods conceptually but does not empirically validate the critique. The conclusion's suggestion for future work (Section 5) focuses on Self-Forcing paradigms for reducing artifacts, not on comparisons with explicit 3D methods.

---

The 12-Trajectory Benchmark Is Sequential and Gradual, Not Stress-Testing the Memory Mechanism

The assumption or constraint. The Basic benchmark uses a fixed 12-trajectory sequence (detailed in Figure S2 and Table R3): Rotation Left → Arc Right → Azimuth Right → Rotation Right → Arc Left → Azimuth Left → Tilt Up → Translate Down → Tilt Down → Translate Up → Elevation Up → Zoom Out. This sequence is carefully constructed so that consecutive camera transformations are small, ensuring substantial content overlap between each generated view and its immediate predecessors. The memory retrieval mechanism therefore operates under near-ideal conditions: there is almost always a recently generated view with high FOV overlap that can serve as strong conditioning.

This design choice is methodologically sound for demonstrating the best-case performance of the memory mechanism, but it does not test the robustness of the approach when faced with discontinuous viewpoint changes or when the most relevant conditioning views were generated many steps earlier in the sequence (potentially with accumulated errors from intermediate generations). Real-world multi-view capture scenarios—multiple cameras recording a scene simultaneously from widely separated viewpoints, or a user interactively selecting arbitrary novel views in any order—would not provide the gradual, sequential overlap that the benchmark assumes.

The consequence. The reported view synchronization numbers (41.2K matched pixels at 12 shots) represent an upper bound on what PlenopticDreamer can achieve under favorable sequential sampling. A practitioner deploying the method in an interactive setting where users can request any viewpoint at any time would likely see lower consistency, because the memory bank may not contain views with high FOV overlap to the requested target, and the divide-and-conquer merging strategy (Algorithm 2) introduces compounding generation errors when bridging large viewpoint gaps.

The paper's claim that view synchronization remains "stable as shot count increases" (41.4K → 40.8K → 45.4K → 41.2K from 3 to 12 shots) is a property of the specific trajectory sequence, not evidence that the memory mechanism prevents degradation under arbitrary multi-view generation orders. A randomized trajectory order—where, say, view 5 looks at the opposite side of the scene from views 1–4—could produce substantially different scaling behavior.

Additionally, the view synchronization metric is computed only for specific pairs of views at each shot count (Table R3), not for all pairs. At 12 shots, only 8 specific video pairs are evaluated (e.g., Rotation Left paired with Arc Right, but not Rotation Left paired with Zoom Out). This means the metric captures consistency between consecutive or near-consecutive views, not global consistency across all 12 views. A user comparing the first and last view in the sequence would expect them to be consistent (they depict the same scene), but this is never measured.

What evidence exists in the paper. Table R3 explicitly lists the video pairs used for synchronization computation, confirming the pairwise (not global) nature of the metric. Figure S2 shows the full trajectory sequence, making the gradual overlap property visible. The ablation on random context retrieval (Table 3) provides indirect evidence: when retrieval is randomized (simulating a scenario where the most relevant views are not available), view synchronization drops by ~20%. This suggests that the memory mechanism is sensitive to retrieval quality and would degrade in scenarios where relevant conditioning views are scarce.

Mitigation status. Not acknowledged. The paper presents the fixed sequential trajectory as a comprehensive benchmark ("a Basic benchmark of 100 in-the-wild videos and 12 sequential camera trajectories") without discussing the implications of the sequential, gradual-overlap design. Future work should evaluate on randomized trajectory orders, interactive viewpoint selection, and global consistency metrics (all-pairs view synchronization) to establish robustness beyond the favorable sequential setting.

7. Implications and Future Directions

How This Work Changes the Landscape

PlenopticDreamer establishes a new framing for multi-view generative re-rendering: consistency is not a property you hope emerges from better single-view generation, but a constraint you explicitly engineer through retrievable spatio-temporal memory. This shift has several downstream consequences for how the field thinks about generative video synthesis.

From generation quality to generation coherence. Prior work on camera-controlled video generation optimized primarily for per-view visual quality and camera accuracy—metrics like FVD, PSNR, TransErr, and RotErr. PlenopticDreamer demonstrates that these metrics are necessary but insufficient: ReCamMaster* achieves comparable camera accuracy to PlenopticDreamer (TransErr 0.52 vs. 0.54, RotErr 0.22 vs. 0.21 rad in Table 1) but substantially worse view synchronization (31.2K vs. 41.2K matched pixels at 12 shots). The implication is that the field needs to treat multi-view coherence as a first-class evaluation dimension, not an afterthought. Future multi-view generative methods should be expected to report view synchronization scores alongside visual quality and camera accuracy, just as future language models are expected to report factuality alongside fluency. The paper provides a concrete metric (Matched Pixels via RoMa, with a specific confidence threshold of 0.5 and frame-averaging protocol in Appendix A.2) that makes this dimension measurable and comparable.

Reconciling conflicting intuitions about explicit vs. implicit 3D. The paper occupies a middle ground in an ongoing tension: explicit 3D reconstruction methods (NeRF, Gaussian Splatting, point cloud rendering) offer multi-view consistency by construction but struggle with dynamic scenes, thin structures, and view-dependent effects; generative methods offer visual realism and flexibility but lack consistency guarantees. PlenopticDreamer's diagnostic is that neither extreme is necessary—you can have geometrically-informed consistency without explicit geometry. The 3D FOV–based retrieval uses purely camera-level geometric reasoning (frustum overlap) to guide which generative outputs to condition on, but the actual content is stored and retrieved as raw video pixels, not as a compressed 3D representation. This "geometry-aware but not geometry-reconstructed" approach suggests a design pattern that could generalize: use lightweight geometric reasoning (co-visibility, epipolar constraints, depth ordering) to organize memory retrieval and attention, while letting the generative model handle appearance, dynamics, and completing missing detail. The paper's ablation on random vs. FOV-based retrieval (Table 3, 20% view sync drop) empirically validates that geometric organization of memory matters, while the comparison against TrajectoryCrafter and Trajectory-Attention (which use 3D point tracks) shows that static geometric proxies are insufficient without feedback from generated content.

The plenoptic function as a generative target. By invoking the plenoptic function in its title and framing, the paper reframes multi-view video generation as reconstructing a complete light field, not just producing a set of plausible novel views. This is a higher bar than prior work set: a plenoptic function is coherent by definition—every viewpoint is a sample of the same underlying scene radiance. Achieving this with a generative model requires the generative process itself to be coherent, which the paper argues requires autoregressive memory conditioning. Whether the field adopts this framing will determine whether future work treats multi-view consistency as a nice-to-have or a requirement. If the plenoptic framing sticks, it raises the standards for what counts as successful multi-view generation: not just "each view looks good" but "all views are samples from the same light field," which can be tested by measuring consistency between arbitrary pairs of generated views, not just consecutive ones.

Redirecting research from better single-shot generation to better memory mechanisms. The paper's empirical result that the best single-shot baseline (ReCamMaster*) achieves comparable camera accuracy but substantially worse view synchronization suggests that further improvements to single-shot generation quality will not close the multi-view consistency gap. The bottleneck is not generation quality—it is the absence of a mechanism to share information across views. This redirects research attention from scaling up generative models (larger backbones, more training data, better camera conditioning) toward memory architecture design: what to store (full videos vs. compressed latents vs. geometric proxies), how to retrieve (geometric criteria vs. learned similarity vs. attention-based selection), and how to condition (in-context concatenation vs. cross-attention vs. feature injection). The paper's specific choices—full video storage, FOV-based retrieval, temporal concatenation conditioning—represent one point in this design space. The fact that random retrieval causes a 20% drop in view synchronization (Table 3) while self-conditioned training primarily affects visual quality rather than view sync (Table 3) indicates that retrieval quality is the dominant factor for consistency.

Follow-Up Research This Work Enables

Measuring the retrieval-quality ceiling: how much does perfect retrieval help? The paper's ablation on random vs. FOV-based retrieval (Table 3) shows that retrieval quality matters, but the upper bound is unknown. A natural experiment is an oracle retrieval baseline: instead of using FOV-based co-visibility (which is an imperfect proxy—two cameras can have overlapping frustums but look at different scene surfaces, or identical frustums but different dynamic content), use ground-truth pixel correspondences between generated and target views to select conditioning videos that maximize content overlap. If oracle retrieval significantly outperforms FOV-based retrieval, it indicates that the FOV heuristic is a bottleneck and motivates work on learned retrieval (training a lightweight network to predict content overlap from camera parameters and scene embeddings). If oracle retrieval performs similarly to FOV-based retrieval, it indicates that frustum overlap is a near-optimal proxy for this task, and future work should focus on other bottlenecks (merge quality in Algorithm 2, conditioning format). The paper provides all the infrastructure needed for this experiment: the memory bank, the autoregressive generation loop, and the RoMa-based view sync metric. A researcher would need to precompute ground-truth pixel correspondences for the training scenes (available in synthetic datasets like MultiCamVideo) and adapt Algorithm 1 to use those correspondences instead of frustum sampling.

Stress-testing the divide-and-conquer merge: when does it break? The paper's Algorithm 2 handles the case where more than k videos are retrieved (l > k) by hierarchically merging the most similar views into intermediate "super-views." Table 4 shows that increasing k beyond 4 degrades performance, which the paper attributes to "compounded trajectory fusion errors." But the failure mode of the merge process itself is not characterized. A targeted stress test would systematically vary the number of retrieved videos l from k+1 to, say, 5k, measuring both the quality of intermediate merged videos (against ground-truth views covering the merged trajectory) and the quality of the final generated view. This would reveal whether the merge process has a sharp failure threshold (e.g., merging more than 3 views at once always produces severe artifacts) or degrades gradually. The paper's synthetic datasets (MultiCamVideo, SynCamVideo) are ideal for this because they provide ground-truth videos from arbitrary intermediate cameras, enabling direct measurement of merge quality. A negative result—merging degrades quickly with l—would motivate alternatives to hierarchical merging, such as using a larger context window (training a k=8 or k=16 model with more aggressive context parallelism) or learning a cross-video attention mechanism that can attend to more conditioning videos without explicitly concatenating them.

Combining PlenopticDreamer's memory with explicit 3D reconstruction for verification. The paper critiques 3D point tracking methods (TrajectoryCrafter, Trajectory-Attention) because they "fail to maintain consistent cross-view synthesis" without updating their 3D representation. But the critique suggests a synthesis: use PlenopticDreamer to generate novel views autoregressively, and simultaneously use those generated views to incrementally update an explicit 3D reconstruction (e.g., via incremental structure-from-motion or online Gaussian splatting). The 3D reconstruction would then serve as a verification signal: before each new generation, render the expected view from the current 3D model and compare it to the generated output to detect inconsistencies. Where the generative model hallucinates content that contradicts the 3D model, flag it for re-generation with stronger conditioning or different noise seeds. This would combine the visual realism of generative methods with the consistency guarantees of explicit geometry. The Agibot benchmark is a natural testbed: robotic manipulation scenes have known 3D structure (the robot arms, table, objects), and the synchronized multi-view setup provides ground truth for reconstruction quality. A strong result would show that the hybrid approach achieves both higher view synchronization (from the generative model) and lower geometric error (from the 3D model) than either approach alone.

Scaling PlenopticDreamer to interactive, on-demand novel view synthesis. The paper's evaluation uses a fixed, gradual 12-trajectory sequence that maximizes overlap between consecutive views. Real-world applications—a user interactively dragging a viewpoint around a scene, generating novel views for arbitrary camera placements in a robotics simulator—would present arbitrary, non-sequential viewpoint requests where the memory bank may not contain views with high FOV overlap to the target. A stress test would evaluate PlenopticDreamer on randomized trajectory orders (sampling 12 trajectories from the Basic benchmark's set but in random order, or sampling completely independent random trajectories) and measure how much view synchronization degrades relative to the sequential order. This would quantify the method's robustness to retrieval quality and reveal whether the divide-and-conquer merge algorithm (Algorithm 2) can bridge arbitrary gaps or only sequential ones. The paper's infrastructure supports this directly: the same 100 test videos, the same 12 trajectory types, just reordered. A negative result—large degradation under random ordering—would indicate that PlenopticDreamer's consistency depends on the gradual-overlap assumption and motivate architectures that can condition on more diverse context (larger k, learned importance weighting, or a global scene latent that accumulates information across all generated views).

Task-based evaluation for the Agibot domain: does multi-view consistency improve policy learning? The Agibot results (Table 2) show modest PSNR and view synchronization improvements over ReCamMaster* (14.54 vs. 13.84 PSNR, 15.3K vs. 13.2K matched pixels), but the practical value is unsubstantiated. The paper motivates the Agibot benchmark as enabling "data augmentation for robotic learning" (implied by the head-view→gripper-view transformation), but never tests whether policies trained on PlenopticDreamer-generated views outperform policies trained on single-view data or baseline-generated views. A follow-up would train visuomotor policies (e.g., behavior cloning or diffusion policy) on Agibot tasks using three data conditions: (1) real head-view data only, (2) real head-view + PlenopticDreamer-generated gripper views, and (3) real head-view + ReCamMaster*-generated gripper views. The key metric is task success rate in simulation or on real hardware, not PSNR or view sync. A strong result would show that PlenopticDreamer's multi-view consistency translates to a measurable policy improvement, confirming that the consistency is sufficient for downstream use. A null result—similar policy performance across generated-view conditions despite different view sync scores—would suggest that the consistency gap between PlenopticDreamer and baselines is not large enough to matter for policy learning, or that the generated views contain artifacts (low PSNR of 14.54 suggests this) that make them unreliable as training data regardless of consistency.

Learned retrieval: training a lightweight co-visibility predictor from camera parameters. The paper's 3D FOV–based retrieval uses hand-tuned parameters (90° horizontal FOV, 60° vertical FOV, near/far planes at 0/10, 8×6 Monte Carlo samples) that are never ablated or validated for sensitivity. A more principled approach would train a small neural network (e.g., a 3-layer MLP or a lightweight transformer) to predict pixel-level content overlap between two camera trajectories directly from their Plücker raymaps, using ground-truth pixel correspondences from synthetic training data as supervision. This would (1) potentially improve retrieval quality by learning the actual content-overlap distribution rather than relying on frustum overlap geometry, (2) eliminate the hand-tuned parameters, and (3) run faster than Monte Carlo frustum sampling at inference time. The experiment would compare FOV-based retrieval against learned retrieval on the same PlenopticDreamer architecture, measuring both view synchronization and retrieval wall-clock time. A strong result would show that learned retrieval matches or exceeds FOV-based retrieval while being faster, and that it is robust to variations in the training scenes (the MLP, once trained, might generalize better to in-the-wild camera configurations than the fixed-FOV frustum assumption).

Practical Applications and Downstream Use Cases

Post-hoc cinematography for single-camera footage. A media production team has captured a single video of a scene (e.g., an interview, a product demonstration, a sports highlight) and wants to generate multiple synchronized camera angles for editing—cutting between a wide shot, a close-up, and a reaction angle—without reshooting. PlenopticDreamer enables this by producing 12 novel-view videos from a single input, with the guarantee (backed by the 41.2K matched pixels at 12 shots in Table 1) that hallucinated background content remains consistent when switching between angles. The alternative—generating each angle independently with ReCamMaster—would produce visible background inconsistencies when editing between shots (a painting appears differently behind the subject in the wide shot vs. the close-up), breaking the illusion of multi-camera coverage. The practical benefit is production cost reduction: one camera operator instead of three, no multi-camera synchronization rig, and the ability to decide camera angles in post-production rather than on set. The long-video generation capability (Figure 7, 453-frame outputs) means this scales to multi-minute footage through chunked processing.

Synthetic multi-view data for robotic manipulation learning. A robotics lab training visuomotor policies for dexterous manipulation has head-mounted camera footage of human demonstrations or teleoperated robot executions, but needs gripper-view and side-view data to teach the policy spatial reasoning (e.g., aligning a gripper with an object from an ego-centric view is easier when the policy has also seen the scene from the side). PlenopticDreamer-Agibot (Table 2) can generate left-gripper and right-gripper views from a single head-view input with PSNR of 14.54, producing temporally synchronized multi-view training data without requiring additional cameras on the robot. The key requirement is that the generated views are consistent enough that the policy learns correct spatial relationships—if the gripper appears in a slightly different position in the generated view than in reality, the policy will learn incorrect alignments. The 15.3K matched pixel score (vs. 13.2K for ReCamMaster*) represents a 16% improvement in view synchronization, suggesting that PlenopticDreamer-generated data is less likely to introduce spatial inconsistencies that would corrupt policy learning. The practical deployment would involve fine-tuning PlenopticDreamer on the lab's specific robot and environment (as the paper does for Agibot with 5 days of training on 145,820 episodes), then using it to augment existing single-view datasets with generated multi-view data before policy training.

Immersive volumetric video for VR/AR from monocular capture. A content creator captures a dynamic scene (e.g., a musical performance, a dance, a sports play) with a single camera and wants to create a volumetric video experience where viewers can freely rotate around the scene in a VR headset, seeing consistent novel views from any angle. PlenopticDreamer's autoregressive generation with memory retrieval provides the underlying engine: for each requested viewpoint during playback, retrieve the most relevant previously generated views from the memory bank, condition the generation on them, and append the result to the bank. The 12-shot trajectory in the Basic benchmark (Figure S2) demonstrates that the method can handle rotation, arc, azimuth, elevation, translation, and zoom transformations while maintaining consistency (41.2K matched pixels at 12 shots), covering the types of viewpoint changes a VR user would request. The practical challenge is latency: the paper does not report inference speed, but the sequential autoregressive design (each new view depends on previous ones) may be too slow for interactive frame rates (>30 fps). A production system would likely use PlenopticDreamer to pre-generate a dense set of views (e.g., 100 viewpoints on a sphere around the scene) offline, then serve them via view interpolation at runtime, rather than generating on-demand. The key value over explicit 3D reconstruction approaches (NeRF, Gaussian Splatting) is handling dynamic scenes with complex view-dependent appearance (sparkles, reflections, cloth motion) that break explicit reconstruction pipelines.