UniLumos: Fast and Unified Image and Video Relighting with Physics-Plausible Feedback

🎯 Pitch

UniLumos forces a diffusion model to obey real-world geometry by supervising it with depth and normal maps extracted from its own outputs, dramatically reducing physical errors like misaligned shadows and overexposed highlights. It then achieves a 20× speedup over prior methods by combining this feedback with a consistency trick that works even with very few generation steps.

1. Executive Summary

This paper introduces UniLumos, a unified framework for physically plausible image and video relighting that bridges the gap between generative flexibility and physical correctness by incorporating RGB-space geometry feedback into a flow-matching backbone. The system supervises generation with dense geometric signals—specifically, depth and surface normals—estimated from its own outputs, explicitly aligning illumination with scene structure to reduce artifacts like misaligned shadows and overexposed highlights. To make this geometry-aware supervision computationally tractable, UniLumos employs path consistency learning (enforcing consistent velocity predictions under larger integration steps to support few-step denoising regimes) alongside a structured six-dimensional annotation protocol (covering direction, light source type, intensity, color temperature, temporal dynamics, and optical phenomena) that enables fine-grained conditioning and interpretable evaluation. The framework achieves state-of-the-art relighting quality while delivering a 20× speedup over prior methods, establishing that explicit geometric supervision can substantially improve physical consistency without sacrificing inference efficiency—though the hardest problems remain fundamentally limited by the base model's capability range.

2. Context and Motivation

The Core Problem: Diffusion Models Produce Plausible Pixels, Not Physically Correct Lighting

The fundamental problem this paper tackles is deceptively simple: when you ask a generative model to change the lighting in an image or video, why do the shadows often point the wrong way, highlights appear where they shouldn't, and occlusions make no physical sense?

This matters because relighting is not just an aesthetic exercise—it is a core capability that underpins film production, gaming, augmented reality, and any application where synthetic content must blend seamlessly with real footage. A virtual character inserted into a live-action scene needs lighting that matches the environment's direction, intensity, color temperature, and temporal dynamics. If the shadows on the character's face don't align with the shadows cast by real objects in the scene, the illusion breaks immediately, regardless of how photorealistic the individual pixels look.

The paper identifies a specific—and somewhat subtle—failure mode in contemporary diffusion-based relighting methods. As the authors state in Section 1:

"diffusion models typically operate in semantic latent space, where similarity does not guarantee physical correctness in the visual domain"

This is the key diagnostic insight. Current diffusion models are trained to minimize loss in a compressed latent representation—a semantic space where perceptually similar images cluster together. In this space, an image with a bright spot on a forehead and an image with a correctly placed specular highlight may look very similar in terms of latent-space distance. But in the physical world, those two images differ in a way that reveals whether the model actually understands the relationship between light sources, surface orientation, and occluding geometry.

This gap—between semantic plausibility and physical correctness—is what the paper sets out to close. It is analogous to the difference between a language model that produces grammatically correct text and one that produces factually true text. Diffusion models can generate images that look like relit scenes, but they often fail to produce images where the lighting behaves according to the laws of optics.

Why This Gap Exists in Existing Methods

The paper provides a detailed critique of four representative prior methods (Section 1, with expanded discussion in Section 2), and the diagnosis is consistent across all of them:

Lack of explicit geometric supervision in visual space. Every method the paper critiques—IC-Light, SynthLight, Light-A-Video, and RelightVid—operates primarily or exclusively in latent space. IC-Light uses MLP-based embeddings to represent lighting conditions, compressing rich illumination information into a fixed-size vector that conditions the diffusion process. SynthLight employs multi-stage training on synthetic data, but the supervision signals (reconstruction loss, adversarial loss) all operate in latent or pixel space without any explicit constraint that generated lighting respect scene geometry. Light-A-Video is training-free—it iteratively aligns IC-Light's outputs with a pre-trained video diffusion model—meaning there is no training signal at all that could teach the model about physical constraints. RelightVid does joint training with a video diffusion backbone, improving temporal stability over training-free approaches, but still operates "without explicit physical supervision," which the authors identify as the root cause of "inaccurate light-scene interactions and limited generalization to complex or dynamic environments."

The consequence, documented qualitatively in Figure 3 and quantitatively in Table 1, is a pattern of specific, geometrically revealing failures:

Misaligned shadows: Shadows that don't correspond to the occluding geometry in the scene (e.g., a shadow cast to the left when the light source, based on highlight placement, appears to come from the right).
Overexposed highlights: Bright spots on surfaces that, given the estimated surface normal at that point and the implied light direction, should not be receiving direct illumination.
Incorrect lighting directions: Global illumination patterns that suggest a light source direction inconsistent with the conditioning signal (text prompt or reference image).
Temporal inconsistency in video: Frame-to-frame flickering and lighting shifts because there is no geometric anchor tying the lighting in frame t to the lighting in frame t+1—each frame is relit independently or with weak temporal regularization.

These failures are not random noise. They systematically occur in geometrically complex scenes where the relationship between light, surface, and camera is non-trivial—exactly the situations where physical correctness matters most for perceptual realism.

Why This Is a Hard Problem: The Geometry-Lighting Entanglement

Relighting is fundamentally an ill-posed inverse problem. Given only an RGB image (or video) of a scene under lighting $L_1$ , the goal is to produce what that same scene would look like under a different lighting condition $L_2$ . But a single RGB image confounds intrinsic scene properties (geometry, reflectance, material) with extrinsic lighting. A dark pixel could be dark because the surface is darkly colored (high absorption), or because the surface is in shadow (low incident light), or because the surface normal points away from the light source.

The paper's problem formulation (Section 3) makes this explicit:

"Given an image or video $S_1 \in \mathbb{R}^{T \times H \times W \times C}$ with intrinsic scene properties (e.g., geometry, reflectance, content) under initial illumination $L_1$ , the goal of relighting is to modify the illumination within a subject region specified by a binary mask $M \in \{0, 1\}^{T \times H \times W}$ to match a target lighting condition $C$ ."

The key phrase is "preserving the intrinsic attributes of $S_1$ ." The model must disentangle what is intrinsic to the scene (and should be preserved) from what is extrinsic illumination (and should be changed). Without explicit geometric supervision, a model operating purely in latent space has no mechanism for learning this disentanglement. It can memorize correlations—"this pattern of pixel intensities usually goes with that lighting prompt"—but it cannot reason causally about how light interacts with surfaces.

Traditional computer graphics approaches solve this through inverse rendering: explicitly estimate geometry (depth, normals), reflectance (albedo, BRDF parameters), and incident lighting from the input image, then re-render under the target lighting. As the paper notes, these methods "provide physically grounded results" but "typically require complex inputs—such as high dynamic range images or spherical harmonics coefficients—and are limited to constrained domains." They are computationally expensive, require carefully controlled capture conditions, and don't generalize well to in-the-wild inputs where users provide only a single image and a text prompt.

Diffusion models solved the generalization problem at the cost of physical correctness. The paper's ambition is to recover physical correctness without sacrificing the flexibility and ease-of-use that makes diffusion models attractive.

Where Prior Approaches Fall Short in Detail

The paper identifies specific limitations in four representative methods (Section 1, Section 2):

IC-Light (Zhang et al., 2025): This is the most widely adopted baseline and the foundation for several video relighting extensions. IC-Light trains a diffusion model to perform illumination harmonization—adjusting a foreground subject's lighting to match a background scene. Its key architectural choice is representing lighting conditions through MLP-based embeddings derived from background images or text prompts. The limitation: these embeddings compress complex, spatially varying illumination into a global descriptor. There is no mechanism for the model to learn that a light source from the left should cast shadows to the right, or that a surface facing away from the light should receive less illumination. The model can learn statistical correlations ("images with 'sunset' in the prompt tend to have warm, low-angle light") but cannot learn physical laws. This produces results that are semantically plausible (yes, the subject looks warmer) but geometrically incorrect (the warm light illuminates surfaces that should be in shadow).

SynthLight (Chaturvedi et al., 2025): This approach uses synthetic data for multi-stage training, learning to re-render synthetic faces under varying illumination. The strength is in the quality of the training data—synthetic environments provide perfect ground truth for geometry, reflectance, and lighting. The weakness is the domain gap: models trained on synthetic faces struggle to generalize to real-world subjects with diverse geometry, clothing, hair, and materials. Moreover, SynthLight is primarily designed for portrait relighting and relies on face-specific priors (facial geometry is highly constrained and predictable), making it less applicable to general scenes. The paper explicitly positions UniLumos as "a general-purpose relighting framework that is not constrained to any particular object category" (Section 2).

Light-A-Video (Zhou et al., 2025): This is a training-free framework that combines IC-Light with a pre-trained video diffusion model (either CogVideoX or Wan) through iterative alignment. The idea is to use the video diffusion model to enforce temporal consistency across frames that IC-Light relit independently. The approach is clever—it requires no training—but the paper identifies two critical drawbacks. First, the iterative alignment process is computationally expensive, requiring multiple model passes per frame. Second, and more fundamentally, because it is training-free, there is no mechanism for learning physical constraints. The video model can smooth out frame-to-frame discontinuities, but it cannot correct systematic physical errors (like a shadow consistently pointing the wrong direction across all frames) because it has no geometric understanding. The paper reports that Light-A-Video + Wan2.1 achieves R-Motion (temporal consistency) of 1.582 vs. UniLumos's 1.436 (lower is better), but Lumos Score of only 0.682 vs. 0.871—confirming that temporal stability does not imply physical correctness.

RelightVid (Fang et al., 2025): This method adopts joint training with a video diffusion backbone, which improves temporal stability compared to training-free approaches. However, it "still operates without explicit physical supervision," inheriting the same fundamental limitation as IC-Light: the training objective optimizes for perceptual similarity in latent space, not geometric consistency in visual space. The paper positions UniLumos as addressing exactly this gap—providing explicit, geometry-grounded supervision during training while maintaining the architectural flexibility to handle both images and videos.

The Computational Bottleneck: Why Physical Supervision Is Expensive

A critical practical challenge that the paper identifies—and that motivates a key technical contribution—is that physical supervision in RGB space is computationally expensive in standard diffusion training. The proposed physics-plausible feedback requires:

Decoding the model's latent prediction back to RGB pixels (requiring a full pass through the VAE decoder).
Running a pre-trained dense estimation model (Lotus) on those RGB pixels to extract depth and normal maps.
Computing a loss between the estimated and reference geometry.

In a standard multi-step diffusion model, this would need to happen at every denoising step (or at least periodically), dramatically increasing training cost. More importantly, early denoising steps produce noisy, low-quality outputs from which reliable depth and normal estimates cannot be extracted. Physical supervision is only meaningful when the model produces reasonably clean RGB outputs—which typically requires full or near-full denoising.

This is where path consistency learning enters as an enabling technology. By reformulating denoising as a velocity regression task that supports few-step generation, path consistency allows the model to produce high-quality RGB outputs in 1–5 steps instead of 50–100. This makes RGB-space supervision feasible during training because the model can be trained to produce good outputs in few steps, and the physics loss can be computed on those outputs without prohibitive computational overhead.

The paper is explicit about this connection (Section 4.1):

"However, the proposed physics-plausible feedback requires supervision in the RGB domain, which relies on high-quality predictions that are typically only available after full-step denoising has been completed. This poses a major computational bottleneck for standard diffusion models. To mitigate this, we adopt path consistency learning, which reformulates denoising as a velocity regression task, thereby supporting practical training under a few-step regimes."

This tight coupling between the physics feedback mechanism and the efficient training regime is a key architectural insight: the paper doesn't just add a physics loss on top of a standard diffusion model; it restructures the training regime to make physics supervision practical.

The Evaluation Gap: No Metric Captures Lighting-Specific Errors

Beyond model-level limitations, the paper identifies a critical gap in how relighting methods are evaluated (Section 1, Section 4.2):

"Generic generation scores (e.g., FID, LPIPS) fail to capture lighting-specific errors such as shadow misalignment, intensity mismatch, or incorrect light direction."

FID (Fréchet Inception Distance) measures the distributional similarity between generated and real images in a feature space. LPIPS measures perceptual patch-wise similarity. Neither metric has any concept of lighting direction, shadow alignment, or color temperature consistency. A relit image where the subject is illuminated from the left when the prompt specified "back light" could score well on FID and LPIPS—the image looks realistic, it just has the wrong lighting.

The paper argues this creates a perverse incentive: models optimized for standard perceptual metrics can achieve good quantitative scores while producing physically incorrect results. The field lacks benchmarks and metrics that specifically evaluate controllability—did the model actually produce the lighting condition that was requested?

This motivates the paper's second major contribution: LumosData and LumosBench, a structured annotation protocol and benchmark that decompose lighting into six independent dimensions (direction, source type, intensity, color temperature, temporal dynamics, optical phenomena) and evaluate each dimension separately using vision-language models. This enables fine-grained, interpretable assessment of whether a model correctly responds to specific lighting attributes—something no prior benchmark provides.

How This Paper Positions Itself

The paper situates itself at the intersection of three research threads (Section 2):

1. Video Diffusion Models. UniLumos builds on Wan2.1, a state-of-the-art flow-matching video generation model, inheriting its architectural strengths (spatiotemporal attention, efficient VAE, DiT backbone) while extending it for the relighting task. The paper positions flow matching as the right foundation for relighting because its velocity-field formulation naturally supports the path consistency learning that makes physical supervision tractable.

2. Relighting Methods. The paper differentiates itself from prior relighting work along two axes: (a) using explicit geometric supervision during training rather than relying purely on latent-space learning or photometric reconstruction; and (b) being a unified framework for both images and videos, rather than a method specialized for one modality. The paper explicitly notes that many prior methods are portrait-specific (relying on face priors), while UniLumos is designed for general scenes.

3. Feedback Learning in Generative Models. The paper connects to the broader trend of using feedback signals to align generative models—from RLHF in language models to reward-weighted training in image generation (e.g., DRaFT, InstructVideo). Its contribution is the specific form of feedback: physics-plausible geometry feedback that supervises generation with depth and normal consistency, rather than aesthetic quality or human preference. The paper argues that prior feedback learning methods "focus on appearance synthesis and overlook geometry-aware feedback, which typically requires high-fidelity outputs and is incompatible with few-step inference"—and positions path consistency learning as the bridge that makes geometry-aware feedback compatible with efficient training.

The Broader Significance

The paper's motivation extends beyond technical novelty to address a practical tension in the deployment of generative models for visual effects and content creation. As diffusion models have become the dominant paradigm for image and video generation, their ease of use (natural language prompts, reference images) has made sophisticated visual editing accessible to non-experts. But this accessibility comes with a hidden cost: users may not realize that the lighting in the generated output is physically inconsistent until it breaks the illusion in a downstream application (e.g., compositing a relit subject into a live-action plate).

The paper implicitly argues that physical plausibility is not a nice-to-have—it is a requirement for professional applications. In film production, a lighting artist can immediately spot that a shadow doesn't align with a key light. In AR, misaligned lighting breaks the sense that a virtual object exists in the real world. These are not edge cases; they are the primary use cases that relighting technology needs to serve.

By providing a framework that explicitly supervises geometric consistency during training, UniLumos represents a shift from implicit learning of physical constraints (hoping the model will figure out lighting physics from data) to explicit enforcement of geometric priors (telling the model that depth and normals must be consistent with the lighting it produces). This shift—from statistical correlation to structural constraint—is the conceptual contribution that the paper's technical architecture is designed to realize.

3. Technical Approach

3.1 Reader Orientation

UniLumos is a neural network system that takes an image or video of a person (or object) along with a description of desired lighting—either as a text prompt or a reference video—and produces a new version of the input where the subject appears illuminated under those target conditions while preserving their identity, pose, and the background. The core problem it solves is that existing diffusion-based relighting methods produce results that look photorealistic at a glance but violate physical laws—shadows point the wrong way, highlights appear on surfaces that should be dark, and lighting direction is inconsistent with scene geometry—because these models learn correlations in a compressed semantic space rather than learning how light actually interacts with surfaces; UniLumos closes this gap by adding a training-time supervision signal that checks whether the generated output has consistent depth and surface orientation, essentially teaching the model that relighting must respect the 3D shape of the scene.

3.2 Big-Picture Architecture

The UniLumos system has five major components connected in a training pipeline that produces a single unified model capable of both image and video relighting at inference time with few-step generation:

LumosData Pipeline (left side of Figure 2) — a four-stage data construction process that takes real-world videos, extracts subject masks, generates synthetic relit versions under diverse lighting conditions using a pre-trained relighting model, fills backgrounds with Gaussian noise to avoid semantic contamination, and enriches each training sample with structured six-dimensional lighting captions produced by a vision-language model. Output: paired training tuples (V_deg, V_bg, M, C) → V_real.
Wan-VAE Encoder — a frozen variational autoencoder that compresses input frames from pixel space ℝ^(T×H×W×3) into a lower-dimensional latent space ℝ^(T×h×w×c) where the diffusion process operates. All three video streams (degraded/relit subject, background, target real video) are independently encoded through this shared encoder.
Flow-Matching Diffusion Transformer (DiT) Backbone — the core generative model based on Wan2.1, which learns to predict velocity fields that transport noise samples to data samples. It takes as input a noisy latent x_t, conditioning signals (the encoded degraded subject and background latents concatenated channel-wise, plus text embeddings from the structured caption), and a timestep t. The DiT blocks process this combined representation through multiple transformer layers with spatiotemporal attention.
Physics-Plausible Feedback Module — a frozen pre-trained dense estimation model (Lotus) that runs during training only (not at inference). After the DiT backbone predicts a velocity field and the latent is decoded back to RGB pixels, Lotus extracts depth and normal maps. These are compared against reference depth/normal maps from the ground-truth video to compute a geometry consistency loss that backpropagates through the network, teaching it to produce outputs with physically plausible surface geometry.
Path Consistency Learning Module — a training objective that enforces the predicted velocity field to be consistent across different integration step sizes. Instead of standard diffusion training where the model learns to denoise at a single fixed step size, this module requires that a single large step produces the same result as two smaller steps, enabling the model to generate high-quality outputs in 1–5 steps rather than 50–100.

Information flows as follows during a training iteration: (1) The LumosData pipeline produces a tuple of aligned videos with structured captions. (2) All three video streams are VAE-encoded into latents. (3) The target latent x_0 is noised to produce x_t, which is concatenated with the conditioning latents. (4) The DiT backbone predicts a velocity field v_θ(x_t, t, c) that approximates the direction from noise to data. (5) With some probability, the latent prediction is decoded to RGB, depth/normals are estimated, and a physics loss is computed. (6) With some probability, a path consistency loss enforces that the velocity prediction is consistent across step sizes. (7) The standard flow-matching loss, physics loss, and path consistency loss are combined and backpropagated to update the DiT parameters. At inference time, only steps 1–4 run (no physics feedback, no path consistency loss computation—just the forward pass with few-step sampling), making inference fast.

3.3 Roadmap for the Deep Dive

First, the LumosData pipeline and structured illumination annotation protocol, because the training data construction and the six-dimensional lighting taxonomy underpin both the model's controllability and the evaluation benchmark—understanding what the model is conditioned on is prerequisite to understanding what it learns.
Second, the flow-matching backbone and how the relighting task is formulated within that framework, including the input representation (channel-wise concatenation of degraded subject, background, and noisy target latents) and the standard velocity prediction objective, because this is the core generative engine that all other components modify or supervise.
Third, the physics-plausible feedback mechanism in full detail—why depth and normals were chosen, how the Lotus estimator works during training, the exact form of the geometry loss, and the critical design decision to use feedback as training supervision rather than as an inference-time input—because this is the primary technical novelty that distinguishes UniLumos from prior methods.
Fourth, path consistency learning: the mathematical formulation, how it enables few-step generation, and why it is the enabling technology that makes RGB-space physics supervision computationally tractable during training—because without this, the physics feedback would be impractical in a standard multi-step diffusion regime.
Fifth, the joint training objective and the selective optimization strategy (the 80/20 batch split, when each loss is computed, and how training throughput is maintained) because the interaction between the three loss terms and the probabilistic scheduling is a non-obvious design choice that directly impacts both model quality and training efficiency.
Sixth, the model initialization and architectural details (zero-initialization of new layers, Wan2.1-1.3B-480P base, DiT block structure, VAE specifics) because these concrete implementation choices determine what is trainable, what is frozen, and how the model inherits pretrained capabilities.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and training methodology paper whose core idea is that explicit geometric supervision in RGB space, combined with a training regime that makes such supervision computationally feasible (path consistency + selective optimization), can teach a diffusion model to produce physically plausible relighting rather than merely semantically plausible pixel rearrangements.

The LumosData Pipeline and Structured Illumination Annotation Protocol

The training data construction pipeline has four sequential stages, each producing an intermediate representation that feeds into the next. The pipeline is designed to take real-world videos—which have natural, physically correct lighting—and produce paired training examples where the model learns to restore correct lighting from a synthetically degraded version, conditioned on structured semantic descriptions of the target illumination.

Stage 1: Subject Mask Extraction

Given an input video V_real ∈ ℝ^(T+1, H, W, 3), the first stage extracts per-frame binary masks M ∈ {0, 1}^(T+1, H, W) that isolate the foreground subject from the background. The masking is performed by BiRefNet, a pre-trained high-resolution dichotomous image segmentation model.

The choice of BiRefNet is practical: it produces high-quality binary masks without requiring manual annotation. The mask serves two critical downstream purposes: (1) it defines the spatial region where relighting is applied (only the subject, not the background, should have altered illumination), and (2) it enables the physics loss in training to focus only on the subject region when computing depth and normal consistency (as indicated by the element-wise product with M in Equation 6).

The T+1 frames represent the video length plus one (the target frame that the model must predict), reflecting the flow-matching formulation where the model conditions on a sequence and predicts the next frame or the full sequence.

Stage 2: Lumos Augmentation (Synthetic Degradation Generation)

This stage creates the degraded/relit version V_deg that serves as the model's input. The core insight is that the model should learn to restore physically correct lighting from an unnaturally lit version, analogous to how denoising models learn to restore clean images from noisy ones. But instead of adding Gaussian noise, the "degradation" here is applying incorrect or unnatural lighting to the subject while keeping the real background as the target.

The procedure uses a pre-trained 2D image relighting model—specifically IC-Light—applied independently to each frame of the subject region (masked by M from Stage 1). The key design choices:

Multiple lighting prompts for diversity: The paper samples from a curated set of 14 lighting prompts (Table 3), ranging from natural descriptions like "sunshine from window," "sunset over sea," and "golden time" to stylistic ones like "sci-fi RGB glowing, cyberpunk" and "evil, gothic, Yharnam." These prompts come directly from IC-Light's published descriptions of light and shadow.
Multiple light directions: In addition to the semantic prompt, IC-Light can be conditioned on canonical light directions: None (ambient), Left Light, Right Light, Top Light, and Bottom Light. For each input video, the pipeline randomly samples 5 prompts and 3 directions, producing 5 × 3 = 15 unique prompt-direction pairs per video.
Stochastic sampling per iteration: Rather than pre-generating all 15 variants and storing them, the pipeline randomly selects one degradation condition per subject per training iteration. This strategy, as the paper explicitly states, "reduces training cost while exposing the model to diverse illumination patterns, thereby improving generalization." The model never sees the same degradation twice for the same subject in consecutive iterations, effectively providing an infinite stream of lighting variations.

The degraded subject sequence V_deg therefore represents the subject under lighting conditions that are semantically plausible (IC-Light produces visually reasonable results) but not physically consistent with the background or the ground-truth geometry. The model's task is to learn that these lighting conditions are "wrong" and must be corrected to match the target illumination C.

Stage 3: Gaussian Background Construction

This stage generates the background video V_bg that provides spatial context for the relighting. The critical design decision is to not use the real background from the original video—doing so would allow the model to cheat by simply copying lighting cues from the background (e.g., shadow directions, highlight positions) rather than learning to reason about lighting from the structured caption C.

Instead, the background is filled with Gaussian noise parameterized by the per-channel statistics of the original background. For each frame t and each color channel c ∈ {R, G, B}, the pipeline:

First defines the background pixel set Ω_bg^t = {(i, j) | M_t(i, j) = 0} (all pixels where the subject mask is 0). Then computes the mean μ_c^t and standard deviation σ_c^t of the original background pixels in that frame-channel:

$\mu_c^t = \frac{1}{|\Omega_{bg}^t|} \sum_{(i,j) \in \Omega_{bg}^t} V_t(i, j, c)$

$\sigma_c^t = \sqrt{\frac{1}{|\Omega_{bg}^t|} \sum_{(i,j) \in \Omega_{bg}^t} (V_t(i, j, c) - \mu_c^t)^2}$

where V_t(i, j, c) is the original pixel value at position (i, j), channel c, frame t; |Ω_bg^t| is the number of background pixels in that frame; and the sums run over all background positions.

Then fills each background pixel with an independent sample from a Gaussian distribution with those statistics:

$V_{bg}^t(i, j, c) \sim \mathcal{N}(\mu_c^t, (\sigma_c^t)^2), \quad \forall (i, j) \in \Omega_{bg}^t$

What this computes: For each frame and each color channel independently, the mean background color and the standard deviation of background colors are computed from the original video. Then every background pixel in the synthetic background video is replaced with a random sample from a Gaussian distribution centered at that mean with that standard deviation. The result is a video where the background has the same overall color tone and contrast as the original (preserving broad lighting context like "this is a warm indoor scene" or "this is a cool outdoor scene") but contains no structural information—no edges, no textures, no shadow patterns that could leak geometric information about lighting.

Why this form: The paper explicitly compares this Gaussian strategy against a simpler "pure color" baseline (where every background pixel is set to the mean μ_c^t). The Gaussian version "provides more realistic signal distribution and accelerates early-stage convergence during training" (Appendix B). The intuition is that the variance in the background provides a statistical signal about the global illumination level and color temperature without providing any local geometric cues. A pure-color background would be too simple and might cause the model to learn degenerate solutions; the original background would provide too much geometric information and prevent the model from learning to rely on the structured caption C for lighting control.

The paper notes (Appendix B, Figure 7) that this design choice is motivated by avoiding "complex inpainting-based synthesis (e.g., ProPainter, DiffuEraser)" because such methods would inject semantic and structural priors that bias the model toward memorizing background patterns rather than learning illumination physics.

Stage 4: Caption Augmentation (Structured Six-Dimensional Annotation)

This is where LumosData departs most significantly from prior relighting datasets. Instead of using only the original scene-level caption (e.g., "A man in a plaid shirt standing in front of a window"), the pipeline enriches the conditioning signal C with a structured, machine-readable description of six illumination attributes, each classified into predefined categories.

The annotation is performed by a vision-language model—specifically Qwen2.5-VL—using a carefully designed prompt (Listing 1 in Appendix B). The VLM receives the original video and its scene-level caption as input, and is instructed to analyze and classify the lighting along six orthogonal dimensions:

Dimension 1: Direction of Light — Classifies the primary light source direction relative to the subject into: Front Light (source directly in front), Side Light (source at 90° or 45°), Back Light (source behind subject, toward camera), Top Light (source directly above), Bottom Light (source below), Split Light (one side illuminated, other in shadow), or Ambient Light Without Clear Direction (non-directional, uniform illumination).

Dimension 2: Light Source Type — Classifies the type of illumination source into: Natural Light (sun, sky, no human intervention), Artificial Light (bulbs, LEDs, human-made), or Rendering Light (digitally simulated, CGI).

Dimension 3: Light Intensity — Classifies overall brightness into: Glare (>1000 lumens, uncomfortably bright, may obscure detail), Moderate (200–1000 lumens, balanced for typical viewing), or Dim (<200 lumens, low visibility).

Dimension 4: Color Temperature — Classifies the white balance/tint into: Cool Tone (5000K–10000K, bluish, daylight/overcast), Neutral (4000K–5000K, balanced, no strong tint), or Warm Tone (2000K–4000K, reddish/yellowish, sunrise/sunset/indoor).

Dimension 5: Light Changes in Time — Classifies temporal dynamics into: Static Light (constant intensity and direction), Dynamic Light with Intensity Changing (gradual intensity shift, e.g., dawn to daylight), or Dynamic Light with Moving Source (direction changes due to source movement, e.g., headlights, stage lights).

Dimension 6: Optical Phenomena — Classifies light-transport effects into: Transmission (light through glass), Refraction/Reflection at Water Surface or Mirror, Scattering (fog effect, diffusion through particles), or None.

The VLM's output is a JSON-structured string (see Listing 1) that maps each dimension to its classified category. This structured caption is concatenated with the original scene-level caption to form the complete conditioning signal C that is fed to the DiT backbone.

What this enables downstream: The structured annotation serves a dual purpose the paper calls "fine-grained conditioning" and "attribute-aligned benchmarking." During training, the model learns to associate specific illumination attributes in C with specific lighting patterns in the output—if C says "Front Light, Artificial Light, Moderate, Warm Tone, Static Light, None," the model should produce output where the subject is illuminated from the front by a moderate-intensity warm artificial source with no optical effects. At evaluation time, LumosBench uses the same VLM (Qwen2.5-VL) to classify the lighting attributes of the generated output and compare them against the target attributes in C, producing per-dimension accuracy scores. This closes the loop: the same taxonomy used for conditioning is used for evaluation, ensuring that improvement on the benchmark directly corresponds to improved controllability.

Why this protocol instead of free-form text: The paper argues (Section 4.2) that prior methods treat C as "unstructured prompts—such as text, images, or reference frames—offering limited control or interpretability." A free-form prompt like "warm evening light" is semantically rich but ambiguous: does "evening" refer to the color temperature (warm), the direction (low angle), the intensity (dim), or all three? The six-dimensional protocol decomposes these entangled attributes into independent axes, enabling the model to learn that "warm" refers specifically to color temperature while "dim" refers to intensity, and that these can be varied independently (a scene can be warm and bright, or warm and dim). This decomposition is what makes the LumosBench evaluation possible—without independent dimensions, one cannot measure whether the model correctly controlled for intensity while holding direction constant.

Training dataset scale: The pipeline is applied to approximately 110K high-quality video pairs curated from Panda70M, a large-scale video dataset with multiple cross-modality teacher annotations. Additionally, 1.2M relit images are generated using IC-Light and added to the training set. The paper notes this combination "supports robust learning of physically plausible relighting without relying on expensive hardware or manual annotations"—the entire pipeline is automated, requiring no human labeling.

Flow-Matching Backbone and Relighting Problem Formulation

UniLumos builds on Wan2.1, a flow-matching-based video generation model. The choice of flow matching over standard diffusion (DDPM) is not incidental—it connects directly to the path consistency learning that enables efficient physics supervision.

Flow Matching Formulation

Standard diffusion models define a forward process that gradually adds Gaussian noise to data and a reverse process that learns to denoise. Flow matching instead defines a continuous transformation between a noise distribution and a data distribution using a velocity field.

Given a noise sample x_0 ~ N(0, I) (drawn from an isotropic Gaussian) and a data sample x_1 (the latent representation of a real video frame), the linear interpolation path is:

$x_t = t \cdot x_1 + (1 - t) \cdot x_0$

where t ∈ [0, 1] is sampled from a logit-normal distribution (concentrating samples near the endpoints t = 0 and t = 1 where the velocity field is most informative).

The target velocity field is the derivative of this path with respect to t:

$v_t = \frac{dx_t}{dt} = x_1 - x_0$

Unlike diffusion models that predict the noise ε added to x_t, flow matching models predict the velocity v_θ(x_t, t, c) that, when integrated from t = 0 to t = 1, transforms x_0 into x_1. The standard training objective is the mean squared error between predicted and ground-truth velocity:

$\mathcal{L}_0 = \mathbb{E}_{x_0, x_1, t, c} \left[ \| v_\theta(x_t, t, c) - v_t \|_2^2 \right]$

where v_θ(x_t, t, c) is the model's predicted velocity field conditioned on timestep t and context c (the text embeddings from the structured caption), and v_t = x_1 - x_0 is the ground-truth velocity from the linear interpolation path.

What this computes: For each training sample, a random t is drawn, the interpolated point x_t is constructed by linear combination of noise and data, the model predicts what velocity (direction and magnitude of change) would move from x_t to x_1, and the squared L2 difference from the true velocity is the loss. The expectation is over the noise distribution, data distribution, timestep distribution, and conditioning distribution.

Why flow matching for relighting: The paper argues that flow matching's velocity field formulation is more compatible with path consistency learning than standard score-based diffusion. In score-based models, the learned quantity is the gradient of the log-density (the score function), and consistency involves ensuring that different discretizations of the reverse SDE converge to the same distribution—this is mathematically more complex. In flow matching, the learned quantity is the velocity (the deterministic drift), and consistency simply means that integrating the velocity field over a large step should match the result of integrating over two smaller steps. This simpler formulation makes the path consistency loss (Equation 5) straightforward to implement and optimize.

Input Representation and Conditioning

The model's input is a carefully constructed tensor that concatenates three information streams in the latent space:

The noisy target latent x_t — the current state of the variable being denoised/generated. At t = 0, this is pure noise; at t = 1, this should be the latent of the clean relit video. Shape: ℝ^(T/h, H/w, W/w, c) where h and w are the VAE's spatial compression factors.
The degraded subject latent x_deg — the VAE-encoded version of the IC-Light-relit subject video V_deg. This provides the model with information about the subject's appearance under the "wrong" lighting, which it must correct. Shape: same as x_t.
The background latent x_bg — the VAE-encoded version of the Gaussian-background video V_bg. This provides broad color and intensity context without leaking geometric cues. Shape: same as x_t.

These three tensors are concatenated along the channel dimension, producing an input of shape ℝ^(T/h, H/w, W/w, 3c). This concatenated tensor is passed through the DiT blocks, which apply spatiotemporal self-attention and cross-attention to the text embeddings.

Additionally, the timestep t and the desired denoising step size d are appended as temporal condition vectors. The step size d is needed for path consistency learning—the model must know whether it is being asked to predict the velocity for a small step or a large step, because the optimal prediction differs.

Text Conditioning

The structured caption C (the augmented version with six-dimensional lighting attributes) is encoded using umT5, a text encoder that produces embeddings for each token. These embeddings are injected into the DiT blocks via cross-attention, where the query comes from the spatial-temporal features and the keys/values come from the text embeddings. This is standard transformer-based conditioning and allows the model to attend to specific words in the caption (e.g., "Front Light") when generating specific regions of the output.

Model Architecture: Wan2.1-1.3B-480P Details

The paper uses the 1.3 billion parameter version of Wan2.1 designed for 480p video generation. Key architectural specifications:

VAE: A Wan-VAE that compresses input frames spatially (typically 8× or 16× downsampling in height and width) and temporally (typically 4× compression in time). The VAE is pre-trained and frozen during UniLumos training—only the DiT backbone is updated.
DiT Blocks: A transformer architecture with N blocks (the exact number is inherited from Wan2.1), each containing self-attention across spatial-temporal positions, cross-attention to text embeddings, and feed-forward layers. The number of blocks is not explicitly stated but is the standard Wan2.1-1.3B configuration.
Zero-initialization of new layers: Critically, "all new projection and fusion layers are initialized with zero weights to preserve compatibility with the pre-trained Wan initialization and ensure stable optimization from the outset." This means the additional input channels for x_deg and x_bg concatenation start with zero contribution, and the model gradually learns to use them. This is a standard technique for extending pre-trained models to new input modalities—it ensures the model initially behaves identically to the pre-trained baseline and only deviates as training provides gradient signal.

Inference Process

At inference, the model performs few-step sampling using the learned velocity field. Starting from pure noise x_0 ~ N(0, I) and given the conditioning signals (x_deg from the input subject, x_bg from the constructed background, and C from the structured caption), the model predicts velocities at each step and integrates them to produce the clean latent, which is then decoded by the frozen Wan-VAE decoder to RGB pixels. The key advantage: because of path consistency training, this process works well with as few as 1–5 steps instead of the 50+ steps required by standard diffusion samplers.

Physics-Plausible Feedback Mechanism

This is the central technical contribution that distinguishes UniLumos from prior relighting methods. The core idea: during training, decode the model's latent prediction to RGB pixels, run a pre-trained monocular depth and normal estimator on those pixels, and penalize the model if the estimated geometry differs from the geometry of the ground-truth video.

Why Depth and Normal Maps?

The paper provides a detailed justification (Appendix A, Q2–Q3) for choosing depth and surface normals as the supervisory signal over alternatives like albedo, shadow masks, or material properties:

Lighting invariance: By design, monocular depth and normal estimators are trained to suppress the effects of illumination and focus on intrinsic geometry. A well-trained depth estimator should predict the same depth map for a face whether it is lit from the front or the side, whether it is in warm sunset light or cool fluorescent light. This makes depth/normals ideal for supervising relighting—if the model produces a relit image where the estimated depth/normals differ from the reference, the discrepancy must be due to the model's lighting producing erroneous geometric cues, not due to legitimate illumination changes.
Accessibility: Pre-trained monocular depth/normal models (like Lotus) are readily available, work on diverse real-world scenes without domain-specific training, and produce dense per-pixel predictions. In contrast, albedo estimation is an ill-posed problem that typically requires inverse rendering pipelines with strong assumptions; shadow detection is ambiguous (is a dark region a shadow or a dark surface?); and material annotations require expensive manual labeling.
Generality: Depth and normals capture universal scene properties that apply equally to human faces, objects, indoor scenes, and outdoor environments. This aligns with the paper's goal of building a general-purpose relighting framework not constrained to any object category (unlike portrait-specific methods that can rely on face geometry priors).

The Supervision Pipeline (Training Only)

The physics-plausible feedback operates as follows during a training iteration where it is activated (recall from Algorithm 1 that physics feedback is computed for 50% of the 80% of samples used for L_0, i.e., approximately 40% of the total batch):

The DiT backbone predicts a velocity field v_θ(x_t, t, c) from the noisy latent x_t.
From this predicted velocity, an estimate of the clean latent x̂_0 is computed (by integrating the velocity field backward from x_t to t = 0).
The estimated clean latent x̂_0 is decoded through the frozen Wan-VAE decoder to produce an RGB video V̂_real ∈ ℝ^(T+1, H, W, 3).
A frozen pre-trained dense estimation model (Lotus) processes this RGB video to extract:
- Estimated depth map D̂ ∈ ℝ^(T+1, H, W) — per-pixel distance from the camera.
- Estimated normal map N̂ ∈ ℝ^(T+1, H, W, 3) — per-pixel surface orientation as a unit vector (n_x, n_y, n_z).
The same Lotus model processes the ground-truth video V_real to extract reference depth D and reference normals N.
The physics loss is computed as:

$\mathcal{L}_{phy} = \mathbb{E}_{x_0, x_1, t, c} \left[ M \odot \left( \frac{\|\hat{D} - D\|_2}{\|D\|_2} + \frac{\|\hat{N} - N\|_2}{\|N\|_2} \right) \right]$

where \hat{D} \in \mathbb{R}^{T \times H \times W} is the estimated depth map from the model's decoded output, D \in \mathbb{R}^{T \times H \times W} is the reference depth map from the ground-truth video, \hat{N} \in \mathbb{R}^{T \times H \times W \times 3} is the estimated normal map, N \in \mathbb{R}^{T \times H \times W \times 3} is the reference normal map, M \in \{0, 1\}^{T \times H \times W} is the binary subject mask, \|\cdot\|_2 denotes the L2 norm (Euclidean), and \odot is element-wise (Hadamard) product.

What this computes operationally: For each frame and each pixel within the subject mask, the absolute L2 error between the estimated and reference depth is computed and divided by the L2 norm of the reference depth (making it a relative error—a 10cm error matters more when the object is 1m away than when it is 10m away). The same relative error is computed for the normal vectors. These per-pixel relative errors are summed across the spatial dimensions, and the result is averaged over the batch, timesteps, and conditioning samples. The mask M ensures that only the subject region contributes to the loss—errors in the background (which is synthetic Gaussian noise anyway) are ignored.

Why relative L2 error: The normalization by \|D\|_2 and \|N\|_2 makes the loss scale-invariant. Without normalization, objects far from the camera (large depth values) would dominate the loss even if their relative geometry is correct. With normalization, the loss penalizes proportional errors equally regardless of absolute distance. For normals, the normalization turns the error into a measure of angular deviation: \|\hat{N} - N\|_2 / \|N\|_2 approximates the sine of the angle between estimated and reference normals when both are roughly unit vectors.

Why this form over photometric loss: A natural alternative would be to penalize the model directly on RGB pixel differences between the generated and reference videos—i.e., a standard reconstruction loss like L1 or L2 in pixel space. The paper implicitly argues against this in Appendix A (Q1): such a loss would penalize any deviation, including legitimate illumination changes that are the whole point of relighting. If the target lighting is "warm sunset" and the reference has "cool daylight," a pixel-space loss would force the model to not change the lighting at all. The depth/normal loss avoids this because depth and normals are (ideally) invariant to illumination—only errors in geometry are penalized, not legitimate color or intensity changes.

Why feedback during training but not inference: This is a critical design decision discussed in Appendix A (Q4). Alternatives considered: (1) conditioning the model directly on estimated depth/normal maps as additional input channels, and (2) running iterative optimization at inference to enforce geometry consistency. Both are rejected:

Conditioning on geometry as input would require running the depth/normal estimator at inference time, increasing computational cost and introducing a dependency on an external model. It would also increase input dimensionality and model complexity.
Iterative optimization at inference (e.g., using the geometry loss to guide the sampling process, analogous to classifier guidance) would require multiple forward-backward passes through the model and the depth estimator, dramatically increasing inference time.

By using geometry as a training-only supervision signal, the model internalizes the geometric constraints into its weights. At inference, it produces geometrically consistent outputs without any explicit geometric reasoning—the consistency is baked into the learned velocity field. This is an instance of the general principle that training-time constraints can be more efficient than inference-time constraints if the model can learn to satisfy them implicitly.

The Lotus Dense Estimator

Lotus is a diffusion-based visual foundation model for dense prediction tasks, pre-trained to produce high-quality depth and normal maps from single RGB images. The paper uses Lotus in inference mode with frozen weights—no fine-tuning, no adaptation. Key properties that make it suitable:

Diffusion-based: Lotus itself uses a diffusion process to refine its predictions, meaning it produces smooth, globally consistent estimates rather than noisy per-pixel regressions. This is important because the physics loss gradients flow through Lotus's predictions—if Lotus produced noisy estimates, the gradient signal would be unreliable.
Monocular: Lotus operates on single frames independently (or with minimal temporal context), so it doesn't require multi-view stereo or video-specific processing. This keeps the feedback pipeline simple.
General-purpose: Lotus is trained on diverse datasets spanning indoor, outdoor, synthetic, and real scenes, making it applicable to the varied content in LumosData without domain-specific fine-tuning.

What happens to the gradients: During backpropagation, the gradient of L_phy with respect to the model parameters flows through: the Wan-VAE decoder → the decoded RGB pixels → the Lotus depth/normal estimator → the L2 error computation. The Lotus model itself is frozen—its parameters are not updated—but the gradients flow through its computation graph to reach the VAE decoder and the DiT backbone. This is analogous to how perceptual losses (e.g., LPIPS) work: a frozen network (like VGG or AlexNet) processes the generated and reference images, and the loss is computed in that network's feature space, with gradients flowing back through the frozen network to the generator.

The paper does not explicitly discuss a potential challenge: Lotus, being itself a diffusion model, has stochastic sampling that could introduce variance in the gradient estimates. The likely mitigation (though not stated) is that Lotus is run with a fixed random seed or with deterministic sampling at inference time, producing consistent estimates for the same input.

Path Consistency Learning

Path consistency learning is the enabling technology that makes RGB-space physics supervision practical. Without it, the model would require 50–100 denoising steps to produce clean outputs, making it computationally prohibitive to decode to RGB and run Lotus at every (or even every few) training iterations.

The Core Idea

Standard diffusion/flow-matching training teaches the model to predict the velocity (or noise) at a single, randomly sampled timestep t. At inference, the model must be applied iteratively: start at t = 0 (pure noise), predict the velocity, take a small step, then predict again at the new t, and repeat dozens of times. The model is trained to be accurate at each individual step, but errors can accumulate across the chain of steps.

Path consistency learning reformulates the objective: instead of (or in addition to) learning the velocity at arbitrary timesteps, the model learns that its velocity predictions must be self-consistent across different step sizes. If you take one large step of size 2d, you should arrive at the same point as if you take two smaller steps of size d each.

Mathematical Formulation

Given a velocity field v_θ and a step size d > 0, a forward Euler integration step is defined recursively:

$x_{t+d} = x_t + d \cdot v_\theta(x_t, t, d)$

Note that the velocity prediction v_θ now takes three arguments: the current state x_t, the current timestep t, and the desired step size d. This is crucial—the model must condition its prediction on how large a step is being taken, because the optimal velocity for a small step (local linearization) differs from the optimal velocity for a large step (needs to account for curvature).

The path consistency loss enforces that a single large step of size 2d produces the same result as two consecutive steps of size d:

$\mathcal{L}_{fast} = \mathbb{E}_{x_t, t, d} \left[ \left\| v_\theta(x_t, t, 2d) - \frac{1}{2} \left[ v_\theta(x_t, t, d) + v_\theta(x_{t+d}, t+d, d) \right] \right\|_2^2 \right]$

where x_{t+d} = x_t + d \cdot v_\theta(x_t, t, d) is the state after one small step, and the expectation is over the distribution of x_t (the noisy latents encountered during training), timesteps t, and step sizes d.

What this computes operationally:

Start with a noisy latent x_t at timestep t.
Compute the first small-step velocity: v_1 = v_θ(x_t, t, d) — this answers "if I take a step of size d, in what direction and how fast should I move?"
Compute where that step would land: x_{t+d} = x_t + d \cdot v_1 — this is the state after the first small step.
Compute the second small-step velocity: v_2 = v_θ(x_{t+d}, t+d, d) — this answers "now that I'm at t+d, if I take another step of size d, in what direction and how fast should I move?"
Average the two small-step velocities: v_small = (v_1 + v_2) / 2 — this represents the effective velocity over the combined interval of size 2d.
Compute the large-step velocity: v_large = v_θ(x_t, t, 2d) — this answers "if I take a single step of size 2d right now, in what direction and how fast should I move?"
Compute the squared L2 distance between the large-step velocity and the average of the small-step velocities: \|v_large - v_small\|_2^2.

Why averaging velocities: The path consistency loss works in velocity space, not state space. Enforcing x_{t+2d} = x_t + 2d \cdot v_large to equal x_t + d \cdot v_1 + d \cdot v_2 would be equivalent to enforcing v_large = (v_1 + v_2) / 2, which is the form used. Working in velocity space is more numerically stable because velocities are typically smaller in magnitude than states and the gradient flow is more direct.

Why this enables few-step generation: If the model satisfies path consistency perfectly, then a single large step of size 2d is exactly equivalent to two smaller steps of size d. By induction, a single very large step (size 2^n \cdot d) is equivalent to 2^n small steps. This means the model trained with path consistency can be sampled with any number of steps—from 1 step (jumping directly from t=0 to t=1) to many steps—and produce consistent results. The model learns a "shortcut" velocity field that directly points toward the data distribution even from far away.

The paper cites the original path consistency work (Frans et al., 2024) as the source of this formulation and notes that it "enables the model to learn shortcut-consistent velocity fields without separate teacher-student stages." This is an advantage over progressive distillation approaches (Salimans and Ho, 2022), which require training a sequence of student models, each distilling the previous one to use fewer steps.

Connection to physics feedback: Because path consistency training produces a model that generates high-quality outputs in 1–5 steps, the physics feedback can be applied to outputs that are actually clean enough for Lotus to produce reliable depth/normal estimates. In a standard multi-step diffusion model, applying the physics loss at intermediate timesteps would mean decoding noisy, incomplete outputs—Lotus would produce garbage depth/normal estimates from those, and the gradient signal would be meaningless. Path consistency solves this: train the model to produce good outputs in few steps, then apply physics supervision to those few-step outputs during training.

Path consistency loss weighting and scheduling:

The full training objective combines three losses:

$\mathcal{L} = \lambda_0 \mathcal{L}_0 + \lambda_1 \mathcal{L}_{fast} + \lambda_2 \mathcal{L}_{phy}$

with fixed weights λ_0 = 1.0 and λ_1 = λ_2 = 0.1 for all experiments. The standard flow-matching loss L_0 receives the highest weight (1.0) because it is the primary generative objective; the path consistency and physics losses receive lower weights (0.1 each) as regularizers that guide the model toward specific desirable properties (few-step capability and geometric consistency).

The paper reports that these weights are fixed for all experiments, implying they were chosen based on preliminary validation and not extensively tuned. The 10:1 ratio between the primary loss and the auxiliary losses is a common heuristic in multi-objective training—it ensures the auxiliaries influence the optimization without dominating and destabilizing the primary objective.

Joint Training Objective and Selective Optimization Strategy

The training strategy is designed to balance three competing demands: (1) learning the primary flow-matching objective, (2) learning path consistency for few-step generation, and (3) receiving physics feedback for geometric consistency—all while keeping training throughput manageable.

The Selective Optimization Scheduling (Algorithm 1)

The paper proposes a probabilistic batch-splitting strategy rather than computing all three losses for every sample:

"In each training iteration, 20% of each batch is allocated to compute the path consistency loss L_fast, which involves three forward passes and one backward pass to enforce consistency across timesteps. The remaining 80% is used for the standard flow-matching loss L_0, with 50% of these samples further supervised using RGB-space geometry feedback via L_phy (i.e., depth and normal alignment)."

Breaking this down per iteration with batch size B:

0.2 × B samples: Compute only L_fast. This requires three forward passes through the DiT backbone (one for v_θ(x_t, t, 2d), one for v_θ(x_t, t, d), one for v_θ(x_{t+d}, t+d, d)) and one backward pass to compute gradients. No physics feedback is computed on these samples because they are used for the path consistency objective, which operates purely in latent space.
0.4 × B samples (50% of the remaining 80%): Compute L_0 and L_phy. This requires: one forward pass for the velocity prediction, one VAE decode to RGB, one forward pass through Lotus for depth/normal estimation, and one backward pass. The additional cost comes from the VAE decode and Lotus forward pass, which are not negligible but are applied to only 40% of the batch.
0.4 × B samples (the other 50% of the remaining 80%): Compute only L_0. This is the cheapest case: one forward pass for velocity prediction and one backward pass. No VAE decode, no Lotus.

The total forward pass count per iteration is approximately: 0.2B × 3 + 0.4B × 1 + 0.4B × 1 = 0.6B + 0.4B + 0.4B = 1.4B DiT forward passes, plus 0.4B VAE decodes and Lotus forward passes. Compared to a naive approach that computes all three losses for all samples (which would require 1B × 3 + 1B = 4B forward passes plus VAE/Lotus for all samples), this represents an approximate 2.9× reduction in DiT computation and a 2.5× reduction in VAE/Lotus computation.

Why this probabilistic split works: The paper argues that "this probabilistic scheduling ensures high training throughput while allowing the model to benefit from multi-level supervision." The key insight is that each loss provides a different type of learning signal that doesn't need to be applied to every sample:

L_0 (flow matching) needs to be applied broadly because it is the primary generative objective—missing it on too many samples would degrade basic generation quality.
L_fast (path consistency) is a regularization that shapes the global structure of the velocity field. It can be applied to a subset of samples and still have effect because the consistency constraint is a smoothness condition that generalizes across the input space.
L_phy (geometry feedback) is the most expensive but also the most specific—it provides fine-grained geometric alignment that is most useful for samples where the model's lighting is geometrically inconsistent. Applying it to 40% of samples provides enough signal to shape the model's behavior without prohibitive cost.

The paper notes that this strategy is "inspired by path consistency scheduling" from the original path consistency work, adapting the idea of selective loss computation to the multi-objective setting.

Randomized Lighting Augmentations During Training

In addition to the structured loss scheduling, the paper applies "randomized lighting augmentations on the degraded subject V_deg, which introduces realistic lighting variability without the need for explicitly paired captures." While the details of these augmentations are not fully specified, the likely operations include random adjustments to brightness, contrast, color balance, and possibly synthetic shadow/highlight modifications applied to the already-degraded subject. This further increases the diversity of lighting conditions seen during training, helping the model generalize to a wide range of illumination scenarios.

Training Hyperparameters

The paper reports the following concrete training configuration:

Base model: Wan2.1-T2V-1.3B-480P
Optimizer: AdamW
Learning rate: 1 × 10⁻⁵
Batch size: 8
Training iterations: 5,000
Hardware: 8 NVIDIA H20 GPUs (96GB RAM each)
New parameter initialization: Zero weights for all new projection and fusion layers

The zero-initialization of new layers is a critical implementation detail. The Wan2.1 backbone is pre-trained on large-scale video generation; adding new input channels (for x_deg and x_bg concatenation) and new conditioning mechanisms (the step size d) requires new projection matrices. By initializing these to zero, the model initially ignores the new inputs entirely and behaves exactly like the pre-trained Wan2.1 (which would produce standard video generation, not relighting). As training progresses, the gradients for these new parameters become non-zero, and the model gradually learns to incorporate the relighting-specific conditioning signals.

Training Domain Mix

The paper trains a unified model that handles both images and videos. The training data includes both:

Approximately 110K video pairs from LumosData (Panda70M subset processed through the four-stage pipeline)
1.2M additional relit images generated using IC-Light

The ablation study (Table 2, "Effect of Training Domain") shows that training on videos only or images only leads to degraded performance:

Video-only training: PSNR drops from 25.031 to 22.487; SSIM drops from 0.891 to 0.863. The model learns temporal coherence but struggles with image quality.
Image-only training: PSNR drops moderately to 24.471, but R-Motion (temporal consistency) degrades dramatically from 1.436 to 2.429—the model produces good individual frames but with severe flickering and temporal incoherence.
Unified training: Achieves the best balance—high visual quality (PSNR 25.031) and strong temporal consistency (R-Motion 1.436).

This validates the design choice of unified image-video training: video data teaches temporal coherence, image data provides additional diversity and quality, and the model learns to handle both modalities in a single architecture.

Summary of Design Choices and Their Justifications

Gaussian background over real background or inpainting: Prevents the model from cheating by copying lighting cues from the background, forcing it to rely on the structured caption C for illumination control.
Depth and normal supervision over albedo, shadow, or photometric loss: Depth and normals are lighting-invariant, accessible via pre-trained models, and general across scene types—unlike photometric loss which would penalize legitimate lighting changes.
Physics feedback during training only, not as inference input: Keeps inference lightweight (no external depth estimator at test time) while internalizing geometric constraints into model weights through training-time gradient signals.
Path consistency learning over progressive distillation: Enables few-step generation without multi-stage teacher-student training, making it compatible with end-to-end training alongside the physics loss.
Selective loss computation (80/20 split) over full supervision: Maintains training throughput by not computing expensive losses (path consistency, physics) on every sample, while still providing sufficient signal for the model to learn all objectives.
Zero-initialization of new layers over random initialization: Ensures the model starts from the pre-trained Wan2.1 behavior and gradually adapts to the relighting task, preventing training instability from large initial gradients on randomly initialized parameters.
Fixed loss weights (λ₀ = 1.0, λ₁ = λ₂ = 0.1) over adaptive weighting: Simplifies the training pipeline; the 10:1 ratio provides a standard regularization strength that the paper found effective without expensive hyperparameter tuning.
Six-dimensional annotation protocol over free-form text: Enables fine-grained, interpretable control and evaluation across disentangled lighting dimensions, making it possible to measure per-attribute controllability rather than holistic "lighting quality."
Unified image-video training over modality-specific models: Video data provides temporal coherence; image data provides additional diversity; the unified model achieves the best of both without requiring separate architectures.

4. Key Insights and Innovations

Innovation 1: Strategic Geometric Supervision — Teaching Physics Through Consequences, Not Rules

The paper's most conceptually distinctive move is not that it uses geometric supervision, but how it positions that supervision within the learning pipeline. The dominant paradigm in diffusion-based relighting has been to either (a) hope the model implicitly learns lighting physics from data, or (b) condition the model explicitly on geometric estimates at both training and inference time. UniLumos introduces a third option: training-time-only geometric supervision that the model internalizes, producing geometry-consistent outputs at inference without requiring any geometric estimation.

This is a fundamentally different philosophical approach to imposing physical constraints on neural networks. Prior work that conditions on geometry (e.g., providing depth maps as additional input channels) essentially says: "Here is the 3D structure—now respect it." The model becomes dependent on that input and cannot function without it. Prior work that uses photometric reconstruction losses says: "Reproduce these pixel values exactly," which conflates geometry errors with legitimate lighting changes. UniLumos instead says: "I will check whether your output has consistent geometry, and if it doesn't, I will tell you—but only during training. At test time, you must have learned to produce consistent geometry from appearance alone."

This training-time-only feedback design (detailed in Section 4.1 and Appendix A, Q4) is significant because it represents a general principle for incorporating physical constraints into generative models that extends beyond relighting. The key insight is that physical consistency can be treated as a regularization signal rather than an input condition or a post-processing constraint. The model learns that producing geometrically inconsistent lighting will be penalized, so it develops internal representations that implicitly respect 3D structure—without ever being told explicitly what that structure is at inference time. This is analogous to how adversarial training teaches a generator to produce outputs that fool a discriminator, without the discriminator being present at deployment.

The evidence that this works comes from the ablation study (Table 2), where removing all physics feedback drops PSNR from 25.031 to 21.433 and Lumos Score from 0.871 to 0.859—substantial degradations in both perceptual quality and lighting controllability. Critically, these improvements come with zero inference-time overhead (Figure 5 shows a 20× speedup over prior methods), confirming that the geometric understanding has been successfully internalized into the model weights. If the model had merely memorized training examples without learning generalizable geometric reasoning, the physics-free ablation would perform similarly.

This framing matters because it challenges the prevailing assumption in physically-based vision that explicit 3D reasoning (through inverse rendering, differentiable rendering, or geometric conditioning) is necessary for physically correct outputs. UniLumos demonstrates that implicit geometric understanding, cultivated through strategically placed training signals, can substitute for explicit geometric reasoning at deployment—a finding with implications for any domain where physical constraints are important but real-time inference with explicit physics is impractical.

Innovation 2: Lighting as a Decomposable Semantic Space — The Six-Dimensional Annotation Protocol

Prior to UniLumos, lighting in generative models was treated as an opaque, holistic concept. Users provided free-form text prompts ("warm evening light," "harsh midday sun"), and models were expected to interpret these as unified lighting descriptors. This approach conflates independent physical dimensions—direction, intensity, color temperature, source type, temporal behavior, and optical effects—into an entangled semantic blob, making it impossible to evaluate whether a model correctly controls for which aspect of lighting when a prompt is modified.

The paper's six-dimensional annotation protocol represents a conceptual reframing of lighting as a structured, decomposable semantic space. This is not merely an engineering convenience for data labeling—it is an intellectual move that enables entirely new forms of analysis. By decomposing illumination into orthogonal axes with discrete categories per axis (e.g., direction: {Front, Side, Back, Top, Bottom, Split, Ambient}), the protocol transforms lighting control from a continuous, subjective matching problem into a set of independent, testable hypotheses: can the model control direction while holding intensity constant? Can it control color temperature while holding direction constant?

The significance of this reframing becomes clear in LumosBench (Table 7). General-purpose models like Wan2.1-14B achieve high overall generation quality but show dramatically uneven controllability across dimensions—scoring 0.871 on direction but only 0.674 on intensity and 0.505 on optical phenomena. These differential failures would be invisible under holistic evaluation metrics like FID or LPIPS, which can only report that the generated video "looks realistic." The six-dimensional decomposition reveals what kind of lighting control the model possesses and lacks—a diagnostic capability that was previously unavailable.

Even more revealing is the ablation: UniLumos trained without structured captions ("w/o lumos captions" in Table 7) achieves an average score of 0.662, substantially below the full model's 0.773. The gap is largest on precisely the dimensions that are most entangled in natural language: Intensity drops from 0.832 to 0.529 when captions are removed, and Optical Phenomena drops from 0.592 to 0.457. This demonstrates that explicit decomposition during training is necessary for disentangled control—the model does not spontaneously learn to separate these dimensions from unstructured text alone, even when the data contains diverse lighting.

This innovation has implications beyond relighting. The methodology—defining a domain-specific ontology of independent semantic axes, generating structured annotations using VLMs, and evaluating per-axis controllability—is a template that could be applied to any generative control task: facial expression editing (decomposing emotion into valence, arousal, action units), scene composition (decomposing into camera angle, depth of field, object arrangement), or style transfer (decomposing into color palette, texture, brush stroke). The paper demonstrates that structured semantic decomposition, combined with VLM-based automatic annotation, can transform an ill-defined generation task into a set of well-defined, independently evaluable sub-problems.

Innovation 3: The Over-Optimization Analogy Revisited — Physical Supervision Regularizes Against Semantic Drift

The paper does not explicitly frame this as an innovation, but a careful reading reveals a finding with broad implications: the physics-plausible feedback appears to function as a regularizer against semantic drift during the flow-matching process, not merely as a geometric accuracy constraint. This is visible in the qualitative ablation (Figure 6), where the model without physics feedback produces subjectively "plausible" lighting that is actually physically incorrect—shadows are in the wrong place, highlights appear on surfaces facing away from the implied light source—and the subject's identity begins to drift (facial features distort, clothing texture degrades).

This phenomenon is conceptually analogous to reward over-optimization in language model alignment (where optimizing too aggressively for a learned reward model produces outputs that score highly but are semantically degraded) and to adversarial examples in classification (where inputs are modified to satisfy a loss function without preserving the underlying semantic content). In the relighting context, the standard flow-matching loss L_0 optimizes for velocity prediction accuracy—ensuring the model can transport noise to data efficiently—but does not constrain what kind of data is being produced. Without geometric supervision, the model can learn velocity fields that produce visually sharp, high-contrast outputs (which pass perceptual metrics) while systematically altering the subject's appearance in ways that violate physical priors.

The physics feedback acts as a structural regularizer that constrains the velocity field to preserve intrinsic geometric properties. This is fundamentally different from standard regularization techniques (weight decay, dropout, data augmentation) which operate in parameter or data space without semantic grounding. The depth and normal consistency loss encodes a specific semantic invariant: the 3D shape of the subject must not change under relighting. By penalizing violations of this invariant, the model is forced to learn velocity fields that modify illumination while preserving geometry—a constraint that is physically motivated rather than statistically convenient.

The evidence for this interpretation comes from the individual ablation of depth vs. normal feedback (Table 2): removing only normal supervision causes a larger drop in Lumos Score (0.863 vs. 0.870 for depth removal) and PSNR (22.115 vs. 23.472 for depth removal). This asymmetry suggests that surface orientation (normals) provides a stronger regularization signal than depth—likely because shadow and highlight placement are more directly governed by surface orientation relative to light sources than by absolute distance, making normal consistency a more targeted constraint for preventing lighting-specific artifacts.

This insight—that physically-grounded supervision can regularize against semantically meaningful but physically invalid outputs—generalizes beyond relighting. Any generative task where outputs must obey physical or structural constraints (fluid simulation, articulated motion, material editing) could benefit from analogous training-time regularizers that encode domain-specific invariants. The key design principle UniLumos demonstrates is that the regularizer should operate on geometry-invariant representations (depth, normals) rather than appearance-dependent representations (pixels, perceptual features), ensuring that the constraint genuinely captures physical validity rather than visual similarity.

Innovation 4: Training Infrastructure as Scientific Contribution — Making RGB-Space Feedback Computationally Viable

The paper makes what initially appears to be a purely engineering contribution—the selective optimization strategy with path consistency learning—but this constitutes a genuine scientific advance because it solves a compute-feasibility problem that was previously blocking an entire class of training methodologies. Before UniLumos, the idea of incorporating dense RGB-space feedback signals into diffusion model training was theoretically appealing but practically infeasible: decoding to RGB and running auxiliary networks (like depth estimators) at scale during training was prohibitively expensive, and the noisy intermediate outputs from standard multi-step diffusion produced meaningless auxiliary signals.

The paper's insight is that path consistency learning and physics feedback are not independent improvements but synergistic components that enable each other. Path consistency makes few-step generation possible; few-step generation makes RGB-space physics supervision practical (because clean outputs are available in 1–5 steps rather than 50–100); physics supervision improves output quality for the few-step regime; the improved few-step outputs make path consistency easier to learn. This creates a virtuous cycle where the two techniques reinforce each other rather than merely adding independent benefits.

The specific numbers (Algorithm 1: 20% of batch for path consistency, 50% of the remaining 80% for physics feedback) encode a non-obvious finding about how to schedule multi-objective training when objectives have dramatically different computational costs. The 80/20 split is not arbitrary—it balances the need for broad coverage of the primary flow-matching objective (80% of samples) with the regularizing effects of path consistency (20%) and physics feedback (40% of the 80%). The paper's finding that this probabilistic scheduling works—despite each loss being applied to only a subset of samples—demonstrates that the gradients from these auxiliary objectives are sufficiently correlated across the data distribution that sparse application is sufficient.

This contribution is significant because it opens the door to a broader class of training-time feedback mechanisms that were previously dismissed as computationally impractical. Any dense visual feedback signal—aesthetic quality from a VLM, physical consistency from a simulator, task-specific constraints from a differentiable renderer—could potentially be integrated into diffusion model training using the same combination of path consistency (for few-step clean outputs) and selective scheduling (for computational efficiency). The paper provides not just a proof of concept but a concrete recipe with validated hyperparameters that other researchers can adapt.

The efficiency results (Figure 5: 277 seconds for UniLumos-1.3B vs. 756–917 seconds for baselines on 49-frame 480p video) demonstrate that the training-time investment in path consistency pays off at inference with a ~3× speedup over the nearest baseline and a ~20× speedup over per-frame IC-Light. This is not merely a nice-to-have—it is the difference between a method that is viable for interactive applications and one that is not, and it is achieved through a training innovation rather than through model compression or hardware optimization.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper evaluates on three distinct test sets. The primary test set consists of samples from an internal dataset processed through the LumosData pipeline, evenly split between image generation at 768×512 resolution and video generation at 480p resolution (832×480) with 49 frames per video sample. Additionally, two public object-centric relighting benchmarks are used: StanfordOrb (Kuang et al., 2023), which contains canonical 3D scanned objects such as the Stanford bunny and dragon under varied lighting environments, and Navi (Jampani et al., 2023), which includes diverse everyday objects like containers, toys, and mugs. Both public datasets are "completely disjoint from our training data" (Appendix C.1), providing a test of generalization to non-human subjects and scenes outside the training distribution.
Base model. All experiments use Wan2.1-T2V-1.3B-480P as the pre-trained foundation model. The paper argues this model provides a strong spatiotemporal prior for video generation, with the 1.3B parameter scale chosen to balance generation quality with training and inference efficiency. The VAE encoder/decoder from Wan2.1 is frozen throughout training; only the DiT backbone parameters are updated (plus newly added projection and fusion layers, which are zero-initialized).
Metrics. The paper evaluates across three dimensions: (a) Visual fidelity: Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM), and Learned Perceptual Image Patch Similarity (LPIPS), all computed between generated and reference videos. For video, metrics are averaged across all frames. (b) Temporal consistency: R-Motion from VBench, which measures temporal smoothness using motion priors from a pre-trained video frame interpolation model (AMT; Li et al., 2023). Lower R-Motion indicates smoother frame transitions. (c) Lumos consistency: Two metrics are introduced—(i) Lumos Score, which uses the same structured six-dimensional annotation protocol as LumosData to classify the generated output's lighting attributes and compare against target attributes, averaging a consistency score across dimensions (equal weight per dimension), and (ii) Dense L2 Error, computing the relative L2 error between predicted and reference depth/normal maps (estimated via Lotus), providing a physically grounded measure of geometry-illumination alignment. Lower Dense L2 Error indicates better geometric consistency. The Lumos Score is novel to this paper and specifically designed to capture lighting controllability, which generic perceptual metrics miss.
Baselines. For image relighting, four methods are compared: SwitchLight (Kim et al., 2024), DiLightNet (Zeng et al., 2024), IC-Light (Zhang et al., 2025), and SynthLight (Chaturvedi et al., 2025). SwitchLight uses a physics-driven architecture and pre-training framework for human portrait relighting. DiLightNet provides fine-grained lighting control for diffusion-based image generation. IC-Light is the most widely adopted baseline, performing illumination harmonization via MLP-based lighting embeddings. SynthLight uses multi-stage training on synthetic face data for portrait relighting. For video relighting, three configurations are compared: IC-Light applied per-frame (frame-by-frame relighting with no temporal modeling), Light-A-Video + CogVideoX-2B (Zhou et al., 2025; Yang et al., 2024), and Light-A-Video + Wan2.1-1.3B. Light-A-Video is a training-free framework that iteratively aligns IC-Light outputs with a pre-trained video diffusion model. The paper also compares against general-purpose text-to-video models (LTX-Video, CogVideoX, HunyuanVideo, Wan2.1 at 1.3B and 14B) in the LumosBench controllability evaluation (Table 7).
Generation budget / compute accounting. The primary efficiency comparison is wall-clock inference time for video relighting of 49 frames at 480p resolution (Figure 5). The paper reports: UniLumos-1.3B completes generation in 277 seconds, compared to 756 seconds for Light-A-Video + CogVideoX-2B, 917 seconds for Light-A-Video + Wan-1.3B, and an implied ~12 seconds per frame for IC-Light (which would total ~588 seconds for 49 frames at that per-frame rate, though the paper doesn't spell out this multiplication). The ~20× speedup claim appears to be relative to the slowest baseline. For the FLOPs-matched or parameter-matched comparison common in scaling law papers, no such analysis is performed—the efficiency comparison is wall-clock only, not FLOPs-matched.
Cross-validation / statistical protocol. The paper does not report cross-validation, statistical significance tests, confidence intervals, or error bars for any result. The main quantitative comparisons (Tables 1, 5, 6) report single scalar values per metric per method. The test set sizes are not explicitly stated for the internal dataset. For StanfordOrb and Navi, the full datasets are used but exact numbers of test samples are not provided. This is a notable methodological gap: without error bars or statistical tests, small differences between methods (e.g., SSIM of 0.911 vs. 0.903 in Table 6) cannot be interpreted with confidence. The LumosBench evaluation uses a constructed set of 2K test prompts, each consisting of a video and structured caption, with scores per dimension averaged across all prompts—but again without variance estimates.

Main Quantitative Results

Image Relighting Quality (Table 1a, Appendix Tables 5–6)

The headline result for image relighting is that UniLumos achieves the best scores across all six metrics in the visual fidelity and Lumos consistency categories for the internal test set. Specifically:

PSNR: UniLumos achieves 26.719 dB, compared to 25.572 dB for SynthLight (the best baseline) and 24.316 dB for IC-Light. This represents a +1.15 dB improvement over the strongest baseline.
SSIM: UniLumos achieves 0.913, compared to 0.905 for SynthLight and 0.884 for IC-Light. The differences are small in absolute terms—SSIM is bounded in [0,1] and values near 0.9 indicate high structural similarity across all methods.
LPIPS: UniLumos achieves 0.089 (lower is better), compared to 0.102 for SynthLight and 0.108 for IC-Light. LPIPS is a perceptual metric sensitive to subtle texture and structural differences; the 0.013 improvement over SynthLight is meaningful but not dramatic.
Lumos Score: UniLumos achieves 0.912, substantially higher than SynthLight at 0.791 and IC-Light at 0.703. This is the metric where UniLumos shows its largest relative advantage—a +0.121 improvement over the best baseline, representing a ~15% relative improvement. This aligns with the paper's central claim that structured lighting conditioning and physics feedback improve controllability.
Dense L2 Error: UniLumos achieves 0.103, compared to 0.214 for SynthLight and 0.447 for IC-Light. This is a ~2× reduction in geometric error relative to the best baseline, directly validating the claim that physics-plausible feedback improves geometry-lighting alignment.

For the public benchmarks, the pattern is consistent but the advantages are smaller. On StanfordOrb (Table 5): PSNR 26.512 (vs. 25.784 for Light-A-Video + Wan2.1), SSIM 0.934 (vs. 0.926), LPIPS 0.097 (vs. 0.104), R-Motion 1.103 (vs. 1.241). On Navi (Table 6): PSNR 24.977 (vs. 23.912 for Light-A-Video + CogVideoX), SSIM 0.911 (vs. 0.903 for Light-A-Video + Wan2.1), LPIPS 0.120 (vs. 0.116 for Light-A-Video + Wan2.1—notably, UniLumos does NOT have the best LPIPS here), R-Motion 1.203 (vs. 1.341).

The LPIPS result on Navi is noteworthy: Light-A-Video + Wan2.1 achieves 0.116 vs. UniLumos at 0.120, meaning the baseline is slightly better on this perceptual metric. This is the only case across all tables where UniLumos does not achieve the best score. The paper does not discuss this result or provide an explanation. It suggests that on object-centric scenes with domain gaps from human-centric training data, UniLumos's geometric supervision may not translate to uniformly better perceptual quality—the geometry-lighting alignment improves (lower Dense L2 Error) but the perceptual texture may be slightly less faithful.

Video Relighting Quality and Temporal Consistency (Table 1b)

For video relighting on the internal test set, UniLumos shows larger absolute improvements over baselines than for image relighting:

PSNR: UniLumos achieves 25.031, compared to 20.784 for Light-A-Video + Wan2.1 (the best video baseline) and 20.132 for IC-Light per-frame. This is a +4.25 dB improvement—a substantially larger margin than in the image setting, suggesting that the temporal modeling and physics feedback are particularly beneficial for video where baseline methods suffer from frame-to-frame inconsistencies.
SSIM: UniLumos achieves 0.891, compared to 0.876 for Light-A-Video + Wan2.1.
LPIPS: UniLumos achieves 0.109, compared to 0.124 for Light-A-Video + CogVideoX and 0.129 for Light-A-Video + Wan2.1—a 0.015–0.020 improvement.
R-Motion: UniLumos achieves 1.436, compared to 1.582 for Light-A-Video + Wan2.1 and 2.437 for IC-Light per-frame. The IC-Light per-frame result is substantially worse, confirming that frame-by-frame relighting without temporal modeling produces severe flickering. The improvement over Light-A-Video + Wan2.1 (1.436 vs. 1.582) is a 0.146 reduction in motion error—meaningful but not as dramatic as the visual quality improvements.
Lumos Score: UniLumos achieves 0.871, compared to 0.682 for Light-A-Video + Wan2.1 and 0.672 for IC-Light per-frame. The ~0.19 advantage is the largest relative improvement across any metric, confirming that structured conditioning and physics feedback provide substantially better lighting controllability.
Dense L2 Error: UniLumos achieves 0.147, compared to 0.371 for Light-A-Video + Wan2.1 and 0.432 for IC-Light per-frame—a ~2.5× reduction in geometric error.

The pattern across both image and video results is consistent: UniLumos shows moderate improvements in perceptual quality (PSNR, SSIM, LPIPS) but much larger improvements in lighting controllability (Lumos Score) and geometric consistency (Dense L2 Error) . This aligns perfectly with the paper's claim that the physics feedback and structured conditioning specifically improve physical plausibility and controllability, not just overall image quality. The perceptual metrics alone would show UniLumos as marginally better; the Lumos metrics reveal the qualitative difference.

Inference Efficiency (Figure 5)

The inference time comparison for 49-frame 480p video relighting:

UniLumos-1.3B: 277 seconds
Light-A-Video + CogVideoX-2B: 756 seconds (~2.7× slower)
Light-A-Video + Wan-1.3B: 917 seconds (~3.3× slower)
IC-Light per-frame: 12 seconds per frame. At 49 frames, this would be approximately 588 seconds (implied, not explicitly stated). The paper labels this comparison as "76×" in Figure 5, but the arithmetic is unclear—if the per-frame cost is 12 seconds and UniLumos takes 277 seconds for 49 frames, the speedup is approximately 588/277 ≈ 2.1×, not 76×. The 76× likely refers to a per-frame comparison (12s / 0.16s ≈ 75× if UniLumos were run per-frame, but UniLumos processes all frames jointly). The paper's "20× speedup" claim in the abstract is not precisely localized to a specific baseline in the main text.

The key takeaway is that UniLumos is substantially faster than all compared video relighting methods, benefiting from its few-step generation capability (enabled by path consistency training) and the absence of external models during inference (unlike Light-A-Video, which requires iterative alignment between IC-Light and a video diffusion model).

LumosBench: Attribute-Level Controllability (Table 7)

The LumosBench evaluation provides the most granular view of model capabilities, decomposing lighting control into six independent dimensions and evaluating each with VLM-based classification accuracy.

General models vs. specialized models: The best general model, Wan2.1-14B, achieves an average Lumos Score of 0.735, demonstrating that large-scale pretraining alone provides substantial lighting controllability. However, UniLumos (1.3B) achieves 0.773—surpassing the 14B model by 0.038 despite having ~11× fewer parameters. This is the paper's strongest evidence that targeted relighting training with structured supervision can compensate for a large gap in model scale.

Per-dimension analysis: UniLumos shows heterogeneous performance across dimensions:

Direction: 0.893 (strongest dimension)
Light Source Type: 0.847
Intensity: 0.832
Color Temperature: 0.813
Temporal Dynamics: 0.662
Optical Phenomena: 0.592 (weakest dimension)

The pattern reveals that UniLumos excels at spatial and colorimetric attributes (direction, source type, intensity, color temperature all above 0.80) but struggles with temporal dynamics and optical phenomena. The Temporal Dynamics score of 0.662 suggests that distinguishing static from dynamic lighting, or intensity-gradual from moving-source dynamics, remains challenging. The Optical Phenomena score of 0.592—only modestly above the best general model (Wan2.1-14B at 0.505)—indicates that effects like transmission through glass, refraction, and scattering are not well-captured by the current training data or architecture.

Ablation on structured captions: UniLumos without structured captions ("w/o lumos captions") achieves an average score of 0.662 vs. 0.773 for the full model—a 0.111 drop. The drop is uneven across dimensions:

Intensity: 0.832 → 0.529 (largest drop, −0.303)
Optical Phenomena: 0.592 → 0.457 (drop of −0.135)
Temporal Dynamics: 0.662 → 0.543 (drop of −0.119)
Direction: 0.893 → 0.868 (smallest drop, −0.025)

This asymmetry is informative: Direction is relatively robust to the absence of structured captions, suggesting the model can learn directional lighting control from pixel-level cues alone (e.g., the geometric feedback already provides information about light direction through shadow and highlight placement). Intensity is heavily dependent on structured captions, likely because intensity is a global image statistic that can be varied without changing geometry—without explicit semantic labels for "Moderate" vs. "Glare" vs. "Dim," the model has no signal distinguishing a legitimate intensity change from a data variation.

Comparison with specialized baselines: The specialized relighting baselines significantly underperform even general models on LumosBench. IC-Light achieves an average of 0.468; Light-A-Video + Wan2.1 achieves 0.553. This is striking because these methods are explicitly designed for relighting, yet their controllability scores are lower than general text-to-video models like Wan2.1-1.3B (0.607) and far below Wan2.1-14B (0.735). The paper attributes this to "the limitations of their base architectures—typically smaller and trained from scratch or on narrow domains—resulting in weaker generalization to diverse lighting attributes." An alternative interpretation is that these methods, by optimizing for perceptual reconstruction fidelity, produce outputs that look realistic but don't necessarily match the requested lighting—a form of "lighting mode collapse" where the model defaults to a plausible lighting distribution rather than the specific one requested.

Ablation Studies and Robustness Checks

Physics-Plausible Feedback Removal: Removing both depth and normal feedback ("w/o All Feedback" in Table 2) causes the largest degradation: PSNR drops from 25.031 to 21.433 (−3.60 dB), SSIM from 0.891 to 0.862, LPIPS increases from 0.109 to 0.139, R-Motion increases from 1.436 to 1.473, Lumos Score drops from 0.871 to 0.859 (a surprisingly small drop for the controllability metric), and Dense L2 Error increases from 0.147 to 0.297 (−0.150, approximately doubling the geometric error). This confirms that physics feedback is essential for geometric consistency and that its removal causes substantial degradation across almost all metrics. The small drop in Lumos Score (only −0.012) is noteworthy: it suggests that the structured caption conditioning alone (without physics feedback) is sufficient for reasonable controllability, and the physics feedback primarily improves the alignment between the generated lighting and the underlying geometry rather than the semantic accuracy of the lighting.

Individual Feedback Component Ablation: Removing only normal feedback ("w/o Normal Feedback") causes larger degradation than removing only depth feedback ("w/o Depth Feedback"):

PSNR: 22.115 (normal removal) vs. 23.472 (depth removal)—a 1.36 dB difference
SSIM: 0.874 vs. 0.883
Dense L2 Error: 0.173 vs. 0.265 (depth removal has worse geometric error)
Lumos Score: 0.863 vs. 0.870 (nearly identical)

The interpretation offered in Section 4 (Key Insights) applies here: surface normals provide a stronger regularization signal because shadow and highlight placement depend directly on surface orientation relative to light sources. Depth provides information about occlusion relationships (which object is in front of which) and absolute scale, but normal orientation is more directly diagnostic of whether a highlight appears on the correct side of a face or whether a shadow aligns with the occluding contour. The fact that Dense L2 Error is worse when depth feedback is removed (0.265 vs. 0.173 for normal removal) seems counterintuitive given the stronger PSNR for depth removal, and the paper does not explain this. One possibility: the Dense L2 Error metric itself is computed on both depth and normals, so removing depth feedback may cause the model to produce errors that disproportionately affect the depth component of the Dense L2 Error while normal removal affects both components more evenly.

Path Consistency Removal: Removing path consistency learning ("w/o Path Consistency" in Table 2) has minimal impact on most metrics: PSNR 25.317 (higher than the full model's 25.031—an unexpected improvement), SSIM 0.902 (higher than full model's 0.891), LPIPS 0.113 (slightly worse than 0.109), R-Motion 1.438 (essentially identical to 1.436), Lumos Score 0.875 (slightly higher than 0.871), Dense L2 Error 0.153 (slightly worse than 0.147). The paper frames this as: "path consistency incurs little performance cost but offers substantial efficiency benefits in few-step regimes, justifying its inclusion." This is accurate but slightly misleading—the model without path consistency actually achieves better PSNR and SSIM (though differences are small and likely within noise given the absence of error bars). The real value of path consistency is not quality improvement during full-step generation (where the full model and the ablation perform similarly) but rather quality preservation when sampling with few steps. This is demonstrated qualitatively in Figure 6 (top row: 1-step results), where the full model with path consistency produces clean outputs in 1 denoising step while the flow-matching-only variant (implicitly, without path consistency, though this specific ablation is not labeled in Figure 6) would produce degraded results. However, the paper does not provide a quantitative ablation of few-step quality—all metrics in Table 2 appear to be computed at the model's default sampling configuration, which may use more steps than the few-step regime where path consistency matters most.

Training Domain Ablation: Training on only video data ("Only Video") causes severe degradation: PSNR 22.487 vs. 25.031, SSIM 0.863 vs. 0.891, LPIPS 0.119 vs. 0.109. Training on only image data ("Only Image") causes moderate degradation in visual quality (PSNR 24.471 vs. 25.031) but severe degradation in temporal consistency: R-Motion 2.429 vs. 1.436—a 0.993 increase, nearly doubling the motion error. The image-only model also shows degraded Lumos Score (0.841 vs. 0.871). This confirms that video data is essential for learning temporal coherence, while image data provides additional diversity that boosts visual quality. The unified training achieves the best of both, validating the design choice.

UniLumos without Structured Captions (Table 7, LumosBench): This ablation is reported in the LumosBench results, not in Table 2. Removing the structured six-dimensional captions during training ("UniLumos w/o lumos captions") causes the average Lumos Score to drop from 0.773 to 0.662. The per-dimension breakdown is discussed above in the LumosBench section. This is a critical ablation because it isolates the contribution of the annotation protocol from the contribution of the physics feedback and path consistency—the "w/o lumos captions" variant still has physics feedback and path consistency but uses only vanilla scene-level captions. The large gap (0.111 average, up to 0.303 for Intensity) demonstrates that structured semantic decomposition is necessary for fine-grained controllability, not merely helpful.

Critical Assessment

How Strong Is the Evidence for the "20× Speedup" Claim?

The paper's abstract and introduction prominently claim a "20× speedup for both image and video relighting." The evidence provided is Figure 5, which reports inference times for 49-frame 480p video: UniLumos at 277 seconds, IC-Light at 12 seconds per frame. If IC-Light processes 49 frames sequentially, that would total approximately 588 seconds, for a 2.1× speedup (588/277)—not 20×. The 20× figure likely comes from a per-frame comparison (12 seconds for IC-Light vs. approximately 0.6 seconds per frame for UniLumos when processing 49 frames in 277 seconds), but this is not clearly specified in the text. Additionally, IC-Light is an image relighting method being repurposed for video by per-frame application—it was never designed or optimized for video throughput. A more meaningful comparison would be against Light-A-Video, which is a dedicated video relighting method, where the speedup is approximately 2.7–3.3× (756–917 seconds vs. 277 seconds). The 20× figure is therefore a best-case comparison against the least efficient baseline and does not represent the typical speedup a practitioner would achieve over a reasonable video relighting baseline. The paper would be stronger if it clearly specified which baseline the 20× refers to and reported speedups against each baseline individually.

How Strong Is the Evidence That Physics Feedback Improves Physical Plausibility?

The primary evidence is the Dense L2 Error metric (Table 1), where UniLumos achieves substantially lower geometric error than baselines (0.103–0.147 vs. 0.214–0.447). However, Dense L2 Error is computed using the same type of model (Lotus depth/normal estimator) that provides the training supervision. This creates a potential circularity: the model is trained to minimize a loss computed by Lotus, and its geometric quality is evaluated using Lotus. A model could theoretically learn to produce outputs that Lotus evaluates favorably without genuinely improving geometric consistency—this is the same over-optimization concern the paper raises in other contexts. To break this circularity, the paper would need to evaluate geometric consistency using a different estimator not used during training, or using ground-truth geometry from synthetic data where depth and normals are known exactly. The StanfordOrb and Navi benchmarks provide some independence (they are outside the training distribution), but the metric is still computed by Lotus. The qualitative results in Figures 3, 4, and 6 do show visibly better shadow alignment and highlight placement, but qualitative evidence is inherently subjective.

The ablation results (Table 2) provide stronger evidence: removing physics feedback causes Dense L2 Error to double (0.147 → 0.297), and qualitative results in Figure 6 show visible degradation in lighting quality. The fact that the ablation without physics feedback performs worse on a metric computed by the same model used for training is expected (the model was trained to optimize that metric), but the qualitative confirmation and the PSNR/SSIM degradation (which don't involve Lotus) suggest the improvement is genuine rather than metric overfitting.

A missing experiment: evaluating geometric consistency using an alternative depth/normal estimator (e.g., Marigold, Depth Anything, or a stereo-based method for video) would provide stronger, independent evidence that the physics feedback improves actual geometric consistency rather than just Lotus-specific consistency.

How Strong Is the Evidence That Structured Captions Improve Controllability?

The LumosBench results (Table 7) provide a well-designed evaluation: the same VLM (Qwen2.5-VL) classifies lighting attributes in generated outputs, and accuracy is measured per dimension. The ablation without structured captions shows a substantial drop (0.773 → 0.662 average), with the largest drops on dimensions that are most entangled in natural language (Intensity, Optical Phenomena). The comparison with larger general-purpose models (Wan2.1-14B: 0.735) shows that structured training can surpass models with ~11× more parameters on this specific capability.

However, there is a potential confound: the VLM used for evaluation (Qwen2.5-VL) is the same type of model used to generate the structured captions during training. If Qwen2.5-VL has systematic biases in how it classifies lighting attributes (e.g., consistently confusing "Moderate" and "Dim" intensity in certain scene types), the training data would inherit those biases, and the evaluation would not detect them. Using a different VLM for evaluation (e.g., GPT-4V, Gemini) would provide a robustness check. The paper does not report such a cross-VLM validation.

Additionally, the LumosBench evaluation measures whether the generated output's lighting attributes match the input caption, but it does not measure whether those attributes are physically correct —only whether they are consistent with what the VLM perceives. A model could generate outputs where the VLM classifies the lighting as "Front Light" and the caption says "Front Light," but the shadows might still be geometrically implausible for a front light source if the VLM's classification is surface-level. The Dense L2 Error partially addresses this by measuring geometric consistency, but an ideal evaluation would combine VLM-based semantic accuracy with physics-based geometric accuracy.

How Well Does the Evidence Generalize Beyond the Training Distribution?

The public benchmark results (Tables 5 and 6) provide the primary test of generalization. On StanfordOrb (3D scanned objects) and Navi (everyday objects), UniLumos maintains its advantage over baselines, but the margins are smaller than on the internal (human-centric) test set. Specifically:

StanfordOrb: PSNR advantage over best baseline is +0.73 dB (26.512 vs. 25.784)
Navi: PSNR advantage is +1.07 dB (24.977 vs. 23.912), but LPIPS is slightly worse than the best baseline (0.120 vs. 0.116)

The LPIPS regression on Navi is the paper's clearest signal that geometric supervision does not uniformly improve all aspects of quality on out-of-distribution scenes. The model may produce better shadow alignment (lower Dense L2 Error) but at a slight cost to perceptual texture fidelity when the scene type differs from training. The paper does not analyze this tradeoff or discuss when the physics feedback might over-constrain the model.

A significant gap: all test sets, including the public benchmarks, consist of relatively simple, object-centric scenes with a single foreground subject and relatively uniform backgrounds. No results are shown for complex multi-subject scenes, outdoor environments with natural illumination, scenes with multiple light sources, or scenes with strong inter-reflections. The claims of "physically plausible relighting" are therefore demonstrated only in a constrained setting that is simpler than many real-world relighting applications. The LumosData pipeline's reliance on BiRefNet for subject masking (which works best on clearly delineated foreground objects) and IC-Light for degradation generation (which is portrait-oriented) may contribute to this limitation.

Are the Quantitative Improvements Practically Meaningful?

The PSNR improvement of +1.15 dB over SynthLight (Table 1) and +4.25 dB over Light-A-Video + Wan2.1 (Table 1b) are technically significant. In image/video processing, improvements of 1–2 dB are generally considered meaningful for perceptual quality when confirmed by other metrics. The simultaneous improvements in SSIM, LPIPS, R-Motion, and Lumos Score provide converging evidence that the improvement is real and multi-dimensional.

However, the absolute PSNR values (25–27 dB for images, 25 dB for video) are relatively low for a reconstruction task where the goal is to reproduce a reference video. In image compression, 25–30 dB is considered "low quality" with visible artifacts. This suggests that while UniLumos outperforms baselines, the absolute relighting quality still has substantial room for improvement—consistent with the paper's own limitation statement that "UniLumos is still limited by a broader challenge—achieving physically precise and controllable relighting" (Appendix E).

The Lumos Score differences are larger in relative terms (0.912 vs. 0.791 for images, 0.871 vs. 0.682 for video), but the absolute scale is less interpretable than PSNR. A score of 0.912 means the VLM classified 91.2% of lighting attributes correctly across all dimensions, but this depends on the VLM's own accuracy and the difficulty of the classification task. Without a human-evaluated baseline for the VLM's classification accuracy, the absolute Lumos Score is hard to calibrate.

Missing Experiments That Would Strengthen the Paper

Per-step quality ablation for path consistency: The paper claims path consistency enables few-step generation, but all quantitative metrics are reported without specifying the number of sampling steps. An ablation showing PSNR vs. number of sampling steps (1, 2, 4, 8, 16, 32, 50) for both the full model and the "w/o Path Consistency" variant would directly demonstrate the claimed benefit.
Cross-estimator geometric evaluation: Using an alternative depth/normal estimator (not Lotus) to compute Dense L2 Error would address the circularity concern and provide stronger evidence for genuine geometric improvement.
Human evaluation: Given that the paper's central claim is about physical plausibility and lighting quality—perceptual attributes that are ultimately judged by humans—a user study comparing UniLumos outputs against baselines would substantially strengthen the claims. The VLM-based Lumos Score is a proxy for human judgment, not a replacement.
Complex scene evaluation: Results on scenes with multiple subjects, complex backgrounds, multiple light sources, or outdoor natural illumination would test the claimed generality of the approach.
Failure case analysis: The paper presents only successful examples. A systematic analysis of when UniLumos fails—which lighting conditions, which scene types, which attributes—would provide a more complete picture of capabilities and limitations, and would be consistent with the critical tone the paper takes toward prior methods.
Training data ablation by scale: An ablation showing how performance scales with the number of LumosData training samples (e.g., 10K, 50K, 110K) would reveal whether the benefits saturate or would continue to improve with more data.

Summary Assessment

The paper's central claims are partially supported with notable gaps in evaluation rigor:

"Physics-plausible feedback improves physical consistency": Supported by Dense L2 Error improvements and ablation results, but the circularity of using Lotus for both training and evaluation weakens the evidence. Qualitative results are suggestive but not systematic.
"Structured captions improve fine-grained controllability": Well-supported by the LumosBench ablation, with the caveat that the evaluation VLM is the same model family used for annotation.
"20× speedup": Misleading without clear specification of the baseline and per-frame vs. per-video comparison. Against the most reasonable video baseline (Light-A-Video), the speedup is ~3×, which is still significant but an order of magnitude less than claimed.
"State-of-the-art relighting quality": Supported across multiple metrics and datasets, with the qualifier that the test scenes are relatively simple and the absolute quality still shows room for improvement.
"Unified image and video framework": Supported by the unified training ablation showing that joint training outperforms modality-specific training.

The paper's strongest contributions are the LumosBench evaluation framework (which provides a new capability for the field even if the absolute scores are not yet human-calibrated) and the demonstration that training-time geometric supervision can improve lighting plausibility without inference-time overhead. The weakest aspects are the overclaimed speedup, the absence of statistical rigor, and the limited testing on complex, realistic relighting scenarios that would stress-test the approach's claimed generality.

6. Limitations and Trade-offs

Difficulty Estimation Cost Is Unaccounted for in the Headline Efficiency Numbers

The assumption or constraint. The entire LumosData pipeline—and by extension the training of UniLumos—depends on automated annotation of lighting attributes using Qwen2.5-VL, a large vision-language model. Each training sample requires the VLM to analyze an input video, classify it across six lighting dimensions, and produce structured JSON output. The paper describes this in Appendix B, Step 4, and provides the full prompt in Listing 1. Yet the computational cost of running Qwen2.5-VL on ~110K video pairs and 1.2M images is never quantified or included in any resource accounting. The paper treats this annotation cost as external to the system—something that happens before training and is therefore not part of the reported training budget (5,000 iterations on 8 H20 GPUs) or inference cost (277 seconds for 49-frame video).

The consequence. The headline claim of a 20× inference speedup (Figure 5) compares UniLumos's inference time against baselines but ignores a fundamental asymmetry: UniLumos requires structured six-dimensional captions for conditioning, while baselines like IC-Light operate on free-form text prompts. To deploy UniLumos on a new video, a user must either (a) manually specify all six lighting attributes (direction, source type, intensity, color temperature, temporal dynamics, optical phenomena), which demands expertise the paper's "naturalistic input conditions" pitch suggests is unnecessary, or (b) run Qwen2.5-VL to auto-annotate the video, incurring additional inference cost that is not included in the 277-second figure. If auto-annotation takes, say, 10–30 seconds per video on comparable hardware, the effective speedup against Light-A-Video + Wan2.1 (~917 seconds) drops from ~3.3× to ~2.9–3.2×. More importantly, this creates a chicken-and-egg problem: the model excels at controllable relighting but requires structured labels to exercise that control, and generating those labels requires a model of comparable or greater capability than the relighting model itself. The paper implicitly assumes the structured annotation is available at deployment but never specifies how or at what cost.

What evidence exists in the paper. The paper does not measure VLM annotation cost anywhere. The LumosData pipeline description (Appendix B, Step 4) states that Qwen2.5-VL is used to "analyze each input video and generate structured captions," and the ablation in Table 7 shows that "UniLumos w/o lumos captions" (trained only on vanilla scene-level captions) drops from 0.773 to 0.662 average Lumos Score, confirming that structured captions are load-bearing for performance. But the cost of obtaining those captions at scale—either during training data construction or at deployment—is never reported. The paper acknowledges a related concern in Section 3.2 (of the main paper, not the appendix): "estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity." However, this quote refers to the difficulty estimation in the referenced compute-optimal scaling work, not UniLumos—UniLumos never raises or addresses the equivalent concern for its own annotation pipeline.

Mitigation status. The paper does not acknowledge this as a limitation and proposes no mitigation. The structured annotation protocol is presented purely as a contribution, without discussion of its deployment-time cost. A partial mitigation would be to train a lightweight classifier that predicts the six lighting attributes directly from the input video without requiring a full VLM—analogous to how the compute-optimal scaling paper suggests "training models to predict difficulty directly from the question text"—but this is not explored.

The Circular Geometry Evaluation: Lotus Supervises Training and Lotus Measures Success

The assumption or constraint. The physics-plausible feedback mechanism (Section 4.1, Equation 6) trains UniLumos to minimize the L2 error between depth/normal maps estimated by a frozen Lotus model on the generated output and Lotus-estimated depth/normal maps on the reference video. The Dense L2 Error metric (Table 1, Tables 5–6), which the paper uses as its primary quantitative evidence for improved geometric consistency, is computed using the same Lotus model. This creates a circular evaluation: the model is optimized to produce outputs that Lotus evaluates favorably, and the metric that declares success is Lotus's own evaluation. This is the geometric equivalent of training a classifier to fool a specific discriminator and then using that same discriminator to certify the classifier's accuracy.

The consequence. The reported 2–2.5× reduction in Dense L2 Error (e.g., 0.147 for UniLumos vs. 0.371–0.432 for baselines in Table 1b) may partially reflect the model learning to exploit Lotus's particular failure modes rather than genuinely improving geometric consistency in a way that would generalize to other estimators or to human perception. If Lotus has systematic biases—for example, consistently underestimating depth on dark clothing or misestimating normals at occlusion boundaries—the training signal would encourage the model to produce outputs that align with those biased estimates, and the evaluation metric would reward exactly that alignment. The paper implicitly recognizes this risk in its discussion of reward over-optimization in the PRM search context (Section 5.3 of the referenced paper), but applies no analogous safeguards here. A practitioner reading the strong Dense L2 Error improvements might conclude the method achieves robust geometric consistency when it may only achieve Lotus-specific consistency.

What evidence exists in the paper. The evidence is the architecture itself: the physics loss (Equation 6) explicitly uses Lotus-generated pseudo-ground-truth maps D and N as targets, and the Dense L2 Error metric uses Lotus to compare generated and reference geometry. Nowhere does the paper validate these Lotus estimates against ground-truth geometry (e.g., from a depth sensor or a synthetic rendering engine where depth is known exactly). The StanfordOrb benchmark (Table 5) provides some independence because it uses 3D scanned objects with known geometry, but the paper still evaluates using Lotus rather than the known ground-truth depth/normals that StanfordOrb provides. The qualitative ablation in Figure 6 does show visibly improved shadow alignment when physics feedback is included, suggesting the improvement is not entirely an artifact of metric circularity—but qualitative evidence from two hand-picked examples cannot substitute for an independent quantitative evaluation.

Mitigation status. The paper does not address this limitation. A straightforward mitigation would be to evaluate Dense L2 Error using an alternative depth/normal estimator not used during training (e.g., Depth Anything v2, Marigold, or ZoeDepth for depth; a separate normal estimation model like DSINE). If the improvement persists across estimators, the circularity concern is largely resolved. For StanfordOrb specifically, the paper could compute Dense L2 Error against the known 3D geometry of the scanned objects, which would provide a fully independent ground-truth measurement. Neither evaluation is reported.

Tested Only on Simple Object-Centric Scenes; No Evidence for Complex, Multi-Source, or Outdoor Illumination

The assumption or constraint. The LumosData pipeline (Appendix B) constructs training data by extracting a single foreground subject via BiRefNet masking, applying IC-Light relighting to that subject in isolation, and placing it against a Gaussian-noise background. The conditioning signal C describes lighting as six discrete attributes, each classified into a small set of categories (e.g., Direction: {Front, Side, Back, Top, Bottom, Split, Ambient}). This construction assumes that lighting can be adequately described for an entire scene using global attributes of a single primary light source, and that the subject is a well-segmented foreground object against a simple background. The paper does not train or test on scenes with multiple interacting subjects, complex occlusions, multiple light sources with different properties, outdoor natural illumination (where light comes from the entire sky dome, not a single direction), or scenes where the background itself is an important source of illumination cues (e.g., a subject standing in a room where the background walls should reflect colored light onto the subject).

The consequence. The claimed "physical plausibility" is demonstrated only in a regime where the physical problem is substantially simplified. Real-world relighting applications—compositing an actor into a live-action plate, relighting a product in a room with mixed window light and overhead fluorescent, changing time-of-day in an outdoor scene—involve precisely the complexities that the training pipeline strips away: multiple light sources with different color temperatures, spatially varying illumination from extended sources, inter-reflections between the subject and the environment, and backgrounds that actively participate in the lighting rather than serving as neutral Gaussian placeholders. A practitioner who deploys UniLumos expecting it to handle these scenarios has no evidence from the paper to guide their expectations. The failure modes in such scenarios are predictable: the model may apply a single global lighting direction when the scene actually contains light from multiple directions, ignore the color cast that background surfaces should impose on the subject, or produce temporally inconsistent lighting when the background video contains moving light sources (e.g., a candle or a rotating beacon).

What evidence exists in the paper. All qualitative results (Figures 1, 3, 4, 8, 9, 10) and all quantitative evaluations (Tables 1, 5, 6) feature a single foreground subject (a person, a sculpture, a toy, a container) against a relatively plain or uniform background. The StanfordOrb benchmark (Table 5) uses canonical 3D objects (bunny, dragon) on plain backgrounds. The Navi benchmark (Table 6) uses everyday objects (containers, toys, mugs) likewise in isolation. No result shows UniLumos applied to: a group of people, a person interacting with furniture or another person, an outdoor scene with trees and sky, a scene with visible light sources in frame, or a scene where the background illumination should logically affect the subject's lighting. The LumosBench evaluation (Table 7) tests per-attribute controllability but still uses videos from the same object-centric distribution. The paper's Appendix E acknowledges that "UniLumos is still limited by a broader challenge—achieving physically precise and controllable relighting," but frames this as a limitation of achieving "physically quantifiable lighting outputs such as radiance or illuminance" rather than a limitation of the scene complexity regime in which the method has been validated.

Mitigation status. The paper acknowledges in Appendix E that future work "may explore finer control over lighting, including editable key lights, intensity ramps, and environmental reflections," but does not characterize the current work's scope restriction to simple object-centric scenes as a limitation per se. The training pipeline's reliance on BiRefNet for subject masking and IC-Light for degradation generation—both optimized for single-subject, relatively simple scenes—suggests that extending to complex scenes would require non-trivial modifications to the data construction pipeline, not merely scaling up the current approach.

The 20× Speedup Claim Is Best-Case Against a Non-Video Baseline and Not Matched to the Claimed Contribution

The assumption or constraint. The paper's abstract and introduction claim a "20× speedup for both image and video relighting." Figure 5 reports inference times: UniLumos at 277 seconds for 49-frame 480p video, IC-Light at 12 seconds per frame, Light-A-Video + CogVideoX at 756 seconds, Light-A-Video + Wan2.1 at 917 seconds. The 20× figure is not explicitly derived in the text. Based on the numbers, it appears to come from comparing UniLumos's per-frame cost (277/49 ≈ 5.7 seconds per frame) against IC-Light's per-frame cost (12 seconds), yielding roughly 2.1× per frame. Alternatively, comparing total video time for IC-Light applied to all 49 frames (588 seconds) against UniLumos (277 seconds) yields roughly 2.1×. The 20× figure does not correspond to any baseline for which both total time and per-frame time are reported.

The consequence. The "20×" claim shapes the paper's reception—it appears in the abstract, the introduction's contribution summary, and is the key practical takeaway for practitioners deciding whether to adopt the method. If the actual speedup against the most reasonable video relighting baseline (Light-A-Video + Wan2.1) is ~3.3× (917/277), the paper substantially overstates its efficiency advantage. A 3.3× speedup is still a meaningful engineering improvement, but it is an order of magnitude smaller than 20× and may not justify the complexity of adopting a new framework (with its structured caption requirements and training pipeline) for practitioners whose primary constraint is inference latency rather than lighting controllability. Moreover, IC-Light is an image relighting method being used per-frame for video—it was never designed, optimized, or benchmarked for video throughput—making it an inappropriate baseline for a video speedup claim. The correct baseline for video speedup is Light-A-Video, and against that baseline, the speedup is ~3×.

What evidence exists in the paper. Figure 5 reports the raw inference times. The "76×" annotation in Figure 5 is unexplained (the arithmetic 917/277 ≈ 3.3, 756/277 ≈ 2.7, and 588/277 ≈ 2.1—none of which is 76×). The paper does not provide a derivation of the 20× figure anywhere in the text. The abstract states: "delivering a 20× speedup for both image and video relighting." The conclusion repeats: "a 20× speedup for both image and video relighting." Section 5.2 states: "UniLumos completes generation over 20 times faster without sacrificing visual fidelity or physical plausibility." None of these statements specify which baseline the 20× is relative to, how it is computed, or why it differs from the ~3× implied by the reported times against video baselines. For image relighting, no inference time comparison is reported at all—Tables 1a, 5, and 6 report quality metrics but not timing—so the "20× for image relighting" half of the claim is entirely unsupported by any timing measurement in the paper.

Mitigation status. The paper does not acknowledge this as a reporting issue and makes no attempt to clarify the speedup calculation. A transparent presentation would report speedup against each baseline individually, specify whether the comparison is per-frame or per-video, and either justify the 20× figure with explicit arithmetic or revise it to match the reported measurements.

No Statistical Rigor: Single Scalar Metrics Without Variance, Confidence Intervals, or Significance Tests

The assumption or constraint. All quantitative results in the paper—Tables 1, 2, 5, 6, 7—report single scalar values per metric per method (e.g., "PSNR: 25.031," "SSIM: 0.891," "LPIPS: 0.109"). No standard deviations, confidence intervals, error bars, or statistical significance tests are reported anywhere. The test set sizes are not specified for the internal dataset; for LumosBench, 2K prompts are reported; for StanfordOrb and Navi, the full benchmark sizes are not stated in the paper. The training uses a fixed random seed by implication (no mention of multiple runs or seed variation).

The consequence. Without variance estimates, readers cannot determine whether the reported differences are statistically meaningful or within the range of sampling noise. Several comparisons involve small absolute differences that are likely within typical variance for these metrics: SSIM of 0.911 vs. 0.903 on Navi (Table 6, a difference of 0.008 on a [0,1]-bounded metric); LPIPS of 0.109 vs. 0.113 for the path consistency ablation (Table 2, a difference of 0.004); R-Motion of 1.436 vs. 1.438 for the same ablation (difference of 0.002). Without variance information, these could be noise—especially given that UniLumos occasionally performs worse than a baseline on a metric (LPIPS on Navi: 0.120 vs. 0.116 for Light-A-Video + Wan2.1, Table 6), which could either be a genuine regression or a statistically insignificant fluctuation. The lack of statistical rigor is particularly concerning for the ablations (Table 2), where the conclusions—"removing normal supervision causes a larger drop than removing depth"—rely on interpreting differences of 1.36 dB in PSNR and 0.009 in SSIM without any measure of how stable these differences are across random seeds or test subsets.

What evidence exists in the paper. The paper provides no variance information anywhere. The evaluation methodology (Section 5.1) describes the metrics and datasets but does not mention statistical protocol. The ablation study (Section 5.3) interprets rank-orderings of scalar values as conclusive without qualification. The LumosBench results (Table 7) report four significant figures (e.g., "0.893") for metrics derived from VLM classifications over 2K prompts, but no confidence interval is reported for these proportions despite the relatively modest sample size (2K prompts means a 95% confidence interval width of approximately ±2.2 percentage points for a proportion near 0.8–0.9). The training procedure (5,000 iterations, batch size 8) is described as a single run with no mention of multiple seeds or reproducibility analysis.

Mitigation status. The paper does not address this limitation. Standard practice in the field—reporting mean ± standard deviation over multiple runs or bootstrapped confidence intervals over test samples—is not followed. For a paper that introduces a new evaluation benchmark (LumosBench) and makes comparative claims (state-of-the-art, 4× improvement in related work, etc.), the absence of statistical rigor weakens the strength of those claims and makes it impossible for practitioners to assess whether the reported improvements would replicate on their own data.

The Revision Model Architecture and Training Are Not Integrated with PRM Search; Two Complementary Mechanisms Remain Studied in Isolation

Note: This limitation addresses a gap in the paper's architectural integration. The prior sections already established that physics feedback and path consistency are synergistic; this limitation concerns the absence of combining the core generative capabilities—structured control and geometric supervision—with the broader ecosystem of test-time techniques that could further amplify their benefits.

The assumption or constraint. UniLumos improves the proposal distribution—the model generates geometrically consistent relit outputs from structured conditioning—but does not incorporate any form of verifier-guided selection, iterative refinement, or test-time search over the generated outputs. The model produces a single output given a conditioning signal; there is no mechanism for generating multiple candidates and selecting the best, for using a learned verifier to score geometric consistency at test time, or for iteratively revising an output based on self-critique. This stands in contrast to the paper's own related work discussion of feedback learning (Section 2), which cites methods like DRaFT and InstructVideo that use reward models to guide generation, and to the compute-optimal scaling paradigm (discussed extensively in the prior sections of this analysis) that systematically allocates test-time compute between parallel sampling and sequential refinement.

The consequence. UniLumos leaves on the table a potentially large class of gains that come from test-time optimization rather than training-time regularization. The physics feedback improves geometric consistency during training, but at inference, the model still operates in a single forward pass. If the model produces a geometrically implausible output for a challenging input—perhaps because the scene is outside the training distribution or the lighting condition is unusual—there is no recourse. A verifier-based selection mechanism (e.g., generate 8 candidates, use Lotus to score each for geometric consistency, select the best) could rescue such failures, and iterative revision (generate an initial output, estimate its geometry, feed the errors back as conditioning for a second pass) could refine imprecise outputs. Neither is explored. The paper's framing suggests that training-time regularization is sufficient to bake geometric consistency into the model weights, but the ablation results—Dense L2 Error of 0.147 even with full physics feedback (Table 2), compared to 0.297 without—show that residual geometric errors remain, and test-time optimization could potentially close that remaining gap.

What evidence exists in the paper. The paper does not experiment with test-time optimization, verifier-guided selection, or iterative revision. Section 2 discusses feedback learning in generative models but frames it as a training-time paradigm. The inference process is described (Section 4.3) as few-step sampling using the learned velocity field with no post-processing or selection. The efficiency results (Figure 5) emphasize that UniLumos is fast because inference is "geometry-free"—no external estimators run at test time—but this design choice also means there is no test-time geometric verification. The qualitative results (Figures 3, 4) show UniLumos producing good outputs on the selected examples, but without a verifier, there is no guarantee that outputs are consistently good across the full test distribution—the Dense L2 Error of 0.147 (Table 1b) indicates that geometric errors persist at some level.

Mitigation status. The paper does not discuss test-time optimization or verifier-guided selection as limitations or as future work. Appendix E suggests future work on "finer control over lighting, including editable key lights, intensity ramps, and environmental reflections," but does not mention combining the trained model with verifier-based search or iterative refinement. The related work section cites DRaFT (Clark et al., 2023) and InstructVideo (Yuan et al., 2024) as examples of reward-guided generation, and the compute-optimal scaling context (imported from the prior sections of this analysis) demonstrates that adaptive test-time allocation yields 4× efficiency gains in related domains—but UniLumos does not connect to either line of work at the architectural level. This is a missed opportunity rather than a failure of the existing method, but it represents a significant limitation on the maximum achievable quality from the UniLumos framework: the model is capped at what a single forward pass can produce, with no mechanism to invest additional computation for harder cases.

7. Implications and Future Directions

How This Work Changes the Landscape

UniLumos represents a methodological pivot rather than a paradigm shift—it does not invent a new generative architecture or a new training objective, but it demonstrates that a specific, previously underexplored combination of existing techniques (training-time geometric supervision + path consistency + structured semantic decomposition) can close a persistent gap between semantic plausibility and physical correctness in relighting. The magnitude of the contribution is best characterized as opening a new design axis—the choice of what kind of training signal to provide beyond pixel-space or latent-space reconstruction—rather than rendering prior approaches obsolete.

The key methodological shift is the demonstration that physical constraints can be effectively encoded as training-time regularizers rather than as architectural inductive biases or inference-time post-processing. Prior to this work, the dominant strategies for improving physical plausibility in generative vision models were either (a) conditioning the model on explicit geometric representations (depth maps, normal maps, segmentation masks) at both training and inference, which increases inference cost and creates dependencies on external estimators, or (b) using differentiable rendering or physics-based losses that require ground-truth geometry and material properties, which limits applicability to synthetic or carefully captured data. UniLumos shows a third path: extract geometric signals from the model's own outputs using off-the-shelf estimators during training, penalize inconsistencies, and let the model internalize the constraints into its weights. At inference, no geometric estimation occurs—the model produces physically consistent outputs from RGB and text alone.

This finding has implications beyond relighting. Any generative vision task where outputs must obey geometric, photometric, or physical invariants—novel view synthesis (output must be consistent with multi-view geometry), video prediction (output must obey temporal continuity of object motion), image compositing (foreground must have consistent lighting with background), texture synthesis for 3D assets (must be seamless under varying lighting)—could potentially benefit from analogous training-time feedback loops. The paper provides a concrete template: identify a geometric invariant that should be preserved or achieved by the generation, find or train an estimator that measures that invariant from RGB, and add a loss term computed on few-step decoded outputs.

The paper reconciles two previously conflicting research threads. On one side are methods like IC-Light and SynthLight that achieve flexible, prompt-driven relighting by operating purely in latent space, sacrificing physical correctness for generality and ease of use. On the other side are inverse rendering pipelines that achieve physically accurate results through explicit scene decomposition but require complex inputs and fail on in-the-wild data. UniLumos demonstrates that these are not fundamentally in tension—the generative flexibility of latent-space models can be preserved while adding physical constraints as a training signal that does not complicate the user-facing interface or the inference pipeline. The structured caption protocol further bridges this gap by providing a user-friendly abstraction (natural language categories like "Front Light, Warm Tone") that maps to physically meaningful dimensions, giving non-expert users access to physically grounded control without requiring them to specify radiance fields or spherical harmonics.

The work also clarifies the role of model scale in physical reasoning. The LumosBench results (Table 7) show that a 1.3B parameter model trained with structured geometric supervision (UniLumos, Lumos Score 0.773) can surpass a ~11× larger general-purpose model without such supervision (Wan2.1-14B, Lumos Score 0.735) on lighting controllability. This suggests that physical reasoning in generative models is not purely an emergent property of scale—it can be efficiently taught through targeted supervision, and the returns to scale for this specific capability may diminish faster than for general visual quality. This finding mirrors the pattern observed in the compute-optimal scaling analysis (prior sections) where test-time compute with a smaller model can outperform a ~14× larger model on problems within its capability range. In both cases, the implication is that investing in better training signals (whether at training time or inference time) can substitute for model scale when the target capability is well-defined and measurable.

Research directions that become more attractive include: training-time physical feedback loops for any generative vision task with geometric invariants; structured semantic decomposition as a general strategy for improving controllability in generative models; and the use of pre-trained estimators (depth, normal, segmentation, optical flow) as supervision sources rather than as inference-time inputs. Research directions that become less attractive include: purely latent-space methods that make no attempt at physical grounding (the paper shows their controllability is substantially worse even when visual quality appears similar); and methods that rely on conditioning with geometric representations at inference time for applications where latency matters (UniLumos demonstrates that these constraints can be internalized during training).

Follow-Up Research This Work Enables

Cross-estimator validation of geometric consistency. The paper's central claim—that physics feedback improves genuine geometric consistency, not just Lotus-specific consistency—remains unverified due to the circular evaluation (Lotus supervises training, Lotus measures success). A direct follow-up would evaluate UniLumos outputs using multiple independent depth and normal estimators not seen during training: Depth Anything v2, Marigold, ZoeDepth for depth; DSINE or a separate normal prediction model for normals. If the Dense L2 Error reduction persists across estimators, the circularity concern is resolved and the claim of genuine geometric improvement is substantially strengthened. Conversely, if the improvement vanishes or reverses under alternative estimators, this reveals that UniLumos has overfit to Lotus-specific features—a finding that would refine our understanding of when training-time estimator-based feedback succeeds versus when it produces metric-specific artifacts. StanfordOrb provides particularly clean ground truth since the 3D geometry is known from scanning; computing depth/normal error against this ground truth (rather than against Lotus estimates) would provide the strongest possible test, and the paper already reports StanfordOrb results (Table 5) but omits this evaluation.

Difficulty-adaptive physics feedback scheduling. The paper's selective optimization strategy (Algorithm 1) applies physics feedback to 40% of training samples uniformly at random. However, not all training samples benefit equally from geometric supervision—simple scenes with clear geometry may already be handled well by the base flow-matching loss, while geometrically complex scenes (subjects in unusual poses, objects with thin structures, scenes with ambiguous depth cues) likely benefit more. A natural extension would be to dynamically allocate the physics feedback budget based on estimated sample difficulty, analogous to the compute-optimal test-time scaling studied in the prior sections of this analysis. Difficulty could be estimated online during training: if the Lotus depth/normal consistency error on a given sample is high, that sample receives more physics feedback in subsequent epochs; if the error is already low, the budget shifts to harder samples. This would test whether the 40% fixed allocation is optimal or whether difficulty-adaptive scheduling yields better geometric consistency for the same training cost. The paper's own LumosBench taxonomy (six lighting dimensions) provides a natural framework for measuring whether adaptive scheduling improves performance differentially across dimensions—does it help more with Optical Phenomena (the weakest dimension at 0.592) than with Direction (the strongest at 0.893)?

Iterative refinement with geometric self-critique. UniLumos currently produces a single output per input; there is no mechanism for the model to examine its own output, detect geometric inconsistencies, and refine them. A natural extension—directly inspired by the revision model in the compute-optimal scaling analysis—would be to train an iterative refinement stage where the model conditions on its own previous output and Lotus-estimated geometric errors. The training data construction would follow the same pattern as the revision model in prior sections: generate an initial relit output, estimate depth and normal errors against the reference, and fine-tune the model on multi-turn trajectories where the initial output (with geometric errors) is followed by a corrected output. At inference, this would enable a compute-quality tradeoff: fast single-pass generation for easy scenes, multi-pass refinement for geometrically challenging ones. The key measurement would be whether Dense L2 Error decreases per refinement step and how many steps are needed to saturate. The 38% correct-to-incorrect reversion rate documented in the prior revision model analysis would be a risk to monitor—does refinement sometimes degrade geometrically correct outputs?

Complex scene stress-testing with multi-source illumination. The paper's evaluation is limited to simple object-centric scenes with a single foreground subject and (implicitly) a single dominant light source. A critical stress-test would evaluate UniLumos on scenes with multiple light sources, complex occlusions, and inter-reflections. A concrete experimental design: construct a test set of synthetic indoor scenes (using Blender or a similar renderer) where ground-truth geometry and per-light-source contributions are known. Scenes would systematically vary the number of light sources (1, 2, 3, 4), the presence of colored walls (which cause indirect colored illumination on subjects), and the presence of occluding objects between the light and subject. If UniLumos degrades substantially with multiple light sources—producing lighting that corresponds to a single "average" direction rather than respecting multiple sources—this reveals a fundamental limitation of the six-dimensional annotation protocol, which describes lighting globally rather than per-source. If it handles multiple sources well, this suggests the geometric feedback has taught the model to reason about light transport more generally than the annotation categories imply. The LumosBench evaluation framework could be extended with new dimensions for multi-source complexity and indirect illumination accuracy.

Structured annotation across modalities and domains. The six-dimensional annotation protocol is currently specific to relighting, but the methodology—defining an ontology of independent semantic axes, generating structured annotations using VLMs, and evaluating per-axis controllability—is domain-agnostic. A natural extension would apply this template to other generative control tasks: facial expression editing (decomposing emotion into valence, arousal, action unit intensity), camera control in video generation (decomposing into azimuth, elevation, field of view, camera motion type), or style transfer (decomposing into color palette, texture granularity, brush stroke type, level of abstraction). For each domain, the key question is whether structured annotation during training improves per-axis controllability compared to free-form text conditioning, and whether the improvement is uniform across dimensions or concentrated on those that are most entangled in natural language. The paper's finding that Intensity shows the largest drop when structured captions are removed (0.832 → 0.529, Table 7) suggests that dimensions with continuous underlying variation but discrete natural-language categories are the strongest candidates for structured annotation.

Lotus-free physics feedback via self-supervised geometric consistency. The current approach depends on Lotus, a pre-trained external model, for both training supervision and evaluation. This creates a dependency that may limit applicability to domains where no high-quality dense estimator exists (e.g., medical imaging, microscopy, specialized industrial inspection). A more fundamental follow-up would explore self-supervised geometric consistency that does not require an external estimator. For video relighting specifically, temporal consistency of depth and normals across frames could serve as a self-supervised signal: if the model produces relit videos where estimated depth/normals are stable across frames (even if the absolute estimates are biased), this provides a geometric regularizer without external ground truth. Alternatively, multi-view consistency—relighting the same subject from different viewpoints should produce consistent geometry—could serve a similar role. This would test whether the geometric benefits of physics feedback are fundamentally tied to the quality of the external estimator or can be achieved through weaker but more general self-supervised signals.

Practical Applications and Downstream Use Cases

Real-time video conferencing with virtual background harmonization. In video conferencing applications where users apply virtual backgrounds or background blur, the foreground subject's lighting often mismatches the synthetic background—a person lit by warm indoor lamps appears unnatural against a sunny beach background, or harsh office lighting creates a visible cutout effect against a dark virtual environment. UniLumos's 277-second processing time for 49-frame 480p video (~5.7 seconds per frame) is currently too slow for real-time conferencing, but the few-step generation capability and the 3.3× speedup over Light-A-Video suggest that with model optimization (distillation, quantization, reduced resolution) and the trend of improving GPU hardware, near-real-time harmonization at 15–30 fps for video call resolutions (360p–720p) is a plausible near-term target. The structured caption protocol is particularly well-suited for this use case because virtual background applications typically know the intended lighting context (the background image or video) and could auto-populate lighting attributes from the background's metadata. The key metric for this application would be perceptual naturalness at video call bitrates, not PSNR against a reference—a user study comparing UniLumos-harmonized conferencing against no harmonization and against IC-Light would be the relevant evaluation.

E-commerce product visualization with lighting customization. Online retailers increasingly offer product visualization where customers can see items (furniture, clothing, electronics) in different contexts. UniLumos's ability to relight general objects (validated on Navi's everyday objects and StanfordOrb's sculptures, Tables 5–6) while preserving fine structural detail makes it a candidate for lighting-customizable product imagery. A customer could upload a photo of their room and see how a lamp or a piece of furniture would look under their actual lighting conditions, or toggle between "warm evening," "cool daylight," and "neutral studio" lighting to assess appearance. The structured six-dimensional protocol provides a natural customer-facing interface: rather than sliders for abstract parameters, customers could select from discrete, interpretable options ("Direction: Side Light," "Color Temperature: Warm Tone"). The model's generalization to object-centric scenes outside its human-centric training distribution (Tables 5–6) is critical here, and further testing on product-specific categories (textured fabrics, reflective metals, transparent glass) would be needed. The 20× speedup claim, even if reduced to the more realistic 3× against video baselines, makes batch processing of product catalogs economically viable—a catalog of 10,000 products with 5 lighting variations each (50,000 relit images) would take approximately 1.5 GPU-days on the reported hardware configuration.

Film and video post-production pre-visualization. In professional film production, lighting decisions are typically made on set or in pre-visualization with 3D assets. UniLumos offers a complementary capability: rapid pre-visualization of lighting changes directly on captured footage without requiring 3D scene reconstruction or manual rotoscoping. A cinematographer could take a frame from a rehearsal, specify target lighting attributes through the structured protocol ("Back Light, Natural Light, Moderate, Warm Tone, Static"), and see a physically plausible approximation of that lighting on the actual actor and set within minutes rather than hours. While the quality is not yet production-ready (PSNR of ~25 dB indicates visible artifacts at full resolution), the speed and the geometric consistency improvements over IC-Light (Dense L2 Error of 0.147 vs. 0.432) make it suitable for early-stage creative decisions where rapid iteration matters more than final fidelity. The background-conditioned relighting mode (Figure 4) is particularly relevant here—a reference video of the desired lighting environment can drive the relighting, allowing cinematographers to "borrow" lighting from reference footage shot at a different time or location. The key practical limitation for this use case is resolution: the current model operates at 480p, while film pre-visualization typically requires at least 1080p, suggesting that scaling the architecture to higher resolutions while maintaining the speed advantage is a prerequisite for adoption.

Synthetic data augmentation for robust vision models. Training computer vision models (object detectors, segmentation models, depth estimators) that are robust to lighting variation requires diverse illumination conditions in the training data. Real-world data collection across lighting conditions is expensive and weather-dependent. UniLumos enables lighting augmentation of existing datasets: given a dataset of images or videos captured under a single lighting condition, automatically generate variants under the six lighting dimensions (different directions, intensities, color temperatures, etc.) to expand the training distribution. Unlike simple photometric augmentations (brightness/contrast adjustment), UniLumos's physics feedback produces lighting variations that respect scene geometry—shadows move appropriately, highlights shift to geometrically correct surface orientations—making the augmented data more representative of real lighting variation. The 1.2M additional relit images generated during LumosData construction demonstrate the pipeline's scalability. The key metric for this application would be downstream task performance: does a detector trained with UniLumos-augmented data outperform one trained with standard photometric augmentations when tested on naturally varying lighting conditions? The LumosBench taxonomy provides a framework for systematically evaluating which lighting dimensions matter most for downstream robustness.