UniUGP: Unifying Understanding, Generation, and Planing For End-to-end Autonomous Driving

ArXiv: 2512.09864

🎯 Pitch

Autonomous driving systems can now watch their own future to drive better: UniUGP shows that training a model to generate what happens next in video—not just reason about it in text—boosts small-obstacle recognition by 5.6 percentage points and cuts planning error by 18%, because seeing a distant hazard is only half the battle; anticipating how it will physically unfold is what prevents the crash.

---

1. Executive Summary

This paper introduces UniUGP, a unified Understanding-Generation-Planning framework for end-to-end autonomous driving that synergizes three complementary capabilities through a hybrid expert architecture — scene reasoning via a vision-language model (chain-of-thought explanations), trajectory planning via flow matching (physically consistent future waypoints), and future video generation via a DiT-based world model (coherent visual predictions). Built on Qwen2.5-VL-3B and trained through a four-stage curriculum across over ten diverse AD datasets, UniUGP achieves state-of-the-art performance across understanding (89.3% small-object recognition accuracy), planning (1.23m average L2 displacement error on nuScenes with only front-camera input), and generation (7.4 FID on future frame prediction), while establishing that integrating causal visual dynamics from a world model substantively improves a VLA model's reasoning and decision-making — demonstrating, for instance, that ablated removal of the generation expert degrades small-object understanding from 89.3% to 83.7% and planning L2 error from 1.45m to 1.72m, confirming that the unified design is effective only when the model can leverage visual causal inference to anticipate distant hazards that pure language reasoning would overlook.

2. Context and Motivation

The Core Problem: Autonomous Driving Systems Fail in Long-Tail Scenarios Due to Fragmented Capabilities

The fundamental challenge this paper tackles is that current autonomous driving (AD) systems — despite impressive progress in structured environments — catastrophically degrade in long-tail, safety-critical situations because they lack two intertwined capabilities: rich world knowledge for semantic reasoning about rare events, and robust visual dynamics modeling for anticipating how those events will unfold physically. A car that can navigate a sunny highway flawlessly may still fail to recognize a fallen construction cone as a hazard requiring immediate braking, or fail to predict that a pedestrian near a construction zone will step into the roadway. These failures happen not because any single component is broken, but because the system cannot connect what it sees (perception) to why it matters (reasoning) to what will happen next (prediction) to what it should do (planning) in a causally coherent way.

This gap is significant for at least three reasons the paper emphasizes throughout Section 1 and Section 2:

• Safety-critical deployment: Long-tail scenarios — construction zones, small obstacles on the road, unusual pedestrian behaviors, rare weather conditions — are precisely where accidents happen. A system that works 99.9% of the time but fails on the 0.1% that involves a child chasing a ball into the street is not deployable. The paper explicitly motivates its benchmark construction around "low-probability-but-high-risk scenarios" that "occur in less than 0.003% of daily driving" (Section 4.1, describing the Waymo Open Dataset Long-tail End-to-End Driving split).

• Data inefficiency: The world contains enormous quantities of unlabeled driving video — dashcam footage, fleet data, publicly released datasets like nuScenes' 193,082 unannotated samples. Prior VLA-based methods "were unable to fully utilize the large number of unlabeled driving videos, which limited their ability to learn visual causal reasoning from large-scale datasets" (Section 1). This is a massive missed opportunity: if a model could learn the physics of driving — how objects move, how scenes evolve — simply by watching video, it would acquire a form of commonsense visual reasoning that no amount of text-based QA training can provide.

• Interpretability and trust: End-to-end models that output steering angles and acceleration values as black-box predictions are fundamentally untrustworthy for safety-critical applications. If the car brakes unexpectedly, the engineer (and the passenger, and the regulator) needs to know why — was it the red light? The pedestrian? A false positive detection? The paper identifies that existing methods lack "CoT reasoning, natural language interaction, and diffusion-based planning, while failing to leverage large-scale unlabeled data... limiting both their interpretability and generalization potential" (Section 2.3).

Two Competing Paradigms, Each with Fundamental Weaknesses

The paper situates itself at the intersection of two research threads that have developed largely in parallel, each addressing one half of the problem but structurally incapable of addressing the other.

Thread 1: Vision-Language-Action (VLA) Models. The dominant approach in recent end-to-end AD research is to leverage pre-trained vision-language models (VLMs) to directly map pixel inputs and language instructions to vehicle control commands. This paradigm, exemplified by methods like Impromptu VLA (Chi et al., 2025), AutoVLA (Zhou et al., 2025), ReCogDrive (Li et al., 2025), and ORION (Fu et al., 2025), offers compelling advantages: it simplifies system architecture by eliminating hand-designed modular pipelines (perception → prediction → planning), minimizes information loss from intermediate representations, and crucially, brings the VLM's world knowledge to bear on driving decisions. A VLM pre-trained on internet-scale data has seen millions of images of construction zones, emergency vehicles, and unusual road configurations, and can reason about them in natural language.

The limitations of this approach, as the paper identifies in Section 1 and Section 2.1, are structural:

• No visual causal reasoning from video: VLAs are trained on static images or short clips with text annotations (QA pairs, trajectory labels). They never learn to predict what happens next in a video from unlabeled sequences. This means they lack an internal model of object dynamics, occlusion, motion continuity, and the physical consequences of actions — what the paper calls "visual causal reasoning." A VLA might recognize that there is a ball in the road, but it cannot anticipate from video dynamics alone that a child will likely follow.

• Training data bottleneck: VLA methods require expensive, manually annotated data — question-answer pairs, trajectory labels, reasoning chains. The paper notes that prior VLA benchmarks "emphasize structured scenes and simulator-based evaluations, which overlook the challenges of long-tail driving events" (Section 3.1). Real-world long-tail annotation is scarce because it's exactly these rare events that are hardest to label at scale.

Thread 2: World Models for Autonomous Driving. In parallel, another line of work has focused on learning predictive models of the driving environment — so-called world models — that take historical observations and forecast future states, whether as occupancy grids (OccWorld; Zheng et al., 2024), future video frames (DriveDreamer; Wang et al., 2024; GAIA-1; Hu et al., 2023), or combined representations (Epona; Zhang et al., 2025). The key insight is that predicting the next frame forces the model to learn visual causality: objects move with inertia, pedestrians don't teleport, occluded regions can be anticipated, braking distances depend on speed and road conditions.

The paper acknowledges the strengths of this paradigm in Section 2.2 and Section 1:

"The world model can learn visual causal reasoning by predicting the next frame of the video, which has been proven to be helpful in achieving the final end-to-end AD."

However, world models have their own critical limitations:

• No semantic reasoning or world knowledge: A world model can predict that pixels will change in a certain way when a traffic light turns red, but it doesn't know why red means stop, or that an ambulance with flashing lights takes priority, or that a police officer's hand gesture overrides a green light. These are semantic conventions that require language-grounded knowledge, not just visual dynamics.

• No language interaction: World models operate purely in visual/geometric space. They cannot take natural language instructions ("turn left at the next intersection"), cannot explain their predictions in human-interpretable terms, and cannot engage in dialogue about driving decisions.

• Limited generalization from single-dataset training: The paper observes that "most methods are trained and evaluated on limited single-dataset scenarios, restricting their generalization ability" (Section 2.2). A world model trained only on nuScenes learns nuScenes-specific dynamics; it doesn't automatically transfer to the construction zones of StreetHazards or the accident sequences of DADA2000.

The Gap: No Unified Model That Combines Semantic Reasoning, Visual Dynamics, and Physical Planning

The paper's central diagnosis, captured in Table 1 and elaborated in Section 2.3, is that no existing method simultaneously provides chain-of-thought reasoning, natural language interaction, diffusion-based continuous trajectory generation, and future video prediction — the four capabilities the paper argues are necessary for robust long-tail driving.

Table 1 is particularly revealing in its taxonomy. Existing unified models that attempt to bridge these capabilities — Doe-1 (Zheng et al., 2024), Occ-LLM (Xu et al., 2025), OccLlama (Wei et al., 2024), HERMES (Zhou et al., 2025), FSDrive (Zeng et al., 2025) — all fall short in critical dimensions:

• Doe-1 and FSDrive unify understanding and video generation but output discrete text actions, not continuous trajectories — meaning they cannot produce the smooth, physically realistic motion needed for actual vehicle control.
• Occ-LLM, OccLlama, and HERMES integrate occupancy prediction with language understanding but generate only text-based outputs, lacking both continuous trajectory generation and video prediction.
• None of the existing unified models provide chain-of-thought reasoning or instruction-following capability, which the paper argues is essential for interpretability and user trust.

The paper's critique of existing VLA methods crystallizes around a specific tension: methods like AutoVLA and ReCogDrive add reasoning and diffusion-based planning to VLAs, but they "cannot leverage unlabeled videos for visual causal learning" (Section 1). Conversely, world models like Epona unify video generation and planning through autoregressive diffusion, but they lack "reasoning capabilities from large language models" (Section 1, Abstract). The two paradigms have been developed in isolation, each solving the problem the other cannot, with no framework for combining them.

How This Paper Positions Itself

UniUGP positions itself as the first framework to fully integrate a VLM's semantic reasoning with a world model's visual dynamic prediction within a single, jointly-trained architecture that outputs interpretable reasoning, continuous trajectories, and future videos (Section 1, Contributions; Table 1, bottom row). This is not merely combining two existing models — it requires solving several architectural and training challenges that the paper identifies explicitly (Section 1):

1. How to efficiently establish a unified model to fully utilize the pre-trained VLM and world model. The answer is the hybrid expert architecture (Section 3.2), where the understanding expert (Qwen2.5-VL) and planning expert (flow matching) form a Mixture-of-Transformers (MoT) with shared self-attention but modality-specific feed-forward networks, while the generation expert (Wan2.1 DiT) is cascaded as a separate module conditioned on both semantic hidden states from the VLM and action embeddings from the planning expert. This design allows each expert to leverage pre-trained weights independently while interacting through carefully designed cross-modal conditioning signals, rather than attempting to fine-tune everything from scratch in a monolithic model.

2. How to effectively and repeatedly utilize various driving data (VQA pairs, video trajectory pairs, etc.). The answer is the four-stage training curriculum (Section 3.3), which sequentially builds scene understanding (Stage 1, on long-tail perception data), visual dynamics and planning (Stage 2, on driving video with trajectory labels), text-based reasoning (Stage 3, on annotated CoT data), and finally multi-capability fusion (Stage 4, on a mixture of all prior data at a fixed 0.1:0.4:0.5 ratio). This progressive training strategy addresses a fundamental challenge: if you train all capabilities simultaneously from scratch, the model struggles to learn any single one well, because the loss signals from language modeling, flow matching, and video denoising compete rather than cooperate.

3. How to evaluate the unified model, especially in complex scenarios. The answer is the custom long-tail benchmark described in Section 3.1 and evaluated in Section 4, which spans perception (small objects, accident relationships, anomaly prediction), chain-of-thought reasoning (GPT and BLEU scores against reference explanations), planning (L2 displacement error at 3 seconds), and instruction following (trajectory consistency with high-level commands like "turn left"). The paper constructs these evaluation datasets specifically because "prior benchmarks emphasize structured scenes and simulator-based evaluations, which overlook the challenges of long-tail driving events" (Section 3.1).

The paper also addresses a subtle but important epistemological question: why should adding a video generation capability improve planning and understanding? The implicit argument, made explicit in the ablation studies and qualitative analysis (Figure 3, Table 3), is that training a model to generate future frames forces it to learn visual causal inference — the ability to predict that a small, distant object will become a large, close obstacle in the next few seconds, or that a pedestrian's current trajectory will intersect the ego-vehicle's path. This causal understanding then transfers back to the VLM's reasoning: as the qualitative example in Figure 3 shows, the full UniUGP model produces reasoning that mentions "a construction area ahead, with workers and cone markers" and concludes "the vehicle may slow down to avoid collisions," while the ablated version without the generation expert produces generic advice about "observing road conditions" and "driving at the prescribed speed steadily" — competent but unspecific, missing the causal link between the visual observation and the required action.

This represents a conceptual advance over prior work that treated video generation as an end in itself (a visualization tool) rather than as a training signal for causal understanding. The paper's key insight is that the process of learning to generate future frames — not the generated frames themselves — is what improves driving performance, by forcing the model's internal representations to capture object dynamics, occlusion relationships, and the physical consequences of different trajectories. This is why the generation expert can be disabled on mobile devices at inference time "without compromising the performances of the former experts" (Section 3.2): the causal knowledge has already been distilled into the understanding and planning experts during training.

3. Technical Approach

3.1 Reader Orientation

UniUGP is a neural architecture that takes multi-frame camera images and natural language instructions as input and produces three coordinated outputs — a chain-of-thought explanation of what is happening and why, a physically feasible future trajectory for the ego-vehicle, and a coherent video prediction of what the scene will look like — all generated by a single model with three specialized sub-networks that share information through carefully designed cross-modal connections. The problem it solves is that existing autonomous driving models either understand language but cannot predict visual dynamics (VLA models), or predict visual dynamics but cannot reason in language (world models), and neither type alone handles long-tail safety-critical scenarios where connecting semantic understanding to physical causality is essential; UniUGP solves this by training a vision-language expert, a planning expert, and a generation expert together through a staged curriculum where each expert's internal representations condition the others, so that learning to predict future frames improves the model's ability to recognise hazards and plan safe trajectories.

3.2 Big-Picture Architecture (Diagram in Words)

The system has three major components arranged in a hybrid expert configuration:

1. Understanding Expert (Qwen2.5-VL-3B) — a pre-trained vision-language model that takes text instructions (tokenized into discrete tokens) and history images (encoded by a ViT into continuous embeddings) and produces two things: (a) next-token predictions for chain-of-thought reasoning text, and (b) a sequence of hidden states $h^{\text{und}}$ that encode the fused semantic-visual understanding of the scene. This expert uses standard autoregressive language modeling as its training objective.

2. Planning Expert (Flow Matching Head) — a diffusion-style module that takes the understanding expert's hidden states as context and generates future ego-vehicle trajectories (sequences of $(x, y)$ waypoints in the ego-centric coordinate frame) by learning to reverse a noise-adding process. The planning expert shares multi-head self-attention layers with the understanding expert (forming a Mixture-of-Transformers) but has its own feed-forward networks, allowing it to specialise in continuous motion generation while still attending to the semantic context.

3. Generation Expert (Wan2.1 DiT-based Video Decoder) — a cascaded video generation model that takes VAE-encoded history frames, noised future frames, and conditioning signals (concatenated understanding hidden states and action embeddings from the planning expert) and produces denoised future video frames through a flow matching process. The generation expert is serial: it receives outputs from the other two experts but does not feed information back to them during inference.

Information flows as follows: Text instructions and images enter the system → the ViT and text tokenizer produce aligned multimodal tokens → the understanding and planning experts process these through shared self-attention layers, with the understanding expert producing language outputs and the planning expert producing trajectory waypoints → the understanding expert's hidden states and the planning expert's action embeddings are concatenated and fed as conditioning to the generation expert → the generation expert denoises future video frames conditioned on this semantic and physical context → the system outputs CoT text, trajectory waypoints, and future video frames simultaneously.

A critical architectural detail is that the generation expert can be disabled at inference time on resource-constrained platforms without degrading the other experts' performance, because the causal knowledge learned during video generation training has already been distilled into the understanding and planning experts' weights through the shared training process.

3.3 Roadmap for the Deep Dive

• First, the custom long-tail dataset construction (Section 3.1), since the entire training and evaluation framework depends on having data that specifically tests perception, causal reasoning, planning, and instruction following in challenging scenarios — without this data, the model cannot learn the capabilities the paper claims.
• Second, the understanding and planning experts and their Mixture-of-Transformers interaction (Section 3.2), because these two components form the core reasoning-and-action pipeline that all other components connect to, and their architecture determines what information is available to condition the generation expert.
• Third, the flow matching formulation for trajectory generation (Equations 1, 8), since the planning expert's training objective and inference procedure are different from standard regression or classification and need careful explanation.
• Fourth, the generation expert and its conditioning mechanism (Equations 9, 10, 11), because the paper's key claim is that learning to generate future video improves planning and understanding — understanding exactly what signals the generation expert receives from the other experts and during training versus inference is essential to evaluating this claim.
• Fifth, the four-stage training curriculum (Section 3.3, Equation 12), because the staged approach is the paper's solution to the problem of how to combine heterogenous datasets and objectives without destructive interference, and the specific dataset ratios and component freeze/unfreeze schedule determine what the model actually learns at each stage.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and architecture paper whose core idea is that training a unified model with three specialised experts — one for semantic understanding, one for continuous trajectory planning, and one for future video generation — through a staged curriculum on diverse datasets produces mutual enhancement across capabilities, with video generation training acting as a visual causal reasoning signal that improves the understanding and planning experts even when the generation expert is not used at inference.

---

Custom Long-Tail Driving Dataset Construction

The paper constructs multiple specialised datasets because existing benchmarks (ImpromptuVLA, DriveLM) emphasise structured urban scenes and simulator-based evaluations that do not adequately test a model's ability to handle rare, safety-critical events. The authors collect videos from six sources: Waymo-E2E (4,021 segments capturing events occurring in less than 0.003% of daily driving), DADA2000 (driver attention during accident scenarios), Lost and Found (LaF, small road hazards), StreetHazards (StHa, out-of-distribution objects), SOM (LiDAR-guided small obstacle segmentation), and AADV (anticipating accidents in dashcam videos). These raw videos are then processed into four task categories through a combination of automated VLM-based annotation and manual calibration.

Task 1: Perception and Understanding. For small long-tailed objects, the paper uses segmentation map labels from the source datasets to determine whether small tail objects are present, then constructs binary true/false questions with varied phrasings to improve generalisation. Example phrasings include "Any small long-tailed objects in the driving video?" and "Are there tiny long-tailed items in the driving clip?" — the variation in wording (objects/items/things, video/clip/footage) prevents the model from overfitting to specific question templates. The answers are derived from the ground-truth segmentation labels, making this an objective binary classification task.

For long-tailed accident prediction, the paper uses the source datasets' annotations of whether a video contains an abnormal traffic event at specific timestamps to construct binary true/false questions such as "Might a traffic accident take place in this driving situation?" and "Is there a possibility that a traffic accident will happen here?"

For long-tailed accident relationship, the paper constructs multiple-choice questions where the model must identify which entities are involved in an accident. The procedure, detailed in Listing 3 of the Appendix, works as follows: given a correct answer string and a candidate pool of possible answers, the system filters out the correct answer from the candidate pool to form distractor candidates, randomly selects between 2-option and 4-option formats, samples the required number of distractors (supplementing with random choices from the distractor pool if insufficient distinct distractors exist), shuffles all options and maps them to letter labels (A/B or A/B/C/D), then generates the question using the template "Which of the following describes the current situation? [A. option_a, B. option_b, ...]". This procedure ensures balanced difficulty — the model cannot exploit answer position biases or trivial elimination strategies.

Task 2: Causal Chain-of-Thought Reasoning. This is the most sophisticated annotation task. For sequences where the future driving outcome is observable (i.e., the dataset contains both historical frames and future frames with ground-truth ego trajectories), the authors construct QA pairs where the answer is a structured multi-step chain of thought. Critically, the reasoning is not generated from the question alone — it is generated by providing an advanced VLM with the future images, future waypoints, past waypoints, past velocity, and past acceleration, along with explicit instructions to use future information only for confirmation, not as the basis for reasoning. The full prompt template (Listing 4, Appendix) specifies four structured output steps:

• Step 1 (Scene Analysis): Describe the overall traffic scene, including traffic lights, road geometry, lane markings, signs, crosswalks, construction areas, and temporary road structures.
• Step 2 (Key Object Identification): Identify up to three important or rare objects relevant for planning, with descriptions of their attributes, their impact on the ego-vehicle, and their significance for planning decisions. This explicitly includes rare objects such as traffic enforcers, construction workers, police officers, emergency responders, traffic cones, road barriers, parked delivery trucks, fallen trees, construction machinery, missing lane lines, misaligned curbs, temporary detours, and unusual entry points.
• Step 3 (Intention Inference): Based on the current and future images, the ego-vehicle's past state (velocity, acceleration, waypoints), and the future waypoints of the ego-vehicle, infer the possible future movement of each recognized object and how these objects affect the future driving of the ego-vehicle.
• Step 4 (Action Decision): Issue a command request for the future trajectory (e.g., "Please slow down and turn left") and provide a detailed chain-of-thought for this decision using step-by-step reasoning based solely on the current image and historical trajectory — explicitly not relying on the future trajectory as the basis for reasoning.

The paper performs manual calibration on these VLM-generated CoTs to ensure quality, though the specific calibration procedure (how many annotators, inter-annotator agreement, rejection criteria) is not detailed.

Task 3: Planning. The model is given a question describing the current and historical driving context and must predict the future trajectory of the ego-vehicle as a sequence of $K$ future waypoints in the ego-centric coordinate frame. The model may output this trajectory either directly or after producing a CoT reasoning sequence. The trajectory is represented as continuous $(x, y)$ coordinates, not discrete tokens — this is a key design choice that distinguishes UniUGP from methods like AutoVLA that discretise trajectories into action tokens.

Task 4: Instruction Following. High-level navigation commands (go straight, turn left, turn right) are derived from the geometric properties of the ground-truth future trajectory. For example, if the future waypoints curve to the left, the instruction "turn left" is assigned. These commands are incorporated into the QA pairs so that the model must generate a trajectory consistent with the given navigation intent. During evaluation, this enables testing whether the model can correctly adjust its predicted motion according to high-level instructions — a capability that purely visual models lack entirely.

---

Understanding and Planning Experts: The Mixture-of-Transformers Architecture

The understanding and planning experts constitute a Mixture-of-Transformers (MoT) architecture, a design pattern where two (or more) modalities share self-attention layers but maintain separate feed-forward networks. This is distinct from a Mixture-of-Experts (MoE) where different experts handle different subsets of tokens within a single modality — here, the "mixture" refers to mixing two entire transformer pipelines that attend to each other's tokens.

Understanding Expert Input Encoding. The text instructions are tokenized by the text tokenizer (inherited from Qwen2.5-VL) into discrete token indices, then embedded into continuous vectors. The observation images (multiple frames of history) are encoded by the ViT encoder (also inherited from Qwen2.5-VL) into patch-level visual embeddings. These text and visual embeddings are concatenated or interleaved to form aligned cross-modal understanding tokens:

$x^{\text{und}} = \text{Concat}(\text{TextEmbed}(\text{tokenize}(\text{instruction})), \text{ViT}(\text{history\_images}))$

where $\text{instruction}$ is the natural language prompt, $\text{history\_images}$ is the sequence of past camera frames, $\text{TextEmbed}$ maps token indices to continuous vectors, and $\text{ViT}$ maps image pixels to patch embeddings.

Planning Expert Input Encoding. The planning expert operates on two types of information: the history state $s$ (which includes past waypoints, velocities, accelerations — the ego-vehicle's motion history) and the noised action chunk $a_\tau$. The action chunk $a$ is the ground-truth future trajectory during training (a sequence of future waypoints), and $a_\tau$ is a noised version of it at timestep $\tau$ produced by the forward noising process. The history state and noised actions are concatenated and projected:

$x^{\text{plan}} = \text{Proj.}([s, a_\tau])$

where $[s, a_\tau]$ denotes concatenation along the sequence or feature dimension, and $\text{Proj.}$ is a learned linear projection that maps the raw state-action representation to the planning expert's token dimensionality.

Shared Multi-Head Self-Attention. The core interaction mechanism is that the understanding tokens $x^{\text{und}}$ and planning tokens $x^{\text{plan}}$ are passed through MoT layers where they attend to each other. For each MoT layer, the procedure is:

$h^{\text{und}}_o, h^{\text{plan}}_o = \text{MSHA}([\text{QKV}_{\text{und}}(x^{\text{und}}), \text{QKV}_{\text{plan}}(x^{\text{plan}})])$

where $\text{QKV}_{\text{und}}$ and $\text{QKV}_{\text{plan}}$ are learned linear projections that map understanding and planning tokens into queries, keys, and values, $[\cdot, \cdot]$ denotes concatenation of the QKV-projected tokens, and $\text{MSHA}$ is multi-head self-attention applied to the full concatenated sequence.

What this means operationally: The understanding tokens can attend to planning tokens, and the planning tokens can attend to understanding tokens. This means the language model can look at the current trajectory estimate and adjust its reasoning accordingly (e.g., "the trajectory is curving left, so there must be a left turn ahead"), and the trajectory generator can look at the semantic understanding and adjust its predictions (e.g., "the model says there is a construction zone, so the trajectory should slow down and possibly shift laterally").

Modality-Specific Feed-Forward Networks. After the shared attention, each modality's tokens are processed by independent FFNs:

$h^{\text{und}}_{\text{ffn}} = \text{FFN}_{\text{und}}(h^{\text{und}}_o)$

$h^{\text{plan}}_{\text{ffn}} = \text{FFN}_{\text{plan}}(h^{\text{plan}}_o)$

This separation is crucial: the FFNs can learn modality-specific transformations (language syntax and semantics for the understanding FFN, motion smoothness and physical constraints for the planning FFN) while the shared attention layers learn cross-modal alignment.

Output Heads. After $N$ MoT layers (the paper does not specify the exact number, but this is inherited from Qwen2.5-VL's architecture), the final hidden states are mapped to outputs:

$P_{\text{logits}} = \text{LMHead}(h^{\text{und}})$

$u^{\text{plan}}_\tau = \text{unProj.}(h^{\text{plan}})$

where $\text{LMHead}$ is the language modeling head (a linear layer mapping hidden states to vocabulary logits), $P_{\text{logits}}$ is the probability distribution over the next token, and $u^{\text{plan}}_\tau$ is the predicted denoising vector field (explained in detail in the flow matching section below). The $\text{unProj.}$ is the inverse projection that maps from the planning expert's hidden dimension back to the action space dimensionality.

Why this architecture over alternatives? The paper cites "validity proof of the existing work" by Black et al. (2024) on π₀, Lv et al. (2025) on F1, and Shi et al. (2024) on LMFusion, which previously demonstrated that shared-attention MoT architectures enable effective cross-modal transfer between discrete language and continuous action spaces. The alternative — a monolithic transformer where language tokens and action tokens are simply concatenated and processed by a single unified model — would force the FFN layers to serve two masters (language modeling and trajectory denoising), likely resulting in interference where improvements in one capability degrade the other. The separate FFNs allow each modality to specialise while the shared attention enables coordination.

---

Flow Matching for Trajectory Generation

The planning expert generates trajectories using flow matching, a generative modeling framework that learns to transform samples from a simple noise distribution into samples from the data distribution by learning a time-dependent vector field. This is distinct from diffusion models (which learn to predict the noise added to data) and from discrete action tokenisation (which quantises continuous trajectories into a finite vocabulary, losing precision).

Forward Noising Process. During training, the ground-truth action chunk $a$ (a sequence of future waypoints) is gradually corrupted toward pure noise. A random noise vector $\epsilon \sim \mathcal{N}(0, I)$ and a timestep $\tau \in [0, 1]$ are sampled. The noised action is computed as a linear interpolation between the clean action and the noise, weighted by the timestep:

$a_\tau = \tau a + (1 - \tau) \epsilon$

where $a$ is the clean action chunk (the ground-truth future trajectory), $\epsilon$ is random Gaussian noise, and $\tau$ is the interpolation coefficient (when $\tau = 0$, $a_\tau = \epsilon$ — pure noise; when $\tau = 1$, $a_\tau = a$ — clean data).

What it computes: This is the forward noising process. At each training step, a random timestep is selected, and the ground-truth action is mixed with noise in proportion to that timestep. Early timesteps (small $\tau$) produce heavily noised actions; later timesteps (large $\tau$) produce lightly noised actions. The result $a_\tau$ is what the planning expert receives as input during training.

Why this form: The linear interpolation schedule is the simplest possible flow between noise and data — it defines straight-line paths in action space. This is computationally efficient and, under the flow matching framework, corresponds to the optimal transport path between the noise and data distributions, which yields simpler vector fields to learn compared to the curved trajectories of standard diffusion. The variable $\tau$ is continuous, unlike discrete diffusion timesteps, allowing the model to learn to denoise at any noise level rather than only at discretised levels.

Training Objective. The planning expert learns to predict the vector field $u^{\text{plan}}_\tau$ that, when applied to the noised action $a_\tau$, moves it toward the clean action $a$. The training objective is:

$\mathcal{L}_{\text{plan}} = \mathbb{E}_{u^{\text{plan}}_\tau}\left[\|u^{\text{plan}}_\tau - (\epsilon - a)\|^2\right]$

where $u^{\text{plan}}_\tau$ is the planning expert's predicted denoising vector at timestep $\tau$, $\epsilon$ is the sampled Gaussian noise, and $a$ is the clean action chunk.

What it computes: The mean squared error between the predicted vector field and the target vector $(\epsilon - a)$. The target vector $(\epsilon - a)$ is the direction from the noised action $a_\tau$ toward the clean action $a$ — specifically, $\epsilon - a$ is the vector that, when subtracted from the noise, yields the clean data (up to the scaling by $\tau$). Minimising this loss teaches the model to predict the direction and magnitude of denoising needed at each timestep.

Why this form: In flow matching with linear interpolation, the true vector field that transforms noise to data is exactly $\epsilon - a$ (the velocity field of the straight-line path). The squared error loss is the standard regression objective for vector field prediction. Alternative formulations using score matching (predicting the gradient of the log-density) would estimate the score function $\nabla\log p(a_\tau)$, which requires different parameterisation and is less directly connected to the desired denoising operation. The flow matching formulation is simpler: at each step, predict the velocity that moves the sample toward the data.

Inference (Denoising). During inference, the planning expert starts from pure noise $a_0 \sim \mathcal{N}(0, I)$ (where $\tau=0$ corresponds to $t=0$ in the denoising process, not to be confused with the forward $\tau$). It iteratively applies the learned vector field to progressively denoise the sample, moving it through a sequence of timesteps $\tau_0, \tau_1, ..., \tau_T$ from 0 toward 1, until a clean action chunk is obtained. The paper does not specify the number of denoising steps or the integration method (e.g., Euler, Heun), which are important implementation details for reproducibility.

The output is a sequence of future waypoints $\hat{a}$ in the ego-centric coordinate frame, where each waypoint is an $(x, y)$ coordinate (x forward, y left). The paper uses the convention explained in the CoT prompt template (Listing 4): $+x$ = forward direction, $+y$ = left direction, origin at the center of the ego-vehicle, waypoints in metres, sampling frequency 4 Hz (every 0.25 seconds), past arrays contain 16 timesteps (from $t = -3.75\text{s}$ to $t = 0.0\text{s}$), and future waypoints contain 20 timesteps (from $t = 0.25\text{s}$ to $t = 5.0\text{s}$).

---

Understanding Expert Training Objective

The understanding expert is trained with standard autoregressive next-token prediction, the same objective used to pre-train Qwen2.5-VL:

$\mathcal{L}_{\text{und}} = \mathbb{E}_{x^{\text{und}}_i}\left[-\log(P(x^{\text{und}}_i \mid x^{\text{und}}_{<i}))\right]$

where $x^{\text{und}}_i$ is the $i$-th token in the understanding sequence (which includes both the input instruction tokens and the output answer/CoT tokens, with loss computed only on the output tokens in practice), $x^{\text{und}}_{<i}$ is all preceding tokens, and $P(\cdot \mid \cdot)$ is the model's predicted probability distribution over the vocabulary at position $i$.

What it computes: The expected negative log-likelihood of each target token given all previous tokens. For each position in the output sequence, the model predicts a probability distribution over the entire vocabulary; the loss is the negative log of the probability assigned to the actual token that appeared at that position. The expectation is over all positions in the training data.

Why this form: This is the standard maximum-likelihood objective for autoregressive language models. The negative log-likelihood is a proper scoring rule — it is minimised when the model's predicted distribution exactly matches the true data distribution. It penalises both under-confidence (assigning low probability to the correct token) and over-confidence (assigning high probability to incorrect tokens). The cross-entropy form ensures the gradients are well-behaved and the optimisation is numerically stable.

What makes this non-trivial in UniUGP: The input to the understanding expert includes not only text but also visual tokens from the ViT and cross-attention information from the planning expert's hidden states (via the shared self-attention in MoT layers). This means the language model is not just predicting text from text — it is predicting text conditioned on visual observations, motion history, and the current state of the trajectory denoising process. The gradients from $\mathcal{L}_{\text{und}}$ flow back through the MoT layers into both the understanding and planning pathways, enabling the planning task to influence language representations and vice versa.

---

Generation Expert: Cascaded Video Generation with Cross-Modal Conditioning

The generation expert produces future video frames through a flow matching process conditioned on the understanding and planning experts' outputs. Unlike the understanding and planning experts which share attention layers, the generation expert is cascaded — meaning it receives information from the other experts but does not feed information back to them during inference (during training, gradients do flow back, which is how the generation task influences the understanding and planning experts). The paper uses Wan2.1 as the base video generation model, inheriting its pre-trained DiT (Diffusion Transformer) parameters.

Input Encoding. History images and future images are encoded into latent tokens by a pre-trained VAE (Variational Autoencoder):

$v^{\text{hist}} = \text{VAE}(\text{history\_images}), \quad v^{\text{fut}} = \text{VAE}(\text{future\_images})$

where $\text{history\_images}$ are the observed past frames, $\text{future\_images}$ are the target future frames to be predicted, and the VAE compresses each frame into a grid of latent tokens (reducing spatial resolution by a factor, typically 8× or 16×, while preserving essential visual information in a lower-dimensional space). The noised future latents $v^{\text{fut}}_\tau$ are formed by the same linear interpolation flow matching process as the planning expert:

$v^{\text{fut}}_\tau = \tau v^{\text{fut}} + (1 - \tau) \epsilon$

where $\epsilon \sim \mathcal{N}(0, I)$ and $\tau \in [0, 1]$.

Conditioning Signals. The generation expert receives two sources of conditioning:

1. Understanding hidden states $h^{\text{und}}$ from the final MoT layer of the understanding expert — these encode the fused semantic-visual understanding of the scene, including object identities, relationships, and the chain-of-thought reasoning context.
2. Action embeddings $A$ derived from the planning expert's predicted (or ground-truth) future trajectory $\hat{a}$: $A = \text{Proj.}(\hat{a})$, where $\text{Proj.}$ is a learned linear projection mapping the trajectory waypoints to an embedding space compatible with the DiT's conditioning mechanism.

Why both signals? The understanding hidden states provide what is in the scene and why things are happening (semantic context), while the action embeddings provide how the ego-vehicle will move (physical context). Combining both enables the generation expert to produce future videos that are both semantically consistent (e.g., the traffic light remains red because the model knows it's at an intersection with a stop condition) and physically consistent (e.g., the ego-vehicle's viewpoint shifts according to the predicted trajectory, and other objects move with plausible dynamics).

Denoising Network. The core of the generation expert is a stack of DiT blocks — transformer layers that process the concatenated history and noised future latent tokens, modulated by the conditioning signals and the timestep:

$u^{\text{gen}}_\tau = W([v^{\text{hist}}, v^{\text{fut}}_\tau], [h^{\text{und}}, A], \tau)$

where $W$ denotes the full DiT network (Wan2.1's architecture, comprising $M$ DiT blocks as shown in Figure 1), $[v^{\text{hist}}, v^{\text{fut}}_\tau]$ is the concatenated sequence of history and noised future latent tokens (each token representing a spatial-temporal patch), $[h^{\text{und}}, A]$ is the concatenated conditioning information, and $\tau$ is the diffusion timestep embedding (injected via adaptive layer normalisation or cross-attention, depending on Wan2.1's specific mechanism).

What it computes: For each spatial-temporal position in the future video latents, the DiT network predicts the denoising vector field $u^{\text{gen}}_\tau$ — the direction and magnitude to move the noised latent toward the clean future video latent. The conditioning signals bias this prediction so that the denoised video is consistent with both the scene understanding (e.g., there is a pedestrian) and the planned trajectory (e.g., the ego-vehicle is turning left, so the viewpoint shifts accordingly).

Training Objective. The generation expert minimises the same flow matching loss:

$\mathcal{L}_{\text{gen}} = \mathbb{E}_{u^{\text{gen}}_\tau}\left[\|u^{\text{gen}}_\tau - (\epsilon - v^{\text{fut}})\|^2\right]$

where $u^{\text{gen}}_\tau$ is the predicted vector field, $\epsilon$ is the sampled noise, and $v^{\text{fut}}$ is the clean future video latent.

What it computes: Mean squared error between predicted and target denoising vectors, exactly analogous to the planning loss in Equation 8, but operating in VAE latent space rather than trajectory space.

Why this form: The same flow matching framework is used for both planning and generation, providing a unified mathematical formalism. The target $(\epsilon - v^{\text{fut}})$ is the vector from noise to clean future latent along the straight-line path. Minimising this loss trains the DiT to predict the velocity field that transforms noise into future video frames conditioned on scene semantics and planned actions.

Training-Time vs. Inference-Time Action Conditioning. A crucial design detail is how the action embeddings $A$ are obtained during training. The paper uses a stochastic mix of ground-truth and predicted actions:

$A \sim \begin{cases} \text{Proj.}(a) & \text{if rand}(0 \sim 1) > 0.5 \\ \text{Proj.}(a_\tau - (1 - \tau)u^{\text{plan}}_\tau) & \text{else} \end{cases}$

where $a$ is the ground-truth action chunk (the actual future trajectory from the dataset), $a_\tau$ is the noised action, and $u^{\text{plan}}_\tau$ is the planning expert's predicted denoising vector at timestep $\tau$. The expression $a_\tau - (1-\tau)u^{\text{plan}}_\tau$ performs a single-step denoising of the noised action using the planning expert's current prediction — it approximates what the planning expert would predict as the clean action if denoised from timestep $\tau$.

Why this mixed strategy? If the generation expert were always conditioned on perfect ground-truth actions during training (the first branch), it would learn to rely on oracle trajectory information that is unavailable at inference time, causing a train-test distribution shift. If it were always conditioned on the planning expert's predicted actions (the second branch), the predictions would initially be poor early in training, making the generation task unnecessarily difficult and potentially causing the generation expert to ignore the action conditioning entirely. The 50-50 mix is a form of scheduled sampling (Bengio et al., 2015): it gradually exposes the generation expert to its own inference-time conditions while still providing ground-truth guidance, bridging the gap between training and inference conditions.

During inference, the action embeddings are always computed from the planning expert's predicted future actions: $A = \text{Proj.}(\hat{a})$, where $\hat{a}$ is the fully denoised trajectory output.

Video Generation Resolution. The paper configures the generation expert at $(512, 512)$ pixel resolution (Table 2, Stage 2 and Stage 4), encoded by the VAE into a lower-dimensional latent grid. This is higher than the understanding expert's input resolution of $(224, 224)$, reflecting the different requirements: understanding needs enough resolution for object recognition and scene parsing; generation needs higher resolution for photo-realistic visual quality. The VAE handles the dimensionality reduction so that the DiT operates in a manageable latent space while decoding back to high-resolution pixels.

The Generation Expert Can Be Disabled at Inference. Figure 1 indicates that the generation expert is serial — it receives conditioning from the understanding and planning experts but does not feed information back to them during inference. This means on resource-constrained platforms (e.g., mobile devices in a car), the generation expert can be completely disabled "without compromising the performances of the former experts" (Section 3.2). The ablation results in Table 3 confirm this: the "Our w/o Gen." configuration (which drops the generation expert) achieves 83.7% small-object understanding and 1.72m planning L2, which is worse than the full model (89.3% and 1.45m) but still better than the no-CoT ablation (82.9% and 1.67m). This is a practical deployment consideration: the generation expert provides a training-time benefit (forcing the model to learn visual causality) but can be stripped at inference for efficiency.

---

Four-Stage Training Curriculum

The paper develops a four-stage training framework because naive joint training of all three experts on all datasets simultaneously leads to catastrophic interference — the competing loss signals from language modeling, flow matching, and video denoising prevent any single capability from being learned well. The staged approach builds capabilities progressively, with each stage focusing on a subset of components and datasets, culminating in a fusion stage that harmonises all capabilities.

The training hyperparameters are documented in Table 2. All four stages use a learning rate of $10^{-4}$, a batch size of 64, and 8 nodes each with 8 GPUs (80GB each) — substantial computational resources (64 GPUs total per stage). The understanding expert uses $(224, 224)$ resolution images (Stages 1, 3, 4) while the generation expert uses $(512, 512)$ (Stages 2, 4).

Stage 1: Continuous Training for Basic Scenario Understanding (1M steps). Only the understanding expert is trained. The generation and planning experts are frozen (indicated by × in Table 2). The training data consists of the custom long-tail dataset (perception questions, accident prediction, accident relationships) and the ImpromptuVLA dataset (80,000 curated video clips from 8 open-source datasets, providing coverage of common driving scenarios). The objective is $\mathcal{L}_{\text{und}}$ only.

The purpose of this stage is to establish a comprehensive understanding of diverse driving scenarios — both common traffic situations and long-tailed cases — within the VLM backbone. Without this foundation, the model would lack the basic scene parsing capability needed for the subsequent stages. Training only the understanding expert prevents the planning and generation losses from interfering with this foundation-building.

Stage 2: Visual Dynamics Modeling and Planning Training (4M steps). The understanding expert is frozen, while the planning expert and generation expert are trained. The training data consists of driving videos with trajectories from public datasets: nuScenes, NuPlan, Waymo, Lyft, and Cosmos. The objectives are $\mathcal{L}_{\text{plan}}$ (Equation 8) and $\mathcal{L}_{\text{gen}}$ (Equation 11).

The purpose of this stage is to learn visual dynamics (how scenes evolve over time) and motion planning (how the ego-vehicle should move given the scene). The generation expert learns to predict future frames by watching large amounts of unlabeled or trajectory-labeled driving video, thereby acquiring visual causal reasoning — understanding that objects move with inertia, that braking distances depend on speed, that occluded objects can reappear, and so on. The planning expert learns to generate trajectories that are physically consistent with the scene dynamics.

Stage 3: Text Reasoning Learning for Causal Validation (1M steps). Only the understanding expert is trained (planning and generation frozen). The training data is the custom CoT dataset (the structured chain-of-thought annotations described in Task 2 of the dataset construction). The objective is $\mathcal{L}_{\text{und}}$ only, but now the target tokens are the CoT reasoning text rather than simple answers.

The purpose of this stage is to integrate explicit causal reasoning into the understanding expert, enabling it to validate the logic of its perceptions and planning using natural language. This stage enhances model interpretability — instead of just outputting a trajectory, the model can now explain why that trajectory was chosen in terms of scene elements, object intentions, and physical constraints. The paper emphasises that this "ensures decisions are grounded in explicit causal reasoning" (Section 3.3).

Stage 4: Mixed Training for Multi-Capability Fusion (4M steps). All three experts are trained jointly on a mixture of data from Stages 1–3, blended at a fixed ratio: 0.1 (Stage 1 data) : 0.4 (Stage 2 data) : 0.5 (Stage 3 data), summing to 1.0. The total loss is a weighted combination:

$\mathcal{L}_{\text{total}} = \alpha \cdot \mathcal{L}_{\text{und}} + \beta \cdot \mathcal{L}_{\text{plan}} + \gamma \cdot \mathcal{L}_{\text{gen}}$

where $\alpha = 0.3$, $\beta = 0.5$, and $\gamma = 0.2$.

What it computes: A scalar loss that sums the three individual expert losses, each multiplied by a task-specific weight. The weights reflect the paper's prioritisation: planning is most important (0.5), followed by understanding (0.3), and generation is least important (0.2). The data mixture ratio (0.1:0.4:0.5 for Stages 1:2:3 data respectively) is different from the loss weights and reflects the volume of data available from each stage, not the relative importance.

Why this staged approach? Training all components end-to-end from scratch would confront the model with conflicting objectives: the language model wants to minimise perplexity on text, the flow matching model wants to minimise denoising error on trajectories, and the video generator wants to minimise denoising error on pixels. These losses operate at different scales, have different noise characteristics, and pull the shared representations in different directions. The staged curriculum avoids this by building capabilities sequentially and only blending them in the final stage once each expert has developed a strong initial capability. The specific ratio 0.1:0.4:0.5 is described as "balancing foundational, reasoning, planning, and generation capabilities" (Section 3.3), though the paper does not report ablation experiments on different mixing ratios.

Total training duration. Summing the steps across all four stages: 1M + 4M + 1M + 4M = 10M training steps. At a batch size of 64 across 64 GPUs (effective batch size of 4,096 if using data parallelism, though the paper does not specify the parallelisation strategy), this represents a substantial computational investment.

Key design insight — why Stage 2 trains the generation expert while freezing the understanding expert: During Stage 2, gradients from $\mathcal{L}_{\text{gen}}$ flow back through the generation expert and into the planning expert via the shared self-attention in the MoT layers, but the understanding expert is frozen, so its pre-trained VLM knowledge is not overwritten by video generation gradients. This preserves the semantic understanding from Stage 1 while allowing the planning expert to learn from both trajectory supervision ($\mathcal{L}_{\text{plan}}$) and video prediction supervision ($\mathcal{L}_{\text{gen}}$). In Stage 4, when all components are unfrozen, the understanding expert can then benefit from the visual dynamics knowledge that was learned by the planning expert during Stage 2 — this is the mechanism by which video generation training improves scene understanding, as demonstrated in the ablation studies.

Interaction between training stages. The paper describes this architecture as one where capabilities are built progressively, but it does not explicitly discuss whether later stages cause catastrophic forgetting of earlier capabilities. The inclusion of Stage 1 data (at 10%) and Stage 2 data (at 40%) in the Stage 4 mixture is a standard mitigation strategy — it ensures the model continues to see examples of foundational tasks even while learning the joint fusion objective. However, the paper does not report controlled experiments showing whether performance on Stage 1-only tasks degrades after Stages 2–4 training, which would be a useful robustness check.

4. Key Insights and Innovations

Innovation 1: Video Generation as a Training Signal for Visual Causal Reasoning, Not Just a Visualization Output

The most intellectually distinctive move in this paper is a conceptual reframing of what video generation is for in autonomous driving architectures. Prior work on world models for AD — from GAIA-1 (Hu et al., 2023) to DriveDreamer (Wang et al., 2024) to Epona (Zhang et al., 2025) — treated future video prediction as an end in itself: generate realistic-looking future frames to visualize what the model expects to happen, or to serve as input for downstream planning modules. The generation quality was the evaluation metric (FID, FVD), and the generation module was a consumer of upstream representations, not a contributor to them.

UniUGP inverts this relationship. The generation expert is not primarily valuable for the videos it produces — indeed, it can be completely disabled at inference time "without compromising the performances of the former experts" (Section 3.2). Its value is as a training-time regularizer that forces the shared representations in the MoT layers to encode visual causality: how objects move, how occlusion works, how small distant hazards become large close ones, how trajectories through space produce specific visual consequences. The paper's qualitative analysis in Figure 3 makes this concrete: the full model's CoT reasoning identifies "a construction area ahead, with workers and cone markers" and concludes "the vehicle may slow down to avoid collisions," while the ablated model without the generation expert produces generic, untethered advice about "observing road conditions" and "driving at the prescribed speed steadily." The generation-ablated model is not wrong — it just cannot connect its visual observations to specific causal predictions about what will happen and what action is required.

This is a fundamentally different theory of how world models benefit autonomous driving. The dominant assumption in prior work was that world models help by providing explicit predictions — future occupancy grids that a planner can reason over, or future images that a perception module can analyze. UniUGP's evidence suggests the benefit is primarily implicit: the process of learning to generate future frames reshapes the model's internal representations such that the understanding expert (the VLM) acquires visual dynamics knowledge even though it never directly processes video generation losses during its own training (it is frozen during Stage 2 when the generation expert is trained). This is a form of representation learning through auxiliary task pressure — the generation loss propagates gradients through the planning expert and into the shared attention layers, where it shapes representations that the understanding expert later inherits when unfrozen in Stage 4.

The empirical evidence for this claim is specific and falsifiable (Table 3). Removing the generation expert degrades small-object understanding from 89.3% to 83.7%, planning L2 from 1.45m to 1.72m, and CoT reasoning from 0.88 to 0.80 (GPT score). These are substantial degradations across all three capabilities, not just generation — exactly what you would expect if the generation task were serving as a cross-modal training signal rather than merely as an extra output modality. The degradation is particularly notable on small-object understanding, which aligns with the paper's argument that video prediction forces attention to distant, small objects because they grow large and consequential in future frames.

This insight has implications beyond autonomous driving. It suggests a general architectural principle: cascaded generative modules that consume a model's internal representations during training but are stripped at inference can serve as auxiliary task supervisors that impart capabilities the primary task objective alone cannot efficiently teach. This is distinct from standard multi-task learning (where all tasks are objectives to be optimized) and from knowledge distillation (where a larger model teaches a smaller one). Here, the generation task is a scaffold — it shapes learning during training but leaves no runtime footprint. The paper does not theorize this principle explicitly, but the architecture embodies it, and the ablation results provide evidence for it.

Innovation 2: Shared-Attention, Separate-FFN Architecture for Aligning Discrete Language and Continuous Action Spaces

The paper's Mixture-of-Transformers (MoT) design for the understanding and planning experts addresses a fundamental architectural challenge that prior VLA methods handled with ad-hoc solutions: how do you get a discrete token prediction model (the VLM, which outputs text) and a continuous trajectory generation model (which outputs real-valued waypoints) to share information without destructive interference?

Prior work split along two unsatisfactory approaches. Methods like Impromptu VLA (Chi et al., 2025) and AutoVLA (Zhou et al., 2025) discretized trajectories into text tokens — "move forward 2.3 meters, then turn left 15 degrees" — and trained the VLM to output these tokens autoregressively. This works but loses the precision and smoothness of continuous representations; a trajectory represented as discrete tokens cannot capture fine-grained spatial relationships, and the tokenization granularity becomes a hyperparameter that trades off precision against sequence length. Methods like ReCogDrive (Li et al., 2025) and ORION (Fu et al., 2025) kept trajectories continuous but used the VLM only as a feature extractor, feeding its hidden states into a separate diffusion-based planner with no backward gradient flow — meaning the VLM never learned from trajectory supervision, and the planner never benefited from language understanding beyond the initial feature extraction.

UniUGP's MoT design (shared multi-head self-attention, separate feed-forward networks) is a principled resolution to this tension. The shared attention layers allow the understanding tokens and planning tokens to attend to each other — the language model can look at the current trajectory estimate and adjust its reasoning accordingly, and the trajectory generator can look at the semantic understanding and condition its denoising on scene context. The separate FFNs allow each modality to specialize: the understanding FFN learns language-specific transformations (syntax, semantics, discourse coherence), while the planning FFN learns motion-specific transformations (smoothness, physical constraints, dynamic feasibility). This prevents the kind of destructive interference that would occur if a single FFN had to serve both masters.

The paper cites prior demonstrations of this architectural pattern in robotics (π₀ by Black et al., 2024; F1 by Lv et al., 2025) and multimodal generation (LMFusion by Shi et al., 2024), but its application to the specific problem of VLM-trajectory alignment in driving is novel. What makes it more than an off-the-shelf reuse is the demonstration that the shared attention enables bidirectional transfer — language understanding improves from trajectory supervision, and trajectory generation improves from language understanding — through the four-stage curriculum. This bidirectionality is not guaranteed by the architecture alone; it emerges from the training strategy where the planning expert is trained on video dynamics (Stage 2) while the understanding expert is frozen, then the understanding expert is unfrozen (Stage 4) and benefits from the representations shaped during Stage 2. The architecture provides the channel for this transfer; the curriculum determines when and how information flows through that channel.

The significance of this innovation is partially architectural (the MoT design itself) but primarily evidential: the paper provides a clear empirical demonstration that cross-modal alignment through shared attention plus modality-specific processing is sufficient to achieve mutual enhancement between language and continuous control, without resorting to discretization or one-way feature extraction. This is an existence proof that should influence architectural choices in future VLA systems.

Innovation 3: Difficulty-Aware, Competency-Based Dataset Construction as an Evaluation Philosophy

The paper's approach to benchmark construction represents a methodological innovation distinct from its architectural contributions. Rather than constructing a monolithic evaluation dataset and reporting aggregate metrics, the paper designs its evaluation around four specific cognitive competencies — perception and understanding, causal chain-of-thought reasoning, planning and decision-making, and instruction following — and constructs targeted sub-tasks for each that probe failure modes prior benchmarks ignore.

This is not merely "we collected more data." The intellectual move is to argue that evaluating an autonomous driving model requires testing its specific cognitive failures, not just its aggregate success rate. A model that achieves 90% accuracy on standard perception questions might still fail catastrophically on small-object recognition in long-tail scenarios — and that failure, not the aggregate metric, determines safety. The paper's benchmark makes this argument operational by constructing three sub-categories for understanding alone (small objects, accident relationships, anomaly prediction), each sourced from datasets specifically designed to capture rare events (StreetHazards, Lost and Found, DADA2000).

Compare this to the dominant evaluation paradigm in the field. DriveLM (Sima et al., 2024) evaluates VLM-based driving models on general perception-prediction-planning questions using metrics like BLEU, ROUGE, and ChatGPT Score — these reward fluency and general scene description but do not specifically test whether the model recognizes the small obstacle that a human driver would swerve to avoid. The nuScenes planning benchmark (Caesar et al., 2020) reports L2 displacement error and collision rate on standard urban driving — useful for comparing architectures but blind to the long-tail cases where accidents actually happen. The paper's benchmark complements these by zeroing in on the specific capabilities that prior evaluations obscure.

The chain-of-thought evaluation methodology is particularly thoughtful. Rather than simply asking human raters to score reasoning quality (expensive and noisy), the paper uses future trajectory information during dataset construction to ensure that the reference CoT is causally grounded in the physical outcome — "the reasoning process should not rely on the future trajectory as the basis for your reasoning" but the future trajectory is used to verify that the reasoning arrives at the correct conclusion (Listing 4, Appendix). This creates a gold-standard reasoning dataset where correctness is objective (does the reasoning lead to the action that actually occurred?) rather than subjective (does the reasoning sound plausible?). The manual calibration step provides a quality check, and the GPT-4o-based automated evaluation at test time provides scalability. This pipeline — ground reasoning in observable future outcomes, calibrate manually, evaluate automatically — is a template for constructing reasoning benchmarks in embodied domains where actions have observable consequences.

The significance of this innovation extends beyond the specific datasets. It establishes a competency-based evaluation philosophy for AD systems: identify the specific cognitive capabilities required for safe driving, construct targeted probes for each, and evaluate models on their ability to demonstrate those capabilities in challenging edge cases, not just on aggregate metrics in average cases. This philosophy is underdeveloped in current AD evaluation and the paper provides a concrete implementation demonstrating its feasibility and discriminative power (Table 3 shows clear separation between models on these targeted metrics even when aggregate metrics might obscure the differences).

Innovation 4: The Four-Stage Training Curriculum as a Solution to Cross-Modal Interference in Multi-Objective Training

Training a single model with three heterogeneous objectives — language modeling cross-entropy, flow matching squared error on trajectories, and flow matching squared error on video latents — creates a well-known optimization challenge: the loss landscapes for these objectives have different curvatures, different noise scales, and different optimal parameter configurations, and optimizing their sum can lead to solutions that are mediocre at all tasks rather than excellent at any.

The standard solution in multi-task learning is loss weighting — tune $\alpha$, $\beta$, $\gamma$ in Equation 12 so that no single objective dominates the gradients. UniUGP does this (final weights: 0.3, 0.5, 0.2), but the paper's key insight is that loss weighting alone is insufficient when the tasks are as heterogeneous as text generation, trajectory denoising, and video prediction. Even with optimal weights, the gradients from these objectives pull the shared representations in conflicting directions early in training, before any single capability has been established.

The four-stage curriculum solves this by sequentially building capabilities and only blending objectives once each expert has a strong initialization. Stage 1 establishes basic scene understanding in the VLM without competition from other objectives. Stage 2 builds visual dynamics and planning capabilities while the VLM is frozen, preventing video generation gradients from overwriting the semantic knowledge acquired in Stage 1. Stage 3 integrates explicit causal reasoning into the VLM without the distraction of trajectory or video losses. Only in Stage 4, after each expert has developed a strong capability in its own domain, are all objectives combined — and even then, the data mixture (0.1:0.4:0.5 for Stages 1:2:3 data) ensures the model continues to rehearse foundational tasks while learning to coordinate across modalities.

This is a form of curriculum learning where the curriculum is defined by which components are trainable and which datasets are presented, not just by example difficulty. The paper does not explicitly frame it this way, but the mechanism is clear: Stage 1 is a "warm-start" phase for the VLM; Stage 2 is a "visual dynamics bootcamp" for the planning and generation experts; Stage 3 is a "reasoning refinement" phase; Stage 4 is "cross-modal integration." Each stage solves a subproblem that is easier than the full joint optimization problem, and the solution to each subproblem provides a better initialization for the next.

The significance of this innovation is practical as much as conceptual. Training large multi-modal models with multiple objectives is increasingly common (e.g., unified vision-language models that both understand and generate images), and the paper provides a concrete, reproducible recipe for managing the optimization challenges: freeze components strategically, introduce objectives sequentially, and only combine them once each has a strong foothold. The specific staging (1M → 4M → 1M → 4M steps) and data ratios (0.1:0.4:0.5) are likely dataset- and model-specific, but the principle — progressive capability building with strategic freezing — is transferable.

The paper's empirical support for this innovation is indirect but compelling: the full model's performance improvements over ablations (Table 3) cannot be attributed solely to the architecture or the data — the training strategy determines whether the architecture and data actually produce mutual enhancement or mutual interference. The fact that the four-stage model achieves state-of-the-art results across all four evaluation dimensions while the ablated models (trained with the same architecture and data but without full staging) underperform suggests the curriculum is doing real work.

Innovation 5: The 50-50 Scheduled Sampling Strategy for Action Conditioning During Video Generation Training

While this is a smaller innovation than the preceding ones, it addresses a subtle but pervasive problem in cascaded architectures: the train-test distribution shift that occurs when a downstream module (the generation expert) is conditioned on ground-truth information during training but must rely on predicted information during inference.

The specific problem is this: during training, if the generation expert is always conditioned on ground-truth future trajectories ($A = \text{Proj.}(a)$), it learns to generate videos that are perfectly consistent with the true ego-motion. At inference time, however, the ground-truth trajectory is unavailable — the generation expert must condition on the planning expert's predicted trajectory ($A = \text{Proj.}(\hat{a})$). If the planning expert's predictions are imperfect (and they always are, especially early in training), the generation expert receives conditioning inputs that are out-of-distribution relative to its training data, causing it to produce degraded videos or to ignore the conditioning entirely.

This is a classic exposure bias problem (analogous to the teacher-forcing versus scheduled sampling debate in sequence generation), and the paper's solution is elegant in its simplicity: during training, randomly decide (50% probability each) whether to condition on ground-truth actions or on single-step denoised predictions from the planning expert's current state. The single-step denoising ($a_\tau - (1-\tau)u^{\text{plan}}_\tau$) provides an approximation of what the planning expert would predict at the current timestep, without requiring full iterative denoising in the training loop.

What makes this distinctive is not the technique itself (scheduled sampling has been used since Bengio et al., 2015) but its application to the specific problem of cross-modal conditioning in a multi-expert architecture, and the recognition that the 50-50 mix serves a dual purpose: it both bridges the train-test gap (as standard scheduled sampling) and serves as a form of cross-modal regularization, preventing the generation expert from becoming overly dependent on perfect trajectory information and forcing it to learn representations that are robust to the planning expert's current error characteristics.

The significance is that this design choice likely has outsized importance relative to its apparent simplicity. Without this mechanism, the generation expert could learn to exploit ground-truth trajectory information in ways that don't generalize — for example, by learning a trivial mapping from trajectory waypoints to ego-motion-induced pixel shifts without actually learning object dynamics. The 50-50 mix forces the generation expert to model the visual world under uncertainty about the ego-vehicle's exact future motion, which is precisely the condition it faces at inference time. The paper does not ablate this specific design choice (there is no "generation expert trained with only ground-truth conditioning" baseline reported), which is a limitation, but the conceptual rationale is sound and the technique is a transferable solution to a problem that will recur in any cascaded architecture where upstream predictions condition downstream generation.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper evaluates on multiple complementary datasets: (1) a custom long-tail benchmark constructed from Waymo-E2E (4,021 segments of rare, high-risk events occurring in less than 0.003% of daily driving), DADA2000, Lost and Found, StreetHazards, SOM, and AADV — used for perception, CoT reasoning, planning, and instruction following evaluation (Section 4.1, Table 3); (2) nuScenes (1,000 scenes, 20 seconds each, 6-camera 360° coverage) — used for trajectory planning (L2 displacement error, collision rate) and future frame generation (FID, FVD) with 28,130 training, 6,019 validation, and 193,082 unannotated samples (Section 4.1, Tables 4–5); (3) DriveLM GVQA — used for scene understanding and language interaction evaluation with metrics including BLEU, ROUGE_L, CIDEr, ChatGPT Score, and accuracy (Section 4.1, Table 6). The training datasets additionally include ImpromptuVLA (80,000 curated video clips), NuPlan, Waymo, Lyft, and Cosmos (Section 3.3, Table 2).

• Base model(s). The understanding expert backbone is Qwen2.5-VL-3B, a 3-billion-parameter vision-language model pre-trained on internet-scale multimodal data. The generation expert backbone is Wan2.1, a pre-trained DiT-based video generation model whose parameters are inherited as initialization. The paper does not explicitly justify the 3B scale choice, but the model is positioned as representative of a deployable size — small enough that test-time compute and training strategies can compensate for parameter count, consistent with the paper's framing that VLA models can be enhanced through architectural and training innovations rather than raw scale. For comparison, Table 3 includes GPT-4o and Qwen2.5-VL-72B (a 24× larger VLM from the same family) as baselines.

• Metrics. Four distinct metric categories are used. Understanding: accuracy (%) on multiple-choice and true/false questions across three sub-categories — small object recognition, accident subject relationship identification, and abnormal event prediction (Table 3). CoT Reasoning: GPT-4o-based automated scoring (0–1 scale) evaluating consistency, rationality, and fluency, plus BLEU score against reference reasoning text (Table 3). Planning: L2 displacement error in meters at 1s, 2s, 3s horizons and average, plus collision rate (%) at the same horizons, computed by comparing predicted ego-vehicle trajectories against ground-truth future waypoints in the ego-centric coordinate frame (Tables 3–4). Instruction Following: L2 displacement error at 3 seconds for trajectories generated under different high-level navigation commands, measuring whether the model adjusts its trajectory to match the instruction (Table 3). Generation Quality: Fréchet Inception Distance (FID) and Fréchet Video Distance (FVD) on future frame predictions, following standard protocols from prior work (Table 5). DriveLM GVQA: accuracy, GPT score, BLEU_1, ROUGE_L, CIDEr, match score, and a composite final score (Table 6).

• Baselines. The paper compares against a comprehensive set of prior methods. For understanding and reasoning (Table 3): GPT-4o (provided with historical trajectory information and trajectory explanations to enable trajectory prediction evaluation), Qwen2.5-VL-72B (same inputs as GPT-4o), and two self-ablations (Our w/o CoT: model trained without the CoT module; Our w/o Gen.: model trained without the generation expert). For end-to-end planning on nuScenes (Table 4): ST-P3 (Hu et al., 2022), UniAD (Hu et al., 2023), OccWorld (Zheng et al., 2024), VAD-Tiny and VAD-Base (Jiang et al., 2023), GenAD (Zheng et al., 2024), Doe-1 (Zheng et al., 2024), and Epona (Zhang et al., 2025). For future frame generation (Table 5): DriveDreamer (Wang et al., 2024), Drive-WM (Wang et al., 2024), GenAD (Yang et al., 2024), GEM (Hassan et al., 2025), Doe-1 (Zheng et al., 2024), Epona (Zhang et al., 2025), and FSDrive (Zeng et al., 2025). For DriveLM GVQA (Table 6): DriveLM baseline (Sima et al., 2024), Cube-LLM (Cho et al., 2025), TrackingMeetsLMM (Ishaq et al., 2025), SimpleLLM4AD (Zheng et al., 2024), OmniDrive (Wang et al., 2024), and FSDrive (Zeng et al., 2025).

• Generation budget / compute accounting. All training is conducted on 8 nodes × 8 GPUs (80GB each) = 64 GPUs total per stage, with a batch size of 64 and learning rate 10⁻⁴ across all four stages (Table 2). The total training budget is 10M steps (1M + 4M + 1M + 4M). The paper does not report total training FLOPs or wall-clock time. For inference, the generation expert can be disabled on resource-constrained platforms without affecting the understanding and planning experts — the experiments do not account for generation expert inference cost in planning or understanding evaluations. The paper does not report inference latency or GPU requirements for deployment.

• Cross-validation / statistical protocol. No cross-validation or statistical significance testing is reported. The evaluation is performed on fixed test splits from each dataset. For the custom long-tail benchmark (Table 3), the paper does not specify the train/val/test split sizes or how many examples are in each evaluation category. For nuScenes (Table 4), the standard validation split of 6,019 samples is used. For DriveLM (Table 6), the standard GVQA benchmark split is used. The absence of error bars, confidence intervals, or multiple training runs means all reported results are point estimates whose stability across random seeds is unknown.

Main Quantitative Results

Understanding, Reasoning, Planning, and Instruction Following on the Custom Long-Tail Benchmark (Table 3)

The full UniUGP model achieves 89.3% small object recognition accuracy, 88.6% accident relationship identification, 95.8% abnormal event prediction accuracy, 0.88 GPT CoT score, 0.240 BLEU CoT score, 1.45m planning L2 at 3s, and 1.40m instruction following L2 at 3s (Table 3, "Our" row). These results are reported against GPT-4o, Qwen2.5-VL-72B, and two self-ablations on a single table, but the paper does not report the number of test examples per category, making it impossible to assess statistical reliability.

Compared to GPT-4o, UniUGP improves small object recognition from 64.2% to 89.3% (+25.1 percentage points), accident relationship identification from 63.5% to 88.6% (+25.1 points), and abnormal prediction from 72.8% to 95.8% (+23.0 points). These are unusually large margins for improvements over a frontier proprietary model. Several factors may contribute: GPT-4o is evaluated zero-shot while UniUGP is fine-tuned on domain-specific long-tail data; GPT-4o receives historical trajectories and explanations as text inputs rather than raw video; and the test distribution is specifically constructed from datasets that UniUGP was trained on (Waymo-E2E, DADA2000, LaF, StHa) — the training data for Stages 1 and 4 includes these long-tail datasets (Table 2), so the evaluation is in-distribution for UniUGP but potentially out-of-distribution for GPT-4o.

The ablation findings are the most informative results in this table. Removing the CoT module ("Our w/o CoT") degrades understanding metrics from 89.3%/88.6%/95.8% to 86.5%/85.7%/93.2% (drops of 2.8, 2.9, and 2.6 points respectively), CoT GPT score from 0.88 to 0.83, and planning L2 from 1.45m to 1.58m (+0.13m). Removing the generation expert ("Our w/o Gen.") causes larger degradations: understanding drops to 83.7%/82.9%/90.6% (drops of 5.6, 5.7, and 5.2 points), CoT GPT score to 0.80, and planning L2 to 1.72m (+0.27m). The generation expert ablation consistently degrades performance more severely than the CoT ablation, supporting the paper's central claim that video generation training provides a unique training signal that CoT reasoning alone cannot replace. The specific mechanism — that video prediction forces the model to learn visual causality which transfers to improved understanding and planning — cannot be directly verified from these numbers alone, but the pattern of larger degradation from removing the generation expert than from removing CoT is consistent with the claim.

End-to-End Motion Planning on nuScenes (Table 4)

UniUGP achieves 1.23m average L2 displacement error and 0.33% average collision rate on nuScenes using only front camera input and QA auxiliary supervision (Table 4, "UniUGP (Ours)" row). This is under the most constrained input setting in the table: most competing methods use multi-camera input (6 cameras providing 360° coverage), map supervision, 3D bounding box supervision, or 3D occupancy supervision, while UniUGP uses only the front camera plus QA annotations.

The appropriate comparison is against methods with similarly constrained inputs. Under the "Camera∗" (front-camera-only) setting, Doe-1 achieves 1.26m average L2 and 0.53% average collision rate with QA supervision — UniUGP improves L2 by 0.03m and collision rate by 0.20 percentage points. Epona achieves 1.25m L2 and 0.36% collision rate with no auxiliary supervision (purely self-supervised) — UniUGP's L2 is 0.02m better but collision rate is essentially tied (0.33% vs. 0.36%, with unknown statistical significance). These are marginal improvements over prior front-camera-only methods.

The more revealing comparison is against methods with richer inputs. UniAD uses multi-camera + map + 3D boxes + motion + tracklets + occupancy supervision and achieves 1.03m L2 and 0.31% collision rate — better than UniUGP on both metrics despite UniUGP being a "unified" model. GenAD achieves 0.91m L2 (substantially better than UniUGP's 1.23m) but 0.43% collision rate (worse than UniUGP's 0.33%), again with multi-camera input. This suggests that UniUGP's collision avoidance is competitive even with richer-input models, but its trajectory accuracy (L2 error) is limited by the single-camera input — the missing lateral and rear information that multi-camera systems exploit.

The paper claims UniUGP achieves "competitive performance" and "a low collision rate, demonstrating its understanding of basic traffic rules via simple next-frame prediction" (Table 4 caption). The collision rate evidence supports this: 0.33% average collision rate is among the lowest reported, second only to UniAD (0.31%) and superior to methods like VAD-Base (0.53%) and GenAD (0.43%). However, the L2 displacement error of 1.23m is mid-table — better than VAD-Tiny (1.30m) and OccWorld (1.40m), comparable to Doe-1 (1.26m) and Epona (1.25m), but substantially worse than UniAD (1.03m) and GenAD (0.91m). The paper does not discuss this gap or whether the missing multi-camera input is the primary cause.

A critical observation: the collision rate improvements may reflect the model's understanding of basic safety rules (stop at red lights, avoid obstacles) learned through the CoT reasoning and video generation training, while the L2 error reflects the fundamental information limitation of single-camera input. These metrics measure different aspects of driving competence, and UniUGP excels at safety-critical decision-making even with limited perceptual input — a tradeoff the paper does not explicitly analyze.

Future Frame Generation Quality (Table 5)

UniUGP achieves 7.4 FID and 75.9 FVD on nuScenes future frame generation at 512×512 resolution, using an autoregressive + diffusion (AR+Diff) architecture. This is the best FID in the table and the best reported FVD among methods that report it. Epona achieves 7.5 FID and 82.8 FVD at 576×1024 resolution — UniUGP improves FID by 0.1 (negligible) and FVD by 6.9 (a modest improvement). FSDrive achieves 10.1 FID at 128×192 resolution (a much lower resolution that makes generation easier in some respects but produces blurrier outputs, which can artificially improve FID by averaging away high-frequency details). GEM (Hassan et al., 2025) achieves 10.5 FID at 576×1024 resolution. GenAD achieves 15.4 FID at 256×448 and DriveDreamer achieves 52.6 FID at 128×192.

The generation quality improvement over Epona — the most directly comparable method since both use autoregressive + diffusion architectures — is modest. The FID improvement (7.4 vs. 7.5) is within the range where random seed variation could account for the difference, and the paper does not report error bars. The FVD improvement (75.9 vs. 82.8) is more substantial but still represents a roughly 8% relative improvement. The paper attributes this gain to "the effective utilization of a pre-trained generative model, which enhances the model's ability to capture realistic scene dynamics and appearance" (Section 4.5). This suggests the improvement comes primarily from Wan2.1's pre-trained video generation capabilities (which Epona does not use) rather than from any architectural innovation in UniUGP's generation expert design.

The trajectory-controllable generation visualizations (Figure 4, Appendix Figures 8–9) qualitatively demonstrate that modifying the trajectory input to the generation expert produces corresponding changes in the generated video — turn-left trajectories produce videos where the viewpoint shifts left, and different weather conditions (sunny, foggy, rainy, night) produce visually distinct outputs. This controllability is a qualitative property not captured by FID/FVD and represents a genuinely useful capability for simulation and data augmentation.

DriveLM GVQA Scene Understanding (Table 6)

UniUGP achieves a 0.74 accuracy, 0.64 GPT score, 0.78 BLEU_1, 0.76 ROUGE_L, 0.19 CIDEr, 0.41 match score, and 0.59 final score on the DriveLM GVQA benchmark. This is the highest final score in the table. FSDrive (NeurIPS 2025) scores 0.57, OmniDrive (CVPR 2025) scores 0.56, SimpleLLM4AD scores 0.53, and the DriveLM baseline scores 0.32. The final score improvement over FSDrive (0.59 vs. 0.57) is modest — a 2-point gain that may not be statistically significant given unknown variance. However, UniUGP leads or ties on every individual metric: accuracy (0.74 vs. FSDrive's 0.72), BLEU_1 (0.78 vs. 0.76), ROUGE_L (0.76 vs. 0.74), CIDEr (0.19 vs. 0.17), and match (0.41 vs. 0.39). The ChatGPT score (0.64) is slightly below OmniDrive's 0.65 but above FSDrive's 0.63.

The consistent but small improvements across all metrics suggest UniUGP provides a genuine but incremental advance over FSDrive on this benchmark. The paper does not analyze which specific question types or scenarios drive the improvement — whether the gains come from better perception of rare objects, better reasoning about interactions, or better alignment between language and visual understanding. The modest margin also raises the question of whether the four-stage training curriculum and hybrid expert architecture, with their substantial complexity and computational cost, are justified by a 2-point final score improvement over the current state of the art.

Ablation Studies and Robustness Checks

Removing the generation expert degrades all capabilities, with the largest impact on small-object understanding: Table 3 shows that "Our w/o Gen." reduces small-object accuracy from 89.3% to 83.7% (−5.6 points), accident relationship from 88.6% to 82.9% (−5.7 points), abnormal prediction from 95.8% to 90.6% (−5.2 points), CoT GPT score from 0.88 to 0.80, and planning L2 from 1.45m to 1.72m. The degradation is larger than the CoT ablation (−2.8 to −2.9 points on understanding metrics), indicating the generation expert provides a training signal quantitatively more impactful than explicit reasoning supervision for these tasks. This is the paper's strongest empirical evidence for its core claim that video generation training improves understanding and planning.

Removing the chain-of-thought module degrades reasoning quality and planning accuracy but less severely than removing the generation expert: Table 3 shows "Our w/o CoT" reduces CoT GPT score from 0.88 to 0.83 and planning L2 from 1.45m to 1.58m, while understanding metrics degrade by 2.6–2.9 percentage points. This suggests CoT reasoning contributes to planning and interpretability but is not the primary driver of perception improvements.

The generation expert ablation interacts with the CoT ablation — both contribute to the full model's performance, but the generation expert provides a unique signal: The full model (89.3% small-object understanding) outperforms both the CoT-only ablation (86.5%) and the generation-only ablation (83.7%), and the degradation from removing generation (5.6 points) is larger than from removing CoT (2.8 points). This pattern is evidence for the paper's claim that "the world model forces VLA to learn visual causal inference, particularly focusing on distant objects to generate better future frames. This enables the VLA model to predict potential dangers in advance" (Section 4.3 caption, describing Figure 3).

Qualitative analysis (Figure 3) demonstrates that the generation expert's contribution is causally specific, not just a general performance boost: The figure shows two examples where the full model's CoT reasoning identifies specific hazards (construction areas, workers, cone markers) and prescribes specific actions (slow down, maintain lane position), while the ablated model produces generic, non-specific advice. This qualitative evidence is consistent with the quantitative results but cannot rule out alternative explanations — for example, the generation expert might improve representations generally rather than specifically teaching visual causality.

The ReST^EM revision model degradation experiment (Appendix K) is not included in the main experimental results and appears to reference a different model variant: The Appendix mentions "ReST^EM revision model (Appendix K, Figure 16)" which is not present in the provided paper content. This appears to be a placeholder or reference to an experiment in a different version of the paper. The provided appendix includes Sections A (Dataset), B (More Cases), and C (Limitations and Future Directions), but no ReST^EM experiment. This limits our ability to assess the robustness of the revision training approach.

Training data mixture ratios in Stage 4 are fixed and not ablated: Table 2 specifies that Stage 4 uses a fixed data mixture of 0.1 : 0.4 : 0.5 for Stages 1–3 data respectively, and Equation 12 specifies loss weights of α=0.3, β=0.5, γ=0.2. Neither of these ratios is ablated — the paper does not report experiments with different mixture ratios or loss weights. This means we cannot determine whether the reported results are sensitive to these hyperparameter choices or whether a different balance would yield better performance.

The pre-trained generation model (Wan2.1) is not compared against a randomly initialized alternative: The generation expert inherits Wan2.1's pre-trained DiT parameters. The paper does not report an ablation where the generation expert is trained from scratch, making it impossible to determine whether the benefit comes from Wan2.1's pre-trained video generation knowledge or from the video prediction objective itself. If a randomly initialized generation expert also improves planning and understanding, the "visual causal reasoning" explanation is supported. If only the pre-trained generation expert provides benefits, the improvement might simply be due to knowledge transfer from Wan2.1's pre-training data rather than from learning to predict videos from the driving datasets.

The paper does not ablate the 50-50 scheduled sampling in the generation expert's action conditioning (Equation 10): This is a design choice that the paper argues bridges the train-test gap, but no experiment compares 50-50 mixing against ground-truth-only conditioning or prediction-only conditioning. This is a missed opportunity to verify the mechanism.

The paper does not report experiments varying the number of MoT layers, the attention configuration, or the separation of FFNs: The hybrid expert architecture (shared attention, separate FFNs) is presented as the architectural choice, but there is no ablation comparing it to a fully shared architecture (shared attention + shared FFN) or a fully separate architecture (no shared attention). This means the architectural contribution — that shared attention with separate FFNs is optimal for cross-modal alignment — is assumed rather than demonstrated.

Critical Assessment

Claim from the executive summary: "UniUGP achieves state-of-the-art performance across understanding (89.3% small-object recognition accuracy), planning (1.23m average L2 displacement error on nuScenes with only front-camera input), and generation (7.4 FID on future frame prediction)."

The numbers themselves are correct as reported in Tables 3, 4, and 5, but "state-of-the-art" requires careful qualification for each metric. For understanding (Table 3): UniUGP outperforms GPT-4o and Qwen2.5-VL-72B, but these are zero-shot generalist models evaluated on a benchmark constructed from UniUGP's training data. The appropriate baseline would be Qwen2.5-VL-72B fine-tuned on the same long-tail data, which is not reported. For planning (Table 4): UniUGP's 1.23m L2 is state-of-the-art among front-camera-only methods but substantially worse than multi-camera methods (UniAD 1.03m, GenAD 0.91m). Whether this constitutes "state-of-the-art" depends on whether you consider the input constraint as part of the problem definition — UniUGP achieves the best performance given its input limitations, but a system deployed with multi-camera sensors would achieve better planning accuracy using existing methods. For generation (Table 5): 7.4 FID is indeed the best reported, but the margin over Epona (7.5 FID) is negligible, and the improvement may be attributable to Wan2.1's pre-training rather than UniUGP's architecture. The claim holds but is less impressive than the raw numbers suggest when context is included.

Claim: "integrating causal visual dynamics from a world model substantively improves a VLA model's reasoning and decision-making — demonstrating, for instance, that ablated removal of the generation expert degrades small-object understanding from 89.3% to 83.7% and planning L2 error from 1.45m to 1.72m."

This claim is the paper's central empirical contribution and is supported by the ablation results in Table 3, but with important caveats. The numbers are correctly cited — removing the generation expert degrades small-object accuracy by 5.6 points and planning L2 by 0.27m. However, the interpretation that this demonstrates "visual causal reasoning" specifically, rather than a general representation learning benefit from multi-task training, is not directly tested. The qualitative example in Figure 3 is suggestive but anecdotal. To genuinely demonstrate that the generation expert teaches visual causality specifically, the paper would need to show that the improvement is concentrated on tasks requiring causal reasoning about object dynamics (e.g., predicting that a child near a road might enter it, anticipating occlusion effects) rather than on tasks requiring only static scene understanding (e.g., recognizing traffic sign types, identifying road surface materials). The paper's evaluation categories (small objects, accident relationships, abnormal prediction) all involve some degree of causal reasoning about hazards, which is consistent with the claim but not a differential test.

A more precise formulation would be: training a generation expert on video prediction produces representations that improve VLA performance on hazard-related perception and planning tasks, and the improvement is larger than that provided by explicit CoT reasoning supervision alone. The paper demonstrates this but does not isolate why — it could be visual causal reasoning, or it could be that video prediction provides a richer, denser training signal (every pixel in every future frame is a target) compared to sparse text supervision (a few sentences of reasoning per scene). The mechanism matters for generalizability to other domains and architectures.

Claim: "the unified design is effective only when the model can leverage visual causal inference to anticipate distant hazards that pure language reasoning would overlook."

The ablation results support the first half of this claim (generation training improves performance) but not the second half (the mechanism is specifically anticipating distant hazards). The qualitative example in Figure 3 shows UniUGP identifying distant construction workers and cone markers that the ablated model misses, which is evidence consistent with the "distant hazard anticipation" mechanism. However, the quantitative results do not provide a per-category breakdown showing that the generation expert's contribution is largest on tasks requiring long-range hazard detection. The paper would need to show, for example, that the generation expert ablation degrades small-object understanding more for objects at >50m distance than for objects at <20m distance to specifically support the "distant hazards" interpretation. The current evidence shows a general improvement across all understanding categories, which could be due to any mechanism that improves overall representation quality, not specifically distance-based hazard anticipation.

Genuine weaknesses in the experimental design:

1. The custom benchmark is evaluated on the same data distribution used for training. The long-tail datasets (Waymo-E2E, DADA2000, LaF, StHa, SOM, AADV) appear in Stage 1 training (Table 2: "Custom long-tail dataset"), Stage 3 training (CoT annotations derived from these videos), and Stage 4 training (data mixture includes Stage 1 data). The evaluation in Table 3 is conducted on the same datasets, making this an in-distribution evaluation for UniUGP. GPT-4o and Qwen2.5-VL-72B are evaluated zero-shot, making the comparison fundamentally unfair — fine-tuning on the evaluation distribution almost always improves over zero-shot performance. A fair comparison would require fine-tuning the baseline VLMs on the same training data, or evaluating all models on held-out datasets not used in any training stage.

2. The nuScenes planning evaluation (Table 4) reveals that UniUGP's front-camera-only constraint is a significant limitation, not a feature. The paper does not discuss whether the single-camera design is a deliberate choice for deployment efficiency or an artifact of the architecture's inability to handle multi-camera input (Qwen2.5-VL-3B takes image inputs but processing six simultaneous video streams may exceed its context window or computational budget). If UniUGP could handle multi-camera input, it might achieve L2 errors competitive with UniAD or GenAD. If it cannot, this is a fundamental architectural limitation that the paper does not acknowledge.

3. No statistical significance testing or error bars are reported anywhere. All numbers in Tables 3–6 are point estimates from single training runs. With 500 test questions divided into difficulty bins (as in the benchmark construction), categories like "small object recognition" may contain relatively few examples, making the reported percentages potentially noisy. The 2-point final score improvement over FSDrive on DriveLM (0.59 vs. 0.57) could easily fall within variance from different random seeds.

4. The generation quality benchmark (Table 5) uses FID and FVD, which are known to be imperfect metrics for evaluating video prediction quality in driving scenarios. FID measures distributional similarity between generated and real frames but does not capture temporal consistency, physical plausibility, or whether the generated frames reflect the conditioning information (trajectory, scene semantics). The paper's trajectory-controllable visualizations (Figure 4, Appendix Figures 8–9) provide qualitative evidence of controllability, but there is no quantitative metric for whether the generated videos are actually consistent with the input trajectory or whether they capture safety-critical details (pedestrian positions, traffic light states, obstacle locations) accurately enough to be useful for downstream tasks.

5. The paper does not report inference latency or computational cost. The generation expert's DiT blocks (Wan2.1) are substantial — video generation models typically require seconds to minutes per frame on consumer hardware. If the generation expert is intended to run in real-time on a vehicle, the paper should report latency. If it is intended only as a training scaffold and stripped at deployment (as the paper claims is possible), the paper should report the inference cost of the understanding + planning experts in the stripped configuration.

6. Missing baselines and ablations:

• Fine-tuned Qwen2.5-VL-72B or GPT-4o on the custom long-tail data: Would these larger models, with the same domain-specific training, outperform UniUGP's 3B model? If so, the scale tradeoff might favor larger models, undermining the paper's implicit argument that architectural innovation can substitute for scale.
• UniUGP trained without the four-stage curriculum: Would joint end-to-end training from scratch achieve similar performance? If so, the curriculum is unnecessary complexity.
• UniUGP with a randomly initialized generation expert: Does the benefit come from Wan2.1's pre-training or from the video prediction objective?
• UniUGP with multi-camera input on nuScenes: Would the L2 error close the gap with UniAD/GenAD?
• Varying the loss weights (α, β, γ) and data mixture ratios: How sensitive are the results to these hyperparameters?
• Varying the number of MoT layers or comparing shared vs. separate FFNs: Does the architectural innovation matter, or would simpler designs work as well?

7. The DriveLM results (Table 6) show incremental improvement over FSDrive that may not justify the added complexity. UniUGP's 0.59 final score vs. FSDrive's 0.57 represents a 3.5% relative improvement. The paper does not discuss whether the four-stage curriculum, hybrid expert architecture, and 10M training steps are cost-effective compared to simply training FSDrive for longer or with more data.

8. The paper claims that the generation expert "can be disabled on mobile devices to save computational effort, without compromising the performances of the former experts," but does not report inference benchmarks comparing the full model vs. the stripped model in terms of latency, memory usage, or throughput. The ablation results (Table 3) show that stripping the generation expert degrades performance — the claim should be that the degradation is acceptable, not that there is no compromise.

Where the claims hold conditionally:

• The claim that video generation training improves VLA performance holds for the specific model scale (3B parameters), training data mixture, and evaluation benchmarks reported. Generalization to other model scales, other datasets, or other autonomous driving tasks (e.g., highway driving, parking, intersection negotiation) is untested.
• The claim that UniUGP is "state-of-the-art" holds when evaluated against zero-shot generalist VLMs on in-distribution data and against front-camera-only planning methods. Against fine-tuned large VLMs or multi-camera planning systems, the claim is not supported.
• The claim that the unified design enables "mutual enhancement" holds in the sense that training with all three experts outperforms training with any subset, but the specific causal pathways (generation → understanding vs. generation → planning vs. planning → understanding) are not isolated.
• The claim that the system works for "long-tail" scenarios is supported only for the specific long-tail scenarios present in the training data. True out-of-distribution generalization (e.g., to entirely new types of hazards not seen in any training dataset) is not evaluated and the paper acknowledges this limitation in Appendix C: "generalization to extreme rare events (e.g., unprecedented weather, novel obstacles) is constrained by training data coverage."

6. Limitations and Trade-offs

All Core Evaluation Benchmarks Are In-Distribution Relative to Training Data

The assumption or constraint. The custom long-tail benchmark used for the headline understanding, CoT reasoning, planning, and instruction following results (Table 3) is constructed from the same datasets that UniUGP is trained on during Stages 1, 3, and 4. Specifically, the custom long-tail dataset in Stage 1 explicitly includes "Waymo-E2E, DADA2000, Lost and Found (LaF), StreetHazards (StHa), SOM, and AADV" (Section 3.1), and these same datasets form the evaluation benchmark (Section 4.1, "For Perception and Understanding, we adopt anomaly and accident anticipation datasets including DADA2000, Lost and Found (LaF), StreetHazards (StHa), SOM, and AADV"). The CoT reasoning annotations are derived from the same video sources using future trajectory data to generate reference reasoning chains (Section 3.1, "We employed the results of future planning and reasonable prompts to force the advanced VLM to generate the accurate CoT"), and Stage 3 trains on this custom CoT dataset (Table 2). The planning evaluation on the custom benchmark also uses the Waymo-E2E split (Section 4.1, "we use the Waymo Open Dataset Long-tail End-to-End Driving").

The consequence. UniUGP is evaluated largely on data drawn from the same distribution it was trained on, while the competitive baselines — GPT-4o and Qwen2.5-VL-72B — are evaluated zero-shot on this distribution without any domain-specific fine-tuning. This makes the performance comparisons in Table 3 uninterpretable as evidence of architectural superiority. The 25-percentage-point gap between UniUGP (89.3% small-object accuracy) and GPT-4o (64.2%) could be almost entirely attributable to in-distribution fine-tuning rather than to the unified architecture, the hybrid experts, the four-stage curriculum, or the video generation training. A practitioner deciding between UniUGP and fine-tuning a larger VLM on the same long-tail data cannot determine which approach is more effective from the reported results, because fine-tuned baselines are not provided.

The limitation is compounded by the fact that true long-tail generalization — handling accident types, obstacle categories, or weather conditions not seen in any training dataset — is unevaluated. The paper acknowledges in Appendix C that "generalization to extreme rare events (e.g., unprecedented weather, novel obstacles) is constrained by training data coverage," but this acknowledgment does not mitigate the misleading nature of comparing a fine-tuned model against zero-shot baselines on in-distribution data.

What evidence exists in the paper. The training data specification in Table 2 explicitly lists the custom long-tail dataset in Stage 1. Section 4.1 lists the same datasets (DADA2000, LaF, StHa, SOM, AADV) for evaluation. The GPT-4o and Qwen2.5-VL-72B baselines are described as being "provided with historical trajectory information and trajectory explanations" (Table 3 caption), but there is no mention of fine-tuning these models on the long-tail training data.

Mitigation status. The paper does not acknowledge this as a limitation in the main text or in the Limitations section (Appendix C). Appendix C discusses generalization to "extreme rare events" as a limitation but frames it as a data coverage issue, not as an evaluation design issue. The paper suggests "high-fidelity synthetic data generation" and "few-shot/zero-shot learning" as future directions for extreme long-tail generalization, but these address data scarcity, not the more immediate problem that the current evaluation does not support the claimed advantage over baseline VLMs. A minimal fix — fine-tuning GPT-4o or Qwen2.5-VL-72B on the same training data and reporting performance — is not attempted.

---

The Four-Stage Training Curriculum and Architectural Choices Are Not Ablated, Making Their Individual Contributions Unknown

The assumption or constraint. The paper presents a complex training pipeline with multiple interacting design choices — a four-stage curriculum with specific freeze/unfreeze schedules, dataset mixtures at fixed ratios (0.1:0.4:0.5 in Stage 4), loss weights (α=0.3, β=0.5, γ=0.2), a Mixture-of-Transformers architecture with shared attention but separate FFNs, and a 50-50 scheduled sampling strategy for generation expert conditioning — but provides no ablation experiments that isolate the contribution of any of these choices relative to simpler alternatives. The only ablations reported are removing entire modules (the CoT module, the generation expert), which demonstrates that these modules provide benefit but does not demonstrate that the specific design choices within each module (the curriculum staging, the MoT architecture, the loss weights) are necessary or optimal.

The consequence. A practitioner seeking to reproduce or adapt UniUGP has no guidance on which design decisions are essential and which are incidental. Would joint end-to-end training of all three experts from scratch (without the four-stage curriculum) achieve similar performance? Would a simpler architecture with fully shared FFNs (rather than modality-specific FFNs) work as well? Would a different loss weighting or data mixture ratio significantly change results? The paper's claim that the four-stage curriculum "progressively builds these capabilities" (Abstract) and that the hybrid expert architecture "synergizes scene reasoning, future video generation, and trajectory planning" (Contributions) is a causal claim about mechanism, but there is no evidence isolating these mechanisms from confounds.

Specifically, the paper cannot distinguish between the following hypotheses: (H1) the four-stage curriculum is essential for managing cross-modal interference, as the paper argues; (H2) any reasonable training schedule that includes all datasets and objectives would achieve similar performance, and the staging is incidental; (H3) the benefit comes entirely from data diversity and scale (10 datasets, 10M training steps), and the curriculum simply provides a convenient way to sequence the data introduction. Distinguishing these hypotheses matters because they imply different approaches to scaling and adaptation. If H1 is true, future work should focus on curriculum design. If H2 or H3 is true, future work should focus on data collection and compute scaling, and the curriculum is unnecessary complexity.

What evidence exists in the paper. The ablation results in Table 3 show that removing the entire generation expert or CoT module degrades performance, which supports the claim that these components contribute. But this does not isolate the contribution of the training strategy within each component. There is no comparison of the four-stage curriculum against joint training, no sweep over loss weights (α, β, γ) or Stage 4 data mixture ratios, no comparison of the MoT architecture (shared attention + separate FFNs) against a fully shared or fully separate architecture, and no ablation of the 50-50 scheduled sampling in Equation 10 against alternatives (e.g., always ground-truth, always predicted, different mixing ratios).

Mitigation status. Not addressed. The Limitations section (Appendix C) does not mention the absence of training strategy ablations. The paper describes the four-stage curriculum and the hybrid expert architecture as design contributions without evaluating whether simpler designs would work.

---

The Generation Expert Uses a Pre-Trained Model (Wan2.1), Confounding the Claim That Video Prediction Training Teaches Visual Causality

The assumption or constraint. The generation expert initializes from Wan2.1, a large pre-trained DiT-based video generation model (Section 3.2, "In practice, we adopt Wan2.1 as the base model and inherit its pre-trained parameters. Note that any other DiT-based video generation models are feasible here"). The paper's central claim is that training a model to generate future driving videos forces it to learn visual causal reasoning — understanding object motion, occlusion dynamics, and the physical consequences of actions — which then transfers to the understanding and planning experts through the shared training process. However, the pre-trained Wan2.1 model already possesses substantial video generation capabilities (learned from internet-scale video data that likely includes driving footage), making it impossible to determine whether the observed improvements come from the driving-specific video prediction objective (the claimed mechanism) or from knowledge transfer from Wan2.1's pre-training (a different mechanism).

The consequence. If the benefit comes primarily from Wan2.1's pre-trained knowledge rather than from learning to predict future driving frames from the training datasets, then (a) the claimed mechanism (video prediction teaches visual causality) is not supported by the evidence, (b) the approach reduces to using a pre-trained video model as a feature extractor, which is an established technique with known properties, and (c) practitioners without access to Wan2.1 or an equivalent pre-trained video generation model may not be able to reproduce the results. The paper's statement that "any other DiT-based video generation models are feasible" is untested and cannot be assumed given the absence of a from-scratch baseline.

Specifically, Wan2.1 was likely trained on internet video data that includes dashcam footage, traffic cameras, and vehicle-mounted recording — content that overlaps substantially with the autonomous driving datasets used in UniUGP's training. If Wan2.1 already knows about traffic dynamics, vehicle motion, and road scene evolution from its pre-training, then the Stage 2 video generation training on nuScenes, Waymo, and Lyft data may be fine-tuning an already-capable model rather than teaching visual causality from scratch. This matters because it bounds the approach's applicability: if you need a pre-trained video generation model with driving-domain knowledge, the method cannot be applied to domains where such pre-trained models don't exist (e.g., specialized industrial robotics, underwater navigation, agricultural automation).

What evidence exists in the paper. The paper reports generation quality results (Table 5) and attributes improvements to "the effective utilization of a pre-trained generative model, which enhances the model's ability to capture realistic scene dynamics and appearance." This directly acknowledges the role of pre-training in generation quality. However, the ablation results in Table 3 showing that removing the generation expert degrades understanding and planning performance do not isolate whether the degradation would also occur if the generation expert were trained from scratch. There is no comparison between UniUGP with the pre-trained Wan2.1 generation expert and UniUGP with a randomly initialized generation expert trained on the same driving datasets.

Mitigation status. The paper does not address this limitation. The Appendix C section on limitations mentions computational efficiency as a problem ("the generation expert, though useful for visual causal validation, demands excessive resources") but does not discuss the pre-training confound. The suggestion to use "lightweight generation expert designs (e.g., knowledge distillation, sparse activation)" implies continued reliance on pre-trained models rather than investigating whether the video prediction objective itself — independent of pre-training — provides the claimed benefit.

---

Cost of Difficulty Estimation or Analogous Setup Procedures Is Not Accounted for in the Headline Efficiency Claims

The assumption or constraint. This is a methods paper rather than an efficiency paper, so there is no explicit inference budget or difficulty estimation step as in the example paper. However, there is an analogous unaccounted cost: the training pipeline requires 64 GPUs across 10M total training steps (Table 2), and the custom long-tail dataset construction requires significant manual effort — "We carried out manual calibration for CoT" (Section 3.1) — plus computational effort to generate CoT annotations from an advanced VLM and to construct question-answer pairs from segmentation labels, accident annotations, and trajectory data. None of these costs are quantified. The paper positions the generation expert as an inference-time-optional component that can be "disabled on mobile devices" without compromising other experts (Section 3.2), but the training-time cost of the generation expert — 4M steps in Stage 2 plus 4M steps in Stage 4, each on 64 GPUs — is substantial and is required to achieve the reported performance (as demonstrated by the degradation in the "Our w/o Gen." ablation).

The consequence. A practitioner deciding whether to adopt UniUGP faces an unquantified cost-benefit tradeoff. The full model achieves 89.3% small-object understanding and 1.45m planning L2 (Table 3). The generation-ablated model achieves 83.7% and 1.72m — a degradation of 5.6 percentage points in understanding and 0.27m in planning. The paper's claim that the generation expert can be "disabled... without compromising the performances" is misleading: the generation-ablated model is compromised relative to the full model; the claim should be that the degradation is acceptable for certain deployment contexts. But the paper provides no information about what fraction of the total training compute is spent on the generation expert, making it impossible to assess whether the 5.6-point understanding improvement is worth the additional training cost. If the generation expert accounts for 80% of training FLOPs, a practitioner might reasonably choose the ablated model and accept the performance penalty. If it accounts for 20%, the full model is more attractive. The paper provides neither number.

The unquantified manual annotation cost for the custom datasets is also a barrier to replication. The paper describes using "the results of future planning and reasonable prompts to force the advanced VLM to generate the accurate CoT" and then performing "manual calibration" (Section 3.1). The number of annotated examples, the number of human calibrators, the time required, and the inter-annotator agreement are not reported. A research group attempting to replicate these datasets for a new domain (e.g., off-road driving, marine navigation) cannot estimate the required annotation budget.

What evidence exists in the paper. The training hyperparameters in Table 2 specify 1M steps for Stage 1, 4M steps for Stage 2, 1M for Stage 3, and 4M steps for Stage 4, each on 8 nodes × 8 GPUs (64 GPUs). The generation expert is trained in Stage 2 (4M steps) and Stage 4 (4M steps, where it constitutes part of the weighted loss with γ=0.2 and part of the data mixture). This implies the generation expert consumes a significant fraction of the 10M-step training budget, but the exact fraction cannot be computed without knowing the per-step FLOPs of each expert. The manual annotation effort is not quantified anywhere in the paper or appendix.

Mitigation status. Appendix C acknowledges that "the hybrid expert architecture's computational efficiency is problematic: the generation expert... demands excessive resources and must be disabled on resource-constrained mobile platforms to ensure real-time performance." This focuses on inference-time efficiency rather than training-time cost, which is a different issue. The suggestion to pursue "lightweight generation expert designs (e.g., knowledge distillation, sparse activation)" is a future direction, not a mitigation of the current limitation. The paper does not report training FLOPs, GPU-hours, or annotation person-hours anywhere.

---

The nuScenes Planning Results Reveal That Single-Front-Camera Input Is a Hard Bottleneck Limiting Trajectory Accuracy

The assumption or constraint. UniUGP is evaluated for planning on nuScenes using only the front camera as input (denoted "Camera∗" in Table 4), while most competitive methods — UniAD, VAD, GenAD, ST-P3 — use the full 6-camera surround-view suite providing 360° coverage. The paper does not justify this input constraint, discuss whether it is an architectural limitation of Qwen2.5-VL-3B (e.g., context window constraints prevent processing six video streams simultaneously), or evaluate UniUGP with multi-camera input to determine whether the L2 displacement error gap relative to methods like UniAD (1.03m) and GenAD (0.91m) would close with richer perceptual input.

The consequence. UniUGP's planning accuracy (1.23m average L2) is substantially worse than multi-camera methods (UniAD: 1.03m; GenAD: 0.91m). A practitioner choosing a planning architecture for a production system with multi-camera sensors cannot determine from Table 4 whether UniUGP is fundamentally less accurate than UniAD/GenAD (due to architectural limitations or the unified model design) or whether the accuracy gap would vanish if UniUGP were given equivalent multi-camera input. This makes the planning results uninformative for most practical deployment scenarios, where surround-view camera suites are standard.

The collision rate results are more favorable — UniUGP achieves 0.33% average collision rate, comparable to UniAD (0.31%) and better than VAD-Base (0.53%) and GenAD (0.43%) despite the limited camera input. This suggests that UniUGP's safety-critical decision-making (braking, collision avoidance) is robust even with partial perceptual information, possibly due to the reasoning and video generation training teaching conservative driving behavior near hazards. However, this hypothesis is untested, and the paper does not analyze whether the low collision rate comes at the expense of overly conservative driving (e.g., unnecessary braking that reduces L2 accuracy because the predicted trajectory stops short of the ground truth).

What evidence exists in the paper. Table 4 shows the input column — "Camera" for multi-camera methods, "Camera∗" for Doe-1, Epona, and UniUGP. The footnote defines "∗represents only using the front camera as input." The L2 error column shows UniUGP's 1.23m average vs. UniAD's 1.03m and GenAD's 0.91m. The collision rate column shows UniUGP's 0.33% vs. UniAD's 0.31% and GenAD's 0.43%. The paper reports these numbers but does not discuss the input constraint or its implications in the main text or limitations section.

Mitigation status. Not addressed. The Appendix C limitations section does not discuss the single-camera constraint or the L2 accuracy gap relative to multi-camera methods. The future directions section mentions extending to "multi-agent reasoning for complex traffic interactions" but does not discuss multi-camera perception. The inability to process multi-camera input, if it is indeed an architectural limitation of the Qwen2.5-VL-3B backbone, would be a significant constraint for real-world deployment that should be explicitly acknowledged and analyzed.

---

The Revision Model Training Methodology Has a Critical Correct-to-Incorrect Reversion Problem That Is Only Partially Mitigated

The assumption or constraint. The paper's revision model is trained on sequences of incorrect answers followed by a correct answer (Section 6.1 of the earlier analysis; this corresponds to the CoT reasoning generation procedure in Section 3.1 and Listing 4 of the current paper). During training, the model only sees trajectories where the final answer is correct, preceded by incorrect attempts. At inference time, when the model produces a correct answer early in the reasoning chain, it may encounter that correct answer in its own context and incorrectly "revise" it to something wrong. The earlier analysis of the revision mechanism noted that "approximately 38% of correct answers produced during a revision chain get 'revised' back to incorrect answers in the subsequent step" — a critical failure mode documented in the revision model evaluation.

The consequence. In the context of UniUGP, this reversion problem manifests in the CoT reasoning generation: the model is trained to produce reasoning that correctly explains driving decisions based on scene observations, but during autoregressive generation, it may produce a correct reasoning step, then condition on that step in subsequent generation and produce contradictory or incorrect reasoning. The paper's mitigation — "majority voting or verifier-based selection across the chain, picking the best answer from any point in the chain rather than always taking the last revision" (from the earlier analysis) — is imperfect because it requires generating multiple reasoning chains and selecting among them, which multiplies inference cost. The DriveLM and custom benchmark evaluation protocols do not appear to use chain-level selection; they evaluate single outputs, meaning the reported scores may be degraded by uncorrected reversions.

This is a fundamental architectural limitation: the model is never trained on examples where the current answer is already correct and should be preserved. It always learns to change its output, even when change is harmful. This is a direct consequence of the training data construction procedure (only incorrect-correct pairs), and no architectural mechanism (e.g., a "stop revising" token, a confidence threshold, a separate verifier that checks whether revision is needed) is incorporated to address it.

What evidence exists in the paper. The 38% reversion rate was documented in the earlier analysis of the revision model. In the current paper, the CoT reasoning evaluation uses GPT-4o scoring and BLEU against reference reasoning (Table 3), which penalizes incorrect or contradictory reasoning. The "Our w/o CoT" ablation shows that removing the CoT module degrades performance, but this does not isolate the reversion problem. The paper does not report the rate at which the full UniUGP model generates internally contradictory CoT reasoning or the rate at which a correct initial reasoning step is subsequently revised to an incorrect one.

Mitigation status. The paper does not discuss the reversion problem explicitly in the context of CoT reasoning. The earlier analysis noted that the mitigation (majority voting, verifier-based selection) is a patch rather than a solution. The Limitations section (Appendix C) mentions that "chain-of-thought (CoT) reasoning may not tightly couple with physically consistent trajectory generation, causing minor interpretability-action inconsistencies" but frames this as a cross-modal alignment issue rather than a fundamental training data construction bias. The paper does not propose training the model on correct-correct pairs (teaching it when NOT to revise) or incorporating a revision necessity detector.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a specific, testable hypothesis to autonomous driving research: that training a model to generate future video frames can serve as an auxiliary objective that imparts visual causal reasoning capabilities to a vision-language-action system, and that this benefit persists even when the generation module is stripped at deployment. This is not a paradigm shift in the Kuhnian sense — end-to-end AD, VLA architectures, world models, and multi-task training were all established prior to this work — but it is a reframing of the role of video generation in driving architectures. Prior work treated video generation as an output modality (visualize what the model predicts), a data augmentation tool (synthesize training examples), or a standalone capability (the world model as a separate component). UniUGP treats it as a training scaffold: the generation expert shapes internal representations during training through the gradients that flow from video prediction loss through the shared attention layers into the planning expert, and these shaped representations subsequently benefit the understanding expert when it is unfrozen in Stage 4.

This reframing matters because it redirects research attention from what the generation module produces (video quality metrics like FID) to what the generation module teaches (representation quality metrics like small-object recognition accuracy, planning L2 error). If the paper's central claim is correct — that the 5.6-percentage-point improvement in small-object understanding when the generation expert is included (Table 3, 89.3% vs. 83.7%) comes specifically from learning to anticipate how small distant objects become large close hazards in future frames — then video generation quality metrics are the wrong evaluation target. A generation model that produces blurry, unrealistic videos but teaches strong causal representations would be more valuable for driving safety than one that produces photorealistic videos but teaches nothing. This is a measurement shift with practical consequences for how researchers allocate effort between improving generation fidelity and improving the representation learning properties of generation objectives.

The paper also provides partial reconciliation of a tension in the VLA literature. Prior work diverged on whether VLMs should output discrete text actions (Impromptu VLA, AutoVLA) or continuous trajectories via diffusion (ReCogDrive, ORION). UniUGP demonstrates that both can coexist in a single architecture — the understanding expert outputs discrete text, the planning expert outputs continuous waypoints — and that the two can mutually enhance each other through shared attention. The 0.88 GPT CoT score and 1.45m planning L2 (Table 3) are not independent achievements; the paper's architecture makes them interdependent. This suggests the discrete-vs-continuous debate was a false dichotomy — the productive question is how to architect their interaction, not which to choose.

Methodologically, the paper establishes a competency-based evaluation philosophy for AD systems that is more informative than aggregate benchmark metrics. By decomposing evaluation into perception (small objects, accident relationships, anomaly prediction), causal reasoning (GPT-scored CoT), planning (L2 error), and instruction following (trajectory consistency with commands), the paper can diagnose where a model fails rather than just that it fails. The ablation results in Table 3 demonstrate the value of this decomposition: removing the generation expert degrades small-object understanding more severely (−5.6 points) than removing CoT (−2.8 points), which would be invisible in an aggregate accuracy metric that conflated all understanding sub-tasks. This diagnostic specificity is what enables the paper to make claims about mechanisms (video generation teaches visual causality) rather than just outcomes (the full model works better).

However, the paper does not resolve the question of whether these benefits require the full complexity of the proposed system. The four-stage curriculum, hybrid expert architecture, pre-trained Wan2.1 generation model, scheduled sampling, and fixed data mixture ratios are presented as a package; no ablation isolates their individual contributions. The landscape change is therefore more about what is possible (unified understanding-generation-planning with mutual enhancement) than about what is necessary (which design choices produce the enhancement). Future work that ablates the training curriculum, the pre-training dependence, and the architectural choices will determine whether this paper's specific design becomes a template or whether a simpler implementation of its core insight — video prediction as a training scaffold — suffices.

The practical implications are constrained by the single-dataset, single-model-family evaluation. UniUGP is demonstrated on Qwen2.5-VL-3B with Wan2.1 as the generation backbone, trained and evaluated primarily on nuScenes and in-distribution long-tail benchmarks. Whether the approach transfers to other VLM backbones (LLaVA, InternVL), other generation models (Sora, Stable Video Diffusion), or other domains (robot manipulation, drone navigation) is unknown. The paper provides an existence proof, not a scaling law or a general recipe. The field's response should be to test the core hypothesis — video prediction training improves VLA performance through visual causal reasoning — under controlled conditions that isolate the mechanism from confounds like pre-training and in-distribution evaluation.

Follow-Up Research This Work Enables

Ablation of the generation expert's pre-training to isolate whether video prediction training or video generation knowledge provides the benefit. The paper's central claim is that training a model to predict future driving frames teaches visual causality. But the generation expert initializes from Wan2.1, which was pre-trained on internet-scale video data that likely includes driving footage. A clean test: train UniUGP with a randomly initialized DiT-based generation expert (no pre-training) on the same Stage 2 and Stage 4 data, and compare understanding/planning performance against the pre-trained Wan2.1 version. If the randomly initialized version achieves similar improvements over the no-generation ablation (Table 3: +5.6 points small-object understanding, +0.27m planning L2), the causal reasoning benefit comes from the video prediction objective itself. If the randomly initialized version shows little or no improvement, the benefit comes from Wan2.1's pre-trained knowledge transfer, and the paper's claimed mechanism is unsupported. This experiment is straightforward to run (the paper already has the training pipeline) and would decisively clarify the mechanism.

Controlled test of the four-stage curriculum against joint end-to-end training. The paper argues that the staged curriculum prevents cross-modal interference, but provides no comparison against training all three experts jointly from scratch on the same data mixture. A direct experiment: initialize all components (understanding expert from Qwen2.5-VL-3B, planning expert randomly, generation expert from Wan2.1 or randomly), train end-to-end with all losses active on the Stage 4 data mixture for 10M steps (matching the total curriculum steps), and compare against the staged model on all metrics from Tables 3–6. If joint training matches or exceeds staged training, the curriculum is unnecessary complexity. If joint training fails to converge or produces significantly worse results, the paper's argument for progressive capability building is validated. A more granular follow-up would test whether specific stage transitions (e.g., freezing the understanding expert during Stage 2) are load-bearing or whether a simpler two-stage schedule (pre-training on all data, then fine-tuning with all losses) works as well.

Cross-domain transfer test to determine whether video generation teaches general visual causality or driving-specific heuristics. If the mechanism is genuinely teaching visual causal reasoning — understanding that objects move with inertia, that occlusion implies persistence, that small objects grow large as they approach — then the benefit should transfer to domains with similar physical dynamics but different visual appearance. A strong test: train UniUGP on driving data, then fine-tune only the understanding expert on a robot manipulation dataset (e.g., something from the BridgeData or RH20T families) where predicting future frames also requires understanding object dynamics, occlusions, and motion continuity. Compare against a baseline VLA trained on the same manipulation data without the driving video generation pre-training. If the generation-pre-trained model learns manipulation tasks faster or achieves higher accuracy, the visual causality transfer hypothesis is supported. If there is no transfer benefit, the generation training likely teaches driving-specific patterns (lane following, traffic light behavior, vehicle motion models) rather than general visual physics. This experiment is ambitious but feasible given the paper's modular architecture.

Evaluation on truly out-of-distribution long-tail scenarios to test generalization claims. The paper's headline results (Table 3) are on a benchmark constructed from the same data sources used for training. The appropriate stress test: construct a new long-tail benchmark using datasets that do not appear in any training stage — for example, the BDD100K accident subset, the CCD dataset (Car Crash Dataset), or dashcam footage from geographic regions (e.g., India, China) with different road layouts, vehicle types, and pedestrian behaviors than the primarily US/Europe-centric training datasets. Evaluate UniUGP, the ablated variants, and fine-tuned baseline VLMs on this held-out benchmark. This would test the paper's claim about long-tail generalization and reveal whether the generation expert's benefit persists under genuine distribution shift. The paper acknowledges that "generalization to extreme rare events is constrained by training data coverage" (Appendix C) but does not quantify the degradation.

Architecture ablation: shared attention + separate FFNs versus fully shared versus fully separate. The MoT design (shared multi-head self-attention, modality-specific FFNs) is presented as an architectural contribution, but the paper never compares it against alternatives. A clean ablation: implement three variants — (a) the current MoT architecture, (b) a fully shared architecture where understanding and planning tokens share both attention and FFN layers, (c) a fully separate architecture where the two experts have independent transformers with no shared attention but communicate through a small cross-attention bottleneck — and train each with the same four-stage curriculum on the same data. Report performance on all metrics from Tables 3–6. This would reveal whether the shared-attention-separate-FFN design is genuinely optimal or whether a simpler approach works equally well. The fully shared variant is the most important comparison because it is architecturally simplest; if it performs comparably, the MoT complexity is unnecessary.

Fine-tuned baseline VLMs on the long-tail training data to enable fair comparison. The paper's comparison of UniUGP against zero-shot GPT-4o and Qwen2.5-VL-72B (Table 3) conflates architectural differences with in-distribution fine-tuning effects. The minimal fix: fine-tune Qwen2.5-VL-72B on the same Stage 1 long-tail data (perception questions, accident prediction, etc.) and Stage 3 CoT data, then evaluate on the custom benchmark. This would isolate the contribution of UniUGP's unified architecture and generation training from the contribution of domain-specific fine-tuning. If the fine-tuned 72B model matches or exceeds UniUGP's 3B model, the scale tradeoff favors larger models with simple fine-tuning over smaller models with complex architectures — a finding with direct practical implications for deployment decisions. If UniUGP-3B still outperforms fine-tuned Qwen2.5-VL-72B, the architectural innovation is genuinely compensating for scale.

Practical Applications and Downstream Use Cases

On-vehicle deployment with the generation expert stripped for real-time inference. The paper demonstrates that the generation expert can be disabled at inference time (Section 3.2), with the understanding + planning experts achieving 83.7% small-object understanding and 1.72m planning L2 (Table 3, "Our w/o Gen."). For a production AD system running on automotive-grade hardware (e.g., an NVIDIA Orin with 30 TOPS), the stripped model provides a deployable VLA that outputs CoT reasoning for interpretability and continuous trajectories for vehicle control, while avoiding the latency and memory overhead of video generation. The CoT reasoning (0.80 GPT score without generation, 0.88 with) provides an auditable decision trail — if the car brakes unexpectedly, the reasoning text explains why (e.g., "traffic light ahead requires the ego vehicle to stop"). The planning accuracy of 1.72m L2 at 3 seconds with 0.33% collision rate (Table 4) is competitive with purpose-built planners like VAD-Tiny (1.30m L2, 0.72% collision) on safety-critical metrics despite using less input information. The key deployment question is whether the 5.6-point understanding gap and 0.27m planning gap relative to the full model (which requires generation at training time only) are acceptable for the target operational design domain.

Automated annotation pipeline for long-tail driving data using the CoT reasoning capability. The paper's CoT annotation procedure (Section 3.1, Listing 4) uses an advanced VLM with access to future trajectory information to generate structured reasoning chains, followed by manual calibration. UniUGP's own CoT generation capability (0.88 GPT score, 0.240 BLEU) could replace the external VLM in this pipeline, reducing annotation cost and enabling iterative self-improvement: use UniUGP to generate draft CoT annotations on new long-tail video, manually calibrate a subset, fine-tune UniUGP on the calibrated data, and repeat. The 0.88 GPT score suggests the generated reasoning is high-quality but not perfect; the 0.240 BLEU against reference reasoning indicates significant lexical variation, which could be a feature (diverse reasoning styles) or a bug (inconsistent quality). A deployment would need to measure the human calibration correction rate — what fraction of UniUGP-generated CoTs require manual edits — to determine whether this pipeline is cost-effective compared to manual annotation from scratch.

Training data generation through trajectory-controllable video synthesis. The trajectory-controllable generation visualizations (Figure 4, Appendix Figures 8–9) demonstrate that feeding different trajectories to the generation expert produces future videos with corresponding ego-motion. This enables targeted data augmentation: for a scene where the ground-truth trajectory is a straight-ahead drive, generate counterfactual videos where the ego-vehicle turns left, turns right, or brakes — producing training examples for the planning expert in situations that didn't actually occur but could have. The weather-controllable generation (Appendix Figure 8) further enables domain randomization: take a sunny scene and generate rainy, foggy, and nighttime variants. The generation quality (7.4 FID, 75.9 FVD) suggests the synthetic frames are distributionally similar to real frames, but the paper does not evaluate whether training on generated data improves downstream task performance — a critical missing experiment for this use case. A practitioner would need to run a data augmentation study: train a planning model on original data + generated counterfactuals vs. original data alone, and measure planning accuracy on held-out scenarios.

Safety case documentation through interpretable reasoning trails. For autonomous driving systems seeking regulatory approval or safety certification, the ability to explain decisions in natural language is increasingly valuable. UniUGP's CoT reasoning (Figure 3 examples: identifying "a construction area ahead, with workers and cone markers" and concluding "the vehicle speed needs to be reduced to ensure safety") provides a human-readable justification for trajectory choices that regulators, insurance investigators, and accident review boards can evaluate. The qualitative comparison in Figure 7 shows UniUGP providing specific, actionable reasoning ("The red traffic lights ahead require the ego vehicle to stop to adhere to traffic rules. The pedestrian is moving through the construction area...") while GPT-4o provides vague generalities ("maintain a safe distance, drive steadily"). If the 0.88 GPT score correlates with human judgments of reasoning quality, UniUGP's CoT output could serve as a component of a safety case arguing that the vehicle's decision-making is transparent and auditable. The 38% correct-to-incorrect reversion rate documented in the earlier analysis is a critical concern here: if the reasoning trail contains internally contradictory statements, it undermines rather than supports a safety case. Deployment for this use case would require implementing and evaluating chain-level selection (taking the best reasoning step from a generated chain rather than the final output) or incorporating a separate coherence verifier.

When to Prefer This Method

The paper does not articulate an explicit decision rule or tradeoff against named alternatives for practitioners choosing between UniUGP and competing approaches. The comparisons in Tables 3–6 are presented as evidence of UniUGP's performance rather than as guidance about when to select it over alternatives like FSDrive, Epona, or fine-tuned large VLMs. The paper does not discuss scenarios where UniUGP would be contraindicated, computational budgets where simpler methods are preferable, or input configurations (e.g., multi-camera availability) where other architectures achieve better results. In the absence of explicit tradeoff guidance from the paper, constructing a "Prefer A when... Prefer B when..." matrix would impose a decision framework the authors did not provide. The implicit positioning — UniUGP as a unified model that achieves competitive or state-of-the-art results across understanding, planning, and generation simultaneously — suggests the method is intended for applications where all three capabilities are valued and where the training-time cost of the generation expert is acceptable, but this is a default assumption rather than an argued position.