LLM Systems2025~119 minarXiv:2502.09980

V2V-LLM: Vehicle-to-Vehicle Cooperative Autonomous Driving with Multimodal Large Language Models

score 2Hsu-kuang Chiu, Ryo Hachiuma, Chien-Yi Wang +3NVIDIA↑ 6★ 11Feb 14, 2025cs.CVhttps://github.com/eddyhkchiu/V2VLLM

🎯 Pitch

A single multimodal LLM that fuses perception data from multiple connected vehicles can answer driving questions and plan trajectories, cutting collision rates nearly in half compared to non-LLM cooperative baselines in occluded scenarios. This turns cooperative perception from a sensor fusion problem into a unified language-driven reasoning task.

1. Executive Summary

This paper introduces a novel problem setting that integrates a Multimodal Large Language Model into cooperative autonomous driving, proposing the Vehicle-to-Vehicle Multimodal Large Language Model (V2V-LLM) as a baseline method alongside a new Vehicle-to-Vehicle Question-Answering (V2V-QA) dataset and benchmark. V2V-LLM fuses scene-level feature maps and object-level feature vectors shared by multiple Connected Autonomous Vehicles (CAVs) via V2V communication and uses an LLM — specifically LLaVA-v1.5-7b with a PointPillars LiDAR encoder — to answer driving-related questions spanning grounding, notable object identification, and planning tasks. On the V2V-QA benchmark built from the V2V4Real and V2X-Real cooperative perception datasets (1.45M QA pairs across 48K frames), V2V-LLM achieves the lowest collision rate (3.00% vs. 3.55–4.57% for other fusion baselines on the V2V-split) and the best planning L2 error (4.99m vs. 6.20–7.08m), while also outperforming a non-LLM cooperative baseline combining CoBEVT detection with a BEV-planner (3.00% vs. 11.59% average collision rate), establishing that LLM-based fusion of cooperative perception data can unify perception and planning for multiple CAVs within a single model architecture, though the approach currently assumes a centralized computing node with all CAVs sharing features and has only been validated on LiDAR-only, two-vehicle real-world driving scenarios with ground-truth trajectory supervision.

2. Context and Motivation

Every autonomous vehicle deployed today faces a fundamental physical constraint: its perception of the world is limited to what its own sensors can observe. This becomes a safety-critical vulnerability when those sensors are blocked. Consider a self-driving car approaching an intersection behind a delivery truck. The truck's large body occludes a substantial portion of the scene — pedestrians crossing from the far side, a cyclist in the adjacent lane, or another vehicle running a red light. The ego-vehicle's LiDAR and cameras simply cannot see these hazards, and without that perception input, the planning system cannot react to them. The paper identifies this as the central motivation:

"the perception and planning systems of autonomous vehicles in daily operation rely mainly on their local LiDAR sensors and cameras to detect notable nearby objects and plan for future trajectories. This approach may encounter safety-critical problems when the sensors are occluded by nearby large objects."

This is not a rare edge case. Occlusions are pervasive in urban driving — by trucks, buses, buildings, vegetation, and even other passenger vehicles in dense traffic. Each occlusion creates a blind spot, and each blind spot represents a potential collision that the vehicle has no way to anticipate or avoid using only its own sensors.

The problem compounds when we consider planning. Even if a vehicle detects all visible objects, it must predict where it can safely navigate in the future. Planning errors — whether from undetected obstacles, poor trajectory prediction, or misjudged distances — translate directly into collision risk. The paper's framing makes clear that perception failures cascade into planning failures, and that the two cannot be cleanly separated in safety analysis. A vehicle that correctly detects 99% of objects but misses the one pedestrian in an occluded crosswalk has failed catastrophically, regardless of how sophisticated its planning algorithm is for the 99% of cases it handles correctly.

Vehicle-to-Vehicle Communication: The Intuitive Solution

The natural solution to the occlusion problem is sharing: if one vehicle cannot see around a truck, perhaps another vehicle nearby can. This is the premise of cooperative perception via Vehicle-to-Vehicle (V2V) communication. Multiple Connected Autonomous Vehicles (CAVs) driving in proximity share their individual perception data — raw sensor readings, extracted features, or detected objects — and fuse this information to construct a more complete understanding of the scene than any single vehicle could achieve alone.

The intuition is powerful: the truck blocking CAV-EGO's view of the crosswalk is not blocking CAV-1's view from the adjacent lane. By combining CAV-1's unobstructed observation of the pedestrian with CAV-EGO's observation of the truck and road geometry, both vehicles gain a comprehensive scene understanding that eliminates the blind spot. Cooperative perception effectively creates a distributed sensor network where the collective field of view covers regions that individual sensors cannot reach.

The Gap: Prior Work Stopped at Perception

The paper identifies a critical limitation of existing cooperative driving research: it has focused almost exclusively on perception tasks (object detection and tracking) without addressing how these perception improvements translate to downstream planning. The authors state this gap explicitly:

"to date, cooperative driving research and datasets have mostly focused on perception tasks. How these state-of-the-art cooperative perception models can be connected with the downstream planning models to generate good cooperative planning results is still under-explored."

This is not merely a matter of missing evaluation metrics — it reflects a structural gap in the research pipeline. The cooperative perception community has developed increasingly sophisticated fusion architectures (AttFuse, V2X-ViT, CoBEVT) and released multiple datasets (OPV2V, V2X-Sim, V2XSet, V2V4Real, V2X-Real, DAIR-V2X, TUMTrafV2X), but these efforts treat perception as the terminal output. A cooperative perception model might produce excellent detection results on occluded pedestrians, but there is no established methodology for feeding those detections into a planning system, and no benchmark exists for evaluating whether cooperative perception actually improves planning outcomes.

This gap matters for two reasons. First, from a systems perspective, the ultimate goal of autonomous driving is safe motion planning — not accurate object detection per se. Detection is an intermediate task. If cooperative perception improves detection but those improvements do not translate into safer trajectories (because, for example, the planning module does not trust or properly incorporate cooperative perception outputs), then the safety benefits of V2V communication remain unrealized. Second, from a research direction perspective, the absence of planning benchmarks for cooperative driving means the field lacks a north star. Researchers optimize detection metrics without knowing whether their architectural innovations will actually reduce collision rates in deployment.

The datasets themselves reflect this perception-only focus. As Table I shows, existing cooperative perception datasets — OPV2V, V2X-Sim, V2XSet, DAIR-V2X, V2V4Real, TUMTrafV2X, V2X-Real — provide no Question-Answer pairs for planning tasks. They are designed to train and evaluate detection and tracking models, with no mechanism for asking "given what all CAVs see, where should CAV-EGO drive to avoid collisions?"

Where LLM-Based Driving Methods Fall Short

In parallel with cooperative perception research, a separate community has explored using Large Language Models and Multimodal LLMs to build end-to-end autonomous driving systems for individual vehicles. The motivation is compelling: LLMs possess common-sense reasoning capabilities and generalization ability from large-scale pretraining that could help autonomous vehicles handle the diversity and ambiguity of real-world driving scenarios. Rather than building separate perception, prediction, and planning modules with engineered interfaces between them, an LLM-based system encodes raw sensor inputs into visual tokens, processes them alongside language instructions, and directly outputs driving decisions — actions, trajectories, or answers to driving-related questions.

The paper acknowledges this trend and its initial successes:

"Other recent research has attempted to use LLM-based methods to build end-to-end perception and planning algorithms for an individual autonomous vehicle due to their common-sense reasoning and generalization ability from large-scale pre-trained data."

Several datasets have emerged to support this paradigm: NuScenes-QA, Lingo-QA, DriveLM, TOKEN, and OmniDrive — all listed in Table I. These datasets pair driving scenes with Question-Answer pairs covering perception, prediction, and planning tasks. For example, DriveLM includes planning QAs that ask the model to generate future waypoints, while TOKEN focuses on object-level knowledge to address long-tail events.

However, the paper identifies a critical limitation: all existing LLM-based driving research and datasets are designed for single-vehicle scenarios. They assume one autonomous vehicle with its own sensor suite, processing its own observations, and making decisions independently. None address the cooperative driving setting where multiple vehicles share perception information and an LLM must reason across potentially conflicting or complementary observations from different viewpoints.

The authors make this limitation explicit when describing the differences between their work and prior LLM-based driving research:

"First, our LLM can fuse multiple perception information from different CAVs and provide answers to different questions from any CAV, rather than just serving a single self-driving car."

This is not a trivial extension. Fusing perception data from multiple CAVs introduces challenges that single-vehicle LLM driving systems never encounter:

Viewpoint alignment: Each CAV observes the scene from a different position and orientation. The LLM must spatially align observations — understanding that CAV-1's "vehicle on the left" and CAV-2's "vehicle ahead" might refer to the same physical object.
Complementary and conflicting information: One CAV might detect an object that another misses entirely (complementary), or two CAVs might provide slightly different location estimates for the same object (conflicting). The LLM must resolve these cases.
CAV-conditional responses: The same fused perception data should produce different answers depending on which CAV is asking. A vehicle's planned trajectory will differ from another vehicle's because their positions, headings, and destinations differ — even though both queries are answered from the same underlying fused scene representation.
Occlusion-aware questioning: The questions themselves should target the specific blind spots of each CAV. A CAV should ask about regions it cannot see, not about regions already visible to its own sensors. This is a fundamentally new type of question design that prior single-vehicle QA datasets do not address.

The paper's second design distinction reinforces this point:

"Second, our grounding questions are specially designed to focus on the potential occluded regions of each individual CAV."

The Missing Intersection: Why Cooperative Perception + LLM Planning Matters

The paper positions itself at the intersection of two previously disconnected research threads: cooperative perception (which has perception datasets but no planning integration) and LLM-based driving (which has planning capabilities but no multi-vehicle cooperation). The gap is striking when visualized:

Research Thread	Handles Multiple Vehicles?	Handles Planning?	Unified Architecture?
Cooperative perception	✓ (but only detection/tracking)	✗	✗ (separate perception + planning)
LLM-based driving	✗ (single vehicle only)	✓	✓ (end-to-end)
This paper's aim	✓	✓	✓

This positioning is not merely additive — it represents a qualitative shift in what is possible. By using an LLM as the fusion and reasoning engine, the architecture can perform perception tasks (answering "is there an object behind this truck?") and planning tasks (answering "what trajectory should I follow?") within a unified model. There is no separate perception module that exports detections to a separate planning module — the same LLM that reasons about occluded objects also generates driving trajectories, with both capabilities emerging from the same underlying model.

The practical motivation is safety: if cooperative perception can reveal hazards invisible to individual vehicles, and if LLM-based planning can generate safer trajectories than modular pipelines, then combining both capabilities should produce the safest possible driving behavior — especially in occlusion-heavy scenarios that are currently the most dangerous.

How the Paper Frames Its Contribution

The paper explicitly presents three contributions that together define the novel problem setting:

The V2V-QA dataset: A new Question-Answer dataset supporting LLM-based cooperative autonomous driving, with 1.45M QA pairs across grounding, notable object identification, and planning tasks. Unlike prior cooperative perception datasets (detection/tracking only) or LLM-driving datasets (single vehicle only), V2V-QA is specifically designed for the intersection: multiple CAVs, planning evaluation, and occlusion-aware questioning.
The V2V-LLM baseline: A fusion architecture where each CAV independently extracts scene-level feature maps and object-level feature vectors from its LiDAR point cloud (using PointPillars), shares these features with a centralized LLM (LLaVA-v1.5-7b), and the LLM fuses the features to answer driving-related questions from any CAV. This is positioned as a new fusion category — "LLM fusion" — distinct from traditional early, intermediate, and late fusion approaches.
A benchmark for cooperative LLM-based driving: Experimental comparisons against no-fusion, early-fusion, intermediate-fusion (AttFuse, V2X-ViT, CoBEVT), and non-LLM baselines (CoBEVT + BEV-planner) on all three task types, establishing V2V-LLM as a strong initial baseline and demonstrating the viability of LLM-based architectures for cooperative driving.

The paper's ambition is stated clearly in its conclusion: to establish V2V-LLM as "a unified multimodal foundation model that can effectively perform perception and planning tasks for cooperative autonomous driving."

Practical Significance and Implicit Assumptions

The paper operates under several assumptions that shape both the problem formulation and the solution architecture:

Centralized compute model. The proposed setting assumes a "centralized LLM computing node" that receives perception data from all CAVs, processes it, and returns answers. This is a reasonable starting point — it simplifies the coordination problem and allows the LLM to see all available information — but it raises deployment questions about single-point-of-failure risk, communication latency at scale, and whether the communication cost scales acceptably as the number of CAVs grows (the paper addresses the latter partially, showing the LLM node's communication cost scales as $0.203N_v + 0.0004N_qN_v$ MB, but does not deeply explore latency or reliability implications).

Two-vehicle scenarios. The underlying datasets (V2V4Real and V2X-Real) involve exactly two vehicles driving simultaneously. While the problem formulation supports an arbitrary number of CAVs, all experimental validation is on two-vehicle scenarios. The paper does not claim generalization to larger vehicle networks (3+, which would appear in the simulated OPV2V and V2XSet datasets but are not used here because the paper prioritizes real-world data over simulation).

LiDAR-only perception. The input features are derived exclusively from LiDAR point clouds. Cameras — which provide complementary semantic information (traffic light states, sign text, lane markings) — are not used. This is consistent with the chosen base datasets (V2V4Real and V2X-Real are LiDAR-focused) but means the model cannot benefit from visual cues that humans and many production autonomous driving systems rely on.

Ground-truth trajectory supervision. For the planning task, the ground-truth answers are the actual future trajectories driven by human operators during data collection. This raises an open question: are these trajectories truly optimal, or do they merely represent what one human driver chose to do? The model is trained to imitate human driving behavior, not necessarily to generate the safest possible trajectory. In cases where the human driver made a suboptimal choice (e.g., following too closely, taking an inefficient path), the model learns to replicate that behavior.

Open-loop evaluation. The planning evaluation is open-loop: the model predicts a trajectory given the current scene, but that trajectory is not executed in simulation to observe its consequences. The collision rate is computed by checking whether the predicted waypoints intersect with annotated object locations, not by simulating the vehicle's motion through the environment. This is standard practice in the autonomous driving literature (NuScenes, OmniDrive, DriveLM all use open-loop evaluation) but potentially overestimates safety, since it does not account for how the model's trajectory might interact with the future motion of other agents or how the vehicle's dynamics constrain feasible trajectories.

Why Now? Convergence of Enabling Technologies

The paper's timing reflects the convergence of several enabling factors that make LLM-based cooperative driving feasible for the first time:

Mature cooperative perception foundations: Datasets like V2V4Real (2023) and V2X-Real (2024) provide real-world cooperative driving data with synchronized multi-vehicle LiDAR and ground-truth annotations, eliminating the need for simulation-only studies.
LLaVA and similar architectures: LLaVA-v1.5-7b provides a strong, open-source multimodal LLM that can be fine-tuned with domain-specific perception encoders and task data, making it practical to build a cooperative driving model using accessible pre-trained weights.
Parameter-efficient fine-tuning: LoRA enables training the LLM component on cooperative driving tasks without updating all 7 billion parameters, making the training computationally feasible on 8 A100 GPUs rather than requiring industrial-scale compute clusters.
QA-driven evaluation in autonomous driving: The emergence of NuScenes-QA, DriveLM, Lingo-QA, TOKEN, and OmniDrive has established QA benchmarks as a legitimate evaluation paradigm for autonomous driving, providing a template that V2V-QA extends to the cooperative setting.

3. Technical Approach

3.1 Reader Orientation

This paper is primarily a systems-and-dataset paper that proposes a concrete architecture for using a multimodal large language model as the central reasoning engine in a cooperative autonomous driving pipeline. The system being built is a centralized question-answering machine that ingests perception features from multiple connected autonomous vehicles (CAVs), fuses them inside a large language model, and outputs natural-language answers to driving-related questions — including where objects are, what hazards lie along a planned trajectory, and what future path a vehicle should follow.

The core problem it solves is how to convert shared perception data from multiple vehicles into actionable driving decisions using a single unified model, rather than a fragmented pipeline of separate detection, tracking, fusion, and planning modules. The solution's shape is: each CAV runs a lightweight 3D object detector locally to extract features, these features are transmitted to a central LLM, and the LLM directly produces task-specific answers without intermediate detection outputs or hand-designed fusion rules.

3.2 Big-Picture Architecture (Diagram in Words)

The V2V-LLM system has five major components, connected in a feedforward pipeline:

Per-CAV LiDAR Sensor: Each Connected Autonomous Vehicle (CAV) captures a 3D point cloud of its surrounding environment using its onboard LiDAR sensor. This is raw sensor data — a set of (x, y, z, intensity) points in the vehicle's coordinate frame.
Per-CAV 3D Object Detector (PointPillars): Each CAV independently runs the same PointPillars object detector on its own point cloud. The detector produces two types of features: a scene-level feature map (S_EGO or S_1), which is a dense spatial representation of the entire scene, and object-level feature vectors (O_EGO or O_1), which are compact embeddings of individual detected objects (their locations, sizes, orientations, and class information). These features are what get shared — not raw point clouds and not final detection outputs.
V2V Communication Channel: Each CAV transmits its scene-level feature map and object-level feature vectors to a centralized computing node. The paper calculates that one scene feature map costs ≤0.2 MB and one set of object detection parameters costs ≤0.003 MB per CAV per timestep. Questions and answers add negligible overhead (≤0.0002 MB each). Importantly, each CAV only sends its features once per timestep regardless of how many questions are asked, because the LLM node can cache and reuse them.
Multimodal LLM (LLaVA-v1.5-7b with Vicuna backbone): This is the central reasoning engine. It receives perception features from all CAVs (after projection into language embedding space), plus a natural-language question from any CAV, and generates a natural-language answer. The LLM is responsible for fusing the multi-vehicle perception data — resolving complementary and conflicting observations — and producing the appropriate task output.
Projector Network: A multi-layer perceptron (MLP) that maps perception features from the PointPillars embedding space into the LLM's token embedding space. This is the only learned interface between the frozen perception encoder and the frozen LLM backbone (along with LoRA adapters injected into the LLM).

Information flow at inference time: A CAV captures a point cloud → PointPillars extracts scene and object features → features are transmitted to the centralized LLM → a question arrives from any CAV (e.g., "Is there anything behind the object at [x, y]?") → the projector maps perception features to language embedding tokens → the LLM processes perception tokens + question tokens jointly → the LLM autoregressively generates an answer string.

3.3 Roadmap for the Deep Dive

The technical breakdown proceeds in this order:

First, the V2V-QA dataset construction pipeline — what the five question types are, how they are generated automatically from ground-truth annotations, and what evaluation metrics they use — because the dataset defines what the model must learn to do and constrains the architecture design.
Second, the perception feature extraction — how PointPillars converts raw LiDAR point clouds into scene-level feature maps and object-level feature vectors — because these are the shared representations that all downstream fusion and reasoning depends on.
Third, the multimodal LLM architecture — how perception features are projected, combined with language tokens, and processed by the LLM — because this is the novel fusion mechanism that distinguishes V2V-LLM from prior cooperative perception approaches.
Fourth, the training procedure — what gets frozen, what gets fine-tuned, and with what hyperparameters — because the training design choices (LoRA, pre-trained initialization, multi-task QA training) directly determine what the model can learn.
Fifth, the baseline fusion methods for comparison — no fusion, early fusion, intermediate fusion — because understanding what V2V-LLM is compared against clarifies what "LLM fusion" means and why it is positioned as a distinct category.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems-and-dataset paper whose core idea is that a multimodal LLM can serve as the fusion and reasoning engine for cooperative autonomous driving, replacing manually designed fusion architectures with a learned, language-mediated integration of multi-vehicle perception data, and that this unified architecture can perform both perception tasks (grounding, object identification) and planning tasks (trajectory generation) within a single model.

V2V-QA Dataset Construction

The V2V-QA dataset is built on top of two existing real-world cooperative perception datasets: V2V4Real (Xu et al., CVPR 2023) and V2X-Real (Xiang et al., ECCV 2024). The paper creates two splits — V2V-split (from V2V4Real) and V2X-split (from V2X-Real) — and follows the same training/testing partition boundaries as the original datasets.

Source data. V2V4Real contains 20K frames of synchronized LiDAR data from two vehicles driving simultaneously, with 3D bounding box annotations for other objects. The training set has 32 driving sequences (7,105 frames per CAV) and the testing set has 9 driving sequences (1,993 frames per CAV). V2X-Real contains 33K frames from two vehicles plus roadside infrastructure LiDAR in some scenes. Its training set has 43 driving sequences (5,772 frames per CAV) and its testing set has 9 driving sequences (1,253 frames per CAV). The frame rate is 10 Hz.

QA generation methodology. For each frame, the authors use three sources of information to automatically generate questions and answers: (1) the ground-truth 3D bounding box annotations, (2) each CAV's ground-truth future trajectory, and (3) each CAV's individual detection results from a pre-trained PointPillars model. They apply manually designed geometric rules and text templates — not human annotation and not LLM generation — to produce the QA pairs. This guarantees that all answers are factually correct with respect to the ground-truth labels, avoiding hallucination in the training data. The text templates are shown in Figures 4 and 5 of the paper, producing structured natural-language outputs like "Yes, there is a car at the location. Its center location is [x, y]."

Q1: Grounding at a reference location (Figure 2a). The LLM receives a specific 2D location [x_q, y_q] and must determine whether any object occupies that location. If yes, it returns the object's center location; if no, it returns a negative response. The query locations are drawn from two pools: the center locations of all ground-truth bounding boxes (ensuring coverage of all actual objects) and the center locations of each CAV's individual PointPillars detection outputs (including both true positives and false positives). This dual-sampling strategy is designed — the paper states — to "focus more on evaluating each model's cooperative grounding ability on the potential false positive and false negative detection results." In other words, by including detection outputs (which may be wrong) as query locations, the dataset explicitly tests whether cooperative fusion can correct individual perception errors — confirming true detections that a single CAV might doubt, or rejecting false positives that a single CAV might accept.

Q2: Grounding behind a reference object at a location (Figure 2b). This question type simulates an occlusion scenario. The question provides the center location of a reference object [x_ref, y_ref] — which represents a nearby large object that might be blocking the asking CAV's view — and asks whether there is another object behind it. The LLM must use the fused perception data from all CAVs to determine if an occluded object exists. If so, it returns that object's location; if not, it returns a negative response.

The geometric rule for generating the answer works as follows: from the relative pose of the asking CAV and the reference object location, the system draws a sector region extending behind the reference object (the paper does not specify the exact angular width or distance threshold of this sector in the main text, though it is presumably defined in the released code). The closest ground-truth object whose center falls within this sector region is selected as the answer target. The reference locations in the questions are drawn from each CAV's individual detection results — again, emphasizing the evaluation of whether cooperative fusion can reveal occluded objects that a single vehicle would miss or misjudge.

Q3: Grounding behind a reference object in a direction (Figure 2c). This extends Q2 by replacing the explicit reference location with a directional keyword — "front," "back," "front left," "front right," "back left," or "back right." The system first identifies the closest detected object to the asking CAV in each of these 6 directions to serve as the reference object. Then it follows the same sector-based geometric rule as Q2 to find the closest ground-truth object behind that reference object. This question type evaluates the LLM's language-and-spatial understanding ability — it must map a directional word like "front right" to a geometric region, identify which detected object occupies that region, and then reason about occlusions behind that object using multi-vehicle data.

Q4: Notable object identification (Figure 2d). This question provides a CAV's planned future trajectory as a sequence of 6 waypoints (extracted from the ground-truth trajectory over the next 3 seconds, evenly spaced at 0.5-second intervals) and asks: "Is there anything I need to be aware of?" The answer must identify up to the 3 closest ground-truth objects within 10 meters of the reference trajectory. This task requires the model to compute spatial proximity between a trajectory and candidate objects using the fused multi-vehicle perception data.

Q5: Planning (Figure 2e). This question asks the LLM to output a suggested future trajectory to avoid collisions. The answer is the ground-truth future trajectory: 6 waypoints evenly distributed over the next 3 seconds (again at 0.5-second intervals). The paper notes that V2V-QA's planning task is "more challenging than other NuScenes-based LLM-driving related works" for two reasons:

Multi-CAV support: The LLM must generate different trajectories depending on which CAV is asking, even though the underlying fused perception data is the same. Prior datasets like NuScenes-QA and DriveLM only need to plan for a single ego-vehicle.
Diverse driving scenarios: V2V-QA includes both urban and highway driving, while NuScenes only contains urban driving. Highway scenarios involve higher speeds, longer planning horizons, and different interaction patterns (merging, lane following at speed, larger inter-vehicle distances).

Dataset scale. Table II summarizes the resulting QA pairs: 1,454,790 total across both splits, averaging 30.2 QA pairs per frame. This is substantially denser than prior LLM-based driving datasets — NuScenes-QA has 13.5 QA pairs per frame, Lingo-QA has 15.3, and TOKEN has 15.5. The density reflects the combinatorial expansion from generating multiple question types per frame and, within each type, generating multiple instances using different reference locations, reference objects, and directional keywords.

Evaluation metrics. The paper follows the evaluation approach established by OmniDrive and TOKEN:

Grounding (Q1, Q2, Q3) and notable object identification (Q4): The primary metric is F1 score, which is the harmonic mean of precision and recall. For grounding questions, precision measures how many of the model's claimed objects are actually present at the correct locations, while recall measures how many of the ground-truth objects the model correctly identifies. The paper reports both P, R, and F1 in the detailed tables (Tables IX and X in the appendix).
Planning (Q5): The primary metrics are L2 distance error (in meters) and collision rate (as a percentage). The L2 error is computed by comparing each predicted waypoint to the corresponding ground-truth waypoint and averaging across all 6 waypoints, then reported at 1-second, 2-second, and 3-second horizons (Table XI and XII). The collision rate is computed by checking whether any predicted waypoint falls within the bounding box of a ground-truth annotated object.

The L2 error is a standard trajectory accuracy metric — lower is better. The collision rate is a safety metric — zero would mean the predicted trajectory never intersects any object's position. Together, they capture both how closely the model imitates human driving behavior and how safe its proposed trajectories are.

Perception Feature Extraction (PointPillars)

Every CAV processes its own LiDAR point cloud using PointPillars (Lang et al., CVPR 2019), a 3D object detector that the paper selects "for fair comparisons" with prior cooperative perception baselines that also use PointPillars. The choice is driven by compatibility with V2V4Real and V2X-Real, both of which provide PointPillars-based baseline implementations.

What PointPillars produces. Given a LiDAR point cloud P (a set of (x, y, z, reflectance) points), PointPillars performs two operations relevant to V2V-LLM:

Scene-level feature map (S): PointPillars voxelizes the point cloud into a grid of vertical "pillars" in the bird's-eye view (BEV) plane. Each pillar aggregates the points falling within it using a simplified PointNet, producing a fixed-dimensional feature vector. The collection of all pillar features forms a 2D feature map S over the ground plane. This feature map captures dense spatial information about the scene — where surfaces are, what the local point distributions look like, and coarse semantic cues — without committing to specific object detections.
Object-level feature vectors (O): PointPillars also includes a detection head that processes the scene features to produce 3D bounding box predictions. Each predicted box comes with a feature vector encoding its location (center x, y, z), dimensions (length, width, height), orientation (yaw angle), and class probability distribution. The set of all such feature vectors forms O. Unlike the scene-level features, these are sparse and object-centric — each vector compactly represents a candidate object that the detector believes exists.

Why both feature types? The paper's ablation study (Table VIII) shows that using only scene-level features or only object-level features degrades performance compared to using both. Scene-level features preserve dense geometric information that might be lost in the discretization to discrete object detections — a partially occluded pedestrian might not trigger a confident detection but might still produce a distinctive pattern in the scene feature map. Object-level features provide explicit spatial priors (object locations, sizes, orientations) that the LLM can reason about symbolically — they are easier for the language model to "understand" because they are already in a quasi-symbolic format. The paper notes that "object-level-only model outperforms the scene-level-only model," suggesting object-level features are generally more informative for the LLM, but the two are complementary — scene-level features help with cases where detection is uncertain.

Coordinate frames. Each CAV processes its point cloud in its own ego-centric coordinate frame. The scene features and object features are therefore in different coordinate frames for different CAVs. The paper does not explicitly describe a coordinate transformation step before feeding features to the LLM — this transformation is implicitly encoded in the feature extractor (PointPillars outputs features in the ego-vehicle frame) and the LLM must learn to relate features from different CAVs through the fusion process. This is a non-trivial learned capability: the model must understand that CAV-1's feature map centered at (0, 0) and CAV-2's feature map centered at a different (0, 0) correspond to the same physical world, just viewed from different positions.

Communication cost of features. The paper quantifies the data transmitted per CAV per timestep: the scene-level feature map is ≤0.2 MB and the object-level feature vectors are ≤0.003 MB (Table IV, Appendix VIII). For comparison, raw LiDAR point clouds from a typical 64-beam sensor produce several megabytes per frame, making raw data sharing prohibitive for V2V communication with current bandwidth. The feature-level sharing approach reduces communication by roughly an order of magnitude while preserving task-relevant information.

Inclusion of roadside infrastructure. In the V2X-split, some driving sequences include LiDAR point clouds from roadside infrastructure (RSUs). When available, these are treated identically to CAV point clouds — they are fed through the same PointPillars detector, producing scene-level and object-level features that are shared with the centralized LLM. The paper does not distinguish between CAV-derived and RSU-derived features in the model architecture; both are simply additional perception inputs to be fused.

Multimodal LLM Architecture

The core reasoning engine is built on LLaVA-v1.5-7b (Liu et al., NeurIPS 2023), which uses Vicuna (Chiang et al., 2023) as its 7-billion-parameter language model backbone. LLaVA is originally designed for image-based visual question answering, using a CLIP vision encoder to convert RGB images into visual tokens that are concatenated with language tokens and processed by the LLM. V2V-LLM replaces the CLIP image encoder with the LiDAR-based feature extraction pipeline described above, adapting LLaVA to the point cloud domain.

Input representation. The input to the LLM at inference time consists of two components concatenated into a single token sequence:

Perception tokens: The scene-level feature map S_i and object-level feature vectors O_i from each CAV i are independently projected into the LLM's token embedding space through the projector network. The projected tokens from all CAVs are concatenated into a single sequence of visual tokens. The paper does not specify the exact tokenization scheme (e.g., whether each spatial position in the scene feature map becomes one token, or whether pooling is applied), but the standard LLaVA approach is to use a learned projection that maps each spatial grid cell of the vision encoder's output to a single embedding vector, preserving the spatial structure.
Language tokens: The question from the asking CAV — for example, "Is there anything behind the front object?" — is tokenized by the LLM's standard text tokenizer.

Projector network. The projector is a multi-layer perceptron (MLP) trained to align the PointPillars feature space with the LLM's token embedding space. This is necessary because the feature dimensions and semantics of the LiDAR encoder (trained for 3D object detection) differ substantially from the CLIP image features that LLaVA was pre-trained with (trained for vision-language alignment). The projector serves as a learned adapter, transforming d_pointpillars-dimensional features into d_llm-dimensional embeddings that the LLM can process alongside text tokens.

The specific MLP architecture (number of layers, hidden dimensions, activation functions) is not detailed in the paper, but it follows the LLaVA approach of using a simple projection head rather than a complex cross-attention mechanism. This design choice keeps the perception-language interface lightweight, preserving most of the model capacity in the LLM itself where the fusion and reasoning occur.

LLM processing. Once the concatenated token sequence (perception tokens from all CAVs + question tokens) enters the LLM, processing proceeds via standard autoregressive transformer decoding. The LLM's self-attention mechanism operates across all tokens in the sequence, meaning that:

Each CAV's perception tokens can attend to every other CAV's perception tokens — this is where cross-vehicle fusion occurs. The model can learn to compare features from different viewpoints, identify correspondences (this object detected by CAV-1 is the same as that object detected by CAV-2), and resolve conflicts (CAV-1 says there is an object here; CAV-2 saw nothing — what is the correct interpretation?).
Each token can attend to the question tokens — this is where task conditioning occurs. The same fused perception representation gets interpreted differently depending on what is being asked. This enables a single model to perform multiple tasks (grounding, identification, planning) from the same underlying features.
The question tokens can attend to the perception tokens — allowing the model to ground the language question in the specific spatial context of the driving scene. "Behind the front-right object" only has meaning relative to the asking CAV's pose and the detected objects in the scene.

Output generation. The LLM autoregressively generates a natural-language answer, token by token, until a stop condition is met. The answer follows the structured templates shown in Figures 4 and 5: for grounding questions, the format is "Yes/No, there is a [class] at [location]" with optional location coordinates; for notable object identification, it lists objects with their locations; for planning, it outputs the 6 waypoint coordinates.

Design choice: why LLM-based fusion over traditional approaches. The paper positions its approach as a new fusion category — LLM fusion — distinct from traditional early, intermediate, and late fusion. The key advantage is the LLM's ability to perform non-linear, context-dependent fusion using the full representational capacity of a 7-billion-parameter transformer. Traditional intermediate fusion methods (AttFuse, V2X-ViT, CoBEVT) use attention mechanisms to aggregate feature maps, but these attention operations are designed primarily for detection — they learn to suppress noisy features and amplify informative ones to improve bounding box regression. They do not perform the kind of spatial reasoning, occlusion inference, and trajectory planning that the LLM can do. By using the LLM as the fusion mechanism, the architecture can handle the full task diversity (grounding, identification, planning) within a single learned component rather than requiring task-specific fusion heads.

Design choice: why centralized over decentralized. The paper assumes a centralized LLM computing node rather than deploying an LLM on each CAV. This is primarily a simplification for initial exploration — it avoids the complexity of distributed consensus and enables the LLM to see all available data simultaneously. The paper does acknowledge decentralized alternatives in Appendix VIII: "one can also consider a decentralized setting that deploys one LLM in each CAV," where each CAV would receive features from all other CAVs and run inference locally. The communication cost in the decentralized setting is 0.203(N_v - 1) MB per CAV (receiving others' features but not transmitting questions/answers). The paper does not experimentally evaluate this setting.

Design choice: why pre-trained LLaVA weights. The model is initialized from the publicly released LLaVA-v1.5-7b checkpoint rather than training the LLM from scratch. This leverages LLaVA's pre-trained vision-language understanding — even though LLaVA was trained on RGB images from datasets like COCO and LAION, the general capability to process visual tokens alongside language tokens transfers to the LiDAR domain after fine-tuning. The ablation in Table VIII confirms this: training from scratch (random initialization) yields substantially worse performance across all tasks compared to fine-tuning from LLaVA's checkpoint, indicating that the pre-trained language and cross-modal reasoning capabilities are beneficial even with a different input modality.

Significance of supporting multiple CAVs. A subtle but important architectural capability: the same LLM instance answers questions from any CAV at the same timestep using the same cached perception features. Because the perception features are independent of the question, the LLM node can receive features from all CAVs once, cache them, and then process multiple questions from different CAVs sequentially (or in parallel batches) without re-transmitting or re-processing the perception data. This is why the communication cost formula in Table IV includes N_q (number of questions per CAV per timestep) — the feature transmission cost is paid once regardless of N_q.

Training Procedure

The training procedure freezes most of the model and fine-tunes only two components: the projector network and LoRA adapters injected into the LLM.

Frozen components:

The LLM backbone (Vicuna-7b): All 7 billion parameters of the transformer are kept frozen. This preserves the pre-trained language modeling and reasoning capabilities that were learned from LLaVA's vision-language training and Vicuna's instruction tuning.
The PointPillars feature encoder: The 3D object detector weights are kept frozen at their pre-trained values from V2V4Real and V2X-Real's released checkpoints. This ensures a fair comparison with intermediate fusion baselines that also use frozen PointPillars encoders (the detector quality is held constant; only the fusion and reasoning differ).

Trainable components:

The projector network: The MLP that maps PointPillars features to LLM embedding space is trained from scratch (or from LLaVA's pre-trained projector, adapted to the new input dimension). This is the primary adaptation mechanism — it learns to represent LiDAR features in a way that the frozen LLM can effectively process.
LoRA (Low-Rank Adaptation) adapters: Following Hu et al. (ICLR 2022), low-rank weight matrices are inserted into the LLM's attention layers and trained while the original weights remain frozen. LoRA dramatically reduces the number of trainable parameters — rather than updating the full d × d weight matrices, it trains two low-rank matrices A (size d × r) and B (size r × d) where r ≪ d, and the effective weight update is ΔW = AB. The paper does not specify the LoRA rank r or which specific layers receive adapters, but uses the same LoRA configuration as LLaVA-v1.5-7b's standard fine-tuning recipe.

Training data. The model is trained on all QA pairs from the V2V-QA training splits: 429,439 pairs from V2V-split and 704,272 pairs from V2X-split. The model sees all five question types (Q1–Q5) during training, making this a multi-task learning setup where the same model learns grounding, notable object identification, and planning simultaneously. The paper does not specify whether different question types are weighted differently in the loss, suggesting uniform weighting across all QA pairs.

Hyperparameters. The paper specifies the following training configuration:

Hardware: 8 NVIDIA A100-80GB GPUs (indicating a total of 640 GB of GPU memory, though the effective batch size per GPU and gradient accumulation settings are not specified).
Batch size: 32 (presumably total across all GPUs, meaning 4 per GPU with 8-way data parallelism).
Optimizer: Adam.
Learning rate: 2 × 10⁻⁵ (0.00002) starting rate, with a cosine learning rate scheduler.
Warm-up: 3% warm-up ratio (the learning rate linearly increases from 0 to 2 × 10⁻⁵ over the first 3% of training steps, then cosine-decays to 0).
Initialization: The model is initialized by loading the pre-trained LLaVA-v1.5-7b checkpoint.
Other settings: "For all other training settings and hyperparameters, we use the same ones from LLaVA-v1.5-7b" — this includes parameters like weight decay, gradient clipping, sequence length, and number of training epochs, none of which are specified in the paper but can be found in the LLaVA repository.

Loss function. The paper does not explicitly state the training objective, but since V2V-LLM generates free-form text answers (not classification labels or regression targets), the standard approach in LLaVA is next-token prediction with cross-entropy loss. For each QA pair, the model is trained to predict the answer tokens given the perception tokens and question tokens. The loss is computed only on the answer portion of the sequence (the question is conditioned on but not predicted). This is the standard autoregressive language modeling objective:

$\mathcal{L} = -\sum_{t=1}^{T} \log p(y_t \mid y_{<t}, \text{perception\_tokens}, \text{question\_tokens})$

where T is the number of answer tokens, y_t is the ground-truth token at position t, and p(y_t | ...) is the model's predicted probability for that token.

What it computes: The sum of negative log-probabilities for each ground-truth answer token, averaged over tokens (or summed — the paper doesn't specify). At each generation step, the model produces a probability distribution over its vocabulary; the loss measures how much probability mass it assigned to the correct token. Lower loss means the model is more confident about the correct answer and less likely to generate incorrect alternatives.

Why this form: This is the standard maximum-likelihood objective for autoregressive sequence generation. It directly optimizes the model to produce the exact ground-truth answer string, token by token. An alternative would be to extract structured outputs (e.g., parse the generated coordinates and apply an L2 regression loss), but this would require the model to learn a two-stage process (generate text, then parse coordinates for loss computation). The end-to-end text generation approach is simpler and leverages the LLM's native output format, but it means the model is not directly optimized for spatial accuracy — it learns to output correct coordinates only insofar as doing so reduces the token-level cross-entropy.

Training from scratch ablation (Table VIII). The paper compares V2V-LLM initialized from LLaVA pre-training against the same architecture trained from scratch (random weight initialization). The from-scratch model achieves substantially worse performance: on grounding, 37.1 F1 vs. 40.7 (V2V-LLM); on notable object identification, 49.3 F1 vs. 59.7; on planning, 6.30m L2 error and 5.01% collision rate vs. 4.99m and 3.00%. This confirms that the LLaVA pre-training provides transferable capabilities even though the input modality (LiDAR vs. RGB) and task domain (driving QA vs. general VQA) differ from pre-training.

Baseline Fusion Methods for Comparison

To establish a benchmark for V2V-QA, the paper implements four alternative fusion approaches using the same LLM architecture (LLaVA-v1.5-7b with projector and LoRA fine-tuning). The only difference is how the perception features are extracted and combined before being fed to the LLM.

No fusion. Only a single CAV's LiDAR point cloud is processed by PointPillars. The resulting scene and object features are fed to the LLM. This represents the lower bound — an individual autonomous vehicle with no cooperative perception. The LLM can only answer questions based on what one CAV sees. In occluded scenarios, this model should fail when the query target is invisible to the ego-vehicle's sensors.

Early fusion. The raw LiDAR point clouds from all CAVs are merged into a single combined point cloud before being processed. Specifically, the 3D points from CAV-1 and CAV-2 (and potentially roadside infrastructure) are concatenated, with appropriate coordinate transformations to express all points in a common reference frame (presumably the asking CAV's ego frame, though the paper does not specify the reference frame choice). A single PointPillars detector processes this merged point cloud to produce one set of scene-level and object-level features, which are then fed to the LLM.

Early fusion gives the detector access to all raw sensor data simultaneously, potentially producing the most complete perception. However, it has severe practical drawbacks: transmitting raw point clouds requires high communication bandwidth (several MB per frame per CAV), and latency is introduced by the need to wait for all CAVs' data before processing begins. The paper quantifies the communication cost as 1.9208 MB (Table III), roughly five times higher than intermediate fusion methods and nearly five times higher than V2V-LLM.

Intermediate fusion. Each CAV independently processes its point cloud through a cooperative detection model that fuses feature maps from multiple CAVs using attention mechanisms. The paper uses three published intermediate fusion architectures:

AttFuse (Xu et al., ICRA 2022): Uses attention to aggregate features from different CAVs in the BEV space. The attention weights are learned to emphasize informative features and suppress noisy or redundant ones.
V2X-ViT (Xu et al., ECCV 2022): Uses a Vision Transformer architecture adapted for multi-agent feature fusion. Each CAV's features are treated as tokens, and self-attention across all tokens enables global feature aggregation.
CoBEVT (Xu et al., CoRL 2022): Uses sparse transformers for cooperative BEV feature fusion, designed to be computationally efficient while maintaining strong detection performance.

In all three cases, the cooperative detection model outputs scene-level and object-level features (derived from the fused feature maps), which are then fed to the LLM. The communication cost for intermediate fusion is 0.4008 MB (Table III), primarily from transmitting feature maps rather than raw point clouds. CoBEVT is not evaluated on the V2X-split because V2X-Real did not release a CoBEVT baseline model.

Key distinction from LLM fusion. In intermediate fusion, the feature aggregation (the attention-based fusion of multi-CAV features) happens inside the detection model, which was trained specifically for detection tasks. The LLM receives already-fused features and performs question answering on top of them. In LLM fusion (V2V-LLM), the features are not pre-fused — each CAV's features are independently extracted and concatenated as separate token groups, and the LLM itself performs the fusion through its self-attention layers. This means the fusion mechanism can adapt to the specific question being asked: when answering a grounding question about an occluded region, the LLM can attend more heavily to the CAV with a clear view of that region; when planning a trajectory, it can attend to all CAVs equally to build a comprehensive scene understanding.

Non-LLM baseline (Table V). To assess whether an LLM-based architecture is necessary at all, the paper also compares against a traditional modular pipeline: CoBEVT for cooperative detection (producing 3D bounding boxes) + BEV-planner (Li et al., CVPR 2024) for trajectory planning using BEV features from the same CoBEVT model. For the grounding task (Q1), the CoBEVT detection outputs are evaluated directly against ground-truth annotations using the same F1 metric. For planning (Q5), the BEV-planner is trained on the same training data and evaluated on L2 error and collision rate. This provides a non-LLM lower bound for the unified architecture.

4. Key Insights and Innovations

Innovation 1: A New Box on the Research Map — LLM-Based Cooperative Autonomous Driving

This paper does not merely propose a better model for an existing problem; it defines a new problem setting at the intersection of three previously disconnected research communities: cooperative perception, LLM-based driving, and multimodal question answering. Before V2V-LLM, these communities operated in parallel. Cooperative perception researchers built fusion architectures (AttFuse, V2X-ViT, CoBEVT) for detection and tracking, but their pipelines terminated at bounding boxes — there was no mechanism to translate improved perception into planning decisions, and no benchmark to evaluate whether cooperative perception actually made driving safer. LLM-based driving researchers (DriveLM, TOKEN, OmniDrive) built end-to-end perception-planning models that accepted sensor data and output driving actions, but they assumed a single ego-vehicle with its own sensors — they never addressed how an LLM should reason across data from multiple vehicles with different viewpoints, complementary fields of view, and potentially conflicting observations. The multimodal QA community (NuScenes-QA, Lingo-QA) developed question-answering benchmarks for autonomous driving, but these assumed a single vehicle asking questions about its own sensor data — they never designed question types that specifically target the occlusion scenarios that cooperative perception is meant to solve.

What makes this intellectually distinctive is not the V2V-LLM architecture itself (which adapts LLaVA in a relatively straightforward way), but the conceptual move of recognizing that these three research threads are solving complementary sub-problems of a unified challenge. Cooperative perception provides the what (shared sensor data), LLM-based planning provides the how (end-to-end architecture), and cooperative QA provides the why (occlusion-aware safety questions that define the task). By bringing them together, the paper creates a research direction that did not previously exist: language-mediated cooperative driving where the same model that reasons about occluded objects also generates collision-avoiding trajectories.

The significance is measured not by architectural novelty but by what becomes possible for the field. Prior to this paper, if a researcher wanted to build a cooperative perception system, they had no standard way to evaluate whether it actually improved driving safety — they could only report detection metrics and hope those translated to better planning. Conversely, if a researcher wanted to build an LLM-based planner for cooperative driving, no dataset existed to train or evaluate it. The V2V-QA dataset and benchmark fill this gap, creating the infrastructure for a new subfield. Table I makes this explicit: V2V-QA is the only dataset with checkmarks in both the "Point Cloud" and "Planning" columns, and the only LLM-based AD dataset with a non-zero number of CAVs.

The move is fundamental rather than incremental because it expands the ontology of cooperative driving research. Prior work asked: "How well can we detect objects using shared sensor data?" This paper asks: "How can we use shared sensor data to answer safety-critical questions about what to attend to and where to drive?" The shift from detection output to question-answering output changes what "good performance" means — from localization accuracy to a balance of spatial precision, collision avoidance, and natural language generation — and makes the evaluation directly relevant to the ultimate goal of autonomous driving (safety) rather than an intermediate goal (detection mAP).

Evidence for the distinctiveness of this setting comes from the experimental design itself. The baseline methods include both cooperative perception architectures (AttFuse, V2X-ViT, CoBEVT) and a non-LLM planning pipeline (CoBEVT + BEV-planner), explicitly comparing against what the field would have built without the LLM-based approach. The fact that V2V-LLM achieves a 3.00% collision rate versus 11.59% for the non-LLM baseline (Table V) is not just a performance improvement — it demonstrates that the unified LLM-based architecture finds cooperative driving solutions that a modular pipeline, even using state-of-the-art components, cannot replicate. This suggests the problem setting is genuinely distinct: cooperative planning through language-mediated fusion is not equivalent to cooperative detection followed by separate planning.

Innovation 2: LLM Fusion as a Distinct Fusion Paradigm with Task-Conditioned Integration

The paper proposes a new category of multi-vehicle data fusion — LLM fusion — and positions it as qualitatively different from the three traditional fusion paradigms (early, intermediate, and late) that have dominated cooperative perception research. This is not a purely taxonomic contribution; it identifies a specific capability that traditional fusion lacks: task-conditioned feature integration.

Traditional fusion methods perform the same aggregation regardless of what question will be asked. Early fusion merges all point clouds into one before detection regardless of downstream use. Intermediate fusion (AttFuse, V2X-ViT, CoBEVT) learns attention weights to aggregate feature maps, but these weights are optimized for detection accuracy — they are fixed at inference time and do not adapt based on whether the system will be asked to ground an object, identify notable objects near a trajectory, or plan a future path. Late fusion runs detection independently on each CAV and applies NMS to merge results, again with no task dependence.

LLM fusion is different in a specific, mechanistic way that the paper's architecture makes possible: the fusion occurs inside the LLM's self-attention layers, which operate on the concatenation of all CAVs' perception tokens plus the question tokens. Because self-attention computes pairwise token interactions, the LLM can learn to attend more strongly to CAV-1's features when answering a question about a region that CAV-2 cannot see, and attend more evenly to all CAVs when planning a trajectory that requires comprehensive scene understanding. The fusion is implicitly parameterized by the question — "what should I fuse and how?" depends on "what am I being asked to do?"

The significance of this distinction goes beyond the mechanism. It means that V2V-LLM does not need a separate fusion module with manually designed rules or detection-only training objectives. The fusion is learned jointly with the task objectives (grounding, identification, planning) through the same next-token prediction loss that trains the entire model end-to-end. This has a subtle but important implication: the model can discover fusion strategies that are optimal for planning even if they are suboptimal for detection. A traditional pipeline that optimizes intermediate fusion for detection accuracy may discard information that would have been useful for planning — information about road geometry, free space, or subtle occlusion patterns that don't affect bounding box quality but matter for trajectory generation. In V2V-LLM, the fusion weights are shaped by all tasks simultaneously, allowing the representation to preserve planning-relevant information that a detection-only fusion might suppress.

The evidence is suggestive rather than definitive — the paper does not provide an attention visualization showing question-dependent fusion patterns — but the performance results support the claim. In Table III, V2V-LLM achieves the best F1 on Q4 (notable object identification) and the lowest collision rate and L2 error on Q5 (planning) in both V2V-split and V2X-split, while being competitive on grounding tasks. The intermediate fusion baselines (particularly CoBEVT) are stronger on pure grounding but weaker on planning, consistent with the hypothesis that detection-optimized fusion is not optimal for trajectory generation. The fact that V2V-LLM's advantage is largest on the tasks furthest from detection (planning, notable object identification) and smallest on the tasks closest to detection (Q1 grounding at a location) supports the claim that LLM fusion provides task-adaptive integration that traditional fusion cannot.

This is a fundamental contribution in the sense that it reframes fusion from a pre-processing step (do fusion, then ask questions) to an integral part of reasoning (ask questions that guide fusion). It is not an incremental improvement to existing fusion methods — it replaces the fusion architecture entirely with a learned, language-conditioned process inside a pre-trained model.

Innovation 3: A Dataset Design Philosophy That Tests Cooperative Capability, Not Individual Perception

The V2V-QA dataset contains a design choice that is easy to overlook but constitutes the intellectual backbone of the benchmark: the grounding questions are deliberately constructed to probe the specific failure modes that cooperative perception is meant to address. This transforms the dataset from a general-purpose driving QA collection into a diagnostic instrument for cooperative capability.

The key mechanism is in how Q1 (grounding at a reference location) selects its query locations. Rather than sampling uniformly from the drivable area or from ground-truth object locations only, the dataset draws query locations from two pools: (1) ground-truth object centers, and (2) each CAV's individual PointPillars detection outputs, including false positives and false negatives. A single-vehicle QA dataset like NuScenes-QA or DriveLM would only need ground-truth locations — asking "what is at this location?" where the location is known to contain an object tests basic grounding ability. But by including locations that a single CAV's detector incorrectly identifies as containing an object (false positives) or locations where a real object exists but a single CAV's detector missed it (false negatives), the dataset creates questions that can only be answered correctly by leveraging multi-vehicle information. If CAV-1's detector hallucinates an object at location [x, y], but CAV-2 and roadside infrastructure confirm nothing exists there, a model with effective cooperative fusion should correctly answer "there is nothing" — whereas a single-vehicle model fed only CAV-1's data would likely echo the false positive. Conversely, if CAV-1 misses an object that CAV-2 clearly detects, cooperative fusion should recover the missed detection.

The same design philosophy extends to Q2 and Q3 (grounding behind a reference object). These questions simulate occlusion scenarios — the reference object (typically a large vehicle) is positioned between the asking CAV and the query region. The question explicitly targets situations where the asking CAV cannot see the answer from its own sensors, but another CAV can. A single-vehicle model has no way to answer correctly; it is fundamentally information-limited. Only by fusing perception data from the unobstructed CAV can the model succeed. This makes Q2 and Q3 direct probes of cooperative capability — performance on these question types measures how effectively the model uses shared data, not how good the individual detectors are.

Why this matters beyond dataset design: it defines what "good cooperative performance" means in a way that is robust to differences in individual perception quality. A model could achieve high Q1 F1 scores by having very accurate individual detectors and simply ignoring shared data — the metric cannot distinguish between models that are genuinely cooperative and models that are individually excellent. But Q2 and Q3 cannot be answered without cooperation. The gap between Q4 (notable object identification, which can benefit from cooperation) and Q5 (planning, which requires reasoning about the fused scene) similarly provides a spectrum of cooperative dependence. This creates a more informative benchmark than aggregate metrics alone — it reveals where cooperation helps and where models fall back to individual perception.

The evidence for this design's diagnostic value appears in the performance patterns across question types. In Table III, the no-fusion baseline achieves 66.6 F1 on Q1 (V2V-split) — surprisingly high, because Q1 includes many ground-truth objects visible to a single CAV. But on Q2, no-fusion drops to 22.6 F1, and on Q3 to 17.2 F1 — the occlusion-targeted questions correctly penalize the single-vehicle approach. Intermediate fusion methods improve Q2 and Q3 substantially (CoBEVT: 29.3 Q2, 21.3 Q3), demonstrating that cooperative fusion provides genuine value for occlusion reasoning. This graded difficulty pattern validates the dataset design: if all question types were equally easy or equally hard for all methods, the benchmark would not discriminate cooperative capability from individual perception quality.

This is an incremental innovation in the sense that it applies a well-established evaluation design principle (diagnostic test construction) to cooperative driving, but it is fundamental to the paper's contribution because without this design, the benchmark would measure the wrong thing. A cooperative driving benchmark that did not explicitly test occlusion scenarios would likely be dominated by models with strong individual perception, masking the value of cooperation entirely. The paper's decision to design question types that isolate cooperative capability is what makes V2V-QA a meaningful contribution rather than just another driving QA dataset.

Innovation 4: Evidence That Unified Perception-Planning Outperforms Modular Pipelines in Cooperative Settings

The paper provides the first head-to-head comparison in cooperative driving between a unified LLM-based architecture (V2V-LLM) and a modular pipeline using the best available cooperative perception and planning components (CoBEVT + BEV-planner). The result — V2V-LLM achieves 3.00% average collision rate versus 11.59% for the modular baseline (Table V) — is not just a performance delta but an architectural finding that challenges the dominant paradigm in autonomous driving system design.

The autonomous driving industry has largely converged on modular architectures: separate perception, prediction, and planning modules connected by defined interfaces (typically bounding boxes from perception feed into prediction, which feeds into planning). This modularity is appealing for engineering reasons — modules can be developed, tested, and improved independently — and for safety reasons — failures can be isolated to specific components. The cooperative driving research community adopted this same modular philosophy: build better cooperative perception modules, then plug them into existing planning stacks.

What V2V-LLM demonstrates is that the interface between perception and planning may be a bottleneck in cooperative settings. The CoBEVT + BEV-planner baseline uses CoBEVT's detection outputs and BEV features as input to the planner. CoBEVT is, by detection metrics, a strong cooperative perception model — it achieves the best Q1 F1 (72.2) among all methods in Table III. Yet when its outputs feed into a separate planner, the resulting collision rate is nearly 4× worse than V2V-LLM. This suggests that the information CoBEVT preserves — optimized for detection — is not the information that a planner needs to generate safe trajectories. The planner may need fine-grained spatial information about occluded regions, uncertainty estimates about object locations, or contextual cues about the relationships between objects that are present in the scene-level feature maps but discarded when the output is reduced to a set of bounding boxes.

The unified architecture avoids this interface bottleneck because the same features used for perception are directly accessible to the planning computation. There is no information bottleneck imposed by an intermediate representation (bounding boxes) designed for a different task. The LLM's self-attention over perception tokens can attend to whatever spatial regions are relevant for trajectory generation — including regions that contain no detected objects but are important for understanding free space, lane geometry, or potential occlusion shadows.

This finding is conceptually significant because it questions the modularity assumption in safety-critical cooperative systems. If a modular pipeline with state-of-the-art components underperforms a unified model by a factor of 4× on collision rate, then the standard argument for modularity — that it enables independent optimization of components — is undercut by evidence that the components cannot be independently optimized to achieve the same system-level safety. The information that matters for cooperative planning may be inherently non-decomposable into detection-relevant and planning-relevant subsets.

This is an empirical finding rather than a theoretical innovation, but it is a fundamental one for the cooperative driving field because it redirects architectural thinking. Prior to this paper, the natural approach to cooperative planning would be: pick the best cooperative detection model, extract its outputs, and train a planner on those outputs. After this paper, that approach looks highly suboptimal. The implication is that cooperative perception and planning should be co-designed — either through unified architectures like V2V-LLM or through modular architectures with richer interfaces (e.g., sharing feature maps rather than bounding boxes between perception and planning). The paper also provides a plausible mechanism for why the modular approach fails (information loss at the detection-to-planning interface), which is testable in future work.

The caveat is that this comparison involves only one modular baseline and one LLM architecture, both trained on the same amount of data. It is possible that a different modular combination (e.g., a stronger planner, or a different feature representation at the interface) would narrow the gap. The paper does not claim that unified architectures are universally superior — only that the straightforward modular approach that the field would have defaulted to is substantially worse than the LLM-based unified approach on this benchmark.

5. Experimental Analysis

Evaluation Methodology

Dataset. All experiments use the V2V-QA dataset, which contains 1,454,790 QA pairs across 48K frames built from the V2V4Real and V2X-Real cooperative perception datasets. The dataset has two splits: V2V-split (from V2V4Real: 32 training sequences / 7,105 frames per CAV, 9 testing sequences / 1,993 frames per CAV) and V2X-split (from V2X-Real: 43 training sequences / 5,772 frames per CAV, 9 testing sequences / 1,253 frames per CAV). The paper follows the original training/testing partition boundaries from V2V4Real and V2X-Real. The QA pairs cover 5 question types: Q1 (grounding at a reference location), Q2 (grounding behind a reference object at a location), Q3 (grounding behind a reference object in a direction), Q4 (notable object identification), and Q5 (planning).
Base model(s). The primary model is V2V-LLM, built on LLaVA-v1.5-7b with a Vicuna-7b LLM backbone and using PointPillars as the LiDAR point cloud feature encoder. The LLM and PointPillars encoder are frozen during training; only the projector network and LoRA adapters injected into the LLM are fine-tuned. The model is initialized from the publicly released LLaVA-v1.5-7b checkpoint. PointPillars was chosen for compatibility with prior cooperative perception baselines (V2V4Real and V2X-Real both use PointPillars), enabling fair comparison where only the fusion and reasoning mechanism differs. All baseline methods use the same LLaVA-v1.5-7b architecture with the same projector and LLM configuration, differing only in how perception features are extracted and combined before being fed to the LLM.
Metrics. The paper follows the evaluation conventions established by OmniDrive and TOKEN. For grounding questions (Q1, Q2, Q3) and notable object identification (Q4), the primary metric is F1 score, computed as the harmonic mean of precision and recall. The appendix (Tables IX, X, XV) reports both P and R alongside F1 for full transparency. For the planning question (Q5), two metrics are used: L2 distance error (in meters), computed by comparing each predicted waypoint to the corresponding ground-truth waypoint and averaging across all 6 waypoints, reported at 1-second, 2-second, and 3-second horizons as well as an overall average; and collision rate (CR, as a percentage), computed by checking whether any predicted waypoint falls within the bounding box of a ground-truth annotated object, reported at the same three temporal horizons plus an overall average. For the grounding average (QGr), the paper reports the mean F1 across Q1, Q2, and Q3. Communication cost (Comm) is reported in megabytes (MB) per timestep.
Baselines. The paper compares against four classes of baselines, all using the same LLM architecture (LLaVA-v1.5-7b) and training protocol with only the perception feature extraction differing:
- No fusion: Only a single CAV's LiDAR point cloud is processed by PointPillars to produce scene-level and object-level features, which are fed to the LLM. This represents a non-cooperative lower bound.
- Early fusion: Raw LiDAR point clouds from all CAVs are merged into a single combined point cloud (with coordinate transformation to a common reference frame), then processed by a single PointPillars detector to produce one set of scene and object features. Communication cost: 1.9208 MB.
- Intermediate fusion: Each CAV independently processes its point cloud, and features are fused using attention-based cooperative detection models before being fed to the LLM. Three published architectures are used:
  - AttFuse (Xu et al., ICRA 2022): Attention-based feature aggregation in BEV space.
  - V2X-ViT (Xu et al., ECCV 2022): Vision Transformer for multi-agent feature fusion.
  - CoBEVT (Xu et al., CoRL 2022): Sparse transformers for cooperative BEV feature fusion. Note: CoBEVT is evaluated only on V2V-split because V2X-Real did not release a CoBEVT baseline model. Communication cost for all intermediate fusion methods: 0.4008 MB.
- Non-LLM baseline (Table V only): A modular pipeline consisting of CoBEVT for cooperative detection (producing 3D bounding boxes) plus BEV-planner (Li et al., CVPR 2024) for trajectory planning. CoBEVT detection outputs are evaluated directly for Q1; BEV-planner is trained separately and evaluated on Q5.
LLM fusion (V2V-LLM) is categorized as a distinct fusion type, where each CAV independently extracts features using PointPillars, shares both scene-level and object-level features with the centralized LLM, and the LLM itself performs fusion through its self-attention layers. Communication cost: 0.4068 MB.
Generation budget / compute accounting. There is no explicit generation budget or beam search scaling analysis in this paper — V2V-LLM is not a search-based method. The paper evaluates models at inference time on the fixed test sets, with all models producing a single answer per question. Communication cost is used as the compute-equivalent metric for comparing efficiency across fusion approaches, quantified in MB per timestep. The paper does not compare models at varying computational budgets (unlike the search scaling paper), so there is no budget-sweep analysis. Training compute is reported descriptively: 8 NVIDIA A100-80GB GPUs, batch size 32, using LLaVA-v1.5-7b pre-trained weights as initialization.
Cross-validation / statistical protocol. There is no cross-validation protocol reported in this paper. The training and testing splits are fixed partitions inherited directly from V2V4Real and V2X-Real. Performance is reported as single-point estimates on the held-out test sets (1,993 frames for V2V-split, 1,253 frames for V2X-split, with multiple QA pairs per frame). The paper does not report confidence intervals, standard deviations, or statistical significance tests for the differences between methods. The robustness experiments (Tables VI and VII) apply varying levels of communication latency and positional noise to the test set and report the resulting performance degradation, but these are single-point measurements.

Main Quantitative Results

Grounding Performance (Q1, Q2, Q3)

The grounding results are reported in Table III, with detailed precision and recall breakdowns in Tables IX (V2V-split) and X (V2X-split) in the appendix.

Aggregate grounding (QGr = average of Q1, Q2, Q3 F1). On the V2V-split, V2V-LLM achieves a QGr of 40.7 F1, which is competitive with the best intermediate fusion method (CoBEVT at 40.9 F1) and substantially better than no fusion (35.5 F1) and early fusion (39.2 F1). The differences among the top methods (CoBEVT 40.9, V2X-ViT 40.5, V2V-LLM 40.7) are small — within 0.4 F1 — suggesting that on aggregate grounding, LLM fusion performs similarly to detection-optimized intermediate fusion. On the V2X-split, V2V-LLM achieves 37.5 QGr F1, outperforming all other methods: intermediate fusion methods achieve 36.4–36.6 F1, early fusion achieves 36.4 F1, and no fusion trails at 34.1 F1. Here V2V-LLM leads by a larger margin (0.9–1.1 F1 over intermediate fusion), suggesting better generalization to the V2X-Real data distribution.

Per-question-type patterns. The grounding results show a clear difficulty gradient across question types that aligns with the dataset's diagnostic design:

Q1 (grounding at a location): This is the easiest task, testing whether the model can confirm or deny the presence of an object at a specific coordinate. On V2V-split, CoBEVT achieves the best Q1 F1 at 72.2, followed by early fusion at 73.5, V2V-LLM at 70.0, and V2X-ViT at 70.8. On V2X-split, V2V-LLM achieves 60.5 F1, leading over the next-best method (V2X-ViT at 59.6). The no-fusion baseline achieves 66.6 (V2V) and 55.7 (V2X) — relatively high because many Q1 queries involve objects visible to a single CAV's sensors. The fact that no fusion is only ~4–5 F1 behind cooperative methods on Q1 (V2V-split) versus ~8–18 F1 behind on Q2 and Q3 confirms that Q1 is less dependent on cooperation.
Q2 (grounding behind a reference object at a location): This task requires identifying occluded objects, making it fundamentally cooperative. On V2V-split, V2V-LLM achieves the best F1 at 30.8, followed by CoBEVT at 29.3, V2X-ViT at 28.0, AttFuse at 26.4, early fusion at 23.3, and no fusion at 22.6. The gap between V2V-LLM and no fusion is 8.2 F1 — substantially larger than the Q1 gap, confirming that Q2 is more cooperation-dependent. On V2X-split, V2V-LLM achieves 25.3 F1, again the best, with the next-best methods at 24.2 (V2X-ViT) and 23.9 (AttFuse). The no-fusion baseline sits at 21.4 — a 3.9 F1 gap.
Q3 (grounding behind a reference object in a direction): This is the hardest grounding task, combining occlusion reasoning with language understanding (mapping directional keywords to spatial regions). On V2V-split, V2X-ViT achieves the best F1 at 22.6, followed by V2V-LLM at 21.2, CoBEVT at 21.3, early fusion at 20.8, AttFuse at 18.4, and no fusion at 17.2. On V2X-split, V2V-LLM achieves 26.7 F1, leading over AttFuse (26.3), V2X-ViT (26.1), early fusion (26.1), and no fusion (25.2). The performance is low across all methods (17–27 F1 range), indicating that Q3 is genuinely challenging — the combination of directional language understanding and cooperative occlusion reasoning pushes all models close to their limits.

Key pattern: V2V-LLM's advantage grows with task difficulty. On V2V-split, V2V-LLM ranks 4th on Q1 (easiest), 1st on Q2 (medium), and 2nd on Q3 (hardest). On V2X-split, V2V-LLM ranks 1st on all three grounding tasks. This pattern — stronger relative performance on harder tasks — is consistent with the hypothesis that LLM fusion's task-conditioned attention provides benefits that detection-optimized intermediate fusion cannot match, especially when the task requires reasoning beyond simple object presence/absence.

Precision-recall tradeoffs. The detailed tables (IX and X) reveal different precision-recall profiles across methods. On V2V-split Q1, no fusion has high precision (77.9) but low recall (58.2), while V2V-LLM has high precision (80.1) and moderate recall (62.2). For Q2, V2V-LLM achieves the highest precision (36.3) and highest recall (26.7) simultaneously — it is not trading off one for the other. For Q3, V2V-LLM's precision (21.5) is lower than V2X-ViT's (25.2), but its recall (20.8) is higher, suggesting the LLM is more willing to predict objects in uncertain scenarios (higher recall, lower precision) while V2X-ViT is more conservative.

Notable Object Identification (Q4)

The notable object identification results are reported in Table III, with precision-recall details in Tables IX and X.

V2V-split. V2V-LLM achieves 59.7 F1, the best performance by a margin of 2.1 F1 over the next-best method (CoBEVT and V2X-ViT both at 57.6). The full ranking: V2V-LLM (59.7) > AttFuse (56.9) > CoBEVT (57.6) = V2X-ViT (57.6) > early fusion (53.9) > no fusion (47.3). The gap from best to worst is 12.4 F1 — much larger than the aggregate grounding gap, indicating that Q4 performance is more sensitive to the choice of fusion method.

V2X-split. V2V-LLM achieves 69.3 F1, again the best, with a margin of 1.7 F1 over the next-best method (early fusion at 67.6). The ranking: V2V-LLM (69.3) > early fusion (67.6) > AttFuse (65.9) > V2X-ViT (65.0) > no fusion (64.4). The no-fusion baseline is surprisingly competitive on V2X-split Q4 (64.4 vs. 69.3 for V2V-LLM) — a gap of only 4.9 F1, compared to 12.4 F1 on V2V-split. This suggests that the V2X-Real test scenes may have fewer occlusion-heavy scenarios that make cooperative perception critical for notable object identification, or that individual vehicle perception is more reliable in those scenes.

Interpretation. Q4 requires spatial reasoning — computing which objects are within 10 meters of a planned trajectory — which is more complex than Q1's binary presence/absence at a location. V2V-LLM's strong performance (best on both splits, with margins of 2.1 and 1.7 F1) suggests that LLM-based fusion is particularly advantageous for tasks that require relational reasoning (object-to-trajectory distance) rather than simple detection. The precision-recall breakdown shows V2V-LLM achieves the highest precision on both splits (61.9 on V2V, 71.9 on V2X) — it identifies notable objects more accurately than alternatives, even when recall is comparable.

Planning Performance (Q5)

Planning results are reported in Table III (aggregate), with detailed horizon breakdowns in Tables XI (V2V-split) and XII (V2X-split). The planning performance with temporal inputs (3 consecutive frames vs. 1 frame) is reported in Table XIV (Appendix IX).

V2V-split planning (Table III, Table XI). V2V-LLM achieves an average L2 error of 4.99 meters and an average collision rate of 3.00% — both the best across all methods. The full L2 error ranking: V2V-LLM (4.99m) < early fusion (6.20m) < no fusion (6.55m) < CoBEVT (6.72m) < AttFuse (6.83m) < V2X-ViT (7.08m). The collision rate ranking: V2V-LLM (3.00%) < early fusion (3.55%) < CoBEVT (3.88%) < AttFuse (4.12%) < V2X-ViT (4.33%) < no fusion (4.57%).

The L2 error advantage of V2V-LLM over the next-best method (early fusion) is 1.21m (4.99 vs. 6.20), a 19.5% reduction. The collision rate advantage is 0.55 percentage points (3.00% vs. 3.55%), a 15.5% relative reduction. Compared to no fusion, V2V-LLM reduces L2 error by 1.56m (23.8%) and collision rate by 1.57 percentage points (34.4%).

Temporal horizon breakdown (Table XI). The L2 error and collision rate worsen monotonically with prediction horizon for all methods, as expected. At the 1-second horizon, V2V-LLM achieves 2.96m L2 error and 0.55% collision rate — errors are small and collisions are rare because the near-future trajectory is constrained by current vehicle dynamics. At the 3-second horizon, L2 error grows to 7.05m and collision rate to 5.25%. The relative advantage of V2V-LLM over other methods persists across all horizons. For example, at 3 seconds: V2V-LLM 7.05m vs. early fusion 8.74m vs. no fusion 9.30m (L2), and V2V-LLM 5.25% vs. early fusion 5.83% vs. no fusion 7.63% (CR).

V2X-split planning (Table III, Table XII). V2V-LLM achieves 1.71m average L2 error and 6.89% average collision rate — again the best. The L2 error values are substantially lower than V2V-split (all methods in 1.71–2.31m range vs. 4.99–7.08m), likely because V2X-Real scenes involve different driving conditions (possibly simpler trajectories, lower speeds, or more straight-line driving). The collision rate pattern is notable: V2V-LLM achieves 6.89%, while other methods range from 8.39% to 9.21%. The collision rate increases more sharply with horizon on V2X-split: at 1 second, V2V-LLM has 2.17% CR; at 3 seconds, 11.71%. This rapid degradation suggests that the planning task is genuinely challenging even on V2X-split, despite lower L2 errors.

Non-LLM baseline comparison (Table V). The CoBEVT + BEV-planner modular pipeline achieves 65.7 F1 on Q1 (using CoBEVT detection outputs directly) and 5.82m L2 error / 11.59% average collision rate on Q5. V2V-LLM outperforms significantly on both: 70.0 Q1 F1 (vs. 65.7) and 4.99m L2 / 3.00% CR (vs. 5.82m / 11.59%). The gap is especially large on collision rate — V2V-LLM's 3.00% is 3.86× lower than the modular baseline's 11.59% — suggesting that the unified LLM-based architecture substantially improves planning safety beyond what a modular pipeline using the same underlying detection model (CoBEVT features) can achieve.

Planning with temporal inputs (Table XIV). When feeding visual features from 3 consecutive frames (current + previous 2) instead of a single frame, V2V-LLM achieves 4.82m L2 error and 2.93% collision rate on V2V-split, improving upon the single-frame results (4.99m, 3.00%). All baseline methods also improve with temporal context, but V2V-LLM remains the best. The relative ordering of methods is preserved: V2V-LLM > early fusion > CoBEVT ≈ AttFuse > V2X-ViT > no fusion. This indicates that V2V-LLM's advantage is not specific to single-frame input — it generalizes to temporal sequences — but multi-frame input provides only modest gains (0.17m L2 reduction, 0.07 percentage point CR reduction for V2V-LLM).

Communication Cost Analysis

Communication cost results are reported in Table III (Comm column) and analyzed in detail in Table IV and Appendix VIII.

Cost comparison across fusion methods. The communication cost per timestep for V2V-LLM is 0.4068 MB — composed of the scene-level feature map (≤0.2 MB) plus object-level feature vectors (≤0.003 MB) from each CAV, plus negligible question/answer overhead (≤0.0002 MB each). This is only 1.5% higher than intermediate fusion methods (0.4008 MB), which transmit fused feature maps of similar size. Early fusion costs 1.9208 MB (4.7× higher) because it transmits raw point clouds. No fusion costs 0 MB in cooperative communication (but achieves worse performance).

Scaling with number of CAVs and questions (Table IV, Appendix VIII). In the centralized setting with Nv CAVs and Nq questions per CAV per timestep:

Each CAV's communication cost: 0.203 + 0.0004Nq MB (features transmitted once regardless of Nq, questions/answers scale linearly).
Centralized LLM node's communication cost: 0.203Nv + 0.0004NqNv MB (receiving features from all CAVs once, plus all questions/answers).

The key insight is that feature transmission cost is paid once per CAV per timestep regardless of how many questions are asked. This means the per-question cost amortizes favorably as Nq increases — the system can answer many questions (from the same or different CAVs) at essentially the same communication cost as answering one question, since only the question/answer strings add marginal bandwidth (0.0004 MB per QA pair).

Decentralized alternative (Appendix VIII). In a decentralized setting where each CAV runs its own LLM, each CAV receives features from all other CAVs but does not transmit questions/answers. Communication cost per CAV: 0.203(Nv - 1) MB. This scales linearly with the number of CAVs but eliminates the single-point-of-failure concern of the centralized approach. The paper does not experimentally evaluate this setting.

Robustness to Communication Latency and Sensor Noise

The robustness assessment is reported in Tables VI (latency) and VII (positional errors), evaluated on V2V-split.

Communication latency (Table VI). To simulate V2V communication delays, the paper applies varying levels of latency (0.1 to 1.0 seconds) during inference. As latency increases from 0 to 1.0 seconds:

Q1 F1 drops from 70.0 to 69.3 (0.7 decrease).
QGr (average grounding F1) drops from 40.7 to 39.3 (1.4 decrease).
Q4 F1 drops from 59.7 to 55.0 (4.7 decrease).
Q5 L2 error increases from 4.99m to 5.26m (0.27m increase, 5.4% relative).
Q5 collision rate increases from 3.00% to 4.09% (1.09 percentage point increase, 36.3% relative).

The degradation is gradual rather than catastrophic — at 0.2 seconds latency (a realistic V2V communication delay), Q5 L2 error is 5.05m and collision rate is 3.21%, only slightly worse than the zero-latency setting (4.99m, 3.00%). This suggests V2V-LLM is reasonably robust to moderate communication delays. The planning tasks (Q4, Q5) are more sensitive to latency than grounding tasks (Q1-Q3), likely because planning requires precise positional information that becomes stale under delay.

Sensor noise / positional errors (Table VII). To simulate GPS/ localization errors, Gaussian noise with varying standard deviations (0.1 to 1.0 meters) is added to the positional information of perception features. As positional error STD increases from 0 to 1.0m:

Q1 F1 drops from 70.0 to 69.8 (negligible — Q1 is robust because it asks about specific coordinate locations).
QGr F1 drops from 40.7 to 40.5 (minimal degradation).
Q4 F1 drops from 59.7 to 57.2 (2.5 decrease).
Q5 L2 error increases from 4.99m to 5.21m (0.22m increase).
Q5 collision rate increases from 3.00% to 3.86% (0.86 percentage point increase, 28.7% relative).

As with latency, planning tasks are more sensitive to positional noise than grounding tasks. The collision rate increase is notable — a 1.0m positional error STD causes collision rate to rise from 3.00% to 3.86%, suggesting that the model relies on accurate spatial alignment between CAVs' coordinate frames for safe trajectory generation. However, even at 1.0m noise STD (a relatively large error for modern localization systems), performance remains substantially better than the no-fusion baseline (which achieves 4.57% CR without noise), indicating cooperative fusion still provides net benefits despite degraded positional accuracy.

Ablation Studies and Robustness Checks

All ablation results are reported in Table VIII (main paper) and Table XV (appendix, with full P/R breakdowns), evaluated on V2V-split.

Scene-level features only vs. object-level features only (Table VIII, Table XV): Removing object-level features (scene-level only) causes a substantial degradation across all tasks: QGr drops from 40.7 to 34.4 (6.3 F1 decrease), Q4 drops from 59.7 to 43.2 (16.5 F1 decrease — the largest drop), and Q5 L2 error increases from 4.99m to 7.21m with collision rate jumping from 3.00% to 15.55%. Removing scene-level features (object-level only) causes milder but still significant degradation: QGr drops to 37.8 (2.9 F1 decrease), Q4 drops to 52.6 (7.1 F1 decrease), Q5 L2 increases to 5.24m with CR at 7.78%. The object-level-only model outperforms the scene-level-only model across all tasks, supporting the paper's claim that "object-level features are easier for LLM to digest." However, both feature types contribute — the full model with both features achieves the best results across all metrics, confirming complementarity. The planning task (Q5) is most dependent on scene-level features (collision rate nearly triples from 3.00% to 15.55% without them), likely because scene-level feature maps capture free-space and geometric information critical for trajectory planning that object-level features abstract away.
Communication cost of feature variants (Table XV): Scene-level-only and full V2V-LLM both require 0.4008–0.4068 MB communication (the scene feature map dominates), while object-level-only costs only 0.0068 MB (two orders of magnitude less). This creates a practical tradeoff: deploying object-level-only features dramatically reduces communication bandwidth but incurs a 2.22 percentage point increase in collision rate (3.00% → 7.78%). The paper does not discuss this bandwidth-safety tradeoff explicitly.
Training from scratch vs. pre-trained initialization (Table VIII, Table XV): Initializing from LLaVA-v1.5-7b pre-trained weights versus random initialization reveals the value of pre-training transfer. Training from scratch achieves: QGr 37.1 (vs. 40.7, 3.6 F1 lower), Q4 49.3 (vs. 59.7, 10.4 F1 lower — the largest gap), Q5 L2 6.30m (vs. 4.99m, 1.31m higher), Q5 CR 5.01% (vs. 3.00%, 2.01 percentage points higher). The gap is largest on Q4 and Q5 — the tasks furthest from standard object detection — suggesting that LLaVA's pre-trained vision-language reasoning capabilities are especially valuable for the relational and planning tasks. Interestingly, the from-scratch model still outperforms the no-fusion baseline on planning (5.01% CR vs. 4.57%, and 6.30m vs. 6.55m L2), indicating that even without pre-training, the LLM-based fusion architecture provides some cooperative benefit.
Robustness across question types under ablation: The detailed Table XV reveals that Q2 (grounding behind a reference object) is most sensitive to removing object-level features: 30.8 F1 (full) → 15.4 (scene-only, a 50% drop) vs. 26.9 (object-only, a 12.7% drop). This suggests that for occlusion reasoning (Q2), object-level features are critical — the model needs explicit object position information to reason about what is behind what, and dense scene features alone do not provide sufficient object-level spatial priors.
Communication cost scaling with latency (implicit robustness check): Table VI shows that at 0.1s latency, V2V-LLM achieves 40.5 QGr, 59.4 Q4 F1, 5.02m L2, and 3.05% CR — performance nearly indistinguishable from the zero-latency setting (40.7, 59.7, 4.99m, 3.00%). This suggests that the current model is not overfitting to perfectly synchronized multi-vehicle data and can tolerate small temporal misalignments typical of real V2V communication.

Critical Assessment

Do the grounding results support the claim that V2V-LLM provides effective cooperative perception?

The paper claims that V2V-LLM "has a promising capability of fusing perception features from multiple CAVs to answer grounding questions." The grounding results partially support this but reveal a more nuanced picture. On V2V-split, V2V-LLM's QGr (40.7 F1) is essentially tied with CoBEVT (40.9) and V2X-ViT (40.5) — the differences are within 0.4 F1. On V2X-split, V2V-LLM leads (37.5 vs. 36.4–36.6), but the margin is modest (0.9–1.1 F1). For Q1 specifically — the most basic grounding task — V2V-LLM ranks 4th out of 6 methods on V2V-split (70.0 vs. CoBEVT's 72.2). This suggests that for straightforward object presence/absence queries, detection-optimized intermediate fusion may be as good as or better than LLM-based fusion.

Where V2V-LLM distinguishes itself is on the occlusion-targeted grounding tasks: Q2 (grounding behind a reference object) — best on both splits — and Q3 (directional grounding) — second on V2V, first on V2X. These are precisely the tasks that the dataset was designed to make cooperation-dependent. The pattern is consistent with the hypothesis that LLM fusion provides benefits specifically when the task requires reasoning about occlusion relationships, not just detecting objects. However, the absolute performance on Q2 (30.8 F1) and Q3 (21.2 F1) remains low — most occlusion-grounding queries are answered incorrectly even by the best method. This raises the question of whether the remaining failures are due to inadequate perception (missed detections in the fused features) or inadequate reasoning (the LLM cannot correctly infer occlusion relationships from the available features). The paper does not disentangle these failure modes.

A missing experiment: the paper does not report performance separately for positive and negative Q1 queries (objects present vs. absent at the query location). Given that Q1 queries are drawn from both ground-truth locations and detection outputs (including false positives), it would be informative to know whether V2V-LLM's cooperative fusion is especially good at rejecting false positive detections from individual CAVs (a key cooperative capability). The dataset statistics in Tables XVI and XVII show that negative cases are plentiful (e.g., 137,417 negative Q1 training instances in V2V-split), but the evaluation does not break out performance by case type.

Do the planning results support the claim that V2V-LLM is a promising unified model for cooperative planning?

The planning results provide the strongest evidence for V2V-LLM's effectiveness. The paper claims V2V-LLM can serve as "a unified multimodal foundation model that can effectively perform perception and planning tasks for cooperative autonomous driving." Several pieces of evidence support this:

Magnitude of improvement. On V2V-split, V2V-LLM's 3.00% collision rate is substantially lower than all baselines: no fusion (4.57%), early fusion (3.55%), and all intermediate fusion methods (3.88–4.33%). The gap between V2V-LLM and the next-best (early fusion, 3.55%) is 0.55 percentage points — a 15.5% relative reduction in collision rate. This is a meaningful safety improvement.

Non-LLM baseline comparison (Table V). The 3.86× reduction in collision rate compared to the modular CoBEVT + BEV-planner pipeline (3.00% vs. 11.59%) is the paper's most striking result. This demonstrates that the unified architecture finds planning solutions that a state-of-the-art modular pipeline cannot — a finding with real implications for system architecture design in cooperative driving.

Cross-split consistency. V2V-LLM achieves the best planning performance on both V2V-split and V2X-split, with the V2X-split advantage being even larger in relative terms (6.89% CR vs. 8.39–9.21% for baselines). This suggests the result is not dataset-specific.

Temporal horizon generalization. The advantage persists across all three temporal horizons (1s, 2s, 3s) shown in Tables XI and XII, with V2V-LLM achieving the best L2 error and collision rate at every horizon.

However, there are important caveats that qualify the strength of this claim:

Absolute collision rates remain high. A 3.00% collision rate means that in 3 out of every 100 test frames, V2V-LLM's predicted trajectory intersects a ground-truth object's location. For a safety-critical system, this is unacceptably high. The 3-second collision rate of 5.25% is even worse. The paper does not discuss whether these collisions occur in easily avoidable scenarios (e.g., the model drives straight into a clearly visible obstacle) or in genuinely difficult edge cases (e.g., highly occluded scenes where even cooperative fusion provides incomplete information). Without this analysis, it's unclear whether the remaining failures are addressable through architectural improvements or represent fundamental limitations of the LiDAR-only, open-loop approach.

Open-loop evaluation limitation. The collision rate is computed by checking whether predicted waypoints fall within annotated bounding boxes at the corresponding future timestep. This does not account for how the ego-vehicle's motion would interact with the future motion of other agents — the other vehicles and pedestrians are assumed to remain at their current annotated positions, which is unrealistic. An open-loop collision may not correspond to an actual collision in closed-loop simulation (because the other agent might move away, or the ego-vehicle might react to it). Conversely, the model might avoid annotated objects but collide with objects it fails to detect, which would not be captured. This is standard practice in the field (OmniDrive, TOKEN, DriveLM all use open-loop metrics) but it means the collision rate should be interpreted as a relative ranking metric rather than an absolute safety estimate.

Ground-truth trajectory as target. The model is trained to imitate human driving trajectories, which may not be optimal or collision-free themselves. If human drivers in the training data occasionally follow too closely or take suboptimal paths, the model learns to replicate those behaviors. The paper does not analyze whether the ground-truth trajectories in V2V4Real and V2X-Real are themselves collision-free or close to any safety boundaries.

Missing closed-loop evaluation. The paper would be substantially strengthened by even a simple closed-loop evaluation: feed the predicted trajectory to a vehicle dynamics model, simulate the ego-vehicle's motion for 3 seconds in the static scene, and check for collisions. This would validate whether the open-loop collision rate metric correlates with actual safety outcomes and would reveal whether the L2 errors (which measure trajectory imitation quality) translate into collision avoidance.

Do the baseline comparisons support the claim that LLM fusion is a distinct and advantageous fusion paradigm?

The paper positions LLM fusion as a new category of multi-vehicle data fusion. The evidence partially supports this distinction but leaves open questions about whether the advantages come from the LLM itself or from other design choices.

Evidence for distinctiveness. The performance patterns show that V2V-LLM behaves differently from intermediate fusion methods: it is weaker on pure detection-style grounding (Q1) but stronger on relational reasoning (Q4, Q5). This task-dependent performance profile is consistent with the claim that LLM fusion performs question-conditioned integration rather than fixed, detection-optimized aggregation. If LLM fusion were simply a different implementation of the same thing as intermediate fusion, one would expect correlated performance across all tasks.

Confounding factors in the comparison. There are several differences between V2V-LLM and the intermediate fusion baselines beyond the fusion mechanism itself:

Feature content: Intermediate fusion methods use only scene-level feature maps. V2V-LLM uses both scene-level and object-level features. The ablation (Table VIII) shows that object-level features alone outperform scene-level features alone (QGr 37.8 vs. 34.4, Q4 52.6 vs. 43.2). This means V2V-LLM has access to richer input features than the intermediate fusion baselines — it's not a fair comparison of fusion paradigms when the input information differs. A fairer comparison would feed both feature types to all methods.
Pre-training: V2V-LLM benefits from LLaVA's vision-language pre-training, while the intermediate fusion baselines use detection-only pre-training (the cooperative detection models were trained on V2V4Real / V2X-Real detection tasks). The from-scratch ablation (Table VIII) shows that pre-training provides a ~3.6 QGr F1 advantage. If the intermediate fusion baselines received equivalent pre-training on vision-language tasks, their performance might improve substantially, narrowing or eliminating V2V-LLM's advantage.
Model capacity: The LLM backbone (Vicuna-7b, ~7B parameters) provides substantially more capacity for learning task-specific representations than the attention fusion modules in AttFuse, V2X-ViT, and CoBEVT, which are relatively lightweight architectures designed for efficient feature aggregation. V2V-LLM's advantage might partly reflect model scale rather than architectural paradigm.

Missing comparison: late fusion with LLM. The paper does not include a late fusion baseline that would cleanly isolate the value of LLM-based fusion. A natural late fusion approach would be: each CAV independently runs detection and produces object lists, these lists are merged via NMS or bipartite matching, and the merged object list (plus scene features) is fed to the same LLM. This would use the same LLM architecture as V2V-LLM but with fusion happening outside the LLM (via geometric association) rather than inside the LLM (via self-attention). Comparing LLM fusion to late fusion + LLM would directly test whether fusion inside the LLM provides benefits beyond what geometric fusion + LLM reasoning can achieve. The absence of this baseline is a significant gap in the experimental design.

Do the ablation studies cover the most important design choices?

The paper includes three ablations: feature type (scene vs. object), training initialization (pre-trained vs. scratch), and robustness to latency/noise. Several important design choices are not ablated:

Number of CAVs. All experiments use exactly two CAVs. The paper's problem formulation supports an arbitrary number, but the architecture's scaling behavior with 3, 4, or more CAVs is unknown. Does performance improve monotonically with more CAVs (more viewpoints = better fused perception), or does it degrade (more features = harder for the LLM to attend to relevant information)? The communication cost scaling analysis (Table IV) shows linear growth with Nv, but performance scaling is not evaluated.

LLM scale. The paper uses a 7B-parameter LLM backbone. How would performance change with a smaller LLM (e.g., 1.3B or 3B parameters, which would reduce inference latency and computational cost) or a larger one (e.g., 13B parameters)? In LLM research, scaling behavior is often non-trivial. The paper does not explore whether the cooperative driving capability emerges only at certain model scales.

LoRA rank. The paper uses LLaVA's default LoRA configuration but does not specify the rank or ablate it. A higher LoRA rank increases trainable parameters and may improve adaptation to the LiDAR driving domain at the cost of more training compute and memory.

Projector architecture. The projector is described as an MLP without details on depth, width, or activation function. Alternative projector designs (e.g., cross-attention, perceiver resampler, linear projection) are not compared. Given that the projector is the only learned interface between the frozen perception encoder and frozen LLM, its design could significantly impact performance.

Training data composition. The model is trained on all five question types simultaneously. There is no ablation on whether multi-task training is beneficial or whether training on individual question types separately would improve per-task performance. There is also no ablation on the effect of dataset scale — whether training on a subset of the 1.45M QA pairs would achieve similar performance.

Camera data. The paper uses LiDAR-only inputs. No ablation compares LiDAR-only vs. LiDAR+Camera or Camera-only performance. Cameras provide complementary information (color, texture, traffic light states, sign text) that could improve planning and object identification, and many production autonomous driving systems use both modalities.

Are the reported results statistically reliable?

The paper does not report any measure of statistical uncertainty. All results in Tables III, V, VI, VII, VIII, IX, X, XI, XII, XIV, and XV are single-point estimates with no confidence intervals, standard deviations, or statistical tests. Several factors make this concerning:

Test set sizes. The V2V-split test set has 1,993 frames and ~147K QA pairs (Table II). The V2X-split test set has 1,253 frames and ~174K QA pairs. While the number of QA pairs is large, they are not independent — multiple QA pairs come from the same frame and the same driving sequence. The effective sample size for collision rate estimation is closer to the number of frames (or sequences) than the number of QA pairs. A 3.00% collision rate on V2V-split corresponds to approximately 60 frames where a collision is predicted — the statistical uncertainty around this number (given 1,993 test frames) could be several tenths of a percentage point, making the ranking between closely-performing methods (e.g., V2V-LLM at 3.00% vs. early fusion at 3.55%) potentially unstable.

Sequence-level correlations. The 9 testing sequences per split likely contain correlated frames (consecutive frames from the same driving scenario). If a particular sequence contains unusually challenging occlusion scenarios, it could disproportionately influence the aggregate metrics. The paper does not report per-sequence performance or account for sequence-level clustering in any statistical analysis.

No cross-validation or multiple random seeds. The model is trained once and evaluated once on the fixed test split. There is no report of training with different random seeds to assess training stability, and no cross-validation across different train/test splits within the dataset.

Given these concerns, the performance differences among the top methods on grounding tasks (QGr: 40.7 vs. 40.9 vs. 40.5 — differences of 0.2–0.4 F1) are likely within statistical noise. The paper's claim that V2V-LLM "achieves competitive results in the grounding tasks" is appropriately cautious, but stronger claims about V2V-LLM's grounding superiority would be unwarranted without statistical backing. The planning results (collision rate gaps of 0.55–1.57 percentage points) are larger and more likely to be statistically meaningful, but this cannot be confirmed without the relevant tests.

Do the robustness experiments adequately assess real-world deployability?

The latency and noise experiments (Tables VI, VII) are a good start but address only two of many real-world challenges:

Bandwidth constraints. The communication cost analysis assumes 0.4068 MB per timestep at 10 Hz = ~4 MB/s per CAV. This is feasible for modern V2V communication (DSRC supports 3–27 Mbps, C-V2X supports higher), but the analysis does not consider packet loss, variable latency, or channel congestion when many CAVs share the same spectrum. No experiment degrades communication quality below the full-feature transmission.

CAV dropout. What happens if one CAV suddenly stops transmitting (e.g., due to communication failure or sensor malfunction)? The paper does not test robustness to missing CAVs at inference time, which would be common in real deployments.

Heterogeneous sensor quality. The underlying datasets use identical LiDAR sensors on both CAVs. Real cooperative driving would involve vehicles with different sensor suites (different LiDAR models, different camera configurations, some vehicles with only cameras). The paper does not test robustness to heterogeneous perception quality across CAVs.

Adversarial scenarios. A malicious or malfunctioning CAV could transmit corrupted features designed to confuse the LLM. The paper's noise experiments add Gaussian noise to positions — a relatively benign perturbation. More adversarial perturbations (systematic bias, swapped detections, deleted objects) are not tested.

Computational latency. The paper focuses on communication latency but does not measure the LLM's inference time. A 7B-parameter model processing features from multiple CAVs and generating a trajectory may require significant compute (hundreds of milliseconds to seconds), which could exceed the real-time requirements of autonomous driving (100ms for planning updates is a common target). The paper does not report inference latency on the target hardware (A100 GPUs) or any optimized deployment configuration.

6. Limitations and Trade-offs

Centralized Architecture with Unresolved Latency and Reliability Concerns

The paper assumes a "centralized LLM computing node" that receives perception features from all CAVs, processes them, and returns answers. The authors state this assumption explicitly in Section III-A:

"In this setting, we assume there are multiple Connected Autonomous Vehicles (CAVs) and a centralized LLM computing node."

This architectural choice carries significant deployment consequences that the paper acknowledges only in passing. A centralized node represents a single point of failure: if the LLM computing node crashes, loses connectivity, or suffers a hardware fault, all CAVs lose their cooperative perception and planning capability simultaneously. In a safety-critical application like autonomous driving, this failure mode is unacceptable without redundant fallback mechanisms — each CAV would need a local planning system that can operate independently when the central node is unavailable, which the paper does not design, test, or discuss.

The latency implications are also unexamined. The paper measures communication latency robustness (Table VI) but does not report the end-to-end inference latency of the centralized LLM. A 7-billion-parameter model (Vicuna-7b) processing perception tokens from multiple CAVs and autoregressively generating multi-token answers (including trajectory coordinates) likely requires hundreds of milliseconds to multiple seconds on A100 GPUs, depending on batching and optimization. Autonomous driving planning typically requires updates at 10 Hz (100ms cycles) — if the centralized LLM cannot meet this deadline, the system cannot operate in real time. The paper provides no inference latency measurements on any hardware configuration, not even the 8× A100-80GB training setup. The fact that communication latency experiments (Table VI) only go up to 1.0 second — and that at 0.4 seconds latency, Q5 collision rate already rises from 3.00% to 3.49% — suggests that total system latency (communication + inference) could easily exceed acceptable bounds.

What evidence exists: Table VI measures communication latency separately from inference latency. The paper states the training hardware (8× A100-80GB) in Section IV-B but does not report inference throughput or latency. Appendix VIII discusses a decentralized alternative where each CAV runs its own LLM, but this is described conceptually without experimental validation:

"one can also consider a decentralized setting that deploys one LLM in each CAV... Which setting works best in terms of communication costs is beyond the current focus of our work."

Mitigation status: Not addressed. The paper treats the centralized assumption as a simplifying choice for initial exploration. The decentralized alternative is mentioned only in the appendix as a possibility, not evaluated. There is no discussion of redundancy, failover, or how the system degrades when the central node is unavailable. This is a fundamental architectural limitation that any practical deployment would need to resolve, and the paper provides no empirical guidance on how to do so.

Open-Loop Planning Evaluation Masks True Safety Performance

All planning evaluation in the paper is open-loop: the model predicts a sequence of 6 waypoints over 3 seconds, and these waypoints are compared against ground-truth trajectories (for L2 error) and annotated object bounding boxes at the corresponding future timesteps (for collision rate). The vehicle is never simulated executing the predicted trajectory, and the model never observes the consequences of its own actions. This is standard practice in the autonomous driving literature — OmniDrive, TOKEN, and DriveLM all use open-loop metrics — but it fundamentally limits what the collision rate metric can tell us about real-world safety.

The specific problem is that open-loop collision checking assumes a static world. When the paper reports a 3.00% collision rate for V2V-LLM on V2V-split (Table III), it means that in 3% of test frames, at least one predicted waypoint intersects a ground-truth annotated object's position at that future timestep. But in reality, the ego-vehicle's motion and other agents' motions are coupled — if the ego-vehicle swerves to avoid a detected obstacle, the obstacle might also move, or the ego-vehicle's new position might create secondary risks not captured in the static annotation. An open-loop "collision" might not occur in closed-loop simulation (the other agent might have moved by the time the ego-vehicle reaches that location), and conversely, an open-loop "safe" trajectory might cause a collision in closed-loop (the ego-vehicle might move into the path of an agent whose future motion was not considered).

The L2 error metric has a complementary weakness: it measures how well the model imitates human driving, not how safe the trajectory is. A model could achieve low L2 error by closely matching human trajectories that are themselves suboptimal (e.g., following too closely, taking unnecessarily sharp turns). The paper does not analyze whether the ground-truth trajectories in V2V4Real and V2X-Real are collision-free or represent optimal driving behavior. The 3.00% collision rate for V2V-LLM (Table III) occurs despite training to imitate these trajectories, suggesting some degree of mismatch between human driving and collision-free planning.

What evidence exists: The paper reports L2 error and collision rate at 1s, 2s, and 3s horizons (Tables XI, XII), with collision rates increasing substantially over longer horizons (e.g., V2V-LLM on V2V-split: 0.55% at 1s → 5.25% at 3s). This temporal degradation is expected in open-loop evaluation — predicting further into the future is harder — but it does not tell us whether these "collisions" would manifest in actual driving. The paper provides no closed-loop simulation results, no ablation comparing open-loop collision predictions to any form of dynamic validation, and no discussion of how the open-loop metrics relate to real-world safety outcomes.

Mitigation status: Not addressed. The paper acknowledges the open-loop nature of the evaluation only implicitly (by using the standard metrics and citing OmniDrive/TOKEN) but does not discuss its limitations or suggest closed-loop validation as future work. This is a significant gap because the paper's headline result — 3.86× lower collision rate than the non-LLM baseline (Table V) — depends entirely on the open-loop metric. If open-loop collision rate is not a reliable proxy for closed-loop safety (which is plausible given the static-world assumption), the practical significance of this result is uncertain.

The Approach Cannot Handle Hard Perception Failures (Difficulty Bin 5 Equivalent)

Across all experiments, the paper demonstrates that cooperative fusion improves perception and planning, but it does not characterize when cooperation fails to help. This is the "difficulty bin 5" problem from the test-time compute scaling paper: some scenarios are beyond the base capability of the system, and no amount of additional computation or data fusion can compensate. For V2V-LLM, the equivalent question is: when both CAVs fail to detect an object, does the LLM have any mechanism to infer its presence?

The answer appears to be no. V2V-LLM's perception capabilities are fundamentally bounded by the quality of the individual CAV feature extractors (PointPillars). If an object is invisible to all CAVs' LiDAR sensors — for example, a pedestrian occluded by multiple large vehicles in a dense urban intersection, or a small object at long range below the LiDAR's resolution limit — the object leaves no trace in any CAV's scene-level feature maps or object-level feature vectors. The LLM receives no signal about this object and cannot hallucinate its presence through reasoning. The low absolute performance on occlusion-targeted grounding tasks supports this: Q2 F1 is only 30.8 for V2V-LLM on V2V-split (Table III), meaning that in ~69% of occlusion-grounding queries, the model fails to identify the occluded object that the best-positioned CAV could see. In some fraction of these failures, even the best-positioned CAV simply cannot see the object due to its own occlusions or sensor limitations.

This limitation is fundamental because it defines a hard ceiling on what cooperative perception can achieve: cooperation can only reveal objects that at least one CAV can detect. It cannot create information that does not exist in any sensor stream. The paper never discusses this boundary, never analyzes failure cases to distinguish "object visible to at least one CAV but fusion failed" from "object invisible to all CAVs," and never quantifies the fraction of objects that fall into each category in the V2V-QA test sets.

What evidence exists: The performance numbers indirectly reveal this ceiling. Q3 (directional grounding behind a reference object) achieves only 21.2 F1 for V2V-LLM on V2V-split (Table III), and the best method (V2X-ViT) achieves only 22.6. These low absolute scores, even with cooperative fusion, suggest that many occluded objects are simply undetectable. The appendix limitation discussion (Section XIII) focuses on planning failures (wrong-lane trajectories) and proposes HD maps as a solution, not on the more fundamental perception ceiling. The paper does not provide occlusion-level analysis, per-scenario difficulty breakdowns, or any characterization of when cooperation helps versus when it cannot.

Mitigation status: Not addressed. The paper does not analyze failure cases by occlusion severity, does not distinguish between fusion failures and sensor limitations, and does not discuss the fundamental information-theoretic bounds on cooperative perception. The planning limitation discussed in Appendix XIII (wrong-lane trajectories) suggests that some failures might be addressed by adding HD map information, but this would not help with the core perception ceiling — no map can tell you about a pedestrian that no sensor detected. This is a capability boundary that practitioners need to understand: deploying V2V-LLM does not eliminate occlusion risk; it only mitigates it for objects that at least one CAV can partially observe.

LiDAR-Only Input Severely Limits Perception Modality Coverage

All experiments use LiDAR point clouds as the sole sensor modality. The paper inherits this constraint from the V2V4Real and V2X-Real datasets, which are LiDAR-focused, and from the choice of PointPillars as the feature encoder. The authors do not discuss this as a limitation, but it has major consequences for what the system can perceive and how safely it can plan.

LiDAR provides accurate 3D geometry but lacks several types of information critical for autonomous driving that cameras capture: color and texture (distinguishing a concrete barrier from a hedge, or a stationary vehicle from a building facade), traffic light states (red/yellow/green cannot be read from LiDAR intensity), sign text and symbols (speed limits, stop signs, directional signs), lane markings (painted lines on the road surface), and fine-grained object classification (distinguishing a child from an adult, or a cyclist from a motorcyclist, based on visual appearance). These missing modalities have direct safety implications: the model cannot obey traffic lights, cannot adjust speed based on posted limits, cannot recognize that a lane is ending based on pavement markings, and cannot distinguish a stationary vehicle (okay to pass) from a pedestrian standing still (must stop) if both have similar 3D shapes.

The planning task (Q5) is particularly affected. The model generates trajectories solely from LiDAR features — it has no access to lane geometry (unless implicitly learned from point distributions), no traffic rule information, and no semantic understanding of drivable vs. non-drivable surfaces beyond what the 3D point distribution reveals. The failure cases shown in Figure 31 — where "the model generates future trajectories in the lane of the opposite traffic direction" — are likely symptoms of this modality gap. A camera-based system would see lane markings and directional signs indicating which side of the road is the correct driving direction; a LiDAR-only system can only infer road structure from curb heights, barrier positions, and other vehicles' locations, which is far less reliable.

The paper acknowledges this implicitly in the limitation discussion (Appendix XIII), stating that "including HD map information" could help with wrong-lane failures, but HD maps are a post-hoc fix, not a real-time perception solution. They are expensive to maintain, unavailable in many regions, and cannot capture dynamic changes (construction, accidents, temporary lane shifts). Camera input would provide real-time visual cues that complement LiDAR geometry without requiring pre-mapped infrastructure.

What evidence exists: The model architecture and all experiments use only LiDAR input (Section IV-A, IV-B). The base datasets (V2V4Real, V2X-Real) do not provide synchronized camera data usable for the QA tasks. Table I shows that none of the prior cooperative perception datasets (OPV2V, V2X-Sim, V2XSet, DAIR-V2X, V2V4Real, TUMTrafV2X, V2X-Real) include planning QA — the field has not yet built multi-modal cooperative driving datasets that combine LiDAR, camera, and planning annotation. The failure case visualization (Figure 31) shows wrong-lane trajectories that are plausibly caused by missing lane/perception information. No ablation compares LiDAR-only to any multi-modal configuration.

Mitigation status: Not addressed in the main paper. Appendix XIII mentions HD maps as a potential solution for wrong-lane planning failures, but this is not evaluated and does not address the broader perception gaps (traffic lights, signs, object semantics). The limitation is acknowledged only as a specific failure mode (wrong-lane trajectories), not as a fundamental modality constraint that affects all tasks. A practitioner deploying this system would need to integrate camera inputs and train a multi-modal version, which the paper provides no guidance for.

Training Data Quality: Ground-Truth Trajectories Are Human Driving, Not Optimal or Safety-Critical Driving

The planning task (Q5) is trained to imitate ground-truth human driving trajectories extracted from the V2V4Real and V2X-Real datasets. The paper states this in Section III-C:

"we extract 6 future waypoints, evenly distributed in the next 3 seconds, from each CAV's ground-truth future trajectory as the answer."

This means the model learns to replicate what human drivers actually did during data collection — not what an optimal or maximally safe driver would do. There are several concerning implications:

Human drivers are not collision-optimized. The human drivers in the datasets were navigating normally, not executing emergency avoidance maneuvers. Their trajectories may include behaviors that are legal and common but suboptimal for safety: following at close distance, accelerating through yellow lights, making rolling stops, or taking cornering lines that leave small safety margins. The model learns to reproduce these patterns.

No adversarial or safety-critical scenarios. The datasets were collected during normal driving — not during near-misses, emergency braking events, or collision-avoidance maneuvers. The QA pairs for Q4 and Q5 ask about objects near the planned trajectory, but the trajectories themselves may never come close enough to objects to create genuine safety-critical decisions. If the training data contains no examples of the model needing to swerve, brake hard, or take evasive action, it cannot learn these behaviors. At test time, if a genuinely dangerous scenario occurs (an object is directly in the planned path and braking alone is insufficient), the model may generate a trajectory that imitates normal driving rather than executing an emergency maneuver.

Open-loop training mimics, not optimizes. Because training uses next-token prediction (standard language modeling loss) rather than a safety-aware objective, the model is not explicitly penalized for generating trajectories that collide with objects — it is only penalized for generating tokens that differ from the ground-truth answer string. If the ground-truth human driver happened to pass close to an object (but not collide), the model learns that such proximity is acceptable. In a different scene where that proximity would actually cause a collision, the model has no safety-aware training signal to tell it to stay further away.

Evidence in the paper: The collision rate of V2V-LLM is 3.00% on V2V-split (Table III). Given that the model was trained to imitate collision-free human trajectories, one might expect near-zero collision rate on the training distribution. The fact that even the best model collides in 3% of test frames suggests that imitation learning alone does not guarantee collision avoidance when generalizing to new scenes. The collision rate gap between V2V-LLM and the no-fusion baseline (3.00% vs. 4.57%) demonstrates that cooperative perception helps reduce collisions, but the residual 3.00% — combined with the 5.25% 3-second collision rate (Table XI) — indicates that the imitation objective has not solved the safety problem.

Mitigation status: Not addressed. The paper does not discuss the quality or safety properties of the ground-truth trajectories, does not analyze whether collisions in the model's outputs correspond to scenarios where the human driver would also have been at risk, and does not explore alternative training objectives (e.g., safety-aware reward modeling, reinforcement learning from human feedback, or adversarial training with collision penalties). The planning task is framed as trajectory prediction from fused perception data, not as safety-critical decision making — the distinction is important for deployment but absent from the paper's analysis.

The V2V-LLM vs. Intermediate Fusion Comparison Is Confounded by Feature Asymmetry

The paper's central experimental claim is that LLM fusion (V2V-LLM) outperforms intermediate fusion methods (AttFuse, V2X-ViT, CoBEVT) on notable object identification and planning tasks (Table III, Section V-B). However, this comparison contains a confounding variable that prevents clean attribution of the performance difference to the fusion paradigm: V2V-LLM receives both scene-level feature maps and object-level feature vectors as input, while all intermediate fusion baselines receive only scene-level feature maps.

The intermediate fusion methods (AttFuse, V2X-ViT, CoBEVT) are cooperative detection architectures that fuse feature maps from multiple CAVs through attention mechanisms. Their standard implementation — as released by the V2V4Real and V2X-Real authors and used in this paper — outputs fused scene-level feature maps. They do not produce object-level feature vectors in the format that V2V-LLM uses (explicit detection result parameters: location, size, orientation, class). The paper feeds only the fused scene-level features from these models to the LLM.

V2V-LLM, by contrast, receives both scene-level features (from per-CAV PointPillars) and object-level features (per-CAV detection results). The paper's own ablation (Table VIII) demonstrates that object-level features alone outperform scene-level features alone: on V2V-split, object-only achieves QGr 37.8 vs. scene-only 34.4, Q4 52.6 vs. 43.2, and Q5 collision rate 7.78% vs. 15.55%. The full model (both feature types) achieves 40.7 QGr, 59.7 Q4, and 3.00% Q5 CR — better than either alone. This means the intermediate fusion baselines are operating at a systematic information disadvantage: they lack access to explicit object location priors that V2V-LLM leverages, and the ablation shows these priors are highly valuable.

The fair comparison would be to provide the same feature types to all methods. This could be done by: (1) running per-CAV detection (as V2V-LLM does) and concatenating the resulting object-level features with the intermediately-fused scene features before feeding to the LLM, or (2) using only scene-level features for V2V-LLM (the scene-only ablation in Table VIII) and comparing that against the intermediate fusion baselines on equal footing. When we look at the scene-only V2V-LLM versus intermediate fusion (both using only scene features), the ordering changes dramatically: scene-only V2V-LLM achieves QGr 34.4, compared to CoBEVT 40.9, V2X-ViT 40.5, and AttFuse 38.5 on V2V-split. For Q5, scene-only V2V-LLM achieves 15.55% collision rate versus CoBEVT's 3.88% and early fusion's 3.55%. On scene features alone, V2V-LLM underperforms intermediate fusion substantially. The full V2V-LLM's advantage over intermediate fusion is therefore largely (perhaps entirely) attributable to the inclusion of object-level features — not to the LLM fusion mechanism itself.

The paper does not acknowledge this confound. It presents V2V-LLM's full-model results against intermediate fusion baselines as evidence that "LLM fusion" is advantageous, without noting that the comparison is between models with different input information. The communication cost analysis (Table III) also reflects this asymmetry: V2V-LLM's 0.4068 MB includes both scene and object features, while intermediate fusion's 0.4008 MB includes only scene features. If intermediate fusion methods were augmented with equivalent object-level features, their communication cost would increase and their performance would likely improve.

Evidence in the paper: The confounding is visible by comparing: (a) Table III full V2V-LLM vs. intermediate fusion (V2V-LLM leads on Q4, Q5), (b) Table VIII scene-only V2V-LLM vs. Table III intermediate fusion (intermediate fusion leads dramatically on all metrics), and (c) Table VIII scene-only vs. object-only vs. full V2V-LLM (object features provide large gains over scene features alone). The paper does not draw attention to these cross-table comparisons or discuss the confound.

Mitigation status: Not addressed. The paper does not run the intermediate fusion baselines with object-level features, does not compare scene-only V2V-LLM directly against intermediate fusion in the main results table, and does not discuss this as a limitation or alternative explanation for the observed performance differences. A practitioner reading only the main results (Table III) would conclude that LLM fusion is better than intermediate fusion for planning, without realizing that the comparison is not between equivalent information inputs. This is a significant methodological weakness that undermines the paper's primary architectural claim.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not shift a paradigm so much as create one where none existed before. Prior to V2V-LLM, cooperative autonomous driving research had a clear structure: perception researchers built fusion architectures (AttFuse, V2X-ViT, CoBEVT) that improved detection and tracking metrics, while planning researchers worked with single-vehicle sensor streams. The connection between these activities was assumed but never tested — the field operated on the implicit premise that better cooperative perception would naturally translate into safer driving. V2V-LLM demonstrates that this premise is empirically false in the straightforward modular form: the best cooperative detection model (CoBEVT) fed into a standard planner (BEV-planner) achieves an 11.59% average collision rate, while the unified LLM-based architecture achieves 3.00% (Table V). This 3.86× gap is not a marginal improvement — it indicates that the interface between cooperative perception and planning is a first-order bottleneck, not a minor engineering detail.

The conceptual shift is therefore from cooperative perception as an independent sub-problem to cooperative perception-planning as a joint design problem. This reframes what "good performance" means in cooperative driving. A cooperative perception paper that reports state-of-the-art detection mAP but never connects to planning is now, after V2V-LLM, answering only half the question. The field must demonstrate that perception improvements survive the transition to planning decisions — or, better yet, design architectures where perception and planning are trained jointly so that perception features are optimized for planning usefulness, not merely detection accuracy.

The paper also resolves a latent tension in the autonomous driving architecture debate. The industry has largely converged on modular designs — separate perception, prediction, and planning modules with defined interfaces — for reasons of interpretability, independent testing, and engineering team organization. The LLM-based driving community (DriveLM, TOKEN, OmniDrive) has challenged this with evidence that end-to-end architectures can outperform modular pipelines on single-vehicle tasks. V2V-LLM extends this challenge to cooperative driving with the strongest evidence yet: in the cooperative setting, the modular penalty appears to be larger than in the single-vehicle setting. The CoBEVT + BEV-planner collision rate of 11.59% versus V2V-LLM's 3.00% (Table V) represents a larger relative gap than what OmniDrive or TOKEN report for their single-vehicle unified-vs-modular comparisons. This suggests that the information loss at the perception-planning interface is especially damaging in cooperative scenarios — perhaps because occluded objects that cooperative perception recovers are precisely the ones that standard modular interfaces are worst at communicating to planners (they appear as low-confidence detections, partially reconstructed from another vehicle's viewpoint, with positional uncertainty that a bounding-box interface poorly captures).

An important reframing concerns what an LLM is contributing to cooperative driving. The paper's ablation (Table VIII) reveals that object-level features alone outperform scene-level features alone across all tasks, and the combination of both feature types yields the strongest results. This suggests a specific role for the LLM: it excels at symbolic-spatial reasoning over explicit object representations (locations, sizes, orientations, class labels) while also benefiting from dense geometric context (scene feature maps). This is more nuanced than "LLMs bring common sense to driving." The LLM is acting as a learned fusion and reasoning engine that can resolve multi-vehicle observations into a coherent scene representation and answer structured queries — it is not relying on pre-trained world knowledge about traffic norms or social conventions (the training data provides ground-truth trajectories as targets, not open-ended reasoning). Future work that assumes LLMs contribute general driving knowledge to cooperative scenarios should test this assumption explicitly, as V2V-LLM's success may be primarily attributable to its capacity for high-dimensional feature integration rather than to any driving-specific pre-training.

The paper also shifts the evaluation paradigm for cooperative driving. Prior datasets (OPV2V, V2X-Sim, V2XSet, V2V4Real, V2X-Real) provide only detection and tracking benchmarks. V2V-QA establishes that cooperative driving systems can and should be evaluated on end-to-end question answering that includes planning. This is not merely additive — it changes the incentive structure for researchers. A cooperative perception architecture that achieves +2 mAP on detection but introduces artifacts that confuse a downstream planner might now look worse than an architecture with +0 mAP that preserves planning-relevant information. V2V-QA's design, with its occlusion-targeted grounding questions (Q2, Q3) and planning task (Q5), provides the measurement tools to detect such tradeoffs. The fact that CoBEVT achieves the best Q1 F1 (72.2) but worse Q5 collision rate than early fusion (3.88% vs. 3.55%, Table III) is exactly the kind of perception-planning tradeoff that detection-only benchmarks would miss and that V2V-QA captures.

Finally, the paper creates research pressure on a specific technical problem: closing the information gap between cooperative perception and planning without sacrificing modularity. The stark CoBEVT + BEV-planner result (Table V) suggests that modular pipelines using standard bounding-box interfaces are fundamentally limited for cooperative planning. This makes two research directions more attractive than they were before: (1) unified architectures that avoid the interface entirely (the V2V-LLM approach), and (2) richer modular interfaces that preserve more than bounding boxes — feature maps, uncertainty estimates, and raw point clusters from occluded regions. Conversely, the paper makes incremental improvements to detection-only cooperative perception less attractive unless accompanied by planning validation. A new cooperative detection architecture that achieves +1 AP on V2V4Real but is never evaluated on a planning task is now answering a question the field increasingly recognizes as incomplete.

Follow-Up Research This Work Enables

Closed-loop evaluation of V2V-LLM in a simulator with dynamic agents. The paper's planning evaluation is entirely open-loop: waypoints are compared against static ground-truth annotations without simulating the vehicle's motion or other agents' reactions. A strong follow-up would deploy V2V-LLM in a cooperative driving simulator (e.g., CARLA with multi-vehicle support, or the OPV2V simulator configured for planning) where the model's predicted trajectories are executed, other vehicles react, and true collision outcomes are measured over multi-step rollouts. The specific question: does the 3.00% open-loop collision rate (Table III) translate to a similar rate in closed-loop, or is it substantially higher (because static-world checking misses dynamic conflicts) or lower (because the vehicle can course-correct)? This experiment would also reveal whether V2V-LLM's imitation-learned trajectories are dynamically feasible — the open-loop evaluation ignores vehicle kinematics, so waypoints that look reasonable on paper might require impossible accelerations or steering angles. The experiment should compare V2V-LLM against the CoBEVT + BEV-planner baseline in the same closed-loop setting to confirm whether the 3.86× collision rate ratio (Table V) is robust to dynamic simulation. A negative result — V2V-LLM's advantage shrinking or reversing in closed-loop — would be just as informative as a positive one, as it would reveal that open-loop metrics reward trajectory-matching rather than safety.

Adding camera inputs to V2V-LLM and measuring the modality contribution. The paper's LiDAR-only approach inherits the modality limitation of the underlying datasets. A critical follow-up would integrate camera features alongside LiDAR features in the V2V-LLM architecture and measure the incremental benefit per task. The specific experimental question: does adding camera input reduce the 3.00% collision rate (V2V-split, Table III) or the wrong-lane planning failures shown in Figure 31? Camera features should help most with tasks that LiDAR alone cannot solve: recognizing traffic light states, reading speed limit signs, detecting lane markings, and classifying fine-grained object types (pedestrian vs. cyclist, stationary vehicle vs. building). The failure cases in Figure 31 — trajectories into opposite-direction lanes — are precisely the kind of error that lane marking perception should eliminate. This experiment requires a dataset that provides synchronized multi-vehicle LiDAR and camera data with planning annotations. Since no such dataset currently exists (V2V4Real and V2X-Real are LiDAR-only for the cooperative setting), building it would itself be a contribution. The experiment should ablate: LiDAR-only, camera-only, LiDAR+camera, and measure the collision rate and L2 error delta attributable to each modality. A negative finding — cameras providing negligible planning improvement despite helping with perception tasks — would reveal that V2V-LLM's planning failures are primarily geometric rather than semantic, redirecting research toward better trajectory optimization rather than multi-modal fusion.

Measuring whether LLM fusion recovers from individual CAV perception failures. The paper designs Q1 queries to include false positive and false negative detection locations from individual CAVs, but the evaluation aggregates all queries together and does not report performance specifically on the subset where cooperative fusion should make the biggest difference. A targeted follow-up would: (1) identify the subset of Q1 queries where the asking CAV's individual detector made an error (false positive or false negative at the query location), (2) measure V2V-LLM's F1 on this subset versus the same metric for no-fusion and intermediate-fusion baselines, and (3) compare against a hypothetical "oracle fusion" that always correctly resolves individual errors using all available information. This would directly quantify how much of the gap between V2V-LLM and no-fusion on Q1 (70.0 vs. 66.6 F1, V2V-split, Table III) comes from correcting individual perception errors versus simply having additional views of already-visible objects. The experiment would also reveal whether V2V-LLM's cooperative correction capability is symmetric — does it equally well reject false positives (CAV-1 hallucinates an object, V2V-LLM correctly says "nothing there") and recover false negatives (CAV-1 misses an object, V2V-LLM correctly identifies it)? Asymmetric performance would indicate specific failure modes — e.g., the model might be better at accepting additional evidence (recovering missed detections) than at rejecting misleading evidence (overriding false positives), or vice versa. This analysis requires no new data, only a stratified evaluation on the existing V2V-QA test set.

Training a "difficulty estimator" for cooperative driving scenarios. The paper provides no analysis of when cooperative fusion helps versus when it cannot — the equivalent of the "difficulty bin 5" problem from the test-time compute scaling paper, where some scenarios are beyond the system's capability regardless of budget. A natural extension would train a lightweight classifier to predict, from the CAVs' individual perception features alone (before fusion), whether cooperative fusion is likely to improve planning safety for a given scene. The training signal: for each frame in V2V-QA, compute the delta between V2V-LLM's collision probability and the no-fusion baseline's collision probability. Frames where this delta is large are "high-cooperation-benefit" scenarios; frames where it is near zero are "cooperation-irrelevant" (because both vehicles already see everything, or because neither vehicle can see the critical hazard). The classifier would take per-CAV feature statistics (e.g., the fraction of the asking CAV's field of view occluded by nearby objects, the overlap between the two CAVs' fields of view, the number of objects detected by only one CAV) and predict the cooperation benefit. This would enable adaptive V2V communication: in low-benefit scenarios, CAVs could reduce transmission bandwidth or skip sharing entirely, saving communication resources for the occlusion-heavy scenarios where cooperation matters most. The experiment should measure: (a) the classifier's accuracy at identifying high-benefit frames, (b) the collision rate when communication is gated by the classifier (sharing only when predicted benefit exceeds a threshold) versus always-on communication, and (c) the communication bandwidth saved. A strong result would show that selective communication achieves near-identical safety to always-on sharing while using a small fraction of the bandwidth.

Scaling the number of CAVs and measuring diminishing returns. All experiments use exactly two CAVs, inherited from the V2V4Real and V2X-Real datasets. A critical scaling experiment would measure V2V-LLM's performance as the number of CAVs increases from 2 to 5+ — something that would require either a new real-world data collection effort or repurposing the simulated OPV2V and V2X-Sim datasets (which support 2–7 and 2–5 CAVs respectively, though these datasets lack planning QA pairs). The specific question: does planning performance (collision rate, L2 error) improve monotonically with more CAVs, or does it saturate after a certain number? More CAVs provide more viewpoints, potentially revealing occluded objects that even two vehicles cannot see jointly, but they also increase the communication cost linearly (Table IV: 0.203Nv + 0.0004NqNv MB for the centralized node) and may introduce conflicting observations that challenge the LLM's fusion capacity. If performance saturates at 3–4 CAVs, this would provide practical guidance for V2V network design — there is no benefit to connecting more vehicles than this saturation point. If performance degrades beyond a certain number (due to attention dilution or feature overload), this would reveal a fundamental limitation of the current LLM fusion approach that might require hierarchical fusion or attention routing mechanisms. The experiment should also measure inference latency scaling with Nv, as the LLM's self-attention complexity grows quadratically with total token count, potentially making large-Nv deployment infeasible in real time.

Testing whether pre-trained driving knowledge transfers, or whether V2V-LLM learns only from V2V-QA supervision. The paper initializes V2V-LLM from LLaVA-v1.5-7b weights and demonstrates that training from scratch performs substantially worse (Table VIII: 49.3 vs. 59.7 Q4 F1, 5.01% vs. 3.00% Q5 CR). But LLaVA was pre-trained on general visual question answering (COCO, LAION, instruction-tuning data), not on driving-specific data. An informative follow-up would compare V2V-LLM's LLaVA initialization against initialization from a driving-specific VQA model (e.g., one fine-tuned on DriveLM or NuScenes-QA before transfer to V2V-QA). The question: does exposure to single-vehicle driving QA during pre-training provide additional benefit for cooperative driving beyond what general vision-language pre-training provides? If yes, it suggests that some driving-relevant capabilities (spatial reasoning about vehicles, understanding of traffic scenarios, trajectory prediction) transfer from single-vehicle to cooperative settings, and pre-training on larger single-vehicle driving datasets would be a cost-effective way to improve cooperative performance. If no — if general VQA pre-training matches driving-specific pre-training — it suggests that the cooperative task requires fundamentally different reasoning (multi-viewpoint integration, occlusion inference) that single-vehicle driving data does not teach, and efforts should focus on scaling cooperative-specific training data rather than leveraging single-vehicle datasets. This experiment also has methodological value: it tests whether the LLaVA initialization advantage (over scratch training) is a generic pre-training effect or specific to the task domain.

Practical Applications and Downstream Use Cases

Safety-focused cooperative driving assistant in occlusion-heavy urban environments. The most direct application is deploying V2V-LLM as a real-time safety advisor in urban driving scenarios where occlusion risk is high: intersections with limited sight lines, dense city traffic with large vehicles blocking views, and multi-lane roads where adjacent vehicles create persistent blind spots. In this setting, each CAV runs lightweight PointPillars detection locally and transmits compact features (0.203 MB per CAV per timestep, Table IV) to a centralized or edge-deployed LLM node. When a CAV approaches an intersection with a blocked view — for example, a delivery truck occluding the crosswalk — it queries the LLM with Q2-style questions ("Is there anything behind the object at [x, y]?") and receives an answer with the location of any occluded pedestrian or cyclist within tens of milliseconds of communication latency (Table VI shows the model tolerates up to 0.2s latency with minimal degradation: 3.21% CR vs. 3.00% at zero latency). The benefit is measured directly in collision avoidance: the no-fusion baseline (single-vehicle perception) achieves 4.57% collision rate on V2V-split, while V2V-LLM achieves 3.00% (Table III) — a 34.4% relative reduction. In a fleet of 1,000 vehicles operating in urban environments where occlusion-related collisions account for, say, 15% of incidents, a 34% reduction translates to approximately 50 fewer collisions annually, assuming baseline collision rates typical of urban autonomous driving pilots. The system could also serve as an aftermarket safety add-on for human-driven connected vehicles, providing occlusion warnings to the driver via a display or audio alert without requiring full autonomous control.

Edge-based cooperative traffic management for smart intersections. V2V-LLM's architecture naturally extends to infrastructure-based deployment where roadside LiDAR units (already present in the V2X-Real portion of the dataset) serve as additional perception sources alongside vehicle-mounted sensors. In a smart intersection scenario, multiple roadside LiDARs provide persistent, wide-angle coverage of the intersection, while passing CAVs contribute their mobile viewpoints. The centralized LLM node fuses all available data and can answer planning queries from any approaching vehicle: "What is the suggested future trajectory to avoid collision with nearby objects?" The roadside infrastructure eliminates the occlusion problem almost entirely for the intersection itself (since overhead or pole-mounted sensors have clear sight lines), while the CAVs extend coverage to the intersection approaches where roadside sensors may have blind spots. The planning benefit is substantial: V2V-LLM achieves 1.71m L2 error and 6.89% collision rate on V2X-split (which includes roadside infrastructure in some scenes, Table III), compared to 2.31m and 9.21% for no-fusion. For intersection scenarios specifically — where NHTSA data shows roughly 40% of all crashes occur — the collision rate reduction from infrastructure-augmented cooperative planning could be even larger than the aggregate numbers suggest, since intersections concentrate the occlusion scenarios that cooperative perception is designed to address. The communication cost scaling (Table IV) is favorable: each CAV still transmits only 0.203 MB per timestep regardless of how many infrastructure sensors are present, and the centralized node's cost scales as 0.203Nv + 0.0004NqNv, meaning even with 10 CAVs asking 5 questions each per timestep, total bandwidth is only ~2 MB per timestep — well within modern V2X communication standards.

Offline data generation for training cooperative planning models. V2V-LLM can serve as a data generation engine for training specialized cooperative planning models that are smaller and faster than the 7B-parameter LLM. The workflow: run V2V-LLM on a large corpus of cooperative driving scenes (potentially including simulated data from CARLA or OPV2V extended with V2V-QA-style questions), collect the generated planning trajectories that pass a safety filter (collision rate below some threshold, dynamically feasible according to a vehicle model), and use these trajectories as training data for a lightweight student model (e.g., a CNN-based trajectory generator with 1–10M parameters) that can run in real time on embedded hardware. The value is that V2V-LLM's 3.00% collision rate on V2V-split (Table III) represents an upper bound on the student model's achievable safety — the student can be trained via distillation to imitate V2V-LLM's outputs, potentially achieving similar collision avoidance with dramatically lower inference latency and without requiring a GPU-equipped central node. The key numbers: V2V-LLM's communication cost is 0.4068 MB per timestep (Table III), but a distilled student model deployed on-vehicle would require 0 MB of V2V communication for the LLM component (only the per-CAV feature sharing at 0.203 MB per CAV remains). This application is particularly attractive for production deployment where the centralized LLM node is impractical due to latency or reliability concerns, but the cooperative perception features are still available via V2V communication.

When to Prefer This Method

The paper frames V2V-LLM against four specific alternative fusion approaches (no fusion, early fusion, intermediate fusion via AttFuse/V2X-ViT/CoBEVT) and one non-LLM modular pipeline (CoBEVT + BEV-planner). The tradeoffs are articulated through the performance results in Tables III and V, communication cost analysis in Table IV, and robustness experiments in Tables VI and VII. Based on these comparisons, the decision boundaries are:

Prefer V2V-LLM (LLM fusion) when: the primary objective is planning safety (collision rate) or notable object identification accuracy, and the deployment can tolerate a centralized computing node with GPU inference capability. The method achieves the lowest collision rate (3.00% V2V-split, 6.89% V2X-split, Table III) and best Q4 F1 (59.7 V2V-split, 69.3 V2X-split), while adding only 1.5% communication overhead over intermediate fusion (0.4068 vs. 0.4008 MB, Table III). It is especially advantageous when multiple question types (grounding, identification, planning) must be handled by a single model, since the LLM architecture unifies these tasks without task-specific heads. The method also degrades gracefully under moderate communication latency (0.2s: 3.21% CR vs. 3.00% at 0s, Table VI) and positional noise (0.2m STD: 3.10% CR vs. 3.00%, Table VII), making it viable for real-world V2V communication conditions.
Prefer intermediate fusion (AttFuse, V2X-ViT, CoBEVT) when: the primary objective is grounding accuracy (Q1–Q3), especially basic object presence/absence queries (Q1), and the deployment is bandwidth-constrained. CoBEVT achieves the best Q1 F1 on V2V-split (72.2 vs. V2V-LLM's 70.0, Table III), though note the confounding factor: intermediate fusion baselines use only scene-level features while V2V-LLM uses both scene and object features. For pure detection-style tasks where explicit object location priors (provided by V2V-LLM's object-level features) may not be necessary, intermediate fusion provides competitive or superior performance at slightly lower communication cost (0.4008 vs. 0.4068 MB).
Prefer early fusion when: raw point cloud bandwidth is not a constraint (e.g., wired infrastructure-to-vehicle connections or dedicated short-range high-bandwidth links) and the deployment values simplicity — a single merged point cloud processed by one detector is architecturally simpler than coordinating per-CAV feature extraction with LLM fusion. Early fusion achieves the second-best collision rate on V2V-split (3.55% vs. V2V-LLM's 3.00%, Table III) and the best Q1 F1 (73.5), but at 4.7× the communication cost of intermediate or LLM fusion (1.9208 MB), making it impractical for bandwidth-limited V2V wireless links.
Prefer no fusion when: V2V communication is unavailable or unreliable, and the CAV must operate independently. The no-fusion baseline achieves 66.6 Q1 F1, 47.3 Q4 F1, and 4.57% collision rate on V2V-split (Table III) — substantially worse than cooperative methods but still functional as a fallback. The paper does not test a hybrid system that uses V2V-LLM when communication is available and falls back to no-fusion when it is not, but the results suggest such a system would degrade to the no-fusion baseline rather than failing catastrophically.

The modular non-LLM pipeline (CoBEVT + BEV-planner) is not recommended based on the paper's evidence: it achieves 11.59% collision rate versus V2V-LLM's 3.00% on V2V-split (Table V), with worse L2 error (5.82m vs. 4.99m) and lower Q1 F1 (65.7 vs. 70.0). The 3.86× collision rate gap suggests that the standard modular architecture — cooperative detection feeding bounding boxes to a separate planner — is fundamentally suboptimal for cooperative planning, at least with the components tested.

Dependency tree

Papers this one builds on, walking backwards through the same domain and shared topics.

Hunyuan3D 2.0: Scaling Diffusion Models for High Resolution Textured 3D Assets Generation2025· llm-systems← in-domain predecessor
- NVILA: Efficient Frontier Visual Language Models2024· llm-systems← in-domain predecessor

What it influenced

Later papers in the same chain. Direction reversed.

MolmoAct: Action Reasoning Models that can Reason in Space2025· uncategorized→ LLM-judged similar
- SIMA 2: A Generalist Embodied Agent for Virtual Worlds2025· agents→ LLM-judged similar
- GR-Dexter Technical Report2025· agents→ LLM-judged similar
- VLA-0: Building State-of-the-Art VLAs with Zero Modification2025· architecture→ LLM-judged similar
- UI-Ins: Enhancing GUI Grounding with Multi-Perspective Instruction-as-Reasoning2025· reasoning→ LLM-judged similar
- Robix: A Unified Model for Robot Interaction, Reasoning and Planning2025· uncategorized→ LLM-judged similar
- From Spatial to Actions: Grounding Vision-Language-Action Model in Spatial Foundation Priors2025· uncategorized→ LLM-judged similar
- Learning Vision-Driven Reactive Soccer Skills for Humanoid Robots2025· uncategorized→ LLM-judged similar
- RynnVLA-002: A Unified Vision-Language-Action and World Model2025· uncategorized→ LLM-judged similar
- V2V-GoT: Vehicle-to-Vehicle Cooperative Autonomous Driving with Multimodal Large Language Models and Graph-of-Thoughts2025· vision→ LLM-judged similar
- Visual Spatial Tuning2025· vision→ LLM-judged similar
- Vector Prism: Animating Vector Graphics by Stratifying Semantic Structure2025· vision→ LLM-judged similar
- NitroGen: An Open Foundation Model for Generalist Gaming Agents2026· agents→ LLM-judged similar
- VLingNav: Embodied Navigation with Adaptive Reasoning and Visual-Assisted Linguistic Memory2026· agents→ LLM-judged similar
- Fast-ThinkAct: Efficient Vision-Language-Action Reasoning via Verbalizable Latent Planning2026· agents→ LLM-judged similar
- RynnBrain: Open Embodied Foundation Models2026· agents→ LLM-judged similar
- World Guidance: World Modeling in Condition Space for Action Generation2026· training-methods→ LLM-judged similar
- World Action Models are Zero-shot Policies2026· uncategorized→ LLM-judged similar
- Towards Affordance-Aware Robotic Dexterous Grasping with Human-like Priors2025· uncategorized→ in-domain predecessor
- EmbeddingGemma: Powerful and Lightweight Text Representations2025· llm-systems→ topic thread
- SAIL-VL2 Technical Report2025· multimodal→ topic thread
- A Survey of Reinforcement Learning for Large Reasoning Models2025· reasoning→ topic thread
Gemini Embedding: Generalizable Embeddings from Gemini2025· llm-systems→ in-domain predecessor
- A Comprehensive Survey on Long Context Language Modeling2025· llm-systems→ in-domain predecessor
- Gemini Robotics: Bringing AI into the Physical World2025· multimodal→ topic thread
DeepMMSearch-R1: Empowering Multimodal LLMs in Multimodal Web Search2025· llm-systems→ similar topic
- Unify-Agent: A Unified Multimodal Agent for World-Grounded Image Synthesis2026· agents→ LLM-judged similar
- Llama-Embed-Nemotron-8B: A Universal Text Embedding Model for Multilingual and Cross-Lingual Tasks2025· llm-systems→ in-domain predecessor
- MolmoWeb: Open Visual Web Agent and Open Data for the Open Web2026· llm-systems→ similar topic