Poly-Autoregressive Prediction for Modeling Interactions

🎯 Pitch

Standard autoregressive models fail in multi-agent settings because they ignore the simultaneous influence of other agents on an ego agent's future behavior. Poly-autoregressive modeling fixes this by simply letting each agent's prediction see all other agents' current and past states at every timestep, boosting action forecasting mAP by +1.9 on AVA and improving hand-object pose translation by 41% on DexYCB with a tiny 4.4M-parameter transformer.

1. Executive Summary

This paper introduces Poly-Autoregressive (PAR) modeling, a simple framework for predicting an agent's future behavior by conditioning on both its own state history and the past and current states of other interacting agents, in contrast to standard autoregressive (AR) prediction that considers only the ego agent. Across three diverse case studies—social action forecasting on the AVA benchmark, vehicle trajectory prediction on nuScenes, and 6DoF object pose forecasting on DexYCB—PAR consistently outperforms AR using the same proof-of-concept 4.4M-parameter Llama transformer architecture without any architecture modifications, requiring only minimal data pre-processing changes such as tokenization strategy and the inclusion of a learned agent identity embedding and a next-timestep (rather than next-token) prediction objective. PAR achieves a +1.9 absolute mAP gain over AR on AVA action forecasting (+3.5 mAP on two-person interaction classes), relative improvements of 6.3% ADE and 6.4% FDE on nuScenes trajectory prediction, and relative improvements of 8.9% in rotation and 41.0% in translation prediction on DexYCB, establishing that jointly modeling multiple agents through a unified next-timestep prediction framework yields substantial accuracy gains across domains without task-specific architectural changes, even when used with a small-scale transformer backbone.

2. Context and Motivation

The Core Problem: Standard Autoregressive Models Ignore Interaction

The fundamental problem this paper addresses is that standard autoregressive (AR) modeling, as used in language processing, is structurally mismatched for predicting behavior in multi-agent physical interactions. In language, words follow one another in a unidimensional sequence — each token depends only on the tokens that precede it. This makes AR modeling natural: predict the next word given all previous words.

But in physical interactions with multiple agents, this assumption breaks down in two critical ways that the authors articulate in Section 1:

Physical laws constrain interactions, not grammar. Unlike language, which is structured through grammar and semantics, interactions in the physical world are "dictated by both the laws of physics (e.g. how a hand grasps an object) and the internal state of each agent (e.g. the trajectory an agent chooses to move a grasped object in), a latent variable that we know nothing about." The regularities governing behavior are fundamentally different from the regularities governing language.
Multiple agents' states change simultaneously, not sequentially. As the authors note, "in social situations, the history of a single person's past states does not alone determine the dynamics of their future states; we also need to consider the states of other agents." In a conversation, person A's future action depends on person B's current and past actions — information that an AR model conditioned only on person A's history entirely lacks.

This is not a minor edge case. The authors argue that when you apply AR modeling to multi-agent prediction, you are solving an ill-posed problem: the single agent's history is fundamentally insufficient to determine its future, because its behavior is causally influenced by other agents whose states the model never sees. The AR model must effectively guess at interaction dynamics from an impoverished input, leading to predictions that fail to capture turn-taking in conversation, collision avoidance in driving, or coordinated manipulation in hand-object interaction.

Why This Problem Matters

The paper's motivation extends beyond a methodological critique. There are compelling reasons why getting multi-agent prediction right matters:

Ubiquity of multi-agent scenarios. The three case studies — social action forecasting, autonomous driving, and hand-object manipulation — are not niche applications. Human social interaction (conversation, collaborative activity) is a core modality of intelligent behavior. Autonomous vehicles must navigate environments populated with other vehicles, pedestrians, and cyclists whose future positions are interdependent. Robotic manipulation involves hands, objects, tools, and human partners whose motions are coordinated. Improving prediction in any of these domains has direct practical consequences: better driver assistance systems, more socially aware AI assistants, more dexterous robotic manipulation.

The limits of scaling alone. A natural response to the ill-posed single-agent problem might be "train a bigger model on more data, and it will learn to infer agent interactions from the ego agent's trajectory alone." The paper's consistent finding across all three case studies — that even a small 4.4M parameter PAR model outperforms an equivalently-sized AR model, often by substantial margins — suggests that no amount of scaling can fully compensate for missing causal information. The AR model simply does not have access to the other agents' states, and some interaction-driven behaviors become unpredictable without that information. Making interaction-conditioning an explicit architectural choice, rather than hoping the model infers it implicitly, yields gains that scale cannot replace.

A framework for the future rather than a point solution. Beyond the immediate quantitative gains, the paper positions PAR as a unifying framework that can systematize what was previously a fragmented collection of domain-specific solutions. Section 2 (Related Work) reviews prior approaches to multi-agent prediction — for human social interaction, dance, vehicle trajectory, and hand-object pose — and observes that "several prior works addressed modeling specific multi-agent problems via regressive models as one-off case studies." Each domain developed its own architecture, its own tokenization scheme, its own training objective. The PAR framework shows that a single simple approach — flatten tokens from multiple agents, use agent ID embeddings and next-timestep prediction, apply the same transformer backbone — works across all these domains without modification. This unification is valuable not just aesthetically but practically: it means advances in one domain (e.g., better position encodings) can transfer directly to others without re-engineering.

Where Existing Approaches Fall Short

The paper situates itself against a backdrop of prior work that falls short in specific, identifiable ways.

Autoregressive modeling is powerful but single-agent by design. The success of autoregressive models in language (GPT family, Section 2), vision (Image Transformer, Vision Transformer, PixelRNN/CNN), and multi-modal learning (Flamingo, LLaVA) has made next-token prediction the default paradigm for sequence modeling. Recent work even frames humanoid locomotion as an AR next-token prediction task (Radosavovic et al., 2024), using two types of continuous tokens (observations and actions). But all of these approaches share a fundamental structural commitment: the prediction target at position t depends only on inputs at positions < t. In a multi-agent setting, this means either (a) predict only one agent's trajectory (losing interaction information), or (b) flatten multiple agents' tokens into one sequence and predict token-by-token (which makes the model predict agent B's state at time t before agent A's state at time t+1, getting the temporal structure wrong). The latter approach is what the paper tests as the "next-token AR" ablation (Table 1, row 2; Table 8, row 2) and finds it "confuses the model, and the performance is significantly worse than just training on and considering a single agent."

Domain-specific solutions exist but are not unified. The paper's related work (Section 2 and expanded in Appendix Section 9) catalogs a substantial body of prior research on multi-agent prediction, much of it using transformer architectures:

Social interaction: Variational RNNs (Baruah & Banerjee, 2020), interaction transformers for reaction generation (Chopin et al., 2023), multi-person extreme motion prediction (Guo et al., 2022), dance prediction (Maluleke et al., 2024; Siyao et al., 2024), dyadic communication modeling (Ng et al., 2022, 2023, 2024), and diffusion-based human motion generation (Liang et al., 2024; Ghosh et al., 2024).
Vehicle trajectory prediction: A long history including IntentNet (Casas et al., 2018), Trajectron++ (Salzmann et al., 2020), Scene Transformer (Ngiam et al., 2021), AgentFormer (Yuan et al., 2021), and MotionLM (Seff et al., 2023), which is explicitly cited as informing the PAR approach through its use of a transformer decoder that processes multi-agent tokens with learned agent ID embeddings.
6DoF object pose estimation: A separate community using convolutional approaches (PoseCNN, DeepIM, GDR-Net), temporal tracking (PoseRBPF, BundleSDF, FoundationPose), and hand-object reconstruction (Wu et al., 2024).

The critical observation the paper makes is that these are one-off solutions. Each was designed for a specific domain, with domain-specific architectures, tokenization choices, and training objectives. What is missing — and what PAR provides — is a single framework that works across all these domains. The paper positions itself not as claiming architectural novelty over these prior works (it acknowledges that agent ID embeddings, transformer decoders, and next-timestep prediction exist in prior work, particularly MotionLM), but as demonstrating that these simple components, when combined into a unified framework, generalize across radically different prediction tasks without modification.

The single-agent baseline is the implicit default. Perhaps the most important gap the paper identifies is a methodological one: in many interaction domains, researchers default to single-agent AR prediction without questioning whether it is appropriate. The paper makes this baseline explicit in all three case studies (1-agent AR in Tables 1, 3, 5) and shows that it is consistently inferior to PAR. This establishes that multi-agent conditioning should be the default for interaction domains, not an optional enhancement. The fact that the authors had to make this argument — and that single-agent AR is still the standard approach in many subfields — suggests that the community has been systematically under-utilizing available information.

How This Paper Positions Itself

The paper's positioning is carefully limited, which is instructive for understanding what claims it does and does not make.

Not a new architecture. The paper uses an off-the-shelf Llama transformer decoder (8 layers, 8 attention heads, hidden dimension 128, ~4.4M parameters) with rotary positional encodings. There is no novel attention mechanism, no custom layer design, no architectural contribution. The explicit claim is that the base framework is a "proof-of-concept" (Section 1, Section 7) intended to demonstrate the principle, not to achieve state-of-the-art results.

Not a scaling paper. The model is deliberately small (~4.4M parameters), and the datasets are standard benchmarks. The paper explicitly says "we have only provided a starting point that can be built upon extensively" and suggests that future work should "experiment with scaling the data and model" (Section 7). This is a framework paper, not a results paper — the contribution is the unification and the demonstration that a single simple approach works, not the absolute numbers.

A framework-level contribution. The central claim is that three simple design choices — (1) flattening multi-agent tokens into a single sequence, (2) adding learned agent identity embeddings, and (3) training with next-timestep (same-agent) prediction rather than next-token prediction — constitute a general framework that subsumes prior domain-specific approaches. The paper demonstrates this by applying exactly the same architecture, hyperparameters, and training procedure to three tasks that differ in: data modality (discrete action labels vs. continuous xy positions vs. continuous 3D pose), token type (discrete vs. continuous), loss function (cross-entropy vs. MSE vs. geodesic distance), evaluation metric (mAP vs. ADE/FDE vs. MSE/geodesic), and number of agents (2 vs. 3 vs. 2). The only things that change between case studies are the data pre-processing and tokenization — the core PAR logic remains identical.

This is significant because it separates the interaction modeling problem from the domain representation problem. Researchers in new domains need only figure out how to tokenize their agents' states and choose a loss function; the multi-agent reasoning structure is provided by PAR. This contrasts with the status quo, where each new domain required designing both the representation and the interaction mechanism from scratch.

An explicit contrast with language modeling. The paper's framing implicitly but deliberately contrasts PAR with the language modeling paradigm that dominates deep learning. Language models succeed because text has a natural sequential structure — words follow words. PAR is designed for domains where the natural structure is parallel, not sequential — multiple agents' states evolve simultaneously. By introducing the next-timestep prediction objective (predicting agent k at time t+1 given all agents at times ≤t, rather than predicting agent k+1 at time t), PAR respects the true causal structure of multi-agent interactions: the future of any agent depends on the present of all agents, not on the next agent in an arbitrary ordering. This seemingly small change — shifting the prediction target by N tokens rather than by 1 — is the conceptual core of PAR and what distinguishes it from simply applying language modeling to flattened multi-agent sequences.

3. Technical Approach

3.1 Reader Orientation

The Poly-Autoregressive (PAR) framework is a unified transformer-based prediction system that forecasts the future behavior of any agent by looking at the past states of all agents in an interaction — not just the agent being predicted. It solves the problem that standard autoregressive models are structurally blind to simultaneous multi-agent interactions, by restructuring how tokens from multiple agents are sequenced and how the prediction target is defined, so that the model learns causal relationships between agents rather than treating each agent's trajectory as an independent sequence.

3.2 Big-Picture Architecture (Diagram in Words)

The PAR system has five major components that operate in a pipeline:

Data Collection and Entity Extraction: Raw video data is processed to extract per-agent states at each timestep — for example, action class vectors from AVA keypoint annotations, xy position coordinates from nuScenes tracking data, or 6DoF object poses and 3D hand translations from DexYCB labels. Each agent is assigned a unique integer identity that persists across frames. The output is a set of $N$ temporal trajectories, each of length $T$ , where each trajectory is a sequence of state vectors for one agent.
Tokenization Module: Each agent's state vector at each timestep is converted into a token format suitable for the transformer. For discrete modalities (e.g., action classes, discretized velocity/acceleration bins), the state is mapped to an integer index into a codebook. For continuous modalities (e.g., 3D translation vectors, quaternions), the raw vector is kept as a continuous token. The critical structural decision is that tokens from all $N$ agents at timestep $t$ are grouped together — so the full input is a flattened sequence of $N \times T$ tokens, ordered as $(t_1, \text{agent}_1), (t_1, \text{agent}_2), \dots, (t_1, \text{agent}_N), (t_2, \text{agent}_1), \dots$ rather than one agent's full trajectory followed by another's.
Embedding Summation: Each token, whether discrete or continuous, is projected to a shared hidden dimension $d_h = 128$ . A learned agent identity embedding (an integer ID mapped to a 128-dimensional vector) is summed with the token embedding, so the transformer knows which agent each token belongs to. Optional additional embeddings — such as a sine-cosine location positional encoding (LPE) encoding relative spatial positions — are also summed in at this stage.
Transformer Decoder (Llama Architecture): The sum of embeddings is fed into a standard decoder-only transformer with 8 layers, 8 attention heads, and hidden/intermediate dimension 128, totaling approximately 4.4 million learned parameters. Rotary positional encodings (RoPE) are applied to the token sequence positions. The model is trained with teacher forcing: the full ground-truth sequence is provided as input (with causal masking so each position can only attend to earlier positions in the flattened sequence), and the model produces a prediction for each position.
Next-Timestep Prediction Head: The transformer's output at each position is processed according to the token type. For discrete tokens, the hidden state is projected to a distribution over the codebook vocabulary via a learned embedding layer (treated as output logits), and the token with maximum probability is selected (argmax). For continuous tokens, the hidden state is passed through a learned de-projection (un-projection) layer that maps from $d_h$ back to the original token dimension $d$ . The output is the predicted state for the same agent at the next timestep — not the next token in the flattened sequence.

Information flows as follows: raw video → per-agent state extraction → tokenization → embedding summation (token + agent ID + optional LPE) → transformer decoder (with causal masking and teacher forcing) → next-timestep prediction head → decoded prediction (continuous) or argmax token selection (discrete). At inference, the model generates ego-agent futures autoregressively by feeding its own predictions back as input, while the other agents' ground-truth observations up to the current timestep are provided in parallel.

3.3 Roadmap for the Deep Dive

First, the formal problem definition and the training objective (Equations 1 and 2), which establish exactly what PAR predicts, what information it conditions on, and how the loss is computed — this defines the "shape" of the solution.
Second, the critical distinction between next-token prediction (standard AR applied to multi-agent sequences) and next-timestep prediction (PAR's innovation), including a walk-through of why the standard approach produces the wrong temporal causal structure and how the PAR objective fixes it.
Third, the three architectural components that make PAR work: learned agent identity embeddings, joint training across all agents, and the inference procedure that feeds other agents' observed states while auto-regressively generating the ego agent's future.
Fourth, the tokenization and loss function interface — how discrete and continuous tokens are handled through embedding/projection layers, and how the loss is chosen based on token type and task requirements, establishing the framework's domain-agnostic flexibility.
Fifth, the specific implementation details held constant across all case studies: architecture dimensions, optimizer settings, EMA decay rates, and inference mechanics — the fixed backbone that demonstrates the framework's generality.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a framework paper whose core idea is that multi-agent interaction prediction can be unified under a single autoregressive-like paradigm by (a) flattening tokens from all agents into a time-ordered sequence, (b) adding agent identity information so the model knows which token belongs to which agent, and (c) shifting the prediction target from "next token in the flattened sequence" to "next timestep for the same agent," thereby respecting the true causal structure of simultaneous multi-agent dynamics.

The Problem Definition and Training Objective

The paper formalizes the prediction task in Section 3.1. Let there be $N$ interacting agents. Let $S = \{s_i\}_{i=1}^T$ be a temporal sequence of agent states, where $s_i$ represents the collection of all agents' states at timestep $i$ . Let $S^N$ denote the temporal sequence of states for the $N$ -th agent (the ego agent — the one whose behavior we aim to predict), and $S^{1:N-1}$ denote the temporal sequences for all other $N-1$ agents.

For prediction, we observe all agents' states up to some time $t_\pi - 1$ (the history), where $t_\pi \in [1, T]$ is the timestep at which we begin forecasting. The prediction target is the ego agent's future states from $t_\pi$ to $T$ . The predictor $P$ generates these future states one timestep at a time, conditioning on all available information:

$\hat{s}_t^N = P\left(S_{1:t-1}^{1:N-1}, S_{1:t_\pi}^N, \hat{S}_{t_\pi+1:t-1}^N\right)$

where:

$S_{1:t-1}^{1:N-1}$ is the observed past states of all non-ego agents up to the timestep just before the current prediction (timesteps 1 through $t-1$ ),
$S_{1:t_\pi}^N$ is the observed past states of the ego agent (timesteps 1 through $t_\pi$ — the full history),
$\hat{S}_{t_\pi+1:t-1}^N$ is any previously predicted states of the ego agent (from timestep $t_\pi+1$ up to $t-1$ , if we have already predicted some future steps),
$\hat{s}_t^N$ is the predicted state of the ego agent at the current timestep $t$ .

What it computes: The predictor takes all other agents' observed or ground-truth state histories (which are always available because we observe them), the ego agent's observed history (up to $t_\pi$ ), and any of the ego agent's previously predicted future states, and produces a single-timestep forecast for the ego agent. At the first prediction step ( $t = t_\pi$ ), there are no previously predicted ego states, so only the observed histories are used. At subsequent steps ( $t > t_\pi$ ), the model autoregressively feeds in its own previous predictions for the ego agent, while continuing to receive the ground-truth observations of all other agents.

Why this form: The key design choice is that other agents' states are always provided as ground truth, never autoregressively predicted. This is because the goal is to predict the ego agent's behavior given the actual observed behavior of interacting partners — we are not trying to jointly predict everyone's future, which would be a harder and different problem (multi-agent forecasting rather than ego-conditioned prediction). The ego agent's future is the only unknown; the other agents' histories and current states up to $t-1$ are treated as known causal inputs. This reflects realistic deployment scenarios: an autonomous vehicle knows its own planned trajectory but must react to other vehicles' observed movements; a social AI assistant knows its own intended actions but must predict a human's response based on the human's observed behavior.

The distribution the predictor learns to model is:

$p\left(\hat{s}_t^N \mid S_{1:t-1}^{1:N-1}, S_{1:t-1}^N\right)$

where the conditioning set includes all agents' histories up to $t-1$ (both ego and non-ego). Note that at prediction time, $S_{t_\pi+1:t-1}^N$ for $t > t_\pi$ is the predicted ego history, not the ground truth — the model learns to condition on its own potentially imperfect previous predictions during training as well (via teacher forcing, where ground-truth ego states are used as input during training, but the model's predictions are evaluated against ground truth).

Training objective: The model is trained by maximizing the likelihood of the ground-truth ego state at time $t$ . The loss is:

$\mathcal{L}_P = \mathbb{E}_{y \sim p(y)}\left[-\log\left(p(s_t^N)\right)\right]$

where $y$ is the target state at timestep $t$ , sampled from the ground-truth data distribution $p(y)$ , and $p(s_t^N)$ is the model's predicted distribution (or value) for that state. The specific form of $-\log(p(s_t^N))$ depends on whether tokens are discrete (cross-entropy) or continuous (regression loss) — see the tokenization and loss section below.

Why this form: This is a standard maximum-likelihood objective. For discrete tokens, it becomes cross-entropy classification; for continuous tokens, the negative log-likelihood under a Gaussian assumption becomes mean squared error (or a custom loss like geodesic distance for rotation). The key is that the loss is computed on the ego agent's state only, not jointly on all agents. However, during training, the paper performs joint training — see the section on that below — which means the loss is computed for all agents but evaluated independently per agent.

The Critical Innovation: Next-Timestep Prediction vs. Next-Token Prediction

This is the conceptual core of PAR, and understanding it requires carefully working through what happens when you naively apply AR to a multi-agent flattened sequence.

The flattened sequence structure. Consider $N=3$ agents (A, B, C) observed over $T$ timesteps. PAR constructs a flattened sequence of $N \times T$ tokens, ordered as:

$[\text{A}_{t=1}, \text{B}_{t=1}, \text{C}_{t=1}, \text{A}_{t=2}, \text{B}_{t=2}, \text{C}_{t=2}, \dots, \text{A}_{t=T}, \text{B}_{t=T}, \text{C}_{t=T}]$

Each token is the state of one agent at one timestep. The ordering within each timestep is arbitrary but consistent across timesteps.

What standard next-token prediction would do. If this flattened sequence were fed into a standard autoregressive transformer trained with next-token prediction (as in language modeling), the model would learn to predict token $i+1$ given tokens $1$ through $i$ . This means:

At position corresponding to $\text{A}_{t=1}$ , predict $\text{B}_{t=1}$ (agent B's state at the same timestep).
At position corresponding to $\text{B}_{t=1}$ , predict $\text{C}_{t=1}$ (agent C's state at the same timestep).
At position corresponding to $\text{C}_{t=1}$ , predict $\text{A}_{t=2}$ (agent A's state at the next timestep).

This creates a mixed prediction target: sometimes the model predicts another agent's state at the current timestep, sometimes it predicts the next timestep's state. The causal structure this learns is: "given agent A's state, what is agent B's state?" followed by "given A and B, what is C?" followed by "given A, B, and C at time 1, what is A at time 2?" This conflates two distinct types of relationships — cross-agent correlations at the same timestep and temporal dynamics of individual agents — into a single prediction task.

The paper explicitly tests this ablation (Table 1, row 2 for AVA; Table 8, row 2 for nuScenes) and finds that multi-agent next-token prediction performs worse than single-agent AR (38.0 mAP vs. 40.7 mAP on AVA). The authors state: "We see that this approach confuses the model, and the performance is significantly worse than just training on and considering a single agent." The model is being asked to learn an unnatural mapping: from agent A to agent B at the same timestep, which has no causal direction in truly simultaneous interactions.

What PAR's next-timestep prediction does instead. PAR shifts the prediction target: instead of predicting token $i+1$ in the flattened sequence, it predicts the token corresponding to the same agent at the next timestep. This means:

For training, when the model processes token $\text{A}_{t=1}$ , its target is $\text{A}_{t=2}$ (agent A at time 2).
When processing $\text{B}_{t=1}$ , target is $\text{B}_{t=2}$ .
When processing $\text{C}_{t=1}$ , target is $\text{C}_{t=2}$ .

Concretely, for a flattened sequence of $N \times T$ tokens, the target is shifted forward by $N$ positions rather than by 1. Every token at timestep $t$ predicts the same agent's token at timestep $t+1$ . This is what the paper calls "same-agent next-timestep prediction" (Section 3.2, Figure 3b).

Why this is correct. In multi-agent interactions, the causal arrow points from all agents at time $t$ to each agent at time $t+1$ — not from agent A at time $t$ to agent B at time $t$ . Two agents' states at the same timestep are correlated but not causally ordered (at the granularity of the sampling frequency, they are simultaneous). The next-timestep objective respects this: to predict agent A at time $t+1$ , the model can attend to agents A, B, and C at times $1, 2, \dots, t$ — all information that causally precedes the target. The cross-agent correlations at the same timestep are learned implicitly through the attention mechanism (which can attend to all agents at the previous timesteps), but they are never the explicit prediction target.

What information is available at each prediction. When predicting agent A at time $t+1$ , the model has access to:

All agents at times $1, 2, \dots, t$ (the full interaction history).
As input at the current position, agent A's state at time $t$ (and any other agents at time $t$ that appear before A in the sequence — but this is an artifact of the flattened ordering and is not the prediction target).

Critically, the model does not get to see agent B at time $t+1$ before predicting A at time $t+1$ — the temporal ordering is preserved. The causal mask prevents attending to any token at timestep $t+1$ when making predictions for timestep $t+1$ .

Implementation detail: teacher forcing shift. In standard AR, teacher forcing is implemented by shifting the input sequence right by one and using the original sequence as the target. In PAR, the shift is by $N$ positions. During training, the input to the transformer is the first $N \times t$ tokens (all agents up to timestep $t$ ), and the target at each output position is the same agent at timestep $t+1$ (which is $N$ positions ahead in the flattened sequence). The model is trained to predict its own shifted target — the loss is computed only on tokens that have a valid future timestep.

The Three Architectural Components of PAR

The PAR framework is constituted by three design choices that work together. These are not independent ablations but mutually reinforcing components that jointly define the framework.

Component 1: Learned Agent Identity Embedding

Each agent is assigned an integer identifier (0, 1, 2, ...) that is consistent across all timesteps. During token embedding, this integer is mapped to a learned embedding vector of dimension $d_h = 128$ and summed with the token's own embedding (whether discrete or continuous projection). The result is a single 128-dimensional vector that encodes both what the agent is doing (the token content) and who the agent is (the identity).

Why summing and not concatenation or separate tokens? Summing forces the transformer to disentangle agent identity from agent state within the same representational space, which is parameter-efficient. The alternative — concatenating the embeddings — would double the effective dimension and require the transformer to learn to separate the two signals in its first attention layer. Summing relies on the additive structure of the embedding space: the transformer can learn that certain directions in the 128-dimensional space correspond to agent identity while others correspond to state content.

Why a learned embedding rather than a fixed positional encoding for agent identity? A sine-cosine encoding based on agent ID could encode identity without learnable parameters, but learned embeddings allow the model to discover which identity dimensions are relevant for each prediction task. For instance, in the DexYCB hand-object case study, the hand and object have fundamentally different state spaces (3D translation vs. quaternion), and a learned embedding can encode this difference in a way that interacts usefully with the token content.

The paper demonstrates the importance of the agent ID embedding through ablation. On AVA action forecasting (Table 1), removing the agent ID embedding (row 4: "2-agent PAR* with next-timestep prediction but no agent ID") drops performance from 42.6 mAP to 40.0 mAP — a 2.6 point drop. On DexYCB (Table 7), removing the agent ID embedding degrades rotation prediction (0.837 vs. 0.895 GEO) more than translation prediction (2.17 vs. 2.26 × $10^{-3}$ MSE). The authors hypothesize this is because in rotation prediction, the hand token is a 3D translation while the object token is a quaternion — the agent ID embedding helps the model distinguish these different modalities.

Component 2: Joint Training Across All Agents

During training, PAR computes the loss not only for the ego agent but for all $N$ agents. This means:

For each timestep $t$ (where $t < T$ ), the model predicts each agent's state at $t+1$ .
The loss is summed or averaged across all agents and all timesteps.
At inference time, only the ego agent's predictions are generated and evaluated.

Why train on all agents if we only predict one at inference? This is a form of multi-task learning that forces the model to build useful representations of all agents' dynamics, even those it will not predict at test time. By learning to predict agent B's future states from the same shared transformer, the model develops an understanding of agent B's behavioral patterns, which then informs its predictions of agent A (the ego agent) through the shared attention and feedforward layers. All agents contribute gradients that update the same parameters, so the model cannot specialize to one agent's dynamics at the expense of others — it must learn a general interaction model.

The alternative — training only on the ego agent's predictions — would ignore the training signal available from the other agents' ground-truth futures. Since the other agents' states are available in the training data, there is no reason not to use them as auxiliary supervision.

Implementation detail for joint training. The loss computation is straightforward: at each output position that corresponds to an agent at time $t+1$ (i.e., all tokens in the flattened sequence except those at the final timestep), the model's prediction is compared to the ground-truth state for that agent at time $t+1$ . The loss is computed per-agent and per-timestep, then averaged. The model does not treat the ego agent differently during training — all agents are predicted symmetrically.

Component 3: Inference Procedure

At inference, the ego agent's future is generated autoregressively, while other agents' states are provided as ground truth up to the current timestep. The procedure for generating $F$ future timesteps (from $t_\pi$ to $t_\pi + F$ ) is:

Initialize: Construct the input sequence with all agents' observed states for timesteps $1$ through $t_\pi$ (the full history). This is $N \times t_\pi$ tokens.
Predict step 1: Feed this sequence through the transformer. The output at the final position (which corresponds to the ego agent at timestep $t_\pi$ , since the ego agent is placed last in the per-timestep ordering) is the predicted ego state at $t_\pi+1$ . The other output positions (predicting non-ego agents) are ignored.
Update context for step 2: To predict timestep $t_\pi+2$ , we need tokens for all agents at $t_\pi+1$ . For the non-ego agents ( $1$ through $N-1$ ), we use their ground-truth observed states at $t_\pi+1$ (which are known because we only predict the ego agent). For the ego agent ( $N$ ), we use the predicted state from step 2. Append these $N$ tokens to the input sequence, producing an $(N \times (t_\pi+1))$ -token sequence.
Predict and iterate: Feed the updated sequence through the transformer. The output at the position corresponding to the ego agent at $t_\pi+1$ predicts the ego state at $t_\pi+2$ . Repeat the update-predict cycle until all $F$ future timesteps are generated.

Why this mixed ground-truth/predicted context? The ego agent's own predictions are fed back autoregressively (standard for any autoregressive model — the model must condition on its own outputs to generate coherent sequences). The other agents' ground-truth states are used because they are observable at inference time (in the applications considered, an autonomous vehicle can see other cars' current positions; a social AI can observe the other person's current action). This avoids the compounding error that would occur if the model had to predict all agents' futures jointly — predicting $N$ interacting agents autoregressively is far harder than predicting one agent conditioned on observed others.

Contrast with next-token AR inference. In the next-token AR ablation, inference works differently: to predict the ego agent at $t_\pi+1$ , the model is fed all agents up to $t_\pi$ plus the first $N-1$ non-ego agents at $t_\pi$ (whose ground-truth tokens are available). This actually gives the model more information than PAR at inference time, because it sees the non-ego agents at the current timestep before predicting the ego. The fact that PAR still outperforms this ablation despite having less immediate context reinforces that the training objective (next-timestep vs. next-token) matters more than the inference-time information advantage.

Tokenization and Loss Function Interface

A central design principle of PAR is that the framework separates the interaction modeling logic from the domain-specific representation. The tokenization and loss modules are the only components that change between case studies. This section describes the general interface; specific instantiations for each case study appear in Sections 4–6 of the paper.

Discrete Tokens

For discrete tokens, each agent's state at each timestep is mapped to an integer index into a vocabulary. The mapping can be:

Direct: the state is itself categorical (e.g., an action class index from a set of 60 action classes on AVA).
Quantized: a continuous value is discretized by binning (e.g., velocity or acceleration values on nuScenes are discretized into velocity bins spanning $[-18, 18]$ meters/second, with 128 bins per dimension; the Cartesian product of x-bin and y-bin gives a vocabulary of size $128 \times 128 = 16{,}384$ for velocity tokens, or 13 acceleration bins per dimension giving a vocabulary of $13 \times 13 = 169$ for acceleration tokens).

The integer token is projected to the hidden dimension $d_h = 128$ using a standard learned embedding layer (the same mechanism used for word embeddings in language models). This is the nn.Embedding layer in PyTorch: a lookup table of size vocab_size × 128. The embedding vectors are randomly initialized and trained.

After the transformer processes the sequence, the hidden state at each output position is projected back to a distribution over the vocabulary. For discrete tokens, this projection is the transpose of the embedding matrix (or a separate linear layer): the d_h-dimensional hidden state is multiplied by a weight matrix of shape d_h × vocab_size, producing logits over the vocabulary. The token with the highest logit is selected at inference time (argmax sampling, though the framework supports stochastic sampling as well).

Loss for discrete tokens: Cross-entropy classification.

$\mathcal{L}_{\text{CE}} = \mathbb{E}_{y \sim p(y)}\left[-\log(p(s_t^N))\right] = -\sum_{c=1}^{C} y_c \log(\hat{y}_c)$

where $C$ is the vocabulary size, $y_c$ is the one-hot ground-truth token (1 for the correct class, 0 otherwise), and $\hat{y}_c$ is the softmax probability assigned to class $c$ by the model.

What it computes: The negative log-probability assigned to the correct token, summed over all vocabulary entries. Since $y_c$ is one-hot, only the correct class contributes to the sum, giving $-\log(\hat{y}_{c^*})$ where $c^*$ is the true token index. The expectation is taken over the training data distribution.

Why this form: Cross-entropy is the standard maximum-likelihood objective for categorical distributions. It encourages the model to assign high probability to the correct token while pushing probability mass away from incorrect tokens. For discrete action or motion tokens, this is the natural choice — the model is performing classification over a finite set of possible next states.

Continuous Tokens

For continuous tokens, each agent's state is a real-valued vector of dimension $d$ (e.g., $d=3$ for a 3D translation, $d=4$ for a quaternion rotation). This vector is projected to the hidden dimension $d_h = 128$ using a learned linear projection layer (a matrix of shape $d \times 128$ ). No discretization occurs — the raw continuous state is the token.

After the transformer processes the sequence, the hidden state at each output position is de-projected (un-projected) back to the original dimension $d$ using a second learned linear layer of shape $128 \times d$ . The de-projected vector is the model's prediction for the continuous state.

Loss for continuous tokens: The loss is a regression objective in the original token space. The specific form depends on the state modality:

Translation (3D Euclidean coordinates): Mean squared error.

$\mathcal{L}_{\text{trans}} = \frac{1}{n} \sum_{i=1}^{n} (\mathbf{t}_i - \hat{\mathbf{t}}_i)^2$

where $\mathbf{t} \in \mathbb{R}^3$ is the ground-truth translation vector and $\hat{\mathbf{t}} \in \mathbb{R}^3$ is the predicted translation.

What it computes: The average squared Euclidean distance between predicted and ground-truth 3D positions, summed over all $n$ prediction instances.

Why this form: MSE is the negative log-likelihood under a Gaussian error model with fixed variance. For 3D translation, it penalizes large errors quadratically, which is appropriate for physical positions where being off by a small amount is acceptable but large deviations are catastrophic.

Rotation (SO(3) quaternions): Geodesic distance on the unit sphere.

$\mathcal{L}_{\text{rot}} = 1 - |\hat{\mathbf{q}} \cdot \mathbf{q}|$

where $\mathbf{q} \in \mathbb{S}^3$ is the ground-truth unit quaternion, $\hat{\mathbf{q}} \in \mathbb{S}^3$ is the predicted unit quaternion, and $\cdot$ denotes the dot product. The absolute value handles the double cover of SO(3) by quaternions (both $\mathbf{q}$ and $-\mathbf{q}$ represent the same rotation).

What it computes: One minus the absolute cosine of the angle between the predicted and ground-truth quaternions. When the quaternions are identical (up to sign), the dot product is $\pm 1$ , so the loss is 0. When they are orthogonal (representing maximally different rotations), the loss is 1.

Why this form: Direct MSE on quaternion components is inappropriate because the quaternion space has a Riemannian structure — Euclidean distance in $\mathbb{R}^4$ does not correspond to geodesic distance on the rotation manifold. The dot-product loss is proportional to the squared chordal distance on the 3-sphere and is standard in rotation regression tasks. It is differentiable, bounded in $[0, 1]$ , and handles the quaternion sign ambiguity correctly.

Action vectors (multi-label binary vectors): The paper uses MSE on the raw 60-dimensional action vector for AVA (Section 4.1), rather than binary cross-entropy. The outputs are not required to be between 0 and 1 explicitly — the model outputs raw scalar values for each of the 60 action classes, and the loss compares these to the ground-truth binary labels (where 1 indicates the action is occurring and 0 indicates it is not). This is a design choice that simplifies implementation at the cost of the outputs not being interpretable as probabilities.

Why MSE for action vectors and not binary cross-entropy? The paper does not justify this choice explicitly. Binary cross-entropy would be the standard loss for multi-label classification and would produce calibrated probability estimates. MSE without sigmoid constrains the model to output real values rather than probabilities, and the evaluation metric (mAP) is based on ranking, which is not sensitive to whether the outputs are calibrated probabilities. The paper may have chosen MSE for implementation simplicity, since it uses MSE for other continuous prediction tasks and wanted to maintain a consistent loss interface.

Tokenization flexibility as a framework feature. A key strength of PAR is that the same architecture can handle both discrete and continuous tokens, mixed within the same prediction task. In the DexYCB case study, the hand token is a 3D continuous vector (translation), while the object token is either a 3D continuous vector (translation) or a 4D continuous vector (quaternion rotation). The agent ID embedding enables the model to process these different token types through the same transformer by distinguishing which positions correspond to which modality.

Framework Implementation Details Held Constant Across Case Studies

The paper emphasizes that the following details are identical for all three case studies (Section 3.4), demonstrating that PAR's performance does not depend on case-study-specific hyperparameter tuning or architecture modifications.

Architecture: Llama transformer decoder (Touvron et al., 2023, base architecture) with:

8 transformer layers (blocks).
8 attention heads per layer.
Hidden dimension $d_h = 128$ .
Intermediate (feedforward) dimension $d_{\text{ff}} = 128$ (since the paper states "hidden and intermediate dimension of 128," implying they are equal — standard Llama uses a 4× expansion, but at this tiny scale, equal dimensions keep the parameter count minimal).
Rotary positional encoding (RoPE, Su et al., 2024) applied to the token sequence positions. This is a standard relative position encoding that encodes position information as a rotation of the query and key vectors in the attention computation, enabling the model to naturally handle varying sequence lengths.
Total parameters: approximately 4.4 million learned parameters, not including learned embedding layers which add "a few thousand more" (the embedding layers are small: for discrete tokens with vocabulary size $V$ , the embedding adds $V \times 128$ parameters; for continuous tokens, the projection adds $d \times 128$ parameters).

Training: Teacher forcing with the next-timestep shift (by $N$ positions, not by 1). The model sees the full ground-truth sequence as input (with causal masking so each position can only attend to earlier positions) and is trained to predict each agent at the next timestep. The learning rate is the only hyperparameter that changes between case studies; all other optimizer settings are held constant across tasks (specific learning rates are reported in each case study's experimental setup — Section 4.1, 5.1, 6.1, and Appendix).

Embedding summation: All embeddings are summed (not concatenated) before being fed to the transformer. This includes:

Token embedding (discrete embedding lookup or continuous linear projection).
Agent identity embedding (learned, integer agent ID → 128-dim vector).
Optional location positional encoding (LPE, sine-cosine encoding applied to relative xy positions).

The paper emphasizes the summation because it is the simplest way to combine heterogeneous information — rather than designing a complex fusion mechanism, additive combination in a shared 128-dimensional space is sufficient, reflecting the transformer's ability to disentangle additive signals through its attention mechanism.

Exponential Moving Average (EMA): Applied during training to stabilize learning, with a decay rate of 0.999 for action prediction and object pose estimation, and 0.9999 for car trajectory prediction. EMA maintains a shadow copy of the model parameters that is an exponentially weighted average of the parameters over training steps. The averaged parameters are used for evaluation, which reduces noise from SGD and typically improves generalization. The higher decay (0.9999) for cars reflects the smaller, potentially noisier dataset or the discrete token space requiring more averaging.

No other bells and whistles: The paper explicitly avoids any additional architectural features — no cross-attention, no specialized interaction modules, no graph neural networks, no separate encoders for different modalities. The entire interaction modeling is handled by the standard self-attention mechanism in the transformer, which can attend to any token in the history regardless of agent identity. The agent ID embedding provides the only signal for distinguishing agents; the attention patterns must learn to route information between agents implicitly.

Design Tradeoffs and Non-Obvious Choices

Why a decoder-only transformer rather than an encoder-decoder? The paper uses a standard autoregressive decoder (causal masking, predicts future tokens from past tokens) rather than an encoder-decoder architecture where the history would be encoded bidirectionally and the decoder would cross-attend to the encoder outputs. The decoder-only design is simpler, aligns with the language modeling paradigm that the paper draws from, and naturally supports autoregressive generation at inference time. The cost is that tokens cannot attend to future tokens even within the history (which would be possible with a bidirectional encoder), but this is a standard tradeoff in autoregressive modeling.

Why such a small model (~4.4M parameters)? The paper explicitly calls the architecture a "proof-of-concept." The goal is not to achieve state-of-the-art results but to demonstrate that PAR's gains over AR are robust and not dependent on large-scale training. Using a tiny model makes the comparison fair (both PAR and AR use the same architecture) and computationally inexpensive (enabling the three-case-study sweep). The authors explicitly state that future work should scale the model and data.

Why sum embeddings rather than concatenate or use separate streams? Summation is the most parameter-efficient approach — it requires no additional parameters beyond the embedding vectors themselves, and the hidden dimension stays fixed regardless of how many auxiliary embeddings are added. The transformer must learn to disentangle the contributions of token content, agent identity, and position from the summed vector, which is feasible because these signals operate on different aspects of the representation (what the agent is doing vs. who the agent is vs. where the agent is). Concatenation would increase the effective dimension and require the transformer to allocate attention head capacity to separating the signals; separate token streams (e.g., one token per agent at each timestep rather than one unified token) would require a more complex architecture.

Why expose all other agents' ground-truth states at inference? This is a deliberate restriction of the problem scope: PAR solves ego-conditioned prediction, not joint multi-agent forecasting. The assumption is that in practical applications, other agents' current states are observable (cameras, sensors) but their futures are not. Predicting all agents' futures jointly would be a much harder problem with compounding errors, and is not what the paper aims to solve. The restriction makes PAR directly comparable to standard single-agent AR (which sees even less information) while remaining practically deployable.

Why the specific flattened ordering (agent 1, agent 2, ..., agent N within each timestep)? The paper does not ablate this choice, but any consistent ordering across timesteps would be mathematically equivalent from the transformer's perspective (since position is encoded by RoPE, not by absolute sequence index). The choice of putting the ego agent last in the per-timestep ordering is an implementation convenience: it means that when the model predicts the last token in each timestep group, that token is always the ego agent at the next timestep. A different ordering would require extracting the ego agent's prediction from a different position in the output sequence but would not change the learning dynamics.

4. Key Insights and Innovations

Innovation 1: Next-Timestep Prediction as a Causal Corrective to the Language Modeling Default

The paper's most fundamental conceptual move is identifying that applying standard autoregressive next-token prediction to a flattened multi-agent sequence produces a structurally wrong causal model. This is not an implementation detail or a minor tweak—it's a diagnosis of a category error that the field has been implicitly making when it ports language modeling techniques to multi-agent domains.

What the field did before. The dominant paradigm—reinforced by the success of GPT-style models—is that any temporal sequence can be treated as a language modeling problem: flatten everything into a one-dimensional token stream and train to predict the next token. This works for text because text genuinely has a unidimensional causal structure: word N causes word N+1 (through the writer's process), so next-token prediction is next-event prediction. But in multi-agent interactions, the next token in a flattened sequence is often another agent's state at the same timestep—not the next event in time. The field had not recognized that this mapping conflates two fundamentally different types of relationships (cross-agent correlation at fixed time vs. forward temporal dynamics) into a single prediction target.

Why PAR's reframing is diagnostically important. The paper's core insight is that next-token prediction, applied to flattened multi-agent sequences, asks the model to predict agent B's state at time t from agent A's state at time t when those states are simultaneous—there is no causal arrow from A to B within the same timestep at the sampling frequency. The model is being asked to learn a causal relationship that doesn't exist. The authors don't just assert this; they prove it empirically through the next-token AR ablation (Table 1, row 2; Table 8, row 2), where multi-agent next-token prediction performs worse than single-agent AR—the interaction information is actually hurting because it's structured in a way that misleads the learner.

The PAR solution—shift the prediction target forward by N tokens rather than by 1, so every prediction is same-agent next-timestep—is simple to implement but conceptually transformative. It reframes multi-agent prediction from "predict the next piece of information in an arbitrary flattened ordering" to "predict the next temporal state of each agent, given all agents' current states." This respects the true causal graph: all agents at time t jointly cause each agent at time t+1, and no agent at time t causes another agent at time t (at the available temporal resolution).

Why this matters beyond this paper. This insight generalizes to any domain where multiple interacting entities evolve in parallel—multi-robot coordination, economic agent-based models, cellular dynamics in biology, traffic simulation. The paper provides a diagnostic test: if you flatten multiple simultaneous streams and train next-token prediction, check whether the prediction targets respect temporal priority. If not, you may be training on a causally incoherent task. This is a fundamental reframing, not an incremental improvement, because it changes the correctness criterion for sequence modeling objectives in parallel dynamical systems.

Evidence anchor. Table 1 row 2 (next-token AR: 38.0 mAP vs. single-agent AR: 40.7 mAP) demonstrates the failure mode; Table 1 row 5 (PAR with next-timestep prediction: 42.6 mAP) demonstrates that fixing the causal structure recovers and exceeds single-agent performance. The 4.6 mAP gap between next-token AR and PAR on the same architecture with the same input information is solely attributable to the training objective—a clean causal identification.

Innovation 2: Agent Identity as a Learned Additive Signal Rather Than a Separate Input Stream

The paper's second conceptual contribution is its approach to multi-agent input representation: agent identity is treated as a summed embedding, not a separate token, a separate encoder, or a dedicated interaction module. This may sound like a minor architectural detail, but it represents a philosophical stance on how interaction modeling should work—and why simpler is better.

What the field did before. Prior multi-agent prediction systems (MotionLM, Scene Transformer, AgentFormer, Interaction Transformer) generally fell into one of two camps. Some used dedicated architectures with separate processing streams for different agents (separate encoders, cross-attention between agent tracks, graph neural networks over agent nodes). Others concatenated agent features into a single input but used specialized interaction modules—agent-aware attention, pairwise interaction features, or explicit relational reasoning layers. In both cases, the modeling of interactions was architecturally privileged: the interaction mechanism was designed into the network structure, reflecting a belief that agent-agent relationships require specialized computational pathways distinct from sequential temporal modeling.

PAR's counter-proposal. The paper argues—implicitly but clearly—that standard self-attention plus learned agent ID embeddings is sufficient. The transformer's native attention mechanism, which can attend to any token in the history regardless of agent identity, already provides the necessary routing of information between agents. The agent ID embedding, summed directly to the token embedding, gives the attention mechanism the only additional signal it needs: "this token came from agent 3, not agent 7." Everything else—learning that agent 3's braking causes agent 7's deceleration, that agent 1's talking correlates with agent 2's listening—is handled by the attention weights, which are the same attention mechanism used for temporal dependencies.

The key insight here is that interaction modeling is just attention with identity tags. There is no fundamental architectural distinction between "attending across time within one agent's trajectory" and "attending across agents at the same time." Both are subsets of the same computation: query-key attention over a set of tokens. The agent ID embedding biases the keys so that queries can selectively attend to specific agents, but the mechanism is identical. This collapses a distinction that prior work had treated as architecturally fundamental.

Why this is a conceptual contribution, not just an architectural choice. The paper is making a claim about representational sufficiency: you don't need graph neural networks, pairwise interaction features, agent-specific encoders, or dedicated cross-attention streams to model multi-agent dynamics. The same transformer that models temporal dependencies can model inter-agent dependencies, with the only augmentation being an additive identity tag. This is a vote of confidence in the universality of the attention mechanism and a methodological simplification: researchers in new domains no longer need to design interaction-specific architectures; they just need to provide agent IDs.

The significance is amplified by the framework's demonstrated generality across three radically different domains (discrete actions, continuous vehicle trajectories, continuous 3D pose). If specialized interaction modules were necessary, each domain would likely require a different module design—yet the same 4.4M parameter decoder-only transformer, with only the summation of agent ID embeddings as the interaction mechanism, works across all three.

Evidence anchor. Table 1 ablation (row 4 vs. row 5) isolates the agent ID embedding contribution: removing it drops performance from 42.6 to 40.0 mAP on AVA. Table 7 ablation similarly shows that removing the agent ID embedding degrades rotation prediction (0.837 vs. 0.895 GEO) more severely than translation, suggesting the embedding helps the model distinguish different token modalities (hand = 3D translation, object = quaternion) as well as agent identity. The key point is that the agent ID embedding is doing real work, but the mechanism is additive embedding summation, not architectural specialization.

Innovation 3: A Unified Framework as an Empirical Claim About Generality

The paper's third distinctive contribution is methodological rather than technical: it demonstrates that a single simple framework, applied without modification across diverse prediction tasks, outperforms domain-specific defaults. This turns "generality" from an aspiration into an empirical finding.

What the field did before. As the paper documents in Section 2 and Appendix Section 9, multi-agent prediction was fragmented into domain-specific approaches. Social interaction prediction used variational RNNs or interaction transformers with specialized architectures. Vehicle trajectory prediction used graph neural networks or agent-aware transformer variants with lane information and scene encoders. 6DoF pose tracking used particle filters or convolutional pose refinement networks. Each community developed bespoke solutions, and there was no evidence that a single approach could work across all of them—not because anyone had tried and failed, but because no one had tried.

The default assumption in each subfield was that the interaction mechanisms needed to be tailored to the domain: social interactions require modeling turn-taking and joint attention, driving requires modeling collision avoidance and lane following, hand-object manipulation requires modeling contact dynamics and kinematic constraints. A single framework handling all three would need to be either (a) so abstract as to be vacuous, or (b) implicitly capturing universal properties of multi-agent dynamics.

PAR's empirical claim. The paper demonstrates (b): PAR captures something universal. The framework is not abstract—it makes concrete design choices (flattened token ordering, agent ID embedding, next-timestep prediction). Yet these concrete choices, fixed across all case studies, suffice for discrete classification (AVA actions), continuous regression with geodesic loss (DexYCB rotation), and discrete motion token classification (nuScenes velocity/acceleration). The only changes between tasks are data pre-processing (what counts as an agent, how states are tokenized) and the loss function (determined by token type). The architecture, training procedure, hyperparameters (except learning rate), and inference logic are identical.

This is an empirical finding, not a theoretical proof. The paper doesn't claim that PAR is optimal for all tasks—only that it works, and works better than the single-agent baseline that is often the default. The significance is that it refutes the implicit assumption that multi-agent prediction requires domain-specific interaction architectures. If a 4.4M parameter generic transformer can handle social actions, driving, and manipulation, then the barrier to entry for multi-agent prediction in new domains is much lower than the prior literature suggests.

Why this matters methodologically. The paper is arguing—through demonstration, not rhetoric—for a separation of concerns: the interaction modeling problem (how to structure multi-agent prediction) is distinct from the domain representation problem (how to tokenize states, what loss to use). Researchers can solve each independently. This is analogous to how the transformer separated the sequence modeling problem from the modality representation problem in NLP/CV, enabling rapid progress through transfer of architectural innovations across domains. PAR proposes the same separation for multi-agent prediction.

The limits of this claim. The paper is careful not to overclaim. It explicitly states the model is a "proof-of-concept" and that scaling should be attempted. The three case studies are all relatively small-scale and involve only two or three agents at a time (the nuScenes 10-agent experiment shows no improvement over 3-agent, suggesting diminishing returns). Whether PAR scales to dozens of agents, longer horizons, or more complex interaction dynamics (team sports, crowd simulation) is untested. But as a methodological contribution, the demonstration that "one framework works across three very different domains" is sufficient to establish PAR as a viable default.

Evidence anchor. Tables 1, 3, and 5 all show PAR outperforming AR with the same architecture, with the only changes between tables being the dataset, token type, and loss function. The framework's architecture and training procedure are described as identical in Section 3.4 and implemented with a shared codebase (released with the paper). This is the empirical basis for the generality claim.

Innovation 4: The Diagnostic Value of Single-Agent AR Baselines in Interaction Domains

A subtler but important contribution is the paper's methodological insistence on comparing PAR to a properly implemented single-agent AR baseline—and the finding that such a baseline, despite being the implicit default in many subfields, is consistently and substantially suboptimal. This establishes a diagnostic: if your prediction task involves interacting agents, you should measure how much performance you leave on the table by ignoring the other agents.

What the field did before. In many interaction domains, single-agent prediction was not just a baseline but the standard approach. Action forecasting often conditions only on the target person's past actions. Object pose tracking often conditions only on the object's pose history. These defaults persisted not because anyone had proven them sufficient, but because incorporating interaction information required additional architectural engineering that was domain-specific. The methodological lesson from PAR is that a simple framework makes the baseline comparison trivial—since PAR requires no architecture changes, implementing an AR baseline is just a matter of removing the other agents' tokens and dropping the agent ID embedding. The comparison can be done in an afternoon, and the paper demonstrates that it should be.

The consistent finding—PAR outperforms AR by 1.9 mAP on AVA, 6.3% ADE on nuScenes, 8.9% GEO on DexYCB rotation, and 41% MSE on DexYCB translation—is not just a performance result. It's a diagnostic signal: in each of these domains, a substantial fraction of the prediction problem is socially or physically determined by other agents, and models that ignore this are structurally limited. The magnitude varies by domain—the 41% translation improvement on DexYCB suggests that object motion is overwhelmingly determined by hand motion, which is intuitive, while the 6.3% ADE improvement on nuScenes suggests that vehicle trajectories have a significant but more modest dependence on neighboring vehicles.

Why this is an innovation beyond the specific results. The paper is not just reporting gains; it's making a methodological argument that the field should treat single-agent prediction as a straw man in interaction domains—something to beat immediately, not a serious contender. This reframes the baseline expectations for future work. A new trajectory prediction method that only considers the ego vehicle should be compared against a PAR baseline that incorporates neighboring vehicles, with the understanding that any method ignoring interaction information is solving a strictly harder problem with strictly less information.

Evidence anchor. Tables 1, 3, and 5 each present single-agent AR as the primary baseline, and PAR outperforms it in every case. Table 2 (AVA baselines) shows that standard non-learned baselines (random, nearest neighbor) perform far worse than AR, confirming that the AR baseline is a meaningful learned competitor, not a straw man. The key methodological claim is not that PAR is state-of-the-art (it isn't—the model is too small), but that the AR vs. PAR comparison reveals how much information is structural in the interaction and missing from the single-agent view.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper evaluates on three distinct benchmarks to demonstrate cross-domain generality: (1) AVA (Atomic Visual Actions) for social action forecasting — 235 training and 64 validation 15-minute movie clips annotated at 1 Hz with 60 action classes and person bounding boxes/tracks; clips are selected with continuous action sequences spanning at least 4 seconds (split at 12 seconds maximum) and the first half is used as history to predict the second half. (2) nuScenes for vehicle trajectory prediction — input of 2 seconds of positions to forecast 6 seconds ahead, using only vehicles as agents and the trajdata interface for loading/visualization. (3) DexYCB for 6DoF object pose forecasting — 1000 videos (800 train, 40 val, 160 test) of subjects manipulating 20 YCB objects, using one of 8 camera views, with labels including object SO(3) rotation, 3D translation, and hand 3D translation.

Base model. All experiments use an identical proof-of-concept Llama transformer decoder (Touvron et al., 2023) with 8 layers, 8 attention heads, hidden and intermediate dimensions of 128, totaling approximately 4.4 million parameters (not counting small learned embedding layers adding "a few thousand more"). The architecture uses rotary positional encodings (RoPE; Su et al., 2024). This deliberately small model is chosen to demonstrate that PAR's gains over AR are not dependent on large-scale training — the goal is to establish the framework's validity, not to achieve state-of-the-art absolute performance. The learning rate is the only hyperparameter that varies between case studies; all other architectural and optimizer settings remain constant (Appendix Section 10).

Metrics. Three distinct evaluation regimes correspond to the three tasks: (1) AVA: mean average precision (mAP) over the 60 action classes, computed on the validation set since test set annotations are unreleased; actions are multi-label (a person can perform multiple actions simultaneously), and outputs are 60-dimensional real-valued vectors. (2) nuScenes: standard average displacement error (ADE) and final displacement error (FDE) in meters, measuring the average Euclidean distance between predicted and ground-truth xy positions over the full 6-second horizon (ADE) and at the final timestep (FDE). (3) DexYCB: mean squared error (MSE) for 3D translation prediction and geodesic distance (GEO) in radians for rotation prediction (computed as the angle of the relative rotation between predicted and ground-truth quaternions).

Baselines. The paper implements a consistent set of non-learned and learned baselines across case studies (Section 3.3): Random Token (pick random tokens from the available token space), Random Trajectory (pick a random training trajectory as the prediction), Nearest Neighbor (NN) (given ego agent's history, find the closest training trajectory by distance metric — Hamming for AVA actions, MSE for nuScenes positions — and use that agent's future), Multiagent NN (for two-agent settings: find the closest training trajectory for the non-ego agent and use that agent's partner's future), and Mirror (in two-agent settings, use the other agent's ground-truth future as the prediction). Additionally, the primary learned baseline is single-agent AR: the same transformer architecture trained with standard next-token prediction on only the ego agent's trajectory, using teacher forcing and greedy decoding at inference. For nuScenes, the paper also compares velocity token vs. acceleration token variants of single-agent AR.

Generation budget / compute accounting. The paper does not compare methods under a shared FLOPs or generation budget constraint — PAR and AR use the same architecture, so per-timestep computation is identical; PAR processes more tokens (all agents vs. one agent) at each timestep but makes predictions for only the ego agent at inference. The comparison is at equal model scale (~4.4M parameters) and equal training data, with the difference being the conditioning structure (multi-agent vs. single-agent) and the prediction objective (next-timestep vs. next-token). There is no compute-optimal scaling analysis or budget sweep in the generation count — the paper focuses on demonstrating that multi-agent conditioning improves accuracy, not on characterizing the cost-accuracy tradeoff.

Cross-validation / statistical protocol. The paper does not employ cross-validation or statistical significance testing. For AVA, the evaluation is on the full validation set (test set annotations are unreleased). For nuScenes, evaluation uses the validation set (since the test set requires leaderboard submission). For DexYCB, evaluation uses the 160-video test split. The paper reports single-run results without error bars or confidence intervals. The ablation studies (removing agent ID embedding, switching between next-token and next-timestep prediction) are run as single experiments. The primary claim of robustness rests on the fact that the same architecture and training procedure produce consistent PAR-over-AR gains across three independent datasets and tasks, which serves as an implicit cross-validation: the pattern is replicated across domains rather than tuned to one.

Main Quantitative Results

The headline result (Table 1): 2-agent PAR achieves 42.6 mAP on the AVA validation set, compared to 40.7 mAP for 1-agent AR — an absolute gain of +1.9 mAP. This is set against non-learned baselines (Table 2) where the strongest, single-agent NN, reaches only 13.17 mAP and multiagent NN reaches only 5.10 mAP, confirming that both AR and PAR are substantially above simple retrieval methods. When restricting to the 13 two-person action classes enumerated in Figure 5, PAR achieves 39.8 mAP versus AR's 36.3 mAP — a +3.5 mAP gain. Figure 11 shows that PAR also outperforms AR on the majority of single-person action classes, with notable gains on walking (+11.3 mAP), listening to music (+5.4 mAP), and watching TV (+1.7 mAP).

The ablation results in Table 1 decompose the PAR framework's contributions. Row 2 (2-agent AR with next-token prediction, no next-timestep) achieves only 38.0 mAP — substantially worse than single-agent AR (40.7 mAP), demonstrating that naïve multi-agent next-token prediction actively harms performance. Row 3 (next-timestep prediction but no agent ID embedding) reaches 40.2 mAP, and row 4 (agent ID embedding but no next-timestep prediction, i.e., next-token) reaches 40.0 mAP — both near single-agent AR, showing that each component alone recovers AR-level performance but the combination (row 5: both next-timestep and agent ID embedding at 42.6 mAP) is required for the gain. The key insight from these ablations is that multi-agent information is only beneficial when structured correctly (next-timestep prediction respecting temporal causality) and identifiable (agent ID embedding distinguishing agents).

Qualitative results (Figure 4) illustrate the mechanism: in a conversation scene where a man and woman alternate talking and listening, AR predicts the man will continue talking (extrapolating his own history), while PAR recognizes the woman is talking and correctly predicts turn-taking — the man will listen. Figure 5 quantifies this across classes: "listen to" gains +7.0 mAP, "talk to" gains +4.4 mAP, "kiss" gains +8.3, "fight/hit" gains +5.7, "hug" gains +5.7, and "hand shake" gains +4.0.

Vehicle Trajectory Prediction on nuScenes

The headline result (Table 3): 3-agent PAR with acceleration tokens and location positional encoding (LPE) achieves an ADE of 1.35 m and FDE of 3.34 m, compared to 1-agent AR with acceleration tokens at 1.44 m ADE and 3.57 m FDE — relative improvements of 6.3% ADE and 6.4% FDE. The non-learned baselines (Table 4) show that NN achieves 1.80 m ADE and 4.13 m FDE, substantially worse than all learned models, confirming that the AR baseline is a meaningful competitor.

Tokenization matters: acceleration tokens consistently outperform velocity tokens for both AR and PAR. With velocity tokens, 1-agent AR achieves 1.50 m ADE / 3.64 m FDE, while 3-agent PAR achieves 1.45 / 3.51 (a 3.3% ADE improvement). Switching to acceleration tokens improves 1-agent AR to 1.44 / 3.57 and 3-agent PAR (without LPE) to 1.40 / 3.44. The authors hypothesize that acceleration tokens benefit from a much smaller vocabulary (13 bins per dimension, 13×13=169 vocabulary size) compared to velocity tokens (128 bins per dimension, 128×128=16,384), making the discrete classification task easier to optimize.

The location positional encoding (LPE) provides an additional boost: adding LPE to the 3-agent acceleration-token PAR reduces ADE from 1.40 to 1.35 and FDE from 3.44 to 3.34. This is a meaningful contribution because the model otherwise has no spatial awareness — it knows agents' speeds (through discrete motion tokens) but not their positions, which is critical for collision avoidance.

The ablation in Table 8 isolates the PAR components for the car trajectory setting. 3-agent AR with next-token prediction (row 2) achieves 1.36 / 3.37 — surprisingly close to full PAR (row 5: 1.35 / 3.34), unlike in the AVA case where next-token AR collapsed. The authors note that next-token AR actually has more information at inference (it sees non-ego agents at the current timestep before predicting the ego), which may compensate for the mismatched training objective. However, full PAR with both components still provides a small consistent edge.

Scaling to 10 agents (Table 9) does not improve performance: 10-agent PAR with LPE achieves 1.35 / 3.35, essentially identical to 3-agent PAR with LPE at 1.35 / 3.34. The authors hypothesize that for highway driving, the two most influential agents are the car directly ahead and the car in the adjacent lane, and additional agents provide marginal information on the relatively simple nuScenes dataset. This is not a failure of PAR — the model handles 10 agents without degradation — but suggests diminishing returns for simply adding more agents without a mechanism to weigh their importance.

Qualitative results (Figure 6) show that the 3-agent PAR model handles situations where single-agent predictions might lead to collisions (Scenario A), uses other agents' behavior to better adhere to road boundaries without environment data (Scenario A, C), and adjusts speed based on the behavior of nearby cars (Scenario D).

Object Pose Forecasting on DexYCB

The headline results (Table 5): For rotation forecasting, 2-agent PAR achieves a geodesic distance of 0.837 rad, compared to 1-agent AR at 0.919 rad — a relative improvement of 8.9%. For translation forecasting, 2-agent PAR achieves an MSE of 2.17 × 10⁻³ m², compared to 1-agent AR at 3.68 × 10⁻³ m² — a relative improvement of 41.0%. The non-learned baselines (Table 6) are substantially worse: random token achieves 0.244 m² translation MSE and 2.196 rad rotation GEO; the best non-learned baseline (Mirror for translation) achieves only 1.20 × 10⁻² m².

The dramatic asymmetry between rotation (8.9% improvement) and translation (41.0% improvement) is informative. Translation is heavily constrained by hand motion — the object moves where the hand moves — so incorporating hand position as a second agent provides a strong predictive signal. Rotation, by contrast, depends more on object-specific dynamics (how the object is grasped, its shape, contact forces) that may not be fully captured by hand translation alone (the dataset does not provide hand rotation, only 3D hand translation). The substantially larger gain for translation validates the core PAR hypothesis: when an agent's state is causally influenced by another observable agent, conditioning on that agent improves prediction.

The agent ID embedding ablation (Table 7) shows that removing the agent ID embedding degrades rotation more than translation: rotation GEO increases from 0.837 to 0.895 (a 6.9% relative degradation), while translation MSE increases from 2.17 × 10⁻³ to 2.26 × 10⁻³ (a 4.1% relative degradation). The authors hypothesize that this is because in rotation prediction, the hand token (3D translation) and object token (quaternion) are different modalities with different dimensionalities and semantic types, so the agent ID embedding is crucial for the model to distinguish them. In translation prediction, both tokens are 3D vectors, so the token content alone provides some agent-typing signal.

Qualitative results show the spatial impact: Figures 7 and 8 project 3D predictions onto images, isolating rotation (with ground-truth translation shown) and translation (with ground-truth rotation shown). The PAR predictions align more closely with ground-truth object poses, and Figures 12 and 13 (Appendix) show sampled frames over full prediction sequences, confirming that the advantage persists across the full forecasting horizon.

Summary of Cross-Domain Gains

Across all three case studies, PAR consistently outperforms single-agent AR despite using the identical architecture and training procedure:

Domain	Metric	AR	PAR	Improvement
AVA (actions)	mAP	40.7	42.6	+1.9 absolute
AVA (2-person)	mAP	36.3	39.8	+3.5 absolute
nuScenes	ADE (m)	1.44	1.35	6.3% relative
nuScenes	FDE (m)	3.57	3.34	6.4% relative
DexYCB (rot)	GEO (rad)	0.919	0.837	8.9% relative
DexYCB (transl)	MSE (m²)	3.68×10⁻³	2.17×10⁻³	41.0% relative

Ablation Studies and Robustness Checks

Next-token vs. next-timestep prediction (AVA, Table 1, rows 2 vs. 5): The most critical ablation. 2-agent AR with standard next-token prediction achieves 38.0 mAP, which is 2.7 points below single-agent AR (40.7) and 4.6 points below PAR (42.6). This demonstrates that naïvely applying language-modeling-style next-token prediction to flattened multi-agent sequences is not just unhelpful but actively harmful — the model performs worse with more information because that information is structured in a causally incorrect way. This is the cleanest evidence that PAR's next-timestep objective is not a minor tweak but a categorical correction.

Agent identity embedding (AVA, Table 1, rows 4–5; DexYCB, Table 7): On AVA, removing the agent ID embedding from PAR drops mAP from 42.6 to 40.0 — a 2.6-point loss that brings performance back to roughly single-agent AR levels, indicating that multi-agent information is useless without identity tags to distinguish agents. On DexYCB, removing the embedding has asymmetric effects: rotation GEO degrades from 0.837 to 0.895 (6.9% relative increase in error), while translation MSE degrades from 2.17 × 10⁻³ to 2.26 × 10⁻³ (4.1% relative increase). The asymmetry is attributed to the different token modalities (hand = 3D vector vs. object = quaternion) making agent identity more critical for rotation prediction.

Next-timestep prediction + agent ID embedding on nuScenes (Table 8): The ablation is less dramatic than on AVA. 3-agent AR with next-token prediction (row 2: 1.36 / 3.37) is close to full PAR (row 5: 1.35 / 3.34), and removing either component individually (rows 3–4) produces nearly identical results. The authors note that next-token AR actually has more information at inference (sees non-ego agent at current timestep), which likely compensates for the weaker training objective. This suggests that on domains where the temporal dynamics are simpler (cars mostly maintain velocity), the structural advantage of next-timestep prediction is less pronounced. However, the location positional encoding (Section 5.1) provides an additive gain beyond the PAR components: 3-agent PAR without LPE achieves 1.40 ADE, with LPE achieves 1.35 ADE — a 3.6% relative improvement, confirming that spatial awareness is complementary to identity-aware multi-agent conditioning.

Velocity vs. acceleration tokenization (nuScenes, Table 3): Both AR and PAR benefit from acceleration tokens over velocity tokens. Single-agent AR: velocity 1.50 / 3.64 vs. acceleration 1.44 / 3.57 (4.0% ADE improvement). 3-agent PAR: velocity 1.45 / 3.51 vs. acceleration 1.40 / 3.44 (3.4% ADE improvement). The authors attribute this to vocabulary size — acceleration bins (13 per dimension, 169 total) are much smaller than velocity bins (128 per dimension, 16,384 total), making the classification problem easier to optimize. This is an important practical finding: the choice of tokenization granularity matters substantially for discrete token prediction, and finer quantization is not always better.

Number of agents (nuScenes, Table 9): Scaling from 3 agents to 10 agents does not improve performance (3-agent with LPE: 1.35 / 3.34; 10-agent with LPE: 1.35 / 3.35). Without LPE, 3-agent achieves 1.40 / 3.44 and 10-agent achieves 1.39 / 3.43. The model handles the increased complexity without degradation (PAR can scale to 10 agents), but the additional agents do not provide predictive value on the nuScenes validation set. This is a negative result that is sensibly interpreted: for highway driving, only the nearest one or two vehicles strongly influence ego behavior, and the other agents are either too far away or their dynamics are already captured by the closest agents' behavior. The paper acknowledges this as a limitation and suggests that tasks with richer group interactions (team sports) might benefit from more agents.

ReST^EM revision model for object pose (Appendix K, Figure 16, cited in Section 6 of the main paper): An attempt to use reinforcement learning with self-generated data (ReST^EM; Singh et al., 2024) to further optimize a revision model for object pose estimation. This experiment is described in the revision model section of the paper, not in PAR — it represents an attempt to improve a sequential revision approach for DexYCB. The result is negative: additional sequential revisions substantially hurt performance. At 256 generations, fully sequential performance dropped to approximately 33.5% (in unspecified units) compared to roughly 38.5% at the optimal sequential-to-parallel ratio. The authors hypothesize that on-policy data collection exacerbated spurious correlations in the revision trajectories, causing the model to fail to learn the revision task properly. This is relevant to PAR as a cautionary note: the benefits of multi-agent conditioning are robust under supervised learning, but attempts to optimize the framework with RL-style methods may backfire if the data distribution shifts.

Cross-domain robustness (implicit ablation): The strongest robustness check is the paper's structure itself: applying exactly the same architecture, hyperparameters (except learning rate), and training procedure to three tasks with different modalities, token types, loss functions, and evaluation metrics. This is not a controlled ablation in the traditional sense, but it serves the same function: demonstrating that no single result is dependent on a case-study-specific design choice. The consistent PAR-over-AR pattern across all three domains — despite using MSE for AVA actions, cross-entropy for nuScenes motion tokens, geodesic distance for DexYCB rotation, and MSE for DexYCB translation — strongly suggests that the gains are due to the multi-agent conditioning structure, not to domain-specific tuning.

Critical Assessment

Does the evidence support the claim that PAR is a "simple unifying approach"?

The paper's central claim is that three design choices — flattening multi-agent tokens, adding learned agent ID embeddings, and training with next-timestep prediction — constitute a unifying framework that works across diverse multi-agent prediction tasks "without modifications aside from domain-specific data processing" (Section 1). The experimental evidence supports this claim in the narrow sense demonstrated: the same architecture with the same hyperparameters (except learning rate) produces consistent PAR-over-AR improvements on three benchmark tasks spanning discrete classification, continuous regression, and mixed-modality prediction.

However, there are important qualifiers:

The tasks share a structural commonality that limits the generality claim. All three case studies involve purely predictive tasks with clean state representations extracted from video datasets — the paper assumes access to perfect tracking and state labels. In the real world, agent states must be estimated from noisy perception, and tracking failures are common. The paper does not test PAR on raw sensor data (images, lidar) or under state estimation uncertainty. This is an acknowledged scope limitation, not a flaw, but it means the claim of "unification" applies to symbolically-extracted interaction data, not to end-to-end learning from pixels.

The number of agents and the complexity of interactions are limited. AVA uses 2 agents, DexYCB uses 2 agents (hand + object), and nuScenes uses 3–10 agents but shows no benefit from more than 3. All interactions are pairwise or small-group — there is no test on crowd dynamics, team sports, or scenarios where dozens of agents interact simultaneously. The paper acknowledges this limitation (Section 7) and suggests it as future work. The framework might scale to more agents, but this is untested.

The model scale is deliberately tiny. At 4.4M parameters, this is several orders of magnitude smaller than production models in any of these domains. The claim is that PAR provides a "proof-of-concept" — and it does — but the paper does not demonstrate that the PAR-over-AR advantage persists at larger scales. It is possible that larger AR models with more capacity can learn to infer interaction dynamics from ego-only trajectories (through implicit world modeling), reducing or eliminating the PAR advantage. The paper cannot rule this out. The consistent finding that even a tiny PAR model beats an equivalently-sized AR model does not necessarily mean the gap would persist at 100M or 1B parameters.

The experimental protocol lacks statistical rigor. All results are reported as single numbers without confidence intervals, error bars, or statistical tests. The test sets are relatively small: AVA validation is 64 videos (though the number of action sequences after filtering is not specified), nuScenes validation has several thousand scenes but the paper uses only vehicles, and DexYCB test is 160 videos. A 1.9 mAP improvement on AVA with 60 classes might or might not be statistically significant — the paper provides no way to assess this. The cross-validation is implicit (replication across three datasets) rather than statistical within each dataset. This is not unusual for proof-of-concept papers but limits the strength of any specific numerical claim.

Does the evidence support the claim that next-timestep prediction is the critical innovation?

The ablation in Table 1 (AVA) provides the strongest evidence: next-token AR with 2 agents scores 38.0 mAP, substantially worse than single-agent AR (40.7), while next-timestep PAR scores 42.6. The 4.6-point swing from worst to best across the ablation is clean and convincing. On nuScenes (Table 8), the evidence is weaker: next-token AR (1.36 / 3.37) is very close to PAR (1.35 / 3.34), with the difference being ~0.01 ADE. The authors note that next-token AR actually has an information advantage at inference (seeing non-ego agents at the current timestep), which may compensate. This means the strongest evidence for next-timestep prediction comes from the domain where the prediction targets are most complex (multi-label action vectors with 60 classes), and the evidence weakens in domains with simpler dynamics (cars mostly maintain velocity). This is a nuanced finding that the paper could have discussed more explicitly: the benefit of causally-correct training objectives is most pronounced when the prediction problem is sufficiently complex that learning from a mismatched objective causes genuine confusion, and less pronounced when the domain dynamics are simple enough that even a mismatched objective can be optimized to reasonable performance.

Does the evidence support the claim that PAR "outperforms AR across three scenarios"?

Yes, with the caveat that "outperforms" means "shows consistent numerical improvement" rather than "is practically better." The absolute gains: +1.9 mAP on AVA (a 4.7% relative improvement), 0.09 m ADE on nuScenes (6.3% relative), 0.082 rad on DexYCB rotation (8.9% relative), 1.51 × 10⁻³ m² on DexYCB translation (41% relative). These are real improvements but are they practically significant?

On AVA, a +1.9 mAP gain on a 60-class problem with a tiny model is encouraging but not transformative — larger models (SlowFast, MViT, Hiera, LART) achieve substantially higher mAP on AVA action recognition/detection by operating on pixels. The PAR paper explicitly does not compare to these, since it operates on ground-truth action labels as input. On nuScenes, 1.35 m ADE from a 4.4M parameter model using only position information (no maps, no images, no scene context) is comparable to simple constant-velocity baselines; the state-of-the-art on nuScenes prediction (using full sensor data and large models) achieves ADE well below 1.0 m. The comparison that matters is AR vs. PAR, not PAR vs. state-of-the-art, and on that metric PAR consistently wins. But the practical relevance of these gains would be stronger if the paper demonstrated that they persist at a model scale and feature set closer to what is used in real systems.

On DexYCB, the 41% translation improvement is practically very significant — reducing MSE from 3.68 to 2.17 × 10⁻³ represents a large gain, and the qualitative examples (Figures 7, 8, 12, 13) show visually obvious improvements. This is the domain where the interaction signal is most dominant (object motion is almost entirely determined by hand motion), and consequently the PAR advantage is largest. This validates the paper's core argument that when interactions are strong, ignoring them is costly.

Missing experiments that would strengthen the paper

Scaling study. The most important missing experiment is a scaling curve: does the PAR-over-AR gap widen, narrow, or stay constant as model size increases from 4.4M to larger scales? If larger AR models can implicitly learn to infer interactions from ego-only behavior, the PAR advantage might diminish — or conversely, if PAR enables more efficient use of parameters (since less capacity is needed for implicit interaction modeling), the advantage might grow. The paper explicitly calls for scaling experiments in Section 7 but does not conduct them.

Comparison to domain-specific multi-agent methods. The paper positions PAR as a unifying framework, but it does not compare PAR against prior domain-specific multi-agent methods — it only compares against single-agent AR. On nuScenes, comparing against MotionLM (Seff et al., 2023), which also uses multi-agent tokens with agent ID embeddings and discrete motion tokens, would test whether PAR's specific design choices (next-timestep prediction, the particular token ordering) matter relative to a prior state-of-the-art multi-agent approach. On AVA, comparing against interaction transformers or relational action forecasting (Sun et al., 2019) would test whether the simple PAR architecture matches purpose-built interaction modules. The paper's claim is not that PAR beats these methods — it doesn't — but testing this would clarify whether the unification comes at a performance cost.

Ablation on token ordering within timesteps. The paper uses a fixed ordering (agent 1, agent 2, ..., agent N) within each timestep, with the ego agent placed last. Does the ordering matter? The paper argues that any consistent ordering would be equivalent (since position is encoded by RoPE, not absolute sequence index), but this is not tested. An ablation comparing different orderings would confirm this claim and provide guidance for practitioners.

Ablation on the number of difficulty levels / interaction strength. The paper does not analyze when PAR helps most. On AVA, Figure 5 shows that 2-person interaction classes benefit more than single-person classes, which is consistent with the hypothesis that PAR captures interaction dynamics. But within 2-person classes, are there classes where PAR does not help? And within single-person classes (Figure 11), some show large gains (walk: +11.3) while others show slight decreases (answer phone: -2.0, drive: -3.4, enter: -2.2, climb: -2.3). The paper does not analyze these failures — what is different about "answer phone" or "drive" that makes multi-agent conditioning unhelpful or harmful? A per-class analysis of when interaction conditioning helps vs. hurts would strengthen the diagnostic value of the framework.

Inference-time cost analysis. PAR processes more tokens per prediction step than AR (N agents vs. 1 agent), which increases per-step computation and memory. The paper does not report inference latency, FLOPs, or memory usage for PAR vs. AR. For deployment, the accuracy gain must be weighed against the computational cost, and this tradeoff is uncharacterized. Since the model is tiny (4.4M parameters), the absolute cost is small, but the relative cost increase from AR to PAR could be substantial (roughly N× more tokens processed per prediction step, plus the cost of attending over a longer sequence). For a fair comparison, one might want to give AR a proportionally larger budget or compare at iso-FLOPs, neither of which is done.

Conditional validity of the claims

"PAR outperforms AR" — holds across all three benchmarks for the specific 4.4M parameter Llama architecture, but the claim is conditional on: (a) the existence of interaction partners whose states are observable, (b) the interaction being strong enough to influence behavior (the DexYCB translation gain of 41% vs. the nuScenes ADE gain of 6.3% illustrates the range), (c) the task being purely predictive with no exploration or intervention, and (d) the model scale being small (the scaling behavior is unknown).

"Unifying framework without modifications" — holds for the three tested domains but is conditional on: (a) the availability of clean per-agent state trajectories extracted from video (the paper does not handle raw perception), (b) the ability to define a tokenization and loss function for the domain (which is a non-trivial design choice — the paper shows that tokenization choices like velocity vs. acceleration matter), and (c) the interaction being small-group (2–3 agents; scaling to many agents shows no benefit and may require additional mechanisms like attention sparsification or agent importance weighting).

"Proof-of-concept" — this is the paper's own characterization, and it is accurate. The experiments demonstrate the viability of the PAR approach at a small scale with clean, symbolic inputs. They do not demonstrate that PAR would be practically useful in a deployed system (where perception noise, tracking failures, larger agent counts, and latency constraints apply), nor do they characterize the scaling behavior or compare against domain-specific multi-agent methods. The paper succeeds at its stated goal — providing a starting point for unified multi-agent interaction modeling — but the gap between proof-of-concept and practical deployment is substantial, and the experiments do not address it.

6. Limitations and Trade-offs

The Cost of Difficulty Estimation Is Not Accounted For in the Reported Efficiency Gains

The assumption or constraint. The PAR framework, as presented, always assumes access to ground-truth states of non-ego agents at every timestep during inference — including future timesteps that have not yet occurred at prediction time. Specifically, Equation 1 conditions on S_{1:t-1}^{1:N-1}, meaning the model requires all other agents' states up to the timestep just before the current prediction. At step t = t_π + 1 (the first prediction step), this requires the non-ego agents' states at timestep t_π, which is the final history timestep and is observed. But at step t = t_π + 2, the model requires non-ego agent states at t_π + 1 — a timestep that is in the future relative to the prediction start. The paper explicitly acknowledges this in Section 3.1: at inference, "we provide the observed ground truth states of other agents" — meaning the model has access to information that, in a real deployment, would not be available because it has not happened yet.

The consequence. This is not merely a minor implementation detail. It fundamentally changes what problem PAR solves. Rather than predicting the ego agent's future given only information available at the decision point, PAR predicts the ego agent's future given oracle knowledge of what all other agents will do during the prediction horizon. In the autonomous driving case study, this means PAR knows where neighboring cars will be at t_π + 1, t_π + 2, ..., t_π + F — information that the ego vehicle cannot observe because those positions have not yet occurred. The comparison against single-agent AR is therefore asymmetric: AR predicts from impoverished but realistically available information (only the ego agent's history), while PAR predicts from information that includes partial future ground truth. The paper's headline comparisons (Tables 1, 3, 5) are thus comparing a model with an oracle information advantage against one without it, making the reported gains an upper bound on what would be achievable in a realistic deployment where non-ego agent futures must also be predicted or estimated.

A subtler but equally important consequence: the PAR framework sidesteps the core difficulty of multi-agent interaction modeling — joint futures are interdependent. In real interactions, the ego agent's future behavior influences other agents' future behavior (e.g., if ego brakes, the car behind also brakes). By treating non-ego futures as given rather than co-determined, PAR assumes away the circular causality that makes multi-agent prediction genuinely hard. The framework models reaction to observed others, not interaction where all agents mutually adapt.

What evidence exists in the paper. The paper does not measure the impact of this assumption directly. There is no ablation comparing PAR with oracle non-ego futures to PAR with predicted non-ego futures (generated autoregressively or by some other means). There is no experiment measuring how performance degrades as the prediction horizon extends (and thus the model must condition on its own predictions for non-ego agents at more timesteps). The inference procedure (Section 3.4) explicitly states that non-ego agents' ground-truth states at the current timestep are used — this is the only inference procedure described, and no alternative that predicts non-ego agents jointly is evaluated.

The nuScenes 10-agent experiment (Table 9) provides indirect evidence of the issue: adding more agents does not improve performance (3-agent with LPE: 1.35 ADE; 10-agent with LPE: 1.35 ADE), which the authors attribute to the nearest one or two vehicles being the primary causal influences on ego behavior. But it could also reflect that providing ground-truth futures for distant agents is providing "free information" that a deployed system would not have, and the model learns to rely on this rather than developing a more robust interaction model.

Mitigation status. The paper does not address this limitation. Section 7 mentions that future work could explore "incorporating environmental context or tokenizing pixel patches, especially as a way to relax our assumption on high-quality tracking," but does not discuss the oracle futures assumption or how to remove it. The paper frames the problem as ego-conditioned prediction rather than joint multi-agent forecasting, which is a deliberate scope restriction, but it does not acknowledge that this scope restriction makes the comparison to single-agent AR fundamentally asymmetric — AR is solving a strictly harder version of the task (predict the future from past information alone) than PAR (predict the future from past information plus future non-ego observations).

Single Benchmark per Domain and Single Model Architecture Prevent Generalization Claims

The assumption or constraint. All three case studies use a single dataset per domain: AVA for social action forecasting, nuScenes for vehicle trajectory prediction, and DexYCB for object pose forecasting. All experiments use a single model architecture (4.4M-parameter Llama decoder) with fixed hyperparameters (8 layers, 8 heads, hidden dimension 128, rotary positional encoding). There is no cross-domain evaluation (e.g., testing the AVA-trained model on a different social interaction dataset), no architecture comparison (e.g., does PAR also help with encoder-decoder transformers or RNNs), and no attempt to characterize how the PAR-over-AR gap varies with dataset properties or model scale.

The consequence. The paper's central claim — that PAR is a "unifying framework" that generalizes across domains — rests on the observed pattern that PAR > AR on three specific benchmarks with one specific architecture. But this pattern could be an artifact of dataset-specific properties that happen to favor the PAR conditioning structure, or of the specific model scale (4.4M parameters) where the limited capacity of the AR model makes the missing interaction information especially costly. Without a second dataset per domain, we cannot know whether PAR's advantage on AVA actions transfers to other action forecasting benchmarks (e.g., EPIC-Kitchens, Something-Something) or whether it is specific to the movie-domain interactions and action distribution in AVA. Without a model scale sweep, we cannot know whether a 400M-parameter AR model with greater capacity would learn to implicitly infer interaction dynamics from ego-only behavior, closing the gap.

The choice of a single, deliberately small model architecture introduces a related concern: perhaps the PAR advantage is partly an artifact of the model being too small to learn interaction patterns from ego-only trajectories. Larger models are known to develop emergent world models and may implicitly track other agents' states from ego observations. If the PAR-over-AR gap shrinks with model scale, the practical significance of PAR depends on the deployment scale — it might be most valuable for small, efficient models and less important for large, compute-heavy models.

What evidence exists in the paper. The paper provides no scaling curve (model size vs. PAR-AR gap) and no second dataset per domain. The cross-domain pattern (PAR > AR on tasks with very different interaction strengths — 41% translation improvement on DexYCB vs. 6.3% ADE improvement on nuScenes) is consistent with the hypothesis that PAR helps most when interactions are strongest, but this is a single data point per interaction-strength regime. The authors explicitly acknowledge the scaling limitation: "we have only provided a starting point that can be built upon extensively. For instance, incorporating environmental context or tokenizing pixel patches... are avenues for further research using PAR. It would be interesting to experiment with scaling the data and model" (Section 7). However, they do not acknowledge that the lack of scaling experiments means the practical value of PAR at deployment-typical model sizes is unknown.

Mitigation status. The paper partially mitigates this through its explicit framing as a "proof-of-concept" (Section 1, Section 7). The authors are transparent that the model is deliberately small and that scaling is future work. However, the paper's title ("Poly-Autoregressive Prediction for Modeling Interactions"), abstract ("We introduce a simple framework... PAR outperforms AR across these three scenarios"), and introduction all present PAR as a general finding without qualifying that the finding is conditional on model scale and dataset choice. The mitigation is in the framing language ("starting point," "proof-of-concept") but the strength of the generalization claims in the main text is not similarly qualified.

Only Symbolic, Perfectly Observed Agent States Are Tested — No Raw Perception or State Uncertainty

The assumption or constraint. All three case studies operate on symbolically extracted agent states that are assumed to be perfectly observed at every timestep. The inputs to PAR (and AR) are ground-truth labels from the dataset: action class vectors from AVA annotations, xy position coordinates from nuScenes tracking, 6DoF object poses and 3D hand translations from DexYCB labels. The paper never tests PAR on raw sensor data (video frames, lidar point clouds, RGB-D images) and never introduces state estimation noise, tracking failures, or missing observations. The pipeline diagram in Figure 2 shows "Data Collection → Entity Extraction → PAR Training and Inference → Evaluation," with entity extraction producing clean symbolic representations from dataset labels or "computer vision techniques" — but the experiments only use dataset labels, not the output of a perception system.

The paper briefly notes this assumption: "We assume that each agent in the dataset is detected at each frame and is associated with an agent ID" (Section 3.3, under "Data"). But this understates the gap between symbolic extraction from ground-truth labels and real perception. In a deployment setting, agent detection, tracking, and state estimation are themselves error-prone machine learning problems — tracking IDs can be lost or swapped, pose estimates are noisy, action recognition is imperfect, and occlusions cause missing observations. PAR's performance when fed such imperfect inputs is entirely uncharacterized.

The consequence. This limitation affects both the practical deployability of PAR and the interpretation of its advantage over AR. If agent state perception is noisy, does PAR still outperform AR, or does conditioning on noisy non-ego states introduce errors that cancel the benefit of the additional information? If tracking fails and agent IDs are temporarily lost, does PAR degrade gracefully or catastrophically? If occlusion prevents observing the interacting agent for several timesteps, can PAR fall back to AR-style ego-only prediction or does it produce nonsensical outputs?

The agent ID embedding, in particular, is trained assuming consistent agent identity across timesteps. If a tracking system produces ID swaps (agent 1 becomes agent 2), the PAR model has never seen such inputs during training and may produce unpredictable outputs. Single-agent AR, which has no agent identity concept, is immune to this failure mode.

A more subtle consequence: operating on symbolic states rather than raw perception means PAR avoids learning a visual representation of interactions — it cannot learn that certain visual patterns (eye gaze, body orientation, hand contact) are predictive of interaction outcomes. The framework, as tested, depends entirely on the quality of the symbolic state extraction, which in turn depends on the quality of the underlying perception system. The paper's claim that PAR is a "unifying framework" is thus more accurately stated as a unification of symbolic interaction modeling, not end-to-end interaction prediction from raw sensory data.

What evidence exists in the paper. None. There is no experiment that adds noise to the input states, simulates tracking failures, or evaluates on the output of a perception pipeline. The paper does not report the quality of the underlying perception for any dataset (e.g., how accurate the AVA action labels are, the tracking error on nuScenes, the pose estimation error on DexYCB). The evaluation assumes the inputs are ground truth.

Mitigation status. The paper partially acknowledges this limitation in Section 7 when it suggests "tokenizing pixel patches, especially as a way to relax our assumption on high-quality tracking." But this is framed as a future extension, not as a characterization of the current framework's limitations. The assumption of perfect symbolic extraction is never listed as a limitation of the current results. The paper does not discuss how PAR would perform under state uncertainty or how the framework could be extended to handle noisy perception (e.g., by training on noisy inputs, by incorporating uncertainty estimates, or by joint state estimation and prediction).

The Single-Agent AR Baseline Is Not Given the Same Inference-Time Information as PAR

The assumption or constraint. The primary comparison in the paper is 1-agent AR vs. N-agent PAR, where AR sees only the ego agent's history and PAR sees all agents' histories (including partial future ground truth for non-ego agents — see first limitation above). But this comparison is not iso-information: PAR gets strictly more information at inference time than AR. The paper does not evaluate an alternative baseline where AR is given a summary of non-ego agent information — for instance, by encoding the non-ego agents' histories into a fixed-length context vector that is appended to the ego agent's input, or by giving AR the non-ego agents' histories as additional input tokens at each timestep while keeping the standard next-token prediction objective. Such a baseline would test whether the performance gain is due to having more information (which any model would benefit from) or due to the specific PAR conditioning structure (next-timestep prediction + agent ID embedding).

The consequence. The paper cannot cleanly attribute the PAR-over-AR improvement to the next-timestep prediction objective or the agent ID embedding specifically, because any improvement could be partially or entirely due to the additional input information. The ablation in Table 1 (2-agent AR with next-token prediction scores 38.0 mAP, worse than single-agent AR at 40.7 mAP) suggests that simply adding multi-agent tokens with a naive objective hurts performance, which is evidence that the PAR structure matters. But this ablation uses the same flattened sequence structure as PAR and changes only the prediction target — it does not test whether a different way of incorporating multi-agent information into an AR model (e.g., a separate encoder for non-ego agents) would recover or exceed PAR's performance.

This is particularly important for the practical takeaway: if a practitioner can achieve similar gains by simply concatenating non-ego agent histories to the ego agent's input and training with standard AR, then the PAR framework's specific innovations (next-timestep prediction, joint training) are unnecessary. The paper does not provide evidence against this simpler alternative.

What evidence exists in the paper. Only the next-token AR ablation (Table 1, row 2; Table 8, row 2), which tests a specific straw-man alternative (flattened sequence with standard next-token prediction). The paper does not test:

AR with non-ego agent history provided as additional input tokens at each timestep (but standard next-token prediction within the ego-only sequence).
AR with non-ego agent states encoded by a separate encoder and concatenated to the ego representation.
AR with an interaction feature (e.g., relative positions, distances) computed from non-ego agents and provided as auxiliary input.
PAR without joint training (i.e., train to predict only the ego agent, not all agents).

The last ablation is particularly notable: if joint training (predicting all agents during training) is independently beneficial, then an AR model trained to also predict non-ego agents from shared representations might show improvement without requiring the full PAR prediction structure. This ablation is not performed.

Mitigation status. The paper does not acknowledge this baseline gap. The next-token AR ablation is presented as the multi-agent AR alternative, and all other comparisons are against single-agent AR. The paper's claim that PAR "outperforms AR" is true for the specific AR baselines tested, but whether it would outperform a more sophisticated AR model that uses multi-agent information in a different way is unanswered. The paper's position — that PAR's specific innovations (next-timestep prediction and agent ID embedding) are what enable multi-agent conditioning to be beneficial — is supported by within-PAR ablations (Table 1) but not by comparisons against alternative ways of achieving multi-agent conditioning within an AR framework.

No Analysis of Inference-Time Computational Cost Relative to Accuracy Gains

The assumption or constraint. The paper compares PAR and AR at equal model scale (both use the same 4.4M-parameter architecture) and equal training data, but it does not compare them at equal inference-time computation. PAR processes N× more tokens per timestep than single-agent AR (since it includes all N agents' states in its input sequence), which increases both the per-step floating-point operations (more tokens to embed and process through self-attention, which scales quadratically in sequence length) and the memory footprint (storing activations for a longer sequence during autoregressive generation). The paper reports accuracy metrics (mAP, ADE/FDE, MSE, GEO) without reporting inference latency, FLOPs per prediction step, or peak memory usage for any method.

The consequence. A practitioner deciding whether to adopt PAR cannot perform a cost-benefit analysis. If PAR achieves a 6.3% ADE improvement on nuScenes but requires 3× more compute per prediction step (since it processes 3 agents rather than 1) and has 3× the memory footprint, the practical value of the improvement depends on the deployment constraints: is a 6.3% ADE reduction worth 3× the inference cost? For a real-time autonomous driving system with strict latency budgets, the answer might be no if the latency increase pushes the system above the maximum allowed inference time. For an offline batch prediction system, the answer might be yes. The paper provides no data to inform this decision.

The cost asymmetry is particularly relevant because the comparison to single-agent AR is already information-asymmetric (PAR gets oracle non-ego futures). If PAR is both information-privileged and compute-intensive relative to AR, then the accuracy gains conflate two advantages: better information and more computation. A fairer comparison would give AR additional compute budget (e.g., an ensemble of AR models, or a deeper AR model with more layers) to match PAR's inference FLOPs, asking whether compute spent on multi-agent conditioning is more valuable than compute spent on a larger single-agent model. This experiment is not performed.

What evidence exists in the paper. None. No latency measurements, no FLOPs counts, no memory profiling. The paper's model is deliberately small (4.4M parameters), so absolute costs are low and the relative increase from adding agents may not be a practical concern at this scale. But the paper positions PAR as a framework to be scaled ("we have only provided a starting point that can be built upon extensively"), and at the scales used in production systems (hundreds of millions to billions of parameters), the per-step cost of processing multiple agents' tokens could be substantial — particularly since self-attention complexity is O(n²) in sequence length, and PAR's flattened sequence is N× longer than AR's.

The joint training objective also has a cost implication: during training, PAR computes losses for all N agents at every timestep, while a single-agent AR model only computes loss for the ego agent. This increases the training compute (more loss evaluations, more gradients flowing through the model) but is not accounted for in any comparison. The paper does not report training time or total FLOPs for PAR vs. AR training.

Mitigation status. The paper does not discuss inference cost. The only mention of computational considerations is in the context of model size, not inference latency per se — the paper emphasizes that the model is small and therefore the experiments are cheap to run, but it does not characterize how cost scales with the number of agents or sequence length. This is a significant gap for a paper that positions itself as providing a framework for practical multi-agent prediction. A cost analysis is entirely missing, and there is no suggestion in Section 7 that such an analysis would be valuable future work.

All Results Are from a Small Number of Runs Without Statistical Reliability Estimates

The assumption or constraint. The paper reports all results as single-point estimates without any measure of statistical uncertainty. There are no confidence intervals, no error bars, no standard deviations across training runs with different random seeds, and no statistical significance tests comparing PAR to AR performance. The test sets are relatively small: AVA validation set (64 videos, though the number of extracted action sequences is not reported), DexYCB test set (160 videos), and nuScenes validation set (size not explicitly reported but typically several thousand scenes for the vehicle-only subset used here). With small test sets and no error characterization, the reported numerical differences (e.g., +1.9 mAP on AVA, 1.35 vs. 1.44 ADE on nuScenes) could be within the range of sampling variability or seed sensitivity.

The consequence. The paper's headline quantitative claims are not statistically grounded. If a practitioner re-ran the same experiment with a different random seed and got a smaller (or zero) PAR-over-AR gap, would that be surprising? The paper provides no way to answer this question. For the AVA result specifically, a +1.9 mAP improvement over a 60-class problem evaluated on a subset of the validation set (clips with continuous action sequences ≥ 4 seconds) could be sensitive to clip selection, data split, or model initialization. The nuScenes result, with ADE differences of 0.09 meters (1.44 vs. 1.35), is a roughly 6% relative improvement — but standard practice in the trajectory prediction literature is to report results across multiple runs or with confidence intervals, especially for such small absolute differences.

The ablation results suffer from the same issue. Table 1 shows a progression of rows with mAP values of 40.7, 38.0, 40.2, 40.0, and 42.6. The ordering and the conclusions drawn from it (e.g., "the combination of both components is required for the gain") depend on the assumption that these differences exceed run-to-run variability. If the standard deviation across seeds is, say, ±1.5 mAP, then the difference between 40.2 and 40.0 is noise, and the conclusion that both components matter is not supported. The paper provides no evidence that the differences are robust.

What evidence exists in the paper. The paper provides no statistical characterization. There is no mention of random seeds, no report of variability, no statement about how many training runs were performed. The hyperparameter details in Appendix Section 10 and Section 12 specify learning rates and optimizer settings but do not mention the number of random seeds or whether results are averaged across runs. The main text and tables report single precision values (e.g., 42.6 mAP, not 42.6 ± 1.2 mAP), and there are no error bars in any figure.

The paper's implicit defense against the statistical reliability concern is cross-domain replication: the PAR-over-AR pattern appears in all three case studies, which is less likely to be a statistical fluke than a single-dataset result. This is a reasonable argument — three independent datasets showing the same qualitative pattern is evidence against the results being purely due to random variation. However, cross-domain replication does not provide within-domain confidence estimates; it tells us the pattern is consistent across domains but not how precise any individual number is.

Mitigation status. The paper does not address this limitation. Section 7 (Discussion) does not mention statistical reliability or the need for multiple runs. The "proof-of-concept" framing partially mitigates the concern — a proof-of-concept paper might reasonably report single-run results to demonstrate feasibility, with the expectation that follow-up work will establish statistical reliability at scale. But the paper's abstract and introduction present the results as established findings (e.g., "PAR outperforms AR with relative improvements of 6.3% ADE and 6.4% FDE") without qualifying that these are single-run point estimates whose precision is unknown. A brief statement about the number of training runs and a note that statistical characterization is future work would address this concern without requiring additional experiments.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not claim to advance the state-of-the-art on any individual benchmark — its 4.4M-parameter model is deliberately tiny, and the absolute numbers are far below what large-scale domain-specific systems achieve on AVA, nuScenes, or DexYCB. The contribution that shifts the landscape is methodological unification: PAR demonstrates that three design choices — flattening multi-agent tokens with consistent per-timestep ordering, summing learned agent identity embeddings, and shifting the prediction target from next-token to same-agent next-timestep — suffice to turn a generic transformer decoder into a multi-agent interaction predictor that works across discrete action classification, continuous vehicle trajectory regression, and mixed-modality 6DoF pose forecasting without any architectural modification. The magnitude of this shift is a reframing and a new diagnostic, not a paradigm shift. It does not introduce fundamentally new mathematics or a novel learning principle; instead, it identifies that the field has been systematically misapplying next-token prediction to flattened multi-agent sequences (producing a causally incoherent training objective) and shows that a simple correction — predict the same agent's next state, not the next token in an arbitrary flattened ordering — recovers the proper causal structure and yields consistent gains.

The paper reconciles a latent contradiction in the multi-agent prediction literature. Prior work had developed domain-specific interaction architectures — interaction transformers for social behavior, agent-aware attention mechanisms for trajectory prediction, separate processing streams with cross-attention for hand-object dynamics — implicitly suggesting that modeling interactions requires specialized computational pathways. PAR refutes this: standard self-attention, augmented only with additive agent identity tags, is sufficient. The contradiction is not that prior methods were wrong, but that they were over-engineered for the interaction modeling component — the heavy lifting of interaction reasoning can be done by the same attention mechanism that handles temporal dependencies, once the tokens are structured to respect temporal causality and agent identity. This reframing redirects research effort away from designing ever-more-complex interaction modules and toward the problem of structuring multi-agent data correctly — a shift from architectural innovation to representational hygiene.

The paper also establishes a new diagnostic for the field: if you are applying autoregressive modeling to a multi-agent domain, you should test whether a PAR-style reformulation (next-timestep + agent ID embedding) improves over your current approach. The availability of a simple, architecture-agnostic recipe lowers the barrier to testing this — a researcher can take their existing transformer, restructure their flattened sequence to group by timestep rather than by agent, add an agent ID embedding, shift their loss target by N tokens, and measure the impact. The paper demonstrates that this diagnostic reveals substantial performance left on the table: 41% relative translation error reduction on DexYCB, 8.9% rotation error reduction, 6.3% ADE reduction on nuScenes, and +1.9 mAP on AVA from the same small model with the same training data. Any multi-agent prediction paper that does not report this comparison going forward will have an incomplete evaluation.

Research directions that become more attractive: (1) scaling PAR to model sizes and datasets typical of production systems, to determine whether the PAR-over-AR gap persists or closes; (2) developing methods to remove PAR's reliance on oracle non-ego futures during inference; (3) extending PAR to raw perception inputs rather than symbolic states; (4) exploring mechanism design for dynamically selecting which agents to attend to, since the nuScenes 10-agent experiment showed no benefit from additional agents. Research directions that become less attractive: designing specialized interaction modules (graph neural networks over agent nodes, pairwise interaction features, dedicated cross-attention streams) as first approaches to a new multi-agent prediction domain — PAR provides a simpler default that should be tried first, with specialized architectures justified only if PAR proves insufficient.

Follow-Up Research This Work Enables

Characterizing the PAR-over-AR gap as a function of model scale. The paper uses a 4.4M-parameter model and explicitly calls for scaling experiments. A critical open question is whether larger models close the gap: a 400M or 4B-parameter AR model with greater capacity might learn to implicitly infer interaction dynamics from ego-only trajectories, making explicit multi-agent conditioning unnecessary at scale. A strong follow-up would train PAR and AR models at sizes spanning 4M to 400M parameters on nuScenes (where the gap is modest at 6.3% ADE) and DexYCB translation (where the gap is large at 41%), measuring the PAR-minus-AR gap at each scale. If the gap shrinks monotonically toward zero, PAR is most valuable for edge-deployment small models and less important for datacenter-scale systems. If the gap persists or widens, PAR provides a permanent architectural advantage. The paper's existing data cannot distinguish these hypotheses.

Removing the oracle non-ego futures assumption via joint autoregressive prediction or learned summary encodings. PAR currently requires ground-truth non-ego agent states at every timestep during inference (Section 3.1), including future timesteps. This makes the comparison against AR asymmetric. A direct extension would train PAR to jointly predict all agents' futures autoregressively, feeding all agents' predicted states (including non-ego) back as context, and measure how much of the PAR-over-AR gain survives when the oracle information is removed. A more sophisticated alternative would train a separate encoder that compresses non-ego agent histories into a fixed-length summary vector, and provide only that summary (not future ground truth) to the PAR decoder. The key metric is PAR's performance with predicted vs. oracle non-ego futures, broken down by prediction horizon — this would reveal whether PAR's gains are primarily from knowing what other agents will do (oracle advantage) or from the causal structure of the training objective (which would persist even with predicted non-ego states).

Testing PAR on raw sensor inputs with state estimation uncertainty. The paper operates entirely on symbolic states extracted from dataset labels. A natural stress test is to replace ground-truth states with the output of a perception system — e.g., a pretrained object detector and tracker for nuScenes vehicles, or a 6DoF pose estimator for DexYCB. This would characterize how PAR degrades under state noise, tracking failures, and ID swaps. A specific experiment: take the nuScenes PAR model, replace ground-truth vehicle positions with positions from a standard tracker (with known error characteristics), and measure the ADE/FDE degradation as a function of tracking noise level (controlled by adding synthetic Gaussian noise to ground-truth positions in a sweep). This would inform whether PAR's agent ID embedding — which assumes consistent identity across timesteps — is robust to ID swaps or requires an identity-matching mechanism. The paper's current results cannot speak to deployment viability because perception-quality inputs are untested.

Dynamic agent selection via learned importance weighting. The nuScenes 10-agent experiment (Table 9) shows no benefit over 3 agents, plausibly because distant vehicles have negligible causal influence on ego behavior. This suggests PAR would benefit from a mechanism that learns to attend selectively to the most relevant agents rather than processing all agents equally. A concrete experiment: augment PAR with a learned gating module that takes the sequence of all agents' states and outputs an attention mask (or importance weight) for each non-ego agent at each timestep, trained end-to-end with a sparsity penalty on the number of active agents. The prediction accuracy vs. number-of-agents tradeoff curve would characterize when additional agents are worth the computational cost, directly addressing the diminishing returns observed in Table 9. This also connects to the inference cost limitation — if the model can dynamically prune irrelevant agents, the computational overhead of PAR relative to AR shrinks.

Cross-dataset transfer of the PAR conditioning structure. The paper's three case studies use one dataset per domain, preventing any assessment of whether the PAR conditioning structure transfers across datasets within the same domain. A strong follow-up would train a PAR model on nuScenes vehicle trajectories and evaluate it zero-shot or fine-tuned on a different driving dataset (e.g., Argoverse, Waymo Open Dataset) with different sensor setups, road geometries, and traffic patterns. The metric is the PAR-over-AR gap on the transfer dataset — if the gap transfers, the PAR structure captures universal interaction dynamics rather than dataset-specific correlations; if the gap disappears, the paper's results may be specific to the training distribution. This experiment is cheap to run (the 4.4M model trains quickly) and directly tests the generality claim that is central to the paper's contribution.

Failure mode analysis: when does adding agents hurt? The AVA per-class results (Figures 5 and 11) reveal that while PAR helps on most classes, some single-person action classes show small degradations (answer phone: -2.0 mAP, drive: -3.4, enter: -2.2, climb: -2.3). The paper does not analyze these failures. A diagnostic study would characterize the properties of failure cases: are they classes where the second agent is poorly tracked? Where the second agent's behavior is uncorrelated with the ego? Where the training data contains spurious correlations between the two agents? Understanding when PAR fails is as valuable as understanding when it succeeds, because it defines the boundary conditions for the framework's applicability and suggests mitigations (e.g., a gating mechanism that learns to ignore the second agent when it adds noise rather than signal).

Practical Applications and Downstream Use Cases

Edge-deployment behavior prediction with small models. The paper's consistent PAR-over-AR gains using a 4.4M-parameter model suggest that multi-agent conditioning is particularly valuable for resource-constrained settings where model capacity is limited. A concrete scenario: an on-device social assistant (smart glasses, hearing aid) that needs to predict a conversation partner's next action (speaking, listening, gesturing) from the wearer's own action history and the partner's observed actions. With only 4.4M parameters, PAR achieves 42.6 mAP vs. AR's 40.7 mAP on AVA — a +4.7% relative improvement. The absolute numbers are modest, but the improvement comes from better use of available information without increasing model size, which is exactly the regime where edge deployment operates. The inference cost increase (processing 2 agents vs. 1) is small in absolute terms at this scale.

Offline data generation for multi-agent imitation learning. When generating training data for imitation learning in multi-agent settings (e.g., generating plausible human motion for animation, creating synthetic driving scenarios for simulation), prediction quality directly affects downstream policy performance. PAR's 41% relative translation error reduction on DexYCB (MSE: 3.68 → 2.17 × 10⁻³) is practically significant for robotic manipulation data generation — reducing object pose prediction error by nearly half when generating synthetic hand-object interaction trajectories means imitation-learned grasping policies are trained on more accurate data. The PAR framework can be integrated into a data generation pipeline with minimal engineering: the same code that extracts per-agent state trajectories can be repurposed to generate synthetic futures by feeding PAR's predictions autoregressively.

Multi-agent trajectory prediction for autonomous systems with limited sensor suites. The nuScenes PAR model (1.35 m ADE / 3.34 m FDE) uses only xy positions — no maps, no images, no lane geometry. This is relevant for autonomous systems (drones, delivery robots) that lack high-definition maps or heavy perception stacks but have access to basic multi-agent tracking (e.g., from lidar or cooperative perception). PAR's 6.3% ADE improvement over AR with the same minimal input demonstrates that conditioning on neighboring agents' trajectories provides a predictive signal that partially substitutes for missing environmental context. In practice, a lightweight trajectory predictor that takes tracked positions from a simple detector and runs PAR on a small accelerator could provide socially-aware forecasting without the infrastructure required for full-scene understanding.