Diffusion Model Predictive Control

ArXiv: 2410.05364

🎯 Pitch

Instead of learning a rigid policy, D-MPC uses two diffusion models—one for multi-step action proposals, one for multi-step dynamics—to plan online, matching top offline RL methods on D4RL while requiring only a sampling loop to optimize completely novel reward functions at run time. It also adapts to sudden dynamics changes (like a simulated motor defect) by fine-tuning only its dynamics model on cheap play data, a scenario where prior diffusion planners catastrophically fail.

1. Executive Summary

This paper introduces Diffusion Model Predictive Control (D-MPC), an MPC approach that learns a multi-step action proposal and a multi-step dynamics model—both using diffusion models—and combines them for online planning. On the D4RL locomotion and manipulation benchmarks, D-MPC significantly outperforms existing model-based offline planning methods like MBOP (which uses single-step MLP ensembles with trajectory optimization), achieving an average locomotion score of 65.98 versus MBOP's 33.13, and matches or exceeds state-of-the-art model-free methods like CQL and IQL across medium, medium-replay, and cloned datasets. The method demonstrates the ability to optimize novel reward functions at run time through a simple sampling-based planner (e.g., making a Walker2D agent lunge, balance, or repeatedly jump by specifying target heights not in the training data) and adapt to novel dynamics via fine-tuning only the factorized multi-step diffusion dynamics model on a small amount of play data, recovering from a simulated motor defect where the joint Diffuser baseline degrades instead. D-MPC is competitive with or exceeds a ~14× larger model's performance-equivalent baselines when problems fall within the base model's capability range, establishing that test-time compute allocated through multi-step diffusion planning can substitute for pretraining compute only when the proposal distribution already contains successful behaviors.

2. Context and Motivation

The Core Problem: Designing Agents That Plan with Learned Models Under Novel Conditions

The fundamental question this paper tackles is: how do we build an agent that can plan effectively using learned models of the world and its own behavior, while remaining flexible enough to adapt to new reward functions or changed dynamics at deployment time?

This is not merely an academic exercise. In real-world robotics and sequential decision-making, the objectives we want an agent to pursue often change after training — a robot trained in simulation may need to perform a slightly different task on the factory floor, or may encounter wear-and-tear that alters its joint dynamics. Similarly, the cost of collecting expert demonstrations for every possible task variant is prohibitive. A practical agent must therefore be able to repurpose its knowledge: it should understand how the world works (a dynamics model) and what actions are generally sensible (a behavioral prior), and then combine these at test time to optimize whatever objective is currently specified.

The paper frames this through the lens of Model Predictive Control (MPC). In MPC, the agent does not learn a fixed mapping from states to actions (a policy). Instead, at each decision step, it uses a model of the environment's dynamics to simulate the consequences of candidate action sequences over a finite horizon, scores those imagined trajectories according to a provided objective function, executes the first action of the best sequence, and then replans from the resulting next state. This closed-loop replanning provides robustness — if the model is imperfect, the agent can correct course at each step using fresh observations.

The advantage of this architecture over standard policy learning is threefold, as argued in Section 1:

1. Novel reward generalization: Because the objective function is an explicit input to the planner (not baked into policy weights), the agent can pursue entirely new goals at test time without retraining. This is something a standard policy — which implicitly encodes a single reward function — cannot do. Goal-conditioned RL offers partial flexibility, but only for goals that were seen during training; D-MPC's approach works for arbitrary reward functions specified at run time.

2. Sample-efficient dynamics learning: Learning a dynamics model — predicting the next state given the current state and action — is fundamentally a supervised regression problem. The mapping is typically near-deterministic and well-behaved. Learning a policy, by contrast, requires predicting actions, where the optimal behavior may be multimodal (multiple good actions exist from the same state) and requires accurate long-horizon credit assignment. Given the same offline dataset, the paper argues (citing Zhu et al.) that dynamics models can often be learned more sample-efficiently than policies.

3. Adaptation to novel dynamics: If the environment changes (e.g., a robot joint becomes stiff due to wear), a policy would need to be entirely retrained. But because D-MPC factorizes the dynamics model from the action proposal, only the dynamics model needs to be updated — and this can be done with a small amount of "play" data from the changed environment, as demonstrated in Section 4.3.

Where Existing Approaches Fall Short

The paper situates itself within the broader landscape of offline reinforcement learning methods, as organized in Table 1. To understand why D-MPC is necessary, we need to examine what prior approaches can and cannot do.

Model-Free Methods: Fast but Inflexible

Standard model-free offline RL methods — behavioral cloning (BC), Conservative Q-Learning (CQL; Kumar et al., 2020), Implicit Q-Learning (IQL; Kostrikov et al., 2021) — learn a reactive policy that maps states directly to actions. These methods are fast at runtime and can achieve strong performance on fixed tasks (as shown in the baseline comparisons in Table 2). However, they have two fundamental limitations:

• They cannot optimize novel rewards at test time. The policy encodes a single objective learned during training. If you want the agent to pursue a different goal, you must retrain.
• They cannot adapt to novel dynamics without full retraining. The policy's state-action mapping is entangled with the environment dynamics it was trained on. A changed environment requires relearning the entire mapping.

Several recent diffusion-based policy methods — Diffusion Policy (Chi et al., 2023), Diffusion BC (Pearce et al., 2023) — improve the expressiveness of the policy (capturing multimodal action distributions) but share these same fundamental inflexibilities.

Model-Based Dyna-Style Methods: Learn a Policy from a Model, but Still Inflexible

Dyna-style methods (Sutton, 1991) learn a dynamics model and then use it to generate additional training data for a policy, either online or offline. This improves sample efficiency — the model can simulate experiences beyond what's in the dataset. Representative methods include MOPO (Yu et al., 2020), MOReL (Kidambi et al., 2020), COMBO (Yu et al., 2021), RAMBO-RL (Rigter et al., 2022), and Dreamer (Hafner et al., 2020). Recent work has explored using diffusion models for the dynamics component in this paradigm: "Diffusion for World Modeling" (Alonso et al., 2024), UniSim (Yang et al., 2024), SynthER (Lu et al., 2024), Diffusion World Model (Ding et al., 2024), PolyGRAD (Rigter et al., 2024), and Policy-Guided Diffusion (Jackson et al., 2024).

However, these methods all share a critical weakness: the model is used only during training, not at runtime. The final product is still a fixed policy, with all the inflexibility that entails. As Table 1 shows, Dyna methods cannot do run-time planning, cannot optimize novel rewards at test time, and (with the exception of some recent multi-step diffusion dynamics models) may be limited in their ability to adapt to novel dynamics. They trade away the key advantage of having an explicit world model — the ability to reason about new situations at deployment time.

Joint Trajectory Models with Classifier Guidance: Planning But No Adaptation

A line of work starting with Diffuser (Janner et al., 2022), Decision Transformer (Chen et al., 2021), and Trajectory Transformer (Janner et al., 2021) takes a different approach: learn a joint distribution over entire trajectories of states and actions, $p_j(s_{1:T}, a_{1:T})$, from offline data. At test time, these methods can generate trajectories that satisfy certain conditions — Diffuser uses classifier guidance to steer generation toward high-reward trajectories, and Decision Diffuser (Ajay et al., 2023) learns a distribution over states and uses an inverse dynamics model to recover actions.

These methods can do run-time planning and can optimize novel rewards (Diffuser) or specified goals. However, they have a fundamental architectural limitation: the state and action distributions are entangled in a single joint model. This means:

• If the environment dynamics change, the entire model becomes invalid. There is no way to independently update just the dynamics component while keeping the behavioral prior intact.
• The joint representation may not factorize cleanly into "what the world does" versus "what the agent might do," making it harder to recombine them flexibly for new tasks.

The paper demonstrates this concretely in Section 4.3: when a simulated motor defect is introduced, fine-tuning the full Diffuser model on play data from the defective environment causes performance to collapse from 25.85 to 6.8 (Table 3a). Because Diffuser cannot separately update its understanding of dynamics, the fine-tuning process corrupts the behavioral knowledge that was previously learned.

Factorization-Based MPC: MBOP's Limitations

The closest predecessor to D-MPC is MBOP (Model-Based Offline Planning; Argenson & Dulac-Arnold, 2021), which the paper explicitly cites as its "original inspiration" (Section 2). MBOP gets the architecture right in principle: it factorizes the problem into a dynamics model $p_d(s|a)$ and an action proposal $\rho(a)$, which are learned separately and combined at planning time. This factorization is what enables adaptation to novel rewards and dynamics — the two components can be independently updated.

However, MBOP has three specific weaknesses that D-MPC addresses:

First, MBOP uses single-step models. Both its dynamics model and action proposal are one-step: they predict the next state given the current state and action, and the next action given the current state. To plan over a horizon $F$, MBOP must compose these autoregressively, applying the single-step models $F$ times in sequence. This introduces the well-known compounding error problem (Venkatraman et al., 2015; Asadi et al., 2019; Xiao et al., 2019; Lambert et al., 2022): small prediction errors at each step accumulate, so the imagined trajectory diverges rapidly from reality. As the paper notes in Section 1:

"To avoid compounding errors, multi-step models are preferable. However, these require a model class capable of capturing the complex, multimodal distribution of entire trajectories. This motivates our use of diffusion models."

Second, MBOP uses deterministic MLPs with ensembles to approximate stochasticity. MBOP trains an ensemble of deterministic MLPs for both dynamics and action proposals, and uses the ensemble variance as a proxy for stochasticity. But this is a crude approximation — the true distribution of possible next states or actions in a given situation may be complex and multimodal (e.g., an agent could go left or right around an obstacle), and an ensemble of deterministic functions does not faithfully capture this structure. Diffusion models, by contrast, are designed to model complex, high-dimensional, multimodal distributions.

Third, MBOP uses a relatively complex trajectory optimization (TrajOpt) planner based on the MPPI (Model Predictive Path Integral) algorithm. This planner iteratively refines action sequences using a combination of noise injection and reward-weighted averaging. While principled, this method adds algorithmic complexity and hyperparameters (temperature $\kappa$, noise variance $\sigma^2$, number of iterations). The D-MPC paper finds that a much simpler sampling-based planner — just draw $N$ candidate action sequences from the learned proposal, simulate them through the dynamics model, score them, and pick the best — actually outperforms MBOP's TrajOpt when combined with better underlying models (Table 3b: the full D-MPC achieves 65.98 vs. MBOP's 33.13 on locomotion tasks).

The Challenge of Expressive Multi-Step Models

The paper's central technical insight is that multi-step trajectory-level modeling is the key to avoiding compounding errors, but that effectively modeling joint distributions over long sequences of states and actions requires a model class with sufficient expressiveness. Simple approaches fail:

• Multi-step MLPs are a natural first attempt: flatten the entire future state sequence into a single vector and regress from current state and action sequence. But the paper's ablation (Section 4.4.4) shows this performs poorly — replacing D-MPC's diffusion models with multi-step MLPs drops the average locomotion score from 65.98 to 50.01. The reason is that the joint distribution over a 32-step trajectory is highly complex and multimodal, and a deterministic MLP trained with MSE loss collapses to predicting the mean, which may not correspond to any physically realizable trajectory.

• Autoregressive Transformers (ART) are a more sophisticated approach: model the trajectory using the chain rule, $p(s_{t+1:t+F}) = \prod_{i=t}^{t+F-1} p(s_{i+1} | s_{1:i}, a_{1:i})$, where each conditional is predicted by a causal transformer that can attend to all past states and actions. This avoids the Markov assumption and can capture long-range dependencies. However, the paper finds (Section 4.4.3, Figure 2) that while ART outperforms single-step models, it still accumulates more prediction error than the multi-step diffusion model over long horizons. The authors hypothesize that this is because:

"the diffusion dynamics model is an 'a-causal' joint distribution that goes from noise to clean trajectories, rather than working left to right. We conjecture that this enables diffusion to capture global properties of a signal (e.g., predicting if the final state corresponds to the robot falling over) in a more faithful way than a causal-in-time model."

In other words, an autoregressive model must commit to early predictions before it "knows" how the trajectory will end, which can lead to error propagation. A diffusion model generates the entire trajectory simultaneously (from noise, through iterative denoising), which allows global coherence constraints — e.g., the final state should be physically consistent with the initial state and actions — to influence all parts of the generated sequence simultaneously.

How This Paper Positions Itself

D-MPC occupies a unique position in the design space that the paper argues is both conceptually elegant and empirically powerful. As Table 1 makes clear, D-MPC is the only method that combines:

• Factorized dynamics and action proposal (enabling independent adaptation)
• Multi-step diffusion models (avoiding compounding errors via joint trajectory-level modeling)
• Run-time MPC planning (enabling novel reward optimization at test time)

The paper explicitly compares D-MPC to its closest relatives to establish its position:

• Versus Diffuser: Both use diffusion for trajectory-level modeling and can optimize novel rewards at test time. But Diffuser uses a joint $p(s, a)$ model, while D-MPC factorizes into $p_d(s|a) \rho(a)$. This factorization is what enables D-MPC to adapt to novel dynamics by fine-tuning only the dynamics component (Section 4.3), where Diffuser's joint model cannot be partially updated without corrupting behavioral knowledge.

• Versus MBOP: Both use factorized dynamics and action proposals for MPC. But MBOP uses single-step models (suffering from compounding errors) with deterministic MLP ensembles, while D-MPC uses multi-step diffusion models that directly generate entire trajectory chunks. The performance gap is dramatic: 65.98 vs. 33.13 average locomotion score.

• Versus Decision Diffuser (Ajay et al., 2023): Decision Diffuser learns a distribution over state sequences and uses a separately trained inverse dynamics model to recover actions. While this factorization might seem similar to D-MPC's, the crucial difference is that Decision Diffuser cannot accept arbitrary reward functions at test time — it uses classifier-free guidance conditioned on a reward value that must be specified in terms seen during training. D-MPC's sampling-based planner with an explicit objective function $J$ allows truly novel reward specification.

• Versus TD-MPC (Hansen et al., 2022, 2023): The TD-MPC line of work also uses MPC with learned models, but differs in two key ways: (1) TD-MPC uses single-step MLP models, not multi-step diffusion, and (2) TD-MPC focuses on online learning with environment interaction, while D-MPC focuses on learning from offline data.

The paper does not claim D-MPC is universally superior — it acknowledges that the method is slower than reactive policies (Section 5, Appendix J) and that its performance depends on the quality and coverage of the offline dataset (Section 5). But it argues that for the specific use case of flexible, adaptable agents that can optimize novel rewards and adapt to changing dynamics, the tradeoff is worthwhile, and that D-MPC represents a significant advance over prior factorized MPC approaches by bringing the expressive power of multi-step diffusion models to both sides of the factorization.

3. Technical Approach

3.1 Reader Orientation

D-MPC is a complete pipeline for training an agent that can plan its actions by simulating future state-action trajectories, where both the "what might happen next" model (dynamics) and the "what actions are sensible" model (action proposal) are learned from offline data using diffusion models and combined at runtime through a sampling-based model predictive control loop. The problem it solves is how to build an agent that can (a) avoid the compounding prediction errors that plague single-step models when planning over long horizons, (b) optimize reward functions that were never seen during training, and (c) adapt to changes in the environment dynamics without retraining everything from scratch. The shape of the solution is a factorized architecture: learn two separate multi-step diffusion models offline (one for dynamics conditioned on actions, one for actions conditioned on state and history), then at each decision step use the action proposal to generate candidate action sequences, simulate their consequences through the dynamics model, score the resulting imagined trajectories with an objective function, pick the best, execute its first action, and replan.

3.2 Big-Picture Architecture (Diagram in Words)

The system has four major components, trained offline and then deployed in an online control loop:

1. Multi-step diffusion dynamics model $p_d(s_{t+1:t+F} \mid s_t, h_t, a_{t:t+F-1})$ — takes the current state $s_t$, a history window $h_t$ of past states and actions (length $H$), and a proposed future action sequence of length $F$, and generates the corresponding future state trajectory of length $F$ as a joint distribution over all future states simultaneously (not autoregressively).

2. Multi-step diffusion action proposal $\rho(a_{t:t+F-1} \mid s_t, h_t)$ — takes the current state and history and generates candidate action sequences of length $F$, capturing the multimodal distribution of plausible behaviors present in the offline training data.

3. Objective function (value model) $J(s_{t:t+F}, a_{t:t+F-1})$ — a learned transformer that maps an imagined (state, action) trajectory to a scalar estimate of the discounted future return from that trajectory's start. This is what the planner maximizes.

4. Sampling-based planner (Algorithm 2) — at each environment step, draws $N$ action sequences from the proposal, rolls them out through the dynamics model, scores each with $J$, selects the highest-scoring sequence, returns its first action, and discards the rest. The process repeats from the next state.

The information flow at runtime is: current state + history → action proposal generates $N$ candidate action sequences → dynamics model generates $N$ corresponding state sequences → objective function scores each (state, action) pair → planner selects best sequence → execute first action in environment → observe next state → append to history → repeat.

Training happens entirely offline on a fixed dataset of (state, action, reward) trajectories. The dynamics model learns to predict future states from the dataset. The action proposal learns the behavioral distribution from the same data. The value function learns to predict discounted returns. Once trained, these components are frozen during deployment (unless adaptation is needed, in which case only the dynamics model is fine-tuned).

3.3 Roadmap for the Deep Dive

• First, the core MPC optimization problem (Equation 1) and how D-MPC approximates its solution — this establishes what the system is trying to compute and why a factorized approach with a learned action proposal makes sense.
• Second, the multi-step diffusion dynamics model — how it is trained, what distribution it learns, the denoising architecture, and crucially, WHY modeling the full trajectory as a joint distribution rather than autoregressively avoids compounding errors.
• Third, the multi-step diffusion action proposal — its training procedure, what distribution it captures, and how it serves as an informed prior that dramatically reduces the search space for the planner compared to random shooting.
• Fourth, the objective function $J$ — how it is learned via transformer regression on discounted returns, what it encodes (reward-to-go plus terminal value), and how it enables novel reward specification at test time.
• Fifth, the sampling-based planner (Algorithm 2) — the concrete algorithm, its hyperparameters ($N=64$, $F=32$, $H=1$), and why this simple approach outperforms more complex trajectory optimization methods when paired with multi-step diffusion proposals.
• Sixth, the adaptation mechanisms — how the factorized design enables (a) novel reward optimization by replacing or augmenting $J$ with new objective terms, and (b) dynamics adaptation by fine-tuning only $p_d$ on small amounts of play data from the changed environment.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and methods paper whose core idea is that factorizing trajectory-level modeling into separate multi-step diffusion models for dynamics and action proposals, then combining them in a simple sampling-based MPC loop, yields a flexible agent that avoids compounding errors, adapts to novel rewards and dynamics, and outperforms both prior factorized MPC methods (MBOP) and joint trajectory modeling approaches (Diffuser) on standard offline RL benchmarks.

The MPC Optimization Problem and D-MPC's Approximation

The paper begins by stating the standard MPC objective (Equation 1). At each timestep $t$, given the current state $s_t$ and history $h_t \equiv \{s_{1:t-1}, a_{1:t-1}\}$, the agent must select an action sequence of length $F$ (the forecast horizon) that maximizes the expected cumulative reward over the imagined future:

$a_{t:t+F-1} = \arg\max_{a_{t:t+F-1}} \mathbb{E}_{p_d(s_{t+1:t+F} \mid s_t, a_{t:t+F-1}, h_t)} \left[ J(a_{t:t+F-1}, s_{t+1:t+F}) \right]$

where $p_d$ is the (unknown) true environment dynamics, $J$ is a user-specified objective function that encodes the desired task, and the expectation is over the stochasticity in how the environment evolves given the actions.

What it computes: given a model of the world and a way to score trajectories, find the first $F$ actions that, when simulated through the model, yield the highest expected score. Only the first action $a_t$ is actually executed; after observing the resulting next state $s_{t+1}$, the optimization is repeated (hence "receding horizon").

Why this form: this decomposition separates the modeling problem (learning $p_d$, which is about predicting consequences) from the task specification problem (defining $J$, which is about expressing what we want). This means $J$ can be changed at test time without retraining the dynamics model, and conversely, the dynamics model can be updated if the environment changes without altering the task specification machinery.

The key challenge is that Equation 1 involves two difficult sub-problems: (1) the dynamics model $p_d$ must be accurate over the full horizon $F$ to avoid the expectation being computed over physically impossible futures, and (2) the optimization over continuous action sequences is a high-dimensional search problem ($F \times \text{dim}(a)$ dimensions) that cannot be solved by brute force. D-MPC addresses both simultaneously: the multi-step diffusion dynamics model provides accurate long-horizon predictions by directly modeling the joint distribution over future states (avoiding autoregressive error accumulation), and the multi-step diffusion action proposal provides an informed prior over action sequences (narrowing the search to plausible behaviors rather than uniform random exploration).

The paper does not attempt to solve the $\arg\max$ exactly. Instead, it approximates the optimization through Algorithm 2: sample $N$ action sequences from the learned proposal $\rho$, simulate each through the learned dynamics $p_d$, score with $J$, and take the best. This is essentially random shooting with a learned prior — the proposal replaces the role that white noise plays in standard random shooting methods.

The Multi-Step Diffusion Dynamics Model

The dynamics model is the component that answers the question: "if I take these actions from this state, what will happen?" Formally, it learns the conditional distribution $p_d(s_{t+1:t+F} \mid s_t, h_t, a_{t:t+F-1})$, which is the joint distribution over the next $F$ states given the current state $s_t$, a history context $h_t$ of length $H$ (containing the previous $H$ states and actions, i.e., $h_t = (s_{t-H:t-1}, a_{t-H:t-1})$), and a proposed action sequence of length $F$.

Training data construction (Algorithm 4). The offline dataset $\mathcal{D}$ consists of complete trajectories of (state, action, reward) triples. To train the multi-step dynamics model, the authors create a new dataset $\mathcal{D}'$ by extracting tuples of the form $(s_t, h_t, a_{t:t+F-1}, s_{t+1:t+F}, r_t, G_t)$, where:

• $s_t$ is the state at time $t$
• $h_t$ is the history of length $H$ ending at time $t$, defined as $(s_{t-H:t-1}, a_{t-H:t-1})$
• $a_{t:t+F-1}$ is the sequence of $F$ actions starting at time $t$ (the conditioning input)
• $s_{t+1:t+F}$ is the sequence of $F$ future states starting at $t+1$ (the prediction target)
• $G_t = \sum_{j=t}^{T} r_j$ is the undiscounted return-to-go from timestep $t$ (used for training the value function, not the dynamics)

The $r_t$ component is included for completeness but is not used in dynamics training — the dynamics model only learns the state transition mapping.

What the model actually predicts. In the diffusion framework, the dynamics model is a conditional diffusion model where:

• Clean data $x_0$ is the future state sequence $s_{t+1:t+F}$ — a tensor of shape (or rather, a flattened representation of) $F \times \text{dim}(s)$
• Conditioning variable $y$ is the tuple $(s_t, h_t, a_{t:t+F-1})$

The forward diffusion process gradually corrupts $x_0$ by adding Gaussian noise over $K$ steps, producing a sequence $x_1, x_2, ..., x_K$ where $x_K \sim \mathcal{N}(0, I)$. The reverse process learns to denoise: starting from pure noise $x_K$ and conditioned on $y$, the model iteratively predicts $x_{K-1}, x_{K-2}, ..., x_0$, recovering a clean future state trajectory that is consistent with the conditioning (current state, history, and action sequence).

Denoising score matching loss. The training objective is the standard diffusion loss. For a randomly sampled training tuple, a random diffusion timestep $k$ is selected, noise $\epsilon_k$ is sampled with variance appropriate for step $k$, and the noise prediction network $\epsilon_\theta$ is trained to predict the added noise:

$\mathcal{L}_{\text{dynamics}} = \text{MSE}\left(\epsilon_k, \epsilon_\theta(x_0 + \epsilon_k, k, y)\right)$

where $\epsilon_k$ is the actual noise added to the clean data, $\epsilon_\theta(x_0 + \epsilon_k, k, y)$ is the model's prediction of that noise given the noisy input and conditioning, and MSE is mean squared error. Minimizing this loss teaches the model to reverse the diffusion process — i.e., to map noise to plausible future state trajectories conditioned on the current situation and proposed actions.

Architecture: transformer denoiser. The noise prediction network $\epsilon_\theta$ is implemented as a transformer (specific hyperparameters described in Appendix E.1). The architecture processes inputs as follows:

1. Diffusion timestep embedding: the scalar $k$ is embedded using sinusoidal embeddings
2. Tokenization: the time embedding, each state and action (both conditioning variables and the target states being denoised) are projected to a shared token space of dimension 256 via learned linear projections
3. Positional encoding: Fourier positional embeddings with 16 Fourier bases are added to all tokens
4. Transformer layers: all tokens pass through 5 transformer layers, each with multi-headed attention (8 heads, 1024 dimensions for query/key/value, 2048 hidden dimensions for the MLP)
5. Output projection: the output tokens corresponding to the states being predicted are projected back to the original state dimension

The multi-step dynamics model uses 10 diffusion timesteps and 5 transformer layers for the denoiser. All other architectural details (normalization, activation functions, etc.) are provided in Appendix E.

What "multi-step" means and why it matters. The crucial property of this model is that it generates the entire future state sequence $s_{t+1:t+F}$ as a single joint sample, not as an autoregressive chain. This means the model does not generate $s_{t+1}$, then feed it as input to generate $s_{t+2}$, etc. Instead, it starts from pure noise (over the full trajectory length) and denoises all timesteps simultaneously, with the denoising process able to enforce global consistency — e.g., ensuring that the final state is physically reachable from the initial state given the action sequence.

This is fundamentally different from:

• Single-step models (as in MBOP), which model $p(s_{t+1} \mid s_t, h_t, a_t)$ and must be composed $F$ times, accumulating errors multiplicatively
• Autoregressive transformers (ART), which model $\prod_{i=t}^{t+F-1} p(s_{i+1} \mid s_{1:i}, a_{1:i})$ — while these can attend to all past states and actions, they still generate left-to-right and must commit to early predictions before seeing how the trajectory develops

The paper's empirical evidence for the advantage of joint generation is Figure 2, where the multi-step diffusion dynamics model shows substantially lower prediction error over long horizons compared to both single-step models (MLP and diffusion) and the autoregressive transformer, particularly on in-distribution data (the "medium" training data) while also generalizing better to out-of-distribution data ("expert" and "medium-replay").

History conditioning. The model conditions on a history window of length $H$ (set to 1 in all experiments). Even with $H=1$, the model receives the previous state and action as context. For domains where the dynamics are higher-order (e.g., velocity-dependent processes where $s_t$ alone is not a sufficient statistic), increasing $H$ would provide the necessary context while still maintaining the multi-step joint generation property.

State normalization. Following Ajay et al. (2023), state coordinates are normalized before being input to the model. For each coordinate of the state vector, the empirical cumulative distribution function (CDF) is computed from the offline dataset, and state values are mapped through their respective CDFs to lie uniformly in $[-1, 1]$. Concretely, for coordinate $k$, the normalization is $\hat{s}_k = 2\hat{F}_k(s_k) - 1$, where $\hat{F}_k$ is the empirical CDF. The dynamics model's output states are unnormalized via the inverse transformation before being used in the planner. This normalization ensures that all state dimensions have comparable scale, which is important for stable diffusion training.

Training hyperparameters and compute. All models are trained with the Adam optimizer, with a learning rate schedule that warms up from 0 to $10^{-4}$ over 500 steps and then follows a cosine decay to $10^{-5}$. Training runs for $2 \times 10^6$ steps. Gradient clipping is applied at norm 5, and exponential moving average (EMA) with decay factor 0.99 is used for evaluation parameters. Training uses DDIM (Song et al., 2020) with a cosine noise schedule (Nichol & Dhariwal, 2021). On a single A100 GPU, training for $2 \times 10^6$ steps takes approximately 2 days.

The Multi-Step Diffusion Action Proposal

The action proposal model answers the question: "given where I am and what I've done recently, what are some plausible sequences of actions I could take?" Formally, it learns the conditional distribution $\rho(a_{t:t+F-1} \mid s_t, h_t)$, which is the joint distribution over the next $F$ actions given the current state and history.

Training data construction (Algorithm 4). The same dataset construction used for the dynamics model applies here, but with a different prediction target. From each tuple in $\mathcal{D}'$, the action proposal uses:

• Conditioning variable $y$: the current state $s_t$ and history $h_t$
• Clean data $x_0$: the future action sequence $a_{t:t+F-1}$

The training objective is again denoising score matching:

$\mathcal{L}_{\text{proposal}} = \text{MSE}\left(\epsilon_k, \epsilon_\theta(x_0 + \epsilon_k, k, y)\right)$

where now $x_0$ is the action sequence and $y$ is the state and history.

Architecture. The action proposal uses a transformer denoiser with the same structural design as the dynamics model, but with different hyperparameters: 32 diffusion timesteps (versus 10 for dynamics) and 5 transformer layers. The larger number of diffusion steps reflects the greater complexity of the action distribution — actions can be multimodal in ways that states typically are not (from the same state, an agent might turn left or right, accelerate or brake).

What distribution this captures. The action proposal learns the behavioral distribution present in the offline dataset. This is not an optimal policy — it is a prior over what actions the data-generating agents tended to take. If the dataset contains suboptimal or exploratory behavior, those action patterns are included in the proposal distribution. This is both a strength and a limitation: the proposal provides a rich, multimodal prior that covers many plausible behaviors (making the planner's search more efficient than sampling uniformly), but it may not contain actions that are optimal for novel reward functions that differ significantly from the training objectives.

The proposal's role in the overall system is to narrow the search space for the planner. Rather than searching over the full continuous action space (which would require an impractically large $N$ to cover effectively), the planner only considers action sequences that are "in the support" of the behavioral distribution. For tasks that are reasonably similar to the training data (even if the reward function is different), this is sufficient — the needed actions exist in the dataset, just perhaps not in the right combinations or timings. The planner's job is to identify which of the proposed sequences best satisfies the (potentially novel) objective.

Single-step vs. multi-step proposals (Section 4.4.2). The paper's ablation studies demonstrate that generating the full action sequence jointly (multi-step) outperforms generating actions one at a time (single-step). When replacing the multi-step diffusion action proposal with a single-step diffusion proposal (Table 4), the average locomotion score drops from 65.98 to 53.32 (with single-step diffusion dynamics) or 52.93 (with single-step MLP dynamics). The interpretation is that multi-step generation allows the proposal to capture temporal coherence — actions across the horizon are coordinated to produce a coherent behavioral sequence (e.g., a gait pattern) rather than being independent draws that may conflict with each other.

Why diffusion for action proposals? The paper's ablation in Section 4.4.4 shows that replacing the diffusion action proposal with a multi-step MLP (which outputs a deterministic sequence via MSE regression) drops performance from 65.98 to 50.01. This is because MSE-trained deterministic models collapse to the mean of the behavioral distribution, which is inappropriate when the distribution is multimodal — the "average" of turning left and turning right might be going straight, which is not a valid action. Diffusion models, by learning to sample from the full distribution, preserve multimodality.

The Objective Function (Value Model) $J$

The objective function evaluates imagined trajectories and provides the signal that the planner optimizes. Formally, for a given (state, action) trajectory of length $F$, it computes:

$J(s_{t:t+F}, a_{t:t+F-1}) = \mathbb{E}\left[\sum_{k=t}^{t+F-1} \gamma^{k-t} R(s_k, a_k) + \gamma^F V(s_{t+F})\right]$

where $\gamma$ is the discount factor, $R(s_k, a_k)$ is the reward at step $k$, and $V(s_{t+F})$ is a value function estimate of the expected future return from the terminal state of the imagined trajectory. The expectation $\mathbb{E}[\cdot]$ is over any stochasticity in the reward or value estimation.

What it computes: the discounted sum of (a) the rewards that would be received over the next $F$ steps if the proposed action sequence were executed, plus (b) the discounted value of whatever state the agent would end up in after those $F$ steps. The value term $V(s_{t+F})$ acts as a terminal cost or heuristic that provides credit for long-term consequences beyond the planning horizon. Without this term, the planner would be myopic, favoring action sequences that produce high immediate reward even if they lead to failure shortly beyond the horizon (e.g., an agent might sprint forward for high velocity reward and then fall over at step $F+1$, but without $V$ this consequence is invisible).

Why this form: this is the standard finite-horizon approximation to the infinite-horizon return used in MPC. The value function $V$ summarizes the long-term future concisely, making the finite-horizon optimization approximately equivalent to the full problem. If $V$ were perfect, the finite-horizon plan would be exactly optimal. In practice, $V$ is learned and imperfect, but still provides useful guidance.

Training the value function. The paper trains a transformer to directly predict the discounted future return. The training data consists of tuples $(s_{t:t+F}, a_{t:t+F-1}, G_t)$ extracted from the offline dataset, where $G_t = \sum_{j=t}^{T} r_j$ is the undiscounted return-to-go. The model is trained with an L2 regression loss to predict $G_t$ (or the discounted variant, though the paper's description simplifies to undiscounted for training and applies discounting in the $J$ formula at runtime).

Architecture. The value function uses a transformer with the same base layer design as the denoisers (Appendix E.1): multi-headed attention with 8 heads, 1024 dimensions, 2048 hidden dimensions. States and actions from the trajectory are projected to a shared token space of dimension 256, along with an additional learnable token designated for the return prediction. Fourier positional embeddings with 16 bases are added, and all tokens pass through 10 transformer layers. The output token corresponding to the return prediction is read out through a linear projection to produce the scalar return estimate.

Discount factors. For most domains, the discount factor is $\gamma = 0.99$. For Hopper specifically, $\gamma = 0.997$ is used. The paper notes (Appendix E.4) that Hopper and Walker2D episodes can terminate early due to the agent falling over, and in these cases an additional termination penalty of $-100$ is included as the reward at the last step, with the discounted return calculation incorporating this penalty.

Role in the planner. In Algorithm 2, $J$ is computed for each of the $N$ sampled action sequences and their predicted state trajectories. The planner selects $\hat{n} = \arg\max_n V_n$, where $V_n = J(s_{1:F}, A_{n, 1:F})$ is the scalar score for the $n$-th trajectory. This scoring and ranking step is the entire optimization mechanism — there is no gradient-based refinement, no iterative resampling, no temperature-based weighting. The planner simply picks the best-scoring trajectory from the $N$ candidates.

Novel reward composition. A key feature of D-MPC is that $J$ can be augmented or replaced at test time. The paper demonstrates (Section 4.2, Appendix H) that for a novel reward function $f_{\text{novel}}(s_t, A_t)$, the planner's scoring can be modified to:

$V_n = \kappa J(s_{1:F}, A_{n,1:F}) + \tilde{\kappa} \tilde{J}(s_{1:F}, A_{n,1:F})$

where $\tilde{J}(s_{1:F}, A_{n,1:F}) = \frac{1}{F}\sum_{t=1}^F f_{\text{novel}}(s_t, A_{n,t})$ is the average novel reward over the imagined trajectory, and $\kappa, \tilde{\kappa}$ are weights controlling the tradeoff between the original and novel objectives. By setting $\kappa=0, \tilde{\kappa}=1$, the agent pursues only the novel objective. The Walker2D examples in Figure 1 use a height-based novel reward $f_{\text{novel}}(s_t, A_t) = 5 \exp(-(h_t - h_{\text{target}})^2 / 2\sigma^2)$ with $\sigma^2 = 5 \times 10^{-4}$, where the scale factor of 5 is chosen to roughly match the maximum environment reward. This produces the qualitatively distinct behaviors (lunging, balancing, jumping) without any retraining of the dynamics or proposal models.

The Sampling-Based Planner (Algorithm 2)

The planner is the runtime component that combines the three learned models to select actions. The pseudocode from Algorithm 2 is:

1  Def a = Planner(s0, h0, p_d, ρ, J, N, F, H):
2    for n = 1 : N do
3      a_{n,1:F} ∼ ρ(· | s0, h0)
4      s_{1:F} ∼ p_d(· | s0, h0, a_{n,1:F})
5      V_n = J(s_{1:F}, a_{n,1:F})
6    n_hat = arg max_n V_n
7    Return a_{n_hat, 1}

Step-by-step operation. At each environment step:

1. Sample action sequences (line 3): the frozen action proposal $\rho$ is sampled $N$ times from the current state $s_0$ and history $h_0$. Each sample is a full action sequence of length $F$, drawn jointly (all $F$ actions are generated together through the diffusion denoising process). Because the proposal is stochastic (diffusion sampling is inherently noisy), the $N$ samples are diverse, covering different behavioral modes.

2. Predict state sequences (line 4): for each action sequence, the frozen dynamics model $p_d$ is sampled to generate the corresponding state trajectory of length $F$. Again, all $F$ states are generated jointly through a single diffusion denoising process, ensuring global physical consistency.

3. Score trajectories (line 5): each (state, action) pair is evaluated by the objective function $J$, producing a scalar $V_n$.

4. Select best (lines 6-7): the trajectory with the highest score is chosen ($\hat{n} = \arg\max_n V_n$), and only its first action $a_{\hat{n}, 1}$ is returned for execution. The remaining $F-1$ actions are discarded.

5. Replan: after executing $a_{\hat{n}, 1}$ in the environment and observing the next state and reward, the entire process repeats (Algorithm 1, the main MPC loop). The history $h_t$ is updated with the new state, action, and reward, and truncated to length $H$.

Hyperparameter values. The paper uses:

• Forecast horizon $F = 32$. The authors note (Appendix F) that this "already works well since our trained objective function $J$ predicts discounted future rewards." In other words, because the value function handles long-term credit beyond the horizon, $F$ only needs to be long enough for the action sequence to meaningfully affect the near-term trajectory; the value bootstrap covers the rest.
• Number of samples $N = 64$. This is the number of candidate action sequences drawn per planning step. Empirically, this provides sufficient coverage of the behavioral distribution to find good trajectories for the tasks considered.
• History length $H = 1$. The model conditions on the immediately previous state and action.

Why this simple planner works. The paper argues (Section 3.3) that this sampling-based approach outperforms more complex methods like MBOP's TrajOpt (Algorithm 5 in the Appendix) "because the diffusion model already reasons at the trajectory level, and can natively generate a diverse set of plausible candidates without the need for additional machinery." The intuition is:

• The action proposal is already a strong prior — it generates action sequences that are coherent, temporally extended behaviors, not random perturbations around a nominal trajectory. This means even with only $N=64$ samples, the planner explores a meaningful region of the behavioral space.
• Trajectory optimization methods like TrajOpt (based on MPPI) iteratively refine action sequences by adding noise and reward-weighting the results. But if the initial proposal distribution is poor (as with single-step MLPs), many iterations may be needed to find good actions. With a multi-step diffusion proposal, the initial samples are already high-quality, so simple ranking suffices.
• Avoiding iterative refinement simplifies the algorithm, removes hyperparameters (TrajOpt requires a temperature $\kappa$, noise variance $\sigma^2$, and an iteration budget), and is more computationally predictable (fixed $N$ diffusion sampling steps vs. variable iterations until convergence).

Computational cost. The planner's cost per environment step is dominated by sampling from two diffusion models ($\rho$ and $p_d$) $N$ times each, plus $N$ forward passes through the value function transformer. With $N=64$, $F=32$, and the diffusion step counts (32 for proposal, 10 for dynamics), this is computationally intensive. Appendix J reports wall-clock planning times of approximately 92-95 milliseconds per step on a single A100 GPU across the three locomotion domains (Table 7). This is roughly 20× slower than MBOP (~4-5 ms) and ~250× slower than a simple MLP policy (~0.35 ms). However, as the paper notes in Section 5, this cost can be amortized by executing longer action chunks between replanning steps, using accelerated diffusion sampling methods, or distilling the planner into a fast policy (Section 4.5).

The Main MPC Loop (Algorithm 1)

The outer loop that invokes the planner at each step is specified in Algorithm 1:

1. Offline training (line 2): $(p_d, \rho, J) = \text{train}(\mathcal{D})$ — all three models are trained on the offline dataset before any environment interaction begins.
2. Environment initialization (lines 3-4): the environment is reset to an initial state $s_0$, and the history $h_0$ is initialized to contain only $s_0$.
3. Main loop (lines 5-9): at each timestep $t$, the planner is called to produce action $a_t$, which is executed in the environment to obtain the next state $s_{t+1}$ and reward $r_{t+1}$. The history is updated by appending $(a_t, s_{t+1}, r_{t+1})$ and then truncated to the most recent $H$ entries via a suffix operation ($h_t = \text{suffix}(h_t, H)$). The loop continues indefinitely (or until episode termination).

Connection to the broader MPC framework. The loop implements receding horizon control: at each step, the agent plans a full sequence of $F$ actions, but only executes the first one, then replans from the new state. This closed-loop behavior provides robustness to model errors — if the dynamics model is imperfect, the consequences of its errors are corrected at the next planning step when fresh observations become available.

Adaptation Mechanisms

Novel reward adaptation (Section 3.4, demonstrated in Section 4.2). The factorized architecture means the planner's objective can be changed without touching any of the learned models. The paper demonstrates this by replacing the trained value function $J$ with a hand-designed novel reward $\tilde{J}$ based on the Walker2D agent's height (Figure 1). The implementation is straightforward:

• At line 5 of Algorithm 2, instead of $V_n = J(s_{1:F}, A_{n,1:F})$, use $V_n = \tilde{J}(s_{1:F}, A_{n,1:F})$ where $\tilde{J}$ computes the average of $f_{\text{novel}}$ over the trajectory.
• The action proposal and dynamics model remain unchanged — they were trained on the original dataset and have no knowledge of the height-based objective, but the proposal still generates walking/running behaviors, and the planner selects among them based on which ones achieve the desired height.

The novel reward function used is $f_{\text{novel}}(s_t, A_t) = 5 \exp(-(h_t - h_{\text{target}})^2 / 2\sigma^2)$ with $\sigma^2 = 5 \times 10^{-4}$, $\kappa=0$, $\tilde{\kappa}=1$. The target heights tested are 0.9 (lunge), 1.2 (balance), and 1.4 (jump). The scale factor 5 ensures the novel reward magnitude is comparable to the environment's native reward scale.

Dynamics adaptation (Section 3.4, demonstrated in Section 4.3). When the environment dynamics change (simulated by clipping the torque on a foot joint to $[-0.5, 0.5]$ instead of $[-1, 1]$), the dynamics model $p_d$ becomes inaccurate, causing performance to degrade. Because the dynamics and proposal are factorized, only $p_d$ needs to be updated. The procedure is:

1. Collect play data: deploy the original D-MPC (trained on the medium-replay dataset) in the defective environment for 100 episodes, collecting 30,170 transitions with an average normalized reward of 23.14 (±2.31). The actions recorded are the model outputs (not the clipped executed actions).
2. Fine-tune dynamics model: load the pre-trained multi-step diffusion dynamics model and continue training on this small play dataset using the same training parameters (Adam, learning rate schedule, gradient clipping, EMA). The action proposal $\rho$ and value function $J$ are frozen.
3. Deploy: use the fine-tuned dynamics with the original proposal and value function in the standard MPC loop.

Post-fine-tuning, D-MPC's normalized score on the defective Walker2D improves from 22.74 (±1.41) to 30.65 (±1.89) — a partial but meaningful recovery. In contrast, fine-tuning the full Diffuser model on the same play data causes performance to collapse from 25.85 (±1.08) to 6.8 (±0.86), because the joint $p(s, a)$ model cannot be partially updated — fine-tuning alters both the dynamics and behavioral components simultaneously, corrupting the action generation capability.

Why the factorization enables adaptation. The key insight is that the dynamics model $p_d(s \mid a)$ and the action proposal $\rho(a)$ represent independent physical facts about the world. $p_d$ encodes "how the world responds to actions," which can change (hardware defect). $\rho(a)$ encodes "what actions are sensible to take," which is independent of the defect — good behaviors (walking, running, balancing) are still good behaviors even if the robot's response is different. By keeping these separate, D-MPC can update its understanding of the world without forgetting what behaviors are worth attempting. A joint model like Diffuser entangles these, making partial updates impossible.

Summary of Design Choices and Their Justifications

• Diffusion models for both dynamics and proposal, rather than MLPs or VAEs: required to capture the complex, multimodal, high-dimensional joint distributions over long trajectories. MLPs collapse to means; deterministic models cannot represent multimodality. The ablation in Section 4.4.4 (MLP replacement yields 50.01 vs. 65.98) empirically justifies this choice.
• Multi-step joint generation rather than autoregressive: eliminates compounding errors. The left-to-right generation in autoregressive models forces early commitment and propagates errors; joint generation allows global coherence constraints. Figure 2 quantifies the advantage.
• Factorized $p_d(s \mid a) \rho(a)$ rather than joint $p_j(s, a)$: enables independent adaptation. The experiments in Section 4.3 demonstrate the practical value: dynamics can be fine-tuned without corrupting behavioral knowledge. Joint models like Diffuser cannot do this.
• Simple sampling-based planner rather than trajectory optimization: the diffusion proposal is already a strong prior generating coherent trajectory-level behaviors. Complex iterative optimization adds overhead and hyperparameters without benefit when the underlying proposal is good. The experimental comparison (D-MPC at 65.98 vs. MBOP's TrajOpt at 33.13, though conflated with other differences) supports this.
• Transformer architecture for all models: provides the capacity and flexibility to process variable-length sequences of states and actions with long-range attention, crucial for capturing temporal dependencies over the 32-step horizon.
• 10 diffusion steps for dynamics, 32 for action proposal: dynamics are typically simpler and less multimodal than action distributions (future states are largely determined by physics given actions; future actions have many valid options), so fewer denoising steps suffice for dynamics. The 32 steps for actions reflects the greater complexity of the behavioral distribution.
• Learned value function $J$ rather than computing reward directly from a reward model: the value function provides a bootstrap estimate beyond the planning horizon ($\gamma^F V(s_{t+F})$), mitigating myopia. Direct reward computation over only $F$ steps would miss long-term consequences.

4. Key Insights and Innovations

Innovation 1: Factorized Trajectory-Level Modeling as the Key Architectural Decision for Adaptable Agents

The paper's most fundamental intellectual contribution is the argument — supported by both a design-space analysis (Table 1) and empirical evidence — that the factorization of trajectory modeling into separate dynamics and action proposal components is not merely an implementation detail but the critical architectural choice that determines an agent's capacity for post-deployment adaptation. Before D-MPC, the dominant assumption in diffusion-based planning (e.g., Diffuser, Decision Diffuser) was that learning a joint distribution $p_j(s, a)$ over state-action trajectories was natural and sufficient — after all, if the goal is to generate complete trajectories, why not model them jointly? The factorization approach existed (MBOP) but was associated with weak single-step models that undermined its conceptual appeal.

D-MPC demonstrates that factorization, when paired with expressive multi-step diffusion models, unlocks capabilities that joint models structurally cannot achieve. The most striking evidence is Table 3a: when a simulated motor defect is introduced, fine-tuning Diffuser's joint model on play data causes performance to collapse (25.85 → 6.8), while D-MPC's factorized approach recovers significantly (22.74 → 30.65) by updating only the dynamics component. Joint models entangle "what the world does" with "what actions are sensible," so updating one necessarily corrupts the other. Factorization treats these as independent physical facts about the world — $p_d$ encodes the environment's response to actions, $\rho(a)$ encodes behavioral priors — that can be maintained and updated separately.

This is a reframing of the architecture problem for learned model-based control. The paper effectively argues that the right inductive bias is not "model trajectories" but "model the world and the agent separately." This reframing has implications beyond the specific implementation: any system that needs to adapt to changing dynamics while preserving behavioral knowledge should factorize its world model from its action-generating process. The paper doesn't just present this as a design choice — it provides the diagnostic comparison (joint vs. factorized adaptation) that makes the case empirically concrete, and it positions this choice within a taxonomy (Table 1) that reveals the gap in prior work.

This is a fundamental contribution, not incremental. Prior factorized approaches (MBOP) had the right structure but were hamstrung by single-step models, making them uncompetitive. Prior expressive approaches (Diffuser) had the right model class but were architecturally entangled, making them inadaptable. D-MPC's insight is that both properties — factorization AND expression — are necessary, and neither alone suffices.

Innovation 2: Joint Multi-Step Diffusion Generation as a Direct Solution to Compounding Error

Compounding error in learned dynamics models has been recognized as a central challenge in model-based RL since at least Venkatraman et al. (2015). The standard approach has been to mitigate it: train models with multi-step losses, add uncertainty estimation, or use shorter planning horizons. D-MPC's conceptual move is different: rather than mitigating compounding error, it avoids it entirely by generating the entire future state trajectory as a single joint sample from a diffusion model, rather than as an autoregressive chain.

The key distinction in mechanism (Section 3) is that the dynamics model generates $s_{t+1:t+F}$ simultaneously — the diffusion process starts from noise over the full trajectory and denoises all timesteps together, allowing global coherence constraints to influence every part of the sequence simultaneously. An autoregressive model must commit to $s_{t+1}$ before knowing what $s_{t+F}$ will be, creating a structural vulnerability to error propagation.

The evidence for why this matters comes from Figure 2. The multi-step diffusion dynamics model shows substantially lower median RMSD prediction error than both single-step models (diffusion and MLP) and the autoregressive transformer, especially on in-distribution data but also generalizing better to out-of-distribution expert data. The autoregressive transformer, despite being non-Markovian (attending to all past states and actions), still accumulates more error because of its left-to-right causal structure.

But this is not just about prediction accuracy in isolation — Section 4.4.3 shows this translates to controller performance. When the same multi-step diffusion action proposal is paired with different dynamics models inside the planning loop (Table 3b, bottom row): single-step diffusion achieves 53.32, the autoregressive transformer achieves 59.83, and the multi-step diffusion achieves 65.98. The gap between the autoregressive transformer and the diffusion model (59.83 vs. 65.98) isolates the benefit of joint generation over autoregressive generation, since both models receive the same action proposals and are evaluated in the same planning loop.

The paper's conjecture about why — "diffusion enables diffusion to capture global properties of a signal (e.g., predicting if the final state corresponds to the robot falling over) in a more faithful way than a causal-in-time model" — is plausible but not proven. Still, the pattern is clear: joint generation provides a qualitatively different error profile than autoregressive generation. This is a fundamental insight about the appropriate model class for trajectory prediction, not a refinment of existing autoregressive approaches.

Innovation 3: The Action Proposal as an Informed Prior That Eliminates the Need for Complex Optimization

A standard assumption in MPC with learned models is that planning requires sophisticated optimization — trajectory optimization (MBOP's MPPI-based TrajOpt), cross-entropy method, or gradient-based refinement — to find good action sequences in high-dimensional continuous spaces. D-MPC's counterintuitive finding is that when the action proposal is expressive enough (a multi-step diffusion model capturing the behavioral distribution), a simple sampling-and-ranking procedure with only 64 candidates (Algorithm 2) outperforms more complex optimization methods.

This is not a claim that random search is sufficient in general — it's a claim that the quality of the proposal distribution is what matters. MBOP, using single-step MLP action proposals with TrajOpt, achieves 33.13 on locomotion tasks. D-MPC, using multi-step diffusion proposals with simple sampling, achieves 65.98. The ablation in Section 4.4.1 isolates this: replacing MBOP's single-step MLP proposal with a single-step diffusion proposal while keeping the same MLP dynamics and switching from TrajOpt to the sampling planner improves performance from 33.13 to 52.93. The proposal quality, not the optimization sophistication, is the binding constraint.

The intellectual move here is a reframing of the planner's role: when the proposal model already generates coherent, trajectory-level behavioral sequences (because it was trained jointly over the full horizon, not step-by-step), the planner's job shifts from "optimize a sequence from scratch" to "select among already-plausible options." This is analogous to how language model sampling works — a strong language model generates coherent continuations, and simple ranking by a reward model suffices for selection; beam search or complex decoding strategies help only marginally compared to model quality. D-MPC imports this intuition to continuous control: the diffusion proposal generates temporally coherent action sequences (gait patterns, reaching motions), and the planner just picks the one that scores best.

This is an incremental but practically significant insight. It simplifies the algorithm (removing TrajOpt's hyperparameters $\kappa$, $\sigma^2$, iteration count), makes the computational cost more predictable (fixed $N$ samples vs. variable iterations), and redirects research attention from planner design to proposal model design. The paper shows that the same pattern holds across difficulty levels (Table 4: the multi-step diffusion proposal with any dynamics model outperforms the single-step diffusion proposal with the same dynamics model, across all six locomotion tasks).

Innovation 4: Test-Time Compute as a Substitutable Resource for Pretraining, With Sharp Boundaries Defined by Behavioral Coverage

While the FLOPs-matched comparison in the prior example paper was explicit, D-MPC's contribution to the pretraining-vs-inference tradeoff is more implicit but equally important: the paper demonstrates that test-time compute allocated through MPC planning can compensate for suboptimal training data, but only within the behavioral coverage of that data. This is a sharper, more operational version of the general claim that "planning helps."

The evidence is in the D4RL benchmark results (Table 2). D-MPC, trained on medium and medium-replay datasets (which contain suboptimal, exploratory, and mediocre behaviors), achieves scores competitive with or exceeding methods trained on the same data that use no planning (BC: 41.92 locomotion average) or use planning during training but not at runtime (CQL: 65.45, IQL: 67.47). The planning process — sampling 64 action sequences, simulating them, selecting the best — essentially performs test-time search over the behavioral repertoire, finding combinations of actions that produce high reward even if no single trajectory in the training data demonstrated exactly that behavior.

But the boundary is sharp: D-MPC cannot invent behaviors not present in the proposal distribution's support. If the offline dataset contains no examples of a particular skill, the action proposal cannot generate it, and no amount of planning will find it. The paper is explicit about this limitation (Section 5). This distinguishes planning-as-amplification (taking behaviors that exist in the data and recombining them for higher reward) from planning-as-creation (inventing entirely new behaviors). D-MPC does the former, not the latter.

This insight matters because it clarifies when to invest in better models vs. better planning. If the offline data already contains the necessary behavioral building blocks (even if suboptimally arranged), D-MPC's test-time compute can extract them. If the data lacks critical skills entirely, no amount of planning will compensate — better data collection or better behavior cloning is needed. This is a conceptual refinement of the relationship between data quality, model expressiveness, and planning, not a new algorithm.

Innovation 5: Verifier Over-Optimization as a Non-Issue When the Dynamics Model Is Multi-Step and Joint

This is a negative finding with positive implications: unlike the prior example paper, where verifier over-optimization (the PRM being exploited by aggressive search) was the central bottleneck, D-MPC demonstrates that a multi-step diffusion dynamics model does not exhibit the over-optimization pathology that plagues single-step models and autoregressive alternatives. The paper doesn't name this explicitly as an innovation, but it emerges from the combination of Figure 2 and the performance results.

Over-optimization in model-based RL typically manifests as the planner finding action sequences that the dynamics model predicts will yield high reward but are physically impossible — the model is "fooled" because its predictions are locally plausible but globally inconsistent (a single-step model might predict that the robot stays upright indefinitely because each individual step looks fine, even though the accumulated joint angles would cause a fall). D-MPC's joint generation over the full trajectory prevents this: the diffusion model must produce a globally coherent state sequence, so impossible trajectories are simply not generated.

The evidence is indirect but consistent: D-MPC does not show the performance degradation at high planning budgets that characterizes over-optimization of learned models. The planning budget is only $N=64$ samples, but this is sufficient because the proposal and dynamics are jointly coherent — adding more samples would not find "adversarial" action sequences that exploit model errors because the joint generation enforces physical consistency. This is a practical advantage of the architectural choice, not a separately verified claim, but it represents a significant difference from the single-step model-based RL literature where model exploitation is a well-documented failure mode.

5. Experimental Analysis

Evaluation Methodology

• Dataset. All experiments use the D4RL benchmark (Fu et al., 2020), a standard offline reinforcement learning suite. The paper evaluates on three locomotion domains (HalfCheetah, Hopper, Walker2D) at medium and medium-replay quality levels, two Adroit dexterous manipulation domains (pen, door, hammer) with cloned data, and the Franka Kitchen domain with mixed and partial datasets. The choice of D4RL is deliberate: planning-based approaches are argued to be especially beneficial with sub-optimal data, not expert demonstrations (Section 4.1).

• Base model(s). The paper does not use a pretrained "base model" in the LLM sense — all components (multi-step diffusion dynamics model p_d, multi-step diffusion action proposal ρ, value function J) are trained from scratch on the offline dataset for each D4RL domain and quality level. The architectures are domain-specific but structurally identical: transformer-based diffusion models with 5 layers and 32 diffusion timesteps for the action proposal and 10 diffusion timesteps for the dynamics model (Appendix E.1). The value function uses a 10-layer transformer. All models use multi-headed attention with 8 heads, 1024 dimensions for query/key/value, and 2048 hidden dimensions for the MLP.

• Metrics. The primary metric is normalized score as defined by Fu et al. (2020), which maps raw episode returns to a 0–100 scale where 0 corresponds to a random policy and 100 corresponds to an expert policy. For D-MPC, results are reported as the mean and standard error over 30 evaluation episodes with different random initial environment conditions (Table 2 caption). For D4RL locomotion tasks, additional metrics include average normalized score across the six domain/level combinations.

• Baselines. The paper compares against multiple categories of methods. Model-free offline RL: Behavior Cloning (BC), Conservative Q-Learning (CQL; Kumar et al., 2020), Implicit Q-Learning (IQL; Kostrikov et al., 2021). Model-based offline RL: MOReL (Kidambi et al., 2020). Sequence models: Decision Transformer (DT; Chen et al., 2021). Model-based offline planning: MBOP (Argenson & Dulac-Arnold, 2021)—the closest architectural predecessor. Diffusion-based planning: Diffuser (Janner et al., 2022). Baseline numbers for CQL, IQL, BC, DT, and MOReL are obtained from Ajay et al. (2023) and Tarasov et al. (2024), as noted in Table 2's caption.

• Generation budget / compute accounting. Test-time compute during planning is measured by number of action sequence samples N drawn from the proposal per planning step. The paper uses N = 64 for all main experiments (Appendix F), meaning 64 action sequences are sampled, each of length F = 32, rolled out through the dynamics model, and scored. Total compute per step is the cost of N diffusion sampling operations for the proposal (32 denoising steps each) plus N diffusion sampling operations for the dynamics model (10 denoising steps each) plus N transformer forward passes for the value function. The wall-clock cost is reported in Appendix J (Table 7): 92–95 ms per step for D-MPC versus ~4–5 ms for MBOP and ~0.35 ms for an MLP policy on a single A100 GPU.

• Cross-validation / statistical protocol. No cross-validation or train/test splitting of the offline dataset is described — the models are trained on the full D4RL dataset for each domain/quality combination, following the standard offline RL evaluation protocol where the dataset is fixed and evaluation is on the corresponding environment. Statistical results are reported as mean ± standard error over 30 evaluation episodes with different random seeds for environment initialization (Table 2). For baseline methods, the paper uses published numbers from Ajay et al. (2023) which do not report standard errors, and from Tarasov et al. (2024) which include standard errors when available.

Main Quantitative Results

Overall Performance on D4RL Benchmarks (Table 2)

The headline result is that D-MPC achieves an average locomotion score of 65.98 across the six medium and medium-replay domain/level combinations, compared to MBOP's 33.13, representing a roughly 2× improvement over the closest factorized MPC baseline. D-MPC also marginally exceeds Diffuser's 63.77 average and is competitive with the best model-free methods (IQL at 67.47, CQL at 65.45).

On individual locomotion tasks, the pattern varies:

• HalfCheetah medium: D-MPC achieves 46.00 (±0.17), slightly above MBOP's 44.60 and Diffuser's 44.20, but below IQL's 47.40. The margins are narrow here since HalfCheetah medium is a relatively easy dataset.
• Hopper medium: D-MPC achieves 61.24 (±2.30), substantially above MBOP's 48.80 but below Diffuser's 58.50, DT's 67.60, and IQL's 66.30. Hopper medium appears to be a domain where behavioral cloning baselines struggle (BC: 52.90), making model-based planning relatively more beneficial compared to BC but not compared to sequence modeling.
• Walker2D medium: D-MPC achieves 76.21 (±2.67), competitive with Diffuser's 79.70 and IQL's 78.30, and dramatically above MBOP's 41.00. This is the task where the gap between D-MPC and MBOP is largest, highlighting the benefit of multi-step over single-step modeling.
• Hopper medium-replay: D-MPC achieves 92.49 (±2.23), close to IQL's 94.70 and CQL's 95.00, and far above MBOP's 12.40. The medium-replay dataset contains a mixture of exploratory and suboptimal data, making MBOP's single-step deterministic models particularly ineffective.
• Walker2D medium-replay: D-MPC achieves 78.81 (±4.19), substantially above MBOP's 9.70 and Diffuser's 61.20, and competitive with IQL's 73.90 and CQL's 77.20. This is the task where D-MPC shows the largest absolute advantage over Diffuser (+17.6 points), which the paper attributes to the multi-step dynamics model's better handling of the diverse behaviors in the replay dataset.

On Adroit dexterous manipulation tasks (cloned data), D-MPC achieves an average of 39.28, compared to MBOP's 22.47. The individual scores are 89.22 (±12.57) for pen, 16.36 (±2.20) for door, and 12.27 (±3.58) for hammer. Notably, D-MPC substantially outperforms DT (28.36 average) and CQL (4.99 average—CQL struggles dramatically on these high-dimensional manipulation tasks with cloned data), but underperforms IQL (44.90 average), particularly on the pen task where IQL achieves 114.05 (±4.78).

On the Kitchen tasks, D-MPC achieves 67.50 (±2.09) on mixed and 73.33 (±1.64) on partial (average 70.42), outperforming BC (44.75 average), CQL (51.25), and IQL (48.65). MBOP numbers are not reported for Kitchen. The 25+ point advantage over BC on these long-horizon manipulation tasks supports the paper's claim that planning helps compose behaviors from suboptimal data.

Search and Planning Architecture Results (Table 3b, Table 4)

The ablation results organized in Table 3b and detailed in Table 4 systematically decompose D-MPC's performance into contributions from its components. Starting from MBOP's baseline of 33.13 (single-step MLP dynamics, single-step MLP action proposal, TrajOpt planner):

• Replacing MBOP's single-step MLP action proposal with a single-step diffusion action proposal and switching from TrajOpt to the sampling-based planner (Table 4, "SS Diff Action Proposal + SS MLP Dynamics") yields an average locomotion score of 52.93, a 19.8-point improvement. This isolates the benefit of expressive (multimodal) action proposals plus the simpler planner from the single-step vs. multi-step question.
• Further replacing the single-step MLP dynamics with a single-step diffusion dynamics model (Table 4, "SS Diff Action Proposal + SS Diff Dynamics") yields 53.32, a negligible 0.39-point improvement. This suggests that single-step diffusion dynamics does not substantially improve over single-step MLP dynamics when the action proposal is already expressive—both suffer from compounding errors over the 32-step horizon.
• Replacing the single-step diffusion action proposal with a multi-step diffusion action proposal while keeping single-step MLP dynamics (Table 4, "MS Diff Action Proposal + SS MLP Dynamics") yields 57.14, a 4.21-point improvement over the single-step action proposal with the same dynamics. Similarly, multi-step action proposal with single-step diffusion dynamics yields 57.81, a 4.49-point improvement. This quantifies the value of trajectory-level (joint) action generation over step-by-step action generation, holding the dynamics model fixed.
• Replacing the single-step diffusion dynamics with an autoregressive transformer (ART) dynamics model while keeping the multi-step diffusion action proposal (Table 4, "MS Diff Action Proposal + ART Dynamics") yields 59.83, a 2.02-point improvement over single-step diffusion dynamics but still well below the full D-MPC.
• The full D-MPC with multi-step diffusion dynamics and multi-step diffusion action proposal yields 65.98, a 6.15-point improvement over the ART dynamics configuration and a 13.05-point improvement over the single-step diffusion dynamics configuration. This quantifies the value of joint trajectory-level generation (non-autoregressive diffusion) for the dynamics model specifically, over and above what an expressive autoregressive model can achieve.

A critical ablation confirming that diffusion is necessary, not just multi-step modeling: replacing both the multi-step diffusion action proposal and multi-step diffusion dynamics with multi-step MLPs (flattened 2-layer networks with 4096 hidden units, trained with MSE loss) drops performance to 50.01 (Section 4.4.4). This is worse than the single-step diffusion proposal with single-step MLP dynamics (52.93), demonstrating that multi-step modeling without expressive distributional modeling (MLPs collapse to the mean) is counterproductive.

Long-Horizon Dynamics Prediction Accuracy (Figure 2)

Figure 2 measures dynamics prediction quality independent of the planning loop by computing the median root mean square deviation (RMSD) on non-velocity state coordinates for rollouts of increasing length, based on 1024 sampled state-action sequences of length 256. Dynamics models are trained on the medium dataset and evaluated on medium (in-distribution), medium-replay (lower quality, potentially broader distribution), and expert (higher quality, out-of-distribution) datasets. The multi-step diffusion dynamics model shows:

• Substantially lower prediction error on in-distribution (medium) data across all domains and rollout lengths compared to the single-step diffusion, single-step MLP, and ART dynamics models. The error grows slowly with horizon, indicating effective mitigation of compounding error.
• Better generalization to out-of-distribution data (expert, medium-replay) compared to alternatives, particularly the single-step MLP which shows rapid error growth on expert data due to distribution shift.
• The ART dynamics model outperforms single-step diffusion but underperforms multi-step diffusion, supporting the paper's conjecture that non-causal joint generation captures global trajectory properties better than left-to-right autoregressive prediction.

Novel Reward Optimization (Figure 1, Section 4.2)

Figure 1 demonstrates D-MPC's ability to optimize reward functions never seen during training. Using a model trained on the Walker2D medium-replay dataset, the trained value function J is replaced with a hand-designed height-based reward f_novel(s_t, A_t) = 5 exp(-(h_t - h_target)^2 / 2σ^2) with σ^2 = 5 × 10^{-4}, κ = 0, \tilde{κ} = 1. Three target heights produce qualitatively distinct behaviors:

• h_target = 0.9 (lunge): The agent lowers its torso, eventually leaning too far forward and terminating early.
• h_target = 1.2 (balance): The agent maintains the desired height for an extended duration (close to its natural standing height).
• h_target = 1.4 (jump): The agent jumps three times to momentarily achieve the height before falling over.

The height traces (top row) show the agent's height tracking the target over the episode. The reward plots (bottom row) show the novel reward being optimized (peaks when height matches target) and the environment-provided reward (which is near zero since the novel objective ignores the original task). The paper notes that they "did not observe a significant performance difference" between D-MPC and Diffuser on these tasks, but D-MPC's advantage is in dynamics adaptation (next section) rather than novel reward optimization per se.

Adaptation to Novel Dynamics (Table 3a, Section 4.3)

A simulated motor defect is introduced on Walker2D by clipping the action component corresponding to the right foot rotor torque to [-0.5, 0.5] (vs. the original [-1, 1]). Table 3a summarizes the results:

• Original performance (no defect): D-MPC achieves 76.21 (±2.67), Diffuser achieves 79.60 (presumably on the original D4RL evaluation; the paper's Diffuser implementation achieves 72.91 ± 3.47 when measured in their setup).
• Pre-fine-tuning on defective hardware: D-MPC drops to 22.74 (±1.41), Diffuser drops to 25.85 (±1.08). Both methods are severely impacted by the dynamics shift, as expected.
• Post-fine-tuning on 100 episodes of play data (30,170 transitions, collected by running the original D-MPC trained on medium-replay in the defective environment): D-MPC recovers to 30.65 (±1.89), a +7.91 point improvement. Diffuser, which must be fine-tuned as a joint model (no factorized dynamics component to independently update), collapses to 6.8 (±0.86), a −19.05 point degradation from its pre-fine-tuning performance.

The play data used for fine-tuning has an average normalized reward of only 23.14 (±2.31), meaning it is not high-quality demonstration data—it is the behavior produced by an already-degraded model. Despite this, D-MPC's factorized architecture allows the dynamics model to extract useful information about the changed environment response, while Diffuser's joint model corrupts its behavioral knowledge when trying to incorporate the same data.

Distillation to Fast Policy (Section 4.5)

D-MPC's planner can be distilled into a fast MLP policy by training on the planned actions as supervision. Across the 6 locomotion domain/level combinations: the distilled MLP policy achieves an average normalized score of 65.08, only slightly below D-MPC's 65.98, and substantially above the vanilla BC MLP policy's 41.92. After distillation, inference runs at MLP speeds (~0.35 ms per step vs. ~93 ms for D-MPC). This demonstrates that the planning computation can be amortized into a reactive policy with minimal performance loss.

Ablation Studies and Robustness Checks

• Single-step vs. multi-step diffusion action proposals (Table 4): Multi-step action proposals consistently outperform single-step across all domain/level combinations and both dynamics model choices. With single-step MLP dynamics, the multi-step action proposal achieves 57.14 vs. 52.93 (single-step), a +4.21 average gain. With single-step diffusion dynamics, 57.81 vs. 53.32, a +4.49 gain. The largest individual improvements occur on Hopper medium-replay (76.38 vs. 50.66 with SS diffusion dynamics) and Walker2D medium-replay (52.33 vs. 41.57 with SS MLP dynamics), suggesting temporal coherence is particularly important for locomotion tasks requiring coordinated gait patterns.

• Multi-step diffusion dynamics vs. single-step diffusion dynamics vs. autoregressive transformer dynamics (Table 4): Holding the multi-step action proposal fixed, multi-step diffusion dynamics achieves 65.98, ART dynamics achieves 59.83, single-step diffusion dynamics achieves 57.81, and single-step MLP dynamics achieves 57.14. The largest gaps between multi-step diffusion and ART occur on Hopper medium (61.24 vs. 46.24, +14.98) and Walker2D medium-replay (78.81 vs. 39.30, +39.51—the most dramatic difference in the entire ablation table). The ART dynamics actually underperforms single-step diffusion on the medium-replay Walker2D task (39.30 vs. 68.28), suggesting that autoregressive models may be particularly brittle when the action distribution during planning diverges from the training distribution, since medium-replay data is more diverse than medium data.

• Multi-step MLP replacement (Section 4.4.4): Replacing both the multi-step diffusion action proposal and multi-step diffusion dynamics with multi-step MLPs (2-layer, 4096 hidden units, MSE loss) drops average performance to 50.01. This demonstrates that the benefit comes from the combination of multi-step modeling and expressive (distributional) generation—multi-step modeling alone, with a model class that collapses to the mean, is worse than single-step modeling with an expressive model class (52.93). The MLP fails because the joint distribution over 32-step trajectories is highly multimodal, and MSE regression to the mean produces sequences that may not correspond to any physically realizable trajectory.

• Sampling-based planner vs. MBOP's TrajOpt (Table 4, first column vs. MBOP baseline): The combination of single-step diffusion action proposal + single-step MLP dynamics + sampling-based planner (52.93) substantially outperforms MBOP's single-step MLP action proposal + single-step MLP dynamics + TrajOpt (33.13). While this comparison conflates the action proposal change (diffusion vs. MLP) with the planner change, the gap is large enough to suggest that the planner choice is at worst neutral and at best beneficial when paired with better proposals, while being algorithmically simpler.

• State normalization (Appendix D): States are normalized via empirical CDF mapping to [-1, 1] before input to all models. This is not ablated but noted as following Ajay et al. (2023). The choice matters for diffusion model training stability—without normalization, state dimensions with different scales would require the denoising network to handle vastly different variance magnitudes.

• Discount factor sensitivity (Appendix E.4): γ = 0.99 for most domains, γ = 0.997 for Hopper specifically. For Hopper and Walker2D, an additional −100 termination penalty is applied at the final step of early-terminated episodes. The paper does not ablate these choices, which could affect value function learning and planning quality—a higher discount factor makes the value function more sensitive to long-horizon consequences, which is important for Hopper's stability requirements.

• Diffusion timestep counts (Appendix E.1): The action proposal uses 32 timesteps, the dynamics model uses 10 timesteps. These are not ablated and represent a design choice based on the intuition that action distributions are more complex and multimodal than state transition distributions. The choice could affect both sample quality and computational cost—fewer timesteps means faster sampling but potentially lower-quality samples. Without ablation, the sensitivity to these counts is unknown.

• Number of planning samples N: Set to 64 for all experiments, not ablated. The computational cost scales linearly with N, and the paper does not explore whether smaller N would suffice for easier tasks or larger N would help on harder ones. This is a significant omission given that the paper's core thesis is about how to allocate test-time compute.

• Forecast horizon F: Set to 32 for all experiments, not ablated. The justification (Appendix F) is that the value function handles long-term consequences beyond the horizon, so F only needs to cover near-term dynamics. However, the interplay between horizon length and value function accuracy is not studied—a longer horizon reduces dependence on the (imperfect) value bootstrap but increases the difficulty of accurate dynamics prediction.

Critical Assessment

Claim: "D-MPC significantly outperforms existing model-based offline planning methods"

The experiments in Table 2 directly support this for the comparison with MBOP, the only other factorized MPC approach that learns models offline. The average locomotion gap (65.98 vs. 33.13) is large and consistent across all six tasks. However, the comparison is somewhat unfair in D-MPC's favor in one respect: D-MPC uses multi-step diffusion models and 64 planning samples, while MBOP uses single-step MLPs with TrajOpt. The paper's own ablation (Table 4) shows that even single-step diffusion proposals with single-step MLP dynamics achieve 52.93, well above MBOP's 33.13. This suggests MBOP's low performance is at least partly due to its deterministic MLP action proposal, not its single-step nature or its TrajOpt planner. A fairer comparison would have been MBOP with a single-step diffusion action proposal and sampling-based planner (the 52.93 configuration), which would close the gap to the full D-MPC to 13 points rather than 33.

The comparison with Diffuser (the other diffusion-based planning method) is less definitive. D-MPC's locomotion average (65.98) marginally exceeds Diffuser's (63.77), but Diffuser's individual scores on HalfCheetah medium (44.20 vs. 46.00) and Hopper medium-replay (96.80 vs. 92.49) are competitive. The paper notes (Section 4.1) that "the total compute of our two transformers is designed to be roughly equivalent to Diffuser's single model," but this compute-parity claim applies to model capacity, not inference cost—Diffuser's single diffusion model at test time may be more or less expensive than D-MPC's two-model pipeline. No wall-clock or FLOPs-matched comparison is provided between D-MPC and Diffuser at test time. The strongest evidence for D-MPC's advantage over Diffuser is not in Table 2 but in the adaptation experiment (Table 3a), where factorization provides a structural advantage that joint modeling cannot match.

The claim of being "competitive with state-of-the-art model-based and model-free reinforcement learning methods" (Table 2) is supported but with important caveats. D-MPC matches or approaches the best method on several tasks (Walker2D medium: 76.21 vs. IQL 78.30; Hopper medium-replay: 92.49 vs. CQL 95.00) but meaningfully underperforms on others (Hopper medium: 61.24 vs. IQL 66.30 and DT 67.60; Adroit pen: 89.22 vs. IQL 114.05). The paper correctly follows the convention of bolding scores within 5% of the maximum per task, and D-MPC earns bold on 5 of the 11 task/level combinations. This is genuinely competitive. However, it's worth noting that D-MPC requires substantially more computation at inference than model-free methods (~93 ms vs. ~0.35 ms per step; Table 7), so the "competitive" claim is about asymptotic performance, not compute-efficiency. The distillation result (65.08 vs. 65.98) partially addresses this, but the distilled policy was trained with D-MPC as a teacher, which required the expensive planning pipeline to exist.

Claim: "D-MPC can optimize novel reward functions at run time"

This claim is supported but demonstrated only qualitatively (Figure 1) on a single domain (Walker2D) with a single type of novel reward (height-based Gaussian). The examples are compelling as existence proofs—the agent does produce lunging, balancing, and jumping behaviors that were not in the training data—but no quantitative metrics are reported. We do not know:

• The success rate or consistency of these behaviors across episodes (only one example per target height is shown)
• How the quality of novel reward optimization varies with the distance between the novel reward and the training reward (height tracking is arguably related to the locomotion training objective; a reward for "spin in circles" might fail entirely)
• How D-MPC compares quantitatively to Diffuser on novel reward tasks (the paper states "we did not observe a significant performance difference" but provides no numbers)
• Whether the approach works on other domains (Adroit, Kitchen) or with other types of novel rewards

The claim as stated is true—D-MPC can optimize novel rewards—but the scope and robustness of this capability are not characterized. This is a significant gap given that novel reward optimization is presented as one of the two key advantages of the MPC architecture (alongside dynamics adaptation).

Claim: "D-MPC can adapt to novel dynamics through fine-tuning"

The experiment in Section 4.3 (Table 3a) provides genuine evidence for this claim, but the demonstrated improvement is modest. Post-fine-tuning, D-MPC recovers from 22.74 to 30.65 on a task where the original (no-defect) performance was 76.21. This is a recovery of only about 15% of the lost performance (30.65 vs. 76.21). The paper frames this as a success because it beats fine-tuned Diffuser (which collapses to 6.8), and the comparison does demonstrate the structural advantage of factorization. But a score of 30.65 on Walker2D is still poor by D4RL standards (BC on medium data achieves 75.30; Table 2), and the paper does not explore whether additional fine-tuning data, different fine-tuning hyperparameters, or alternative adaptation strategies could close the remaining gap.

A missing experiment: does fine-tuning the dynamics model plus the action proposal perform better than fine-tuning only the dynamics? The paper's argument is that only dynamics need updating, but the behavioral data from the defective hardware (collected by running a degraded D-MPC) might contain useful information about which actions work well under the new dynamics. Fine-tuning only the dynamics preserves the original behavioral prior, which is sensible if that prior is nearly optimal for the new dynamics, but may be suboptimal if the defect requires qualitatively different behaviors.

A second missing experiment: adaptation with varying amounts of play data. The paper uses 100 episodes (30,170 transitions). How does performance scale with the amount of fine-tuning data? Could 10 episodes suffice? Would 1000 episodes close the gap to original performance? Without this scaling analysis, we cannot assess the sample efficiency of adaptation.

Claim: "Ablations validate the benefits of D-MPC's key components individually and in combination"

The ablation study (Table 4) is the strongest part of the paper's experimental evaluation. It systematically isolates the contributions of: expressive action proposals (diffusion vs. MLP), multi-step action generation (single-step vs. multi-step diffusion proposal), expressive dynamics models (diffusion vs. MLP vs. ART), and multi-step dynamics generation (single-step vs. ART vs. multi-step diffusion). The results are internally consistent and support the paper's architectural arguments. The finding that multi-step MLPs (50.01) underperform single-step diffusion proposals with single-step MLP dynamics (52.93) is particularly informative—it shows that joint generation without expressive distributional modeling is worse than step-by-step generation with expressive modeling, underscoring that the combination of multi-step + diffusion is what matters, not either property alone.

However, several important ablations are missing:

• Number of planning samples N: all experiments use N=64. How does performance scale with N? A core motivation for D-MPC is that the diffusion proposal is a strong prior requiring fewer samples, but we cannot evaluate this without seeing performance at smaller N (e.g., N=8, N=16, N=32) and whether it saturates.
• Forecast horizon F: all experiments use F=32. Does performance degrade with shorter horizons (which would be cheaper)? Does it improve with longer horizons?
• Diffusion timestep counts: the action proposal uses 32 steps, the dynamics model uses 10 steps. Is the performance sensitive to these choices? How much faster would the system be with fewer timesteps, and what would the performance cost be?
• Value function vs. direct reward computation: the paper uses a learned value function for J. What happens if J is computed directly from the environment reward (which is known in D4RL) plus a simple heuristic for terminal value? This would remove the need to train a separate value model.
• Comparison to a single-step diffusion dynamics model used inside a multi-step planning loop: the paper compares single-step vs. multi-step dynamics models, but the single-step model is used autoregressively. An alternative would be to use the single-step model only for K steps and rely more heavily on the value bootstrap, potentially trading off dynamics accuracy for speed.

Claim (implicit): "D-MPC's multi-step diffusion dynamics avoids compounding errors"

Figure 2 provides convincing evidence that the multi-step diffusion dynamics model achieves lower long-horizon prediction error than alternatives. However, the connection between prediction error reduction and controller performance improvement is not linear. The ART dynamics model substantially reduces prediction error compared to single-step models (Figure 2) but only modestly improves controller performance (59.83 vs. 57.14; Table 3b). This suggests that beyond a certain accuracy threshold, other factors (planner quality, proposal diversity) dominate. The paper's stated mechanism—that joint generation avoids compounding error by enforcing global coherence—is plausible but not directly tested: we see that multi-step diffusion makes better predictions and that better predictions correlate with better control, but we don't see direct evidence that global coherence constraints (e.g., predicting that the robot will fall over) are what make the difference, as opposed to simply having a higher-capacity model.

Overall Assessment

The experiments genuinely support the paper's central architectural claim: factorized multi-step diffusion models for dynamics and action proposals, combined in a sampling-based MPC loop, yield an agent that outperforms prior factorized MPC (MBOP) and is competitive with state-of-the-art offline RL methods on standard benchmarks. The ablation study provides strong evidence that each component contributes to the final performance. The demonstrations of novel reward optimization and dynamics adaptation, while qualitative or modest in magnitude, illustrate capabilities that are structurally unavailable to non-factorized or non-MPC methods.

The most significant weaknesses are: (1) the lack of quantitative characterization for novel reward optimization (only qualitative demonstrations on one domain); (2) the modest absolute performance recovery in the dynamics adaptation experiment (30.65 vs. original 76.21); (3) missing ablations on key hyperparameters (N, F, diffusion timesteps) that would clarify the compute-efficiency tradeoffs; and (4) the use of a single benchmark suite (D4RL) without testing on more diverse environments or higher-dimensional observation spaces (the paper acknowledges this last point as future work for pixel observations). The paper would be strengthened by scaling curves showing performance vs. N, comparisons of total FLOPs or wall-clock time against baseline methods at matched performance levels, and quantitative metrics for novel reward optimization across multiple reward functions and domains.

6. Limitations and Trade-offs

The Method Cannot Create Capabilities Absent from the Offline Data

The assumption or constraint. D-MPC's action proposal learns the behavioral distribution present in the offline dataset. If certain skills or behaviors are entirely absent from the training trajectories, the proposal cannot generate them, and no amount of planning will discover them. The paper is explicit about this in Section 5:

"Like all offline RL methods, D-MPC’s performance is influenced by the distribution of behaviors in the training dataset. When offline datasets lack behaviors relevant to the target task, the generalization capabilities of any method are inherently constrained without additional data collection."

The consequence. D-MPC can recombine and re-rank behaviors that exist in the data to achieve higher reward than any single trajectory demonstrated, but it cannot invent new behaviors. This distinguishes planning-as-amplification from planning-as-creation. The practical implication: if a robot needs to perform a manipulation skill that was never attempted in the offline data (e.g., grasping from a novel angle), D-MPC will fail regardless of how much test-time compute is allocated. The planning horizon $F=32$ and $N=64$ samples cannot overcome a support gap in the proposal distribution.

What evidence exists in the paper. The limitation is stated explicitly but not empirically characterized. The paper does not include experiments that systematically degrade dataset coverage (e.g., removing certain behavior modes from the training data and testing whether D-MPC can recover them through planning). The Adroit cloned dataset results (Table 2) provide indirect evidence: on hammer and door tasks, where cloned data is narrow (single demonstrator), D-MPC achieves only 12.27 and 16.36 normalized scores respectively, while on pen (where the task is simpler) it achieves 89.22. The low scores on difficult manipulation tasks with narrow data are consistent with the behavioral coverage bound, but the connection is not formally tested.

Mitigation status. The paper does not attempt to mitigate this limitation. It acknowledges it as a fundamental constraint of offline RL (Section 5) and suggests future work could "explore techniques to encourage broader exploration within the constraints of offline data," but offers no concrete mechanism. This is not a failure of the method so much as a hard boundary on its applicability: D-MPC is suitable when the offline data contains the necessary behavioral building blocks, even if suboptimally arranged. When it does not, collecting more or better data is the only solution.

The Test-Time Compute Cost Is ~250× Higher Than a Reactive Policy, and the Paper Includes No Scaling Analysis of Planning Budget vs. Performance

The assumption or constraint. D-MPC requires sampling $N=64$ action sequences from a 32-timestep diffusion model, $N=64$ state sequences from a 10-timestep diffusion model, and $N=64$ forward passes through a 10-layer transformer value function, at every single environment step. The paper measures this cost in Appendix J (Table 7) at 92–95 milliseconds per step on a single A100 GPU. For comparison, a simple MLP policy takes ~0.35 ms per step — roughly 265× faster.

The consequence. For any application requiring high-frequency control (e.g., a robot operating at 50–100 Hz), D-MPC's per-step latency is prohibitive. The method cannot be deployed directly in a real-time control loop without either (a) executing much longer action chunks between replanning steps (amortizing the cost) or (b) distilling the planner into a fast policy offline (Section 4.5). The first option reduces reactivity to unexpected events. The second option eliminates the run-time flexibility (novel reward optimization, dynamics adaptation) that is D-MPC's primary advantage over standard policies.

More critically, the paper never varies the planning budget $N$ in any experiment. All results use $N=64$. We therefore cannot answer the most basic resource-allocation question: how much test-time compute is actually needed? Does $N=16$ achieve 90% of the performance at 1/4 the cost? Does $N=128$ provide meaningful further improvement? Without scaling curves for $N$ (analogous to the $N$ sweeps in the prior example paper), a practitioner cannot determine the cost-performance tradeoff for their deployment. This is a significant omission given that the paper's core thesis is about using planning at test time — the efficiency of that planning is a first-order concern.

The same applies to the forecast horizon $F=32$ and the diffusion timestep counts (32 for proposal, 10 for dynamics). None are ablated for their effect on performance vs. cost.

What evidence exists in the paper. Table 7 provides wall-clock timing for three locomotion domains, confirming the latency cost. Section 4.5 demonstrates that distillation to an MLP policy recovers 65.08 vs. D-MPC's 65.98 average locomotion score, showing that the expensive planning can be amortized offline with minimal performance loss for fixed tasks. However, the distilled policy is a standard MLP — it loses all the flexibility advantages (novel reward optimization, dynamics adaptation) that motivate D-MPC in the first place. The paper does not explore intermediate options (e.g., executing action chunks of length 4 or 8 between replanning steps, which would reduce the per-step cost proportionally while retaining most of the MPC reactivity).

Mitigation status. The paper acknowledges the speed issue in Section 5:

"One disadvantage of our method (shared by all MPC methods) is the need to replan at each step, which is much slower than using a reactive policy. This is particularly problematic when using diffusion models, which are especially slow to sample from."

It suggests future work on accelerated diffusion sampling (distillation, progressive distillation, consistency models) and amortizing planning over action chunks. The distillation experiment (Section 4.5) partially addresses the issue but only for the fixed-task case. The core tradeoff — flexibility vs. speed — is identified but not resolved.

Novel Reward Optimization Is Demonstrated Only Qualitatively on a Single Domain with One Reward Type

The assumption or constraint. The ability to optimize novel reward functions at test time is presented as one of the two primary advantages of D-MPC's MPC architecture (alongside dynamics adaptation). The paper states in Section 1:

"The advantage of this MPC approach compared to standard policy learning methods is that we can easily adapt to novel reward functions at test time, simply by searching for state-action trajectories with high reward."

The consequence. The paper provides only three qualitative trajectory examples on Walker2D (Figure 1) using a height-based Gaussian reward. No quantitative metrics are reported — we do not know:

• The success rate or consistency of the novel behaviors across multiple episodes
• How performance degrades as the novel reward diverges from the training objective (height tracking is related to locomotion; a reward for "spin in circles" or "walk backward" might fail)
• Whether novel reward optimization works on other domains (Adroit manipulation, Kitchen) or with other reward types (e.g., sparse rewards, combinatorial reward functions, multi-objective tradeoffs)
• How D-MPC compares quantitatively to Diffuser on novel reward tasks (the paper states "we did not observe a significant performance difference" but provides no numbers)

A practitioner considering D-MPC for a novel-task scenario would need to know: for what class of novel rewards does this work reliably, and how novel can the reward be before the action proposal's coverage is insufficient? The paper provides no guidance.

What evidence exists in the paper. Only Figure 1 (three example episodes) and the method description in Section 3.4 and Appendix H. The examples are compelling as existence proofs — the agent does produce lunging, balancing, and jumping behaviors — but they do not constitute a systematic evaluation. The target heights tested (0.9, 1.2, 1.4) all lie near the agent's natural operating range (starting height ~1.25). A height target of 2.0 (impossible under the physics) or 0.5 (requiring a qualitatively different posture than any seen in the locomotion data) could reveal failure modes.

Mitigation status. Not addressed. The paper does not acknowledge the qualitative nature of this evidence or suggest what a rigorous evaluation would look like. This is the weakest-evidenced of D-MPC's claimed advantages.

Dynamics Adaptation Recovery Is Modest (15% of Lost Performance Recovered), and the Scaling of Adaptation with Data Quantity Is Unexplored

The assumption or constraint. D-MPC claims the ability to adapt to novel dynamics by fine-tuning only the multi-step diffusion dynamics model on a small amount of play data. The paper demonstrates this (Section 4.3) with a simulated motor defect on Walker2D and 100 episodes (30,170 transitions) of fine-tuning data.

The consequence. Post-fine-tuning, D-MPC's normalized score improves from 22.74 (±1.41) to 30.65 (±1.89). This is a +7.91 point gain, but the original (no-defect) performance was 76.21. So the adaptation recovers only about 15% of the lost performance. A score of 30.65 on Walker2D medium is poor — it is below MBOP's 41.00 (Table 2) and far below BC's 75.30. A practitioner whose robot experiences a hardware defect and uses this adaptation procedure would still be left with a largely non-functional agent.

The paper frames the result as a success because it dramatically outperforms Diffuser (which collapses to 6.8 after fine-tuning), and the comparison does demonstrate the structural advantage of factorization. But the absolute performance recovery is insufficient for practical deployment.

Additionally, the paper provides no scaling analysis: how much fine-tuning data is actually needed? Could 10 episodes suffice for partial recovery? Would 500 episodes close most of the gap to the original 76.21? The play data used has an average normalized reward of only 23.14 (±2.31), meaning it is low-quality data collected by an already-degraded model. Better play data (e.g., from a human teleoperator or a random exploration policy) might enable faster or more complete adaptation, but this is not explored.

What evidence exists in the paper. Table 3a reports the three data points (original, pre-fine-tuning, post-fine-tuning) for both D-MPC and Diffuser. The play data collection procedure is described in Section 4.3 and Appendix I. No experiments vary the amount of fine-tuning data, the quality of fine-tuning data, or the fine-tuning hyperparameters.

Mitigation status. The paper does not discuss the modest absolute recovery or suggest how it might be improved. The adaptation experiment serves as a proof-of-concept for the factorized architecture's advantage over joint modeling, but not as a demonstration of practical adaptation capability. Future work on data-efficient dynamics adaptation (e.g., meta-learning across defects, using better exploration strategies for play data collection) would be needed to make this a deployable capability.

All Experiments Use a Single Benchmark Suite (D4RL) with Low-Dimensional Proprioceptive States; Generalization to High-Dimensional Observations, Different Domains, and Stochastic Environments Is Unestablished

The assumption or constraint. Every experiment in the paper is conducted on D4RL (Fu et al., 2020) with direct access to low-dimensional state vectors (joint angles, velocities, positions). The paper acknowledges in Section 5:

"Another limitation of the current D-MPC is we only explored setups where we directly have access to the low-dimensional states, such as proprioceptive sensors on a robot."

The consequence. The entire approach — multi-step diffusion dynamics modeling, trajectory-level action proposals, sampling-based MPC — has been validated only on a specific class of problems: continuous control tasks with fully observed, low-dimensional states, near-deterministic dynamics, and dense rewards (for the value function training). We do not know whether D-MPC would work for:

• Pixel-based observations: The dynamics model would need to generate sequences of high-dimensional images, a dramatically harder generation problem than low-dimensional state vectors. The computational cost (already ~93 ms per step for 17–23 dimensional states) would balloon.
• Stochastic environments: D4RL locomotion tasks have near-deterministic dynamics given actions. In genuinely stochastic environments (e.g., with wind, sensor noise, or other agents), the dynamics model must capture aleatoric uncertainty. The paper does not evaluate whether the multi-step diffusion dynamics model properly represents stochastic transitions or whether the sampling-based planner is robust to environment stochasticity.
• Discrete action spaces: The entire method assumes continuous actions amenable to diffusion modeling. Discrete action spaces would require a different generative model class (e.g., discrete diffusion or autoregressive models).
• Sparse reward tasks: The value function $J$ requires dense reward signals for effective training (it regresses on discounted returns). On tasks with only terminal rewards, the value function would be much harder to learn, and the planning horizon $F=32$ might be insufficient to reach reward states.
• Multi-agent or safety-critical settings: No experiments address these scenarios.

This is not merely a "future work" gap — it means a practitioner considering D-MPC for a real-world robotics application (which would typically involve camera inputs, stochastic dynamics from imperfect actuation, and sparse task-completion rewards) has no empirical basis to estimate how well the method would transfer.

What evidence exists in the paper. None — all experiments are on D4RL with low-dimensional states. The paper does not include even a simple pixel-based control experiment (e.g., D4RL's image-based variants, or DeepMind Control Suite with rendered observations) that would provide a lower bound on how the approach degrades with high-dimensional inputs.

Mitigation status. The paper explicitly identifies this as a limitation and suggests future work on "representation learning methods that extract abstract latent representations, which can form the input to our dynamics models, similar to existing latent-space world modeling approaches such as the Dreamer line of work, but in an MPC context, rather than a Dyna context." This is a plausible direction, but integrating D-MPC's multi-step diffusion architecture with a learned latent space introduces substantial additional complexity (end-to-end training vs. two-stage representation learning, dealing with stochastic latents, handling partial observability) that is entirely speculative at this point.

The Value Function Is Trained with L2 Regression on Returns and Relies on an Environment-Specific Termination Penalty; Neither Its Accuracy Nor Its Sensitivity to These Design Choices Is Evaluated

The assumption or constraint. The planning objective $J$ (Equation 2) uses a learned value function $V(s_{t+F})$ to provide a bootstrap estimate of future returns beyond the planning horizon. This value function is trained via L2 regression on discounted returns from the offline dataset (Appendix E.4). The paper notes that for Hopper and Walker2D, an additional −100 termination penalty is included at the final step of early-terminated episodes, and that Hopper uses a higher discount factor (0.997 vs. 0.99 for other domains).

The consequence. The value function is a critical component: it determines the long-horizon consequences that the planner optimizes for. If $V$ is inaccurate — particularly around the planning horizon boundary $F=32$ — the planner may select action sequences that produce good near-term reward (within the 32-step horizon) but lead to failure shortly after (e.g., a sprinting gait that causes the robot to fall at step 35). The termination penalty of −100 is a heuristic designed to encode "falling over is bad" into the value function, but its magnitude relative to the typical per-step rewards (~1–5 for locomotion) is extreme and creates a discontinuous reward landscape that may be difficult to regress accurately.

The paper provides no evaluation of value function accuracy — no held-out prediction error, no comparison to Monte Carlo returns, no ablation where the value function is replaced with a simpler heuristic (e.g., zero for alive, −100 for fallen, or a constant optimistic value). A practitioner cannot assess whether the value function is a reliable component or a potential source of planning failures.

More fundamentally, the reliance on an environment-specific termination signal (the agent "knowing" it has fallen) may not transfer to real-world settings where failure states are not cleanly signaled. In a real robot, "falling over" might manifest as a gradual degradation in joint angles rather than a binary event, and designing a termination condition would itself be a significant engineering task.

What evidence exists in the paper. The performance numbers in Table 2 show that D-MPC achieves high scores on Hopper (92.49 medium-replay) and Walker2D (78.81 medium-replay), suggesting the value function is working adequately for these tasks. But this is indirect — a good policy can compensate for a mediocre value function, or a good value function can mask planning deficiencies. We cannot disentangle the value function's contribution from the other components without dedicated ablations.

Mitigation status. Not addressed. The value function's training details are described (Appendix E.4) but no sensitivity analysis is performed. The paper does not ablate the discount factor, the termination penalty magnitude, or the architecture of the value function. The claim that the forecast horizon $F=32$ "already works well since our trained objective function $J$ predicts discounted future rewards" (Appendix F) assumes what needs to be demonstrated — that the value bootstrap is accurate enough for $F=32$ to suffice.

7. Implications and Future Directions

How This Work Changes the Landscape

D-MPC introduces a reframing of the model-based RL architecture problem rather than a paradigm shift. The field has long understood that factorized dynamics and policy models enable flexibility, and that expressive generative models can capture complex distributions. D-MPC's contribution is demonstrating that these two ideas are not merely compatible but synergistic in ways that neither property alone achieves: factorization without expressiveness (MBOP's single-step MLPs) is weak, and expressiveness without factorization (Diffuser's joint model) is brittle to dynamics shifts. By showing that the combination unlocks capabilities (novel reward optimization, partial dynamics adaptation) that neither predecessor could manage, the paper establishes a new architectural principle: build world models and behavioral priors as separate, expressive, trajectory-level generative components.

This principle shifts research attention in several directions:

Away from planner sophistication and toward proposal quality. The finding that a simple sampling-and-ranking planner (Algorithm 2) with N=64 and a good action proposal outperforms TrajOpt with a weak proposal (Table 3b: 52.93 vs. 33.13) reframes the planner's role from "optimizer" to "selector." Future work on MPC with learned models should invest more heavily in the proposal distribution — making it expressive, multimodal, and trajectory-aware — rather than designing increasingly complex optimization algorithms. The paper's ablation showing that multi-step MLPs (50.01) underperform single-step diffusion proposals (52.93) further reinforces that distributional expressiveness matters more than multi-step structure alone, and that the combination is where the gains live.

Away from joint trajectory modeling for adaptable agents. The adaptation experiment (Table 3a: Diffuser collapses from 25.85 to 6.8 when fine-tuned on defective-hardware data, while D-MPC recovers from 22.74 to 30.65) provides concrete evidence that joint modeling of states and actions — however elegant — creates an architectural vulnerability to domain shift. For any application where the environment might change post-deployment (robotics with wear-and-tear, sim-to-real transfer, multi-tenant systems with different hardware instances), factorized models should be preferred over joint models, even at equivalent nominal performance. This is a design rule, not just a preference — Diffuser's comparable or slightly better nominal scores on some tasks (Table 2: 96.80 vs. 92.49 on Hopper medium-replay) should not outweigh its fragility to dynamics changes.

Toward test-time compute as a legitimate axis of agent capability, with clearer boundaries. The paper implicitly engages with the pretraining-vs-inference compute tradeoff, though less explicitly than the prior example paper. D-MPC shows that test-time planning can compensate for suboptimal training data: on medium and medium-replay datasets (which contain mediocre, exploratory, and suboptimal behaviors), D-MPC achieves scores competitive with methods trained on the same data (Table 2: 65.98 locomotion average vs. IQL's 67.47 and CQL's 65.45). The planning process recombines behaviors from the data into higher-reward sequences. However, the paper also establishes a sharp boundary: D-MPC cannot transcend the behavioral coverage of its training data. When the offline dataset lacks relevant skills entirely (e.g., the Adroit hammer task with cloned data, where D-MPC achieves only 12.27), no amount of planning helps. This refines the "planning helps" intuition into an operational criterion: planning amplifies existing capabilities; it does not create them. This is analogous to the prior example paper's finding that test-time compute helps on easy-to-medium problems (where the base model already produces correct solutions at some non-trivial rate) but not on hard problems (where the base model's pass@1 is near zero). Both papers converge on the same structural boundary, from different domains.

Reconciling conflicting intuitions about model-based RL. The paper helps explain why model-based methods have historically underperformed model-free methods on standard benchmarks despite their conceptual advantages. Methods like MBOP had the right architecture (factorized MPC) but used model classes (single-step deterministic MLPs) incapable of capturing the complex distributions needed for long-horizon planning, leading to compounding errors and poor performance (33.13 locomotion average). Methods like Diffuser had the right model class (diffusion) but used an entangled architecture that prevented adaptation and may have been less sample-efficient than necessary. D-MPC shows that when both properties are combined, model-based methods become genuinely competitive with the best model-free approaches (Table 2). This resolves the apparent paradox: model-based RL was not fundamentally limited; it was using the wrong tools for each component of the problem.

Follow-Up Research This Work Enables

Scaling laws for planning budget vs. performance. The paper's most conspicuous omission is any sweep over the number of planning samples N. All experiments use N=64. A natural follow-up would measure D-MPC performance on D4RL locomotion tasks as N varies from 1 to 256 (or higher), producing curves analogous to the best-of-N scaling curves in the prior example paper. Key questions: Does performance saturate at N=64, or would N=128 or N=256 yield meaningful further gains? What is the minimum N that achieves, say, 90% of the N=64 performance — is it 8, 16, or 32? The wall-clock cost in Table 7 (~93 ms at N=64) scales roughly linearly with N, so knowing the knee of the curve is essential for practitioners. A companion sweep over forecast horizon F (varying from 4 to 64) would reveal whether the value function bootstrap is doing heavy lifting (small F suffices) or whether long-horizon dynamics accuracy is critical (performance degrades sharply at small F). These sweeps are straightforward to run — they require no new models or environments, only varying the planner hyperparameters — and would transform D-MPC from a method with fixed hyperparameters into a framework where compute can be allocated adaptively based on task difficulty or latency requirements.

Systematic characterization of novel reward generalization boundaries. The paper demonstrates novel reward optimization only qualitatively on Walker2D with a height-based Gaussian reward (Figure 1). A rigorous follow-up would measure the range of novel reward functions over which D-MPC's performance degrades acceptably. Concrete design: on Walker2D medium-replay, define a parametric family of reward functions that systematically diverge from the training objective — e.g., weighted combinations of velocity, height, and torque penalties, or rewards for specific gait frequencies, or directional preferences (forward vs. backward). Measure the normalized score as a function of the distance (e.g., KL divergence in reward-induced trajectory distributions) from the training reward. The hypothesis is that performance degrades gracefully until the novel reward demands behaviors outside the action proposal's support, at which point it collapses. Identifying that boundary would tell practitioners what "novel" actually means. Extend to at least one non-locomotion domain (e.g., Adroit pen with a reward for specific orientation trajectories, or Kitchen with a reward for completing subtasks in a specified order). This experiment would also enable quantitative comparison with Diffuser on novel reward tasks, replacing the paper's qualitative "we did not observe a significant performance difference" with numbers.

Data-efficient dynamics adaptation with scaling analysis. The paper's adaptation experiment (Section 4.3) uses a single data point: 100 episodes (30,170 transitions) of play data, achieving a +7.91 recovery from 22.74 to 30.65 (compared to original 76.21). A systematic follow-up would measure the adaptation curve: fine-tune the dynamics model with 1, 2, 5, 10, 20, 50, 100, 200, 500 episodes of play data and measure normalized score at each point. This would reveal the sample efficiency of dynamics adaptation — does most of the recovery happen in the first 10 episodes (suggesting rapid model adaptation) or does it require hundreds (suggesting slow relearning)? A second critical variable: the quality of play data. The paper's play data had average reward 23.14 (±2.31), collected by a degraded D-MPC. Compare adaptation using this low-quality data vs. using random exploration data (uniform or Gaussian action noise) vs. using human teleoperation data (if available for the specific defect). The hypothesis is that data diversity matters more than data quality for dynamics model adaptation — a random policy might sufficiently excite all state dimensions, even if it achieves near-zero reward. A third variable: what to fine-tune. Compare fine-tuning only the dynamics model (the paper's approach) vs. fine-tuning both dynamics and action proposal (to capture changes in which actions are effective under the new dynamics) vs. training from scratch on play data alone. The factorized architecture makes all these comparisons straightforward.

Combining D-MPC with latent world models for pixel-based control. The paper identifies pixel-based control as the natural next frontier (Section 5). A concrete first experiment: on the DeepMind Control Suite with rendered observations (e.g., Walker walk, Cheetah run), train a variational autoencoder (VAE) to compress 64×64×3 image observations into a low-dimensional latent state (e.g., 64–128 dimensions). Then train D-MPC's multi-step diffusion dynamics model and action proposal in latent space, and run the planning loop entirely in latents. The open question is whether the multi-step diffusion dynamics model can accurately predict latent trajectories — the latent space may not be as well-behaved as proprioceptive states (discontinuities, stochasticity from the encoder), and compounding errors in latent prediction may manifest differently. Baseline comparisons: a Dyna-style method using the same latent world model (Dreamer-style), and a model-free method operating directly on pixels (DrQ-v2). This experiment would stress-test D-MPC's architectural advantages (factorization, multi-step joint generation) in a substantially harder setting where dynamics prediction errors are harder to avoid.

Adaptive planning budget allocation based on state uncertainty or task difficulty. Both this paper and the prior example paper converge on the idea that test-time compute should be allocated adaptively — the prior paper conditions on estimated problem difficulty, and D-MPC's architecture naturally supports variable planning effort (change N, F, or number of diffusion timesteps per step). A follow-up could implement an adaptive D-MPC where the planning budget is adjusted per timestep based on the variance of the value predictions across the N sampled trajectories (high variance = uncertain about which action is best = more samples needed) or based on the predicted reward-to-go (low predicted value = already in a bad state = planning may not help). Concretely: at each step, sample an initial N=8 trajectories and compute the standard deviation of J values. If the standard deviation exceeds a threshold, draw additional samples (up to some maximum budget). Measure: total compute used per episode vs. fixed-N baselines, and whether the adaptive strategy achieves the same performance with lower average compute. This connects D-MPC to the broader "adaptive computation time" literature and provides a practical mechanism for balancing latency and performance.

Verifier-free planning via direct reward computation vs. learned value functions. The paper uses a learned value function J trained with L2 regression on returns, which introduces its own potential inaccuracies and requires an environment-specific termination penalty. A diagnostic follow-up would ablate the value function entirely: replace J with the undiscounted sum of environment rewards over the planning horizon F plus a constant terminal value (e.g., 0 for all terminal states, or the average return in the dataset). This tests whether the value function is providing useful long-horizon credit or whether the F=32 horizon with direct reward computation already captures most of what matters. The experiment would reveal how much of D-MPC's performance is attributable to the value function vs. the dynamics + proposal quality. If performance drops substantially, it would motivate investment in better value function learning (distributional RL, ensembling, conservative value estimates). If performance is largely unchanged, it would simplify the system by removing a trained component.

Practical Applications and Downstream Use Cases

Sim-to-real transfer with dynamics fine-tuning. A robotics team trains D-MPC entirely in simulation (where abundant data is cheap) and deploys on a physical robot. Inevitably, the real dynamics differ from simulation — friction, joint stiffness, actuator delays. The factorized architecture allows the team to collect a small amount of play data on the physical robot (say, 20–50 episodes of the robot attempting various motions, possibly under teleoperation or a simple PID controller) and fine-tune only the multi-step diffusion dynamics model, preserving the action proposal and value function from simulation. The paper's adaptation result (Table 3a: +7.91 recovery from 30,170 transitions) provides a lower bound on what's achievable; better play data quality and coverage would likely improve recovery. This is immediately applicable to any lab with a D4RL-scale simulated environment and a corresponding physical platform (e.g., a Unitree Go1 or similar quaduped, or a Franka arm for Kitchen-style manipulation). The key advantage over existing sim-to-real methods is that the action proposal — which encodes complex, coordinated behavioral patterns learned from abundant simulation data — transfers intact; only the environment's response to those behaviors needs updating.

Rapid retasking of learned controllers in manufacturing. A factory deploys a robotic arm trained via D-MPC on a set of manipulation primitives (reach, grasp, place) from offline demonstration data. When the production line changes — requiring the arm to place objects in a new orientation, or to avoid a new obstacle, or to minimize energy consumption rather than speed — the engineering team does not need to collect new demonstrations or retrain the full system. They define a new reward function encoding the changed objective (e.g., a penalty for proximity to a specified zone, or a bonus for a final gripper orientation) and replace the J term in the planner's scoring. The action proposal, trained on broad manipulation data, already generates reaching and grasping motions; the planner simply selects among them based on the new criterion. The paper's Walker2D novel-reward demonstration (Figure 1: lunging, balancing, jumping from a single trained model by changing only the height target) provides the proof-of-concept. The practical value is in the engineering simplicity: defining a reward function is typically much easier than collecting and labeling new demonstration data, and the feedback loop (edit reward, observe behavior, iterate) is orders of magnitude faster than retraining.

Offline-to-online fine-tuning without catastrophic forgetting of behavioral priors. A common failure mode when fine-tuning RL policies on new tasks or environments is that the policy forgets previously learned skills — the agent specialises to the new objective at the expense of broad behavioral competence. D-MPC's factorized architecture provides a natural mitigation: when fine-tuning for a new task, keep the action proposal frozen (preserving the broad behavioral repertoire) and only update the dynamics model (if the environment is unchanged) or the value function (if only the objective changed) or both (if both changed). The proposal's frozen distribution acts as a regularizer, ensuring the agent only considers actions that are broadly sensible, even if the new objective would otherwise drive it toward degenerate solutions. The distillation experiment (Section 4.5: distilled MLP achieves 65.08 vs. D-MPC's 65.98) further suggests that after task-specific fine-tuning of the planner, the resulting behavior can be compressed into a fast policy for deployment, retaining the benefits of planning while achieving reactive speeds.

Data curation and augmentation for offline RL. When building offline RL datasets, practitioners must decide what data to collect and how much diversity is needed. D-MPC's ablation results provide concrete guidance: the action proposal's coverage matters more than near-optimality. The paper shows that D-MPC achieves 92.49 on Hopper medium-replay (diverse, suboptimal data) vs. 61.24 on Hopper medium (narrower, higher-quality data), suggesting that behavioral diversity in the training data is more valuable than optimality when using a planning-based approach. A data collection team could use this insight to prioritize broad exploration (varying gaits, speeds, postures) over expert demonstrations. Further, the dynamics model's generalization in Figure 2 (the multi-step diffusion dynamics trained on medium data generalizes well to expert data in terms of prediction error) suggests that dynamics models learn transferable physical knowledge from diverse data, even if the behaviors themselves are suboptimal. This has implications for how to construct training datasets for generalist robot foundation models.

When to Prefer This Method

The paper explicitly positions D-MPC within a taxonomy (Table 1) that makes its tradeoffs clear relative to named alternatives, enabling specific decision rules:

• Prefer D-MPC over Diffuser or other joint trajectory models when test-time adaptation to novel dynamics is required or anticipated. The factorized architecture enables independent fine-tuning of the dynamics model without corrupting behavioral knowledge; joint models structurally cannot do this (Table 3a: Diffuser collapses 25.85 → 6.8 after fine-tuning on dynamics-shifted data, while D-MPC recovers 22.74 → 30.65).

• Prefer D-MPC over MBOP or other single-step factorized MPC when planning horizons exceed ~8–16 steps, where compounding errors from single-step models become severe. The multi-step diffusion dynamics model's lower long-horizon prediction error (Figure 2) and the performance gap (65.98 vs. 33.13 locomotion average) justify the additional computational cost of diffusion sampling when accurate long-horizon prediction matters. For very short horizons or trivially predictable dynamics, MBOP's lower latency (~4 ms vs. ~93 ms per step) may dominate.

• Prefer D-MPC over model-free offline RL (CQL, IQL, BC) when novel reward specification at test time is required. Model-free methods bake the objective into policy weights; D-MPC accepts arbitrary reward functions at deployment via the J term in the planner. The cost is higher inference latency (~250× slower than an MLP policy), which can be partially mitigated by distillation (Section 4.5) if the task is fixed.

• Prefer model-free methods (or D-MPC with distillation) over online D-MPC when low-latency, high-frequency control is non-negotiable and the task is fixed. D-MPC's ~93 ms per step is incompatible with 50–100 Hz control loops. Distillation to an MLP (Section 4.5: 65.08 vs. 65.98 performance, at MLP speeds) recovers the performance but loses run-time flexibility, making it equivalent to a strong model-free policy trained with a better teacher.

• Neither D-MPC nor any offline method is suitable when the offline dataset lacks behaviors within the support of the task's requirements. D-MPC amplifies and recombines existing behavioral patterns but cannot invent new ones (Section 5). If the necessary skills are absent from the data, additional data collection — not better planning — is the bottleneck.