Natural Questions: A Benchmark for Question Answering Research

ArXiv: 1901.04473

🎯 Pitch

A recurrent neural network policy learns to land spacecraft on Mars despite random engine failure and on asteroids with unknown gravity—no system identification needed. It even flies blind using raw Doppler radar returns, a task impossible for classical guidance laws.

---

1. Executive Summary

This paper proposes a novel adaptive guidance system developed using reinforcement meta-learning with a recurrent policy and value function approximator—the recurrent network's hidden state adapts in real time to environmental forces acting on the agent (e.g., unknown gravity, engine failure, mass variation)—and evaluates it against a standard DR/DV guidance law and a non-recurrent (MLP) policy across four challenging landing tasks with highly variable dynamics. The recurrent policy consistently achieves the best performance, including enabling integrated guidance and navigation using only noisy Doppler radar altimeter returns—a task the DR/DV baseline cannot attempt—while the DR/DV law fails catastrophically under engine failure (123 m mean terminal position error and 20 m/s mean terminal velocity) and asteroid landing with unknown dynamics, establishing that recurrent-network-based policies can adapt to substantial environmental uncertainty during deployment without requiring explicit system identification.

2. Context and Motivation

The Core Problem: Guidance Laws Cannot Handle Unmodeled Dynamics

The fundamental problem this paper addresses is deceptively simple: how do you design a spacecraft guidance system that works reliably when the actual dynamics of the environment differ substantially from what was modeled during design? Standard guidance laws—the algorithms that translate a spacecraft's current state (position, velocity) into control commands (thrust direction and magnitude)—are designed assuming a specific, known dynamical model: a known gravitational field, known vehicle mass properties, known actuator capabilities, and known external forces. When any of these assumptions break during the actual mission, the guidance law's performance degrades, potentially catastrophically.

The paper opens with a cascade of examples that make this gap tangible and pressing:

Orbital refueling. When two spacecraft dock and transfer fuel, the inertia tensor of the combined system changes continuously. The mass distribution shifts, and the dynamics that the guidance and control system was designed for no longer match reality. As the authors note (citing Reference 1), this makes the combined system "difficult to control"—a polite way of saying that standard controllers can become unstable when the plant they're controlling drifts outside their design envelope.

Exoatmospheric kill vehicles (EKVs). An EKV's wet mass consists largely of fuel. As divert thrusts deplete this fuel, the center of mass migrates, and the thrust vectors are no longer orthogonal to the velocity vector. The result is wasted fuel and degraded intercept performance—exactly the kind of gradual, continuous parameter shift that fixed-gain controllers handle poorly.

Asteroid missions with poorly characterized targets. An asteroid's gravitational field, rotational velocity, and local solar radiation pressure may not be accurately known before arrival. A mission might encounter an asteroid whose properties differ significantly from Earth-based estimates, and the guidance system must adapt on the fly rather than relying on pre-computed parameters.

Hypersonic re-entry. Aerodynamic models at hypersonic speeds are imperfect. The actual forces experienced during re-entry can deviate from predicted values due to atmospheric variability, ablation effects, and modeling approximations.

Sensor degradation. State estimation is never perfect. Sensors introduce bias, noise, and occasionally drop out entirely. A guidance system that assumes perfect state knowledge will degrade when fed corrupted estimates.

Actuator failure. Perhaps the most dramatic case: an engine fails, instantly and permanently changing the mapping from control commands to applied forces. The spacecraft now has reduced thrust capability along one axis, and the guidance law must compensate using the remaining healthy thrusters.

What unifies these scenarios is that they all involve time-varying or unknown system dynamics that are imperfectly modeled prior to the mission. The paper's central claim is that existing guidance approaches—which rely on explicit models of these dynamics—are fundamentally brittle in such settings, and that a different paradigm is needed.

Why This Problem Matters: Practical and Theoretical Significance

The practical significance is straightforward: space missions are expensive and risky, and guidance failures are catastrophic. A Mars lander that crashes because its guidance law couldn't handle a mass estimate error or unexpected wind represents not just a scientific loss but potentially billions of dollars and years of planning. The ability to deploy a single guidance system that works across a wide range of environmental conditions—without requiring per-mission re-tuning or explicit system identification—would represent a substantial improvement in mission robustness.

But the problem also carries theoretical weight. Guidance laws like DR/DV (Reference 5) are derived from optimal control theory: they solve a constrained optimization problem assuming a known dynamical model. The theoretical guarantees (optimality, stability, convergence) hold only when the model matches reality. What the paper is implicitly challenging is the entire paradigm of model-based guidance: if the model is wrong—and it almost always is, to some degree—then the "optimal" guidance law computed from that model may be far from optimal for the actual system.

This connects to a broader tension in control theory between robust control (designing controllers that maintain stability and performance within known bounds of model uncertainty) and adaptive control (designing controllers that estimate unknown parameters online and adjust their behavior accordingly). The paper's approach—using a recurrent neural network policy trained via reinforcement learning—represents a third path: learned adaptation, where the adaptation mechanism itself is trained from experience rather than designed analytically.

The paper frames this as an instance of meta-learning (learning to learn): the agent learns, across many training episodes with randomized dynamics, a strategy for quickly adapting to novel dynamics at test time. This is not just a practical engineering contribution—it's a conceptual reframing of what a guidance system is. Instead of being a hand-designed function from state to control, it becomes a learned dynamical system whose internal state evolves to capture unobserved environmental parameters.

Where Prior Approaches Fall Short

The paper identifies several lines of prior work and explains why they don't fully solve the problem.

Standard guidance laws (DR/DV). The DR/DV guidance law (Reference 5) is the baseline against which the paper compares its method. DR/DV is an optimal guidance law for planetary landing derived from a quadratic cost function with terminal constraints. Like most optimal guidance laws, it assumes a known dynamical model: known gravity, known vehicle mass and mass rate, known thrust limits. The paper gives the DR/DV baseline access to ground-truth gravitational force and the lander mass at the start of each episode—information the RL agent does not receive—yet the DR/DV law still fails catastrophically when dynamics deviate from nominal (Section "Experiments"). This establishes that even with privileged information, a model-based law has a brittle performance boundary that can be crossed by realistic perturbations.

Training with domain randomization (non-recurrent). The authors' own prior work (Reference 4) showed that training a standard feedforward (MLP) policy with randomized system parameters can improve robustness to parameter uncertainty. This is an important baseline: it demonstrates that simply exposing a policy to varied dynamics during training can produce a policy that works across a range of conditions. However, the paper notes that this approach works only "provided the parameter uncertainty is not too extreme." The MLP policy is essentially learning a single fixed mapping from observations to actions that must work across the entire range of training conditions. When the parameter variation becomes large, the optimal action for one set of dynamics may conflict with the optimal action for another, and the MLP policy—lacking any mechanism to condition its behavior on inferred dynamics—must settle for a compromise that degrades everywhere.

This is the critical limitation: an MLP policy cannot adapt at test time. Its behavior is entirely determined by its fixed network weights. It cannot infer, from the history of observations and actions, that "the current dynamics seem to have high gravity" or "the left engine appears to have failed" and modify its behavior accordingly. It treats every observation in isolation (or, at best, with a short concatenated history of observations as in Reference 6).

Explicit system identification + RL. Reference 2 proposes a two-stage approach: use a recurrent neural network to explicitly learn model parameters through real-time interaction with the environment, then use these estimated parameters to augment the observation for a standard RL algorithm. This is closer in spirit to the paper's approach but separates the system identification task from the policy task. The recurrent network learns to output parameter estimates (e.g., mass, gravity), and these estimates are then fed as additional inputs to the policy. The paper's approach is more integrated: the recurrent hidden state is not trained to output explicit parameter estimates but rather to evolve in whatever way is useful for the policy's task. This is a subtle but important distinction: the hidden state can capture any unobserved information that helps predict future rewards, not just the parameters an engineer would think to identify.

Recurrent policies for robotics (domain randomization). Reference 3 uses a recurrent policy and value function with a modified deep deterministic policy gradient (DDPG) algorithm to learn a policy for a robotic manipulator using real camera images. This work demonstrates the effectiveness of recurrent policies with domain randomization in a robotics context but focuses on sim-to-real transfer of a manipulation task. The paper builds on this idea but applies it to the quite different domain of spacecraft guidance, with its own specific challenges (long time horizons, terminal constraints, high-stakes sparse rewards).

Inverse reinforcement learning for sparse rewards. The paper mentions that the standard approach for handling sparse reward problems in RL is inverse reinforcement learning (Reference 14), where a per-timestep reward function is learned from expert demonstrations. The paper explicitly chooses not to use this approach, instead engineering a shaped reward function (Equations 14a–14e, 15) inspired by the gaze heuristic from biological interception behavior. This is a significant methodological choice: inverse RL requires expert demonstrations, which may not be available for novel mission scenarios, while the paper's hand-designed reward shaping is general and can be applied to any landing scenario given basic mission parameters (target location, desired terminal velocity constraints).

Multiple discount rates. The paper identifies an unresolved conflict in the RL literature: terminal rewards benefit from large discount rates (so the terminal bonus has significant influence on early actions), while shaping rewards benefit from smaller discount rates (to avoid overweighting distant shaping rewards). The authors state that "to our knowledge, a method of resolving this conflict has not been reported in the literature" and introduce their own framework for accommodating multiple discount rates (Equations 17a–17b). This is a small but novel contribution to the RL methodology.

How This Paper Positions Itself

The paper positions its approach—recurrent policy and value function trained with PPO under domain randomization—as a form of reinforcement meta-learning. The key insight is in the training setup: by training on randomized MDPs (where each episode samples different dynamics), the policy is forced to learn a strategy for adapting rather than a fixed behavioral mapping. The recurrent network's hidden state acts as an implicit representation of the current MDP's unknown parameters, learned not through supervised system identification but through the pressure of the RL objective: the hidden state must capture whatever information from the history of observations and actions is useful for selecting actions that lead to high rewards.

The contrast with prior work is sharpest on three dimensions:

1. Integration of system identification and policy. Unlike Reference 2, which separates parameter estimation from policy learning, the paper's recurrent policy learns an end-to-end mapping from observation history to actions. The hidden state is not forced to represent physically meaningful parameters—it can represent any latent information that helps the policy perform. This is both a strength (more flexible, can capture unanticipated patterns) and a limitation (less interpretable, harder to verify).

2. Robustness as a trained property vs. an engineered property. Unlike robust control theory, which provides formal guarantees within bounded uncertainty sets, the paper's approach provides empirical robustness—the policy works well on the test distribution of dynamics but with no theoretical guarantee on out-of-distribution scenarios. The paper implicitly argues that for the types of highly variable dynamics encountered in space missions, empirical robustness trained on wide randomization ranges is more practical than attempting to derive formal robustness guarantees.

3. Guidance and navigation as an integrated learned capability. Experiment 2 (landing using only Doppler radar altimeter returns) represents a significant departure from traditional aerospace practice. Conventionally, guidance and navigation are separate subsystems: the navigation filter estimates the vehicle state, and the guidance law takes that state estimate as input. By training end-to-end from raw sensor observations to control actions, the paper demonstrates that a recurrent policy can learn to implicitly perform navigation—extracting state information from sensor history—and guidance simultaneously. This blurs the traditional architectural boundary and suggests that for sufficiently complex sensing scenarios, learning an integrated policy may outperform the traditional separated design.

4. PPO with recurrent architectures and unrolling. The paper makes specific implementation contributions around training recurrent policies with PPO at scale. The challenge is that recurrent networks must be unrolled through time during the forward pass to capture temporal dependencies, but this unrolling interferes with the parallel batch processing that makes PPO efficient. The paper's solution—capturing hidden states during rollout collection and using them to initialize the unrolled segments during training (illustrated in Figure 1)—is a practical contribution that enables training recurrent policies on long-horizon tasks without sacrificing computational efficiency.

The paper's overall framing is that the combination of domain randomization during training and recurrent architecture at deployment produces a policy that adapts in real time to unknown dynamics, and that this approach outperforms both classical model-based guidance (DR/DV) and non-adaptive learned policies (MLP) across a range of challenging, dynamics-rich scenarios. The four experiments are chosen to stress-test different aspects of this claim: unknown environmental parameters (asteroid), partially observable state (radar altimetry), actuator failure (engine loss), and large internal parameter variation (mass depletion).

3. Technical Approach

3.1 Reader Orientation (Approachable Technical Breakdown)

The system being built is a learned adaptive guidance policy — a neural network that directly maps sensor observations to spacecraft thrust commands and continues to change its behavior in real time after deployment as it accumulates experience with the current environment. It solves the problem of landing a spacecraft safely when the environmental dynamics (gravity, mass, external forces, actuator health) are unknown at design time and must be inferred from the stream of observations during the descent. The "shape" of the solution is a recurrent neural network trained via reinforcement learning across thousands of simulated landings, each with randomly sampled dynamics, so that the network's internal memory (its hidden state) learns to extract implicit estimates of the unknown dynamics from the history of observations and actions, enabling the policy to behave appropriately for the actual environment it encounters rather than some nominal average.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major interacting components:

1. The Environment Simulator: A 3-DOF translational dynamics model (Equations 1a–1c) that propagates the lander's position, velocity, and mass forward in time given a thrust command. At the start of each training episode, it randomly samples environmental parameters (gravity, rotation, solar radiation pressure, initial mass, engine failure status) from wide uniform distributions. It also simulates sensor observations — either clean position/velocity-derived features or raw Doppler radar altimeter returns from a digital terrain map.

2. The Policy Network (Actor): A four-layer neural network with a gated recurrent unit (GRU) as its second layer. It takes an observation vector (e.g., velocity error, altitude, time-to-go, or raw altimeter readings) and outputs the parameters of a Gaussian distribution over thrust actions. The recurrent layer maintains a persistent hidden state that evolves based on the sequence of observations and actions, implicitly capturing unobserved environmental parameters.

3. The Value Function Network (Critic): A structurally identical four-layer network, also with a GRU as its second layer. It takes the same observation and outputs a scalar estimate of the expected sum of future discounted rewards from that state. This estimate is used to compute advantages — how much better or worse a particular action was compared to the average — which guides policy updates.

4. The PPO Training Loop: Proximal Policy Optimization (Equations 8–13) updates both networks using batches of trajectories (rollouts) collected from the environment. The policy is updated to increase the probability of actions that led to higher-than-expected rewards, while a clipping mechanism prevents destructively large updates. The value function is updated to better predict actual returns.

5. The Reward Function (Equations 14–15): A hand-engineered per-timestep reward that provides dense feedback by penalizing deviation from a target velocity field (derived from the gaze heuristic) and control effort, while adding a constant progress incentive and a large terminal bonus for successful soft landing.

Information flows as follows: the environment samples dynamics parameters → generates an initial observation → the policy's GRU hidden state processes this observation (augmented by its persistent memory) → the policy outputs thrust commands → the environment propagates the dynamics and computes a reward → the next observation is generated → the GRU hidden state updates based on the new observation and previous action → the cycle repeats until landing or constraint violation. During training, batches of these trajectories are fed through the unrolled recurrent networks (Figure 1) to compute policy and value function updates.

3.3 Roadmap for the Deep Dive

• First, the equations of motion (Equations 1a–1c), because every subsequent component — the environment sampling, the reward design, the policy's input features — depends on what dynamics the policy must control.
• Second, the reinforcement learning formulation: what PPO is, why it was chosen, how the advantage function and clipped objective work, and — critically — the novel multi-discount-rate framework (Equations 17a–17b) that resolves the tension between terminal and shaping rewards.
• Third, the recurrent policy and value function architecture (Table 1, Figure 1), including the specific implementation challenge of unrolling recurrent layers through time while maintaining parallel batch computation and the solution using captured hidden states.
• Fourth, the reward function design (Equations 14a–14e, 15): the gaze heuristic inspiration, the velocity field formulation, the piecewise structure for terminal descent, and the rationale for each term's coefficient.
• Fifth, the observation design (Equation 16) and why the chosen features — relative velocity error, altitude, time-to-go — produce a policy that generalizes beyond the training state distribution.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a methodology and empirical evaluation paper whose core idea is that a recurrent neural network policy, trained via PPO with domain randomization across randomized dynamical systems, learns to adapt its behavior in real time to unobserved environmental parameters through the evolution of its hidden state — and that this adaptive capability substantially outperforms both classical model-based guidance laws and non-recurrent learned policies when deployed in environments with highly variable or unknown dynamics.

---

Equations of Motion (The Environment Being Controlled)

The landing dynamics are modeled in 3 degrees of freedom (3-DOF), meaning the lander can translate in three dimensions (downrange, crossrange, elevation) but rotational dynamics are not explicitly modeled — the lander is treated as a point mass that can apply thrust in any direction. This is a simplification that keeps the control problem tractable while capturing the essential challenges of planetary landing: managing velocity, position, and fuel consumption under time-varying forces.

The translational motion is governed by:

$\dot{\mathbf{r}} = \mathbf{v}$

$\dot{\mathbf{v}} = \frac{\mathbf{T} + \mathbf{F}_{\text{env}}}{m} + \mathbf{g} + 2\dot{\mathbf{r}}_a \times \boldsymbol{\omega} + (\boldsymbol{\omega} \times \mathbf{r}_a) \times \boldsymbol{\omega}$

$\dot{m} = -\frac{\|\mathbf{T}\|}{I_{sp} g_{\text{ref}}}$

where $\mathbf{r}$ is the lander's position vector in the target-centered reference frame (origin at the desired landing site), $\mathbf{v}$ is the velocity vector in the same frame, $\mathbf{T}$ is the thrust vector applied by the lander's engines (the control input), $\mathbf{F}_{\text{env}}$ is a vector of normally distributed random variables representing environmental disturbances such as wind and atmospheric density variations, $m$ is the lander's current mass, $\mathbf{g}$ is the gravitational acceleration vector (nominally $[0, 0, -3.7114]^T$ m/s² for Mars), $\mathbf{r}_a$ is the lander's position in the planet-centered (or asteroid-centered) reference frame, $\boldsymbol{\omega}$ is the rotational velocity vector of the central body, $I_{sp} = 225$ s is the engine specific impulse, and $g_{\text{ref}} = 9.8$ m/s² is the reference gravitational acceleration used to convert specific impulse to effective exhaust velocity.

What the position equation computes: the simple kinematic relationship that velocity is the time derivative of position. The position in the target frame updates by integrating the velocity.

What the velocity equation computes: Newton's second law divided by mass, giving the acceleration. The four terms represent: (1) the control acceleration from thrust and environmental disturbances, (2) the gravitational acceleration, (3) the Coriolis acceleration $2\dot{\mathbf{r}}_a \times \boldsymbol{\omega}$ (which arises because the target-centered frame rotates with the central body), and (4) the centrifugal acceleration $(\boldsymbol{\omega} \times \mathbf{r}_a) \times \boldsymbol{\omega}$ (also from frame rotation). For the Mars landing experiments, $\boldsymbol{\omega} = \mathbf{0}$, so the Coriolis and centrifugal terms vanish and only thrust, disturbance, and gravity matter. For the asteroid landing (Experiment 1), $\boldsymbol{\omega}$ is non-zero and randomly sampled, making these rotational terms a significant source of unknown dynamics.

What the mass equation computes: the rate at which fuel is consumed, proportional to the magnitude of thrust (the rocket equation in differential form). The denominator $I_{sp} g_{\text{ref}}$ is the effective exhaust velocity — for the nominal Mars lander with $I_{sp} = 225$ s, this is $225 \times 9.8 = 2205$ m/s, meaning each Newton of thrust consumes approximately $1/2205 \approx 0.00045$ kg/s of fuel. For the high-mass-variation experiment (Experiment 4), the specific impulse is divided by a factor of 6, increasing fuel consumption sixfold and making the mass change a much larger fraction of the initial mass during the descent.

Why this form: the 3-DOF formulation captures the essential control challenges — managing position and velocity in three dimensions under thrust, gravity, and rotation — without the additional complexity of attitude dynamics (orientation and angular velocity). This is a standard simplification for guidance law development: the guidance system outputs a desired thrust vector, and a separate attitude control system (not modeled here) orients the spacecraft to apply that thrust. The inclusion of the rotational terms ($\boldsymbol{\omega}$) is specifically motivated by the asteroid landing scenario, where the asteroid's rotation can produce Coriolis and centrifugal forces comparable to or larger than the gravitational force, creating a genuinely unknown dynamical environment. The environmental disturbance term $\mathbf{F}_{\text{env}}$ is modeled as normally distributed random variables rather than a structured wind model, which provides stochasticity that encourages robust policy learning without requiring a detailed atmospheric model.

Key constraints on the thrust: for Mars landing, the thrust magnitude is constrained to the range $[2000, 15000]$ N, and the nominal wet mass is 2000 kg. This gives a maximum acceleration of $15000/2000 = 7.5$ m/s² at the start of the descent, which increases as fuel is consumed. For the asteroid landing, the lander uses pulsed thrusters with 2 N capability along each axis in the target-centered frame, and the wet mass is randomly sampled between 450 and 500 kg, giving a maximum acceleration of approximately $2/450 \approx 0.0044$ m/s² per axis — extremely low, making precise control essential.

---

Proximal Policy Optimization (PPO) — The Core Learning Algorithm

The paper uses Proximal Policy Optimization (PPO, Reference 9) as the reinforcement learning algorithm to train both the policy and value function. PPO is a policy gradient method, meaning it directly optimizes the parameters of the policy (the neural network weights) to maximize expected cumulative reward, rather than learning a value function and deriving the policy from it (as in Q-learning). The choice of PPO is significant because it addresses a fundamental challenge in policy gradient methods: how to take large enough update steps to learn efficiently without taking steps so large that the policy collapses (a phenomenon known as "catastrophic forgetting" in the RL context).

To understand PPO, we need to build up from the basic RL objective. The paper frames the problem as optimizing:

$J(\theta) = \mathbb{E}_{p(\tau)} \left[ r(\tau) \right] = \int_T r(\tau) p_\theta(\tau) d\tau$

where $J(\theta)$ is the expected total reward over trajectories, $\theta$ are the policy network parameters, $\tau = [\mathbf{x}_0, \mathbf{u}_0, ..., \mathbf{x}_T, \mathbf{u}_T]$ is a trajectory (sequence of observations and actions), $r(\tau) = \sum_{i=0}^T \gamma^i r_k(\mathbf{x}_k, \mathbf{u}_k)$ is the sum of discounted rewards along the trajectory with discount factor $\gamma \in [0, 1)$, and $p_\theta(\tau)$ is the probability of the trajectory under the current policy.

What it computes: the expected cumulative discounted reward when following the policy parameterized by $\theta$. The expectation is over trajectories, which are generated by the interaction of the policy's action distribution and the environment's transition dynamics.

Why this form: this is the standard objective in episodic reinforcement learning. The discount factor $\gamma$ serves two purposes: it makes the infinite-horizon sum finite (though all episodes here are finite), and it controls the temporal credit assignment horizon — smaller $\gamma$ means the agent cares more about immediate rewards, larger $\gamma$ means it values distant rewards nearly as much as immediate ones. The paper uses multiple discount rates (Equations 17a–17b), which we will address separately.

The policy gradient theorem (Reference 8) gives the gradient of this objective with respect to the policy parameters:

$\nabla_\theta J(\theta) \approx \sum_{i=0}^M \sum_{k=0}^T A^\pi_w(\mathbf{x}^i_k, \mathbf{u}^i_k) \nabla_\theta \log \pi_\theta(\mathbf{u}^i_k | \mathbf{x}^i_k)$

where $M$ is the number of trajectory samples (the batch size), $T$ is the trajectory length, $A^\pi_w(\mathbf{x}_k, \mathbf{u}_k) = Q^\pi(\mathbf{x}_k, \mathbf{u}_k) - V^\pi(\mathbf{x}_k)$ is the advantage function (how much better action $\mathbf{u}_k$ is compared to the average action at state $\mathbf{x}_k$), $\pi_\theta(\mathbf{u}_k | \mathbf{x}_k)$ is the probability of taking action $\mathbf{u}_k$ given observation $\mathbf{x}_k$ under the current policy, and $\nabla_\theta \log \pi_\theta$ is the score function gradient.

What it computes: an estimate of the gradient of expected reward with respect to policy parameters. For each action taken in each trajectory, we check whether it was better than expected (positive advantage) or worse (negative advantage), and adjust the policy to make better actions more likely and worse actions less likely. The sum over trajectories $i$ and timesteps $k$ is a Monte Carlo estimate of the expectation.

Why the advantage function rather than raw reward: subtracting the state-dependent baseline $V^\pi(\mathbf{x}_k)$ reduces the variance of the gradient estimate without introducing bias. Without this subtraction, a policy that always receives positive rewards would increase the probability of all actions taken — including mediocre ones — making learning slow and unstable. The advantage tells us which actions are genuinely better than typical for that state.

Now PPO: the key innovation of PPO over simpler policy gradient methods is how it constrains the size of policy updates. The problem with unrestricted policy gradients is that a single batch of data might suggest a large parameter update that actually degrades performance on the next batch — the policy "overfits" to the current batch. PPO addresses this by optimizing a clipped surrogate objective:

$L(\theta) = \mathbb{E}_{p(\tau)} \left[ \min \left( p_k(\theta), \text{clip}(p_k(\theta), 1 - \epsilon, 1 + \epsilon) \right) A^\pi_w(\mathbf{x}_k, \mathbf{u}_k) \right]$

where $p_k(\theta) = \frac{\pi_\theta(\mathbf{u}_k | \mathbf{x}_k)}{\pi_{\theta_{\text{old}}}(\mathbf{u}_k | \mathbf{x}_k)}$ is the probability ratio — how much more (or less) likely the current policy makes action $\mathbf{u}_k$ compared to the old policy that collected the data, $\text{clip}(p_k(\theta), 1 - \epsilon, 1 + \epsilon)$ constrains this ratio to the interval $[1-\epsilon, 1+\epsilon]$, and $\epsilon$ is a hyperparameter controlling the maximum policy change per update (the paper dynamically adjusts $\epsilon$ to target a KL divergence between policy updates of 0.001).

What it computes: a conservative estimate of the policy improvement. For each action, if the advantage is positive, the objective wants to increase $p_k(\theta)$ (make the action more likely), but the clip prevents $p_k(\theta)$ from exceeding $1 + \epsilon$. If the advantage is negative, the objective wants to decrease $p_k(\theta)$, but the clip prevents it from going below $1 - \epsilon$. The $\min$ operation ensures we never get credit for moving the ratio beyond the clipped range if doing so would improve the objective — this is the "pessimistic" part of PPO that prevents the policy from exploiting the objective function by making extreme changes.

Why this form: the clipping mechanism is PPO's key contribution over TRPO (Trust Region Policy Optimization, Reference 10). TRPO enforces a constraint on the KL divergence between old and new policies using a second-order optimization procedure that is computationally expensive and complex to implement. PPO approximates the same constraint with a simple first-order clipping mechanism, achieving similar stability at much lower computational cost. In the paper's implementation, the clipping parameter $\epsilon$ is dynamically adjusted to target a KL divergence of 0.001 between updates — this adaptive scheme eliminates the need to manually tune $\epsilon$ and ensures that the effective step size remains appropriate throughout training as the policy landscape changes.

The value function $V^\pi_w$ is trained concurrently using a mean squared error loss:

$L(w) = \sum_{i=1}^M \left( V^\pi_w(\mathbf{x}^i_k) - \left[ \sum_{\ell=k}^T \gamma^{\ell-k} r(\mathbf{u}^i_\ell, \mathbf{x}^i_\ell) \right] \right)^2$

where $V^\pi_w(\mathbf{x}^i_k)$ is the value function's prediction for observation $\mathbf{x}^i_k$, and the term in brackets is the empirical return — the sum of discounted rewards actually received from timestep $k$ to the end of the episode.

What it computes: the squared error between predicted and actual returns. The value function learns to predict how much total discounted reward the agent can expect to receive starting from a given observation and following the current policy thereafter.

Why this form: accurate value prediction is essential for computing good advantage estimates. If the value function systematically overestimates or underestimates, the advantages will be biased and the policy gradient will point in the wrong direction. MSE is the standard regression loss and works well in practice for value function fitting.

The updates are performed using gradient descent on the value function and gradient ascent on the policy:

$\mathbf{w}^+ = \mathbf{w}^- - \beta_w \nabla_w L(w)|_{w=w^-}$

$\boldsymbol{\theta}^+ = \boldsymbol{\theta}^- + \beta_\theta \nabla_\theta J(\theta)|_{\theta=\theta^-}$

where $\beta_w$ and $\beta_\theta$ are the learning rates for the value function and policy, respectively.

The policy's action distribution: the paper uses a Gaussian distribution with mean $\pi_\theta(\mathbf{x}_k)$ (the output of the policy network) and a diagonal covariance matrix. Actions are sampled from this distribution during training (exploration), and during deployment the mean is used deterministically (exploitation). Because the log probabilities used in the policy gradient are calculated using the exploration variance, "the degree of exploration automatically adapts during learning such that the objective function is maximized" — meaning the variance itself can be a learned parameter that the policy adjusts to control its exploration-exploitation tradeoff.

---

Multi-Discount-Rate Framework (A Novel Methodological Contribution)

The paper identifies and resolves a previously unreported conflict in RL reward design: the optimal discount rate for terminal rewards differs from the optimal discount rate for shaping rewards. The tension arises because:

• Terminal rewards (the bonus $\eta$ for a successful landing) should have a large discount factor (close to 1) so that the terminal bonus has significant influence on actions taken early in the episode. If the discount factor is too small, the terminal bonus is heavily discounted and provides negligible learning signal for early decisions.

• Shaping rewards (the velocity tracking error, control effort, and progress incentive) should have a smaller discount factor to avoid overweighting distant shaping rewards. Since shaping rewards are given at every timestep, a large discount factor would make early actions care too much about shaping rewards far in the future, which can distort the learning signal.

The paper states: "to our knowledge, a method of resolving this conflict has not been reported in the literature." Their solution is a framework that assigns different discount rates to different components of the reward function. Let $r_1(k)$ be the reward term associated with the terminal bonus coefficient $\eta$, and $r_2(k)$ be the sum of all other reward terms ($\alpha$ velocity tracking error, $\beta$ control effort, $\gamma$ progress incentive). Let $\gamma_1$ be the discount rate for $r_1$ and $\gamma_2$ be the discount rate for $r_2$. Then:

Advantage function with multiple discounts:

$A^\pi_w(\mathbf{x}_t, \mathbf{u}_t) = \left[ \sum_{\tau=t}^n \gamma_1^{\tau-t} r_1(\mathbf{u}_\tau, \mathbf{x}_\tau) + \gamma_2^{\tau-t} r_2(\mathbf{u}_\tau, \mathbf{x}_\tau) \right] - V^\pi_w(\mathbf{x}_k)$

Value function loss with multiple discounts:

$J(w) = \sum_{i=1}^M \left( V^\pi_\theta(\mathbf{x}_k) - \left[ \sum_{\tau=t}^n \gamma_1^{\tau-t} r_1(\mathbf{u}_\tau, \mathbf{x}_\tau) + \gamma_2^{\tau-t} r_2(\mathbf{u}_\tau, \mathbf{x}_\tau) \right] \right)^2$

where the notation is identical to Equations 10–11, except that two discount factors are used instead of one, applied separately to the two reward components.

What it computes: the advantage and value target are now computed as sums of discounted rewards where different reward components are discounted at different rates. The terminal bonus $r_1$ is discounted by $\gamma_1$ (large, close to 1), while the shaping rewards $r_2$ are discounted by $\gamma_2$ (smaller). At each timestep $\tau$, the total discounted reward contribution is $\gamma_1^{\tau-t} r_1 + \gamma_2^{\tau-t} r_2$.

Why this form: this decouples the horizons over which different reward signals propagate. The terminal bonus can have a long effective horizon (influencing early actions) while the shaping rewards have a shorter effective horizon (focusing learning on immediate velocity tracking). This addresses the fundamental tension that a single discount rate forces a compromise between these competing needs. The paper reports that this simple modification provides "significant" performance improvement, exceeding the improvement from using Generalized Advantage Estimation (GAE, Reference 16) — a more complex method for reducing advantage variance — both with and without multiple discount rates. Moreover, the authors note that "without the use of multiple discount rates, the performance was actually worsened by including the terminal reward term," indicating that the terminal bonus and shaping rewards can actively interfere with each other under a single discount rate.

This is a small but genuine methodological contribution: prior work using shaped rewards with terminal bonuses either accepted the compromise of a single discount rate or used explicit reward annealing schedules, but the idea of running multiple discount factors in parallel and summing their discounted contributions is, to the authors' knowledge, novel.

---

Recurrent Policy and Value Function Architecture

The central technical innovation is the use of recurrent neural network layers (specifically, Gated Recurrent Units, GRUs) in both the policy and value function networks. The architecture is given in Table 1 of the paper:

Policy Network:

• Hidden layer 1: $10 \times \text{obs\_dim}$ units, tanh activation
• Hidden layer 2 (recurrent): $\sqrt{nh_1 \times nh_3}$ units, tanh activation (where $nh_1$ is the size of hidden layer 1 and $nh_3$ is the size of hidden layer 3)
• Hidden layer 3: $10 \times \text{act\_dim}$ units, tanh activation
• Output layer: $\text{act\_dim}$ units, linear activation

Value Function Network:

• Hidden layer 1: $10 \times \text{obs\_dim}$ units, tanh activation
• Hidden layer 2 (recurrent): $\sqrt{nh_1 \times nh_3}$ units, tanh activation (with $nh_3 = 5$ for the value function's hidden layer 3)
• Hidden layer 3: 5 units, tanh activation
• Output layer: 1 unit, linear activation

The observation dimension $\text{obs\_dim}$ is 8 for the standard Mars landing tasks (3 velocity error components, 3 target velocity components? — the paper is slightly ambiguous here — plus altitude and time-to-go, making 8 total features given the observation vector in Equation 16 has $\mathbf{v}_{\text{error}} \in \mathbb{R}^3$, $r_2 \in \mathbb{R}$, and $t_{\text{go}} \in \mathbb{R}$, which would be 5 features; the 8 likely comes from including both $\mathbf{v}$ and $\mathbf{v}_{\text{targ}}$ separately). The action dimension $\text{act\_dim}$ is 3 (thrust in downrange, crossrange, and elevation directions).

How the recurrent layer produces adaptation: the GRU layer maintains a persistent hidden state vector that is updated at each timestep based on the current input (the output of hidden layer 1) and the previous hidden state. This hidden state is not reset between episodes during deployment — it evolves continuously as the lander descends. The crucial insight is:

"during training, the hidden state of a network's recurrent network evolves differently depending on the observed sequence of observations from the environment and actions output by the policy"

Since the ground-truth dynamics (mass, gravity, external forces, engine health) affect the mapping from actions to next observations, different dynamics produce different sequences of observations for the same policy. The hidden state, which processes this sequence, will therefore evolve to different values depending on the actual dynamics. Because the policy's weights (including the GRU weights) are optimized to maximize the likelihood of actions that lead to high advantages, the hidden state is implicitly trained to capture whatever unobserved information is useful for selecting good actions. The paper explicitly states:

"the trained policy's hidden state captures unobserved information such as external forces and the current lander mass, as well as the past history of observations and actions, as this information is useful in minimizing the cost function"

After training, the adaptation continues: "although the recurrent policy's network weights are frozen, the hidden state will continue to evolve in response to a sequence of observations and actions, thus making the policy adaptive." This is the key distinction from an MLP (non-recurrent) policy: "In contrast, an MLP policy's behavior is fixed by the network parameters at test time." An MLP can only respond to the current observation; it has no memory of past observations that might indicate, for example, that thrust commands are producing less acceleration than expected (implying higher mass or engine degradation).

Why GRU rather than LSTM or simple RNN: the paper implements the recurrent layers as Gated Recurrent Units (GRU, Reference 13) specifically "to avoid issues with exploding / vanishing gradients when we back propagate through the unrolled recurrent layer." GRUs use gating mechanisms (reset and update gates) that allow the network to learn to maintain information over long sequences without the gradient instability that plagues simple RNNs. Compared to LSTMs, GRUs have fewer parameters (two gates instead of three) and have been shown to perform comparably on many sequence tasks, making them a practical choice.

The parallel unrolling challenge (Figure 1): training recurrent policies at scale presents a computational challenge. To compute gradients through the recurrent layer, the network must be "unrolled" through time — the same GRU cell is applied sequentially to each timestep, and gradients flow backward through this temporal chain. However, PPO processes batches of trajectories in parallel for efficiency. The conflict: if we want to unroll the network for $T$ steps, the hidden state at step $k+1$ depends on the output at step $k$, creating a sequential dependency that prevents parallel computation across timesteps.

The paper's solution (illustrated in Figure 1): during data collection (rollout), capture the GRU hidden state at each timestep and store it alongside the observation, action, and reward. During training, instead of unrolling continuously from step 0 to step $T$, segment the trajectories into chunks and use the stored hidden state as the initial state for each chunk. Specifically:

"prior to the recurrent layer, the network unrolls the output of the previous layer, reshaping the data from $\mathbb{R}^{m \times n}$ to $\mathbb{R}^{T \times m/T \times n}$, where $m$ is the batch size, $n$ the feature dimension, and $T$ the number of steps we unroll the network"

At step zero of each unrolled segment, the hidden state from the rollouts is injected as the initial state; for subsequent steps, the hidden state evolves according to the current parameterization of the GRU. This allows different segments of different trajectories to be processed in parallel batches while still maintaining temporal continuity within each segment.

Why this matters: without this technique, training a recurrent policy on long trajectories (60–200 timesteps per episode) with large batch sizes would be prohibitively slow — the forward pass would need to be computed sequentially timestep by timestep for each trajectory, losing the parallelism that makes GPU training efficient. The paper's approach recovers parallelism by treating each $T$-step unrolled segment as an independent computation that can be batched, with the stored hidden states providing the temporal linkage between segments.

Several implementation details are crucial to get this right:

• Padding: rollouts must be padded so that $m/T$ is an integer (the batch size must be divisible by the unroll length), but the padding must be removed before computing the loss to avoid contaminating the gradient with padded values.
• No shuffling: "It is also important to not shuffle the training data, as this destroys the temporal association in the trajectories captured in the rollouts." Standard ML training shuffles data to break correlations, but for recurrent networks the sequential order within a trajectory is essential information.
• Unroll length as a hyperparameter: The paper experiments with unrolling lengths of 1, 20, 60, 120, and 200 steps (referred to as "T-step RNN"). Longer unrolling captures longer temporal dependencies but increases computational cost and may suffer from vanishing/exploding gradient issues. The results show that longer unrolling generally improves performance, particularly for tasks with long temporal dependencies (Experiment 2, radar altimetry) and that performance can stall or degrade if unrolling is too long relative to the effective temporal horizon (the 120-step RNN in Experiment 2 initially improves but then stalls).

Recurrent policies as meta-learning: the paper frames this approach as a form of reinforcement meta-learning. In meta-learning, an agent learns across a distribution of tasks (MDPs) a strategy for quickly adapting to a new task. Here, each randomized dynamical system (different gravity, mass, rotation, engine failure status) is a different MDP. The recurrent policy is trained across this distribution and learns to use its hidden state to encode a representation of the current MDP — not by being explicitly trained to output parameter estimates, but by being trained to output actions that maximize reward, which requires implicitly inferring the dynamics that produced the observed history.

Recurrent policies for POMDPs: the paper also notes that recurrent policies can handle partially observable Markov decision processes (POMDPs), where the observation does not satisfy the Markov property — the optimal action depends on more than just the current observation. Experiment 2 (radar altimetry) is an explicit POMDP: "there are multiple ground truth positions and velocities that could correspond to a given observation, making the optimal action a function of the history of past altimeter readings." The recurrent hidden state can integrate information over time, similar to how a recursive Bayesian filter (e.g., a Kalman filter) can estimate velocity from a history of position measurements. But the paper claims an advantage: "a recurrent network has the ability to capture much longer temporal dependencies than a Kalman filter" — the non-linear, high-capacity GRU can learn to extract information from observation sequences in ways that go beyond the linear-Gaussian assumptions of classical filtering.

---

Reward Function Design — The Gaze Heuristic and Velocity Field Shaping

The reward function is arguably the most critical engineering component of the system, because reinforcement learning with sparse rewards — where the agent only receives a signal upon successful landing — would require an astronomically large number of episodes to discover a successful policy through random exploration. The paper notes: "If we only reward the agent for making a soft pinpoint landing at the correct attitude and with close to zero rotational velocity, the agent would never see the reward within a realistic number of episodes, as the probability of achieving such a landing using random actions in a 3-DOF environment with realistic initial conditions is exceedingly low."

The standard solution in RL is inverse reinforcement learning (Reference 14), where a per-timestep reward function is learned from expert demonstrations. The paper explicitly rejects this approach in favor of a hand-engineered reward shaping function based on the gaze heuristic.

The gaze heuristic: this is a biological interception strategy used by predators (hawks, cheetahs) and humans (baseball players catching fly balls) where the interceptor keeps the line-of-sight angle to the target constant. If the line-of-sight angle is not changing, the interceptor is on a collision course. This is also the basis of proportional navigation (PN), the standard homing-phase guidance law for missiles. The paper adapts this idea for planetary landing, where the target is not moving (it's fixed on the surface) but there is the additional constraint of achieving a soft (low-velocity) landing.

The paper's key insight: "since the target is not moving in the target-centered reference frame, the target's future position is its current position, and the optimal action is to head directly towards the target." This simplifies the gaze heuristic: the lander should keep its velocity vector aligned with the line-of-sight vector (the vector pointing from the lander to the target). To achieve a soft landing, the lander should reduce its targeted velocity as the time-to-go decreases, where time-to-go is estimated as the ratio of range to speed.

The reward function implements this heuristic by defining a target velocity field — for each position of the lander, there is a desired velocity vector — and penalizing the lander for deviating from this target. The target velocity is given by:

$\mathbf{v}_{\text{targ}} = -v_o \left( \frac{\hat{\mathbf{r}}}{\|\hat{\mathbf{r}}\|} \right) \left( 1 - \exp\left( -\frac{t_{\text{go}}}{\tau} \right) \right)$

$t_{\text{go}} = \frac{\|\hat{\mathbf{r}}\|}{\|\hat{\mathbf{v}}\|}$

where $\mathbf{v}_{\text{targ}}$ is the target velocity vector (what the lander's velocity should be at its current position), $v_o = \|\mathbf{v}_o\|$ is the magnitude of the lander's velocity at the start of powered descent (a scaling factor), $\hat{\mathbf{r}}$ is a modified position vector (defined below in Equation 14c), $\|\hat{\mathbf{r}}\|$ is the distance to the target, $t_{\text{go}}$ is the estimated time-to-go (range divided by current speed), and $\tau$ is a time constant controlling how aggressively the velocity target decreases.

What it computes: a desired velocity vector that always points toward the target (the negative sign and the unit vector $-\hat{\mathbf{r}}/\|\hat{\mathbf{r}}\|$) with a magnitude that starts at $v_o$ when far away (when $t_{\text{go}} \gg \tau$, the exponential term is near zero and the velocity target approaches $v_o$) and decreases smoothly to zero as the lander approaches the target (as $t_{\text{go}} \to 0$, the exponential approaches 1 and the velocity target approaches zero). The time constant $\tau$ controls the "aggressiveness" of the deceleration — smaller $\tau$ means the lander slows down later and more abruptly.

Why this form: the exponential shaping provides a smooth velocity profile that naturally transitions from high-speed approach to low-speed terminal descent. A linear velocity taper ($\mathbf{v}_{\text{targ}} \propto t_{\text{go}}$) would produce constant deceleration, which is fuel-inefficient for planetary landing (it wastes fuel fighting gravity early in the descent). The exponential profile concentrates deceleration near the end, which is more fuel-efficient and also more robust to navigation errors early in the descent.

The piecewise position and velocity definitions: the target landing point is not at the surface — it's 15 meters above the desired landing site. This is encoded in the piecewise definitions:

$\hat{\mathbf{r}} = \begin{cases} \mathbf{r} - \begin{bmatrix} 0 \\ 0 \\ 15 \end{bmatrix}, & \text{if } r_2 > 15 \\ \begin{bmatrix} 0 \\ 0 \\ r_2 \end{bmatrix}, & \text{otherwise} \end{cases}$

$\hat{\mathbf{v}} = \begin{cases} \mathbf{v} - \begin{bmatrix} 0 \\ 0 \\ -2 \end{bmatrix}, & \text{if } r_2 > 15 \\ \mathbf{v} - \begin{bmatrix} 0 \\ 0 \\ -1 \end{bmatrix}, & \text{otherwise} \end{cases}$

where $r_2$ is the elevation (z-component) of the lander's position.

What these compute: above 15 meters altitude, the target position is offset 15 meters above the landing site (the lander aims for a point in the sky above the target), and the target velocity has a -2 m/s downward component (the lander should be descending at 2 m/s). Below 15 meters, the target position's downrange and crossrange components are zeroed out (encouraging the lander to be directly above the landing site), and the target velocity's downrange and crossrange components are zeroed out (encouraging pure vertical descent), with a reduced downward velocity target of -1 m/s.

Why this form: the 15-meter handoff altitude creates a two-phase descent: a powered approach phase where the lander navigates to a point above the target while managing its velocity, and a terminal vertical descent phase where the lander drops straight down at low speed. This is how real planetary landers operate (e.g., the Mars Science Laboratory's skycrane maneuver involved a powered descent to ~20 meters altitude followed by a constant-velocity descent). The piecewise structure encodes this operational concept into the reward function. The reduction from -2 m/s to -1 m/s at low altitude provides an additional safety margin for touchdown.

The time constant is also piecewise:

$\tau = \begin{cases} \tau_1, & \text{if } r_2 > 15 \\ \tau_2, & \text{otherwise} \end{cases}$

with the Mars landing hyperparameters set to $\tau_1 = 20$ s and $\tau_2 = 100$ s (Table 2). The larger $\tau_2$ below 15 meters means the velocity target decreases more slowly during terminal descent, producing a gentler deceleration.

The total per-timestep reward:

$r = \alpha \|\mathbf{v} - \mathbf{v}_{\text{targ}}\| + \beta \|\mathbf{T}\| + \gamma + \eta \cdot \mathbb{1}(r_2 < 0 \text{ and } \|\mathbf{r}\| < r_{\text{lim}} \text{ and } \|\mathbf{v}\| < v_{\text{lim}} \text{ and } gs > gs_{\text{min}})$

where $\alpha = -0.01$ weights the velocity tracking error penalty, $\beta = -0.05$ weights the control effort penalty (fuel consumption), $\gamma = 0.01$ is a constant positive progress incentive, $\eta = 10$ is the terminal bonus for a successful landing, $r_{\text{lim}} = 5$ m is the maximum allowable position error at touchdown, $v_{\text{lim}} = 2$ m/s is the maximum allowable velocity magnitude at touchdown, and $gs_{\text{min}} = 5$ rad/s is the minimum glideslope (the angle between the velocity vector and the horizontal plane — requiring $gs > 5$ rad/s ensures the velocity is directed predominantly downward at touchdown).

What each term does:

1. Velocity tracking penalty ($\alpha \|\mathbf{v} - \mathbf{v}_{\text{targ}}\|$): this is the main shaping term that drives the lander toward the desired velocity field. It penalizes the Euclidean distance between the lander's current velocity and the target velocity for its current position. It is always negative (since $\alpha$ is negative), so the agent is incentivized to minimize this distance.

2. Control effort penalty ($\beta \|\mathbf{T}\|$): penalizes the magnitude of thrust, incentivizing fuel-efficient trajectories. Since $\beta$ is negative, using more thrust yields more negative reward. This term creates the fundamental trade-off: the agent must balance tracking the target velocity (which requires thrust) against conserving fuel.

3. Progress incentive ($\gamma$): a constant small positive reward at every timestep. Since all other terms are negative (penalties), without this term the optimal policy for maximizing discounted reward would be to violate a constraint as quickly as possible and terminate the episode — since all rewards are negative, a shorter episode has a less negative cumulative reward. The constant positive term "encourages the agent to keep making progress along the trajectory" and prevents this pathological behavior.

4. Terminal bonus ($\eta \cdot \mathbb{1}(\text{success})$): a large positive reward given once at the end of the episode if the lander achieves a soft landing within the specified constraints. The indicator function checks four conditions simultaneously: altitude below zero (the lander has reached the surface), position error within 5 meters, velocity magnitude below 2 m/s, and glideslope above 5 rad/s (velocity directed predominantly downward).

Why these specific coefficient values: the relative magnitudes encode the designer's priorities. $\beta$ is five times larger in magnitude than $\alpha$ ($-0.05$ vs. $-0.01$), placing strong emphasis on fuel efficiency. $\gamma = 0.01$ is small enough not to distort the velocity tracking objective but large enough to prevent the short-episode pathology. $\eta = 10$ is substantially larger than the per-timestep rewards, making the terminal bonus the dominant learning signal — but because it only appears at the end of successful episodes, it would be useless without the shaping rewards to guide exploration toward success. This is a carefully balanced ecosystem of reward components.

The asteroid landing uses modified shaping: for the asteroid environment (Equations 18a–18b), the target velocity formula is identical but the hyperparameters change: $v_o = 1$ m/s, $\tau = 300$ s, $\alpha = -1.0$, $\beta = -0.01$, $\gamma = 0.01$, $\eta = 10$, $r_{\text{lim}} = 1$ m, $v_{\text{lim}} = 0.2$ m/s, and $gs_{\text{min}} = 0$ (no glideslope constraint, since the asteroid's irregular shape means "vertical" is ill-defined). The much larger $\alpha$ magnitude ($-1.0$ vs. $-0.01$) reflects the different scale of velocities (cm/s vs. m/s) and the need for precise velocity tracking in the ultra-low-gravity environment. The single $\tau = 300$ s (no piecewise split) reflects the simpler two-phase descent strategy.

---

Observation Design for Generalization

The observation given to the agent during both training and testing is:

$\text{obs} = \begin{bmatrix} \mathbf{v}_{\text{error}} \\ r_2 \\ t_{\text{go}} \end{bmatrix}$

where $\mathbf{v}_{\text{error}} = \mathbf{v} - \mathbf{v}_{\text{targ}}$ is the velocity tracking error, $r_2$ is the lander's altitude (elevation), and $t_{\text{go}} = \|\hat{\mathbf{r}}\| / \|\hat{\mathbf{v}}\|$ is the time-to-go estimate.

What this observation encodes: the velocity error tells the agent how far its current velocity is from the target velocity and in what direction. The altitude provides absolute position context (how close to the surface). The time-to-go provides a normalized measure of distance that accounts for current speed — it's a single scalar that captures whether the lander is approaching too fast (small $t_{\text{go}}$ relative to altitude) or too slow (large $t_{\text{go}}$).

What is NOT in the observation: "aside from the altitude, the lander translational coordinates do not appear in the observation." The lander's absolute position (downrange and crossrange) is not provided, nor is the lander's mass, nor the gravitational acceleration, nor the external forces. This is a deliberate design choice that produces "a policy with good generalization in that the policy's behavior can extend to areas of the full state space that were not experienced during learning."

Why this works: by expressing the control problem in terms of velocity error and time-to-go, rather than absolute position and velocity, the policy learns a behavior that is invariant to the specific location. A lander at position (1000, 500, 2000) with a certain velocity error and time-to-go should behave the same as a lander at position (-500, 300, 1500) with the same velocity error and time-to-go, because the dynamics (the equations of motion) are translation-invariant in the downrange-crossrange plane. By removing absolute position from the observation, the policy cannot overfit to specific regions of the state space and is forced to learn this translation-invariant behavior.

The DR/DV guidance law, by contrast, is given privileged information: it has access to "the ground truth gravitational force, as well as the lander mass at the start of an episode." The RL agent never sees the mass or gravity, and must infer them implicitly from the sequence of observations through the recurrent hidden state. This makes the comparison particularly stark: the DR/DV law has more information but still performs worse when dynamics deviate from nominal, while the recurrent policy infers the necessary information from observation history and adapts accordingly.

---

Summary of Design Choices and Their Justifications

• GRU over LSTM or simple RNN: gating mechanism prevents vanishing/exploding gradients during backpropagation through unrolled time steps, and fewer parameters than LSTM reduce computational cost.
• PPO over TRPO or vanilla policy gradient: clipped objective provides TRPO-level stability with first-order optimization, avoiding the computational complexity of second-order methods while still preventing catastrophic policy updates.
• Multi-discount-rate framework over single discount with GAE: resolves the inherent tension between terminal and shaping reward horizons; simpler to implement than GAE and empirically outperforms it in this domain.
• Gaze-heuristic reward shaping over inverse RL: does not require expert demonstrations, generalizes to any landing scenario given basic mission parameters, and provides dense learning signal in a principled way through a velocity field potential.
• Observation as velocity error, altitude, time-to-go rather than full state: enforces translational invariance, forcing the policy to learn position-independent behaviors that generalize beyond the training distribution; also reduces input dimensionality compared to providing full position and velocity.
• Captured hidden states for parallel unrolling: enables efficient batch training of recurrent policies on GPU hardware by breaking the sequential dependency across unrolled segments while maintaining temporal continuity within segments.
• Domain randomization during training (wide uniform distributions over dynamics parameters): the mechanism that forces the recurrent hidden state to become an implicit dynamics estimator — because the policy must perform well across all sampled dynamics, the hidden state must learn to capture whatever unobserved information distinguishes one set of dynamics from another.

4. Key Insights and Innovations

Innovation 1: Recasting Guidance as Reinforcement Meta-Learning Rather Than Explicit System Identification

The paper's most distinctive intellectual contribution is the framing that repositions adaptive guidance from a system-identification-plus-control paradigm into a meta-learning paradigm where adaptation emerges as a trained property of a recurrent neural network rather than an engineered module. This is not simply "using an RNN for control" — it is a fundamental shift in what counts as the unit of learning during training.

Prior to this work, the dominant approach for handling unknown dynamics in learned control was to train a separate system identification module — typically another neural network — to explicitly estimate the unknown parameters (mass, gravity, disturbance forces) and feed those estimates into the policy as additional inputs. Reference 2 (Yu et al., 2017) exemplifies this separation: a recurrent network learns model parameters through online interaction, and these parameters are then concatenated with the observation for a standard RL algorithm. This is a natural, engineering-intuitive decomposition: identify the system, then control it. The identification module is trained with supervised learning against ground-truth parameter values, and the policy is trained with RL using the estimated parameters as input.

What this paper does is collapse that distinction entirely. The recurrent hidden state is never told what the mass or gravity or engine health is — it is only told to produce actions that maximize reward. The hidden state learns to represent whatever latent information is useful for action selection, which may include physically meaningful parameters but may also include emergent features that an engineer would never think to identify explicitly. The paper makes this point directly:

"the trained policy's hidden state captures unobserved information such as external forces and the current lander mass, as well as the past history of observations and actions, as this information is useful in minimizing the cost function."

The word "as well as" is doing subtle work here. The hidden state is not constrained to represent physically interpretable quantities; it is free to represent any latent structure in the observation-action history that correlates with future reward. This is a representation-learning perspective on adaptation: the recurrent state is a learned sufficient statistic for the unknown MDP parameters, trained end-to-end with the control objective.

Why is this more than just a "neural network does everything" argument? Because it changes the training signal. In the system-identification approach, the identification module receives supervised gradients from ground-truth parameter values. This requires knowing which parameters vary, measuring them during training, and designing a loss function. In the meta-learning approach, the hidden state receives RL gradients only — which are noisier and sparser but require no privileged information. The hidden state's representational capacity is directed solely at what improves control, not at what reconstructs parameters accurately. A parameter that varies during training but has negligible impact on optimal actions will be ignored by the meta-learning approach (correctly) but faithfully estimated by the system-identification approach (wastefully).

The evidence for this framing as genuinely meta-learning comes from the training regime: by randomizing dynamics across episodes, the policy sees a distribution of MDPs. The only way to achieve high average reward across this distribution is to develop an internal mechanism that adapts to each MDP within the episode. The frozen weights after training encode a learning algorithm — a procedure for updating the hidden state given new observations such that the policy's behavior converges to something appropriate for the current dynamics. The paper explicitly connects this to meta-learning's definition: "an agent learns through experience on a wide range of Markov decision processes (MDP) a strategy for quickly learning (adapting) to novel MDPs."

This framing is significant beyond performance gains because it provides a principled vocabulary for discussing why recurrent policies work: they are not just "policies with memory" but learned adaptation algorithms whose training objective naturally aligns the hidden state representation with control-relevant latent variables. It also explains the regime where this approach should be expected to fail — when the training distribution of MDPs does not cover the deployment MDP, the adaptation algorithm has no relevant experience to draw on, and the hidden state will evolve in ways that do not correspond to useful adaptation.

Innovation 2: Empirical Demonstration That Hidden-State Adaptation Outperforms Privileged-Information Model-Based Guidance

The paper's second major intellectual contribution is a diagnostic result rather than a methodological one: through a carefully designed comparison, it demonstrates that a recurrent policy with no access to ground-truth dynamics parameters consistently outperforms the DR/DV optimal guidance law that does have access to those parameters, sometimes catastrophically so. This is not a claim that RL beats classical control in general — it is a specific claim about what happens when the model used by the model-based controller is wrong.

The comparison design is critical and non-obvious. The DR/DV guidance law (Reference 5) is an optimal guidance law for planetary landing derived from a quadratic cost function with terminal constraints. It assumes a known dynamical model: known gravity, known vehicle mass, known mass depletion rate. To ensure a fair comparison — and indeed to bias the comparison in favor of DR/DV — the authors give the DR/DV law access to "the ground truth gravitational force, as well as the lander mass at the start of an episode." The RL agent, by contrast, receives neither: "the RL agent only has access to observations that are a function of the lander's position and velocity."

Yet the results tell a stark story:

• Asteroid landing (Experiment 1, Table 7): DR/DV achieves a maximum terminal position error of 1811 meters — a catastrophic miss that would destroy any real lander — while the recurrent policies achieve maximum errors under 2 meters. The mean DR/DV position error (1.9 m) is deceptively low because it averages over mostly successful landings with occasional catastrophic failures — the standard deviation of 54.6 m and maximum of 1811 m reveal the true picture of unreliability.

• Engine failure (Experiment 3, Table 12): DR/DV achieves a mean terminal position error of 123 meters with a standard deviation of 186 meters and maximum error of 1061 meters. The mean terminal velocity is 20.07 m/s with a maximum of 59.29 m/s — far above the 2 m/s threshold for a safe landing. The 20-step recurrent policy achieves 0.3 m mean position error and 0.99 m/s mean velocity, with maximums of 0.9 m and 1.15 m/s respectively.

• High mass variation (Experiment 4, Table 13): DR/DV shows a maximum position error of 19.6 m and velocity of 9.56 m/s (unsafe), while the recurrent policy achieves 1.1 m and 1.21 m/s maximums.

Why is this result intellectually significant rather than merely "RL beats classical control on these tasks"? Because it isolates model mismatch as the specific failure mode. The DR/DV law is optimal — provably optimal — for the nominal dynamics. When the actual dynamics deviate from nominal (unknown asteroid gravity and rotation, engine failure, mass variation beyond expected bounds), the optimality guarantee evaporates, and the law can produce actively dangerous commands. The DR/DV law does not degrade gracefully — it produces catastrophic failures at the tails of the error distribution, which is exactly where reliability matters most for high-stakes space missions.

The MLP policy provides an intermediate data point. It substantially outperforms DR/DV on all experiments (e.g., 0.7 m mean position error in Experiment 3 vs. DR/DV's 123 m), showing that domain randomization alone — training on varied dynamics — produces a robust fixed policy even without adaptation. But the MLP policy still falls short of the recurrent policy on the hardest tasks (e.g., in Experiment 2 with radar altimetry, the MLP achieves 131 m mean position error vs. 59 m for the 200-step RNN), and in Experiment 3 the MLP achieves 0.7 m mean position error vs. 0.3 m for the recurrent policy. The recurrent policy's additional gain can be attributed specifically to online adaptation, since both policies are trained with the same domain randomization.

This result challenges a deep assumption in control engineering: that if you know the true dynamics parameters, an optimal controller derived from those parameters will outperform a learned controller that must infer those parameters from data. The paper shows the opposite — the learned controller that infers dynamics implicitly through its hidden state outperforms the optimal controller that receives those parameters explicitly. The likely explanation is that the DR/DV law's optimality is fragile: it assumes the parameters are not just known but constant (or at least vary according to the model), and when parameters change during the descent (mass depletion) or differ from the nominal model in unanticipated ways (asteroid rotation creating Coriolis forces that DR/DV's cost function was not designed to handle), the "optimal" action computed from wrong-model assumptions can be worse than the action computed by a policy that has learned, through experience, what works across the actual distribution of dynamics.

This is not a claim that model-based control is obsolete. It is a claim that for problems with substantial, hard-to-characterize dynamics uncertainty, the adaptivity learned by a recurrent policy from broad domain randomization can provide a more robust form of "optimality" than a model-based law's theoretical optimality on a nominal model it will never exactly encounter during deployment.

Innovation 3: The Multi-Discount-Rate Framework as a Solution to the Terminal-vs-Shaping Reward Tension

This innovation is more narrowly technical than the previous two but represents a genuine, previously unreported solution to a problem that affects any RL application combining dense shaping rewards with sparse terminal bonuses. The authors state explicitly: "to our knowledge, a method of resolving this conflict has not been reported in the literature."

The conflict is structural: in standard RL, a single discount factor γ controls the effective horizon over which rewards influence policy updates. Shaping rewards — per-timestep penalties for velocity error, control effort, etc. — are most effective with relatively small γ, so that the policy focuses on near-term tracking performance rather than being distracted by distant shaping signals. Terminal rewards — a large bonus for successful landing — require large γ (close to 1), so that the bonus has meaningful discounted value when propagated back to early actions in the episode. For a 60-second descent at 10 Hz, the terminal bonus at step 600 is discounted by γ⁶⁰⁰; if γ = 0.95, the effective weight is 0.95⁶⁰⁰ ≈ 4.6 × 10⁻¹⁴, rendering the bonus irrelevant for early decisions.

The standard workarounds are unsatisfying: use a single γ as a compromise (degrading both shaping and terminal credit assignment), use reward annealing (start with small γ for shaping, increase toward 1 for terminal — complex to tune), or use Generalized Advantage Estimation (GAE, Reference 16) which smooths the advantage estimate but does not fundamentally decouple the discount horizons. The paper reports that "without the use of multiple discount rates, the performance was actually worsened by including the terminal reward term" — meaning the terminal bonus actively interfered with learning under a single discount rate, likely by creating conflicting gradient signals between immediate velocity tracking and long-horizon terminal bonus maximization.

The solution — separate discount factors applied to different reward components within the same advantage sum (Equations 17a–17b) — is simple in retrospect but required recognizing that the discount factor need not be a property of the MDP but can be a property of the reward component. The mathematical justification is that the advantage sum is a linear combination of discounted rewards; linearity means each component can be discounted independently and summed. This is a small formal insight — essentially applying superposition to the discounting operation — but it resolves a practical tension that prior RL work with shaped rewards had either worked around or ignored.

The significance is broader than this paper's domain. Any RL problem with a mixture of dense progress incentives and sparse completion bonuses — robotic manipulation, game playing, autonomous navigation, dialogue systems with per-turn coherence rewards and end-of-conversation success signals — faces the same tension. The multi-discount framework provides a clean, principled resolution: assign each reward component its own discount rate based on the timescale over which that reward signal should influence decisions, and sum their separately discounted contributions. The paper reports that this approach "exceed[s] that of using generalized advantage estimation (GAE) both with and without multiple discount rates," though specific numbers are not provided — this is noted as a comparative claim in the RL formulation section rather than a tabled result, which is a minor limitation of the evidence.

Innovation 4: Integrated Guidance and Navigation as an Emergent Property of Recurrent Policies in POMDPs

Experiment 2 (landing using only Doppler radar altimeter returns) represents a conceptual advance that goes beyond performance improvement: it demonstrates that a recurrent policy can dissolve the traditional architectural boundary between guidance and navigation by learning to perform implicit state estimation and control simultaneously from raw sensor data.

In conventional aerospace engineering, guidance and navigation are separate subsystems designed by different teams using different mathematical frameworks. Navigation uses filtering (Kalman filters, particle filters) to estimate the vehicle's state (position, velocity, attitude) from noisy sensor measurements. Guidance takes that state estimate as input and computes control commands. This separation is deeply embedded in engineering practice, textbooks, and certification processes. It is justified by modularity (each subsystem can be developed and tested independently) and by the availability of rigorous performance guarantees for each component.

Experiment 2 challenges this separation at a fundamental level. The observation is not a (noisy) state vector but four raw Doppler altimeter readings — distances from the lander to terrain features at four offset angles. These readings do not uniquely determine the lander's position and velocity: "there are multiple ground truth positions and velocities that could correspond to a given observation, making the optimal action a function of the history of past altimeter readings." This is a formally partially observable Markov decision process (POMDP), and the recurrent policy's hidden state serves as a learned belief state — an implicit representation of the probability distribution over possible true states given the observation history.

What is striking about the results (Tables 9 and 11) is not that the recurrent policy works at all — it is that it works without any explicit navigation module and achieves landing accuracy that, while not suitable for a multi-billion-dollar flagship rover (59 m mean position error with 200-step RNN in the target-pointing variant, Table 11, last row), is qualitatively impressive given that the agent is navigating using only four distance readings in an environment where "the measurement error becomes quite extreme at lower elevations" (Table 8: at 100 m elevation, the mean altimeter error is 2600 m with a standard deviation of 3200 m and 61% of readings missing the terrain entirely).

The paper frames this as evidence that "a recurrent network has the ability to capture much longer temporal dependencies than a Kalman filter." This is an important claim about representation capacity: Kalman filters maintain a fixed-size state estimate (mean and covariance) and update it with a linear-Gaussian model; GRUs maintain a high-dimensional hidden state and update it with a learned nonlinear function. The GRU can, in principle, learn to represent multi-modal belief distributions, exploit non-Gaussian sensor noise characteristics, and integrate information over arbitrarily long temporal horizons — all properties that classical filters cannot achieve without significant engineering effort.

This insight carries implications beyond spacecraft guidance. For any control problem where the sensor stream is high-dimensional and the state is only partially observable — autonomous driving from camera images, robotic manipulation from tactile sensors, drone flight from monocular video — the conventional wisdom of separating perception (state estimation) from control may sacrifice performance that an end-to-end recurrent policy can recover. The paper does not claim that integrated policies should replace separated architectures in safety-critical applications — the 59 m landing error would not be acceptable for a real Mars mission — but it opens the conceptual space for architectures where the boundary between perception and control is blurred by learned temporal integration.

The trend in the results is also informative: as the unrolling length increases (1 → 20 → 60 → 120 → 200 steps in Table 9), mean terminal position error drops monotonically (114 → 78 → 72 → 72 → 59 m in the non-target-pointing case; 3.3 → 0.3 → 0.4 → 0.6 → N/A in the target-pointing case, though the 1-step case is an outlier at 3.3 m), suggesting that the effective temporal dependencies in this navigation task span a significant fraction of the ~60-second descent. This is an empirical finding about the problem structure — the information needed to disambiguate the lander's state from altimeter readings accumulates over tens of seconds of flight — that would be difficult to derive analytically from the sensor model.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper uses four custom-designed landing scenarios rather than a pre-existing benchmark dataset. Each scenario involves a simulated 3-DOF landing environment with randomized initial conditions and dynamical parameters. The Mars landing experiments use a digital terrain map (DTM) of the Mars surface in the vicinity of Uzbois Valis, doubled in size by reflection and joining (Figure 5). For the asteroid landing, dynamics parameters are sampled from the ranges given in Table 4. Initial conditions for each scenario are specified in Tables 3 (Mars landing), 5 (asteroid landing), and 10 (Mars landing with target-pointing altimeters). The training distribution consists of episodes with randomly sampled parameters; testing is conducted over 10,000 episodes per experiment with the trained policy frozen and exploration turned off.

• Base model(s). The paper compares three guidance approaches: a DR/DV optimal guidance law (Reference 5), an RL agent with a non-recurrent (MLP) policy, and an RL agent with a recurrent policy. The RL agents both use the same PPO training algorithm, the same reward function (Equations 14–15 for Mars, 18 for asteroid), and the same observation vector (Equation 16), differing only in the architecture of the policy and value function networks. The recurrent architectures use GRU layers with unrolling lengths of 1, 20, 60, 120, or 200 steps during the forward pass, referred to as "T-step RNN." The DR/DV baseline is given privileged information — "the ground truth gravitational force, as well as the lander mass at the start of an episode" — which the RL agents do not receive. The choice of DR/DV as baseline is motivated by its status as a standard optimal guidance law for planetary landing.

• Metrics. Four terminal performance metrics are reported for each experiment: terminal position error (Euclidean distance from the targeted landing site, in meters), terminal velocity magnitude (m/s), terminal glideslope (radians per second, measuring whether velocity is directed predominantly downward at touchdown), and fuel consumption (kg). Each metric is reported with mean (µ), standard deviation (σ), and maximum (max) across the 10,000 test episodes. For Experiment 2, an additional metric "miss %" at various altitudes quantifies the fraction of altimeter readings that fail to intersect the terrain entirely (Table 8). The terminal constraints for a successful Mars landing are: position error < 5 m, velocity < 2 m/s, glideslope > 5 rad/s. For the asteroid landing, the constraints are tighter: position error < 1 m, velocity < 0.2 m/s, with no glideslope constraint.

• Baselines. Three baselines are used across experiments:

  • DR/DV guidance law (Reference 5): an optimal guidance law for planetary landing derived from a quadratic cost function with terminal constraints. It is given access to ground-truth gravitational acceleration and initial lander mass, information not provided to the RL agents.
  • MLP policy: an RL agent with a non-recurrent (feedforward) policy and value function, trained with PPO under the same domain randomization as the recurrent policy. This baseline isolates the contribution of recurrence to performance, since both MLP and recurrent policies receive identical training.
  • 1-step RNN: a recurrent policy where the GRU is unrolled for only 1 timestep during the forward pass. This serves as an intermediate point between the MLP (no temporal memory) and the longer-unrolled RNNs (substantial temporal memory), helping to distinguish whether benefits come from the GRU architecture itself or from the temporal depth of its hidden state.

  Note that no baseline is available for Experiment 2 with the DR/DV law "for obvious reasons" — DR/DV requires position and velocity estimates as inputs and cannot operate directly on raw Doppler altimeter returns.

• Generation budget / compute accounting. The paper does not use a "generation budget" in the sense of controlling the number of candidate solutions (as in LLM best-of-N sampling). Instead, all policies are limited by the physical dynamics of the landing: each episode runs until the lander reaches zero altitude (either successfully or crashing) or until a constraint violation terminates the episode. The Mars landing initial conditions (Table 3) produce descents of approximately 60 seconds at the 10 Hz simulation rate, yielding roughly 600 timesteps per episode. The asteroid landing (Table 5) uses different dynamics and initial conditions, resulting in episodes of different but comparable lengths. Training compute is measured implicitly by the number of episodes (visible in the learning curves of Figures 4, 6, and 7), but test-time compute is identical across all methods for a given scenario — each policy must control the lander through the same descent physics. The key comparison is therefore performance per landing attempt, not performance per FLOP. The only computational asymmetry is the DR/DV law's access to privileged information (ground truth mass and gravity), which makes its underperformance more damning.

• Cross-validation / statistical protocol. No formal cross-validation or statistical significance testing is reported. The evaluation protocol is: train a policy using PPO with domain randomization until convergence (as shown in the learning curves), freeze the policy weights, and then evaluate on 10,000 test episodes with different random seeds for the randomized parameters. Exploration is turned off during testing. The large number of test episodes (10,000) provides reasonably stable estimates of means and standard deviations, though confidence intervals are not reported. The learning curves in Figures 4, 6, and 7 show statistics (terminal position, terminal velocity) computed over the 30 episodes used to generate rollouts for each PPO update, giving a coarse picture of learning progress but not a rigorous evaluation of generalization during training. The asteroid experiment mentions that "we tuned the nominal gravity parameter for the DR/DV policy for best performance," indicating some manual optimization of the baseline, but no systematic hyperparameter search is described for any method.

---

Main Quantitative Results

The paper organizes results by experiment, with each of the four experiments targeting a distinct type of dynamical uncertainty. I follow this structure, presenting the headline numbers for each experiment and drawing comparisons across methods.

Experiment 1: Asteroid Landing with Unknown Dynamics

This experiment tests whether the recurrent policy can adapt to an environment where gravity, rotational velocity, and solar radiation pressure are sampled from wide uniform distributions at the start of each episode. The lander uses pulsed thrusters with 2 N capability per axis, a wet mass randomly chosen between 450–500 kg, and targets a landing site on the asteroid's pole 250 m from the center of rotation. The environmental parameter ranges (Table 4) are chosen to be "limited only by the lander's thrust capability" — the rotational velocities produce Coriolis and centrifugal forces up to the point where the lander can no longer physically overcome them.

DR/DV fails catastrophically at the tails. Table 7 shows that the DR/DV guidance law achieves a seemingly acceptable mean terminal position error of 1.9 m, but this average conceals extreme unreliability: the standard deviation is 54.6 m and the maximum error over 10,000 episodes is 1811 m — a complete miss that would destroy the lander. The maximum terminal velocity is 47 cm/s (well above the 20 cm/s threshold), and the maximum position error indicates that "these typically occur after less than 1000 episodes of testing," meaning the failures are not rare outliers but occur with roughly 10% probability. The paper attributes these failures to DR/DV's inability to handle the wide parameter variation: even with ground-truth gravity and mass, the optimal guidance law assumes a model structure (point-mass gravity, no rotation terms) that deviates from the actual dynamics when asteroid rotation is significant.

All RL policies achieve safe landings, but recurrence improves fuel efficiency. All RL-derived policies (MLP and RNN variants) achieve mean terminal position errors of 0.2–0.3 m and mean terminal velocities of 3.8–4.2 cm/s, well within the 1 m and 20 cm/s constraints. The maximum errors are also contained: the worst RL landing across all variants has a position error of 1.7 m (20-step RNN) and a velocity of 11.0 cm/s (also 20-step RNN). This is a qualitatively different reliability profile from DR/DV.

The key differentiator among RL methods is fuel efficiency. As the unrolling length increases, mean fuel consumption decreases monotonically: MLP uses 1.62 kg, 1-step RNN uses 1.48 kg, 20-step RNN uses 1.42 kg, 60-step RNN uses 1.44 kg, and 120-step RNN uses 1.43 kg. The standard deviation of fuel consumption also decreases with longer unrolling (0.92 kg for MLP → 0.35 kg for 120-step RNN), indicating more consistent efficiency across episodes with different dynamics. The maximum fuel consumption follows a similar trend: 3.32 kg (MLP) → 2.85 kg (120-step RNN). This pattern — improved fuel efficiency with longer recurrent memory, despite all methods meeting the terminal constraints — is consistent with the interpretation that the GRU hidden state captures the specific dynamics parameters of the current episode (asteroid rotation rate, gravity magnitude, SRP direction) and tailors the trajectory to exploit them efficiently, whereas the MLP policy must use a one-size-fits-all strategy that burns additional fuel as a safety margin.

Learning curves (Figure 4) show rapid convergence. The learning curves plot terminal position error and terminal velocity as a function of training episode. Both metrics drop sharply in the first ~2000 episodes and then stabilize, with all RL variants converging to similar terminal accuracy by episode 5000. The recurrent variants do not show a clear advantage in convergence speed over the MLP — the benefit of recurrence appears in the asymptotic performance (fuel efficiency) rather than in learning speed.

Asteroid-specific modifications matter. The paper tuned the DR/DV policy's nominal gravity parameter "for best performance" and found that "the optimal setting for this parameter is quite a bit higher than the actual environmental gravity." This is an important diagnostic: DR/DV's performance depends on a single tunable parameter that must be set to a value that does not correspond to any physical quantity, essentially turning the optimal guidance law into a heuristic controller with an opaque tuning knob. The RL policies, by contrast, require no per-asteroid tuning — the same trained policy handles the entire distribution.

Experiment 2: Mars Landing Using Radar Altimeter Observations

This experiment tests two related but distinct capabilities: (1) whether a recurrent policy can perform guidance using only raw sensor returns rather than estimated state, and (2) whether longer recurrent unrolling captures the temporal dependencies needed to disambiguate the lander's state from ambiguous altimeter readings. The observations are simulated Doppler altimeter returns from four beams with equal offset angles (π/8 radians) from a central direction vector that averages the lander's velocity vector and the downward direction. The observations do not satisfy the Markov property because multiple ground-truth positions and velocities could produce the same altimeter readings (e.g., being high over a valley vs. low over a hill).

The altimeter model introduces substantial errors that worsen at lower altitudes (Table 8). At 100 m elevation, the mean error is 2600 m, standard deviation 3200 m, and 61% of readings miss the terrain entirely (the beam either points at the sky or the intersection algorithm fails). At 400 m, errors remain large (mean 513 m, 22% miss rate). Accuracy improves above 600 m (mean error 25 m, 6% miss rate at 600 m; 4 m mean error and 2% miss rate at 800 m). This error profile means that the most critical phase — terminal descent below 100 m — has the worst sensor data, forcing the policy to rely heavily on its memory of higher-altitude observations to estimate its position and velocity during final approach.

The baseline results (non-target-pointing, Table 9) show steadily improving mean position error with longer unrolling. The MLP policy achieves a mean terminal position error of 131 m with a standard deviation of 101 m — substantially better than random but far from a pinpoint landing. The 1-step RNN improves to 114 m mean, 95 m standard deviation. The 20-step RNN reduces this to 78 m mean and 41 m standard deviation. The 120-step RNN achieves 72 m mean and 40 m standard deviation, and the 200-step RNN achieves the best result: 59 m mean, 42 m standard deviation, and a maximum error of 288 m.

This monotonic improvement with unrolling length (131 → 114 → 78 → 72 → 59) is the strongest single piece of evidence for the paper's central claim about temporal dependencies. Each increase in the number of steps through which the GRU is unrolled during training gives the policy access to longer observation histories, and each increase produces better landing accuracy. The paper explicitly notes: "this implies that the temporal dependencies for this task probably span a significant fraction of a single episode." Since episodes last roughly 600 timesteps and the improvement continues out to 200-step unrolling, the relevant temporal horizon appears to be on the order of 20–30 seconds of flight.

Terminal velocity also improves with unrolling, but the pattern is different. Mean terminal velocity drops from 48 m/s (MLP) to 37 m/s (1-step RNN) to 26 m/s (20-step RNN) to 28 m/s (120-step RNN) to 23 m/s (200-step RNN). The improvements between 20 and 120 steps are small, suggesting that velocity estimation requires shorter temporal context than position estimation. The glideslope metric shows inconsistent patterns — the 200-step RNN achieves a mean glideslope of 2.55 rad/s, which is below the 5 rad/s constraint for a "good" landing in the standard Mars scenario, indicating that the velocity vector is not predominantly downward at touchdown for many landings. This is not necessarily a failure — the altimeter-based policy may be actively managing horizontal velocity to navigate toward the target, and touchdown may occur with significant horizontal component if position accuracy is prioritized over vertical descent.

The target-pointing variant (Table 11) dramatically improves performance across all methods. When the altimeter beams are assumed to remain pointed at the target (bracketing the landing site), and initial condition uncertainty is reduced (Table 10: downrange 0–1000 m vs. 0–2000 m, velocity range reduced), all methods improve substantially. The MLP achieves 1.4 m mean position error, 2.9 m standard deviation. The 20-step RNN achieves the best mean at 0.3 m, with 1.2 m standard deviation. The 60-step RNN achieves 0.4 m mean, 1.5 m standard deviation, and the 120-step RNN achieves 0.6 m mean, 1.6 m standard deviation.

Several patterns emerge from the target-pointing results. First, the 20-step RNN is the best performer by mean position error, not the longest unrolling — suggesting that when sensor quality is high (beams remain on target, terrain diversity is maintained), the optimal temporal integration window may be shorter. Second, the standard deviations and maxima remain substantial (the 20-step RNN has a maximum position error of 116.0 m, and the MLP has a maximum of 179.2 m), indicating that even with target-pointing, some episodes produce large errors due to the altimeter inaccuracies at low altitude. Third, fuel consumption increases slightly with longer unrolling (211 kg MLP → 223 kg 120-step RNN), the opposite of Experiment 1 — this may reflect that longer-memory policies are more aggressive about position correction, burning fuel to achieve better accuracy rather than accepting larger errors to save fuel.

The paper notes that performance degrades when the landing site is moved to a region with lower terrain diversity (south of the DTM), though no quantitative results are provided for this variation. This sensitivity to terrain characteristics is expected — if the altimeter readings are similar across a wide area (flat terrain), the disambiguation problem becomes harder regardless of temporal memory.

Learning curves (Figures 6 and 7) reveal a significant difference between short and long unrolling. For the 1-step RNN (Figure 6), terminal position and velocity errors decrease gradually and somewhat noisily over ~80,000 episodes, never fully stabilizing. For the 120-step RNN (Figure 7), initial progress is rapid — terminal position error drops sharply in the first ~20,000 episodes — but then stalls: "the optimization initially makes good progress, but then stalls, likely due to the highly inaccurate altimeter readings at lower altitudes." This stalling is a negative result that reveals a limitation: longer unrolling enables faster initial learning (the policy quickly discovers how to use history for coarse navigation) but hits a performance ceiling imposed by sensor quality rather than temporal reasoning capacity.

Experiment 3: Mars Landing with Engine Failure

This experiment tests adaptation to a discrete, persistent change in dynamics: an actuator failure. The maximum thrust is increased to 24,000 N. At the start of each episode, with probability 0.5, a failure occurs: either the downrange or crossrange thrust capability is halved, and the vertical (elevation) thrust capability is reduced by a factor of 1.5 (i.e., to two-thirds of nominal). The failure persists for the entire episode. The RL agent is not told whether a failure has occurred or which axis is affected — it must infer this from the observed response to its thrust commands through the recurrent hidden state.

DR/DV fails catastrophically (Table 12). The mean terminal position error is 123 m with a standard deviation of 186 m and a maximum of 1061 m. The mean terminal velocity is 20.07 m/s with a maximum of 59.29 m/s — an order of magnitude above the 2 m/s safety threshold. The fuel consumption is 213 kg, comparable to the nominal case (Table 13 baseline), but this is irrelevant given that the landing is not survivable. DR/DV's failure mode is clear: it assumes symmetric thrust capability, and when one axis is suddenly limited, the guidance law commands thrust vectors that the damaged actuator cannot produce, leading to loss of control authority and crash.

The MLP policy achieves a reasonable but imperfect landing. Mean terminal position error is 0.7 m, standard deviation 0.2 m, maximum 1.8 m — within the 5 m constraint. Mean terminal velocity is 0.99 m/s, maximum 4.67 m/s — within the 2 m/s constraint on average but with occasional violations. The glideslope mean is 13.46 rad/s, indicating predominantly vertical descent. Fuel consumption is 302 kg. This is a substantial improvement over DR/DV, demonstrating that domain randomization alone (training on episodes with and without engine failures) produces a fixed MLP policy that can handle both nominal and failed conditions reasonably well. However, the maximum velocity of 4.67 m/s and the glideslope standard deviation of 5.35 rad/s indicate that the MLP policy is not fully reliable — there are episodes where the landing is unsafe.

The 1-step RNN performs similarly to the MLP. Mean position error 0.9 m, mean velocity 0.95 m/s, fuel consumption 298 kg. The 1-step unrolling provides minimal temporal context (essentially just the current observation with a GRU transformation), so this result is expected — the GRU architecture without temporal depth offers no advantage over a feedforward network for adaptation to engine failure.

The 20-step and 60-step RNNs achieve safe landings with improved consistency. The 20-step RNN achieves mean position error 0.3 m, standard deviation 0.1 m, maximum 0.9 m. Mean velocity 0.99 m/s with standard deviation 0.06 m/s and maximum 1.15 m/s. The 60-step RNN achieves nearly identical results: mean position error 0.3 m, maximum 1.1 m; mean velocity 1.00 m/s, maximum 1.16 m/s. Both variants keep the maximum velocity below the 2 m/s safety threshold, meaning no unsafe landings occurred in 10,000 test episodes (with 50% failure probability, corresponding to roughly 5,000 engine-failure episodes).

The key improvement is not in the means — the MLP's means are already acceptable — but in the tails. The 20- and 60-step RNNs virtually eliminate the outlier episodes where the MLP produced 4.67 m/s touchdown velocities. The standard deviation of velocity drops from 0.62 m/s (MLP) to 0.06 m/s (20-step RNN), a factor of ~10 reduction. This is precisely the signature of adaptation: the recurrent policy detects the engine failure early in the descent (through the mismatch between commanded and achieved acceleration) and adjusts its subsequent commands to compensate, while the MLP policy, lacking memory, treats each observation independently and occasionally commands thrust vectors that the damaged system cannot execute safely.

Glideslope metrics show a non-obvious pattern. The 20-step RNN has a mean glideslope of 49.99 rad/s with a standard deviation of 82.14 — enormous variance. The 60-step RNN has 24.98 rad/s mean, 12.70 standard deviation. These values are far above the 5 rad/s minimum, indicating that the landers are descending nearly vertically at touchdown (high glideslope = steep descent angle). The large variance for the 20-step RNN may reflect that the policy sometimes overshoots the target and needs to correct with horizontal thrust near the surface, producing more variable descent angles.

Fuel consumption is nearly identical across RL methods (302 kg MLP, 298 kg 1-step, 295 kg 20-step, 295 kg 60-step), suggesting that fuel use is dominated by the physics of compensating for the engine failure (requiring longer burns with the remaining healthy thrusters) rather than by policy differences.

Experiment 4: Mars Landing with High Mass Variation

This experiment tests adaptation to a continuously varying internal parameter: the lander's mass. By dividing the engine specific impulse by a factor of 6, fuel consumption increases to approximately 1200–1600 kg per landing (compared to ~200–300 kg in nominal conditions). Since the wet mass is 2000 kg and dry mass is 200 kg, the lander loses 60–80% of its initial mass during the descent. This creates a difficult control problem because the same thrust command produces very different accelerations at the beginning (mass ~2000 kg) and end (mass ~400–800 kg) of the descent. The agent never receives the ground-truth mass — it must infer it from how the lander's velocity responds to thrust commands.

DR/DV degrades but does not fail catastrophically (Table 13). The mean terminal position error is 0.4 m with a maximum of 19.6 m. The mean terminal velocity is 0.63 m/s with a maximum of 9.56 m/s — the maximum velocity is unsafe (above 2 m/s), indicating occasional loss of control during the final approach. The glideslope has a mean of 36.89 rad/s but a standard deviation of 180.6, indicating extreme variability in the descent angle at touchdown. Fuel consumption is 1362 kg. DR/DV's partial success here (compared to its catastrophic failure in Experiments 1 and 3) reflects that mass variation is a parameter DR/DV was designed to handle — the guidance law includes a mass depletion model — but the extreme rate of mass change (6× normal) pushes the law outside its design envelope, producing occasional unsafe landings.

The MLP policy improves over DR/DV but still produces unsafe landings. Mean position error 0.7 m, maximum 3.7 m. Mean velocity 0.92 m/s, maximum 5.25 m/s — again, the maximum exceeds the safety threshold. The glideslope mean is 10.12 rad/s with standard deviation 2.09, indicating more consistent descent geometry than DR/DV but with occasional problematic episodes.

The 1-step RNN performs similarly to the MLP. Mean position error 0.4 m, maximum 0.8 m. Mean velocity 0.98 m/s, maximum 6.48 m/s — the maximum velocity is actually worse than the MLP, suggesting that the GRU architecture without temporal depth may introduce harmful state dynamics for this task. Glideslope mean is 20.05 rad/s, standard deviation 4.20. Fuel consumption is 1229 kg — slightly lower than the MLP's 1235 kg.

The 20-step and 60-step RNNs achieve consistently safe landings. The 20-step RNN achieves mean position error 0.6 m, maximum 1.0 m. Mean velocity 1.17 m/s with a maximum of 1.36 m/s — safely below 2 m/s. The 60-step RNN achieves mean velocity 1.06 m/s with a maximum of 1.21 m/s. Both variants keep all 10,000 test episodes within safe velocity limits, and the maximum position errors (1.0 m and 1.1 m) are well within the 5 m constraint.

The key improvement is again in the tails of the velocity distribution. The MLP produces a small fraction of landings with velocities above 5 m/s; the recurrent policies eliminate these entirely. The mechanism is presumably the GRU's ability to track the lander's evolving mass implicitly through the observed acceleration response to thrust commands — if a given thrust command produces less deceleration than expected, the hidden state can infer that the mass is higher than average and adjust subsequent commands to be more aggressive, and vice versa as mass decreases.

Fuel consumption decreases slightly with longer unrolling (1235 kg MLP → 1229 kg 1-step → 1224 kg 20-step → 1227 kg 60-step), though the differences are small relative to the total (a ~1% reduction from MLP to 60-step). The real benefit is in consistency and safety, not fuel savings.

The 1-step RNN anomaly: The 1-step RNN's maximum velocity (6.48 m/s) is worse than the MLP's (5.25 m/s), despite both having effectively no temporal memory. This is a minor negative result that suggests the GRU cell's state dynamics, even when not unrolled through time in the forward pass, can introduce instability that a purely feedforward architecture avoids. The paper does not comment on this anomaly, but it serves as a caution that recurrent architectures are not universally beneficial — they must be used with sufficient temporal depth (unrolling length) to be helpful.

---

Ablation Studies and Robustness Checks

The paper does not conduct formal ablation studies in the traditional sense (varying one component of the method while holding others fixed to measure its contribution). However, several structural comparisons across experiments serve as implicit ablations:

Recurrence depth (unrolling length): The comparison of 1-step, 20-step, 60-step, 120-step, and 200-step RNN variants across all four experiments serves as an ablation of temporal memory length. The consistent finding is that longer unrolling improves performance, though with diminishing returns and occasional regressions:

• In Experiment 1 (asteroid), fuel efficiency improves from 1.62 kg (MLP) → 1.48 kg (1-step) → 1.42 kg (20-step) → 1.44 kg (60-step) → 1.43 kg (120-step) — most of the gain occurs between MLP and 20-step, with small further improvements.
• In Experiment 2 (radar altimetry, non-target-pointing), position error improves monotonically: 131 → 114 → 78 → 72 → 59 m — gains continue out to 200 steps.
• In Experiment 3 (engine failure), the critical improvement (elimination of velocity outliers) occurs between 1-step and 20-step, with 60-step similar to 20-step.
• In Experiment 4 (mass variation), the critical improvement again occurs between 1-step and 20-step, with 60-step similar.

The pattern suggests that 20–60 steps of temporal context captures the essential dynamics of most landing scenarios, with the radar altimetry task (POMDP requiring state estimation from ambiguous observations) being the exception that benefits from longer horizons.

MLP vs. recurrent policy: The comparison between the MLP policy and the recurrent policies serves as an ablation of the GRU architecture and hidden state memory. Across all experiments, the recurrent policy (with sufficient unrolling) outperforms the MLP, but the margin varies dramatically by task:

• Asteroid landing: MLP performs acceptably (0.2 m, 4.1 cm/s); recurrent improves fuel efficiency but not terminal accuracy.
• Radar altimetry: MLP performs poorly (131 m); recurrent improves to 59 m — a ~55% reduction in error.
• Engine failure: MLP produces unsafe velocity outliers (max 4.67 m/s); recurrent eliminates them entirely (max 1.15 m/s).
• Mass variation: Similar pattern — MLP has unsafe outliers (max 5.25 m/s); recurrent eliminates them (max 1.36 m/s).

This pattern is consistent with the paper's theoretical framing: the value of recurrence is proportional to the amount of relevant unobserved information that must be inferred from observation history. When the only unknowns are slowly varying external parameters (asteroid gravity, rotation), the MLP can find a fixed policy that works across the distribution. When the unknowns include discrete, persistent changes (engine failure) or continuous identification problems (mass variation, state estimation from ambiguous sensors), recurrent memory becomes essential.

DR/DV with privileged information vs. RL without: The comparison between DR/DV (which receives ground-truth mass and gravity) and the RL agents (which do not) serves as an ablation of the value of explicit dynamics knowledge. The fact that DR/DV performs worse in every experiment — despite having more information — is the paper's strongest argument for learned adaptation over model-based guidance with uncertain models.

Multiple discount rates (implicit ablation): The paper mentions that "without the use of multiple discount rates, the performance was actually worsened by including the terminal reward term" and that the multi-discount approach "exceed[s] that of using generalized advantage estimation (GAE) both with and without multiple discount rates." However, no quantitative results are presented for single-discount-rate baselines, making this claim impossible to evaluate from the reported data. This is a significant gap — the multi-discount framework is presented as a methodological contribution, but the evidence for its effectiveness is anecdotal rather than empirical.

Reward function design (implicit ablation): The paper describes experimenting "with several potential functions with no success" before arriving at the gaze-heuristic-based velocity field. No results are shown for these failed attempts, making it difficult to assess how sensitive the overall approach is to reward function design. The paper's claim that the specific reward formulation in Equations 14–15 is critical to success is plausible but unsupported by comparative data.

Training stability across unrolling lengths (Figure 7 vs. Figure 6): The learning curves for the 1-step RNN (Figure 6) show gradual, noisy improvement over ~80,000 episodes. The learning curves for the 120-step RNN (Figure 7) show rapid initial improvement followed by stalling. This is an unplanned ablation demonstrating that longer unrolling affects not just final performance but also the dynamics of learning — longer temporal context enables faster initial progress but can also lead to premature convergence when sensor quality limits further improvement.

Target-pointing vs. not (Experiment 2 variations): The comparison between Tables 9 and 11 serves as an ablation of sensor quality and initial condition uncertainty. Reducing initial condition spread (Table 10 vs. Table 3) and keeping the altimeter beams on target dramatically improves performance across all methods (MLP: 131 m → 1.4 m; 20-step RNN: 78 m → 0.3 m). This demonstrates that the difficulty of the radar altimetry task is driven more by beam placement and initial condition uncertainty than by the fundamental ambiguity of altimeter readings — when the beams bracket the target throughout the descent, even an MLP can achieve landing accuracy approaching that of the state-observation experiments.

Asteroid DR/DV tuning (implicit sensitivity analysis): The paper notes that the DR/DV nominal gravity parameter was tuned "for best performance" and found the optimal setting was "quite a bit higher than the actual environmental gravity." This is an informal sensitivity analysis showing that DR/DV's performance is brittle with respect to its model parameters — the "optimal" setting is a heuristic value that bears no relationship to the physical environment.

---

Critical Assessment

The experiments collectively support the paper's central claim — that a recurrent policy trained with domain randomization can adapt in real time to unknown dynamics and substantially outperform both a classical guidance law and a non-recurrent learned policy — but with important boundaries on when and how much recurrence helps.

Claim 1: The recurrent policy consistently achieves the best performance across all tasks. This claim is supported, with an important nuance. The recurrent policy (with sufficient unrolling, typically 20–60 steps) achieves the best means and, more importantly, the tightest tails (smallest standard deviations and maxima) in every experiment. However, the magnitude of the improvement varies dramatically by task:

• In the asteroid landing (Experiment 1), the recurrent policy's advantage over the MLP is modest — all RL methods achieve safe landings, and the recurrent policy's main benefit is ~12% better fuel efficiency (1.42 kg vs. 1.62 kg). This is a real but incremental improvement.
• In the engine failure scenario (Experiment 3), the recurrent policy provides a qualitative improvement — eliminating catastrophic velocity outliers that the MLP occasionally produces. This is the difference between a policy that is "usually safe" and one that is "always safe" in 10,000 trials, which matters enormously for mission assurance.
• In the radar altimetry task (Experiment 2), the recurrent policy provides a large quantitative improvement — reducing mean position error from 131 m to 59 m. But 59 m is still far from the 5 m target constraint, meaning the policy is not achieving "pinpoint landing" in the non-target-pointing case.

A fair summary: the recurrent policy is always the best option among the tested methods, but the degree of superiority ranges from "modest efficiency gain" to "enables the task at all" depending on how much unobserved information must be inferred from observation history.

Claim 2: The DR/DV guidance law fails under unknown dynamics. This claim is strongly supported for the conditions tested. DR/DV produces unsafe landings in three of three tested scenarios (Experiments 1, 3, and 4), with catastrophic failures (position errors exceeding 1000 m) in the engine failure case. However, a critic could argue that the DR/DV implementation is not state-of-the-art — it does not include any online parameter estimation, adaptive gain scheduling, or robust control modifications that a careful engineer would add for missions with known parameter uncertainty. The paper's comparison is against a "bare" optimal guidance law, not against an engineered adaptive version of DR/DV. This is a fair comparison for establishing that naive model-based guidance fails, but it does not demonstrate that all model-based approaches would fail — an adaptive DR/DV with online mass estimation might perform competitively with the MLP policy, if not the recurrent policy.

Additionally, DR/DV was given two specific pieces of privileged information (ground-truth gravity and initial mass) but was not given other information that might have helped (e.g., the actual thrust limits after engine failure, the actual rotational velocity for the asteroid). The authors' intent was clearly to bias the comparison in favor of DR/DV by providing information the RL agent does not receive, but a skeptic could argue that providing partial privileged information is not the same as providing all relevant information — DR/DV might perform better if also told about engine failures explicitly, though this would violate the spirit of the unknown-dynamics scenario.

Claim 3: The recurrent policy enables integrated guidance and navigation from raw sensor data (Experiment 2). This claim is supported in a proof-of-concept sense but not in a deployment-ready sense. The 59 m mean position error (200-step RNN, non-target-pointing, Table 9) is remarkable given the sensor quality (altimeter errors of hundreds to thousands of meters at low altitude, 61% miss rate at 100 m), and it genuinely demonstrates that the recurrent policy is extracting usable state information from the altimeter history. However, 59 m is not accurate enough for a real Mars landing (where the target might be a specific safe zone within a hazardous terrain field). The target-pointing variant achieves much better accuracy (0.3 m mean for the 20-step RNN, Table 11), but this variant makes strong assumptions (beams remain pointed at target, reduced initial condition uncertainty) that may not hold in practice without additional engineering.

The paper is appropriately cautious: "you would not want to entrust an expensive rover to this integrated guidance and navigation algorithm." This is a candid acknowledgment that the result is a research demonstration, not a flight-ready system. The contribution is the concept that integrated guidance and navigation can emerge from recurrent policy training, not that the specific implementation achieves operational accuracy.

Missing experiments that would strengthen the paper:

1. Single discount rate baselines. The multi-discount-rate framework is presented as a contribution, but no results are shown for a single-discount-rate baseline on any experiment. Without this comparison, the reader cannot assess whether the multi-discount innovation is essential to the reported performance or merely a minor tweak.

2. Generalized Advantage Estimation (GAE) comparison. The paper claims the multi-discount approach exceeds GAE performance, but no GAE results are reported. This is a missing ablation that would contextualize the contribution.

3. Varying the amount of domain randomization. All RL agents are trained on the same wide randomization ranges (Tables 4, 3). An experiment that varied the randomization width would reveal whether performance degrades gracefully as the training distribution narrows, and whether the recurrent policy's advantage over the MLP grows with randomization width (as the adaptation problem becomes harder).

4. Out-of-distribution dynamics. All test episodes sample dynamics from the same distributions used during training. An experiment where test dynamics are drawn from a different distribution (e.g., training on Mars gravity ±5% but testing on ±20%) would test whether the adaptation mechanism is genuine meta-learning (adapting to novel MDPs) or merely memorizing a policy that covers the training distribution well.

5. Computational cost comparison. The paper does not report training time, number of episodes to convergence, or inference latency for any method. For practical adoption, the computational overhead of unrolling a GRU for 60–200 steps versus running a feedforward MLP matters. Similarly, the DR/DV law is computationally trivial (a few arithmetic operations per timestep), making its poor performance a clear case of "cheap but wrong" — but the recurrent policy's computational cost is unquantified.

6. Hidden state analysis. The paper claims the GRU hidden state captures "unobserved information such as external forces and the current lander mass" but provides no analysis of what the hidden state actually represents. A simple diagnostic — e.g., training a linear decoder from hidden state to ground-truth parameters and measuring decoding accuracy — would substantiate this claim and provide engineering confidence.

7. Robustness to sensor noise beyond altimeter errors. The experiments either use clean state-derived observations (velocity error, altitude, time-to-go) or raw altimeter readings. A middle ground — state observations with additive noise, or intermittent sensor dropouts — would test whether the recurrent policy's benefit is specific to the extreme partial observability of altimetry or generalizes to more typical state estimation uncertainty.

Limitations on the evidence:

• 10,000 test episodes per experiment provide stable means and maxima, but the maxima in particular are sensitive to the specific random seed. A single landing at 1811 m error for DR/DV in Experiment 1 drives the maximum, and a different random seed might produce an even larger or smaller maximum. Confidence intervals on the maxima are not reported.

• Single-engine-failure mode. Experiment 3 models engine failure as a factor-of-2 reduction in one horizontal axis and factor-of-1.5 in the vertical axis. Real engine failures can be partial (any fraction of thrust loss), can affect multiple engines simultaneously, and can evolve over time. The paper's failure model is a single, fixed reduction — a useful test case but not a comprehensive exploration of actuator failure modes.

• 3-DOF assumption. All experiments use translational dynamics only (no attitude control). Real landers must orient their thrust vector, which adds rotational degrees of freedom and couples attitude control with guidance. The paper acknowledges this limitation and states future work will explore "more realistic 6-DOF environments," but the current results cannot guarantee that the recurrent policy's advantage persists when attitude dynamics are added.

• No comparison to online system identification + adaptive control. The paper positions its approach against Reference 2's explicit system identification method but does not implement that method as a baseline. A comparison against a policy that receives explicit parameter estimates from a separately trained identification network would directly test the paper's claim that implicit adaptation in the hidden state is superior to explicit parameter estimation.

Conditional nature of the findings:

The paper's results hold under the specific conditions of:

• Training with domain randomization that covers the test distribution (no out-of-distribution dynamics tested).
• Sufficient unrolling length (20–60 steps for most tasks; 120–200 steps for the radar altimetry POMDP).
• The engineered gaze-heuristic reward function — performance with sparse rewards or alternative shaping functions is unknown.
• 3-DOF dynamics with thrust vector control — rotational dynamics and actuator pointing constraints are not modeled.
• Episodic tasks with clear terminal conditions (landing or crash) — the approach may not directly transfer to continuous control tasks without natural episode boundaries.

The central message — that recurrent policies provide a form of learned adaptation that outperforms fixed policies and classical model-based laws when dynamics vary substantially — is well-supported by the reported experiments. The magnitude and generality of this advantage beyond the specific scenarios tested remains an open question that the paper appropriately flags for future work.

6. Limitations and Trade-offs

The Paper Provides No Quantitative Evidence for the Multi-Discount-Rate Framework

The multi-discount-rate framework (Equations 17a–17b) is presented in Section 3 as a methodological contribution — a novel solution to the previously-unreported conflict between terminal and shaping reward discounting. The authors state explicitly that "to our knowledge, a method of resolving this conflict has not been reported in the literature" and claim that without it, "the performance was actually worsened by including the terminal reward term." They further assert that this approach "exceed[s] that of using generalized advantage estimation (GAE) both with and without multiple discount rates."

The consequence: none of these claims are supported by any reported numbers, tables, or figures. A practitioner attempting to replicate or build on this work cannot determine:

• Whether the multi-discount framework is essential to the paper's reported performance or merely a minor improvement.
• How much worse a single-discount-rate PPO baseline performs on any of the four experiments.
• Whether GAE with a single discount rate would match or exceed the multi-discount approach — the comparison is claimed but not quantified.
• What specific values of γ₁ and γ₂ were used in the experiments (the paper never reports the actual discount factors), making exact replication impossible without guessing.

This is a significant evidentiary gap because reward design choices are known to be among the most sensitive hyperparameters in deep RL, and the paper's central engineering contribution — the shaped reward function — depends on this framework to avoid harmful interference between the terminal bonus and the velocity-tracking penalty. Without baseline comparisons, the reader cannot distinguish between "the multi-discount framework is critical to making shaped rewards work" and "PPO with a single well-tuned discount factor would perform similarly."

What evidence exists in the paper: none. The claims about performance degradation without multiple discounts and about superiority over GAE appear only as prose statements in Section 3's discussion of the RL formulation. No ablation experiment, no learning curve, and no table compares single-discount to multi-discount PPO on any task.

Mitigation status: not attempted. The paper treats this framework as a given methodological tool rather than as a hypothesis requiring empirical validation. Given that the paper is primarily an empirical evaluation of recurrent policies for guidance, and the multi-discount framework is proposed as a novel contribution, the absence of evidence for its effectiveness is a conspicuous gap that weakens the paper's methodological claims and complicates replication efforts.

---

The DR/DV Baseline Is a Weak Strawman That Does Not Represent State-of-the-Art Model-Based Adaptive Control

The paper's headline result — that the recurrent policy substantially outperforms the DR/DV optimal guidance law — depends critically on the comparison being against a "bare" DR/DV implementation with no online adaptation, no parameter estimation, no robust control modifications, and no gain scheduling. The paper states that DR/DV is given "the ground truth gravitational force, as well as the lander mass at the start of an episode" — privileged information the RL agent does not receive — yet the comparison is still unfavorable to DR/DV in all three applicable experiments (1, 3, and 4).

The consequence: the paper's framing as "recurrent RL vs. classical optimal guidance" overstates the strength of the classical baseline. A fairer comparison against a model-based adaptive controller would include at minimum:

• Online mass estimation from the observed acceleration response to thrust commands (a standard technique in powered descent guidance, where mass depletion follows a known differential equation driven by the commanded thrust magnitude).
• Adaptive gain scheduling that adjusts the guidance law's parameters as the estimated mass changes.
• For the asteroid scenario, inclusion of the Coriolis and centrifugal terms in the guidance model (these are known physics; a well-designed guidance law for a rotating body would include them, estimating ω online if needed).

A model-based adaptive controller implementing these features might close much of the performance gap with the MLP policy, and might approach the recurrent policy's performance on the mass-variation and engine-failure tasks — because those tasks involve estimating parameters (mass, thrust limits) that are physically meaningful and estimable with standard techniques. The paper demonstrates that naive model-based guidance fails, which is an unsurprising result that does not constitute a strong argument against model-based approaches in general.

What evidence exists in the paper: the DR/DV performance numbers in Tables 7, 12, and 13 provide evidence only about the specific (non-adaptive) implementation tested. The paper acknowledges that the asteroid DR/DV was tuned: "we tuned the nominal gravity parameter for the DR/DV policy for best performance; it turns out that the optimal setting for this parameter is quite a bit higher than the actual environmental gravity." This tuning itself reveals brittleness — the optimal parameter value bears no relationship to the physical environment — but it does not establish that all model-based approaches would suffer the same brittleness.

Mitigation status: none. The paper does not implement or compare against any adaptive version of DR/DV, nor does it discuss what features such a version would need. The initial comparison setup (Section "Introduction") states that DR/DV is given privileged information "to improve its performance" and to make the comparison fair, but the fairness argument cuts both ways: giving partial privileged information to a non-adaptive law does not test whether model-based adaptation can match learned adaptation. This is a significant limitation for practitioners deciding between RL-based and classical approaches for a real mission — the paper provides evidence against one specific classical method but not against the broader class of adaptive model-based controllers that an experienced GNC engineer would actually consider deploying.

---

Difficulty Estimation and/or Domain Randomization Coverage Is Assumed but Not Validated Out-of-Distribution

The entire approach depends on domain randomization during training: the recurrent policy learns to adapt across the distribution of dynamics parameters (mass, gravity, rotation, engine failures) that it encounters during training, and its deployed performance depends on the test dynamics falling within or near that training distribution. The paper evaluates only in-distribution: all 10,000 test episodes for each experiment sample from the same uniform distributions used during training (Tables 3, 4, 5, 10).

The consequence: the paper's claim that the recurrent policy "adapts in real time to unknown dynamics" is imprecise. What the experiments actually demonstrate is that the policy adapts to dynamics sampled from a known distribution that matches the training distribution. This is an important distinction. A genuine test of adaptation to unknown dynamics would evaluate the policy on dynamics drawn from a different distribution — e.g., training on Mars gravity ±5% variation but testing on gravity drawn from ±20% variation, or training on engine failures with probability 0.5 and testing with probability 0.1 or 0.9, or training on asteroid rotation up to 1e-3 rad/s and testing up to 2e-3 rad/s.

Without out-of-distribution evaluation, the paper cannot distinguish between two possible explanations for the recurrent policy's strong performance:

1. Genuine meta-learning: the hidden state learns a general-purpose adaptation algorithm that can infer and compensate for any dynamics within a broad class, including dynamics not seen during training.
2. Distribution covering: the hidden state learns to recognize which region of the training distribution the current episode falls into and retrieves the appropriate behavior for that region — essentially, memorizing a set of parameter-conditioned policies without the ability to generalize to novel parameter values.

The difference matters enormously for deployment. If the mechanism is distribution covering (explanation 2), then a mission encountering dynamics outside the training range — e.g., an asteroid with unexpectedly fast rotation, or an engine failure more severe than the factor-of-2 modeled — could experience catastrophic performance degradation that the in-distribution test results give no warning about. The paper's engine failure experiment (Experiment 3) uses a specific failure model (factor-of-2 reduction in one horizontal axis, factor-of-1.5 in vertical), and the performance on, say, a factor-of-3 reduction or a failure affecting two axes simultaneously is unknown and potentially unsafe.

What evidence exists in the paper: none directly. All test distributions match the training distributions. The asteroid experiment comes closest to an out-of-distribution test in that the parameter ranges (Table 4: ω up to 1e-3 rad/s each axis, g up to 1e-4 m/s²) are described as "wide" and "limited only by the lander's thrust capability," but they are still explicitly bounded uniform distributions that are identical between training and testing. The learning curves (Figures 4, 6, 7) show training performance only; they provide no information about generalization to different parameter ranges.

Mitigation status: not attempted. The paper does not discuss out-of-distribution evaluation as a limitation or propose it as future work. The closest acknowledgment is the paper's statement that the hidden state captures "unobserved information such as external forces and the current lander mass" — a claim about what is represented, but not about whether that representation generalizes beyond the training range. For a method positioned as enabling missions where "environmental dynamics... are likely to be imperfectly modeled prior to the mission," the absence of out-of-distribution testing is a significant gap: the very scenario that motivates the work (encountering dynamics that were not anticipated during design) is precisely the scenario on which the method's performance is untested.

---

The Radar Altimetry Experiment Demonstrates Proof-of-Concept but Not Operational Accuracy, and Terrain Dependence Is Unexplored

Experiment 2 (landing using only Doppler radar altimeter returns) is presented as a demonstration that a recurrent policy can integrate guidance and navigation — implicitly performing state estimation from raw sensor history. The results in Table 9 show that the best-performing variant (200-step RNN, non-target-pointing) achieves a mean terminal position error of 59 meters with a standard deviation of 42 meters and a maximum of 288 meters. These errors are 12× the 5-meter position constraint used to define a successful landing in the other Mars experiments.

The consequence: while the result is intellectually interesting as a proof-of-concept — the policy is extracting usable state information from sensors with errors measured in thousands of meters at low altitude (Table 8: mean altimeter error 2600 m at 100 m elevation, 61% miss rate) — it does not constitute a usable integrated guidance and navigation system for planetary landing. A 59-meter mean position error means the lander would, on average, miss its targeted safe zone by a substantial margin, and the 288-meter maximum means some landings would miss by nearly 300 meters. In terrain with hazards (rocks, craters, slopes), these errors would translate to high probability of landing failure.

The paper acknowledges this candidly: "you would not want to entrust an expensive rover to this integrated guidance and navigation algorithm." But the acknowledgment is followed by "the performance is remarkably good given the altimeter inaccuracy at lower elevations" — which invites the reader to attribute the errors to sensor quality rather than to fundamental limitations of the recurrent policy's state estimation capability. The distinction matters because a practitioner needs to know whether investing in better sensors would bring the 59-meter error down to an operationally useful level, or whether the recurrent policy's implicit state estimation has inherent accuracy limits that better sensors would not overcome.

What evidence exists in the paper: the target-pointing variant (Table 11) provides partial evidence. When the altimeter beams are assumed to remain pointed at the target (improving the relevance of the altimeter readings) and initial condition uncertainty is reduced (Table 10), the 20-step RNN achieves a mean position error of 0.3 meters — a ~200× improvement over the non-target-pointing case. This suggests that the poor accuracy in the main experiment is indeed driven by sensor quality and beam geometry, not by a fundamental limit of the recurrent policy. However, the target-pointing variant imposes strong assumptions (the beams perfectly track the target, initial conditions are tightly constrained) that may not hold in practice without additional engineering (a separate beam-pointing control system), partially defeating the purpose of an integrated end-to-end approach.

The paper also notes — without providing quantitative results — that "when we repeat the experiment for a landing site further to the south (bottom of DTM), we find that performance degrades." This terrain-dependence is acknowledged but not characterized: the reader does not know whether degradation means 70 m instead of 59 m, or 300 m instead of 59 m. This is a significant gap because it suggests the integrated guidance and navigation capability is fragile with respect to terrain characteristics (e.g., terrain diversity, which affects how much information each altimeter reading provides about position).

Mitigation status: partial. The target-pointing variant shows that performance can improve dramatically under better sensing conditions, suggesting a path toward operational viability if beam-pointing can be achieved. The terrain-dependence observation is flagged but not addressed: no quantitative sensitivity analysis is provided, and no terrain-robustness techniques are proposed. The paper suggests future work on "different sensor models for observations such as simulated camera images, and flash LIDAR for asteroid close proximity missions" (Conclusion), which would address the sensor-quality limitation but not the terrain-sensitivity limitation directly.

---

Training and Architecture Hyperparameters Are Substantial but Not Ablated, and the 1-Step RNN Anomaly Suggests GRU Instability Without Sufficient Temporal Depth

The paper specifies a complex architecture with numerous design choices: four-layer networks with GRU as the second layer, specific layer size formulas (10× obs_dim, $\sqrt{nh_1 \times nh_3}$, 10× act_dim), tanh activations throughout, the captured-hidden-state unrolling scheme with specific unroll lengths, the dynamic ϵ adjustment targeting KL divergence of 0.001, and the multi-discount-rate framework with unspecified γ₁ and γ₂ values. None of these choices are ablated — there is no experiment showing sensitivity to layer sizes, activation functions, unroll length beyond the tested values (1, 20, 60, 120, 200), or the KL target.

The consequence: a practitioner attempting to apply this method to a different landing scenario or a different dynamics problem cannot determine which architectural choices are essential and which are incidental. Should they use exactly 10× obs_dim for the first layer, or would 5× or 20× work equally well? Is the GRU specifically important, or would an LSTM perform comparably? Is the $\sqrt{nh_1 \times nh_3}$ sizing formula for the recurrent layer important, or is it an arbitrary heuristic? Without sensitivity analysis, the method is a recipe rather than a principled architecture — follow the exact recipe and it may work; deviate and performance is unpredictable.

This limitation is compounded by an anomalous result in Experiment 4 (high mass variation, Table 13): the 1-step RNN produces a worse maximum terminal velocity (6.48 m/s) than the MLP (5.25 m/s). Both the 1-step RNN and the MLP have effectively no temporal memory (unrolling for 1 step means the GRU processes only the current observation, functioning essentially as a nonlinear feedforward transformation with an irrelevant hidden state). Yet the 1-step RNN is worse than the simpler MLP on the worst-case velocity metric. The paper does not comment on this anomaly, but it suggests that the GRU architecture, when deprived of meaningful temporal context, can introduce harmful state dynamics that a purely feedforward network avoids. This is a practical concern: if the necessary temporal horizon for a new task is unknown, and a practitioner deploys a GRU policy with insufficient unrolling, they may get worse performance than if they had used a simpler MLP.

What evidence exists in the paper: the anomaly appears in Table 13 (maximum velocity: MLP 5.25 m/s vs. 1-step RNN 6.48 m/s). A similar pattern may exist in Experiment 3 (Table 12: maximum velocity MLP 4.67 m/s vs. 1-step RNN 4.76 m/s), though the difference is smaller. No other architecture ablations are reported.

Mitigation status: not attempted. The paper does not discuss the 1-step RNN anomaly, does not perform any architecture sensitivity analysis, and does not provide guidance on how to select unrolling length for a new task beyond the empirical observation that "the amount of steps we unroll the recurrent network in the forward pass has a large impact on optimization performance" (Experiment 2 discussion). For a method that relies on a specific recurrent architecture as its central innovation, the absence of architectural ablations limits confidence that the reported performance is robust to implementation details and complicates transfer to new problems.

---

The 3-DOF Simplification Avoids Attitude Control Coupling, and Performance in 6-DOF with Realistic Actuator Constraints Is Unknown

All experiments model the lander as a point mass with direct thrust vector control in three translational degrees of freedom (Equations 1a–1c). The lander can apply thrust in any direction instantaneously, with no modeling of attitude dynamics (orientation, angular velocity, moment of inertia), no actuator slew rate limits, and no coupling between the direction the lander is facing and the direction it can thrust. This is a standard simplification for guidance law development — the guidance system produces a desired thrust vector, and a separate attitude control system (assumed perfect) orients the spacecraft to realize it — but it eliminates a major source of complexity that interacts with the unknown-dynamics problem.

The consequence: several of the paper's key scenarios would be substantially harder in 6-DOF. Consider:

• Engine failure (Experiment 3): in 3-DOF, an engine failure simply reduces the available thrust magnitude along one axis. In 6-DOF, a thruster failure on one side of the spacecraft creates a torque imbalance — the remaining healthy thrusters produce a net torque that rotates the spacecraft unless compensated by other thrusters. The guidance system must not only adjust its commanded thrust vector to account for reduced authority but must also ensure that the thrust vector remains realizable given the spacecraft's current attitude and the available healthy thrusters. This couples the guidance problem to the attitude control problem in a way that the 3-DOF model completely elides.

• Large mass variation (Experiment 4): as mass depletes, the lander's moment of inertia changes (mass is leaving the tanks, shifting the center of mass and changing the inertia tensor). In 3-DOF, only the translational acceleration response changes. In 6-DOF, the changing inertia affects the rotational acceleration response to attitude control torques, creating a time-varying rotational dynamics problem that compounds the translational mass-variation challenge. The recurrent policy would need to track two time-varying parameters (mass for translation, inertia tensor for rotation) from the same observation stream.

• Asteroid landing (Experiment 1): the asteroid's rotation creates Coriolis and centrifugal forces that depend on the lander's position in the rotating frame. In 3-DOF, these are simply additional acceleration terms in the translational equations. In 6-DOF, the lander's attitude relative to the rotating frame matters — gravity-gradient torques (differential gravitational pull on different parts of the spacecraft) can cause the lander to tumble if not actively controlled, especially for elongated spacecraft in the weak, irregular gravity field of a small asteroid.

The paper acknowledges the 3-DOF limitation in the Conclusion: "Future work will explore the adaptability of recurrent policies in more realistic 6-DOF environments." This is appropriate, but it means that the paper's central claim — that recurrent policies provide adaptive guidance that outperforms classical methods under unknown dynamics — has been demonstrated only in a simplified dynamics setting where attitude control is assumed solved. The claim may or may not hold when the adaptation problem includes rotational dynamics.

What evidence exists in the paper: none for 6-DOF. All equations, all experiments, and all results are for 3-DOF translational dynamics with direct thrust vector control. The paper does not discuss how the observation space, action space, or reward function would need to change for 6-DOF.

Mitigation status: the limitation is acknowledged as future work. However, the acknowledgment does not address the specific ways in which 6-DOF dynamics would challenge the recurrent policy's adaptation capability. A practitioner considering this approach for a real mission would need to assess whether the demonstrated adaptation capability — which involves inferring unobserved translational parameters (mass, external forces) from observation history — would extend to inferring rotational parameters (inertia tensor, thruster geometry, attitude-dependent disturbances) from an observation stream that would need to include attitude information (e.g., from star trackers, IMUs). This is a non-trivial extension that the paper does not address.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a conceptual reframing of adaptive guidance as a meta-learning problem rather than a system-identification-plus-control decomposition. The shift is not that neural networks can control spacecraft — prior work had established that (Reference 4) — but rather that the mechanism of adaptation itself can be a trained property of a recurrent architecture rather than an engineered module. This is an architectural insight about where the adaptation should live: not in an explicit parameter estimator feeding a controller, but in the hidden state dynamics of a single end-to-end network trained with domain randomization.

The magnitude of this reframing should not be overstated. This is not a paradigm shift that overturns control theory — the paper does not provide theoretical guarantees, the experiments are in simplified 3-DOF dynamics, and the results on the hardest task (radar altimetry, 59 m mean error without target-pointing) are not operationally viable. Rather, it is a demonstration of viability that opens a design space previously underexplored: learned recurrent policies as implicit adaptive controllers for aerospace guidance. The paper's contribution is best understood as providing the first systematic empirical evidence, across four distinct types of dynamical uncertainty, that this design pattern works and where it works — the engine failure and mass variation experiments (Tables 12–13) show that recurrence provides qualitative safety improvements (eliminating velocity outliers) that non-recurrent learned policies cannot match, while the asteroid experiment (Table 7) shows that the benefit is more modest (~12% fuel savings) when the adaptation problem involves slowly-varying external parameters that a fixed MLP policy can handle.

Reconciling prior tensions. The paper resolves a latent tension between two lines of prior work. Reference 4 (the authors' own prior work) showed that training MLP policies with domain randomization produces robust fixed controllers, but noted limitations when "parameter uncertainty is not too extreme." Reference 2 showed that explicit system identification with a recurrent network can improve RL performance, but required a separate supervised training phase for the system identifier. This paper's results clarify when each approach is sufficient: the MLP policy performs adequately when unknown parameters vary slowly and continuously around a nominal value (asteroid gravity ±5%, mass ±10%), but recurrent memory becomes essential when the unknown involves a discrete persistent change (engine failure) or requires continuous inference from ambiguous observations (mass variation without direct measurement, radar altimetry for state estimation). The 1-step RNN's consistently intermediate performance — neither as good as longer-unrolled RNNs nor as bad as the MLP on hard tasks — confirms that the benefit comes from temporal depth, not merely from the GRU architecture's nonlinearity.

Research directions that become more attractive. The paper makes three directions newly compelling: (1) investigating what the recurrent hidden state represents through post-hoc analysis (decoding physical parameters from hidden activations), (2) developing formal verification methods for recurrent policy behavior under out-of-distribution dynamics, and (3) exploring whether the meta-learning framing generalizes to 6-DOF with realistic actuator constraints. The paper also makes one direction less attractive: developing increasingly sophisticated explicit system identification modules for RL-based control. If a recurrent policy can learn implicit identification end-to-end — without supervised parameter labels and without architectural separation — the engineering effort of building and tuning separate identification networks may be better spent on improving the recurrent policy's training distribution and reward design.

Research directions that become less attractive. The paper's negative result with DR/DV — catastrophic failure under engine failure (1061 m max position error, 59.29 m/s max velocity) and asteroid landing (1811 m max position error) despite receiving privileged ground-truth information — should discourage the strategy of "take an optimal guidance law derived for nominal dynamics and hope it degrades gracefully." The paper demonstrates empirically that graceful degradation is not a property of model-based optimal controllers when the model mismatch is large. This redirects effort away from robustifying classical guidance laws through ad-hoc parameter tuning (the asteroid DR/DV required a tuned gravity parameter "quite a bit higher than the actual environmental gravity") and toward learned controllers that are trained for robustness from the start.

---

Follow-Up Research This Work Enables

Post-hoc decoding of the GRU hidden state to verify what physical parameters are being tracked. The paper claims that the recurrent hidden state "captures unobserved information such as external forces and the current lander mass" but provides no evidence about what the hidden state actually encodes. A natural follow-up would train a linear decoder from the hidden state activations (recorded during test episodes) to ground-truth parameters (mass, gravity vector, engine health status, rotational velocity) and measure decoding accuracy. This would serve two purposes: (1) it would verify the paper's claim about information capture, and (2) it would reveal whether the hidden state represents physically interpretable quantities (which would increase trust for safety-critical deployment) or task-specific abstractions (which would suggest the meta-learning interpretation is accurate but complicate verification). A strong result would show that the hidden state linearly encodes mass with high accuracy and engine failure status with a simple threshold, while a null result (no linear decodability) would suggest the hidden state uses distributed, non-interpretable representations — important information for deciding whether this approach can be certified for flight.

Out-of-distribution generalization stress test across dynamics parameters. The paper evaluates only in-distribution (test dynamics sampled from the same uniform distributions as training). This leaves open whether the recurrent policy's adaptation is genuine meta-learning (a learned algorithm for inferring any dynamics within a broad class) or distribution covering (memorizing parameter-conditioned behaviors for the training range). A stress test would train policies on one range of dynamics (e.g., gravity ±5%, engine failure probability 0.5) and evaluate on shifted ranges (gravity ±20%, engine failure probability 0.1 or 0.9, or failure magnitudes 3× reduction instead of 2×). The key measurement is whether the performance gap between the recurrent policy and the MLP widens, narrows, or stays constant as the test distribution shifts further from training. If the recurrent policy maintains its advantage on moderate out-of-distribution shifts but collapses on extreme shifts, that would establish a "generalization horizon" for the learned adaptation — practically important for mission design, where the actual dynamics will almost certainly differ from any finite training set. A particularly informative negative result would be if the recurrent policy degrades faster than the MLP out-of-distribution, suggesting that the hidden state dynamics overfit to the training distribution in ways that produce actively harmful adaptation when confronted with unfamiliar observations.

Comparison against online model-based adaptive control (extended Kalman filter + DR/DV with parameter estimation). The paper's DR/DV baseline is a non-adaptive implementation, which makes the comparison informative about naive model-based control but not about the broader class of adaptive model-based methods that an experienced GNC engineer would consider. A strong follow-up would implement an online mass and thrust-limit estimator (e.g., an extended Kalman filter using the known mass depletion dynamics in Equation 1c and the observed acceleration response to thrust commands) feeding an DR/DV law with parameters updated in real time, and compare this against the recurrent policy on the engine failure and mass variation experiments. This comparison would directly test whether the recurrent policy's implicit adaptation outperforms explicit estimation when the parameters being estimated are physically meaningful and estimable with standard techniques. The prediction from the paper's framing is that the recurrent policy would still hold an advantage on the engine failure task (where the failure is discrete and requires rapid detection from limited data, a regime where the GRU's nonlinear filtering may outperform a Kalman filter's linear-Gaussian assumptions) but the explicit estimator might match or approach the recurrent policy on the mass variation task (where the parameter varies continuously according to known dynamics). Establishing this boundary would clarify when the meta-learning approach is worth its additional complexity and opacity.

Scaling analysis: how does the required unrolling length grow with episode duration and sensor degradation? The paper experiments with unrolling lengths of 1, 20, 60, 120, and 200 steps but does not provide a principled answer to the question: "for a new landing scenario, what unrolling length should I use?" The results suggest that 20–60 steps suffices for tasks where the unknown parameters are constant or slowly varying (Experiments 1, 3, 4), while 120–200 steps helps for the POMDP task (Experiment 2) where disambiguating state from ambiguous observations requires integrating information over ~20–30 seconds of flight. A systematic scaling study would vary episode duration (by changing initial altitude or velocity in Table 3) and sensor degradation (by varying the altimeter error characteristics in Table 8) and measure the optimal unrolling length for each condition. The hypothesis is that required unrolling length scales with the information integration timescale — the time needed for the observation history to disambiguate the unknown parameters or hidden state. If this timescale can be estimated from sensor models and dynamics uncertainty bounds before training, it would provide a principled way to set the unrolling hyperparameter for new scenarios without the trial-and-error that the paper implicitly relies on.

Training a recurrent policy on camera images or flash LIDAR for asteroid close-proximity operations. The paper's Conclusion explicitly mentions "simulated camera images, and flash LIDAR for asteroid close proximity missions" as future work. Experiment 2 (radar altimetry) demonstrates that recurrent policies can extract state information from raw range measurements, but camera images and LIDAR point clouds provide much richer but higher-dimensional observations. A concrete follow-up would replicate Experiment 1 (asteroid landing) but replace the state-derived observation vector (Equation 16) with rendered images from a nadir-pointing camera or simulated flash LIDAR point clouds of the asteroid surface, and train a recurrent policy with convolutional layers preceding the GRU. The specific question is whether the recurrent hidden state can learn to perform visual odometry and terrain-relative navigation simultaneously with guidance — integrating the functions of three traditional spacecraft subsystems (optical navigation, hazard detection, and guidance) into one learned policy. The paper's radar altimetry results (59 m error without target-pointing, 0.3 m with target-pointing, Table 11) provide a baseline: if visual observations enable accuracy below 1 m without requiring beam-pointing assumptions, that would demonstrate a substantial advance in autonomous asteroid landing capability. A negative result — that the high dimensionality of visual observations makes end-to-end recurrent policy training unstable or requires prohibitive amounts of simulated experience — would be equally informative, suggesting that the approach works best with low-dimensional, information-dense sensor streams and that visual perception may need to remain a separate preprocessing stage.

---

Practical Applications and Downstream Use Cases

Mars lander backup guidance system for engine-out scenarios. The paper's Experiment 3 (Table 12) demonstrates that a 20-step recurrent policy achieves safe landings (max velocity 1.15 m/s, max position error 0.9 m) under a 50% probability of engine failure reducing thrust by a factor of 2 on one horizontal axis and 1.5 on the vertical axis — conditions under which the baseline DR/DV guidance law fails catastrophically (max position error 1061 m, max velocity 59.29 m/s). A practical application is deploying this policy as a backup guidance mode that activates when the primary guidance system detects an actuator fault (e.g., through unexpected acceleration residuals). The benefit is risk reduction: for a flagship Mars mission where landing failure represents a multibillion-dollar loss, a backup system that converts a 50% probability of catastrophic failure under engine loss to a near-zero probability (0 unsafe landings in 10,000 test episodes for the 20-step RNN) provides substantial expected value, even if the recurrent policy uses 295 kg of fuel compared to the MLP's 302 kg under nominal conditions. The 3-DOF limitation means this would need to be paired with an attitude control system that can handle the torque imbalance from asymmetric thruster failure — the guidance policy assumes thrust can be directed arbitrarily, which may not hold under realistic engine-out geometry.

Asteroid surface exploration with a single policy that adapts to each new body. The paper's Experiment 1 (Table 7) trains a single recurrent policy that handles asteroid landings across a wide range of gravitational accelerations (up to 1e-4 m/s², roughly 1% of Earth's gravity), rotational velocities (up to 1e-3 rad/s per axis), and solar radiation pressures — and achieves 1.42 kg mean fuel consumption with a maximum position error of 1.7 m at test time, without any per-asteroid tuning. This suggests an operational concept where a fleet of small landers is deployed to multiple asteroids with poorly characterized physical properties, and all landers run the same pre-trained recurrent guidance policy. Each lander's policy adapts its hidden state to the local dynamics within the first few seconds of descent (using the observed acceleration response to its initial thrust commands), then executes a tailored fuel-efficient trajectory. The practical benefit is eliminating the need for pre-arrival characterization campaigns (which are expensive and sometimes impossible for fast flyby missions) and eliminating the risk of landing failure due to incorrect pre-arrival parameter estimates. The fuel savings relative to a fixed MLP policy (1.42 kg vs. 1.62 kg mean, with 0.38 kg vs. 0.92 kg standard deviation) translate to either reduced wet mass requirements or extended surface operations capability.

Self-improving lander using telemetry from previous flights. The paper notes in the RL Overview section that "the safe method of continuous learning in the field is to have the lander send back telemetry data, which can be used to improve the environment's dynamics model, and update the policy via simulated experience." This suggests a practical pipeline: after each landing (or after each landing attempt, successful or not), the lander transmits its recorded trajectory (observations, actions, rewards) to Earth. Engineers use this data to refine the dynamics simulator (e.g., updating the asteroid gravity model, characterizing the altimeter error statistics for the specific terrain encountered), then retrain the recurrent policy in the updated simulator and uplink the new policy weights to the next lander. Over multiple missions to similar bodies, the policy improves without requiring on-board learning (which carries risks of policy degradation and is difficult to verify pre-flight). The paper's domain randomization framework makes this pipeline natural: each new batch of telemetry narrows the randomization ranges for subsequent training, progressively improving policy performance without changing the architecture or training algorithm. The specific benefit is that the first lander in a campaign handles the widest dynamics uncertainty (like the paper's Experiment 1 ranges), while subsequent landers benefit from progressively tighter priors.

---

When to Prefer This Method

The paper articulates a clear tradeoff between recurrent and non-recurrent (MLP) learned policies, and between learned policies and classical model-based guidance (DR/DV). These tradeoffs are grounded in specific experimental results rather than stated as general principles, but they imply decision rules:

• Prefer a recurrent policy over an MLP when the unknown dynamics involve discrete state changes (engine failure, Table 12: MLP max velocity 4.67 m/s vs. 20-step RNN max 1.15 m/s), large continuous parameter variation that must be tracked from observation history (mass variation, Table 13: MLP max velocity 5.25 m/s vs. 60-step RNN max 1.21 m/s), or partially observable state requiring temporal integration of ambiguous sensor data (radar altimetry, Table 9: MLP mean position error 131 m vs. 200-step RNN 59 m). The recurrent policy's advantage grows with the information integration requirement — how much of the relevant hidden state can only be inferred from observation sequences rather than from the current observation alone.

• Prefer an MLP policy over a recurrent policy when the dynamics uncertainty is limited to modest continuous parameter variation around a nominal value (asteroid landing, Table 7: MLP achieves 0.2 m mean position error and 4.1 cm/s velocity, already within constraints; recurrent's advantage is only ~12% fuel savings). The MLP is architecturally simpler, computationally cheaper at inference time (no hidden state maintenance or GRU gating computations), and avoids the 1-step RNN anomaly observed in Experiment 4 (Table 13: 1-step RNN max velocity 6.48 m/s worse than MLP's 5.25 m/s). If the task does not require temporal memory, the MLP is the safer choice.

• Prefer a learned policy (MLP or recurrent) over DR/DV when the deployment dynamics are expected to deviate substantially from the model used to derive the guidance law, and failures are catastrophic. The DR/DV law fails catastrophically under engine failure (max error 1061 m, Table 12) and asteroid dynamics (max error 1811 m, Table 7) despite receiving privileged ground-truth information that the learned policies do not receive. This is not a claim that learned policies always outperform model-based control — DR/DV would likely perform well on nominal dynamics — but rather that model-based optimality guarantees are brittle when the model is wrong, and learned policies trained with domain randomization provide a more robust form of near-optimality across the expected distribution of model mismatch.

• Do not deploy the recurrent policy as an integrated guidance and navigation system without additional engineering if operational accuracy requirements are tight (<5 m position error). The non-target-pointing radar altimetry results (Table 9: 59 m mean error for the 200-step RNN) demonstrate proof-of-concept but not operational viability for precision landing. The target-pointing variant (Table 11: 0.3 m mean error) shows what is possible with improved sensor geometry, but achieving that geometry in practice requires a separate beam-pointing system that partially defeats the purpose of end-to-end integration. For missions requiring precision landing on hazardous terrain, the current approach would need to be augmented with either better sensors, a dedicated navigation filter, or a target-pointing mechanism — that is, it would need to move back toward the traditional separated architecture that the paper's Experiment 2 challenges.