VLingNav: Embodied Navigation with Adaptive Reasoning and Visual-Assisted Linguistic Memory

ArXiv: 2601.08665

🎯 Pitch

Navigation agents typically waste equal computation on every step, but VLingNav learns to reason explicitly on only 2.1% of moves—triggering deep planning precisely when needed. This selective cognition, paired with a linguistic memory that compresses past visuals into searchable text, lets a single model jump 12 points ahead on MP3D ObjectNav and operate zero-shot on a real robot navigating entirely new tasks.

---

1. Executive Summary

VLingNav proposes a Vision-Language-Action model for embodied navigation that integrates two cognitively-inspired mechanisms: an Adaptive Chain-of-Thought (AdaCoT) mechanism, which dynamically triggers explicit reasoning only at critical decision points—activating on just 2.1% of steps on average—and a Visual-Assisted Linguistic Memory (VLingMem) module, which distills visual observations into persistent cross-modal linguistic summaries to prevent redundant exploration and track spatial history across long horizons. Trained on Nav-AdaCoT-2.9M, the largest embodied navigation dataset with reasoning annotations to date (2.9M trajectory steps, 472K CoT annotations), and further refined through an online expert-guided reinforcement learning stage, VLingNav achieves state-of-the-art performance across ObjectNav, Embodied Visual Tracking, and ImageNav benchmarks—improving HM3Dv1 success rate by 5.4 points over Uni-NaVid (79.1% vs. 73.7%) and MP3D success rate by 12.3 points over CogNav (58.9% vs. 46.6%)—while also transferring zero-shot to a real-world quadruped robot, establishing that adaptive reasoning combined with linguistic memory enables robust sim-to-real generalization across both seen and unseen navigation tasks.

2. Context and Motivation

The Core Problem: Reactive Navigation Without Reasoning or Memory

The fundamental problem this paper tackles is that existing Vision-Language-Action (VLA) models for embodied navigation are fundamentally reactive systems: they map observations directly to actions without engaging in explicit reasoning about why those actions are chosen, and without maintaining a persistent semantic memory of what they have already experienced during a trajectory. This limitation manifests concretely in two failure modes that the authors identify as pervasive across current approaches.

First, current VLA navigation models cannot modulate their computational effort based on situational complexity. Whether navigating a straight, empty corridor or deciding which direction to take at a complex intersection, the model spends the same fixed amount of computation. This is fundamentally misaligned with how effective decision-making works in practice: easy situations require quick, reflexive responses, while ambiguous or critical situations demand deeper deliberation. The paper frames this as a missing capability for adaptive reasoning — the agent should be able to decide when to think carefully versus when to act immediately.

Second, existing models lack persistent semantic memory for tracking spatial history over long trajectories. While many recent VLA approaches (NaVid, Uni-NaVid, NaVILA, StreamVLN) encode historical visual frames as inputs, this provides only implicit visual memory — compressed visual features that lose semantic fidelity over time. The consequence is well-documented in the literature and reproduced in the paper's ablation studies: agents exhibit redundant exploration, looping behaviors, and poor adaptation to dynamic environments because they cannot explicitly remember which rooms they have already searched or what objects they have already encountered. As the authors state in Section 1:

"Without a mechanism to retain historical context, agents struggle to track their progress over extended trajectories, resulting in redundant exploration, looping behaviors, and poor adaptation to dynamic changes in the environment."

These two gaps — no adaptive reasoning and no explicit memory — are not merely inconveniences. They fundamentally cap the performance of VLA navigation models on long-horizon tasks where both capabilities become essential. A robot searching for a specific object in a large, multi-room environment must remember where it has already looked and must think carefully when it encounters ambiguity (e.g., "Is this the refrigerator the instruction asked for, or a different appliance that looks similar?"). Current models fail precisely at these critical junctures.

Why This Problem Matters: From Theory to Deployment

The significance of addressing these gaps extends beyond incremental benchmark improvements and touches on fundamental questions about what makes embodied agents robust, interpretable, and deployable in real-world settings.

Practical deployment considerations. The paper's real-world experiments (Section 6.3) make concrete why memory matters: a robot deployed in a home environment searching for a microwave (Section 6.3.2) must avoid re-checking the same rooms repeatedly — each redundant traversal wastes time, drains the battery, and degrades the user's trust in the system's competence. The 15.4% success rate of the "w/o Memory" ablation on HM3D OVON (Table 7) versus 50.1% with the full VLingMem module quantifies this gap starkly: without explicit memory, agents succeed less than a third as often on open-vocabulary navigation tasks.

Similarly, the adaptive reasoning capability directly addresses a tension between inference latency and decision quality. A robot that runs chain-of-thought reasoning at every step would be prohibitively slow for real-time navigation (the paper shows that dense per-step CoT actually degrades performance — 25.3% SR vs. 36.2% without CoT on ObjNav, Table 6). But never reasoning at all leaves the agent unable to handle novel or ambiguous situations. The AdaCoT mechanism's ability to achieve state-of-the-art performance while activating reasoning on only 2.1% of steps demonstrates that this tension is resolvable: intelligent agents should allocate cognitive resources adaptively, not uniformly.

Scientific significance. From a research perspective, the paper addresses a deeper gap in how VLA models are understood. Current approaches treat VLMs primarily as perception-to-action translators — the VLM processes visual input and generates action tokens, but its internal reasoning capabilities (the very thing that makes LLMs powerful on text-based tasks) remain largely untapped. As the authors note in Section 2.1:

"existing navigation VLA models rely solely on action labels for finetuning and thus fail to exploit the inherent reasoning capabilities of VLMs"

This is a significant missed opportunity. VLMs like LLaVA-Video-7B (which VLingNav extends) have been trained on vast amounts of visual reasoning data and can, in principle, perform sophisticated spatial reasoning, object identification, and task decomposition. The failure to leverage these capabilities in navigation models represents an architectural gap between what VLMs can do and what VLA models ask them to do.

Where Prior Approaches Fall Short

The paper identifies specific limitations across four categories of existing work, each of which VLingNav is designed to address.

VLA Navigation Models: Video-Based but Memory-Limited

The most directly comparable prior work is the recent generation of video-based VLA navigation models. NaVid (Zhang et al., 2024a) pioneered this approach, demonstrating that video-conditioned VLMs could be fine-tuned for navigation with strong generalization. Uni-NaVid (Zhang et al., 2025b) extended this to multiple navigation tasks and introduced video-stream compression to control token count. NaVILA (Cheng et al., 2025) and StreamVLN (Wei et al., 2025) incorporated large-scale open-world data and KV-cache optimizations for efficiency. JanusVLN (Zeng et al., 2025) fused 3D spatial features from VGGT for improved instruction following.

However, all of these models share a critical limitation: they maintain history exclusively through implicit visual features. As the compressed visual tokens are repeatedly processed through the VLM, semantic information degrades — the model progressively loses track of what it has seen, even if the visual features retain some where information. Section 2.3 makes this explicit:

"such implicit memory can hinder learning to focus on key regions, and semantic information is further degraded as visual features are repeatedly compressed"

The consequence is that these models perform well on short-horizon tasks but degrade significantly on long-horizon navigation where revisiting decisions and tracking explored areas becomes essential. The paper's experimental evidence supports this: while Uni-NaVid achieves 73.7% SR on HM3Dv1 (comparable to VLingNav's 70.6% at the SFT stage), its performance on the longer-horizon MP3D benchmark and the memory-intensive OVON open-vocabulary task lags substantially behind VLingNav's final model (Tables 2 and 3).

Embodied Chain-of-Thought: Fixed Schedules, No Adaptivity

Several recent works have attempted to bring chain-of-thought reasoning to embodied tasks, but the paper argues they suffer from a common flaw: fixed reasoning schedules that cannot adapt to situational demands.

Embodied-CoT (Zawalski et al., 2024) pioneered structured textual reasoning for manipulation tasks. CoT-VLA (Zhao et al., 2025) and VPP (Hu et al.) integrated reasoning via future image prediction. π₀.₅ (Black et al., 2025) performed task decomposition through text. ChatVLA-2 (Zhou et al.) enhanced complex visual reasoning through additional pre-training data. ThinkAct (Huang et al., 2025) designed a dual-system framework bridging high-level reasoning with low-level action.

Critically, all of these are limited to tabletop manipulation tasks. Only OctoNav (Gao et al., 2025) extended CoT reasoning to navigation in open spaces, but it did so with a fixed frequency — executing CoT at predetermined intervals regardless of whether reasoning was actually needed at that step. The paper identifies this as a fundamental limitation:

"the requirement for manual configuration of the CoT frequency impedes the full exploitation of CoT's potential" (Section 2.2)

Aux-Think (Wang et al., 2025c) observed empirically that excessive reasoning can actually degrade performance, a finding that VLingNav's ablation studies reproduce dramatically: dense per-step CoT drops ObjNav success rate from 36.2% to 25.3% (Table 6). This paradoxical result — that reasoning too much hurts navigation — underscores why adaptivity is not merely an efficiency concern but a performance necessity.

NavA³ (Zhang et al., 2025c) attempted to use GPT-4o for reasoning and 3D spatial localization, but suffers from impractically long inference latency, making real-robot deployment infeasible. VLingNav positions adaptive CoT as the synthesis that addresses all three limitations: it reasons only when necessary (unlike fixed-interval methods), it avoids the performance degradation of excessive reasoning (unlike dense CoT), and it maintains inference efficiency suitable for real-time deployment (unlike NavA³).

Memory in VLA Models: Coarse Latents or Implicit Features

Memory mechanisms in VLA models have evolved along several lines, but the paper argues each approach has fundamental limitations for navigation.

RoboFlamingo (Li et al., 2023) compresses vision-language representations into latent tokens and propagates them through an LSTM. The resulting latent representations are coarse-grained, losing fine-grained perceptual history. MemoryVLA (Shi et al., 2025) integrates cognitive semantics and perceptual details in a unified framework but uses only a single implicit cognitive token as semantic memory — this single vector cannot capture the rich, structured information (what rooms were visited, what objects were seen, what decisions were made) that long-horizon navigation requires.

On the navigation side, video-based VLA models (NaVid, Uni-NaVid, NaVILA, StreamVLN, TrackVLA) provide implicit visual memory by encoding historical frames, but as discussed above, the visual features degrade semantically through repeated compression. Mem2Ego (Zhang et al., 2025e) and MapNav (Zhang et al., 2025d) incorporate global map information as memory, but VLM backbones lack native support for map-format inputs, and map representation design for VLAs remains under-explored.

The paper's key insight about memory is that language is the natural modality for memory in VLA systems. As stated in Section 2.3:

"Compared with latent-, vision-, or map-based memories, language memory is better aligned with the VLA framework, thanks to large-scale language pretraining."

This motivates VLingMem's design: use visual features as auxiliary signals to ground and enrich linguistic summaries, but let the memory itself be represented in language — the modality that VLMs process most naturally and retain most robustly. The ablation in Table 7 confirms this: language-only memory provides some benefit (18.8% SR vs. 15.4% without memory), visual-only is substantially better (45.2%), but the combination in VLingMem (50.1%) significantly outperforms either alone.

Training Paradigms: Imitation Learning Ceilings and the RL Gap

The paper identifies a limitation in how VLA navigation models are trained. Most current approaches rely on pure supervised fine-tuning via imitation learning on expert demonstrations. This has well-known failure modes:

• Covariate shift: during deployment, the agent encounters states not represented in the training distribution, and errors compound because the policy has never practiced recovering from its own mistakes.
• Causal confusion: the model may learn spurious correlations in the expert data rather than genuine decision-making strategies.
• Ceiling effect: the model can only perform at the level of the demonstrations, never discovering superior strategies through exploration.

While reinforcement learning has proven transformative for LLMs and VLMs on complex reasoning tasks (DeepSeek-R1, Video-R1), its application to VLA navigation remains preliminary. OctoNav, VLN-R1 (Qi et al., 2025), and Nav-R1 (Liu et al., 2025b) have convergently integrated GRPO into navigation VLAs for discrete autoregressive actions. ActiveVLN (Zhang et al., 2025h) uses outcome-based RL with GRPO by caching historical actions into tokens.

However, the paper identifies a critical gap: existing VLA-RL frameworks are confined to discrete action spaces. As noted in Section 2.4:

"The aforementioned work remains confined to autoregressive action outputs, failing to support more advanced continuous action prediction."

Continuous action prediction (trajectories of $(x, y, \theta)$ waypoints) offers finer-grained control and higher action quality compared to discrete action tokens, but brings RL training challenges that prior work hasn't addressed. ReinFlow (Zhang et al., 2025g) addresses continuous control through flow matching as an MDP, but at the cost of slow inference due to iterative denoising.

VLingNav positions its expert-guided online RL as filling this gap: it supports continuous action prediction through a probabilistic policy head (multivariate Gaussian parameterized by the VLM backbone), while the hybrid rollout strategy (alternating naive policy rollouts with expert-guided recovery demonstrations) addresses the exploration efficiency and instability challenges that plague pure RL in sparse-reward, long-horizon navigation tasks.

How VLingNav Positions Itself

The paper's positioning emerges clearly from the synthesis of these four gaps. It is not proposing a radically new single mechanism, but rather an integrated architecture that combines adaptive reasoning, linguistic memory, and RL-based policy refinement in a way that addresses the limitations of each prior approach:

• Against video-based VLAs (NaVid, Uni-NaVid, NaVILA): VLingNav adds explicit linguistic memory (VLingMem) to complement implicit visual memory, and adds adaptive reasoning (AdaCoT) to leverage the VLM's reasoning capabilities that these prior works leave untapped. The key differentiator is not the VLM backbone (LLaVA-Video-7B is standard) but the cognitive architecture built around it.

• Against embodied CoT (Embodied-CoT, OctoNav, CoT-VLA): VLingNav's AdaCoT is adaptive rather than fixed-schedule, learning from data when to think rather than relying on human-configured frequencies. The 2.1% reasoning activation rate versus the performance degradation of dense CoT (Table 6) provides empirical validation that adaptivity matters decisively.

• Against prior memory mechanisms (RoboFlamingo, MemoryVLA, Mem2Ego): VLingMem uses language as the primary memory modality with visual features as auxiliary signals, better aligned with the VLM's pretraining and more robust to information decay than latent vectors or compressed visual features.

• Against existing VLA-RL (OctoNav, Nav-R1, SimpleVLA-RL): VLingNav supports continuous action RL with an expert-guided hybrid rollout, avoiding the discrete-action limitation of prior work while maintaining inference efficiency (unlike flow-based approaches).

The paper frames these contributions through a unifying metaphor drawn from cognitive science — the dual-process theory of fast and slow thinking — which provides a principled motivation for why adaptivity matters and how memory should support both modes of cognition. This theoretical grounding distinguishes VLingNav from prior work that added CoT or memory as engineering heuristics rather than as components of a coherent cognitive architecture.

3. Technical Approach

3.1 Reader orientation

VLingNav is a complete system for embodied robot navigation that takes video frames from an egocentric camera plus a natural language instruction and produces continuous motion trajectories — sequences of $(x, y, \theta)$ waypoints — to guide a mobile robot through unseen environments toward specified goals. The system solves the problem of reactive, memory-less navigation by building a cognitive architecture around a pre-trained Vision-Language Model that can decide when to think carefully about what it sees, remember what it has already encountered in explicit language summaries, and learn beyond imitation through trial-and-error exploration guided by expert recovery demonstrations.

3.2 Big-picture architecture (diagram in words)

The system has five major interconnected components, arranged as a processing pipeline from visual input to robot actuation:

1. Dynamic Visual Observation Encoder — receives the continuous video stream from the robot's egocentric camera and applies an Ebbinghaus-forgetting-curve-inspired sampling policy that retains recent frames at high temporal resolution while aggressively downsampling older frames (both in frame rate and in spatial pooling stride). It embeds each sampled frame into visual tokens using a frozen SigLIP-400M vision encoder, projects them into the VLM's latent space through a two-layer MLP, and prepends temporal-aware indicator tokens (RoPE-encoded timestamps) so the model can perceive time intervals between frames.

2. Adaptive Chain-of-Thought (AdaCoT) Controller — a learned gating mechanism that, at each timestep, first predicts a binary CoT indicator token (<think_on> or <think_off>). If <think_on>, the VLM autoregressively generates a structured reasoning block (enclosed in thinking... response tags) containing perception analysis, task decomposition, visited-place assessment, and next-action planning, followed by an environmental summary (enclosed in <summary>...</summary> tags) that becomes persistent linguistic memory.

3. Visual-Assisted Linguistic Memory (VLingMem) Module — maintains a growing buffer of <summary> tokens produced by previous AdaCoT activations. These linguistic memories are concatenated into the VLM's input sequence at each step, providing explicit, semantically-rich recall of what rooms have been visited, what objects were seen, and what decisions were made — supplementing the implicit visual memory from encoded frames.

4. Probabilistic Continuous Action Head — a lightweight MLP that takes the hidden state of the VLM's final predicted token as input and parameterizes a multivariate Gaussian distribution over the next trajectory $\tau = \{a_1, a_2, ..., a_n\}$, where each $a \in \mathbb{R}^3$ is an $(x, y, \theta)$ waypoint. During inference, actions are drawn deterministically (the mean) for validation or stochastically (sampled from the Gaussian) for exploration during online RL training.

5. Expert-Guided Online RL Post-Training Loop — starts from the SFT checkpoint and alternates between two rollout modes: naive rollouts where the current policy interacts independently (only successful episodes kept) and expert-guided rollouts where a shortest-path planner takes control when the agent oscillates or gets stuck (triggered after $k=15$ steps of irrational behavior), producing corrective demonstrations. Both sources feed a hybrid buffer. The policy is updated with a composite loss: $\lambda$-weighted PPO-style policy gradient (using REINFORCE++ for advantage estimation) plus $(1-\lambda)$-weighted SFT imitation loss with $\lambda=0.01$.

Information flows as: video stream → dynamic FPS sampling → SigLIP encoding → grid pooling → MLP projection into VLM latent space → concatenation with instruction tokens, temporal tokens, and linguistic memory tokens → VLM autoregressive prediction (first CoT indicator token, then optionally CoT content) → hidden state of final token → action MLP → Gaussian mean and log-standard-deviation → sampled or deterministic trajectory → NMPC controller on robot.

3.3 Roadmap for the deep dive

I will explain the technical components in the order they are encountered during an inference step, which also corresponds to the architectural dependency chain:

• First, the Dynamic Observation Encoding pipeline (Section 3.3.1), because all downstream reasoning and memory depend on how visual information is sampled, compressed, and represented. This includes the forgetting-curve FPS policy, the time-dependent grid pooling, the temporal-aware indicator tokens with RoPE, and the MLP projector — the full chain from raw pixels to VLM-ready tokens.
• Second, the AdaCoT reasoning mechanism and VLingMem memory module together (Section 3.3.2), because they are tightly coupled: AdaCoT produces the linguistic summaries that VLingMem stores, and VLingMem provides the historical context that AdaCoT reasons about. This includes the autoregressive prediction of the CoT indicator, the structured format of CoT outputs, and how summaries accumulate across timesteps.
• Third, the Action Model (Section 3.3.3), which takes the VLM's final hidden state and produces trajectories. This includes the MLP architecture, the probabilistic Gaussian parameterization for RL, and the deterministic mean-output mode for inference.
• Fourth, the Autonomous Data Labeling Pipeline (Section 4.1.2), which produces the 2.9M-step Nav-AdaCoT-2.9M dataset. Without understanding how adaptive CoT annotations are generated, the training recipe makes limited sense.
• Fifth, the Three-Stage Training Recipe (Sections 5.1–5.3), covering pre-training on open-world adaptive CoT video data, SFT with combined navigation + video data and dual loss (MSE for actions, CE for text), and the expert-guided online RL stage with hybrid rollouts and composite PPO+SFT objective.
• Sixth, the online inference algorithm (Algorithm 1), which ties all components together in the deployment loop.

3.4 Detailed, sentence-based technical breakdown

This is primarily a systems paper that integrates multiple cognitively-inspired architectural innovations into a VLA navigation model. Its core idea is that embodied navigation demands two capabilities missing from prior work — adaptive deliberation (thinking hard only when needed) and explicit linguistic memory (remembering what you've seen in words) — and that these capabilities can be bootstrapped from a VLM backbone through carefully designed data, training, and architectural components.

---

Dynamic Visual Observation Encoding

The observation encoding pipeline transforms a growing stream of egocentric video frames into a fixed-budget set of VLM-compatible tokens, solving the fundamental tension between temporal context (you want many historical frames for spatial reasoning) and computational cost (each additional frame adds hundreds of visual tokens that the VLM must process autoregressively). The paper's key insight is that not all historical frames are equally informative: recent frames matter for precise obstacle avoidance and short-term dynamics, while older frames matter for coarse spatial orientation and visited-place memory. The encoding pipeline reflects this through three coordinated mechanisms.

Dynamic FPS sampling with exponential decay. The core idea is to sample historical frames at a rate that decays exponentially with their temporal distance from the current frame $t$, formalized through an Ebbinghaus-forgetting-curve analog:

$f_s(i) = f_s^{max} e^{-\frac{\Delta T}{s}}$

where $f_s(i)$ is the sampling rate (in frames per second) applied to frame $i$, $f_s^{max}$ is the maximum sampling rate (applied to the most recent frames, though the exact value is not specified in the paper), $\Delta T = t - i$ is the time interval from the current frame $t$ to historical frame $i$ (in seconds), and $s$ is a stability parameter that controls how quickly memory fades (larger $s$ means slower decay, retaining more distant frames).

What it computes: For each historical frame $i$ in the buffer, the decay function outputs a target sampling rate. Frames with $f_s(i)$ below a threshold (determined by the desired total frame budget) are dropped entirely. The surviving frames form a non-uniform temporal sampling: dense near the present, sparse in the distant past.

Why this form: The exponential decay captures the intuition from human memory research (Ebbinghaus, 1885/2013) that the rate of forgetting is proportional to the amount of information retained — recent experiences are remembered in detail while distant ones fade gradually rather than disappearing abruptly. A uniform subsampling (as used in NaVILA) would either lose critical short-term detail (at low sampling rates) or retain excessive long-term redundancy (at high sampling rates). A fixed-window approach would discard all context beyond the window boundary, creating a hard cliff where spatial memory disappears entirely. The exponential form provides a smooth, principled trade-off parameterized by a single interpretable scalar $s$.

Time-dependent grid pooling. After selecting which frames to keep, the paper applies a second level of compression: spatial downsampling of the visual feature maps, also keyed to temporal distance. For each retained frame at time $t_i$, a grid pooling stride $g(i)$ is computed:

$g(i) = \lfloor e^{-\frac{\Delta T}{g}} \rfloor$

where $g$ is a decay parameter (analogous to $s$, with the paper using a different symbol to indicate it controls spatial rather than temporal compression), and $\lfloor \cdot \rfloor$ floors to the nearest integer stride. The feature map $\mathbf{V}_{t_i} \in \mathbb{R}^{N \times C}$ (where $N=729$ patches from SigLIP with patch size yielding $27 \times 27$ grid for a $384 \times 384$ input, and $C=1152$ embedding dimension) is then spatially downsampled by the grid pooling operation $\mathcal{G}$:

$\mathbf{V}_{t_i}' = \mathcal{G}(\mathbf{V}_{t_i}, g(i))$

where $\mathcal{G}$ applies average pooling over $g(i) \times g(i)$ spatial blocks, reducing the number of visual tokens for frame $t_i$ by a factor of $g(i)^2$.

What it computes: Recent frames ($\Delta T \approx 0$) have $g(i) \approx 1$, retaining full $27 \times 27 = 729$ tokens. Older frames have $g(i) > 1$, producing coarser spatial representations with fewer tokens (e.g., $g(i) = 2$ yields $13 \times 13 \approx 169$ tokens; $g(i) = 3$ yields $9 \times 9 = 81$ tokens). The combination of temporal subsampling (dropping frames) and spatial subsampling (coarsening kept frames) controls the total token budget.

Why this form: This dual-decay strategy mirrors how biological memory works: distant memories are both less frequently accessed and less detailed when recalled. It is more principled than the token-merging approaches used in Uni-NaVid (which can distort semantic features by averaging unrelated visual patches) because it preserves the spatial structure of the feature map (adjacent patches remain adjacent in the pooled output). It is more flexible than uniform subsampling because recent observations — which may contain obstacles, dynamic objects, or navigation-relevant details — retain full spatial resolution.

Temporal-aware indicator tokens with RoPE. The dynamic FPS sampling creates an irregular temporal spacing between retained frames that could confuse the VLM's understanding of temporal order and velocity. To eliminate this ambiguity, the paper introduces explicit temporal encoding:

$\mathbf{E}^T(\Delta T) = \mathbf{E}^T_{base} + \text{RoPE}(\Delta T)$

where $\mathbf{E}^T_{base} \in \mathbb{R}^C$ is a learnable base embedding for temporal indicators (initialized randomly and trained), $\Delta T = t - t_i$ is the absolute time difference from the current frame to the historical frame in seconds, and $\text{RoPE}(\cdot)$ applies Rotary Position Embedding (Su et al., 2024) to encode the scalar $\Delta T$ as a rotation in the embedding space.

What it computes: RoPE encodes the scalar time difference by rotating pairs of dimensions in $\mathbf{E}^T_{base}$ by angles proportional to $\Delta T$. Specifically, for dimension pairs $(2k, 2k+1)$, the rotation angle is $\Delta T \cdot \theta_k$ where $\theta_k = 10000^{-2k/C}$ follows the standard RoPE frequency schedule. This means that frames separated by the same time interval will have the same pairwise angular difference in embedding space regardless of their absolute times, enabling the model to learn temporal-distance-sensitive operations through the VLM's self-attention mechanism.

Why this form: Without temporal indicators, the model would see a sequence of visual tokens with no explicit information about how much real time elapsed between them — it could not distinguish "these two frames are 0.2 seconds apart" from "these two frames are 10 seconds apart." This matters because the appropriate action depends on the temporal scale: if the robot hasn't moved far in 10 seconds, it may be stuck. RoPE encoding is preferred over learned absolute position embeddings because it naturally handles variable spacing (the rotation is a continuous function of $\Delta T$) and generalizes to time intervals not seen during training. The additive combination with a learnable base embedding allows the model to simultaneously learn what a "temporal indicator token" means (through the base) and how to compare temporal distances (through the rotation).

Visual feature projection. After temporal encoding, the spatially-pooled visual features $\mathbf{V}_{t_i}'$ are projected into the VLM's latent space through a two-layer MLP projector $\mathcal{P}$:

$\mathbf{E}^V_{t_i} = \mathcal{P}(\mathbf{V}_{t_i}')$

where $\mathbf{E}^V_{t_i}$ is the projected visual token for frame $t_i$, matching the dimensionality of the VLM's token embeddings. This follows the standard VLM projector design from LLaVA (Liu et al., 2023), which the paper uses without modification.

Full input sequence construction. For the current timestep $t$, the VLM's input is formed by concatenating, in order: instruction tokens $\mathbf{E}^I$ (from tokenizing the navigation instruction), then for each retained historical frame $t_i$ a tuple of $[\mathbf{E}^T(\Delta T_i), \mathbf{E}^V_{t_i}]$ (temporal indicator followed by visual tokens), then linguistic memory tokens $\mathbf{E}^M$ (from VLingMem, discussed below). The VLM processes this sequence autoregressively to predict the CoT indicator and optionally the CoT content.

---

Adaptive Chain-of-Thought (AdaCoT) and Visual-Assisted Linguistic Memory (VLingMem)

These two components are architecturally intertwined — AdaCoT generates the linguistic content that VLingMem stores, and VLingMem provides the historical context that AdaCoT reasons about — so I will explain them jointly, tracking the flow of information through both systems during a single inference step.

The CoT gating mechanism. At each timestep $t$, after the input sequence has been assembled and processed through the VLM's transformer layers, the model autoregressively predicts a CoT indicator token as its first output. This is a binary decision: either <think_on> or <think_off>. This prediction is generated by the standard autoregressive next-token mechanism — the VLM's output distribution over its vocabulary, conditioned on the full multimodal input, is sampled (or greedily decoded) to produce the indicator.

What happens when <think_off> is predicted: The model stops text generation immediately. The hidden state corresponding to this indicator token becomes the input to the action model (Section 3.3.3), and the robot executes the predicted trajectory. No new linguistic memory is created. This is the "fast thinking" path — the model has determined that the current situation is routine and does not require explicit deliberation.

What happens when <think_on> is predicted: The model continues autoregressive generation, producing structured reasoning content. The output follows a strict format with three tagged sections:

1. Reasoning content, enclosed in thinking ... response tags: This contains free-form natural language reasoning addressing four specific aspects that the training data was designed to induce (Section 4.1.2): perception of the current visual observation (what objects, structures, and people are visible), task decomposition and progress assessment (what sub-goal am I working on, have I found the target), assessment of whether the current location has been visited before (leveraging VLingMem's linguistic summaries), and determination of the next action and its justification.

2. Environmental summary, enclosed in <summary> ... </summary> tags: This is a concise linguistic description of the current scene, designed to be stored in VLingMem and injected into future inputs. It captures semantically meaningful information such as "a large kitchen with a central island, checked refrigerator and cabinets on north wall" or "entered corridor B, doors on left lead to offices 1–3, tracking target wearing red jacket moved toward end of corridor."

The generation of these two sections is a single autoregressive sequence: the model generates thinking, then the reasoning text, then ``, then <summary>, then the summary text, then </summary>. After generation completes, the summary text is extracted and appended to the VLingMem buffer $\mathcal{M}$.

VLingMem buffer management. The memory module maintains a persistent list of all <summary> segments generated during the current episode. At each subsequent timestep, all accumulated summaries are tokenized into $\mathbf{E}^M$ and prepended to the VLM input (before the temporal and visual tokens). There is no explicit forgetting or summarization of the memory buffer described in the paper — the buffer grows unboundedly over the episode, though in practice episode lengths are limited by the navigation benchmarks (typically well under 100 steps for successful episodes).

What AdaCoT+VLingMem jointly achieve: When the model encounters a situation requiring deliberation (triggering <think_on>), it has access to a complete linguistic record of what it has already observed and done. The reasoning block can explicitly reference this history: "I previously checked the kitchen and found no microwave — should explore the adjacent dining room." The summary produced becomes part of that record for future steps. When the model does not need deliberation (<think_off>), it still benefits from the linguistic memory as context for its implicit (non-verbalized) decision-making, because $\mathbf{E}^M$ tokens are always included in the input regardless of the CoT indicator.

Why this design over alternatives:

• Why a learned gating mechanism rather than heuristics? A heuristic trigger (e.g., "think every 20 steps") cannot adapt to episode-specific complexity. A straight corridor needs no thinking; a crowded room with ambiguous targets might need thinking at multiple consecutive steps. The learned gate can also be optimized during RL post-training — the model discovers through experience when deliberation pays off. The 2.1% activation rate in the final model (Table 6) is an emergent property of training, not a manually configured parameter.

• Why generate both reasoning and summary in one autoregressive block? This design forces the reasoning to be about the summary that will inform future decisions — the model cannot produce a summary that contradicts its own reasoning without detectable incoherence. It also ensures that the summary is grounded in the same visual observation and reasoning context, rather than being an afterthought.

• Why linguistic memory rather than visual feature cache? Linguistic summaries are more robust to information decay because they capture semantic abstractions (what rooms were visited, what objects were seen) rather than low-level visual features that degrade through repeated compression. They are also directly interpretable and debuggable, which matters for safety-critical deployment. However, the paper acknowledges that visual features still provide complementary information — the summary cannot capture everything (e.g., the exact spatial layout of a cluttered room), and the visual tokens fill this gap. Hence "visual-assisted" linguistic memory: language for semantic recall, vision for spatial detail.

• Why <summary> tags rather than free-form memory injection? The structured tagging allows clean extraction at test time — the memory buffer can be built without any additional parsing or filtering. It also provides a clear training signal: the model learns to distinguish reasoning (which is ephemeral and not stored) from memory content (which persists).

Training data for AdaCoT. The model learns when to think and what to think about from Nav-AdaCoT-2.9M, described in detail in the Data Collection section below. The critical property of this dataset is that only a fraction of steps (approximately 16% — 472K CoT annotations out of 2.9M total steps) contain CoT annotations. Steps without CoT annotations are trained with the target indicator token <think_off>, teaching the model that not every situation requires explicit reasoning. The model thus learns to predict <think_on> only when the input resembles situations where the training data contains CoT, and to predict <think_off> otherwise.

---

Action Model

The action model bridges the VLM's text-centric representations and the robot's continuous control space. Unlike prior VLA navigation models that discretize actions into vocabulary tokens (NaVid, Uni-NaVid) or use computationally expensive generative models like diffusion (TrackVLA, which uses an anchor-based diffusion policy), VLingNav uses a lightweight probabilistic MLP that directly outputs distributions over continuous waypoints.

Architecture. The action model $\mathcal{A}_\theta$ is a multi-layer perceptron (exact depth and width not specified) that takes as input the hidden state vector $\mathbf{h}_t^{pred} \in \mathbb{R}^D$ (where $D$ is the VLM's hidden dimension, 4096 for LLaMA-7B architectures) corresponding to the final token predicted by the VLM backbone at timestep $t$. This final token is either:

• The <think_off> token (if CoT was not triggered)
• The final token of the CoT generation (if <think_on> was triggered — this would be the token after </summary>)

The action model outputs two vectors: a mean trajectory $\boldsymbol{\mu}_t \in \mathbb{R}^{3n}$ and a log-standard-deviation $\log \boldsymbol{\sigma}_t \in \mathbb{R}^{3n}$, where $n$ is the trajectory horizon (the number of future waypoints to predict) and each waypoint $a \in \mathbb{R}^3$ represents $(x, y, \theta)$ in the robot's local coordinate frame — $x$ forward displacement, $y$ lateral displacement, $\theta$ heading change.

Probabilistic formulation (for RL training): The policy is parameterized as a multivariate Gaussian with diagonal covariance:

$\pi_\theta(\mathbf{a}_t | \mathbf{s}_t) = \mathcal{N}\left(\boldsymbol{\mu}_\theta(\mathbf{h}_t), \text{diag}(\boldsymbol{\sigma}_\theta(\mathbf{h}_t)^2)\right)$

where $\mathbf{a}_t = \tau_t = (a_{t,1}, a_{t,2}, ..., a_{t,n})$ is the full predicted trajectory (flattened into a $3n$-dimensional vector), $\boldsymbol{\mu}_\theta(\mathbf{h}_t)$ is the predicted mean trajectory, and $\text{diag}(\boldsymbol{\sigma}_\theta(\mathbf{h}_t)^2)$ is a diagonal covariance matrix with predicted variances $\sigma^2_{t,j}$ for each dimension $j$ of the trajectory.

What it computes: The action head projects the VLM's multimodal representation $\mathbf{h}_t$ (which encodes the instruction, visual observations, temporal structure, linguistic memory, and any CoT reasoning) into two quantities: where the robot should go (the mean) and how uncertain the model is about this prediction (the variance). During online RL exploration, actions are sampled $\mathbf{a}_t \sim \pi_\theta(\cdot | \mathbf{s}_t)$, introducing stochastic exploration driven by the learned uncertainty. During deterministic evaluation (validation, testing, and real-world deployment), the mean is used directly: $\mathbf{a}_t = \boldsymbol{\mu}_\theta(\mathbf{h}_t)$.

Why this form: The diagonal Gaussian is the simplest continuous distribution that supports both exploration (through variance) and deterministic execution (through the mean). A full-covariance Gaussian would capture correlations between waypoint dimensions (e.g., forward displacement and heading change are likely correlated when turning) but would require $O((3n)^2)$ parameters, which is expensive for long horizons. The diagonal parameterization is a standard efficiency-expressivity trade-off in continuous control. The log-standard-deviation parameterization ensures variance is always positive (since $\sigma = e^{\log \sigma} > 0$) without requiring constrained optimization.

Why continuous action prediction over discrete tokens: The paper argues in Section 2.1 that discrete action tokenization "leads to inefficient action quality and weak adaptability in dynamic scenarios." Concretely, discretizing a continuous $(x, y, \theta)$ space into a vocabulary of, say, 1000 tokens introduces quantization error — the robot can only move to a finite set of pre-defined waypoints. This is particularly problematic for tracking tasks where precise relative positioning matters. Continuous prediction eliminates this quantization bottleneck. Compared to diffusion-based approaches (TrackVLA), the MLP-based approach is dramatically faster at inference time because it requires a single forward pass rather than iterative denoising steps.

Why condition on the final hidden state rather than pooling? The final token's hidden state $\mathbf{h}_t^{pred}$ contains the VLM's full processed representation of the entire input sequence after autoregressive generation. If CoT reasoning was performed, this hidden state encodes the deliberation's conclusions; if not, it encodes the VLM's implicit assessment. Using a single vector is computationally efficient and aligns with standard practice in VLA models (the hidden state of the last predicted token is the natural "summary" of the VLM's processing).

Training objectives for the action model (SFT phase): During supervised fine-tuning, the action model is trained jointly with the VLM's text predictions using a composite loss:

$\min_\theta \mathcal{L}_{\text{SFT}}(\theta) = \alpha \mathcal{L}_{\text{MSE}}(\hat{\tau}_t, \tau_t^{gt}) + (1 - \alpha) \mathcal{L}_{\text{CE}}(E_t^{pred}, E_t^{gt})$

where $\alpha = 0.5$ is a balancing hyperparameter, $\mathcal{L}_{\text{MSE}}$ is the Mean Squared Error between the predicted trajectory $\hat{\tau}_t$ and the ground-truth trajectory $\tau_t^{gt}$ (from expert demonstrations or shortest-path planners), and $\mathcal{L}_{\text{CE}}$ is the standard Cross-Entropy loss supervising all text outputs (CoT indicator, reasoning content if present, summary content, and VQA responses for video data).

What it computes: The total loss is a weighted sum of two terms. The MSE term $\frac{1}{3n} \sum_{j=1}^{3n} (\hat{\tau}_{t,j} - \tau_{t,j}^{gt})^2$ penalizes the predicted mean trajectory for deviating from the expert's path. The CE term $\sum_{k} -\log p_\theta(e_k^{gt} | \text{context})$ penalizes the model for incorrect text predictions at each token position $k$. The $\alpha = 0.5$ weighting means both objectives contribute equally to the gradient, though the paper notes this was determined by loss scale rather than grid search.

Why this form: This is a multi-task learning objective where the VLM backbone is shared between text generation and action prediction, and gradients from both tasks flow into the backbone's parameters. This means the action model benefits from the VLM's text-based reasoning (better language understanding → better hidden states → better trajectories), and the VLM benefits from action supervision (the need to produce actionable hidden states regularizes the language representations to be spatially grounded). The equal weighting reflects an empirical observation that the MSE and CE losses have similar magnitudes in this setup, avoiding the need for careful loss scaling.

---

Autonomous Adaptive CoT Data Labeling Pipeline

The Nav-AdaCoT-2.9M dataset is constructed through an automated pipeline that uses a large VLM (Qwen2.5-VL-72B) to generate adaptive CoT annotations for existing navigation trajectory data. This pipeline is critical because manually annotating 2.9M steps with reasoning would be infeasible, and the quality of these annotations directly determines what the model learns about when and how to reason.

Input data sources: The pipeline processes trajectory data from six existing benchmarks that collectively cover ObjectNav, visual tracking, and ImageNav tasks (see Table 1 for exact scene counts and data volumes). For ObjectNav, data comes from HM3D ObjNav (human demonstrations from Habitat-Web), MP3D ObjNav (shortest-path trajectories), and HM3D OVON (shortest-path trajectories for open-vocabulary navigation). For tracking, data comes from EVT-Bench (multi-person indoor tracking). For ImageNav, data comes from HM3D Instance ImageNav (shortest-path trajectories with step-by-step action labels).

The labeling prompt design. For each step in each trajectory, the pipeline constructs a composite prompt to Qwen2.5-VL-72B containing five components:

1. Navigation instruction — the original task instruction (e.g., "Find the microwave" for ObjectNav, or the tracking target description for EVT).
2. Egocentric visual stream — the most recent 10 frames from the robot's egocentric camera, provided as images. The paper notes that this number is chosen "to reduce the computational load of the VLM" — feeding the full video would be prohibitively expensive for a 72B model operating over 2.9M steps.
3. Prior memory content — the linguistic summaries from previous steps (if any have been generated), enabling the VLM to produce CoT that is consistent with the agent's history.
4. Expert trajectory at the current step — the ground-truth action that the expert (human demonstrator or shortest-path planner) took from this state. This is crucial because the VLM needs to generate reasoning that justifies the expert's action, not reasoning that simply describes the scene.
5. Explicit formatting requirements — instructions to output reasoning in thinking ... tags and environmental summaries in <summary> ... </summary> tags.

VLM inference and output structure. For each step, Qwen2.5-VL-72B processes this prompt and generates structured output. The VLM is not asked to decide whether to reason — instead, the pipeline determines adaptivity through a separate mechanism: CoT annotations are generated only for a subset of steps (approximately 16% — 472K out of 2.9M), presumably those where the VLM's reasoning would be non-trivial or where the expert's action requires explanation. Steps without CoT annotations receive only the <think_off> target during training.

Two-stage filtering pipeline. The raw VLM outputs undergo automated quality control:

1. Rule-based checks: Incomplete responses (missing closing tags, truncated generations) and logically inconsistent outputs (reasoning that contradicts the expert action, summaries that are inconsistent with the visual input or prior memory) are discarded. The paper does not specify the exact rules used to detect inconsistency.

2. Quality verification: The generated reasoning is cross-validated against the expert navigation trajectory. Specifically, the paper states "Decisions were cross-validated against expert navigation trajectories to ensure accuracy," though the verification procedure (e.g., whether it uses automated metrics or a separate VLM judge) is not detailed.

What the pipeline produces: After filtering, the dataset contains 2.9M step-level training samples, each consisting of: visual observations (the dynamically sampled frames), the navigation instruction, any prior linguistic memory, the target CoT indicator token (<think_on> or <think_off>), the target reasoning content and summary (if <think_on>), and the ground-truth trajectory $\tau_t^{gt}$ for action supervision. Of these, 472K samples include CoT annotations (the <think_on> path).

Why this design over alternatives:

• Why use a separate large VLM rather than human annotators? Scale. Manually annotating reasoning for 2.9M navigation steps would be astronomically expensive and slow. The automated pipeline enables dataset creation at a scale that matches the needs of VLM fine-tuning.

• Why provide the expert action to the labeling VLM? If the VLM generates reasoning without knowing what action was taken, it may produce plausible-sounding but action-irrelevant reasoning (e.g., describing the scene without justifying the decision). Providing the expert action forces the reasoning to be causally connected to behavior — the VLM must explain why the expert did what they did, which teaches the downstream model to produce reasoning that is grounded in actionable decisions.

• Why use only the most recent 10 frames? This is purely a computational constraint — processing 200+ frames per step with a 72B VLM over millions of steps would be infeasible. The 10-frame window is sufficient for short-term spatial reasoning (obstacle avoidance, immediate navigation decisions) but misses longer-term context, which is partially compensated by including prior linguistic memory in the prompt.

• Why filter through cross-validation with expert trajectories? Without this step, VLMs can hallucinate reasoning that sounds plausible but is factually wrong (e.g., claiming the robot turned left when the expert trajectory shows it turned right). Cross-validation catches these hallucinations and ensures the training data teaches the model to produce reasoning that is accurate, not just fluent.

• Why the 16% CoT annotation rate? This was likely determined empirically. If too many steps have CoT, the model learns to reason constantly (dense CoT), which degrades performance (Table 6: 25.3% SR vs. 36.2% without CoT). If too few steps have CoT, the model never learns when reasoning is beneficial. The 16% rate produces an emergent 2.1% activation rate at test time (Table 6), suggesting the model learns that reasoning is needed in only a small fraction of situations — those that resemble the CoT-annotated steps in training.

---

Three-Stage Training Recipe

VLingNav's training proceeds in three sequential stages, each with distinct data, objectives, and purposes. The progression from pre-training → SFT → online RL reflects a deliberate curriculum: first teach the model general adaptive visual reasoning, then teach it navigation-specific skills through imitation, then let it refine those skills through trial-and-error exploration.

Stage 1: Model Pre-training (Adaptive CoT on Open-World Video)

Objective: Endow the LLaVA-Video-7B backbone with the foundational ability to perform adaptive visual reasoning — deciding whether a given visual input warrants explicit chain-of-thought deliberation.

Data: The custom open-world adaptive CoT video dataset described in Section 4.2, comprising 1.6M samples from three sources:

• LLaVA-Video-178K (Zhang et al., 2024b): general video understanding data
• Video-R1 (Feng et al., 2025): challenging video QA pairs, formatted as CoT-annotated subset
• ScanQA (Azuma et al., 2022): 3D scene understanding QA

The data is organized by difficulty: challenging subsets (Video-R1) receive CoT annotations, while easier subsets (LLaVA-Video-178K, ScanQA) are formatted as non-CoT samples, teaching the model that reasoning should be reserved for complex inputs.

Training details: The model is fine-tuned for a single epoch using standard cross-entropy loss at the token level. All videos are sampled at 1 FPS "to reduce redundancy between consecutive frames." Only the visual encoder's parameters are frozen; all other components (VLM backbone, projector, and the newly-initialized action head) are updated. The paper describes this stage as "consistent with standard VLM practices" and notes it runs on 128 NVIDIA A100 GPUs (though per-GPU batch size is not specified).

What this stage achieves: After pre-training, the model can process video inputs and decide whether to generate CoT reasoning based on input complexity, but it has no navigation-specific knowledge — it doesn't yet know what actions to take or how to navigate. This stage establishes the adaptive reasoning "scaffolding" that subsequent stages build upon.

Stage 2: Supervised Fine-Tuning (Navigation + Video Co-Training)

Objective: Establish robust navigation skills while retaining the general visual reasoning capabilities from Stage 1.

Data: A combined dataset mixing all embodied navigation data from Nav-AdaCoT-2.9M (2.9M samples) with all open-world video data from Stage 1 (1.6M samples), randomly shuffled. This co-training strategy ensures the model does not catastrophically forget its general visual understanding while acquiring navigation-specific skills.

Training details:

• Training duration: 20K steps with total batch size 512 (across 128 A100 GPUs, implying per-GPU batch size of 4)
• Loss function: the composite SFT loss from Equation 6 with $\alpha = 0.5$
• Optimizer: AdamW (the paper mentions this in context of standard VLM training but does not restate specific learning rates or weight decay for this stage)
• Frozen components: only the visual encoder (SigLIP-400M) is frozen; VLM backbone, projector, and action head are updated
• One epoch ≈ 10K training steps (from the ablation in Figure 11, which shows performance scaling positively with training steps up to 20K, with diminishing returns and eventual degradation beyond that)

The composite loss in detail: At each training step, a batch contains both navigation samples (with ground-truth trajectories for action supervision) and video QA samples (with only text targets). The total loss is:

$\min_\theta \mathcal{L}_{\text{SFT}}(\theta) = 0.5 \cdot \mathcal{L}_{\text{MSE}}(\hat{\tau}, \tau^{gt}) + 0.5 \cdot \mathcal{L}_{\text{CE}}(E^{pred}, E^{gt})$

For navigation samples, both terms are active: the model must predict correct trajectories (MSE) and correct text outputs — CoT indicator, reasoning content if <think_on>, and summary content (CE). For video QA samples, only the CE term is active (there are no trajectories to predict). The shared VLM backbone receives gradients from both terms, learning representations that simultaneously support spatial reasoning for navigation and general video understanding.

What this stage achieves: After SFT, the model can perform navigation tasks but is limited to the quality of its training demonstrations. It has never experienced the consequences of its own actions — it has only learned to imitate what experts did, not to recover from mistakes or discover novel strategies. This is where Stage 3 becomes critical.

Stage 3: Online Expert-Guided Reinforcement Learning Post-Training

Objective: Overcome the limitations of pure imitation learning — covariate shift, causal confusion, and the ceiling effect — by letting the agent interact with environments, collect its own experience, and improve through a combination of outcome-driven policy gradients and expert-guided supervision.

Environments for online interaction: The agent interacts with the training splits of three benchmarks: HM3D OVON (open-vocabulary ObjectNav), HM3D Instance ImageNav, and EVT-Bench DT (distracted tracking). These three environments provide diverse navigation challenges (open-vocabulary search, image-goal navigation, dynamic tracking with distractors).

Probabilistic action model for exploration: During online rollouts, the action model operates in stochastic mode. Given the VLM's hidden state $\mathbf{h}_t$, the head produces $\boldsymbol{\mu}_\theta(\mathbf{h}_t)$ and $\log \boldsymbol{\sigma}_\theta(\mathbf{h}_t)$. Actions are sampled from the Gaussian:

$\mathbf{a}_t \sim \pi_\theta(\cdot | \mathbf{s}_t) = \mathcal{N}(\boldsymbol{\mu}_\theta(\mathbf{h}_t), \text{diag}(\boldsymbol{\sigma}_\theta(\mathbf{h}_t)^2))$

This stochasticity is the engine of exploration — the model tries actions that deviate from its current best guess (the mean), and if those deviations lead to better outcomes, the policy gradient reinforces them. If they lead to worse outcomes, the policy is adjusted away from them. During validation and testing, the model switches to deterministic mode: $\mathbf{a}_t = \boldsymbol{\mu}_\theta(\mathbf{h}_t)$.

Hybrid rollout strategy: The central challenge in RL for long-horizon navigation is that sparse rewards (success/failure at the end of a potentially very long episode) make exploration extremely inefficient — the agent may wander randomly for hundreds of steps without receiving any learning signal. The paper addresses this through a hybrid data collection strategy that alternates between two modes:

Naive rollout: The current policy $\pi_\theta$ interacts with the environment completely independently, generating a full trajectory $\tau = \{(\mathbf{s}_t, \mathbf{a}_t, r_t)\}$ by sampling actions at each step. Only successful trajectories are retained and added to the hybrid buffer. This filtering is important: unsuccessful trajectories would provide negative examples (actions that led to failure), but with sparse binary rewards in long-horizon tasks, it is difficult to assign credit — which specific action caused the failure? By keeping only successful trajectories, the naive rollout provides positive examples of the policy's own discovered strategies.

Expert-guided rollout: When the agent exhibits irrational behavior — defined as oscillating or being stuck for $k=15$ consecutive steps, or when the episode eventually fails — an expert policy $\pi^*$ takes control. The expert is implemented as a Shortest Path planner in simulation, which has access to the ground-truth map and can compute optimal paths to the goal. The expert demonstrates a recovery path from the stuck/failed state to successful completion. This corrective trajectory is added to the hybrid buffer, providing high-quality examples of how to escape difficult situations.

Iteration schedule: The policy undergoes 10 rollout iterations of updates. For each iteration:

1. The current policy collects 128 episodes of on-policy data (split between naive and expert-guided rollouts; the paper does not specify the ratio)
2. These episodes are added to the hybrid buffer (which may retain data from previous iterations, though the paper does not specify buffer size or retention policy)
3. The model is updated using the composite post-training loss

Composite post-training loss: The optimization objective combines a PPO-style policy gradient (for outcome-driven learning from interaction data) with the SFT imitation loss (for stabilizing supervision from expert demonstrations):

$\min_\theta \mathcal{L}_{\text{post}}(\theta) = \lambda \mathcal{L}_{\text{RL}}(\theta) + (1 - \lambda) \mathcal{L}_{\text{SFT}}(\theta)$

where $\lambda = 0.01$ is a small weight on the RL term (determined by the scale of different losses), $\mathcal{L}_{\text{SFT}}$ is the same composite loss from Equation 6, and $\mathcal{L}_{\text{RL}}$ is the PPO-style clipped surrogate objective:

$\mathcal{L}_{\text{RL}}(\theta) = -\mathbb{E}_t \left[ \min\left(r_t(\theta) A_t, \text{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon) A_t\right) \right]$

where $r_t(\theta) = \frac{\pi_\theta(\mathbf{a}_t | \mathbf{s}_t)}{\pi_{\theta_{\text{old}}}(\mathbf{a}_t | \mathbf{s}_t)}$ is the probability ratio between the current and old policies (measuring how much the policy has changed), $A_t$ is the advantage estimate computed using REINFORCE++ (Hu, 2025), $\epsilon$ is the PPO clipping parameter (standard value 0.1–0.2, not specified in the paper), and the min and clip operations prevent destructively large policy updates.

What the composite loss computes: The RL term $\mathcal{L}_{\text{RL}}$ encourages the policy to increase the probability of actions that led to better-than-expected outcomes (positive advantage) and decrease the probability of actions that led to worse-than-expected outcomes (negative advantage), while the clipping prevents the policy from changing too much in a single update. The SFT term $\mathcal{L}_{\text{SFT}}$ pulls the policy back toward the expert demonstrations, acting as a regularizer that prevents catastrophic forgetting of the SFT-learned behaviors and provides a stable learning signal even when the RL advantage estimates are noisy.

Why $\lambda = 0.01$? This extreme weighting (RL is 100× smaller than SFT) reflects the paper's empirical finding that pure RL (naive rollout without expert guidance) fails to improve performance (Figure 11). The sparse rewards and long horizons make advantage estimates unreliable, so aggressive policy updates based on noisy RL signals would destroy the SFT policy. The small $\lambda$ means RL provides a gentle exploration bias — "try things slightly different from the expert, and if they work better, shift the policy slightly in that direction" — rather than attempting to learn navigation from scratch through trial and error.

REINFORCE++ for advantage estimation: The paper adopts REINFORCE++ (Hu, 2025) rather than standard GAE (Generalized Advantage Estimation) because REINFORCE++ is designed for the autoregressive token-generation setting where the "action" is a sequence of tokens rather than a single continuous vector. Specifically, REINFORCE++ computes advantages by treating the VLM's text generation (CoT indicator, reasoning, summary) and the action model's trajectory prediction as a joint sequence, and rewards are assigned at the episode level (success/failure). The advantage $A_t$ for a particular step is computed as the difference between the Monte Carlo return (discounted sum of future rewards from that step) and a learned value baseline.

Why this three-stage recipe over alternatives:

• Why pre-train on open-world video before navigation data? Without Stage 1, the model would need to learn both general visual reasoning and navigation simultaneously from Stage 2, which is a harder optimization problem. Pre-training on diverse video data establishes strong visual representations and reasoning priors that transfer to navigation, making the SFT stage more sample-efficient. The ablation in Table 8 confirms this: co-training with open-world video improves HM3D OVON success rate from 43.1% to 50.1%.

• Why expert-guided RL rather than pure RL? Pure RL with sparse rewards in long-horizon navigation is notoriously difficult — the credit assignment problem is severe because the final success/failure signal must be propagated back through potentially hundreds of steps. The Naive Rollout ablation in Figure 11 confirms this: pure RL fails to improve performance at all. Expert-guided RL provides a scaffold: the expert demonstrations show the model how to recover from mistakes (addressing credit assignment by providing step-by-step corrective actions), while the naive rollouts allow the model to discover strategies the expert didn't demonstrate (addressing the imitation ceiling).

• Why continuous action RL with PPO rather than discrete action RL with GRPO? GRPO (used in OctoNav, Nav-R1, VLN-R1) is designed for discrete action spaces and compares relative advantages within a group of sampled outputs. VLingNav's continuous action space requires a policy gradient method that can handle continuous distributions. PPO with a Gaussian policy is the standard choice for continuous control and is well-supported by existing infrastructure. The paper's contribution is not the RL algorithm itself but the demonstration that continuous-action RL can be successfully applied to VLA navigation when combined with expert guidance.

• Why only 10 RL iterations? Each iteration requires 128 episodes of online interaction, collecting on-policy data in simulation. More iterations would provide more data but at increasing computational cost and risk of the policy diverging from useful behaviors. The performance curves in Figure 11 show that improvements plateau after several iterations, suggesting 10 is near the point of diminishing returns.

---

Online Inference Algorithm

Algorithm 1 in the paper provides a formal specification of the complete inference loop, which I will walk through step by step to show how all components interact during deployment.

Initialization (lines 1–2): The algorithm receives two inputs: the video stream $\mathcal{O} = \{\mathbf{o}_1, \mathbf{o}_2, ..., \mathbf{o}_t\}$ (which grows continuously as the robot moves) and the navigation instruction $I$ (text). It initializes two empty structures: the linguistic memory buffer $\mathcal{M} \leftarrow \emptyset$ and the visual feature cache $\mathcal{V} \leftarrow \emptyset$.

Main loop (line 4): The inference runs in a continuous while true loop until a stop condition is triggered (the robot reaches the goal, exceeds a maximum step limit, or the model explicitly outputs a stop token).

Step 1: Encode instruction (line 5). The navigation instruction $I$ is tokenized once (it doesn't change across timesteps): $\mathbf{E}^I \leftarrow \text{Tokenizer}(I)$.

Step 2: Process current visual frame (lines 6–8). The latest egocentric frame $\mathbf{o}_t$ is encoded by the frozen SigLIP vision encoder into visual features $\mathbf{v}_t$. These features are appended to the visual cache $\mathcal{V}$. The cache now contains features for all historical frames (or a bounded window if the cache size is limited). The Sampling&Pooling function applies the dynamic FPS sampling (Equation 1) to select which cached frames to use, then applies time-dependent grid pooling (Equations 2–3) to spatially compress each selected frame's features, producing the final set of visual tokens $\mathbf{E}^V$ for this timestep.

Step 3: Create temporal indicator tokens (line 9). For each retained historical frame, a temporal-aware indicator token is computed using RoPE encoding of the time difference (Equation 4): $\mathbf{E}^T \leftarrow \text{RoPE}(\Delta t)$.

Step 4: Tokenize linguistic memory (line 10). The accumulated linguistic summaries from previous AdaCoT activations are tokenized: $\mathbf{E}^M \leftarrow \text{Tokenizer}(\mathcal{M})$. If no CoT has been triggered yet, $\mathcal{M}$ is empty and this produces no tokens.

Step 5: VLM forward pass to predict CoT indicator (line 11). The VLM backbone processes the concatenated input $[\mathbf{E}^I, \mathbf{E}^T, \mathbf{E}^V, \mathbf{E}^M]$ and autoregressively generates the first token: $\mathbf{E}^{\text{CoT}} \leftarrow \text{LLM.forward}(\dots)$. This token is either <think_on> or <think_off>.

Step 6: Conditional CoT generation (lines 12–15). If the predicted indicator is <think_on>, the model continues autoregressive generation to produce the full CoT content $c_t$ (reasoning in thinking..., summary in <summary>...</summary>). The summary is extracted and appended to the memory buffer: $\mathcal{M} \leftarrow \text{UpdateMemory}(\mathcal{M}, c_t)$. If the indicator is <think_off>, no further text is generated and the memory buffer remains unchanged.

Note that the paper's pseudocode shows UpdateMemory being called with the full $c_t$, but in practice only the content between <summary> tags is stored — this is implicit in the mention that "this summary is incorporated into subsequent inputs as linguistic memory" (Section 3.3.2).

Step 7: Extract hidden state for action prediction (line 16). The hidden state corresponding to the final predicted token (either <think_off> or the last token of the CoT generation) is extracted: $\mathbf{h}_t^{pred} \leftarrow \mathbf{E}^{\text{CoT}}[-1]$. This vector encodes the VLM's full processed understanding of the current situation.

Step 8: Predict trajectory (line 17). The action model takes this hidden state and produces a trajectory: $\hat{\tau}_t \leftarrow \mathcal{A}_\theta(\mathbf{h}_t^{pred})$. During deployment (validation, testing, real-world), this is the deterministic mean trajectory $\hat{\tau}_t = \boldsymbol{\mu}_\theta(\mathbf{h}_t^{pred})$.

Step 9: Execute or stop (lines 18–22). If the predicted trajectory is the special stop token (indicating the model believes the goal has been reached), the loop breaks. Otherwise, the trajectory is executed on the robot via whatever low-level controller is available. The loop then repeats with the next observed frame.

Key implementation details for real-world deployment (Section 6.3.1): Several optimizations make this loop practical on real hardware:

• Visual token caching: The visual features for historical frames are cached, so at each new step, only the latest frame's features need to be computed. The Sampling&Pooling step reuses cached features without re-encoding. This reduces per-step computation from scaling with trajectory length to being approximately constant.
• Image compression for network transmission: On the real robot (Unitree Go2), images are compressed before transmission over Wi-Fi to the remote server (NVIDIA RTX 4090). The paper reports that including communication overhead (~100 ms), the system achieves ~2.5 FPS inference during long-horizon experiments.
• Inference latency: The model maintains under 300 ms inference latency across 500 video frames, meaning the computational cost does not grow significantly with episode length despite the growing visual cache and memory buffer.
• NMPC trajectory tracking: The predicted trajectory $\hat{\tau}_t$ (sequence of waypoints) is tracked by a Nonlinear Model Predictive Control (NMPC) module (Grandia et al., 2023) that solves an optimization problem based on a kinematic unicycle model to compute optimal linear and angular velocities over a receding horizon.

Why this inference design over alternatives:

• Why cache visual features rather than re-encode the full video each step? Without caching, inference time would grow linearly with episode length — a 500-step episode would require encoding 500 frames at every step, which is $O(T^2)$ computation. Caching makes it $O(T)$: each step adds one new frame's encoding cost. This is essential for real-time deployment.

• Why generate the CoT indicator as the first token rather than as a separate classifier? Tying the gating decision to the autoregressive generation enables it to be optimized through the same cross-entropy loss as all other text outputs, and allows it to be conditioned on the full multimodal context through the VLM's attention mechanism. A separate binary classifier would require an additional training objective and might not capture the same nuanced, context-dependent gating behavior.

• Why deterministic execution during deployment but stochastic during RL training? Stochastic exploration is essential for RL — without it, the policy would never try actions different from its current predictions and could never discover improvements. But during deployment, stochastic actions would make the robot's behavior unpredictable and potentially unsafe. The mean trajectory represents the model's best estimate of where to go given its current knowledge.

4. Key Insights and Innovations

Innovation 1: Adaptive Reasoning as a Learned Gating Problem, Not a Fixed Schedule

The paper's most conceptually distinctive move is reframing chain-of-thought reasoning in embodied agents from a schedule design problem to a learned gating problem. Prior work that applied CoT to embodied tasks treated reasoning frequency as a hyperparameter to be manually configured: OctoNav executes CoT at a fixed interval determined by the human designer; NavA³ uses a monolithic reasoning-VLM that deliberates on every input, incurring prohibitive latency. The implicit assumption across all of these is that the human knows best when the model should think — if CoT reasoning helps, do it more often; if it's too slow, do it less often.

VLingNav's AdaCoT mechanism rejects this framing entirely. Instead of asking "at what fixed frequency should the model reason?", it asks "can the model learn from data to decide for itself when reasoning is needed?" The answer, as evidenced by the 2.1% reasoning activation rate in Table 6, is a decisive yes — and the model arrives at this rate through training, not through a human-chosen threshold. The gating decision is produced autoregressively as the first token of the VLM's output (<think_on> or <think_off>), meaning it is optimized through the same cross-entropy loss and conditioned on the same rich multimodal context as all other predictions. There is no separate classifier, no hand-tuned threshold on uncertainty, no heuristic trigger based on environment features — just a token prediction that the model learns to make correctly through exposure to a dataset where reasoning is annotated on only a subset of steps.

The intellectual significance of this reframing extends beyond the performance gains. It transforms CoT reasoning from an external augmentation bolted onto a policy into an intrinsic cognitive capability that the model can deploy or withhold based on its own assessment of situational demands. This aligns with the dual-process theory (fast vs. slow thinking) that the paper invokes as inspiration, but the connection is more than metaphorical: the model actually learns to instantiate the fast/slow distinction through the gating mechanism, with <think_off> corresponding to System 1 (fast, intuitive, low-compute) and <think_on> corresponding to System 2 (slow, deliberative, high-compute). The fact that this behavior emerges from data rather than being programmed is what makes it cognitively interesting rather than merely an engineering trick.

The ablation in Table 6 provides striking evidence for why adaptivity matters in a way that fixed schedules cannot match. Dense CoT (reasoning at every step) degrades ObjNav success rate from 36.2% to 25.3% — reasoning too much actively hurts performance. Fixed-interval reasoning at k=5 (20% activation) improves to 42.5%, and at k=20 (5% activation) to 39.7%. But adaptive CoT reaches 50.1% with only 2.1% activation — dramatically outperforming all fixed schedules while reasoning on far fewer steps. This pattern reveals that the value of reasoning is highly non-uniform across steps: some critical decision points benefit enormously from deliberation, while routine navigation actually suffers from it. A fixed schedule inevitably either reasons too little at critical junctures or too much during routine traversal. Only adaptivity can place reasoning precisely where it matters.

This is a fundamental conceptual shift, not an incremental refinement. Prior CoT-for-embodiment work treated reasoning as a capability to be applied uniformly; VLingNav treats it as a resource to be allocated sparingly and strategically. The fact that the model learns to activate reasoning on only 2.1% of steps — far below what any human designer would likely configure — suggests that the model's learned allocation strategy is genuinely non-obvious and that manual schedule design is fundamentally limited.

Innovation 2: Language as the Primary Memory Modality in VLA Navigation Systems

The paper introduces a modality-level design principle for VLA memory: language, not compressed visual features or latent vectors, should serve as the primary medium for storing and recalling semantic history. This is not an obvious choice — the dominant paradigm in video-based VLA navigation (NaVid, Uni-NaVid, NaVILA, StreamVLN, TrackVLA) is to encode historical visual frames and propagate them as implicit visual memory, while work on general VLA memory (RoboFlamingo, MemoryVLA) uses compressed latent tokens. The assumption underlying both approaches is that the model's internal representations, whether visual or latent, are the appropriate substrate for memory because they preserve information in a form the model can directly process.

VLingNav's VLingMem module challenges this assumption with a specific argument: language is better aligned with the VLM's pretraining than any latent representation could be. The VLM backbone (LLaVA-Video-7B) was trained on massive corpora of text and image-text pairs; its most robust, generalizable processing capabilities operate over linguistic tokens, not over ad-hoc latent vectors that must be learned from scratch during navigation fine-tuning. When visual features are repeatedly compressed and passed through transformer layers, their semantic content degrades — the model progressively loses track of what it saw even if it retains some where information. Linguistic summaries, by contrast, are discrete, stable, and directly interpretable by the same text-processing machinery that the VLM uses for instruction understanding and reasoning.

This is fundamentally a representation choice argument, not a mechanism-level innovation. The mechanism itself — storing <summary> tokens in a buffer and injecting them into the input sequence — is straightforward once you've decided that memory should be linguistic. The intellectual contribution is the argument for that decision, backed by an ablation (Table 7) that systematically compares memory modalities. The "Language-only" condition (18.8% SR) substantially underperforms "Visual-only" (45.2%), but "VLingMem" — which combines both (50.1%) — significantly outperforms either alone. This pattern reveals that language and vision contribute complementary information: language provides semantic, categorical memory (what rooms were visited, what objects were seen, what decisions were made) that resists degradation, while vision provides spatial, geometric detail (exactly where obstacles are, the precise layout of the current room) that language cannot fully capture. The term "visual-assisted linguistic memory" is carefully chosen: language is the primary modality, vision assists.

This reframing has downstream implications that the paper begins to explore but doesn't fully develop. Linguistic memory is interpretable and auditable — a human operator can inspect the <summary> buffer and understand what the robot believes it has experienced, which matters enormously for safety-critical deployment and debugging. Linguistic memory is also composable with reasoning — when the model triggers <think_on>, it can explicitly reference its own memory summaries ("I previously checked the kitchen and found no microwave"), creating a tight coupling between memory and deliberation that latent-vector memories cannot support because their content is opaque and inaccessible to linguistic reasoning processes.

The significance of this innovation is that it proposes a design principle for future VLA memory systems — ground memory in the modality the model processes best — rather than just introducing a new memory mechanism. It is a conceptual advance with practical consequences, not a raw performance gain, and it applies to any VLA system built on a language-model backbone, which is essentially all current VLA architectures.

Innovation 3: Verifier Over-Optimization as a First-Class Phenomenon in Test-Time Scaling

This section number and title appear to be a copy-paste artifact from the reference example and do not apply to this paper. I will skip to the actual innovations in VLingNav.

Innovation 3: Expert-Guided RL as a Scaffold for Continuous-Action Policy Refinement

The paper's online post-training stage addresses a capability gap in the VLA training landscape: existing RL approaches for VLA navigation are confined to discrete action spaces (OctoNav, Nav-R1, VLN-R1 all use GRPO with autoregressive action tokens), while continuous-action VLA systems lack any RL-based policy refinement and are limited to pure imitation learning. VLingNav bridges this gap with a specific architectural choice — a probabilistic Gaussian action head that supports both stochastic exploration (for RL) and deterministic execution (for deployment) without iterative denoising — and a specific training strategy: expert-guided hybrid rollouts with a heavily SFT-weighted composite loss (λ = 0.01).

The intellectual contribution here is not the RL algorithm itself (PPO with REINFORCE++ advantage estimation is standard) but the demonstration that naive RL fails catastrophically in long-horizon navigation and that expert guidance is not merely helpful but essential for making continuous-action RL work in this setting. Figure 11 shows that Naive Rollout (pure RL without expert demonstrations) fails to improve over the SFT baseline at all, while Expert Rollout (DAgger-like) provides substantial gains, and Hybrid Rollout achieves the best performance. This is a diagnostic finding: the credit assignment problem in sparse-reward, long-horizon navigation is severe enough that standard policy gradient methods cannot learn from scratch, and expert demonstrations serve as a necessary scaffold that provides step-level corrective signals where outcome-level rewards are too sparse to be informative.

The heavily asymmetric weighting (λ = 0.01, meaning the RL gradient is 100× smaller than the SFT gradient) is itself an interesting finding. In typical applications of PPO + demonstration data, the RL and imitation terms are more balanced (often λ ≈ 0.1–0.5). The extreme SFT dominance here suggests that the RL signal is so noisy in this domain that it can only provide a very gentle exploration bias — "try things slightly different from the expert, and if they work better, shift the policy slightly in that direction" — rather than substantially reshaping the policy. The fact that even this gentle bias produces significant improvements (Table 2: 79.1% vs. 70.6% SR on HM3Dv1; Table 3: 59.3% vs. 45.9% SR on HM3D OVON val seen) suggests that the SFT policy is close to optimal in its broad structure but benefits from fine-grained adjustments that pure imitation cannot provide.

This is a pragmatic engineering insight with broader implications: as VLA systems move toward continuous action spaces for finer-grained control, the RL training challenge shifts from discrete-token optimization (where GRPO-style group-relative comparisons work well) to continuous-distribution optimization (where sparse rewards become much harder to propagate). Expert-guided hybrid rollouts with heavily SFT-weighted composite objectives represent a practical solution to this challenge that is likely to generalize to other continuous-action VLA domains beyond navigation.

Innovation 4: Multi-Task Navigation Training as a Source of Emergent Cross-Task Capabilities

A finding that emerges from the experimental results rather than being presented as a core architectural contribution is that joint training on ObjectNav, Embodied Visual Tracking, and ImageNav produces capabilities that no single-task model possesses and that transfer compositionally across tasks. This is documented in Section 6.4, particularly in the real-world experiments where the model performs behaviors it was never explicitly trained for: tracking targets specified by image goals (despite tracking training data containing only language instructions), searching for a language-described target and then switching to tracking it, and searching for an image-specified target and then tracking it.

The intellectual significance of this finding is that it challenges the common assumption in VLA research that navigation tasks are distinct skills requiring task-specific architectures or fine-tuning. The fact that a single set of model weights achieves state-of-the-art or competitive performance across ObjectNav, EVT, and ImageNav (Tables 2–5) while also exhibiting zero-shot composition of skills across these tasks suggests that the shared VLM backbone, trained on diverse navigation data, is learning generalizable navigation priors rather than task-specific heuristics. The multi-task synergy ablation in Figure 12 confirms this quantitatively: models trained on a single task consistently underperform the multi-task model even on their own specialized benchmarks.

This is more than a "multi-task learning works" finding. The specific compositional behaviors — tracking from image goals, switching between search and tracking — demonstrate that the model has learned to decouple the goal specification modality (language description vs. image) from the navigation behavior (search vs. track). This decoupling is what enables cross-task transfer: once the model understands that "find and follow the person in the red jacket" and "find and go to the location in this image" both involve recognizing a target and moving toward it, the specific modality of the target specification becomes incidental rather than determinative of the behavior. This is a form of conceptual abstraction that goes beyond what SFT on task-specific data typically produces, and it likely arises from the shared VLM backbone's pretraining on diverse vision-language data interacting with the multi-task navigation fine-tuning.

The practical implication is significant: future VLA navigation systems may not need task-specific architectures, training pipelines, or even explicit task-identification mechanisms. A single model trained on diverse navigation data with a unified architecture can internally route different goal specifications to shared navigation capabilities, producing more robust and flexible behavior than purpose-built task-specific systems. The result also provides empirical support for the paper's broader thesis that linguistic-driven cognition (AdaCoT + VLingMem) provides a sufficiently general substrate that task-specific specialization becomes unnecessary — the cognitive architecture itself, rather than task-specific engineering, enables the observed generality.

5. Experimental Analysis

Evaluation Methodology

• Dataset. VLingNav is evaluated on multiple public embodied navigation benchmarks spanning three task categories. For Object Goal Navigation: HM3Dv1 ObjNav (Ramakrishnan et al., 2021), HM3Dv2 ObjNav, MP3D ObjNav (Chang et al., 2017), and HM3D OVON (Yokoyama et al., 2024b) — the latter featuring three splits (val seen, val seen synonyms, val unseen) for open-vocabulary evaluation. For Embodied Visual Tracking: EVT-Bench (Wang et al., 2025b) with two splits (Single Target Tracking and Distracted Tracking). For Image Goal Navigation: HM3D Instance ImageNav (Krantz et al., 2022). A single shared model checkpoint is used across all benchmarks with no task-specific fine-tuning. All simulation experiments use the standard test splits provided by these benchmarks; the paper does not report exact test set sizes for each benchmark but these are well-established community standards (HM3D ObjNav: 2,000 episodes; MP3D ObjNav: 2,000+ episodes; HM3D OVON: three splits each with several hundred episodes; EVT-Bench: not specified; Instance ImageNav: not specified, but the original benchmark defines standard test splits).

• Base model(s). VLingNav extends LLaVA-Video-7B (Zhang et al., 2024b), a video-based Vision-Language Model with 7 billion parameters. The vision encoder is a frozen SigLIP-400M (Zhai et al., 2023) that produces 729 visual tokens per frame (27×27 grid, embedding dimension 1152). The paper argues this model is representative of current VLM capabilities and sits in a useful regime where pre-trained visual reasoning abilities can be leveraged for navigation while remaining computationally tractable for real-time robotic deployment (Section 3.2, Section 6.3.1 reports <300ms inference latency on an RTX 4090 across 500-frame episodes). For data labeling, Qwen2.5-VL-72B (Bai et al., 2025) is used as the annotation VLM.

• Metrics. The paper uses standard evaluation metrics from each benchmark's established protocol. For ObjectNav and ImageNav: Success Rate (SR) — the fraction of episodes where the agent stops within a threshold distance of the goal (exact thresholds follow each benchmark's standard) — and Success-weighted Path Length (SPL) — SR multiplied by the ratio of optimal path length to actual path length, penalizing inefficient trajectories. For Embodied Visual Tracking: SR, Tracking Rate (TR) — the fraction of timesteps where the target is within the camera's field of view and correctly identified — and Collision Rate (CR) — the fraction of timesteps where the agent collides with obstacles or other agents. For the open-vocabulary HM3D OVON benchmark, SR and SPL are reported separately for val seen (trained categories), val seen synonyms (synonyms of trained categories), and val unseen (completely novel categories). All metrics are computed by the respective benchmark evaluation scripts, ensuring comparability with prior published results.

• Baselines. The paper compares against three categories of prior methods. Modular approaches that separate perception, mapping, and planning: VLFM (Yokoyama et al., 2024a), SG-Nav (Yin et al., 2024), L3MVN (Yu et al., 2023), UniGoal (Yin et al., 2025), Habitat-Web (Ramrakhya et al., 2022), InstructNav (Long et al., 2024), ApexNav (Zhang et al., 2025f), OVRL (Yadav et al., 2023b), OVRL-v2 (Yadav et al., 2023a), LFG (Shah et al., 2023), CogNav (Cao et al., 2025), FiLM-Nav (Yokoyama and Ha, 2025), GOAT (Chang et al., 2023), Mod-IIN (Krantz et al., 2023), and MTU3D (Zhu et al., 2025). End-to-end small-scale models that use pre-trained visual feature extractors with learned policy networks: PirlNav (Ramrakhya et al., 2023), Habitat-Web (mentioned above), PoliFormer (Zeng et al.), EVT (Zhong et al., 2024), IBVS (Gupta et al., 2016), PSL (Sun et al., 2024), OVRL-v2-IIN (Yadav et al., 2023a), among others. VLA models that use VLM backbones for navigation: Uni-NaVid (Zhang et al., 2025b), TrackVLA (Wang et al., 2025b), TrackVLA++ (Liu et al., 2025a), NavFoM (Zhang et al., 2025a), Nav-R1 (Liu et al., 2025b), JanusVLN (Zeng et al., 2025), and TANGO (Ziliotto et al., 2025). For real-world experiments, the paper directly compares against Uni-NaVid (for ObjectNav and EVT) and UniGoal (for ImageNav). Baseline numbers are taken directly from published results except where the paper explicitly notes re-evaluation.

• Generation budget / compute accounting. The paper does not use a "generation budget" concept analogous to LLM test-time compute scaling work. Instead, all methods are evaluated under each benchmark's standard protocol: agents operate with a fixed maximum number of steps per episode (benchmark-specific, typically 500–1000 steps) and the primary efficiency metric is SPL (path efficiency) rather than inference FLOPs. For fair comparison against VLA baselines, the paper uses the same visual input (monocular RGB at 1280×800 resolution) and the same action space (continuous trajectories of $(x, y, \theta)$ waypoints) as the most comparable prior work (TrackVLA, TrackVLA++, Uni-NaVid). The online RL post-training uses 128 episodes per iteration over 10 iterations, totaling 1,280 environment interactions for policy improvement — this cost is borne during training only. The paper reports inference latency (Section 6.3.1: <300ms across 500-frame episodes on an RTX 4090, achieving ~2.5 FPS including ~100ms communication overhead) as the relevant compute metric for deployment, but does not systematically compare FLOPs or wall-clock time against baselines during benchmark evaluation.

• Cross-validation / statistical protocol. For simulation benchmarks, results are reported on the standard test splits with the standard evaluation protocol — the paper does not describe custom cross-validation or multiple random seeds for the main results, instead following each benchmark's established single-evaluation-pass convention. For real-world experiments (Section 6.3), each target object or scenario is evaluated with 10 repeated trials to mitigate randomness, and success rates are reported as averages over these trials. For the ablation studies (Section 6.5), the paper adheres to the same training procedures and evaluation settings as the full model to ensure comparability, though specific statistical significance testing (confidence intervals, standard errors) is not reported. For the compute-optimal CoT strategy selection, the paper uses the full training dataset (Nav-AdaCoT-2.9M) across all training stages, with the adaptive CoT behavior emerging from training rather than being selected via held-out validation. The RL post-training uses a fixed 10-iteration schedule rather than early stopping on validation performance.

Main Quantitative Results

Object Goal Navigation (Tables 2 and 3)

Closed-vocabulary benchmarks (Table 2). VLingNav achieves state-of-the-art performance across all three closed-vocabulary ObjectNav benchmarks. On HM3Dv1, VLingNav reaches 79.1 SR and 42.9 SPL, improving over the previous best video-based VLA model Uni-NaVid (73.7/37.1) by +5.4 SR (+7.3%) and +5.8 SPL (+15.6%) . On HM3Dv2, VLingNav attains 83.0 SR and 40.5 SPL, surpassing FiLM-Nav (77.0/41.3) by +6.0 SR (+7.8%) while noting that FiLM-Nav's slightly higher SPL (41.3 vs. 40.5) stems from its architectural advantage — FiLM-Nav selects frontier waypoints and relies on a shortest-path planner, whereas VLingNav directly outputs trajectory-based actions, which is more challenging for path efficiency metrics in simulation. On MP3D — the benchmark with the longest-horizon exploration scenarios — VLingNav achieves 58.9 SR and 26.5 SPL, dramatically outperforming the prior best methods CogNav (46.6/16.1) and ApexNav (39.2/17.8). This represents a +26.4% relative improvement in SR and a +32.8% relative improvement in SPL over CogNav, which the paper attributes to VLingNav's robust exploration and memory capabilities being particularly impactful in complex, multi-room environments where looping and redundant exploration (the failure modes that VLingMem addresses) are most prevalent.

The SFT-only checkpoint (row "VLingNav (SFT)" in Table 2) provides an intermediate comparison: on HM3Dv1 it achieves 70.6/38.2, on HM3Dv2 76.4/32.6, and on MP3D 47.4/25.8. The gap between SFT and final VLingNav (e.g., +8.5 SR on HM3Dv1, +11.5 SR on MP3D) quantifies the contribution of the online RL post-training stage. Notably, the SFT-only model already approaches Uni-NaVid's performance on HM3Dv1 (70.6 vs. 73.7 SR) and exceeds it on MP3D (47.4 vs. not reported for Uni-NaVid), suggesting that even without RL, the AdaCoT+VLingMem architecture provides substantial benefits for long-horizon tasks.

Open-vocabulary benchmark (Table 3). On HM3D OVON, which tests generalization to novel object categories, VLingNav achieves best performance across all three splits. On val seen: 59.3 SR / 29.7 SPL, improving over Nav-R1 (58.4/26.3) by +0.9 SR (+1.5%). On val seen synonyms: 56.8 SR / 30.1 SPL, improving over Nav-R1 (48.1/23.1) by +8.7 SR (+18.1%) . On val unseen: 50.1 SR / 24.6 SPL, improving over NavFoM (43.6/31.3) in SR by +6.6 SR (+15.1%) while noting that NavFoM achieves higher SPL (31.3 vs. 24.6) — the paper acknowledges this SPL gap but does not analyze its cause, though it may relate to NavFoM's multi-view setting providing better spatial awareness for path optimization. The SFT-only checkpoint (45.9/26.5 on val seen, 44.8/27.1 on synonyms, 41.5/22.4 on unseen) again shows substantial gains from RL post-training, particularly on the val seen split (+13.4 SR).

The pattern across OVON splits reveals an interesting finding about generalization: VLingNav's advantage over prior methods is smallest on val seen (+1.5% over Nav-R1) and largest on val seen synonyms (+18.1%). This suggests that VLingNav's linguistic memory and adaptive reasoning are particularly beneficial for semantic generalization — recognizing that "settee" means the same thing as "couch" and navigating accordingly — rather than providing uniform improvements across all difficulty levels. The robust performance on val unseen (50.1 SR, where categories are entirely novel) further supports the claim that multi-task training with open-world video co-training produces genuinely generalizable navigation capabilities rather than overfitting to training categories.

Embodied Visual Tracking (Table 4)

On EVT-Bench, VLingNav demonstrates state-of-the-art performance with a particularly strong showing in the more challenging Distracted Tracking scenario. On Single Target Tracking: 88.4 SR / 81.2 TR / 2.07 CR, matching or slightly exceeding the previous best methods NavFoM (88.4/80.7 with multi-view setting, noted as NavFoM*) and TrackVLA++ (86.0/81.0/2.10). On Distracted Tracking: 67.6 SR / 73.5 TR / 5.51 CR, representing a +1.1 SR (+1.7%) and +4.7 TR (+6.8%) improvement over TrackVLA++ (66.5/68.8/4.71).

Two aspects of these results warrant attention. First, VLingNav matches NavFoM's multi-view performance while using only a monocular camera, which is a substantially more challenging setting — NavFoM benefits from a wider field of view and depth information from multiple cameras, while VLingNav must track targets and avoid obstacles from a single egocentric viewpoint. The paper highlights this as evidence that "robust tracking and precise recognition" can be achieved without multi-view inputs when combined with linguistic memory and adaptive reasoning. Second, VLingNav's collision rate is slightly higher than TrackVLA++ on both splits (2.07 vs. 2.10 on Single Target; 5.51 vs. 4.71 on Distracted), suggesting that the current system's trajectory prediction may sacrifice some obstacle avoidance precision in favor of tracking persistence — a trade-off that the paper's Section 8 acknowledges as a limitation to be addressed through future dual-system architectures with high-frequency obstacle avoidance.

The Distracted Tracking split, which introduces multiple similar-looking distractors, is the more diagnostic test of VLingNav's cognitive capabilities. Here, the ability to maintain linguistic memory of the target's appearance ("person in red jacket, medium height") and to trigger deliberate reasoning when the target is occluded or confused with a distractor should provide the greatest advantage. The +6.8% TR improvement supports this interpretation, though the paper does not provide a breakdown of when AdaCoT triggers occur during tracking episodes, which would strengthen the causal attribution.

Image Goal Navigation (Table 5)

On HM3D Instance ImageNav, VLingNav achieves 60.8 SR / 37.4 SPL, narrowly exceeding UniGoal (60.2/23.7) in SR by +0.6 (+1.0%) but dramatically outperforming it in SPL by +13.7 (+57.8%) . This SPL improvement is the largest relative gain observed across all benchmarks and requires careful interpretation.

UniGoal leverages the LightGlue (Lindenberger et al., 2023) keypoint-matching algorithm as an additional explicit criterion for determining when the goal image matches the current view — this provides a strong, hand-designed signal for goal confirmation that VLingNav does not use, instead relying solely on the VLM's implicit visual reasoning. The fact that VLingNav achieves comparable SR despite lacking this explicit matching mechanism suggests that its adaptive reasoning can learn to recognize goal similarity from data. The dramatically higher SPL (37.4 vs. 23.7) indicates that VLingNav takes substantially more direct paths to the goal — it is not just finding the target at similar rates, but finding it much more efficiently. This efficiency gain likely stems from VLingMem: the agent remembers which areas it has already explored (avoiding redundant search) and can reason about the spatial layout (inferring that an office chair is likely in an office rather than a kitchen, and planning its exploration accordingly).

The SFT-only checkpoint (51.1/32.6) is particularly revealing on this benchmark: even without RL post-training, VLingNav achieves an SPL that already exceeds UniGoal by 37.5% (32.6 vs. 23.7), though SR lags behind (51.1 vs. 60.2). This suggests that the memory and reasoning architecture provides most of the path-efficiency benefit, while the RL stage contributes primarily to goal-finding accuracy — the agent learns through interaction to better recognize when it has reached the target location.

Visualization Results (Figure 6)

Figure 6 provides qualitative examples of VLingNav's behavior across simulation benchmarks, showing the robot's egocentric visual observations, a top-down scene map, input instructions, and the outputs of adaptive CoT alongside predicted trajectories. These visualizations illustrate the interpretability benefit of AdaCoT: when the model triggers <think_on>, the generated reasoning is displayed, making the agent's decision-making process transparent. The paper does not provide quantitative analysis of these visualizations (e.g., what fraction of CoT activations occur at decision points like intersections vs. during straight-line navigation), but they serve as qualitative validation that the reasoning content is sensible and task-relevant.

Real-World Experiments (Figures 7, 8, 9)

The real-world experiments on a Unitree Go2 quadruped robot (Figure 7) test zero-shot transfer of the simulation-trained VLingNav checkpoint to physical environments across three task categories.

Object Goal Navigation (Section 6.3.2, Figure 8 left). Evaluated across three environments (home, office, outdoor) with three target objects each and 10 repeated trials per target. VLingNav achieves "significantly higher success rate than Uni-NaVid across all tested scenarios." The paper presents success rates as bar charts in Figure 8 but does not report exact numerical values, making precise comparison difficult. Qualitatively, the improvements are most pronounced in the outdoor environment, where lighting variation, dynamic objects, and less structured layouts present greater sim-to-real challenges.

Embodied Visual Tracking (Section 6.3.3, Figure 8 middle). Evaluated across three scenarios: single-target tracking in open spaces, single-target tracking in cluttered indoor environments, and distracted tracking in crowded scenes with frequent occlusions. VLingNav "consistently outperforms Uni-NaVid in tracking success rate, with the largest margins appearing in the distracted setting where transient occlusions and target switches are common." Again, exact numerical values are not reported. The paper attributes this advantage to adaptive reasoning enabling re-identification after occlusion (the model reasons about target appearance when <think_on> is triggered) and precise trajectory control from continuous action prediction (as opposed to Uni-NaVid's discrete action tokens).

Image Goal Navigation (Section 6.3.4, Figure 8 right). Evaluated across three environment categories (home, office, outdoor) with two image-specified goals each and 10 repeated trials per goal. VLingNav achieves "substantially higher success rate than UniGoal in all categories." The paper suggests that multi-task training induces robust cross-modal grounding and that the combination of Adaptive CoT and linguistic memory supports reliable localization to visually specified targets despite variations in camera intrinsics, viewpoints, and lighting.

A critical caveat: The real-world results, while impressive as a demonstration of zero-shot sim-to-real transfer, are reported without numerical precision. The bar charts in Figure 8 show clear advantages for VLingNav over baselines, but the lack of exact success rates, confidence intervals, or statistical tests makes it impossible to assess the reliability or significance of these differences. Additionally, the 10 trials per condition provide limited statistical power, especially for binary success/failure outcomes — a difference of 2 successes out of 10 trials could appear large in a bar chart but would not reach statistical significance under standard tests. The qualitative examples in Figure 9, showing successful navigation in office, household, and outdoor settings, are selected success cases; the paper does not show failure cases or analyze failure modes in real-world deployment beyond the general statements in Section 8 about limitations.

Emergent Cross-Task and Cross-Domain Capabilities (Section 6.4)

Section 6.4 documents behaviors that VLingNav exhibits despite never being explicitly trained for them, which the paper frames as evidence of emergent generalization.

Cross-task performance (Section 6.4.1). VLingNav can track targets specified by image goals in zero-shot, despite tracking training data containing only language-format instructions. Moreover, it composes behaviors: searching for a language-described target then switching to tracking it, or searching for an image-specified target and subsequently tracking it after locating it. These compositional capabilities arise from the unified architecture and multi-task co-training, but the paper does not provide quantitative evaluation of these behaviors — they are described qualitatively based on real-world observations in Figure 9.

Cross-domain performance (Section 6.4.2). VLingNav reliably tracks dynamic non-human targets despite being trained only to track humans. It also successfully localizes and navigates to out-of-distribution objectives specified by fine-grained textual instructions, including category-ambiguous objects disambiguated by color, spatial constraints, or detailed descriptions. Again, these are qualitative observations without quantitative metrics, making it difficult to assess how robust or frequent these behaviors are in practice.

While these emergent capabilities are scientifically interesting and consistent with the paper's thesis that linguistic-driven cognition enables generalization, the absence of systematic evaluation (controlled experiments with held-out task compositions, quantitative success rates for cross-task behaviors, comparison against baselines on these emergent tasks) weakens the strength of the claims. The observed behaviors could represent genuine emergent generalization or could be selection effects from qualitative cherry-picking — without systematic analysis, the reader cannot distinguish between these possibilities.

Ablation Studies and Robustness Checks

All ablations in Section 6.5 are conducted on three evaluation settings: HM3D OVON val unseen for ObjectNav, EVT-Bench Distracted Tracking for EVT, and HM3D Instance ImageNav val for ImageNav. The paper states these were chosen for consistency with the full model's evaluation settings, though the choice of the most challenging splits (val unseen rather than val seen for OVON, Distracted rather than Single Target for tracking) provides a stringent test of each component's contribution.

Adaptive CoT strategy (Table 6). This ablation compares five reasoning strategies against the same underlying architecture, varying only when CoT is triggered:

• w/o CoT (never reason): 36.2 SR / 16.5 SPL on ObjNav, 62.7 SR / 68.5 TR on tracking, 56.3 SR / 27.3 SPL on ImageNav. This establishes the baseline without any explicit reasoning.
• Dense CoT (reason at every step, 100% activation): 25.3 SR / 13.0 SPL on ObjNav, 59.8 SR / 70.1 TR on tracking, 19.6 SR / 13.2 SPL on ImageNav. This is the most striking result in the ablation: reasoning at every step significantly degrades performance compared to never reasoning, and the degradation is most severe on ImageNav (19.6 vs. 56.3 SR — a 65% reduction) and ObjNav (25.3 vs. 36.2 — a 30% reduction). The collision rate on tracking jumps from 6.28 to 26.3, suggesting that constant deliberation interferes with the reactive, low-latency decisions needed for dynamic obstacle avoidance.
• Fixed Interval (k=5) (reason every 5 steps, 20% activation): 42.5 SR / 23.5 SPL on ObjNav, 68.5 SR / 74.2 TR on tracking, 48.2 SR / 28.7 SPL on ImageNav. Substantial improvement over both no-CoT and dense CoT, but still well below adaptive.
• Fixed Interval (k=20) (reason every 20 steps, 5% activation): 39.7 SR / 19.4 SPL on ObjNav, 66.2 SR / 70.8 TR on tracking, 51.3 SR / 31.2 SPL on ImageNav. Interesting inversion on ImageNav: k=20 outperforms k=5 (51.3 vs. 48.2 SR), suggesting that ImageNav benefits from sparser reasoning, possibly because image-goal matching requires less frequent deliberation than object search.
• Adaptive CoT (learned gating, 2.1% activation): 50.1 SR / 24.6 SPL on ObjNav, 67.6 SR / 73.5 TR on tracking, 60.8 SR / 37.4 SPL on ImageNav. Best performance on all benchmarks while using the least reasoning. The 2.1% activation rate is an emergent property — the model was trained on data where ~16% of steps had CoT annotations, but at test time it activates reasoning even more sparingly.

The key insight: reasoning is not uniformly beneficial. Dense CoT actively harms performance, most dramatically on ImageNav where the task requires visual matching that may be disrupted by verbal deliberation (the "verbal overshadowing" effect known from cognitive psychology, though the paper does not invoke this term). Fixed-interval reasoning helps but cannot place reasoning precisely at the critical decision points — the adaptive strategy achieves 50.1 SR on ObjNav with 2.1% activation while k=5 (20% activation) achieves only 42.5, meaning the adaptive strategy is 10× more selective with reasoning while being 18% more effective. This strongly supports the paper's thesis that "deliberate thought at a small fraction of key steps is sufficient to substantially boost overall task success" (Section 7).

Visual-assisted linguistic memory (Table 7). This ablation decomposes VLingMem into its component modalities:

• w/o Memory: 15.4 SR / 3.5 SPL on ObjNav, 37.5 SR / 59.1 TR / 1.90 CR on tracking, 21.0 SR / 3.7 SPL on ImageNav. The catastrophic drop on ObjNav (15.4 vs. 50.1 SR) and ImageNav (21.0 vs. 60.8 SR) confirms that memory is essential for search-based navigation in large environments. The paper notes that without memory, "agents frequently get stuck in loops or revisit dead ends" — the 3.5 and 3.7 SPL values (near-zero path efficiency) are consistent with looping behavior.
• Visual-only (implicit visual memory through cached frame features): 45.2 SR / 20.3 SPL on ObjNav, 66.8 SR / 70.6 TR on tracking, 57.9 SR / 33.7 SPL on ImageNav. This partially recovers performance, confirming that implicit visual memory (the approach used by prior VLA models like Uni-NaVid and TrackVLA) provides substantial benefit.
• Language-only (only linguistic summaries, no visual history): 18.8 SR / 4.4 SPL on ObjNav, 40.2 SR / 55.2 TR on tracking, 23.3 SR / 7.5 SPL on ImageNav. Surprisingly poor performance — language-only memory alone barely improves over no memory at all, and is dramatically worse than visual-only memory. This is a non-obvious finding: the paper argues that linguistic memory is the primary modality, but this ablation shows that language alone fails without accompanying visual features. The interpretation: linguistic summaries provide categorical, semantic memory (what rooms, what objects) but lack the spatial precision needed for moment-to-moment navigation decisions. The visual features provide this spatial grounding, and the combination (VLingMem: 50.1 SR) significantly outperforms either alone, confirming the "visual-assisted" design — language for semantic recall, vision for spatial grounding.

The tracking results show a different pattern: w/o Memory (37.5 SR) vs. Language-only (40.2 SR) is a small gap, and Visual-only (66.8 SR) already recovers most of the full model's performance (67.6 SR). This suggests that for tracking, where the primary challenge is maintaining line-of-sight to a dynamic target rather than remembering spatial layouts, visual memory is the dominant component — language summaries of target appearance help re-identification after occlusion but are less critical than continuous visual tracking.

Open-world video co-training (Table 8). This ablation removes the open-world video data (1.6M samples from LLaVA-Video-178K, Video-R1, ScanQA) from training:

• w/o Co-training (navigation data only): 43.1 SR / 20.6 SPL on ObjNav, 66.5 SR / 70.2 TR on tracking, 50.2 SR / 32.7 SPL on ImageNav
• w/ Co-training (full VLingNav): 50.1 SR / 24.6 SPL on ObjNav, 67.6 SR / 73.5 TR on tracking, 60.8 SR / 37.4 SPL on ImageNav

The co-training provides substantial gains on ObjNav (+7.0 SR, +16.2%) and ImageNav (+10.6 SR, +21.1%), with a smaller effect on tracking (+1.1 SR). The paper attributes this to "enriching the model's semantic priors, thereby enhancing cross-modal grounding and generalization capabilities while notably reducing the sim-to-real gap." The asymmetry — large gains on tasks requiring visual scene understanding and spatial reasoning, small gains on tracking — is consistent with open-world video data primarily improving general visual perception rather than task-specific tracking skills.

SFT training steps (Figure 11, which appears to be mislabeled as Figure 10 in the paper's text — the figures are numbered 10, 11, 12 in the paper but the text references them as 11, 11, 12). The paper investigates how performance scales with SFT training duration: "model performance scales positively with the number of training steps (where 1 epoch ≈ 10K training steps). The success rate rises steadily as the model is exposed to more data. Notably, we found that excessive training leads to diminishing returns and eventual performance degradation, likely due to overfitting on the simulation data." The optimal appears to be around 20K steps (2 epochs), with degradation beyond this point. The paper does not report exact numbers at each checkpoint, showing only curves in the figure.

Online post-training iteration steps (Figure 11, right). This ablation compares three rollout strategies for the RL stage:

• Naive Rollout (pure RL, no expert guidance): fails to improve over SFT baseline. The paper attributes this to "sparse reward signals and the long-horizon nature of the task making value estimation too difficult" — the policy gradient signal is too noisy to drive meaningful improvement.
• Expert Rollout (DAgger-like, only expert-guided demonstrations): provides substantial improvement over SFT, demonstrating that corrective demonstrations alone can refine the policy.
• Hybrid Rollout (the paper's proposed strategy, combining naive and expert rollouts): achieves the best performance, outperforming expert-only rollouts. The paper argues this shows that on-policy data enables the model to "explore and discover better strategies" beyond what the expert demonstrates, while expert data "corrects faulty behaviors" and stabilizes learning.

This ablation reveals that the naive rollout component, while unable to drive improvement on its own, provides complementary benefits when combined with expert guidance — a finding consistent with the demonstration-augmented RL literature (Rajeswaran et al., 2017) but newly demonstrated for continuous-action VLA navigation.

Multi-task synergy (Figure 12). Models trained on a single navigation task (ObjectNav only, EVT only, or ImageNav only) consistently underperform the multi-task model on all benchmarks, including their own specialized tasks. The paper reports this as evidence that "multi-task learning facilitates the transfer of skills across different domains" and "enhances the model's ability to reason and plan across diverse modalities, tasks, and target categories." The figure shows bar charts comparing single-task vs. multi-task performance, but exact numbers are not quoted in the text. The mechanism for this synergy is not investigated in depth — the paper suggests it arises from shared navigation priors (exploration strategies, obstacle avoidance, spatial reasoning) that transfer across tasks, but does not provide the kind of task-similarity analysis or gradient conflict measurement that would strengthen this claim.

Critical Assessment

Does VLingNav genuinely demonstrate adaptive reasoning, or just infrequent reasoning?

The headline result — that AdaCoT activates on only 2.1% of steps while achieving state-of-the-art performance — is compelling, but the evidence that this represents genuine adaptivity to situational demands (rather than simply learning to reason on a sparse, fixed subset of step types) is incomplete. The paper does not analyze when the model triggers reasoning: does <think_on> consistently occur at intersections, decision points, and occluded-target scenarios, or is the 2.1% activation distributed essentially randomly across step types? Without this analysis, an alternative explanation is possible: the model has learned that reasoning on approximately 1 in 50 steps is optimal for benchmark performance regardless of context, and the specific steps where reasoning occurs are determined by noise in the gating prediction rather than genuine situational assessment. The qualitative visualizations in Figure 6 show CoT activations at plausible decision points, but these are selected examples.

A stronger test would be to intervene on the environment (e.g., make a previously simple corridor become complex by adding obstacles, or make a previously complex intersection become simple by removing choice) and measure whether the gating probability adapts accordingly. The paper's use of the dual-process theory metaphor is appealing but the experimental evidence for genuine, context-sensitive adaptivity (as opposed to sparse-but-random activation) is suggestive rather than conclusive.

How much does linguistic memory contribute versus the visual features that accompany it?

The ablation in Table 7 produces a paradoxical result for the paper's central thesis: language-only memory (18.8% SR on ObjNav) barely outperforms no memory at all (15.4%), while visual-only memory (45.2%) recovers most of the full model's performance (50.1%). This suggests that visual features are doing the heavy lifting, and linguistic memory provides a modest incremental benefit on top of a strong visual baseline. The paper's framing — "visual-assisted linguistic memory" — positions language as the primary modality with vision as auxiliary, but the ablation tells a different story: vision is the workhorse, and language provides a small but significant increment.

This does not invalidate the VLingMem contribution — the 4.9 SR point gain from adding language to visual-only (45.2 → 50.1) is meaningful — but it recalibrates the narrative. The real finding may be that linguistic memory serves a specific, narrow function (preventing catastrophic looping and revisitation in large environments) while visual features handle the continuous stream of spatial navigation decisions. The fact that w/o Memory drops to 15.4 SR but visual-only recovers to 45.2 suggests that the primary memory failure mode is spatial disorientation (forgetting where you've been in geometric space), which visual features can partially address, while the additional 4.9 SR from language comes from semantic disambiguation (remembering that this specific room was already searched for the target object). This is a more nuanced and defensible interpretation than "language is the primary memory modality," but the paper does not explicitly make this distinction.

The SFT vs. RL contribution is confounded by benchmark-specific effects.

Tables 2–5 show that RL post-training provides substantial gains over the SFT checkpoint across all benchmarks, but the magnitude varies dramatically: +8.5 SR on HM3Dv1, +1.0 SR on ImageNav (from 51.1 to 60.8, but see below), +13.4 SR on HM3D OVON val seen. These variations are not discussed or explained. The ImageNav result is particularly noteworthy: the SFT-only checkpoint achieves 51.1 SR, which is already quite strong, and RL adds +9.7 SR to reach 60.8. But the SFT-only ImageNav SR is reported as 51.1 in Table 5, while the ablation in Table 6 reports the full model's ImageNav SR as 60.8 — these are consistent. What's missing is an analysis of whether the RL gains come from better exploration (finding the goal more reliably), better stopping (recognizing the goal when reached), or better path efficiency (taking shorter routes, which would manifest in SPL gains). The SPL improves from 32.6 to 37.4, so path efficiency improves, but the decomposition is not analyzed.

Real-world results lack quantitative rigor.

The real-world experiments (Section 6.3) are the paper's most compelling demonstration of practical utility — zero-shot transfer to a physical quadruped robot is genuinely impressive. However, the reporting falls short of scientific standards:

• No numerical results reported: Figure 8 shows bar charts but the text never states exact success rates. The reader cannot determine whether VLingNav achieved 80% vs. 60% success, or whether the "significantly higher" performance is 3/10 vs. 2/10 trials.
• Small sample sizes: 10 trials per condition with binary outcomes provides very limited statistical power. The paper reports no confidence intervals or significance tests.
• No failure analysis: Figure 9 shows only successful examples. Understanding when and why the zero-shot transfer fails is arguably more informative for future work than knowing that it sometimes succeeds.
• No comparison to simulation performance: The paper does not report whether the 2.1% CoT activation rate observed in simulation holds in the real world, or whether the gating behavior changes when faced with real-world visual complexity.

These limitations do not negate the achievement — zero-shot sim-to-real transfer for VLA navigation is non-trivial and the qualitative demonstrations are persuasive — but they prevent the real-world results from serving as rigorous evidence for the paper's specific claims about adaptive reasoning and linguistic memory. A skeptic could argue that the real-world success is primarily attributable to the LLaVA-Video backbone's strong visual representations and the continuous action space, with AdaCoT and VLingMem providing minor incremental benefits that happen to accumulate to an advantage over a relatively weak baseline (Uni-NaVid, which uses discrete actions and lacks memory).

Missing experiments that would strengthen the paper.

Several experiments would substantially strengthen the paper's claims but were not conducted:

1. CoT timing analysis. When does AdaCoT trigger? A distribution of <think_on> activations over environment features (intersections, open corridors, near obstacles, after occlusion, near goal) would directly test the adaptivity claim. This is purely an analysis of existing model behavior and requires no new training.

2. Memory visualization. What do the <summary> tokens contain, and do they accumulate meaningful information over long trajectories? The paper shows qualitative examples in figures but provides no quantitative analysis of memory content: average summary length, semantic diversity, factual accuracy (do summaries correctly describe visited rooms?), or degradation over time.

3. Scaling with environment size. The MP3D benchmark shows the largest gains, which the paper attributes to memory benefits in large environments. A controlled experiment varying environment size (small vs. medium vs. large layouts) while holding other factors constant would directly test whether the VLingMem benefit scales with spatial complexity as claimed.

4. RL contribution decomposition. The RL stage combines exploration (naive rollouts) with expert correction (expert rollouts) and a composite PPO+SFT loss. Ablating each component separately (PPO-only, SFT-only, naive-only, expert-only, different λ values) is partially done in Figure 11, but the paper does not ablate the SFT component of the composite loss — does pure PPO on the hybrid buffer (without the SFT loss term) also improve, or is the SFT regularization essential?

5. Comparison to VLA models with discrete actions on the same benchmarks. VLingNav uses continuous actions while Uni-NaVid and TrackVLA use discrete or anchor-based actions. The SR improvements could partially reflect the action space advantage rather than the cognitive architecture. A controlled comparison where VLingNav's action head is replaced with discrete action tokens (keeping AdaCoT+VLingMem) would isolate the contribution of continuous actions to the overall performance gains.

6. Sensitivity to CoT annotation quality. The autonomous labeling pipeline uses Qwen2.5-VL-72B and a two-stage filtering process. How sensitive is VLingNav's performance to the quality of these annotations? Would a smaller labeling VLM, or fewer CoT annotations, or unfiltered raw annotations produce similar results? This matters for the reproducibility and scalability of the approach.

Which claims are well-supported, and which are provisional?

Well-supported claims:

• VLingNav achieves state-of-the-art performance on standard embodied navigation benchmarks. The simulation results in Tables 2–5 are comprehensive, use established benchmarks, and compare against relevant baselines. The gains are substantial on most benchmarks.

• Adaptive CoT outperforms fixed-interval and dense CoT while using far less reasoning. Table 6 provides clean, controlled evidence for this claim across three benchmarks. The 2.1% activation rate achieving better performance than 20% fixed-interval reasoning is a striking and well-documented result.

• VLingMem combining visual and linguistic memory outperforms either modality alone. Table 7 supports this. The language-only result (18.8% vs. 15.4% for no memory) is weaker than one might expect given the paper's narrative, but the combined benefit is clear.

• Online RL post-training improves over SFT. Tables 2–5 consistently show RL > SFT across all benchmarks and metrics (with minor exceptions like SPL on some benchmarks).

• Multi-task training produces better performance than single-task training. Figure 12 supports this, though the absence of exact numbers in the text limits precise quantification.

Provisional claims requiring stronger evidence:

• "Adaptive reasoning dynamically responds to situational demands." The evidence (2.1% activation, qualitative examples) is consistent with adaptivity but does not rule out alternative explanations (learned sparse-but-random gating). A timing analysis is needed.

• "Linguistic memory is the primary memory modality." The language-only ablation in Table 7 (18.8% vs. 45.2% for visual-only) directly contradicts this framing — visual memory provides the bulk of the benefit. The claim should be refined to acknowledge that language provides a specific, complementary function (semantic episodic memory) rather than being the primary modality.

• "Zero-shot transfer demonstrates strong cross-domain and cross-task generalization." The real-world results demonstrate zero-shot transfer, which is genuinely impressive, but the lack of numerical rigor and systematic cross-task evaluation (beyond qualitative examples) makes the strength of this generalization difficult to assess. The emergent cross-task behaviors (tracking from image goals, search-then-track composition) are shown only in selected examples.

• "VLingNav bridges the sim-to-real gap through cognitive architecture." The paper demonstrates sim-to-real transfer, but does not provide evidence that the cognitive architecture (rather than the VLM backbone, the continuous action space, or the open-world video co-training) is responsible for bridging the gap. An ablation removing AdaCoT and VLingMem from the real-world deployment would test this, but such an experiment is not reported.

Overall, the experimental analysis is thorough for simulation benchmarks and provides strong support for the paper's core technical contributions (adaptive CoT outperforms fixed schedules, VLingMem improves over visual-only and language-only baselines, RL improves over SFT). The real-world results, while impressive as a demonstration, lack the quantitative rigor needed to serve as strong evidence for the paper's broader claims about generalization and cognitive architecture. The missing analyses — particularly the CoT timing distribution and the memory content analysis — would substantially strengthen the paper's central narrative without requiring additional experiments.

6. Limitations and Trade-offs

The Monocular Egocentric Input Constrains Spatial Awareness and Obstacle Handling

The assumption or constraint. VLingNav relies on a single forward-facing RGB camera (Intel RealSense D457 at 1280×800 resolution, 90° HFOV) as its sole perceptual input. The paper acknowledges this explicitly in Section 8:

"the current model primarily relies on monocular egocentric observations as input. Due to the limited field of view (FOV) inherent in monocular vision, such input constrains the model's perceptual capabilities."

This is not merely a deployment detail — it is a fundamental architectural constraint: the model never sees what is behind, beside, or in some cases even immediately adjacent to the robot unless it turns to look. The entire AdaCoT reasoning and VLingMem memory architecture operates over this narrow visual slice.

The consequence. Three concrete failure modes arise directly from monocular input:

1. Obstacle avoidance blind spots. The robot cannot perceive obstacles outside its forward FOV. The Distracted Tracking collision rates (Table 4: VLingNav achieves 5.51 CR vs. TrackVLA++ at 4.71 CR, and 2.07 vs. 2.10 on Single Target) suggest VLingNav experiences slightly more collisions than the best prior method. In real-world deployment, collisions with unseen obstacles (furniture legs, low objects, people approaching from the side) could cause physical damage, safety incidents, or simply terminate the mission — failures that better perceptual coverage could prevent.

2. Inefficient search due to limited situational awareness. In ObjectNav, the agent must physically rotate to survey rooms, consuming steps that a multi-view system could process simultaneously. The SPL gap on certain benchmarks — notably ImageNav where VLingNav achieves 60.8 SR but UniGoal with explicit keypoint matching and modular vision achieves 60.2 SR (Table 5) — may partially reflect this exploration tax: the agent spends steps turning to look at things that a multi-camera system would see continuously. The paper notes that NavFoM, which uses multi-view input, achieves higher SPL on HM3D OVON val unseen (31.3 vs. 24.6, Table 3), consistent with the interpretation that wider perceptual coverage improves path efficiency.

3. Tracking vulnerability to occlusions and target departure from FOV. In Embodied Visual Tracking, the target can exit the monocular FOV through lateral movement. The model must then reason about where the target went and execute a search — a fragile recovery compared to a multi-view system that might maintain visual contact throughout. The paper's qualitative claim that adaptive reasoning helps re-identification after occlusion (Section 6.3.3) implicitly acknowledges this fragility: the reasoning is needed because the monocular system loses the target in the first place.

What evidence exists in the paper. The collision rate data in Table 4 provides the most direct quantitative evidence, but the paper does not systematically compare VLingNav's monocular performance against multi-view baselines on metrics that would isolate the perceptual bottleneck (e.g., average time to find target after losing visual contact, or SR specifically on episodes where the target exits the FOV). The Section 8 discussion of this limitation is purely qualitative.

Mitigation status. The paper explicitly identifies this as a limitation and proposes future work: "Following recent work, we will explore integrating multi-view observations to improve navigation efficiency" (Section 8). No experiments with multi-view inputs are reported, and the paper does not estimate how much of the remaining performance gap to NavFoM (multi-view) is attributable to perceptual coverage versus other architectural differences. This is a recognized but unaddressed limitation.

---

No Mechanism for High-Frequency Reactive Control or Dynamic Obstacle Avoidance

The assumption or constraint. VLingNav operates as a single inference pipeline at a fixed control frequency (~2.5 FPS in real-world deployment, Section 6.3.1): the VLM processes multimodal input, optionally generates CoT, and predicts a trajectory. There is no separate fast-reacting subsystem for obstacle avoidance, no reactive safety layer, and no mechanism to abort or modify the predicted trajectory mid-execution if the environment changes. The paper acknowledges this architectural limitation in Section 8:

"the current model adopts a single-system architecture, which restricts its prediction frequency. This limitation impedes rapid decision-making and obstacle handling in highly dynamic environments."

The consequence. The 2.5 FPS control rate means the robot commits to a trajectory (sequence of $(x, y, \theta)$ waypoints predicted at time $t$) and executes it via NMPC without visual feedback until the next inference cycle completes. This creates a reaction-time vulnerability: a person stepping into the robot's path, a door closing, or an object falling into the trajectory between inference cycles will not be detected until the next cycle. At 2.5 FPS, the minimum reaction time is ~400ms plus NMPC execution latency — far slower than the sub-100ms reactions expected of safe mobile robots in human environments.

The problem compounds when AdaCoT activates <think_on>. The paper reports <300ms inference latency across 500-frame episodes (Section 6.3.1), but this is an average — when the model generates reasoning text autoregressively (potentially hundreds of tokens of CoT content), the inference latency for that step will be substantially higher. The paper does not report the latency distribution conditional on the CoT indicator, but an additional 1–2 seconds of text generation (reasonable for a 7B model generating detailed reasoning) would create a dangerous control gap in dynamic environments — the robot would be essentially blind and uncontrolled while "thinking."

The collision rate data (Table 4) is consistent with this vulnerability: VLingNav's collision rate on Distracted Tracking (5.51) is higher than TrackVLA++ (4.71), despite VLingNav's superior tracking metrics. In a crowded hallway with moving pedestrians, a 5.5% collision rate at 2.5 FPS would translate to a collision every ~7 seconds of operation — clearly unacceptable for real-world deployment.

What evidence exists in the paper. The paper reports aggregate inference latency (~300ms at 2.5 FPS) in Section 6.3.1 and collision rates in Table 4, but does not report latency conditional on CoT activation, worst-case latency, or the relationship between inference frequency and collision rate. There is no ablation comparing collision rates at different control frequencies, and no analysis of whether collisions occur disproportionately during or immediately after CoT-activated steps. The real-world experiments (Section 6.3) are conducted in relatively static, controlled environments — the paper does not test the system in truly dynamic scenarios with fast-moving obstacles.

Mitigation status. The paper identifies this as a limitation and proposes a dual-system architecture for future work: "we plan to upgrade VLingNav to a dual-system structure that supports high-frequency action outputs, thereby enhancing fundamental navigation performance, such as obstacle avoidance" (Section 8). This would presumably involve a fast reactive subsystem (e.g., a lightweight collision-avoidance policy running at 30+ Hz) that operates in parallel with the slower VLM-based reasoning system. However, no such architecture is implemented or evaluated, and the paper provides no evidence that the current system's performance would be preserved in a dual-system design (the interaction between fast reactive control and slow deliberative planning often introduces new failure modes, such as the reactive system overriding deliberate navigation decisions). This is a recognized limitation that the paper correctly identifies as critical for real-world deployment but does not address experimentally.

---

The Difficulty Estimation and Data Labeling Pipeline Incur Substantial Unaccounted Costs

The assumption or constraint. The Nav-AdaCoT-2.9M dataset — which enables VLingNav's adaptive reasoning — is constructed using Qwen2.5-VL-72B, a 72-billion-parameter VLM, running inference over 2.9 million navigation steps. The paper describes this as an "autonomous adaptive CoT labeling pipeline" (Section 4.1.2) and uses it to generate 472K CoT annotations. The computational cost of this labeling — running a 72B VLM on 2.9M multimodal inference calls, each processing 10 video frames plus instruction text and prior memory — is never quantified or accounted for in the paper's cost analysis.

The consequence. This omission matters for two reasons. First, it affects reproducibility: a research group seeking to reproduce or extend VLingNav must either (a) replicate the Qwen2.5-VL-72B labeling pipeline, which requires substantial computational resources (rough estimate: 2.9M inferences × ~1 second each on high-end hardware = ~800 GPU-hours, plus storage and filtering overhead), or (b) release the pre-labeled dataset, which the paper does not mention doing. Second, it affects scalability claims: the paper presents VLingNav as a general framework (Section 7, point 4: "generality and real-world generalization"), but the data labeling cost scales linearly with the number of environments and tasks. Extending VLingNav to new domains would require running the costly labeling pipeline on new navigation data, which may not be feasible for resource-constrained research groups or for domains where suitable labeling VLMs (with Qwen2.5-VL-72B-level capabilities) are not available.

The paper also does not compare the cost of the labeling pipeline against alternative approaches. Could a smaller VLM (e.g., 7B parameters) produce usable CoT annotations? Could rule-based heuristics combined with template-based reasoning achieve comparable downstream performance at a fraction of the cost? Without such comparisons, the reader cannot assess whether the 72B labeling VLM is necessary or merely convenient.

What evidence exists in the paper. The paper describes the labeling pipeline in detail (Section 4.1.2, Figure 4) and provides dataset statistics (Table 1: 2.9M steps, 472K CoT annotations), but nowhere reports the computational cost of annotation. The Qwen2.5-VL-72B model is identified by name, confirming the use of a 72B VLM, but inference time, GPU-hours, or cost estimates are absent. There is no ablation varying the labeling VLM size or quality to assess sensitivity. The two-stage filtering procedure (rule-based checks + quality verification) is described qualitatively but its cost (what fraction of annotations are rejected? how much re-labeling is needed?) is not reported.

Mitigation status. The paper does not acknowledge this as a limitation and does not propose future work to reduce labeling cost. The "Limitation" section (Section 8) focuses on perceptual and control limitations without mentioning data costs. This is a significant oversight: the paper's core technical contribution (adaptive CoT) depends on a costly labeling pipeline whose expense is invisible in the reported results. A practitioner evaluating whether to adopt VLingNav needs to know that the headline performance includes an implicit capital investment in dataset construction that is not amortized in any reported metric.

---

Performance Collapses on the Hardest Generalization Splits and Under Domain Shift

The assumption or constraint. VLingNav is trained on data from specific simulated environments (HM3D, MP3D, EVT-Bench) and evaluated on held-out test splits from those same environments — a standard protocol in embodied navigation research. However, the assumption that performance on these test splits predicts real-world generalization is tested only qualitatively through small-scale real-world experiments (Section 6.3) with limited statistical rigor.

The consequence. The model shows clear signs of simulation-specific overfitting and domain sensitivity. On the HM3D OVON benchmark (Table 3), which explicitly tests generalization to novel object categories, a revealing gradient emerges: VLingNav achieves 59.3 SR on val seen (trained categories), 56.8 SR on val seen synonyms (synonyms of trained categories), and 50.1 SR on val unseen (completely novel categories). The drop from seen to unseen is -9.2 SR (-15.5%), which is substantial but comparable to prior methods. However, the synonyms split reveals something more concerning about linguistic generalization: the gap between val seen (59.3) and val seen synonyms (56.8) is -2.5 SR, while the gap between val seen synonyms (56.8) and val unseen (50.1) is -6.7 SR. This suggests that VLingNav's linguistic memory and reasoning provide some benefit for semantic generalization (synonyms), but performance degrades sharply when the object category is entirely novel — the model has not learned a generalizable concept of "find objects of category X" but rather has partially memorized category-specific navigation patterns.

The real-world experiments, while demonstrating successful deployment, reveal another dimension of this limitation: the paper reports that VLingNav tracks dynamic non-human targets despite training only on human tracking data (Section 6.4.2), but provides only qualitative examples without quantitative success rates. If this cross-domain generalization were robust, the paper would likely quantify it — the absence of numbers suggests the behavior may be unreliable or infrequent. Similarly, the compositional cross-task behaviors (search-then-track, image-goal tracking) are described qualitatively (Section 6.4.1) but never evaluated systematically.

What evidence exists in the paper. The OVON benchmark results (Table 3) provide the clearest quantitative evidence: the 50.1 SR on val unseen, while state-of-the-art, still means the model fails on half of all episodes involving novel object categories. The SFT training steps ablation (Section 6.5.4, Figure 11) shows "excessive training leads to diminishing returns and eventual performance degradation, likely due to overfitting on the simulation data" — this is direct evidence that the model is susceptible to simulation-specific overfitting, and the paper's remedy (stopping at 20K steps) is a mitigation, not a solution. The real-world experiments (Section 6.3) use only 10 trials per condition and report no numerical results, making it impossible to quantify the sim-to-real transfer gap.

Mitigation status. The paper employs open-world video co-training (Section 4.2, Table 8) specifically to "reduce the sim-to-real transfer gap," and the ablation shows that removing this co-training reduces ImageNav SR from 60.8 to 50.2 (-17.4%) — confirming that domain diversity in training data is essential for generalization. However, even with co-training, the model's performance on out-of-distribution categories and real-world scenarios is far from ceiling. The paper's Section 7, point 4 acknowledges the generalization challenge implicitly by celebrating the zero-shot transfer results, but does not characterize the conditions under which transfer fails. The limitation section (Section 8) does not discuss generalization failures or domain sensitivity at all, focusing instead on perceptual and architectural limitations. This is a missed opportunity: understanding when and why the model fails to generalize would be more informative for future work than the qualitative success cases the paper emphasizes.

---

The RL Post-Training Gains Are Unexplained and Potentially Brittle

The assumption or constraint. The online expert-guided RL post-training stage (Section 5.3) is presented as a key contribution that "enables the model to surpass pure imitation learning and to acquire more robust, self-explored navigation behaviors" (Abstract). The composite loss (Equation 8) uses an extreme weighting of λ = 0.01, meaning the RL policy gradient is weighted 100× lower than the SFT imitation loss. The paper's stated rationale is that this weighting was "determined by the scale of different losses" (Section 5.4.1), but no systematic study of λ sensitivity is reported.

The consequence. The extreme SFT dominance (λ = 0.01) raises questions about what the RL stage actually contributes and whether the reported gains are robust:

1. Is RL providing genuine policy improvement, or is it mostly continued SFT with slightly more data? The hybrid buffer includes expert-guided trajectories — essentially additional SFT data generated on-policy but from the expert, not from the current policy. The RL loss term, being 100× smaller than the SFT term, may contribute negligible gradients. The observed performance improvements could be primarily attributable to (a) additional training data from expert rollouts (more imitation data, not RL) and (b) the on-policy data collection providing a better match to the model's own state distribution (addressing covariate shift through data diversity rather than through policy gradient optimization). The paper does not ablate the RL term from the composite loss — the Expert Rollout ablation in Figure 11 uses the same composite loss rather than pure SFT on expert data — so the contribution of the RL gradient specifically cannot be isolated.

2. The λ = 0.01 choice is unexplained and potentially brittle. The paper states λ was "determined by the scale of different losses," suggesting it was chosen to balance gradient magnitudes rather than through a principled hyperparameter search. If this weighting is specific to the loss scales in this particular setup (which depend on the MSE loss magnitude, the CE loss magnitude, the advantage estimation method, and the batch composition), then reproducing or extending the method requires re-tuning λ for each new domain or task — a costly process requiring online interaction data.

3. The 10-iteration schedule is arbitrary. The paper provides no justification for 10 iterations versus 5 or 20. Figure 11 shows performance plateauing, but does not report whether further iterations cause degradation (as the SFT steps experiment in Figure 10 shows overtraining can hurt). The 10-iteration choice may be hitting a sweet spot that would not generalize to other tasks or environments.

What evidence exists in the paper. Figure 11 provides the most relevant evidence, comparing Naive Rollout, Expert Rollout, and Hybrid Rollout strategies. Naive Rollout (pure RL without expert guidance) fails to improve — confirming that expert data is essential. Expert Rollout (DAgger-like) provides substantial improvement. Hybrid Rollout achieves the best performance. However, none of these conditions ablates the λ parameter or isolates the RL gradient's contribution. The paper also does not report training curves (loss vs. iteration, advantage estimates over time, policy entropy over time) that would help diagnose whether the RL term is actively shaping the policy or serving as negligible noise.

Mitigation status. The paper does not acknowledge this as a limitation and does not propose future work to better understand or improve the RL training dynamics. The limitation section (Section 8) focuses on perceptual and control limitations without discussing training methodology. This is a significant omission: the RL post-training stage is presented as a core methodological contribution, but its operating mechanism is not adequately characterized, making it difficult for practitioners to adopt the approach with confidence or to extend it to new domains.

---

The Model Assumes a Structured, Tag-Based Output Format That Constrains Reasoning Flexibility

The assumption or constraint. AdaCoT's reasoning output follows a rigid template enforced during training: CoT content must be enclosed in thinking ... response tags, and memory summaries must be enclosed in <summary> ... </summary> tags (Section 3.3.2). The Nav-AdaCoT-2.9M dataset enforces this format through the labeling prompt's "explicit formatting requirements" (Section 4.1.2), and the SFT loss (Equation 6) supervises exact tag reproduction. There is no mechanism for the model to deviate from this structure — to produce reasoning without a summary, to update memory without full deliberation, or to intermix reasoning and action generation.

The consequence. This rigid structure creates two practical problems:

1. The reasoning and summary are always generated together or not at all. When <think_on> is triggered, the model must produce both reasoning text and an environmental summary in a single autoregressive block. There is no mechanism for the model to update its linguistic memory without full deliberation (e.g., "I just passed a door on the left — note for later" without analyzing the full scene), nor to deliberate without storing a summary (e.g., "Is that the target? Let me think — no, wrong color" without creating a permanent memory). This coupling may explain the qualitative examples in Figures 6 and 9: CoT activations appear at major decision points but never for brief, targeted updates. The model may be forgoing useful partial updates because the compute cost of generating both reasoning AND summary outweighs the benefit, even when either alone would be worthwhile.

2. The model cannot use reasoning to directly modify actions. The action model takes the hidden state $\mathbf{h}_t^{pred}$ of the final predicted token as input (Section 3.3.3, Equation 5). If CoT is triggered, this hidden state encodes the full reasoning text; if not, it encodes only the <think_off> token. But in neither case can the CoT reasoning explicitly reference or modify the action trajectory — it can only influence it through the shared hidden representation. This is an indirect, opaque coupling: the model learns through gradient descent that certain reasoning patterns correlate with certain trajectory predictions, but there is no mechanism for the reasoning to explicitly say "move 0.5m forward" or "turn left 30 degrees," or for the action model to explicitly condition on specific reasoning conclusions. This limits the interpretability benefit: even when the model produces sensible reasoning, there is no guarantee that the actual trajectory reflects those conclusions (and vice versa — the trajectory might be sensible even if the reasoning is nonsense).

What evidence exists in the paper. The paper presents no analysis of the relationship between reasoning content and predicted trajectories. The qualitative examples (Figures 6, 9) show CoT text alongside trajectory visualizations, but the connection is asserted, not demonstrated. There is no ablation comparing the structured tag format against alternative designs (e.g., interleaved reasoning and action tokens, separate reasoning and memory generation, optional summary without full reasoning). The 2.1% CoT activation rate (Table 6) is interpreted as evidence of selective deliberation, but could alternatively reflect that the rigid format makes reasoning too expensive for all but the most critical situations — the model may want to reason more often but cannot afford the computational cost of generating both reasoning and summary.

Mitigation status. The paper does not discuss this as a limitation. The structured format is presented as a design choice without analysis of its constraints on model behavior. The limitation section (Section 8) does not mention the rigidity of the reasoning format, and future work proposals focus on multi-view perception and dual-system control rather than on more flexible reasoning architectures. This is a missed opportunity: the paper's core thesis — that adaptive reasoning is essential for embodied navigation — would be strengthened by an exploration of how reasoning should be structured, not just when it should occur. The current rigid format is a reasonable first design but is treated as fixed rather than as a dimension for future optimization.

7. Implications and Future Directions

How This Work Changes the Landscape

VLingNav advances the field of VLA-based embodied navigation by demonstrating that explicit cognitive mechanisms—adaptive deliberation and linguistic memory—can be bootstrapped from a VLM backbone through carefully designed data and training, rather than requiring fundamentally new model architectures. This is not a paradigm shift in the sense of replacing VLMs as the foundation for navigation, but it is a significant reframing of what VLA models should do: they should not merely translate perception to action but should actively manage cognitive resources (deciding when to think) and maintain semantic state (remembering what they've encountered in language).

Three specific shifts emerge from this work:

First, chain-of-thought reasoning in embodied agents is recharacterized from a schedule-design problem to a learned gating problem. Prior work treated CoT frequency as a hyperparameter to be set by the human designer (OctoNav's fixed intervals, NavA³'s always-on reasoning). VLingNav's AdaCoT mechanism demonstrates that a model can learn from data when reasoning is beneficial, and that this learned schedule is dramatically more efficient than any fixed schedule—achieving state-of-the-art performance with reasoning activated on only 2.1% of steps (Table 6). The implication for future work is clear: researchers building embodied CoT systems should invest in data that teaches models when to reason, not in better heuristics for scheduling reasoning. The finding that dense per-step CoT actively degrades performance (25.3% vs. 36.2% SR on ObjNav, Table 6) provides a strong negative result that should discourage the "reason everywhere" approach implicit in some prior work.

Second, the paper provides the first systematic evidence that linguistic memory and visual memory serve complementary rather than redundant functions in VLA navigation. The ablation in Table 7—where language-only memory barely outperforms no memory (18.8% vs. 15.4% SR on ObjNav) while visual-only memory recovers most performance (45.2%) and the combination achieves the best results (50.1%)—establishes a functional decomposition: visual features handle continuous spatial navigation decisions, while linguistic summaries prevent catastrophic forgetting of semantic episodic history (which rooms were visited, what objects were seen). This resolves a tension in the prior literature between approaches that relied purely on implicit visual memory (Uni-NaVid, NaVILA, TrackVLA) and those that argued for explicit memory structures (Mem2Ego, MapNav). The answer is not one or the other—it's both, for different purposes. Future memory systems for VLA navigation should be designed with this decomposition in mind, allocating representational capacity accordingly rather than treating "memory" as a monolithic capability.

Third, the paper demonstrates that continuous-action RL post-training is viable for VLA navigation when combined with expert guidance, but that naive RL fails catastrophically. This is a diagnostic finding that recalibrates expectations for RL in embodied VLA systems. The failure of Naive Rollout to improve over SFT (Figure 11) and the extreme SFT dominance in the composite loss (λ = 0.01, meaning the RL gradient is 100× weaker than the SFT gradient) suggest that current RL algorithms cannot overcome the credit assignment problem in sparse-reward, long-horizon navigation without expert scaffolds. This makes research directions focused on pure RL for navigation (scaling up GRPO without demonstrations, outcome-based rewards without step-level guidance) less attractive, and makes directions focused on better expert data generation, more informative reward shaping, or hybrid imitation-RL objectives more attractive. The paper's specific recipe—probabilistic continuous action head, hybrid rollouts, heavily SFT-weighted composite loss—provides a concrete starting point that future work can refine or challenge.

The paper also reconciles an apparent contradiction in the embodied CoT literature. Aux-Think (Wang et al., 2025c) found that "excessive reasoning affects the model's efficiency and performance," while OctoNav (Gao et al., 2025) showed that CoT reasoning at fixed intervals improves navigation. These findings seemed contradictory: does reasoning help or hurt? VLingNav's results explain the discrepancy through the lens of reasoning frequency adaptivity. Fixed-interval reasoning (OctoNav) provides some benefit over no reasoning, but at suboptimal efficiency. Dense reasoning (the extreme case of Aux-Think's "excessive reasoning" finding) actively hurts. Adaptive reasoning achieves the best of both worlds: the benefits of deliberation at critical junctures without the interference cost of constant verbalization. This reframing—reasoning is beneficial only when appropriately targeted—should guide future work away from binary "reasoning vs. no reasoning" comparisons toward more nuanced analyses of when and what to reason about.

Follow-Up Research This Work Enables

Characterizing when AdaCoT activates and whether activations are genuinely context-sensitive. The paper reports that AdaCoT activates on 2.1% of steps (Table 6) and shows qualitative examples of plausible activations at decision points (Figure 6), but provides no systematic analysis of when reasoning is triggered. A strong follow-up study would instrument VLingNav to log CoT indicator predictions alongside environment features (distance to nearest obstacle, number of visible navigation options, whether the current room matches a previously visited room, time since last target sighting, semantic complexity of the scene as measured by object detector outputs). This would answer: does <think_on> occur reliably at intersections, at target-ambiguous moments, and after occlusion events, or is the 2.1% activation distributed essentially randomly? A causal intervention experiment would further strengthen this: manipulate the environment mid-episode (e.g., close a previously open door, add a distractor object, remove the target from view) and measure whether the gating probability increases appropriately. Without this analysis, the "adaptive" claim remains plausible but unverified—the model might have learned a sparse-but-fixed reasoning schedule that happens to work well on benchmark distributions without genuinely responding to situational demands.

Measuring the semantic accuracy and utility of linguistic memory summaries over long trajectories. The paper presents VLingMem as a core contribution and shows it improves performance (Table 7), but never analyzes what the summaries contain or whether they remain factually accurate over long episodes. A follow-up study would extract the <summary> buffer from VLingNav across hundreds of episodes and evaluate: (a) factual accuracy—do summaries correctly describe visited rooms, seen objects, and taken actions, or do they hallucinate (e.g., claiming to have searched a room that was never entered)? (b) Information density—do summaries accumulate redundant information over time ("entered kitchen" appearing 3 times across a 50-step episode) or do they efficiently compress novel observations? (c) Causal importance—if summaries from specific steps are ablated (removed from the memory buffer), does performance degrade more for critical decision points than for routine navigation? This analysis would validate whether VLingMem serves the episodic memory function the paper claims or whether the performance gains in Table 7 are attributable to a simpler mechanism (e.g., the summaries provide useful task-reminder cues rather than genuine spatial history). It would also inform whether the unbounded-growth buffer design (no forgetting, no summarization of summaries) is sustainable for very long episodes (500+ steps) or whether memory management mechanisms are needed.

Scaling VLingNav to multi-view perception and measuring the contribution of wider perceptual coverage. The paper acknowledges monocular input as a limitation (Section 8) and proposes multi-view integration as future work, but does not estimate how much of the remaining performance gap is attributable to narrow FOV. A direct follow-up experiment would train VLingNav variants with 1, 2, and 4 cameras (all other architecture held constant) and measure performance across benchmarks, with specific attention to collision rates (Table 4), SPL (where multi-view baselines like NavFoM show advantages on HM3D OVON, Table 3), and tracking robustness when the target exits the forward FOV. This would decompose the performance gap between VLingNav and multi-view methods (NavFoM, NavFoM*) into the portion explainable by perceptual coverage versus architectural differences (AdaCoT, VLingMem, continuous actions). The hypothesis—that wider FOV primarily improves SPL (fewer exploration steps wasted on turning to look around) and collision rates (obstacles visible sooner) while AdaCoT and VLingMem contribute primarily to SR (better decisions at critical junctures and memory of searched areas)—would give practitioners clear guidance on where to invest engineering effort for specific deployment requirements.

Stress-testing the sim-to-real transfer by evaluating on systematically varied real-world conditions. The paper's real-world experiments demonstrate zero-shot transfer (Section 6.3) but use only 10 trials per condition, report no numerical results, and test in relatively controlled environments. A rigorous follow-up would define a real-world generalization benchmark with controlled variation along dimensions known to challenge sim-to-real transfer: lighting (bright daylight, dim indoor, backlighting), environment clutter (empty corridor, moderately furnished room, densely cluttered space), dynamic obstacles (static, slow-moving people, fast-moving people), and viewpoint variation (camera at standard height, elevated, tilted). For each condition, measure SR, SPL, collision rate, and AdaCoT activation rate over 50+ trials. The key question: does AdaCoT's activation rate increase under challenging real-world conditions (suggesting genuine adaptivity to perceptual difficulty), or does it remain at ~2.1% regardless (suggesting the gating behavior is fixed by simulation training)? This experiment would simultaneously validate the adaptive reasoning claim and characterize the robustness boundary of the current system—identifying which real-world conditions VLingNav can handle zero-shot and which require additional real-world fine-tuning or architectural changes.

Investigating whether the structured reasoning format constrains model behavior and whether more flexible formats improve performance. The paper's AdaCoT uses a rigid template: when <think_on> is triggered, the model must generate both reasoning text AND an environmental summary in a single autoregressive block (Section 3.3.2). This coupling means the model cannot update memory without full deliberation, nor deliberate without creating permanent memory. A follow-up study would train variants with decoupled formats: (a) independent gating—separate <think_on> and <remember_on> tokens allowing the model to store a summary without reasoning, reason without storing, or do both; (b) interleaved action and reasoning—allow the model to generate reasoning tokens and action tokens in the same output sequence, enabling explicit connections like "I see the target at coordinates X, therefore I will move forward 0.5m"; (c) hierarchical reasoning—allow the model to generate brief reasoning (1-2 sentences) or detailed reasoning (full paragraph) depending on a predicted verbosity level. The evaluation would measure SR, SPL, CoT activation rate, and inference latency for each variant. The hypothesis—that decoupled formats enable more frequent but lighter-weight cognitive operations (brief memory updates without full deliberation, quick reasoning checks without permanent storage) leading to higher overall performance with similar or lower total compute—would directly test whether the current format is optimal or merely a reasonable first design.

Testing whether the expert-guided RL recipe generalizes to manipulation tasks or different robot embodiments. The paper's RL post-training approach—probabilistic continuous action head, hybrid rollouts with expert takeover after k=15 stuck steps, composite PPO+SFT loss with λ=0.01—is developed and validated exclusively for navigation. The question of whether this recipe transfers to other VLA domains is both practically important and scientifically informative about the generality of the credit assignment challenge. A follow-up would apply the same RL recipe (with minimal adaptation) to a tabletop manipulation benchmark (e.g., SIMPLER, CALVIN, or RLBench) using a comparable VLA backbone, and measure: (a) whether naive RL also fails to improve over SFT (suggesting the credit assignment problem is fundamental to embodied RL, not navigation-specific); (b) whether the λ=0.01 weighting transfers or requires re-tuning; (c) whether the k=15 stuck-detection threshold needs domain-specific adjustment. A negative result—the recipe fails on manipulation—would be equally informative, suggesting that navigation's specific structure (spatial exploration with clear progress metrics) makes expert-guided RL more tractable than in domains with more complex contact dynamics or longer action horizons.

Practical Applications and Downstream Use Cases

Deployment of service robots in multi-room indoor environments with minimal per-deployment fine-tuning. The paper's results on MP3D (58.9% SR, +12.3 points over prior best) and HM3D OVON val unseen (50.1% SR on completely novel object categories) are directly relevant to commercial service robots (hotel delivery, hospital logistics, office concierge) that must navigate large, multi-room buildings and find specified objects or locations. The key practical benefit is that a single VLingNav model, trained entirely in simulation, can be deployed zero-shot in new physical environments without environment-specific mapping, per-room fine-tuning, or reconfiguration. The 2.5 FPS inference speed on an RTX 4090 (Section 6.3.1) means the computational requirements are compatible with either on-premise edge GPU deployment or low-latency cloud offloading over Wi-Fi. For a hotel delivery robot operating across 50 rooms, VLingNav's VLingMem module (which prevents redundant room re-checking—the "w/o Memory" ablation drops to 15.4% SR, Table 7) is the critical capability: without explicit memory, the robot would repeatedly search the same areas, draining battery and failing to complete deliveries within acceptable time windows.

Autonomous mobile camera operators for event coverage and security patrol. The Embodied Visual Tracking results on Distracted Tracking (67.6% SR, 73.5% TR, Table 4) demonstrate that VLingNav can maintain continuous tracking of a specified target in crowded environments with multiple similar-looking distractors. A practical deployment scenario is an autonomous camera robot at conferences, sports events, or security patrol that follows a designated person (speaker, athlete, security guard) through crowds while avoiding collisions. The AdaCoT mechanism provides a specific operational advantage here: when the target is temporarily occluded (passes behind a pillar, is blocked by another person), the model triggers <think_on> to reason about the target's likely location based on last-known position and movement direction, rather than immediately switching to a distractor or wandering randomly. The continuous action space (trajectories of $(x, y, \theta)$ waypoints) provides smoother camera motion than discrete-action alternatives, which matters for video quality—jerky discrete turns would produce unusable footage. The practical limitation is the monocular FOV: a camera operator robot would benefit substantially from a wider-angle or multi-camera setup to maintain visual contact with the target more consistently without physical rotation, reducing the need for occlusion-recovery reasoning in the first place.

Data generation for training downstream navigation policies through automated exploration and annotation. The autonomous CoT labeling pipeline (Section 4.1.2) that produced Nav-AdaCoT-2.9M—using Qwen2.5-VL-72B to annotate 2.9M navigation steps with reasoning—is itself a practical capability beyond the specific VLingNav model. Organizations building custom navigation systems for specialized environments (warehouses, farms, construction sites) could use a similar pipeline to generate training data for their domain: deploy a VLA model for data collection, record trajectories (visual observations + actions), then use a large VLM to retrospectively annotate those trajectories with adaptive CoT reasoning. The resulting domain-specific dataset could fine-tune a smaller, deployment-efficient VLA model. The key numbers from the paper: 472K CoT annotations from 2.9M steps (16% annotation rate) was sufficient to train effective adaptive gating (2.1% activation at test time, Table 6). This provides a rough target for practitioners: annotating approximately 15-20% of trajectory steps with CoT is sufficient; denser annotation is unnecessary and may be counterproductive (the Dense CoT result of 25.3% SR, Table 6). The two-stage filtering pipeline (rule-based checks + expert trajectory cross-validation) provides a template for quality control that balances automation with accuracy.