ArXiv: 2602.14979
🎯 Pitch
For robots to truly reason about the physical world, their internal 'thoughts' must be tied to concrete spatial coordinates, not just abstract text. RynnBrain demonstrates this by interleaving natural language reasoning with explicit point and bounding box predictions—a method called Chain-of-Point reasoning—yielding a 7% jump in trajectory prediction over text-only counterparts. The open-source model family outperforms existing embodied foundation models across 28 benchmarks by grounding perception, planning, and action in unified spatiotemporal representations.
1. Executive Summary
This paper introduces RynnBrain, an open-source spatiotemporal foundation model for embodied intelligence that unifies perception, reasoning, and planning within a physically grounded framework across three model scales—2B, 8B, and 30B-A3B MoE—and four post-trained variants (RynnBrain-Nav, RynnBrain-Plan, RynnBrain-VLA, and RynnBrain-CoP). Trained on a 20-million-sample corpus constructed through a human–model collaborative data flywheel, RynnBrain integrates four core capabilities—comprehensive egocentric understanding (e.g., fine-grained video QA and OCR), diverse spatiotemporal localization (e.g., object, area, affordance, and trajectory prediction across episodic memory), physically grounded reasoning via Chain-of-Point (CoP) reasoning (interleaving textual inference with explicit spatial grounding primitives such as points and bounding boxes), and physics-aware planning (embedding location information directly into planning outputs). Across 28 benchmarks, RynnBrain largely outperforms existing embodied foundation models, with the CoP reasoning variant improving complex spatiotemporal tasks such as trajectory prediction by approximately 7%, establishing that embodiment-agnostic foundational pretraining with explicit spatial grounding substantially benefits downstream navigation, manipulation, and VLA tasks—but only when reasoning traces remain anchored to observable physical evidence rather than conducted in purely textual space.
2. Context and Motivation
The Core Problem: Embodied Agents Lack a Physically Grounded "Brain"
The fundamental challenge this paper addresses is the absence of a unified, physically grounded foundation model for embodied intelligence. Current AI systems for robotics and embodied agents fall into two camps, each with a critical deficiency:
On one side, general-purpose vision-language models (VLMs) like GPT-4o and Gemini possess broad semantic understanding, strong reasoning capabilities, and the ability to transfer knowledge across diverse domains—skills inherited from massive multimodal pretraining on internet-scale data. However, these models are not intrinsically grounded in physical dynamics. They struggle with spatio-temporal consistency (reasoning about how objects move and relate over time), physical reasoning (understanding material properties, affordances, and 3D spatial relationships), and actionable planning (producing outputs that a robot can directly execute in the physical world). Their reasoning typically occurs in an abstract textual space, disconnected from the concrete spatial coordinates and temporal constraints that govern real-world interaction.
On the other side, embodied models trained primarily on action-centric data—such as robot trajectories, manipulation demonstrations, or navigation episodes—excel at producing physically executable outputs but sacrifice high-level semantic abstraction. By training narrowly on task-specific action distributions, these models lose the broad generalization capabilities that VLMs acquire from large-scale pretraining. They can execute specific motions they have seen but fail to reason about novel objects, unfamiliar environments, or tasks that require compositional understanding beyond their training distribution.
The paper's thesis is that progress toward general-purpose embodied intelligence requires bridging this gap: a model must preserve the semantic breadth of VLMs while being explicitly structured around physical space, temporal dynamics, and embodiment constraints. Such a model would serve as a high-level cognitive "brain" for perception, reasoning, and decision-making, while remaining adaptable to downstream control systems that handle low-level execution.
Why This Problem Matters
The importance of this problem spans both practical deployment and fundamental AI research:
Practical impact. Robots capable of operating in unstructured human environments—homes, hospitals, warehouses—must handle enormous variability in scenes, objects, tasks, and interaction modalities. A robot that can only perform tasks it was explicitly trained on is economically infeasible; what is needed is a system that can generalize from high-level instructions ("tidy up the sink") to concrete actions in a novel kitchen it has never seen. The hierarchical architecture this paper advocates—a generalizable "brain" that produces physically grounded plans, paired with a "body" policy that executes them—is a promising path toward such general-purpose robots.
Research significance. The paper argues that existing VLMs fail on embodied tasks not because they lack knowledge, but because their reasoning is disconnected from physical grounding. A VLM might correctly reason that "to pour coffee, I need to locate the handle of the coffee pot," but if it cannot output the spatial coordinates of that handle in the current egocentric view, the downstream robot controller has no actionable input. Conversely, a pure action-prediction model might learn to approach handles from demonstration data but cannot adapt when the handle is in an unexpected position or when the task requires reasoning about which object to grasp first among multiple candidates. The missing piece is a model that explicitly produces spatial grounding primitives (points, bounding boxes, trajectories) interleaved with textual reasoning, so that the reasoning process remains anchored to observable physical evidence.
Prior Approaches and Their Limitations
The paper situates itself against several recent efforts to build embodied foundation models, each of which makes progress but falls short in specific ways:
RoboBrain 2.0 (robobrain2.0) unified understanding, localization, and planning within a single VLM to facilitate complex embodied tasks. This was an important step toward integrated systems, but the paper identifies three limitations: (1) its egocentric cognitive capabilities remain narrow because training is confined to limited task categories or perception modalities, restricting robustness in complex environments; (2) spatial reasoning is grounded in static image inputs, lacking coherent spatio-temporal representations necessary for global scene awareness (understanding how objects and the agent relate across time, not just in a single frame); and (3) high-level reasoning and planning are conducted primarily in a purely textual space, leading to hallucinations and inconsistencies with physical constraints—the model might describe a plan that is physically impossible without the feedback mechanism of anchoring each step to actual spatial coordinates.
Robix (Robix) emphasized more natural human-robot interaction during execution but shares similar limitations. The interaction is richer, but the underlying reasoning still lacks explicit physical grounding.
MiMo-Embodied, Pelican-VL, and Cosmos-Reason2 represent additional efforts at embodied foundation models. The paper evaluates against all of these in Tables 3 and 4, showing that while they achieve non-trivial performance on certain benchmarks, they consistently underperform RynnBrain across the board, particularly on tasks requiring fine-grained spatial localization and spatio-temporal reasoning—the dimensions the paper identifies as critical missing pieces.
General VLM reasoning approaches represent another category of prior work that the paper critiques. Systems like Video-R1, DeepSeek-VL, and MMR1 (leng2025mmr1) rely on purely textual reasoning paradigms—the model thinks in words alone. Other approaches such as TACO, DeepEyes, and ReFocus incorporate auxiliary tools like region zooming to alleviate visual recognition challenges, but the reasoning process itself remains largely detached from physical spatial structure. Alternative methods that explore "visual imagination" during reasoning (PARM, VPRL, GoT-R1) actually worsen the problem by introducing hallucinated visual content, undermining physical consistency rather than strengthening it. The paper's core observation is that for embodied agents operating in real-world environments, reasoning must be grounded in observable physical evidence—not in imagined or purely textual abstractions.
How RynnBrain Positions Itself
The paper distinguishes itself through four key design choices that directly address the limitations of prior work:
1. Explicit spatio-temporal representation across episodic memory. Unlike prior systems that reason about static images or short clips, RynnBrain maintains representations across extended video sequences (up to 2048 frames), enabling it to track objects, predict trajectories, and reason about spatial relationships over time. This is a qualitative shift from "what is in this image" to "how does the world change as the agent moves through it."
2. Physically grounded output space via discrete coordinate tokens. Rather than treating spatial quantities as free-form text (e.g., "the cup is on the left side of the table"), RynnBrain discretizes all spatial entities—bounding boxes, points, trajectory waypoints—into integer tokens normalized to [0, 1000]. This converts continuous spatial prediction into a classification problem compatible with the autoregressive language modeling objective, while producing outputs that are directly consumable by downstream robot controllers. The model outputs the equivalent of "the handle is at pixel coordinates (342, 567)" rather than "the handle is somewhere on the right."
3. Chain-of-Point (CoP) reasoning. This is the paper's signature innovation for physically grounded cognition. Rather than reasoning in pure text (like a standard chain-of-thought) or with purely imagined visual elements, CoP interleaves textual reasoning steps with explicit spatial grounding primitives. A CoP trace might look like: "I need to grasp the handle of the coffee pot. <object><frame 47>: (234, 512), (389, 601)</object> The handle is oriented vertically, so I should approach from the side. <affordance><frame 47>: (310, 556)</affordance> ..." Each reasoning step is anchored to concrete visual evidence with precise spatial coordinates, making the trace verifiable against the actual video and resistant to hallucination.
4. Human-model collaborative data flywheel. The paper acknowledges that a fundamental bottleneck for embodied foundation models is the scarcity of high-quality training data. To address this, it designs dedicated data pipelines that strategically leverage the priors of pretrained foundation models (e.g., Qwen2.5-VL for object detection, Qwen3-VL-235B for chain-of-thought generation) while introducing human supervision only at critical decision points. This enables scaling the training corpus to over 20 million samples while maintaining annotation quality under constrained labeling budgets.
The paper's contribution is thus not a single architectural innovation but rather a systematic integration of spatio-temporal representation, physically grounded output space, interleaved reasoning, and scalable data construction—all unified under the thesis that embodied intelligence requires reasoning traces to remain firmly anchored in observable physical evidence. The four post-trained variants (CoP, Nav, Plan, VLA) demonstrate that this foundation transfers effectively to diverse downstream tasks, from spatial reasoning to navigation to manipulation to direct action prediction.
3. Technical Approach
3.1 Reader Orientation
RynnBrain is a vision-language model that has been explicitly trained to understand and reason about the physical world from an egocentric (first-person) perspective, producing not just text but also precise spatial outputs like bounding boxes, points, and trajectories. The paper addresses the fundamental problem that standard vision-language models can describe a scene but cannot ground their reasoning in physical coordinates, while robot-specific models can produce actions but lack broad semantic understanding; RynnBrain's solution is to train a single model that does both simultaneously—using a unified autoregressive framework where spatial coordinates are represented as discrete text tokens interleaved with natural language, enabling the model to think and communicate in physically grounded terms.
3.2 Big-Picture Architecture (Diagram in Words)
The RynnBrain system has five major components arranged in a standard vision-language pipeline, with several key modifications for physical grounding:
-
Vision Encoder — processes raw visual inputs (single images, multi-view images, or video frames) into a sequence of visual feature tokens. Inherited from Qwen3-VL, this component uses Interleaved MRoPE (Multimodal Rotary Position Embeddings) to encode both spatial position within frames and temporal position across frames, enabling the model to track objects and motion over time.
-
Vision-Language Projector — a learned mapping that translates visual tokens into the same embedding space as the language model, allowing visual and textual information to be processed jointly. This is a standard adapter layer initialized from Qwen3-VL checkpoints.
-
Large Language Model Backbone — a decoder-only transformer that processes the combined sequence of visual tokens, text tokens, and special coordinate tokens autoregressively (predicting one token at a time, each conditioned on all previous tokens). Available in three scales: a 2-billion-parameter dense model, an 8-billion-parameter dense model, and a 30-billion-parameter mixture-of-experts model with 3 billion active parameters (30B-A3B MoE). This is the "brain" that performs understanding, reasoning, and planning.
-
Unified Coordinate Token Vocabulary — a special set of integer tokens representing normalized spatial coordinates in the range [0, 1000]. These tokens are embedded in the same vocabulary as text tokens, so bounding boxes become sequences like
(234, 512), (389, 601), trajectory waypoints become ordered sequences of(x, y)pairs, and affordance points become single coordinate pairs. This is the key design choice that enables the model to produce spatially grounded outputs using the same next-token prediction mechanism it uses for language. -
Training Data Pipeline — a human-model collaborative system that generates the 20-million-sample pretraining corpus across multiple data categories (general MLLM data, cognition data, localization data, planning data). This pipeline uses pretrained foundation models (Qwen2.5-VL, Qwen3-VL-235B) to propose annotations and generates questions, then introduces human verification at critical decision points to ensure quality.
Information flows as follows: multimodal input (images, video frames, text instructions) enters the system → the vision encoder converts visual data into token sequences with spatio-temporal position encodings → the projector maps these into the LLM's embedding space → the combined token sequence is processed by the decoder-only transformer → the model autoregressively generates a mixed output sequence of text tokens (reasoning traces, answers, plans) and coordinate tokens (bounding boxes, points, trajectories) → downstream systems (navigation policies, manipulation controllers, VLA models) consume these grounded outputs for execution.
The training process follows a staged approach: (1) large-scale pretraining on the 20-million-sample corpus to establish general embodied understanding, (2) cold-start supervised fine-tuning for the Chain-of-Point reasoning variant, (3) reinforcement learning with GRPO to align the model with physically grounded reasoning tasks, and (4) task-specific post-training for navigation, planning, and VLA variants.
3.3 Roadmap for the Deep Dive
-
First, the unified spatio-temporal representation and physically grounded output space (Section 3.1), because these are the foundational design decisions that distinguish RynnBrain from standard VLMs and enable all downstream capabilities. Understanding how coordinate tokens work is prerequisite to understanding everything else.
-
Second, the pretraining data pipeline (Section 3.2), because the quality and diversity of training data is the primary bottleneck the paper identifies for embodied foundation models. I will walk through each data category—general MLLM data, multi-dimensional cognition data, spatio-temporal localization data, and physics-aware planning data—explaining what each contributes and how it is constructed through the human-model collaborative flywheel.
-
Third, the Chain-of-Point (CoP) reasoning mechanism and training procedure (Section 4), because this is the paper's signature innovation for physically grounded cognition. This includes the cold-start SFT data construction (how interleaved reasoning traces are generated), the GRPO reinforcement learning objective (how the model is aligned to produce spatially grounded reasoning), and the reward design for trajectory, affordance, and area prediction tasks.
-
Fourth, the post-training recipes for downstream embodied tasks (Section 5), covering how the pretrained foundation model is adapted for vision-language navigation (RynnBrain-Nav), manipulation planning (RynnBrain-Plan), and vision-language-action modeling (RynnBrain-VLA). Each adaptation reveals different aspects of how the foundation model's capabilities transfer to specific embodiments.
-
Fifth, the training infrastructure optimizations (Section 2.2), including the online load-balancing pipeline for handling variable-length sequences and the per-sample loss formulation that eliminates communication overhead, because these engineering decisions are critical to making training at this scale practically feasible.
3.4 Detailed, Sentence-Based Technical Breakdown
This is primarily a systems and training methodology paper whose core idea is that embodied intelligence requires a foundation model that produces spatially grounded outputs interleaved with textual reasoning, and that such a model can be built by representing physical coordinates as discrete tokens in a unified autoregressive framework, training on a large-scale corpus constructed through human-model collaboration, and fine-tuning with reinforcement learning to align the model's reasoning traces with physical evidence.
Unified Spatio-Temporal Representation
The foundation of RynnBrain's architecture is a unified representation that treats all visual inputs—static images, multi-view images, and video sequences—as a single modality parameterized by the number of frames. Formally, a visual input $\mathbf{V}$ is represented as a sequence of frames $\{I_t\}_{t=1}^T$, where $T=1$ for static images and $T>1$ for videos. For video inputs, frames are uniformly sampled to preserve temporal continuity—meaning the time gap between consecutive frames is approximately constant, so the model can learn consistent motion patterns.
Each frame is independently encoded into visual tokens by the vision encoder, but critically, temporal positional embeddings are added to encode the frame order. This means that token $j$ from frame $t$ receives a position encoding that captures both its spatial location within the frame (via Interleaved MRoPE, which applies rotary position embeddings to 2D spatial coordinates) and its temporal index $t$ (via an additional temporal position encoding). The result is that the model can distinguish between "the same object at the same spatial position but at different times," which is essential for reasoning about motion, trajectories, and temporal relationships.
The paper also employs DeepStack (meng2024deepstack), a technique that stacks multiple visual tokens into a single token representation before feeding them to the LLM, effectively compressing the visual sequence length. This addresses the practical bottleneck that videos with many frames produce extremely long token sequences (2048 frames at typical encoding rates would produce millions of tokens), which would exceed the model's context window and make training prohibitively expensive. DeepStack aggregates spatial neighbors or temporally adjacent visual tokens, reducing the sequence length while preserving the essential visual information.
Why this representation: Treating images and videos as a single modality with temporal position encodings is cleaner than maintaining separate encoders or processing pipelines for static versus dynamic visual inputs. The alternative—having separate image and video encoders—would prevent the model from transferring knowledge between the two (e.g., object recognition learned from static images wouldn't automatically apply to video frames). The unified representation ensures that all visual understanding capabilities are shared across all input types.
Physically Grounded Output Space
This is the defining technical innovation of RynnBrain: rather than treating spatial quantities as free-form text descriptions (e.g., "the cup is on the left"), the model outputs explicit coordinate tokens that represent precise physical locations. All spatial entities—bounding boxes $\mathcal{B}$, points $\mathcal{P}$, and trajectory waypoints $\mathcal{T}$—are normalized to the range $[0, 1000]$ and encoded as integer tokens.
Normalization to $[0, 1000]$ means that a coordinate $x$ that is halfway across the image becomes the integer token 500, and a coordinate at the right edge becomes 1000. This discretization converts continuous spatial prediction into a classification problem over 1001 possible values per coordinate, which is directly compatible with the standard autoregressive next-token prediction objective used to train language models. The model predicts spatial coordinates token by token, exactly as it predicts text tokens: given the input and all previously generated tokens, it outputs a probability distribution over the 1001 possible coordinate values for the next position.
A bounding box $\mathcal{B} = \{(x_0, y_0, x_1, y_1)\}$ is represented as a sequence of four coordinate tokens (top-left x, top-left y, bottom-right x, bottom-right y). A point $\mathcal{P} = (x, y)$ is two coordinate tokens. A trajectory $\mathcal{T} = \{(x_i, y_i)\}_{i=1}^m$ is a sequence of $2m$ coordinate tokens, representing $m$ waypoints in order. These coordinate tokens are interleaved with text tokens in the model's output, producing mixed sequences that might read: "The handle is located at <object><frame 47>: (234, 512), (389, 601)</object>. I should grasp it at <affordance><frame 47>: (310, 556)</affordance>."
The special XML-like tags (<object>, <affordance>, <area>, etc.) serve as structured delimiters that distinguish coordinate tokens from surrounding text and indicate the semantic type of the spatial entity. This structured output format ensures that downstream systems (navigation controllers, manipulation policies) can parse the model's outputs unambiguously.
Why this form: The alternative—outputting coordinates as free-form text (e.g., "x=234, y=512")—would be ambiguous and error-prone. Free-form text coordinates would be generated one character at a time, meaning the model would need to learn that "2" followed by "3" followed by "4" represents the number 234, a level of numerical understanding that language models struggle with. Discrete coordinate tokens treat each integer as an atomic symbol, eliminating the need for the model to compose multi-digit numbers. Furthermore, bounding the range to $[0, 1000]$ creates a fixed vocabulary size (1001 entries per dimension) that can be efficiently embedded, whereas unnormalized pixel coordinates could span thousands of values and would make the classification problem intractable.
The normalization to $[0, 1000]$ rather than $[0, 1]$ or $[0, 100]$ is a practical choice: 1001 values provide sub-pixel precision when mapped back to typical image resolutions (e.g., a 1000-pixel-wide image maps 1-to-1, while a 2000-pixel-wide image maps each integer to 2 pixels, which is still fine-grained enough for most manipulation tasks), while keeping the vocabulary small enough that the classification head over 1001 classes remains computationally tractable.
Next-Token Prediction Objective
RynnBrain is trained end-to-end using a standard autoregressive language modeling objective applied to the mixed sequence of text tokens and coordinate tokens. The training loss is defined as:
where $\mathbf{V}$ denotes the visual input (a sequence of frames with temporal position encodings), $\mathbf{y} = (y_1, y_2, \ldots, y_L)$ is the target sequence of mixed text and coordinate tokens, $y_{<i}$ denotes all tokens before position $i$, and $\mathbf{\Theta}$ represents all trainable model parameters (vision encoder, projector, and LLM weights).
What it computes: For each position $i$ in the target sequence, the model takes all previous tokens and the visual input, computes a probability distribution over the entire token vocabulary (text tokens plus the 1001 coordinate tokens), and the loss is the negative log probability assigned to the correct token $y_i$. Summing over all positions gives the total loss for one training sample. Minimizing this loss encourages the model to assign high probability to the correct next token at every position, which is equivalent to maximum likelihood estimation under the model's autoregressive factorization.
Why this form: This is the standard causal language modeling objective used to train virtually all modern LLMs, which means RynnBrain can inherit the training infrastructure, optimization techniques, and initialization from Qwen3-VL without modification. The key insight is that coordinate tokens are treated identically to text tokens in the loss computation—the model receives the same learning signal for predicting (234, 512) correctly as it does for predicting "the" correctly. This unified objective is what enables the model to learn spatial grounding and language understanding jointly, rather than treating them as separate tasks with separate loss functions. The alternative—having a separate regression head for coordinate prediction with an L2 loss—would create a multi-task optimization problem where the model must balance conflicting gradients from the language loss and the coordinate loss, a well-known difficulty in multi-task learning that the unified token approach elegantly avoids.
Training hyperparameters across model scales (Table 1):
| Parameter | RynnBrain-2B | RynnBrain-8B | RynnBrain-30B-A3B |
|---|---|---|---|
| Base Model | Qwen3-VL-2B-Instruct | Qwen3-VL-8B-Instruct | Qwen3-VL-30B-A3B-Instruct |
| Optimizer | AdamW | AdamW | AdamW |
| Learning Rate (LLM) | 5e-6 | 2e-6 | 2e-6 |
| Learning Rate (Vision) | 1e-6 | 2e-6 | 2e-6 |
| Global Batch Size | 512 | 1024 | 1024 |
| Warmup Ratio | 0.03 | 0.03 | 0.03 |
The vision encoder receives a separate, typically lower learning rate to prevent catastrophic forgetting of the pretrained visual features while the language model adapts to the embodied domain. The 2B model uses a higher LLM learning rate (5e-6 vs. 2e-6) likely because smaller models have more capacity headroom from their Qwen3-VL initialization and can tolerate more aggressive updates. The warmup ratio of 0.03 means that the learning rate linearly increases from zero to its peak value over the first 3% of training steps, a standard practice to avoid destabilizing the pretrained weights early in training.
Infrastructure: Online Load-Balancing Pipeline
A practical challenge in training RynnBrain is that the training data span a wide range of sequence lengths—from short localization queries with single frames to long-form video understanding with thousands of frames. Since the transformer's computational cost scales quadratically with sequence length in the self-attention layers, a naive distribution of samples across data-parallel (DP) workers would cause a severe straggler effect: workers that happen to receive long sequences would become throughput bottlenecks, forcing all other workers to idle while waiting for them to finish.
The paper solves this with an online load-balancing pipeline that redistributes samples across DP workers to minimize the maximum cumulative sequence length assigned to any single worker. The algorithm works as follows: (1) before training, pre-compute estimated sequence lengths for all samples based on image sizes and text token counts; (2) during batch sampling, rather than assigning an equal number of samples to each DP worker, aggregate all samples across the DP group; (3) redistribute using a greedy algorithm that sorts sequences in descending order of length and iteratively assigns each sequence to the worker with the smallest current total sequence length. This is executed during data prefetching, so it does not block the training loop.
Why this approach: The naive alternative—assigning equal numbers of samples to each worker—would result in some workers receiving predominantly long sequences and others predominantly short ones, producing up to several-fold differences in per-step computation time. Pre-processing the data to create balanced batches offline would eliminate this flexibility but would require re-processing whenever training hyperparameters or datasets change. The online greedy assignment is both efficient (sorting a batch of samples is cheap relative to forward/backward passes) and adaptive, maintaining balance regardless of the data mixture.
Infrastructure: Per-Sample Loss Reduction
The standard practice for distributed training with variable-length sequences is to compute a per-token loss and normalize by the total number of tokens across all workers:
where $n$ is the number of data-parallel workers, $b_i$ is the local batch size on worker $i$, $s_{ij}$ is the sequence length of the $j$-th sample on worker $i$, and $l_{ijk}$ is the per-token loss at position $k$. The denominator $\sum_{i=1}^{n}\sum_{j=1}^{b_i} s_{ij}$ is the global total number of tokens, which requires an all-gather communication operation across all DP workers to compute.
The paper replaces this with a per-sample loss reduction:
where $b = \sum_{i=1}^{n} b_i$ is the global batch size, a constant known to every worker without communication.
What it computes: Instead of averaging over all tokens globally, this formulation first averages the loss within each sample (dividing by $s_{ij}$), then averages across samples (dividing by the global batch size $b$). This means each sample contributes equally to the gradient regardless of its sequence length, unlike the per-token formulation where longer sequences contribute proportionally more to the total loss and therefore receive larger gradient updates.
Why this form: The per-sample formulation eliminates the need for the all-gather operation to compute the global token count, reducing communication overhead and improving training efficiency (the paper reports this "doubles training efficiency"). However, it changes the optimization semantics: in the per-token formulation, every token is weighted equally in the objective, which is arguably correct because the model should learn equally from all tokens. In the per-sample formulation, a short sample with 10 tokens receives the same total gradient contribution as a long sample with 10,000 tokens, which means the model effectively downweights information from longer sequences. The paper reports that this does not harm convergence or stability, likely because the batch size $b$ is large enough that the averaging across samples is well-behaved, and because the load-balancing ensures that sequence lengths within each worker are already somewhat balanced.
Infrastructure: Memory Optimization for Large Models
For the 2B and 8B models, the paper uses ZeRO-1 optimizer (deepspeed) and per-block gradient checkpointing. ZeRO-1 partitions the optimizer states (momentum and variance buffers for AdamW) across data-parallel workers, reducing per-GPU memory consumption. Per-block gradient checkpointing trades computation for memory: instead of storing all intermediate activations for backpropagation, it stores only the inputs to each transformer block and recomputes activations within the block during the backward pass. Together, these enable fitting the 2B and 8B models on a single GPU.
For the larger 30B-A3B MoE model, the paper employs ZeRO-2 (which additionally partitions gradients across workers), expert parallelism (EP) with a world size of 2 (meaning the experts in the MoE layers are split across 2 GPUs, and tokens are dispatched to the appropriate expert's GPU), and grouped linear operations for MoE layers using NVIDIA CUTLASS kernel templates. Cross-GPU token dispatching for expert parallelism uses DeepEP (deepseekv3), a communication library optimized for the all-to-all communication patterns that arise in MoE inference and training. Additionally, to reduce memory consumption from the final output projection (which produces logits over the full vocabulary of ~150K tokens for every position in the sequence), the paper selectively filters out tokens that do not require loss calculation—such as multimodal tokens (visual tokens, special delimiter tokens)—during the forward pass of the output head.
General MLLM Pretraining Data (4.8M samples)
To ensure RynnBrain retains the broad multimodal understanding capabilities of its Qwen3-VL initialization, the pretraining corpus includes 4.8 million samples from general-purpose multimodal datasets spanning both images and videos. These include LLaVA-OV-SI (instruction-tuning data for open-vocabulary recognition), LLaVA-Video (video understanding conversations), ShareGPT-4o-video (GPT-4o generated video captions and QA), VideoGPT-plus (detailed video descriptions), FineVideo (fine-grained video understanding), CinePile (movie understanding), ActivityNet (human activity recognition), YouCook2 (cooking video understanding), LLaVA-SFT (image-text instruction data), and VideoLLaMA 3 (video-language alignment).
These datasets collectively support open-vocabulary object recognition, conversational video understanding, long-horizon temporal reasoning, and standard image-text alignment. Their inclusion acts as a form of catastrophic forgetting prevention: by mixing general MLLM data with embodied-specific data during pretraining, the model maintains its ability to understand diverse visual concepts and follow general instructions, rather than overfitting to the narrow distribution of embodied tasks.
Why this is necessary: A model trained exclusively on embodied data would likely lose its general visual understanding capabilities—for instance, the ability to recognize rare objects, understand abstract visual concepts, or follow complex multi-step instructions—because embodied datasets predominantly feature indoor scenes, common household objects, and task-oriented language. The general MLLM data provides a regularization effect, ensuring that the visual and language representations remain broadly competent while the embodied data adds the specialized spatial grounding capabilities.
Object Understanding Data (1.1M samples)
The object understanding dataset trains RynnBrain to perform fine-grained object recognition and reasoning about specific object instances. Each sample includes a video frame with a bounding box annotation identifying a particular object, formatted as <object><frame n>: (coordinates) </object>, paired with a question about the object's attributes (category, color, material, shape, state, position, function, surface detail, size). The model must answer based on the visual evidence within the specified bounding box at the specified frame.
The data construction pipeline for the egocentric subset (712K samples) works as follows: (1) Object detection: Qwen2.5-VL identifies objects in key frames of indoor videos. (2) Bounding box generation: Grounding DINO 1.5 predicts bounding boxes for each identified object. (3) Tracking: SAM2 segments and tracks each object across frames, enabling consistent object identity over time. (4) Redundancy reduction: To prevent the model from overfitting to frequently occurring object categories, each video is limited to at most two instances per object category—so a video of a kitchen might include at most two mugs, even if five appear. (5) QA generation: Qwen2.5-VL generates object-centric question-answer pairs conditioned on the bounding box, asking about specific attributes. (6) Manual filtering: Human annotators filter the generated QA pairs for quality, removing hallucinated attributes or questions that don't require visual inspection.
The remaining samples come from public datasets (RefCOCO, Google Refexp, Osprey-724K, DAM, VideoRefer-700k) that provide referential expressions and bounding box annotations.
Why structured as <object><frame n>: (coordinates) </object>: Explicitly encoding the frame index and bounding box coordinates in a structured format teaches the model to condition its responses on precise spatio-temporal references. When the model sees a question about "the color of this object" preceded by an <object> tag with coordinates, it learns to attend to the specified region in the specified frame rather than the whole image. This is critical downstream: when a planning model outputs "grasp the <object><frame 47>: (234, 512), (389, 601)</object>," the model has been trained to understand that the coordinates precisely identify which object is being referenced.
Spatial Understanding Data (2.5M samples)
Spatial reasoning—understanding 3D relationships, distances, relative positions, and object layouts—is identified as a critical weakness of existing VLMs. The paper curates 2.5 million spatial instruction samples in two categories:
General spatial understanding (from public datasets): Sensenova-SI-800K, VLM-3R, and VSI-590K provide generic spatial reasoning QA pairs covering relative positions ("to the left of"), depth ordering ("in front of"), and spatial relationships.
Fine-grained object-centric spatial reasoning (855K video + 272K image samples): This subset is constructed from self-collected indoor videos using a 3D reconstruction pipeline. MASt3R-SLAM processes RGB video to reconstruct 3D point clouds and estimate camera extrinsics (the position and orientation of the camera for each frame). Instance-level segmentations (from SAM2 or similar) are then projected into the reconstructed 3D space, assigning each pixel to a specific object instance in 3D. The point cloud is realigned using RANSAC to detect the ground plane and enforce a gravity-aligned world coordinate system—this corrects for any drift in the SLAM reconstruction and ensures that "up" is consistently defined across the scene.
Given these calibrated 3D scenes, the paper generates spatial QA pairs using template-based generation with computed ground-truth answers. Crucially, the answers are not generated by an LLM (which might hallucinate), but are computed directly from the 3D geometry. For example, a template might be "How far is the [object A] from the [object B]?" and the answer is the Euclidean distance between their 3D centroids, computed from the point cloud. Other templates cover relative heights ("Which is higher, [A] or [B]?"), spatial containment ("Is [A] inside [B]?"), and directional relationships.
Why 3D geometry is necessary: 2D spatial relationships in a single image are ambiguous—an object that appears to the left of another might actually be behind it but offset in the image plane. 3D reconstruction resolves this ambiguity by providing metric distances and true spatial relationships, enabling the model to learn genuine 3D spatial reasoning rather than 2D image-plane heuristics. The gravity alignment is particularly important for embodied tasks: a robot needs to understand "up" and "down" in world coordinates, not image coordinates, to know that a cup is on a table (supported from below) rather than merely appearing above it in the image.
Counting Data (264K samples)
The counting dataset trains RynnBrain to estimate object quantities in complex visual scenes, a capability that is surprisingly challenging for vision-language models. The public component consists of 222K samples from Molmo2's counting subset, which provides diverse scenes with reliable quantity annotations. The embodied component adds 42K counting QA pairs from self-collected egocentric videos, with all annotations manually verified.
Counting in embodied contexts is distinct from generic object counting because it often requires reasoning about task-relevant subsets ("how many clean plates are on the drying rack?") and dealing with occlusion, clutter, and varied viewing angles typical of egocentric video. The manual verification is important because automated counting from object detectors is unreliable in cluttered scenes—a detector might miss partially occluded objects or double-count objects that span detection boundaries.
OCR Data (1M samples)
Text recognition in embodied environments is critical for tasks like reading labels, instructions, or signage, yet is often overlooked in embodied benchmarks. The OCR dataset is constructed from egocentric videos sourced from Ego4D, Charades-Ego, and EPIC-KITCHENS, representing typical first-person scenarios where text appears.
The construction pipeline: (1) Text detection: GoMatching detects scene text in video frames. (2) Video segmentation: Videos are segmented into clips of 3 to 15 seconds based on text appearance patterns—when text appears, changes, or disappears, clip boundaries are inserted. This yields 85,324 text-containing segments. (3) Human annotation: For each segment, annotators label the first frame where the text appears, the frame where it is clearest (for optimal OCR), the text transcription, and bounding polygons around each text region. (4) QA generation via two complementary strategies:
-
GPT-5.2 contextual QA (256K samples): GPT-5.2 generates goal-oriented, first-person questions grounded in practical text understanding. For instance, given a video of someone in a kitchen with a recipe card visible, GPT-5.2 might generate "What temperature does the recipe say to preheat the oven to?" This tests whether the model can locate, read, and contextually interpret text in service of a task. Generated questions are filtered to ensure visual perception is required (i.e., the answer cannot be guessed without seeing the video).
-
Template-based QA (722K samples): Structured templates produce questions covering text reading ("What does the text on the box say?"), temporal localization ("In which frame does the 'EXIT' sign first appear?"), verification ("Is there a 'STAFF ONLY' sign visible?"), and multiple-choice recognition.
The OCR data is provided in two formats: normal video QA (893K samples) where the model predicts textual answers, and area prediction QA (85K samples) where the model outputs the frame index and bounding coordinates of the text—the latter being particularly important for embodied agents that need to physically locate and interact with text-bearing objects.
Why two formats: The area prediction format forces the model to ground its text recognition in spatial coordinates, which is necessary for downstream tasks like "go to the sign that says 'Conference Room B'" where a navigation policy needs the coordinates of the sign, not just its textual content. The normal QA format ensures the model can also extract semantic meaning from text, which is necessary for tasks like "read the instructions on the detergent bottle."
Egocentric Task Understanding Data (2.77M samples)
To provide broad coverage of egocentric task comprehension, the paper aggregates several public egocentric video QA datasets: Env-QA (environmental understanding from egocentric video), EgoTaskQA (task-oriented question answering), RoboVQA (robot-relevant visual QA), EgoRe-5M (egocentric reasoning), QAEgo4D (Ego4D-based QA), Robo2VLM (robot-to-VLM transfer), and ShareRobot (shared robot data). Videos shorter than 3 seconds are excluded to ensure sufficient temporal context for task-level reasoning—a 1-second clip of a hand reaching for an object provides insufficient context to determine what task is being performed.
This category serves as a catch-all for general egocentric understanding that does not fit into the more specialized object, spatial, counting, or OCR categories. It exposes the model to a diverse range of egocentric activities (cooking, assembly, social interaction, navigation) and task-oriented question types, contributing to the model's ability to understand first-person video and reason about human activities from an embodied perspective.
Object Location Data (1.2M samples)
Object localization—finding where a described object is in an image or video—is a core capability for embodied agents. Each sample is represented as $(\mathbf{V}, Q, \mathcal{B}, t)$, where $\mathbf{V} = \{I_t\}_{t=1}^T$ is the visual input (image if $T=1$, video if $T>1$), $Q$ is a textual query describing the target object (e.g., "the red mug on the counter"), $\mathcal{B} = \{(x_0, y_0, x_1, y_1)\}$ is the bounding box of the target with normalized coordinates in $[0, 1000]$, and $t$ is the index of the key frame where the object is most clearly visible.
The dataset aggregates 900K samples from public grounding datasets (ADE20K, COCOStuff, Mapillary, PACO-LVIS, PASCAL-Part, Visual Genome, RoboAfford++) and adds 300K egocentric samples constructed using the same segmentation pipeline as object understanding. Referring expressions for the egocentric subset are generated by Qwen3 and fall into two categories: simple expressions based on category or position ("the white cup on the table"), and situational expressions requiring task-level inference ("the object you would use to pour coffee"). All samples are manually filtered.
The inclusion of situational expressions is important: in real embodied tasks, objects are often referenced by their function or role in a task, not by their category alone. For example, "the thing I stir with" requires the model to reason about which object in the scene serves that function (a spoon), which is a more complex inference than category-based reference.
Area Location Data (3.37M samples)
Area localization addresses a distinct capability: identifying non-object regions such as surfaces, empty spaces, or functional areas. Unlike objects, which are typically represented by bounding boxes, areas are represented by sets of points: each sample is $(\mathbf{V}, Q, \mathcal{P}, t)$, where $\mathcal{P} = \{(x_i, y_i)\}_{i=1}^n$ is a set of normalized points indicating the target area (e.g., a set of points scattered across a table surface), and $t$ is the keyframe index.
This is the largest single localization category at 3.37M samples, constructed from multiple sources: 6K video segments with LLM-generated instructions and human-selected point annotations, 222K video samples from Molmo2-VideoPoint, 448K static image samples from indoor images, and 2.2M image samples from RoboAfford++ and RefSpatial.
Why point sets rather than bounding boxes: Areas are often irregularly shaped (table surfaces, countertops, floor regions) and cannot be accurately represented by axis-aligned rectangles. Point sets provide a flexible representation that can approximate arbitrary shapes—a dense set of points on a table surface effectively delineates the usable area for placing objects. For the model, predicting a set of points rather than bounding box corners requires learning to output variable-length sequences of coordinate pairs, which reinforces the general capability of spatial output generation.
Affordance Location Data (1.13M samples)
Affordance localization identifies actionable points on objects—where a robot should grasp, press, pull, or interact. Each sample is $(\mathbf{V}, Q, p, t)$, where $p = (x, y)$ is a single normalized point (e.g., the center of a handle, a button, or a door pull), and $t$ is the keyframe where the affordance is most relevant. Unlike area prediction which outputs a set of points, affordance prediction typically outputs a single precise location where interaction should occur.
The dataset follows a construction pipeline similar to area localization: 6K video segments with LLM-generated instructions and human-labeled points, 476K image-based samples from 500K indoor images, and 260K samples from RoboAfford++. The focus on "actionable points" is what distinguishes affordances from general object localization—the model must predict not just where an object is, but where on the object the interaction should occur, which requires understanding of object function and geometry.
Trajectory Location Data (563K samples)
Trajectory localization trains the model to predict the path a robot end-effector or manipulated object should follow. Each sample is $(\mathbf{V}, Q, \mathcal{T}, t_s)$, where $\mathcal{T} = \{(x_i, y_i)\}_{i=1}^m$ is an ordered sequence of up to 10 normalized trajectory points, and $t_s$ is the starting frame. For example, the query might be "show how to pour from the kettle into the cup," and the trajectory would trace a curve from the kettle's spout to the cup's opening.
The dataset comprises 6K video-based samples with human-annotated trajectories (emphasizing cross-frame reasoning—the trajectory must be consistent with object motion across frames), 507K image-based trajectory samples from indoor scenes, and 13K samples from the FSD dataset.
Why trajectories are limited to 2D: The trajectory points are predicted in 2D image coordinates, not 3D world coordinates. While 3D trajectories would be more directly useful for robot control, predicting 3D coordinates from a 2D image is an ill-posed problem (depth is ambiguous from monocular views). The 2D trajectories serve as a "grounded suggestion" that a downstream 3D-aware policy can lift into 3D using depth information. Additionally, 2D trajectories are sufficient for many tasks—when a human demonstrates "wipe the counter in an S-pattern," the 2D trajectory captures the spatial pattern even without depth.
Grasp Pose Location Data (1.3M samples)
Grasp pose prediction is a specialized form of spatial output oriented toward robotic manipulation: given an image and an object description, predict the 2D oriented rectangle representing where and at what angle the gripper should close. Each sample is $(I, Q, \mathcal{G})$, where $I$ is an RGB image, $Q$ is a query specifying the target object and grasping task, and $\mathcal{G} = \{(x_i, y_i)\}_{i=1}^4$ are the four corner points of the grasp rectangle.
The data comes from Grasp-Anything, which provides grasp annotations parameterized by center $(c_x, c_y)$, dimensions $(w, h)$, and rotation angle $\theta$. The paper converts this parameterization to four corner points via rotation, so that the model's output format is consistent with other spatial primitives (sequences of coordinate tokens). This avoids having a separate output head or format for grasp poses, maintaining the unified output space design principle.
From approximately 995K images, the paper constructs 1.3M training samples (some images contain multiple objects, each yielding a separate sample). Prompts are generated using a weighted template strategy: 40% object-centric ("Grasp the red cup"), 30% scene-aware ("On the cluttered desk, grasp the red cup"), and 30% task-oriented ("To prepare coffee, grasp the red cup"). This linguistic diversity ensures the model learns to interpret grasping instructions in varied contexts.
Why oriented rectangles: A grasp is defined not just by a position but by an orientation—the gripper must approach the object from the correct angle. Representing the grasp as four corner points (rather than center + angle) keeps the output format consistent with bounding boxes and enables the model to use the same coordinate token prediction mechanism. The tradeoff is that the four corners are not independent (they must form a rectangle), but the model implicitly learns this constraint from training data.
Physics-Aware Planning Data (160K samples)
The planning dataset teaches RynnBrain to decompose high-level tasks into grounded sub-tasks. Each sample is $(\mathbf{V}, Q, \mathcal{M})$, where $\mathbf{V}$ is the visual context preceding the current step, $Q$ is a high-level instruction (e.g., "Please help me tidy up the sink"), and $\mathcal{M}$ is the current sub-task plan expressed as a mixed sequence of text and grounding annotations.
The key design choice is that each sub-task includes explicit spatial grounding: target object bounding boxes $\mathcal{B}$, placement area points $\mathcal{P}$, and affordance points $p$. For example, a sub-task "Place the sponge in the holder" would include the bounding box of the sponge, the area points of the holder, and the affordance point for grasping the sponge. This means the planning output is directly consumable by a downstream manipulation policy—the policy receives not just "grasp the sponge" but "grasp the sponge at (234, 512)".
The data construction process: (1) Long-horizon tasks are decomposed into temporally ordered sub-tasks using an in-house model (likely a fine-tuned LLM), with human verification to ensure correctness. (2) Following the Hi Robot framework, atomic actions (e.g., grasp, place, push) are used as minimal planning units. (3) Spatial annotations are added on randomly sampled frames from public datasets (AgibotWorld Alpha, Open X-Embodiment) formatted as single-turn planning dialogues.
The relatively small size of the planning dataset (160K samples vs. millions for localization) reflects the difficulty of obtaining high-quality manipulation planning data with spatial annotations. The paper compensates by formatting the data as multi-turn dialogues during post-training (Section 5.2), enabling the model to maintain planning context across steps.
Chain-of-Point Reasoning: Cold-Start SFT Data Construction
Chain-of-Point (CoP) reasoning is RynnBrain's mechanism for physically grounded cognition. Unlike standard chain-of-thought (CoT) which consists entirely of text, CoP interleaves textual reasoning steps with explicit spatial grounding primitives. The training data for CoP is constructed through a multi-stage pipeline operating on the core spatio-temporal localization datasets (Area, Affordance, and Trajectory):
Stage 1: Reasoning chain generation. Given the original task instruction and video frames, Qwen3-VL-235B pre-generates a step-by-step textual reasoning chain. This chain walks through the logical steps needed to solve the task, and marks potential entities (objects or areas that need to be localized) using square brackets: "I need to find the [white flower-patterned wallpaper] and then point to the [empty space on the counter]." The bracketed phrases are candidates for visual grounding.
Stage 2: Entity classification. An in-house model classifies each bracketed entity as either "area" or "object" based on the textual context. This classification determines what type of spatial primitive will be used: areas receive point sets, objects receive bounding boxes.
Stage 3: Human annotation and grounding. Human annotators review the reasoning chain and entity classifications. For each identified entity, they select the most relevant and clear frame from the video sequence, then perform precise spatial annotation: bounding boxes for objects, representative point sets for areas. These grounding annotations are inserted back into the reasoning text using structured tags: <object><frame 47>: (234, 512), (389, 601)</object> replaces "[white flower-patterned wallpaper]" and <area><frame 52>: (100, 300), (150, 310), ...</area> replaces "[empty space on the counter]".
Result: Each training sample extends the standard localization tuple to $(V, Q, \mathcal{P}_{final}, t_s, R)$, where $R$ is the interleaved reasoning chain containing both text and grounding annotations. The model is trained to generate this entire chain—text and coordinates together—given the visual input and task instruction.
Why this matters: This data construction teaches the model that reasoning about physical tasks should be anchored to specific visual evidence at every step. Instead of reasoning abstractly ("I should find the handle and grasp it"), the model learns to produce reasoning that is verifiable against the video ("I should find the handle, which is at coordinates (234, 512) in frame 47, and grasp it at affordance point (310, 556)"). The structured interleaving ensures that spatial grounding is not an afterthought but an integral part of the reasoning process.
Chain-of-Point Reasoning: Cold-Start SFT Training
RynnBrain-CoP is initialized from the pretrained RynnBrain checkpoint and fine-tuned on the CoP reasoning data using full-parameter supervised fine-tuning. The training configuration: AdamW optimizer with cosine learning rate schedule, peak learning rate $1 \times 10^{-5}$ for the language model and projector, $2 \times 10^{-6}$ for the vision encoder (again lower to preserve pretrained visual features), 3% warmup period, 1 epoch of training, global batch size of 128.
To handle long egocentric videos with CoP reasoning traces (which can be substantially longer than simple QA because they include full reasoning chains plus coordinate sequences), frames are sampled at 2 FPS with up to 2048 frames, and the maximum context length is set to 16,384 tokens. DeepSpeed ZeRO-1 is used for memory efficiency. The 2 FPS sampling represents a tradeoff: higher frame rates would provide finer temporal resolution but exceed the context window; 2 FPS (capturing one frame every 0.5 seconds) is sufficient for most household manipulation tasks where motions occur on the timescale of seconds.
Chain-of-Point Reasoning: GRPO Reinforcement Learning
After cold-start SFT, RynnBrain-CoP undergoes reinforcement learning using Group Relative Policy Optimization (GRPO) to further align its outputs with physically grounded reasoning. GRPO is chosen over standard PPO because it eliminates the need for a separate value function (critic) model, which would double memory consumption.
The GRPO objective for a query $q$ and a group of $G$ sampled outputs $\{o_1, \ldots, o_G\}$ from the old policy $\pi_{\theta_{old}}$ is:
where $\rho_i = \frac{\pi_{\theta}(o_i|q)}{\pi_{\theta_{old}}(o_i|q)}$ is the importance sampling ratio (how much more likely output $i$ is under the new policy versus the old policy), $A_i$ is the advantage of output $i$ (how much better it is than the group average), $\epsilon$ controls the clipping range, $\beta$ controls the KL penalty coefficient that prevents the policy from deviating too far from the reference model $\pi_{\text{ref}}$, and $\mathbb{D}_{KL}$ is the Kullback-Leibler divergence.
What it computes: For each query, the model generates $G = 5$ candidate outputs. Each output receives a reward from the task-specific reward functions (described below). The advantage $A_i$ for output $i$ is computed by normalizing rewards within the group:
This normalization means that advantages are zero-centered and scaled by the group's standard deviation—outputs better than average get positive advantages, outputs worse than average get negative advantages, and the magnitude depends on how much better or worse they are relative to the group's spread.
The clipped surrogate objective $\min(\rho_i A_i, \text{clip}(\rho_i, 1-\epsilon, 1+\epsilon)A_i)$ is the standard PPO clip: it prevents the policy from changing too much in a single update by clipping the importance ratio $\rho_i$ to $[1-\epsilon, 1+\epsilon]$ (with $\epsilon$ set to 0.2–0.28). The KL penalty $\beta \mathbb{D}_{KL}$ provides an additional constraint, regularizing the policy toward the reference model to prevent reward hacking or catastrophic forgetting.
Why GRPO over PPO: Standard PPO requires learning a value function $V(s)$ alongside the policy to estimate advantages ($A_i = r_i + \gamma V(s') - V(s)$). This value function is typically a neural network of comparable size to the policy, which doubles GPU memory consumption. GRPO instead estimates the baseline from the group mean of multiple samples, avoiding the need for a separate value model entirely. This is particularly important for large vision-language models where memory is already constrained. However, the group-based baseline is higher-variance than a learned value function (because it averages over only 5 samples rather than a learned expectation), which is why the KL penalty $\beta$ is important—it provides additional regularization to compensate for the noisier advantage estimates.
Training configuration: initialized from the cold-start SFT model, SGLang inference engine for efficient rollout generation (SGLang optimizes LLM serving throughput for batched generation), group size $G = 5$, 10 epochs, batch size 128, cosine learning rate schedule starting at $2 \times 10^{-6}$ with 3% warmup, clipping range $\epsilon \in [0.2, 0.28]$, KL coefficient $\beta = 0.02$, maximum sequence length 16,384 tokens.
Chain-of-Point Reasoning: Reward Design
The GRPO training uses task-specific rule-based reward functions that evaluate how well the model's spatial outputs match the ground truth. All coordinates are normalized to $[0, 1]$ before reward computation (a different normalization than the $[0, 1000]$ used for tokenization, likely to make distances scale-invariant).
Trajectory Reward:
The trajectory reward evaluates both the shape and sequential alignment of predicted paths. The predicted sequence $\mathcal{P} = (p_1, \ldots, p_M)$ and ground truth $\mathcal{G} = (g_1, \ldots, g_N)$ are first resampled to have the same number of points uniformly spaced by arc length—this ensures that the comparison is scale-invariant and that temporal misalignment (the prediction being slightly ahead or behind the ground truth) does not unfairly penalize shape similarity.
The Discrete Fréchet Distance (DFD) between the resampled sequences is computed. Formally, let $c(i, j)$ be the coupling distance between prefixes $p_{1:i}$ and $g_{1:j}$, defined recursively:
with $c(0,0) = \|p_1 - g_1\|_2$. The final DFD is $D_F = c(M, N)$.
What it computes: The Fréchet distance is often described as the minimum leash length needed for a person and a dog to walk along their respective paths—it captures the maximum spatial separation between the two trajectories when both are traversed monotonically from start to end. The recursive definition means that at each step, the algorithm considers advancing along either trajectory or both, and keeps the maximum Euclidean distance encountered along the coupling. The min over coupling choices finds the tightest possible alignment.
The reward is an exponential decay with distance:
When $D_F = 0$ (perfect match), the reward is 1. As $D_F$ increases, the reward decays exponentially toward 0.
Why Fréchet distance over MSE: Mean squared error between corresponding points assumes a known temporal alignment (which point in the prediction corresponds to which point in the ground truth). For trajectories, this alignment is not known a priori—the model might predict the correct shape but at a different speed. Fréchet distance simultaneously solves for the optimal temporal alignment and the spatial error, making it a more principled metric for trajectory similarity. The exponential reward shaping provides smooth gradients (as opposed to a hard threshold) and naturally saturates for very poor predictions.
Affordance Reward:
For affordance prediction, the model outputs a set of predicted interaction points $\mathcal{P}$ and is evaluated against ground truth points $\mathcal{G}$ using the Bidirectional Mean Euclidean Distance, a symmetric variant of Chamfer distance:
The reward is: $r_{\text{aff}} = \exp(-\lambda_{\text{aff}} \cdot D_{\text{bidir}})$
What it computes: The first term (precision term) averages the distance from each predicted point to its nearest ground truth point—this penalizes predictions that are far from any valid affordance. The second term (recall term) averages the distance from each ground truth point to its nearest predicted point—this penalizes predictions that miss some of the annotated affordance regions. The average of the two terms gives a balanced metric that captures both false positives (predicted points in the wrong place) and false negatives (missed affordance regions).
Why bidirectional: Unidirectional Chamfer distance (predictions to ground truth only) would reward the model for predicting a single point at the centroid of all affordance regions—this would achieve low distance but misses the coverage goal. The bidirectional formulation forces the model to cover all annotated affordance regions, which is important when multiple interaction points are equally valid (e.g., you can grasp a handle anywhere along its length).
Area Reward:
For area identification, the model predicts a set of points $\mathcal{P}$ and the reward is the fraction of predicted points that fall within the ground truth polygon $S_{\mathcal{G}}$:
where $\mathbb{I}(\cdot)$ is the indicator function (1 if the point is inside the polygon, 0 otherwise).
What it computes: A strict accuracy metric: each predicted point is either inside the valid area or not. The reward is simply the proportion of points that are correctly placed. This is appropriate for area tasks because the goal is to identify the usable region, and any point outside the region is an error regardless of how close it is to the boundary (a point 1 pixel outside a table edge is functionally equivalent to a point 100 pixels outside—both would cause a robot to miss the table).
Why indicator rather than distance: For area tasks, being "close" is not good enough—placing an object 1 cm off the edge of a table will cause it to fall, just as placing it 1 m off would. The binary in/out metric directly captures the functional requirement: all predicted points must be within the valid area. The tradeoff is that this reward is non-smooth (a small perturbation can change a point from inside to outside), which can make optimization harder. The cold-start SFT initialization likely provides a good enough starting point that the policy does not need to navigate this non-smoothness from scratch.
Chain-of-Point Reasoning: RL Data Filtering
The RL training dataset is a curated subset of 30K samples drawn from the larger spatio-temporal localization data. The filtering strategy is difficulty-aware: samples that are too easy (the SFT model already gets near-perfect scores) provide no learning signal because the advantage is always near zero (all outputs in the group already achieve high reward). Samples that are too hard or noisy provide unreliable reward signals that can destabilize training.
Each candidate sample is scored by the pretrained SFT model using the same evaluation metrics described above, and only samples with intermediate difficulty (scores between 40 and 80, where 100 would be perfect) are retained. Additionally, a subset of failure cases where the SFT model incorrectly selects key frames is included to specifically improve temporal localization—these are cases where the model's spatial prediction might be accurate but applied to the wrong frame.
Why difficulty filtering: In RL, learning progress requires a meaningful spread of advantages within each group. If all outputs in a group score near 100 (easy samples), the advantages are near zero (all outputs are similarly good), and the policy receives no gradient signal. If all outputs score near 0 (impossible samples), the policy also receives no signal because no output is meaningfully better than any other; the model has no positive examples to reinforce. Intermediate difficulty provides the gradient signal needed for policy improvement: some outputs are better than others, and the model can learn from the difference.
Vision-Language Navigation: RynnBrain-Nav
The VLN post-training adapts RynnBrain from a general embodied understanding model to a navigation agent that produces discrete actions. The task is formalized as: at time $t$, given the history of visual observations $O = \{o_0, \ldots, o_t\}$ (each $o_i$ is an RGB image from the agent's perspective) and a natural language instruction $Q$, predict the next action $a_t \in \mathcal{A} = \{\uparrow, \leftarrow, \rightarrow, \text{STOP}\}$ representing moving forward 30 cm, turning left 15 degrees, turning right 15 degrees, or halting.
The training data is organized as interleaved observation-action sequences: $\{o_0, a_0, o_1, a_1, \ldots, o_n, a_n\}$, formatted as a multi-turn conversation following StreamVLN's methodology. The training objective is to predict $a_i$ given $o_i$ and all preceding conversation history.
Data collection: 450K video clips from R2R, R2R-EnvDrop, and RxR trajectories across 60 Matterport3D environments, augmented with 300K samples from ScaleVLN. Multi-turn DAgger (Dataset Aggregation) is used to collect additional on-policy data: the current model is deployed to generate actions, a human (or oracle) corrects any mistakes, and the corrected trajectories are added to the training set. This iterative process reduces the distribution shift between training (where the model sees ground-truth observations) and inference (where it sees its own generated observations, which may diverge from ground truth).
Fine-tuning: Full-parameter supervised fine-tuning with AdamW, cosine schedule, peak learning rate $2 \times 10^{-5}$ for LLM and projector, $2 \times 10^{-6}$ for vision encoder, 3% warmup, 1 epoch, global batch size 256, frames sampled at 2 FPS (up to 2048 frames), max context length 16,384 tokens, DeepSpeed ZeRO-1.
Why this adaptation works: The pretrained RynnBrain already understands egocentric video, can recognize objects and spatial relationships, and can follow instructions. The VLN fine-tuning primarily teaches it to map this understanding onto the discrete action space of the navigation task—essentially learning the interface between semantic understanding and low-level movement. The broad pretraining means the model generalizes to novel environments and instructions better than a model trained from scratch on navigation data alone.
Manipulation Planning: RynnBrain-Plan
RynnBrain-Plan adapts the foundation model for long-horizon manipulation planning by leveraging the planning-centric data already present in the pretraining corpus. The key challenge is maintaining effective memory across many planning steps: as the plan unfolds, the model must remember what it has already done, what objects have been moved, and what sub-tasks remain.
The solution is to format planning as a multi-turn dialogue, where each turn corresponds to one planning step and the conversation history serves as an explicit memory buffer. Grounding annotations (bounding boxes, area points, affordance points) are applied only to the final frame of each dialogue turn, ensuring that current decisions are conditioned on both the immediate observation and the accumulated planning history.
A striking finding is the data efficiency: fine-tuning with only a few hundred samples (a tiny in-house dataset) is sufficient to endow the model with robust long-horizon planning. This is possible because the pretraining already provides the core capabilities (object recognition, spatial grounding, task decomposition)—the fine-tuning primarily teaches the conversational format and memory maintenance behavior.
Vision-Language-Action: RynnBrain-VLA
RynnBrain-VLA closes the loop from planning to physical execution by adapting the 2B model to directly predict robot actions. The architecture uses a flow matching framework following $\pi_0$: the VLM backbone serves as a single-stream Diffusion Transformer (DiT) that takes a packed sequence containing the visual condition (images, task instruction, spatial grounding information) and noisy actions, and predicts the denoised action chunk for the current step.
Three linear projections are added to align the dimensionality of (1) input noise, (2) timestamp embeddings, and (3) output actions to the VLM's hidden size. The model preserves the native conversation format: instructions include pointing information in text-based format (e.g., <affordance> (x, y) </affordance> and <object> (x0, y0), (x1, y1) </object>), with the task's initial frame prepended. Actions are positioned at the end of the sequence to enable KV-cache reuse during inference, following $\pi_0$'s design.
Fine-tuning: A dataset of six pick-and-place tasks with three distinct objects, collected via manual teleoperation on a Franka Emika arm, is annotated with target objects and aligned to the planning model's output format. The 2B model is fine-tuned for 60K steps with learning rate $2 \times 10^{-5}$ and batch size 32, with images proportionally resized to a short-side dimension of 384 pixels.
Why RynnBrain-2B for VLA: The 2B model is chosen for its low inference latency—critical for real-time robot control where decisions must be made at 10–50 Hz. The larger models (8B, 30B) would provide better understanding but would be too slow for closed-loop control. The finding that even the 2B model achieves reliable grasping indicates that the pretraining on fine-grained spatial grounding (trajectories, affordances, grasp poses) transfers effectively to physical execution, even though the pretraining was embodiment-agnostic (it never saw the specific robot arm or gripper used in the VLA fine-tuning).
4. Key Insights and Innovations
Innovation 1: Moving Embodied Reasoning from "Abstract Thought" to "Physically Anchored Trace"
The fundamental conceptual move in this paper is the diagnosis that existing vision-language models fail at embodied tasks not because they lack knowledge, but because their reasoning occurs in a spatially unmoored abstraction layer. A model might correctly chain "I need the handle → handles are for grasping → grasp from the side," but without explicit, verifiable spatial anchors at each step, this trace is indistinguishable from hallucination — the model has no mechanism to check whether the handle it is reasoning about actually exists in the current egocentric view, or whether its proposed grasp point corresponds to empty space.
The field's default assumption, encoded in approaches like Video-R1, DeepSeek-VL, and MMR1 (leng2025mmr1), has been that reasoning is a purely textual process — chain-of-thought is a string of words. Auxiliary visual mechanisms like TACO's region zooming or DeepEyes' visual inspection tools augment the input to reasoning but leave the reasoning trace itself textual. The alternative of "visual imagination" (PARM, VPRL, GoT-R1) actually makes the problem worse by introducing hallucinated spatial content that can't be verified against the environment.
RynnBrain's Chain-of-Point redefines the reasoning trace as a first-class integration of language and spatial coordinates. A CoP trace isn't text with coordinates appended — it's a structure where each logical step is jointly expressed as a natural language clause and its corresponding spatial grounding, connected by XML-like delimiters that make the binding explicit and machine-parseable. The significance is that reasoning becomes falsifiable: every claim about an object's location or an interaction point can be compared against the actual video. This transforms reasoning from an act of generation to an act of grounded inference, where the model must produce evidence for its claims as it makes them.
The evidence that this matters comes from Section 6's evaluation and the CoP-specific improvements (approximately 7% on trajectory prediction tasks). But the deeper claim is not about percentage points — it's about a new interface between cognition and perception. Prior work treated these as separate modules (a reasoner that outputs text, a localizer that outputs coordinates), which creates a brittle interface where errors in one module cannot be corrected by the other. CoP makes spatial grounding an integral part of the reasoning process, so that a failure to localize correctly is a failure of reasoning, not just a downstream perception error. This reframing has implications beyond this paper: it suggests that any embodied AI system that separates "thinking" from "seeing" is architecturally flawed, and that the right unit of cognition is the grounded reasoning step, not the abstract thought.
Innovation 2: The Unified Token Approach as a Distributional Hack That Eliminates Multi-Task Optimization
At first glance, representing bounding boxes as integer tokens in [0, 1000] and training them with a standard language modeling loss looks like an implementation trick — a convenient way to repurpose existing transformer training infrastructure. But this characterization misses the deeper insight: this representation converts a hard multi-task optimization problem into a single-task classification problem, and in doing so avoids a class of failure modes that plague embodied AI systems.
The standard approach to spatial prediction in vision-language models is to add a separate regression head that predicts continuous coordinates, trained with an L2 or smooth-L1 loss alongside the language modeling loss. This creates a multi-task optimization landscape where the model must balance gradients from two objectives with different scales, different convergence rates, and potentially conflicting requirements on the shared representation. The well-known difficulty of tuning loss weights in multi-task learning means that practitioners often end up with systems that are good at either language or localization, but not both simultaneously.
By encoding coordinates as atomic tokens in the same vocabulary as words, RynnBrain eliminates the need for a separate output head, a separate loss function, and any loss-weight tuning. The gradient that teaches the model to predict "the" at position is structurally identical to the gradient that teaches it to predict "512" at position — both come from the same softmax cross-entropy over the same vocabulary. What makes this more than an implementation convenience is that it forces the model's internal representations to serve both purposes through a single bottleneck: the hidden state that predicts the next token must encode information that is simultaneously useful for language generation and spatial localization. This shared representation is what enables the model to naturally interleave text and coordinates without mode collapse or catastrophic interference.
The significance is practical but fundamental: it suggests that many "modality fusion" problems in embodied AI might be soluble not through carefully engineered multi-objective training schemes, but through a unifying tokenization scheme that recasts heterogeneous outputs as a single sequence prediction problem. The approach has limits — it requires discretizing continuous quantities, which introduces quantization error — but the paper's choice of [0, 1000] normalization provides sub-pixel precision at typical image resolutions, making the tradeoff favorable for most embodied tasks. This insight is likely to influence subsequent work on unified architectures for embodied AI, particularly as models expand to handle more output modalities (3D coordinates, joint angles, force commands) that currently require separate prediction heads.
Innovation 3: Difficulty-Aware Reinforcement Learning for Spatial Reasoning — A Negative Result in Disguise
The GRPO fine-tuning procedure for CoP reasoning contains a design choice whose significance is easy to overlook: the difficulty-aware filtering of the RL training data. The paper retains only samples where the SFT model achieves intermediate scores (40–80 out of 100), discarding both too-easy samples (where all outputs already score near-perfect) and too-hard samples (where no output achieves meaningful reward). This is positioned as a data efficiency measure, but it actually encodes a deeper insight about the limits of RL for spatial reasoning.
The standard assumption in RL fine-tuning of language models (RLHF, DPO, etc.) is that more data is better — you collect a large preference dataset and let the reward model or policy gradient sort out which examples are informative. The difficulty-filtering strategy in RynnBrain challenges this assumption by demonstrating that for spatially grounded tasks, the distribution of helpful learning signals is sharply concentrated in an intermediate difficulty regime. Easy samples provide no gradient (all group members have near-identical rewards, so advantages are ~0). Hard samples provide misleading gradients (the reward signal is dominated by noise, and the policy learns spurious correlations rather than genuine improvements).
This is more than a data curation trick — it's an empirical finding about the structure of spatial reasoning tasks that distinguishes them from purely textual reasoning. Textual chain-of-thought can often be improved on "easy" samples by making the reasoning more detailed or more rigorous, providing a meaningful gradient even when the final answer is correct. But spatial outputs (coordinates, trajectories, affordance points) have a crisp correctness criterion: a point is either on the handle or it isn't. There is no "more correct" version of a correct affordance point to optimize toward, and no informative gradient from a completely wrong trajectory. The learning signal is concentrated in the intermediate regime where the model is approximately right but not precise — close enough to the correct answer that small improvements produce meaningful reward changes, but far enough that there is room to improve.
This finding has negative implications that are as important as the positive results: it suggests that RL fine-tuning for embodied spatial reasoning has a fundamental ceiling determined by the fraction of "intermediate difficulty" samples in the training distribution. If the SFT model is already highly accurate on most tasks, RL will have almost nothing to learn from. If the SFT model is poor, RL will have almost no reliable signal to learn from. The effectiveness of RL is contingent on the SFT model landing in the right performance regime, which is not guaranteed for novel tasks or environments. This explains why the RL improvement is relatively modest (~7% on trajectories) despite the sophistication of the reward design — the window of learnable improvement is intrinsically narrow for spatial reasoning tasks with crisp correctness criteria.
Innovation 4: Embodiment-Agnostic Pretraining as Genuine Transfer — Not Just Representation Learning
The post-training results for RynnBrain-Nav, RynnBrain-Plan, and RynnBrain-VLA collectively make a claim that goes beyond standard transfer learning: that embodiment-agnostic pretraining on spatial grounding tasks transfers capabilities that are qualitatively different from what navigation- or manipulation-specific pretraining provides. This is not the standard "pretrain on broad data, fine-tune on narrow data" story — it's a claim about what kind of knowledge is being transferred.
Standard transfer learning from VLMs to embodied tasks typically transfers semantic knowledge (what objects look like, what words mean) and basic visual reasoning. RynnBrain's pretraining adds several capabilities that are specifically valuable for downstream embodiment: (a) the ability to output spatial coordinates in a structured format, (b) the understanding that text and coordinates can be interleaved to form coherent instructions, and (c) the ability to reason about 3D spatial relationships from 2D egocentric views. When RynnBrain-Nav achieves state-of-the-art results on R2R and RxR, the improvement comes not just from "better visual features" but from the model already understanding that navigation instructions should be grounded in spatial awareness — the fine-tuning teaches it which spatial outputs to produce, not how to produce spatial outputs.
The most striking evidence is the VLA result: the 2B model, fine-tuned on only six pick-and-place tasks with three objects, achieves reliable grasping that consistently outperforms models fine-tuned from π₀.₅. Six tasks is a trivially small fine-tuning dataset by modern standards — it would be completely insufficient to teach a model object recognition, spatial localization, or grasp affordance prediction from scratch. The fact that it works implies that these capabilities were genuinely learned during the embodiment-agnostic pretraining phase, and the fine-tuning only teaches the model to map its existing spatial reasoning onto the specific action space of the Franka arm.
This has significant implications for how the field thinks about embodied AI training. The dominant paradigm has been to train models on embodiment-specific data — robot trajectories, manipulation demonstrations, navigation episodes — which is expensive to collect and doesn't generalize across robots. RynnBrain's results suggest an alternative: invest heavily in a single, embodiment-agnostic foundation model that learns general spatial reasoning and grounding, then cheaply adapt it to specific embodiments with minimal fine-tuning data. If this result generalizes to other robots, tasks, and environments, it could substantially reduce the data burden for deploying AI on novel robotic platforms.
The limitation, which the paper is honest about, is that the VLA evaluation is small-scale (six tasks, one robot arm, lab setting). Whether the transfer holds for dexterous manipulation, dynamic tasks, or contact-rich interactions remains an open question. But even as an existence proof, the result challenges the assumption that effective robot policies require robot-specific training data at scale.
5. Experimental Analysis
Evaluation Methodology
-
Dataset. RynnBrain is evaluated on a total of 28 benchmarks: 20 embodied benchmarks spanning object cognition, spatial understanding, grounding, pointing, navigation, and planning, plus 8 general vision understanding benchmarks. The paper also introduces RynnBrain-Bench, a custom evaluation suite comprising 3,616 video clips (577,998 frames) and 12,000 manually verified open-ended questions covering four evaluation dimensions (Object Cognition, Spatial Cognition, Grounding, Pointing) and 21 specialized sub-capabilities. The benchmark is constructed from self-collected egocentric indoor videos with rigorous multi-round human annotation and cross-validation, and includes internal data balancing across sub-capabilities to ensure evaluation fairness. For navigation, standard benchmarks R2R (anderson2018vision) and RxR (ku2020room) are used. For general visual understanding, benchmarks include AI2D, ChartQA, DocVQA, MVBench, RealWorldQA, InfoVQA, EgoSchema, and VideoMME (without subtitles).
-
Base model(s). The RynnBrain family comprises three foundation model scales initialized from Qwen3-VL-Instruct checkpoints: RynnBrain-2B (2 billion parameters, dense), RynnBrain-8B (8 billion parameters, dense), and RynnBrain-30B-A3B (30 billion total parameters with 3 billion active via mixture-of-experts). Four post-trained variants are evaluated: RynnBrain-CoP (Chain-of-Point reasoning, fine-tuned from pretrained RynnBrain with SFT + GRPO reinforcement learning), RynnBrain-Nav (vision-language navigation, fine-tuned on VLN data with DAgger), RynnBrain-Plan (manipulation planning via multi-turn dialogue fine-tuning), and RynnBrain-VLA (vision-language-action, fine-tuned from RynnBrain-2B on 60K steps of pick-and-place data). The 2B scale is specifically chosen for VLA to maintain low inference latency for real-time control.
-
Metrics. The paper employs a diverse set of task-specific metrics based on output type. For textual QA (object cognition, spatial understanding, egocentric task understanding): responses are scored by GPT-4o on a scale from 0 to 1, using either binary scoring or a multi-level system with 0.2-point increments. For numerical spatial questions: Mean Relative Accuracy (MRA) and Rotational Accuracy (RoA) following RynnEC (rynnec). For object grounding: Acc@0.5, defined as the indicator that the model selects a valid key frame (ground truth exists) AND the IoU between predicted and ground truth bounding box exceeds 0.5 — formally,
Acc@0.5 = 1ifG_t ≠ ∅andIoU(B, G_t) > 0.5. For trajectory prediction: Discrete Fréchet Distance (DFD) after resampling both predicted and ground truth sequences to 15 uniformly spaced points, with reward computed asexp(-λ_traj · D_F). For area prediction: proportion of predicted points falling within the ground truth polygon (strict point-in-polygon accuracy). For affordance prediction: exponential decay of the bidirectional mean Euclidean distance (Chamfer variant) between predicted and ground truth point sets. For general vision benchmarks: standard task-specific metrics (accuracy, exact match, etc.) as defined by each benchmark. For navigation: success rate (reaching the target within threshold distance) and SPL (Success weighted by inverse Path Length) on R2R and RxR. For VLA: grasping success rate. -
Baselines. RynnBrain is compared against several existing embodied foundation models evaluated at comparable parameter scales. For models below 8B parameters (Table 3): MiMo-Embodied (7B), RoboBrain 2.0 (7B; robobrain2.0), Pelican-VL (7B; pelican-VL), Cosmos-Reason2 (8B; azzolini2025cosmos), and Qwen3-VL (8B; qwen3vl). For models above 30B parameters (Table 4): RoboBrain 2.0 (72B and 32B), MiMo-Embodied (72B), Cosmos-Reason2 (72B), and Qwen3-VL (30B-A3B and 235B). Note that Qwen3-VL serves as both the initialization base for RynnBrain and as an independent baseline representing what the base VLM achieves without embodied-specific pretraining. For VLA experiments, the baseline is models fine-tuned from π₀.₅ (pi0). For the RynnBrain-Bench evaluation, all models are evaluated using the same protocol: open-ended generation with GPT-4o scoring. Where results are marked with
*in the tables, they were obtained from the authors' own reproduction rather than reported by the original model developers. -
Generation budget / compute accounting. The paper does not use a generation budget or compute-matching protocol in the main evaluation tables — comparisons are made at the model level (each model evaluated under its standard inference protocol) rather than at equal inference FLOPs. This differs from the reference example paper (which matched FLOPs between pretraining and test-time compute). The post-training experiments (VLA, Nav, Plan) use task-specific evaluation protocols: navigation uses success rate and SPL on standard benchmark test splits; VLA uses physical grasping success rate on a Franka Emika arm. There is no explicit FLOPs or latency matching between RynnBrain and baselines of different scales.
-
Cross-validation / statistical protocol. For RynnBrain-Bench, annotations are cross-validated by human annotators across multiple rounds to ensure correctness and quality. Internal data balancing is performed across sub-capabilities within each evaluation dimension, and object-related questions are balanced against real-world object distributions for authenticity. For navigation benchmarks, standard test splits of R2R and RxR are used. For the RL training data (Section 4.2.3), samples are filtered by difficulty based on SFT model scores (only scores between 40 and 80 out of 100 are retained), but this is a training data curation protocol, not an evaluation protocol. The paper does not report confidence intervals, standard deviations, or statistical significance tests for any results.
Main Quantitative Results
Embodied Cognition Benchmarks (Tables 3 and 4)
The most comprehensive comparison is presented in Tables 3 and 4, which evaluate all models across multiple embodied cognition and general vision understanding benchmarks. I will analyze these results grouped by capability dimension.
Object cognition and egocentric understanding. On the in-house RynnBrain-Object benchmark, RynnBrain-2B achieves 70.7 and RynnBrain-8B achieves 71.2, while the next best model (Qwen3-VL-8B) achieves only 41.8 — a gap of nearly 30 percentage points. MiMo-Embodied-7B scores 39.0, RoboBrain 2.0-7B scores 24.7, and Pelican-VL-7B scores 30.8. This stark difference suggests that the object-centric QA data construction pipeline (712K egocentric samples with bounding-box-conditioned questions) provides capabilities that general VLM training does not. On MindCube (a benchmark for 3D spatial reasoning from egocentric views), RynnBrain-8B scores 56.6 versus Qwen3-VL-8B at 36.0, with MiMo at 43.1 and RoboBrain 2.0 at 38.9. The margin here (20.6 points over the base model) indicates substantial improvement from the spatial understanding data.
Spatial reasoning. On VSI-Bench (visual spatial intelligence), RynnBrain-2B achieves 70.5 and RynnBrain-8B achieves 71.0, compared to Qwen3-VL-8B at 60.3, Cosmos-Reason2-8B at 53.7*, and RoboBrain 2.0-7B at 36.1. On MMSI (multi-modal spatial intelligence), RynnBrain-8B reaches 39.6 versus Qwen3-VL-8B at 29.6 — a 10-point advantage. On RoboSpatial, RynnBrain-8B scores 73.1 versus the next best (MiMo) at 61.8. On the in-house RynnBrain-Spatial benchmark, RynnBrain-8B achieves 59.9 versus Qwen3-VL-8B at 35.0 — a 24.9-point gap that directly measures the contribution of the 2.5M spatial instruction samples, particularly the 3D-reconstructed spatial QA data.
Egocentric task QA and OCR. On EgoTaskQA, RynnBrain-2B achieves 73.9 and RynnBrain-8B achieves 72.5, slightly behind MiMo-Embodied-7B (unreported in Table 3 for this metric, but Figure 3 shows RynnBrain-8B at 72.5 and MiMo at 58.7*). On EgoTextVQA (indoor subset), RynnBrain-8B scores 31.6 for accuracy and 2.28 for a secondary metric (likely verbatim text match), compared to Qwen3-VL-8B at 38.9/2.64* — notably, the base Qwen3-VL outperforms RynnBrain on this benchmark, suggesting that the OCR data construction (1M samples from egocentric video) may not have captured the specific distribution of EgoTextVQA's indoor questions, or that the base model's general OCR capabilities were partially degraded during embodied pretraining (a potential case of catastrophic forgetting on text-heavy tasks). On Open-X VQA, RynnBrain-8B achieves 74.0 versus Qwen3-VL-8B at 59.8, Cosmos-Reason2 at 55.0*, and Pelican-VL at 44.6.
A pattern across cognition benchmarks. The results consistently show RynnBrain-2B and RynnBrain-8B outperforming all baselines on embodied cognition tasks, with particularly large margins on benchmarks that directly test capabilities emphasized in the pretraining data (object-centric QA, spatial reasoning, egocentric understanding). The relative ordering of baselines is not consistent — MiMo-Embodied leads on some tasks, Cosmos-Reason2 on others — suggesting that these models specialize differently depending on their training data mixtures. RynnBrain's consistent top or near-top performance across all categories supports the claim that the diverse pretraining corpus provides broad capability coverage.
Embodied Localization Benchmarks (Tables 3 and 4)
Localization — the ability to output precise spatial coordinates — is RynnBrain's most distinctive capability relative to standard VLMs. The evaluation here is particularly informative because it tests the core design choice of unified coordinate tokens.
Object grounding and area localization. On the in-house RynnBrain-Grounding benchmark (which uses Acc@0.5), RynnBrain-2B achieves 79.1 and RynnBrain-8B achieves 81.6, compared to Qwen3-VL-8B at 62.8, Cosmos-Reason2-8B at 60.0, MiMo-Embodied-7B at 49.8, and RoboBrain 2.0-7B at only 18.6. The 62.8 for Qwen3-VL indicates that the base model already has some localization capability (likely from its general VLM training), but the nearly 19-point improvement demonstrates that the 1.2M object location samples substantially sharpen this ability. On RynnBrain-Area, RynnBrain-8B scores 56.2 versus Qwen3-VL-8B at 30.0 — a 26.2-point gap that reflects the 3.37M area localization samples, by far the largest single localization category. Pelican-VL-7B (46.5) and MiMo-Embodied-7B (49.4) perform notably better on area tasks than on object grounding, suggesting area prediction may be an easier spatial reasoning format or that these models' training emphasized area-like spatial outputs.
Affordance and trajectory prediction. On RynnBrain-Affordance, RynnBrain-8B achieves 90.4 versus Qwen3-VL-8B at 82.9 — a 7.5-point improvement. The high absolute scores across all models (even Qwen3-VL at 82.9) suggest that affordance prediction from static images is relatively tractable for general VLMs. On RynnBrain-Trajectory, RynnBrain-2B achieves a DFD (lower is better, indicated by ↓ in the table) of 0.34 and RynnBrain-8B achieves 0.35, compared to Cosmos-Reason2-8B at 0.36*, Qwen3-VL-8B at 0.37, and RoboBrain 2.0-7B at 0.55 — the latter being substantially worse, indicating that trajectory prediction across video frames is particularly challenging without spatio-temporal pretraining. On ShareRobot-Affordance, RynnBrain-8B scores 44.7 versus Qwen3-VL-8B at 37.0, with Cosmos-Reason2 at 37.1* and Pelican-VL at 11.3. On ShareRobot-Trajectory, RynnBrain-8B achieves 0.35 (DFD) versus RoboBrain 2.0-7B at 0.55.
Grasp pose prediction. On Cornell-Grasp, RynnBrain-8B achieves 26.6 versus Qwen3-VL-8B at 21.2* and Cosmos-Reason2-8B at 18.1*. On VMRD-Grasp, RynnBrain-8B scores 14.1 versus Cosmos-Reason2-8B at 13.7*. The absolute scores here are notably low across all models — even the best model gets grasp pose prediction correct only about 27% of the time on Cornell-Grasp — indicating that predicting oriented grasp rectangles from single RGB images is inherently difficult. The gap between RynnBrain and baselines is modest (5.4 points on Cornell-Grasp), suggesting that the 1.3M grasp pose samples provide some benefit but do not dramatically solve this challenging problem.
RefSpatial-Bench. RynnBrain-8B achieves 59.2 versus Qwen3-VL-8B at 53.4 and MiMo-Embodied-7B at 48.0. RefSpatial tests spatial referring expressions, and the 5.8-point gap over Qwen3-VL indicates meaningful improvement from the pretraining, though the base model already performs reasonably well.
Summary of localization results. Across all localization benchmarks, RynnBrain-2B and RynnBrain-8B consistently outperform all baselines. The margins are largest on object grounding (Acc@0.5 of 81.6 vs. 62.8 for Qwen3-VL, a 30% relative improvement) and area localization (56.2 vs. 30.0, an 87% relative improvement), and more modest on affordance and trajectory tasks. This pattern aligns with the pretraining data volumes: the largest datasets (area at 3.37M, object location at 1.2M) correspond to the largest improvements, while smaller datasets (affordance at 1.13M, trajectory at 563K) correspond to more modest gains. The grasp pose results (1.3M samples, but only modest improvement) suggest that data volume alone does not guarantee improvement — the difficulty of the task and the quality of the annotations also matter.
Chain-of-Point (CoP) Reasoning Results
The CoP reasoning variant (RynnBrain-CoP) is evaluated on the RynnBrain-Bench spatio-temporal tasks, which require the model to both reason about video content and produce spatially localized outputs. The paper reports that the interleaved grounding-reasoning paradigm of CoP "improves performance on complex spatio-temporal tasks (e.g., trajectory prediction) by approximately 7%" (Section 1). This figure appears to represent the improvement of the CoP variant (after SFT + GRPO reinforcement learning) over the pretrained RynnBrain without CoP reasoning, but the paper does not provide a detailed breakdown of CoP-specific results in tables — the main Tables 3 and 4 report only the base pretrained models, not the CoP variant.
The paper states that CoP's design "yields superior performance on tasks requiring precise localization, counting, and other embodied perceptual reasoning capabilities" (Section 1), but specific per-task CoP results (e.g., CoP vs. base RynnBrain on area prediction, affordance prediction, and trajectory prediction individually) are not provided in the evaluation section. This is a significant gap: the 7% figure is the only quantitative CoP result reported, and it lacks granularity and context (7% relative improvement? 7 percentage points? On which exact benchmarks?).
RynnBrain-Nav: Vision-Language Navigation Results
The paper reports that RynnBrain-Nav "achieves state-of-the-art results and consistently surpasses Qwen3-VL-based counterparts across model scales" on the R2R and RxR benchmarks (Section 1). However, specific numerical results for RynnBrain-Nav (success rates, SPL, exact comparisons with prior methods at each model scale) are not provided in the main paper text or in Tables 3–4. The evaluation tables focus on the pretrained foundation model's capabilities (cognition, localization) rather than the post-trained variants' task-specific performance. This absence means that the claimed state-of-the-art navigation results cannot be verified from the information provided in the paper — a significant omission for what is presented as one of the four key post-training directions.
RynnBrain-Plan: Manipulation Planning Results
For RynnBrain-Plan, the paper describes two evaluation protocols: VLMs-UMI (which "directly measures the accuracy and efficiency of high-level planning") and VLMs-VLA (which "evaluates how RynnBrain's physics-aware, spatially explicit plans enhance downstream VLA execution, thereby strengthening the robustness of the hierarchical embodied architecture"). However, as with RynnBrain-Nav, no specific numerical results are reported in the main evaluation section. The paper states that fine-tuning with "only a few hundred samples is sufficient to endow the model with robust long-horizon planning and generalization capabilities" (Section 5.2), but provides no quantitative evidence to support this claim (e.g., planning accuracy, execution success rate, comparison to baselines). The "further details and quantitative evaluations" are deferred to Section 6 but are not present in the provided content.
RynnBrain-VLA: Vision-Language-Action Results
The VLA evaluation is described qualitatively: "In high-complexity grasping scenarios, RynnBrain-VLA consistently outperforms models fine-tuned from π₀.₅" (Section 1). The paper states that "by leveraging RynnBrain's embodied understanding and precise localization, this simple adaptation achieves accurate interpretation of point-based instructions and reliable grasping" (Section 5.3). The fine-tuning uses six pick-and-place tasks with three distinct objects on a Franka Emika arm. However, no quantitative metrics are provided — no grasp success rates, no comparison tables against π₀.₅ baselines, no breakdown by object or task type. The sample size (six tasks, one robot) is extremely small, and without numerical results the strength of this evidence is impossible to assess.
General Visual Understanding (Tables 3 and 4)
To verify that embodied pretraining does not degrade general visual capabilities (catastrophic forgetting), RynnBrain is evaluated on 8 standard vision-language benchmarks. The results show that RynnBrain largely preserves general visual understanding:
- AI2D (diagram understanding): RynnBrain-8B scores 86.3 versus Qwen3-VL-8B at 85.7 — essentially no degradation.
- ChartQA: RynnBrain-8B scores 86.5 versus Qwen3-VL-8B at 89.6 — a small 3.1-point regression.
- DocVQA (document QA): RynnBrain-8B scores 96.2 versus Qwen3-VL-8B at 96.4 — negligible difference.
- MVBench (multi-view): RynnBrain-8B scores 69.5 versus Qwen3-VL-8B at 68.7 — slight improvement.
- RealWorldQA: RynnBrain-8B scores 67.3 versus Qwen3-VL-8B at 71.5 — a 4.2-point regression, the largest gap.
- InfoVQA: RynnBrain-8B scores 83.4 versus Qwen3-VL-8B at 83.1 — essentially flat.
- EgoSchema (egocentric video): RynnBrain-8B scores 69.7 versus Qwen3-VL-8B at 69.7 — identical.
- VideoMME (video understanding, without subtitles): RynnBrain-8B scores 70.7 versus Qwen3-VL-8B at 71.4 — nearly identical.
The overall pattern is that RynnBrain maintains general visual understanding within a few points of its Qwen3-VL initialization across all benchmarks, with no catastrophic collapse on any metric. The largest regressions (RealWorldQA at -4.2, ChartQA at -3.1) are modest and may reflect the domain shift in pretraining data (predominantly indoor/egocentric) relative to the general-domain nature of these benchmarks, rather than genuine forgetting of visual capabilities. This is a positive result for the paper's approach: the 20M-sample embodied pretraining corpus successfully adds spatial grounding and egocentric understanding without destroying the general visual capabilities inherited from Qwen3-VL.
FLOPs-Matched Comparison
Unlike the reference example paper, RynnBrain does not conduct a FLOPs-matched comparison between smaller models using test-time compute and larger models using less compute. The evaluation focuses on demonstrating that the embodied pretraining improves capabilities relative to same-scale baselines, rather than investigating compute-optimal allocation between model size and inference compute. This is a different kind of contribution — a demonstration of capability improvement from data and training methodology, not an analysis of compute efficiency tradeoffs.
Ablation Studies and Robustness Checks
The paper does not include a dedicated ablation study section with systematic evaluation of design choices. Several implicit comparisons and isolated results scattered across sections 3–6 can be interpreted as partial ablations:
Embodied pretraining vs. base VLM (Tables 3–4 across all benchmarks). The comparison of RynnBrain against Qwen3-VL at the same scale serves as the primary "ablation" — it measures the effect of the entire 20M-sample embodied pretraining corpus added on top of Qwen3-VL's general pretraining. Across embodied cognition benchmarks, the improvement is substantial (e.g., +29.4 points on RynnBrain-Object for 2B, +24.9 points on RynnBrain-Spatial for 8B). Across general visual understanding, performance is roughly preserved (within ~4 points). This demonstrates that the embodied pretraining adds specific capabilities without significant forgetting, but it does not isolate which components of the pretraining data or training recipe contribute most.
Chain-of-Point reasoning effect. The paper reports that CoP reasoning improves complex spatio-temporal tasks (e.g., trajectory prediction) by approximately 7% (Section 1). However, this is not presented as an ablation (e.g., base RynnBrain vs. RynnBrain-CoP on each task) and the exact evaluation protocol is unclear — the 7% figure appears in the introduction without specific task breakdown in Section 6.
GRPO reinforcement learning effect. The GRPO training is applied after cold-start SFT for CoP reasoning, but the relative contribution of RL over SFT alone is not separately evaluated. The paper does not report SFT-only CoP performance versus SFT+GRPO CoP performance, making it impossible to attribute the 7% improvement specifically to RL.
Difficulty-aware filtering for RL data. The paper describes filtering the RL training data to retain only samples with SFT model scores between 40 and 80 (Section 4.2.3), but provides no ablation comparing this filtering strategy to unfiltered data or alternative filtering thresholds. This is a significant omission because the filtering strategy is presented as important for RL effectiveness, yet its actual impact is not quantified.
Revision model comparison (analogous to the example paper). The example paper studied sequential vs. parallel sampling tradeoffs for revisions. RynnBrain does not study revision or self-correction mechanisms, and there is no analogous analysis of how different inference strategies affect performance.
Multi-turn dialogue format for planning. The paper reports that formatting manipulation planning as multi-turn dialogues enables long-horizon generalization with only a few hundred samples, but provides no comparison to single-turn planning or alternative memory mechanisms. The claim of data efficiency is not quantitatively substantiated.
Per-sample vs. per-token loss reduction. The paper describes switching from per-token to per-sample loss reduction to eliminate communication overhead (Section 2.2), but provides no ablation comparing the two formulations in terms of convergence speed, final performance, or training stability. The claim that "the holistic approach doubles training efficiency while preserving model stability and convergence properties" is asserted without quantitative evidence.
MoE model comparison (30B-A3B). Table 4 reports results for large-scale models above 30B parameters, but the specific performance of RynnBrain-30B-A3B relative to RynnBrain-8B and RynnBrain-2B is not separately analyzed. There is no scaling analysis showing how performance improves with model scale for the same pretraining data, which would be informative for understanding whether the embodied pretraining benefits scale with model capacity.
Missing ablations. Several design choices that the paper presents as important are not empirically validated:
- The unified coordinate token representation (discrete tokens in [0, 1000]) versus alternative representations (continuous regression, free-form text coordinates, different normalization ranges) is not ablated.
- The 3D reconstruction pipeline for spatial QA data versus using only 2D spatial annotations is not compared.
- The contribution of each pretraining data category is not measured (e.g., removing object data, removing area data, removing OCR data).
- The DeepStack token compression technique is not ablated.
- The online load-balancing pipeline's effect on training throughput and convergence is stated but not quantitatively verified.
Critical Assessment
Claim 1: RynnBrain "largely outperforms existing embodied foundation models by a significant margin."
Assessment: This claim is supported by Tables 3 and 4 for the benchmarks where comparisons are available. On embodied cognition and localization tasks, RynnBrain consistently achieves the highest scores among compared models at both the sub-8B and 30B+ scales. The margins are particularly large on benchmarks that directly test the capabilities emphasized in RynnBrain's pretraining data: object grounding (81.6 vs. 62.8 for Qwen3-VL), area localization (56.2 vs. 30.0), and object-centric QA (71.2 vs. 41.8).
However, the strength of this evidence is qualified by several factors. First, RynnBrain is initialized from Qwen3-VL, which itself receives additional embodied pretraining data — the comparison to the base Qwen3-VL is fundamentally a measure of how much the 20M-sample corpus adds to an already-strong VLM. The comparison to other embodied foundation models (MiMo, RoboBrain 2.0, Pelican-VL) is confounded by different base model initializations, different pretraining data scales, and different training recipes. It is not possible to determine whether RynnBrain's advantage comes from better base initialization, larger pretraining data, better data quality, better training methodology, or some combination.
Second, the set of baselines is incomplete. Several prominent embodied AI models (e.g., RT-2, Octo, OpenVLA, SpatialVLM) are not compared. The paper's scope is limited to "embodied brain" models (high-level reasoning and planning) rather than end-to-end policy models, but this distinction is not consistently maintained — RynnBrain-VLA is an end-to-end policy, and the baselines should reflect that.
Third, the evaluation on RynnBrain's in-house benchmarks (RynnBrain-Object, RynnBrain-Spatial, RynnBrain-Grounding, RynnBrain-Area, RynnBrain-Affordance, RynnBrain-Trajectory) is potentially confounded by distribution overlap with the pretraining data. The paper states that RynnBrain-Bench is constructed from self-collected egocentric indoor videos with rigorous annotation and balancing, but the pretraining data also includes self-collected egocentric indoor videos. If the training and evaluation data are drawn from the same or similar distributions, the reported numbers may overstate generalization to genuinely novel environments. The paper does not describe how training and evaluation data are separated (e.g., by physical environment, by video recording session, or by object instances).
Claim 2: Chain-of-Point reasoning "improves performance on complex spatio-temporal tasks (e.g., trajectory prediction) by approximately 7%."
Assessment: This claim is essentially unverifiable from the provided paper content. The 7% figure appears in Section 1 and is referenced in Section 4, but Section 6 (Evaluation) does not contain a dedicated CoP evaluation table or a comparison of RynnBrain vs. RynnBrain-CoP on specific benchmarks. The paper states that CoP "yields superior performance on tasks requiring precise localization, counting, and other embodied perceptual reasoning capabilities," but provides no data to substantiate this beyond the single 7% figure.
Even if the 7% figure is accurate, the paper does not specify: (a) what metric this represents (absolute accuracy improvement? relative improvement over baseline?); (b) which benchmarks it applies to (trajectory prediction? all spatio-temporal tasks?); (c) how it is calculated (mean across tasks? weighted? best task only?); (d) whether it compares CoP to base RynnBrain or CoP to standard chain-of-thought prompting. Without this information, the claim cannot be evaluated.
Claim 3: RynnBrain-Plan "validates the effectiveness of the fine-grained manipulation-planning paradigm" and fine-tuning with "only a few hundred samples" is sufficient.
Assessment: This claim is not supported by the provided evaluation. Section 5.2 describes the planning post-training methodology and makes qualitative claims about effectiveness, but Section 6 contains no planning results. The VLMs-UMI and VLMs-VLA evaluation protocols are mentioned by name but no quantitative results are reported. The claim of data efficiency (few hundred samples) is intriguing but unsubstantiated — without a learning curve showing performance vs. number of fine-tuning samples, or a comparison to models fine-tuned on larger planning datasets, the claim cannot be evaluated.
Claim 4: RynnBrain-VLA "consistently outperforms models fine-tuned from π₀.₅" in "high-complexity grasping scenarios."
Assessment: This claim is not quantitatively substantiated. The VLA fine-tuning uses only six pick-and-place tasks with three objects, and the paper provides no grasp success rates, no comparison tables against π₀.₅ baselines, and no definition of "high-complexity grasping scenarios." The sample size (six tasks, one robot, one lab) is insufficient to draw general conclusions about VLA performance. The claim is more accurately described as a preliminary demonstration that RynnBrain's pretraining can be adapted for VLA, not evidence of consistent superiority.
Claim 5: RynnBrain-Nav "achieves state-of-the-art results and consistently surpasses Qwen3-VL-based counterparts across model scales" on R2R and RxR.
Assessment: As with the planning and VLA claims, this claim is not supported by the provided evaluation section. The paper states the claim in Section 1 but provides no navigation benchmark results in Section 6. R2R and RxR are standard benchmarks with well-defined metrics (success rate, SPL, nDTW), and the absence of these numbers is a significant gap. The paper's evaluation tables focus exclusively on the pretrained foundation model's capabilities — cognition and localization — and do not include task-specific results for the post-trained variants.
Claim 6: The general visual understanding capabilities are preserved after embodied pretraining.
Assessment: This claim is well-supported by the general visual understanding results in Tables 3 and 4. RynnBrain's performance on 8 standard benchmarks is within ~4 points of its Qwen3-VL initialization, with no benchmark showing catastrophic regression. This is genuine evidence that the 20M-sample embodied pretraining corpus does not cause significant forgetting of general visual capabilities. However, the paper does not investigate whether the preservation holds for more diverse or challenging benchmarks beyond the 8 reported, and does not analyze whether specific sub-capabilities (e.g., fine-grained object recognition, OCR on non-embodied text) might be disproportionately affected.
Missing Experiments
Several experiments that would substantially strengthen the paper's claims are absent:
-
Per-data-category ablation. The paper would be much stronger if it showed how performance changes when individual pretraining data categories are removed (e.g., training without area data, without OCR data, without trajectory data). This would identify which data categories are most critical for which capabilities and provide practical guidance for future work. The current evaluation measures the effect of the entire 20M-sample corpus but cannot attribute improvements to specific design choices.
-
Scaling analysis across model sizes. Tables 3 and 4 report results for 2B, 8B, and 30B+ models, but there is no analysis of how performance scales with model size for the same pretraining data. Does the gap between RynnBrain and Qwen3-VL grow with model scale (suggesting that embodied pretraining benefits compound with capacity) or shrink (suggesting that larger models already capture some embodied understanding)? This analysis is absent.
-
Out-of-distribution generalization. All evaluation benchmarks appear to involve indoor, household, or navigation environments. There is no evaluation of how RynnBrain performs on outdoor environments, industrial settings, or domains substantially different from the training distribution. The claim that RynnBrain is a "foundation model for embodied intelligence" implies broad generalization, but the evaluation does not test this boundary.
-
Human evaluation of planning and reasoning quality. The GPT-4o scoring protocol for textual QA provides automated evaluation, but the quality of physically grounded reasoning and planning outputs is not assessed by human judges. Given that the paper's core claim is about the value of spatial grounding for reasoning, human evaluation of whether the CoP reasoning traces are actually more useful, interpretable, or actionable than purely textual reasoning would be highly informative.
-
Latency and computational cost analysis. For the VLA variant especially, inference latency is critical for real-time control. The paper states that the 2B model is chosen for low latency but provides no latency measurements, no comparison of inference time across model scales, and no analysis of whether the 2B model can run at the control frequencies required for the Franka arm tasks. For navigation and planning, the computational cost of processing long video sequences (up to 2048 frames) is not discussed.
-
Comparison to task-specific state-of-the-art. The paper compares RynnBrain to other "embodied brain" models but not to task-specific state-of-the-art systems. For navigation, how does RynnBrain-Nav compare to dedicated VLN models (not just other VLM-based approaches)? For manipulation, how does RynnBrain-Plan compare to task-and-motion-planning (TAMP) systems? Without these comparisons, it is difficult to assess whether the foundation model approach is competitive with specialized systems or merely competitive with other generalist approaches.
Summary Assessment
The paper's central contribution — that a vision-language model can be trained to produce spatially grounded outputs interleaved with textual reasoning, and that this capability transfers to downstream embodied tasks — is partially supported by the evaluation. The evidence is strongest for the pretrained foundation model's capabilities on cognition and localization benchmarks (Tables 3 and 4), where RynnBrain consistently outperforms comparable embodied foundation models. The evidence is weakest for the post-trained variants (CoP, Nav, Plan, VLA), where quantitative results are largely absent from the provided evaluation section.
The paper would be substantially strengthened by: (1) providing complete evaluation results for all four post-trained variants with standard metrics, (2) conducting ablations that isolate the contribution of specific pretraining data categories and design choices, (3) evaluating out-of-distribution generalization, (4) including human evaluation of reasoning quality, and (5) comparing to task-specific state-of-the-art systems. In its current form, the paper demonstrates a promising approach to embodied pretraining but does not fully validate its claims about downstream task performance.
6. Limitations and Trade-offs
Absence of Evaluation Results for Post-Trained Variants
The assumption or constraint. The paper presents four post-trained variants (RynnBrain-CoP, RynnBrain-Nav, RynnBrain-Plan, RynnBrain-VLA) as central contributions—the introduction frames them as demonstrating "two key potentials of the RynnBrain foundation model: (i) enabling physically grounded reasoning and planning, and (ii) serving as a strong pretrained backbone that can be efficiently adapted to diverse embodied tasks." However, the paper provides almost no quantitative evaluation for any of these variants in Section 6. The sole numerical result is the claim that CoP reasoning "improves performance on complex spatio-temporal tasks (e.g., trajectory prediction) by approximately 7%" (Section 1), without specifying the metric, the baseline, or the benchmarks over which this is computed. The paper states that RynnBrain-Nav "achieves state-of-the-art results and consistently surpasses Qwen3-VL-based counterparts across model scales" on R2R and RxR (Section 1), but reports no success rates, SPL scores, or comparison tables. RynnBrain-Plan "adopts two online evaluation protocols, VLMs-UMI and VLMs-VLA" (Section 1), but the evaluation section contains no planning results. RynnBrain-VLA "consistently outperforms models fine-tuned from π₀.₅" on "high-complexity grasping scenarios" (Section 1), but no grasp success rates or comparison tables are provided.
The consequence. The post-trained variants are positioned as the primary evidence that the foundation model transfers effectively to downstream tasks. Without quantitative results, none of the paper's claims about navigation, planning, or VLA performance can be verified. A practitioner deciding whether to adopt RynnBrain for these tasks has no basis for comparison against existing methods. The abstract and introduction make strong claims about these variants, but the evaluation section effectively treats them as non-existent—if the results exist but were excluded from the paper, the paper's evidentiary basis is incomplete; if the results were not collected, the claims are unsubstantiated. This is the single most consequential limitation because it undermines the paper's core narrative that embodiment-agnostic pretraining transfers to downstream embodied tasks.
What evidence exists in the paper. None. Section 6 (Evaluation) contains Tables 3 and 4 covering the pretrained foundation model's cognition and localization capabilities, plus the general visual understanding benchmarks. The CoP, Nav, Plan, and VLA variants are not evaluated. The in-house RynnBrain-Bench (Section 6.1) is described as an evaluation suite for "cognition and localization capabilities," not for navigation or planning. The paper defers "further details and quantitative evaluations" to Section 6 (Section 5.2), but Section 6 does not provide them.
Mitigation status. Not addressed. The authors do not acknowledge this gap or explain why the post-training results are absent. There is no statement that these results will appear in a future version or an appendix. The limitation is not mentioned anywhere in the paper.
Difficulty Estimation Cost Is Unaccounted for in the RL Training Pipeline
The assumption or constraint. The Chain-of-Point RL training data is curated using a difficulty-aware filtering strategy (Section 4.2.3): each candidate sample is scored by a pretrained SFT model using the evaluation metrics from Section 6.1, and only samples with intermediate difficulty (scores between 40 and 80 out of 100) are retained. This requires running full inference with the SFT model on every candidate sample in the spatio-temporal localization dataset—potentially hundreds of thousands of evaluations—before RL training begins. The paper does not account for this cost in any training budget calculation.
The consequence. The difficulty filtering strategy is presented as an important data curation step that "reduces hallucinations and promotes more reliable reasoning during reinforcement learning" (Section 4.2.3), but its practicality depends on whether the up-front inference cost is amortizable. If the spatio-temporal localization pretraining data is large (millions of samples, per Table 2), scoring every sample with the SFT model could consume comparable or greater compute than the RL training itself. A practitioner attempting to replicate this pipeline would need to invest substantial compute before the first RL update, and would need to determine whether this up-front cost is justified relative to simpler strategies (e.g., random subsampling, training on all data and letting the GRPO advantage normalization handle difficulty variation). Additionally, the filtering thresholds (40 and 80) appear to be chosen heuristically without sensitivity analysis—different thresholds would produce different training datasets with unknown effects on final performance.
What evidence exists in the paper. None. The paper states the filtering strategy (Section 4.2.3) and reports that the resulting RL dataset contains 30K samples, but does not report what fraction of the original spatio-temporal localization data was retained (i.e., how many samples were filtered out), what the up-front scoring cost was in GPU-hours, or whether alternative filtering strategies were tested. There is no ablation comparing difficulty-filtered RL training to RL training on unfiltered data.
Mitigation status. Not addressed. The paper does not acknowledge the up-front scoring cost as a practical limitation or suggest cheaper difficulty estimation methods. The filtering thresholds are presented as a design choice without justification or sensitivity analysis.
In-House Benchmark May Overlap with Training Distribution
The assumption or constraint. The paper introduces RynnBrain-Bench (Section 6.1), a custom evaluation suite constructed from "self-collected egocentric indoor videos and object-centric Q/A pairs initially generated with foundation model priors, followed by the rigorous human-in-the-loop annotation pipeline detailed in Section 3.2." The pretraining data construction (Section 3.2) also draws from "self-collected egocentric indoor videos" and uses the same or similar foundation model priors (Qwen2.5-VL, Qwen3) for QA generation. The paper states that RynnBrain-Bench annotations are "cross-validated by human annotators across multiple rounds" and that "internal data balancing across sub-capabilities within each foundational dimension" is performed, but does not describe how the evaluation data is separated from training data to prevent contamination.
The consequence. If the self-collected videos used for RynnBrain-Bench overlap with those used for pretraining data construction—even if the specific QA pairs differ—the reported performance on RynnBrain-Object (70.7/71.2 for 2B/8B), RynnBrain-Spatial (57.2/59.9), RynnBrain-Grounding (79.1/81.6), RynnBrain-Area (54.6/56.2), RynnBrain-Affordance (89.4/90.4), and RynnBrain-Trajectory (66.6/64.5) may overstate generalization to genuinely novel environments. The margins on these in-house benchmarks are substantially larger than on public benchmarks: RynnBrain-8B scores 71.2 on RynnBrain-Object versus 41.8 for Qwen3-VL-8B (a 29.4-point gap), but on the public VSI-Bench, the gap is 10.7 points (71.0 vs. 60.3). This pattern is consistent with—though not proof of—some degree of training-evaluation overlap inflating the in-house benchmark numbers. A practitioner evaluating whether RynnBrain's strong in-house results will transfer to their own environment cannot assess this risk without knowing whether the training and evaluation environments are distinct.
What evidence exists in the paper. The paper does not describe the separation protocol between training and evaluation data. There is no statement about whether the self-collected videos are split by physical environment, recording session, or any other criterion that would ensure the evaluation scenes, objects, and layouts are unseen during training. The data construction pipeline in Section 3.2 describes QA generation using foundation model priors that are also used for the benchmark construction, raising the possibility of systematic similarity between training and evaluation examples even if the raw videos differ.
Mitigation status. Partially addressed through annotation quality control ("cross-validated by human annotators across multiple rounds" and "data balancing") but not through data separation. The paper does not claim that RynnBrain-Bench environments are held out from training, and does not provide a benchmark-specific datasheet or contamination analysis. This is a standard concern for in-house benchmarks, and the lack of explicit separation documentation is a significant omission for a paper that relies heavily on in-house benchmarks for its strongest quantitative claims.
Single Model Family and Domain: All Results Are on Qwen3-VL Initialization in Indoor/Household Environments
The assumption or constraint. All RynnBrain models are initialized from Qwen3-VL-Instruct checkpoints (Section 3.1, Table 1). The entire 20M-sample pretraining corpus consists of indoor, household, or navigation environments—egocentric videos from Ego4D, Charades-Ego, EPIC-KITCHENS, self-collected "indoor videos" and "house-touring video segments," tabletop grasping scenes from Grasp-Anything, and navigation trajectories in Matterport3D environments (Sections 3.2.1–3.2.4). The evaluation benchmarks (Section 6) are similarly indoor-focused, covering household object cognition, indoor spatial reasoning, indoor navigation (R2R, RxR in Matterport3D), and tabletop manipulation.
The consequence. The paper's findings are unambiguously limited to (a) models initialized from Qwen3-VL, and (b) indoor/household/navigation domains. There is no evidence that the pretraining recipe would transfer to other base VLM architectures (e.g., LLaVA, InternVL, GPT-4V derivatives) or to substantially different domains (outdoor navigation, construction sites, warehouses, agricultural settings, underwater). The indoor bias is particularly important because indoor environments share structural regularities—flat floors, orthogonal walls, standard furniture layouts, consistent lighting—that may make spatial reasoning substantially easier than in unstructured outdoor environments. Similarly, the Qwen3-VL initialization provides a specific set of visual and linguistic capabilities that may interact with the embodied pretraining in ways that do not generalize to other base models. A practitioner deploying RynnBrain in a warehouse or outdoor setting has no evidence that the reported capabilities will transfer.
What evidence exists in the paper. The paper does not claim cross-domain or cross-architecture generalization and does not evaluate on outdoor or industrial benchmarks. The domain restriction is evident from the data sources listed in Table 2 and the benchmark descriptions in Section 6. The model family restriction is explicit in Table 1 (all variants initialized from Qwen3-VL). The general visual understanding benchmarks (Table 3, bottom) provide some evidence of broad visual capability preservation, but these are standard VLM benchmarks (AI2D, ChartQA, DocVQA) that do not test embodied reasoning in novel domains.
Mitigation status. Not addressed. The paper does not acknowledge the domain or model-family restriction as a limitation, and does not suggest that future work should evaluate on outdoor or industrial benchmarks or with alternative base VLMs. The title "Open Embodied Foundation Models" and the framing as a general-purpose embodied intelligence system imply broader applicability than the evaluation supports.
Absence of Systematic Ablation Studies for Core Design Choices
The assumption or constraint. The paper introduces several architectural and training design choices that are presented as important innovations—the unified coordinate token representation (discrete tokens in [0, 1000] rather than continuous regression), the 3D reconstruction pipeline for spatial QA data, the per-sample loss reduction formula, the DeepStack visual token compression, the online load-balancing pipeline, and the difficulty-filtered RL data curation. However, the paper provides no systematic ablation experiments that isolate the contribution of any of these choices. The evaluation in Tables 3 and 4 measures the effect of the entire pretraining pipeline (20M samples added on top of Qwen3-VL), but cannot attribute improvements to specific components.
The consequence. A practitioner seeking to replicate or adapt RynnBrain's approach cannot determine which of the many design choices are load-bearing and which are incidental. For example, if the per-sample loss reduction formula is the key to training stability with variable-length sequences, adopting a different formula could prevent convergence; if it is merely an implementation convenience, it can be replaced without consequence. Similarly, if the 3D reconstruction pipeline for spatial QA data is critical for spatial reasoning improvements, replication requires deploying a SLAM system like MASt3R-SLAM—a substantial engineering undertaking. If similar spatial reasoning improvements could be achieved with cheaper 2D-only spatial annotations, the pipeline is unnecessarily complex. Without ablations, the paper provides a recipe ("do all of these things and you get these results") rather than an analysis ("these specific components cause these specific improvements"), which substantially reduces its value as a research contribution.
What evidence exists in the paper. None. The paper does not contain an ablation study section. The only implicit comparison is the full RynnBrain model versus Qwen3-VL (the initialization), which measures the aggregate effect of the entire 20M-sample pretraining corpus plus all architectural modifications. Within this aggregate, the relative contributions of different data categories (object data vs. spatial data vs. OCR data vs. localization data vs. planning data), different training techniques (per-sample loss, load-balancing, DeepStack), and different output representations (discrete coordinate tokens vs. alternatives) are completely unknown.
Mitigation status. Not addressed. The paper does not acknowledge the absence of ablations as a limitation, does not provide ablation results even for a subset of design choices, and does not suggest that future work should systematically evaluate which components of the pipeline are necessary versus optional. The paper's length (substantial space devoted to data pipeline descriptions, training infrastructure, and post-training recipes) suggests that the authors view the full system specification as the contribution, but without ablations, the reader cannot determine which specifications matter.
7. Implications and Future Directions
How This Work Changes the Landscape
RynnBrain does not introduce a single algorithmic breakthrough that will be immediately adopted across the field—it introduces no new architecture, no new loss function, no new optimization technique. What it does is integrate previously separate capabilities into a unified framework and, in doing so, establish a new baseline for what an embodied foundation model should be able to do. This is a systems-level contribution whose primary impact is to shift the field's expectations about what constitutes a well-rounded embodied intelligence model.
The key reframing is this: prior work treated spatial grounding as a property of the interface between reasoners and executors—a reasoning module produces text, a grounding module converts text to coordinates, a policy module converts coordinates to actions. RynnBrain argues, through its architecture and training design, that spatial grounding is a property of the reasoning process itself. When a model produces a Chain-of-Point trace, it does not first think and then localize; it thinks through localization, with each reasoning step anchored to concrete spatial evidence. This is a qualitative shift from the modular paradigm, and its adoption would mean that future embodied models are evaluated not just on whether their final answers are correct, but on whether their intermediate reasoning steps are verifiable against the physical world.
The paper's second consequential reframing is about data strategy for embodied AI. The dominant paradigm has been to collect embodiment-specific data—robot trajectories, teleoperation demonstrations, navigation episodes—and train models that are tightly coupled to particular hardware. RynnBrain demonstrates an alternative: invest heavily in an embodiment-agnostic foundation model that learns general spatial reasoning, object understanding, and physical grounding from diverse visual data (egocentric video, 3D reconstructions, grasp annotations), then cheaply adapt to specific robots and tasks with minimal fine-tuning data. If the VLA result—reliable grasping from only six pick-and-place tasks after embodiment-agnostic pretraining—generalizes beyond this paper, it implies that the marginal cost of deploying AI on a new robot platform could drop dramatically, because most of the required spatial intelligence is acquired during the shared pretraining phase. This would shift resource allocation in robotics labs from "collect 10,000 demonstrations per task" toward "invest in one large-scale pretraining run and collect 50 demonstrations per task."
A third, more subtle contribution is the paper's negative result about the difficulty-dependence of reinforcement learning for spatial reasoning. The difficulty-aware filtering strategy for GRPO training—retaining only samples where the SFT model achieves intermediate scores—encodes an empirical finding that is important even though the paper does not frame it as such: for spatially grounded tasks with crisp correctness criteria, the window of learnable improvement from RL is narrow. Easy tasks provide no gradient (all outputs already correct), hard tasks provide no reliable signal (all outputs are wrong in ways not meaningfully differentiated by the reward). RL only helps in the intermediate regime where the model is "close but not precise." This finding, if replicated, has direct implications for how RL should be applied to embodied reasoning—it suggests that the standard practice of training on broad data distributions and expecting the policy gradient to sort out informative from uninformative samples is inefficient at best and counterproductive at worst. Future work on RL for embodied AI should explicitly characterize the difficulty distribution of their training data and report performance as a function of difficulty, not just aggregate numbers.
The paper also resolves a tension in the literature about whether VLMs can serve as effective backbones for embodied tasks. Prior work has shown both that VLMs transfer useful semantic knowledge to robotics (e.g., RT-2, SayCan, PaLM-E) and that VLMs struggle with spatial precision and physical consistency (e.g., the observation that standard VLMs fail to produce accurate bounding boxes or understand 3D relationships). RynnBrain demonstrates that these are not contradictory findings—standard VLMs fail at spatial grounding because they were not trained for it, and adding the right training data (3D-reconstructed spatial QA, coordinate token prediction, trajectory annotation) can add these capabilities without destroying general visual understanding. The 8 general vision benchmarks showing performance within ~4 points of Qwen3-VL provide the cleanest evidence in the paper for this claim.
Finally, the paper redirects research attention toward data construction methodology as a first-class research problem. The 20M-sample pretraining corpus is not scraped from the internet—it is constructed through a human-model collaborative flywheel with foundation model priors for proposal generation and human supervision at critical decision points. The quality of this data, particularly the 3D-reconstructed spatial QA, the OCR annotations linked to video frames, and the CoP reasoning traces with manual grounding verification, is likely a larger contributor to RynnBrain's performance than any architectural innovation. This suggests that future progress in embodied AI may depend more on innovations in scalable data annotation pipelines than on novel model architectures—a shift analogous to what occurred in general NLP after the observation that data quality often dominates architectural choices for LLM fine-tuning.
The directions that become more attractive after this work: (1) investing in large-scale, embodiment-agnostic pretraining with diverse spatial grounding data; (2) designing evaluation benchmarks that test intermediate reasoning grounding, not just final answer accuracy; (3) exploring the difficulty-dependent dynamics of RL for spatial reasoning tasks; (4) building hierarchical systems where a general-purpose spatial reasoner feeds grounded instructions to embodiment-specific low-level policies. The directions that become less attractive: (1) training embodied models narrowly on robot-specific demonstration data without general spatial pretraining; (2) evaluating embodied models primarily on text-only QA benchmarks that do not require spatial output; (3) relying on purely textual chain-of-thought for tasks that require physical grounding.
Follow-Up Research This Work Enables
Systematic ablation of pretraining data categories to identify which capabilities transfer, and to which downstream tasks. RynnBrain's 20M-sample pretraining corpus spans general MLLM data, object understanding, spatial understanding, counting, OCR, egocentric task understanding, five types of localization (object, area, affordance, trajectory, grasp), and physics-aware planning. The paper reports aggregate performance on 28 benchmarks but provides no evidence about which data categories drive which capabilities. A strong follow-up would train multiple variants of RynnBrain, each omitting one major data category, and measure the effect on every benchmark in Tables 3 and 4. Concretely: does removing the 3D-reconstructed spatial QA data (855K video + 272K image samples from Section 3.2.2) degrade performance on VSI-Bench and RynnBrain-Spatial, or can general spatial understanding be learned from the 2D-only spatial data? Does removing the object understanding data (1.1M samples) affect grounding performance (RynnBrain-Grounding), or is grounding learned primarily from the dedicated localization datasets? Does the grasp pose data (1.3M samples) transfer to affordance prediction, or are these independent capabilities? This ablation matrix would transform RynnBrain from a monolithic recipe into a modular toolkit where practitioners can select data categories based on their target application.
Replication of the VLA transfer result on a broader range of robots, tasks, and environments. The RynnBrain-VLA result—reliable grasping on six pick-and-place tasks after embodiment-agnostic pretraining—is the paper's most tantalizing claim and also its least substantiated. A rigorous follow-up would: (1) replicate the result with a different robot arm (e.g., a Kinova or Universal Robots arm rather than the Franka Emika) to test whether the transfer is genuinely embodiment-agnostic; (2) expand from 3 objects to 20+ objects covering diverse geometries (deformable objects, transparent objects, articulated objects), since the current objects are unspecified and likely simple rigid bodies; (3) include contact-rich tasks (peg insertion, drawer opening, cloth folding) that require more than pick-and-place; (4) compare RynnBrain-VLA not just to π₀.₅ fine-tuned on the same six tasks, but to RynnBrain-VLA fine-tuned without the embodiment-agnostic pretraining (i.e., starting from Qwen3-VL directly), to isolate the contribution of the spatial pretraining. The key measurement would be the sample efficiency curve: how many task-specific demonstrations does RynnBrain-VLA need to reach 80% success rate, versus baselines? If the pretraining provides a ~5× sample efficiency improvement that transfers across robots, that is a practically important result even if absolute success rates are modest.
Investigation of whether Chain-of-Point reasoning reduces hallucination in spatial outputs, using controlled perturbation experiments. The paper claims that CoP reasoning "anchors intermediate reasoning steps to concrete spatial references" and is "hallucination-resistant" (Section 4), but provides no evidence that CoP actually reduces spatial hallucinations compared to standard chain-of-thought followed by a separate localization step. A strong follow-up would construct a benchmark where the video contains deliberately ambiguous or misleading visual cues (e.g., two identical objects in different locations, an object partially occluded by another) and measure whether CoP traces correctly express uncertainty or point to the wrong location less often than textual chain-of-thought. Additionally, one could perturb the input video (e.g., remove the target object from some frames, add distractor objects) and measure whether CoP's spatial anchors make the reasoning more robust to visual noise—if CoP's intermediate grounding steps create verifiable checkpoints, the model should be able to detect when its spatial predictions conflict with visual evidence and backtrack, whereas a purely textual reasoner might continue confidently along a wrong path.
Extension of the difficulty-aware RL filtering insight to other spatial reasoning tasks and reward formulations. The paper's finding that RL training for spatial reasoning benefits from difficulty filtering (Section 4.2.3) is presented as a data curation detail, but it may reflect a more general principle about the interaction between reward function sharpness and policy gradient effectiveness. A follow-up study would systematically vary the difficulty filtering threshold (e.g., keeping samples with SFT scores from 0–20, 20–40, 40–60, 60–80, 80–100) across multiple spatial reasoning tasks (trajectory, affordance, area, grasp) and multiple reward formulations (Fréchet distance vs. MSE for trajectories, Chamfer vs. Hausdorff for affordances, strict point-in-polygon vs. distance-to-boundary for areas). The key hypothesis to test: is there a universal "effective difficulty band" for each reward function, or does the optimal filtering threshold depend on the specific reward's sensitivity to small versus large errors? If the effective difficulty band is reward-function-specific, this would provide practical guidance for reward design in embodied RL—one would choose a reward function whose gradient is informative in the difficulty regime where the SFT model operates, rather than choosing the most "principled" metric.
Development of a continuous or adaptive difficulty estimation mechanism to replace the static binning used in RL data curation. The paper's RL data filtering uses fixed thresholds (SFT scores between 40 and 80) applied uniformly to all tasks. This is brittle—if the SFT model improves during RL training, samples that were intermediate difficulty at the start become easy, and the effective training set shrinks. A dynamic filtering approach that periodically re-evaluates sample difficulty and adjusts the training distribution would be more efficient. This connects to curriculum learning and self-paced learning literatures, but the novel aspect is the specific interaction with GRPO's group-based advantage normalization: how does the difficulty distribution within each group of G=5 outputs affect the quality of the advantage estimate? If all 5 outputs in a group are similarly difficult (e.g., all scores between 40–50), the advantage noise is low but the signal for preferring one output over another is also low. If the group mixes difficulties (e.g., scores of 20, 40, 60, 80, 100), the advantage signal is strong but may be dominated by the easy/hard extremes. Determining the optimal group composition for GRPO in spatial reasoning tasks would inform both data curation and sampling strategies for future work.
Evaluation of RynnBrain's spatial grounding capabilities on out-of-distribution environments—specifically outdoor, industrial, and dynamic settings. The paper's evaluation is exclusively indoor/household/navigation. A critical stress-test would evaluate RynnBrain on: (1) outdoor egocentric video (e.g., Ego4D outdoor subsets, or specialized datasets like TITAN for traffic scenes) to test whether spatial understanding transfers to environments without orthogonal walls, consistent lighting, and structured furniture layouts; (2) industrial settings (e.g., factory floors, construction sites) where objects are large, metallic, or irregularly shaped, and spatial relationships involve heavy machinery rather than household items; (3) dynamic environments where objects move independently of the agent (e.g., crowded spaces, sports scenarios), testing whether the trajectory prediction and temporal reasoning capabilities learned from relatively static indoor scenes transfer to settings with independent object motion. The key comparisons would be RynnBrain versus Qwen3-VL on these out-of-distribution benchmarks—if the gap narrows or reverses, this identifies the boundary of the indoor pretraining's effectiveness. If RynnBrain maintains a substantial advantage, it suggests that the spatial reasoning capabilities are more general than the training domain.
Practical Applications and Downstream Use Cases
Hierarchical robot control with a cloud-based RynnBrain planner and an on-device low-level policy. The architecture that RynnBrain enables is one where a large model (RynnBrain-8B or 30B-A3B) runs remotely, processing egocentric video streams and producing physically grounded plans (e.g., "grasp the sponge <affordance><frame 47>: (310, 556)</affordance> and wipe the counter following <trajectory><frame 47>: (100, 200), (150, 250), ...</trajectory>"), while a lightweight on-device policy (potentially a small VLA model distilled from RynnBrain-VLA) executes these plans using the explicit spatial coordinates as waypoints. The benefit is that the on-device model does not need to perform high-level reasoning, object recognition, or task decomposition—it only needs to track the specified spatial targets and generate motor commands, which is a substantially simpler learning problem. RynnBrain's performance on localization benchmarks (81.6 Acc@0.5 on grounding, 90.4 on affordance, 64.5 DFD on trajectory for the 8B model) suggests that the spatial instructions would be precise enough for downstream execution, though this has not been demonstrated end-to-end. The key practical advantage is that the cloud model can be updated continuously (improving reasoning and spatial understanding) while the on-device policy remains fixed, decoupling cognitive improvement from physical deployment.
Automated spatial annotation for robot training data. RynnBrain's localization capabilities—particularly on area prediction (56.2 on RynnBrain-Area for 8B, versus 30.0 for Qwen3-VL) and affordance prediction (90.4, versus 82.9)—make it a candidate for automated or semi-automated labeling of robot manipulation data. In a typical robot data collection pipeline, human operators must annotate where objects are, where to grasp them, and where to place them. RynnBrain could pre-annotate video frames with proposed bounding boxes, affordance points, and area points, reducing human annotation effort to verification rather than de novo labeling. The 1.3M grasp pose samples in the pretraining data specifically train this capability. A deployment would involve running RynnBrain on videos of manipulation demonstrations, generating spatial annotations, and having humans correct only the ones flagged as low-confidence. The practical metric would be annotation time saved: if RynnBrain's proposals are correct 80% of the time and require only minor adjustment for the remaining 20%, this could reduce annotation costs several-fold for large-scale robot data collection efforts.
Egocentric video understanding for assistive technology. The OCR and egocentric task understanding capabilities—specifically the 1M-sample OCR dataset combining contextual QA (256K GPT-5.2-generated questions) with template-based structured questions (722K samples)—position RynnBrain for assistive applications where a wearable camera processes first-person video and answers user queries about the environment. For example, a visually impaired user could ask "What does the sign on the door say?" or "Is there an empty seat at this table?" and receive both a textual answer and spatial coordinates (so the answer is actionable—"the empty seat is at <area>: (coordinates)"). The EgoTextVQA indoor result (31.6 for 8B) shows there is room for improvement, but the structured output format (text + coordinates) provides a capability that no other model in the comparison table offers—other models might correctly answer that an empty seat exists, but cannot tell the user where it is in their field of view. The practical barrier is inference latency: processing video at 2 FPS (the paper's training configuration) may be too slow for real-time assistive applications, and the 8B or 30B models may not run on wearable hardware. Distillation to a smaller, faster model specialized for assistive queries would be a natural next step.
When to Prefer This Method
The paper does not articulate a clear tradeoff against named alternatives in a way that would support a structured decision rule. It positions RynnBrain as generally superior to existing embodied foundation models on the benchmarks it evaluates (Tables 3 and 4), and as a drop-in replacement for Qwen3-VL in embodied applications, without specifying conditions under which a practitioner should prefer MiMo-Embodied, RoboBrain 2.0, Pelican-VL, Cosmos-Reason2, or a task-specific system. The post-training variants (Nav, Plan, VLA) are presented as demonstrations of transfer rather than as competitors to state-of-the-art specialized systems, and no head-to-head comparisons with specialized navigation, planning, or manipulation systems are provided.
The paper's implicit guidance is: if you need a single model that jointly performs embodied understanding, spatial localization, and physically grounded reasoning, and you can afford the inference cost of a 2B–8B VLM, RynnBrain outperforms other embodied foundation models on these capabilities. Beyond this, the paper does not provide the evidence needed to make a principled choice between RynnBrain and alternatives for specific downstream tasks, because those task-specific evaluations are absent. A practitioner deciding between RynnBrain-Nav and a specialized VLN model, or between RynnBrain-Plan and a task-and-motion-planning system, would find no quantitative basis for their decision in this paper. This is not a flaw in the method but a gap in the evaluation that limits the paper's practical utility for downstream technology selection.
