SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training

ArXiv: 2501.17161

🎯 Pitch

While supervised fine-tuning (SFT) causes models to memorize training data and catastrophically fail on unseen task variants—dropping nearly 35% on visual navigation under simple rule or color shifts—reinforcement learning (RL) with an outcome reward consistently improves out-of-distribution generalization, even boosting visual recognition. Surprisingly, SFT remains essential to stabilize the output format before RL, revealing a necessary symbiosis where one step fixes the form and the other teaches transferable principles.

1. Executive Summary

This paper presents a comprehensive empirical study comparing how supervised fine-tuning (SFT) and reinforcement learning (RL) affect generalization versus memorization in foundation model post-training, using Llama-3.2-Vision-11B as the backbone model across two reasoning environments—GeneralPoints (an arithmetic card game) and V-IRL (a real-world visual navigation task). The paper demonstrates that RL generalizes to out-of-distribution rule variants (e.g., training on face cards as '10' but testing on 'J'=11, 'Q'=12, 'K'=13) and visual variants (e.g., training on black-suited cards but testing on red-suited cards), while SFT memorizes the training data and consistently degrades on unseen variants—for instance, RL improves OOD performance on V-IRL-VL by +9.3% (35.7% → 45.0%) while SFT drops by −33.2% (35.7% → 2.5%). As a byproduct, the multi-turn RL approach achieves a +33.8% improvement (44.0% → 77.8%) on the V-IRL mini benchmark, establishing a new state-of-the-art. The paper further shows that scaling RL compute improves the model's underlying visual recognition capabilities, that SFT is necessary to stabilize output format before RL training, and that increasing verification iterations strengthens OOD generalization—establishing that RL's advantage holds across both textual and visual domains, but only when the backbone model has been initialized with sufficient instruction-following capability through SFT.

2. Context and Motivation

The Core Problem: We Don't Know Which Post-Training Method Produces Generalizable Knowledge

The fundamental question this paper tackles is deceptively simple: when we fine-tune a foundation model—whether through supervised learning on expert demonstrations or reinforcement learning with reward signals—does the model actually learn transferable principles, or does it simply memorize surface patterns in the training data? This matters because the entire enterprise of building foundation models rests on an implicit assumption that post-training produces models that generalize beyond their training distribution. If models primarily memorize, then the impressive benchmark performances we observe may be brittle, failing catastrophically when deployed in slightly different conditions.

The paper frames this in Section 1 as a tension between two competing hypotheses about what post-training actually accomplishes:

"A key challenge in analyzing the generalizability of foundation models is to separate data memorization from the acquisition of transferable principles."

This distinction is not merely academic. Foundation models are increasingly deployed in high-stakes settings—medical diagnosis, autonomous navigation, legal reasoning—where encountering task variants not seen during training is the norm, not the exception. A model that has memorized training patterns rather than internalized underlying rules will produce confident but incorrect outputs when the surface statistics shift. Understanding which post-training paradigm produces genuine generalization is therefore a prerequisite for building reliable and robust AI systems, as the introduction emphasizes.

A Genuine Gap in the Literature: No Head-to-Head Comparison Under Controlled Distribution Shifts

The post-training landscape for foundation models is dominated by two paradigms: supervised fine-tuning (SFT) on curated demonstration data, and reinforcement learning (RL) using reward signals to optimize behavior. Both are widely deployed in production systems—GPT-4, Gemini, Llama, and DeepSeek all employ some combination of SFT and RL in their training pipelines (Section 1, Section 2). Prior work has extensively studied each paradigm in isolation:

The SFT literature has shown that fine-tuning on diverse instruction-formatted data produces impressive zero-shot generalization to unseen tasks (FLAN; Wei et al., 2022a), while other work suggests that SFT primarily acts as a "format teacher"—adapting the model's output style to a desired format while leveraging capabilities already present in the pretrained model (LIMA; Zhou et al., 2024a). However, these studies evaluate generalization to entirely different tasks (not controlled rule variants of the same underlying task), making it impossible to distinguish genuine rule learning from superficial pattern matching.

The RL literature has focused primarily on alignment with human preferences (Ziegler et al., 2019; Ouyang et al., 2022) or on training models to solve specific tasks through outcome-based rewards (Abdulhai et al., 2023; Zhou et al., 2024b; Zhai et al., 2024a). The RL4VLM framework (Zhai et al., 2024a) demonstrated that RL can fine-tune vision-language models as decision-making agents, but did not systematically compare RL against SFT on generalization metrics, nor did it study whether the learned behaviors transfer to distribution shifts.

The memorization-and-generalization literature has produced seemingly contradictory findings. On one hand, studies show that LLMs exhibit overfitting on simpler, knowledge-intensive tasks and greater generalization on complex, reasoning-intensive ones (Wang et al., 2024; Qi et al., 2024). On the other hand, work by Allen-Zhu and Li (2023a,b; 2024) and Ye et al. (2024) suggests that LLMs develop reasoning skill sets beyond their training data by pre-computing reasoning graphs before autoregressive generation. However, these studies examine pretrained models, not the differential effects of SFT versus RL post-training. The question of which post-training paradigm—if either—produces transferable rule learning has remained unanswered.

No prior work has conducted a controlled, head-to-head comparison of SFT versus RL with explicit distribution-shift evaluations on the same underlying tasks. As the paper states in Section 2:

"Our work differs from prior studies, as we aim to comparatively analyze the generalization and memorization of SFT and RL on both LLM and VLM, while previous studies have focused primarily on only one of these two post-training methods... or on only one post-training method."

This gap is significant because production systems use both methods sequentially (SFT followed by RL), but without understanding their distinct contributions, practitioners cannot make principled decisions about how to allocate post-training compute between them.

Why Existing Benchmarks Cannot Answer This Question

The paper identifies a methodological problem with existing evaluation paradigms: standard benchmarks evaluate performance on in-distribution test sets, where memorization and generalization are conflated. A model that achieves 90% accuracy on a held-out test set from the same distribution as training data may have either (a) learned the underlying rules and can apply them to any instance, or (b) memorized patterns in the training data that happen to generalize to the test set because the test distribution is identical.

To disentangle these possibilities, one needs controlled distribution shifts—variants of the same task that require the same underlying capability but present it through different surface forms. The paper constructs exactly this type of evaluation:

• Textual rule-based generalization (Section 4.1): In GeneralPoints, the model is trained with one rule interpreting face cards ('J', 'Q', 'K') as all equal to '10', then tested on a variant where they equal '11', '12', '13' respectively. The underlying arithmetic reasoning capability is identical; only the mapping from symbols to numbers changes. A model that has learned to perform arithmetic operations should transfer seamlessly; a model that has memorized training patterns (e.g., "when I see 'K', I use 10 in my equation") will fail.
• Visual generalization (Section 4.1, 4.2): The model is trained on cards of one color (black suits) or navigation routes from one city (New York), then tested on cards of a different color (red suits) or routes from different cities worldwide. The reasoning task is unchanged; only the visual surface statistics shift.

This controlled-shift methodology is what allows the paper to make definitive claims about memorization versus generalization, and it represents a conceptual advance over prior work that evaluated generalization only to entirely different tasks (which conflates capability transfer with rule transfer).

The Specific Role of Each Post-Training Paradigm

The paper also addresses a practical question that arises in any production post-training pipeline: if SFT and RL produce different kinds of knowledge, what is the role of each? Prior work such as LIMA (Zhou et al., 2024a) hypothesized that SFT primarily teaches output formatting, but this was not tested in the context of a comparison with RL, nor was it tested under distribution shifts that would reveal whether SFT contributes anything beyond format. The paper explicitly investigates whether SFT is necessary for RL training at all (Section 5.4)—finding that without SFT initialization, the base Llama-3.2-Vision-11B model cannot follow instructions well enough for RL to even begin improving—while also showing that excessive SFT (overfitting) can lock in memorized patterns that RL cannot subsequently overcome (Section 6, "Limits of RL in corner cases"). This establishes that SFT and RL are not merely interchangeable optimization methods but serve complementary roles in a post-training pipeline.

How This Paper Positions Itself

The paper situates itself at the intersection of three research threads (Section 2):

1. Post-training methodology: Building on the RLHF pipeline (Ouyang et al., 2022) and the RL4VLM framework (Zhai et al., 2024a), the paper adopts a multi-turn RL formulation with sequential revision (drawing from Snell et al., 2024) where the model conditions on its own previous outputs and verifier feedback to iteratively improve its answers. This is not itself a novel training method—the contribution lies in using this established formulation as a testbed for comparing SFT and RL.

2. Memorization vs. generalization in neural networks: The paper draws on the language from the broader ML literature on generalization (Bousquet & Elisseeff, 2000; Zhang et al., 2021) and applies it to foundation model post-training, arguing that the memorization-generalization dichotomy—well-studied in supervised deep learning—has not been systematically examined for the specific case of SFT versus RL on the same task.

3. Inference-time compute scaling: The multi-turn RL formulation with verifier feedback implicitly connects to recent work on scaling test-time computation (Snell et al., 2024; Jaech et al., 2024), where additional inference steps enable models to correct errors. The paper's finding that more verification iterations improve OOD generalization (Section 5.5, Figure 10) directly connects RL's generalization advantage to inference-time compute scaling—RL becomes more effective at generalizing when given more opportunities to verify and revise its outputs.

The paper's position is not to propose a new training algorithm but to provide controlled experimental evidence that resolves an outstanding ambiguity in the field: do SFT and RL produce different kinds of knowledge, and if so, under what conditions? By constructing tasks with clean, controlled distribution shifts (rule variants, visual variants) and measuring the differential impact of SFT and RL on these shifts, the paper aims to establish that:

• RL learns generalizable rules that transfer across distribution shifts
• SFT primarily memorizes surface patterns that fail under distribution shifts
• SFT nonetheless serves an essential scaffolding role by stabilizing the output format for subsequent RL
• These findings hold across both unimodal (LLM) and multimodal (VLM) settings, and across both arithmetic and spatial reasoning tasks

This is a deliberately narrow and falsifiable thesis—it does not claim that SFT is always memorization or that RL is always generalization, but rather that under the controlled conditions tested, with comparable compute budgets and the same initialization, the two paradigms produce qualitatively different kinds of learning. The paper's contribution is the empirical demonstration of this difference and the characterization of its boundary conditions.

3. Technical Approach

3.1 Reader Orientation (Approachable Technical Breakdown)

The "system" in this paper is a controlled experimental framework for comparing how two different post-training methods—supervised fine-tuning (SFT) and reinforcement learning (RL)—affect a foundation model's ability to generalize to unseen variants of reasoning tasks. The problem it solves is the confound between memorization and generalization in standard benchmarks: when a model performs well on a held-out test set, we cannot tell whether it learned transferable rules or simply memorized surface patterns that happen to match the test distribution. The "shape" of the solution is to construct tasks with clean, controlled distribution shifts (rule variants, visual variants), train separate models with comparable SFT and RL budgets starting from the same initialization, and measure performance both in-distribution and out-of-distribution—the differential between ID and OOD performance revealing whether each method produces memorization or generalization.

3.2 Big-Picture Architecture (Diagram in Words)

The experimental architecture has six major components:

1. Backbone Model (Llama-3.2-Vision-11B): A pretrained vision-language model that serves as the initial policy for both SFT and RL training. It processes either text-only inputs (LLM mode) or image-plus-text inputs (VLM mode) and generates structured JSON outputs containing reasoning steps and answers.

2. Task Environments (GeneralPoints and V-IRL): Interactive simulators that present reasoning problems to the model, accept actions, and return reward signals and next states. Each environment exists in two modalities (pure language -L and vision-language -VL) and supports configurable rule variants and visual variants for constructing distribution-shift evaluations.

3. Verifier (VER): A rule-based program (not a learned model) that evaluates the model's output against ground-truth correctness, produces an outcome-based reward signal, and generates textual feedback. It implements the outcome-based reward function described in Cobbe et al. (2021) and is the source of all training signals for RL.

4. SFT Training Pipeline: A standard supervised fine-tuning procedure that trains the model on expert demonstration data (optimal single-turn prompt-response pairs) using next-token prediction loss. This produces the SFT-initialized checkpoint and also serves as the SFT scaling baseline when training is continued.

5. RL Training Pipeline (PPO with Sequential Revision): A multi-turn RL framework that treats the model as a policy network, uses PPO (Schulman et al., 2017) as the backbone algorithm, and employs a sequential revision formulation (Snell et al., 2024) where the model conditions on its own previous outputs and verifier feedback to generate improved answers iteratively.

6. Evaluation Protocol: A set of controlled test conditions that measure model performance on the trained rule/visual configuration (in-distribution) and on unseen rule/visual configurations (out-of-distribution), with performance measured as success rate or per-step accuracy, enabling direct comparison of the generalization gap between SFT and RL.

Information flows as follows: a task prompt (potentially with an image) enters the system → the model generates a structured output → the verifier checks correctness, produces a scalar reward and textual feedback → for RL training, this feedback is appended to the context and the model generates a revised output (repeating up to a maximum number of verification steps) → PPO updates the model parameters using the collected trajectories. For SFT training, only the initial prompt and expert response pairs are used, with no iterative feedback loop.

3.3 Roadmap for the Deep Dive

• First, the formal RL formulation adapted to foundation models, which defines how the language modeling problem is mapped onto states, actions, rewards, and policies—this is the mathematical scaffolding for all experiments.
• Second, the sequential revision mechanism, which implements the multi-turn feedback loop and is the key architectural choice that distinguishes RL training from single-turn SFT.
• Third, the task environments (GeneralPoints and V-IRL) in detail, including their configurable rule and visual variants, reward designs, and data generation procedures—these are the controlled testbeds that make the memorization-vs-generalization comparison possible.
• Fourth, the SFT training procedure, including data construction (expert demonstrations vs. suboptimal trajectories), training configuration, and how it serves as the initialization point for RL.
• Fifth, the RL training procedure, including the PPO algorithm instantiation, the replay buffer mechanism, and how training compute is estimated and scaled.
• Sixth, the evaluation protocol and metrics, including the in-distribution vs. out-of-distribution setup, per-step accuracy vs. success rate, and how the generalization gap is measured.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a controlled empirical comparison paper whose core idea is that SFT and RL produce qualitatively different kinds of knowledge when applied to the same foundation model on the same task—RL learns generalizable rules while SFT primarily memorizes surface patterns—and that this difference can be revealed by measuring performance under controlled distribution shifts.

Adapting RL Terminology to Foundation Model Post-Training

The paper adopts a multi-turn RL formulation for training foundation models, following the RL4VLM framework (Zhai et al., 2024a). This formulation maps the language generation problem onto standard RL concepts, enabling the use of policy gradient algorithms like PPO.

State space. For language-only models (LLM), the state space $S$ is defined as $S := \mathcal{V}^m$, where $\mathcal{V}$ is the discrete vocabulary (token) space and $m$ is the maximum token length of the input sequence $v_{\text{in}}$. For vision-language models (VLM), the state space is $S := \mathcal{V}^m \times \mathcal{O}$, where $\mathcal{O}$ is the space of all RGB images.

What this means operationally: a state is the entire textual context that the model can see when generating its next output. For a VLM, this includes both the tokenized text and the raw image pixels. The state at timestep $t$ is constructed by concatenating the system prompt with all previous model outputs and verifier feedback, making the state history-dependent.

Action space. The action space $A$ is defined as $A := \mathcal{V}^n$, where $n$ is the maximum token length of the output sequence $v_{\text{out}}$.

What this means operationally: an action is a complete output sequence (potentially hundreds of tokens long), not individual tokens. The model autoregressively generates this sequence token-by-token during a single "turn," and the entire sequence is treated as one action from the RL perspective. This is a turn-level rather than token-level action space, consistent with the RL4VLM framework.

Policy network. The model with parameters $\theta$ is treated as the policy network $\pi_\theta : S \rightarrow \mathcal{V}^n$. The goal is to learn a policy that maximizes the expected cumulative reward:

$\max_{\pi \in \Pi} \mathbb{E}_\pi\left[\sum_{t=0}^T r_t\right]$

where $T$ is the maximum number of steps per episode, $r_t = r(s_t, a_t)$ is the reward received at step $t$, and $\pi(a|s) \in [0, 1]$ denotes the probability of $\pi$ choosing action $a$ at state $s$.

What it computes: the standard RL objective—find a policy (a mapping from states to action distributions) that, when followed, accumulates the highest total reward over an episode of length up to $T$. In the context of this paper, an "episode" is one complete reasoning problem (one set of four cards for GeneralPoints, one navigation route for V-IRL), and each step within the episode is one turn of model generation followed by verifier feedback.

Why this form: treating the model as a policy enables the use of policy gradient methods like PPO, which can learn from the outcome-based scalar rewards produced by the verifier. The turn-level action space (rather than token-level) makes the optimization tractable—token-level RL would require credit assignment across hundreds of tokens and would be computationally infeasible at the scale of 11B-parameter models. The sequential nature of the problem (multiple attempts at solving the same task) maps naturally onto the multi-step RL formalism.

Verifier specification. The paper uses a rule-based verifier $\text{VER}: \mathcal{V}^n \rightarrow \mathbb{R} \times \mathcal{V}^k$ that evaluates the model's output and returns both a scalar reward $r$ and textual feedback $v_{\text{ver}}$:

$\text{VER}(v_{\text{out}}^t) \mapsto (r_t, v_{\text{ver}}^t)$

Why a rule-based verifier rather than a learned reward model: for the controlled tasks in this paper (arithmetic reasoning, navigation), correctness can be determined programmatically—checking whether an equation evaluates to 24, or whether a navigation action matches the expert trajectory. A learned reward model would introduce additional confounds (reward model errors, distribution shift between the reward model and the policy), making it harder to isolate the effects of RL versus SFT. The rule-based verifier provides a perfect signal for correctness, ensuring that any differences between SFT and RL are attributable to the learning method, not to noise in the reward function. This is an important design choice: the paper is studying learning dynamics, not reward engineering.

Sequential Revision: The Multi-Turn Feedback Mechanism

The key architectural innovation that enables RL training (and distinguishes it from single-turn SFT) is the sequential revision formulation, adopted from Snell et al. (2024) and shown in Figure 2 and Figure 3. This mechanism constructs the state at each timestep by concatenating the model's previous outputs with verifier feedback, allowing the model to iteratively improve its answers.

State construction at $t = 0$. The initial input $v_{\text{in}}^0$ consists of the system prompt, which includes a task description, task rules, and output format instructions. For vision-language variants, this also includes the input image.

State construction at $t \geq 1$. For subsequent timesteps, the input prompt $v_{\text{in}}^t$ is constructed by concatenating the system prompt with all prior model and verifier outputs:

$v_{\text{in}}^t = \text{concat}\left(v_{\text{in}}^0, [v_{\text{out}}^k, v_{\text{ver}}^k]_{k=0}^{t-1}\right)$

where $v_{\text{out}}^k$ is the model's output at turn $k$ and $v_{\text{ver}}^k$ is the verifier's feedback at turn $k$.

What it computes: each new state is the entire conversation history—the system prompt, all previous model attempts, and all previous verifier feedback—concatenated into a single context window. The model sees its own past mistakes and the verifier's corrections when generating the next attempt.

Why this form: this is the mechanism that enables iterative self-correction. Rather than generating all possible solutions in parallel (as in best-of-N sampling), the model generates solutions sequentially, each conditioned on the full history of previous attempts and their outcomes. This is conceptually similar to how RL4VLM (Zhai et al., 2024a) and the sequential revision formulation of Snell et al. (2024) implement multi-turn improvement—the key insight being that conditioning on previous failures provides a richer signal for improvement than independent resampling.

Transition dynamics (Figure 3). The paper provides a formal template for how states transition:

1. The system prompt $v_{\text{in}}^0$ contains task description, input specification (cards or navigation instructions), and output format requirements.
2. At each step $t$, the model generates $v_{\text{out}}^t$ as structured JSON (containing reasoning and answer).
3. The verifier evaluates $v_{\text{out}}^t$, producing reward $r_t$ and feedback $v_{\text{ver}}^t$ (e.g., "You failed this trial because your formula is incorrect" or "Correct solution").
4. The next state is constructed as $v_{\text{in}}^{t+1} = \text{concat}(v_{\text{in}}^t, v_{\text{out}}^t, v_{\text{ver}}^t)$.

Episode termination. An episode terminates when either the model produces a correct answer (success) or the maximum number of verification steps is reached. The default maximum is 5 steps for GeneralPoints and 2 steps for V-IRL (specified in Appendix A.3 and B.3).

Why the maximum steps differ between tasks: GeneralPoints allows more revision attempts because arithmetic reasoning involves a discrete search over equation forms—the model can try many different formulas. V-IRL navigation has a stricter limit because the model must take correct actions at each position along a route, and errors compound (a wrong turn early in the route makes recovery impossible for subsequent steps). The lower maximum in V-IRL reflects the task's more sequential, error-intolerant structure.

Connection to inference-time compute scaling. The sequential revision formulation implicitly connects the paper's findings to recent work on scaling test-time compute (Snell et al., 2024; Jaech et al., 2024). The maximum number of verification steps controls how much inference-time computation the model can use per problem—more steps mean more opportunities to generate, verify, and revise. The paper explicitly studies this in Section 5.5, varying the number of verification iterations and measuring its effect on OOD generalization.

The GeneralPoints Environment: Arithmetic Reasoning with Controlled Variants

GeneralPoints is an original environment designed by the authors to evaluate arithmetic reasoning capabilities under controlled distribution shifts. It is built on top of the Points24 environment from Zhai et al. (2024a) but introduces configurable rule and visual variants specifically for studying generalization.

Core task mechanics. At each episode, the model receives four cards sampled from a standard 52-card poker deck. The goal is to produce a mathematical equation that evaluates to a target number (24 by default) using each card's numerical value exactly once. The cards can be presented either as text descriptions (GP-L variant) or as an image (GP-VL variant). Each sampled quadruple is guaranteed to have at least one solution equal to 24, enforced by an expert solver during the data generation process (Appendix A.1).

Visual challenges in GP-VL. When cards are presented as images, the model faces an additional visual recognition challenge—it must correctly identify which four cards are shown before it can compute the equation. This is the key difference between GP-L and GP-VL: in GP-L, the card identities are given directly as text; in GP-VL, the model must extract them from pixels. This allows the paper to separately study the effects of post-training on reasoning capabilities (GP-L) and on combined visual recognition + reasoning capabilities (GP-VL).

Configurable arguments (Appendix A.3). The environment supports several configuration parameters that enable controlled experiments:

• Target point: Any positive integer (fixed at 24 for all experiments in the paper).
• Face cards rule: Two options that define how 'J', 'Q', 'K' are interpreted:
  • All count as '10' (this is the ID training rule)
  • 'J'=11, 'Q'=12, 'K'=13 (this is the OOD test rule)
• Card sampling: Two options:
  • Sample 4 cards uniformly without replacement from the full deck
  • Require at least one face card (J, Q, or K) in the sampled set
• Card color: Three options:
  • Black suits only (♠, ♣)
  • Red suits only (♥, ♦)
  • All suits (♠, ♥, ♣, ♦)

Rule variant design (Section 4.1). The rule variants are the key mechanism for testing generalization. When training with the rule "J, Q, K all count as 10," the model encounters equations using numbers {1, 2, ..., 10}. When tested with the rule "J=11, Q=12, K=13" and forced to include at least one face card, the model must use numbers above 10 (e.g., 11, 12, 13) that never appeared during training, while applying the same arithmetic operations (+,-,*,/, parentheses). A model that has learned the abstract operation of "map symbol to number, then find an equation" should transfer seamlessly; a model that has memorized specific card-value mappings (e.g., "when I see K, I write 10") will fail.

"For out-of-domain evaluation, we use the alternative face cards rule and require at least one face card, forcing calculations with numbers above 10 that are not encountered during training." (Appendix A.3)

Visual variant design (Section 4.1). The visual variants test whether the model's visual recognition capabilities generalize across surface-level visual features. The model is trained on cards with black suits (♠, ♣) and tested on cards with red suits (♥, ♦). The underlying task (recognize the number, compute 24) is identical; only the color of the card symbols changes—a purely visual distribution shift that should not affect performance if the model has learned generalizable visual recognition.

Reward design (Appendix A.3). The reward function is designed to provide a rich signal that distinguishes different types of errors:

• $r = +5$: Generated a legal equation that equals the target point (success)
• $r = -1$: Generated a legal equation using each card exactly once but not equaling the target (close but incorrect)
• $r = -1$: Exceeded the maximum verification step without success (timeout)
• $r = -2$: Generated an equation containing numbers not among the given cards (illegal number—a reasoning error)
• $r = -3$: Generated all other illegal equations (format errors, parsing failures)
• $r = -1.5$ (GP-VL only): Failed to correctly recognize the given cards (visual recognition error—applied as additional penalty on top of whatever the equation reward would be)

Why these specific reward magnitudes: the graduated penalties create a reward landscape that distinguishes between "close" errors (-1, where the model understands the task but miscalculates) and "fundamental" errors (-2, -3, where the model doesn't understand the constraints). This is important for RL training because it provides shaped rewards that guide the policy toward improvement even when it cannot immediately achieve the +5 success reward. The additional -1.5 penalty for visual recognition failure in GP-VL ensures that the model cannot achieve high reward by guessing equations without correctly recognizing the cards—the visual perception and reasoning components are jointly optimized.

Output format (Figure 11, Figure 12). The model is required to produce structured JSON output:

{
    "cards": ["A", "3", "K", "6"],
    "number": [1, 3, 13, 6],
    "formula": "(1+6)*3+13=24"
}

The cards field requires the model to identify which four cards are present (a visual recognition subtask in GP-VL). The number field requires mapping card symbols to their numerical values according to the active rule. The formula field requires constructing a valid arithmetic expression. This structured output format enables the rule-based verifier to programmatically check correctness at each level: did the model recognize the right cards? Did it map them to the right numbers? Did it produce a valid equation that evaluates to 24?

The V-IRL Environment: Spatial Reasoning with Real-World Visual Input

V-IRL (Yang et al., 2024a) is an open-world visual navigation environment that the paper adopts to study spatial reasoning generalization. Unlike the synthetic arithmetic domain of GeneralPoints, V-IRL uses real-world street-view imagery and natural-language navigation instructions, providing a more ecologically valid test of visual and spatial reasoning.

Core task mechanics. The model receives a sequence of navigation instructions (e.g., "First, turn slightly right towards the northeast...") and a 2×2 grid of street-view images showing the current location from four cardinal directions. At each step, the model must observe the visual input, identify landmarks mentioned in the instructions, and output one of several possible navigation actions. The goal is to navigate from a starting point to a destination by following the instructions, requiring the model to recognize landmarks in the visual input, relate them to the textual instructions, and select appropriate spatial actions.

The concept of a route (Appendix B.3). A route is the fundamental navigation object in V-IRL. Each route corresponds to a real-world path and contains:

• Destination: the target location to reach
• Starting point: the initial position
• Turning points: intersections where the navigator must change direction
• Straight roads: road segments connecting turning points, the start, and the destination
• Street views: 360-degree panoramic images (presented as 2×2 grids) at each movable point
• Oracle information: expert observation data for each point (what landmarks are visible from which direction)
• Expert trajectory: the correct sequence of actions to reach the destination
• Instruction: natural language description of the route

Action space variants (Section 4.2). The environment supports two distinct action space configurations that form the rule variants for generalization testing:

• Absolute orientation: Actions are cardinal/intercardinal directions: turn_direction(x) where $x \in \{\text{'north', 'northeast', 'east', 'southeast', 'south', 'southwest', 'west', 'northwest'}\}$, plus forward() and stop(). This action space uses a fixed global coordinate system.
• Relative orientation: Actions are egocentric turns: turn_direction(x) where $x \in \{\text{'left', 'right', 'slightly left', 'slightly right'}\}$, plus forward() and stop(). The relative actions adjust the current orientation by 90° (left/right) or 45° (slightly left/slightly right).

What makes this a distribution shift: the two action spaces require fundamentally different spatial reasoning. In absolute orientation, the model must maintain a global mental map (knowing that "northeast" means something specific regardless of current facing direction). In relative orientation, the model reasons egocentrically (knowing that "left" depends on which way it's currently facing). A model trained only on absolute directions must learn to reinterpret the same spatial relationships through a completely different coordinate frame. As the paper notes in Section 4.2:

"This relative configuration adjusts the current orientation by 90 degrees or 45 degrees to the left or right, respectively."

This is a more dramatic shift than the GeneralPoints rule variant—it changes not just the mapping of symbols to values but the entire spatial reasoning framework.

Visual challenges in V-IRL-VL. The key visual challenge is landmark recognition: the model must identify specific businesses (restaurants, shops, landmarks like "The Dutch," "Lola Taverna," "Shuka") from the 2×2 street-view image grid and match them to the landmarks named in the instructions. The visual input is a 2×2 grid with headings [front, right, back, left], requiring the model to understand both the visual content and the spatial layout of the grid.

Language-only variant (V-IRL-L). The paper also uses a pure-language version where the visual observations are replaced with text descriptions (e.g., "Hotel 32One is on your right behind; You observe an intersection"). This is generated by parsing the visual input into text using the original V-IRL pipeline. The V-IRL-L variant isolates the spatial reasoning component from the visual recognition component, analogous to how GP-L isolates arithmetic reasoning from card recognition.

Simplifications from the original V-IRL (Appendix B.3). The paper makes two modifications to accommodate RL training:

1. Eliminating the 2-stage navigation pipeline: The original V-IRL used a separate visual detector for street-view processing before the language model. The paper removes this, making the VLM responsible for all visual processing end-to-end. This simplification is necessary because RL training requires a differentiable (or at least end-to-end optimizable) pipeline.
2. Removing online queries: The original system queried external APIs during navigation. The paper pre-computes all necessary information, reducing training time and eliminating external dependencies.

Maximum straight road length: The paper introduces a configurable parameter to limit the number of consecutive forward() actions on a straight road. Without this limit, long straight segments would require the model to output many repetitive forward() actions, which is uninteresting from a reasoning perspective and wastes training compute.

Reward design (Appendix B.3). The reward function is simpler than GeneralPoints because navigation actions are either correct or incorrect:

• $r = +1$: Generated a correct action at the current coordinate (matched the expert trajectory)
• $r = -1$: Generated a wrong action at the current coordinate
• $r = -1$: Exceeded the maximum verification step without reaching the destination
• $r = -1.5$: Failed to detect landmarks correctly (visual recognition error)

The episode terminates when either the agent calls stop() at the destination or the maximum verification step of 2 is reached.

Why only 2 maximum verification steps in V-IRL vs. 5 in GeneralPoints: navigation is a sequential task where errors compound—if the model takes a wrong turn, it's now in a completely different location than the instructions assume, and subsequent actions are meaningless regardless of correctness. Two steps provides one chance for self-correction without wasting compute on unrecoverable trajectories. In contrast, GeneralPoints equation-solving is a "reset-free" task where each attempt is independent—five attempts are sensible because failure doesn't corrupt the state.

Training and evaluation data (Appendix B.1). The training database contains 1000 unique routes from New York City. Rule-variant experiments evaluate on randomly sampled routes from this same database (ID uses the same action space as training, OOD uses the alternative action space). Visual-variant experiments evaluate on the V-IRL VLN mini benchmark (Yang et al., 2024a), which consists of 18 distinct routes across 9 cities worldwide (Milan, New Delhi, Buenos Aires, London, Hong Kong, New York, Melbourne, Lagos, San Francisco), with 2 routes per city. The NYC routes in the benchmark do not overlap with the training data.

SFT Training: Supervised Fine-Tuning on Expert Demonstrations

Supervised fine-tuning (SFT) is the first stage of the post-training pipeline and serves dual purposes: it produces the SFT-trained models used in the comparison against RL, and it initializes the model parameters before RL training.

Training data construction (Appendix C.1). SFT data is created by pairing each task's system prompt (which includes task description, rules, and input) with the corresponding expert response. For GeneralPoints, the expert response is the correct equation in the structured JSON format. For V-IRL, the expert response is the correct action at each position along the route.

Optimal single-turn data (default). The primary SFT experiments use optimal single-turn prompt-response pairs—the system prompt is paired with the correct answer directly, without any verification steps, error messages, or revision history. This means the SFT model is trained only on the "happy path": prompt → correct answer.

What this choice implies: the SFT model never sees examples of the model making mistakes and then correcting them. It only learns the mapping from problem statements to correct solutions. This is a deliberate design choice that makes the SFT-RL comparison a fair test of the learning paradigm, not the data. SFT sees expert trajectories; RL sees its own (initially suboptimal) trajectories with reward signals. If SFT with expert-only data could generalize as well as RL with verifier feedback, that would be evidence that explicit correction examples aren't necessary. The finding that SFT does not generalize suggests that merely observing correct solutions is insufficient—the model needs to learn from its own mistakes.

Suboptimal trajectory SFT (ablation, Appendix C.1, Figure 15). To test whether SFT's failure to generalize is due to the data (expert-only single-turn) rather than the learning paradigm itself, the paper also conducts experiments with SFT on sub-optimal trajectories—training examples that include the model's errors and verifier feedback, similar to what the RL model sees during training. The results show that even with this richer data, "SFT still merely memorizes the training data with degraded out-of-distribution performance." This is a critical control experiment: it demonstrates that the memorization effect is a property of the SFT training paradigm (maximum likelihood on fixed data), not an artifact of using only expert demonstrations.

Training configuration. All experiments use Llama-3.2-Vision-11B (Dubey et al., 2024) as the backbone model. SFT training tunes all model components with a shared learning rate per experiment. The paper reports ablating learning rates in the range $\{1 \times 10^{-4}, 1 \times 10^{-6}, 5 \times 10^{-7}, 1 \times 10^{-7}\}$ for the GP-VL experiments (Appendix D.1, Figure 16), with additional experiments freezing the vision encoder or freezing both the vision encoder and adapter. Training is conducted on 8 H800 GPUs (80GB each).

SFT as initialization for RL. In all RL experiments reported in the main body, the RL training starts from an SFT-initialized checkpoint, not from the raw pretrained model. This follows the standard RLHF pipeline (Ouyang et al., 2022) and RL4VLM (Zhai et al., 2024a). The reason is investigated in Section 5.4 (and discussed further below under RL training).

The SFT scaling baseline. When the paper refers to "scaling SFT" or "SFT performance" in the main results (Figure 5), this means continuing SFT training beyond the initialization point—training on more expert demonstration data. The RL scaling baseline starts from the same initialization and then applies RL training. Both are measured in terms of total training FLOPs, enabling a fair comparison of how each method uses additional compute.

RL Training: PPO with Sequential Revision and Outcome-Based Rewards

RL training is the second stage of the experimental pipeline and is the primary method of interest for studying generalization.

Algorithm choice: PPO. The paper uses Proximal Policy Optimization (PPO; Schulman et al., 2017) as the backbone RL algorithm. PPO is a policy gradient method that updates the policy (the model parameters) using collected trajectories while constraining the update size to prevent destructive large parameter changes. The choice of PPO is consistent with prior work on RL for language models (Ziegler et al., 2019; Ouyang et al., 2022; Zhai et al., 2024a).

Why PPO over other RL algorithms: PPO is the standard choice for fine-tuning large language models with RL because (a) it is relatively stable compared to vanilla policy gradient, (b) it handles the high-variance reward signals that arise from sparse outcome-based rewards, and (c) it has been extensively validated in the RLHF literature. The paper does not claim algorithmic novelty—PPO is adopted as a well-understood tool for studying the generalization properties of RL training.

Training dynamics. The RL training proceeds in iterative stages of replay buffer collection and optimization:

1. Rollout collection: The current policy (model) interacts with the environment for multiple episodes, generating trajectories of states, actions, and rewards using the sequential revision formulation. Each episode consists of up to $T$ turns (where $T$ is the maximum verification steps: 5 for GeneralPoints, 2 for V-IRL), with each turn generating a complete output sequence and receiving verifier feedback.
2. Replay buffer: The collected trajectories (states, actions, rewards, verifier feedback) are stored in a replay buffer. The buffer contains both successful and unsuccessful episodes, providing both positive and negative examples for policy improvement.
3. Policy optimization: PPO updates the model parameters using the trajectories in the replay buffer, computing advantage estimates and applying clipped policy updates.
4. Repeat: Steps 1–3 are repeated, with the updated policy generating new trajectories for the next iteration. This is the standard on-policy RL loop: the policy improves, generates better trajectories, improves further, and so on.

Replay buffer size and inference cost (Appendix C.3). The PPO algorithm requires additional inference computation beyond the training FLOPs because each iteration involves generating new trajectories from the current policy. The paper estimates this cost as:

$D_{\text{buffer}} \approx \frac{E \cdot \bar{d}_i \cdot \bar{d}_o}{D_{\text{RL}}} \cdot D_{\text{RL}} = \lambda D_{\text{RL}}$

where $E \in \mathbb{N}$ is the number of autoregressive generation processes, $\bar{d}_i$ and $\bar{d}_o$ are the average input and output token counts, and $D_{\text{RL}}$ is the number of tokens used during RL optimization.

Why this matters for the FLOP comparison: the total RL training compute includes both the optimization FLOPs (updating model parameters) and the inference FLOPs (generating trajectories). The paper estimates $\lambda \approx 6$ for GeneralPoints and $\lambda \approx 5.1$ for V-IRL, meaning the inference cost is roughly 5–6 times the optimization cost. This is factored into the total training FLOPs estimates used in Figure 5, ensuring that the SFT-vs-RL comparison is fair—both methods are measured by their total computational cost, not just their optimization cost.

Total training FLOP estimation (Appendix C.3). The paper estimates FLOPs following the standard scaling laws formulation (Hoffmann et al., 2023; Snell et al., 2024):

$X_{\text{train}} = 6 N D_{\text{train}}$ $X_{\text{inference}} = 2 N D_{\text{inference}}$

where $N$ is the number of model parameters and $D$ represents token counts. For SFT:

$X_{\text{SFT}} = 6 N (D_{\text{init}} + D_{\text{SFT}})$

For RL (including inference):

$X_{\text{RL}} = 6 N (D_{\text{init}} + D_{\text{RL}}) + 2 N D_{\text{buffer}}$

where $D_{\text{init}}$ is the number of tokens used in pretraining the base model. The factor 6 in the training term comes from the forward pass (2) and backward pass (4) operations per token; the factor 2 in the inference term comes from forward pass only.

What this computes: the total floating-point operations consumed by each training method, starting from the pretrained checkpoint. This enables the apples-to-apples comparison in Figure 5, where the x-axis is "Training Computation (GFLOPs)" and both SFT and RL curves start from the same initialization point.

Why this FLOP accounting matters: without accounting for RL's inference cost, one might incorrectly conclude that RL is more compute-efficient than it actually is. By including the replay buffer generation cost, the paper ensures that any RL advantage in generalization is achieved at a comparable (or even higher) computational cost to SFT—ruling out the explanation that RL simply sees more data.

Hyperparameter search (Appendix D.1). For RL experiments on GP-VL, the paper searched learning rates $\{2 \times 10^{-6}, 1 \times 10^{-6}\}$ and found that "finding suitable hyperparameters for RL experiments requires minimal effort" (Figure 17). All model components (vision encoder, adapter, language model) are tunable during RL training.

The necessity of SFT initialization (Section 5.4, Figure 9). A critical finding is that RL training fails entirely when applied to the raw pretrained Llama-3.2-Vision-11B model without SFT initialization. The paper reports:

"Without SFT, the base model suffers from poor instruction following capability. A detailed failure case is provided in Figure 20 (in Appendix D.3), revealing that the base Llama-3.2-Vision-11B model tends to generate long, tangential, and unstructured responses. This issue makes it impossible to retrieve task-related information and rewards for RL training."

The failure mode is specific: the pretrained model does not produce the structured JSON output required by the verifier. Instead, it generates free-form text (e.g., "To solve this problem, we can use a brute force approach..."), sometimes attempting to write Python code, and rarely finishing within the context length. Since the verifier expects specific JSON fields (cards, number, formula), it cannot extract reward signals from unstructured outputs, and RL receives no useful training signal.

Why this is an important boundary condition: it demonstrates that RL's generalization capability is not unconditional—it requires a model that can already follow the basic output format and task structure. SFT provides this scaffolding by teaching the model the expected output format, which then enables RL to optimize the content of that output. The paper explicitly notes that this finding does not contradict DeepSeekAI et al. (2025), which found SFT unnecessary for downstream RL; the difference is attributed to the backbone model (Llama-3.2-Vision-11B vs. DeepSeek's models).

Limits of RL from overfitted checkpoints (Section 6, Figure 19). Conversely, the paper finds that RL cannot recover OOD performance when starting from an extremely overfitted SFT checkpoint where the model has memorized the training rule so strongly that it cannot adapt. In V-IRL-VL, when initialized from a checkpoint with less than 1% OOD per-step accuracy, RL training produces no improvement in OOD performance. Figure 21 shows a failure case where the model "collapses to the training rule"—it outputs absolute-direction actions (turn_direction(northwest)) even when the task requires relative-direction actions (left, right). The SFT memorization has effectively locked in the training distribution's action format, and RL's exploration mechanism cannot escape this local optimum.

The "Goldilocks zone" for SFT initialization. Taken together, these findings define a narrow window of SFT initialization quality that enables effective RL: the SFT must be sufficient to teach the output format and basic task structure (so RL can receive reward signals), but not so extensive that the model overfits to the training distribution and loses the capacity to adapt to new variants. The paper characterizes this as an open question: "Further research is needed to delineate the conditions under which SFT facilitates effective RL" (Section 6).

Evaluation Protocol: Measuring Generalization Through Controlled Distribution Shifts

The evaluation protocol is the methodological core of the paper—it is what converts the SFT-vs-RL comparison from a simple performance benchmark into a study of generalization versus memorization.

In-distribution (ID) evaluation. For each task and variant, ID evaluation measures performance on the same configuration used during training. For GeneralPoints: training and evaluation both use the rule where J/Q/K count as 10. For V-IRL: training and evaluation both use the absolute orientation action space. ID performance measures how well each method solves the task it was trained on—this is the standard benchmark metric, and both SFT and RL should improve it with more compute.

Out-of-distribution (OOD) evaluation: rule variants. OOD evaluation changes the task rules while keeping the underlying reasoning capability the same:

• GeneralPoints (Section 4.1): Train with J/Q/K all as 10; test with J=11, Q=12, K=13, and require at least one face card. The arithmetic operations are identical; only the symbolic-to-numeric mapping changes.
• V-IRL (Section 4.2): Train with absolute orientation actions (north, northeast, etc.); test with relative orientation actions (left, right, slightly left, slightly right). The spatial reasoning is identical (navigate from start to destination following instructions); only the coordinate system changes.

Out-of-distribution evaluation: visual variants (Section 5.2). OOD evaluation changes visual surface features while keeping the task and rules identical:

• GeneralPoints: Train on black-suited cards (♠, ♣); test on red-suited cards (♥, ♦). The task and rules are identical; only the color of the card symbols changes.
• V-IRL: Train on navigation routes in New York City; test on routes from 9 cities worldwide (Milan, New Delhi, Buenos Aires, London, Hong Kong, Melbourne, Lagos, San Francisco). The navigation task is identical; only the visual appearance of streets, buildings, and landmarks changes.

The generalization gap metric. The key analytical metric is the difference between ID and OOD performance—the "generalization gap." A method that learns generalizable rules should show:

• ID performance improving (or at least not degrading) with more compute
• OOD performance also improving (or at least not degrading) with more compute
• A narrow or closing gap between ID and OOD

A method that memorizes should show:

• ID performance improving with more compute
• OOD performance degrading (or failing to improve) with more compute
• A widening gap between ID and OOD

The paper's central empirical claim is that RL exhibits the first pattern and SFT exhibits the second.

Metrics (Appendix C.3).

• Success rate (GeneralPoints, V-IRL overall): For GeneralPoints, success is defined as producing a correct equation at least once during the inference-time verification steps. For V-IRL overall success, the model must take correct actions at every movable point on the route—an extremely demanding metric since errors compound multiplicatively across steps.
• Per-step accuracy (V-IRL): An individual step is considered correct when the model's chosen action matches the expert trajectory action at that position. This metric is more granular than overall success rate and is used as the primary metric for V-IRL in Figures 5 and 6 because the overall success rate is too low to show meaningful differentiation between methods (the paper reports in Appendix D.2, Figure 18 that both SFT and RL achieve <1% overall success rate on V-IRL-VL OOD).
• Visual recognition accuracy (GP-VL, Section 5.3): The fraction of episodes where the model correctly identifies the four cards from the input image. This is computed separately from the equation-solving success rate to disentangle visual perception from arithmetic reasoning in the analysis of Section 5.3.

Computation estimation for fair comparison (Appendix C.3). The x-axis in all performance-over-compute plots (Figures 5, 7, 8, 10) is "Training Computation (GFLOPs)," estimated using the FLOP formulas described above. Both SFT and RL curves start from the same initialization checkpoint (marked as "Init" in the figures), and the computation cost includes all training operations from that point onward. For RL, this includes the inference cost of replay buffer generation. This ensures the comparison answers the question: "Given the same compute budget beyond the initialization point, does SFT or RL produce better generalization?"

Cross-validation and statistical reporting. The paper states that all evaluated data points are assumed to follow a binomial distribution, with standard error approximated as $\sqrt{P(1-P)/N}$, where $P$ is the decimal success rate and $N$ is the number of samples. Line plots use Savitzky–Golay filtering with polynomial order 3 for smoothing visualization. The paper does not report train/test splits within the evaluation data (the tasks are procedurally generated, so train/test contamination is addressed by the distribution-shift design rather than by a fixed held-out set).

Verification iterations as an evaluation parameter (Section 5.5). During evaluation, the number of verification iterations (the maximum turns allowed) can be varied independently of the training configuration. The paper studies this in Figure 10, where the same RL-trained model is evaluated with $\text{VIter} \in \{1, 3, 5, 10\}$ verification steps. This tests whether the generalization benefits of RL are amplified by scaling inference-time compute—more verification steps mean more opportunities to self-correct using the skills acquired during RL training. The finding that OOD improvement grows from +0.48% (1 step) to +5.99% (10 steps) suggests that RL's generalization advantage compounds with inference-time computation, connecting the paper's findings to the broader literature on test-time compute scaling.

Summary of Design Choices and Their Justifications

Rule-based verifier over learned reward model: eliminates reward model error as a confound; ensures that differences between SFT and RL are attributable to the learning paradigm, not noise in the training signal.

Turn-level (not token-level) action space: makes RL tractable for 11B-parameter models by reducing the effective horizon and enabling credit assignment at the granularity of complete reasoning attempts.

Sequential revision over parallel sampling for RL: enables the model to learn from its own mistakes iteratively, which is hypothesized to be the mechanism through which RL acquires generalizable rules—by repeatedly correcting errors, the model learns the invariant structure of the task rather than surface patterns.

Expert-only SFT data (with suboptimal-trajectory ablation): isolates the effect of the learning paradigm from the data distribution. The suboptimal-trajectory ablation confirms that SFT's memorization is a property of maximum-likelihood training, not of seeing only correct answers.

Two complementary task domains (arithmetic + spatial navigation): tests whether the SFT-memorization / RL-generalization finding is domain-specific or a general property of these post-training paradigms. The consistent results across both domains strengthen the paper's central claim.

Controlled distribution shifts (rules + visuals) rather than new tasks: enables clean measurement of generalization by keeping the core capability identical while changing surface features. Generalization to entirely new tasks would confound capability transfer with rule learning.

FLOP-matched comparison with buffer cost included: ensures the SFT-vs-RL comparison is fair—both methods are allocated comparable total compute, and RL's inference cost is not hidden.

SFT initialization for all RL experiments: follows standard practice (RLHF, RL4VLM) and enables studying the complementary roles of the two methods. The ablation showing RL fails without SFT validates this design choice and characterizes an important boundary condition.

Multiple verification iterations during evaluation: connects the paper's findings to test-time compute scaling and shows that RL's generalization advantage is not just a training phenomenon but compounds with additional inference-time reasoning.

4. Key Insights and Innovations

Innovation 1: A Controlled Experimental Framework for Disentangling Memorization from Generalization in Post-Training

The paper's most fundamental contribution is not any single empirical result but rather the experimental methodology itself—a clean, controlled framework for testing whether a post-training paradigm produces transferable knowledge or surface-level memorization. Prior work on memorization versus generalization in foundation models (Carlini et al., 2022; Wang et al., 2024; Qi et al., 2024) largely studied pretrained models or evaluated generalization to entirely different tasks, which confounds capability transfer with rule learning. The question of what SFT and RL each contribute to generalization remained unanswered because the field lacked experimental designs that could isolate these effects.

What makes this paper's approach distinctive is the construction of distribution shifts that preserve the underlying reasoning capability while varying only surface features. In GeneralPoints, the shift from "J/Q/K all equal 10" to "J=11, Q=12, K=13" changes the symbolic-to-numeric mapping but preserves the arithmetic reasoning requirement. In V-IRL, the shift from absolute to relative orientation action spaces changes the coordinate system but preserves the spatial navigation requirement. These are not new tasks—they are the same task with a different interface. A model that has learned the invariant structure (arithmetic operations, spatial relationships) transfers; a model that has learned surface correlations (specific card-to-number mappings, specific direction words) fails.

This represents a fundamental methodological advance over prior comparisons. Previous studies comparing SFT and RL (or studying each in isolation) relied on in-distribution test sets where memorization and generalization are indistinguishable. The paper's framework provides what amounts to a litmus test for generalization: measure performance on a training-distribution variant, then measure performance on a rule or visual variant that requires the same underlying capability. The difference between these measurements—the generalization gap—reveals what the model actually learned. This framework is generalizable to any reasoning task with configurable rules, making it a reusable tool for the field rather than a one-off analysis.

The evidence for this framework's diagnostic power appears throughout Figures 5–8, where SFT and RL show dramatically different generalization gaps despite comparable in-distribution performance. The framework's validity is further supported by the suboptimal-trajectory SFT ablation (Appendix C.1, Figure 15), which demonstrates that SFT's failure to generalize persists even when trained on data containing errors and corrections—ruling out the confound that SFT's memorization was an artifact of seeing only expert demonstrations.

Innovation 2: The Empirical Finding That RL and SFT Produce Qualitatively Different Kinds of Knowledge—and the Characterization of Their Complementary Roles

Prior work in the post-training literature operated under an implicit assumption that SFT and RL are alternative optimization methods for achieving similar ends—both ultimately fine-tune the model toward correct behavior, just through different training signals (demonstrations versus rewards). FLAN (Wei et al., 2022a) showed SFT produces zero-shot generalization; RLHF (Ouyang et al., 2022) showed RL produces aligned behavior. But no prior study systematically asked: do these methods teach the model fundamentally different things?

This paper's central empirical finding is that they do. RL learns generalizable rules that transfer across distribution shifts; SFT primarily memorizes surface patterns that fail under distribution shifts. This is not a matter of degree—it is a qualitative difference in what is learned. The evidence is stark: across both GeneralPoints and V-IRL, in both unimodal (LLM) and multimodal (VLM) settings, RL consistently improves OOD performance while SFT consistently degrades it (Figure 6). For instance, on V-IRL-L, RL improves OOD per-step accuracy from 80.8% to 91.8%, while SFT collapses from 80.8% to 1.3%. These are not incremental differences—they represent opposite directions of effect.

What makes this finding intellectually significant beyond the raw numbers is that it resolves an ambiguity that pervaded the post-training literature. When prior work reported that SFT produced "good generalization" (FLAN) or that RL was "effective for alignment" (RLHF), these claims were made without distinguishing whether the model had learned rules or patterns. This paper shows that SFT's impressive in-distribution performance is largely a product of memorization—which happens to work on test sets drawn from the same distribution—while RL's performance reflects genuine rule acquisition. This reframes how the field should interpret post-training results: in-distribution accuracy is an unreliable proxy for capability, and any claim of generalization must be tested under controlled distribution shifts.

The complementary roles finding (Section 5.4) adds further depth. SFT is not merely memorization—it serves an essential scaffolding function by stabilizing the model's output format, without which RL cannot even begin training (Figure 9). This finding, combined with the observation that excessive SFT locks in memorized patterns that RL cannot overcome (Section 6, Figures 19, 21), defines a Goldilocks zone for SFT initialization: enough SFT to teach the output format, but not so much that the model overfits to the training distribution. This is a practically actionable insight for anyone building post-training pipelines—SFT and RL are not interchangeable stages to be tuned independently but must be carefully balanced to achieve the best of both: format adherence from SFT and rule generalization from RL.

Innovation 3: RL as a Mechanism for Improving Visual Recognition in VLMs—A Byproduct of Generalizable Learning

A distinctive and somewhat surprising finding is that RL training improves the VLM's underlying visual recognition capabilities, even when the reward signal only evaluates the final reasoning output (equation correctness, navigation action correctness), not visual accuracy directly. This is shown in Figure 8 (Section 5.3), where scaling RL compute simultaneously improves both visual recognition accuracy (correctly identifying the four cards from the image) and overall task success rate, across both in-distribution and OOD settings. Scaling SFT, in contrast, degrades visual recognition accuracy.

This finding is conceptually significant because it inverts the expected causal direction. Prior work on improving VLM visual capabilities focused on the SFT stage—curating better visual data (Chen et al., 2023; Liu et al., 2024), improving the training recipe by unfreezing visual encoders (Liu et al., 2023; Tong et al., 2024a), or combining multiple vision encoders (Tong et al., 2024d; Kar et al., 2025). The implicit assumption was that visual perception must be improved before or during SFT, and that RL merely optimizes decision-making given already-perceived visual inputs.

The paper's finding challenges this assumption: RL, trained only with an outcome-based reward on the final reasoning task, produces improvements in visual recognition as a byproduct. The mechanism is not explicitly verified, but the implication is clear—the model learns that accurate visual recognition is instrumentally necessary for achieving high reward, and the RL optimization process strengthens the visual representations accordingly. This is an emergent property of the RL training dynamics, not something explicitly programmed into the reward function (the GP-VL reward does include a -1.5 penalty for recognition failure, but this is just a scalar signal; the model must discover the connection between visual accuracy and reward on its own).

The significance extends beyond this paper: it suggests that RL may be an underexplored tool for improving multimodal perception in foundation models, and that the conventional pipeline of "perfect perception, then optimize decision-making" may be unnecessarily sequential. RL can jointly optimize perception and reasoning, with improvements in each component reinforcing the other—a finding that could reshape how multimodal post-training pipelines are designed.

Innovation 4: The Identification of Verification Iterations as a Multiplier on RL's Generalization Advantage

The paper establishes a direct connection between inference-time compute and generalization through the finding that scaling verification iterations amplifies RL's OOD improvement (Section 5.5, Figure 10). Under the same RL training budget, models evaluated with 10 verification steps show +5.99% OOD improvement, while those evaluated with only 1 verification step show a marginal +0.48% improvement. This is not simply "more computation at test time helps"—that has been shown before in the test-time compute scaling literature (Snell et al., 2024; Jaech et al., 2024). What is distinctive is the interaction between training method and inference-time computation: the benefit of additional verification steps is conditional on the model having been trained with RL rather than SFT.

The implication is that RL does not just learn generalizable rules; it learns a capacity for iterative self-correction that compounds with inference-time compute. SFT, by memorizing surface patterns, produces models that cannot effectively use additional verification steps because their errors are not structured—the model lacks a generalizable error-correction strategy that transfers across distribution shifts. RL, by learning the underlying task structure through trial-and-error, develops a self-correction capability that scales with the number of revision opportunities.

This finding connects the paper's work to two major research directions simultaneously. First, it provides a mechanistic explanation for why RL generalizes: the sequential revision process during RL training teaches the model to use feedback to correct errors, and this meta-skill transfers to OOD settings. Second, it suggests that inference-time compute scaling laws may be fundamentally intertwined with training methodology—the optimal allocation of test-time compute may depend on whether the model was trained with RL or SFT, a consideration absent from prior work on compute-optimal inference (Snell et al., 2024), which studied models post-trained with a single method.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper uses two environments with procedurally generated data rather than a fixed static dataset. GeneralPoints samples four-card quadruples from a standard 52-card poker deck, with an expert solver ensuring each sampled set has at least one solution equaling 24 (Appendix A.1). The number of unique training samples is not explicitly reported but is effectively unbounded since cards are sampled procedurally during training. The V-IRL training database consists of 1000 unique routes from New York City, with the VLN mini benchmark (Yang et al., 2024a) used for visual OOD evaluation comprising 18 distinct routes across 9 cities (Milan, New Delhi, Buenos Aires, London, Hong Kong, New York, Melbourne, Lagos, San Francisco), with 2 routes per city. The NYC routes in this benchmark do not overlap with the training set (Appendix B.1). The SFT training data for both environments consists of optimal single-turn prompt-response pairs, pairing each system prompt with its corresponding expert response (Appendix C.1).

• Base model(s). All experiments use Llama-3.2-Vision-11B (Dubey et al., 2024), an approximately 11 billion parameter vision-language model. The authors do not explicitly justify this choice beyond it being a representative open-source VLM, but the implicit rationale is clear: the model is large enough to perform non-trivial reasoning on these tasks (the SFT initialization achieves non-zero success rates in all settings), yet small enough that training on 8 H800 GPUs is feasible. The model supports both text-only input (used for GP-L and V-IRL-L variants) and image-plus-text input (used for GP-VL and V-IRL-VL variants), enabling the modality comparison.

• Metrics. Three metrics are reported across different experiments (Appendix C.3):

  • Success rate (GeneralPoints, V-IRL overall): For GeneralPoints, success is defined as producing a correct equation (evaluating to 24) at least once during the inference-time verification steps. An episode with 5 verification steps that produces a correct answer on step 3 counts as a success. For V-IRL overall success, the model must take correct actions at every movable point on the route—a trajectory-level metric where a single wrong action fails the entire episode. The paper reports that both SFT and RL achieve <1% overall success rate on V-IRL-VL OOD, making the metric too sparse for meaningful comparison (Appendix D.2, Figure 18).
  • Per-step accuracy (V-IRL): An individual step is considered correct when the model's chosen action matches the expert trajectory action at that position. This metric treats each intermediate verification step as an independent sample, providing greater statistical resolution than overall success rate for V-IRL. This is the primary metric reported for V-IRL in the main figures.
  • Visual recognition accuracy (GP-VL): The fraction of episodes where the model correctly identifies all four cards from the input image, assessed from the cards field in the JSON output. This metric isolates visual perception from arithmetic reasoning in the Section 5.3 analysis. Standard error for all metrics is approximated assuming binomial distributions: √(P(1-P)/N) where P is the decimal success rate and N is the number of samples (Appendix C.3). Line plots apply Savitzky–Golay filtering with polynomial order 3 for smoothing.

• Baselines. The paper establishes the SFT-initialized checkpoint (marked "Init" in all figures) as the shared starting point for both SFT and RL scaling. SFT performance is measured as SFT training continues from this initialization; RL performance is measured as RL training proceeds from the same initialization. This makes the comparison a direct test of how each method uses additional compute beyond the shared starting point. The paper does not include external baseline methods (e.g., a different RL algorithm, a different SFT configuration) beyond the suboptimal-trajectory SFT ablation. For the V-IRL visual OOD experiment, the previous state-of-the-art from Yang et al. (2024a) is included as a reference point (44.0% on the VLN mini benchmark, using a two-stage VLM-LLM collaboration with GPT-4).

• Generation budget / compute accounting. All performance-over-compute plots use "Training Computation (GFLOPs)" on the x-axis, estimated using the scaling laws formulation from Hoffmann et al. (2023) and Snell et al. (2024) (Appendix C.3). The FLOP estimation is:

  • For SFT: X_SFT = 6N(D_init + D_SFT) where N is model parameters, D_init is pretraining tokens, and D_SFT is supervised fine-tuning tokens.
  • For RL: X_RL = 6N(D_init + D_RL) + 2N·D_buffer, where the additional 2N·D_buffer term accounts for the inference cost of generating trajectories during the on-policy PPO replay buffer collection. The paper estimates D_buffer ≈ λ·D_RL with λ ≈ 6 for GeneralPoints and λ ≈ 5.1 for V-IRL, based on average input/output token counts and the number of autoregressive generation processes per training iteration. This FLOP accounting ensures the comparison is fair—RL's additional inference cost is not hidden, and both SFT and RL curves start from the same D_init point. Training is conducted on 8 H800 GPUs (80GB each). The paper does not report wall-clock time, only FLOPs.

• Cross-validation / statistical protocol. The paper does not use traditional train/validation/test splits because both environments generate data procedurally. For GeneralPoints, each evaluation episode draws a new set of four cards, and distribution shift is achieved by changing the sampling parameters (e.g., requiring face cards for OOD) rather than holding out examples. For V-IRL, the training data is the 1000-route NYC database, and visual OOD evaluation uses entirely separate routes from different cities. The paper reports standard errors assuming binomial distributions and applies Savitzky–Golay filtering for visualization, but does not report confidence intervals on the compute-optimal scaling curves or conduct statistical significance tests between SFT and RL performance at matched compute budgets. The GP-VL hyperparameter search (Figure 16) tests 10 different SFT configurations (varying learning rates and frozen components) and 2 RL configurations (learning rates 2×10⁻⁶, 1×10⁻⁶), confirming that no SFT configuration reaches the in-distribution success rate of RL.

Main Quantitative Results

Rule-Based Generalization: RL Generalizes, SFT Memorizes

The headline result from Figure 5 and Figure 6 is that RL consistently improves OOD performance across all four task variants (GP-L, V-IRL-L, GP-VL, V-IRL-VL), while SFT consistently degrades OOD performance, even as both methods improve in-distribution performance.

GeneralPoints language-only (GP-L). Figure 5, top-left panel (ID) and bottom-left panel (OOD), with final values in Figure 6 (leftmost bar chart):

• SFT initialization (Init): 11.5% OOD success rate
• After scaling SFT compute: OOD drops to 3.4% (a −8.1 percentage point decrease)
• After scaling RL compute (matched total FLOPs to SFT): OOD rises to 15.0% (a +3.5 percentage point increase)

The in-distribution performance (Figure 5, top-left) tells a more nuanced story: SFT initially improves ID performance more rapidly than RL at low compute budgets, but the ID gap narrows as compute scales. At the highest compute levels shown (approximately 1×10¹⁰ GFLOPs), RL's ID performance appears competitive with or slightly below SFT's ID performance—but RL achieves this while simultaneously improving OOD, whereas SFT's ID gains come at the cost of OOD collapse.

V-IRL language-only (V-IRL-L). Figure 5, second column from left; Figure 6, second bar chart from left:

• Init: 80.8% OOD per-step accuracy
• After scaling SFT: OOD collapses to 1.3% (−79.5 percentage points)
• After scaling RL: OOD rises to 91.8% (+11.0 percentage points)

This is the most dramatic gap in the entire paper. SFT's near-total collapse on V-IRL-L OOD—from 80.8% to 1.3%—deserves careful attention. The V-IRL-L OOD shift from absolute to relative orientation action space represents a more fundamental change than the GeneralPoints rule shift: the model must completely restructure its action generation, mapping spatial instructions onto an egocentric rather than allocentric coordinate frame. The 80.8% initialization performance suggests the base SFT model had already acquired some ability to handle absolute directions; SFT's further training on absolute-direction data actively destroys this ability for relative directions, while RL training strengthens it.

GeneralPoints vision-language (GP-VL). Figure 5, third column; Figure 6, third bar chart:

• Init: 11.2% OOD success rate
• After scaling SFT: 5.6% (−5.6 percentage points)
• After scaling RL: 14.2% (+3.0 percentage points)

The SFT performance on GP-VL shows an unusual pattern in the ID panel (Figure 5, top row, third column): SFT ID performance fails to reach the levels achieved by RL. This is the only task variant where SFT's in-distribution performance is notably inferior to RL's. The paper explicitly acknowledges this as a "failure of SFT on GP-VL" in Section 6 and hypothesizes that "SFT locally overfits to reasoning tokens while neglecting recognition tokens, possibly due to the higher frequency of reasoning tokens." The ID gap is visible in Figure 5 top row: SFT plateaus at roughly 20–25% while RL reaches approximately 40–55% at comparable compute.

V-IRL vision-language (V-IRL-VL). Figure 5, fourth column; Figure 6, rightmost bar chart:

• Init: 35.7% OOD per-step accuracy
• After scaling SFT: 2.5% (−33.2 percentage points)
• After scaling RL: 45.0% (+9.3 percentage points)

The V-IRL-VL results are reported as per-step accuracy rather than success rate because the overall success rate is too low to show meaningful differentiation—Appendix D.2, Figure 18 shows both methods achieving <1% overall success rate on V-IRL-VL OOD. The per-step accuracy metric provides sufficient resolution to observe the SFT/RL divergence.

Key cross-task patterns. Several patterns emerge consistently across all four task variants:

1. RL always improves OOD performance relative to initialization, with gains ranging from +3.0% (GP-VL) to +11.0% (V-IRL-L).
2. SFT always degrades OOD performance relative to initialization, with drops ranging from −5.6% (GP-VL) to −79.5% (V-IRL-L).
3. The initialization point (SFT checkpoint before scaling either method) provides a meaningful baseline—it achieves non-trivial OOD performance in all cases (ranging from 11.2% to 80.8%), confirming that the base SFT initialization has not yet overfit to the training rule.
4. The SFT degradation is most severe on V-IRL tasks (−79.5% and −33.2%) compared to GeneralPoints tasks (−8.1% and −5.6%), suggesting that the memorization effect is amplified when the distribution shift is more dramatic (changing the entire coordinate system vs. changing a few card values).

Visual Generalization: RL Generalizes to Unseen Visual Domains

Section 5.2 tests whether the RL-generalization / SFT-memorization pattern extends to visual distribution shifts—changes in surface visual features while the task rules remain identical. The headline results from Figure 7:

GeneralPoints visual OOD (GP-VL, train on black suits, test on red suits).

• Init: 23.6% OOD success rate
• After scaling SFT: 13.7% (−9.9 percentage points)
• After scaling RL: 41.2% (+17.6 percentage points)

The visual shift in GP-VL is relatively mild—only the color of the card symbols changes, not the card identities or the arithmetic task. Despite this, SFT still degrades, though less severely than in the rule-shift case (−9.9% vs. −5.6% for the same task's rule shift). RL's improvement is substantial (+17.6%), nearly double the gain seen in GP-VL rule OOD (+3.0%). This asymmetry is noteworthy: RL benefits more from visual generalization than rule generalization in GP-VL, while SFT's degradation is worse under visual shift.

V-IRL visual OOD (V-IRL-VL, train on NYC routes, test on worldwide routes).

• Init: 16.7% success rate (note: this is success rate, not per-step accuracy—Figure 7 reports success rate for V-IRL-VL visual OOD, unlike Figure 6 which reports per-step accuracy for rule OOD)
• After scaling SFT: 11.1% (−5.6 percentage points)
• After scaling RL: 77.8% (+61.1 percentage points)
• Previous state-of-the-art (Yang et al., 2024a): 44.0%

The +61.1 percentage point improvement from RL on V-IRL visual OOD is the single largest gain reported in the paper, and it establishes a new state-of-the-art on the V-IRL VLN mini benchmark by a margin of +33.8% (44.0% → 77.8%). The paper notes that the previous SOTA used a "two stage VLM-LLM collaboration technique and tailored prompt engineering on closed-sourced model (GPT-4)," while the RL approach "enables an open-sourced model (Llama-3.2-Vision-11B) to reach superior performance" (Section 5.2). This is the only external baseline comparison in the paper, and it substantially strengthens the claim that RL produces genuinely useful generalization, not just relative advantage over a weak SFT baseline.

Interpreting the visual OOD results in context. The visual shift in V-IRL is a more realistic and challenging test of generalization than the GeneralPoints color shift: the model must recognize landmarks in entirely different cities with different architectural styles, signage, lighting conditions, and street layouts. The fact that RL achieves 77.8% success rate—meaning the model successfully navigates 14 out of 18 routes correctly end-to-end—on routes from cities it was never trained on, suggests that RL has learned something quite general about the relationship between visual observations, textual instructions, and navigation actions.

However, the paper does not fully disentangle how much of this 77.8% comes from the multi-turn RL formulation (which allows iterative self-correction during evaluation) versus the specific learning dynamics of RL training. The previous SOTA of 44.0% was achieved with a single-stage evaluation, so the comparison is not perfectly like-for-like. The paper's multi-turn evaluation (up to 2 verification steps in V-IRL) gives the RL-trained model an advantage—it can recover from an initial mistake, while the GPT-4 baseline (presumably) could not. This does not invalidate the result, but it should be understood as reflecting both the training method advantage and the inference-time compute advantage.

Visual Recognition Improvements: A Byproduct of RL Training

Section 5.3 investigates why RL improves visual OOD performance by directly measuring visual recognition accuracy alongside task success rate in GP-VL. Figure 8 presents a combined visualization showing both metrics as a function of training compute, with separate panels for rule-variant and visual-variant OOD settings.

Key observations from Figure 8:

• RL improves visual recognition as a byproduct of task-level RL training. The red points (in-distribution) and blue points (OOD) for RL both show simultaneous increases in recognition accuracy (y-axis) and success rate (x-axis) as training compute increases (indicated by increasing point opacity). The final RL checkpoint achieves both the highest recognition accuracy and the highest success rate.
• SFT degrades visual recognition. The SFT trajectory shows decreasing recognition accuracy as compute scales, moving leftward and downward. This is consistent with the hypothesis that SFT overfits to reasoning-specific patterns while the visual representations degrade.
• Recognition and success rate are correlated. The connected (⋆−◦) pairs in Figure 8, which represent the same checkpoints evaluated under both ID and OOD conditions, show that recognition accuracy tracks roughly linearly with success rate. The paper states: "the VLM's visual recognition accuracy largely affects the overall performance, which was similarly observed in Zhong et al. (2024)."

The paper hypothesizes that SFT's visual degradation occurs because "SFT locally overfits to reasoning tokens while neglecting recognition tokens, possibly due to the higher frequency of reasoning tokens" (Section 6, referencing Figure 11). The visual recognition tokens (identifying cards in the image) appear only once per episode, while reasoning tokens (constructing the equation) dominate the output. Maximum-likelihood training on next-token prediction naturally allocates capacity to predicting the most frequent token patterns—which in this case are the reasoning tokens, not the recognition tokens.

For RL, the mechanism is different: the reward function penalizes both recognition errors (−1.5 penalty) and reasoning errors (−1, −2, −3 penalties), so the policy optimization process jointly improves both capabilities. Since the verifier can only compute the equation reward if the cards are correctly identified, the RL training effectively discovers that visual accuracy is a prerequisite for task success, and the gradient signal flows back through the entire generation process, including the visual recognition step encoded in the cards field of the JSON output.

The Role of SFT in RL Training: Necessary but Insufficient

Section 5.4 addresses a practical question: is SFT necessary, or can RL be applied directly to the pretrained model? Figure 9 shows three RL training runs on GP-L starting from the raw pretrained Llama-3.2-Vision-11B (no SFT initialization), with different PPO hyperparameter settings. All three runs fail entirely—the success rate remains at approximately 0.0–0.4% throughout training, showing no improvement trend.

The failure mode is documented in Figure 20 (Appendix D.3): the base model generates long, unstructured responses that do not conform to the required JSON output format. An example output begins: "To solve this problem, we can use a brute force approach by generating all possible combinations of the cards and checking if any of them can be used to form an equation that equals 24. Here is a Python solution for this problem..." The model attempts to write code rather than output structured JSON, and "fails to finish within finite context length." Since the rule-based verifier expects specific JSON fields (cards, number, formula), it cannot extract any reward signal from these outputs, and RL receives no useful training gradient.

The complementary roles. The paper establishes a clear division of labor:

• SFT teaches the output format and basic task structure, making RL training possible.
• RL teaches generalizable rules that SFT cannot acquire, enabling OOD transfer.

This is not merely a restatement of LIMA's "format teacher" hypothesis (Zhou et al., 2024a)—it extends it by showing that SFT's formatting role is necessary for RL but that SFT cannot itself produce the generalizable knowledge that RL can. The paper explicitly notes that this finding does not contradict DeepSeekAI et al. (2025)'s claim that SFT is unnecessary for downstream RL, attributing the difference to the backbone model choice.

The overfitting boundary (Section 6, Figure 19). Conversely, the paper shows that RL cannot recover from an overfitted SFT checkpoint. When initialized from a V-IRL-VL checkpoint with <1% OOD per-step accuracy (achieved by excessive SFT), RL training produces flat OOD performance—no improvement whatsoever. Figure 21 shows the failure mode: the model outputs absolute-direction actions (turn_direction(northwest)) even when the task prompt specifies relative-direction actions (left, right, slightly left, slightly right). The SFT training has locked in the training distribution's action format so strongly that RL's exploration cannot escape.

This defines a feasible window for RL: the SFT initialization must be strong enough to produce parseable outputs (above the "format threshold") but not so strong that it memorizes the training distribution's specifics (below the "overfitting threshold"). The paper does not characterize this window quantitatively—no metrics are provided for measuring "sufficient but not excessive" SFT—but the boundary conditions are clearly demonstrated.

Verification Iterations: Scaling Inference-Time Compute Amplifies RL's Generalization

Section 5.5 tests whether the number of verification steps (the maximum turns allowed per episode during evaluation) affects the generalization performance of RL-trained models. Figure 10 reports the OOD performance growth (y-axis) versus ID performance growth (x-axis) for RL experiments on GP-L with VIter ∈ {1, 3, 5, 10}, as training compute scales (indicated by point opacity).

The key finding: more verification iterations produce larger OOD improvements for the same RL training budget. Specifically:

• VIter = 1: +0.48% OOD improvement (marginal)
• VIter = 3: +2.15% OOD improvement
• VIter = 5: +2.99% OOD improvement
• VIter = 10: +5.99% OOD improvement

The ID growth also increases with more verification steps, but the relationship is less pronounced—the ID growth for VIter=10 reaches approximately 12% while VIter=1 reaches approximately 4%, suggesting that ID performance also benefits from more verification opportunities, but OOD benefits disproportionately. The paper describes this as "scaling up verification improves generalization" (Section 5.5).

Connection to inference-time compute. This finding links the paper's work to the test-time compute scaling literature (Snell et al., 2024; Jaech et al., 2024). The number of verification iterations is an inference-time compute parameter—more iterations mean more tokens generated and more verifier calls per problem. The finding that OOD generalization improves with more verification iterations suggests that RL's advantage is not solely a training-time phenomenon but is amplified by how the model is used at inference time. A model trained with RL and evaluated with many verification steps achieves substantially better OOD performance than the same RL-trained model evaluated with few verification steps.

The paper does not explicitly compare SFT with varying verification iterations (i.e., does SFT also benefit from more verification steps, just less than RL?), which would strengthen the claim that the interaction between training method and inference-time compute is specific to RL. Figure 10 only shows RL experiments.

Ablation Studies and Robustness Checks

Suboptimal-trajectory SFT (Appendix C.1, Figure 15): Training SFT on data that includes model errors, verifier feedback, and revision steps—similar to what RL sees during training—does not rescue SFT's generalization. The results show that SFT trained on these suboptimal trajectories still "merely memorizes the training data with degraded out-of-distribution performance," with ID performance improving to ~90% while OOD remains flat or degrades. This ablation rules out the confound that SFT's memorization is caused by seeing only expert demonstrations—even when SFT sees failure-and-correction trajectories, maximum-likelihood training fails to extract generalizable rules from them.

GP-VL SFT hyperparameter sweep (Appendix D.1, Figure 16): Ten different SFT configurations for GP-VL were tested, varying learning rates (1×10⁻⁴ to 1×10⁻⁷) and which components are trainable (all components, frozen vision encoder, frozen vision encoder + adapter). None of the ten configurations achieves an in-distribution success rate above approximately 30%, and none shows an increasing trend comparable to RL's trajectory. This comprehensive sweep addresses the concern that the GP-VL SFT failure might be a hyperparameter artifact—the degradation appears robust across a wide range of SFT settings.

GP-VL RL hyperparameter sweep (Appendix D.1, Figure 17): In contrast to SFT's sensitivity, RL achieves strong and consistent improvements with minimal tuning. Two learning rates (2×10⁻⁶ and 1×10⁻⁶) were tested, and both show similar increasing trends, with the model reaching approximately 40–50% in-distribution success rate. The paper notes that "finding suitable hyperparameters for RL experiments requires minimal effort."

Overall success rate for V-IRL-VL (Appendix D.2, Figure 18): The paper reports the trajectory-level success rate (all actions correct from start to destination) as complementary to the per-step accuracy metric. Both SFT and RL achieve <1% overall success rate on V-IRL-VL OOD, while RL achieves slightly higher ID success rate. The near-zero OOD overall success rate for both methods highlights the difficulty of V-IRL as a sequential decision-making task—even RL's per-step accuracy of 45.0% translates to essentially zero trajectory-level success because errors compound multiplicatively across ~10 steps.

RL from overfitted SFT checkpoint (Section 6, Figure 19): As described in the main results, this negative result shows that RL cannot recover OOD performance when starting from a severely overfitted SFT checkpoint. The per-step accuracy remains flat at <1% across approximately 3.5×10⁹ GFLOPs of RL training compute. A representative failure case in Figure 21 shows the model generating turn_direction(northwest) when the task requires relative-direction actions—the overfitted SFT has locked the model into the training action space format.

End-to-end RL without SFT (Section 5.4, Figure 9): All three RL runs starting directly from the pretrained Llama-3.2-Vision-11B (without SFT initialization) fail with success rates near zero throughout training. The failure mode in Figure 20 shows the model cannot follow the required output format. This ablation establishes that SFT's role as a format stabilizer is necessary for the specific backbone model used—Llama-3.2-Vision-11B does not have sufficient instruction-following capability out of the box for RL to succeed.

Replay buffer FLOP accounting (Appendix C.3): The paper's FLOP estimation for RL explicitly includes the inference cost of replay buffer generation, with estimated multipliers λ ≈ 6 for GeneralPoints and λ ≈ 5.1 for V-IRL. This accounting prevents the comparison from being biased in RL's favor by hiding its additional inference cost. The paper does not verify these λ estimates empirically (e.g., by measuring actual inference FLOPs during training), but the methodology is transparent.

Critical Assessment

Claim: RL generalizes to OOD rule variants while SFT memorizes training data.

This is the paper's central claim, and the evidence supporting it is strong across all four task variants. The consistency of the pattern—RL always improves OOD, SFT always degrades OOD—across two fundamentally different reasoning tasks (arithmetic, spatial navigation) and two modalities (LLM, VLM) makes the finding unlikely to be an artifact of any particular experimental configuration. The suboptimal-trajectory SFT ablation (Figure 15) strengthens the claim by ruling out the most obvious confound (that SFT fails because it only sees correct answers).

However, two limitations temper the claim's generality:

First, the "SFT memorizes" characterization is demonstrated for a specific SFT recipe: continued fine-tuning on the same in-distribution data beyond the initialization point, with all model parameters tuned, on a fixed set of expert demonstrations. The paper does not explore SFT variants that might produce better generalization—for instance, SFT on a mixture of ID and OOD examples, SFT with data augmentation (paraphrased prompts, varied card presentations), or SFT with early stopping before the overfitting occurs. These are not exotic techniques—they are standard practices in the fine-tuning literature. The paper's finding that SFT can memorize does not demonstrate that SFT necessarily memorizes; it demonstrates that under the specific training protocol used, memorization is the default outcome.

Second, the paper does not characterize the generalization performance of the SFT initialization checkpoint (the "Init" point in all figures) before any scaling. The Init model is itself an SFT-trained model, and in several cases it achieves non-trivial OOD performance (80.8% on V-IRL-L, 35.7% on V-IRL-VL, 23.6% on GP-VL visual OOD). This suggests that the initial SFT training (before the scaling comparison begins) did produce some generalizable knowledge. The paper frames the comparison as "scaling SFT vs. scaling RL from the same initialization," but the initialization itself demonstrates that SFT is not inherently incapable of generalization—only that continued SFT beyond a certain point erodes it. An experiment tracking the full trajectory of SFT from the pretrained model (not just from the shared initialization onward) would clarify whether there is an optimal SFT stopping point where generalization peaks before the memorization regime begins.

Claim: RL improves visual recognition capabilities in VLMs.

The evidence from Figure 8 is suggestive but correlational. The paper shows that recognition accuracy and success rate improve together under RL and degrade together under SFT, but it does not demonstrate a causal mechanism. Specifically, the paper does not:

• Measure recognition accuracy for intermediate RL checkpoints to trace the trajectory of visual improvement
• Conduct an ablation where the RL reward function excludes the visual recognition penalty (−1.5) to test whether visual improvement is driven by the explicit recognition reward or emerges from task-level optimization
• Compare RL against a baseline that explicitly optimizes visual recognition (e.g., SFT with a recognition-specific auxiliary loss) to see whether RL's visual improvement is competitive with direct supervision

The claim in Section 5.3 that "scaling RL compute also improves visual recognition accuracy, as a byproduct of its generalization capability" is stated rather than demonstrated—the word "byproduct" implies a causal mechanism that the experiments do not isolate. The correlation could equally support the reverse causal direction: RL improves task performance by improving visual recognition, and the task improvement is not a "byproduct" of generalization but the primary mechanism through which RL achieves its gains.

Claim: The multi-turn RL approach achieves state-of-the-art on V-IRL VLN mini benchmark (+33.8%).

This claim is accurate in absolute terms—77.8% beats 44.0%—but the comparison is not like-for-like in at least one important respect. The previous SOTA (Yang et al., 2024a) used GPT-4 with a two-stage pipeline and tailored prompt engineering, but the paper does not specify whether that system also had access to multi-turn verification or iterative self-correction during evaluation. The RL-trained model benefits from up to 2 verification steps during evaluation (as specified in the sequential revision formulation), meaning it can recover from an initial wrong action. If the GPT-4 baseline was evaluated with single-turn generation only, then part of the 33.8% improvement is attributable to the inference-time compute advantage (verification + revision during evaluation) rather than purely to RL training. A fairer comparison would report single-turn performance of the RL-trained model or allow multi-turn evaluation for the GPT-4 baseline.

Additionally, the paper does not report whether the GPT-4 baseline had access to the same training data (the 1000 NYC routes). If GPT-4 was evaluated zero-shot on the worldwide routes, while the RL model was trained on 1000 NYC routes (which, while not overlapping with the test routes, provide relevant navigation experience), then the comparison conflates training data availability with training methodology.

Claim: SFT is necessary for effective RL training on Llama-3.2.

The evidence for this claim is direct and compelling: RL without SFT initialization fails completely (Figure 9, Figure 20). However, the paper's interpretation—that SFT teaches output formatting—is inferred from the failure mode (the model generates unstructured text) rather than demonstrated through an experiment that explicitly teaches output formatting through an alternative means. If SFT's role is purely format stabilization, then a lightweight formatting intervention (e.g., few-shot prompting with the correct JSON format, or a small amount of format-only SFT data) should suffice to enable RL. The paper does not test this hypothesis, leaving open the possibility that SFT contributes more than just format—perhaps it also provides the initial policy with some task-relevant knowledge that RL subsequently refines and generalizes.

Claim: Scaling verification iterations improves generalization.

Figure 10 demonstrates this for RL on GP-L, but the paper does not show a corresponding SFT curve. Without knowing whether SFT also benefits from more verification iterations (even if from a lower baseline), the claim that verification iterations specifically amplify RL's generalization advantage is incomplete. If SFT with VIter=10 also shows improvement over SFT with VIter=1 (just less than RL's improvement), the interaction between training method and inference-time compute would be quantitative rather than qualitative. The paper's narrative implies a qualitative interaction but only provides one side of the comparison.

Missing experiments that would strengthen the paper:

• Verification iteration sweep for SFT. Parallel to Figure 10 but for SFT-trained models, to test whether the interaction between verification steps and OOD performance is specific to RL.
• RL with alternative reward functions. For GP-VL, training RL without the visual recognition penalty (−1.5) to test whether visual improvement is reward-driven or emergent.
• SFT with data diversity interventions. Training SFT on a mixture of ID and OOD data, on augmented data (paraphrased prompts), or with regularization (early stopping, weight decay) to test whether "SFT memorizes" is an inherent property or a consequence of the specific SFT protocol.
• Single-turn evaluation of the RL model on V-IRL. To disentangle training-method effects from inference-time compute effects in the SOTA comparison.
• Full SFT trajectory from pretrained model. To identify whether there exists an SFT checkpoint (earlier than the shared initialization) where generalization is optimal, characterizing the overfitting dynamics more completely.
• Statistical significance tests or confidence intervals. The paper reports standard errors for individual data points but never tests whether the differences between SFT and RL at matched compute budgets are statistically significant. With large per-step-accuracy differences like 80.8% → 1.3% (SFT) vs. 80.8% → 91.8% (RL), significance is likely not in doubt, but formal reporting would strengthen the quantitative claims.

Overall assessment. The paper's experimental design is clean and its central empirical finding—that RL and SFT produce qualitatively different generalization behavior under the tested conditions—is well-supported. The consistency across four task variants and the inclusion of several important controls (suboptimal-trajectory SFT, hyperparameter sweeps, FLOP-matched compute accounting) make the core claim persuasive. However, the paper's interpretation sometimes extends beyond what the experiments directly demonstrate, particularly regarding the causal mechanisms behind RL's visual improvement and the claimed necessity of SFT's formatting role. The paper is best understood as establishing a robust phenomenon—RL generalizes better than SFT under distribution shifts—while leaving the mechanisms as hypotheses supported by circumstantial evidence rather than direct causal tests.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Not Accounted for in the Generalization Gains

The assumption or constraint. The entire compute-optimal framework rests on estimating prompt difficulty before allocating the inference-time budget. The paper's method for doing so—generating 2048 samples per question and averaging verifier scores—is extraordinarily expensive, consuming more compute than the largest test-time budgets studied. The authors acknowledge this in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

This is a significant gap between the reported efficiency gains and practical deployment cost.

The consequence. The 4× efficiency gains reported in Figures 4 and 8 are computed after difficulty is known, without amortizing the cost of acquiring that knowledge. In a realistic deployment, the total cost would be difficulty estimation + strategy execution, and the former—requiring 2048 samples per prompt at minimum—could dominate. If difficulty estimation costs more than the test-time budget being optimized, the efficiency gains are illusory. The paper frames this as an exploration-exploitation tradeoff but provides no empirical quantification of the total cost when estimation is included, leaving practitioners unable to assess the net benefit.

What evidence exists in the paper. The paper provides no experiment that includes difficulty estimation cost in the budget accounting. The predicted (non-oracle) difficulty bins perform nearly as well as oracle bins (Figures 4 and 8), which demonstrates that the approach is feasible without ground-truth labels, but it does not address the cost of obtaining those predictions. The 2048-sample estimation protocol is described in Section 3.2 but never included in any FLOPs budget calculation.

Mitigation status. The authors explicitly flag this as "a key avenue for future work" (Section 3.2), suggesting that future systems could train models to predict difficulty directly from the question text. However, no such model is developed, trained, or evaluated. The paper also mentions the possibility of amortizing difficulty estimation across multiple evaluations of the same prompt, but this applies only to repeated-use scenarios and not to one-off inference. Until the exploration cost is reduced or amortized, the reported efficiency gains should be understood as an upper bound rather than a realized deployment advantage.

All Results Are on a Single Model Family and Two Reasoning Tasks

The assumption or constraint. Every experiment in the paper uses Llama-3.2-Vision-11B (Dubey et al., 2024) as the backbone model, evaluated exclusively on two tasks: GeneralPoints (arithmetic reasoning) and V-IRL (spatial navigation). The authors state they "believe this model is representative of the capabilities of many contemporary LLMs" (Section 4), but this claim is untested.

The consequence. Several aspects of the findings could be model-specific:

• The PRM's over-optimization behavior depends on PaLM 2-S*'s output distribution. A model with different calibration properties might exhibit different difficulty-dependent scaling curves, potentially changing which strategies are optimal at which difficulty levels.
• The revision model's ability to learn from incorrect examples depends on the base model's in-context learning capabilities, which vary substantially across model families. A model with stronger or weaker in-context learning might show different sequential-vs-parallel tradeoffs.
• The finding that SFT is necessary for RL training (Section 5.4) is explicitly tied to Llama-3.2-Vision-11B's poor instruction-following out of the box. The authors acknowledge this is model-specific by noting that DeepSeekAI et al. (2025) found SFT unnecessary for their models, but the paper provides no evidence about where the boundary lies.

Both tasks involve symbolic or spatial reasoning with well-defined correctness criteria. It is unclear whether the difficulty-dependent patterns (beam search hurting easy problems, revisions helping easy problems) generalize to other reasoning domains—code generation, logical deduction, scientific QA—or to tasks requiring factual knowledge rather than inference. The MATH benchmark used for all experiments consists exclusively of competition-level math problems; the paper does not test on other reasoning benchmarks or on knowledge-intensive tasks.

What evidence exists in the paper. The paper provides no cross-model or cross-benchmark validation. All figures (Figures 3–10) are based on the same backbone model. The paper does not report whether preliminary experiments on other models or tasks were attempted or what the results were. The claim of representativeness is an assertion, not an empirical finding.

Mitigation status. Not addressed. The authors do not claim to have tested other models or benchmarks, and they do not discuss this as a limitation in Section 6 or Section 8 (future work). A practitioner considering deploying this approach with a different model family (e.g., LLaMA, Qwen, DeepSeek) or on a different task domain has no empirical basis for predicting whether the difficulty-conditioned allocation policies would transfer.

The Difficulty Binning Is Coarse, Static, and the Policy Selection Sample Size Is Small

The assumption or constraint. The paper discretizes continuous difficulty into five quintiles and selects a fixed strategy per bin. This coarse discretization means that questions at opposite ends of the same bin receive the identical strategy regardless of potentially meaningful difficulty differences. Furthermore, difficulty is estimated once and treated as static—there is no mechanism for dynamically adjusting strategy mid-computation based on intermediate verifier feedback. Most critically, the compute-optimal policy is selected based on a small sample: the 500-question test set is split into 5 difficulty bins of approximately 100 questions each, then further split by two-fold cross-validation, meaning the "best strategy" per bin is selected based on approximately 50 questions per fold.

The consequence. With only ~50 questions per fold per bin, the selected compute-optimal strategy may be sensitive to sampling noise. A few unusually easy or hard questions in a bin could shift the apparent optimal strategy, leading to suboptimal allocation when deployed on new questions. The paper does not report confidence intervals on the compute-optimal scaling curves (Figures 4, 8), making it impossible to assess whether the observed gains are statistically reliable at this sample size. The coarseness of the binning also means a question at the easy end of bin 3 and one at the hard end of bin 3 receive the same strategy, even though their true optimal strategies might differ—the discretization necessarily leaves some efficiency on the table.

What evidence exists in the paper. The paper reports results based on this binning scheme throughout (Figures 3 right, 4, 7 right, 8) but never reports the variance of the estimated optimal strategy or the sensitivity of results to the number of bins. The cross-validation protocol is described in Section 3.2 but the impact of fold size on strategy selection variance is not analyzed. Appendix C mentions binomial standard errors for individual data points but does not propagate this uncertainty to the strategy selection step.

Mitigation status. Not addressed. The paper does not experiment with alternative bin counts, continuous difficulty estimates, or dynamic adjustment policies. Section 8 mentions "adaptive difficulty estimation" as future work but does not discuss the small-sample issue. A practitioner with a larger or differently-distributed test set would need to re-derive the optimal policies, and the paper provides no guidance on minimum sample sizes needed for reliable policy selection.

Verifier Over-Optimization Is Documented but Unresolved, Capping the Scaling Ceiling

The assumption or constraint. The paper identifies verifier over-optimization as the primary bottleneck preventing unbounded improvements from additional test-time compute: beam search degrades easy-problem performance at high budgets (Figure 3, right), lookahead search paradoxically underperforms simpler methods (Figure 3, left), and qualitative examples show degenerate outputs that score highly under the PRM (Appendix M, Figures 29, etc.). The compute-optimal policy mitigates this by routing easy problems away from aggressive search, but it does not address the underlying problem: the verifier is not robust to adversarial optimization.

The consequence. On medium-difficulty problems where beam search is deployed, over-optimization still limits the scaling ceiling—the beam search curves in Figure 3 flatten and sometimes decline well before the generation budget is exhausted (e.g., beam search M=4 plateaus around 34% at 256 generations while best-of-N weighted continues improving to ~38%). This means the compute-optimal approach is fundamentally bounded by verifier quality, not by the search algorithm or the allocation policy. Improving the PRM—through better training data, adversarial robustness, or ensemble methods—would likely shift the difficulty thresholds and change the optimal policy. The paper provides no analysis of how sensitive the optimal allocation is to verifier quality, nor does it characterize the verifier's failure modes beyond qualitative examples.

Furthermore, the revision model's separate verifier (the revision-specific ORM described in Appendix J) was necessary because the base PRM did not transfer well to revision model outputs due to distribution shift (Figure 15a). This means the current system requires maintaining two separate verifiers—one for search and one for revisions—with no unified scoring mechanism. The paper does not explore whether this dual-verifier requirement is fundamental or mitigable.

What evidence exists in the paper. Figure 3 (right) directly shows over-optimization on easy problems: beam search accuracy decreases from approximately 78% to 77% as budget increases from 4 to 256 on difficulty bin 1. Figure 3 (left) shows lookahead search consistently underperforming. Appendix M provides qualitative examples of degenerate outputs. Figure 15a documents the distribution shift problem requiring a separate revision verifier.

Mitigation status. The paper explicitly acknowledges this in Section 8: "improving verifier robustness is the key bottleneck for further scaling test-time compute, not improving search algorithms." However, no experiments address verifier robustness—no adversarial training, no ensemble verification, no constrained search with KL penalties. The dual-verifier requirement is documented but not resolved. The paper identifies the bottleneck without taking steps to widen it, leaving the scaling ceiling exactly where it found it.

Sequential Revision Latency Is Ignored Despite Favoring Sequential-Heavy Strategies on Easy Problems

The assumption or constraint. The paper measures computation in "generations" (number of complete solutions sampled), which is a reasonable proxy for total FLOPs but ignores wall-clock latency. Sequential revisions—where each revision conditions on the previous one—are inherently serial, while parallel best-of-N can be executed simultaneously given sufficient hardware. The compute-optimal policy on easy problems (Figure 7, right, bins 1–2) favors highly sequential strategies: easy questions perform best with purely sequential revisions, and even medium questions show optimal ratios with more sequential than parallel sampling.

The consequence. A strategy that allocates 128 generations as 64 sequential × 2 parallel takes approximately 64× longer wall-clock time than one that runs 128 parallel samples simultaneously. For latency-sensitive applications—interactive assistants, real-time decision-making, live tutoring systems—the sequential-heavy strategies favored by the compute-optimal policy on easy and medium problems may be impractical regardless of their FLOPs efficiency. The paper's 4× efficiency claim is measured in generations (FLOPs), not in time-to-answer, and the two measures diverge substantially when serial dependencies dominate.

This tradeoff is particularly sharp because the compute-optimal policy routes easy problems—which are presumably the most common in many deployments—toward the most sequentially-dependent strategies. A system that is 4× more FLOP-efficient but 64× slower may be unacceptable in practice, effectively limiting the approach to batch or offline settings where latency is not a constraint.

What evidence exists in the paper. The paper never reports wall-clock time, latency measurements, or throughput numbers. The revision model's serial dependency is described in Section 6.1 and illustrated in Figure 5 (right panel), but the latency implications are never quantified or discussed. The FLOPs-matched comparison in Section 7 and all figures use generation count or FLOPs as the cost metric.

Mitigation status. Not addressed. The paper does not mention latency, throughput, or time-to-answer as considerations. Section 8 (future work) discusses combining search and revisions and improving verifier robustness, but does not mention latency-aware allocation. A practitioner building a real-time system would need to re-derive compute-optimal policies with a wall-clock-time cost model rather than a generation-count cost model, and the optimal policies would likely shift toward more parallel strategies, potentially reducing the reported efficiency gains.

The 14× Larger Model Baseline in the FLOPs-Matched Comparison Is Not Compute-Optimally Trained

The assumption or constraint. The FLOPs-matched comparison in Section 7 scales model parameters by approximately 14× while holding training data fixed, following the LLaMA paradigm (Touvron et al., 2023) rather than the Chinchilla-optimal paradigm where both parameters and data are scaled equally (Hoffmann et al., 2022). The authors acknowledge this:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

Additionally, the 14× larger model is evaluated using only greedy decoding—no best-of-N, no majority voting, no search, no revisions. This is a deliberately weak baseline: the larger model is given no test-time compute budget, while the smaller model receives a substantial one.

The consequence. A Chinchilla-optimal model trained with 14× more total FLOPs (scaling both parameters and training tokens optimally) would likely outperform a parameter-only-scaled model, making the pretraining baseline stronger than the one used. The reported advantages of test-time compute over pretraining—for example, +27.8% relative improvement on easy questions at R ≪ 1 for revisions (Figure 9, left; Figure 1, top-right bar chart)—may shrink or reverse against a properly compute-optimal larger model. Similarly, giving the larger model even a modest test-time compute budget (e.g., best-of-8 or 4 sequential revisions) would create a much stronger baseline that is never tested. The paper demonstrates that test-time compute can beat a 14× larger model evaluated greedily, not that it can beat a 14× larger model that is itself augmented with test-time compute.

What evidence exists in the paper. The FLOPs-matched results are in Figure 9 and the bar charts in Figure 1. The paper is transparent about the parameter-only scaling choice but does not provide evidence about how much the results would change under compute-optimal pretraining. The greedy-decoding baseline for the larger model is not explicitly justified; it appears to be assumed as the default evaluation protocol.

Mitigation status. The authors flag this as future work in Section 7. However, the paper's central framing—that "test-time compute can be more effective than scaling pretraining" (Section 1, Section 7 takeaway box)—is stated without the caveat that the pretraining scaling was suboptimal. A more accurate framing would be: "test-time compute with a smaller model can outperform a 14× larger parameter-only-scaled model using greedy decoding, on easy-to-medium problems at low inference-to-pretraining token ratios." The current framing risks overstating the finding's generality to practitioners who assume "scaling pretraining" means compute-optimal scaling.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the conversation around foundation model post-training from a practice-oriented question ("which method should I use?") toward a mechanistic question about the nature of knowledge acquisition: do different training paradigms produce qualitatively different kinds of knowledge from the same data? The finding that SFT and RL diverge not in degree but in kind—SFT producing surface-level memorization that collapses under distribution shift while RL produces transferable rule learning that strengthens—represents a reframing of the post-training problem. It is not a paradigm shift in the sense of introducing a new algorithm or architecture, but it is a diagnostic advance that changes how the field should evaluate and interpret post-training results.

The specific shift is this: in-distribution accuracy is no longer a sufficient proxy for post-training quality. Prior work routinely reported SFT and RL performance on held-out test sets from the same distribution, implicitly treating both methods as optimizing the same objective (correctness) through different means. This paper demonstrates that this conflation masks a fundamental asymmetry—SFT's in-distribution gains are largely a product of memorization that happens to transfer to identically-distributed test examples, while RL's gains reflect genuine rule acquisition. A researcher reading this paper should come away questioning whether any reported post-training improvement, absent controlled distribution-shift evaluation, actually represents learned capability rather than memorized patterns.

The paper also provides a unifying explanation for apparently contradictory findings in the literature. Prior work documented both that SFT produces impressive zero-shot generalization (FLAN; Wei et al., 2022a) and that SFT primarily serves as a "format teacher" (LIMA; Zhou et al., 2024a). Prior work on memorization versus generalization showed LLMs overfit on simple tasks and generalize on complex ones (Wang et al., 2024; Qi et al., 2024). The paper's difficulty-conditioned framework reconciles these: SFT's generalization in FLAN was measured across entirely different tasks, where surface-level memorization cannot transfer, forcing the model to rely on pretrained capabilities that SFT merely formats. In contrast, this paper tests generalization across controlled variants of the same task, where surface patterns are partially preserved, revealing SFT's reliance on those patterns. The contradictory findings were an artifact of different evaluation protocols—new-task generalization versus rule-variant generalization—and this paper provides the conceptual vocabulary to distinguish them.

Perhaps most consequentially, the paper redefines the role of SFT in the post-training pipeline. It establishes that SFT serves an essential but bounded function: stabilizing output format to enable subsequent RL. SFT cannot itself produce generalizable knowledge (Section 5.1), but RL cannot function without SFT's formatting scaffolding (Section 5.4), and excessive SFT locks in memorized patterns that RL cannot subsequently overcome (Section 6). This three-part characterization—format stabilizer, not generalizer, and a double-edged sword that can prevent subsequent generalization—gives practitioners a concrete framework for reasoning about SFT-RL tradeoffs that did not exist before. The finding that DeepSeekAI et al. (2025) reached a different conclusion (SFT unnecessary for RL) using different backbone models suggests that the "format threshold" varies by model family, opening a line of inquiry into which pretrained models require SFT scaffolding and which do not.

The paper also strengthens the case for RL as a first-class tool for improving multimodal perception. The finding that task-level RL training improves visual recognition accuracy—even when the reward function provides no direct supervision on visual outputs—challenges the conventional wisdom that visual perception must be fixed during SFT before decision-making optimization begins. If RL can jointly optimize perception and reasoning through outcome-based rewards alone, the standard "perceive, then reason, then act" pipeline may be unnecessarily sequential. This finding is too preliminary to be considered a settled result—the mechanism is not demonstrated, only correlated—but it opens a research direction that could reshape multimodal training pipelines.

Finally, the paper connects two previously-separate research threads: post-training methodology and inference-time compute scaling. The finding that verification iterations amplify RL's OOD generalization (Section 5.5, Figure 10) suggests that the optimal allocation of test-time compute depends on how the model was trained, not just on the problem difficulty (as in Snell et al., 2024). A model trained with RL benefits disproportionately from additional verification steps because it has learned a generalizable error-correction strategy; a model trained with SFT, having memorized surface patterns, cannot effectively use additional revision opportunities. This implies that inference-time scaling laws are not training-method-agnostic—a consideration absent from prior work that studied test-time compute scaling on models post-trained with a single method.

Follow-Up Research This Work Enables

Causal isolation of the RL generalization mechanism through reward-function ablations. The paper demonstrates that RL generalizes and SFT does not, but it does not isolate why. A direct follow-up would decompose the RL training process to identify which component drives generalization. Concretely: train RL variants on GP-L where (a) the reward function provides only binary success/failure (r = +5 for correct, r = 0 otherwise), removing the graduated penalty structure that gives shaped rewards for "close" errors; (b) the sequential revision context is ablated—the model sees only the current prompt without previous attempts and verifier feedback, transforming multi-turn RL into repeated single-turn RL; (c) the verifier provides only scalar reward without textual feedback, removing the informational content of the verifier messages. If RL still generalizes under (a) but fails under (b) or (c), the mechanism is the iterative self-correction process rather than reward shaping. If RL fails under (a), then the graduated reward structure—not RL per se—is responsible for generalization, which would substantially narrow the paper's claimed advantage for RL. This experiment is directly suggested by the paper's framework but not executed.

Optimal SFT stopping point for maximal RL generalization. The paper demonstrates that RL from the shared SFT initialization produces generalization while continued SFT from the same point produces memorization (Figure 5), and that RL from an overfitted SFT checkpoint fails entirely (Section 6, Figure 19). These findings define a window but do not characterize it quantitatively. A systematic follow-up would: starting from the pretrained Llama-3.2-Vision-11B, train SFT checkpoints at regular intervals (e.g., every 1×10^9 GFLOPs), then run RL from each checkpoint and measure both ID and OOD performance after fixed RL budgets. This would produce a curve showing OOD generalization after RL as a function of SFT initialization quality, identifying the precise SFT checkpoint that maximizes downstream RL generalization. The paper's current evidence shows the endpoints (no SFT = RL fails; too much SFT = RL cannot recover) but not the shape of the function between them. If the optimal SFT checkpoint occurs very early—just enough to teach output formatting—that would suggest SFT's role is purely format-stabilization and that minimal SFT is ideal, directly informing production pipeline design.

Cross-model replication of the SFT-memorization / RL-generalization finding. The paper's findings are from a single model family (Llama-3.2-Vision-11B). A replication study testing 3–5 different pretrained VLMs of varying scales (e.g., LLaVA-1.5-7B, LLaVA-1.5-13B, Qwen-VL, InternVL2, and the base Llama-3.2 in 1B and 3B variants) on the same GeneralPoints OOD protocol would answer: (a) Is the SFT-memorization effect universal across model families and scales, or is it specific to Llama architectures or the 11B scale? (b) Does the format-stabilization threshold (where SFT becomes sufficient to enable RL) vary systematically with model size—do larger models need less SFT because they follow instructions better out of the box? (c) Does RL's generalization advantage diminish for models that already generalize well from pretraining, or does it compound with stronger base capabilities? The paper's claim that Llama-3.2 is "representative" (Section 4) currently lacks evidence. A negative result—finding a model family where SFT generalizes as well as RL—would not invalidate the paper but would refine the boundary conditions and identify which architectural or pretraining properties confer generalization robustness under SFT.

Extension of the verification-iteration scaling analysis to SFT-trained models and to V-IRL. Figure 10 shows that more verification iterations improve OOD generalization for RL on GP-L, but no corresponding SFT curve is provided, and the analysis is limited to GP-L. A direct extension would: (a) evaluate SFT-trained GP-L models with VIter ∈ {1, 3, 5, 10}, producing a companion figure to Figure 10 that reveals whether SFT also benefits from verification iterations (even from a lower baseline) or whether the benefit is specific to RL; (b) replicate the verification-iteration sweep on GP-VL and both V-IRL variants to test whether the interaction between training method and inference-time compute is task-general. If SFT shows zero benefit from additional verification iterations across all tasks, this would confirm a qualitative interaction—RL teaches a capability for iterative self-correction that SFT does not. If SFT shows some benefit (just less than RL), the interaction is quantitative, and the optimal verification budget becomes a joint function of training method and task difficulty—this would directly inform compute-optimal inference strategies that are training-method-aware.

Combined RL and SFT post-training with OOD data exposure. The paper's SFT protocol uses only in-distribution data (the training rule, the training visual domain). A natural follow-up tests whether SFT's memorization can be mitigated by exposing the model to distribution-shifted examples during SFT—not as a replacement for RL but as a supplementary data intervention. Specifically: compare (a) RL-only from the standard SFT initialization (the current best method), (b) SFT on a mixture of 80% ID / 20% OOD data (exposing the model to rule variants during SFT without RL), and (c) SFT on the same mixture followed by RL. The hypothesis—suggested but not tested by the paper—is that OOD exposure during SFT might prevent the overfitting that locks in memorized patterns, potentially enabling RL to achieve even better OOD generalization from the resulting checkpoint, or might itself produce modest generalization without RL. If (b) performs comparably to (a), the paper's central claim—that RL is necessary for generalization—would be falsified in favor of a data-diversity explanation. If (c) outperforms (a), SFT with OOD data exposure becomes a practical preprocessing step for RL pipelines.

Adversarial evaluation of the RL generalization: compound distribution shifts. The paper tests generalization along single axes: rule shift OR visual shift, never both simultaneously. A stress-test would evaluate RL-trained models under compound shifts—for GP-VL, testing with both the OOD rule (J=11, Q=12, K=13) AND the OOD visual domain (red suits) simultaneously, neither of which appeared together during training. For V-IRL-VL, testing relative-orientation actions on a new city (a combined rule + visual shift). The paper's framework makes this trivially implementable given the existing environment configuration options. If RL's generalization is truly compositional—the model has learned arithmetic operations abstracted from card values AND color-invariant visual recognition as separable skills—then compound-shift performance should be approximately the product of the two single-shift performances. If performance collapses disproportionately, it suggests RL's generalization is more brittle than it appears, relying on partial rather than complete abstraction. This stress-test directly addresses whether RL learns factorized generalizable knowledge or merely broader-but-still-correlated patterns, distinguishing between "RL generalizes" (the paper's claim) and "RL generalizes only when at least one surface feature matches training" (a weaker, more qualified claim).

Practical Applications and Downstream Use Cases

Production post-training pipelines for instruction-following VLMs. Organizations deploying VLMs for tasks with configurable rules or variable visual environments—navigation assistants, document understanding systems, visual question answering in specialized domains—should structure their post-training as minimal SFT followed by extensive RL, rather than the common practice of heavy SFT with optional RL at the end. The paper provides concrete numbers for the cost of getting this wrong: on V-IRL-L, scaling SFT beyond the initialization point destroys OOD performance from 80.8% to 1.3% (Figure 6), while RL improves it to 91.8%. For a navigation system that must operate in new cities or handle varied instruction formats, this difference is the gap between functional and useless. The practical takeaway is that SFT should be treated as a format initialization step—train only until the model produces reliably parseable outputs in the expected structure, then stop and switch to RL, monitoring OOD performance on a held-out rule or visual variant to detect the onset of SFT overfitting. The paper's failure case (Figure 20) provides a diagnostics template: if the base model without SFT generates unstructured text, continue SFT; if the SFT checkpoint shows decreasing OOD performance on a variant task, SFT has gone too far.

Visual recognition improvement without visual supervision. The finding that RL training on an outcome-based reward improves visual recognition accuracy (Figure 8, Section 5.3) has immediate implications for domains where visual ground-truth labels are expensive or unavailable. In medical imaging, for instance, annotating radiology images with bounding boxes or segmentation masks requires specialist expertise, but outcome labels (did the patient develop the condition?) are often available from electronic health records. The paper's results suggest that RL trained on outcome rewards alone could improve a VLM's ability to recognize clinically relevant features in images, even without explicit visual supervision, because the model discovers that accurate visual perception is instrumentally necessary for correct outcome prediction. The GP-VL result—where RL improved recognition accuracy on unseen card colors despite receiving only equation-correctness rewards—provides a proof of concept that this emergent visual improvement is possible. A cautious interpretation is warranted: the paper did not causally isolate the mechanism, and the visual shift tested (card color) is simpler than most real-world visual distribution shifts. But the finding is strong enough to motivate pilot studies in outcome-supervised visual domains.

Open-source VLMs matching or exceeding closed-source systems on spatial reasoning tasks. The V-IRL visual OOD result—Llama-3.2-Vision-11B achieving 77.8% on the VLN mini benchmark versus GPT-4's 44.0% (Section 5.2, Figure 7)—demonstrates that multi-turn RL with outcome-based rewards can enable an open-source 11B-parameter model to substantially outperform a closed-source system on a challenging spatial reasoning task. For organizations that cannot use proprietary APIs due to cost, privacy, or latency constraints, this result provides a concrete recipe: take an open-source VLM, apply minimal SFT for format stabilization, then train with RL using a task-specific verifier and multi-turn sequential revision. The 77.8% success rate means the model correctly navigates 14 out of 18 complex, multi-step routes in entirely unseen cities—a level of performance that makes the system practically deployable for applications like assistive navigation for visually impaired users, autonomous inventory management in warehouses, or virtual tourism guides. The key limitation is that this approach requires building a task-specific verifier and environment simulator for RL training; the paper's rule-based verifiers for GeneralPoints and V-IRL are relatively simple, but constructing equivalent verifiers for more open-ended tasks remains an engineering challenge.

Offline data generation for self-improvement through RL-filtered trajectories. The paper's finding that RL-trained models produce correct solutions on OOD variants—while SFT-trained models fail—suggests a data generation strategy for self-improvement loops. Specifically: use an RL-trained model to generate solutions on OOD task variants (where it achieves non-trivial success rates, e.g., 45.0% on V-IRL-VL OOD per-step accuracy), filter for correct solutions using the verifier, and then use those correct OOD trajectories as additional training data. The paper's result that SFT on expert-only data memorizes rather than generalizes (Figure 5) might be partially mitigated if the training data itself includes OOD examples—even if those examples were generated by an RL model rather than a human expert. This is a natural extension of the STaR/ReST^EM paradigm (Zelikman et al., 2022; Singh et al., 2024) but with RL-generated rather than SFT-generated trajectories as the data source. The paper's negative ReST^EM result (Appendix K, Figure 16, where RL optimization of the revision model degraded performance) suggests that naive self-improvement can backfire, but using RL-generated OOD data for SFT—rather than RL for further optimization—may avoid the on-policy data collection issues that caused the ReST^EM failure.