SAIL-RL: Guiding MLLMs in When and How to Think via Dual-Reward RL Tuning

🎯 Pitch

MLLMs often stumble by either overthinking simple tasks or arriving at correct answers through flawed logic—both rewarded under standard outcome-only training. SAIL-RL fixes this with a dual reward that separately scores reasoning quality and adaptively decides when deep thinking is needed, slashing token waste on easy tasks by over 3x while still beating GPT-4o on math and vision benchmarks.

1. Executive Summary

SAIL-RL introduces a reinforcement learning post-training framework that teaches multimodal large language models when and how to think by replacing outcome-only supervision with a dual-reward system — a Thinking Reward that evaluates reasoning quality across logical coherence, factual grounding, and answer consistency, and a Judging Reward that adaptively determines whether deep reasoning or direct answering is appropriate for a given task. Applied to the state-of-the-art SAIL-VL2 at both 4B and 8B scales, SAIL-RL achieves a new open-source state-of-the-art average of 59.3 on multimodal reasoning benchmarks — a +20.0 improvement over the SAIL-VL2-8B baseline — while delivering competitive performance against commercial closed-source models such as GPT-4o, and the Judging Reward reduces token usage on perception-heavy tasks like OCRBench from 4.7× to 1.2× compared to always-thinking baselines without sacrificing accuracy, establishing that adaptive reasoning depth can simultaneously improve both effectiveness and efficiency when the model is explicitly supervised on when reasoning is necessary rather than uniformly applied.

2. Context and Motivation

The Core Problem: Reasoning Quality and Efficiency Are in Tension

The fundamental challenge this paper addresses is that current reinforcement learning methods for multimodal large language models create a false choice between reasoning quality and reasoning efficiency. When MLLMs are trained with outcome-only supervision — where rewards depend solely on whether the final answer is correct — they develop two intertwined pathologies that undermine both their reliability and their practical deployability.

The first pathology is unsound reasoning masquerading as success. Because the reward signal only cares about the final answer, models can arrive at correct answers through logically flawed, factually inaccurate, or internally inconsistent reasoning chains. The paper provides concrete examples of this phenomenon: in Figure 1 (bottom), a model correctly solves a geometry problem but does so through a reasoning path that contains factual errors about the underlying mathematical relationships. This is what the authors term "accidental correctness" or "lucky success" — the model gets the right answer, but for the wrong reasons. Under outcome-only supervision, such instances are reinforced as positive examples during RL training, creating a perverse incentive structure where the model learns that reasoning quality doesn't matter as long as the final output matches the ground truth.

This is not merely a theoretical concern about the purity of reasoning. The paper argues that this pathology has cascading practical consequences: models trained this way exhibit increased hallucination rates because they have not been penalized for fabricating facts or inventing logical connections during their reasoning process, reduced trustworthiness because users cannot rely on the reasoning trace to verify correctness, and diminished robustness because the model has not internalized the structural patterns of valid reasoning but rather surface-level correlations between problem features and answers.

The second pathology is the uniform application of reasoning depth across all tasks. Current MLLM training paradigms — particularly those inspired by the success of models like DeepSeek-R1 and OpenAI's o1 in text reasoning — typically train models to always engage in extended chain-of-thought reasoning before producing any answer. The paper identifies this as a fundamental mismatch with how humans approach problems: we naturally modulate our cognitive effort based on task difficulty, spending seconds on simple perceptual judgments and minutes or hours on complex analytical reasoning. Forcing MLLMs to apply the same deep reasoning process to every query regardless of complexity creates two opposing failure modes.

On simple tasks — the paper's Figure 1 (top) shows a handwritten number recognition problem — the model engages in elaborate, unnecessary reasoning chains that are computationally wasteful and, more insidiously, can introduce errors. The model in the example overanalyzes whether digits might be misread, generates alternative interpretations, and ultimately produces a chain of thought that is longer than the problem warrants. This "overthinking" problem has been observed across the literature (Zhang et al., 2025; Zhou et al., 2025) but has not been addressed through the training objective itself — current approaches either apply reasoning uniformly or rely on post-hoc routing mechanisms that are not optimized during the training process.

On complex tasks, the same rigidity causes underthinking: the model applies a reasoning process that is too shallow for the problem's difficulty, producing incorrect answers not because the model lacks the underlying knowledge or capability, but because it has not been trained to allocate sufficient reasoning depth to harder problems. The paper's framing is that this is fundamentally a resource allocation problem — the model needs to learn a meta-cognitive skill of assessing task difficulty and adjusting its computational budget accordingly.

Why This Problem Matters

The paper's motivation extends beyond academic interest in reasoning quality. The practical stakes are substantial and operate on multiple dimensions:

Reliability and trustworthiness in high-stakes applications. When MLLMs are deployed in domains where reasoning errors have real consequences — medical diagnosis, legal analysis, scientific research, education — the distinction between "gets the right answer" and "gets the right answer for the right reasons" becomes critical. A model that achieves 90% accuracy through a mixture of sound reasoning and lucky guesses is fundamentally less trustworthy than one that achieves the same accuracy with verified logical chains, because there is no way to distinguish which answers are reliable without auditing the reasoning process. The paper argues that outcome-only supervision creates exactly this opacity — users cannot know whether a correct answer was produced through valid inference or through a reasoning path that happened to converge on the right output despite containing errors.

Computational efficiency in production deployments. The always-thinking paradigm imposes a uniform inference cost regardless of task difficulty. For large-scale deployments where the query distribution includes many simple perceptual questions (object recognition, basic OCR, simple counting), the cost of generating unnecessary reasoning tokens dominates the total inference budget. The paper's experiments in Table 5 quantify this: forced thinking on OCRBench requires 4.7× more tokens than direct answering while actually reducing accuracy (88.7 vs. 90.5 for direct answering), meaning the extra computation is not just neutral but actively harmful. At deployment scale — millions of queries per day — these token overheads translate directly to GPU hours, latency, and operational cost. An adaptive model that can bypass reasoning on simple tasks while still engaging deep analysis on complex ones offers a path to maintaining high accuracy across all difficulty levels while dramatically reducing average inference cost.

The hallucination-reasoning connection. The paper identifies a specific mechanism by which uniform reasoning policies increase hallucination. When models are forced to generate reasoning chains for simple perceptual tasks, they often invent spurious details, alternative interpretations, or justifications that are not grounded in the visual evidence. This is not random — it is a direct consequence of the mismatch between the task's information requirements and the model's generative process. Simple recognition tasks have low entropy in their correct outputs (there is essentially one right answer), but the reasoning generation process has high entropy (there are many possible chains of pseudo-analysis the model could produce). This entropy creates opportunities for the model to generate content that isn't supported by the input, which in turn can contaminate the final answer. The paper argues that by teaching the model when reasoning is unnecessary, the Judging Reward closes this hallucination vector at its source.

Bridging the gap between open-source and closed-source models. The paper's benchmarks (Tables 1 and 2) show that at the time of writing, open-source MLLMs at the 7-8B scale lag significantly behind closed-source systems like GPT-4o and Gemini-2.0-Pro on reasoning benchmarks. The SAIL-VL2-8B baseline achieves 39.3 average on reasoning benchmarks versus 56.6 for Gemini-2.0-Pro — a gap of over 17 points. The paper positions SAIL-RL as demonstrating that part of this gap is not due to fundamental model capacity limitations but rather to suboptimal post-training strategies. By improving how and when the model reasons rather than just whether it produces correct answers, SAIL-RL achieves 59.3 — surpassing both closed-source models. This has significant implications for the democratization of advanced reasoning capabilities: if a principled post-training framework can extract dramatically better reasoning from existing model architectures, the barrier to competitive reasoning performance is more about training methodology than about training ever-larger models.

Where Existing Approaches Fall Short

The paper situates its contributions against three broad classes of prior work, identifying specific limitations in each:

Outcome-only RL for MLLMs. The dominant paradigm for applying reinforcement learning to multimodal reasoning — exemplified by recent work from major labs (Team et al., 2025a; Team et al., 2025b; Deng et al., 2025; Chen et al., 2025a) — follows a straightforward recipe: define a reward based on whether the final answer matches a ground truth (or passes a verifier), and use this reward to train the model to produce correct answers via RL algorithms like PPO or GRPO. The problem with this approach, as the paper systematically argues, is not that it fails to improve answer accuracy — it demonstrably does — but that it does so by optimizing a proxy objective that is misaligned with the true goal of building reliable reasoners. The reward signal treats all correct answers identically, regardless of the reasoning quality that produced them.

This creates what the paper identifies as a form of reward hacking specific to reasoning tasks: models learn to exploit statistical regularities in the problem-answer mapping without necessarily learning valid reasoning procedures. The "lucky success" phenomenon in Figure 1 (bottom) is a direct consequence — the model finds paths through the reasoning space that lead to correct answers but are not constrained by logical validity, and since the reward function cannot distinguish valid from invalid reasoning, these paths are reinforced alongside genuinely sound reasoning.

The paper also notes a more subtle failure mode: during RL training, the model's reasoning quality can actually degrade even as answer accuracy improves. The training dynamics analysis in Figure 4 shows this directly — the answer-only baseline's "consistency score" (measuring whether the reasoning chain actually supports the final answer) drops over the course of training, even as its accuracy on the final answer rises. This is the "reasoning collapse" phenomenon the authors describe: the model learns that producing the right answer is rewarded regardless of whether the reasoning supports it, so it progressively decouples the thinking process from the final output. The thinking reward is designed specifically to prevent this decoupling by imposing a logical consistency constraint that makes answer-correctness rewards contingent on reasoning quality.

Process reward models for reasoning verification. A second line of work — including VisualPRM (Wang et al., 2025b) and URSA (Luo et al., 2025) — attempts to address the reasoning quality problem by training separate process reward models that evaluate individual reasoning steps. These PRMs can then be used either to filter training data, to provide step-level rewards during RL, or to guide search at inference time. The paper acknowledges that this approach can improve reasoning quality but identifies two significant limitations.

First, training effective multimodal PRMs is itself a difficult and data-intensive problem. PRMs require step-level supervision (correctness labels for each intermediate reasoning step), which is expensive to obtain at scale and introduces its own reliability issues — the PRM can be wrong, and when it is, it propagates errors into the model's training. This creates a circular dependency: you need good reasoning to train a good PRM, but you need a good PRM to train good reasoning.

Second, and more fundamentally, process reward models address how to think but not when to think. A PRM can verify that each step of a reasoning chain is correct, but it does not answer the prior question of whether a reasoning chain should be generated at all for a given task. The paper argues that efficiency and effectiveness must be addressed simultaneously, not sequentially — a model that produces perfectly verified reasoning chains for simple perceptual tasks is still inefficient, and a model that produces verified but shallow reasoning on complex tasks is still inaccurate. The dual-reward design is a direct response to this limitation, treating when and how as interdependent dimensions that should be optimized jointly.

Routing and mode-switching approaches. Recent work has begun to explore dynamic reasoning strategies through architectural routing mechanisms. Zhang et al. (2025) propose OThink-R1, which learns to switch between fast (System 1) and slow (System 2) thinking modes. Zhou et al. (2025) apply similar ideas in the context of vision-language-action models for autonomous driving. These approaches typically train a separate router module or use threshold-based heuristics to decide when to engage deep reasoning.

The paper identifies two shortcomings in these routing approaches. The first is that routing decisions are typically optimized independently of reasoning quality — the router learns to minimize a cost function (e.g., expected token count) subject to an accuracy constraint, but this optimization is decoupled from the process that generates the actual reasoning content. The router can make correct mode-selection decisions while the reasoning process within the selected mode remains flawed, or vice versa. No joint optimization exists.

The second shortcoming is more subtle: routing mechanisms trained via supervised learning on human-annotated difficulty labels are fundamentally limited by the quality and coverage of those labels. Human judgments about whether a task "requires reasoning" are noisy, context-dependent, and often inconsistent across annotators. The paper's Judging Reward addresses this by making the when-to-think decision part of the RL optimization process itself — the model learns from reward signals what constitutes a "complex" versus "simple" task, and this learning co-evolves with the model's actual reasoning capabilities. As the model gets better at reasoning, its definition of what requires reasoning can adapt — a problem that was complex for the base model might become simple after training, and the model can learn to shift its judgment accordingly.

The gap SAIL-RL fills. The paper explicitly positions itself as addressing what it sees as a fragmentation in prior work: approaches that improve reasoning quality (thinking rewards, PRMs) don't address efficiency, and approaches that improve efficiency (routing, mode switching) don't address reasoning quality in a jointly optimized way. The dual-reward mechanism — particularly through the cascading product formulation in Equation 1 — is designed to create a single optimization objective where the quality of reasoning, the correctness of the mode decision, and the accuracy of the final answer are all interdependent. A model cannot receive high reward by being efficient but wrong, nor by being correct but inefficient, nor by making the right mode decision but producing poor reasoning within that mode. The cascading product functions as a logical AND gate across all three dimensions, which the paper argues is the key innovation that prior works miss by treating these as separable optimization problems.

How This Paper Positions Itself

The paper frames SAIL-RL not as an incremental improvement over existing RL methods but as a paradigm shift in what the reward function supervises. The core conceptual move is expanding the supervision target from a single scalar (answer correctness) to a structured evaluation of the entire reasoning episode — the decision about whether to reason, the quality of the reasoning if engaged, and the fidelity between reasoning and answer.

This reframing has several implications for how the paper positions its contributions. First, it argues that SAIL-RL is architecture-agnostic — the dual-reward mechanism can be applied to any MLLM that can be trained with RL, and the paper demonstrates this through experiments on both SAIL-VL2 (at 2B, 4B, and 8B scales) and Qwen2.5-VL (at 3B and 7B scales), showing consistent improvements across model families (Table 10). This positions SAIL-RL as a general training methodology rather than a model-specific technique.

Second, the paper emphasizes that the dual-reward system is self-contained within the RL training loop — unlike approaches that require separate verifier models (PRMs) or external difficulty classifiers, SAIL-RL uses a strong pre-trained LLM (Gemini-2.5-Pro in the main experiments, but the paper shows robustness to reward model choice in Table 9) as a reward judge, and the thinking/judging rewards are computed as part of the rollout evaluation. This makes the approach more practical to implement since it doesn't require training auxiliary models or collecting additional labeled data beyond the standard RL dataset.

Third, the paper positions the cascading product reward formulation (Equation 1) as a principle-driven design choice rather than an empirical hack. The multiplicative structure is motivated by the logical requirement that all components must succeed jointly — it is not enough to be partially good at reasoning, partially correct in mode selection, and partially accurate in answers. The authors explicitly contrast this against additive reward combinations (Table 6) and show that additive rewards allow reward hacking by letting the model compensate for failures in one dimension with successes in another. The cascading product eliminates this compensation pathway, which the paper frames as essential for stable optimization when rewards have multiple interacting components.

Finally, the paper situates its work within the broader trajectory of the field from "alignment with human preferences" to "thinking before speaking." The introduction characterizes this as a pivotal paradigm shift in RL for language models, citing DeepSeek-R1 and similar works as exemplars of the "thinking before speaking" approach. SAIL-RL's contribution to this trajectory is adding adaptive metacognition — not just thinking before speaking, but deciding whether to think before speaking, and ensuring that when thinking is engaged, it is genuinely sound. This positions the work as advancing the System 1 / System 2 cognitive paradigm that the paper invokes in its related work discussion (Section 2), where the goal is not to replace System 1 (fast, intuitive) with System 2 (slow, deliberative) but to build models that can fluidly and appropriately employ both modes depending on task demands, with the mode selection itself being a learned capability rather than a hard-coded heuristic.

3. Technical Approach

3.1 Reader Orientation

SAIL-RL is a reinforcement learning post-training framework that teaches multimodal large language models to produce high-quality reasoning only when tasks actually require it, rather than either blindly reasoning through everything or blindly optimizing for correct final answers without regard for how they were obtained. The system solves the dual problem of unsound reasoning (getting right answers through flawed logic) and inefficient reasoning (wasting computation on simple tasks or under-thinking hard ones) by introducing two specialized reward signals—one that grades the quality of reasoning and one that grades the necessity of reasoning—combined through a multiplicative reward structure that forces the model to succeed at both simultaneously or receive no credit at all.

3.2 Big-Picture Architecture (Diagram in Words)

The SAIL-RL framework has six major components operating in a two-stage training pipeline:

Base MLLM (e.g., SAIL-VL2-8B): The pretrained multimodal model with vision encoder and language model backbone. This is the starting point that will be taught when and how to reason.
LongCoT SFT Dataset (400K samples): A curated dataset of multimodal problems structured in the judge-think-answer format. Every sample contains a <judge> tag indicating whether reasoning is needed, a \think section containing the reasoning trace (empty for simple tasks), and a final answer in \boxed{} tags. This dataset teaches the model the basic format and meta-cognitive structure.
RL Training Dataset (70K samples): A mixed dataset of 50K challenging STEM problems and 20K general QA samples. STEM problems undergo difficulty filtering to retain only problems within an optimal difficulty range (removing trivial and impossible ones). This is the dataset over which RL optimization occurs.
Dual-Reward System: The core innovation. For each model response generated during RL rollouts, a reward judge (Gemini-2.5-Pro by default) evaluates four dimensions:
- Format Reward: Whether the output is structurally parseable.
- Answer Reward: Whether the final answer matches ground truth.
- Thinking Reward: A composite score evaluating logical coherence, factual grounding, and answer consistency of the reasoning trace.
- Judging Reward: Whether the model correctly decided if reasoning was needed.
Cascading Reward Combiner: A multiplicative formula R_total = α · (R_judge · R_think · R_answer) + (1-α) · R_format that multiplies the three core rewards together, functioning as a logical AND gate. If any component fails (e.g., correct answer but wrong mode decision), the product is zero, preventing reward hacking.
DAPO RL Optimizer: The reinforcement learning algorithm (a variant of PPO with dynamic clipping and no KL penalty) that updates the model parameters using the cascading reward signal, exploring the space of reasoning behaviors through multiple rollouts per prompt.

Information flow: A multimodal input (image + question) enters the model → the model generates a response in judge-think-answer format → the reward judge evaluates the response across all four dimensions → the cascading combiner produces a single scalar reward → the DAPO algorithm computes a policy gradient and updates model parameters → the process repeats for three epochs over the 70K RL dataset.

3.3 Roadmap for the Deep Dive

First, the LongCoT SFT stage (Section 3.4.1): how the training data is constructed to teach the model the judge-think-answer format and establish the foundation for meta-cognitive reasoning, since this structured format is what the RL stage will optimize within.
Second, the Thinking Reward (Section 3.4.2): the three sub-rewards (logical coherence, factual grounding, answer consistency) and how they are evaluated by the LLM judge, since this is the component that addresses the "how to think" problem and is the most technically novel evaluation mechanism.
Third, the Judging Reward (Section 3.4.3): how the model learns when reasoning is necessary through binary alignment with ground-truth complexity labels, since this is the component that enables adaptive efficiency.
Fourth, the Cascading Reward System (Section 3.4.4): the multiplicative formulation and why it functions as a logical AND gate, since this is the mechanism that forces joint optimization of all dimensions and prevents the reward hacking possible under additive combinations.
Fifth, the RL training stage (Section 3.4.5): the dataset preparation, the DAPO algorithm configuration, and the specific hyperparameter choices, since these operational details determine whether the dual-reward signals can be effectively optimized.
Sixth, the design rationale and contrasts with alternatives (Section 3.4.6): why discrete rather than continuous rewards, why equal weighting of thinking sub-components, and why the cascading product over additive combination, since these design choices are empirically validated in the ablation studies and understanding them reveals the principles behind the framework.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a training methodology paper whose core idea is that MLLM reasoning can be simultaneously made more reliable (sound reasoning leading to correct answers) and more efficient (adaptive reasoning depth based on task difficulty) by expanding the RL reward function from a single scalar (answer correctness) to a structured evaluation of the entire reasoning episode—including whether reasoning was appropriate, whether it was logically sound, factually grounded, and consistent with the final answer—all combined through a multiplicative reward that enforces joint success across all dimensions.

3.4.1 LongCoT SFT: Teaching the Judge-Think-Answer Format

The first stage of SAIL-RL is supervised fine-tuning on a specially constructed dataset that teaches the model the structural format required for the subsequent RL stage. This is not merely a standard instruction-tuning step; it establishes the meta-cognitive template that the RL stage will optimize within.

Dataset construction. The LongCoT SFT dataset consists of 400,000 high-quality multimodal samples, each structured as a sequence of three labeled sections: <judge> (a binary decision about whether reasoning is required), \think (the reasoning trace itself, empty for simple tasks), and <answer> (the final answer in \boxed{} tags). The dataset is built through a multi-stage curation pipeline:

Data collection. The authors collect samples from diverse sources spanning two categories: complex logical problems from datasets like VisualWebInstruct (Jia et al., 2025) and MathV360K (Shi et al., 2024), and simple perception-based questions from datasets like LLaVA-CoT (Xu et al., 2024). This diversity is intentional—the model must learn to distinguish between tasks that genuinely require analytical reasoning and those that can be answered through direct perceptual recognition.

Data cleaning. A cleaning pass removes extraneous content that could confuse the format learning: system prompts are stripped, conflicting hints are removed, and the entire dataset is deduplicated based on unique image-question pairs to prevent the model from memorizing specific instances rather than learning the general judge-think-answer pattern.

Conditional annotation. This is the critical step that encodes the when-to-think signal into the training data. Each sample is annotated based on its complexity:

For complex problems requiring reasoning, a strong teacher model (the paper doesn't specify which, but the context suggests a capable LLM like Gemini) uses guided prompting to generate a detailed chain-of-thought for the \think section, and the <judge> tag is set to indicate that reasoning is necessary. The thinking trace is a full step-by-step derivation leading to the answer.
For simple perceptual tasks, the <judge> tag is set to indicate the question can be answered directly, and the \think section is intentionally populated with an empty string (\n\n). The model sees examples where <judge>This question does not require thinking...</judge> is immediately followed by an empty thinking block and then the direct answer.
The final answer for all samples is standardized within \boxed{} tags, ensuring the model learns a consistent answer extraction format.

Quality filtering. The annotated dataset undergoes rigorous filtering: a redundancy filter penalizes trivial reasoning by measuring token overlap between the thought process and the final answer (if the "reasoning" is just a restatement of the answer, it is removed), and a length-balancing step ensures varied representation of reasoning chain lengths so the model doesn't learn to always produce reasoning of a fixed length.

Training objective. The model is fine-tuned using standard next-token prediction loss over the full sequence. The objective function is:

$\mathcal{L}_{\text{LongCoT-SFT}} = -\frac{1}{|\mathcal{D}_{\text{CoT}}|} \sum_{(I, J, T, A) \in \mathcal{D}_{\text{CoT}}} \log P_\theta(J \circ T \circ A \mid I)$

where $I$ is the multimodal input (image + question text), $J$ is the judgment text (the content of the <judge> tags), $T$ is the reasoning process (the content of the \think section), $A$ is the final answer (the content of the <answer> tags with \boxed{} formatting), and $\circ$ denotes string concatenation of these three sections in order.

What it computes: the standard autoregressive language modeling loss summed over all tokens in the judge-think-answer sequence. For each position in the concatenated output $J \circ T \circ A$ , the model predicts the next token given all previous tokens (including the image and question). The loss is the negative log-likelihood of the ground-truth tokens, averaged over the entire 400K-sample dataset.

Why this form: the standard next-token prediction objective is the maximum-likelihood estimator for autoregressive sequence generation, which is the appropriate loss for teaching the model to produce outputs in a specific format. Training on the full sequence $J \circ T \circ A$ rather than just $A$ ensures the model learns the complete judge-think-answer structure as a single coherent generation behavior—it must learn to first produce the judgment, then condition the thinking on that judgment, then condition the answer on the thinking. If training were done on answers alone, the model would have no signal about when to produce judgments or reasoning traces. The concatenation $\circ$ means the model sees examples where $J$ is immediately followed by $T$ is immediately followed by $A$ , learning the sequential dependency: judgment → thinking → answer.

Training configuration. The SFT stage uses full-parameter fine-tuning (all model weights are updated) for one epoch over the 400K samples. Specific hyperparameters: maximum sequence length of 20,000 tokens (accommodating long reasoning chains), global batch size of 1024, learning rate of $1 \times 10^{-6}$ , AdamW optimizer with cosine learning rate schedule, executed on 64 NVIDIA A100 GPUs.

Why full-parameter SFT rather than LoRA or partial fine-tuning: the stage needs to fundamentally reshape the model's output format from whatever structure it learned during pretraining to the judge-think-answer paradigm. This is a substantial distribution shift in output format, and parameter-efficient methods may not have sufficient capacity to restructure the generation behavior. The one-epoch duration suggests the authors found that a single pass through the curated data is sufficient to establish the format without overfitting, though they don't explicitly discuss this choice.

What this stage produces: a model that has learned to (1) generate a judgment about whether reasoning is needed, (2) condition the subsequent thinking trace on that judgment, and (3) produce a final answer that follows from the thinking. However, the quality of the judgments and reasoning at this stage is purely a reflection of the teacher model's outputs—there has been no optimization for correctness, logical soundness, or efficiency. This is where the RL stage takes over.

3.4.2 The Thinking Reward: Evaluating How to Think

The Thinking Reward is the component that moves SAIL-RL beyond outcome-only supervision by directly evaluating the quality of the reasoning process itself. Rather than rewarding the model solely for producing correct final answers, the Thinking Reward assesses whether the reasoning trace is logically sound, factually grounded, and consistent with the answer it produces. This is implemented as an LLM-based evaluation where a strong judge model (Gemini-2.5-Pro in the main experiments) analyzes the model's output and produces structured assessments.

The three sub-rewards. The Thinking Reward decomposes reasoning quality into three independent dimensions, each evaluated by the judge model using a specific prompt (detailed in Appendix C of the paper). The prompts are extensive—each is roughly a page of instructions specifying the judge's role, the evaluation criteria, edge cases to consider, and the exact JSON output format required.

Sub-reward 1: Logical Coherence Reward ( $d_1$ ). This evaluates whether the model "thinks clearly"—whether its reasoning process is internally valid from initial modeling to final conclusion. The judge model performs two sequential checks:

Structural Soundness: Does the model correctly map the image and question into a valid logical or mathematical model? This means verifying that the chosen formulas, logical frameworks, or problem representations correctly capture what the problem is asking. For example, in a geometry problem, the judge checks whether the model correctly identified the relevant geometric relationships (e.g., recognizing that two angles are supplementary) and set up the appropriate equation. Failure here—using the wrong formula or misidentifying the problem structure—yields $d_1 = 0$ regardless of subsequent execution.
Deductive Soundness: Given the model established in the structural check, is the step-by-step execution free of calculation errors, contradictions, or invalid inferences? This check verifies that the algebraic manipulation is correct, that inferences follow from their premises, and that no step contradicts an earlier step. For example, if the model sets up $2x = 10$ and concludes $x = 4$ , the deductive check catches the arithmetic error.

The logical coherence score is binary: $d_1 = 1$ only if both structural and deductive checks pass; $d_1 = 0$ if either fails. This is a strict AND—a perfect calculation based on a wrong model is still unsound reasoning.

Sub-reward 2: Factual Grounding Reward ( $d_2$ ). This evaluates whether the model "thinks truthfully"—whether its reasoning is grounded in verifiable evidence rather than hallucinated claims. The judge performs hierarchical fact-checking across three sources, applied in priority order:

Visual Grounding: Claims about objects, attributes, relationships, or text in the image are cross-referenced against the actual image content. If the model states "there are three cats" but the image shows two, this check fails.
Textual Grounding: Claims are checked against the input question for consistency with stated constraints or provided data. If the question states a specific numerical value and the model's reasoning uses a different value, this check fails.
World Knowledge: For claims not verifiable from image or text, the judge checks against established facts (historical, scientific, geographical). Critically, the prompt includes an exception: if the question or image deliberately presents a hypothetical or counterfactual scenario (e.g., "Imagine the sky is green"), the provided context overrides world knowledge—the model should follow the context, not "correct" it to match reality.

Any contradiction at any of these stages yields $d_2 = 0$ ; only if all claims are supported by at least one verification source does $d_2 = 1$ . This hierarchical structure means the judge prioritizes direct evidence (image, then text) over general knowledge, and the context-override exception prevents the judge from penalizing the model for correctly following the premises of a hypothetical problem.

Sub-reward 3: Answer Consistency Reward ( $d_3$ ). This evaluates whether the model "thinks faithfully"—whether the final answer is truly derived from the preceding reasoning rather than being asserted independently. The judge verifies that the reasoning trace fully justifies the answer without gaps, disconnects, reliance on unstated information, or unsupported logical leaps.

A critical design choice specified in the prompt: the judge is explicitly instructed to not evaluate whether the thinking is factually correct. The consistency check is purely about logical flow: if the thinking says $1 + 1 = 3$ and the answer says $3$ , this is consistent (even though factually wrong). If the thinking calculates $x = 5$ and $y = 10$ , so $x + y = 15$ , but the answer says $20$ , this is inconsistent. This separation of concerns—factual correctness is handled by $d_2$ , logical flow by $d_3$ —prevents the judge from conflating "the reasoning is wrong" with "the reasoning doesn't support the answer."

The score is binary: $d_3 = 1$ if the answer is a direct summary or derivation of the final reasoning step without introducing new entities, contradictions, or logical gaps; $d_3 = 0$ otherwise.

Aggregation into the Thinking Reward. The three sub-rewards are combined through simple averaging:

$R_{\text{think}} = \frac{1}{3} \sum_{i=1}^{3} d_i$

where $d_1$ is the logical coherence score, $d_2$ is the factual grounding score, and $d_3$ is the answer consistency score, each binary $\{0, 1\}$ .

What it computes: the fraction of the three reasoning quality dimensions that the model satisfied. Since each $d_i$ is binary, $R_{\text{think}}$ takes one of four values: $0$ (no dimension satisfied), $1/3$ (one dimension), $2/3$ (two dimensions), or $1$ (all three). This is a coarse but interpretable signal—the model knows exactly which fraction of the quality checks it passed.

Why this form—equal weighting: the paper explicitly ablated this choice (Table 8), testing biased weighting schemes that emphasized individual dimensions (e.g., $w_L = 1/2, w_H = 1/4, w_C = 1/4$ to bias toward logic). These biased schemes produced a "see-saw effect"—gains in one domain at the cost of losses in others, with maximum variance across benchmarks of 0.7-0.9 points. The equal-weight scheme provides the most robust performance across diverse benchmarks because the three dimensions are genuinely complementary: a logically sound but factually hallucinated reasoning chain is as useless as a factually grounded but logically inconsistent one. The simple average treats them as equally necessary conditions, which matches the intuition that reasoning quality requires all three.

Why binary rather than continuous sub-rewards: the paper argues (Section 4.5) that binary $\{0, 1\}$ signals provide sharper guidance for RL optimization than continuous $[0, 1]$ scores. The reasoning is twofold: (1) LLM judges have difficulty producing consistent fine-grained scores—distinguishing between 0.7 and 0.8 on reasoning quality introduces subjective noise—while binary decisions ("Is this step valid? Yes/No") are more reproducible; (2) RL training is sensitive to reward variance, and noisy continuous scores increase the variance of advantage estimates, destabilizing the optimization. Binary rewards provide a high-discrimination signal that cleanly separates good from bad behaviors, making the policy gradient more reliable.

3.4.3 The Judging Reward: Learning When to Think

The Judging Reward teaches the model to make appropriate decisions about whether to engage in deep reasoning for a given task. Rather than applying reasoning uniformly to all inputs, the model learns to assess task complexity and switch between a full reasoning mode and a direct-answer mode.

Mechanism. The model is required to output a thinking decision before generating its response, in the <judge> tag format learned during SFT. This decision is evaluated against ground-truth complexity labels that classify each problem as either requiring reasoning or being answerable through direct perception. The reward is binary:

$R_{\text{judge}} = d_{\text{judge}}$

where $d_{\text{judge}} = 1$ if the model's judgment aligns with the ground truth (choosing "thinking mode" for complex tasks or "no-thinking mode" for simple tasks), and $d_{\text{judge}} = 0$ otherwise.

What it computes: a simple binary indicator of whether the model made the correct mode-selection decision. There is no partial credit—a model that correctly identifies a complex task but unnecessarily triggers thinking for a simple task receives $d_{\text{judge}} = 0$ on that simple task.

Why this form—binary reward with ground-truth labels: the paper uses pre-assigned complexity labels for each problem in the RL training dataset. The STEM problems (50K) are classified as requiring reasoning, while the general QA problems (20K from LLaVA-OneVision) are classified based on their content—perceptual tasks are labeled as not requiring reasoning, analytical tasks as requiring it. The binary reward is appropriate because the decision space is discrete and the ground truth is categorical: there are exactly two valid choices (think or don't think), and any deviation is incorrect. A continuous reward (e.g., rewarding the model proportionally to how "close" its judgment reasoning is to a correct justification) would introduce noise from the judge's interpretation of justification quality.

How ground-truth labels are generated. The paper doesn't fully detail the annotation process for the RL dataset's complexity labels, but the SFT data pipeline (Appendix A.1) describes the approach: for complex problems requiring reasoning, the <judge> tag is set to indicate thinking is necessary; for simple perceptual tasks, it is set to indicate direct answering. The RL dataset inherits this labeling scheme. Additionally, the RL data curation involves difficulty-based filtering—problems are evaluated using the SFT model's pass@4 score (the fraction of 4 attempts that produce a correct answer), and problems that are trivially easy (pass@4=1, meaning the model always gets them right) or impossibly hard (pass@4=0, meaning the model never gets them right) are removed. This filtering ensures the RL dataset contains problems in an "optimal difficulty range" where the model has some chance of success but isn't already perfect, which is critical for RL to have a meaningful training signal.

What behavior this reward shapes. The Judging Reward penalizes two symmetric failure modes:

Under-thinking: the model chooses direct-answer mode for a complex problem, producing a shallow or incorrect answer because it skipped necessary reasoning. The $d_{\text{judge}} = 0$ penalty means the model cannot get full reward even if it accidentally produces the correct answer through a lucky guess without reasoning.
Over-thinking: the model chooses reasoning mode for a simple perceptual task, generating unnecessary (and potentially noise-introducing) reasoning chains. The $d_{\text{judge}} = 0$ penalty means the model is disincentivized from wastefully expending computation on tasks that don't benefit from it.

The paper's experiments (Figure 5, Table 5) demonstrate that this reward successfully shapes adaptive behavior: after training, the model triggers reasoning on 99-100% of reasoning benchmark problems but only 7.5% of OCRBench problems, with a smooth gradient of trigger rates across benchmarks of varying complexity.

3.4.4 The Cascading Reward System: Enforcing Joint Success

The individual rewards—Thinking, Judging, Answer, and Format—must be combined into a single scalar that the RL algorithm can optimize. The paper's core design choice is a multiplicative cascading structure rather than a simple weighted sum.

The cascading product formulation:

$R_{\text{total}} = \alpha \cdot (R_{\text{judge}} \cdot R_{\text{think}} \cdot R_{\text{answer}}) + (1 - \alpha) \cdot R_{\text{format}}$

where $R_{\text{judge}} \in \{0, 1\}$ is the judging reward (was the mode decision correct?), $R_{\text{think}} \in \{0, 1/3, 2/3, 1\}$ is the thinking reward (fraction of reasoning quality dimensions satisfied), $R_{\text{answer}} \in \{0, 1\}$ is the answer reward (is the final answer correct?), $R_{\text{format}} \in \{0, 1\}$ is the format reward (is the output structurally parseable?), and $\alpha = 0.9$ is a balancing coefficient.

What it computes: a weighted combination where 90% of the reward comes from the cascade $R_{\text{judge}} \cdot R_{\text{think}} \cdot R_{\text{answer}}$ and 10% comes from the format regularizer $R_{\text{format}}$ . The cascade term is a three-way product: if any of the three components is zero, the entire cascade term is zero. This means the model receives high reward only when it simultaneously (1) makes the correct mode decision, (2) produces high-quality reasoning if reasoning was chosen (or doesn't need to if it wasn't), and (3) produces the correct final answer. The format term provides a small incentive for structural compliance regardless of content quality.

Why this form—the logical AND gate property. The multiplicative structure enforces a strict conditional dependency that additive combinations cannot provide. Consider an additive reward $R_{\text{additive}} = \frac{1}{3}(R_{\text{judge}} + R_{\text{think}} + R_{\text{answer}})$ . Under this formulation, a model could:

Skip reasoning on a complex problem ( $R_{\text{judge}} = 0$ ), produce no thinking trace ( $R_{\text{think}} = 0$ by default if reasoning is expected), but somehow guess the correct answer ( $R_{\text{answer}} = 1$ ), receiving $R_{\text{additive}} = 1/3$ —a non-trivial reward for lucky guessing.
Or correctly decide to think ( $R_{\text{judge}} = 1$ ), produce perfect reasoning ( $R_{\text{think}} = 1$ ), but make a calculation error in the final answer ( $R_{\text{answer}} = 0$ ), receiving $R_{\text{additive}} = 2/3$ —a substantial reward for a wrong answer.

Both scenarios represent reward hacking: the model is incentivized to optimize the components independently rather than jointly. The multiplicative cascade $R_{\text{judge}} \cdot R_{\text{think}} \cdot R_{\text{answer}}$ eliminates these failure modes:

Lucky guessing ( $R_{\text{judge}} = 0$ ) yields $0 \cdot R_{\text{think}} \cdot R_{\text{answer}} = 0$ regardless of answer correctness.
Great reasoning with wrong answer ( $R_{\text{answer}} = 0$ ) yields $R_{\text{judge}} \cdot R_{\text{think}} \cdot 0 = 0$ regardless of reasoning quality.
Correct answer through flawed reasoning ( $R_{\text{think}} < 1$ ) yields $R_{\text{judge}} \cdot (\text{partial}) \cdot R_{\text{answer}} < R_{\text{judge}} \cdot 1 \cdot R_{\text{answer}}$ , providing a partial but diminished reward that still incentivizes improving reasoning quality.

This is what the paper calls a "zero-tolerance penalty" for error propagation—any failure in the chain nullifies the entire reward, forcing the model to learn coherent end-to-end reasoning behaviors rather than compensating for weaknesses in one area with strengths in another.

Why $\alpha = 0.9$ rather than 1.0. The 10% allocation to $R_{\text{format}}$ serves as a regularizer that ensures the model maintains proper output structure even when the content rewards might be zero. Without this term, a model that consistently fails at reasoning (getting zero cascade reward) might also drift away from the judge-think-answer format, making it harder to recover. The format reward provides a weak stabilizing signal for structural compliance. The choice of 0.9 (rather than, say, 0.95 or 0.8) is not ablated in the paper, but it represents a strong emphasis on content quality while still providing a non-zero gradient for format learning.

Why the product includes $R_{\text{think}}$ rather than just $R_{\text{judge}} \cdot R_{\text{answer}}$ . The thinking reward's inclusion is what distinguishes SAIL-RL from a pure efficiency-focused approach. A reward of $R_{\text{judge}} \cdot R_{\text{answer}}$ would incentivize correct mode decisions and correct answers but would not care about reasoning quality—the model could still engage in lucky guessing or produce unsound but accidentally correct reasoning. Including $R_{\text{think}}$ in the product means that even when the model correctly decides to reason and gets the right answer, it receives less-than-maximum reward if its reasoning was flawed (since $R_{\text{think}} < 1$ reduces the product). This creates a continuous incentive to improve reasoning quality even after answer accuracy is high, explaining why the thinking reward training curves in Figure 4 show sustained improvement in consistency and logic scores throughout training rather than plateauing when accuracy saturates.

3.4.5 RL Training Stage: Optimizing with DAPO and the Dual-Reward System

The second stage of SAIL-RL applies reinforcement learning to optimize the SFT model's reasoning behaviors using the dual-reward system. This is where the model learns to actually improve the quality and adaptivity of its reasoning, moving beyond simply mimicking the teacher's format.

RL dataset preparation. The RL stage uses a curated dataset of 70,000 samples, carefully constructed to provide a training environment where the dual-reward signals can meaningfully shape behavior. The dataset composition reflects the paper's goal of simultaneously improving reasoning quality and efficiency:

50,000 STEM-focused problems sourced from diverse public benchmarks spanning mathematics (MM-Math, MathVerse, WeMath), puzzles (PuzzleVQA), science (ScienceQA), OCR, and counting tasks (CLEVR). These problems are intended to exercise the model's reasoning capabilities—they require multi-step logical deduction, mathematical calculation, or analytical inference.
20,000 general QA samples from LLaVA-OneVision (Li et al., 2024). These provide a mix of perceptual tasks (recognition, description) and analytical tasks, ensuring the model encounters both simple and complex queries during RL training. The diversity is crucial for the Judging Reward to learn meaningful complexity boundaries—if all training data were complex reasoning problems, the model would never learn when reasoning is unnecessary.

Difficulty-based filtering. The STEM problems undergo a two-stage filtering pipeline designed to optimize for RL training stability:

Format conversion: Multiple-choice questions are reformatted into open-ended, free-response formats. This prevents reward hacking through answer format exploitation—if the model learns to game multiple-choice answer patterns rather than actually solving problems, the reward signal becomes decoupled from reasoning quality. Free-response formats force the model to generate the answer content rather than select from options.
Difficulty curriculum filtering: Problems are evaluated using the SFT model's pass@4 score—the model generates 4 complete solution attempts for each problem, and pass@4 is the fraction of those 4 attempts that produce the correct answer. Problems with pass@4 = 1 (the model always gets them right—too easy) and pass@4 = 0 (the model never gets them right—too hard) are removed. This retains only problems in an "optimal difficulty range" where the model has some non-zero but imperfect success rate.

The rationale for difficulty filtering: RL training is most effective when the model has a meaningful probability of both success and failure on each problem. If a problem is trivially easy (pass@4 = 1), the model already knows how to solve it, and RL provides no learning signal—the reward is always 1 regardless of reasoning quality improvements. If a problem is impossibly hard (pass@4 = 0), the model never receives positive reward, providing no gradient for improvement—the policy cannot distinguish between "slightly wrong" and "completely wrong" reasoning when both yield zero reward. The optimal difficulty range ensures that improvements in reasoning quality and mode decisions translate into higher reward probabilities, creating a meaningful optimization landscape.

RL algorithm: DAPO (with modifications). The paper uses the DAPO algorithm (Yu et al., 2025), which is a variant of Proximal Policy Optimization (PPO) adapted for language model training. The key hyperparameters and modifications are:

Algorithm: DAPO with policy gradient optimization. DAPO is a PPO-style algorithm that clips policy updates to prevent destructive large parameter changes while still allowing sufficient exploration.
No KL divergence penalty: The standard KL divergence term that penalizes the policy for deviating from the reference (SFT) model is removed. This is a significant departure from conventional RLHF practice. The rationale (implied but not fully explicit in the paper) is that the SFT model's distribution may be suboptimal—it was trained to mimic a teacher, not to optimize the dual-reward objective—and penalizing deviation from this suboptimal distribution would slow down or prevent the model from learning better reasoning behaviors. Removing the KL penalty encourages exploration, allowing the model to discover reasoning strategies that the SFT model wouldn't generate but that achieve higher dual-reward scores.
Dynamic clipping: The clipping value $\varepsilon$ (which bounds how much the policy ratio can change in a single update) is dynamically adjusted within the range $[0.20, 0.28]$ . This is higher than the standard PPO clipping of 0.2, providing more room for policy updates. The dynamic adjustment likely responds to training statistics (e.g., reducing clipping when the policy is changing rapidly to prevent instability, increasing it when progress is slow), though the paper doesn't detail the specific adjustment schedule.
Policy learning rate: $1 \times 10^{-6}$ with a cosine schedule. This is the same learning rate as the SFT stage, suggesting the authors found it appropriate for both supervised and RL optimization.
Global PPO batch size: 256. This is the number of rollouts collected before each policy update, balancing update frequency (more updates per epoch with smaller batches) against gradient estimate quality (lower variance with larger batches).
Rollouts per sample: 5. For each prompt in the RL dataset, the model generates 5 complete responses. These 5 rollouts provide the sample-based estimate of the advantage function—by comparing the reward of each rollout to the average reward across rollouts for the same prompt, the algorithm can estimate which responses are better than the model's current average, providing a baseline-normalized training signal.
Training duration: 3 epochs over the 70K RL dataset. With 5 rollouts per sample, this means the model sees each problem in the dataset 3 times, generating 5 responses each time, for a total of $70,000 \times 3 \times 5 = 1,050,000$ reward-evaluated rollouts.
Maximum sequence length: 20,000 tokens total, split as 16,000 for the input (image + question + generation context) and 4,000 for the generated output (judge + think + answer). This constraint ensures the model's reasoning traces don't exceed manageable length while still providing substantial room for detailed step-by-step analysis.
Hardware: 64 NVIDIA A100 GPUs, using the VeRL framework (Sheng et al., 2025) for RL training. The parallel rollout collection (5 rollouts per prompt) benefits significantly from multi-GPU parallelism.

Why DAPO over standard PPO or GRPO. The paper doesn't explicitly justify DAPO over alternatives, but DAPO's key features—designed for language model RL at scale, with efficient rollout batching and dynamic clipping—are well-suited to the computational demands of generating 5 rollouts per prompt for 70K prompts across 3 epochs. The dynamic clipping range $[0.20, 0.28]$ is particularly relevant given the removed KL penalty: without the KL constraint to prevent large policy changes, the clipping mechanism becomes the primary guardrail against destructive updates, and the slightly higher upper bound (0.28 vs. standard 0.2) accommodates the larger policy improvements possible without KL regularization.

What happens during one RL training step:

Sampling: A batch of prompts (size 256) is sampled from the RL dataset. Each prompt consists of an image and a question.
Rollout generation: For each prompt, the current policy model generates 5 complete responses in the judge-think-answer format. These responses are produced by autoregressive sampling from the model's output distribution (the paper doesn't specify sampling temperature, but typical PPO implementations use temperature 1.0 for rollouts to maintain exploration).
Reward computation: Each of the 5 × 256 = 1,280 generated responses is evaluated by the reward judge (Gemini-2.5-Pro). The judge produces structured JSON outputs for each of the four reward components:
- Format: binary parseability check.
- Answer: comparison against ground-truth answer.
- Thinking: the three sub-scores (logic, hallucination, consistency) are computed using the specialized prompts from Appendix C, then averaged.
- Judging: comparison of the model's judgment against the ground-truth complexity label.
Cascading combination: The four component rewards are combined via the cascading product formula to produce a single scalar reward $R_{\text{total}}$ for each of the 1,280 responses.
Advantage estimation: For each prompt, the mean reward across its 5 rollouts is computed. The advantage for each rollout is its reward minus the mean reward for that prompt. This baseline normalization is crucial: it means the model is rewarded for generating responses that are better than its current average for that specific prompt, not for generating responses that happen to have high absolute reward. This prevents the model from simply memorizing which prompts yield high rewards and instead incentivizes improvement relative to its current capability.
Policy gradient computation: The DAPO algorithm computes the policy gradient using the advantage estimates, with the clipping mechanism bounding the policy ratio to prevent destructive updates.
Parameter update: Model parameters are updated using the AdamW optimizer with learning rate $1 \times 10^{-6}$ .

This process repeats across all prompts in the 70K dataset for 3 epochs, with 5 rollouts per prompt per epoch.

3.4.6 Design Rationale: Why Specific Choices Over Alternatives

The paper validates several key design choices through ablation studies (Section 4.5, Tables 6-8), but the rationale for these choices is grounded in the paper's broader argument about what makes RL for reasoning effective. Here we synthesize the principles behind the design.

Discrete vs. continuous reward signals (Table 7). The Thinking Reward's sub-components ( $d_1, d_2, d_3$ ) are binary rather than continuous $[0, 1]$ scalars. The paper's ablation shows discrete rewards outperform continuous ones by substantial margins (+2.1 on MathVision, +4.4 on LogicVista, +3.3 on MMMU). The rationale combines two arguments:

Judge calibration: Current LLM judges cannot reliably produce consistent fine-grained scores for reasoning quality. Asking an LLM to distinguish between a reasoning quality of 0.7 versus 0.8 introduces subjective noise—different judges, or even the same judge on different runs, will produce different continuous scores for the same reasoning trace. Binary decisions ("Is this step logically sound? Yes/No") are significantly more reproducible because they reduce the evaluation to a clear criterion rather than a scalar judgment.

RL optimization stability: Policy gradient methods are sensitive to reward variance. Noisy continuous scores increase the variance of advantage estimates, which in turn increases the variance of the policy gradient, slowing convergence and potentially causing training instability. Binary rewards provide a high signal-to-noise ratio: the difference between "good reasoning" (reward 1) and "bad reasoning" (reward 0) is maximally discriminative, while continuous scores compress this difference (0.7 vs. 0.8 represents a much smaller signal). In the extreme, if continuous scores for "good" and "bad" reasoning overlap significantly due to judge noise, the RL algorithm receives a weak or misleading training signal.

This is a non-obvious finding with significant practical implications: it suggests that for LLM-as-judge reward modeling, the conventional wisdom that "more granular feedback is better" may be wrong. Binary pass/fail signals, when the pass/fail criteria are well-defined, can provide more effective training guidance than noisy fine-grained scores.

Equal weighting of thinking sub-components (Table 8). The three sub-rewards (logic, hallucination, consistency) are combined with equal weights $1/3$ each, rather than biased toward any single dimension. The ablation shows that biasing weights creates a "see-saw effect": emphasizing logic (weights $1/2, 1/4, 1/4$ ) slightly improves MathVision (+0.4) but degrades LogicVista (-0.5) and MMMU (-0.3); emphasizing consistency (weights $1/4, 1/4, 1/2$ ) slightly improves MMMU (+0.4) but degrades the other two. The maximum variance across benchmarks under biased weighting is 0.7-0.9 points—modest but consistent enough to suggest a trade-off.

The rationale for equal weighting is that the three dimensions are genuine complements rather than substitutes: a reasoning chain needs all three properties to be useful. A logically sound but hallucinated chain is unreliable; a factually grounded but logically inconsistent chain is incoherent; a logically sound and factual chain that doesn't support the answer defeats the purpose of reasoning. Equal weighting reflects this complementarity—no single dimension can compensate for failures in another, so no single dimension should dominate the reward signal.

This design choice also has a practical benefit: it is robust across diverse tasks. The paper shows (Table 8) that equal weighting achieves the highest or near-highest performance across all three evaluated benchmarks, without the domain-specific tuning that biased weights would require. For a general-purpose training framework intended to work across diverse multimodal tasks, this robustness is valuable.

Cascading product vs. additive combination (Table 6). The multiplicative cascade $R_{\text{judge}} \cdot R_{\text{think}} \cdot R_{\text{answer}}$ substantially outperforms the additive alternative $\frac{1}{3}(R_{\text{judge}} + R_{\text{think}} + R_{\text{answer}})$ (+3.3 on MathVision, +3.1 on LogicVista, +2.3 on MMMU). The rationale is reward hacking prevention:

Under additive combination, the model can partially compensate for failures: a correct answer with wrong mode decision still yields $1/3$ reward, which provides a non-zero training signal that reinforces the lucky-guess behavior. Over many training steps, the model may learn to optimize the easiest-to-improve component (typically answer accuracy, since it has the clearest ground-truth signal) while neglecting the harder-to-learn components (reasoning quality and mode decisions).

Under multiplicative combination, the reward is zero if any component fails, regardless of success in other components. This eliminates the compensation pathway: there is no reward for lucky guessing, no reward for correct mode decisions with wrong answers, and no reward for correct answers with flawed reasoning. The only way to achieve non-zero reward is joint success across all three dimensions. This forces the model to learn coherent end-to-end behaviors where the mode decision, reasoning quality, and answer accuracy are all mutually consistent.

The paper conceptualizes this as turning the reward structure into a "logical AND gate." In digital logic terms, the output is 1 only when all inputs are 1. Translating this to RL: the policy gradient only receives positive reinforcement when every component of the reasoning episode is correct. This is a strong inductive bias that matches the paper's goal of building reliable reasoners—a model that sometimes thinks well but sometimes guesses, or that reasons correctly but gets wrong answers, is not reliable, and the AND-gate reward structure reflects this.

The role of $\alpha = 0.9$ and the format reward. The 10% weight on $R_{\text{format}}$ serves a pragmatic purpose that the additive-vs-multiplicative ablation doesn't fully capture. If the cascade term is frequently zero (as it will be early in training when the model hasn't yet learned good reasoning), the model receives zero reward on most rollouts. Without the format term, the gradient would be zero for these rollouts, meaning the model gets no signal at all—not even about producing structurally correct outputs. The format term provides a weak but non-zero gradient that maintains structural compliance even when content quality is poor. This prevents the model from drifting into unparseable output formats during the early stages of RL, which would make it harder to evaluate subsequent rollouts and could lead to training collapse.

The choice of 0.9 (rather than, say, 0.99 or 0.5) balances two concerns: giving enough weight to the cascade to make it the dominant optimization target (90%), while reserving enough weight for the format term to provide a reliable fallback gradient (10%). The paper doesn't ablate this value, so the exact choice may be somewhat arbitrary, but the principle—a small format bonus to maintain structural stability—is well-motivated.

Removing the KL penalty. The decision to remove the standard KL divergence penalty between the policy and the reference (SFT) model is a significant departure from standard RLHF practice that reflects the paper's goals. In standard RLHF, the KL penalty prevents the policy from diverging too far from the human-aligned SFT model, ensuring the model doesn't optimize the reward function at the cost of fluency, coherence, or safety. The paper removes this penalty because the SFT model's distribution is exactly what they want to escape: the SFT model was trained to mimic a teacher's reasoning, not to optimize reasoning quality or adaptivity. Penalizing deviation from this suboptimal distribution would constrain the RL optimization, preventing the model from discovering reasoning strategies that differ from the teacher's but achieve higher dual-reward scores.

The removal of the KL penalty is compensated for by the dynamic clipping mechanism (range $[0.20, 0.28]$ ), which provides a different form of regularization: rather than penalizing deviation from a reference distribution, clipping bounds how much the policy can change in a single update. This allows the policy to drift substantially from the SFT initialization over many updates (since each individual update is small) while preventing the kind of catastrophic single-update changes that can cause training collapse. This combination—no KL penalty, dynamic clipping—represents a deliberate trade-off prioritizing exploration over stability, appropriate when the initial policy is known to be suboptimal.

4. Key Insights and Innovations

Innovation 1: Reasoning Quality as a Trainable Objective, Not a Byproduct of Accuracy

The paper's most fundamental intellectual contribution is the reframing of reasoning quality from an emergent property that hopefully co-occurs with correct answers into a directly supervised objective within the RL training loop. Prior work in multimodal RL—exemplified by DeepSeek-R1 style training (Guo et al., 2025; Team et al., 2025a; Chen et al., 2025a)—operates under the implicit assumption that optimizing for answer correctness will naturally pull reasoning quality along with it. The intuition is simple: "if the model learns to get right answers more often, it must be reasoning better." SAIL-RL's diagnostic contribution is demonstrating that this assumption is empirically false and identifying the mechanism by which it breaks down.

The evidence for this claim is not merely the accuracy gains in Tables 1-3 but the training dynamics analysis in Figure 4. Under answer-only reward, the model's answer accuracy does improve over training, but the consistency score—measuring whether the reasoning trace actually supports the final answer—degrades. This is the "reasoning collapse" phenomenon: the policy gradient finds a path to higher answer reward that does not require maintaining logical fidelity between the reasoning process and the final output. The model learns, in effect, that it can produce reasoning as a decorative prefix that need not constrain the answer. The Thinking Reward's inclusion in the cascading product prevents this decoupling because $R_{\text{think}} < 1$ reduces the total reward even when $R_{\text{answer}} = 1$ , creating a continuous pressure to maintain reasoning quality alongside answer accuracy.

What makes this idea distinctive is not the technical mechanism—process supervision has been explored before in text-only settings (Lightman et al., 2023) and multimodal settings (Wang et al., 2025b; Luo et al., 2025)—but the diagnostic insight that reasoning and accuracy can diverge under RL optimization, and that this divergence is the expected outcome of outcome-only supervision rather than an anomaly. Prior work on process reward models treats step-level supervision as a way to improve reasoning beyond what answer-only training achieves. SAIL-RL's contribution is showing that answer-only training can actually reduce reasoning quality from its starting point, and that the Thinking Reward is therefore a corrective for an active degradation process rather than merely an amplifier of existing capabilities. This reframes the role of process supervision from "nice to have" to "necessary for stable optimization," which is a stronger claim with broader implications for how multimodal RL should be designed.

The significance of this insight extends beyond the paper's specific implementation. If reasoning quality and answer accuracy can decouple under RL, then any system that relies on LLM-generated reasoning traces for downstream purposes—explainability, verification, human oversight, or self-improvement loops—is vulnerable to a silent degradation where the model appears to be improving (accuracy goes up) while actually becoming less reliable (reasoning becomes decorrelated from answers). This has direct implications for the deployment of reasoning models in high-stakes domains and suggests that process-level evaluation should be a standard component of RL training pipelines, not an optional enhancement.

Innovation 2: The Cascading Product as a Logical AND Gate for Reward Hacking Prevention

The second distinctive contribution is the conceptualization of reward combination as a logical constraint rather than a weighted trade-off. The dominant approach to multi-objective RL in language model training—seen across RLHF (Ouyang et al., 2022), DPO variants (Rafailov et al., 2023), and most multimodal RL work—is additive reward aggregation: different reward components are weighted and summed to produce a scalar that the policy optimizes. This treats the components as partially substitutable: a weakness in one dimension can be compensated by strength in another.

SAIL-RL's cascading product $R_{\text{judge}} \cdot R_{\text{think}} \cdot R_{\text{answer}}$ rejects this substitutability premise. By multiplying the three core rewards together, it enforces what the paper characterizes as a "logical AND gate": the total reward is zero if any component fails, regardless of success in other components. This is a fundamentally different optimization landscape. Under additive combination, a model that correctly decides to think, produces flawless reasoning, but makes an arithmetic error in the final answer receives $2/3$ of maximum reward. Under multiplicative combination, it receives zero. The gradient signal is qualitatively different: rather than telling the model "you did well on two out of three things," it says "you failed."

The ablation in Table 6 validates that this difference matters substantially: cascading product outperforms additive combination by +3.3 on MathVision, +3.1 on LogicVista, and +2.3 on MMMU. The magnitude of these gaps—consistent across diverse benchmarks—suggests this is not a minor tuning difference but a structural property of multi-objective RL for reasoning tasks.

What makes this idea intellectually distinctive is that it inverts the standard approach to reward design. Rather than asking "how should we weight these components to express our preferences?", the cascading product asks "which components are logically necessary conditions for success, and how can we encode that necessity in the reward structure?" This shifts the design problem from preference elicitation (trading off objectives) to constraint encoding (identifying dependencies). The logical AND gate is not an empirical finding about what weights work best—it is a principled design derived from the structure of the reasoning task itself, where the correctness of the mode decision, the quality of the reasoning, and the accuracy of the answer are genuinely non-substitutable. A perfect mode decision with zero-quality reasoning is useless; perfect reasoning with a wrong answer is useless; a correct answer through a wrong-mode decision is a lucky guess that undermines reliability.

This contribution has implications beyond SAIL-RL's specific application. The cascading product is a general template for reward design in any domain where multiple success conditions must be satisfied jointly, and where allowing compensation between conditions would create reward hacking incentives. The paper's explicit contrast with additive combination provides a clear diagnostic: whenever reward components are logical necessities rather than preferences, multiplicative combination is the appropriate structure. This insight, while simple in retrospect, represents a conceptual advance in how the field thinks about multi-dimensional reward design for language model training.

Innovation 3: Making Metacognition a First-Class Training Objective

The third distinctive contribution is treating the decision about whether to reason not as an architectural feature (a separate router module) or a post-hoc optimization (efficiency tuning after reasoning capability is established) but as an integral part of the RL training objective that co-evolves with reasoning capability. This is the Judging Reward's conceptual contribution: it transforms "when to think" from a separate system design problem into a learned behavior that is optimized jointly with reasoning quality.

Prior work on adaptive reasoning in MLLMs—notably OThink-R1 (Zhang et al., 2025) and mode-switching architectures (Zhou et al., 2025)—treats the routing decision as a separate optimization problem. A router module is trained (typically via supervised learning on human-annotated difficulty labels) to classify tasks as requiring System 1 (fast) or System 2 (slow) processing, and this router gates whether the model engages in chain-of-thought reasoning. The reasoning capability itself is trained independently, and the router is bolted on afterward. This separation creates two problems: the router's difficulty assessment cannot adapt as the model's reasoning capability improves (a problem that was "hard" for the base model might become "easy" after training, but the router doesn't know this), and the router's errors propagate without feedback—if the router misclassifies a complex task as simple, the resulting wrong answer provides no signal back to the router.

SAIL-RL's Judging Reward addresses both problems by making the mode decision part of the same RL optimization loop as the reasoning quality. Because $R_{\text{judge}}$ appears in the cascading product with $R_{\text{think}}$ and $R_{\text{answer}}$ , the model receives joint feedback: a wrong mode decision nullifies the reward even if the reasoning (had it been produced) would have been perfect and the answer correct. This means the model cannot learn to produce correct answers in one part of the parameter space while maintaining a separate, unoptimized mode-decision mechanism—the gradient flows through the mode decision and forces it to co-adapt with reasoning capability. As the model gets better at reasoning, its understanding of what constitutes a "complex" task can evolve, because the reward signal reflects the actual relationship between task difficulty, reasoning effort, and answer accuracy rather than a static human annotation.

The evidence that this joint optimization produces genuinely adaptive behavior is in Figure 5 and Table 5. The model achieves near-saturated reasoning trigger rates on mathematical reasoning benchmarks (99-100%) while dropping to 7.5% on OCRBench, with a smooth gradient of trigger rates across benchmarks of intermediate complexity. Critically, this adaptivity does not come at the cost of accuracy on simple tasks—Table 5 shows that SAIL-RL achieves 91.3 on OCRBench, higher than the always-thinking baseline (88.7) and even slightly above the never-thinking baseline (90.5), suggesting the model has learned when its reasoning would be counterproductive and avoids it selectively rather than just learning a blanket bias toward or against reasoning.

This contribution is significant because it establishes that metacognitive control—deciding how much cognitive effort to expend—can be learned through the same RL framework that learns the cognitive operations themselves, without requiring a separate architectural component or training pipeline. This has implications for building more autonomous AI systems that can manage their own computational resources appropriately, and it opens the door to more sophisticated forms of learned resource allocation (e.g., deciding not just whether to reason but how deeply, or deciding when to seek external information vs. relying on internal knowledge).

Innovation 4: Empirical Demonstration That Uniform Reasoning Can Be Actively Harmful

While the inefficiency of applying deep reasoning to simple tasks is intuitive—it wastes computation—the paper provides a stronger and more counterintuitive finding: forced reasoning on perception-heavy tasks actively reduces accuracy. This is not merely an efficiency concern but an effectiveness one, and it challenges the implicit assumption in the System 2 reasoning literature that "more thinking is never worse, just more expensive."

The evidence is in Table 5. On OCRBench, the always-thinking baseline achieves 88.7 accuracy using 4.7× more tokens than the never-thinking baseline, which achieves 90.5. In other words, forcing the model to reason through a simple OCR task makes it less accurate by 1.8 percentage points while costing nearly 5 times as much computation. SAIL-RL's adaptive approach achieves 91.3—higher than both extremes—by selectively engaging reasoning only when it is beneficial. Similarly, Table 4 shows that always-thinking degrades performance on HallusionBench (58.3 vs. 61.5 for SAIL-RL, a +3.2 gap) and MMBench (88.6 vs. 90.4, +1.8).

The mechanism behind this harmful effect is not fully unpacked in the paper, but the qualitative examples and the design of the Factual Grounding Reward suggest an explanation: when forced to generate reasoning for simple perceptual tasks, the model produces chains of thought that are not constrained by the task's actual information requirements. Simple recognition tasks have low entropy in their correct outputs—there is essentially one right answer—but the reasoning generation process has high entropy because there are many possible pseudo-analytical justifications the model could invent. This entropy creates opportunities for hallucination: the model generates spurious observations, alternative interpretations, or invented details that are not grounded in the image, and these can contaminate the final answer. The Judging Reward prevents this by allowing the model to skip the high-entropy reasoning generation step entirely for tasks where the direct perceptual pathway is more reliable.

This finding has significant implications for how the field thinks about reasoning in MLLMs. The dominant narrative—reinforced by the success of models like OpenAI's o1 and DeepSeek-R1—is that more reasoning is better, and the primary constraint is cost. SAIL-RL's results suggest a more nuanced picture: reasoning is a tool that is appropriate for some tasks and counterproductive for others, and forcing it uniformly creates a previously underappreciated accuracy penalty on perception-heavy tasks. This reframes the efficiency argument for adaptive reasoning into an effectiveness argument: it's not just that we can save compute by skipping reasoning on simple tasks, but that skipping reasoning actually makes the model more accurate on those tasks. This is a stronger claim with clearer practical implications for deployment.

5. Experimental Analysis

Evaluation Methodology

Dataset. The evaluation uses VLMEvalKit (Duan et al., 2024) across two categories of benchmarks. Multimodal Reasoning is assessed on DynaMath, LogicVista, MathVerse, MathVision, MathVista, and WeMath — these are competition-level math and logic benchmarks requiring multi-step deduction from visual inputs. General Multimodal Understanding is assessed on MMMU (expert-level multimodal reasoning across disciplines), MMBench (general VQA), MME (perception and cognition), ChartQA (chart comprehension), AI2D (diagram understanding), OCRBench (optical character recognition), and HallusionBench (hallucination detection). The evaluation uses GPT-4o-Mini as the model judge for answer grading.
Base model(s). The primary model is SAIL-VL2 (Yin et al., 2025) at both 8B and 2B scales, which integrates the AimV2 (Fini et al., 2025) vision encoder with the Qwen3 (Yang et al., 2025a) language model backbone. The 8B variant is the paper's flagship configuration because it represents the scale where open-source models typically compete with each other and where the reasoning-vs-understanding trade-off is most apparent. To demonstrate cross-architecture generalization, experiments are also conducted on Qwen2.5-VL (Bai et al., 2025) at 3B and 7B scales (Table 10). Comparison models include proprietary systems (GPT-4o, Gemini-2.0-Pro) and open-source competitors at similar scales (InternVL3-8B, Keye-VL-8B-Thinking, Kimi-VL-A3B-Thinking, WeThink-7B, and several others listed in Tables 1-2).
Metrics. The primary metric is accuracy (%) — the fraction of test questions for which the model's final answer matches the ground truth as graded by GPT-4o-Mini. The paper reports per-benchmark accuracy and an average score across reasoning benchmarks (Table 1) and general understanding benchmarks (Table 2) separately. For the efficiency analysis, thinking trigger rate (%) measures the fraction of test instances where the model activates the \think reasoning mode, and normalized token usage reports inference cost relative to a never-thinking baseline (where 1.0× represents the direct-answering token count). These are reported in Table 5.
Baselines. The paper compares against multiple internal and external baselines:
- SAIL-VL2-8B-Instruct and SAIL-VL2-2B-Instruct: the base instruction-tuned models without any CoT or RL post-training, representing the starting point before reasoning enhancement.
- SAIL-VL2-8B-LongCoT and SAIL-VL2-2B-LongCoT: models after the SFT stage (judge-think-answer format training) but before RL optimization, isolating the contribution of the RL stage over SFT alone.
- Answer-only RL baseline (referenced in Tables 3-4 and Figure 4): SAIL-VL2-8B trained with RL using only the answer reward ( $R_{\text{answer}}$ ), representing the outcome-only supervision paradigm that SAIL-RL argues against. This is the primary internal ablation baseline for the Thinking Reward.
- Always-thinking baseline (Tables 4-5): a variant where the model is forced to generate reasoning for every input, representing the uniform reasoning strategy common in prior work. This is the primary internal ablation baseline for the Judging Reward.
- Never-thinking baseline (Table 5): a variant where the model directly answers without any reasoning, representing the System 1 extreme.
- Judge-without-reward baseline (Table 5): the model produces judgments during the LongCoT SFT stage but receives no explicit judging reward during RL, testing whether the judgment capability emerges from SFT alone.
- For external comparison, the paper reports published results for GPT-4o-latest, Gemini-2.0-Pro, InternVL3-8B, Qwen2.5-VL-7B, VL-Rethinker-7B, VLAA-Thinker-7B, Keye-VL-8B-Thinking, Kimi-VL-A3B-Thinking, and WeThink-7B, all evaluated under the same VLMEvalKit protocol with GPT-4o-Mini judging.
Generation budget / compute accounting. The paper does not use a FLOPs-based compute budget in the style of scaling laws papers. Instead, efficiency is measured through normalized token usage (Table 5), where the token count for each strategy is normalized by the never-thinking baseline's token count (1.0×). This captures the practical inference cost since MLLM generation time scales approximately linearly with output token count. For the RL training stage, the computational budget is reported in terms of hardware configuration (64 NVIDIA A100 GPUs), training epochs (3 over 70K samples), rollouts per sample (5), and PPO batch size (256), providing a complete accounting of training cost even if not directly comparable to inference-time budgets.
Cross-validation / statistical protocol. The paper does not report cross-validation or confidence intervals for benchmark results. The evaluation is a single-pass inference on standard test sets using VLMEvalKit with a fixed judge model (GPT-4o-Mini). For the ablation studies in Section 4.5, specific benchmarks (MathVision, LogicVista, MMMU) are selected as representative probes for reasoning and understanding capabilities, but results are reported as point estimates without variance information. The RL training dynamics curves in Figure 4 are monitored throughout training to confirm convergence and stability, and the paper reports that training is run for 3 epochs with consistent improvement trends, but formal statistical testing (e.g., paired bootstrap, significance tests) is not employed.

Main Quantitative Results

Reasoning Benchmark Performance (Table 1)

SAIL-VL2-8B-Thinking achieves an average score of 59.3 across the six multimodal reasoning benchmarks (DynaMath, LogicVista, MathVerse, MathVision, MathVista, WeMath), representing a +20.0 point improvement over the SAIL-VL2-8B-Instruct baseline (39.3). This is the highest reported result among open-source models at comparable scales, surpassing Keye-VL-8B-Thinking (56.6) by 2.7 points and Kimi-VL-A3B-Thinking (52.1) by 7.2 points. Among proprietary systems, SAIL-VL2-8B-Thinking outperforms GPT-4o (54.8) by 4.5 points and Gemini-2.0-Pro (56.6) by 2.7 points.

The gains are not uniform across benchmarks. The largest absolute improvements over the SAIL-VL2-8B-Instruct baseline occur on MathVerse (+32.2: from 32.9 to 65.1) and WeMath (+22.4: from 35.8 to 58.2), both of which test mathematical reasoning from visual diagrams and require multi-step derivations. DynaMath shows a +20.5 improvement (from 17.8 to 38.3), LogicVista +18.8 (from 45.0 to 63.8), and MathVision +21.8 (from 27.6 to 49.4). MathVista, where the base model already achieves relatively high performance (76.4), shows a more modest +4.5 gain to 80.9, consistent with the paper's difficulty-dependent analysis — tasks where the base model already performs well have less room for improvement through better reasoning strategies.

The intermediate LongCoT SFT stage (SAIL-VL2-8B-LongCoT) achieves an average of 52.1, meaning the SFT stage alone accounts for approximately 64% of the total improvement over the Instruct baseline (12.8 points out of 20.0). The remaining 7.2 points come from the RL stage with the dual-reward system. This decomposition is visible in the per-benchmark results: LongCoT alone brings MathVerse from 32.9 to 53.1 (+20.2), and RL adds another 12.0 points to 65.1. For LogicVista, LongCoT contributes +13.2 (45.0 → 58.2) and RL adds +5.6 (58.2 → 63.8). The RL stage's marginal contribution varies by benchmark, being largest on MathVerse and smallest on MathVista, which may reflect differences in how much reasoning quality improvement translates to answer accuracy gains across different problem types.

At the 2B scale, SAIL-VL2-2B-Thinking achieves an average of 44.6, a +13.6 improvement over SAIL-VL2-2B-Instruct (31.0). This is a larger relative improvement (44% increase) than at 8B (51% increase), suggesting that smaller models may benefit proportionally more from improved reasoning strategies.

General Understanding Benchmark Performance (Table 2)

SAIL-VL2-8B-Thinking achieves an average score of 80.8 across seven general multimodal understanding benchmarks (MMMU, MMBench, MME, ChartQA, AI2D, OCRBench, HallusionBench), representing a +3.6 point improvement over SAIL-VL2-8B-Instruct (77.2) and a +2.7 point improvement over SAIL-VL2-8B-LongCoT (78.1). This is the highest open-source result at the 8B scale among the compared systems, exceeding Kimi-VL-A3B-Thinking (78.4) by 2.4 points and Keye-VL-8B-Thinking (77.6) by 3.2 points. Against proprietary models, SAIL-VL2-8B-Thinking's 80.8 surpasses both Gemini-2.0-Pro (77.9) and GPT-4o (79.0).

The moderate gains on general understanding benchmarks compared to reasoning benchmarks (+3.6 vs. +20.0) are expected: general benchmarks include many perception-heavy tasks where reasoning is less critical, and the base model's performance is already relatively high (MMBench at 90.2, AI2D at 87.7). The largest improvements within this category occur on HallusionBench (+6.4: from 55.1 to 61.5) and MMMU (+10.7: from 55.4 to 66.1), both of which require some level of analytical reasoning beyond simple perception, suggesting that the Thinking Reward's benefits are most pronounced when the task demands integration of visual information with logical inference. OCRBench shows a modest +0.8 improvement (90.5 to 91.3) while ChartQA gains +3.3 (90.3 to 93.6).

A notable pattern is that the LongCoT SFT stage provides only marginal gains on general benchmarks (+0.9 average: 77.2 → 78.1) compared to the RL stage (+2.7: 78.1 → 80.8). This is the inverse of the reasoning benchmark pattern, where SFT provided the majority of gains. The likely explanation is that on perception-heavy tasks, the SFT stage's judge-think-answer format training primarily teaches the model when to skip reasoning rather than how to reason better, and the RL stage's Judging Reward provides the optimization signal needed to actually learn the correct mode-selection policy. On reasoning tasks, the SFT stage's chain-of-thought data provides substantial improvements in reasoning procedure, with RL refining quality.

Thinking Reward Ablation: Reasoning Quality Improvement (Table 3, Figure 4)

The Thinking Reward's contribution is isolated by comparing SAIL-RL (answer + thinking reward) against an answer-only RL baseline using SAIL-VL2-8B. On four STEM benchmarks, SAIL-RL outperforms the answer-only baseline by: +3.2 on WeMath (58.2 vs. 55.0), +2.4 on LogicVista (63.8 vs. 61.4), +2.1 on MathVision (49.4 vs. 47.3), and +1.6 on DynaMath (38.3 vs. 36.7). The consistent positive gains across diverse benchmarks (mathematical reasoning, logical deduction, visual proof comprehension) confirm that the Thinking Reward provides a signal that meaningfully improves reasoning beyond what answer-correctness optimization alone achieves.

The training dynamics in Figure 4 reveal the mechanism behind these gains. Four metrics are tracked over the course of RL training for both the SAIL-RL (answer+thinking) and answer-only configurations:

Logic score (think_logic_score): Both configurations start at approximately 0.55-0.60 and improve over training. The SAIL-RL curve consistently runs higher, ending near 0.67 vs. approximately 0.60 for the answer-only baseline. The gap widens in the later training stages, suggesting that sustained pressure from the logical coherence sub-reward prevents the plateau that the answer-only baseline experiences.
Hallucination score (think_hallucination_score): Similar pattern — both start near 0.55, SAIL-RL reaches approximately 0.67, answer-only plateaus near 0.58. The hallucination mitigation is notably important because outcome-only supervision has no mechanism to penalize fabricated facts in reasoning, so the Thinking Reward's factual grounding component provides a gradient that answer-only training lacks entirely.
Consistency score (think_answer_consistency_score): This is where the divergence is most dramatic and most diagnostically important. The answer-only baseline's consistency score starts at approximately 0.85-0.90 and declines over training to approximately 0.80, while SAIL-RL's consistency score remains near 0.95 throughout. This is the "reasoning collapse" phenomenon described in Section 3: without explicit supervision on reasoning-to-answer fidelity, the model progressively learns that correct answers can be produced without coherent reasoning, and the consistency degrades. The Thinking Reward's answer consistency sub-reward ( $d_3$ ) directly penalizes this decoupling, maintaining alignment between the reasoning trace and the final output.
Accuracy score (acc_score): Both configurations start at approximately 0.45-0.50 and improve to approximately 0.58 (answer-only) and 0.60 (SAIL-RL). The narrower gap on accuracy compared to the reasoning quality metrics demonstrates the paper's central argument: answer-only training achieves similar accuracy improvements but through degraded reasoning quality, while SAIL-RL achieves better accuracy and maintains or improves reasoning quality across all dimensions.

The consistency score's decline under answer-only training is the most consequential finding in this ablation. It provides direct evidence for the "accidental correctness" or "lucky success" pathology that the paper identifies as the core failure mode of outcome-only supervision: the model learns that the reasoning trace is not causally connected to the answer under the reward function, so it progressively decouples them. The Thinking Reward prevents this by making the answer reward contingent on reasoning quality through the cascading product — if consistency is zero, the product $R_{\text{think}} \cdot R_{\text{answer}}$ is reduced even when $R_{\text{answer}} = 1$ , creating a gradient that pulls the model back toward coherent reasoning.

Judging Reward Ablation: Adaptive Efficiency (Tables 4-5, Figure 5)

The Judging Reward's contribution is isolated by comparing SAIL-RL against an "always thinking" baseline that forces the model to engage in full reasoning for every input, regardless of complexity. Table 4 reports the comparison on five general benchmarks:

MMMU: SAIL-RL 66.1 vs. always-thinking 64.5 (+1.6)
MMBench: SAIL-RL 90.4 vs. always-thinking 88.6 (+1.8)
MME: SAIL-RL 86.0 vs. always-thinking 83.8 (+2.2)
OCRBench: SAIL-RL 91.3 vs. always-thinking 88.7 (+2.6)
HallusionBench: SAIL-RL 61.5 vs. always-thinking 58.3 (+3.2)

The pattern is striking: forced reasoning degrades performance on every single benchmark, despite the always-thinking baseline being identical to SAIL-RL in architecture and base training — the only difference is the Judging Reward during RL. The degradation is largest on HallusionBench (+3.2) and OCRBench (+2.6), both of which are perception-heavy tasks where hallucination risk from unnecessary reasoning is highest. This confirms the paper's claim that uniform reasoning is not merely inefficient but actively harmful to accuracy.

Table 5 provides the efficiency breakdown. On MathVision (a reasoning-intensive benchmark), all strategies that engage reasoning achieve substantially higher accuracy than never-thinking (27.6): always-thinking reaches 48.7, judge-without-reward reaches 47.5, and SAIL-RL reaches 49.4. The token overhead is substantial — always-thinking uses 5.4× the tokens of never-thinking, but the accuracy gain of +21.1 points justifies the cost on this reasoning-heavy task. SAIL-RL uses 5.1× tokens (slightly less than always-thinking because on the rare simple MathVision problems it may skip reasoning) while achieving the highest accuracy.

On OCRBench (a perception-heavy benchmark), the picture reverses. Never-thinking achieves 90.5 accuracy at 1.0× tokens. Always-thinking incurs a 4.7× token overhead while degrading accuracy to 88.7 — the extra computation actively hurts. Judge-without-reward partially moderates this, triggering thinking on only 47.6% of samples, using 2.9× tokens, and achieving 89.8 accuracy — better than always-thinking but still worse than never-thinking. SAIL-RL achieves the best of both worlds: it triggers thinking on only 7.5% of OCRBench samples, uses only 1.2× tokens (barely more than never-thinking), and achieves 91.3 accuracy — higher than all other strategies. The model has learned to selectively engage reasoning on the small fraction of OCRBench problems that actually benefit from it while efficiently processing the rest through direct perception.

Figure 5 visualizes the thinking trigger rates across all evaluation benchmarks, revealing a sophisticated learned difficulty gradient. The reasoning benchmarks (blue) show near-saturated trigger rates: 100.0% on LogicVista, 99.8% on MathVision, 99.2% on MathVerse, 99.1% on WeMath, 97.6% on DynaMath, and 94.0% on MathVista. The general understanding benchmarks (purple) show a smooth spectrum reflecting task complexity: MMMU triggers reasoning on 99.3% of samples (this benchmark requires expert-level multimodal reasoning), AI2D on 93.3%, MMBench on 87.8%, MMStar on 84.3%, MMBench on 77.2%, MMVet on 75.2%, and OCRBench on only 7.5%. This gradient suggests the model has learned a nuanced internal representation of task difficulty that aligns with human intuitions — OCR is almost always simple perception, chart comprehension and diagram understanding sometimes require analysis, and MMMU-level expert problems almost always require deep reasoning.

The judge-without-reward baseline (Table 5) provides an important negative result: without explicit reward supervision on the mode decision, the SFT stage's judgment training does not alone produce optimal adaptivity. On MathVision, judge-without-reward triggers thinking on 90.4% of samples (much higher than optimal, which should be near 100%) and achieves only 47.5 accuracy vs. SAIL-RL's 49.4. On OCRBench, it triggers on 47.6% (far too high) and achieves 89.8 vs. SAIL-RL's 91.3. This demonstrates that the Judging Reward during RL optimization is essential — the model needs reward feedback on its mode decisions to calibrate the appropriate trigger rates, and SFT alone provides the format but not the optimization pressure to get the decisions right.

Reward Mechanism Design Ablations (Tables 6-8)

Three design choices in the reward system are ablated to validate their contribution to final performance.

Cascading product vs. additive combination (Table 6): The multiplicative reward $R_{\text{judge}} \cdot R_{\text{think}} \cdot R_{\text{answer}}$ is compared against the additive baseline $\frac{1}{3}(R_{\text{judge}} + R_{\text{think}} + R_{\text{answer}})$ . On MathVision, the cascading product achieves 49.4 vs. 46.1 for additive (+3.3). On LogicVista, 63.8 vs. 60.7 (+3.1). On MMMU, 66.1 vs. 63.8 (+2.3). The consistency and magnitude of these gaps — averaging +2.9 points across three diverse benchmarks — confirm that the logical AND gate property of multiplicative combination prevents the reward hacking that additive combination allows. Under additive combination, the model can receive partial reward for lucky guesses (correct answer despite wrong mode decision, yielding 1/3 reward) or for good reasoning with wrong answers (2/3 reward), diluting the optimization signal toward truly joint success.

Discrete vs. continuous reward signals (Table 7): Binary $\{0, 1\}$ rewards for the Thinking Reward's sub-components are compared against continuous $[0, 1]$ scalars. On MathVision, discrete rewards achieve 49.6 vs. 47.5 for continuous (+2.1). On LogicVista, 63.8 vs. 59.4 (+4.4). On MMMU, 66.1 vs. 62.8 (+3.3). The average improvement of +3.3 points is substantial, and the gap is largest on LogicVista (+4.4), which is notable because logical coherence evaluation is where continuous scoring would seem most natural (logic isn't perfectly binary) yet performs worst. The paper attributes this to judge calibration issues — LLM judges cannot produce consistent fine-grained scores — and to RL optimization stability — binary rewards provide lower-variance advantage estimates. This is a practically significant finding because it suggests that for LLM-as-judge reward modeling, coarse binary signals should be preferred over fine-grained continuous ones, contrary to the intuition that more granular feedback is better.

Weighting scheme for thinking sub-components (Table 8): Equal weighting (1/3 each for logic, hallucination, and consistency) is compared against three biased schemes: biased-logic (1/2, 1/4, 1/4), biased-hallucination (1/4, 1/2, 1/4), and biased-consistency (1/4, 1/4, 1/2). The results reveal a "see-saw effect" where emphasizing one dimension produces small gains in some benchmarks but losses in others. Equal weighting achieves 49.6 on MathVision, 63.8 on LogicVista, and 66.1 on MMMU. The biased schemes produce maximum variances across benchmarks of 0.8 (biased-logic), 0.9 (biased-hallucination), and 0.7 (biased-consistency) — modest but consistent enough to suggest a genuine trade-off. No single biased scheme dominates equal weighting across all three benchmarks, confirming that the three dimensions are genuine complements where all are necessary for robust reasoning. Equal weighting provides the most stable performance without requiring per-benchmark weight tuning.

Cross-Architecture Generalization (Table 10)

To demonstrate that SAIL-RL is not specific to the SAIL-VL2 architecture, the framework is applied to Qwen2.5-VL at both 3B and 7B scales. The experiment compares three training configurations: SFT baseline (standard instruction tuning), answer-only RL (+Answer Reward), and SAIL-RL (dual-reward). Results are reported on MathVision, LogicVista, and MMMU — one reasoning-heavy benchmark, one logic benchmark, and one general understanding benchmark.

For Qwen2.5-VL-3B:

MathVision: SFT baseline 18.1 → +Answer Reward 25.2 → SAIL-RL 27.3 (+2.1 over answer-only)
LogicVista: SFT baseline 36.0 → +Answer Reward 41.2 → SAIL-RL 42.9 (+1.7 over answer-only)
MMMU: SFT baseline 48.1 → +Answer Reward 49.2 → SAIL-RL 51.7 (+2.5 over answer-only)

For Qwen2.5-VL-7B:

MathVision: SFT baseline 25.4 → +Answer Reward 27.1 → SAIL-RL 30.2 (+3.1 over answer-only)
LogicVista: SFT baseline 47.9 → +Answer Reward 51.7 → SAIL-RL 53.4 (+1.7 over answer-only)
MMMU: SFT baseline 58.1 → +Answer Reward 61.2 → SAIL-RL 63.1 (+1.9 over answer-only)

SAIL-RL consistently outperforms both baselines across all model scales and benchmarks. The average improvement of SAIL-RL over the answer-only RL baseline is approximately +2.1 points at 3B and +2.2 points at 7B, comparable to the improvements observed on SAIL-VL2. The gains over the SFT baseline are larger: at 7B, SAIL-RL improves MathVision from 25.4 to 30.2 (+4.8), which is particularly notable given that the answer-only baseline only improves to 27.1 (+1.7 over SFT), suggesting that for this model-task combination, outcome-only supervision provides limited benefit while the dual-reward system unlocks substantially more improvement.

An interesting pattern: the relative benefit of SAIL-RL over answer-only RL is larger at 7B than at 3B for MathVision (+3.1 vs. +2.1), suggesting that larger models may be better able to leverage the richer reward signal because they have more capacity to simultaneously optimize the multiple reward dimensions. However, on MMMU, the 3B model shows a slightly larger gain (+2.5 vs. +1.9), so the pattern is not monotonic with scale.

Reward Model Robustness (Table 9)

SAIL-RL's sensitivity to the choice of reward judge model is evaluated by comparing Gemini-2.5-Pro (the default), GPT-5, and Qwen2.5-VL-32B as the judge model for computing the Thinking and Judging Rewards. On MathVision, the results are 49.7 (GPT-5), 49.4 (Gemini-2.5-Pro), 48.4 (Qwen2.5-VL-32B). On LogicVista: 63.5, 63.8, 62.7. On MMMU: 66.4, 66.1, 64.9.

All three reward models produce results that substantially outperform the SFT baseline (27.6, 45.0, 55.4 on MathVision, LogicVista, MMMU respectively), demonstrating that SAIL-RL's dual-reward mechanism is not brittle to the specific judge model used. The ranking across benchmarks is consistent — GPT-5 performs slightly best overall, Gemini-2.5-Pro very close behind, Qwen2.5-VL-32B somewhat lower but still delivering strong improvements — with the performance gap between the best and worst judge model being approximately 1.3 points on average. This 1.3-point spread is modest compared to the 20+ point gains over the SFT baseline, suggesting that SAIL-RL's training framework is robust to reasonable variations in judge model quality and that the structural design of the reward system (cascading product, binary signals, equal weighting) is more important than the specific judge model used.

The paper notes that "the slight advantage of top-tier models highlights that RL training inherently benefits from more accurate and stable reward signals," which is an expected finding — a better judge provides clearer gradients — but the key takeaway is that even a weaker open-source judge model (Qwen2.5-VL-32B) provides sufficient signal quality for SAIL-RL to deliver substantial improvements, which lowers the practical barrier to adopting the framework since organizations don't need access to the most expensive proprietary models as reward judges.

Ablation Studies and Robustness Checks

Beyond the core reward mechanism ablations in Tables 6-9, the paper includes several additional analyses that validate specific aspects of the framework:

LongCoT SFT as a necessary precursor to RL: The consistent performance gap between SAIL-VL2-8B-LongCoT (SFT only) and SAIL-VL2-8B-Thinking (SFT + RL) across all benchmarks demonstrates that RL provides non-trivial gains beyond format training. On reasoning benchmarks, the SFT stage improves the average from 39.3 (Instruct) to 52.1, and RL adds another 7.2 points to reach 59.3. On general benchmarks, SFT provides only +0.9 (77.2 → 78.1), while RL adds +2.7 (78.1 → 80.8). The SFT stage is therefore critical for establishing the baseline reasoning capability that RL then optimizes, particularly on reasoning-intensive tasks, but RL is essential for both the final accuracy gains and the adaptive efficiency that SFT alone cannot provide.

Difficulty-based filtering for RL data: The RL dataset construction involves removing problems where pass@4 = 0 (too hard) or pass@4 = 1 (too easy) based on the SFT model's performance (Appendix A.1). While the paper does not ablate this filtering step directly, it reports that the filtering creates an "optimal difficulty range" of approximately 50K STEM problems from a larger initial pool. The rationale — that RL requires problems where the model has a meaningful probability of both success and failure to provide a useful training signal — is well-motivated by the RL theory but not empirically validated within this paper. This is a limitation of the current analysis, as the filtering step could be removing problems where the Thinking Reward and Judging Reward would be most informative (e.g., on very hard problems where the model consistently fails, the Judging Reward might teach the model to recognize its own limitations and respond appropriately).

5 rollouts per sample for advantage estimation: The RL training uses 5 rollouts per prompt to estimate the advantage function (the difference between a rollout's reward and the average reward for that prompt). The paper does not ablate this number, but the choice of 5 represents a trade-off: more rollouts provide lower-variance advantage estimates but increase the computational cost of each PPO iteration (since each rollout must be generated and evaluated by the reward judge). The paper's training configuration (64 GPUs, 256 PPO batch size) suggests that 5 was the maximum feasible number given hardware constraints and training time budgets. Whether 3 or 7 rollouts would produce meaningfully different results is not explored.

Removing the KL penalty: As discussed in Section 3.4.5, the paper removes the standard KL divergence penalty between the policy and reference model, which is unusual in RLHF practice. The paper justifies this as encouraging exploration beyond the SFT model's suboptimal distribution, and the dynamic clipping mechanism (range [0.20, 0.28]) provides an alternative form of regularization. However, the paper does not ablate this choice — there is no comparison of SAIL-RL with and without the KL penalty — so the contribution of this design decision to the final results cannot be isolated. This is a notable gap, as the KL penalty removal is a significant departure from standard practice, and its effect on training stability, final performance, and the model's tendency to drift from the SFT initialization is unquantified within this paper.

General QA data in the RL mix: The RL dataset includes 20K general QA samples alongside the 50K STEM problems. The paper does not ablate the effect of including vs. excluding these general QA samples. Their inclusion is motivated by the need to teach the Judging Reward across diverse task types — if RL training only included complex STEM problems, the model would never encounter examples where the correct judgment is "don't think," and the Judging Reward would provide no signal to learn adaptive mode selection. However, the specific ratio (50K:20K, approximately 2.5:1 STEM-to-general) and its effect on the learned trigger rates and accuracy-adaptivity trade-off is not systematically explored.

Critical Assessment

This section evaluates whether the reported experiments genuinely support the paper's three central claims, identifies specific weaknesses and missing experiments, and characterizes the precise conditions under which the claims hold.

Claim 1: SAIL-RL achieves state-of-the-art reasoning performance among open-source models while maintaining competitive general understanding.

The experimental evidence for this claim is strong and well-documented. Tables 1 and 2 provide direct comparisons against a broad set of open-source and closed-source competitors under a standardized evaluation protocol. The 59.3 average on reasoning benchmarks and 80.8 on general understanding benchmarks are the highest reported open-source results at the 8B scale. However, several caveats should be noted:

The comparison set is incomplete. The paper compares against models published before or contemporaneously with its work, but the fast-moving nature of the field means that new models achieving higher scores may have emerged since the paper's submission. This is not a weakness of the paper's methodology but limits the durability of the "state-of-the-art" claim.
The evaluation protocol (GPT-4o-Mini as judge) introduces an uncontrolled variable. While the paper uses the same protocol across all models to ensure fairness, GPT-4o-Mini's own biases and inconsistencies as an evaluator are not characterized. The paper does not report inter-judge agreement rates or compare GPT-4o-Mini's judgments against human evaluation. The absolute accuracy numbers should therefore be interpreted as relative rankings within a consistent evaluation framework rather than absolute measures of model capability.
The base model advantage is not fully isolated. SAIL-VL2-8B-Instruct achieves 39.3 on reasoning benchmarks, which is itself competitive with several compared models (Qwen2.5-VL-7B at 40.1, InternVL3-8B at 41.5). SAIL-RL's 59.3 represents a +20.0 point improvement over this base, but it's unclear how much of the advantage over competitors is due to SAIL-RL's training methodology versus the SAIL-VL2 base model's inherent capabilities. The cross-architecture experiments on Qwen2.5-VL (Table 10) partially address this by showing consistent gains when applying SAIL-RL to a different model family, but these results are limited to three benchmarks rather than the full evaluation suite, making it difficult to determine whether the 59.3 average on SAIL-VL2 would translate to a similarly competitive position if SAIL-RL were applied to other base architectures.
The 2B-scale results are less comprehensively benchmarked. Table 1 reports 2B-scale results on the full reasoning benchmark suite, but the general understanding results for SAIL-VL2-2B-Thinking in Table 2 (74.1 average) are compared against fewer external baselines, and the 2B-scale cross-architecture experiments are limited to three benchmarks (Table 10). The claim of generalization across scales is supported but the evidence is thinner at smaller scales.

Claim 2: The Thinking Reward substantially improves reasoning quality beyond outcome-only supervision, and this improvement is mediated by preventing reasoning collapse during RL training.

The experimental evidence for this claim is strong and includes the paper's most diagnostically important result. The training dynamics analysis in Figure 4 provides direct evidence for the mechanism: the answer-only baseline's consistency score declines over training while SAIL-RL's remains high, confirming the reasoning collapse hypothesis. The Thinking Reward ablation in Table 3 quantifies the performance impact (+1.6 to +3.2 points depending on benchmark), and the thinking sub-component ablations in Tables 7-8 validate the specific design choices.

Specific strengths:

The four-panel Figure 4 provides a rare window into how reasoning quality evolves during RL training, moving beyond the typical black-box reporting of final benchmark scores. The decoupling of consistency from accuracy under answer-only training is a genuinely important finding with implications beyond this paper.
The ablation of discrete vs. continuous rewards (Table 7) is practically valuable and non-obvious, providing actionable guidance for practitioners building LLM-as-judge reward systems.
The cascading product vs. additive combination ablation (Table 6) validates that the multiplicative structure is not merely a design choice but produces quantitatively superior results, supporting the paper's theoretical argument about reward hacking prevention.

Specific weaknesses and missing experiments:

The reasoning quality metrics (logic, hallucination, consistency scores) are themselves evaluated by an LLM judge (Gemini-2.5-Pro), which introduces circularity. The Thinking Reward is computed by the same judge model (or a similar one) that produces the training signals. Figure 4 shows that the RL training optimizes the model to produce higher scores on these judge-evaluated metrics, but it's possible that the model is learning to produce reasoning that looks good to the judge rather than reasoning that is genuinely better by some ground-truth standard. This is the perennial challenge of LLM-as-judge evaluation — without human-validated ground truth for reasoning quality, improved judge scores could reflect either genuine improvement or reward hacking of the judge itself. The qualitative examples in Appendix B (Figures 6-7) provide some face validity, but systematic human evaluation of reasoning quality would substantially strengthen this claim.
The ablation of thinking sub-component weighting (Table 8) is limited to three biased schemes plus equal weighting. The paper does not explore whether a learned weighting (e.g., via a small neural network or a dynamic weighting that adapts based on task characteristics) could outperform the static equal-weight scheme. The "see-saw effect" across benchmarks suggests that different tasks might genuinely benefit from different emphasis on logic vs. hallucination mitigation vs. consistency, but the paper's static weighting cannot capture this.
The Thinking Reward's generalization to non-STEM domains is not tested. All reasoning benchmarks in Table 1 are mathematical or logical reasoning tasks. Whether the Thinking Reward's dimensions (logical coherence, factual grounding, answer consistency) are appropriate and effective for other forms of reasoning — causal reasoning, moral reasoning, strategic planning, creative problem-solving — is untested. The general understanding benchmarks in Table 2 include some analytical tasks (MMMU), but these are not broken down by reasoning quality metrics.
The relationship between the three sub-rewards and final accuracy is not deeply analyzed. The paper reports that SAIL-RL achieves higher accuracy than the answer-only baseline (Table 3), but does not analyze which sub-rewards contribute most to accuracy gains on which types of problems. For example, does logical coherence primarily help on multi-step derivation tasks while factual grounding primarily helps on visual recognition tasks? Such an analysis would provide more actionable guidance for practitioners who might want to emphasize specific sub-rewards for specific application domains.

Claim 3: The Judging Reward enables adaptive reasoning that simultaneously improves efficiency and accuracy, and forced reasoning is actively harmful on perception-heavy tasks.

The evidence for this claim is compelling and represents the paper's most distinctive contribution. Table 5 provides the cleanest demonstration: on OCRBench, SAIL-RL achieves 91.3 accuracy using only 1.2× the tokens of the never-thinking baseline, while always-thinking achieves only 88.7 accuracy at 4.7× token cost. This is a clear case where adaptive reasoning improves both accuracy and efficiency simultaneously — the model is not trading one off against the other but achieving a Pareto improvement over both extremes. Figure 5's trigger rate analysis shows that the adaptivity is nuanced, with a smooth gradient of reasoning engagement across benchmarks rather than a binary switch.

Specific strengths:

The judge-without-reward baseline (Table 5) is a critical control experiment that isolates the contribution of the Judging Reward during RL from the judgment format training during SFT. The finding that SFT alone produces suboptimal trigger rates (47.6% on OCRBench vs. optimal 7.5%) demonstrates that the RL optimization of the mode decision is essential, not merely nice-to-have.
The per-benchmark trigger rates in Figure 5 show a sophisticated learned difficulty spectrum that aligns with human intuition about task complexity, providing face validity that the model has learned a genuine meta-cognitive capability rather than a brittle heuristic.
The across-the-board degradation of always-thinking vs. SAIL-RL on general benchmarks (Table 4, every benchmark shows a positive gap for SAIL-RL) provides strong evidence that the "overthinking is harmful" phenomenon is robust and not limited to one or two outlier benchmarks.

Specific weaknesses and missing experiments:

The ground-truth complexity labels used for the Judging Reward are not described in detail. The paper states that problems are labeled as requiring reasoning or not, but the labeling process, inter-annotator agreement, and potential ambiguities (e.g., problems that are on the boundary between simple and complex) are not discussed. If the ground-truth labels contain noise or systematic biases, the Judging Reward would propagate those biases into the model's learned mode-selection policy. The paper's finding that SAIL-RL outperforms both always-thinking and never-thinking baselines suggests the labels are informative, but the sensitivity of results to label quality is not ablated.
The "harmful" effect of overthinking is demonstrated but not explained. Tables 4-5 show that forced reasoning degrades accuracy, but the paper does not provide quantitative analysis of why — e.g., does overthinking introduce hallucinations (as the Factual Grounding Reward is designed to prevent), does it cause the model to talk itself out of correct initial intuitions, or does it simply consume output token budget that could have been used for a more careful direct answer? The qualitative example in Figure 7 (Appendix B) shows one instance of overthinking noise, but systematic categorization of overthinking failure modes would substantially strengthen the claim.
The trigger rate analysis (Figure 5) reports test-time behavior but does not guarantee causality. The smooth gradient of trigger rates across benchmarks is consistent with the model having learned to assess task complexity, but it's also consistent with simpler explanations — e.g., the model might learn to trigger reasoning based on superficial features of the question format (presence of mathematical notation, question length, benchmark-specific formatting artifacts) rather than genuine task complexity. An experiment that tests the model's adaptivity on out-of-distribution tasks (e.g., mathematical problems formatted like simple perception questions, or simple questions dressed in mathematical notation) would distinguish between genuine meta-cognition and shallow heuristics.
The efficiency analysis is limited to token counts and does not consider latency. Sequential reasoning generation adds latency beyond the token count overhead because it requires iterative autoregressive decoding. A model that generates 100 reasoning tokens followed by 20 answer tokens takes approximately 6× longer wall-clock time than one that generates 20 direct-answer tokens, even though the token ratio is 6:1. In latency-sensitive applications, the efficiency gains from the Judging Reward may be larger than token counts alone suggest. Conversely, parallel batching of multiple queries can amortize the token overhead differently. The paper does not discuss these practical deployment considerations.
The interaction between the Thinking and Judging Rewards on individual problems is not analyzed. Does the model ever correctly decide to skip reasoning but then produce a wrong answer because it should have reasoned? Does it ever correctly decide to reason but produce flawed reasoning that the Thinking Reward would penalize? A confusion matrix of mode-decision correctness × answer correctness × reasoning quality would characterize the model's failure modes more precisely than the aggregate accuracy and trigger rate statistics.

Cross-cutting concern: evaluation reliance on a single judge model (GPT-4o-Mini) without validation. The entire benchmark evaluation in Tables 1-2 uses GPT-4o-Mini as the answer grader. The paper does not report correlation between GPT-4o-Mini's grading and human evaluation, nor does it validate that the grading function is equally accurate across the diverse benchmarks used (math problems with structured answers, open-ended VQA, chart comprehension, OCR). Grading errors in any of these benchmarks would propagate into the reported accuracy numbers, and if grading errors are systematically correlated with model characteristics (e.g., GPT-4o-Mini might be more lenient toward certain answer formats), the rankings could be distorted. The consistency of the paper's internal comparisons (SAIL-RL vs. its own baselines) is robust to uniform grading bias, but the external comparisons against published results from other papers — which may have used different grading protocols — are less reliable.

Missing experiment: joint optimization of Thinking and Judging Rewards vs. sequential optimization. The paper trains both rewards simultaneously through the cascading product, but does not compare against a sequential approach where the model is first trained with the Thinking Reward alone (improving reasoning quality uniformly) and then trained with the Judging Reward to learn adaptivity. A sequential approach might achieve different trade-offs — for example, better reasoning quality at the cost of less refined adaptivity — and the comparison would reveal whether the joint optimization provides unique benefits beyond the convenience of single-stage training.

Missing experiment: the effect of the cascading product's $\alpha = 0.9$ parameter. The paper states that $\alpha = 0.9$ prioritizes content reward while preserving format compliance, but this value is not ablated. The format reward's 10% weight could be too high (wasting optimization budget on format when content is poor) or too low (insufficient to maintain structural compliance). The sensitivity of results to this parameter — within a reasonable range like [0.8, 0.95] — is unknown.

Overall assessment of experimental support: The paper's experiments provide strong support for its core claims, with the training dynamics analysis (Figure 4) and the adaptive efficiency demonstration (Tables 4-5, Figure 5) being particularly well-executed and informative. The main limitations are: (1) the reliance on LLM judges for both training rewards and evaluation without human validation of reasoning quality or grading accuracy, which introduces potential circularity; (2) the limited analysis of why overthinking is harmful beyond the qualitative example in Appendix B; (3) the absence of experiments testing whether the learned adaptivity generalizes to out-of-distribution task formats; and (4) several missing ablations that would clarify the sensitivity of results to specific design choices (KL penalty removal, $\alpha$ parameter, difficulty filtering threshold, number of rollouts per sample). These limitations do not undermine the paper's main contributions but represent opportunities for more rigorous validation in future work.

6. Limitations and Trade-offs

6.1 The Cascading Product's Zero-Tolerance Penalty Creates a Cold-Start Problem with No Partial Credit

The assumption or constraint. The cascading reward formulation $R_{\text{total}} = \alpha \cdot (R_{\text{judge}} \cdot R_{\text{think}} \cdot R_{\text{answer}}) + (1 - \alpha) \cdot R_{\text{format}}$ implements what the paper explicitly terms a "zero-tolerance penalty" (Section 3.3): if any of the three core components is zero, the entire cascade term is zero regardless of success in the other dimensions. This means the model receives exactly the same reward — zero — for a response that gets the mode decision right, produces flawless reasoning, but makes a minor arithmetic error in the final answer, as it does for a response that gets everything wrong. The reward function provides no signal about how close the model was to success.

The consequence. This creates a significant practical challenge for early-stage RL training. When the model has not yet learned to reliably produce correct answers with sound reasoning, the cascade reward will be zero on the vast majority of rollouts. In standard RL, zero reward across most samples means the advantage estimates are near-zero for those samples, providing negligible gradient signal for policy improvement. The $R_{\text{format}}$ term (10% weight) provides a weak non-zero gradient for structural compliance, but it offers no signal about reasoning quality or mode decisions. In effect, the model must discover successful (judge, think, answer) triplets through essentially random exploration before the reward signal can begin to shape behavior, and the zero-tolerance structure means that near-misses — which in continuous or additive reward schemes would provide directional gradients toward better solutions — provide no learning signal at all. This is the reinforcement learning analog of a cold-start problem: the reward function provides excellent discrimination between perfect and imperfect responses but minimal guidance for improving imperfect responses toward perfection.

A secondary consequence is that the cascading product may unintentionally penalize problems where the "correct" mode decision is genuinely ambiguous. The paper's Judging Reward uses binary ground-truth complexity labels (Section 3.2), but many real-world tasks fall in a gray area where either mode could be reasonable — a medium-complexity chart question might be answerable through direct perception by an expert model but benefit from reasoning for a weaker model. When the ground-truth label is arbitrary for such edge cases, the model receives zero reward 50% of the time regardless of response quality, creating a noisy and potentially misleading training signal.

What evidence exists in the paper. The paper's training dynamics (Figure 4) provide indirect evidence for this limitation. The accuracy score acc_score starts at approximately 0.45-0.50, meaning that even at the beginning of RL training — after the SFT stage — the model only produces correct answers on roughly half of rollouts. Since the cascading product requires correct answers and correct mode decisions and sound reasoning, the fraction of rollouts receiving non-zero cascade reward at the start of training is substantially below 50%, likely in the 10-30% range depending on how often the model passes all three conditions simultaneously. The training curves do show improvement over time, indicating that sufficient non-zero reward samples exist to drive learning, but the paper does not report the actual proportion of non-zero rewards during training, making it impossible to assess how severe the cold-start problem is in practice. The paper also does not report whether training occasionally encounters episodes where the reward is zero for an extended sequence of rollouts, which could cause temporary plateaus or instability.

Mitigation status. The paper partially mitigates this through the difficulty-based filtering of the RL dataset (Appendix A.1): problems where the SFT model's pass@4 is 0 (too hard — never gets the right answer) or 1 (too easy — always gets it right) are removed. This ensures that the RL dataset contains problems where the model has a "meaningful probability of both success and failure," which increases the fraction of rollouts that can potentially receive non-zero cascade rewards. However, this filtering addresses only the $R_{\text{answer}}$ component — the model must still get the mode decision and reasoning quality correct to receive non-zero reward, and these are not filtered for. The paper does not discuss whether additional warm-start strategies (e.g., curriculum learning where the cascade constraint is gradually tightened, or initial training with an additive reward that is annealed toward the multiplicative form) would improve training efficiency. The format reward's 10% weight provides a fallback gradient, but 10% is a weak signal and the paper does not ablate whether higher format weights during early training (with annealing toward 0.9/0.1) would accelerate convergence.

6.2 Difficulty Estimation for Judging Reward Relies on Undisclosed Ground-Truth Labels, Limiting Scalability

The assumption or constraint. The Judging Reward requires ground-truth complexity labels for every problem in the RL training dataset — a binary classification of whether each problem "requires reasoning" or can be answered directly. The paper states that the model's decision is "evaluated against ground-truth complexity labels" (Section 3.2: "This decision is evaluated against ground-truth complexity labels. The reward d_judge is binary: it is set to 1 if the model's decision aligns with the ground truth (i.e., choosing thinking mode for complex tasks or no-thinking mode for simple tasks); otherwise, it is 0"). The paper does not describe how these labels are generated for the RL dataset beyond noting that the SFT data pipeline (Appendix A.1) assigns complexity labels during the conditional annotation step. Specifically: "For complex problems requiring reasoning, we use a guided-prompting strategy to generate a detailed chain-of-thought for the \think section, and the <judge> tag is set to indicate that thinking is necessary. For simple perceptual tasks, the <judge> tag is set to indicate that the question can be answered directly."

The consequence. This labeling dependency creates two significant scalability barriers. First, the annotation process — whether done by human annotators or by a strong LLM judge — introduces its own costs and potential biases. If labels are generated by the same class of models used as the reward judge (e.g., Gemini-2.5-Pro), there is a circularity: the training signal for "when to think" is determined by what a strong closed-source model thinks is complex, and the student model learns to mimic this judgment rather than developing its own genuinely grounded complexity assessment. If the teacher's complexity judgments contain systematic errors — e.g., classifying certain types of visual puzzles as "simple" when they actually require careful analysis — those errors are baked into the training signal and the student model will learn to replicate them. Second, and more fundamentally, this labeling approach does not scale to domains where complexity is inherently subjective or continuous. Tasks that fall in a gray area between "simple perception" and "complex reasoning" — which constitute a large fraction of real-world multimodal queries — must be forced into a binary classification, and the training signal for such edge cases will be noisy and potentially misleading.

The paper does not report inter-annotator agreement rates, the proportion of problems where complexity classification was ambiguous, or any analysis of whether the learned mode-selection policy is robust to systematic label noise. This makes it difficult to assess how much of the Judging Reward's effectiveness depends on high-quality labeling versus the structural properties of the reward mechanism itself.

What evidence exists in the paper. The judge-without-reward baseline in Table 5 provides partial evidence about the labeling issue. When the model receives the SFT judgment training but no explicit Judging Reward during RL (the "Judge-w/o-reward" row), its trigger rates are poorly calibrated: 90.4% on MathVision (close to correct, since MathVision requires near-universal reasoning) but 47.6% on OCRBench (far too high; the optimal is 7.5%). This demonstrates that the SFT labels alone — without the RL optimization pressure from the Judging Reward — do not produce well-calibrated adaptivity. However, the paper does not analyze whether the RL optimization corrects label errors or merely amplifies whichever signal the labels provide. If the labels for OCRBench were systematically biased (e.g., labeling 20% of OCRBench problems as "requires reasoning" when they are actually simple), the model's learned 7.5% trigger rate could be either correcting label errors (good generalization) or fitting a noisy signal in a way that happens to work for OCRBench but would not generalize (brittle overfitting). The paper provides no analysis to distinguish these possibilities.

Mitigation status. The paper does not address this limitation explicitly. Section 8 (Conclusion) frames SAIL-RL as "a robust and scalable paradigm" without discussing the labeling bottleneck. The cross-architecture experiments (Table 10) demonstrate that SAIL-RL's training recipe works on different base models, but they use the same labeling pipeline and the same RL dataset construction method, so they do not test sensitivity to label quality. A potential mitigation — using the model's own evolving capability to generate complexity labels in a self-supervised or semi-supervised manner, where the Judging Reward becomes a function of whether the model's mode decision leads to correct answers rather than matching static labels — is not explored. Such an approach would eliminate the need for pre-assigned labels but would change the reward structure fundamentally, since the Judging Reward signal would depend on answer correctness, creating a circular dependency with the cascading product.

6.3 Single Benchmark Domain (STEM + General VQA) with No Evidence of Transfer to Other Reasoning Modalities

The assumption or constraint. All evaluation benchmarks in Tables 1 and 2 fall into two categories: mathematical/logical reasoning (DynaMath, LogicVista, MathVerse, MathVision, MathVista, WeMath, MMMU) and general multimodal understanding (MMBench, MME, ChartQA, AI2D, OCRBench, HallusionBench). These cover a range of visual reasoning tasks — mathematical derivation from diagrams, logical deduction, chart comprehension, diagram understanding, hallucination detection — but they are entirely within the domain of static image + text question → text answer. The paper provides no evidence about whether the dual-reward framework transfers to fundamentally different reasoning modalities: video understanding (temporal reasoning across frames), interactive or multi-turn reasoning (where the model must update its understanding based on new information), embodied reasoning (where actions have consequences that must be anticipated), code generation from visual specifications, or open-ended creative reasoning where there is no single ground-truth answer.

This is not merely a "more benchmarks needed" observation — it reflects a structural limitation of the Thinking Reward's three sub-dimensions. Logical coherence ( $d_1$ ) evaluates "structural soundness" and "deductive soundness" (Appendix C), which presumes that reasoning can be decomposed into well-defined steps with clear correctness criteria. This maps naturally onto mathematical derivation and formal logic but may not apply to narrative reasoning, analogical reasoning, or abductive inference (inferring the most plausible explanation from incomplete evidence) where the notion of a "correct step" is ill-defined. Factual grounding ( $d_2$ ) checks claims against the image, the question text, and world knowledge — but for tasks requiring counterfactual reasoning ("what would happen if..."), hypothetical scenario analysis, or reasoning about fictional worlds, the verification sources may not contain the relevant ground truth. Answer consistency ( $d_3$ ) checks whether the answer follows from the reasoning, but for open-ended generation where the "answer" is a paragraph of analysis rather than a boxed expression, consistency is a matter of degree rather than a binary property.

The consequence. A practitioner deciding whether to adopt SAIL-RL for a specific application must extrapolate from the paper's benchmark results to their domain of interest without evidence about which aspects of the framework will transfer. If the application involves mathematical or logical reasoning from images, the transfer is plausible. If it involves video understanding, interactive dialogue, code generation, or open-ended analysis, there is essentially no evidence either way. The risk is that the framework's components — particularly the thinking sub-rewards and the LLM judge prompts — encode assumptions about task structure that are valid for STEM-style problems but break down in other domains, and a practitioner would discover this only after investing in the full training pipeline.

A more subtle consequence: even within the paper's domain, the type of reasoning evaluated is predominantly deductive (deriving conclusions from premises using formal rules). Inductive reasoning (generalizing from examples), analogical reasoning (mapping structures between domains), and causal reasoning (inferring cause-effect relationships) are either absent or present only as minor components of the evaluated benchmarks. The paper's claims about improving "reasoning quality" should be understood as specifically about formal deductive reasoning quality, not reasoning in the broader cognitive sense.

What evidence exists in the paper. None. The paper does not discuss domain generalization as a limitation. The related work section (Section 2) frames the approach within the context of multimodal mathematical reasoning (citing VisualPRM, URSA, and similar works) but does not claim applicability beyond this domain. The evaluation benchmarks are entirely drawn from the VLMEvalKit suite, which is the standard toolkit for multimodal evaluation, but which does not include video reasoning, interactive reasoning, or code generation tasks. The cross-architecture experiments (Table 10) test different base models but on the same three benchmarks (MathVision, LogicVista, MMMU), providing no evidence of generalization to new task types.

Mitigation status. The paper does not address this limitation beyond framing the work as targeting "multimodal reasoning" (which in practice means the VLMEvalKit reasoning subset). The conclusion states that SAIL-RL is "a robust and scalable paradigm for training the next generation of reliable, meta-cognitive MLLMs," which implies broader applicability than the experiments support. Future work on extending the framework to other reasoning modalities would need to address at minimum: (1) whether the three thinking sub-rewards are sufficient or need to be modified for non-deductive reasoning; (2) whether the judge-think-answer format transfers to tasks with non-boxed outputs; (3) whether the Judging Reward's binary complexity labels are meaningful for tasks with continuous difficulty spectra.

6.4 The Thinking Reward's LLM Judge Evaluation Creates a Potential Circularity Between Training Signal and Capability

The assumption or constraint. The Thinking Reward is computed by an LLM judge (Gemini-2.5-Pro in the main experiments) that evaluates the model's generated reasoning across three dimensions using structured prompts (detailed in Appendix C). The paper relies on this judge to provide training signals about reasoning quality — signals that are then used to update the model's parameters via RL. This creates a potential circularity: if the judge has systematic biases in how it evaluates reasoning, the model will be trained to produce reasoning that looks good to the judge rather than reasoning that is genuinely superior by some ground-truth standard. The judge's evaluations are treated as ground truth for the purpose of computing $d_1, d_2, d_3$ , but no independent validation of the judge's accuracy is provided.

The specific nature of this circularity depends on the judge model's failure modes. If the judge tends to approve reasoning that uses certain linguistic patterns (e.g., explicit step labeling, formal mathematical notation, hedging language) regardless of actual correctness, the model will learn to produce reasoning with those patterns — a form of reward hacking where the model optimizes for judge preferences rather than reasoning quality. If the judge is lenient on certain types of factual errors (e.g., plausible-sounding but incorrect mathematical claims that the judge cannot verify), the model will learn that those errors are not penalized. If the judge's logical coherence evaluation is superficial — checking for the presence of reasoning structure rather than actually verifying the logic — the model will learn to produce well-structured but potentially invalid reasoning.

The consequence. The paper's central claim — that SAIL-RL improves reasoning quality, not just answer accuracy — rests on the validity of the judge's evaluations. If the judge's scores are systematically inflated or biased, the reported improvements in logic score, hallucination score, and consistency score (Figure 4) may reflect improved judge alignment rather than improved reasoning. This is not merely a hypothetical concern: the LLM-as-judge literature has documented systematic biases including position bias (favoring responses in certain positions), verbosity bias (favoring longer responses), and self-enhancement bias (favoring responses that match the judge's own generation style). The paper's judge prompts are carefully designed to mitigate these biases (e.g., by specifying binary criteria with clear pass/fail conditions), but without human validation of judge accuracy, the effectiveness of these mitigations is unknown.

A secondary consequence: even if the judge is reasonably accurate on average, its errors are not uniform across problem types. If the judge is less accurate at evaluating factual grounding for problems requiring specialized knowledge (e.g., advanced physics or chemistry), the $d_2$ signal will be noisier for those problems, and the model will receive weaker supervision on factual groundedness in exactly the domains where grounding is most important. The paper's benchmark results (Tables 1-2) aggregate across problem types, potentially masking domain-specific weaknesses in reasoning quality improvement.

What evidence exists in the paper. The reward model robustness experiment (Table 9) provides partial evidence. When the judge model is varied across Gemini-2.5-Pro, GPT-5, and Qwen2.5-VL-32B, the final model performance varies by approximately 1.3 points on average across MathVision, LogicVista, and MMMU. This suggests that the training framework is not catastrophically sensitive to the specific judge model, which provides some reassurance against the most extreme form of circularity (where only one specific judge's biases are learned). However, all three judge models are strong LLMs likely sharing similar biases about what constitutes good reasoning — they are all trained on similar corpora, use similar architectural paradigms, and were developed by organizations with overlapping research cultures. The fact that three similar judges produce similar reward signals does not validate those signals against an independent ground truth. The paper does not report human evaluation of reasoning quality for the trained models, nor does it report agreement rates between the LLM judge and human annotators on a sample of reasoning traces.

The paper's qualitative examples (Appendix B, Figures 6-7) provide illustrative cases where SAIL-RL's reasoning appears genuinely better than the baselines. In Figure 6, the answer-only baseline produces a reasoning chain with a logical error ("Since −2 ≥ 2, we have found the smallest positive integer n") while SAIL-RL produces a correct derivation recognizing the alternating pattern and computing the correct threshold. This is a compelling example but represents a single curated case — it does not constitute systematic validation of judge accuracy.

Mitigation status. The paper partially addresses this through its design choices. The use of binary rather than continuous rewards (Table 7) reduces the judge's burden from fine-grained scoring to binary decisions, which the paper argues are more reproducible and less susceptible to calibration drift. The detailed judge prompts in Appendix C specify concrete criteria for each sub-reward, reducing ambiguity. However, these mitigations improve reliability (consistency of judge decisions) without addressing validity (whether the judge's decisions correspond to genuine reasoning quality). The paper does not discuss this distinction, nor does it propose validation procedures such as human annotation of reasoning quality, comparison against formal verification (for problems where reasoning can be mechanically checked), or adversarial testing of the judge's robustness to stylistic variations that preserve semantic content.

6.5 The Removal of KL Divergence Penalty Is Neither Ablated Nor Theoretically Justified, Creating Unknown Stability Risks

The assumption or constraint. The paper removes the standard KL divergence penalty between the RL policy and the reference (SFT) model, stating: "To encourage exploration and stabilize training, we remove the standard KL divergence and dynamically adjust the clipping value ε within the range of [0.20, 0.28]" (Appendix A.2). This is a significant departure from standard RLHF practice (where the KL penalty prevents the policy from diverging too far from the human-aligned SFT model) and from the DAPO algorithm's default configuration. The paper justifies this as encouraging exploration beyond the SFT model's suboptimal distribution, and compensates with dynamic clipping that bounds how much the policy can change in a single update.

The consequence. The removal of the KL penalty creates two risks that the paper does not quantify or discuss. First, training instability: without the KL penalty to prevent large policy changes, the model may drift far from the SFT initialization over the course of 3 epochs × 70K samples × 5 rollouts = 1,050,000 reward-evaluated updates. The dynamic clipping mechanism (range [0.20, 0.28]) limits per-update changes, but the cumulative effect of many small unconstrained updates could still produce a policy that is substantially different from the SFT model. This is not inherently undesirable — the goal is to improve upon the SFT model — but without the KL penalty as a diagnostic, there is no signal about how far the policy has drifted and whether it has entered a region of parameter space where the model's fundamental language capabilities, safety properties, or instruction-following behavior have degraded. In standard RLHF, a rising KL divergence serves as an early warning that the policy is overfitting to the reward function at the cost of general capabilities; with the KL penalty removed, this warning signal is absent.

Second, reward hacking amplification: the cascading product already provides strong incentives for joint success, but without the KL penalty, the model may discover pathological strategies that achieve high reward by exploiting judge weaknesses rather than genuinely improving reasoning. For example, if the judge's logical coherence evaluation is partially influenced by the presence of explicit step labeling (e.g., "Step 1: ..., Step 2: ..."), the model could learn to produce reasoning that is mostly step labels and minimal substantive content — a strategy that the SFT model's distribution would penalize (since SFT training data doesn't look like that) but that unconstrained RL optimization might discover. The KL penalty would prevent such distributional drift; without it, only the clipping mechanism and the natural constraints of the model architecture prevent this.

What evidence exists in the paper. None directly. The paper does not report any metrics that would reveal the consequences of KL penalty removal: the KL divergence between the final policy and the SFT model, performance on held-out tasks not in the RL training distribution (which would detect capability degradation), or the rate at which the policy changes over training (which would indicate whether dynamic clipping is sufficient to maintain stability). The training dynamics in Figure 4 show smooth improvement in all metrics, suggesting that training is stable in practice, but these metrics are reward-related — they show that the model is getting better at the tasks the reward function measures. They do not show whether performance on unrelated capabilities (fluency, factual knowledge, safety guardrails) is maintained.

The paper's cross-benchmark results (Tables 1-2) show that SAIL-RL maintains strong performance on general understanding tasks while improving reasoning, which provides some reassurance that catastrophic forgetting or capability degradation has not occurred. However, this evidence is weak because the general understanding benchmarks (MMBench, MME, ChartQA, AI2D) are similar in format to portions of the RL training data (the 20K general QA samples from LLaVA-OneVision), so maintained performance might reflect training data overlap rather than genuine stability of capabilities not represented in the RL mix.

Mitigation status. The paper does not acknowledge this as a limitation. The removal of the KL penalty is presented as a deliberate design choice ("To encourage exploration... we remove the standard KL divergence") without discussion of the trade-offs involved. The dynamic clipping mechanism is presented as a sufficient substitute for KL regularization, but no evidence is provided that dynamic clipping alone achieves the same stability guarantees that KL penalties provide. Future work could address this by: (1) ablating SAIL-RL with and without the KL penalty to measure its effect on training stability and final performance; (2) reporting the implicit KL divergence during training even if it is not used as a penalty, to provide a diagnostic of policy drift; (3) evaluating the trained model on held-out capability benchmarks to verify that removing the KL penalty does not cause degradation on tasks outside the RL training distribution. Until such evidence is available, practitioners adopting SAIL-RL should consider monitoring policy drift and evaluating on their own out-of-distribution tasks to ensure that capability preservation is maintained.

6.6 The RL Dataset's Difficulty Filtering (Pass@4 Bounds) Is Not Ablated, and Its Effect on the Learned Difficulty Threshold Is Unknown

The assumption or constraint. The RL training dataset undergoes difficulty-based filtering where problems are removed if the SFT model's pass@4 is exactly 0 or exactly 1 (Appendix A.1): "The second stage implements a difficulty-based curriculum filtering, using our SFT model's pass@4 score to retain only problems within an optimal difficulty range by removing the easiest (pass@4=1) and hardest (pass@4=0) instances." This filtering serves a clear purpose — RL requires problems where the model has a meaningful probability of both success and failure — but it introduces an unexamined dependency: the filtering thresholds are defined relative to the SFT model's capabilities, and the filtering determines which problems the model ever sees during RL training.

The consequence. The pass@4 filtering removes exactly the problems that are most informative for the Judging Reward. Problems with pass@4 = 1 (the SFT model always gets them right) are precisely the problems where the model should learn to skip reasoning — they are trivially easy when answered directly, and forced reasoning on them would introduce the "overthinking noise" that the Judging Reward is designed to prevent. By removing these from RL training, the model never receives Judging Reward signals on the easiest problems, and its learned trigger rates for such problems are determined entirely by the SFT stage's format training plus whatever generalization occurs from the moderate-difficulty problems that remain. If the SFT model's trigger behavior on pass@4 = 1 problems is suboptimal (e.g., it triggers reasoning on them too often), RL training cannot correct this because those problems are absent.

Similarly, problems with pass@4 = 0 (the SFT model never gets them right) are the problems where the model should learn to recognize its own limitations — potentially learning to say "I cannot solve this" or to engage in extra-deep reasoning. By removing these, the model never receives Judging Reward signals about how to handle problems at the edge of or beyond its capability. The filtering may artificially bound the model's learned difficulty threshold, preventing it from developing appropriate responses to very hard problems (either deeper reasoning or honest acknowledgment of uncertainty).

A secondary consequence is that the filtering threshold is defined relative to the SFT model, but the optimal threshold for RL training is an empirical question. Perhaps problems with pass@4 = 1 could provide valuable training signal for the Thinking Reward (the model gets them right but might improve reasoning quality on them). Perhaps pass@4 = 0.25 problems (the model succeeds 1 in 4 attempts) provide a better training signal than pass@4 = 0.5 problems because they offer more room for improvement. The paper's choice of [0.25, 0.75] as the effective retained range (excluding 0 and 1, retaining everything in between) is a reasonable default but is not justified through experimentation.

What evidence exists in the paper. None directly. The paper does not ablate the filtering thresholds (e.g., comparing pass@4 thresholds of 0/1 vs. 0.2/0.8 vs. no filtering), does not report what fraction of the initial STEM problem pool was removed by each threshold, and does not analyze whether the retained problems differ systematically from the removed ones in ways that might bias the model's learned behavior (e.g., if pass@4=1 problems are predominantly simple perception tasks and pass@4=0 problems are predominantly tasks requiring specialized knowledge, removing them skews the training distribution toward medium-complexity deduction tasks). The filtering is described as an implementation detail rather than as a consequential design choice.

The evaluation results provide indirect evidence that the filtering is not catastrophically harmful — SAIL-RL achieves strong performance across benchmarks that include problems spanning the full difficulty range. On OCRBench, SAIL-RL triggers reasoning on only 7.5% of problems (Table 5), which includes both problems that were in the RL training set and problems that were not (since OCRBench is an evaluation benchmark, not the training data). This suggests the model has generalized its mode-selection policy to problems not seen during RL training, including potentially very easy ones that would have been filtered out. However, this generalization is not guaranteed — it depends on the similarity between the filtered-out problems and the retained problems, which is not characterized.

Mitigation status. The paper does not address this as a limitation. A potential mitigation — using the SFT model's pass@4 not as a binary filtering criterion but as a continuous weight in the RL objective (up-weighting problems where the model has intermediate success probability) — would retain all problems while still focusing training on the most informative ones, but this approach is not explored. Alternatively, a multi-stage curriculum where the filtering thresholds are progressively widened as training proceeds — starting with a narrow difficulty band for stable early learning, then expanding to include easier and harder problems — would address the concern about missing edge-case training, but the paper's single-stage training with fixed filtering does not implement this.

7. Implications and Future Directions

How This Work Changes the Landscape

SAIL-RL's primary conceptual contribution is reframing the RL training signal for MLLM reasoning from a single scalar—answer correctness—into a structured evaluation of the entire reasoning episode. This shift matters because it converts reasoning quality from an emergent property that hopefully co-occurs with correct answers into a directly supervised objective that the optimization process actively maintains. The diagnostic evidence for why this reframing is necessary rather than merely helpful comes from Figure 4: under outcome-only supervision, the model's answer accuracy improves while its consistency score—measuring whether the reasoning trace actually supports the final answer—degrades. This "reasoning collapse" phenomenon is not a hypothetical concern but an empirically observed dynamic that occurs during standard RL training, and SAIL-RL's Thinking Reward directly prevents it by making answer reward contingent on reasoning quality through the cascading product.

This finding should change how practitioners think about RL for reasoning models. The dominant paradigm, inherited from DeepSeek-R1 and similar works, operates on the implicit assumption that optimizing answer accuracy will naturally pull reasoning quality along with it. SAIL-RL demonstrates that this assumption is false in a specific and consequential way: the policy gradient finds paths to higher answer reward that do not require maintaining logical fidelity between reasoning and answer, and these paths are preferred because they are easier to discover than genuine reasoning improvement. The practical implication is that process-level supervision should be considered a necessary component of RL training for reasoning models, not an optional enhancement. Any system that relies on MLLM-generated reasoning traces for downstream purposes—explainability, verification, human oversight, self-improvement—is vulnerable to silent degradation where the model appears to be improving (accuracy rises) while actually becoming less reliable (reasoning becomes decorrelated from answers). SAIL-RL provides both the diagnostic tool (monitoring consistency scores during training) and the corrective mechanism (the cascading product) to prevent this.

The second landscape shift concerns the efficiency-effectiveness relationship in reasoning models. Prior work treated the trade-off as inherent: you can have accurate reasoning or you can have fast inference, and you must choose. SAIL-RL's Judging Reward results (Tables 4-5) demonstrate that this is a false dichotomy created by uniform reasoning policies. On OCRBench, SAIL-RL achieves 91.3% accuracy at 1.2× token cost, while an always-thinking variant achieves only 88.7% at 4.7× cost. In other words, adaptive reasoning simultaneously improves both accuracy and efficiency—a Pareto improvement over both extremes. This finding should redirect the conversation around System 1/System 2 architectures: the goal is not to build a router that minimizes cost subject to an accuracy constraint, but to train the model end-to-end so that the mode decision, reasoning quality, and answer accuracy are jointly optimized within a single objective. The cascading product $R_{\text{judge}} \cdot R_{\text{think}} \cdot R_{\text{answer}}$ provides a template for this joint optimization that prior routing-based approaches lack.

The paper also resolves a tension in the literature between works finding that self-correction and reasoning improve MLLM performance and works finding that forced reasoning introduces hallucinations and degrades perception. SAIL-RL's difficulty-dependent trigger rate analysis (Figure 5) shows that both findings can be simultaneously true: reasoning helps on complex tasks (99-100% trigger rates on math benchmarks, with substantial accuracy gains) and hurts on simple tasks (7.5% trigger rate on OCRBench, where always-thinking degrades accuracy by 2.6 points). The resolution is that the effectiveness of reasoning is conditional on task difficulty, and a model that does not adapt its reasoning depth to task demands will experience domain-dependent benefits and harms. This provides a unified explanation for contradictory prior results and establishes that adaptive reasoning is not merely an efficiency optimization but a correctness requirement for models deployed across diverse task distributions.

Research directions that become more attractive as a result of this work include: (1) developing process-level supervision mechanisms that do not require expensive LLM judges, since SAIL-RL demonstrates the necessity of such supervision but relies on a strong proprietary model to provide it; (2) exploring more sophisticated difficulty estimation that goes beyond binary complexity labels to continuous or multi-level difficulty assessments; (3) investigating whether the cascading product formulation generalizes to other multi-dimensional reward problems in language model training beyond reasoning quality. Directions that become less attractive include: (1) purely architectural routing solutions that separate the mode-decision mechanism from the reasoning mechanism, since SAIL-RL shows that joint optimization provides better calibration; (2) outcome-only RL approaches that scale compute without addressing reasoning quality, since Figure 4 demonstrates that such approaches can actively degrade reasoning fidelity even as accuracy improves.

Follow-Up Research This Work Enables

Self-supervised difficulty estimation to replace ground-truth complexity labels. The Judging Reward currently requires pre-assigned complexity labels for every training problem—a labeling bottleneck that limits scalability to new domains and introduces potential teacher bias (Section 6.2). A follow-up could replace static labels with a dynamic estimate derived from the model's own behavior: define a problem as "requiring reasoning" if the model's direct-answer accuracy is below a threshold and "not requiring reasoning" if it is above. The Judging Reward would then compare the model's mode decision against this empirically determined label, which co-evolves with the model's capability. A strong experiment would compare this self-supervised variant against the original SAIL-RL on the full benchmark suite, measuring whether the adaptive behavior (Figure 5 trigger rates) is preserved without external labels. The critical metric is whether the self-supervised variant can learn appropriate trigger rates on perception-heavy benchmarks like OCRBench without ever seeing human-annotated complexity labels for those tasks.

Human validation of LLM-judge reasoning evaluations to quantify circularity risk. The Thinking Reward relies entirely on an LLM judge (Gemini-2.5-Pro) to evaluate reasoning quality, with no independent validation against human judgments (Section 6.4). A follow-up study should collect human annotations of reasoning quality—logical coherence, factual grounding, and answer consistency—for a representative sample of SAIL-RL-generated reasoning traces and baseline traces. The key measurement is the correlation between LLM-judge scores and human scores, broken down by problem type (math vs. perception, easy vs. hard). If the correlation is high (>0.8), the circularity concern is mitigated. If it is low or systematically biased (e.g., the judge overrates reasoning that uses formal notation), then SAIL-RL's reported reasoning improvements (Figure 4) partially reflect judge alignment rather than genuine improvement, and future work should focus on better judge calibration. A negative result—finding that human evaluations do not confirm the judge's reported improvements—would substantially refine our understanding of SAIL-RL's mechanism and would motivate research into judge-free process supervision.

Extending the cascading product to multi-level difficulty decisions. The Judging Reward currently enforces a binary choice: think or don't think. Real tasks exist on a continuous difficulty spectrum, and optimal reasoning should allocate variable depth rather than binary engagement. A natural extension replaces the binary $R_{\text{judge}}$ with a multi-level reward that incentivizes the model to allocate reasoning depth proportional to estimated difficulty. Concretely: the model could output a budget estimate (e.g., "this problem requires approximately 200 tokens of reasoning") and receive reward proportional to the accuracy achieved given that budget, penalized for both under-allocation (insufficient reasoning causing errors) and over-allocation (excessive reasoning on simple tasks). The experiment would compare a multi-level SAIL-RL against the binary version on the benchmark suite, measuring both accuracy and average token usage. The hypothesis is that multi-level allocation would further improve efficiency on the "medium" benchmarks (MMBench, AI2D) where Figure 5 shows intermediate trigger rates of 75-93%, suggesting these tasks sometimes benefit from reasoning but don't always need the full reasoning budget.

Combining SAIL-RL with test-time compute scaling to study joint allocation of training and inference compute. SAIL-RL optimizes the training objective to teach adaptive reasoning; the model then deploys with fixed behavior. A complementary line of work—exemplified by the test-time compute scaling literature—optimizes how inference compute is allocated per-query after training. Combining these paradigms would mean: (1) train the model with SAIL-RL to produce high-quality reasoning and calibrated mode decisions; (2) at inference time, allow the model to also vary its reasoning budget dynamically based on its own uncertainty estimates. The experiment would measure whether a SAIL-RL-trained model can further improve accuracy by allocating additional test-time compute (e.g., generating multiple reasoning chains and selecting the best) on problems where its initial mode decision was "think" but its internal confidence is low. The key question is whether SAIL-RL's training—which already teaches the model to calibrate reasoning effort—produces better uncertainty estimates that enable more efficient test-time compute allocation than models trained with outcome-only supervision. This connects the paper's training-time adaptivity to the broader inference-time scaling literature and could reveal whether training for adaptive reasoning and test-time compute scaling are complementary or partially redundant.

Stress-testing the Thinking Reward on adversarial reasoning challenges where the judge is likely to fail. The Thinking Reward's three sub-dimensions (logical coherence, factual grounding, answer consistency) are evaluated by an LLM judge that may have systematic weaknesses. A diagnostic follow-up would construct a benchmark of reasoning problems specifically designed to expose judge failures: problems where the reasoning appears logically sound but contains subtle fallacies (testing $d_1$ ), problems where plausible-sounding factual claims are actually false (testing $d_2$ ), and problems where the reasoning is correct but the answer is intentionally mismatched in a way that looks superficially consistent (testing $d_3$ ). Training SAIL-RL on this adversarial benchmark and measuring whether the resulting model has learned genuine reasoning or judge-pleasing patterns would provide a strong test of the framework's robustness. If the model achieves high judge scores but low ground-truth accuracy on the adversarial problems, it indicates that SAIL-RL's improvements are partly attributable to exploiting judge weaknesses—a finding that would motivate research into more robust process supervision mechanisms.

Ablating the KL penalty removal to quantify stability and capability preservation trade-offs. The paper removes the standard KL divergence penalty to encourage exploration (Section 6.5) but provides no ablation or stability analysis. A direct follow-up experiment would train SAIL-RL variants with KL penalties at different strengths (0.0, 0.01, 0.05, 0.1) and measure: (1) final benchmark performance on the full suite; (2) training stability metrics (policy update magnitudes, reward variance over time); (3) performance on held-out capabilities not in the RL training distribution (e.g., pure text reasoning benchmarks, factual knowledge probes, safety evaluations) to detect whether KL penalty removal causes capability drift. The hypothesis is that some KL regularization is beneficial for preserving general capabilities, and the optimal strength is non-zero—but the paper's removal of the penalty means we don't know whether SAIL-RL's strong reasoning results come at the cost of degraded performance on capabilities not measured by the evaluation suite. If a modest KL penalty (e.g., 0.01-0.05) maintains or improves reasoning performance while preserving out-of-distribution capabilities, it would be the safer recommendation for practitioners.

Practical Applications and Downstream Use Cases

Cost-efficient deployment of MLLMs for mixed-difficulty query streams. In production settings where MLLMs serve queries spanning a wide difficulty range—from simple product recognition in e-commerce to complex document analysis in legal tech—SAIL-RL's adaptive reasoning directly translates to inference cost reduction without accuracy loss. The OCRBench result in Table 5 quantifies this: SAIL-RL uses 1.2× the tokens of direct answering while achieving 91.3% accuracy, versus 4.7× tokens and 88.7% accuracy for an always-thinking model. For a deployment processing 10 million queries per day with a difficulty distribution similar to the general understanding benchmarks (where Figure 5 shows average trigger rates around 50-70%), replacing a uniform-reasoning model with SAIL-RL would reduce token generation by approximately 40-60% while maintaining or improving accuracy on perception-heavy queries. At current API pricing for models at the 7-8B scale, this translates to tens of thousands of dollars in monthly savings for high-volume deployments.

Training data generation for self-improvement pipelines with verified reasoning quality. When MLLMs are used to generate training data for themselves or smaller models—a common paradigm in iterative self-improvement—the quality of the generated reasoning traces directly determines the quality of the resulting student models. SAIL-RL's Thinking Reward ensures that generated reasoning is logically coherent, factually grounded, and consistent with answers, making it substantially more valuable as training data than reasoning traces from outcome-only-trained models (which Figure 4 shows suffer from consistency degradation). A practical pipeline would: (1) train a teacher model with SAIL-RL; (2) use the teacher to generate judge-think-answer triplets on a large corpus of unlabeled multimodal data; (3) filter triplets where the Thinking Reward (computed by the same judge used during training) is below a threshold; (4) fine-tune a smaller student model on the filtered data. The key advantage over using outcome-only-trained teachers is that the generated reasoning traces are more likely to be actually instructive—the student learns from logically sound derivations rather than from "lucky success" traces that happen to reach correct answers through flawed reasoning.

Hallucination reduction in high-stakes multimodal QA systems. In domains where factual accuracy is critical—medical image analysis, legal document review, scientific figure interpretation—the Factual Grounding sub-reward ( $d_2$ ) provides a mechanism for explicitly training models to avoid fabricating claims unsupported by the visual or textual evidence. The paper's HallusionBench results (Table 4: SAIL-RL achieves 61.5 vs. 58.3 for always-thinking, a +3.2 point improvement) suggest that the combination of adaptive reasoning (skipping reasoning when it would introduce hallucination risk) and factual grounding supervision (penalizing unsupported claims in reasoning traces) produces measurable reductions in hallucination. A practical deployment would fine-tune a domain-specific SAIL-RL variant on in-domain data (e.g., radiology images with structured reports), using the Factual Grounding sub-reward to penalize reasoning steps that contradict the image or introduce unsupported medical claims. The resulting model would be both more accurate (due to improved reasoning) and more trustworthy (because its reasoning traces can be audited for factual grounding) than a model trained with outcome-only supervision. The 3.2-point HallusionBench improvement, while modest in absolute terms, represents a meaningful reduction in a failure mode that is disproportionately costly in high-stakes applications.

When to Prefer This Method

The paper explicitly positions SAIL-RL against two alternatives: outcome-only RL (represented by the answer-only baseline in Tables 3-4 and Figure 4) and uniform reasoning (represented by the always-thinking and never-thinking baselines in Tables 4-5). The trade-offs are clearly characterized by the paper's experiments:

Prefer SAIL-RL over outcome-only RL when the deployment requires both answer accuracy and verifiable reasoning quality, or when the model's reasoning traces will be used for downstream purposes (explainability, human verification, self-improvement data generation). The evidence is Figure 4: outcome-only training achieves similar accuracy improvements but at the cost of degrading reasoning consistency, creating a silent reliability failure. SAIL-RL maintains or improves all reasoning quality dimensions while achieving higher accuracy (Table 3: +1.6 to +3.2 points over answer-only on STEM benchmarks).
Prefer SAIL-RL over uniform always-thinking when the query distribution includes a substantial fraction of perception-heavy or simple tasks where reasoning is unnecessary or harmful. The evidence is Tables 4-5: always-thinking degrades accuracy on every general benchmark tested (by 1.6-3.2 points) while consuming 4-5× more tokens than necessary. SAIL-RL's adaptive trigger rates (Figure 5) automatically allocate reasoning only to tasks that benefit from it, achieving the highest accuracy on both reasoning and perception benchmarks while using 1.2× tokens on simple tasks.
Prefer uniform never-thinking only when the query distribution consists almost entirely of simple perceptual tasks (OCR, basic recognition, simple counting) and inference latency is the primary constraint. Table 5 shows never-thinking achieves 90.5 on OCRBench at 1.0× tokens, marginally below SAIL-RL's 91.3 at 1.2×. If the 0.2× token overhead is unacceptable in a latency-critical deployment and the 0.8-point accuracy gap is tolerable, never-thinking is the appropriate choice. The paper's results suggest this scenario is rare for general-purpose deployments—SAIL-RL's adaptive approach achieves the highest accuracy on every benchmark tested—but it may apply in narrow-domain, high-throughput settings.

🎯 Pitch

1. Executive Summary

2. Context and Motivation

The Core Problem: Reasoning Quality and Efficiency Are in Tension

Why This Problem Matters

Where Existing Approaches Fall Short

How This Paper Positions Itself

3. Technical Approach

3.1 Reader Orientation

3.2 Big-Picture Architecture (Diagram in Words)

3.3 Roadmap for the Deep Dive

3.4 Detailed, Sentence-Based Technical Breakdown

3.4.1 LongCoT SFT: Teaching the Judge-Think-Answer Format

3.4.2 The Thinking Reward: Evaluating How to Think

3.4.3 The Judging Reward: Learning When to Think

3.4.4 The Cascading Reward System: Enforcing Joint Success

3.4.5 RL Training Stage: Optimizing with DAPO and the Dual-Reward System

3.4.6 Design Rationale: Why Specific Choices Over Alternatives

4. Key Insights and Innovations

Innovation 1: Reasoning Quality as a Trainable Objective, Not a Byproduct of Accuracy

Innovation 2: The Cascading Product as a Logical AND Gate for Reward Hacking Prevention

Innovation 3: Making Metacognition a First-Class Training Objective

Innovation 4: Empirical Demonstration That Uniform Reasoning Can Be Actively Harmful

5. Experimental Analysis

Evaluation Methodology

Main Quantitative Results

Reasoning Benchmark Performance (Table 1)

General Understanding Benchmark Performance (Table 2)

Thinking Reward Ablation: Reasoning Quality Improvement (Table 3, Figure 4)

Judging Reward Ablation: Adaptive Efficiency (Tables 4-5, Figure 5)

Reward Mechanism Design Ablations (Tables 6-8)

Cross-Architecture Generalization (Table 10)

Reward Model Robustness (Table 9)

Ablation Studies and Robustness Checks

Critical Assessment

6. Limitations and Trade-offs

6.1 The Cascading Product's Zero-Tolerance Penalty Creates a Cold-Start Problem with No Partial Credit

6.2 Difficulty Estimation for Judging Reward Relies on Undisclosed Ground-Truth Labels, Limiting Scalability

6.3 Single Benchmark Domain (STEM + General VQA) with No Evidence of Transfer to Other Reasoning Modalities

6.4 The Thinking Reward's LLM Judge Evaluation Creates a Potential Circularity Between Training Signal and Capability

6.5 The Removal of KL Divergence Penalty Is Neither Ablated Nor Theoretically Justified, Creating Unknown Stability Risks

6.6 The RL Dataset's Difficulty Filtering (Pass@4 Bounds) Is Not Ablated, and Its Effect on the Learned Difficulty Threshold Is Unknown

7. Implications and Future Directions

How This Work Changes the Landscape

Follow-Up Research This Work Enables

Practical Applications and Downstream Use Cases

When to Prefer This Method

Adjacent in domain

DeepSeekMath-V2: Towards Self-Verifiable Mathematical Reasoning

TimeSearch-R: Adaptive Temporal Search for Long-Form Video Understanding via Self-Verification Reinforcement Learning