LLaVA-o1: Let Vision Language Models Reason Step-by-Step

ArXiv: 2411.10440

🎯 Pitch

A 11B vision-language model, fine-tuned on just 100K examples that structure reasoning into summary–caption–reasoning–conclusion stages, defeats GPT‑4o‑mini and Gemini‑1.5‑pro on six multimodal reasoning benchmarks. It does this by generating multiple candidates per reasoning stage at test time and backtracking when they all fail a reward threshold—enabling self‑correction without any external verifier and showing continued performance gains even after 10,000 seconds of compute.

---

1. Executive Summary

This paper introduces LLaVA-CoT, a vision-language model fine-tuned on the LLaVA-CoT-100k dataset that performs autonomous multistage reasoning through four structured stages—summary, caption, reasoning, and conclusion—each demarcated with specialized tags to enforce systematic, step-by-step answer generation. The authors further propose stage-wise retracing search (SWIRES), a test-time scaling method that generates multiple candidates per reasoning stage and retraces to earlier stages for regeneration when all candidates fall below a reward threshold, enabling self-reflection and error correction during inference. Built on Llama-3.2-11B-Vision-Instruct and evaluated across six multimodal reasoning benchmarks, LLaVA-CoT with SWIRES achieves a 9.4% average improvement over its base model and outperforms larger open-source and closed-source models including Gemini-1.5-pro and GPT-4o-mini, while demonstrating continued scaling behavior on MMStar beyond 10,000 seconds where best-of-N search plateaus, establishing that structured reasoning stages combined with stage-level retracing enable more effective test-time compute scaling than coarse full-response search—though the gains concentrate primarily in reasoning-intensive skill areas rather than perceptual tasks.

2. Context and Motivation

The Core Problem: Vision Language Models Cannot Reason Systematically

The fundamental problem this paper tackles is that current Vision-Language Models (VLMs), despite their strong performance on many visual understanding tasks, lack the ability to conduct systematic, structured reasoning. When faced with complex visual questions requiring multi-step logical deduction—mathematical word problems with diagrams, scientific reasoning about visual phenomena, or compositional counting tasks—existing VLMs typically resort to one of two failure modes (Section 1, Figure 2, Appendix A):

1. Rushing to judgment: The model generates a direct answer without properly organizing the problem context or available visual information, often producing a premature conclusion and then attempting to justify it retroactively. Since LLMs generate token-by-token, once an erroneous conclusion appears, the model commits to a flawed reasoning trajectory from which it cannot recover.

2. Losing the thread: Even when VLMs are prompted with Chain-of-Thought (CoT), they frequently produce reasoning chains containing errors, hallucinations, or logical inconsistencies. The reasoning is present but insufficiently structured—the model does not clearly delineate what it knows from the image, what the problem requires, and how each reasoning step follows from the previous.

The authors frame this as a dual deficiency: existing VLMs are neither systematic (they don't decompose problems into distinct reasoning stages) nor structured (they don't explicitly identify which reasoning stage they're in and what task each stage addresses). These are distinct issues—a model could produce multi-step reasoning (systematic) without clear stage boundaries (unstructured), or it could use structured formatting without genuine logical progression between stages.

This problem matters for two interconnected reasons. On the practical side, VLMs are increasingly deployed as general-purpose assistants where users expect reliable reasoning across diverse visual contexts—from helping students solve geometry problems to interpreting scientific diagrams to answering compositional questions about photographs. A model that produces confident-sounding but logically flawed answers undermines trust and limits real-world deployability. On the theoretical side, visual reasoning represents a distinct challenge from text-only reasoning because the model must integrate information from two modalities with fundamentally different structures: continuous visual features and discrete linguistic concepts. Understanding how to make this integration systematic is a core open question in multimodal AI.

Why This Gap Persists: The Direct-Response Paradigm and Its Limitations

Early open-source VLMs, as the authors note (Section 1), predominantly employ a direct prediction approach: given an image and a question, generate a brief answer immediately. This paradigm—exemplified by models like LLaVA [35] and its variants—is architecturally simple and works adequately for straightforward visual questions ("What color is the car?" "Is there a dog in this image?"). However, it fundamentally fails on tasks requiring logical reasoning because there is no intermediate state where the model can organize information, test hypotheses, or verify its own reasoning before committing to an answer.

The introduction of Chain-of-Thought prompting [56] partially addressed this by encouraging step-by-step reasoning. However, the authors observe that even with CoT prompting, VLMs still exhibit significant reasoning failures (Section 1, Appendix A). They identify a root cause that prior work has not adequately addressed: the generated reasoning chains lack structural discipline. When a VLM produces a stream-of-consciousness reasoning sequence, there is no guarantee that it will:

• First summarize what the problem asks and what information is needed
• Then extract relevant visual details before attempting logical operations on them
• Then conduct step-by-step inference with clear logical dependencies
• Finally produce a conclusion that synthesizes rather than contradicts the reasoning

Without this structure, the model's reasoning is fragile. A hallucinated visual detail early in the chain cannot be caught because the model never explicitly separates "what I see" from "what I infer." A logical error midway through cannot be localized because the reasoning is a continuous text block rather than a sequence of labeled stages. The authors' diagnostic insight—supported by their ablation showing that removing structured tags causes significant performance drops (Table 2, row "LLaVA-CoT (w/o Structured Tags)")—is that the format of reasoning is not cosmetic; it causally contributes to reasoning quality.

Conflicting Signals in Prior Work

The paper positions itself at the intersection of several active research threads, each with notable gaps:

Structured prompting for VLMs. Prior works like Prism [45] manually decompose the VLM's process into separate perception and reasoning stages via prompting. MSG [8] pioneered forced chain-of-thought through structured prompts. However, these approaches rely on external prompt engineering—the model doesn't inherently understand or internalize the stage structure. When the prompt is removed, the structured behavior disappears. The LLaVA-CoT insight is that by training the model with stage-tagged data, the structured reasoning becomes an intrinsic capability that the model autonomously maintains without any test-time prompting intervention (Section 3.1.1: "each stage is initiated at the model's discretion, without any additional prompting").

Distillation of reasoning into smaller models. Works like Distilling CoT [22] and Visual Program Distillation [24] use larger teacher models to generate reasoning traces, then fine-tune smaller models on those traces. The LLaVA-CoT-100k dataset follows this general paradigm (using GPT-4o as the teacher), but the authors emphasize that they are not simply distilling GPT-4o's capabilities. In Appendix F (Table 7), they show that prompting GPT-4o with their structured CoT format yields improvements over GPT-4o's standard CoT (67.6 vs. 66.0 on MMStar-R), suggesting the format itself provides benefits beyond the teacher model's baseline reasoning. When they instead perform multi-task training using GPT-4o's responses without the CoT structure, performance is significantly worse (57.7 vs. 63.1), confirming that the structure matters more than denser supervision alone.

Test-time scaling methods are coarse and inflexible. The paper identifies specific limitations in existing test-time scaling approaches (Section 2.3):

• Best-of-N search [4, 55] generates N complete responses and selects the best. The problem is granularity: errors early in a response propagate through the remainder, and evaluating entire responses requires the verifier to assess very different reasoning paths holistically—a challenging judgment call.

• Majority voting [25] works for problems with standard answers but fails on open-ended questions where multiple phrasings may all be correct.

• Beam search [19, 52] generates multiple candidates and selects the best at intermediate points. However, existing beam search for language models typically triggers search at fixed token or sentence intervals [67], which has no semantic correspondence to the reasoning process. A search step after the 50th token may cut mid-thought, while a search step after every sentence may be too frequent for simple sections and too sparse for complex sections. The authors' key observation is that the granularity of search should align with reasoning stages, not arbitrary text boundaries.

None of these methods incorporate error correction through backtracking. If a best-of-N response is wrong at step 2 of 5, the remaining steps are wasted. If beam search selects a suboptimal candidate at an early stage, later stages cannot recover because they build on the flawed partial output. The authors argue that what's missing is a mechanism for recognizing when a reasoning stage has produced poor outputs and retreating to an earlier stage for regeneration—a form of test-time self-reflection that mirrors how humans reconsider earlier assumptions when their current reasoning isn't working.

How This Paper Positions Itself

LLaVA-CoT makes two primary moves relative to prior work:

Move 1: From external prompting to intrinsic structured reasoning. Rather than prompting the model to follow a structure at test time (which prior work shows is fragile and requires careful prompt design), LLaVA-CoT embeds the structure into the model through supervised fine-tuning on data that explicitly delineates reasoning stages with dedicated tags. The model learns not just what to reason about but how to organize its reasoning process. This transforms structured reasoning from a prompting trick into a learned capability. The analogy is instructive: prompting-based structure is like giving a student instructions before each exam question; training-based structure is like teaching the student a problem-solving methodology they can apply independently.

Move 2: From coarse to stage-aligned test-time scaling with self-correction. Rather than applying search at arbitrary intervals or only on complete responses, LLaVA-CoT's SWIRES method performs search at stage boundaries (after summary, caption, reasoning, conclusion), ensuring that each search decision corresponds to a semantically meaningful unit. The retracing mechanism adds a meta-cognitive layer: when all candidates at a stage score below a reward threshold, the system infers that the preceding stage likely produced an error and retraces to regenerate it. This is analogous to what the test-time scaling literature [51] calls "sequential revision" but applied at the granularity of reasoning stages within a single forward pass, rather than to complete solution attempts.

The authors explicitly connect to the broader test-time compute scaling framework (Section 2.3, citing Snell et al. [51]), positioning SWIRES as a method that makes test-time compute more effective at the same budget by allocating it to stages where correction is most needed, rather than uniformly increasing sample count. The empirical claim is that this targeted allocation yields stronger scaling behavior: while best-of-N and stage-wise beam search plateau around 10,000 seconds of compute, SWIRES continues to improve (Figure 5).

What this work does NOT do (boundaries). It's important to note what the paper explicitly leaves unaddressed. First, the structured reasoning stages are fixed and predefined—the model cannot dynamically decide how many stages a problem needs or what kind of stages are appropriate. Second, the SWIRES method is applied only to the caption and reasoning stages (the summary stage typically produces high-quality outputs, as noted in Section 3.2.2), meaning the retracing mechanism does not fully explore the space of possible error recovery patterns. Third, the method is evaluated exclusively on visual reasoning benchmarks with verifiable answers—extension to open-ended dialog or creative generation is not explored. The authors frame these as directions for future work rather than limitations of the current approach.

3. Technical Approach

3.1 Reader Orientation

LLaVA-CoT is a vision-language model fine-tuned to reason through visual questions in four explicit, tagged stages—summary, caption, reasoning, and conclusion—rather than producing a single undifferentiated response. The system solves the problem that VLMs typically rush to judgment or produce logically tangled reasoning chains by embedding a structured problem-solving methodology directly into the model through supervised fine-tuning on staged reasoning data, then amplifies its accuracy at test time through a stage-aligned search procedure that can backtrack and regenerate earlier stages when the current reasoning quality is poor.

3.2 Big-Picture Architecture (Diagram in Words)

The LLaVA-CoT system has three major components, deployed in two phases:

Training Phase:

1. LLaVA-CoT-100k Dataset Generator — Uses GPT-4o, prompted with a structured four-stage format, to produce reasoning annotations for 99k image-question pairs drawn from ten diverse VQA datasets. A verification step filters out refusals and answers inconsistent with the original dataset's ground truth.
2. Supervised Fine-Tuning Pipeline — Takes the base model (Llama-3.2-11B-Vision-Instruct) and performs full-parameter fine-tuning on the LLaVA-CoT-100k dataset for 3 epochs, teaching the model to autonomously generate responses with <SUMMARY>, <CAPTION>, <REASONING>, and <CONCLUSION> tags without any external prompting.

Inference Phase:

3. Stage-wise Retracing Search (SWIRES) — At test time, the fine-tuned model generates multiple candidate responses at each reasoning stage. A reward model (InternLM-XComposer2.5-Reward) scores each candidate. If at least one candidate exceeds a statistical threshold, the top N proceed to the next stage. If all candidates fall below the threshold, the system retraces to the previous stage, regenerates its outputs, and re-generates candidates for the current stage—up to a maximum of C retracing attempts. This continues through all stages, with the final answer selected as the highest-scoring conclusion across all generated candidates.

The information flow during inference: question + image → model generates summary (1 candidate) → model generates M captions → reward model scores captions → if threshold met, select top N captions; if not, regenerate summary and repeat → for each selected caption, model generates M/N reasonings → reward model scores reasonings → if threshold met, select top N reasonings; if not, retrace to caption regeneration → for each selected reasoning, model generates 1 conclusion → reward model scores all conclusions → highest-scoring conclusion returned as final answer.

3.3 Roadmap for the Deep Dive

• First, the four reasoning stages and their tag structure, because this defines what "structured reasoning" means concretely—the semantic contract each stage enforces and how the model transitions between them autonomously.
• Second, the LLaVA-CoT-100k dataset construction pipeline, because the training data is what teaches the model this structured behavior—the prompting strategy for GPT-4o, the filtering procedure, and the composition of source datasets.
• Third, the supervised fine-tuning procedure and hyperparameters, because this is the mechanism that converts the base model from an unstructured responder into a staged reasoner.
• Fourth, the stage-wise beam search algorithm, because SWIRES builds on it—understanding the baseline search method clarifies what retracing adds.
• Fifth, the full stage-wise retracing search (SWIRES) algorithm, including the reward threshold calculation, the backtracking logic, and the hyperparameter choices—this is the core test-time contribution.
• Sixth, the design choices and their justifications, synthesizing why each decision (tags, stage order, reward model choice, threshold formula) was made relative to alternatives.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a training methodology and inference algorithm paper whose core idea is that decomposing VLM reasoning into explicitly tagged stages—and aligning test-time search granularity with those stage boundaries while adding a retracing mechanism—yields more systematic reasoning and better test-time compute scaling than either prompting-based structure or coarse full-response search.

---

The Four Reasoning Stages and Their Tag Structure

LLaVA-CoT decomposes the answer generation process into four sequential stages, each with a dedicated purpose and demarcated by specialized XML-style tags that the model learns to produce autonomously during a single forward pass (Section 3.1.1). The stages are:

Summary Stage (<SUMMARY>...</SUMMARY>). This is the initial phase where the model provides "a high-level summary interpretation of the question, outlining the primary aspects of the problem it intends to address." The summary is not the answer itself but a meta-cognitive preview: what kind of problem is this, what information will I need, what approach will I take. For example, in Figure 2's first example, the summary states: "I will examine the image to identify and count all the objects, focusing on the tiny shiny balls and purple objects. Then, I will subtract these objects from the total count and determine the number of remaining objects." This serves as a planning step that constrains the subsequent reasoning by establishing the problem-solving framework before detailed work begins.

Caption Stage (<CAPTION>...</CAPTION>). Here the model provides "a concise overview of the visual elements relevant to the question, helping to understand multimodal input." Critically, this is not a generic image caption but a task-directed description that filters visual information through the lens of the question. In the engineering problem in Figure 2, the caption describes the block's dimensions, the applied force, and the displacement—all parameters needed for the shear modulus calculation—rather than irrelevant details about colors or materials. This stage explicitly separates perception (what do I see?) from inference (what follows from what I see?), preventing the model from conflating observation with deduction.

Reasoning Stage (<REASONING>...</REASONING>). Building on the summary's plan and the caption's observations, this stage "conducts structured, logical reasoning to derive a preliminary answer." This is where the actual chain-of-thought lives—mathematical derivations, logical deductions, counting operations, comparative analysis. In Figure 2's second example, the reasoning stage contains the full shear modulus formula, unit conversions, and arithmetic steps. The reasoning is informed by both the summary (which set the approach) and the caption (which provided the numbers), creating a dependency chain: summary → caption → reasoning.

Conclusion Stage (<CONCLUSION>...</CONCLUSION>). In this final stage, the model "synthesizes an answer based on the preceding reasoning." The conclusion adapts to user requirements: for multiple-choice questions, it outputs only the option letter (e.g., "B"); for open-ended questions, it provides a thorough response (e.g., the full biographical introduction in Figure 8). The authors emphasize that the conclusion is "the direct response provided to the user, while the prior three stages are internal 'hidden stages' representing LLaVA-CoT's reasoning process."

Autonomous stage transitions. A crucial design property is that "each stage is initiated at the model's discretion, without any additional prompting, and all stages are completed by the model in a single inference pass." This means there is no external controller switching between stages—the model's forward generation naturally produces the tag transitions as part of its learned behavior. The authors report they "have not observed any instances where the model fails to adhere to the designated stage format," indicating that the structured format is robustly internalized through training rather than being a fragile prompted behavior.

Why this specific stage decomposition? The four-stage structure captures what the authors identify as the essential reasoning sub-tasks: (1) understanding the problem, (2) gathering visual evidence, (3) performing logical inference on that evidence, and (4) presenting the result. The ordering matters—the ablation in Appendix F (Table 7, row "LLaVA-CoT (reorder)") shows that training with shuffled stage orders (e.g., reasoning before caption) leads to almost no improvement over the base model (58.2 vs. 56.9 average on reasoning benchmarks), confirming that "only when the four stages follow a natural reasoning process does the reasoning become effective." This is not arbitrary formatting; the stage sequence mirrors how a careful human solver would approach a visual reasoning problem.

Tags as semantic anchors. The tags (<SUMMARY>, </SUMMARY>, etc.) serve multiple functions. During training, they provide explicit supervision targets—the model learns to emit these tokens at specific reasoning boundaries. During inference, they segment the output into semantically meaningful units for the SWIRES algorithm, enabling search to operate at stage boundaries rather than arbitrary token counts. During evaluation, they make the reasoning process inspectable and auditable. The ablation in Table 2 shows that removing the structured tags from training data (row "LLaVA-CoT (w/o Structured Tags)") drops average benchmark performance from 62.4 to 60.9, with particularly large drops on MathVista (54.8 → 54.1) and HallusionBench (47.8 → 45.0)—benchmarks heavily dependent on reasoning.

---

The LLaVA-CoT-100k Dataset Construction Pipeline

Since existing VQA datasets lack the detailed staged reasoning annotations needed for training, the authors construct a new dataset by prompting GPT-4o to generate structured responses for 99k image-question pairs drawn from ten source datasets (Section 3.1.2, Table 1). The construction pipeline has three sequential phases:

Phase 1: Source dataset selection and curation. The authors select samples from ten datasets spanning two categories:

General VQA Datasets (71.9k samples total):

• ShareGPT4V [9]: 31.3k samples — multi-turn question-answering data from GPT-4V interactions, providing diverse conversational visual questions.
• ChartQA [41]: 17.2k samples — chart and graph interpretation questions, requiring extraction of numerical information from visual data representations.
• A-OKVQA [48]: 16.1k samples — questions requiring external knowledge beyond visible content, testing the model's ability to integrate world knowledge with visual observation.
• DocVQA [42]: 4.0k samples — document-based questions requiring textual comprehension from images of forms, invoices, and reports.
• PISC [31]: 1.0k samples — social relationship understanding from images of people.
• CLEVR [27]: 0.5k samples — compositional questions about object properties, spatial relationships, and counting in synthetic scenes.

Science-Targeted VQA Datasets (28.9k samples total):

• GeoQA+ [7]: 11.4k samples — geometric reasoning problems requiring understanding of shapes, angles, and spatial configurations.
• AI2D [28]: 11.4k samples — scientific diagram interpretation questions from textbook-style illustrations.
• ScienceQA [37]: 5.6k samples — multimodal science questions covering physics, biology, and chemistry concepts.
• CLEVR-Math [14]: 0.5k samples — arithmetic analysis in visual contexts, extending CLEVR with mathematical operations.

The dataset composition deliberately balances general visual understanding with science-targeted reasoning, ensuring the model learns structured reasoning across diverse visual domains. The total is described as "99k image QA pairs (each pair may include one or multiple rounds of questioning)."

Phase 2: GPT-4o generation with structured prompting. For each image-question pair, the authors prompt GPT-4o with a carefully designed instruction that enforces the four-stage format. The prompt (reproduced in full in Appendix B) specifies:

• The exact four-section structure with explicit tag formats
• The purpose of each section (summarize approach, describe relevant image content, provide step-by-step reasoning, state answer in matching format)
• The critical constraint that "the final answer in the CONCLUSION matches the standard correct answer precisely"
• Formatting instructions for multiple-choice questions ("the conclusion should only include the option without repeating what the option is")
• A reminder to include the closing </CONCLUSION> tag

The prompt is applied separately for each of the four stages (as shown in Figure 3's process flow diagram), meaning GPT-4o generates each stage's content independently with focused instructions rather than producing all four stages in a single response. The process flow in Figure 3 shows: for the summary, GPT-4o is prompted with "Briefly explain what steps you'll take…"; for the caption, "Describe the contents of the image…"; for the reasoning, "Outline a step-by-step thought process…"; and for the conclusion, "State the final answer in a clear format…". Each stage's output is wrapped in its respective tags.

Phase 3: Filtering and verification. After generation, a two-stage filtering process ensures data quality:

• Format compliance check: The authors "verify whether the data generated by GPT-4o adheres to the prescribed format and filter out any data that does not comply." This catches cases where GPT-4o fails to produce properly tagged responses or generates malformed XML.

• Semantic consistency verification: The conclusion content is extracted and compared against the original dataset's standard answer using a second GPT-4o prompt (Appendix B). This verification prompt evaluates whether the assistant's response "is not a refusal and it aligns with the standard answer in meaning." Responses that are refusals ("cannot recognize a specific person/object or refuses to answer") or "differs from the standard answer in a meaningful way" are marked invalid and filtered out. The prompt includes an explicit instruction: "Do not consider a response to be a refusal just because it includes the word 'no' or other negative terms," preventing overly aggressive filtering of legitimate negative answers.

Why GPT-4o as the teacher? The authors acknowledge that "no multimodal model currently exists that can directly produce systematic, structured reasoning," making GPT-4o the pragmatic choice for generating training data. However, they explicitly address the concern that this is mere distillation: in Appendix F (Table 7), prompting GPT-4o with the structured CoT format yields a 1.6-point improvement over GPT-4o's standard performance on MMStar-R (67.6 vs. 66.0), while prompting the base Llama model with the same structured format yields no improvement (56.9 vs. 56.9). This asymmetry shows that the structured format provides benefits beyond the teacher's baseline capabilities—GPT-4o already has strong reasoning that the format helps organize, while Llama lacks the underlying reasoning capacity that the format alone cannot create, but can learn through fine-tuning on structured examples.

---

Supervised Fine-Tuning Procedure

The LLaVA-CoT-100k dataset is used to fine-tune the base model through standard supervised fine-tuning (SFT) with full parameter updates (Section 3.1.2). The training configuration is:

Base model. The authors select Llama-3.2-11B-Vision-Instruct [43] as the starting checkpoint. This is an 11-billion parameter vision-language model with native image understanding capabilities. The authors note this model has "the lowest average score" among compared models (Figure 1, Table 5: 56.9 on reasoning benchmarks), making it a challenging base that provides a strong test of whether structured training can lift weak reasoning.

Training hyperparameters (Table 6, Appendix C). All values are reported verbatim from the paper:

The training is "conducted on a single node with 8 H100 GPUs." Full-parameter fine-tuning means all 11 billion parameters are updated, rather than using parameter-efficient methods like LoRA.

What the model learns. During training, the model's next-token prediction objective is applied to the entire structured response including the tags. This means the model learns to:

1. Recognize that after a question, it should begin with <SUMMARY>
2. Generate a planning-oriented summary appropriate to the question type
3. Close the summary with </SUMMARY> and open <CAPTION>
4. Extract and describe task-relevant visual details
5. Close the caption and transition to <REASONING>
6. Produce step-by-step logical inference connecting the caption's observations to the summary's plan
7. Close the reasoning and transition to <CONCLUSION>
8. Produce the final answer in the format appropriate to the question type
9. Close the conclusion with </CONCLUSION>

The authors report that after training, "LLaVA-CoT is capable of seamlessly transitioning between different stages without any external intervention," and they "have not observed any instances where the model fails to adhere to the designated stage format." This robustness is noteworthy because it means the tag structure is not a brittle prompted behavior but an internalized generation pattern—the model has learned that structured reasoning is how it answers questions, not just a format it can optionally follow.

Why 100k samples? The paper does not provide an explicit justification for the dataset size, but the fact that only 100k samples (with 3 epochs of training) produces substantial improvements suggests that the structured format provides a strong inductive bias—the model doesn't need to see millions of examples to learn the reasoning structure because the tag format itself constrains the generation space. The ablation comparing LLaVA-CoT to the model trained directly on original Q&A pairs without the structured format (Table 2, row "LLaVA-CoT (with Direct Training)") shows the structured format provides 3.4 points of average improvement (62.4 vs. 59.0), confirming that the structure itself—not just additional training data—drives the gains.

---

Stage-wise Beam Search

Before describing the full SWIRES algorithm, it's essential to understand the stage-wise beam search that it extends. Stage-wise beam search addresses a granularity problem in traditional beam search for language models (Section 3.2.1).

The granularity problem. Traditional beam search in text generation operates at fixed intervals—after every K tokens or after every sentence boundary. For visual reasoning, this is problematic because reasoning complexity varies dramatically between problems and between stages within a single problem. A search step after 20 tokens might split a mathematical formula mid-expression, while a search step after every sentence might be too coarse for multi-sentence reasoning chains where errors in the first sentence propagate through all subsequent candidates.

Stage-wise beam search procedure. The algorithm operates as follows (Section 3.2.1, Figure 4 middle panel):

1. Generate initial stage. Produce one candidate for the first stage (summary). The paper notes that the summary stage "typically produces high-quality outputs," so search is not applied here.
2. At each subsequent stage:
   • Generate M candidate responses for the current stage.
   • Score each candidate using a reward model.
   • Select the top N candidates by reward score.
   • Each selected candidate then generates M/N new candidates in the next stage (maintaining a constant total of M candidates per stage).
3. Repeat through all stages. After the conclusion stage, select the highest-scoring final response.

Why M/N branching? The branching factor M/N ensures that the total computational budget (number of generations per stage) remains constant at M across all stages, regardless of how many candidates N are retained. If N=2 and M=4, each of the 2 selected captions generates 2 reasonings, maintaining 4 total reasonings. If N=1 and M=4, the single selected caption generates 4 reasonings. This design allows the beam width (N) to be tuned without changing the total compute budget (M), enabling fair comparison across different search widths.

The local optima problem. As the authors identify (Section 3.2.1), stage-wise beam search has a critical weakness: "since the reward model selects the highest-scoring response based only on the current stage, it may suffer from local optima or selection biases." A concrete example: if the caption stage produces a response that scores well according to the reward model but contains a subtle factual error (e.g., misreading "400 mm" as "200 mm" in the engineering problem), subsequent reasoning stages building on that caption will produce wrong answers regardless of how well they reason. Because the beam search only looks forward—selecting the best candidates at each stage without the ability to reconsider earlier choices—the error is locked in. The authors frame this as a need for "self-reflection and error correction" through retracing.

---

Stage-wise Retracing Search (SWIRES)

SWIRES enhances stage-wise beam search with a backtracking mechanism that enables the model to recognize when current-stage outputs are poor—implying an error in an earlier stage—and retrace to regenerate the previous stage's outputs (Section 3.2.2). The full algorithm is presented as pseudocode in Appendix D (Algorithm 1) with hyperparameters in Table 8.

Algorithm overview. The SWIRES procedure operates as follows, with step numbers matching Algorithm 1:

Step 1: Generate initial summary. The model generates exactly one response for the summary stage (denoted "first stage" in the pseudocode). The paper notes empirically that the summary stage typically produces high-quality outputs, so retracing is not applied here, and only one candidate is generated.

Step 2: Initialize tracking variables. A backtracking counter c is initialized to 0. A reasoning candidates list Cand and a corresponding score list Score are initialized as empty.

Step 3: Generate caption candidates and select top N. The model generates M captions, where M=4 (from Table 8). Each caption is evaluated using the reward model (InternLM-XComposer2.5-Reward [64]). The top N=2 captions by reward score are selected for progression to the next stage.

Step 4: Generate reasoning candidates. For each of the N=2 selected captions, the model generates M/N = 4/2 = 2 reasonings, producing a total of M=4 reasoning candidates. Each reasoning is evaluated by the reward model, and both the reasoning text and its score are appended to the Cand and Score lists respectively.

Step 5: Check threshold condition. If the generated reasonings satisfy the preset condition, the loop breaks and the algorithm proceeds to conclusion generation. If not, the backtracking counter increments (c = c + 1), and the algorithm retraces to the caption stage (returning to Step 3). This retracing loop continues until either the preset condition is met or the counter reaches the maximum C=3 iterations.

Step 6: Select top N reasonings. After breaking from the loop (either by satisfying the condition or exhausting retracing attempts), the top N=2 reasonings are selected from all accumulated candidates based on their reward scores.

Step 7: Generate conclusions. For each of the top N reasonings, the model generates exactly one conclusion (producing N total conclusions). All conclusions are evaluated by the reward model.

Step 8: Return best conclusion. The conclusion with the highest reward score is returned as the final answer.

The reward threshold condition. The "preset condition" that determines whether to retrace is a statistical threshold on the reward scores of the top candidates (Appendix D). The threshold is calculated as:

$\text{backtrack\_cutoff} = \text{reward\_mean} + Z \times \text{reward\_std}$

where $\text{reward\_mean} = -0.77$ is the mean reward score, $\text{reward\_std} = 2.08$ is the standard deviation of reward scores, and $Z = 0.2533$ is a coefficient from the standard normal distribution.

What this condition means operationally. The preset condition is satisfied when the top candidate in the current stage's candidate set has a reward score exceeding this cutoff value. The $\text{reward\_mean}$ and $\text{reward\_std}$ are derived from "statistically analyzing the reward scores output by the reasoning phase reward model on the MMStar dataset," and the authors note that "the distribution of the reward model output in this phase is close to a Gaussian distribution." The Z value of 0.2533 is described as "a special coefficient in the standard normal distribution, where values greater than 0.2533 × std + mean account for 40% of the distribution."

Why this threshold design. The authors' reasoning (Appendix D): "if we retain N reasonings, having one of them being sufficiently good will ensure that the remaining reasoning, when selecting one conclusion from three, will provide enough relevant reference." In other words, the threshold is set so that if at least one of the generated reasonings is in the top 40% of expected reasoning quality, the system considers the current stage satisfactory and proceeds. If no reasoning meets this bar, the system infers that the preceding stage (caption) may have been flawed, since even the best reasoning built on that caption is subpar. The threshold thus serves as an early fault detection mechanism—rather than completing the entire reasoning chain and discovering an error at the end, SWIRES identifies likely upstream errors early and corrects them before propagating.

Retracing mechanics and termination. When retracing is triggered (no candidate exceeds the cutoff), the algorithm does not simply discard the current stage's results. Instead, it "retains the current stage's search results as candidates" but returns to the previous stage to generate new outputs. The previous stage is regenerated, and these new outputs are used to generate M new candidates for the current stage. If any of these new candidates now exceed the threshold, the search terminates immediately and the top N among all accumulated candidates proceed. If not, the new candidates are added to the candidate pool and retracing continues (up to C=3 times). After C retracing attempts, the algorithm proceeds with the best available candidates regardless of whether the threshold was met—this prevents infinite loops on problems that are genuinely too difficult for the model.

Why retracing works (theoretical intuition). The retracing mechanism is a form of test-time self-reflection that operates at the granularity of reasoning stages. When all reasoning attempts built on a particular caption are poor, the system hypothesizes that the caption itself contains an error (e.g., misreading a number, missing a relevant object, describing an irrelevant detail). By regenerating the caption, the model gets a chance to produce a different visual interpretation that might enable correct reasoning. This is fundamentally different from best-of-N, where errors in early parts of a complete response are uncorrectable because there is no mechanism to isolate which part of the response is wrong. It's also different from standard beam search, which can select a different candidate at the current stage but cannot revise its selection at a previous stage.

Hyperparameter configuration (Table 8). The specific values chosen are:

• M = 4 (candidates generated per stage)
• N = 2 (candidates retained per stage)
• C = 3 (maximum retracing attempts)
• reward_mean = -0.77
• reward_std = 2.08
• Z = 0.2533

The values of M, N, and C represent a tradeoff between exploration breadth (more candidates) and depth (more retracing attempts) under a fixed computational budget. M=4 and N=2 means each stage maintains 4 total candidates while preserving 2 distinct "branches," providing enough diversity to explore different reasoning paths while keeping the top-2 selection manageable. C=3 means the system will attempt up to 3 caption regenerations before giving up—enough to potentially correct a faulty initial observation without excessive computation on problems genuinely beyond the model's capability.

The reward model choice. The authors use InternLM-XComposer2.5-Reward [64] as the reward model for all stages, noting that "there are currently not many open-source reward models in the multi-modal field that align with human preferences, and this model bridges this gap with a simple yet effective multi-modal reward model that aligns LVLMs with human preferences" (Appendix D). The reward model takes as input the current stage's output (e.g., a caption or reasoning text) and outputs a scalar score. The same reward model is used across all stages, meaning it must evaluate diverse output types—short captions, lengthy reasoning chains, and brief conclusions—with a single scoring function.

Why SWIRES scales better than alternatives (Figure 5). The scaling curve in Figure 5 shows that best-of-N and stage-wise beam search plateau around 10,000 seconds, while SWIRES continues to improve. The authors' explanation: best-of-N operates at the coarsest level, evaluating complete responses. If an error occurs early in a response, the remaining computation is wasted because the entire response is scored poorly even if later sections are correct. Stage-wise beam search isolates errors to individual stages, but errors in early stages still propagate because there's no correction mechanism. SWIRES can detect and correct these early-stage errors through retracing, meaning additional compute (more retracing iterations) is spent on genuine error correction rather than on generating more variations of a fundamentally flawed reasoning chain. This targeted allocation of compute to where errors are most likely explains the continued scaling behavior.

---

Design Choices and Their Justifications

Choice 1: XML-style tags over prompting-based structure. The authors could have achieved structured reasoning through test-time prompting (as in Prism [45]), but chose to embed the structure into the model via fine-tuning. The justification is robustness: prompting-based structure depends on the model correctly interpreting the prompt, and fails when the prompt is removed or when the model's instruction-following is imperfect. Training-based structure makes the format an internalized behavior that the model autonomously maintains. The ablation in Appendix F confirms this: prompting the base Llama model with structured CoT format yields no improvement (56.9 with and without CoT prompting), while training on structured data yields 6.2 points of improvement (63.1 vs. 56.9).

Choice 2: Four specific stages rather than a different decomposition. The summary-caption-reasoning-conclusion structure mirrors cognitive models of problem-solving: orientation (understand the problem), information gathering (collect relevant data), computation (perform inferences), and response generation (produce answer). Alternative decompositions are possible—for instance, separating mathematical calculation from logical deduction, or adding a verification stage. The authors validate their specific decomposition through the shuffle ablation (Appendix F, Table 7): reordering the stages (e.g., reasoning before caption) reduces the average score from 63.1 to 58.2, barely above the base model's 56.9. This confirms that the specific ordering—and by extension the semantic roles of each stage—is causally important, not just the presence of stages.

Choice 3: GPT-4o as teacher rather than human annotation. Human annotation of 99k structured reasoning traces would be prohibitively expensive for a research project. GPT-4o provides a scalable alternative. The authors mitigate the risk of simply distilling GPT-4o's errors by (a) using a rigorous filtering pipeline that verifies answer consistency with ground truth, and (b) showing that the structured format itself provides benefits beyond GPT-4o's standard outputs (Appendix F, Table 7: GPT-4o with structured CoT outperforms GPT-4o standard). The concern about teacher-student distribution shift is acknowledged but not directly addressed in the current work—future work might explore RL-based refinement to reduce dependence on the teacher's specific reasoning patterns.

Choice 4: Z=0.2533 for the reward threshold. This specific value corresponds to the 60th percentile of a standard normal distribution (since 40% of the distribution lies above mean + 0.2533 × std). The authors' implicit justification is that requiring the top candidate to be in the top 40% of expected quality provides a reasonable balance between being too permissive (never retracing, missing error correction opportunities) and too strict (always retracing, wasting computation on problems that are fundamentally hard). The threshold is calibrated on the MMStar dataset's reward distribution—different datasets or reward models would require recalibration.

Choice 5: Retracing only from caption stage onward. The summary stage is generated as a single candidate without retracing, because the authors observe it "typically produces high-quality outputs." This is a pragmatic optimization: the summary is a high-level plan that is less likely to contain factual errors (it doesn't reference specific image details or perform calculations), so the expected benefit of retracing it is low relative to the computational cost. However, this means the system cannot recover from fundamentally wrong problem interpretations—if the summary misunderstands what the question is asking, all subsequent stages will be misdirected.

Choice 6: Single reward model for all stages. Using the same reward model (IXC-2.5-Reward) across all stages is simpler than training separate reward models for captions, reasonings, and conclusions. However, it requires the reward model to evaluate qualitatively different outputs—a caption's quality criteria (accuracy, relevance) differ from a reasoning's quality criteria (logical validity, mathematical correctness). The paper does not explore whether stage-specific reward models would improve performance, leaving this as a potential optimization for future work.

4. Key Insights and Innovations

Innovation 1: Structure Is Not Just a Prompting Trick — It's a Learned Capability That Causal Improves Reasoning

The dominant approach to improving VLM reasoning prior to this work treated structured thinking as an external constraint imposed at test time through carefully engineered prompts. Prism [45] manually decomposes the VLM process into perception and reasoning stages via prompting. MSG [8] enforces a forced Chain-of-Thought format through structured prompts. Visual CoT [49] generates bounding boxes alongside reasoning. These methods all share an assumption: the model already knows how to reason systematically, it just needs the right prompt to elicit that behavior.

LLaVA-CoT fundamentally challenges this assumption. The authors show that prompting alone is insufficient to make a model reason structurally — when they apply their carefully designed four-stage prompt to the base Llama-3.2-11B-Vision-Instruct, performance does not improve at all (Appendix F, Table 7: 56.9 with structured CoT prompting vs. 56.9 without). The model cannot be coaxed into systematic reasoning through prompt engineering because it lacks the underlying capability. It's not a problem of elicitation; it's a problem of capability absence.

What makes this insight distinctive is its reversal of the causal arrow. Rather than viewing structure as something you extract from an already-capable model through prompting, the authors treat structure as something you embed into the model through training, which then causally improves its reasoning quality. The evidence for causality is the ablation showing that training on the same 100k GPT-4o-labeled data without structured tags yields significantly worse performance (Table 2: 62.4 with tags vs. 60.9 without, a 1.5-point drop on average), and training with shuffled stage orders yields almost no improvement over the base model (Appendix F, Table 7: 58.2 for reordered stages vs. 56.9 for base). Structure isn't just a formatting preference — it's a reasoning scaffold that, when internalized through training, enables the model to organize information more effectively, separate observation from inference, and avoid the premature-conclusion pathology that plagues direct-response VLMs.

This is a fundamental reframing, not an incremental improvement. Prior work operated in a "better prompting" paradigm — the research question was "how do we design prompts that make the model think step by step?" LLaVA-CoT shifts to a "capability building" paradigm — the research question becomes "how do we train models to have structured thinking as an intrinsic behavior?" The distinction matters for the field's trajectory: if structure were just a prompting phenomenon, the path forward would be prompt optimization. Since structure requires training to be effective, the path forward is curriculum design and training data engineering that teaches models how to organize thought, not just what to think.

The significance extends beyond performance gains. This finding implies that many reasoning failures attributed to model capacity limits may actually be failures of reasoning organization — the knowledge is present but disorganized. The structured training acts as a form of metacognitive scaffolding that helps the model deploy its existing knowledge more effectively. This is consistent with the difficulty-bin analysis in Table 3, where LLaVA-CoT's improvements concentrate in reasoning-intensive areas (instance reasoning: +5.6 points, logical reasoning: +7.2 points, math: +18.8 points, science & technology: +12.0 points) rather than perceptual tasks (coarse perception: +2.8 points, fine-grained perception: +0.4 points). The model's visual perception capabilities haven't dramatically improved — what's changed is its ability to use perceptual information in structured logical chains.

Innovation 2: The Granularity of Search Must Align with Semantic Reasoning Stages, Not Arbitrary Text Boundaries

The test-time compute scaling literature — from the foundational work on beam search [19, 52] to recent LLM scaling analyses [51] — has predominantly applied search at fixed text boundaries: after every K tokens, after every sentence, or after complete response generation. The implicit assumption has been that any reasonably fine-grained search interval is adequate, and that the key variable is how much search you do (beam width, number of samples), not where you apply it.

LLaVA-CoT identifies a more nuanced problem: search granularity must match the semantic structure of the reasoning process to be effective. The paper's diagnostic observation is that fixed-interval search creates a mismatch — a search step after N tokens might split a mathematical expression mid-derivation (too fine), while a search step after every sentence might miss compounding errors within a multi-sentence reasoning chain (too coarse). This isn't a minor implementation detail; it's a structural mismatch between the search algorithm's operational units and the reasoning process's semantic units.

The stage-wise beam search proposed in Section 3.2.1 operationalizes this insight by anchoring search decisions to the explicit stage boundaries that the model learns to emit (</SUMMARY>, </CAPTION>, </REASONING>). Each search step now corresponds to a complete semantic unit — evaluating whether the entire caption stage's output is good, rather than whether a particular sentence fragment scores well. This is conceptually analogous to the difference between evaluating a student's essay sentence-by-sentence versus evaluating each paragraph as a complete thought. The former might flag an awkward sentence that's actually correct in context; the latter judges whether the paragraph as a whole achieves its communicative goal.

What elevates this from an engineering optimization to a genuine insight is that the tags themselves provide the search boundaries. The model's structured output format — originally designed to improve reasoning quality during training — serendipitously creates natural breakpoints for test-time search. This is an elegant design synergy: the same structure that helps the model produce better reasoning also helps the search algorithm evaluate that reasoning at appropriate granularity. The authors don't need to heuristically determine when to trigger search; the model tells them through its tag emissions.

Comparing to prior work sharpens the distinction. In the analyze paper's framework [51], test-time compute allocation is optimized over search algorithm choice (best-of-N vs. beam search) and sequential-vs-parallel sampling ratios, but always operates on complete solution attempts or fixed token windows. The LLaVA-CoT contribution adds a new dimension to the optimization space: search granularity alignment. This dimension wasn't visible in text-only LLM reasoning because text-only models don't naturally segment their reasoning into explicit stages — the continuous stream of tokens has no intrinsic structure to align with. Vision-language reasoning, with its necessary alternation between perception and inference, makes the need for stage-aligned search more apparent.

The empirical validation comes from Figure 5, where stage-wise beam search consistently outperforms best-of-N at comparable computation times. Best-of-N at 15,000 seconds achieves approximately 60.5% accuracy on MMStar, while stage-wise beam search achieves approximately 60.8% at roughly 5,000 seconds — similar performance at 3× less compute. This efficiency gain isn't from a better search algorithm in the abstract; it's from applying search at the right level of granularity for the task structure. This finding suggests that future test-time compute scaling methods — for VLMs and possibly for LLMs on structured reasoning tasks — should co-design the model's output format and the search granularity rather than treating them as independent design choices.

Innovation 3: Retracing as Test-Time Self-Correction Through Early Error Detection

The field's dominant approaches to test-time error correction operate at the level of complete solution attempts: generate a full response, evaluate it, and if it's wrong, generate a new full response (best-of-N) or condition on the previous attempt to produce a revision (sequential revision in [51]). These methods share a fundamental limitation — they can only recognize errors after the entire reasoning chain is complete. By that point, the computational cost of the flawed chain is already sunk, and there's no mechanism to identify where in the chain the error occurred to avoid repeating it.

LLaVA-CoT introduces a different paradigm: stage-level early error detection with targeted retracing. The SWIRES algorithm doesn't wait until the final answer is produced to judge quality. Instead, at each intermediate reasoning stage, it evaluates whether the current stage's outputs are satisfactory and, critically, infers that subpar current-stage outputs likely indicate an error in the previous stage (Section 3.2.2). This is a form of abductive reasoning about error sources: "my reasoning isn't working → the caption I'm reasoning from is probably wrong → let me regenerate the caption."

What makes this conceptually distinctive is the upstream error attribution mechanism. Standard beam search can select a different candidate at the current stage but cannot revise a suboptimal selection at a previous stage — it's strictly forward-looking. Best-of-N has no internal error localization at all — a wrong answer could be wrong because of a perceptual error in step 2, a calculation error in step 7, or a conclusion error in step 10, and the system has no way to distinguish these. SWIRES uses the reward model's stage-level scores as a diagnostic signal: low scores on all reasoning candidates suggest the caption (the shared input to all those reasonings) is the culprit, not the reasoning process itself.

This is a fundamental shift in how test-time compute can be deployed. Rather than spending compute on more attempts (breadth) or longer revision chains (depth), SWIRES spends compute on targeted correction of specific reasoning substages. The retracing budget (up to C=3 attempts) is allocated precisely where the diagnostic signal indicates it's most needed — the caption stage when reasoning is poor, rather than uniformly across all stages or at the end. This is analogous to debugging code by running unit tests on intermediate functions rather than only testing the final output — when a unit test fails, you know which function to fix, and you can fix it before it corrupts downstream computations.

The empirical evidence for this innovation's significance is the continued scaling behavior in Figure 5. Best-of-N search plateaus around 10,000 seconds and even slightly declines beyond that point (approximately 60.5% at 15,000 seconds vs. 60.8% at 10,000 seconds), consistent with the over-optimization phenomenon documented in [51] where additional samples eventually amplify verifier errors rather than finding genuinely better solutions. Stage-wise beam search plateaus near the same region (approximately 61.0% at 10,000 seconds). SWIRES, by contrast, continues to improve: from 60.8% at roughly 5,000 seconds to 62.5% at roughly 15,000 seconds. This continued scaling isn't magic — it's the consequence of compute being allocated to error correction rather than to generating more variations of fundamentally flawed reasoning. When the system detects a faulty caption, retracing gives it a genuine chance to fix the root cause rather than trying to reason correctly from wrong premises.

A subtle but important implication: this result suggests that test-time compute scaling for reasoning is bottlenecked not by search algorithm sophistication but by error attribution granularity. Detailed search algorithms (MCTS, lookahead search) that don't isolate error sources will eventually plateau because they're optimizing the wrong thing — selecting among candidates that all build on shared early-stage errors. SWIRES's retracing mechanism is conceptually simple (just regenerate and check a threshold) but powerful because it targets the cause of errors rather than their symptoms. This aligns with the finding in [51] that sequential revision — which can correct early errors by revisiting the entire solution — outperforms parallel search on easy problems where errors tend to be localized. SWIRES generalizes this intuition to fine-grained, per-stage revision within a single solution attempt.

Innovation 4: A Positive Result That Small Data + Structured Training Can Close the Gap to Much Larger Models on Reasoning Tasks

The scaling laws literature [51] has established that test-time compute can sometimes substitute for model size, but these analyses typically assume access to models with strong base reasoning capabilities and substantial training data. The LLaVA-CoT results present a striking counterpoint: with only 100k training samples and a relatively small 11B-parameter base model that starts as the weakest performer among all compared models (Figure 1, Table 5: 56.9 average on reasoning benchmarks), the structured fine-tuning approach produces a model that outperforms:

• Comparable-size open-source models like Qwen2-VL-7B (65.9 vs. 66.3), InternVL2-8B (64.0 vs. 66.3), and Ovis1.5-Gemma2-9B (64.0 vs. 66.3)
• Larger open-source models like Llama-3.2-90B-Vision-Instruct (62.3 vs. 66.3), VILA-1.5-40B (56.9 vs. 66.3), and Deepseek-VL2 with 27B MoE parameters (66.0 vs. 66.3)
• Closed-source models like Gemini-1.5-Pro (63.6 vs. 66.3) and GPT-4o-mini (63.8 vs. 66.3)

What makes this significant beyond a leaderboard entry is the efficiency of the intervention. The base model (Llama-3.2-11B-Vision-Instruct) requires 3 epochs of fine-tuning on 100k samples — a training cost that is negligible compared to pretraining a 90B-parameter model from scratch. The entire fine-tuning runs on a single node with 8 H100 GPUs. Yet the resulting model matches or exceeds models with 4× to 8× more parameters that were trained on orders of magnitude more data. This suggests that structured reasoning is a high-leverage capability — relatively small investments in teaching a model how to organize its reasoning yield disproportionate gains compared to scaling parameters or data volume.

This insight challenges a common assumption in the VLM literature: that reasoning improvements require either larger models, more training data, or more sophisticated architectures. The LLaVA-CoT results suggest that for many reasoning tasks, the bottleneck is not raw capacity but reasoning methodology. A small model that follows a systematic problem-solving procedure can outperform a much larger model that reasons in an unstructured, stream-of-consciousness fashion. This is consistent with the skill-level analysis in Table 3, where the base model already has reasonable perceptual capabilities (66.0 coarse perception, 46.4 fine-grained perception) but weak reasoning (50.8 logical reasoning, 45.2 math). The structured training doesn't dramatically improve perception (68.8 and 46.8, roughly flat) but substantially improves reasoning (58.0 logical reasoning, 64.0 math). The model already knew enough to solve these problems; it just didn't know how to deploy that knowledge systematically.

The implication for the field is a reprioritization of research investment. If teaching reasoning methodology yields higher returns than scaling model size for a fixed compute budget, then the path to better reasoning VLMs may lie in better training curricula and data engineering rather than in ever-larger architectures. This is not to say scaling doesn't matter — the paper explicitly shows that on the hardest problems, even structured training provides limited gains (Appendix J notes failure cases where "the input image may be overly complex and exceed the model's visual understanding capabilities"). But for the substantial fraction of reasoning problems that are within a model's capability range but are answered incorrectly due to disorganized reasoning, structured training provides an efficient path to improvement that complements rather than replaces scaling.

A nuance worth highlighting: this result doesn't mean that 100k samples of structured data is all you need for any VLM. The base model (Llama-3.2-11B-Vision-Instruct) has already been instruction-tuned on a much larger corpus, giving it strong language understanding and basic visual reasoning. The 100k structured samples build on this foundation, refining the model's reasoning process rather than teaching it new facts or visual concepts. For a model with weaker base capabilities, the same 100k samples would likely be insufficient. The insight is about leverage: given a reasonably capable base model, structured training is a highly efficient way to improve reasoning that would otherwise require orders of magnitude more unstructured data to achieve.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The primary evaluations use six multimodal reasoning benchmarks, accessed through VLMEvalKit [16]: MMStar [10], MMBench V1.1 [36], MMVet [62], MathVista [38], AI2D [28], and HallusionBench [20]. MMStar, MMBench, and MMVet evaluate general visual question-answering capabilities, while MathVista and AI2D focus on mathematical and scientific reasoning. HallusionBench specifically assesses handling of language hallucinations and visual illusions. For MMBench, the V1.1 test set is used; MathVista uses the testmini set; the remaining benchmarks each have a single test set. For the state-of-the-art comparison (Table 5), the authors create filtered "reasoning-only" subsets—MMStar-R, MMBench-R, and MMVet-R—by removing tasks requiring only perception and OCR, retaining tasks that demand reasoning. Details of the filtering criteria are provided in Appendix E.

• Base model. All experiments use Llama-3.2-11B-Vision-Instruct [43] as the starting checkpoint. This model is the weakest performer among all compared models in Table 5 (56.9 average on reasoning benchmarks), making it a challenging base that provides a strong test of whether structured training and test-time scaling can lift weak reasoning. The authors select this model to demonstrate that their method improves reasoning capability rather than relying on an already-strong base.

• Metrics. The primary metric is accuracy (%), reported per-benchmark and as an average across all six benchmarks. For the MMStar skill-level breakdown (Table 3), accuracy is reported within each of six skill categories: coarse perception, fine-grained perception, instance reasoning, logical reasoning, math, and science & technology. All evaluations are conducted using VLMEvalKit [16] to ensure fairness and reproducibility. The performance metrics of all baseline models are derived from VLMEvalKit's testing results.

• Baselines. The paper compares against multiple categories of models. Closed-source models: GPT-4o-0806 [3], GLM-4v-Plus [18], Claude3.5-Sonnet-0620 [2], Gemini-1.5-Pro [46], and GPT-4o-mini-0718 [44]. Larger open-source models: Llama-3.2-Vision-Instruct-90B [43], VILA-1.5-40B [33], and Deepseek-VL2 (MoE, 27B parameters) [57]. Comparable-size open-source models: Qwen2-VL-7B [54], InternVL2-8B [11], Ovis1.5-Gemma2-9B [39], MiniCPM-V2.6 [60], Prism [45], and VisCoT-7b-336 [49]. For the training-time ablation (Table 2), baselines include the base model itself, LLaVA-CoT trained directly on original Q&A pairs without structured tags ("with Direct Training"), and LLaVA-CoT trained on the LLaVA-CoT-100k dataset with structured tags removed ("w/o Structured Tags").

• Generation budget / compute accounting. For the test-time scaling experiments (Section 5, Figure 5), computation is measured in wall-clock time (seconds) on a single A800 node, using a logarithmic scale for the x-axis. The three methods scale in different ways: best-of-N scales by increasing N (the number of complete response options), with N values of 1, 3, 4, and 8 for the four experimental points. Stage-wise beam search scales by expanding the number of candidates per stage, with values of 1, 4, 6, and 19. SWIRES scales by increasing the maximum number of retracing iterations, with values of 0, 1, and 3 for the three experimental points (Appendix G). This means the scaling curves compare methods under approximately matched total inference time rather than matched generation counts, since the per-generation cost differs between methods.

• Cross-validation / statistical protocol. The paper does not report cross-validation or statistical significance testing for benchmark results. Ablation comparisons in Tables 2 and 3 are presented as point estimates (single accuracy values) without confidence intervals or standard deviations. The reward model threshold parameters (Appendix D, Table 8) are calibrated on the MMStar dataset's reasoning phase reward score distribution, with the mean and standard deviation derived from statistical analysis described as "close to a Gaussian distribution," but no details on sample size or distribution fitting methodology are provided. This lack of statistical rigor is a notable weakness—the 500-question test sets split into skill categories of varying sizes (e.g., MMStar's six skills across an unspecified number of questions) mean that some accuracy figures in Table 3 are estimated from small samples, where point estimates alone can be misleading.

---

Main Quantitative Results

Post-Training Performance: LLaVA-CoT Without Test-Time Scaling

The headline result from training alone (Table 2, row "LLaVA-CoT") is an average benchmark score of 62.4 across the six reasoning benchmarks, representing a 5.8-percentage-point improvement over the base model's 56.6 average. The per-benchmark breakdown shows improvements on every benchmark: MMStar (49.8 → 57.6, +7.8 points), MMBench (65.8 → 75.0, +9.2 points), MMVet (57.6 → 60.3, +2.7 points), MathVista (48.6 → 54.8, +6.2 points), AI2D (77.3 → 78.7, +1.4 points), and HallusionBench (40.3 → 47.8, +7.5 points). The gains are most substantial on MMBench and MMStar, and smallest on AI2D, suggesting the structured reasoning approach helps more on benchmarks requiring systematic inference than on diagram understanding.

Comparison to direct training without structure (Table 2, row "LLaVA-CoT (with Direct Training)"). Training on the same 100k question-answer pairs but without the four-stage structured format yields an average of 59.0—only 2.4 points above the base model and 3.4 points below the full LLaVA-CoT. Critically, performance on MMVet actually degrades relative to the base model (49.9 vs. 57.6, a drop of 7.7 points), while the full LLaVA-CoT improves MMVet by 2.7 points. The authors attribute this to MMVet requiring "more detailed responses," suggesting that unstructured training on dense answers may actually harm performance on tasks requiring nuanced generation, while the structured format preserves and improves this capability.

Comparison to structured training without tags (Table 2, row "LLaVA-CoT (w/o Structured Tags)"). Removing the XML-style tags from training data while keeping the four-stage content drops the average to 60.9—1.5 points below the full model. The performance degradation is uneven across benchmarks: MathVista drops from 54.8 to 54.1 (-0.7), HallusionBench drops from 47.8 to 45.0 (-2.8), and MMStar drops from 57.6 to 55.7 (-1.9). This confirms that the tags themselves—not just the semantic content of the stages—contribute to reasoning quality. The authors interpret this as evidence that "structured tagging facilitates reasoning and improves model performance" (Section 4.3), presumably because the explicit stage boundaries help the model maintain focus on the current reasoning subtask rather than blurring between observation, inference, and conclusion.

Skill-level analysis on MMStar (Table 3). The breakdown by cognitive skill reveals that LLaVA-CoT's improvements are concentrated in reasoning-intensive areas rather than perceptual tasks:

The math and science gains are dramatic—18.8 and 12.0 points respectively—while perception gains are negligible (0.4–2.8 points). This pattern directly supports the paper's central claim that structured reasoning specifically improves reasoning capabilities rather than generally boosting all VLM performance. It also suggests that the base model already had adequate visual perception; its failures on reasoning benchmarks were primarily failures of logical organization, not failures of seeing. The structured training teaches the model to deploy its existing perceptual capabilities within a systematic reasoning framework.

Comparison to training with shuffled stage order (Appendix F, Table 7). Training with the four stages presented in a non-natural order (e.g., reasoning before caption) yields an average score of 58.2 on the reasoning benchmarks—barely above the base model's 56.9 and substantially below LLaVA-CoT's 63.1. This ablation is critical because it distinguishes between the contribution of having multiple stages and the contribution of having stages in the correct logical sequence. The shuffled model still receives the same GPT-4o supervision and still produces outputs with multiple labeled sections, but the unnatural order (captioning after reasoning, or concluding before summarizing) disrupts the logical dependency chain that makes structured reasoning effective.

Comparison to multi-task training (Appendix F, Table 7). Training separate models or heads for captioning, summarization, reasoning, and conclusion generation—without chaining them in a CoT format—yields an average of 57.7, essentially identical to the base model (56.9). This controls for the possibility that LLaVA-CoT's gains come simply from denser GPT-4o supervision rather than from the structured CoT process itself. The multi-task model receives the same amount of GPT-4o-generated content as LLaVA-CoT, but without the sequential stage structure connecting the outputs, the supervision doesn't translate into improved reasoning. This is strong evidence that the CoT structure is the active ingredient.

GPT-4o prompting comparison (Appendix F, Table 7). When prompted with the same structured CoT format used to generate LLaVA-CoT-100k, GPT-4o's performance improves from 71.8 to 74.1 average on reasoning benchmarks (+2.3 points). This is a meaningful gain for a model that already reasons well, suggesting the structured format provides benefits even at high capability levels. However, the same structured CoT prompt applied to the base Llama model yields no improvement (56.9 with and without CoT prompting). This asymmetry validates the paper's approach: Llama-3.2-11B-Vision-Instruct lacks the reasoning capability to benefit from structure through prompting alone, but can learn to reason structurally through fine-tuning on structured examples. The structure provides a scaffold that the model can internalize through training even though it cannot exploit it through zero-shot prompting.

Test-Time Scaling Performance (Section 5)

Scaling curve comparison (Figure 5). Tested on MMStar, all three methods—best-of-N search, stage-wise beam search, and SWIRES—show improved accuracy with additional test-time computation, but with substantially different scaling trajectories:

• Best-of-N search: Starts at approximately 57.6 at the lowest time point (~1,000 seconds, N=1), rises to approximately 60.5% at ~10,000 seconds (N=4), and then plateaus and slightly declines to approximately 60.1% at ~15,000 seconds (N=8). The decline at high compute is consistent with the verifier over-optimization phenomenon documented in the test-time scaling literature—additional samples eventually amplify reward model errors rather than finding genuinely better solutions.

• Stage-wise beam search: Starts at approximately 57.6 at ~1,000 seconds (1 candidate), rises to approximately 60.8% at ~5,000 seconds (4 candidates total), reaches approximately 61.2% at ~10,000 seconds (6 candidates total), and then plateaus at approximately 60.8% at ~15,000 seconds (19 candidates total). Stage-wise beam search consistently outperforms best-of-N at comparable time budgets, with the efficiency advantage most pronounced at moderate compute levels: it achieves ~60.8% at roughly 5,000 seconds, which best-of-N requires ~10,000 seconds to approach.

• SWIRES (stage-wise retracing search): Starts at approximately 57.6 at ~2,000 seconds (0 retracing iterations), rises to approximately 60.8% at ~5,000 seconds (1 retracing iteration), and continues scaling to approximately 62.5% at ~15,000 seconds (3 retracing iterations). Unlike both baselines, SWIRES shows no plateau within the tested compute range—the scaling curve continues upward, suggesting that additional retracing iterations would yield further improvements.

Key comparisons at matched time budgets. At approximately 5,000 seconds, SWIRES achieves ~60.8% while stage-wise beam search achieves ~60.5% and best-of-N achieves ~58.7%—a 2.1-point advantage over best-of-N. At approximately 10,000 seconds, SWIRES reaches ~61.0% compared to ~61.2% for stage-wise beam search (roughly tied) and ~60.5% for best-of-N. At approximately 15,000 seconds, SWIRES achieves ~62.5% compared to ~60.8% for stage-wise beam search and ~60.1% for best-of-N—a 2.4-point advantage over the best baseline. The relative advantage of SWIRES grows with compute budget, consistent with the hypothesis that retracing enables more effective use of additional computation by targeting error correction rather than generating redundant variations.

Full benchmark results with scaling (Table 4). Comparing LLaVA-CoT without scaling to LLaVA-CoT with SWIRES scaling across all six benchmarks:

SWIRES provides additional gains on top of the training-time improvements, with the largest absolute gains on MMStar (+4.9 points) and MMVet (+4.6 points), and the smallest on HallusionBench (+1.3 points) and AI2D (+2.3 points). This pattern suggests SWIRES is most beneficial on benchmarks where reasoning errors are correctable through retracing—MMStar and MMVet contain multi-step reasoning problems where a single misstep can compound—and less beneficial on benchmarks where errors stem from fundamental perceptual limitations (HallusionBench's visual illusions) or where the answer is relatively straightforward once correct perception is achieved (AI2D).

State-of-the-Art Comparison (Table 5)

On reasoning-focused benchmarks (MMStar-R, MMBench-R, MMVet-R, MathVista, AI2D, HallusionBench). LLaVA-CoT with SWIRES scaling achieves an average of 66.3, compared to the base model's 56.9—a 9.4-point improvement that moves the model from the bottom of the comparison table to a competitive position.

Versus closed-source models. LLaVA-CoT (w/ scaling) at 66.3 outperforms GPT-4o-mini (63.8) by 2.5 points and Gemini-1.5-Pro (63.6) by 2.7 points. It remains behind Claude3.5-Sonnet (66.7, -0.4), GPT-4o-0806 (71.8, -5.5), and GLM-4v-Plus (72.5, -6.2). The fact that an 11B open-source model can outperform two major closed-source APIs on reasoning benchmarks is a significant practical result, though the gap to the strongest closed-source models remains substantial.

Versus larger open-source models. LLaVA-CoT (w/ scaling) at 66.3 outperforms Llama-3.2-90B-Vision-Instruct (62.3, +4.0), VILA-1.5-40B (56.9, +9.4), and draws essentially even with Deepseek-VL2 (66.0, +0.3). Outperforming a 90B model from the same family with an 11B model is particularly striking and directly supports the paper's implicit claim that structured reasoning methodology can substitute for model scale.

Versus comparable-size open-source models. LLaVA-CoT (w/ scaling) edges out Qwen2-VL-7B (65.9 vs. 66.3, +0.4), InternVL2-8B (64.0 vs. 66.3, +2.3), Ovis1.5-Gemma2-9B (64.0 vs. 66.3, +2.3), and MiniCPM-V2.6 (63.3 vs. 66.3, +3.0). It substantially outperforms Prism-7B (48.3, +18.0) and VisCoT-7b-336 (31.7, +34.6), though these models represent earlier-generation approaches.

Where LLaVA-CoT does not lead. On individual benchmarks, LLaVA-CoT (w/ scaling) is not best-in-class on any single benchmark—it achieves strong but not dominant performance across the board. For instance, Qwen2-VL-7B outperforms it on MathVista (61.4 vs. 57.7) and AI2D (83.0 vs. 81.0), while InternVL2-8B outperforms it on MMStar-R (62.5 vs. 63.0—but note that the MMStar-R figure for LLaVA-CoT w/ scaling in Table 5 is 63.0 vs. InternVL2-8B's 62.5, so LLaVA-CoT actually leads here). The strength of LLaVA-CoT is in balanced performance across diverse reasoning tasks rather than dominance on any specific benchmark type.

---

Ablation Studies and Robustness Checks

Structured tags removed from training data (Table 2, row "LLaVA-CoT (w/o Structured Tags)"). Training on the LLaVA-CoT-100k dataset with the semantic content of the four stages preserved but without the XML-style demarcation tags reduces average performance from 62.4 to 60.9 (-1.5 points). The drop is uneven: HallusionBench is most affected (47.8 → 45.0, -2.8 points), while MMVet shows essentially no change (60.3 → 57.0, -3.3 points—actually, 60.3 is the value without scaling for the full model, but the untagged model gets 57.0, a 3.3-point drop). The authors interpret this as evidence that the tags themselves facilitate reasoning, not just the multi-stage content, though the mechanism is not directly tested. Possible explanations include: the tags provide clear task boundaries that help the model switch between reasoning modes, or the tags serve as anchors during training that enable more effective credit assignment during SFT.

Direct training on original Q&A pairs (Table 2, row "LLaVA-CoT (with Direct Training)"). Using the same 100k examples but training on the original dataset's question-answer format (without the four-stage decomposition) yields only 59.0 average, a 3.4-point deficit to the full model. This controls for the data quantity and source: the full model's advantage is attributable to the structured reasoning format, not simply to seeing more training examples. The MMVet result is particularly informative: direct training degrades MMVet by 7.7 points (57.6 → 49.9), while structured training improves it by 2.7 points (57.6 → 60.3). This suggests that fine-tuning on dense, unstructured answers may inadvertently train the model to produce verbose but less accurate responses, while the structured format constrains the model to a more disciplined generation pattern that preserves or enhances answer quality.

Stage order shuffled (Appendix F, Table 7, row "LLaVA-CoT (reorder)"). Training with the four stages presented in non-natural order yields an average reasoning-benchmark score of 58.2, compared to 63.1 for the full LLaVA-CoT and 56.9 for the base model. This is the most important ablation for the paper's central claim that structured reasoning with the correct logical order is the causal mechanism. The shuffled model has the same number of stages, the same tags, the same GPT-4o supervision, and the same training procedure—only the sequential dependency structure is disrupted. The near-complete elimination of gains (58.2 vs. 56.9, only +1.3 points over base) demonstrates that the dependency chain (summary informs caption, caption informs reasoning, reasoning informs conclusion) is essential, not just the presence of multiple labeled stages. This is strong causal evidence for the paper's theoretical framing.

Multi-task training without CoT (Appendix F, Table 7, row "LLaVA-CoT (multi-task)"). Training on the same GPT-4o-generated content but formatted as independent tasks (summarization, captioning, reasoning, conclusion generation) without chaining them into a sequential CoT format yields 57.7 average, nearly identical to the base model (56.9). This rules out the hypothesis that LLaVA-CoT's gains come from multi-task learning or from denser supervision from GPT-4o. The multi-task model receives the same information as LLaVA-CoT but cannot learn the transitional dependencies between stages—it doesn't learn that captions should inform reasoning or that summaries should guide caption focus. The result is a model that can produce individual stage-like outputs when prompted but cannot integrate them into a coherent reasoning chain, yielding essentially no reasoning improvement.

Structured CoT prompting without training (Appendix F, Table 7). Prompting the base Llama-3.2-11B-Vision-Instruct model with the same four-stage format yields 56.9 average—identical to the base model without structured prompting. This is a crucial negative result: the structured format alone, without training, provides zero benefit. In contrast, prompting GPT-4o with the same format improves its performance from 71.8 to 74.1 (+2.3 points). The asymmetry reveals that structured reasoning is a capability that must be learned—stronger models like GPT-4o already possess it and can be prompted to deploy it, while weaker models like Llama-3.2-11B-Vision-Instruct lack the capability and cannot simulate it through prompting. This directly validates the paper's core approach of training rather than prompting for structured reasoning.

Skill-level ablation (Table 3). Breaking down MMStar performance by cognitive skill reveals that structured training's benefits are entirely concentrated in reasoning-intensive areas: math (+18.8 points), science & technology (+12.0 points), logical reasoning (+7.2 points), and instance reasoning (+5.6 points) show large gains, while coarse perception (+2.8 points) and fine-grained perception (+0.4 points) show minimal improvement. The untagged model ("w/o Structured Tags") shows a similar but slightly attenuated pattern. This ablation addresses an important alternative explanation: if LLaVA-CoT's gains came from generally better training (more data, better optimization), we would expect improvements across all skill categories. The fact that gains are concentrated specifically in reasoning skills—and are absent from perceptual skills—provides convergent evidence that the structured reasoning format specifically improves reasoning rather than generally boosting model performance.

Test-time scaling method comparisons at fixed approximate compute (Figure 5). The three scaling methods—best-of-N, stage-wise beam search, and SWIRES—are compared at varying time budgets on MMStar. This is not a controlled-compute experiment in the strict sense, since wall-clock time on an A800 node can be affected by implementation efficiency, batch processing, and other factors not directly related to algorithmic quality. However, it provides a practical comparison of what each method achieves given the same inference hardware budget. The key finding is not just that SWIRES outperforms at the maximum budget (62.5 vs. 60.8 vs. 60.1 at ~15,000 seconds), but that SWIRES continues to scale where the other methods plateau—the slope of the SWIRES curve remains positive at the right edge of Figure 5, while best-of-N and stage-wise beam search have flattened or declined.

Comparison with and without test-time scaling on individual examples (Figure 7, Appendix H). The qualitative comparison in Figure 7 demonstrates a specific case where SWIRES corrects an error that the base LLaVA-CoT model (without scaling) makes. In the wine pricing example, without scaling the model produces "$149.99" (incorrect), while with scaling it produces "$249.98" (correct). The figure shows that the unscaled model's reasoning stage contains a truncated response ending with a wrong calculation, while the scaled model's selected reasoning contains the full correct calculation. This single qualitative example illustrates the mechanism: test-time scaling enables selection of better reasoning from multiple candidates, and retracing enables correction when all candidates from a given stage are poor. However, a single example does not establish the mechanism's reliability—the paper does not provide aggregate statistics on how often SWIRES successfully corrects errors versus how often it retraces without finding a better answer.

Limitation analysis (Appendix J). The authors acknowledge that LLaVA-CoT "sometimes gets lost during retracing or starts hallucinating in order to reach an answer." They attribute failure cases to "overly complex" input images that "exceed the model's visual understanding capabilities," noting that "even after retracing, the model may still fail to produce the correct answer." This is a qualitative observation without quantitative characterization—the paper does not report what fraction of failures are attributable to perceptual limitations versus reasoning errors versus retracing failures. This makes it difficult to assess whether the primary bottleneck for further improvement is better perception, better reasoning, or better retracing heuristics.

---

Critical Assessment

Does structured training improve reasoning, or does it improve benchmark performance through other mechanisms?

The experiments convincingly demonstrate that LLaVA-CoT achieves higher benchmark scores than the base model, and the skill-level analysis (Table 3) shows that gains concentrate in reasoning-intensive categories. However, the causal claim—that structured training improves reasoning—requires stronger evidence than benchmark accuracy improvements alone. The paper does not include a human evaluation of reasoning quality (e.g., rating the logical validity, factual accuracy, and completeness of generated reasoning traces). The GPT-4o-based filtering during dataset construction ensures training data quality, but the same filtering is not applied to evaluate model outputs at test time. It is possible that the structured format simply produces outputs that are formatted better (clearer organization, explicit step labeling) without being substantively more accurate in their reasoning. The MMVet result is suggestive here: direct training on unstructured answers degrades MMVet, which evaluates response quality holistically, while structured training improves it. But this is indirect evidence.

A more rigorous test would be to evaluate the intermediate reasoning traces (summary, caption, reasoning) for correctness independently of the final answer. If the structured model's captions are more accurate than the base model's implicit visual descriptions, and its reasoning steps are more logically valid, that would directly support the claim of improved reasoning rather than improved answer formatting. The paper's reliance on end-to-end benchmark accuracy leaves open the possibility that the model has learned to produce better-structured but not fundamentally more accurate reasoning.

Is the 5.8-point training gain substantial given the 100k sample investment?

The base model (56.9 average) improves to 62.4 (+5.8 points) after fine-tuning on 100k structured examples. This is a meaningful but not dramatic gain—the model moves from worst-in-class to roughly mid-pack among comparable-size models (Table 5). The paper's claim that LLaVA-CoT "achieves marked improvements" (abstract) and "demonstrates notable improvements" (Section 4.2) is fair given the base model's low starting point and the small training data investment. However, the +5.8-point gain should be contextualized: it represents closing roughly 35% of the gap between the base model (56.9) and the strongest comparable open-source model, Qwen2-VL-7B (65.9), but still leaves LLaVA-CoT 3.5 points behind that model before scaling is applied. The structured training meaningfully improves a weak base but does not, by itself, produce a best-in-class model.

Does SWIRES actually scale better, or does the scaling curve reflect different parameterizations?

Figure 5 shows SWIRES continuing to improve while best-of-N plateaus. However, the three methods are scaled through different mechanisms: best-of-N by increasing N (1, 3, 4, 8), stage-wise beam search by increasing candidates (1, 4, 6, 19), and SWIRES by increasing retracing iterations (0, 1, 3). These are not directly comparable scaling dimensions—retracing fundamentally uses computation differently than generating independent samples. The continued scaling of SWIRES may reflect that the tested retracing iteration counts (0–3) are still in a regime where additional retracing provides genuine error correction benefits, and that SWIRES would also plateau at higher retracing counts (e.g., 10 or 20 iterations) once all correctable errors are exhausted. Without testing higher retracing budgets, the claim of "superior scaling capability" (Section 5.2) is supported only for the tested range, not as an asymptotic property.

Additionally, the stage-wise beam search curve shows an unusual pattern: at ~15,000 seconds (19 candidates), it achieves the same accuracy as at ~5,000 seconds (4 candidates)—roughly 60.8%. This suggests that beyond a certain candidate count, additional beam width provides no benefit and may even be counterproductive (consistent with verifier over-optimization). The paper does not explore why 19 candidates perform similarly to 4 candidates, which limits our understanding of when and why stage-wise beam search saturates.

The absence of compute-matched comparisons between scaling methods.

Figure 5 compares methods at matched wall-clock times on the same hardware, but the x-axis values (1000–15000 seconds) are sparse and unevenly sampled—SWIRES has only 3 data points, stage-wise beam search has 4, and best-of-N has 4. The curves are constructed by connecting these sparse points, but the underlying function shapes are uncertain. A more rigorous comparison would measure total FLOPs or number of model forward passes and compare methods at matched computational budgets, which would eliminate implementation-specific timing artifacts. The paper's use of wall-clock time as the compute metric is pragmatic but makes it difficult to determine whether SWIRES's advantage comes from algorithmic efficiency or from implementation factors (e.g., caching of intermediate states, batch processing).

The state-of-the-art comparison has a significant confound: the base model difference.

Table 5 compares LLaVA-CoT (with scaling) at 66.3 to Qwen2-VL-7B at 65.9 and InternVL2-8B at 64.0, claiming superiority. However, LLaVA-CoT uses an 11B-parameter base model while Qwen2-VL uses 8B parameters and InternVL2 uses 8B. The 3B parameter difference (11B vs. 8B) is nontrivial—LLaVA-CoT has roughly 37.5% more parameters than the "comparable size" models it's compared against. The paper's narrative emphasizes that LLaVA-CoT starts from "the weakest performer" (56.9) and surpasses these models, which is true of the specific base model they chose. But a fairer comparison would ask: if the same structured training were applied to Qwen2-VL-7B or InternVL2-8B as the base, would the resulting model outperform LLaVA-CoT? The paper cannot answer this without cross-model-family experiments, which are absent.

Missing experiments that would strengthen the paper.

The paper would benefit from several additional experiments:

1. Reward model ablation: How does SWIRES perform with a different reward model, or with no reward model (e.g., using self-consistency through majority voting of final answers)? The paper uses IXC-2.5-Reward throughout and never tests whether the retracing mechanism's effectiveness depends on reward model quality. If a simpler reward model or majority voting achieved similar results, the complexity of the threshold calibration would be unnecessary.

2. Retracing depth ablation: The paper tests 0, 1, and 3 retracing iterations (Figure 5) but doesn't test intermediate values (2, 4, 5) to characterize the scaling curve more precisely. The jump from 1 to 3 iterations with no datapoint at 2 makes it impossible to assess whether retracing benefits are linear or saturating in this range.

3. Per-benchmark scaling curves: Figure 5 shows scaling only on MMStar. The paper reports full benchmark results with scaling in Table 4 but doesn't show per-benchmark scaling curves. Given that SWIRES shows different absolute gains on different benchmarks (MMStar +4.9, HallusionBench +1.3), scaling curves for each benchmark would reveal whether SWIRES scales on all benchmarks or only on certain types.

4. Difficulty-level analysis for scaling: The base model's improvement from training is analyzed by skill (Table 3), but the scaling results (Figure 5, Table 4) are aggregated without difficulty or skill breakdowns. Understanding whether SWIRES helps more on easy, medium, or hard questions—as the test-time scaling literature [51] would predict—is essential for characterizing when and why retracing works.

5. Direct comparison to prompting-based structured reasoning: The paper argues that training-based structure is superior to prompting-based structure, but never directly compares LLaVA-CoT to the base model + structured prompting on the same benchmarks in the main results. Appendix F provides this comparison for reasoning benchmarks but not for the full benchmark suite. A side-by-side comparison in the main results would strengthen the claim.

What the experiments do and do not demonstrate.

The experiments do demonstrate that fine-tuning an 11B VLM on 100k structured reasoning examples improves benchmark performance by 5.8 points, with gains concentrated in reasoning-intensive skill areas and dependent on both the presence of stage tags and the correct logical ordering of stages. The experiments also demonstrate that SWIRES provides additional gains (+3.1 points on average) and shows continued scaling on MMStar in a regime where best-of-N plateaus.

The experiments do not demonstrate that LLaVA-CoT's reasoning is genuinely more accurate, logical, or faithful than the base model's—only that its final answers are more often correct. They do not demonstrate that SWIRES's retracing mechanism specifically corrects caption or reasoning errors (as opposed to simply generating more candidates and selecting better ones through the reward model). They do not characterize the conditions under which structured training or retracing fail, beyond a qualitative note about image complexity. And they do not establish whether the approach transfers to other base model families, other visual reasoning domains (video, embodied AI), or non-reasoning VLM tasks.

The paper's central narrative—that structured, stage-based reasoning is a learnable capability that improves VLM reasoning and enables more effective test-time scaling—is supported by the experimental evidence presented, but with important gaps in causal evidence, generalizability, and mechanistic understanding that future work should address.

6. Limitations and Trade-offs

6.1 The Difficulty Estimation Cost Is Unaccounted for in the Compute Budget

The assumption or constraint. The SWIRES algorithm depends on a reward model (InternLM-XComposer2.5-Reward) to evaluate candidate quality at every stage and to determine whether retracing is necessary via a calibrated threshold. The paper calibrates this threshold by "statistically analyzing the reward scores output by the reasoning phase reward model on the MMStar dataset" (Appendix D), deriving a mean of -0.77 and standard deviation of 2.08. The authors state that "the distribution of the reward model output in this phase is close to a Gaussian distribution." This calibration is performed on the same benchmark used to evaluate scaling performance. Additionally, the SWIRES hyperparameters (M=4, N=2, C=3, Z=0.2533) are fixed across all benchmarks without any reported per-benchmark tuning or sensitivity analysis.

The consequence. The threshold calibration is benchmark-specific. If the reward score distribution on other benchmarks (MathVista, HallusionBench, AI2D) differs substantially from MMStar's distribution—which is likely given that these benchmarks test qualitatively different reasoning skills—the threshold will be misaligned. A threshold that is too low means SWIRES rarely retraces and degenerates to stage-wise beam search; a threshold that is too high means SWIRES retraces excessively, wasting computation on problems where retracing cannot help. The paper reports scaling results only on MMStar (Figure 5) and aggregated benchmark results with scaling (Table 4), but never shows per-benchmark scaling curves or analyzes whether the same hyperparameters are optimal across benchmarks. A practitioner deploying SWIRES on a new domain would need to recalibrate the reward threshold and potentially all hyperparameters—a process requiring ground-truth labels, reward model access, and substantial computation that is not accounted for in the reported efficiency gains.

What evidence exists in the paper. The paper provides the calibration methodology and hyperparameter values in Appendix D and Table 8, but does not report: (1) whether these values were tuned on MMStar test data (creating potential data leakage between tuning and evaluation), (2) the sensitivity of SWIRES performance to each hyperparameter, (3) per-benchmark reward score distributions to validate the Gaussianity assumption or the transferability of the calibrated threshold, or (4) any experiment showing that the chosen hyperparameters are near-optimal rather than cherry-picked. The scaling curve in Figure 5 has only three data points for SWIRES (0, 1, and 3 retracing iterations, as listed in Appendix G), providing almost no information about the shape of the scaling function or the optimal retracing depth.

Mitigation status. Not addressed. The paper acknowledges that the reward model's score distribution is used for threshold calibration but treats the calibrated values as fixed constants without discussing cross-benchmark transfer or sensitivity. No ablation studies vary M, N, C, or Z to assess robustness. The authors do not suggest future work on adaptive or benchmark-agnostic threshold setting.

---

6.2 SWIRES Scaling Is Evaluated Only on MMStar, with Sparse and Incomparable Data Points

The assumption or constraint. The paper's central scaling claim—that SWIRES "demonstrates the strongest scaling effect as computation time increases" and "continues to scale beyond this time scale, highlighting its superior scaling capability" (Section 5.2)—is based entirely on Figure 5, which shows results on a single benchmark (MMStar) with a single metric (accuracy) and uses wall-clock time on a single A800 node as the compute axis. The three methods are scaled through different mechanisms: best-of-N by increasing N (1, 3, 4, 8), stage-wise beam search by increasing candidates (1, 4, 6, 19), and SWIRES by increasing retracing iterations (0, 1, 3) (Appendix G). The x-axis values are not aligned across methods—each method has data points at different approximate time values—making direct comparisons at matched budgets impossible without interpolation.

The consequence. The claim of superior scaling is supported by exactly three SWIRES data points and one benchmark. This is insufficient to establish a general property of the algorithm. The sparse sampling means we cannot distinguish between a genuinely superior scaling law (SWIRES continues improving asymptotically) and a more mundane explanation (SWIRES simply hasn't exhausted its effective retracing budget at C=3, and would plateau at C=5 or C=10 just as best-of-N plateaus at N=8). The use of wall-clock time rather than FLOPs or forward passes means that implementation-specific factors—Python loop overhead, GPU utilization efficiency, batch processing optimizations—may systematically advantage or disadvantage certain methods in ways unrelated to algorithmic quality. A method that is faster on one hardware configuration could be slower on another, making the scaling comparison hardware-dependent rather than algorithmic.

The scaling curves in Figure 5 also reveal an anomaly the paper does not discuss: stage-wise beam search at 19 candidates (~15,000 seconds) achieves the same accuracy (60.8%) as at 4 candidates (~5,000 seconds). This suggests that stage-wise beam search saturates very early and additional candidates provide zero marginal benefit—but since the method comparison uses different scaling parameters, we cannot determine whether SWIRES's continued improvement represents genuine superiority or simply the fact that stage-wise beam search was tested in a saturated regime while SWIRES was tested in an unsaturated regime.

What evidence exists in the paper. Figure 5 is the sole source of scaling evidence. Table 4 reports aggregated benchmark results with scaling but does not show per-benchmark scaling curves, making it impossible to assess whether SWIRES scales on all six benchmarks or only on MMStar. Appendix G provides the scaling parameter values for each data point, confirming the sparse and uneven sampling. The paper does not report error bars on the Figure 5 measurements, making it impossible to assess whether the ~60.8% vs. ~61.2% vs. ~62.5% differences are statistically significant or within measurement noise.

Mitigation status. The paper acknowledges that retracing is applied "starting from the caption stage" (Section 3.2.2) because the summary stage typically produces high-quality outputs, implicitly recognizing that retracing has differential benefits across stages. However, this is not extended to a recognition that scaling behavior may differ across benchmarks, difficulty levels, or problem types. The authors do not discuss the limitations of single-benchmark scaling analysis or suggest broader scaling studies as future work.

---

6.3 The Base Model Is the Weakest in Its Class; Cross-Family Generalization Is Untested

The assumption or constraint. All experiments use a single base model: Llama-3.2-11B-Vision-Instruct. The authors explicitly note this model has "the lowest average score" among compared models (Section 6, Table 5: 56.9 on reasoning benchmarks). The paper's claim—that structured training improves reasoning capabilities and enables effective test-time scaling—is validated only on this specific model family (Llama 3.2) and architecture. No experiments are conducted with other base VLMs such as Qwen2-VL, InternVL2, or LLaVA-OneVision to test whether the structured training methodology transfers across model families, architectures, or pretraining distributions.

The consequence. The selection of the weakest base model creates a ceiling effect that works in the paper's favor: any improvement appears large in absolute terms because the starting point is so low. The same structured training applied to a model that already reasons well (e.g., Qwen2-VL-7B at 65.9) might yield diminishing returns, since the base model already possesses much of the reasoning capability that LLaVA-CoT aims to teach. This is directly evidenced by the GPT-4o prompting experiment (Appendix F, Table 7): GPT-4o, which already reasons well, gains only +2.3 points from structured CoT prompting (71.8 → 74.1), while the base Llama model gains zero points from the same prompt (56.9 → 56.9). This suggests that the benefits of structured training are largest for models with poor baseline reasoning and may shrink substantially for stronger base models.

A practitioner choosing between (a) fine-tuning Qwen2-VL-7B with structured data and (b) using Qwen2-VL-7B as-is has no evidence from this paper about which choice yields better performance. The paper's state-of-the-art comparison (Table 5) compares LLaVA-CoT to these models but does not apply the same fine-tuning recipe to them, making the comparison between LLaVA-CoT (trained on structured data) and other models (trained on their proprietary recipes) confounded by both the base model and the training methodology.

What evidence exists in the paper. Table 5 shows LLaVA-CoT (w/ scaling) at 66.3 average vs. Qwen2-VL-7B at 65.9, but this is a cross-model-family comparison where the base models differ in architecture, pretraining data, and instruction tuning. The GPT-4o vs. Llama structured prompting comparison in Appendix F (Table 7) provides indirect evidence that structured reasoning benefits are larger for weaker base models. The paper does not include any experiment testing the same structured training procedure on a different base model.

Mitigation status. Not addressed. The paper does not claim cross-model generalization, but it also does not acknowledge this as a limitation. The authors frame the base model choice as demonstrating that their method "can mainly improve reasoning capabilities of the model" (Section 4.3), which is valid for the tested model but leaves open the question of whether the method improves reasoning in general or only improves reasoning in models that are particularly deficient at it.

---

6.4 SWIRES Introduces Latency Bubbles That Are Hidden by Wall-Clock Time Aggregation

The assumption or constraint. The SWIRES algorithm is inherently sequential at two levels: (1) each stage depends on the output of the previous stage, and (2) retracing creates a serial dependency loop where the caption stage must be regenerated and re-scored before new reasonings can be generated. The paper measures test-time computation in total wall-clock seconds on a single A800 node (Figure 5) and reports that SWIRES achieves the highest accuracy at the maximum time budget (~15,000 seconds). This aggregate time measurement obscures the latency distribution: a method that spends 15,000 seconds producing one answer through sequential retracing loops has fundamentally different deployment characteristics than a method that spends 15,000 seconds generating parallel samples that could be batched across multiple GPUs.

The consequence. For latency-sensitive applications—interactive assistants, real-time visual reasoning, embodied AI—the serial dependencies in SWIRES impose a hard lower bound on response time that cannot be reduced by adding more hardware. If the model takes 5,000 seconds (over 83 minutes) to produce a single answer through SWIRES, it is unusable for any application requiring sub-minute response times, regardless of accuracy. This contrasts with best-of-N, which can be parallelized across N GPUs to achieve the same total FLOPs in 1/N the wall-clock time. The paper's comparison of methods at matched total wall-clock time on a single GPU therefore systematically favors methods that use sequential computation efficiently (retracing) over methods that could benefit from parallelization (increased beam width, more samples). On a multi-GPU deployment where best-of-N could be run with N=8 in roughly the time of a single generation, the scaling curves in Figure 5 would shift substantially in best-of-N's favor.

The paper reports that at the highest SWIRES budget, the system performs up to 3 retracing iterations, each of which requires regenerating the caption stage (M=4 captions), re-scoring all captions with the reward model, regenerating reasonings (M=4 total), and re-scoring those. In the worst case (3 retracing iterations, none meeting the threshold), the system generates 4 caption sets and 4 reasoning sets sequentially before proceeding—a latency multiplier of approximately 4× over a single forward pass, independent of total FLOP count.

What evidence exists in the paper. Figure 5 shows all three methods on the same time axis, implicitly treating wall-clock time as the relevant cost metric. The paper never discusses latency, throughput, or parallelization potential. Appendix G provides the scaling parameters, confirming that SWIRES uses sequential retracing iterations rather than parallel exploration. The paper does not report the latency distribution, the average or worst-case number of retracing iterations triggered, or the fraction of problems that trigger retracing at all.

Mitigation status. Not addressed. The paper does not acknowledge the latency-throughput tradeoff, does not discuss deployment scenarios where latency matters, and does not suggest whether SWIRES could be modified to parallelize retracing attempts or to limit maximum latency for time-sensitive applications. This is a significant gap for a method presented as a practical test-time scaling approach.

---

6.5 The Structured Reasoning Format Is Rigid—Four Fixed Stages for All Problem Types

The assumption or constraint. LLaVA-CoT decomposes all visual reasoning problems into exactly four stages: summary, caption, reasoning, and conclusion. This decomposition is applied uniformly regardless of problem type—the same four-stage structure is used for geometry problems, chart interpretation, hallucination detection, counting tasks, and scientific reasoning. The model has no mechanism to skip unnecessary stages (e.g., omitting the caption stage when the image is irrelevant to the answer), add additional stages (e.g., a verification stage to check arithmetic), or reorder stages based on problem demands. The authors acknowledge this implicitly by noting that the summary stage generates "exactly one response" and retracing is not applied there (Appendix D).

The consequence. The fixed structure creates inefficiencies and potential failure modes. For simple visual questions where the answer is immediately obvious from the image ("What color is the car?"), the model still generates a summary, caption, reasoning, and conclusion—four stages where one would suffice. This wastes inference compute and, more importantly, may introduce errors: a caption that hallucinates an irrelevant detail could mislead the reasoning stage even when the answer was obvious from the question alone. The ablation showing that shuffled stage orders destroy performance (Appendix F, Table 7: 58.2 vs. 63.1) demonstrates that the fixed order is essential, but also implies that the model cannot dynamically adapt its reasoning structure to the problem—it must follow the trained sequence regardless of whether a different structure would be more appropriate.

For problems requiring iterative refinement (e.g., geometric proofs that require trying multiple constructions, or compositional reasoning where intermediate conclusions need verification before proceeding), the linear four-stage structure is fundamentally mismatched. The model cannot loop back from reasoning to caption, or from conclusion to reasoning, within a single forward pass—the structure is strictly feedforward. SWIRES partially addresses this through retracing, but retracing is a test-time search mechanism external to the model's generation, not an internal capability to recognize and correct its own errors mid-generation. The model itself never learns to think "my reasoning suggests the caption is wrong; let me re-examine the image."

What evidence exists in the paper. The structured format is defined in Section 3.1.1 with fixed stages and fixed ordering. The shuffle ablation (Appendix F, Table 7) confirms the importance of the specific order but also shows that alternative orders are not learned—the model performance collapses when trained on non-standard orders, indicating it cannot flexibly deploy different structures for different problems. The paper reports that the model autonomously produces the four tags "in a single inference pass" and the authors "have not observed any instances where the model fails to adhere to the designated stage format" (Section 3.1.1). This perfect format adherence could be interpreted as robustness or as rigidity—the model always follows the same structure even when it's suboptimal.

Mitigation status. The paper does not discuss structural flexibility or adaptive stage selection. The authors do not propose mechanisms for the model to determine how many stages a problem needs, to skip stages, or to insert problem-specific stages. The fixed structure is presented as a feature (ensuring systematic reasoning) rather than a limitation, but the lack of adaptability means the approach cannot handle the full diversity of visual reasoning problems with the optimal reasoning structure for each.

---

6.6 No Evidence That the Intermediate Reasoning Stages Are Actually Correct

The assumption or constraint. The paper evaluates LLaVA-CoT exclusively on end-to-end answer accuracy—whether the final conclusion matches the ground-truth answer. This is the standard evaluation paradigm for VQA benchmarks and is used consistently across all experiments (Tables 2, 3, 4, 5). However, LLaVA-CoT's key innovation is structured intermediate reasoning—the summary, caption, and reasoning stages are presented as the mechanism by which the model achieves better final answers. The paper assumes that improved final-answer accuracy implies improved intermediate reasoning quality, but never directly evaluates whether the summaries, captions, or reasoning chains are factually accurate, logically valid, or faithful to the model's actual decision process.

The consequence. Without intermediate evaluation, alternative explanations for the accuracy gains cannot be ruled out. The structured format could be improving final-answer accuracy through mechanisms unrelated to reasoning quality: (1) the tags might simply cause the model to allocate more tokens to each problem, effectively increasing test-time compute in a way that benefits accuracy independent of content; (2) the structured training might teach the model to better calibrate its confidence or to format answers more consistently with the expected benchmark format, improving grader matching without improving underlying reasoning; (3) the model might learn to produce plausible-sounding intermediate stages that are post-hoc rationalizations of an answer it would have produced anyway, rather than stages that causally determine the answer (the "unfaithful explanation" problem documented in [53, 29]).

The hallucination risk is particularly acute for the caption stage. If the model's caption contains visual hallucinations—describing objects or relationships not present in the image—and the subsequent reasoning relies on these hallucinations, the final answer may be correct by coincidence or incorrect due to the hallucination. Without evaluating caption accuracy independently, we cannot distinguish between a model that genuinely uses accurate visual perception to inform reasoning from a model that guesses the answer correctly despite faulty perception. The failure cases noted in Appendix J—where "the input image may be overly complex and exceed the model's visual understanding capabilities"—suggest that perceptual failures do occur, but their frequency and impact on intermediate stages are unmeasured.

What evidence exists in the paper. All evaluation metrics are final-answer accuracy. The paper provides qualitative examples of generated reasoning traces (Figures 2, 7, 8; Appendix A, H) that show plausible-looking intermediate stages, but these are cherry-picked success cases. No quantitative evaluation of caption accuracy, reasoning logical validity, or summary relevance is reported. The data filtering pipeline (Appendix B) verifies that GPT-4o's generated training data has correct conclusions, but does not verify the accuracy of the intermediate stages in the training data or in the model's outputs. The skill-level analysis in Table 3 shows that gains are concentrated in reasoning-intensive areas, which is consistent with improved reasoning but could also reflect improved test-taking strategies (better answer formatting, elimination of obviously wrong options) that manifest disproportionately on reasoning questions.

Mitigation status. The paper does not address the faithfulness or accuracy of intermediate reasoning stages. This is a common limitation in the CoT literature—evaluating reasoning chains requires human annotation or automated metrics that are themselves imperfect—but it is particularly salient here because the four-stage structure is the paper's central contribution. Without evidence that the stages are causally responsible for the improved answers (rather than epiphenomenal), the paper's theoretical framing—that structured, systematic reasoning is what improves performance—remains a plausible hypothesis rather than an empirically established mechanism. Future work on evaluating the faithfulness and accuracy of structured reasoning traces would substantially strengthen the claims, and the authors do not flag this as a limitation or a direction for future research.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a capability-building paradigm for VLM reasoning that stands in contrast to the dominant prompting-based paradigm. Prior to LLaVA-CoT, the field's approach to improving VLM reasoning treated structured thinking as something to be elicited from an already-capable model through carefully engineered prompts—Prism [45] decomposes the VLM process into perception and reasoning stages via prompting, MSG [8] enforces CoT through structured prompts, and Visual CoT [49] generates bounding boxes alongside reasoning. The implicit assumption was that models possess latent reasoning capabilities that the right prompt can unlock.

LLaVA-CoT refutes this assumption directly. When the authors apply their carefully designed four-stage prompt to the base Llama-3.2-11B-Vision-Instruct, performance does not improve at all (Appendix F, Table 7: 56.9 with structured CoT prompting vs. 56.9 without). The identical prompt applied to GPT-4o yields a 2.3-point gain (71.8 → 74.1). This asymmetry reveals that structured reasoning is not a latent capability waiting to be prompted—it is a learned skill that weaker models lack entirely and must acquire through training. The methodological shift is from "how do we design prompts that make models reason better?" to "how do we train models to have structured reasoning as an intrinsic behavior?"

This reframing has practical consequences for how the field allocates research effort. If prompting alone cannot teach structured reasoning to models that lack it, then the path to better reasoning VLMs lies in training data engineering and curriculum design rather than in prompt optimization. The paper shows that 100k carefully constructed structured examples, combined with 3 epochs of full-parameter fine-tuning on a single 8-GPU node, can lift an 11B model from worst-in-class (56.9) to competitive with far larger models (66.3 with scaling, surpassing Llama-3.2-90B-Vision-Instruct at 62.3). The efficiency of this intervention—100k examples vs. the billions of tokens used in pretraining—implies that reasoning methodology is a high-leverage capability where small investments in teaching how to think yield disproportionate returns compared to scaling parameters or data volume.

The paper also reconciles a tension between prompting-based and training-based CoT results. Prior work showed that CoT prompting improves LLM reasoning [56] but struggled to demonstrate consistent gains on VLMs. The authors' finding that structured CoT prompting helps GPT-4o but not Llama-3.2-11B-Vision-Instruct explains this discrepancy: the effectiveness of prompted CoT depends on the base model's pre-existing reasoning capability. Strong models benefit from structured prompts; weak models do not. Training bridges this gap, enabling weak models to acquire the capability that strong models already possess. This suggests that the mixed results in the VLM CoT literature reflect heterogeneity in base model strength rather than inherent limitations of CoT for visual reasoning.

On the test-time compute side, the paper introduces stage-level error attribution as a new dimension in the test-time scaling design space. The existing framework [51] optimizes over search algorithm choice (best-of-N vs. beam search) and sequential-vs-parallel sampling ratios, but always operates at the granularity of complete solution attempts or fixed token windows. SWIRES demonstrates that aligning search granularity with semantic reasoning stages—and adding a retracing mechanism that attributes poor current-stage outputs to likely upstream errors—yields continued scaling in a regime where coarser methods plateau (Figure 5: SWIRES reaches 62.5% at ~15,000 seconds while best-of-N plateaus at ~60.1%). This suggests that test-time compute scaling for reasoning is bottlenecked not by search algorithm sophistication but by error attribution granularity—the ability to identify which part of a reasoning chain is wrong and target correction there, rather than generating more variations of fundamentally flawed chains.

The paper also provides the first evidence in the VLM literature that stage-level beam search with retracing outperforms best-of-N at matched compute budgets. On MMStar, stage-wise beam search achieves ~60.8% at ~5,000 seconds, which best-of-N requires ~10,000 seconds to approach—a 2× efficiency improvement. SWIRES extends this to ~62.5% at ~15,000 seconds, beyond where both baselines plateau. This establishes that finer-grained search with error correction is not just a theoretical improvement but yields practical gains, and it opens the door to more sophisticated stage-level search strategies (MCTS over reasoning stages, learned roll-out policies for retracing decisions).

Research directions that become more attractive after this work:

• Structured training data engineering for reasoning: The LLaVA-CoT-100k dataset construction pipeline (GPT-4o generation with staged prompts + format verification + answer consistency filtering) provides a template that can be extended to new domains, harder problems, or more diverse reasoning structures.
• Learned retracing policies: The fixed Z=0.2533 threshold is a heuristic. Learning when to retrace—potentially through RL or through training a separate retracing decision model—could improve efficiency by retracing only when it's likely to help.
• Reward model quality for stage-level evaluation: The paper uses a single general-purpose reward model (IXC-2.5-Reward) across all stages. Training stage-specific reward models—one for captions that evaluates visual accuracy, one for reasoning that evaluates logical validity—could substantially improve search and retracing decisions.
• Co-design of output format and search granularity: The tags that LLaVA-CoT learns to emit serendipitously provide search boundaries. Future models could be explicitly designed with search-friendly output formats—stage markers, confidence indicators, or explicit dependency annotations—to enable more effective test-time scaling.

Research directions that become less attractive:

• Prompting-only approaches for teaching structured reasoning to weak VLMs: The paper's negative result—zero improvement from structured CoT prompting on the base Llama model (Appendix F, Table 7)—suggests that prompt engineering alone is insufficient for models lacking baseline reasoning capability. Resources are better spent on training data construction.
• Coarse-grained best-of-N as a default test-time scaling strategy: The plateau and decline of best-of-N in Figure 5 (60.5% → 60.1% from ~10,000 to ~15,000 seconds) and the consistent underperformance relative to stage-level methods suggest that best-of-N is fundamentally limited for multi-step reasoning tasks, and finer-grained search should be the default for future work.
• Scaling only model size for reasoning improvements: The paper shows an 11B model with structured training matching or exceeding 90B and 40B models on reasoning benchmarks (Table 5). While size still matters for the hardest problems, the marginal return on additional parameters for reasoning—given equivalent training data investment—appears lower than previously assumed.

Follow-Up Research This Work Enables

1. Stage-level accuracy evaluation of intermediate reasoning. The paper's central claim—that structured stages improve reasoning—is supported only by end-to-end answer accuracy, not by direct evaluation of the summaries, captions, or reasoning chains. A critical follow-up would annotate LLaVA-CoT's intermediate outputs for factual accuracy and logical validity on a subset of benchmark questions (e.g., 200 samples from MMStar and MathVista). For captions: does the model correctly identify all task-relevant visual elements, and does it avoid hallucinating objects or relationships not present in the image? For reasoning: are the logical steps valid, do they follow from the caption's observations, and do they lead to the stated conclusion? For summaries: does the plan accurately characterize the problem and propose an appropriate solution strategy? This annotation would distinguish between the hypothesis that structured training improves reasoning quality (more accurate captions, more valid logical chains) and the alternative that structured training improves answer formatting or grader alignment without improving intermediate correctness. If captions are frequently inaccurate but final answers are correct, that would suggest the model is not genuinely using the caption stage for perception—it would be producing post-hoc rationalizations rather than causally effective intermediate reasoning.

2. Cross-model-family structured training to test boundary conditions. The paper validates LLaVA-CoT on a single base model (Llama-3.2-11B-Vision-Instruct) that is explicitly the weakest in its class. A direct follow-up would apply the same LLaVA-CoT-100k dataset and SFT recipe to Qwen2-VL-7B and InternVL2-8B—models that already reason well (65.9 and 64.0 on reasoning benchmarks, respectively, vs. LLaVA-CoT's 66.3 with scaling). This experiment would answer: does structured training provide ceiling-bounded benefits (larger gains for weaker base models, smaller gains for stronger ones), or does it provide consistent additive improvements regardless of baseline? If Qwen2-VL-7B with structured training achieves, say, 69.0, that would suggest structured reasoning is a general capability that benefits all models. If it achieves 66.5 (marginal gain), that would suggest the benefits saturate once a model already reasons competently, and the paper's large gains are partly an artifact of the unusually weak base model. This experiment would also test whether the structured format transfers across model families or whether it interacts with architecture-specific inductive biases.

3. Adaptive difficulty-based stage selection and retracing depth. The paper applies the same four-stage structure and the same retracing hyperparameters (M=4, N=2, C=3, Z=0.2533) to all problems regardless of difficulty. The test-time scaling literature [51] establishes that optimal strategy depends on problem difficulty. A natural extension would train a lightweight difficulty classifier (perhaps a linear probe on the base model's hidden states at the question-encoding stage) that predicts whether a problem is easy, medium, or hard, and then vary the reasoning structure and retracing budget accordingly: easy problems might skip the caption stage (or use only 1 candidate) and disable retracing; medium problems use the full four-stage structure with moderate retracing (C=2); hard problems use expanded beam width (M=8) and deeper retracing (C=5). The classifier could be trained on the pass@1 rate from the base model, following the oracle difficulty estimation approach in the test-time scaling literature. The key measurement would be whether adaptive allocation improves the efficiency-accuracy Pareto frontier relative to uniform application of the same structure and budget to all problems. This would directly connect the LLaVA-CoT structured reasoning framework to the compute-optimal test-time scaling paradigm.

4. Retracing policy learning via reinforcement learning or offline optimization. The current SWIRES threshold (Z=0.2533, calibrated on MMStar reward score statistics) is a heuristic with no optimality guarantees. A more principled approach would learn a retracing policy: given the current stage's candidate scores, the previous stage's candidate scores, and problem features (question length, number of visual elements, benchmark category), should the system retrace or proceed? This could be framed as a contextual bandit problem where the action is {retrace, proceed}, the state is the reward score distribution and problem metadata, and the reward is whether the final answer is correct. Training data could be collected by running SWIRES with random retracing decisions on a large set of training problems and recording outcomes. A learned policy could discover patterns that the heuristic misses—for instance, retracing might be more valuable on geometry problems (where caption errors about spatial relationships are recoverable) than on chart interpretation problems (where misreading a number is often uncorrectable regardless of retracing). The baseline comparison would be the fixed-threshold policy vs. the learned policy, evaluated on held-out benchmarks. If the learned policy achieves higher accuracy at the same average compute budget, it would demonstrate that retracing decisions can be optimized beyond heuristic calibration.

5. Extension to domains without clean verifiability signals. The paper evaluates exclusively on benchmarks with ground-truth answers that can be checked for exact match. Many important VLM applications—open-ended visual dialog, image-grounded creative writing, visual explanation generation—lack such clean correctness signals, making the reward model's role even more critical and more error-prone. A stress-test follow-up would apply LLaVA-CoT to open-ended visual reasoning tasks such as Visual Storytelling, Visual Commonsense Reasoning (VCR) with justification generation, or instruction-following in visual environments (e.g., ALFRED). The key question is whether the structured reasoning format provides benefits when the "correctness" of intermediate stages cannot be verified against ground truth—the reward model must judge caption quality and reasoning validity based on learned preferences rather than answer matching. If structured training degrades performance on open-ended tasks (because the rigid format restricts creative or context-appropriate responses), that would establish an important boundary condition: structured reasoning helps for verifiable, closed-form problems but hurts for open-ended generation. If it helps regardless, it suggests the structure provides general metacognitive benefits beyond answer-verification tasks.

6. Combining structured training with reinforcement learning for self-improvement. The paper uses SFT on GPT-4o-generated structured data, which means the model learns to imitate a teacher's reasoning patterns. The authors note that "future research could explore the application of reinforcement learning to further improve complex multimodal reasoning" (Section 7). A concrete follow-up would implement a ReST-style self-improvement loop: (1) use the SFT-trained LLaVA-CoT to generate structured reasoning traces on a large set of unlabeled visual questions, (2) filter for traces where the final answer is correct (verified against ground truth or a strong verifier model), (3) fine-tune the model on these self-generated successful traces, (4) repeat. This would test whether the model can improve beyond its teacher (GPT-4o) by learning from its own successful reasoning patterns. The key measurement would be whether iteration 2 outperforms iteration 1 on held-out benchmarks, and whether the improvement comes from better intermediate reasoning (more accurate captions, more valid logical steps) or simply from exposure to more training data. The failure mode to watch for is the "correct-to-incorrect reversion" problem documented in the revision model literature [51]—if the model learns to produce reasoning traces that look structured but are substantively hollow (e.g., captions that are generic rather than task-specific), self-improvement could backfire.

Practical Applications and Downstream Use Cases

1. Structured training for on-device reasoning VLMs. The paper demonstrates that an 11B model with structured fine-tuning matches a 90B model on reasoning benchmarks (Table 5: LLaVA-CoT w/ scaling at 66.3 vs. Llama-3.2-90B-Vision-Instruct at 62.3). For on-device deployment—where model size is constrained by memory and compute, and cloud offloading is undesirable for latency or privacy reasons—this represents a concrete recipe: take a small base VLM (8B–11B parameters), fine-tune on 100k structured reasoning examples, and deploy with stage-wise beam search (without retracing, if latency is critical). The 5.8-point training gain on the base model (56.9 → 62.4) combined with the additional 3.1-point gain from scaling (62.4 → 65.5) yields a model that provides competitive reasoning on-device without requiring cloud connectivity. The primary cost is dataset construction (GPT-4o API calls for 100k examples) and fine-tuning compute (8× H100 GPUs for 3 epochs), both of which are one-time investments amortized over all deployed devices.

2. Test-time compute allocation for batch VLM evaluation. Organizations running large-scale VLM evaluation—grading student answers to visual science problems, processing insurance claims with photo evidence, or auditing VLM outputs for compliance—can use the stage-wise search and retracing framework to allocate compute adaptively per query. Instead of applying uniform best-of-N to every query, the system can: (1) generate the summary and caption stages (cheap, since the summary is single-candidate and the caption stage with M=4 is moderate cost), (2) check the reward model's score on the caption candidates, (3) if caption scores are high, proceed with minimal reasoning budget (N=1 or 2 reasonings, no retracing); if caption scores are borderline, allocate additional budget (more reasoning candidates, enable retracing). The paper's Figure 5 shows that stage-wise beam search at ~5,000 seconds achieves comparable accuracy to best-of-N at ~10,000 seconds—a 2× efficiency gain that directly translates to cost savings in batch processing. The paper's fixed hyperparameters would need per-domain calibration, but the overall framework of stage-granularity compute allocation is immediately applicable.

3. Data generation for VLM self-improvement pipelines. When using VLMs to generate training data for themselves—a common paradigm in the LLM literature (STaR, ReST) that is beginning to transfer to VLMs—the quality of generated reasoning traces determines the quality of the resulting model. LLaVA-CoT's structured output format and SWIRES search provide a quality-control mechanism for self-generated data: run SWIRES with a moderate retracing budget (C=2 or 3) on each training question, and only include traces where the reward model's stage-level scores exceed thresholds at all stages (not just the final conclusion). This filters for examples where the model was confident in its perception (high caption scores), its reasoning (high reasoning scores), and its answer (high conclusion score), rejecting traces where the model was uncertain or produced low-quality intermediate stages. The structured format makes this filtering more granular than simple answer-correctness filtering—a trace could have a correct final answer but a hallucinated caption, and the stage-level scores would flag it as low-quality even though end-to-end accuracy would accept it. This could improve the signal-to-noise ratio in self-generated training data, addressing a known failure mode in self-improvement loops.