SIM-CoT: Supervised Implicit Chain-of-Thought

ArXiv: 2509.20317

🎯 Pitch

Implicit chain-of-thought training collapses when you give it more room to think—latent states lose their diversity and all reasoning steps blur together. SIM-CoT fixes this with step-level supervision during training, boosting accuracy by over 8% on small models and letting them beat explicit CoT with 2.3× fewer tokens.

1. Executive Summary

This paper introduces SIM-CoT (Supervised Implicit Chain-of-Thought), a plug-and-play training module that addresses a core latent instability issue in implicit CoT methods — where scaling the number of reasoning tokens causes latent representations to collapse into homogeneous states, losing operator information and semantic diversity. Working on the GSM8K-Aug math reasoning dataset with GPT-2 and LLaMA model families, SIM-CoT introduces step-level supervision through an auxiliary decoder that aligns each implicit latent token with its corresponding explicit reasoning step during training (e.g., mapping latent₁ to "0.3×120=36", latent₂ to "120−36=84"), then discards the decoder at inference to preserve efficiency. The method boosts Coconut by +8.2% on GPT-2 and CODI by +3.0% on LLaMA-3.1 8B, surpasses the explicit CoT baseline by 2.1% on GPT-2 with 2.3× greater token efficiency, and remains stable at 8–16 implicit tokens where prior methods collapse — establishing that step-level supervision can bridge the implicit-explicit performance gap while maintaining stability, though the approach's reliance on explicit step annotations during training means its benefits are tied to domains where such structured reasoning traces are available.

2. Context and Motivation

The Core Problem: Implicit CoT Methods Are Token-Efficient but Unstable

The fundamental tension this paper tackles is one that has quietly frustrated the chain-of-thought reasoning literature: explicit reasoning chains work well but are expensive, while implicit (continuous) reasoning chains are cheap but fall apart when you try to scale them. This is not a trivial performance gap — it reflects a genuine instability in how latent representations behave when they are asked to carry multi-step reasoning without step-by-step textual guidance.

The practical stakes are clear. Explicit chain-of-thought prompting (Wei et al., 2022) and its trained variants require models to verbalize every intermediate calculation, equation, or logical deduction as discrete text tokens before producing a final answer. For a math problem requiring four reasoning steps, this might mean generating 40–100 tokens of intermediate text before the answer appears. This verbosity has two painful consequences: (1) inference cost scales with reasoning chain length, making deployment expensive for complex problems, and (2) verbalization from a fixed vocabulary forces the model to commit to a single reasoning path, precluding the exploration of alternative solution strategies or the encoding of concepts that don't have clean textual representations (Li et al., 2025; Zhang et al., 2025b).

Implicit CoT methods attempt to solve both problems simultaneously. Instead of generating a sequence of discrete text tokens for reasoning, they operate in the model's continuous hidden state space — inserting a fixed number of "latent tokens" (dense vectors) that carry forward the intermediate computational state. Because these latent vectors can in principle encode richer, more compressed information than individual text tokens, they offer the promise of fast, token-efficient reasoning that preserves the multi-step structure of explicit CoT without the verbal overhead.

The problem — and this is the core gap the paper identifies — is that this promise breaks down when you try to scale the latent reasoning budget. As the authors demonstrate in Figure 1(a), increasing the number of implicit tokens from the default three (established by Coconut, Hao et al., 2025) to five causes training to become unstable and sometimes collapse entirely, with accuracy plummeting to 12.5%. This is not a gradual degradation; it is a phase transition from functioning reasoning to complete failure. The significance of this finding extends beyond any single method: if implicit reasoning cannot scale to handle longer chains, it fundamentally cannot replace explicit CoT for complex problems, regardless of how efficient it is on simple ones.

Why This Problem Matters: More Than Just Math

The latent instability issue has implications that reach well beyond the GSM8K math benchmark. First, it touches on a fundamental question about representation learning in transformers: can a model learn to maintain distinct, semantically meaningful representations across multiple steps of internal computation without explicit step-by-step textual grounding? The paper's analysis suggests the answer is "not without help" — without sufficient supervision, the latent space collapses toward homogeneity, losing the diversity needed to encode distinct reasoning operations (numbers, operators, subgoals).

Second, the problem is architecturally universal. The instability occurs in autoregressive latent reasoning methods (the most promising class of implicit CoT), which work by having the model generate its own hidden states as inputs for subsequent computation steps. This is the approach taken by Coconut (Hao et al., 2025), CODI (Shen et al., 2025b), and SoftCoT (Xu et al., 2025). All of these methods face the same underlying challenge: the model must learn to produce latent vectors that are simultaneously useful for downstream computation and sufficiently distinct from each other to avoid redundancy. When this balance fails, the entire reasoning chain degrades.

Third, the practical deployment calculus hinges on this stability question. If implicit reasoning only works reliably for 2–3 step problems but collapses on 4–5 steps, then it cannot serve as a general replacement for explicit CoT in production systems where problem difficulty varies. The instability creates an uncomfortable trade-off: deploy explicit CoT and pay the token cost, or deploy implicit CoT and risk catastrophic failures on harder problems. SIM-CoT's intervention — showing that step-level supervision stabilizes the latent space — provides a path out of this dilemma.

Prior Approaches and Where They Fall Short

The paper situates itself relative to three generations of implicit reasoning work, each with a specific supervision strategy that the analysis shows to be insufficient:

Answer-level supervision (Coconut). Coconut (Hao et al., 2025) pioneered the autoregressive latent reasoning paradigm that SIM-CoT inherits. The model replaces explicit reasoning steps with a fixed number of continuous latent tokens, training end-to-end with only the final answer as a supervisory signal. The latent tokens themselves receive no direct guidance — they are optimized solely through the gradient flowing back from answer correctness. This is elegantly simple but, as Figure 1(b–d) reveals, critically underconstrained. When the number of latent tokens grows, the model faces a credit assignment problem: it must learn to distribute distinct reasoning operations across an increasing number of latent positions, with no signal about which latent should encode which operation. The result is the collapse documented in Figure 1: latent tokens become homogeneous (overlapping in semantics), lose operator information (decoding primarily to numbers), and drift away from the vocabulary embedding space that anchors them to meaningful token-level semantics.

Trajectory-level supervision (CODI). CODI (Shen et al., 2025b) improves on Coconut by adding a distillation objective: the full sequence of implicit latent tokens is aligned with the last hidden state of an explicit CoT trajectory, providing a coarse trajectory-level signal. This is a step in the right direction — it tells the model that the latent chain as a whole should resemble the information content of an explicit reasoning trace. However, it remains coarse-grained: the model still receives no per-step guidance about which latent corresponds to which reasoning substep. The alignment is between two aggregate representations, not between individual latent steps and individual reasoning steps. This limits the granularity of supervision and means that within the latent chain, individual tokens can still drift toward homogeneity as the chain lengthens. The paper demonstrates (Table 1, Table 2) that CODI indeed performs better than Coconut, but the gap to explicit CoT remains — and the latent instability issue is not fundamentally resolved, merely pushed to higher latent counts.

Training-free latent construction (Soft Thinking). A parallel line of work (Zhang et al., 2025b; Wu et al., 2025) constructs latent tokens on-the-fly at inference time by taking a probability-weighted mixture of vocabulary embeddings, without any training. This is computationally appealing — no fine-tuning required — but has an inherent limitation: the latent tokens are effectively interpolations in the existing vocabulary space, meaning they can only represent concepts that are already expressible as mixtures of existing tokens. They lack the capacity to learn entirely new, task-optimized latent representations that could encode reasoning operations more efficiently than text. The paper treats soft thinking as complementary rather than competing (Appendix C shows that adding soft thinking on top of SIM-CoT provides small additional gains), but the fundamental limitation remains: without learning, the latent space is bounded by the pretrained embedding geometry.

What all prior approaches share is insufficient per-step supervision. Whether it's answer-level (Coconut), trajectory-level (CODI), or no training at all (Soft Thinking), none of these methods tells the model explicitly: "latent number 3 should encode the multiplication step, and latent number 4 should encode the subtraction step." This is the gap that SIM-CoT fills — and the paper's central diagnostic contribution is showing that this gap is not just a performance optimization but a stability requirement for scaling the latent reasoning chain.

How the Paper Positions Itself

SIM-CoT is presented as a plug-and-play supervision module, not a new reasoning architecture. This is a deliberate framing choice with specific implications. The method does not alter the autoregressive latent generation mechanism (Equation 1), the explicit answer decoding (Equation 3–4), or the inference procedure. It adds precisely one thing: during training only, each latent token is fed into an auxiliary decoder that generates the corresponding natural language reasoning step, and the decoder's cross-entropy loss (Equation 6) provides a per-step gradient signal back to the latent representations. At inference, the decoder is removed entirely — the forward pass is identical to Coconut.

This positioning accomplishes several things:

It isolates the contribution to supervision granularity. By keeping the architecture identical to Coconut and adding only a training-time auxiliary loss, the paper can attribute all observed improvements (stability, accuracy, semantic diversity) specifically to the step-level supervision mechanism, not to any architectural innovation. This makes the experimental comparisons unusually clean: the only difference between Coconut and SIM-CoT+Coconut is the presence of the decoder and its loss.

It emphasizes compatibility over novelty of mechanism. The auxiliary decoder is architecturally identical to the base LLM — there is no new network design, no novel attention pattern, no exotic training objective. The innovation is entirely in what gets supervised (individual latent steps) and how that supervision is structured (one decoder per latent, with shared embeddings). The paper explicitly tests this compatibility by applying SIM-CoT on top of both Coconut and CODI (Tables 1, 2), showing gains in both settings.

It reframes the implicit CoT problem as a supervision design problem, not an architecture problem. Prior work implicitly treated latent instability as a scaling challenge that might require architectural fixes — different recurrence patterns, longer training, or alternative latent injection methods. SIM-CoT's results suggest a simpler diagnosis: the latent space collapses because it lacks per-step grounding, and providing that grounding through an auxiliary decoder is sufficient to stabilize it. This shifts the research question from "how do we design a better implicit reasoning architecture?" to "how do we structure supervision to maintain representational diversity across latent steps?" — a fundamentally different and potentially more tractable direction.

It provides an interpretability benefit as a byproduct. Because the auxiliary decoder maps each latent to explicit text during training, it can be reused at inference time to visualize what each latent step has encoded (Figure 4, Appendix I). This is not the primary contribution — the paper is clear that performance and stability are the goals — but it addresses a long-standing criticism of implicit methods: that their reasoning process is opaque and unverifiable. The decoded latent steps (e.g., "0.3 × 120 = 36", "120 − 36 = 84") provide a window into the model's implicit computation, enabling error diagnosis and building trust in the reasoning process.

The paper also positions itself carefully relative to the explicit CoT baseline. Unlike some prior implicit CoT work that implicitly treats explicit CoT as the gold standard to be matched, SIM-CoT's framing is more nuanced. The paper shows that on GPT-2, SIM-CoT actually surpasses the explicit CoT supervised fine-tuning baseline by 2.1% (Table 1) with 2.3× fewer tokens — suggesting that well-supervised implicit reasoning can be not just more efficient but potentially more accurate than explicit verbalization, perhaps because it avoids the constraint of forcing reasoning into discrete token sequences. On larger models (LLaMA 8B), the gap narrows but does not entirely close (Table 3), which the paper presents honestly without overclaiming.

The Diagnostic Analysis as a Contribution in Its Own Right

Section 2 of the paper deserves special attention because it constitutes a systematic failure analysis of implicit CoT that was previously absent from the literature. Prior papers had observed that implicit methods underperform explicit CoT, but the mechanism of that underperformance — the specific ways in which latent representations degrade — had not been characterized. The paper's four-part diagnostic (Figure 1) provides this characterization:

Latent instability (Figure 1a): The training dynamics themselves become erratic as latent count increases, with accuracy fluctuating wildly rather than smoothly improving or plateauing. This suggests an optimization problem, not just a capacity problem.
Information loss (Figure 1b): By separately measuring accuracy on numbers, operators, and final answers, the paper shows that operator information is lost first and most severely — the model can still extract the relevant numbers from the problem but cannot represent the operations that combine them. This is a specific, interpretable failure mode that points toward the need for operator-level supervision.
Shifted distance (Figure 1c): The geometric analysis quantifies two simultaneous failures: a reduction in inter-latent distance (the latent vectors become too similar to each other, losing distinctiveness) and an increase in distance to the vocabulary center (the latents drift out of the semantic space that anchors them to interpretable concepts). This is a clean, measurable signature of collapse that the paper uses as a diagnostic metric throughout (Table 4b, Figure 6).
Semantic homogenization (Figure 1d): Qualitative decoding of latent tokens shows that in failed models, distinct latent positions decode to nearly identical content — mostly numbers, with no operators or logical connectives. This directly confirms that the collapse is not just a geometric artifact but represents a genuine loss of semantic diversity.

This diagnostic framework is significant because it provides measurable, falsifiable predictions about what good implicit reasoning should look like: high inter-latent distances, moderate distances to the vocabulary center, and semantically diverse decoded content. SIM-CoT's effectiveness is then evaluated not just on accuracy but on whether it restores these healthy geometric properties (as Table 4b and Figure 6 show it does).

In summary, the paper addresses a specific gap — latent instability in scaling implicit CoT — that matters because it is the primary barrier to implicit reasoning serving as a general replacement for explicit CoT. Prior work approached the problem through coarser supervision (answer-level, trajectory-level) or no training at all, all of which proved insufficient to maintain representational diversity across a growing latent chain. SIM-CoT positions step-level supervision as the missing ingredient, providing a plug-and-play module that stabilizes the latent space without architectural modification, while simultaneously enabling the visualization and diagnosis of implicit reasoning steps.

3. Technical Approach

3.1 Reader Orientation

This is primarily a method paper that introduces a training-time supervision module, not a new model architecture. The system being built is a pipeline that trains a language model to perform multi-step mathematical reasoning in a continuous latent space, where each latent step is explicitly supervised to correspond to a specific reasoning operation (e.g., "multiply these two numbers," "subtract this from that") through an auxiliary decoder that is discarded at inference time. The core problem it solves is the latent instability that causes implicit chain-of-thought methods to collapse when the number of reasoning steps increases — the latent representations lose their semantic diversity, becoming homogeneous vectors that encode only numbers and drift away from the vocabulary space, making complex reasoning impossible. The solution's "shape" is deceptively simple: instead of supervising only the final answer (as Coconut does) or the entire trajectory as a blob (as CODI does), provide per-step textual grounding signals during training so that each latent learns to encode distinct, meaningful reasoning content, then remove the grounding machinery at inference to preserve the efficiency of implicit reasoning.

3.2 Big-Picture Architecture (Diagram in Words)

The SIM-CoT system has four major components, three of which participate in training and two in inference:

Base Language Model (F_θ) — This is the pretrained transformer (GPT-2, LLaMA 1B/3B/8B) that forms the backbone of the system. It operates in two modes: in implicit mode, it processes the question tokens followed by a sequence of continuous latent vectors z₁, z₂, …, z_K that it generates autoregressively from its own hidden states; in explicit mode, it switches back to standard autoregressive token decoding to produce the final answer. The base model's parameters θ are updated during training by both answer-level and step-level loss signals.
Latent Construction Mechanism — This is not a separate network but a specific operational procedure executed by the base model: after processing the input question, the model's last-layer hidden state at the final position is taken as the first implicit latent z₁, which is then concatenated to the sequence as if it were a token embedding; the model re-processes the augmented sequence, and the new last-layer hidden state becomes z₂, and so on for K steps. This autoregressive latent generation is what distinguishes the implicit CoT paradigm: reasoning proceeds by repeatedly feeding the model's own compressed representations back as inputs, creating a recurrent computation loop in the depth dimension.
Auxiliary Decoder (p_φ) — This is architecturally identical to the base LLM (same transformer configuration) but serves a single specialized purpose during training only. For each latent step k, the decoder takes ONLY the single latent vector z_k as a conditioning prefix (not the full sequence history) and autoregressively generates the corresponding textual reasoning step s_k (e.g., "12 × 3 = 36"). The decoder's cross-entropy loss on these step tokens provides the per-step gradient signal that flows back through z_k into the base model, teaching it to encode step-specific semantic content. The decoder shares the base model's token embedding matrix E but has its own transformer parameters φ and its own output projection W_dec. Crucially, the decoder is removed entirely at inference — it exists only to provide training-time supervision.
Answer Decoding Head — This is the standard language model head W_o already present in the base model. After the K implicit latents are constructed, the model switches to explicit mode and uses this head to autoregressively decode the final answer tokens a_1, a_2, …, a_{L_a}. The answer generation is conditioned on the full prefix: question tokens + all K implicit latents.

The information flow during training is: question tokens → autoregressive generation of z₁ → autoregressive generation of z₂ (conditioned on question + z₁) → … → autoregressive generation of z_K (conditioned on question + z₁, …, z_{K-1}) → autoregressive generation of answer a (conditioned on everything). In parallel, each z_k is independently fed into the auxiliary decoder to generate s_k, with no cross-step conditioning in the decoder. Two loss terms are computed: L_ans-lm from the answer language modeling objective, and L_step from the decoder's step-generation cross-entropy. Gradients from both flow back into z₁, …, z_K and into the base model. At inference, the decoder and L_step are absent; only the base model forward pass and answer decoding occur.

3.3 Roadmap for the Deep Dive

I will explain SIM-CoT's technical approach in the following order, building from notation to mechanism to objective:

First, the notation and formalism (Section 3.1): I will define the vocabulary V, the embedding matrix E, the question x, the reasoning steps s_k, the answer a, the base model F_θ, and the decoder p_φ. This vocabulary is necessary because every subsequent equation depends on it, and the paper's notational conventions (e.g., H_θ(U) for last-layer hidden state, ⊕ for time-axis concatenation) are used throughout.
Second, the implicit phase (Section 3.2): I will explain Equation 1, which defines how latent tokens are constructed autoregressively from hidden states. This is the core mechanism that SIM-CoT inherits from Coconut and that distinguishes implicit from explicit reasoning — it is the "engine" that generates the latent chain, and understanding it is prerequisite to understanding what the step-level supervision is supervising.
Third, the explicit phase and answer decoding (Section 3.3): I will explain Equations 2–4, which describe how the model switches back from continuous latents to discrete token decoding once the implicit reasoning chain is complete. This establishes what happens at inference time and what the answer-level loss term optimizes.
Fourth, the training-time decoder and step-level supervision (Section 3.4): This is the novel contribution — I will explain Equation 5 and its parameterization, covering how the decoder conditions on each individual z_k (not the full history), how it injects z_k as a prefix that initializes its hidden state, how it generates step tokens autoregressively, and how the step-level cross-entropy loss is computed over only the textual tokens (not the latent prefix). I will also explain the critical design choice of using one latent per reasoning step (each latent decodes to exactly one explicit step) and why no cross-step context is provided to the decoder.
Fifth, the combined training objective (Section 3.5): I will explain Equations 6–8, which combine the step-level decoder loss and the answer-level language modeling loss into a weighted sum, and detail how gradients from each term propagate through the network to shape the latent representations and the base model parameters.
Sixth, the curriculum and implementation details: I will cover how K (the number of implicit steps) is gradually increased during training following a schedule adapted from Coconut, the inference procedure (exactly what the forward pass looks like without the decoder), and the key hyperparameter choices. I will also explain the training data structure — how the GSM8K-Aug dataset's structured mathematical expressions serve as the ground-truth step sequences s_k.

3.4 Detailed, Sentence-Based Technical Breakdown

The paper introduces SIM-CoT as a training-time supervision mechanism that modifies how gradients flow to the latent representations in autoregressive implicit chain-of-thought models, without changing the inference-time architecture or computational cost. The core idea is that by aligning each latent token with a specific textual reasoning step through an auxiliary decoder during training, the latent representations are forced to encode distinct, meaningful information that prevents the collapse into homogeneity observed when scaling the number of implicit tokens.

Notation and Formalism

I begin by establishing the notational framework used throughout the method, as every subsequent equation builds on these definitions.

Vocabulary and embeddings. Let V denote the vocabulary of the base language model — the set of all possible discrete tokens the model can produce or consume. Let E ∈ R^{|V| × d} be the token embedding matrix, where |V| is the vocabulary size and d is the model's hidden dimension (e.g., 768 for GPT-2, 2048 for LLaMA 1B). Each row E_v ∈ R^d is the learned embedding vector for token v. The embedding function e(·) maps a token to its embedding: e(v) = E_v. Note that the paper uses e(·) rather than subscripting E directly, which emphasizes that embeddings are vectors being fed as inputs to the transformer, not indices into a lookup table.

Input representation. A question x is a sequence of tokens (x_1, ..., x_T) where each x_t ∈ V and T is the question length. The embedded prefix — the initial input to the transformer before any reasoning has occurred — is defined as a sequence of d-dimensional vectors:

$U^{(0)} = (e(x_1), ..., e(x_T))$

where e(x_t) ∈ R^d is the embedding of token x_t. The superscript (0) indicates that this is the sequence before any latent tokens have been appended; as latent tokens are added, the superscript increments. This U^{(k)} notation (a sequence of d-dimensional vectors) is crucial because it emphasizes that the transformer operates on a mix of token embeddings and continuous latent vectors — from the model's perspective, there is no architectural difference between them; both are d-dimensional vectors fed through the same self-attention and feedforward layers.

Last-layer hidden state function. The paper introduces the notation H_θ(U) to denote a specific operation: given a prefix sequence U = (u_1, ..., u_m) of d-dimensional vectors (where each u_i may be either a token embedding or a latent vector), run the autoregressive transformer F_θ on this sequence and extract the last-layer hidden state at the final position m. Formally:

$H_θ(U) ∈ R^d$

This is the d-dimensional vector that would normally be multiplied by the output projection matrix W_o to produce logits for predicting the next token — but in the implicit phase, it is instead used directly as the next latent token. The notation H_θ(U) abstracts away all the internal transformer mechanics (attention, feedforward layers, residual connections) and exposes only the interface that matters for the method: "given this sequence, what continuous vector does the model produce at the final position?"

What this notation enables. By defining H_θ as a function from sequences-of-vectors to single-vectors, the paper can describe the autoregressive latent construction process compactly without repeatedly invoking the full transformer forward pass. It also emphasizes that the latent tokens are produced by exactly the same mechanism that produces hidden states during normal token generation — there is no separate "latent generation network." The latents are simply hidden states that are fed back as inputs rather than being projected to vocabulary logits. This is the essential insight of the autoregressive latent reasoning paradigm: the model's internal computation (hidden states) becomes the medium for carrying forward reasoning state, bypassing the need to discretize through the vocabulary.

Reasoning steps and answers for supervision. For training, each question x is paired with a sequence of K textual reasoning steps and a final answer. The k-th reasoning step is a token sequence s_k = (y_{k,1}, ..., y_{k,L_k}) where each y_{k,t} ∈ V and L_k is the length of step k in tokens. The final answer is a token sequence a = (a_1, ..., a_{L_a}). The auxiliary decoder has its own parameters φ; the base language model has parameters θ. The decoder is trained to generate s_k from only z_k (no access to the question or to other latents), while the base model is trained to generate a from the question and the full latent chain z_{1:K}.

Implicit Phase: Latent Construction by Last Hidden States

Equation 1 defines the core mechanism that generates the implicit chain-of-thought as a sequence of continuous vectors. The process operates as a recurrence over k = 1, ..., K, where K is a fixed hyperparameter chosen before training:

$z_k = H_θ(U^{(k-1)}) ∈ R^d$

$U^{(k)} = U^{(k-1)} ⊕ z_k$

where ⊕ denotes concatenation along the time axis (i.e., appending the vector z_k to the end of the sequence U^{(k-1)} to create a sequence that is one position longer), z_k is the k-th implicit latent token — a d-dimensional continuous vector, U^{(k-1)} is the sequence prefix before step k, consisting of the original question embeddings plus the first k-1 latent tokens, and H_θ(U^{(k-1)}) is the last-layer hidden state of the transformer after processing U^{(k-1)}.

What the equation computes operationally. For step k = 1: the transformer processes the embedded question tokens U^{(0)} = (e(x_1), ..., e(x_T)). The hidden state at the final position becomes z_1. This vector z_1 is then treated as if it were the embedding of a special "latent token" and appended to the sequence, producing U^{(1)} = (e(x_1), ..., e(x_T), z_1). For step k = 2: the transformer re-processes this augmented sequence U^{(1)}. The final hidden state becomes z_2, which is appended to produce U^{(2)}, and so on. After K steps, the sequence U^{(K)} contains the question embeddings plus K continuous latent vectors, and the model switches to explicit answer decoding.

The autoregressive dependency structure. This is the critical property of the construction: each z_k depends on all previous latents plus the question, because the transformer processes the entire growing prefix U^{(k-1)} to produce each new hidden state. The dependency chain is z_1 ← question, z_2 ← question + z_1, z_3 ← question + z_1 + z_2, and so on. This is why the paradigm is called "autoregressive latent reasoning": the latent tokens are generated sequentially, with each one conditioned on all previous latent tokens plus the question, exactly as text tokens are generated autoregressively in standard language modeling.

Why this form. The autoregressive construction has three essential properties. First, it allows the latent chain to represent a sequence of computational steps, where each step can build on the results of previous steps — exactly the structure needed for multi-step reasoning. If all latents were generated independently or in parallel, there would be no way to compose operations across steps. Second, it uses the model's existing autoregressive mechanism and attention patterns — no new architectural components are needed; the model already knows how to condition on prefixes of varying length. Third, it maintains the property that z_k is a single vector per step, not a sequence of vectors, which is what enables the token efficiency: K latent vectors replace potentially hundreds of explicit reasoning tokens.

The relationship between K and reasoning steps. The paper sets K to match the number of explicit reasoning steps in the training data (determined by parsing the structured mathematical expressions in GSM8K-Aug — see Section 4.1 and Figure 5). Each latent token corresponds to two implicit tokens in the sequence (a detail inherited from Coconut: "One implicit latent corresponds to two implicit tokens," Appendix E.1). This means that if K = 4, the model inserts 8 positions into the sequence — 4 latent "tokens" that each occupy 2 vector slots. The motivation for using 2 vectors per latent is empirical: Coconut found that this provides sufficient representational capacity per reasoning step without excessive sequence expansion. SIM-CoT does not alter this convention.

The shifting nature of U^{(k)} during training. As training progresses, the model's parameters θ change, which changes H_θ, which changes the latent vectors z_k, which changes the inputs U^{(k)} for subsequent steps. This creates a moving-target problem: the distribution of latent vectors that the model encounters during training is non-stationary because it depends on the model's own evolving parameters. This is a known challenge in autoregressive latent methods and one of the reasons that answer-level supervision alone (Coconut) becomes unstable — the model must simultaneously learn to produce useful latents and to interpret latents that are continuously changing. SIM-CoT's step-level supervision helps mitigate this by providing a more direct, per-step gradient signal that anchors each latent to a specific semantic target, reducing the co-adaptation problem.

Explicit Phase: Answer Decoding over the Vocabulary

After the K implicit latent steps have been constructed and appended to the sequence, the model switches from continuous latent mode to standard discrete token decoding to generate the final answer. This switch is not architectural — there is no mode flag or gating mechanism — but rather procedural: instead of taking the final hidden state as z_{K+1}, the model passes it through the language model head W_o to get a probability distribution over the vocabulary, and samples or argmaxes a token, then continues autoregressively until an end-of-sequence token is generated or a maximum length is reached.

The generation process. Let h_{T+K+t} denote the last-layer hidden state at position T+K+t — that is, after the T question tokens, the K latent tokens, and the first t-1 answer tokens have been processed. With teacher forcing during training (where the ground-truth partial answer a_{<t} is provided as input), the hidden state is:

$h_{T+K+t} = H_θ(U^{(K)} ⊕ e(a_{<t}))$

What this computes. Starting from the latent-augmented prefix U^{(K)} (which contains question embeddings and all K latent vectors), the model appends the embeddings of the ground-truth answer tokens one by one and extracts the hidden state at each position. Each h_{T+K+t} is a d-dimensional vector encoding the model's representation after seeing the question, all K latent steps, and the first t-1 answer tokens.

This hidden state is then projected to vocabulary logits via the standard output projection matrix W_o ∈ R^{|V| × d}:

$p_θ(a_t | x, z_{1:K}, a_{<t}) = \text{softmax}(W_o · h_{T+K+t})_{a_t}$

What this computes. The softmax over W_o · h_{T+K+t} produces a probability distribution over the vocabulary for the next answer token, given the entire history (question, latent chain, and preceding answer tokens). The subscript a_t indexes this distribution to extract the probability assigned to the ground-truth token. The full answer probability is the product over all answer tokens:

$p_θ(a | x, z_{1:K}) = \prod_{t=1}^{L_a} p_θ(a_t | x, z_{1:K}, a_{<t})$

Why this form. The explicit phase uses standard autoregressive language modeling with no architectural modifications — this is the same mechanism used for any text generation task. The only difference from standard generation is that the prefix U^{(K)} contains continuous latent vectors rather than discrete token embeddings, but this is transparent to the transformer since both are d-dimensional vectors processed identically by attention and feedforward layers. The key property is that the answer is conditioned on the full latent chain, meaning the model must learn to extract and use the reasoning information encoded in z_{1:K} to produce the correct answer. This is what creates the gradient path from answer correctness back to the latent representations — if z_3 encodes wrong information, the answer will be wrong, and the resulting loss will penalize the parameters that produced z_3.

During inference, exactly the same process occurs without teacher forcing: the model generates answer tokens one by one, each conditioning on previously generated tokens, until a stopping criterion is met.

Training-Time Decoder and Step-Level Supervision

This is the novel contribution of SIM-CoT — the mechanism that distinguishes it from Coconut and CODI by providing per-step, rather than per-trajectory or per-answer, supervision for the latent representations.

The decoder's role and architecture. During training, a separate decoder network p_φ is introduced. This decoder is architecturally identical to the base LLM — it has the same number of transformer layers, the same hidden dimension, the same attention mechanism — but with its own parameters φ (separate from the base model's θ). The decoder shares the base model's token embedding matrix E (so both models map the same vocabulary tokens to the same embedding vectors) but has its own language model head W_dec ∈ R^{|V| × d}. The decoder is used exclusively during training; at inference it is discarded entirely, meaning the inference-time model is identical in architecture and computational cost to standard implicit CoT methods.

How the decoder conditions on latent tokens. Unlike the base model, which processes the full sequence history to produce each latent, the decoder receives only a single latent vector as context — no question tokens, no other latents. For step k, the decoder is given the latent z_k and must autoregressively generate the corresponding textual reasoning step s_k = (y_{k,1}, ..., y_{k,L_k}). The key design choice is that the decoder sees z_k in isolation, without access to the question or to previous steps. This forces z_k to be a self-contained representation of the k-th reasoning step — it must encode all the information needed to reconstruct that step without relying on context from other latents. If the decoder could see the question text, z_k might learn to encode only a pointer to relevant question tokens rather than the actual computational content of the step. If the decoder could see z_{k-1}, the latents might learn to share information across positions, reducing their distinctiveness and undermining the diversity that SIM-CoT aims to enforce.

The decoder input sequence construction. The latent z_k cannot be treated as a standard token in the decoder's vocabulary because it is a continuous vector, not a discrete index. The paper handles this through a specific injection mechanism: z_k is prepended to the step token sequence as a prefix that initializes the decoder's hidden state for step generation. Concretely, the decoder's input sequence for step k is:

$U_k^{\text{dec}} = [z_k; e(y_{k,1}), ..., e(y_{k,L_k})]$

where the notation [z_k; ...] means that z_k occupies the first position of the sequence, and the token embeddings e(y_{k,t}) occupy the subsequent positions. The decoder processes this sequence autoregressively: at position 0, it receives z_k as input; at position 1, it receives e(y_{k,1}); at position 2, e(y_{k,2}); and so on. The crucial property is that the loss is computed only over the token positions — z_k itself is not predicted and does not contribute to the cross-entropy loss. The latent serves purely as a conditioning signal that biases the decoder's hidden state trajectory toward generating the correct step tokens. This is analogous to how encoder-decoder models condition on encoder outputs: z_k acts as a learned "encoding" of the step, and the decoder learns to "decode" it into text.

The autoregressive step token generation. Let h_{k,t}^dec be the decoder's hidden state at position t (where t = 0 corresponds to z_k, and t ≥ 1 corresponds to the token positions). The probability distribution over the next token given the latent and previous step tokens is:

$p_φ(y_{k,t} | z_k, y_{k,<t}) = \text{softmax}(W^{\text{dec}} · h_{k,t}^{\text{dec}})_{y_{k,t}}$

What this computes. At each position t, the decoder's hidden state h_{k,t}^dec (encoding the latent z_k and the preceding step tokens y_{k,1}, ..., y_{k,t-1}) is projected through the decoder's output matrix W_dec to produce a probability distribution over the vocabulary, and the probability assigned to the ground-truth token y_{k,t} is extracted. The overall probability of generating the full step sequence s_k from latent z_k is the product over all step tokens:

$p_φ(s_k | z_k) = \prod_{t=1}^{L_k} p_φ(y_{k,t} | z_k, y_{k,<t})$

Why the decoder processes one step at a time rather than the full chain. A natural alternative would be to have a single decoder that generates the entire reasoning chain s_1, s_2, ..., s_K from the full latent sequence z_1, ..., z_K. The paper explicitly rejects this design in favor of independent per-step decoding. The reason is directly tied to the goal of preventing latent collapse: if the decoder saw all latents simultaneously, it could learn to reconstruct the reasoning chain by distributing information redundantly across latents — z_1 and z_2 might both encode overlapping numerical information, and the decoder could piece together the full reasoning from this redundant representation without each latent being individually informative. The independent decoding constraint — each z_k must alone suffice to generate s_k — forces each latent to be a complete, self-contained representation of its corresponding step, which enforces diversity and prevents the homogenization that characterizes failed models (Figure 1d).

The loss for a single step. For a given step k with latent z_k and ground-truth step tokens s_k, the step-level supervision loss is the negative log-likelihood of the step tokens under the decoder:

$\mathcal{L}_{\text{step}, k} = -\sum_{t=1}^{L_k} \log p_φ(y_{k,t} | z_k, y_{k,<t})$

What this computes. For each token in the step sequence, the model computes the log-probability assigned by the decoder to the correct token given the latent and previous step tokens, then sums the negative of these log-probabilities. This is standard autoregressive cross-entropy, exactly the same loss used for language modeling pre-training, but with the crucial difference that the conditioning prefix is a continuous vector z_k rather than a sequence of discrete tokens. The loss is non-negative, reaches zero only when the decoder assigns probability 1.0 to every correct token (which never happens in practice), and increases as the decoder assigns lower probability to the correct tokens.

Why cross-entropy rather than a regression or contrastive objective. The paper uses autoregressive cross-entropy because it naturally handles variable-length step sequences (L_k varies across steps depending on the complexity of the arithmetic expression), provides a dense gradient signal (every token position contributes to the loss), and is the standard objective for sequence generation tasks. A regression objective (e.g., minimizing the L2 distance between z_k and some target representation of step s_k) would require pre-defining a target representation, which is itself a hard representational learning problem. A contrastive objective (e.g., making z_k similar to a pooled representation of s_k and dissimilar to representations of other steps) would provide a sparser signal and might not capture the sequential structure of the step tokens.

Gradient flow from the decoder to the base model. This is the mechanism that makes SIM-CoT work. The decoder loss L_step,k is computed as a function of z_k (via the decoder's forward pass). The gradient of this loss with respect to z_k — denoted ∂L_step,k / ∂z_k — tells us how changing z_k would change the decoder's ability to generate the correct step tokens. This gradient is backpropagated through the base model: since z_k = H_θ(U^{(k-1)}), the gradient flows into the transformer parameters θ and into the representations of earlier latents that influenced U^{(k-1)}. Concretely, ∂L_step,k / ∂θ measures how adjusting the base model's parameters would change z_k in a direction that makes it easier for the decoder to reconstruct the correct step. During optimization, θ is updated to minimize both the step and answer losses simultaneously, so the base model learns to produce latents that are simultaneously good for reconstructing intermediate steps (via the decoder) and for producing correct final answers (via the LM head).

The parameter sharing and separation. The decoder has its own transformer parameters φ and its own output projection W_dec. These are trained jointly with θ but serve different purposes: φ learns to decode latent vectors into reasoning text, while θ learns to produce latent vectors that encode reasoning content. The base model's embedding matrix E is shared between the base model and the decoder, which means that both models map the same vocabulary tokens to the same embedding vectors. This sharing is important because it ensures that the latent representations learned by the base model live in the same semantic space as the vocabulary — without this, the latents might drift into a representation space that is incommensurate with the token embeddings, making the transfer of information from latent to answer decoding more difficult. The sharing of E is one factor that helps maintain the geometric property of moderate "distance to vocabulary center" that the paper uses as a stability diagnostic (Table 4b).

The decoder size ablation and its implications. Appendix B reports an experiment where the decoder size is varied while keeping the base model fixed at 1B parameters. A 1B-scale decoder yields the best performance; larger decoders (3B or 8B) slightly degrade performance. The paper hypothesizes that excessive decoder capacity may introduce optimization difficulties or representation misalignment — since the 1B encoder (base model) and 1B decoder originate from the same pretrained model, they share a compatible representation space that facilitates learning. Larger decoders with different pretrained representations may require additional projection layers to align with the encoder, introducing training instability. This finding supports the design choice of matching the decoder capacity to the base model size.

Training Objectives and Gradient Flow

The total training objective for SIM-CoT combines two complementary loss terms — one for step-level supervision through the decoder, and one for final answer supervision through the base language model — into a weighted sum.

Step-level supervision loss (aggregated over all steps). The per-step losses are summed over all K reasoning steps to form the total step-level supervision signal:

$\mathcal{L}_{\text{step}} = -\sum_{k=1}^{K} \sum_{t=1}^{L_k} \log p_φ(y_{k,t} | z_k, y_{k,<t})$

What this computes. For each of the K latent steps, for each token in the corresponding explicit reasoning step, the negative log-probability of the correct token under the decoder is computed, and all these values are summed. The resulting L_step is a single scalar that quantifies how well the entire latent chain z_{1:K} can be decoded into the full reasoning trace s_{1:K}. A low value indicates that each z_k contains sufficient information for the decoder to faithfully reconstruct step s_k; a high value indicates information loss.

Why sum over steps rather than average. Summing (rather than averaging) means that problems with more reasoning steps contribute proportionally more to the total loss, which is appropriate because longer chains are precisely where latent instability is most severe and where per-step supervision is most needed. The sum also means that the gradient magnitude from L_step scales with the total number of step tokens across all steps, which provides a natural balancing against L_ans-lm (which scales with answer length). The paper does not experiment with alternative aggregation strategies (e.g., weighted sum, max), so the choice of simple summation is empirical.

Answer supervision loss. After the K implicit steps, the base model generates the final answer tokens using standard autoregressive language modeling. The answer-level loss is:

$\mathcal{L}_{\text{ans-lm}} = -\sum_{t=1}^{L_a} \log p_θ(a_t | x, z_{1:K}, a_{<t})$

What this computes. For each token in the ground-truth answer sequence a, the model computes the negative log-probability assigned by the base LM head W_o to the correct token, given the question, all latent tokens, and the preceding answer tokens. This is identical to the loss used in Coconut and standard supervised fine-tuning — it optimizes the model to produce correct answers.

Why both losses are necessary. The step-level loss L_step alone could in principle train the model, since if the latents encode the correct reasoning steps, the answer should follow. But relying solely on L_step would introduce an indirect optimization path: the model would learn to produce latents that make the decoder happy, but the decoder might learn to produce correct-looking step text from latents that don't actually encode generalizable reasoning (a form of "decoder overfitting" where the decoder learns to map degenerate latents to memorized step patterns). The answer-level loss L_ans-lm provides a direct, end-task-relevant signal: it forces the latents to be useful for producing correct answers through the base model's LM head, not just for reconstructing steps through the decoder. Conversely, L_ans-lm alone (as in Coconut) provides insufficient per-step structure, leading to the latent instability documented in Section 2. The combination ensures that latents encode both step-level semantic content and answer-relevant computational information.

The combined objective. The total loss is a weighted sum:

$\mathcal{L} = \lambda_{\text{step}} \mathcal{L}_{\text{step}} + \lambda_{\text{lm}} \mathcal{L}_{\text{ans-lm}}$

where λ_step and λ_lm are scalar hyperparameters controlling the relative importance of the two objectives. The paper does not report the specific values used for these weights, which is a notable omission — these coefficients could significantly affect the balance between step reconstruction fidelity and answer accuracy. In practice, the two terms operate in the same units (both are sums of negative log-probabilities) and have comparable magnitudes (since step token count and answer token count are typically within the same order of magnitude for GSM8K problems), so equal weighting (λ_step = λ_lm = 1) is a plausible default, but the paper does not confirm this.

Gradient flow from the combined loss. The total loss L is differentiated with respect to all trainable parameters. The gradients flow through two distinct paths:

Path 1 (via the decoder): ∂L/∂z_k receives contributions from both L_step (through the decoder's forward pass) and L_ans-lm (through the base model's answer token predictions, since z_k influences U^{(K)} which conditions the answer). These gradients are then backpropagated through the base model's transformer layers to update θ. Additionally, ∂L/∂φ updates the decoder parameters to improve step reconstruction, and ∂L/∂W_dec updates the decoder's output projection.
Path 2 (via the LM head): ∂L/∂θ receives direct contributions from L_ans-lm through the standard language modeling gradient path (question → latents → answer). This updates the base model to produce better answers and, indirectly, better latents.
Path 3 (via the shared embeddings): Since the embedding matrix E is shared between the base model and the decoder, gradients from both L_step (through the decoder's token embeddings) and L_ans-lm (through the base model's token embeddings) flow into E. This joint update ensures the embedding space remains useful for both the base model's and the decoder's generation tasks.

The interaction between these gradient paths is what enforces the dual constraint on the latent representations: z_k must simultaneously be decodable into step s_k (enforced by Path 1) and useful for answer generation (enforced by Path 2). This dual constraint is the mechanism by which SIM-CoT prevents latent collapse — if the latents degenerate into homogeneous numerical vectors, L_step will increase because the decoder cannot distinguish which step each vector corresponds to, and L_ans-lm will increase because the base model cannot extract the necessary operator information from homogeneous latents. The gradients from both terms push the latents toward diversity and semantic grounding.

Discarding the decoder at inference. The most important practical property of SIM-CoT is that the decoder p_φ, its parameters φ, and the step-level loss L_step exist only during training. At inference, the forward pass is:

Process the question tokens to produce U^{(0)}.
For k = 1, ..., K: compute z_k = H_θ(U^{(k-1)}), append to sequence.
Generate answer tokens autoregressively from the final prefix U^{(K)} using the standard LM head W_o.

This is identical to Coconut's inference procedure — no decoder, no step generation, no additional computation beyond the K extra forward positions for the latent tokens. The training-time decoder and its loss serve solely to shape the latent representations during optimization; once trained, the model's θ parameters have internalized the step-level structure, and the latents z_{1:K} naturally encode diverse, step-specific information without needing the decoder to enforce it at test time. This is the "plug-and-play" property: SIM-CoT modifies training, not inference, so any model trained with it can be deployed identically to a Coconut-trained model.

Curriculum for the Number of Implicit Steps

The paper inherits Coconut's curriculum learning strategy for gradually increasing the number of implicit latents K during training, with modifications to accommodate the step-level supervision. The curriculum is defined by a schedule function:

$K^{(e)} = \min\left(K_{\max}, \left\lfloor \frac{e}{\Delta e} \right\rfloor\right)$

where K^{(e)} is the number of implicit latent steps used at training epoch e, K_max is the maximum number of latents (set to 8 in the main experiments, based on the reasoning step distribution in Figure 5 where most problems involve 2–6 steps with a long tail of harder cases), and Δe is the update interval in epochs (set to 3 for GPT-2 and LLaMA 1B, following Coconut).

What this schedule does. Starting from K^{(0)} = 0 (no implicit latents, standard explicit CoT training), every Δe epochs, one additional latent step is introduced. At epoch e = Δe, the model switches from K = 0 to K = 1 — one explicit reasoning step is replaced by an implicit latent. At epoch e = 2Δe, K increases to 2, and so on, until K = K_max is reached, after which K remains fixed for the remainder of training. The total training duration is K_max × Δe + 15 additional epochs after reaching K_max for Coconut-based training (the "+15" epochs are specified in Appendix E.1; CODI-based training uses different epoch counts as detailed in Table 6).

Why a curriculum is necessary. Directly training with K = K_max from epoch 1 would require the model to simultaneously learn (a) how to produce useful latent vectors, (b) how to interpret latent vectors as inputs for subsequent steps, and (c) how to map latent vectors to reasoning text through the decoder — all from a cold start. The curriculum decomposes this into a gradual progression: first learn to reason with mostly explicit steps and a few latents, then gradually internalize more reasoning into the latent space as the model becomes proficient. This is directly analogous to how iCoT (Deng et al., 2024) used "stepwise internalization" to progressively remove explicit reasoning steps. The paper's key contribution is showing that even with this curriculum, Coconut collapses at K = 5 (Figure 1a), while SIM-CoT with step-level supervision remains stable through K = 8 and beyond (Figure 3), indicating that the curriculum is necessary but not sufficient — step-level supervision is the missing ingredient that enables scaling to longer latent chains.

How the curriculum interacts with the step-level supervision. As K increases, the decoder must learn to decode more latents, and the base model must learn to produce more latents. The step-level loss L_step provides a direct per-latent signal at every value of K, which means that when a new latent is introduced (e.g., going from K = 3 to K = 4), the model immediately receives feedback on whether the new latent encodes useful information. Without this per-step signal (as in Coconut), the model must infer the role of the new latent solely from the answer-level loss, which is a much weaker and more delayed signal — leading to the instability observed when K reaches 5. The curriculum thus works synergistically with the step-level supervision: the curriculum controls the pace at which new latents are introduced, and the supervision ensures that each new latent learns to encode distinct step content from the moment it is introduced.

The maximum K and its relationship to problem difficulty. The paper sets K_max = 8 based on the distribution of reasoning steps in GSM8K-Aug (Figure 5), where most problems require 2–4 steps and a small fraction require 6 or more. For problems with fewer than K_max steps (the majority), the "extra" latent steps are still generated but may encode redundant or empty content — the paper's case studies (Appendix K) note that "implicit reasoning continues to produce latent tokens even after the correct answer has been reached," with trailing latents simply repeating the final prediction. This is not harmful to accuracy but does represent wasted computation. An interesting future direction would be dynamic K selection based on estimated problem difficulty, though the paper does not explore this. The key finding is that even with a fixed K_max = 8, SIM-CoT remains stable and continues to improve over Coconut at all latent counts (Figure 3, blue line consistently above orange line), demonstrating that the method does not require precisely matching K to the ground-truth step count.

Implementation details. The specific hyperparameters for the curriculum and training are detailed in Appendix E (Table 6). For GPT-2 with Coconut backbone: learning rate 1 × 10^{-4}, 15 epochs after reaching K_max, Adam optimizer with β_1 = 0.9, β_2 = 0.999, weight decay 0.1, batch size 128. For LLaMA 1B with Coconut backbone: same learning rate and curriculum schedule as GPT-2. For CODI-based training on GPT-2: learning rate 3 × 10^{-3}, 40 epochs, batch size 128. For CODI on LLaMA 1B: learning rate 8 × 10^{-4}, 10 epochs. The decoder uses the same optimizer configuration as the base model and is trained jointly from scratch (no pretrained decoder weights are loaded — the decoder is randomly initialized). The embeddings E are initialized from the pretrained base model's embeddings and updated during training.

Inference Procedure

At inference, SIM-CoT's procedure is architecturally identical to standard implicit CoT methods like Coconut. The auxiliary decoder p_φ is completely removed — its parameters are not loaded, and no step generation occurs. The inference forward pass consists of exactly three stages:

Stage 1: Question encoding. The input question tokens x = (x_1, ..., x_T) are embedded to form the initial prefix U^{(0)} = (e(x_1), ..., e(x_T)). This is a standard transformer forward pass that produces key-value caches for all question tokens, which will be reused in subsequent stages.

Stage 2: Implicit reasoning. For k = 1, ..., K (where K is the fixed number of implicit steps determined during training, typically 3–8 depending on the model configuration): compute z_k = H_θ(U^{(k-1)}), then append z_k to the sequence to produce U^{(k)}. Each of these K steps requires one additional forward pass through the transformer (or, with KV-caching, one additional position's worth of computation per step). This is K forward positions total for the implicit reasoning, which replaces the potentially much larger number of positions that would be needed for explicit CoT text generation (e.g., 40–100 tokens for a typical 4-step GSM8K problem).

Stage 3: Answer generation. Starting from the final prefix U^{(K)} (question + K latent vectors), the model autoregressively generates answer tokens a_1, a_2, ... using the standard language model head W_o and a decoding strategy (greedy decoding, beam search, or sampling — the paper uses greedy decoding for all reported results unless otherwise specified, as is standard for evaluation). Generation continues until an end-of-sequence token is produced or a maximum answer length is reached.

Token efficiency quantification. The paper reports token counts in the "# Tokens" columns of Tables 1–3. These counts represent the average total number of token positions (question + reasoning + answer) across the test set. For implicit methods (Coconut, CODI, SIM-CoT), the count includes the question tokens, the implicit latent positions (each latent counts as 2 tokens in the sequence, since "one implicit latent corresponds to two implicit tokens" per Appendix E.1), and the answer tokens. For explicit CoT (SFT-CoT), the count includes question tokens, all explicit reasoning step tokens, and answer tokens. For GPT-2 (Table 1), SIM-CoT+Coconut uses 11.4 tokens for reasoning (in-domain average) compared to SFT-CoT's 24.7 tokens — a 2.3× reduction. For LLaMA 1B (Table 2), the reduction is 13.4 vs. 23.1 tokens — a 1.7× reduction. The larger relative gain on GPT-2 reflects that smaller models require proportionally longer explicit reasoning traces to achieve reasonable accuracy, making the implicit compression more impactful.

Latency considerations not addressed in the paper. The paper measures compute in "token positions" (number of forward positions through the transformer), which is a reasonable proxy for total FLOPs but ignores two latency-relevant factors. First, the implicit phase requires K sequential forward passes that each depend on the previous one (the autoregressive latent generation creates a chain of data dependencies), making the implicit reasoning inherently sequential and preventing parallelization — exactly the same latency constraint as generating K text tokens. Second, the absence of the decoder at inference means SIM-CoT adds zero latency overhead compared to Coconut or other implicit methods, but the base latency of K sequential forward passes remains. For applications where K = 4 replaces 40 explicit reasoning tokens, the latency improvement is approximately 10× (since each token generation requires one forward pass). This is a significant practical advantage, though the paper does not report wall-clock timing measurements.

4. Key Insights and Innovations

Innovation 1: Diagnosing Latent Collapse as a Measurable, Mechanistic Failure Mode

The paper's most foundational contribution is not a method but a diagnostic framework that transforms "implicit CoT is unstable" from a vague empirical observation into a specific, measurable, and interpretable failure mode with geometric and semantic signatures. Prior work treated the underperformance of implicit methods as a black-box accuracy gap — models trained with answer-level (Coconut, Hao et al., 2025) or trajectory-level (CODI, Shen et al., 2025b) supervision simply scored lower than explicit CoT, and the mechanism of that underperformance went uncharacterized. The field lacked a language for describing how implicit reasoning breaks when it breaks, which in turn obscured why certain supervision strategies succeed or fail.

SIM-CoT's Section 2 provides that language through a four-part decomposition of the collapse phenomenon, each dimension backed by a specific, quantifiable metric:

Latent instability as an optimization phenomenon (Figure 1a). Rather than reporting only final accuracy, the paper traces training dynamics as the number of implicit tokens scales from one to five. The transition is not gradual — accuracy first improves (suggesting more latent capacity helps), then catastrophically collapses at five tokens, with training becoming erratic rather than smoothly plateauing. This distinguishes the problem from mere underfitting (which would show saturating but stable accuracy) and identifies it as an optimization instability — the training signal itself becomes unreliable as the latent chain lengthens.

Information loss as operator-specific degradation (Figure 1b). By separately evaluating accuracy on numerical extraction, operator identification, and final answer computation, the paper pinpoints what information is lost during collapse: operator information disappears first and most severely, while numerical information persists longer. This is not an obvious finding — one might expect all information to degrade uniformly — and it has direct implications for supervision design. It explains why answer-level supervision (Coconut) is insufficient: the answer loss provides a weak signal about operator correctness (since getting numbers right but operators wrong still produces a wrong answer, but the gradient doesn't distinguish which error caused the failure), whereas step-level supervision explicitly requires the model to encode operators in the latent representation to reconstruct the step text.

Shifted distance as a geometric signature of collapse (Figure 1c, Table 4b). The paper introduces two continuous metrics — inter-latent distance (average pairwise L2 distance between latent vectors) and distance to vocabulary center (average distance from each latent to the mean of the token embedding matrix). In healthy models, inter-latent distance is high (latents are distinct from each other) and distance to the vocabulary center is moderate (latents remain anchored near meaningful semantic space). In collapsed models, inter-latent distance plummets (latents become nearly identical) while distance to the vocabulary center spikes (latents drift into uninterpretable regions of the embedding space). The paper quantifies this: from 28.34 inter-latent distance in a normal 5-latent model to 4.21 in the failed case, and from 28.34 to 39.39 for vocabulary center distance. These are not subtle shifts — they are order-of-magnitude changes that provide a clear, computable diagnostic for whether an implicit reasoning model is healthy without requiring human inspection of decoded tokens.

Semantic homogenization as the qualitative correlate (Figure 1d). The geometric collapse has a direct semantic interpretation: when latents become too similar, they decode to nearly identical content. The paper shows that in failed models, distinct latent positions that should encode different reasoning steps instead all decode to the same numbers, with operators and logical structure absent. This connects the quantitative metrics to human-interpretable reasoning quality: the model hasn't just learned suboptimal representations — it has lost the ability to represent multi-step computation entirely, defaulting to a "bag of numbers" representation that cannot support complex reasoning.

Why this diagnostic framework is conceptually significant. It transforms the implicit CoT research agenda from trial-and-error (try a new supervision strategy, measure accuracy, iterate) to hypothesis-driven engineering (measure inter-latent distance and vocabulary drift during training, diagnose whether collapse is occurring, and design interventions that prevent the specific geometric failure mode). The paper uses this framework not just to diagnose Coconut's failures but to validate SIM-CoT's success: Table 4b shows that after applying SIM-CoT, inter-latent distance increases to 32.81 (even higher than the healthy 5-latent baseline) while vocabulary center distance remains moderate at 29.80 — demonstrating that step-level supervision doesn't just improve accuracy but specifically restores the geometric properties associated with healthy, diverse latent representations. Figure 6 visualizes this restoration across three rows (normal, failed, SIM-CoT-recovered), making the geometric argument visually concrete.

This is a fundamental contribution in the sense that it provides concepts and metrics that outlive any particular method. Future work on implicit reasoning can and should use inter-latent distance and vocabulary center distance as standard diagnostic tools, much as NLP researchers use perplexity or BLEU score. The paper effectively establishes that these geometric properties are necessary (though perhaps not sufficient) conditions for effective implicit reasoning, and any method that claims to solve the instability problem should be evaluated on whether it restores them.

Innovation 2: Reframing Implicit CoT Instability as a Supervision Granularity Problem

Before SIM-CoT, the dominant assumption in the implicit reasoning literature — implicit in the design of Coconut, CODI, and related methods — was that latent instability was fundamentally an architecture or optimization challenge. The default approach was to treat the model as a black box that needed to learn to compress reasoning into a latent space, with the training signal provided at whatever granularity was convenient: final answer correctness (Coconut), trajectory-level alignment (CODI), or no training at all (soft thinking methods from Zhang et al., 2025b; Wu et al., 2025). When these methods underperformed or collapsed, the natural response was to seek better architectures (recurrent depth scaling, Geiping et al., 2025; looped transformers, Saunshi et al., 2025) or better optimization curricula (progressive internalization, Deng et al., 2024).

SIM-CoT makes a fundamentally different argument: the problem is not that we lack the right architecture to make implicit reasoning work, but that we have been providing supervision at the wrong granularity. The paper demonstrates that adding step-level textual grounding — without any architectural modification to the autoregressive latent generation mechanism — is sufficient to stabilize training, prevent collapse, and close most of the gap to explicit CoT. This is a reframing of the problem space, not an incremental improvement to an existing method.

What makes this reframing distinctive. The paper's choice to implement step-level supervision through a training-only auxiliary decoder, discarded at inference, is not just an engineering convenience — it is a conceptual statement. It says: "the latent representations need better guidance during training, but the inference mechanism is fine." This stands in contrast to approaches that modify the model architecture to make latent reasoning more stable (e.g., adding recurrence, changing attention patterns, or introducing dedicated latent processing layers). SIM-CoT's results imply that the autoregressive latent construction mechanism inherited from Coconut is not the bottleneck — the bottleneck is the quality of the training signal that shapes the latent representations.

This reframing is supported by the paper's experiments with CODI as a backbone (Tables 1–3). CODI already adds trajectory-level supervision, which represents an intermediate point on the supervision granularity spectrum between Coconut's answer-level signal and SIM-CoT's step-level signal. The fact that SIM-CoT improves CODI by +3.4% on LLaMA 1B and +3.0% on LLaMA 8B (Table 3) indicates that trajectory-level supervision is not the ceiling — there are gains to be had specifically from per-step grounding, even when trajectory-level alignment is already in place. This is not an obvious result: one might reasonably hypothesize that trajectory-level distillation already provides sufficient per-step information (since the trajectory contains all steps), but the data shows otherwise. The coarse-grained alignment doesn't enforce that each latent encodes a distinct step; the model can learn to distribute information redundantly across latents while still achieving good trajectory-level alignment. Only explicit per-step decoding forces each latent to be individually informative.

The "supervision spectrum" as an organizing concept. By implementing and comparing answer-level (Coconut), trajectory-level (CODI), and step-level (SIM-CoT) supervision within the same architectural framework, the paper implicitly establishes a supervision granularity spectrum for implicit reasoning. This spectrum provides a conceptual tool for thinking about the design space: on one end, minimal supervision (only final answer) leads to representational collapse as the chain lengthens because there is no signal to maintain diversity; on the other end, maximal supervision (full step-by-step textual decoding of every latent) enforces diversity but requires step-level annotations and introduces an auxiliary training cost; in between, trajectory-level supervision provides a partial signal that helps but doesn't fully prevent collapse. The paper doesn't explicitly name this spectrum, but its experimental design — testing SIM-CoT on both Coconut and CODI, and measuring geometric properties at each supervision level — makes it the central organizing principle of the work.

The significance beyond this paper. This reframing has implications for how researchers approach latent reasoning more broadly. If supervision granularity is the key variable, then future work should focus on (a) developing methods to obtain step-level annotations more cheaply (e.g., through automatic parsing of reasoning traces, weak supervision from smaller models, or self-supervised step proposal), and (b) exploring supervision granularities between trajectory-level and step-level (e.g., grouping steps into "subgoals" and supervising at the subgoal level, or using a dynamic granularity that varies with problem difficulty). The reframing also suggests that architectural innovations for implicit reasoning — which have been an active research area — may be solving a problem that is better addressed through supervision design, potentially redirecting research effort toward data and annotation strategies.

Innovation 3: Demonstrating That Implicit Reasoning Can Surpass Explicit CoT Under the Right Supervision

A persistent narrative in the implicit reasoning literature has been that implicit methods are "fast but inaccurate" — they offer token efficiency at the cost of a performance gap relative to explicit chain-of-thought. This narrative was supported by consistent empirical results: Coconut underperformed SFT-CoT of comparable training effort, CODI narrowed but did not close the gap, and training-free implicit methods were even further behind. The field had largely accepted this as a fundamental trade-off: the compression of reasoning into a latent space inevitably loses some information or flexibility, and the best one could hope for was to minimize the gap.

SIM-CoT challenges this narrative directly with a result that, while specific to a particular model scale, carries broader implications: on GPT-2, SIM-CoT+Coconut outperforms the explicit CoT supervised fine-tuning baseline by 2.1% (44.8% vs. 42.7%, Table 1) while using 2.3× fewer tokens for reasoning (11.4 vs. 24.7 tokens). This is not just closing the gap — it is reversing it. The implicit method is simultaneously more accurate and more efficient.

Why this result is surprising and significant. The conventional wisdom would predict that compressing 24.7 tokens of explicit reasoning into 11.4 token positions' worth of latent vectors would lose information, and that this information loss would manifest as reduced accuracy, not increased accuracy. SIM-CoT's result suggests the opposite: under certain conditions, the latent representation is not just a compressed version of explicit reasoning but a better medium for reasoning than text. The paper doesn't fully explain this mechanism — it's an empirical finding, not a theoretical result — but the implication is provocative: the constraint of verbalizing reasoning in discrete natural language tokens may actually limit reasoning quality by forcing the model to commit to representations that are optimized for human readability rather than computational utility. The continuous latent space allows the model to encode information in forms that don't correspond to any single token or phrase, potentially enabling more compact and more accurate representations of numerical relationships.

This interpretation is consistent with prior theoretical work suggesting that transformers can perform more efficient computation in continuous latent space than in discrete token space (Zhu et al., 2025; Xu and Sato, 2025), but SIM-CoT provides the first empirical demonstration that this theoretical advantage can translate to practical accuracy gains over explicit CoT at a given model scale. The paper is appropriately cautious about this claim — it doesn't assert that implicit reasoning is universally superior, and the result is demonstrated on a relatively small model (GPT-2) where explicit CoT's accuracy is modest (42.7%). On larger models (LLaMA 8B), SIM-CoT achieves "performance comparable to SFT-CoT" (96% of explicit CoT accuracy, Section 4.2) but does not surpass it, suggesting that the implicit-explicit trade-off may shift with model scale. This scaling-dependent relationship is itself an important finding: it implies that the representational advantages of latent reasoning may be most pronounced when the model's capacity is constrained relative to the task complexity, and that as models grow larger, the advantage of explicit verbalization (structured, interpretable, easier to supervise) may dominate.

The out-of-domain generalization perspective. Even on larger models where SIM-CoT doesn't surpass explicit CoT in-domain, it shows competitive or superior out-of-domain performance. On LLaMA 8B (Table 3), SIM-CoT matches or exceeds SFT-CoT on MultiArith (100.0 vs. 98.3) and SVAMP (79.4 vs. 73.1) — benchmarks that test generalization to problem variations and different wording. The paper attributes this to the "moderate form of supervision" provided by step-level alignment: unlike SFT-CoT, which forces the model to mimic deterministic natural language annotations (potentially overfitting to annotation style), SIM-CoT's step-level supervision "ensures the plausibility of each reasoning step while preserving the diversity of reasoning trajectories, thereby improving generalization to unseen inputs." This is a subtle but important point: explicit CoT training can teach the model to reproduce specific phrasings and reasoning patterns from the training data, which may not transfer to problems with different wording or structure. SIM-CoT's latent representations, supervised only on the semantic content of each step (decoded through the auxiliary decoder) rather than the exact token sequence, may learn more abstract reasoning operations that generalize better.

The implications for the implicit-explicit debate. SIM-CoT's result that implicit reasoning can surpass explicit CoT on GPT-2 — even if only at that specific scale and on that specific benchmark — reframes the narrative from "implicit reasoning is a necessary efficiency compromise" to "implicit reasoning may be the better approach, and our previous failures were due to insufficient supervision, not fundamental limitations of latent representations." This shifts the burden of proof: rather than asking whether implicit reasoning can ever catch up to explicit CoT, the question becomes under what conditions and at what scales implicit reasoning is the superior choice. The paper doesn't answer this question comprehensively — GPT-2 is one data point, GSM8K-Aug is one dataset — but it establishes the existence proof that was previously missing.

Innovation 4: Interpretable Implicit Reasoning Through Latent Decoding

A long-standing criticism of implicit reasoning methods has been their opacity. When a model reasons in continuous latent space, there is no direct way to inspect whether its intermediate steps are correct, logical, or even relevant to the problem. This opacity has practical consequences: it makes error diagnosis difficult (if the model produces a wrong answer, is it because of a flaw in step 2 or step 4?), hinders trust in high-stakes applications, and complicates the development process (researchers cannot easily determine whether a new training method is improving reasoning quality or just memorizing answer patterns).

SIM-CoT provides a solution to this opacity as a byproduct of its training design. Because the auxiliary decoder is trained to map each latent z_k to its corresponding explicit reasoning step s_k, it can be reused at inference time — even though it's not needed for answer generation — to project each implicit latent back into human-readable text. The case studies in Figure 4 and Appendix K demonstrate this capability: for a GSM8K problem, the decoder produces step-by-step reasoning (e.g., "0.3 × 120 = 36", "120 − 36 = 84", "3/4 × 84 = 63", "84 − 63 = 21") that mirrors the structure of explicit chain-of-thought while being generated from the continuous latent representations.

Why this is conceptually significant beyond convenience. The ability to decode latent steps into text transforms implicit reasoning from an opaque computation into a verifiable process. For the first time in the autoregressive latent reasoning paradigm, researchers and practitioners can inspect individual latent tokens and ask: "What does the model think it's doing at step 3? Is that step mathematically correct? Does it follow logically from step 2?" The paper's case studies show that the decoded steps can be directly compared against ground-truth reasoning, enabling per-step error analysis that was previously impossible for implicit methods.

This interpretability has two distinct audiences with different implications. For researchers, it enables debugging of training dynamics — if a particular latent consistently decodes to nonsense text while others decode to correct reasoning, that indicates a problem with how supervision is distributed across latent positions, potentially informing better curriculum design or loss weighting. For practitioners, it provides a trust mechanism — in a deployed system, the decoded steps could be shown to users or logged for audit, providing transparency into the model's reasoning process even though the actual computation occurred in continuous space. This partially addresses the "black box" criticism that has limited implicit reasoning's adoption in applications where reasoning transparency is required.

The relationship to explicit CoT interpretability. It's worth noting that the interpretability provided by SIM-CoT's decoder is different in kind from explicit CoT interpretability. In explicit CoT, the reasoning text is the computation — the model generates tokens that simultaneously serve as the reasoning mechanism and the explanation. In SIM-CoT, the decoded text is a projection of the latent computation into human language, not the computation itself. This means the decoded text might not perfectly capture everything the latent representations encode — the decoder may fail to express certain numerical relationships or may simplify the reasoning in ways that lose nuance. The paper doesn't quantify the fidelity of the latent-to-text mapping (e.g., by measuring whether the decoded steps are consistent with the final answer), which leaves open the question of whether the decoded text can be fully trusted as a faithful representation of the model's internal reasoning. This is a limitation worth noting, but it doesn't diminish the practical value of having some interpretability where previously there was none.

The decoder as a diagnostic tool for latent health. Appendix I and Figure 6 use the decoder not just for step-by-step visualization but also for geometric diagnostics of the latent space. By decoding the content of latent tokens and measuring their semantic diversity, the paper can distinguish healthy models (where different latents decode to distinct, meaningful content — numbers, operators, subgoals) from collapsed models (where all latents decode to similar numerical tokens). This diagnostic use of the decoder goes beyond the simple "does the model get the right answer?" evaluation and provides insight into the quality of the internal representations, which could be used for early stopping, hyperparameter tuning, or detecting training instability before it manifests as accuracy degradation.

This innovation is incremental rather than fundamental in the sense that it's a byproduct of the training design rather than a primary goal, but it addresses a specific, well-known limitation of implicit reasoning methods (opacity) that has been a barrier to adoption. The fact that interpretability comes "for free" — no additional training, no separate interpretation model, just reusing the already-trained decoder — makes it a particularly practical contribution.

5. Experimental Analysis

Evaluation Methodology

Dataset. All experiments use the GSM8K-Aug dataset (Deng et al., 2024) for training, an augmented version of GSM8K (Cobbe et al., 2021) expanded from 8.5k to approximately 385k training examples through GPT-4 paraphrasing, numerical resampling, and synthetic generation. The reasoning chains in GSM8K-Aug are stripped of natural language, preserving only structured mathematical expressions (e.g., <<12*3=36>><<9*2=18>>), with each expression logically linked to the previous step. For evaluation, the paper uses the GSM8K-Aug test set as the in-domain benchmark and three out-of-domain benchmarks: SVAMP (1,000 elementary-level arithmetic word problems with controlled wording variations; Patel et al., 2021), GSM-Hard (a modified GSM8K test split where numbers are replaced with larger magnitudes to increase difficulty; Gao et al., 2022), and MultiArith (600 multi-step arithmetic word problems; Roy & Roth, 2015). The distribution of reasoning steps in GSM8K-Aug (Figure 5) shows most problems require 2–4 steps, with a long tail extending to 6 or more steps.
Base model(s). The paper evaluates on four model families at different scales: GPT-2 (Radford et al., 2019), LLaMA 3.2 1B, LLaMA 3.2 3B, and LLaMA 3.1 8B (Meta, 2024). GPT-2 is chosen as the primary testbed for method development and detailed ablation because its modest scale (and correspondingly modest baseline accuracy on GSM8K-Aug) leaves substantial room for improvement, making it easier to measure the effect of interventions. The LLaMA models test scalability to larger, more capable backbones, with the 1B model providing a direct comparison point to GPT-2 and the 3B/8B models evaluating whether gains persist as pretrained capability increases. The paper notes that "curriculum learning in larger models leads to catastrophic forgetting" (Section 4.2), which motivates the choice of CODI as the backbone for larger models due to its KL-regularized objective that constrains training not to deviate too far from the original model distribution.
Metrics. The primary metric is accuracy (%) — the fraction of test questions for which the model's generated final answer matches the ground-truth answer. For in-domain evaluation, this is computed on the GSM8K-Aug test set; for out-of-domain evaluation, on GSM-Hard, MultiArith, and SVAMP. The paper also reports token efficiency as the average number of tokens generated per question (question + reasoning + answer), shown in the "# Tokens" columns of Tables 1–3 and Appendix Tables 4–5. For implicit methods, each latent token counts as two tokens in the sequence (a convention inherited from Coconut, Appendix E.1: "One implicit latent corresponds to two implicit tokens"). Geometric diagnostics include inter-latent distance (average pairwise L2 distance between latent vectors, Equation 9) and distance to vocabulary center (average L2 distance from each latent to the mean of all token embeddings, Equation 10), reported in Table 4b. The paper also reports per-component accuracy on numbers, operators, and final answers in the diagnostic Figure 1b, though these are not used as primary metrics in the main experiments.
Baselines. The paper compares against five methods:
1. SFT-CoT: Supervised fine-tuning on CoT-annotated data, where the model is trained to generate explicit intermediate reasoning steps followed by the final answer. This is the explicit CoT baseline and represents the accuracy ceiling for reasoning methods.
2. No-CoT: Supervised fine-tuning on direct answers only, without producing intermediate steps. This represents the lower bound — what the model can achieve without any reasoning chain.
3. iCoT (Deng et al., 2024): A curriculum learning method based on "Stepwise Internalization" that injects CoT reasoning patterns into the model's internal representations through progressive removal of explicit steps during training. This is an earlier implicit reasoning method that predates Coconut.
4. Coconut (Hao et al., 2025): The primary implicit CoT baseline using answer-level supervision with curriculum learning that gradually replaces explicit reasoning steps with implicit latent tokens. Serves as the backbone for most SIM-CoT experiments.
5. CODI (Shen et al., 2025b): A distillation-based method where explicit CoT acts as the teacher and implicit CoT as the student, aligning the last hidden states of the full reasoning trajectory. Represents the state-of-the-art implicit reasoning method prior to SIM-CoT, particularly on larger models where it alleviates catastrophic forgetting through KL regularization.
Generation budget / compute accounting. The paper measures compute in two complementary ways. First, token count — the total number of token positions processed during inference (question + reasoning + answer), with each implicit latent counting as 2 token positions. This is reported in the "# Tokens" columns of all result tables and provides a direct measure of inference cost. Second, implicit token count — the number of latent tokens K (which corresponds to 2K actual positions in the sequence). This is the primary variable in scaling studies (Figure 3, where K ranges from 1 to 8). The paper does not report wall-clock time, FLOPs, or latency measurements. All methods are compared at the same implicit token count K within each experiment, ensuring fair comparison of the supervision strategy rather than the compute budget.
Cross-validation / statistical protocol. The paper does not report cross-validation, statistical significance tests, confidence intervals, or error bars for any of the main results. Accuracy is reported as single-point estimates (e.g., "44.8" in Table 1). For the ablation on implicit token count (Figure 3), each configuration reports "the best performance" achieved during training, which is common practice but means the results represent peak rather than average performance and may overstate reliability. The absence of variance estimates is a notable limitation given that the test sets are relatively small (GSM8K-Aug test set, SVAMP with 1,000 problems, MultiArith with 600 problems) and performance differences between methods are sometimes modest (e.g., +0.6% gain for SIM-CoT on CODI with GPT-2 in Table 1, +0.2% for soft thinking additions in Table 5).

Main Quantitative Results

In-Domain Results on GPT-2 (Table 1)

Table 1 reports accuracy on the GSM8K-Aug test set for GPT-2, the primary development model. The headline result is that SIM-CoT+Coconut achieves 44.8% accuracy, surpassing the explicit SFT-CoT baseline (42.7%) by 2.1 percentage points while using 2.3× fewer tokens (11.4 vs. 24.7 tokens). This is the first reported instance where a training-based implicit CoT method outperforms explicit CoT on this benchmark with this model scale.

Breaking down the comparisons against individual baselines:

SIM-CoT+Coconut (44.8%) vs. Coconut (36.6%): SIM-CoT provides an +8.2 percentage point improvement, a 22.4% relative gain. This is the largest single improvement in the table and directly quantifies the benefit of adding step-level supervision to answer-level-supervised implicit reasoning.
SIM-CoT+CODI (42.6%) vs. CODI (42.0%): The gain is +0.6 percentage points, a much smaller improvement than on Coconut. This suggests that CODI's trajectory-level distillation already captures much of the benefit that step-level supervision provides, with the remaining gap being relatively modest. However, the fact that there is any improvement at all indicates that trajectory-level alignment does not fully substitute for per-step grounding.
SIM-CoT+Coconut (44.8%) vs. SIM-CoT+CODI (42.6%): The Coconut-based version outperforms the CODI-based version by 2.2 points, which is unexpected given that CODI alone (42.0%) substantially outperforms Coconut alone (36.6%). This suggests a potential interaction effect: SIM-CoT's step-level supervision may work more synergistically with Coconut's simpler training setup than with CODI's already-regularized objective, possibly because CODI's trajectory-level constraint limits the flexibility of latent representations and reduces the room for step-level supervision to reshape them. The paper does not explore this interaction in detail.

Against non-implicit methods, SIM-CoT+Coconut (44.8%) substantially outperforms No-CoT (19.1%) and iCoT (30.1%), confirming that implicit reasoning with step-level supervision provides reasoning capability far beyond direct answer prediction or simpler internalization methods.

Out-of-Domain Results on GPT-2 (Table 1, columns 3–6)

Across the three out-of-domain benchmarks, SIM-CoT+Coconut achieves an average accuracy of 46.9%, compared to 42.6% for Coconut alone (+4.3 points) and 45.2% for SFT-CoT (+1.7 points over explicit CoT). The per-benchmark breakdown reveals non-uniform improvements:

GSM-Hard: SIM-CoT+Coconut achieves 12.2% vs. Coconut's 12.2% — no improvement. GSM-Hard tests robustness to larger numbers, and the identical scores suggest that step-level supervision does not help the model handle numerical magnitude shifts, possibly because the latent representations encode reasoning operations rather than numerical values, and the operations are the same regardless of number magnitude.
MultiArith: SIM-CoT+Coconut achieves 9.3% vs. Coconut's 8.1% (+1.2 points). This is a modest gain on a dataset that tests multi-step operation sequencing.
SVAMP: SIM-CoT+Coconut achieves 90.8% vs. Coconut's 83.5% (+7.3 points). This is by far the largest out-of-domain improvement and is notable because SVAMP specifically tests robustness to superficial changes in problem wording and structure. The paper attributes this to the "moderate form of supervision" that "ensures the plausibility of each reasoning step while preserving the diversity of reasoning trajectories, thereby improving generalization to unseen inputs" (Section 4.2). The fact that the largest gain appears on the benchmark most explicitly designed to test generalization supports this interpretation.

For SIM-CoT+CODI on GPT-2, out-of-domain improvements are minimal: 48.0% vs. 48.3% average, with the small gain (+0.3 points) concentrated in SVAMP (42.6% vs. 41.7%). This again suggests CODI already captures most of the generalization benefit that step-level supervision provides on GPT-2.

Results on LLaMA 3.2 1B (Table 2)

Table 2 shows results for LLaMA 3.2 1B, a substantially more capable pretrained model than GPT-2 (SFT-CoT baseline: 58.4% vs. 42.7%). The key findings demonstrate that SIM-CoT's benefits scale to larger models but with important shifts in the pattern of improvements:

SIM-CoT+Coconut (42.2%) vs. Coconut (33.2%): An +9.0 percentage point improvement, which is actually larger than the +8.2 point gain on GPT-2. This indicates that Coconut's answer-level supervision is even more insufficient on the larger model — possibly because the 1B model's stronger pretrained representations create a larger representational space that is harder to constrain with answer-level signals alone.
SIM-CoT+CODI (56.1%) vs. CODI (52.7%): An +3.4 percentage point improvement, substantially larger than the +0.6 point gain on GPT-2. This is a critical finding: on the larger model, CODI's trajectory-level supervision leaves more room for improvement via step-level grounding than it does on GPT-2. The paper notes that SIM-CoT+CODI achieves "96% of [SFT-CoT's] accuracy" (56.1% vs. 58.4%), characterizing this as "performance comparable to SFT-CoT" — a significant milestone for implicit reasoning on larger models.
SIM-CoT+Coconut (42.2%) vs. SIM-CoT+CODI (56.1%): The CODI-based version strongly outperforms the Coconut-based version by 13.9 points, reversing the pattern seen on GPT-2 where Coconut-based SIM-CoT was superior. This is consistent with the paper's claim that CODI's KL-regularized objective is necessary for larger models to prevent catastrophic forgetting during curriculum learning. SIM-CoT benefits from CODI's training stability while adding step-level supervision on top.

Out-of-domain, SIM-CoT+CODI achieves an average of 56.8% vs. CODI's 55.8% (+1.0 points), with improvements on all three benchmarks: GSM-Hard (12.7% vs. 11.9%, +0.8), MultiArith (96.2% vs. 95.0%, +1.2), SVAMP (61.5% vs. 60.6%, +0.9). The consistency across benchmarks suggests that step-level supervision provides a genuine generalization benefit rather than benchmark-specific effects.

Results on Larger LLaMA Models (Table 3)

Table 3 reports results for LLaMA 3.2 3B and LLaMA 3.1 8B, using CODI as the backbone (Coconut is not tested on these scales, consistent with the paper's observation about catastrophic forgetting in larger models with curriculum learning). The results confirm that SIM-CoT scales to larger models while maintaining consistent, though modest, gains:

LLaMA 3.2 3B: SIM-CoT+CODI achieves 62.3% vs. CODI's 60.8% (+1.5 points). This is smaller than the +3.4 point gain on 1B, suggesting diminishing returns from step-level supervision as model scale increases. On out-of-domain benchmarks, SIM-CoT improves SVAMP (74.9% vs. 73.3%, +1.6) and MultiArith (98.8% vs. 98.7%, +0.1), with no improvement on GSM-Hard (14.6% vs. 14.3%). Compared to SFT-CoT (71.5% in-domain), SIM-CoT reaches 87% of explicit CoT accuracy.
LLaMA 3.1 8B: SIM-CoT+CODI achieves 64.1% vs. CODI's 61.1% (+3.0 points). The gain is larger on 8B than on 3B (+3.0 vs. +1.5), which is somewhat counterintuitive — one might expect the benefit of step-level supervision to decrease monotonically with scale as the model's internal representations become richer. The paper does not comment on this non-monotonic pattern. On out-of-domain benchmarks, SIM-CoT improves MultiArith (100.0% vs. 99.5%, achieving perfect score), SVAMP (79.4% vs. 78.1%, +1.3), and GSM-Hard (16.3% vs. 15.5%, +0.8). Notably, SIM-CoT matches or exceeds SFT-CoT (71.7%) on MultiArith (100.0 vs. 98.3) and SVAMP (79.4 vs. 73.1), while underperforming on GSM-Hard (16.3 vs. 27.4) and in-domain (64.1 vs. 71.7).

The out-of-domain pattern — where SIM-CoT excels on MultiArith and SVAMP but not on GSM-Hard — is consistent across all model scales (GPT-2, 1B, 3B, 8B). GSM-Hard tests robustness to larger numbers, and the consistent underperformance suggests that implicit reasoning methods, even with step-level supervision, struggle with numerical generalization in ways that explicit CoT handles better (perhaps because explicit text generation allows the model to "write down" intermediate numerical values and recompute from them, while latent representations may lose precision in representing large numbers).

Token Efficiency (Tables 1–3, "# Tokens" columns)

Across all experiments, SIM-CoT maintains the same token efficiency as the implicit method it builds on — the auxiliary decoder is removed at inference, so SIM-CoT adds zero token overhead. The efficiency gains come from the implicit reasoning paradigm itself, not from SIM-CoT's innovation. The key comparisons:

GPT-2 (Table 1): SIM-CoT+Coconut uses 11.4 tokens vs. SFT-CoT's 24.7 tokens — a 2.2× speedup. SIM-CoT+CODI uses 12.6 tokens vs. SFT-CoT's 24.7 — a 2.0× speedup.
LLaMA 1B (Table 2): SIM-CoT+Coconut uses 11.9 tokens vs. SFT-CoT's 23.1 — a 1.9× speedup. SIM-CoT+CODI uses 13.4 tokens vs. 23.1 — a 1.7× speedup.
LLaMA 3B (Table 3): CODI/SIM-CoT uses 7.5 tokens vs. SFT-CoT's 22.4 — a 3.0× speedup. The larger speedup on 3B (compared to 1B) reflects that CODI's 3B configuration uses fewer implicit tokens (7.5 vs. 13.4), suggesting the model can reason in a more compressed latent space at larger scales.
LLaMA 8B (Table 3): Same 7.5 vs. 22.2 tokens — a 3.0× speedup.

The out-of-domain token counts show similar efficiency gains. The paper frames these as demonstrating that "SIM-CoT retains or even enhances the performance of explicit CoT while substantially reducing inference cost" (Section 4.2).

Scaling the Number of Implicit Tokens (Figure 3)

Figure 3 presents an ablation on GPT-2 comparing SIM-CoT+Coconut against Coconut as the number of implicit latents (and corresponding tokens) increases from 1 latent/2 tokens to 8 latents/16 tokens. The x-axis uses notation like "1-2" (1 latent, 2 tokens) through "8-16" (8 latents, 16 tokens). This experiment directly tests the paper's central claim about stability:

Coconut (orange line): Performance initially improves from 1 latent to approximately 3–4 latents, then degrades substantially at higher counts, with visible instability in the training process (the paper notes this in Section 2, referencing Figure 1a where training "becomes unstable and sometimes collapses" at 5 latents). The curve shows Coconut's well-known brittleness to latent count — too few latents provide insufficient reasoning capacity, too many cause collapse.
SIM-CoT+Coconut (blue line): Performance is consistently above Coconut across all latent counts and continues to improve or remain stable at counts where Coconut degrades. The gap widens at higher latent counts (4–8 latents), directly demonstrating that step-level supervision prevents the collapse that afflicts answer-level supervision. The paper emphasizes that "SIM-CoT provides more stable training and achieves consistent gains over Coconut, indicating that step-level implicit supervision scales effectively with larger latent capacity" (Section 4.3).

The figure shows results across the four benchmarks (GSM8k-Aug in-domain plus three out-of-domain). While the exact accuracy values vary by benchmark, the qualitative pattern — SIM-CoT above Coconut, with the gap widening at higher latent counts — is consistent across all four evaluation settings, confirming that the stability benefit generalizes beyond the training distribution.

Soft Thinking Integration (Table 5)

Table 5 reports an ablation where soft thinking (Zhang et al., 2025b; Wu et al., 2025) — a training-free method that constructs latent tokens as weighted mixtures of vocabulary embeddings — is combined with both Coconut and SIM-CoT on GPT-2. The results show:

Coconut + Soft Thinking: GSM8k-Aug: 36.7% (vs. 36.6% for Coconut alone), GSM-Hard: 8.3% (vs. 8.1%), MultiArith: 85.2% (vs. 83.5%), SVAMP: 36.0% (vs. 36.2%). The improvements are small and inconsistent across benchmarks, with SVAMP actually showing a slight decrease.
SIM-CoT + Soft Thinking: GSM8k-Aug: 45.0% (vs. 44.8%), GSM-Hard: 9.4% (vs. 9.3%), MultiArith: 91.5% (vs. 90.8%), SVAMP: 40.8% (vs. 40.7%). The improvements are consistently positive but very small (+0.1 to +0.7 points across benchmarks).

The paper interprets these results as showing that "soft thinking complements training-based implicit reasoning" and that "combining training-free construction with training-based supervision provides gains beyond either approach in isolation" (Appendix C.4). However, the effect sizes are minimal — the largest gain is +0.7 points on MultiArith for SIM-CoT — and it would be difficult to argue that soft thinking provides a practically meaningful benefit on top of SIM-CoT. The more important finding is implicit: SIM-CoT's training-based approach (44.8–45.0%) dramatically outperforms Coconut+Soft Thinking (36.0–36.7%), confirming that learned latent representations with step-level supervision are far more effective than vocabulary-space interpolation without training.

Ablation Studies and Robustness Checks

Number of implicit tokens: As shown in Figure 3 (discussed in the Main Results above), SIM-CoT maintains stable or improving performance from 1 to 8 latent tokens (2–16 actual tokens), while Coconut degrades at higher counts. This is the most important ablation because it directly validates the paper's central claim that step-level supervision prevents the latent instability that limits prior methods. The paper reports "best performance" for each configuration rather than final performance, which may overstate absolute accuracy but the relative pattern (SIM-CoT consistently above Coconut) is unlikely to be sensitive to this choice.

Decoder size: Table 4a reports an experiment on LLaMA 1B where the auxiliary decoder size is varied (1B, 3B, 8B parameters) while keeping the base model fixed at 1B. The baseline (no decoder, i.e., CODI alone) achieves 52.7% on GSM8k-Aug; adding a 1B decoder (SIM-CoT default) improves to 56.1%; upgrading to a 3B decoder drops to 50.4% (worse than baseline); an 8B decoder drops further to 50.0%. Out-of-domain benchmarks show the same inverted-U pattern: the 1B decoder provides consistent gains, while the 3B and 8B decoders degrade performance below the no-decoder baseline. This is a non-obvious finding with practical implications: larger decoders do not provide better supervision and can actively harm training. The paper hypothesizes two explanations: (1) "excessively large decoders may introduce optimization difficulties or misalignment with the backbone," and (2) "the 1B encoder and 1B decoder originate from the same model, thereby sharing a more compatible representation space that facilitates learning" (Appendix B). This finding supports the design choice of matching decoder capacity to base model size.

Model scale (GPT-2 vs. LLaMA 1B/3B/8B): The consistent improvements across model scales (Tables 1–3) serve as a robustness check for SIM-CoT's effectiveness: +8.2% on GPT-2 Coconut, +9.0% on LLaMA 1B Coconut, +1.5% on LLaMA 3B CODI, +3.0% on LLaMA 8B CODI. The gains are universally positive but non-monotonic in scale (larger on 1B than 3B, then larger again on 8B). The paper does not explain this non-monotonicity, which may reflect differences in backbone (Coconut vs. CODI), training hyperparameters (Table 6 shows different learning rates and epoch counts per model), or genuine scale-dependent effects of step-level supervision. The key robustness finding is that SIM-CoT never hurts performance — even the smallest gains (+0.6% on GPT-2 CODI, +1.5% on LLaMA 3B) are positive, and there are no reported cases where adding step-level supervision degrades accuracy.

Training-free augmentation (soft thinking): As discussed above (Table 5), adding soft thinking to SIM-CoT provides small, consistent improvements (+0.1 to +0.7 points), confirming that training-based and training-free approaches can be combined without negative interference. However, the gains are small enough that they could plausibly be within the range of run-to-run variance (not reported), making this a weak positive result rather than a strong confirmation of complementarity.

Backbone method (Coconut vs. CODI): SIM-CoT is tested on both Coconut and CODI across GPT-2 and LLaMA 1B (Tables 1–2), and on CODI alone for LLaMA 3B/8B (Table 3). The consistent improvements on both backbones confirm that SIM-CoT is genuinely plug-and-play — it does not depend on a specific implicit reasoning architecture. However, the interaction between backbone and SIM-CoT's effectiveness varies: on GPT-2, SIM-CoT benefits Coconut more than CODI (+8.2 vs. +0.6), while on LLaMA 1B, SIM-CoT benefits CODI substantially (+3.4) and Coconut even more (+9.0). This suggests that the choice of backbone matters for the magnitude of SIM-CoT's benefit, and that SIM-CoT may be most impactful when the backbone's supervision is coarsest (Coconut's answer-level supervision leaves the most room for improvement). The paper does not systematically investigate this interaction.

Geometric diagnostics (Table 4b): The paper provides geometric measurements of the latent space under different configurations as a form of representation-quality ablation. In the failed 5-latent case, inter-latent distance collapses to 4.21 and distance to vocabulary center spikes to 39.39. After applying SIM-CoT, inter-latent distance increases to 32.81 (higher than the healthy 5-latent baseline of 28.34) and vocabulary center distance stabilizes at 29.80 (comparable to healthy configurations). These measurements confirm that SIM-CoT restores the geometric properties associated with diverse, stable latent representations — high distinctiveness between latents combined with moderate anchoring to the vocabulary space. The paper does not report these metrics across different random seeds or training runs, so the stability of these geometric properties is uncertain.

Omitted ablation: loss weighting (λ_step vs. λ_lm). The paper does not report experiments varying the relative weight of the step-level and answer-level losses in Equation 8. This is a notable gap because the balance between the two terms could significantly affect the trade-off between step reconstruction fidelity and answer accuracy. The default weighting (presumably equal) may not be optimal, and understanding the sensitivity to this hyperparameter would strengthen the practical guidance for applying SIM-CoT.

Omitted ablation: decoder training from scratch vs. pretrained initialization. The paper states that the decoder is randomly initialized (not loaded from a pretrained checkpoint), but does not compare this against using a pretrained language model as the decoder. Given that the decoder is architecturally identical to the base model, initializing it from the same pretrained weights could potentially accelerate training or improve step reconstruction quality. The finding that decoder size matters (Table 4a) but initialization strategy is not explored leaves open the question of whether SIM-CoT's performance could be further improved with better decoder initialization.

Omitted ablation: cross-step context in the decoder. The paper's design choice to have each decoder process only a single latent z_k in isolation (Section 3.4) is motivated by the goal of enforcing per-latent semantic distinctiveness. However, the paper does not report an ablation where the decoder sees multiple latents or the full latent chain, which would test whether the independence constraint is actually necessary for preventing collapse or whether it merely provides a convenient inductive bias. This is a theoretically important ablation because it would distinguish between "any decoder-based supervision helps" and "specifically independent per-step supervision is required."

Critical Assessment

The paper makes several central claims, and the experiments support them to varying degrees. I assess each in turn.

Claim 1: SIM-CoT addresses a "latent instability issue" where scaling implicit tokens causes training collapse, and this collapse is caused by insufficient step-level supervision. The evidence for this claim is strong but incomplete. The diagnostic analysis in Figure 1 is thorough and convincing: it shows (a) training instability at 5 latents, (b) loss of operator information specifically, (c) geometric collapse of latent representations, and (d) semantic homogenization. The geometric measurements in Table 4b provide quantitative confirmation of the collapse signature (inter-latent distance dropping from 28.34 to 4.21, vocabulary distance spiking from 28.34 to 39.39). SIM-CoT's restoration of healthy geometric properties (inter-latent distance 32.81, vocabulary distance 29.80) and stable scaling to 8 latents (Figure 3) demonstrate that the intervention works as hypothesized.

However, the causal claim — that insufficiency of step-level supervision causes the collapse — is supported by correlation (adding supervision fixes the problem) rather than by a controlled experiment that varies only the supervision granularity while holding all else equal. Coconut, CODI, and SIM-CoT differ in more than just supervision granularity: they use different training objectives (answer-only cross-entropy vs. distillation vs. auxiliary decoder), different loss functions, and different optimization dynamics. The paper would be stronger with an experiment that adds the auxiliary decoder loss but with trajectory-level targets (i.e., the decoder generates the full reasoning chain from the combined latent sequence) to isolate whether it's the decoder mechanism or the per-step granularity that matters. The failure to run this ablation means the paper demonstrates that SIM-CoT works but doesn't conclusively prove that step-level granularity is the necessary and sufficient condition.

Claim 2: SIM-CoT is a plug-and-play module that improves existing implicit CoT methods. This claim is well-supported across multiple backbones (Coconut, CODI), multiple model scales (GPT-2, LLaMA 1B/3B/8B), and multiple benchmarks (in-domain + three out-of-domain). The improvements are universally positive — SIM-CoT never degrades performance relative to the backbone method. The plug-and-play nature is validated by the fact that SIM-CoT requires no architectural changes to the backbone and adds zero inference overhead.

The caveat is that the magnitude of improvement varies substantially across settings: from +8.2% on GPT-2 Coconut to +0.6% on GPT-2 CODI, from +9.0% on LLaMA 1B Coconut to +1.5% on LLaMA 3B CODI. The paper does not provide guidance on when to expect large vs. small gains, which limits the practical utility of the plug-and-play claim — a practitioner cannot know in advance whether SIM-CoT will provide a dramatic improvement or a marginal one for their specific model, dataset, and backbone. Additionally, SIM-CoT was only tested with autoregressive latent reasoning methods (Coconut, CODI) and not with other implicit reasoning paradigms (e.g., architectural modification methods like looped transformers, training-free methods as backbones), so the "plug-and-play" scope is narrower than the term implies.

Claim 3: SIM-CoT surpasses the explicit CoT baseline on GPT-2 by 2.1% with 2.3× greater token efficiency. This claim is factually correct based on Table 1: SIM-CoT+Coconut achieves 44.8% vs. SFT-CoT's 42.7% on the GSM8K-Aug in-domain test, with 11.4 vs. 24.7 tokens. The token efficiency claim is straightforward — the numbers are directly reported. However, the significance of surpassing explicit CoT requires qualification:

The result is on a single model (GPT-2) and a single benchmark (GSM8K-Aug in-domain). On larger models (LLaMA 1B, 3B, 8B), SIM-CoT does not surpass SFT-CoT in-domain, reaching at best 96% of explicit CoT accuracy (Table 2, 56.1% vs. 58.4%). On out-of-domain benchmarks, the picture is mixed — SIM-CoT sometimes surpasses SFT-CoT (e.g., MultiArith 100.0 vs. 98.3 on LLaMA 8B, Table 3) but sometimes underperforms substantially (GSM-Hard 16.3 vs. 27.4 on LLaMA 8B). The paper's claim of surpassing explicit CoT therefore holds specifically for the combination of GPT-2 scale and in-domain evaluation, not as a general property of the method.
The explicit CoT baseline uses the same training data (GSM8K-Aug) and the same base model (GPT-2). This is a fair comparison, but it's worth noting that SFT-CoT at 42.7% on GPT-2 is a relatively weak baseline — GPT-2 is a 124M parameter model that was not designed for mathematical reasoning. The fact that implicit reasoning can beat explicit CoT at this scale is interesting but may not generalize to models with stronger explicit reasoning capabilities.
The 2.3× token efficiency is measured in "token positions" through the transformer. In practice, the latency gain may be smaller or larger depending on implementation details (KV-caching, batch size, hardware). The paper does not report wall-clock time, so the practical speedup is uncertain.

Claim 4: SIM-CoT provides interpretability of implicit reasoning by projecting latent tokens onto explicit reasoning vocabulary. This claim is supported by qualitative case studies (Figure 4, Figure 7) showing that the decoder can produce human-readable reasoning steps from latent tokens. The case studies are convincing as existence proofs: it is possible to decode implicit reasoning into interpretable text. However, the paper provides no quantitative evaluation of this interpretability:

No measure of decoding fidelity: What fraction of decoded steps are mathematically correct? What fraction are consistent with the final answer? If the decoder produces plausible-looking but incorrect reasoning (e.g., correct arithmetic operations applied to wrong numbers), the interpretability could be misleading rather than helpful.
No comparison to ground-truth reasoning: The case studies in Figures 4 and 7 show decoded steps that look reasonable, but the paper doesn't systematically compare decoded steps against ground-truth reasoning annotations to measure reconstruction accuracy.
No evaluation of whether decoded steps explain errors: If the model produces a wrong answer, do the decoded steps reveal why the error occurred (e.g., a specific step was miscalculated), or do they show plausible reasoning that doesn't align with the model's actual computation? The paper doesn't test this diagnostic use case.

The interpretability claim is thus qualitatively supported but quantitatively unvalidated. A stronger evaluation would measure the BLEU score or exact-match accuracy of decoded steps against ground-truth reasoning, or would conduct a human study to assess whether the decoded steps help users understand and trust model outputs.

Claim 5: SIM-CoT remains stable when scaled to 8 or 16 implicit tokens where prior methods collapse. Figure 3 directly supports this claim for scaling to 8 latents (16 tokens) on GPT-2: SIM-CoT maintains accuracy while Coconut degrades. The paper does not report results for 16 latents (32 tokens) — the maximum tested is 8 latents (16 tokens). The claim about "16 implicit tokens" in the abstract ("while previous implicit CoT approaches (e.g., Coconut) collapse when scaled to 8 or 16 implicit tokens") refers to 16 token positions (8 latents × 2 tokens per latent), not 16 latents. This is potentially confusing because the paper alternates between counting "latents" and "tokens" (where 1 latent = 2 tokens). The key result is stability at 8 latents/16 tokens, where Coconut collapses. Scaling to even higher latent counts (e.g., 12 or 16 latents) is not tested, so the upper bound of SIM-CoT's stability is unknown.

Additional weaknesses in the experimental design:

Single training dataset (GSM8K-Aug). All experiments train on GSM8K-Aug and evaluate on math reasoning benchmarks. This is a natural choice for studying reasoning, but it means the findings may not generalize to other reasoning domains (code generation, logical deduction, scientific reasoning) or to tasks where explicit step annotations are harder to obtain. GSM8K-Aug's structured mathematical expressions (<<12*3=36>>) provide clean, easily parsed step boundaries — real-world reasoning tasks may have fuzzier step delineations.
Small test sets for some benchmarks. GSM-Hard has approximately 1,319 examples and MultiArith has 600 examples. Performance differences of 1–2 percentage points on these benchmarks represent very small absolute numbers of questions (e.g., +1.2 points on MultiArith for SIM-CoT+CODI on LLaMA 1B represents roughly 7 additional correct answers out of 600). Without confidence intervals, it's difficult to determine whether these differences are statistically reliable.
No multiple random seeds or error bars. All results are reported as single-point estimates. The training of implicit reasoning models with curriculum learning is known to be sensitive to random seeds, and the absence of variance estimates makes it impossible to assess whether the reported improvements (especially the smaller ones, like +0.6% on GPT-2 CODI or +1.5% on LLaMA 3B) are robust or within the range of run-to-run variation.
The oracle problem in step annotation. SIM-CoT requires step-level ground-truth annotations (s_1, ..., s_K) for training. In GSM8K-Aug, these are provided by the dataset's structured mathematical expressions. On datasets without such annotations, obtaining step-level supervision would require either manual annotation (expensive) or automatic parsing (potentially noisy). The paper does not discuss the cost or feasibility of obtaining step-level annotations for new domains, which limits the practical applicability of the method beyond curated reasoning benchmarks.
No comparison to simply training with more explicit CoT data. SIM-CoT outperforms SFT-CoT on GPT-2, but the comparison is between methods trained on the same GSM8K-Aug data. An alternative approach to closing the implicit-explicit gap would be to train explicit CoT on more data or with better data augmentation. The paper doesn't establish whether SIM-CoT's gains are larger than what could be achieved by simply improving the explicit CoT baseline with the same engineering effort.
Decoder training cost not accounted for. SIM-CoT requires training an auxiliary transformer decoder during training, which doubles the parameter count being optimized and adds the computational cost of the decoder's forward and backward passes. The paper emphasizes that the decoder is removed at inference (so inference cost is identical to standard implicit methods), but the training-time computational overhead is not quantified. For resource-constrained settings where training cost matters, this overhead could be significant.
Missing experiment on dynamic K. The paper uses a fixed number of implicit latents K set before training (or determined by curriculum). Problems with fewer reasoning steps than K generate redundant latents that "simply repeat the final prediction" (Appendix K). An experiment with dynamic K (e.g., training the model to emit a "stop reasoning" signal) would test whether the fixed-K approach wastes computation and whether adaptive latent counts could further improve efficiency. The paper doesn't explore this.

Overall, the experiments convincingly demonstrate that SIM-CoT improves implicit reasoning accuracy and stability across a range of settings, but the magnitude and generality of these improvements are more limited than the abstract's framing suggests. The method works best when the backbone's supervision is weakest (large gains on Coconut, small gains on CODI), when the model is smaller (GPT-2 > LLaMA 3B), and on in-domain or structurally similar out-of-domain data (large gains on SVAMP, small or no gains on GSM-Hard). The central diagnostic contribution — identifying and measuring latent collapse — is well-supported and may prove more impactful than the specific SIM-CoT method itself.

6. Limitations and Trade-offs

Limitation 1: Step-Level Annotations Are Required for Training and May Not Be Available in New Domains

SIM-CoT's training procedure requires explicit, step-by-step reasoning annotations (s_1, ..., s_K) for every training example — each implicit latent z_k must be paired with a corresponding textual reasoning step to compute the decoder's cross-entropy loss (Equation 6). In the paper's experiments, these annotations come from GSM8K-Aug's structured mathematical expressions (<<12*3=36>><<9*2=18>>), where step boundaries are cleanly delineated by the angle-bracket notation and each expression represents a single arithmetic operation. This is a highly favorable setting: step boundaries are unambiguous, steps are semantically self-contained (each expression can be understood in isolation), and the annotations are machine-generated at scale (385k examples via GPT-4 augmentation).

The paper does not discuss the cost, feasibility, or methodological challenges of obtaining comparable step-level annotations for other reasoning domains. In settings where such annotations are absent — code generation (where reasoning steps interleave with code blocks and natural language), multi-hop question answering (where steps are logical deductions rather than arithmetic expressions), scientific reasoning (where steps may involve qualitative inference rather than discrete operations), or any domain without pre-existing structured reasoning traces — the method cannot be applied directly. The authors acknowledge this implicitly by never testing on a benchmark without pre-segmented reasoning steps, but do not discuss it as a limitation.

The consequence is that SIM-CoT's applicability is tied to the availability of structured, step-segmented reasoning data. A practitioner wanting to apply the method to a new domain faces a significant data engineering challenge: they must either manually annotate step boundaries (expensive for large datasets), develop automatic segmentation heuristics (potentially noisy, and the paper provides no guidance on what constitutes a valid step boundary), or use a teacher model to generate step annotations (which reintroduces the dependency on explicit CoT that SIM-CoT aims to circumvent). The paper's claim that SIM-CoT is "plug-and-play" (Section 1, abstract) refers to its compatibility with different implicit reasoning backbones (Coconut, CODI), not to its data requirements — a distinction that may not be obvious to readers and could lead to overestimation of the method's ease of adoption.

No experiments in the paper test SIM-CoT in a setting where step annotations are noisy, automatically generated, or differently structured. The ablation on decoder size (Table 4a) varies decoder capacity but does not vary annotation quality. The out-of-domain evaluations (GSM-Hard, MultiArith, SVAMP) test generalization of the trained model to new problems, but all models are trained with the same high-quality step annotations from GSM8K-Aug, so the experiments provide no evidence about how annotation quality or availability affects training.

The paper does not attempt to mitigate this limitation, and the authors do not propose solutions for obtaining step annotations in new domains. The "Future Directions" (Appendix J) focus on multimodal extension, multi-path reasoning, RLHF integration, and theoretical foundations — none of which address the data annotation bottleneck. This is a significant gap because it means SIM-CoT's core benefit (stabilizing implicit reasoning through step-level supervision) is only accessible in domains that already have structured reasoning traces — precisely the domains where explicit CoT is most effective and where the efficiency argument for implicit reasoning is strongest. In domains without such traces, the cost of creating them may outweigh the inference-time efficiency gains.

Limitation 2: Difficulty Estimation Cost for Step Annotation Is Unaccounted for in Training Budget

Even when step-level annotations exist (as in GSM8K-Aug), SIM-CoT introduces substantial training-time computational overhead that is not captured by the paper's headline efficiency metrics. The paper emphasizes inference-time token savings (2.3× fewer tokens than SFT-CoT on GPT-2, Table 1) and zero inference overhead from the discarded decoder. However, the training process requires:

Training an auxiliary decoder p_φ with the same architectural capacity as the base model (the paper uses a 1B decoder for a 1B base model, per Appendix B). This approximately doubles the parameter count being optimized during training and adds the full forward and backward pass of the decoder to each training step.
Computing per-step decoder losses for all K steps on every training example. For a 4-step problem with 5 tokens per step, this means 20 additional autoregressive generation steps through the decoder per training example, each with its own forward and backward pass.
The decoder must generate full step sequences from each latent — unlike the base model, which only generates a single answer sequence, the decoder generates K separate sequences (one per step), increasing the total generation length during training by a factor proportional to the number of reasoning steps.

The paper does not report training wall-clock time, GPU memory usage, or FLOPs for SIM-CoT compared to Coconut or CODI. The training efficiency cost is entirely unquantified. This matters for practitioners because the training overhead may be prohibitive in resource-constrained settings. Coconut already requires a curriculum training schedule (incrementally increasing K from 0 to K_max, then training for 15 additional epochs — Appendix E.1). Adding a same-scale decoder effectively doubles the per-step computation during each of these epochs. For the LLaMA 8B experiments (Table 3), this means training an additional 8B-parameter decoder alongside the 8B base model — a substantial computational investment.

The consequence is that SIM-CoT's practical efficiency gains are less favorable than the inference-time token counts suggest when total computational cost (training + inference) is considered. This is a well-known trade-off in methods that shift computation from inference to training (knowledge distillation, model compression), but SIM-CoT adds training cost without reducing inference cost relative to the implicit reasoning backbone — it only prevents the degradation that would occur without it. A practitioner must invest additional training compute to achieve accuracy that Coconut cannot reach, and the paper provides no cost-benefit analysis to inform this decision. For deployment scenarios where training cost is amortized over many inference queries (e.g., a deployed API serving millions of requests), the training overhead may be negligible. For scenarios where models are frequently retrained or where training resources are the primary constraint, the overhead could be decisive.

The authors do not acknowledge this limitation. The only efficiency numbers reported are inference-time token counts ("# Tokens" columns in Tables 1–3) and token efficiency ratios ("2.3× greater token efficiency," abstract). The training procedure is described in Appendices E–G without any cost quantification. The ablation on decoder size (Table 4a) tests whether larger decoders improve accuracy, but does not report the training cost scaling. The decoder's training cost is treated as an implementation detail rather than a first-class trade-off.

The paper does not propose methods to reduce training overhead, such as training the decoder on only a subset of steps, using a smaller decoder (the 1B decoder works well; whether a smaller decoder would suffice is not tested), or sharing layers between the base model and the decoder (they share embeddings E but not transformer parameters). These are natural directions for making the method more training-efficient, but they are not explored.

Limitation 3: SIM-CoT Does Not Help on the Hardest Problems and Degrades Relative to Explicit CoT on Numerical Generalization

The paper's difficulty-binned analysis is less granular than the framework we saw in the Coconut scaling laws paper, but the pattern is clear from the out-of-domain benchmarks: SIM-CoT provides large gains on benchmarks testing robustness to superficial variations (SVAMP: +7.3 points on GPT-2 Coconut, Table 1) but minimal or no gains on benchmarks testing harder numerical reasoning (GSM-Hard: 0.0 point improvement on GPT-2 Coconut, 12.2% vs. 12.2%). This pattern persists across model scales:

GPT-2 (Table 1): SIM-CoT+Coconut vs. Coconut on GSM-Hard: 12.2% vs. 12.2% (no improvement). On MultiArith: 9.3% vs. 8.1% (+1.2). On SVAMP: 90.8% vs. 83.5% (+7.3).
LLaMA 1B (Table 2): SIM-CoT+CODI vs. CODI on GSM-Hard: 12.7% vs. 11.9% (+0.8). On MultiArith: 96.2% vs. 95.0% (+1.2). On SVAMP: 61.5% vs. 60.6% (+0.9).
LLaMA 8B (Table 3): SIM-CoT+CODI vs. SFT-CoT on GSM-Hard: 16.3% vs. 27.4% (−11.1 point deficit to explicit CoT). On MultiArith: 100.0% vs. 98.3% (+1.7 over explicit CoT). On SVAMP: 79.4% vs. 73.1% (+6.3 over explicit CoT).

The contrast between GSM-Hard (where numbers are replaced with larger magnitudes) and SVAMP (where problem wording is varied) reveals a specific failure mode: SIM-CoT's latent representations may not robustly encode numerical precision. Explicit CoT can "write down" intermediate numerical results as discrete tokens, which provides a form of error correction — the model can attend to the written number and recompute from it. Implicit reasoning, even with step-level supervision, must carry numerical values through continuous vectors, which may lose precision or fail to generalize to number magnitudes not seen during training. The paper does not analyze this hypothesis or measure numerical precision in the latent representations.

The consequence is that SIM-CoT's benefits are concentrated on problems that are structurally similar to training examples, while the most challenging generalization — to harder numerical values — sees no improvement or even degradation relative to explicit CoT. For a practitioner, this means SIM-CoT is most useful when the deployment distribution is similar to the training distribution in numerical complexity, and least useful when problems may involve numbers outside the training range. This is particularly relevant for educational applications where problem difficulty is varied by changing numerical values, or for financial/scientific applications where numbers span orders of magnitude.

The paper does not explicitly analyze this failure mode. The out-of-domain results are reported in aggregate (average across three benchmarks) and discussed in general terms ("SIM-CoT consistently outperforms SFT-CoT, with an average improvement of +4.3 points," Section 4.2), which obscures the flat GSM-Hard performance. The per-benchmark breakdown is available in Tables 1–3 but is not highlighted or discussed as a limitation. The geometric diagnostics (Table 4b, Figure 6) measure inter-latent distance and vocabulary center distance, but do not measure numerical encoding fidelity or robustness to value shifts.

The paper does not attempt to mitigate this limitation. There is no experiment explicitly targeting numerical generalization, no analysis of latent representation sensitivity to input number magnitudes, and no suggestion for how implicit reasoning methods might better handle numerical variation. This is a missed opportunity given that the GSM8K-Aug training data itself includes numerical resampling (the augmentation process varies numbers), which would have enabled a controlled study of numerical generalization.

Limitation 4: All Results Are on a Single Task Family (Grade-School Math) with a Single Training Dataset

The paper's entire empirical evaluation — training, in-domain testing, and out-of-domain testing — operates within the narrow domain of grade-school arithmetic word problems. The training data is GSM8K-Aug (augmented from GSM8K). The in-domain test is GSM8K-Aug test split. The out-of-domain tests are SVAMP (elementary arithmetic with wording variations), GSM-Hard (GSM8K with harder numbers), and MultiArith (multi-step arithmetic) — all of which are variations on the same underlying task: read a short word problem, identify the arithmetic operations needed, compute the answer. There is no evaluation on qualitatively different reasoning tasks: no code generation, no logical deduction, no commonsense reasoning, no multi-hop QA, no symbolic manipulation, no theorem proving.

This is acknowledged implicitly by the paper's scope (the abstract and introduction frame the work around mathematical reasoning) but never discussed as a limitation. The authors do not claim that SIM-CoT generalizes to other domains, but they also do not provide any evidence about domain specificity. The key question for a practitioner is: are the findings about latent instability, step-level supervision, and geometric collapse specific to arithmetic reasoning, or do they reflect fundamental properties of implicit reasoning that would appear in any domain?

The consequence of this narrow evaluation is uncertainty about SIM-CoT's broader applicability. Several features of arithmetic reasoning could make it an especially favorable domain for SIM-CoT:

Clean step boundaries: Arithmetic steps are discrete operations (add, subtract, multiply, divide) with clear inputs and outputs, making step segmentation straightforward.
Compositional structure: Arithmetic reasoning is strictly sequential (each step's output feeds into the next step's input), matching SIM-CoT's autoregressive latent generation structure.
Fixed vocabulary: Arithmetic operators and numbers form a constrained vocabulary, which may make the decoder's step reconstruction task easier than in domains with open-ended reasoning language.
Objective correctness: Answers are single numbers, making evaluation unambiguous. This is not true for many reasoning domains (summarization, argument generation, creative problem-solving).

In domains with more fluid reasoning structure — e.g., legal reasoning where steps involve citing precedents and weighing evidence, or medical reasoning where steps involve differential diagnosis and test interpretation — the notion of a "reasoning step" is fuzzier, step boundaries are harder to define, and the mapping from latent to text may require generating much longer and more varied text per step. SIM-CoT's architecture (one latent vector decoding to one step) assumes each step can be compressed into a single d-dimensional vector, which may not hold for complex, information-rich reasoning steps. The paper provides no evidence about whether the method would work when steps are longer, more variable, or less self-contained.

The paper does not propose validation on other datasets or domains as future work. The "Future Directions" (Appendix J) mention multimodal extension, multi-path reasoning, RLHF integration, and theoretical foundations, but do not mention broader reasoning domain evaluation. This is a notable gap because expansion to new domains would be the natural next step for demonstrating the method's generality, and its absence suggests the authors may view SIM-CoT as primarily a math reasoning method rather than a general reasoning method.

Limitation 5: No Statistical Significance or Variance Estimates Are Reported for Any Result

The paper reports all accuracy numbers as single-point estimates with no confidence intervals, error bars, standard deviations, or statistical tests. The test sets are modest in size: GSM8K-Aug test split (size not explicitly stated but the original GSM8K test set has 1,319 examples; the augmented version may differ), SVAMP (1,000 examples), GSM-Hard (approximately 1,319 examples), and MultiArith (600 examples). On these test sets, the reported improvements range from large (+8.2% on GPT-2 Coconut, Table 1) to very small (+0.1 to +0.7 points for soft thinking integration, Table 5; +0.6% on GPT-2 CODI, Table 1; +0.3% on GPT-2 CODI out-of-domain, Table 1).

For the smaller improvements, the absence of variance estimates is a significant concern. The training of implicit reasoning models involves curriculum learning (Appendix E.1), random initialization of the auxiliary decoder, and sensitive hyperparameters (learning rates ranging from 1 × 10^{-4} to 3 × 10^{-3} across models, Table 6). These factors introduce run-to-run variance that could easily account for differences of 0.5–1.0 percentage points. Without multiple runs or statistical testing, a reader cannot determine whether, for example, SIM-CoT's +0.6% improvement on GPT-2 CODI (Table 1) reflects a genuine benefit of step-level supervision or is within the range of training variance. Similarly, the claim that soft thinking provides "consistent improvements" (Appendix C.4) is based on differences of +0.1 to +0.7 points that could be noise.

The consequence is that the paper's more modest claims — particularly the CODI-based improvements on GPT-2 (+0.6% in-domain, +0.3% out-of-domain) and the soft thinking ablation — are not statistically reliable based on the evidence presented. A practitioner trying to decide whether the additional training complexity of SIM-CoT is worthwhile for a CODI-based system would want to know whether the expected gain is reliably positive or could be zero in practice. The paper's evidence does not answer this question.

This limitation also affects the geometric diagnostics (Table 4b). The inter-latent distance and vocabulary center distance are reported as single values (e.g., "32.81" for inter-latent distance after SIM-CoT). Without variance estimates, it's unclear whether these values are stable across training runs or whether the difference between the healthy 5-latent baseline (28.34) and the SIM-CoT value (32.81) is reliable. The geometric measurements are a key part of the paper's diagnostic framework, and uncertainty about their stability weakens their utility as monitoring tools.

The paper does not acknowledge this limitation. No mention is made of statistical significance, confidence intervals, or multiple random seeds anywhere in the main text or appendices. The training procedure descriptions (Appendix E) specify hyperparameters in detail but do not mention random seed values or whether results are averaged across runs. This is consistent with common practice in the implicit reasoning literature (Coconut and CODI also report single-point estimates), but it limits the strength of the conclusions that can be drawn from the smaller effect sizes.

The paper does not attempt to mitigate this limitation. Running multiple seeds for the key comparisons (SIM-CoT vs. backbone on each model scale) and reporting means with standard deviations would require modest additional compute (3–5 runs per configuration) and would substantially strengthen the reliability of the conclusions. The absence of this analysis is particularly notable given that the paper's primary contribution is demonstrating a stabilizing effect — measuring variance would directly test whether SIM-CoT not only improves mean accuracy but also reduces run-to-run variance in training outcomes, which would be strong evidence for the claimed stability benefit.

Limitation 6: The Decoder's Interpretability Is Unvalidated — Decoded Steps May Not Faithfully Represent Internal Reasoning

SIM-CoT's auxiliary decoder provides a mechanism for projecting implicit latent tokens into human-readable reasoning steps, which the paper presents as a key benefit: "It also provides interpretability by projecting each latent token onto an explicit reasoning vocabulary, enabling per-step visualization and diagnosis" (abstract). The case studies in Figure 4 and Appendix K (Figure 7) show examples where decoded steps correspond to plausible reasoning operations. However, the paper provides no quantitative evaluation of the fidelity, accuracy, or diagnostic utility of these decoded steps.

The specific unvalidated claims are:

Semantic fidelity: Do the decoded steps accurately represent what the latent tokens encode, or does the decoder learn to produce plausible-looking text from latents that encode different (or no) information? Since the decoder is trained to minimize cross-entropy on step tokens, it will learn to generate text that maximizes likelihood given each latent — but this is a generative model, not a veridical interpretation. A collapsed latent that encodes only numbers might still be decoded into an operator-containing step if the decoder has learned to "hallucinate" the missing operator information from the number pattern (e.g., seeing "12, 3" and generating "12 × 3 = 36" because that pattern is common in the training data).
Diagnostic utility for errors: When the model produces a wrong final answer, do the decoded steps reveal where the reasoning failed? The paper shows only correct examples in the case studies. There is no demonstration of using decoded steps to diagnose an error — e.g., showing that step 2 decoded to a wrong operation, or that step 3 used a number from step 1 instead of step 2.
Consistency with the final answer: Are the decoded steps mathematically consistent with the model's final answer? If the model computes the answer through a different path than what the decoder produces, the decoded steps would be misleading as an explanation. The paper does not measure step-answer consistency.
Comparison to ground-truth reasoning: For examples where ground-truth step annotations exist (the training data has them), the paper does not report the exact-match accuracy or BLEU score of decoded steps against the ground truth. This would be a straightforward evaluation but is absent.

The consequence is that the interpretability claim — while plausible based on the qualitative examples — remains unsubstantiated. A practitioner who adopts SIM-CoT for the interpretability benefit (e.g., to build user trust by showing intermediate reasoning, or to debug model errors in production) has no evidence about whether the decoded steps are trustworthy. Worse, plausible-looking but incorrect decoded steps could be actively misleading — a user might trust a model more because they see "reasonable" reasoning steps, not realizing those steps don't reflect the model's actual computation. This is a known failure mode in explainable AI where post-hoc explanations can increase trust without increasing explanation fidelity (the "illusion of explanatory depth").

The paper's geometric diagnostics (Table 4b, Figure 6) partially address this concern by showing that SIM-CoT's latents are more diverse and better anchored to the vocabulary space than collapsed latents. However, this is indirect evidence: diverse, vocabulary-proximate latents are more likely to encode recoverable semantic information, but the geometric metrics don't measure whether the decoder actually recovers the correct semantic information. A latent could have healthy geometric properties (high inter-latent distance, moderate vocabulary distance) yet still encode information that the decoder misinterprets (e.g., encoding "multiply" as a vector that the decoder maps to "add" due to training data biases).

The paper does not acknowledge this limitation. The interpretability section (Section 4.3, Appendix I) describes the decoding mechanism and shows qualitative examples, but does not discuss the possibility of decoder hallucination or unfaithful explanation. The claim that SIM-CoT "affords interpretability" (Section 1) is stated as an accomplished fact rather than a hypothesis requiring validation.

The paper does not propose validation methods for interpretability. A minimal validation would involve: (1) measuring exact-match accuracy of decoded steps against ground-truth step annotations on a held-out set, (2) measuring whether decoded steps are mathematically consistent with the model's final answer (do the operations in the decoded steps, when computed, produce the final answer?), and (3) testing whether decoded steps for incorrect answers reveal distinct error patterns compared to decoded steps for correct answers. These evaluations could be done with the existing GSM8K-Aug data and would substantially strengthen the interpretability claim. The paper's "Future Directions" (Appendix J) mention "integration with RLHF" and "theoretical foundations" but do not mention interpretability validation.

7. Implications and Future Directions

How This Work Changes the Landscape

SIM-CoT's primary conceptual contribution is a reframing rather than a paradigm shift: it recasts the implicit chain-of-thought instability problem from an architectural or optimization challenge into a supervision granularity problem. Prior to this work, the dominant response to implicit CoT's limitations was to seek better architectures — looped transformers (Saunshi et al., 2025), recurrent depth scaling (Geiping et al., 2025), or architectural modifications that add dedicated latent processing capacity (Chen et al., 2025; Cheng & Van Durme, 2024). The implicit assumption was that autoregressive latent generation as implemented in Coconut (Hao et al., 2025) was fundamentally limited, and that making implicit reasoning work would require changing how latents are generated, not just what signal guides them.

SIM-CoT challenges this assumption with a clean negative result: the autoregressive latent generation mechanism is not the bottleneck. By keeping the inference architecture identical to Coconut and intervening only in the training signal — adding per-step textual grounding through an auxiliary decoder that is discarded at inference — the paper achieves stability at 8 latent tokens where Coconut collapses and surpasses explicit CoT on GPT-2 by 2.1 percentage points (Table 1). The mechanism of latent generation (Equation 1: z_k = H_θ(U^{(k-1)})) is unchanged. The improvement comes entirely from what each z_k is taught to encode, not from how it is computed.

This reframing redirects research attention from architecture design to supervision design. The paper's diagnostic framework (Section 2) provides the language for this shift: rather than asking "does the model architecture support multi-step latent computation?", researchers can now ask "does the training signal maintain inter-latent distance above collapse threshold and vocabulary distance within the stable range?" The specific thresholds from Table 4b — inter-latent distance collapsing from 28.34 to 4.21 in failed models, vocabulary center distance spiking from 28.34 to 39.39 — give concrete, measurable targets for evaluating any implicit reasoning method, regardless of architecture.

Reconciling contradictory findings in the implicit reasoning literature. The paper indirectly reconciles a tension that has been present but unarticulated in the literature: why do some implicit reasoning methods show promising scaling trends (Coconut improving from 1 to 3 latents in Figure 3) while others report catastrophic degradation at longer chains (the paper's own Figure 1a showing collapse at 5 latents)? SIM-CoT's diagnostic framework provides the reconciliation: these are not contradictory findings about latent reasoning capacity — they are observations of the same underlying phenomenon (latent collapse) at different points on the supervision granularity spectrum. Answer-level supervision (Coconut) provides sufficient signal for short chains (1–3 latents) where the model can distribute distinct information across a small number of latent positions purely from the answer-level gradient, but fails beyond this threshold because the gradient becomes too diluted to enforce per-position semantic distinctiveness. Trajectory-level supervision (CODI) pushes the collapse threshold higher but doesn't eliminate it because coarse trajectory alignment still permits redundant information distribution across latents. Step-level supervision (SIM-CoT) eliminates the collapse entirely within the tested range (up to 8 latents) because the per-step reconstruction loss directly penalizes redundancy. The paper doesn't frame this reconciliation explicitly, but it is a direct consequence of the supervision granularity analysis.

Which research directions become more attractive. The paper's findings make supervision design for latent representations a high-priority research area. Specific directions that gain plausibility include: (a) developing methods to automatically segment reasoning traces into semantically meaningful steps without human annotation (making SIM-CoT's step-level supervision cheaper to obtain), (b) exploring intermediate granularities — e.g., grouping reasoning steps into "subgoals" and supervising at the subgoal level rather than the individual operation level, testing whether the stability benefit saturates before full per-step granularity is reached, (c) designing dynamic supervision strategies that allocate more supervision to earlier latent positions (where errors compound) and less to later positions (which are closer to the answer and may receive stronger answer-level signal), and (d) investigating whether step-level supervision enables latent reasoning to scale to chain lengths far beyond what is tested in this paper (e.g., 20–50 latent steps), potentially making implicit reasoning viable for tasks like multi-hop QA or long-form code generation.

Which research directions become less attractive. The paper's results reduce the case for purely architectural approaches to latent reasoning stability. If step-level supervision alone can prevent collapse without any architectural modification, then architectural innovations that aim to solve the same problem (e.g., adding recurrence to latent processing, designing specialized latent attention patterns, or introducing separate latent processing modules) may be solving a problem that is better addressed through supervision design. This does not mean architectural work on latent reasoning is obsolete — the paper does not test whether SIM-CoT combined with architectural innovations could push performance even higher, and there may be ceilings that supervision alone cannot break — but it does mean that future architectural proposals should demonstrate benefits above and beyond what simpler supervision improvements (like step-level grounding) can achieve. The default baseline for new implicit reasoning architectures should be SIM-CoT-style supervision, not Coconut-style answer-only training.

The significance of surpassing explicit CoT, and its limits. The result that SIM-CoT outperforms explicit CoT on GPT-2 (44.8% vs. 42.7%, Table 1) is methodologically important because it disproves the hypothesis that implicit reasoning is fundamentally information-lossy relative to explicit verbalization. Prior work had consistently found implicit methods underperforming explicit CoT, leading to an implicit assumption that compressing reasoning into continuous vectors inevitably sacrifices some reasoning fidelity. SIM-CoT's result shows this is not a necessary trade-off: with sufficient per-step supervision, the continuous latent space can be a better medium for reasoning than discrete text, at least at the GPT-2 scale. This has a subtle but important implication: it suggests that the fixed vocabulary of natural language may constrain reasoning by forcing the model to commit to specific token sequences that are optimized for human communication rather than computational efficiency. The continuous latent space, by contrast, can represent numerical relationships, intermediate values, and operation types in forms that don't correspond to any single token, potentially enabling more compact and more accurate computation.

However, the fact that this advantage does not persist at larger scales (SIM-CoT reaches 96% of SFT-CoT on LLaMA 1B, Table 2; 87% on LLaMA 3B; 89% on LLaMA 8B, Table 3) tempers the implication. The paper does not explain this scaling-dependent reversal, but a plausible hypothesis is that larger models have stronger language modeling capabilities that make explicit verbalization less constraining — they can fluently generate diverse reasoning paths and are less hampered by the "fixed vocabulary" limitation. The implicit reasoning advantage may be most pronounced when model capacity is constrained relative to task complexity, making latent compression a genuine efficiency gain rather than a convenience. Testing this hypothesis would require systematically evaluating SIM-CoT against explicit CoT across a wider range of model scales and task difficulties, which the paper does not do.

The geometric diagnostic framework as a lasting contribution. Perhaps the most durable contribution of the paper is not SIM-CoT itself but the measurable signature of latent collapse it introduces: inter-latent distance (Equation 9) and distance to vocabulary center (Equation 10). These metrics are architecture-agnostic, supervision-agnostic, and computable from any autoregressive latent reasoning model during or after training. They transform "the model collapsed" from a subjective judgment into a quantitative diagnosis. The paper demonstrates their utility by tracking them across configurations (Table 4b) and showing that SIM-CoT restores healthy values (inter-latent distance 32.81, vocabulary distance 29.80). Future work on implicit reasoning — regardless of architecture or supervision strategy — can and should report these metrics alongside accuracy, enabling the field to build a shared understanding of what geometric properties are necessary for effective latent computation. This is analogous to how the deep learning community uses gradient norm and activation statistics to diagnose training instabilities, and it may prove more impactful than any single method.

Follow-Up Research This Work Enables

Quantifying the supervision granularity spectrum: at what granularity does the stability benefit saturate? SIM-CoT demonstrates that step-level supervision prevents collapse where answer-level and trajectory-level supervision fail, but the paper does not explore intermediate points on the granularity spectrum. A natural follow-up experiment would systematically vary supervision granularity while holding all other factors constant: train models where the decoder loss groups reasoning steps at different resolutions — individual steps (SIM-CoT default), pairs of consecutive steps, subgoals (groupings of 2–3 steps that form a logical unit), and full trajectories (approximating CODI's distillation but through the same decoder mechanism). The dependent variables would be: (a) maximum stable latent count, (b) inter-latent distance and vocabulary center distance at various latent counts, (c) in-domain and out-of-domain accuracy. The key question is whether the stability benefit of step-level supervision is linear (each increment of granularity provides proportional benefit) or exhibits a threshold effect (per-step supervision is necessary; anything coarser fails). If a threshold exists at the individual-step level, it would validate SIM-CoT's design choice as necessary rather than merely beneficial. If intermediate granularities (e.g., subgoal-level) provide similar stability with lower annotation cost, it would open a more practical path to applying the method in domains where clean step boundaries are unavailable. GSM8K-Aug's structured expressions make this experiment straightforward: the dataset already contains step-level annotations that can be grouped into coarser units by simple concatenation.

Testing whether decoder fidelity correlates with reasoning accuracy: can decoded steps be trusted? The paper claims interpretability as a benefit of SIM-CoT but provides no quantitative validation of decoder fidelity. A rigorous follow-up would measure: (a) exact-match accuracy of decoded steps against ground-truth reasoning annotations on the GSM8K-Aug test set, (b) mathematical consistency between decoded steps and the model's final answer (do the operations and values in the decoded steps, when computed, yield the final answer?), and (c) whether decoded steps for incorrect answers exhibit systematic error patterns (e.g., step 3 consistently decodes to the wrong operation when the problem involves subtraction). The last point is particularly important for the claimed diagnostic utility: if decoded steps for wrong answers show plausible-looking but incorrect reasoning that doesn't actually reflect what went wrong, then the interpretability is misleading rather than helpful. Conversely, if error patterns in decoded steps correlate with ground-truth error types (wrong operation vs. wrong number vs. wrong step ordering), then the decoder provides genuine diagnostic value. This experiment would require comparing decoded step sequences against ground-truth annotations on a per-step basis, which is feasible with GSM8K-Aug's structured format. The strong version of the interpretability claim — that decoded steps reveal the model's actual internal computation — would be supported if step-level exact-match accuracy is high for both correct and incorrect answers, and if errors in decoded steps predict errors in the final answer with high precision.

Scaling test-time compute for implicit reasoning through dynamic latent allocation. One of the paper's acknowledged limitations is that SIM-CoT uses a fixed number of latent tokens K set before training, causing redundant latents on simple problems ("trailing latents simply repeat the final prediction," Appendix K). This is analogous to the uniform compute allocation problem that the Coconut scaling laws paper addressed for explicit CoT — spending the same reasoning budget on every problem regardless of difficulty. A follow-up could develop a dynamic K mechanism where the model learns to emit a "stop reasoning" signal in the latent space, indicating that sufficient computation has been performed. Concretely: during training, the model would be trained with variable-length latent chains (matching the ground-truth number of steps for each problem, rather than padding all problems to K_max), and the decoder would be applied only to the actual steps present. At inference, the model would generate latent tokens until a learned stopping criterion is met (e.g., a dedicated "reasoning complete" latent state, or a confidence threshold on the answer distribution). The evaluation would measure whether dynamic K improves both accuracy (by preventing redundant latents from introducing noise) and efficiency (by reducing average latent count per problem). GSM8K-Aug's step count distribution (Figure 5) makes this experiment natural: most problems require 2–4 steps, with a long tail of harder cases, so the efficiency gain from not always using K = 8 could be substantial. The paper's geometric diagnostics (Table 4b) provide the monitoring framework for whether dynamic K maintains latent health as the chain length varies per example.

Expanding to non-arithmetic reasoning domains to test the generality of the latent instability diagnosis. The paper's entire evaluation is on grade-school arithmetic word problems. The diagnostic framework — latent instability caused by insufficient supervision, manifesting as low inter-latent distance and high vocabulary center distance — is presented as a general property of autoregressive latent reasoning, but this claim is untested outside arithmetic. A critical follow-up would replicate the diagnostic analysis (Figure 1) and SIM-CoT training on a qualitatively different reasoning domain. Strong candidates include: (a) Last Letter Concatenation (a synthetic reasoning task where the model must extract and concatenate the last letters of words in a sequence — tested in Coconut but not with SIM-CoT's diagnostic framework), (b) CSQA or StrategyQA (commonsense reasoning requiring multi-hop inference over facts, where "steps" are less crisply defined than arithmetic operations), or (c) MBPP or HumanEval for code generation (where reasoning steps are subgoals in program synthesis, and step annotations could be extracted from code comments or intermediate variable assignments). The key measurement would be whether the same geometric collapse signature (inter-latent distance plummeting, vocabulary distance spiking) appears at similar latent counts in these domains, and whether step-level supervision (with domain-appropriate step annotations) prevents it. A negative result — latent instability does not occur in non-arithmetic domains, or step-level supervision does not help — would indicate that the paper's findings are specific to the compositional, operator-structured nature of arithmetic reasoning, which would significantly limit the claimed generality. A positive result would establish latent instability as a domain-general phenomenon and SIM-CoT-style supervision as a domain-general solution, substantially expanding the method's significance.

Combining SIM-CoT with test-time compute scaling strategies for implicit reasoning. The paper demonstrates that SIM-CoT stabilizes implicit reasoning when scaling the number of latent tokens during training, but does not explore test-time scaling — using additional latent tokens at inference beyond what was seen during training to improve accuracy on hard problems. This is directly analogous to the test-time compute scaling strategies explored in the Coconut scaling laws paper (best-of-N, beam search, revisions), but applied in the implicit reasoning paradigm where "compute" means latent token count rather than sampled solution count. A follow-up experiment would train SIM-CoT with a moderate K (e.g., 4 latents) using the curriculum schedule, then at inference test whether increasing K beyond the training value (e.g., to 8, 12, or 16 latents) improves accuracy without causing collapse — essentially testing whether step-level supervision during training produces latent representations that are robust to out-of-distribution chain lengths at test time. The geometric diagnostics would be crucial here: does inter-latent distance remain healthy when K is extended beyond the training maximum, or does collapse re-emerge? This experiment would connect the implicit reasoning stability literature to the broader test-time compute scaling literature and could reveal whether implicit reasoning has a scaling law analogous to the explicit reasoning scaling laws — does accuracy follow a power law in latent token count, and does step-level supervision shift the exponent?

Measuring the training-compute cost of step-level supervision and developing cheaper alternatives. The paper's largest unquantified cost is the training-time overhead of the auxiliary decoder — architecturally identical to the base model, trained jointly from scratch, processing K separate step sequences per training example. A practical follow-up would systematically measure this cost and develop cheaper alternatives. The measurement would report: (a) wall-clock training time for SIM-CoT vs. Coconut vs. CODI at each model scale (GPT-2, LLaMA 1B/3B/8B), (b) peak GPU memory usage during training for each configuration, and (c) total FLOPs for training to convergence. The alternatives to test would include: (i) using a much smaller decoder (e.g., 2–4 transformer layers instead of matching the base model depth) to test whether full decoder capacity is necessary or whether a lightweight decoder provides sufficient step-level signal, (ii) sharing transformer layers between the base model and the decoder (not just embeddings) to reduce parameter count, (iii) using a non-autoregressive decoder that predicts all step tokens simultaneously from z_k (reducing the sequential generation cost from L_k forward passes to 1), (iv) replacing the decoder's autoregressive cross-entropy loss with a contrastive loss that only requires encoding the step text (not generating it), eliminating the decoder's autoregressive generation cost entirely. The evaluation criterion would be whether the cheaper alternative maintains the stability and accuracy benefits of full decoder training. The paper's Table 4a (decoder size ablation) already provides suggestive evidence that a 1B decoder is optimal for a 1B base model, but does not test whether a decoder smaller than 1B would suffice — testing a 100M or 300M decoder on a 1B base model would directly answer the question of whether full-scale decoding is necessary for the supervision signal, or whether even a lightweight decoder can provide sufficient per-step gradient information to prevent collapse.

Practical Applications and Downstream Use Cases

Cost-efficient batch mathematical reasoning at scale. For organizations that process large volumes of mathematical word problems — educational technology platforms grading student submissions, tutoring systems generating step-by-step solutions, or automated problem generators creating practice materials — SIM-CoT offers a concrete cost reduction. The paper demonstrates on GPT-2 that SIM-CoT matches or exceeds SFT-CoT accuracy (44.8% vs. 42.7%, Table 1) while using 2.3× fewer inference tokens (11.4 vs. 24.7 average tokens per problem). For a batch of 1 million problems, this translates from approximately 24.7 million generated tokens to 11.4 million — a savings of roughly 13 million tokens of transformer forward passes. At the LLaMA 8B scale (Table 3), the savings are even larger in absolute terms: 7.5 tokens per problem for SIM-CoT vs. 22.2 for SFT-CoT, a 3.0× reduction. For deployment on GPU clusters where inference cost scales with token count, this directly reduces operational expenditure. The caveat is that the accuracy trade-off varies by model scale: on GPT-2, SIM-CoT actually improves accuracy over explicit CoT (so the cost reduction comes with a quality improvement), while on LLaMA 8B, it reaches 89% of SFT-CoT's in-domain accuracy (64.1% vs. 71.7%, Table 3), so the cost savings must be weighed against the accuracy reduction. The application is most compelling when the problem distribution is similar to GSM8K-Aug in difficulty and structure, and when the cost of occasional errors is low relative to the inference cost savings.

Interpretable reasoning for educational and audit-sensitive applications. SIM-CoT's auxiliary decoder — trained during the step-level supervision process and usable at inference for visualization without additional training — enables a capability that was previously impossible for implicit reasoning methods: showing step-by-step reasoning to end users or auditors. In educational settings (e.g., an AI math tutor that explains its work to students) or compliance-sensitive settings (e.g., financial calculations that must be auditable), the opacity of standard implicit reasoning methods has been a barrier to adoption. SIM-CoT's decoder produces human-readable reasoning steps from the latent tokens (Figure 4, Figure 7), enabling deployment scenarios where: (a) a student sees not just the final answer but the sequence of operations that produced it, building trust and enabling learning, (b) an auditor can verify that a financial or legal calculation followed the correct procedure by inspecting decoded intermediate steps, or (c) a developer debugging model errors can examine which specific reasoning step produced an incorrect intermediate value, enabling targeted fixes. The paper does not quantify the accuracy of the decoded steps (a limitation discussed in Section 6), but the qualitative examples in Figure 4 and Appendix K show decoded steps that match the ground-truth reasoning structure: 0.3 × 120 = 36, 120 − 36 = 84, 3/4 × 84 = 63, 84 − 63 = 21. The practical deployment workflow would be: train SIM-CoT with step-level annotations aligned to the target domain's reasoning structure, deploy with the decoder kept available for optional step visualization (not used in the answer computation, so no latency impact on the critical path), and optionally validate decoder fidelity on a held-out set before exposing decoded steps to users.

Edge deployment of reasoning-capable models under token budget constraints. For on-device or edge deployments where inference compute is severely constrained — mobile assistants, embedded systems in vehicles or IoT devices, offline-capable educational tools — the token efficiency of implicit reasoning methods is a hard requirement, not an optimization. A device that can afford to generate 10–15 tokens per reasoning problem cannot run explicit CoT models that generate 25–50 reasoning tokens. Prior to SIM-CoT, the choice was between running Coconut (efficient but unstable at the latent counts needed for complex problems) or running explicit CoT (accurate but too expensive). SIM-CoT enables a middle path: train with step-level supervision to achieve stability at the latent counts needed for the target problem complexity, then deploy the trained model (with decoder removed) using exactly the same inference budget as Coconut. On GPT-2, SIM-CoT achieves 44.8% accuracy with 11.4 tokens per problem (Table 1) — compared to Coconut at 36.6% with the same token budget. The 8.2 percentage point improvement comes with zero additional inference cost, making it purely a training-time investment that pays off at deployment. For a mobile app processing thousands of math problems per day on-device, the accuracy improvement could mean the difference between a usable product and one that makes too many errors. The practical constraint is that SIM-CoT's step-level annotations must be obtained for the target domain — for grade-school math, GSM8K-Aug provides these, but for other on-device reasoning tasks (e.g., unit conversion, tip calculation, schedule optimization), step annotations would need to be generated or curated.

Self-improvement pipelines where implicit reasoning generates training data for explicit models. A less obvious but potentially impactful application is using SIM-CoT as a data generation engine for self-improvement loops. The implicit reasoning paradigm's efficiency advantage makes it attractive for generating large volumes of reasoning traces that can then be used to train or fine-tune explicit CoT models. The workflow would be: (1) train SIM-CoT on available step-annotated data (e.g., GSM8K-Aug), (2) use the trained model to generate solutions (answers + decoded reasoning steps via the auxiliary decoder) for a large corpus of unannotated problems, (3) filter generated solutions for correctness (by checking final answers against ground truth if available, or by consistency checks), and (4) fine-tune an explicit CoT model on the filtered generated reasoning traces. The advantage over directly using explicit CoT for generation is throughput: SIM-CoT generates answers with 2–3× fewer tokens per problem (Tables 1–3), so for a fixed inference budget, it can process 2–3× more problems. The decoded steps from the auxiliary decoder serve as the "reasoning traces" for the downstream explicit model, and the paper's case studies suggest these traces are coherent enough to serve as training data (though quantitative validation is needed). This application is analogous to how STaR (Zelikman et al., 2022) and ReST^EM (Singh et al., 2024) use explicit reasoning models to generate their own training data, but with an efficiency multiplier from the implicit generation step. The paper itself suggests "distilling the outputs of applying additional test-time compute back into the base LLM" (Section 8 analog in this paper's limitations), and SIM-CoT's decoder provides the mechanism for converting implicit reasoning into explicit training data for that distillation step.