Training Large Language Models to Reason in a Continuous Latent Space

🎯 Pitch

Instead of forcing language models to verbalize every reasoning step as words, Coconut feeds the model's own hidden states back as input, creating a “continuous thought.” This allows the model to encode multiple alternative next steps simultaneously—effectively performing an emergent breadth-first search—and significantly outperforms chain-of-thought on logical reasoning that requires backtracking.

1. Executive Summary

This paper introduces Coconut (Chain of Continuous Thought), a new paradigm that enables LLMs to reason in an unconstrained continuous latent space rather than generating reasoning steps as word tokens. The method feeds the last hidden state of the LLM directly back as the next input embedding — bypassing the language model head — creating a "continuous thought" that can encode multiple alternative next steps simultaneously, enabling an emergent breadth-first search pattern over reasoning paths. Experiments on GSM8k, ProntoQA, and the newly proposed ProsQA dataset using pre-trained GPT-2 show that Coconut achieves superior accuracy-efficiency tradeoffs over chain-of-thought — on ProsQA, latent reasoning reduces hallucinations and wrong-target errors while generating fewer tokens, and on GSM8k it substantially mitigates the accuracy drop from internalizing reasoning steps (achieving 34.1% accuracy vs. 16.5% for no-CoT), establishing that continuous latent reasoning outperforms language-based CoT only when guided by a multi-stage training curriculum that progressively replaces language steps with continuous thoughts.

2. Context and Motivation

The Core Problem: Language May Be a Suboptimal Medium for Reasoning

The fundamental question this paper tackles is whether natural language is the right representational format for LLM reasoning. Every major approach to getting LLMs to solve complex problems — chain-of-thought prompting (Wei et al., 2022), training on reasoning traces (Yue et al., 2023; Yu et al., 2023), reinforcement learning for step-by-step solutions (Wang et al., 2024; Havrilla et al., 2024) — forces the model to articulate its intermediate reasoning as a sequence of discrete word tokens. The paper argues this constraint is not merely incidental but actively harmful to reasoning performance for two specific reasons that are baked into the architecture of current LLMs.

Reason 1: Token-level compute misallocation. The transformer architecture allocates roughly the same computational budget to every token prediction — each forward pass through the full depth of the network, each output distribution computed over the entire vocabulary. Yet the cognitive importance of tokens within a reasoning chain varies enormously. As the paper notes in Section 1:

"Most tokens in a reasoning chain are generated solely for fluency, contributing little to the actual reasoning process. By contrast, some critical tokens require complex planning and pose huge challenges to LLMs."

This is not merely an efficiency concern. When a model must commit computational resources to generating function words, articles, and other fluency markers that carry no reasoning content, it has fewer effective compute cycles available for the genuinely difficult decisions — which node in the logical graph to explore next, which arithmetic operation to perform, which intermediate variable to compute. The paper cites Madaan and Yazdanbakhsh (2022), who classified CoT tokens into symbols, patterns, and text, and proposed guiding models toward concise reasoning — but this approach still operates within the language space and cannot eliminate the fundamental overhead of serializing reasoning into words.

Reason 2: Premature commitment to deterministic paths. When reasoning is expressed as a linear sequence of words, each token choice is a discrete, irreversible decision. Once the model generates "Every brimpus is a lempus," it has committed to that reasoning step and all subsequent reasoning builds on it. There is no mechanism in standard autoregressive generation to maintain multiple alternative hypotheses simultaneously, to backtrack from a wrong turn, or to defer a decision until more evidence accumulates. This is precisely the pattern the paper demonstrates in Figure 4's case study: the CoT-trained model, having made a wrong initial choice, hallucinates a non-existent edge ("Every yumpus is a rempus") because backtracking in language is unnatural and difficult. The paper frames this as a fundamental limitation:

"the autoregressive generation nature makes it challenging to mimic human reasoning on more complex problems, which typically require planning and search."

This connects to a broader critique articulated by LeCun (2022) and Hao et al. (2023): human reasoning on complex problems involves maintaining and evaluating multiple hypotheses, which the sequential, verbal nature of CoT poorly approximates. Prior work has addressed this gap by equipping LLMs with explicit tree search algorithms — Tree of Thoughts (Yao et al., 2023), beam search with self-evaluation (Xie et al., 2023), or training on search trajectories (Lehnert et al., 2024; Gandhi et al., 2024; Su et al., 2024). But these approaches all operate over language tokens — they treat individual words or sentences as the nodes being searched, which retains the inefficiency of the language medium and adds significant computational overhead from the search process itself.

Why This Problem Matters: Theoretical and Practical Significance

The paper's motivation has both a theoretical and a practical dimension.

The theoretical dimension draws on evidence from human cognitive neuroscience. The authors cite neuroimaging studies (Amalric and Dehaene, 2019; Monti et al., 2012, 2007, 2009; Fedorenko et al., 2011) showing that:

"the language network – a set of brain regions responsible for language comprehension and production – remains largely inactive during various reasoning tasks."

Further evidence indicates that "human language is optimized for communication rather than reasoning" (Fedorenko et al., 2024). The implication is stark: if the human brain — the only known system capable of sophisticated reasoning — does not use language as its reasoning substrate, why should we force artificial systems to do so? The biological evidence suggests that reasoning operates over richer, more flexible representations that can encode multiple alternatives, partial evidence, and graded uncertainty in ways that discrete words cannot. Coconut is an attempt to give LLMs access to a representation with similar freedom.

The practical dimension concerns the expressivity limits of transformers. Recent theoretical work (Feng et al., 2023; Merrill and Sabharwal, 2023; Li et al., 2024) has established that chain-of-thought increases the effective depth of transformers by feeding generated outputs back as inputs — each reasoning step effectively adds another computational layer through the recursive processing of previous outputs. The paper explicitly builds on this insight:

"By employing CoT, the effective depth of the transformer increases because the generated outputs are looped back to the input."

Coconut preserves this recursive depth-increasing property (continuous thoughts are fed back as input embeddings) but removes the bottleneck of projecting through the discrete vocabulary, potentially allowing more information to flow between reasoning steps. If a language token can only carry ~log₂(vocabulary_size) bits of information (e.g., ~15 bits for a 50k vocabulary), a continuous vector of dimensionality d (typically 768 or larger) can carry orders of magnitude more, enabling each reasoning step to transmit richer state information.

Where Prior Approaches Fall Short

The paper positions itself against four categories of prior work, each with specific limitations:

1. Language-based CoT methods (Wei et al., 2022; and many subsequent works). These are effective — the paper is clear about this — but suffer from the token-level compute misallocation and premature commitment problems described above. Even "concise CoT" approaches (Madaan and Yazdanbakhsh, 2022) remain in the language space. Tree search over language tokens (Yao et al., 2023; Hao et al., 2024) addresses the premature commitment problem but amplifies the computational cost through explicit branching and rollouts, and still operates over discrete, lossy representations.

2. Pause tokens and filler tokens. Goyal et al. (2023) proposed inserting learnable <pause> tokens into the training corpus, giving the model additional computational capacity without forcing it to generate meaningful text. Pfau et al. (2024) explored using filler tokens like "..." for similar purposes, finding they work well for "highly parallelizable problems." However, both approaches have a fundamental limitation that Coconut aims to transcend. The paper explicitly cites Pfau et al.'s observation:

"these methods do not extend the expressivity of the LLM like CoT; hence, they may not scale to more general and complex reasoning problems."

The reason is architectural: pause tokens are inserted as discrete tokens with fixed embeddings. They don't carry state forward from the model's previous computation — they're blank slates that the model can compute over, but they don't create the recursive depth-increasing loop that CoT creates. Coconut's continuous thoughts, by contrast, are hidden states fed back as embeddings, which directly implements the same recursive computation that gives CoT its expressivity benefits, but without projecting through the vocabulary bottleneck.

3. Distillation and internalization approaches. Two closely related lines of work attempt to internalize CoT reasoning into the model's forward pass. Deng et al. (2023) used knowledge distillation to train a model to directly output answers that previously required explicit CoT, essentially compressing the reasoning into the model weights. Deng et al. (2024) developed iCoT, a training curriculum that gradually removes tokens from the beginning of the reasoning chain — by the final stage, the model goes from question directly to answer with no explicit reasoning. Yu et al. (2024b) distilled reasoning from complex search algorithms into a model that reasons latently.

These approaches demonstrate that latent reasoning is possible, but they suffer from a critical limitation: the reasoning is compressed into a single forward pass through a fixed-depth network. There is no mechanism for variable-depth reasoning — the model can't "think longer" about harder problems. Coconut inherits the multi-stage curriculum idea from iCoT (which the paper explicitly credits: "inspired by iCoT") but adds a crucial capability: continuous thoughts can be chained, with each thought's hidden state becoming the next thought's input embedding, creating a variable-depth reasoning process analogous to CoT but in the continuous space. This means Coconut can scale computation to problem difficulty — a property that internalization approaches lack.

4. Alternative architectures for latent reasoning. The paper briefly acknowledges looped transformers (Giannou et al., 2023; Fan et al., 2024) and diffusion models in embedding space (Barrault et al., 2024), but notes these are not focused on multi-step reasoning or on understanding the properties of latent vs. language reasoning. Coconut's contribution is not a new architecture per se — it's a training paradigm applied to standard transformer LLMs — but rather an investigation of what happens when you free reasoning from the language constraint.

How This Paper Positions Itself

The paper's positioning can be understood as a synthesis of three insights from prior work, combined with a novel empirical claim:

From CoT theory (Feng et al., 2023): The recursive feedback of outputs as inputs increases transformer expressivity. Coconut preserves this mechanism but in continuous space.
From iCoT (Deng et al., 2024): A multi-stage curriculum that gradually removes language reasoning steps can guide the model toward latent reasoning. Coconut adopts and extends this training strategy.
From pause tokens (Goyal et al., 2023): Giving the model extra computation without forcing language generation can be beneficial. Coconut replaces discrete pause tokens with continuous hidden states that carry forward reasoning state.

The novel empirical claim — and the paper's central contribution beyond the method itself — is that continuous latent reasoning exhibits qualitatively different and superior reasoning patterns compared to language-based reasoning. Specifically, Section 4 demonstrates that continuous thoughts can encode multiple alternative next steps simultaneously (a form of superposition), enabling an emergent breadth-first search (BFS) over reasoning paths that is not explicitly trained but arises from the architecture and training objective. This BFS pattern explains why Coconut outperforms CoT on planning-intensive tasks: it avoids premature commitment, maintains multiple hypotheses, and defers decisions until more information is available.

The paper does not claim to have solved latent reasoning. It explicitly frames itself as an initial exploration:

"We believe that these findings underscore the potential of latent reasoning and could provide valuable insights for future research."

The experiments are conducted on pre-trained GPT-2 with small-scale reasoning datasets, not on frontier models. The paper's goal is to establish proof of concept and characterize the properties of continuous-space reasoning — the emergent search behavior, the efficiency gains, the difficulty-dependent benefits — rather than to achieve state-of-the-art on any benchmark. This is important context for interpreting the results: the paper is arguing that latent reasoning is a promising direction, not a finished solution.

A subtle but important aspect of positioning: the paper is careful to distinguish Coconut from "internalizing" CoT. While the training procedure (multi-stage curriculum replacing language steps with continuous thoughts) superficially resembles iCoT's step removal, the resulting behavior is fundamentally different. iCoT removes reasoning steps entirely — the model computes the answer in one shot. Coconut replaces language steps with continuous steps, preserving the chained reasoning structure. This means Coconut can scale to longer reasoning chains (more continuous thoughts) for harder problems, a capability that internalization approaches lack. The paper demonstrates this scaling behavior empirically: increasing the number of continuous thoughts per reasoning step (the hyperparameter c) improves performance on GSM8k (Figure 8, II), and increasing the number of latent reasoning steps (k) on ProsQA increases accuracy and reduces hallucinations (Figure 3).

Limitations the Paper Acknowledges at the Outset

Even before presenting results, the paper flags two important constraints on its scope:

Training is supervised by language reasoning chains. The multi-stage curriculum requires explicit CoT data to bootstrap latent reasoning. In the "w/o curriculum" ablation (Table 1), the model trained directly with continuous thoughts and only question-answer supervision achieves 14.4% on GSM8k — worse than no-CoT (16.5%). The paper is honest about this:

"While the multi-stage training used for Coconut has proven effective, further research is definitely needed to develop better and more general strategies for learning reasoning in latent space, especially without the supervision from language reasoning chains."

This means Coconut, as presented, is not a replacement for CoT data but a way to deploy the knowledge learned from CoT data more efficiently at inference time.

Training efficiency is a challenge. The sequential nature of generating multiple continuous thoughts (n+1 forward passes for n thoughts) limits parallelism:

"the sequential nature of the multiple forward passes poses challenges for parallelism."

The paper notes this as an important direction for future work but does not solve it. This creates a practical tradeoff: Coconut reduces inference-time tokens but increases the number of sequential forward passes, which may affect wall-clock time differently depending on hardware and batch size (as partially addressed in Appendix B).

These acknowledged limitations are important for understanding the paper's contribution. Coconut is not claiming to be a drop-in replacement for CoT that works without reasoning data — it's a method that, given CoT training data, can learn to reason in a more efficient and structurally different (BFS-like) manner than the language chains it was trained on.

3. Technical Approach

3.1 Reader Orientation

This paper introduces Coconut (Chain of Continuous Thought), a training and inference method that allows a standard transformer language model to perform multi-step reasoning in its own continuous latent space — the high-dimensional vector space of its hidden states — rather than generating intermediate reasoning steps as sequences of discrete word tokens. The system solves the problem of language being a suboptimal medium for reasoning by feeding the model's last hidden state directly back as the next input embedding, creating a "continuous thought" that can carry rich, multi-path information between reasoning steps, and using a multi-stage training curriculum that progressively replaces explicit language reasoning chains with these continuous thoughts.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five interconnected components:

Base Transformer Language Model (pre-trained GPT-2) — the core neural network that processes input sequences and produces hidden states. It operates in two modes: language mode (normal autoregressive token generation via the LM head) and latent mode (hidden states bypass the LM head and are fed back as embeddings).
Continuous Thought Mechanism — the central innovation: when in latent mode, the last hidden state $h_t$ at position $t$ is taken after the final layer normalization and used directly as the input embedding for position $t+1$ , instead of mapping it through the vocabulary projection and embedding lookup. This preserves the full $d$ -dimensional information content (768 dimensions for GPT-2) rather than bottlenecking through a single discrete token.
Special Boundary Tokens (<bot> and <eot>) — two learned tokens that demarcate the beginning and end of latent reasoning. <bot> is inserted after the question to signal the transition into latent mode; <eot> signals the return to language mode for answer generation. During training, the model learns to associate these boundaries with the mode switch.
Multi-Stage Training Curriculum — a procedure that teaches the model to reason continuously by starting with full language chain-of-thought training and progressively replacing early language reasoning steps with continuous thoughts across successive training stages. This provides a scaffolding that prevents the model from failing to learn (as happens with direct end-to-end training of latent reasoning).
Inference Controller — a simple mechanism that determines how many continuous thoughts to use at test time. The paper uses a constant-length approach: pad the latent mode to a fixed number of continuous thoughts (matching the final training stage), then generate the remaining reasoning and answer in language mode.

Information flows as follows: question tokens enter the model → <bot> token signals transition to latent mode → for each continuous thought, the last hidden state is extracted and fed back as the next embedding (repeated $k$ times) → <eot> token signals return to language mode → model generates remaining reasoning steps and final answer autoregressively. During training, the loss is computed only on the language tokens after the continuous thoughts (question tokens and continuous thought positions are masked), and the multi-stage curriculum gradually increases $k$ while removing the corresponding language reasoning steps from the training data.

3.3 Roadmap for the Deep Dive

First, the core continuous thought mechanism — how hidden states replace token embeddings, the precise equations governing the mode switch, and why this preserves the recursive depth-increasing property of CoT while removing the vocabulary bottleneck.
Second, the special tokens and mode-switching protocol — how <bot> and <eot> are integrated into the token vocabulary and training, the two strategies for determining when to exit latent mode, and the tradeoffs between them.
Third, the multi-stage training curriculum — the stage-by-stage procedure, the hyperparameter $c$ that controls how many continuous thoughts replace one language step, the loss masking strategy, and why this curriculum is necessary (the "w/o curriculum" ablation result).
Fourth, the inference process — how the model operates at test time, the challenge of determining latent mode length, and the constant-length solution used in experiments.
Fifth, the mathematical formulation — the explicit equations defining the model's computation in both modes, the training objective, and the gradient flow through continuous thoughts.
Sixth, the ProsQA latent tree search analysis framework — the probing methodology used in Section 4, how the model is forced to decode partial language paths from intermediate continuous thoughts, and how the value function is defined and computed (since this is central to the paper's core claim about emergent BFS).

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a methods paper with an analytical component whose core idea is that by modifying the autoregressive generation loop to feed hidden states back as embeddings (rather than discretizing through the vocabulary), and training this modified process with a curriculum that gradually replaces language reasoning with continuous reasoning, the model learns to perform a form of reasoning that exhibits emergent breadth-first search behavior and is more efficient than language-based chain-of-thought.

Continuous Thought Mechanism: The Feed-Forward Feedback Loop

The fundamental technical operation in Coconut is deceptively simple but has profound implications. In a standard transformer language model at inference time, the computation for predicting the next token given an input sequence $x = (x_1, ..., x_T)$ proceeds as:

$H_t = \text{Transformer}(E_t)$

$M(x_{t+1} \mid x_{\leq t}) = \text{softmax}(W h_t)$

where $E_t = [e(x_1), e(x_2), ..., e(x_t)]$ is the matrix of token embeddings for all positions up to $t$ , $H_t \in \mathbb{R}^{t \times d}$ is the matrix of last hidden states for all positions up to $t$ , $h_t = H_t[t, :]$ is the last hidden state at position $t$ (a $d$ -dimensional vector, where $d = 768$ for the GPT-2 base model used in experiments), $e(\cdot)$ is the token embedding function (the learned embedding lookup table that maps token indices to $d$ -dimensional vectors), and $W$ is the parameter matrix of the language model head (the linear projection from hidden states to vocabulary logits).

What this computes: the transformer processes the sequence of embeddings through its full depth (multiple self-attention and feed-forward layers), producing a contextualized hidden state $h_t$ at the final position. This hidden state is then projected through the LM head $W$ to produce a probability distribution over the entire vocabulary, from which the next token is sampled or greedily selected. The chosen token's embedding then becomes $e(x_{t+1})$ in the next step's embedding matrix $E_{t+1}$ .

Why this form for standard LMs: this architecture is designed for language generation, where each output must be a discrete token that can be decoded by humans and fed back into the model. The vocabulary projection $\text{softmax}(W h_t)$ compresses the rich $d$ -dimensional hidden state into a distribution over (for GPT-2) 50,257 tokens — an information bottleneck of roughly $\log_2(50257) \approx 15.6$ bits if the distribution were deterministic, and substantially less when the distribution is diffuse (as it typically is during reasoning). This bottleneck is acceptable when the goal is generating fluent text, but it discards information that could be useful for downstream reasoning steps.

The Coconut modification. In latent mode, the paper replaces the LM head projection and embedding lookup with a direct feedback loop. When the model is in latent mode (between positions $i$ and $j$ , where $x_i = \texttt{<bot>}$ and $x_j = \texttt{<eot>}$ ), the input embedding at position $t$ (where $i < t < j$ ) is not a token embedding but rather the previous position's hidden state:

$E_t = [e(x_1), e(x_2), ..., e(x_i), h_i, h_{i+1}, ..., h_{t-1}]$

The paper notes an important practical detail: "the last hidden states have been processed by the final normalization layer, so they are not too large in magnitude." This matters because the embedding layer $e(\cdot)$ produces vectors with a certain scale (determined by the learned embedding matrix and any layer normalization preceding the transformer layers), and feeding raw hidden states could cause a scale mismatch that destabilizes training. The final layer normalization ensures the hidden states are in a comparable range to the learned token embeddings.

After latent mode finishes ( $t \geq j$ ), the input reverts to token embeddings:

$E_t = [e(x_1), e(x_2), ..., e(x_i), h_i, h_{i+1}, ..., h_{j-1}, e(x_j), ..., e(x_t)]$

What this computes operationally: for each continuous thought step, the model executes a full forward pass through all transformer layers. The final hidden state $h_t$ at the last position of this forward pass is extracted, and then immediately fed back as the input embedding at the same position index $t+1$ in the next forward pass. This creates a recursive computation chain where the hidden state evolves through repeated self-attention and feed-forward transformations, but without ever being projected to a discrete token. The model can, in principle, encode arbitrary real-valued information in the $d = 768$ dimensions of the hidden state, representing multiple reasoning hypotheses simultaneously, partial evidence, or gradient-like information that doesn't correspond to any single word.

Why this preserves CoT's expressivity benefit. The theoretical work of Feng et al. (2023) demonstrated that chain-of-thought reasoning increases the effective depth of a transformer because each generated token's hidden state becomes part of the input context for all subsequent tokens, and the model processes this expanded context through the full transformer depth. In a standard transformer with $L$ layers processing a CoT of length $T_{\text{CoT}}$ , the effective depth (the number of non-linear transformations a token can undergo through the attention mechanism's ability to route information across positions) scales with $L \times T_{\text{CoT}}$ . Coconut preserves exactly this property: each continuous thought step processes the previous hidden state through all $L$ layers, so $k$ continuous thoughts provide $L \times k$ layers of effective computation. The difference is that the information passed between steps is a full $d$ -dimensional vector rather than a single discrete token, removing the vocabulary bottleneck while maintaining the depth-increasing recursion.

What would have been wrong: an alternative approach would be to simply increase the number of transformer layers (making the model deeper) and hope that internal computation suffices for reasoning. This fails because, without the recursive feedback, the model's computation is strictly feed-forward through a fixed-depth network — it cannot adapt the amount of computation to problem difficulty. Another alternative is pause tokens (Goyal et al., 2023), which insert learnable but static embedding vectors that don't carry forward the model's evolving reasoning state. These give the model more positions to compute over but don't create the recursive feedback loop that makes CoT (and Coconut) fundamentally more expressive. The paper cites Pfau et al. (2024)'s acknowledgment that filler tokens "do not extend the expressivity of the LLM like CoT."

Mode Switching: `<bot>` and `<eot>` Tokens

The transition between language and latent modes is managed by two special tokens — <bot> (beginning of thought) and <eot> (end of thought) — which are added to the model's vocabulary as learned embeddings. Their role is to provide explicit signals to the model about which mode it should be operating in at each position.

Token integration. The <bot> and <eot> tokens are not counted toward the hyperparameter $c$ that controls how many continuous thoughts replace each language reasoning step. They serve purely as mode markers. During training, their embeddings are learned alongside all other parameters through the standard gradient descent process. The tokens appear in the training sequences at the positions determined by the multi-stage curriculum (described in the next section).

Latent mode protocol. When the model encounters a <bot> token in the input sequence, it processes it normally through the transformer (the <bot> embedding goes through all layers, producing a hidden state). On the next position after <bot>, instead of taking the hidden state $h_{i}$ from position $i$ (where $x_i = \texttt{<bot>}$ ) and projecting it through the LM head to get a token, the system directly uses $h_i$ as the input embedding for position $i+1$ . This is the first continuous thought. For each subsequent position in latent mode, the hidden state from the previous position's forward pass becomes the embedding for the current position:

$e_{\text{latent}}(x_t) = h_{t-1} \quad \text{for} \quad i < t < j$

where $h_{t-1}$ is obtained by running a full transformer forward pass with embeddings $E_{t-1} = [e(x_1), ..., e(x_i), h_i, ..., h_{t-2}]$ . This means that generating $k$ continuous thoughts requires $k+1$ forward passes: one to process the prefix up through <bot>, then one forward pass per continuous thought, each building on the hidden states from the previous pass.

Exiting latent mode. The <eot> token signals the return to language mode. When the model generates (or the training data specifies) an <eot> token, the next position's input reverts to using the standard token embedding function $e(\cdot)$ applied to the language token that follows. The hidden state from the <eot> position is not fed back as an embedding — the <eot> token's embedding goes through the transformer, producing a hidden state that is then projected through the LM head to begin generating language tokens normally.

The challenge of determining when to exit. During inference, the model must decide how many continuous thoughts to use before switching back to language mode to produce the answer (and possibly remaining reasoning steps). The paper considers two strategies:

Strategy a) Learn to decide autonomously. Train a binary classifier on top of the hidden states during latent mode that predicts whether to continue reasoning or exit to language mode. This would allow the model to adapt the number of continuous thoughts to problem difficulty — easy problems might need only a few, hard ones many. This is conceptually appealing but adds complexity (training the classifier, integrating it with autoregressive generation).

Strategy b) Pad to constant length. Always use a fixed number of continuous thoughts, determined by the final training stage. If the model was trained with $k$ continuous thoughts in the final stage, inference always uses $k$ continuous thoughts. The paper states: "We found that both approaches work comparably well. Therefore, we use the second option in our experiment for simplicity, unless specified otherwise."

The constant-length approach has an important implication: the model must learn to compress its reasoning into exactly the prescribed number of thoughts, regardless of problem difficulty. For problems requiring fewer reasoning steps than the latent budget, the model can "waste" some continuous thoughts on identity-like transformations; for problems requiring more steps, the model must pack more reasoning into each thought. The fact that this works (Coconut with constant length outperforms CoT on logical reasoning tasks) suggests that continuous thoughts are substantially more information-dense than language tokens.

What happens to $M(x_{t+1} \mid x_{\leq t})$ during latent mode. The paper notes that the standard LM head output $\text{softmax}(W h_t)$ "is not defined when $i < t < j$ , since the latent thought is not intended to be mapped back to language space. However, $\text{softmax}(W h_t)$ can still be calculated for probing purposes (see Section 5)." This is a subtle but important distinction: during training and inference, the model does not use the LM head output at latent positions (no loss is computed there, no token is sampled). But the projection exists (it's part of the model architecture), and the paper exploits this for analysis — by computing $\text{softmax}(W h_t)$ at intermediate continuous thought positions, they can decode what language concepts the hidden state would map to if forced, which is the basis for the tree search analysis in Section 4.

Multi-Stage Training Curriculum

The training curriculum is the mechanism by which Coconut learns to reason in continuous space. The central challenge is that continuous thoughts have no direct supervision — there is no "correct" hidden state to compare against, only the downstream correctness of predicted language tokens. The paper demonstrates that attempting to learn continuous reasoning directly from question-answer pairs (the "w/o curriculum" ablation) fails catastrophically: on GSM8k, it achieves 14.4% accuracy compared to 16.5% for simply predicting the answer with no reasoning at all (no-CoT). The model needs guidance to discover useful continuous representations.

The curriculum borrows its structure from iCoT (Deng et al., 2024) but with a critical difference. In iCoT, language reasoning tokens are progressively removed from the beginning of the chain, and no replacement is added — the model learns to compute the answer in fewer and fewer tokens, eventually in a single forward pass (the reasoning is "internalized" into the model parameters). In Coconut, removed language reasoning steps are replaced with continuous thoughts — the chain structure is preserved, but the medium changes from discrete language to continuous vectors.

Stage-by-stage procedure. Let $N$ be the total number of training stages (including the initial stage 0). The maximum number of reasoning steps in the training data determines $N$ : for ProsQA and ProntoQA, the maximum is 6 steps, so $N = 7$ stages (stages 0 through 6, with stage 0 being the initial CoT stage). The curriculum proceeds as:

Stage 0 (Initial): Train the model on standard chain-of-thought instances — full language reasoning chains followed by answers. The model learns to generate reasoning in language. This is standard supervised fine-tuning on the reasoning dataset.
Stage k (for k = 1, 2, ..., N-1): Replace the first $k$ language reasoning steps with $k \times c$ continuous thoughts, where $c$ is a hyperparameter controlling the "compression ratio" — how many continuous thoughts substitute for one language step. The continuous thoughts are encapsulated by <bot> and <eot> tokens. The remaining language reasoning steps (from step $k+1$ onward) and the answer remain as language tokens in the training sequence.
Final stage: All language reasoning steps are replaced with continuous thoughts. The model sees only the question, <bot>, $k_{\text{max}} \times c$ continuous thoughts, <eot>, and the answer. No intermediate language reasoning remains. For the GSM8k experiments, the paper uses an additional stage beyond this to handle the long-tail distribution of problems requiring more than 3 reasoning steps: "We then include an additional stage where still $3 \times c$ continuous thoughts are used as in the previous stage, but with all the remaining language reasoning chain removed."

The compression hyperparameter $c$ . This controls how many continuous thoughts replace one language reasoning step. For logical reasoning tasks (ProntoQA, ProsQA), the paper sets $c = 1$ — one continuous thought per language step. For math reasoning (GSM8k), they set $c = 2$ — two continuous thoughts per language step. The choice reflects the different nature of the reasoning: logical reasoning steps involve relational inference ("Stella is a zumpus. Zumpuses are gorpuses. Therefore, Stella is a gorpus.") which may be compactly encodable, while math reasoning steps involve arithmetic computation ("4 - 2 = 2") that may benefit from more latent computation. The paper experiments with varying $c$ on GSM8k (Figure 8, panel II), showing performance improvement when increasing from $c = 0$ to $c = 1$ to $c = 2$ . At $c = 3$ , they observe instability: "adding three continuous thoughts at once – particularly during the final stage transition – leads to a sharp spike in training loss, causing instability."

Training hyperparameters. For all experiments, the base model is pre-trained GPT-2. The learning rate is set to $1 \times 10^{-4}$ with an effective batch size of 128. The paper resets the optimizer state (Adam moments) when switching between training stages, following the practice of Deng et al. (2024). This is important because the training data distribution changes significantly between stages (different numbers of continuous thoughts, different lengths of remaining language chains), and carrying over momentum from the previous stage's gradient landscape could interfere with learning in the new configuration.

Stage-specific training durations:

GSM8k: 6 epochs in stage 0 (initial CoT), 3 epochs in each subsequent stage. The model processes 385,620 training examples per epoch.
ProntoQA and ProsQA: 5 epochs per stage for all stages (stages 0 through 6), with the model staying in the final stage until 50 total epochs. The best checkpoint on the validation set is selected from this extended final-stage training period. This extended final stage allows the model to refine its continuous reasoning after the curriculum has fully transitioned to latent mode.

Loss masking and training objective. The training objective is the standard negative log-likelihood (cross-entropy) loss on next-token prediction, but with a critical masking strategy:

$\mathcal{L} = -\sum_{t \in \mathcal{T}_{\text{active}}} \log P(x_t \mid x_{<t})$

where $\mathcal{T}_{\text{active}}$ is the set of positions where loss is computed. The loss is masked (set to zero contribution) on:

Question tokens — the model should not be penalized for "predicting" the question, since it's given as input.
Continuous thought positions — there is no target token for these positions, since the model is not producing language output. The hidden states are fed forward but no vocabulary distribution is compared against.

The loss is computed only on the remaining language tokens after the continuous thoughts — the language reasoning steps (in early training stages) and the final answer tokens. This means the continuous thoughts receive gradient signals indirectly: the only way the model can improve its prediction of the downstream language tokens is by encoding useful information in the continuous hidden states that get fed forward. There is no explicit objective that says "make the continuous thought similar to the language tokens it replaced" — the continuous thought is free to discover any representation that helps predict the remaining reasoning and answer.

Why this curriculum works (and why direct training fails). The "w/o curriculum" ablation provides crucial evidence. When the model is trained directly with continuous thoughts and only question-answer pairs (no intermediate language supervision), it achieves 14.4% on GSM8k — worse than the no-CoT baseline (16.5%). The interpretation is that the continuous thought space is too unconstrained for the model to discover useful representations from scratch using only the weak signal from final-answer correctness. The gradient from answer prediction back through $k$ iterations of the transformer is too diffuse to guide the model toward encoding the right information in each thought.

The curriculum solves this by providing progressively more challenging intermediate targets. In stage 1, the model only needs to encode the first reasoning step in continuous form while still seeing the remaining steps in language — the language steps provide strong, local supervision. As more steps are replaced, the model must learn to encode more information in continuous form, but it has already developed useful representations from earlier stages. The paper characterizes this as the continuous thought learning to "facilitate the prediction of future reasoning" rather than compress the removed language:

"It is important to note that the objective does not encourage the continuous thought to compress the removed language thought, but rather to facilitate the prediction of future reasoning. Therefore, it's possible for the LLM to learn more effective representations of reasoning steps compared to human language."

This is a crucial design insight: the continuous thought doesn't need to encode the same information as the language step it replaced. It can encode something different — possibly more compact, possibly encoding multiple alternatives — as long as it helps predict the subsequent tokens.

Optimizer state reset. The paper resets the Adam optimizer state (first and second moment estimates) at each stage transition. This prevents the optimizer from applying updates based on gradient statistics accumulated on a different data distribution. The motivation follows Deng et al. (2024): when the training data distribution shifts abruptly (as it does when language steps are replaced with continuous thoughts), the gradient magnitudes and directions can change dramatically. Momentum from the previous stage would push parameters in directions that were useful for the old distribution but may be harmful for the new one.

Handling variable-length reasoning chains. The training data contains problems with reasoning chains of different lengths. If a problem's reasoning chain has fewer than $k$ steps in stage $k$ , the paper's protocol is: "all the language thoughts will be removed." That is, if a problem requires only 2 reasoning steps but the curriculum is at stage 3, all language steps are replaced with $3 \times c$ continuous thoughts, even though this provides more latent computation than the original problem needed. This is consistent with the constant-length inference approach: the model learns to handle variable-difficulty problems with a fixed latent budget by using only as many continuous thoughts as needed and "padding" the rest with near-identity transformations.

Inference Process

At inference time, Coconut operates as a modified autoregressive generation procedure that interleaves latent mode (hidden state feedback) with language mode (standard token generation).

Step-by-step inference procedure:

Encode the question. The question tokens are processed through the transformer normally, producing a sequence of hidden states. The last hidden state after the question represents the model's encoding of the problem.
Insert <bot> token. The <bot> token embedding is appended to the input sequence and processed through the transformer, producing a hidden state $h_{\text{bot}}$ . This serves as the initial state for latent reasoning.
Generate continuous thoughts. For $t = 1, 2, ..., k$ (where $k$ is the predetermined number of continuous thoughts):
- Use the previous hidden state $h_{\text{prev}}$ (initially $h_{\text{bot}}$ ) as the input embedding for the current position.
- Execute a full transformer forward pass, producing a new hidden state $h_t$ at the current position.
- Store $h_t$ for the next iteration.
Each forward pass uses the full context of all previous tokens and continuous thoughts through the self-attention mechanism — the continuous thought at step $t$ attends to the question tokens, the <bot> token, and all previous continuous thoughts.
Insert <eot> token. After $k$ continuous thoughts, the <eot> token embedding is appended to the input sequence.
Generate language tokens. Continue with standard autoregressive generation: project the hidden state through the LM head to get a vocabulary distribution, sample or greedily select the next token, embed it, and repeat. The model generates remaining reasoning steps (if any) and the final answer.

Determining $k$ at inference time. The paper uses the constant-length strategy: $k$ is set to match the final training stage. For GSM8k with $c = 2$ and 3 stages of replacement, $k = 3 \times 2 = 6$ continuous thoughts (plus the additional final stage with all language removed). For ProsQA with $c = 1$ and 6 stages, $k = 6$ continuous thoughts in the final stage. The model generates these thoughts regardless of problem difficulty.

For the analysis experiments in Section 4 (where they "manipulate the model to switch between fully latent reasoning and fully language reasoning"), they vary $k$ at inference time across $\{0, 1, 2, 3, 4, 5, 6\}$ while using the same model weights (trained with $k=6$ in the final stage). When $k < 6$ , the model is forced to exit latent mode early and generate the remaining reasoning in language. This works because the multi-stage curriculum has exposed the model to all intermediate values of $k$ during training, so the model can handle any $k$ from 0 to 6 at inference time.

Greedy decoding. All experiments use greedy decoding (selecting the highest-probability token at each step) rather than sampling. This eliminates variance from stochastic generation and makes the results directly attributable to the model's learned reasoning capabilities rather than sampling luck.

The sequence of forward passes. For an inference with $k$ continuous thoughts, the total number of transformer forward passes is: 1 (to process the question and <bot> through to produce the first hidden state for feedback) + $k$ (one per continuous thought) + 1 (to process the <eot> and all subsequent language generation). If the model generates $T_{\text{lang}}$ language tokens after the continuous thoughts, the total passes are $k + 2$ , which is independent of $T_{\text{lang}}$ because the language tokens are generated incrementally within a single autoregressive loop using a KV cache — each new language token requires only one forward pass through the transformer (processing the new token in the context of the cached keys and values from all previous positions, including the continuous thoughts).

This means the computational cost of the continuous thoughts is $k$ forward passes that don't produce any language output — a "pure reasoning" overhead. The paper trades this overhead against the reduction in language tokens generated: Coconut generates far fewer language tokens than CoT (Table 1 shows 8.2 vs. 25.0 on GSM8k, 9.0 vs. 92.5 on ProntoQA, 14.2 vs. 49.4 on ProsQA), so the total forward passes may actually decrease if the reduction in language generation exceeds the continuous thought overhead. The clock-time measurements in Appendix B confirm this: Coconut is faster than CoT on all three datasets (0.09s vs. 0.26s on GSM8k, 0.11s vs. 0.85s on ProntoQA, 0.15s vs. 0.47s on ProsQA).

Mathematical Formulation of the Training Objective

The training objective for Coconut is a masked negative log-likelihood loss that operates on the language tokens following the continuous thoughts. Let the full training sequence be $x = (x_1, ..., x_T)$ , where the subsequence from position $i$ to $j$ (inclusive of boundary tokens) contains the latent reasoning: $x_i = \texttt{<bot>}$ , positions $i+1$ through $j-1$ contain $k = j - i - 1$ continuous thoughts, and $x_j = \texttt{<eot>}$ . The remaining positions $t > j$ contain language reasoning steps and the final answer.

The loss for a single training example is:

$\mathcal{L} = -\sum_{t \in \mathcal{T}_{\text{active}}} \log P(x_t \mid x_{<t})$

where $\mathcal{T}_{\text{active}} = \{t : t \text{ is a language token after the continuous thoughts}\}$ — that is, positions $t > j$ that correspond to actual token predictions. Positions $t \leq j$ (question, <bot>, continuous thought positions, <eot>) are excluded from the loss.

The probability $P(x_t \mid x_{<t})$ is computed differently depending on whether position $t-1$ was in latent or language mode, because the input embedding at position $t$ is constructed differently:

If position $t-1$ was a language token (including <eot>): $P(x_t \mid x_{<t}) = \text{softmax}(W h_{t-1})[x_t]$ — the standard LM head projection applied to the hidden state from the previous token.
If position $t-1$ was a continuous thought: the hidden state $h_{t-1}$ is used as the embedding for position $t$ , so the computation of $P(x_t \mid x_{<t})$ depends on $h_{t-1}$ indirectly through the transformer's processing of the input sequence $E_t$ which includes $h_{t-1}$ as the embedding at position $t$ .

What this computes operationally: for each active position $t$ , the model's predicted probability for the ground-truth token $x_t$ is extracted, the negative log is taken, and these are summed. The gradient of this loss with respect to all model parameters (including the continuous thoughts) is computed via backpropagation. Because the continuous thoughts are intermediate activations in the computation graph (the hidden state $h_{t-1}$ is a function of all previous parameters and inputs), gradients flow through them: $\frac{\partial \mathcal{L}}{\partial \theta} = \sum_{t \in \mathcal{T}_{\text{active}}} \frac{\partial \mathcal{L}}{\partial P(x_t)} \cdot \frac{\partial P(x_t)}{\partial h_{t-1}} \cdot \frac{\partial h_{t-1}}{\partial \theta}$ (plus gradients through the chain of continuous thoughts back to earlier parameters).

Why this form: the loss is the standard language modeling objective, restricted to positions where we have ground-truth tokens. The masking of continuous thought positions is necessary because there is no target to compare against — the model is not trying to predict a specific token at those positions. The masking of question tokens is standard in instruction fine-tuning: the model should learn to generate the answer conditioned on the question, not to regenerate the question.

The key property of this loss is that it provides no direct supervision on what the continuous thoughts should represent. The only signal is whether the continuous thoughts help predict the downstream tokens correctly. This is both a strength (the model can discover representations that are more effective than language for facilitating downstream reasoning) and a challenge (without the curriculum, the signal is too weak to guide learning). The multi-stage curriculum addresses this by initially providing strong local supervision (most reasoning steps in language, only the first step replaced), then gradually withdrawing that supervision as the model develops useful continuous representations.

The paper explicitly contrasts their loss with what an "obvious" alternative would be: training the continuous thoughts to reconstruct the language tokens they replaced (an autoencoder-like objective). They argue this would be suboptimal because:

"the objective does not encourage the continuous thought to compress the removed language thought, but rather to facilitate the prediction of future reasoning."

A reconstruction loss would force the continuous thought to encode the same information as the language tokens, merely in a different format. The actual loss allows the continuous thought to encode different information — possibly richer, possibly encoding multiple alternatives — as long as it helps with downstream prediction.

The forward pass count for training. When $k$ continuous thoughts are scheduled in the current training stage, the training procedure requires $k + 1$ forward passes: the first $k$ passes to compute each continuous thought sequentially (each building on the hidden state from the previous pass), and one final forward pass through the language tokens to compute the loss. The paper notes:

"While we can save any repetitive computing by using a KV cache, the sequential nature of the multiple forward passes poses challenges for parallelism."

The sequential dependency (each continuous thought depends on the hidden state from the previous thought's forward pass, which depends on the previous thought, etc.) means these $k$ forward passes cannot be parallelized — they form a chain of length $k$ . This is the primary training efficiency bottleneck.

Probing and Analysis Framework for Latent Tree Search (Section 4 Methodology)

Section 4 of the paper presents a detailed analysis of what the continuous thoughts encode and how the model uses them for reasoning. This is not merely an interpretive overlay but a specific experimental methodology that the paper develops to characterize the emergent reasoning behavior. Understanding this methodology is essential for grasping the paper's central empirical claim (that continuous reasoning enables BFS).

The core probing technique: forced language decoding from intermediate continuous thoughts. The key idea is to interrupt the latent reasoning process at an intermediate step and force the model to "translate" its current continuous thought into language. Given a model trained with the multi-stage curriculum (which can handle any $k$ from 0 to $k_{\text{max}}$ ), the paper inserts an <eot> token after exactly $k$ continuous thoughts (for some $k < k_{\text{max}}$ ), causing the model to exit latent mode and begin generating language tokens. The language tokens produced immediately after the <eot> reveal what reasoning step the model would have taken at that point, allowing reconstruction of the "reasoning tree" that the continuous thought implicitly represents.

Example from the paper (Figure 5). For the ProsQA problem shown in Figure 4, the initial reasoning step requires identifying which immediate child node of "Alex" to explore next. The candidates are {lempus, sterpus, zhorpus, grimpus} (since Alex is a rempus, and various rules connect rempus to other concepts). After one continuous thought ( $k=1$ ), the model is forced to generate language. The paper computes the probability of each candidate concept being generated as the first concept after the <eot> token.

How the "value function" is defined and computed. For a candidate next concept (e.g., "lempus"), the model's estimated value is:

$V(\text{concept}) = \prod_{w \in \text{tokens(concept)}} P(w \mid \text{context}, \text{previous tokens of concept})$

where $\text{tokens(concept)}$ decomposes the concept word into its constituent subword tokens (for GPT-2's BPE tokenizer), and the probabilities are computed autoregressively conditioned on the full prefix (question, <bot>, continuous thoughts, <eot>) and any previously generated tokens.

What this computes operationally: rather than generating greedily and taking the single highest-probability concept, the paper computes the joint probability of each complete concept string as the product of its token-level conditional probabilities. For a single-token concept, this is just the probability of that token at the first generation step. For multi-token concepts, it's the product of probabilities across the tokens, capturing the model's assessment that this particular concept (not just its first token) is the right next step.

Why this is interpreted as a value function: the probability $V(\text{concept})$ represents the model's estimate, given the current continuous thought, of how likely this concept is to be the correct next step toward the answer. It's not a reinforcement learning value function in the formal sense (expected future reward), but rather an implicit assessment encoded in the model's generation probabilities. Higher $V$ means the model considers this concept a more promising direction for further reasoning.

Constructing the latent search tree (Figure 5). The analysis proceeds in two steps:

First step ( $k=1$ continuous thought): The model is probed after one continuous thought. The candidates are all immediate children of "Alex" that are reachable via the stated rules. The paper computes $V$ for each candidate. In the example, the probabilities are: lempus (0.33), sterpus (0.15), zhorpus (0.07), grimpus (0.27), with the remaining probability mass distributed over other tokens and concepts.

Second step ( $k=2$ continuous thoughts): The model is probed after two continuous thoughts. Now the candidates include not just immediate children but also grandchildren — all nodes reachable within two steps from Alex. The probabilities are recomputed, showing a shift: rorpus (a child of grimpus) now has the highest value (0.87), while lempus (which was highest after one thought) has dropped significantly.

The BFS interpretation. The key observation is that the model does not simply pick the highest-value node at step 1 and commit to it. If it were doing greedy search, after identifying lempus as the best first step (0.33), it would only explore lempus's children in step 2. Instead, the model's second continuous thought assigns the highest value to rorpus — a child of grimpus, which was only the second-highest at step 1. This means the continuous thought has maintained information about multiple candidates (at least lempus and grimpus) and allowed the model to evaluate their children before committing to a path. This is the signature of breadth-first search: explore multiple branches before deciding.

The parallelism analysis (Figure 6). To quantify how many alternatives the model maintains, the paper computes, for each test example, the cumulative value of the top-k candidate nodes:

$V_{\text{cumulative}}(k) = \sum_{i=1}^{k} V(\text{node with }i\text{-th highest }V)$

They plot $V_{\text{cumulative}}(1)$ , $V_{\text{cumulative}}(2)$ , and $V_{\text{cumulative}}(3)$ across all test examples, sorted by $V_{\text{cumulative}}(1)$ . For the first continuous thought (Figure 6, left), there are substantial gaps between the top-1, top-2, and top-3 curves, indicating that the top-1 candidate accounts for significantly less probability mass than the combination of top-2 or top-3 — the model is spreading its probability across multiple alternatives. For the second continuous thought (Figure 6, right), the gaps narrow, indicating the model is converging on a smaller set of candidates.

The node height analysis (Figure 7). To explain why BFS is beneficial, the paper defines the height of a node in the DAG as its shortest distance to any leaf node (a node with no outgoing edges). Nodes closer to leaves (lower height) have fewer possible downstream paths and should be easier to evaluate. Nodes farther from leaves (higher height) have many possible downstream paths and are inherently more ambiguous.

The paper classifies nodes as "correct" (can lead to the target answer) or "incorrect" (cannot reach the target), and plots the model's estimated value $V$ against node height. The result: for correct nodes, $V$ is low and variable at high heights (the model is uncertain) and increases and stabilizes at low heights (the model becomes confident). For incorrect nodes, $V$ decreases at low heights (the model correctly identifies dead ends). This provides a mechanistic explanation for the BFS benefit: by deferring commitment until more information accumulates (nodes at lower heights are evaluated more accurately), the model avoids the early mistakes that plague greedy CoT.

Why this probing methodology works with the constant-length training. The multi-stage curriculum trains the model on all intermediate values of $k$ (stage 1 uses $k = c$ , stage 2 uses $k = 2c$ , etc.). This means the model's weights are adapted to handle any number of continuous thoughts from 0 to $k_{\text{max}}$ . At inference time, the paper can set $k$ to any value in this range, and the model will behave appropriately — generating language from the point it would have been at in the corresponding training stage. This enables the "interrupted" probing that reveals the internal search tree.

The fact that the model's behavior at intermediate $k$ values produces coherent, interpretable language reasoning (as shown in Figure 4's case study and Figure 5's tree visualization) is itself evidence that the continuous thoughts are encoding meaningful reasoning state, not just arbitrary vectors that happen to produce correct final answers. The model has learned a genuine multi-step reasoning process in continuous space, where each thought corresponds to a stage of exploration and evaluation analogous to search algorithms.

Design Choices and Their Justifications

Why continuous thoughts instead of pause tokens? Pause tokens (Goyal et al., 2023) are learned embedding vectors inserted at fixed positions but not generated by the model's own computation. They provide additional "compute budget" — more positions for the transformer to process — but they don't create the recursive feedback loop that CoT relies on for expressivity. The hidden state after a pause token is processed through the transformer and may influence subsequent positions via attention, but it is not fed back as the next input — the next input is another independent pause token embedding. Coconut's continuous thoughts, by feeding the hidden state back as the embedding, create a chain where each step's output becomes the next step's input, exactly mirroring the recursive structure that gives CoT its depth-increasing property. The paper's ablation ("pause as thought") shows that pause tokens with the same multi-stage curriculum achieve lower accuracy than continuous thoughts on GSM8k (24.1% vs. 34.1%) and ProsQA (96.6% vs. 97.0%), supporting the claim that the feedback mechanism matters beyond just providing additional computation positions.

Why the multi-stage curriculum instead of end-to-end training? The "w/o curriculum" ablation is the key evidence: direct training of continuous thoughts from question-answer pairs fails completely (14.4% vs. 16.5% for no-CoT on GSM8k). The gradient signal from final-answer correctness, backpropagated through $k$ iterations of the full transformer, is too weak and noisy to guide the model toward useful continuous representations. The curriculum provides progressively sparser language supervision — early stages have most of the reasoning in language (strong, local gradient signal), later stages have less (weaker but now the model has learned useful representations to build on). This is analogous to curriculum learning in reinforcement learning: start with dense rewards, then gradually make them sparser as the policy improves.

Why replace from the beginning of the chain rather than the end? The curriculum replaces reasoning steps from the beginning (first step replaced in stage 1) rather than the end. This is inherited from iCoT but has a specific justification in Coconut: the early reasoning steps are where the model needs to encode multiple alternatives (as Section 4 demonstrates). If the model were to replace the last steps with continuous thoughts while keeping the first steps in language, the model would still commit to a single path in language early on, losing the BFS benefit. By replacing early steps, the model can use continuous thoughts for the exploration-heavy phase of reasoning and switch to language only for the final verification and answer generation.

Why $c = 1$ for logical reasoning and $c = 2$ for math? The paper doesn't provide an explicit ablation comparing $c$ values across tasks, but the choice reflects an intuition about the information density of different reasoning types. A logical reasoning step ("Stella is a zumpus. Zumpuses are gorpuses. Therefore, Stella is a gorpus.") involves primarily relational inference — connecting two known facts — which may be representable compactly in continuous space. A math step ("4 - 2 = 2") involves arithmetic computation over numerical values, which may benefit from more computational steps (each continuous thought is one forward pass through the transformer, providing an opportunity for iterative refinement of numerical representations). The GSM8k results in Figure 8 confirm that increasing $c$ from 1 to 2 improves performance, but the instability at $c = 3$ suggests this is not simply "more is better."

Why constant-length inference rather than learned termination? The paper states both approaches "work comparably well" but chooses constant length for simplicity. This is a practical choice that avoids the complexity of training a separate termination classifier. However, it means the model cannot adapt its reasoning depth to problem difficulty — it always uses the same number of continuous thoughts whether the problem is trivial or extremely hard. The fact that this still outperforms CoT suggests that continuous thoughts are flexible enough to handle this variability (possibly by encoding "no-op" thoughts for easy problems), but a learned termination mechanism would likely improve efficiency further.

4. Key Insights and Innovations

Innovation 1: Continuous Representations Enable an Emergent Search Process That Is Qualitatively Different from Language-Based Reasoning

The paper's most important intellectual contribution is not the continuous thought mechanism itself — that is a straightforward engineering modification — but the empirical discovery that reasoning in continuous space yields a fundamentally different reasoning pattern from language-based chain-of-thought, even when neither the architecture nor the training objective explicitly encourages this pattern.

What makes this discovery significant is that it was not designed. The model is trained with a standard next-token prediction loss masked over the continuous thought positions (Section 3). There is no search supervision, no reinforcement learning reward for exploring multiple paths, no architectural component that explicitly models branching. The training data provides only language reasoning chains — linear sequences of reasoning steps — as the initial supervision (stage 0 of the curriculum). Yet when the model transitions to continuous reasoning, it spontaneously develops a breadth-first search (BFS) strategy in which early continuous thoughts encode multiple candidate next steps simultaneously (Figure 5, left: values of 0.33, 0.27, 0.15, 0.07 spread across four candidates) and later thoughts converge on the most promising path after evaluating downstream consequences (Figure 5, right: the highest value shifts from lempus at 0.33 to rorpus at 0.87, a node that was not the top-ranked candidate at the first step).

This is a fundamentally different kind of reasoning from what CoT produces. In language-based CoT, the autoregressive generation process forces a greedy commitment at each step: the model picks one next token, then builds on that choice. The case study in Figure 4 makes this concrete: the CoT model, after committing to an incorrect initial path ("Every rempus is a yumpus"), hallucinates a non-existent edge ("Every yumpus is a rempus") because backtracking in language is unnatural and the model has no mechanism to maintain the alternative path it rejected. In contrast, Coconut with k = 2 continuous thoughts maintains both the lempus and grimpus branches in parallel, evaluates their children (rorpus vs. lempus's downstream nodes), and converges on the correct path — a behavior that mirrors search algorithms but emerges from gradient-based optimization of a language modeling objective.

The quantitative evidence for this being a systematic phenomenon rather than an anecdotal quirk comes from Figure 6. The parallelism analysis shows that across the entire ProsQA test set, the gap between top-1, top-2, and top-3 cumulative values is substantial for the first continuous thought (indicating broad parallel exploration) and narrows for the second continuous thought (indicating convergence toward a specific path). This pattern — parallel early, focused late — is the signature of BFS and appears consistently across examples.

Comparison to prior work: Previous approaches to search-augmented reasoning in LLMs — Tree of Thoughts (Yao et al., 2023), beam search with self-evaluation (Xie et al., 2023), stream of search (Gandhi et al., 2024) — all implement search over language tokens as an explicit external algorithm, requiring the model to generate, evaluate, and branch on complete text strings at each step. The search process is orchestrated by an external controller (Python code managing the tree) and the model serves as a proposal/evaluation subroutine. Coconut achieves search-like behavior internally, within the model's own representations, without any external tree management. The continuous thought vector simultaneously encodes information about multiple candidates in its high-dimensional space (768 dimensions for GPT-2), allowing the model to "simulate" branching and comparison through its standard feedforward and attention operations rather than through explicit token generation and scoring. This is a conceptual shift: search moves from being an algorithm implemented with the model to being a computation performed by the model.

The theoretical work of Zhu et al. (2025b), conducted in parallel with this paper, provides a formal framework for understanding this phenomenon. They demonstrate that continuous CoT can represent multiple reasoning paths in superposition states — linear combinations of path representations that the model can simultaneously maintain and differentiate — making it more efficient than discrete CoT for tasks requiring exploration of alternatives. Coconut's empirical findings (the BFS pattern, the parallelism analysis, the height-dependent value function) provide experimental evidence for this theoretical claim, establishing that the superposition property is not merely a mathematical possibility but emerges in practice from standard training.

Why this is fundamental, not incremental: This finding changes our understanding of what LLMs can do when freed from the language generation constraint. The dominant assumption in the field has been that LLMs reason by generating language — that the language tokens are the reasoning. Coconut demonstrates that language generation can be replaced by a more expressive continuous process that supports search-like computation, and that this replacement yields qualitative improvements in reasoning behavior (fewer hallucinations, less premature commitment) that are not achievable by simply making language CoT more efficient or concise. The BFS pattern is not a minor improvement on CoT — it is a different category of reasoning algorithm, one that was previously thought to require explicit external search machinery.

Innovation 2: The Multi-Stage Curriculum as a Bootstrap for Discovering Useful Continuous Representations Without Direct Supervision

The paper's second major contribution is methodological: it demonstrates that a multi-stage curriculum that gradually replaces language reasoning steps with continuous thoughts can successfully teach a model to reason in latent space, even though direct end-to-end training of continuous thoughts fails entirely. This is not merely an engineering trick — it reveals something important about the learning dynamics of continuous representations and provides a template for future work on latent reasoning.

The intellectual significance lies in the nature of the learning problem that the curriculum solves. The continuous thought mechanism has no direct supervision — there is no "correct" hidden state to compare against, no autoencoding loss that forces the thought to reconstruct the replaced language tokens, no auxiliary objective that shapes what information the thought should encode. The only training signal comes from the downstream language tokens: the model must predict the remaining reasoning steps and final answer correctly, and gradient flows backward through the continuous thoughts to update the parameters that produced them. In the "w/o curriculum" setting, this signal is insufficient — it must propagate through k iterations of the full transformer (each continuous thought is a complete forward pass), and the gradient becomes too diffuse to guide the model toward useful representations. The result is complete failure: on GSM8k, Coconut w/o curriculum achieves 14.4% accuracy, worse than simply predicting the answer with no reasoning at all (no-CoT: 16.5%).

The curriculum solves this by providing a learning scaffold that progressively weakens the supervision signal as the model develops competence. In stage 1, only the first reasoning step is replaced with c continuous thoughts; the remaining steps are still in language and provide strong, local gradient signals (the model can directly learn to predict step 2 from the continuous thought that replaced step 1). In stage 2, the first two steps are replaced, forcing the continuous thoughts to carry more information while still benefiting from language supervision on steps 3+. By the final stage, all language steps are gone, but the model has developed useful continuous representations through the earlier stages and can refine them using only the final-answer signal.

Comparison to prior work: The curriculum is explicitly inspired by iCoT (Deng et al., 2024), which also gradually removes language reasoning tokens to internalize reasoning into the model's forward pass. But there is a crucial architectural difference that makes Coconut's curriculum solve a harder problem. In iCoT, removed tokens are simply deleted — the model processes fewer tokens, and the internalized reasoning happens through the transformer's existing depth (the model learns to compute the answer in fewer forward passes). There is no new mechanism to learn; the model just gets better at using its existing computation. In Coconut, removed tokens are replaced with a new mechanism (continuous thoughts) that the model must learn to use effectively from scratch. The curriculum must simultaneously teach the model what to encode in continuous thoughts and how to process them, while gradually removing the crutch of language supervision.

The ablation Coconut w/o thought (Table 1) isolates this difference. This variant uses the same curriculum schedule as Coconut but without inserting continuous thoughts — it's essentially iCoT with Coconut's exact stage timings. On GSM8k, it achieves 21.6% accuracy, which is better than no-CoT (16.5%) but substantially worse than full Coconut (34.1%). This demonstrates that the continuous thoughts are providing a benefit beyond simply removing tokens and relying on the model's internal computation — the feedback mechanism matters.

The Coconut pause as thought ablation (Table 1) further sharpens the contribution. This variant uses the same curriculum but replaces continuous thoughts with discrete <pause> tokens (static learned embeddings). It achieves 24.1% on GSM8k, better than no-thought (21.6%) but still far below continuous thoughts (34.1%). This confirms that feeding hidden states back as embeddings — creating the recursive feedback loop — provides a learning benefit that additional static computation positions cannot replicate. The gradient can shape the content of continuous thoughts in a way that it cannot shape fixed pause token embeddings (which only receive gradient in the final forward pass, not through the recursive chain).

The broader significance: This curriculum approach establishes a general principle for learning latent representations in autoregressive models: when direct end-to-end training fails due to insufficient gradient signal, a curriculum that provides intermediate supervision through increasingly compressed explicit representations can bootstrap learning. This principle extends beyond reasoning — it could apply to any setting where one wants an LLM to develop internal representations that support downstream tasks without explicit intermediate supervision. The paper's specific finding — that the continuous thought is free to discover representations that are different from and more effective than the language tokens it replaces — challenges the implicit assumption in distillation-based approaches (Deng et al., 2023; Yu et al., 2024b) that the optimal latent representation approximately encodes the same information as the explicit reasoning chain.

This is fundamental rather than incremental because it opens a new axis of model design: rather than choosing between explicit CoT (expensive but trainable) and implicit reasoning (efficient but hard to learn), practitioners can use a curriculum to transition from explicit to implicit while retaining the benefits of both. The curriculum is not just a training trick — it is a way to solve a learning problem that was previously unsolved (direct latent reasoning training fails) and that the field had no general approach for.

Innovation 3: Empirical Characterization of When and Why Latent Reasoning Succeeds — The Difficulty-Dependent Advantage and the Height Hypothesis

The paper's third distinctive contribution is its diagnostic analysis of the conditions under which continuous reasoning outperforms language reasoning, specifically the finding that latent reasoning's advantage is concentrated on problems requiring planning and search over complex logical graphs, and the mechanistic explanation linking this advantage to the model's ability to defer decisions until easier-to-evaluate nodes are reached.

This contribution is not the empirical result itself — other papers report accuracy numbers on benchmarks — but the conceptual framework the paper builds to interpret those numbers. The paper constructs a new dataset (ProsQA, described in Appendix A) specifically designed with a directed acyclic graph structure that forces models to explore multiple paths and avoid dead ends. The graph has an average of 23 nodes, 36 edges, a shortest path length of 3.8 steps, and crucially, 1.6 shortest paths on average (Table 2) — meaning there are typically multiple valid reasoning chains to the correct answer, but also many seductive wrong turns. This is a deliberate stress test for the premature-commitment problem: a model that greedily picks one path and commits will likely pick a wrong one because the correct choice is ambiguous early in the graph.

The results (Figure 3) show a monotonic improvement in both final-answer accuracy and reasoning-process correctness as the number of continuous thoughts k increases from 0 (fully language CoT) to 6 (fully latent reasoning). The accuracy rises from approximately 78% to 97%, and the rate of correct paths/labels increases from approximately 60% to 97%, while hallucinations (non-existent edges) and wrong-target paths (valid but irrelevant) drop dramatically. This is not a marginal gain — it is a qualitative transformation in the model's reasoning capability on this task.

Why this is more than a benchmark result: The paper uses this data to develop and test a specific mechanistic hypothesis about why latent reasoning helps — the "height hypothesis" formalized in Figure 7. The hypothesis states that nodes closer to the graph's leaves (lower height, fewer downstream paths to explore) are easier for the model to evaluate accurately because there is less ambiguity about where they lead. Nodes farther from leaves (higher height) are inherently harder to evaluate because many possible downstream paths exist and the model must reason through them to assess the node's quality. In language CoT, the model must evaluate and commit to nodes at high heights (early in the reasoning chain) where its evaluations are least reliable — this is the premature-commitment problem. In continuous reasoning, the model can defer commitment, maintaining multiple candidates (as shown by the BFS pattern) until it reaches nodes at lower heights where its evaluations are more accurate.

The empirical test of this hypothesis is the correlation shown in Figure 7: for correct nodes (those that can lead to the target), the model's estimated value V is low and highly variable at high heights and increases toward 1.0 at low heights; for incorrect nodes, V decreases toward 0.0 at low heights. This means the model can distinguish correct from incorrect nodes, but only when they are close to terminal states — exactly the condition the height hypothesis predicts. By allowing the model to explore multiple high-height nodes in parallel and evaluate them only after descending to lower heights (via their children), continuous reasoning aligns the model's decision points with its evaluative competence.

This is a diagnostic contribution rather than a performance contribution. The paper is not just claiming "Coconut works better" — it is explaining why it works better in terms of a falsifiable hypothesis about the relationship between graph structure, representation fidelity, and reasoning strategy. This is the kind of mechanistic understanding that enables prediction: given a new reasoning task, one can analyze its graph structure (or more generally, its branching factor and depth) and predict whether continuous reasoning will provide benefits over language CoT. Tasks with high branching factors early in the reasoning process (many alternatives, ambiguity about which is correct) should benefit most; tasks with linear reasoning chains (each step has one obvious successor) should benefit less.

Comparison to prior work: Previous analyses of CoT reasoning — such as the faithfulness literature (Wang et al., 2022; Turpin et al., 2024) showing that models' verbalized reasoning does not always reflect their actual decision process, or the expressivity analyses (Feng et al., 2023; Merrill and Sabharwal, 2023) — focus on what language CoT can and cannot do. They do not propose or analyze alternative reasoning substrates. The height hypothesis provides a lens for understanding why a different substrate might be better, connecting graph-theoretic properties of the reasoning task to representational properties of the reasoning medium. It also connects to the superposition analysis of Zhu et al. (2025b), which predicts that continuous representations should be more efficient specifically when multiple alternatives must be maintained — exactly the high-height regime.

Significance assessment: This is a fundamental conceptual contribution because it reframes the question from "does latent reasoning work?" (a binary empirical question) to "under what structural conditions does latent reasoning provide benefits, and why?" (a mechanistic question with predictive power). The height hypothesis is specific, testable, and connects the empirical phenomenon (BFS-like behavior) to the computational advantage (deferred decision-making). It provides a theoretical foundation for future work: if the hypothesis is correct, then continuous reasoning should benefit any task whose underlying structure has high branching early and discriminative information late — a property that extends beyond logical reasoning DAGs to planning, multi-hop QA, code generation with multiple possible approaches, and other domains.

Innovation 4: Demonstrating That Latent Reasoning Can Scale Computation to Problem Difficulty While Remaining More Token-Efficient Than Language CoT

The paper's fourth contribution is an empirical characterization of the efficiency-accuracy tradeoff between continuous and language reasoning, demonstrating that Coconut achieves a Pareto-superior frontier: for a given accuracy level, it uses fewer tokens; for a given token budget, it achieves higher accuracy. This is more than a speed improvement — it reveals that continuous representations are a more information-dense medium for reasoning than natural language.

The evidence comes from two complementary analyses. First, Table 1 shows that on ProntoQA, Coconut achieves 99.8% accuracy while generating 9.0 tokens, compared to CoT's 98.8% accuracy with 92.5 tokens — a ~10× reduction in generated tokens with higher accuracy. On ProsQA, Coconut achieves 97.0% accuracy with 14.2 tokens versus CoT's 77.5% with 49.4 tokens — roughly 3.5× fewer tokens with a 19.5 percentage point accuracy gain. These are not marginal efficiency improvements; they represent a step change in the information-per-token ratio of the reasoning process.

Second, Figure 8 (panel I) shows what happens when CoT models are forced to become more "efficient" by internalizing early reasoning steps — the standard iCoT approach. As the number of internalized steps m increases from 0 to ALL, the CoT models' accuracy drops rapidly (from ~43% at m = 0 to ~30% at m = 1 to well below 20% at m = ALL), and their token count drops commensurately. These points trace out a tradeoff curve in the accuracy-vs-tokens plane. Coconut's points on the same plot (with c = 0, 1, 2) lie substantially above this curve: for the same token count, Coconut achieves significantly higher accuracy. This demonstrates that continuous thoughts are not merely "compressing" language reasoning (which would lie on the same tradeoff curve) but are providing genuinely more efficient representations — encoding more useful reasoning information per unit of computation than the language tokens they replace.

The probing analysis in Figure 9 provides qualitative support for this interpretation. When the first continuous thought is decoded into language tokens (by projecting through the LM head), it often corresponds to intermediate variables needed for computation (in the math example, the continuous thought decodes to «60», «20», «10» — intermediate values in the calculation). This suggests that continuous thoughts can directly encode numerical and relational content without the syntactic overhead that language requires to express the same content (e.g., "The total is 60. Each part is 20. Therefore...").

Comparison to prior work: The efficiency advantage of continuous reasoning stands in contrast to previous approaches for efficient CoT. Madaan and Yazdanbakhsh (2022) proposed guiding models to generate concise CoT by removing redundant text patterns, but this remains in the language space and cannot escape the fundamental overhead of serializing reasoning into words. Deng et al. (2024)'s iCoT internalizes reasoning into the model's forward pass, achieving high efficiency but losing the ability to scale computation to problem difficulty — the model uses fixed computation regardless of problem hardness. Coconut combines the best of both: it can scale computation (more continuous thoughts for harder problems, as shown by the monotonic improvement with increasing k in Figure 3) while using less "token overhead" than language CoT because each continuous thought carries more information than a word token.

It is important to note what the paper does not claim. It does not claim that Coconut surpasses CoT on all metrics — on GSM8k, CoT achieves 42.9% while Coconut achieves 34.1%. The claim is about the tradeoff: Coconut offers a different operating point on the accuracy-efficiency frontier that may be preferable when inference cost matters. The fact that Coconut's efficiency advantage is most pronounced on tasks with high branching (ProsQA) and least pronounced on tasks with more linear computation (GSM8k, where arithmetic steps have low branching) is consistent with the BFS interpretation from Innovation 1 — the efficiency gain comes from avoiding the generation of exploration tokens that would be needed in language to evaluate multiple paths.

Why this is an innovation rather than just a metric: The efficiency result challenges the implicit assumption that reasoning requires language. The field has largely accepted that CoT's verbosity — generating dozens or hundreds of tokens to solve a problem — is a necessary cost of getting LLMs to reason. Coconut demonstrates that this verbosity is an artifact of the language medium, not an inherent property of multi-step reasoning. The continuous thought can represent in one 768-dimensional vector what takes multiple tokens to express in language, because it doesn't need to respect syntax, fluency, or the sequential constraints of text. This reframes the efficiency question: rather than asking "how can we make CoT more concise?", the question becomes "can we replace language with a more efficient reasoning medium while retaining the benefits of step-by-step computation?" Coconut provides an affirmative answer and a concrete method.

This is incremental-adjacent-to-fundamental. The specific efficiency numbers (3.5× fewer tokens on ProsQA) are incremental relative to prior efficiency work. But the demonstration that changing the reasoning medium can shift the entire accuracy-efficiency tradeoff curve (not just move along it, as iCoT does) is a more fundamental finding. It suggests that the efficiency frontier for LLM reasoning may be determined more by the choice of representation than by model size or training data — a hypothesis that, if validated at scale, would redirect research effort from "bigger models with longer CoT" toward "better representations for the computation that matters."

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper evaluates on three datasets: GSM8k (Cobbe et al., 2021), a grade school-level math word problem dataset with 385,620 training, 500 validation, and 1,319 test examples; ProntoQA (Saparov and He, 2022), a logical reasoning dataset with 9,000 training, 200 validation, and 800 test examples; and ProsQA, a newly proposed dataset (Section 4, Appendix A) with 17,886 training, 300 validation, and 500 test examples. For GSM8k, the paper uses a synthetic training set generated by Deng et al. (2023) rather than the original GSM8k training data. ProsQA is constructed with directed acyclic graphs (DAGs) of logical relationships, with an average of 23 nodes, 36 edges, a shortest path length of 3.8 steps, and 1.6 shortest paths on average (Table 2), designed to require substantial planning and search.
Base model(s). All experiments use a pre-trained GPT-2 model (Section 4.1, Section 5.1). The paper also reports smaller-scale experiments with Llama 3.2-3B and Llama 3-8B (Dubey et al., 2024) in Appendix C.2, but the main experimental results in Tables 1 and the figures in Sections 4 and 5 are all on GPT-2. The choice of GPT-2 is deliberate: it establishes proof of concept on a model small enough for extensive controlled experimentation while being "representative of the capabilities of many contemporary LLMs" (Section 4). The GSM8k training uses the synthetic dataset from Deng et al. (2023) rather than the original GSM8k training data, a detail that matters for replicability.
Metrics. The primary metric is final-answer accuracy (percentage of test questions where the model's output matches the ground-truth answer), computed using greedy decoding throughout all experiments (Section 5.1). For ProsQA specifically, the paper defines a second set of fine-grained process metrics (Section 4.1): Correct Path (output is one of the shortest valid paths to the correct answer), Longer Path (a valid but non-shortest path to the correct answer), Hallucination (path includes non-existent edges or is disconnected), Wrong Target (a valid path in the graph but to the wrong destination node), Correct Label (directly outputs the correct answer with no path), and Incorrect Label (directly outputs the wrong answer). These six categories are exhaustive and mutually exclusive. For partial reasoning chains (when Coconut is probed at intermediate k values), the paper classifies them based on whether a valid explanation can complete the partial path. Additionally, the paper reports number of generated tokens (Table 1) as an efficiency metric, computed as the average number of tokens generated during inference per test example, and inference time in seconds (Appendix B, Table 4) measured on an Nvidia A100 GPU with batch size 1.
Baselines. The paper compares against four baselines:
- CoT (chain-of-thought): The model is trained with full language reasoning chains and generates a complete reasoning chain during inference (Wei et al., 2022; see Section 4.1).
- No-CoT: The model is trained with only question-answer pairs (no reasoning steps) and outputs the final answer directly during inference (Section 4.1).
- iCoT (Deng et al., 2024): The model is trained with language reasoning chains and follows a carefully designed curriculum that gradually removes tokens from the beginning of the reasoning chain until only the answer remains. During inference, the model directly predicts the answer. The paper reports iCoT results from Deng et al. (2024) for GSM8k (noted with an asterisk in Table 1) and trains their own iCoT models for ProntoQA and ProsQA.
- Pause Token (Goyal et al., 2023): The model is trained using only question-answer pairs with special <pause> tokens inserted between the question and answer. The number of <pause> tokens is set equal to the number of continuous thoughts in Coconut. The <pause> tokens are learned embeddings that provide additional computational positions but do not create the recursive feedback loop of continuous thoughts.
Generation budget / compute accounting. The paper measures computational cost in two ways: number of generated tokens (efficiency metric, Table 1, Figure 8) and inference time in seconds (Appendix B, Table 4). For the continuous thought mechanism, the cost model is implicit: each continuous thought requires one additional transformer forward pass beyond what language generation would require. The paper's training procedure requires n + 1 forward passes when n continuous thoughts are scheduled (Section 3). For the number of continuous thoughts, the paper compares variants of Coconut with k ∈ {0, 1, 2, 3, 4, 5, 6} on ProsQA (all sharing the same model weights, varying only at inference time), and compares c ∈ {0, 1, 2} on GSM8k (varying the compression ratio between continuous thoughts and language steps). The "generation budget" is not explicitly equated between methods in terms of FLOPs — the paper reports tokens and clock time, leaving precise FLOP accounting to implicit comparison.
Cross-validation / statistical protocol. For all datasets, the paper selects the best checkpoint based on accuracy on the validation set (Section 5.1). Results on GSM8k are reported with standard deviation across presumably multiple runs (Table 1 shows 42.9 ±0.2 for CoT, 34.1 ±1.5 for Coconut), though the number of runs is not explicitly stated. The two-fold cross-validation protocol used in the reference paper (Snell et al., 2024) is not employed here — the paper does not report cross-validation for strategy selection or hyperparameter tuning. The ProsQA analysis in Figures 3, 5, 6, and 7 aggregates results across the 500-example test set. No confidence intervals are reported for the ProsQA results or the tree-search analyses.

Main Quantitative Results

Overall Performance Across Three Datasets

Table 1 presents the headline results. On GSM8k (math reasoning):

Coconut (Ours) achieves 34.1% ±1.5 accuracy while generating 8.2 tokens on average.
CoT achieves 42.9% ±0.2 accuracy with 25.0 tokens — higher accuracy but ~3× more tokens.
No-CoT achieves only 16.5% ±0.5 with 2.2 tokens — demonstrating that continuous thoughts provide substantial reasoning benefit beyond direct answer prediction.
iCoT achieves 30.0% with 2.2 tokens (result from Deng et al., 2024) — Coconut outperforms iCoT by 4.1 percentage points while using more tokens (8.2 vs. 2.2), suggesting the extra continuous thought computation is beneficial.
Pause Token achieves 16.4% ±1.8 with 2.2 tokens — effectively no improvement over No-CoT, confirming that static pause tokens do not provide the same reasoning benefit as continuous thoughts.

On ProntoQA (logical reasoning):

Coconut achieves 99.8% ±0.2 accuracy with 9.0 tokens — near-perfect accuracy with ~10× fewer tokens than CoT.
CoT achieves 98.8% ±0.8 with 92.5 tokens.
No-CoT achieves 93.8% ±0.7 with 3.0 tokens.
iCoT achieves 99.8% ±0.3 with 3.0 tokens — roughly matching Coconut's accuracy with fewer tokens.
Pause Token achieves 77.7% ±21.0 — substantially worse and with high variance (standard deviation of 21.0 percentage points), indicating instability.

On ProsQA (complex planning logical reasoning):

Coconut achieves 97.0% ±0.3 accuracy with 14.2 tokens — substantially outperforming CoT in both accuracy and efficiency.
CoT achieves 77.5% ±1.9 with 49.4 tokens — a 19.5 percentage point accuracy gap.
No-CoT achieves 76.7% ±1.0 with 8.2 tokens — essentially the same as CoT in accuracy, indicating that language reasoning chains do not help on this task without the ability to search.
iCoT achieves 98.2% ±0.3 with 8.2 tokens — slightly higher accuracy than Coconut (by 1.2 percentage points) with fewer tokens.
Pause Token achieves 75.9% ±0.7 with 8.2 tokens — essentially matching No-CoT, again showing pause tokens alone are insufficient.

The pattern across datasets reveals that Coconut's advantage is most pronounced on the task requiring extensive planning and search (ProsQA, +19.5 points over CoT), substantial on math reasoning (GSM8k, +17.6 points over No-CoT, though not matching CoT), and present but smaller on simpler logical reasoning (ProntoQA, +1.0 point over CoT with dramatically better efficiency). The iCoT baseline is competitive on the logical reasoning tasks but substantially weaker on GSM8k (30.0% vs. Coconut's 34.1%), suggesting that continuous thoughts provide particular benefit for tasks requiring more complex computation beyond simple deductive chaining.

Difficulty-Dependent Scaling of Latent Reasoning on ProsQA

Figure 3 decomposes the ProsQA results by the number of continuous thoughts used at inference time (k ∈ {0, 1, 2, 3, 4, 5, 6}), with all variants using the same model weights (trained with the full multi-stage curriculum ending at k = 6). The left panel shows final-answer accuracy, and the right panel shows reasoning process categories.

Final-answer accuracy (Figure 3, left):

At k = 0 (fully language CoT, equivalent to generating the full reasoning chain in language after no continuous thoughts), accuracy is approximately 77.5% (matching the CoT baseline in Table 1).
As k increases, accuracy monotonically improves: k = 1 reaches roughly 82%, k = 2 reaches roughly 90%, k = 3 reaches roughly 93%, k = 4 reaches roughly 95%, k = 5 reaches roughly 96%, and k = 6 reaches roughly 97%.
The largest gains occur at low k — moving from k = 0 to k = 2 improves accuracy by approximately 12.5 percentage points, while moving from k = 2 to k = 6 adds another ~7 points. This suggests that the first few continuous thoughts (which replace the early, high-branching reasoning steps) provide the majority of the benefit.

Reasoning process categories (Figure 3, right):

At k = 0 (CoT): Correct Path accounts for approximately 55%, Hallucination for approximately 20%, Wrong Target for approximately 15%, and the remaining ~10% distributed across other categories. The high hallucination and wrong-target rates confirm that language CoT frequently commits to incorrect early choices and fails to recover.
As k increases: Correct Path rises to approximately 65% at k = 1, 82% at k = 2, and stabilizes around 85-90% for k ≥ 3. Correct Label (direct answer without partial path) rises from near-zero at low k to approximately 10% at k = 6, reflecting that at high k, the model sometimes skips explicit language reasoning entirely and outputs only the answer.
Hallucination drops dramatically from ~20% at k = 0 to near zero at k ≥ 3. Wrong Target similarly drops from ~15% to near zero. These error categories virtually disappear once the model has access to 3+ continuous thoughts.
At k = 6 (fully latent reasoning), the model either generates correct answers directly (Correct Label, ~10%) or produces partial paths that can be completed to correct solutions (Correct Path, ~87%), with almost no hallucinations or wrong targets.

This monotonic improvement with increasing k is direct evidence that "chaining" continuous thoughts enhances reasoning — each additional continuous thought provides an additional round of latent computation that the model can use to refine its search over the reasoning graph. The sharp reduction in Hallucination and Wrong Target errors at k = 2 (where they drop from ~20% and ~15% to near zero) coincides with the model's ability to evaluate children of multiple candidate nodes (as shown in the Figure 5 case study), supporting the interpretation that two continuous thoughts are sufficient for the BFS pattern to disambiguate correct from incorrect paths.

Efficiency-Accuracy Tradeoff (Figure 8)

Figure 8 consists of two panels: Panel I compares the accuracy-vs-tokens tradeoff between Coconut and language-based internalization, and Panel II shows how Coconut's performance scales with the compression hyperparameter c.

Panel I (Efficiency comparison): The paper trains a series of CoT models that progressively internalize the first m ∈ {0, 1, 2, 3, ALL} reasoning steps (following the iCoT approach). These models trace out a tradeoff curve in the accuracy-versus-generated-tokens plane. At m = 0 (full CoT with no internalization), the model has the highest accuracy (~43%) but generates the most tokens (~25). As m increases, accuracy drops rapidly: m = 1 achieves roughly 30% accuracy with ~15 tokens, m = 2 achieves roughly 22% with ~10 tokens, m = ALL achieves roughly 16% with ~2 tokens. These points form a roughly linear tradeoff: fewer tokens, lower accuracy.

Coconut's points are overlaid on the same plot for c = 0 (no continuous thoughts, equivalent to m = ALL), c = 1 (one continuous thought per language step), and c = 2 (two continuous thoughts per language step). The Coconut points lie substantially above the language tradeoff curve:

Coconut c = 1 achieves approximately 30% accuracy with roughly 8 tokens — matching the m = 1 language model's accuracy (30%) but with ~7 fewer tokens, or alternatively, achieving 8 percentage points higher accuracy than the m = 2 language model (22%) at a similar token count.
Coconut c = 2 achieves approximately 34% accuracy with roughly 8 tokens — 12 points higher accuracy than the equivalently-token-budget language model (m = 2 at ~22%).

This is the paper's key efficiency claim visualized: Coconut shifts the entire tradeoff curve outward, achieving higher accuracy for any given token budget (or fewer tokens for any given accuracy target) compared to language-based internalization. The improvement is not simply moving along the same curve (as iCoT does) — it represents a genuinely more efficient representation.

Panel II (Effect of compression ratio c): On GSM8k, accuracy improves monotonically with c: c = 0 achieves approximately 21.6% (the w/o thought variant), c = 1 achieves approximately 30%, and c = 2 achieves approximately 34.1%. The paper notes that at c = 3, "we observe a slight performance drop accompanied by increased variance" and "adding three continuous thoughts at once – particularly during the final stage transition – leads to a sharp spike in training loss, causing instability" (Appendix C.1). This non-monotonicity at c = 3 suggests a training stability limitation rather than a fundamental representational ceiling.

Ablation Studies and Robustness Checks

Multi-stage curriculum (w/o curriculum): Removing the multi-stage training and directly training with continuous thoughts and only question-answer supervision causes catastrophic failure. On GSM8k, Coconut w/o curriculum achieves 14.4% ±0.8 (Table 1), which is worse than No-CoT (16.5%). On ProntoQA, it achieves 52.4% ±0.4 (vs. 93.8% for No-CoT and 98.8% for CoT). On ProsQA, it achieves 76.1% ±0.2 (essentially matching No-CoT at 76.7%). This ablation demonstrates that the continuous thought mechanism cannot be learned from scratch using only final-answer supervision — the gradient signal through multiple transformer iterations is too weak to guide the model toward useful continuous representations. The curriculum is essential for bootstrapping latent reasoning.
Continuous thoughts vs. no replacement (w/o thought): This variant follows the same multi-stage training schedule as Coconut but removes language reasoning steps without inserting continuous thoughts — essentially iCoT implemented with Coconut's exact stage structure. On GSM8k, it achieves 21.6% ±0.5 (Table 1), which is better than No-CoT (16.5%) but 12.5 points lower than Coconut (34.1%). On ProntoQA, it achieves 99.9% ±0.1 (slightly higher than Coconut at 99.8%). On ProsQA, it achieves 95.5% ±1.1 (lower than Coconut at 97.0%). This ablation isolates the contribution of continuous thoughts beyond simply removing tokens: on tasks requiring substantial computation (GSM8k, ProsQA), the continuous thought feedback mechanism provides significant additional benefit; on simpler tasks where model capacity is less limiting (ProntoQA), simply internalizing computation into the existing depth is sufficient.
Continuous thoughts vs. pause tokens (pause as thought): This variant replaces continuous thoughts with discrete <pause> tokens (fixed learned embeddings) while following the same multi-stage curriculum. On GSM8k, it achieves 24.1% ±0.7 — better than w/o thought (21.6%) but 10 points lower than continuous thoughts (34.1%). On ProntoQA, it achieves 100.0% ±0.1 (essentially perfect). On ProsQA, it achieves 96.6% ±0.8 (slightly lower than Coconut at 97.0%). The gap between pause tokens and continuous thoughts on GSM8k is the key finding: pause tokens provide additional computational positions (the model can process more "blank" inputs), but they lack the recursive feedback mechanism where each step's output becomes the next step's input. The 10-point gap demonstrates that the feedback property — not just additional computation budget — is responsible for the majority of Coconut's benefit on math reasoning.
Effect of the number of continuous thoughts on ProsQA (Figure 3): As detailed above, varying k ∈ {0, 1, 2, 3, 4, 5, 6} at inference time (using the same model weights) shows monotonic improvement in both accuracy and reasoning quality. At k = 0 (fully language CoT): ~77.5% accuracy, ~55% Correct Path, ~20% Hallucination, ~15% Wrong Target. At k = 6 (fully latent): ~97% accuracy, ~87% Correct Path + ~10% Correct Label, near-zero Hallucination and Wrong Target. Notably, the Variants at k = 1 and k = 2 show partial improvements: k = 1 reduces Hallucination to roughly 10% and increases Correct Path to roughly 65%; k = 2 further reduces Hallucination to near zero. All these variants use the same model trained with the full curriculum — the model has learned to handle any k value because the curriculum exposed it to all intermediate stages.
Compression ratio c scaling on GSM8k (Figure 8, panel II): Accuracy on GSM8k increases from c = 0 (21.6%) to c = 1 (30.1%) to c = 2 (34.1%). At c = 3, training instability causes a performance drop with increased variance, suggesting that the current curriculum (which adds c thoughts all at once at each stage transition) becomes unstable when the jump is too large. The paper hypothesizes that finer-grained schedules (adding thoughts one at a time) could mitigate this, following iCoT's token-by-token removal approach.
Larger model experiments (Appendix C.2, Table 5): On Llama 3.2-3B, Coconut achieves 31.7% vs. No-CoT's 26.0% on GSM8k — a 5.7 point improvement. On Llama 3-8B, Coconut achieves 43.6% vs. No-CoT's 42.2% — a 1.4 point improvement. The gains are positive but smaller than with GPT-2, which the paper attributes to larger models having "already undergone extensive language-focused pre-training, making the transition to latent reasoning more challenging" (Appendix C.2). However, these experiments use c = 1 (not the optimal c = 2 found for GPT-2), and are trained for only 3 epochs in stage 0 and 1 epoch per subsequent stage (vs. 6 and 3 for GPT-2), making the comparison to the GPT-2 results potentially confounded by undertraining.
Decoding continuous thoughts (Figure 9): When the first continuous thought is decoded into language tokens by projecting through the LM head, the resulting tokens often correspond to intermediate variables in the computation (e.g., "60", "20", "10" in a math word problem). This qualitatively supports the claim that continuous thoughts encode semantically meaningful reasoning content, not arbitrary vectors.

Critical Assessment

Claim 1: Continuous representations enable an emergent breadth-first search pattern that improves reasoning. The evidence for this claim comes entirely from the ProsQA analysis in Section 4. The qualitative case study (Figure 4) shows one example where Coconut (k = 2) finds the correct path while CoT hallucinates — this demonstrates possibility but not systematicity. The probing analysis (Figure 5) shows that after one continuous thought, probability mass is distributed across multiple candidates (lempus: 0.33, grimpus: 0.27, sterpus: 0.15, zhorpus: 0.07), and after two continuous thoughts, the correct downstream node (rorpus, child of grimpus) receives the highest value (0.87). This is consistent with BFS behavior — maintaining multiple alternatives, then converging on the correct one after evaluating children — but it is demonstrated on a single example. The paper does not report aggregate statistics on how often this pattern occurs across the test set (e.g., what fraction of examples show value redistribution from a non-highest initial candidate to a correct final candidate, or what fraction of examples show top-2 cumulative values exceeding some threshold).

The parallelism analysis in Figure 6 provides the only aggregate evidence: the gap between top-1, top-2, and top-3 cumulative values is larger for the first thought than the second. This is consistent with broader exploration early and convergence late, which is characteristic of BFS. However, the figure reports cumulative values without showing the raw distribution — it doesn't tell us how many distinct candidates the model actually considers (is it always 2-3, or sometimes 5-10?), or whether the convergence pattern is universal across examples or driven by a subset.

The height hypothesis in Figure 7 provides a mechanistic explanation for why BFS would be beneficial: correct nodes at high heights receive uncertain value estimates, and these become more accurate at low heights, so deferring commitment until lower heights (via maintaining multiple paths) improves accuracy. This is a correlation, not a causal demonstration. The paper does not manipulate node height independently (e.g., by constructing graphs with systematically varied heights) to show that the BFS advantage is causally driven by height rather than some other graph property. The correlation is consistent with the hypothesis but does not rule out alternative explanations (e.g., that continuous thoughts simply encode the correct path's information more efficiently regardless of graph structure).

The paper also does not compare Coconut's search behavior to explicit search methods (e.g., Tree of Thoughts with GPT-2 on ProsQA), which would help calibrate how effective the emergent BFS is relative to explicit search. Without such a comparison, we cannot assess whether the emergent BFS is as effective as explicit search, or whether it's a partial but useful approximation.

Claim 2: The multi-stage curriculum is necessary for learning continuous reasoning. The w/o curriculum ablation (Table 1) strongly supports this: on GSM8k, 14.4% vs. 16.5% for No-CoT; on ProntoQA, 52.4% vs. 93.8% for No-CoT; on ProsQA, 76.1% vs. 76.7% for No-CoT. The failure is most dramatic on ProntoQA, where the w/o curriculum variant collapses to near-random performance despite the task being solvable with high accuracy by No-CoT. This is strong evidence that the curriculum is necessary.

However, the paper does not ablate the curriculum structure itself — it only compares "full curriculum" to "no curriculum." It does not test whether the specific number of stages (6 for logical reasoning, 3+1 for GSM8k) is optimal, whether replacing from the beginning is better than replacing from the end, whether the optimizer state reset at stage boundaries is necessary, or whether alternative curricula (e.g., gradually increasing k without explicitly removing language steps) would work. The contribution is the demonstration that some curriculum is necessary, but the specific curriculum design choices are not validated as optimal or even particularly important beyond the iCoT-inspired structure.

Claim 3: Continuous reasoning is more token-efficient than language CoT. Table 1 provides clear evidence: on ProsQA, Coconut achieves 97.0% accuracy with 14.2 tokens vs. CoT's 77.5% with 49.4 tokens — both more accurate and more efficient. On ProntoQA, Coconut achieves 99.8% with 9.0 tokens vs. CoT's 98.8% with 92.5 tokens — roughly equivalent accuracy with ~10× fewer tokens. Figure 8 (panel I) shows Coconut's Pareto improvement over language internalization on GSM8k. These results are clear and well-supported.

The caveats: the efficiency advantage is measured in generated tokens, which is a reasonable proxy for inference cost but ignores the cost of the continuous thought forward passes. Coconut with k continuous thoughts requires k additional forward passes beyond standard generation. The paper addresses this in Appendix B with clock-time measurements (Table 4): Coconut is faster than CoT on all three datasets (0.09s vs. 0.26s on GSM8k, 0.11s vs. 0.85s on ProntoQA, 0.15s vs. 0.47s on ProsQA), so the continuous thought overhead is more than compensated by the reduction in generated tokens. However, these measurements are on a single A100 with batch size 1 — the tradeoff may differ in batched inference where the sequential forward passes for continuous thoughts become the bottleneck.

Additionally, the token efficiency comparison against CoT is favorable in part because CoT generates full verbose reasoning chains (92.5 tokens on ProntoQA), which is the standard CoT baseline but represents an upper bound on CoT's verbosity. A "concise CoT" baseline (Madaan and Yazdanbakhsh, 2022) that generates shorter reasoning chains would narrow the efficiency gap and would help disentangle whether the efficiency gain comes from the continuous representation or simply from being less verbose.

Claim 4: Coconut's continuous reasoning does not match CoT on GSM8k. This is an honest negative result: Coconut achieves 34.1% vs. CoT's 42.9% on GSM8k (Table 1). The paper acknowledges this and frames it as a tradeoff — Coconut offers better efficiency but lower absolute accuracy on this task. However, several factors complicate this interpretation:

The GSM8k training data is a synthetic dataset generated by Deng et al. (2023), not the original GSM8k training set. The paper does not specify the size or quality of this synthetic dataset, making it impossible to compare these numbers to standard GSM8k results reported in the literature. CoT on original GSM8k with GPT-2 would presumably achieve different numbers.
The c = 3 experiments on GSM8k showed instability (Appendix C.1), suggesting the training procedure may not have been fully optimized for this task. With better tuning (finer-grained curriculum, different learning rates, longer training), Coconut might close the gap with CoT — the current result represents what was achieved with a specific set of hyperparameters, not an upper bound.
The larger model experiments (Appendix C.2) show smaller gains from Coconut on Llama models (31.7% vs. 26.0% on Llama 3.2-3B; 43.6% vs. 42.2% on Llama 3-8B), and neither approaches the CoT baseline for these models (which is not reported). This raises the question of whether the continuous thought benefits demonstrated on GPT-2 will scale to larger, more capable models.

Overall assessment. The experiments support the paper's central conceptual claims — that continuous reasoning enables search-like behavior, that the curriculum is necessary, that efficiency improves — but the evidence is stronger for some aspects than others. The systematicity of the BFS pattern is demonstrated on a single case study with aggregate parallelism analysis that is consistent with BFS but does not directly quantify how often the pattern occurs or how it compares to explicit search. The necessity of the curriculum is convincingly demonstrated, but the specific curriculum design is not ablated. The efficiency advantage is clear in token space and clock time at batch size 1, but may not extend to batched inference or to comparisons with optimized CoT baselines. The GSM8k result shows that continuous reasoning is not universally superior to language CoT — the advantage is task-dependent, concentrated on planning-intensive problems with high branching factors (ProsQA) rather than more linear computational tasks (GSM8k, where arithmetic dominates over search). The paper's primary contribution is establishing the feasibility and characterizing the properties of latent reasoning at a small scale, with the recognition that significant further work is needed to make it competitive with or superior to CoT across all tasks and scales.

Missing experiments that would strengthen the paper:

A comparison of Coconut's emergent BFS to explicit tree search methods (e.g., Tree of Thoughts, beam search) on ProsQA, to calibrate how effective the latent search is relative to explicit search.
Aggregate statistics on the BFS pattern across the full test set (not just one example): frequency of multi-path maintenance, correlation between BFS behavior and accuracy improvement.
Ablations on the curriculum structure: replacing from end vs. beginning, varying number of stages, necessity of optimizer resets.
Comparison to a "concise CoT" baseline (Madaan and Yazdanbakhsh, 2022) to disentangle efficiency gains from continuous representation vs. reduced verbosity.
Experiments on more model families and scales (only GPT-2 for main results, only preliminary Llama 3 experiments).
Batched inference efficiency measurements, since the sequential forward passes for continuous thoughts may limit throughput in deployment settings where batch size > 1.
On GSM8k, reporting CoT baselines with the same synthetic training data and a direct comparison to standard GSM8k results to contextualize the numbers.
A c = 3 experiment on GSM8k with a finer-grained curriculum (adding one thought per sub-stage) to test whether the instability is a curriculum design problem rather than a fundamental limitation.

6. Limitations and Trade-offs

6.1 Continuous Reasoning Training Requires Language Chain-of-Thought Supervision, Making It Dependent on the Very Medium It Seeks to Replace

The assumption or constraint. Coconut's multi-stage curriculum requires explicit chain-of-thought reasoning data — full language reasoning chains — to bootstrap the learning of continuous thoughts. The model is initially trained on standard CoT instances (stage 0), and subsequent stages progressively replace language steps with continuous thoughts while still using the remaining language steps as training targets. The paper is transparent about this dependency:

"While the multi-stage training used for Coconut has proven effective, further research is definitely needed to develop better and more general strategies for learning reasoning in latent space, especially without the supervision from language reasoning chains." (Section 5.3)

The consequence. This creates a fundamental circularity in the method's value proposition. Coconut is motivated by the claim that language is a suboptimal medium for reasoning (Section 1), and the experiments demonstrate that continuous representations enable superior reasoning patterns (Section 4). Yet the method cannot learn these superior representations without first obtaining high-quality language reasoning chains — the very artifact it claims to improve upon. For any new reasoning domain where CoT data does not exist or is expensive to produce, Coconut cannot be applied directly. Moreover, the quality of the learned continuous reasoning is bounded by the quality of the language CoT supervision: if the CoT data encodes suboptimal reasoning strategies (e.g., greedy depth-first search rather than breadth-first search), the continuous thoughts may inherit those limitations even if the continuous medium is in principle capable of better strategies. The emergent BFS pattern documented in Section 4 is therefore not purely an emergent property of continuous reasoning — it is shaped by the specific language reasoning chains used during training, and different CoT data might produce different (potentially worse) continuous reasoning behavior.

What evidence exists in the paper. The w/o curriculum ablation in Table 1 provides direct evidence for this limitation. When trained directly with continuous thoughts and only question-answer pairs (no intermediate language supervision at any stage), Coconut achieves 14.4% on GSM8k — worse than the No-CoT baseline at 16.5%. On ProntoQA, it collapses to 52.4% (vs. 93.8% for No-CoT), and on ProsQA, it achieves 76.1% (matching No-CoT's 76.7%). These results demonstrate that the model cannot discover useful continuous representations from final-answer supervision alone — the language CoT scaffolding is strictly necessary. The paper does not test whether alternative supervision signals (e.g., process rewards, outcome verifiers, reinforcement learning) could substitute for language CoT data, nor does it characterize how the quality of the CoT data affects the quality of the learned continuous reasoning.

Mitigation status. The paper does not attempt to mitigate this limitation. It explicitly flags it as an important direction for future work (Section 5.3) and acknowledges that developing training strategies that do not require language reasoning chains is an open problem. The multi-stage curriculum, while effective, is presented as a first-generation approach that works but is not claimed to be the final solution. The paper does not explore whether continuous reasoning could be learned from weaker signals (e.g., only final-answer correctness with more sophisticated optimization, or from process-level reward models rather than full language chains) or whether pretraining on latent reasoning objectives could reduce the dependency on language supervision at fine-tuning time.

6.2 Training and Inference Require Sequential Forward Passes That Cannot Be Parallelized, Limiting Throughput at Scale

The assumption or constraint. Each continuous thought requires a full forward pass through the transformer, and these passes are sequentially dependent: the hidden state from thought t becomes the input embedding for thought t+1, so thought t+1 cannot begin until thought t completes. The paper acknowledges this explicitly:

"the sequential nature of the multiple forward passes poses challenges for parallelism. Further optimizing the training efficiency of Coconut remains an important direction for future research." (Section 3, Training Details)

For k continuous thoughts, training requires k+1 sequential forward passes (Section 3), and inference requires k+2 (Section 3.4, Inference Process).

The consequence. This sequential dependency creates a tension between the method's intellectual appeal (rich latent computation) and its practical deployment characteristics. In standard autoregressive language generation, each token prediction requires one forward pass, but these can be batched across multiple independent sequences (batch parallelism) and, within a sequence, can use KV caching to make each forward pass efficient. Coconut's continuous thoughts add forward passes that do not produce any output tokens — they are "pure computation" overhead. While the paper shows that the total wall-clock time is lower than CoT at batch size 1 (Appendix B, Table 4: 0.09s for Coconut vs. 0.26s for CoT on GSM8k), this advantage may not persist in batched inference. In a batched setting with B independent sequences, CoT's token generation can exploit the fact that all B sequences generate tokens in parallel within each forward pass (batch parallelism over the sequence dimension). Coconut's continuous thoughts must be executed sequentially for each sequence, and while different sequences' thoughts at the same depth could in principle be batched together, the varying number of continuous thoughts needed per problem (if adaptive termination were used) would create load-balancing challenges. Moreover, for latency-critical applications where a single query must be answered as quickly as possible, the k sequential forward passes add latency that is proportional to k, regardless of how many tokens are ultimately generated. On GSM8k with c=2 and 3 stages, k=6 continuous thoughts add 6 sequential forward passes of latency before any output token is produced — this may be acceptable if it reduces total generation time, but it means the time-to-first-token is substantially higher than for CoT, which begins generating immediately.

What evidence exists in the paper. Appendix B, Table 4 provides clock-time measurements at batch size 1 on an A100 GPU. These show Coconut is faster than CoT overall (0.09s vs. 0.26s on GSM8k; 0.11s vs. 0.85s on ProntoQA; 0.15s vs. 0.47s on ProsQA). However, these measurements are at batch size 1 and on a single GPU — the paper provides no batched inference measurements, no throughput analysis (queries per second at various batch sizes), and no latency breakdown (time spent in continuous thought forward passes vs. language generation). The paper acknowledges the parallelism challenge as a direction for future work but does not characterize how the tradeoff changes with scale (e.g., larger models where each forward pass is more expensive, or deployment scenarios where multiple queries arrive simultaneously).

Mitigation status. The paper does not attempt to mitigate this limitation beyond noting the clock-time advantage at batch size 1. It does not explore techniques that could reduce the sequential bottleneck, such as: using a smaller model for the continuous thought computations (disentangling reasoning depth from generation capacity), applying distillation to reduce the number of continuous thoughts needed, or designing architectures where some continuous thought computations can be parallelized (e.g., predicting multiple future hidden states simultaneously rather than sequentially). The paper frames training efficiency optimization as future work.

6.3 All Core Results Use a Single Small Model (GPT-2) on Small-Scale Reasoning Benchmarks; Scalability to Larger Models and Real-World Tasks Is Unproven

The assumption or constraint. The paper's main experimental results are conducted entirely on pre-trained GPT-2 (124M parameters) using datasets of modest size: GSM8k (synthetic training data generated by Deng et al., 2023, with 1,319 test examples), ProntoQA (800 test examples), and ProsQA (500 test examples). The paper states:

"We believe that these findings underscore the potential of latent reasoning and could provide valuable insights for future research." (Section 1)

This framing positions the work as a proof of concept — it demonstrates that continuous latent reasoning is possible and characterizes its properties at a small scale, without claiming that the method would directly transfer to production-scale models or tasks.

The consequence. The central empirical claims of the paper — that continuous reasoning enables emergent BFS, that it improves accuracy-efficiency tradeoffs, that it reduces hallucinations on planning-intensive tasks — are established only within a narrow experimental regime. Several aspects of the results may not generalize to larger models or more diverse tasks:

Model scale: The paper's own preliminary experiments with Llama 3.2-3B and Llama 3-8B (Appendix C.2, Table 5) show substantially smaller gains from Coconut than with GPT-2. On Llama 3.2-3B, Coconut achieves 31.7% vs. No-CoT's 26.0% on GSM8k — a 5.7 percentage point improvement, compared to 17.6 points for GPT-2. On Llama 3-8B, the gap shrinks to 1.4 points (43.6% vs. 42.2%). The paper hypothesizes that "larger models have already undergone extensive language-focused pre-training, making the transition to latent reasoning more challenging," but this explanation is speculative. It could also be that the training recipe (3 epochs in stage 0, 1 epoch per subsequent stage) is undertraining the larger models, or that the benefits of continuous reasoning relative to No-CoT diminish as the base model's direct reasoning capability improves.
Task diversity: All three benchmarks are structured reasoning tasks with clear correct answers: grade-school math (GSM8k), deductive logical reasoning from given premises (ProntoQA), and graph-based logical planning (ProsQA). The paper does not test on tasks requiring factual knowledge retrieval, multi-hop question answering over unstructured text, code generation, commonsense reasoning, or open-ended generation. The BFS pattern demonstrated on ProsQA depends on the task having a graph structure with branching paths — it is unclear whether continuous reasoning would provide similar benefits on tasks where the reasoning structure is more linear or where the challenge is knowledge integration rather than search.
Training data scale: The largest training set (GSM8k synthetic) has 385,620 examples. Modern LLM training datasets for reasoning often contain millions of examples. The multi-stage curriculum's effectiveness at larger data scales (where early stages may see the model converge quickly, leaving later stages with most of the training budget) is unexamined.

What evidence exists in the paper. The GSM8k experiments use GPT-2 and a synthetic training set generated by Deng et al. (2023), not the original GSM8k data — making it impossible to compare Coconut's 34.1% accuracy directly to standard GPT-2 CoT results on GSM8k reported elsewhere. The Llama 3 experiments (Appendix C.2, Table 5) use c=1 (not the optimal c=2 found for GPT-2), are trained for only 3+1×N epochs (vs. 6+3×N for GPT-2), and do not report CoT baselines for comparison — only No-CoT is shown. These are preliminary experiments that raise more questions than they answer about scalability. The paper acknowledges this limitation implicitly through its framing as "initial exploration" and "potential of latent reasoning" (Section 1, Section 6), but does not provide a systematic scalability analysis (varying model size while controlling for training compute, measuring how the continuous thought benefit changes with model scale and pretraining quality).

Mitigation status. The paper makes no attempt to demonstrate scalability beyond the preliminary Llama experiments in Appendix C.2. It does not discuss what engineering or algorithmic changes would be needed to apply Coconut to models with hundreds of billions of parameters, how the multi-stage curriculum would interact with large-scale pretraining, or whether continuous thought training could be integrated into pretraining rather than applied post-hoc. The paper mentions future work on "pretraining in latent space" (Section 6, Appendix C.2) as a potential path to improving scalability, but this is speculative.

6.4 The Latent Tree Search Interpretation Relies Heavily on a Single Case Study; Systematic Evidence for the BFS Claim Is Limited

The assumption or constraint. Section 4's central claim — that continuous reasoning enables an emergent breadth-first search (BFS) pattern where the model "can encode multiple alternative next reasoning steps" and "perform a breadth-first search to solve the problem, rather than prematurely committing to a single deterministic path like CoT" — is supported primarily by a detailed analysis of one example (Figure 4 case study, Figure 5 tree visualization, Figures 6 and 7 aggregate analysis). The probing methodology (forcing language generation after intermediate continuous thoughts and computing value functions over candidate next concepts) is applied systematically (Figures 6 and 7 are aggregate across the test set), but the specific BFS claim — that the model maintains multiple alternatives, evaluates their children, and converges on the correct path — is demonstrated in detail only for the single case study.

The consequence. The BFS interpretation may not capture the full range of reasoning behaviors that continuous thoughts enable. Several alternative interpretations are consistent with the aggregate data:

The model might encode a ranked list of candidates rather than simultaneously representing multiple alternatives — Figure 5 shows probability mass distributed across candidates at step 1, but this could simply be a softmax distribution over a single representation that assigns higher probability to the most likely candidate, not a genuine maintenance of multiple distinct paths.
The redistribution of probability mass from step 1 to step 2 (lempus drops, rorpus rises) could result from the model re-evaluating a single representation with additional computation rather than from exploring multiple branches of a search tree — the second continuous thought provides more computation, and this additional computation may simply refine the model's estimate of which path is correct, without the model ever explicitly encoding the alternative.
The parallelism analysis in Figure 6 shows that the gap between top-1, top-2, and top-3 cumulative values narrows from the first to the second thought, which is consistent with convergence from exploration to exploitation. However, this pattern is also consistent with a model that is simply uncertain at step 1 (flat probability distribution, high entropy) and becomes more certain at step 2 (peaked distribution, low entropy) — no explicit multi-path maintenance is required.

These alternative interpretations do not contradict the empirical findings (the model does improve accuracy, and it does distribute probability mass across multiple candidates early), but they represent different mechanistic accounts of how continuous thoughts achieve this. The BFS account claims the model explicitly maintains and evaluates multiple paths; the uncertainty-resolution account claims the model maintains a single integrated representation that is refined through additional computation. Disambiguating these accounts would require probing the internal geometry of the continuous thoughts (e.g., using linear probes to detect whether distinct path representations are linearly separable in the hidden state, or intervening on the hidden state to remove specific candidate paths and observing whether downstream probabilities shift accordingly). The paper does not conduct such analyses.

What evidence exists in the paper. Section 4 provides three pieces of evidence:

Qualitative case study (Figure 4, Figure 5): One example where CoT hallucinates, k=1 Coconut produces a wrong-target path, and k=2 Coconut finds the correct path. The probing shows probability distribution at step 1 across four candidates and redistribution at step 2 favoring the correct downstream node. This is a single example — the paper does not report how many examples in the test set show a similar pattern of value redistribution from a non-highest initial candidate to a correct final candidate.
Parallelism analysis (Figure 6): Aggregate over the test set showing the cumulative values of top-1, top-2, and top-3 candidates. The gaps between these curves are larger for the first thought than the second. This suggests the model is less concentrated on a single candidate early on, but it does not directly measure how many distinct paths the model maintains or whether those paths correspond to different branches of the reasoning graph.
Height hypothesis (Figure 7): Shows that correct nodes at lower heights receive higher values, and incorrect nodes at lower heights receive lower values. This explains why deferring decisions (via multi-step reasoning) would be beneficial, but it does not demonstrate that continuous thoughts achieve this by maintaining multiple paths rather than by refining a single path estimate.

The paper uses the BFS language throughout Section 4 ("breadth-first search," "parallel exploration," "multiple alternative next reasoning steps"), but the gap between this claim and the evidence is a significant interpretative limitation. The aggregate data is consistent with BFS but does not uniquely support it.

Mitigation status. The paper does not address these alternative interpretations or provide analyses that would specifically differentiate BFS from uncertainty resolution. The probing methodology could be extended to test these alternatives (e.g., by intervening on the first continuous thought to remove one candidate and measuring whether the second continuous thought's values shift accordingly), but no such experiments are reported. The theoretical work of Zhu et al. (2025b), which the paper cites, provides a framework for understanding how superposition states could encode multiple reasoning paths simultaneously — but the paper does not empirically validate whether this theoretical mechanism is actually what the model learns. The BFS claim remains an intriguing hypothesis supported by suggestive evidence rather than a rigorously established mechanism.

6.5 The Difficulty Estimation and Adaptive Budget Allocation Framework from Prior Test-Time Compute Scaling Work Is Absent; Coconut Uses a Fixed Latent Reasoning Budget for All Problems

The assumption or constraint. Coconut uses a constant-length latent reasoning budget: the number of continuous thoughts at inference time is fixed to match the final training stage (k = k_max for all problems, regardless of difficulty). The paper states:

"we insert a <bot> token immediately following the question tokens. For <eot>, we consider two potential strategies: a) train a binary classifier on latent thoughts to enable the model to autonomously decide when to terminate the latent reasoning, or b) always pad the latent thoughts to a constant length. We found that both approaches work comparably well. Therefore, we use the second option in our experiment for simplicity, unless specified otherwise." (Section 3, Inference Process)

The consequence. This design choice means Coconut cannot adapt its reasoning depth to problem difficulty — every problem, whether trivially easy or extremely hard, receives exactly the same number of continuous thoughts (6 for ProsQA/ProntoQA, 6 for GSM8k with c=2). This is inefficient in two directions. For easy problems that require few or no reasoning steps, the model must "waste" continuous thoughts on identity-like transformations — computation that doesn't improve the answer and adds latency. For very hard problems that might benefit from extended reasoning (beyond 6 continuous thoughts), the model is capped at its training budget and cannot "think longer." The paper's own analysis in Section 4.3 and Figure 5 suggests that the model's reasoning converges within 2 continuous thoughts on the example problem (the value redistribution from step 1 to step 2 identifies the correct path), implying that the remaining 4 continuous thoughts in the k=6 configuration may be unnecessary for that specific problem.

Moreover, this fixed-budget approach misses an opportunity that the paper's own framework enables. The multi-stage curriculum trains the model to handle any number of continuous thoughts from 0 to k_max — the model weights support variable-length latent reasoning. A natural extension would be to adaptively select k per problem, using more continuous thoughts for harder problems and fewer for easier ones, analogous to the compute-optimal test-time scaling framework of Snell et al. (2024). The paper does not explore this direction, even though the ProsQA analysis in Figure 3 shows that the accuracy-vs-k curve is concave — most of the gain comes from the first 2-3 continuous thoughts, with diminishing returns thereafter. This suggests that an adaptive policy allocating fewer thoughts to most problems and reserving the full budget for the hardest cases could achieve similar accuracy with substantially lower average inference cost.

What evidence exists in the paper. Figure 3 (left) shows that on ProsQA, accuracy increases from ~77.5% at k=0 to ~90% at k=2 to ~97% at k=6 — the accuracy gain from k=2 to k=6 is roughly 7 percentage points, compared to 12.5 points from k=0 to k=2. This diminishing returns pattern suggests that many problems are already solved with 1-2 continuous thoughts, and the additional thoughts primarily help a subset of harder problems. The paper does not report the distribution of "minimum k needed for correct answer" across the test set, which would directly quantify the potential efficiency gain from adaptive allocation. The constant-length decision is described as a simplification ("for simplicity"), and the paper acknowledges the alternative (learned termination) but does not pursue it.

Mitigation status. The paper acknowledges the possibility of learned termination (strategy a) but does not implement or evaluate it beyond stating that "both approaches work comparably well" — without reporting numbers for the learned termination approach. The lack of adaptive budget allocation is not presented as a limitation but rather as a simplification. Future work on dynamic reasoning budgets is not explicitly discussed, though it is a natural extension of the paper's framework. The connection to compute-optimal test-time scaling (Snell et al., 2024) is not drawn in the paper, despite the conceptual parallel: both works study how to allocate inference computation to maximize accuracy under a budget constraint, and both find that uniform allocation is suboptimal.

6.6 The Method Has Not Been Tested on Tasks Without Clear Correctness Signals or Ground-Truth Answers, Limiting Its Demonstrated Applicability to Closed-Form Reasoning Problems

The assumption or constraint. All experiments in the paper are on tasks with unambiguous ground-truth answers that can be evaluated for correctness with exact matching or logical verification: GSM8k (numerical answers), ProntoQA (True/False), and ProsQA (binary choice between two concepts, e.g., "Is Tom a lempus or scrompus?"). The training objective is the standard next-token prediction loss on the answer tokens (and remaining language reasoning steps during the curriculum), which requires knowing exactly what the correct answer tokens are for each training example.

The consequence. This evaluation scope leaves open the question of whether continuous reasoning would provide benefits on tasks where correctness is ambiguous, multi-dimensional, or subjective. Many important reasoning tasks lack clean answer verification: open-ended generation (essay writing, creative problem-solving), multi-step planning with multiple valid solutions (e.g., "Plan a trip to Paris under a budget"), dialogue and negotiation, and tasks where the evaluation metric is a learned reward model or human preference rather than an exact match. For these tasks, the training signal for continuous thoughts would need to come from a different source — perhaps a process reward model, reinforcement learning from human feedback, or self-consistency measures — but the paper provides no evidence that continuous reasoning can be learned from such signals. The current training procedure depends on the existence of language CoT data with correct final answers, which is feasible for math and formal logic but expensive or impossible for open-ended domains.

Furthermore, the probing methodology used to interpret continuous thoughts (decoding through the LM head to see what language tokens they correspond to, as in Figures 5 and 9) depends on the model's vocabulary being able to express the relevant reasoning concepts as discrete tokens. For reasoning that involves continuous quantities, perceptual features, or abstractions that don't map neatly to single words, the decoding-based interpretability approach would break down, making it harder to understand what the continuous thoughts encode or to diagnose failures.

What evidence exists in the paper. None. The paper tests exclusively on three reasoning benchmarks with closed-form answers: GSM8k (math word problems with numerical answers), ProntoQA (deductive logic with True/False answers), and ProsQA (graph-based logic with binary concept disambiguation). The paper does not discuss the applicability of Coconut to open-ended reasoning tasks, does not propose alternative training signals, and does not address the interpretability challenge for reasoning that does not decompose neatly into language concepts.

Mitigation status. The paper does not address this limitation. It is not flagged as a limitation or a direction for future work. The paper's framing emphasizes the potential of latent reasoning as a general paradigm, but the experimental scope is restricted to tasks where correctness is well-defined and language CoT supervision is available. Extending Coconut to tasks with weak or learned reward signals would require developing new training objectives (e.g., reinforcement learning where the reward is a learned verifier rather than exact match, or contrastive objectives that distinguish good from bad reasoning without requiring token-level supervision), none of which are explored. The recent work cited in the paper's conclusion on latent space pretraining (Geiping et al., 2025; Barrault et al., 2024; Gladstone et al., 2025) suggests possible paths forward, but these are not integrated with Coconut in the current work.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a new axis of design for LLM reasoning systems that has been largely absent from the field: the choice of representation medium for intermediate computation. Prior to Coconut, the dominant assumption was that LLMs reason by generating language — chain-of-thought is language, tree-of-thought operates over language, verifier-guided search scores language, and internalization approaches (iCoT, distillation) remove language entirely but also remove the chained computation structure. Coconut demonstrates that there is a third option: preserve the chained, multi-step reasoning structure that gives CoT its expressivity benefits, but replace the discrete language medium with a continuous latent medium that is more information-dense and supports qualitatively different reasoning patterns. This is not an incremental improvement on CoT — it is a demonstration that the reasoning medium itself is a design choice with significant consequences for both the efficiency and the algorithmic nature of the reasoning process.

The magnitude of this shift should be understood carefully. Coconut does not establish latent reasoning as universally superior to language CoT. The GSM8k results (34.1% for Coconut vs. 42.9% for CoT on the same base model) make clear that the method, as presented, does not match CoT's absolute accuracy on all tasks. The paper's contribution is not "latent reasoning beats CoT" but rather "latent reasoning enables a different point on the accuracy-efficiency frontier that is preferable under certain conditions, and those conditions can be characterized in terms of task structure." The BFS analysis in Section 4 identifies those conditions: tasks with high branching factors early in the reasoning process where greedy commitment to a single path causes irrecoverable errors. For such tasks (ProsQA), Coconut provides a ~19.5 percentage point accuracy gain while generating ~3.5× fewer tokens. For tasks with more linear reasoning chains where arithmetic or deduction dominates over search, the advantage narrows or reverses. This task-dependent characterization is the paper's most important conceptual contribution — it provides a framework for predicting when latent reasoning will help, rather than treating it as a universal upgrade.

The paper also reframes the role of the language model head and embedding layer from essential components of the reasoning architecture to optional interfaces that can be bypassed when communication (to humans or to other systems) is not the immediate goal. This is a conceptual shift with practical implications: the standard transformer architecture can be used in two distinct modes — a language generation mode that projects through the vocabulary, and a latent computation mode that feeds hidden states forward directly. The fact that the same model weights support both modes, and that the latent mode can be learned through a curriculum that bootstraps from language supervision, suggests that future LLMs could be designed with this dual-mode operation as a first-class capability rather than a post-hoc modification.

The work also reconciles a tension in the literature between two approaches to efficient reasoning. On one side, pause tokens (Goyal et al., 2023) and filler tokens (Pfau et al., 2024) give models extra computation without language generation but do not extend expressivity — they are "flat" computation that doesn't create the recursive depth-increasing loop that CoT provides. On the other side, internalization methods (Deng et al., 2023; Deng et al., 2024) compress reasoning into the model's existing depth but lose the ability to scale computation to problem difficulty. Coconut synthesizes these: it provides the expressivity benefits of recursive computation (like CoT) with the language-free efficiency of pause tokens (no vocabulary bottleneck), while retaining the ability to scale reasoning depth by chaining more continuous thoughts. The key ablation (pause as thought: 24.1% vs. continuous thoughts: 34.1% on GSM8k, Table 1) demonstrates that the recursive feedback — not just additional computation positions — is responsible for the majority of the benefit, and the w/o thought ablation (21.6%) shows that continuous thoughts provide benefit beyond simply removing tokens.

The research directions that become more attractive after this work include: developing training objectives that can learn latent reasoning without language CoT supervision (the paper's most critical limitation), scaling continuous reasoning to larger models and more diverse tasks (the Llama experiments in Appendix C.2 show diminishing returns that need to be understood), designing architectures where the latent reasoning mode is a native capability rather than a training-time modification, and building adaptive systems that can dynamically switch between language and latent reasoning based on problem characteristics. The research directions that become less attractive are those that treat language as the only viable reasoning medium and focus exclusively on making language CoT more efficient through prompting or compression — Coconut demonstrates that changing the medium can shift the entire efficiency-accuracy tradeoff curve outward, making medium-change a more fundamental improvement than same-medium optimization.

Follow-Up Research This Work Enables

Directly measuring whether continuous thoughts encode genuinely multi-path representations or merely refined single-path estimates. The paper's BFS claim is supported by a single case study (Figures 4, 5) and aggregate parallelism analysis (Figure 6) that is consistent with BFS but also consistent with a model that maintains a single integrated representation refined through additional computation. A strong follow-up would use linear probing or intervention methods to test whether distinct reasoning paths are linearly separable in the continuous thought hidden states. Specifically, for ProsQA problems, train linear classifiers on the first continuous thought's hidden state to predict whether the model will eventually output a path through lempus vs. grimpus. If distinct path representations are separable (high probe accuracy), this supports the multi-path encoding claim. If not, the benefit likely comes from iterative refinement of a single representation. An intervention experiment would modify the first continuous thought by subtracting the probe direction for a specific path and measuring whether downstream probabilities for that path's children decrease — causal evidence for multi-path encoding. This experiment is now tractable because Coconut provides exactly the mechanism (intermediate hidden states that can be probed and intervened upon) that makes this analysis possible, and the multi-stage curriculum ensures the model can handle probing at any k value.

Characterizing how CoT data quality shapes the emergent continuous reasoning strategy. The paper shows that continuous reasoning depends on language CoT supervision (the w/o curriculum ablation fails catastrophically), but it does not test how the content of that supervision matters. A natural experiment would train Coconut variants where the stage-0 CoT data is manipulated: (1) CoT that always follows a greedy depth-first search through the reasoning graph, (2) CoT that explicitly explores multiple alternatives before committing (breadth-first style language CoT), (3) CoT with varying amounts of noise (some hallucinations injected). Then measure whether the emergent BFS pattern in continuous thoughts differs across conditions. The hypothesis from the paper's framework would be that continuous thoughts can transcend the search strategy encoded in the language data — the "greedy CoT" condition might still produce BFS-like continuous reasoning because the continuous medium inherently supports it. If instead the continuous reasoning strategy faithfully mirrors the language supervision strategy regardless of continuous medium capability, this would reveal a fundamental limitation: continuous reasoning can only learn what language CoT demonstrates, not discover superior strategies on its own. This experiment would clarify whether Coconut's BFS is an emergent property of the continuous medium or a learned imitation of something implicit in the CoT data.

Developing and evaluating learned termination policies for continuous reasoning length. The paper uses constant-length latent reasoning at inference time (Section 3), but notes that a learned binary classifier to decide when to exit latent mode "works comparably well" — without reporting numbers. A direct follow-up would train such a classifier on the continuous thought hidden states, using the final training stage's checkpoint, and evaluate whether adaptive termination can match or exceed the constant-length approach while using fewer continuous thoughts on average. The evaluation should include: (1) accuracy vs. average k used (efficiency-accuracy Pareto frontier for adaptive vs. constant policies), (2) calibration of the termination classifier (does the model exit early on problems it gets wrong, or does it continue thinking when uncertain?), (3) comparison to an oracle policy that terminates at the minimum k needed for correct answer (to bound the best possible adaptive policy). This is important because the paper's own analysis shows diminishing returns beyond k=2-3 on ProsQA (Figure 3: ~90% at k=2, ~97% at k=6), suggesting significant efficiency potential from adaptive termination. The iCoT comparison (which achieves 98.2% on ProsQA with 8.2 tokens vs. Coconut's 97.0% with 14.2 tokens, Table 1) further motivates this — iCoT essentially implements extreme adaptive termination (always k=0 for the logical reasoning tasks), and a learned policy could find a middle ground that allocates continuous thoughts only when they provide marginal benefit.

Testing whether continuous reasoning benefits transfer to reinforcement learning-based training objectives. The paper trains Coconut exclusively with supervised next-token prediction loss on language reasoning chains and answers (Section 3). A critical follow-up would replace the language CoT supervision with a reinforcement learning signal — for example, using GRPO-style training (Guo et al., 2025) where the model generates continuous thoughts, then produces an answer, and receives a reward only for final-answer correctness (or a process reward from a trained PRM). The specific experiment: train Coconut variants on GSM8k where the multi-stage curriculum is replaced with RL fine-tuning, starting from a base model that has been pretrained with some continuous thought exposure. The key question is whether the gradient signal from RL (which avoids the need for language CoT data) can substitute for the curriculum in teaching useful continuous representations. The w/o curriculum ablation shows that supervised learning from question-answer pairs alone fails (14.4% on GSM8k), but RL provides a different optimization dynamic (exploration through sampling, credit assignment through reward, potential for self-correction through rollouts) that might succeed where supervised learning fails. A positive result (RL-trained Coconut matching or exceeding curriculum-trained Coconut) would remove the paper's most significant limitation — dependency on language CoT data — and open the door to applying latent reasoning to domains where CoT data doesn't exist. A negative result (RL also fails) would suggest that the semantic scaffolding from language supervision is fundamentally necessary, redirecting research toward alternative scaffolding signals rather than alternative optimization methods.

Scaling latent reasoning to multi-modal or continuous-answer tasks where language CoT is unnatural. The paper evaluates on text-based reasoning with discrete symbolic answers (numbers, True/False, concept labels). An ambitious extension would test continuous reasoning on tasks where the reasoning target is itself continuous or multi-modal — for instance, predicting a trajectory for robot planning, generating an image that satisfies complex constraints, or outputting a structured JSON object from a complex specification. The continuous thought mechanism is naturally suited for these domains because it doesn't force intermediate reasoning through a discrete vocabulary — a continuous thought could encode spatial coordinates, pixel values, or abstract relational features that would be cumbersome to express in language. A specific experiment: use Coconut's continuous thoughts for a visual reasoning task where the input is a diagram and the output is a set of spatial coordinates (e.g., "where should the next block be placed to eventually reach the goal configuration?"). Train with language CoT in early stages (descriptions of spatial relationships), then transition to continuous thoughts in later stages, and measure whether the latent reasoning can encode spatial information more efficiently than language descriptions. This would test the limits of the "language is optimized for communication, not reasoning" hypothesis (Fedorenko et al., 2024) that motivates the paper — if continuous reasoning provides larger benefits on spatial/continuous tasks than on symbolic reasoning tasks, this would strengthen the argument that language is a bottleneck specifically for non-linguistic reasoning content.

Analyzing the internal geometry of continuous thoughts across the chain to understand what information is encoded at each step. The paper's probing analysis (Section 4) decodes continuous thoughts into language tokens to reveal the reasoning tree, but this captures only what is expressible in the model's vocabulary — it misses any information that doesn't correspond to word tokens. A deeper analysis would use unsupervised methods to characterize the geometry and information flow across continuous thoughts: (1) measure the effective dimensionality of continuous thought hidden states (using PCA or intrinsic dimension estimation) at each position in the chain — does dimensionality decrease as the model converges on a solution, consistent with the BFS-to-focused transition? (2) compute representational similarity between continuous thoughts at different chain positions — do early thoughts cluster by the set of alternatives they encode, while later thoughts cluster by the specific path chosen? (3) use mutual information estimation between continuous thoughts and input features (e.g., which graph edges are relevant) to quantify what information is preserved vs. discarded at each step. This analysis would provide mechanistic detail beyond what the language-decoding probe can reveal, and would help answer whether continuous thoughts genuinely encode "superposition states" of multiple reasoning paths (as Zhu et al., 2025b theorize) or whether they implement a different computational mechanism entirely. The paper's infrastructure (multi-stage curriculum producing a model that can be probed at any k) makes this analysis straightforward to implement.

Practical Applications and Downstream Use Cases

Cost-efficient batch inference for structured reasoning tasks with high branching factors. For organizations running large-scale logical reasoning evaluations — such as automated theorem proving, compliance checking against regulatory rule sets, or knowledge graph traversal for question answering — Coconut offers a concrete efficiency advantage over language CoT. On ProsQA, Coconut achieves 97.0% accuracy with 14.2 generated tokens, compared to CoT's 77.5% with 49.4 tokens (Table 1). For a batch of 100,000 problems, this translates to generating roughly 1.42 million tokens with Coconut vs. 4.94 million tokens with CoT, while simultaneously achieving ~19.5 percentage points higher accuracy. At standard API pricing for language model inference (where cost scales with output tokens), this represents a ~3.5× cost reduction with improved quality. The clock-time measurements (Appendix B, Table 4) show 0.15 seconds per problem for Coconut vs. 0.47 seconds for CoT on a single A100 — a ~3.1× speedup that compounds in batch processing. The key caveat is that this advantage is demonstrated on reasoning tasks with search-intensive graph structures; on more linear reasoning tasks (GSM8k), Coconut's accuracy is lower than CoT, so the deployment decision should be conditioned on task analysis. The paper's BFS analysis (Section 4.3) provides a diagnostic: if the task's underlying reasoning structure has multiple plausible paths that require evaluation before commitment, Coconut is likely to provide both accuracy and efficiency gains.

Deployment of smaller models for complex reasoning through latent computation scaling. The paper's core finding — that replacing language reasoning steps with continuous thoughts can dramatically improve reasoning capability on search-intensive tasks — suggests a deployment architecture where a smaller base model is augmented with latent computation rather than scaling to a larger model for reasoning capability. On ProsQA with GPT-2 (124M parameters), Coconut achieves 97.0% accuracy with 6 continuous thoughts (Figure 3, k=6). A larger model without latent reasoning might be needed to match this accuracy on graph-search tasks, but Coconut shows that the smaller model can reach high accuracy through more sophisticated inference-time computation. The practical deployment scenario: on-device or edge deployment where model size is constrained by memory or latency, but the reasoning task has known graph-search structure (e.g., a mobile app for logical puzzle solving, an embedded system for fault diagnosis in a system with known dependencies). By training the small model with the Coconut curriculum, the system can use latent computation at inference time to compensate for limited parameter count, trading off the sequential latency of continuous thoughts against the memory and loading cost of a larger model. The paper's current results bound this tradeoff: on a single A100, Coconut adds ~0.06 seconds of latency for 6 continuous thoughts (Table 4: 0.15s for Coconut vs. 0.08s for No-CoT), but improves accuracy from 76.7% to 97.0% on ProsQA. For applications where this accuracy gain is mission-critical and 0.07 seconds of additional latency is acceptable, Coconut enables deployment of a model that would otherwise be inadequate.

Continuous thought-based data generation for self-improvement pipelines targeting search-intensive reasoning. When using LLMs to generate training data for reasoning tasks — as in STaR, ReST^EM, or rejection sampling fine-tuning — the quality and diversity of generated reasoning traces matter. Coconut offers a mechanism for generating higher-quality reasoning paths on search-intensive problems: use the Coconut model to produce correct answers via latent reasoning, then decode the continuous thoughts into language (or use the model's final language output) as training data for a student model. On ProsQA, Coconut with k=6 achieves ~87% Correct Path and ~10% Correct Label (Figure 3, right) — meaning ~97% of generated answers come with valid, hallucination-free reasoning (either explicit paths or directly decoded answers). CoT on the same task achieves only ~55% Correct Path with ~20% Hallucination — meaning 20% of generated data would contain incorrect reasoning traces that could poison a student model's training. Using Coconut as the data generator would produce cleaner training data, and the variable-k capability (trained once, probed at any k) means the generator could be configured to produce reasoning of varying depth by adjusting the number of continuous thoughts. The practical setup: train Coconut on ProsQA-style data, use it to generate 100,000 verified correct reasoning traces (by filtering for Correct Path outputs), and fine-tune a larger model on these traces. The paper's ablation that shows the curriculum is necessary for learning continuous reasoning (Table 1, w/o curriculum) suggests that the student model in this pipeline would still need to be trained with explicit language reasoning, not continuous thoughts — but the training data quality improvement from Coconut-generated traces could substantially boost the student's CoT performance on search-intensive tasks.