Inference Optimization2025~127 minarXiv:2504.13171

Sleep-time Compute: Beyond Inference Scaling at Test-time

Kevin Lin, Charlie Snell, Yu Wang +4Apr 17, 2025cs.AI

🎯 Pitch

By letting LLMs 'think' offline about shared contexts before a user asks, Sleep-time Compute shifts up to 5× of the reasoning cost out of the critical path without hurting accuracy—and actually boosts it by up to 18% on hard math tasks. This breaks the assumption that test-time scaling must incur high latency and cost, showing that amortizing context understanding across queries can slash per-query cost by 2.5×.

1. Executive Summary

This paper introduces sleep-time compute, a paradigm that shifts inference computation to an offline "sleep" phase — after a context is available but before a user query arrives — by prompting the model to generate inferences and re-represent the context into a form more amenable to rapid test-time answering (e.g., pre-computing intermediate quantities for math problems, summarizing relevant codebase structure for software engineering tasks). Evaluated on modified reasoning benchmarks — Stateful GSM-Symbolic and Stateful AIME, which split existing math problems into context and query portions — and on a new agentic software engineering benchmark (SWE-Features), sleep-time compute shifts the pareto frontier of test-time compute versus accuracy, reducing the test-time tokens needed by ~5× to match equivalent accuracy and improving accuracy by up to 13% on Stateful GSM-Symbolic and 18% on Stateful AIME when sleep-time compute itself is scaled. On the Multi-Query GSM-Symbolic dataset, amortizing sleep-time compute across 10 related queries about the same context reduces the average cost per query by 2.5×, establishing that the method's efficiency benefits compound with query multiplicity only when contexts are shared across queries.

2. Context and Motivation

The Core Problem: Test-Time Compute Assumes Statelessness

The paper addresses a simple but pervasive assumption in current approaches to scaling inference-time computation: that every user query arrives as an atomic, self-contained unit composed of both the background context (e.g., a codebase, a document corpus, conversation history) and the specific question, presented together at the moment of interaction. In the standard paradigm, when a user asks a question about a document, the model receives the document and the question simultaneously, reasons through the entire problem from scratch, and returns an answer. When a second question about the same document arrives moments later, the model repeats the process—rereading the document, re-deriving the same intermediate inferences, and incurring the same latency and cost for reasoning that is effectively redundant.

The paper terms this the stateless assumption: queries are treated as independent, and any computation invested in understanding the context is discarded after each response, forcing the model to recompute useful inferences anew for every subsequent question. This is not a limitation of any specific model or reasoning strategy—it is a structural assumption baked into how inference pipelines are designed and evaluated. Benchmarks for reasoning (GSM8K, MATH, AIME) present each problem as a monolithic block where all information needed to answer the question is provided at once. This makes them convenient for measuring raw reasoning capability but entirely obscures the efficiency losses from redundant context processing in stateful deployments.

Why This Matters: Latency, Cost, and Deployment Realities

The practical stakes of this problem have grown sharply with the emergence of test-time scaling as a dominant paradigm for improving LLM performance. Models like OpenAI's o1, DeepSeek-R1, and Claude's extended thinking mode demonstrate that spending more tokens at inference—generating longer chain-of-thought traces, exploring multiple solution paths, or applying verifier-guided search—can push accuracy on difficult problems far beyond what a single forward pass can achieve. However, the costs of this strategy are severe and well-documented in the paper's framing (Section 1):

Latency: Users may wait "potentially several minutes for answers." For interactive applications—coding assistants, conversational AI, document Q&A—this latency is often unacceptable regardless of accuracy gains. The paper notes that o1-pro costs "up to tens of dollars per query," and while it does not quote exact latency figures, the implication is clear: waiting minutes for a response is a non-starter for many real-world use cases.
Cost: The token overhead of extended reasoning traces compounds with query volume. In high-throughput settings where many users ask questions about shared resources (e.g., a team querying a codebase, customers asking questions about product documentation), the same context-level reasoning is paid for repeatedly.
The stateful reality of most deployments: As the paper argues, "many LLM applications are inherently stateful, and work in conjunction with persisted, re-used context." Document question-answering systems hold a corpus of documents that contextualize all user questions. Coding agents operate over a repository that persists across debugging sessions. Conversational assistants maintain dialogue history. In all these cases, the context exists before the user's query and persists after the response. The paper's insight is that this interval—when the model is "otherwise idle in sleep-time"—represents a wasted opportunity to reason about the context and prepare for likely questions.

The paper does not just identify an inefficiency; it argues that this inefficiency is the primary bottleneck preventing test-time compute from being practical in stateful, interactive settings. The problem is not that scaling test-time compute doesn't work—it demonstrably does—but that its latency and cost make it incompatible with many deployment scenarios. Sleep-time compute is proposed as a way to preserve the accuracy benefits of test-time scaling while dramatically reducing its runtime burden.

Prior Approaches and Their Limitations

The paper engages with three lines of prior work, identifying specific gaps that sleep-time compute addresses:

Test-time scaling (sequential and parallel). The dominant approaches to improving LLM performance through additional inference compute are sequential scaling (extended chain-of-thought, as in o1 and R1, where the model generates longer reasoning traces before answering) and parallel scaling (best-of-N or pass@k sampling, where multiple independent solutions are generated and a verifier or majority vote selects the best one). The paper acknowledges the impressive accuracy gains from these methods—they are the techniques sleep-time compute builds upon—but identifies two structural weaknesses:

All computation happens after the query arrives. In both sequential and parallel scaling, every token of reasoning is generated while the user is waiting. The paper notes (Section 2): "sequential test-time scaling has demonstrated impressive performance improvements, parallel test-time scaling has the advantage of scaling test-time compute without increasing latency." But even parallel scaling, which avoids serial latency because all samples can be generated simultaneously, still requires the user to wait for the slowest sample to finish before a final answer is selected. Neither approach exploits the fact that in stateful settings, the context is already available and could have been pre-processed.
No mechanism for sharing computation across related queries. When multiple users ask different questions about the same document, or when a single user asks follow-up questions about the same codebase, standard test-time compute treats each query independently. The paper describes this as the model carrying out "independent reasoning processes for each $q_i$ , even if they are related to the same context $c$ ." This is the core inefficiency that amortization (Section 5.3) targets.

Speculative decoding. The paper draws a comparison to speculative decoding (Leviathan et al., 2023; Stern et al., 2018), which reduces latency by using a smaller draft model to generate candidate tokens that a larger target model verifies in parallel. The key insight shared with sleep-time compute is speculation—performing work ahead of time based on predictions about what will be needed. However, the paper identifies a critical difference (Section 2): "unlike speculative decoding, the generated tokens are used as an input regardless of the user's actual query." In speculative decoding, if the draft model's predictions don't match the target model's actual output, the draft tokens are discarded and the computation is wasted. In sleep-time compute, the pre-computed inferences about the context $c'$ are retained and provided to the model as additional input even if the user's specific question wasn't perfectly anticipated. The paper argues that reasoning about a context—deriving intermediate quantities, identifying structural patterns, summarizing key facts—is useful for a wide range of possible queries about that context, not just a single predicted query. This makes sleep-time compute more robust to mis-prediction than speculative decoding is to mis-speculation.

Pre-computation and caching in traditional systems. The paper situates itself in a lineage of pre-computation strategies that extends beyond LLMs, citing memory caches (Smith, 1982), data cubes for OLAP workloads (Gray et al., 1997), and the concept of pre-fetching in operating systems. The common thread is a trade-off between pre-computation overhead and query latency—invest resources upfront to reduce cost at query time. The paper's extension to LLMs builds specifically on Packer et al. (2023)'s work on MemGPT, which introduced the idea of LLMs as operating systems with persistent memory management. However, that work focused on memory retrieval and context management, not on pre-computing inferences about the stored context. Sleep-time compute adds the active reasoning component: the model does not just store context more efficiently but actively transforms it into a form optimized for downstream query answering.

How This Paper Positions Itself

The paper frames sleep-time compute not as a replacement for test-time scaling but as an orthogonal dimension along which inference compute can be allocated. The standard test-time compute paradigm has exactly one degree of freedom: how much to spend after the query arrives. Sleep-time compute introduces a second: how much to spend before the query arrives, on the context alone. The paper's characterization in Section 3 formalizes this:

Standard test-time compute: $T_B(q, c) \rightarrow a$ — a method $T$ with budget $B$ maps query $q$ and context $c$ to an answer $a$ .
Sleep-time compute: $S(c) \rightarrow c'$ followed by $T_b(q, c') \rightarrow a$ — a sleep-time transformation $S$ pre-processes the context into $c'$ , and then a much smaller test-time budget $b \ll B$ suffices to answer the query, because the heavy reasoning about the context has already been done.

The paper is careful not to claim that this always improves performance. It frames sleep-time compute as an efficiency mechanism that shifts the pareto frontier of test-time compute versus accuracy: for a given accuracy target, less test-time compute is needed; for a given test-time budget, higher accuracy is achievable. This is a nuanced claim—sleep-time compute is not uniformly better but allows better points on the accuracy-efficiency tradeoff curve, which the paper empirically demonstrates in Figures 3, 4, and 11.

A key positioning move is the explicit framing of sleep-time compute as "representation learning over tokens" (Section 7). The paper draws an analogy to traditional representation learning (Bengio et al., 2014), where raw data is transformed into features more amenable to downstream tasks. However, where traditional representation learning operates in parameter or activation space (training embeddings, fine-tuning weights), sleep-time compute forms representations in the space of natural language tokens. The "representation" $c'$ is a string of text—reorganized, enriched with inferences, anticipating likely questions—that the model can read and reason over more efficiently than the raw context $c$ . This connects sleep-time compute to recent work on using LLMs to implement statistical modeling techniques in natural language (Zhong et al., 2022, 2025) and positions the paper as contributing a new application of that paradigm: pre-computed natural language representations for efficient inference.

The paper also positions sleep-time compute as distinct from simply predicting the user's question and answering it early. The context-only baseline in Appendix I directly tests this: the model is given only $c$ and must "guess the most likely question and output the answer." On both Stateful GSM-Symbolic (Figure 21) and Stateful AIME (Figure 22), sleep-time compute significantly outperforms the context-only baseline, demonstrating that the questions in the datasets "are not trivially predictable from the context." Sleep-time compute's $c'$ is not a pre-answered question; it is enriched context that makes answering any question about the context faster, without committing to a specific predicted query.

Conceptual Innovation: Amortization Across Queries

Beyond the offline/online shift, the paper introduces amortization as a second key benefit of sleep-time compute. Section 3 formalizes this: since $c'$ can be "shared across different queries $q_i$ about the same context," the cost of computing $c'$ is divided among all queries that use it. In the standard test-time paradigm, each query pays the full cost of reasoning about $c$ . In the sleep-time paradigm, the sleep-time cost is paid once (or can be incrementally refined) and the test-time cost per query is dramatically reduced. Figure 9 demonstrates this empirically: for a single query per context, the cost of sleep-time compute can outweigh its benefits; but as queries per context increase (2, 5, 10), the amortization effect kicks in and the average cost per query drops, reaching a 2.5× reduction at 10 queries per context.

This insight reframes the deployment decision. If a context will only be queried once, the efficiency case for sleep-time compute requires that the sleep-time pre-computation be sufficiently cheaper than test-time computation (or that latency constraints make test-time compute infeasible). If a context will be queried many times—as in a shared codebase, a frequently referenced document, or a persistent conversation—sleep-time compute becomes increasingly attractive regardless of the relative cost of sleep-time versus test-time tokens.

Summary of Gaps Addressed

The paper identifies and addresses a specific set of interlocking problems:

The stateless assumption in test-time compute research ignores that real deployments have persistent context available before queries arrive, leaving a window of idle model time unexploited.
The latency-cost barrier of test-time scaling makes it impractical for interactive applications, creating a need for mechanisms that preserve accuracy gains while reducing runtime overhead.
The redundancy problem in multi-query settings—where the same context-level reasoning is paid for with each query—is unaddressed by current inference strategies.
The parallel between model inference and system design (pre-computation, caching, pre-fetching) had not been systematically explored for LLM reasoning, despite the growing importance of inference efficiency as models are deployed in stateful, interactive contexts.

3. Technical Approach

3.1 Reader Orientation

The paper proposes a two-phase inference architecture for LLMs: during an offline "sleep-time" phase, when only the background context (e.g., a codebase, a document, a conversation history) is available but no user query has yet arrived, the model actively pre-computes useful inferences about that context and rewrites it into an enriched, re-represented form $c'$ ; then at test-time, when the user's query does arrive, the model uses this pre-processed context to answer with far less computation than standard test-time scaling would require. The system solves the problem that test-time compute is expensive and high-latency because all reasoning about the context happens after the query arrives, by shifting the heavy context-level reasoning to an offline phase when the model would otherwise be idle, producing a new context that can be reused across multiple queries about the same context to amortize costs further.

3.2 Big-Picture Architecture (Diagram in Words)

The architecture has four major components connected in a two-phase pipeline:

Raw Context $c$ — the persisted, pre-existing background information available before any user query arrives. This could be a math problem statement minus the final question, a code repository, a document corpus, or conversation history. It is the input to the sleep-time phase.
Sleep-time Processor $S$ — a function that takes only the raw context $c$ (no query) and outputs a re-represented context $c'$ . Implemented by prompting the model to "re-organize and consolidate memories" by calling a rethink_memory function iteratively, drawing inferences, anticipating possible queries, and computing intermediate results. This phase runs offline between user interactions and can be scaled by running parallel generation chains or by increasing reasoning effort (for reasoning models like o1/o3-mini).
Re-Represented Context $c'$ — the output of the sleep-time phase, stored and shared across all subsequent queries about the same $c$ . It is a natural-language string containing intermediate inferences, restructured information, and anticipated question-answer patterns that make answering actual queries faster.
Test-time Answerer $T_b$ — a function that takes the user's query $q$ and the enriched context $c'$ and produces an answer $a$ using a small test-time budget $b \ll B$ , where $B$ is what would have been needed with the raw context. The model is instructed to use the "rethink memory block" (the $c'$ ) to answer directly without recomputing anything already present. For reasoning models, $b$ is controlled by varying the reasoning effort parameter or budget-forcing prompt.

Information flows: sleep-time phase → $c \rightarrow S(c) \rightarrow c'$ ; test-time phase (triggered by user query) → $(q, c') \rightarrow T_b(q, c') \rightarrow a$ . When multiple queries $q_1, q_2, ..., q_N$ arrive about the same $c$ , the same $c'$ is reused for each, splitting the sleep-time cost across $N$ queries.

3.3 Roadmap for the Deep Dive

First, the formal problem setting and notation (Section 3 of the paper), which defines the two operations $S(c)$ and $T_b(q, c')$ and clarifies the relationship to standard test-time compute. This notation is essential for understanding what the system actually does and what the efficiency claims mean.
Second, the sleep-time computation mechanism — how $S(c)$ is implemented via iterative function calling (rethink_memory), what prompt engineers the model to do during sleep-time, and how the budget for sleep-time compute is controlled (parallel generations for non-reasoning models, reasoning effort for reasoning models). This is the core technical contribution.
Third, the test-time computation mechanism — how $T_b(q, c')$ is implemented, particularly the prompts that instruct the model to use the pre-computed context without recomputing. This explains how varying $b$ creates the test-time compute vs. accuracy curves.
Fourth, the cost model and amortization framework — how the paper models the cost of tokens at sleep-time versus test-time, the linear cost model with weight factor $t = 10$ , and how this leads to the amortization analysis across multiple queries. This is what makes the efficiency argument quantitative rather than just conceptual.
Fifth, the construction of the evaluation datasets — the procedures for splitting GSM-Symbolic and AIME into context-query pairs, and generating the Multi-Query GSM-Symbolic dataset with synthetic additional questions. Understanding these constructions is necessary to interpret the experimental results.
Sixth, the baseline configurations and test-time scaling mechanisms — how the paper varies test-time compute (verbosity prompts for non-reasoning models, reasoning effort and budget forcing for reasoning models, pass@k for parallel scaling) to produce the pareto curves against which sleep-time compute is compared.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical systems paper whose core idea is that LLM inference in stateful settings can be decomposed into an offline context-preprocessing phase and an online query-answering phase, and that doing so shifts the pareto frontier of test-time compute versus accuracy by exploiting idle model time and enabling computation reuse across queries.

Formal Problem Setting and Notation

The paper defines two phases of computation with explicit functional notation in Section 3.

Standard test-time compute. In the conventional paradigm that the paper critiques, the user provides a query $q$ together with context $c$ at the same moment, and the model applies a test-time scaling method $T$ with total budget $B$ to produce a reasoning trace followed by a final answer $a$ :

$T_B(q, c) \rightarrow a$

where $T_B$ is any method for scaling test-time compute with budget $B$ (which could be measured in tokens, reasoning steps, or number of parallel samples), $q$ is the user's specific question, $c$ is the background context available for answering the question, and $a$ is the final answer produced by the model after spending the full budget.

What this notation means operationally: when a user asks "How many marked indigo tennis balls are there?" and provides a paragraph describing the juggler's balls, the model receives both simultaneously, reasons through the full problem from the raw text, and outputs an answer, spending $B$ tokens of reasoning in the process. If the user then asks a second question about the same context (e.g., "How many tennis balls are there?"), the model repeats the entire reasoning process from scratch, spending another $B$ tokens, re-deriving the same intermediate quantities (e.g., total tennis balls = 200) that were already computed in the first query. The paper's central critique is that this wastes computation by treating each query as stateless and independent.

Sleep-time compute. The paper introduces a two-phase alternative. First, during sleep-time (when the model has access to $c$ but not yet to $q$ ), a sleep-time processor $S$ transforms the raw context into an enriched form $c'$ :

$S(c) \rightarrow c'$

where $S$ is any standard test-time scaling technique applied toward pre-processing the context at sleep-time, $c$ is the raw context available before the query arrives, and $c'$ is the re-represented context containing inferences, anticipated question structures, and pre-computed intermediate results (all expressed in natural language).

Then at test-time, the model answers the query using the enriched context with a much smaller budget $b$ :

$T_b(q, c') \rightarrow a$

where $b \ll B$ is a significantly reduced test-time compute budget, possible because the heavy reasoning about $c$ has already been performed during sleep-time and is directly available in $c'$ as pre-computed text that the model can reference rather than recompute.

Why this decomposition works: the premise is that a substantial fraction of the reasoning tokens in standard test-time compute are spent on understanding and processing the context ( $c$ ) rather than on the query-specific reasoning ( $q$ ). For example, in the juggler problem, a large portion of the reasoning — computing that there are 200 tennis balls, 100 indigo tennis balls, and 10 marked indigo tennis balls — depends only on the context (the initial paragraph) and not on which specific question is asked (total marked balls? total tennis balls?). Sleep-time compute pre-computes these context-dependent quantities, encodes them in $c'$ , and presents them to the model at test-time so that the test-time reasoning only needs to handle the query-specific portion (e.g., selecting which pre-computed quantity answers the specific question asked).

The multi-query extension. When there are multiple queries $q_1, q_2, ..., q_N$ about the same context $c$ , the same $c'$ is reused for all of them. The sleep-time cost is paid once, and each test-time query pays only the small per-query cost $b$ . The paper formalizes the cost model using a linear weighting factor: tokens generated at test-time cost $t$ times as much as tokens generated at sleep-time, reflecting that latency-optimized inference at test-time can be "roughly 10× more expensive" (Section 5.3). The total amortized cost per query for $N$ queries is:

$\text{Cost per query} = \frac{\text{Cost}(S(c)) + t \cdot \sum_{i=1}^N \text{Cost}(T_b(q_i, c'))}{N}$

where $\text{Cost}(S(c))$ is the token cost of the sleep-time preprocessing (charged at the cheaper sleep-time rate), $t = 10$ is the cost multiplier for test-time tokens (reflecting higher latency-optimized pricing), and $\text{Cost}(T_b(q_i, c'))$ is the test-time token cost per query. As $N$ grows, the amortized sleep-time cost per query approaches zero, leaving only the (small) per-query test-time cost.

Why a linear cost model with $t = 10$ : the paper justifies this choice by citing that latency-optimized inference can be roughly 10× more expensive (footnote referencing Databricks documentation). The linear model is a simplification — it doesn't account for non-linear utility functions where users might place higher value on faster responses — but it provides a tractable framework for analyzing the tradeoff. The paper explicitly notes that the analysis "can be generalized to different cost functions that consider non-linear user-utility," flagging the linear model as a starting point rather than a claim of universal applicability.

The Sleep-Time Computation Mechanism

The sleep-time phase is the paper's primary technical contribution. It transforms the raw context $c$ into an enriched form $c'$ through a structured process of iterative inference and memory consolidation, implemented via function calling.

Implementation via function calling (Section 3 and Appendix K). The sleep-time processor $S(c)$ is realized by giving the model access to two functions:

rethink_memory(new_memory: str, source_block_label: str, target_block_label: str) -> None: This function takes a new memory string and replaces the current contents of the target memory block with this string, optionally integrating information from a source block. It does not return anything — it is a side-effect operation that updates the model's persistent memory state. The model calls this function iteratively, each time refining or expanding its understanding of the context.
finish_rethinking_memory() -> None: This function terminates the sleep-time computation process. The model calls it when it has finished drawing inferences and re-organizing the context. It returns nothing and signals that $c'$ is complete and ready for test-time use.

The sleep-time prompt structure (Appendix A, Figures 17-18). The model is given a system prompt that establishes its role as "Letta-Offline-Memory" — a digital companion whose task is to "re-organize and consolidate memories by calling rethink memory at every single step." The prompt provides detailed instructions about the structure of the memory system:

Read-Only Blocks: These contain the core persona information and essential user details, providing foundational context that the model can reference but not modify. They include a "persona" sub-block (guiding how the model behaves) and a "human" sub-block (storing key details about the user).
Read-Write Blocks: The "Rethink Memory Sub-Block" is the target for $c'$ . The model is instructed to "take current information in the rethink memory block and select a single memory block to integrate information from, producing a new memory for the rethink memory block."

The prompt instructs the model to draw "all logical conclusions and potential hypotheses possible" from the given context. If uncertain, the model should "use internal monologue to consider what the possible conclusions are, and then state the most likely new facts that would replace the old facts."

Domain-specific sleep-time prompts. For mathematical reasoning tasks (Stateful AIME), the paper provides an additional domain-specific instruction (Appendix A, Figure 18):

"You will be given part of an AIME math problem. You will receive the rest of the problem later. Make as many inferences as possible about the part of the problem you are given so as to help yourself answer the fully problem more quickly once it is given to you later."

This prompt explicitly frames the task as speculative pre-computation: the model should "try to predict possible ways the rest of the problem might go and compute results that could be helpful for reaching the final answer more quickly once the rest of the problem is given." The important design choice here is that the model is not asked to predict the exact question and answer it — that would be the context-only baseline tested in Appendix I. Instead, it is asked to compute intermediate quantities, explore possible continuations, and structure the context in ways that would be helpful regardless of which specific question follows.

Budget control for sleep-time compute. The paper describes two mechanisms for scaling the amount of sleep-time compute, depending on the model type:

For non-reasoning models (GPT-4o, GPT-4o-mini): Sleep-time compute is scaled by running $k$ parallel generations, each independently producing a version of $c'$ , resulting in $c'_1, c'_2, ..., c'_k$ . At test-time, the model receives all $k$ versions concatenated together as the enriched context. The paper sweeps $k \in \{1, 2, 5, 10\}$ and reports results in Figure 7. This approach scales sleep-time compute without increasing serial latency (all $k$ generations run in parallel), analogous to how parallel test-time scaling (pass@k) works but applied at sleep-time rather than test-time.
For reasoning models (o1, o3-mini): Sleep-time compute is scaled by varying the reasoning effort parameter exposed by the API. The paper sweeps three levels — "low reasoning effort," "medium reasoning effort," and "high reasoning effort" — and reports results in Figure 8. This controls the amount of internal chain-of-thought the model generates during the sleep-time rethink_memory calls, effectively increasing the depth and breadth of the pre-computed inferences.

The model is limited to 10 rethink_memory calls (Appendix K): "The model is allowed to call the function rethink memory for up to 10 times." This cap prevents unbounded sleep-time computation and represents a design choice about the maximum number of refinement iterations the model can perform on the context.

Why function calling rather than simple prompting: the iterative rethink_memory mechanism with separate read-only and read-write blocks enforces a structured memory consolidation process rather than a single-pass re-write. Each call can focus on integrating information from a specific source block (the "persona" block, the "human" block, or the previous "rethink memory" block), producing incremental refinements. This is intended to prevent the model from producing a superficial re-write and instead force deliberate, step-by-step inference — a form of chain-of-thought but directed at restructuring memory rather than answering a question. The finish_rethinking_memory function serves as an explicit termination condition, preventing the model from continuing indefinitely.

The Test-Time Computation Mechanism

After sleep-time has produced $c'$ , the test-time phase answers user queries using this enriched context with a dramatically reduced compute budget.

Test-time prompt structure (Appendix A, Figures 12-16). The paper constructs a set of five test-time prompts — labeled "Verbosity 0" through "Verbosity 4" (though the paper uses the terms "level 0" through "level 4" in the figure captions) — that control how much computation the model performs at test-time. These prompts share a common structure but vary the instructions about reasoning depth:

All prompts instruct the model to use the pre-computed context: Every prompt contains the key instruction: "You check the 'rethink memory block' for potential questions and answers and intermediate reasoning traces that can help answer the question. You use the information in the rethink memory block to answer the questions rather than thinking on the spot. Do not recompute anything that already exists in the rethink memory block." This is the mechanism by which sleep-time compute reduces test-time computation — the model is explicitly told to reference pre-computed results rather than deriving them anew.
Verbosity 0 (Figure 12): "You respond directly with a single sentence by saying 'The answer is ' followed by the numerical answer." This is the most aggressive test-time compute reduction, instructing the model to produce essentially no reasoning trace and output only the final answer.
Verbosity 1 (Figure 13): "You answer with one short sentence of explanation, followed by a sentence that starts with 'The answer is' and a numerical answer." Allows a single explanatory sentence before the answer.
Verbosity 2 (Figure 14): "You end response with a final numerical answer at the end of the message, and no reasoning after that." The prompt does not explicitly limit the amount of reasoning before the final answer, but instructs the model to answer "using only the number of tokens necessary and none more," creating a soft constraint on verbosity.
Verbosity 3 (Figure 15): Identical text to Verbosity 2 in the provided figures — the paper appears to have a distinction between these levels that is not captured in the appendix text. Based on the pattern, Verbosity 3 may represent a moderate level of allowed reasoning.
Verbosity 4 (Figure 16): "You always reason out loud before using any information. You explain each step, of what your reasoning is. If you use any numbers from the rethink memory block you first recompute and double check your answers." This is the highest test-time compute setting, instructing the model to verify the pre-computed information rather than blindly trusting it, producing a substantial reasoning trace.

Why five verbosity levels: the paper needs to produce a continuous pareto curve of test-time compute versus accuracy to demonstrate the shift from sleep-time compute. By varying the prompt from "output only the answer" to "reason step-by-step and verify everything," the paper can sweep a range of test-time token budgets and measure accuracy at each point. The same set of prompts is used for both the sleep-time compute condition and the standard test-time compute baseline, ensuring a fair comparison — the only difference between conditions is whether the model receives the raw context $c$ or the enriched context $c'$ .

For reasoning models (o1, o3-mini, Claude 3.7 Sonnet, DeepSeek-R1): The test-time compute budget is varied differently because these models have built-in reasoning mechanisms that are controlled by API parameters or prompting techniques:

For o1 and o3-mini: The paper scales test-time compute "based on what is available in the API," meaning it uses the reasoning effort parameter (low/medium/high) to control the amount of internal chain-of-thought tokens the model generates before answering.
For Claude 3.7 Sonnet: The paper similarly uses the extended thinking mechanism exposed through the API to control test-time compute.
For DeepSeek-R1: Since the API "does not provide a way to control test-time compute," the paper applies the "budget forcing" and extension prompt technique from Muennighoff et al. (2025). Budget forcing is a technique where the model's reasoning trace is truncated at a specified token limit, and an extension prompt (e.g., "wait" or "continue") is appended to force the model to keep reasoning, effectively controlling the total reasoning length.

Temperature and sampling: For non-reasoning models on Stateful GSM-Symbolic, the paper uses "temperature 0 for generation," meaning deterministic greedy decoding. For reasoning models, the paper averages results over 3 runs for o1, o3-mini, and DeepSeek-R1, and over 10 runs for Claude 3.7 Sonnet "as we observed more noise in initial experiments." This increased averaging for Claude suggests that the extended thinking mechanism introduces more variance in outputs, possibly due to non-deterministic internal reasoning paths.

The Cost Model and Amortization Framework

The paper's quantitative efficiency claims depend on a cost model that distinguishes between sleep-time and test-time tokens and accounts for query multiplicity.

Token cost asymmetry. The paper asserts that tokens generated at test-time are more expensive than tokens generated at sleep-time because test-time inference must meet stricter latency requirements. The specific cost model (Section 5.3) is:

$\text{Total Cost} = \text{SleepTokens} + t \cdot \text{TestTokens}$

where $\text{SleepTokens}$ is the total number of tokens generated during the sleep-time phase (all rethink_memory calls), $\text{TestTokens}$ is the total number of tokens generated at test-time (the model's answer and any reasoning trace), and $t$ is the cost multiplier for test-time tokens. The paper sets $t = 10$ based on the observation that "latency optimized inference can be roughly 10× more expensive" (Section 5.3, citing Databricks documentation).

Why $t = 10$ : This figure reflects the reality that low-latency inference — where the model must respond within milliseconds to maintain interactivity — often requires provisioning dedicated high-throughput hardware (e.g., more GPUs to parallelize attention computation) or using speculative decoding, both of which increase cost per token relative to batch inference where latency is not constrained. During sleep-time, there is no user waiting, so the model can use cheaper batch inference. The paper does not measure $t$ empirically — it is an assumed parameter — and explicitly notes that the analysis "can be generalized to different cost functions that consider non-linear user-utility."

Amortization across multiple queries. When $N$ queries share the same context, the sleep-time cost is paid once and divided across all queries. The average cost per query is:

$\text{Average Cost Per Query} = \frac{\text{SleepTokens} + t \cdot \sum_{i=1}^N \text{TestTokens}_i}{N}$

This model produces the curves in Figure 9, where the x-axis is "Total Inference Cost / Query" and different curves correspond to different numbers of queries per context (1, 2, 5, 10). The key observable: when $N = 1$ , the sleep-time compute curves sit to the right of the test-time-only baseline (higher cost for similar accuracy) because the sleep-time token overhead is not amortized. As $N$ increases, the sleep-time compute curves shift leftward (lower cost per query), eventually crossing and then significantly improving over the baseline. At $N = 10$ , the paper reports "decrease the average cost per query by up to 2.5×" compared to the single-query baseline.

What "2.5× reduction" means operationally: If the test-time-only baseline achieves a certain accuracy at a cost of 300 total inference cost units per query, the sleep-time compute approach with 10 queries per context achieves the same accuracy at approximately 120 total inference cost units per query. The actual numbers depend on the specific accuracy target, but the 2.5× figure represents the maximum reduction observed in the swept cost-accuracy curves.

Construction of Stateful Evaluation Datasets

The paper creates two categories of evaluation datasets — Stateful GSM-Symbolic and Stateful AIME — by splitting existing reasoning benchmarks into context-query pairs, and additionally creates Multi-Query GSM-Symbolic by generating synthetic additional questions for existing contexts.

Stateful GSM-Symbolic construction (Section 4.1). The paper derives this dataset from the GSM-Symbolic dataset (Mirzadeh et al., 2024), which itself extends GSM8K (Cobbe et al., 2021) by adding clauses to increase difficulty. The original GSM-Symbolic problems are presented as a single paragraph containing all information followed by a question. The paper splits each problem into a context (all statements except the final question) and a query (the final question). The example in Figure 2 illustrates this: the original problem "A juggler can juggle 800 balls. 1/4 of the balls are tennis balls, and 1/2 of the tennis balls are indigo of which 1/10 are marked. How many marked indigo tennis balls are there?" becomes:

Context: "A juggler can juggle 800 balls. 1/4 of the balls are tennis balls, and 1/2 of the tennis balls are indigo of which 1/10 are marked."
Query: "How many marked indigo tennis balls are there?"

The dataset uses two difficulty splits: P1, which adds one clause to the original GSM8K problems (5,000 examples), and P2, which adds two clauses (2,500 examples). The additional clauses insert distractor information or intermediate steps, increasing the complexity of the context. For example, a P2 problem might include additional quantities that need to be tracked and computed, making the context richer and the pre-computation of intermediate results more valuable.

Stateful AIME construction (Section 4.1 and Appendix J). The derivation from AIME 2024 and 2025 (60 total questions) follows a more nuanced procedure because AIME problems have varied structures:

Statement-level splitting: Each AIME problem is broken into "punctuation separated sentences in the problem." The paper then uses "all but the last statement as the context, and the final statement as the query."
Manual rearrangement for edge cases: In some problems, "the question is posed in e.g. the second to last statement rather than the last statement." In these cases, the authors "manually rearrange the statements to ensure the query being used corresponds to the question." This manual intervention ensures the query actually contains the problem's question rather than an intermediate statement that happens to be last.
Single-statement problems: For the few cases where "there is only one statement in the problem," the context is empty. These represent a degenerate case where sleep-time compute has no context to process and would be equivalent to standard test-time compute.
LaTeX figure handling: AIME includes LaTeX representations of geometric figures, but these "can leak information about the answer: for example, these latex figures can contain exact information about the lengths of the sides in a geometry problem, giving away the answer." The paper "first ensure[s] that the problem is solvable without the figure and then manually strip[s] the figure latex from the problem context." This is an important data quality decision: if figures were left in the context, the model could potentially extract answer-critical information from them during sleep-time, inflating the apparent benefit of sleep-time compute by giving the model access to information that would ordinarily require geometric reasoning to derive.

Multi-Query GSM-Symbolic construction (Section 4.1 and Appendix C). To study amortization across queries, the paper generates additional question-answer pairs for each context in Stateful GSM-Symbolic:

Selection of base examples: The paper samples "one instance from each template from the GSM-Symbolic dataset" — GSM-Symbolic is generated from templates with variable substitutions, so sampling one instance per template ensures diversity across the underlying problem structures while keeping the dataset size manageable.
Synthetic question generation: Using o3-mini, the paper generates additional questions from each context-question pair. The generation prompt (Appendix C, Figure 19) instructs the model: "Your task is to generate a list of questions and answers about the context at the same difficult level that could plausibly be asked about that context. Make sure that the newly generated questions have the same number of reasoning steps required as the example question."
Format constraint: Generated questions must "have the same format as the example, where the answer first contains reasoning and then is the final answer comes after \n####." This enforces consistency with the original GSM-Symbolic format.
Dataset statistics (Table 1 in Appendix C): For P1, the dataset contains 1,095 contexts with a total of 12,043 questions (1,095 original + 10,948 generated, averaging ~10 generated questions per original question). For P2, there are 500 contexts with 5,497 total questions (500 original + 4,997 generated, also averaging ~10 per context). The paper reports using up to 10 questions per context in the amortization experiments (Figure 9).

Quality of generated questions: The paper includes an example in Figure 20 (Appendix C) showing a context about Sofia's toys and the original question "How many bouncy balls came in the tube?" The generated questions include both direct queries about quantities explicitly computable from the context ("How many action figures does the pack contain?", "What is the total number of stickers in the sticker book?") and more complex derived questions ("If Sofia had received a tube with 10 extra bouncy balls, what would be the new total number of items?"). This demonstrates that the generation process produces questions at varying levels of difficulty and requiring different subsets of the pre-computable information in the context.

Baseline Configurations and Test-Time Scaling Mechanisms

To establish the pareto frontier against which sleep-time compute is compared, the paper needs methods for varying test-time compute in both the sleep-time and standard conditions.

Standard test-time compute baseline. For the standard test-time compute condition, the model receives the raw context $c$ and query $q$ simultaneously at test-time, with no pre-processing. The test-time budget is varied using the same verbosity prompts described above (Verbosity 0–4 for non-reasoning models; reasoning effort/budget forcing for reasoning models). This produces the "test-time compute only" curves in Figures 3 and 4 — for each prompt level, the paper measures the average test-time tokens per question and the accuracy, plotting one against the other.

Parallel test-time scaling baseline (pass@k). In addition to the sequential scaling baseline, the paper compares against parallel test-time compute in Figures 5 and 6. The pass@k metric measures whether the correct answer appears in any of $k$ independently sampled solutions. The paper notes that pass@k "makes the unrealistic assumption of having oracle query access to a ground truth verifier at test-time, an assumption which we do not make with sleep-time compute." This makes pass@k a strong baseline — it represents an upper bound on what parallel sampling could achieve with a perfect verifier — and outperforming it would be a meaningful result.

The paper applies parallel scaling "to the lowest sequential compute setting on each task, since scaling pass@k with higher sequential compute settings would quickly reach token budgets that exceed that of sleep-time compute in the maximum sequential setting." In other words, at each accuracy level, the paper ensures the token budgets are comparable between sleep-time compute and pass@k, preventing situations where pass@k uses far more tokens at test-time and thus makes the comparison unfair.

Context-only baseline (Appendix I). To verify that the questions are not trivially predictable from the context (which would make sleep-time compute essentially equivalent to guessing the question and answering it early), the paper includes a baseline where the model receives only the context $c$ and is "tasked with directly guessing an answer to the question it guesses is most likely to come next." This is compared against sleep-time compute in Figures 21 and 22. Sleep-time compute significantly outperforms this context-only baseline — on Stateful GSM-Symbolic P1, accuracy with sleep-time compute is approximately 0.4–0.8 depending on test-time budget, while the context-only baseline would be substantially lower (the exact numbers are not quoted, but the visual gap in Figure 21 is large). This demonstrates that $c'$ is not simply a pre-answered predicted question but genuinely enriches the context in a way that helps with the actual (unanticipated) query.

Why these specific baselines: The paper chooses baselines that represent different points on the test-time scaling spectrum — sequential scaling at varied verbosity (the primary baseline for latency-sensitive applications), parallel scaling with oracle verification (the strongest possible parallel baseline), and context-only prediction (to validate that the task is not trivial). Together, they establish that sleep-time compute's benefits are not attributable to simply predicting the question, to using more total tokens, or to an unfair comparison against a weak baseline.

Sleep-Time Compute Scaling Configurations

The paper sweeps two orthogonal dimensions — sleep-time compute and test-time compute — to produce the pareto shift analyses.

Sleep-time scaling for non-reasoning models (Section 5.2). The paper varies the number of parallel sleep-time generations $k \in \{1, 2, 5, 10\}$ . The procedure:

During sleep-time, $k$ independent calls to the LLM are made, each given the same context $c$ and the sleep-time prompt (Figure 17). Each call produces its own version of $c'$ through the rethink_memory / finish_rethinking_memory loop.
At test-time, the $k$ versions are concatenated — "the model then receives the inputs concatenated $c'_1, \dots, c'_k$ to generate the final answer." The test-time prompt (one of the five verbosity levels) is appended to this concatenated context.
The total sleep-time compute cost scales linearly with $k$ (each generation produces roughly the same number of tokens), while test-time cost is measured separately and reported on the x-axis of Figures 7 and 26.

Sleep-time scaling for reasoning models (Section 5.2). For o1 and o3-mini, the paper varies the sleep-time reasoning effort by "varying the reasoning effort for o1 and for o3-mini when applying the sleep-time compute prompt." The three levels — low, medium, high reasoning effort — are API-exposed parameters that control how many internal chain-of-thought tokens the model generates during each rethink_memory call. The test-time reasoning effort is also varied to produce the two-dimensional sweep shown in Figure 8.

Key observation about diminishing returns: The paper finds that for non-reasoning models on Stateful GSM-Symbolic, "5 parallel generations generally outperforms 10" (Section 5.2). This suggests that beyond a certain point, additional sleep-time generations produce redundant or conflicting inferences that do not further improve test-time accuracy. The paper does not deeply analyze why this saturation occurs, but possible explanations include: (a) the concatenated context becomes too long, exceeding the model's effective context utilization; (b) later parallel generations produce inferences that are already captured by earlier ones; or (c) the test-time model struggles to integrate information from too many parallel pre-computation chains.

SWE-Features: A Case Study in Agentic Software Engineering

The paper includes a case study (Section 6) applying sleep-time compute to a realistic multi-turn agentic task, which requires a different implementation than the math benchmarks.

SWE-Features benchmark construction (Appendix D). The benchmark collects pull requests (PRs) from large open-source repositories (specifically Aider-AI/aider and comfyanonymous/ComfyUI) with the following filtering process:

File count filter: "We identify all pull requests that modify at least three files with filenames ending in .py or .js." This ensures the tasks are non-trivial — modifying multiple files requires understanding cross-file dependencies and repository structure, making pre-computation of the repository architecture valuable.
Content filter using GPT-4o-mini: The PRs are filtered "based on their title and body, retaining only those that meet the following criteria: (a) the title and body clearly describe the PR; (b) the PR introduces new functionality rather than fixing bugs; and (c) the PR is independent and not obviously linked to other issues." This filtering ensures the benchmark consists of well-specified, self-contained feature additions — tasks where the required changes can be inferred from the PR description alone, without external issue tracker context.
Resulting dataset: 33 examples total — 18 from Aider-AI/aider and 15 from comfyanonymous/ComfyUI.

Clustering for context selection. For each target PR that the agent needs to solve, the paper needs to provide a relevant context $c$ — other PRs that share structural or functional similarities. The paper uses Claude 3.7 Sonnet to "cluster pull requests (PRs) from the ComfyUI and Aider repositories into several groups," then provides other PRs from the same cluster as context. For example, "in the ComfyUI repository, PR #5293 and PR #931 are grouped into the same cluster. Thus, when processing PR #931, we organize the title, body, and changed files of PR #5293 to serve as contextual information during sleep-time."

The resulting clusters (Appendix D) group PRs by functional theme — e.g., "Dynamic Typing and Workflow Control" for ComfyUI, "Model Configuration" for Aider. This clustering provides a realistic proxy for how a deployed coding assistant might have access to historically completed PRs that are relevant to a new feature request.

Sleep-time implementation for SWE-Features. The sleep-time phase works differently here than in the math benchmarks:

Context provision: During sleep-time, the agent is given "the content of PR #5293" (the related PR) and is allowed to "explore the repository and summarize its understanding ahead of time." This exploration involves reading files, understanding the codebase structure, and identifying patterns from the related PR that might apply to the target PR.
Test-time provision: At test-time, the agent receives "the title and body of PR #931" (the target PR). In the sleep-time condition, the agent has already explored the repository and built a mental model of the relevant codebase areas; in the baseline condition, the agent sees the related PR content for the first time at test-time alongside the target PR.

Test-time budget control. The paper controls test-time budget in SWE-Features by fixing the total number of agent steps (API calls). The implementation uses the Letta framework's max_chaining_steps parameter. The system prompt includes:

"You have a strict budget of {max chaining steps} steps, which means you need to finish your edits within these steps. Every time you get queried, you will see a count of how many steps you have left in the form of '[Current Step / Max Steps]'. If you exceed this budget, your response will be cut off."

This step-based budget is analogous to the token-based budgets in the math experiments but adapted to an agentic setting where each step could involve reading files, writing code, or running tests. The paper notes that "explicitly indicating the current and total steps significantly improves agent performance, especially in low-budget settings."

Evaluation metric for SWE-Features. Unlike the math benchmarks where answers are graded for exact match, SWE-Features uses a file-level F1 score: "For each PR, we compare the set of files predicted to be modified with the ground truth list of modified files." Specifically, the evaluation considers files with status "modified" (not newly created files), and computes precision and recall over the set of files the agent edits versus the set of files actually modified in the ground-truth PR. The F1 score harmonizes these into a single metric. The paper notes that the agent is "still instructed to implement the required functionality in a Docker environment and write test functions to validate the implementations," but "after the agent makes the modifications, we extract the modified files and calculate the F1 score." This means functional correctness of the implementation is not directly evaluated — only whether the agent correctly identifies which files need to be changed.

Why F1 rather than functional correctness: The paper acknowledges the limitation that "there are not straightforward tests to use for evaluation" since the PRs are scraped from GitHub. A full functional evaluation would require reproducing the exact repository state at the time of each PR, running the modified code, and verifying that the feature works correctly — which is significantly more complex than file-set comparison. The F1 metric captures a meaningful dimension of task understanding (knowing which files need modification is a prerequisite for implementing the feature correctly) while being tractable to compute across all 33 examples.

4. Key Insights and Innovations

Innovation 1: Decomposing Inference into Context-Preprocessing and Query-Answering Phases

The paper's foundational conceptual move is not a new algorithm but a reframing of what inference-time computation means in stateful settings. Before this work, the field's implicit assumption was that reasoning about context and reasoning about the query are inseparable — that the model must process both simultaneously at the moment the user asks a question. Test-time scaling research (Snell et al., 2024; OpenAI, 2024; DeepSeek-AI, 2024) operated entirely within this assumption: the question is how much compute to spend after the prompt arrives, not when or how to sequence it relative to context availability.

Sleep-time compute challenges this by recognizing that context processing is temporally and functionally separable from query processing. The paper formalizes this through the decomposition $S(c) \rightarrow c'$ followed by $T_b(q, c') \rightarrow a$ , but the intellectual contribution is not the notation — it is the recognition that a significant fraction of test-time reasoning tokens (the authors do not quantify exactly what fraction, but the ~5× reduction in test-time tokens from Figure 3 provides an empirical bound) are spent on computations that depend only on the context and not on the specific query. In the standard paradigm, these context-dependent computations are re-executed for every query, even when multiple queries share the same context. In the sleep-time paradigm, they are executed once, offline, and the results are materialized as natural-language text that can be referenced cheaply at query time.

This reframing connects to a long lineage of systems thinking — pre-computation in databases (Gray et al., 1997), caching in operating systems (Smith, 1982), pre-fetching — but the paper adapts the principle to a domain where the "computation" is reasoning rather than data retrieval, and the "cache" is natural-language text rather than structured query results. The distinction is critical: reasoning is not idempotent in the way that data retrieval is. A pre-computed database view returns the same rows regardless of how the user phrases a query; a pre-computed reasoning trace must be interpretable by the model across a range of possible questions. The paper demonstrates that natural-language inferences — "there are 200 tennis balls," "there are 100 indigo tennis balls," "there are 10 marked indigo tennis balls" for the juggler problem — serve this role effectively, acting as a learned intermediate representation that compresses the context into query-agnostic facts.

Prior assumption challenged: That inference compute must be allocated entirely at query time, with the model starting from a blank reasoning state for each interaction. The paper shows this is not a fundamental constraint but an artifact of how benchmarks are structured (monolithic problem presentations) and how inference pipelines are designed (stateless per-request processing).

Significance assessment: This is a fundamental reframing, not an incremental improvement. It does not propose a better way to do chain-of-thought or a more efficient search algorithm — it changes when the computation happens. This has implications beyond the specific method: it suggests that future LLM system design should explicitly model the temporal structure of information availability (context available at T0, query at T1, follow-up at T2) and allocate compute budgets across these phases adaptively. The paper's discussion of optimal allocation between sleep-time and test-time compute (Section 7) points toward a broader framework where inference compute budgeting becomes a multi-phase optimization problem.

Evidence anchor: The ~5× reduction in test-time tokens reported in Section 5.1 (Figures 3 and 4) is the quantitative manifestation of this reframing. On Stateful GSM-Symbolic, GPT-4o-mini with sleep-time compute achieves at Verbosity 0 approximately the same accuracy as the standard baseline at Verbosity 2-3, representing a dramatic compression of test-time reasoning. On Stateful AIME (Figure 4), o3-mini with sleep-time compute achieves ~0.5 accuracy at ~1000 test-time tokens, while the standard baseline requires ~3000+ tokens to reach comparable accuracy. These are not small refinements but order-of-magnitude shifts in the test-time compute budget — precisely what you would expect if a substantial fraction of the baseline's reasoning was context-level processing that sleep-time compute offloads.

A natural consequence of the sleep-time decomposition is that the enriched context $c'$ can be shared across any number of queries about the same raw context $c$ . While this follows logically from the architecture, the paper elevates it from an implementation detail to a distinct conceptual contribution by formalizing the amortization analysis (Section 5.3) and demonstrating that the efficiency gains compound with query multiplicity — a property that standard test-time compute inherently lacks.

In standard test-time compute, the cost structure is linear in the number of queries: $N$ queries cost $N \times B$ reasoning tokens, with no sharing. Sleep-time compute introduces a fixed upfront cost (the sleep-time generation of $c'$ ) plus a small per-query cost $b$ , yielding a total cost that grows sublinearly as $N$ increases. The paper's key finding is not merely that amortization exists, but that it creates a threshold effect: at low query multiplicity ( $N = 1$ ), sleep-time compute can be worse than the baseline in terms of total cost (the sleep-time overhead is not recouped); at moderate multiplicity ( $N = 2-5$ ), it breaks even or offers modest gains; at high multiplicity ( $N = 10$ ), it provides up to 2.5× cost reduction per query (Figure 9). This threshold behavior has direct practical implications for deployment: sleep-time compute is not universally beneficial but is specifically advantageous when a context will be queried many times, which is exactly the scenario in shared codebases, frequently referenced documents, and persistent conversations.

Prior assumption challenged: That inference cost scales linearly with query volume, and that the only way to reduce per-query cost is to use a smaller model or reduce reasoning depth. The amortization result shows that architectural decisions about when to compute can change the scaling relationship entirely — from linear to sublinear — for contexts with high query multiplicity.

Significance assessment: The amortization insight is conceptually distinct from the phase decomposition (Innovation 1) because it depends on computation reuse rather than computation shifting. One could imagine a sleep-time system that regenerates $c'$ for each query (no reuse), which would capture the latency benefits but not the amortization benefits. Conversely, one could imagine a system that caches previous test-time reasoning traces and reuses them for related queries (computation reuse without sleep-time), which would be a different approach to the same problem. The paper's contribution is demonstrating that the sleep-time architecture naturally enables reuse as a consequence of its design, and that this reuse compounds the efficiency gains beyond what phase decomposition alone provides.

The connection to query predictability (Section 5.4) adds nuance: amortization is most valuable when queries are predictable from the context, because the pre-computed $c'$ is more likely to contain exactly the intermediate quantities and reasoning patterns that the actual queries will need. When queries are unpredictable, the fixed sleep-time cost may be paid for inferences that no query ever uses, reducing or negating the amortization benefit. This inverts the standard intuition: rather than predicting the exact question and answering it (the context-only baseline, which fails), the system predicts a distribution of useful inferences from the context, and the amortization gains materialize when the actual queries draw from that distribution.

Evidence anchor: Figure 9 shows the threshold behavior directly. On Stateful GSM-Symbolic P1, the sleep-time compute curve for $N = 1$ falls to the right of the test-time-only baseline (higher cost at comparable accuracy). As $N$ increases to 5 and then 10, the sleep-time curve shifts progressively leftward, crossing and then clearly dominating the baseline. The reported 2.5× cost reduction at $N = 10$ is the quantitative anchor. The predictability analysis (Figure 10) provides the mechanistic explanation: in the most predictable quintile, the accuracy gap between sleep-time and test-time-only is substantially larger than in the least predictable quintile (approximately 0.35–0.40 versus 0.05–0.15 on P2), confirming that amortization benefits scale with query predictability.

Innovation 3: Representation Learning in Token Space as an Inference-Time Strategy

The paper's most theoretically provocative framing — articulated in Section 7 as "sleep-time compute as representation learning over tokens" — is that the enriched context $c'$ functions as a learned representation that makes downstream query answering more efficient, but learned in the space of natural language tokens rather than in parameter or activation space as in traditional representation learning (Bengio et al., 2014). This reframing connects the empirical efficiency gains to a well-established theoretical framework while highlighting what is novel: the model is not learning to compress the context into a dense vector embedding (as an encoder would) or fine-tuning its weights to be better at answering questions (as supervised fine-tuning would). Instead, it is learning to rewrite the context in a form that a frozen model can process more efficiently — a representation that is human-readable, interpretable, and transferable across different queries and even different models (though the paper does not test cross-model transfer).

Why this is distinctive: Traditional representation learning operates on the assumption that the representation (embedding, activation, feature map) is what the downstream model consumes directly. The representation is typically not intended to be human-interpretable and is tightly coupled to the model architecture that produced it. Sleep-time compute's representation is text — it can be inspected, debugged, and potentially edited by humans or other models. It is model-agnostic in principle (any model that can read text can consume it), though the paper only tests within the same model family. This opens an unusual design space: the "representation learner" (the sleep-time processor $S$ ) can be a different model than the "downstream task solver" (the test-time answerer $T_b$ ), potentially enabling a powerful model to pre-compute representations that a weaker or faster model uses at test-time — a form of model distillation without weight transfer.

Prior assumption challenged: That making inference more efficient requires either compressing the model (quantization, pruning, distillation) or reducing the computation the model performs (shorter reasoning traces, smaller search budgets). The paper demonstrates a third path: keep the model and the total reasoning fixed, but reorganize when and in what form the reasoning occurs to align better with deployment constraints (latency at test-time, cost amortization across queries). This is a form of computational efficiency that does not sacrifice model capability or reasoning depth — it just restructures the timeline of computation to exploit idle periods and query commonalities.

Connection to prior work: The paper explicitly cites Zhong et al. (2022, 2025) as recent work that "implements statistical modeling techniques in the space of natural language using modern LLMs." Sleep-time compute can be viewed as applying this paradigm to the problem of efficient inference: the model performs a kind of "natural-language feature extraction" on the raw context, producing interpretable intermediate features that downstream question-answering can leverage. This is distinct from traditional feature extraction because the features are generated by the same type of model that will consume them, using the same language interface — a closed loop where the model teaches itself what is useful to know.

Significance assessment: The representation-learning framing is a theoretical contribution more than an empirical one — the paper does not compare against alternative representation learning methods or demonstrate that $c'$ transfers to other models. But it provides a conceptual vocabulary for understanding why sleep-time compute works, beyond the systems-level argument of "we moved computation earlier." It suggests that the enriched context $c'$ is not merely pre-computed answers but a genuine compression of the context that retains query-relevant structure while discarding irrelevant detail. The fact that concatening $k = 5$ parallel sleep-time generations outperforms $k = 10$ (Section 5.2) supports this: more representation is not always better; there is a sweet spot where the representation captures sufficient structure without introducing noise or redundancy that the test-time model struggles to parse.

Evidence anchor: The representation-learning framing is most directly supported by the scaling experiments in Figures 7 and 8, where increasing sleep-time compute (more parallel generations or higher reasoning effort) produces systematic improvements in test-time accuracy that diminish rather than saturate abruptly — a pattern consistent with representation quality improving with more compute, up to the point where the representation captures all query-relevant structure in the context. The predictability analysis (Figure 10) also supports this: if $c'$ were simply pre-computed answers, then it would help equally across all predictability levels (since the answers would be present regardless). The fact that it helps more when queries are predictable suggests that $c'$ encodes a distribution of useful information, and queries sample from that distribution — precisely what a good representation should enable.

Innovation 4: Verifier-Free Pareto Improvement Over Parallel Test-Time Scaling with Oracle Access

A striking empirical finding that carries conceptual weight is that sleep-time compute outperforms pass@k parallel scaling at matched test-time token budgets (Figures 5 and 6), despite pass@k having an unrealistic advantage: access to a ground-truth verifier that can perfectly select the correct answer from among $k$ independent samples. In standard test-time compute analysis, pass@k represents an upper bound on what parallel sampling can achieve — it measures whether the correct answer exists in the model's output distribution, assuming perfect selection. For a method that does not use verification at all (sleep-time compute selects a single answer via greedy decoding at temperature 0) to outperform this oracle-assisted baseline is counterintuitive and requires explanation beyond simple efficiency gains.

What makes this finding significant: It suggests that the enriched context $c'$ does more than pre-compute answers — it fundamentally alters the model's output distribution at test-time, making the correct answer more probable in the first place rather than relying on post-hoc selection among many samples. In the pass@k regime, the model's per-sample probability of correctness remains low (otherwise a small $k$ would suffice), and the gains come from sampling until a correct answer appears by chance. In the sleep-time regime, the model's per-sample probability of correctness is increased because it receives $c'$ — a context that already contains the intermediate reasoning — reducing the opportunity for the model to make errors in those early reasoning steps. This is a proposal distribution improvement (to use the terminology from Snell et al., 2024, referenced in the related work) rather than a verifier improvement, and the fact that it beats perfect verification of the original proposal distribution is a strong signal that the proposal distribution itself has been meaningfully improved.

Prior assumption challenged: That parallel test-time scaling with oracle verification represents a ceiling that methods without verification cannot exceed. The paper shows that by preprocessing the context, a model can achieve higher accuracy at the same test-time token budget than generating many independent samples from the raw context and selecting the best — even with perfect selection. This challenges the implicit assumption in the test-time scaling literature that the primary axis of improvement is better search/selection methods for navigating the model's output distribution, suggesting instead that modifying the input to shift the output distribution can be more effective than sophisticated post-hoc selection.

Significance assessment: This is an empirical finding with theoretical implications — it was not an obvious or predicted result, and the paper does not attempt to explain it mechanistically beyond showing that it occurs consistently across tasks and models (GSM-Symbolic with GPT-4o and GPT-4o-mini in Figure 5; AIME with o3-mini, o1, Claude 3.7 Sonnet, and DeepSeek-R1 in Figure 6). The finding suggests a previously underappreciated relationship between context preprocessing and output distribution quality: when a model is given a well-structured, inference-rich context, its default reasoning path (even at low temperature, even with minimal test-time compute) is more likely to be correct than when it must derive all inferences from raw text. This is not simply a matter of "reducing the work the model needs to do" — if it were, pass@k with the raw context should match sleep-time compute, since the model is ultimately doing the same total amount of reasoning (sleep-time + test-time versus test-time-only). The difference is that sleep-time compute sequentially decomposes the reasoning into a context-processing phase and a query-answering phase, and this decomposition somehow produces a more reliable overall inference than doing all the reasoning at once. Understanding why this decomposition improves reliability — whether through reduced cognitive load, better attention allocation, or reduced error propagation across reasoning steps — is an open question that the paper's results motivate but do not resolve.

Evidence anchor: Figure 5 shows the comparison most clearly. On Stateful GSM-Symbolic P1 with GPT-4o-mini, the sleep-time compute curve (blue squares) consistently sits above the pass@k curve at matched test-time token budgets. At approximately 100 test-time tokens per question, sleep-time compute achieves ~0.55 accuracy while pass@k (with perfect verification) achieves ~0.45. On P2 with GPT-4o, the gap is even larger: at 200 test-time tokens, sleep-time compute reaches ~0.65 accuracy versus ~0.50 for pass@k. Figure 6 shows the same pattern for reasoning models on Stateful AIME: for o3-mini, the sleep-time compute curve dominates pass@k across nearly the entire budget range. The consistency across models (non-reasoning and reasoning), tasks (GSM-Symbolic and AIME), and difficulty levels (P1 and P2) strengthens the finding beyond a model-specific artifact.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper evaluates on two categories of custom-constructed datasets. Stateful GSM-Symbolic is derived from the GSM-Symbolic benchmark (Mirzadeh et al., 2024) by splitting each problem into a context (all statements except the final question) and a query (the final question), producing two difficulty splits: P1 (5,000 examples, adding one distractor clause to original GSM8K problems) and P2 (2,500 examples, adding two clauses). Stateful AIME combines 60 questions from AIME 2024 and 2025, split into context and query by treating all but the last punctuation-separated statement as context and the final statement as query, with manual rearrangement for edge cases where the question appears in a non-final statement (Section 4.1, Appendix J). For amortization experiments, Multi-Query GSM-Symbolic extends Stateful GSM-Symbolic by generating 10 additional question-answer pairs per context using o3-mini, resulting in 12,043 total questions across 1,095 contexts for P1 and 5,497 questions across 500 contexts for P2 (Appendix C, Table 1). SWE-Features provides 33 real-world software engineering pull requests (18 from Aider-AI/aider, 15 from comfyanonymous/ComfyUI) filtered to modify at least three .py or .js files, introduce new functionality rather than fix bugs, and be self-contained (Appendix D).
Base model(s). Experiments span two categories of models chosen to represent points on the capability spectrum where test-time compute scaling yields meaningful improvements. For Stateful GSM-Symbolic, the paper uses GPT-4o-mini and GPT-4o — non-reasoning models where test-time compute is controlled through verbosity prompting. For Stateful AIME, the paper uses reasoning models with built-in test-time scaling: OpenAI o1, OpenAI o3-mini, Anthropic Claude 3.7 Sonnet (with extended thinking), and DeepSeek-R1 (DeepSeek-AI, 2024). The SWE-Features case study uses Claude 3.7 Sonnet. The choice of both non-reasoning and reasoning model classes is deliberate: it tests whether sleep-time compute benefits generalize across fundamentally different approaches to test-time scaling (externally prompted chain-of-thought versus internally trained reasoning procedures).
Metrics. The primary metric across all math reasoning experiments is accuracy — the fraction of questions for which the model's final answer matches the ground-truth answer, using GSM-Symbolic's standard grading for math problems. For reasoning models (o1, o3-mini, DeepSeek-R1), results are averaged over 3 runs; for Claude 3.7 Sonnet, results are averaged over 10 runs "as we observed more noise in initial experiments" (Section 5.1). For SWE-Features, the metric is F1 score between the set of files the agent predicts to modify and the ground-truth set of modified files in the actual pull request, computed over files with "modified" status only (not newly created files) — a proxy for task understanding rather than functional correctness, used because "there are not straightforward tests to use for evaluation" on GitHub-scraped PRs (Appendix D). For amortization analysis, the metric is total inference cost per query, computed using a linear cost model where test-time tokens are weighted 10× more than sleep-time tokens (Section 5.3), with accuracy plotted against this cost-weighted x-axis rather than raw token counts.
Baselines. The paper employs several baselines designed to test different aspects of the sleep-time compute claim. Standard test-time compute only: the model receives raw context $c$ and query $q$ simultaneously at test-time with no preprocessing, using the same test-time verbosity/reasoning-effort variations as the sleep-time condition — this is the primary pareto frontier baseline (Figures 3, 4). Pass@k parallel scaling (Brown et al., 2024): $k$ independent samples are generated from the lowest sequential compute setting, and the answer is selected by an oracle verifier — this represents "an upper bound on what parallel sampling could achieve with a perfect verifier" and is applied at matched test-time token budgets (Figures 5, 6). Context-only baseline (Appendix I): the model receives only $c$ without $q$ and must guess the most likely question and output its answer — this tests whether Stateful datasets contain trivially predictable questions that would make sleep-time compute equivalent to early answering. Majority voting is not used as a baseline; the paper focuses on comparing against the test-time scaling approaches dominant in the literature rather than simpler aggregation methods.
Generation budget / compute accounting. The paper measures test-time compute in average test-time tokens per question for math benchmarks, computed by averaging the number of output tokens (including reasoning traces) across all test-set questions at each prompt/reasoning-effort setting. This provides a continuous x-axis against which accuracy is plotted, producing the pareto curves in Figures 3–8. For sleep-time compute, only the test-time tokens (tokens generated after the query arrives) are counted on the x-axis; sleep-time tokens are accounted for separately in the amortization analysis using the cost model. For amortization experiments (Figure 9), the x-axis shifts to total inference cost per query, computed as $(\text{SleepTokens} + t \cdot \text{TestTokens}) / N$ where $t = 10$ and $N$ is the number of queries per context, reflecting that test-time tokens are "roughly 10× more expensive" due to latency constraints (Section 5.3). For SWE-Features, the budget is measured in agent steps (API calls) with a hard cap enforced by the Letta framework's max_chaining_steps parameter, and the x-axis reports average test-time tokens consumed across those steps (Figure 11). The paper sweeps test-time budgets at powers of 2 for parallel scaling (e.g., $N = 1, 2, 4, 8$ ) and at discrete reasoning-effort or verbosity levels for sequential scaling.
Cross-validation / statistical protocol. The paper does not employ k-fold cross-validation on its test sets. Instead, results on Stateful AIME are reported as averages over 3 runs for o1, o3-mini, and DeepSeek-R1, and 10 runs for Claude 3.7 Sonnet (to account for observed noise). On Stateful GSM-Symbolic, generation uses temperature 0 (deterministic decoding), so no repeated sampling is performed. For the SWE-Features case study with 33 examples, the paper reports a single evaluation without cross-validation splits, making these results more exploratory. The difficulty quintile analysis (Section 5.4) bins questions into five equally sized bins based on query predictability scores after the fact, with no cross-validation on bin boundaries. The paper does not report confidence intervals, standard errors, or statistical significance tests for any accuracy comparisons — all results are presented as point estimates on the pareto curves, which limits the ability to assess whether the observed shifts (e.g., "up to 13% improvement") are statistically reliable or within noise for the 500-question GSM-Symbolic test set and the 60-question AIME test set. This is a meaningful limitation given the small test sets, particularly for AIME (60 questions) and SWE-Features (33 examples).

Main Quantitative Results

Sleep-Time Compute Shifts the Test-Time Compute vs. Accuracy Pareto Frontier (Section 5.1)

The headline finding is that sleep-time compute produces a pareto improvement in the tradeoff between test-time compute and accuracy across all tasks and models tested, enabling the same accuracy with substantially fewer test-time tokens. On Stateful GSM-Symbolic (Figure 3), GPT-4o-mini with sleep-time compute achieves at the lowest test-time budget (Verbosity 0, ~50 test-time tokens/question on P1) approximately 0.55 accuracy on P1 — roughly matching the standard test-time compute baseline at Verbosity 2–3 (~150–200 test-time tokens), representing a ~3–5× reduction in test-time tokens needed. GPT-4o shows similar behavior: with sleep-time compute at Verbosity 0 (~50 tokens) it achieves approximately 0.70 accuracy on P1, matching the standard baseline at ~150–200 tokens. On P2 (the harder split), the same pattern holds: GPT-4o-mini with sleep-time compute at ~50 tokens achieves approximately 0.40 accuracy, matching the standard baseline at ~200 tokens — again roughly a 4× reduction.

The authors explicitly state in Section 5.1: "at lower test-time budgets, the performance of sleep-time compute is significantly better than the baseline, achieving performance comparable to that of the baseline with 5× less test-time tokens" (Section 5.1). However, they also note a qualification: "at the test-time compute budgets, the test-time compute only baseline slightly outperforms sleep-time compute." This refers to the highest-verbosity settings (Verbosity 3–4) on P1 with GPT-4o-mini, where the standard baseline curve edges slightly above sleep-time compute at ~400+ test-time tokens (~0.78 vs ~0.75). The paper hypothesizes that "this may be because the standard test-time compute only has the content relevant to the specific question, so there is less distracting information in the prompt" — a tradeoff where the enriched context $c'$ may include inferences that, while generally useful, become noise when the model has ample test-time budget to perform all reasoning from scratch.

On Stateful AIME (Figure 4), the pareto shift is more dramatic. For o3-mini, the sleep-time compute curve reaches approximately 0.50 accuracy at ~1,000 test-time tokens, while the standard baseline requires roughly 3,000–4,000 tokens to reach comparable accuracy — a 3–4× reduction. At higher accuracy levels (~0.65), sleep-time compute achieves this at ~2,000 tokens where the baseline requires ~6,000+. For DeepSeek-R1, the baseline curve is relatively flat (accuracy improving slowly from ~0.20 to ~0.50 as tokens increase from ~1,000 to ~6,000), while sleep-time compute jumps to ~0.55 accuracy at ~1,000 tokens and stays consistently above the baseline across the entire budget range. For Claude 3.7 Sonnet, sleep-time compute at ~2,500 test-time tokens achieves ~0.42 accuracy versus ~0.35 for the baseline at the same budget, and reaches ~0.425 at ~5,000 tokens where the baseline is at ~0.38. The exception is o1, which "demonstrates limited gains" — Figure 4 shows the sleep-time and baseline curves largely overlapping for o1, with a small separation at very low test-time budgets (~0.60 sleep-time vs ~0.55 baseline at ~2,000 tokens) that mostly converges at higher budgets. The paper does not analyze this exception in detail, but it may relate to o1's already-extensive internal reasoning making context preprocessing less additive.

Pass@k comparison (Figures 5 and 6). On Stateful GSM-Symbolic (Figure 5), sleep-time compute consistently outperforms pass@k parallel scaling at matched test-time token budgets. For GPT-4o-mini on P1, at ~100 test-time tokens per question, sleep-time compute reaches ~0.55 accuracy versus ~0.45 for pass@k. On P2 with GPT-4o, the gap widens: at ~200 test-time tokens, sleep-time compute achieves ~0.65 versus ~0.50 for pass@k. The dominant pattern holds for GPT-4o-mini on P2 as well (sleep-time at ~100 tokens reaches ~0.40 vs pass@k at ~0.30). On Stateful AIME (Figure 6), the pattern replicates across reasoning models. For o3-mini, the sleep-time compute curve sits above pass@k from ~2,000 to ~7,000 tokens, with a peak gap of approximately 0.10 accuracy at ~3,000 tokens. For o1, the gap is smaller but sleep-time compute still dominates across most of the budget range. For Claude 3.7 Sonnet, sleep-time compute at ~2,500 tokens reaches ~0.40 versus pass@k at ~0.35; at ~10,000 tokens, sleep-time reaches ~0.43 versus pass@k at ~0.37. For DeepSeek-R1, sleep-time compute starts at ~0.55 accuracy at ~1,000 tokens versus pass@k at ~0.30, and reaches ~0.58 at ~5,000 tokens versus pass@k at ~0.50.

The pass@k comparison is particularly significant because pass@k assumes oracle verification — a perfect ability to identify the correct answer when it exists among $k$ samples. The paper explicitly acknowledges this advantage: pass@k "makes the unrealistic assumption of having oracle query access to a ground truth verifier at test-time, an assumption which we do not make with sleep-time compute" (Section 5.1). That sleep-time compute outperforms this oracle-assisted baseline suggests that the enriched context $c'$ does more than pre-compute answers — it fundamentally improves the model's output distribution, making the correct answer more probable in a single sample than it is across multiple independent samples from the raw context with perfect selection.

Scaling Sleep-Time Compute Further Shifts the Pareto Frontier (Section 5.2)

Non-reasoning models (Figure 7). On Stateful GSM-Symbolic, increasing the number of parallel sleep-time generations ( $k \in \{1, 2, 5, 10\}$ ) produces progressive outward shifts of the test-time compute vs. accuracy curve. On P1 with GPT-4o-mini, increasing from $k = 1$ to $k = 5$ improves accuracy by approximately 5–8 percentage points across test-time budgets (e.g., at ~100 test-time tokens, $k = 1$ achieves ~0.60 while $k = 5$ achieves ~0.68). Moving from $k = 5$ to $k = 10$ provides minimal additional gain and sometimes slightly reduces accuracy — the paper explicitly states that "5 parallel generations generally outperforms 10" (Section 5.2). On P2, the gains from scaling sleep-time compute are larger: GPT-4o with $k = 5$ achieves approximately 0.75 accuracy at ~100 test-time tokens versus ~0.58 for $k = 1$ , a ~17 percentage point improvement. The paper reports that scaling sleep-time compute "improving performance by up to 13% on Stateful GSM-Symbolic" — this 13% figure represents the maximum gain observed across all configurations, specifically for GPT-4o on P2 when scaling from $k = 1$ to $k = 5$ at certain test-time budgets.

The paper highlights that "the largest gains on more difficult tasks with stronger models (eg. on P2 with 'gpt-4o')" (Section 5.2), suggesting that as the context becomes more complex (P2 adds more distractor clauses than P1) and the model has greater reasoning capacity (GPT-4o vs GPT-4o-mini), the returns to additional sleep-time compute increase. This is an intuitive result: richer contexts have more structure worth pre-computing, and stronger models are more capable of generating useful inferences that weaker models cannot reliably produce, making the pre-computed inferences from a stronger model more valuable.

Reasoning models (Figure 8 and Appendix M, Figures 25–26). On Stateful AIME, scaling sleep-time compute by varying reasoning effort (low → medium → high) for o1 and o3-mini also shifts the pareto curve outward. For o3-mini on AIME 2024 (Figure 25), high reasoning effort sleep-time reaches ~0.72 accuracy at ~1,500 test-time tokens versus ~0.65 for low effort sleep-time — a gain of approximately 7 percentage points. On AIME 2025 (Figure 26), high reasoning effort sleep-time reaches ~0.70 accuracy at ~1,000 test-time tokens versus ~0.62 for low effort. For o1, the gains are more modest: on AIME 2024, high effort sleep-time reaches ~0.58 accuracy at ~2,000 test-time tokens versus ~0.54 for low effort. The paper reports that scaling sleep-time compute improves accuracy "by up to 18% on Stateful AIME" (Section 5.2) — this represents the maximum gain across years and models, likely for o3-mini on the 2025 split where the accuracy difference between lowest and highest sleep-time effort at certain test-time budgets reaches this magnitude.

Key observation about diminishing returns. The finding that $k = 5$ outperforms $k = 10$ for non-reasoning models suggests diminishing (and eventually negative) returns to parallel sleep-time generations. The paper does not deeply analyze this phenomenon, but it implies an information-theoretic limit: concatenating too many independent pre-computation chains may introduce redundancy and conflicting inferences that the test-time model cannot effectively integrate, or may exceed the model's effective context window utilization, diluting the signal from genuinely useful pre-computed facts.

Amortization Across Multiple Queries Reduces Per-Query Cost (Section 5.3)

Figure 9 presents the amortization analysis on Multi-Query GSM-Symbolic using the cost model where test-time tokens cost 10× more than sleep-time tokens. The key finding is that sleep-time compute's cost-efficiency relative to standard test-time compute depends strongly on the number of queries per context $N$ :

At $N = 1$ : The sleep-time compute curve sits to the right of the test-time-only baseline on both P1 and P2 — sleep-time compute has higher total cost per query at matched accuracy. On P1 at ~0.50 accuracy, sleep-time compute costs approximately 200 total inference cost units per query versus ~150 for the baseline. On P2 at ~0.40 accuracy, the gap is even larger. This is because the sleep-time token overhead is fully borne by a single query, and the cost savings from reduced test-time tokens (even weighted at 10×) do not compensate for the added sleep-time tokens.
At $N = 2$ and $N = 5$ : The sleep-time compute curves shift leftward as the fixed sleep-time cost is amortized. On P1 at $N = 2$ , sleep-time compute roughly breaks even with the baseline — at ~0.60 accuracy, both conditions cost approximately 100–120 units per query. At $N = 5$ , sleep-time compute becomes clearly cheaper: the same ~0.60 accuracy costs approximately 60–80 units with sleep-time compute versus ~100 with the baseline. On P2, the crossing point occurs at slightly higher $N$ : at $N = 5$ , sleep-time compute achieves ~0.45 accuracy at ~120 units versus ~150 for the baseline.
At $N = 10$ : Sleep-time compute achieves clear dominance. On P1, at ~0.55 accuracy, sleep-time compute costs approximately 50 units per query versus ~100 for the baseline — a 2× reduction. The paper reports that "we can decrease the average cost per query by up to 2.5× when there are 10 queries per context, compared to the single-query baseline" (Section 5.3). This 2.5× figure represents the maximum reduction across accuracy levels: at ~0.35 accuracy on P2, sleep-time compute with $N = 10$ costs roughly 60 units versus ~150 for $N = 1$ , yielding approximately 2.5× reduction.

Multi-Query GSM-Symbolic for GSM-Symbolic. The amortization impact spans both P1 and P2 and affects the cost-accuracy tradeoff consistently: the sleep-time compute curves for high $N$ are shifted not just to the left (cheaper) but also achieve higher accuracy ceilings than the baseline at the same per-query cost. On P2, at a cost of approximately 200 units per query, the $N = 10$ sleep-time curve reaches ~0.55 accuracy while the baseline reaches only ~0.42 — demonstrating that amortization improves both cost-efficiency and the accuracy ceiling at matched per-query budgets.

Why this matters beyond cost: The amortization result reframes when sleep-time compute is worth deploying. For one-off queries, the upfront sleep-time cost makes it potentially worse than standard test-time compute in total expenditure. For contexts queried many times — shared codebases, popular documentation pages, long-running conversations — the fixed cost is rapidly amortized and the per-query savings compound. The paper's demonstration that this threshold exists (somewhere between $N = 1$ and $N = 5$ on P1, and higher on P2) provides practical guidance: sleep-time compute is not a universal improvement but a strategy tailored to high-query-multiplicity contexts.

Query Predictability Correlates with Sleep-Time Compute Benefit (Section 5.4)

Figure 10 bins GSM-Symbolic questions into five quintiles based on the log-probability of the question given the context under the Llama2-70B base model — a measure of how predictable the query is from the context alone. The paper reports the accuracy delta between sleep-time compute and standard test-time compute (both at the lowest test-time compute budget, Verbosity 0) within each bin. On P1 with GPT-4o-mini:

Bin 1 (least predictable): The accuracy delta is near zero — sleep-time compute provides essentially no benefit, and may even slightly underperform (the bar shows a small positive value, approximately 0.05, but close to zero within the scale).
Bin 5 (most predictable): The accuracy delta is approximately 0.40–0.45 — sleep-time compute substantially outperforms the standard baseline.

The bars show a clear upward trend from Bin 1 to Bin 5 (approximately 0.05 → 0.15 → 0.25 → 0.35 → 0.45 on P1), confirming the hypothesis that "sleep-time compute is most effective in settings where the query is more predictable from the context" (Section 5.4). On P2, the same trend holds but with a wider spread: Bin 1 shows a delta near zero, while Bin 5 shows approximately 0.35–0.40.

Examples of predictable and unpredictable questions (Appendix E). The most predictable P1 question asks "How many grapes can Yusuf harvest in 2 years?" given a context describing Yusuf's grape field yields — the question follows directly from the numbers provided. The least predictable P1 question asks "How much longer do they have to get to grocery store without being late, if they have already wasted 48 minutes to get a coffee before their walk?" given a context about walking times to various landmarks — the question introduces a new twist (coffee delay) not directly derivable from the context alone. These examples illustrate that predictability in this analysis captures whether the query follows a straightforward template from the context (predictable) or introduces novel variables or reasoning steps (unpredictable). The paper notes that "our notion of question predictability generally aligns with the intuition that contexts where the query pattern is more predictable benefit most from sleep-time compute" (Section 5.4).

This analysis provides a mechanistic explanation for when sleep-time compute works: it is most beneficial when the kinds of inferences that can be pre-computed from the context (intermediate quantities, structural relationships) are precisely the inferences that the query requires. When the query is unpredictable — asking about something not obviously foreshadowed by the context, or requiring reasoning steps that don't follow from the information provided — the pre-computed $c'$ may not contain the relevant inferences, and the model must essentially perform the full reasoning from scratch (or may be distracted by the pre-computed but irrelevant inferences). The analysis does not, however, establish a causal relationship — predictability and sleep-time benefit could both be driven by a common factor (e.g., problem simplicity) rather than predictability directly enabling sleep-time compute's effectiveness.

SWE-Features Case Study (Section 6)

Figure 11 shows the results of applying sleep-time compute to the SWE-Features benchmark with Claude 3.7 Sonnet. The F1 score (file-level correctness of predicted modifications) is plotted against average test-time tokens per question, with the sleep-time and standard test-time compute curves overlaid:

At low test-time budgets (~3,000–4,000 test-time tokens): Sleep-time compute achieves F1 scores of approximately 0.48–0.50 versus ~0.42–0.44 for the standard baseline. The paper reports this as "up to roughly a 1.5× decrease in test-time tokens" — meaning sleep-time compute at ~3,000 tokens matches the baseline accuracy at ~4,500+ tokens (Section 6).
At high test-time budgets (~7,000–10,000 tokens): The standard test-time compute curve crosses above sleep-time compute, reaching approximately 0.52–0.54 F1 at ~10,000 tokens versus ~0.48–0.50 for sleep-time compute. The paper reports that "when the test-time compute budget is high, using only test-time compute can perform better" (Section 6).

The paper offers a qualitative analysis of why this reversal occurs: "using only test-time compute tends to begin editing files earlier and usually edits fewer files overall. In contrast, the agent with sleep-time compute, having explored more files during the test-time phase, tends to edit more files, which may lead to slightly lower precision." In other words, the sleep-time agent, having built a richer mental model of the codebase during its pre-test-time exploration phase, identifies a broader set of potentially relevant files and proposes edits to more of them. At low budgets, this broader exploration captures correct files that a time-constrained standard agent misses (higher recall). At high budgets, the standard agent can afford to explore more thoroughly and correctly identify the relevant files without pre-computation, while the sleep-time agent's broader edits start to include false positives (lower precision), dragging down the F1 score.

Caveats about SWE-Features. The sample size is small (33 examples total), and the F1 metric captures only file identification correctness, not functional correctness of the implemented feature. The paper acknowledges that the agent is "still instructed to implement the required functionality in a Docker environment and write test functions to validate the implementations" but evaluates only on file sets. This makes SWE-Features a proof of concept rather than a rigorous evaluation — the trends align with the math benchmarks (sleep-time compute helps at low test-time budgets, standard compute catches up or exceeds at high budgets), but the limited scale and indirect metric mean these results are suggestive rather than conclusive.

Ablation Studies and Robustness Checks

Context-only prediction baseline (Appendix I, Figures 21–22): To verify that Stateful datasets do not contain trivially predictable questions that would make sleep-time compute equivalent to simply guessing the question, the paper evaluates a baseline where the model receives only $c$ (no query) and must output an answer to whatever question it predicts. On Stateful GSM-Symbolic (Figure 21), sleep-time compute substantially outperforms the context-only baseline across all test-time budgets (e.g., on P1 with GPT-4o-mini, sleep-time at ~50 tokens achieves ~0.55 accuracy versus context-only near zero). On Stateful AIME (Figure 22), the gap is similarly large: for o3-mini, sleep-time compute at ~1,000 test-time tokens reaches ~0.50 accuracy versus context-only at ~0.30–0.35. This demonstrates that the questions "are not trivially predictable from the context" and that sleep-time compute's $c'$ is genuinely enriched context rather than a covert pre-answer.

Scaling number of parallel sleep-time generations (Figure 7, Section 5.2): The sweep across $k \in \{1, 2, 5, 10\}$ for non-reasoning models reveals that $k = 5$ generally outperforms $k = 10$ , and $k = 10$ sometimes performs worse than $k = 2$ (e.g., GPT-4o-mini on P2 at high test-time budgets where $k = 2$ achieves ~0.75 versus $k = 10$ at ~0.72). This negative result establishes that more sleep-time compute is not always better — there is an optimal level of sleep-time investment beyond which additional pre-computed inferences become noisy or redundant. The paper does not analyze this threshold in detail, but it represents a meaningful robustness check against the naive hypothesis of monotonic improvement.

Low vs. medium vs. high reasoning effort for sleep-time on reasoning models (Figure 8, Appendix M): Varying the sleep-time reasoning effort for o1 and o3-mini on Stateful AIME shows consistent but diminishing returns. On AIME 2024 with o3-mini, high effort sleep-time outperforms medium, which outperforms low, but the gap between medium and high is smaller than between low and medium (approximately 3–4 percentage points vs. 7–8 percentage points at matched test-time budgets), suggesting diminishing returns. On AIME 2025 with o1, the three effort levels are more tightly clustered (~0.55, ~0.56, ~0.58 at ~2,000 test-time tokens), suggesting that for o1, additional sleep-time reasoning effort provides minimal incremental benefit. This model-specific variation is an important negative result: sleep-time compute scaling is not uniformly effective across models, with o1 showing "limited gains" across all sleep-time scaling experiments.

Year-by-year AIME breakdown (Appendices L and M, Figures 23–26): The paper disaggregates Stateful AIME results by year (2024 vs. 2025) in the appendix. The gap between sleep-time compute and the standard baseline is generally larger on AIME 2025 than on AIME 2024 for o3-mini and o1 (e.g., o3-mini on 2025: sleep-time reaches ~0.65 at ~1,500 tokens vs. baseline at ~0.45; on 2024: sleep-time reaches ~0.55 at ~1,500 tokens vs. baseline at ~0.40). The year-by-year consistency — sleep-time compute shifting the pareto frontier outward in both years, for all models except o1 — serves as a robustness check: the effect is not specific to a single test set draw. However, the 2024 and 2025 AIME splits are small (~30 questions each), so the disaggregated results have higher variance.

Multi-Query dataset generation quality (Appendix C, Figure 20): The paper provides example generated questions for a context about Sofia's toys, showing that the synthetic question generation produces both straightforward queries ("How many action figures does the pack contain?") and more complex ones ("If Sofia divided the 49 bouncy balls equally into 7 baskets, how many balls would each basket contain?"). This serves as a qualitative validation that the Multi-Query dataset contains questions spanning a range of difficulty and requiring different subsets of the pre-computable context information, rather than being trivially similar to the original question.

Critical Assessment

Claim 1: "Sleep-time compute reduces the test-time compute needed to achieve the same accuracy by ~5×" — The evidence supports this claim at low to moderate test-time budgets but with important caveats. On Stateful GSM-Symbolic (Figure 3), the ~5× reduction is most clearly visible when comparing sleep-time compute at Verbosity 0 (the most aggressive test-time budget reduction) against the standard baseline at Verbosity 2–3. However, the paper does not report a single "~5×" figure as a precise metric — it is an approximate characterization of the visual gap in Figure 3. On Stateful AIME (Figure 4), the reduction factor varies substantially by model: for o3-mini it is approximately 3–4×; for DeepSeek-R1 it is larger at low accuracy levels; for o1 it is minimal. The ~5× figure should be understood as the best case across tasks and models at the low end of the test-time budget spectrum, not as a uniform improvement. Furthermore, the paper's own observation that at the highest test-time budgets "the test-time compute only baseline slightly outperforms sleep-time compute" (Section 5.1) means the ~5× reduction does not hold at all accuracy levels — it is specifically a low-budget phenomenon where the standard baseline requires substantial computation to reach the accuracy that sleep-time compute achieves with minimal test-time tokens.

A significant limitation is that the comparison is between test-time token counts, but the total system compute (sleep-time + test-time) is not equalized. Sleep-time compute achieves the same accuracy with 5× fewer test-time tokens, but it adds sleep-time tokens that are not counted on the x-axis of Figures 3 and 4. The paper acknowledges this through the separate amortization analysis (Figure 9), which uses the weighted cost model and shows that at $N = 1$ (single query per context), sleep-time compute can actually be more expensive in total cost. The ~5× test-time token reduction is an accurate statement about the x-axis of Figure 3, but it does not represent a 5× reduction in total system cost — it represents a shift in when the computation occurs, with the efficiency gains materializing only when contexts are reused across multiple queries.

Claim 2: "By scaling sleep-time compute we can further increase accuracy by up to 13% on Stateful GSM-Symbolic and 18% on Stateful AIME" — These figures refer to the accuracy gains from increasing sleep-time compute (e.g., from $k = 1$ to $k = 5$ parallel generations) at matched test-time budgets, not absolute accuracy improvements over the standard baseline. On Stateful GSM-Symbolic (Figure 7), the 13% figure corresponds to the gain for GPT-4o on P2 when moving from 1 to 5 parallel sleep-time generations — at some test-time budgets, $k = 5$ achieves ~0.75 versus $k = 1$ at ~0.62, representing a 13 percentage point improvement. On Stateful AIME (Section 5.2, Figure 8, and Appendix M), the 18% figure corresponds to the gain for o3-mini on certain AIME splits when moving from low to high reasoning effort sleep-time. These are valid upper-bound characterizations, but they represent the maximum improvement across all configurations tested — not average or expected gains. The actual improvement varies substantially by model, task, and test-time budget. For o1 on AIME 2024, the gain from low to high effort sleep-time is approximately 4 percentage points (0.54 → 0.58), far below the 18% ceiling.

Moreover, the "up to" framing masks an important caveat: these gains are achieved by spending more total compute (more sleep-time tokens for higher $k$ or higher reasoning effort). The paper does not plot these results on a total-compute x-axis, so the reader cannot assess whether the accuracy gains from scaling sleep-time compute are cost-effective compared to simply scaling test-time compute further. The results demonstrate that sleep-time compute can be scaled to achieve higher accuracy, but do not address whether this is the most compute-efficient way to achieve that accuracy.

Claim 3: "By amortizing sleep-time compute across multiple queries about the same context, we can decrease the average cost per query by 2.5×" — This claim is well-supported by Figure 9 but applies only at high query multiplicity ( $N = 10$ ) and under the specific cost model assumption $t = 10$ . The 2.5× figure compares the cost per query of sleep-time compute with $N = 10$ against $N = 1$ (not against the standard test-time baseline), and represents the maximum reduction observed across accuracy levels. At $N = 2$ , the reduction is negligible or non-existent (sleep-time compute roughly breaks even with the standard baseline). At $N = 5$ , the reduction is approximately 1.5–2× (visual estimate from Figure 9). The amortization benefit is therefore a continuous function of $N$ and the cost multiplier $t$ , and the 2.5× figure is a point on this continuum at $N = 10$, $t = 10$ . Changing $t$ would shift the threshold: if test-time tokens are less than 10× more expensive than sleep-time tokens, larger $N$ would be required to achieve the same amortization benefit. The paper's choice of $t = 10$ ` is justified by a citation to Databricks documentation on latency-optimized inference pricing, but the sensitivity of the results to this parameter is not analyzed.

Weaknesses that limit the generality of all claims:

Small test sets, no statistical rigor. Stateful AIME has 60 total questions (30 per year for year-by-year breakdowns). SWE-Features has 33 examples. Stateful GSM-Symbolic is larger (5,000 for P1, 2,500 for P2), but the paper does not report confidence intervals, standard errors, or significance tests for any comparison. The pareto curves in Figures 3–8 are plotted as lines connecting point estimates, with no error bands. On AIME with 60 questions, a difference of 5 percentage points corresponds to 3 questions — within the range of sampling noise. The increased averaging for Claude 3.7 Sonnet (10 runs vs. 3 runs for other models) suggests the authors observed high variance, but they do not quantify it. Without error estimates, it is impossible to assess whether the observed pareto shifts are statistically reliable or consistent with noise.
Single cost model parameter, no sensitivity analysis. The amortization results depend entirely on the assumption $t = 10$ — test-time tokens cost 10× more than sleep-time tokens. The paper does not analyze how results change for $t = 5$ , $t = 20$ , or $t = 1$ . If $t = 1$ (equal cost for sleep-time and test-time tokens), the amortization benefit would be dramatically smaller or non-existent, since the cost asymmetry is what makes offloading computation to sleep-time attractive. The paper's claim to "decrease the average cost per query by 2.5×" is conditional on the $t = 10$ assumption and would change substantially with different cost ratios.
No model diversity beyond OpenAI/Anthropic/DeepSeek. All experiments use commercial API models (GPT-4o, GPT-4o-mini, o1, o3-mini, Claude 3.7 Sonnet, DeepSeek-R1). The paper does not test on open-weight models (Llama, Mistral, Qwen), which would be relevant for on-device or self-hosted deployments where the sleep-time/test-time cost asymmetry might differ substantially from the API-based $t = 10$ assumption. The paper's recommendation that sleep-time compute is a "representation learning" strategy that could transfer across models (Section 7) is not tested — all experiments use the same model for sleep-time and test-time phases.
Sleep-time compute is not compared against alternatives that achieve similar effects. The paper does not compare against simpler approaches that might capture some of the same benefits: (a) prompting the model to produce a summary of the context at test-time before answering (which would add test-time tokens but might capture the benefit of structured context without sleep-time pre-computation); (b) using retrieval-augmented generation to fetch relevant pre-computed facts from a database rather than generating them via sleep-time model calls; (c) multi-turn decomposition where the model is first asked to process the context and then asked the query in a second turn (which shifts the computation boundary but still occurs at test-time). Without these comparisons, it is unclear whether the benefits come specifically from offline pre-computation or simply from explicit context structuring regardless of when it occurs.
The pass@k outperformance result, while striking, is not contextualized. Sleep-time compute outperforms pass@k with oracle verification (Figures 5, 6), but the paper does not analyze why beyond a brief suggestion that it implies "a proposal distribution improvement." The pass@k baseline uses the lowest sequential compute setting (Verbosity 0 for non-reasoning models), while sleep-time compute draws on the enriched context $c'$ — but critically, the generation of $c'$ itself involves significant computation (sleep-time tokens) that is not accounted for in the test-time token budget. The comparison effectively allows sleep-time compute to use additional total tokens (sleep-time + test-time) compared to the pass@k baseline's test-time-only budget. This makes the outperformance less surprising — it's not that sleep-time compute achieves higher accuracy with the same total tokens, but that it achieves higher accuracy by shifting some token expenditure to an uncounted phase. A fairer comparison would plot total tokens (sleep-time + test-time) on the x-axis for both conditions.
SWE-Features is exploratory, not conclusive. The 33-example case study, with evaluation limited to file identification F1 and no functional correctness testing, cannot support strong claims about sleep-time compute's applicability to agentic software engineering. The paper frames it as a "case study," which is appropriate, but the results (Figure 11) should not be given equal evidentiary weight to the math benchmark results. The finding that sleep-time compute hurts precision at high test-time budgets because the agent "tends to edit more files" is an interesting hypothesis but is based on qualitative observation rather than systematic analysis.
Difficulty estimation is not addressed. Unlike the Snell et al. (2024) paper on compute-optimal test-time scaling (which this paper cites), sleep-time compute does not include a mechanism for estimating whether a given context will benefit from preprocessing or for adaptively allocating compute between sleep-time and test-time phases. The paper applies sleep-time compute uniformly to all contexts, but the predictability analysis (Section 5.4) shows that the benefit is concentrated in predictable queries. This suggests that adaptive allocation — identifying which contexts have predictable queries and investing sleep-time compute only in those — could improve efficiency, but no such mechanism is developed or evaluated. The paper acknowledges this in Section 7 as future work: "identifying which contexts may have predictable questions and optimally allocating inference compute between sleep-time and test-time across different contexts and queries."
No latency measurements. The paper's motivation emphasizes latency reduction as a key benefit of sleep-time compute ("waiting potentially several minutes for answers"), but no latency measurements are reported. The x-axis measures token counts, not wall-clock time. For parallel test-time scaling (pass@k), token count and wall-clock time are similar (all samples run in parallel), but for sequential test-time scaling, token count and wall-clock time are directly proportional. Sleep-time compute replaces a long test-time sequential reasoning trace with a short one, which should reduce wall-clock latency, but the paper does not quantify this reduction. Without latency measurements, the claim that sleep-time compute addresses the "significant increase in latency" from test-time compute (Section 1) is supported only indirectly through token count reductions.

What experiments would have strengthened the paper:

Total-compute-equalized comparisons: Plotting all results on an x-axis of total tokens (sleep-time + test-time) rather than test-time tokens alone, to make explicit the tradeoff between shifting computation and reducing it.
Latency measurements: Reporting wall-clock time for both sleep-time and test-time phases, demonstrating the claimed latency benefits concretely.
Cross-model sleep-time compute: Testing whether $c'$ generated by GPT-4o can improve test-time accuracy for GPT-4o-mini, testing the "representation learning" hypothesis that $c'$ is a transferable natural-language representation.
Sensitivity analysis on the cost multiplier $t$ : Showing how the amortization threshold (the $N$ at which sleep-time compute becomes cost-effective) varies with $t$ , providing guidance for practitioners with different cost structures.
Comparison to context summarization at test-time: A baseline where the model is prompted to first summarize the context and then answer the query — both at test-time — to test whether the benefit is from pre-computation specifically or from context restructuring generally.

6. Limitations and Trade-offs

The Headline Efficiency Gains Exclude Sleep-Time Compute from the Budget

The assumption or constraint. The primary experimental results in Figures 3–8 present test-time compute (x-axis) against accuracy (y-axis), demonstrating that sleep-time compute achieves the same accuracy with ~5× fewer test-time tokens. However, the x-axis explicitly counts only test-time tokens — the tokens generated after the query arrives. The sleep-time compute phase itself consumes a substantial number of tokens (the iterative rethink_memory calls, the parallel generations, or the reasoning effort expended during sleep-time), and this cost is excluded from the pareto frontier comparisons in Sections 5.1 and 5.2. The paper transparently addresses this in Section 5.3: "we model the total cost of inference between both sleep-time and test-time, by up-weighing the cost of test-time tokens" — but this cost analysis is presented separately (Figure 9) and operates under a specific cost asymmetry assumption that is not integrated into the primary claims about test-time token reduction.

The consequence. A practitioner reading the headline ~5× reduction might reasonably conclude that sleep-time compute makes the overall inference pipeline 5× cheaper. This is incorrect. The reduction applies strictly to test-time tokens — the tokens the user waits for — but the total system token expenditure (sleep-time + test-time) may be comparable, higher, or even much higher than standard test-time compute alone. The amortization analysis (Figure 9) reveals that at N = 1 query per context, sleep-time compute can be more expensive in total cost-weighted tokens than standard test-time compute, because the sleep-time overhead is not recouped. The ~5× figure is best understood as a latency and per-query responsiveness improvement — the user sees a much faster response — not as a 5× reduction in total compute bill. For deployment scenarios where sleep-time GPU hours are as expensive as test-time GPU hours (e.g., self-hosted models on fixed hardware where idle GPUs represent sunk cost, or cloud inference where all tokens are priced similarly), the total-cost picture is substantially less favorable than the token-count reduction suggests.

What evidence exists in the paper. Figure 9 is the critical evidence: the x-axis shifts from "Avg. Test Time Tokens / Question" to "Total Inference Cost / Query," integrating sleep-time tokens with a 10× discount. At N = 1, the sleep-time compute curve sits to the right of the standard test-time compute baseline on both P1 and P2, indicating higher total cost at matched accuracy. At N = 10, the sleep-time curve shifts leftward and achieves up to 2.5× cost reduction — but the 2.5× figure is relative to N = 1 sleep-time compute, not relative to the standard test-time baseline. The paper does not report the absolute number of sleep-time tokens consumed in these experiments, making it impossible for a reader to independently assess the total-compute tradeoff. The paper acknowledges this gap explicitly in Section 5.3: "we consider a simple linear model where tokens generated at test-time are a factor t the cost of the tokens at sleep-time" — the analysis is explicitly conditioned on t = 10 and the paper notes that it "can be generalized to different cost functions," but no sensitivity analysis on t is provided.

Mitigation status. Partially addressed through the separate amortization analysis (Figure 9), which explicitly models the total cost. However, this analysis is not integrated into the primary pareto plots — a reader must read Section 5.3 carefully to understand that the ~5× figure is a test-time-only metric. The paper does not report the absolute sleep-time token counts or provide total-compute-equalized versions of Figures 3 and 4. Future work on adaptive allocation between sleep-time and test-time compute (Section 7) could address this by making the sleep-time budget explicit and optimizing jointly.

The Difficulty Estimation and Query Predictability Problem Is Unresolved

The assumption or constraint. Sleep-time compute applies the same S(c) → c' transformation to every context, regardless of whether that context will lead to queries that benefit from pre-computation. The paper's analysis in Section 5.4 demonstrates that the benefit of sleep-time compute is highly concentrated in contexts where the query is predictable: the accuracy delta between sleep-time and standard test-time compute grows from near zero in the least predictable quintile to 0.35–0.45 in the most predictable quintile (Figure 10). However, the paper's experimental design bakes in predictability as a post-hoc analysis rather than as a pre-computation gating mechanism. At deployment time, a system using sleep-time compute would need to decide — before the query arrives — whether to invest computation in pre-processing a given context. The paper provides no method for making this decision.

The paper acknowledges this gap in Section 7: "An interesting direction for future work is identifying which contexts may have predictable questions and optimally allocating inference compute between sleep-time and test-time across different contexts and queries." The current difficulty estimation method — computing the log-probability of questions under Llama2-70B — requires access to the actual future query, making it circular for the decision it is meant to inform.

The consequence. Without a mechanism for predicting ex ante whether a context will yield predictable queries, a deployment must either (a) apply sleep-time compute uniformly to all contexts, wasting compute on contexts where queries will be unpredictable (where the pre-computed c' provides no benefit and the test-time-only baseline would be equally accurate at lower total cost), or (b) deploy a heuristic for gating sleep-time compute whose performance is unknown and unvalidated. The paper's own evidence (Figure 10) shows that in the lowest predictability quintile, the accuracy delta is approximately 0.05 — sleep-time compute may provide essentially zero benefit while still incurring the sleep-time token cost. At deployment scale, processing a large corpus of contexts uniformly through sleep-time compute when only a fraction will receive predictable queries represents a significant waste of compute.

This limitation is structurally similar to the difficulty estimation problem identified in Snell et al. (2024), where the compute-optimal test-time scaling strategy requires estimating problem difficulty before allocating the inference budget. Snell et al. (2024) addresses this with a PRM-based difficulty estimator, but notes that the estimation cost itself is substantial. Sleep-time compute inherits this difficulty — the decision of whether to apply sleep-time compute is itself a meta-decision that requires information not available until queries start arriving. The paper's suggestion of "identifying which contexts may have predictable questions" (Section 7) is an open problem, not a solved one.

What evidence exists in the paper. Figure 10 and the discussion in Section 5.4 provide clear evidence that the benefit of sleep-time compute is not uniform. The text states: "sleep-time compute is most effective in settings where the query is more predictable from the context" and "The gap between sleep-time compute and standard test-time inference widens as the question becomes more predictable from the context." Appendix E provides examples of predictable versus unpredictable questions, showing that the predictability metric captures intuitive differences in whether the query follows a straightforward template from the context or introduces novel elements. However, the predictability analysis is entirely post-hoc — it uses the actual query to bin the results, which cannot be done in advance at deployment time. The paper does not evaluate any method for predicting query predictability from the context alone.

Mitigation status. Not addressed. The paper identifies this as future work in Section 7: "identifying which contexts may have predictable questions and optimally allocating inference compute between sleep-time and test-time across different contexts and queries." No method, heuristic, or baseline is provided for this gating decision. A practitioner deploying sleep-time compute today would need to develop their own heuristic or accept the waste of applying it uniformly.

The Method Requires Clean Context-Query Decomposability, Which Fails for Many Real-World Tasks

The assumption or constraint. The entire sleep-time compute framework rests on the assumption that an interaction can be cleanly decomposed into a pre-existing context c (available before the query) and a user query q (arriving later), where c contains information that can be fruitfully pre-processed to accelerate answering q. The paper's experimental evaluation constructs datasets that enforce this decomposition: Stateful GSM-Symbolic and Stateful AIME are created by splitting existing math problems into a context (all statements except the final question) and a query (the final question), and Multi-Query GSM-Symbolic is generated by prompting o3-mini to produce "questions and answers about the context at the same difficult level that could plausibly be asked about that context" (Appendix C). The SWE-Features case study clusters related PRs to serve as context for a target PR, assuming that the structure and patterns in the related PRs provide useful pre-computable information.

The paper acknowledges in Section 7 that this is a simplification: "In our experiments, we make the simplifying assumption that interactions fall into two phases: sleep-time and test-time. However, real-world LLM use cases can be more complex, with multiple rounds of interaction and context modifications between rounds (e.g. multiple edits to a code-base)."

The consequence. For many deployed LLM applications, the context-query decomposition is either leaky (the "context" changes between queries, invalidating the pre-computed c') or non-existent (the query itself transforms the relevant context, as in multi-turn dialogue where each turn modifies the conversational state). In a coding assistant, the user might edit files between questions, rendering pre-computed codebase inferences stale. In a multi-turn conversation, the "context" is the dialogue history, but each user utterance simultaneously serves as a query and as new context for future turns — there is no clean temporal separation. In document question-answering, the document corpus is static, satisfying the decomposition, but for interactive tasks like debugging or collaborative writing, the assumption breaks down.

Even when the decomposition holds structurally, the value of sleep-time compute depends on whether c can be usefully pre-processed without knowing q. The paper's own context-only baseline (Appendix I) shows that models cannot trivially predict the question from the context — accuracy is far below sleep-time compute accuracy. This means c' must encode a distribution of potentially useful inferences rather than a targeted answer to a predicted question. For contexts where the space of plausible queries is large, diverse, and difficult to anticipate, the pre-computed c' may contain many inferences that no actual query ever uses, diluting the value-to-cost ratio.

The Multi-Query GSM-Symbolic construction itself illustrates the constraint: the synthetic questions are generated by o3-mini to be "at the same difficult level that could plausibly be asked about that context," which means the set of queries is constructed to be predictable from the context. Real user query distributions may not share this property — users ask unanticipated questions, follow-up questions that build on previous answers, or questions that require information not derivable from the static context alone. The paper's evaluation on Multi-Query GSM-Symbolic thus demonstrates amortization in a setting designed for high query predictability, which may overstate the amortization benefit in less structured query distributions.

What evidence exists in the paper. The context-only baseline in Appendix I (Figures 21–22) provides indirect evidence: models cannot trivially guess the question from the context, meaning that the pre-computation in c' must be robust to query variation. The predictability analysis (Figure 10) provides direct evidence that the benefit of sleep-time compute degrades as queries become less predictable. For the least predictable quintile, the accuracy delta between sleep-time and test-time compute is near zero — even though the context-query decomposition is enforced by the dataset construction, the unpredictable queries do not benefit. This suggests that even in settings where the decomposition holds structurally, the functional benefit depends on the query distribution.

Mitigation status. Acknowledged but not addressed. The paper states that "real-world LLM use cases can be more complex, with multiple rounds of interaction and context modifications between rounds" and that "the length of the sleep-time may also vary significantly between interactions." The paper suggests that "future work should extend sleep-time compute paradigm to more elegantly handle these scenarios" (Section 7), but provides no mechanism for incremental updates to c' as context evolves, or for handling overlapping sleep-time and test-time phases. The SWE-Features case study is a step toward more complex settings but still imposes a clean sleep-time/test-time split: the agent pre-processes related PRs during sleep-time, then receives the target PR at test-time. A deployed coding agent would face continuous context evolution as the user edits code, runs tests, and asks follow-up questions — a regime not evaluated.

The SWE-Features Case Study Is Too Small and Too Narrow to Support Agentic Claims

The assumption or constraint. The paper's only evaluation beyond synthetic math benchmarks is the SWE-Features case study (Section 6), consisting of 33 pull requests across two repositories. The evaluation metric is F1 score over predicted file modifications — a proxy for task understanding rather than a measure of functional correctness. The paper explicitly states the limitation: "Since the PRs are scraped from GitHub, there are not straightforward tests to use for evaluation. Instead, we compare the predicted set of modified files with the ground truth list of modified files" (Section 6). The agents are "still instructed to implement the required functionality in a Docker environment and write test functions to validate the implementations," but this implementation is not evaluated for correctness. The paper frames this as a "case study" rather than a rigorous benchmark evaluation, which is honest, but the findings from this section are then used to support claims about sleep-time compute's applicability to "realistic agentic" tasks.

The consequence. Practitioners in software engineering or agentic LLM deployment should treat the SWE-Features results as suggestive rather than actionable. The sample size (33 PRs, 18 from Aider, 15 from ComfyUI) is far too small to draw statistical conclusions — a difference of 2 percentage points in F1 corresponds to less than one example. The file-level F1 metric captures whether the agent identifies the correct set of files to modify, which is necessary but not sufficient for successful task completion — an agent could correctly identify all files while implementing the feature incorrectly, or could correctly implement the feature while misidentifying one file (e.g., placing a function in a new utility file rather than an existing one). The paper's own qualitative analysis reveals a precision-recall tension: "the agent with sleep-time compute, having explored more files during the test-time phase, tends to edit more files, which may lead to slightly lower precision" (Section 6). This means the F1 metric conflates two competing effects — sleep-time compute may increase recall (finding more relevant files) at the cost of precision (editing irrelevant files), and the F1 score obscures this tradeoff.

The repository clustering used to provide context (related PRs) is itself a potential confound. The paper uses Claude 3.7 Sonnet to cluster PRs, but clustering quality is not evaluated. If the clusters are noisy — grouping PRs that are only superficially related — the "context" provided during sleep-time may be irrelevant to the target PR, and the sleep-time agent may build a mental model of the codebase that is actively misleading. The paper does not report how often the provided context PR was actually helpful, or provide examples of failures attributable to poor context selection.

Finally, the task specificity matters: both repositories are Python-based AI/LLM tools (ComfyUI is a Stable Diffusion interface, Aider is an AI pair programming tool). This is a narrow slice of software engineering — there is no evaluation on web applications, data pipelines, systems code, or other common software categories. The repository structure (relatively modular Python code) may be more amenable to file-level prediction than, say, a monolithic C++ codebase or a distributed microservice architecture.

What evidence exists in the paper. Figure 11 shows the F1 score vs. test-time tokens for the SWE-Features case study with Claude 3.7 Sonnet. The sleep-time compute curve achieves approximately 0.48–0.50 F1 at ~3,000–4,000 tokens versus ~0.42–0.44 for the baseline, but drops below the baseline at higher budgets (~10,000 tokens). The paper provides a qualitative explanation: "using only test-time compute tends to begin editing files earlier and usually edits fewer files overall. In contrast, the agent with sleep-time compute, having explored more files during the test-time phase, tends to edit more files, which may lead to slightly lower precision." This explanation is based on observation rather than systematic analysis — the paper does not report precision and recall separately, nor does it provide examples or counts of false-positive versus false-negative file predictions.

Mitigation status. The paper frames this as a "case study" rather than a core result, which is appropriate. However, the limitation is not mitigated in the evaluation. No statistical measures (confidence intervals, significance tests) are reported. No ablation is performed on the clustering quality or the specific related PRs used as context. No evaluation of functional correctness is provided. The paper does not identify specific failure modes beyond the precision-recall observation, and does not provide guidance for when a practitioner should expect the SWE-Features trends to generalize versus fail.

The Paper Provides No Cross-Model Sleep-Time Compute Results, Limiting the "Representation Learning" Claim

The assumption or constraint. The paper frames sleep-time compute in Section 7 as "representation learning over tokens" and draws an explicit analogy to Bengio et al. (2014): "We first transform the context into a representation that is more amenable to answering test-time queries, and then we utilize that representation at test-time to rapidly answer queries." The paper further connects this to work by Zhong et al. (2022, 2025) on "implementing statistical modeling techniques in the space of natural language." A central premise of representation learning is that good representations are transferable — a representation learned by one model (or for one task) should be useful for another model (or another task). If c' is genuinely a "natural language representation," then a c' generated by GPT-4o during sleep-time should, in principle, improve test-time accuracy for GPT-4o-mini, or vice versa.

The paper tests no cross-model transfers. In every experiment, the same model family is used for both sleep-time (S(c) → c') and test-time (T_b(q, c') → a). For Stateful GSM-Symbolic, GPT-4o-mini sleep-time is paired with GPT-4o-mini test-time, and GPT-4o sleep-time with GPT-4o test-time. For Stateful AIME, o1 sleep-time is paired with o1 test-time, o3-mini with o3-mini, etc. The SWE-Features case study uses Claude 3.7 Sonnet for both phases.

The consequence. Without cross-model transfer experiments, the characterization of sleep-time compute as "representation learning" remains a metaphor rather than a substantiated claim. A representation that only works within the same model that produced it may simply be the model's own intermediate reasoning externalized — a form of self-consistency rather than a general-purpose transformation of the input. This has practical implications: if a deployment has access to a powerful model during sleep-time (e.g., a large model running on batch hardware overnight) but must use a smaller model at test-time (e.g., an on-device model for low-latency response), can sleep-time compute help? The paper provides no evidence either way. Conversely, if the representation is not transferable, then sleep-time compute's value is bounded by the cost of running the same model in both phases — if the model is expensive at test-time, it is also expensive at sleep-time, and the amortization argument must carry the full weight of the efficiency case.

The absence of cross-model experiments is particularly notable given the paper's own motivation (Section 1): "on-device deployment" is listed as a potential beneficiary of test-time compute strategies. On-device models are typically much smaller than their cloud counterparts (e.g., a 7B parameter model on a phone versus a 405B parameter model in a datacenter). If sleep-time compute could allow a large cloud model to pre-process a context into c' that a small on-device model uses at test-time, this would directly address the on-device deployment use case. The paper does not evaluate this configuration.

What evidence exists in the paper. No experiments test cross-model sleep-time compute. The paper's related work discussion (Section 2) contrasts sleep-time compute with speculative decoding, noting that "the generated tokens are used as an input regardless of the user's actual query" — implying that sleep-time tokens are meant to be useful as context, not as draft tokens that a specific verifier model checks. But this does not establish that the tokens are useful as context for a different model. The PRM training discussion in Snell et al. (2024) found that PRMs trained on one model's output distribution transferred poorly to another model's outputs due to distribution shift — an analogous concern applies here: a c' generated by GPT-4o may reflect reasoning patterns and assumptions specific to GPT-4o that confuse or mislead a different model at test-time.

Mitigation status. Not addressed. The paper does not acknowledge this as a limitation, nor does it suggest cross-model transfer as future work. The representation-learning framing in Section 7 is presented as a conceptual contribution without empirical backing for the transfer property that would distinguish true "representation learning" from simple self-consistency. A practitioner should assume that the reported benefits apply only when the same model (or a very similar model from the same family) is used for both sleep-time and test-time phases, until cross-model experiments demonstrate otherwise.

No Latency Measurements Are Reported, Undermining the Central Motivation

The assumption or constraint. The paper's introduction and motivation ground the case for sleep-time compute in latency reduction: "improved performance from test-time compute comes at a significant increase in latency and cost, waiting potentially several minutes for answers" (Section 1). The core value proposition is that by moving computation to an offline phase, the user experiences a much faster response at test-time. However, the paper reports no latency measurements — no wall-clock times for either sleep-time or test-time phases, no end-to-end latency comparisons between sleep-time compute and standard test-time compute at matched accuracy, and no analysis of how latency scales with the chosen test-time budget or sleep-time scaling factor.

The consequence. The paper's central claim about latency cannot be directly substantiated from the reported results. The evidence is indirect: sleep-time compute achieves the same accuracy with fewer test-time tokens (Figures 3–4), and all else being equal, fewer output tokens means lower latency. But "all else being equal" does not hold — the test-time prompts for sleep-time compute instruct the model to reference the rethink memory block and "not recompute anything that already exists," which could change the relationship between token count and wall-clock time. For example, if the sleep-time test-time prompt leads the model to spend more time per token (e.g., because it must attend to a longer context c' versus the raw c), the latency reduction might be less than the token-count reduction suggests. For reasoning models (o1, o3-mini, R1), the test-time compute budget is controlled through API reasoning effort parameters that do not have a transparent mapping to wall-clock time — higher reasoning effort may increase latency beyond the token-count increase due to internal processing not exposed as output tokens.

The latency claims also interact with the practical deployment model. The paper envisions sleep-time compute running "between interactions with the model while it would otherwise be idle" (Section 1). But "idle" time between user interactions may be measured in seconds (between messages in a conversation) or hours (overnight batch processing). The latency of the sleep-time phase matters: if sleep-time compute takes minutes to run, it cannot be completed between rapid-fire user messages in a conversational setting; the system would need to either delay the user's next query or serve it with a stale c'. The paper provides no guidance on sleep-time latency, making it difficult for practitioners to assess whether the sleep-time window in their application is sufficient to complete the pre-computation before the next query arrives.

What evidence exists in the paper. None. Every figure reports token counts, not wall-clock time. The paper's methodology (Section 4.2) notes that for non-reasoning models, "We use temperature 0 for generation" and controls test-time compute through verbosity prompts, but no latency is reported. For reasoning models, the paper uses API-exposed reasoning effort parameters and "budget forcing" for DeepSeek-R1, but the mapping from these parameters to wall-clock latency is not provided. The cost model in Section 5.3 uses t = 10 based on latency-optimized inference being "roughly 10× more expensive," but this is a pricing assumption, not a latency measurement — it reflects the provider's pricing of low-latency inference, not the actual wall-clock time experienced by the user.

Mitigation status. Not addressed. The paper does not acknowledge this as a limitation, report any latency measurements, or provide guidance on sleep-time latency requirements. A practitioner must infer that token-count reductions translate proportionally to latency reductions, which is a reasonable first approximation but should be validated for the specific models and deployment configurations of interest. For the parallel sleep-time scaling experiments (k = 1, 2, 5, 10 parallel generations), the paper notes that these run in parallel, but does not report whether this parallelism reduces sleep-time wall-clock time compared to sequential generation, or what hardware is assumed for the parallel deployment.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a new temporal dimension to the inference compute allocation problem. Before this work, the field's entire conversation about scaling inference compute—whether through sequential chain-of-thought (OpenAI, 2024; DeepSeek-AI, 2024), parallel best-of-N sampling (Brown et al., 2024), or compute-optimal adaptive strategies (Snell et al., 2024)—operated within a single assumption: all computation happens after the query arrives, when the user is waiting. Test-time scaling research was essentially one-dimensional: how much compute to spend, what method to use, how to allocate it across difficulty levels. Sleep-time compute introduces a second dimension: when the computation occurs relative to query arrival.

This is a reframing rather than a paradigm shift—it does not replace test-time scaling but adds an orthogonal degree of freedom to it. The paper's formalism makes this explicit: standard test-time compute is T_B(q, c) → a; sleep-time compute decomposes this into S(c) → c' followed by T_b(q, c') → a where b << B. The method S is not a new algorithm for reasoning—it is the same test-time scaling techniques (extended chain-of-thought, parallel generation, reasoning effort control) applied at a different time and to a different input (context only, no query). The innovation is architectural, not algorithmic: it restructures when the model does its work, not how it does it.

The magnitude of this reframing becomes clear when considering what it changes about system design. The paper's decomposition creates a natural point of computation reuse: c' can be shared across any number of queries about the same context, enabling the amortization benefits demonstrated in Figure 9. Standard test-time compute has no analogous mechanism—each query pays the full reasoning cost independently, even when the underlying context is identical. This shifts the inference cost structure from linear in query count (each query costs B tokens) to sublinear (a fixed upfront cost plus small per-query cost b). For deployments with high query multiplicity—shared codebases, frequently referenced documents, persistent conversations—this changes the economics of test-time compute from "expensive and latency-bound" to "amortized and potentially latency-free."

The paper also forces a reconciliation of the latency and accuracy tradeoff. Prior work on test-time scaling (Snell et al., 2024; Brown et al., 2024) treated latency as an unavoidable cost of higher accuracy: to get better answers, you must wait longer. Sleep-time compute breaks this coupling for the class of problems where context-level reasoning dominates query-specific reasoning. By shifting the heavy reasoning to an offline phase when no user is waiting, the method preserves the accuracy benefits of extended reasoning while dramatically reducing user-perceived latency. This is not a marginal improvement—it is a qualitative change in the nature of the tradeoff, from "accuracy vs. latency" to "accuracy vs. pre-computation cost," where the pre-computation cost is amortizable and may be zero marginal cost if it uses otherwise-idle GPU cycles.

The paper resolves an implicit tension in the test-time compute literature. Sequential test-time scaling (OpenAI, 2024; DeepSeek-AI, 2024) has shown impressive accuracy gains but faces criticism for making LLMs impractical for interactive use due to multi-minute latency. Parallel test-time scaling (Brown et al., 2024) addresses latency by running samples simultaneously, but requires expensive verification mechanisms (learned verifiers or oracle access) and still consumes substantial compute at query time. Sleep-time compute offers a third path: it achieves the accuracy of sequential scaling with the latency profile closer to single-pass inference, while naturally enabling computation reuse that neither sequential nor parallel scaling supports. The paper's empirical demonstration that sleep-time compute outperforms pass@k with oracle verification (Figures 5, 6) is particularly significant here—it shows that the method beats even the strongest version of parallel scaling while requiring no verification mechanism at test-time, because it improves the model's output distribution (its "proposal distribution," in Snell et al.'s terminology) rather than relying on post-hoc sample selection.

However, the paper's contribution should not be overstated as universal. It applies specifically to stateful settings where a persistent context exists independently of the query—math problems split into context and question, codebases with related historical pull requests, document corpora. For stateless interactions where the entire input arrives at once (e.g., a standalone math problem, a single-turn factual question with no shared background), sleep-time compute offers no benefit because there is no "sleep-time" to exploit—the context and query arrive simultaneously, and the decomposition S(c) → c' followed by T_b(q, c') → a collapses back to T_B(q, c) → a. The paper does not claim otherwise, but the scope of applicability is narrower than the framing might suggest to a casual reader.

The paper also makes specific research directions more or less attractive:

More attractive: Methods for structuring, compressing, and sharing model-generated inferences between queries become high-priority. The paper shows that natural-language intermediate representations can serve as effective "cached reasoning," and improving the quality and transferability of these representations is a natural next step. Techniques from retrieval-augmented generation (RAG) are obvious candidates: rather than a single c' string, a structured database of pre-computed facts about a context could be queried and assembled at test-time based on the specific query, combining the amortization benefits of sleep-time compute with the adaptability of retrieval.
Less attractive: Purely algorithmic improvements to test-time reasoning—more sophisticated search, better verification, more efficient chain-of-thought—remain valuable but are now understood to operate within a paradigm that is fundamentally limited by its assumption of statelessness. The paper's results suggest that for many practical deployments, architectural changes to when computation occurs may yield larger efficiency gains than marginal improvements to how computation is performed. A dollar spent on better context preprocessing infrastructure may return more than a dollar spent on a 5% improvement to a search algorithm.
More attractive: The intersection of inference compute allocation and systems design—pre-fetching, caching, scheduling—becomes recognized as a first-class research area. The paper explicitly connects sleep-time compute to a lineage of systems thinking (caching, pre-fetching, OLAP data cubes) that has been largely absent from the LLM reasoning literature. This opens the door for applying well-understood systems principles (cache invalidation policies, cost-benefit analysis for pre-computation, adaptive resource allocation) to LLM inference pipelines.
Less attractive: The paper's finding that the benefit of sleep-time compute is concentrated in predictable queries (Figure 10) and that the hardest problems see limited gains (in the most unpredictable quintile, the accuracy delta is near zero) means that research focused exclusively on harder problems may find diminishing returns from this approach. The method is most powerful for routing, structured tasks where the query distribution is concentrated—not for open-ended reasoning where every query is novel and unpredictable.

Follow-Up Research This Work Enables

Adaptive gating: predicting query predictability from context alone. The paper's most actionable unresolved question is: can a model predict—before any queries arrive—whether a given context will receive predictable queries that benefit from sleep-time compute? The predictability analysis in Figure 10 uses the actual query (via Llama2-70B log-probability) to bin results, which is circular for deployment. A follow-up study should train a classifier that takes only the context c as input and predicts the expected benefit of sleep-time compute—perhaps using features like context length, presence of enumerable quantities, template-like structure, or an LLM's own estimate of "how many distinct questions could plausibly be asked about this context." The dependent variable would be the accuracy delta from sleep-time compute measured on a held-out set of queries. A negative result (no reliable ex-ante predictor exists) would be equally informative: it would mean sleep-time compute must be applied uniformly, and the wasted compute on unpredictable contexts is an unavoidable tax. A positive result would enable adaptive deployment where sleep-time compute is triggered only when its expected benefit exceeds a threshold, directly addressing the cost-effectiveness concern raised by the N = 1 case in Figure 9. The Multi-Query GSM-Symbolic dataset provides natural training data: each of the 1,095 P1 contexts has 10+ associated queries with varying predictability, and the measured accuracy delta at Verbosity 0 could serve as the ground-truth benefit signal.

Cross-model sleep-time compute: testing the "representation learning" claim. The paper frames c' as a "natural language representation" (Section 7) and draws an analogy to representation learning, but never tests whether c' generated by one model transfers to another. A direct experiment: take the Stateful GSM-Symbolic P1 test set, generate c' using GPT-4o with 5 parallel sleep-time generations (the best configuration from Figure 7), and evaluate test-time accuracy using GPT-4o-mini with the same c'. Compare against (a) GPT-4o-mini test-time compute only (the standard baseline), (b) GPT-4o-mini with c' generated by GPT-4o-mini itself (the same-model sleep-time condition), and (c) GPT-4o with its own c' (the within-model upper bound). If GPT-4o-mini benefits from GPT-4o-generated c' nearly as much as from its own c', the representation-learning framing is validated and the practical implication is significant: powerful models can batch-process contexts offline to accelerate weaker, cheaper, or on-device models at test-time. If transfer fails (GPT-4o-mini performs no better with GPT-4o's c' than with raw context), this reveals that c' encodes model-specific reasoning patterns rather than general context structure—an important negative result that would narrow the scope of sleep-time compute to same-model deployments. The experiment should also test the reverse direction (GPT-4o using GPT-4o-mini's c'), and should measure whether performance degrades gracefully or catastrophically when models are mismatched.

Latency-annotated evaluation with wall-clock measurements. The paper's central motivation—reducing user-perceived latency—is entirely unvalidated by measurement. A follow-up study should replicate the Stateful GSM-Symbolic and Stateful AIME experiments end-to-end, reporting not just token counts but wall-clock latency for both sleep-time and test-time phases, under realistic deployment constraints. Key measurements include: (a) the wall-clock time to generate c' for a single context (both with and without parallel sleep-time generations), which determines whether sleep-time compute can complete between rapid user interactions; (b) the end-to-end latency from query arrival to answer delivery for sleep-time compute versus standard test-time compute at matched accuracy, to validate that the 5× token-count reduction translates to a ~5× latency reduction; (c) latency variance across queries, since a long-tail latency distribution would undermine the user experience even if average latency is low. The study should test on both API-based deployment (where token generation speed is provider-controlled) and self-hosted deployment (where the researcher controls batching and parallelism), since the cost asymmetry t = 10 is specific to API pricing. A natural extension: measure how sleep-time latency varies with the number of parallel generations k and whether the parallelism assumed in the paper (all k generations run simultaneously) is practically achievable on typical GPU configurations. If parallel sleep-time generations cannot actually run in parallel due to memory constraints or sequential attention computation, the reported scaling benefits for k > 1 may not translate to wall-clock improvements.

Structured c' formats beyond single-block natural language. The paper's implementation uses a single rethink_memory block that is iteratively rewritten—a flat string of natural-language inferences. This is the simplest possible representation, and likely leaves efficiency on the table. A follow-up could explore more structured forms of c': (a) a tagged format where inferences are organized by category (e.g., "DERIVED_QUANTITY: total_tennis_balls = 200", "POTENTIAL_QUESTION: 'How many marked indigo tennis balls are there?' ANSWER: 10") that allows the test-time model to selectively attend to relevant portions; (b) a retrieval-based approach where sleep-time compute produces a database of (inference, embedding) pairs, and test-time compute retrieves the top-k most relevant entries based on query embedding similarity, avoiding the context-length issues that likely cause the k = 10 degradation in Figure 7; (c) a multi-turn sleep-time protocol where the model alternately generates inferences and self-critiques them, iteratively refining c' until a stopping criterion is met, analogous to how self-consistency or self-refinement works at test-time but applied to context processing. The Stateful GSM-Symbolic and Multi-Query GSM-Symbolic datasets are ideal for this exploration because the ground-truth answers provide a clean signal for which pre-computed inferences are "correct" (match the quantities needed to answer actual queries) and which are noise (derived quantities that no query uses). A study that measures the precision and recall of c'—what fraction of pre-computed inferences are actually used by downstream queries, and what fraction of query-required inferences were successfully pre-computed—would provide mechanistic insight into why sleep-time compute works and how to improve it.

Sleep-time compute for retrieval-augmented generation (RAG) pipelines. The paper evaluates only on reasoning datasets (math, code) where the context contains all information needed to answer queries. But the most natural real-world deployment of sleep-time compute may be in RAG settings where the "context" is a large corpus of documents, and the "queries" are user questions about those documents. The sleep-time phase could pre-process the entire corpus: summarizing each document, extracting key entities and relationships, pre-computing answers to anticipated questions, and structuring the results in a format optimized for rapid test-time lookup. A follow-up study should construct a Stateful version of a standard RAG benchmark (e.g., Natural Questions, HotpotQA, or a multi-document QA dataset) by treating the document collection as c and individual questions as q_i, then measure: (a) whether sleep-time pre-processing of the corpus (generating document summaries, extracting facts, anticipating questions) reduces the test-time retrieval and reasoning cost; (b) whether amortization benefits materialize when multiple users query the same corpus; (c) how sleep-time compute interacts with the retrieval step—does c' supplement the raw documents, replace them, or both? This experiment would stress-test the assumption that context is "available before queries" in a realistic setting (document corpora are static between updates, satisfying the assumption) while testing whether the method scales to much larger contexts than the paragraph-length math problem statements used in the paper.

Sleep-time compute as a synthetic data generation strategy. The paper briefly mentions in Section 7 that "the output of sleep-time compute itself could serve as a form of synthetic data." This is a promising direction that the paper does not develop. A follow-up could test whether c'—the enriched, inference-laden context—can be used as training data to improve a model's ability to answer queries about similar contexts without requiring sleep-time compute at test-time. Concretely: take the Multi-Query GSM-Symbolic training set, generate c' for each context using a strong model (e.g., GPT-4o with k = 5 parallel sleep-time generations), and fine-tune a smaller model (e.g., GPT-4o-mini) on pairs of (c', q) → answer where the answer is the ground-truth. Then evaluate the fine-tuned model on the test set (a) with c' provided at test-time (measuring whether the model learns to better utilize pre-computed inferences) and (b) with only the raw c provided at test-time (measuring whether training on c' induces the model to internally generate similar inferences even without explicit pre-computation, i.e., whether the model learns to "think" like the sleep-time processor). This connects sleep-time compute to the literature on distillation and synthetic data generation (Bansal et al., 2024; DeepSeek-AI et al., 2025), testing whether the enriched context representations can be "compiled" into model weights rather than explicitly provided at inference time.

Practical Applications and Downstream Use Cases

Document Q&A over static corpora. The most direct application is question-answering over a fixed document collection—product documentation, legal contracts, scientific papers, internal wikis. In this setting, the document corpus is the context c, which changes infrequently (e.g., when documentation is updated). During low-traffic periods (overnight, weekends), sleep-time compute pre-processes each document: extracting key facts, anticipating common questions, computing intermediate quantities, and structuring the enriched context c' for rapid retrieval. When user queries arrive during business hours, the system retrieves the relevant pre-processed documents and answers using the enriched context with minimal test-time reasoning. The amortization benefit is maximal here because the same documents are queried by many users—the paper's Figure 9 shows that at 10 queries per context, the cost per query drops by up to 2.5× compared to single-query sleep-time compute. For a documentation site serving thousands of queries daily, the total cost reduction could be substantial. The system could also tier its approach: use sleep-time compute for all documents, but apply additional test-time compute (higher verbosity or reasoning effort) only for queries that the pre-computed context cannot answer confidently, detected through a lightweight confidence score.

On-device coding assistants with cloud pre-processing. A coding assistant running on a developer's laptop (e.g., in an IDE) faces strict latency constraints—the developer expects sub-second responses after asking a question or requesting a code modification. However, the codebase is available and static between editing sessions. A practical architecture: during "sleep-time" (overnight, or when the developer is not actively editing), a cloud-based reasoning model (e.g., o3-mini or Claude 3.7 Sonnet) processes the repository, exploring its structure, identifying architectural patterns, pre-computing dependency graphs, and generating summaries of key modules. The enriched context c' is stored locally. When the developer asks a question at test-time, a small on-device model (e.g., a 7B parameter model running locally) answers using c' with minimal test-time reasoning, achieving accuracy comparable to a much larger model running extended chain-of-thought in the cloud. This architecture delivers the accuracy benefits of powerful models with the latency of local inference, and the paper's findings directly support it: sleep-time compute achieves comparable accuracy to standard test-time compute with ~5× fewer test-time tokens (Figure 3), and the pass@k outperformance (Figure 5) suggests the enriched context genuinely improves the model's output distribution. The key unknown is whether cross-model sleep-time compute works (see Follow-Up Research above)—if GPT-4o's c' transfers to an on-device Llama model, this application is immediately practical; if not, the cloud model would need to generate c' using a smaller model's reasoning patterns, which may reduce the quality of pre-computed inferences.

Multi-turn conversational agents with persistent memory. In a long-running conversation (customer support, tutoring, therapy), the dialogue history serves as the context c, and each user utterance is a new query q. Between user messages—whether seconds (while the user types) or hours (between sessions)—the model can pre-process the conversation history: summarizing key points, inferring the user's goals or emotional state, anticipating likely follow-up questions, and structuring this enriched context for rapid retrieval when the next message arrives. The paper's finding that sleep-time compute is most effective when queries are predictable (Figure 10) maps naturally onto structured conversations: in customer support, the set of possible next questions after a given exchange is often limited (clarifying a policy, asking for a refund, escalating to a manager), and pre-computing answers to these anticipated questions can dramatically reduce response latency. The amortization benefit applies across turns: the same conversation history is queried multiple times (once per user message), so the sleep-time cost per turn decreases as the conversation lengthens. The challenge—which the paper identifies but does not solve—is that the context evolves with each turn (new messages are added), requiring incremental updates to c' rather than full regeneration. A practical system might use a sliding window: regenerate c' from scratch every N turns, and for intermediate turns, append new messages to c' with a lightweight "incorporate this new information" prompt rather than full re-processing.

Batch inference pipelines for evaluation and data generation. Organizations that run large-scale batch inference—evaluating models on benchmarks, generating training data via rejection sampling, or scoring candidate solutions—often face a tradeoff between accuracy (using extended chain-of-thought or best-of-N) and throughput (minimizing tokens per example). Sleep-time compute offers a way to shift this tradeoff: pre-process all contexts in the batch during a low-priority "sleep-time" phase (e.g., overnight on spare GPU capacity), then run the test-time phase at high throughput with minimal per-example reasoning. For a benchmark like MATH (12,500 problems), pre-processing each problem statement as c' overnight and then evaluating with Verbosity 0 at test-time could achieve accuracy comparable to Verbosity 3 evaluation without pre-processing, reducing evaluation cost by ~3–5× per the token-count reductions in Figure 3. The amortization analysis (Figure 9) shows that this is most cost-effective when the sleep-time compute is discounted relative to test-time compute—which is exactly the case when sleep-time uses spare overnight capacity at lower priority/pricing, and test-time uses peak-demand capacity.

Dependency tree

Papers this one builds on, walking backwards through the same domain and shared topics.

Hogwild! Inference: Parallel LLM Generation via Concurrent Attention2025· inference-optimization← in-domain predecessor
- Break the Sequential Dependency of LLM Inference Using LOOKAHEAD DECODING2024· inference-optimization← LLM-judged similar
- Quantization for OpenAI's Whisper Models: A Comparative Analysis2025· inference-optimization← in-domain predecessor
- Star Attention: Efficient LLM Inference over Long Sequences2024· inference-optimization← similar topic
Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters2024· inference-optimization← similar topic
- An Efficient Inference Framework for Early-exit Large Language Models2024· inference-optimization← in-domain predecessor
- Can 1B LLM Surpass 405B LLM? Rethinking Compute-Optimal Test-Time Scaling2025· inference-optimization← similar topic

What it influenced

Later papers in the same chain. Direction reversed.

SageAttention3: Microscaling FP4 Attention for Inference and An Exploration of 8-Bit Training2025· inference-optimization→ in-domain predecessor
- Bridging the Gap Between Promise and Performance for Microscaling FP4 Quantization2025· inference-optimization→ LLM-judged similar
- Sparse VideoGen2: Accelerate Video Generation with Sparse Attention via Semantic-Aware Permutation2025· inference-optimization→ in-domain predecessor
- An Empirical Study of Qwen3 Quantization2025· low-precision→ topic thread
- Prompt Cache: Modular Attention Reuse for Low-Latency Inference2023· inference-optimization→ similar topic
- SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Acceleration2024· inference-optimization→ similar topic
- Full Attention Strikes Back: Transferring Full Attention into Sparse within Hundred Training Steps2026· inference-optimization→ similar topic
Distributed Speculative Inference (DSI): Speculation Parallelism for Provably Faster Lossless Language Model Inference2024· inference-optimization→ similar topic
- A Simple and Effective $L_2$ Norm-Based Strategy for KV Cache Compression2024· inference-optimization→ in-domain predecessor
- Fast Inference from Transformers via Speculative Decoding2022· inference-optimization→ similar topic
- Accelerating LLM Inference with Staged Speculative Decoding2023· inference-optimization→ similar topic
- Cascade Speculative Drafting for Even Faster LLM Inference2023· inference-optimization→ similar topic
Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters2024· inference-optimization→ similar topic
- Can 1B LLM Surpass 405B LLM? Rethinking Compute-Optimal Test-Time Scaling2025· inference-optimization→ LLM-judged similar
- When to Ensemble: Identifying Token-Level Points for Stable and Fast LLM Ensembling2025· llm-systems→ LLM-judged similar
- CyberV: Cybernetics for Test-time Scaling in Video Understanding2025· vision→ LLM-judged similar
- Recurrent-Depth VLA: Implicit Test-Time Compute Scaling of Vision-Language-Action Models via Latent Iterative Reasoning2026· architecture→ LLM-judged similar
- From Scale to Speed: Adaptive Test-Time Scaling for Image Editing2026· diffusion→ LLM-judged similar
- Train-to-Test (T2) Scaling Laws: Integrating Pretraining and Test-Time Compute2026· scaling-laws→ LLM-judged similar
- Writing in the Margins: Better Inference Pattern for Long Context Retrieval2024· inference-optimization→ in-domain predecessor