Skeleton-of-Thought: Large Language Models Can Do Parallel Decoding

ArXiv: 2307.15337

🎯 Pitch

Current LLMs are slowed down by their sequential decoding approach, but this paper shows you can make them answer up to 2.39× faster by first generating a skeleton outline and then expanding each point in parallel. Beyond speed gains, this structured prompting surprisingly improves answer quality across multiple LLMs like GPT-4 and ChatGPT for decomposable questions—planning the answer structure in language is both faster and better.

1. Executive Summary

This paper proposes Skeleton-of-Thought (SoT), a prompting strategy that reduces end-to-end generation latency of LLMs by eliciting an answer skeleton first, then expanding each skeleton point in parallel through batched decoding or parallel API calls. Evaluated across 12 LLMs—including Vicuna, LLaMA2-Chat, Claude, and GPT-4—on the Vicuna-80 and WizardLM assistant-style datasets, SoT achieves speed-ups up to 2.39× while maintaining or improving answer quality on question categories whose answers can be decomposed into independently expandable points (e.g., knowledge, generic, and common-sense questions). The paper further introduces SoT with router (SoT-R), which adaptively falls back to normal sequential decoding for questions unsuitable for parallel expansion—such as math or coding problems requiring step-by-step reasoning—establishing that structured parallel generation improves both efficiency and quality only when the answer's constituent points lack strong sequential dependencies.

2. Context and Motivation

The Core Problem: Sequential Decoding Is a Structural Bottleneck

The fundamental problem this paper tackles is the end-to-end generation latency of large language models (LLMs). Despite the remarkable capabilities of modern LLMs, their inference process remains slow, and the paper identifies three contributing factors (Section 1):

1. Large model size demanding substantial memory, memory access, and computation — for example, the FP16 weights of a 175B GPT-3 model alone require 350GB of GPU memory.
2. The attention mechanism in transformer architectures, which is I/O-bound and has quadratic complexity in sequence length.
3. The sequential decoding approach — the practice of generating tokens one by one, where the generation of the $k+1$-th token cannot begin until the $k$-th token is produced.

While the first two axes — model size and attention efficiency — have attracted extensive research, the paper argues that the third axis, the sequential decoding assumption itself, has been treated as an inherent constraint rather than a design choice that can be questioned. The paper asks: do LLMs actually have to perform fully sequential decoding?

This matters for both practical and conceptual reasons. On the practical side, slow inference directly degrades user experience in interactive applications. The paper provides concrete examples: answering a single question takes 22 seconds for Claude (accessed via Slack API) and 43 seconds for Vicuna-33B V1.3 running locally on an NVIDIA A100 GPU (Section 1). For applications where users expect near-instantaneous responses — chatbots, coding assistants, or agent-to-agent interactions — latencies in the tens of seconds are prohibitive. On the conceptual side, the sequential decoding assumption fundamentally limits the achievable inference speed for any given hardware configuration because the decoding phase is bottlenecked by memory I/O of model weights rather than by computation. The paper reports that when decoding one token at a time on an NVIDIA A100 GPU, the actual computing performance is only 0.31 TFLOPS — a mere 0.1% of the GPU's peak FP16 performance (Appendix E, Table 5).

Where Existing Approaches Fall Short

The paper locates its contribution relative to three classes of existing work, arguing that each addresses the efficiency problem from a different level of abstraction but none questions the sequential decoding assumption at the level of how the LLM organizes its own output content.

Model-level optimization (compression, architecture redesign). Prior work addresses model size and attention bottlenecks through techniques including quantization (Xiao et al., 2022; Frantar et al., 2022; Lin et al., 2023), weight and activation sparsification (Mishra et al., 2021; Wang et al., 2021), efficient attention designs like multi-query attention (Shazeer, 2019) and linear-complexity attention (Kitaev et al., 2020; Wang et al., 2020), and mixture-of-experts architectures (Lepikhin et al., 2021; Fedus et al., 2022). These approaches either require expensive retraining or, in the case of compression methods, fine-tuning to recover quality after reducing model complexity. Critically, they leave the sequential token-by-token generation paradigm untouched — they make each sequential step faster but cannot parallelize token generation across different parts of the answer.

System-level optimization (computational graph optimization, batching, scheduling). Serving systems improve throughput by packing multiple queries into batches (Fang et al., 2021), optimizing memory management (Kwon et al., 2023), and exploiting model parallelism across devices (Rajbhandari et al., 2020; Zheng et al., 2022). Techniques like FlashAttention (Dao et al., 2022) fuse operations to reduce off-chip I/O during the prefilling phase. However, the paper notes a crucial distinction: these methods primarily target throughput (queries per second across many users), not end-to-end latency for a single user. The decoding phase remains a sequential bottleneck per query. As the paper states in Appendix E, this phase is "heavily bottlenecked by the I/O of weights" — all 7B+ parameters must be loaded from HBM to the GPU chip to produce each single token — so system-level batching helps throughput but does not fundamentally reduce the wall-clock time a single user waits for a response.

Decoding-level optimization (speculative decoding, non-autoregressive generation). These are the closest in motivation to SoT, as they directly address the sequential decoding bottleneck. Speculative decoding methods (Stern et al., 2018; Leviathan et al., 2022; Chen et al., 2023a) use a smaller, cheaper assistant model to generate candidate token sequences sequentially, then have the target LLM verify these candidates in parallel, accepting the longest prefix that matches. Non-autoregressive generation (NAG) methods (Gu et al., 2018; Xiao et al., 2023) sample and refine consecutive tokens in parallel, typically requiring specially designed models and training schemes. The paper identifies a fundamental difference: these methods rely on either external assisting models or modified model architectures, and they operate at the level of consecutive tokens within a continuous output sequence. They do not exploit the semantic structure of the content itself to identify naturally parallelizable segments. As the paper puts it (Appendix D.1), "SoT prompts the LLM itself to plan the contents in a way that permits the parallel generation of tokens in different segments, by exploiting the emerging instruction-following and planning ability of LLMs."

A Missing Paradigm: Data-Centric Optimization for Efficiency

The paper identifies a gap in the research landscape: while data-centric techniques for improving answer quality are gaining traction — including prompting methods like Chain-of-Thought (Wei et al., 2022; Kojima et al., 2022), Tree-of-Thoughts (Yao et al., 2023), and multi-modality task orchestration (Shen et al., 2023) — the potential of data-centric techniques for improving inference efficiency remains entirely unexplored. All existing work on efficient LLM inference operates at the model level (changing the architecture or weights), system level (changing how computation is scheduled on hardware), or decoding level (changing how tokens are sampled and verified). No prior work asks: can we guide the LLM to structure its output in a way that makes inference inherently faster?

This gap is becoming increasingly important because the capabilities of LLMs are evolving in ways that make such guidance feasible. Modern instruction-tuned LLMs demonstrate strong planning and instruction-following abilities — they can be prompted to produce structured outputs (e.g., JSON, bullet points, step-by-step reasoning), decompose complex tasks, and even orchestrate multi-modality workflows. The paper's bet is that these emerging capabilities can be repurposed for efficiency: an LLM that can be prompted to produce a high-quality structured answer can also be prompted to produce that answer in a format that permits parallel processing. As the paper states in Section 1, "This novel perspective is becoming feasible and is expected to grow in importance, owing to the evolving capabilities of state-of-the-art LLMs."

How This Paper Positions Itself

The paper positions SoT as the first attempt at data-centric efficiency optimization — a new "content co-organization for efficiency" paradigm (Appendix D.3) that sits alongside, and can synergize with, existing model-level and system-level techniques. Rather than competing with quantization or FlashAttention, SoT offers an orthogonal lever: it trades off some throughput and token overhead for reduced end-to-end latency, and the paper explicitly notes that system-level batching techniques can be applied within SoT's point-expanding stage, creating opportunities for better latency-throughput tradeoffs in future serving systems (Section 5).

The paper's grounding intuition draws an analogy to human cognition (Section 1): when humans answer complex questions, we typically do not compose responses purely sequentially. Instead, for many question types — consultancy, essay writing, test-taking — we first derive a skeleton or outline according to organizational protocols, then flesh out each point with evidence and details. This process permits parallel work on independent points. The paper asks whether LLMs, with their instruction-following capabilities, can be guided to adopt this same strategy.

A key design choice that distinguishes SoT from prior hierarchical text generation work (Li et al., 2015; Shao et al., 2019; Fan et al., 2018) is that SoT uses explicit, free-form language planning rather than implicit latent representations or closed-form planning modules. Prior hierarchical models trained dedicated modules to reorder input keywords or maintain sentence-level latent variables, requiring custom architectures and training procedures. SoT instead exploits the LLM's own emerging planning capability via prompting — the skeleton is produced as natural language by the same model, and the parallel expansion is done by the same model conditioned on that skeleton. This means SoT can be applied to any off-the-shelf instruction-tuned LLM without weight modification, fine-tuning, or access to training data — a practical advantage that broadens its applicability to proprietary API-based models like GPT-4 and Claude (Section 5).

The paper also introduces the router extension (SoT-R) as a necessary bridge to practical deployment. The core SoT approach only works well for questions whose answers can be decomposed into independently expandable points — roughly five of the nine categories in Vicuna-80 (knowledge, generic, common-sense, roleplay, counterfactual). For math, coding, and fermi problems requiring step-by-step reasoning where later steps depend on earlier results, parallel expansion is fundamentally inappropriate. The router — implemented as either a prompted GPT-4 classifier or a fine-tuned RoBERTa model trained on labeled question-answer data — determines whether to trigger SoT or fall back to normal sequential decoding, making the overall system general across question types.

What SoT Does NOT Attempt

It is important to clarify the scope: SoT does not aim to improve answer quality as its primary goal, though the paper finds it sometimes does so (Section 3.2). SoT does not modify model weights, require access to training data or gradients, or depend on specialized hardware. It does not reduce the total computational FLOPs required to generate an answer — in fact, it often increases them due to duplicated prompts and parallel processing overhead (Appendix H). The speed-up comes entirely from converting serial latency into parallel execution: rather than generating $N$ tokens sequentially, SoT generates $B$ parallel segments each of length roughly $N/B$, completing all segments in the time of the longest one plus the skeleton stage overhead. This is a latency-throughput tradeoff, and the paper is transparent about scenarios where it is and is not appropriate.

3. Technical Approach

3.1 Reader Orientation

This paper introduces Skeleton-of-Thought (SoT), a prompting strategy that makes large language models (LLMs) generate answers faster by structuring their output into independently expandable parts before writing the full response. The core problem is that standard LLM decoding generates tokens one-by-one, creating a serial bottleneck — SoT solves this by having the LLM first produce a concise answer outline (the "skeleton"), then expand each outline point in parallel using batched decoding or simultaneous API calls, reducing wall-clock latency by roughly a factor of 2 on suitable questions without modifying model weights, architecture, or hardware.

3.2 Big-Picture Architecture (Diagram in Words)

The SoT system has three major stages connected sequentially, with parallel execution in the middle stage:

1. Skeleton Stage (sequential): A specially designed prompt instructs the LLM to produce only the skeleton of the answer — a numbered list of short phrases (3–5 words each, 3–10 points total) that capture the major dimensions of the response. This stage produces $B$ skeleton points from a single inference call.

2. Point-Expanding Stage (parallel): For each of the $B$ skeleton points, a second prompt template is filled with the original question, the complete skeleton, and the specific point's index and text. All $B$ requests are processed simultaneously — either as parallel API calls for proprietary models or as a single batched inference for locally run open-source models. Each request generates a short expansion (1–2 sentences) for its assigned point.

3. Aggregation Stage (zero-compute): The $B$ point expansions are concatenated in order (point 1, point 2, ..., point B) with their skeleton headers to form the final answer. No additional model inference is required.

Information flows strictly forward: question → skeleton request → skeleton response → $B$ point-expanding requests → $B$ point-expanding responses → concatenated final answer. The key parallelism is that all $B$ point-expanding inferences run concurrently, with total latency determined by the skeleton stage plus the slowest point expansion rather than the sum of all expansion times.

3.3 Roadmap for the Deep Dive

• First, the two prompt templates — skeleton and point-expanding — because they are the core mechanism that elicits structured, parallelizable output from an unmodified LLM.
• Second, the parallel execution mechanics — how batched decoding achieves speed-ups on open-source models and how parallel API calls work for proprietary models — because the speed-up depends entirely on the hardware efficiency properties of the decoding phase.
• Third, the SoT-R router extension — the prompting-based router (using GPT-4 as a classifier) and the trained router (a fine-tuned RoBERTa model) — because the base SoT method is only suitable for a subset of question types, and the router is what makes the system practical across arbitrary user queries.
• Fourth, the efficiency profiling methodology — how latencies are measured or estimated for both open-source and API-based models — because the speed-up claims depend on a specific, reproducible measurement protocol.
• Fifth, the answer quality evaluation framework — the FastChat and LLMZoo metrics, the GPT-4 judge, and the net win rate metric — because the paper claims that SoT not only accelerates generation but also sometimes improves answer quality, and this claim requires careful evaluation machinery.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a prompt engineering and systems paper whose core idea is that the sequential decoding assumption can be circumvented by eliciting answer structure from the LLM itself, then exploiting that structure for parallel generation. The technical contribution spans prompt design, parallel execution strategies, adaptive triggering via a router, and a careful measurement methodology.

Skeleton Prompt Template

The skeleton prompt is the first inference call in the SoT pipeline and is responsible for extracting a concise, structured outline of the answer from the LLM. The template appears as Prompt 1 (for GPT-4, without demonstrations) and Prompt 3 (for all other models, with two-shot demonstrations) in the paper.

Template structure (Prompt 1, zero-shot version for GPT-4):

[User:] You're an organizer responsible for only giving the skeleton
(not the full content) for answering the question.
Provide the skeleton in a list of points (numbered 1., 2., 3., etc.)
to answer the question. Instead of writing a full sentence, each
skeleton point should be very short with only 3∼5 words. Generally,
the skeleton should have 3∼10 points. Now, please provide the
skeleton for the following question.
{question}
Skeleton:
[Assistant:] 1.

What this prompt does: It establishes a role ("organizer"), constrains the output format (numbered list, 3–5 words per point, 3–10 points total), and provides a partial answer ("1.") for the LLM to continue. The partial answer is a critical implementation detail — by starting the assistant's response with "1.", the prompt forces the model into autoregressively continuing in the desired numbered-list format, which makes downstream parsing reliable.

Two-shot variant (Prompt 3): For all models except GPT-4, the template prepends two demonstration examples before the target question. The first demonstration asks "What are the typical types of Chinese dishes?" and provides a skeleton with 8 points (Dumplings, Noodles, Dim Sum, etc.). The second asks "What are some practical tips for individuals to reduce their carbon emissions?" and provides a 6-point skeleton (Energy conservation, Efficient transportation, etc.). These demonstrations serve as in-context examples that teach the model the desired output format — short noun phrases without elaboration, strictly following the numbered format. The paper notes that GPT-4 can work without these demonstrations, implying stronger zero-shot instruction-following capability.

Skeleton extraction: After the LLM generates the skeleton response, a regular expression (\d+)\.\s?([\s\S]+?)(?=\n|\n*$) extracts the point index and point skeleton text for each line. The paper states this works "in most cases" (Appendix B.1), indicating occasional parsing failures where the model deviates from the expected format.

Design rationale for the skeleton format: The 3–5 word constraint and 3–10 point range are chosen to balance two competing objectives. Shorter skeletons (fewer words per point) make extraction more reliable and reduce the skeleton stage latency, but risk being too vague for the point-expanding stage to generate useful content. More points provide finer granularity for parallelization (more parallelism = more potential speed-up), but each point gets a shorter expansion and the overhead of the skeleton prompt template (which includes the full skeleton in every point-expanding request) grows with $B$. The paper does not formalize this tradeoff mathematically but reports empirical averages: across all models on Vicuna-80, the average number of points $B$ ranges from 4.4 (LLaMA2-Chat-13B) to 9.7 (StableVicuna-13B), with an overall average of 6.8 (Figure 11a).

Point-Expanding Prompt Template

The point-expanding prompt is used $B$ times — once per skeleton point — and is responsible for generating the detailed content for each point without elaboration on other points. The template appears as Prompt 2:

[User:] You're responsible for continuing the writing of one and
only one point in the overall answer to the following question.
{question}
The skeleton of the answer is
{skeleton}
Continue and only continue the writing of point {point index}.
Write it **very shortly** in 1∼2 sentence and
do not continue with other points!
[Assistant:] {point index}. {point skeleton}

Template parameters: Four slots are filled per point:

• {question}: the original user question (identical across all $B$ requests).
• {skeleton}: the complete skeleton text from stage 1 (identical across all $B$ requests — this is what causes the prefilling token overhead discussed below).
• {point index}: the integer index of the current point (e.g., "1", "2").
• {point skeleton}: the short phrase extracted for this point from the skeleton response.

Partial answer mechanism: The assistant prefix {point index}. {point skeleton} repeats the skeleton point and forces the model to continue from that text. This is functionally similar to the skeleton stage's "1." partial answer — it constrains the output format by making the model autoregressively complete the point expansion rather than generating a preamble or restating the skeleton. The paper notes two implementation variants: for open-source models, the partial answer is placed at the end of the prompt string where the model continues writing; for API-based models (ChatGPT-3.5, Claude, GPT-4), the partial answer is either provided as the last assistant message in the chat history (ChatGPT-3.5) or as an explicit instruction to "start your answer from..." (Claude and GPT-4) since API constraints differ.

Conciseness instruction: The phrase "Write it very shortly in 1∼2 sentence" is critical for achieving the speed-up. If the model generates long point expansions, the longest expansion (which determines the parallel stage latency) approaches the length of a normal sequential answer, eliminating the speed-up. The paper reports that Claude and GPT-4 follow this instruction well enough that the "very shortly" emphasis is actually removed from their prompts to avoid overly terse answers (Appendix B.1). Conversely, StableVicuna-13B "fails to adhere to the 'Write it very shortly' instruction" (Section 3.1.1), with its longest point-expanding responses being "as lengthy as the overall normal answer," explaining why SoT achieves only a 1.13× speed-up on this model compared to >2× on most others.

System message handling: The paper notes that system messages are not included for open-source models except LLaMA2, which uses a specific conversation template. For other models (Vicuna, UltraLM, OpenChat), the prompts use the model-specific conversation templates from the FastChat codebase, which handle the formatting of user/assistant message separators.

Parallel Point-Expanding: Batched Decoding for Open-Source Models

The speed-up for locally run open-source models comes from processing all $B$ point-expanding requests as a single batch during the decoding phase. Understanding why this works requires understanding the two-phase structure of LLM inference and the bottleneck characteristics of each phase.

The two phases of LLM inference (Appendix E):

1. Prefilling phase: The input prompt is processed to generate the key-value (KV) cache — intermediate representations that will be reused during token generation. This phase is compute-bound: the model performs matrix multiplications over the entire input sequence at once, achieving relatively high GPU utilization. The paper reports 43 TFLOPS (13.8% of peak) for LLaMA-7B during prefilling with a 128-token input.

2. Decoding phase: Tokens are generated one-by-one autoregressively. For each new token, the model must load all its weights from GPU high-bandwidth memory (HBM) onto the compute units, perform a relatively tiny amount of computation (one token's worth of matrix multiplications), and store the result. This phase is memory-I/O-bound, not compute-bound: the paper reports only 0.31 TFLOPS (0.1% of peak) for LLaMA-7B during decoding — a 139× drop from prefilling utilization (Table 5).

Why batching doesn't increase per-token latency much: When decoding with batch size $B > 1$, the model still loads the same weights from HBM once, but now applies them to $B$ independent sequences simultaneously. Since weight loading dominates the latency and the weights are the same regardless of batch size, the per-token latency increases only slightly with $B$ (Figure 10a shows sub-linear growth: for LLaMA-33B, latency goes from ~5500ms at B=1 to ~6000ms at B=9). The GPU utilization increases roughly linearly with $B$ (Figure 10b) because the same weight I/O cost is amortized over more computation. This means that if we can decode $B$ segments in parallel, we get roughly $B \times$ more tokens generated per unit time compared to sequential decoding.

The speed-up arithmetic: For a final answer of length $N$ tokens, if SoT cuts it into $B$ segments each of length roughly $N/B$, and decodes them as a batch, the decoding time is approximately:

$T_{\text{SoT}} \approx T_{\text{skeleton}} + T_{\text{prefill\_batch}}(B) + T_{\text{decode\_batch}}(B, N/B)$

where $T_{\text{skeleton}}$ is the skeleton stage latency (prefill + decode for one short response), $T_{\text{prefill\_batch}}(B)$ is the cost of prefilling $B$ point-expanding requests simultaneously, and $T_{\text{decode\_batch}}(B, N/B)$ is the cost of decoding $N/B$ tokens for each of $B$ sequences in the batch. The paper notes that $T_{\text{decode\_batch}}(B, N/B) \approx T_{\text{decode\_single}}(1, N/B)$ — decoding $N/B$ tokens with batch size $B$ costs roughly the same as decoding $N/B$ tokens with batch size 1. The sequential baseline would cost $T_{\text{decode\_single}}(1, N)$. Therefore, the ideal speed-up approaches $B \times$, reduced by the skeleton overhead and the imbalance between point lengths.

Imbalance reduces practical speed-up: If the $B$ point expansions have unequal lengths, the batching is bottlenecked by the longest one — shorter expansions finish early and their computation slots go idle. The paper measures the standard deviation of point expansion lengths as an "imbalance degree" (Figure 11e). LLaMA2 and API-based models generate more balanced expansions (lower standard deviation), while some open-source models show high variance. This imbalance explains why the actual speed-up (e.g., 2.39× for LLaMA2-Chat-7B) is substantially less than the average number of points (e.g., 6.4 for LLaMA2-Chat-7B).

Memory overhead: The paper reports that peak memory overhead from batching is modest because model weights dominate the memory footprint (Figure 10c). For LLaMA-33B, peak memory grows from ~65GB at B=1 to ~70GB at B=9 — roughly a 7.7% increase. This means SoT can be run on a single GPU without additional memory optimization techniques for all tested models.

Common prefix optimization: The paper implements a simple trick to reduce the prefilling overhead: since all $B$ point-expanding requests share the same question and skeleton text, the common prefix can be prefilled once with batch size 1, and only the point-specific suffixes (the point index and point skeleton) need to be processed with batch size $B$ (Appendix H, Table 7). This reduces the prefilling token overhead ratio from 30–38× (naive) to a smaller value, though exact after-trick ratios are not reported.

Parallel Point-Expanding: API Calls for Proprietary Models

For API-based models (Claude, ChatGPT-3.5, GPT-4), parallelization is achieved by issuing multiple simultaneous API calls — one for the skeleton stage and $B$ concurrent calls for the point-expanding stage. The latency is computed as:

$T_{\text{SoT\_API}} = T_{\text{skeleton\_call}} + \max_{b=1,\ldots,B} T_{\text{point\_call\_b}}$

where $T_{\text{skeleton\_call}}$ is the end-to-end latency of the single skeleton API request and $T_{\text{point\_call\_b}}$ is the latency of the $b$-th point-expanding API request. The paper records latencies using Python's time.time() before and after each API call.

Tradeoffs: API-based parallelization achieves latency reduction at the cost of increased total API requests ($1 + B$ versus 1) and increased total tokens processed (the skeleton and question are repeated in each of the $B$ point-expanding requests). The paper reports substantial prefilling token overhead: for GPT-4, the ratio of total prefilling tokens (SoT) to normal prefilling tokens is 89.20× (Table 6). This matters because many API services charge per token — SoT may increase monetary cost even as it reduces wall-clock time. The paper notes this as a concern and suggests that prompt tuning techniques could design shorter SoT prompts in the future (Section 6).

SoT-R Router: Adaptive Triggering

The base SoT method is only suitable for questions whose answers can be decomposed into independently expandable points. For questions requiring step-by-step reasoning (math, coding, fermi problems) or continuous prose (some writing tasks), parallel expansion either fails (because later points depend on earlier ones) or degrades quality (because the list format is inappropriate). SoT-R adds a router module before the SoT pipeline that decides whether to trigger SoT or fall back to normal sequential decoding (Section 4).

Router as a binary classifier: The router takes a question $q$ as input and outputs a binary decision: "suitable for SoT" (positive) or "not suitable" (negative). If positive, the system proceeds with the full SoT pipeline (skeleton → parallel point expansion → aggregation). If negative, the system falls back to standard sequential generation (the question is fed directly to the LLM with no skeleton or point-expanding prompts).

Implementation 1: Prompting router (Section 4.1). This router uses GPT-4 itself as the classifier via a structured prompt (Prompt 4). The prompt presents the question and asks whether the desired answer is:

• Option A: "Organize the answer as a list of points or perspectives... and the points or perspectives can be answered independently without referring to the contents of the previous points."
• Option B: "Organize the answer as a list of points or perspectives... and the contents of later points or perspectives cannot be answered independently without referring to the contents of the previous ones."
• Option C: "Do not organize the answer as a list of points or perspectives."

If the response is "A", SoT is triggered; if "B" or "C", normal decoding is used. The prompt explicitly instructs "Just say A, B, or C. Do not explain. Do not provide an answer to the question." to keep latency and token cost minimal. GPT-4 is chosen for this role "given its strong capability" (Section 4.1). No training data or weight modification is required.

Implementation 2: Trained router (Section 4.2). This router is a fine-tuned RoBERTa-base model (120M parameters) that performs sequence classification on the question text. The training pipeline has three steps:

1. Annotation: The LIMA dataset (Zhou et al., 2023), containing 1,030 Q&A pairs from Stack Exchange, wikiHow, and Reddit, is annotated for SoT suitability. Two of the paper's authors independently label each question as positive (can be answered with SoT) or negative (requires sequential generation), using GPT-4's analysis of the question and answer to inform their decision, then discuss inconsistencies. A positive label requires that (a) the answer contains a list of independently expandable points, and (b) each point provides sufficient detail that the point-expanding stage would achieve a speed-up (i.e., expansions aren't trivially short).

2. Training: The RoBERTa model is fine-tuned on the annotated LIMA data using the AdamW optimizer with weight decay 0.01. The learning rate warms up during the first 1% of iterations to $5 \times 10^{-5}$ and then decays linearly. Training runs for 2 epochs with batch size 32. Input sequences are padded or truncated to 512 tokens.

3. Loss function: The paper uses Tversky loss with parameters $\alpha = 0.7$ and $\beta = 0.3$, which penalizes false positives (incorrectly triggering SoT when it would degrade quality) more heavily than false negatives (missing a SoT opportunity, which only loses a speed-up). Label smoothing with factor $\epsilon = 0.2$ is also applied. The entire fine-tuning process takes 2 minutes on an NVIDIA A100 GPU.

Router overhead: The paper reports that the prompting router has an average latency of 0.65 seconds on Vicuna-80 (range: 0.39s–1.37s) and 0.80 seconds on WizardLM (range: 0.36s–2.22s). The trained router has an average latency of 0.04 seconds on Vicuna-80 (range: 0.008s–1.55s) and 0.03 seconds on WizardLM (range: 0.009s–2.52s) (Appendix G.2). The trained router is substantially faster on average because RoBERTa-base is a much smaller model than GPT-4, though it can occasionally be slower due to hardware variability or queuing effects.

Router consistency analysis (Appendix C.3): On Vicuna-80, all three routers (human, GPT-4 prompting, RoBERTa trained) show "notable consistency" (Table 3). The GPT-4 router achieves 0 false positives and 5 false negatives relative to human annotations; the trained router has 6 false positives and 5 false negatives. On WizardLM, discrepancies increase: the GPT-4 router produces 66 false positives (triggering SoT when it shouldn't), while the trained router produces only 25 false positives, showing better alignment with human annotations (Table 4).

Concurrent execution proposal (Appendix C.4): The paper sketches but does not implement an optimization where the router, normal generation, and SoT generation all start concurrently, with the rejected path aborted once the router decides. This would reduce latency further at the cost of wasted token generation on the aborted path.

Efficiency Profiling Methodology

The paper uses different latency measurement approaches for open-source and API-based models.

Open-source models — profiling-based estimation (Appendix F): Rather than timing each SoT run individually (which would require running all $1+B$ inferences sequentially to measure, defeating the purpose), the paper builds a latency profiling table for each LLaMA architecture (7B, 13B, 33B) on the target GPU (NVIDIA A100-80G or RTX 3090). The profiling captures:

• Decoding latency $t^D_B(k)$: the time to decode the $(k+1)$-th token with batch size $B$, profiled for all $k = 1, \ldots, 1024$ and $B = 1, \ldots, 16$. Each measurement runs three times and takes the geometric mean.
• Prefilling latency $t^P_B(k)$: the time to prefill an input of token length $k$ with batch size $B$, profiled for $k = 1, 11, 21, \ldots, 691$ (every 10 tokens) and $B = 1, \ldots, 16$. Each measurement runs seven times with the first two as warmup, taking the geometric mean of the last five.

Given a request with $l_i$ input tokens and desired $l_o$ output tokens at batch size $B$, the estimated latency is:

$T(l_i, l_o, B) = \tilde{t}^P_B(l_i) + \sum_{k=l_i}^{l_i+l_o-1} t^D_B(k)$

where $\tilde{t}^P_B(l_i)$ is estimated as $t^P_B(\lfloor l_i/10 \rfloor \times 10 + 1)$ since prefilling is profiled only every 10 token lengths.

The SoT latency is then:

$L_{\text{SoT}} = T(l_i^s, l_o^s, 1) + T(l_i^{pe}, l_o^{pe}, B)$

where $l_i^s, l_o^s$ are the skeleton stage input/output token lengths, $l_i^{pe}, l_o^{pe}$ are the longest point-expanding input/output token lengths, and $B$ is the number of points. The point-expanding latency uses the longest expansion because the batch is bottlenecked by the slowest sequence.

API-based models — wall-clock measurement: Latency is measured by recording time.time() before and after each API call. The total SoT latency is the skeleton call latency plus the maximum point-expanding call latency.

Actual latency validation (Appendix G.1.4): The paper also validates the profiling-based estimates by running actual batch tests on 9 open-source models, with five repeated runs per model. The actual speed-ups are consistent with the profiling-based estimates: SoT achieves >2× speed-up on 6 of 9 models (Figure 14).

GPU performance calculation (Appendix F): The paper computes average GPU computing performance during decoding as:

$P^D(l_i, l_o, B) = \frac{\sum_{k=l_i}^{l_i+l_o-1} f^D_B(k)}{\sum_{k=l_i}^{l_i+l_o-1} t^D_B(k)}$

where $f^D_B(k)$ is the FLOPs of decoding one token with context length $k$, calculated using DeepSpeed's FLOPs profiler. This metric is used to demonstrate the low utilization of the decoding phase (0.31 TFLOPS for LLaMA-7B at B=1) and the linear improvement with batch size (Figure 10b).

Answer Quality Evaluation Framework

The paper evaluates answer quality using two LLM-based evaluation frameworks that compare SoT-generated answers against normal sequential generation.

FastChat evaluation (Zheng et al., 2023): A GPT-4 judge is presented with a question and a pair of answers (one from SoT, one from normal generation) and asked to declare a preference. FastChat provides a single "general quality" metric, with specialized evaluation prompts for coding and math questions to ensure appropriate assessment criteria.

LLMZoo evaluation (Chen et al., 2023c): Similar pairwise comparison by a GPT-4 judge, but with five detailed metrics beyond general quality: coherence (logical flow between points), diversity (range of perspectives covered), immersion (how well the answer adopts the assigned role's tone and manner), integrity (completeness without gaps), and relevance (closeness to the question without redundancy). LLMZoo does not provide specialized prompts for math and coding, so these categories are excluded from LLMZoo results.

Bias mitigation via double evaluation: To avoid order bias (the GPT-4 judge might systematically prefer the first or second answer presented), each comparison is run twice with swapped answer ordering. Each evaluation assigns +1 (SoT wins), 0 (tie), or -1 (SoT loses). The final outcome is determined by the sum of the two scores: positive sum = SoT wins, zero sum = tie, negative sum = SoT loses. For example, if SoT wins in one ordering and ties in the other (sum = +1), the final is "win"; if SoT wins in one and loses in the other (sum = 0), the final is "tie".

Net win rate metric: To summarize answer quality across many questions, the paper defines:

$\text{Net win rate} = \frac{\#\text{win} - \#\text{lose}}{\text{total number of questions}}$

This metric ranges from -100% (SoT loses on all questions) to +100% (SoT wins on all questions), with 0% indicating competitive performance (equal wins and losses). A positive net win rate means SoT improves answer quality on more questions than it degrades it.

Judge models: The main paper uses GPT-4 as the judge. Appendix I.4 replicates results with ChatGPT-3.5 as judge, finding similar qualitative conclusions but different absolute values (ChatGPT-3.5 tends to be more optimistic about SoT's quality).

Human evaluation absence: The paper explicitly notes that human evaluation was not conducted because "it is easy for a human to tell whether an answer is generated with SoT due to its distinctive pattern, which might cause evaluation bias" (Section 6). The structured list format with skeleton headers is a visible signature that would make blinding impossible — a human evaluator would know which answer is SoT and which is normal, introducing potential confirmation bias.

4. Key Insights and Innovations

Innovation 1: Introducing "Content Co-Organization for Efficiency" as a New Paradigm for LLM Inference Optimization

The most fundamental conceptual move in this paper is not any specific prompt design or routing mechanism, but the reframing of inference efficiency as a data-level problem rather than (only) a model-level or system-level problem. This is what the paper terms the "content co-organization for efficiency" paradigm (Section 5, Appendix D.1).

To understand why this is a significant intellectual shift, consider the landscape before SoT. All prior work on efficient LLM inference operated at one of three levels: (1) the model level — compressing weights via quantization (Frantar et al., 2022; Lin et al., 2023), pruning activations or weights (Mishra et al., 2021; Wang et al., 2021), or redesigning architectures for cheaper attention (Kitaev et al., 2020; Shazeer, 2019); (2) the system level — optimizing computational graph execution (Dao et al., 2022; Zhai et al., 2022), scheduling across devices (Sheng et al., 2023), or batching for throughput (Fang et al., 2021); and (3) the decoding level — speculative decoding with assistant models (Leviathan et al., 2022; Chen et al., 2023a) or non-autoregressive generation with modified architectures (Gu et al., 2018). The implicit shared assumption across all three levels is that what the model generates — the content, its structure, its organization — is a given. The efficiency engineer can change how that content is computed but not how the content is organized. The content is treated as exogenous to the efficiency problem.

SoT breaks this assumption. It asks: can we guide the LLM to structure its output in a way that makes inference inherently faster? The answer is a structural parallelism that arises not from model architecture or system scheduling, but from the semantic organization of the content itself. If the answer can be decomposed into a skeleton of independently expandable points, these points can be decoded in parallel — not because the hardware or model supports parallel decoding of arbitrary tokens, but because the content has been deliberately organized to permit it. The key move is making content structure an endogenous variable in the efficiency optimization, not an exogenous constraint.

This is significant beyond the immediate speed-up numbers because it opens an entirely new axis for efficiency research that scales with LLM capabilities rather than with hardware improvements. As LLMs become better at instruction-following, planning, and structured generation, the space of possible "content co-organizations" that they can reliably produce expands. The paper gestures toward this in Section 6 with the "Graph-of-Thoughts" concept — a generalization where the answer structure is not a flat list of independent points but a directed acyclic graph with explicit dependency edges, where each node is decoded conditioned on the content of its ancestors but parallelized with respect to its siblings. This vision requires stronger planning and dependency-tracking abilities from LLMs than currently exist, but the trajectory of capability improvement suggests it may become feasible. The data-centric efficiency paradigm thus has a natural scaling path — it gets more powerful as base models get more capable, making it a complementary bet to model- and system-level approaches whose benefits may saturate.

The paper is careful to position this not as a replacement for existing techniques but as an orthogonal lever (Section 5): SoT can be combined with quantization, FlashAttention, and serving-system batching. The point-expanding stage's batch of requests can itself be processed with all the throughput optimizations that serving systems provide. This means SoT converts a throughput-oriented optimization into a latency-benefiting one — a non-obvious synergy that arises precisely because SoT creates a workload pattern (multiple independent requests in a batch) that throughput-optimized systems are designed to handle efficiently.

Innovation 2: Identifying and Exploiting a Structural Latency-Throughput Tradeoff Specific to the Decoding Phase

The paper's efficiency analysis (Appendix E) provides a crisp, empirically grounded explanation for why batched parallel point-expanding achieves speed-ups, and this analysis itself constitutes a conceptual contribution: it makes explicit a latency-throughput tradeoff that is intrinsic to the memory-I/O-bound nature of the decoding phase and shows how content-level parallelism can exploit it.

The critical empirical observation is the staggering utilization gap between the prefilling and decoding phases. As reported in Table 5, LLaMA-7B achieves 43 TFLOPS (13.8% of peak A100 FP16 performance) during prefilling but only 0.31 TFLOPS (0.1% of peak) during decoding — a 139× drop. The decoding phase is so severely underutilized because generating a single token requires loading all model weights from HBM onto the GPU compute units, and the computation performed on those weights (one token's worth of matrix multiplications) is minuscule relative to the I/O cost. This means the decoding phase leaves enormous GPU compute capacity idle — capacity that can be filled by decoding multiple sequences simultaneously without proportionally increasing per-token latency (Figure 10a).

This is not a new observation about transformer inference — the memory-bound nature of autoregressive decoding is well-documented. What is new is the recognition that content-level parallelism can exploit this specific utilization gap without requiring any changes to the model, system, or hardware. Prior work attempted to fill this utilization gap through mechanisms that are external to the content: speculative decoding uses an assistant model to generate candidate tokens for parallel verification, non-autoregressive generation uses modified architectures to sample multiple tokens at once, and serving systems use batching across independent user queries. SoT fills it by making a single user's query generate multiple independent decoding workloads — a transformation from serial to parallel work that happens at the semantic level.

The paper quantifies this tradeoff clearly. The peak memory overhead of batching is modest — for LLaMA-33B, peak memory grows from ~65GB at batch size 1 to ~70GB at batch size 9 (Figure 10c, ~7.7% increase) — because model weights dominate the memory footprint regardless of batch size. This means the latency reduction comes at a small memory cost, making it practical on single-GPU setups. The countervailing cost is in throughput and total tokens: SoT increases the total number of prefilling tokens by 30–89× (Tables 6, 7) and the total number of generated tokens by 1.1–6.8× relative to normal generation (Figure 11f). These overheads mean SoT is not "free" — it trades throughput and computational cost for latency reduction, and this tradeoff is favorable only when the system has spare compute capacity (underutilized GPUs during off-peak periods, or edge deployments with a single user).

This analysis provides a principled framework for thinking about when SoT is appropriate — a question the paper addresses in Appendix H by distinguishing saturated from unsaturated serving scenarios. When there is an unsaturated number of concurrent queries (single-user edge applications, centralized services during off-peak hours), SoT reduces latency and improves GPU utilization simultaneously. When the system is saturated with concurrent queries, SoT's throughput overhead becomes the dominant concern, and its value shifts from efficiency to quality improvement (since structured answers may be better). This framing of SoT as context-dependent — its value proposition changes with system workload — is a practically important nuance that goes beyond simple "SoT is faster" claims.

Innovation 3: Demonstrating That Structured Prompting Can Simultaneously Improve Efficiency and Answer Quality, With a Clear Diagnostic of When and Why

A significant empirical finding that emerges across the evaluation is that SoT does not just accelerate generation — it also improves answer quality on specific question categories and specific quality dimensions, and the pattern of improvement is diagnostic rather than accidental. The paper shows, through detailed per-category and per-metric analysis, that SoT's quality effects are tightly linked to whether the question's ideal answer benefits from explicit structure, and that the quality improvements come from the structured planning that the skeleton stage enforces.

The quality improvement is not uniform. On question categories where answers naturally decompose into independent points — knowledge, generic, common-sense, roleplay, counterfactual — SoT achieves positive net win rates (Figures 5, 22). On categories requiring sequential reasoning — math, coding, fermi — SoT performs substantially worse. The per-metric breakdown from LLMZoo (Figure 6) reveals which aspects improve and degrade: SoT substantially improves diversity (win rate 61.4% vs. lose 11.3%, net +50.1%) and relevance (win 99.9% vs. lose 0.1%, net +99.8%, though the magnitude here suggests the metric may be nearly tautological for structured answers), while degrading immersion (win 23.2% vs. lose 42.1%, net -18.9%) and coherence (win 29.8% vs. lose 39.6%, net -9.8%). Integrity sits in the middle (net +3.5%).

The mechanism for quality improvement is the forced planning step. In normal sequential generation, the LLM produces tokens one by one without explicit high-level organization — the answer may be rambling, miss important perspectives, or contain filler before getting to substantive points. The skeleton stage forces the LLM to commit to a concise, multi-perspective outline before generating any detailed content. This constraint acts as a kind of structural regularizer: it prevents the model from fixating on one aspect, forces breadth of coverage, and eliminates preamble. The paper's answer examples illustrate this concretely — in the "How can I improve my time management skills?" example (Appendix I.1.2), ChatGPT-3.5's normal answer provides six brief bullet points without elaboration, while SoT's answer expands eight distinct strategies with 1–2 sentence explanations each. The skeleton forced the model to generate a more comprehensive and detailed answer than its default sequential generation.

The quality improvements are especially interesting because they are not the primary motivation for SoT. The paper explicitly states that SoT "does not aim to improve answer quality as its primary goal" and that the quality benefits are a "potential" side effect (Section 3.2 introduction). This distinguishes SoT from the extensive literature on prompting for quality improvement — Chain-of-Thought (Wei et al., 2022), Tree-of-Thoughts (Yao et al., 2023), self-consistency (Wang et al., 2022) — all of which design prompts and decoding strategies specifically to enhance reasoning or answer correctness. SoT's quality improvements emerge as a byproduct of structuring for efficiency, suggesting that the relationship between answer structure, reasoning quality, and inference efficiency is more intertwined than prior work recognized.

The degradation dimensions are equally diagnostic. Immersion suffers because the structured list format with embedded skeleton headers reads as less "in-character" for roleplay and narrative tasks — a chef describing their signature dish should not sound like they're reading bullet points. Coherence suffers because the independent expansion of points loses the transitional language and narrative flow that sequential generation naturally produces. These degradations are not failures of the approach so much as manifestations of a fundamental tension: the same structural decomposition that enables parallelism can break the linear coherence that sequential prose relies on. The paper doesn't resolve this tension, but identifying it clearly is a contribution — it tells future work exactly where to focus (improving transition generation, allowing the LLM to decide whether to include skeleton headers in the final answer, or combining sequential and parallel generation for different parts of the response).

Innovation 4: The Router as a Practical Bridge to Deployment, Validating That Adaptive Prompting Strategies Can Generalize Across Question Types

The SoT-R router extension (Section 4) is more than an engineering add-on — it embodies a meta-strategy for adaptive prompting that addresses a fundamental brittleness in single-strategy prompting approaches. The core insight is that no single prompting strategy is optimal across all question types, and a lightweight classifier can route each question to the appropriate strategy without requiring the user to know which strategy to apply.

This matters because it converts SoT from a method that works on ~60% of questions (the fraction suitable for independent-point decomposition, per Appendix K.1) into one that works on all questions without degradation. The key evidence is Figure 8: for question categories where base SoT performs poorly (math, coding, writing, fermi), SoT-R recovers to approximately 0% net win rate — meaning it performs competitively with normal generation — by falling back to sequential decoding on those questions. On categories where base SoT already performs well (knowledge, generic, common-sense, roleplay, counterfactual), SoT-R maintains the quality improvements. The router successfully separates the questions into "SoT-appropriate" and "not," with the confusion matrices in Appendix C.3 showing reasonable alignment between the routers and human annotations.

The comparison between the prompting router (GPT-4 as a zero-shot classifier) and the trained router (fine-tuned RoBERTa-base) reveals an interesting tradeoff that speaks to the broader question of when to use LLMs versus small specialized models for classification tasks. The trained router achieves 0.04 seconds average latency versus 0.65 seconds for the prompting router — a 16× speed-up. It also achieves better alignment with human annotations on the more diverse WizardLM dataset (25 false positives vs. 66 for GPT-4; Table 4), suggesting that while GPT-4 has strong zero-shot classification ability, it struggles with the nuanced boundary cases present in a diverse question distribution. The training cost for the trained router is minimal — 2 minutes on a single A100 — making it a practical choice even for modest deployment budgets. This finding is not just about SoT; it suggests a general pattern where routing between prompting strategies benefits from a dedicated, fine-tuned classifier rather than relying on general-purpose LLM judgment, especially when the boundary between "suitable" and "not suitable" is subtle.

An intriguing result is that the routers occasionally surpass human annotations in downstream quality outcomes (Figure 8, roleplay category). This means human annotators are not perfect at predicting which questions will benefit from SoT — some questions that humans classify as unsuitable actually yield better answers with SoT according to the GPT-4 judge. This suggests that the relationship between question type and prompting strategy effectiveness is not fully captured by human intuition, and that data-driven routing may discover non-obvious strategy assignments.

The router design also reflects a practical prioritization: the Tversky loss with α = 0.7, β = 0.3 penalizes false positives (triggering SoT when it would degrade quality) more than false negatives (missing a SoT opportunity). This is the right choice for a user-facing system — a speed-up missed is a minor inconvenience; a quality degradation is a user-visible failure. This asymmetric loss function is a small but important design detail that signals awareness of deployment realities, not just benchmark metrics.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The primary evaluation uses Vicuna-80 (Chiang et al., 2023), a set of 80 handcrafted questions spanning nine categories: coding, math, fermi, roleplay, writing, knowledge, generic, counterfactual, and common-sense. This is supplemented by WizardLM (Xu et al., 2023), with 218 questions across 29 more fine-grained categories including Physics, Biology, Law, Medicine, and Code Debug. For router training data, the paper annotates the LIMA dataset (Zhou et al., 2023), which contains 1,030 Q&A pairs sourced from Stack Exchange, wikiHow, and Reddit (Section 3, Appendix C.2.1). The datasets are chosen to represent the assistant-style conversational questions that are the target use case, with diversity in both topic and required answer structure.

• Base model(s). The paper evaluates 12 instruction-tuned LLMs spanning three access tiers (Table 1, Appendix A). Nine open-source models based on the LLaMA architecture (Touvron et al., 2023a;b): LLaMA2-Chat-7B, LLaMA2-Chat-13B, OpenChat-13B (Wang et al., 2023a), Vicuna-7B V1.3, Vicuna-13B V1.3, Vicuna-33B V1.3 (Chiang et al., 2023), StableVicuna-13B (Phung, 2023), UltraLM-13B (Ding et al., 2023), and Vicuna-7B V1.1 (Chiang et al., 2023). Three API-based models: Claude (Anthropic, 2023, accessed via Slack), ChatGPT-3.5 (Azure OpenAI, gpt-35-turbo 0301), and GPT-4 (OpenAI, gpt-4-0613). The open-source model scale ranges from 7B to 33B parameters, and the inclusion of both weak and strong instruction-followers allows analysis of how model capability affects SoT's performance.

• Metrics. The paper evaluates two dimensions:

  • Efficiency: End-to-end generation latency, measured in seconds. For open-source models, latency is estimated via pre-built profiling tables (Appendix F) and validated with actual batch tests (Appendix G.1.4). For API-based models, latency is measured via time.time() around API calls. The primary efficiency metric is speed-up: the ratio of normal sequential generation latency to SoT (or SoT-R) latency, where SoT latency = skeleton stage latency + max(slowest point-expanding latency), and normal latency = latency of a single sequential generation call for the same question.
  • Answer quality: Evaluated by a GPT-4 judge (ChatGPT-3.5 in Appendix I.4) using two frameworks: FastChat (Zheng et al., 2023), which provides a single general quality metric with specialized prompts for coding and math, and LLMZoo (Chen et al., 2023c), which provides five detailed metrics — coherence, diversity, immersion, integrity, and relevance — plus a general quality metric. Each comparison runs twice with swapped answer ordering to mitigate position bias. The aggregate metric is net win rate = (#SoT wins − #SoT losses) / total questions, ranging from −100% to +100% (Section 3.2).

• Baselines. The sole baseline is normal sequential generation: the LLM is prompted with only the user question (no skeleton or point-expanding prompts) and generates the answer token-by-token in the standard autoregressive manner. This is the universal default for all LLM inference and represents the status quo that SoT aims to improve upon. For quality comparisons, SoT answers are evaluated pairwise against normal answers for the same question; for efficiency comparisons, SoT latency is compared against normal generation latency for the same question and model.

• Generation budget / compute accounting. For open-source models, SoT spends a total generation budget of l_o^s + B × (l_o^{pe}) output tokens (skeleton output plus B point expansions of varying length), while normal generation spends l_o^{normal} output tokens. The paper does not constrain the two to equal token budgets — SoT answers are often 1.1–6.8× longer than normal answers (Figure 11f). The speed-up comes from converting serial latency into parallel latency, not from reducing total computation. The paper acknowledges this explicitly by reporting the prefilling token overhead ratios (30–89× for API models, Tables 6 and 7) and discussing the throughput tradeoff (Appendix H). There is no FLOPs-matched comparison between SoT and normal generation.

• Cross-validation / statistical protocol. For the trained router, the LIMA dataset (1,030 questions) is annotated for SoT suitability and used as the training set. The router is evaluated on Vicuna-80 and WizardLM, which are entirely separate datasets — there is no training on the evaluation data. For latency measurements, open-source model profiling uses 3–7 repeated measurements with geometric mean aggregation (Appendix F), and actual latency tests on 9 models use 5 repeated runs with box plots (Figure 14). For quality evaluation, each pairwise comparison is run twice with swapped ordering; the paper does not report confidence intervals or statistical significance tests for net win rates, but the two-fold evaluation protocol (two orderings per comparison) provides a basic consistency check.

Main Quantitative Results

Speed-Up Results: Aggregate and Per-Model

The headline efficiency result is that SoT achieves a >2× speed-up on 8 out of 12 tested models, with a maximum of 2.39× for LLaMA2-Chat-7B on Vicuna-80 (Figure 2a). The speed-up varies substantially by model:

• Highest speed-ups: LLaMA2-Chat-7B (2.39×), LLaMA2-Chat-13B (2.38×), Vicuna-7B V1.1 (2.30×), OpenChat-13B (2.28×), Vicuna-33B V1.3 (2.24×), UltraLM-13B (2.18×), Vicuna-7B V1.3 (2.01×), GPT-4 (2.00×).
• Moderate speed-ups: ChatGPT-3.5 (1.97×), Vicuna-13B V1.3 (1.91×).
• Limited speed-ups: Claude (1.31×) and StableVicuna-13B (1.13×).

The paper attributes the variation to differences in how well models follow the point-expanding prompt's conciseness instruction. StableVicuna-13B generates point-expanding responses that can be "as lengthy as the overall normal answer" (Section 3.1.1, Figure 11c), meaning its longest parallel segment is not significantly shorter than the full sequential answer. In contrast, API-based models (Claude, ChatGPT-3.5, GPT-4) follow the instruction well and generate shorter expansions (Figure 11c), but Claude's lower speed-up is partly due to its already-fast normal generation and partly due to fewer skeleton points on average (B = 5.0 for Claude vs. 7.3 for Vicuna-7B V1.3; Figure 11a).

Speed-Up Results: Per-Category

When broken down by question category across all models, SoT achieves speed-ups on every category, but the magnitude varies (Figure 2b):

• Highest speed-ups: knowledge (2.33×), generic (2.31×), writing (2.26×), common-sense (2.24×), coding (2.06×).
• Moderate speed-ups: roleplay (1.95×), counterfactual (1.89×), fermi (1.69×).
• Lowest speed-up: math (1.34×).

However, the paper marks in green the five categories where SoT provides high-quality answers (knowledge, generic, common-sense, roleplay, counterfactual) and in red the categories where quality degrades (writing, fermi, math, coding; Section 3.1.2). The green categories achieve speed-ups of 1.89× to 2.33×, meaning SoT simultaneously accelerates generation and maintains or improves quality on these. For the red categories, the speed-up exists but is less meaningful in practice because it comes with quality degradation — the SoT-R router addresses this by falling back to normal generation on these question types.

Determinants of Speed-Up: Point Count, Balance, and Token Lengths

Figure 11 provides the detailed statistics that explain the per-model and per-category speed-up variation:

• Average number of points B (Figure 11a): Ranges from 4.4 (LLaMA2-Chat-13B) to 9.7 (StableVicuna-13B). GPT-4 and StableVicuna-13B generate the most points (~9), while LLaMA2 models and ChatGPT-3.5 generate fewer (<6). More points create more parallelism (higher potential speed-up) but also more token overhead from repeating the skeleton in each point-expanding request.
• Maximum point-expanding response length (Figure 11c): API-based models (Claude, ChatGPT-3.5, GPT-4) generate short expansions (48–67 tokens on average), closely following the "1–2 sentence" instruction. Open-source models generate substantially longer expansions: StableVicuna-13B averages 216 tokens, and UltraLM-13B averages 120 tokens. The maximum expansion length is the bottleneck for the parallel stage — it directly limits speed-up by determining T_decode_batch(B, max_length).
• Ratio of maximum expansion length to normal answer length (Figure 11d): For most models, this ratio is 0.2–0.9, meaning the longest parallel segment is shorter than the full sequential answer. For StableVicuna-13B, the ratio is 0.9 on average and exceeds 2.0 for some categories, meaning its longest expansion can be longer than the normal answer — completely negating the parallelism advantage.
• Imbalance degree (Figure 11e): Measured as the standard deviation of point expansion token lengths. LLaMA2 and API-based models generate more balanced expansions (std dev 7–20 tokens), while some open-source models show high variance (e.g., Vicuna-7B V1.3 std dev = 30 tokens). High imbalance means the batch is bottlenecked by a single long expansion while shorter ones complete early, reducing effective parallelism.
• Overall SoT-to-normal length ratio (Figure 11f): SoT answers are 1.1× to 6.8× longer than normal answers on average. StableVicuna-13B shows the most extreme blowup (6.8×), while Claude, ChatGPT-3.5, and GPT-4 generate answers 0.5–1.5× the normal length. This length increase reflects SoT's tendency to produce more comprehensive, multi-perspective answers, which is a quality benefit but increases total token generation and prefilling overhead.

Latency Breakdown: Prefilling vs. Decoding Phases

Figure 12 decomposes the absolute latencies of normal and SoT generation into prefilling and decoding components across models (Figure 12a) and categories (Figure 12b). The key finding is that the decoding phase overwhelmingly dominates end-to-end latency for both normal and SoT generation. For example, across all models, the normal generation decoding phase accounts for ~90–95% of total latency. SoT's skeleton stage adds some prefilling overhead (the skeleton prompt is longer than a bare question), and the point-expanding stage adds substantial prefilling overhead (B requests prefilled as a batch), but this overhead has "negligible impact on the overall latency" (Appendix G.1.2) because decoding still dominates. The speed-up comes from reducing the decoding phase time: SoT decodes ~N/B tokens per sequence in a batch of size B, versus N tokens serially. The speed-up is visible as the dramatic reduction in the blue "decode" bar height from normal to SoT in Figure 12.

Figure 13 replicates this analysis on an RTX 3090 GPU for the three 7B models that fit on a consumer GPU. Speed-ups are 1.94× (Vicuna-7B V1.3), 2.26× (Vicuna-7B V1.1), and 2.40× (LLaMA2-Chat-7B), confirming that the efficiency gains are not specific to datacenter GPUs.

Figure 14 validates the profiling-based latency estimates with actual batch testing: SoT achieves >2× speed-up on 6 of 9 open-source models, with maximum 2.88× for Vicuna-7B V1.1. The agreement between profiling-based estimates (Figure 2a) and actual measurements (Figure 14a) confirms the reliability of the estimation methodology. Minor discrepancies exist — for example, actual Vicuna-7B V1.1 achieves 2.88× versus an estimated 2.30× — likely due to the profiling table's interpolation between measured points and variations in system load.

Answer Quality Results: Aggregate

Figure 3 presents the overall win/tie/lose rates of SoT versus normal generation across all models and questions using two metrics. Under the FastChat general quality metric, SoT wins 29.5%, ties 29.3%, and loses 41.2% of comparisons — SoT is not worse than normal generation in ~59% of cases. Under the LLMZoo general quality metric, SoT wins 45.8%, ties 19.6%, and loses 34.5% — SoT is not worse in ~65% of cases. The discrepancy between the two metrics (29.5% vs. 45.8% win rates) reflects differences in their evaluation prompts and criteria, but the consistent finding is that SoT maintains competitive quality with normal generation in the majority of cases, despite being faster. The paper notes that a more accurate view comes from the per-category analysis, since aggregate win rates average across question types where SoT is known to be unsuitable.

Answer Quality Results: Per-Model

Figure 4 shows net win rates per model. The models fall into three groups:

• High net win rates (SoT improves quality): StableVicuna-13B, UltraLM-13B, Vicuna-13B V1.3 — all with positive net win rates under both FastChat and LLMZoo metrics. The paper hypothesizes that these models are "good enough to understand SoT prompts" but their normal sequential generation "has a larger room for improvement" than stronger models (Appendix I.1.1).
• Moderate net win rates: GPT-4, LLaMA2-Chat-7B, Vicuna-33B V1.3, Vicuna-7B V1.3, ChatGPT-3.5 — mixed or near-zero net win rates.
• Low net win rates (SoT degrades quality): LLaMA2-Chat-13B, OpenChat-13B, Vicuna-7B V1.1, Claude — consistently negative net win rates. The paper identifies two distinct failure modes (Appendix I.1.1): (a) weak models (OpenChat-13B, Vicuna-7B V1.1) that cannot follow the SoT prompts precisely — they either generate malformed skeletons (e.g., completing skeleton points in the skeleton stage) or produce empty point expansions; (b) strong models (Claude) whose normal generation is already so good that the structured format provides no quality advantage, and the list format's immersion/coherence penalties dominate.

The per-model, per-category heatmap in Figure 22 provides a granular view: for example, Vicuna-13B V1.3 achieves high net win rates on counterfactual (+30%), generic (+10%), knowledge (+60%), and roleplay (+30%) but strongly negative on coding (−100%), math (−50%), and fermi (−30%). This pattern — positive on independently-expandable categories, negative on step-by-step reasoning categories — is consistent across most models.

Answer Quality Results: Per-Category

Figure 5 reveals the central per-category pattern. Using the FastChat general quality metric:

• Positive net win rates: counterfactual (+17%), generic (+6%), common-sense (+3%), knowledge (+2%), roleplay (+2%). These are the five categories where SoT is suitable.
• Negative net win rates: fermi (−18%), writing (−25%), math (−57%), coding (−75%). These are categories where parallel expansion is inappropriate.

The LLMZoo metric (Figure 5b) shows similar relative ordering but more optimistic absolute values — e.g., counterfactual at +37% vs. +17%. The key qualitative insight from answer inspection (Appendix I.1.2) is that SoT excels when the ideal answer covers several relatively independent perspectives. The forced skeleton stage prevents rambling, ensures breadth of coverage, and eliminates filler. For math and coding, the skeleton cannot capture sequential dependencies — later reasoning steps need results from earlier steps, but the point-expanding stage processes all points independently, so the model either makes errors (using uncomputed intermediate values) or falls back to embedding the full computation in the skeleton itself (which some models do adaptively, as shown in the fermi example in Appendix I.1.2).

Answer Quality Results: Per-Metric

Figure 6 decomposes quality into the five LLMZoo dimensions:

• Diversity: SoT wins 61.4%, ties 11.3%, loses 27.3% — net +34.1%. The skeleton stage forces the model to consider multiple perspectives, directly improving breadth of coverage.
• Relevance: SoT wins 99.9%, ties 0.1%, loses 0.0% — nearly perfect. The paper hypothesizes this is because the structured format eliminates tangential content and keeps each point focused on its assigned topic. The near-100% figure suggests the LLMZoo relevance metric may be nearly definitional for structured list answers — any answer that explicitly addresses each skeleton point will be rated as highly relevant.
• Integrity: SoT wins 40.5%, ties 34.6%, loses 24.9% — roughly balanced.
• Coherence: SoT wins 29.8%, ties 30.6%, loses 39.6% — net −9.8%. The list format with embedded skeleton headers breaks narrative flow, makes transitions between points abrupt, and lacks introduction/conclusion paragraphs. This is the main quality cost of parallelization.
• Immersion: SoT wins 23.2%, ties 34.6%, loses 42.1% — net −18.9%. The bullet-point structure reads as less "in-character" for roleplay, narrative, or conversational tasks (Appendix I.1.3 provides the chef signature dish example where the normal answer opens with "Bonjour honored judges..." while SoT produces stiff labeled points).

These per-metric results show that SoT's quality effects are multi-dimensional and trade off against each other: the structural organization that improves diversity and relevance simultaneously degrades coherence and immersion. The paper does not claim SoT is universally better — it is better on specific quality dimensions and worse on others, and the net effect depends on which dimensions matter more for the question type.

SoT-R Router Results: Speed-Up

Figures 7, 15, and 16 present the speed-ups of SoT-R with both prompting and trained routers compared to base SoT. The key findings:

• SoT-R speed-ups are lower than SoT because the router falls back to normal generation for some questions, and the router itself adds a small latency overhead (0.04s for trained router, 0.65s for prompting router on Vicuna-80; Appendix G.2). On Vicuna-80 across all categories (Figure 7), SoT-R with trained router achieves 1.14× to 1.82× speed-up, while base SoT achieves 1.13× to 2.39×. The reduction is largest for models where many questions are unsuitable for SoT.
• The trained router and prompting router produce similar speed-ups on Vicuna-80 (Figures 15a vs. 15b), with the trained router slightly higher for 7 of 12 models (Figure 7). On WizardLM, the prompting router generally achieves higher speed-ups (Figures 16, 17), particularly on GPT-4 (2.41× vs. 1.74×), because it triggers SoT more aggressively — but this comes with more false positives and quality degradation (discussed below).
• Per-category speed-ups with SoT-R (Figures 18, 19) show that SoT-R still achieves substantial speed-ups on the suitable categories: with the trained router, common-sense (2.10×), counterfactual (1.79×), knowledge (1.87×), generic (2.26×), and roleplay (1.23×). On unsuitable categories, speed-ups are near 1.0× (normal generation is used), except for a few false positives.

SoT-R Router Results: Answer Quality

Figures 8 and 23 present the answer quality of SoT-R compared to base SoT. The central finding:

• SoT-R significantly improves quality on unsuitable categories by falling back to normal decoding. Using the FastChat metric (Figure 8), base SoT achieves net win rates of −75% (coding), −57% (math), −25% (writing), and −18% (fermi). SoT-R with the trained router improves these to approximately 0% for coding, math, and writing, and slightly positive for fermi — meaning it is now competitive with normal generation on these categories.
• SoT-R maintains quality on suitable categories: counterfactual, generic, common-sense, knowledge, and roleplay all remain positive or near-zero net win rates with SoT-R. The router successfully separates the questions without losing the quality benefits on questions where SoT works.
• The trained router occasionally outperforms the human router: on the roleplay category (Figure 8), the trained router achieves a positive net win rate while both the prompting router and human router are near zero. This means the trained router triggers SoT on some roleplay questions that humans thought were unsuitable but that actually benefit from structured generation — and it correctly refrains from SoT on roleplay questions where structured format would hurt immersion. This is an example of data-driven routing discovering non-obvious strategy assignments.
• On WizardLM, the prompting router shows more false positives (Figure 24, Appendix I.2): categories like Code Debug, Complex Format, and Multilingual show negative net win rates for the prompting router (−50% to −20%) but near zero for the trained router. This aligns with the confusion matrix in Table 4, where GPT-4 produces 66 false positives (triggering SoT when it shouldn't) versus 25 for the trained router. The WizardLM dataset's more diverse and ambiguous questions expose the brittleness of zero-shot GPT-4 classification.

Quality vs. Speed-Up Tradeoff Visualization

Figures 1 (right) and 25 synthesize the dual objectives into a single scatter plot, with speed-up on the x-axis and net win rate on the y-axis. The baseline point (1.0, 0.0) represents normal generation — no speed-up, neutral quality. SoT-R moves models up and to the right: most models achieve both speed-up >1.0 and net win rate >0, placing them in the "better and faster" quadrant. The paper notes that models like Vicuna-13B V1.3 and UltraLM-13B achieve particularly favorable tradeoffs (~2.0× speed-up with ~+20% net win rate), while Claude achieves modest speed-up (~1.15×) with approximately neutral quality. GPT-4 achieves 1.54× speed-up with slightly positive quality — a notable result given that GPT-4's normal generation is already considered very high quality.

Comparison with Longer Normal Answers

To address the concern that GPT-4 judges may prefer SoT answers simply because they are longer (SoT answers are 1.1–6.8× longer on average), the paper adds a control experiment (Appendix I.3): normal generation is prompted with "Please give a slightly long answer" (ChatGPT-3.5) or "Please give a long answer" (LLaMA2-Chat-7B) to approximately match SoT answer lengths (Figure 27). When lengths are comparable, SoT still achieves competitive quality: on ChatGPT-3.5, SoT wins 32.4% vs. 24.3% lose against the longer normal baseline (net +8.1%), and on LLaMA2-Chat-7B, SoT wins 2.7% vs. 5.4% lose (net −2.7%). The paper reports this as evidence that SoT's quality is not purely a length artifact, though the sample is limited (only questions manually labeled as suitable for SoT) and the results are close to competitive rather than clearly superior.

Ablation Studies and Robustness Checks

• Router type comparison: prompting vs. trained vs. human: The three routers are compared on both efficiency and quality across Vicuna-80 and WizardLM. On Vicuna-80 (Figures 7, 8), the trained router achieves similar speed-ups and quality to the prompting router, with slight advantages on specific categories (e.g., roleplay). On WizardLM (Figures 17, 24), the trained router significantly outperforms the prompting router on quality (fewer false positives on categories like Code Debug, Complex Format, Multilingual) while achieving lower speed-ups (since it triggers SoT less aggressively). The human router serves as an upper bound on quality but is occasionally surpassed by the trained router, suggesting human intuition about SoT suitability is imperfect. Consistency is analyzed via confusion matrices (Tables 3, 4): on Vicuna-80, GPT-4 achieves 0 false positives and 5 false negatives; the trained router has 6 false positives and 5 false negatives. On WizardLM, GPT-4 has 66 false positives and 3 false negatives; the trained router has 25 false positives and 31 false negatives.

• Judge model robustness: GPT-4 vs. ChatGPT-3.5: Appendix I.4 replicates all quality evaluations using ChatGPT-3.5 instead of GPT-4 as the judge. The aggregate win/tie/lose rates differ in absolute value (FastChat general quality: 12.4% win / 69.2% tie / 18.4% lose with ChatGPT-3.5 vs. 29.5% / 29.3% / 41.2% with GPT-4), but the qualitative conclusions are consistent: SoT performs well on knowledge, generic, common-sense, roleplay, and counterfactual, and poorly on coding, math, fermi, and writing (Figures 29 vs. 5). The per-model rankings are preserved: Vicuna-13B V1.3, StableVicuna-13B, and UltraLM-13B have high net win rates; OpenChat-13B, Vicuna-7B V1.1, and Claude have low net win rates (Figures 30 vs. 4). The per-metric breakdown also replicates: SoT improves diversity and relevance, degrades immersion and coherence (Figure 32 vs. 6). This replication across two different judge models strengthens confidence that the findings are not artifacts of a particular judge's biases.

• Quantization compatibility: Appendix J evaluates SoT-R combined with 4-bit weight-only quantization (GPTQ; Frantar et al., 2022) on 8 open-source models. Two comparisons are reported: (a) SoT-R on quantized models vs. normal generation on quantized models — measuring how much SoT accelerates already-accelerated quantized inference (Figures 33, 34); (b) SoT-R on quantized models vs. normal generation on unquantized models — measuring the compound speed-up from quantization + SoT (Figures 35, 36). In comparison (a), SoT-R achieves 1.08× to 1.99× speed-ups on quantized models (Figure 33). In comparison (b), the compound speed-up reaches 1.54× to 2.07× across models (Figure 35) and up to 3.41× on the generic category with the trained router (Figure 36b). This demonstrates that SoT is orthogonal to and composable with model-level compression techniques — a practically important finding for deployment scenarios where both techniques would be applied.

• GPU type robustness: A100 vs. RTX 3090: The paper evaluates SoT on both a datacenter GPU (NVIDIA A100-80G) and a consumer GPU (NVIDIA RTX 3090). On the three 7B models tested on RTX 3090, SoT achieves 1.94× to 2.40× speed-ups (Figure 13), comparable to the A100 results (e.g., LLaMA2-Chat-7B: 2.39× on A100, 2.40× on RTX 3090). The memory-I/O-bound nature of decoding is a property of the GPU architecture, not the specific GPU tier, so SoT's mechanism transfers across GPU types as long as the model fits in memory.

• Dataset robustness: Vicuna-80 vs. WizardLM: While the main paper focuses on Vicuna-80, Appendix G and I report speed-ups and quality on the WizardLM dataset (218 questions, 29 categories). The per-model speed-ups (Figure 17) show similar patterns: SoT provides >1.5× speed-up on most models, with GPT-4 achieving the highest (2.41× with prompting router). The per-category speed-ups (Figure 21) show substantial variation, from near 1.0× on Reasoning, Physics, and Math to >2.2× on Counterfactual, Economy, and Technology. The quality results (Figure 24) replicate the Vicuna-80 finding that SoT-R reduces quality degradation on unsuitable categories compared to base SoT, though the WizardLM results are more varied due to the dataset's greater diversity.

• Latency estimation validation: The profiling-based latency estimation method (used for all open-source model speed-up calculations in the main paper) is validated against actual batch testing for 9 models on Vicuna-80 (Figure 14). The reported speed-ups from profiling (Figure 2a) and actual testing (Figure 14a) are consistent in their relative ordering: the models with highest profiling speed-ups (LLaMA2-Chat-7B, LLaMA2-Chat-13B, Vicuna-7B V1.1) also have the highest actual speed-ups. Absolute values differ slightly: profiling estimates 2.39× for LLaMA2-Chat-7B vs. actual 2.20×; 2.30× for Vicuna-7B V1.1 vs. actual 2.88×. These discrepancies arise from interpolation in the profiling tables (prefilling latency profiled only every 10 token lengths) and natural system variability across runs. The paper uses profiling estimates rather than full actual tests because profiling is more efficient for sweep studies across many models, categories, and budget settings.

• Answer quality with longer normal baseline (partial ablation of the length confound): As discussed above, the comparison against normal generation with explicit "give a long answer" instructions (Appendix I.3) partially controls for the length confound. The sample is limited (only manually-labeled suitable SoT questions on two models), but the result that SoT remains competitive (net +8.1% for ChatGPT-3.5, net −2.7% for LLaMA2-Chat-7B) suggests length alone does not explain SoT's quality. A more comprehensive ablation — running this comparison across all models and categories, or systematically controlling for output length — is not performed.

Critical Assessment

The paper makes three central claims that the experiments aim to support: (1) SoT reduces end-to-end generation latency by up to 2.39× across a diverse set of LLMs, (2) SoT maintains or improves answer quality on question categories whose answers can be decomposed into independently expandable points, and (3) the SoT-R router extension enables adaptive deployment that achieves speed-ups on suitable questions while falling back to normal generation on unsuitable questions, making the system practical for general-purpose use.

Claim 1 (speed-up): Strongly supported for the tested models and hardware, but with important caveats about generalizability. The speed-up numbers are well-documented with transparent methodology: Figure 2a shows 2.39× maximum, 8 of 12 models above 2×. The profiling-based estimation is validated against actual measurements (Figure 14), tested on two GPU types (A100 and RTX 3090), and decomposed in detail (Figures 11, 12). The paper is transparent about why some models achieve lower speed-ups (StableVicuna-13B's failure to follow conciseness instructions, Claude's already-fast base latency).

However, several caveats are under-explored. First, the speed-ups are measured on a dataset of 80 questions, with per-category sample sizes as small as 3 (for math on Vicuna-80; Figure 37 shows only 0–2 math questions are suitable for SoT). Speed-up estimates on categories with ~3 questions have very high variance and should not be interpreted as reliable point estimates. The paper reports averages without confidence intervals, making it impossible to assess whether, for example, the 1.34× math speed-up is meaningfully different from 1.0×.

Second, the paper acknowledges but does not quantify the sensitivity of speed-ups to the specific SoT prompts. The skeleton prompt template differs slightly across models (two-shot for most models, zero-shot for GPT-4; "very shortly" removed for Claude and GPT-4; partial answer mechanism differs for API vs. open-source). These prompt variations are necessary for making SoT work across different model interfaces and capabilities, but they mean the speed-up for any given model is partly a function of prompt engineering effort. How much would LLaMA2-Chat-7B's 2.39× change if the prompt demonstrations were different? The paper provides no prompt ablation.

Third, the speed-up measurement for open-source models relies on the profiling table methodology, which assumes the point-expanding requests are processed as a single batch. In a real serving system with concurrent users and dynamic batching, the actual batching behavior depends on the serving system's scheduling policy. The paper does not test SoT integrated into a serving system like vLLM or TensorFlow Serving, so the reported speed-ups represent an idealized single-user scenario.

Claim 2 (quality improvement on suitable categories): Supported with qualifications about evaluation methodology and limited to specific quality dimensions. The per-category net win rates (Figure 5) show positive values for five categories (counterfactual, generic, common-sense, knowledge, roleplay) and negative for four (fermi, writing, math, coding). The per-metric breakdown (Figure 6) reveals that quality improvement is concentrated in diversity and relevance, with degradation in coherence and immersion. The answer examples in Appendix I.1 provide convincing qualitative evidence: SoT generates more comprehensive, better-organized answers on questions that benefit from multi-perspective coverage.

The qualifications are substantial. The quality evaluation relies entirely on LLM judges (GPT-4, ChatGPT-3.5) with no human evaluation. The paper explicitly acknowledges this limitation (Section 6) and argues that human evaluation is infeasible because SoT's distinctive format makes blinding impossible. This is a genuine constraint — any human evaluator can trivially identify which answer is SoT-generated — but it means the quality claims ultimately rest on the assumption that GPT-4's pairwise preferences correlate with human preferences. The paper cites Li et al. (2023b) for GPT-4's alignment with human judgment but does not validate this alignment on the specific Vicuna-80 questions or SoT's distinctive answer format. It is possible that GPT-4 judges have their own biases toward structured, list-format answers, which would inflate SoT's apparent quality.

The net win rate framing is intuitive but masks effect magnitude. A +2% net win rate (e.g., knowledge and roleplay in Figure 5a) means SoT wins on approximately 1–2 more questions than it loses out of 80 total. Without confidence intervals, it is impossible to assess whether these small positive net win rates are statistically distinguishable from zero. The paper would benefit from reporting raw counts and binomial confidence intervals rather than only aggregate percentages.

The length confound is partially addressed (Appendix I.3) but not eliminated. The control experiment shows SoT remains competitive when normal answers are artificially lengthened, but this is only tested on two models and a subset of questions. The more fundamental issue is that SoT's quality advantage may be entirely attributable to the implicit "give a structured, multi-perspective answer" instruction embedded in the skeleton prompt, rather than to the skeleton-then-expand mechanism itself. A proper ablation would compare SoT against a baseline where the LLM is simply prompted to "answer in a structured list of points with detailed explanations" but generates the full answer sequentially. If that baseline achieves similar quality to SoT, then the quality improvement is a prompting effect, not a structural benefit of parallel expansion. This ablation is not performed.

Claim 3 (router enables general-purpose deployment): Supported for the tested datasets, but the router's generalization to arbitrary user queries is untested. SoT-R successfully routes unsuitable questions to normal generation, recovering competitive quality on math, coding, writing, and fermi (Figure 8). The router comparison reveals that a small trained classifier (RoBERTa-base) can match or exceed GPT-4's routing accuracy while being 16× faster (0.04s vs. 0.65s). The confusion matrices (Tables 3, 4) quantify the tradeoff between false positives (quality degradation) and false negatives (missed speed-up), and the Tversky loss prioritizing false positive avoidance is a sensible design choice for user-facing systems.

The generalization concern is that the router is trained on LIMA (1,030 questions from Stack Exchange, wikiHow, Reddit) and evaluated on Vicuna-80 and WizardLM. These datasets share distributional properties — they are all English-language, assistant-style Q&A drawn from similar domains. The paper does not evaluate the router on out-of-distribution question types (e.g., creative writing prompts, multi-turn dialogues, code completion, non-English queries). The GPT-4 router's 66 false positives on WizardLM (Table 4) suggests that zero-shot routing degrades significantly on a more diverse dataset; the trained router's better WizardLM performance (25 false positives) is encouraging but still leaves room for error. In a production deployment where users ask arbitrary questions, false positives (triggering SoT on unsuitable questions) could produce noticeably degraded answers, eroding user trust. The paper's recommended asymmetric loss helps but does not eliminate this risk.

Missing experiments that would strengthen the paper:

1. Prompt ablation: How do speed-up and quality change if the skeleton prompt is modified (e.g., different demonstration examples, different point count range, different conciseness instructions)? This would establish the sensitivity of results to the specific prompt design and guide practitioners in adapting SoT to new models.

2. Comparison to sequentially generated structured answers: A baseline where the LLM is prompted to produce a structured, multi-perspective answer in a single sequential generation would disentangle the quality benefits of the skeleton structure from the benefits of parallel expansion. If structured prompting alone achieves similar quality, then SoT's value is purely in latency reduction.

3. Evaluation on a more diverse benchmark: Vicuna-80's 80 questions (with some categories having only 3–8 questions) is a small sample. Evaluation on a larger, more diverse benchmark (e.g., AlpacaEval, MT-Bench) would provide more reliable per-category estimates and test generalization beyond the Vicuna distribution.

4. Human evaluation with a blinding workaround: While true blinding is impossible given SoT's distinctive format, a relative comparison design could mitigate bias: present human evaluators with SoT answer A and SoT answer B (with different prompts or configurations) rather than SoT vs. normal, then ask which is better. This doesn't compare against normal generation but would provide human judgments on SoT variants.

5. Latency measurements under realistic serving system conditions: Integrating SoT into an existing serving framework (e.g., vLLM, TensorRT-LLM) with dynamic batching and concurrent users would test whether the theoretical speed-ups materialize in practice or are eroded by scheduling overhead, queuing, and resource contention.

6. Token cost analysis for API models: For API-based models, the paper reports prefilling token overhead ratios (Table 6) but does not translate these into monetary cost. A simple calculation of total API cost for SoT vs. normal generation at current per-token pricing would clarify the practical tradeoff for users who pay per token.

7. Statistical significance testing: Net win rates are reported as point estimates without confidence intervals. Given the small test set (80 questions, with 3–10 questions per category), binomial confidence intervals would reveal whether observed differences are likely to be real or could arise from sampling noise.

Strengths of the experimental design:

The paper's evaluation is commendably thorough in several respects. The testing across 12 models — spanning three orders of magnitude in model size and including both open-source and API-based models — establishes that SoT's benefits are not specific to a single model family. The per-category analysis (rather than only aggregate reporting) reveals the crucial interaction between question type and SoT effectiveness, which is the paper's most important empirical finding. The latency measurement methodology is transparent and validated against actual measurements. The per-metric quality decomposition (diversity, coherence, immersion, relevance, integrity) provides a nuanced picture of where SoT helps and hurts, avoiding oversimplified "better/worse" claims. The router comparison quantifies the precision-recall tradeoff concretely and demonstrates that a cheap trained classifier can outperform GPT-4 for this routing task. The quantization compatibility experiment shows that SoT composes with existing model-level optimizations, which is practically important.

Overall assessment: The experiments support the paper's core claim that SoT provides meaningful latency reduction on a broad set of models and question types while maintaining competitive answer quality on questions amenable to structured decomposition. The evidence is strongest for the efficiency claims (speed-up) and weakest for the quality claims (due to reliance on LLM judges, small sample sizes, and lack of human evaluation). The router extension is shown to be a practical solution for handling unsuitable question types, though its generalization to arbitrary user queries remains uncertain. The paper's primary contribution is the conceptual framework — data-centric efficiency optimization — and the experiments serve primarily to demonstrate its feasibility rather than to provide definitive benchmarks. The speed-up numbers and quality patterns establish that the approach is worth pursuing, but the exact magnitudes should be interpreted as preliminary estimates that depend on model capability, prompt design, and question distribution.

6. Limitations and Trade-offs

SoT Fundamentally Cannot Accelerate Questions Requiring Sequential Reasoning

The assumption or constraint. SoT's parallel point-expanding mechanism assumes that each skeleton point can be expanded independently — that is, the content of point $k$ does not depend on the computed results or reasoning from points $1, \ldots, k-1$. The paper states this explicitly in Section 6: "SoT currently ignores the dependencies between points." For question categories where answers require step-by-step reasoning with sequential dependencies — math, coding, fermi problems — "it is fundamentally challenging to apply SoT" because "the latter steps require the details from the earlier steps" (Section 3.2.3).

The consequence. On math, coding, and fermi questions, SoT produces answers that are either incorrect (the model makes reasoning errors because it cannot reference previously computed intermediate results) or degenerate (the model falls back to embedding the full computation in the skeleton stage, skipping the point-expanding stage entirely). The evidence in Appendix I.1.2 shows concrete failure modes: ChatGPT-3.5 generates a correct skeleton for solving $3x + 10 = 5(x - 2)$ but makes arithmetic errors in point 2 because it does not have access to the result from point 1; the normal sequential answer achieves the correct solution. The net win rates for these categories are strongly negative: −57% for math, −75% for coding, −18% for fermi under the FastChat metric (Figure 5a). This is not a marginal degradation — SoT breaks on these question types. The speed-ups on these categories (Figure 2b: math 1.34×, coding 2.06×) are therefore practically unusable because they come with unacceptable quality.

What evidence exists in the paper. Figure 5 (per-category net win rates), Figure 22 (per-model, per-category heatmap), and the qualitative answer analysis in Appendix I.1.2. The paper also shows that this limitation is robust across models: strong models (GPT-4, ChatGPT-3.5) can sometimes produce correct skeletons but fail at independent point expansion; weak models (Vicuna-7B V1.1) fail even at the skeleton stage (Appendix I.1.2, math example).

Mitigation status. The SoT-R router partially mitigates this limitation by detecting unsuitable questions and falling back to normal generation. The router improves net win rates on math, coding, and fermi to approximately 0% (Figures 8, 23) — i.e., quality equivalent to normal generation. However, this means SoT provides zero speed-up on these question types; the overall system still depends on the base model to handle them sequentially. The paper does not solve the fundamental problem of parallelizing sequential reasoning; it only avoids applying SoT where it would fail. The "Graph-of-Thoughts" concept (Section 6) is proposed as future work to handle dependencies explicitly, but no implementation or results are provided.

Speed-Up Magnitudes Are Highly Sensitive to Model Instruction-Following Ability and Are Not a Predictable or Controllable Property

The assumption or constraint. SoT's speed-up depends critically on the LLM's ability to follow two specific prompt instructions: (1) produce a skeleton with 3–10 points, each 3–5 words, and (2) expand each point in 1–2 sentences without elaborating on other points. The paper acknowledges that "the acceleration ratio of SoT depends on the SoT prompt, the model, and the question, and thus not as predictable and controllable as model- or system-level techniques, which might hinder the practical adoption" (Section 6).

The consequence. The speed-up varies by a factor of ~2.1× across models tested on the same dataset (1.13× for StableVicuna-13B vs. 2.39× for LLaMA2-Chat-7B; Figure 2a). This is not a variance that can be tolerated in deployment scenarios requiring predictable latency guarantees. The root cause is that instruction-following is an emergent capability that varies substantially across models, across prompts, and even across individual queries for the same model. StableVicuna-13B fails to follow the "Write it very shortly" instruction (Section 3.1.1), producing point expansions as long as the full normal answer (Figure 11c), which eliminates the parallelism benefit. Some models (OpenChat-13B, Vicuna-7B V1.1) produce malformed skeletons or empty point expansions (Appendix I.1.1), which simultaneously hurt speed-up and quality. The paper provides no way to guarantee a minimum speed-up for a given model without testing it on the specific question distribution.

The imbalance between point expansion lengths (Figure 11e) further compounds this unpredictability: even when a model generally follows instructions, the longest point expansion for any given question is a random variable. The speed-up is bottlenecked by the longest expansion, meaning a single overly verbose point can eliminate the parallelism benefit for that question. This makes per-query latency highly variable in ways that are hard to anticipate.

What evidence exists in the paper. Figures 2a and 11 provide the model-by-model breakdown of speed-up, point counts, expansion lengths, and imbalance. The paper transparently discusses StableVicuna-13B's failure case (Section 3.1.1), Claude's relatively low speed-up due to "already-fast normal generation and fewer skeleton points" (Appendix I.1.1), and the per-category speed-up variation (Figure 2b). The router consistency analysis (Tables 3, 4) shows that even GPT-4 makes substantial routing errors on diverse datasets (66 false positives on WizardLM), indicating that the question-suitability classification itself introduces unpredictability.

Mitigation status. The paper does not attempt to make SoT's speed-up predictable or controllable. The suggestion in Section 6 is that "prompt tuning techniques" and future LLM capability improvements could make the behavior more consistent and allow for shorter prompts. The SoT-R router mitigates the quality risk (by avoiding SoT on unsuitable questions) but does not address the speed-up variability on questions where SoT is triggered — the speed-up on those questions remains model- and query-dependent. The paper provides no guidance for practitioners on how to estimate expected speed-up for a new model or question distribution without running the full SoT pipeline.

Prefilling Token Overhead and API Cost Are Not Accounted for in Headline Speed-Up Numbers

The assumption or constraint. The paper measures SoT latency as skeleton stage time plus the maximum point-expanding stage time, but the throughput cost (total tokens processed) is substantially higher than normal generation. The paper states this explicitly: "SoT may lead to higher costs" for API-based models (Section 6), and reports prefilling token overhead ratios of 60–89× for API models (Table 6) and 30–39× for open-source models (Table 7, before the common prefix optimization).

The consequence. For API-based models that charge per token (GPT-4, ChatGPT-3.5, Claude), the monetary cost of SoT can be dramatically higher than normal generation. The skeleton is repeated in each of $B$ point-expanding requests, the question is repeated in each request, and the SoT prompts themselves are longer than a bare user question. Table 6 reports that for GPT-4, the total prefilling tokens under SoT are 89.20 times the normal prefilling tokens. While the paper does not translate tokens into dollars, a back-of-the-envelope calculation is revealing: if SoT uses ~90× the input tokens and generates ~1.5× the output tokens (Figure 11f for GPT-4), and GPT-4 pricing is per-token with input tokens often cheaper than output tokens, the total API cost could be ~10–30× higher than normal generation for the same question. The latency reduction is real, but the user pays for it in increased API spend.

For open-source models, the cost is in throughput rather than dollars. Each SoT request consumes ~30× more prefilling tokens than a normal request. In a serving system with multiple concurrent users, this means SoT consumes more GPU compute per query, potentially reducing the maximum queries per second the system can handle. The paper acknowledges this in Appendix H: "this computational overhead remains a concern, especially during periods of high system workload." The paper suggests that SoT is appropriate during unsaturated periods (low user load) but may be detrimental during saturated periods. This context-dependence is not reflected in the headline speed-up numbers.

What evidence exists in the paper. Tables 6 and 7 quantify the prefilling token overhead. Figure 11f shows that SoT answers are 1.1–6.8× longer than normal answers. Appendix H discusses the throughput implications and proposes a common prefix optimization for open-source models (prefilling the shared question+skeleton once with batch size 1, then only the point-specific suffixes with batch size $B$). The paper explicitly notes in Section 6 that "this concern" warrants attention and suggests prompt tuning (Jiang et al., 2023) as a future direction for reducing prompt length.

Mitigation status. Partial mitigation exists for open-source models (the common prefix trick, Appendix H) but is not quantified — the paper does not report the after-optimization prefilling token ratio. For API-based models, no mitigation is provided; the paper only suggests future prompt tuning. The SoT-R router reduces overhead by falling back to normal generation on unsuitable questions, which avoids wasting tokens on questions where SoT would not help anyway, but does not reduce the overhead on questions where SoT is triggered. The paper does not report the total token cost (or dollar cost) of SoT vs. normal generation on the evaluation datasets, which would be a practically useful complement to the latency measurements.

SoT Is Evaluated on a Single Small Test Set (80 Questions) With Limited Per-Category Statistical Power

The assumption or constraint. All main-paper results are on the Vicuna-80 dataset, containing 80 questions across 9 categories. Some categories have very few questions: math has 3 questions, coding has 3 questions, and the other categories have 5–10 questions each (Figure 37 shows human-labeled suitable counts of 0 for math, 0 for writing, 2 for coding, 4 for fermi). The paper acknowledges the evaluation scope limitation indirectly (Section 6: "Our answer quality evaluation is far from perfect due to the limited prompt set"), but does not discuss the statistical implications of the small test set.

The consequence. Per-category speed-up and net win rate estimates are based on 3–10 data points and reported as point estimates without confidence intervals. The observed speed-up of 1.34× for math (Figure 2b) is an average over 3 questions; the true expected speed-up could easily be 1.0×, and the 1.34× could be driven by a single outlier question where the skeleton stage happened to produce a complete answer. The net win rate of −75% for coding (Figure 5a) means SoT lost on roughly 2–3 of the 3 coding questions. Binomial sampling variance on such small samples is enormous — an observed −75% with n=3 is consistent with a true win probability anywhere from ~0% to ~40%. This makes the per-category ordering (e.g., "SoT performs relatively well on counterfactual but relatively poorly on writing") less reliable than the paper's presentation suggests.

The WizardLM dataset (218 questions, Appendix G and I) provides a larger sample but is relegated to the appendix. The WizardLM results (Figure 24) show qualitatively similar patterns to Vicuna-80 — positive net win rates on some categories, negative on others — but with substantial variance across categories and more router errors (GPT-4 router produces 66 false positives; Table 4). The fact that router performance degrades significantly from Vicuna-80 to WizardLM suggests that the Vicuna-80 results may not generalize well to other question distributions.

What evidence exists in the paper. Figure 37 shows the per-category counts of questions suitable for SoT. Appendix G.2 and I.2 report WizardLM results. The paper does not report confidence intervals, standard errors, or statistical significance tests anywhere in the evaluation.

Mitigation status. The paper makes no attempt to address sample size limitations statistically (e.g., bootstrapping, binomial confidence intervals). The WizardLM evaluation in the appendix provides some cross-dataset validation but does not solve the fundamental small-sample problem for per-category analysis. The paper does not claim statistical significance and presents results as preliminary evidence for the SoT concept rather than as definitive benchmarks. However, the per-category conclusions are presented with a level of confidence ("SoT performs relatively well on... relatively poorly on...") that is not fully supported by the sample sizes.

Answer Quality Is Assessed Solely by LLM Judges Without Human Validation, on a Response Format the Judges May Be Biased Toward

The assumption or constraint. All answer quality evaluation relies on GPT-4 (or ChatGPT-3.5 in Appendix I.4) as a pairwise judge comparing SoT answers against normal generation answers. The paper explicitly acknowledges: "Currently, we did not conduct human evaluation since it is easy for a human to tell whether an answer is generated with SoT due to its distinctive pattern, which might cause evaluation bias" (Section 6). The distinctive pattern — numbered skeleton headers followed by prose expansions — makes blinding impossible for human evaluators.

The consequence. Two validity threats arise. First, the GPT-4 judge may itself have preferences for or against structured, list-format answers that do not align with human preferences. If GPT-4 systematically prefers structured, multi-perspective answers regardless of their actual quality or appropriateness for the question, then SoT's quality advantages (e.g., +17% net win rate on counterfactual; Figure 5a) may be artifacts of judge bias rather than genuine quality improvement. The paper's argument that GPT-4 aligns with human judgment (citing Li et al., 2023b) is a general claim that may not hold for the specific format characteristics that distinguish SoT answers.

Second, the LLMZoo per-metric results partially contradict the overall quality picture. SoT substantially improves diversity (+34.1% net) and relevance (+99.9% net, which is suspiciously large and may indicate a near-tautological metric for structured answers), but degrades coherence (−9.8% net) and immersion (−18.9% net) (Figure 6). Whether the net quality effect is positive depends on how humans weight these dimensions relative to each other. For a roleplay question, immersion may be the dominant quality dimension, making SoT's diversity gain irrelevant. For a knowledge question, diversity and relevance may dominate. The GPT-4 judge's aggregation of these dimensions is opaque and uncalibrated against human preferences.

The replication of results with ChatGPT-3.5 as judge (Appendix I.4) shows consistent relative ordering of categories and models but substantially different absolute win rates (e.g., FastChat general quality: 12.4% win rate with ChatGPT-3.5 vs. 29.5% with GPT-4; Figure 28 vs. Figure 3). This sensitivity to the judge model's identity further undermines confidence in the absolute quality claims.

What evidence exists in the paper. All quality results in Section 3.2, Figures 3–6, and Appendices I.1–I.4. The paper is transparent about the reliance on LLM judges and the absence of human evaluation (Section 6).

Mitigation status. The paper makes several efforts to improve evaluation reliability — double evaluation with swapped ordering to mitigate position bias, replication with two different judge models (GPT-4 and ChatGPT-3.5), and per-metric decomposition to reveal the dimensions of improvement and degradation. The "longer normal answer" control experiment (Appendix I.3) partially addresses length bias but is limited to two models and a subset of questions. However, the fundamental validity threat — that LLM judges may be systematically biased toward structured-format answers — is not addressed. The authors' argument that human evaluation is infeasible due to the impossibility of blinding is valid, but this means the quality claims remain fundamentally unvalidated against ground-truth human preferences. The paper does not propose an alternative methodology (e.g., third-party human evaluation of anonymized answer pairs with format indicators removed, or a relative comparison design comparing two SoT variants rather than SoT vs. normal).

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a new conceptual category for LLM inference optimization that sits alongside model-level and system-level techniques, but operates on a fundamentally different axis: the content itself. Before SoT, efficiency research treated what the model generates as exogenous to the efficiency problem — you could compress the model, optimize the attention kernel, or batch across users, but the sequential structure of any single answer was taken as a given. SoT makes answer structure an endogenous variable that can be deliberately shaped for parallelism. This is not an incremental contribution to an existing line of work; it opens a line of work that did not previously exist.

The shift in perspective matters because it changes what kinds of gains are possible. Model-level techniques like quantization and pruning face diminishing returns — you can only compress weights so far before quality degrades. System-level techniques like FlashAttention already approach the hardware efficiency ceiling for the prefilling phase. But content-level parallelism has a scaling property that these other techniques lack: it becomes more powerful as base models become more capable. A model with stronger instruction-following can produce better skeletons, respect point-expanding constraints more reliably, and handle more complex dependency structures. The data-centric efficiency paradigm thus rides the capability curve upward, while model and system techniques face saturation.

The paper's results also force a re-examination of what "inference efficiency" means. The field has largely conflated throughput (tokens per second across many users) with latency (wall-clock time for a single user), and optimized primarily for the former. SoT demonstrates that these two objectives can trade against each other in non-obvious ways: SoT often increases total tokens generated and total FLOPs consumed while reducing end-to-end latency, because it converts serial work into parallel work. This means efficiency metrics that track only total compute are blind to the latency gains that content-level parallelism provides. The paper provides a concrete, quantified example of this distinction, which may encourage more nuanced efficiency benchmarking that disaggregates latency from throughput rather than reporting only one or the other.

The paper also reconciles a tension in the prompting literature that has been latent but unexplored. On one hand, prompting techniques like Chain-of-Thought (Wei et al., 2022) and Tree-of-Thoughts (Yao et al., 2023) improve answer quality by structuring the model's reasoning process, often increasing the number of tokens generated and therefore the latency. On the other hand, efficiency techniques aim to reduce latency. SoT shows that structure and efficiency are not necessarily in tension — the same structural decomposition that improves answer diversity and relevance (because it forces multi-perspective coverage) also enables parallel decoding (because the perspectives are independent). This suggests a broader principle: eliciting structure from LLMs can simultaneously improve quality and enable parallelization, as long as the structure captures genuinely independent substructures. The finding that SoT improves quality on some categories (counterfactual +17% net win rate) while accelerating them (1.89–2.33×) provides the first empirical evidence for this dual benefit.

On the other side, SoT clarifies the boundary where parallel generation fails. The sharp drop in quality on math (−57% net win rate), coding (−75%), and fermi (−18%) questions (Figure 5) establishes that sequential reasoning dependencies are a hard constraint that content-level parallelism cannot circumvent — at least with the current flat-list skeleton structure. This boundary is diagnostically useful because it tells the field where to look for more sophisticated content organizations (graphs, trees, adaptive structures) versus where to accept that sequential decoding is unavoidable. The negative results on these categories are as informative as the positive results on knowledge and generic questions: they define the current frontier of what content-level parallelism can achieve.

Finally, the router extension validates a meta-strategy of adaptive prompting that has implications beyond SoT. The finding that a 120M-parameter RoBERTa model, fine-tuned in two minutes, can match or exceed GPT-4 at classifying question suitability for a prompting strategy suggests that lightweight, strategy-specific routers are a practical design pattern for deploying multi-strategy prompting systems. Rather than relying on a single prompt for all inputs — which inevitably performs poorly on some subset — systems can maintain a portfolio of strategies and route each input to the most appropriate one. The paper provides a concrete recipe for this: annotate a modest training set, train a small classifier with asymmetric loss (penalizing false positives more than false negatives), and deploy with negligible latency overhead (0.04 seconds). This is a generalizable pattern that applies to any setting where multiple prompting strategies have complementary strengths.

Follow-Up Research This Work Enables

Graph-of-Thoughts with explicit dependency edges. The paper's central acknowledged limitation is that SoT assumes all skeleton points are independent, which fails catastrophically on sequential reasoning tasks (Figure 5: −57% net win rate on math). The natural generalization is to replace the flat list skeleton with a directed acyclic graph where nodes are reasoning steps or answer components and edges encode dependencies. In such a Graph-of-Thoughts, each node $v$ would be decoded conditioned on the content of its ancestor nodes (preventing the dependency-breaking errors documented in Appendix I.1.2), but nodes with no path between them could be expanded in parallel. The key open question is whether current LLMs can be prompted to produce such dependency graphs reliably — the paper's skeleton prompt already extracts structure, and extending it to produce "1. [Depends on: none] ... 2. [Depends on: 1] ..." would be a minimal modification. A strong follow-up would measure: (a) can LLMs produce correct dependency annotations for math and coding problems (evaluated against ground-truth solution graphs), (b) does conditioned expansion (providing ancestor content in the point-expanding prompt) fix the sequential reasoning errors SoT currently makes, and (c) what speed-up is achievable given that only independent nodes can be parallelized (the parallelism is limited by the graph's width, not the total number of nodes)? The MATH benchmark or GSM8K would be appropriate testbeds since they have clear step-by-step solution structures.

Difficulty-aware adaptive content organization. SoT currently applies the same skeleton-then-expand structure to all questions deemed "suitable" by the router. But even within suitable questions, the optimal degree of parallelism (number of skeleton points) may vary. For simple questions, 3–5 coarse points may suffice; for complex questions, 8–10 fine-grained points may provide better quality. Moreover, the optimal structure may depend on the available compute budget and system load — during off-peak periods with idle GPU capacity, more aggressive parallelization (more points, longer expansions) trades throughput for latency reduction; during saturated periods, a shallower structure with fewer points and shorter expansions minimizes token overhead. This suggests a compute-aware content planner that decides not just whether to use SoT but with what parameters (number of points, expansion length target, sequential vs. parallel ratio for different answer sections). The difficulty estimation could leverage the PRM-based approach from the compute-optimal test-time scaling literature (which this paper does not engage with), or could be learned directly from the question text as in the SoT trained router but with a continuous or multi-class output. A concrete experiment would measure the Pareto frontier of latency vs. quality as a function of skeleton granularity for different question categories, producing a lookup table analogous to Figure 7 in the SoT paper but for structural parameters rather than sequential-to-parallel ratios.

Closed-loop SoT with self-consistency verification. SoT's parallel point expansion creates an opportunity that sequential generation does not: because all point expansions are generated independently (conditioned only on the shared skeleton), they can be verified against each other for consistency. If point 3's expansion implicitly contradicts point 1's expansion (e.g., different assumed values in a fermi calculation, or conflicting factual claims in a knowledge answer), the system could detect this inconsistency and either re-generate the conflicting points with additional context about the contradiction, or flag the answer as potentially unreliable. This is a form of self-consistency (Wang et al., 2022) applied at the content level rather than the answer level — instead of sampling multiple complete solutions and taking a majority vote, sample multiple expansions of the same skeleton and check for internal coherence. This would directly address SoT's coherence degradation (−9.8% net win rate; Figure 6), which the paper identifies as arising from the independent expansion of points that should be mutually consistent. A strong follow-up would measure whether consistency verification + targeted re-generation can close the coherence gap with sequential generation without sacrificing speed-up, and whether the verification signal can be used to improve the skeleton itself (e.g., if points 3 and 5 consistently conflict, the skeleton may be poorly structured and should be revised).

SoT as a data generation engine for self-improvement. The paper notes in Section 6 that "it is interesting to explore how the SoT answers can be used to fine-tune LLMs to generate more structured answers in a self-improving way." This connects to the broader STaR/ReST literature (Zelikman et al., 2022; Singh et al., 2024) where models generate their own training data. SoT offers a specific mechanism: use SoT to generate high-quality structured answers (which the paper shows are rated as more diverse and relevant; Figure 6), then fine-tune the base model on SoT-style question-and-structured-answer pairs, with the goal of making the model internalize the structured answering pattern so it produces multi-perspective, well-organized answers even under normal sequential decoding. This would effectively amortize the SoT overhead — you pay the parallel generation cost once during data creation, then the fine-tuned model produces structured answers at normal sequential cost during deployment. The key experiment would compare a SoT-fine-tuned model against the base model on both answer quality metrics and the tendency to produce structured answers (measurable as the fraction of answers that contain explicit section headers, bullet points, or numbered lists). A positive result would mean SoT enables a one-time compute investment that yields persistent quality improvement at zero additional inference cost.

Integration with serving systems for dynamic workload-adaptive batching. The paper evaluates SoT in an idealized single-user setting (Appendix E) and acknowledges that behavior under realistic serving system workloads is unexplored (Appendix H). A practically important follow-up would integrate SoT into an existing LLM serving framework (e.g., vLLM, TensorRT-LLM, or an open-source equivalent) and measure SoT's latency, throughput, and tail latency under varying concurrent user loads. The key design question is dynamic triggering: the serving system should activate SoT when GPU utilization is low (there is spare compute capacity that can be filled by parallel point expansions without competing with other users' requests) and deactivate it when utilization is high (to avoid throughput degradation from SoT's token overhead). This requires the router to take an additional input — current system load — and output a continuous or multi-level decision rather than a binary suitable/unsuitable. The experiment would characterize the throughput-latency Pareto frontier as a function of load and routing policy, producing operational guidance for when SoT should be enabled in production deployments. The paper's profiling-based latency estimation methodology (Appendix F) provides a foundation for building a simulator of SoT under load, which could be validated against real serving system measurements.

Prompt optimization for minimal SoT overhead. The token overhead ratios reported in Tables 6 (60–89× for API models) and 7 (30–39× for open-source models) are substantial enough to limit SoT's practical adoption in cost-sensitive settings. The paper gestures at prompt tuning (Jiang et al., 2023) but does not pursue it. A concrete follow-up would apply discrete prompt optimization (e.g., AutoPrompt-style token search; Shin et al., 2020) to find shorter skeleton and point-expanding prompts that elicit equivalent answer quality. The objective would be to minimize (skeleton prompt length + $B$ × point-expanding prompt length) subject to a constraint that answer quality (as measured by the GPT-4 judge) does not degrade below some threshold relative to the full-prompt SoT. This is a constrained discrete optimization problem that is newly tractable because the paper provides a reproducible evaluation pipeline (SoT on Vicuna-80, FastChat metric). A 2–3× reduction in prompt length would substantially reduce the throughput overhead and API cost, making SoT more competitive in saturated or cost-sensitive deployment scenarios. The paper's finding that GPT-4 can use a shorter, zero-shot skeleton prompt (Prompt 1 vs. Prompt 3) while other models need two-shot demonstrations already hints that prompt length can be reduced for more capable models — systematic optimization could push this further.

Practical Applications and Downstream Use Cases

Edge-side and single-user deployments where latency is the binding constraint. The paper explicitly identifies single-user edge applications as a scenario where SoT's latency-throughput tradeoff is favorable: "When there is an unsaturated number of concurrent queries, SoT can effectively reduce latency and enhance GPU utilization" (Section 6). This includes on-device LLM inference on laptops, phones, or dedicated edge hardware where only one user is querying the model at a time. In these settings, the GPU or NPU sits idle between tokens during sequential decoding (0.1% utilization; Table 5), and SoT's parallel point expansion fills that idle capacity without competing with other users. The paper's RTX 3090 results (Figure 13) demonstrate that SoT works on consumer-grade hardware: Vicuna-7B V1.3 achieves 1.94× speed-up, LLaMA2-Chat-7B achieves 2.40× speed-up. For a user waiting 15 seconds for a response, a 2× speed-up reduces that to 7.5 seconds — the difference between a tolerable interaction and a frustrating one. The SoT-R router's trained variant (0.04 seconds overhead; Appendix G.2) adds negligible latency, making adaptive SoT deployable on edge devices where the routing decision runs locally on the same hardware.

Interactive agent-to-agent communication where response latency gates overall system speed. The paper briefly notes that "reduced end-to-end latency can significantly benefit emerging application scenarios like agent-agent interaction" (Appendix L). In multi-agent systems where one LLM's output is another LLM's input (e.g., HuggingGPT-style task orchestration; Shen et al., 2023, or multi-agent debate frameworks), the end-to-end system latency is the sum of sequential agent latencies. If each agent's response time can be reduced by ~2× through SoT, the overall system speed-up compounds across the agent chain. For a pipeline of 3 sequential agent interactions, a 2× speed-up per agent yields an 8× reduction in total latency versus a 2× reduction for a single-agent scenario. The SoT structure is particularly well-suited to agent communication because agents often need to provide structured outputs (task plans, status reports, multi-part responses) that naturally decompose into independently expandable points. The key deployment consideration is whether the agent's output format tolerates the skeleton headers in the final answer — if downstream agents parse the output programmatically, the headers may need to be stripped via post-processing.

Batch answer generation for dataset creation and model evaluation. When using LLMs to generate training data (instruction tuning datasets, synthetic Q&A pairs, or evaluation benchmarks), throughput matters — you need to generate many answers quickly. But SoT's parallel mechanism can be applied even in throughput-oriented settings if the generation pipeline is restructured: rather than generating 1,000 answers sequentially, generate 1,000 skeleton stages in parallel (as separate queries in a batch), then generate all point expansions for all 1,000 questions as a single large batch. The point-expanding stage would contain $1,000 × B_{avg}$ requests, where $B_{avg}$ is the average number of skeleton points per question. This batched parallel generation would have higher GPU utilization than sequential generation while maintaining SoT's structured-answer quality benefits (improved diversity and relevance; Figure 6). The paper's finding that SoT answers are 1.1–6.8× longer (Figure 11f) is actually an advantage in data generation contexts where comprehensive, multi-perspective answers are more valuable training examples than terse ones. The tradeoff is that SoT-generated training data has a distinctive format (skeleton headers, bullet-point structure) that may bias models fine-tuned on it toward producing structured outputs — whether this is a feature or a bug depends on the downstream application.