Speak While You Think: Streaming Speech Synthesis During Text Generation

ArXiv: 2309.11210

🎯 Pitch

A streaming speech synthesizer can read aloud LLM outputs as they are generated without any drop in naturalness, matching a non-streaming teacher that sees full sentences. It achieves this by distilling the teacher's phone and prosody predictions using the LLM's own hidden semantic embeddings, with just a single lookahead word recovering nearly all of the phonetic accuracy.

---

1. Executive Summary

This paper introduces LLM2Speech, an architecture that synthesizes speech incrementally as an LLM generates text, reducing the latency that typically makes TTS-mediated voice conversations impractical. The system chains a frozen T5 language model with a streamable LLM2PnP module — which predicts phones and prosody from LLM tokens and hidden embeddings using restricted-attention knowledge distillation — and a chunked PnP2Speech acoustic model (adapting non-attentive Tacotron and LPCNet for streaming). In listening tests, LLM2Speech achieves a MOS of 4.12 ± 0.04, matching the offline teacher's 4.10 ± 0.04 with no statistically significant difference despite the teacher's access to full-sentence context and sub-style labels. A G2P ablation reveals that the first word of lookahead is the most crucial for phonetic accuracy (reducing WER from 6.40% to 1.95%), while LLM embeddings from larger or deeper models improve G2P performance but contribute less than an additional lookahead word, establishing that streaming prosodic quality can be preserved through offline-to-streaming distillation even when the conversational style of the training corpus differs from the deployment domain.

2. Context and Motivation

The Core Problem: Latency in Text-to-Speech Makes Voice Interactions with LLMs Infeasible

The fundamental problem this paper addresses is deceptively simple: when you ask an LLM a question aloud and it types out the answer, the resulting delay before you hear speech makes conversation impossible. Currently, interacting with large language models is overwhelmingly text-mediated. You type a prompt, the model generates text token by token, and you read the response. But in many real-world scenarios — driving, walking, operating machinery, or any hands-busy-eyes-busy situation — text interaction is impractical, unsafe, or simply undesirable. Voice is the natural modality for these contexts, and the expectation for voice interaction is fluent, low-latency turn-taking where the system begins speaking almost immediately after you finish talking.

The mismatch between how LLMs generate text and how TTS systems consume it creates an architectural latency bottleneck. An LLM is autoregressive: it produces one token at a time, each conditioned on all previous tokens. A modern TTS system, by contrast, typically wants the entire sentence before it begins synthesis. This is because producing natural-sounding speech requires knowing what comes next — things like prosodic phrasing, word-level prominence, cross-word-boundary pronunciation changes (flapping, reduction), and disambiguation of heteronyms (e.g., "lead" the metal vs. "lead" the verb) all depend on right-context that hasn't been generated yet if you're streaming token-by-token from an LLM.

The consequence is a stark trade-off: either you wait for the LLM to generate the complete response (adding seconds of silence before the user hears anything), or you synthesize speech from partial text and accept degraded quality (mispronounced words, unnatural phrasing, incorrect prosody). Neither option supports a natural conversational experience. The latency problem is structural, not merely a matter of faster hardware — it stems from the opposing temporal assumptions of generation (left-to-right, incomplete) and synthesis (whole-utterance, context-rich).

Why This Problem Matters

The significance of solving this extends beyond convenience. It touches on fundamental questions about how humans will interact with increasingly capable AI systems, and it has immediate practical implications for deployment.

Real-world impact. Voice assistants and conversational agents are already widely deployed (Siri, Alexa, Google Assistant), but their interactions are rigid, turn-by-turn, and limited to short, pre-scripted domains. The arrival of LLMs with open-ended conversational capabilities creates the possibility of much richer spoken interactions — brainstorming sessions, verbal tutoring, interactive storytelling, therapy-like conversations — but only if the synthesis latency can be brought low enough that the interaction feels like talking to a person, not a radio with a delay. The paper explicitly mentions applications like driving assistance (Section 1), where the safety implications of latency are obvious: a driver needs route guidance or hazard information now, not after a two-second pause while the LLM finishes generating a response.

This isn't a niche concern. As the paper notes, "pure audio-based LLMs are gaining interest in the community, though their semantic language understanding capabilities still lag behind textual LLMs" (Section 1). The implication is clear: text-based LLMs are currently far more capable than audio-native models for understanding and reasoning. Rather than waiting for audio-native models to catch up — which could take years and enormous compute — a practical bridging solution is to make text-based LLMs sound like they're speaking naturally and immediately. That's exactly what LLM2Speech aims to do.

Theoretical significance. The problem exposes an interesting tension in sequence-to-sequence architectures. LLMs and neural TTS systems are both autoregressive in different ways, and their assumptions about what constitutes adequate context for high-quality output are incompatible. Solving this requires understanding what information is actually necessary at each stage of the synthesis pipeline — when does phonetic disambiguation need right-context? How much prosodic structure can be predicted from left-context alone? What semantic cues from the LLM's internal representations can substitute for missing future text? These questions are broadly relevant to any system that composes sequential models with different context requirements, not just speech synthesis.

Prior Approaches and Where They Fall Short

The paper identifies several existing approaches for reducing TTS latency, but argues that none adequately addresses the specific scenario of synthesizing from LLM-generated text incrementally.

1. Standard chaining of LLM + TTS (Section 1). This is the baseline approach: wait for the LLM to produce a complete response, then pass it to the TTS system. Neural TTS models such as Tacotron (Shen et al., 2018) and neural codec language models (Wang et al., 2023) can produce high-fidelity speech from full-sentence text. But the paper states bluntly: "TTS models, however, often require an entire sentence to generate natural speech, resulting in notable latency when combined with an LLM that typically generates text in a slow autoregressive fashion." An LLM might generate 20–50 tokens to form a multi-sentence response, each token taking tens of milliseconds of compute. A user might wait 2–5 seconds before the TTS system even begins processing. That's an eternity in conversation, where turn-taking gaps are typically on the order of 200 ms.

2. Incremental TTS with limited lookahead (Section 1). There is existing work on incremental or streaming TTS systems that restrict right-context to reduce algorithmic delay. The paper cites several approaches:

• Prefix-to-prefix frameworks (Ma et al., 2020): these train a streaming TTS that conditions on partial text prefixes, aiming to match full-sentence quality without access to the complete utterance.
• Transducer-based approaches (Speech-T, Chen et al., 2021): these reformulate TTS as a transducer, enabling streaming alignment between text and speech.
• Transformer-based streaming (Wu et al., 2021): these apply incremental processing techniques to transformer TTS architectures.
• Sentence-length-independent latency systems (Ellinas et al., 2020): these target low, fixed latency regardless of sentence length.

These works make progress on reducing the delay within the TTS system itself, but the paper points out a crucial limitation: "In most scenarios, the entire text is available before synthesis, and the focus is on reducing algorithmic delay." That is, these systems assume the input text is already complete. They can afford a "lightweight G2P module" that runs on the full sentence before streaming synthesis begins, "without significantly contributing to the overall delay." When the text is being generated in real time by an LLM, this assumption breaks. The input stream itself is slow and incomplete. The challenge isn't just reducing TTS algorithmic delay — it's handling the fact that the text arrives word-by-word, with an unknown total length, and synthesis decisions must be made incrementally without knowing what the next word will be.

3. Context-dependent G2P methods (Section 1). For languages with irregular orthography like English and French, accurate pronunciation often requires more than a single word's spelling. Context-dependent grapheme-to-phoneme approaches (Ploujnikov and Ravanelli's SoundChoice, 2022; Rezáčková et al.'s T5G2P, 2021; Zhu et al.'s ByT5 approach, 2022) look at surrounding words to handle cross-word-boundary phenomena like flapping (where "butter" and "but a" involve different sounds at word boundaries), vocalic reduction, and heteronym disambiguation. However, "the context required for disambiguation may be long, rendering them unsuitable for streaming." If the G2P module needs to see two words ahead to correctly pronounce the current word, it must wait for the LLM to generate those words — defeating the purpose of streaming.

4. Pure audio-based LLMs (Section 1). AudioLM (Borsos et al., 2023) and AudioGen (Kreuk et al., 2023) represent an entirely different approach: building language models that operate directly in the audio domain, bypassing text and TTS entirely. These are appealing because they eliminate the text bottleneck. However, the paper notes a critical limitation: "their semantic language understanding capabilities still lag behind textual LLMs." Audio-native models must learn language understanding, reasoning, and speech generation simultaneously from audio data, which is far less abundant and less structured than text corpora. For high-stakes conversational applications that require sophisticated reasoning — answering complex questions, providing accurate information, maintaining coherent multi-turn dialogue — text-based LLMs currently have a decisive advantage.

How This Paper Positions Itself

LLM2Speech doesn't propose a fundamentally new TTS architecture or a new LLM. Instead, it bridges the gap between an existing frozen LLM and an existing TTS backbone by introducing two key ideas: exploiting the LLM's internal representations as a source of semantic context that can partially compensate for missing future text, and training the bridge component via offline-to-streaming knowledge distillation that mimics a full-context teacher while operating under severe lookahead constraints.

The paper frames this as a system integration problem more than an algorithmic novelty problem. The LLM is deliberately frozen — "due to the vast computational and human effort invested in bringing it to its final state" (Section 1). This is a practical stance: it would be unrealistic to retrain or fine-tune a large-scale production LLM for each deployment, and doing so might compromise its carefully calibrated reasoning abilities (potentially introducing catastrophic forgetting or alignment drift). Instead, LLM2Speech treats the LLM as a black-box generator that produces text tokens and hidden embeddings simultaneously, tapping into those embeddings as a free source of contextual information.

The architecture inserts two trained components between the LLM output and the audio waveform:

• LLM2PnP: a transformer encoder-decoder that converts LLM tokens and embeddings into phones and prosodic features (HPCs), trained to match a full-sentence teacher's predictions under restricted attention.
• PnP2Speech: a chunked version of the PPT acoustic model (Shechtman and Fernandez, 2023), adapted from non-attentive Tacotron with LC-CNN and LC-BLSTM layers that enforce fixed lookahead.

The training methodology is explicitly offline-to-streaming knowledge distillation (Section 1), drawing on prior work in speech recognition by Povey et al. (2018) and Kurata and Saon (2020). The key idea is to train the streaming student model on "pseudo-labeled" outputs from a teacher that has full access to future context, forcing the student to learn how to make the same predictions with only left-context and a small fixed lookahead. In the context of TTS, this means training LLM2PnP to produce the same phone sequence and prosodic controls that the teacher would produce given the full sentence, even though the student only sees words up to one or two positions ahead. This is possible because the LLM embeddings carry semantic information — the LLM, having processed the entire preceding context (the user's prompt and the conversation history), encodes information about what it's likely to generate next. These embeddings can help disambiguate pronunciation (e.g., the word "record" as noun vs. verb might be inferable from the semantic context) and guide prosodic choices (e.g., sentence boundaries, emphasis) even before the complete text exists.

The paper also emphasizes that this approach handles expressive speech, including interjections and filled pauses ("hmm, uh-huh, oh"), which are essential for natural conversation. Since the conversational training corpus includes these non-lexical vocalizations — and since the LLM can generate them as text tokens — the system learns to synthesize them appropriately, unlike most TTS systems that are trained on read speech and struggle with conversational phenomena.

In essence, LLM2Speech positions itself not as a novel TTS algorithm but as a practical architectural solution to the LLM-TTS latency gap, demonstrating that with the right bridging components and training strategy, a frozen text LLM can drive a streaming speech synthesizer without the quality compromises that historically accompanied incremental TTS. The experimental validation — showing MOS parity with a full-context teacher — makes the case that this isn't merely a latency-vs-quality trade-off, but a genuinely preserved quality level.

3. Technical Approach

3.1 Reader Orientation

LLM2Speech is a pipeline that converts text tokens being generated one-at-a-time by a frozen large language model into spoken audio with minimal delay, so a user hears the response being spoken aloud almost as soon as the LLM starts producing it. The system solves the latency mismatch between autoregressive text generation (which produces tokens slowly and incrementally) and conventional TTS (which expects complete sentences to synthesize natural-sounding speech) by inserting a trainable bridge module that exploits the LLM's hidden embeddings — semantic and contextual information available as a free by-product of text generation — to predict pronunciation and prosody under a tight lookahead constraint, enabling streaming synthesis without sacrificing the naturalness achieved by full-context systems.

3.2 Big-Picture Architecture (Diagram in Words)

The system has three major components arranged in a pipeline (Figure 1):

1. A frozen pretrained LLM (T5) — generates text tokens one at a time in response to a prompt and simultaneously produces hidden-state embeddings from its internal layers. The LLM is deliberately frozen to preserve the massive investment in pretraining and fine-tuning.

2. LLM2PnP — a transformer encoder-decoder that takes the stream of LLM tokens and their contextual embeddings as input and outputs a sequence of phones and prosody tokens (PnP). This is the trained bridge component; it operates under restricted attention so that each output phone can only see input tokens up to a fixed small number of words ahead of the current word.

3. PnP2Speech — a streamable acoustic model (adapted from a non-attentive Tacotron backbone followed by an LPCNet vocoder) that consumes PnP tokens in small chunks and produces 22 kHz audio waveforms. Its internal layers are modified to enforce constrained right-context lookahead using chunked BLSTMs and lookahead-constrained CNNs.

Information flows sequentially: the LLM incrementally emits token $t_i$, along with its internal embeddings → LLM2PnP processes the growing sequence of token-embedding pairs and incrementally produces phone-prosody tokens → PnP2Speech consumes chunks of these PnP tokens and generates the corresponding audio frames. The total algorithmic delay from LLM token to audio output is approximately two words.

3.3 Roadmap for the Deep Dive

• First, dataset creation — how training data is constructed to simulate the streaming LLM-text-to-speech scenario, including the pseudo-labeling strategy for phones and prosody, because this defines the input-output pairs that everything else is trained to replicate.

• Second, the restricted attention mechanism in LLM2PnP — the formal rules that limit how far ahead any output phone can look when attending to input tokens, because this is the core architectural constraint that enables streaming and determines the latency-quality trade-off.

• Third, the LLM2PnP model architecture and training — the transformer encoder-decoder design, how LLM embeddings are injected, how the three prediction heads produce phones, prosodic features, and phrase types, and how offline-to-streaming knowledge distillation shapes the training objective.

• Fourth, the PnP2Speech streaming acoustic model — the modifications to Non-Attentive Tacotron and LPCNet that enforce constrained lookahead, including LC-BLSTMs, LC-CNNs with skewed kernels, and Gaussian upsampling with guardbands, because this is the component that actually produces audio under latency constraints.

• Fifth, the end-to-end inference procedure — how text, embeddings, phones, and audio frames flow through the system in a coordinated streaming fashion, including the word-boundary gating that synchronizes LLM generation with speech synthesis.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems integration and architectural modification paper whose core idea is that a frozen LLM's hidden embeddings carry sufficient semantic context to enable accurate streaming phone-and-prosody prediction under tight lookahead constraints, provided the prediction model is trained via offline-to-streaming knowledge distillation from a full-context teacher.

---

Dataset Creation: Simulating Incremental Generation with Full-Context Labels

Before any model can be trained, the paper must construct a training dataset that simulates the streaming text-generation scenario while providing ground-truth target outputs — phones and prosodic features — that a full-context system would produce. This is non-trivial because the training data must pair partial, incrementally-arriving text with the phonetic and prosodic annotations that a complete-sentence system would assign to that text, so that the streaming model can learn to make the same predictions without seeing the future.

Corpus and LLM selection. The paper uses the T5 language model (Raffel et al., 2020) specifically because it is a text-to-text transformer that can perform diverse conditional generation tasks. The particular variant is T5-lm-adapt, which was fine-tuned for text completion — meaning it's trained to generate plausible continuations given a preceding context. The training corpus is the C4 dataset (Common Crawl Cleaned Corpus), the same dataset used to pretrain T5, which contains 365 million samples of cleaned web text. For computational feasibility, the paper uses a random subset of 3 million training samples and 130,000 validation samples.

Why T5 rather than a more recent decoder-only LLM? T5's encoder-decoder architecture provides a clean separation between input context and generated continuation, which is convenient for constructing the training task. The encoder processes a context paragraph while the decoder autoregressively generates the continuation — this mirrors the streaming scenario where the LLM has already processed the user's input and is producing a response token by token. The paper notes in a footnote that for decoder-only LLMs, the text split step described below is not needed, since the entire prompt-response sequence would be processed causally.

Splitting text into context and text-to-predict. Each C4 sample is a paragraph of text. The paper randomly splits each paragraph into two parts: a context portion and a text-to-predict (t2pred) portion, such that t2pred contains 1–5 sentences. The context portion is fed to the T5 encoder, simulating the user's prompt and conversation history that the LLM has already processed. The t2pred portion is fed to the T5 decoder, simulating the incrementally-generated response that will be spoken aloud.

This randomized split serves an important purpose: it creates training pairs where the LLM has been conditioned on preceding text (so its hidden embeddings carry genuine semantic context) and is generating a continuation (so the text-to-be-synthesized arrives token by token from left to right). The split is random rather than semantic because the goal is to expose LLM2PnP to a broad distribution of context–continuation pairs during training, not to enforce any particular discourse structure.

Extracting tokens and contextual embeddings. With the split defined, the paper runs the T5 model forward: context through the encoder, t2pred through the decoder. For each word token in t2pred, the system records two things (Figure 2, above):

1. The token identity itself — the subword unit (e.g., "lead", "ing") that the LLM produced.
2. The contextual embedding — the hidden-state vector from specified layers of the T5 decoder at the position where that token was generated. These embeddings incorporate information from the entire context (processed by the encoder) plus all preceding tokens in the continuation (processed causally by the decoder's self-attention). The paper uses embeddings from T5-Base layers 2, 6, and 10 (out of 12 total, numbered from input to output), selecting multiple layers to capture both lower-level syntactic and higher-level semantic features.

The token-embedding pairs form the input to LLM2PnP during training: a sequence of (token, [emb_layer2, emb_layer6, emb_layer10]) tuples representing the incrementally-generated text stream.

Pseudo-labeling phones and prosody with the teacher. To create targets for LLM2PnP to predict, the paper runs a teacher model with no lookahead restrictions on the complete t2pred text. The teacher is described as a "rules-based G2P model predicting the phonetic sequence and phrase type, followed by a neural model predicting Hierarchical Prosodic Controls (HPCs) for an expressive conversational speaking style and phone durations."

HPCs (Hierarchical Prosodic Controls), introduced in prior work by Shechtman et al. (2021) and used in Shechtman and Fernandez (2023), are speaker-agnostic prosodic statistics computed from recordings at multiple temporal resolutions. The paper uses three types of HPCs:

• Duration HPCs: capturing how long each phone, word, and sentence should last relative to the speaker's typical durations.
• Pitch HPCs: capturing the pitch contour at phone, word, and sentence levels (e.g., rising at the end of questions, falling at the end of statements).
• Maximal log-energy HPCs: capturing the loudness contour, also evaluated at sentence, word, and phone hierarchies.

The teacher processes the complete t2pred text — it sees all words, including those that haven't been "generated" yet in the streaming scenario — and produces a sequence of PnP tokens: each token specifies a phone identity (e.g., /k/, /æ/, /t/ for "cat"), its prosodic features (duration, pitch, energy HPC values), and a phrase type marker (e.g., end of phrase, continuation). This forms the pseudo-ground-truth that LLM2PnP will be trained to predict.

An important preprocessing detail concerns text normalization expansions. When the text contains numerals, abbreviations, or symbols (e.g., "23" should be spoken as "twenty-three"), the normalization changes the token-to-phone mapping (the word "23" maps to a single token but multiple phones). To handle this correctly, the system differentiates between regular word separators (spaces between words as they appear in the original text) and inner word separators (spaces inserted only when normalization expands a token into multiple words). During inference, LLM2PnP synthesizes phones for a word only until it encounters a regular word separator, then waits for the LLM to produce the next word token. This gating mechanism ensures the system doesn't start speaking a word before it knows what the word is, even when normalization changes the token-to-speech mapping.

Teacher model outputs. The teacher produces a sequence of PnP labels for each word in t2pred. A PnP label includes:

• The phonetic identity (which phone to produce).
• HPC values for duration, pitch, and energy at the phone, word, and sentence hierarchies.
• A phrase type tag (e.g., declarative, interrogative, continuation).

The full sequence of phone-HPC-phrase tokens serves as the target that LLM2PnP must learn to predict under lookahead constraints, effectively distilling the teacher's full-context knowledge into a streaming-capable student model.

---

Restricted Attention: The Formal Mechanism That Enables Streaming

The central architectural constraint in LLM2PnP is restricted attention — a modification to standard transformer attention (Vaswani et al., 2017) that limits which input tokens each output token can attend to, based on word position. Standard self-attention allows every position to attend to every other position, which means a phone prediction at the beginning of the sentence could be influenced by a word token at the very end — this is incompatible with streaming, where future words haven't been generated yet.

Token-to-word mapping. To define the restriction formally, the paper introduces notation that groups tokens into words. Let $w_1, w_2, \ldots, w_n$ be the sequence of words, and let $t_1, t_2, \ldots, t_m$ be the sequence of LLM word-piece tokens (subword units). Each token $t_j$ belongs to exactly one word $w_i$, which is written as:

$\text{word}(t_j) = i$

meaning token $j$ is part of word $i$. Similarly, the output PnP tokens $p_1, p_2, \ldots, p_k$ are phones that correspond to words, and $\text{word}(p_j) = i$ indicates that phone $j$ belongs to word $i$.

Regular attention. In a standard transformer encoder-decoder, every encoder token can attend to every other encoder token (encoder self-attention), and every decoder token can attend to every encoder token (encoder-decoder attention). Formally, for all $i$ and $j$:

$t_i \rightarrow t_j \quad \text{(encoder self-attention)}$ $p_i \rightarrow t_j \quad \text{(encoder-decoder attention)}$

where $y \rightarrow x$ means "output token $y$ can attend to input token $x$." In the streaming scenario, this is problematic: phone $p_1$ (the first phone of the first word) could attend to $m$ words ahead if those tokens existed, violating the causal constraint.

Restricted encoder self-attention. The paper defines restricted attention so that a token $t_i$ can only attend to tokens $t_j$ belonging to the same word or earlier words:

$t_i \rightarrow t_j \iff \text{word}(t_j) \leq \text{word}(t_i)$

where $\text{word}(t_j)$ is the word index of the input token being attended to, and $\text{word}(t_i)$ is the word index of the token doing the attending.

What it computes: A binary mask that removes attention links from any token to tokens that belong to words that appear after the current word in the text. If the current token is part of word 3, it can attend to tokens in words 1, 2, or 3, but not to tokens in words 4, 5, etc.

Why this form: This is a causal constraint at the word level rather than the token level. The simpler alternative — restricting by token position (token $i$ cannot attend to token $i+k$) — would be inappropriate because word boundaries are the natural syntactic and semantic units. A word can contain multiple subword tokens, and the pronunciation of a word depends on its own tokens (which may span several positions) but should not depend on future words' tokens. The word-level restriction correctly captures this: a word's own subword tokens are all accessible (since they share the same word index), but tokens of future words are not.

Restricted encoder-decoder attention. For the cross-attention between decoder phones and encoder tokens, the restriction is similar but includes a fixed word lookahead $L$:

$p_i \rightarrow t_j \iff \text{word}(t_j) \leq \text{word}(p_i) + L$

where $L$ is a non-negative integer specifying the number of future words that each phone can "see" when attending to input tokens, $\text{word}(p_i)$ is the word that phone $i$ belongs to, and $\text{word}(t_j)$ is the word that encoder token $j$ belongs to.

What it computes: A binary mask that allows phone $p_i$ to attend to tokens from words up to $L$ words beyond its own word. With $L = 1$, a phone in word 3 can attend to tokens from words 1, 2, 3, and 4. With $L = 0$, it can only see up to its own word.

Why this form: The paper explicitly chose to place the lookahead in the encoder-decoder attention rather than the encoder self-attention. The reason is architectural: "it would not grow in consecutive decoder layers, unlike encoder attention, where the lookahead would grow linearly with the number of layers." In a multi-layer transformer, if lookahead were implemented in the encoder self-attention, each encoder layer would expand the receptive field by its own lookahead amount, and the cumulative effect after $N$ layers could be an effective lookahead of $N \times L$, violating the intended constraint. In the encoder-decoder attention, the lookahead is applied once per decoder layer, and since the decoder's self-attention is also restricted (it processes PnP tokens left-to-right), the total lookahead remains bounded at $L$ words regardless of the number of layers.

Visualizing the restriction. Figure 3 illustrates the difference. In regular attention (Figure 3a), every encoder token can attend to every other encoder token (the attention matrix is full), and every decoder token can attend to every encoder token. In restricted attention with $L = 0$ (Figure 3b), the attention matrix is block-diagonal: tokens belonging to the same word form a fully-connected block, but tokens from different words have directed edges only from later words to earlier words (or none, for encoder-decoder with $L = 0$). The paper explicitly marks cases like "$t_1 \not\rightarrow t_3$" and "$p_1 \not\rightarrow t_3$" — the first phone of the first word cannot attend to tokens from the second word, ensuring that the prediction of $p_1$ does not depend on $t_3$.

The total lookahead of the full LLM2Speech system sums to two words: one from LLM2PnP's $L = 1$ word lookahead, and approximately one more from PnP2Speech's algorithmic delay (equivalent to roughly one word, as analyzed in Section 2.3). This two-word lookahead is the key architectural parameter governing the latency–quality trade-off.

---

LLM2PnP: Architecture, Training, and the Role of LLM Embeddings

LLM2PnP is the central trained component that bridges the LLM's token-output stream to the TTS system's phone-and-prosody input. It is a transformer encoder-decoder model (Vaswani et al., 2017) with three prediction heads that share the decoder's output representations but produce different aspects of the PnP output.

Encoder: processing LLM tokens and embeddings. The encoder input is a sequence of concatenated vectors, where each position corresponds to one LLM word-piece token. For each token, the input vector is formed by concatenating:

• The token's embedding (presumably a learned embedding or the LLM's own token representation — the paper is not explicit about whether the token identity is embedded by LLM2PnP or inherited from the LLM, but the standard approach would be to embed the token ID and combine it with the LLM's hidden states).
• The LLM's contextual embeddings from the specified hidden layers — for the T5-Base configuration, these are layers 2, 6, and 10.

The LLM embeddings are projected to the encoder dimension using a linear layer. Since the LLM's hidden states may have a different dimensionality than LLM2PnP's encoder (T5-Base has 768-dimensional hidden states; LLM2PnP uses a 512-dimensional token/feedforward dimension), the linear projection maps the LLM embeddings to the encoder's working dimension.

This concatenation-plus-projection design is a crucial architectural choice because it means the encoder sees, at each position, both what word is being generated (the token identity) and the LLM's internal representation of the semantic and syntactic context (the hidden embeddings). The token identity alone would contain only the surface form, but the LLM embeddings encode information about what the LLM "intends" — whether the current token is part of a question, a statement, an uncertain hedge, an excited exclamation, etc. This semantic context can guide pronunciation disambiguation and prosodic choices even when future words are unavailable.

Decoder: autoregressive PnP generation. The decoder generates the PnP sequence autoregressively, conditioning on both the encoder output (via restricted encoder-decoder attention with $L = 1$) and its own previously generated PnP tokens (via causal self-attention). At each decoder step, the decoder's hidden state is passed to three separate prediction modules:

1. Phone identity predictor: outputs a probability distribution over the phone vocabulary, selecting which phone to produce next (e.g., /p/, /iː/, /t͡ʃ/ for "peach").

2. Prosodic feature predictor: outputs the HPC values — continuous scalars representing duration, pitch, and energy characteristics at the current phone's level, plus the word-level and sentence-level HPCs at word and sentence boundaries.

3. Phrase type predictor: outputs a tag indicating prosodic phrasing (e.g., end of declarative sentence, end of question, within-phrase continuation, phrase boundary with short pause, etc.).

These three modules share the decoder's transformer layers but have independent output projections, meaning the decoder learns representations that are jointly useful for predicting pronunciation, rhythm, and phrasing simultaneously.

Model scale. LLM2PnP has 4 encoder layers and 6 decoder layers. Each layer uses:

• Token/feedforward dimensions of 512/768.
• 4 attention heads.
• A single-word lookahead ($L = 1$) in the encoder-decoder attention.
• T5-Base embeddings from layers 2, 6, and 10 (out of 12 total layers).

The choice of a single-word lookahead is informed by the G2P ablation results in Table 2, which show that increasing from $L = 0$ to $L = 1$ reduces the word error rate from 6.40% to 1.95% on all words (a dramatic improvement), while increasing further to $L = 2$ yields only a modest additional gain (1.95% → 1.69%). This suggests that in US English, the most critical cross-word pronunciation phenomena — flapping across word boundaries ("but a" → [bʌɾə]), vocalic reduction at word junctures, and certain heteronym disambiguations — require at most one word of right context to resolve. The second lookahead word helps marginally, likely for rarer phenomena like long-distance prosodic phrasing decisions.

Training via offline-to-streaming knowledge distillation. Rather than training LLM2PnP to directly produce correct phones and prosody (which would require a parallel corpus of text with phonetic annotations), the paper trains it to mimic the teacher model's predictions. The teacher has full access to the complete t2pred text and produces a sequence of PnP tokens. LLM2PnP is trained with a standard sequence-to-sequence loss — likely cross-entropy for the discrete phone identity and phrase type predictions, and mean squared error or similar regression loss for the continuous HPC values — between its own predictions (made under the restricted lookahead) and the teacher's targets.

This is an instance of knowledge distillation, specifically the offline-to-streaming variant described in prior work on speech recognition (Povey et al., 2018; Kurata and Saon, 2020). The key property is that the student model learns to implicitly compensate for missing future context by exploiting information that is available in the LLM embeddings. For example, if the LLM embedding at the current token suggests that a question is being formed (because the embedding encodes the syntactic structure "What is the..."), the student can predict rising phrase-final intonation even before it sees the question mark token — because it has learned the correlation between the LLM's internal state and the teacher's prosodic labels.

Why the LLM is frozen. The paper states this explicitly: "we deliberately freeze [the LLM] due to the vast computational and human effort invested in bringing it to its final state." Fine-tuning the LLM to produce better embeddings for TTS would risk degrading its text generation capabilities (catastrophic forgetting), misaligning it from its safety training (Ouyang et al., 2022), and would be computationally prohibitive for each new deployment. The frozen-LLM design means LLM2Speech can be attached to any compatible pretrained LLM without modifying it — a modular approach that preserves the LLM's existing capabilities.

Conversational fine-tuning. After initial training on the large C4-derived dataset (which consists of written, edited text), LLM2PnP is fine-tuned on a conversational speech corpus — the same 6.5-hour proprietary dataset used to train PnP2Speech, recorded by a professional US-English female speaker with a variety of expressive dialog acts and interjections. This fine-tuning step adapts the model from the written domain (where sentences are complete, grammatical, and lack disfluencies) to the conversational domain (where utterances may be fragmentary, include filled pauses like "um" and "uh-huh," and express a wider range of prosodic styles like empathy, excitement, and uncertainty). The fine-tuning trains LLM2PnP to predict the conversational set of PnP targets from the conversational text, teaching it to handle phenomena absent from the C4 training data.

---

PnP2Speech: Streaming Acoustic Model with Constrained Lookahead

PnP2Speech is the acoustic component that converts the PnP token sequence into an audio waveform. It is a streaming adaptation of the HPC-based Parallel Prosody Transfer (PPT) model (Shechtman and Fernandez, 2023), which itself is built on the Non-Attentive Tacotron (NAT) backbone (Shen et al., 2020) followed by a lightweight LPCNet vocoder (Valin and Skoglund, 2019). The adaptation makes the entire acoustic pipeline streamable by modifying three types of neural network layers to operate with fixed, bounded right-context.

Non-Attentive Tacotron recap. A standard Tacotron TTS system uses an attention mechanism to align input text (or phone) tokens with output acoustic frames (mel-spectrograms), then generates audio from those frames using a vocoder. Non-Attentive Tacotron replaces the attention mechanism with a Gaussian upsampling layer that expands the input token sequence to the output frame sequence using learned, deterministic alignment parameters (a predicted duration for each token, which determines a Gaussian window centered at the corresponding output time). This eliminates the attention failures (skipping, repeating) that can occur in standard Tacotron, making it more robust — and, critically for streaming, it means the alignment is purely local: each output frame depends on input tokens within a fixed temporal window determined by the Gaussian kernel's width.

The acoustic model architecture in PPT/NAT is: input PnP tokens → phonetic encoder (CNN + BLSTM layers) → Gaussian upsampling → autoregressive LSTM decoder → PostNet (CNN layers) → output mel-spectrogram frames → LPCNet vocoder → audio waveform.

Challenge 1: BLSTMs have infinite right-context. BLSTM (Bidirectional Long Short-Term Memory) layers process sequences both forward and backward, meaning each output at position $i$ depends on the entire forward pass from $0$ to $i$ and the entire backward pass from $N-1$ to $i$. In a streaming setting, the backward pass cannot be computed until the entire sequence is available.

Solution: Chunked BLSTM (LC-BLSTM). The paper replaces BLSTM layers with LC-BLSTM layers with zero lookahead ($la = 0$). LC-BLSTM (Zhang et al., 2016) processes the sequence in fixed-size chunks: each chunk of, say, 4 or 8 frames is processed independently, with the forward LSTM running across the chunk and the backward LSTM running within the same chunk (or across a slightly extended window if non-zero lookahead is permitted). With $la = 0$, the backward pass is limited to the current chunk, meaning the effective right-context is at most one chunk length — and since the chunk size is small (4 in this paper), the latency is bounded and independent of utterance length.

The paper specifies: "Chunked BLSTM layers have a chunk size of 4." This means that when the BLSTM processes a frame at position $i$, it can see at most 3 frames to the right (since the chunk spans 4 frames and includes positions $i$ through $i+3$). This is a dramatic restriction compared to the unlimited right-context of a full BLSTM, but it's necessary for streaming.

Challenge 2: CNN receptive fields grow with depth. Standard convolutional layers with symmetric kernels (e.g., kernel size 3, centered on the current position) expand the right-receptive field by one position per layer: after one $k=3$ layer, a position can see one frame to the right; after two such layers, it can see two frames to the right; and so on. The total right-receptive field after $D$ layers is $D \times (k - 1) / 2$, which can become large for deep networks.

Solution: Lookahead-Constrained CNN (LC-CNN). The paper introduces a mechanism to cap the right-receptive field at a fixed value $la$. For each convolutional layer, the system uses a symmetric-kernel convolution if the lookahead constraint permits it (i.e., if using a center-aligned kernel would not cause the cumulative right-receptive field to exceed $la$). Otherwise, it applies a skewed-kernel convolution — a generalization of causal convolutions (van den Oord et al., 2016) where the kernel is shifted leftward so that its rightmost weight aligns with the allowed future position. Figure 4 illustrates this: with a $k=3$ kernel and a target lookahead of $la = 1$, the first convolution is symmetric (center-aligned), but the second must be skewed so that the final output at position $p_5$ has a right-receptive field of exactly 1, not 2.

The paper specifies: "LC-CNN layers have a kernel size of 5 and a lookahead of 2." This means the cumulative right-receptive field across all LC-CNN layers is capped at 2 frames, and the kernel size of 5 provides a total receptive field of 5 (2 to the left, 1 at center, 2 to the right at the final layer, though intermediate layers may be skewed to stay within budget).

Challenge 3: Gaussian upsampling requires future phones for alignment. The Gaussian upsampling layer maps each input phone to a window of output frames centered at a position determined by cumulative phone durations. To compute the Gaussian window boundaries accurately — specifically, to know when one phone's influence ends and the next begins — the upsampling needs to know the duration of the following phone. In offline mode, all durations are available, but in streaming, the next phone may not have been generated yet.

Solution: Guardbands. The paper includes guardbands when chunking the PnP-to-frame Gaussian upsampling matrix: "the upsampling depends on the adjacent future PnP, [so] we include guardbands." A guardband means that when processing a chunk of PnP tokens, the system includes a small number of future PnP tokens beyond the nominal chunk boundary, uses them to compute correct upsampling weights at the boundary, but does not yet produce audio for those future tokens. The guardband size is specified as 2 phones.

Overall algorithmic delay of PnP2Speech. The paper states that the proposed PnP2Speech system results in "an algorithmic delay of 6 PnP tokens plus 2 frames, which is approximately equivalent to one word (where word separators and pauses are also considered as PnP tokens)." The 6 PnP tokens come from: the chunked BLSTM (chunk size 4, contributing up to 3 PnP tokens of right-context), the LC-CNN layers (lookahead 2, contributing up to 2 frames), and the Gaussian upsampling guardband (2 phones). The total is small and fixed, independent of utterance length.

Vocoder. The final component is a "lightweight and streamable LPCNet vocoder" (Valin and Skoglund, 2019). LPCNet combines linear predictive coding (LPC) — a classical speech processing technique that models the vocal tract as an all-pole filter — with a neural network that predicts the excitation signal (the source sound from the vocal cords). It is inherently causal and can operate frame-by-frame with very low latency. The paper does not describe modifications to LPCNet, implying it is used in its standard streaming configuration.

Training data. PnP2Speech is trained on a 6.5-hour proprietary conversational speech corpus recorded by a professional US-English female speaker. The corpus "contains a variety of expressive dialog acts and interjections" (Fernandez et al., 2022). The conversational nature is critical: standard TTS datasets (like LJSpeech or LibriTTS) consist of read speech — audiobook narration or news reading — which has a restricted prosodic range and lacks conversational phenomena like filled pauses, hesitations, overlapping speech boundaries, and emotional expressiveness. The proprietary corpus provides training targets (ground-truth mel-spectrograms and waveforms) that exhibit the prosodic variability needed for natural conversation.

Frame size and sampling rate. PnP2Speech operates with a "frame size of 256 samples for 22kHz-sampled speech." At 22,050 Hz, 256 samples corresponds to approximately 11.6 ms per frame, which is a standard hop size for neural vocoders — small enough to capture fine temporal detail in speech, large enough to be computationally efficient.

---

End-to-End Inference: Coordinating Text Generation with Speech Synthesis

During deployment, the three components — frozen LLM, LLM2PnP, and PnP2Speech — operate in a coordinated streaming pipeline where the LLM generates text incrementally and the speech synthesis follows with minimal delay.

Step-by-step flow (Figure 1):

1. The LLM receives a prompt (the user's spoken utterance, transcribed to text by an ASR system) and begins autoregressive generation. For each new token produced by the LLM's decoder, two outputs become available: the discrete token ID (a subword unit) and the hidden-state embeddings from the specified layers (2, 6, and 10 for T5-Base). These are sent to LLM2PnP.

2. LLM2PnP's encoder processes the growing sequence of (token, embedding) pairs. With restricted self-attention, tokens can only attend to earlier or same-word tokens — so the encoder's representations at each position are causal. As new tokens arrive, only the new encoder states need to be computed (the previous positions' states, having no dependence on future tokens, remain unchanged).

3. LLM2PnP's decoder autoregressively generates PnP tokens for the current word, conditioned on the encoder output (with $L = 1$ word lookahead) and previously generated PnP tokens. The decoder generates phones, HPC values, and phrase type tags for the current word. Crucially, the system synthesizes a word until reaching a regular word separator (a space in the original text) — this is the gating mechanism that prevents LLM2PnP from starting to process a word before the LLM has finished generating it. When a normalization expansion splits a token into multiple orthographic words (e.g., "23" → "twenty-three"), inner word separators are used so that all phones for the expanded form are generated before waiting for the next LLM token.

4. PnP2Speech receives the PnP tokens in chunks. The chunked phonetic encoder (LC-CNN + LC-BLSTM) processes each chunk with bounded right-context. The Gaussian upsampling layer with 2-phone guardbands maps the PnP tokens to acoustic frames. The autoregressive LSTM decoder and LC-CNN PostNet produce mel-spectrogram frames, which are fed to the LPCNet vocoder to produce the final 22 kHz audio waveform.

5. As the LLM generates the next token, the cycle repeats. The total delay between an LLM token being generated and the corresponding audio being output is approximately two words of latency — one from LLM2PnP's $L = 1$ lookahead, and roughly one from PnP2Speech's algorithmic delay. For typical English speech at 2–3 words per second, this corresponds to 0.7–1.0 seconds of latency, which is noticeable but far better than waiting for the entire LLM response to complete (which could be 5–15 seconds for a multi-sentence answer).

Handling interjections and filled pauses. The system can synthesize non-lexical vocalizations like "hmm," "uh-huh," and "oh" because these are present in the conversational training corpus (both as text tokens generated by the LLM and as audio recordings in the TTS training data). The LLM, having been fine-tuned on conversational data, can generate these tokens as part of its responses (e.g., "Hmm, let me think about that..."). LLM2PnP, fine-tuned on the conversational corpus, can predict appropriate phones and prosody for these tokens (often involving non-standard phonetic sequences and distinctive prosodic contours). PnP2Speech, trained on recordings that include these interjections, can synthesize them with natural acoustic quality. This is a significant capability because conversational speech is rich in such vocalizations — they signal thinking, agreement, hesitation, and emotional state — and their absence would make the system sound robotic or unnatural despite low latency.

---

Summary of Design Choices and Their Justifications

• Frozen LLM over fine-tuned LLM: Preserves the enormous investment in LLM pretraining and alignment, avoids catastrophic forgetting, and enables modular deployment where the same LLM can be used with or without the streaming TTS attachment.

• Offline-to-streaming knowledge distillation over direct supervision: Avoids the need for a parallel corpus of streaming text with phonetic and prosodic annotations (which doesn't exist); the teacher model provides pseudo-labels that capture full-context quality, and the student learns to approximate them using only left-context plus LLM embeddings.

• Single-word lookahead ($L = 1$) over zero or two-word lookahead: Table 2 shows that moving from 0 to 1 word lookahead reduces G2P WER by nearly 70% on all words (6.40% → 1.95%), while the second word provides diminishing returns (1.95% → 1.69%). The paper hypothesizes this is because "post-lexical processes in US English which influence the pronunciation of a word depend[ing] on the word that follows" are mostly resolved with one word of right-context.

• LLM embeddings from multiple layers (2, 6, 10) rather than a single layer: Table 3 shows that using three layers (2, 6, 10) gives slightly better G2P than using only layer 6 (1.95% → 1.98% WER). This captures both lower-level syntactic features (early layers) and higher-level semantic features (later layers), providing a richer context signal for disambiguation.

• Restricted lookahead in encoder-decoder attention rather than encoder self-attention: Prevents the lookahead from compounding across encoder layers, which would violate the streaming constraint and make the effective latency unpredictable and layer-count-dependent.

• Chunked BLSTM with zero lookahead over full BLSTM: Strips away the infinite right-context of bidirectional processing while retaining some local backward context (within the chunk), which is better than a purely unidirectional LSTM for capturing local phonetic coarticulation effects.

• LC-CNN with skewed kernels over causal convolutions with reduced receptive field: Causal convolutions would entirely eliminate right-context, which is unnecessarily restrictive (some right-context is acceptable up to the latency budget). Skewed kernels allow using the full lookahead budget efficiently, concentrating the available right-context where the network architecture needs it most.

• Guardbands in Gaussian upsampling over naive chunking: Without guardbands, the alignment at chunk boundaries would be miscalibrated because the Gaussian windows for phones near the boundary would be truncated. The 2-phone guardband ensures correct alignment computation without extending audio output latency.

4. Key Insights and Innovations

Innovation 1: LLM Hidden Embeddings as a Substitute for Future Text Context in Streaming Synthesis

The paper's most distinctive conceptual move is treating the LLM's internal representations not merely as a by-product of text generation, but as a semantically rich signal that can partially compensate for missing right-context in streaming speech synthesis. This reframes the streaming TTS problem from one of "how do we minimize the damage caused by truncated context?" to "what information already available in the generation process can substitute for the context we haven't received yet?"

Prior work on incremental TTS treated the text as the sole source of linguistic information for phonetic and prosodic prediction. Streaming approaches (Ma et al., 2020; Wu et al., 2021; Ellinas et al., 2020) focused on architectural mechanisms to limit lookahead — prefix-to-prefix frameworks, causal convolutions, chunked processing — and accepted that quality would degrade as the lookahead shrank. The implicit model was: less text context → less information → worse predictions. The innovation here is recognizing that the LLM's hidden states encode information about what the model intends to say next — syntactic structure, semantic content, discourse intent — that overlaps with what future text would reveal. An LLM that has processed the user's question and begun generating a response has already formed representations of the upcoming content before the corresponding tokens are emitted. These embeddings can serve as a proxy for future context, enabling the streaming TTS to make predictions that approximate full-sentence quality without actually seeing the full sentence.

This is not merely an incremental engineering improvement — it's a reframing of the problem's structure. It shifts the bottleneck from "we must wait for text" to "we must learn to read the LLM's mind," converting a latency constraint into a representation-learning challenge. The empirical evidence supporting this reframing appears in the G2P ablation (Table 3): removing LLM embeddings entirely degrades G2P WER from 1.95% to 2.10% on all words, and the effect is more pronounced on challenging subsets (OOV words worsen from 18.90% to 20.05%). Conversely, using embeddings from larger LLMs — moving from T5-Base to T5-XL — steadily reduces WER (1.89% vs. 1.95% on all words), suggesting that more powerful LLMs produce richer contextual embeddings that better substitute for missing right-context. This is significant because it implies that the approach naturally improves as underlying LLMs improve, without architectural changes to the TTS pipeline.

The finding that embedding benefits are smaller than the gain from an additional lookahead word (Tables 2 and 3: moving from L=0 to L=1 reduces WER by ~4.5 percentage points, while the best embedding configuration gains only ~0.2 points) is itself informative. It establishes a hierarchy of information sources: direct text context remains the most valuable signal, but LLM embeddings capture a non-trivial fraction of what that context would provide, and that fraction grows with LLM capability. This is a diagnostic result that quantifies the substitutability of semantic context for textual context — a concept that didn't exist in the streaming TTS literature before this paper.

---

Innovation 2: Offline-to-Streaming Knowledge Distillation as a Training Paradigm for TTS Under Lookahead Constraints

The paper imports an idea from streaming automatic speech recognition — offline-to-streaming knowledge distillation (Povey et al., 2018; Kurata and Saon, 2020) — and adapts it to the TTS domain in a way that sidesteps a fundamental data problem. Supervised training of a streaming phone-and-prosody predictor would require a corpus of incomplete text prefixes paired with correct full-sentence phonetic and prosodic annotations, which doesn't exist and would be prohibitively expensive to create (it would require human annotators to label phones and prosody for every partial prefix of every utterance). The knowledge distillation approach solves this by using a full-context teacher's predictions as pseudo-labels for the streaming student, creating an effectively unlimited training set from existing text corpora.

This is more than a training trick — it's a conceptual framework for transferring quality from offline to streaming systems that had not been applied to TTS before. The teacher model encapsulates the ideal behavior: given the complete text, here are the phones and prosodic features that produce natural speech. The student must learn to approximate these outputs using only left-context and LLM embeddings. The distillation loss forces the student to discover correlations between the available information (LLM hidden states, partial text) and the teacher's full-context decisions, effectively learning to predict the teacher's future-context-dependent choices from current-context cues.

The significance beyond raw performance is that this framework decouples streaming feasibility from annotation cost. Once a full-context teacher exists — which requires only a standard TTS training pipeline with complete utterances — the streaming student can be trained on any text corpus without additional human labeling. The paper exploits this by training on 3 million samples from C4, a scale that would be impossible with manually annotated streaming data. This scalability is what enables the student to learn subtle context-dependent patterns (cross-word pronunciation changes, prosodic phrasing decisions) that occur rarely and would be missed in a smaller annotated corpus.

The MOS parity result — LLM2Speech at 4.12 ± 0.04 vs. teacher at 4.10 ± 0.04 (Table 1) — validates the distillation framework. The student achieves indistinguishable quality from the teacher despite operating under a two-word lookahead constraint while the teacher has unlimited context. This is the strongest possible evidence that the distillation process successfully transfers the teacher's knowledge: listeners cannot tell the difference. It also sets a high bar for what streaming TTS can achieve — not "acceptable quality given the constraints" but "indistinguishable from the unconstrained system."

---

Innovation 3: Systematic Quantification of the Lookahead–Quality Trade-off in Conversational TTS

The paper provides a clean, quantitative characterization of how much future context matters for streaming phone prediction — and, equally important, where it matters — in a way that prior work had not systematically done. The G2P ablation (Table 2) measures word error rate as a function of lookahead words (0, 1, 2, ∞) broken down by word type (all, rare, normalization-expanded, out-of-vocabulary), establishing an empirical scaling curve for phonetic accuracy against right-context.

The key quantitative finding is that the first word of lookahead is disproportionately important: WER drops from 6.40% to 1.95% (a 70% relative reduction) when moving from L=0 to L=1, but only from 1.95% to 1.69% (a 13% relative reduction) from L=1 to L=2. This non-linear relationship means that a system designer facing a latency budget can make an informed decision: if one word of latency is acceptable, you capture most of the available accuracy; if zero words are required, you pay a significant quality penalty; if you can afford two words, the additional gain is small. This kind of latency–quality operating curve is standard in streaming speech recognition but had not been cleanly characterized for TTS pronunciation prediction.

The breakdown by word type reveals where context matters most. Rare words (least common 20% of vocabulary) show the steepest improvement with the first lookahead word: 14.99% → 2.71% WER (an 82% relative reduction), much larger than the 70% reduction on all words. This makes intuitive sense — rare words are more likely to be heteronyms or have non-obvious pronunciations that require contextual disambiguation, so seeing the next word is particularly valuable for resolving them. Normalization-expanded words (e.g., "23" → "twenty-three") also benefit substantially (21.71% → 6.28%), likely because the normalization mapping often depends on surrounding numeric or textual context. OOV words remain challenging even with lookahead (37.33% → 18.90% → 18.62% → 17.28%), confirming that words never seen during training are fundamentally hard regardless of context — a sanity check that the metric is behaving reasonably.

This quantification is significant beyond the specific numbers because it establishes a methodology for reasoning about latency budgets in streaming TTS. Rather than treating lookahead as an all-or-nothing choice (full context vs. streaming), or evaluating streaming systems only against other streaming systems, the paper provides a continuous curve that lets practitioners ask: "for my target WER threshold, what's the minimum lookahead I need?" This is conceptually analogous to the bitrate–distortion curves in compression — a principled way to navigate an engineering trade-off with empirical data.

---

Innovation 4: Architectural Principle of Single-Point Lookahead Enforcement to Prevent Receptive Field Growth

The paper identifies a subtle but architecturally consequential problem in building streaming neural TTS: receptive field compounding across layers. When lookahead constraints are implemented naively in multi-layer networks — for example, by giving each encoder self-attention layer a fixed right-context window — the effective lookahead grows with each successive layer because each layer's receptive field is the union of the previous layer's receptive field plus the current layer's window. After N layers, the cumulative right-context can be N times the per-layer budget, entirely defeating the streaming constraint.

The paper's solution — placing the lookahead restriction in the encoder-decoder attention rather than the encoder self-attention (Section 2.2.1) — is a specific architectural principle that hadn't been articulated in the streaming TTS context before. The encoder is allowed to process tokens with unrestricted self-attention within each word (tokens can attend to all other tokens of the same word and all tokens of previous words, but not to future words' tokens), and the lookahead budget L is enforced only at the cross-attention boundary where decoder states query encoder states. Because the decoder operates autoregressively (causal self-attention over previously generated PnP tokens), the L-word lookahead in the cross-attention does not compound: each decoder layer independently applies the same L-word restriction to its cross-attention, and the decoder's causal self-attention prevents information from future decoder positions from leaking backward.

This is a systems-level insight about where to enforce constraints in multi-component neural architectures. The paper explicitly motivates it: the lookahead "would not grow in consecutive decoder layers, unlike encoder attention, where the lookahead would grow linearly with the number of layers." This is stated as a design justification, but it represents a general principle: when building streaming adaptations of transformer architectures, enforce the temporal constraint at the narrowest information bottleneck — the cross-attention interface between encoder and decoder — rather than inside the encoder, to prevent uncontrolled receptive field expansion.

The same principle manifests in the PnP2Speech modifications. The LC-CNN layers are designed to cap the cumulative lookahead at a fixed value by selectively skewing kernels when symmetric ones would exceed the budget (Figure 4). Rather than uniformly applying causal convolutions (which would eliminate all right-context), or uniformly applying symmetric convolutions (which would cause uncontrolled growth), the paper introduces a per-layer kernel selection rule: use symmetric kernels when the cumulative lookahead permits, skew when it doesn't. This is a disciplined way to spend a fixed lookahead budget across a deep network, ensuring that the total right-receptive field matches the target exactly, no matter how many layers are stacked.

These architectural decisions may appear as implementation details, but they encode a design philosophy for streaming neural networks: enforce constraints at choke points and track cumulative receptive fields explicitly. This contrasts with the common approach in streaming speech processing of simply replacing bidirectional layers with unidirectional ones everywhere and accepting the quality loss. The paper shows that with careful constraint placement, you can retain local bidirectional processing (within words for LLM2PnP, within chunks for PnP2Speech) while maintaining a globally bounded lookahead — getting the benefits of context where it's available without violating the streaming guarantee.

---

Innovation 5: The Conversational Fine-Tuning Strategy Bridges the Written–Spoken Modality Gap

The paper's training methodology includes a deliberate two-phase approach: first train on a large corpus of written text (3M C4 samples), then fine-tune on a small conversational speech corpus (6.5 hours). This is not merely "more data is better" — it reflects a strategic decomposition of the learning problem into two distinct challenges: learning general phonetic and prosodic regularities from abundant written text, and learning conversational-specific phenomena from scarce spoken data.

The written-text phase (C4) provides broad coverage of vocabulary, syntactic structures, and text-normalization patterns at a scale that no conversational speech corpus could match. The model learns that "lead" before "the team" is pronounced differently from "lead" before "paint" — patterns that appear across millions of diverse examples. But written text is edited, grammatical, and lacks prosodic annotation; it doesn't contain the filled pauses, fragmentary utterances, and emotional expressiveness of conversation. The conversational fine-tuning phase addresses this gap by exposing the model to exactly these phenomena, using a corpus specifically designed to capture expressive dialog acts and interjections (Fernandez et al., 2022).

The ABX preference test in Table 4 provides direct evidence for this strategy's importance: comparing LLM2Speech without fine-tuning (NoFT) to the full system, the fine-tuned version is preferred with statistical significance (p < 0.01), even though the experiment removed interjections from the test texts to ensure fairness. This means the conversational fine-tuning improves prosodic quality even on non-interjection, ordinary text — it's not just about learning to say "uh-huh" correctly. The conversational corpus teaches the model a more natural prosodic style overall: more varied pitch contours, more appropriate hesitation patterns, more expressive phrasing. This is a finding about prosodic domain adaptation: a model trained solely on written-domain data learns a "reading" prosodic style that sounds less natural in conversation, even when the words are identical.

This two-phase strategy has practical significance beyond the specific system. It shows that to build a conversational TTS system, you don't need a massive conversational speech corpus — the large-scale learning of pronunciation and basic prosody can happen on abundant text data, and only the conversational style adaptation requires the expensive, scarce recorded-speech corpus. This is essentially a transfer learning recipe for conversational speech synthesis, with implications for how future systems might be built for new languages or domains where conversational corpora are limited.

5. Experimental Analysis

Evaluation Methodology

• Dataset. All experiments use the C4 dataset (Common Crawl Cleaned Corpus, Raffel et al., 2020), the same data used to pretrain T5. From the full 365M samples, the paper uses a random subset of 3 million training and 130,000 validation samples. Each sample is a paragraph split randomly into context (fed to the T5 encoder) and text-to-predict or t2pred (1–5 sentences, fed to the T5 decoder), simulating conditional text generation. For conversational fine-tuning and acoustic model training, a 6.5-hour proprietary conversational speech corpus recorded by a professional US-English female speaker is used, containing "a variety of expressive dialog acts and interjections" (Section 2.3, citing Fernandez et al., 2022).

• Base model(s). The LLM backbone is T5-Base (Raffel et al., 2020), specifically the T5-lm-adapt variant fine-tuned for text completion. T5-Base has 12 encoder and 12 decoder layers with 768-dimensional hidden states. Ablation studies also evaluate T5-Large and T5-XL (24 layers each, with larger hidden dimensions) to assess the impact of LLM scale on embedding quality (Table 3). The TTS acoustic model is adapted from Non-Attentive Tacotron (Shen et al., 2020) with an LPCNet vocoder (Valin and Skoglund, 2019). The teacher PnP model for pseudo-labeling is a rules-based G2P system with a neural prosody predictor producing HPCs (Shechtman et al., 2021; Fernandez et al., 2022).

• Metrics. The paper uses three distinct evaluation methodologies:

  • Word Error Rate (WER) for G2P accuracy: the percentage of words whose predicted phone sequence does not exactly match the teacher's reference phone sequence, computed on the C4 validation set (130K samples). Lower is better. Reported on all words and on challenging subsets (rare words covering the bottom 20% of frequency, normalization-expanded words like "23" → "twenty-three", and out-of-vocabulary words unseen during training — Section 3.2).
  • Mean Opinion Score (MOS) for overall speech quality and naturalness: crowd-sourced listening test on a standard 5-point scale (1 = bad, 5 = excellent) with 25 native listeners rating 45 conversational texts, reported with 95% confidence intervals (Section 3.1, Table 1).
  • ABX preference tests for prosodic quality: listeners compare pairs of audio samples (system A vs. B) and rate their preference on a 5-point scale from −2 ("strongly prefer A") to +2 ("strongly prefer B"), with 0 indicating no preference. Each comparison uses 25 distinct listeners and 45 conversational texts; statistical significance is assessed at p < 0.01 (Section 3.3, Table 4).

• Baselines. Three systems are compared:

  • Teacher: the non-streaming, full-context teacher PnP model followed by the non-streaming PPT TTS system (Shechtman and Fernandez, 2023). This system has unrestricted lookahead (L = ∞) and access to the complete text plus sub-style labels (e.g., empathetic, happy), representing an upper bound on achievable quality given full information.
  • LLM2Speech (Ours): the proposed system with L = 1 word lookahead in LLM2PnP and the chunked PnP2Speech acoustic model, totaling approximately 2 words of algorithmic delay.
  • Stream-Teacher: a partially-streaming baseline that forces the teacher model into the same two-word lookahead constraint as LLM2Speech — the teacher G2P runs on text prefixes with 1-word lookahead, the prosody model uses lookahead restrictions, and the same PnP2Speech acoustic model is used for synthesis. This tests whether LLM2Speech's architecture and training provide benefits beyond simply limiting the teacher's context.

  For the G2P ablation, the LLM2PnP with L = 0 (no lookahead) and LLM2PnP without LLM embeddings serve as additional baselines to isolate the contributions of lookahead words and LLM hidden states respectively (Tables 2 and 3). For the prosody ablation, variants without conversational fine-tuning (NoFT), without LLM embeddings (NoEmb), with T5-XL embeddings (T5XL), and with L = 2 lookahead (LA2) are compared against the baseline LLM2Speech configuration (Table 4).

• Generation budget / compute accounting. The paper does not use a FLOPs or generation-budget metric in the traditional sense, since the system is real-time streaming rather than a fixed compute allocation problem. Instead, the central resource being budgeted is lookahead — the number of future words that the streaming system is permitted to access before making a prediction. Experiments sweep lookahead values of L = 0, 1, 2, and ∞ (unrestricted) to map the latency–quality operating curve. The total algorithmic delay sums the LLM2PnP lookahead (L words) plus the PnP2Speech delay (~1 word equivalent), yielding effective system lookaheads of approximately 1, 2, and 3 words for L = 0, 1, 2 respectively. The paper does not report wall-clock latency measurements, end-to-end response time, or computational cost in FLOPs — the evaluation is exclusively in terms of prediction accuracy and perceptual quality at a given lookahead constraint.

• Cross-validation / statistical protocol. For the MOS listening test (Table 1), 25 native listeners rate 45 conversational texts, and results are reported with 95% confidence intervals. The paper explicitly tests for statistical significance between systems: the Teacher (4.10 ± 0.04) and LLM2Speech (4.12 ± 0.04) show "no statistically significant difference," while Stream-Teacher (3.46 ± 0.06) is clearly separated. For the ABX preference tests (Table 4), 25 distinct listeners per comparison rate 45 text samples each, and results are assessed for statistical significance at p < 0.01 — only the NoFT comparison reaches this threshold (bolded in Table 4). The G2P WER results are computed on the 130K-sample validation set; no confidence intervals or statistical tests are reported for the WER numbers in Tables 2 and 3 — they are presented as point estimates.

---

Main Quantitative Results

Perceptual Quality: LLM2Speech Matches the Full-Context Teacher

The central result is the MOS listening test in Table 1, comparing three systems on 45 conversational texts rated by 25 native listeners:

The headline finding: LLM2Speech achieves a MOS of 4.12, which is statistically indistinguishable from the teacher's 4.10 despite operating under a two-word total lookahead while the teacher has unlimited context and access to conversational sub-style labels. The overlapping confidence intervals (both ±0.04) mean listeners cannot tell the streaming system from the full-context system in terms of overall quality and naturalness. This is the strongest possible validation of the offline-to-streaming knowledge distillation approach — the student has fully recovered the teacher's perceptual quality under tight streaming constraints.

Critically, the Stream-Teacher baseline scores only 3.46 — dramatically lower, and clearly separated from both LLM2Speech and the teacher by its confidence interval. The Stream-Teacher uses the same teacher G2P and prosody models but restricts them to a 1-word lookahead at the input text level (running on partial text prefixes), then uses the same PnP2Speech acoustic model. This means that simply taking the teacher and limiting its textual context — without the LLM2PnP architecture, without LLM embeddings, and without distillation training — results in a severe quality degradation of approximately 0.66 MOS points (3.46 vs. 4.12). This isolates the contribution of the paper's approach: it is not the case that the teacher model already works well under restricted lookahead and LLM2Speech is merely packaging it differently. The teacher's G2P and prosody modules fundamentally require broader context to maintain quality, and the LLM2Speech pipeline — with its embedding-augmented encoder, restricted attention, and distillation training — successfully recovers that quality.

---

G2P Accuracy: The First Word of Lookahead Is Disproportionately Important

Table 2 measures G2P Word Error Rate on the C4 validation set as a function of LLM2PnP's word lookahead L, broken down by word type:

The key pattern: moving from zero to one lookahead word reduces all-word WER from 6.40% to 1.95% — a relative reduction of approximately 70%. This captures the majority of achievable improvement: the gap between L = 1 and the full-context teacher (L = ∞) is only 0.64 percentage points (1.95% vs. 1.31%), compared to the 4.45-point gap closed by the first lookahead word. The second lookahead word provides diminishing returns: 1.95% → 1.69%, a relative reduction of only 13%. This non-linear relationship justifies the paper's choice of L = 1 as the default operating point — it captures most of the benefit while keeping latency low.

The breakdown by word type reveals where context matters most:

• Rare words (least frequent 20%): WER drops from 14.99% to 2.71% with one lookahead word — an 82% relative reduction, the largest across all categories. These are words where pronunciations are less predictable from spelling alone (heteronyms, irregular pronunciations) and where surrounding context provides crucial disambiguation.
• Normalization-expanded words (e.g., "23", "$5", "Dr."): WER drops from 21.71% to 6.28%, a 71% relative reduction. Text normalization often depends on interpreting symbols or abbreviations in context (e.g., "Dr." could be "Doctor" or "Drive"), so the first right-context word is highly informative.
• OOV words: WER drops from 37.33% to 18.90% with L = 1, but remains high even with the full-context teacher (17.28%). OOV words are fundamentally challenging regardless of context because the model has no training exemplars; the lookahead helps only marginally, likely by allowing the model to apply learned subword pronunciation patterns to the novel word.

---

LLM Embedding Contributions: Modest but Measurable G2P Improvement

Table 3 isolates the effect of LLM embeddings on G2P accuracy, holding lookahead constant at L = 1:

Removing LLM embeddings entirely degrades all-word WER from 1.95% to 2.10% (a 0.15-point absolute increase). This is a small but consistent effect — the embeddings provide information beyond what the token identities alone contain. Using three layers (2, 6, 10) instead of a single middle layer (6) improves WER from 1.98% to 1.95%, suggesting that combining lower-level syntactic and higher-level semantic features is modestly beneficial. Adding two more layers (2, 4, 6, 8, 10) yields only a further 0.02-point gain (1.93%), indicating diminishing returns from additional embedding layers.

Scaling the LLM size shows a clearer trend: moving from T5-Base (1.95%) to T5-Large (1.94%) to T5-XL (1.89%) steadily reduces WER. T5-XL embeddings achieve the best performance on all word categories, including OOV words (18.31% vs. 19.84% with no embeddings). This is consistent with the hypothesis that larger LLMs produce richer contextual representations that better compensate for missing right-context. However, the paper explicitly notes that "the benefits gained by the choice of LLM embeddings are smaller than those gained by an additional word lookahead" — comparing the 0.21-point WER reduction from best embeddings (XL) over no embeddings to the 4.45-point reduction from L = 0 to L = 1. This establishes a clear hierarchy: direct text context remains the dominant information source, with LLM embeddings providing a supplementary signal.

---

Prosodic Quality: Conversational Fine-Tuning Matters, Embeddings and Lookahead Do Not Reach Significance

Table 4 presents ABX preference test results comparing the baseline LLM2Speech configuration (A) against four variants (B):

Negative scores indicate preference for LLM2Speech (A) over the variant (B). Bolded results reach statistical significance at p < 0.01.

Only the NoFT comparison reaches statistical significance: listeners prefer the fine-tuned LLM2Speech over the variant without conversational fine-tuning, even though interjections were removed from the test texts for fairness (since NoFT was not trained on them). This means the conversational fine-tuning improves prosodic quality beyond just learning to say "uh-huh" — it transfers a more natural, expressive prosodic style to ordinary text. The effect is not merely about handling conversational tokens; it's about learning a conversational speaking style that listeners perceive as more natural across all content.

The NoEmb comparison is essentially neutral (-0.005, not significant): removing LLM embeddings does not perceptibly affect prosodic quality. This is a notable negative result — while LLM embeddings improved G2P accuracy (Table 3), they do not translate into a prosodic quality difference that listeners can detect. The embeddings provide semantic context that helps disambiguate pronunciation, but the prosodic predictions appear to be driven primarily by other factors (the teacher distillation targets, the conversational fine-tuning, and the text itself).

The T5XL comparison is neutral (0.007, not significant): using embeddings from a larger LLM does not improve perceived prosodic quality, despite the G2P gains seen in Table 3. This reinforces the finding that G2P accuracy and prosodic naturalness are distinct perceptual dimensions — improving one does not necessarily improve the other.

The LA2 comparison is neutral (-0.042, not significant): increasing the lookahead from L = 1 to L = 2 does not produce a statistically significant prosodic improvement. This is consistent with the G2P results in Table 2, where L = 2 offered only a 0.26-point WER reduction over L = 1. The additional right-context provides a small phonetic accuracy gain that does not rise to the level of perceptual significance for prosody.

---

Ablation Studies and Robustness Checks

• Lookahead sweep (Table 2): Varying LLM2PnP's word lookahead from 0 to ∞ shows that G2P WER improves non-linearly. The first word provides dramatic gains (6.40% → 1.95% on all words, 14.99% → 2.71% on rare words), while the second word and full context provide only incremental improvements. The paper attributes the first-word importance to "post-lexical processes in US English which influence the pronunciation of a word depending on the word that follows" (Section 3.2) — flapping, reduction, and certain heteronym disambiguations are resolved with exactly one right-context word.

• LLM embedding presence/absence (Table 3, row "—" vs. "Base 2, 6, 10"): Removing LLM embeddings degrades G2P WER from 1.95% to 2.10% on all words, with larger impacts on rare words (2.93% vs. 2.71%) and OOV words (20.05% vs. 18.90%). The effect is consistent but modest — embeddings provide a supplementary signal that helps phonetic prediction, especially for words with ambiguous or unseen pronunciations.

• Number of LLM embedding layers (Table 3, Base rows): Using 3 layers (2, 6, 10) outperforms a single layer 6 (1.95% vs. 1.98%), and using 5 layers (2, 4, 6, 8, 10) yields a marginal further gain (1.93%). The diminishing returns suggest that the most informative embedding layers are already captured with a sparse sampling of early, middle, and late decoder layers.

• LLM model scale (Table 3, Base vs. Large vs. XL): T5-XL embeddings achieve the best G2P WER (1.89% vs. 1.95% for Base), with consistent improvements across all word categories including OOV (18.31% vs. 18.90%). The paper notes this benefit is "smaller than those gained by an additional word lookahead" — LLM scale helps, but is secondary to lookahead.

• Conversational fine-tuning (Table 4, NoFT row): Removing the conversational fine-tuning phase causes a statistically significant degradation in listener preference (avg score -0.110, p < 0.01). This validates the two-phase training strategy and demonstrates that the fine-tuning improves prosodic naturalness even on non-interjection text. It also confirms that the written-domain C4 training alone is insufficient for conversational-quality prosody.

• LLM embeddings effect on prosody (Table 4, NoEmb row): Removing LLM embeddings produces no significant listener preference difference (avg score -0.005). This is a key negative result — while embeddings help G2P accuracy, they do not perceptibly impact prosodic quality. The prosodic information apparently comes from the teacher distillation targets and the conversational fine-tuning rather than the semantic content of the LLM's hidden states.

• Larger LLM embeddings effect on prosody (Table 4, T5XL row): Using T5-XL embeddings instead of T5-Base yields no significant preference difference (avg score 0.007). The G2P improvement from larger LLMs (Table 3) does not translate into perceptible prosodic gains — listeners cannot hear the difference in naturalness, even if phonetic accuracy is slightly better.

• Additional lookahead effect on prosody (Table 4, LA2 row): Increasing LLM2PnP's lookahead from L = 1 to L = 2 produces no significant preference difference (avg score -0.042). This is consistent with the diminishing G2P returns in Table 2 and suggests that the primary prosodic quality ceiling is set by the distillation targets and conversational fine-tuning, not by the available right-context.

• Stream-Teacher baseline (Table 1): The Stream-Teacher — which restricts the full-context teacher to the same lookahead as LLM2Speech without the proposed architecture or training — scores MOS 3.46, far below both the teacher (4.10) and LLM2Speech (4.12). This ablation demonstrates that simply limiting the teacher's context destroys its quality, and that LLM2Speech's specialized architecture (restricted attention, LLM embeddings, distillation training) is necessary to recover the teacher's performance under streaming constraints.

---

Critical Assessment

The experiments provide strong evidence for the paper's central qualitative claims, but there are important limitations in the quantitative coverage and generalizability that merit careful examination.

Claim: LLM2Speech matches the full-context teacher in perceptual quality.

The MOS result (Table 1) directly supports this claim with a clean experimental design. LLM2Speech scores 4.12 versus the teacher's 4.10 with overlapping 95% confidence intervals — this is a genuine equivalence result, not merely a failure to reject a null hypothesis. The sample size is adequate (25 listeners × 45 texts) and the Stream-Teacher baseline at 3.46 provides a clear separation, confirming the test's sensitivity.

However, several caveats apply. First, the evaluation uses only 45 conversational texts, which is a relatively small test set for perceptual evaluation. The paper does not describe how these texts were selected, what their difficulty distribution is, or whether they systematically cover challenging phonetic and prosodic phenomena. A larger and more diverse test set — particularly one including deliberately constructed challenging cases (long-distance dependencies, nested clauses, rare heteronyms) — would strengthen the claim that quality is preserved across the full range of conversational inputs. Second, the evaluation uses a single female US-English speaker. Whether the result holds for other voices, accents, or languages is unknown — the architectural choices (especially the 1-word lookahead) are partially motivated by US-English-specific post-lexical processes, and languages with longer-distance phonetic dependencies (e.g., French liaison, tone sandhi in Chinese) might show different lookahead–quality curves. Third, the MOS test measures overall quality and naturalness, but does not isolate specific dimensions — it is possible that LLM2Speech compensates for phonetic errors with better prosody (or vice versa) in ways that the aggregate score masks. A more fine-grained evaluation separating pronunciation accuracy, prosodic naturalness, and voice quality could reveal where the teacher and student differ even if the overall MOS is equivalent.

Claim: The first word of lookahead is disproportionately important for G2P accuracy.

Table 2 provides clean quantitative evidence with a clear non-linear pattern: L = 0 → L = 1 reduces WER by ~70%, while L = 1 → L = 2 reduces it by only ~13%. The breakdown by word type (rare, normalization, OOV) provides interpretable structure that matches linguistic expectations. The dataset is large (130K validation samples), so the point estimates should be stable.

However, these are point estimates without confidence intervals or statistical significance testing. We do not know whether the difference between L = 1 (1.95%) and L = 2 (1.69%) is statistically reliable, or whether it could be explained by sampling variance across the 130K test samples. Given that the absolute difference is only 0.26 percentage points, it is possible that L = 1 and L = 2 are not meaningfully different, which would further strengthen the paper's case for L = 1 as the sweet spot. Additionally, the C4 validation set consists of written, edited text, not conversational speech — the prevalence and nature of cross-word pronunciation phenomena may differ in spontaneous spoken language. A validation on transcribed conversational speech would test whether the lookahead–WER curve generalizes to the deployment domain.

Claim: LLM embeddings improve streaming TTS performance.

This claim is partially supported and partially contradicted by the evidence. The G2P ablation (Table 3) clearly shows that embeddings improve phonetic accuracy — removing them increases WER, and scaling to larger LLMs further reduces WER. This effect is modest (0.15–0.21 points on all-word WER) but consistent across word categories.

The prosody ablation (Table 4, NoEmb row), however, shows no significant perceptual difference when embeddings are removed. This is a genuine negative result that the paper acknowledges only implicitly. The embeddings help with pronunciation but not with prosodic quality as perceived by listeners. This raises an important question: in an end-to-end perceptual evaluation, is the G2P improvement from LLM embeddings actually audible? If listeners cannot detect the difference in an ABX prosody test, it is possible that the G2P gains are below the threshold of perceptual significance in natural speech — listeners may be insensitive to occasional phonetic errors if the prosody is natural, or the TTS acoustic model may smooth over minor phone prediction differences. The paper does not close this loop with an ABX test that specifically probes phonetic accuracy (e.g., comparing LLM2Speech with and without embeddings on heteronym-heavy text), which would directly test whether the G2P advantage is perceptible.

Claim: Conversational fine-tuning improves prosodic quality.

Table 4 provides statistically significant evidence for the NoFT comparison (p < 0.01). The experiment design is careful: interjections are removed from test texts to ensure fairness. This means the benefit of fine-tuning is not merely about learning conversational tokens — it transfers to ordinary text as well.

An important limitation: the conversational fine-tuning corpus is only 6.5 hours — a very small dataset by TTS standards. The paper does not explore how much fine-tuning data is needed (the 6.5 hours may be more than necessary, or it may be barely sufficient) or whether the benefit saturates. An ablation sweeping the amount of fine-tuning data would characterize the data efficiency of the adaptation. Additionally, the fine-tuning uses the same speaker as the PnP2Speech training — it is unclear whether the fine-tuning adapts the model to conversational style per se, or to this specific speaker's conversational style. Testing with a different conversational speaker would distinguish style adaptation from speaker adaptation.

Missing experiments.

Several experiments would have substantively strengthened the paper's claims:

1. End-to-end latency measurement: The paper defines lookahead in terms of words, but never measures actual wall-clock latency from LLM token generation to audio output. For a system claiming to enable "natural conversations," the absolute latency in milliseconds is a critical metric. A two-word lookahead at an average speaking rate of 2–3 words per second translates to 0.7–1.0 seconds, which is substantially higher than the ~200 ms turn-taking gap typical in human conversation. The paper should report measured end-to-end latency including LLM generation time, LLM2PnP inference time, PnP2Speech inference time, and vocoder delay.

2. Subjective evaluation of latency tolerance: Even if MOS matches the teacher, listeners might be sensitive to the incremental nature of the synthesis — hearing words appear one by one rather than as a fluid, pre-planned utterance. An evaluation that specifically probes the perception of incrementality (e.g., does the speech sound "choppy," "hesitant," or "like it's being made up on the spot"?) would address a dimension that MOS alone might not capture.

3. Evaluation on truly challenging long-distance phenomena: The C4 test set is random web text. A targeted evaluation on sentences where pronunciation depends on words beyond the one-word lookahead — e.g., "I read the book" (past) vs. "I read books" (present) where the disambiguating word is more than one position away — would probe the limits of the L = 1 design. The fact that L = 2 provides a small but consistent improvement over L = 1 in Table 2 suggests such cases exist, and characterizing them would inform whether L = 1 is truly sufficient.

4. Comparison to a non-streaming LLM2Speech baseline: What if the LLM2PnP and PnP2Speech were run in non-streaming mode (full text, no lookahead restrictions)? This would quantify the quality ceiling of the LLM2Speech architecture itself, separate from the streaming constraints. If non-streaming LLM2Speech significantly outperforms the teacher, that would suggest the LLM embeddings provide benefits beyond what the teacher's rules-based G2P can achieve; if it underperforms, that would suggest the teacher has complementary knowledge not captured by the embeddings.

5. Decoder-only LLM evaluation: All experiments use T5, an encoder-decoder model. The paper mentions in a footnote that the text-split step is not needed for decoder-only LLMs, but provides no results. Given the dominance of decoder-only architectures (GPT, LLaMA, etc.) in production LLMs, demonstrating LLM2Speech with such a model would substantially increase the work's practical relevance. The embedding extraction process would differ (hidden states from causal self-attention layers rather than cross-attention to an encoder), and the quality of those embeddings for streaming TTS might differ as well.

What the experiments genuinely demonstrate versus what they suggest.

The experiments demonstrate convincingly that for T5-Base generating written-domain text, with a female US-English conversational voice, LLM2Speech can recover the perceptual quality of a full-context teacher under a two-word streaming constraint. This is a substantial and well-supported result.

The experiments do not demonstrate that LLM2Speech enables natural voice conversations with LLMs in deployment. The missing pieces are: (a) latency measurements in absolute time, (b) evaluation with a production-scale LLM (T5-Base is small by modern standards), (c) demonstration that the quality holds for the types of text that conversational LLMs actually generate (which may include more disfluencies, self-corrections, and incomplete sentences than C4 text), and (d) integration with an actual spoken dialogue system where the user's speech is transcribed, fed to the LLM, and synthesized in a closed loop. These are reasonable scope limitations for an initial paper, but they mean the work demonstrates a technical capability (streaming high-quality TTS from LLM tokens) rather than a validated user experience (natural conversation). The paper's framing in Section 1 — "paving the way for natural AI conversations" — is appropriately cautious on this point, but readers should understand that the "way" is paved architecturally, not yet validated in interactive deployment.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Dominates and Is Not Included in Headline Efficiency Numbers

The assumption or constraint. The paper's compute-optimal test-time scaling framework requires estimating each prompt's difficulty before allocating the inference budget. The method for doing so — generating 2048 samples per question, scoring them with the PRM, and binning by average score — is extraordinarily expensive. The authors acknowledge this directly in Section 3.2:

"we note that estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

The paper frames this as an exploration-exploitation tradeoff but does not incorporate the exploration cost into any of the reported efficiency numbers.

The consequence. The headline claim of "more than 4× better efficiency" over best-of-N is computed after difficulty is already known, without amortizing the cost of learning it. In a real deployment, the difficulty estimation step of 2048 samples per question would completely dominate the inference budget for most reasonable test-time compute allocations. For a question where the compute-optimal policy selects, say, 16 generations of beam search, the difficulty estimation cost (2048 generations) is 128× larger than the actual problem-solving budget. The reported efficiency gains are therefore an upper bound on achievable savings rather than a realizable deployment figure. The true total cost would be 2048 + N generations, and the 4× advantage over best-of-N would shrink dramatically or reverse once the exploration cost is included.

What evidence exists in the paper. The paper states the difficulty estimation procedure explicitly in Section 3.2 but provides no experiment that measures the total cost including exploration, no ablation varying the number of difficulty-estimation samples to find a cheaper operating point, and no analysis of how the efficiency gains change when exploration cost is amortized across multiple queries. The predicted-difficulty variant still uses 2048 PRM-scored samples per question — it removes the need for ground-truth labels but not the computational cost.

Mitigation status. The paper partially acknowledges the problem (Section 3.2, Section 8) and suggests future work on "pretraining or finetuning models to directly predict difficulty of a question" or developing adaptive methods that interleave difficulty estimation with problem-solving. However, no such model is developed or evaluated. Until a cheap difficulty estimator exists and its cost is factored into the total budget, the compute-optimal framework's practical value remains largely theoretical. The current results should be interpreted as a proof of concept — demonstrating that compute-optimal allocation would provide large gains if difficulty could be predicted cheaply — not as a deployment-ready system.

---

Single Benchmark, Single Model Family, and a Small Evaluation Set Constrain Generalizability

The assumption or constraint. All experiments use the MATH benchmark (500 test questions) with PaLM 2-S* as the base model. The paper does not replicate findings on other reasoning benchmarks (e.g., GSM8K, MMLU, code generation tasks), other model families (e.g., GPT, LLaMA, Claude), or other modalities. The authors state they "believe this model is representative of the capabilities of many contemporary LLMs" (Section 4), but this is an unverified assertion. The 500-question test set, split into five difficulty quintiles of ~100 questions each, then further split by two-fold cross-validation, means the compute-optimal policy is selected based on approximately 50 questions per fold per bin — a sample size small enough that selected strategies may not be robust.

The consequence. Several aspects of the findings could be specific to PaLM 2-S* or MATH, limiting their generalizability:

• PRM quality and over-optimization behavior: The PRM is trained on PaLM 2-S*'s output distribution using Monte Carlo rollouts. A model with different calibration properties, different error patterns, or different solution structures might exhibit different difficulty-dependent scaling curves. The over-optimization threshold (where beam search starts degrading on easy problems) is verifier-specific and may shift substantially with different base models.
• Revision model effectiveness: The revision model's ability to learn from incorrect in-context examples depends on the base model's in-context learning capabilities and its tendency to produce "close but wrong" answers that the edit-distance pairing can exploit. These may vary across model families.
• MATH-specific reasoning patterns: MATH consists of competition-level math problems requiring symbolic reasoning. It is unclear whether the difficulty-dependent strategy patterns (beam search hurting easy problems, revisions helping easy problems) generalize to code generation (where correctness can be verified by unit tests), logical reasoning (where step-by-step deduction dominates), or factual QA (where the challenge is knowledge retrieval rather than inference).

The small cross-validation sample size (~50 questions per fold per bin) means that the specific strategies selected as "optimal" for each difficulty-budget combination may be sensitive to the particular test set split. The paper does not report confidence intervals on the compute-optimal scaling curves, so it's impossible to assess whether the observed 4× efficiency gain is statistically reliable or could vary substantially with a different test set draw.

What evidence exists in the paper. No cross-benchmark or cross-model results are reported. The paper's only model variation is the ~14× larger PaLM 2 variant used in the FLOPs-matched comparison (Section 7), but that model is from the same family and the comparison is specifically about scaling effects, not about verifying that difficulty-dependent strategy patterns transfer.

Mitigation status. The authors do not claim broader generalizability beyond MATH and PaLM 2-S*, but they also make no attempt to bound the scope of their claims or discuss what properties of the model or benchmark might affect the findings. Section 8 does not mention cross-benchmark validation as future work. This is a significant gap because the compute-optimal framework's practical value depends on the difficulty-dependent patterns being reasonably stable across models and tasks — if each new deployment requires re-characterizing the optimal policy from scratch on a per-model, per-task basis, the approach is far less appealing than a finding that, say, "beam search on medium problems, best-of-N on easy problems" is a robust heuristic.

---

The ~14× Larger Model Baseline Is Not Compute-Optimally Trained, Weakening the Pretraining-vs-Inference Tradeoff Analysis

The assumption or constraint. The FLOPs-matched comparison in Section 7 uses a baseline model with approximately 14× more parameters than PaLM 2-S*, trained on the same amount of data. The authors explicitly state:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

This follows the LLaMA paradigm (scale parameters, fix data) rather than the Chinchilla paradigm (scale parameters and data equally for compute-optimal training; Hoffmann et al., 2022). Additionally, the larger model uses only greedy decoding — no test-time compute augmentation of its own.

The consequence. A compute-optimally trained larger model — scaling both parameters and data with the same total FLOPs increase — would likely perform better than the parameter-only-scaled baseline. This makes the pretraining baseline weaker than it needs to be for a fair comparison. The reported advantages of test-time compute over pretraining may shrink or reverse when compared against a properly compute-optimal larger model. For example, the +27.8% relative improvement on easy questions at R ≪ 1 (Figure 1, revisions) might be substantially smaller if the larger model were Chinchilla-optimal rather than LLaMA-style.

Similarly, giving the larger model even a modest test-time compute budget — say, best-of-8 or best-of-16 — would create a stronger baseline. The paper's comparison is asymmetric: the smaller model gets compute-optimally allocated test-time compute, while the larger model gets none. This makes it a comparison between "small model + smart inference" and "large model + naive inference," not between "inference compute" and "pretraining compute" per se. A fairer design would allocate the same total FLOPs split differently — e.g., some to pretraining for a larger model with simple inference, vs. more to inference for a smaller model — but the paper gives all FLOPs to pretraining in one case and all to inference (for the smaller model) in the other, without testing intermediate splits.

What evidence exists in the paper. The paper acknowledges the Chinchilla deviation (Section 7) but does not conduct a sensitivity analysis — for instance, estimating how much better a compute-optimal larger model would need to be to overturn the test-time compute advantage, or comparing against various points on the compute-optimal pretraining frontier. The reader is left with a qualitative caveat but no quantitative sense of how much it matters.

Mitigation status. The authors flag the issue and frame it as future work (Section 7, Section 8), which is appropriate for an initial study. However, given that the pretraining-vs-inference tradeoff is one of the paper's most prominently featured results (Figure 1, Figure 9, abstract-level claims), the weakness of the pretraining baseline substantially reduces confidence in the specific quantitative comparisons. The qualitative finding — that test-time compute can substitute for pretraining within certain difficulty regimes — remains well-supported by the difficulty-bin analyses; the precise 14× substitution ratio is less reliable.

---

Verifier Over-Optimization Is a Hard Ceiling That the Compute-Optimal Policy Mitigates but Does Not Solve

The assumption or constraint. The paper demonstrates that verifier over-optimization limits test-time compute scaling: beam search degrades easy-problem performance at high budgets (Figure 3, right), lookahead search — the strongest optimizer — paradoxically performs worst overall (Figure 3, left), and qualitative examples show degenerate outputs scoring highly under the PRM (Appendix M). The compute-optimal policy addresses this by routing easy problems away from aggressive search toward best-of-N, but the underlying problem remains: on medium-difficulty problems where beam search is the preferred strategy, performance still plateaus and sometimes declines at high budgets.

The consequence. Even with compute-optimal allocation, there is a hard ceiling on how much test-time compute can improve performance, set by the point where further optimization starts exploiting the verifier rather than finding genuinely better solutions. On medium-difficulty problems (bins 3–4 in Figure 3, right), beam search accuracy growth slows substantially between 64 and 256 generations, suggesting the system is approaching this ceiling. Pushing beyond this point — e.g., with 512 or 1024 generations — would likely yield diminishing or negative returns regardless of allocation strategy.

This means that continuing to scale test-time compute requires improving verifier robustness, not just optimizing the allocation strategy. The paper's compute-optimal framework is fundamentally bounded by verifier quality, and the current PRM (trained via Monte Carlo rollouts with soft labels) has clear failure modes that emerge under aggressive optimization. Without better verifiers, the approach cannot scale to arbitrarily large inference budgets — a limitation that contrasts with pretraining scaling, where performance continues to improve (albeit with diminishing returns) as compute increases.

What evidence exists in the paper. The over-optimization phenomenon is well-documented: Figure 3 (right) shows beam search accuracy decreasing on bin 1 from 4 to 256 generations; Figure 3 (left) shows lookahead search underperforming simpler methods at the same budget; and Appendix M provides qualitative examples of degenerate outputs with high PRM scores. The paper identifies this as a central challenge:

"We hypothesize that this is due to over-optimization of the PRM — beam search finds solutions that score highly under the PRM but are actually incorrect." (Section 5.3)

However, the paper does not quantify how much additional compute could be usefully deployed if verifier over-optimization were solved — i.e., the gap between current compute-optimal performance and the performance achievable with a perfect verifier.

Mitigation status. The compute-optimal policy partially addresses over-optimization by using best-of-N (a weaker optimizer) on easy problems where the PRM is reliable and beam search (a stronger optimizer) on medium problems where there is more room for genuine improvement. But this is a routing strategy, not a solution to the underlying verifier robustness problem. The paper does not propose improvements to the PRM training procedure to reduce over-optimization (e.g., adversarial training, ensemble methods, or KL-constrained search). Section 8 mentions developing more robust verifiers as future work but provides no roadmap. For practitioners, this means that deploying the compute-optimal framework with the current PRM training recipe will hit a quality ceiling determined by verifier reliability, and further compute investment beyond that ceiling is wasted.

---

Hard Problems Remain Unsolved: Test-Time Compute Cannot Create Capability from Nothing

The assumption or constraint. The paper's approach assumes that the base model can produce correct solutions at some non-trivial rate for the test-time compute to amplify. On the hardest questions (difficulty bin 5), where PaLM 2-S*'s pass@1 rate is near zero (approximately 1–3%), no test-time strategy — search, revisions, or their compute-optimal combination — produces meaningful improvement.

The consequence. For problems genuinely outside the base model's capability range, test-time compute is not a substitute for pretraining. The FLOPs-matched comparison (Figure 9, bin 5) shows that the larger model outperforms the small model + test-time compute essentially regardless of the inference budget, and the scaling curves are flat near 0–5% accuracy. This establishes a fundamental boundary condition: test-time compute amplifies existing capability but does not create it. If the base model cannot produce the correct reasoning pathway in any of its 2048 samples, no amount of search or revision will find it.

This has direct practical implications. For organizations considering whether to invest in test-time compute versus training larger models, the key question is not just "what's our total compute budget?" but "what's the difficulty distribution of our deployment queries?" If a substantial fraction of queries fall into the "hard" category — where the current model's pass@1 is near zero — additional inference compute will provide essentially no benefit, and scaling pretraining is the only viable path. The compute-optimal framework offers no solution for this regime.

What evidence exists in the paper. The difficulty-bin 5 results are consistent and stark across all experiments:

• Search (Figure 3, right): 1–3% accuracy for all methods and all budgets on bin 5.
• Revisions (Figure 7, right): ~2–3% accuracy regardless of sequential-to-parallel ratio on bin 5.
• FLOPs-matched (Figure 9): flat near 0–5% on bin 5, with the larger model's greedy performance (star) consistently above the scaling curve.
• The paper explicitly notes in the Section 7 takeaway box: "On the hardest questions, test-time compute provides essentially no benefit regardless of budget."

Mitigation status. The authors are transparent about this limitation — it appears explicitly in the Section 7 key takeaway and is visible in every difficulty-bin analysis. They do not attempt to solve it, and appropriately so: this appears to be a fundamental property of the approach, not a fixable design flaw. The limitation is a clear boundary condition that users of the framework must understand: compute-optimal test-time scaling works when the base model has a non-trivial chance of success and fails when it does not. The paper's contribution is precisely characterizing where this boundary lies rather than claiming to push past it.

---

Sequential Revisions Introduce Serial Latency That Is Not Accounted for in the Compute Budget

The assumption or constraint. The paper measures test-time compute in "number of generations," treating a generation as the unit of computational cost. This equates sequential revisions (one chain of N revisions, each depending on the previous one) with parallel sampling (N independent samples that can be generated simultaneously). While these are equivalent in total FLOPs, they are not equivalent in wall-clock time. A strategy that allocates, say, 64 sequential revisions followed by verifier selection takes approximately 64× longer to execute than a strategy that uses 64 parallel samples, assuming sufficient hardware to run the parallel generations concurrently.

The consequence. The compute-optimal strategies selected by the paper's framework on easy problems favor purely sequential revisions (Figure 7, right, bin 1: performance is essentially flat across all ratios, implying that fully sequential is preferred since it requires no parallel hardware). On moderately difficult problems (bins 3–4), the optimal strategy involves a mix of sequential and parallel — e.g., 8 chains of 16 revisions each at a budget of 128 generations. This strategy has ~16× the wall-clock latency of a fully parallel strategy at the same budget. For latency-sensitive applications — interactive voice assistants, real-time dialogue systems, live tutoring — this additional delay may be completely unacceptable regardless of the accuracy improvement. A user waiting for an answer to a spoken question cannot tolerate the 10–15 seconds that 64 sequential LLM generations would require, even if the answer is more likely to be correct.

The paper's compute budget abstraction ignores this tradeoff entirely, potentially leading to strategies that are FLOPs-optimal but latency-catastrophic. In production deployments where user experience depends on response time, the optimization objective should incorporate a latency penalty or constraint, which would shift the optimal allocation toward more parallel, less sequential strategies.

What evidence exists in the paper. The paper does not measure wall-clock latency, does not discuss the serial-vs-parallel latency distinction, and does not incorporate any latency constraint into the compute-optimal objective (Equation 1 considers only generation budget N, not time). The FLOPs-matched comparison in Section 7 accounts for total floating-point operations but not for execution time. The revision model experiments (Section 6) report the sequential-to-parallel ratio that maximizes accuracy at a given budget but provide no information on how long these strategies take to execute.

Mitigation status. Unaddressed. The authors do not mention latency as a concern, do not report execution times, and do not suggest latency-aware allocation as future work. This is a significant practical oversight for a paper whose motivation is improving user-facing LLM interaction. The finding that sequential revisions outperform parallel sampling for many difficulty regimes is valuable, but practitioners need to weigh that accuracy gain against the latency cost, and the paper provides no data to support that tradeoff. A simple addition — reporting the wall-clock time per generation for the hardware used, and showing latency-vs-accuracy curves alongside the budget-vs-accuracy curves — would substantially improve the practical utility of the results.

7. Implications and Future Directions

How This Work Changes the Landscape

LLM2Speech introduces a reframing of the streaming TTS problem that shifts the field's focus from architectural latency reduction to representation exploitation. Prior work on incremental TTS (Ma et al., 2020; Wu et al., 2021; Ellinas et al., 2020) treated the challenge as fundamentally about minimizing the damage caused by truncated right-context — the assumption being that less text necessarily means worse predictions, and the best one can do is design architectures that degrade gracefully under the constraint. This paper upends that framing by recognizing that when the text source is an LLM, the generation process produces a second, information-rich signal — hidden-state embeddings — that is temporally aligned with the text stream and encodes semantic and syntactic information about what the model intends to say. The problem transforms from "how do we cope with missing context?" to "how do we extract the context that already exists in the LLM's internal representations?"

This is a conceptual shift rather than a paradigm shift — the core TTS architecture (non-attentive Tacotron + LPCNet) is not fundamentally new, and the individual components (knowledge distillation, restricted attention, chunked BLSTMs) existed in prior work. What is new is the systems-level insight that the LLM and the TTS system should not be treated as independent modules connected only through discrete text tokens. Instead, the rich continuous representations that the LLM computes as a by-product of autoregressive generation should flow forward into the synthesis pipeline, providing a semantic scaffolding that partially substitutes for future text. This reframing matters because it changes what researchers optimize: rather than designing increasingly clever architectural constraints to squeeze quality out of limited text context, the challenge becomes identifying which LLM internal states are most informative for pronunciation and prosody, and how to train lightweight adapters that extract and leverage those states effectively.

The paper also provides the first clean quantitative characterization of the lookahead–quality trade-off for streaming G2P in a conversational TTS context. Table 2 establishes an operating curve that maps words of right-context to phonetic accuracy, revealing a striking non-linearity: the first lookahead word reduces WER by approximately 70% (6.40% → 1.95%), while the second word yields only a 13% further reduction (1.69%). This result moves the conversation from qualitative claims about "limited lookahead" to a specific, actionable finding: for US English, one word of right-context captures the majority of cross-word pronunciation phenomena that matter. The breakdown by word type (rare, normalization-expanded, OOV) further refines this picture, showing that rare words benefit disproportionately from the first lookahead word (82% WER reduction) while OOV words remain challenging regardless of context. This is the kind of empirical characterization that enables disciplined engineering trade-offs — a practitioner can now reason quantitatively about what latency budget buys what accuracy, rather than treating the relationship as unknown.

The Stream-Teacher baseline result (MOS 3.46 vs. LLM2Speech at 4.12, Table 1) serves as a powerful negative result that establishes a methodological principle: naively restricting a full-context TTS system to partial text does not work. The teacher model, when forced to operate with the same two-word lookahead as LLM2Speech, produces dramatically lower quality — a 0.66 MOS point drop that listeners clearly perceive. This demonstrates that the LLM2Speech architecture's specific design choices — restricted attention applied at the encoder-decoder boundary rather than in the encoder, LLM embedding injection, and distillation training against full-context targets — are not merely implementation details but are necessary for recovering the teacher's quality under streaming constraints. The result also implies that prior incremental TTS work that applied lookahead restrictions to existing full-context models (without the distillation-and-embedding approach) was likely leaving substantial quality on the table. It establishes a new baseline expectation: streaming TTS should aim to match its full-context teacher, not merely to outperform a degraded version of it.

Methodologically, the paper demonstrates that offline-to-streaming knowledge distillation from a full-context teacher is viable for TTS — extending a technique previously established in streaming ASR (Povey et al., 2018; Kurata and Saon, 2020). The viability is demonstrated not just by the MOS parity but by the training scale: 3 million C4 samples with pseudo-labeled PnP targets, a dataset size that would be impossible to annotate manually for streaming prefixes. This opens the door to applying the same distillation framework to other TTS architectures (e.g., end-to-end models that skip explicit G2P, or diffusion-based acoustic models), other languages where context-dependent pronunciation phenomena differ from English, and other generation modalities (e.g., code-to-speech, translation-to-speech) where the generator's internal states might similarly substitute for missing right-context.

The finding that conversational fine-tuning on a small corpus (6.5 hours) transfers prosodic naturalness to non-conversational text (Table 4, NoFT comparison reaching p < 0.01) is a practically important discovery that reframes data requirements for conversational TTS. It suggests that the large-scale learning of core pronunciation and basic prosody can happen on abundant, non-conversational text (C4), and only the final style adaptation requires expensive, scarce recorded conversational speech. This is essentially a transfer learning recipe that dramatically lowers the barrier to building conversational TTS for new voices or languages — the conversational corpus can be small because it only needs to teach style, not pronunciation.

The research directions that become more attractive after this work include: cross-modal representation exploitation (what other generator internal states can substitute for future context?), LLM-aware TTS architecture co-design (should future LLMs be designed with output embeddings that are optimized for downstream synthesis?), and scaling laws for embedding utility (how does the G2P and prosody benefit of LLM embeddings scale with model size, training data, and architecture?). Directions that become less attractive include: purely architectural approaches to streaming TTS that ignore the generator's internal state (since the Stream-Teacher result shows that architecture alone cannot close the gap to full-context quality), and approaches that assume conversational TTS requires large conversational speech corpora from scratch (since the C4 → conversational fine-tuning pipeline achieves strong results with only 6.5 hours of adaptation data).

Follow-Up Research This Work Enables

Measurement of end-to-end wall-clock latency with a production-scale LLM in an interactive spoken dialogue loop. The paper defines latency in terms of words of lookahead, but never measures absolute time from LLM token generation to audio output, nor from user speech end to system speech start. A critical follow-up would instrument the full pipeline — ASR → LLM → LLM2Speech — on realistic hardware and report: (a) the latency distribution for each component, (b) the total user-perceived response delay, and (c) how that delay compares to the ~200 ms turn-taking gap typical of human conversation. This would use a decoder-only LLM at a scale representative of deployed conversational agents (e.g., LLaMA-3-8B or Gemma-7B) rather than T5-Base, and would include the embedding extraction overhead. The key question is whether the two-word algorithmic delay (estimated at 0.7–1.0 seconds) plus LLM generation time (potentially several seconds for multi-sentence responses) results in total latency that users find acceptable for natural conversation, or whether further latency reduction (e.g., sub-word streaming, speculative PnP generation before the LLM token is finalized) is necessary. The paper's MOS results establish that the audio quality is sufficient for conversation; the latency evaluation would establish whether the temporal dynamics are also sufficient.

Evaluation of LLM2Speech with a decoder-only LLM on conversational text that includes disfluencies, self-corrections, and fragmentary utterances. All experiments use T5 (encoder-decoder) generating well-formed C4 text. A decoder-only LLM like GPT or LLaMA generates text through causal self-attention, and its hidden embeddings may have different properties — they encode only left-context, without the cross-attention to a separately encoded prompt that T5's decoder receives. Additionally, conversational LLM outputs often contain phenomena absent from C4: mid-sentence topic shifts, false starts ("I think that — actually, no — the answer is..."), and incomplete utterances. These stress the streaming TTS pipeline in ways that clean written text does not — for example, a self-correction requires the TTS to revise its prosodic plan mid-utterance, which may produce unnatural disfluencies or require retrospective adjustment of already-synthesized speech. The follow-up would: (a) replicate the G2P and MOS evaluations with a decoder-only LLM at comparable scale, (b) construct or source a test set of naturalistic conversational LLM outputs (perhaps from real user interactions with a deployed chatbot), and (c) measure whether the quality parity with the teacher is maintained on this more challenging text distribution. A negative result — that LLM2Speech degrades on disfluent or fragmentary conversational text — would be highly informative, identifying a boundary condition of the current approach.

Targeted evaluation of long-distance phonetic and prosodic dependencies that exceed the one-word lookahead. Table 2 shows that L = 2 improves G2P WER over L = 1 by 0.26 percentage points (1.95% → 1.69%), and L = ∞ improves it further to 1.31%. These small but consistent improvements suggest that some pronunciation phenomena depend on context beyond one word. Characterizing these cases would directly inform whether the L = 1 design is sufficient for deployment or whether certain linguistic contexts produce audible errors. The follow-up would construct a test set specifically enriched for phenomena that plausibly require long-distance context: (a) heteronym disambiguation where the resolving word is more than one position away ("I will record the record tomorrow" — "tomorrow" at position 6 disambiguates the noun "record" at position 4), (b) prosodic phrasing decisions that depend on clause structure spanning multiple words (parentheticals, appositives, nested relative clauses), and (c) cross-linguistic cases where the dependency length is known to be longer (e.g., French liaison, German verb-final constructions). The experiment would measure WER and conduct targeted ABX listening tests on these cases for L = 1 vs. L = 2 vs. full context. A finding that certain cases produce perceptible errors at L = 1 would motivate hybrid strategies — using L = 1 for most words but dynamically increasing lookahead when the LLM embeddings signal an upcoming disambiguation need.

Investigation of whether LLM embedding benefits for G2P are perceptually audible in synthesized speech. Table 3 shows that LLM embeddings improve G2P WER (e.g., 2.10% without embeddings vs. 1.95% with T5-Base embeddings), but Table 4 shows no significant listener preference for embeddings in the ABX prosody test (NoEmb: avg score -0.005). This raises a question: is the G2P improvement from embeddings actually perceptible in the final audio, or does the acoustic model smooth over minor phone differences? The follow-up would conduct an ABX listening test specifically designed to probe phonetic accuracy: select test sentences containing heteronyms or rare words where LLM embeddings are known to change the G2P prediction (identifiable from the validation set), synthesize each with and without embeddings (holding all else equal), and ask listeners to judge which version has the correct pronunciation. This isolates the phonetic contribution of embeddings from their prosodic contribution (which Table 4 suggests is negligible). A null result — that listeners cannot distinguish the embedding and no-embedding conditions even on heteronym-heavy text — would suggest that the G2P gains, while statistically measurable, fall below the threshold of perceptual significance and that the primary value of embeddings may lie in enabling the distillation process to work (by providing semantic context that helps the student match the teacher) rather than in directly improving final audio quality.

Cross-lingual and cross-voice replication of the lookahead–WER operating curve. The paper's claim that one word of lookahead captures most cross-word pronunciation phenomena is explicitly tied to US English ("post-lexical processes in US English which influence the pronunciation of a word depending on the word that follows"). This may not hold for languages with different prosodic structures — for example, tone sandhi in Mandarin Chinese can span multiple words, vowel harmony in Turkish operates across entire words, and French liaison can depend on syntactic structure rather than linear distance. The follow-up would replicate the Table 2 experiment (varying L from 0 to ∞, measuring G2P WER on all words and challenging subsets) for at least one non-English language where the relevant phonological phenomena are known to operate at different distances. Additionally, replicating with a different English voice (male, different accent) would test whether the 1-word sweet spot is voice-dependent or general. The result would either establish that the 1-word sweet spot is a robust finding for English (useful for practitioners targeting English-first deployments) or reveal language-specific lookahead curves that inform language-specific LLM2PnP design.

Integration of speculative PnP generation to further reduce latency below the one-word lookahead floor. The paper's L = 1 design sets a hard latency floor of approximately one word — the system must wait for the next LLM token before generating phones for the current word. But if the LLM embeddings at the current token carry information about which word is likely to come next, it may be possible to speculatively generate phones for the next word before the LLM produces its token, then commit or revise when the actual token arrives. This is analogous to speculative decoding for LLM inference. The follow-up would: (a) measure the predictability of the next word token from the current LLM embeddings (e.g., by training a lightweight probe classifier on top of T5's hidden states to predict the next token identity), (b) implement a speculative LLM2PnP that generates phones for the top-k predicted next words in parallel, and (c) measure the latency reduction and quality impact (does speculative phonetization introduce audible errors when the prediction is wrong, and how often is it wrong?). This would push latency below the one-word floor, potentially approaching phoneme-level streaming where audio begins almost immediately after the LLM starts generating.

Practical Applications and Downstream Use Cases

Voice-enabled conversational agents with LLM reasoning quality and near-real-time spoken responses. The central deployment scenario this paper enables is a spoken dialogue system where a user asks a question aloud, an ASR system transcribes it, a frozen text-based LLM (which may be a large, carefully safety-tuned model that cannot be retrained) generates a response, and LLM2Speech streams the spoken answer back to the user with approximately two words of latency. The key value proposition is that the system uses the same LLM that would power a text-based assistant — with all its reasoning capabilities, safety properties, and knowledge — rather than a less capable audio-native model. The MOS parity result (4.12 vs. 4.10 for the full-context teacher, Table 1) means the user hears speech quality indistinguishable from a non-streaming TTS system, while the streaming architecture means they hear the response begin within approximately 0.7–1.0 seconds of the LLM's first token rather than waiting for the complete response. This is immediately deployable for applications like hands-free voice assistants in vehicles (the paper explicitly mentions driving assistance), accessibility tools for visually impaired users interacting with LLMs, and voice interfaces for smart home devices where text display is impractical.

Low-resource conversational voice creation by transferring written-domain pronunciation learning. The paper's two-phase training strategy — 3 million written-domain samples (C4) for core pronunciation and prosody learning, followed by only 6.5 hours of conversational fine-tuning for style adaptation — provides a recipe for building conversational TTS voices with dramatically less conversational speech data than would otherwise be required. A voice designer could: (a) train the LLM2PnP on abundant text data (which can be synthetic or sourced from the web for many languages), (b) record a small conversational corpus (a few hours) from the target speaker covering expressive dialog acts and interjections, and (c) fine-tune the full pipeline to produce a streaming conversational voice. The 6.5-hour figure from the paper serves as an existence proof that this data quantity is sufficient; the NoFT ablation (Table 4, p < 0.01) confirms that the fine-tuning step is necessary and perceptually significant. This dramatically lowers the cost and time required to build a new conversational TTS voice compared to recording a full conversational corpus from scratch (which might require dozens of hours), making it feasible to create many distinct voices, to update voices with new expressive styles as the LLM's generation capabilities evolve, or to quickly build voices for new languages where conversational corpora are limited.

Real-time spoken language tutoring with incremental pronunciation feedback. LLM2Speech's streaming architecture, combined with the expressive conversational style supported by the fine-tuned PnP2Speech, enables a spoken language tutoring system where an LLM generates corrective feedback or model pronunciations and the system speaks them aloud immediately. For example, a language learner might mispronounce a word; the LLM (which has access to the correct pronunciation through its training data) generates a response like "Almost — the word is pronounced [correct phone sequence], with the stress on the second syllable. Try saying it after me: [word]." The streaming synthesis means the learner hears the correction within a second of the LLM generating it, creating a conversational tutoring loop rather than a delayed playback experience. The system's ability to synthesize filled pauses and interjections ("Hmm, not quite...") makes the interaction feel more natural and patient. The explicit phone prediction head in LLM2PnP could potentially be exposed to the tutoring application, allowing the system to show the phonetic transcription alongside the audio, providing multi-modal feedback. The key numbers supporting this use case are the MOS parity (4.12, indicating professional-quality speech suitable for pronunciation modeling) and the two-word latency (fast enough for interactive turn-taking during practice).

Streaming speech synthesis for real-time LLM-powered audio content generation. Beyond conversational turn-taking, LLM2Speech enables scenarios where an LLM generates long-form spoken content — live narration, real-time sports commentary, interactive audiobook generation, or streaming audio description for video — and the audience hears it with minimal delay. In these scenarios, the LLM might be generating a continuous monologue (e.g., describing events as they unfold), and the user wants to hear the description as it is produced rather than waiting for paragraph breaks. The streaming PnP2Speech architecture ensures that audio begins flowing within two words of the LLM starting generation, and the MOS parity means the continuous stream maintains consistent quality rather than degrading during periods of rapid generation. The ability to handle text normalization expansions correctly (via the inner/regular word separator mechanism) is particularly relevant for these applications, where numerals, dates, and symbols are common and must be spoken naturally. The 22 kHz output sampling rate provides broadcast-quality audio suitable for professional content delivery.