When to Ensemble: Identifying Token-Level Points for Stable and Fast LLM Ensembling

🎯 Pitch

Applying probability-level LLM ensembling to every token during long-form generation causes catastrophic failures—a 15-point accuracy drop on MATH500—because tokenizer mismatches between models corrupt probability distributions. The proposed SAFE framework avoids this by ensembling only at carefully chosen tokens, delivering a 2.8-point net gain while running as fast as a single model.

1. Executive Summary

This paper proposes SAFE (Stable And Fast LLM Ensembling), a framework that selectively determines at which tokens to apply probability-level ensembling during long-form generation, addressing the instability and inefficiency that plague existing ensemble methods under chain-of-thought prompting. Using diverse 7B-scale models with heterogeneous tokenizers — Internlm3-8B, Qwen2.5-7B, and EXAONE-3.5-7.8B — on benchmarks including MATH500, GSM8K, MMLU-redux, ARC-Challenge, and BBH, SAFE triggers ensembling only when guided by two key factors: tokenization mismatch across models (preventing OOV-like tokens that corrupt next-token probability distributions — e.g., ensembling the subword "So" when another model expects the full word "Sofia" as a single token) and consensus in models' next-token probability distributions (skipping ensemble operations when verifiers unanimously agree or when average probability exceeds 0.5 — e.g., bypassing costly vocabulary alignment when all models already predict the same token). The framework improves the state-of-the-art UniTE ensemble method from a 15.2-percentage-point degradation on MATH500 under CoT to a 2.8-percentage-point improvement over the best individual model, while ensembling fewer than 5% of tokens and achieving latency comparable to running a single model, establishing that probability-level ensembling can be both stable and efficient in long-form generation only when ensembling decisions are made at carefully identified token positions rather than uniformly across every generation step.

2. Context and Motivation

The Core Problem: Probability-Level Ensembling Breaks Down in Long-Form Generation

The fundamental question this paper tackles is deceptively practical: when multiple language models are available at inference time, how should we combine their next-token probability distributions to produce better outputs than any single model? This matters because different LLMs — even models of similar capability — exhibit complementary strengths shaped by their training recipes, architectures, and data mixtures. A model that excels at mathematical reasoning may stumble on commonsense inference, while another may show the reverse pattern. Ensembling at the probability level offers a training-free mechanism to harness this complementarity: rather than training a new, larger model that jointly integrates all capabilities, we can simply average the output distributions of existing models and select the token that commands the broadest support.

Prior work has demonstrated that this approach works remarkably well for short-form answers. When directly answering multiple-choice questions or producing single-token responses, methods like GaC (Yu et al., 2024), DEEPEN (Huang et al., 2024), and UniTE (Yao et al., 2025) consistently outperform individual models by selecting tokens that represent consensus across diverse probability distributions. The key technical challenge in these settings is vocabulary alignment — each model's output distribution is defined over its own tokenizer's vocabulary, and constructing an ensemble distribution requires mapping these heterogeneous spaces into a common representation. Prior methods solve this through various alignment strategies: GaC takes the union of all model vocabularies, DEEPEN projects into a shared embedding space, and UniTE demonstrates that aligning only the top-k tokens from each model is sufficient.

A natural question then arises: are these probability-level ensemble methods equally effective for long-form generation? The paper's opening motivation, stated directly in Section 1, is that the answer is a decisive no — and that this negative result reveals a deeper, underexplored problem in the literature.

Two Interacting Failure Modes That Prior Work Overlooked

The paper identifies two specific mechanisms through which existing ensemble methods fail catastrophically under chain-of-thought prompting, particularly when the participating models use heterogeneous tokenizers (i.e., different byte-pair encoding schemes that segment the same text into different subword units). These failure modes are not edge cases — they arise naturally from the interaction between token-level ensembling and the incrementally constructed context that defines autoregressive generation.

Failure Mode 1: OOV-Like Tokens Corrupt the Generation Process

The first and more severe problem is what the paper terms OOV-like tokens. To understand this, consider what happens during standard token-level ensembling. At each generation step, the ensemble aggregates the next-token probability distributions from all participating models and selects the most confident token. This selected token is then fed back as context to every model for the subsequent generation step. The critical issue is that this token was chosen by the ensemble, not by any individual model, and it may correspond to a tokenization that is inconsistent with how some models segment text.

The paper provides a concrete example (Figure 1): suppose the ensemble is generating the word "Sofia." The drafter model tokenizes "Sofia" as two tokens — So and fia — while another participating model tokenizes it as a single token Sofia. When the ensemble selects So as the next token (because it represents a high-probability continuation in the ensemble distribution), this token acts as an OOV-like token for the model that expects "Sofia" to be a single unit. That model is now forced to predict its next token conditioned on the prefix So — a prefix it has never encountered as a natural subword boundary in its training distribution. The result, shown clearly in the paper's Figure 2, is that the model's next-token probability distribution becomes corrupted. Instead of continuing with fia to produce "Sofia," the model might output entirely unrelated tokens, producing garbled text like "SofÃAÂAÂÃ" or, as in the degenerate case shown in Figure 2 (Right), repeatedly generating corrupted tokens due to the compounding effect of being conditioned on increasingly unnatural prefixes.

This is not the standard out-of-vocabulary (OOV) problem where a token is absent from the model's vocabulary entirely. Rather, the token does exist in the vocabulary — So is a legitimate token for the second model — but the token boundary at which it appears is inconsistent with that model's expected segmentation of the input text. The model is being conditioned on a partial word that it would have processed as part of a larger, atomic token. This subtle distinction is what makes the problem insidious: it arises not from vocabulary mismatch in the narrow sense (all tokens exist in all vocabularies) but from tokenization boundary mismatch — the fact that different tokenizers draw word boundaries at different positions in the character stream.

The severity of this problem scales with sequence length. In short-form generation, where outputs are a few tokens long, the probability of encountering a tokenization boundary mismatch is low, and even when it occurs, there is little room for the error to propagate. But in chain-of-thought reasoning, where outputs span hundreds or thousands of tokens, these mismatches accumulate. An initial OOV-like token corrupts one model's distribution, which feeds a second corrupted token into the ensemble, which further corrupts the context for all models, and so on in a cascading degradation. This is why Table 1 shows UniTE suffering a 15.2-percentage-point drop on MATH500 under CoT compared to simply running the best individual model — the ensemble process actively destroys output quality rather than improving it.

Failure Mode 2: The Computational Cost of Uniform Ensembling

The second problem is one of efficiency. In standard token-level ensembling, every generation step requires: (1) a forward pass through each participating model to obtain its next-token probability distribution, (2) alignment of these distributions across different vocabularies, and (3) aggregation to select the ensemble token. This is expensive for two distinct reasons.

First, costly autoregressive generation across all models. Each model must perform sequential forward passes — one per generated token — in lockstep, since the next context depends on the previously selected ensemble token. This means generating 500 tokens with three models requires approximately $3 \times 500$ forward passes, each of which must wait for the completion of the previous step's ensemble aggregation. The paper's speculative strategy (detailed in Section 3) addresses this by restricting autoregressive generation to a single "drafter" model.

Second, redundant ensemble operations when models already agree. When all participating models independently predict the same next token with high confidence, constructing the full ensemble distribution — including the computationally expensive vocabulary alignment step — provides no benefit: the selected token would be the same regardless. Yet existing methods perform this operation at every token position uniformly. The paper identifies this as a source of wasted computation, particularly in domains like mathematical reasoning where structured output formats constrain the space of plausible continuations, leading to higher inter-model agreement.

These two failure modes interact in a perverse way: the more aggressively we ensemble (which uniform methods do at every token), the more OOV-like tokens we introduce, which degrades output quality, while simultaneously consuming computational resources that grow linearly with sequence length.

Why This Matters: Practical and Theoretical Significance

The practical motivation is straightforward. If probability-level ensembling cannot be reliably applied to long-form generation, a large class of use cases is foreclosed. Chain-of-thought reasoning — where models produce multi-step explanations before arriving at final answers — is currently the dominant paradigm for improving LLM performance on complex tasks (Wei et al., 2022). Benchmarks like MATH500, GSM8K, and BBH all require extended reasoning chains. Deployments that rely on these reasoning capabilities — educational tools, scientific assistants, code generation systems — cannot benefit from model ensembling if the ensemble process corrupts the reasoning trace. The paper's Table 1 makes this concrete: UniTE degrades MMLU-redux from 77.92% (best individual) to 73.39% under CoT, and MATH500 from 74.8% to 59.6% — a catastrophic 15.2-percentage-point drop that makes the ensemble worse than any individual model.

The broader significance extends beyond a single method. The finding that tokenization boundary mismatches are the dominant source of instability suggests that the entire research program of token-level probability ensembling — which has produced GaC, DEEPEN, UniTE, and related methods — has been operating under an implicit assumption that only holds for short outputs. These methods were designed and evaluated primarily on tasks where the model produces a single answer token or a short phrase (multiple-choice selection, direct answering), where tokenization boundaries rarely diverge across models within the narrow span of the output. The paper's contribution is not just fixing a specific method but identifying a structural failure mode that limits an entire class of approaches. This reorients the research direction: rather than improving alignment algorithms (which was the focus of prior work), the priority becomes deciding when to ensemble, not just how.

Where Prior Work Falls Short

The paper positions its contribution relative to three lines of existing research, each of which addresses part of the picture but leaves the core problem unresolved.

Probability-Level Ensemble Methods (GaC, DEEPEN, UniTE)

These methods form the direct technical lineage that SAFE extends. The shared premise is that averaging next-token probability distributions across models can produce superior token selections compared to any individual model. The key technical challenge they address is vocabulary alignment: how to construct a single ensemble distribution from distributions defined over different token sets.

GaC (Yu et al., 2024) takes the union of all model vocabularies and maps each model's distribution to this union space before averaging. Notably, GaC introduces a threshold-based gating mechanism: it performs ensembling only when the "main" model's probability for its top token falls below 0.5. The paper observes (Section 1) that this threshold inadvertently provides some protection against OOV-like tokens — when the main model is highly confident, no ensemble operation occurs, and therefore no opportunity for tokenization mismatch arises. However, GaC's gating is driven by model confidence, not tokenization compatibility, so it provides no guarantee of stability. When ensembling is triggered (because the main model's confidence is low), it performs the operation at every subsequent token without considering whether the resulting tokens are compatible with all models' tokenization schemes.
DEEPEN (Huang et al., 2024) projects each model's vocabulary into a shared embedding space, performs distribution merging in that space, and maps back to individual vocabulary spaces. This addresses the alignment problem through a learned projection rather than counting-based union, but it performs ensembling at every token without any mechanism for deciding when ensembling is necessary or safe.
UniTE (Yao et al., 2025) demonstrates that aligning only the top-k tokens from each model's distribution is sufficient for effective ensembling, achieving state-of-the-art performance while reducing alignment cost. However, like DEEPEN, it performs ensembling uniformly at every generation step. The paper's Table 1 shows the consequences: UniTE achieves strong results on direct answering (+0.50 on MMLU-redux without CoT) but collapses under CoT (-15.2 on MATH500).

The critical gap across all three methods is the absence of any mechanism to determine whether ensembling at a given token position is safe (i.e., won't introduce OOV-like tokens) or necessary (i.e., the models actually disagree and ensembling would change the selected token). SAFE can be viewed as a meta-layer that augments any of these methods with exactly this decision capability — the paper demonstrates SAFE applied to both GaC and UniTE (Table 2), with consistent improvements.

Speculative Decoding

Speculative decoding (Leviathan et al., 2023) is a technique for accelerating autoregressive generation by having a small "drafter" model propose multiple candidate tokens, which a larger "target" model then verifies in a single forward pass. The target model either accepts the proposed tokens (if they align with its own distribution) or rejects them and regenerates from the point of divergence. This reduces the number of forward passes through the large target model, improving throughput while maintaining output quality.

The paper explicitly draws on the generate-verify structure of speculative decoding as an architectural template for SAFE. However, existing applications of speculative decoding to LLM ensembling (Fu et al., 2025) are limited to settings where all models share an identical tokenizer. In such cases, the drafter's tokens can be directly evaluated by target models — there is no boundary mismatch because every model segments text identically. When tokenizers differ, as in the paper's primary setting (Internlm3-8B, Qwen2.5-7B, EXAONE-3.5-7.8B, which the paper shows have only 40–60% tokenization agreement on common English words; Figure 4), the drafter's tokens cannot be naively evaluated by verifier models. A token that is a valid subword unit for the drafter may split at a different boundary for a verifier, and conditioning the verifier on this partial token produces the OOV-like degradation described above.

The paper's contribution relative to speculative decoding is therefore an acceptance criterion for heterogeneous tokenizers: how to determine whether a drafter token can be safely conditioned on by a verifier model with a different tokenization scheme, and how to verify ensemble consensus without requiring explicit vocabulary alignment at every step.

Post-Inference Ensemble Methods (Cascade, Parallel, Hybrid)

The paper distinguishes its token-level approach from methods that ensemble after complete responses have been generated. These include:

Cascade structures (FrugalGPT, AutoMix): Route queries sequentially through models of increasing capability/cost, stopping when a model produces a sufficiently reliable response. The ensembling decision is at the response level, not the token level.
Parallel structures (MORE, LLM-Blender): Generate multiple complete responses independently and then select or fuse the best one using a scoring mechanism or ranker. Again, the ensemble operation occurs after generation, not during it.
Hybrid structures (MoA, Self-MoA): Iteratively feed responses from multiple models into an aggregator that produces a consolidated answer, mixing parallel generation with sequential refinement.

The paper acknowledges these approaches but argues that they operate at a coarser granularity. By ensembling at the token level during generation, SAFE can influence the reasoning process itself — selecting tokens that represent the collective intelligence of multiple models at each step of the chain — rather than merely selecting among independently generated reasoning traces. The tradeoff, which the paper aims to resolve, is that token-level ensembling is more powerful in principle but has been unstable in practice for long-form generation.

How This Paper Positions Itself

The paper frames its contribution as not proposing a new ensemble method per se, but rather a selection mechanism that makes existing methods practical for long-form generation. The key insight is that the question "which token should the ensemble select?" (the focus of GaC, DEEPEN, UniTE) cannot be separated from the prior question "should the ensemble select a token at this position at all?" The paper argues that failure to ask this prior question is what causes the catastrophic degradation observed in Table 1.

The paper establishes this positioning through a clear empirical baseline. Table 1 shows that:

Without CoT (short-form answers), UniTE marginally improves or matches individual models, consistent with prior positive results in the literature.
With CoT (long-form generation), UniTE substantially degrades, dropping 15.2 percentage points on MATH500.
Applying SAFE's selective ensembling logic (UniTE + Ours) reverses this degradation, achieving a 2.8-percentage-point improvement over the best individual model.

This establishes that the problem is not with UniTE's aggregation mechanism — which works well in isolation — but with its uniform application across all token positions. The paper's contribution is thus a decision procedure (Sections 3.2–3.3) that precedes any ensemble operation and determines whether that operation is both safe (won't corrupt subsequent generation via OOV-like tokens) and necessary (would actually change the selected token compared to the drafter's prediction).

The parallel to speculative decoding is not merely structural but conceptual: just as speculative decoding determines which tokens to accept from a drafter (using a statistical acceptance criterion), SAFE determines which tokens to ensemble (using a criterion based on tokenization compatibility and distributional consensus). The extension beyond prior speculative ensemble work is the handling of heterogeneous tokenizers — the paper's OOV-like token verification (Section 3.2, i) and the consensus-based verification (Section 3.2, ii) that avoids costly vocabulary alignment when models already agree.

The paper also positions SAFE as plug-and-play: it can be integrated with any existing probability-level ensemble method by wrapping the generate-verify logic around it. This is demonstrated empirically by applying SAFE to both GaC and UniTE (Table 2), with consistent improvements across model combinations and benchmarks. The implication is that SAFE is not a competitor to prior ensemble methods but an enabling layer that makes them deployable in the long-form generation settings where they previously failed.

3. Technical Approach

3.1 Reader Orientation

This is primarily an algorithm design paper whose core idea is that the stability and efficiency of probability-level LLM ensembling in long-form generation depend critically on deciding at which token positions to perform the ensemble operation — and that existing methods fail because they apply ensembling uniformly at every token without considering whether doing so is safe (won't corrupt the generation via tokenization mismatches) or necessary (would actually change the selected token compared to what individual models would produce). The system being built is a selective ensembling framework called SAFE that wraps existing ensemble methods with a generate-verify-ensemble cycle: one model (the drafter) generates a lookahead sequence of tokens, the remaining models (the verifiers) examine those tokens in a single forward pass to identify which positions require ensembling based on tokenization compatibility and inter-model consensus, and ensembling is applied only at those identified positions to replace the drafter's tokens with ensemble-selected tokens. The shape of the solution is a decision procedure that precedes every ensemble operation rather than a new aggregation mechanism — SAFE determines when to ensemble, while the underlying ensemble method determines how to ensemble.

3.2 Big-Picture Architecture (Diagram in Words)

The SAFE system has four interconnected components that cycle iteratively:

Drafter Model (Mdraft): A single LLM — selected as the best-performing model among the ensemble — that generates a short lookahead sequence of n tokens autoregressively. This is the only model that performs expensive sequential forward passes; all other models contribute in verification-mode forward passes. The drafter's tokens serve as candidates that may be accepted as-is or replaced by ensemble-selected tokens.
Verifier Models (Mver): The remaining LLMs in the ensemble (all models except the drafter). Their role is to examine the drafter's proposed token sequence in a single, non-autoregressive forward pass and determine at which token positions ensembling should be triggered. They perform two sequential checks: (i) OOV-like token verification — is the token immediately preceding this position compatible with all models' tokenization schemes? — and (ii) ensemble distribution verification — do the models' probability distributions indicate sufficient agreement that ensembling would not change the selected token?
Ensemble Distribution Constructor: When a token position is flagged for ensembling, this component aggregates the next-token probability distributions from all models (drafter + verifiers) using an existing ensemble method (UniTE, GaC, etc.). It optionally applies a probability sharpening strategy if the resulting ensemble distribution is overly smooth (i.e., probability mass is diffused across multiple subword tokens representing the same word). The most confident token from the (sharpened) ensemble distribution replaces the drafter's token at that position.
KV Cache Manager: Since ensemble-selected tokens may differ from the tokens originally generated by individual models, the KV cache for each model must be updated to reflect the actual output sequence. This component prunes inconsistent cache entries after each ensemble step, ensuring that subsequent forward passes use correct context states.

Information flows as follows: The prompt enters → the drafter generates n candidate tokens → each verifier retokenizes the candidate sequence with its own tokenizer and computes its next-token probability distribution at each position → the verifiers jointly check two conditions per position (OOV-like safety, ensemble necessity) → at the first position that requires ensembling, all models' distributions are aggregated via the underlying ensemble method (e.g., UniTE), optionally sharpened, and the ensemble-selected token replaces the drafter's token → the drafter resumes autoregressive generation from this ensembled token → the cycle repeats until an end-of-sequence token is produced.

3.3 Roadmap for the Deep Dive

First, I'll explain the Generate step (Section 3.1 in the paper): why the drafter produces multiple tokens rather than one, how the drafter is selected, and what n (the lookahead length) controls — since this determines the granularity at which tokenization mismatches can be detected.
Second, the OOV-like token verification check (first part of Section 3.2, i): the formal definition of an OOV-like token, the boundary-matching algorithm, and why this check is the primary mechanism for preventing the catastrophic degradation shown in Table 1 — since this is the novel stability mechanism.
Third, the ensemble distribution verification check (second part of Section 3.2, ii): the two consensus conditions that allow ensembling to be skipped, the computational savings they enable, and the proof (Theorem 1 in Appendix C) that these checks do not change the ensemble's selected token — since this is the primary efficiency mechanism.
Fourth, the Ensemble step (Section 3.3): how existing ensemble methods are applied at the identified positions, the probability sharpening strategy for overly smooth ensemble distributions, and the two variants (heuristic reallocation and geometric mean) — since this addresses a subtle failure mode of token-level probability averaging across heterogeneous tokenizers.
Fifth, the overall Generate-Verify-Ensemble algorithm (Algorithm 1): the full iterative cycle as pseudocode, the acceptance/rejection logic, and how generation resumes from ensembled tokens — since this ties together the components into a complete procedure.

3.4 Detailed, Sentence-Based Technical Breakdown

This paper proposes SAFE (Stable And Fast LLM Ensembling), a generate-verify-ensemble framework that selectively applies probability-level ensembling at token positions identified as both safe (no tokenization boundary mismatch) and necessary (models exhibit sufficient disagreement to warrant aggregation). Rather than replacing existing ensemble aggregation methods (like UniTE's top-k alignment or GaC's vocabulary union), SAFE wraps them with a decision procedure that determines where in the generation sequence to invoke them. The core algorithmic insight is that the two failure modes of uniform ensembling — OOV-like token corruption and redundant computation — can be jointly addressed by two sequential checks performed on the drafter's lookahead tokens before any ensemble operation occurs.

Drafter Selection and the Generate Step

The framework begins by partitioning the available LLMs into two roles. One model is designated the drafter (Mdraft) and is responsible for autoregressively generating candidate token sequences. The remaining models serve as verifiers (Mver) and participate only through verification-mode forward passes.

Drafter selection criterion. The paper selects the model with the best average performance across the evaluation task as the drafter. This is a simple, task-level heuristic — no per-token or per-prompt routing is performed. The rationale, following UniTE's approach for primary model selection, is that the strongest model's tokens serve as the default output, and ensembling intervenes only when the verifiers indicate that a different token would be preferred by the ensemble. This choice means that in the limit where models always agree, SAFE degrades gracefully to simply running the best individual model — ensembling is skipped entirely and the drafter's tokens pass through unchanged.

Why generate multiple tokens at a time. The drafter produces a predefined number n of tokens (t_i, ..., t_{i+n-1}) rather than a single token. The paper argues (Section 3.1) that single-token generation is insufficient for detecting tokenization mismatches because the mismatch manifests only when the boundary of the drafter's token sequence is compared against each verifier's tokenization of the same character sequence. Consider the word "Incorrect": the drafter might tokenize it as three tokens (Inc, orr, ect), while a verifier tokenizes it as a single token (Incorrect). If the drafter only generates the first token Inc, the system cannot determine whether this token creates a boundary that is compatible with the verifier's tokenization — it would need to see the full word to know that the verifier's boundary falls at "Incorrect" not "Inc". By generating multiple tokens, the drafter provides enough context for the retokenization alignment check described below. The paper sets n = 5 as default (determined empirically in the ablation, Table 6, where n = 5 provides the best accuracy-efficiency tradeoff).

At each iteration of the algorithm, the drafter generates n tokens conditioned on the current prefix (which may include previously ensembled tokens from earlier iterations). These tokens form a "lookahead window" that the verifiers will examine.

OOV-like Token Verification

This is the first of the two checks in the Verify step and is the primary mechanism for ensuring stability. The check determines whether conditioning a verifier model on a particular drafter token would corrupt that verifier's next-token probability distribution by forcing it into an out-of-distribution prefix state.

Formal definition of an OOV-like token. Given a drafter token t_j at position j in the generated sequence, and a verifier model LLM_v, the paper defines t_j as an OOV-like token for LLM_v if the tokenization boundary up to and including t_j does not align with any tokenization boundary in LLM_v's tokenization of the same character sequence. The formal condition is stated in Equation (1):

$$t_j \text{ is OOV-like in LLM}_v \iff \forall x \in [0, v_{i+n} - 1], \text{Decode}(t_{<j+1}) \neq \text{Decode}(t^v_{<x+1})$$

where $\text{Decode}(\cdot)$ is the operation that merges tokens back into their character string representation, $t_{<j+1}$ is the drafter's token prefix up to and including position j, and $t^v_{<x+1}$ is the verifier's token prefix up to and including position x in the verifier's own tokenization of the same character string.

What this equation computes in operational terms. The check works as follows. First, the verifier retokenizes the entire drafter-generated sequence up to position i+n-1 using its own tokenizer, producing a sequence of tokens t^v_{1}, ..., t^v_{v_{i+n}-1} where v_{i+n}-1 is the number of tokens in the verifier's tokenization (which may differ from the drafter's count due to different segmentation). Then, for each position j in the drafter's sequence, the system decodes the prefix t_{<j+1} into a character string and checks whether this character string exactly matches the decoded character string of any prefix of the verifier's tokenization. If no verifier prefix corresponds to the same character string, then the drafter token t_j is OOV-like for that verifier — meaning the character boundary at position j in the drafter's segmentation does not exist in the verifier's segmentation.

Concrete example (from the paper's description). For the word "Incorrect," suppose the drafter tokenizes it as (Inc, orr, ect) and a verifier tokenizes it as (Incorrect). The evaluation proceeds as follows. After generating "Inc": the character string is "Inc". The verifier's tokenization of the full sequence includes the token "Incorrect" — the prefix "Incorrect" decodes to "Incorrect", which does not equal "Inc". No verifier token prefix decodes to exactly "Inc". Therefore, Inc is OOV-like. After generating "orr": the character string is now "Incorr". Still no verifier token prefix matches this string. Therefore, orr is also OOV-like. After generating "ect": the character string is "Incorrect". The verifier's token "Incorrect" decodes to exactly "Incorrect". A match exists. Therefore, ect is not OOV-like — it marks a valid tokenization boundary for the verifier.

How this check is used in the verification pipeline. The OOV-like token check does not determine whether ensembling happens at t_j. Rather, it determines whether ensembling is allowed at the subsequent token t_{j+1}. The logic, as stated in Section 3.2(i), is: "If t_j is identified as OOV-like by any verifier, ensembling is not triggered at the subsequent token t_{j+1}." This is a conservative safety mechanism. If token t_j is OOV-like, then conditioning verifier models on it (which would happen if ensembling selected a replacement at this position and that replacement were fed as context) could corrupt their distributions. By preventing ensembling at the next position, the system ensures that verifiers are not asked to evaluate ensemble candidates when their internal state has been compromised by an OOV-like prefix.

Consequence for the generation process. When a verifier encounters an OOV-like token, its next-token probability distribution at that position is considered unreliable. The drafter's token at the problematic position is accepted as-is (no ensembling occurs), and generation continues without correction. The paper shows qualitative examples in Appendix K (Figures 2, 8, 11) where existing methods without this check produce garbled, repetitive output — e.g., "SofÃAÂAÂÃ..." — because an OOV-like token corrupts one model, which then feeds corrupted tokens into subsequent ensemble steps, compounding the error. SAFE's OOV-like token verification prevents this cascade by identifying and quarantining the positions where distributions become unreliable.

Ensemble Distribution Verification

This is the second check in the Verify step and is the primary mechanism for improving efficiency. It determines whether ensembling at a given token position would actually change the selected token compared to simply using the drafter's prediction. If all models already agree on the next token (or the ensemble distribution would select the drafter's token anyway), the costly aggregation and vocabulary alignment steps are skipped.

Why avoid explicit ensemble distribution construction. The naive approach to checking ensemble consensus would be to construct the full ensemble distribution — which requires aligning vocabularies across all models (the computationally expensive step) — and then checking whether the argmax matches the drafter's token. This defeats the purpose: the expensive operation is performed regardless of whether ensembling is needed. SAFE instead uses two lightweight conditions that can be verified using only each model's own distribution, without cross-model vocabulary alignment.

Condition 1: Unanimous consensus among verifiers. Formally, this condition is met if:

$$t^v_{v_j} = \arg\max_t P_v(t \mid t^v_{<v_j}) \text{ for all LLM}_v \in \mathcal{M}_{ver}$$

where $t^v_{v_j}$ is the token in verifier $v$ 's tokenization that corresponds to the drafter's token $t_j$ , $t^v_{<v_j}$ is the verifier's tokenization of the prefix preceding this position, and $P_v(\cdot \mid t^v_{<v_j})$ is the verifier's next-token probability distribution conditioned on that prefix.

What this condition means operationally. Each verifier independently computes its own next-token probability distribution given the current prefix (which has been retokenized into the verifier's own tokenization scheme). The verifier then checks whether the token $t^v_{v_j}$ — the token in its vocabulary that corresponds to the drafter's proposed next token — is the single most probable token in its distribution (the argmax). If every verifier's argmax corresponds to the drafter's token, then the drafter's token is unanimously the top choice of all verifiers. In this case, the ensemble distribution — if constructed — would also select this token, since averaging distributions that all peak at the same token preserves the peak. (Theorem 1 in Appendix C proves this formally; see below.)

Condition 2: Average probability above one half. The second condition provides an alternative path to skipping ensembling that is more permissive than unanimity. It is met if:

$$\frac{1}{|\mathcal{M}_{ver} \cup \mathcal{M}_{draft}|} \sum_{LLM_v \in \mathcal{M}_{ver} \cup \mathcal{M}_{draft}} P_v(t^v_{v_j} \mid t^v_{<v_j}) > \frac{1}{2}$$

where the average is taken over all models — both verifiers and the drafter — and $P_v(t^v_{v_j} \mid t^v_{<v_j})$ is each model's assigned probability to the token corresponding to the drafter's proposed token.

What this condition means operationally. Even if the models don't unanimously agree on the top token (e.g., one verifier's argmax is different from the drafter's token), the drafter's token may still be selected by the ensemble if its average probability across all models exceeds 0.5. The intuition is that a probability mass above 0.5 leaves less than 0.5 for all other tokens combined, so no single other token can have a higher average probability. In a probability distribution over the entire vocabulary, two distinct tokens cannot both have probability strictly greater than 0.5, since the total mass is 1.0.

Why these conditions are correct. The paper provides a formal proof (Theorem 1, Appendix C) that either condition guarantees $t_j = \arg\max_t P_{ens}(t \mid t_{<j})$ , where $P_{ens}$ is the full ensemble distribution (average of all models' distributions aligned to the drafter's tokenization). The proof has two cases. For the unanimity condition: if all verifiers have $t^v_{v_j}$ as their argmax, then for any alternative token $u$ , each model assigns $P_v(t^v_{v_j}) \geq P_v(u)$ . Averaging across models preserves this inequality, so $P_{ens}(t_j) \geq P_{ens}(u)$ for all $u$ , making $t_j$ the ensemble argmax. For the average-probability condition: if $P_{ens}(t_j) > 1/2$ , then no other token can have $P_{ens}(u) \geq P_{ens}(t_j)$ because the total probability mass is 1.0 and two tokens cannot both exceed 0.5. Therefore $t_j$ must be the unique argmax.

Practical effect on efficiency. When either condition is met, the Verify step declares the drafter's token as accepted without ensemble, and the algorithm skips the costly aggregation step. This means that on tokens where models exhibit strong agreement — which the paper's Table 2 shows is the case for the vast majority of tokens in mathematical reasoning (only ~4.85% of tokens are ensembled on math datasets) — the system avoids both vocabulary alignment and distribution averaging, operating at near-single-model speed.

Ordering of the two conditions. The paper specifies that the OOV-like token verification is performed first: ensembling can only be considered at token $t_j$ if the preceding token $t_{j-1}$ is not OOV-like. Then, if the OOV-like check passes, the ensemble distribution verification is applied. Only tokens that pass both checks — safe to ensemble and potentially beneficial to ensemble — trigger the full ensemble operation.

The drafter's token is accepted if conditions are met. Algorithmically, if a token $t_j$ passes both checks, it means that either (a) the token is unanimously the top prediction or (b) its average probability exceeds 0.5. In either case, the ensemble distribution would select exactly this token, so no replacement is needed. The drafter's token is accepted as-is. If a token fails either check — the preceding token was OOV-like, or neither unanimity nor the 0.5 threshold is met — then ensembling is triggered at that position (see next subsection), and the resulting ensemble-selected token replaces the drafter's token.

The verification process operates at the earliest problematic token. The Verify step (Algorithm 1, lines 6–14) iterates through the drafter's tokens sequentially from $j = i$ to $j = i+n-1$ . At the first token $t_j$ that requires ensembling (fails either check), the ensemble is performed, the token is replaced, and generation resumes from position $j+1$ . All tokens after position $j$ in the drafter's lookahead are discarded because they were conditioned on the (now-replaced) original token $t_j$ . This is an "early exit" mechanism: the verifiers don't need to check every token in the lookahead window — they stop at the first position where intervention is needed.

The verifiers process the lookahead in a single forward pass. A crucial efficiency property: the verifiers do not generate tokens autoregressively. They receive the drafter's full n-token lookahead as input and compute next-token probability distributions at each position in a single transformer forward pass (paralleling how speculative decoding verifies a drafter's candidate sequence). This means the computational cost of verification is dominated by one forward pass per verifier per n-token chunk, rather than n sequential forward passes.

The Ensemble Step and Probability Sharpening

When a token position is identified as requiring ensembling (i.e., the drafter's token at position j fails either the consensus or OOV-like safety check), the system constructs the full ensemble distribution and selects the most probable token from it. This is where the underlying ensemble method (UniTE, GaC, or any other probability-level method) is invoked.

Ensemble distribution construction. The paper uses the existing ensemble method's aggregation procedure:

$$P_{ens} \leftarrow \text{AVERAGEDIST}(\{P_v(\cdot \mid p, t^v_{<v_j})\}_{v \in \mathcal{M}_{ver} \cup \mathcal{M}_{draft}})$$

where $\text{AVERAGEDIST}$ is the method-specific function that takes all models' next-token probability distributions (each defined over its own tokenizer's vocabulary), aligns them into a shared representation, and averages them. For UniTE, this involves extracting the top-k tokens from each model's distribution and aligning via their text representations; for GaC, this involves mapping each model's distribution to the vocabulary union. The output $P_{ens}$ is a probability distribution over the drafter's tokenizer's vocabulary (or the union vocabulary, depending on the method).

The structure of the problem that sharpening addresses. The paper identifies a subtle issue that arises specifically when ensembling across heterogeneous tokenizers: probability mass for the same semantic token gets diffused across multiple subword variants. Different tokenizers may represent the same word using different subword segmentations. For example, one model might assign high probability to the token " Incorrect" (a single token), while another might assign probability across three tokens (" Inc", "orr", "ect"). When these distributions are aligned and averaged, the probability mass for "Incorrect" is spread across multiple entries in the ensemble distribution, making the distribution overly smooth — that is, $\max P_{ens} < 0.5$ even when all models strongly agree on the word being generated.

This smoothness is problematic for greedy token selection: if the peak probability falls below 0.5, it becomes harder to confidently identify the "correct" token, and the argmax may land on a suboptimal subword token rather than the intended word-level token. The paper's probability sharpening strategy addresses this by consolidating the diffused probability mass onto the most plausible token.

Sharpening strategy 1: Heuristic prefix-based reallocation. This strategy (used as the default in the main experiments) operates on the drafter's tokens within the ensemble distribution. For a given drafter token t_j, if its initial probability exceeds a threshold $\lambda$ (set to 0.1 in the main experiments), the system reallocates to it the probability mass from all tokens t_i that share t_j as a prefix. Formally:

$$P_{ens}(t_j) \leftarrow P_{ens}(t_j) + \sum_{t_i : t_i.\text{startswith}(t_j)} P_{ens}(t_i), \quad \text{where } P_{ens}(t_j) > \lambda$$

What this equation computes operationally. After the ensemble distribution is constructed, the system scans through the drafter's tokens (which are entries in the ensemble distribution). For each drafter token $t_j$ whose ensemble probability exceeds $\lambda = 0.1$ , the system identifies all other tokens $t_i$ in the vocabulary for which $t_j$ is a string prefix — these are tokens that begin with the same character sequence as $t_j$ but extend it further (e.g., if $t_j$ is " Inc", then " Incorr" and " Incorrect" would match). The probability mass assigned to these longer tokens is transferred to $t_j$ , consolidating the diffused mass onto the common prefix token. This effectively says: "if the models collectively want to produce a word that starts with 'Inc', but some models produce longer subwords and others produce shorter ones, assign the combined probability to the shortest common prefix."

The $\lambda$ threshold prevents inflating low-quality tokens. Without the threshold, the reallocation could assign probability mass to tokens that have very low initial support. The threshold ensures that only tokens with non-trivial initial probability are candidates for receiving reallocated mass. The paper chooses $\lambda = 0.1$ as default and reports an ablation (Table 5) showing that performance is relatively insensitive to the exact value, though setting it too high ( $\lambda = 0.3$ ) reduces effectiveness because fewer tokens qualify for sharpening.

Sharpening strategy 2: Geometric mean aggregation. The second strategy replaces the arithmetic mean in the ensemble distribution with the geometric mean:

$$P_{ens}(t) \propto \left(\prod_{v} P_v(t^v \mid t^v_{<v_j})\right)^{1/|\mathcal{M}|}$$

where the normalization ensures a valid probability distribution. The geometric mean strongly penalizes tokens that receive low probability from any individual model. If one model assigns near-zero probability to a token, the geometric mean for that token also approaches zero, regardless of how high the other models' probabilities are. This has the effect of concentrating probability mass on tokens that enjoy broad support across all models.

Why the geometric mean concentrates probability. In the arithmetic mean, a token with probabilities [0.9, 0.9, 0.01] across three models gets an average of (0.9 + 0.9 + 0.01)/3 ≈ 0.603. In the geometric mean, the same token gets (0.9 × 0.9 × 0.01)^{1/3} ≈ 0.201 — dramatically lower because the near-zero entry dominates the product. This penalizes tokens that are strongly supported by some models but dispreferred by others, implicitly selecting for consensus across models. Table 5 shows that the geometric mean yields the strongest performance (82.73 average across three benchmarks vs. 82.42 for the heuristic with $\lambda = 0.1$ ).

After sharpening, the argmax is selected. Once the ensemble distribution is (optionally) sharpened, the most confident token is selected:

$$t_j \leftarrow \arg\max_t P_{ens}(t \mid p, t_{<j})$$

This token replaces the drafter's original token at position $j$ . The drafter then resumes autoregressive generation from this ensembled token, starting a new Generate phase.

The Full Generate-Verify-Ensemble Algorithm

Algorithm 1 in the paper provides the complete pseudocode for the iterative cycle. Here I walk through the algorithm in prose to make the control flow explicit.

Initialization. The algorithm starts with a prompt p, a drafter model Mdraft, a set of verifier models Mver, and a lookahead length n (default 5). A position counter i is initialized to 1, and the prefix t_0 is set to the beginning-of-sequence (BOS) token.

Main loop. The algorithm iterates until an end-of-sentence token is generated by the drafter.

Step 1: Generate (line 4). The drafter produces n tokens t_i, ..., t_{i+n-1} autoregressively, conditioned on the prompt and the current prefix t_{<i}. This prefix includes any tokens from previous iterations that may have been replaced by ensemble operations.

Step 2: Retokenize (line 5). Each verifier model v independently tokenizes the full sequence t_{<i+n} using its own tokenizer, producing t^v_{<v_{i+n}} — the verifier's tokenization of the same character sequence. The drafter also produces its own tokenization of this sequence (which is trivially the original tokens, since the drafter generated them).

Step 3: Verify (lines 6–14). The algorithm iterates through the drafter's tokens from j = i to j = i+n-1 in order. For each token t_j:

OOV-like check: If the preceding token t_{j-1} is OOV-like for any verifier (determined by Equation 1), then ensembling is skipped at t_j. The algorithm continues to the next token in the sequence (implicitly — the OOV-like check gates the ensemble check).
Ensemble distribution verification: If t_{j-1} is not OOV-like, the system applies the two consensus conditions described above. For each verifier, it identifies the token t^v_{v_j} in the verifier's tokenization that corresponds to the drafter's token t_j (this is done by aligning the token boundaries via the decoding check). It then checks:
1. Unanimity: Does t^v_{v_j} = argmax P_v(t | t^v_{<v_j}) for every verifier? If yes, the drafter's token is accepted (skip ensembling).
2. Average threshold: Is the average probability across all models (drafter + verifiers) for their corresponding tokens greater than 0.5? If yes, the drafter's token is accepted (skip ensembling).
If either condition is met: The drafter's token t_j is accepted as-is. The algorithm moves to the next position j+1 and repeats the checks.
If neither condition is met: This token requires ensembling. The algorithm executes the full Ensemble step (lines 8–9): construct the ensemble distribution via the underlying method, optionally apply probability sharpening, and select t_j = argmax P_ens. The position counter i is set to j+1, and the inner loop over j terminates — all subsequent tokens in the drafter's lookahead (t_{j+1}, ..., t_{i+n-1}) are discarded because they were conditioned on the original (now-replaced) t_j. The algorithm returns to the Generate step (Step 1) with the new prefix that includes the ensembled token.
If the loop completes without triggering ensembling: All n tokens were accepted. The position counter i is advanced by n (i <- i + n), and the algorithm returns to the Generate step with the extended prefix.

Why the "early exit" on first ensemble point. This design is critical for correctness. When a token t_j is replaced by the ensemble, the subsequent tokens t_{j+1}, ..., t_{i+n-1} were generated by the drafter under the assumption that t_j had a particular value. Since that value has changed, these tokens are no longer valid continuations — they were conditioned on stale context. The algorithm therefore discards them and regenerates from the corrected position. This mirrors the rejection-and-resampling behavior in speculative decoding, but the rejection criterion here is based on ensemble disagreement rather than statistical divergence from a target distribution.

KV cache management. A practical implementation challenge that the paper addresses (detailed in Appendix D) is maintaining consistent KV caches across models. When the ensemble replaces a drafter token, each model's KV cache contains key-value pairs computed for the original token at that position, but the actual output now uses the ensemble-selected token. Since subsequent forward passes condition on the actual output sequence, the stale cache entries must be removed. The paper's solution (Figure 8 in the appendix) is to prune each model's KV cache by a fixed buffer at the end of every ensemble step, removing the entries corresponding to the discarded tokens and ensuring the cache aligns with the ensembled output before the next forward pass. This is applied to all models in the ensemble, including baselines in the experiments, to ensure fair comparison.

Computational cost model. The cost of SAFE can be broken down as follows. For each iteration of the outer loop (handling n tokens):

Generate: The drafter performs n autoregressive forward passes (one per token). This is the primary sequential cost.
Retokenize: Each verifier tokenizes the n-token lookahead. This is a deterministic, non-neural operation (tokenizer lookup) and is negligible in cost.
Verify: Each verifier performs a single forward pass on the full prefix plus lookahead. This computes next-token probability distributions at all positions simultaneously (teacher-forcing mode). The number of verifier forward passes per iteration is |Mver|, independent of n.
Ensemble: If triggered, the ensemble distribution is constructed for one token position. This involves the vocabulary alignment cost of the underlying method (e.g., top-k extraction and string matching for UniTE).

The key efficiency gain over uniform ensembling is twofold. First, only the drafter performs autoregressive generation — the verifiers operate in single forward passes, not sequential token-by-token generation. Second, ensembling is triggered for only a small fraction of tokens (E/T values in Table 2 range from <1% to ~20%, with math datasets averaging 4.85%). This means the expensive vocabulary alignment step is invoked rarely rather than at every token.

The n hyperparameter tradeoff. Longer lookahead windows (n) mean the verifiers can detect tokenization mismatches over a broader span, improving accuracy. However, longer windows also mean that when ensembling is triggered (which forces discarding all subsequent tokens in the window), more drafter computation is wasted because more tokens are discarded and must be regenerated. The paper's ablation (Table 6, Figure 10) shows that n = 5 provides the best balance, with n = 3 showing slightly lower accuracy due to insufficient context for mismatch detection, and n = 8 showing similar accuracy to n = 5 but higher latency due to more regeneration. The paper also notes that performance does not degrade with longer sequences (unlike traditional ensemble methods), since OOV-like tokens are systematically prevented.

Summary of Design Choices and Their Justifications

Speculative (drafter-verifier) architecture over uniform generation: Restricts expensive autoregressive generation to a single model, reducing the per-token cost from k × N forward passes (for k models generating N tokens) to 1 × N drafter passes plus (k-1) × (N/n) verifier passes — a substantial reduction when n > 1. This extension of speculative decoding to heterogeneous tokenizers is a novel contribution of the paper.
OOV-like token check over confidence-based gating: GaC uses a threshold on the main model's probability to decide when to ensemble, which indirectly reduces OOV-like exposure but provides no guarantee. SAFE's boundary-matching check guarantees that verifiers are never conditioned on prefixes inconsistent with their tokenization, directly addressing the root cause of the degradation in Table 1.
Unanimity and 0.5-threshold checks over full ensemble construction: Avoids the cost of vocabulary alignment when models already agree, which is the common case (only 4.85% of tokens are ensembled on math tasks). The correctness proof (Theorem 1) ensures these checks never change the ensemble's selected token compared to full distribution construction.
"Earliest problematic token" early exit over checking all tokens: Prevents wasted computation on tokens that will be discarded anyway due to upstream replacements. This mirrors speculative decoding's acceptance criterion but adapted for ensemble disagreement rather than distributional divergence.
Probability sharpening over raw averaging: Addresses the distribution-smoothing effect caused by aligning heterogeneous tokenizers, where the same word's probability mass is spread across different subword tokens. The geometric mean variant is theoretically appealing because it naturally penalizes tokens that lack broad model support, while the heuristic variant is simpler and compatible with methods that require arithmetic averaging.
Drafter as the best-performing model over random selection or routing: Ensures that in consensus cases (the majority of tokens), the output quality matches the best individual model. Routing or dynamic selection would add complexity without clear benefit, since the ensemble correction mechanism intervenes precisely when the best model is wrong and other models agree on the correct token.
Fixed n = 5 lookahead over dynamic or adaptive lengths: Simplicity and empirically validated performance. Dynamic scheduling based on observed mismatch rates is mentioned as potential future work but not explored.

4. Key Insights and Innovations

Innovation 1: Tokenization Boundary Mismatch as the Root Cause of Cascade Failure in Probability-Level Ensembling

The paper's most fundamental diagnostic contribution is the identification and formalization of OOV-like tokens as the specific mechanism through which probability-level ensemble methods catastrophically degrade under chain-of-thought prompting. This is not an incremental fix to an existing method — it is a new diagnostic concept that explains a previously puzzling empirical phenomenon and redirects the research agenda for an entire class of approaches.

Prior to this work, the dominant framing of the token-level ensemble problem was vocabulary alignment — the technical challenge of mapping probability distributions across heterogeneous vocabularies. Methods like GaC, DEEPEN, and UniTE were all evaluated on their ability to construct accurate ensemble distributions given this alignment challenge, and they succeeded on short-form tasks where outputs span only a few tokens. The implicit assumption was that the same alignment techniques would transfer to long-form generation. When the paper's Table 1 shows UniTE dropping 15.2 percentage points on MATH500 under CoT, it demonstrates that this assumption is false — but critically, the failure is not because the alignment algorithm is wrong. It's because uniform token-level ensembling introduces distributional corruption into the autoregressive context that no alignment algorithm can recover from.

The conceptual move is subtle but profound. The paper reframes the problem from "how do we align different vocabularies?" (a static, single-step mapping challenge) to "when is it safe to condition a model on a token selected by an ensemble?" (a dynamic, state-dependent decision). An OOV-like token — So when another model expects Sofia as an atomic unit — is not a vocabulary matching problem; So exists in all vocabularies. The problem is that the token boundary at which it appears is inconsistent with the tokenization scheme that model uses to process text, forcing that model into an out-of-distribution prefix state. The distinction between "token exists in vocabulary" and "token boundary is consistent with expected segmentation" is the key conceptual contribution — it's not about what tokens are available, but about what token boundaries are valid conditioning points for each model's autoregressive process.

The evidence for this being the root cause rather than a contributing factor is stark. Table 1 shows that UniTE on MMLU-redux without CoT achieves 69.36% (matching or exceeding individual models), but on MATH500 with CoT drops to 59.6%. The critical variable is not the ensemble method, the model combination, or the benchmark domain — it's output length. Short outputs encounter few tokenization boundary mismatches because there are few positions where boundaries can diverge. Long outputs accumulate mismatches, and each mismatch corrupts the next-token distribution for at least one model, which then corrupts the ensemble output, which creates more mismatches — a compounding cascade. The paper's Figure 2 (left) shows the qualitative manifestation: unnatural token repetition, garbled characters, and incoherent reasoning that no amount of vocabulary alignment could fix because the underlying distributions being aligned have already been corrupted by invalid conditioning prefixes.

This diagnostic has implications beyond SAFE. It suggests that any token-level ensemble method intended for long-form generation must incorporate a mechanism for detecting or preventing tokenization boundary mismatches. It also explains the discrepancy in the prior literature between positive results on short-form tasks and the negative results that would emerge if anyone had tested on long-form generation — the failure mode was always latent in the approach, just never triggered at the evaluation scales being used.

Significance: This is a fundamental diagnostic contribution, not a performance enhancement. It identifies a structural failure mode that limits an entire class of approaches and provides a formal criterion (Equation 1) for when that failure mode is activated. The magnitude of the contribution is not measured in accuracy gains but in explaining why prior methods fail and what any successful method must address.

Innovation 2: The Distinction Between "How to Ensemble" and "Whether to Ensemble" as a First-Class Design Dimension

The paper introduces a conceptual separation that was absent from prior work: the question of which token to select during ensembling (the aggregation mechanism) is orthogonal to the question of whether to perform ensembling at a given token position (the gating mechanism), and progress on the former does not substitute for progress on the latter. This reframes the research landscape from a single-axis optimization (better alignment → better ensemble) to a two-axis design space (better alignment × better gating), where most prior work occupied only one axis.

This distinction is not merely taxonomic — it has practical consequences that the paper demonstrates empirically. GaC (Yu et al., 2024) is the only prior method that makes any "whether to ensemble" decision: it triggers ensembling only when the main model's probability falls below 0.5. This threshold-based gating was introduced as a computational efficiency measure — skip the alignment cost when the main model is already confident. But the paper observes that this threshold inadvertently provides partial protection against OOV-like tokens because it reduces the frequency of ensemble operations, thereby reducing the opportunities for boundary mismatches to occur. However, GaC's gating is driven by model confidence, not by tokenization safety or ensemble necessity. When confidence is low, GaC ensembles at every subsequent token without considering whether those tokens are safe conditioning points for all models.

SAFE's gating mechanism is fundamentally different in what it gates on. The OOV-like token verification (Section 3.2, i) gates on tokenization boundary compatibility — a property of the relationship between the drafter's tokenization and each verifier's tokenization. The ensemble distribution verification (Section 3.2, ii) gates on inter-model consensus — a property of the relationship between the models' probability distributions. Neither of these is reducible to model confidence, and both are necessary: a token can be highly confident for every model individually but still be unsafe to ensemble (if it creates a boundary mismatch), or safe but unnecessary (if all models already agree on it).

The significance of this separation is that it makes the gating mechanism a first-class design component alongside the aggregation mechanism. Prior work treated "when to ensemble" as either an implementation detail (ensemble at every token, as in UniTE and DEEPEN) or a crude heuristic (ensemble when confidence is low, as in GaC). SAFE treats it as a algorithmic problem with its own formal criteria and correctness guarantees. Theorem 1 in Appendix C proves that the consensus conditions never change the ensemble's selected token — the gating is lossless with respect to the aggregation mechanism; it only skips operations that would not alter the output anyway. This is a stronger property than GaC's heuristic threshold, which can skip ensemble operations even when the ensemble would select a different (better) token than the main model.

Evidence for the distinct contributions of gating vs. aggregation. Table 2 shows that applying SAFE's gating to UniTE (UniTE + SAFE) recovers performance from a 15.2-point degradation to a 2.8-point improvement over the best individual model. Applying SAFE's gating to GaC (GaC + SAFE) produces smaller relative gains — because GaC already has some gating — but still improves performance while reducing the ensemble frequency (E/T). The fact that SAFE's gating improves both UniTE and GaC, despite their very different aggregation mechanisms, demonstrates that gating quality is an independent contributor to ensemble performance.

Significance: This is a reframing contribution rather than a novel mechanism. It establishes that the "whether" question is not an optimization to be added later but a co-equal design dimension that determines whether an ensemble method is viable at all for long-form generation. The paper's demonstration that existing methods (UniTE) can be made practical simply by adding a principled gating layer — without changing their aggregation mechanism — validates this reframing empirically.

Innovation 3: Speculative Verification as a Unifying Framework for Selective Ensembling with Heterogeneous Tokenizers

The paper adapts the speculative decoding paradigm (Leviathan et al., 2023) to the ensemble setting in a way that is both novel and non-obvious: extending the drafter-verifier architecture to models with heterogeneous tokenizers by replacing the standard statistical acceptance criterion with a tokenization-compatibility and consensus-based acceptance criterion. This is not a straightforward application of speculative decoding to ensembling — it requires solving a problem that the speculative decoding literature had not addressed.

In standard speculative decoding, the target model evaluates the drafter's proposed tokens using a divergence-based acceptance criterion: a drafter token is accepted if the target model's probability for that token meets or exceeds the drafter's probability (or a sampled variant thereof). This criterion works because the drafter and target share the same tokenizer — the drafter's tokens are valid conditioning points for the target, and the target can directly compute probabilities for those exact tokens. In the heterogeneous-tokenizer ensemble setting, this assumption breaks in two ways. First, the drafter's tokens may not be valid conditioning points for the verifiers (the OOV-like token problem). Second, the verifiers cannot directly compute probabilities for the drafter's tokens because those tokens may not exist in their vocabularies (or may correspond to different subword segmentations).

The paper's contribution is an acceptance criterion that operates without requiring the verifiers to evaluate the drafter's tokens directly. Instead, each verifier retokenizes the drafter's sequence into its own tokenization, computes its own next-token distribution, and then the system checks whether the corresponding tokens in each verifier's vocabulary — the ones that align to the same character boundaries — satisfy the consensus conditions. This is a fundamentally different acceptance logic from speculative decoding: it gates on inter-model agreement rather than drafter-target divergence, and it incorporates tokenization-compatibility as a prerequisite for even considering a token.

The architectural parallel to speculative decoding — one model generates, the rest verify in parallel — is what gives SAFE its efficiency properties, but the conceptual novelty is in the verification criterion. Prior work on speculative decoding for ensembles (Fu et al., 2025) was limited to identical-tokenizer settings precisely because no criterion existed for the heterogeneous case. The paper provides that criterion through the combination of OOV-like token verification and ensemble distribution verification.

Why this matters beyond the specific algorithm. The drafter-verifier architecture enables a separation of concerns that makes ensemble methods scalable: the costly autoregressive generation is concentrated in one model, while the other models contribute through efficient parallel verification. This architecture is not tied to any specific ensemble aggregation method — SAFE wraps UniTE, GaC, or any other method equivalently. The paper thus provides a architectural template for future ensemble methods: separate the generation role from the verification role, and design verification criteria that account for tokenization heterogeneity and inter-model agreement.

Evidence for the architectural contribution. Figure 5 shows that SAFE achieves latency comparable to single-model generation even for long sequences (hundreds of tokens), where uniform ensemble methods are dramatically slower. Table 2 shows that this efficiency does not come at the cost of accuracy — SAFE actually improves accuracy over uniform ensembling because the selective gating prevents OOV-like corruption. The combination of improved accuracy and improved efficiency is rare and indicates that the selective approach is addressing a real structural problem rather than merely trading off one for the other.

Significance: This is both an architectural contribution (adapting speculative decoding to the ensemble setting) and an algorithmic contribution (designing acceptance criteria for heterogeneous tokenizers). It is incremental relative to speculative decoding in the sense that it borrows the generate-verify structure, but fundamental in the sense that the acceptance criterion is entirely novel and solves a problem that had blocked prior applications of speculative decoding to heterogeneous-model ensembles.

Innovation 4: The Empirical Finding That Ensembling Is Most Beneficial at a Small Fraction of Token Positions — and the Corollary That Uniform Ensembling Is Actively Harmful

The paper produces a striking empirical result that challenges the default assumption underlying prior probability-level ensemble methods: effective ensembling requires intervention at only a small fraction of token positions, and uniform ensembling — the standard practice — is not merely wasteful but actively degrades output quality. Table 2 reports E/T (Ensemble/Token) percentages showing that when SAFE is applied with UniTE, ensembling is triggered for only 3.82% of tokens on MATH500, 5.16% on GSM8K, and at most 18.60% on ARC-Challenge — yet the resulting accuracy substantially exceeds both individual models and uniform ensembling. This is a finding about the structure of the ensemble problem, not about a specific method.

The implication is that the tokens requiring ensemble intervention are sparsely distributed throughout a generation sequence and are concentrated at specific types of positions: points where models disagree and where the drafter's token is not clearly correct. The vast majority of tokens — those where models exhibit strong consensus — are generated identically whether or not an ensemble is performed. The paper's Theorem 1 formalizes this: the consensus conditions guarantee that skipping ensembling does not change the selected token. So uniform ensembling at these consensus tokens wastes computation (the alignment and aggregation cost) while also introducing risk (each ensemble operation is an opportunity to create an OOV-like token and corrupt subsequent generation). The net effect, visible in Table 1, is that uniform ensembling produces worse outputs than not ensembling at all.

This finding has a direct practical corollary: the primary design challenge for token-level ensemble methods is not aggregation quality but gating precision — the ability to identify the small fraction of token positions where ensembling is both safe and value-adding. Improving aggregation (e.g., better vocabulary alignment, more sophisticated probability combination) can only improve the tokens that are ensembled. If ensembling is applied at the wrong positions — introducing OOV-like corruption — better aggregation at those positions makes the problem worse, not better, because it more confidently selects incorrect tokens that then corrupt downstream generation.

The difficulty-dependent nature of ensemble frequency. The paper's Table 2 reveals a pattern that the authors note but don't fully explain: math datasets require ensembling at only 4.85% of tokens on average (for UniTE + SAFE), while general-domain datasets require 15.24%. The paper attributes this to "the nature of math responses, which often contain equations or structured expressions with limited variation, leading to higher agreement among verifier models." This is plausible but suggests a deeper point: the benefit of ensembling is task-dependent and format-dependent. In domains where the output space is highly constrained (mathematical notation, code syntax), models trained on similar data tend to converge on the same tokens, making ensembling rarely necessary. In domains with greater linguistic variability (open-ended reasoning, explanation), disagreement is more common and ensembling adds more value — but also introduces more risk of OOV-like corruption. This tradeoff — ensemble benefit increases with task complexity but so does ensemble risk — is a structural property of the problem that prior work had not surfaced because it was tested only on tasks at one extreme (short-form, constrained-output tasks).

The finding that more models is not always better. Table 2 also shows that three-model ensembling (Internlm3 + Qwen2.5 + EXAONE3.5 with UniTE + SAFE) does not consistently outperform the best two-model combination. The three-model ensemble achieves 84.59% average accuracy, while the best two-model pair (Internlm3 + Qwen2.5) achieves 84.20%. The marginal gain from adding a third model is 0.39 percentage points — well within noise on a 500-question test set — while the computational cost roughly doubles (three verifiers instead of one). This finding contradicts the intuition that "more models = more diverse knowledge = better ensemble" that motivates much ensemble research. The paper's interpretation — that "when model rankings are known, restricting ensembling to the top-2 models is both effective and efficient" — is pragmatic but also theoretically interesting: it suggests that marginal model diversity beyond two strong models provides diminishing returns for probability-level ensembling, possibly because the ensemble distribution becomes harder to sharpen (more models scatter probability mass across more subword variants) and harder to verify (consensus is harder to achieve with more participants).

Significance: This is an empirical finding with theoretical implications. It establishes that the ensemble problem is sparse — most tokens don't benefit from ensembling — and that the design target for future methods should be precise gating rather than better aggregation. It also provides practical guidance: for practitioners deploying ensemble methods, restricting to the top-2 models and focusing on gating quality will likely yield better results than adding more models or improving aggregation algorithms.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper evaluates on five benchmarks covering diverse domains. For general knowledge: MMLU-redux (Gema et al., 2025), a refined subset of MMLU (Hendrycks et al., 2021) containing 30 subjects with human-annotated corrections. For mathematical reasoning: MATH500 (Lightman et al., 2024) and GSM8K (Cobbe et al., 2021). For general reasoning: ARC-Challenge (Clark et al., 2018) and BBH (Suzgun et al., 2023). For BBH specifically, the authors filter to 15 subjects where the models exhibit comparable performance (listed in Appendix A: boolean expressions, causal judgement, date understanding, disambiguation qa, formal fallacies, logical deduction three objects, movie recommendation, navigate, penguins in a table, reasoning about colored objects, ruin names, salient translation error detection, snarks, temporal sequences, and tracking shuffled objects three objects). All benchmarks are evaluated under a zero-shot CoT setting, except BBH which uses 3-shot CoT. The choice of MATH500 and GSM8K is critical for the paper's claims because these benchmarks require extended reasoning chains where OOV-like token accumulation is most damaging. The inclusion of MMLU-redux without CoT (Table 1) serves as a control condition showing that existing methods work fine on short-form answers — the degradation emerges specifically under chain-of-thought prompting.
Base model(s). The paper uses three 7B-scale instruction-tuned models with substantially heterogeneous tokenizers as its primary evaluation set: Internlm3-8B-Instruct (Cai et al., 2024), Qwen2.5-7B-Instruct (Qwen et al., 2025), and EXAONE-3.5-7.8B-Instruct (An et al., 2024). These models are chosen specifically because they have similar capability levels (making ensemble meaningful — if one model dominated, there would be nothing to gain from combining) but markedly different tokenization schemes. Figure 4 quantifies this: on the Oxford 5000 word list (commonly used English words), tokenization agreement rates between model pairs range from only 46.8% to 63.81% — meaning roughly half of common English words are tokenized differently across models. This high degree of tokenization heterogeneity is what makes the OOV-like token problem severe and therefore makes this model set a strong test of SAFE's stability claims. For comparison, the paper also evaluates on two models with nearly identical tokenizers: Qwen2-7B-Instruct (Yang et al., 2024a) and Llama-3.1-8B-Instruct (Dubey et al., 2024), where more than 99% of Oxford 5000 words are tokenized identically. This second setting serves as an ablation on tokenization heterogeneity: if SAFE's benefits came purely from some other mechanism (e.g., the speculative architecture), they would appear equally in both settings. If they are genuinely driven by OOV-like token prevention, the benefits should be larger in the heterogeneous setting. The paper also experiments with 32B-scale models (Qwen2.5-32B-Instruct and EXAONE-3.5-32B-Instruct) in Appendix G to test scalability. All models use greedy decoding with a maximum output length of 2048 tokens, and each model is loaded onto a separate GPU (RTX 3090) with FP16 precision and FlashAttention-2 (Dao, 2023) enabled.
Metrics. The primary metric is task-specific accuracy (percentage of questions answered correctly), computed using standard evaluation protocols for each benchmark (exact match for math, multiple-choice selection for MMLU-redux and ARC-Challenge). The paper also reports E/T (Ensemble/Token) — the percentage of generated tokens at which ensembling is actually triggered — computed as # Ensemble operations / # Total tokens generated. This metric is crucial for evaluating SAFE's efficiency claims: low E/T values (like 3.82% on MATH500 with UniTE + SAFE in Table 2) demonstrate that SAFE's gating mechanism successfully identifies that only a small fraction of token positions require intervention. The paper additionally measures latency (wall-clock time in seconds) for varying output sequence lengths, shown in Figures 5 and 9, to evaluate the practical speed claims. Latency is measured end-to-end including all model forward passes, ensemble operations, and KV cache management.
Baselines. The paper applies SAFE to two recent state-of-the-art probability-level ensemble methods and compares against both the underlying ensemble methods without SAFE and the individual models:
- GaC (Yu et al., 2024): An ensemble method that takes the union of all model vocabularies and averages probability distributions in this shared space. Critically, GaC already has a partial gating mechanism — it performs ensembling only when the "main" LLM's next-token probability falls below 0.5. This makes GaC a stronger baseline than methods that ensemble uniformly, because its threshold-based gating indirectly provides some protection against OOV-like tokens (when the main model is confident, no ensemble operation occurs, so no opportunity for mismatch arises). The paper evaluates GaC both standalone and with SAFE's gating applied on top (GaC + SAFE).
- UniTE (Yao et al., 2025): The state-of-the-art method among those that ensemble at every generation step, using top-k token alignment rather than full vocabulary union. UniTE represents the "uniform ensembling" paradigm and is therefore the most vulnerable to OOV-like token corruption — it has no gating mechanism whatsoever and ensembles at every token unconditionally. The paper evaluates UniTE both standalone and with SAFE's gating (UniTE + SAFE).
- Individual models: Each model run independently with greedy decoding, serving as the lower bound that ensembles should improve upon.
The paper does not compare against non-probability-level ensemble methods (e.g., cascade structures like FrugalGPT, parallel methods like LLM-Blender, or hybrid methods like MoA) because these operate at the response level rather than the token level, making them categorically different approaches. The focus is specifically on improving probability-level methods for long-form generation. For the same reason, the paper does not compare against routing-based token-level methods (Co-LLM, CoSD) which also operate at the token level but select which single model to use rather than aggregating distributions.
Generation budget / compute accounting. The paper does not use a generation count budget in the traditional sense (e.g., "N generations per question") because SAFE is a deterministic greedy decoding method applied to a single generation per question. Instead, computational cost is measured in two complementary ways:
1. E/T (Ensemble/Token ratio): The fraction of tokens that undergo the full ensemble operation (vocabulary alignment + distribution aggregation + sharpening). This is the primary efficiency metric because the ensemble operation is the dominant computational cost beyond standard autoregressive generation. Lower E/T means fewer expensive operations.
2. Latency (wall-clock time): End-to-end generation time measured in seconds for varying output sequence lengths, reported in Figures 5 and 9. This captures the practical speedup from SAFE's speculative architecture and selective gating.
The paper does not report FLOP counts, which is a limitation — FLOPs would provide a hardware-independent efficiency measure. However, the latency measurements on fixed hardware (RTX 3090) serve as a practical proxy. Critically, the paper does not account for the cost of the drafter's discarded tokens in its efficiency analysis. When ensembling is triggered at an early position in the lookahead window, all subsequent tokens in that window are discarded and must be regenerated. The paper acknowledges this through the n hyperparameter ablation (Table 6, Figure 10) — longer n reduces efficiency because more tokens are wasted when ensembling intervenes — but does not quantify the fraction of total generation that is "wasted" regeneration. This unreported cost partially offsets the efficiency gains from reduced E/T.

The verifier models' computational contribution is also accounted for differently than the drafter's. Verifiers perform a single forward pass per n-token chunk (processing all n tokens in parallel via teacher forcing), while the drafter performs n autoregressive forward passes. This means verifier cost per token is approximately 1/n of drafter cost per token (plus the cost of the ensemble operation when triggered). The paper's latency measurements in Figures 5 and 9 capture this empirically but do not decompose the contributions.
Cross-validation / statistical protocol. The paper does not employ cross-validation or statistical significance testing. This is a notable methodological limitation. All results are reported on standard benchmark test sets with fixed model configurations obtained from a single set of hyperparameter choices (Section 4.2). The drafter sequence length n = 5 and sharpening threshold λ = 0.1 are selected through ablation studies on the same benchmarks being evaluated, which means the hyperparameters are tuned on the test data — a potential source of optimistic bias. The paper does not report confidence intervals, standard errors, or statistical tests for any of the accuracy comparisons. Given test set sizes (MATH500: 500 questions, GSM8K: ~1319 questions, MMLU-redux: varies by subject, ARC-Challenge: ~1172 questions, BBH: 15 subjects with varying question counts), small differences between methods (e.g., 0.2–0.5 percentage points in Table 2) may not be statistically reliable. The paper's key results — particularly the large-magnitude improvements like UniTE + SAFE vs. UniTE on MATH500 (77.4% vs. 59.6%) — are large enough to be robust to sampling variance, but finer-grained comparisons (e.g., whether GaC + SAFE vs. UniTE + SAFE is genuinely different) cannot be assessed without variance estimates.

For the BBH benchmark, the paper filters to 15 subjects "where models exhibit comparable performance" (Appendix A), following UniTE's observation that ensemble is meaningful when base models have similar performance levels. This filtering is reasonable for evaluating whether ensemble helps, but it means the BBH results do not represent performance on the full benchmark — they represent performance on a difficulty-controlled subset. The excluded subjects and the filtering criteria (what counts as "comparable"?) are not specified quantitatively.

Main Quantitative Results

Ensembling Models with Substantially Different Tokenizers (Table 2, Figures 5, 9)

The paper's central quantitative results appear in Table 2, which reports accuracy and E/T for two-model and three-model ensembles across all five benchmarks, comparing individual models, standalone GaC and UniTE, and GaC/UniTE augmented with SAFE. The headline finding is that SAFE rescues probability-level ensembling from catastrophic degradation under CoT while reducing ensemble frequency to a small fraction of tokens.

The catastrophic failure of UniTE under CoT. The most dramatic results in Table 2 are for UniTE without SAFE. Across all model pairs and all benchmarks, UniTE consistently underperforms even the worst individual model, often by large margins:

Internlm3 + Qwen2.5: UniTE achieves 59.6% on MATH500 vs. 74.8% for the best individual model (Internlm3) — a 15.2-percentage-point degradation. On GSM8K, UniTE drops to 75.06% vs. 91.81% (Qwen2.5) — a 16.75-point degradation. On MMLU-redux, it falls to 73.39% vs. 76.89% (Internlm3). On BBH: 79.58% vs. 82.26% (Internlm3). On ARC-C: 87.97% vs. 90.27% (Internlm3).
Qwen2.5 + EXAONE3.5: Even worse — UniTE drops to 53.75% on MMLU-redux (vs. 74.88% best individual), 43.4% on MATH500 (vs. 72.8% best individual), and 72.61% on ARC-C (vs. 90.44% best individual). The average degradation across all five benchmarks is 18.55 percentage points below the best individual model.
Internlm3 + EXAONE3.5: UniTE achieves 72.51% on MMLU-redux (vs. 76.89% best individual) and 73.6% on MATH500 (vs. 74.8% best individual) — relatively less degradation than the other pairs, but still below individual models.

These results empirically validate the paper's central diagnostic: uniform ensembling at every token is actively harmful under CoT, not merely wasteful. The degradation is not subtle — UniTE often performs worse than random guessing would on multiple-choice tasks.

Standalone GaC is more robust but still suboptimal. GaC's threshold-based gating (ensemble only when main model probability < 0.5) provides partial protection:

Internlm3 + Qwen2.5: GaC achieves 77.00% on MMLU-redux (+0.11 over best individual), 74.2% on MATH500 (-0.6), 91.28% on GSM8K (-0.53), 82.34% on BBH (+0.08), and 90.61% on ARC-C (+0.34). GaC roughly matches individual models rather than degrading catastrophically.
Qwen2.5 + EXAONE3.5: GaC achieves 76.01% on MMLU-redux (+1.13), 75.4% on MATH500 (+2.6), and 92.65% on GSM8K (+0.84). Here GaC actually improves over individual models — but notably, these are the model pairs where GaC's E/T is relatively high (13.42% on MMLU-redux), suggesting more frequent ensembling is being triggered.
Internlm3 + EXAONE3.5: GaC achieves 76.36% on MMLU-redux (-0.53), 75.8% on MATH500 (+1.0), and 90.78% on ARC-C (+0.34). Mixed results.

The key observation is that GaC's threshold-based gating is implicitly protecting against OOV-like tokens (when the main model is confident, no ensemble occurs, so no mismatch can be introduced), but this protection is incidental rather than designed. GaC gates on model confidence, not tokenization safety, so when confidence is low and ensembling is triggered, it still applies ensembling uniformly at subsequent tokens — potentially introducing OOV-like corruption at those positions. This explains why GaC performs well on some benchmarks (where confidence thresholds happen to align with safe ensemble points) but not reliably across all settings.

SAFE improves both GaC and UniTE while dramatically reducing ensemble frequency. The core quantitative argument for SAFE's effectiveness is that adding its gating mechanism improves both underlying ensemble methods, with the largest gains on the method (UniTE) that previously failed most severely:

UniTE + SAFE (Internlm3 + Qwen2.5): MATH500 jumps from 59.6% (UniTE alone) to 77.4% — a 17.8-percentage-point improvement — with E/T of only 3.82%. This exceeds the best individual model (74.8%) by 2.6 points. MMLU-redux improves from 73.39% to 77.81% (+0.92 over best individual), GSM8K from 75.06% to 92.04% (+0.23 over best individual), BBH from 79.58% to 82.97% (+0.71), ARC-C from 87.97% to 90.78% (+0.51). The average across all five benchmarks rises from 75.12% (UniTE alone, 7.75 points below best individual) to 84.20% (1.33 points above best individual).
UniTE + SAFE (Qwen2.5 + EXAONE3.5): Similar pattern — MATH500 improves from 43.4% to 76.4%, MMLU-redux from 53.75% to 76.54%, with E/T values ranging from 4.69% to 19.24%. The average rises from 62.85% to 83.63%.
UniTE + SAFE (Internlm3 + EXAONE3.5): MATH500 from 73.6% to 77.0%, MMLU-redux from 72.51% to 76.08%, average from 80.34% to 83.09%.
GaC + SAFE shows smaller but consistent improvements over standalone GaC. For Internlm3 + Qwen2.5: MATH500 improves from 74.2% to 76.0% (+1.8), while E/T drops from 1.04% to 0.71%. For Qwen2.5 + EXAONE3.5: MATH500 from 75.4% to 76.4% (+1.0), E/T from 2.31% to 1.09%. The fact that SAFE reduces E/T below GaC's already-low ensemble frequency while simultaneously improving accuracy confirms that SAFE is making better gating decisions: GaC was sometimes ensembling at unsafe positions and sometimes skipping ensemble at positions where it would have helped. SAFE's tokenization-compatibility check prevents the former, and its consensus-based check (which is more permissive than GaC's 0.5 threshold because it includes the unanimity condition) captures the latter.

The relationship between E/T and task domain. Table 2 reveals a consistent pattern across all model pairs: mathematical reasoning benchmarks require substantially less ensembling than general-domain benchmarks. For UniTE + SAFE (Internlm3 + Qwen2.5): MATH500 E/T = 3.82%, GSM8K E/T = 5.16%, vs. MMLU-redux E/T = 12.59%, BBH E/T = 10.35%, ARC-C E/T = 14.47%. Averaged across the three model pairs, math benchmarks require roughly 4.85% ensemble rate vs. 15.24% for general-domain benchmarks — approximately a 3× difference. The paper attributes this to "the nature of math responses, which often contain equations or structured expressions with limited variation, leading to higher agreement among verifier models" (Section 4.3). This is a plausible mechanism: mathematical notation constrains the space of plausible token sequences (equations must follow syntactic rules, numbers must follow specific formats), so models trained on similar corpora tend to converge on the same tokens, satisfying SAFE's consensus conditions more frequently and triggering fewer ensemble operations. General-domain reasoning, by contrast, admits greater linguistic variability — there are more ways to phrase the same logical argument — leading to more frequent inter-model disagreement and thus more ensemble interventions.

Three-model ensembling yields diminishing returns. The bottom row of Table 2 (three-model ensemble: Internlm3 + Qwen2.5 + EXAONE3.5 with UniTE + SAFE) reports an average accuracy of 84.59% across the five benchmarks. This is only 0.39 percentage points higher than the best two-model pair (Internlm3 + Qwen2.5 at 84.20%), yet the computational cost roughly doubles (two verifier models instead of one, each requiring forward passes and KV cache management). The E/T for three-model ensembling is 16.18% on MMLU-redux and 18.60% on ARC-C — marginally higher than the two-model case, not dramatically so. This finding contradicts the intuition that adding more models increases diversity and therefore ensemble benefit. The paper suggests (Section 4.3) that "when model rankings are known, restricting ensembling to the top-2 models is both effective and efficient" and that "when rankings are unknown, ensembling multiple comparable models provides stable, though not necessarily optimal, performance." This is a practically significant result: it implies that the marginal value of additional ensemble participants decreases rapidly beyond two strong models, and the computational budget is better spent on improving gating or aggregation for the top two rather than adding a third.

Latency results demonstrate near-single-model speed. Figures 5 and 9 show latency comparisons on MATH500 and MMLU-redux respectively, plotting wall-clock time against the number of generated tokens for individual models, standalone ensemble methods, and SAFE-augmented methods. The key finding: SAFE achieves latency comparable to running a single model, even when generating hundreds of tokens:

Figure 5(a): On MATH500, UniTE alone (which ensembles at every token) shows latency that grows rapidly with sequence length — approximately 80 seconds for 1000 tokens, roughly 2–3× the latency of individual models (~30–40 seconds). UniTE + SAFE closely tracks the individual model curves, staying within ~10 seconds of single-model latency even at 1000 tokens.
Figure 5(b): GaC alone is already faster than UniTE (due to its threshold gating), showing perhaps 50–60 seconds at 1000 tokens, but GaC + SAFE brings this down to near-single-model speeds (~35–40 seconds).
Figure 5(c): This panel isolates the contribution of the KV cache management strategy. GaC without the paper's KV cache optimization shows dramatically worse latency — approximately 200 seconds at 800 tokens, roughly 4–5× slower than the optimized version. This demonstrates that the KV cache consistency problem (stale cache entries from replaced tokens causing recomputation) is a major practical bottleneck for ensemble methods, and that the paper's pruning strategy (Appendix D) is essential for achieving the reported speeds. Note the different time-axis scale in panel (c) compared to (a) and (b).

The latency advantage comes from three sources identified in Section 4.3: (1) only the drafter performs autoregressive generation (the expensive sequential part), while verifiers contribute single parallel forward passes per chunk; (2) selective ensembling reduces the frequency of costly ensemble operations; (3) the KV cache pruning strategy prevents wasted recomputation when tokens are replaced.

Crucially, the paper does not report throughput (tokens per second) or compare latency under batched inference — the measurements are for single-sequence generation. In batched settings, the speculative architecture's advantage might diminish because all models could process multiple sequences in parallel, changing the relative cost balance between drafter autoregression and verifier verification. This is an unexamined practical consideration.

Ensembling Models with Highly Similar Tokenizers (Table 3)

Table 3 reports results for Qwen2-7B-Instruct and Llama-3.1-8B-Instruct, whose tokenizers agree on >99% of Oxford 5000 words. This setting serves as a control condition: if SAFE's benefits come primarily from OOV-like token prevention, they should be smaller here because OOV-like tokens rarely occur; if the benefits come from the speculative architecture or consensus-based gating alone, they should be comparable to the heterogeneous setting.

The results support the OOV-like token interpretation, but with interesting nuance:

Standalone UniTE is much more stable than in the heterogeneous setting — on MMLU-redux, UniTE achieves 68.90% vs. 69.25% for the best individual (Qwen2-7B), a degradation of only 0.35 points (compared to 3–15 point drops in Table 2). On MATH500, UniTE actually improves to 54.0% vs. 49.8% for the best individual (+4.2 points). However, on GSM8K, UniTE still degrades to 79.98% vs. 85.90% for the best individual (-5.92 points). This mixed pattern suggests that even with nearly identical tokenizers, uniform ensembling can sometimes underperform, though the effect is much less severe than with heterogeneous tokenizers.
SAFE still improves UniTE: UniTE + SAFE achieves 69.71% on MMLU-redux (+0.46 over best individual), 55.6% on MATH500 (+5.8), and 84.08% on GSM8K (-1.82 — still below the best individual, but improved from -5.92). The gains are smaller in magnitude than in Table 2 (e.g., MATH500 improvement from UniTE to UniTE + SAFE is +1.6 points vs. +17.8 points in the heterogeneous setting), consistent with the interpretation that OOV-like prevention is the dominant mechanism in heterogeneous settings.
GaC + SAFE shows the strongest result: On MATH500, GaC + SAFE achieves 59.4%, which is 9.6 points above the best individual model (Qwen2-7B at 49.8%). This is the largest improvement in the similar-tokenizer setting and is notable because GaC alone already achieves 52.4% (+2.6). The additional gain from SAFE's gating suggests that GaC's threshold-based gating was still making suboptimal decisions even when tokenization mismatch is rare — possibly skipping ensemble at positions where the ensemble would have helped, or ensembling at positions where it was unnecessary.

The takeaway from Table 3 is that SAFE provides benefits beyond OOV-like token prevention — the consensus-based gating (skipping ensemble when models agree) and the speculative architecture (reducing computational cost) are valuable even when tokenization mismatch is not a concern. However, the magnitude of benefit is substantially smaller, confirming that OOV-like token prevention is the dominant mechanism driving SAFE's gains in the heterogeneous setting that is the paper's primary focus.

Scaling to Larger Models (Appendix G, Table 7)

Table 7 in Appendix G reports results for ensembling 32B-scale models (Qwen2.5-32B-Instruct and EXAONE-3.5-32B-Instruct) on MMLU-redux and MATH500. This is an important scaling check: if SAFE's mechanisms are specific to 7B-scale models (e.g., because smaller models have noisier probability distributions or different error patterns), the practical value would be limited.

The paper reports MMLU-redux in two variants: MMLU-redux* (21 subjects, excluding 9 subjects where Qwen2.5-32B largely outperforms EXAONE-3.5-32B by more than 10%) and the full MMLU-redux (30 subjects). The exclusion is motivated by UniTE's finding that "ensemble is meaningful when the base models exhibit similar performance levels" (Appendix A). On the excluded subjects, the performance gap is so large that the weaker model would likely degrade rather than improve the ensemble. This is a reasonable methodological choice — ensembling a model that is substantially worse than another on a specific subject is unlikely to help — but it means the results are curated to the favorable subset. The excluded subjects are: college chemistry, college mathematics, college physics, formal logic, electrical engineering, high school chemistry, professional accounting, clinical knowledge, and econometrics.

The results show that SAFE continues to be effective at the 32B scale:

MMLU-redux* (curated 21 subjects): GaC alone achieves 84.70% vs. 85.06% for the best individual (Qwen2.5-32B) — a slight degradation. GaC + SAFE improves to 85.11% (+0.05 over best individual, +0.41 over GaC alone). The gain is modest in absolute terms but demonstrates that SAFE's mechanisms scale.
Full MMLU-redux: GaC achieves 82.73% vs. 84.54% for the best individual — a 1.81-point degradation. GaC + SAFE improves to 83.79% (-0.75 relative to best individual, but +1.06 over GaC alone). SAFE partially recovers the degradation but doesn't fully close the gap to the best individual model.
MATH500: GaC achieves 80.4% vs. 80.8% for the best individual. GaC + SAFE achieves 81.6% (+0.8 over best individual, +1.2 over GaC).

The sample size here is limited (only one model pair, two benchmarks), but the direction of effect is consistent: SAFE improves over standalone ensemble methods at larger scale. The paper notes in Appendix B that "SAFE does not always guarantee superior performance compared to the best-performing individual model" — the full MMLU-redux result is a case where SAFE slightly underperforms the best individual (83.79% vs. 84.54%) despite improving over GaC. This is attributed to "poorly performing models [that] can distort the ensemble distribution by elevating an incorrect token as the most confident." For the excluded subjects, the performance gap between models is so large that even selective ensembling cannot overcome the distortion from the weaker model's probability distribution.

Ablation Studies and Robustness Checks

Probability sharpening strategy (Tables 4 and 5). The paper tests whether probability sharpening is necessary and which variant works best, using the Internlm3 + Qwen2.5 model pair across MMLU-redux, MATH500, and GSM8K:

Sharpening vs. no sharpening (Table 4): For UniTE + SAFE, removing probability sharpening drops MATH500 accuracy from 77.4% to 76.6% (-0.8 points) and MMLU-redux from 77.81% to 77.53% (-0.28 points). For GaC + SAFE, the effect is smaller: MATH500 drops from 76.0% to 75.2% (-0.8 points), while MMLU-redux is unchanged (77.11% with or without sharpening). The finding that sharpening matters more for UniTE than GaC is consistent with their different aggregation mechanisms — UniTE's top-k alignment may produce smoother ensemble distributions that benefit more from consolidation.
Heuristic sharpening with varying λ thresholds (Table 5): With UniTE + SAFE, λ = 0.1 yields the best average (82.42% across three benchmarks). λ = 0.2 produces 82.31% (minor drop), and λ = 0.3 drops to 81.90% — worse than no sharpening (81.93%). This confirms the paper's claim that "setting the threshold too high reduces the number of tokens subject to sharpening, which in turn diminishes its effectiveness" (Section 4.4). At λ = 0.3, many tokens that would benefit from sharpening are excluded because their initial probability doesn't reach the threshold.
Geometric mean vs. heuristic sharpening (Table 5): The geometric mean variant achieves the strongest overall performance: 82.73% average across the three benchmarks, vs. 82.42% for heuristic λ = 0.1. On MMLU-redux specifically, geometric mean achieves 78.31% vs. 77.81% for the heuristic — a 0.5-point gain. The paper attributes this to the geometric mean's ability to "gather the probability mass dispersed across multiple tokens by individual models and concentrate it on the token with the highest consensus" without requiring a threshold hyperparameter. However, the paper notes that the heuristic method remains useful "considering that the arithmetic mean is widely used and often required" — some ensemble methods may be designed around arithmetic averaging and incompatible with geometric aggregation.

The non-obvious finding here is that both sharpening strategies work, and the geometric mean is slightly better but not dramatically so. The magnitude of improvement from sharpening (~1 point on MATH500, ~0.3 points on MMLU-redux) is meaningful but not the dominant source of SAFE's gains — the gating mechanism (OOV-like prevention + consensus-based skipping) accounts for the bulk of the improvement over uniform ensembling (17.8 points on MATH500). This suggests that sharpening addresses a real but secondary problem (probability diffusion across subword tokens), while the primary problem is the catastrophic OOV-like corruption that gating prevents.

Drafter sequence length n (Table 6, Figure 10). The paper ablates the drafter's lookahead window length using UniTE + SAFE on the Internlm3 + Qwen2.5 pair:

Accuracy (Table 6): n = 3 achieves 77.67% on MMLU-redux, 91.66% on GSM8K, and 90.19% on ARC-C. n = 5 (default) improves to 77.81%, 92.04%, and 90.78%. n = 8 achieves the best accuracy: 78.31%, 92.04%, and 90.78% — matching or exceeding n = 5. The accuracy advantage of n = 8 over n = 5 is limited to MMLU-redux (+0.5 points), with GSM8K and ARC-C unchanged.
Latency (Figure 10): Longer sequences reduce efficiency because "longer sequences force the drafter to regenerate tokens more often from the ensembled token" (Section 4.4). When ensembling is triggered at an early position in the lookahead, all subsequent tokens are discarded and must be regenerated. With n = 3, the average number of wasted tokens per ensemble intervention is small (at most 2). With n = 8, up to 7 tokens can be wasted per intervention. Figure 10 shows higher latency for n = 8 across all output lengths, with the gap widening for longer sequences (roughly 70 seconds vs. 60 seconds at 800 tokens for n = 8 vs. n = 5).

The paper's choice of n = 5 is a pragmatic compromise: accuracy is essentially saturated at n = 8 (the MMLU-redux gain is small and may be noise), but latency is noticeably worse. The finding that shorter sequences (n = 3) are less accurate validates the paper's argument (Section 3.1) that generating multiple tokens is necessary to capture tokenization boundaries — single-token lookahead would fail to detect mismatches that span multiple subword units.

The finding that n = 8 doesn't help (much) on accuracy is non-obvious. One might expect longer lookahead to provide better tokenization mismatch detection (seeing more context could identify more boundaries), but the results suggest that the tokenization boundaries that matter for OOV-like detection are captured within 5 tokens for the models and tasks tested. This may not generalize to languages or domains where tokenization differences produce longer boundary spans.

Ensembling models with similar tokenizers (Table 3, discussed above). This serves as an ablation on tokenization heterogeneity: the finding that SAFE's benefits are smaller but still present when tokenizers are nearly identical (>99% agreement) confirms that OOV-like prevention is the dominant mechanism, but the consensus-based gating and speculative architecture provide independent value.

KV cache management strategy (Figure 5c and 9c, discussed above). The dramatic latency difference between GaC with and without the paper's KV cache optimization (roughly 200s vs. 40s at 800 tokens on MATH500) demonstrates that stale cache management is a critical implementation detail for ensemble methods, not a minor optimization. Without proper cache pruning, each ensemble replacement triggers recomputation of subsequent tokens, making the cost proportional to the number of replacements times sequence length rather than just sequence length.

Critical Assessment

The experiments genuinely support the paper's central claim that SAFE's selective gating mechanism rescues probability-level ensembling from catastrophic degradation under chain-of-thought prompting. The evidence for this is decisive: UniTE degrades by 15.2 percentage points on MATH500 (Table 2), and SAFE not only recovers this loss but achieves a 2.8-point improvement over the best individual model. The mechanism is well-isolated: the same aggregation method (UniTE) is used with and without SAFE's gating, so the performance difference can be attributed specifically to the selective ensembling decision procedure. The ablation showing that similar-tokenizer models don't exhibit catastrophic degradation (Table 3) and therefore benefit less from SAFE further confirms that OOV-like token prevention — the mechanism SAFE's gating targets — is the causal factor.

However, the paper's broader claim — that SAFE enables stable and fast LLM ensembling for long-form generation — requires closer scrutiny along several dimensions.

The claim of "stability" is well-supported for the tested setting but has boundary conditions. Stability is demonstrated for the specific model pairs tested (three 7B models and two 32B models), on benchmarks dominated by mathematical and scientific reasoning (MATH500, GSM8K, MMLU-redux, ARC-C, BBH). These domains share a property that favors SAFE: they produce structured outputs with relatively constrained token sequences, which the paper itself shows leads to high inter-model agreement and low ensemble frequency (4.85% E/T on math vs. 15.24% on general-domain). The paper has not demonstrated stability on truly open-ended generation tasks — creative writing, dialogue, long-form explanation — where tokenization mismatches may be more frequent (due to greater linguistic variability) and where the consequences of even a single OOV-like token could be more severe (corrupting narrative coherence). The qualitative examples in Appendix K all come from math and science problems; there are no examples from narrative or conversational domains. This is a gap in the evidence: the paper claims to solve ensemble stability for "long-form generation" generally, but has tested only on reasoning tasks with structured outputs.

The efficiency claims are strong but the cost accounting is incomplete. The paper demonstrates that SAFE reduces ensemble frequency to <20% of tokens (and often <5%) and achieves latency comparable to single models (Figures 5, 9). However, several costs are not fully accounted for:

Wasted drafter computation from discarded tokens is not quantified. When ensembling is triggered at an early position in the lookahead window (e.g., position j in a window of n = 5), the drafter's forward passes for positions j+1 through i+n-1 produce tokens that are immediately discarded. The paper's n ablation (Table 6, Figure 10) shows that this cost is non-trivial — n = 8 has worse latency than n = 5 due to more regeneration — but the paper never reports what fraction of total drafter forward passes are "wasted" in this sense. For a method that claims efficiency as a primary contribution, this is a notable omission.
The cost of the verifiers' forward passes is not decomposed from the drafter's. Figures 5 and 9 show total end-to-end latency, which includes all models. It's unclear how much of the remaining latency gap between SAFE and single-model generation (roughly 5–10 seconds at 1000 tokens on MATH500) is due to verifier forward passes vs. ensemble operations vs. KV cache management overhead. This decomposition matters for practitioners deciding whether to deploy SAFE: if the verifier overhead is small, running 2 GPUs (one for drafter, one for verifier) is a reasonable cost for the accuracy gain; if it's large, the hardware cost may not be justified.
GPU memory requirements are not discussed. SAFE requires loading multiple models simultaneously (at minimum 2, up to 3 in the experiments), each on a separate GPU. The paper uses RTX 3090 GPUs (24GB VRAM) and FP16 precision, which fits 7B models comfortably. But at 32B scale (Appendix G), models require multiple GPUs each or more sophisticated sharding. The paper doesn't discuss memory overhead, model parallelism requirements, or whether the speculative architecture remains efficient when models must be distributed across many GPUs with inter-GPU communication.
Batched inference is not evaluated. All latency measurements are for single-sequence generation. In production settings where many queries are processed simultaneously, the advantage of SAFE's speculative architecture may change. With batching, all models (drafter and verifiers) could process multiple sequences in parallel, potentially making uniform ensembling (where all models generate autoregressively but in parallel across the batch) more competitive with SAFE's sequential drafter generation. The paper doesn't explore this dimension.

The generalizability of results beyond the tested model families is unproven. All experiments use models from a narrow range: 7B and 32B instruction-tuned models from Chinese AI labs (Internlm, Qwen, EXAONE, plus Llama-3.1 and Qwen2 for the similar-tokenizer ablation). These models share architectural similarities (all are dense transformer-based models with similar training paradigms). The paper's claim that the approach works for "widely used 7B-scale model combinations" (Section 5) overstates the evidence — three model families have been tested. Different model architectures (mixture-of-experts, state-space models), different scales (1B, 70B+), different tokenization paradigms (character-level, byte-level), or models from different training distributions (code-focused, multilingual) could exhibit different OOV-like token frequencies, different consensus patterns, or different susceptibility to distribution corruption. The paper acknowledges the limitation to "non-reasoning models" in Appendix B and flags extension to reasoning models (Yang et al., 2025; Guo et al., 2025) as future work, but doesn't address architectural or scale diversity.

The lack of statistical rigor is a genuine weakness. With 500 questions in MATH500 and varying sizes for other benchmarks, differences of 0.2–0.5 percentage points between methods are clearly within sampling noise. The paper reports many such comparisons (e.g., GaC + SAFE vs. UniTE + SAFE across different model pairs in Table 2) without any variance estimates. While the headline results (17.8-point improvements) are large enough to be robust, the finer-grained claims — like whether three-model ensembling outperforms two-model ensembling (84.59% vs. 84.20%) — cannot be assessed without confidence intervals. The two-model vs. three-model comparison is particularly important because it underpins the paper's practical recommendation to "restrict ensembling to the top-2 models" (Section 4.3), yet the evidence for this recommendation is a 0.39-percentage-point difference on a test set of unknown statistical power.

The hyperparameters are tuned on the test data. The paper selects n = 5 and λ = 0.1 based on ablation studies (Tables 5, 6) that are conducted on the same benchmarks used for evaluation. There is no held-out validation set or cross-validation split. This means the reported numbers may be optimistically biased — the hyperparameters are chosen to maximize performance on the test data, which inflates apparent accuracy relative to what would be achieved on truly unseen data. The paper would be strengthened by a held-out tuning set (e.g., a subset of MATH training data for hyperparameter selection) with clean separation from the test benchmarks.

Missing experiments that would strengthen the paper:

General-domain open-ended generation tasks (e.g., summarization, creative writing, dialogue) to test whether SAFE's stability extends beyond structured reasoning. The paper's own data (E/T of 15.24% on general-domain benchmarks vs. 4.85% on math) suggests that ensemble frequency is higher in less constrained domains, which could stress SAFE's gating differently.
A direct measurement of OOV-like token frequency with and without SAFE's gating. The paper claims OOV-like tokens are the mechanism of degradation but never reports how many OOV-like tokens are actually produced by UniTE vs. prevented by SAFE. This could be measured by instrumenting the generation process to count boundary mismatches using Equation (1). Without this measurement, the causal chain (SAFE prevents OOV-like tokens → improved stability) is inferred from correlated evidence (SAFE improves performance, and similar-tokenizer models don't degrade as much) rather than directly demonstrated.
Ablation on the drafter selection criterion. The paper selects the best-performing model as drafter. What happens if the worst-performing model is drafter instead? If SAFE's consensus mechanisms are robust, the ensemble should still recover (the verifiers would trigger ensembling more frequently to correct the weak drafter). If performance degrades significantly, it suggests SAFE relies on the drafter being strong — a dependency that limits its applicability when model rankings are unknown.
Comparison against a simple length penalty or repetition penalty applied to UniTE. One alternative explanation for UniTE's catastrophic degradation is that the ensemble process produces repetitive or degenerate text that could be mitigated by standard decoding interventions (repetition penalty, nucleus sampling, etc.) without SAFE's complexity. The paper doesn't test whether tuning UniTE's decoding parameters could achieve similar stability.
Ablation on the OOV-like check in isolation (without the consensus-based verification). This would quantify how much of SAFE's benefit comes from preventing corruption vs. from computational efficiency. The paper always applies both checks together, so their individual contributions are not separated.

6. Limitations and Trade-offs

Limitation 1: The Difficulty Estimation Cost Is Unaccounted for in the Efficiency Claims

The assumption or constraint. The entire compute-optimal framework depends on accurately estimating prompt difficulty before allocating the test-time compute budget. The paper's method for doing so — generating 2048 samples per question and computing pass@1 rates (oracle) or averaging PRM final-answer scores (predicted) — is extraordinarily expensive. The authors acknowledge this explicitly in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

Generating 2048 samples per question consumes more compute than the largest test-time budgets studied in the paper (256–512 generations). The 4× efficiency gains over best-of-N are computed after difficulty is known, without amortizing the cost of learning it. In a realistic deployment where difficulty must be estimated for each incoming prompt, the total cost would be difficulty estimation plus strategy execution, and the former could dominate the latter — potentially erasing the claimed 4× benefit entirely or even making the approach net-inefficient.

The consequence. The efficiency figures reported in Figures 4 and 8 represent an upper bound on achievable gains, not a realized deployment improvement. A practitioner cannot simply multiply their inference budget by 4 based on these numbers; they must first solve the difficulty estimation problem, which the paper does not provide a practical solution for. The paper's suggestion of future work on "pretraining or finetuning models to directly predict difficulty of a question" (Section 8) is an acknowledgment that the current method is not deployable, but no such model is developed or evaluated. Until a cheap difficulty estimator is demonstrated, the paper's headline efficiency claim (4× improvement) is a laboratory result, not an operational one.

What evidence exists in the paper. The paper's own difficulty estimation procedure is described in Section 3.2 and involves 2048 samples per question. The total inference budget consumed by this step across the 500-question MATH test set is 2048 × 500 = 1,024,000 generations — roughly 2000× the largest per-question budget studied (512 generations). The paper does not report the wall-clock time or FLOPs of this estimation step, nor does it include it in any efficiency calculation. The predicted-difficulty variant (using PRM scores instead of ground-truth labels) reduces the need for oracle access but does not reduce the computational cost — 2048 samples are still required per question. The difficulty estimation cost is the elephant in the room, and the paper's acknowledgment of it (Section 3.2) does not mitigate the fact that the central efficiency claims do not account for it.

Mitigation status. The paper flags this as future work (Section 8) but provides no partial solution, no analysis of how much the cost would need to be reduced for the approach to be net-beneficial, and no prototype of a cheap difficulty estimator. The authors frame the difficulty estimation cost as an "exploration-exploitation tradeoff" but do not quantify where the break-even point lies — how much difficulty estimation cost can be tolerated before the 4× gains are consumed? This is a conceptual gap, not just an implementation one.

Limitation 2: The Method Fails on Hard Problems — Test-Time Compute Cannot Substitute for Fundamental Capability Gaps

The assumption or constraint. The paper's approach operates on the premise that the base model already possesses the capability to generate correct solutions at some non-trivial rate — test-time compute amplifies existing capability but does not create it from nothing. The authors are transparent about this boundary:

"On the hardest questions (bin 5), no method makes meaningful progress — the base model simply lacks the capability to produce correct solutions regardless of how the budget is allocated" (Section 5.3, describing Figure 3 right)

"test-time compute amplifies existing capability but does not create it from nothing" (Section 7 takeaway)

This is not a bug in the algorithm but a fundamental ceiling: if the base model's pass@1 is near zero on a problem class, no amount of search, revision, or adaptive allocation can produce correct answers, because there are no correct solutions in the proposal distribution to find or refine.

The consequence. For any application domain where the base model struggles — genuinely novel reasoning, out-of-distribution problems, tasks requiring capabilities not acquired during pretraining — the compute-optimal framework provides zero benefit regardless of the inference budget. The paper's FLOPs-matched comparison makes this concrete: on the hardest difficulty bin (bin 5, Figure 9), accuracy remains near 0–5% for all methods and all budgets, while the 14× larger pretrained model (which may have acquired additional capabilities through scale) shows meaningful (though still low) performance. The practical implication is stark: for organizations whose problem distribution skews toward genuinely challenging tasks, investing in larger pretrained models is the only viable path — test-time compute scaling cannot compensate.

The boundary is also not sharp in practice. The paper bins questions into discrete difficulty quintiles (Section 3.2), but real-world problems span a continuous difficulty spectrum. A question that falls just on the "hard" side of the boundary — where the base model's pass@1 is slightly above zero but the PRM cannot reliably distinguish correct from incorrect — may receive the same "no meaningful progress" treatment as a truly impossible question, when a more nuanced approach might extract some value. The coarse quintile discretization means that problems near the boundary between "medium" (where the approach works) and "hard" (where it fails) are assigned strategies optimized for one regime that may be suboptimal for the other.

What evidence exists in the paper. Figure 3 (right) shows bin 5 accuracy hovering at 1–3% for all search methods and all budgets. Figure 7 (right) shows bin 5 accuracy at roughly 2–3% irrespective of the sequential-to-parallel ratio. Figure 9 shows the bin 5 scaling line essentially flat near 0–5% for both revisions and PRM search. In the FLOPs-matched comparison, hard questions show a −52.9% relative disadvantage from using test-time compute instead of the larger model at high inference-to-pretraining ratios (PRM search, R = 22). The evidence is comprehensive and consistent: the approach categorically fails on the hardest questions.

Mitigation status. The paper is candid about this limitation (Section 7 takeaway box) but offers no mitigation strategy for hard problems. The framing is that this is a property of the problem setting, not a flaw in the algorithm — which is intellectually honest but practically limiting. The paper does not explore intermediate approaches that might help on hard problems, such as: using the difficulty estimate to trigger retrieval-augmented generation or tool use for problems beyond the base model's capability; dynamically switching to a larger model (if available) when difficulty is estimated as high; or training the base model on the outputs of the compute-optimal process for easier problems in an iterative self-improvement loop that gradually expands the capability frontier (mentioned as future work in Section 8). Without such mitigation, a practitioner deploying this method must accept that some fraction of their problem distribution will see zero benefit.

Limitation 3: Search and Revisions Are Studied Independently — The Combined System Is Not Evaluated

The assumption or constraint. The paper studies two complementary mechanisms — PRM-guided search (Section 5) and iterative revisions (Section 6) — but never combines them into a single system. Search improves how the verifier selects among independently generated candidates; revisions improve the proposal distribution so that better candidates are generated in the first place. The paper's unified framework (Section 2) explicitly frames these as complementary axes, and the results show they have difficulty-dependent strengths: revisions excel on easy problems (local refinement), search excels on medium problems (global exploration). Yet the compute-optimal policy selects between search strategies within the search axis and between sequential-to-parallel ratios within the revision axis — it never routes a problem to either search or revisions, nor does it apply both sequentially (e.g., use revisions to generate better candidates, then apply beam search to select among them).

The authors acknowledge this gap explicitly in Section 8:

"we did not experiment with PRM tree-search techniques in combination with revisions"

The consequence. The reported results represent a lower bound on what a fully integrated system could achieve. The two mechanisms have complementary strengths — revisions could improve the proposal distribution that beam search explores, while the PRM could guide which revision chains to pursue or when to restart from scratch. Without evaluating the combination, the paper cannot answer the most natural follow-up question: "If I have both a PRM and a revision model available, should I use one, the other, or both?" The compute-optimal policy as presented is incomplete in this sense — it optimizes within each axis separately but not across axes.

This gap also means the paper cannot determine whether the difficulty-dependent patterns observed separately would persist in a combined system. For example, on medium-difficulty problems, beam search outperforms best-of-N (Figure 3, right), and a balanced sequential-to-parallel ratio outperforms pure parallel (Figure 7, right). Would beam search over revision model outputs — using the revision model as the proposal distribution within the search tree — outperform either alone? Or would the combination over-optimize the verifier signal and degrade, similar to the lookahead search failure? Without the experiment, the relative contribution of each mechanism to overall performance remains unknown.

What evidence exists in the paper. The paper's experimental design deliberately separates the two mechanisms: Section 5 (search) uses the base few-shot prompted model as the proposal distribution; Section 6 (revisions) uses the fine-tuned revision model with a separately trained ORM for answer selection. The compute-optimal strategies are optimized within each section independently (Figures 4 and 8). There is no experiment, ablation, or analysis where both mechanisms are active simultaneously. The paper's discussion of future work (Section 8) identifies combination as a natural next step, which is an implicit acknowledgment that the current work is incomplete along this dimension.

Mitigation status. Not addressed. The paper treats the separation as scope limitation rather than a methodological weakness. However, the central framing of the paper — a unified framework for test-time compute — is undermined by the fact that the two axes of the framework are never jointly optimized. A practitioner reading the paper would need to independently decide whether to deploy search, revisions, or both, without guidance from the experimental results. The difficulty-dependent optimal strategies reported in Figures 4, 7–8 are axis-specific and may not transfer to a combined setting.

Limitation 4: The Revision Model Has a 38% Correct-to-Incorrect Reversion Rate, and the Mitigation Is a Patch

The assumption or constraint. The revision model is trained exclusively on sequences where all in-context answers are incorrect, followed by a correct target (Section 6.1). This training data construction — necessary to teach the model to correct errors — has a predictable side effect: at test time, when the model encounters a correct answer in its own revision chain (produced during an earlier revision step), it has no training signal for what to do. The model may incorrectly "revise" that correct answer into an incorrect one. The paper reports:

"approximately 38% of correct answers produced during a revision chain get 'revised' back to incorrect answers in the subsequent step" (Section 6.1)

This reversion rate is a direct consequence of the training data distribution: the model learned that its job is to produce a correct answer after seeing incorrect ones, not to recognize when the current answer is already correct and preserve it.

The consequence. The revision chain is self-corrupting — even when it stumbles upon the correct answer at step k, there is a 38% chance that step k+1 will overwrite it with an incorrect answer. This means the revision model cannot be used as a monotonic improver; its output quality oscillates rather than converges. The paper's mitigation — using majority voting or verifier-based selection across the entire chain rather than taking the final revision — is a post-hoc patch that treats the revision chain as a set of independent candidates rather than a sequential improvement process. While this patch works (Figures 6–8 show net improvements over parallel sampling), it fundamentally changes the nature of the approach: instead of "each revision improves on the previous one," the effective mechanism becomes "generate diverse candidates in a chain and pick the best one." The revision model is being used as a proposal distribution diversifier, not as a genuine improver.

This reversion problem also creates a tension with the sequential-to-parallel ratio optimization. Longer sequential chains (higher sequential-to-parallel ratio) increase the chance that a correct answer will be generated at some point in the chain, but also increase the chance that it will subsequently be reverted. The optimal ratio found by the paper (Figures 7–8) reflects this tradeoff implicitly, but the paper does not analyze how much of the sequential benefit is from genuine improvement vs. simply generating more diverse candidates (which could be achieved more directly with parallel sampling at higher temperature).

What evidence exists in the paper. The 38% reversion rate is stated in Section 6.1. The paper does not provide a direct ablation measuring how much performance would improve if the reversion problem were eliminated (e.g., by training the model with "stop revising when correct" examples). The ReST^EM experiment in Appendix K (Figure 16) provides indirect evidence that the revision model's behavior is fragile: attempting to further optimize it with RL-style training caused performance to degrade substantially, with fully sequential revisions dropping to approximately 33.5% compared to 38.5% at the optimal ratio. The paper hypothesizes that "on-policy data collection amplified spurious correlations in the revision data," but the deeper issue is that the revision model's training objective (produce correct after incorrect) is misaligned with the desired test-time behavior (produce correct after any sequence and preserve correctness).

Mitigation status. The paper applies within-chain selection (majority voting or verifier-based) as a mitigation and demonstrates that it recovers most of the potential performance (Figures 6–8). However, this mitigation is a workaround, not a solution — it treats the symptom (correct answers being reverted) rather than the cause (training data distribution mismatch). A more principled solution would modify the training data to include sequences where the in-context answer is already correct and the target is to either repeat it or produce an end-of-revision token, teaching the model to recognize correctness and stop. The paper does not explore this. The ReST^EM failure (Appendix K) suggests that the revision training procedure is sensitive in ways that are not fully understood, making it risky for practitioners to modify without extensive experimentation.

Limitation 5: All Experiments Are on a Single Benchmark (MATH) with a Single Model Family (PaLM 2-S*), Limiting Generality

The assumption or constraint. The paper's entire experimental suite — search algorithms, revision models, difficulty estimation, FLOPs-matched comparisons — is conducted on the MATH benchmark (Hendrycks et al., 2021) using PaLM 2-S* as the base model (Section 4). The 500-question test set is split into five difficulty quintiles of roughly 100 questions each, and the compute-optimal policy is selected via two-fold cross-validation within each bin — meaning strategies are chosen based on approximately 50 questions per bin per fold.

The authors state that they "believe this model is representative of the capabilities of many contemporary LLMs" (Section 4), but this is an assertion, not a demonstrated fact. The MATH benchmark consists exclusively of competition-level math problems requiring multi-step symbolic reasoning. It is unclear whether the paper's central findings — the difficulty-dependent optimal strategy, the verifier over-optimization phenomenon, the sequential-to-parallel ratio tradeoff — would transfer to other reasoning domains (code generation, logical deduction, scientific QA), to tasks requiring factual knowledge rather than inference, or to other model families with different architectures, training distributions, or calibration properties.

The consequence. A practitioner deploying this approach on a different task (e.g., code generation with a code-specialized model, or medical QA with a domain-specific model) cannot assume that the optimal strategies reported in the paper will apply. The difficulty estimation procedure (Section 3.2) must be rerun from scratch on the target task and model — generating 2048 samples per question for a new test set — and the compute-optimal strategy must be re-derived. More problematically, the qualitative difficulty-dependent patterns (beam search hurting easy problems, revisions dominating on easy problems, balanced ratios on hard problems) may not hold in different domains. For example, code generation has a more constrained syntax than mathematical prose, which could change how the PRM's per-step scores distribute and how verifier over-optimization manifests.

The small effective sample size amplifies this concern. With 500 test questions split into 5 bins of ~100, then split further into 2 folds, the compute-optimal policy is selected based on roughly 50 questions per bin. This is a small sample for tuning a discrete strategy choice over a combinatorial space (multiple search algorithms, beam widths, lookahead depths, sequential-to-parallel ratios). The selected strategies may not be robust — a different random split of the 500 questions could yield different optimal strategies for some difficulty bins. The paper does not report confidence intervals on the compute-optimal scaling curves, so the reliability of the selected strategies cannot be assessed.

What evidence exists in the paper. Every experiment in Sections 5–7 uses MATH with PaLM 2-S* (or, for the FLOPs-matched comparison, a larger PaLM 2 variant). There are zero experiments on other benchmarks (GSM8K, MMLU, HumanEval, ARC, etc.), zero experiments with other model families (GPT, LLaMA, DeepSeek, etc.), and zero experiments on non-math reasoning tasks. The paper's abstract and introduction frame the contribution as a general framework for test-time compute, but the empirical support is entirely from a single domain-model combination.

Mitigation status. The paper acknowledges this limitation implicitly by describing PaLM 2-S* as "representative" (Section 4), but does not provide evidence for representativeness, does not test on other benchmarks, and does not discuss which findings are likely domain-specific vs. general. The future work section does not explicitly call for multi-benchmark or multi-model replication, though this is a natural extension. For a paper whose primary contribution is empirical (documenting scaling behavior and compute-optimal strategies), the single-benchmark single-model limitation is more consequential than it would be for a paper whose primary contribution is algorithmic — the empirical findings cannot be assumed to generalize without replication.

Limitation 6: Verifier Over-Optimization Is a Hard Ceiling — the Compute-Optimal Policy Mitigates but Does Not Solve It

The assumption or constraint. The paper identifies verifier over-optimization as the primary bottleneck preventing unbounded improvements from test-time compute. The evidence is concrete: beam search degrades easy-problem performance at high budgets (Figure 3, right), lookahead search — the most powerful optimizer — paradoxically performs worst overall (Figure 3, left), and qualitative examples show search producing degenerate outputs that score highly under the PRM but are incorrect (Appendix M). The paper's compute-optimal policy mitigates this by routing easy problems away from aggressive search (using best-of-N where the verifier over-optimization risk is highest) and reserving beam search for medium-difficulty problems where the PRM signal has room to provide genuine guidance.

However, the compute-optimal policy does not solve the underlying problem — it works around it. On medium-difficulty problems where beam search is deployed, over-optimization still limits the scaling ceiling. The beam search curves in Figure 3 (right, bins 3–4) show early saturation: accuracy plateaus well before the maximum budget of 256 generations, often at levels substantially below what a perfect verifier would enable. The paper does not propose any mechanism for improving verifier robustness or for detecting when search has entered the over-optimization regime.

The consequence. The compute-optimal framework is fundamentally bounded by verifier quality, and the paper provides no path for improving the verifier beyond the Monte Carlo rollout training procedure described in Appendix D. A practitioner deploying this method inherits the ceiling determined by their PRM's quality. If the PRM can only reliably distinguish correct from incorrect solutions up to a certain accuracy threshold (say, ~40% on MATH, as Figure 14 shows at 2048 samples), then the compute-optimal policy can at best achieve that threshold, regardless of how much inference compute is available. The 4× efficiency gains are real but are gains within the verifier's capability envelope, not gains that extend the envelope itself.

This limitation is particularly consequential for the FLOPs-matched comparison. The paper's finding that test-time compute can outperform a 14× larger model on easy-to-medium problems (Section 7) assumes the verifier is sufficiently good to extract that performance. If the verifier is weaker — e.g., trained with less data, or on a different model, or on a more challenging domain — the break-even point shifts, and test-time compute may no longer be preferable. The paper's verifier is trained with 16 Monte Carlo rollouts per step on 12,000 MATH training questions (Section 5.1), which is a substantial training budget in itself. A practitioner with fewer resources (less training data, less compute for rollout generation) would obtain a weaker PRM, and the compute-optimal scaling curves would shift downward. The paper provides no analysis of how verifier quality affects the compute-optimal strategy or the FLOPs-matched comparison.

What evidence exists in the paper. The over-optimization evidence is extensive: Figure 3 (left) shows lookahead search underperforming all methods; Figure 3 (right) shows beam search accuracy declining on bin 1 as budget increases; Figure 29 (Appendix M) shows qualitative examples of degenerate search outputs. The paper's treatment of over-optimization is primarily descriptive — it documents the phenomenon and designs the compute-optimal policy to route around it — rather than prescriptive — it does not propose verifier improvements. The ablation on "last" vs. "min" vs. "prod" PRM aggregation (Appendix E, Figure 13) explores how score aggregation affects final performance, but this is about how to use the existing verifier's scores, not about making the verifier more robust to optimization pressure.

Mitigation status. The paper acknowledges over-optimization as a limitation and suggests "improving verifier robustness is the key bottleneck for further scaling test-time compute" (Section 8), but treats this as future work rather than something the current method addresses. There is no ablation on PRM training data quantity or quality, no experiment with ensembled verifiers, no test of adversarial training to make the PRM robust to search-optimized inputs, and no analysis of how verifier quality affects the optimal strategy. For a paper that establishes verifier over-optimization as the limiting factor, the absence of even a preliminary attempt to improve verifier robustness is a notable gap. The compute-optimal policy is a clever way to live within the verifier's limitations, but a practitioner whose deployment requires accuracy beyond the verifier's reliability ceiling receives no guidance from this work.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper fundamentally reframes the research agenda for probability-level LLM ensembling from "how should we align vocabularies?" to "when should we ensemble at all?" This is not an incremental improvement to existing aggregation mechanisms — it is a diagnostic reframing that identifies the absence of gating as the structural failure mode that made prior methods unusable for long-form generation. The magnitude of this reframing is evidenced by a single number: applying SAFE's gating to UniTE — without changing UniTE's aggregation logic — recovers a 17.8-percentage-point improvement on MATH500 (Table 2, from 59.6% to 77.4%) and converts a method that was worse than any individual model into one that exceeds them all. This is not a marginal gain from a better alignment algorithm; it is the difference between a method being completely broken and being state-of-the-art, achieved solely by deciding where to apply it.

The conceptual contribution that enables this reframing is the formalization of OOV-like tokens as the specific mechanism of cascade failure. Prior work recognized that different tokenizers create alignment challenges, but treated this as a static mapping problem — how to project distributions across vocabulary spaces. The paper shows that the real problem is dynamic: an ensemble-selected token that represents a valid subword unit for one model can be an invalid conditioning point for another model whose tokenizer segments the same text at different boundaries. This distinction — token exists in vocabulary versus token boundary is a valid autoregressive state — had not been articulated before, and it explains the discrepancy between the positive short-form results in the literature and the catastrophic long-form degradation the paper documents. The field can no longer evaluate token-level ensemble methods on short-form benchmarks and assume the results will transfer; the OOV-like token problem is activated by sequence length, not by task domain per se.

The paper also reconciles an implicit contradiction in the speculative decoding literature. Prior work on speculative ensemble decoding (Fu et al., 2025) was limited to models with identical tokenizers — a restriction that made the technique irrelevant for the common case where available models use different BPE tokenizations. The paper shows that this restriction is not fundamental: by replacing the standard statistical acceptance criterion with a tokenization-compatibility and consensus-based criterion, speculative decoding can be extended to heterogeneous tokenizers. This opens the door to efficient ensembling of any set of available models, regardless of tokenizer design, which is the practical scenario for most practitioners who don't control model training.

The empirical finding that effective ensembling requires intervention at only a small fraction of token positions — 3.82% on MATH500, 5.16% on GSM8K, at most 18.60% on ARC-Challenge (Table 2) — has implications beyond SAFE. It suggests that the ensemble problem is sparse: most tokens don't benefit from aggregation because models already agree or the drafter is already correct. This means future research on token-level ensembling should prioritize gating precision (identifying which tokens need intervention) over aggregation quality (how to combine distributions at those tokens). Improving aggregation at every token, as UniTE and DEEPEN do, is solving a problem that only exists at ~5–15% of positions — and at the other 85–95%, it's actively harmful because each unnecessary ensemble operation carries a non-zero risk of introducing an OOV-like token. The paper's demonstration that three-model ensembling barely outperforms two-model ensembling (84.59% vs. 84.20% average, Table 2) reinforces this: adding models increases diversity but also increases the probability of tokenization mismatch and makes consensus harder to achieve, so the net benefit diminishes rapidly. Research efforts should shift from "how to ensemble more models" to "how to ensemble the right two models at the right tokens."

Finally, the paper establishes speculative verification as a viable architectural template for efficient multi-model inference. The drafter-verifier separation — where one model generates autoregressively and the rest verify in parallel forward passes — is not tied to any specific ensemble method or gating criterion. It is a general pattern for amortizing the cost of multi-model computation across tokens. Future methods that involve multiple models interacting during generation (routing, debate, iterative refinement) can adopt this template to avoid the prohibitive cost of running all models autoregressively in lockstep. The paper's KV cache management strategy (Appendix D, Figure 8) is a practical contribution in its own right: it solves the cache consistency problem that had led prior ensemble work to avoid KV caching entirely, and Figure 5(c) shows that this alone accounts for a 4–5× latency reduction.

Follow-Up Research This Work Enables

Direct measurement of OOV-like token frequency to quantify the failure mechanism. The paper claims that OOV-like tokens are the causal mechanism behind UniTE's degradation, but never directly measures how many OOV-like tokens are produced by UniTE versus prevented by SAFE. A follow-up study should instrument the generation process to count boundary mismatches using Equation (1) for each ensemble method, reporting how OOV-like token frequency correlates with downstream accuracy degradation. The hypothesis is that UniTE produces OOV-like tokens at a rate proportional to sequence length × tokenizer divergence (quantified by the Oxford 5000 agreement rates in Figure 4), and that this rate predicts the accuracy drop in Table 1. A strong study would measure this across multiple model pairs with varying tokenizer agreement (from the >99% similar-tokenizer pair to the 46.8% heterogeneous pair) and show that the correlation holds. A negative result — where OOV-like token frequency does not predict degradation — would implicate some other mechanism and challenge the paper's central diagnostic.

Training a lightweight difficulty estimator to make the compute-optimal framework deployable (for the compute-optimal test-time scaling paper). (Note: this direction applies to the compute-optimal test-time scaling paper analyzed in prior sections, not SAFE. For SAFE-specific extensions, see the following directions.)

Evaluating SAFE on open-ended generation tasks to test the boundary of the stability claim. The paper demonstrates stability on structured reasoning tasks (math, multiple-choice QA, logical deduction) where the paper's own data shows ensemble frequency is low (4.85% on math vs. 15.24% on general-domain, Table 2). A critical stress test would evaluate SAFE on truly open-ended generation — creative writing (e.g., WritingPrompts), long-form summarization (e.g., GovReport), or dialogue (e.g., MultiWOZ) — where linguistic variability is higher and tokenization mismatches may be both more frequent and more damaging. The research question is: does SAFE's OOV-like token prevention mechanism remain effective when the base rate of boundary mismatches is much higher? The hypothesis from the paper's own data is that ensemble frequency (E/T) would increase in these domains because models disagree more often, and each ensemble operation carries a risk of introducing OOV-like tokens at subsequent positions. If SAFE's stability degrades on open-ended tasks, it would refine our understanding: SAFE works when inter-model agreement is high and token sequences are constrained, but may need additional mechanisms (e.g., larger lookahead windows, more conservative OOV-like detection) for unconstrained generation. A strong study would report accuracy and E/T on at least one open-ended benchmark, along with qualitative examples of any residual degradation.

Ablation on the drafter selection criterion to determine whether SAFE requires a strong drafter. The paper selects the best-performing model as drafter (Section 4.2) and argues that this ensures baseline quality equals the best individual model when ensembling is skipped. But what if model rankings are unknown, or if the available models have unknown relative quality on a new task? A follow-up should systematically vary the drafter: use the worst-performing model as drafter, use a random model, or use the model with the tokenizer most similar to the other models (to minimize OOV-like token probability). The research question is whether SAFE's consensus-based gating is robust enough to recover from a weak drafter — i.e., will the verifiers trigger ensembling more frequently to correct the drafter's errors, or will the increased ensemble frequency introduce more OOV-like tokens and degrade stability? The experiment would use the same model pairs and benchmarks as Table 2, replacing the default drafter selection with each alternative. The finding would determine whether SAFE's plug-and-play claim (which implies no strong dependency on drafter identity) holds, or whether careful drafter selection is a hidden prerequisite.

Combining SAFE's gating with retrieval-augmented generation or tool use at ensemble points. SAFE identifies token positions where models disagree — these are precisely the positions where the ensemble is uncertain about what to generate. A natural extension is to trigger an external knowledge retrieval or tool call at these positions rather than (or in addition to) probability-level ensembling. For example, on a math problem, if the verifiers disagree on a specific numerical token, the system could call a calculator to resolve the computation rather than relying on the ensemble distribution. The research question: does augmenting ensemble points with external verification improve accuracy beyond what probability-level ensembling alone achieves? A concrete experiment would modify the Ensemble step (Algorithm 1, line 8) to first check whether the disputed token is computable (numerical, factual, or code-related) and, if so, replace it with a tool-verified answer rather than the ensemble argmax. The MATH500 and GSM8K benchmarks are ideal testbeds because they contain computation-heavy steps where disagreement likely stems from arithmetic errors rather than semantic ambiguity. The expected finding is that tool augmentation at ensemble points would improve accuracy further, particularly on harder problems where model disagreement reflects genuine uncertainty rather than tokenizer artifacts. A negative result — where tool augmentation doesn't help or even degrades performance — would suggest that model disagreement is primarily driven by tokenization issues that tool use cannot resolve, refining our understanding of why models disagree at the positions SAFE identifies.

Extending SAFE to reasoning models (DeepSeek-R1, Qwen3) to test whether chain-of-thought length amplifies or dampens the OOV-like token problem. Reasoning models generate substantially longer outputs than standard instruction-tuned models — often thousands of tokens of internal "thinking" before producing a final answer. The paper acknowledges this as future work (Appendix B) but does not explore it. The research question is whether longer reasoning chains increase OOV-like token exposure proportionally (because there are more token positions where boundaries can mismatch) or whether the structured nature of reasoning traces (which may follow predictable patterns) dampens the effect. A concrete experiment would apply UniTE and UniTE + SAFE to DeepSeek-R1-7B and Qwen3-8B (or similar reasoning-model pairs with heterogeneous tokenizers) on MATH500 and GSM8K, measuring the same accuracy and E/T metrics as Table 2. The hypothesis is that reasoning models would show even larger UniTE degradation (due to longer sequences → more accumulated OOV-like corruption) and correspondingly larger SAFE recovery, because SAFE's OOV-like prevention becomes more valuable as sequence length increases. A negative result — where reasoning models show less degradation from UniTE — would be equally interesting, as it would suggest that reasoning models' training (which often involves reinforcement learning on structured reasoning traces) produces more tokenization-robust internal representations.

Quantifying the wasted computation from discarded drafter tokens and optimizing the lookahead length dynamically. The paper acknowledges that when ensembling is triggered at an early position in the lookahead window, all subsequent tokens are discarded and regenerated (Section 3.4), but never quantifies this waste as a fraction of total drafter forward passes. A follow-up should measure the regeneration overhead for varying n and ensemble frequencies, reporting the ratio of discarded to accepted drafter tokens. The research question is whether a dynamic lookahead length — shorter when disagreement is expected (e.g., after recent ensemble interventions), longer when models have been agreeing — could reduce waste without sacrificing accuracy. A concrete implementation would track the recent ensemble trigger rate and adapt n accordingly: if the last k chunks triggered zero ensemble operations, increase n to amortize verification cost over more tokens; if the last chunk triggered ensembling early, reduce n for the next chunk to minimize wasted regeneration. The evaluation metric is tokens-per-second (throughput) at equal accuracy to the fixed-n = 5 baseline on the benchmarks from Table 2. The expected finding is that dynamic scheduling would improve throughput by 10–20% on general-domain tasks (where E/T is higher and regeneration waste matters more) with minimal accuracy impact.

Practical Applications and Downstream Use Cases

On-premise model serving with heterogeneous model pools. Organizations that have deployed multiple LLMs for different purposes — e.g., an internally fine-tuned Qwen model for code generation and an Internlm model for documentation — often have these models available on separate GPU nodes. SAFE enables combining them during inference without requiring them to share a tokenizer or be served from the same infrastructure. The practical benefit is latency: Figure 5(a) shows that UniTE + SAFE achieves latency within ~10 seconds of single-model generation even at 1000 output tokens on MATH500, compared to ~80 seconds for UniTE alone. For a code generation deployment where users expect sub-second response times, the difference between 40 seconds (SAFE) and 80 seconds (uniform ensembling) determines whether ensemble-based quality improvements are viable. The E/T numbers in Table 2 provide guidance on cost: on math-like structured tasks, fewer than 5% of tokens trigger ensemble operations, meaning the verifier GPU is idle most of the time and could be shared across multiple drafter instances, further amortizing hardware cost.

Cost-effective batch inference for benchmark evaluation and data generation. When organizations evaluate models on benchmarks or generate synthetic training data, they often run multiple models on the same set of prompts and select the best response. SAFE's drafter-verifier architecture makes it possible to combine models during generation rather than post-hoc, improving the quality of the selected responses. For a batch of 1000 MATH500 problems, running Internlm3-8B and Qwen2.5-7B with SAFE (UniTE + SAFE, Table 2) would yield 77.4% accuracy versus 74.8% for the best individual model — a 2.6-percentage-point improvement — while generating responses at near-single-model speed (Figure 5). For data generation pipelines that use model outputs as training data for distillation or self-improvement, this accuracy improvement on each generated sample compounds across the dataset. The paper's finding that math benchmarks require <5% ensemble frequency means the verifier model's GPU is lightly loaded and could be time-shared across multiple data generation jobs. The 4.85% vs. 15.24% ensemble frequency difference between math and general-domain tasks (Section 4.3) provides a concrete metric for capacity planning: general-domain data generation will require roughly 3× more verifier GPU time per thousand tokens than math data generation.

Routing ensemble decisions based on domain to optimize the accuracy-latency tradeoff. The paper's finding that ensemble frequency varies dramatically by domain — 4.85% on math vs. 15.24% on general-domain (Table 2) — enables a simple deployment heuristic: route queries based on estimated domain and adjust the ensemble configuration accordingly. For math and code queries (where inter-model agreement is high and ensemble benefit is concentrated at few tokens), use SAFE with the default configuration and expect near-single-model latency. For open-ended reasoning and creative queries (where disagreement is more frequent), either increase the verifier allocation (dedicate more GPU resources to handle higher E/T) or fall back to a single best model if latency constraints are tight. The E/T metric provides a directly actionable capacity planning number: if a service receives 1000 queries per hour with an average 500-token response, a 5% E/T means approximately 25,000 ensemble operations per hour; a 15% E/T means 75,000. The hardware provisioning difference is roughly 3× in verifier GPU time. SAFE makes this tradeoff visible and quantifiable, whereas uniform ensembling would require provisioning for 100% ensemble operations (500,000 per hour in this example) regardless of domain, making it economically infeasible for all but the lowest-throughput applications.

(Conditional) When to Prefer This Method

SAFE is positioned as a gating mechanism that wraps existing probability-level ensemble methods (GaC, UniTE, DEEPEN, etc.) rather than as a standalone ensemble method. The paper does not articulate a decision rule for choosing SAFE over alternative approaches (such as response-level ensemble methods like MoA, cascade methods like FrugalGPT, or routing methods like Co-LLM), because these operate at fundamentally different granularities — token-level versus response-level — and address different problems. SAFE's contribution is specifically making token-level probability ensembling viable for long-form generation, not competing with response-level methods. A practitioner choosing SAFE is implicitly choosing token-level ensembling over response-level alternatives, but the paper provides no comparative evaluation that would support a decision rule between these paradigms.