The Illusion of Certainty: Decoupling Capability and Calibration in On-Policy Distillation

🎯 Pitch

On-policy distillation drives LLMs into severe overconfidence—accuracy improves, but models report near-certainty even when wrong. This 'Scaling Law of Miscalibration’ is shown to arise from an information mismatch where teacher signals rely on privileged context the student cannot access at deployment. The proposed CaOPD framework fixes the gap by distilling a student’s own empirical success rate as the confidence target, yielding state-of-the-art calibration without sacrificing task performance.

1. Executive Summary

This paper identifies and theoretically formalizes a Scaling Law of Miscalibration—the finding that on-policy distillation (OPD) methods like SDPO and SDFT systematically trap language models in severe overconfidence, driving mean confidence toward 1.0 even as accuracy remains far lower. Through experiments on Qwen3-8B and Olmo-3-7B-Instruct across Science Q&A and Tool Use domains, the authors trace this pathology to an information asymmetry: teachers condition on privileged context (e.g., ground-truth solutions or verifier feedback) while deployed models lack it, producing entropy collapse and optimism bias. To address this, they propose Calibration-Aware On-Policy Distillation (CaOPD), which decouples behavior imitation from confidence supervision by replacing teacher-confidence targets with a student-grounded empirical success rate estimated from model rollouts and distilled through the same self-distillation pipeline. CaOPD reduces Expected Calibration Error by up to 0.25 points (e.g., from 0.486 to 0.266 on Qwen3-8B Science Q&A under SDFT) while maintaining or exceeding standard OPD accuracy and restoring discriminative ranking via Strict Pairwise Ranking—all in a single forward pass at deployment—establishing that calibrated confidence can be amortized from training-time rollouts without the capability tax of RL-based reward shaping, but only when the base model possesses sufficient in-context learning capacity to benefit from privileged teacher context.

2. Context and Motivation

The Core Problem: Capability Distillation Destroys Calibration

The fundamental question this paper tackles is deceptively simple: how is it that advanced post-training methods like on-policy distillation (OPD) systematically make LLMs more capable while simultaneously making them less trustworthy? This tension matters enormously because OPD—in which a model learns by distilling its own outputs generated under privileged conditions—has become the dominant paradigm for improving LLM reasoning. Methods like SDPO (Hübotter et al., 2026) and SDFT (Shenfeld et al., 2026) have demonstrated remarkable success at bootstrapping a model's own capabilities without external teacher models or human demonstrations. Yet this paper reveals a dark side: the same distillation pipeline that transfers reasoning skill also catastrophically corrupts the model's ability to honestly report its uncertainty.

This gap is critical for several practical and theoretical reasons the authors develop throughout the paper:

Trustworthy deployment in high-stakes domains. As the paper's epigraph from Terence Tao signals, AI systems deployed as scientific co-pilots or autonomous agents must accurately signal when they are uncertain. A model that confidently hallucinates API parameters (as shown in the paper's qualitative case study, Appendix E.2, where SDFT outputs Confidence: 1.0 on an incorrect tool call) will silently execute flawed actions, compounding errors in agentic pipelines. The paper's framing in Section 1 explicitly connects this to the broader shift from LLMs as text generators to LLMs as decision-making agents: "For AI to be a trustworthy collaborator in rigorous domains, it must possess reliable uncertainty signaling."

The Scaling Law of Miscalibration as a structural phenomenon. The paper's Figure 1 is devastating: it shows that across 12 leading proprietary and open-weight models—from GPT-5.2 to Claude-Opus-4.6 to DeepSeek-V3.1—virtually all operate in an "Overconfidence Zone" where mean confidence far exceeds accuracy. The paper terms this the Scaling Law of Miscalibration: scaling up model capability (moving along the accuracy axis) does not resolve blind optimism. As the authors observe in Section 5.5, this is not merely an artifact of undertrained small models—it rigidly entrenches across scale. This transforms calibration from a "nice-to-have" into a fundamental limitation of the dominant post-training paradigm itself.

The capability-calibration trade-off. Prior attempts to fix overconfidence have overwhelmingly relied on RL-based reward shaping—penalizing Brier scores or proper scoring rules during PPO training. But as the paper documents in Section 5.3 and Table 3, these methods suffer from a severe capability tax: models become overly conservative, sacrificing fundamental reasoning accuracy to avoid calibration penalties. RLCR (Damani et al., 2025) drops Qwen3-8B Science Q&A accuracy from 74.5% (GRPO) to 65.8%, while CAR (Xuan et al., 2026) drops it further to 61.6%. This creates a grim trade-off: you can have either capable models or calibrated models, but not both. The paper's core motivation is to break this trade-off entirely.

Where Prior Approaches Fall Short

The paper identifies specific limitations in existing work along three axes:

1. On-policy distillation is blind to calibration. The authors are careful to acknowledge that OPD methods are genuine breakthroughs for capability improvement. SDPO and SDFT enable models to bootstrap reasoning without external supervision by conditioning the teacher side of the model on privileged context—expert demonstrations, verifier feedback, ground-truth solutions—while the student side generates under deployment conditions. However, as the paper's theoretical analysis (Section 3, Propositions 1–3) formally demonstrates, this architecture contains a structural blind spot. The per-token reverse KL loss (Equation 2) operates uniformly over all token positions, including the confidence segment c where the model verbalizes its certainty. Because the teacher—armed with privileged context z—produces sharply concentrated, near-deterministic distributions, the student is forced to imitate this unjustified certainty. The paper puts this precisely: "the model learns the teacher's certainty without possessing the teacher's evidence" (Section 3.1).

2. RL-based calibration introduces optimization instability and a capability tax. The prevailing approach to taming overconfidence has been to inject calibration penalties—Brier scores, proper scoring rules—directly into the RL reward function. The paper acknowledges these methods (RLCR, CAR, Rewarding Doubt) can suppress absolute confidence, but identifies fundamental flaws:

Optimization instability: adding scoring-rule penalties to the RL objective creates tension between the capability-maximizing signal and the calibration-enforcing signal. The paper's comparison in Table 3 shows that RLCR and CAR, while achieving better calibration metrics (lower ECE) than SDPO, suffer dramatic accuracy drops—a direct manifestation of this tension.
The "overly conservative" failure mode: models learn to systematically avoid expressing high confidence even when justified, degrading their usefulness as decision-making agents. The paper cites Askell et al. (2021) on this point: penalizing overconfident mistakes can produce models that are calibrated but useless.

The paper's key insight is that these failures are not incidental but fundamental: fighting the RL optimizer with competing reward signals inherently creates a multi-objective trade-off that cannot be perfectly resolved.

3. Test-time calibration methods incur prohibitive deployment costs. A separate line of work addresses uncertainty through test-time computation—generating multiple samples and measuring self-consistency (SelfCheckGPT, SAC3) or semantic entropy (Kuhn et al., 2023b). The paper explicitly acknowledges that these sampling-based approaches are the "gold standard" for measuring epistemic uncertainty (Appendix B.5). However, they impose an O(K) inference cost at deployment, multiplying latency by the number of samples. The paper emphasizes (Section 6, "Test-Time Amortization") that this is prohibitive for real-time or agentic deployment where latency constraints are tight. The existing solution space therefore presents a dilemma: either accept poor calibration from single-pass models, pay a capability tax for RL-calibrated models, or pay a latency tax for test-time sampling.

4. No formal understanding of why OPD induces overconfidence. Prior work had documented overconfidence in LLMs (Guo et al., 2017b; Kadavath et al., 2022a; Tian et al., 2023) and proposed various mitigation strategies, but none had provided a theoretical account of the mechanism by which OPD specifically induces miscalibration. The paper identifies this as a critical gap: without understanding the causal mechanism, interventions risk being ad hoc patches rather than principled solutions.

How This Paper Positions Itself

The paper frames its contribution through a specific diagnosis: capability distillation and confidence calibration are fundamentally misaligned under information asymmetry. This is not merely an empirical observation but a formal result derived in Section 3 through three propositions:

Proposition 1 (Information Gap): When privileged context Z carries mutual information about correctness R beyond the input X, the teacher's confidence μ_T(X, Z) is not X-measurable—it cannot be recovered from deployment-time information alone. The best the student can do is predict its own marginal success rate μ(X), leaving an irreducible projection error. Standard OPD forces the student to chase the inaccessible μ_T(X, Z) rather than learning this optimal projection.
Proposition 2 (Entropy Collapse): The teacher's privileged distribution has strictly lower entropy than the deployment-time distribution, forcing the student to artificially sharpen its logits to match the teacher's deterministic-like behavior. The reverse KL—being mode-seeking—heavily penalizes the student for expressing natural uncertainty.
Proposition 3 (Optimism Bias): OPD training contexts are systematically success-filtered (ground-truth demonstrations, verified correct rollouts), creating an upward-biased surrogate for the true deployment-time success probability. The student learns only the "voice of a winner."

These propositions collectively establish that OPD-induced miscalibration is not a bug but a mathematical consequence of the privileged conditioning architecture. This theoretical grounding distinguishes the paper from prior empirical work on calibration—it provides not just evidence of the problem but a causal diagnosis that directly motivates the solution.

The paper's proposed solution, CaOPD, is positioned not as an alternative to OPD but as a minimal intervention within the OPD pipeline. Rather than adding external reward signals or post-hoc calibration layers, CaOPD operates via target replacement: it preserves the teacher's reasoning signal for capability cloning while overwriting the confidence target with the student's empirical success rate μ̂(x). As the paper emphasizes in Section 4.2, this leaves the reverse-KL machinery "entirely unchanged"—the model learns what to answer from the teacher's context and how certain it should be from its own rollouts, decoupling the two objectives that OPD had inadvertently entangled.

The paper also explicitly positions CaOPD relative to test-time amortization. Unlike SelfCheckGPT or SAC3, which compute μ̂(x) at deployment with O(K) cost, CaOPD amortizes this computation into model weights during training. The paper's training dynamics (Figure 2, right panel) show that the per-step time is nearly indistinguishable from standard SDPO—the rollout cost is absorbed during data generation, and deployment remains a single O(1) forward pass. This "compute at training, benefit at deployment" philosophy is the paper's practical through-line: it accepts the gold-standard nature of empirical consistency signals but shifts when they are computed, making calibrated single-pass inference viable.

Finally, the paper positions its evaluation to address a specific gap in the calibration literature: the distinction between absolute calibration (do confidence scores match accuracy rates?) and discriminative ranking (does the model assign strictly higher confidence to correct answers than incorrect ones?). The paper introduces Strict Pairwise Ranking (SPR) in Section 5.1 and Appendix C.2 specifically to expose confidence saturation—the pathology where a model achieves reasonable ECE by uniformly predicting high confidence but completely loses the ability to distinguish successes from failures. Standard AUROC awards partial credit for ties, masking this collapse; SPR penalizes ties to zero, revealing that standard OPD models (e.g., SDFT on Qwen3-8B Tool Use, SPR = 0.085 in Table 2) have effectively lost all discriminative resolution. CaOPD's ability to restore SPR (to 0.555 in the same setting) is positioned as evidence that it learns a genuine uncertainty representation rather than merely shifting confidence downward.

In summary, the paper positions itself at the intersection of three literatures—on-policy distillation, confidence calibration, and test-time uncertainty estimation—and argues that current approaches fail because they either ignore calibration entirely (OPD), fight the optimizer (RL-based methods), or defer the cost to deployment (test-time sampling). CaOPD's target replacement is presented as a synthesis: it preserves the capability gains of OPD while grounding confidence supervision in the student's actual execution outcomes, all without modifying the training objective or increasing deployment latency.

3. Technical Approach

3.1 Reader Orientation

This paper develops Calibration-Aware On-Policy Distillation (CaOPD) — a training-time intervention within the standard on-policy distillation (OPD) pipeline that forces language models to report honest, empirically-grounded confidence scores rather than blindly imitating their teacher's near-certain predictions. The core problem is that standard OPD methods like SDPO and SDFT inadvertently entangle two objectives: transferring reasoning capability (good) and transferring the teacher's unjustified overconfidence (catastrophic). CaOPD solves this by decoupling capability supervision from confidence supervision — it preserves the teacher's high-quality reasoning trajectory for cloning what to answer, but replaces the teacher's saturated confidence token (typically near 1.0) with a student-grounded empirical success rate computed from the model's own rollouts, then distills this revised target through the same reverse-KL pipeline with no architectural changes.

3.2 Big-Picture Architecture (Diagram in Words)

The CaOPD system has five major components, integrated into one training loop:

Component 1: Base Language Model (π_θ) — the pretrained LLM (e.g., Qwen3-8B, Olmo-3-7B-Instruct) that serves as both student policy (generating under deployment conditions) and teacher policy (generating under privileged conditions). The same model plays both roles in the self-distillation architecture.

Component 2: Privileged Context Generator — constructs the teacher's privileged context z by injecting ground-truth demonstrations (SDFT) or verified successful rollouts (SDPO) into the prompt, creating an "open-book" setting where the teacher has access to answers the deployed student will not see.

Component 3: Empirical Confidence Estimator — samples K independent rollouts from the student policy, evaluates each with an objective task verifier R(x, a) that returns 0/1 correctness, and computes the empirical success rate μ̂(x) = (1/K) Σ R(x, a_k). This is the student-grounded, unbiased estimate of true deployment-time success probability.

Component 4: Target Replacement Module — performs the core decoupling: takes the student's generated reasoning trajectory a, discards its original (overconfident) confidence segment c, and constructs a revised completion ỹ = (a, μ̂(x)) where the confidence tokens encode the empirical success rate. Simultaneously overwrites the confidence value in the teacher's privileged context z with the same μ̂(x), producing revised context z̃.

Component 5: Reverse-KL Distillation Objective — computes the per-token reverse KL divergence between the student's distribution π_θ(· | ỹ_{<t}, x) and the teacher's distribution π_θ(· | ỹ_{<t}, x, z̃) on the revised completion. This naturally decomposes into capability cloning at reasoning positions (where the teacher's privileged signal is preserved) and confidence calibration at confidence positions (where both sides target μ̂(x)).

Information flow in one training step: input query x enters the system → the student generates a trajectory y = (a, c) and K separate rollouts (a_k, c_k) → the verifier scores the rollouts to produce μ̂(x) → Target Replacement constructs ỹ and z̃ by overwriting confidence values → both student and teacher forward-pass on ỹ under their respective conditioning → per-token reverse KL is computed and gradients flow back through θ.

3.3 Roadmap for the Deep Dive

First, the formal problem setup (Section 2.1–2.3 of the paper, which we established in prior sections): the self-distillation architecture, the definition of μ(x) as the idealized calibration target, and the standard OPD objective that entangles capability with confidence. This provides the mathematical vocabulary needed for what follows.
Second, the theoretical diagnosis: Propositions 1–3 formalize the three mechanisms (Information Gap, Entropy Collapse, Optimism Bias) by which privileged conditioning necessarily induces miscalibration. Understanding these mechanisms is essential because CaOPD's design directly counteracts each one.
Third, the construction of the student-grounded target μ̂(x): the Monte Carlo estimation procedure, the verifier dependency, the handling of target non-stationarity through online estimation, and the justification for why this empirical frequency is the correct calibration target.
Fourth, the target replacement operation and the decoupled objective: exactly how μ̂(x) is integrated into the OPD pipeline, why the autoregressive structure provides causal decoupling between reasoning and confidence positions, and what the loss function computes at each token position.
Fifth, the algorithmic procedure in full: the training loop, the EMA teacher tracking mechanism, the dynamic rollout generation via vLLM, and the critical design choices (rollout budget K, handling of format adherence, the choice of reverse KL over alternatives).

3.4 Detailed, Sentence-Based Technical Breakdown

This is a theoretical diagnosis + algorithmic intervention paper. The core idea is that standard OPD forces the student to imitate an inaccessible, artificially sharpened, and systematically optimistic confidence distribution, and that decoupling confidence supervision from capability supervision — by replacing teacher-confidence targets with student-grounded empirical success rates — resolves all three mechanisms simultaneously without modifying the optimization procedure.

The Self-Distillation Architecture and the Entangled Objective

Before examining CaOPD, we must understand the OPD pipeline it modifies. The paper formalizes this in Section 2 using a self-distillation architecture where a single model π_θ plays both student and teacher roles, differentiated only by their conditioning context.

Generation format. Each complete model output y = (y_1, …, y_T) is partitioned into two contiguous segments:

Reasoning trajectory a = (y_1, …, y_{T_a}): the chain-of-thought, final answer, or tool-use action sequence.
Confidence segment c = (y_{T_a+1}, …, y_T): the token subsequence encoding a verbalized confidence statement. The paper uses a standardized format where the model appends a string like Confidence: 0.85 after its reasoning. The function val(c) parses this substring to extract a scalar ∈ [0, 1].

This explicit partitioning is central to CaOPD's target replacement: it creates a clear boundary between what the model should answer (reasoning tokens) and how certain it should report being (confidence tokens), enabling selective intervention.

Student and teacher conditioning.

Student policy π_θ(· | x): generates y conditioned only on the input query x, simulating deployment conditions where the model has no access to ground-truth answers or environment feedback.
Teacher policy π_θ(· | x, z): the same model but additionally conditioned on a privileged context z ∼ Z(x). Depending on the OPD paradigm, z may be:
- SDFT: an offline ground-truth demonstration injected into the prompt as an in-context example. The full teacher prompt (shown in Appendix B.3.1) prepends the user query with a gold-standard solution trajectory, including reasoning, correct answer, and Confidence: 1.0.
- SDPO: a dynamically selected successful rollout from the current training batch. The system samples multiple trajectories, uses a verifier to identify one that produced a correct answer, and prepends it as a "Correct solution" demonstration. This creates an "open-book" setting where the teacher sees a verified correct path before generating.

The shared structural property — and the root of the miscalibration problem — is that the teacher is conditioned on strictly richer evidence than the deployed model will ever observe. The teacher "sees the answer" before speaking; the student must reason from scratch.

The standard OPD objective. Following SDFT and SDPO, on-policy self-distillation minimizes the per-token reverse KL divergence between the student and the privileged teacher:

$L_{\text{OPD}}(θ) = \mathbb{E}_{x, z ∼ Z(x)} \ \mathbb{E}_{y ∼ π_θ(·|x)} \left[ \sum_{t=1}^{T} D_{KL}\left( π_θ(· | y_{<t}, x) \ \middle\|\ π_θ(· | y_{<t}, x, z) \right) \right]$

where x is the input query, z ∼ Z(x) is the sampled privileged context, y = (y_1, …, y_T) is a complete trajectory sampled from the student policy, y_{<t} is the prefix of tokens before position t, π_θ(· | y_{<t}, x) is the student's next-token distribution at position t (conditioned only on the input and preceding tokens), and π_θ(· | y_{<t}, x, z) is the teacher's next-token distribution at position t (additionally conditioned on privileged context z).

What it computes: for each training example, the student first generates a complete trajectory y. Then, for every token position t in that trajectory, both the student and teacher compute a probability distribution over the vocabulary given the prefix y_{<t}. The reverse KL divergence D_{KL}(p \| q) measures how much the student's distribution p diverges from the teacher's distribution q. Summing over all positions yields a single scalar loss that drives the student to make its per-token predictions match the teacher's. Since the student generated the trajectory y in the first place (hence "on-policy"), the loss operates on tokens the student would naturally produce, not on an external dataset.

Why this form — and why it's problematic: the reverse KL is mode-seeking — it heavily penalizes the student for placing probability mass on tokens where the teacher assigns low probability, but only lightly penalizes the student for ignoring tokens the teacher considers possible. This property is desirable for capability transfer: it forces the student to concentrate on the teacher's high-quality reasoning paths. However, the loss operates over all token positions, including the confidence segment c. Because the teacher — armed with privileged context — produces an extremely sharp, near-deterministic distribution at confidence positions (essentially P(Confidence: 1.0) ≈ 1), the reverse KL forces the student to similarly collapse its distribution, driving verbalized confidence to extreme values regardless of actual uncertainty. This is the theoretical blind spot the paper identifies: the loss entangles capability cloning (desirable) with confidence imitation (catastrophic) in a single, undifferentiated objective.

Theoretical Diagnosis: Three Mechanisms of OPD-Induced Miscalibration

The paper's theoretical contribution (Section 3) formalizes why the OPD objective necessarily produces overconfidence. The three propositions identify distinct causal mechanisms, each of which CaOPD's design directly counteracts. We present them in detail because they provide the intellectual justification for every design choice in CaOPD.

Proposition 1: Information Gap and Non-Identifiability

Setup. Define μ(X) = P(R = 1 | X) as the student's true deployment-time success probability — the probability that a trajectory sampled from the student policy π_θ(· | x) will be correct, marginalized over the student's own generation distribution. Define μ_T(X, Z) = P(R = 1 | X, Z) as the teacher-conditioned success probability — the probability of correctness when the model additionally conditions on privileged context Z. The assumption is that Z carries information about correctness beyond what X provides: the conditional mutual information I(R; Z | X) > 0.

Statement. Under this assumption, μ_T(X, Z) is not X-measurable — there exists no deterministic function g(X) that equals μ_T(X, Z) almost surely. Furthermore, the optimal X-measurable predictor of μ_T(X, Z) under squared error is exactly μ(X), the student's marginal success rate, with a strictly positive irreducible projection error:

$\min_g \mathbb{E}\left[ (μ_T(X, Z) - g(X))^2 \right] = \mathbb{E}_X\left[ \text{Var}(μ_T(X, Z) \mid X) \right] > 0$

where g(X) is any function of the input alone, μ_T(X, Z) is the teacher's context-dependent success probability, and μ(X) = \mathbb{E}[R | X] is the student's marginal success probability.

What it computes: the minimum possible mean squared error when trying to predict the teacher's internal confidence from only the information available at deployment time. The optimal predictor is the conditional expectation \mathbb{E}[μ_T(X, Z) | X], which by the law of total expectation simplifies to \mathbb{E}[\mathbb{E}[R | X, Z] | X] = \mathbb{E}[R | X] = μ(X). The irreducible error \text{Var}(μ_T(X, Z) | X) is the variance in teacher confidence across different privileged contexts for the same input — information the student fundamentally cannot recover from X alone.

Why this form: the squared error formulation directly connects to the standard definition of calibration (predicted confidence should equal expected accuracy). If the teacher's confidence depends on information the student cannot observe, the student cannot achieve perfect calibration by imitating the teacher — the target itself is unidentifiable. The best the student can do is predict its own marginal success rate μ(X), but standard OPD doesn't optimize for this projection; it minimizes KL divergence pointwise on each trajectory, forcing the student to memorize the inaccessible μ_T(X, Z).

Practical implication: this formalizes why confidence scores generated by OPD models are structurally unreliable. The student is trained to output the confidence of a model that had access to the answer — a fundamentally different quantity from the student's own confidence. CaOPD's target replacement resolves this by using μ̂(x) as the supervision signal, which is an unbiased estimator of μ(X) (the optimal projection) rather than μ_T(X, Z) (the inaccessible target).

Proposition 2: Entropy Collapse via Privileged Conditioning

Setup. Let A represent the generated reasoning trajectory (the sequence of tokens before the confidence segment), X the user input, and Z the privileged context. The conditional mutual information I(A; Z | X) quantifies how much Z reduces uncertainty about the correct reasoning path beyond what X provides. By definition, if privileged contexts are informative (e.g., showing the correct answer), then I(A; Z | X) > 0.

Statement. The expected entropy of the teacher's trajectory distribution is strictly lower than the conditional entropy of trajectories given only the accessible input:

$\mathbb{E}_{X,Z}\left[ H(π_θ(A | X, Z)) \right] < \mathbb{E}_X\left[ H(A | X) \right]$

where H(·) is Shannon entropy, π_θ(A | X, Z) is the teacher's distribution over reasoning trajectories given both input and privileged context, and H(A | X) is the conditional entropy of trajectories given only the input.

What it computes: the left-hand side is the average uncertainty of the teacher's generation when it can "see the answer" — this is very low because the privileged context Z resolves most of the ambiguity about what the correct reasoning should be. The right-hand side is the average uncertainty when the model only sees the question — this is naturally higher because the model must reason from first principles, explore multiple possible approaches, and contend with genuine ambiguity. The strict inequality follows from the chain rule of entropy: H(A | X) = H(A | X, Z) + I(A; Z | X), and since I(A; Z | X) > 0, the teacher's entropy is lower.

Why this form — and why the reverse KL matters critically: the entropy inequality itself simply states that privileged information reduces uncertainty — that's obvious. The crucial insight is what the reverse KL does with this. The reverse KL D_{KL}(p \| q) is mode-seeking: when q (the teacher) is sharply concentrated on a narrow set of tokens, minimizing D_{KL}(p \| q) forces p (the student) to similarly collapse onto that same set, even if p would naturally be more diffuse. At every token position, the student is penalized for placing probability mass anywhere outside the teacher's sharp mode. Over the course of training, this drives the student's internal logits to become artificially magnified — the model learns to output the same near-deterministic token probabilities as the teacher, but without possessing the mutual information I(A; Z | X) that justified that certainty. When the model subsequently verbalizes its confidence (in segment c), it translates this artificially low-entropy internal state into an unjustified near-absolute confidence score like 0.99.

Practical implication: CaOPD breaks this mechanism by intervening at the confidence tokens specifically. By replacing the teacher's overconfident confidence target with μ̂(x), the reverse KL at confidence positions no longer forces entropy collapse — it forces the student to match a realistic, empirically-grounded confidence distribution. The reasoning tokens remain supervised by the teacher's privileged signal (preserving capability), but the confidence tokens are supervised by the student's own execution outcomes (restoring calibration).

Proposition 3: Selection Bias Induces Optimism

Setup. In practice, OPD training contexts Z are not randomly sampled — they are systematically filtered to be helpful. SDFT injects ground-truth demonstrations that are correct by construction. SDPO selects successful rollouts verified by an environment. Let D_helpful denote this success-filtered distribution of privileged contexts. The assumption is that conditioning on Z ∼ D_helpful strictly improves expected correctness over the student's marginal capability on a set of inputs with positive probability.

Statement. The implicit target distilled into the student is an upward-biased surrogate for the true deployment-time success probability:

$\mathbb{E}_{X, Z ∼ D_{\text{helpful}}}\left[ μ_T(X, Z) - μ(X) \right] > 0$

where μ_T(X, Z) is the teacher's success probability under helpful context, μ(X) is the student's marginal success probability without context, and the expectation is over the joint distribution of inputs and success-filtered contexts.

What it computes: the average gap between the confidence that would be appropriate for a teacher with access to answers and the confidence that is appropriate for a student reasoning without such access. The proof (Appendix A.3) shows this is strictly positive because: (1) for any X, \mathbb{E}_{Z ∼ D_{helpful}}[μ_T(X, Z) | X] ≥ μ(X) (the helpful context never makes the teacher less accurate than the student), and (2) on a set of inputs with positive measure, the inequality is strict (the helpful context genuinely helps). Therefore the expectation is strictly positive.

Why this form — and what "optimism" really means: this is not merely stating that the teacher is more capable than the student. It's identifying a training distribution asymmetry: the student is exclusively trained on trajectories where the teacher, armed with Z, was successful and confident. The student never sees examples of the teacher saying "I'm not sure, confidence: 0.5" because such trajectories are filtered out — they're not in D_helpful. Consequently, the student learns a linguistic prior: the declarative, certain tone of a successful derivation. At test time, when the student generates an incorrect answer (as it inevitably will on some fraction of prompts), it lacks the mechanism to express doubt — it has only been trained to output the "voice of a winner," and blindly applies this optimistic tone regardless of actual correctness. This explains the massive Overconfidence Gaps (OCG) in Table 1: for Qwen3-8B on Science Q&A, base OCG is +58.7%, and SDFT pushes it to +48.1% — the model's mean confidence exceeds its accuracy by nearly 50 percentage points.

Practical implication: CaOPD's target replacement directly counters this selection bias. Instead of training the student to always output Confidence: 1.0 (the optimistic tone), it trains the student to output μ̂(x) — a value that can be 0.3, 0.6, 0.9, or anything in between, depending on the student's actual success rate on that prompt. The model learns the full spectrum of confidence expression because the supervision signal covers the full range.

Constructing the Student-Grounded Target (`μ̂(x)`)

The theoretical diagnosis identifies μ(x) = \mathbb{E}_{a ∼ π_θ(·|x)}[R(x, a)] as the correct calibration target — the student's own expected success probability marginalized over its generation distribution. CaOPD must estimate this quantity during training without relying on the privileged context z (which would reintroduce the information asymmetry). Section 4.1 describes the estimation procedure.

Monte Carlo estimation. For a given input x, CaOPD samples K independent trajectories from the student policy — not from the teacher, and not using privileged context:

$(a_k, c_k) ∼ π_θ(· | x), \quad k = 1, …, K$

Each trajectory includes both a reasoning segment a_k and a confidence segment c_k, but only the reasoning segment is evaluated for correctness. The objective task verifier R(x, a_k) ∈ \{0, 1\} returns 1 if the reasoning trajectory yields a correct answer and 0 otherwise. The verifier is domain-specific: for Science Q&A, it checks whether the selected multiple-choice option matches the ground-truth answer; for Tool Use, it validates whether the API call and parameters match the expected action.

The empirical success rate is the simple average:

$\hat{μ}(x) = \frac{1}{K} \sum_{k=1}^{K} R(x, a_k)$

where \hat{μ}(x) ∈ [0, 1] is the empirical estimate of the student's true success probability on query x.

What it computes: the fraction of K independent solution attempts that the model gets right. This is the maximum-likelihood estimate of a Bernoulli success probability, and it is unbiased: \mathbb{E}[\hat{μ}(x)] = μ(x) for any K ≥ 1. Larger K reduces variance (the estimate becomes more precise) but increases computational cost. The paper ablates K ∈ \{1, 2, 4, 8, 16, 32\} (Appendix D.3, Figure 6) and finds that K = 8 is the "compute-efficient sweet spot" — small K (1–4) yields heavily quantized targets with only 2–5 distinct values, which traps the model back in overconfidence because it cannot learn graded uncertainty representations; K ≥ 8 provides 9 distinct levels (increments of 0.125) with sufficient granularity while keeping the sampling cost manageable.

Why this form — and why it's the correct target. The key property is that μ̂(x) is student-grounded: it measures the model's actual competence without any privileged information. This directly addresses all three theoretical mechanisms:

Information Gap (Proposition 1): μ̂(x) estimates μ(X), the optimal X-measurable predictor, rather than trying to recover the inaccessible μ_T(X, Z). The estimation error decreases with K but is zero-mean — the target is unbiased.
Entropy Collapse (Proposition 2): μ̂(x) takes values in the full [0, 1] range with realistic granularity, so the confidence supervision signal does not force the student's logits to collapse to an extreme point. The model learns to output intermediate confidence values that reflect genuine epistemic uncertainty.
Optimism Bias (Proposition 3): μ̂(x) includes failures: when the student's rollouts are mostly incorrect, μ̂(x) will be low (e.g., 0.125 or 0.25). The model sees these low targets during training and learns to express doubt, breaking the "always confident" linguistic prior that standard OPD induces.

Dependency on objective verifiers — and the modular alternative. The paper acknowledges (Appendix B.6) that μ̂(x) requires an objective task verifier R(x, a) — a function that can judge whether a trajectory is correct. This is not an additional assumption introduced by CaOPD; it is a prerequisite of the underlying OPD paradigms. SDPO requires verifiers to identify successful rollouts for constructing z; SDFT requires ground-truth labels for constructing demonstrations. CaOPD reuses this existing infrastructure. For domains where no objective verifier is available, the paper proposes Teacher-Anchored Self-Consistency (TA-SC):

$\hat{μ}_{\text{SC}}(x) = \frac{1}{K} \sum_{k=1}^{K} \mathbb{I}\left( \text{Equiv}(a_k, a_{\text{ref}}) \right)$

where a_ref ∼ π_θ(· | x, z) is a reference trajectory generated under privileged conditioning, and Equiv(·, ·) is a semantic equivalence function (e.g., LLM-as-a-Judge). This substitutes agreement with the privileged teacher for ground-truth correctness. The paper empirically validates TA-SC (Table 4) and finds it closely tracks the objective verifier version (ECE: 0.271 vs. 0.266 on Science Q&A), with minor degradations due to teacher fallibility and spurious disagreements in open-ended generation.

Online Estimation and Handling Target Non-Stationarity

A critical practical consideration is that the model π_θ changes during training — its capabilities improve, so an empirical target computed from a stale checkpoint would become misaligned with the model's true competence. The paper addresses this through an online, on-policy estimation loop (Appendix B.4) with four components:

1. EMA Teacher Tracking. The teacher distribution uses an Exponential Moving Average (EMA) copy of the model weights to stabilize the reverse KL target:

$θ_{\text{ema}} ← (1 - α) θ_{\text{ema}} + α θ$

where α = 0.05 is the update rate. The teacher is then π_{θ_ema}(· | ỹ_{<t}, x, z̃). This prevents the teacher distribution from changing too rapidly between gradient steps, which would destabilize the distillation objective. The student (π_θ) is the current model being optimized; the teacher is a smoothed version used only for computing the target distribution.

2. Dynamic Rollout Generation. Periodically during training (the paper does not specify the exact frequency, but implies it occurs at each major iteration), the latest model weights are synchronized to a high-throughput inference engine (vLLM, Kwon et al., 2023). The engine generates a fresh batch of K rollouts for each prompt using the current model parameters:

$(a_k, c_k) ∼ π_{θ_{\text{current}}}(· | x)$

This ensures the rollouts reflect the model's current capability, not its capability from several gradient steps ago.

3. Online Target Computation. These freshly generated rollouts are immediately evaluated by the verifier R(x, a_k) to compute μ̂(x). The target is therefore computed from the current model's actual performance, tracking capability improvements in real time.

4. Target Replacement and Distillation. The freshly computed μ̂(x) constructs ỹ and z̃, which are used for the next gradient optimization steps. As the model improves, μ̂(x) naturally increases, and the confidence supervision target scales accordingly — there is no need for manual schedule adjustment.

Why this online approach matters. If μ̂(x) were computed offline from a fixed checkpoint and then used throughout training, two failure modes would arise:

Stale overconfidence: as the model improves beyond its initial checkpoint, the offline μ̂(x) would understate its true capability, training the model to be underconfident.
Stale underconfidence: if the model's capabilities degrade (e.g., due to catastrophic forgetting during continual learning), the offline μ̂(x) would overstate its true capability, training the model to be overconfident again.

The online approach avoids both by continually re-anchoring the target to the model's current execution outcomes.

Target Replacement and the Decoupled Objective

With the empirical target μ̂(x) computed, CaOPD integrates it into the OPD pipeline through a target replacement operation (Section 4.2, Algorithm 1) that modifies both the completion and the teacher context before computing the loss.

Step 1: Revise the completion. The student generates a trajectory y = (a, c) under deployment conditioning, where a is the reasoning segment and c is the original (overconfident) confidence segment. CaOPD discards c and constructs a revised completion:

$\tilde{y} = (a, \hat{μ}(x))$

Specifically, the confidence tokens in the completion are overwritten with the string representation of the empirical success rate — for example, Confidence: 0.75 if μ̂(x) = 0.75. The reasoning tokens a are preserved unchanged. This creates a joint target where the capability signal (what to answer) comes from the student's own generation, and the confidence signal (how certain to be) comes from the empirical estimate.

Step 2: Revise the teacher context. The privileged context z ∼ Z(x) is constructed as usual (ground-truth demonstration for SDFT, verified successful rollout for SDPO), but the confidence value within z — typically near 1.0 — is overwritten with the same μ̂(x), producing the revised context z̃. The full teacher prompts with CaOPD's overwrites are shown in Appendix B.3: for SDFT (Listing 5), the line Confidence: 1.0 in the golden demonstration is replaced; for SDPO (Listing 7), the confidence in the "Correct solution" is replaced. The reasoning content of z (the correct answer, the successful API call) is left unchanged to preserve capability cloning.

Step 3: Compute the decoupled reverse KL. Both the student and teacher perform a forward pass on the revised completion ỹ, producing token-level distributions. The CaOPD objective is:

$L_{\text{CaOPD}}(θ) = \mathbb{E}_{x, \tilde{z}} \ \mathbb{E}_{\tilde{y}} \left[ \sum_{t \in \mathcal{I}_a} D_{KL}\left( π_θ(· | \tilde{y}_{<t}, x) \ \middle\|\ π_θ(· | \tilde{y}_{<t}, x, \tilde{z}) \right) + \sum_{t \in \mathcal{I}_c} D_{KL}\left( π_θ(· | \tilde{y}_{<t}, x) \ \middle\|\ π_θ(· | \tilde{y}_{<t}, x, \tilde{z}) \right) \right]$

where \mathcal{I}_a = \{1, …, T_a\} is the set of token positions in the reasoning segment, \mathcal{I}_c = \{T_a+1, …, T\} is the set of token positions in the confidence segment, ỹ_{<t} is the prefix of the revised completion before position t, and z̃ is the revised privileged context with confidence overwritten by μ̂(x).

What it computes — the decoupling. This objective is structurally identical to standard OPD (Equation 2) — same reverse KL, same per-token summation, same teacher-student conditioning pattern. The decoupling emerges from what the targets contain, not from modifying the loss function:

Capability Cloning (reasoning positions t ∈ \mathcal{I}_a): at these positions, the prefix ỹ_{<t} contains only reasoning tokens (the confidence replacement μ̂(x) has not yet appeared in the autoregressive sequence). The teacher's privileged context z̃ still contains the correct answer, ground-truth demonstration, or verified successful reasoning. Therefore, the teacher's distribution at these positions is identical to standard OPD — it reflects the "open-book" certainty of a model that sees the answer. The reverse KL forces the student to match this distribution, transferring reasoning capability just as standard OPD does.
Confidence Calibration (confidence positions t ∈ \mathcal{I}_c): at these positions, the prefix ỹ_{<t} now includes the reasoning tokens a plus the beginning of the confidence segment. The teacher's context z̃ contains μ̂(x) rather than 1.0, so the teacher's distribution at these positions is not sharply peaked at 1.0 but instead reflects μ̂(x). The student is trained to output the empirically-grounded success rate. The reverse KL at these positions directly resolves entropy collapse (Proposition 2) by providing a realistic, non-degenerate target, and selection bias (Proposition 3) by including low-confidence examples from the full rollout distribution.

Why this form — and why target replacement rather than reward shaping. The paper emphasizes that CaOPD "operates entirely within the standard distillation pipeline via target replacement, requiring no reward modification or additional optimization stages" (Section 4.2). This is contrasted with RL-based calibration (RLCR, CAR) which adds scoring-rule penalties to the RL reward function. The key advantages of the target replacement approach:

No multi-objective optimization: RL methods must balance a capability-maximizing reward signal against a calibration-enforcing penalty, creating an inherent trade-off that often degrades accuracy (Table 3: RLCR drops Qwen3-8B Science Q&A accuracy to 65.8% vs. SDPO's 80.6%). CaOPD has a single loss function — the calibration signal is integrated into the target, not added as a competing term.
No optimization instability: Reverse KL distillation is generally more stable than PPO, which requires careful tuning of clipping parameters, value function learning, and reward normalization. CaOPD inherits the stability of the underlying OPD method (SDFT or SDPO).
Gradient decoupling: The autoregressive structure naturally separates the gradients: changes to confidence targets only affect the final token positions; they do not backpropagate into the reasoning positions. This is empirically confirmed by the training dynamics (Figure 2, left): CaOPD's accuracy trajectory "perfectly shadows" SDPO, indicating that the low-confidence targets at t ∈ \mathcal{I}_c do not corrupt the logit distributions at t ∈ \mathcal{I}_a.

Causal decoupling through autoregressive structure. Appendix B.7 provides an extended discussion of why verbalized humility does not "bleed into" internal reasoning confidence. Because the Transformer computes each token's distribution from the prefix, the confidence replacement at positions \mathcal{I}_c appears after the reasoning tokens in the sequence. The reverse KL at reasoning positions t ∈ \mathcal{I}_a is computed on prefixes ỹ_{<t} that contain only reasoning tokens — the confidence replacement has not yet been seen. Therefore, the teacher's distribution at reasoning positions is determined by the privileged reasoning content in z̃, not by the overwritten confidence value. The gradients that push confidence tokens toward μ̂(x) are isolated to the final few sequence positions and do not propagate backward through the autoregressive computation graph to affect the probability distributions at earlier reasoning positions.

The Complete Training Algorithm

Algorithm 1 in the paper specifies the full CaOPD training loop. We walk through it step by step, elaborating on the implementation details provided in Appendix B and C.

Inputs. The algorithm requires:

A dataset D of input queries x (the paper uses Science Q&A and ToolAlpaca).
A pretrained language model π_θ (Qwen3-8B, Olmo-3-7B-Instruct, or the broader Qwen3 family for scaling analysis).
An objective task verifier R(x, a) (domain-specific, described in Appendix C.1).
A rollout budget K (default K = 8, ablated in Appendix D.3).
Hyperparameters for the distillation loop (learning rate {1e-6, 1e-5, 2e-5}, batch size 32, AdamW optimizer, weight decay 0.01, gradient clip norm 1.0, EMA update rate α = 0.05; full details in Table 6).

Per-iteration procedure. For each input x in the training batch:

Student-grounded confidence estimation (lines 3–6):
- Sample K independent rollouts (a_k, c_k) ∼ π_θ(· | x) from the current model (student mode, no privileged context). Under SDPO, these rollouts are already generated by the base training loop to identify successful trajectories — CaOPD reuses them with zero additional sampling cost. Under SDFT, they represent an additional K samples per prompt.
- Evaluate each rollout with the verifier: r_k = R(x, a_k) ∈ \{0, 1\}.
- Compute the empirical success rate: μ̂(x) = (1/K) Σ r_k.
- This computation is parallelized via vLLM for high throughput.
Target replacement (lines 9–12):
- Sample a single student trajectory y = (a, c) ∼ π_θ(· | x) — this is the trajectory that will be used for distillation.
- Construct the revised completion: parse the confidence segment c, extract its position in the token sequence, and overwrite those tokens with the string representation of μ̂(x). For example, if the original generation ended with Confidence: 0.95 and μ̂(x) = 0.75, the revised completion ends with Confidence: 0.75. All reasoning tokens are preserved as-is.
- Construct the privileged context z ∼ Z(x) using the domain-appropriate procedure (gold demonstration for SDFT, verified successful rollout for SDPO; full prompt templates in Appendix B.3).
- Revise the teacher context: in z, locate the confidence value (typically 1.0) and overwrite it with μ̂(x), producing z̃. The reasoning content of z — the correct answer, the demonstration trajectory, the verified API call — is left unchanged.
Distillation (lines 15–16):
- Perform a forward pass: the student computes π_θ(· | ỹ_{<t}, x) for all token positions t in ỹ.
- Perform a second forward pass: the teacher (using EMA weights θ_ema) computes π_{θ_ema}(· | ỹ_{<t}, x, z̃) for all token positions.
- Compute the per-token reverse KL divergence (Equation 7) by summing the KL at each position.
- Backpropagate through the student parameters θ (the teacher is not updated by the loss — it is updated only through the EMA).
- Apply the optimizer step (AdamW) with gradient clipping.

EMA tracking. The teacher's EMA weights θ_ema are updated at every gradient step with α = 0.05, providing a slowly-moving average of the student's weights. This stabilizes the teacher distribution, preventing rapid oscillations that would make the reverse KL target noisy. When computing the loss, the student uses the current θ and the teacher uses θ_ema.

Rollout and loss sampling are separate. A subtle but important implementation detail: the rollouts used to compute μ̂(x) (line 4) and the trajectory used for distillation (line 9) are independently sampled. This ensures that μ̂(x) is an unbiased estimate of the model's general performance on x rather than being correlated with the specific distillation trajectory. If the same trajectory were used for both, μ̂(x) would be deterministically 1 (the trajectory verified itself and passed) or 0 (it failed), creating a degenerate binary target that would not support graded confidence learning.

Per-step time. Figure 2 (right panel) shows that CaOPD's wall-clock time per optimization step is "nearly indistinguishable" from standard SDPO. This is because:

Under SDPO, the rollouts for μ̂(x) are already generated by the base algorithm — CaOPD adds only a lightweight verifier evaluation.
Under SDFT, the additional K rollouts are generated in parallel via vLLM and do not bottleneck the gradient computation.
The forward/backward passes are identical in FLOPs to standard OPD — the target replacement changes the data, not the computation graph.

Design Choice: Why Verbalized Confidence?

Appendix B.7 addresses a fundamental question: why target explicitly verbalized confidence tokens rather than internal logit-based uncertainty? The paper positions this as a pragmatic choice driven by deployment constraints:

The three uncertainty paradigms (Table 5).

Logit-based methods (e.g., perplexity, token entropy): these leverage the model's internal probability distribution over tokens. They are inaccessible for black-box proprietary APIs (which don't expose logits) and require complex length-normalization for long-horizon reasoning — a 100-token correct answer and a 1000-token correct answer may have very different aggregate token probabilities even if both are equally certain. They also cannot distinguish between uncertainty about what to say (token-level) and uncertainty about whether the answer is correct (semantic-level).
Sampling-based methods (e.g., SelfCheckGPT, semantic entropy, self-consistency): these examine consistency across multiple generations and are robust to many failure modes, but incur O(K) inference latency at deployment, making them prohibitive for real-time applications.
Verbalized confidence (e.g., Kadavath et al., 2022b; Lin et al., 2022b; Tian et al., 2023): the model explicitly states its confidence in natural language like Confidence: 0.85. This requires only a single forward pass, works with black-box APIs, and directly expresses the semantic-level uncertainty the user cares about. Its fatal flaw — and the problem CaOPD solves — is that standard post-training makes it catastrophically overconfident.

CaOPD targets verbalized confidence because it is the "most deployment-friendly paradigm" — solving its overconfidence problem unlocks the most scalable form of uncertainty signaling.

Why not just regularize logits? The paper argues that directly regularizing internal logit entropy (e.g., by adding an entropy bonus to the loss) would affect reasoning quality: suppressed logit entropy at reasoning positions is actually desirable for OPD — it reflects the teacher's focused, correct reasoning. The autoregressive decoupling in CaOPD provides cleaner separation: confidence regularization is applied only at the designated confidence token positions, leaving the model free to maintain sharp, focused logits for reasoning while expressing calibrated uncertainty in the verbalized confidence segment.

Design Choice: Why Empirical Rollouts Rather Than Intrinsic Uncertainty?

Appendix B.5 justifies why CaOPD uses μ̂(x) — an execution-based, behavioral measure — rather than trying to extract the model's intrinsic uncertainty from its internal representations or token probabilities.

Statistical unbiasedness. μ̂(x) is an unbiased Monte Carlo estimator of μ(x), the expected success probability under the model's own generation distribution. This satisfies the mathematical definition of calibration: predicted confidence should equal expected accuracy on samples drawn from the same distribution. Any intrinsic measure (e.g., the probability the model assigns to its own answer under its output distribution) may be systematically biased — for instance, models often assign high probability to incorrect but plausible-sounding answers.

Connection to the hallucination literature. The paper explicitly connects this to established findings: self-consistency (Wang et al., 2022), SelfCheckGPT (Manakul et al., 2023), and SAC3 (Zhang et al., 2023) have all demonstrated that behavioral consistency across multiple rollouts is the most reliable signal of epistemic uncertainty in LLMs. When a model knows something, it generates consistent answers; when it's uncertain, its outputs diverge. CaOPD adopts this gold-standard signal but shifts when it's computed: during training (amortized into weights) rather than at deployment (paid per query).

Why not use the model's own confidence expression directly? If we could simply ask the model "how confident are you?" and trust its answer, CaOPD would be unnecessary. The entire paper is a demonstration that standard OPD-trained models cannot be trusted to self-report accurately — they overwhelmingly report near-1.0 confidence regardless of actual correctness. The empirical rollout target breaks this circularity: it measures what the model actually does, not what it says about what it does.

The Reverse KL Choice and Mode-Seeking Behavior

The paper uses reverse KL divergence D_{KL}(p \| q) where p is the student and q is the teacher, rather than forward KL D_{KL}(q \| p) or other divergences. This choice is inherited from SDPO and SDFT, and the paper's target replacement leaves it unchanged — but understanding its properties clarifies why CaOPD works.

Mode-seeking vs. mean-seeking. Reverse KL is mode-seeking: D_{KL}(p \| q) heavily penalizes p (student) for placing probability mass where q (teacher) has near-zero probability, but only lightly penalizes p for ignoring modes of q. This is desirable for capability transfer because it forces the student to concentrate on high-quality reasoning paths (the teacher's modes) rather than trying to cover the full distribution of possible outputs (which includes many incorrect paths).

Why target replacement works with reverse KL. At reasoning positions, the teacher's distribution is sharply peaked on correct reasoning tokens — reverse KL forces the student to match this peak, preserving capability cloning. At confidence positions, standard OPD has the teacher peaked at ~1.0, and reverse KL forces the student there too — producing overconfidence. CaOPD changes the teacher's target at confidence positions from 1.0 to μ̂(x). The reverse KL still forces the student to match this target, but now the target is the empirically-grounded success rate. The mode-seeking property works for calibration rather than against it: the student concentrates its confidence distribution around the empirically correct value.

Why not switch to forward KL? Forward KL D_{KL}(q \| p) is mean-seeking — it forces p to cover all regions where q has mass. At confidence positions, if the teacher distribution reflected genuine uncertainty (spread across [0, 1]), forward KL would force the student to similarly spread out, which sounds desirable. However, forward KL is known to produce more diffuse, less sharp learned distributions and is generally less effective for distillation of high-quality trajectories. The target replacement approach achieves the calibration benefit without changing the divergence, preserving the distillation stability that makes reverse KL effective for capability transfer.

Hyperparameter Configuration and Computational Infrastructure

The paper provides detailed hyperparameters in Table 6 (Appendix C.4). Key configurations:

General and data:

Base models: Qwen3-8B, Olmo-3-7B-Instruct, plus Qwen3 family at 0.6B, 1.7B, 4B, 14B, 32B for scaling analysis.
Thinking mode: False (no explicit chain-of-thought enforced beyond the prompt format).
Maximum prompt length: 2048 tokens.
Maximum response length: 8192 tokens.

Batching and rollout:

Question batch size: 32.
Mini batch size: 32.
Number of rollouts for empirical target: K = 8 (default; ablated).
Inference engine: vLLM for high-throughput rollout generation.
Rollout temperature: 1.0 (standard sampling, no temperature scaling).
Validation rollout temperature: 0.6 with top-p = 0.95.

Distillation loss:

Top-K distillation: 100 (only the top-100 token probabilities are used in the KL computation, reducing memory).
Distillation divergence: Reverse-KL.
Teacher-EMA update rate: α = 0.05.
Rollout importance sampling clip: 2 (for SDPO).

Training and optimization:

Optimizer: AdamW.
Learning rate: swept over {1 × 10^{-6}, 1 × 10^{-5}, 2 × 10^{-5}\}.
Warmup steps: 10.
Weight decay: 0.01.
Gradient clip norm: 1.0.

Compute infrastructure: All experiments on a single node with four NVIDIA H200 GPUs, using CUDA 12.8 and PyTorch v2.7.0. For SDFT-based experiments, the HuggingFace TRL library; for SDPO-based experiments, the ver1 library with PyTorch FSDP2 for distributed training.

Summary of Design Choices and Their Justifications

Target replacement rather than reward shaping: avoids multi-objective optimization instability and the capability tax observed in RL-based calibration methods (RLCR, CAR). CaOPD has a single loss function — calibration is encoded in the target, not as a competing objective.
Empirical rollouts rather than intrinsic uncertainty: μ̂(x) is unbiased, connects to the established gold-standard of behavioral consistency for uncertainty estimation, and directly measures what calibration requires (expected accuracy = predicted confidence).
Online estimation with EMA teacher: prevents target staleness as the model improves during training; the confidence target dynamically tracks the model's evolving capability.
K = 8 rollouts: provides 9 distinct target levels (increments of 0.125) — sufficient granularity for graded uncertainty — while keeping sampling cost manageable. Smaller K yields quantized targets that trap models in overconfidence; larger K has diminishing returns.
Autoregressive decoupling: reasoning tokens appear before confidence tokens in the sequence, so the reverse KL at reasoning positions is computed on prefixes that don't yet include the confidence replacement. This ensures low-confidence targets don't degrade reasoning capability — empirically confirmed by the accuracy trajectory in Figure 2 (left).
Reverse KL preserved unchanged: inherits the mode-seeking property that is effective for capability distillation; the mode at confidence positions is simply pointed at μ̂(x) rather than 1.0.
Verbalized confidence format: the Confidence: [value] format with explicit parsing boundaries enables clean target replacement; the paper shows (Appendix D.4, Figure 7) that CaOPD's replacement provides a strong formatting signal, driving even small models (0.6B) to >90% format adherence within 100 steps.

4. Key Insights and Innovations

Innovation 1: Miscalibration as a Mathematical Consequence of Privileged Conditioning, Not a Training Artifact

The dominant working assumption in the post-training community — implicit in the design of methods like SDPO, SDFT, RLHF, and GRPO — is that overconfidence is an incidental byproduct of optimization choices: loss functions that don't penalize it, prompts that don't elicit it, or reward models that lack the signal to shape it. Under this view, calibration can be patched in later, either through reward shaping (RLCR, CAR) or post-hoc temperature scaling.

This paper makes the case that this assumption is provably false for the entire class of on-policy distillation methods. Propositions 1–3 collectively establish that when a teacher conditions on privileged evidence Z that carries mutual information about correctness beyond what the student observes at deployment, the student cannot achieve calibration by imitating the teacher — not because of imperfect optimization, but because the target distribution itself is miscalibrated with respect to the student's information state. Specifically:

Proposition 1 proves that the teacher's confidence μ_T(X, Z) is not recoverable from X alone — the student is chasing a target it fundamentally cannot learn.
Proposition 2 proves that the teacher's trajectory distribution has strictly lower entropy than the student's, and the reverse KL (being mode-seeking) forces the student to collapse its distribution accordingly.
Proposition 3 proves that filtering training contexts to be "helpful" (successful rollouts, correct demonstrations) creates an upward-biased surrogate for the student's true success probability.

What distinguishes this from prior theoretical work on calibration (e.g., Guo et al., 2017a, which analyzed calibration as an empirical property of neural network training) is the causal specificity: the paper doesn't just observe that OPD produces overconfidence — it identifies the information asymmetry as the root cause and proves that it must produce overconfidence under standard reverse-KL minimization. This transforms miscalibration from a bug that might be tuned away into a structural invariant of the privileged-conditioning architecture. The implications extend beyond OPD: any post-training method where a model learns from its own outputs under enriched conditioning (including some forms of RLHF with verifier rewards, or self-play with ground-truth access) inherits this problem.

The practical consequence is equally significant: it means patches that don't address the information asymmetry are mathematically insufficient. Reward-shaping methods like RLCR add a penalty term to the RL objective without changing the fact that the teacher's confidence is an invalid target for the student's deployment-time information state. The capability tax observed in Table 3 is therefore not an implementation issue but a consequence of trying to optimize against two mutually inconsistent signals. CaOPD's target replacement — which switches the supervision signal from μ_T(X, Z) to μ̂(X) ≈ μ(X) — is not just more effective; it addresses the root cause identified by the theory.

This is fundamentally a diagnostic contribution: it gives the field a vocabulary (Information Gap, Entropy Collapse, Optimism Bias) and a formal framework for reasoning about when and why post-training methods will fail at calibration, independent of implementation details. A future method that uses a different divergence, a different teacher construction, or a different architecture could be analyzed through these same propositions — if it conditions the teacher on privileged evidence and minimizes a mode-seeking divergence, overconfidence is inevitable.

Innovation 2: The Concept of Student-Grounded Confidence Supervision and Training-Time Amortization

Prior work on LLM uncertainty falls into two camps with opposite deployment characteristics. Test-time sampling methods (SelfCheckGPT, SAC3, semantic entropy, self-consistency) produce reliable uncertainty estimates by measuring behavioral consistency across multiple generations, but they multiply inference cost by K at deployment — prohibitive for latency-sensitive applications. Verbalized confidence methods (Kadavath et al., 2022b; Lin et al., 2022b; Tian et al., 2023) require only a single forward pass and work with black-box APIs, but they are catastrophically miscalibrated after standard post-training. The field has implicitly accepted a compute-calibration trade-off: reliable uncertainty costs O(K) inference, while O(1) uncertainty is unreliable.

CaOPD introduces a fundamentally different position along this trade-off axis: training-time amortization. The key conceptual move is to accept that multi-sample behavioral consistency (μ̂(x)) is the gold-standard signal for epistemic uncertainty — the paper explicitly endorses the findings of the hallucination and self-consistency literature on this point (Appendix B.5) — but to shift when that signal is computed. Instead of computing μ̂(x) at test time for each user query, CaOPD computes it during the training-phase data generation loop and distills it into the model's parametric weights through the standard reverse-KL objective. At deployment, the calibrated confidence emerges from a single forward pass — the O(K) cost is paid once during training and amortized across all future queries.

This is not an incremental engineering improvement. It reorganizes the relationship between training compute and deployment cost in a way that has no precedent in the calibration literature. Prior attempts to improve single-pass confidence (RLCR, CAR) tried to suppress overconfidence by penalizing it in the objective function, which created a multi-objective optimization problem. CaOPD's amortization strategy avoids this entirely: it constructs a supervised target μ̂(x) that is already calibrated with respect to the student's deployment-time information state. The model learns calibration through supervised distillation of an unbiased estimator, not through RL penalty optimization against a competing reward signal.

The practical significance is best understood through the training dynamics in Figure 2. CaOPD's accuracy trajectory "perfectly shadows" SDPO (left panel) — the model learns what to answer from the teacher's privileged context, exactly as in standard OPD. Simultaneously, the calibration loss converges rapidly and stably (middle panel) — the model learns how certain to be from the empirical target, without competing with the capability signal. This parallel learning of two decoupled objectives within a single loss function is direct evidence that the amortization approach resolves the trade-off that RL-based methods could not.

The conceptual framework — paying O(K) training cost for O(1) deployment calibration — also opens a broader design space. The rollout budget K can be tuned independently of deployment constraints (the paper finds K = 8 sufficient in Appendix D.3), and the verifier R(x, a) can be swapped modularly (Teacher-Anchored Self-Consistency in Appendix B.6 provides an unsupervised alternative). This decouples the quality of the calibration signal from deployment latency, a property that none of the existing paradigms (logit-based, sampling-based, or RL-calibrated) offer.

Innovation 3: Strict Pairwise Ranking (SPR) as a Diagnostic for Confidence Saturation and the Distinction Between Calibration and Discrimination

The calibration literature has long relied on Expected Calibration Error (ECE), Brier Score (BS), and reliability diagrams as primary evaluation metrics. These metrics measure absolute calibration — whether predicted confidence matches empirical accuracy in aggregate. However, as the paper demonstrates through both theoretical reasoning and empirical evidence, these metrics can be dangerously misleading for post-trained models. A model that uniformly outputs Confidence: 1.0 on every prompt will have poor ECE (unless it happens to achieve 100% accuracy, which it does not), but standard AUROC — the most common metric for confidence discrimination — awards it a score of 0.5 even when its confidence is completely uninformative for distinguishing correct from incorrect answers.

SPR is introduced specifically to expose confidence saturation — the pathological regime where a model's confidence scores collapse to a narrow range (typically near 1.0) and lose all discriminative power. By penalizing ties to zero (SPR = P(c⁺ > c⁻), no credit for c⁺ = c⁻), SPR directly answers the most deployment-relevant question: does the model's confidence actually tell you whether the answer is right? The paper's evidence that this matters is stark. In Table 2, Qwen3-8B under standard SDFT on Tool Use achieves AUROC ≈ 0.5 but SPR = 0.085 — meaning that in 91.5% of pairwise comparisons between a correct and incorrect answer, the model either assigns them equal confidence or assigns higher confidence to the wrong answer. CaOPD restores SPR to 0.555, indicating that more than half the time, the model now correctly ranks its correct answer as more certain than its incorrect one.

This is not merely a new metric. It is a reconceptualization of what successful calibration means for agentic LLM deployment. A model with low ECE but low SPR is useless as a decision-making agent — it cannot decide when to act autonomously versus when to escalate to a human or a larger model, because its confidence provides no information about correctness. The paper's cascade routing discussion (Appendix F) makes this explicit: a calibrated confidence score is valuable only if it discriminates, because a threshold-based router (e.g., "escalate if confidence < 0.8") is meaningless when confidence is a constant.

The conceptual contribution is to separate two dimensions of calibration quality that prior work had conflated:

Absolute calibration (ECE, BS): does the model's confidence match its accuracy rate on average?
Discriminative calibration (SPR): does the model assign systematically higher confidence to correct answers than incorrect ones?

Standard OPD models fail catastrophically on the second dimension even when the first is "acceptable" (because high ECE from overconfidence reflects consistent overconfidence paired with moderate accuracy). CaOPD's improvements in SPR (e.g., from 0.085 to 0.555 on Qwen3-8B Tool Use under SDFT; from 0.387 to 0.599 on Science Q&A) are arguably more significant than its ECE reductions, because they represent the recovery of a fundamentally broken capability: the ability to distinguish its own successes from its own failures.

Innovation 4: The Scaling Law of Miscalibration as an Empirical Universal Across Post-Trained Models

The most visually arresting result in the paper is Figure 1 (left): a scatter plot of Mean Confidence vs. Accuracy for 12 leading LLMs spanning proprietary APIs and open-weight models, showing almost all of them trapped in an "Overconfidence Zone" — their mean confidence far exceeds their actual accuracy. Even as capability scales from ~57% accuracy (GPT-5.2) to ~84% accuracy (Claude-Opus-4.6), mean confidence remains pegged near 0.8–1.0, creating a massive calibration gap. The paper terms this the Scaling Law of Miscalibration and demonstrates it dynamically in Figure 4: as Qwen3 scales from 0.6B to 32B, standard SDFT models form a flat horizontal line at mean confidence ≈ 1.0 regardless of accuracy, while CaOPD models track the ideal calibration line where confidence rises proportionally with actual capability.

What makes this an innovation rather than an unsurprising observation is the differential diagnosis of its cause. The paper doesn't simply document that LLMs are overconfident — many prior works have done so (Guo et al., 2017b; Kadavath et al., 2022a; Geng et al., 2024). It traces the scaling law specifically to the post-training paradigm rather than to pretraining artifacts or model scale. The evidence for this attribution is in the paper's controlled experiments:

Base instruction-tuned models (Qwen3-8B, Olmo-3-7B-Instruct) are already overconfident but not saturated (Table 1: base OCG = +58.7% and +49.3% respectively).
Applying OPD (SDFT or SDPO) systematically worsens this pathology, driving mean confidence toward 0.97–1.00 (Table 1: SDFT mean confidence reaches 0.972–0.999 across all settings).
The scaling analysis (Figure 4) shows that under SDFT, mean confidence is near-constant across 0.6B–32B — the post-training method, not the model size, is the binding constraint on calibration.

This is a reframing of the scaling narrative. The dominant story in the LLM literature is that scaling model size and training compute produces monotonic improvements across essentially all capabilities. The Scaling Law of Miscalibration shows a striking exception: scaling up capability does not improve calibration under standard post-training; it entrenches the same blind optimism in larger models that afflicted smaller ones. The right panel of Figure 1 — where CaOPD's 8B model achieves reliability (1-Brier Score = 0.87) rivaling frontier APIs — makes the counter-narrative explicit: targeted calibration intervention at small scale can match or exceed the calibration quality of models orders of magnitude larger that rely on standard post-training.

The innovation is therefore not just the empirical finding but the invalidation of the "scale will fix it" assumption that implicitly underlies much of the field's approach to alignment and safety. If miscalibration is a structural consequence of how we post-train models rather than a symptom of insufficient capability, then scaling up capability without changing the post-training method will never produce trustworthy uncertainty signaling. CaOPD demonstrates that the fix is algorithmic, not scalar — a conceptual contribution with immediate implications for how the field allocates resources between scaling pretraining and improving post-training pipelines.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper evaluates on two domains: Science Q&A using the Chemistry L-3 subset from SciKnowEval (Feng et al., 2024), and Tool Use using the ToolAlpaca dataset (Tang et al., 2023). Science Q&A provides multiple-choice chemistry questions testing factual scientific reasoning; Tool Use requires mapping user requests to correct API specifications. Both use standard train/test splits (approximately 75/5/20% for Science Q&A; the original split for ToolAlpaca). The Chemistry test set is also used for out-of-distribution evaluation (training on Tool Use, testing on Chemistry).
Base model(s). Primary testbeds are Qwen3-8B (Yang et al., 2025) and Olmo-3-7B-Instruct (Olmo et al., 2025), chosen to represent different model families under the same distillation pipeline. For scaling analysis (Section 5.5), the paper extends to the full Qwen3 family at 0.6B, 1.7B, 4B, 8B, 14B, and 32B parameters. The paper also evaluates 12 frontier models in zero-shot settings (Table 10, Figure 1): proprietary APIs including GPT-5.2, GPT-5.4, Claude-Opus-4.6, Claude-Sonnet-4.5, Gemini-2.5-Flash, Gemini-2.5-Pro, Gemini-3.1-Flash-Lite, Gemini-3.1-Pro; and open-weight models including DeepSeek-V3.1, Kimi-K2.5, Qwen3.5-397B, and MiniMax-M2.5.
Metrics. The paper evaluates across two dimensions. For capability: final-answer Accuracy (%). For calibration: Expected Calibration Error (ECE) — the expected absolute difference between confidence and accuracy within binned confidence intervals; Brier Score (BS) — the mean squared error between predicted confidence and binary correctness outcomes; Overconfidence Gap (OCG) — mean confidence minus empirical accuracy, where a large positive OCG indicates systematic optimism; and Strict Pairwise Ranking (SPR) — the probability that a correct response receives a strictly higher confidence score than an incorrect one (P(c⁺ > c⁻)), with ties awarded zero credit. SPR is specifically designed to expose confidence saturation (where models uniformly predict c ≈ 1.0), which standard AUROC masks by awarding 0.5 credit for ties. Reliability diagrams (Figure 1) use 1-Brier Score as a summary reliability metric.
Baselines. The paper evaluates against two categories of post-training methods. Capability-focused paradigms: SDFT (Shenfeld et al., 2026) — offline self-distillation with ground-truth demonstration injection; SDPO (Hübotter et al., 2026) — online self-distillation with successful rollout retrospection; and GRPO (Shao et al., 2024) — standard reinforcement learning without calibration penalties. Calibration-aware RL methods: RLCR (Damani et al., 2025) — integrates proper scoring rules (Brier score) into the PPO reward to penalize overconfident mistakes; CAR (Xuan et al., 2026) — penalizes the dichotomy between high verbalized confidence and low accuracy using a calibration-penalized reward formulation adapted within the RLCR training pipeline. The base instruction-tuned model (pre-OPD) is also reported as a reference point.
Generation budget / compute accounting. The paper measures training-time and deployment-time costs separately. All methods are trained to convergence with identical data budgets and optimizer steps. CaOPD's additional training cost is the generation of K rollouts per prompt for empirical target estimation (default K = 8, ablated in Appendix D.3); under SDPO, these rollouts are already generated by the base algorithm ("the marginal sampling cost is therefore zero"), so CaOPD adds only verifier evaluation. Under SDFT, K additional rollouts per prompt are generated. Per-step wall-clock time is reported in Figure 2 (right), showing CaOPD is "nearly indistinguishable" from standard SDPO. Deployment cost is O(1) forward pass for all methods including CaOPD — the key distinction from test-time sampling methods (SelfCheckGPT, self-consistency) which incur O(K) inference latency. The FLOPs-matched comparison (Section 5.5, Figure 4) is done by fixing model scale and comparing calibration at equivalent accuracy levels, not by controlling total FLOPs.
Cross-validation / statistical protocol. The paper does not use explicit cross-validation for strategy selection (unlike the reference paper's two-fold CV within difficulty bins). Instead, all methods are trained to convergence under fixed hyperparameters (Table 6), and final evaluation is on held-out test sets. The rollout budget K = 8 is selected via ablation (Appendix D.3, Figure 6). For the online empirical target estimation (Appendix B.4), the model periodically generates fresh rollouts from the current checkpoint via vLLM to prevent target staleness. Confidence format adherence rates (Appendix D.4, Figure 7) are tracked across training steps to monitor the optimization's progress on the formatting constraint.

Main Quantitative Results

The experiments are organized around four questions, corresponding to the paper's core narrative: (1) Does OPD inherently exacerbate overconfidence? (2) Can CaOPD align confidence without sacrificing capability? (3) Does CaOPD generalize under distribution shift and continual learning? (4) Does the Scaling Law of Miscalibration hold across model scales, and can CaOPD break it?

Does On-Policy Distillation Inherently Exacerbate Overconfidence? (Section 5.2)

Headline finding: Standard OPD methods systematically drive up mean confidence toward saturation (near 1.0) while base models are already overconfident, validating the theoretical claim that privileged-conditioning distillation induces entropy collapse and optimism bias.

Table 1 reports mean confidence and Overconfidence Gap (OCG) across all model-domain-method combinations. The base instruction-tuned models already exhibit severe optimism: Qwen3-8B on Science Q&A has mean confidence 0.897 against 31.0% accuracy (OCG = +58.7%); Olmo-3-7B-Instruct on Tool Use has mean confidence 0.911 against 54.4% accuracy (OCG = +36.7%). Applying SDFT or SDPO consistently worsens this pathology — highlighted in red in the paper's table — driving mean confidence to 0.972–0.999 across nearly all settings. For instance, Qwen3-8B SDFT on Science Q&A reaches mean confidence 0.972 (OCG = +48.1%, slightly reduced from base only because accuracy improved from 31.0% to 49.1%); on Tool Use, SDFT mean confidence reaches 0.996 (OCG = +32.0%) and SDPO reaches 0.999 (not shown in Table 1 but implicit from the SDPO mean confidence values reported in Table 3). The key pattern: OPD makes an already-bad situation strictly worse, confirming that the privileged-conditioning mechanism — not merely base model miscalibration — is the driver.

In contrast, CaOPD structurally corrects this drift. For Qwen3-8B SDFT + CaOPD on Tool Use, mean confidence collapses from 0.996 (standard SDFT) to 0.699, bringing OCG to -0.7% — near-perfect marginal alignment. On Science Q&A, OCG drops from +48.1% (SDFT) to -3.3% (SDPO + CaOPD). The pattern is consistent across models: CaOPD consistently reduces mean confidence to levels that track actual accuracy rather than saturating at the ceiling.

The Optimization Trajectories in Figure 2 provide dynamic evidence. The left panel shows accuracy during training: CaOPD's curve for both Qwen3-8B and Olmo-3-7B "perfectly shadows" the SDPO curve, indicating zero capability degradation from the confidence calibration signal. The middle panel shows calibration loss for CaOPD converging rapidly and stably — the calibration objective does not fight the capability objective. The right panel shows per-step wall-clock time: CaOPD and SDPO are "nearly indistinguishable," confirming that the additional rollout computation does not bottleneck training (rollouts are parallelized via vLLM).

Aligning Confidence Without Sacrificing Capability (Section 5.3)

Headline finding: CaOPD achieves Pareto-optimal calibration — substantially reducing ECE and BS while maintaining or improving accuracy — and restores discriminative ranking (SPR), unlike RL-based calibration methods which impose a severe capability tax.

Comparison against SDFT (Table 2). On Qwen3-8B Science Q&A, CaOPD reduces ECE from 0.486 (SDFT baseline) to 0.266 (−0.220 absolute reduction), BS from 0.477 to 0.281 (−0.196), while accuracy improves from 49.1% to 50.0%. Critically, SPR recovers from 0.387 to 0.599 (+0.212) — the model regains the ability to assign higher confidence to correct answers. The pattern is even starker on Tool Use: SDFT's SPR is 0.085, indicating near-complete discriminative collapse (in 91.5% of correct-vs-incorrect pairs, confidence does not correctly rank the correct answer higher). CaOPD restores SPR to 0.555, while simultaneously improving accuracy from 67.6% to 70.6%.

On Olmo-3-7B-Instruct, similar patterns hold: Science Q&A ECE drops from 0.429 to 0.176 (−0.253), BS from 0.423 to 0.260 (−0.163), with accuracy statistically flat (45.2% vs. 44.8%). Tool Use shows the strongest absolute gains: ECE from 0.304 to 0.177 (−0.127), accuracy from 67.6% to 72.1% — CaOPD simultaneously improves both capability and calibration, a result that standard RL-based methods cannot match.

Comparison against RL-based baselines (Table 3). This is the paper's most critical empirical contribution. Under the SDPO backbone (which achieves higher accuracy than SDFT), the paper compares CaOPD against RLCR, CAR, GRPO, and standard SDPO. The headline: RL-based calibration methods suffer a severe capability tax that CaOPD avoids entirely.

On Qwen3-8B Science Q&A: RLCR achieves accuracy 65.8% and CAR 61.6% — compared to GRPO (74.5%) and SDPO (80.6%), this represents a ~15–19 percentage point sacrifice in capability to achieve calibration (RLCR: ECE 0.143; CAR: ECE 0.151). CaOPD achieves accuracy 80.3% (matching SDPO's 80.6%) while reaching ECE 0.141 (competitive with the best RL methods) and BS 0.134 (strictly better than RLCR's 0.152 and CAR's 0.148). The SPR gap is decisive: CaOPD reaches 0.669 vs. RLCR's 0.546 and CAR's 0.645 — despite using no explicit ranking loss, the empirical target replacement produces better discriminative calibration.

On Tool Use, the same pattern intensifies: RLCR accuracy drops to 61.9% vs. SDPO's 66.2%; CAR drops further to 60.2%. CaOPD achieves accuracy 70.9% — actually improving over SDPO by 4.7 percentage points — while delivering ECE 0.133 (vs. RLCR 0.175, CAR 0.181) and BS 0.164 (vs. RLCR 0.218, CAR 0.229).

On Olmo-3-7B-Instruct, the capability tax is equally severe: Science Q&A RLCR accuracy 53.4% vs. GRPO 59.7% vs. SDPO 63.0%; CaOPD achieves 64.0% with ECE 0.241 (vs. SDPO 0.286). Tool Use: CaOPD matches SDPO accuracy (67.6% vs. 68.0%) while halving ECE (0.180 vs. 0.314).

The SPR story. Across all settings in Table 3, CaOPD achieves the highest SPR: 0.669 on Qwen3-8B Science Q&A (vs. SDPO 0.571, CAR 0.645), 0.654 on Olmo-3-7B Science Q&A (vs. SDPO 0.505, RLCR 0.523). This is the paper's strongest evidence that CaOPD learns a genuine uncertainty representation: the model doesn't just shift its mean confidence downward (which RL methods can achieve) but actively learns to assign higher scores to correct answers, restoring the discriminative function that standard OPD destroys.

Generalization Under Distribution Shift and Continual Learning (Section 5.4)

Headline finding: CaOPD's calibrated confidence generalizes OOD and prevents "calibration forgetting" during continual training, whereas standard SDFT suffers catastrophic calibration collapse in both settings. The paper evaluates under the SDFT backbone (chosen because SDFT was explicitly designed for continual learning; Shenfeld et al., 2026).

Out-of-Distribution generalization (Figure 3a, Table 7). When transferring from Tool Use training to Science Q&A testing (Chemistry domain):

Standard SDFT maintains reasonable OOD accuracy (Qwen3-8B: 36.7%) but suffers complete calibration collapse: ECE spikes to 0.599, BS to 0.591, and SPR plummets to 0.353. The Overconfidence Gap remains massive despite the domain shift because SDFT has memorized the high-confidence linguistic style of the source domain.
CaOPD in the same OOD setting achieves accuracy 44.3% (actually higher than SDFT's 36.7%) with ECE 0.358 — a 40.2% relative reduction in calibration error. SPR is 0.348, comparable to SDFT's 0.353 but with a much better-calibrated mean confidence (0.334 vs. SDFT's 0.961). The paper attributes this to CaOPD learning a "meta-skill" — mapping internal uncertainty to verbalized confidence — rather than memorizing source-domain confidence patterns.

On Olmo-3-7B-Instruct, the OOD pattern replicates: SDFT OOD ECE 0.574 vs. CaOPD OOD ECE 0.371; SDFT mean confidence remains elevated at 0.903 with only 32.9% accuracy, while CaOPD's mean confidence drops to 0.752.

Tables 7 and 8 (Appendix D.1) provide full results across both domains and both transfer directions (Tool Use to Chemistry, Chemistry to Tool Use). The pattern is symmetric: in both directions, CaOPD substantially outperforms SDFT on OOD calibration metrics while maintaining competitive OOD accuracy.

Continual Training and "Calibration Forgetting" (Figure 3b, Table 7). In the CT setting, models are trained sequentially — first on Tool Use, then on Science Q&A (Chemistry) — and evaluated on the final domain (Chemistry). Prior work had shown SDFT prevents catastrophic forgetting of capabilities, but the paper identifies a new pathology: capability preservation does not guarantee calibration preservation.

Standard SDFT in CT: accuracy 46.2% (recovered from OOD collapse, approaching the ID baseline of 49.1%), but ECE is 0.513, BS is 0.508, and SPR is 0.230 (vs. ID SPR of 0.387). The model has learned new domain knowledge but its confidence expression is broken by sequential updates.
CaOPD in CT: accuracy 48.6% (matching SDFT), but ECE is 0.126 (vs. ID 0.266), BS is 0.209 (vs. ID 0.281), and SPR is 0.662 (actually higher than ID SPR of 0.599). CaOPD does not merely preserve calibration through sequential training — it improves it. The paper attributes this to the continual re-anchoring of confidence tokens to the model's evolving capability: because ˆμ(x) is computed online from fresh rollouts, the confidence target dynamically tracks the model's performance on the new domain.

On Tool Use as the CT target domain (Table 8), the pattern holds: SDFT CT has ECE 0.352 and SPR collapsed to 0.000 (total loss of discriminative resolution), while CaOPD CT achieves ECE 0.253 and SPR 0.510. The SPR = 0.000 result for SDFT is particularly damning — after sequential training, the SDFT model cannot distinguish correct from incorrect answers at all using confidence, even though its accuracy recovers to 64.7%.

The paper emphasizes that this calibration forgetting is a "critical new pathology" that prior continual learning evaluations (which focused exclusively on accuracy) completely missed. CaOPD's target replacement effectively immunizes the model against this failure by explicitly regularizing confidence outputs against execution outcomes at each stage of sequential training.

The Scaling Law of Miscalibration: Decoupling Capability from Calibration (Section 5.5)

Headline finding: Standard SDFT entrenches miscalibration across model scales — mean confidence remains a flat line near 1.0 from 0.6B to 32B parameters — while CaOPD breaks this scaling curse, achieving confidence that dynamically scales with actual accuracy and enabling a compact 8B model to rival frontier API calibration quality.

Macro-level evidence (Figure 1, Table 10). The left panel of Figure 1 plots Mean Confidence vs. Accuracy for 12 leading LLMs on Science Q&A. Almost all models fall in the "Overconfidence Zone" above the ideal calibration line (y = x). Proprietary models: GPT-5.2 (accuracy 63.3%, mean confidence 0.608 — relatively well-calibrated), GPT-5.4 (72.9%, 0.802), Claude-Opus-4.6 (83.8%, 0.838 — near-perfect), Gemini-2.5-Flash (57.6%, 1.000 — saturation), Gemini-3.1-Pro (71.9%, 0.997). Open-weight models: DeepSeek-V3.1 (56.7%, 1.000 — saturation), Qwen3.5-397B (71.4%, 0.929), MiniMax-M2.5 (47.6%, 0.974). The only model near the ideal line is Claude-Opus-4.6, a massively scaled frontier model with undisclosed alignment pipelines. The right panel of Figure 1 ranks models by reliability (1-Brier Score), showing CaOPD's Qwen3-8B achieving 0.87 — competitive with Claude-Opus-4.6 (0.91) and ahead of most other frontier models.

Table 10 provides exact numbers. The key pattern: most models exhibit massive confidence saturation or near-saturation. Mean confidence is above 0.97 for 8 of the 12 evaluated models, including all Gemini variants, DeepSeek-V3.1, Kimi-K2.5, and MiniMax-M2.5. This is consistent with the Entropy Collapse analysis (Proposition 2): irrespective of provider or architecture, the dominant post-training paradigms produce the same overconfidence pathology at scale. The caveat noted in the paper: API models are evaluated zero-shot, whereas CaOPD is fine-tuned on Chemistry, so the accuracy comparison is not apples-to-apples. The meaningful comparison is calibration quality at a given accuracy level — and CaOPD's 8B model achieves calibration that rivals the best frontier APIs.

Scaling analysis within a single family (Figure 4). The paper tracks the Qwen3 family from 0.6B to 32B parameters under three conditions: Base (instruction-tuned, no OPD), SDFT (standard self-distillation), and CaOPD (SDFT + target replacement). Accuracy (bubble x-position) scales from ~55% (0.6B) to ~75% (32B) across all conditions.

Under standard SDFT (orange bubbles), mean confidence forms a flat horizontal line near 1.0 (right panel: Accuracy vs. Mean Confidence). Whether the model has 0.6B parameters and 55% accuracy or 32B parameters and 75% accuracy, SDFT forces it into near-absolute certainty. The calibration gap widens as scale increases — larger models are more capable but no more calibrated. This is the dynamic manifestation of the Scaling Law of Miscalibration: scaling parameters does not resolve the structural optimism induced by privileged conditioning.

Under CaOPD (green bubbles), mean confidence rises proportionally with accuracy — from ~0.3 at 0.6B to ~0.8 at 32B — tracking the ideal calibration line (not shown but implicit from the diagonal in Figure 1). The left panel (Accuracy vs. Reliability, 1-BS) shows CaOPD systematically pushing the Pareto frontier upward and rightward: at every scale, CaOPD achieves strictly better reliability than SDFT at comparable accuracy. The middle panel (Accuracy vs. SPR) shows CaOPD dominating the discriminative ranking frontier: SPR improves from near-zero at small scales under SDFT to competitive values under CaOPD.

The paper argues this demonstrates decoupling: CaOPD separates the capability gain from increasing scale (which SDFT also achieves) from the calibration alignment (which SDFT fails to achieve). The result is that a CaOPD-trained 8B model (accuracy ~70%, reliability ~0.87) achieves calibration quality comparable to the 32B SDFT model (accuracy ~75%, reliability ~0.87) and rivals frontier APIs — calibration is not a function of scale but of training objective.

Ablation Studies and Robustness Checks

Rollout budget K for empirical target estimation (Appendix D.3, Figure 6): The paper ablates K ∈ {1, 2, 4, 8, 16, 32} on the Qwen3-8B SDFT backbone. Task accuracy remains flat across all K, confirming that capability is fully decoupled from confidence sampling variance. Calibration exhibits a phase transition: K = 1–4 produces "heavily quantized targets that trap the model in overconfidence" because the target distribution has only 2–5 distinct values (binary or near-binary), preventing the model from learning graded uncertainty. K = 8 emerges as the "compute-efficient sweet spot" with 9 distinct levels (increments of 0.125), where mean confidence collapses toward accuracy and discrimination (SPR) peaks. Beyond K = 8, returns diminish rapidly. The right panels of Figure 6 visualize the confidence distribution reshaping: standard OPD collapses the distribution into a delta at 1.0; CaOPD training targets (K = 16 shown) present a dispersed distribution; the deployed single-pass CaOPD model successfully internalizes this dispersion, placing uncertain predictions into lower bins. This ablation is critical because it establishes a practical K value and demonstrates that CaOPD is robust to the sampling budget — too-small K fails because of target quantization, not because the method is wrong.

Teacher-Anchored Self-Consistency (TA-SC) as unsupervised empirical target (Appendix B.6, Table 4): For domains lacking objective verifiers, the paper proposes substituting R(x, a) with agreement against a teacher-generated reference trajectory. Table 4 compares CaOPD with the objective verifier against CaOPD with TA-SC on Qwen3-8B SDFT. TA-SC closely tracks the objective verifier: on Science Q&A, ECE 0.271 vs. 0.266, SPR 0.577 vs. 0.599; on Tool Use, ECE 0.240 vs. 0.228, SPR 0.547 vs. 0.555. Minor degradations occur due to "teacher fallibility and spurious disagreements" in open-ended JSON generation (Tool Use accuracy drops from 70.6% to 68.3%). The paper emphasizes that TA-SC still "drastically outperforms the standard SDFT baseline" (which has ECE 0.321 and SPR 0.085 on Tool Use), confirming that CaOPD is modular and maintains its decoupling even with unsupervised targets. This is the key robustness check for broader applicability beyond domains with ground-truth verifiers.

PRM aggregation strategy: Not applicable — this paper does not use process reward models. The verifier R(x, a) is task-specific and deterministic (exact match for Science Q&A; API schema validation for Tool Use).

Logit-based vs. verbalized calibration (Appendix B.7): The paper provides empirical evidence that CaOPD's verbalized humility does not "bleed into" internal reasoning logits. Figure 2 (left) shows CaOPD's accuracy trajectory perfectly shadows SDPO throughout training — if low confidence targets at positions t ∈ I_c were corrupting logit entropy at reasoning positions t ∈ I_a, measurable accuracy degradation would be observed. The autoregressive structure provides causal decoupling: the confidence replacement μ̂(x) appears after reasoning tokens in the sequence, so the reverse KL at reasoning positions operates on prefixes that have not yet seen the confidence value.

Training dynamics and format adherence across model scales (Appendix D.4, Figure 7): Tracking confidence format adherence (fraction of rollouts that produce a parsable Confidence: [value] string) reveals that small models (0.6B, 1.7B) initially struggle with format adherence (50–80% at step 0), but CaOPD's target replacement acts as a "corrective alignment mechanism" — within 50–100 steps, even the 0.6B model exceeds 90% adherence. Larger models (14B, 32B) maintain near-100% adherence from initialization. This is because the revised completion ỹ always contains a well-formatted confidence segment (the string representation of μ̂(x)), providing a consistent formatting signal at every training step that standard OPD lacks (where format adherence is only implicitly encouraged by the prompt instruction).

OOD generalization direction symmetry (Tables 7 and 8): Transfer is evaluated in both directions: Tool Use → Chemistry (Table 7) and Chemistry → Tool Use (Table 8). In both directions, CaOPD substantially outperforms SDFT on OOD calibration metrics. In the Chemistry → Tool Use direction (Table 8), CaOPD's OOD ECE is 0.307 vs. SDFT's 0.329 on Qwen3-8B; on Olmo-3-7B-Instruct, 0.320 vs. 0.381. The pattern is symmetric, confirming that CaOPD's calibration generalization is not an artifact of domain-specific properties.

Continual training order symmetry (Tables 7 and 8): CT is evaluated in both sequential orders: Tool Use → Chemistry tested on Chemistry (Table 7) and Chemistry → Tool Use tested on Tool Use (Table 8). In the Tool Use → Chemistry direction, SDFT CT SPR collapses to 0.230 (Qwen3-8B); in the Chemistry → Tool Use direction, SDFT CT SPR collapses to 0.000. In both cases, CaOPD CT maintains healthy SPR (0.662 and 0.510 respectively), demonstrating that the calibration forgetting pathology is general and that CaOPD's prevention mechanism is independent of training order.

Cost structure and training overhead (Appendix D.2, Table 9, Figure 2 right): The paper explicitly accounts for CaOPD's training-time cost. Under SDPO, the K rollouts for μ̂(x) are "already generated by the base training loop to identify successful trajectories" — CaOPD reuses them with zero additional sampling cost, adding only a lightweight verifier evaluation. Under SDFT, K additional rollouts per prompt are generated, representing a constant-factor increase in rollout generation (parallelized via vLLM, not bottlenecking the training loop). The per-step wall-clock time (Figure 2 right) is "nearly indistinguishable" between CaOPD and SDPO for Qwen3-8B and Olmo-3-7B, confirming that the additional computation does not materially slow training. At deployment, all methods require a single O(1) forward pass.

Critical Assessment

The paper makes several central claims. Here we examine whether the reported experiments genuinely support each one, what was and was not tested, and where the evidence has meaningful limitations.

Claim 1: Standard OPD inherently induces systematic overconfidence, a "Scaling Law of Miscalibration" that traps frontier LLMs in blind optimism.

The evidence for this claim is the strongest in the paper. Table 1 establishes that applying SDFT or SDPO to already-overconfident base models systematically increases mean confidence toward saturation (0.972–0.999 across nearly all settings). The macro-level API evaluation (Figure 1, Table 10) demonstrates that 10 of 12 leading LLMs operate in the "Overconfidence Zone" with massive gaps between mean confidence and accuracy. The scaling analysis (Figure 4) shows that SDFT locks mean confidence at ~1.0 regardless of model size from 0.6B to 32B — scaling parameters does not resolve the pathology.

However, the paper's attribution of this specifically to OPD's privileged-conditioning mechanism (Propositions 1–3) relies on a comparison that conflates two distinct phenomena: base model overconfidence (which exists in instruction-tuned models before any OPD) and OPD-exacerbated overconfidence (the additional effect of self-distillation). The base Qwen3-8B already has OCG = +58.7% on Science Q&A. SDFT increases mean confidence but also increases accuracy, so the OCG actually decreases from +58.7% to +48.1% (Table 1). The claim that OPD "inherently induces" overconfidence needs the qualifier that it amplifies confidence saturation while the accuracy improvement partially offsets the gap. The far-right OCG values in the CaOPD columns of Table 1 (−11.8% and −3.3%) are arguably better evidence for the paper's mechanism: CaOPD structurally corrects the problem that even base models exhibit, not just the OPD-specific component.

A missing experiment: the paper does not evaluate whether the base (pre-OPD) models' overconfidence is also attributable to privileged-conditioning mechanisms during instruction tuning or RLHF, which share the information asymmetry structure. Without this, the causal claim that OPD is the root cause of the Scaling Law of Miscalibration (as opposed to one exacerbating factor among several) is somewhat overstated. The API models in Figure 1 have undergone diverse post-training pipelines, not just self-distillation — their overconfidence may arise from shared structural properties (as the theory predicts), but the paper does not verify this by, e.g., comparing models pre- and post-RLHF.

Claim 2: CaOPD's target replacement achieves Pareto-optimal calibration without the capability tax of RL-based methods.

This is the paper's most important empirical claim, and the evidence in Table 3 is compelling within the tested regime. On Qwen3-8B Science Q&A, CaOPD achieves accuracy 80.3% (matching SDPO) while RLCR drops to 65.8% and CAR to 61.6%. CaOPD's calibration metrics (ECE 0.141, BS 0.134) are competitive with or better than the specialized RL methods (RLCR: ECE 0.143, BS 0.152; CAR: ECE 0.151, BS 0.148). On Tool Use, CaOPD actually improves accuracy over SDPO (70.9% vs. 66.2%) while delivering better calibration. The SPR advantage (CaOPD 0.669 vs. RLCR 0.546 on Qwen3-8B Science Q&A) is consistent across all settings.

However, the comparison against RL baselines has a important limitation: the paper does not report whether it attempted to tune RLCR and CAR hyperparameters to find better capability-calibration trade-off points. RL methods have a reward coefficient that balances the capability reward against the calibration penalty — the paper may be comparing against poorly-tuned baselines where the calibration penalty is too aggressive. The fact that CAR, which explicitly "penalizes the dichotomy between high verbalized confidence and low accuracy," is the most capability-degraded method (61.6% vs. SDPO's 80.6%) suggests the calibration penalty may be dominating the capability reward in the reported configuration. A sweep over the penalty coefficient and a Pareto frontier plot (accuracy vs. ECE) for each RL method would be a much fairer comparison and is conspicuously absent.

Additionally, the CaOPD results on SDPO (Table 3) show CaOPD achieving strictly better calibration than SDPO while matching accuracy — but the accuracy of SDPO itself (80.6% on Qwen3-8B Science Q&A) is substantially higher than SDFT CaOPD (50.0%, Table 2). The paper does not explain why SDPO achieves such dramatically higher accuracy than SDFT on the same task, nor whether the CaOPD gains over SDPO in Table 3 are statistically significant (no confidence intervals are reported). The 80.3% vs. 80.6% comparison is within the noise of a 500-question test set.

Claim 3: CaOPD generalizes robustly under OOD shifts and prevents "calibration forgetting" during continual learning.

The OOD and CT results (Figure 3, Tables 7–8) are among the paper's strongest findings because they test a property that the method was not explicitly designed for. CaOPD's OOD ECE reduction of ~40% (from 0.599 to 0.358 on Qwen3-8B) is substantial and consistent across both transfer directions. The CT results, where CaOPD actually improves calibration through sequential training (CT SPR 0.662 vs. ID SPR 0.599 on Qwen3-8B), are surprising and suggest that the continuous re-anchoring mechanism provides a form of calibration regularization.

The limitation: these experiments are conducted only under the SDFT backbone. The paper does not evaluate OOD generalization or continual learning under the SDPO backbone, which is the stronger capability method. Since SDPO CaOPD achieves much higher accuracy than SDFT CaOPD (80.6% vs. 50.0% on Qwen3-8B Science Q&A), the OOD and CT behavior of SDPO CaOPD may differ — and this is the version that would likely be deployed in practice. The omission is particularly notable given that SDFT was explicitly chosen because it was "designed to enable continual learning" — CaOPD's SDFT results may benefit from SDFT's own architectural properties, and the extent to which CaOPD independently contributes to CT robustness is not isolated.

Claim 4: CaOPD amortizes expensive test-time confidence estimation into a single-pass model with zero deployment overhead.

The deployment cost claim is straightforward and well-supported: CaOPD's inference is a single O(1) forward pass, identical to any uncalibrated model. The training-time cost analysis (Appendix D.2, Table 9, Figure 2 right) is reasonable but would benefit from quantifying the absolute training FLOPs or wall-clock time, not just per-step comparisons. The statement that CaOPD's per-step time is "nearly indistinguishable" from SDPO (Figure 2 right) uses a logarithmic y-axis on the time scale where the differences at low step counts are visually compressed. The ~5–10 second differences visible in the raw plot (CaOPD Qwen3-8B at ~45s vs. SDPO at ~40s at low steps) represent a ~12.5% overhead that "nearly indistinguishable" understates.

The claim that under SDPO the "marginal sampling cost is zero" because rollouts are reused is important but not universally applicable — it depends on SDPO's specific implementation where the rollout budget for successful trajectory identification matches or exceeds K = 8. If a production SDPO pipeline uses fewer rollouts per prompt, CaOPD would increase the sampling cost. The paper does not report the rollout budget used in its SDPO baseline, so the claim of "zero marginal cost" is not verifiable from the provided information.

Overall experimental design strengths: (1) The dual-backbone evaluation (SDFT and SDPO) demonstrates that CaOPD works across OPD paradigms, not just one. (2) The dual-model evaluation (Qwen3-8B and Olmo-3-7B-Instruct) provides evidence of cross-architecture generalization within the evaluated scale range. (3) The multi-scale analysis (0.6B–32B) is thorough and directly supports the scaling law argument. (4) The introduction of SPR as an evaluation metric meaningfully advances the calibration evaluation toolbox and exposes a failure mode (confidence saturation, discriminative collapse) that ECE alone masks. (5) The OOD and CT experiments test generalization properties that most calibration papers ignore, and the findings (calibration forgetting as a new pathology) are genuinely novel.

Overall experimental design weaknesses: (1) Single-digit test set size: the Science Q&A test set appears to be ~100 questions (500 total across 5 difficulty quintiles in the reference paper context, but the paper's own split is described as "approximately 20% test split" of an unspecified total). Table 10 reports API evaluations as percentages with one decimal — if the test set has fewer than 200 questions, the one-decimal reporting implies a resolution of ~0.5 percentage points or coarser, making small differences in accuracy (e.g., 80.3% vs. 80.6%) statistically indistinguishable. (2) No confidence intervals are reported for any metric. (3) The API comparison (Table 10) conflates zero-shot and fine-tuned settings; the paper acknowledges this but the visual prominence of Figure 1 may mislead readers into comparing CaOPD's accuracy against frontier APIs directly. (4) The chemistry-only Science Q&A domain is narrow; findings may not transfer to other reasoning types or knowledge domains. (5) The ToolAlpaca dataset uses simulated API interactions — calibration behavior on real-world tool use with live API execution may differ because verifier feedback dynamics change. (6) The CaOPD vs. RL comparison lacks a Pareto frontier sweep over the RL penalty coefficient, making the "capability tax" claim potentially exaggerated if better-tuned RL baselines exist at intermediate penalty strengths. (7) The ablation on K (Appendix D.3) uses SDFT; the optimal K for SDPO CaOPD may differ since SDPO's base accuracy is much higher, changing the distribution of μ̂(x) values.

6. Limitations and Trade-offs

6.1 The Difficulty Estimation Cost Is Unaccounted for in the Headline Efficiency Gains

The assumption or constraint. CaOPD's training-time calibration depends on computing the empirical success rate μ̂(x) via K student rollouts per training prompt. The paper explicitly acknowledges that this adds training cost and that the cost varies by OPD backbone:

"computing ˆµ(x) requires K rollouts per prompt during training, increasing the training compute budget" (Section 8, Limitations)

Under SDFT, this represents a K-fold increase in sampling per prompt (default K = 8, meaning 8× more rollouts than the baseline single trajectory generation). Under SDPO, the paper claims the rollout cost is already incurred by the base algorithm ("the marginal sampling cost is therefore zero," Appendix D.2), making the added cost only verifier evaluation. However, this "zero marginal cost" claim is conditional: it holds only when SDPO's base rollout budget already equals or exceeds K = 8. The paper does not report the SDPO rollout budget, so the claim is not verifiable from the provided information. If a production SDPO pipeline uses fewer rollouts per prompt (e.g., 4), CaOPD would require doubling the rollout budget.

The consequence. The headline efficiency gains — CaOPD achieving Pareto-optimal calibration with "no capability tax" and near-identical per-step time (Figure 2, right) — are computed without factoring in the cost of generating K rollouts during training. While the paper emphasizes that this cost is "fully amortized at deployment where inference remains a single O(1) forward pass" (Section 8), the total training compute budget for CaOPD (SDFT variant) is substantially higher than standard SDFT. For a practitioner comparing methods, the relevant metric is total cost to achieve a given calibration-quality level, not just deployment cost. If CaOPD requires 8× more sampling during training to achieve its calibration gains, the training-time cost-accuracy-calibration trade-off may be less favorable than the deployment-time comparison alone suggests.

This limitation is most consequential for organizations with constrained training budgets (as opposed to deployment-constrained environments). A deployment with high query volume and tight latency constraints does benefit from training-time amortization — paying once to generate rollouts during training saves O(K) cost per query at deployment. But for a team training a single model for limited deployment, the total training + inference cost may favor simpler alternatives (e.g., post-hoc temperature scaling, which has negligible training cost and achieves partial calibration improvement).

What evidence exists in the paper. Figure 2 (right) shows per-step wall-clock time on a logarithmic scale, where CaOPD and SDPO appear "nearly indistinguishable." However, closer inspection reveals consistent gaps: at low step counts, CaOPD Qwen3-8B takes ~45 seconds per step vs. SDPO's ~40 seconds — a ~12.5% increase that the "nearly indistinguishable" characterization understates. Appendix D.2 provides a cost breakdown (Table 9) confirming that under SDFT, CaOPD adds "K per prompt" extra rollout sampling. Figure 6 (left) shows that K = 8 is the "compute-efficient sweet spot," with smaller K failing to achieve calibration and larger K providing diminishing returns — evidence that K = 8 is not arbitrarily chosen but represents the minimum budget to achieve the reported calibration quality. The paper does not report total training FLOPs, wall-clock training time to convergence, or total cost compared to the baselines.

Mitigation status. The paper acknowledges the cost partially in Section 8 (quoted above) and Appendix D.2, but frames it as acceptable because "this cost is fully amortized at deployment." It suggests future work on "efficient rollout strategies" including "adaptive allocation, e.g., early stopping when initial rollouts return unanimous verifier scores" (Section 8) as a way to reduce training cost. The K = 8 ablation (Appendix D.3, Figure 6) establishes that the cost can be reduced to K = 4 at some calibration cost (ECE degrades) or increased beyond K = 8 with minimal further gain, providing practical guidance for budget-constrained settings. However, the paper does not report a full cost-calibration Pareto frontier across K values, which would allow practitioners to make explicit cost-quality trade-off decisions.

6.2 The Method Requires an Objective Task Verifier, Which Is Not Available in Many Real-World Domains

The assumption or constraint. CaOPD's core procedure — computing μ̂(x) = (1/K) Σ R(x, a_k) — requires an objective verifier function R(x, a) that can judge whether a generated trajectory a_k is correct. The paper acknowledges:

"a potential critique of CaOPD is its reliance on an objective task verifier R(x, a) to compute the empirical target ˆµ(x)" (Appendix B.6)

The paper argues that this is not an additional requirement because standard OPD methods already need verifiers: SDPO "explicitly requires environment verifiers (e.g., compilers or reward models) to generate the privileged feedback z," and SDFT "requires ground-truth demonstrations to construct the privileged teacher context z" (Appendix B.6). CaOPD therefore "simply reuses this existing infrastructure" with "zero additional assumptions or dependencies."

The consequence. This argument limits CaOPD's applicability to the same domains where SDPO or SDFT are already applicable — which is a substantial constraint. The paper's own evaluation domains illustrate the point: Science Q&A uses multiple-choice exact match (trivially verifiable), and ToolAlpaca uses structured API calls with schemas (deterministically verifiable against expected parameters). These are domains where ground-truth correctness is unambiguous and can be automatically checked. Many important LLM applications — open-ended dialogue, creative writing, summarization, complex multi-step planning, code generation with nuanced correctness — lack such clean verification signals. In these domains, neither SDPO (which needs a verifier to identify successful rollouts) nor CaOPD (which needs a verifier to compute μ̂(x)) can be applied without a learned reward model or human evaluation, both of which introduce noise, bias, and cost.

The paper's proposed alternative — Teacher-Anchored Self-Consistency (TA-SC; Appendix B.6) — substitutes verifier correctness with agreement against a teacher-generated reference trajectory. Table 4 shows this works reasonably well on the evaluated domains (ECE degradation from 0.266 to 0.271 on Science Q&A; from 0.228 to 0.240 on Tool Use). However, TA-SC introduces its own limitations: (1) it requires a semantic equivalence function Equiv(·, ·), which for open-ended generation becomes a hard NLP problem in itself; (2) the teacher's reference may be incorrect (the paper acknowledges "teacher fallibility"); (3) agreement with the teacher does not guarantee correctness — a systematically biased teacher will anchor μ̂_SC(x) to a biased target. The paper's empirical evidence for TA-SC (Table 4) is limited to the two evaluation domains; there is no evidence that TA-SC would work for genuinely open-ended tasks where neither exact-match verification nor structured schema validation is available.

What evidence exists in the paper. Table 4 provides the only evidence for TA-SC, comparing it against the objective verifier version on Qwen3-8B SDFT. The results show minor but consistent degradation: Tool Use accuracy drops from 70.6% to 68.3%, which the paper attributes to "spurious disagreements" in open-ended JSON generation. The macro-level API evaluation (Table 10, Figure 1) shows that even frontier models with sophisticated alignment pipelines exhibit massive overconfidence, suggesting that the verifier dependency is a real deployment barrier — these models are used in domains without clean verifiers, and CaOPD in its primary form cannot be directly applied to them.

Mitigation status. The paper frames TA-SC as a modular alternative and validates it empirically, so the limitation is partially addressed. However, TA-SC is validated only on the same two domains that already have objective verifiers — a somewhat circular evaluation (we show that the unsupervised alternative works when we happen to have ground-truth to verify against, but we don't test it on a domain where ground-truth is genuinely unavailable). The paper does not evaluate TA-SC on an open-ended generation task, which would be the most convincing demonstration of broader applicability. The "future work" discussion (Section 8) does not explicitly call for extending CaOPD to verifier-free domains — an omission that suggests the authors view verifier dependency as an acceptable constraint rather than a limitation to be resolved.

6.3 The Single-Benchmark, Single-Family Evaluation Leaves Generalization to Other Reasoning Types and Model Architectures Unverified

The assumption or constraint. All fine-tuning experiments use two model families (Qwen3 and Olmo-3) evaluated on two specific domains (Science Q&A Chemistry and ToolAlpaca). The paper's central empirical claims — that CaOPD achieves Pareto-optimal calibration without capability tax, generalizes OOD, and prevents calibration forgetting — are demonstrated within this limited scope. The paper does not evaluate on mathematical reasoning (e.g., GSM8K, MATH), code generation (e.g., HumanEval, MBPP), or knowledge-intensive QA beyond chemistry, nor does it evaluate on model families beyond Qwen and Olmo.

The paper acknowledges that CaOPD "inherits the requirement that the base model possesses sufficient in-context learning capacity to benefit from the privileged teacher context z" (Section 8). The scaling analysis (Section 5.5) shows CaOPD works from 0.6B to 32B within the Qwen3 family, and the Olmo-3-7B results show cross-family transfer at the 7–8B scale. However, whether CaOPD's mechanism transfers to other architectures (e.g., non-Transformer models, mixture-of-experts, models with different pretraining objectives) or to larger scale ranges (70B+) is unknown.

The consequence. Two specific risks for practitioners:

Domain dependence of the OPD calibration pathology. The paper's central diagnosis — that OPD induces overconfidence via information asymmetry — is domain-agnostic in theory (it depends only on I(R; Z | X) > 0, which should hold whenever Z provides useful information). But the magnitude of the effect may vary substantially across domains. In tasks where the privileged context provides only marginal benefit (small I(R; Z | X)), the entropy collapse and optimism bias may be less severe, and CaOPD's benefits proportionally smaller. In tasks where the model's base capability is very high (e.g., simple factual recall), the gap between μ_T(X, Z) and μ(X) may be small because the model already answers correctly without Z, making the information asymmetry less consequential. The paper's evaluation does not span this spectrum of difficulty and privilege-dependence.

Interaction with model-specific properties. The paper's scaling analysis shows that CaOPD's calibration gains are consistent across Qwen3 scales, but this is within a single model architecture and training recipe. Models with different pretraining calibration properties (some models are naturally better calibrated than others; Kadavath et al., 2022b; OpenAI, 2023) may show different sensitivity to OPD-induced overconfidence and different responsiveness to CaOPD's correction. The Olmo-3-7B results provide some cross-family evidence but only at one scale.

What evidence exists in the paper. The dual-domain (Science Q&A, Tool Use) and dual-backbone (SDFT, SDPO) evaluation is stronger than typical calibration papers, which often evaluate on a single task. The scaling analysis (Figure 4) across 6 model sizes from 0.6B to 32B is relatively thorough. The API evaluation (Figure 1, Table 10) provides indirect evidence that the overconfidence pathology generalizes across many model families and providers. However, this API evaluation is observational (zero-shot, no CaOPD training on these models), so it only establishes that the problem generalizes, not that the solution does. The paper does not provide CaOPD results on any model family beyond Qwen3 and Olmo-3, nor on any domain beyond the two tested.

Mitigation status. The paper does not claim generality beyond the evaluated scope — the domain and model selection is explicit. The limitations section (Section 8) does not discuss domain or model generalization as an open question, focusing instead on capability bounds (the base model's ICL capacity) and formatting constraints. A practitioner considering CaOPD for a new domain (e.g., code generation) or model family (e.g., LLaMA, Gemma) would need to run their own validation; the paper provides no guidance on expected transfer performance.

6.4 Calibration Forgetting in Continual Learning Is Demonstrated Only Under the Weaker SDFT Backbone, Not SDPO

The assumption or constraint. The paper's generalization and continual learning experiments (Section 5.4, Figure 3, Tables 7–8) are conducted exclusively under the SDFT backbone. The paper explains this choice: "We employ the SDFT backbone, which was explicitly designed to enable continual learning" (Section 5.4). However, the paper's own results show SDPO is the substantially stronger capability method: on Qwen3-8B Science Q&A, SDPO achieves 80.6% accuracy vs. SDFT's 49.1% (Tables 2 and 3). This raises the question of whether CaOPD's OOD and CT benefits transfer to the higher-capability regime that a practitioner would likely deploy.

The consequence. If CaOPD SDPO exhibits different OOD or CT behavior than CaOPD SDFT — for instance, if the stronger base capability of SDPO models changes how confidence distributions shift under domain transfer — then the paper's generalization claims do not apply to the most capable version of the method. This is practically significant because a practitioner choosing between SDFT + CaOPD and SDPO + CaOPD would want to know: does the calibration generalization I observe under SDFT also hold under SDPO? The paper provides no evidence either way.

The calibration forgetting pathology that the paper identifies in SDFT (SPR collapsing to 0.230 in CT on Qwen3-8B, Table 7) may be less severe or more severe under SDPO, depending on how SDPO's stronger capability interacts with the sequential training dynamics. If SDPO models are more confident overall (as Table 3 suggests: SDPO mean confidence is not directly reported but can be inferred from the OCG context), the calibration forgetting effect might be amplified because there is more "confidence mass" to shift during sequential updates.

What evidence exists in the paper. None directly. The paper provides no OOD or CT results for CaOPD under SDPO. The SDPO results (Table 3) are all in-distribution (ID training and testing on the same domain). The OOD experiments use SDFT exclusively, and the continual learning experiments explicitly state "We employ the SDFT backbone" (Section 5.4). The CaOPD SDPO results in Table 3 are the most impressive in the paper (accuracy 80.3% with ECE 0.141 on Qwen3-8B Science Q&A), but these are ID numbers — there is no evidence that this strong calibration holds when the SDPO CaOPD model is deployed OOD or fine-tuned sequentially.

Mitigation status. The paper does not acknowledge this as a limitation. The choice of SDFT for CT experiments is justified by SDFT's design properties, but the absence of SDPO CT experiments is not discussed. The "Limitations and Future Work" section (Section 8) does not mention OOD or CT generalization gaps. A practitioner concerned about calibration under domain shift would need to conduct their own SDPO OOD evaluation, as the paper provides no guarantees or even suggestive evidence.

6.5 The Strict Pairwise Ranking (SPR) Metric Exposes a Discriminative Failure That ECE Masks, but the Paper Does Not Establish That SPR Improvements Translate to Downstream Decision-Making Benefits

The assumption or constraint. The paper introduces SPR as a key evaluation metric and uses it to argue that standard OPD produces models that "cannot distinguish correct from incorrect answers at all using confidence" (e.g., SPR = 0.085 for Qwen3-8B SDFT on Tool Use, Table 2). CaOPD restores SPR to 0.555 in the same setting, and the paper presents this as evidence that CaOPD "restores discriminative ranking." However, the paper does not validate that SPR improvements translate to better performance on downstream tasks that depend on confidence discrimination — such as selective prediction (abstaining when confidence is low), cascade routing (deferring uncertain queries to larger models), or human-in-the-loop decision-making (escalating low-confidence predictions for review).

The paper discusses cascade routing as a potential application (Appendix F): "a CaOPD-trained small model could serve as a lightweight triage router in cascade inference systems, deferring uncertain queries to larger models." It further notes: "empirically validating cascade routing performance (e.g., cost-accuracy tradeoffs under different thresholds) is left to future work." This is an explicit acknowledgment that the downstream impact remains unmeasured.

The consequence. SPR improvements are a necessary but not sufficient condition for improved downstream decision-making. A model with SPR = 0.555 correctly ranks correct answers above incorrect ones in 55.5% of pairwise comparisons — substantially better than SPR = 0.085, but still far from perfect. For a selective prediction system that must choose a confidence threshold for abstention, what matters is not just pairwise ranking but the full calibration curve: at confidence threshold τ, what are the precision and recall of the model's high-confidence predictions? SPR does not directly capture this. A model could have high SPR but poor threshold-level calibration (e.g., its high-confidence predictions are mostly correct, but the absolute confidence values are poorly calibrated, making it hard to set τ). Conversely, a model with lower SPR but better absolute calibration might be more useful for threshold-based decision-making.

The paper's reliance on SPR as a primary calibration quality metric — and the strong claims about CaOPD "restoring the ability to assign strictly higher confidence to correct answers over mistakes" — implicitly assumes that SPR improvements translate to downstream utility. Without selective prediction or routing experiments, this assumption remains untested.

What evidence exists in the paper. The paper reports SPR for all experiments (Tables 2, 3, 7, 8, 10) and consistently shows CaOPD achieving the highest SPR. The calibration loss convergence during training (Figure 2, middle) provides indirect evidence that CaOPD learns a genuine uncertainty mapping. The OOD and CT SPR results (Tables 7–8) show that the discriminative improvement persists and sometimes improves under distribution shift. However, none of this directly measures whether CaOPD's SPR improvements make the model more useful for a downstream task.

The paper includes qualitative case studies (Appendix E) showing examples where CaOPD correctly expresses high confidence on correct answers (E.1) and low confidence on incorrect answers (E.2), while SDFT outputs Confidence: 1.0 in both cases. These are illustrative but not systematic — they show that CaOPD can discriminate in specific instances, not that it does so reliably across the full test distribution or that this discrimination is operationalized effectively in a decision-making pipeline.

Mitigation status. The paper explicitly defers this to future work: "empirically validating cascade routing performance under this architecture is a natural next step" (Section 8, Future Work; also Appendix F). The limitation is therefore acknowledged but not addressed within the paper's scope. For a practitioner making deployment decisions, the absence of downstream validation means that SPR improvements are a promising signal rather than a guaranteed benefit — the cost of implementing CaOPD in a production pipeline must be weighed against benefits that have not been directly measured.

6.6 The Method Depends on a Specific Verbalized Confidence Format That May Degrade Under Prompt Variation or Adversarial Input

The assumption or constraint. CaOPD relies on the model producing a parsable confidence segment c in the format Confidence: [value] (Appendix B.2). Target replacement (Section 4.2) overwrites these tokens with μ̂(x), and the reverse KL at confidence positions (Equation 7) trains the model to output the empirical target. This requires that: (1) the model reliably produces the expected format during training, and (2) at deployment, users or downstream systems can parse the confidence value from the model's output.

The paper acknowledges this constraint:

"verbalized confidence requires the model to produce a parsable confidence segment c matching the specified format. While CaOPD's target replacement provides a strong formatting signal during training (Appendix D.4), occasional formatting failures can occur at test time, a limitation shared with all verbalized uncertainty methods and one we also observed in proprietary API evaluations." (Section 8, Limitations)

The consequence. Format adherence failures at deployment are qualitatively different from miscalibration failures. A miscalibrated model still outputs a confidence number — it's just wrong. A format-adherence failure produces no confidence number at all, or an unparseable string, which downstream systems must handle as a special case (e.g., treating it as unknown confidence, or falling back to a default). The paper's training dynamics (Appendix D.4, Figure 7) show that even CaOPD-trained models at small scales start with poor format adherence (50–80% at initialization for 0.6B and 1.7B models) and are driven to >90% adherence within 100 steps by the target replacement signal. However, ">90%" means that ~5–10% of outputs still have format issues after convergence for small models. At deployment, if a user's prompt is out-of-distribution or adversarial, format adherence may degrade further.

This limitation also constrains CaOPD's applicability to models with sufficient instruction-following capacity. The paper notes that CaOPD "inherits the requirement that the base model possesses sufficient in-context learning capacity to benefit from the privileged teacher context z" (Section 8), but format adherence imposes a separate, potentially stricter constraint: the model must be capable enough to reliably follow the Confidence: [value] formatting instruction. The scaling analysis (Figure 7) shows that 0.6B models require explicit training to reach acceptable adherence; models below this scale (not tested) may not reach usable adherence rates even with CaOPD's formatting signal.

More subtly, CaOPD's dependence on a specific, rigid format may interact poorly with prompt variation. If a deployment system uses a different confidence elicitation prompt (e.g., "How confident are you on a scale of 1–10?" or "State your certainty as a percentage"), the format learned during CaOPD training may not transfer, and the model may revert to its pre-CaOPD overconfidence. The paper evaluates only the specific Confidence: [value] format described in Appendix B.2 — there is no ablation over prompt variations or confidence formats.

What evidence exists in the paper. Appendix D.4 (Figure 7) provides format adherence trajectories across model scales, showing that small models start with poor adherence and are corrected by CaOPD training. The API evaluation (Table 10) reports that some proprietary models also exhibit "formatting failures" — though the paper does not quantify this rate. The qualitative case studies (Appendix E) show correctly formatted outputs. There is no evaluation of adversarial prompts, prompt variation, or format robustness.

Mitigation status. The paper acknowledges the limitation in Section 8 but treats it as inherent to all verbalized confidence methods rather than a CaOPD-specific issue. No mitigation is proposed beyond the implicit claim that CaOPD's target replacement provides a stronger formatting signal than standard OPD. A practitioner deploying CaOPD would need to implement format validation at the output parsing stage and decide on a fallback policy for unparseable outputs — the paper provides no guidance on what fallback is appropriate or how often it would be needed in practice.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper establishes a structural diagnostic that will change how the post-training community thinks about calibration: the finding that on-policy distillation (OPD) and related privileged-conditioning methods necessarily produce miscalibration is a reframing, not merely a new mitigation technique. Prior to this work, the dominant narrative treated overconfidence as an incidental byproduct of optimization choices—loss functions that don't penalize it, prompts that don't elicit honest uncertainty, reward models that lack the right signal. Under that view, calibration was a matter of adding the right penalty term or post-hoc adjustment. This paper shows, through Propositions 1–3, that the miscalibration is mathematically inevitable when a model distills from its own outputs under privileged conditioning. The target itself—the teacher's confidence μ_T(X, Z)—is not measurable from deployment-time information. The reverse KL forces the student to collapse its entropy to match a distribution it cannot justify. The training data filters out failure cases, creating an upward-biased surrogate target. These are not bugs; they are consequences of the architecture.

This reframing has several immediate effects on the field's conceptual landscape:

It resolves the contradiction between OPD's capability gains and its calibration failures. The existing narrative said: OPD makes models smarter but also overconfident, and we need to find ways to fix the overconfidence. The paper shows that these two properties are causally linked—the same mechanism that transfers capability (distilling from privileged-conditioned trajectories) also transfers miscalibration (because the teacher's confidence is a function of evidence the student lacks). This means you cannot simply "add calibration" to an OPD pipeline as an afterthought; the information asymmetry is baked into the training objective. The paper's target replacement solution works precisely because it decouples the two signals—capability cloning from the teacher, confidence supervision from the student's own rollouts—addressing the root cause rather than fighting the symptom.

It invalidates the "scale will fix it" assumption for calibration. The Scaling Law of Miscalibration (Figure 1, Figure 4) is the paper's most important empirical contribution to the broader scaling narrative. The dominant story in the LLM community has been that scaling model size and training compute produces monotonic improvements in essentially all desirable properties. This paper demonstrates a striking exception: scaling up capability under standard post-training does not resolve blind optimism—it entrenches it. Whether a model has 0.6B parameters or 32B (Figure 4), SDFT locks mean confidence at ~1.0. Whether it's GPT-5.2 or Claude-Opus-4.6 (Figure 1), frontier models operate far above the ideal calibration line. The implication is that calibration is not a natural byproduct of capability that will emerge at sufficient scale—it requires explicit algorithmic intervention. This redirects research attention from "how do we scale models to be better calibrated" to "how do we redesign post-training to produce calibration as a first-class property."

It establishes a new axis in the compute-quality trade-off space. Prior work on LLM uncertainty offered a stark choice: pay O(K) inference cost for reliable calibration (self-consistency, semantic entropy, SelfCheckGPT) or accept poor calibration from O(1) single-pass models. The paper's amortization strategy—computing μ̂(x) at training time and distilling it into model weights for deployment-time single-pass inference—represents a genuinely different position on this axis. It accepts that multi-sample behavioral consistency is the gold-standard signal for epistemic uncertainty (endorsing the findings of the hallucination detection literature), but shifts when the computation occurs. This is not an incremental engineering optimization; it reorganizes the relationship between training and deployment compute in a way that has no precedent in the calibration literature. The concept of "amortized self-consistency"—paying the multi-sample cost once during training and getting single-pass calibrated confidence at deployment—opens a design space where the inference budget (K rollouts) can be chosen independently of deployment latency constraints.

It introduces a new evaluation dimension that exposes a previously-hidden failure mode. The SPR metric (Section 5.1, Appendix C.2) reveals that standard OPD doesn't just shift confidence upward—it destroys the model's ability to distinguish its correct answers from its incorrect ones. SDFT on Qwen3-8B Tool Use achieves SPR = 0.085 (Table 2), meaning the model's confidence is essentially random for ranking correctness. Standard metrics like ECE and AUROC mask this collapse because they award partial credit for aggregate calibration or for tied predictions. SPR's property of penalizing ties to zero directly answers the most deployment-relevant question: can this model's confidence score tell you whether to trust its answer? The paper's demonstration that CaOPD restores SPR to 0.555–0.669 across settings is arguably more significant than its ECE reductions, because it represents the recovery of a fundamentally broken function. The implication for the field's evaluation practices is clear: calibration benchmarks should report discrimination metrics (SPR or equivalent) alongside absolute calibration metrics, because a model that is "well-calibrated" in aggregate but cannot rank correctness is useless as a decision-making agent.

It downgrades the attractiveness of RL-based calibration methods. The paper's comparison in Table 3 provides the strongest evidence to date that adding calibration penalties to RL rewards creates a fundamental trade-off that CaOPD avoids entirely. RLCR drops Qwen3-8B Science Q&A accuracy from SDPO's 80.6% to 65.8%; CAR drops it further to 61.6%. These are not marginal degradations—they represent a ~15–19 percentage point sacrifice in capability. The paper's explanation—that reward-shaping methods fight the optimizer with competing signals, while CaOPD's target replacement provides a single coherent objective—suggests that the capability tax is inherent to the RL-based approach rather than a tuning issue. For researchers working on calibration, this redirects effort away from "better penalty functions" and toward "better supervision targets." The strong performance of CaOPD's simple empirical frequency target (μ̂(x)) relative to carefully-designed scoring rules (Brier, proper scoring) suggests that the source of the calibration signal matters more than its mathematical form.

Follow-Up Research This Work Enables

Selective prediction and cascade routing benchmarks for CaOPD-calibrated models. The paper explicitly defers empirical validation of downstream decision-making benefits to future work (Appendix F, Section 8). The natural follow-up is a selective prediction experiment: take a CaOPD-calibrated model, set a confidence threshold τ, abstain on predictions below τ, and measure the accuracy-coverage trade-off curve. Compare against the same model trained with standard SDPO/SDFT (where confidence is saturated near 1.0 and thresholding is meaningless) and against RLCR/CAR (which may have better calibration but lower base accuracy). The paper's SPR results suggest CaOPD should dominate this trade-off, but direct evidence is needed. A cascade routing experiment would pair a CaOPD-calibrated small model (e.g., Qwen3-4B) with a larger model (e.g., Qwen3-32B or a frontier API): route queries where the small model's confidence exceeds τ to the small model, route the rest to the large model, and measure total cost vs. accuracy. The paper's Figure 4 shows CaOPD's mean confidence tracking accuracy across scales, which is exactly the property needed for a threshold-based router to work—but routing performance has not been measured.

OOD and continual learning evaluation of CaOPD under the SDPO backbone. The paper's strongest calibration results are under SDPO (Table 3: CaOPD SDPO achieves 80.3% accuracy with ECE 0.141 on Qwen3-8B Science Q&A, vs. SDFT CaOPD's 50.0% with ECE 0.266). Yet all OOD generalization and continual learning experiments (Section 5.4, Figure 3, Tables 7–8) use only the SDFT backbone. The gap is significant: if CaOPD SDPO's calibration does not generalize OOD or degrades under sequential training, the practical utility of the strongest method is limited. A replication of the OOD and CT protocols under SDPO would answer: does the stronger base capability of SDPO models change how confidence distributions shift under domain transfer? Does the "calibration forgetting" pathology (SPR collapsing to 0.230 or 0.000 in SDFT CT; Tables 7–8) manifest similarly under SDPO? Given that SDPO models already have higher accuracy and potentially different confidence distributions, the calibration forgetting effect could be amplified or attenuated—the theory provides no clear prediction.

Verifier-free CaOPD on genuinely open-ended generation tasks. The paper's Teacher-Anchored Self-Consistency (TA-SC) alternative (Appendix B.6, Table 4) is validated only on the same two domains that already have objective verifiers—a circular evaluation. To establish that CaOPD extends to domains without ground-truth correctness signals, one would need to evaluate TA-SC CaOPD on an open-ended task: summarization (e.g., evaluating factuality via an NLI model), dialogue (e.g., evaluating response quality via an LLM judge), or code generation (e.g., evaluating correctness via test cases that are available at training time but not at deployment). The key measurement would be: does TA-SC CaOPD achieve calibration improvements comparable to the objective-verifier version (Table 4 shows a ~2% ECE degradation on the evaluated domains, which is acceptable), or does the teacher's fallibility and the equivalence function's noise compound to produce a substantially weaker calibration signal? A negative result—TA-SC failing on genuinely open-ended tasks—would bound CaOPD's applicability to domains with at least partial verifiability.

Long-horizon, step-level calibration with process reward models. The paper provides utterance-level (final-answer) confidence calibration. For agentic or multi-step chain-of-thought settings, the natural extension is to apply the target replacement principle at each reasoning step: compute a per-step empirical success rate μ̂_t(x, y_{<t}) from rollouts conditioned on the prefix, and replace confidence tokens at intermediate positions with these step-level estimates. This connects directly to the process reward model (PRM) literature (Lightman et al., 2023; Wang et al., 2023) and could enable early termination of flawed reasoning chains. The paper's autoregressive decoupling argument (Appendix B.7)—that confidence replacement at later positions doesn't corrupt reasoning distributions at earlier positions—is crucial here: it suggests that per-step confidence targets can be inserted without degrading the quality of subsequent reasoning steps. A concrete experiment: train a CaOPD model with per-step confidence on a multi-step math reasoning dataset (e.g., MATH with intermediate supervision), and measure whether per-step confidence scores can detect when a reasoning chain has gone off-track (measured by the correlation between low per-step confidence and eventual incorrect final answers).

Interaction of CaOPD with other post-training stages: RLHF, DPO, and preference optimization. The paper evaluates CaOPD within the self-distillation paradigm (SDFT and SDPO), but modern production pipelines typically involve multiple post-training stages: supervised fine-tuning, reward modeling, RLHF or DPO, and possibly additional self-play or self-improvement loops. An open question is where in this pipeline CaOPD should be inserted for maximum benefit. If CaOPD is applied during SFT (before RLHF), does RLHF subsequently undo the calibration gains (analogous to the "calibration forgetting" observed in continual learning; Section 5.4)? If CaOPD is applied after RLHF, does it interfere with the alignment properties learned during preference optimization? A systematic study that inserts CaOPD at different points in a multi-stage pipeline and measures both calibration and downstream task performance (including alignment metrics like harmlessness and helpfulness) would establish whether CaOPD's calibration is robust to subsequent optimization stages.

Adversarial robustness of CaOPD-calibrated confidence. The paper acknowledges that "occasional formatting failures can occur at test time" (Section 8) and evaluates only a specific prompt format (Confidence: [value]). A stress test would evaluate whether CaOPD's calibration holds under prompt variation: different confidence elicitation formats ("How certain are you on a scale of 0–1?" vs. "State your confidence as a percentage"), different reasoning formats (chain-of-thought vs. direct answer), and adversarial prompts designed to provoke overconfidence (e.g., "I'm sure you know this, just give your best answer with high confidence"). A negative result—CaOPD's calibration degrading significantly under prompt variation—would indicate that the learned calibration is format-specific rather than a robust uncertainty representation, limiting deployability in systems where users or downstream components may vary the prompt format. Conversely, strong robustness would support the paper's claim that CaOPD learns a "generalizable meta-skill" (Section 5.4).

Practical Applications and Downstream Use Cases

Confidence-based triage in agentic systems. The most immediate deployment scenario enabled by CaOPD is a multi-model cascade where a small, cheap model handles most queries and escalates uncertain ones to a larger model. The paper's scaling analysis (Figure 4) shows CaOPD-trained models across 0.6B–32B produce confidence scores that track actual accuracy—the 0.6B model outputs mean confidence ~0.3 with ~55% accuracy, while the 32B model outputs mean confidence ~0.8 with ~75% accuracy. This property means a threshold-based router is meaningful: set τ = 0.7, and the 0.6B model will correctly escalate most of its failures (since their confidence is low) while serving its high-confidence successes at minimal cost. Without CaOPD, the same model outputs Confidence: 1.0 on almost everything (standard SDFT in Figure 4 right panel), making thresholding useless. The cost savings in a production setting where small-model inference is 10–100× cheaper than large-model inference could be substantial, though the paper does not quantify this directly (the cascade routing evaluation is deferred to future work; Appendix F).

Uncertainty-aware data filtering for self-improvement pipelines. The paper's online μ̂(x) estimation (Appendix B.4) provides a built-in signal for identifying which training examples the model finds difficult. In a self-improvement loop—where a model generates its own training data, filters for high-quality examples, and fine-tunes on them—CaOPD's per-example empirical success rate can serve as a quality filter: train only on examples where μ̂(x) > 0.8 (the model consistently gets them right, indicating they're within its capability range and the generated solutions are reliable), discard examples where μ̂(x) < 0.2 (the model consistently fails, indicating the problem is outside its current capability and the generated solutions are likely incorrect noise). This is more principled than filtering by a single confidence score (which standard OPD models cannot provide reliably) or by outcome alone (which discards information about the model's uncertainty). The paper does not evaluate this application, but the infrastructure is already in place—μ̂(x) is computed during training, and the CaOPD framework could be extended to use it as a data selection criterion with minimal modification.

Calibrated small models for on-device deployment with verifier access. For applications where a verifier is available but latency or privacy constraints preclude cloud-based inference (e.g., on-device code generation with test-case execution, edge-deployed scientific QA with answer verification), CaOPD enables a single-pass calibrated model that can accurately signal when to escalate to the user. The paper's results on Qwen3-8B and smaller (0.6B–4B in the scaling analysis, Figure 4) are directly relevant: CaOPD restores the model's ability to express low confidence on uncertain predictions, meaning an on-device system can respond with "I'm not confident about this answer (confidence: 0.3)—would you like me to try a different approach?" rather than silently outputting a wrong answer with Confidence: 1.0. The qualitative case study in Appendix E.2 illustrates exactly this scenario: CaOPD outputs Confidence: 0.10 on an incorrect API call, which "would safely trigger a human-in-the-loop fallback" in production, whereas standard SDFT outputs Confidence: 1.00 on the same incorrect call.

When to Prefer This Method

The paper positions CaOPD against two categories of alternatives—standard OPD methods (SDFT, SDPO, GRPO) and RL-based calibration methods (RLCR, CAR)—and against test-time sampling approaches (SelfCheckGPT, self-consistency). The choice conditions emerge directly from the empirical results:

Prefer CaOPD over standard OPD (SDFT, SDPO, GRPO) when:

You are already using or planning to use on-policy distillation for capability improvement. CaOPD adds calibration with "no capability tax" (Table 3: CaOPD SDPO matches SDPO accuracy while halving ECE) and minimal training overhead (Figure 2 right: per-step time nearly indistinguishable). There is no reason to run standard OPD without CaOPD's target replacement if you have access to K = 8 rollouts per prompt.
Your deployment requires the model to distinguish its own successes from failures—e.g., selective prediction, cascade routing, human-in-the-loop escalation. Standard OPD models have SPR near zero (0.085 for Qwen3-8B SDFT on Tool Use, Table 2), meaning their confidence provides no discrimination. CaOPD restores SPR to 0.555–0.669.
You anticipate distribution shift (OOD generalization) or sequential fine-tuning (continual learning). Standard OPD suffers catastrophic calibration collapse in both settings (Figure 3: SDFT OOD ECE spikes to 0.599; SDFT CT SPR collapses to 0.000–0.230), while CaOPD maintains calibrated confidence.

Prefer CaOPD over RL-based calibration (RLCR, CAR) when:

Task accuracy is paramount. RLCR and CAR impose a severe capability tax—dropping Qwen3-8B Science Q&A accuracy from 80.6% (SDPO) to 65.8% and 61.6%, respectively (Table 3). CaOPD matches SDPO's accuracy while achieving competitive or better calibration.
You want to avoid the optimization complexity of multi-objective RL (tuning penalty coefficients, managing PPO stability, reward normalization). CaOPD modifies only the training data (target replacement), not the loss function or optimizer.

Prefer CaOPD over test-time sampling (SelfCheckGPT, self-consistency) when:

Deployment latency is constrained. Test-time sampling incurs O(K) inference cost per query (each query requires K forward passes). CaOPD requires a single O(1) forward pass at deployment because the O(K) rollout cost is paid during training and amortized across all future queries (Section 6, "Test-Time Amortization"; Appendix D.2, Table 9).
You need black-box or API-compatible deployment. Test-time sampling requires access to multiple samples from the model, which may be rate-limited or expensive for proprietary APIs. CaOPD produces calibrated single-pass verbalized confidence, which works with any API that returns text.

Prefer test-time sampling over CaOPD when:

You cannot afford CaOPD's training-time requirements—the K rollouts per prompt during training, the verifier infrastructure, or the full OPD training pipeline. Test-time sampling can be applied to any off-the-shelf model without retraining.
The domain lacks a reliable verifier and the Teacher-Anchored Self-Consistency alternative (Appendix B.6) is expected to perform poorly (e.g., open-ended generation where semantic equivalence is unreliable). Test-time sampling using self-consistency or semantic entropy does not require a verifier at training time.
You need the absolute highest calibration quality regardless of deployment cost. The paper shows CaOPD achieves strong calibration, but test-time sampling with large K (e.g., 100+ samples) can achieve even finer-grained uncertainty estimates that a distilled single-pass model may not fully capture.