ArXiv: 2510.13998

🎯 Pitch

Fine-tuning LLMs to 1.58-bit usually fails, and the performance gap surprisingly grows with model size—from 13.9 to 15.3 points as you scale from 0.6B to 4B. BitDistill eliminates this scalability failure with a simple continued pre-training warm-up, matching full-precision accuracy while cutting memory 10× and speeding up CPU inference 2.65×.

1. Executive Summary

This paper introduces BitNet Distillation (BitDistill), a three-stage pipeline that fine-tunes off-the-shelf full-precision LLMs into 1.58-bit ternary-weight models for specific downstream tasks, achieving task performance comparable to the FP16 original while delivering up to 10× memory savings and 2.65× faster CPU inference. Using the Qwen3 model family at 0.6B–4B scales on GLUE classification benchmarks and CNN/DailyMail summarization, BitDistill integrates three key techniques: modeling refinement with SubLN layers (inserting additional normalization before attention and FFN output projections to stabilize activation variance), a continued pre-training warm-up step using only 10B tokens (which reshapes the weight distribution to resemble that of a BitNet trained from scratch, preventing convergence to suboptimal local minima), and multi-head attention distillation based on MiniLM (training the quantized student to mimic the teacher's query, key, and value relation matrices at a single carefully selected layer). The paper demonstrates that directly fine-tuning FP16 LLMs into 1.58-bit precision (BitNet-SFT) yields a performance gap that widens with model scale — from 13.9 percentage points at 0.6B to 15.3 at 4B — establishing that this scalability failure is not inherent to extreme quantization but rather a consequence of insufficient adaptation, which BitDistill's continued pre-training stage resolves by enabling the weights to rapidly enter a distribution favorable to ternary optimization before task-specific distillation begins.

2. Context and Motivation

The Core Problem: Fine-Tuning LLMs to 1.58-Bit for Downstream Tasks Remains Unstable and Unreliable

The fundamental question this paper tackles is: can we take an existing, pre-trained full-precision LLM and efficiently convert it into a 1.58-bit ternary model that performs on par with its FP16 original on specific downstream tasks? The answer, prior to this work, was effectively "no" — at least not without prohibitive computational cost or unacceptable performance degradation.

This matters because the deployment landscape for LLMs is increasingly bifurcated. On one side, massive models like GPT-4, DeepSeek-R1, and Qwen3 demonstrate ever-improving capabilities, but their size makes them impractical for resource-constrained environments such as smartphones, edge devices, and embedded systems. On the other side, the techniques designed to compress these models for deployment — quantization — struggle at the extreme low-bit regime (specifically 1.58-bit, where weights are restricted to three values: -1, 0, and 1). The paper identifies a specific failure mode: directly fine-tuning an existing FP16 LLM into a 1.58-bit model for a downstream task produces a substantial performance gap that scales poorly with model size. As shown in Figure 1, this gap grows from 13.9 percentage points at 0.6B parameters to 15.3 at 4B (measured as accuracy loss on classification tasks), revealing that simply applying quantization-aware training to a pre-trained model does not work well and in fact gets worse as models grow larger — a counterintuitive and concerning finding that suggests the adaptation process is fundamentally broken, not just insufficient.

Why This Problem Matters: The Training-From-Scratch Chasm

The paper is motivated by a genuine practical impossibility. The 1.58-bit BitNet architecture — introduced in prior work (Ma et al., 2024, 2025) — has demonstrated that ternary-weight LLMs can achieve competitive performance. However, achieving that performance requires pretraining from scratch on massive corpora (approximately 4 trillion tokens, per the Ma et al. 2025 BitNet b1.58 2B4T report). This creates an enormous barrier:

  • Computational cost: Training a 1.58-bit model from scratch on trillions of tokens requires resources comparable to pretraining a full-precision model — defeating the purpose of seeking efficiency in deployment.
  • Energy overhead: The energy consumption associated with multi-trillion-token pretraining undermines the sustainability arguments for deploying efficient models.
  • Practical infeasibility for most organizations: Only well-resourced labs can afford BitNet-from-scratch training, locking out the broader community that might benefit most from efficient on-device deployment.

The paper frames the alternative — fine-tuning existing FP16 models into 1.58-bit — as critically underexplored. If such conversion could be made to work, it would unlock a drastically cheaper path to efficient deployment: start from the wealth of existing pre-trained models (Qwen, Gemma, LLaMA families), spend a modest additional compute budget adapting them to 1.58-bit for a specific downstream task, and deploy. The paper's explicit goal is to close the gap between this vision and the current reality where direct fine-tuning (BitNet-SFT) produces models 13–15 percentage points below their FP16 teachers.

Where Prior Approaches Fall Short

The paper identifies specific inadequacies in existing methods along two axes: quantization techniques and distillation techniques.

Quantization-aware training (QAT) lacks a recipe for 1.58-bit conversion. The paper acknowledges that QAT has been studied for low-bit LLMs (e.g., EfficientQAT, LLM-QAT) and that post-training quantization (PTQ) methods like GPTQ and AWQ are widely used. However, these efforts have focused on higher bit-widths (4-bit, 3-bit, sometimes 2-bit) and on general language modeling capabilities rather than specific downstream task performance. At 1.58-bit — which is qualitatively different from even 2-bit quantization because the weight space collapses to just three values — the existing QAT recipes break down. The paper demonstrates this directly in Figure 1 and Table 1: BitNet-SFT (direct QAT conversion of FP16 Qwen models to 1.58-bit) yields catastrophic accuracy drops, e.g., 88.01 → 74.09 on MNLI for the 0.6B model. This is not a minor degradation — it is a failure of the approach.

The paper also notes a more subtle problem with PTQ methods: "PTQ always suffer[s] from significant performance degradation, especially when quantization bits are lower than 4 bits" (Section 5, citing Dettmers et al., 2022). The paper does not claim to replace PTQ but rather positions QAT as the necessary path for extreme low-bit regimes, while acknowledging that existing QAT methods have not been designed with 1.58-bit conversion of pre-trained models in mind.

Knowledge distillation for quantization is underexplored at 1.58-bit. The paper cites several works that use distillation to improve quantized LLMs. TSLD (Kim et al., 2023) employs layer-to-layer distillation for ternary quantization but focuses on general language modeling. BitDistiller (Du et al., 2024) uses self-distillation for sub-4-bit LLMs but again targets broad LM capabilities rather than downstream task performance. The paper's critique is specific: "most existing methods primarily target general language modeling capabilities and still exhibit noticeable performance gaps in downstream applications compared to their full-precision counterparts" (Section 5). In other words, distillation techniques exist but have not been validated as a solution for the task-specific 1.58-bit conversion problem.

The instability problem is architectural. The paper identifies a mechanism-level failure that prior work did not adequately address for conversion scenarios. Full-precision models are initialized with activation variance maintained in a stable range by standard initialization schemes. When the same architecture is quantized to 1.58-bit, the activation variance can explode because the ternary weights interact differently with inputs — the quantization function fundamentally changes the scale of the weight matrix's output. The original BitNet architecture addressed this with SubLN layers placed throughout the transformer block, but an off-the-shelf FP16 model (like Qwen3) was not designed with these normalization layers. Directly quantizing such a model creates an architectural mismatch: the normalization regime is wrong for ternary computation. The paper shows in Figure 3(a) that without SubLN insertion, training loss is unstable and converges to higher values, directly linking architectural design to optimization stability.

The "scalability issue" is a previously unidentified phenomenon. The paper's most intriguing diagnostic finding is that the performance gap between FP16 and 1.58-bit BitNet-SFT widens with model size (Figure 1). The paper explicitly states this as "poor scalability" and identifies it as a key challenge not previously characterized. The authors hypothesize that larger models have more parameters that need to adapt from the FP16 weight distribution (which is approximately Gaussian) to a distribution suitable for ternary quantization (which concentrates weights near the transition boundaries between quantization levels). With a limited number of downstream task training tokens, the adaptation is insufficient, and — counterintuitively — larger models adapt worse per parameter. This is not an issue that prior quantization literature addressed, as most work either trained from scratch (where the weight distribution develops naturally) or used PTQ (which doesn't involve training dynamics).

How This Paper Positions Itself

The paper positions BitDistill as the first framework specifically designed for fine-tuning pre-trained full-precision LLMs into 1.58-bit BitNet models for downstream tasks. The authors make this claim explicit in their first contribution: "To the best of our knowledge, we are the first to investigate fine-tuning pre-trained full-precision LLMs into 1.58-bit BitNet for specific downstream tasks, and we identify key challenges including: performance degradation, poor scalability, and training instability."

The framework is structured as a three-stage pipeline that directly addresses each identified failure mode:

  • Stage 1 (Modeling Refinement): Inserting SubLN layers addresses the activation variance instability problem. This is not a novel architectural contribution — SubLN comes from the original BitNet work — but the paper applies it as a surgical modification to existing FP16 architectures to make them compatible with ternary computation, which is a practical engineering contribution rather than a theoretical one.

  • Stage 2 (Continual Pre-Training): Training the SubLN-modified model on 10B tokens of general corpus before task-specific fine-tuning directly addresses the scalability issue by giving the model space to adapt its weight distribution from the FP16-typical Gaussian shape toward the BitNet-typical shape (where weights concentrate near the -1/0 and 0/1 transition boundaries). The paper provides visual evidence in Figure 2, comparing weight distributions of BitNet-trained-from-scratch and BitDistill-after-continued-training, showing they converge to similar patterns. The insight is that continued pre-training serves as a distributional adaptation step that prevents the downstream fine-tuning from getting trapped in suboptimal local minima — a specific mechanistic hypothesis that the paper tests and supports.

  • Stage 3 (Distillation-Based Fine-Tuning): Combining logits distillation and multi-head attention distillation (adapted from MiniLM's relation-based approach) recovers the remaining accuracy. The attention distillation is applied at only a single layer rather than all layers, based on the hypothesis that "conferring greater optimization flexibility to the 1.58-bit student BitNet often yields superior downstream performance" — a non-obvious design choice validated by the ablation in Figure 3(b).

The paper does not claim to advance the theory of quantization or distillation. Instead, it positions itself as a practical systems contribution: a recipe that combines known components (SubLN, MiniLM-style distillation, continued pre-training) in a specific sequence with specific hyperparameter settings to solve a previously unaddressed deployment problem. The evidence is empirical: across three model scales (0.6B, 1.7B, 4B), two task types (classification, summarization), and multiple base model families (Qwen3, Qwen2.5, Gemma), BitDistill achieves performance within approximately 0–1 percentage points of FP16 fine-tuned baselines — a claim that, if robust, would make 1.58-bit deployment practical for downstream applications in a way that training from scratch or direct QAT do not.

The paper also positions itself as illuminating the interaction between low-bit quantization and optimization dynamics. The investigation in Section 4.4 into why continued pre-training works — showing that the weight distribution reshapes to concentrate near quantization boundaries, enabling more frequent value transitions with small gradient steps — provides a mechanistic explanation that generalizes beyond the specific pipeline. This explanatory contribution distinguishes the paper from a pure engineering report and connects it to broader questions about how extreme quantization interacts with learning dynamics.

3. Technical Approach

3.1 Reader Orientation

What the system is: BitDistill is a three-stage training pipeline that takes an off-the-shelf full-precision LLM and converts it into a 1.58-bit ternary-weight model fine-tuned for a specific downstream task, producing a deployable artifact that runs 2.65× faster on CPUs with 10× memory savings while matching the original model's task accuracy.

What problem it solves and the "shape" of the solution: The system addresses three distinct failure modes that occur when naively quantizing a pre-trained FP16 model to 1.58-bit and fine-tuning it — activation variance instability, poor scalability of the performance gap with model size, and residual accuracy loss from precision reduction — by sequentially applying architectural modification (inserting normalization layers into the transformer blocks to stabilize training), distributional adaptation (a brief continued pre-training phase on 10B tokens of general corpus that reshapes the weight distribution to be compatible with ternary optimization), and knowledge transfer (logits distillation plus single-layer attention relation distillation from the FP16 teacher to the quantized student during task-specific fine-tuning). Each stage targets a specific degradation mechanism identified in the paper's diagnostic analysis, and the pipeline is designed so that skipping any individual stage measurably degrades final performance.

3.2 Big-Picture Architecture (Diagram in Words)

The system has four major components connected in a sequential pipeline:

  1. Base FP16 LLM (e.g., Qwen3) — the off-the-shelf pre-trained full-precision model that serves as both the starting point for conversion and, after task-specific fine-tuning, the teacher for knowledge distillation. This model has a standard transformer architecture with pre-normalization but without the internal normalization layers needed for stable ternary computation.

  2. Modeling Refinement Module (Stage 1) — a surgical architectural modification that inserts SubLN normalization layers into each transformer block at two specific positions: immediately before the attention output projection and immediately before the FFN output projection. This produces a structurally modified FP16 model whose activation variance remains bounded under ternary weight computation, enabling stable training in subsequent stages.

  3. Continual Pre-Training Process (Stage 2) — a distributional adaptation phase that trains the SubLN-modified FP16 model on 10B tokens of general corpus (sampled from the FALCON dataset) using standard next-token prediction, still in full precision, to reshape the weight distribution from the original approximately Gaussian form toward a distribution where mass concentrates near the quantization boundaries (the transitions between -1 and 0, and between 0 and 1). This phase produces a model whose weights are "ready" for ternary quantization — they sit in positions where small gradient updates during quantization-aware training can easily toggle between ternary values, preventing the optimizer from getting trapped in suboptimal configurations.

  4. Distillation-Based Fine-Tuning Module (Stage 3) — a multi-objective training process that simultaneously quantizes the model to 1.58-bit (using the absmean quantization function and straight-through gradient estimation), trains it on downstream task data via cross-entropy loss, and distills knowledge from the task-specific FP16 teacher through two channels: logits distillation (KL divergence between teacher and student output distributions, softened with temperature τ=5.0) and multi-head attention relation distillation (KL divergence between the scaled dot-product relation matrices computed from the Q, K, and V projections at a single carefully selected transformer layer). The module outputs the final 1.58-bit model with ternary weights and 8-bit activations.

Information flow: Take an off-the-shelf FP16 LLM → surgically insert SubLN layers before attention and FFN output projections (Stage 1, architecture modified but weights unchanged) → train the modified model on 10B general-domain tokens to reshape weight distributions (Stage 2, weights updated in FP16, quantization not yet active) → simultaneously quantize weights to {-1, 0, 1} using STE, fine-tune on downstream task data, and distill from a separately task-fine-tuned FP16 copy of the original model using logits KL divergence and single-layer attention relation KL divergence (Stage 3, weights now ternary, activations 8-bit) → deploy the quantized model on CPU with BitNet-compatible kernels achieving 2.65× speedup and 10× memory reduction.

3.3 Roadmap for the Deep Dive

  • First, the quantization function itself — how weights are mapped from FP16 to {-1, 0, 1} and activations to 8-bit integers — because every subsequent stage assumes this mapping as the computational substrate and the straight-through gradient estimator as the mechanism for training through it.

  • Second, the modeling refinement (Stage 1) — the SubLN insertion and its specific positions — because this architectural change is necessary to make the quantization-compatible model stable enough for the training in Stages 2 and 3, and understanding where the normalizations go explains why they prevent activation variance explosion.

  • Third, the continual pre-training mechanism (Stage 2) — the objective, the data, the scale (10B tokens), and the hypothesized mechanism (weight distribution reshaping toward quantization boundaries) — because this is the paper's most novel diagnostic and solution contribution, and understanding it requires the context of the weight visualization in Figure 2 and the loss curves in Figure 3(a).

  • Fourth, the distillation-based fine-tuning (Stage 3) — the combined loss function (cross-entropy + logits distillation + attention relation distillation), the specific formulations of each distillation term, the MiniLM-inspired attention relation computation (Algorithm 1), and the temperature, layer selection, and coefficient choices — because this is where the final accuracy recovery happens and the design choices (single-layer distillation, Q/K/V relation matrices, specific coefficients) represent the accumulated engineering knowledge that makes the pipeline work.

  • Fifth, the full training configuration — hyperparameters, learning rate selection strategy, sequence length, batch size, hardware, and the specific dataset splits — to make the pipeline reproducible and to understand the scale of compute required relative to training-from-scratch (10B tokens vs. 4T tokens, a 400× reduction).

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and empirical methods paper whose core idea is that converting pre-trained FP16 LLMs to 1.58-bit for downstream tasks fails because of three specific, addressable mechanisms — activation instability, poor weight distribution adaptation, and residual precision loss — and that a three-stage pipeline targeting each mechanism sequentially can recover FP16-comparable performance without requiring BitNet training from scratch.

The 1.58-Bit Quantization Function and Gradient Approximation

The computational foundation of the entire pipeline is the mapping that converts full-precision weights and activations into low-bit representations. The paper adopts the quantization scheme from prior BitNet work (Ma et al., 2024, 2025), which uses two different quantization functions for weights and activations.

Weight quantization to ternary values. The weight quantization function $Q_w(W)$ maps a full-precision weight matrix $W_{\text{FP16}}$ to ternary values in {-1, 0, 1} using per-tensor absmean scaling:

Qw(W)=Δ⋅RoundClip(WFP16Δ+ϵ,−1,1)Q_w(W) = \Delta \cdot \text{RoundClip}\left(\frac{W_{\text{FP16}}}{\Delta + \epsilon}, -1, 1\right)

Δ=mean(∣W∣)\Delta = \text{mean}(|W|)

RoundClip(Y,a,b)=min⁡(max⁡(⌊Y⌉,a),b)\text{RoundClip}(Y, a, b) = \min\left(\max\left(\lfloor Y \rceil, a\right), b\right)

where $W_{\text{FP16}}$ is the full-precision weight tensor, $\Delta$ is a scalar computed as the mean of the absolute values of all elements in $W$, $\epsilon$ is a small constant added for numerical stability (preventing division by zero), $\lfloor Y \rceil$ is the nearest-integer rounding operator, and $a = -1$ and $b = 1$ are the clipping bounds.

What it computes: The function first computes a single scaling factor $\Delta$ as the average magnitude of all weights in the tensor. It then divides every weight by this scaling factor (with $\epsilon$ preventing division by zero if all weights are exactly zero), rounds the result to the nearest integer, and clips to the range [-1, 1]. Finally, it multiplies by $\Delta$ to restore scale. The net effect is that every weight is mapped to exactly one of three values: $-\Delta$, $0$, or $+\Delta$, where $\Delta$ is a per-tensor (not per-channel or per-filter) scalar computed from the data. This means all weights in a given tensor share the same scale, and the only degrees of freedom are which of the three quantization levels each weight occupies and the single scalar $\Delta$.

Why this form: Per-tensor quantization with absmean scaling is chosen over per-channel quantization because it is computationally simpler and maps directly to efficient hardware kernels (ternary matrix multiplication can be implemented with additions and subtractions rather than full multiplications). The absmean function $\Delta = \text{mean}(|W|)$ is a specific design choice: it uses the L1 mean rather than the maximum absolute value (which would be more conservative but waste dynamic range for outlier-insensitive distributions) or the standard deviation (which would center the distribution differently). The absmean places the quantization boundaries at approximately the expected magnitude of weights, which empirical evidence from prior BitNet work shows provides good signal preservation. The RoundClip operation with bounds [-1, 1] produces exactly three values after scaling, which is the defining characteristic of 1.58-bit quantization — unlike 2-bit quantization which would use four values, or 1-bit which would use two.

Activation quantization to 8-bit integers. Activations are quantized to 8-bit integers using per-token absmax scaling:

QINT8(X)=γ127⋅RoundClip(127γ+ϵXFP16,−128,127)Q_{\text{INT8}}(X) = \frac{\gamma}{127} \cdot \text{RoundClip}\left(\frac{127}{\gamma + \epsilon} X_{\text{FP16}}, -128, 127\right)

γ=max⁡(∣XFP16∣)\gamma = \max(|X_{\text{FP16}}|)

where $X_{\text{FP16}}$ is the full-precision activation tensor, $\gamma$ is the maximum absolute value of all elements in $X$ (computed per token, meaning each token's activation vector gets its own scaling factor), and the scaling and clipping map the activation values to the representable range of 8-bit signed integers, from -128 to 127.

What it computes: For each token's activation vector, the function finds the maximum absolute value $\gamma$, then scales all activations so that $-\gamma$ maps to -128 and $+\gamma$ maps to 127 (hence the factor $127/(\gamma + \epsilon)$). The result is rounded to the nearest integer and clipped to the 8-bit signed range. This means each token's activations are quantized with a per-token dynamic range — tokens with larger activations get a larger $\gamma$ and hence coarser quantization, while tokens with smaller activations get finer quantization. This is a per-token scheme (unlike the per-tensor scheme for weights) because activation magnitudes vary significantly across tokens in a sequence, and using a single per-tensor scale would either clip informative activations or waste precision on near-zero activations.

Why this form: Per-token absmax quantization is standard in LLM activation quantization because activation distributions are heavily sequence-position-dependent — early tokens often have larger magnitudes than later tokens, and attention patterns can concentrate activation mass differently across positions. The absmax function guarantees that no activation is clipped (the maximum value maps exactly to 127 or -128), which preserves the full dynamic range at the cost of potentially underutilizing the quantization bins if the activation distribution has heavy tails. The 8-bit width (rather than ternary, like the weights) is a practical choice: activations are the intermediate representations that propagate information through the network, and restricting them to 3 values would destroy the representational capacity of the model. The asymmetry — ternary weights but 8-bit activations — reflects the different roles: weights store knowledge and can be highly compressed, while activations carry input-dependent computation and need more precision.

Gradient approximation via Straight-Through Estimator (STE). The RoundClip operation in both quantization functions has zero gradient almost everywhere (it is a step function with flat regions and discontinuities at the rounding boundaries). This means standard backpropagation cannot propagate gradients through the quantization nodes. The paper adopts the Straight-Through Estimator, which simply passes the gradient through the quantization function unchanged:

∂L∂WFP16≈∂L∂Qw(W)\frac{\partial \mathcal{L}}{\partial W_{\text{FP16}}} \approx \frac{\partial \mathcal{L}}{\partial Q_w(W)}

In plain language: when computing gradients, the quantization function is treated as an identity function — the gradient of the loss with respect to the quantized weight is copied directly as the gradient with respect to the full-precision weight, ignoring the quantization operation entirely. The full-precision weights (often called "latent weights" or "shadow weights" in the quantization literature) are updated by gradient descent, but the forward pass uses their quantized versions. This means the optimizer sees the loss landscape of the quantized model (because the forward pass uses $Q_w(W)$) but computes gradients as if the model were full-precision (because the backward pass ignores quantization). This is an approximation — the true gradient would account for the quantization error — but it works in practice because the gradient steps are small and the quantization function is piecewise constant, so the "direction" of improvement for the full-precision weights tends to align with what would reduce the loss of their quantized counterparts.

Why this form: The STE is used rather than more sophisticated gradient estimators (e.g., REINFORCE, Gumbel-Softmax, or learned quantization) because it is simple, computationally cheap, and empirically effective for weight quantization. The alternative — treating quantization as a stochastic node and using score-function estimators — would introduce variance that slows training. The STE's bias (it ignores the quantization error in the gradient) is tolerable because the quantization function is deterministic and the full-precision weights are free to move continuously, with the quantization "snapping" them to discrete values only in the forward pass. This is the standard approach in the quantization literature (Bengio et al., 2013) and is used consistently across the BitNet family.

Stage 1: Modeling Refinement via SubLN Insertion

The first stage of BitDistill addresses the problem of activation variance instability in quantized transformers. The paper identifies that 1.58-bit models suffer from "excessively large activation variance, which results in optimization instability and degraded convergence," and that this problem does not exist in full-precision models because standard initialization schemes keep activation variance within a stable range. The solution is to insert additional LayerNorm-style normalization layers, called SubLN, at specific positions inside each transformer block.

The problem mechanism. In a standard transformer block, the residual stream adds the output of each sublayer (attention or FFN) to the input without re-normalizing the sum. In full precision, this works because the weight matrices are initialized to produce outputs with controlled variance, and the residual connection preserves the scale. However, when weights are quantized to {-1, 0, 1}, the output scale becomes dependent on the quantization scaling factor $\Delta$ and the structure of which weights are non-zero. A ternary weight matrix can produce outputs with variance that is very different from its full-precision counterpart — if many weights are quantized to ±1, the output magnitude is proportional to the input magnitude times the number of active weights, which may be much larger than the full-precision case where weights are typically small floating-point values. This variance can compound across layers through the residual stream, leading to activation explosion that makes training unstable.

Where SubLN layers are inserted. The paper modifies the transformer block computation by inserting SubLN layers at two specific positions, using Qwen3 as the reference architecture. The modified computations for layer $l$ are:

Yl=Xl+SubLN(Concat(heads))WoutMHSAY_l = X_l + \text{SubLN}\left(\text{Concat}(\text{heads})\right) W^{\text{MHSA}}_{\text{out}}

Xl+1=Yl+SubLN((YlWupFFN)⊙σ(YlWgateFFN))WdownFFNX_{l+1} = Y_l + \text{SubLN}\left((Y_l W^{\text{FFN}}_{\text{up}}) \odot \sigma(Y_l W^{\text{FFN}}_{\text{gate}})\right) W^{\text{FFN}}_{\text{down}}

where the multi-head attention heads are computed as:

heads={Softmax(QiKi⊤d)Vi  |  Qi=XWQ,iMHSA,  Ki=XWK,iMHSA,  Vi=XWV,iMHSA}\text{heads} = \left\{\text{Softmax}\left(\frac{Q_i K_i^\top}{\sqrt{d}}\right) V_i \;\middle|\; Q_i = X W^{\text{MHSA}}_{Q,i},\; K_i = X W^{\text{MHSA}}_{K,i},\; V_i = X W^{\text{MHSA}}_{V,i}\right\}

What each equation computes, operationally. In the first equation, $X_l$ is the input to the $l$-th transformer block (a residual stream from previous layers). The multi-head attention mechanism computes query, key, and value projections for each head, performs scaled dot-product attention, and concatenates the per-head outputs. The critical modification is $\text{SubLN}(\text{Concat}(\text{heads}))$ — a normalization layer applied to the concatenated attention outputs before the output projection matrix $W^{\text{MHSA}}_{\text{out}}$ multiplies them. The normalized and projected result is added to the residual stream $X_l$ to produce $Y_l$.

In the second equation, $Y_l$ enters the feed-forward network. The specific FFN architecture here is the gated variant (used in Qwen3), where the input is projected through an up-projection $W^{\text{FFN}}_{\text{up}}$ and a gate projection $W^{\text{FFN}}_{\text{gate}}$, the gate projection is passed through a sigmoid activation $\sigma$, and the element-wise product $\odot$ of the up-projected and gated values forms the intermediate activation. Again, the critical modification is $\text{SubLN}(\cdot)$ applied to this intermediate activation before the down-projection matrix $W^{\text{FFN}}_{\text{down}}$ multiplies it. The result is added to $Y_l$ to produce $X_{l+1}$, the input to the next layer.

The semantics of SubLN placement. SubLN is applied immediately before the output projection of each sublayer (attention and FFN), not at the sublayer input. This is a specific design choice with a concrete rationale: the output projections ($W^{\text{MHSA}}_{\text{out}}$ and $W^{\text{FFN}}_{\text{down}}$) are the weight matrices that will eventually be quantized to ternary values. By normalizing their inputs, SubLN ensures that the activations entering these quantized projections have controlled variance — they are scaled to have approximately zero mean and unit variance (or some learned affine transformation thereof). This prevents the quantized weight matrices from receiving inputs with wildly varying magnitudes, which would cause the ternary outputs to have unpredictable scale that propagates forward through the residual stream. If SubLN were placed at the sublayer input instead, it would normalize the residual stream but not protect the quantized projections from internal variance accumulation within each sublayer.

What SubLN is. The paper does not provide the explicit SubLN formula, but following the original BitNet work (Wang et al., 2023), SubLN is a LayerNorm variant: given an input vector $x$, it computes $\text{SubLN}(x) = \gamma \odot (x - \mu) / (\sigma + \epsilon) + \beta$, where $\mu$ and $\sigma$ are the mean and standard deviation of $x$, and $\gamma$ and $\beta$ are learnable scale and shift parameters. The "Sub" prefix indicates that this normalization is applied to the sub-layer output (the internal representations within a transformer block) rather than to the block input (which already has pre-normalization in modern architectures like Qwen3). This means each transformer block in the modified architecture has three normalization points: the existing pre-normalization before attention and FFN (inherited from Qwen3), plus the two new SubLN normalizations before the output projections. This is a substantial increase in normalization — effectively normalizing the representations at every point where they interact with a matrix that will be quantized — and the paper's loss curves in Figure 3(a) demonstrate that this is necessary for stable 1.58-bit training.

Why this design over alternatives. The paper could have chosen to not insert SubLN and instead rely on more careful learning rate tuning or gradient clipping to manage activation variance. The evidence in Figure 3(a) — showing a large gap in training loss between "w/ SubLN" and "w/o SubLN" for 1.58-bit models — demonstrates that these alternative approaches are insufficient: the activation variance problem is structural, not just a hyperparameter sensitivity issue. The specific choice of SubLN placement (before output projections, not before input) follows the original BitNet architecture design and has been validated in the context of training ternary models from scratch. The paper's contribution is not the SubLN technique itself but the recognition that it must be surgically inserted into off-the-shelf FP16 architectures before they can be stably quantized and fine-tuned — a practical insight that is not obvious from the BitNet literature, which assumes architecture design from scratch.

A subtle consequence of SubLN insertion. The SubLN layers add learnable parameters (the scale and shift vectors $\gamma$ and $\beta$) and increase the total parameter count slightly. These additional parameters are in FP16 during Stages 1 and 2 (the staging phases) and are quantized along with everything else during Stage 3. They also change the effective computation graph: the normalization operation is not trivially fusible with the preceding operations in all hardware backends, which could affect inference speed in practice. The paper does not report the parameter count increase or its inference impact, but the overall speed and memory numbers (2.65× faster, 10× memory reduction) include these overheads and demonstrate that the net effect is still strongly positive.

Stage 2: Continual Pre-Training as Distributional Adaptation

The second stage addresses the paper's most novel diagnostic finding: the scalability failure where the performance gap between FP16 and 1.58-bit BitNet-SFT widens with model size (growing from 13.9 to 15.3 percentage points going from 0.6B to 4B, as shown in Figure 1). The authors hypothesize that directly fine-tuning a model whose weights are distributed approximately Gaussian (the typical distribution after FP16 pre-training) on a small number of downstream task tokens is insufficient to adapt the weights to a distribution that is favorable for ternary quantization. The continued pre-training stage provides a larger volume of general-domain training data to achieve this adaptation before task-specific optimization begins.

The training objective. The continued pre-training is a standard autoregressive language modeling objective on a general-domain corpus:

LCT=−1N∑i=1N∑t=1Tilog⁡Pθ(ci,t∣ci,<t)\mathcal{L}_{\text{CT}} = -\frac{1}{N} \sum_{i=1}^{N} \sum_{t=1}^{T_i} \log P_\theta(c_{i,t} \mid c_{i,<t})

where $N$ is the number of documents in the continued pre-training corpus, $T_i$ is the number of tokens in document $i$, $c_{i,t}$ is the $t$-th token of document $i$, $c_{i,<t}$ is the prefix of document $i$ up to but not including position $t$, and $P_\theta(\cdot \mid \cdot)$ is the probability distribution over the next token produced by the model with parameters $\theta$.

What it computes: For each position $t$ in each document, the model takes the preceding tokens as context, produces a probability distribution over the vocabulary for the next token, and is penalized by the negative log-probability assigned to the actual next token $c_{i,t}$. This is the standard teacher-forcing objective for language modeling: the model learns to predict each token given all previous tokens, averaged over all positions in all documents. The loss is a scalar representing the average per-token cross-entropy.

Why this form: This is the standard pre-training objective for LLMs and is chosen because it provides a dense training signal — every token position contributes a gradient update — and it is domain-agnostic, encouraging the model to adapt its weight distribution to the general statistical structure of language without overfitting to any specific downstream task. The alternative of directly fine-tuning on downstream data (which is what BitNet-SFT does) provides a much sparser signal because downstream datasets are typically small (tens of thousands of examples, vs. billions of tokens in the continued pre-training corpus), and the task-specific distribution may not cover the full vocabulary and syntactic patterns needed to reshape all weights. Using a general-domain corpus ensures that all parts of the model receive gradient updates.

The data and scale. The continued pre-training uses only 10 billion tokens sampled from the FALCON corpus (Penedo et al., 2023), which is a filtered web-text dataset. The paper emphasizes that this is "virtually negligible" compared to the cost of pretraining a 1.58-bit BitNet from scratch, which requires approximately 4 trillion tokens (Ma et al., 2025) — a 400× reduction. At this stage, the model is still in FP16 (quantization has not yet been applied), so the continued pre-training is standard full-precision training with the SubLN-modified architecture. This means Stage 2 can use standard training infrastructure without quantization-aware kernels.

The hypothesized mechanism: weight distribution reshaping. The paper provides a specific mechanistic hypothesis for why continued pre-training helps, supported by the weight visualizations in Figure 2. The argument proceeds in several linked claims:

  1. Initial state (Gaussian weights). When an FP16 LLM is loaded, its weight distributions across different projection matrices (attention Q, K, V, O and FFN gate, up, down) are approximately Gaussian or Gaussian-like, centered near zero with moderate variance. This is a natural consequence of standard initialization and regularization during pre-training. The top two rows of Figure 2 show the weight distributions of a BitNet trained from scratch: the FP16 versions (before quantization) have a distinctive shape where mass is concentrated near the quantization boundaries (the transitions between -1 and 0, and between 0 and 1), with relatively less mass in the centers of the quantization bins. This shape develops naturally during from-scratch BitNet training because the optimizer learns to place weights where small perturbations can most easily change the ternary value — at the decision boundaries of the quantization function.

  2. The adaptation problem. When the same model is directly loaded from FP16 pre-training and immediately quantized and fine-tuned (BitNet-SFT), the weights start in the Gaussian distribution, not the boundary-concentrated distribution. With only downstream-task data (limited number of tokens), the optimizer does not have enough signal to move all weights from their initial positions (distributed across the full range of the Gaussian) to positions near the quantization boundaries. The result is that many weights get "stuck" in suboptimal configurations: they are far from the boundaries, meaning that gradient steps are too small to toggle their ternary values, and the model cannot flexibly adapt to the downstream task. This explains the scalability failure: larger models have more weights to adapt, and with a fixed downstream data budget, the adaptation per weight is lower for larger models, hence the gap grows.

  3. The solution (continued pre-training). By training the SubLN-modified FP16 model on 10B tokens of general-domain data, the optimizer gets enough signal to move the weight distribution toward the boundary-concentrated shape. The bottom two rows of Figure 2 show this visually: after continued pre-training, the weight distributions of the BitDistill model closely resemble those of the BitNet trained from scratch — mass is concentrated near the quantization boundaries, with less mass in the bin centers. This means that when quantization is applied in Stage 3, the weights are already positioned where they can effectively toggle between ternary values in response to gradient signals from the downstream task. The continued pre-training serves as a distributional initialization step: it gets the weights into the right region of the parameter space for ternary optimization, after which the downstream fine-tuning can do the precise task-specific adjustment.

  4. Why boundary concentration helps. The paper provides a further mechanistic explanation: "the unique distribution concentrates more weights near the transition boundaries between 0 and -1 as well as between 0 and 1. Such placements allow the quantized values to shift more frequently with small gradient steps, thereby enhancing the 1.58-bit BitNet's ability to fit downstream data and reducing the risk of being trapped in suboptimal local minima." In operational terms, if a weight's full-precision value is near 0.5 (the boundary between ternary 0 and ternary 1), a small gradient step that pushes it to 0.51 will cause the quantized value to flip from 0 to 1 — a large change in the model's function with a small parameter update. Conversely, if a weight starts at 0.1, a gradient step of the same magnitude might push it to 0.15, which still rounds to 0 — no effective change. The boundary-concentrated distribution maximizes the probability that a given gradient update will actually change the ternary weight, making the optimization landscape less "flat" and reducing the risk of getting stuck where no ternary weight changes occur despite ongoing gradients.

The empirical support. The paper provides several pieces of evidence for this mechanism. Figure 3(a) shows that SubLN insertion alone (without continued pre-training) improves training loss but does not close the performance gap — suggesting that activation stability and weight distribution adaptation are distinct problems that both need solving. Table 5 shows that continued pre-training alone (without distillation, i.e., Stage 1 + Stage 2 but no Stage 3) takes the MNLI accuracy from 74.09 (BitNet-SFT baseline, no stages) to 86.73 — recovering most of the gap to the FP16 baseline of 88.01. This demonstrates that the distributional adaptation is the single largest contributor to performance recovery, with distillation (Stage 3) providing the final 1.4 percentage points. The weight visualizations in Figure 2 provide the mechanistic visual evidence.

Why this is a key insight. The continued pre-training stage is the paper's most original contribution to understanding. Prior work on quantizing pre-trained models focused on PTQ (which doesn't involve training) or QAT directly on task data (which the paper shows fails). The recognition that a brief general-domain training phase is necessary to reshape the weight distribution before quantization identifies a previously undocumented adaptation bottleneck. This is not "more training data helps" — it is a specific claim about what kind of training data (general-domain, not task-specific) and what it accomplishes (distribution reshaping, not task learning) and why it matters at scale (to prevent the scalability failure). This insight would not be accessible from a pure engineering approach and requires the combination of careful diagnostics and weight visualization that the paper presents.

Stage 3: Distillation-Based Fine-Tuning

The third stage applies quantization-aware training on the downstream task data, combining three loss terms to simultaneously fit the task, preserve the FP16 teacher's output distribution, and preserve the teacher's internal attention patterns. This stage takes the SubLN-modified, distribution-adapted FP16 model from Stage 2, applies the quantization functions from the preliminaries, and trains end-to-end on the downstream dataset with the combined objective.

The total loss function. The training objective for Stage 3 is:

L=LCE+λLLD+γLAD\mathcal{L} = \mathcal{L}_{\text{CE}} + \lambda \mathcal{L}_{\text{LD}} + \gamma \mathcal{L}_{\text{AD}}

where $\mathcal{L}_{\text{CE}}$ is the standard cross-entropy loss on the downstream task, $\mathcal{L}_{\text{LD}}$ is the logits distillation loss, $\mathcal{L}_{\text{AD}}$ is the multi-head attention distillation loss, and $\lambda$ and $\gamma$ are scalar coefficients controlling the relative weight of each distillation term.

The cross-entropy loss $\mathcal{L}_{\text{CE}}$. This is the standard supervised fine-tuning objective on the downstream dataset:

LCE=−1N∑i=1N∑t=1∣yi∣log⁡Pθ(yit∣xi)\mathcal{L}_{\text{CE}} = -\frac{1}{N} \sum_{i=1}^{N} \sum_{t=1}^{|y_i|} \log P_\theta(y_i^t \mid x_i)

where $N$ is the number of training examples, $|y_i|$ is the length of the target sequence for example $i$, $y_i^t$ is the $t$-th token of the target, $x_i$ is the input, and $P_\theta(\cdot \mid \cdot)$ is the student model's probability distribution. This is the standard maximum-likelihood objective: the model is trained to assign high probability to the correct output tokens given the input.

What it computes: the standard teacher-forcing loss for sequence generation — for classification tasks (MNLI, QNLI, SST-2), the target $y_i$ is typically a single label token or short phrase, and the loss reduces to per-example negative log-likelihood of the correct class; for summarization (CNNDM), the target is a multi-token summary, and the loss sums over all summary tokens.

Why this form: it provides the primary task-specific training signal and is the default objective for fine-tuning. Without this term, the model would only receive distillation signals and might not learn to perform the task at all — distillation transfers how to think about the task but does not directly optimize task performance.

Logits Distillation

The logits distillation term encourages the student's output probability distribution to match the teacher's softened output distribution:

LLD=1N∑i=1NDKL(PθFP16(yi∣xi)  ∥  Pθ1.58-bit(yi∣xi))\mathcal{L}_{\text{LD}} = \frac{1}{N} \sum_{i=1}^{N} D_{\text{KL}}\left(P^{\text{FP16}}_\theta(y_i \mid x_i) \;\middle\|\; P^{\text{1.58-bit}}_\theta(y_i \mid x_i)\right)

Pθ(⋅)(y∣x)=exp⁡(zy(x;θ)/τ)∑y′exp⁡(zy′(x;θ)/τ)P^{(\cdot)}_\theta(y \mid x) = \frac{\exp(z_y(x; \theta) / \tau)}{\sum_{y'} \exp(z_{y'}(x; \theta) / \tau)}

where $D_{\text{KL}}(\cdot \|\cdot)$ is the Kullback-Leibler divergence, $P^{\text{FP16}}_\theta$ is the teacher's softened probability distribution over output tokens, $P^{\text{1.58-bit}}_\theta$ is the student's softened probability distribution, $z_y(x; \theta)$ is the unnormalized logit (pre-softmax score) for token $y$ given input $x$ under model $\theta$, and $\tau$ is the temperature parameter controlling the softness of both distributions.

What it computes: For each training example, both the FP16 teacher and the 1.58-bit student produce logits (unnormalized scores) over the output vocabulary. These logits are divided by the temperature $\tau$ (making the distributions softer — higher $\tau$ means more uniform distributions, lower $\tau$ means peakier distributions), passed through softmax, and then compared via KL divergence. The KL divergence $D_{\text{KL}}(P^{\text{teacher}} \| P^{\text{student}})$ measures how much information is lost when using the student's distribution to approximate the teacher's: it is the expected log-ratio of probabilities under the teacher, $\sum_y P^{\text{teacher}}(y) \log(P^{\text{teacher}}(y) / P^{\text{student}}(y))$. The loss is averaged over all $N$ training examples.

Why this form and why temperature $\tau = 5.0$: Logits distillation (also called "dark knowledge distillation," from Hinton et al., 2015) transfers not just the teacher's top-1 prediction but the full distribution over the output space, which encodes information about which outputs are "almost correct" or "plausible." For example, in sentiment classification, the teacher might assign 0.7 probability to "positive" and 0.25 to "neutral" and 0.05 to "negative" — the student learns that "neutral" is a more reasonable alternative than "negative," which is richer information than just the label "positive." The temperature $\tau$ controls the softness: at $\tau = 1$, the distribution matches the standard softmax; at higher $\tau$, the distribution is more uniform, giving more weight to non-top predictions and exposing more of the teacher's internal uncertainty structure. The paper uses $\tau = 5.0$, which is a moderately high temperature — it substantially softens the distribution so that the student learns from the relative ordering of many tokens, not just the most likely one. This is particularly important for 1.58-bit models because their limited weight precision may make it hard to exactly replicate the teacher's peaky distribution; a softer target provides a smoother optimization landscape. The KL divergence is chosen over cross-entropy because it is asymmetric: it penalizes the student for placing low probability on tokens the teacher considers likely, but is less punitive when the student places high probability on tokens the teacher considers unlikely — this focuses the distillation on preserving the teacher's "knowledge" (which tokens are good) rather than exactly matching the shape of the distribution.

Coefficient $\lambda$ settings. For classification tasks, $\lambda = 10$; for summarization, $\lambda = 1$. This means classification places much higher relative weight on logits distillation (10× the cross-entropy weight) compared to summarization (equal weight). The paper does not provide an explicit justification for this difference, but it is consistent with the nature of the tasks: classification has a small output space (typically 2–5 classes for the GLUE datasets), so the teacher's full distribution is information-dense and the student can easily match it; summarization has a vocabulary-sized output space (tens of thousands of tokens), and the teacher's per-token distribution is very peaky (most tokens have near-zero probability), making the KL divergence less informative per position and potentially conflicting with the cross-entropy signal that must get the exact token right.

Multi-Head Attention Distillation

The attention distillation term transfers the teacher's internal attention patterns to the student, going beyond output-level matching to align the intermediate representations. This is based on the MiniLM approach (Wang et al., 2020, 2021), which distills the relation matrices derived from the query, key, and value projections rather than the attention weights directly.

The attention relation matrix. The core idea is to compute, for a given projection matrix (Q, K, or V), the pairwise similarities between all token positions in the sequence — a relation matrix of shape $|x| \times |x|$ — and then distill these relation matrices from teacher to student. The relation matrix for projection type $j$ at layer $i$ for model variant $m$ (FP16 or 1.58-bit) is:

Ri,j,a,tm=Softmax(Ai,j,a,tm(Ai,j,a,tm)⊤dr)R^{m}_{i,j,a,t} = \text{Softmax}\left(\frac{A^{m}_{i,j,a,t} (A^{m}_{i,j,a,t})^\top}{\sqrt{d_r}}\right)

where $A^{m}_{i,j,a,t}$ is the $t$-th row of the projected representations for attention head $a$ (after splitting heads and projecting through Q, K, or V), with shape $|x| \times d_r$ (sequence length by head dimension), and $d_r$ is the head dimension used for computing the relation matrix (which may differ between teacher and student — the teacher uses $d_r$ and the student uses $d'_r$). The result $R^{m}_{i,j,a,t}$ is a probability distribution over the $|x|$ token positions, representing how strongly position $t$ relates to every other position according to projection $j$ at head $a$.

What it computes, operationally: For each attention head and each projection type (Q, K, V), take the projected representations — a matrix where rows correspond to token positions and columns to feature dimensions — compute the dot-product similarity between every pair of rows (the $A A^\top$ term), divide by $\sqrt{d_r}$ for scaling (analogous to the scaling in standard attention), and apply softmax along the last dimension to get a valid probability distribution per row. The result is, for each token position, a distribution indicating which other token positions have similar Q, K, or V projections. This captures the structural relationships in the representation space: two tokens with similar query projections will have high similarity in the Q relation matrix, indicating they "attend to similar things" from the perspective of the attention mechanism.

Why Q, K, and V rather than attention weights: The MiniLM approach uses the raw Q, K, and V projections rather than the final attention weights ($\text{Softmax}(QK^\top/\sqrt{d})V$) because the relation matrices from Q, K, and V each capture different aspects of the representation. The Q relation matrix captures which tokens have similar "query patterns" (which tokens ask similar questions); the K relation matrix captures which tokens have similar "key patterns" (which tokens provide similar information); the V relation matrix captures which tokens have similar "value patterns" (which tokens contribute similar content). Distilling all three provides a richer supervisory signal than distilling only attention weights, which combine Q and K (through their dot product) but lose information about the V space. This richer signal is particularly valuable for 1.58-bit models, which have limited representational capacity and benefit from learning the fine-grained structure of the teacher's internal representations.

The attention distillation loss. The full loss aggregates KL divergences over the selected layers, projection types, attention heads, and token positions:

LAD=1∣Υ∣∑i=1∣Υ∣∑j=1∣Φ∣αi1Ar∣x∣∑a=1Ar∑t=1∣x∣DKL(Ri,j,a,tFP16  ∥  Ri,j,a,t1.58-bit)\mathcal{L}_{\text{AD}} = \frac{1}{|\Upsilon|} \sum_{i=1}^{|\Upsilon|} \sum_{j=1}^{|\Phi|} \alpha_i \frac{1}{A_r |x|} \sum_{a=1}^{A_r} \sum_{t=1}^{|x|} D_{\text{KL}}\left(R^{\text{FP16}}_{i,j,a,t} \;\middle\|\; R^{\text{1.58-bit}}_{i,j,a,t}\right)

where $\Upsilon$ is the set of layers selected for distillation (with $|\Upsilon| = 1$ in the recommended configuration — see below), $\Phi = \{Q, K, V\}$ is the set of projection types (so $|\Phi| = 3$), $\alpha_i$ are layer-specific coefficients (set to 1.0 for all experiments), $A_r$ is the number of attention heads used for computing the relation matrices (which may differ from the model's actual number of heads — the split_heads parameter in Algorithm 1), and $|x|$ is the sequence length.

What it computes: For each selected layer, for each of the three projection types (Q, K, V), for each attention head, for each token position, compute the KL divergence between the teacher's relation distribution and the student's relation distribution. The KL divergence for a single token position measures how different the student's pattern of similarities to other tokens is from the teacher's — it penalizes the student for assigning low similarity to token pairs that the teacher considers highly similar. The losses are averaged over all heads, positions, projection types, and selected layers, producing a single scalar.

Why this form (KL divergence on relation distributions): The KL divergence is a natural measure of distributional divergence, and the relation matrices are valid probability distributions (each row sums to 1 after softmax). Using KL divergence means the distillation penalizes the student most when it places low probability on token pairs the teacher considers highly related — it preserves the teacher's notion of which tokens are similar. Alternative choices like mean-squared error on the raw similarity scores would be scale-sensitive and would not naturally treat the relations as distributions. The softmax normalization in the relation matrix computation ensures that the distillation focuses on relative similarity patterns rather than absolute similarity magnitudes, which may differ between FP16 and 1.58-bit models due to precision limitations.

Single-layer distillation ($|\Upsilon| = 1$). The paper makes a specific design recommendation: distill attention relations at only a single layer rather than all layers. The justification, stated in Section 3.3, is that this "confers greater optimization flexibility to the 1.58-bit student BitNet" — forcing the student to match the teacher's attention patterns at every layer would over-constrain the optimization, preventing the student from developing its own internal representations that compensate for precision loss. By distilling only one layer, the student gets a strong structural signal at one point in the network while retaining freedom at other layers to adapt to the ternary weight constraints. Figure 3(b) empirically validates this: distilling from a single layer outperforms distilling from all layers, and the choice of which single layer matters significantly, with later layers (closer to the output) performing better than earlier layers. The paper's finding that "layers located in the later stages of the model tend to deliver better distillation performance" is consistent with the intuition that later layers capture higher-level semantic patterns that are more directly relevant to the task output, while early layers capture low-level syntactic patterns that the student might learn adequately from the task data alone.

The split_heads parameter. Algorithm 1 introduces a split_heads parameter that controls how the Q, K, V projections are reshaped before computing relation matrices. Specifically, the projected representations (shape [B, num_heads, L, head_dim], where B is batch size and L is sequence length) are reshaped to [B, split_heads, L, D], where D = heads * head_dim / split_heads. This means the relation matrices are computed using a different number of "effective heads" than the model's actual attention heads. The paper sets split_heads = 1 in the pseudo-code (the default value), which means all heads are merged into a single effective head per projection type — the relation matrix is computed from the full concatenated Q, K, or V space. This is consistent with the MiniLM approach, where using fewer (or one) effective heads simplifies the distillation target while still capturing the overall relational structure. The dimensionality $d_r$ (or $d'_r$) in Eq. 12 is then $D$, the size of the effective head dimension after reshaping.

Algorithm 1 details. The pseudo-code in Algorithm 1 provides the concrete implementation:

  1. Inputs: student_states and teacher_states, each of shape [3, B, num_heads, seq_len, head_dim], containing the Q, K, and V projections (indexed by the first dimension) for a single distillation layer. distill_layer is the index of the layer (used for selecting which layer's states to distill). split_heads controls the head merging.

  2. Head reshaping: For each projection type (loop over range(3) for Q, K, V), the states are transposed and reshaped from [B, num_heads, L, head_dim] to [B, L, split_heads, D] (via transpose to move heads next to batch, then reshape to merge original heads into split_heads effective heads with dimension D), then transposed again to [B, split_heads, L, D]. Both student and teacher states are L2-normalized along the last dimension (F.normalize(..., dim=-1)) so that the dot product becomes cosine similarity.

  3. Relation matrix computation: The relation matrix is torch.matmul(s_values, s_values.transpose(-2, -1)), which computes $A A^\top$ — for each effective head, the [L, D] matrix is multiplied by its transpose [D, L] to produce an [L, L] matrix of pairwise dot products. This is done for both student and teacher.

  4. Temperature scaling and softmax: The relation matrices are divided by a temperature (the pseudo-code uses variable temperature, which is not explicitly defined in the paper text but corresponds to $\sqrt{d_r}$ or $\sqrt{d'_r}$ from Eq. 12), reshaped to [-1, L] (flattening batch and head dimensions), and passed through softmax along the last dimension to produce probability distributions over positions. The student probabilities are clamped to a minimum of 1e-8 before computing KL divergence, ensuring numerical stability (log of zero is undefined).

  5. KL divergence: F.kl_div(torch.log(s_prob), t_prob, reduction="batchmean", log_target=False) computes the KL divergence with the student log-probabilities as input and teacher probabilities as target. The reduction="batchmean" averages over all rows in the flattened batch-heads-times-positions dimension.

Why L2 normalization before computing relations: The F.normalize step converts the dot product into cosine similarity (since both vectors have unit norm after normalization, their dot product equals their cosine similarity). This is important because the teacher and student may have different activation magnitudes (the student's ternary weights produce outputs at different scales than the teacher's FP16 weights). By normalizing, the relation matrix captures only the directional similarity of representations, not their magnitudes, making the distillation target invariant to scale differences introduced by quantization. The split_heads reshaping then computes relations in the high-dimensional space of the merged heads, which captures cross-head interactions that would be lost if relations were computed per-head and then averaged.

Coefficient $\gamma$ settings. For classification, $\gamma = 1 \times 10^5$ — an extremely large weight, making attention distillation the dominant loss term. For summarization, $\gamma = 1 \times 10^3$ — still large but two orders of magnitude smaller. This dramatic difference likely reflects the fact that classification tasks have very short output sequences (often a single token), so the cross-entropy loss provides limited per-example signal, and the attention distillation (which operates over the full input sequence) becomes the primary learning signal. Summarization has longer target sequences and a more distributed cross-entropy signal, so attention distillation is proportionally less critical. The paper does not provide ablation results for these specific coefficient values, but they represent the outcome of the "greedy search strategy to select the optimal learning rate and training epochs" mentioned in the training settings.

Training Configuration and Hyperparameters

Hardware and base models. All training is conducted on servers with 8× AMD Mi300X GPUs. The primary base models are the Qwen3 series at scales 0.6B, 1.7B, and 4B parameters. Additional experiments use Qwen2.5-0.5B and Gemma3-1B to test robustness across model families.

Data and preprocessing. The maximum training sequence length is fixed to 512 tokens across all stages and tasks. The batch size is set to 32. For both classification and summarization, a "greedy search strategy" is used to select the optimal learning rate and training epochs — meaning that multiple configurations are tried and the best-performing one on a validation split is selected. The paper does not report the specific learning rates found by this search, which is a limitation for reproducibility.

Continued pre-training data (Stage 2). Only 10 billion tokens are used, sampled from the FALCON corpus (RefinedWeb). The paper emphasizes that this is approximately 0.25% of the 4 trillion tokens required for training a BitNet from scratch. The data is general web text, not task-specific.

Distillation hyperparameters (Stage 3). The temperature for logits distillation is $\tau = 5.0$. For classification tasks, $\lambda = 10$ (logits distillation weight) and $\gamma = 1 \times 10^5$ (attention distillation weight). For summarization, $\lambda = 1$ and $\gamma = 1 \times 10^3$. The attention distillation coefficients $\alpha_i$ are set to 1.0 for all layers. Attention distillation is applied at a single layer (the specific layer index is determined empirically per model — Figure 3(b) shows that later layers perform better, but the paper does not report the final selected layer indices for each model scale).

Inference settings. For evaluation, top-p sampling is set to 1.0 (effectively disabled, meaning all tokens with non-zero probability are considered) and temperature is set to 0 (greedy decoding). For summarization, the maximum generation length is set to 4096 tokens. Inference speed is measured on CPU with 16 threads, reporting tokens per second.

Teacher model preparation. The FP16 teacher is the same base model (e.g., Qwen3-0.6B) fine-tuned on the downstream task using standard supervised fine-tuning (denoted FP16-SFT in the tables). This means the teacher and student start from the same pre-trained checkpoint, ensuring that the distillation transfers task-specific knowledge acquired during FP16 fine-tuning, not general pre-training knowledge that both already share. The paper also experiments with using larger teachers (Qwen3-1.7B and 4B FP16 models as teachers for the 0.6B 1.58-bit student), as shown in Figure 3(c), demonstrating that larger teachers can push student performance beyond what the same-sized teacher achieves — the 0.6B BitDistill student with a 4B teacher outperforms the 0.6B FP16 baseline.

Reproducibility considerations. Several hyperparameters are not reported: the specific learning rate for each model configuration, the number of training epochs, the optimizer choice (likely AdamW given standard practice, but not confirmed), the learning rate schedule, and weight decay settings. The paper also does not specify how the 10B tokens are sampled from FALCON (random sampling, specific subset, deduplication) or the exact layer indices chosen for attention distillation in each experiment. These omissions limit exact reproducibility but are common in systems papers where the focus is on the framework design rather than exhaustive hyperparameter reporting.

Summary of Design Choices and Their Justifications

  • Three-stage pipeline (SubLN insertion → continued pre-training → distillation fine-tuning): Each stage targets a specific failure mode of direct QAT (activation instability, poor weight distribution adaptation, residual precision loss), and the sequential design ensures that each problem is solved before the next is introduced — SubLN stabilizes training before distribution adaptation begins, distribution adaptation positions weights for effective quantization before task-specific distillation begins.

  • SubLN before output projections, not before sublayer inputs: Normalizes the inputs to the weight matrices that will be quantized, directly preventing activation variance explosion at the point where ternary computation happens, rather than indirectly through residual stream normalization.

  • 10B tokens of general-domain continued pre-training rather than more task data or from-scratch pretraining: General-domain data provides dense per-token signal to reshape all weights, unlike sparse task-specific data; the 10B scale is sufficient for distribution adaptation but orders of magnitude cheaper than from-scratch BitNet training (400× less data).

  • Logits distillation with $\tau = 5.0$: A moderately high temperature softens the teacher's output distribution to expose more of the dark knowledge (relative ordering of non-top predictions), providing a smoother optimization landscape for the capacity-constrained 1.58-bit student.

  • Attention distillation on Q, K, and V relation matrices rather than attention weights: Captures three distinct aspects of the teacher's internal representations (query patterns, key patterns, value patterns), providing a richer supervisory signal than attention weights alone, which is critical for the severely capacity-constrained ternary student.

  • Single-layer attention distillation rather than all-layer: Gives the student optimization flexibility at most layers to develop quantization-compatible representations, while still transferring structural knowledge at one strategically chosen layer (typically a late layer closer to the task output).

  • Very high $\gamma$ for attention distillation in classification ($\gamma = 10^5$): Classification tasks have short output sequences (often single-token), making the cross-entropy signal sparse; the attention distillation over the full input sequence becomes the primary learning signal, justifying the extreme coefficient weight.

  • Per-tensor absmean weight quantization and per-token absmax activation quantization: Follows the established BitNet quantization scheme, chosen for hardware efficiency (ternary weights enable addition/subtraction-based matrix multiplication) and because these specific scaling functions have been empirically validated in prior BitNet work.

4. Key Insights and Innovations

Innovation 1: The Scalability Failure in Direct 1.58-Bit QAT Is an Identifiable, Surgically Fixable Distributional Problem — Not a Fundamental Limitation of Ternary Weights

What makes this distinctive at the idea level. The paper's most important conceptual move is diagnosing why direct quantization-aware fine-tuning of pre-trained FP16 models to 1.58-bit fails, and showing that the failure mechanism is not what the field might have assumed. The naive assumption would be that 1.58-bit precision simply cannot represent the knowledge in a full-precision model — that the capacity loss is fundamentally irrecoverable. The paper refutes this by demonstrating that a BitNet trained from scratch at 1.58-bit achieves competitive performance (Ma et al., 2024, 2025), proving the architecture is capable. The question the paper asks is narrower and sharper: why does the conversion path fail, and can it be fixed without retraining from scratch?

The diagnostic insight is that the performance gap exhibits poor scalability — it widens from 13.9 percentage points at 0.6B to 15.3 at 4B (Figure 1). This is counterintuitive. Under a pure capacity-limitation hypothesis, one might expect the gap to stay roughly constant with model size (both FP16 and 1.58-bit benefit from more parameters) or even narrow (larger models have more redundancy and can better absorb quantization error). The fact that it widens instead points to a dynamic, not static, bottleneck — something about the training process becomes less effective per parameter as scale increases.

The paper's mechanistic explanation, supported by the weight visualizations in Figure 2, is that the problem is distributional mismatch: FP16 pre-trained weights are approximately Gaussian-distributed (mass spread across a continuous range), while effective ternary optimization requires weights to concentrate near the quantization decision boundaries (the transitions between -1 and 0, and between 0 and 1). With a fixed downstream dataset, the optimizer has a fixed number of gradient steps to move weights from their initial positions. In a larger model, the per-parameter number of gradient updates is lower (the same data budget must update more weights), so fewer weights successfully migrate from the Gaussian center regions to the boundary regions. The scalability failure is thus a ratio problem — the ratio of available gradient signal to number of weights that need distributional adaptation — not an inherent representational limitation.

This is a fundamental reframing. It transforms the conversion problem from "can ternary weights represent the knowledge?" (structural) to "can the optimizer find the ternary representation before running out of data?" (dynamical). The former is a property of the architecture; the latter is a property of the training recipe. This distinction is what makes the solution possible: if the problem were structural, no amount of clever training would fix it; since it is dynamical, a modest amount of additional data — carefully chosen to provide general-domain signal rather than task-specific signal — can resolve it.

Contrast with prior assumptions. Prior work on quantizing pre-trained LLMs focused either on PTQ (GPTQ, AWQ), which avoids training entirely and accepts a performance hit, or on QAT directly on task data (EfficientQAT, LLM-QAT), which treats the problem as one of recovering from precision loss during task-specific optimization. Neither framework considered the possibility that the bottleneck is a pre-training distribution adaptation gap. The PTQ approach implicitly assumes weights are already in reasonable positions and only need rounding; the QAT approach assumes the task signal is sufficient to drive recovery. This paper shows both assumptions break at 1.58-bit with limited task data, and identifies a third category of problem — distributional initialization — that neither PTQ nor standard QAT addresses.

Significance beyond raw performance. This diagnostic is the intellectual foundation that makes the whole pipeline coherent. Without it, Stage 2 (continued pre-training on 10B general-domain tokens) would appear to be an ad hoc "add more data" trick rather than a principled intervention. With it, Stage 2 is recognized as a distributional adaptation phase — fundamentally different from either pre-training from scratch or task-specific fine-tuning — that targets a specific, identified bottleneck. The concept of a distributional initialization gap that grows with model scale is a new diagnostic category for the quantization literature, and it suggests that other extreme compression regimes (binary networks, sparse networks) might suffer from analogous dynamical bottlenecks that are invisible under capacity-limitation analyses.

The paper's evidence for this diagnosis comes from multiple angles converging: the scalability trend in Figure 1 (quantitative), the weight distribution visualizations in Figure 2 (visual/structural), the ablation in Table 5 showing that continued pre-training alone (without distillation) recovers most of the accuracy gap — 74.09 → 86.73 on MNLI for the 0.6B model, closing ~85% of the gap to the FP16 baseline of 88.01 (quantitative mechanism attribution), and the training loss curves in Figure 3(a) showing that SubLN addresses a separate instability problem (ruling out the alternative hypothesis that activation variance alone explains the gap). This multi-source convergence makes the distributional-mismatch diagnosis robust: if continued pre-training were merely providing more task-relevant data, it wouldn't explain why 10B tokens of general corpus outperform direct task-specific fine-tuning with the same architecture.

Incremental or fundamental? This is a fundamental diagnostic contribution that changes how the field should think about extreme quantization of pre-trained models. It is not a new algorithm or a new architecture — it is a new problem category with a clearly identified mechanism and a validated intervention strategy. The fact that the intervention (distributing 10B general-domain tokens before task training) is simple does not diminish the insight; it confirms that the solution was non-obvious enough that prior work missed it while trying more complex approaches.

Innovation 2: Multi-Head Attention Relation Distillation as a Precision-Robust Knowledge Transfer Mechanism for Extreme Quantization

What makes this distinctive at the idea level. The paper demonstrates that distilling the teacher's attention relation matrices — pairwise similarity patterns computed from raw Q, K, and V projections — is a particularly effective form of knowledge transfer when the student has severely constrained representational capacity (ternary weights). This builds on MiniLM (Wang et al., 2020, 2021) but makes a specific, non-obvious adaptation: applying it at only a single layer, with extremely high loss weight (γ = 10^5 for classification), and using it in combination with logits distillation and continued pre-training for the specific purpose of recovering from 1.58-bit precision loss.

The conceptual insight is that the Q, K, and V relation matrices capture something fundamentally different from what logits distillation captures, and that this difference is especially valuable under extreme quantization. Logits distillation transfers the teacher's output-level knowledge — which tokens are plausible outputs. Attention relation distillation transfers the teacher's internal structural knowledge — how the model organizes its representational space to arrive at those outputs. For a capacity-constrained ternary student that cannot perfectly replicate all the teacher's internal computations, knowing how the teacher structures its thinking (which tokens are similar in query space, which in key space, which in value space) is more informative than knowing only what the teacher concludes. The relation matrices provide a blueprint for the internal geometry; the logits provide the final answer. The student can follow the blueprint even if its materials (ternary weights) are cruder, arriving at the same final answer through a structurally similar path.

Contrast with prior distillation for quantization. Prior work using distillation for quantized LLMs — TSLD (Kim et al., 2023) with layer-to-layer distillation, BitDistiller (Du et al., 2024) with self-distillation — primarily focused on output-level or hidden-state-level matching. Attention-level distillation has been explored in full-precision model compression (the MiniLM family) but was not, prior to this work, validated as a key ingredient for extreme (1.58-bit) quantization of pre-trained models for downstream tasks. The paper's contribution is not inventing attention relation distillation, but recognizing that it is disproportionately valuable in the specific context of ternary-weight conversion because it provides dense structural supervision that compensates for the representational poverty of three-valued weights.

The single-layer distillation choice is particularly telling. The paper's finding (Figure 3b) that distilling a single layer outperforms distilling all layers challenges a natural intuition: one might think that more layers of distillation provide more supervision, hence better performance. The counterintuitive result — confirmed empirically — suggests that over-constraining the student's internal representations forces it into a representational regime that ternary weights cannot sustain, creating a conflict between the distillation objectives and the model's actual capacity. By distilling only one layer, the student gets a strong structural "anchor" at one point in the network while retaining freedom to develop quantization-compatible representations elsewhere. This is not a generic distillation trick — it is a capacity-aware distillation strategy that acknowledges the student's representational limitations rather than fighting them.

The layer-choice result — later layers perform better (Figure 3b) — provides further insight: the high-level semantic structure captured in late layers is more transferable under extreme quantization than the low-level syntactic structure in early layers, perhaps because late-layer representations are more abstract and compressible, while early-layer representations require finer-grained distinctions that ternary weights struggle to preserve. The student learns adequate low-level representations from the task data and cross-entropy signal, but benefits most from structural guidance at the semantic level.

Significance beyond raw performance. This finding establishes a principle for distillation under extreme compression: match the student's capacity, don't overwhelm it. The single-layer, high-weight attention distillation is not just a hyperparameter choice — it reflects an understanding that the distillation target should provide maximum structural information with minimum representational constraint. This principle likely generalizes to other extreme compression regimes (binary networks, very low-rank approximations) where the student's internal representational capacity is a binding constraint.

The evidence is clear but scope-limited. Table 6 shows that logits distillation alone (+0.6 percentage points over no-distillation baseline) and attention distillation alone (+0.9 points) both help, but their combination (+1.4 points) is synergistic, confirming they transfer complementary knowledge. Figure 3(b) validates the single-layer strategy. However, the paper does not explore why attention distillation helps so dramatically for classification (γ = 10^5) versus summarization (γ = 10^3), beyond the sequence-length explanation, and does not test whether the Q/K/V relation approach is specifically better than distilling attention weights directly — a potentially important ablation.

Incremental or fundamental? This is an incremental contribution in the sense that the technique (attention relation distillation) is adapted from MiniLM, not invented here. However, the application insight — that attention relation distillation addresses a capacity bottleneck specific to extreme quantization, and that single-layer distillation respects the student's representational constraints — is a conceptual advance in understanding how distillation interacts with quantization severity. It moves the discussion from "distillation helps quantization" (which was known) to "specific forms of distillation help in specific ways that depend on the quantization regime and the student's capacity limits" — a more nuanced and actionable framing.

Innovation 3: SubLN Insertion as a Surgical, Architecture-Agnostic Stabilization Intervention for Quantization Conversion

What makes this distinctive at the idea level. The paper demonstrates that taking an off-the-shelf FP16 transformer architecture and converting it to 1.58-bit requires not just weight adaptation but architectural modification — specifically, inserting normalization layers at precise internal positions (before attention and FFN output projections) that were not present in the original design. This is a bridge insight between two previously disconnected research threads: the architecture co-design literature (which assumes you control the architecture from scratch, as in BitNet) and the quantization conversion literature (which assumes you keep the architecture fixed and only modify weights, as in PTQ and standard QAT).

The conceptual move is recognizing that architecture and quantization are coupled: the normalization regime that works for FP16 computation does not work for ternary computation, because ternary weights amplify activation variance in ways that FP16 initialization schemes were not designed to handle. You cannot simply quantize an existing architecture and expect it to train stably; you must first modify the architecture to be quantization-compatible. This is not a theoretical claim about expressiveness — it is an empirical claim about optimization dynamics: the same set of representable functions may be unreachable by gradient descent if the architecture's normalization structure is wrong for the weight representation.

Contrast with prior approaches. Prior PTQ methods (GPTQ, AWQ) keep the architecture absolutely fixed — the model is quantized post hoc with no architectural changes. Prior QAT methods (EfficientQAT, LLM-QAT) also typically preserve the architecture and focus on training strategies to recover accuracy. The original BitNet work designed the architecture from scratch with SubLN included, so the need for internal normalization was known in the from-scratch context but had not been studied as a conversion requirement. The paper's contribution is identifying SubLN insertion as a necessary pre-condition for stable QAT when converting an existing FP16 model, and providing a concrete recipe for where to insert the layers.

Figure 3(a) provides the key experimental evidence: without SubLN, the 1.58-bit model shows substantially higher training loss and slower convergence; with SubLN, training stabilizes and loss decreases faster. Table 5 confirms that SubLN alone (Stage 1 only, no continued pre-training and no distillation) provides a measurable improvement over the BitNet-SFT baseline — from 74.09 to 76.30 on MNLI for the 0.6B model. The fact that this gain is modest (+2.2 percentage points) while the combination of all three stages provides +14.1 points demonstrates that SubLN is necessary but not sufficient — it fixes the stability problem but does not address the distributional or capacity problems. This is exactly what one would expect from a correctly diagnosed architectural prerequisite.

Significance beyond raw performance. This insight matters because it challenges the implicit assumption in much quantization work that architecture and precision are independent axes — that you can compress any architecture to any bit-width and treat the training process as purely about recovering from precision loss. The paper shows that at extreme bit-widths (1.58-bit), the architecture itself becomes a variable that must be adapted. This has practical implications: it means that model families designed for FP16 deployment (Qwen, Gemma, LLaMA) cannot simply be quantized and fine-tuned — they must first be structurally converted to a quantization-compatible form. It also raises a deeper question that the paper does not explore: are there architectural features of modern LLMs that are actively hostile to quantization, beyond normalization? This could open a line of research on quantization-aware architecture design for pre-trained models.

The paper's approach to SubLN insertion is also notable for being surgical — it adds normalization at exactly two positions per transformer block (before attention out-projection and before FFN down-projection) rather than redesigning the block from scratch. This minimal-intervention philosophy is what makes the conversion pipeline practical: it preserves as much of the original architecture as possible, meaning the conversion maintains architectural compatibility with the pre-trained weights and can leverage existing model-parallel training infrastructure.

Incremental or fundamental? The SubLN technique itself is adopted from prior BitNet work, making the individual component incremental. But the framing — that architecture must be modified for conversion, not just training procedure — and the identification of where to modify (before output projections that will be quantized) represent a practical engineering insight that changes the recipe for anyone attempting extreme quantization of existing models. It is not a theoretical advance, but it is a non-obvious, well-supported diagnostic contribution that the paper's experiments validate clearly.

Innovation 4: Convergence of Convergent Evidence That 1.58-Bit Deployment Is Practically Viable for Downstream Tasks Without From-Scratch Training

What makes this distinctive at the idea level. This is less a single technical insight and more the paper's meta-contribution: by combining SubLN insertion, distributional pre-adaptation, and capacity-aware distillation into an integrated pipeline, the paper provides the first empirical demonstration that converting pre-trained FP16 LLMs to 1.58-bit for specific downstream tasks is not only possible but practical — achieving performance within ~1 percentage point of FP16 baselines across three model scales, three model families, and two task types. This is a feasibility result that changes the deployment calculus for organizations that cannot afford to train BitNet models from scratch.

The conceptual significance is that it collapses a previously assumed trade-off. The dominant narrative in the quantization literature has been: low-bit quantization (especially sub-4-bit) forces a choice between training from scratch (high compute cost, good performance) and PTQ (low compute cost, degraded performance). BitDistill demonstrates a third option: modest continued pre-training (400× less data than from-scratch) plus distillation-based fine-tuning achieves near-FP16 performance. This reframes the problem from "can we quantize without degrading?" to "how efficiently can we adapt pre-trained models to quantization?" — a question of training efficiency rather than representational capacity.

The multi-model-family validation (Table 3: Qwen3, Qwen2.5, Gemma) is a critical component of this contribution. It demonstrates that the conversion pipeline is not overfit to a single architecture or training regimen, which is the kind of robustness evidence that distinguishes a deployable method from a research demonstration. The compatibility with different quantization schemes (Table 4: Block-Quant, GPTQ, AWQ alongside the default absmean quantization) further strengthens the feasibility claim: BitDistill is not tied to a specific quantization function and can incorporate improvements in quantization algorithms.

Contrast with prior expectations. Prior to this work, the dominant approach for deploying efficient LLMs on edge devices was either to use higher-bit quantization (4-bit or 8-bit via GPTQ/AWQ), which provides less memory reduction, or to train efficient architectures from scratch (BitNet, MobileLLM), which requires large-scale pretraining infrastructure. The paper's claim that 10B tokens of continued pre-training — which the authors describe as "virtually negligible" compared to the 4T tokens for from-scratch BitNet training — is sufficient to bridge the gap represents a 400× reduction in the data barrier to entry for 1.58-bit deployment. If validated by the community, this would dramatically expand the set of organizations that can deploy 1.58-bit models for specific downstream applications.

Significance beyond raw performance. The paper's findings have immediate practical implications for resource allocation in LLM deployment pipelines: given an existing FP16 model and a downstream task, the recommended path to efficient deployment is not to train a new efficient model from scratch, nor to use PTQ, but to apply BitDistill's three-stage pipeline with a small general-domain corpus. The paper also provides specific guidance on how to select the distillation layer (Figure 3b: later layers), how to set distillation weights (λ and γ differ substantially between classification and summarization), and that larger teachers can push student performance above same-size baselines (Figure 3c). These are actionable engineering insights that lower the barrier to adopting 1.58-bit deployment.

The paper does not claim that BitDistill makes 1.58-bit deployment universally viable — it is explicitly scoped to downstream tasks where the FP16 baseline already performs well (the GLUE benchmarks and CNNDM summarization are standard fine-tuning tasks, not open-ended generation or complex reasoning). The hardest test of whether 1.58-bit conversion is truly practical would involve tasks where the model is near the edge of its capability — the kind of tasks where small precision losses might compound into large performance drops. The paper's evidence is strong within its scope but leaves open the question of whether BitDistill would work for more demanding tasks or larger models.

Evidence quality. The convergence of evidence across model scales (0.6B, 1.7B, 4B), task types (classification, summarization), model families (Qwen3, Qwen2.5, Gemma), and quantization methods (absmean, Block-Quant, GPTQ, AWQ) provides unusually robust support for a systems contribution. The ablation in Table 5 is critical: it shows that each pipeline stage contributes non-trivially and that the full three-stage combination is necessary for optimal results. The visual evidence in Figure 2 (weight distributions) and Figure 3 (loss curves, layer selection, teacher scaling) provides mechanistic support that goes beyond performance reporting. The main limitation is the absence of statistical confidence measures (standard deviations across runs, significance tests), which makes it difficult to assess whether the ~1 percentage point remaining gap between BitDistill and FP16-SFT is statistically reliable or within noise.

Incremental or fundamental? This is fundamentally a feasibility demonstration rather than an algorithmic or theoretical advance. The individual components (SubLN, continued pre-training, MiniLM distillation, STE quantization) are all adapted from prior work. What is new is the integration into a coherent pipeline, the diagnostic analysis of why direct conversion fails (Innovation 1), and the empirical evidence that the combination works across a range of settings. In the taxonomy of research contributions, this is strongest as a systems and empirical contribution that enables practical deployment, supported by diagnostic insights that explain why the naive approach fails. It does not introduce new theory or architecture, but it provides a recipe that — if it replicates across model families and tasks — could change how the community approaches efficient deployment of pre-trained LLMs.

5. Experimental Analysis

Evaluation Methodology

  • Dataset. The paper evaluates on two task types: text classification using three GLUE benchmark datasets — MNLI (Multi-Genre Natural Language Inference, 433k training examples), QNLI (Question-answering Natural Language Inference, 105k training examples), and SST-2 (Stanford Sentiment Treebank, 67k training examples) — and text summarization using the CNN/DailyMail dataset (CNNDM, 287k training examples). The specific dataset splits used for training and evaluation are the standard ones from the GLUE benchmark and the CNNDM dataset, though the paper does not report the exact training/validation/test sizes used in its experiments.

  • Base model(s). The primary base models are the Qwen3 series at three scales: 0.6B, 1.7B, and 4B parameters (Yang et al., 2025). These are chosen as "representative of a canonical approach" to pre-trained LLM architecture, and because their availability at multiple scales enables the investigation of the scalability issue (the widening performance gap with model size) that is central to the paper's diagnostic contribution. Additional experiments test robustness across model families using Qwen2.5-0.5B (Qwen et al., 2025) and Gemma3-1B (Team et al., 2025), providing evidence that the method is not architecture-specific.

  • Metrics. Classification performance is evaluated using accuracy (percentage of correctly classified examples). Summarization quality is assessed using BLEU (Papineni et al., 2002) and ROUGE (Lin, 2004), specifically ROUGE-1, ROUGE-2, ROUGE-L, and the ROUGE-SUM average (the mean of ROUGE-1, ROUGE-2, and ROUGE-L). Inference efficiency is measured as tokens per second throughput on CPU with 16 threads, and memory usage in gigabytes. The paper reports inference speed and memory for model loading and generation but does not specify whether memory is measured as peak GPU memory, CPU RAM, or model file size — the context of "10× memory savings" and the figures (e.g., 1.20 GB for FP16 vs. 0.11 GB for 1.58-bit at 0.6B scale) suggest model weight storage size, since 0.6B parameters at FP16 would be 1.2 GB (0.6B × 2 bytes) and at 1.58-bit would be approximately 0.11 GB if using the theoretical minimum of log2(3) bits per weight with some overhead.

  • Baselines. Three baselines are used throughout:

    • FP16-SFT: The full-precision base model fine-tuned directly on the downstream task using standard supervised fine-tuning. This is the upper-bound reference and also serves as the teacher model in BitDistill's distillation stage. This baseline is a standard supervised fine-tuning (SFT) approach without any quantization or distillation, representative of typical deployment when compute is not constrained.
    • BitNet-SFT: The full-precision base model quantized to 1.58-bit using the absmean quantization function (Eq. 1) and directly fine-tuned on the downstream task using quantization-aware training with straight-through estimation. This is the naive conversion approach that the paper identifies as failing due to the scalability issue. No SubLN insertion, no continued pre-training, and no distillation are applied.
    • BitDistill (Ours): The full three-stage pipeline: SubLN insertion + 10B-token continued pre-training + distillation-based fine-tuning. Compared against both FP16-SFT (to establish performance parity) and BitNet-SFT (to demonstrate the gap closure).

  • Generation budget / compute accounting. The paper does not use a standardized "compute budget" metric in the sense of FLOPs or generation counts. Instead, the main axis of comparison is model scale (0.6B, 1.7B, 4B) at fixed training configurations, with the continued pre-training budget fixed at 10B tokens. The inference efficiency comparison uses a fixed hardware configuration (CPU with 16 threads) and reports throughput and memory, which serves as the practical compute accounting. The paper does note that the continued pre-training cost (10B tokens) is "virtually negligible" compared to training a BitNet from scratch (~4T tokens), representing a 400× reduction — this is the primary compute-cost justification for the method. However, the paper does not report total FLOPs for any stage, making it impossible to precisely compare the computational cost of BitDistill against alternatives like training a larger FP16 model, using a different quantization method, or doing more extensive task-specific fine-tuning. The training is conducted on 8× AMD Mi300X GPUs, but total training time or GPU-hours are not reported.

  • Cross-validation / statistical protocol. The paper states that "a greedy search strategy" is used to select the optimal learning rate and training epochs for all methods, which "mitigates overfitting while ensuring both strong downstream performance and fair comparisons across methods." No explicit cross-validation protocol (e.g., k-fold, held-out validation set splits) is described beyond this. The paper does not report standard deviations, confidence intervals, or results from multiple random seeds. This means that the reported accuracy numbers — many of which differ between BitDistill and FP16-SFT by less than 0.5 percentage points (e.g., 88.17 vs. 88.01 on MNLI for 0.6B, 94.30 vs. 94.21 on SST-2 for 0.6B) — cannot be assessed for statistical significance. The claim that BitDistill achieves "performance comparable to the FP16 baseline" rests on point estimates alone, without evidence that the small remaining gaps are distinguishable from noise.

Main Quantitative Results

Overall Classification and Summarization Performance

The paper's headline results are presented in Table 1 (classification) and Table 2 (summarization). The central claim is that BitDistill achieves performance comparable to FP16-SFT across all model scales and both task types, in stark contrast to BitNet-SFT which shows catastrophic degradation.

Classification (Table 1). For the Qwen3-0.6B model on MNLI, BitDistill achieves 88.17% accuracy compared to 88.01% for FP16-SFT and 74.09% for BitNet-SFT — a gap closure from -13.92 percentage points (BitNet-SFT) to +0.16 (BitDistill). On QNLI at the same scale, BitDistill reaches 93.66% vs. 93.72% for FP16-SFT (-0.06 points) and 78.32% for BitNet-SFT (-15.40 points). On SST-2, BitDistill achieves 94.30% vs. 94.21% for FP16-SFT (+0.09 points) and 79.92% for BitNet-SFT (-14.29 points).

At the 1.7B scale, the gap between BitNet-SFT and FP16-SFT widens: on MNLI, BitNet-SFT drops to 75.27% while FP16-SFT reaches 89.61% — a gap of 14.34 percentage points, larger than the 13.92-point gap at 0.6B. BitDistill closes this to 89.53% (-0.08 points from FP16-SFT). On QNLI at 1.7B, BitNet-SFT achieves 79.54% vs. FP16-SFT's 95.00% (gap of 15.46 points), and BitDistill recovers to 94.82% (-0.18 points). This directly demonstrates the scalability failure: the BitNet-SFT gap grows with model size, while BitDistill's gap remains consistently small.

At the 4B scale, the pattern holds. MNLI: BitNet-SFT 76.11%, FP16-SFT 91.48%, BitDistill 91.40% (-0.08 points). QNLI: BitNet-SFT 79.97%, FP16-SFT 96.02%, BitDistill 95.93% (-0.09 points). SST-2: BitNet-SFT 82.07%, FP16-SFT 96.57%, BitDistill 96.47% (-0.10 points). The BitNet-SFT gap has grown to 15.37 percentage points on MNLI at 4B (from 13.92 at 0.6B), confirming the scalability diagnosis. BitDistill's gap remains under 0.20 percentage points across all scales and datasets.

The inference efficiency numbers are dramatic: BitDistill and BitNet-SFT both achieve 1,135 tokens/second on CPU compared to 427 tokens/second for FP16-SFT — a 2.65× speedup. Memory drops from 1.20 GB to 0.11 GB for the 0.6B model — a 10.9× reduction. These ratios are identical for BitDistill and BitNet-SFT because the model architecture and quantization at inference time are the same; the performance difference comes from training quality, not inference characteristics.

Summarization (Table 2). Using the Qwen3-0.6B model on CNNDM, BitDistill achieves a BLEU score of 14.41 compared to 13.98 for FP16-SFT and 11.47 for BitNet-SFT. On ROUGE-1, BitDistill achieves 40.21 vs. 40.62 (FP16-SFT) and 37.10 (BitNet-SFT). The ROUGE-SUM average is 27.44 for BitDistill, 27.58 for FP16-SFT (a difference of 0.14 points), and 24.15 for BitNet-SFT (a gap of 3.43 points from FP16-SFT). Only the 0.6B scale is reported for summarization; the paper does not present summarization results at 1.7B or 4B, which limits the assessment of whether the scalability fix extends to generation tasks. The summarization metrics show a pattern where BitDistill occasionally exceeds FP16-SFT (BLEU: 14.41 vs. 13.98) and occasionally falls slightly short (ROUGE-L: 27.49 vs. 27.72), with the overall ROUGE-SUM average showing a tiny deficit (27.44 vs. 27.58). This is consistent with "performance comparable to the FP16 baseline" but the single-scale reporting limits confidence in the claim for generation tasks specifically.

The inference efficiency for summarization (also only reported at 0.6B) shows the same 2.65× throughput improvement (427 vs. 1,135 tokens/s) and 10.9× memory reduction (1.20 vs. 0.11 GB) as classification.

Key takeaway from Tables 1 and 2: BitDistill reduces the BitNet-SFT performance gap by approximately 95-100% across all reported comparisons, bringing 1.58-bit models to within 0.0-0.2 percentage points of FP16 baselines on classification and within 0.0-0.5 points on summarization metrics. The BitNet-SFT gap provides a clear counterfactual: without BitDistill's pipeline, direct QAT produces models that are 13-15 points worse at classification, and 3.4 ROUGE-SUM points worse at summarization. The fact that BitDistill achieves this while matching BitNet-SFT's inference efficiency (since both are the same architecture at deployment) is the paper's primary evidentiary claim: the pipeline recovers the performance without sacrificing the efficiency benefits.

Robustness Across Model Families

Table 3 tests whether BitDistill's performance transfers to different base model architectures. Using Gemma3-1B on the MNLI classification task, BitDistill achieves 89.61% accuracy compared to 89.77% for FP16-SFT and 78.02% for BitNet-SFT — a gap closure from -11.75 percentage points to -0.16. Using Qwen2.5-0.5B, BitDistill achieves 79.98% vs. 79.91% for FP16-SFT and 60.80% for BitNet-SFT — a gap closure from -19.11 to +0.07 percentage points.

Two observations emerge from Table 3. First, the BitNet-SFT gap varies by architecture: 11.75 points for Gemma3-1B vs. 19.11 points for Qwen2.5-0.5B on the same MNLI task, suggesting that some architectures are more amenable to direct QAT than others. The paper does not analyze why this variation occurs — it could relate to differences in weight distribution after pre-training, normalization structure, or training data. Second, BitDistill closes the gap consistently regardless of its initial size: the method works across a 11.75-point gap and a 19.11-point gap, bringing both to within 0.2 points of FP16 baselines. This robustness to architecture is important because it addresses a natural skeptical question: is BitDistill overfit to the Qwen3 architecture's specific properties? The Gemma and Qwen2.5 results suggest not, though the number of tested architectures (two alternatives) is limited.

Compatibility with Different Quantization Methods

Table 4 tests whether the BitDistill framework works when the default absmean quantization function (Eq. 1) is replaced with alternative quantization techniques. On MNLI with Qwen3-0.6B, BitDistill with Block-Quant (Dettmers et al., 2021) achieves 88.23%, with GPTQ (Frantar et al., 2022) achieves 88.05%, and with AWQ (Lin et al., 2024) achieves 88.25%, compared to 88.17% for the default absmean quantization. On QNLI, the respective numbers are 93.74%, 93.63%, 93.70%, and 93.66%. The variation across quantization methods is within 0.2 percentage points on MNLI and 0.1 points on QNLI — essentially negligible. All variants remain close to the FP16-SFT baselines (88.01 on MNLI, 93.72 on QNLI).

This result demonstrates that BitDistill's training pipeline (SubLN + continued pre-training + distillation) is complementary to the choice of quantization function. The paper interprets this as evidence that "BitDistill is complementary to different quantization algorithms, offering a unified procedure that can stably enhance low-bit models." However, the comparison is only performed at Qwen3-0.6B on two classification tasks, so the generality of this compatibility claim across scales and tasks is not fully tested. The paper also does not examine whether more sophisticated quantization functions (GPTQ, AWQ) provide larger benefits at larger model scales where quantization error compounding might be more severe — this is a missed opportunity to understand whether quantization function choice becomes more important as the performance floor rises.

Ablation Studies and Robustness Checks

Effect of each individual stage in BitDistill (Table 5). Using Qwen3-0.6B, the full three-stage pipeline achieves 88.17% on MNLI and 14.41 BLEU / 40.21 ROUGE-1 / 17.47 ROUGE-2 / 27.49 ROUGE-L on CNNDM. Removing Stage 3 (distillation) — i.e., SubLN + continued pre-training only, followed by standard cross-entropy fine-tuning without distillation — drops MNLI to 86.73 (-1.44 points) and CNNDM metrics to 13.96 BLEU / 39.75 ROUGE-1 / 16.47 ROUGE-2 / 26.96 ROUGE-L. Removing Stage 2 (continued pre-training) — i.e., SubLN only, then distillation-based fine-tuning — yields 88.04 on MNLI (-0.13 points) and 13.70 BLEU / 39.92 ROUGE-1 / 16.91 ROUGE-2 / 27.16 ROUGE-L on CNNDM. Removing Stage 1 (SubLN) — i.e., continued pre-training + distillation only — is not tested (this ablation row is absent from Table 5; the "✗ ✗ ✗" row corresponds to the BitNet-SFT baseline with no stages). The intermediate row with only Stage 1 (SubLN alone, no continued pre-training and no distillation, using standard SFT) achieves 76.30 on MNLI (+2.21 points over BitNet-SFT baseline of 74.09) and 11.69 BLEU / 37.81 ROUGE-1 on CNNDM (+0.22 BLEU / +0.71 ROUGE-1 over the 11.47 / 37.10 baseline).

Several non-obvious patterns emerge from Table 5. Most dramatically, Stage 2 (continued pre-training) accounts for the majority of the performance recovery: going from Stage 1 only (76.30 MNLI) to Stage 1+2 (86.73 MNLI) provides a +10.43 point jump, while Stage 3 adds only +1.44 more. On CNNDM, Stage 2 contributes +2.27 BLEU / +1.94 ROUGE-1, while Stage 3 adds +0.45 BLEU / +0.46 ROUGE-1. This confirms the paper's central diagnostic claim: distributional adaptation via continued pre-training is the primary bottleneck that BitDistill addresses, and distillation provides a complementary but smaller incremental gain. The asymmetry between classification and summarization in Stage 3's contribution is also notable — distillation helps more on classification (+1.44 MNLI accuracy, which is ~1.6% relative improvement over the Stage 1+2 baseline) than on summarization (+0.45 BLEU, ~3.2% relative improvement, though the absolute numbers are harder to compare across metrics). The paper does not discuss this difference.

A critical missing ablation is Stage 1 removal (SubLN). Table 5 does not test Stages 2+3 without Stage 1, which would answer the question: does continued pre-training alone (without SubLN insertion) also provide large gains, or is SubLN a necessary prerequisite for the continued pre-training to be effective? The paper's earlier claim that SubLN addresses a separate instability problem (supported by Figure 3a) implies that SubLN should be necessary — without it, the continued pre-training might be unstable and fail to reshape the weight distribution effectively. The absence of this ablation weakens the claim that all three stages are independently necessary.

Effect of different distillation techniques in Stage 3 (Table 6). Using Qwen3-0.6B on MNLI, starting from the Stage 1+2 baseline of 86.73: adding logits distillation only (LD) reaches 87.32 (+0.59 points over baseline), adding attention distillation only (AD) reaches 87.67 (+0.94 points), and adding both reaches 88.17 (+1.44 points). The combined gain (+1.44) is greater than the sum of individual gains (+0.59 + +0.94 = +1.53 would be additive), though the paper claims "their synergy is particularly beneficial." This is a fair characterization since the individual gains are measured against the no-distillation baseline and the combined gain is measured against the same baseline — the fact that both together provide +1.44 while the individual gains are +0.59 and +0.94 suggests that some of the benefit each technique provides is overlapping (if they were fully independent, the combined gain would be approximately +1.53). Attention distillation provides a larger individual gain (+0.94) than logits distillation (+0.59), which is consistent with the much higher coefficient (γ = 10^5 vs. λ = 10 for classification).

A notable limitation: Table 6 only reports MNLI results and does not show the distillation ablation for summarization, where the λ and γ coefficients are dramatically different (λ = 1 and γ = 10^3 for summarization vs. λ = 10 and γ = 10^5 for classification). The paper does not test whether attention distillation remains more important than logits distillation under the summarization coefficient regime, nor does it ablate the coefficient values themselves to determine sensitivity. This is a significant gap because the coefficient values are the most extreme hyperparameter choices in the pipeline, and understanding their sensitivity would inform practitioners about how carefully they need to tune these for new tasks.

Effect of SubLN on training stability (Figure 3a). The training loss curves comparing 1.58-bit BitNet-SFT with and without SubLN insertion show that SubLN leads to lower loss and faster convergence during fine-tuning on the FALCON corpus. The "w/ SubLN" curve sits consistently below the "w/o SubLN" curve throughout training, with the gap widening in early steps and persisting through convergence. This provides visual evidence for the activation variance stabilization hypothesis — SubLN prevents the optimization from being hampered by exploding activations, allowing the model to make more consistent progress per gradient step. The curves are presented without numerical labels on the y-axis, so the absolute loss values cannot be read precisely from the figure, but the relative ordering is clear. The paper does not report final loss values or downstream performance for this specific comparison, relying on the visual trend as qualitative evidence. This is adequate for supporting the mechanistic claim but insufficient for quantifying the magnitude of the stability benefit.

Distillation layer selection strategy (Figure 3b). Testing Qwen3-0.6B on MNLI (without continued pre-training — this ablation is performed after Stage 1 only, as clarified in the text), attention relation distillation is applied at each individual layer separately, and the resulting accuracy is plotted. Distilling at layer 0 (the first transformer layer) yields the lowest accuracy, around 81.5%. Performance improves for middle layers, peaking at layer 21 (out of what appears to be 28 total layers, inferred from the x-axis range of Figure 3b), which achieves approximately 87.7% accuracy. Distilling at the final layer (layer 27) yields approximately 87.2%, slightly below the peak. The "All Layers" baseline is shown at approximately 86.5% — substantially lower than the best single layer and confirming the paper's hypothesis that single-layer distillation provides better optimization flexibility. The variation across individual layers is substantial: roughly 6 percentage points between the worst (layer 0) and best (layer 21), demonstrating that layer choice is a critical hyperparameter. The paper's observation that "layers located in the later stages of the model tend to deliver better distillation performance" is supported, with the top-performing layers concentrated in the second half of the network. However, the paper does not report whether this layer-selection pattern generalizes across model scales or task types — it is tested only at 0.6B on MNLI. At larger scales with more layers, the optimal layer might shift, and the performance penalty for choosing a suboptimal layer might differ. The paper also does not provide guidance on how to select the distillation layer without an expensive per-layer sweep.

Effect of teacher quality (Figure 3c). Testing Qwen3-0.6B as the student on MNLI, the paper compares three teacher configurations: Qwen3-0.6B FP16 as teacher (same-size teacher), Qwen3-1.7B FP16 as teacher (larger teacher), and Qwen3-4B FP16 as teacher (largest teacher). The training curves show accuracy over steps. The 0.6B teacher converges to approximately 88.2% (consistent with the final BitDistill result in Table 1). The 1.7B teacher reaches approximately 89.0% — roughly 0.8 points higher than the same-size teacher. The 4B teacher reaches approximately 89.5% — roughly 1.3 points higher than the same-size teacher and 0.5 points higher than the 1.7B teacher. All configurations use the same training steps (x-axis extends to approximately 2.5k steps, read from the figure). The finding that "our algorithm can effectively extract larger gains from a higher-quality teacher, even surpassing FP16 models of the same size" is supported: the 0.6B BitDistill student with a 4B teacher achieves ~89.5%, which exceeds the 0.6B FP16-SFT teacher's 88.01% (Table 1) by approximately 1.5 points. This is a practically significant result because it suggests that organizations with access to a large FP16 model can use it to train a much smaller and more efficient 1.58-bit model that outperforms what the smaller model could achieve on its own — a form of model compression that goes beyond preservation to potential improvement.

Weight distribution visualization (Figure 2). The figure compares the weight distributions of three scenarios across seven projection matrices (Attention-Q, Attention-K, Attention-V, Attention-O, FFN-Gate, FFN-Up, FFN-Down): a BitNet trained from scratch (top two rows, showing both the quantized weights and the underlying FP16 distributions), and a BitNet after loading from an FP16 LLM and applying continued training with SubLN (bottom two rows). The from-scratch BitNet shows FP16 weight distributions with distinctive shapes: mass concentrated in two peaks near the quantization boundaries (approximately at -0.5 and +0.5 on the normalized weight axis, corresponding to the transitions between -1/0 and 0/1 in the ternary space), with relatively lower density near zero and at the extremes. The quantized weights (shown as discrete bars at -1, 0, and 1) reflect this distribution. In contrast, the "after loading weights from LLMs and performing continued training" rows show that: (1) before continued training, the FP16 weights were approximately Gaussian (not shown in Figure 2 directly but stated in the text and inferable from the bottom-row distributions which show significant mass in the boundary regions), and (2) after continued training, the distributions closely resemble the from-scratch BitNet distributions — mass is concentrated near the quantization boundaries, with similar peak shapes and positions.

This visual evidence is the primary support for the paper's mechanistic hypothesis that continued pre-training works by reshaping the weight distribution to be "quantization-friendly." The paper does not provide quantitative distribution similarity metrics (e.g., KL divergence, Wasserstein distance, or overlap coefficients between the from-scratch and continued-training distributions), relying on qualitative visual comparison. This is a limitation: the claim that the distributions "become more similar" is visually plausible but not quantified, and the degree of similarity varies across the seven projection matrices (some appear more similar than others).

The paper also does not show the weight distribution before continued pre-training (i.e., the distribution immediately after loading from the FP16 LLM and inserting SubLN, but before any training). This missing visualization would provide the baseline against which the distribution shift could be measured, and would clarify whether the FP16 pre-trained weights truly start Gaussian (as stated) or already have some structure that continued pre-training amplifies. The figure shows only the endpoint of Stage 2, not the starting distribution, making it impossible to assess the magnitude of the distribution shift from the figure alone.

Critical Assessment

Claim 1: BitDistill achieves performance comparable to full-precision baselines. This claim is supported by Tables 1, 2, and 3. Across 14 data points (3 scales × 3 classification datasets + 1 scale × 5 summarization metrics + 2 architectures × 1 dataset), BitDistill's accuracy or metric is within 0.0–0.5 of FP16-SFT in every case, with some configurations showing marginal improvements over FP16-SFT and others showing marginal deficits. The evidence is consistent and the number of tested configurations is reasonable for a systems paper.

However, "comparable" is doing important work here that the paper does not interrogate statistically. The differences between BitDistill and FP16-SFT in Table 1 are often very small: -0.01 on MNLI at 0.6B (88.17 vs. 88.01), -0.06 on QNLI at 0.6B (93.66 vs. 93.72), +0.09 on SST-2 at 0.6B (94.30 vs. 94.21). Without standard deviations or significance tests, it is impossible to determine whether these are real differences or noise. If the standard deviation of MNLI accuracy under different random seeds is, say, 0.3 percentage points, then 88.17 and 88.01 are statistically indistinguishable. If the standard deviation is 0.05, then BitDistill is genuinely slightly better on this configuration. The paper provides no information to make this determination, which weakens the precision of the "comparable" claim — we know BitDistill is close to FP16-SFT, but we do not know whether it is identically good or systematically slightly worse by an amount that might matter for high-stakes applications.

A related concern is that the FP16-SFT baseline itself has variance — it is the result of a single fine-tuning run with a greedily-searched learning rate. Multiple runs of FP16-SFT with different random seeds might produce a range of accuracies. BitDistill should be compared against the distribution of FP16-SFT results, not a point estimate, to determine whether it falls within the expected range of FP16 performance or systematically below it.

Claim 2: Direct QAT (BitNet-SFT) exhibits a scalability failure where the performance gap widens with model size. This claim is supported by the pattern across scales in Table 1. The MNLI gap between FP16-SFT and BitNet-SFT grows from 13.92 points at 0.6B (88.01 - 74.09) to 14.34 at 1.7B (89.61 - 75.27) to 15.37 at 4B (91.48 - 76.11). The QNLI gap grows from 15.40 at 0.6B (93.72 - 78.32) to 15.46 at 1.7B (95.00 - 79.54) to 16.05 at 4B (96.02 - 79.97). The SST-2 gap grows from 14.29 at 0.6B (94.21 - 79.92) to 14.06 at 1.7B (95.43 - 81.37) to 14.50 at 4B (96.57 - 82.07). The growth is consistent but small in magnitude: roughly 1.5–2 percentage points increase across a 6.7× scale increase (0.6B to 4B). The term "poor scalability" accurately describes the direction of the trend but overstates its severity — a 1.5-point widening over a 6.7× scale increase is a real effect but not a catastrophic one. The more striking finding is that the gap exists at all (13–16 points), not that it widens slightly. The paper's diagnostic emphasis on the widening is scientifically interesting (it rules out pure capacity-limitation explanations) but the practical significance for deployment is that BitNet-SFT fails badly at all scales, not that it gets slightly worse at larger scales.

The claim is only tested up to 4B parameters. Whether the scalability failure continues to worsen at larger scales (7B, 13B, 70B) is unknown. It is possible that the gap asymptotes, or that it accelerates. The paper cannot extrapolate beyond its tested range, and the term "scalability issue" may imply greater generality than the evidence supports.

Claim 3: Continued pre-training resolves the scalability issue by reshaping weight distributions. This is the paper's most important mechanistic claim, and the evidence is mixed. The strongest evidence is in Table 5: Stage 2 (continued pre-training) contributes +10.43 MNLI points over the Stage 1-only baseline (76.30 → 86.73), which represents ~85% of the total recovery from the BitNet-SFT baseline to the final BitDistill result (74.09 → 88.17). This demonstrates that continued pre-training is the dominant contributor to performance recovery — a necessary condition for the mechanistic claim. The visual evidence in Figure 2 shows that the weight distribution after continued pre-training resembles that of a from-scratch BitNet, which is consistent with the hypothesized mechanism.

However, the causal chain from distribution reshaping to performance recovery is not directly tested. The paper shows a correlation — continued pre-training both reshapes the distribution and improves performance — but does not demonstrate that the distribution reshaping causes the performance improvement. An alternative hypothesis is that continued pre-training helps by exposing the model to more data (any data, regardless of distribution effects) or by providing a better initialization for the downstream optimizer (unrelated to quantization boundary placement). To distinguish these, the paper would need to show that models with weight distributions not reshaped toward quantization boundaries (but trained on the same amount of data) perform worse — a challenging counterfactual to construct. As it stands, the mechanism is plausible and consistent with the evidence but not causally proven.

The paper's explanation of why boundary-concentrated distributions help — that they "allow the quantized values to shift more frequently with small gradient steps" — is a hypothesis about optimization dynamics that is not directly tested. No experiment measures the frequency of ternary value flips during training for boundary-concentrated vs. Gaussian initializations, nor shows that models with more frequent flips achieve better downstream performance. This is a theoretical gap that limits the depth of the mechanistic understanding.

Claim 4: BitDistill enables up to 10× memory savings and 2.65× faster inference on CPUs. This claim is straightforward and well-supported by the inference measurements in Tables 1 and 2. The memory reduction (1.20 GB → 0.11 GB for 0.6B models) and throughput improvement (427 → 1,135 tokens/s) are measured on the same CPU hardware with 16 threads. The speedup and memory ratios are properties of the 1.58-bit representation itself, not BitDistill's training pipeline — BitNet-SFT achieves identical inference efficiency. The claim is therefore true but not distinctive to BitDistill; the paper's contribution is that BitDistill achieves these efficiency gains without sacrificing accuracy, which BitNet-SFT cannot do.

The inference measurements are only reported for the 0.6B model. At larger scales (1.7B, 4B), the absolute throughput and memory would change, and the ratio might not remain exactly 2.65× and 10× — larger models may encounter different memory bandwidth bottlenecks or CPU cache effects that alter the speedup. The paper's extrapolation of these ratios to all scales is implied but not directly measured.

Missing experiments and analyses. Several additional experiments would significantly strengthen the paper's claims:

  • Variance estimates (multiple seeds). Running the full pipeline with 3–5 random seeds and reporting means and standard deviations would allow assessment of whether the BitDistill-vs-FP16 differences are statistically significant. This is the single most important missing piece of evidence for the "comparable performance" claim.

  • Summarization at larger scales. Table 2 only reports CNNDM results for the 0.6B model. Does the scalability fix work for generation tasks at 1.7B and 4B? Without this, the claim that BitDistill resolves the scalability issue is only demonstrated for classification.

  • SubLN ablation (Stages 2+3 without Stage 1). Table 5 does not test whether SubLN is necessary when continued pre-training and distillation are present. The paper's claim that all three stages are necessary rests on the assumption that SubLN addresses a problem (activation instability) that the other stages cannot — but this is not experimentally verified.

  • Continued pre-training data quantity ablation. The paper uses 10B tokens for continued pre-training and claims this is sufficient. How sensitive is performance to this quantity? Would 1B tokens work nearly as well? Would 100B tokens provide further gains? Without this ablation, practitioners cannot determine the minimum viable continued pre-training budget.

  • Distillation coefficient sensitivity. The λ and γ values (10 and 10^5 for classification, 1 and 10^3 for summarization) span five orders of magnitude. How sensitive is performance to these choices? A sweep over λ and γ would reveal whether these are carefully tuned values that must be exactly right, or whether the method is robust to wide ranges.

  • Larger scale testing. The scalability claim is tested up to 4B parameters. Testing at 7B or 13B (if computationally feasible) would more strongly validate the claim that BitDistill "preserves scalability" and that the scalability failure of BitNet-SFT continues at larger scales.

  • Comparison to other extreme compression methods. The paper compares only to its own baseline (BitNet-SFT) and to FP16-SFT. Comparisons to other 1.58-bit training methods (if any exist for conversion rather than from-scratch training), or to alternative compression techniques (pruning + quantization, distillation to smaller FP16 models), would contextualize BitDistill's performance relative to the broader compression landscape.

  • Training cost reporting. The paper does not report GPU-hours for any stage of the pipeline, making it impossible for practitioners to estimate the total computational cost. The claim that continued pre-training on 10B tokens is "virtually negligible" compared to 4T tokens of from-scratch training is directionally true but lacks the precision (e.g., GPU-hours, total FLOPs, or dollar cost estimates) needed for resource planning.

6. Limitations and Trade-offs

The Difficulty of Scaling the Evidence Beyond 4B Parameters on Classification Tasks

The assumption or constraint. The paper's central diagnostic — that BitNet-SFT exhibits a scalability failure where the performance gap widens with model size — is tested only up to 4B parameters on classification tasks (Table 1) and only at 0.6B for summarization (Table 2). The paper does not report summarization results at 1.7B or 4B, nor does it test any configuration beyond the 4B scale. The authors do not explicitly state this as a scope limitation; the scaling axis simply stops at 4B in all reported experiments.

The consequence. A practitioner deploying models at larger scales (7B, 13B, 70B+) cannot determine from this paper whether BitDistill's performance parity with FP16-SFT holds, or whether the scalability failure of BitNet-SFT accelerates at those scales. The paper's framing that BitDistill "preserves scalability" (Figure 1, Section 1) implies that the gap between BitDistill and FP16-SFT remains consistently small as models grow, but this extrapolation is untested beyond the 0.6B–4B range tested. If the scalability failure of BitNet-SFT worsens non-linearly at larger scales (e.g., a 25-point gap at 7B), it is possible that BitDistill's continued pre-training with a fixed 10B-token budget becomes insufficient — the per-parameter gradient signal would be even thinner than at 4B, potentially requiring scaling the continued pre-training budget with model size, which the paper does not address. Conversely, if the scalability failure asymptotes, the paper's emphasis on it as a key challenge might be disproportionate. Without larger-scale evidence, the practical applicability of BitDistill is bounded at the tested scale range.

What evidence exists in the paper. The scalability diagnosis is supported by three data points (0.6B, 1.7B, 4B) on three classification datasets (Table 1), showing the BitNet-SFT gap growing from ~13.9 to ~15.3 percentage points. BitDistill's gap to FP16-SFT remains under 0.2 points at all three scales. For summarization, only the 0.6B model is reported (Table 2), so no scalability evidence exists for generation tasks. The paper provides no extrapolation analysis, no scaling law fitting (e.g., log-log plots of gap vs. parameters), and no discussion of how the continued pre-training budget might need to scale with model size.

Mitigation status. The paper does not address this limitation. There is no suggestion that the scalability property should be tested at larger scales, and the scaling axis terminates at 4B without comment. The Abstract and Introduction claim BitDistill "preserves scalability" without qualifying the tested range. A natural future experiment — testing at 7B or 13B using the same pipeline — is not discussed.

Difficulty Estimation for the Continued Pre-Training Budget: No Guidance on How Much Data Is "Enough"

The assumption or constraint. The continued pre-training stage (Stage 2) uses 10B tokens of general-domain data from the FALCON corpus, which the paper describes as "virtually negligible" compared to the ~4T tokens required for training BitNet from scratch (Section 3.2, Section 4.1). This budget is held fixed across all model scales (0.6B, 1.7B, 4B), all tasks, and all base model families. The paper does not ablate the continued pre-training data quantity — there is no experiment showing what happens with 1B, 5B, 20B, or 50B tokens, nor does it provide a principle for determining how much continued pre-training is necessary for a given model scale or task.

The consequence. A practitioner adopting BitDistill for a new model scale, new base architecture, or new task type cannot determine the minimum viable continued pre-training budget from this paper. The paper's central mechanistic claim — that continued pre-training works by reshaping the weight distribution toward quantization boundaries — implies that the required budget should depend on how far the initial FP16 weight distribution is from the target boundary-concentrated distribution, which could vary across model families (Gemma vs. Qwen vs. LLaMA), model scales, and pre-training recipes. Without a data budget ablation, practitioners face two risks: (1) using too little data, which might leave the weight distribution inadequately adapted and produce a larger BitDistill-to-FP16 gap than reported; (2) using too much data, which wastes compute on a stage the paper itself frames as a cost-reduction measure. The paper's claim that 10B tokens is "virtually negligible" relative to 4T-token from-scratch training is directionally true but provides no actionable guidance for deployment scenarios where 10B tokens of continued pre-training might itself be a significant fraction of the total compute budget (e.g., for smaller organizations or for very large models where per-token training cost is high).

What evidence exists in the paper. Table 5 shows that Stage 2 is the single largest contributor to performance recovery — it provides +10.43 MNLI points over the Stage 1-only baseline (76.30 → 86.73 at 0.6B). This demonstrates that 10B tokens works for this specific configuration, but provides no information about whether a smaller budget would suffice (e.g., 5B tokens yielding 85.0 accuracy) or whether a larger budget would provide further gains (e.g., 50B tokens yielding 88.5, beating FP16-SFT). The weight distribution visualizations in Figure 2 show the result of continued pre-training at 10B tokens but do not show intermediate distributions (e.g., at 1B, 5B tokens) that would reveal the trajectory of distribution adaptation and whether it saturates before 10B. The paper does not present a learning curve for the continued pre-training phase (e.g., downstream task accuracy as a function of continued pre-training tokens), which would be the most direct evidence for budget sensitivity.

Mitigation status. The paper does not acknowledge this as a limitation. The fixed 10B-token budget is presented as a feature (cost reduction relative to from-scratch training) rather than a hyperparameter that itself requires tuning. The suggestion in Section 3.2 that 10B tokens allows the model to "rapidly adapt to the feature space that is better suited for 1.58-bit optimization" implies a threshold effect — that adaptation happens quickly and then saturates — but this is not empirically validated. A data budget ablation is a straightforward experiment (train Stage 2 with multiple token budgets, evaluate downstream performance) that the paper does not report.

The Absence of Statistical Significance Measures Makes the "Comparable Performance" Claim Difficult to Evaluate at the Reported Precision

The assumption or constraint. The paper reports all results as point estimates — single accuracy or metric values for each configuration — without standard deviations, confidence intervals, or results from multiple random seeds (Section 4.1, Tables 1–6). The training protocol uses a "greedy search strategy to select the optimal learning rate and training epochs" but does not specify whether the reported results are from the single best run or from an average over multiple runs. The comparison between BitDistill and FP16-SFT often involves differences smaller than 0.5 percentage points (e.g., MNLI 0.6B: 88.17 vs. 88.01, difference of +0.16; QNLI 0.6B: 93.66 vs. 93.72, difference of -0.06; SST-2 1.7B: 95.26 vs. 95.43, difference of -0.17; MNLI 4B: 91.40 vs. 91.48, difference of -0.08). On the CNNDM summarization task, the ROUGE-SUM average is 27.44 for BitDistill vs. 27.58 for FP16-SFT at 0.6B — a difference of 0.14 points.

The consequence. Without variance estimates, the reader cannot determine whether the BitDistill-vs-FP16 gaps are statistically significant or within the noise of training stochasticity. If the standard deviation of MNLI accuracy under different random seeds is, for example, 0.3 percentage points, then a difference of 0.16 points is not statistically meaningful — BitDistill and FP16-SFT are indistinguishable, and the paper's claim of "comparable performance" is fully vindicated. However, if the standard deviation is 0.02 points, then the small but consistent pattern of BitDistill slightly trailing FP16-SFT on most configurations (BitDistill trails on 9 of the 14 comparisons across Tables 1 and 2, though never by more than 0.25 points) could represent a real, systematic precision loss that is masked by the low-granularity reporting. This matters for high-stakes deployment decisions: if BitDistill is genuinely 0.1–0.2 points worse than FP16 across the board, that might be acceptable for most applications but a dealbreaker for applications where every fraction of a percentage point matters (e.g., competitive benchmarks, safety-critical classification). The paper's reporting precision creates an ambiguity that the experimental protocol is equipped to resolve but does not.

What evidence exists in the paper. All result tables (Tables 1–6) and figures (Figures 1, 3) present single point estimates without error bars or variance information. The paper does not discuss the variance properties of the training procedure or the benchmarks. The training settings (Section 4.1) mention a "greedy search strategy" for hyperparameter selection but do not describe a protocol for final evaluation (e.g., best-of-N runs, average over seeds, cross-validation folds). The paper also does not report independent runs of the FP16-SFT baseline, meaning the baseline itself is a point estimate without known variance. This makes it impossible to distinguish between "BitDistill matches FP16" and "BitDistill is slightly but systematically below FP16" — both are compatible with the reported numbers.

Mitigation status. The paper does not address this limitation. No variance metrics are reported, and no discussion of statistical significance appears in the experimental methodology or results sections. This is a standard expectation in empirical ML papers, and its absence weakens the precision of the central claim. The fact that many BitDistill-vs-FP16 differences are within 0.2 points suggests that the conclusion of comparability is likely robust even accounting for reasonable variance, but this cannot be verified from the reported data alone.

The Continued Pre-Training Cost and Difficulty Estimation Burden Are Not Accounted for in the Headline Efficiency Numbers

The assumption or constraint. The paper's headline efficiency claim — "10× memory savings and 2.65× faster inference on CPUs" — describes only the inference-time characteristics of the final deployed model (Section 1, Tables 1–2). The training cost of the BitDistill pipeline itself is not included in any efficiency comparison. Specifically: (1) the continued pre-training phase (Stage 2) trains the full FP16 model on 10B tokens before any quantization or task-specific fine-tuning occurs, and (2) the distillation phase (Stage 3) requires maintaining both the FP16 teacher and the 1.58-bit student in memory simultaneously, plus computing the attention relation matrices and KL divergences for the distillation losses. The paper does not report GPU-hours, total FLOPs, wall-clock training time, or peak memory usage for any stage of the pipeline (Section 4.1 notes only that training is conducted on 8× AMD Mi300X GPUs). The closest the paper comes to acknowledging this is the statement that the continued pre-training cost is "virtually negligible" compared to training BitNet from scratch on 4T tokens — but this is a relative comparison, not an absolute cost accounting.

The consequence. A practitioner evaluating whether to deploy BitDistill cannot perform a full cost-benefit analysis. The question "is it worth spending X GPU-hours on BitDistill training to achieve a 2.65× inference speedup on my downstream task?" cannot be answered from this paper because X is unknown. If a deployment requires processing billions of inference tokens, the training cost is amortized over many inferences and the inference speedup likely justifies any reasonable training cost. But if the deployment involves a small number of inference tokens (e.g., a research prototype, a low-traffic application), the training cost might dominate the total compute budget, making BitDistill less attractive than direct FP16 deployment or simpler PTQ methods that have near-zero training cost (at the expense of accuracy). The paper provides no framework for reasoning about this tradeoff, which is central to the deployment decisions the paper aims to enable.

Furthermore, the distillation phase (Stage 3) requires a separately fine-tuned FP16 teacher model, which itself incurs training cost (the FP16-SFT baseline). If the task-specific FP16 teacher was going to be trained anyway (as part of standard deployment), this cost is not incremental to BitDistill. But the paper does not clarify whether the FP16-SFT training cost is included in the implicit total, or whether the teacher is assumed to exist already.

What evidence exists in the paper. The paper provides no training cost measurements — no GPU-hours, no FLOPs counts, no wall-clock time for any stage. The only efficiency numbers are inference throughput (tokens/s on CPU) and memory (model size in GB) in Tables 1 and 2, which describe the deployed model and are identical for BitDistill and BitNet-SFT. The paper's discussion of cost in Section 3.2 is solely about the data volume (10B tokens vs. 4T tokens) without translating this into compute cost. For context, training a 0.6B-parameter model on 10B tokens on 8× AMD Mi300X GPUs — while not prohibitive — is a non-trivial compute expenditure for many potential users, and the paper provides no way to estimate it.

Mitigation status. The paper does not address this limitation. The cost framing is entirely relative to from-scratch BitNet training, and the absolute cost of the pipeline is not discussed. No suggestion is made for future work on cost reporting or cost optimization. The omission is significant because the paper's primary audience — practitioners wanting to deploy efficient models — needs cost numbers to make resource allocation decisions, and the paper provides only the inference-time half of the equation.

The Weight Distribution Reshaping Mechanism Is Correlational, Not Causally Validated, and Lacks Quantitative Metrics

The assumption or constraint. The paper's central mechanistic claim is that continued pre-training works by reshaping the weight distribution from an approximately Gaussian form (typical of FP16 pre-trained models) to a boundary-concentrated form (typical of BitNet trained from scratch, where mass concentrates near the transitions between -1/0 and 0/1), and that this boundary concentration "allows the quantized values to shift more frequently with small gradient steps, thereby enhancing the 1.58-bit BitNet's ability to fit downstream data and reducing the risk of being trapped in suboptimal local minima" (Section 4.4). This is a causal hypothesis about optimization dynamics. The paper supports it with (1) visual comparison of weight histograms in Figure 2, showing that the post-continued-training distributions resemble from-scratch BitNet distributions, and (2) the ablation in Table 5 showing that continued pre-training is the dominant contributor to performance recovery. The paper does not provide quantitative distribution similarity metrics, does not measure the frequency of ternary value flips during downstream training, and does not demonstrate that models with boundary-concentrated distributions outperform models with non-boundary-concentrated distributions when controlling for total training data.

The consequence. The mechanism is plausible but not proven. Alternative explanations for why continued pre-training helps are compatible with the evidence: (1) more training data (of any kind) provides a better initialization for the downstream optimizer, regardless of distribution shape — 10B tokens of general-domain training may simply move the model to a flatter region of the loss landscape that is more amenable to task-specific fine-tuning; (2) continued pre-training may improve the model's general language modeling capability (since it is trained on general-domain text), and this improved capability transfers to the downstream task independently of quantization effects; (3) continued pre-training may provide more gradient steps overall, preventing the optimizer from getting stuck in suboptimal minima through sheer optimization volume rather than through the specific mechanism of boundary concentration. The paper does not discriminate between these hypotheses and its preferred mechanism. If the mechanism is not boundary concentration but a more generic "more training helps" effect, then the paper's diagnostic contribution — that the scalability failure is a distributional adaptation problem — is weakened, and the practical implication (that continued pre-training should use general-domain data specifically to reshape the distribution) may be less prescriptive than claimed.

The absence of quantitative distribution metrics is particularly limiting. Figure 2 shows visual similarity between the from-scratch BitNet distributions and the post-continued-training distributions, but "looks similar" is not a measurement. The degree of similarity almost certainly varies across the seven projection matrices shown, and the paper does not quantify whether the matrices that are more "BitNet-like" after continued pre-training correspond to layers that benefit more from the continued pre-training. Without such quantification, the causal chain from distribution reshaping to performance improvement remains correlational.

What evidence exists in the paper. Figure 2 provides the visual evidence for distribution reshaping. Table 5 provides the performance evidence for continued pre-training's importance. The paper does not provide: (1) quantitative similarity metrics between distributions, (2) measurement of ternary value flip frequency during training for boundary-concentrated vs. non-boundary-concentrated initializations, (3) an experiment showing that alternative methods of achieving boundary concentration (e.g., initializing weights near boundaries directly) produce similar performance gains, or (4) a control experiment where continued pre-training is replaced with an equivalent amount of task-specific training data to test whether the general-domain nature of the data is important. The paper's claim about "shifting more frequently with small gradient steps" (Section 4.4) is an interpretation of the visual evidence rather than a measurement of optimizer dynamics.

Mitigation status. The paper does not acknowledge this as a limitation. The mechanism is presented as an explanation supported by evidence, not as a hypothesis requiring further validation. Section 4.4 discusses the mechanism in detail and presents it as the reason continued pre-training works, without qualifying the correlational nature of the evidence or proposing experiments to establish causality. The paper does not discuss alternative explanations or suggest that future work should validate the mechanism through more targeted experiments.

The Distillation Hyperparameters Span Five Orders of Magnitude Without Sensitivity Analysis or Transfer Guidance

The assumption or constraint. The distillation loss coefficients λ and γ take dramatically different values depending on the task type. For classification, λ = 10 and γ = 1 × 10⁵ (the attention distillation weight is five orders of magnitude larger than the logits distillation weight). For summarization, λ = 1 and γ = 1 × 10³ (a 100× reduction in the attention distillation weight and a 10× reduction in the logits distillation weight compared to classification). These values span five orders of magnitude across tasks, with γ in particular being extremely large in both cases (10³ to 10⁵). The paper reports these as the outcome of a "greedy search strategy to select the optimal learning rate and training epochs" but does not ablate the coefficient values themselves, provide sensitivity curves, or offer a principle for selecting them for a new task.

The consequence. A practitioner applying BitDistill to a new downstream task (beyond the GLUE classification and CNNDM summarization tasks tested) has no guidance on how to set λ and γ. The five-order-of-magnitude range between the classification and summarization settings suggests that these coefficients are highly task-dependent and that guesswork or expensive grid search is required to find workable values. Moreover, the extreme magnitude of γ (10⁵ for classification) raises questions about training stability and sensitivity: if the attention distillation loss is weighted 100,000 times more than the cross-entropy loss, the total loss is dominated by the distillation term, and the task-specific cross-entropy signal may be effectively irrelevant early in training. The paper does not discuss whether such an extreme weighting is necessary or whether more moderate values (e.g., γ = 10³ for classification) would produce nearly identical results. If the method is sensitive to γ within an order of magnitude, then practitioners face a high-variance hyperparameter that could cause training to fail if set incorrectly; if it is insensitive, the extreme value is an unnecessary peculiarity that obscures the method's true robustness. The paper provides no information to distinguish these scenarios.

The fact that λ and γ differ so dramatically between classification and summarization also suggests that the optimal values may depend on task characteristics beyond just the task type — factors such as output sequence length, vocabulary size, dataset size, or the relative importance of structural vs. output-level knowledge could all influence the optimal distillation weights. The paper does not analyze these dependencies or provide a transfer strategy.

What evidence exists in the paper. Section 4.1 reports the coefficient values for classification and summarization. Table 6 shows a distillation ablation for classification (comparing no distillation, logits only, attention only, and both) at the chosen coefficient values, demonstrating that both forms of distillation help. However, this ablation does not sweep the coefficient values — it only tests the combination of techniques at the fixed, pre-selected coefficients. The paper does not report experiments varying λ or γ, nor does it report the performance of using the classification coefficients on summarization or vice versa. Figure 3(b) explores layer selection for attention distillation (showing which layer to distill from) but does not explore coefficient sensitivity.

Mitigation status. The paper does not acknowledge this as a limitation. The coefficient values are reported as part of the training settings without qualification, and no analysis of their sensitivity or transferability is provided. The "greedy search" mentioned in Section 4.1 refers only to learning rate and training epochs, not to λ and γ. The paper does not suggest that coefficient selection is a challenge or propose methods for automating it. Given that these coefficients are the most extreme hyperparameter settings in the entire pipeline, the lack of sensitivity analysis is a significant gap for anyone attempting to reproduce or extend the method.

7. Implications and Future Directions

How This Work Changes the Landscape

BitDistill represents a feasibility demonstration that reframes a deployment bottleneck as a training-efficiency problem rather than a fundamental capacity limitation of ternary-weight models. This is not a paradigm shift — the individual components (SubLN, MiniLM-style attention distillation, STE-based QAT) are all adopted from prior work — but it is a diagnostic reframing with substantial practical consequences. Before this paper, the dominant narratives around 1.58-bit deployment were bifurcated: either you trained a BitNet from scratch on trillions of tokens (accurate but computationally prohibitive for most organizations) or you attempted post-training quantization or direct QAT (cheap but degraded, with the paper demonstrating 13-15 point accuracy drops that worsen with scale). BitDistill demonstrates a third path: take an existing pre-trained FP16 model, spend a modest additional training budget (10B tokens of general-domain continued pre-training, ~0.25% of the from-scratch cost, plus task-specific distillation fine-tuning), and arrive at a 1.58-bit model that performs within ~0-0.2 percentage points of the FP16 baseline on downstream tasks.

The shift is in what questions the field should ask about extreme quantization. The paper's central diagnostic — that the bottleneck is distributional adaptation (getting the weight distribution into a quantization-compatible shape) rather than representational capacity (ternary weights being unable to encode the necessary knowledge) — changes the research agenda. Prior work treated quantization as a compression problem: how much can you reduce precision before the model's representable function class loses essential structure? BitDistill reframes it for the conversion setting as an optimization dynamics problem: given that the architecture is capable (proven by from-scratch BitNet training), how do you get the pre-trained weights to a region of parameter space where ternary optimization is effective, without retraining from scratch? This is a fundamentally different question, and it directs attention away from better quantization functions (though Table 4 shows compatibility with GPTQ/AWQ, the gains are marginal) and toward better adaptation strategies — what data, in what order, with what objectives, gets the weights into the right distribution?

The paper also reconciles a latent tension between the BitNet literature and the model compression literature. The BitNet papers (Ma et al., 2024, 2025) demonstrated that ternary-weight LLMs can perform well, but only with from-scratch training — implicitly suggesting that conversion from FP16 was impractical. The QAT literature (EfficientQAT, LLM-QAT, BitDistiller) showed that quantized fine-tuning could recover some accuracy, but the results at sub-4-bit were underwhelming and the gap to FP16 remained substantial. The paper's diagnostic — that the missing ingredient was a distributional pre-adaptation phase using general-domain data — explains why neither approach alone sufficed. From-scratch training naturally develops the right weight distribution but is too expensive; direct QAT tries to recover accuracy without first adapting the distribution and fails. BitDistill's continued pre-training phase fills the gap that neither literature had identified as the primary bottleneck.

The paper makes research on extreme quantization conversion newly tractable by providing a baseline recipe that works. Before BitDistill, a researcher wanting to study 1.58-bit conversion of pre-trained models would face a discouraging initial result: direct QAT produces a 13-15 point accuracy gap. Is this gap because the approach is wrong, or because some specific sub-problem is unsolved? BitDistill answers: the approach can work if you solve the distributional adaptation problem first. This provides a stable baseline from which further improvements (better distillation strategies, more efficient adaptation, dynamic quantization schemes) can be measured, and it isolates the remaining challenges (summarization at larger scales, harder reasoning tasks, the continued pre-training budget question) so that subsequent work can target them directly.

Conversely, the paper reduces the attractiveness of certain research directions. The finding that more sophisticated quantization functions (GPTQ, AWQ, Block-Quant in Table 4) provide essentially no benefit over simple absmean quantization when used within the BitDistill framework suggests that quantization function design is not the binding constraint for 1.58-bit conversion — the training recipe dominates. Similarly, the finding that single-layer attention distillation outperforms all-layer distillation (Figure 3b) suggests that more distillation is not better and that research on ever-denser distillation strategies may be misallocated relative to research on when and where to apply distillation minimally. The paper also implicitly argues against the "just train from scratch" approach for organizations that already have access to pre-trained FP16 models: the 400× data reduction makes BitDistill the economically rational choice for downstream task deployment, and the community's resources might be better spent improving conversion pipelines than funding redundant from-scratch BitNet training runs.

Follow-Up Research This Work Enables

Scaling BitDistill to 7B–70B parameter models to test whether the scalability fix holds and whether the continued pre-training budget needs to scale proportionally. The paper demonstrates BitDistill up to 4B parameters and shows the BitNet-SFT gap widens from 13.9 to 15.3 points across 0.6B→4B, while BitDistill's gap to FP16 remains under 0.2 points at all scales. The critical open question is whether this holds at larger scales where the per-parameter gradient signal from the fixed 10B-token continued pre-training budget becomes even thinner. A strong follow-up would apply BitDistill to Qwen3-7B or LLaMA-3-8B on the same GLUE benchmarks, measuring both the BitNet-SFT gap (does it grow to 20+ points at 7B? does it asymptote?) and the BitDistill gap (does the fixed 10B-token budget remain sufficient, or does it need to scale?). If the gap re-emerges at larger scales, the experiment should sweep continued pre-training budgets (10B, 50B, 100B, 200B tokens) to determine the scaling relationship between model parameters and required adaptation data. This would transform the paper's qualitative "10B tokens works at 0.6B–4B" into a quantitative scaling law for distributional adaptation, analogous to how Chinchilla scaling laws describe pretraining compute allocation.

Measuring ternary value flip frequency during downstream training to causally validate the boundary-concentration mechanism. The paper's most important mechanistic claim — that continued pre-training helps by concentrating weights near quantization boundaries, enabling more frequent ternary value changes with small gradient steps — is supported only by visual correlation (Figure 2) and performance improvement (Table 5). A direct causal test would instrument the training process to count, per gradient step, how many weights flip between ternary values (-1→0, 0→1, etc.) for models with and without continued pre-training, and correlate flip frequency with downstream accuracy. Specifically: take the Qwen3-0.6B model at three checkpoints (no continued pre-training, 1B tokens, 10B tokens of continued pre-training), fine-tune each on MNLI for a fixed number of steps, and record per-layer ternary flip counts. If the mechanism is correct, models with higher flip counts should achieve higher final accuracy, and the flip count should increase monotonically with continued pre-training tokens (saturating when the distribution is fully adapted). A negative result — flip frequency not correlating with accuracy, or flip frequency not changing with continued pre-training — would falsify the mechanism and suggest alternative explanations (e.g., continued pre-training improves general language modeling rather than quantization-specific dynamics). This experiment is straightforward to implement with hooks into the quantization function and would elevate the mechanistic explanation from plausible hypothesis to empirically supported theory.

Testing BitDistill on complex reasoning tasks (GSM8K, MATH, BBH) where small precision losses might compound across multi-step inference. The paper's evaluation is confined to classification (GLUE) and extractive summarization (CNNDM) — tasks where the model produces relatively short outputs and the evaluation metrics are coarse (accuracy, BLEU/ROUGE on token overlap). A critical stress-test is whether BitDistill's 1.58-bit models maintain FP16-comparable performance on tasks requiring multi-step logical reasoning, where precision errors in intermediate computations could compound across steps. The follow-up would apply BitDistill to a base model with known reasoning capability (e.g., Qwen3-7B fine-tuned on GSM8K math word problems) and compare chain-of-thought accuracy between FP16-SFT, BitDistill, and BitNet-SFT. If BitDistill maintains FP16 accuracy on reasoning, it validates the method for a much broader deployment class; if it degrades, it reveals a fundamental limitation — ternary weights may preserve final-answer accuracy for classification but lose the precision needed for intermediate reasoning steps. This experiment would also test whether the attention distillation at a single layer (Figure 3b) remains sufficient when the teacher's internal reasoning chain matters, or whether multi-layer distillation becomes necessary for complex reasoning.

Ablating the continued pre-training data domain to determine whether distributional adaptation requires general-domain text or can be achieved with task-specific or synthetic data. The paper hypothesizes that continued pre-training works by reshaping the weight distribution, and it uses general-domain FALCON corpus data. But the mechanism — moving weights toward quantization boundaries through gradient-based training — might be achievable with any data that provides dense per-token gradients. A follow-up would compare three continued pre-training data sources at equal token budgets (e.g., 10B tokens each): (1) general-domain web text (the paper's FALCON approach, as a replication baseline), (2) task-specific data (e.g., repeated epochs on the MNLI training set, or aggregated GLUE training data), and (3) synthetic data (random token sequences or noise designed only to provide gradient steps without linguistic content). If task-specific data performs comparably, the continued pre-training could be folded into the downstream fine-tuning phase (eliminating a separate stage). If synthetic data works, the mechanism is purely about optimization dynamics (number of gradient steps) rather than data content. If only general-domain data works, it suggests that distributional adaptation requires broad linguistic coverage to reshape all projection matrices, which has implications for the minimum continued pre-training data mixture. This experiment directly tests the paper's claim that continued pre-training is "distributional adaptation" rather than simply "more training."

Developing a lightweight difficulty predictor for the continued pre-training budget to enable cost-aware deployment. The paper demonstrates that 10B tokens works for Qwen3 at 0.6B–4B, but provides no guidance for new model families or scales. A practical follow-up would train a small predictor model that takes as input (a) the base model architecture and scale, (b) a sample of the model's initial weight distribution statistics (mean, variance, kurtosis per projection matrix), and (c) the target downstream task type, and predicts the minimum continued pre-training token budget needed to achieve within-X% of FP16 accuracy. The training data would come from running BitDistill with varied continued pre-training budgets (1B, 2B, 5B, 10B, 20B, 50B tokens) across multiple model families (Qwen, Gemma, LLaMA) and scales, measuring the downstream accuracy at each budget, and fitting a regression from initial weight statistics to the budget-accuracy curve. If such a predictor is feasible, it would transform BitDistill from a fixed recipe into an adaptive pipeline that allocates continued pre-training compute based on measurable properties of the starting model — directly addressing the paper's unexamined assumption that 10B tokens is universally sufficient.

Combining BitDistill with dynamic inference-time quantization (mixed-precision across layers or tokens) to push beyond the 1.58-bit uniform constraint. The paper treats all weights uniformly: every linear projection is quantized to ternary {-1, 0, 1}. But Figure 2 shows that different projection matrices (Q, K, V, O, FFN up/gate/down) have different weight distributions after continued pre-training, and the ablation in Table 5 shows that SubLN (which affects normalization differently per projection) matters. This suggests that not all projections benefit equally from ternary quantization — some might tolerate 1.58-bit with minimal accuracy loss, while others might need higher precision (e.g., 2-bit or 4-bit) to preserve critical information. A follow-up would use the attention distillation loss per layer as a diagnostic: layers where the student's attention relation matrix diverges most from the teacher's (high KL divergence in Eq. 11) are candidates for higher precision, while layers where the student matches the teacher easily can remain at 1.58-bit. The experiment would apply BitDistill with per-projection or per-layer mixed precision, using the distillation loss as a guide for precision allocation, and measure whether this recovers the remaining ~0.1-0.2 point gap to FP16-SFT (Tables 1 and 2) or enables scaling to larger models without increasing the continued pre-training budget. This would test whether the uniform 1.58-bit constraint is optimal or a convenient simplification that leaves performance on the table.

Practical Applications and Downstream Use Cases

On-device text classification for resource-constrained edge devices (smartphones, IoT, embedded systems). The paper demonstrates that 1.58-bit BitDistill models achieve 88-96% accuracy on GLUE classification tasks (MNLI, QNLI, SST-2) across 0.6B–4B scales — within 0.2 points of FP16 baselines — while requiring only 0.11 GB of storage and running at 1,135 tokens/second on CPU with 16 threads. This is a directly deployable configuration for applications like on-device spam detection, sentiment analysis for user feedback, content moderation in messaging apps, or document categorization in mobile productivity suites. The 10× memory reduction (1.20 GB → 0.11 GB for a 0.6B model) means these models fit comfortably within the RAM budgets of mid-range smartphones (typically 4-8 GB total, with perhaps 1-2 GB available for ML models), and the 2.65× CPU speedup reduces latency and battery drain for inference-heavy workloads. A company currently running cloud-based classification APIs could switch to on-device BitDistill models, eliminating per-query API costs and network latency while maintaining accuracy. The paper's demonstration that larger FP16 teachers (Qwen3-4B) can distill into smaller 1.58-bit students (Qwen3-0.6B) that outperform same-size FP16 baselines (Figure 3c: ~89.5% vs. 88.01% on MNLI) means the deployment pipeline can use a powerful but expensive cloud model to train a compact on-device model that is actually better than what the small model could achieve on its own in FP16 — a compression-plus-improvement story rather than a compression-with-degradation one.

Cost-efficient batch summarization for content processing pipelines. On the CNNDM summarization task, BitDistill at 1.58-bit (using Qwen3-0.6B) achieves ROUGE-SUM of 27.44 compared to 27.58 for FP16-SFT — a difference of 0.14 points, well within the range where the summarization quality is likely indistinguishable to human evaluators. For organizations running large-scale document summarization pipelines (news aggregation services, legal document processing, scientific literature summarization), deploying BitDistill models on CPU-based inference clusters provides a 2.65× throughput improvement and 10× memory reduction per model instance, meaning the same hardware can serve ~2.65× more summarization requests per second or support ~10× more concurrent model instances in memory. If a summarization service currently processes 1 million documents per day using FP16 models on 100 CPU servers, switching to BitDistill could reduce the server count to ~38 (maintaining throughput) or increase capacity to 2.65 million documents/day on the same infrastructure. The fact that BitDistill occasionally exceeds FP16-SFT on individual metrics (BLEU: 14.41 vs. 13.98 for FP16-SFT in Table 2) suggests that the distillation process may provide a regularizing effect that improves generation quality beyond the FP16 teacher — a phenomenon worth monitoring in deployment but favorable for the business case. The main deployment risk is the unexamined scalability of summarization performance to larger model scales (the paper only reports 0.6B summarization results), so initial deployment should start at the 0.6B–1.7B range where classification evidence suggests the pipeline is reliable.

Teacher-student model compression for organizations with access to large FP16 models but deployment constraints. The paper's finding that a 0.6B 1.58-bit BitDistill student trained with a 4B FP16 teacher achieves ~89.5% MNLI accuracy — outperforming the 0.6B FP16 teacher's 88.01% (Figure 3c) — has direct implications for organizations that train or have API access to large models but need to deploy smaller ones. A cloud provider offering a 70B-parameter FP16 LLM API could use that model as a teacher to train 1.58-bit BitDistill students at 1.7B or 4B for specific high-volume customer tasks (customer support classification, ticket routing, content tagging), then offer those compressed models for on-premise deployment at a fraction of the inference cost. The 10× memory reduction means a 4B BitDistill model requires only ~0.4 GB of storage (extrapolating from the 0.11 GB for 0.6B reported in Table 1), making it deployable on commodity hardware or even edge servers. The key value proposition is that the student can exceed the performance of a same-sized FP16 model through distillation from a much larger teacher — the compression is not just lossy but potentially augmentative — while still delivering the 2.65× CPU speedup and 10× memory reduction. The paper's multi-family validation (Table 3: Qwen3, Qwen2.5, Gemma) suggests this pipeline is not architecture-locked, so organizations can apply it to their preferred base model family.

Rapid prototyping of task-specific efficient models without from-scratch training infrastructure. The paper's 400× data reduction (10B continued pre-training tokens vs. 4T tokens for from-scratch BitNet training) means that a small research lab or startup with access to 8× AMD Mi300X GPUs (the paper's training hardware, Section 4.1) can convert an existing FP16 model to a deployment-ready 1.58-bit model for a specific downstream task within a timeframe and budget that makes experimentation feasible. Assuming the continued pre-training of a 0.6B model on 10B tokens takes on the order of days on this hardware (the paper does not report exact training times, but this is a reasonable estimate for 512-token sequences with batch size 32), a team could iterate on task-specific model design — try different teacher models, different distillation layer selections, different task formulations — without the multi-week or multi-month timeline of from-scratch BitNet training. This lowers the barrier to entry for 1.58-bit deployment from "only large industrial labs" to "any group with access to a modest GPU cluster and existing pre-trained models," which is the community that most needs efficient deployment solutions. The paper's open-source code release (github.com/microsoft/BitNet, per the abstract) and the detailed Algorithm 1 for attention distillation provide the implementation scaffolding to make this practical.