Streaming DiLoCo with overlapping communication: Towards a Distributed Free Lunch

🎯 Pitch

Distributed training of large models can achieve the same quality as tightly-coupled data parallelism while swapping 400× fewer bits and using two orders of magnitude less peak bandwidth. The trick is to stream sparse parameter synchronizations in the background while training continues, combined with 4‑bit quantization, so that even a 100B-parameter model attains 95% compute utilization over bandwidths as low as 5 Gbit/s.

1. Executive Summary

This paper introduces Streaming DiLoCo, a set of three architectural improvements over the DiLoCo distributed training algorithm: synchronizing only subsets of parameters in sequence rather than all at once (reducing peak bandwidth), overlapping communication with computation (increasing tolerated latency), and quantizing outer gradients to 4-bit precision (reducing total data exchanged). The authors simulate and train Chinchilla-architecture language models from 35M to 4B parameters on C4 and Dolma, demonstrating that these combined innovations achieve comparable evaluation loss and downstream accuracy to bandwidth-costly data-parallel training while exchanging ~400× fewer total bits and reducing peak bandwidth by an order of magnitude—reaching 95% simulated compute utilization at bandwidths as low as ~5 Gbit/s for a 100B-parameter model, versus the ~300 Gbit/s required by data-parallelism. The method establishes that co-located synchronous training can be replaced by loosely-connected distributed training without trading off model quality, but only when the per-step compute time is long enough to absorb the overlapping communication latency, making larger models particularly well-suited to this approach.

2. Context and Motivation

The Core Problem: Data-Parallel Training Demands Intimate Co-Location

To understand what this paper addresses, we need to start with a fundamental constraint of modern deep learning: how do you train a model that is too large to fit on a single hardware accelerator? The standard solution for over a decade has been data-parallel training. In data-parallel training, you place copies of the model across multiple accelerators, split each mini-batch of data among them, have each copy independently compute its forward and backward passes, and then synchronize the gradients across all copies before applying an optimizer update. This means that every single training step requires an all-reduce operation—every device communicates its gradients to every other device, or equivalently, all gradients are summed and broadcast back.

The critical point, which is easy to overlook if you haven't worked with large-scale training, is that this all-reduce must happen very fast relative to the computation time. If communication takes too long, compute resources sit idle, waiting for gradients to arrive. This is measured as compute utilization (CU)—the fraction of wall-clock time actually spent on forward and backward passes, as opposed to waiting for communication. In an ideal world, CU = 1.0. In practice, maintaining high CU requires all accelerators to be:

Physically co-located in the same data center or even the same rack.
Connected by low-latency, high-bandwidth interconnects (e.g., NVLink, InfiniBand, specialized intra-datacenter networking).

The paper cites concrete numbers: for data-parallel training of 1B, 10B, and 100B parameter models, achieving reasonable compute utilization requires approximately 100, 200, and 300 Gbit/s of bandwidth, respectively (Figure 4). These are not consumer internet speeds—they are specialized, expensive, intra-datacenter networking fabrics.

Why This Problem Matters: The Economic and Engineering Cost of Co-Location

The requirement for co-location creates several compounding challenges that grow with model scale. The paper frames these as both economic and engineering problems:

Physical infrastructure costs. Building and maintaining a data center that can house tens of thousands of co-located accelerators is enormously expensive. As the authors note, "modern training runs, for example for large language models (LLM), may use tens of thousands of accelerators, and this number increases year after year. Building and maintaining a data-center that can co-locate that many accelerators is expensive and leads to increasingly complex engineering challenges" (Section 1). This isn't just about buying GPUs—it's about power delivery, cooling, physical space, and the networking fabric to connect them all at the required bandwidth.

Software engineering complexity. Orchestrating the passage of gradients, parameters, and intermediate states between devices at each optimization step while keeping all devices fully utilized is described as "technically challenging from a software engineering perspective" (Section 1). Modern training codebases are deeply complex, in part because they must carefully pipeline computation and communication to hide latency within a single step.

Failure amplification. As the number of devices involved in each synchronous training step grows, so does the probability that any one of them fails. A single device failure in a tightly synchronous data-parallel setup can halt the entire training run or introduce subtle numerical issues. The paper puts this directly: "the more devices that are used for each synchronous training step, the more chances there are that one of them fails, risking halting training, or introducing subtle numerical issues" (Section 1).

The hardware lottery. The paper invokes Hooker's (2020) concept of the "hardware lottery"—the idea that certain research directions win not because they are superior, but because they happen to be well-suited to the available hardware and software ecosystem:

"In our view, the ubiquity of co-located Data-Parallel training is likely due to the hardware lottery, when 'a research idea wins because it is suited to the available software and hardware and not because the idea is superior to alternative research directions'" (Section 5).

The authors argue that data-parallel training has benefited from decades of optimization by thousands of researchers, making it hard to beat through sheer engineering inertia, even though fundamentally more distributed approaches might be preferable.

A vision unrealized. The paper gestures toward a larger ambition: what if we could train models using compute resources that are not co-located? The authors imagine "training modular constellations of small models loosely connected across heterogeneous devices, using compute arbitrage spread world-wide" (Section 5). This vision—sometimes called "swarm training" or "decentralized training"—has been discussed for years but has remained impractical because the communication overhead of standard distributed training algorithms is prohibitive when workers are connected by consumer-grade or cross-data-center networking. Streaming DiLoCo is presented as a concrete step toward making this vision real.

Where Prior Approaches Fall Short

The paper's contributions build on several lines of prior work, each of which partially addresses the co-location problem but leaves critical gaps.

Prior Approach 1: Federated Averaging (FedAvg) and Local SGD

The foundational idea for reducing communication in distributed training comes from Federated Averaging (McMahan et al., 2017) and Local SGD (Stich, 2019). The core mechanism is straightforward: instead of synchronizing gradients every step, let each worker train independently on its local data for multiple steps, and only occasionally average the parameters across all workers. This reduces communication frequency, but the averaging step itself is still a full all-reduce—every parameter must be exchanged. When the averaging happens, it causes a communication burst that demands the same peak bandwidth as data-parallel training.

Why this is insufficient. Reducing communication frequency helps amortize the cost (if you synchronize every 100 steps instead of every step, you reduce total communication by 100×), but it doesn't help with peak bandwidth requirements. The network still needs to handle a full model's worth of parameters being exchanged in a short time window during each synchronization round. For large models, this burst can still overwhelm lower-bandwidth links, causing workers to block while waiting for the communication to complete.

Prior Approach 2: FedOpt and DiLoCo

FedOpt (Reddi et al., 2021) generalized FedAvg by introducing a bi-level optimization framework. Workers use an "inner optimizer" (e.g., SGD) for their local steps and an "outer optimizer" (e.g., Adam) to process the model deltas (pseudo-gradients) computed from the difference between the worker's current parameters and its parameters at the last synchronization. This is more sophisticated than simple averaging and can improve learning efficiency.

DiLoCo (Douillard et al., 2024a) is a specific, empirically successful instantiation of FedOpt applied to language model training. The inner optimizer is AdamW, and the outer optimizer is SGD with Nesterov momentum. DiLoCo demonstrated that training with infrequent outer synchronizations (every H = 500 steps in their original paper) can match data-parallel training quality for language models up to several billion parameters.

The algorithm (presented in the paper as Algorithm 1) works as follows:

$M$ replicas each train independently for $H$ steps.
At synchronization time, each replica computes an outer gradient: $\Delta^{(t)}_m = \theta^{(t-H)}_m - \theta^{(t)}_m$ , which is simply the net change in parameters over the $H$ inner steps.
These outer gradients are averaged across all replicas: $\Delta^{(t)} = \frac{1}{M} \sum_{m=1}^M \Delta^{(t)}_m$ .
An outer optimizer (SGD with Nesterov momentum) uses this averaged outer gradient to update the previously synchronized parameters.

What DiLoCo solves and doesn't solve. DiLoCo addresses the frequency problem: by communicating every $H$ steps instead of every step, it reduces total bandwidth by a factor of roughly $H$ . For $H = 100$ to $500$ , this is a substantial 100–500× reduction in total bits exchanged. However, DiLoCo inherits the same weakness as FedAvg: when communication happens, it's a full-parameter all-reduce. Every synchronization round causes a burst where the entire model's outer gradients must be exchanged. This creates two specific problems the paper identifies (Section 1):

"However, in these approaches, the synchronization typically requires an all-reduce operation which fully synchronizes the model parameters on some step. This all-reduce results in two main issues: 1) a large peak bandwidth, and 2) a blocking of the workers while they wait to receive updated weights."

The peak bandwidth problem means that even though total data volume is reduced, the network link still needs to be fast enough to handle the burst. If workers are connected by, say, a 1 Gbit/s link and the model is 10B parameters (40 GB in FP32), a single synchronization round would take over 300 seconds of pure transfer time—during which no computation happens.

The blocking problem means workers sit idle waiting for the synchronization to complete before they can resume training. This directly reduces compute utilization, which is the very thing we're trying to preserve.

Prior Approach 3: Gradient Compression

A complementary line of work compresses the gradients (or outer gradients) before communication. Techniques include random dropping (FedDropout; Wen et al., 2022), top-k sparsification (keeping only the largest gradient components), low-precision quantization (reducing from FP32 to FP16 or INT8), low-rank compression (PowerSGD; Vogels et al., 2019), and combinations thereof (CocktailSGD; Wang et al., 2023). These methods reduce total bits exchanged and can be applied orthogonally to DiLoCo-style infrequent synchronization.

What gradient compression doesn't address. Compression reduces data volume but doesn't change the synchronization pattern. The communication still happens as a burst, and workers still block during the exchange. More subtly, aggressive compression (especially sparsification-based methods like top-k or random dropping) can hurt model quality if not carefully tuned. The paper's ablation in Figure 11 demonstrates this directly: value-dropping methods (FedDropout, DARE, Top-K) consistently underperform low-precision quantization (FP4) for outer gradient compression in the DiLoCo setting.

Prior Approach 4: Partial Communication

Some prior works have explored communicating only subsets of the model at each synchronization round. FedPart (Arivazhagan et al., 2019) proposed personalization layers where some layers are shared globally and others remain local. WASH (Fournier et al., 2024) and Sparta (Baioumy and Cheema, 2025) proposed randomly sampling subsets of neurons to exchange. A concurrent work, also called FedPart (Wang et al., 2024), independently proposed synchronizing per-layer fragments similar to Streaming DiLoCo but with a critical difference: they argued that layers not being synchronized in a given round should be frozen (not updated during inner optimization).

What partial communication approaches miss. The paper identifies two limitations. First, the random subset approaches (WASH, Sparta) don't provide the structured, predictable communication pattern that enables the overlapping optimization the paper develops. Second, and more specifically for the concurrent FedPart work, freezing non-synchronized layers is "flops-inefficient" (Section 3.3.1). The paper runs a direct comparison: on an 18-layer model with 3 layers per fragment, freezing the 15 non-synchronized layers results in a 20% increase in evaluation loss (3.2145 vs. 2.6749). Since modern LLM training is compute-bound (the cost of FLOPS dominates, not the cost of communication), sacrificing computational efficiency to reduce communication is a bad trade-off.

Prior Approach 5: Asynchronous Variants

A natural extension is to allow workers to operate asynchronously—sending their updates when ready and applying received updates without blocking. Asynchronous Local SGD (Liu et al., 2024a) showed that DiLoCo's outer Nesterov optimizer can handle asynchronicity between workers of different speeds with simple modifications.

What remains unsolved. Asynchrony addresses the blocking problem but not the peak bandwidth problem. Workers still need to exchange full model-sized updates, both for sending and receiving. If the bandwidth is insufficient, the communication time for a single update may exceed the time between successive updates, causing unbounded staleness or dropped updates. Asynchrony alone cannot close the two-order-of-magnitude bandwidth gap the paper targets.

How This Paper Positions Itself

Streaming DiLoCo positions itself as a synthesis and extension of these prior ideas rather than a fundamentally new paradigm. It combines three well-motivated principles—partial communication, overlapping, and compression—into a single algorithm, and then demonstrates through simulation and experiment that the combination achieves something none of the individual techniques could: matching data-parallel training quality while reducing required inter-worker bandwidth by two orders of magnitude.

The paper's framing is explicitly incremental and engineering-focused. The contributions are presented as three modifications to DiLoCo (Section 1), each addressing a distinct limitation:

"Contribution 1: Synchronization. We synchronize subsets of parameters on a schedule, rather than all parameters at once. This contribution reduces the peak required bandwidth."

"Contribution 2: Overlapping. We overlap worker computation and communication of synchronizations. This contribution increases the tolerated latency of communication."

"Contribution 3: Quantization. We compress the outer gradients to four bits per parameters without loss of performance. This contribution reduces the total amount of bits exchanged."

What makes this combination novel—and potentially transformative—is not any single technique in isolation, but the observation that they interact synergistically. Streaming partial updates (Contribution 1) reduces peak bandwidth, but also creates a communication pattern that naturally interleaves with computation. This enables overlapping (Contribution 2), which would be much less effective if communication happened as a single large burst. Quantization (Contribution 3) reduces the data volume further, making the overlapping window more forgiving. Together, they transform DiLoCo from an algorithm that still sporadically demands high peak bandwidth to one that can operate smoothly over low-bandwidth links.

The paper also positions itself within a broader research agenda: bringing ideas from the federated learning literature to large-scale LLM training. The authors note that "the federated learning literature has mainly studied smaller scale models, primarily due to its focus on edge devices. There are huge opportunities for bringing the ideas from the federated learning literature to the new world of large scale training for LLMs" (Section 5). Streaming DiLoCo is presented not as a final solution, but as "a first step towards what we call a distributed free lunch"—the idea that distributed training could eventually offer the same model quality as co-located training without any bandwidth penalty.

A subtle but important aspect of the positioning is the emphasis on larger models benefiting more from the approach. The paper identifies that "the required bandwidth can become lower as the model scale gets larger when overlapping communication with computation, because longer compute step time (forward & backward) will provide more time to perform the synchronization across workers" (Section 3.1). This is described as exploiting the "square-cube law of distributed training" (Ryabinin et al., 2023), where computation scales as $O(n^3)$ (for a square matrix $n \times n$ ) while communication scales as $O(n^2)$ . Since model parameters grow as the square of width/depth, but forward/backward computation grows as the cube, larger models naturally have more computation time available to hide communication latency. This means Streaming DiLoCo is not just a workaround for small-scale distributed training—it actually becomes more effective at the largest scales, which is where the co-location problem is most acute.

3. Technical Approach

3.1 Reader Orientation

Streaming DiLoCo is a distributed training algorithm for large language models that enables multiple groups of accelerators (workers) to train collaboratively without needing to be co-located in the same data center. The system solves the problem that existing distributed methods like DiLoCo still demand high peak bandwidth during synchronization bursts: it replaces the all-at-once full-model exchange with a pipeline that streams small fragments of the model parameters continuously, overlaps communication with computation so workers never wait idly, and quantizes the exchanged data to 4 bits—all without degrading the final model quality.

3.2 Big-Picture Architecture (Diagram in Words)

The system consists of five major components that interact across two distinct timescales—fast inner training and slow outer synchronization:

Multiple DiLoCo replicas (workers): Each worker is a group of closely-located accelerators (e.g., within one rack) running standard data-parallel training with FSDP. Workers hold independent copies of the full model parameters and process different shards of the training data. They are the "computation engines."
Inner optimizer (AdamW): Within each worker, this optimizer performs standard per-step parameter updates on local data for many consecutive steps without any cross-worker communication. It operates exactly as in normal training—computing gradients via backpropagation and applying AdamW updates.
Fragmentation scheme: The model parameters are partitioned into $P$ non-overlapping fragments. A fragment consists of several consecutive or interleaved transformer layers (e.g., 3 layers per fragment). Fragments are the unit of communication—only one fragment is synchronized at a time, never the full model.
Outer optimizer (SGD with Nesterov momentum): When a fragment is synchronized, each worker computes its "outer gradient" (the net change in that fragment's parameters over $H$ inner steps), these are averaged across workers, and the outer optimizer applies a Nesterov momentum update to a globally shared copy of the parameters. Each fragment has its own outer optimizer state, and fragments are synchronized on staggered schedules.
Communication scheduler: A deterministic schedule controls which fragment synchronizes at which step, the per-worker offset delays ( $\tau$ ) that allow computation to overlap communication, and the 4-bit quantization applied to outer gradients before transmission. The schedule is designed so that fragment $p$ synchronizes exactly every $H$ steps, but different fragments are offset from each other so that some fragment is synchronizing at almost every inner step.

Information flow (step-by-step):

All workers start with identical randomly-initialized parameters.
Each worker trains independently for $H$ inner steps using its local data shard.
At step $t$ where a particular fragment $p$ is due for synchronization, each worker computes $\Delta^{(t)}_{m,p}$ (the outer gradient for that fragment), quantizes it to 4 bits, and sends it to all other workers (via an all-reduce).
Workers immediately continue training (they do NOT wait for the communication to complete).
After $\tau$ inner steps of overlap, the worker blocks to receive the averaged outer gradient for fragment $p$ , applies the outer optimizer to update a globally-shared copy of that fragment's parameters, and merges the result back into the active training parameters using a mixing factor $\alpha$ .

The key insight enabling this pipeline: because different fragments synchronize at different times, there is nearly always some fragment communicating in the background while computation proceeds on other fragments. The outer parameters for a given fragment serve as a "globally consistent anchor" that gets refreshed every $H$ steps, while the inner optimizer explores locally between synchronizations.

3.3 Roadmap for the Deep Dive

First, the base DiLoCo/FedOpt algorithm (Section 2.1 material): I'll walk through the exact mechanics of the inner-outer optimization loop, the definition of outer gradients, and why this formulation matters—since all three contributions modify this base algorithm.
Second, streaming partial updates (Contribution 1): How fragments are defined and scheduled, the mathematics of staggered synchronization, and why this reduces peak bandwidth without changing the per-fragment synchronization frequency.
Third, overlapping communication with computation (Contribution 2): The $\tau$ -delayed merge mechanism, the mixing factor $\alpha$ , and why the system remains stable and convergent despite merging stale parameters with actively-trained ones.
Fourth, low-precision outer gradients (Contribution 3): The E3M0 4-bit format, why accumulation in FP32 matters, and the empirical finding that value-dropping methods fail while low-precision quantization succeeds.
Fifth, memory management: How Streaming DiLoCo's 5× memory overhead (parameters + Adam state + outer parameters + outer optimizer state) is reduced to ~2% additional HBM at any moment through CPU offloading, and why this works with deterministic schedules.
Sixth, the compute utilization simulator: How the DAG-based simulation works, what it reveals about the square-cube law advantage for larger models, and the concrete bandwidth numbers that Streaming DiLoCo achieves versus baselines.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and algorithms paper that extends an existing distributed training algorithm (DiLoCo) with three pragmatic modifications targeting peak bandwidth, latency tolerance, and total data volume. The core idea is that by fragmenting the model, staggering synchronization, overlapping communication, and quantizing outer gradients, distributed training can match data-parallel quality while requiring two orders of magnitude less inter-worker bandwidth—and this advantage grows with model scale.

The Base Algorithm: DiLoCo/FedOpt (Algorithm 1)

Before understanding the three modifications, we must understand exactly what they modify. DiLoCo (Douillard et al., 2024a) is a specific instantiation of the FedOpt framework (Reddi et al., 2021) tailored for language model training. The algorithm operates as a bi-level optimization—there are two nested optimizers operating at different timescales.

The inner loop (lines 3–5 of Algorithm 1). For each of $M$ replicas (workers) in parallel, and for $T$ total steps of training:

Sample a minibatch $x$ from the worker's local data shard $\mathcal{D}_m$ .
Compute the loss $\mathcal{L} = f(x, \theta^{(t-1)}_m)$ , where $\theta^{(t-1)}_m$ represents the parameters of worker $m$ at the start of step $t$ .
Update parameters: $\theta^{(t)}_m = \text{InnerOpt}(\theta^{(t-1)}_m, \nabla \mathcal{L})$ , where InnerOpt is AdamW (Adam with decoupled weight decay; Loshchilov and Hutter, 2019).

This is identical to standard training—each worker independently performs SGD steps on its own data. The workers' parameters diverge during these inner steps because they process different data and make independent updates.

The outer loop (lines 6–10 of Algorithm 1). Every $H$ steps (where $H$ is the synchronization frequency, e.g., $H = 100$ ):

Each worker computes its outer gradient:

$\Delta^{(t)}_m = \theta^{(t-H)}_m - \theta^{(t)}_m$

where $\theta^{(t-H)}_m$ is the worker's parameters $H$ steps ago (immediately after the last synchronization), $\theta^{(t)}_m$ is the worker's current parameters after $H$ inner steps, and $\Delta^{(t)}_m$ is the net change over those $H$ steps.

What it computes: The outer gradient is simply the total displacement the worker's parameters have undergone during $H$ steps of local training. It is a vector in parameter space pointing from the synchronized parameters to the current local parameters. Critically, this is NOT the sum or average of per-step gradients—it is the difference between two parameter snapshots. This distinction matters because the outer gradient already encodes the cumulative effect of $H$ AdamW updates with their adaptive learning rates and momentum, condensed into a single update direction.

Why this form: Computing $\Delta^{(t)}_m$ as a parameter-space delta rather than accumulating per-step gradients serves two purposes. First, it is communication-efficient—you communicate one vector per $H$ steps regardless of what happened internally. Second, and more subtly, it captures the full effect of the inner optimizer's state (AdamW's momentum buffers, per-parameter learning rates, weight decay) without needing to communicate any of that state. The outer optimizer sees "what happened" to the parameters, not "what the raw gradients were."
All workers communicate to compute the average outer gradient:

$\Delta^{(t)} = \frac{1}{M} \sum_{m=1}^M \Delta^{(t)}_m$

This averaging is implemented as an all-reduce operation (line 8–9 of Algorithm 1). The paper notes that this communication "is as costly as in Data-Parallel, but instead of being executed at every step, it is done every $H$ steps, thus amortizing the communication cost" (Section 2.1).
Each worker applies the outer optimizer:

$\theta^{(t)}_m = \text{OuterOpt}(\theta^{(t-H)}_m, \Delta^{(t)})$

where OuterOpt is SGD with Nesterov momentum (Sutskever et al., 2013). The outer optimizer takes the pre-synchronization parameters $\theta^{(t-H)}_m$ (not the current $\theta^{(t)}_m$ ) and the averaged outer gradient $\Delta^{(t)}$ , and produces updated parameters. This is an important detail: the outer optimizer operates on the globally synchronized parameters, overwriting the local drift that occurred during inner training.

Why SGD with Nesterov momentum as the outer optimizer? The paper inherits this design choice from DiLoCo (Douillard et al., 2024a) and does not re-ablate it. In the FedOpt framework, the outer optimizer treats $\Delta^{(t)}$ as if it were a gradient, even though it is actually a parameter delta accumulated over many steps. SGD with Nesterov momentum provides a simple, robust update rule that does not require maintaining per-parameter adaptive state (unlike Adam), which would blow up memory and communication costs at the outer level. The Nesterov variant specifically provides accelerated convergence for the convex-like outer optimization landscape.

Key hyperparameters mentioned in the paper:

The outer learning rate is tuned to 0.4 at small scale and kept fixed across all scales (Section 3.2).
The synchronization frequency $H$ is varied: $H = 30$ for most scaling experiments and $H = 100$ for the overtrained Dolma experiments.
The number of replicas $M$ is 2 for most experiments, with ablations up to $M = 8$ and $M = 4$ for Dolma overtraining.

What DiLoCo achieves: By communicating every $H$ steps instead of every step, DiLoCo reduces total communication volume by a factor of approximately $H$ . However, the communication that does happen is a full-model all-reduce—every parameter must be exchanged in a burst. This is the bottleneck that Streaming DiLoCo addresses.

Streaming Partial Updates (Contribution 1)

The core idea. Instead of communicating the full outer gradient vector for all parameters in one burst every $H$ steps, Streaming DiLoCo partitions the model into $P$ fragments and communicates only one fragment at a time, but does so more frequently. Each fragment still synchronizes exactly every $H$ steps, but different fragments are offset from each other so that synchronizations are spread out in time.

Fragment definition. A fragment is a subset of the model's transformer layers. The paper studies two partitioning patterns, illustrated in Figure 2:

Sequential pattern: Each fragment comprises a contiguous block of consecutive transformer layers. For example, with 18 layers and fragment size 3, fragment 1 = layers 1–3, fragment 2 = layers 4–6, ..., fragment 6 = layers 16–18.
Strided pattern: Each fragment comprises interleaved layers. With 18 layers and fragment size 3, fragment 1 = layers {1, 7, 13}, fragment 2 = layers {2, 8, 14}, ..., fragment 6 = layers {6, 12, 18}.

The notation $\theta^{(t)}_{m,p}$ represents the parameters of fragment $p$ of worker $m$ at step $t$ . The embedding layer is treated as part of a fragment (the paper's cosine similarity analysis in Figure 18 shows embeddings have distinct synchronization behavior from other layers).

Why strided over sequential? The paper chooses the strided pattern as the default for three reasons (Section 2.2 and 3.3.1):

Slightly better ML performance at the fragment size they consider (3 layers per fragment, shown in Figure 6a).
Better spreading of up-to-date layers across network depth. In deeper networks, the strided pattern ensures that every fragment contains layers distributed throughout the model, meaning that after each synchronization, the model has recently-synchronized layers at all depths rather than having a contiguous block of fresh parameters at one end and stale parameters at the other.
Improved overlapping schedule. As shown in Figure 14, the strided pattern avoids having multiple early layers need synchronization simultaneously, which makes it easier to overlap their communication with the forward pass of the next step. This is a compute utilization advantage, not a learning advantage.

The scheduling algorithm. The key innovation is the synchronization schedule, formalized in Algorithm 2 (lines 6–9). For each fragment $p$ , a time offset $t_p$ is assigned. The condition for synchronizing fragment $p$ at step $t$ is:

$t - t_p \bmod H = 0$

What this computes: Fragment $p$ synchronizes whenever the step counter $t$ , minus the fragment's offset $t_p$ , is a multiple of $H$ . This means fragment $p$ always synchronizes exactly every $H$ steps, but different fragments synchronize at different absolute steps.

Concrete example from the paper: With $H = 100$ and $P = 2$ fragments, the first fragment has offset $t_{p=1} = 0$ , so it synchronizes at steps 100, 200, 300, ... The second fragment has offset $t_{p=2} = 50$ , so it synchronizes at steps 150, 250, 350, ... In this configuration, some fragment synchronizes every 50 steps, but each individual fragment synchronizes every 100 steps.

What changes for larger models. As the paper increases model scale, the fragment definition is maintained rather than rescaled. At 1B parameters with 24 layers, fragment size of 3 layers yields $P = 8$ fragments. At 10B with 48 layers, $P = 16$ fragments. At 100B with 108 layers, $P = 36$ fragments (Table 4). With $H = 100$ and $P = 36$ fragments, a fragment is synchronized on average every $100/36 \approx 2.8$ steps, meaning communication is nearly continuous, but any given fragment only synchronizes every 100 steps.

Peak bandwidth reduction. The peak communication volume—the amount of data that must be exchanged in a single synchronization event—is reduced from the full model size to the size of one fragment:

$\text{Peak reduction factor} = \frac{|\text{fragment}|}{L}$

where $|\text{fragment}|$ is the number of layers in one fragment and $L$ is the total number of layers. For the paper's chosen fragment size of 3 layers, this yields:

1B model (24 layers): 3/24 = 8× reduction in peak bandwidth
10B model (48 layers): 3/48 = 16× reduction
100B model (108 layers): 3/108 = 36× reduction (approximately; the paper uses these numbers in Table 4)

Total communication volume is unchanged. Streaming partial updates does not reduce the total number of bits exchanged—every fragment still synchronizes every $H$ steps with full outer gradients. It transforms a single large burst into many small bursts, reducing peak bandwidth while keeping total data volume constant. This is why Contributions 2 and 3 (overlapping and quantization) are necessary complements: overlapping makes the smaller bursts easier to hide, and quantization reduces the total volume.

Why not synchronize more frequently with smaller fragments? There is a trade-off visible in Figure 6. Smaller fragments reduce peak bandwidth more, but the paper finds that very small fragments (1 layer) degrade learning performance. The outer gradient for a single layer may not capture enough information for the outer optimizer to make meaningful updates, and the fragmentation may disrupt the model's internal representations. The paper selects 3 layers per fragment as a "desirable trade-off between ML performance and reduction of peak bandwidth" (Section 3.3.1).

Design choice: why soft synchronization? The paper considers and rejects an alternative from concurrent work (FedPart; Wang et al., 2024) that freezes non-synchronized layers during inner optimization. The authors argue this is "flops-inefficient": on an 18-layer model with 3 layers per fragment, 15 out of 18 layers (83%) would be frozen at any given time despite still requiring forward and backward computation. Their direct ablation (Section 3.3.1) shows that freezing non-synchronized layers increases evaluation loss from 2.6749 to 3.2145—a 20% degradation. Since LLM training is typically compute-bound (the limiting factor is FLOPS available, not communication bandwidth), sacrificing computational efficiency for communication reduction is the wrong trade-off.

Overlapping Communication with Computation (Contribution 2)

The problem. Even with streaming partial updates, synchronization events still block the worker: at the step when a fragment must be synchronized, the worker traditionally stops computing, sends its outer gradient, waits to receive the averaged outer gradient, applies the outer optimizer, and then resumes training. For a model with 36 fragments and $H = 100$ , a synchronization happens roughly every 2.8 steps, meaning workers could be blocked ~36% of the time if each synchronization takes non-negligible time.

The solution: delayed merge with a mixing factor. Streaming DiLoCo decouples the communication from the immediate application of the outer update. The mechanism is controlled by two parameters introduced in Algorithm 2 (lines 10–12):

$\tau$ : the number of inner steps that communication is allowed to overlap. $\tau$ must satisfy $0 < \tau < H$ .
$\alpha$ : the mixing factor that blends the locally-trained parameters with the globally-synchronized ones.

The procedure works as follows (using the notation from Algorithm 2):

Asynchronous send (line 8): At step $t$ when fragment $p$ is due for synchronization, the worker computes $\Delta^{(t)}_{m,p} = \theta^{(t-H)}_{m,p} - \theta^{(t)}_{m,p}$ and immediately sends it via async-send. The worker does NOT wait for the all-reduce to complete.
Continue training (lines 3-5): The worker immediately resumes inner optimization for the next $\tau - 1$ steps. During these steps, the outer gradient for fragment $p$ is being communicated in the background. The worker continues updating ALL parameters (including those in fragment $p$ ) using the inner optimizer.
Blocking receive (line 10): After $\tau - 1$ inner steps, the worker executes block-receive to retrieve the averaged outer gradient $\Delta^{(t-\tau)}_p$ . This is the outer gradient that was sent $\tau$ steps ago (the superscript $t-\tau$ indicates this).
Outer optimization (line 11): The worker applies the outer optimizer to update a globally-shared copy of the fragment:

$\tilde{\theta}^{(t)}_{m,p} = \text{OuterOpt}(\theta^{(t-\tau-H)}_{m,p}, \Delta^{(t-\tau)}_p)$

where $\theta^{(t-\tau-H)}_{m,p}$ is the fragment's parameters $H + \tau$ steps ago—the parameters BEFORE the inner training that produced the outer gradient. The OuterOpt (SGD with Nesterov momentum) takes this old snapshot and the averaged outer gradient to produce updated "global" parameters $\tilde{\theta}^{(t)}_{m,p}$ .
Merge with local parameters (line 12):

$\theta^{(t)}_{m,p} = \alpha \theta^{(t)}_{m,p} + (1 - \alpha) \tilde{\theta}^{(t)}_{m,p}$

where the left-hand $\theta^{(t)}_{m,p}$ is the new merged value, the first right-hand term $\theta^{(t)}_{m,p}$ is the worker's current local parameters for fragment $p$ after $\tau$ additional steps of inner training, and $\tilde{\theta}^{(t)}_{m,p}$ is the globally-synchronized parameters produced by the outer optimizer.

What the mixing equation computes: This is a linear interpolation between two versions of fragment $p$ : the "locally trained" version that has undergone $H + \tau$ steps of inner optimization since the last global synchronization, and the "globally synchronized" version that represents what the outer optimizer thinks the parameters should be based on information from all workers $H$ steps ago. The parameter $\alpha$ controls the interpolation:

$\alpha = 1$ : Discard the global synchronization entirely. Workers never exchange information. This is equivalent to independent training.
$\alpha = 0$ : Discard all local progress made during the $\tau$ overlap steps. Use only the globally synchronized parameters.
$\alpha = 0.5$ : Take the uniform average of the two.

Why this form: The mixing equation solves a subtle consistency problem. During the $\tau$ overlap steps, the worker has been training fragment $p$ using gradients computed from the pre-synchronization parameters. These gradients are based on slightly stale information—the worker hasn't yet incorporated knowledge from other workers about this fragment. The global parameters $\tilde{\theta}^{(t)}_{m,p}$ represent the consensus direction, but they're $\tau$ steps out of date. By blending, the worker keeps the benefit of its local exploration (which might have found useful updates) while being pulled back toward the consensus (preventing excessive drift).

The paper's ablation in Figure 8 shows that both $\alpha = 0$ and $\alpha = 0.5$ work well, with negligible degradation up to $\tau = 10$ inner steps of overlap (<0.2% increase in evaluation loss). The paper uses $\tau = 1$ in main experiments "for simplicity" (Section 3.3.2).

Why stopping at $\tau \approx 5$ is sufficient for compute utilization. Figure 9 shows the estimated compute utilization for a 100B model as $\tau$ increases. Compute utilization rises sharply from $\tau = 0$ to $\tau = 5$ and then plateaus—there is "little gain in compute time after an overlap of 5 inner steps" (Section 3.3.2). This is because the communication time for one fragment at 100B scale with streaming is small enough that a few steps of compute are sufficient to fully hide it. The paper therefore "advise[s] practitioners to limit the overlap to 5 inner steps" (Section 3.3.2).

Robustness to heterogeneous workers. A natural concern in distributed training is what happens when workers have different speeds (heterogeneous hardware, variable network latency, stragglers). The paper shows that Streaming DiLoCo handles this gracefully by allowing different workers to use different overlap delays $\tau$ . Figure 10 demonstrates that when worker 1 uses $\tau_1 = 1$ and worker 2 uses $\tau_2$ varying from 1 to 10, the evaluation loss degradation is limited for $\tau_2 \leq 5$ . This means workers can have "some slack" without forcing tight synchronization, making the method suitable for training across heterogeneous device types or environments with variable network conditions.

What happens to the outer parameters during overlap? An important subtlety: the outer optimizer's state (Nesterov momentum buffers) is applied to the snapshot $\theta^{(t-\tau-H)}_{m,p}$ , which is the parameters from $\tau + H$ steps ago. The worker has been training on fragment $p$ in the intervening $\tau$ steps, so there is a mismatch. The mixing step (line 12) resolves this by blending, but the outer optimizer itself only ever sees parameter snapshots spaced exactly $H$ apart—it never sees the intermediate steps. This preserves the clean mathematical structure of FedOpt (the outer optimizer always operates on parameters from exactly $H$ inner steps ago) while allowing overlap in wall-clock time.

Low-Precision Outer Gradients (Contribution 3)

What is being quantized. When a worker sends its outer gradient $\Delta^{(t)}_{m,p}$ to other workers (the async-send in line 8 of Algorithm 2), this vector is compressed before transmission. The paper uses a 4-bit floating-point format called E3M0: 1 sign bit, 3 exponent bits, and 0 mantissa bits (Agrawal et al., 2024). This format can represent only powers of two (the exponent determines the value, and with no mantissa, the precision is coarse). The outer gradient is computed and accumulated in FP32 for stability, but the transmission uses E3M0.

What this accomplishes. FP32 parameters occupy 32 bits each, so E3M0 compression reduces the per-parameter communication cost from 32 to 4 bits—an 8× reduction in data volume per synchronization, on top of the $H \times$ reduction from infrequent synchronization and the $L/|p| \times$ reduction from streaming fragments.

Why E3M0 specifically? The paper's ablation in Figure 11 compares two families of compression methods:

Value-dropping methods: FedDropout (Wen et al., 2022) randomly zeros out elements with probability $p$ , DARE (Yu et al., 2024) applies a similar random drop, and Top-K selection keeps only the K largest-magnitude elements. All three are lossy sparsification techniques.
Low-precision quantization: Reducing from FP32 to lower bit widths, specifically tested at FP4 (E3M0 format).

The results are striking (Figure 11a and 11b): lowering precision to FP4 does not affect performance at all (the C4 evaluation loss and HellaSwag accuracy curves for FP4 overlay nearly perfectly with FP32), while all value-dropping methods "significantly worse, particularly when zero-ing out at random" (Section 3.3.3). The paper also mentions preliminary experiments with TIES-Merging's pruning method (Yadav et al., 2023), which also underperformed.

Why does value-dropping fail? The paper doesn't provide a mechanistic explanation, but the likely reason relates to the structure of outer gradients. In DiLoCo, an outer gradient $\Delta^{(t)}_{m,p}$ is the net displacement of parameters over $H$ inner steps. Unlike per-step gradients (which are typically sparse and dominated by a few large components), accumulated outer gradients may have widespread, small-magnitude structure—every parameter has moved at least slightly after 100 AdamW steps. Randomly zeroing out elements destroys this distributed structure, effectively injecting noise into the global parameter update. Low-precision quantization, by contrast, preserves the structure of every element but reduces the fidelity—every parameter's update direction is retained, just with coarser magnitude representation.

Important detail: accumulation in FP32. The paper explicitly states that "once received by a replica, importantly, the accumulation is done in FP32 for stability" (Section 2.4, emphasis original). This means the all-reduce operation (summing outer gradients across $M$ workers) is performed in full precision. The E3M0 compression applies only to the point-to-point transmission of each worker's individual outer gradient, not to the accumulated result. This prevents quantization error from compounding during summation.

Interaction with streaming and overlapping. Quantization interacts synergistically with the other two contributions:

Streaming reduces each communication burst to a small fragment, but the fragments still need to be transmitted. Quantization makes each fragment 8× smaller, further reducing the transmission time and making it easier to hide via overlapping.
Overlapping requires that communication completes within $\tau$ inner steps. Quantization reduces the raw transmission time, meaning a smaller $\tau$ suffices or, equivalently, that the same $\tau$ can tolerate a lower-bandwidth link.
The combination means that for a given bandwidth budget, all three contributions together achieve far more than any one alone.

Why not quantize further? The paper tests only one quantization level (FP4) and doesn't explore lower bit widths (e.g., 2-bit, 1-bit). The choice of E3M0 specifically provides a sign bit and three exponent bits, which allows representing values across a wide dynamic range (important for outer gradients that may have components at very different scales across layers) while sacrificing all mantissa precision. The fact that FP4 works "without loss of performance" (Section 1) is presented as a positive finding—more aggressive quantization is left as future work.

Memory Management: The 5× Overhead That's Really a 2% Overhead

The memory overhead problem. DiLoCo and Streaming DiLoCo both require additional memory beyond standard data-parallel training. In an SPMD (Single Program, Multiple Data) model:

Data-Parallel memory: parameters (1×) + AdamW state (2×, for first and second moment estimates) = 3× the parameter count.
DiLoCo/Streaming DiLoCo memory: parameters (1×) + AdamW state (2×) + outer global parameters (1×) + outer Nesterov state (1×, Nesterov momentum requires a velocity buffer of the same size as the parameters) = 5× the parameter count.

This is a 66% (5/3) increase in total required memory, which could be prohibitive for large models where memory is already the binding constraint.

The key observation for Streaming DiLoCo. However, Streaming DiLoCo only ever needs a small portion of the outer parameters and outer optimizer state at any given time—specifically, the fragment that is currently being communicated and merged. The rest of the outer parameters and outer optimizer state can be offloaded from high-bandwidth memory (HBM, the GPU/TPU memory) to CPU memory (RAM) and paged in as needed.

Concrete calculation from the paper (Section 2.5). For a 100 billion parameter model with 108 layers and fragment size $|p| = 3$ layers:

Total parameters: 100B parameters × 4 bytes (FP32) = 400 GB
AdamW state: 2 × 400 GB = 800 GB
Total for inner training: 400 + 800 = 1,200 GB (approximately; the paper states 1,117 GB, accounting for some overhead)
Outer parameters (total): 400 GB
Outer Nesterov state (total): 400 GB
Size of one fragment's outer parameters: 400 GB × (3/108) ≈ 11.1 GB
Size of one fragment's Nesterov state: ≈ 11.1 GB
Additional HBM needed at any moment: 11.1 + 11.1 = 22.2 GB, which is approximately 2% of the 1,117 GB baseline.

How offloading works. The communication schedule for Streaming DiLoCo is deterministic and known before training begins (it's determined by $H$ , $P$ , and the fragment offsets $t_p$ ). This means the system knows exactly which fragment will be synchronized at which step. The paper describes the paging mechanism: "we can start the transfer from RAM to HBM of a fragment (and its associated outer optimizer state) while finishing the previous (inner) gradients passes" (Section 2.5). Since a fragment is small (3 layers), and the transfer happens asynchronously while computation proceeds on other layers, the transfer latency is hidden.

Transfer speed. The paper estimates the transfer time for a 100B model's fragment using an H100 GPU with PCIe, characterized by 2 TB/s bandwidth: the transfer of ~22 GB "is done in less than 10 milliseconds" (Section 2.5). Since the step time for a 100B model is approximately 4.9 seconds (Table 4), the transfer time represents about 0.2% of the step—negligible.

Why this matters for the "distributed free lunch" claim. The memory overhead discussion addresses a practical concern: if Streaming DiLoCo required 5× the HBM of data-parallel training, it would be unusable at the largest scales where HBM is the scarce resource. By offloading the outer parameters and optimizer state to CPU memory and paging in only what's needed, the method achieves its bandwidth reductions without a corresponding memory penalty. This is what makes the claim of "similar quality at negligible bandwidth" practically credible—the method doesn't secretly cost extra in memory to achieve the bandwidth savings.

The Compute Utilization Simulator (Section 3.1)

The paper validates the bandwidth benefits of Streaming DiLoCo through simulation before running actual training experiments. The simulation models training as a directed acyclic graph (DAG) of computational and communication nodes.

Simulation components. The DAG has four node types (Figure 3):

Blue nodes (forward pass): Computation of the forward pass through a single transformer layer. Each layer requires one forward node.
Light green nodes (backward w.r.t. activations): Computation of gradients of the loss with respect to the layer's activations.
Dark green nodes (backward w.r.t. parameters): Computation of gradients of the loss with respect to the layer's parameters.
Purple nodes (gradient reduction): Communication of (outer) gradients across workers. For data-parallel training, this is the standard all-reduce of per-step gradients. For DiLoCo and Streaming DiLoCo, this is the all-reduce of outer gradients.

How the graph represents one step. For an $L$ -layer model, one full training step is represented by a graph with $4 \times L - 1$ nodes (the minus one is because the first layer doesn't need backward w.r.t. activations—there's no layer before it to send activations to). Nodes have dependencies: a forward pass for layer $\ell$ must complete before the backward pass for layer $\ell$ can begin, and the backward pass w.r.t. parameters for layer $\ell$ must complete before the gradient reduction for layer $\ell$ can begin.

Graph-level representation for multi-step training. The overall training schedule is a larger DAG formed by connecting these per-step subgraphs. The connectivity between steps depends on the method:

Data-Parallel: Gradient reductions for layer $\ell$ at step $t$ must complete before the forward pass for layer $\ell$ at step $t+1$ (because updated parameters are needed). This enforces tight synchronization.
DiLoCo: Outer gradient reductions only occur every $H$ steps. Between synchronizations, steps are independent. At synchronization steps, the outer gradient reduction for ALL layers creates a burst of purple nodes that must complete before the next step can begin.
Streaming DiLoCo: Outer gradient reductions occur for different fragments at different steps. The purple nodes are staggered across steps.
Streaming DiLoCo w/ overlapping: The purple nodes for fragment $p$ are placed $\tau$ steps earlier than when their results are needed, and computation on other fragments proceeds during those $\tau$ steps.

How compute utilization is calculated. The simulator estimates the wall-clock time to execute the full DAG, accounting for:

The time each blue/green node takes (based on FLOPS profile, MFU assumptions, and hardware FLOPS/s).
The time each purple node takes (based on the amount of data to transfer and the available bandwidth).

Compute utilization (CU) is then:

$\text{CU} = \frac{\text{Time spent in blue + green nodes}}{\text{Total wall-clock time}}$

A CU of 0.8 means 80% of real time is spent computing, 20% waiting for communication. A CU of 1.0 means communication is perfectly hidden.

Model scale assumptions (Table 4). The simulator uses:

1B model: 24 layers, estimated step time = 0.1 seconds
10B model: 48 layers, estimated step time = 0.8 seconds
100B model: 108 layers, estimated step time = 4.9 seconds

These step times are estimated "based on the required flops using Kaplan et al. (2020) rule and using a MFU of 60%" (Table 4 caption). MFU (Model FLOPS Utilization) is the fraction of theoretical peak FLOPS actually achieved in practice—60% is a realistic assumption for well-optimized training.

Simulation results (Figure 4). The key findings:

Data-Parallel (blue curves): Requires very high bandwidth. To reach 95% CU, data-parallel needs approximately 100 Gbit/s for 1B, 200 Gbit/s for 10B, and 300 Gbit/s for 100B. At lower bandwidths, CU drops sharply.
Vanilla DiLoCo (orange curves): Shifts the curves left by roughly 100× (the reduction in communication frequency), but the curves have similar shape—they still require substantial bandwidth to reach high CU because of the burst nature of communication.
Streaming DiLoCo (green curves): Further left-shifts the curves because the peak bandwidth per synchronization is reduced. However, CU still asymptotes below 1.0 because the synchronizations still block computation.
Streaming DiLoCo w/ overlapping (red curves): Can reach CU = 1.0 (theoretically, with sufficient overlap). For the 100B model, 95% CU requires only ~5 Gbit/s.
Streaming DiLoCo w/ overlapping + FP4 (purple curves): An additional 8× left-shift. For the 100B model, 95% CU requires roughly 1–2 Gbit/s.

The square-cube law advantage. The paper observes a counter-intuitive phenomenon: "the required bandwidth can become lower as the model scale gets larger when overlapping communication with computation" (Section 3.1). This is explained by the square-cube law: computation (forward + backward) scales as $O(n^3)$ for $n \times n$ matrix multiplications, while communication (parameter exchange) scales as $O(n^2)$ . As models grow, the ratio of computation time to communication time increases, giving the overlapping mechanism more time to hide the communication.

This is visible in Figure 4: the red curve (Streaming DiLoCo w/ overlapping) shifts left as model scale increases. At 1B, reaching 95% CU requires ~9 Gbit/s. At 10B, it requires ~5 Gbit/s. At 100B, it requires ~5 Gbit/s (the plateau region widens). The purple curve (adding FP4) shifts this further: ~2 Gbit/s at 1B, ~1.4 Gbit/s at 10B, ~1 Gbit/s at 100B.

Simulation for Llama and DeepSeek (Figure 16). The paper extends the simulation to two real-world large-scale architectures:

Llama 405B (Grattafiori et al., 2024): Estimated step time of 26.9 seconds. Streaming DiLoCo with overlapped FP4 can reach near-100% CU at ~5 Gbit/s, compared to ~500 Gbit/s for Data-Parallel.
DeepSeek-V3 (DeepSeek-AI et al., 2024): 671B total parameters, 35B activated per token (mixture-of-experts), estimated step time of 20.1 seconds. Despite only 35B parameters being activated, the full 671B must be synchronized between replicas, massively increasing communication. Streaming DiLoCo with overlapped FP4 can reach near-100% CU at ~4 Gbit/s, compared to ~1 Tbit/s for Data-Parallel.

Limitations of the simulation. The paper is transparent about the simulation's limitations: "such simulation is not perfect because for instance we consider only the bandwidth between datacenters and not the local bandwidth between devices" (Section 3.1, Remark). It models cross-worker communication but assumes intra-worker communication (between devices within a worker) is free, which is a simplification. Nevertheless, they believe "this is still a useful tool to estimate device utilization" (Section 3.1), referencing Bonini's paradox—the idea that a model can be useful without being perfectly accurate.

Why the simulation matters. The simulation serves two purposes. First, it provides a clear, controlled comparison of the bandwidth benefits of each contribution in isolation, isolating the engineering effects from the learning effects. Second, it demonstrates that the benefits are robust across model scales and architectures (including MoE models like DeepSeek-V3, which have disproportionate communication costs), strengthening the claim that Streaming DiLoCo is a general solution rather than a quirk of small-scale experiments.

4. Key Insights and Innovations

Innovation 1: The Streaming Fragment Schedule as a Principled Decoupling of Communication Frequency and Peak Bandwidth

The most intellectually distinctive move in this paper is the observation that communication frequency and peak bandwidth—conflated in all prior DiLoCo/FedAvg-style methods—can be decoupled through a deterministic staggered schedule over model fragments. Before Streaming DiLoCo, the dominant assumption was that reducing communication costs meant reducing how often you communicate (larger H). This helps total volume but leaves peak bandwidth unchanged: when communication happens, the entire model must be exchanged. The paper introduces a different axis of optimization: fragment the model so that any individual synchronization event is small, but synchronizations happen more frequently, keeping the per-fragment interval H constant.

This is not an obvious reframing. The natural instinct when bandwidth is scarce is to communicate less often—make H as large as possible. But larger H has a learning cost: workers drift further apart, outer gradients become stale, and model quality degrades. The streaming approach sidesteps this trade-off entirely. Each fragment still synchronizes every H steps (preserving the learning dynamics that DiLoCo established work well), but the system as a whole communicates a fragment nearly every step for large models. The peak bandwidth is reduced by a factor of L/∣p∣ (layers per fragment), while the learning-relevant synchronization interval H is unchanged.

The conceptual breakthrough is treating the model's layers as independently synchronizable units with staggered schedules, rather than as an indivisible whole that must be synchronized atomically. This is qualitatively different from prior partial-communication approaches like FedPart (Arivazhagan et al., 2019; Wang et al., 2024), where some layers are permanently local and others global—a static partition. Streaming DiLoCo's schedule is dynamic: every layer participates in global synchronization, just at different times. The distinction matters because it preserves the mathematical structure of FedOpt (every parameter sees an outer update exactly every H steps) while transforming the communication pattern from burst to stream.

The evidence supporting this as a genuine innovation rather than an engineering trick is in Figure 4: vanilla DiLoCo (orange) and Streaming DiLoCo (green) both exchange the same total data volume, yet Streaming DiLoCo achieves substantially higher compute utilization at every bandwidth level. The improvement comes purely from reshaping the communication schedule—no bits were saved. This is a fundamental reframing of the distributed training communication problem, not an incremental refinement.

Innovation 2: Overlapping as a Convergence-Preserving Merge Operation, Not Just a Systems Hack

Overlapping communication with computation is an old idea in high-performance computing—pipelining data transfers behind computation to hide latency. What makes the paper's treatment distinctive is that it recognizes overlapping in the DiLoCo setting as a learning-theoretic problem, not merely a systems scheduling problem. The naive approach to overlapping would be: send the outer gradient, keep training, and when the response arrives, overwrite the current parameters with the globally synchronized ones, discarding the H + τ steps of local training that happened during the overlap. This is architecturally simple but learning-theoretically wasteful—why throw away τ steps of useful gradient information?

The paper's solution—the mixing equation θ(t)m,p = αθ(t)m,p + (1 − α)θ̃(t)m,p—is more than a pragmatic averaging trick. It formulates overlapping as a convex combination of two legitimate parameter estimates: the locally-trained parameters (which incorporate τ steps of recent gradient information but lack cross-worker consensus) and the globally-synchronized parameters (which incorporate cross-worker information but are τ steps stale). The mixing factor α is a hyperparameter that controls how much to trust recent local exploration versus slightly-stale global consensus.

What elevates this beyond a standard engineering contribution is the empirical finding in Figure 8: the degradation from overlapping is negligible (under 0.2% loss increase) for overlap windows up to τ = 10 inner steps, and the system is robust to heterogeneous τ across workers (Figure 10). This is non-obvious. One might expect that merging parameters from different points in optimization space would cause destructive interference—the locally-trained parameters have moved in some direction, the global parameters represent a different direction, and averaging them could land in a poor region of the loss landscape. The fact that this doesn't happen, even with τ delays of up to 5–10 steps and heterogeneous workers, suggests something deeper about the loss landscape geometry during DiLoCo training: the outer gradient direction and the local inner updates are sufficiently aligned that blending doesn't catastrophically interfere.

This finding connects implicitly to the linear mode connectivity literature (Frankle et al., 2020; Wortsman et al., 2022) that the paper reviews in Section 4: if independently-trained models can be averaged without a loss barrier, perhaps it's not surprising that parameters separated by only τ steps of divergent training can also be blended. But the paper doesn't lean on this connection explicitly, leaving it as an observation for future theoretical work.

The contribution is conceptual: it reframes overlapping from "hide latency by discarding computation" to "merge complementary parameter estimates from different points in a shared optimization trajectory." This is a fundamental shift in how to think about asynchronous distributed optimization, even though the mechanism is a simple linear interpolation.

Innovation 3: The Square-Cube Law as a Scaling Advantage, Not Just a Scaling Constraint

A widely-held intuition in distributed training is that larger models make communication harder—more parameters mean more data to exchange, higher bandwidth requirements, and worse scaling behavior. The paper inverts this intuition by identifying and exploiting what it calls (following Ryabinin et al., 2023) the square-cube law of distributed training: computation scales as O(n³) for n × n matrix operations, while communication scales as O(n²) for parameter exchange. In practical terms, when you double the model's width and depth, the per-step computation time grows much faster than the number of parameters that need to be communicated.

This is not a new mathematical observation—the asymptotic complexity of matrix multiplication versus parameter count has been understood for decades. What's distinctive is the paper's recognition that the square-cube law reverses the scaling relationship between model size and communication difficulty under the overlapping paradigm. Larger models have longer step times (0.1s → 0.8s → 4.9s for 1B → 10B → 100B in Table 4), which means the overlapping window naturally grows. More computation time per step means more time to hide communication in the background. Rather than larger models demanding more bandwidth, they demand less—the bandwidth required to reach 95% CU with the full Streaming DiLoCo (overlapped FP4) drops from ~2 Gbit/s at 1B to ~1 Gbit/s at 100B (Figure 4, purple curves).

This is a diagnostic concept with broad implications beyond this paper's specific method. It suggests that the entire research agenda around communication-efficient distributed training should be reframed: rather than asking "how can we reduce communication to make distributed training possible," we might ask "at what model scale does distributed training become communication-trivial?" If the trend holds, there exists a crossover point where the per-step computation time exceeds any realistic inter-worker communication latency, making distributed training essentially free in terms of bandwidth—the "distributed free lunch" the paper's title gestures toward.

The paper demonstrates this concretely for DeepSeek-V3 (Figure 16b), a 671B-parameter mixture-of-experts model where only 35B parameters are activated per token but the full 671B must be synchronized. Despite the massive parameter count, Streaming DiLoCo with overlapped FP4 achieves near-100% CU at ~4 Gbit/s—a bandwidth achievable with commodity data-center networking—while data-parallel requires ~1 Tbit/s. The square-cube law transforms a model that would be impossible to train in a distributed setting into one that is practically trainable. This is a fundamental insight about scaling behavior, not merely an incremental performance improvement.

Innovation 4: The Diagnostic Finding That Value-Dropping Fails for Outer Gradients Where Quantization Succeeds

The ablation in Figure 11 comparing compression methods for outer gradients constitutes a negative result with positive implications that qualifies as a genuine insight. Prior work on gradient compression (surveyed in Section 4) has extensively explored value-dropping methods: FedDropout randomly zeros gradient components, Top-K preserves only the largest magnitudes, and various sparsification schemes have been effective for per-step gradient compression in data-parallel training. One might naturally assume these methods would transfer to outer gradient compression in DiLoCo-style training.

The paper shows they don't. Value-dropping methods—random dropping (FedDropout, DARE) and Top-K selection—"significantly worse, particularly when zero-ing out at random" (Section 3.3.3), producing markedly higher evaluation loss compared to both FP32 and FP4 baselines. Meanwhile, aggressive 4-bit quantization (E3M0 format, with effectively only sign and exponent information, no mantissa precision at all) performs identically to FP32 communication. This is surprising and informative.

The diagnostic value lies in what it reveals about the structure of outer gradients. Per-step gradients in deep learning are typically sparse—most parameters receive small or zero gradients, with a few large components driving the update. This is why Top-K sparsification works: you can keep the top 1% of gradient components by magnitude and discard the rest with minimal impact. But outer gradients in DiLoCo are accumulated over H = 100 AdamW steps. Over that many updates, every parameter has moved by at least some amount. The outer gradient vector is dense—it represents the integrated effect of many small updates, not a sparse set of large ones. Random dropping destroys this distributed structure by injecting masking noise. Quantization preserves it, just at lower precision.

This finding implicitly challenges the transferability of techniques from the single-step gradient compression literature to the federated optimization setting. It suggests that outer gradients are a different mathematical object from per-step gradients—dense, structured, representing cumulative parameter displacement rather than instantaneous gradient direction—and compression methods should be designed accordingly. This is a conceptual contribution: a diagnostic finding that clarifies a boundary condition for when different compression strategies are appropriate, with implications for future work on communication-efficient federated and distributed training.

5. Experimental Analysis

Evaluation Methodology

Dataset. The primary dataset is C4 (Raffel et al., 2020), a colossal cleaned version of Common Crawl, used for scaling experiments from 35M to 4B parameters. For the overtraining experiments on a 1B parameter model, the paper uses Dolma (Soldaini et al., 2024), a more recent open corpus of three trillion tokens designed for language model pretraining research that better reflects modern overtrained regimes. The evaluation loss is reported on each dataset's validation split (C4 validation set or Dolma validation set). Downstream task evaluation uses HellaSwag (Zellers et al., 2019) for commonsense reasoning, Piqa (Bisk et al., 2020) for physical commonsense, and Arc-Easy (Clark et al., 2018) for science reasoning—all standard benchmarks in the LLM scaling literature.
Base model(s). All experiments use a Chinchilla-architecture (Hoffmann et al., 2022) decoder-only transformer with QKNorm (Henry et al., 2020) and Z-loss regularization with a factor of 1e−4 (Chowdhery et al., 2023) for training stability, following the setup of Wortsman et al. (2023) and Jaghouar et al. (2024a). Models are trained from scratch at six scales: 35M, 100M, 200M, 300M, 500M, 1B, and 4B parameters, all with a sequence length of 1,024 (C4) or 2,048 (Dolma). Architecture hyperparameters and Chinchilla-optimal token budgets are detailed in Table 2: for example, the 1B model uses 8,192 hidden dimension, 24 layers, 32 heads, and a 25B token budget; the 4B model uses 12,288 hidden dimension, 36 layers, 48 heads, and an 83B token budget. The vocabulary size is 32,000 across all scales. The models are trained with 2 DiLoCo replicas by default, each using FSDP (Zhao et al., 2023) internally across their respective closely-located devices.
Metrics. Three categories of metrics are reported:
1. Evaluation loss: The cross-entropy loss on the C4 or Dolma validation set. Lower is better. This is the primary scientific metric for assessing training quality, as it directly measures the model's language modeling capability without relying on downstream task proxies.
2. Downstream accuracy: Accuracy on HellaSwag, Piqa, and Arc-Easy. These provide evidence that the method does not sacrifice general reasoning capability for improved language modeling loss. HellaSwag is typically reported as percentage accuracy, as are Piqa and Arc-Easy.
3. Simulated compute utilization (CU): The fraction of wall-clock time spent on forward and backward computation versus waiting for communication, as estimated by the DAG simulator (Section 3.1). This is reported as a decimal between 0 and 1, with 1.0 being perfect compute utilization. While not a direct empirical measurement, CU provides the bandwidth-to-utilization mapping that grounds the paper's practical claims.
Baselines. The paper compares against four methods:
1. Data-Parallel: Standard synchronous distributed training where gradients are all-reduced across all workers at every step. This represents the "gold standard" for training quality at the cost of maximum communication.
2. DiLoCo (Douillard et al., 2024a): The base algorithm without any of the three proposed modifications. Uses H = 30 inner steps (for scaling experiments) and communicates full outer gradients via all-reduce at each synchronization.
3. Streaming DiLoCo (ablated variants): Versions of the proposed method with individual contributions removed (e.g., no overlapping, no quantization, sequential vs. strided patterns, different fragment sizes).
4. FedPart (Wang et al., 2024): A concurrent partial-communication method that freezes non-synchronized layers during inner optimization—tested only in ablation (Section 3.3.1) to validate the paper's design choice not to freeze layers.
Generation budget / compute accounting. The paper uses total flops as the primary compute measure for scaling experiments, with each model scale trained for a Chinchilla-optimal number of tokens as specified in Table 2. For instance, a 35M model uses 1.5e17 flops, scaling to 2e21 flops for the 4B model. For the overtraining experiments on Dolma (Table 1), three token budgets are used: 25B (1.9e20 flops), 100B (7.6e20 flops), and 250B tokens (1.9e21 flops) for the 1B model. The compute utilization simulator uses additional accounting: step time is estimated based on flops profile, a 60% MFU (model flops utilization) assumption, and hardware theoretical flops per second. Communication time is estimated based on the volume of data exchanged and available bandwidth. For all DiLoCo variants, the number of inner steps H is a key parameter controlling the communication-computation ratio: H = 30 is used for scaling experiments, H = 100 for overtraining, and H is ablated from 10 to 500 (Figure 13, Appendix).
Cross-validation / statistical protocol. The paper does not employ cross-validation in the standard machine learning sense (no hyperparameter sweep over validation folds). Instead, it uses a fixed parametrization across scales: the outer learning rate is tuned to 0.4 at small scale and kept fixed across all scales "for simplicity, and to show that Streaming DiLoCo is a drop-in replacement of DiLoCo" (Section 3.2). The fragment size of 3 layers is chosen based on the ablation in Figure 6 and fixed across all model scales. The number of replicas M = 2 is used by default, with ablations at M = 4 and M = 8 (Figure 12 and Table 6). The paper explicitly notes that these hyperparameters are held constant rather than re-tuned at each scale, which is both a strength (demonstrating robustness) and a potential weakness (suboptimal performance at some scales may be due to suboptimal hyperparameters rather than fundamental algorithmic limitations).

Main Quantitative Results

Scaling Experiments on C4 (Figure 5, Table 5)

The primary scaling results span models from 35M to 4B parameters on C4 with Chinchilla-optimal token budgets. The headline finding is that Streaming DiLoCo matches Data-Parallel and DiLoCo in both evaluation loss and downstream accuracy across all scales, while requiring substantially less peak bandwidth.

At the 1B parameter scale, the evaluation losses are nearly identical: Data-Parallel achieves 2.49, DiLoCo with H=30 achieves 2.49, and Streaming DiLoCo with overlapped FP4 communication and H=30 achieves 2.48—a difference of 0.01, well within experimental noise. HellaSwag accuracy at the same scale is 46.60% for Data-Parallel, 46.56% for DiLoCo H=30, and 46.60% for Streaming DiLoCo with overlapped FP4 and H=30—essentially identical (Table 5). Streaming DiLoCo with H=100 shows a slight degradation at smaller scales (evaluation loss of 2.50 vs. 2.48 for H=30 at 1B, and HellaSwag of 46.00% vs. 46.60%) but the paper notes that "the loss improves proportionally better as we scale" (Section 3.2.1). The scaling law slopes reported in the text are −0.13149 for Data-Parallel and −0.13539 for Streaming DiLoCo—Streaming DiLoCo actually shows a slightly steeper (better) scaling trend, though the difference is small.

At the 4B parameter scale (the largest trained), Data-Parallel achieves an evaluation loss of 2.25 while Streaming DiLoCo with H=100 achieves 2.26—a gap of only 0.01 (Table 5). HellaSwag accuracy is 59.56% for Data-Parallel versus 59.02% for Streaming DiLoCo H=100—a difference of 0.54 percentage points. Piqa and Arc-Easy show similarly tight alignment (72.42% vs. 72.52% for Piqa, 43.51% vs. 43.16% for Arc-Easy).

These results establish the paper's core learning claim: Streaming DiLoCo is ML-equivalent to Data-Parallel training across a 100× range of model scales (35M to 4B parameters). The three modifications—streaming partial updates, overlapping, and FP4 quantization—do not degrade learning dynamics, even when pushing the synchronization interval to H=100.

Overtraining on Dolma (Table 1)

The C4 experiments use Chinchilla-optimal token budgets, but modern LLMs are typically overtrained—trained on many more tokens than the Chinchilla formula recommends (Gadre et al., 2024). The Dolma experiments test whether Streaming DiLoCo's equivalence to Data-Parallel holds under overtrained conditions. Using a 1B parameter model with token budgets of 25B, 100B, and 250B tokens:

25B tokens: Data-Parallel achieves eval loss of 2.67 and HellaSwag of 42.09%. Streaming DiLoCo with overlapped FP4 achieves 2.66 and 42.08%—slightly better on loss, identical on HellaSwag.
100B tokens: Data-Parallel achieves eval loss of 2.52 and HellaSwag of 49.78%. Streaming DiLoCo achieves 2.51 and 49.98%—again slightly better on both metrics.
250B tokens: Data-Parallel achieves eval loss of 2.45 and HellaSwag of 53.86%. Streaming DiLoCo achieves 2.45 and 54.24%—identical on loss, slightly better on HellaSwag.

Piqa and Arc-Easy follow similar patterns, with Streaming DiLoCo sometimes slightly ahead and sometimes slightly behind Data-Parallel, with differences typically under 1 percentage point. The consistency across all three token budgets demonstrates that the equivalence holds not just for compute-optimal training but for the overtrained regimes typical of modern LLM deployment.

The paper also reports three practical metrics that are the central motivation for the work:

Total terabytes exchanged: Data-Parallel exchanges 441 TB at 25B tokens, 1,767 TB at 100B, and 4,418 TB at 250B. Streaming DiLoCo with overlapped FP4 exchanges 1.10 TB, 4.42 TB, and 11.05 TB respectively—a consistent 400× reduction in total data volume across all three budgets. This is the combined effect of H=100 (100× reduction from infrequent synchronization), 24/3 = 8× reduction from streaming fragments, and 8× reduction from FP4 quantization (100 × 8 × 8 = 6,400 in theory, but the realized 400× reflects that the baseline comparison accounts for gradient communication, not parameter communication, and other factors).
Peak bandwidth reduction: At 1B parameters with 24 layers and fragment size 3, the peak bandwidth is reduced by a factor of num_layers / fragment_size = 24/3 = 8× compared to DiLoCo's full-model bursts (Table 1 notes).
Tolerated latency: Data-Parallel "ideally hopes for a 0 second latency when communicating," while Streaming DiLoCo's overlapping scheme allows "a latency as long as a full forward/backward pass, which is several seconds at large scale" (Table 1 notes). At 1B parameters, the step time is 0.1 seconds, meaning overlapping can tolerate ~100 ms of communication latency with no compute utilization penalty. At 100B, the step time is 4.9 seconds, meaning overlapping can tolerate nearly 5 seconds of latency.

Compute Utilization Simulation (Figure 4, Table 4)

The simulation results quantify the bandwidth requirements for each method to achieve a given level of compute utilization. The key numbers are presented in Table 4:

For a 1B parameter model:

Data-Parallel needs 86.8 Gbit/s to reach 50% CU, 152.6 Gbit/s for 80%, and 569.0 Gbit/s for 99%.
Streaming DiLoCo with overlapped FP4 needs only 0.4 Gbit/s for 50%, 0.9 Gbit/s for 80%, and 3.0 Gbit/s for 99%.
The reduction at 95% CU is from 222.3 Gbit/s to 2.0 Gbit/s—roughly 110× less bandwidth.

For a 10B parameter model:

Data-Parallel needs 104.8 Gbit/s for 50% CU and 471.5 Gbit/s for 99%.
Streaming DiLoCo with overlapped FP4 needs 0.4 Gbit/s for 50% and 1.7 Gbit/s for 99%.
At 95% CU, the reduction is from 268.3 Gbit/s to 1.4 Gbit/s—roughly 190× less bandwidth.

For a 100B parameter model:

Data-Parallel needs 184.2 Gbit/s for 50% CU and 471.5 Gbit/s for 99% (the 99% number is the same as 10B because it's limited by the step time assumption, not communication).
Streaming DiLoCo with overlapped FP4 needs 0.5 Gbit/s for 50% and 1.4 Gbit/s for 99%.
At 95% CU, the reduction is from 390.7 Gbit/s to 1.1 Gbit/s—roughly 350× less bandwidth.

The absolute bandwidth required by the full Streaming DiLoCo method is remarkably low and roughly constant across model scales for a given CU target: approximately 1–2 Gbit/s for 95% CU regardless of model size. This is the "square-cube law" effect the paper emphasizes—larger models have longer step times, which gives more time to hide communication, offsetting the increased parameter count.

Figure 4 visualizes these numbers as curves, showing four key patterns:

All Data-Parallel curves (blue) require high bandwidth and drop sharply below ~50 Gbit/s.
Vanilla DiLoCo (orange) shifts the curves substantially left but retains the same shape—the burst communication pattern still causes sharp CU degradation at lower bandwidths.
Streaming DiLoCo (green) and Streaming DiLoCo with overlapping (red) show progressively better scaling at low bandwidths, with the red curves able to reach CU = 1.0.
Adding FP4 quantization (purple) provides a further leftward shift, with the purple curve for the 100B model reaching near-perfect CU at bandwidths as low as 1 Gbit/s.

Compute Utilization on Large-Scale Architectures (Figure 16)

Extending the simulation to two real-world architectures:

Llama 405B (Figure 16a): With an estimated step time of 26.9 seconds, Streaming DiLoCo with overlapped FP4 can achieve near-100% CU at approximately 5 Gbit/s, while Data-Parallel requires approximately 500 Gbit/s for 95% CU. The two-order-of-magnitude gap is maintained at the largest scales.

DeepSeek-V3 (Figure 16b): Despite having 671B total parameters (with 35B activated per token via mixture-of-experts), the full 671B parameters must be synchronized between replicas, creating a massive communication burden. Streaming DiLoCo with overlapped FP4 achieves near-100% CU at approximately 4 Gbit/s. Data-Parallel requires approximately 1,000 Gbit/s (1 Tbit/s) for 95% CU. The bandwidth reduction here is approximately 250×. This is particularly significant because DeepSeek-V3's MoE architecture makes the communication-to-computation ratio worse than a dense model—the square-cube law works against MoE models because parameters scale faster than computation per token. Yet Streaming DiLoCo still reduces the bandwidth requirement to levels achievable with commodity networking.

Scaling with Variable Number of Replicas (Figure 12, Tables 6–7)

Increasing the number of DiLoCo replicas M is not mathematically equivalent to increasing data-parallel workers, because each replica performs independent local training that causes parameter divergence. The paper tests M = {2, 4, 8} replicas under two settings:

Constant global batch size (Figure 12a, Table 6): Increasing M from 2 to 4 while keeping the global batch size fixed means each replica gets half the local batch size. At 1B parameters with H=30, evaluation loss degrades from 2.48 (M=2) to 2.50 (M=4), and HellaSwag drops from 46.60% to 45.25%. The degradation is modest but consistent across scales—e.g., at 500M parameters, loss goes from 2.67 to 2.70, HellaSwag from 38.10% to 36.95%. With H=100, the degradation is slightly larger: at 1B, loss goes from 2.50 to 2.53, HellaSwag from 46.00% to 44.74%.

Constant local batch size (Figure 12b): Keeping each replica's batch size fixed means increasing M increases the global batch size (more data processed per step) and proportionally reduces the total number of training steps. This configuration shows similar trends—increasing M modestly degrades performance.

Overtraining with M=4 on Dolma (Table 7): At 1B parameters with M=4 replicas, Streaming DiLoCo with overlapped FP4 still matches Data-Parallel closely. At 250B tokens, Data-Parallel achieves eval loss 2.45, HellaSwag 53.86%; Streaming DiLoCo M=4 achieves 2.47, HellaSwag 52.20%. The gap is slightly wider with M=4 than M=2 (which achieved 2.45 and 54.24% respectively), but remains small—roughly 1.6 percentage points on HellaSwag. The terabytes exchanged are even lower with M=4 (0.55 TB vs 1.10 TB at 25B tokens) because each replica's outer gradient contributes less to the total communication when there are more replicas (the all-reduce sums them, but the per-replica send volume is the same).

Outer Gradient Cosine Similarity Analysis (Figures 17–18, Appendix)

This analysis characterizes the degree of agreement between replicas' outer gradients during training, which is diagnostic for understanding why DiLoCo-style methods work. The cosine similarity between replicas' outer gradients for all parameters except embeddings (Figure 17a) starts around 0.1 (slightly positively correlated), hovers near 0.0 through most of training (roughly orthogonal), and ends around −0.1 (slightly anti-correlated) as training enters the fluctuation phase. Larger models show consistently lower cosine similarity throughout training.

The per-layer cosine similarity (Figure 18) reveals a striking pattern: the first transformer layer at every scale has significantly higher similarity than all subsequent layers. The paper does not elaborate on why this occurs, but a plausible interpretation is that the first layer learns task-general representations (token embeddings, basic syntactic patterns) that are more consistent across different data shards, while deeper layers learn more data-specific features that diverge under independent training. The low overall cosine similarity (hovering near zero) indicates that replicas are genuinely exploring different regions of parameter space during their independent training phases, which is essential for the outer averaging step to provide meaningful diversity gain—if replicas always moved in the same direction, the outer synchronization would be redundant with simply training longer on one replica.

Ablation Studies and Robustness Checks

Fragment size (number of synced layers per fragment): Figure 6 shows the trade-off between learning performance and peak bandwidth reduction as fragment size varies. Fragment sizes of 1 layer produce peak bandwidth reduction of 24× but degrade evaluation loss (e.g., approximately 2.73 vs. 2.67 for 3 layers at 500M scale—extrapolating from the figure since exact numbers aren't quoted in text). Fragment sizes of 6 layers show better ML performance but only 4× peak bandwidth reduction. The paper selects 3 layers per fragment as the sweet spot, which they describe as "striking a desirable trade-off between ML performance and reduction of peak bandwidth" and use "across all model scales" (Section 3.3.1). This choice means that larger models naturally have more fragments (8 fragments at 1B, 16 at 10B, 36 at 100B), which increases communication frequency per step but keeps the per-fragment synchronization interval H constant.

Sequential vs. strided fragment patterns: Figure 6a shows that the strided pattern slightly outperforms sequential at smaller fragment sizes, while Figure 7 shows the strided pattern achieves better compute utilization for the 100B model simulation. The paper's Figure 14 provides a visual explanation: in the sequential pattern, multiple early layers can end up in the same fragment, creating a situation where all early layers need synchronization simultaneously, making it harder to overlap communication with the forward pass of the next step. The strided pattern distributes layers evenly, avoiding this clustering. The paper adopts strided as the default based on these combined advantages.

Freezing non-synchronized layers (FedPart comparison): Section 3.3.1 directly compares Streaming DiLoCo with and without the frozen-layer pattern proposed by concurrent work FedPart (Wang et al., 2024). On an 18-layer model with 3 layers per fragment, freezing the 15 layers that won't be synchronized at a given round results in an evaluation loss of 3.2145, compared to 2.6749 for Streaming DiLoCo without freezing—a 20% increase in evaluation loss. The paper attributes this to "flop-inefficiency": 83% of the model's parameters (15/18 layers) undergo forward and backward computation but aren't updated, wasting the majority of compute. Since modern LLM training is compute-bound (the cost is dominated by FLOPS, not communication), this inefficiency is deemed unacceptable.

Number of overlapping steps (τ): Figure 8 shows evaluation loss as τ varies from 1 to 20, with two mixing factors α = 0 (discard local updates during overlap) and α = 0.5 (uniform average). For α = 0, the degradation is "negligible up to an overlap of 10 inner steps (<0.2%)" (Section 3.3.2). For α = 0.5, degradation is similarly minimal. Figure 9 complements this by showing compute utilization for a 100B model as τ increases: CU rises sharply from τ = 0 to τ = 5 and plateaus thereafter. The paper therefore advises practitioners to limit overlap to 5 inner steps, and uses τ = 1 "for simplicity" in the main experiments (Section 3.3.2).

Heterogeneous overlap delays across workers (τ₁ ≠ τ₂): Figure 10 tests robustness when workers use different overlap delays, as might happen with heterogeneous hardware or asynchronous training (Liu et al., 2024a). With τ₁ = 1 fixed and τ₂ varying from 1 to 10, and α = 0.5, the evaluation loss degradation is "limited under a delay of up to 5 inner steps" (Section 3.3.2). At τ₂ = 10, the loss increases more noticeably. This suggests Streaming DiLoCo can tolerate moderate asynchronicity between workers without requiring lockstep synchronization, supporting training across heterogeneous devices or network conditions.

Compression method choice (value-dropping vs. quantization): Figure 11 provides a systematic comparison of compression strategies for outer gradients. Lowering precision from FP32 to FP4 (E3M0 format) shows no performance degradation on either C4 evaluation loss or HellaSwag accuracy—the FP4 curve overlays the FP32 curve nearly perfectly. In contrast, all value-dropping methods are significantly worse: FedDropout (random zeroing) and DARE show elevated loss and reduced accuracy, while Top-K selection degrades even more sharply, particularly when the kept fraction is small. The paper also mentions preliminary experiments with TIES-Merging's pruning (Yadav et al., 2023) that "also underperformed." An important qualification: TIES-Merging "might become advantageous with larger number of replicas M" (Section 3.3.3) since the pruning method resolves sign conflicts that become more prevalent with more diverse replicas.

Synchronization frequency (H): Figure 13 shows that both DiLoCo and Streaming DiLoCo degrade at very small H (<10) and very large H (>100). At small H, outer gradients have small norm (only a few steps of accumulated updates) and are noisy. At large H, replicas drift too far apart, and the outer gradient may no longer provide a useful signal for global optimization. Streaming DiLoCo is "more robust to low values of H" than DiLoCo (Section 3.3.1, caption of Figure 13)—at H = 5, Streaming DiLoCo shows lower loss than DiLoCo, suggesting the frequent small synchronizations of the streaming pattern help stabilize training when outer gradient signal is weak. The paper's main experiments use H = 30 or H = 100, both in the stable region.

Which parameters to evaluate: Table 3 compares three choices for evaluation: the first replica's parameters (θ₁), the average of all replica parameters, and the outer parameters (globally synchronized parameters where each fragment was synchronized at a different point in time). The outer parameters slightly outperform the other options on both eval loss (2.67 vs. 2.68 and 2.77) and HellaSwag (37.78 vs. 37.72 and 37.77). The difference is small, but the paper uses outer parameters for all reported results, noting that evaluating a single replica could understate the method's performance since different replicas may be at different points in their training trajectories.

Number of replicas with constant local batch size: Table 6 reports results for Streaming DiLoCo with overlapped FP4 at M = {2, 4} and H = {30, 100} across all model scales. M = 4 consistently underperforms M = 2 across all scales and both H settings. For example, at 1B with H=30, M=2 achieves eval loss 2.48 and HellaSwag 46.60%, while M=4 achieves 2.50 and 45.25%. The gap widens slightly with H=100: M=2 achieves 2.50 and 46.00%, while M=4 achieves 2.53 and 44.74%. This is expected: with more replicas, the local batch size is smaller (since the paper keeps global batch size constant in these experiments), and each replica has fewer tokens to learn from per step, potentially reducing inner optimization quality.

Overtraining with M=4 on Dolma (Table 7): This extends the overtraining experiments to M=4 replicas. The terabytes exchanged are half those of M=2 (since total data volume scales inversely with M in the all-reduce), while performance remains competitive: at 250B tokens, M=4 achieves 2.47 eval loss and 52.20% HellaSwag compared to 2.45 and 53.86% for Data-Parallel. The gap is slightly wider than M=2 (which achieved 2.45 and 54.24%), but the difference is modest given the 2× additional reduction in communication.

Critical Assessment

Does Streaming DiLoCo match Data-Parallel training quality?

What the experiments demonstrate. The scaling experiments (Table 5, Figure 5) show that across six model scales from 35M to 4B parameters on C4, Streaming DiLoCo achieves evaluation loss and downstream accuracy within a fraction of a percentage point of Data-Parallel. At the largest trained scale (4B), the loss gap is 0.01 (2.25 vs. 2.26) and the HellaSwag gap is 0.54 percentage points (59.56% vs. 59.02%). The overtraining experiments on Dolma (Table 1) with token budgets up to 250B show even closer alignment—Streaming DiLoCo actually achieves slightly better numbers on several metrics. These are the right experiments to run, and the consistency across scales and datasets is convincing.

What is NOT demonstrated. The experiments stop at 4B parameters. The paper makes strong claims about behavior at 10B, 100B, and even 405B parameters, but these are based entirely on simulation (Figures 4, 16), not actual training runs. The simulation models compute utilization, not learning dynamics—it assumes that if the system can achieve high CU, the learning outcome will match Data-Parallel. This is a reasonable assumption given the 35M–4B empirical results, but it is an extrapolation, not a demonstration. At 100B parameters, a fragment would be 3/108 ≈ 2.8% of the model, compared to 3/24 = 12.5% at 1B—the outer gradient for such a small fragment might behave differently, and the frequent tiny synchronizations might affect learning dynamics in ways the small-scale experiments cannot capture.

The 4B model is also trained at Chinchilla-optimal tokens (83B), whereas the 1B overtraining experiments go to 250B tokens. No experiment tests the combination of larger scale (10B+) AND overtraining simultaneously, which would be needed to fully validate the method for frontier-model training.

Does Streaming DiLoCo really reduce required bandwidth by two orders of magnitude?

What the experiments demonstrate. The compute utilization simulation (Table 4, Figure 4) shows that for a 100B model, Streaming DiLoCo with overlapped FP4 achieves 95% CU at approximately 1.1 Gbit/s, while Data-Parallel requires 390.7 Gbit/s—a factor of 355×, or more than two orders of magnitude. The Dolma experiments (Table 1) show 400× reduction in total terabytes exchanged (1.10 TB vs. 441 TB at 25B tokens). These numbers are internally consistent and follow directly from the design: H=100 reduces frequency by 100×, fragment size 3/24 reduces peak by 8×, and FP4 reduces per-bit cost by 8× (the realized 400× is less than the theoretical 100×8×8=6400× because other components of communication overhead don't benefit equally from all three factors).

What is NOT demonstrated (and what the paper acknowledges). The simulation does not account for intra-worker communication (between devices within a worker), which can be substantial for large models using FSDP internally. In a real deployment, the worker contains multiple accelerators that must communicate among themselves at high bandwidth. The paper models only cross-worker communication. If intra-worker bandwidth is also constrained (e.g., if workers are not co-located even internally), the actual bandwidth requirements could be higher.

The paper also does not measure real-world bandwidth consumption in any actual distributed deployment. All bandwidth numbers come from simulation with simplifying assumptions (idealized DAG, no contention, no protocol overhead, uniform bandwidth). Real-world network behavior—TCP overhead, packet loss, congestion, variable latency—could degrade utilization below the simulated predictions. The paper's remark about Bonini's paradox ("this is still a useful tool to estimate device utilization") is an honest acknowledgement, but it also means the headline "two orders of magnitude" figure should be interpreted as an estimated benefit validated in simulation, not a measured benefit from a real multi-datacenter training run.

Are the three contributions truly independent and necessary?

What the experiments demonstrate. The ablation structure cleanly separates the contributions. Figure 11 isolates quantization from value-dropping methods and shows FP4 is uniquely effective. Figure 6 isolates fragment size. Figure 8 isolates overlap delay. Figure 7 compares sequential vs. strided patterns. The simulation in Figure 4 shows the additive benefit of each contribution: Streaming DiLoCo improves over DiLoCo, overlapping improves over Streaming DiLoCo, and FP4 further shifts the curve. The paper can reasonably claim that all three contributions are necessary to achieve the full two-order-of-magnitude reduction—removing any one would increase the required bandwidth by 4–10× based on the simulation curves.

What is NOT demonstrated. The paper does not test whether the overlapping mechanism (Contribution 2) is necessary when streaming is already present. If streaming reduces communication to small, frequent fragments, one could imagine a simpler approach: simply block for each small fragment's communication since it completes quickly enough that the blocking time is negligible. The paper shows (Figure 4, green vs. red curves) that without overlapping, CU asymptotes below 1.0 even at high bandwidth, suggesting blocking still matters. But in absolute terms, the bandwidth needed to reach 90% CU with Streaming DiLoCo (without overlapping) is approximately 9 Gbit/s for a 100B model (green curve in Figure 4c). If 9 Gbit/s is already achievable in a given deployment, overlapping may be unnecessary. The paper's claim that overlapping is needed depends on how low a bandwidth target one aims for.

The paper also does not ablate whether FP4 quantization could be more aggressive (2-bit, 1-bit) or combined with other compression techniques (e.g., FP4 + low-rank compression). The finding that FP4 works perfectly opens the question of how much further compression is possible—the paper leaves this entirely to future work.

Does the method genuinely enable cross-datacenter distributed training (the "distributed free lunch")?

What the experiments demonstrate. The simulation suggests that a 100B model could be trained across multiple data centers connected by 1–2 Gbit/s links with 95%+ compute utilization—bandwidth that is achievable with commodity internet or standard inter-datacenter connections. The memory offloading analysis (Section 2.5) shows that the 5× theoretical memory overhead can be reduced to approximately 2% additional HBM at any moment through CPU offloading, making the method practical even for memory-bound large models. The heterogeneous worker experiment (Figure 10) shows robustness to different worker speeds, which would be essential in a real cross-datacenter deployment with variable latency.

What is NOT demonstrated. The paper trains only with M=2 replicas in its main experiments, and with M=4 and M=8 in ablations. Real distributed training across data centers might involve M=10 or M=100 replicas, which introduces qualitatively different challenges: the all-reduce communication patterns become more complex, network topology matters, and the probability of individual replica failure increases. The paper's M≥4 results (Table 6) show a small but consistent degradation relative to M=2, and it's unclear whether this trend continues or plateaus at larger M.

More fundamentally, all experiments use carefully controlled, identical training environments—the replicas train on the same model architecture with the same hyperparameters on data drawn from the same distribution. This is a "cross-silo" federated learning setting (Kairouz et al., 2021), not a truly heterogeneous deployment. Real distributed training across data centers might involve different hardware generations, different batch sizes due to variable accelerator counts, different software stacks, and network partitions. The paper's robustness to heterogeneous τ delays (Figure 10) is a first step, but it tests only timing asynchrony, not hardware or data heterogeneity.

The paper also does not address fault tolerance. If one replica fails or loses connectivity in a cross-datacenter deployment, how does the training recover? Data-Parallel training is brittle to replica failure because every step requires all replicas. DiLoCo methods are inherently more robust because workers can train independently for H steps, but the paper doesn't test or discuss failure recovery mechanisms.

Are the learning dynamics fully characterized, or are there hidden failure modes?

What the experiments demonstrate. The scaling experiments (Table 5) show consistent behavior across scales, datasets, token budgets, and synchronization frequencies (H=30 and H=100). The cosine similarity analysis (Figures 17–18) shows replicas' outer gradients are nearly orthogonal throughout training, suggesting the method doesn't collapse to a degenerate regime where replicas learn identical functions. The comparison with FedPart's freezing approach (Section 3.3.1) validates the design choice not to freeze layers.

What is NOT demonstrated. There is no systematic study of how the outer learning rate or outer Nesterov momentum interact with the streaming pattern. The paper fixes the outer learning rate at 0.4 based on small-scale tuning and never revisits it. At larger model scales or with different H values, the optimal outer learning rate might differ. If the outer learning rate needs to be retuned per scale, the claim of "drop-in replacement" is weaker.

The paper also doesn't explore what happens with very large H (beyond 500). DiLoCo's original paper (Douillard et al., 2024a) used H=500. Streaming DiLoCo with H=500 and 36 fragments at 100B would mean each fragment synchronizes every 500 steps, but with a fragment synchronizing roughly every 500/36 ≈ 14 steps. This is still relatively frequent for a single fragment, but the outer gradient for a 3-layer fragment after 500 steps of training might become very large in magnitude compared to the inner updates happening during the intervening 14 steps—potentially causing instability when merged via the mixing equation. The paper doesn't test this regime.

Missing experiments that would strengthen the paper

Several experiments would provide stronger evidence for the central claims:

A 10B+ parameter training run with real measured bandwidth utilization and training time. This would validate the simulation predictions against empirical measurement and confirm that the learning equivalence holds at larger scales.
Training with M=8 or M=16 replicas on a realistic distributed setup (e.g., replicas in different cloud regions with measured, variable latency). This would test whether the method works under the conditions it's designed for.
A direct comparison between Streaming DiLoCo and other partial-communication methods beyond FedPart (e.g., WASH, Sparta) at comparable communication budgets. The paper compares only against FedPart (which it convincingly outperforms) but doesn't benchmark against other approaches that communicate subsets of neurons rather than subsets of layers.
An ablation of the mixing factor α at larger scales and larger τ. The paper tests α = 0 and α = 0.5 at τ up to 20, but only at 500M parameters. At 100B with 36 fragments and τ = 5, the optimal α might be different, and the robustness observed at small scale might not transfer.
A fault-tolerance experiment. Deliberately introducing replica failures or network partitions and measuring recovery behavior would address a practical concern the paper motivates (the "failure amplification" problem in data-parallel training) but never demonstrates it solves.

Summary of evidence strength

The paper's strongest contribution—that fragmenting DiLoCo's all-reduce into streaming partial updates, combined with overlapping and quantization, reduces simulated bandwidth requirements by two orders of magnitude without degrading learning at small-to-medium scales—is well-supported by the experiments that ARE run. The consistency across six model scales (35M–4B), two datasets (C4, Dolma), three token budgets (up to 250B), and various ablation configurations makes the core claim credible.

The weakest part of the evidence chain is the leap from 4B empirical results + simulation to claims about 100B+ frontier-model training. The simulation is well-constructed and the square-cube law argument is theoretically sound, but the paper would be stronger if it acknowledged more explicitly that the "two orders of magnitude" figure is a simulation prediction awaiting empirical validation at scale, rather than a demonstrated result. The overtrained 1B experiments on Dolma go some way toward bridging this gap by showing the method works in more realistic (overtrained) conditions, but the scale gap remains substantial.

6. Limitations and Trade-offs

The Scale Gap: Empirical Results Stop at 4B Parameters While Core Claims Extend to 100B+

The assumption or constraint. All actual training experiments—the scaling runs on C4, the overtraining runs on Dolma, the ablations—use models from 35M to 4B parameters. The paper's headline claims about behavior at 10B, 100B, and even 405B parameters rest entirely on simulation (the compute utilization DAG model in Section 3.1), not on measured training outcomes. The paper is transparent about this boundary: the simulation section is explicitly labeled as estimating "what we expect to happen in practice" (Section 3.1, Remark), and the 4B training runs are the largest empirical data points.

The consequence. The simulation models compute utilization—the fraction of wall-clock time spent computing versus waiting for communication—but does not model learning dynamics. It assumes that if Streaming DiLoCo can achieve high CU at a given bandwidth, the resulting model quality will match Data-Parallel. While the 35M–4B empirical results support this assumption, two factors could break it at larger scales:

First, the fragment-to-model ratio shrinks as models grow. At 1B parameters with 24 layers, a 3-layer fragment represents 12.5% of the model. At 100B with 108 layers, a 3-layer fragment represents only 2.8%. The outer gradient for such a small fraction of the model—accumulated over H = 100 steps—may have very different statistical properties than a 12.5% fragment. In the extreme, synchronizing a tiny sliver of parameters while the rest of the model drifts independently could create internal representational inconsistencies between recently-synchronized and stale layers, a failure mode that the small-scale experiments cannot reveal because the fragment-to-model ratio remained large enough.

Second, the synchronization frequency per fragment gets compressed as fragment count grows. At 100B with 36 fragments and H = 100, some fragment synchronizes roughly every 2.8 steps. This near-continuous communication deviates substantially from the DiLoCo conceptual model of "train independently for many steps, then synchronize." The outer optimizer's Nesterov momentum state for each fragment would be updated based on outer gradients that are only H = 100 steps apart, but the fragments themselves are being merged back into an actively-training model every few steps. Whether this tight coupling changes the effective learning dynamics—perhaps making the method behave more like data-parallel with delayed gradients than like DiLoCo—is unknown.

What evidence exists in the paper. The simulation (Table 4, Figure 4) shows that CU improves with model scale due to the square-cube law, which is a positive sign. But the cosine similarity analysis (Figures 17–18) reveals that replicas' outer gradients become less correlated at larger scales (lower cosine similarity), and the paper does not analyze whether the streaming pattern interacts with this trend. The Llama 405B and DeepSeek-V3 simulation results (Figure 16) extend the CU prediction to real architectures but remain simulation-only. No 10B+ training run validates the CU-to-learning-quality link at the scales where the two-order-of-magnitude bandwidth reduction is most impactful.

Mitigation status. The paper partially addresses this through the overtraining experiments on Dolma (Table 1), which demonstrate that the method's equivalence to Data-Parallel holds at 1B parameters even when trained far beyond Chinchilla-optimal tokens, suggesting robustness to training duration. However, this tests overtraining in the token dimension, not scaling in the parameter dimension. The paper acknowledges the gap implicitly by framing itself as "a first step towards what we call a distributed free lunch" (Section 5), suggesting larger-scale validation as future work. No concrete plans or resource estimates for such validation are provided.

Difficulty Estimation Cost Is Not Accounted for in Headline Efficiency Numbers

The assumption or constraint. The compute-optimal framework in this paper has an analog: the compute utilization simulation assumes the communication schedule and fragment pattern are known and fixed before training. But in a real deployment, several practical costs are not accounted for in the headline bandwidth reduction figures:

The outer optimizer state offloading mechanism (Section 2.5) requires transferring fragment-sized chunks of outer parameters and Nesterov momentum buffers between CPU RAM and HBM on a per-synchronization schedule. The paper estimates this at "less than 10 milliseconds" for a 100B model using PCIe at 2 TB/s, which is indeed small relative to a 4.9-second step time. However, this transfer consumes PCIe bandwidth that may be shared with data loading, checkpointing, or other I/O. The paper does not model contention for this bus.
The all-reduce operation itself incurs latency that the simulation abstracts as a function of data volume and link bandwidth, but real all-reduce implementations have startup overheads, protocol inefficiencies, and sensitivity to the number of participating replicas. At M = 4 or M = 8 replicas (tested in Tables 6–7), the all-reduce topology matters. The paper's simple bandwidth-volume model may underestimate communication time for small fragment sizes where latency overhead dominates throughput.
The difficulty of tuning the outer learning rate and other DiLoCo hyperparameters. The paper uses a fixed outer learning rate of 0.4 across all scales "for simplicity, and to show that Streaming DiLoCo is a drop-in replacement of DiLoCo" (Section 3.2). But DiLoCo itself required scale-specific tuning in the original paper (Douillard et al., 2024a). The fact that a single outer learning rate works across 35M to 4B is a positive result, but it doesn't guarantee that this rate remains optimal at 100B, or that the optimal rate doesn't shift when the fragment-to-model ratio changes.

The consequence. The paper's headline metric—"400× fewer bits exchanged" and "two orders of magnitude less bandwidth"—represents the idealized communication reduction in a system where all supporting infrastructure (CPU offloading, all-reduce, hyperparameter selection) operates cost-free. In a real deployment, the effective bandwidth reduction could be lower due to:

PCIe contention between offloading and other I/O, increasing per-step latency.
All-reduce overhead for frequent small-fragment synchronizations, which may have poor throughput efficiency compared to infrequent full-model reductions.
Suboptimal hyperparameters at larger scales requiring additional tuning runs, consuming compute that should be counted against the method's efficiency.

None of these costs appear in the bandwidth reduction calculation, making the "two orders of magnitude" figure an upper bound under idealized conditions.

What evidence exists in the paper. The paper explicitly acknowledges that the simulation is simplified: "Such simulation is not perfect because for instance we consider only the bandwidth between datacenters and not the local bandwidth between devices" (Section 3.1, Remark). The memory offloading analysis (Section 2.5) provides a single-point estimate of transfer time under ideal conditions (10 ms on 2 TB/s PCIe) but does not model contention. The paper never discusses all-reduce topology or per-message overhead.

Mitigation status. The paper partially addresses offloading by arguing that the deterministic schedule allows pre-fetching: "we can start the transfer from RAM to HBM of a fragment (and its associated outer optimizer state) while finishing the previous (inner) gradients passes" (Section 2.5). This hides latency for the offloading but does not eliminate bandwidth contention. The paper does not address all-reduce efficiency or hyperparameter tuning cost at scale. These are left as implicit future engineering work.

Single Hardware and Training Configuration Limits Generalizability

The assumption or constraint. All experiments use a Chinchilla-architecture decoder-only transformer, trained from scratch using the NanoDO codebase (Liu et al., 2024b) with DrJax (Rush et al., 2024) for parallelism, on the C4 and Dolma datasets. The compute utilization simulation uses hardware assumptions based on an H100 GPU with PCIe and 60% MFU (Table 4 caption). The outer optimizer is fixed as SGD with Nesterov momentum, and the inner optimizer is fixed as AdamW. The synchronization frequency H is tested at only two values in the main experiments (30 and 100), with the H=100 setting used for the Dolma overtraining runs.

The consequence. Several aspects of the findings could be specific to this configuration, and the paper provides limited evidence to bound the generalization:

Architecture sensitivity. The Chinchilla architecture is a standard dense transformer. Different architectures—particularly mixture-of-experts (MoE) models like DeepSeek-V3, which the paper simulates in Figure 16b—have different communication-to-computation ratios. MoE models synchronize the full parameter set (all experts) but only activate a subset per token, making the square-cube law work against them. The DeepSeek-V3 simulation shows Streaming DiLoCo still works, but this is simulation-only. Other architectural features not tested include: encoder-decoder architectures, models with shared embedding/head weights, models with different normalization schemes, or models with non-transformer components (retrieval, structured state space layers).
Hardware assumptions. The 60% MFU assumption used to estimate step times (Table 4) may not hold for all hardware configurations. Lower MFU (common at very large scales due to communication overheads within a worker, or on less-optimized hardware) would reduce the step time, which in turn reduces the overlapping window and increases the required bandwidth to hit a target CU. The paper's bandwidth requirements scale inversely with step time: if real MFU is 40% instead of 60%, step times would be ~1.5× longer, which would actually help the method. But if MFU is higher (80% on very optimized hardware), step times shrink and bandwidth requirements increase.
Outer optimizer choice. The paper inherits Nesterov SGD as the outer optimizer from DiLoCo and never ablates it. Other outer optimizers (e.g., Adam as used in FedOpt; Reddi et al., 2021) might interact differently with the streaming pattern. Nesterov momentum maintains a velocity buffer that smooths updates across synchronizations; an adaptive optimizer like Adam might amplify noise from small outer gradients on tiny fragments, potentially requiring retuning or degrading performance.
Dataset dependence. C4 and Dolma are both web-scale text corpora. The paper doesn't test on code, math, multilingual, or multi-modal data. The outer gradient statistics—and thus the effectiveness of FP4 quantization and the optimal H—likely depend on data distribution. More diverse or specialized data might produce outer gradients with different sparsity or magnitude distributions, potentially breaking the FP4 quantization or requiring different fragment sizes.

What evidence exists in the paper. The DeepSeek-V3 simulation (Figure 16b) partially addresses the architecture concern for MoE models, but again, it's simulation-only. The paper tests a range of model scales (35M to 4B), which provides some evidence of robustness across model sizes within the same architecture family, but doesn't test across architecture families. The fragment size ablation (Figure 6) and the strided vs. sequential comparison (Figure 7) show some sensitivity to fragmentation configuration, but within a narrow design space. No experiments vary the outer optimizer type, the inner optimizer, or the training data distribution.

Mitigation status. The paper does not claim generality across architectures, hardware, or outer optimizers. The stated goal is to improve DiLoCo specifically, and DiLoCo was developed for decoder-only transformer LLMs. The paper's positioning as a "first step" implicitly acknowledges that broader validation is future work. The DeepSeek-V3 simulation is a gesture toward generalization but doesn't substitute for empirical validation on non-dense architectures.

The Outer Optimizer's Learning Rate and Hyperparameter Landscape Are Unexplored

The assumption or constraint. The paper fixes the outer learning rate at 0.4, tuned at small scale, and keeps it constant across all model sizes, datasets, synchronization frequencies (H=30 and H=100), and numbers of replicas (M=2 and M=4). The paper presents this as a feature: "for the simplicity, and to show that Streaming DiLoCo is a drop-in replacement of DiLoCo, we used the same outer learning rate, without further hyperparameters tuning" (Section 3.2). However, DiLoCo's original paper (Douillard et al., 2024a) showed that the outer learning rate is a critical hyperparameter that interacts with H, model size, and data distribution.

The consequence. There are several reasons to suspect that the optimal outer learning rate might change under the streaming pattern:

Fragment size creates an effective learning rate scaling. In DiLoCo, the outer optimizer sees the full outer gradient (all parameters) and applies a global learning rate. In Streaming DiLoCo, the outer optimizer sees only a fragment's outer gradient. If the outer gradient for a 3-layer fragment has different typical magnitude than the full-model outer gradient, the effective step size changes. The paper never measures outer gradient magnitudes per fragment, so it's unknown whether this matters.
The mixing factor α introduces a second learning-rate-like parameter. The merge equation (line 12 of Algorithm 2) blends locally-trained and globally-synchronized parameters. The paper tests α = 0 and α = 0.5 (Figure 8) but doesn't explore whether the optimal α interacts with the outer learning rate. At α = 0, the outer optimizer's update completely overwrites the local training during the τ overlap steps, which is equivalent to trusting the global consensus direction fully. At α = 0.5, the blend halves the effective outer learning rate (since half the update comes from outer optimization, half from local training). The paper never comments on this interaction.
The streaming schedule means different fragments are at different points in their outer optimizer trajectories. The outer Nesterov momentum state for fragment 1 (which last synchronized 100 steps ago) may differ substantially from fragment 36 (which last synchronized 2 steps ago). A single global outer learning rate may not be appropriate for all fragments at all times. The paper shows that evaluating with the outer parameters (which are a patchwork of fragments synchronized at different times) works slightly better than evaluating a single replica (Table 3), suggesting this asynchronicity is benign, but doesn't explore whether fragment-specific outer learning rates could improve performance.

What evidence exists in the paper. The learning rate robustness is demonstrated indirectly: across 35M–4B parameters on C4 (Table 5) and up to 250B tokens on Dolma (Table 1), the fixed outer learning rate of 0.4 produces results matching Data-Parallel. This is genuine evidence that the outer learning rate is not catastrophically sensitive. However, it doesn't rule out that re-tuning would recover some of the small gap to Data-Parallel observed at 4B (0.01 loss, 0.54 HellaSwag percentage points) or that the optimal rate would shift at 10B+ parameters.

Mitigation status. The paper does not address this as a limitation. The fixed outer learning rate is presented as a strength (simplicity, drop-in replacement). No learning rate sweep is reported at any scale, even in the appendix. No experiments test whether a scale-dependent or fragment-dependent outer learning rate schedule would improve results. The paper explicitly leaves "how to scale efficiently the number of DiLoCo replicas given an equivalent token budget" as future work (Section 5), but doesn't mention outer hyperparameter tuning as part of that scaling question.

The Practical Deployment Gap: No Fault Tolerance, No Real Network Measurements, No System Integration

The assumption or constraint. The paper motivates Streaming DiLoCo partly through the failure modes of co-located training: "the more devices that are used for each synchronous training step, the more chances there are that one of them fails, risking halting training, or introducing subtle numerical issues" (Section 1). However, the paper neither demonstrates that Streaming DiLoCo is fault-tolerant nor measures its behavior under realistic distributed conditions. All experiments are conducted in controlled environments with identical replicas, stable connectivity, and no failures. The compute utilization simulation (Section 3.1) models idealized, contention-free communication.

The consequence. Several practical deployment questions are left completely unaddressed:

Fault tolerance. If one replica fails or becomes unreachable during training, what happens? In Data-Parallel, a single failure typically halts the entire run (unless checkpointing and restart mechanisms are in place). DiLoCo methods are conceptually more robust because workers can train independently for H steps without communication, but the paper provides no mechanism for handling a permanently failed replica. Would the remaining M−1 replicas continue training? Would the outer optimizer need to adjust for the missing outer gradient? What happens to the failed replica's portion of the data? None of these questions are discussed.
Real network conditions. The simulation assumes fixed, uniform bandwidth with no contention, packet loss, latency jitter, or congestion. Real inter-datacenter links experience all of these. Streaming DiLoCo's overlapping mechanism tolerates communication latency up to the step time (τ ≤ 5 steps is recommended), but this assumes latency is predictable. If latency spikes unpredictably (e.g., due to network congestion), a block-receive that expects data within τ steps might stall indefinitely, blocking not just the fragment being synchronized but all subsequent fragments whose schedules depend on the DAG not deadlocking. The paper does not discuss timeout mechanisms, degraded-mode operation, or how the schedule would recover from a missed synchronization window.
System integration complexity. The paper's experiments use a modified NanoDO codebase with custom DrJax parallelism and jax.vmap annotations for replica parallelism (Section 3.2). This is a research codebase. Integrating Streaming DiLoCo into a production training framework (e.g., Megatron-LM, PyTorch FSDP, JAX with Pathways) would require significant engineering: implementing the deterministic fragment schedule, managing the CPU-to-HBM offloading pipeline with pre-fetching, integrating FP4 quantization into the all-reduce communication path, and ensuring compatibility with other parallelism axes (tensor parallelism, pipeline parallelism, sequence parallelism). The paper demonstrates compatibility with FSDP internally within each replica (Section 3.2) but doesn't address how Streaming DiLoCo's outer loop interacts with other parallelism strategies.
Elasticity. Modern distributed training systems increasingly support elastic scaling—adding or removing workers mid-training to respond to preemption or resource availability changes. Streaming DiLoCo's deterministic schedule assumes a fixed number of replicas M and fragments P known at training start. Changing M mid-training would require recomputing fragment offsets, repartitioning outer optimizer state, and potentially rebalancing data shards. The paper's heterogeneous τ experiment (Figure 10) shows some robustness to timing variation but doesn't address structural changes to the replica set.

What evidence exists in the paper. The heterogeneous τ delay experiment (Figure 10) is the closest the paper comes to testing robustness. It shows that workers can tolerate different overlap delays (τ₁ = 1, τ₂ up to 5) without significant loss degradation, demonstrating some slack in the timing assumptions. The paper also tests varying numbers of replicas (M = 2, 4, 8 in Figure 12 and Tables 6–7), showing that performance degrades modestly with more replicas but remains functional. These are positive signals for robustness but fall far short of a fault-tolerance or real-network evaluation.

The paper explicitly acknowledges its scope: "This is only a simulation of what we expect to happen in practice" (Section 3.1, Remark). It does not overclaim the deployment readiness.

Mitigation status. The paper does not attempt to mitigate these gaps. No fault-tolerance mechanisms, timeout handlers, elastic scaling protocols, or production system integration plans are described. The paper presents itself as a research contribution establishing feasibility, with deployment engineering left to future work. The remark about the "hardware lottery" (Section 5) implicitly acknowledges that bridging from research demonstration to production deployment is a substantial undertaking requiring extensive engineering investment. The paper's final vision—"we hope to see the training of modular constellations of small models loosely connected across heterogeneous devices, using compute arbitrage spread world-wide" (Section 5)—signals that the deployment gap is recognized but not addressed in the current work.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper reframes distributed training from a peak-bandwidth provisioning problem into a latency-tolerance scheduling problem. Before Streaming DiLoCo, the dominant mindset for reducing communication in distributed training was to reduce communication frequency—make H as large as possible, synchronize as rarely as possible. This treats bandwidth as a fundamentally scarce resource that must be conserved. Streaming DiLoCo demonstrates that peak bandwidth and communication frequency are independent axes: you can communicate more frequently (streaming a fragment every few steps) while demanding far less peak bandwidth (only one fragment at a time). This is a genuinely counter-intuitive reframing—most practitioners would assume that communicating more often requires more bandwidth, not less. The paper shows this assumption is wrong when you fragment the model, and the result is a communication pattern that behaves like a continuous low-bitrate stream rather than sporadic high-bitrate bursts.

The magnitude of this shift is substantial but bounded. This is not a paradigm shift in the sense of introducing a new learning algorithm—Streaming DiLoCo preserves the DiLoCo/FedOpt inner-outer optimization structure exactly. It is a systems-level reframing that transforms the engineering constraints around where and how distributed training can be deployed. The paper makes credible the idea that 100B-parameter models could be trained across multiple data centers connected by commodity networking (1–5 Gbit/s) without sacrificing model quality—a prospect that would be dismissed as absurd under the co-location paradigm. This opens a design space that was previously considered closed: training across geographic regions, using spot/preemptible compute from multiple cloud providers, or building decentralized training collectives without requiring specialized interconnect fabrics.

The paper also resolves an implicit tension in the distributed training literature between two strategies for bandwidth reduction: communicate less often (Local SGD, FedAvg, DiLoCo) versus compress more aggressively (gradient sparsification, low-precision quantization). These were often seen as alternatives—you either reduce frequency or you reduce per-message size. Streaming DiLoCo shows they are synergistic: streaming fragments reduces peak bandwidth, which makes overlapping possible, which makes the lower per-message size of quantization more impactful because the overlapping window can be smaller. The combination achieves orders-of-magnitude more than either strategy alone. This suggests that future work on communication-efficient training should think in terms of interacting mechanisms (fragmentation × scheduling × compression) rather than isolated improvements to individual components.

One line of research this paper makes less attractive is the development of ever-more-aggressive value-dropping compression methods (random sparsification, Top-K, DARE) for the outer gradient setting. Figure 11 demonstrates conclusively that these methods fail where simple FP4 quantization succeeds, at least in the DiLoCo outer gradient context. The diagnostic finding—that outer gradients are dense, structured displacement vectors, not sparse instantaneous gradient signals—suggests that the extensive literature on gradient sparsification may not transfer to the federated optimization setting in the ways researchers might assume. Future compression work for DiLoCo-style methods should focus on precision reduction (going below 4 bits, exploring non-uniform quantization, or applying block-wise scaling) rather than value-dropping sparsification.

Follow-Up Research This Work Enables

Training a 10B+ model across two physical data centers with measured end-to-end bandwidth, latency, and model quality. The single largest gap in this paper's evidence chain is the absence of empirical validation at the scales where the bandwidth reduction is most impactful (10B–100B+). A direct follow-up would train a 10B-parameter model using Streaming DiLoCo with M=2 replicas placed in two different cloud regions (or two separate on-premise clusters) connected by a measured, realistic inter-datacenter link (e.g., 5–10 Gbit/s with variable latency). The experiment would report: (a) wall-clock training time compared to a co-located Data-Parallel baseline at the same scale, (b) actual measured compute utilization (not simulated), (c) evaluation loss and downstream accuracy curves, (d) bandwidth utilization traces showing the streaming communication pattern in practice, and (e) sensitivity to real-world latency jitter and packet loss. A negative result—if the model fails to converge or requires significantly more bandwidth than the simulation predicts—would identify where the simulation's abstractions break down and would be as informative as a positive result.

Determining the minimum fragment size before learning degrades, as a function of model depth and width. The paper's fragment size ablation (Figure 6) tests 1, 3, and 6 layers at 500M parameters (18 layers) and selects 3 layers as the sweet spot, but the relationship between fragment size, model depth, and learning stability is unexplored. A systematic study would fix the model architecture and vary the fragment size from 1 layer to L/2 layers, measuring evaluation loss and tracking per-layer gradient statistics (norm, cosine similarity between replicas) to identify the mechanism that causes degradation at very small fragments. The hypothesis to test: small fragments produce outer gradients with low signal-to-noise ratio because the displacement of a single layer over H steps may be dominated by noise rather than genuine learning progress. If confirmed, this would imply a minimum fragment size that scales with model depth and H, providing a principled rather than empirical basis for fragment selection. The paper's observation that the first transformer layer has much higher outer gradient cosine similarity than deeper layers (Figure 18) suggests the optimal fragment size may not be uniform across depth—a mixed-size fragmentation scheme (larger fragments for early layers, smaller for middle layers) could potentially improve the peak-bandwidth-vs-quality tradeoff.

Exploring whether outer gradient quantization can be pushed below 4 bits (to 2 bits or 1 bit) with block-wise scaling or non-uniform quantization. The paper's FP4 finding—that E3M0 quantization works perfectly—raises an obvious follow-up question: how much further can we go? The E3M0 format preserves sign and exponent but discards all mantissa precision. If 4 bits works losslessly, do 2 bits (E2M0: 1 sign, 2 exponent, no mantissa) or even 1 bit (sign only) remain viable? The key challenge is that outer gradients likely have wide dynamic range within a single fragment (embedding layer gradients may differ in magnitude from middle-transformer-layer gradients by orders of magnitude), so a naive 2-bit format that can't represent the full dynamic range would clip or saturate. Block-wise quantization—where each fragment or even each layer within a fragment uses its own scaling factor—could preserve dynamic range while reducing per-element bitwidth. A strong follow-up would measure the per-layer outer gradient magnitude distribution across training, design a block-wise quantization scheme informed by those statistics, and report the minimum bitwidth that matches FP32 performance. If 1–2 bit quantization proves viable, combined with 36-fragment streaming and H=100, the total bandwidth reduction over Data-Parallel could approach 10,000×—making inter-datacenter training practical for models at the frontier scale (1T+ parameters).

Characterizing whether the streaming fragmentation schedule can be made fully dynamic and adaptive, rather than static and predetermined. The paper's schedule is fully deterministic: fragment offsets are fixed before training begins. This works well for homogeneous workers with stable connectivity, but real distributed deployments involve variable worker speeds (heterogeneous hardware, resource contention) and time-varying network conditions (congestion, link failures). A dynamic scheduler would monitor per-worker training speed and per-link latency in real-time, and adaptively decide which fragment to synchronize next—prioritizing fragments whose outer gradients are most stale, or deferring synchronization when a worker falls behind to avoid blocking faster peers. The paper's heterogeneous τ experiment (Figure 10) shows the system can tolerate timing variance up to ~5 steps, which provides a buffer for dynamic scheduling to operate within. A concrete experiment: train with M=4 replicas where one replica runs on 50% slower hardware, and compare a static schedule (which forces the slower replica to miss synchronization windows) against an adaptive schedule that dynamically adjusts fragment offsets and τ values per replica. The metric would be final model quality versus total wall-clock time.

Investigating whether the mixing factor α (line 12 of Algorithm 2) has a principled optimal value that depends on τ, H, and the fragment's position in the network, rather than being a fixed hyperparameter. The paper tests α=0 and α=0.5 at 500M parameters (Figure 8) and finds both work well for τ ≤ 10, but provides no analysis of why averaging two parameter estimates from different points in optimization works so robustly. This is a learning-theoretic question disguised as a hyperparameter: the merge equation blends a locally-trained parameter vector (which incorporates τ steps of recent gradient information but is diverged from the global consensus) with a globally-synchronized vector (which incorporates cross-worker information but is τ steps stale). Under what conditions is the uniform average (α=0.5) optimal, versus trusting the global consensus (α=0) or trusting the local exploration (α=1)? One hypothesis: early in training, when outer gradients have high variance and replicas explore different basins, α should be small (trust the consensus) to prevent divergence. Late in training, when the loss landscape flattens and replicas converge, α can be larger (trust local exploration) to enable fine-tuning. A follow-up experiment would track the ideal α over the course of a full training run by running a sweep at multiple checkpoints and measuring whether the optimal α shifts systematically with training progress. If confirmed, a simple annealing schedule for α (from 0 toward 0.5 over training) could recover slight performance improvements with no additional communication cost.

Stress-testing Streaming DiLoCo under deliberate replica failure, network partitions, and straggler conditions. The paper motivates its work partly through the fragility of co-located training ("the more devices that are used for each synchronous training step, the more chances there are that one of them fails"), but never demonstrates that Streaming DiLoCo is actually robust to failures. A stress-test experiment would introduce controlled failures during training: (a) kill one replica entirely and measure whether the remaining replicas can continue training and recover model quality (potentially by redistributing the failed replica's data shard or simply proceeding with M−1 replicas), (b) introduce a network partition that prevents communication for 2H steps (long enough that the outer gradients become severely stale), then restore connectivity and measure whether the merge mechanism recovers, (c) introduce a straggler replica that runs at 10× slower speed and measure whether the heterogeneous τ mechanism (Figure 10) prevents it from blocking the faster replicas. A negative result—if Streaming DiLoCo fails catastrophically under partition—would reveal that the overlapping mechanism assumes reliable communication within τ steps and has no fallback for missed synchronizations. This would motivate adding timeout-and-skip logic to the algorithm: if a fragment's outer gradient doesn't arrive within τ steps, proceed with α=1 (purely local parameters) for that fragment and flag it for urgent synchronization in the next window.

Practical Applications and Downstream Use Cases

Cross-region training for organizations with compute resources in multiple geographic locations. A company with GPU clusters in two cloud regions (e.g., us-east and eu-west) could pool their compute to train a single model rather than training separate models per region. With Streaming DiLoCo, the inter-region link need only support ~2–5 Gbit/s to maintain 95%+ compute utilization for a 100B-parameter model (Table 4, purple curve), which is achievable with standard cloud inter-region bandwidth (typically 5–25 Gbit/s for dedicated interconnects, and the paper's Figure 4c shows that even 1–2 Gbit/s reaches 95% CU for the 100B model with FP4). The practical benefit is utilizing otherwise-idle compute capacity across regions—GPUs in us-east during off-peak hours could contribute to training alongside GPUs in eu-west during their peak hours—without the prohibitive cost and latency of real-time gradient synchronization. This is the "compute arbitrage spread world-wide" the paper envisions (Section 5), and Streaming DiLoCo provides the first algorithm that makes it bandwidth-feasible for LLM-scale models.

Training on decentralized volunteer computing grids (e.g., a distributed equivalent of Folding@home for LLMs). Volunteer computing projects like Folding@home or SETI@home have demonstrated that millions of consumer devices can contribute to large-scale scientific computation when the workload is embarrassingly parallel and communication-minimal. LLM training has been excluded from this paradigm because Data-Parallel requires synchronous all-reduce at every step, which is impossible over consumer internet connections with asymmetric bandwidth (high download, low upload) and variable latency. Streaming DiLoCo changes this calculus: each volunteer's device acts as a DiLoCo replica, training independently for H=100 steps (requiring zero communication during that time), then asynchronously uploading a 4-bit-quantized fragment-sized outer gradient (e.g., for a 1B model with 3-layer fragments and FP4, approximately 1.5 MB per synchronization event). The per-fragment communication is small enough to complete over consumer upload speeds (10–50 Mbit/s) within a few seconds. The download side (receiving the averaged outer gradient) requires similarly low bandwidth. While the paper doesn't test this setting explicitly (all experiments use M≤8 replicas with controlled bandwidth), the compute utilization simulation (Figure 4) suggests that even at very low bandwidths (sub-1 Gbit/s), the method maintains high CU for larger models. The practical barrier shifts from bandwidth to replica management (handling dropouts, variable availability, data distribution), but the core communication constraint that previously made volunteer LLM training impossible is substantially relaxed by the paper's combined 400× data volume reduction.

Efficient multi-cloud training for organizations that want to avoid vendor lock-in or exploit spot/preemptible pricing. Many organizations are reluctant to commit to a single cloud provider for large training runs due to cost, capacity availability, or strategic reasons. Data-Parallel training effectively forces single-provider commitment because the all-reduce requires all accelerators to be on the same low-latency fabric. Streaming DiLoCo enables a multi-cloud training strategy: place half the replicas on one provider's spot instances (which are cheap but can be preempted with short notice) and the other half on another provider's on-demand instances (which are reliable but expensive). The outer synchronization every H=100 steps means that a spot-instance preemption loses at most H steps of work from one replica, which could be recovered by restarting that replica from the last synchronized outer parameters and replaying its data shard. The paper doesn't implement or test this preemption-recovery mechanism, but the algorithmic structure (independent inner training between synchronizations) naturally supports it in a way that Data-Parallel (where every step requires all workers) does not. The concrete benefit is cost reduction: spot instances are typically 60–90% cheaper than on-demand, and Streaming DiLoCo's tolerance of heterogeneous worker speeds (Figure 10) means the spot replicas don't force the on-demand replicas to wait if they are occasionally slower or temporarily unavailable.

When to Prefer This Method

Prefer Streaming DiLoCo with overlapped FP4 communication when:

Training models at scales where the square-cube law provides sufficient per-step computation time to hide communication latency—this threshold depends on bandwidth, but the simulation (Figure 4) suggests that for models above ~1B parameters with inter-worker bandwidth of 1–5 Gbit/s, overlapping can achieve near-perfect compute utilization with fragment sizes of 3 layers and H=100.
Deploying across multiple data centers, cloud regions, or heterogeneous compute pools where installing specialized high-bandwidth interconnect (InfiniBand, NVLink across racks) is physically impossible or economically prohibitive.
The training run would benefit from elasticity or fault-tolerance—Streaming DiLoCo's independent inner training between synchronizations provides natural resilience to replica slowdowns (Figure 10) and a structural basis for preemption recovery that Data-Parallel lacks.
Overtraining on large token budgets (Table 1), since the method's equivalence to Data-Parallel holds across token budgets up to 250B tokens, and the total communication savings accumulate linearly with training duration (400× fewer bits exchanged across the full run).

Prefer Data-Parallel or vanilla DiLoCo when:

All accelerators are already co-located with high-bandwidth interconnect (e.g., within a single TPU pod or tightly-coupled GPU cluster with NVSwitch). In this setting, the engineering complexity of Streaming DiLoCo's fragment scheduling and CPU offloading may not be justified by bandwidth savings that are already abundant.
Training models smaller than ~100M parameters, where the per-step computation time is so short (<10 ms) that even the reduced per-fragment communication cannot be fully overlapped, and the outer gradient signal from tiny fragments (1–2 layers) may be too noisy for stable outer optimization (the fragment size ablation in Figure 6 shows degradation at 1 layer).
The number of replicas M is very large (>100), approaching a cross-device federated learning setting where the all-reduce itself becomes the bottleneck. Streaming DiLoCo reduces per-message size but not the all-reduce fan-in/fan-out complexity. In this regime, hierarchical aggregation or gossip-based protocols may be necessary complements that the paper does not address.
The training infrastructure lacks CPU memory bandwidth for the offloading scheme (Section 2.5). If the PCIe or equivalent link between CPU RAM and accelerator HBM is saturated by data loading, checkpointing, or other I/O, the fragment offloading overhead may not be absorbable, potentially reducing the effective compute utilization below the simulated predictions.