DiLoCo: Distributed Low-Communication Training of Language Models

🎯 Pitch

Training language models no longer requires co-locating thousands of GPUs: DiLoCo matches or beats fully synchronous training while communicating 500× less, enabling fast training across poorly-connected, geographically distributed islands of devices.

1. Executive Summary

This paper proposes DiLoCo (Distributed Low-Communication), a distributed training algorithm for transformer language models that enables training across poorly-connected islands of devices by communicating only once every few hundred inner optimization steps rather than every step. The method is a variant of Federated Averaging using AdamW as the inner optimizer, Nesterov momentum as the outer optimizer, and a large number of inner steps (default 500) between synchronizations. Evaluated on the C4 dataset with decoder-only transformers up to 400M parameters, DiLoCo with 8 workers matches or exceeds fully synchronous training performance while communicating 500× less, achieving 15.02 perplexity versus the baseline's 16.23 — and even outperforms the baseline using an 8× larger batch size — while training 8× faster in wall-clock time. The method is robust to non-i.i.d. data distributions, varying communication frequencies up to 2000 steps, worker outages with up to 50% communication drops, and dynamically changing numbers of workers, establishing that distributed language model training requires neither co-located devices nor frequent communication only when the total compute budget held across workers is preserved.

2. Context and Motivation

The Core Problem: Modern LLM Training Requires Impractical Hardware Cohabitation

At the time of writing, the dominant paradigm for training large language models is standard mini-batch back-propagation distributed across thousands of tightly coupled accelerators. These devices must be co-located in the same physical facility, interconnected with high-bandwidth, low-latency networking (InfiniBand, NVLink), and carefully orchestrated to exchange gradients, parameters, and intermediate activations at every single optimization step. The paper articulates this difficulty directly in Section 1:

"To start, several thousands of devices need to be powered and be placed at the same physical location; and interconnected with high-bandwidth cables to minimize latency. Careful software engineering is required to orchestrate the passage of gradients, parameters and intermediate states between these devices at each optimization step, keeping all devices fully utilized."

This is an infrastructure constraint masquerading as a technical requirement. The implication is that organizations wishing to train frontier models must either build or rent a single enormous compute cluster — a resource only available to a handful of well-capitalized actors — even if the total global demand for LLM training spans far more compute than any single facility holds.

The problem has several concrete dimensions:

1. Physical co-location is the primary bottleneck. The paper draws a contrast between what is technically possible and what is practically feasible: while it might be difficult to assemble 1,000 GPUs in one building, it might be straightforward to find 10 separate clusters of 100 GPUs each, possibly distributed across different geographic regions, institutions, or cloud providers. The current training paradigm cannot exploit such distributed compute because it demands near-instantaneous gradient synchronization.

2. Failure modes compound with scale. In a tightly synchronized training run spanning thousands of devices, any single device failure threatens to stall the entire job or introduce subtle numerical inconsistencies. The paper notes this in the same passage: "the more devices that are used for each synchronous training step, the more chances there are that one of them fails, risking halting training, or introducing subtle numerical issues." Large-scale training runs routinely encounter hardware faults (GPU memory errors, network drops, thermal throttling), and the synchronous paradigm treats these as catastrophic events requiring checkpoint restoration rather than transient conditions to be absorbed.

3. Resource heterogeneity is poorly leveraged. Different accelerators — TPU v4 pods, A100 clusters, V100 servers — have different speeds, memory capacities, and network topologies. Standard data-parallel training assumes all workers are effectively identical in throughput, since a single slow worker becomes a straggler that delays the entire synchronous step. The paper explicitly flags this in Section 5 (Limitations): "the version of DiLoCo presented here assumes that all workers are homogeneous. However, in practice workers might operate at wildly different speed. In these cases, waiting for all workers to perform the same number of steps is rather inefficient." Even with the paper's synchronous outer-loop design, the ability to run different hardware types on different workers represents a step beyond the fully synchronous status quo.

4. Compute availability fluctuates over time. The paper devotes an entire experiment (Figure 7, "Adaptive compute pool") to a scenario where the number of available workers changes mid-training. This is motivated by real-world conditions: preemptible cloud instances that cycle on and off, university cluster scheduling systems that reallocate resources dynamically, and collaborative compute pools (like Petals or Diskin et al., 2021) where participants join and leave. Standard synchronous training cannot gracefully handle a worker disappearing and reappearing without checkpointing and restarting with a different parallelization configuration.

Why This Matters: The Stakes Beyond Engineering Convenience

The problem is not merely about infrastructure inconvenience — it has genuine implications for who can train large models and how the compute scaling curve progresses.

Democratization of training. If training frontier models requires a single contiguous supercomputer, the set of actors who can participate is limited to large technology companies and well-funded research labs that can either build or rent such facilities. A training algorithm that works across loosely connected, geographically distributed clusters would lower this barrier: a consortium of universities could pool their individual GPU clusters, or a company could run training across multiple regional cloud data-centers without paying premium prices for co-location.

Total compute underutilization. The paper's framing implies that there exists a substantial amount of latent compute capacity distributed across multiple sites that cannot be harnessed for a single training run under the synchronous paradigm — even though, in aggregate, it exceeds what any single site offers. This is not a theoretical concern: many research institutions, mid-size companies, and even departments within large organizations have access to multiple GPU clusters that sit idle or are used for smaller, independent jobs because they cannot be federated into one large training run.

Scaling beyond single-datacenter limits. Even for organizations that can build massive single-site clusters, there are physical limits to how many accelerators can fit in one building (power delivery, cooling, floor space). The paper's approach opens the possibility of scaling training compute beyond these physical constraints by federating across sites.

Prior Approaches and Where They Fall Short

The paper's contribution rests on identifying a gap in the distributed training literature: existing methods either communicate too frequently to be practical across poorly connected workers, or degrade unacceptably when communication is made infrequent enough for such settings. The paper's review of related work in Section 4 provides the landscape.

Standard Data Parallelism: Communication Every Step

In conventional distributed data-parallel training, each of $k$ workers processes a different micro-batch, computes gradients, and then participates in an all-reduce operation to average gradients before the optimizer updates weights. This requires communication at every step — typically every few hundred milliseconds. The paper makes this concrete in Section 6:

"While standard mini-batch methods relying on data and model parallelism require sending data every few hundred milliseconds, DiLoCo does so only every few minutes."

For workers connected via low-bandwidth, high-latency links (e.g., across the public internet, or even across different racks in a data-center with oversubscribed interconnects), this per-step communication cost would dominate the total training time, making the approach infeasible. The paper's Table 2 quantifies this: the data-parallel baseline communicates $8 \times N$ times where $N$ is the total number of training steps, while DiLoCo communicates only $8 \times N / H$ , with $H = 500$ , yielding a $500\times$ reduction.

Local SGD: The Obvious First Attempt

The natural solution to reducing communication is local SGD (also called Federated Averaging, or FedAvg, from McMahan et al., 2017). In local SGD, workers train independently on their local data for some number of steps $H$ , and then their model parameters (or parameter updates) are averaged to produce a shared model, which is re-distributed for the next round of local training. This reduces communication frequency by a factor of $H$ .

However, prior work at meaningful scale had found that local SGD struggles when $H$ is large. The paper directly cites this failure:

"Ortiz et al. (2021) is one of the few works in federated learning / local SGD body of literature that has validated on a large-scale setting. They consider ImageNet with Resnet50 and Resnet101, and found that local SGD struggles at scale. In particular, they reported that fewer inner steps (e.g., $H = 8$ ), no pretraining, and a relatively large number of replicas ( $\geq k = 16$ ) degrade generalization. Thus the authors conclude that 'local SGD encounters challenges at scale.'"

This is a critical counterpoint. Ortiz et al. (2021) tested local SGD on ImageNet with ResNets and found failure modes precisely where DiLoCo claims success: at high inner step counts ( $H = 500$ or $1000$ versus their $H = 8$ ), with no pretraining, and with many replicas ( $k = 64$ versus their $k = 16$ ). The paper emphasizes this contrast explicitly in Section 4.1:

"Instead, we show in section 3 that DiLoCo can robustly operate while communicating 125× less ( $H = 1000$ ), even without pretraining, and using up to 4× more replicas ( $k = 64$ ) both in the i.i.d. and non-i.i.d. settings."

What explains the discrepancy? The paper does not definitively answer this, but several design choices differentiate DiLoCo from the local SGD studied in Ortiz et al. (2021):

Inner optimizer choice. Prior local SGD work typically used SGD (possibly with momentum) as the inner optimizer. DiLoCo uses AdamW, the de facto standard for transformer language models, which may produce updates that are more amenable to outer averaging.
Outer optimizer sophistication. Standard FedAvg (McMahan et al., 2017) does simple parameter averaging (equivalent to outer SGD with learning rate 1.0). The FedOpt framework (Reddi et al., 2021) allows for a more sophisticated outer optimizer. DiLoCo pushes this further by using Nesterov momentum as the outer optimizer, which the paper shows empirically outperforms SGD, SGD with momentum, and Adam on this task (Figure 6).
Task domain. Ortiz et al. (2021) studied vision (ImageNet, ResNets). DiLoCo studies language modeling (C4, transformers). Transformers and CNNs have different loss landscapes and may exhibit different behavior under weight averaging. The paper acknowledges this limitation explicitly in Section 5: "other architectures (e.g., CNNs which are known to be more sensitive to linear mode connectivity) should also be considered."

Advanced Federated Optimizers: Still Too Frequent Communication

Within the federated learning community, several works extended FedAvg with more powerful outer optimizers. Reddi et al. (2021) introduced FedOpt, which applies adaptive optimizers (Adam, YOGI) to the server-side update. Wang et al. (2020) proposed SlowMo, which uses slow momentum — maintaining a momentum buffer on the server that integrates outer gradients across rounds — to stabilize training. Huo et al. (2020) proposed FedMom, which uses Nesterov momentum as the outer optimizer, similar to DiLoCo.

However, these methods were evaluated in settings where communication was still relatively frequent:

"only FedMom (Huo et al., 2020) considers Nesterov as the outer optimizer as we did. While they also tackle a language modeling task, the setting is much smaller (1-layer LSTM), with only 2 replicas, and rather frequent communication (every 20 inner steps)."

The paper emphasizes this as a distinction: the communication frequency in prior federated optimizers ( $H \leq 20$ ) is too high for truly distributed, poorly-connected workers, where each communication round might take minutes rather than milliseconds. A method communicating every 20 steps would still spend most of its time in communication if the link is slow.

The Linear Mode Connectivity Perspective

A parallel thread of research, independent from federated learning, studies linear mode connectivity: the empirical observation that models fine-tuned from a shared initialization can be interpolated in weight space (by averaging their parameters) without encountering a loss barrier — a region where the interpolated model performs worse than either endpoint. Frankle et al. (2020) coined the term; Wortsman et al. (2021, 2022b) demonstrated practical applications like "model soups" for vision.

This literature provides a theoretical intuition for why DiLoCo might work: if independently trained model replicas remain in the same loss basin (linearly connected with low loss), then averaging their parameters is a sound operation. The paper draws on this explicitly:

"Wortsman et al. (2022c) started from a pretrained model, finetuned different replicas on various tasks or choice of hyperparameters (Wortsman et al., 2022b), and then averaged the resulting parameters."

However, works in this tradition typically average only once, at the end of fine-tuning, rather than iteratively during training. The paper notes this distinction:

"The majority of works on linear connectivity considers only averaging once all replicas have been fully finetuned, while we exploit the linear mode connectivity during training."

There are exceptions: BTM (Branch-Train-Merge) (Li et al., 2022) and PAPA (Population Parameter Averaging) (Jolicoeur-Martineau et al., 2023) both perform iterative averaging, but they use outer SGD with learning rate 1.0 (simple averaging). Git-theta (Kandpal et al., 2023) proposes using weight merging for collaboration between teams, but again as a one-shot operation. The paper positions DiLoCo as building on these ideas while (a) using a more sophisticated outer optimizer (Nesterov momentum), (b) operating at much larger inner step counts ( $H = 500$ ), and (c) doing so during training rather than only between independently trained checkpoints.

Accelerating a Single Worker: The Lookahead Connection

The paper also draws a connection to the Lookahead optimizer (Zhang et al., 2019), which interpolates between the starting and ending parameters of a phase of training for a single worker (equivalent to DiLoCo with $k = 1$ ). The paper shows in Figure 9 that this single-worker variant provides both faster convergence and better final generalization than standard training, even without any distributed benefit. This suggests that the outer optimization step has value beyond merely enabling distributed execution — it may act as a form of implicit regularization or exploration in weight space.

How DiLoCo Positions Itself

The paper positions DiLoCo not as a theoretical contribution — the algorithm is directly derived from prior work (FedAvg, FedOpt, FedMom) — but as an empirical validation at a scale and regime that prior work suggested would fail. The key positioning claims are:

1. It works where local SGD was expected to fail. By combining AdamW as inner optimizer, Nesterov momentum as outer optimizer, and large inner step counts ( $H = 500$ or more), DiLoCo achieves robust convergence in regimes that Ortiz et al. (2021) identified as challenging for local SGD. The paper attributes this to the specific optimizer combination and the domain (transformers for language modeling rather than CNNs for vision).

2. It matches or exceeds synchronous baselines while communicating 500× less. The headline result (Figure 2, Table 2) is that DiLoCo with 8 workers achieves better perplexity than a fully synchronous baseline with the same total steps, while training 8× faster in wall-clock time and communicating 500× less. This is a direct refutation of the assumption that frequent synchronization is necessary for good model quality.

3. It is robust across multiple axes of variation. The paper systematically tests robustness to:

Communication frequency (Figure 4): performance degrades only 2.9% when going from $H = 50$ to $H = 1000$ (20× less communication).
Data distribution (Figure 5): non-i.i.d. and i.i.d. data shards achieve comparable final perplexity.
Number of workers (Table 3): up to 64 workers with diminishing returns beyond 8, but no catastrophic degradation.
Dynamic worker pools (Figure 7): total compute matters, not when it's available.
Dropped communication (Figure 8): up to 50% communication failure causes only 2.1% perplexity degradation.

4. It reframes the distributed training problem from infrastructure to algorithm design. The paper's underlying argument is that the assumption of co-located, tightly synchronized training is not a fundamental requirement of stochastic gradient descent but an artifact of how we've implemented it. By borrowing from federated learning and investing in a better outer optimizer, the same model quality can be achieved with dramatically relaxed infrastructure requirements.

In essence, DiLoCo doesn't claim to improve model quality in an absolute sense — the 8× updates baseline in Table 2 (which trains for 8× more steps sequentially) achieves 14.72 perplexity versus DiLoCo's 15.02. Instead, it claims to make a specific Pareto-optimal tradeoff: given access to multiple distributed compute clusters that cannot communicate efficiently, DiLoCo extracts near-optimal model quality from that compute without requiring infrastructure changes.

3. Technical Approach

3.1 Reader Orientation

DiLoCo is a distributed optimization algorithm that trains a single language model across multiple independent groups of devices ("workers") that communicate only occasionally — once every few hundred training steps — rather than at every gradient computation. The core problem it solves is: given access to several physically separated GPU clusters that CANNOT efficiently exchange gradients at every step (due to low bandwidth, high latency, or reliability issues), how can we still train a model as effectively as if all the devices were tightly synchronized in one cluster? The solution's shape is a two-level optimization loop: each worker trains its own model replica independently on local data for many steps using AdamW (the "inner loop"), then all workers send their cumulative weight changes to a central coordinator that aggregates them using Nesterov momentum (the "outer loop"), updates the shared model, and redistributes it — repeating this cycle tens or hundreds of times.

3.2 Big-Picture Architecture (Diagram in Words)

The system has four major components arranged in a hierarchical, synchronized outer-loop-over-inner-loops structure:

$k$ Workers (model replicas). Each worker is an independent compute cluster (e.g., a group of GPUs or TPUs at a specific geographic location) that holds a complete copy of the model parameters. Each worker trains on its own data shard using standard AdamW optimization, completely independently of other workers, for $H$ consecutive steps.
Data Shards. The training dataset is partitioned into $k$ disjoint subsets (shards), one per worker. These shards can be randomly split (i.i.d. setting) or clustered by semantic content (non-i.i.d. setting, created via k-means on sentence embeddings). Each worker only samples from its own shard during inner optimization.
Outer Gradient Computation (centralized). After all workers complete $H$ inner steps, they each compute an outer gradient — the vector difference between the shared model parameters they started with and the parameters they arrived at after $H$ steps of local training. These outer gradients are averaged across workers (potentially weighted by shard size in the non-i.i.d. case). This is the ONLY moment of communication in the entire training pipeline.
Outer Optimizer (Nesterov momentum). The averaged outer gradient is fed into a Nesterov momentum optimizer that updates the shared model parameters. This updated shared model is then redistributed to all workers, which resume their inner optimization from this new starting point. The outer optimizer maintains its own momentum buffer across outer steps.

Information flow, step by step:

The shared model $\theta^{(t-1)}$ (output of the previous outer step) is copied to all $k$ workers.
Each worker $i$ sets its local model $\theta^{(t)}_i = \theta^{(t-1)}$ and runs AdamW optimization for $H$ steps, sampling mini-batches exclusively from its data shard $\mathcal{D}_i$ , producing an updated local model $\tilde{\theta}^{(t)}_i$ .
Each worker computes its outer gradient: $\Delta^{(t)}_i = \theta^{(t-1)} - \tilde{\theta}^{(t)}_i$ — the net parameter change over $H$ steps.
A central coordinator (which can be a lightweight CPU server) collects all $\Delta^{(t)}_i$ , averages them: $\Delta^{(t)} = \frac{1}{k}\sum_{i=1}^k \Delta^{(t)}_i$ .
The coordinator applies the outer optimizer (Nesterov momentum with hyperparameters $\eta_{\text{outer}}$ and momentum $\mu$ ): $\theta^{(t)} = \text{Nesterov}(\theta^{(t-1)}, \Delta^{(t)}, \eta_{\text{outer}}, \mu)$ .
The updated $\theta^{(t)}$ is sent to all workers, and the cycle repeats for $T$ outer steps.

At inference time, the model is a standard transformer identical in architecture and computational cost to any synchronized-trained model. The distribution strategy only affects training.

3.3 Roadmap for the Deep Dive

First, the formal algorithm specification (Algorithm 1) and the core equations governing inner and outer optimization, because every subsequent design choice and ablation references this structure.
Second, the inner optimization loop — AdamW as the inner optimizer, the $H$ hyperparameter, and why AdamW matters for this specific distributed regime.
Third, the outer gradient computation and aggregation, because this is the sole communication interface and its properties (similarity, norm, variance) determine whether DiLoCo converges.
Fourth, the outer optimizer — Nesterov momentum — and the empirical comparison against SGD, SGD with momentum, and Adam, because the choice of outer optimizer is the paper's primary algorithmic contribution beyond standard FedAvg.
Fifth, the pretraining strategy and its relationship to DiLoCo initialization, because the paper shows that pretraining is NOT required but that a warmup period matters.
Sixth, the non-i.i.d. data shard construction (k-means clustering on sentence embeddings) and the weighted averaging scheme, because these are the practical mechanisms for handling heterogeneous data distributions.
Seventh, the hyperparameter configuration table and the full training recipe, because reproducibility requires exact values.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical systems paper whose core idea is that Federated Averaging-style distributed training can be made to work robustly at scale for transformer language models — contrary to prior negative results — by replacing the inner optimizer with AdamW (the standard for LLMs), the outer optimizer with Nesterov momentum, and using very large inner step counts ( $H = 500$ ). The theoretical basis is not novel; the contribution is demonstrating that this specific combination works in a regime where prior work predicted failure, and systematically characterizing its behavior.

The Two-Level Optimization Structure (Algorithm 1)

DiLoCo's algorithm is formally specified in Algorithm 1 of the paper, which defines a nested optimization procedure. The outer loop runs for $T$ steps; each outer step $t$ consists of distributing the current shared model to workers, having each worker run $H$ inner optimization steps independently, collecting the resulting model changes, and applying an outer optimizer update.

The outer loop iteration $t$ (lines 1-15):

For $t = 1, \ldots, T$ :

Distribution (line 3): For each worker $i$ (where $i = 1, \ldots, k$ ), set the worker's local model parameters $\theta^{(t)}_i$ to the current shared parameters $\theta^{(t-1)}$ . This is a copy operation — the worker receives the EXACT same initial state.

Inner optimization (lines 4-9): For $h = 1, \ldots, H$ :

Sample a mini-batch $x \sim \mathcal{D}_i$ from the worker's local data shard.
Compute the loss $\mathcal{L} \leftarrow f(x, \theta^{(t)}_i)$ , where $f$ is the language modeling objective (next-token prediction cross-entropy).
Update local parameters: $\theta^{(t)}_i \leftarrow \text{InnerOpt}(\theta^{(t)}_i, \nabla\mathcal{L})$ , where $\text{InnerOpt}$ is AdamW.

After $H$ steps, the worker has performed $H$ independent AdamW updates, producing a locally optimized parameter vector. Critically, the worker does NOT communicate anything during these $H$ steps — it operates as if it were training a standalone model on its shard.

Outer gradient computation (lines 11-12): Each worker $i$ computes its outer gradient:

$\Delta^{(t)}_i = \theta^{(t-1)} - \theta^{(t)}_i$

where $\theta^{(t-1)}$ is the shared parameters at the START of the inner phase (what the worker received at line 3), and $\theta^{(t)}_i$ is the parameters at the END of $H$ inner steps (after line 8).

What it computes: the net vector displacement in parameter space over $H$ inner training steps on worker $i$ 's local data shard. It is NOT a gradient in the infinitesimal sense — it is the accumulated effect of $H$ full AdamW updates, which includes momentum, adaptive learning rates, and weight decay. The vector points in the direction that the worker's local optimization moved the model.

Why this form: using the net parameter DISPLACEMENT rather than the sum of per-step gradients means the outer update captures the full effect of the inner optimizer's dynamics, not just the raw gradient direction. This is crucial because AdamW involves adaptive per-parameter learning rates and momentum — two workers with identical mini-batches in identical order would produce identical $\Delta^{(t)}_i$ , but the underlying per-step gradient vectors would differ due to the adaptive state. The displacement $\theta^{(t-1)} - \theta^{(t)}_i$ is the sufficient statistic for "what the worker learned."

The outer gradients are then averaged across all $k$ workers:

$\Delta^{(t)} = \frac{1}{k} \sum_{i=1}^k \Delta^{(t)}_i = \frac{1}{k} \sum_{i=1}^k \left(\theta^{(t-1)} - \theta^{(t)}_i\right)$

What it computes: the unweighted mean displacement across all workers. For the non-i.i.d. setting where shards are of unequal size, the paper uses a weighted average where each $\Delta^{(t)}_i$ is scaled by the number of examples in its shard before averaging — this prevents small idiosyncratic shards from over-contributing.

Why this form: uniform averaging (or size-weighted averaging) is the standard aggregation in Federated Averaging (McMahan et al., 2017). The implicit assumption is that each worker's local data provides an unbiased sample of the overall data distribution. In the non-i.i.d. case, this assumption is violated, but the paper empirically demonstrates (Figure 5) that averaging still produces a model that generalizes well — likely because the $H = 500$ inner steps allow each worker to move sufficiently far that their displacements capture generalizable learning rather than shard-specific memorization.

Outer optimization (line 14):

$\theta^{(t)} = \text{OuterOpt}\left(\theta^{(t-1)}, \Delta^{(t)}\right)$

where $\text{OuterOpt}$ is Nesterov momentum with learning rate $\eta_{\text{outer}} = 0.7$ and momentum $\mu = 0.9$ .

What it computes: the new shared model parameters, obtained by applying a momentum-based update to $\theta^{(t-1)}$ using the averaged outer gradient $\Delta^{(t)}$ . The specific Nesterov update is:

$v^{(t)} = \mu \cdot v^{(t-1)} - \eta_{\text{outer}} \cdot \Delta^{(t)}$ $\theta^{(t)} = \theta^{(t-1)} + \mu \cdot v^{(t)} - \eta_{\text{outer}} \cdot \Delta^{(t)}$

where $v^{(t)}$ is the outer momentum buffer (initialized to zero at $t=0$ ).

Why this form: the paper empirically compared SGD, SGD with momentum, Adam, and Nesterov momentum as outer optimizers (Figure 6). SGD (equivalent to standard FedAvg) and Adam both performed poorly — SGD converges too slowly because it lacks momentum to smooth the outer updates, while Adam was "particularly unstable with a high second-order momentum norm" (the paper increased $\epsilon$ to 0.1 to mitigate this instability). Nesterov momentum outperforms standard momentum (SGDM) because Nesterov's "look-ahead" correction — where the gradient is evaluated at the momentum-extrapolated point rather than the current point — provides more accurate updates when the outer gradient spans hundreds of inner steps and thus approximates a macro-level direction rather than a local gradient. The specific values $\eta_{\text{outer}} = 0.7$ and $\mu = 0.9$ were found via grid search (Table 5).

Total training steps: Each worker trains for $\text{total steps} = T \times H$ inner steps. In the main experiments, $T = 128$ outer steps, $H = 500$ inner steps, giving 64,000 DiLoCo steps, plus 24,000 pretraining steps, for 88,000 total.

The Inner Optimizer: Why AdamW Specifically

The inner optimizer is AdamW(Kingma and Ba, 2014; Loshchilov and Hutter, 2019), the standard optimizer for transformer language models. The paper states this choice explicitly:

"We use as inner optimizer (InnerOpt) AdamW, which is the most widely used optimizer for transformer language models."

This is not an arbitrary choice — it carries specific implications for DiLoCo's behavior that differ from the SGD inner optimizers used in prior local SGD literature.

AdamW's update rule at inner step $h$ on worker $i$ :

Compute gradient $g_h = \nabla \mathcal{L}(\theta^{(t)}_{i,h-1})$ on a mini-batch $x \sim \mathcal{D}_i$ .
Update biased first moment estimate: $m_h = \beta_1 m_{h-1} + (1 - \beta_1) g_h$ (with $\beta_1 = 0.9$ ).
Update biased second moment estimate: $v_h = \beta_2 v_{h-1} + (1 - \beta_2) g_h^2$ (with $\beta_2 = 0.999$ , element-wise).
Correct bias: $\hat{m}_h = m_h / (1 - \beta_1^h)$ , $\hat{v}_h = v_h / (1 - \beta_2^h)$ .
Update parameters: $\theta_{i,h} = \theta_{i,h-1} - \eta_{\text{inner}} \cdot \frac{\hat{m}_h}{\sqrt{\hat{v}_h} + \epsilon} - \eta_{\text{inner}} \cdot \lambda \cdot \theta_{i,h-1}$ , where $\eta_{\text{inner}} = 4 \times 10^{-4}$ (inner learning rate), $\epsilon = 10^{-8}$ , and $\lambda = 0.1$ (weight decay).

The final term $-\eta_{\text{inner}} \cdot \lambda \cdot \theta_{i,h-1}$ is the decoupled weight decay (Loshchilov and Hutter, 2019), which regularizes the model independently of the adaptive learning rate normalization.

Why AdamW matters for DiLoCo (implicit in the paper, but structurally important):

Adaptive per-parameter learning rates. AdamW's $\hat{m}_h / \sqrt{\hat{v}_h}$ term normalizes updates so that each parameter moves at roughly the same effective rate, regardless of the local gradient scale. This means the outer gradient $\Delta^{(t)}_i$ captures parameter changes that are already "normalized" — the displacement vector is less noisy because each coordinate has been individually scaled. In contrast, SGD produces displacements that are proportional to raw gradient magnitudes, which can vary wildly across layers, making outer averaging less meaningful.
Momentum smoothing. AdamW's first moment $m$ acts as an exponential moving average of gradients, smoothing out mini-batch noise. Over $H = 500$ steps, this smoothing produces a displacement that reflects the dominant gradient direction rather than high-frequency stochastic fluctuations. This makes the outer gradient more aligned across workers (the paper's appendix Figure 10 shows high cosine similarity between workers' outer gradients in the i.i.d. case), which in turn makes outer averaging stable.
Weight decay integration. AdamW's decoupled weight decay ensures that each inner step applies explicit L2 regularization that shrinks weights toward zero. This prevents the model from drifting to extreme parameter values during $H = 500$ steps of unconstrained local training — without it, non-i.i.d. workers might diverge to very different minima, producing outer gradients that are in opposition rather than complementary.

Inner optimizer state is NOT shared. Each worker maintains its own AdamW optimizer state (the $m$ and $v$ buffers). The paper explicitly considered synchronizing these states:

"DiLoCo synchronizes the parameters of the model, but we also considered synchronizing the inner optimizer states. It did not lead to significant improvements while significantly increasing the communication cost (×3 more data to transmit)."

This means communication during the outer step transmits only the $d$ parameters of the model, not the $2d$ additional AdamW state values. This is significant at scale: for a 400M parameter model, synchronizing states would require transmitting 1.2B floats per worker per outer step. The paper's finding that unsynchronized AdamW states do not harm convergence is consistent with prior work (Ortiz et al., 2021; Wang et al., 2020) where SGD with momentum was used as the inner optimizer.

Inner learning rate schedule. The inner learning rate follows a cosine decay schedule with a linear warmup of 1,000 steps. The paper notes in Section 3.1 that this warmup causes a transient perplexity spike at the start of DiLoCo training (visible in Figure 3 after the vertical dashed lines), because the learning rate is reset to a small value and then ramped up. This spike is benign — performance recovers and the warmup is ultimately beneficial, consistent with findings in continual pretraining (Gupta et al., 2023). The cosine decay means that as training progresses, the inner learning rate approaches zero, making outer gradients naturally smaller — which the paper observes eliminates the need to decay the outer learning rate:

"Since we decay the inner learning rate, the outer gradient norm gets naturally smaller over the course of training, removing the need to further decay the outer learning rate."

The Outer Gradient: Properties and Aggregation

The outer gradient $\Delta^{(t)}_i$ is conceptually the "macro-gradient" that summarizes what worker $i$ learned during $H$ inner steps. Understanding its properties explains DiLoCo's robustness.

Magnitude and scaling. As the inner learning rate decays toward zero over the course of training, the norm of each $\Delta^{(t)}_i$ naturally decreases. This is a desirable property because it means the outer optimizer receives updates of appropriate scale without needing an explicit outer learning rate schedule. The paper tried cosine decay on the outer learning rate and found "similar performance," confirming that the inner decay suffices.

Cosine similarity between workers (Appendix, subsection 6.2, Figure 10). This is one of the paper's most informative diagnostic analyses. The authors compute the average pairwise cosine similarity between outer gradients from different workers:

$\text{sim}(\Delta^{(t)}_i, \Delta^{(t)}_j) = \frac{\langle \Delta^{(t)}_i, \Delta^{(t)}_j \rangle}{\|\Delta^{(t)}_i\| \cdot \|\Delta^{(t)}_j\|}$

for all pairs $i \neq j$ in $\{1, \ldots, k\}$ , and report the mean and standard deviation over the $k(k-1)/2$ pairs.

In the i.i.d. setting (Figure 10a): The similarity is high (close to 1.0) throughout training, with tiny variance. This makes intuitive sense — all workers train on independent samples from the same distribution, so their AdamW trajectories over $H$ steps should produce similar net displacements. The high similarity means averaging is safe: the outer gradient $\Delta^{(t)}$ is a consensus direction that all workers agree on.

In the non-i.i.d. setting (Figure 10b): The similarity is LOWER on average but with much HIGHER variance. Workers whose data shards are semantically different (e.g., one shard has mostly scientific text, another has mostly fiction) produce outer gradients that point in somewhat different directions. However — and this is the crucial empirical finding — the similarity does NOT collapse to zero or go negative. The outer gradients are still positively correlated enough that averaging produces a useful update direction. Interestingly, the variance INCREASES toward the end of training as the inner learning rate decays — the paper speculates:

"Since shards have a different distribution, each local optimization seems to fall in a different nearby loss basin."

In other words, as training converges, each worker finds a local minimum specific to its data distribution, and these minima are distinct but nearby in parameter space. The outer averaging step interpolates between them.

Effect of $H$ on outer gradient similarity (Figure 10). Perhaps counterintuitively, the cosine similarity is HIGHER when $H$ is LARGER. The paper reports similarity for $H = 250$ , $H = 500$ , and $H = 1000$ , and similarity increases with $H$ in both i.i.d. and non-i.i.d. settings. The explanation:

"We surmise that when the number of inner step is larger (up to some extent) model replicas converge towards a similar general direction (Gu et al., 2023) averaging out the noise of stochastic gradient descent."

This is a critical insight: longer local training does NOT cause workers to diverge more; instead, the per-step stochastic noise averages out, and the net displacement reflects a more stable, generalizable learning signal. This is what enables DiLoCo to use $H = 500$ or even $H = 1000$ when prior work (like Ortiz et al., 2021) failed at $H = 8$ — AdamW over 500 steps on language data produces a displacement that is robust rather than noisy.

Effect of number of replicas on similarity (Appendix, Figure 11). As $k$ increases from 4 to 8, the average cosine similarity between outer gradients decreases. This is expected: with more shards, each shard covers a narrower slice of the data distribution, so the shard-specific displacements become more distinct. The paper also notes that the norm of the averaged outer gradient $\|\Delta^{(t)}\|$ scales inversely with $\sqrt{k}$ , which follows from the central limit theorem if the outer gradients are approximately independent and identically distributed around a common mean.

Weighted vs. uniform averaging. For the non-i.i.d. setting, the paper states:

"For the non-i.i.d. data regime, we rescale each outer gradient by the number of examples in its shard. While at $k = 4$ , all clusters are quite balanced, imbalance can be striking at $k = 64$ and giving more importance to larger clusters is beneficial."

This is a simple size-weighted average: $\Delta^{(t)} = \sum_{i=1}^k w_i \Delta^{(t)}_i$ where $w_i \propto |\mathcal{D}_i|$ . This prevents a small, idiosyncratic shard from over-contributing to the outer update.

Compression of outer gradients (Appendix, Table 6). The paper explores whether outer gradients can be further compressed before transmission using per-neuron sign pruning (Yadav et al., 2023): for each parameter in the model, if its outer gradient value falls in the bottom $p\%$ of magnitudes for that neuron's parameter group, it is set to zero. Results:

25% pruning: perplexity 15.01 (vs. 15.02 baseline, a 0.06% improvement — noise-level).
50% pruning: perplexity 15.08 (+0.39% degradation).
75% pruning: perplexity 15.27 (+1.66% degradation).

50% pruning is essentially free, meaning the communication volume can be halved with trivial compression.

The Outer Optimizer: Nesterov Momentum

The choice of outer optimizer is the paper's primary algorithmic innovation over standard Federated Averaging. The paper systematically compared four options in Figure 6:

SGD (OuterOpt = SGD, $\eta_{\text{outer}}$ varied among $\{1.0, 0.7, 0.5, 0.3, 0.1\}$ ): This is equivalent to standard FedAvg (McMahan et al., 2017). The update is simply $\theta^{(t)} = \theta^{(t-1)} - \eta_{\text{outer}} \cdot \Delta^{(t)}$ . The paper found this performed "poorly" — convergence was slow because each outer step is treated independently with no memory of the trajectory. Setting $\eta_{\text{outer}} = 1.0$ makes this exact parameter averaging (souping; Wortsman et al., 2021), which the paper found suboptimal.

SGD with momentum (OuterOpt = SGD with momentum, $\eta_{\text{outer}}$ varied, $\mu = 0.9$ ): The update is $v^{(t)} = \mu v^{(t-1)} + \eta_{\text{outer}} \Delta^{(t)}$ , $\theta^{(t)} = \theta^{(t-1)} - v^{(t)}$ . This performed better than plain SGD because the momentum buffer smooths across outer steps, but still underperformed Nesterov.

Adam (OuterOpt = Adam, $\eta_{\text{outer}}$ varied, $\beta_1 = 0.9$ , $\beta_2 = 0.999$ or $0.95$ ): The paper found Adam to be "particularly unstable with a high second-order momentum norm." The second moment $v$ accumulated rapidly because outer gradients can be large in magnitude (spanning 500 inner steps), causing the effective learning rate $\eta / \sqrt{v}$ to become very small and training to stall. Increasing $\epsilon$ to 0.1 (from the typical $10^{-8}$ ) mitigated but did not fully resolve the issue. Adam performed the worst among all tested.

Nesterov momentum (OuterOpt = Nesterov, $\eta_{\text{outer}} = 0.7$ , $\mu = 0.9$ ): This performed best and was adopted for all experiments. The Nesterov update is:

$v^{(t)} = \mu \cdot v^{(t-1)} - \eta_{\text{outer}} \cdot \Delta^{(t)}$ $\theta^{(t)} = \theta^{(t-1)} + \mu \cdot v^{(t)} - \eta_{\text{outer}} \cdot \Delta^{(t)}$

where $v^{(t)}$ is a velocity buffer (initialized to zero) and $\mu = 0.9$ is the momentum coefficient.

What it computes: Nesterov momentum differs from standard momentum in HOW the gradient is applied. In standard momentum, the parameter update would be $\theta^{(t)} = \theta^{(t-1)} + v^{(t)}$ — i.e., apply the velocity as computed. In Nesterov, the gradient correction $\mu \cdot v^{(t)} - \eta_{\text{outer}} \cdot \Delta^{(t)}$ is applied from $\theta^{(t-1)}$ , which can be interpreted as: "first, take a step in the direction of accumulated momentum, THEN evaluate the gradient and apply a correction." This is the "look-ahead" property.

Why this form for DiLoCo specifically: The paper hypothesizes:

"We hypothesize that Nesterov's gradient correction is particularly helpful with the outer gradient that span hundreds of training steps."

The rationale is that $\Delta^{(t)}$ is not a local gradient — it's a macro-displacement accumulated over $H = 500$ inner steps. Standard momentum would combine this macro-displacement with the previous velocity without adjusting for the fact that the "gradient" was evaluated at the starting point $\theta^{(t-1)}$ but describes the movement FROM that point, not the gradient AT that point. Nesterov's correction accounts for this by effectively evaluating the update at the momentum-projected point, which better handles large, coarse-grained updates.

No outer learning rate decay. The paper tested cosine decay on the outer learning rate and found it unnecessary:

"Since we decay the inner learning rate, the outer gradient norm gets naturally smaller over the course of training, removing the need to further decay the outer learning rate."

This is an important simplification — it means the outer optimizer uses a constant learning rate $\eta_{\text{outer}} = 0.7$ throughout training, with the natural decay of outer gradient magnitudes providing implicit scheduling.

Pretraining Strategy and Initialization

All DiLoCo experiments in the main configuration start from a model $\theta^{(0)}$ that has been pretrained for 24,000 steps on the C4 dataset using standard (single-worker) training with the same inner optimizer (AdamW). The total training budget is 88,000 steps: 24,000 pretraining + 64,000 DiLoCo (with $T = 128$ outer steps and $H = 500$ inner steps per outer step).

Why start from a pretrained model? The paper initially assumed this was necessary, given the linear mode connectivity literature's finding that models finetuned from a shared initialization are more amenable to weight averaging (Wortsman et al., 2022c). However, the ablation in Figure 3 showed that pretraining is NOT strictly required:

"In general, we observe that starting DiLoCo before 24k steps achieves a similar final PPL, demonstrating the robustness of the approach. Interestingly, performance is not degraded even when starting from a randomly initialized network. This result contradicts the findings of prior work on post local-SGD (Lin et al., 2020) and its large-scale study on a vision classification task (Ortiz et al., 2021)."

The Figure 3 results:

No pretraining (starting from random initialization): final perplexity ~15.1 (only -0.1 from the 24k-pretrained baseline).
12k pretraining steps: virtually identical to 24k and 48k.
48k pretraining steps: also performs well, but at this point the DiLoCo phase is short (40,000 steps = 80 outer steps).

The key practical takeaway: DiLoCo can train from scratch, but a modest pretraining period (12k-24k steps) provides a small benefit. This is important because pretraining on a single cluster avoids the distributed communication overhead during the initial, high-learning-rate phase when gradients are largest and most volatile.

Inner learning rate warmup at DiLoCo start. When DiLoCo begins (after pretraining), the inner learning rate is reset to zero and linearly warmed up over 1,000 steps to $4 \times 10^{-4}$ . This causes a transient perplexity spike visible in Figure 3 (the sharp uptick at each vertical dashed line). The paper explicitly addresses this:

"The attentive reader may also note spikes in perplexity after the vertical dashed lines: a warm-up of the inner learning rate is the culprit. Despite the transient spike, such warm up is ultimately beneficial, as previously noted also in the continual pretraining setting by Gupta et al. (2023)."

The warmup prevents the inner optimizer from taking destabilizing large steps on the freshly initialized optimizer state after the outer synchronization has changed the parameters.

Non-I.I.D. Data Shard Construction

The paper tests two data distribution regimes:

I.I.D. (independent and identically distributed): The training dataset is randomly shuffled and partitioned into $k$ equal-sized shards. Each shard is a uniform random sample from the full dataset, so all shards have the same expected data distribution.

Non-I.I.D. (heterogeneous): The training dataset is partitioned such that different shards contain SEMANTICALLY DIFFERENT data. The construction method follows Gururangan et al. (2023):

A pretrained language model (the same model used for DiLoCo initialization) processes every example in the training set and extracts the last-layer hidden representation (before the output projection) for the final token of each document.
These embedding vectors are clustered into $k$ groups using $k$ -means clustering.
Each cluster becomes one data shard $\mathcal{D}_i$ .

This produces shards where each shard contains topically similar documents (e.g., one shard might contain scientific articles, another might contain dialogue, another might contain legal text). The shards are NOT uniformly sized — $k$ -means clustering produces clusters of varying cardinality based on the natural data distribution.

Why $k$ -means on features? This is a stronger test of robustness than random partitioning — it simulates a realistic federated learning scenario where different institutions hold data from different domains. If DiLoCo's outer averaging failed under semantic heterogeneity (workers pulling toward domain-specific minima), the final model would perform poorly on the validation set (which spans all domains). The fact that the i.i.d. and non-i.i.d. curves in Figure 5 converge to the same perplexity demonstrates that DiLoCo successfully integrates diverse optimization trajectories into a single generalizing model.

Weighted averaging for non-i.i.d.: As noted above, the outer gradient aggregation weights each $\Delta^{(t)}_i$ by $|\mathcal{D}_i|$ (the number of examples in that worker's cluster). Without this, small but highly idiosyncratic shards could dominate the average.

Hyperparameter Configuration and Training Recipe

The complete hyperparameter specification is in Table 5 of the paper (Appendix). The values chosen for the main experiments (in bold) are:

Model architecture (Table 1):

150M parameter model: 12 transformer layers, hidden dimension 896, 16 attention heads, key/value size 64, vocabulary size 32,000 (SentencePiece tokenizer).
Comparison models: 60M (3 layers, dim 896, 16 heads) and 400M (12 layers, dim 1536, 12 heads, K/V size 128).

Inner optimization (AdamW):

Inner learning rate: $4 \times 10^{-4}$
Weight decay: $0.1$
Adam $\beta_1$ : $0.9$
Adam $\beta_2$ : $0.999$ (standard, though the outer Adam ablation tested 0.95)
Adam $\epsilon$ : $10^{-8}$ (standard; the outer Adam ablation increased this to 0.1)
Batch size: 512 sequences per worker (inner step)
Sequence length: 1,024 tokens
Number of warmup steps: 1,000
Inner learning rate schedule: cosine decay to 0 after warmup

Outer optimization (Nesterov momentum):

Outer learning rate: $0.7$
Outer momentum: $0.9$
No outer learning rate schedule (constant)

Communication and distribution:

Communication frequency $H$ : 500 inner steps
Number of outer steps $T$ : 128
Number of pretraining steps: 24,000
Number of workers $k$ : 8 (default; tested 1, 4, 16, 64)
Data regime: non-i.i.d. (default; tested i.i.d.)

Training trajectory:

Step 0–24,000: Single-worker pretraining (standard AdamW on full C4 or a subset, depending on the experiment).
Step 24,000–88,000: DiLoCo training with $k$ workers, each performing $T = 128$ rounds of $H = 500$ inner steps.
Total steps: 88,000 (same for ALL baselines to ensure fair comparison in terms of wall-clock time).

Why batch size 512? The baseline comparisons in Table 2 use different batch size configurations to isolate the effect of parallel compute versus distributed communication:

Baseline (single worker): batch size 512, 1× training time.
Baseline, 8× batch size with data parallelism: batch size 512 per worker across 8 workers with per-step gradient synchronization — this is standard distributed data-parallel training. Communicates 8 × total steps times (once per step per worker). 1× training time (8 workers running in parallel).
Baseline, 8× batch size with microbatching: batch size 4096 on a single worker via gradient accumulation over 8 micro-batches of 512. Communicates 0 times (all local). 8× training time (sequential micro-batches).
DiLoCo: batch size 512 per worker across 8 workers, communicating once per 500 steps. 1× training time.

The fair comparison is DiLoCo vs. the data-parallel 8× batch baseline — both use 8 workers in parallel and take the same wall-clock time, but DiLoCo communicates 500× less and achieves better perplexity (15.02 vs. 15.30).

What Makes DiLoCo Different from Standard Federated Averaging

The paper explicitly positions DiLoCo as a variant of FedAvg, but several design choices cumulatively create a substantially different algorithm:

Inner optimizer is AdamW, not SGD. Standard FedAvg (McMahan et al., 2017) and most local SGD literature (Lin et al., 2020; Ortiz et al., 2021; Stich, 2019) use SGD with optional momentum as the inner optimizer. DiLoCo uses AdamW, which is the standard for transformer LLMs. The adaptive learning rates and momentum of AdamW produce outer gradients that are more stable and aligned across workers.
Outer optimizer is Nesterov momentum, not simple averaging. Standard FedAvg uses $\theta^{(t)} = \frac{1}{k} \sum_i \theta^{(t)}_i$ , which is equivalent to outer SGD with $\eta = 1.0$ . FedOpt (Reddi et al., 2021) generalizes this to allow adaptive optimizers, but prior instantiations used SGD with momentum or Adam (which DiLoCo found unstable). DiLoCo's choice of Nesterov momentum with $\eta = 0.7$ and $\mu = 0.9$ is tuned specifically for the regime where outer gradients span hundreds of steps.
$H$ is one to two orders of magnitude larger. Prior local SGD work typically uses $H \leq 20$ (Ortiz et al., 2021 tested $H = 8$ ; FedMom used $H = 20$ ). DiLoCo uses $H = 500$ (default) and shows robustness up to $H = 1000$ with only 2.9% relative perplexity degradation. This is the enabler for truly distributed training across low-bandwidth links — at $H = 500$ , communication happens every few minutes rather than every few seconds.
No inner optimizer state synchronization. FedAvg variants sometimes synchronize optimizer states (momentum buffers) along with parameters. DiLoCo explicitly does NOT, finding that it "did not lead to significant improvements while significantly increasing the communication cost (×3 more data to transmit)."
Target domain: large-scale transformers for language modeling. Most prior local SGD results are on vision benchmarks (ImageNet with ResNets) or very small-scale language tasks (1-layer LSTM). DiLoCo validates at 150M–400M parameter transformers on C4, a realistic scale for modern LM training, and shows that transformers may be more amenable to the approach than CNNs.

Edge Cases: Special Operating Modes

The paper explores several variations of the core algorithm:

Single-worker acceleration (Figure 9, Section 3.1): DiLoCo with $k = 1$ (one worker, no distribution). Every $H = 500$ inner steps, the single worker computes $\Delta^{(t)} = \theta^{(t-1)} - \theta^{(t)}_1$ and applies the Nesterov outer optimizer to update the parameters. This is essentially the Lookahead optimizer (Zhang et al., 2019) with Nesterov momentum replacing SGD in the outer step. The result: faster convergence and better final generalization than standard AdamW training, at ZERO communication cost. This demonstrates that the outer optimization step has value beyond enabling distribution — it acts as a regularizer or exploration mechanism.

Asynchronous communication (Figure 8, Section 3.1): Each worker has a probability $p$ of "dropping" its outer gradient in a given round. If dropped, the worker continues training for the next $H$ steps from its OWN local parameters $\theta^{(t)}_i$ rather than receiving the shared $\theta^{(t)}$ . The outer averaging proceeds with only the gradients from workers that successfully communicated. At $p = 0.5$ (50% drop rate), final perplexity degrades by only 2.1% relative to perfect communication. This means the outer synchronization barrier is not strict — workers do not need to wait for stragglers, and transient network failures do not derail training.

Adaptive compute pool (Figure 7, Section 3.1): The number of workers $k$ is varied over the course of training according to different schedules (constant, doubling, halving, ramping up, ramping down). The finding: "the factor determining the ultimate generalization ability of the model is the total amount of compute given to DiLoCo, but this is robust to how the budget is spread over time." Workers can join or leave mid-training without requiring reconfiguration — from the algorithm's perspective, a worker that disappears simply stops contributing outer gradients, and a worker that appears receives the current shared model and starts training.

4. Key Insights and Innovations

Innovation 1: Reframing Distributed Training as a Problem of Optimizer Design, Not Infrastructure

The paper's most fundamental intellectual move is not algorithmic but conceptual: it reframes the challenge of distributed LLM training from an infrastructure problem (how do we build faster interconnects between co-located devices?) to an optimizer design problem (how do we construct a two-level optimization procedure that tolerates hundreds of steps of isolation between synchronizations?). This reframing is the enabling insight behind every empirical result in the paper.

What the field assumed before this work. The dominant paradigm assumed that frequent gradient synchronization — typically every step, or at most every few steps — is necessary for distributed training to converge properly. This assumption was grounded in real empirical evidence. Ortiz et al. (2021), in what the paper treats as the primary counterpoint, had concluded that "local SGD encounters challenges at scale" after showing that increasing the number of inner steps beyond H = 8 degraded performance on ImageNet with ResNets. The natural interpretation of that result was that models diverge when trained independently for too long, and that frequent resynchronization is a hard requirement for model quality. That interpretation, in turn, kept the field's focus on engineering solutions: faster interconnects, better all-reduce algorithms, gradient compression to fit more frequent communication into limited bandwidth.

What DiLoCo changes. DiLoCo demonstrates that the failure observed by Ortiz et al. (2021) was not a fundamental property of infrequent synchronization, but rather a consequence of the specific optimizer configuration used in those experiments (SGD inner optimizer, simple averaging outer optimizer). By replacing the inner optimizer with AdamW (the standard for transformer language models) and the outer optimizer with Nesterov momentum, DiLoCo achieves robust convergence at H = 500 — a 62.5× increase over the H = 8 where local SGD was previously reported to fail — and maintains acceptable performance even at H = 1000 (Figure 4). This is not an incremental improvement in communication efficiency; it is a qualitative regime change in what communication frequencies are viable.

Significance beyond raw performance. This reframing matters because it redirects research attention. If the problem were infrastructure (interconnect bandwidth, latency), progress would require hardware advances — faster networking equipment, better data-center topology, tighter physical integration. If the problem is optimizer design, progress can come from algorithmic innovation — better inner optimizers, outer optimizers, learning rate schedules, and initialization strategies — which is cheaper to iterate on and applicable to existing hardware. The paper does not build a single new hardware component; all experiments run on standard A100 GPUs. The 500× reduction in communication is achieved entirely through a change in the training algorithm. This makes distributed LLM training a software problem rather than a hardware one, substantially lowering the barrier to entry for organizations that cannot build or rent monolithic supercomputing clusters.

Evidence anchoring. The contrast with Ortiz et al. (2021) is explicit throughout the paper and serves as the primary baseline for the reframing claim. Table 2 quantifies the practical implication: DiLoCo with 8 workers communicates 8 × N / H times (where H = 500) versus 8 × N for the data-parallel baseline — a 500× reduction — while achieving better perplexity (15.02 vs. 15.30). Figure 4 shows that communication can be further reduced to once every 1,000 steps with only a 2.9% relative increase in perplexity.

Is this fundamental or incremental? This is a fundamental reframing, not an incremental optimization. The paper does not propose a novel algorithm in the sense of previously unknown mathematics — the components (FedAvg, Nesterov momentum, AdamW) all exist in the literature. The contribution is demonstrating that a specific, previously untested combination of these components operates in a regime that the field had empirically written off. This changes what researchers believe is possible, which is more impactful than a new algorithm that operates within the existing belief structure. However, the reframing is conditional on the domain: it was validated only on transformer language models, and the paper explicitly acknowledges that CNNs (the domain of Ortiz et al., 2021) may genuinely be more sensitive to infrequent synchronization.

Innovation 2: The Outer Gradient Cosine Similarity as a Diagnostic for When Distributed Training Will Succeed

The paper introduces a diagnostic tool — the cosine similarity between outer gradients produced by different workers — that explains why DiLoCo works and when it should be expected to work. This is not presented as a theoretical contribution (no convergence bounds are derived), but as an empirical diagnostic that provides conceptual clarity for a phenomenon that prior work had characterized only by its failures.

What the field lacked before this work. Prior local SGD literature treated the degradation at high H or high k as an empirical observation without a clear mechanistic explanation. The language was about "divergence" or "loss barriers" (Frankle et al., 2020), but there was no quantitative, per-training-run diagnostic that could predict whether a given configuration would succeed. This made hyperparameter tuning for distributed training a blind search: try different H values, see which one works, without understanding why one value fails and another succeeds.

What DiLoCo adds. The cosine similarity analysis in Appendix subsection 6.2 (Figures 10 and 11) provides a direct, interpretable measure of whether workers' local training trajectories are compatible. The key findings:

In the i.i.d. setting (Figure 10a), outer gradient similarity is consistently high (near 1.0) with minimal variance — workers are essentially moving in the same direction despite training on different samples. This is the regime where averaging is maximally safe.
In the non-i.i.d. setting (Figure 10b), similarity is lower on average and more variable, but remains positive throughout training. Workers never enter a regime where they are pulling in opposite directions; they simply pull in slightly different directions.
Counterintuitively, similarity increases with larger H (comparing H = 250, H = 500, H = 1000 in Figure 10). Longer isolated training does not cause divergence — it allows stochastic noise to average out, making each worker's net displacement more reflective of a stable, generalizable direction.

This third finding is particularly significant because it inverts the intuition from prior work. Ortiz et al. (2021) implicitly assumed that more inner steps → more divergence → worse averaging. DiLoCo shows the opposite: more inner steps (up to some threshold) → more stable outer gradients → better averaging. The paper's explanation invokes Gu et al. (2023): "model replicas converge toward a similar general direction averaging out the noise of stochastic gradient descent."

Significance as a research tool. The cosine similarity diagnostic transforms distributed training from a black-box optimization problem into something that can be monitored and debugged. If similarity drops near zero or goes negative at a particular outer step, that signals a problem — perhaps the learning rate is too high, the data distribution is too heterogeneous for the current number of workers, or the outer optimizer's momentum is causing oscillation. This is analogous to how gradient norm and loss curves serve as diagnostics for single-worker training. The paper does not develop this into a full monitoring methodology, but the data in Figure 10 and 11 provide a template that future work can adopt.

Evidence anchoring. Figure 10 shows the average pairwise cosine similarity and standard deviation across outer steps for both i.i.d. and non-i.i.d. settings at three different H values. Figure 11 shows how similarity decreases as the number of replicas increases from k = 4 to k = 8 in the non-i.i.d. setting, with the additional observation that the outer gradient norm scales as 1/sqrt(k). These figures are in the appendix, which is appropriate — they are diagnostic, not headline results — but they encode the mechanistic insight that the main paper's robustness claims depend on.

Is this fundamental or incremental? This is a conceptual advance (a new diagnostic framing) but incremental as a technical contribution (cosine similarity is a standard metric; applying it to outer gradients is a natural step). Its significance lies in how it changes the way practitioners think about and debug distributed training, not in any mathematical novelty.

Innovation 3: The Outer Optimizer as a First-Class Design Choice, Not an Afterthought

The paper elevates the outer optimizer — the update rule applied to the averaged parameter displacements — from a minor implementation detail to a first-class design axis that can determine whether distributed training succeeds or fails. This is a departure from both the Federated Averaging tradition (where the outer step is simple averaging) and the local SGD literature (where the outer step is typically SGD with a fixed learning rate).

What the field assumed before this work. In standard FedAvg (McMahan et al., 2017) and most follow-ups, the outer update is parameter averaging: θ_new = (1/k) * Σ θ_i. This is equivalent to outer SGD with learning rate 1.0. The FedOpt framework (Reddi et al., 2021) generalized this to allow other outer optimizers, but the tested instantiations (SGD with momentum, Adam) were applied in low-H regimes where the outer gradient behaves more like a conventional stochastic gradient. Prior work did not systematically compare outer optimizers at very large H, nor did it identify Nesterov momentum as a particularly good fit for the large-H regime.

What DiLoCo adds. Figure 6 provides a clean empirical comparison of four outer optimizers — SGD, SGD with momentum, Adam, Nesterov momentum — on the same DiLoCo training task with H = 500. The results are stark: SGD is slow, Adam is unstable (requiring ε = 0.1 to avoid second-moment explosion), and Nesterov momentum decisively outperforms all others. The paper attributes this to Nesterov's "look-ahead" correction being particularly suited to outer gradients that span hundreds of inner steps — these are not local gradients in the conventional sense, but macro-displacements accumulated over extended optimization trajectories. The standard momentum update would combine the current macro-displacement with the previous velocity without accounting for the fact that the "gradient" describes movement FROM the starting point, not the gradient AT the starting point. Nesterov's correction implicitly adjusts for this.

This contribution is subtle but important: it establishes that the outer optimizer is not merely a learning rate schedule for averaging, but a mechanism whose dynamics must be matched to the structure of the outer gradient. The paper's outer learning rate of 0.7 (rather than the 1.0 of standard averaging) and momentum of 0.9 are not arbitrary hyperparameters — they are the result of a principled comparison that reveals which optimizer class can handle coarse-grained, infrequent updates.

Significance beyond this paper. This finding suggests a research direction: outer optimizer design for large-H distributed training. If Nesterov momentum outperforms Adam and SGD by a wide margin in this regime, there may be other outer optimizers — perhaps incorporating adaptive per-layer outer learning rates, or outer learning rate warmup, or outer gradient clipping — that perform even better. The paper opens this design space rather than closing it. It also implies that prior negative results on local SGD at scale (like Ortiz et al., 2021) might be partially explained by suboptimal outer optimizer choice rather than an inherent limitation of infrequent communication.

Evidence anchoring. Figure 6 shows the validation perplexity curves for all four outer optimizers. Nesterov achieves the lowest perplexity and converges fastest. The paper additionally notes that Nesterov with η_outer = 0.7 and μ = 0.9 was "very robust" — adopted for all remaining experiments — and that decaying the outer learning rate was tested but found unnecessary because the inner learning rate decay naturally shrinks outer gradient magnitudes.

Is this fundamental or incremental? This is an incremental empirical finding (comparing optimizers is a standard ablation) with fundamental implications for how the field designs distributed training algorithms. The finding itself — Nesterov > SGD > Adam in this regime — is specific and potentially task-dependent. But the implication — "the outer optimizer is a critical design choice, not an afterthought" — shifts how future work in local SGD and federated learning should approach algorithm design.

Innovation 4: Robustness to Dynamic Compute as a First-Class Property, Not an Afterthought

The paper demonstrates that DiLoCo's model quality depends on total compute expended, not on when that compute is available, establishing robustness to dynamic worker pools as an inherent property of the algorithm rather than a special failure-handling mode. This is a conceptual shift from how distributed training typically treats resource variability.

What the field assumed before this work. Standard synchronous distributed training treats worker count as fixed. If a worker fails, training stalls until the worker recovers or is replaced. If a worker joins mid-training, the parallelization strategy must be reconfigured (requiring a training restart or complex dynamic resharding). The implicit assumption is that the training system controls the compute resources, not vice versa. Even in the federated learning literature, where worker availability is acknowledged as a challenge (mobile devices going offline), the typical framing is mitigation — how to handle worker dropout without catastrophic degradation — rather than indifference — worker count can vary arbitrarily and model quality depends only on the integral of compute over time.

What DiLoCo adds. Figure 7 and the associated analysis in Section 3.1 ("Adaptive compute pool") show that when the number of workers is varied over training according to different schedules — constant (8 workers), doubling (4→8), halving (8→4), ramping up (1→8), ramping down (8→1) — the final perplexity is determined by the total amount of compute (total inner steps across all workers), not by the schedule. Specifically:

Doubling Compute and Halving Compute use the same total compute (different timing) and achieve "similar performance."
Ramping Up and Ramping Down also achieve similar performance to each other, despite using exactly opposite schedules.
The ranking of final perplexity across schedules corresponds to total compute, not to any particular allocation pattern.

The paper interprets this as a form of temporal flexibility: the algorithm does not care whether compute arrives early or late, as long as the cumulative sum of inner steps across workers matches the budget. This is fundamentally different from standard synchronous training, where a worker that is absent for the first half of training cannot simply "catch up" by contributing more later — the fixed worker grid must be present from start to finish.

Significance beyond this paper. This property has direct practical implications that the paper flags in its motivation: preemptible cloud instances that cycle on/off, university clusters with dynamic scheduling, and collaborative compute pools where participants join and leave. If model quality depends only on total compute, organizations can assemble training runs from heterogeneous, intermittently available resources without worrying about timing. A lab could start training on 4 GPUs, add 4 more when a colleague's job finishes, lose 2 to preemption overnight, and finish on the remaining 6 — and get the same model quality as if all 8 had been available continuously, provided the total inner-step count is preserved. The paper does not test every possible dynamic schedule, but the range of schedules in Figure 7 — including extreme cases like ramping from 1 to 8 or vice versa — suggests the property is robust.

Connection to the outer gradient aggregation. The mechanism underlying this robustness is straightforward given DiLoCo's design but worth making explicit: each outer step is an independent averaging of whatever outer gradients are available at that moment. If k is small, the averaged outer gradient is noisier (higher variance) but still unbiased if the workers that ARE present provide a reasonable sample of the data distribution. Over T outer steps, the noise averages out in the Nesterov momentum buffer, provided total inner steps across all workers over all outer steps is sufficient. The asynchronous communication experiment (Figure 8) provides complementary evidence: even when individual workers drop communication with 50% probability, final perplexity degrades by only 2.1%. The algorithm absorbs transient worker unavailability as naturally as it absorbs different worker counts.

Evidence anchoring. Figure 7 plots perplexity over training steps for six different compute schedules. The key comparison is between schedule pairs that use equal total compute but different timing (Doubling vs. Halving; Ramping Up vs. Ramping Down). The curves converge to similar final values, supporting the claim. Figure 8 provides the related evidence for communication drops, showing that even in the more challenging non-i.i.d. setting, 50% communication failure causes only minor degradation.

Is this fundamental or incremental? This is an empirical property of the algorithm that the paper demonstrates but does not theoretically guarantee. It is not a new mechanism (the averaging scheme is unchanged) but a new claim about the algorithm's behavior that contradicts the standard assumption that worker availability must be carefully managed. The finding is significant because it converts a set of practical headaches (preemption, scheduling, heterogeneous hardware) into non-issues from the algorithm's perspective — provided the total compute budget is met. This is a stronger claim than "handles worker dropout" (which implies graceful degradation); it is "indifferent to when compute arrives" (which implies no degradation at all, only dependence on the integral).

5. Experimental Analysis

Evaluation Methodology

Dataset. All experiments use the C4 dataset (Raffel et al., 2020), a large corpus derived from Common Crawl web text. The paper reports perplexity on the C4 validation set. The training set is partitioned into shards for distributed workers; the specific training/validation split follows the standard C4 release.
Base model(s). Three sizes of decoder-only transformer, adapted from the Chinchilla architecture (Hoffmann et al., 2022): 60M parameters (3 layers, hidden dim 896, 16 heads), 150M (12 layers, hidden dim 896, 16 heads), and 400M (12 layers, hidden dim 1536, 12 heads, K/V size 128). All use a vocabulary of 32,000 tokens with a SentencePiece tokenizer. The 150M model is the primary testbed; the 60M and 400M models are used for scaling ablations. The paper states these models are "adapted from the Chinchilla architecture" (Table 1), meaning they follow the architectural choices (e.g., relative position encodings, pre-norm, GeGLU activations) of Hoffmann et al. (2022) but at reduced scale.
Metrics. Perplexity on the C4 validation set, reported against the total number of training steps. Perplexity is the standard metric for language modeling: exp(cross-entropy loss). A second implicit metric is wall-clock training time, treated as proportional to the number of inner training steps, because communication is infrequent enough that communication time is negligible compared to compute time. The paper does not report absolute wall-clock times (no hardware specification beyond "machines hosting 16 A100 GPUs"), but treats steps as a proxy for time in the baseline comparisons (Table 2).
Baselines. The paper defines four baselines, all evaluated on the same 150M model and total step count of 88,000 (Section 3, Figure 2, Table 2):
1. Baseline (single worker): Standard training on a single worker with batch size 512 for 88,000 steps. No distribution. Communication: 0. Time: 1×. Perplexity: 16.23.
2. Baseline, 8× batch size with data parallelism: Starts from the 24,000-step pretrained model (like DiLoCo), then trains for 64,000 additional steps using 8 workers with per-step gradient synchronization and an effective batch size of 4,096 (8 × 512). This is standard distributed data-parallel training. Communication: 8 × N (once per step per worker). Time: 1× (8 workers running in parallel). Perplexity: 15.30.
3. Baseline, 8× batch size with microbatching: Starts from the 24,000-step pretrained model, trains for 64,000 additional steps on a single worker with an effective batch size of 4,096 achieved via gradient accumulation over 8 micro-batches of 512. No distribution — all work is sequential on one worker. Communication: 0. Time: 8× (sequential micro-batches). Perplexity: 15.30.
4. Baseline, 8× updates: Starts from the 24,000-step pretrained model, trains for 8 × 64,000 = 512,000 additional steps on a single worker with batch size 512. This is the "train longer" baseline that uses 8× the training steps (and thus 8× the compute) at the standard batch size. Perplexity: 14.72.
The DiLoCo configuration (8 workers, H = 500, T = 128, starting from the same 24,000-step pretrained model, batch size 512 per worker) achieves perplexity 15.02, placing it between baselines 2/3 (15.30, better) and baseline 4 (14.72, worse) while matching baseline 2/3's wall-clock time and using 500× less communication.
Generation budget / compute accounting. The paper measures compute in two ways that serve different purposes:

Training steps is the primary progress metric on all x-axes — total number of inner optimization steps executed per worker × number of workers. Since each inner step processes a batch of 512 sequences of 1,024 tokens, total FLOPs are proportional to total inner steps × model size × batch size × sequence length. The paper keeps model architecture, batch size, and sequence length fixed across all comparisons, so steps are a reliable proxy for total compute.

Communication frequency is measured by H (inner steps per outer synchronization). The total number of communication events is T = total_steps / H. The headline comparison (Table 2) quantifies communication cost as "number of gradient transmissions": the data-parallel baseline communicates 8 × N times (once per step per worker), while DiLoCo communicates 8 × N / H times, yielding a 500× reduction when H = 500.

Wall-clock time is treated as proportional to the number of sequential inner training steps that cannot be parallelized. DiLoCo runs 8 workers in parallel, each executing H steps per outer round, so the wall-clock time per outer round is H steps (not 8 × H). The baselines are compared on this basis in Table 2: DiLoCo takes 1× time (same as the data-parallel baseline), while the microbatching baseline takes 8× time and the 8× updates baseline takes 8× time.
Cross-validation / statistical protocol. The paper does not report cross-validation or multiple random seeds — results are single-run perplexity curves. Hyperparameter search is performed on the 150M model (the paper states "Hyper-parameters were tuned on the 150M model, which may be sub-optimal for the other model sizes" in the context of the model size scaling experiment in Table 4). The outer optimizer comparison (Figure 6) sweeps outer learning rates over {1.0, 0.7, 0.5, 0.3, 0.1} for each optimizer type, with outer momentum sweeps over {0.95, 0.9, 0.8} for Nesterov. The chosen values (η_outer = 0.7, μ = 0.9) are the best on the 150M model. No test-set contamination concern is flagged — the C4 validation set is standard.

For the non-i.i.d. data shard construction, the k-means clustering is performed once using a pretrained model's last-layer features. The paper does not discuss the sensitivity of results to different clusterings or different k-means seeds.

Main Quantitative Results

Headline Result: DiLoCo vs. Synchronous Baselines (Figure 2, Table 2)

The paper's central result is that DiLoCo with 8 workers achieves better perplexity than a fully synchronous baseline using the same number of workers and batch size, while communicating 500× less and training 8× faster than the sequential equivalent. The specific numbers (all at 88,000 total steps, 150M model, C4 validation perplexity):

"DiLoCo (blue) using 8 workers yields lower perplexity, even compared to the baseline using 8 times bigger batch size, while being 8 times faster in wall-clock time and communicating 500 times less."

The breakdown from Table 2:

Method	Comm. Cost	Time	Compute & Data	Perplexity
Baseline (single worker)	0	1×	1×	16.23
Baseline, 8× batch (data parallel)	`8 × N`	1×	8×	15.30
Baseline, 8× batch (microbatching)	0	8×	8×	15.30
Baseline, 8× updates	0	8×	8×	14.72
DiLoCo (8 workers)	`8 × N/500`	1×	8×	15.02

The comparison that the paper emphasizes is DiLoCo versus the data-parallel 8× batch baseline: both use 8 workers in parallel (same wall-clock time), both consume 8× the compute and data of the single-worker baseline, but DiLoCo communicates 500× less and achieves 15.02 perplexity versus 15.30 — a 0.28 absolute improvement in perplexity. The paper does not report a perplexity difference as dramatic as this number might imply for downstream task performance; 0.28 perplexity is a meaningful but not enormous gap at this scale.

What Figure 2 shows visually: The y-axis is validation perplexity (lower is better); the x-axis is total training steps. All DiLoCo baselines start from a 24,000-step pretrained model (indicated by the vertical dashed line at step 24,000). The DiLoCo curve (blue) drops below the 8× batch baselines (purple) early in the DiLoCo phase and maintains a gap through step 88,000. The "from scratch" baseline (red) is substantially worse throughout, as expected since it doesn't benefit from the pretraining phase. The 8× updates baseline (not shown in Figure 2 but reported in Table 2 as 14.72 perplexity) outperforms DiLoCo by 0.30 perplexity but requires 8× the sequential training time.

The key tradeoff is explicit in Table 2: DiLoCo sacrifices some absolute model quality (15.02 vs. 14.72 for the 8× updates baseline) to achieve massive wall-clock speedup and communication reduction. This is a Pareto improvement over the data-parallel baseline (better quality, same time, far less communication) but not a Pareto improvement over "just train longer" (better quality achievable with more time).

Robustness to Communication Frequency (Figure 4)

The paper varies H — the number of inner steps between outer synchronizations — across {50, 100, 250, 500, 1000, 2000} in the non-i.i.d. data regime with the 150M model. This is a stress test of the algorithm's tolerance to infrequent communication.

Headline findings:

More frequent communication always improves perplexity. At H = 50, final perplexity is approximately 14.7–14.8 (extrapolating from Figure 4's curves; the paper does not quote exact final values for each H). At H = 500 (the default), perplexity is 15.02.
Diminishing returns after H = 500. The paper states:

"communicating more frequently than H = 500 steps leads to diminishing returns. Moreover, the performance degradation is very mild up to H = 1000 steps. For instance, when H = 1000 the perplexity increases by only 2.9% relative to H = 50, despite communicating 20× less."

The 2.9% relative increase is computed as (PPL_H=1000 - PPL_H=50) / PPL_H=50. At the absolute level, this means going from ~14.8 to ~15.23 — a 0.43 absolute perplexity increase for a 20× communication reduction.
At H = 2000 (green curve), performance degrades further but the model still trains — perplexity appears to be roughly 15.8–16.0 at step 88,000, which is still competitive with the single-worker baseline (16.23) despite communicating only 44 times total over 88,000 steps.

Interpretation: The communication frequency is a smooth efficiency-quality tradeoff, not a binary "works/fails" threshold. If bandwidth is extremely constrained, H = 1000 or even H = 2000 is viable. If some bandwidth is available, H = 500 captures most of the benefit of frequent communication. The paper chooses H = 500 as "a good trade-off between generalization performance and communication cost."

Robustness to Data Distribution: I.I.D. vs. Non-I.I.D. (Figure 5)

The paper compares DiLoCo with k = 8 workers when data shards are randomly partitioned (i.i.d.) versus clustered by semantic content using k-means on sentence embeddings (non-i.i.d.).

Headline findings:

I.i.d. training converges faster early in training — the red curve in Figure 5 is lower than the blue curve for roughly the first half of the DiLoCo phase (steps 24,000–56,000).
By the end of training (step 88,000), the two curves converge to equivalent perplexity. The paper states:

"Despite the latter converging faster early on in training, the final generalization performance of the two settings is comparable."

The final perplexity difference is not explicitly quoted but appears negligible from Figure 5 (both curves terminate at approximately 15.02).

This is a stronger result than it might appear. In standard federated learning, non-i.i.d. data distributions are known to cause model divergence or severe degradation (the paper cites Gao et al., 2022 on this point). DiLoCo's near-indifference to the data distribution — achieving the same final quality with semantically clustered shards as with random shards — is unexpected. The paper's appendix analysis (Figures 10–11) provides a mechanistic explanation: even in the non-i.i.d. setting, workers' outer gradients remain positively correlated (cosine similarity > 0), and longer inner training (H = 500) actually increases this similarity by averaging out stochastic noise.

Caveat: The paper only tests one degree of non-i.i.d.-ness (k-means clustering on sentence embeddings). More extreme heterogeneity — such as shards with entirely different languages, or shards constructed adversarially to minimize outer gradient similarity — might produce different results. The paper does not explore this.

Scaling the Number of Workers: Table 3

The paper varies the number of replicas k from 1 to 64, keeping total inner steps per worker constant (so total compute scales linearly with k). Results are reported for both i.i.d. and non-i.i.d. settings:

`k`	i.i.d. PPL	non-i.i.d. PPL
1 (baseline)	16.23	—
4	15.23	15.18
8	15.08	15.02
16	15.02	14.91
64	14.95	14.96

Headline findings:

Monotonic improvement in perplexity as k increases from 1 to 8, with diminishing returns beyond 8 workers. Going from k = 1 to k = 8 reduces perplexity by ~1.15 (in i.i.d.) or more. Going from k = 8 to k = 64 reduces perplexity by an additional 0.06–0.13 — a much smaller gain.
No performance degradation at any k in either data regime. This directly contradicts Ortiz et al. (2021), who reported that local SGD degraded at k ≥ 16. The paper emphasizes:

"Unlike what is reported in prior work in the vision domain on ImageNet (Ortiz et al., 2021), we do not observe significant performance degradation by increasing the number of replicas."

The non-i.i.d. setting is slightly better than i.i.d. at k = 8 (15.02 vs. 15.08) and k = 16 (14.91 vs. 15.02), though the differences are small and may not be statistically significant. The paper does not comment on this reversal.

Interpretation: Adding more workers provides more total compute and more data diversity, both of which improve the model — but the marginal benefit of additional workers shrinks because each worker sees a smaller, potentially less representative shard, and the outer gradient averaging becomes noisier as similarity between workers' outer gradients decreases (Figure 11).

Scaling Model Size: Table 4

The paper tests DiLoCo with 8 workers and non-i.i.d. shards on three model sizes: 60M, 150M, and 400M parameters. For each size, the improvement over the corresponding single-worker baseline is reported:

Model Size	Relative Improvement	Absolute PPL Improvement
60M	4.33%	1.01
150M	7.45%	1.21
400M	7.49%	1.01

Headline findings:

Larger models show larger relative improvements from DiLoCo (7.45–7.49% for 150M and 400M vs. 4.33% for 60M), though the absolute improvement is similar (1.01–1.21 PPL across all sizes).
The improvement from 150M to 400M is marginal in relative terms (7.45% → 7.49%), suggesting diminishing returns to scale at this model size range — but this may be confounded by suboptimal hyperparameters (the paper notes that "Hyper-parameters were tuned on the 150M model, which may be sub-optimal for the other model sizes").

The paper's interpretation is speculative but plausible:

"We surmise that (1) in an overtrained setting with large amount of steps, larger models are more efficient at fitting the same amount of data, and (2) as the linear connectivity literature (Ilharco et al., 2022) suggests, larger models are less subject to interference when averaging their parameters."

Point (2) is the more interesting claim: If larger models have "wider" loss basins where different minima are more linearly connected, then outer gradient averaging should be more effective at larger scales. This would predict that DiLoCo's advantage over synchronous training increases with model size — a hypothesis the paper flags for future work but does not test beyond 400M parameters.

Robustness to Pretraining: Figure 3

The paper varies the number of pretraining steps (single-worker training before DiLoCo distribution begins) across {0, 12,000, 24,000, 48,000} in the non-i.i.d. setting with k = 8, keeping total steps fixed at 88,000.

Headline findings:

DiLoCo works even from random initialization. The "no pretraining" curve (teal) achieves final perplexity approximately 15.1, only ~0.1 worse than the 24,000-step pretrained baseline (~15.02). This is the finding the paper emphasizes:

"Interestingly, performance is not degraded even when starting from a randomly initialized network. This result contradicts the findings of prior work on post local-SGD (Lin et al., 2020) and its large-scale study on a vision classification task (Ortiz et al., 2021)."

The curves for 12k, 24k, and 48k pretraining steps are virtually indistinguishable at step 88,000 (all converging to ~15.0–15.1). The vertical dashed lines show a perplexity spike at the start of each DiLoCo phase, caused by inner learning rate warmup.
The 48k pretraining curve (orange) actually shows a small degradation when DiLoCo begins (a sharper spike and slower recovery), possibly because the model has already converged substantially and the distribution of training is disruptive. The paper does not discuss this.

Interpretation: Pretraining is helpful but not necessary. A modest pretraining period (12k–24k steps) provides a small final quality benefit (~0.1 PPL) and avoids the early-training instability that occurs when DiLoCo starts from scratch (the teal curve is noisier in the first 20,000 DiLoCo steps). For practical deployment, pretraining on a single cluster before distributing is recommended because it avoids the communication overhead during the high-learning-rate phase when outer gradients are largest, but the system degrades gracefully without it.

Outer Optimizer Comparison: Figure 6

The paper compares four outer optimizers — SGD, SGD with momentum, Adam, Nesterov momentum — each tuned over a range of learning rates and momentum values (Table 5). The experiment uses 8 workers, non-i.i.d. shards, H = 500, 24,000 pretraining steps.

Headline findings:

Nesterov momentum decisively outperforms all alternatives. The Nesterov curve (with η_outer = 0.7, μ = 0.9) achieves the lowest perplexity and the fastest convergence.
SGD (FedAvg) performs poorly. The paper states it "performed poorly" — the learning curves in Figure 6 show SGD converging substantially slower and to a higher final perplexity.
Adam is unstable. The paper reports:

"Adam was particularly unstable with a high second order momentum norm. We alleviated the issue by increasing the ε factor to 0.1."

Even with this mitigation, Adam's final perplexity is the worst among the four tested. The instability arises because outer gradients spanning H = 500 inner steps have large norms, causing Adam's second-moment accumulator to grow rapidly and the effective learning rate to become very small, stalling training.

SGD with momentum occupies an intermediate position — better than plain SGD but worse than Nesterov. The gap between momentum and Nesterov quantifies the value of the Nesterov correction specifically.

Interpretation: The outer optimizer choice is not a minor hyperparameter — it qualitatively determines whether DiLoCo converges well. Adam, which is the standard optimizer for single-worker training, is actively harmful in the outer loop. This finding is specific to the large-H regime and suggests that outer optimizer design for distributed training with infrequent communication is a distinct problem from standard optimizer design.

Single-Worker Acceleration: Figure 9

DiLoCo is applied with k = 1 (one worker, no distribution), comparing it against standard AdamW training (identical total steps, all other hyperparameters equal). This is essentially the Lookahead optimizer (Zhang et al., 2019) with Nesterov momentum.

Headline findings:

DiLoCo (k = 1, H = 500) converges faster than standard AdamW: the perplexity curve is lower at every step.
DiLoCo (k = 1) achieves a lower final perplexity than standard AdamW. The exact values are not quoted but are visible in Figure 9 — the gap at step 88,000 appears to be approximately 0.3–0.5 perplexity.
This occurs at zero communication cost (no distribution, no data transfer) — the outer optimization step is purely a local computation that interpolates between the starting and ending parameters of each phase.

Interpretation: The outer optimization step has intrinsic value beyond enabling distribution. It acts as a regularizer or an implicit learning rate schedule: by periodically "resetting" the parameters to a Nesterov-smoothed version of the trajectory, DiLoCo may avoid overfitting to the current data window or escape shallow local minima. The paper does not explore this mechanism further (it is a side result), but it is consistent with the linear mode connectivity literature's finding that weight averaging improves generalization.

Adaptive Compute Pool: Figure 7

The paper simulates dynamic worker availability by varying the number of workers k over the course of training according to different schedules, all in the i.i.d. setting:

Constant Local: k = 1 throughout (baseline).
Constant Distributed: k = 8 throughout (standard DiLoCo).
Doubling Compute: k = 4 for first half, k = 8 for second half.
Halving Compute: k = 8 for first half, k = 4 for second half.
Ramping Up: k = 1 → k = 8 linearly over training.
Ramping Down: k = 8 → k = 1 linearly over training.

Headline findings:

The final perplexity is determined by total compute, not by the schedule. Specifically:
- Doubling Compute and Halving Compute use the same total compute (8 × half + 4 × half) and achieve "similar performance."
- Ramping Up and Ramping Down also achieve similar performance to each other, despite opposite schedules.
- Constant Distributed (8 workers throughout, maximum total compute) performs best. Constant Local (1 worker throughout, minimum total compute) performs worst.
- The intermediate schedules (Doubling, Halving, Ramping Up, Ramping Down) lie between these extremes in proportion to their total compute.
The paper concludes:

"Models quality is affected by the total amount of compute, but not as much by how such compute is allocated over time."

Interpretation: DiLoCo is indifferent to temporal allocation of compute resources. Workers can join, leave, or change count arbitrarily, and the model quality depends only on the cumulative sum of inner steps across all workers. This is not a theoretical guarantee but an empirical observation across the tested schedules. The paper does not test schedules where the worker count changes within a single outer round (e.g., a worker drops mid-way through H inner steps), which would require a different handling mechanism than the current synchronous outer-loop design.

Robustness to Dropped Communication: Figure 8

The paper simulates communication failures by dropping each worker's outer gradient with probability p ∈ {0%, 10%, 30%, 50%}. A dropped worker continues training from its own local parameters for the next H steps (i.e., it does not receive the shared model update), and its outer gradient for that round is excluded from the average. This is tested in both i.i.d. (Figure 8a) and non-i.i.d. (Figure 8b) settings.

Headline findings:

Higher drop rates cause more training instability, visible as transient perplexity spikes in Figure 8 (the orange and red curves are noisier than teal).
Even at p = 50% (each worker misses communication half the time on average) in the non-i.i.d. setting, the final perplexity degradation is only 2.1% relative to perfect communication:

"Even in the extreme non-i.i.d. setting where each replica has 50% probability of dropping communication, the degradation of perplexity relative to perfect communication is only 2.1%."

The degradation is slightly worse in the non-i.i.d. setting than in the i.i.d. setting (the paper does not quote exact numbers for i.i.d. at p = 50%, but the curves suggest slightly less degradation).

Interpretation: DiLoCo does not require a synchronization barrier where all workers must communicate. Workers that fail to communicate simply continue training locally and rejoin when possible. The outer optimization proceeds with whatever gradients are available. This is important for practical deployments because it eliminates the need for all workers to wait for stragglers — training can proceed at the pace of the fastest workers, with slower or temporarily disconnected workers catching up on subsequent rounds. The paper notes this implication explicitly:

"Consequently, with robustness to communication failure, the need of a synchronization barrier is less critical and thus training can be accelerated without having to wait all replicas."

Ablation Studies and Robustness Checks

Inner optimizer state synchronization: The paper considered synchronizing AdamW's first and second moment buffers (m and v) in addition to model parameters during the outer step. The finding: "It did not lead to significant improvements while significantly increasing the communication cost (×3 more data to transmit)." This justifies the design choice of communicating parameters only, which keeps communication proportional to the model size d rather than 3d. The paper cites prior work (Ortiz et al., 2021; Wang et al., 2020) as having similar findings in SGD-with-momentum contexts.
Outer gradient compression via sign pruning (Appendix, Table 6): Outer gradients are pruned by setting the smallest-magnitude values (within each neuron's parameter group) to zero, following the sign-based pruning heuristic of Yadav et al. (2023). Results:

Pruning % Perplexity Relative Change
0% 15.02 0%
25% 15.01 −0.06%
50% 15.08 +0.39%
75% 15.27 +1.66%

50% pruning is essentially free — the 0.39% relative perplexity increase is negligible — meaning communication volume can be halved with a trivial compression method. 75% pruning begins to hurt noticeably but is still far from catastrophic. The paper notes that more sophisticated compression methods (e.g., structured sparsity, quantization) could further reduce communication.
Weighted vs. uniform outer gradient averaging (Appendix, subsection 6.1): For non-i.i.d. shards of unequal size (due to k-means clustering), each outer gradient is weighted by the number of examples in its shard: Δ = Σ (|D_i| / Σ|D_j|) · Δ_i. The paper notes this matters particularly at large k: "While at k = 4, all clusters are quite balanced, imbalance can be striking at k = 64 and giving more importance to larger clusters is beneficial." The uniform average is used for i.i.d. data.
Sign-based merging (disjoint merge) vs. uniform averaging (Appendix, subsection 6.1): The paper tested the disjoint merge heuristic from Yadav et al. (2023), which uses a sign-based selection rule for which parameters to average. Result: "we got slightly worse results" than uniform averaging. This is a minor negative result but reinforces that simple uniform averaging is a strong baseline in this setting.
Outer learning rate decay: The paper tested cosine scheduling of the outer learning rate and found "similar performance" to the constant learning rate. The reason: "Since we decay the inner learning rate, the outer gradient norm gets naturally smaller over the course of training, removing the need to further decay the outer learning rate." This is a simplifying design choice — one fewer hyperparameter to tune.
Inner learning rate warmup at DiLoCo start: The paper explicitly flags the perplexity spikes visible at the start of each DiLoCo phase in Figure 3 as being caused by inner learning rate warmup (resetting the learning rate to zero and linearly ramping to 4 × 10^{-4} over 1,000 steps). Despite the transient spike, "such warm up is ultimately beneficial, as previously noted also in the continual pretraining setting by Gupta et al. (2023)." This is a negative result in the sense that the spike looks alarming but is benign — a practical note for anyone replicating DiLoCo.

Pruning %	Perplexity	Relative Change
0%	15.02	0%
25%	15.01	−0.06%
50%	15.08	+0.39%
75%	15.27	+1.66%

Critical Assessment

Do the Experiments Support the Central Claims?

The paper's executive summary makes three core claims:

Claim 1: "DiLoCo on 8 workers performs as well as fully synchronous optimization while communicating 500 times less."

Supported, with a qualification on what "as well" means. The data in Table 2 show that DiLoCo (15.02 PPL) outperforms the data-parallel 8× batch baseline (15.30 PPL) — which is the most natural comparison since both use 8 workers for the same wall-clock time. This is actually better than "as well as" — DiLoCo is strictly better on both perplexity and communication cost. However, the 8× updates baseline (which trains for 8× longer sequentially) achieves 14.72 PPL, beating DiLoCo by 0.30 PPL. So DiLoCo does not match the best possible synchronous result — it matches the time-equivalent synchronous result and exceeds it. This is the correct comparison for the paper's stated goal of enabling distributed training without infrastructure changes, but readers should not interpret "as well as" to mean "as well as any synchronous configuration regardless of time."

What strengthens this claim: The 500× communication reduction is not exaggerated — H = 500 inner steps means 1 communication event per 500 training steps, versus 1 per step for data parallelism. The comparison is exact, not an order-of-magnitude approximation.

What weakens it: The comparison is against a data-parallel baseline with the SAME batch size (512 per worker), not with a batch size that might be optimal for the data-parallel configuration. The paper does not explore whether the data-parallel baseline could achieve better perplexity with different per-worker batch sizes. Additionally, the baseline is on the same 150M model — there is no evidence that DiLoCo's advantage holds at larger model sizes where communication cost is proportionally higher.

Claim 2: "DiLoCo exhibits great robustness to the data distribution of each worker."

Strongly supported. Figure 5 shows that i.i.d. and non-i.i.d. data distributions converge to the same final perplexity, which is a genuinely surprising result given the federated learning literature's emphasis on non-i.i.d. data as a major challenge. The appendix analysis (Figure 10) provides a mechanistic explanation (positive outer gradient similarity even in non-i.i.d. case) that makes the result credible rather than coincidental.

Caveat: "Great robustness" is tested only along one axis of non-i.i.d.-ness — semantic clustering via k-means on sentence embeddings. This is a realistic form of heterogeneity (different workers get different types of text), but it does not cover other important forms: class imbalance (some workers have mostly high-quality text, others mostly low-quality), temporal shift (workers get data from different time periods), or adversarial partitioning (malicious workers with corrupted data). These are standard concerns in the federated learning literature and are not addressed.

Claim 3: "It is also robust to resources becoming unavailable over time, and vice versa, it can seamlessly leverage resources that become available during training."

Supported for the tested schedules. Figure 7 demonstrates that total compute, not timing, determines final model quality across six qualitatively different schedules (constant, doubling, halving, ramping up, ramping down). Figure 8 demonstrates that communication failures up to 50% cause only 2.1% degradation.

What is genuinely demonstrated vs. what is claimed:

Demonstrated: The model is insensitive to when each unit of compute is contributed, as long as total compute is held constant. This is shown for schedules where worker count changes at outer-step boundaries.
Claimed but not directly tested: That the system can "seamlessly leverage resources that become available during training" in the sense of a fully dynamic, asynchronous system where workers join and leave at arbitrary times (not just at outer-step boundaries). The current experiments only change worker count at the start of outer steps. If a worker joins mid-way through H = 500 inner steps, the paper provides no mechanism for integrating it until the next outer round. This is a practical limitation but likely acceptable since outer rounds are minutes apart, not hours.

What's missing: The paper does not test a schedule where the worker count changes within an outer round (e.g., a worker starts late and only completes 300 of 500 inner steps before the outer synchronization). Handling this case would require partial outer gradients or a timeout mechanism, neither of which is discussed.

Genuine Weaknesses in Experimental Design

1. Single dataset, single task (language modeling). All experiments are on C4, a web-text corpus. The paper acknowledges this in Section 5: "First, we only considered a single task, namely language modeling, and a single architecture, a transformer. Other datasets, domains (e.g. vision), and other architectures (e.g., CNNs which are known to be more sensitive to linear mode connectivity (Jordan et al., 2023)) should also be considered." This is a substantial limitation because the paper's positive results may be specific to language modeling or to transformers, and the negative results from Ortiz et al. (2021) were on ImageNet with ResNets — exactly the domain the paper acknowledges may not transfer.

2. Small model scale. State-of-the-art language models at the time of writing have 3–4 orders of magnitude more parameters than the 60M–400M range tested here. The paper acknowledges this (Section 5): "at the time of writing state-of-the-art language models use 3 orders of magnitude more parameters." The hypothesis that DiLoCo improves with scale ("larger models are less subject to interference when averaging their parameters") is plausible but untested. At billion-parameter scales, communication cost becomes a much larger fraction of total training time, which could make DiLoCo's advantages more dramatic — or could reveal new failure modes (e.g., outer gradients from billion-parameter models might be too large in norm for the Nesterov outer optimizer to handle stably).

3. No downstream task evaluation. The paper reports only perplexity on the C4 validation set. Perplexity is a standard language modeling metric but does not always correlate with downstream task performance (e.g., on question answering, summarization, or reasoning benchmarks). The paper does not evaluate DiLoCo-trained models on any downstream tasks, so it is unknown whether the perplexity improvements translate to practical task performance.

4. Single-run results, no error bars. All perplexity curves are single training runs with no reported variance across random seeds. This is standard for large-scale training experiments (running multiple seeds would multiply compute cost), but it means the reported differences — particularly the small gaps like DiLoCo (15.02) vs. 8× batch baseline (15.30) — may not be statistically significant. The paper does not discuss seed sensitivity.

5. Hyperparameters tuned on the evaluation model. Outer optimizer hyperparameters were tuned on the 150M model and then applied to 60M and 400M models. The paper acknowledges this suboptimality for the model size scaling experiment (Table 4) but does not adjust or report sensitivity. The strong Nesterov momentum values (η = 0.7, μ = 0.9) may not transfer to other model sizes, architectures, or datasets.

6. No comparison to gradient compression methods. The paper compares DiLoCo to fully synchronous and fully local baselines, but not to a third class of approaches: data-parallel training with gradient compression (quantization, sparsification, low-rank approximations). These methods also reduce communication while maintaining per-step synchronization, and they are widely used in practice. The paper's sign-pruning ablation (Table 6) shows that simple compression works well on top of DiLoCo, but does not compare DiLoCo + compression against data-parallel + compression at the same total communication budget.

7. Pretraining cost is not fully accounted for. All main DiLoCo experiments start from a 24,000-step pretrained model. In the compute accounting of Table 2, the pretraining cost is shared across all methods (except the from-scratch baseline), so it does not bias the comparison. However, the pretraining phase is done on a single cluster with full synchronization — it is not distributed across the poorly-connected workers. For a deployment where ALL training must be distributed (e.g., no single cluster is large enough even for the initial 24,000 steps), the paper's no-pretraining result (Figure 3) shows DiLoCo works from scratch, but with a small perplexity penalty (~0.1 PPL). The paper does not discuss whether pretraining could itself be distributed via DiLoCo (i.e., using DiLoCo from step 0), which would be necessary in a fully distributed setting.

Missing Experiments That Would Strengthen the Paper

Scaling to billion-parameter models. The paper's core hypothesis — that DiLoCo improves at larger scales because larger models have more linear connectivity — is untested. An experiment at 1B or 7B parameters would directly validate or refute this.
Combination with model parallelism. The paper treats each worker as a single device group that can independently use data and model parallelism. But it does not test this combination. At billion-parameter scales, each worker WOULD need model parallelism internally, and the interaction between DiLoCo's outer optimization and internal model-parallelism strategies is unexplored.
Downstream evaluation. Perplexity on C4 is a narrow metric. Evaluating DiLoCo-trained models on standard NLP benchmarks (GLUE, SuperGLUE, or generation tasks) would demonstrate whether the perplexity improvements are meaningful.
Vision domain. Given that the paper's primary counterpoint (Ortiz et al., 2021) is on ImageNet with ResNets, a direct replication of DiLoCo on that exact benchmark would be highly informative. Does the AdamW + Nesterov combination help in vision, or is DiLoCo's success specific to transformers?
Heterogeneous worker speeds. The paper acknowledges that all workers are assumed homogeneous: "the version of DiLoCo presented here assumes that all workers are homogeneous. However, in practice workers might operate at wildly different speeds." An experiment where workers have different throughputs (some completing H steps faster than others) would test the robustness claim in a more realistic setting. The current synchronous outer loop would force fast workers to idle, which could erase the wall-clock time advantage.
Asynchronous outer updates. The paper's dropped communication experiment (Figure 8) is a step toward asynchrony, but a fully asynchronous variant — where workers update the shared model independently without waiting for an outer synchronization barrier — is not tested. This is flagged as future work in Section 5 and would be more realistic for heterogeneous worker pools.
Varying H during training. The paper uses constant H = 500. Dynamic H — starting small when gradients are large and volatile, increasing as training stabilizes — could improve the communication-quality tradeoff further. The cosine similarity analysis (Figure 10) suggests outer gradients become more similar over training, supporting the idea that later outer steps could use larger H.

Conditional Nature of the Claims

The paper's claims hold empirically for the tested configuration but several important conditions apply:

The claims are specific to transformer language models at 60M–400M scale on C4. Generalization to other architectures (CNNs), domains (vision), scales (billions of parameters), and datasets is hypothesized but not demonstrated.
The 500× communication reduction is relative to a specific data-parallel baseline with equal per-worker batch size. If the baseline could achieve the same quality with a larger per-worker batch size and gradient accumulation (reducing its own communication frequency), the gap would narrow.
The robustness to worker count variation (Figure 7) is demonstrated for changes at outer-step boundaries only. Mid-round worker changes are not tested.
The robustness to non-i.i.d. data (Figure 5) is demonstrated for semantic clustering heterogeneity only. More extreme forms of heterogeneity (adversarial, temporal, quality-based) are not tested.
The adaptation of DiLoCo to a single worker retains overhead. While Figure 9 shows faster convergence and better final perplexity, the k = 1 variant still performs an outer optimization step every H inner steps, which adds a small computational overhead relative to standard training. The paper does not quantify this overhead.

Overall, the experimental section is thorough for the scale and domain studied, but the leap from "works on 150M-parameter transformers on C4" to "enables distributed LLM training across poorly-connected devices at scale" requires extrapolation that the paper acknowledges is untested. The ablation studies are comprehensive along the axes the paper cares about (communication frequency, data distribution, worker count, outer optimizer choice, pretraining, dynamic compute), and they consistently support the paper's robustness claims within the tested regime. The main gap is the absence of larger-scale validation that would confirm the paper's speculation that DiLoCo's advantages increase with model size.

6. Limitations and Trade-offs

6.1 The Empirical Validation Is Confined to a Single Task, Modality, and Architecture Family at Modest Scale

The assumption or constraint. Every experiment in the paper — all robustness ablations, all scaling studies, all optimizer comparisons — is conducted on the C4 language modeling dataset with decoder-only Chinchilla-style transformers, at model sizes of 60M, 150M, and 400M parameters. The paper explicitly acknowledges this scope in Section 5:

"First, we only considered a single task, namely language modeling, and a single architecture, a transformer. Other datasets, domains (e.g. vision), and other architectures (e.g., CNNs which are known to be more sensitive to linear mode connectivity (Jordan et al., 2023)) should also be considered."

The paper further notes the scale limitation:

"at the time of writing state-of-the-art language models use 3 orders of magnitude more parameters. Therefore, it would be interesting to see how DiLoCo works at larger scale."

The consequence. This is not a minor "future work" item — it is a structural uncertainty that affects the paper's central claim of contradiction with prior work. Ortiz et al. (2021), the paper's primary counterpoint, concluded that local SGD fails at scale on image classification with ResNets, communicating every H = 8 steps. DiLoCo's success at H = 500 on language modeling with transformers does not directly refute that finding — it demonstrates success in a different setting. The mechanisms that make DiLoCo work may be specific to:

Transformers vs. CNNs. The linear connectivity literature has shown that CNNs are more sensitive to weight averaging, exhibiting higher loss barriers between independently trained replicas than transformers of comparable parameter count. DiLoCo's outer gradient averaging depends on workers remaining within the same loss basin; if CNNs diverge into disconnected basins more readily, the approach could fail on vision tasks.
Language modeling vs. other objectives. Language modeling has a dense, token-level supervision signal (every token contributes to the loss). This may produce outer gradients that are more stable and aligned across workers than tasks with sparse or episodic rewards (e.g., reinforcement learning, structured prediction).
150M–400M vs. billion-parameter scales. The paper hypothesizes that "larger models are less subject to interference when averaging their parameters" (Table 4), which would predict that DiLoCo improves with scale. But the opposite hypothesis — that outer gradients from billion-parameter models have enormous norms that destabilize the Nesterov outer optimizer, or that communication costs in absolute terms grow to dominate even with 500× reduction — is equally plausible and entirely untested. The 400M experiment in Table 4 shows diminishing returns from 150M to 400M (7.45% to 7.49% relative improvement), but this is at a scale too small to extrapolate to the 70B–1T parameter regime.

What evidence exists in the paper. The paper provides NO experiments outside C4 language modeling, NO experiments with non-transformer architectures, and NO experiments above 400M parameters. The claims of robustness to data distribution, communication frequency, and worker count are ALL conditioned on this specific setting. Section 5 explicitly lists these as limitations.

Mitigation status. The authors acknowledge the limitation transparently and frame it as future work rather than a resolved issue. They speculate about scale ("our initial extrapolation indicate that DiLoCo might perform even better at larger scales") and architecture generality ("CNNs which are known to be more sensitive to linear mode connectivity... should also be considered") but provide no evidence. A practitioner considering deploying DiLoCo for, say, vision model training or for frontier-scale (70B+) language model training would be operating outside the validated regime entirely.

6.2 The Comparison Against Synchronous Baselines Does Not Include Practical Alternatives That Also Reduce Communication

The assumption or constraint. The paper's headline comparison (Table 2, Figure 2) pits DiLoCo against three baselines: a single-worker baseline, a fully synchronous data-parallel baseline communicating at every step, and a sequential micro-batching baseline. From this comparison, the paper claims that DiLoCo achieves better perplexity than synchronous training "while communicating 500 times less." But this framing implicitly assumes that the synchronous baseline MUST communicate at every step — an assumption that is not true in practice.

The consequence. An entire class of practical alternatives is excluded from the comparison: synchronous data-parallel training with gradient compression (quantization, sparsification, low-rank approximation) or gradient accumulation with periodic synchronization. These methods reduce communication WITHOUT changing the optimizer structure, simply by compressing the gradients before transmission or by synchronizing every few steps rather than every step. The paper's sign-pruning ablation (Table 6) demonstrates that even trivial compression (50% pruning) works well with DiLoCo, but it NEVER compares DiLoCo against a synchronous baseline with the SAME total communication budget. The fair question is: if we have a fixed communication budget of, say, 8 × N/100 transmissions total, should we use DiLoCo with H = 100 infrequent but uncompressed communication, or standard data-parallel training with 100× gradient compression communicating every step? The paper provides no evidence to answer this.

This matters because gradient compression is a mature technology widely deployed in practice. The 500× communication reduction figure is impressive only if the alternative genuinely requires communicating 500× more — but if standard data-parallel training with 8-bit quantization reduces communication by 32×, and gradient sparsification reduces it by another 10×, and local gradient accumulation over 4 steps reduces it by another 4×, the combined reduction is 1280× — more than DiLoCo's 500× — using standard tools without changing the training algorithm. The paper's contribution is an algorithmic approach to reducing communication, but it does not establish that this approach is better than the systems-level approaches that are the standard alternative.

What evidence exists in the paper. None. The paper does not mention gradient compression, quantization, or gradient accumulation as baselines. The related work section (Section 4) cites Tang et al. (2023) as a survey of communication-efficient distributed deep learning, but only in the context of potential future enhancements to DiLoCo's outer gradient transmission, not as a competing paradigm. The sign-pruning ablation (Table 6) shows compression can be applied ON TOP of DiLoCo, but does not compare DiLoCo to compressed synchronous training.

Mitigation status. Not addressed. The paper positions DiLoCo exclusively against uncompressed, per-step-synchronized baselines. A more rigorous evaluation would include a "synchronous training with communication budget matched to DiLoCo" baseline — for instance, data-parallel training with H-step gradient accumulation and/or gradient compression, communicating the same total volume as DiLoCo. Without this, readers cannot assess whether DiLoCo's algorithmic complexity (two-level optimization, Nesterov outer optimizer, inner optimizer state management) is justified relative to simpler systems-level solutions.

6.3 Wall-Clock Time Is Reported as Proportional to Inner Steps, But This Ignores Outer Optimization Overhead and Straggler Effects

The assumption or constraint. The paper treats wall-clock training time as directly proportional to the number of inner optimization steps, arguing that "communication across workers is rather infrequent" and therefore negligible. Table 2 reports DiLoCo as taking "1×" time, equal to the data-parallel baseline, because both execute the same number of inner steps across 8 workers in parallel. The paper states:

"Therefore, if each communication step takes a lot of time, DiLoCo converges much faster in terms of wall-clock time."

This framing implicitly assumes that the outer optimization step — collecting outer gradients from all workers, averaging them, running the Nesterov update, and redistributing parameters — is either instantaneous or equally fast as the communication in the synchronous baseline. It also assumes all workers complete their H inner steps at exactly the same rate.

The consequence. In a genuinely distributed setting where workers are "islands of devices that are poorly connected" (Section 1), the outer communication step is NOT negligible — it is precisely the bottleneck that DiLoCo is designed to mitigate. Even at H = 500, transmitting a full set of model parameters (for a 150M model, ~600MB in float32) over a low-bandwidth link between, say, two cloud regions or two university campuses, could take minutes to hours depending on available bandwidth. The synchronous baseline would spend this time at EVERY step; DiLoCo spends it every 500 steps. But the claim that DiLoCo is "8× faster in wall-clock time" (Figure 2 caption) assumes that 500 × fewer communication events translates to negligible communication time — an assumption that depends on the specific bandwidth and latency of the inter-worker links, which are never specified.

Furthermore, the synchronous outer loop imposes a straggler problem: all workers must complete their H inner steps before the outer gradient averaging can proceed. If one worker is slower (due to older hardware, shared cluster load, or network congestion), all other workers idle. The paper acknowledges this in Section 5 with respect to heterogeneous hardware:

"In practice workers might operate at wildly different speed. In these cases, waiting for all workers to perform the same number of steps is rather inefficient."

But it does not quantify this overhead. If workers complete H = 500 inner steps in times that vary by ±20%, then every outer round incurs a 20% idle time penalty, and the "1× time" claim in Table 2 becomes inaccurate. The asynchronous communication experiment (Figure 8) partially addresses this — workers that drop communication continue training without waiting — but this only handles the case where a worker NEVER communicates in a given round, not the case where a worker is simply slower.

What evidence exists in the paper. The paper provides no absolute wall-clock measurements, no specification of inter-worker network bandwidth or latency, and no straggler simulation where workers complete inner steps at different rates. The "Adaptive compute pool" experiment (Figure 7) varies worker count over training but always assumes all active workers complete their steps at the same rate within each outer round. The infrastructure section (Appendix subsection 6.1) states:

"The empirical validation of this work was performed on machines hosting 16 A100 GPUs. These machines were not necessarily co-located in the same geographic region. The outer optimization step is performed on a CPU server connected to the local machines."

This provides no bandwidth or latency numbers, and the qualification "not necessarily co-located" is insufficient to determine whether the communication time was significant in the reported experiments.

Mitigation status. The paper acknowledges the heterogeneity limitation in Section 5 but does not measure it or propose a solution within the current framework. The suggestion is for future work: "Another avenue of future work is then to extend DiLoCo to the asynchronous setting, whereby workers update the global parameter without ever waiting for any other worker." A practitioner evaluating DiLoCo for deployment across genuinely disparate hardware (e.g., a mix of A100 and V100 clusters) would need to measure and account for straggler overhead that the paper does not quantify.

6.4 The Difficulty Estimation Equivalent — Data Shard Similarity — Is Not Characterized as a Precondition for Convergence

The assumption or constraint. DiLoCo's robustness to non-i.i.d. data distributions is demonstrated for one specific kind of heterogeneity: shards created by k-means clustering on last-layer sentence embeddings of a pretrained model (Appendix subsection 3.1). Each shard contains semantically similar documents, but all shards are drawn from the same C4 corpus — they share vocabulary, writing style, and data quality, differing primarily in topic. The paper shows (Figure 5) that final perplexity is equivalent between this non-i.i.d. setting and random i.i.d. partitioning.

The consequence. The paper's strong claim — "DiLoCo exhibits great robustness to the data distribution of each worker" — has only been validated for mild, topic-level heterogeneity within a single curated dataset. In realistic federated scenarios, data heterogeneity can be much more severe:

Different languages. If one worker trains on English Wikipedia and another on Chinese Wikipedia, the token distributions, sentence structures, and optimal parameter values may be substantially different, and outer gradient similarity could drop dramatically.
Different data quality. If one worker trains on web text and another on curated, edited text (news articles, books), the loss landscapes may have different local geometry — the web text worker may converge to a noisier minimum that, when averaged with the clean-data worker, produces a model worse than either alone.
Temporal or distribution shift. If workers collect data at different times or from different geographic regions, the data distributions may shift in ways that are not captured by simple topic clustering.
Adversarial shards. If a worker's data is poisoned or adversarially constructed to produce outer gradients that point in a harmful direction, uniform averaging would incorporate this without detection.

The paper's cosine similarity analysis (Figure 10b) shows that non-i.i.d. workers' outer gradients have lower similarity and higher variance than i.i.d. workers, but remain positively correlated. This positive correlation is the mechanism enabling convergence. If more extreme heterogeneity drove the cosine similarity to near zero or negative values, the outer averaging could become destructive — averaging vectors pointing in orthogonal directions produces a vector that points in neither worker's direction of improvement, potentially stalling training or causing divergence. The paper provides no characterization of HOW MUCH heterogeneity is tolerable before this breakdown occurs.

What evidence exists in the paper. Only the k-means semantic clustering on C4. The cosine similarity analysis (Figure 10) is insightful but describes a single degree of heterogeneity. The paper does not vary the "strength" of non-i.i.d.-ness — for instance, by adjusting the number of clusters or by constructing shards with deliberately low cross-shard similarity — to identify the boundary at which DiLoCo fails.

Mitigation status. Not addressed. The paper presents the non-i.i.d. result as evidence of robustness but does not define the limits of that robustness. A practitioner deploying DiLoCo in a federated setting with substantially different data per worker (e.g., different medical institutions with different patient populations) would not know from the paper's evidence whether the outer gradient similarity would remain positive enough for convergence. The paper's only diagnostic — monitoring cosine similarity between outer gradients — requires running the full training to detect failure, not predicting it from data characteristics beforehand.

6.5 DiLoCo Trades Off Data Efficiency and Compute Efficiency for Wall-Clock Speed

The assumption or constraint. DiLoCo's design philosophy, as articulated throughout the paper, prioritizes wall-clock training time — achieving good model quality quickly by parallelizing across many workers that communicate infrequently. The 8× updates baseline in Table 2 reveals the cost of this prioritization: a model trained sequentially for 8× more steps on a single worker achieves perplexity 14.72, versus DiLoCo's 15.02. The paper acknowledges this explicitly in Section 5:

"DiLoCo attains fast convergence in terms of wall-clock time. However, the distributed nature of the computation reduces the FLOP and data efficiency of the model, as shown by the 8× updates row in Table 2. At a high level, this is because the outer updates have effectively too large a batch size; but naively reducing the outer-update batch size would result in the workers being destabilized because their batch-size is too small."

The consequence. This is a fundamental tradeoff, not an implementation detail that can be optimized away. DiLoCo achieves its wall-clock speedup by consuming more total FLOPs and more total data tokens than a model trained to the same quality sequentially. The mechanism is well-understood from the large-batch training literature: when k workers each process batch size B for H steps before synchronizing, the effective outer gradient is computed from k × H × B tokens. This large effective batch size means each outer gradient is a very precise estimate of the gradient over the combined data shards — but precision in gradient estimation has diminishing returns for optimization. A sequential model using the same total tokens in smaller batches takes more noisy steps, each less precise but more numerous, often converging to a better minimum for the same total data.

Table 2 quantifies this concretely: DiLoCo uses 8× the compute and data of the single-worker baseline (88,000 steps × 512 batch size vs. 88,000 steps × 512 batch size × 8 workers), achieves much better perplexity (15.02 vs. 16.23), but the 8× updates baseline uses the SAME 8× compute and achieves EVEN better perplexity (14.72). So DiLoCo leaves ~0.30 perplexity "on the table" relative to what the same total compute could achieve if spent sequentially. This gap may be acceptable for wall-clock-constrained training, but it means DiLoCo is not compute-optimal — if the goal is the best possible model for a given total FLOPs or data budget, DiLoCo is strictly dominated by training longer on a single worker.

What evidence exists in the paper. Table 2 directly reports the gap. The paper provides no analysis of how this gap scales with the number of workers or with H — would DiLoCo with k = 64 workers have an even LARGER efficiency gap relative to the sequential compute-equivalent? The paper's diminishing returns with more workers (Table 3: going from 8 to 64 workers improves perplexity by only ~0.06–0.13) suggest that compute efficiency degrades further at higher k, but this is not explicitly discussed.

Mitigation status. The paper acknowledges the limitation in Section 5 and suggests future work: "balancing wall-clock time efficiency with compute efficiency and data efficiency, among other quantities of interest. In particular, we believe asynchronous variants of local SGD may allow distributed training with relatively more data-efficient updates." No concrete proposal or experiment is offered. For practitioners, the practical implication is clear: DiLoCo is the right choice when time-to-model-matters (you need a good model quickly and have access to multiple parallel compute clusters), but it is the wrong choice when compute or data budget is the binding constraint (you have limited GPUs or limited data and want to maximize final model quality).

6.6 The Outer Optimizer Configuration Was Tuned on the Same Model Scale and Task as the Main Experiments, With No Evidence of Transferability

The assumption or constraint. The outer optimizer — Nesterov momentum with learning rate η_outer = 0.7 and momentum μ = 0.9 — is the paper's primary algorithmic innovation over standard FedAvg, and the paper shows (Figure 6) that it decisively outperforms SGD, SGD with momentum, and Adam on the 150M model with H = 500. All subsequent experiments (varying worker count, data distribution, pretraining, dynamic compute) use these fixed outer optimizer hyperparameters. The paper states:

"In particular, the setting with outer learning rate equal to 0.7 and outer momentum equal to 0.9 is very robust, and it is adopted for all our experiments throughout."

The consequence. There is a circularity in the experimental validation: the outer optimizer was tuned on the exact same model size (150M), dataset (C4), and communication frequency (H = 500) that constitute the main experimental configuration. The paper then reports that this configuration works well — which it should, because it was selected to work well on this configuration. The question that matters for practitioners is whether these hyperparameters TRANSFER to other settings:

Different H values. Figure 4 varies H but uses the SAME outer optimizer hyperparameters tuned at H = 500. Would a different outer learning rate be optimal at H = 50 (where outer gradients are smaller and more frequent) or H = 2000 (where outer gradients are much larger and rarer)? The Nesterov momentum that works for step sizes of 500 inner updates may be suboptimal for step sizes of 50 or 2000 inner updates.
Different model sizes. Table 4 applies the 150M-tuned outer optimizer to 60M and 400M models. The authors acknowledge: "Hyper-parameters were tuned on the 150M model, which may be sub-optimal for the other model sizes." If larger models have outer gradients with very different norms (plausibly larger or smaller depending on architecture), the fixed outer learning rate of 0.7 could be meaningfully suboptimal, and the reported model size scaling (7.45% → 7.49% relative improvement from 150M to 400M) could understate the true improvement at 400M if outer hyperparameters were re-tuned.
Different datasets or tasks. If deployed on a different corpus with different token distribution and loss landscape, the optimal outer optimizer configuration might differ. The paper provides no evidence either way.

This is not a minor tuning issue — the outer optimizer is the core mechanism that distinguishes DiLoCo from standard FedAvg and from the failed local SGD configurations reported in prior work. If the outer optimizer hyperparameters require task-specific tuning, DiLoCo's practical adoption becomes more complex: each new deployment would require a hyperparameter search over outer optimizer type and parameters before distributed training can begin.

What evidence exists in the paper. The paper provides an ablation over outer optimizer TYPES (Figure 6) but not over their hyperparameters as a FUNCTION of H, model size, or data distribution. The grid search (Table 5) sweeps outer learning rates {1.0, 0.7, 0.5, 0.3, 0.1} and outer momentum values {0.95, 0.9, 0.8} on the 150M model only. The finding that η = 0.7 and μ = 0.9 is "very robust" is asserted but not demonstrated across configurations — robustness here means "works well across the ablations we ran," not "insensitive to the configuration it's applied to."

Mitigation status. The paper does not address this as a limitation. The outer optimizer hyperparameters are presented as fixed and general, with no discussion of whether they should be re-tuned for different settings. The speculation that "Nesterov's gradient correction is particularly helpful with the outer gradient that span hundred of training steps" (Section 3.1) suggests that the optimal outer optimizer might indeed depend on H — if outer gradients at H = 50 have different statistical properties than at H = 500, the optimal outer optimizer might differ. But this is not explored. A practitioner deploying DiLoCo at a new scale or on a new task should expect to perform at least a coarse outer optimizer sweep, which adds a non-trivial cost to the training pipeline.

7. Implications and Future Directions

How This Work Changes the Landscape

DiLoCo represents a reframing, not a paradigm shift. The paper does not introduce a fundamentally new optimization algorithm — its components (Federated Averaging, Nesterov momentum, AdamW) all exist in the literature. What changes is the field's empirical understanding of the viable operating regime for infrequent-communication training. Before this work, the dominant narrative, crystallized by Ortiz et al. (2021), was that local SGD degrades at scale: with more than a handful of inner steps, with many replicas, or without pretraining, model quality collapses. DiLoCo demonstrates that this narrative was an artifact of the specific optimizer configuration tested (SGD inner optimizer, simple averaging outer optimizer) rather than a fundamental property of infrequent communication. By swapping in AdamW as the inner optimizer and Nesterov momentum as the outer optimizer, the paper shows robust convergence at H = 500 — a 62.5× increase over the H = 8 where prior work reported failure — with no degradation up to 64 workers, even from random initialization.

The practical consequence of this reframing is that distributed LLM training becomes a software problem rather than a hardware problem. The paper's 500× reduction in communication is achieved entirely through algorithmic choices, not through faster interconnects or better networking equipment. This means organizations with access to multiple geographically distributed GPU clusters — a common situation for universities, mid-size companies, and even departments within large tech companies — can consider pooling them for a single training run without investing in dedicated co-location or high-bandwidth links. The 500× figure is not a theoretical bound but an empirical measurement of what the specific AdamW + Nesterov combination achieves on 150M-parameter transformers on C4, and the paper shows that acceptable performance persists even at H = 1000 (20× less communication than the default, for only 2.9% relative perplexity degradation, Figure 4).

The paper also resolves a tension in the local SGD literature that had been left unexplained. Some prior work found local SGD beneficial (Lin et al., 2020; Stich, 2019), while others found it degraded at scale (Ortiz et al., 2021). DiLoCo's outer gradient cosine similarity analysis (Appendix subsection 6.2, Figures 10–11) provides a mechanistic explanation for when local SGD should be expected to work: if workers' outer gradients remain positively correlated (cosine similarity > 0), averaging is beneficial; if they become orthogonal or negatively correlated, averaging becomes destructive. The paper shows that for transformer language models, the AdamW inner optimizer produces outer gradients that remain positively correlated even at H = 500 and even with non-i.i.d. data distributions. The counterintuitive finding that cosine similarity increases with larger H (comparing H = 250, 500, 1000 in Figure 10) — because longer local training averages out stochastic noise — provides a concrete diagnostic that future work can adopt to predict and monitor distributed training stability. This transforms distributed training from a black-box "try different H values and see what works" problem into something that can be debugged with a specific metric.

Which research directions become more attractive as a result:

Outer optimizer design for large-H regimes. Figure 6 shows that the outer optimizer choice qualitatively determines success or failure (Nesterov outperforms Adam by a wide margin). This opens a design space — outer optimizers specifically tailored to macro-gradients spanning hundreds of steps — that was previously invisible because prior work did not operate in the large-H regime where the distinction matters.
Scale extrapolation of distributed training. The paper's hypothesis that larger models benefit more from weight averaging (because they have "wider" loss basins with less interference) makes scaling DiLoCo to billion-parameter models a high-expected-value experiment. If the hypothesis holds, DiLoCo's advantages over synchronous training would increase with model size.
Combining algorithmic and systems-level communication reduction. Table 6 shows that simple sign-pruning can halve DiLoCo's already-reduced communication with negligible quality loss. This suggests a multiplicative effect: if DiLoCo reduces communication by 500× and compression adds another 2×, the combined reduction is 1000×, putting distributed training across genuinely low-bandwidth links (e.g., public internet) within reach.
Fully asynchronous variants. The paper's dropped communication experiment (Figure 8) and adaptive compute pool experiment (Figure 7) are steps toward asynchrony, but the current design still uses a synchronous outer loop where all workers must reach the outer step boundary before averaging proceeds. A fully asynchronous variant — where workers push outer gradients independently and the shared model updates continuously — would eliminate the straggler overhead and enable truly heterogeneous worker pools, but requires re-engineering the outer optimization to handle stale or partial updates.

Which directions become less urgent:

Incremental improvements to per-step gradient synchronization protocols. If DiLoCo's approach of infrequent synchronization with a sophisticated outer optimizer proves general, the effort invested in making all-reduce faster (better topologies, in-network aggregation, ring algorithms) becomes less critical — communication frequency drops to the point where bandwidth is rarely the bottleneck.
Hardware co-location as a training requirement. The paper demonstrates that model quality is not intrinsically tied to co-location. While co-location will remain desirable for latency-sensitive inference and may still be the most practical approach for the largest frontier training runs (where even 500× less communication of billion-parameter models is substantial), the paper establishes that it is not a hard requirement for good model quality — a meaningful shift for organizations operating under infrastructure constraints.
Simple one-shot model averaging (souping) as a training-time strategy. Figure 6 shows that DiLoCo's iterative averaging with Nesterov momentum substantially outperforms the one-shot averaging that the linear mode connectivity literature has focused on. While one-shot averaging remains useful for post-hoc model combination, the paper demonstrates that iterative, optimizer-driven averaging during training extracts more value from distributed workers than waiting until the end.

However, the shift is conditional on the domain. The paper explicitly acknowledges that CNNs (the focus of Ortiz et al., 2021's negative results) may genuinely be more sensitive to infrequent synchronization, and all experiments are on transformers with language modeling. The reframing is therefore not "local SGD works at scale, period" but rather "local SGD works at scale for transformer language models when configured with AdamW and Nesterov momentum, and the field should re-examine whether prior failures were optimizer-specific rather than fundamental." A direct replication of DiLoCo on ImageNet with ResNets — the exact setting where local SGD was previously reported to fail — would test the generality of the reframing and is one of the most important experiments the paper enables but does not perform.

A final subtle implication: the paper's single-worker acceleration result (Figure 9, k = 1, H = 500) shows that the outer optimization step improves model quality even without any distribution. This suggests that the outer optimizer is not merely a mechanism for integrating distributed workers, but may be a generally useful training technique — an implicit regularizer or exploration mechanism that periodically "steps back" from the current optimization trajectory. This is conceptually similar to the Lookahead optimizer (Zhang et al., 2019) but with Nesterov momentum replacing SGD in the outer step, and it implies that future single-worker training runs might benefit from DiLoCo-style outer optimization even when distribution is not needed. This is a low-cost enhancement (no communication, only a periodic local interpolation step) that could be tested broadly.

Follow-Up Research This Work Enables

Direct replication of DiLoCo on ImageNet with ResNets to determine whether prior local SGD failures were optimizer-specific or domain-specific. Ortiz et al. (2021) — the paper's primary counterpoint — concluded that local SGD fails at scale on image classification with ResNets, communicating every H = 8 steps. DiLoCo's success at H = 500 on language modeling with transformers does not directly refute this finding; it demonstrates success in a different setting. A direct replication would train a ResNet-50 or ResNet-101 on ImageNet using DiLoCo's exact configuration — AdamW inner optimizer, Nesterov momentum outer optimizer with η = 0.7 and μ = 0.9, H = 500, 8+ workers, and i.i.d./non-i.i.d. data shards — and compare against the synchronous baseline and against Ortiz et al.'s reported results for standard local SGD. If DiLoCo succeeds, it demonstrates that the failure was optimizer-specific, substantially strengthening the paper's reframing claim. If DiLoCo fails, the paper's findings are shown to be domain-specific (transformers benefit from weight averaging in ways CNNs do not), which is equally informative because it establishes the boundary of applicability. Either outcome advances understanding. The experiment would require careful attention to non-i.i.d. shard construction for ImageNet (Dirichlet partitioning by class is standard) and would ideally report outer gradient cosine similarity (analogous to Figure 10) to diagnose whether failure, if it occurs, is due to lower outer gradient similarity in the CNN setting.

Scaling DiLoCo to billion-parameter language models to test the hypothesis that larger models benefit more from weight averaging. The paper speculates that "larger models are less subject to interference when averaging their parameters" (Table 4 discussion, citing Ilharco et al., 2022), which would predict that DiLoCo's perplexity improvement over synchronous training increases with model scale. The 60M → 150M → 400M scaling experiment (Table 4) shows relative improvement increasing from 4.33% to 7.45% to 7.49%, but the 150M → 400M step shows diminishing returns, and the scale is far too small to extrapolate to the 1B–70B regime where frontier models operate. A direct experiment at 1B, 7B, or 70B parameters — using the same C4 dataset and Chinchilla-style architecture, but scaled up — would test the hypothesis. Key measurements: (1) relative perplexity improvement of DiLoCo over a synchronous time-equivalent baseline at each scale, (2) outer gradient cosine similarity as a function of model size (does it increase with scale, as the hypothesis predicts?), and (3) communication-to-computation ratio at each scale (does the 500× reduction in communication events translate to a PRACTICAL wall-clock advantage when the absolute size of parameter transmissions grows?). The experiment would require access to clusters large enough to run a synchronous baseline for comparison, which is a practical barrier — but it is the single most important experiment for determining whether DiLoCo matters at the scale where communication cost is most painful.

Comparing DiLoCo against synchronous data-parallel training with gradient compression at matched communication budgets. The paper's headline 500× communication reduction is relative to an uncompressed, per-step-synchronized baseline. A critical follow-up would pit DiLoCo (H = 500, no gradient compression during outer steps) against a synchronous data-parallel baseline that synchronizes every step but uses gradient compression — e.g., 8-bit quantization (32× reduction), gradient sparsification at 99% (100× reduction), or a combination — to achieve a similar total communication volume. The question is not whether DiLoCo reduces communication (it does), but whether the algorithmic approach to communication reduction (infrequent synchronization with a sophisticated outer optimizer) outperforms the systems-level approach (frequent synchronization with compressed gradients) at the same total communication budget. If compressed synchronous training matches or exceeds DiLoCo's perplexity, the case for DiLoCo's algorithmic complexity is weakened; if DiLoCo outperforms, the two-level optimization structure is shown to provide benefits beyond what simple compression can achieve. The experiment should sweep communication budgets (e.g., 10×, 100×, 500×, 1000× reduction from the uncompressed baseline) and compare Pareto frontiers of perplexity vs. total bytes transmitted for both approaches.

Varying H dynamically during training based on outer gradient statistics. The paper uses a fixed H = 500 throughout training, but the cosine similarity analysis (Figure 10) shows that outer gradient properties change over the course of training — specifically, similarity increases as the inner learning rate decays. This suggests a natural extension: start with a small H (e.g., 50–100) early in training when gradients are large, volatile, and less aligned across workers, then progressively increase H (to 500, 1000, or beyond) as training stabilizes and outer gradients become more similar. The trigger for increasing H could be based on outer gradient cosine similarity exceeding a threshold, or on the outer gradient norm dropping below a threshold. This dynamic schedule would potentially improve the communication-quality tradeoff by communicating more frequently when it matters most (early training) and less when it matters less (late training), without requiring manual tuning of H. A concrete experiment: train the 150M model on C4 with k = 8 and a schedule like H = {100, 250, 500, 1000} triggered at cosine similarity thresholds, and compare the perplexity-vs.-communication Pareto curve against the fixed-H baselines from Figure 4. A negative result — dynamic H doesn't beat the best fixed H — would also be informative, suggesting that the current H = 500 is near-optimal throughout.

Fully asynchronous DiLoCo where workers update the shared model independently, eliminating the synchronization barrier entirely. The paper's dropped communication experiment (Figure 8) and adaptive compute pool experiment (Figure 7) demonstrate robustness to workers missing outer rounds and to worker count changing over time, but the outer loop itself remains synchronous — all active workers must complete H steps before averaging proceeds. A fully asynchronous variant would allow each worker to push its outer gradient to a central parameter server (or a decentralized equivalent) as soon as it completes H inner steps, independently of other workers' progress. The shared model would be updated continuously using the Nesterov outer optimizer, but now receiving outer gradients at irregular intervals from workers operating at different speeds. Key challenges: (1) How does the Nesterov momentum buffer behave when outer gradients arrive at varying intervals with varying staleness? (2) Does the outer optimizer need to be modified to handle the fact that an outer gradient computed over H steps by a fast worker represents a different "amount of progress" than one computed over H steps by a slow worker? (3) What is the tradeoff between asynchrony (which eliminates straggler idle time) and model quality (since the shared model parameters may have changed between when a worker started its inner loop and when it submits its outer gradient)? The experiment would compare wall-clock time to a target perplexity for fully synchronous DiLoCo vs. the asynchronous variant under simulated straggler distributions (e.g., workers with speeds drawn from a log-normal distribution). This is the most natural extension for heterogeneous hardware pools and is flagged as future work in Section 5.

Combining DiLoCo's outer optimization with model parallelism within each worker, targeting the regime where each worker is itself a large distributed system. The paper treats each worker as a monolithic compute unit, but at billion-parameter scales each worker would itself require model parallelism (tensor parallelism + pipeline parallelism) to fit the model in memory. The interaction between intra-worker model parallelism strategies and DiLoCo's outer optimization is unexplored. Specifically: (1) When a worker uses pipeline parallelism, its inner optimization is already asynchronous across pipeline stages — does this interact with the outer gradient computation? (2) The outer gradient Δ_i = θ^{(t-1)} - θ_i^{(t)} requires the starting parameters θ^{(t-1)} and the ending parameters θ_i^{(t)} to be in the same coordinate frame, but if the worker's model parallelism involves data-dependent computation ordering, the effective parameter trajectory may not be a simple vector difference. (3) Does model parallelism change the effective batch size per inner step in ways that affect outer gradient quality? A concrete experiment: train a 7B-parameter model where each of k = 4 workers uses 8-way tensor parallelism and 4-stage pipeline parallelism internally, comparing DiLoCo against fully synchronous training with the same total device count, measuring both perplexity and wall-clock time. This bridges the gap between DiLoCo's current small-scale validation and the regime where distributed training is most needed.

Practical Applications and Downstream Use Cases

Collaborative LLM training across academic consortia with pooled GPU resources. The most direct application enabled by DiLoCo is federated training across multiple institutions that each own a modest GPU cluster (e.g., 16–64 A100s or V100s) but collectively possess enough compute to train a substantial language model if they could combine resources. Under the standard synchronous paradigm, this is infeasible because inter-institution network links have latencies of tens to hundreds of milliseconds and bandwidths of 1–10 Gbps — far too slow for per-step gradient synchronization across hundreds of gigabytes of model parameters. DiLoCo reduces the communication requirement to once every 500 steps. For a 150M-parameter model (~600MB in float32), transmitting outer gradients over a 1 Gbps link takes approximately 5 seconds. If each inner step takes ~0.5 seconds on a 16-GPU worker, H = 500 inner steps take 250 seconds. The communication overhead is therefore 5 seconds per 250 seconds of compute — approximately 2% overhead. For a consortium of 5 universities each with a 64-GPU cluster, this would enable training runs using 320 GPUs total on a 1 Gbps shared internet link, with quality matching the synchronous equivalent (Table 2: 15.02 vs. 15.30 perplexity). The practical benefit is that DiLoCo makes possible what was previously architecturally impossible without dedicated networking infrastructure. The paper's adaptive compute pool results (Figure 7) further mean that if one university's cluster goes offline for maintenance, training continues without reconfiguration — the remaining workers simply average their outer gradients and proceed.

Training across preemptible cloud instances in different regions or from different providers to reduce cost. Cloud providers offer substantial discounts (60–90%) for preemptible or spot instances, but these instances can be reclaimed at any time with short notice. Standard synchronous training cannot use preemptible instances without frequent checkpointing and restarting, because losing even one worker stalls the entire training step. DiLoCo's robustness to worker count variation (Figure 7) and communication drops (Figure 8) means preemptible instances can be treated as transient contributors: when an instance is reclaimed, its outer gradient for that round is simply excluded from the average, and the remaining workers continue. When a new preemptible instance becomes available, it receives the current shared model and joins at the next outer step. The total compute budget is what matters — not whether each unit of compute was contributed continuously. Concretely, an organization training a 400M-parameter model on C4 could assemble a pool of 64 preemptible A100 instances across three cloud regions and three providers (AWS, GCP, Azure) with no dedicated inter-region networking beyond standard public internet links. At 50% average preemption rate, Figure 8 shows only 2.1% perplexity degradation. The cost savings from preemptible pricing — combined with the ability to use cheapest-available instances rather than co-locating — could reduce training cost by 5–10× relative to dedicated instances in a single facility, potentially making large-scale language model training accessible to groups that cannot afford dedicated supercomputer time. The paper does not demonstrate this at billion-parameter scale, and the absolute communication times would be larger (a 7B-parameter model in float32 is ~28GB, taking ~224 seconds over 1 Gbps), but the 500× reduction in communication events means the overhead remains manageable even at larger scales as long as H is set appropriately.

Geographically distributed fine-tuning of a shared base model on domain-specific data without data centralization. Many organizations possess domain-specific text data that they cannot or will not share due to privacy, regulatory, or competitive concerns — hospitals with clinical notes, law firms with case documents, financial institutions with transaction records. Fine-tuning a shared language model on this data typically requires either centralizing the data (which breaks privacy guarantees) or training separate models per organization (which fails to benefit from the combined data scale). DiLoCo enables federated fine-tuning: each organization trains a model replica on its own data shard for H steps, the outer gradients (not the data) are transmitted to a central coordinator, an updated shared model is returned, and the process repeats. The paper's non-i.i.d. robustness (Figure 5: i.i.d. and non-i.i.d. shards achieve equivalent final perplexity) directly supports this scenario, because different organizations' data is naturally non-i.i.d. — a hospital's clinical notes and a law firm's case documents are semantically very different. The key practical advantage is that only model parameter differences, not training data, ever leave each organization's premises. For a 150M-parameter model, each outer gradient is ~600MB; for T = 128 outer rounds, the total data transmitted per organization is ~75GB — a manageable amount for overnight transmission on institutional internet connections, and 500× less than what synchronous federated training would require. The paper's weighted averaging scheme for non-i.i.d. data (weighting outer gradients by shard size) naturally handles organizations with different data volumes. A concrete deployment scenario: 5 hospitals each with 10–50GB of de-identified clinical text, collaboratively fine-tuning a 150M-parameter LM for medical language understanding, with each hospital's data never leaving its firewall. The paper's results suggest the resulting model would match the perplexity of a centrally trained model on the combined data, with the outer gradient cosine similarity analysis (Figure 10b) providing a real-time diagnostic of whether the hospitals' data distributions are compatible enough for effective collaboration.

Training on intermittently available volunteer compute through a distributed computing network. Projects like Folding@home, SETI@home, and distributed.net have demonstrated that millions of volunteers will contribute spare compute cycles to scientific projects. Applying this model to LLM training has been proposed (Diskin et al., 2021; Presser, 2020; Ryabinin et al., 2021; Borzunov et al., 2022) but has been limited by the communication requirements of synchronous training — volunteers' machines are behind residential internet connections with limited upload bandwidth and high latency, making per-step gradient synchronization impossible. DiLoCo reduces the communication requirement to transmitting model parameters once every few minutes (at H = 500, with a consumer-grade NVIDIA GPU completing an inner step in ~1–2 seconds, H = 500 takes 8–16 minutes). A volunteer contributing a single RTX 3060 would: (1) download the current shared model from a central server (~600MB for 150M parameters), (2) train locally on an assigned data shard for ~10 minutes, (3) upload the resulting outer gradient (~600MB) — a total of ~1.2GB transferred per ~10 minutes of compute, or ~7GB/hour. This is within the capabilities of typical residential broadband (10–100 Mbps upload) and would not saturate a volunteer's connection. The paper's robustness to worker dropout (Figure 8: 50% communication failure causes 2.1% degradation) is critical here because volunteers will go offline unpredictably. The adaptive compute pool result (Figure 7: total compute determines quality, not timing) means that volunteers joining and leaving at arbitrary times does not degrade the final model — only the total aggregate compute matters. A practical system would need to handle security (verifying that volunteers' outer gradients are genuine, not adversarial), data distribution (assigning shards to volunteers and ensuring they train on the assigned data), and incentive design (why would volunteers contribute?), but DiLoCo provides the algorithmic foundation that makes the communication requirements feasible. This application remains speculative — the paper does not demonstrate DiLoCo with thousands of workers or with the extreme heterogeneity (GPU types, internet speeds, availability patterns) that a volunteer network would exhibit — but it is a direct extrapolation of the properties the paper has demonstrated.

When to Prefer DiLoCo

The paper itself does not provide an explicit decision framework comparing DiLoCo against named alternatives, and a formulaic "prefer A when X, prefer B when Y" matrix would go beyond what the experimental evidence supports. The paper demonstrates that DiLoCo provides a specific Pareto improvement over synchronous data-parallel training with the same batch size — better perplexity (15.02 vs. 15.30), same wall-clock time, 500× less communication — but does not compare against compressed synchronous training, against asynchronous distributed training, or against sequential training with the same total compute budget. The paper does acknowledge one clear tradeoff: DiLoCo achieves wall-clock speed at the cost of compute and data efficiency, as shown by the 8× updates baseline in Table 2 (14.72 perplexity vs. DiLoCo's 15.02, using the same total compute sequentially). For practitioners, the decision depends on whether wall-clock time or total compute budget is the primary constraint, and whether the practitioner has access to multiple distributed clusters or only a single cluster — a set of considerations the paper's experiments inform but do not fully resolve. The limitations section of this analysis (Section 6) details the relevant tradeoffs and conditions under which the paper's claims hold; a more prescriptive decision framework would require head-to-head comparisons against practical alternatives (compressed synchronous training, asynchronous variants) that the paper does not provide.

🎯 Pitch

1. Executive Summary

2. Context and Motivation

The Core Problem: Modern LLM Training Requires Impractical Hardware Cohabitation

Why This Matters: The Stakes Beyond Engineering Convenience

Prior Approaches and Where They Fall Short

Standard Data Parallelism: Communication Every Step

Local SGD: The Obvious First Attempt

Advanced Federated Optimizers: Still Too Frequent Communication

The Linear Mode Connectivity Perspective

Accelerating a Single Worker: The Lookahead Connection

How DiLoCo Positions Itself

3. Technical Approach

3.1 Reader Orientation

3.2 Big-Picture Architecture (Diagram in Words)

3.3 Roadmap for the Deep Dive

3.4 Detailed, Sentence-Based Technical Breakdown

The Two-Level Optimization Structure (Algorithm 1)

The Inner Optimizer: Why AdamW Specifically

The Outer Gradient: Properties and Aggregation

The Outer Optimizer: Nesterov Momentum

Pretraining Strategy and Initialization

Non-I.I.D. Data Shard Construction

Hyperparameter Configuration and Training Recipe

What Makes DiLoCo Different from Standard Federated Averaging

Edge Cases: Special Operating Modes

4. Key Insights and Innovations

Innovation 1: Reframing Distributed Training as a Problem of Optimizer Design, Not Infrastructure

Innovation 2: The Outer Gradient Cosine Similarity as a Diagnostic for When Distributed Training Will Succeed

Innovation 3: The Outer Optimizer as a First-Class Design Choice, Not an Afterthought

Innovation 4: Robustness to Dynamic Compute as a First-Class Property, Not an Afterthought

5. Experimental Analysis

Evaluation Methodology

Main Quantitative Results

Headline Result: DiLoCo vs. Synchronous Baselines (Figure 2, Table 2)

Robustness to Communication Frequency (Figure 4)

Robustness to Data Distribution: I.I.D. vs. Non-I.I.D. (Figure 5)

Scaling the Number of Workers: Table 3

Scaling Model Size: Table 4

Robustness to Pretraining: Figure 3

Outer Optimizer Comparison: Figure 6

Single-Worker Acceleration: Figure 9

Adaptive Compute Pool: Figure 7

Robustness to Dropped Communication: Figure 8

Ablation Studies and Robustness Checks

Critical Assessment

Do the Experiments Support the Central Claims?

Genuine Weaknesses in Experimental Design

Missing Experiments That Would Strengthen the Paper

Conditional Nature of the Claims

6. Limitations and Trade-offs

6.1 The Empirical Validation Is Confined to a Single Task, Modality, and Architecture Family at Modest Scale

6.2 The Comparison Against Synchronous Baselines Does Not Include Practical Alternatives That Also Reduce Communication

6.3 Wall-Clock Time Is Reported as Proportional to Inner Steps, But This Ignores Outer Optimization Overhead and Straggler Effects

6.4 The Difficulty Estimation Equivalent — Data Shard Similarity — Is Not Characterized as a Precondition for Convergence

6.5 DiLoCo Trades Off Data Efficiency and Compute Efficiency for Wall-Clock Speed

6.6 The Outer Optimizer Configuration Was Tuned on the Same Model Scale and Task as the Main Experiments, With No Evidence of Transferability

7. Implications and Future Directions

How This Work Changes the Landscape

Follow-Up Research This Work Enables

Practical Applications and Downstream Use Cases

When to Prefer DiLoCo

Adjacent in domain

Parameter-Efficient Orthogonal Finetuning via Butterfly Factorization

Fine-tuning Language Models for Factuality