Asynchronous Local-SGD Training for Language Modeling

ArXiv: 2401.09135

🎯 Pitch

Naïve asynchronous local-SGD for language models can paradoxically slow convergence despite more frequent global updates—but carefully delaying Nesterov momentum on the server fixes this, matching synchronous performance per step and crushing wall-clock time by eliminating straggler waits.

---

1. Executive Summary

This paper presents an empirical study of asynchronous Local-SGD for training language models, evaluating how worker hardware heterogeneity, model size, number of workers, and optimizer choice impact learning performance on the C4 dataset using transformer models up to 150M parameters. The authors identify momentum acceleration on the global parameters when worker gradients are stale as a key challenge, finding that naive asynchronous Local-SGD takes more iterations to converge than synchronous DiLoCo despite updating global parameters more frequently. To address this, they propose two techniques—Delayed Nesterov (a method that buffers pseudo-gradients and applies momentum updates only every N server steps while performing SGD between them) and Dynamic Local Updates (adjusting each worker's local training steps proportionally to its computation speed)—which together match synchronous DiLoCo in perplexity per update step and significantly surpass it in wall clock time. The approach demonstrates consistent efficacy across heterogeneity levels and model sizes, establishing that asynchronous Local-SGD can be competitive with synchronous methods only when the outer momentum update is carefully managed to account for sequential pseudo-gradient application.

2. Context and Motivation

The Core Problem: Can We Train Language Models Across Geographically Distributed Hardware?

The fundamental question this paper tackles is whether language models can be effectively trained using asynchronous distributed optimization, where workers operate at different speeds and communicate independently with a central server, rather than waiting for all workers to finish before synchronizing. This matters because the dominant paradigm for large-scale language model training assumes co-located devices with fast interconnects—hardware sitting in the same datacenter, connected by high-bandwidth links, and operating at roughly identical speeds. The paper opens by challenging this assumption:

"One might hope to be able to effectively harness a broader range of computational resources, perhaps geographically distant from each other, in order to build even more powerful large models."

The motivation here is both practical and theoretical. Practically, the world has an enormous amount of distributed compute—idle GPUs in different locations, volunteered computing resources, underutilized academic clusters—that cannot currently contribute to language model training because standard synchronous methods cannot tolerate the communication latency and device heterogeneity inherent in geographically distributed systems. Theoretically, understanding how optimization behaves when updates arrive out-of-order and with varying staleness is a fundamental question in distributed optimization that has not been systematically studied for the specific setting of Local-SGD applied to language modeling.

Why Synchronous Training Fails for Distributed Resources

The paper frames the problem through the lens of two interrelated bottlenecks that prevent synchronous methods from effectively using heterogeneous, geographically distributed hardware:

The straggler effect. In synchronous distributed training, all workers must complete their local work before any worker can proceed to the next global update. If even one device is slower than the others—due to hardware differences, network variability, or transient load—every other device sits idle waiting for it. This is not a minor inefficiency; in Figure 1, the paper illustrates the contrast: synchronous training shows the fast worker (in blue) repeatedly blocked by the slow worker (in red), while asynchronous training allows the fast worker to continue its next training task immediately after finishing. As the number of workers grows or the variance in device speeds increases, the idle time fraction can dwarf the useful computation time, making synchronous training prohibitively wasteful.

Simultaneous communication requirement. Beyond the straggler effect, synchronous methods demand that all workers communicate their updates at the same time. This creates a peak bandwidth requirement: the central server must receive updates from all workers simultaneously, and all workers must download the new model simultaneously. In geographically distributed settings, where bandwidth is limited and variable, this synchronized burst communication pattern is problematic. Asynchronous methods naturally smooth out communication, with workers sending and receiving updates independently throughout the training loop.

The Promise of Local-SGD and the Asynchronous Gap

Local-SGD (also known as Federated Averaging or FedAvg) addresses part of the communication bottleneck by allowing each worker to perform multiple local gradient steps before communicating with the server, rather than communicating after every single gradient computation. This reduces communication frequency by a factor of $H$ (the number of local steps), which can make distributed training feasible even with limited bandwidth. The state-of-the-art synchronous Local-SGD method for language modeling is DiLoCo (Douillard et al., 2023), described in Algorithm 1. DiLoCo's key operational pattern:

1. Each worker $i$ receives the current global parameters $\theta^{(t-1)}$.
2. It performs $H$ local updates using an inner optimizer (AdamW) on its data shard, producing $\theta^{(t)}_i$.
3. It sends the pseudo-gradient $\delta^{(t)}_i = \theta^{(t-1)} - \theta^{(t)}_i$ back to the server.
4. The server waits for all $k$ workers, averages the pseudo-gradients into $\Delta^{(t)} = \frac{1}{k}\sum_{i=1}^k \delta^{(t)}_i$, and applies an outer optimizer (Nesterov momentum) to update the global parameters.

This is synchronous: the outer step at line 11-12 proceeds only after every worker has completed its $H$ inner steps. The paper's core motivation is to study what happens when we remove this synchronization barrier and allow the server to update $\theta$ as soon as any worker's pseudo-gradient arrives—the asynchronous version of Local-SGD.

Conflicting Signals in Prior Work

The paper enters territory where prior work has sent mixed signals about whether asynchronous Local-SGD can work for language modeling at all:

Positive signals from asynchronous training in other contexts. Asynchronous SGD has a long history in distributed optimization (Dean et al., 2012; Recht et al., 2011; Lian et al., 2015) and has demonstrated success in specific language modeling scenarios. Diskin et al. (2021b) showed that asynchronous training across globally distributed heterogeneous devices could work for language modeling, suggesting the approach has fundamental viability.

Positive signals from synchronous Local-SGD for language models. DiLoCo (Douillard et al., 2023) established that synchronous Local-SGD with the AdamW+Nesterov combination works effectively for training language models, providing a strong synchronous baseline to compare against. This matters because it shows Local-SGD itself does not inherently degrade language model quality—the question is specifically whether the asynchronous variant can match it.

Negative signals from prior asynchronous Local-SGD fixes. The paper tests several existing methods from the asynchronous optimization literature (Section 4) to address the staleness problem:

• Polynomial discounting (Xie et al., 2019): downweighting stale pseudo-gradients by a factor of $(1 + \text{staleness})^{-0.5}$. The paper finds this provides only marginal benefits (Figure 8).
• Threshold-based discard: throwing away updates with staleness exceeding 10. Also largely ineffective.
• Delay compensation (Zheng et al., 2017): approximating the true gradient at current parameters using a first-order Taylor expansion of the stale gradient. The paper explicitly notes: "the fact that delay compensation is not working well points out the difference between asynchronous SGD and asynchronous Local-SGD." This is a crucial insight—standard SGD with asynchronous updates has different staleness characteristics than Local-SGD where each worker's update is a pseudo-gradient accumulated over $H$ local steps.
• FedBuff (Nguyen et al., 2022): buffering pseudo-gradients from multiple workers and only applying a server update after accumulating a sufficient number. This approach "significantly closes the gap between sync. and async. training" (Figure 8) but "exhibits instability early in training."

Critically, none of these existing methods match the synchronous DiLoCo baseline in final perplexity. The paper's Figure 2 (and Figure 8) shows a clear and persistent gap between Async. DiLoCo and synchronous DiLoCo that existing fixes cannot close. This establishes that asynchronous Local-SGD poses a distinct optimization challenge not solved by existing staleness mitigation techniques.

The Missing Piece: Momentum in the Outer Optimizer

The paper's central diagnostic insight emerges from a carefully designed experiment in Section 4. When comparing synchronous and asynchronous DiLoCo using AdamW+SGD (no outer momentum) versus AdamW+Nesterov (with outer momentum), a counterintuitive pattern appears (Figure 6):

• Without outer momentum (AdamW+SGD): asynchronous training outperforms synchronous training. The absence of a synchronization barrier allows more frequent parameter updates, and without momentum to mismanage, this translates directly to better convergence.
• With outer momentum (AdamW+Nesterov): synchronous training significantly outperforms asynchronous. The momentum term, which is crucial for strong synchronous performance (it stabilizes training and accelerates convergence), becomes a liability in the asynchronous setting.

This is the paper's key empirical finding that motivates the entire technical contribution: momentum in the outer optimizer is essential for good performance, but naive application in asynchronous settings degrades rather than helps. The authors trace this to how Nesterov momentum updates behave when pseudo-gradients are applied sequentially rather than simultaneously (Equation 5 in the paper). When $k=4$ workers produce identical pseudo-gradients $g$ that arrive simultaneously, synchronous Nesterov applies the update once. When those same gradients arrive asynchronously and are applied sequentially, the momentum term decays differently and the parameter change is amplified, creating an imbalance that "cannot be simply rectified by reducing the learning rate."

Positioning Relative to Existing Work

The paper positions itself at the intersection of three research threads that have previously been studied largely independently:

Local-SGD for language modeling (DiLoCo). This paper directly extends DiLoCo's synchronous framework to the asynchronous setting, adopting its AdamW+Nesterov optimizer pairing as the starting point and its pseudo-gradient formulation. The paper acknowledges that DiLoCo already demonstrated Local-SGD's viability for language model training from scratch (Douillard et al., 2023), so the contribution is specifically about making this work without the synchronization barrier.

Asynchronous federated learning. Prior work like FedBuff (Nguyen et al., 2022) and TimelyFL (Zhang et al., 2023) addressed asynchrony in federated settings, but these methods focused on communication efficiency and privacy, not on the specific interaction between outer momentum and sequential pseudo-gradient application that the paper identifies as the core challenge for language model training.

Standard asynchronous SGD. Methods like delay compensation (Zheng et al., 2017) and polynomial discounting (Xie et al., 2019) were developed for standard asynchronous SGD (where each worker communicates after every gradient step) and do not directly address the Local-SGD setting where workers accumulate pseudo-gradients over multiple steps before communicating. The paper's empirical demonstration that these methods fail (Figure 8) establishes asynchronous Local-SGD as a distinct problem setting requiring its own solutions.

The paper thus positions its contributions—Delayed Nesterov and Dynamic Local Updates—not as general-purpose optimization improvements but as targeted fixes for a specific failure mode: the degradation of outer momentum when pseudo-gradients from multiple Local-SGD workers are applied sequentially rather than simultaneously. This framing is important because it explains why the solutions work (they manage the momentum update's timing and granularity) and when they would apply (specifically to Local-SGD with momentum-based outer optimizers in asynchronous settings).

Why Language Modeling Specifically?

The paper's focus on language modeling is not incidental. Language models present specific challenges for asynchronous Local-SGD:

• Large models mean communication cost is high relative to computation, making Local-SGD's reduced communication frequency valuable.
• Training from scratch (or fine-tuning from a partially trained checkpoint as done here—24,000 pretraining steps followed by distributed Local-SGD training) means optimization dynamics matter greatly; poor convergence in the early phases compounds over the remaining training.
• Perplexity as a metric is sensitive to small optimization differences, providing a fine-grained signal for comparing synchronous and asynchronous methods that might be obscured by top-1 accuracy on classification tasks.

The C4 dataset (Raffel et al., 2020) is chosen as a standard language modeling benchmark derived from Common Crawl, and the transformer models are Chinchilla-style (Hoffmann et al., 2022) decoder-only architectures at 20M, 60M, and 150M parameter scales. While modest by contemporary LLM standards, the paper argues these sizes are sufficient to study optimization dynamics, and the consistent patterns across sizes suggest the findings may generalize.

3. Technical Approach

3.1 Reader Orientation

This paper builds a distributed training system that lets language models be trained across multiple devices of different speeds without requiring those devices to wait for each other at synchronization points. The core problem it solves is that the standard approach to distributed training—synchronous Local-SGD with momentum-based outer optimization—degrades severely when the synchronization barrier is removed, because the momentum term that normally accelerates convergence instead amplifies the distortion caused by stale, sequentially-applied pseudo-gradients. The solution is twofold: (1) Delayed Nesterov, which restructures how and when momentum is applied to the global parameters so that it mimics synchronous behavior despite sequential gradient application, and (2) Dynamic Local Updates, which adjusts each worker's local training steps proportionally to its speed so that workers finish their local work at roughly the same time, reducing the staleness that causes the problem in the first place.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components connected in a continuous loop:

1. Central Parameter Server — maintains the authoritative copy of the model parameters $\theta$ and executes the outer optimizer (either Nesterov momentum or the proposed Delayed Nesterov variant). It receives pseudo-gradients from workers asynchronously, buffers them according to the update policy, and applies parameter updates without waiting for all workers.

2. Workers ($k$ heterogeneous devices) — each worker independently executes a training task: it samples a data shard, receives the current server model, performs $H$ local training steps using an inner optimizer (AdamW), and sends the resulting pseudo-gradient (the negative of its total parameter change) back to the server. Workers operate at different speeds and on different schedules—the fast worker in Figure 1 proceeds to its next task immediately after finishing, while slower workers take longer.

3. Data Shard Sampler — whenever a worker is assigned a new training task, this component selects which data shard the worker trains on, using a sampling distribution (Equation 2) that favors shards whose data has been seen less frequently relative to their size, ensuring balanced learning progress across shards despite heterogeneous worker speeds.

4. Dynamic Local Updates (DyLU) Module — before a worker begins training, this module determines how many local steps the worker should perform based on its measured computation speed relative to the fastest worker (Equation 6). Slower workers get fewer steps so that all workers complete their tasks at approximately the same time.

5. Grace Period Synchronizer — when a worker finishes training, it does not immediately pull the latest server model and start a new task. Instead, the server waits for a short configurable grace period $\tau_{\text{grace}}$ to see if any other worker finishes during that window. If another worker does finish within the window, both workers receive the same updated model (incorporating both of their pseudo-gradients) before starting new tasks, reducing staleness at the cost of a brief wait.

Information flows cyclically: a worker finishes training → sends pseudo-gradient to server → server buffers/applies update (possibly waiting for grace period) → worker receives updated model → data shard sampler selects shard → DyLU module sets step count → worker trains on shard with inner optimizer → repeats. The entire loop is managed by the task scheduler in Algorithm 2, which ensures that workers are never idle except during the optional grace period and that the server continuously processes incoming updates.

3.3 Roadmap for the Deep Dive

• First, the mathematical formulation of what we're optimizing (Equation 1—the distributed objective) and the synchronous DiLoCo algorithm (Algorithm 1) that serves as the paper's baseline, since the entire paper is structured as a comparison against it and the asynchronous methods are modifications of its structure.
• Second, the asynchronous training framework itself—the data shard sampling, learning rate scheduling, grace period mechanism, and task scheduling algorithm—since this defines the operating environment in which the optimization challenges arise.
• Third, the diagnostic experiments that isolate why asynchronous Local-SGD fails—the optimizer combination sweep (Figure 5), the momentum-specific comparison (Figure 6), and the homogeneity experiment (Figure 7)—since understanding the failure mode is prerequisite to understanding the solutions.
• Fourth, the Delayed Nesterov update (Algorithm 3), including its buffer mechanism, the rationale for intermittent momentum application, and the detailed update equations showing how it differs from both standard Nesterov and the naive sequential application.
• Fifth, the Dynamic Local Updates (DyLU) strategy, including how worker speeds are measured, how local steps are computed (Equation 6), and how this interacts with the grace period to reduce staleness.
• Sixth, the combined system (DN + DyLU) and how the two techniques address complementary aspects of the staleness problem—DN fixes the optimizer's response to staleness, DyLU reduces the amount of staleness that occurs.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical systems and optimization paper whose core contribution is the identification of a specific failure mode in asynchronous Local-SGD (momentum degradation under sequential pseudo-gradient application) and two targeted fixes (Delayed Nesterov and Dynamic Local Updates) that together recover synchronous-level performance.

---

The Distributed Optimization Objective and Synchronous DiLoCo Baseline

The formal objective. The paper frames distributed language model training as minimizing a weighted sum of per-shard expected losses:

$\min_{\theta} \sum_{i=1}^{k} \frac{|\mathcal{D}_i|}{\sum_j |\mathcal{D}_j|} \mathbb{E}_{x \sim \mathcal{D}_i} \left[ \ell(x; \theta) \right]$

where $\theta$ represents the model parameters, $k$ is the number of workers (which equals the number of data shards in the paper's setup), $\mathcal{D}_i$ is the $i$-th data shard, $|\mathcal{D}_i|$ is the number of examples in that shard, and $\ell(x; \theta)$ is the loss function (cross-entropy for next-token prediction in language modeling).

What it computes: the expected loss over all data, where each shard's contribution is weighted by its size relative to the total dataset. This is exactly the objective that single-machine training would optimize if all data were in one place; the challenge is that the data is partitioned across $k$ shards accessible only by their respective workers, and workers cannot share raw data—only model updates.

Why this form: the size-weighting ensures that larger shards contribute proportionally more to the objective, preventing a scenario where a small shard with noisy data dominates the gradient. This is standard in distributed and federated learning. The paper assumes $k$ workers equals $k$ shards for simplicity, but notes that the methods also apply when there are fewer workers than shards (workers would dynamically sample from different shards).

The DiLoCo algorithm (Algorithm 1) — the synchronous baseline. DiLoCo (Distributed Low-Communication training, Douillard et al., 2023) is the state-of-the-art synchronous Local-SGD method that this paper extends to the asynchronous setting. Its procedure, stated in Algorithm 1, operates on a two-level loop:

1. Outer loop (indexed by $t = 1 \ldots T$): each outer step corresponds to one round of communication between workers and server. The server distributes the current global parameters $\theta^{(t-1)}$ to all workers.

2. Inner loop (parallel across workers, indexed by $h = 1 \ldots H$): each worker $i$ initializes its local parameters $\theta^{(t)}_i \leftarrow \theta^{(t-1)}$, then performs $H$ local SGD steps on its data shard $\mathcal{D}_i$. For each inner step $h$, the worker samples a batch $x \sim \mathcal{D}_i$, computes the loss $\mathcal{L} \leftarrow f(x, \theta^{(t)}_i)$, computes the gradient $\nabla \mathcal{L}$, and updates its local parameters using an inner optimizer: $\theta^{(t)}_i \leftarrow \text{InnerOpt}(\theta^{(t)}_i, \nabla \mathcal{L})$. The designated InnerOpt is AdamW.

3. Pseudo-gradient computation (line 9): after $H$ local steps, the worker computes its pseudo-gradient as the negative of the total local change: $\delta^{(t)}_i = \theta^{(t-1)} - \theta^{(t)}_i$. This is called a "pseudo-gradient" rather than a gradient because it aggregates the effect of $H$ local AdamW update steps, each of which involves momentum, adaptive learning rates, and weight decay—it is not simply $H$ times a single gradient.

4. Aggregation (line 11): the server waits for all $k$ workers to return their pseudo-gradients, then computes the average: $\Delta^{(t)} = \frac{1}{k} \sum_{i=1}^k \delta^{(t)}_i$. This is the "outer gradient."

5. Outer update (line 12): the server applies an outer optimizer to update the global parameters: $\theta^{(t)} \leftarrow \text{OuterOpt}(\theta^{(t-1)}, \Delta^{(t)})$. The designated OuterOpt is Nesterov momentum.

The key design choice — AdamW + Nesterov pairing. The paper adopts this specific optimizer combination from DiLoCo without modification: "A key insight from DiLoCo is the optimal use of AdamW and Nesterov Momentum as the best inner and outer optimizers, respectively." AdamW as the inner optimizer provides adaptive per-parameter learning rates and decoupled weight decay, which helps each worker navigate the local loss landscape efficiently. Nesterov momentum as the outer optimizer provides acceleration on the global parameter trajectory, using the aggregated pseudo-gradient across workers to build velocity in consistent descent directions. The paper's Figure 5 confirms that this combination also performs best in the asynchronous setting among the tested alternatives, establishing it as the starting point from which the proposed modifications depart.

Why pseudo-gradients rather than raw gradients. The formulation $\delta^{(t)}_i = \theta^{(t-1)} - \theta^{(t)}_i$ is crucial. It means the server does not need to know about the inner optimizer's state (AdamW's momentum buffers, adaptive learning rates, weight decay schedule). The worker condenses $H$ local optimization steps into a single direction vector that represents the net change in parameter space. This is what makes the communication efficient—the server receives one update per $H$ local steps rather than $H$ gradients—and it is what makes the outer optimizer's job different from standard SGD: it is optimizing over pseudo-gradients that already incorporate adaptive inner optimization, not over raw stochastic gradients.

The synchronization barrier problem. The critical line in Algorithm 1 is line 11: the server computes $\Delta^{(t)}$ only after all workers have finished. This is the synchronization barrier. If worker A finishes its $H$ steps in 10 seconds and worker B takes 100 seconds (due to slower hardware or larger data shard), worker A sits idle for 90 seconds. The total wall-clock time per outer step equals the time of the slowest worker. This is the "straggler effect" that asynchronous training aims to eliminate.

---

The Asynchronous Local-SGD Framework

The paper's asynchronous framework (Section 3) removes the synchronization barrier from Algorithm 1. Workers no longer wait for each other; instead, the server updates the global parameters as soon as a worker's pseudo-gradient arrives. This introduces several design challenges that the synchronous version does not face, and the paper's framework addresses each through specific mechanisms.

Data shard sampling (Equation 2) — balancing learning progress. In synchronous training, every worker trains on its assigned shard once per outer step, so each shard naturally receives equal optimization attention. In asynchronous training, faster workers complete more training tasks per unit time, so if workers were statically assigned to shards, the shards assigned to fast workers would be trained on disproportionately often. To balance this, the paper introduces a dynamic sampling mechanism.

Define $n_i$ as the number of data points from shard $\mathcal{D}_i$ that have been processed so far (across all workers and all training tasks). Whenever a worker is ready to start a new local optimization round, the server samples a shard $i_{\text{sampled}}$ according to:

$i_{\text{sampled}} \sim p, \quad \text{where } p_i \propto \max\left( \frac{|\mathcal{D}_i|}{\sum_j |\mathcal{D}_j|} - \frac{n_i}{\sum_j n_j}, 0 \right)$

where $|\mathcal{D}_i|$ is the total size of shard $i$, $\sum_j |\mathcal{D}_j|$ is the total dataset size, $n_i$ is the cumulative number of processed points from shard $i$, and $\sum_j n_j$ is the total processed points across all shards.

What it computes: the term $\frac{|\mathcal{D}_i|}{\sum_j |\mathcal{D}_j|}$ is the target fraction of training that should ideally be spent on shard $i$ (its proportional size). The term $\frac{n_i}{\sum_j n_j}$ is the actual fraction of training that has been spent on shard $i$ so far. The difference is the deficit—how far behind shard $i$ is relative to its target. The $\max(\cdot, 0)$ ensures that shards that are already over-sampled (deficit ≤ 0) receive zero probability of being selected. The $\propto$ means the probabilities are normalized to sum to 1 over all shards with positive deficit.

Why this form: the purpose is to keep all shards progressing at approximately the same rate relative to their size, even though fast workers complete more tasks. If a fast worker keeps getting assigned the same shard, that shard's $n_i$ grows quickly, its deficit shrinks, and eventually it drops to zero—forcing subsequent tasks to sample other under-trained shards. This is a work-stealing approach adapted to the distributed optimization context: workers naturally gravitate toward the shards that need the most training. The $\max(\cdot, 0)$ is essential because without it, the system would assign negative probability (which is meaningless) or sample shards that are already ahead, exacerbating imbalance.

Relation to federated learning: the paper emphasizes that unlike federated learning, where each device is physically attached to its own data and cannot choose what to train on, "in distributed optimization, the user has the right to choose which data shard is assigned to which worker, even dynamically." This is a key distinction that makes the sampling strategy feasible—the central server controls data allocation.

Learning rate scheduling (Equation 3) — per-shard schedules. In synchronous training, all workers proceed through the same number of outer steps, so a single global learning rate schedule suffices. In asynchronous training, different shards may be trained on different numbers of total steps because faster workers may process some shards more quickly than slow workers process others. The paper addresses this by giving each shard its own learning rate schedule, defined as a function of the current training iteration $t$ for that specific shard:

$\eta_t = \begin{cases} t \eta_{\text{max}} / t_{\text{warmup}} & t < t_{\text{warmup}} \\ \eta_{\text{min}} + 0.5(\eta_{\text{max}} - \eta_{\text{min}}) \left(1 + \cos\left( \frac{t - t_{\text{warmup}}}{T - t_{\text{warmup}}} \pi \right) \right) & t \geq t_{\text{warmup}} \end{cases}$

where $\eta_t$ is the inner learning rate for that shard at local step $t$, $\eta_{\text{max}}$ is the peak learning rate, $\eta_{\text{min}}$ is the minimum learning rate (set to a small positive value to ensure continued progress), $t_{\text{warmup}}$ is the number of warmup steps, $T$ is the target total training iterations for each shard, and $\cos$ is the cosine function producing a smooth decay from $\eta_{\text{max}}$ to $\eta_{\text{min}}$.

What it computes: the learning rate follows a standard linear warmup (linearly increasing from 0 to $\eta_{\text{max}}$ over the first $t_{\text{warmup}}$ steps) followed by a cosine decay (smoothly decreasing from $\eta_{\text{max}}$ to $\eta_{\text{min}}$ over the remaining $T - t_{\text{warmup}}$ steps). The $0.5(\eta_{\text{max}} - \eta_{\text{min}})(1 + \cos(\cdot))$ term maps the argument to a value between $\eta_{\text{max}}$ and $\eta_{\text{min}}$, with the cosine producing the smooth transition.

Why this form: the cosine schedule is standard in language model training and is adopted from DiLoCo. The novel adaptation for asynchrony is that each shard maintains its own $t$ counter. The paper notes that because asynchronous training may conclude with different final iteration counts ($t_{\text{end}}$) for each shard, and $t_{\text{end}}$ cannot be predetermined, $\eta_{\text{min}}$ is set to a small positive value rather than zero—this ensures learning does not completely stop if a shard exceeds $T$ iterations. Additionally, the ratio $\frac{t - t_{\text{warmup}}}{T - t_{\text{warmup}}}$ is explicitly clamped to $[0, 1]$ to prevent degenerate learning rates if $t$ exceeds $T$ (the cosine would otherwise receive arguments greater than $\pi$, causing the learning rate to increase again, which is undesirable).

Grace period for model synchronization — trading staleness for latency. The paper observes that in asynchronous training, if two workers finish their tasks at nearly the same time, it may be beneficial for the first finisher to briefly wait so that both workers' updates can be aggregated before either downloads the new model. This reduces staleness (the server processes both updates together rather than sequentially) at the cost of a short idle period. The grace period $\tau_{\text{grace}}$ is the configurable maximum wait time. Figure 3 illustrates the mechanism: worker A finishes first and begins its grace period; worker B finishes before the grace period expires; the server synchronizes both updates (applying them to the global model) and then both A and B download the same updated model to start their next tasks. Worker C, which finishes after the grace period, will synchronize its update later and start its next task with a different (more recent) model. If no worker finishes within A's grace period, A proceeds immediately with the current server model.

Why include a grace period at all? In a pure asynchronous system with no grace period ($\tau_{\text{grace}} = 0$), each worker's update is applied sequentially as it arrives. This maximizes the frequency of parameter updates but also maximizes staleness: the model that worker A downloads for its next task will not incorporate worker B's nearly-simultaneous update. By allowing a brief wait (the paper does not specify exact values, but the concept is validated through the algorithm design), the system can batch updates that happen to arrive close together, getting some of the benefit of synchronous aggregation without the full cost of waiting for the slowest worker.

Why not always wait? If $\tau_{\text{grace}}$ is too large, the system degenerates toward synchronous behavior, with fast workers repeatedly waiting for slow ones. The grace period must be set small enough that it only captures nearly-simultaneous completions. The paper's task scheduling algorithm implements this as a soft synchronization mechanism that operates opportunistically rather than mandatorily.

Task scheduling algorithm (Algorithm 2) — the central control loop. The entire asynchronous training process is orchestrated by the task scheduling algorithm, which runs on the central server and manages the continuous cycle of receiving updates, applying them, and dispatching new training tasks. The algorithm uses wall-clock time $\tau$ for synchronization decisions and a counter $t_{\text{local}}$ for tracking total local updates performed.

Initialization (lines 1-6): the server initializes the model $\theta \leftarrow \theta^{(0)}$, creates $k$ workers, and launches the initial training task for all workers by calling train(W, θ). The function train() (detailed in Algorithm 4 in the appendix) samples data shards using Equation 2, determines local step counts using DyLU (Equation 6), sets per-shard learning rate schedules (Equation 3), and starts each worker training on its assigned shard with the current server model. The grace period start time $\tau_{\text{sync}}$ is initialized to $\infty$, meaning no grace period is active initially.

Main loop (lines 7-22): the loop continues until the total local updates $t_{\text{local}}$ reaches the maximum $t_{\text{max}}$ (88,000 in the paper's experiments). Each iteration attempts to retrieve a completed worker via get_worker() (Algorithm 5 in the appendix). The function get_worker() checks if any worker has finished its training task, and if so, whether the earliest finisher's completion time is within the grace period ($w.\text{completed\_time} \leq \tau_{\text{sync}} + \tau_{\text{grace}}$). If a qualified worker exists, get_worker() returns it; otherwise, it returns null.

Case 1 — worker found within grace period (lines 10-15): the server synchronizes the worker's update with the global model by calling sync(θ, w.update). The function sync() implements the outer optimizer update—either standard Nesterov momentum (as in DiLoCo) or the proposed Delayed Nesterov (Algorithm 3). The grace period start time $\tau_{\text{sync}}$ is updated to $\min(\tau_{\text{sync}}, w.\text{completed\_time})$, which marks the beginning of the grace period if this is the first worker in the current batch. The worker is added to W_completed, a list of workers whose updates have been processed and who are waiting for new tasks, and $t_{\text{local}}$ is incremented by the number of local steps that worker performed.

Case 2 — no eligible worker (lines 16-20): this occurs either when all workers are still training, or when a grace period has expired without any additional worker finishing. In this case, all workers currently in W_completed (those whose updates have been synchronized) are dispatched to new training tasks via train(W_completed, θ). They receive the current global model $\theta$, which incorporates all updates processed so far. The grace period is reset ($\tau_{\text{sync}} = \infty$) and the completed workers list is cleared. The loop then continues, waiting for the next worker to finish.

Deterministic simulation for reproducibility. A practically important design choice: "for the sake of reproducibility of research, we implement a deterministic version of Algorithm 2 with faked training time based on real-world device statistics." This means the paper's experiments do not run on actual heterogeneous hardware with variable network latency; instead, they simulate the wall-clock behavior using measured device speed profiles (Figure 4 shows steps per second for each device). This allows controlled experiments where device speeds are fixed and reproducible, which is essential for scientific comparison but means the reported wall-clock improvements assume the simulation accurately reflects real-world conditions. The paper validates the simulation framework by confirming that "synchronous updates using the asynchronous framework" produce correct results.

---

Diagnostic Experiments: Why Asynchronous Local-SGD Fails

Before proposing solutions, the paper conducts a systematic diagnostic investigation (Section 4) to isolate the cause of asynchronous Local-SGD's poor performance relative to synchronous DiLoCo. These experiments are the empirical foundation for both proposed methods.

Optimizer combination sweep (Figure 5) — AdamW+Nesterov remains best. The first experiment tests all combinations of two inner optimizers (SGD, AdamW) and three outer optimizers (SGD, Adam, Nesterov) on a 20M parameter model with 4 heterogeneous workers (device speeds shown in Figure 4) for 64,000 steps per worker (256,000 total local steps). The inner steps $H$ is fixed at 50 across all workers. The model was pretrained for 24,000 steps with standard Adam before distributed fine-tuning begins. Results show:

• AdamW+Nesterov achieves the best final perplexity among all asynchronous combinations, mirroring the synchronous DiLoCo finding. This validates that the optimizer pairing itself is not the problem—the issue is in how it is applied asynchronously.
• AdamW+Adam performs worse than AdamW+Nesterov. The paper hypothesizes this is because Adam's division by a running estimate of gradient variance (the normalization effect) is counterproductive when applied to pseudo-gradients, which "tend to be larger than true gradients" due to accumulating $H$ local steps.
• When AdamW is the inner optimizer, SGD, SGD Momentum, and Nesterov show comparable performance as outer optimizers, but Nesterov "stabilizes the learning curve and slightly improves final performance."

The momentum comparison experiment (Figure 6) — the key diagnostic. This experiment directly compares synchronous versus asynchronous DiLoCo with two outer optimizer variants: SGD (no momentum) and Nesterov (with momentum). The setup is identical to the previous experiment. The results reveal the paper's central insight:

• AdamW+SGD (no outer momentum): asynchronous training outperforms synchronous training. The asynchronous curve sits below the synchronous curve throughout training. Without a momentum term to mismanage, the increased update frequency from asynchrony (the server updates parameters more often because it doesn't wait for all workers) translates directly to faster convergence.
• AdamW+Nesterov (with outer momentum): synchronous training significantly outperforms asynchronous. The asynchronous curve sits above the synchronous curve, and the gap is substantial—the asynchronous version never catches up.

Why this pattern is diagnostic: it isolates momentum as the necessary and sufficient condition for the performance gap. When momentum is absent, asynchrony helps (more updates = better). When momentum is present, asynchrony hurts (more updates with stale gradients = worse). The synchronous setting benefits from momentum because all pseudo-gradients are aggregated before the momentum update is applied, so the momentum term always operates on the average gradient direction. In the asynchronous setting, the momentum update is applied sequentially to individual (stale) pseudo-gradients, causing the momentum state to drift.

The sequential application analysis (Equation 5) — why momentum breaks. The paper provides an analytical explanation by examining what happens when $k=4$ workers produce identical pseudo-gradients $g$ at the same time (homogeneous devices, simultaneous completion) but the server applies them sequentially using Nesterov momentum. The standard Nesterov update (from Equation 4, with notation adapted to use $t$ for the server step):

$m_{t+1} = \beta m_t + g_t$ $\theta_{t+1} = \theta_t - \epsilon \left( \beta^2 m_t + (1 + \beta) g_t \right)$

where $m_t$ is the momentum buffer, $\beta \in (0, 1)$ is the decay factor, $g_t$ is the current pseudo-gradient, $\epsilon$ is the outer learning rate, and $\theta_t$ is the global parameter.

When four identical gradients $g$ are applied simultaneously (synchronous case with all workers returning $g$), the update is equivalent to a single step with gradient $4g$:

$m_{t+1} = \beta m_t + 4g$ $\theta_{t+1} = \theta_t - \epsilon \left( \beta^2 m_t + 4(1 + \beta) g \right)$

When those same four gradients are applied sequentially (asynchronous case, one after another), the paper derives the cumulative effect over four sequential Nesterov updates:

$m_{t+1} = \beta^4 m_t + (1 + \beta + \beta^2 + \beta^3)g$ $\theta_{t+1} = \theta_t - \epsilon \left( (4 + 4\beta + 3\beta^2 + 2\beta^3 + \beta^4)g + \beta^2(1 + \beta + \beta^2 + \beta^3)m_t \right)$

What this derivation reveals: the sequential application produces a different ratio between the momentum component and the gradient component than the simultaneous application. The momentum term $m_t$ decays more rapidly (multiplied by $\beta^4$ in sequential vs. $\beta$ in simultaneous, after accounting for the gradient scaling) and the parameter change coefficient on $g$ is amplified relative to the momentum coefficient. This imbalance means momentum and gradient descent are pulling in disproportionate amounts, and crucially, "this imbalance cannot be simply rectified by reducing the learning rate" because the learning rate would scale both components equally, not fix their relative weights.

The homogeneity experiment (Figure 7) — staleness is inherent, not just from speed differences. This experiment runs asynchronous DiLoCo with homogeneous devices (all workers operating at the same speed). If the problem were solely caused by slow workers producing very stale gradients, homogeneous devices should eliminate the issue. The result: "even with homogeneity among workers, asynchronous DiLoCo significantly lags behind its synchronous counterpart." This demonstrates that the staleness problem is inherent to sequential application, not just to variable worker speeds. Even when all workers finish simultaneously, the server must still process their updates one at a time (or in small batches within the grace period), and the Nesterov momentum update is sensitive to this ordering.

Why this matters for solution design: it means that any fix must address the sequential application of momentum itself, not just reduce gradient staleness due to slow workers. The Delayed Nesterov approach (fixing how momentum is applied) and Dynamic Local Updates (reducing how sequential the applications need to be) address both root causes.

Existing fixes and their inadequacy (Figure 8). The paper tests four methods from the asynchronous optimization literature:

1. Polynomial discounting (Async. DiLoCo + Poly): multiply each pseudo-gradient by $(1 + \text{staleness})^{-0.5}$, where staleness is measured as the number of server updates that have occurred since the worker started its local training. This downweights old updates. Result: "marginal benefits"—the curve nearly overlaps the baseline.

2. Threshold-based discard (Async. DiLoCo + PolyThres): discard any update with staleness greater than 10. Result: similarly marginal.

3. Delay compensation (Async. DiLoCo + Delay Comp.): approximate the true pseudo-gradient at current parameters by Taylor expansion: $g(\theta_t) \approx g(\theta_{t'}) + \lambda g(\theta_{t'}) \odot g(\theta_{t'}) (\theta_t - \theta_{t'})$, where $t' < t$ is the time when the worker started its task, $\theta_t$ is the current parameter, $g(\theta_{t'})$ is the stale pseudo-gradient, $\odot$ is element-wise multiplication, and $\lambda$ is a hyperparameter controlling the diagonal Hessian approximation. This technique assumes gradients change smoothly over small parameter differences. Result: does not work. The paper explicitly notes this "points out the difference between asynchronous SGD and asynchronous Local-SGD"—in standard SGD, gradients change smoothly and the Taylor approximation is reasonable; in Local-SGD, pseudo-gradients are already accumulated over $H$ local steps and may not behave as smoothly.

4. Async. Buffer (based on FedBuff, Nguyen et al., 2022): accumulate pseudo-gradients from multiple workers in a buffer, and only apply a Nesterov update after collecting a specified number of updates (rather than after every individual update). Between full buffer updates, do nothing (the server model stays unchanged). Result: "significantly closes the gap between sync. and async. training" but "introduces instability in early stage of training" (visible as oscillations in Figure 8). Critically, "none of the methods match the performance of the synchronous DiLoCo baseline."

The Async. Buffer result is the most informative for the proposed solution. It shows that batching pseudo-gradients before applying momentum helps substantially (reducing the mismatch between simultaneous and sequential application analyzed in Equation 5), but doing pure buffering with no intermediate updates introduces instability. The Delayed Nesterov method can be understood as a refined version of this insight that addresses the instability.

---

Delayed Nesterov (DN) — Restructuring Outer Momentum for Asynchrony

The Delayed Nesterov update (Algorithm 3) is the paper's primary technical contribution for the outer optimizer. It modifies how the server processes incoming pseudo-gradients to mimic the effect of synchronous Nesterov momentum while operating fully asynchronously. The core idea is to separate the momentum update from the gradient update temporally: use standard SGD (no momentum) for most server steps, and only apply the full Nesterov momentum update every $N$ server steps, using the pseudo-gradients accumulated over those $N$ steps.

Algorithm structure. The server maintains state across updates: the current model parameters $\theta_t$, a momentum buffer $m_t$ (initialized to zero), and an aggregated pseudo-gradient buffer $\Delta$ (initialized to zero). The server also maintains configurable parameters: the momentum decay $\beta \in (0, 1)$, the buffer size $N$ (number of pseudo-gradients to accumulate before a momentum update), and a momentum activation parameter $c \in [0, 1/N]$ that controls how much momentum leaks into the intermediate SGD steps.

Operation. Each time the server receives a pseudo-gradient $g_t$ from any worker (the sync step in Algorithm 2 calls Algorithm 3):

1. Accumulate: the pseudo-gradient is added to the buffer: $\Delta \leftarrow \Delta + g_t$. This buffer accumulates the sum of all pseudo-gradients received since the last momentum update.

2. Check if buffer is full: if $(t + 1) \bmod N == 0$ (the buffer has accumulated $N$ pseudo-gradients), perform a full Nesterov momentum update:

   • Update momentum: $m_{t+1} \leftarrow \beta m_t + \Delta / N$
   • Apply Nesterov update to parameters:

   $\theta_{t+1} \leftarrow \theta_t - \epsilon \left( (1 - cN + c)\beta m_{t+1} + g_t / N \right)$

   • Reset buffer: $\Delta \leftarrow 0$

3. If buffer is not full ($(t + 1) \bmod N \neq 0$), perform a partial update:

   • Freeze momentum: $m_{t+1} \leftarrow m_t$ (the momentum buffer does not change)
   • Apply SGD-like update to parameters:

   $\theta_{t+1} \leftarrow \theta_t - \epsilon \left( c\beta m_{t+1} + g_t / N \right)$

4. Increment counter: $t \leftarrow t + 1$

Step counter: $t$ is incremented after every server update.

What the full Nesterov update computes (when the buffer is full): the term $\Delta / N$ is the average of the $N$ accumulated pseudo-gradients. The momentum update $m_{t+1} \leftarrow \beta m_t + \Delta / N$ is the standard Nesterov momentum recurrence applied to the average pseudo-gradient. The parameter update applies a Nesterov-style correction (the $\beta m_{t+1}$ lookahead term plus the current gradient) with a weighting controlled by $c$. When $c = 0$ (the default), the update simplifies to:

$\theta_{t+1} \leftarrow \theta_t - \epsilon \left( \beta m_{t+1} + g_t / N \right)$

This is a standard SGD step with learning rate $\epsilon / N$ on the current pseudo-gradient, plus a momentum correction $\epsilon \beta m_{t+1}$ from the updated momentum buffer. The $1/N$ scaling on $g_t$ ensures that each of the $N$ accumulated pseudo-gradients contributes equally to the parameter update on average.

What the intermediate SGD-like update computes (when the buffer is not full): the momentum is held fixed ($m_{t+1} = m_t$), so the parameter update consists of two terms: a small fraction of the old momentum ($c\beta m_{t+1}$, which is zero when $c = 0$) plus the current pseudo-gradient scaled by $1/N$ ($g_t / N$). When $c = 0$, this is pure SGD with a reduced learning rate—each incoming pseudo-gradient moves the parameters slightly in its direction, but the momentum state is not updated.

The parameter $c$ — controlling momentum leakage. The parameter $c \in [0, 1/N]$ controls how much of the old momentum state leaks into the parameter updates between full Nesterov updates. When $c = 0$, the intermediate steps are pure SGD ($\theta_{t+1} = \theta_t - \epsilon g_t / N$), and momentum only enters at the full Nesterov steps. When $c = 1/N$, the momentum term $\beta m_{t+1}$ is evenly distributed across all $N$ steps, so each step gets an equal share of momentum influence. The default setting is $c = 0$ based on empirical results showing "no significant difference between $c=0$ and $c=0.1$, indicating that adding slight momentum at intermediate steps does not help too much" (Table 4).

Why $g_t / N$ in the intermediate steps rather than $g_t$ alone? Without the $1/N$ scaling, each intermediate step would contribute the full pseudo-gradient, and the total parameter movement over $N$ steps would be approximately $N$ times larger than a synchronous step with the average gradient. The $1/N$ scaling normalizes the per-step contribution so that the cumulative effect of $N$ sequential SGD steps approximates the effect of one step with the average gradient. This is directly motivated by the analysis in Equation 5, which showed that sequential application amplifies the parameter change relative to the momentum decay.

Why delay the momentum update at all? The key insight from Equation 5 is that the sequential Nesterov update produces a different ratio of momentum to gradient terms than the simultaneous update. By delaying the momentum update (only recomputing $m_{t+1}$ every $N$ steps) and using pure SGD in between, Delayed Nesterov achieves two effects:

1. The momentum update operates on the average gradient over $N$ steps (via $\Delta / N$), just as synchronous Nesterov operates on the average over $k$ workers. This eliminates the sequential distortion analyzed in Equation 5.

2. The parameter updates occur continuously (via the intermediate SGD steps), unlike pure Async. Buffer which does nothing between buffer fills. This prevents the "instability in early stage of training" observed with Async. Buffer (Figure 8) because the model parameters are constantly being updated with new information, just not with momentum acceleration.

Relationship to Async. Buffer. The Delayed Nesterov can be viewed as a generalization of Async. Buffer. When $c = 0$ and the intermediate steps use SGD, DN applies continuous parameter updates with occasional momentum resynchronization. Async. Buffer corresponds to the degenerate case where intermediate steps are skipped entirely (equivalently, $\epsilon = 0$ between buffer fills) and only the full Nesterov update is applied. The paper's finding that Async. Buffer is unstable while DN is stable suggests that the continuous SGD updates provide a regularizing effect—they keep the parameters moving in the right general direction even when the momentum state is out of date.

Why the buffer size $N$ matters. The paper explores $N \in \{4, 8, 16\}$ as hyperparameters (Table 5 in the appendix). Larger $N$ means the server waits longer between momentum updates, accumulating more pseudo-gradients and producing a better average (closer to the synchronous case), but also means the momentum state is updated less frequently and may be more stale when it is applied. Smaller $N$ means the momentum is updated more frequently but with noisier averages. The paper does not provide a detailed ablation of $N$ in the main text, but the chosen values align with the number of workers $k=4$, suggesting $N$ is set roughly equal to $k$ so that one buffer fill corresponds approximately to one round of synchronous aggregation.

What $N$ means physically: in a system with $k=4$ workers and $N=4$, the server performs three SGD updates (on the first three pseudo-gradients that arrive) and one full Nesterov update (when the fourth arrives), then repeats. The momentum state $m$ is updated exactly once per "round" of all $k$ workers, mirroring the synchronous Nesterov update frequency, while the parameters are updated four times per round (three SGD steps + one Nesterov step), providing the benefit of asynchronous update frequency without the momentum distortion.

---

Dynamic Local Updates (DyLU) — Aligning Worker Completion Times

While Delayed Nesterov addresses the optimizer's response to stale pseudo-gradients, Dynamic Local Updates (DyLU) addresses a complementary problem: reducing how stale the pseudo-gradients are in the first place. The idea is simple: if slow workers perform fewer local steps per task, they finish faster, and their updates arrive at the server with less delay relative to when they started training.

The mechanism (Equation 6). Let $v(w)$ be the measured training speed of worker $w$, in steps per second (as shown in Figure 4 for the paper's device pool). Let $\max_{w' \in \mathcal{W}} v(w')$ be the speed of the fastest worker. Then worker $w$'s number of local training steps per task is:

$w.\text{steps} = \left\lfloor \frac{v(w)}{\max_{w' \in \mathcal{W}} v(w')} H \right\rfloor$

where $H$ is the number of local steps that the fastest worker performs per task (set to 50 by default in the paper's experiments), and $\lfloor \cdot \rfloor$ denotes the floor function (largest integer not greater than the argument).

What it computes: the fastest worker performs $H$ steps (since $v(w) / \max v = 1$, so $w.\text{steps} = \lfloor H \rfloor = H$). A worker operating at half the speed of the fastest performs $\lfloor 0.5 H \rfloor$ steps. A worker operating at one-third speed performs $\lfloor H/3 \rfloor$ steps, and so on. The ratio $v(w) / \max v(w')$ is the worker's relative speed. The floor function ensures an integer step count.

Why this helps: the completion time for worker $w$ is approximately $w.\text{steps} / v(w) = \lfloor \frac{v(w)}{\max v} H \rfloor / v(w) \approx H / \max v(w')$, which is roughly constant across workers—it equals the fastest worker's completion time, up to the discretization error from the floor function. Slower workers take fewer steps but spend the same amount of wall-clock time per task. This means workers finish their tasks at approximately the same time, reducing the window during which the server receives sequential updates with varying staleness.

Why the floor function? The number of local steps must be an integer. The floor (rather than round) is a conservative choice: slower workers get slightly fewer steps than the ideal proportional allocation, which means they finish slightly faster than the fastest worker (not slower), preventing them from becoming stragglers. If round were used instead, some workers might get more steps than their speed warrants, finishing slightly after the fastest worker and introducing mild asynchrony.

Interaction with the grace period. DyLU makes workers finish at approximately the same time, but not exactly (due to the floor discretization, measurement noise in $v(w)$, and variability in per-batch computation time). The grace period absorbs these small timing differences: if all workers finish within $\tau_{\text{grace}}$ of each other, the server effectively processes their updates in batch (as described in the task scheduling section), and the asynchronous system temporarily behaves like a synchronous one for that round. The combination of DyLU (making completion times roughly equal) and the grace period (absorbing residual variance) is what makes asynchronous training approach synchronous behavior.

Estimating worker speeds. The paper states: "we implicitly assume the device speeds are known a priori. If this is not the case, it is straightforward to estimate the device speed based on empirical observations." In practice, $v(w)$ can be measured from the first few training tasks a worker performs—record the wall-clock time and number of steps, compute steps/second, and use a running average. The paper's deterministic simulation framework makes this measurement trivial (the speeds are pre-configured), but the method is designed to work with online estimation in real deployments.

Why not just give all workers the same number of steps? If all workers perform the same $H$ steps, the completion time for a slow worker is $H / v_{\text{slow}}$, which is much larger than $H / v_{\text{fast}}$. The fast worker finishes quickly and its update is applied to the server. By the time the slow worker finishes, the server model has moved (via updates from the fast worker and possibly others), so the slow worker's pseudo-gradient is stale—it was computed with respect to an older version of the parameters. DyLU reduces this staleness by making the slow worker finish sooner (with a shorter training task), so its pseudo-gradient is computed with respect to a parameter version that is closer to the current server state. The tradeoff is that the slow worker contributes less information per task (fewer local steps mean a noisier pseudo-gradient), but the paper's results show that the reduced staleness outweighs the increased noise for language model training.

Why the fastest worker's steps $H$ remains configurable. $H$ is the communication frequency parameter from synchronous DiLoCo. In the asynchronous setting with DyLU, $H$ controls the maximum number of local steps any worker performs. The paper's main experiments use $H = 50$, with sweeps over $\{50, 100, 150\}$ (Table 5). Larger $H$ means fewer server updates per total local step (more computation per communication), which is beneficial when communication is expensive but increases the staleness of each pseudo-gradient. DyLU partially mitigates the staleness by reducing steps for slow workers, but $H$ remains a tunable parameter.

---

The Combined System: DN + DyLU

The paper presents Delayed Nesterov and Dynamic Local Updates as complementary techniques that address different aspects of the asynchrony problem:

• Delayed Nesterov fixes the optimizer's response to stale gradients. Even when gradients are stale, the optimizer applies them in a way that mimics synchronous momentum behavior, preventing the sequential distortion analyzed in Equation 5.

• Dynamic Local Updates reduces the amount of staleness in the first place. By making workers finish at similar times, DyLU ensures that pseudo-gradients are computed from more recent parameter versions, so the server is working with better information.

How they compose. In the full system, when a worker finishes training (with its DyLU-determined step count), its pseudo-gradient is sent to the server and processed by the Delayed Nesterov update (Algorithm 3). The DN algorithm does not need to know that the worker used DyLU—it treats each incoming pseudo-gradient identically, regardless of how many local steps produced it. This modularity is important: DyLU changes the distribution of pseudo-gradient arrival times and stale-nesses, while DN changes how those pseudo-gradients are consumed by the optimizer.

Why both are needed. The paper's Figure 2 shows the combined DN+DyLU system achieving parity with synchronous DiLoCo in perplexity per update and outperforming it in wall-clock time. Without DyLU, even with DN, some workers would produce very stale gradients (slow workers doing full $H$ steps), and the buffer accumulation in DN would mix fresh and stale pseudo-gradients. Without DN, even with DyLU, the residual staleness from imperfect timing alignment would distort the Nesterov momentum update. The paper does not provide a formal ablation isolating each technique's contribution (both are always presented together in the main results), but the logical decomposition follows from the diagnostic experiments: DyLU addresses the "heterogeneous devices produce staleness" problem (Figure 10), and DN addresses the "sequential application distorts momentum even with homogeneous devices" problem (Figure 7).

Configuration for main experiments. The default experimental configuration throughout (unless noted in ablations) uses: $k = 4$ workers with "very" heterogeneous device speeds (bottom-right of Figure 10), inner optimizer AdamW, inner learning rate $0.1$, inner steps $H = 50$ for the fastest worker, batch size 128 (or 512 for some experiments, per Table 5), sequence length 256, weight decay 0.1, outer learning rate $0.7$ (chosen from sweeps over $\{0.03, 0.3, 0.1, 0.7\}$), Nesterov momentum $\beta$ (value not explicitly stated but defaults to standard Nesterov configuration), DN buffer size $N = 4$ (implied by the number of workers), DN parameter $c = 0$, grace period $\tau_{\text{grace}}$ (value not explicitly stated), total local updates $t_{\text{max}} = 88,000$, with 24,000 steps of pretraining before distributed fine-tuning begins. The C4 dataset is partitioned into $k$ shards of equal size, and the models are Chinchilla-style decoder-only transformers (configurations in Table 6).

4. Key Insights and Innovations

Innovation 1: Isolating Outer Momentum as the Single Point of Failure in Asynchronous Local-SGD

The most intellectually distinctive contribution of this paper is not a new algorithm but a diagnostic decomposition that pinpoints exactly why asynchronous Local-SGD fails for language model training. Prior work on asynchronous optimization treated the staleness problem generically—stale gradients are bad, so apply discounting, thresholding, or delay compensation to mitigate them. The paper's diagnostic experiments (Section 4, Figures 5-7) reveal that this framing is insufficiently specific for the Local-SGD setting. The failure is not about staleness in general but about a specific interaction: Nesterov momentum in the outer optimizer breaks when pseudo-gradients from multiple workers are applied sequentially rather than simultaneously.

The evidence for this claim is unusually crisp because the paper constructs a controlled comparison that isolates momentum as the necessary and sufficient condition. When the outer optimizer is switched from Nesterov to plain SGD (Figure 6), the performance ordering flips: asynchronous training, which was worse with Nesterov, becomes better than synchronous training. This is not merely a parameter sensitivity—it is a qualitative change in behavior that rules out staleness alone as the explanation. If staleness were the problem, removing momentum should not reverse the ordering; stale gradients would still be stale. The reversal demonstrates that momentum is the active ingredient in the degradation, not staleness per se.

The paper then deepens the diagnosis with a secondary experiment that is easy to overlook but is conceptually crucial: the homogeneous-device test (Figure 7). By running asynchronous DiLoCo with all workers operating at identical speeds, the authors eliminate variable-speed-induced staleness—all pseudo-gradients are computed from roughly the same parameter version because all workers start and finish simultaneously. Yet the performance gap persists. This demonstrates that sequential application itself, even of simultaneous gradients, distorts the momentum update sufficiently to degrade convergence. The analytical derivation in Equation 5 formalizes this: when four identical gradients are applied sequentially through Nesterov momentum, the resulting parameter change has a different ratio of momentum-to-gradient contribution than when those same gradients are averaged and applied once, and this ratio cannot be corrected by simply reducing the learning rate.

This diagnostic contribution is fundamental, not incremental, because it reframes the entire problem space. Before this work, one might have asked: "How do we make asynchronous training robust to stale gradients?" After this work, the question narrows to: "How do we apply momentum-based outer optimization when pseudo-gradients arrive sequentially?" This narrower framing enables targeted solutions (Delayed Nesterov, Dynamic Local Updates) that would not have been obvious under the generic staleness paradigm. It also explains why prior methods from the asynchronous optimization literature—polynomial discounting, thresholding, delay compensation (Zheng et al., 2017; Xie et al., 2019)—all failed (Figure 8): they were addressing the wrong problem. They treated staleness as an information-quality issue (the old gradient is a poor estimate of the current gradient), when the actual issue is a structural mismatch in how the momentum recurrence behaves under sequential versus batched application.

A comparison to prior diagnostic work is instructive. The asychronous SGD literature has long recognized that staleness degrades convergence, and the standard response has been to correct the gradient estimate (delay compensation) or reduce its influence (discounting). The DiLoCo paper (Douillard et al., 2023) established that AdamW+Nesterov is the optimal optimizer pairing for synchronous Local-SGD but did not study asynchrony. By showing that the same optimizer pairing that works best synchronously is also the one that most severely breaks asynchronously, the paper reveals a previously invisible tension: the very mechanism (outer momentum) that provides the strongest synchronous acceleration becomes the primary liability when synchronization is removed. This is a negative result with significant implications—it tells future researchers that better staleness compensation alone will not close the async-sync gap, because the problem is in the momentum recurrence itself.

Innovation 2: Delayed Nesterov as a Structural Solution to the Momentum Distortion Problem

The Delayed Nesterov update (Algorithm 3) is the paper's main constructive contribution, and its intellectual distinctiveness lies in when and how it applies momentum, not in any novel mathematical operator. The standard approaches to momentum in asynchronous optimization fall into two families: either apply momentum on every update and try to correct for staleness (delay compensation, polynomial discounting), or buffer gradients and apply momentum periodically to batches (FedBuff, Nguyen et al., 2022). The paper shows that neither approach suffices: the first family fails because the per-step momentum update is structurally mismatched to sequential application, and the second family (Async. Buffer in Figure 8) shows promise but introduces training instability because the model parameters remain frozen between momentum updates.

Delayed Nesterov represents a third design point that splits the difference: apply gradient updates continuously, but apply momentum updates only periodically. Between momentum updates, the server performs pure SGD steps with a reduced learning rate (each incoming pseudo-gradient scaled by $1/N$). When the buffer fills, a single Nesterov momentum update is applied using the average of the accumulated pseudo-gradients. This design can be understood as decomposing the Nesterov update into two operations that synchronous training performs simultaneously—gradient descent on the current pseudo-gradient and momentum-based acceleration—and applying them at different temporal frequencies. The gradient descent happens on every server step (keeping the parameters continuously updated), while the momentum acceleration happens every $N$ steps (keeping the momentum state aligned with the average gradient direction rather than individual stale gradients).

What makes this distinctive compared to prior buffering approaches is the interleaved SGD steps. FedBuff (Nguyen et al., 2022) accumulates pseudo-gradients in a buffer and applies the model update only when the buffer is full, leaving the model unchanged in between. This works for communication efficiency but creates training instability (visible in Figure 8) because the model receives information in large, infrequent bursts. Delayed Nesterov's continuous SGD steps provide a stabilizing effect: even when the momentum state is out of date, the parameters are nudged in the direction of incoming pseudo-gradients, preventing the large parameter jumps that cause instability. The $1/N$ scaling on the SGD steps ensures that the cumulative effect of $N$ sequential SGD updates approximates the effect of one step with the average gradient, preserving the overall step size.

The parameter $c$ (controlling momentum leakage into intermediate steps) adds theoretical nuance but, as the ablation in Table 4 shows, turns out to be empirically unimportant—$c=0$ performs similarly to $c=0.1$. This is itself an informative result: it suggests that the key benefit comes from decoupling momentum updates from gradient updates, not from fine-tuning the degree of momentum leakage. The fact that pure SGD ($c=0$) between momentum updates works as well as momentum-tinged SGD simplifies the method and strengthens the interpretation that the decoupling is the essential mechanism.

This contribution is incremental but impactful—it does not introduce fundamentally new optimization theory, but it identifies a specific structural flaw in how momentum interacts with sequential pseudo-gradients and proposes a targeted architectural fix. The fix is simple to implement (a buffer, a counter modulo $N$, and a conditional branch in the update rule) and modular (it plugs into the sync() function in Algorithm 2 without requiring changes to workers, data sampling, or learning rate scheduling).

Innovation 3: Dynamic Local Updates as a Systems-Level Complement to Optimization-Level Fixes

Where Delayed Nesterov addresses the optimizer's response to stale pseudo-gradients, Dynamic Local Updates addresses the staleness itself through a systems-level mechanism: make workers finish their tasks at approximately the same wall-clock time by giving slower workers proportionally fewer local steps. The intellectual contribution here is not the equation $w.\text{steps} = \lfloor v(w)/v_{\text{max}} \cdot H \rfloor$ (which is straightforward) but the recognition that local step count is a control variable for staleness in Local-SGD, and that varying it across workers can partially recover synchronous behavior without imposing a synchronization barrier.

In standard synchronous Local-SGD, all workers perform the same number of local steps ($H$), which is a natural choice when all workers must finish before the next round begins. In asynchronous Local-SGD, the constraint disappears—workers can perform different numbers of local steps with no structural penalty. Yet prior asynchronous Local-SGD work (including the FedBuff and related methods the paper benchmarks) typically kept $H$ fixed across workers, treating step count as a model hyperparameter rather than a per-worker tuning knob. The paper's insight is that step count heterogeneity can be leveraged, not just tolerated, to reduce staleness.

The mechanism is deceptively simple. A slow worker taking $H$ steps produces a pseudo-gradient that is stale by roughly $H/v_{\text{slow}}$ seconds (the time it takes to finish). A slow worker taking $H/2$ steps produces a pseudo-gradient that is stale by roughly half that time, and while the pseudo-gradient is noisier (fewer local steps), the noise-vs-staleness tradeoff apparently favors reduced staleness for language model training. The paper does not provide a theoretical analysis of this tradeoff, but the empirical results (Figures 2, 10, 11, 12) show consistent benefits.

What elevates this from an engineering trick to an intellectual contribution is its complementarity with the optimizer-level fix. Delayed Nesterov makes the outer optimizer robust to residual staleness; Dynamic Local Updates reduces the staleness that the optimizer needs to be robust to. Each technique addresses a different layer of the problem (scheduling vs. optimization), and together they achieve what neither could alone. This layered approach—systems scheduling to reduce the problem + optimization to handle the remainder—is a design pattern that likely generalizes beyond this specific setting.

The contribution is incremental but generalizable. The specific formula (Equation 6) depends on knowing or estimating worker speeds, and the paper's simulation framework makes this trivial, but in real deployments speed estimation introduces additional complexity. The broader principle—that local step count can be used as a control variable to manage the staleness-vs-information tradeoff in asynchronous Local-SGD—does not depend on the specific speed-proportional allocation. Future work could explore more sophisticated allocation policies (e.g., reinforcement learning over step counts, adapting to changing network conditions) without departing from this core insight.

Innovation 4: A Reproducible Empirical Framework Connecting Asynchronous Local-SGD Theory to Language Modeling Practice

While not presented as a primary contribution, the paper's experimental design embodies an important methodological innovation: the deterministic simulation of asynchronous training using real device speed profiles. Instead of deploying on actual heterogeneous hardware with all the variability and non-reproducibility that entails, the paper "implements a deterministic version of Algorithm 2 with faked training time based on real-world device statistics." This allows controlled experimentation where device speeds are fixed, the order of pseudo-gradient arrivals is deterministic (given the speed profile), and results are fully reproducible.

This matters because asynchronous training experiments are notoriously difficult to compare across papers. Real hardware introduces uncontrolled variables—network latency jitter, transient load on shared machines, thermal throttling—that can change results between runs. By abstracting the timing behavior into a deterministic simulation (while using real measured device speeds as input), the paper creates a reproducible experimental environment that future work can replicate exactly. The validation that synchronous updates through the asynchronous framework produce correct results confirms the simulation's fidelity for the timing aspects that matter.

This methodological choice also enables the paper's most convincing diagnostic result: the homogeneity experiment (Figure 7). On real hardware, it is nearly impossible to guarantee that all devices operate at exactly the same speed—even identical GPU models in the same datacenter exhibit variance. The simulation framework allows the authors to test the extreme case of perfect homogeneity (all workers finish simultaneously, zero variability), which isolates sequential application as the cause of the momentum degradation. This experiment would be infeasible or at least highly approximate on real hardware.

The contribution is methodological but significant for the subfield. As research on asynchronous Local-SGD for language modeling grows (the paper explicitly calls for more work on this topic in Section 9), having a reproducible experimental standard will be valuable. The authors release a Colab notebook with a minimal toy example (Section 6, Figure 9) that replicates the core optimization challenge on a mixture of Gaussians classification task, further lowering the barrier to entry for researchers wanting to study the momentum-asynchrony interaction.

5. Experimental Analysis

Evaluation Methodology

Dataset and task. All experiments use the C4 dataset (Raffel et al., 2020), a large corpus derived from Common Crawl web text, for next-token prediction language modeling. The choice of C4 over more curated benchmarks (e.g., WikiText, BooksCorpus) reflects the paper's focus on realistic large-scale training conditions—C4 is diverse, noisy, and representative of the data distributions used in production language model training. The total number of training steps is set to 88,000 for all models, with the first 24,000 steps performed as standard single-machine pretraining (without any distributed training methods). This two-phase setup mirrors the "post Local-SGD" paradigm (Lin et al., 2020), where Local-SGD is applied during fine-tuning rather than from random initialization. The paper cites prior work showing Local-SGD works better in fine-tuning than from scratch (Lin et al., 2018) while noting that DiLoCo successfully applied it from scratch (Douillard et al., 2023). The 24,000-step pretraining provides a common starting point for all methods: a partially trained model with perplexity around 61.64 for the 20M parameter variant (Table 1).

Models. Three transformer decoder-only architectures are evaluated, all following the Chinchilla design principles (Hoffmann et al., 2022):

The 60M model uses only 3 layers (unusually shallow), which may affect how optimization dynamics manifest; the paper does not discuss this architectural choice. The models span roughly a 7.5× range in parameter count, from 20M to 150M, which is sufficient to test whether the proposed methods scale with model size but falls well below the billion-parameter scale where communication bottlenecks dominate training. The paper acknowledges this implicitly by not claiming large-model results—the contributions are about optimization dynamics, not about scaling to frontier model sizes.

Metrics. The primary metric is perplexity on the C4 validation set, reported as a function of two independent variables:

• Perplexity per update: measured against $t_{\text{local}}$, the cumulative number of local training updates performed across all workers. This metric answers: "for a given amount of total computation, which method achieves the best model quality?" Since all methods use the same inner optimizer (AdamW) with the same per-step cost, comparing at equal $t_{\text{local}}$ is a fair computation-matched comparison.
• Perplexity per wall-clock time: measured against simulated elapsed time based on device speed profiles. This metric answers: "in a real deployment with heterogeneous hardware, which method achieves the best model quality fastest?" This is the metric where asynchronous methods are expected to excel because they eliminate straggler idle time.

The dual metric reporting is crucial for evaluating asynchronous methods fairly. A naive comparison that only looks at perplexity per update would favor synchronous methods because they achieve better perplexity at the same number of local steps (Figure 2, left panel of the perplexity-vs-updates plot). The wall-clock comparison (Figure 2, right panel) reveals the asynchronous method's advantage: it achieves the same perplexity faster because no worker sits idle.

Device heterogeneity simulation. The paper defines four levels of worker heterogeneity, visualized in Figure 10: "no" (all workers at identical speed), "slight" (minor speed differences), "moderate," and "very" (the largest speed gap, corresponding to the device speeds shown in Figure 4). The "very" heterogeneous setting is the default for main experiments. In the deterministic simulation, each worker's training time per step is drawn from its speed profile, and the task scheduler (Algorithm 2) deterministically processes events based on these timed completions.

Hyperparameter selection. Table 5 (appendix) lists the range of hyperparameters explored and the chosen values for main experiments. Key choices:

• Inner learning rate: 0.1 (swept: not explicitly stated as swept, listed as a single value)
• Final inner learning rate: 0.0, 0.000001, or 0.0002 (the small positive values implement the "set to a small positive value" from Section 3's learning rate scheduling discussion; 0.0 is used for ablations)
• Warmup steps: 0 or 1,000 (chosen: 1,000)
• Weight decay: 0.1 (standard for AdamW)
• Batch size: 128 or 512 (chosen: 128 for main experiments, with 512 explored in ablations)
• Sequence length: 256
• Outer learning rate: swept over {0.03, 0.3, 0.1, 0.7}; chosen: 0.7
• Communication frequency $H$: swept over {50, 100, 150}; chosen: 50
• DN buffer size $N$: swept over {4, 8, 16} (appendix Table 5); the default appears to be 4 based on the $k=4$ worker count matching
• DN parameter $c$: evaluated at {0, 0.1} (Section 7 ablation); chosen: 0
• Grace period $\tau_{\text{grace}}$: value not explicitly stated in the paper or appendix
• Nesterov $\beta$: value not explicitly stated; standard Nesterov momentum uses $\beta$ around 0.9

The paper reports that hyperparameters for each optimizer combination in Figure 5 were "tuned separately," but the specific tuning methodology (grid search, random search, sequential) is not described. For the main DN+DyLU experiments, the chosen values are highlighted in bold in Table 5.

Baselines. The paper evaluates against five baselines, each representing a different point in the design space:

1. Finetune 1 worker ($4\times$ batch size): standard single-machine training with a proportionally larger batch size to match the total data processed per outer step. This baseline tests whether distributed training provides any benefit beyond what can be achieved by simply scaling the batch size on a single device. For $k=4$ workers, the batch size is multiplied by 4 (so 512 if the distributed batch size is 128). This baseline removes all communication and staleness issues at the cost of requiring a single device with sufficient memory for the larger batch.

2. DiLoCo (Douillard et al., 2023): the synchronous Local-SGD baseline with AdamW inner optimizer and Nesterov outer optimizer. This is the paper's primary performance target—the goal is to match DiLoCo's perplexity without the synchronization barrier.

3. Async. DiLoCo: the naive asynchronous version of DiLoCo where workers operate independently and the server applies Nesterov updates as pseudo-gradients arrive. This baseline isolates the effect of removing synchronization without any mitigation techniques.

4. Async. DiLoCo + [Poly / PolyThres / Delay Comp. / Buffer]: augmented versions of Async. DiLoCo using existing staleness mitigation methods from the literature, as described in Section 4.

5. Pretrained checkpoint (24K steps): the model state before any distributed training begins, providing a lower bound on perplexity that any method should improve upon.

Inner optimizer state handling. Following DiLoCo, when a worker picks up a data shard that another worker has just finished training on, the AdamW optimizer state (momentum buffers, adaptive learning rate estimates) is kept locally on the worker and not communicated. This means the inner optimizer state persists across training rounds on the same worker, even if the worker switches data shards. The inner learning rate schedule is global (per-shard, per Equation 3) and continues across rounds—it is not reset when a new training task begins. This design choice reduces communication overhead (no need to transmit optimizer state) but means the inner optimizer operates on potentially stale state when a worker resumes training on a shard it has not seen recently.

Cross-validation and statistical reporting. The paper does not report confidence intervals, error bars, or multiple random seeds for any experiment. The deterministic simulation framework means that results are exactly reproducible given the same configuration, but the sensitivity to random data ordering, data shard assignment, or pretraining initialization is not assessed. This is a limitation of the current experimental design—we cannot distinguish between genuine method effects and noise from a particular training run.

---

Main Quantitative Results

Diagnostic Results: Isolating the Momentum Failure (Section 4)

Optimizer combination sweep (Figure 5). On a 20M model with 4 heterogeneous workers and 64,000 steps per worker (256,000 total local steps):

• AdamW+Nesterov achieves the lowest final perplexity among all asynchronous combinations. The exact perplexity value is not stated in the text, but Figure 5 shows Async. AdamW+Nesterov terminating around 44 on the perplexity axis (reading from the figure).
• AdamW+Adam performs noticeably worse, confirming the DiLoCo finding that Adam's normalization is counterproductive for outer optimization in Local-SGD.
• The ranking correlates strongly with the synchronous DiLoCo results, establishing that the optimizer pairing wisdom transfers to the asynchronous setting—the problem is not the choice of optimizers but how they interact with asynchrony.

Momentum comparison — synchronous vs. asynchronous (Figure 6). The key diagnostic result, measured on the same setup:

• AdamW+SGD (no outer momentum): Async. DiLoCo reaches lower perplexity than Sync. DiLoCo throughout training. Reading from Figure 6, Async. AdamW+SGD terminates around perplexity 44.5, while Sync. AdamW+SGD terminates around 45.0. The gap is modest but consistent—approximately 0.5 perplexity points favoring asynchronous.
• AdamW+Nesterov (with outer momentum): Sync. DiLoCo significantly outperforms Async. DiLoCo. From Figure 6, Sync. AdamW+Nesterov terminates around 41.3 (consistent with Table 1), while Async. AdamW+Nesterov terminates around 44.3—a gap of approximately 3 perplexity points.

This reversal is the paper's most important single empirical result. It demonstrates that removing outer momentum makes asynchrony beneficial, while adding it back makes asynchrony harmful, all else equal. The paper does not report statistical significance, but the gap magnitude (3 perplexity points is a substantial difference in language modeling) and the qualitative reversal make the result convincing.

Homogeneity experiment (Figure 7). With 4 workers operating at identical speeds:

• Sync. DiLoCo (AdamW+Nesterov) terminates around 41.3 perplexity.
• Async. DiLoCo (AdamW+Nesterov) terminates around 44.3 perplexity—the same 3-point gap as with heterogeneous devices.

This result is particularly striking because with homogeneous devices, all workers finish their $H=50$ local steps simultaneously. The server receives all 4 pseudo-gradients at the same moment. The only difference from synchronous training is that the server must still process them sequentially (one at a time through the Nesterov update) rather than averaging them first and applying once. The 3-point gap is therefore attributable entirely to sequential application of simultaneous gradients, not to variable staleness.

Existing fixes (Figure 8). Evaluated on the same 4-worker setup:

• Async. DiLoCo (baseline): terminates around 44.3.
• Async. DiLoCo + Poly: marginally better, perhaps 44.1–44.2.
• Async. DiLoCo + PolyThres: similar to Poly.
• Async. DiLoCo + Delay Comp.: slightly worse than baseline (around 44.5), suggesting delay compensation actually hurts in the Local-SGD setting.
• Async. Buffer: reaches approximately 42.5–43.0 perplexity, significantly closer to Sync. DiLoCo (~41.3), but shows visible oscillations early in training (the perplexity curve wiggles up and down rather than smoothly decreasing). These oscillations are the "instability" the paper refers to.

None of the existing methods match Sync. DiLoCo's final perplexity. Async. Buffer comes closest but at the cost of training instability.

---

Main Results: Delayed Nesterov + Dynamic Local Updates (Section 7)

Headline comparison (Figure 2, Table 1). On a 20M model with 4 "very" heterogeneous workers, comparing DN+DyLU against baselines:

Numbers from Table 1, "very" heterogeneity column.

Note: The final perplexity values are shown twice — verify if this is intended from the original.

What these numbers mean:

• DN+DyLU matches synchronous DiLoCo in final perplexity (41.13 vs. 41.35—the 0.22 difference is small and may not be statistically significant given the single-run reporting). This is the paper's primary claim: asynchronous training with the proposed fixes achieves parity with synchronous training in model quality.
• DN+DyLU substantially outperforms naive Async. DiLoCo (41.13 vs. 44.27—a 3.14 perplexity point improvement). The entire gap between asynchronous and synchronous training is recovered.
• DN+DyLU slightly outperforms single-worker fine-tuning (41.13 vs. 42.47), confirming that distributed Local-SGD provides a benefit beyond simply scaling the batch size.
• DiLoCo outperforms single-worker fine-tuning (41.35 vs. 42.47), replicating the Local-SGD generalization benefit reported in prior work (Gu et al., 2023).

Perplexity per update curve (Figure 2, left panel). The DN+DyLU curve closely tracks the DiLoCo curve throughout training. From the figure (reading approximate values):

• At 20,000 updates: DiLoCo ~46, DN+DyLU ~46, Async. DiLoCo ~47.5
• At 40,000 updates: DiLoCo ~44, DN+DyLU ~44, Async. DiLoCo ~46
• At 60,000 updates: DiLoCo ~42.5, DN+DyLU ~42.5, Async. DiLoCo ~45
• At 80,000 updates (final): DiLoCo ~41.3, DN+DyLU ~41.1, Async. DiLoCo ~44.3

DN+DyLU and DiLoCo are nearly indistinguishable throughout, while Async. DiLoCo consistently trails by 1.5–3 perplexity points.

Perplexity per wall-clock time (Figure 2, right panel). This is where DN+DyLU's advantage appears. The x-axis is simulated wall-clock time (not update count). DN+DyLU reaches any given perplexity threshold earlier than DiLoCo because asynchronous training eliminates straggler idle time. Reading from the figure:

• To reach perplexity 45: DiLoCo requires approximately 19,000 time units, DN+DyLU requires approximately 18,000 time units—a roughly 5% speedup.
• To reach perplexity 43: DiLoCo requires approximately 22,500 time units, DN+DyLU requires approximately 20,000 time units—a roughly 11% speedup.
• The gap widens at lower perplexities because DiLoCo's straggler penalty compounds over more training rounds.

The wall-clock advantage depends on the degree of device heterogeneity. With homogeneous devices, DiLoCo and DN+DyLU would have identical wall-clock times (since all workers finish simultaneously, the straggler effect vanishes). The paper's "very" heterogeneous setting represents a favorable scenario for asynchronous methods.

---

Ablation Studies

Varying worker heterogeneity (Figure 10, Table 1). DN+DyLU is evaluated across four heterogeneity levels ("no," "slight," "moderate," "very"):

Note: The original text shows the same values with "slight" and "moderate" and "very" variations. This appears to be a table representation issue. The original data should be verified.

The key findings:

1. DN+DyLU performs consistently across all heterogeneity levels. Final perplexity varies from 41.09 to 41.27—a range of only 0.18, suggesting the method is robust to the degree of speed variation.
2. Async. DiLoCo is similarly invariant to heterogeneity level (44.27–44.38), but consistently worse. This reinforces the Figure 7 finding that the core problem is not speed heterogeneity per se but sequential application of momentum.
3. DN+DyLU slightly outperforms DiLoCo when there is no heterogeneity (41.27 vs. 41.35). The paper attributes this to "numerical error, as the two methods reduce to the same and the training curves match almost perfectly"—with homogeneous devices and DyLU giving all workers the same step count, DN+DyLU should be mathematically equivalent to DiLoCo when the grace period batches all workers together. The 0.08 difference is within the range of implementation-level numerical variation.

Varying number of workers (Figure 11, Table 2). Evaluated on 20M model with $k \in \{4, 8, 16\}$:

The key findings:

1. The benefit of Local-SGD diminishes as $k$ increases. With 16 workers, single-worker fine-tuning with 16× batch size achieves 40.60 perplexity, outperforming both DiLoCo (41.25) and DN+DyLU (40.98). This is consistent with prior observations that large-batch training becomes more competitive as the batch size grows (Lin et al., 2018), and that Local-SGD's advantages are most pronounced at moderate worker counts.

2. DN+DyLU continues to match or slightly outperform DiLoCo across worker counts. The gap between DN+DyLU and DiLoCo is 0.22 (4 workers), 0.21 (8 workers), and 0.27 (16 workers)—small, consistent differences favoring DN+DyLU that may not be significant.

3. Async. DiLoCo's performance is invariant to worker count (~44.24 across all settings), suggesting the momentum degradation is a per-update phenomenon that does not worsen with more workers (since each worker's pseudo-gradient is still applied sequentially, regardless of how many workers exist).

4. Single-worker fine-tuning improves substantially with more workers (42.47 → 41.28 → 40.60) because the batch size increases proportionally (4× → 8× → 16×). This is a standard observation: larger batches improve training efficiency up to a point, though they require more memory. The fact that Local-SGD methods stop beating large-batch training at $k=16$ suggests the communication reduction benefit is offset by the optimization challenge of training across more independent workers.

The paper notes that "the advantage becomes more pronounced during the later stages of convergence" for Local-SGD methods, referencing prior work on Local-SGD's generalization benefits (Gu et al., 2023). Figure 11 shows the perplexity curves separating in the final 20,000–30,000 updates, with DiLoCo and DN+DyLU continuing to decrease while single-worker fine-tuning plateaus.

Varying model size (Figure 12, Table 3). Evaluated with 4 "very" heterogeneous workers on 20M, 60M, and 150M models:

The key findings:

1. DN+DyLU matches DiLoCo across all model sizes. The gap between DN+DyLU and DiLoCo is 0.22 (20M), 0.02 (60M), and 0.03 (150M)—essentially identical at 60M and 150M. This is the critical scaling result: the proposed fixes do not degrade as models grow.

2. Local-SGD outperforms single-worker fine-tuning across all sizes. The gap between DiLoCo/DN+DyLU and single-worker fine-tuning is 1.12–1.34 perplexity points at 20M, 0.25–0.27 at 60M, and 0.21–0.24 at 150M. The gap shrinks with model size (larger models may benefit less from the data diversity across shards, or the 4× batch size becomes more competitive), but the advantage persists.

3. Async. DiLoCo's gap to DiLoCo is 2.92 (20M), 1.09 (60M), 0.85 (150M). The gap shrinks as model size increases—naive asynchronous training degrades less on larger models. The paper notes this but does not explain why. Possible explanations: larger models have smoother loss landscapes (making stale pseudo-gradients less harmful), or the pseudo-gradients from larger models are better-aligned across workers (reducing the sequential distortion), or the pretraining starting point is better for larger models (22.80 vs. 61.64 perplexity), leaving less room for optimization differences to manifest. The paper explicitly states: "It's important to note that the performance disparity between synchronous and asynchronous DiLoCo does not diminish even as the model size increases"—but Table 3 shows it does diminish (2.92 → 1.09 → 0.85), so this statement appears to be an error or refers to a different comparison.

4. Larger models start from better pretrained checkpoints (30.19 at 60M, 22.80 at 150M vs. 61.64 at 20M) and achieve correspondingly better final perplexity. The absolute improvement from distributed training is larger for the 20M model (~20 perplexity points) than for 150M (~5.5 points), largely because the 20M model had more room to improve from its weaker pretrained state.

Varying $c$ in Delayed Nesterov (Table 4). An ablation comparing $c=0$ (pure SGD between Nesterov updates) and $c=0.1$ (slight momentum leakage) across varying worker counts and model sizes:

Note: The following text represents a reconstruction attempt of Table 4 data.

Varying workers (k):

Varying model size:

The differences between $c=0$ and $c=0.1$ are negligible (within 0.16 perplexity points across all settings), with no consistent direction of advantage. The paper concludes that "adding slight momentum at intermediate steps does not help too much" and sets $c=0$ as default. This result simplifies the method and supports the interpretation that the core mechanism is the decoupling of momentum from gradient updates, not the fine-tuning of momentum leakage.

---

Assessment: Do the Experiments Support the Claims?

Claim 1: DN+DyLU matches synchronous DiLoCo in perplexity per update. Supported, with qualifications. Table 1 shows DN+DyLU achieving 41.13 vs. DiLoCo's 41.35 on the 20M model—a difference of 0.22 perplexity points that is likely not statistically significant (though no error bars are reported). The perplexity curves in Figure 2 (left) are nearly overlapping. At 60M and 150M scales, the gap narrows further (0.02 and 0.03 respectively). The claim holds across all tested heterogeneity levels and worker counts.

The qualification is that all results come from single training runs with a deterministic simulation—we cannot assess variance. A single anomalous run could produce a 0.22 difference. The consistency across model sizes and heterogeneity levels strengthens the case, but proper statistical validation (multiple seeds, confidence intervals) is absent.

Claim 2: DN+DyLU significantly surpasses synchronous DiLoCo in wall-clock time. Supported, conditional on the simulation fidelity. Figure 2 (right) shows DN+DyLU reaching any given perplexity level earlier than DiLoCo, with the gap widening at lower perplexities. The magnitude of the advantage depends entirely on the device heterogeneity profile—with homogeneous devices, there would be no wall-clock advantage (since no straggler effect exists). The paper's "very" heterogeneous setting (device speeds in Figure 4) represents a plausible heterogeneity scenario, but the paper does not characterize how typical this profile is for real-world distributed training setups.

The simulation's use of "faked training time based on real-world device statistics" is well-motivated for reproducibility, but it abstracts away several real-world factors that could affect wall-clock time: network latency variability, queuing delays when multiple workers communicate simultaneously, and competition for shared resources. The reported wall-clock improvements should be understood as upper bounds—real deployments would likely see smaller gains.

Claim 3: The core challenge is momentum in the outer optimizer, not staleness in general. Strongly supported. The reversal in Figure 6 (async better than sync with SGD, sync better than async with Nesterov) and the persistence of the gap with homogeneous devices (Figure 7) are the paper's most convincing results. They isolate momentum as the necessary and sufficient condition for the asynchronous degradation, which is a clean and falsifiable claim. No alternative explanation (staleness, data imbalance, learning rate) could produce the reversal pattern while also explaining the homogeneity result.

Claim 4: Existing staleness mitigation methods from the literature are insufficient for asynchronous Local-SGD. Supported. Figure 8 shows that polynomial discounting, thresholding, delay compensation, and FedBuff-style buffering all fail to match synchronous DiLoCo. The failure of delay compensation is particularly informative—it works for standard asynchronous SGD (Zheng et al., 2017) but not for Local-SGD, confirming that pseudo-gradients behave differently from raw gradients in ways that matter for staleness correction.

Potential weaknesses in the experimental design:

• No combination of DN and DyLU with individual ablations. The paper always presents DN and DyLU together in the main results. There is no experiment showing DN alone (without DyLU), DyLU alone (without DN), or a formal ablation quantifying each component's contribution. The logical argument that they address different aspects of the problem (DN fixes the optimizer, DyLU reduces staleness) is compelling but unverified. It is possible that one component does all the work and the other is unnecessary.

• Missing hyperparameter: the grace period $\tau_{\text{grace}}$. The paper describes the grace period mechanism in Section 3 and includes it in Algorithm 2, but never states what value is used or provides an ablation varying it. This is a significant omission because the grace period directly affects how many pseudo-gradients get batched together, which interacts with the Delayed Nesterov buffer size $N$. If $\tau_{\text{grace}}$ is large enough to batch all workers, the system is effectively synchronous regardless of DN and DyLU. The paper's claim that DN+DyLU achieves asynchronous training benefits depends on the grace period being small, but this is not verified.

• Limited model scale. The largest model tested is 150M parameters, which is 2–3 orders of magnitude smaller than the models where distributed training is most relevant (1B–100B+ parameters). The paper does not claim these results extend to larger scales, but the motivation (harnessing geographically distributed compute for "even more powerful large models") implies an ambition to scale up. At billion-parameter scales, communication cost dominates and the dynamics of Local-SGD may differ substantially.

• Single dataset (C4). While C4 is a standard language modeling benchmark, it represents only one data distribution. The interaction between data diversity across shards and asynchronous optimization dynamics is unexplored—if shards contain very different types of text, pseudo-gradients may point in more divergent directions, potentially exacerbating or mitigating the momentum distortion.

• No theoretical convergence analysis. The paper explicitly lists this as a limitation (Section 9), but it is worth noting as an experimental weakness: without theoretical guarantees, we cannot predict when DN+DyLU will work beyond the tested configurations. The empirical results show it works for 4–16 workers on C4 with Chinchilla-style transformers at 20M–150M parameters, but the boundaries of applicability are unknown.

• Perplexity as the sole metric. The paper evaluates only language modeling perplexity. It does not measure downstream task performance (zero-shot evaluation, fine-tuning on NLP benchmarks), which is standard in the language modeling literature and would provide evidence that the perplexity improvements translate to useful capabilities. It is possible that the optimization differences between methods affect perplexity but not downstream performance, or vice versa.

• Wall-clock time reporting is simulation-only. No real-hardware wall-clock measurements are provided. The simulation is validated against synchronous behavior (the paper states that synchronous updates through the asynchronous framework produce correct results), but this validates the training logic, not the timing model. Real distributed deployments face unpredictable network delays, stragglers from transient load, and hardware failures that the simulation does not model.

5. Experimental Analysis

Evaluation Methodology

Dataset. All experiments use the C4 dataset (Raffel et al., 2020), a large corpus derived from Common Crawl web text, for next-token prediction language modeling. The paper does not explicitly state the validation split size or how it is constructed from the full C4 dataset, but all perplexity measurements are reported on the C4 validation set. The total training budget is set to 88,000 steps for all models, with the first 24,000 steps performed as standard single-machine pretraining (without distributed methods), followed by distributed Local-SGD fine-tuning for the remaining 64,000 steps.

Base model(s). Three Chinchilla-style (Hoffmann et al., 2022) transformer decoder-only architectures are evaluated: 20M parameters (6 layers, 256 hidden dim, 4 heads), 60M parameters (3 layers, 896 hidden dim, 16 heads), and 150M parameters (12 layers, 896 hidden dim, 16 heads). All use a vocabulary size of 32,000 and key/value size of 64. The models are chosen to span a 7.5× range in parameter count to test whether the proposed methods scale with model size, though the paper acknowledges these are well below billion-parameter scales where communication bottlenecks dominate. All models are pretrained for 24,000 steps before any distributed training begins, providing a common starting point for method comparison.

Metrics. The primary metric is perplexity on the C4 validation set, reported as a function of two independent variables: (1) total local updates ($t_{\text{local}}$), which counts the cumulative number of local training steps performed across all workers and enables computation-matched comparison between methods; and (2) wall-clock time, which uses simulated elapsed time based on device speed profiles (shown in Figure 4) to evaluate whether asynchronous methods translate their lack of idle time into faster convergence. The dual metric reporting is essential because synchronous methods achieve better perplexity at equal update counts (no staleness), while asynchronous methods are expected to complete more updates per unit wall-clock time (no straggler waiting).

Baselines. The paper evaluates against five baselines:

• Finetune 1 worker ($k \times$ batch size): standard single-machine training with the batch size multiplied by $k$ to match the total data processed per outer step across distributed workers. This tests whether distributed Local-SGD provides any benefit beyond simply scaling the batch size on a single device. For $k=4$ workers with distributed batch size 128, the single-worker batch size is 512.
• DiLoCo (Douillard et al., 2023): the synchronous Local-SGD baseline using AdamW as the inner optimizer and Nesterov momentum as the outer optimizer, as described in Algorithm 1. This is the paper's primary performance target.
• Async. DiLoCo: the naive asynchronous version of DiLoCo where workers operate independently and the server applies standard Nesterov updates as each pseudo-gradient arrives, without any staleness mitigation.
• Async. DiLoCo + existing fixes: augmented versions using polynomial discounting (Xie et al., 2019), polynomial discounting with thresholding, delay compensation (Zheng et al., 2017), or FedBuff-style buffering (Nguyen et al., 2022), all described in Section 4.
• Pretrained checkpoint (24K steps): the model state before any distributed training, providing a lower bound on achievable perplexity.

Generation budget / compute accounting. Compute is measured in units of local training steps ($t_{\text{local}}$), where one local step corresponds to one inner optimizer update (AdamW) on one worker processing one batch. Since all methods use the same inner optimizer with the same per-step computational cost, comparing at equal $t_{\text{local}}$ provides a FLOPs-matched comparison. The total local updates across all workers is $t_{\text{max}} = 88,000$, with 24,000 pretraining steps (non-distributed) and up to 64,000 distributed steps per worker. For wall-clock comparisons, the paper uses a deterministic simulation that assigns each worker a fixed processing speed (steps per second, as shown in Figure 4) and simulates task scheduling according to Algorithm 2, with "faked training time based on real-world device statistics." The paper validates the simulation by confirming that synchronous updates through the asynchronous framework produce correct results.

Cross-validation / statistical protocol. The paper does not report confidence intervals, error bars, or results across multiple random seeds. All experiments use a deterministic simulation framework, making results exactly reproducible given the same configuration, but the sensitivity to random data ordering, data shard assignment, or pretraining initialization is not assessed. Hyperparameters for each optimizer combination in Figure 5 are "tuned separately" (Section 4), but the tuning methodology (grid search, random search, sequential) is not specified. For the main experiments, hyperparameter values were selected from sweeps over the ranges listed in Table 5, with chosen values highlighted in bold.

Inner optimizer state handling. Following DiLoCo (Douillard et al., 2023), when a worker picks up a data shard that another worker has just finished training on, the AdamW optimizer state (momentum buffers, adaptive learning rate estimates) is kept locally on the worker and not communicated. The inner optimizer state persists across training rounds, and the inner learning rate schedule (Equation 3) continues globally per-shard across rounds without reset.

Device heterogeneity configuration. Four levels of worker heterogeneity are defined and visualized in Figure 10: "no" (all workers at identical speed), "slight" (minor speed differences), "moderate," and "very" (the largest speed gap, corresponding to the device speed profiles shown in Figure 4). The "very" heterogeneous setting is the default for all main experiments unless otherwise noted. In the deterministic simulation, each worker's training time is computed from its speed profile, and the task scheduler (Algorithm 2) deterministically processes events based on timed completions.

---

Main Quantitative Results

Diagnostic Experiments: Isolating the Momentum Failure Mode

The paper's Section 4 diagnostic results are the empirical foundation for both proposed methods. All experiments in this section use a 20M parameter model with 4 workers, inner steps $H = 50$, and 64,000 steps per worker (256,000 total local steps). The model is pretrained for 24,000 steps before distributed fine-tuning begins.

Optimizer combination sweep (Figure 5). Testing all combinations of inner optimizers {SGD, AdamW} and outer optimizers {SGD, Adam, Nesterov} on asynchronous Local-SGD:

• AdamW+Nesterov achieves the best final perplexity among all asynchronous combinations, consistent with the synchronous DiLoCo finding. Reading from Figure 5, Async. AdamW+Nesterov terminates at approximately 44.3 perplexity.
• AdamW+Adam performs noticeably worse, supporting the DiLoCo observation that Adam's normalization is counterproductive for outer optimization in Local-SGD since pseudo-gradients "tend to be larger than true gradients."
• With AdamW as the inner optimizer, the three outer optimizers (SGD, SGD Momentum, Nesterov) show comparable performance, but Nesterov "stabilizes the learning curve and slightly improves final performance."

Momentum-specific comparison — synchronous vs. asynchronous (Figure 6). This is the paper's pivotal diagnostic result. Comparing AdamW+SGD (no outer momentum) against AdamW+Nesterov (with outer momentum) in both synchronous and asynchronous settings:

• Without outer momentum (AdamW+SGD): Async. DiLoCo outperforms Sync. DiLoCo. Reading from Figure 6, Async. AdamW+SGD terminates at approximately 44.5 perplexity while Sync. AdamW+SGD terminates at approximately 45.0—a consistent gap of roughly 0.5 perplexity points favoring asynchronous training.
• With outer momentum (AdamW+Nesterov): The ordering reverses. Sync. DiLoCo significantly outperforms Async. DiLoCo. Sync. AdamW+Nesterov terminates at approximately 41.3 perplexity (consistent with Table 1), while Async. AdamW+Nesterov terminates at approximately 44.3—a gap of roughly 3 perplexity points.

The qualitative reversal of which method wins when momentum is removed versus added is the paper's strongest single piece of evidence that momentum is the necessary and sufficient condition for the asynchronous degradation.

Homogeneity experiment (Figure 7). Running Async. DiLoCo with all 4 workers operating at identical speeds (homogeneous devices) to eliminate variable-speed-induced staleness:

• Sync. DiLoCo terminates at approximately 41.3 perplexity.
• Async. DiLoCo terminates at approximately 44.3 perplexity—the same 3-point gap observed with heterogeneous devices.

This demonstrates that the performance degradation persists even when all pseudo-gradients are computed from the same parameter version (workers start and finish simultaneously). The gap is attributable solely to sequential application of simultaneous gradients through the Nesterov momentum recurrence.

Existing fixes evaluation (Figure 8). Testing four staleness mitigation methods from the asynchronous optimization literature against Async. DiLoCo:

• Async. DiLoCo + Poly (polynomial discounting): provides "marginal benefits," with the curve nearly overlapping the baseline (final perplexity approximately 44.1–44.2).
• Async. DiLoCo + PolyThres (poly + threshold): similar marginal improvement to Poly alone.
• Async. DiLoCo + Delay Comp. (delay compensation): slightly worse than baseline (approximately 44.5), demonstrating that delay compensation is counterproductive for Local-SGD pseudo-gradients. The paper explicitly notes this "points out the difference between asynchronous SGD and asynchronous Local-SGD."
• Async. Buffer (FedBuff-style): significantly closes the gap, reaching approximately 42.5–43.0 perplexity, but exhibits "instability in early stage of training" visible as oscillations in the perplexity curve.

Critically, none of these methods match Sync. DiLoCo's final perplexity of 41.35, establishing that existing staleness mitigation techniques are insufficient for asynchronous Local-SGD in language modeling.

---

Main Results: DN+DyLU Performance

All main results use 4 workers with "very" heterogeneous device speeds (Figure 4, bottom-right of Figure 10), Delayed Nesterov with buffer size $N = 4$ and $c = 0$, and Dynamic Local Updates with $H = 50$ for the fastest worker.

Headline comparison — 20M model (Figure 2, Table 1). Comparing Async. DN+DyLU against all baselines:

DN+DyLU achieves 41.13 perplexity compared to DiLoCo's 41.35—a difference of 0.22 that makes the methods essentially indistinguishable. Compared to naive Async. DiLoCo (44.27), DN+DyLU recovers the entire 3.14 perplexity-point gap. DN+DyLU also outperforms single-worker fine-tuning (42.47), confirming that distributed Local-SGD with the proposed fixes provides a benefit beyond batch size scaling alone.

Perplexity per update curve (Figure 2, left panel). The DN+DyLU curve closely tracks DiLoCo throughout training, with the two lines nearly overlapping at all points. Reading approximate values from Figure 2:

• At 20,000 updates: DiLoCo and DN+DyLu both around 46.0, Async. DiLoCo around 47.5.
• At 40,000 updates: both around 44.0, Async. DiLoCo around 46.0.
• At 60,000 updates: both around 42.5, Async. DiLoCo around 45.0.
• At 80,000 updates (final): DN+DyLU at 41.13, DiLoCo at 41.35, Async. DiLoCo at 44.27.

The gap between Async. DiLoCo and the other methods widens slightly over training (from ~1.5 points at 20K to ~3 points at 80K), suggesting the momentum distortion compounds over successive outer updates.

Perplexity per wall-clock time (Figure 2, right panel). DN+DyLU reaches any given perplexity threshold earlier than DiLoCo. Reading from Figure 2:

• To reach perplexity 45: DiLoCo requires approximately 19,000 simulated time units, DN+DyLU approximately 18,000—roughly 5% faster.
• To reach perplexity 43: DiLoCo requires approximately 22,500 simulated time units, DN+DyLU approximately 20,000—roughly 11% faster.

The wall-clock advantage widens at lower perplexities because DiLoCo's straggler penalty compounds over more training rounds (more synchronization barriers = more cumulative idle time). The absolute magnitude of the advantage depends on the device heterogeneity profile—with homogeneous devices, DN+DyLU and DiLoCo would have identical wall-clock times since no straggler effect exists.

---

Ablation Studies

Varying worker heterogeneity (Figure 10, Table 1). Evaluating DN+DyLU across four heterogeneity levels on 20M model with 4 workers:

DN+DyLU performs consistently across all heterogeneity levels (perplexity range: 41.09–41.27), demonstrating robustness to the degree of worker speed variation. Async. DiLoCo is similarly invariant (44.27–44.38) but consistently worse, reinforcing the Figure 7 finding that the core problem is sequential momentum application rather than speed heterogeneity per se. DN+DyLU slightly outperforms DiLoCo at "no" heterogeneity (41.27 vs. 41.35), which the paper attributes to "numerical error, as the two methods reduce to the same and the training curves match almost perfectly." With identical-speed devices and DyLU assigning equal steps, DN+DyLU should be mathematically equivalent to DiLoCo.

Figure 10 reveals an additional observation: a "periodic oscillation in performance is observed in certain device groupings" for Async. DiLoCo, further "highlighting the lack of robustness of the original asynchronous algorithm." The DN+DyLU curves do not show these oscillations.

Varying number of workers (Figure 11, Table 2). Evaluating on 20M model with $k \in \{4, 8, 16\}$:

Two trends emerge. First, the benefit of Local-SGD over single-worker fine-tuning diminishes as $k$ increases: with 16 workers, single-worker fine-tuning with 16× batch size achieves 40.60 perplexity, outperforming both DiLoCo (41.25) and DN+DyLU (40.98). Second, DN+DyLU continues to match or slightly outperform DiLoCo across all worker counts, with gaps of 0.22 (4 workers), 0.21 (8), and 0.27 (16). Async. DiLoCo's performance is invariant to worker count (~44.24 across all settings). Figure 11 shows the perplexity curves separating in the final 20,000–30,000 updates, with DiLoCo and DN+DyLU continuing to decrease while single-worker fine-tuning plateaus, consistent with prior observations of Local-SGD's generalization benefits (Gu et al., 2023).

Varying model size (Figure 12, Table 3). Evaluating with 4 "very" heterogeneous workers on 20M, 60M, and 150M models:

DN+DyLU matches DiLoCo across all model sizes, with gaps of 0.22 (20M), 0.02 (60M), and 0.03 (150M)—essentially identical at 60M and 150M. Local-SGD (both sync and async) outperforms single-worker fine-tuning across all sizes, though the gap shrinks from 1.12–1.34 points at 20M to 0.21–0.27 points at 150M. The paper states that "this advantage becomes more pronounced during the later stages of convergence, aligning with findings from previous research that highlight Local-SGD's superior generalization capabilities (Gu et al., 2023)."

A notable observation: the gap between Async. DiLoCo and DiLoCo shrinks from 2.92 points (20M) to 1.09 points (60M) to 0.85 points (150M). The paper states that "the performance disparity between synchronous and asynchronous DiLoCo does not diminish even as the model size increases" — but Table 3 clearly shows a diminishing gap. This appears to be an error in the text. Larger models start from better pretrained checkpoints (30.19 at 60M, 22.80 at 150M vs. 61.64 at 20M), leaving less room for optimization differences to manifest, which may partially explain the smaller gap.

Varying $c$ in Delayed Nesterov (Table 4). Comparing $c = 0$ (pure SGD between Nesterov updates) and $c = 0.1$ (slight momentum leakage) across varying worker counts and model sizes:

For 20M model with varying $k$: $c=0$ achieves 41.13 (4 workers), 41.02 (8), 40.98 (16); $c=0.1$ achieves 41.16 (4), 40.93 (8), 41.04 (16).

For 4 workers with varying model size: $c=0$ achieves 41.13 (20M), 24.53 (60M), 17.26 (150M); $c=0.1$ achieves 41.16 (20M), 24.69 (60M), 17.27 (150M).

The maximum difference between $c=0$ and $c=0.1$ is 0.16 perplexity points (at 60M), with no consistent direction of advantage. The paper concludes that "adding slight momentum at intermediate steps does not help too much" and sets $c=0$ as the default. This result supports the interpretation that the essential mechanism is decoupling momentum from gradient updates, not fine-tuning the degree of momentum leakage. The paper also notes that "setting the value of $c$ does not introduce any overhead to the overall algorithm."

---

Critical Assessment

Does DN+DyLU genuinely match synchronous DiLoCo in perplexity per update?

The experiments in Table 1, Table 2, and Table 3 consistently show DN+DyLU achieving perplexity values within 0.3 points of DiLoCo across all tested configurations (heterogeneity levels, worker counts, model sizes). At 60M and 150M scales, the gap narrows to 0.02–0.03 points—effectively identical. The perplexity curves in Figure 2 overlap almost completely throughout training. This evidence supports the claim that DN+DyLU recovers DiLoCo-level performance.

However, two concerns qualify this conclusion. First, no statistical quantification of variance exists. All results are from single deterministic simulation runs. A 0.22 perplexity-point difference between DN+DyLU (41.13) and DiLoCo (41.35) on the 20M model could be noise from a particular random seed, data ordering, or pretraining initialization. Without multiple runs or confidence intervals, we cannot determine whether DN+DyLU genuinely matches DiLoCo or reliably falls slightly short. The consistency across heterogeneity levels and model sizes (where the gap is always small and in DN+DyLU's favor at 20M, alternating at larger scales) is suggestive but not statistically conclusive.

Second, the paper never ablates DN and DyLU separately. All main results show DN+DyLU combined. It is possible that only one of the two techniques is necessary, or that their contributions are asymmetric (one provides most of the benefit, the other is marginal). The paper provides a logical decomposition—DN addresses optimizer-level momentum distortion, DyLU addresses scheduling-level staleness—but the empirical evidence for this decomposition is indirect (the diagnostic experiments show momentum distortion exists and that variable-speed staleness exists) rather than causally established through ablations. A simple experiment showing DN alone, DyLU alone, and DN+DyLU combined would strengthen the claimed complementarity.

Does DN+DyLU significantly surpass DiLoCo in wall-clock time?

The wall-clock advantage shown in Figure 2 (right panel) demonstrates a real effect—DN+DyLU reaches a given perplexity threshold earlier than DiLoCo—but its magnitude and generality are overclaimed relative to the evidence. The paper states that DN+DyLU "significantly surpasses DiLoCo in terms of perplexity versus wall clock time," which is true for the tested configuration (4 "very" heterogeneous workers) but conditional on that specific heterogeneity profile.

The wall-clock advantage scales with the degree of heterogeneity: with homogeneous devices, there would be no advantage (the methods reduce to the same thing); with extreme heterogeneity, the advantage could be larger. The paper tests only one "very" heterogeneous profile (Figure 4). An ablation showing wall-clock speedup as a function of heterogeneity level would clarify when the advantage is meaningful (e.g., does "slight" heterogeneity produce a 2% speedup, 10% speedup, or negligible difference?). Without this, the claim of "significantly surpasses" is only established for the specific device pool tested, not as a general property of the method.

Additionally, the simulation's use of "faked training time based on real-world device statistics" abstracts away real-world factors that could reduce or eliminate the wall-clock advantage: network latency variability (which could slow asynchronous communication relative to synchronous batch communication), queuing delays (multiple workers communicating simultaneously and competing for bandwidth), and transient hardware issues (a worker that temporarily slows down mid-training). The reported wall-clock improvements should be interpreted as an upper bound under idealized communication conditions, and the paper should be clearer about this.

Is the momentum distortion analysis sufficient to claim this is the "key challenge"?

The diagnostic experiments in Figures 5, 6, and 7 provide strong and well-controlled evidence that outer momentum interacts poorly with sequential pseudo-gradient application. The reversal in Figure 6 (async better than sync without momentum, sync better than async with momentum) is a clean, falsifiable demonstration that momentum is necessary and sufficient for the performance gap. The homogeneity experiment in Figure 7—showing the same gap even with identical-speed devices—rules out variable-speed-induced staleness as the primary cause and implicates sequential application directly.

However, the paper's analytical derivation (Equation 5) showing that sequential Nesterov produces a different momentum-to-gradient ratio than batched Nesterov is presented as an illustrative explanation, not a proof. It assumes all workers produce identical pseudo-gradients $g$—a strong simplification. In practice, pseudo-gradients from different workers on different data shards point in different directions, and the interaction between sequential momentum updates and gradient diversity is more complex than the identical-gradient analysis suggests. The derivation demonstrates that sequential application causes a structural mismatch, but it does not quantify how much of the observed 3-point perplexity gap this mismatch explains versus other factors (gradient diversity, varying staleness, data shard sampling imbalance).

The failure of existing staleness mitigation methods (Figure 8) provides additional corroboration but is not definitive: these methods might fail for reasons unrelated to the momentum problem (e.g., delay compensation's Taylor approximation may be inaccurate for pseudo-gradients that aggregate $H$ local steps of AdamW, regardless of whether momentum is used). The fact that delay compensation "points out the difference between asynchronous SGD and asynchronous Local-SGD" is an important observation, but it does not directly prove that momentum is the sole source of the Local-SGD-specific issue.

What experiments are missing that would strengthen the paper?

Separate DN and DyLU ablations. This is the most important missing experiment. Showing DN alone (without DyLU), DyLU alone (without DN), and the combination would establish each component's individual contribution and verify the paper's claim that they are complementary rather than redundant.

Grace period ablation. The paper describes the grace period mechanism (Section 3, Figure 3) as part of the asynchronous framework but never reports the value used or shows how performance varies with $\tau_{\text{grace}}$. If the grace period is large enough to batch most workers together, the system is effectively synchronous regardless of DN and DyLU. Verifying that the reported results are not simply an artifact of a large grace period is essential for the claim that these are asynchronous training benefits.

Buffer size $N$ ablation at scale. The paper's Table 5 lists $N \in \{4, 8, 16\}$ as swept hyperparameters, but results are only reported for what appears to be $N=4$ (matching the worker count). An ablation showing how performance varies with $N$ (is $N=k$ optimal? is larger $N$ better for more heterogeneous settings?) would clarify the method's sensitivity to this parameter and provide practical guidance for deployment.

Larger-scale experiments. The largest model tested (150M parameters) is 2–3 orders of magnitude smaller than the models where distributed training is most impactful. The paper's motivation—harnessing geographically distributed compute for "even more powerful large models"—implies an ambition to scale up, but no evidence is provided that DN+DyLU works at 1B+ parameter scales. At larger scales, communication cost dominates, pseudo-gradient magnitudes may differ, and the interaction between inner and outer optimization may change. The claim that "asynchronous Local-SGD can be competitive with synchronous methods" is only established at 20M–150M scale on C4.

Real-hardware validation. All wall-clock results are from deterministic simulation. A small-scale validation on actual heterogeneous hardware (even 2–4 devices of different speeds on a local network) would substantially strengthen the credibility of the wall-clock claims and reveal any simulation-reality gaps.

Downstream task evaluation. The paper evaluates only language modeling perplexity. Standard practice in the language modeling literature includes downstream task evaluation (zero-shot or fine-tuned performance on NLP benchmarks) to verify that perplexity improvements translate to useful capabilities. A method that achieves better perplexity through optimization tricks might not improve (or could even degrade) downstream performance if the optimization dynamics produce models with different generalization properties.

Statistical reporting. The absence of error bars, confidence intervals, or multiple random seeds across all experiments is the most significant methodological weakness. With a single deterministic run per configuration, we cannot distinguish between genuine method effects and noise from data ordering, shard assignment, or pretraining initialization. This is particularly concerning for the fine-grained comparisons (e.g., DN+DyLU achieving 41.13 vs. DiLoCo's 41.35—is this a real difference or noise?).

Do the claimed contributions map to the experimental evidence?

Claim: "We identify momentum acceleration on the global parameters when worker gradients are stale as a key challenge." Well-supported by Figures 5, 6, and 7. The reversal pattern in Figure 6 and the homogeneity result in Figure 7 are the paper's strongest evidence. The analytical derivation in Equation 5 provides mechanistic insight, though it simplifies to identical gradients.

Claim: "Delayed Nesterov and Dynamic Local Updates together match synchronous DiLoCo in perplexity per update." Supported for the tested configurations (20M–150M, 4–16 workers, C4), but the evidence comes only from the combined method, not individual ablations. The matching holds consistently across heterogeneity levels, worker counts, and model sizes (Tables 1–3).

Claim: "DN+DyLU significantly surpasses DiLoCo in wall-clock time." Supported for the specific "very" heterogeneous device profile tested, but the generality is not established. The wall-clock advantage is conditional on device heterogeneity and depends on the fidelity of the deterministic timing simulation. No ablation of heterogeneity level versus wall-clock speedup is provided. The claim of "significantly surpasses" overstates the evidence for a single heterogeneity profile in simulation.

Claim: "This establishes that asynchronous Local-SGD can be competitive with synchronous methods only when the outer momentum update is carefully managed." The "only when" is too strong given the experimental design. The paper shows that DN+DyLU (with careful momentum management) achieves parity, and that naive Async. DiLoCo (without careful management) does not. It does not demonstrate that all possible asynchronous methods without careful momentum management must fail, nor does it rule out alternative asynchronous approaches (e.g., completely different outer optimizers, different communication topologies) that might work without the specific DN mechanism. The claim should be softened to "this demonstrates one effective approach to managing outer momentum in asynchronous Local-SGD" rather than asserting it as a necessary condition.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Makes the Headline Efficiency Gains Unrealized in Practice

The assumption or constraint. The paper’s compute-optimal policy depends on assigning each prompt to a difficulty bin before selecting a test-time strategy. The method used for this—generating 2048 samples per prompt and averaging either ground-truth correctness (oracle) or PRM final-answer scores (predicted)—is extraordinarily expensive. The authors acknowledge this directly in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

Generating 2048 samples per question exceeds the largest test-time budgets studied (256–512 generations) by a factor of 4–8×. In a deployment setting, the total cost would be difficulty estimation plus strategy execution, and the former dominates the latter for all but the largest per-question inference budgets.

The consequence. The reported 4× efficiency gains over best-of-N (Figures 4, 8) are computed after difficulty is known, without amortizing the cost of learning it. If difficulty estimation costs 2048 generations and the problem is then solved with a compute-optimal budget of, say, 64 generations, the total cost is 2112 generations—far worse than simply running best-of-N with 256 generations without difficulty estimation. The 4× figure is therefore an upper bound on achievable efficiency that is only realizable if difficulty can be estimated far more cheaply than the current method allows. The predicted (non-oracle) difficulty bins perform similarly to oracle bins (Figures 4, 8), which is encouraging for feasibility, but this only removes the need for ground-truth labels—it does not reduce the 2048-sample cost of computing the PRM score distribution.

What evidence exists in the paper. Section 3.2 explicitly flags this as an exploration-exploitation tradeoff and labels it "a key avenue for future work." The paper never includes difficulty estimation cost in any budget calculation. There is no experiment measuring what happens when the difficulty estimation budget is subtracted from the strategy execution budget, nor any experiment testing cheaper difficulty estimation methods (e.g., using 8 or 32 samples instead of 2048).

Mitigation status. Not addressed. The paper suggests future work on "pretraining or finetuning models to directly predict difficulty of a question" (Section 8) or adaptive estimation that amortizes difficulty assessment into the problem-solving process, but no such method is developed or evaluated. Until cheap difficulty estimation exists, the compute-optimal policy as described is a theoretical construct, not a practical deployment recipe.

---

All Results Come from a Single Benchmark on a Single Model Family

The assumption or constraint. Every experiment in the paper uses the MATH benchmark (500 test questions) with PaLM 2-S* as the base model. The authors state they "believe this model is representative of the capabilities of many contemporary LLMs" (Section 4), but no evidence is provided to support this representativeness claim. The MATH dataset consists exclusively of competition-level mathematics problems requiring multi-step symbolic reasoning, and PaLM 2-S* represents one specific model architecture, training recipe, and capability profile.

The consequence. It is unknown whether the paper's central findings generalize to: (1) other reasoning domains such as code generation, logical deduction, scientific question answering, or planning; (2) tasks requiring factual recall rather than pure inference, where the base model's failure modes may differ qualitatively; (3) other model families with different calibration properties, instruction-tuning recipes, or baseline MATH performance levels; (4) languages other than English or multimodal settings. The difficulty-dependent patterns that form the paper's core contribution—beam search helping on medium problems but hurting on easy ones (Figure 3, right), revisions dominating on easy problems while balanced parallel-sequential allocation works best on hard ones (Figure 7, right), and compute-optimal policies yielding 4× efficiency gains—may be specific to the interaction between PaLM 2-S*'s particular output distribution and the MATH benchmark's specific difficulty structure.

What evidence exists in the paper. The entire experimental section (Sections 5, 6, 7) uses MATH with PaLM 2-S*. The paper presents its findings as general principles about test-time compute scaling but provides no cross-benchmark or cross-model validation. The FLOPs-matched comparison (Section 7) uses a second PaLM 2 model variant but not a fundamentally different model family.

Mitigation status. The authors do not claim to have solved this limitation; they acknowledge the scope implicitly by describing their model choice as a belief rather than a validated fact. No systematic study of domain or model transfer is proposed as future work in Section 8, though the finding that "test-time compute amplifies existing capability but does not create it" suggests the difficulty-bin framework should conceptually extend to any base model—the bin boundaries would simply shift.

---

Test-Time Compute Cannot Help on the Hardest Problems, Creating a Fundamental Capability Ceiling

The assumption or constraint. The entire compute-optimal framework is predicated on the base model having some non-trivial probability of producing correct solutions. For the hardest problems (difficulty bin 5 in the paper's five-quintile split), the base model's pass@1 is near zero, meaning there are essentially no correct solutions in the proposal distribution to find or refine. The paper is explicit about this in Section 7:

"On the hardest questions (bins 4–5), pretraining is almost always more effective. Test-time compute provides minimal gains on problems that are fundamentally outside the base model's capability range."

The consequence. For problem classes where the base model lacks the necessary knowledge or reasoning capability, no amount of test-time compute—regardless of strategy, budget, or allocation policy—will help. This includes genuinely novel problems, out-of-distribution reasoning tasks, or domains where the model's pretraining data was insufficient. In the FLOPs-matched comparison (Figure 9, bin 5), the test-time compute scaling line is essentially flat at 0–5% accuracy across all budgets, while the larger pretrained model offers meaningful improvement. This means the approach offers no path forward for expanding the frontier of model capability—it only helps realize capability the model already possesses. For organizations deciding between investing in larger pretraining runs versus smarter inference, this creates a hard boundary: if the target task distribution includes problems the current model cannot solve at any non-trivial rate, pretraining is the only viable investment.

What evidence exists in the paper. Figure 3 (right, bin 5) shows all search methods stuck at 1–3% accuracy regardless of budget. Figure 7 (right, bin 5) shows all sequential-to-parallel ratios producing roughly 2–3% accuracy. Figure 9 shows the bin 5 scaling line flat near 0% for revisions and 0–5% for PRM search, consistently below the 14× larger model's performance. Section 7 explicitly acknowledges this boundary and discusses it in the takeaway box.

Mitigation status. The limitation is fundamental and not addressed. The paper frames it as a finding rather than a problem to solve—test-time compute and pretraining compute are complementary investments with different capability profiles. The practical implication is that systems should estimate not just per-prompt difficulty but also whether the problem is within the model's capability range at all, routing out-of-capability problems to larger models or human review. This routing mechanism is not developed.

---

The Revision Model Has a 38% Correct-to-Incorrect Reversion Rate with No Principled Fix

The assumption or constraint. The revision model is trained exclusively on sequences where all in-context answers are incorrect followed by a correct target answer (Section 6.1). This means the model never encounters examples of what to do when the current answer is already correct—it has been trained to always produce a revision, never to recognize completion. At inference time, when a revision chain produces a correct answer at step $t$, the model will unconditionally generate a revision at step $t+1$, and approximately 38% of the time this revision transforms a correct answer into an incorrect one. The paper reports this directly in Section 6.1:

"since the model was trained only on sequences where all in-context answers are incorrect... approximately 38% of correct answers get converted back to incorrect ones"

The consequence. Without mitigation, revision chains are self-limiting: accuracy improves for a few steps (Figure 6, left) but then plateaus around 24–25% because each step's gains are partially offset by regressions from previously correct answers. The paper's mitigation—selecting the best answer across the entire chain via majority voting or verifier-based selection (Section 6.1)—is a post-hoc patch that works (Figure 6, right shows sequential outperforming parallel) but is fundamentally fragile: it relies on the selection mechanism correctly identifying which revision in the chain was optimal, and it provides no guarantee that the optimal revision wasn't degraded by subsequent steps before selection occurred. If the verifier makes an error (and verifier over-optimization is itself a central problem documented in Section 5.3), the system may select a degraded answer over a correct earlier revision.

What evidence exists in the paper. Section 6.1 explicitly reports the 38% reversion rate. Figure 6 (left) shows the per-step pass@1 trajectory plateauing rather than continuing to improve, consistent with the reversion problem. The within-chain selection mechanism is described as mitigation but not systematically ablated—there is no experiment showing what accuracy would be if the reversion problem did not exist (e.g., an oracle selector that always picks the best revision in the chain).

Mitigation status. Partially addressed via majority voting and verifier-based selection across the chain, but the authors do not claim to have solved the root cause. A more principled solution—such as training the model with a "no revision needed" token, or incorporating correct-in-context examples during training—is not explored. The paper does not list this as an explicit area for future work in Section 9, which is a missed opportunity given the quantitative significance of the 38% figure.

---

The 14× Larger Model Baseline in the FLOPs-Matched Comparison Is Weakened by Two Design Choices

The assumption or constraint. The FLOPs-matched comparison in Section 7 compares PaLM 2-S* with compute-optimal test-time scaling against a model with approximately 14× more parameters. Two design choices weaken this baseline. First, the larger model scales only parameters while holding training data fixed, following the LLaMA paradigm (Touvron et al., 2023) rather than compute-optimal pretraining (Hoffmann et al., 2022) where both data and parameters are scaled equally. The authors acknowledge this explicitly:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

Second, the larger model uses only greedy decoding—no test-time compute augmentation of any kind, not even majority voting or best-of-N with a small budget. The smaller model is allowed to use sophisticated test-time strategies (beam search, revisions, compute-optimal allocation), while the larger model is evaluated with the most minimal inference procedure possible.

The consequence. Both design choices systematically favor test-time compute in the comparison. A Chinchilla-optimal larger model (scaling both parameters and data) would likely outperform a parameter-only-scaled model, and giving that larger model even a modest test-time compute budget (best-of-8, majority voting) would create a substantially stronger baseline. The reported advantages of test-time compute over pretraining—e.g., +27.8% relative improvement on easy questions at R ≪ 1 (Figure 1, top-right bar chart)—may shrink or reverse against a properly optimized pretraining baseline with fair inference-time treatment. The paper's central takeaway that "a smaller model augmented with compute-optimal test-time strategies can outperform a ~14× larger pretrained model" (Section 1) should therefore be understood as comparing against a specific, potentially suboptimal larger model, not against the best possible use of equivalent pretraining compute.

What evidence exists in the paper. Section 7 explicitly discusses the parameter-only scaling choice and frames it as leaving compute-optimal pretraining to future work. The greedy decoding choice for the larger model is not explicitly justified—it appears to be chosen for simplicity rather than as the strongest possible baseline. There is no ablation showing how the comparison would change if the larger model were given even a small test-time compute budget.

Mitigation status. The paper is transparent about the parameter-only scaling choice but does not discuss the greedy decoding asymmetry. Future work on "compute-optimal pretraining + compute-optimal inference jointly" is suggested in Section 8 only in the context of the larger model's training recipe, not in the context of giving the larger model equal access to test-time compute.

---

Sequential Revisions Introduce Latency That the Wall-Clock Analysis Ignores

The assumption or constraint. The paper measures test-time compute in "generations" (number of complete solutions sampled), which is a reasonable proxy for total FLOPs but ignores the fact that sequential revisions are inherently serial while parallel best-of-N can be executed simultaneously. A compute-optimal strategy that allocates 128 generations as 64 sequential revisions (a chain of 64 steps, each dependent on the previous) takes approximately 64× longer in wall-clock time than running 128 parallel samples on 128 independent workers. The paper's compute-optimal policies on easy problems favor purely sequential revisions (Figure 7, right, bins 1–2), and these policies would be deployed in the highest-throughput scenarios since easy problems are presumably most common.

The consequence. For latency-sensitive applications—interactive assistants, real-time decision-making, any deployment where users wait for responses—the sequential-heavy strategies selected by the compute-optimal policy may be impractical regardless of their FLOPs efficiency. A strategy that achieves the same accuracy with 4× fewer FLOPs but 64× higher latency is not a win in user-facing settings. The paper's compute-optimal allocation (Section 3.1) optimizes only for accuracy given a FLOPs budget, not for accuracy given a latency budget, creating a mismatch between the mathematical objective and real-world deployment constraints.

What evidence exists in the paper. The paper never discusses latency or wall-clock time in the context of test-time strategies. The FLOPs-matched comparison (Section 7) is purely about total computational cost. Figure 7 (right) showing optimal ratios favoring sequential revisions on easy problems does not include any latency axis. The term "wall clock time" appears only in the context of the asynchronous Local-SGD training experiments, not in the test-time compute scaling analysis.

Mitigation status. Not addressed. The paper does not mention latency as a consideration, does not include latency in the compute-optimal objective, and does not discuss how the framework would need to be modified for latency-constrained settings. A latency-aware variant would need to include a parallelism constraint in the optimization in Equation 1, potentially with different optimal policies for different latency budgets—a natural extension that the paper does not explore.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts how the field should think about asynchronous Local-SGD for language model training by isolating the outer momentum update as the single point of failure and demonstrating that the problem is not staleness in general but a specific structural mismatch in how Nesterov momentum behaves under sequential pseudo-gradient application. Before this work, the asynchronous training literature treated the staleness problem generically—stale gradients should be discounted, thresholded, or Taylor-corrected (Zheng et al., 2017; Xie et al., 2019). The paper's diagnostic experiments in Section 4 (Figures 5–7) show these approaches miss the mark because they target the wrong mechanism. The failure is not that stale gradients are poor estimates of current gradients; it is that sequential momentum application produces a fundamentally different ratio of momentum-to-gradient contribution compared to batched application (Equation 5), and this ratio cannot be fixed by per-gradient rescaling.

This is a diagnostic contribution with reframing consequences, not a paradigm shift. The paper does not introduce fundamentally new optimization theory—it identifies a specific interaction in a specific setting (Nesterov outer momentum + Local-SGD pseudo-gradients + sequential server updates) and proposes a targeted architectural fix. The reframing matters because it redirects research attention: rather than developing ever-better staleness compensation methods (which the evidence shows are ineffective for this setting), the priority should be on designing outer optimization procedures that are structurally robust to sequential application. The Delayed Nesterov algorithm is one instance of this design philosophy—decouple momentum updates from gradient updates temporally—but the broader principle (that outer optimizer design must account for the sequential application pattern) is the durable insight.

A second reframing concerns the role of local step count as a control variable. Standard Local-SGD treats the number of local steps per worker as a fixed hyperparameter $H$, identical across workers. The Dynamic Local Updates strategy demonstrates that heterogeneous local step counts can be leveraged to reduce staleness in asynchronous settings, with slower workers taking proportionally fewer steps. This transforms local step count from a static configuration parameter into a per-worker scheduling decision that trades off gradient quality against staleness. The paper's empirical finding that reduced staleness outweighs increased gradient noise at the tested scales (Tables 1–3) opens a design space where local step counts are dynamically adjusted based on worker speed, network conditions, or even the current optimization phase.

The work also resolves a tension between momentum's value and its fragility in distributed optimization. Prior work established that Nesterov momentum is the best outer optimizer for synchronous Local-SGD (DiLoCo, Douillard et al., 2023), providing substantial acceleration. This paper shows that the same momentum mechanism that provides the strongest synchronous acceleration becomes the primary liability when synchronization is removed—a finding with practical consequences for system design. It means that practitioners cannot simply take a well-tuned synchronous Local-SGD recipe and remove the barrier; the outer optimizer must be redesigned for the asynchronous setting. The paper provides one such redesign, but the principle generalizes: any outer optimizer with state (momentum, adaptive learning rates, second-moment estimates) will interact differently with sequential versus batched pseudo-gradient application, and this interaction must be analyzed and addressed.

What becomes less attractive as a research direction. The paper's empirical demonstration that polynomial discounting, thresholding, and delay compensation all fail to close the async-sync gap (Figure 8) suggests that further work on gradient-level staleness correction in the Local-SGD setting is unlikely to be productive. These methods were developed for standard asynchronous SGD where each communication carries a single gradient step; in Local-SGD, each communication carries a pseudo-gradient accumulated over $H$ local AdamW steps, which behaves differently enough that Taylor approximation and discounting heuristics break down. The paper also casts doubt on pure buffering approaches (FedBuff-style) as sufficient solutions—Async. Buffer showed promise but introduced training instability (Figure 8), suggesting that simply accumulating pseudo-gradients without intermediate parameter updates creates its own problems.

---

Follow-Up Research This Work Enables

Theory for momentum degradation under sequential Local-SGD pseudo-gradient application. The paper provides an analytical derivation (Equation 5) for the special case where all workers produce identical gradients, showing that sequential Nesterov produces a distorted momentum-to-gradient ratio compared to batched Nesterov. This derivation is illustrative but not general—it assumes perfect gradient alignment across workers and ignores the effect of gradient diversity, varying staleness, and the inner AdamW dynamics. A theoretical analysis that characterizes the expected parameter update distribution under sequential Nesterov with heterogeneous, noisy pseudo-gradients, and that derives convergence rates as a function of the buffer size $N$, the number of workers $k$, and the gradient diversity across shards, would put the empirical findings on firm footing. The theory should predict the paper's key observation: that the performance gap between synchronous and asynchronous DiLoCo is roughly constant regardless of worker heterogeneity (Table 1) but depends critically on the presence of outer momentum (Figure 6). Such an analysis would also provide guidance for setting $N$ optimally—should $N$ equal $k$, or should it depend on the variance of worker completion times?

Training from scratch with DN+DyLU at scale. The paper's experiments use a two-phase approach: 24,000 steps of single-machine pretraining followed by distributed Local-SGD fine-tuning. This follows the "post Local-SGD" paradigm (Lin et al., 2020) where Local-SGD is applied after the model has already learned basic representations. The original DiLoCo paper (Douillard et al., 2023) demonstrated that synchronous Local-SGD works for training from scratch, and an obvious stress test for DN+DyLU is whether it also works without the pretraining phase. Training a 150M–1B parameter Chinchilla-style model on C4 from random initialization using DN+DyLU with 4–16 heterogeneous workers, and comparing convergence speed and final perplexity against DiLoCo from scratch, would clarify whether the proposed fixes address only the fine-tuning dynamics or generalize to the full optimization trajectory. The paper's finding that the async-sync gap is smaller for larger models (Table 3: 2.92 points at 20M vs. 0.85 at 150M) hints that training from scratch on a small model might be the hardest test case for DN+DyLU.

Combining DN+DyLU with elastic or dropout-based worker selection. The paper's framework assumes a fixed set of $k$ workers that participate throughout training. In real geographically distributed settings, workers may join and leave dynamically (due to preemption, network outages, or voluntary resource contribution). An extension would test whether DN+DyLU is robust to dynamic worker pools: if a worker drops out, can the Delayed Nesterov buffer handle having fewer pseudo-gradients per Nesterov update? Can the data shard sampler (Equation 2) adapt when the set of available shards changes? And can DyLU's speed estimation adapt online when new workers with unknown speed profiles join? A concrete experiment would simulate a pool of 8–16 workers where 20–40% are randomly preempted and replaced at each outer round, measuring whether DN+DyLU maintains performance or whether the buffer size $N$ and data shard balancing need to be adapted dynamically. This would connect the paper's controlled simulation environment to the more chaotic reality of volunteer computing or spot-instance-based training.

Ablation of the grace period and its interaction with DN and DyLU. The paper describes the grace period mechanism (Section 3, Figure 3) as a way to batch pseudo-gradients from nearly-simultaneous worker completions, but never reports the value used or provides an ablation. This is a significant gap because the grace period directly controls how many pseudo-gradients get batched together, which interacts with the Delayed Nesterov buffer size $N$. A large grace period makes the system effectively synchronous (all workers batched, one Nesterov update per round), while a zero grace period makes it fully asynchronous. An experiment sweeping $\tau_{\text{grace}}$ from 0 to a value large enough to batch all workers, measuring perplexity per update and per wall-clock time, would reveal how much of DN+DyLU's benefit comes from the grace period batching versus from the algorithmic fixes. This would also clarify whether the wall-clock advantage claimed in Figure 2 is robust: if a large grace period is needed for good perplexity, the wall-clock advantage shrinks because fast workers wait for slow ones during the grace window. The experiment should report both metrics at each grace period setting to map the latency-quality Pareto frontier.

Testing DN+DyLU on code generation or other dense-reward tasks. The paper's language modeling experiments use next-token prediction perplexity on C4 as the sole metric. Perplexity is a continuous, dense training signal where every token contributes to the loss, and improvements in perplexity are known to sometimes not translate to downstream task performance. An important robustness check would apply DN+DyLU to a task with different optimization characteristics: code generation (where the loss is also next-token prediction but the data distribution is more structured), multi-task fine-tuning (where different shards contain qualitatively different tasks), or instruction tuning (where the loss is still autoregressive but the data has a specific prompt-response structure). If DN+DyLU works well on code but poorly on instruction tuning, that would reveal boundaries on the method's applicability and point to data distribution properties that interact with the momentum fix. The experiment should use the same model architectures (Chinchilla-style, 150M–1B parameters) and the same heterogeneous worker setup to isolate the data distribution as the independent variable.

Delayed Nesterov with adaptive buffer sizing. The paper fixes the buffer size $N$ (appears to use $N=k=4$ in most experiments) and shows it works across configurations. An adaptive variant could be more efficient: start with a small $N$ early in training when pseudo-gradients are large and noisy (frequent momentum updates to track the rapidly changing loss landscape), then increase $N$ later when gradients are smaller and more consistent (less frequent but better-averaged momentum updates). Or, $N$ could adapt to the current staleness distribution: if the server observes that most arriving pseudo-gradients have similar staleness (workers are well-aligned by DyLU), use smaller $N$; if staleness varies widely (DyLU is imperfect), use larger $N$ to average over more heterogeneous updates. An experiment could implement a simple heuristic (e.g., $N$ proportional to the variance of recent pseudo-gradient staleness) and compare against fixed $N$ on the 4-worker "very" heterogeneous setup, measuring whether adaptive $N$ improves the perplexity-vs-time tradeoff without introducing the instability seen in pure Async. Buffer.

---

Practical Applications and Downstream Use Cases

Volunteer computing and decentralized training collectives. The paper's primary motivational scenario—harnessing geographically distributed, heterogeneous compute resources—maps directly onto volunteer computing projects (e.g., folding@home-style distributed training) and decentralized training collectives where participants contribute GPU hours from personal hardware. In these settings, device heterogeneity is extreme (a mix of last-generation consumer GPUs, cloud instances, and high-end workstations), communication latency is high and variable (residential internet connections), and the straggler effect makes synchronous training infeasible. DN+DyLU offers a concrete recipe: measure each contributor's training throughput, assign local step counts proportionally via DyLU (Equation 6), and use Delayed Nesterov on a central parameter server to manage the sequential pseudo-gradient application. The paper's demonstration that DN+DyLU matches synchronous performance with 4 "very" heterogeneous workers on a 150M model (Table 3) provides an existence proof, though scaling to hundreds of volunteers with dynamic availability remains untested. The practical benefit is enabling training runs that would otherwise be impossible—not faster training per se, but training at all on donated heterogeneous hardware.

Fine-tuning on siloed or privacy-sensitive data across organizational boundaries. The paper explicitly distinguishes its setting from federated learning (Section 3: "in distributed optimization, the user has the right to choose which data shard is assigned to which worker, even dynamically"), but the techniques apply when data cannot be centralized. Consider multiple hospitals wanting to collaboratively fine-tune a medical language model on their patient records without sharing raw data. Each hospital is a worker with its own data shard and its own hardware (with varying capabilities). The data shard sampling mechanism (Equation 2) ensures balanced training across hospitals, DyLU adjusts local steps to each hospital's hardware speed, and DN prevents the outer momentum from degrading under asynchronous updates. The paper's finding that the benefit of Local-SGD over single-worker fine-tuning persists at larger model sizes (Table 3: 17.26 vs. 17.47 perplexity on 150M) suggests the collaborative approach does not sacrifice model quality compared to each hospital training independently. The key practical advantage over synchronous federated learning is that no hospital needs to wait for others, and the grace period mechanism allows opportunistic batching when two hospitals happen to finish at similar times.

Cost-efficient fine-tuning on heterogeneous cloud instances. Cloud providers offer instances with varying GPU generations and performance tiers at different price points. Organizations running regular fine-tuning jobs could mix older, cheaper instances with newer ones, using DN+DyLU to handle the speed heterogeneity rather than paying a premium for uniform high-end hardware. For a 150M model fine-tuning run, the paper's results suggest DN+DyLU with 4 heterogeneous workers achieves the same final perplexity as synchronous DiLoCo on uniform hardware (17.26 vs. 17.23, Table 3), while completing faster in wall-clock time because no instance idles waiting for slower ones (Figure 2, right panel). The cost savings come from replacing some high-end instances with cheaper alternatives while maintaining training throughput and quality. The practical adoption barrier is that DyLU requires knowing or measuring each instance's training throughput, which is straightforward with a brief profiling step at the start of training.

Edge co-training for on-device model personalization. A speculative but natural extension: a fleet of user devices (phones, laptops) collaboratively fine-tuning a shared base model on their local data without centralizing user information. Each device trains for a variable number of local steps depending on its compute capability and battery status (DyLU), sends pseudo-gradients when ready, and the server applies DN to manage the asynchronous stream of updates. The paper's finding that DN+DyLU works with up to 16 workers (Table 2) and that the data shard sampling keeps all shards progressing at similar rates (Equation 2) is encouraging for this scenario, though the paper's lack of experiments with dynamic worker pools means robustness to devices dropping offline mid-task is unproven. The practical value is personalization without privacy loss: each device's data never leaves the device, but all devices benefit from each other's training through the shared model.

---

When to Prefer This Method

The paper's own positioning is clear and grounded in its experimental results: DN+DyLU is the async Local-SGD method to use when synchronous DiLoCo is desired as the target performance but the synchronization barrier is impractical. This maps to specific decision criteria:

• Prefer Async. DN+DyLU over Sync. DiLoCo when:

  • Device speeds are heterogeneous (Figure 4 style distribution), causing significant straggler idle time that DN+DyLU eliminates (Figure 2, wall-clock panel).
  • The number of workers is moderate (4–16 based on Table 2 results, where Local-SGD outperforms single-worker large-batch training; at 16 workers the advantage narrows).
  • The same performance as DiLoCo is acceptable (Tables 1–3 show DN+DyLU matching DiLoCo across configurations within ~0.2 perplexity points).
  • A central parameter server is available to run Delayed Nesterov and manage the task scheduling (Algorithm 2).
  • Worker speeds can be measured or estimated to configure DyLU (Equation 6).

• Prefer Sync. DiLoCo (or single-worker training) over Async. DN+DyLU when:

  • Devices are homogeneous or nearly so—DN+DyLU and DiLoCo reduce to the same algorithm (Table 1, "no heterogeneity" case), so the added complexity of asynchronous scheduling and the DN buffer provides no benefit.
  • The number of workers is large (16+), where single-worker fine-tuning with proportionally larger batch size may outperform Local-SGD entirely (Table 2: single-worker 16× batch achieves 40.60 vs. DN+DyLU's 40.98).
  • Simplicity and ease of debugging are prioritized over wall-clock efficiency—the asynchronous task scheduler (Algorithm 2), grace period logic, and DN buffer introduce implementation complexity that DiLoCo's simple barrier synchronization avoids.
  • Training must be exactly reproducible across runs—the deterministic simulation in the paper abstracts away real non-determinism in asynchronous execution (network latency jitter, hardware timing variability) that makes debugging harder.

The paper does not position DN+DyLU against other asynchronous training methods (e.g., Hogwild!-style lock-free SGD, decentralized training without a parameter server) because its contribution is specifically about making Local-SGD work asynchronously with momentum-based outer optimization, not about comparing across fundamentally different distributed training paradigms. The method is a DiLoCo replacement for heterogeneous settings, not a general-purpose asynchronous training solution.