ArXiv: 2410.21680

🎯 Pitch

At 1,024 GPUs, the mean time to failure plummets to just 7.9 hours, guaranteeing that long-running ML training jobs must survive dozens of failures—yet the majority of jobs in a multi-tenant cluster are small, meaning one-size-fits-all reliability strategies leave most workloads over-provisioned. A simple 'lemon node' detection mechanism slashed large-job failures by over 30%, showing that workload-agnostic, software-level mitigation is both effective and cheaper than hardware overhauls.

1. Executive Summary

This paper analyzes 11 months of operational data from two large-scale, multi-tenant ML research clusters—RSC-1 (16k A100 GPUs) and RSC-2 (8k A100 GPUs)—spanning 4 million jobs and over 150 million GPU hours to characterize failure patterns and quantify their impact on training productivity. The authors introduce a failure taxonomy and two named core concepts—Mean Time to Failure (MTTF) projections validated against empirical data up to 4,096 GPUs, and an analytical estimator for Effective Training Time Ratio (ETTR) that models productive runtime as a function of job parameters, checkpointing overhead, and cluster failure rate. Key findings include that MTTF decreases inversely with GPU count (from 47.7 days at 8 GPUs to 7.9 hours at 1,024 GPUs on RSC-1), that hardware-attributed failures impact 19% of GPU runtime but less than 1% of jobs, and that implementing lemon node detection reduced large job failures by over 30%, establishing that while failures disproportionately affect large gang-scheduled jobs, the majority of jobs are small—motivating workload-agnostic reliability techniques that adapt across the full spectrum of job scales.

2. Context and Motivation

The Core Problem: ML Training at Scale Makes Failures Inevitable, Not Exceptional

The paper addresses a fundamental shift in the nature of infrastructure failures as ML training transitions from modest-scale experiments to industrial-scale training runs spanning thousands of GPUs. The problem is not that failures exist—they have always existed in distributed systems—but rather that the statistical properties of failures change qualitatively at scale, and the operational practices developed for smaller clusters do not extend.

The authors frame this concretely with a deceptively simple observation (Section I):

"At such scale, failures are not a matter of if, but a matter of when."

This is not merely a rhetorical flourish. The paper quantifies precisely what "when" means: for a 1,024-GPU job on RSC-1, the Mean Time to Failure is 7.9 hours (Figure 7). For a projected 16,384-GPU job, it drops to 1.8 hours. At 131,072 GPUs, it would be approximately 0.23 hours—or roughly 14 minutes. A training run that takes weeks to complete would encounter hundreds of failures. This transforms failures from rare events that can be handled manually into statistical processes that must be managed automatically and accounted for in system design.

Why is this gap pressing now? The introduction points to the acceleration of LLM development—LLaMa, Megascale, Gemini, GPT-4—as concrete drivers (Section I). These models require training on tens of thousands of accelerators, and the infrastructure investments to support them (16,000–24,000 GPU clusters with high-bandwidth interconnects) represent hundreds of millions of dollars. When a single GPU failure can cause a 4,096-GPU job to restart (due to gang scheduling semantics, Section II-A), the economic cost of inefficiency is enormous: 19% of GPU runtime on RSC-1 is impacted by hardware-attributed failures (Figure 3). This is lost compute that could otherwise be training models.

The Gang Scheduling Amplification Effect

A critical architectural detail that amplifies the reliability challenge is gang scheduling (Section II-A). In ML training, all GPUs allocated to a job must work in lockstep—the SPMD (Single Program Multiple Data) model with Bulk-Synchronous Parallel semantics means that every rank must participate in every collective operation. If one GPU fails, the entire job stalls. As Figure 1 illustrates, a single task failure forces a complete re-allocation of the job. This is fundamentally different from elastic workloads where failed workers can be replaced independently.

The consequence is what the authors call a quadratic goodput cost as a function of job size (Section III, Preemptions and Failure Cascades): larger jobs (1) lose more work per failure because more GPUs sit idle during recovery and (2) fail more frequently because the probability of any component failing scales with the number of components. This is the core scaling challenge that motivates the entire analysis.

The Research Cluster Diversity Gap

A significant portion of the paper's motivation comes from what the authors observe as a gap in the literature's coverage of workload diversity. Existing studies on ML infrastructure failures and LLM training reliability—the authors cite Megascale (Jiang et al., 2024), the Gemini technical report, LLaMa 3 (Meta, 2024), and Unicron (He et al., 2023)—focus predominantly on specialized clusters optimized for a single massive workload. These papers study failures in environments where essentially all resources are devoted to one enormous training job or a small number of very large jobs.

The authors argue this specialization is not representative of research clusters, which they define as environments that must "cater to both large- and small-scale jobs" and "demonstrate diversity in infrastructure needs that is rarely observed in more specialized clusters devoted to only LLMs" (Section I). The data supports this: over 90% of jobs use fewer than 8 GPUs (less than one server), but these small jobs represent less than 10% of GPU time (Figure 6, Observation 7). Meanwhile, 4,096-GPU jobs constitute less than 1% of jobs but consume 12% of GPU resources. This bimodal distribution—many small development and evaluation jobs alongside a few enormous training jobs—creates scheduling dynamics, failure cascades, and optimization tradeoffs that single-workload analyses miss entirely.

Where Prior Work Falls Short

The paper identifies four specific limitations in existing approaches:

1. Scale of analysis. Prior ML infrastructure studies—the authors cite Jeon et al. (2019) and Li et al. (2022)—analyzed clusters with jobs reaching "into the tens of GPUs" (Section VI). The current paper operates at orders of magnitude larger scale (up to 4,096 GPUs per job), which is not just a quantitative difference. At thousand-GPU scale, the failure rate becomes high enough that it fundamentally shapes scheduling strategy, checkpoint frequency, and the statistical properties of job completion.

2. Failure attribution remains noisy and understudied. The paper makes a pointed observation about the difficulty of root-causing failures (Section II-E, Observation 3). An NCCL timeout—one of the most common failure symptoms—can be caused by a network link failure, a GPU hardware error, a user software bug, a filesystem stall, or a deadlocked rank. The symptom is proximal; the cause is often distal. Prior work acknowledges this ambiguity but does not provide a systematic taxonomy or a methodology for differential diagnosis across failure domains. The paper's taxonomy (Table I) explicitly maps symptoms to possible failure domains (user code, system software, hardware infrastructure), arguing that the only way to narrow the hypothesis space is to "rule out unlikely causes" using overlapping health signals.

3. No unified metric for productive training time. The paper notes that previous work uses metrics like job slowdown (Harchol-Balter et al., 2002) and Goodput (Google Cloud's "Runtime Goodput"), but these either fail to account for unproductive scheduled time (e.g., catching up from the last checkpoint after a restart) or conflate wait time with wasted compute. The paper introduces Effective Training Time Ratio (ETTR) as a metric that captures the full pipeline: queue time, restart overhead, checkpoint overhead, and time lost between the most recent checkpoint and a failure. Critically, the authors provide an analytical formulation that allows ETTR to be estimated from aggregate cluster statistics (Equation 1 in Section III), making it a deployable tool rather than just a measurement concept.

4. Correlated and recurring failures are not systematically addressed. The paper observes that certain nodes exhibit above-average failure rates—so-called lemon nodes—due to hardware degradation, misconfiguration, or aging (Section IV-A). While prior work has studied GPU error patterns (Tiwari et al., 2015) and silent data corruption (Dixit et al., 2021; Bonderson, 2021), the paper argues that existing health check mechanisms (which detect failures at the moment they occur) are insufficient for identifying nodes that cause repeated failures. A node that fails once, gets remediated, passes health checks, and is returned to the scheduling pool only to fail again creates an attractor for failures that disproportionately impacts large jobs (which have higher probability of landing on at least one such node). The paper's lemon detection pipeline (Observation 11) addresses this gap by using historical signals (XID error counts, exclusion counts, repair tickets) to identify and isolate these nodes before they cause cascading failures.

How This Paper Positions Itself

The paper positions itself at the intersection of three communities: ML infrastructure operators (who need actionable reliability metrics and mitigations), systems researchers (who need quantitative characterization of failure patterns at scale), and ML practitioners (who need to understand how cluster reliability affects their training productivity). The framing is explicitly operational rather than theoretical—the authors describe their work as sharing "infrastructure experience" and "lessons we have learned in mitigating failures at scale" (Section I).

A key positioning choice is the paper's emphasis on workload-agnostic techniques. Unlike LLM-specific approaches that assume particular parallelization strategies (e.g., Megatron-LM's tensor parallelism, pipeline parallelism mappings to network topology), the paper argues that research clusters must support "constantly changing workloads with potentially unforeseen needs" (Section II). This leads the authors to advocate for reliability mechanisms that operate at the infrastructure level (health checks, lemon detection, adaptive routing) rather than requiring application-level modifications. The scheduler, the health check system, and the network fabric should handle failures transparently, allowing ML researchers to focus on model development rather than distributed systems debugging.

The paper also positions its findings as forward-looking projections, not just retrospective analysis. The MTTF extrapolations (Figure 7), the ETTR estimator (Equation 1), and the checkpoint requirement analysis (Figure 10) are designed to help infrastructure planners answer questions like: What checkpoint overhead would be needed to run a 12,000-GPU job with 90% ETTR on our current cluster? The answer—checkpoint write overhead must be on the order of 10 seconds, or the failure rate must improve by roughly 6×—is a concrete, actionable finding that connects the empirical analysis to future investment decisions.

Finally, the paper distinguishes itself from pure failure analysis papers by also presenting validated mitigations: adaptive routing improved bandwidth under link errors by maintaining 100+ GB/s where non-adaptive routing dropped to near zero (Figure 12a), and lemon node detection reduced large job failures from 14% to 4% (Observation 11). This operational grounding—showing that the characterized problems can be addressed—strengthens the paper's argument that reliability is a solvable engineering challenge rather than an inherent limitation of scale.

Tension Between Generality and Specialization

An implicit tension running through the paper's motivation deserves attention. The authors critique specialized LLM clusters for being insufficiently general, yet they also acknowledge that the most severe reliability challenges come from the largest jobs—which are precisely the LLM training runs that specialized clusters optimize for. The compute profile in Figure 6 shows that 66% of GPU hours on RSC-1 come from jobs of 256+ GPUs, and the goodput loss analysis (Figure 8) shows that nearly all lost compute from failures is attributable to jobs at the 1,024–4,096 GPU scale. This raises the question: if the reliability problem is dominated by large jobs, why not optimize specifically for them?

The paper's answer is pragmatic rather than dogmatic. The failure cascades analysis (Section III, Preemptions and Failure Cascades) shows that 16% of total goodput loss from hardware failures on RSC-1 comes from second-order preemptions of smaller jobs when a large failed job is rescheduled. This means that even if the goal is solely to optimize large job throughput, the scheduler dynamics and the reliability of the entire cluster matter—small jobs cannot simply be ignored because their preemption and requeueing create fragmentation and churn that degrade overall efficiency. The workload-agnostic approach is thus not an ideological preference but a recognition that in a multi-tenant system, reliability is a system-level property, not a property of any single job.

3. Technical Approach

3.1 Reader Orientation

This is an operational analysis and measurement paper whose core idea is that reliability in large-scale ML training clusters must be understood through a combination of quantitative failure characterization, predictive modeling (MTTF and ETTR), and infrastructure-level mitigations (health checks, lemon detection, adaptive routing) that operate transparently across diverse workloads rather than being optimized for any single job type.

The paper does not propose a single novel algorithm or system. Instead, it constructs a measurement and modeling pipeline that ingests 11 months of scheduler logs, health check events, and hardware telemetry from two production clusters, then produces: (1) a failure taxonomy that classifies symptoms by failure domain, (2) validated MTTF projections that predict failure frequency at any scale, (3) an analytical ETTR estimator that models productive training time as a function of cluster parameters, and (4) concrete, deployed mitigations whose efficacy is quantified. The "system" is the cluster infrastructure itself—the paper's contribution is the framework for understanding and improving it.

3.2 Big-Picture Architecture (Diagram in Words)

The system under analysis has five interacting layers, with data flowing from the bottom up and control decisions flowing from the top down:

  1. Hardware substrate (compute + network + storage): DGX A100 servers (8× A100 GPUs each, NVSwitch interconnected), connected via a rail-optimized Infiniband backend fabric arranged in a hierarchical topology (servers → racks → pods → spine switches), with a front-end Ethernet network for control-plane traffic and three storage offerings (NFS, AirStore cache, ObjectStore). This is the physical layer where component failures originate.

  2. Health check infrastructure: A set of periodic checks (every 5 minutes) and job prolog/epilog checks that probe individual nodes for known failure signatures—GPU XID errors, PCIe link status, ECC memory errors, NVLink errors, Infiniband link errors, filesystem mount availability, block device errors, and service liveness. Each check returns success, failure, or warning; high-severity failures immediately trigger node removal and job rescheduling. This layer produces the raw failure telemetry.

  3. Scheduler (Slurm) with gang scheduling: Users submit jobs (shell scripts or Python via submitit) with GPU count requests. Slurm allocates contiguous resources respecting the physical network topology, enforces project-level quotas, and supports preemption after 2 hours with a 7-day maximum job lifetime. When a health check fails on an allocated node, Slurm marks the job as NODE_FAIL and automatically requeues it. This layer generates the job lifecycle events (COMPLETED, FAILED, NODE_FAIL, PREEMPTED, etc.) that form the paper's primary analysis corpus.

  4. Failure attribution and lemon detection pipeline: A post-hoc analysis system that ingests health check events and job termination records, applies heuristics to attribute failures to likely causes (matching health check firings within a ±5–10 minute window of job failure), and runs a lemon node detection algorithm that uses 28-day rolling windows of historical signals (XID error counts, exclusion counts, repair ticket counts, multi-node failure rates) to identify nodes with statistically elevated failure rates. This layer produces the categorized failure data in Figures 4–5 and the lemon node identifications in Table II.

  5. Analytical modeling layer (MTTF + ETTR): Offline models that consume aggregate statistics from the attribution pipeline—node-level failure rate per cluster ($r_f$), job size distribution, queue time distributions, checkpoint overhead estimates—and produce MTTF projections (Figure 7) and ETTR estimates (Equation 1, Figure 9). These models are validated against empirical job run data and used to project requirements for future scales (Figure 10).

Information flow: Hardware failures occur → health checks detect them → Slurm terminates affected jobs and removes nodes → the failure attribution pipeline associates health check events with job failures and classifies them by cause → the lemon detection pipeline identifies repeating-offender nodes using historical data → aggregate statistics feed into the MTTF and ETTR models → the models inform checkpoint strategy, infrastructure investment, and scheduling policy decisions → these decisions feed back into cluster configuration (health check tuning, lemon node remediation, adaptive routing enablement).

3.3 Roadmap for the Deep Dive

  • First, the health check infrastructure and failure attribution mechanism, because all failure data—and therefore all subsequent analysis—depends on how failures are detected and classified. Understanding the detection window, severity levels, and attribution heuristics is essential to interpreting the failure rates in Figures 4–5 and the MTTF projections in Figure 7.

  • Second, the MTTF measurement and projection methodology, including how confidence intervals are constructed, why the theoretical model $\text{MTTF} \propto 1/N_{\text{gpus}}$ is expected to hold, and how the empirical validation against jobs up to 4,096 GPUs works. This is the foundation for understanding failure scaling behavior.

  • Third, the ETTR analytical model, which is the paper's most complex technical contribution. We walk through the derivation of Equation 1 in detail—defining productive runtime, unproductive runtime, queue time, and their relationship to checkpoint interval, restart overhead, and node failure rate—and show how it simplifies under the Daly-Young optimal checkpointing assumption.

  • Fourth, the lemon node detection pipeline, because it is a deployed system whose design choices (signal selection, threshold setting, accuracy evaluation) illustrate the gap between simple health checks and the need for historical pattern analysis.

  • Fifth, the adaptive routing evaluation, which demonstrates a complementary reliability mechanism at the network layer and provides the paper's only controlled experimental results (NCCL All-Reduce benchmarks with and without AR under injected bit errors).

  • Finally, the scheduler-level analysis of failure cascades, which connects individual node failures to cluster-wide goodput loss through preemption chains and demonstrates why workload diversity matters for reliability optimization.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical measurement and modeling paper whose technical contribution is the pipeline for collecting, attributing, and modeling failure data at scale, combined with validated infrastructure mitigations. The core methodological insight is that reliability in large ML clusters must be understood as a statistical process operating at multiple timescales—seconds (health check periodicity), hours (MTTF for large jobs), days (lemon node detection windows), and months (failure mode evolution as hardware ages and software updates roll out)—and that effective mitigations must operate at each of these timescales.

Health Check Infrastructure and Failure Attribution

The health check system is the primary data source for all failure analysis in the paper. It determines what is counted as a failure, how failures are classified, and—critically—what failures are not observed. Understanding its design is therefore foundational to interpreting every quantitative result.

Periodic health checks. Each node in both clusters runs a set of health checks every 5 minutes. Each check is a script that probes a specific hardware or software component and returns a status code: success (the component is healthy), failure (the component is definitively unhealthy), or warning (an intermediate state). The checks cover:

  • GPU errors: XID error codes from NVIDIA GPUs, which encode specific hardware fault types—ECC memory errors (correctable and uncorrectable), NVLink errors, GPU falling off the PCIe bus (XID 79), GSP RPC timeouts (a firmware communication failure), row-remap failures (indicating exhausted spare memory rows), corrupted buffer streams, and context switch timeout errors. Each XID code is a distinct failure signature.
  • PCIe health: PCI link errors and PCIe bus accessibility. Note that 57% of PCIe failures on RSC-1 (37% on RSC-2) co-occur with XID 79 (GPU falling off the bus), because a PCIe failure makes the GPU inaccessible regardless of whether the GPU itself generated the XID event.
  • Network health: Infiniband link errors and link flaps. These are tracked at the NIC level.
  • Memory health: ECC errors on host CPU memory (DIMMs), separate from GPU HBM errors.
  • Storage and filesystem: Block device errors, missing mount points (e.g., NFS mounts becoming stale), and filesystem accessibility.
  • System services: Slurm daemon liveness, LDAP availability, and other infrastructure services.
  • Miscellaneous: BIOS version mismatches, IPMI power supply status, IPMI critical interrupts, and memory size mismatches.

Severity-based response. Health checks are tuned to have a false positive rate below 1%—meaning fewer than 1% of jobs that complete successfully experienced a failed health check during their runtime (Section II-C). This calibration is critical because false positives would cause unnecessary job terminations and node removals, directly reducing cluster capacity and goodput. The cost is that some true failures may be missed (false negatives), but the authors argue this is acceptable because 1) overlapping checks provide redundant coverage, and 2) NODE_FAIL acts as a catch-all for unresponsive nodes.

When a check fails at high severity (GPU not accessible, NVLink error, uncorrectable ECC, failed row-remaps, PCIe or IB link errors, block device errors, missing mount points), the scheduler handler immediately removes the node from the scheduling pool and terminates all jobs running on that node. When a check fails at lower severity, the node is flagged for remediation but is allowed to finish its currently running jobs before being removed—the reasoning being that lower-severity issues may not immediately impact job correctness.

Job prolog/epilog checks. In addition to periodic checks, Slurm runs health checks immediately before a job starts (prolog) and immediately after it finishes (epilog). The prolog check ensures that a node that has degraded between periodic checks is not given to a new job. The epilog check detects failures that occurred during the job's execution but were not caught by periodic checks.

Failure attribution heuristics. When a job terminates with status FAILED (application returned non-zero exit code) or NODE_FAIL (Slurm detected a node failure), the attribution pipeline examines all health check events within a window spanning 10 minutes before the job's start to 5 minutes after the job's end. If a health check failure is found within this window that could plausibly cause the job failure, the failure is attributed to that cause. This temporal window is a design choice: too narrow, and failures detected shortly before job start (which could have caused the job failure) would be missed; too wide, and unrelated health check events would be falsely associated with the job failure.

The paper explicitly notes that failure attribution is inherently noisy (Observation 3). An NCCL timeout—one of the most common job failure symptoms—can be caused by:

  • A genuinely failed network link (hardware)
  • A GPU that fell off the bus (hardware, detectable via PCIe health check)
  • A rank that deadlocked due to a user code bug (application)
  • A filesystem stall that caused one rank to delay entering a collective (infrastructure)
  • A driver or firmware bug (system software)

The health check-based attribution can only identify causes that are detectable at the node level. Failures that occur entirely within the application layer (user code bugs, deadlocks) or within network switches (which lack per-node health checks) will appear as "unattributed" FAILED jobs or, if the node becomes unresponsive, as NODE_FAIL events with no associated health check.

The NODE_FAIL catch-all. NODE_FAIL is a Slurm status indicating that the scheduler lost contact with a node during job execution—typically because the node crashed, became unresponsive, or experienced a kernel panic. Some NODE_FAIL events have no associated health check failure (the node simply disappeared), while others co-occur with health check events. The authors report (Section III) that unattributed NODE_FAIL is a non-trivial category, with some NODE_FAIL events "not associated with any health checks... likely because the node itself became unresponsive."

Overlapping signals as a feature, not a bug. The health check system is designed with intentional redundancy. For example, a PCIe failure that causes a GPU to become inaccessible will trigger both the PCIe health check and potentially an XID 79 event from the GPU, plus possibly an IPMI critical interrupt from the system management controller. The paper reports that 3% and 5% of hardware failures on RSC-1 and RSC-2 respectively have co-occurring events from different checks. The authors view this positively: "even if one check does not fire when it should, another overlapping check would hopefully catch the failure." This redundancy is a practical acknowledgment that individual health checks are imperfect and that failure detection at scale requires defense in depth.

Why this design over alternatives? The paper contrasts its approach with two alternatives: (1) trying to make all failures transparently recoverable by the infrastructure (e.g., live migration of GPU state, transparent network path failover) and (2) leaving failure handling entirely to the application. The first is rejected as having "overhead associated with transparently recovering from them" that is prohibitive at scale. The second is rejected because it places an unreasonable burden on ML researchers, who would need to implement sophisticated fault tolerance for every new model. The health check approach is a cooperative recovery strategy: "the application is still responsible for correctly implementing checkpoint and resume logic," but the infrastructure guarantees that when a failure occurs, the job will be restarted on known-healthy hardware—preventing the worst-case scenario of repeated failures on the same defective node (Observation 2: "no second job failure from a bad node").

Mean Time to Failure (MTTF) Measurement and Projection

MTTF is the foundational reliability metric that all other models depend on. The paper measures it empirically and validates a theoretical model that enables projection to scales not yet observed.

Definition and measurement. MTTF is defined as the total measured system time divided by the number of failures (Section II-D). For a job of a given GPU count, the paper counts failures as:

  • Jobs with status NODE_FAIL
  • Jobs with status FAILED where a health check failure was detected within the attribution window (±10 minutes before job start to +5 minutes after job end)

The total system time is the sum over all jobs in the size category of their runtime (in days) multiplied by the number of nodes allocated. The MTTF is then:

MTTF=∑iruntimei×nodesinumber of failures in category\text{MTTF} = \frac{\sum_i \text{runtime}_i \times \text{nodes}_i}{\text{number of failures in category}}

where $i$ indexes all jobs in the size category, $\text{runtime}_i$ is the wallclock time job $i$ ran before terminating, and $\text{nodes}_i$ is the number of nodes (servers, not GPUs) allocated to job $i$.

What it computes: the average interval between infrastructure-attributed failures for jobs of a given scale, expressed in hours or days. For example, at 1,024 GPUs (128 nodes), the MTTF is 7.9 hours on RSC-1, meaning that on average, a training job of this size will be interrupted by a hardware or infrastructure failure every 7.9 hours of wallclock time.

Why this form: measuring MTTF by aggregating over all jobs in a size category rather than tracking individual long-running jobs is necessary because most ML jobs are short—development jobs, evaluation runs, and hyperparameter sweeps typically last minutes to hours. A single multi-week training run is rare. By pooling all jobs of a given size, the measurement achieves statistical significance despite the heterogeneity of individual job runtimes.

Confidence intervals via Gamma distribution. The paper reports 90% confidence intervals around MTTF estimates (visible as the shaded bands in Figure 7). These are generated by fitting a Gamma distribution to the observed failure times. The Gamma distribution is appropriate for modeling waiting times between events in a Poisson process (which failures approximately follow when aggregated over many independent components). The confidence intervals widen at smaller job sizes because there are fewer observed failures in those categories, and they also widen on RSC-2 due to its lower overall failure rate providing fewer data points.

Theoretical MTTF model. The paper posits that MTTF should scale inversely with the number of nodes (or equivalently, GPUs) in a job:

MTTFtheoretical=(Nnodes⋅rf)−1\text{MTTF}_{\text{theoretical}} = (N_{\text{nodes}} \cdot r_f)^{-1}

where $N_{\text{nodes}}$ is the number of nodes (servers) allocated to the job and $r_f$ is the cluster-wide failure rate per node-day, computed by counting all failures across all jobs larger than 128 GPUs and dividing by the total node-days of runtime in that set.

What it computes: the expected MTTF for a job of any size, assuming (1) failures occur independently across nodes with constant rate $r_f$, and (2) any single node failure causes the entire job to fail (gang scheduling). Under these assumptions, the job-level failure rate is $N_{\text{nodes}} \cdot r_f$ (the sum of per-node failure rates), and the MTTF is the reciprocal of this rate.

Why this form: this is the simplest model consistent with independent component failures. The linear scaling $\text{MTTF} \propto 1/N_{\text{nodes}}$ is the direct consequence of gang scheduling: if each node fails independently at rate $r_f$, then the probability that any of $N_{\text{nodes}}$ nodes fails in a given time interval is approximately $N_{\text{nodes}} \cdot r_f \cdot \Delta t$ (for small $\Delta t$), and the expected time until the first failure is the reciprocal. Any deviation from this scaling would indicate correlated failures (multiple nodes failing together), which would cause faster-than-linear MTTF degradation. The paper uses this model as a baseline to test whether observed MTTF follows the independence assumption.

Node-level vs. GPU-level analysis. The MTTF model is expressed in terms of nodes (servers) rather than GPUs because "failures may manifest for RSC-1 over the last year" at the node level, reflecting the fact that many failure modes (PCIe bus errors, filesystem mounts, network link failures) affect the entire server, not individual GPUs. However, the paper also reports GPU-hour-normalized failure rates in Figure 4, where the per-GPU hourly rate for each failure category is computed by dividing the number of attributed failures by the total GPU-hours across the cluster. This allows comparison of failure rates between clusters of different sizes and between different failure categories on a common scale.

Empirical validation. Figure 7 plots both the observed MTTF (with 90% confidence intervals) and the theoretical curve $(N_{\text{nodes}} r_f)^{-1}$ for both clusters. On RSC-1, with $r_f = 6.50$ failures per thousand node-days, the theoretical curve closely matches observed MTTF for jobs of 32 to 4,096 GPUs. The model slightly underestimates MTTF (predicts more failures than observed) for jobs at the 8–16 GPU scale, which the authors attribute to "experimental usage patterns that cause correlated NODE_FAIL"—meaning that at very small scales, some jobs experience multiple failures not because of independent hardware faults but because of job-specific configuration issues or user behavior.

On RSC-2, with $r_f = 2.34$ failures per thousand node-days, the empirical data is noisier and generally shows slightly higher MTTF (better reliability) than the theoretical projection. The authors attribute this to differences in workload—RSC-2's vision-focused workloads may stress different hardware components than RSC-1's LLM training, resulting in different failure profiles.

Projection to larger scales. Using the validated model, the paper projects MTTF at scales not currently observed: 1.8 hours for 16,384 GPUs and 0.23 hours (approximately 14 minutes) for 131,072 GPUs. These projections are linear extrapolations of $\text{MTTF} = (N_{\text{nodes}} r_f)^{-1}$ and assume that the failure rate per node remains constant as the cluster size increases. The paper does not discuss potential deviations from this assumption at extreme scale—for instance, if power delivery or cooling infrastructure becomes a bottleneck, failures could become correlated across nodes, violating the independence assumption and causing even shorter MTTF than projected.

Effective Training Time Ratio (ETTR) Analytical Model

The ETTR model is the paper's most significant analytical contribution, providing a closed-form expression that relates expected training productivity to measurable cluster parameters. We walk through the derivation, the assumptions, and the validation.

Definition of ETTR. For a complete training run (which may span multiple scheduler jobs due to preemptions and failures), ETTR is:

ETTR=RW=RR+U+Q\text{ETTR} = \frac{R}{W} = \frac{R}{R + U + Q}

where $R$ is the productive runtime (wallclock time during which the model is actually training—forward and backward passes executing), $U$ is the unproductive scheduled time (wallclock time when the job is running but not making forward progress—checkpoint writes, restart initialization, and time spent re-training from the most recent checkpoint after a failure), and $Q$ is the total queue time (wallclock time when the job is eligible to run but waiting for resources). The wallclock time $W = R + U + Q$ is the total elapsed time from job submission to completion.

What ETTR captures that simpler metrics miss. ETTR ranges from 0 (no progress made—the job is perpetually queued or restarting) to 1 (100% productive utilization—every second of wallclock time is spent training). Unlike utilization (which only measures scheduled time), ETTR accounts for queue time. Unlike goodput (which measures productive work per unit time), ETTR normalizes by wallclock time and thus captures the user-facing experience of training duration. The paper differentiates ETTR from the canonical "job slowdown" metric (Harchol-Balter et al., 2002), defined as the ratio of wallclock time to scheduled time, because slowdown ignores unproductive scheduled time (overhead) and inverts the ratio. ETTR also differs from Google Cloud's "Runtime Goodput" in that ETTR explicitly models queue wait time in multi-tenant clusters.

Deriving the expected value. The derivation proceeds in steps. First, the paper defines an intermediate variable $S = (U + Q)/R$, so that $\text{ETTR} = 1/(1 + S)$. By Jensen's inequality, $\mathbb{E}[\text{ETTR}] \geq 1/(1 + \mathbb{E}[S])$, giving a lower bound on expected ETTR that is analytically tractable.

Queue time component. Let $N_{\text{int}}$ be the number of times the job is interrupted (due to failures; preemptions are ignored for high-priority jobs). The total queue time is:

Q=q0+∑j=1NintqjQ = q_0 + \sum_{j=1}^{N_{\text{int}}} q_j

where $q_0$ is the initial wait time after job submission and each $q_j$ is the wait time after the $j$-th interruption. Assuming queue times are independent and identically distributed (i.i.d.) with mean $\bar{q}$, the expected total queue time is:

E[Q]=(1+E[Nint])⋅qˉ\mathbb{E}[Q] = (1 + \mathbb{E}[N_{\text{int}}]) \cdot \bar{q}

Unproductive runtime component. The unproductive time per job segment is the sum of: (1) the fixed restart overhead $u_0$ (loading checkpoints, initializing NCCL, re-establishing distributed state), (2) the time spent writing checkpoints during the segment ($N_{\text{cp}} \cdot w_{\text{cp}}$, where $w_{\text{cp}}$ is the synchronous write cost per checkpoint and $N_{\text{cp}}$ is the number of checkpoints written), and (3) the time lost between the most recent checkpoint and the failure or preemption that terminated the segment. Assuming failure timestamps are uncorrelated with checkpoint timestamps and that the checkpoint interval $\Delta t_{\text{cp}}$ is much smaller than the MTTF, the expected lost work per failure is approximately $\Delta t_{\text{cp}} / 2$—the average distance from a random failure time to the previous checkpoint.

The total expected unproductive time is:

E[U]=(E[Nint]+1)⋅u0+E[Nint]⋅Δtcp2+R⋅wcpΔtcp\mathbb{E}[U] = (\mathbb{E}[N_{\text{int}}] + 1) \cdot u_0 + \mathbb{E}[N_{\text{int}}] \cdot \frac{\Delta t_{\text{cp}}}{2} + \frac{R \cdot w_{\text{cp}}}{\Delta t_{\text{cp}}}

where the last term accounts for checkpoint writes during productive training: with $R$ seconds of productive training and checkpoints every $\Delta t_{\text{cp}}$ seconds, there are $R / \Delta t_{\text{cp}}$ checkpoint writes, each costing $w_{\text{cp}}$ seconds.

Expected number of interruptions. Under the assumption that failures occur independently at rate $r_f$ per node-day and that the job uses $N_{\text{nodes}}$ nodes, the failure rate for the job is $N_{\text{nodes}} r_f$ failures per day of runtime. Since failures can occur during both productive and unproductive time, the expected number of failures during the entire training run satisfies:

E[Nint]=Nnodesrf⋅(R+E[U])\mathbb{E}[N_{\text{int}}] = N_{\text{nodes}} r_f \cdot (R + \mathbb{E}[U])

Substituting the expression for $\mathbb{E}[U]$ and solving for $\mathbb{E}[N_{\text{int}}]$ yields (under the assumption $\Delta t_{\text{cp}}/2 + u_0 \ll (N_{\text{nodes}} r_f)^{-1}$):

E[Nint]≈RNnodesrf⋅1+u0R+wcpΔtcp1−Nnodesrf(u0+Δtcp2)\mathbb{E}[N_{\text{int}}] \approx R N_{\text{nodes}} r_f \cdot \frac{1 + \frac{u_0}{R} + \frac{w_{\text{cp}}}{\Delta t_{\text{cp}}}}{1 - N_{\text{nodes}} r_f (u_0 + \frac{\Delta t_{\text{cp}}}{2})}

What this equation says: the expected number of interruptions scales linearly with the productive runtime $R$, linearly with the number of nodes $N_{\text{nodes}}$, and linearly with the per-node failure rate $r_f$. The numerator's $1 + u_0/R + w_{\text{cp}}/\Delta t_{\text{cp}}$ accounts for the fact that failures can also occur during overhead time, increasing the total failure count slightly above $R N_{\text{nodes}} r_f$. The denominator accounts for a subtle effect: when the MTTF $= (N_{\text{nodes}} r_f)^{-1}$ is not vastly larger than the overhead per failure, the overhead itself consumes time during which additional failures can occur, creating a compounding effect.

Full expected ETTR expression. Substituting all components into $\mathbb{E}[\text{ETTR}] \geq 1/(1 + \mathbb{E}[S])$ gives the paper's Equation 1:

E[ETTR]≳1−Nnodesrf(u0+Δtcp2)1+u0+qˉR+wcpΔtcp+Nnodesrfqˉ(1+wcpΔtcp−Δtcp2R)\mathbb{E}[\text{ETTR}] \gtrsim \frac{1 - N_{\text{nodes}} r_f (u_0 + \frac{\Delta t_{\text{cp}}}{2})}{1 + \frac{u_0 + \bar{q}}{R} + \frac{w_{\text{cp}}}{\Delta t_{\text{cp}}} + N_{\text{nodes}} r_f \bar{q} (1 + \frac{w_{\text{cp}}}{\Delta t_{\text{cp}}} - \frac{\Delta t_{\text{cp}}}{2R})}

What it computes: given a training job with productive runtime requirement $R$, node count $N_{\text{nodes}}$, restart overhead $u_0$, checkpoint write cost $w_{\text{cp}}$, checkpoint interval $\Delta t_{\text{cp}}$, mean queue time $\bar{q}$, and cluster failure rate $r_f$, this expression estimates the lower bound on the fraction of wallclock time that will be productively spent training.

Why this form: the numerator $1 - N_{\text{nodes}} r_f (u_0 + \Delta t_{\text{cp}}/2)$ represents the fraction of time not lost to catching up from failures. The denominator accounts for all sources of overhead: restart initialization ($u_0/R$), queue waiting ($\bar{q}/R$), checkpoint writes ($w_{\text{cp}}/\Delta t_{\text{cp}}$), and the interaction between queue time and failure rate ($N_{\text{nodes}} r_f \bar{q} (\ldots)$). The inequality $\gtrsim$ reflects the use of Jensen's inequality, meaning the true expected ETTR is at least this value.

Simplification for high-priority, long-running jobs. For the highest-priority jobs on RSC-1 and RSC-2 (where queue time is negligible compared to training time, $\bar{q} \approx 0$, and the productive runtime is much larger than overheads, $R \gg u_0 + \Delta t_{\text{cp}}/2$), the expression simplifies to Equation 2:

E[ETTR]≈1−Nnodesrf(u0+Δtcp2)1+wcpΔtcp\mathbb{E}[\text{ETTR}] \approx \frac{1 - N_{\text{nodes}} r_f (u_0 + \frac{\Delta t_{\text{cp}}}{2})}{1 + \frac{w_{\text{cp}}}{\Delta t_{\text{cp}}}}

What this simpler form reveals: ETTR is primarily determined by two ratios. The first is the failure-induced overhead ratio in the numerator: $N_{\text{nodes}} r_f (u_0 + \Delta t_{\text{cp}}/2)$ is the fraction of time lost to restart overhead and catch-up training. When this approaches 1, ETTR collapses—the job spends more time recovering from failures than making progress. The second is the checkpoint overhead ratio in the denominator: $w_{\text{cp}}/\Delta t_{\text{cp}}$ is the fraction of productive time consumed by writing checkpoints. Smaller checkpoint intervals reduce failure-induced loss (smaller $\Delta t_{\text{cp}}/2$), but increase checkpoint overhead (larger $w_{\text{cp}}/\Delta t_{\text{cp}}$). This is the classic tradeoff formalized by Daly and Young.

Daly-Young optimal checkpoint interval. Under the same limiting assumptions (negligible queue time, $\Delta t_{\text{cp}} \gg u_0, w_{\text{cp}}$, and $R \gg u_0, \bar{q}, \Delta t_{\text{cp}}, w_{\text{cp}}$), the checkpoint interval that maximizes ETTR is:

Δtcp∗=2wcpNnodesrf\Delta t^*_{\text{cp}} = \sqrt{\frac{2 w_{\text{cp}}}{N_{\text{nodes}} r_f}}

What it computes: the checkpoint interval (in seconds) that balances the tradeoff between lost work due to failures (which favors frequent checkpointing—small $\Delta t_{\text{cp}}$) and checkpoint write overhead (which favors infrequent checkpointing—large $\Delta t_{\text{cp}}$). The optimal interval grows with the square root of checkpoint cost and shrinks with the square root of the failure rate.

Why this form: the square-root dependence arises because the lost-work cost per failure is $\Delta t_{\text{cp}}/2$ (linear in $\Delta t_{\text{cp}}$), while the checkpoint overhead rate is $w_{\text{cp}}/\Delta t_{\text{cp}}$ (inversely proportional to $\Delta t_{\text{cp}}$). Minimizing the sum of these two terms—a linear term and a reciprocal term—yields a square-root optimum. This is the classic Daly-Young result, derived independently by Young (1974) and Daly (2006).

Validation against empirical data (Figure 9). To validate the model, the paper identifies "job runs" on both clusters—logical training jobs that span multiple Slurm jobs due to failures and preemptions—with at least 48 hours of total training time and highest priority. For each job run, the authors compute the observed ETTR as $R / W$, assuming Daly-Young optimal checkpointing with $u_0 = w_{\text{cp}} = 5$ minutes and using the cluster-wide $r_f$ (6.50 failures per thousand node-days for RSC-1, 2.34 for RSC-2). The predicted $\mathbb{E}[\text{ETTR}]$ is computed from the analytical model using the same parameters plus the average queue time for jobs of that size.

The results (Figure 9) show that predicted and observed ETTR agree "fairly well," with observed ETTR being generally lower than predicted. The authors attribute this conservative bias to the fact that all non-COMPLETED job terminations are treated as infrastructure failures in the empirical calculation, whereas some of these are likely user-initiated cancellations or application bugs. On RSC-1, the largest job runs (>1,024 GPUs) show systematically higher ETTR than predicted, which is explained by Slurm scheduling configurations that give priority preference to larger jobs, resulting in shorter-than-average queue times not captured by the global mean $\bar{q}$.

Projection to 12,288 GPUs (Figure 10). The analytical model is used to generate contour plots showing the checkpoint write overhead required to achieve various ETTR levels (0.70, 0.90, 0.95, 0.99) as a function of cluster failure rate, for a hypothetical 12,288-GPU training run with $u_0 = 5$ minutes. The key finding: to achieve ETTR of 0.9 at RSC-1's current failure rate (6.50 per thousand node-days), checkpoint write overhead $w_{\text{cp}}$ must be approximately 10 seconds—roughly 30× faster than the 5-minute assumption used for current jobs. Alternatively, the failure rate must improve to approximately 1 failure per thousand node-days (a 6.5× improvement). The paper notes that 10-second checkpoint writes are "achievable with asynchronous checkpoint writing strategies," citing Gemini (Wang et al., 2023).

Lemon Node Detection Pipeline

While health checks identify nodes that are currently failing, lemon nodes—servers with above-average failure rates due to degradation, misconfiguration, or manufacturing defects—require historical analysis to detect. A node that fails once, gets remediated, passes health checks, and returns to the scheduling pool only to fail again creates a pattern of repeated job failures that health checks alone cannot prevent. The lemon detection pipeline addresses this gap.

Problem formalization. The goal is to identify nodes where $\mathbb{P}(\text{job fails} | \text{scheduled on node}) \gg \mathbb{P}(\text{job fails} | \text{scheduled on random node})$. Because individual job failures are rare (hardware-attributed failures affect <1% of jobs), detecting this elevation requires aggregating signals over a period long enough to achieve statistical significance—the paper uses a 28-day rolling window.

Detection signals. Seven signals are tracked per node over the 28-day window:

  1. excl_jobid_count: Number of distinct jobs that explicitly excluded this node (users can specify node exclusion lists in Slurm, typically after experiencing a failure on that node).
  2. xid_cnt: Number of unique XID errors recorded on the node—each XID code counted once, regardless of how many times it occurred, to avoid a single persistent error dominating the signal.
  3. tickets: Count of repair tickets created for the node.
  4. out_count: Number of times the node was taken out of the scheduling pool (e.g., for remediation).
  5. multi_node_node_fails: Number of multi-node job failures (jobs spanning more than one server) attributed to this node.
  6. single_node_node_fails: Number of single-node job failures attributed to this node.
  7. single_node_node_failure_rate: Rate of single-node failures on this node (failures per unit time), capturing the failure frequency rather than just the count.

Threshold setting and calibration (Figure 11). The paper plots the cumulative distribution function (CDF) of each signal across all nodes over a 28-day snapshot. Most signals are highly sparse—the majority of nodes have zero occurrences of most signals. For example, the excl_jobid_count signal is shown to have "a large number of nodes... excluded by at least one job," but "did not have a strong correlation with node failures," meaning user exclusion behavior is noisy and not a reliable indicator of actual hardware defects.

Thresholds for lemon classification were set "manually based on accuracy and false positive rate of predicted lemon nodes." The paper does not specify exact threshold values, but reports that the system identified 40 faulty nodes (24 on RSC-1, 16 on RSC-2) with "more than 85% accuracy," meaning that more than 85% of nodes flagged as lemons were genuinely problematic (verified by subsequent diagnosis or repair).

Root cause distribution (Table II). Of the identified lemon nodes, the root causes (determined post-hoc through repair diagnosis) were: GPU issues (28.2%), DIMM (host memory) issues (20.5%), PCIe issues (15.4%), and a long tail of other causes—EUD (7.7%), NIC (7.7%), BIOS (5.1%), PSU (2.6%), Optics CPU (2.6%), and unknown (7.7%). The dominance of GPU and memory issues is consistent with the failure rate breakdown in Figure 4, where GPU ECC errors and PCIe issues are top contributors.

Impact quantification. The identified lemon nodes represented 1.2% of RSC-1's node footprint and 1.7% of RSC-2's. Removing them led to a reduction in large job (512+ GPU) failures from 14% to 4%—a 10 percentage point absolute reduction, which corresponds to a relative reduction of over 70% (from 14% to 4%). The paper frames this as a "10% reduction in large job failures, from 14% to 4%," but carefully parsing: the 14% and 4% appear to be failure rates (percentage of large jobs that fail), and the change from 14% to 4% represents the impact of removing 1.2% of nodes.

Design rationale. The manual threshold approach (rather than a learned classifier) is chosen because the sparsity of the signals (Figure 11 shows that the vast majority of nodes have zero or near-zero values for most features) makes supervised learning challenging—there are too few positive examples to train a reliable model. The 28-day window is chosen to balance two factors: too short a window (e.g., 7 days) and statistical noise dominates—a node that experienced a single unlucky failure looks identical to a genuinely defective node; too long a window (e.g., 90 days) and the system is slow to respond to newly degrading nodes.

Adaptive Routing Evaluation

The adaptive routing analysis provides the paper's only controlled experimental results (as opposed to observational data), demonstrating a network-layer reliability mechanism complementary to the node-level health checks.

Problem. Infiniband links can experience errors ranging from high bit error rates (BER) to flapping (intermittent up/down transitions) to permanent failure. Because physical link replacement requires manual intervention and can take days, fabric-level resilience mechanisms are essential. The default Infiniband self-healing mechanism (SHIELD) coordinates switch-level responses to failed links but may apply conservative thresholds for declaring a link down—allowing degraded links that cause protocol-level retransmissions and bandwidth loss to remain in use.

Adaptive routing mechanism. AR dynamically adjusts per-packet routing decisions based on real-time port load. Unlike static routing (which maps each source-destination pair to a fixed path), AR allows switches to select among multiple viable output ports for each packet, choosing the least congested or avoiding ports connected to degraded links. The mechanism operates at the switch level, requiring no application changes—it is transparent to NCCL collectives and PyTorch.

Experiment 1: resilience to link errors (Figure 12a). The authors inject bit errors (BER) into the fabric using the mlxreg tool to modify port registers, then run an All-Reduce benchmark from NCCL-Tests across 512 GPUs with and without AR enabled. The results show five iterations of the benchmark: with AR, bandwidth stabilizes at approximately 140–200 GB/s, while without AR, bandwidth drops to approximately 100 GB/s—a loss of "as much as 50-75% bandwidth" in the worst case, matching the paper's bring-up phase observation. AR achieves this by routing around the links with injected errors, preventing them from bottlenecking any single NCCL ring.

Experiment 2: performance under contention (Figure 12b). The authors simultaneously run 64 groups of All-Reduce, each across 2 nodes (16 GPUs), creating fabric-wide contention from multiple NCCL rings competing for link bandwidth. With AR enabled, per-group bandwidth shows lower variance (narrower vertical spread in the plot) and higher median bandwidth (averaging around 160–180 GB/s vs. 120–160 GB/s without AR). This is because AR balances traffic across all available links rather than statically mapping each ring to a path that may overlap with other rings on congested links.

Why this matters. The adaptive routing evaluation demonstrates that not all reliability mechanisms need to be reactive (detect failure, then respond). In-network adaptation can mask failures and congestion from applications entirely, preventing the failure from ever manifesting as a job-level event. This complements the health check approach: health checks handle server-level failures that require node removal, while AR handles network-level degradation that can be routed around without job interruption.

Failure Cascade Analysis

The paper quantifies a second-order effect of job failures that is often overlooked: failure cascades through the scheduler.

Mechanism. When a large, high-priority job fails (NODE_FAIL or FAILED with attributed hardware cause), Slurm automatically requeues it. Because the job is high priority, it immediately preempts lower-priority jobs to secure the required resources. Each preempted job loses the work done since its last checkpoint. The paper documents a concrete worst-case example: "a 1024 GPU job NODE_FAIL and subsequently requeue 35 times, causing a total of 548 preemptions (over 7k GPUs)."

Goodput loss accounting (Figure 8). The paper estimates the goodput lost to failures and second-order preemptions by assuming all jobs checkpoint hourly, giving an average of 30 minutes of lost work per preempted job. The lost goodput per job is computed as:

Lost goodput=min⁡(runtime,30 minutes)×GPUs allocated\text{Lost goodput} = \min(\text{runtime}, \text{30 minutes}) \times \text{GPUs allocated}

This is conservative: if a job ran for only 10 minutes since its last checkpoint, only 10 minutes of work is lost, not 30. The goodput loss is then expressed as a fraction of total cluster compute (GPU-hours over the analysis period).

Results (Figure 8). On RSC-1, 0.47% of total cluster compute was lost by the largest job category (2,049–4,096 GPUs) due to attributed failures and preemption cascades. Of this, approximately 16% (roughly 0.08 percentage points) is from second-order preemptions of smaller jobs, while the remainder is direct failure impact on the large jobs themselves. On RSC-2, the total goodput loss is an order of magnitude smaller (0.04% peak for 257–512 GPU jobs), reflecting both its lower failure rate ($r_f = 2.34$ vs. 6.50) and its different job makeup (fewer very large jobs, as shown in Figure 6).

Implication for workload-agnostic design. The 16% figure for second-order preemption costs is the key evidence supporting the paper's argument that optimizing only for large jobs is insufficient. Even if one's sole objective is maximizing large-job throughput, the reliability of small jobs matters indirectly: when small jobs are preempted and requeued, they consume scheduling slots, create resource fragmentation, and delay the availability of resources for future large jobs. The workload-agnostic infrastructure approach—health checks, lemon detection, and adaptive routing that benefit all job sizes—is thus justified not by fairness but by overall system efficiency.

4. Key Insights and Innovations

Innovation 1: Reliability as a Statistical Process Operating at Multiple Timescales, Not a Binary State

The paper's deepest conceptual move is reframing cluster reliability from a binary property ("the cluster is up" vs. "the cluster is down") to a statistical process with characteristic timescales that vary by job scale and failure mode. This is not merely an empirical observation—it is a diagnostic lens that changes what questions an operator asks and what metrics they track.

Prior to this work, the dominant framing in ML infrastructure reliability—both in the research literature and in operational practice—treated failures as discrete, exceptional events to be eliminated. The goal was to make the cluster "reliable enough" that failures became negligible. This framing is visible in the health-check-as-firewall approach (catch failures before they affect jobs) and in the emphasis on mean-time-between-failures (MTBF) as a single-number summary of cluster health. What this framing misses is that at scale, failures are not exceptional—they are continuous. The question is not whether failures will occur during a training run, but how often and with what recovery cost.

The paper's reframing is most visible in three places:

First, the MTTF scaling analysis (Figure 7). By showing that MTTF decreases inversely with GPU count—from 47.7 days at 8 GPUs to 7.9 hours at 1,024 GPUs to a projected 14 minutes at 131,072 GPUs—the paper transforms failure from a qualitative concern into a quantitative design parameter. A team planning a 16,384-GPU training run now knows they will, on average, experience a hardware-induced interruption every 1.8 hours. This is not a "risk" to be mitigated; it is a schedule to be managed. The checkpoint strategy, the restart overhead budget, and the expected training duration all become functions of this number.

This is fundamentally different from how prior LLM training papers (Megascale, LLaMa 3, Gemini) discuss reliability. Those papers report failure statistics, but they treat them as incidents overcome—"we experienced X failures and handled them by doing Y." The current paper treats failure as a property of the system that can be modeled, projected, and designed around before any job is submitted. The analytical ETTR model (Equation 1) operationalizes this by making failure rate an explicit input to the training productivity calculation, alongside checkpoint cost and restart overhead.

Second, the multi-timescale analysis (Figure 5). The paper shows that failure modes evolve on multiple timescales: seconds (health check periodicity), hours (MTTF for large jobs), days (lemon node detection windows), and months (failure mode trends as hardware ages and software updates roll out). The 11-month failure rate evolution plot (Figure 5) shows that the dominant failure category shifts over time—XID errors from a driver bug dominate in late 2023, mount point failures become prominent in spring 2024 after a new health check is added, IB link failures spike in early summer 2024 from a handful of offending nodes. This temporal heterogeneity means that no static reliability configuration is sufficient. The cluster is a living system whose failure profile changes as hardware ages and software evolves, and reliability engineering must be a continuous process, not a one-time setup.

This is a significant departure from prior work like Jeon et al. (2019) and Li et al. (2022), which characterize failure patterns at a single snapshot in time and treat the failure distribution as stationary. The paper's Observation 6—"Cluster failures are dynamic and reducing cluster failure rate is a continuous battle"—is not a throwaway line. It is a conceptual conclusion that implies reliability monitoring must be longitudinal, attribution must be continuously updated, and mitigation strategies must be adaptive.

Third, the ETTR metric itself (Section II-D). By defining ETTR as $R / (R + U + Q)$—productive time over total wallclock time, including queue waiting—the paper unifies three previously separate concerns: hardware reliability (captured in $U$, the unproductive overhead), scheduling policy (captured in $Q$, the queue time), and application-level checkpoint strategy (captured in the relationship between $\Delta t_{\text{cp}}$ and $w_{\text{cp}}$). Prior metrics like utilization only capture $(R + U) / \text{total scheduled time}$, ignoring queue time. Goodput captures productive work per unit time but abstracts away per-job overhead. ETTR's innovation is to collapse the entire training experience—submission, queuing, training, failing, restarting, checkpointing—into a single interpretable number between 0 and 1.

The significance of this unification is practical: it forces tradeoffs to be evaluated holistically. Should you reduce checkpoint interval to lower failure-induced loss (better $\Delta t_{\text{cp}}/2$ term)? That increases checkpoint overhead (worse $w_{\text{cp}}/\Delta t_{\text{cp}}$ term). Should you invest in faster storage to reduce checkpoint write time? That improves the denominator but doesn't affect the numerator's failure-loss term. Should you improve cluster health checks to reduce $r_f$? That affects both the numerator (less catch-up training) and the expected number of interruptions (fewer restarts). ETTR makes these tradeoffs explicit and quantifiable in a way that no prior metric could.

Innovation 2: The Lemon Node Concept—Recurring Failure as a Distinct Failure Mode Requiring Historical Detection

The concept of a lemon node—a server with statistically elevated failure rates that cannot be identified by point-in-time health checks—is a diagnostic innovation that bridges the gap between transient fault detection and systematic hardware degradation. While the underlying idea (some nodes fail more than others) is not new in the systems reliability literature, the paper's contribution is in operationalizing the concept for ML training clusters at scale: identifying which signals are predictive, calibrating detection thresholds from sparse telemetry, and quantifying the impact on large-job completion rates.

Prior work on GPU reliability (Tiwari et al., 2015) characterized error patterns—row-remapping, ECC errors, falling off the bus—but did so retrospectively, analyzing failure logs to understand what had already happened. The lemon detection pipeline in this paper is prospective: it uses a 28-day rolling window of signals to identify nodes that will likely cause future failures. This is a subtle but important shift. The health check infrastructure described in Section II-C is reactive—it detects a failure that is happening right now. Lemon detection is predictive—it identifies a node that, while currently passing health checks, has a historical pattern of failure that makes it a poor scheduling candidate.

The key intellectual move is recognizing that the temporal pattern of failures, not just their existence, carries diagnostic information. A node that fails once and never again is statistically unremarkable—it experienced a transient fault, was remediated, and returned to service. A node that fails five times in a month, each time passing health checks between failures, is systematically defective—perhaps a degraded solder joint, a marginal DRAM cell, or a firmware interaction that manifests only under specific workload conditions. Distinguishing these two cases requires historical data, not just instantaneous measurements.

The signal selection in Section IV-A reveals a critical design insight: the most useful signals are not the ones that seem most directly related to failure (like XID error counts) but rather the ones that capture the system's response to perceived failure—node exclusion counts (excl_jobid_count), repair tickets (tickets), and times taken out of service (out_count). These signals amplify the signal-to-noise ratio because they aggregate human judgment (users and operators who decided "this node seems problematic") alongside automated telemetry. The paper's finding that excl_jobid_count "did not have a strong correlation with node failures" despite many nodes being excluded by at least one job illustrates that user behavior is noisy—but when that noise accumulates across many users and jobs, it becomes a useful feature for lemon classification.

The impact quantification—removing 1.2% of RSC-1's nodes reduced large job failures from 14% to 4%—is disproportionate. This is the paper's strongest evidence that the failure distribution is not uniform across nodes. If all nodes failed at the same rate, removing 1.2% of them would reduce total failures by roughly 1.2%. The observed 71% reduction (from 14% to 4%) implies that the removed nodes were responsible for the vast majority of large-job failures. This is a finding with significant implications: it suggests that reliability improvement efforts should focus on identifying and removing the worst nodes rather than uniformly improving all nodes. A 10x improvement in the failure rate of the median node would be far less impactful than identifying and removing the 99th-percentile lemons.

Innovation 3: The ETTR Analytical Model as an Inference-Time (Planning-Time) Analog of Scaling Laws

The ETTR analytical model is not just a metric definition—it is a closed-form design tool that enables infrastructure planners to make quantitative predictions about training productivity without running experiments. In this sense, it plays a role analogous to what the Chinchilla scaling laws (Hoffmann et al., 2022) did for pretraining compute allocation: it provides a formula that takes measurable inputs and produces actionable predictions about optimal resource allocation.

The innovation here is not the mathematics—the Daly-Young optimal checkpointing result has been known since the 1970s, and queuing theory models of job slowdown are well-established. The innovation is in assembling these components into a single, interpretable framework that connects cluster-level failure statistics to per-job productivity and validating it against real job run data (Figure 9). This closes a gap between the reliability engineering literature (which provides models for checkpoint optimization and failure prediction) and the ML infrastructure literature (which reports failure observations but rarely connects them to training productivity predictions).

The contour plot in Figure 10 is the model's most elegant output. It answers the question: What would it take to run a 12,288-GPU job with 90% ETTR? The answer—checkpoint write overhead must drop to ~10 seconds, or the failure rate must improve by ~6×—is not obvious from raw data. It emerges from the interaction of the model's terms: as GPU count increases, the failure rate term $N_{\text{nodes}} r_f$ grows linearly, forcing the $\Delta t_{\text{cp}}/2$ lost-work term and the $u_0$ restart overhead term to become dominant. You can compensate by writing checkpoints faster (reducing $\Delta t^*_{\text{cp}}$ without increasing $w_{\text{cp}}/\Delta t_{\text{cp}}$) or by making the cluster more reliable (reducing $r_f$). The 10-second checkpoint target is a radical requirement—30× faster than the 5-minute assumption in the validation—and it points directly to asynchronous checkpointing as a necessary technology for exascale training.

What makes this model intellectually distinctive, as opposed to merely useful, is that it converts reliability from a qualitative concern into a quantitative optimization variable. Before this paper, a team planning a large training run might know that "failures will be a problem at that scale" and budget some slack for restarts. After this paper, that team can plug in their expected job size, their cluster's measured $r_f$, their checkpoint system's measured $w_{\text{cp}}$, and their restart procedure's measured $u_0$, and compute their expected ETTR. If the number is too low, they can quantitatively evaluate options: invest in faster storage (reduce $w_{\text{cp}}$), optimize NCCL initialization (reduce $u_0$), implement lemon detection (reduce effective $r_f$), or accept a longer training timeline. This transforms reliability from a source of uncertainty into a design parameter—exactly the transformation that scaling laws achieved for pretraining compute.

The validation in Figure 9 is crucial because it demonstrates that the model works with real, messy cluster data, not idealized assumptions. The predicted $\mathbb{E}[\text{ETTR}]$ agrees with observed ETTR to within the model's conservative bias (observed is lower because all non-COMPLETED terminations are treated as infrastructure failures). The systematic deviation for the largest jobs on RSC-1—where actual queue times are shorter than average—is correctly diagnosed as a priority effect, not a model failure. This kind of validation, where deviations from the model are themselves informative, is characteristic of robust analytical frameworks.

Innovation 4: Failure Cascade Quantification as Evidence for Workload-Agnostic Design

The paper's analysis of failure cascades—the 16% of goodput loss on RSC-1 that comes from preemptions of smaller jobs when large failed jobs are rescheduled—provides the empirical foundation for one of the paper's most important architectural arguments: that cluster reliability is a system-level property that cannot be optimized by focusing solely on the largest jobs.

This finding is subtle because it contradicts a natural engineering instinct. Figure 6 shows that 66% of GPU hours on RSC-1 come from jobs of 256+ GPUs, and Figure 8 shows that nearly all lost goodput from failures is attributable to jobs at the 1,024–4,096 GPU scale. A naive reading of these data would conclude: "Optimize for large jobs; small jobs don't matter." The cascade analysis shows why this is wrong.

The mechanism is a scheduler externality. When a 1,024-GPU job fails and requeues, it preempts potentially dozens of smaller jobs to secure its resource allocation. Each preempted small job loses its progress since the last checkpoint (estimated at 30 minutes of lost work on average, based on hourly checkpointing). These losses are individually small—a 1-GPU job losing 30 minutes of work is negligible compared to a 1,024-GPU job restarting entirely—but they accumulate across hundreds of preemptions. The paper's worst-case example (35 requeues causing 548 preemptions across 7,000+ GPUs) illustrates the multiplier effect.

The intellectual contribution is not the observation that preemption causes lost work—that is obvious. It is the quantification that this lost work is a meaningful fraction of total failure impact (16% on RSC-1) and the implication that optimizing the cluster for large jobs alone leaves a significant source of inefficiency unaddressed. This is an argument for workload-agnostic reliability mechanisms that benefit all job sizes equally—health checks that apply to every node regardless of who is using it, lemon detection that removes defective nodes before any job lands on them, adaptive routing that improves network performance for all traffic. These mechanisms are not "fairness" measures; they are efficiency measures that work by reducing the total failure rate $r_f$, which benefits large jobs most (since their failure rate scales with $N_{\text{nodes}}$) while also preventing the cascade costs that fragment the cluster and delay future large-job scheduling.

This finding also connects to the paper's broader argument about research cluster diversity (Section I). If all jobs were large (as in a specialized LLM training cluster), preemption cascades from large-to-large jobs would be even more destructive, potentially causing cascading failures that bring down the entire cluster's efficiency. The presence of small jobs acts as a buffer—they absorb preemptions that would otherwise hit other large jobs, but at the cost of their own lost work. Understanding this dynamic is essential for scheduler design in multi-tenant ML clusters, and it is a dimension of the reliability problem that single-workload analyses completely miss.

5. Experimental Analysis

Evaluation Methodology

  • Dataset. The analysis is based on 11 months of operational data from two production ML research clusters—RSC-1 and RSC-2—comprising scheduler logs (Slurm job lifecycle events), node-level health check telemetry, hardware error logs (XID codes, PCIe events, ECC errors), and network fabric telemetry. The dataset spans 4 million jobs and over 150 million A100 GPU hours (Section III, Abstract). There is no train/test split; this is an observational measurement study where all data is used for characterization, and cross-validation (two-fold within difficulty bins) is not applicable—the paper instead uses 90% confidence intervals around MTTF estimates via Gamma distribution fitting (Figure 7) and 28-day rolling windows for lemon detection (Section IV-A).

  • Base clusters. Two NVIDIA DGX A100-based clusters: RSC-1 (16,000 A100 GPUs, general ML workloads including prominent LLM training) and RSC-2 (8,000 A100 GPUs, vision-focused workloads). Each server (node) contains dual AMD Rome 7742 CPUs and 8× A100 80GB GPUs connected via NVSwitch, with a rail-optimized Infiniband backend fabric for model gradient exchange and an Ethernet front-end network for control-plane and storage traffic (Section II-B). The clusters run Slurm as the scheduler with gang scheduling semantics, supporting jobs from 1 to 4,096+ GPUs, with 2-hour minimum time-to-preemption and 7-day maximum job lifetime (Section II-A). The authors chose these clusters because they are "state-of-the-art" research environments operating at >80% utilization with workload diversity spanning vision, language, and mixed-modality models—making them representative of production ML infrastructure at scale (Section I).

  • Metrics. Three primary metrics are defined and used throughout (Section II-D):

    • Mean Time to Failure (MTTF): Total measured system time divided by number of failures, expressed in hours or days for jobs of a given GPU count. Computed by summing runtime × nodes over all jobs in a size category and dividing by the count of attributed hardware failures in that category. MTTF ranges from 0 to ∞, decreasing as failure rate increases.
    • Effective Training Time Ratio (ETTR): Productive runtime divided by total wallclock time (productive + unproductive scheduled time + queue time). Ranges from 0 (no progress) to 1 (100% productive). Productive runtime is defined as scheduled time during which forward/backward passes are executing; unproductive runtime includes restart initialization, checkpoint writes, and time spent catching up from the most recent checkpoint after a failure. The paper provides both an analytical expected-value formulation (Equation 1) and an empirical measurement from job run data (Figure 9).
    • Goodput: The amount of productive work completed in aggregate per unit time, normalized by maximum possible goodput to produce a utilization in [0,1]. Used specifically to quantify the cluster-wide impact of failures and preemption cascades (Figure 8), expressed as fraction of total cluster compute (GPU-hours) wasted.

  • Baselines. This is an observational measurement study rather than a methods comparison paper, so there are no algorithmic baselines in the traditional sense. The paper does compare:

    • Observed MTTF vs. theoretical projection: The theoretical model MTTF = (N_nodes × r_f)^(-1) where r_f is the cluster-wide failure rate per node-day, computed from all jobs >128 GPUs (Figure 7). This serves as a null model assuming independent, identically distributed node failures.
    • Observed ETTR vs. analytical prediction: The expected value formulation E[ETTR] from Equation 1, parameterized with cluster-level averages for failure rate and queue time, compared against empirically measured ETTR from actual job runs with ≥48 hours of total training time and highest priority (Figure 9).
    • With vs. without adaptive routing: Controlled experiments running NCCL All-Reduce benchmarks with AR enabled and disabled, both under injected bit errors and under fabric-wide contention from 64 concurrent NCCL rings (Figure 12).
    • Before vs. after lemon node detection: Large job (512+ GPU) failure rates before and after implementing the lemon detection and removal pipeline (Observation 11).

  • Generation budget / compute accounting. The paper measures compute in units appropriate to each analysis layer (Section III):

    • GPU-hours: Total allocated GPU time, used for normalizing failure rates (Figure 4) and quantifying goodput loss (Figure 8). One GPU-hour equals one A100 GPU allocated for one hour.
    • Node-days: Number of servers (nodes) × runtime in days, used for computing the per-node failure rate r_f and for MTTF calculations (Figure 7). One node-day equals one 8-GPU server allocated for one day.
    • Job counts and runtimes: Directly from Slurm logs, used for scheduler job status breakdown (Figure 3) and job size distribution (Figure 6).
    • For the adaptive routing experiments (Figure 12), bandwidth in GB/s is measured using the NCCL-Tests All-Reduce benchmark across 512 GPUs and 64 pairs of 16 GPUs.

  • Cross-validation / statistical protocol. The paper employs several statistical rigor measures:

    • 90% confidence intervals around MTTF: Generated by fitting a Gamma distribution to observed failure times, shown as shaded bands in Figure 7. The Gamma distribution is appropriate for waiting times between events in a Poisson process, which failures approximately follow when aggregated over many independent components.
    • 28-day rolling windows for lemon detection: Feature values (XID counts, exclusion counts, failure rates) are computed over a sliding 28-day window to achieve statistical significance while remaining responsive to newly degrading nodes (Section IV-A). Thresholds were manually calibrated based on accuracy and false positive rate, with reported >85% accuracy on identified lemon nodes.
    • 30-day rolling averages for failure rate evolution: Figure 5 smooths daily failure rates using a 30-day window to reveal long-term trends while suppressing daily noise.
    • Conservative failure attribution: Every non-COMPLETED job termination in the empirical ETTR calculation is treated as an infrastructure failure, producing a lower bound on measured ETTR that the authors acknowledge is an underestimate (Section III, Analysis of ETTR Results).
    • Monte Carlo validation of analytical approximations: For large, long-running hypothetical jobs (e.g., 8k GPUs), the analytical E[ETTR] approximation is compared against Monte Carlo simulations of the various expectations involved, with agreement within ~5% (Section III, Approximating E[ETTR] analytically).

Main Quantitative Results

The paper's experimental results are organized around five axes: failure characterization and attribution, MTTF scaling, ETTR estimation and validation, lemon node detection efficacy, and adaptive routing performance.

Scheduler Job Status and Failure Attribution

Scheduler-level breakdown (Figure 3). Across all jobs on RSC-1 over 11 months, 59.8% completed successfully, 23.7% failed with application-level non-zero exit codes (FAILED status), 9.8% were preempted, 4.2% were cancelled, 1.6% were requeued, 0.6% timed out, 0.1% ran out of memory, and 0.1% failed due to node failure (NODE_FAIL). An additional 0.1% of jobs were marked FAILED with hardware attribution. Together, infrastructure-attributed failures (FAILED (HW) + NODE_FAIL (HW)) represent only 0.2% of jobs by count, but impact 18.7% of GPU runtime—a ~94× amplification factor reflecting that hardware failures disproportionately affect large, long-running jobs.

Failure rates by cause (Figure 4). On RSC-1, the per-GPU hourly failure rates, normalized by total GPU-hours across the cluster, reveal the dominant attributed causes: unattributed NODE_FAIL (1.6 × 10⁻⁵ per GPU-hour, the single largest category), filesystem mount failures (8.6 × 10⁻⁶), Infiniband link failures (6.9 × 10⁻⁶), GPU GSP RPC timeouts (4.8 × 10⁻⁶), PCIe errors (2.2 × 10⁻⁶), and GPU ECC memory errors (1.4 × 10⁻⁶). Multiple co-occurring failure events account for 1.9 × 10⁻⁶ (two events) and 1.1 × 10⁻⁷ (three events). GPU row-remap failures, missing GPUs, NVLink errors, and corrupted buffer streams each contribute between 5.7 × 10⁻⁷ and 1.1 × 10⁻⁶.

On RSC-2, unattributed NODE_FAIL dominates at 4.2 × 10⁻⁵ per GPU-hour—nearly 3× higher than RSC-1—while other categories are generally lower: IB link failures (6.9 × 10⁻⁶, matching RSC-1), GPU ECC memory (3.7 × 10⁻⁶), filesystem mounts (3.0 × 10⁻⁶). The paper notes that RSC-2 is "slightly more reliable" overall than RSC-1, partly attributable to different workloads triggering different failure causes.

Co-occurrence statistics. 3% and 5% of hardware failures on RSC-1 and RSC-2 respectively have co-occurring events of similar priority. PCIe errors co-occur with XID 79 (GPU falling off the bus) in 57% of cases on RSC-1 (37% on RSC-2), and 43% (63%) of PCIe errors have all three event types: PCIe error, XID 79, and IPMI Critical Interrupt. 2% (6%) of IB link failures co-occur with GPU failures such as falling off the bus, suggesting correlated failures through the PCIe subsystem. These co-occurrence rates validate the paper's design choice of redundant health checks—"even if one check does not fire when it should, another overlapping check would hopefully catch the failure" (Section II-C).

Temporal evolution of failure rate (Figure 5). The 30-day rolling average of failure rate for jobs ≥128 GPUs on RSC-1 varies by an order of magnitude over the 11-month period, from approximately 2.5 failures per 1000 node-days to spikes reaching ~17.5 failures per 1000 node-days. The dominant failure mode shifts seasonally: XID errors from a driver bug dominate in late 2023, mount point failures become prominent in spring 2024 after a new health check is added, and IB link failures spike in early summer 2024 from a small number of offending nodes. The addition of new health checks (annotated as vertical lines) causes apparent increases in failure rate because previously undetected failure modes become visible. RSC-2 shows similar dynamics at roughly half the overall failure rate, with peaks reaching ~10 failures per 1000 node-days.

Job Size Distribution and Workload Diversity

Job count vs. GPU-hour distribution (Figure 6). On RSC-1, over 40% of jobs use a single GPU, and more than 90% use fewer than 8 GPUs (less than one server). By GPU hours consumed, however, 66% of total compute goes to jobs of 256+ GPUs, with 12% consumed by the largest category (2,049–4,096 GPUs). On RSC-2, the distribution skews even more toward small jobs—a higher fraction of 1-GPU jobs—with 52% of GPU hours going to 256+ GPU jobs. The largest jobs on RSC-2 reach 1,024–2,048 GPUs, smaller than RSC-1's maximum of 4,096. This bimodal distribution (many tiny development/evaluation jobs, few enormous training jobs) is the empirical foundation for the paper's argument about workload diversity motivating workload-agnostic reliability mechanisms.

MTTF Analysis and Scaling

Empirical MTTF by job size (Figure 7). On RSC-1, MTTF scales from 47.7 days at 8 GPUs (1 node) to 7.9 hours at 1,024 GPUs (128 nodes), representing a ~145× decrease in MTTF for a 128× increase in scale—closely matching the theoretical inverse-linear relationship. At 4,096 GPUs, the observed MTTF is approximately 2–3 hours (read from the figure; the paper quotes the projection rather than the empirical value at this scale). On RSC-2, MTTF decreases from approximately 60–80 days at 8 GPUs to roughly 12–24 hours at 1,024 GPUs. The 90% confidence intervals widen at smaller scales (where fewer failures occur, increasing statistical uncertainty) and are overall wider on RSC-2 (which has a lower total number of failures).

Validation of theoretical model. The theoretical projection MTTF = (N_nodes × r_f)^(-1) with r_f = 6.50 failures per thousand node-days for RSC-1 closely matches empirical MTTF for jobs of 32–4,096 GPUs. The model slightly underestimates MTTF (predicts more failures than observed) for 8–16 GPU jobs, attributed to "experimental usage patterns that cause correlated NODE_FAIL" (Section III, MTTF Decreases at Scale). For RSC-2 (r_f = 2.34), the empirical data shows higher MTTF than the projection, with more fluctuation at 16 GPUs due to "a group of related jobs causing multiple NODE_FAIL." The agreement at larger scales validates the independence assumption: failures across nodes are largely uncorrelated, so job-level failure rate scales linearly with node count.

Projections. Using the validated model, the paper projects MTTF for scales not currently observed: 1.8 hours for 16,384 GPU jobs (2,048 nodes) and 0.23 hours (~14 minutes) for 131,072 GPU jobs (16,384 nodes). These projections assume the per-node failure rate r_f remains constant at larger scales and that failures remain independent across nodes.

Goodput Loss and Failure Cascades

Loss by job size (Figure 8). On RSC-1, the largest job category (2,049–4,096 GPUs) accounts for 0.47% of cluster compute wasted due to failures and second-order preemptions. Of this, approximately 84% (roughly 0.39 percentage points) comes from direct failure impact on the large jobs themselves, while 16% (roughly 0.08 percentage points) comes from preemptions of smaller jobs triggered when failed large jobs are rescheduled. Moderate-sized job categories (513–1,024 GPUs, 257–512 GPUs, 129–256 GPUs) each contribute 0.1–0.2% additional goodput loss. On RSC-2, the absolute goodput loss is an order of magnitude smaller, peaking at 0.04% for the 257–512 GPU category, with each smaller category contributing ~0.005–0.015%.

Cascade mechanism. The paper documents a concrete worst-case example: a 1,024-GPU job that NODE_FAILed and requeued 35 times, causing 548 preemptions across more than 7,000 GPUs. This example illustrates how a single unstable job can create orders-of-magnitude more disruption than its direct failure cost through the preemption cascade mechanism. The estimate of lost goodput per preempted job assumes hourly checkpointing (30 minutes average lost work) and takes the minimum of the job's runtime and 30 minutes, multiplied by the number of GPUs allocated.

ETTR Estimation and Validation

Cluster-level failure rate. The paper computes r_f (the failure rate per node-day used in all ETTR calculations) by counting all NODE_FAIL events plus attributed FAILED events for jobs >128 GPUs, then dividing by total node-days of runtime for those jobs. RSC-1 has r_f = 6.50 failures per thousand node-days; RSC-2 has r_f = 2.34. The difference is corroborated by GPU swap rates: "RSC-1 GPUs are swapped at ~3 times the rate compared to RSC-2," suggesting the failure rate difference is real and workload-related rather than a measurement artifact.

Validation against job runs (Figure 9). For job runs with ≥48 hours total training time and highest priority, assuming Daly-Young optimal checkpointing, u_0 = w_cp = 5 minutes, and cluster-average queue time q̄, the analytical model predicts:

  • On RSC-1: E[ETTR] decreases from approximately 0.92 at 8 GPUs to 0.88 at 512 GPUs, 0.85 at 4,096 GPUs, and approximately 0.78 by extrapolation at 16,384 GPUs. If queue time were zero (idealized high-priority scheduling), E[ETTR] would be approximately 0.94 at 8 GPUs, declining to 0.90 at 16,384 GPUs.
  • On RSC-2: E[ETTR] is consistently higher, decreasing from approximately 0.96 at 8 GPUs to 0.94 at 4,096 GPUs (with zero queue time), reflecting the lower failure rate.

The empirical ETTR measurements (averaged over job runs in each size category, with 90% confidence intervals around the mean) agree with predictions within the conservative bias. Observed ETTR on RSC-1 ranges from approximately 0.85–0.90 for the largest job runs (>1,024 GPUs), systematically higher than the analytical prediction (which uses average queue time) because these large, high-priority jobs experience shorter-than-average queue times. On RSC-2, empirical ETTR is approximately 0.88–0.92 for moderate to large jobs, with wider confidence intervals reflecting fewer large-job observations.

Projection to 12,288 GPUs (Figure 10). The contour plot shows the checkpoint write overhead w_cp required to achieve various ETTR targets (0.70, 0.90, 0.95, 0.99) as a function of cluster failure rate, for a hypothetical 12,288-GPU job with u_0 = 5 minutes. At RSC-1's current failure rate of 6.50 failures per thousand node-days, achieving ETTR of 0.90 requires w_cp of approximately 10 seconds—a 30× improvement over the assumed 5-minute baseline. Alternatively, the failure rate must improve to approximately 1 failure per thousand node-days (a 6.5× improvement) to achieve the same ETTR with 5-minute checkpoint writes. The paper observes that asynchronous checkpointing strategies (cited as Gemini, Wang et al., 2023) can achieve O(10s) checkpoint write overhead, making ETTR of 0.90 feasible at 12k-GPU scale without dramatic reliability improvements.

Lemon Node Detection Efficacy

Detection results. Over the evaluation period, the lemon detection pipeline identified 40 faulty nodes—24 on RSC-1 (1.2% of its 2,000-node footprint) and 16 on RSC-2 (1.7% of its 1,000-node footprint)—with "more than 85% accuracy" (Section IV-A). The paper does not specify the exact false positive rate or precision/recall breakdown, only that thresholds were manually tuned for accuracy and false positive rate.

Root cause distribution (Table II). Of the identified lemon nodes, 28.2% were attributed to GPU issues, 20.5% to DIMM (host memory) issues, 15.4% to PCIe issues, 7.7% each to EUD and unknown causes, 7.7% to NIC issues, 5.1% to BIOS, 2.6% to PSU, and 2.6% to Optics CPU. The dominance of GPU and memory issues is consistent with the failure rate breakdown in Figure 4.

Impact on job failure rates. The removal of identified lemon nodes led to a reduction in large job (512+ GPU) failures from 14% to 4%-a 10 percentage point absolute reduction. The paper characterizes this as a >30% improvement in large job completion rate (Observation 11). Given that lemon nodes represented only 1.2% of RSC-1's footprint, the disproportionate impact implies that the removed nodes were responsible for approximately 71% of large-job failures (from 14% to 4% failure rate, a 10/14 ≈ 71% relative reduction).

Signal analysis (Figure 11). The CDF of detection signals across all nodes over a 28-day window reveals that most features are highly sparse. XID counts (xid_cnt) are zero for the vast majority of nodes, with a long tail; similar sparsity holds for tickets, out-of-service counts, and failure counts. The excl_jobid_count signal (user-initiated node exclusions) was "found not to have a strong correlation with node failures" despite many nodes being excluded by at least one job, motivating the use of automated detection rather than relying on user reports.

Adaptive Routing Performance

Resilience to injected bit errors (Figure 12a). When bit errors are injected into the Infiniband fabric using mlxreg to modify port registers, an NCCL All-Reduce across 512 GPUs shows that AR maintains bandwidth of approximately 140–200 GB/s over five iterations, while the same benchmark without AR drops to approximately 100 GB/s. The paper states that during the bring-up phase of RSC-1, bandwidth degradation without AR was observed to be "as much as 50-75%," consistent with this controlled experiment.

Performance under multi-tenant contention (Figure 12b). When running 64 concurrent NCCL All-Reduce groups (each across 2 nodes / 16 GPUs), AR achieves higher median bandwidth (approximately 160–180 GB/s vs. 120–160 GB/s) and substantially lower variance (tighter vertical spread in the plot) compared to static routing. The reduced variance is because "AR can shield GPUs from being bottlenecked by congested links"—traffic is balanced across all available links rather than statically mapped to paths that may overlap.

Mechanism and deployment. AR operates at the switch level without requiring application changes, dynamically selecting output ports based on real-time port load. The paper reports that AR is deployed and enabled in both clusters to "increase performance predictability" and that it complements server-level health checks by handling network-level degradation transparently, preventing link issues from manifesting as job failures.

Ablation Studies and Robustness Checks

The paper's observational nature means there are no controlled ablations in the traditional ML sense. However, several robustness analyses validate the reliability of the paper's measurements and models:

  • Overlapping health checks as redundant failure detection: The co-occurrence analysis (57% of PCIe failures co-occur with XID 79 on RSC-1; 43% have all three event types) validates that the redundancy built into the health check system provides defense in depth. A failure missed by one check is likely caught by another, reducing the false negative rate of the overall detection system. The paper does not, however, quantify the marginal contribution of each check—how many failures would be missed if any single check were removed.

  • Observed MTTF vs. theoretical independence model at small scales: The deviation from the MTTF ∝ 1/N_nodes model at 8–16 GPU jobs on both clusters (Figure 7), attributed to "experimental usage patterns that cause correlated NODE_FAIL," is a robustness check in the negative sense: it reveals a regime where the independence assumption breaks down. This is not a failure of the MTTF model but a diagnostic finding—small jobs can experience correlated failures due to user behavior (e.g., repeatedly submitting the same configuration to the same problematic node), which the model cannot capture. The implication is that the independence assumption and the inverse-linear MTTF scaling are only reliable for jobs at or above 32 GPUs, where correlated user-induced failures become statistically negligible compared to independent hardware failures.

  • Sensitivity to checkpointing assumptions in ETTR validation: The ETTR validation (Figure 9) assumes Daly-Young optimal checkpoint intervals, u_0 = w_cp = 5 minutes, and that every non-COMPLETED termination is an infrastructure failure. The paper does not vary these assumptions to test sensitivity, but the conservative bias in the failure counting (treating all non-COMPLETED terminations as infra failures) means the empirical ETTR is a lower bound, and the true ETTR is likely higher—consistent with the finding that observed ETTR is sometimes higher than predicted for the largest jobs due to shorter-than-average queue times.

  • Lemon detection signal selection: The analysis of excl_jobid_count (user-initiated node exclusions) in Figure 11 shows that this signal "did not have a strong correlation with node failures" despite being intuitively appealing, and was therefore not used as a primary detection feature. This is a negative result: user-reported node quality is too noisy to be useful for automated lemon detection, motivating the reliance on automated telemetry (XID counts, failure rates, repair tickets) instead.

  • Cluster comparison as an implicit robustness check: The consistent differences between RSC-1 and RSC-2—RSC-1 having ~2.8× higher failure rate (6.50 vs. 2.34 per thousand node-days), ~3× higher GPU swap rate, larger maximum job sizes, and different failure category distributions (Figure 4)—demonstrate that the findings are not an artifact of a single cluster's configuration. The fact that MTTF scaling follows the same theoretical trend on both clusters, despite different absolute failure rates, provides a form of cross-validation.

  • Monte Carlo validation of ETTR approximations: For large, long-running hypothetical jobs (e.g., 8k GPUs), the analytical approximations used in the ETTR derivation (the Δt_cp/2 + u_0 ≪ MTTF assumption, the Jensen's inequality lower bound) are validated against Monte Carlo simulations with agreement within ~5%. This confirms that the closed-form expressions are accurate for the scale regime of interest, though the paper does not show this validation as a separate figure.

Critical Assessment

The experiments provide substantial empirical support for the paper's core claims, but several important caveats attend the interpretation of the results. We examine each major claim in turn.

Claim: Hardware-attributed failures impact 19% of GPU runtime but less than 1% of jobs. This claim (Figure 3) is well-supported by the scheduler log data but requires careful interpretation of what "hardware-attributed" means. The attribution pipeline uses a ±5–10 minute temporal window around job failure to associate health check events—this is a reasonable heuristic, but it cannot distinguish between: (a) a hardware failure that genuinely caused the job to fail, (b) a hardware failure that occurred coincidentally during a job that would have failed anyway due to a user code bug, and (c) a user code bug that manifested as a health check failure (e.g., a GPU memory access violation that triggers an XID error). The paper acknowledges the noisiness of failure attribution (Observation 3, Table I) but does not quantify the false positive rate of the attribution heuristics. If, for instance, 20% of attributed failures are actually user-code-induced, then the true hardware impact on GPU runtime would be closer to 15% rather than 19%. This is a relatively minor uncertainty that does not change the qualitative finding—hardware failures disproportionately affect large jobs—but it matters for precise goodput accounting.

Claim: MTTF decreases inversely with GPU count, with projections to 131,072 GPUs giving 0.23 hours. The empirical validation of inverse-linear MTTF scaling (Figure 7) is strong up to 4,096 GPUs on RSC-1 and up to 1,024 GPUs on RSC-2. The 90% confidence intervals are reasonably tight at these scales. However, the extrapolation to 16,384 and 131,072 GPUs assumes that per-node failure rates remain constant and that failures remain independent. The paper acknowledges neither the possibility of correlated failures at extreme scale (e.g., shared power delivery or cooling infrastructure failures that take down entire racks or pods simultaneously) nor the possibility that failure rates could change as hardware ages beyond the 11-month observation window. The authors reference Erben and Erdil (2024) on hardware failures not limiting AI scaling, but do not incorporate that work's analysis into their own projections. A cautious reading: the projections are the best available given the data, but they are linear extrapolations from a single order-of-magnitude range (32–4,096 GPUs) to two additional orders of magnitude (16,384–131,072 GPUs). This should be understood as a baseline estimate under idealized independence assumptions, not a validated prediction.

Claim: The ETTR analytical model accurately predicts job-level training efficiency. The validation in Figure 9 shows agreement between predicted E[ETTR] and empirical ETTR, but the agreement is qualified by systematic deviations. The largest job runs on RSC-1 have higher ETTR than predicted because their queue times are shorter than average—this is correctly diagnosed but means the model's accuracy depends on having good estimates of job-specific queue time distributions, not just cluster averages. The assumption that u_0 = w_cp = 5 minutes is not validated; the paper states that these are "reasonable values we have encountered anecdotally" (Section II-D), and varying them would shift both the predicted and the Daly-Young optimal checkpoint interval. A sensitivity analysis showing how E[ETTR] varies with u_0 and w_cp would strengthen confidence in the model's robustness. Additionally, the validation set (job runs with ≥48 hours total training time, highest priority) is a selected subset of all jobs—these are the largest, longest-running, and most carefully managed jobs, which may have better checkpointing discipline and faster restart procedures than typical jobs. The model's accuracy for smaller, lower-priority jobs is not tested.

A more subtle issue: the analytical derivation assumes failure timestamps are uncorrelated with checkpoint timestamps. The paper notes (Appendix A) that "if there are e.g. filesystem-related issues where correlations are expected between checkpoint writes and failures, E[W_j - t_cp] may approach Δt_cp rather than Δt_cp/2." This means that in the presence of checkpoint-induced failures (e.g., filesystem overload during coordinated checkpoint writes from many jobs), the unproductive time could be up to 2× higher than the model assumes—a factor not explored in the validation.

Claim: Removing 1.2% of nodes identified as lemons reduced large job failures from 14% to 4%. This is the paper's most dramatic impact claim and also the one with the thinnest methodological documentation. The paper does not specify: (1) the exact threshold values for lemon classification, (2) the false positive rate (only that accuracy is >85%), (3) the time period over which the before-and-after comparison was made, (4) whether other changes (software updates, health check additions, workload shifts) occurred between the "before" and "after" periods that could confound the comparison, or (5) the denominator for the 14% and 4% figures—are these the percentage of large jobs that experience any failure, or the percentage of large job runtime lost to failures? The 14% → 4% reduction is attributed to lemon node removal, but if the cluster failure rate was already declining due to other improvements (as Figure 5 shows the failure rate is highly dynamic), the true impact of lemon detection may be overstated. This is the experiment that most needs a controlled comparison—e.g., identifying a set of candidate lemon nodes, randomly remediating half and leaving the other half in service, and comparing failure rates—but such an experiment is operationally difficult and the paper does not attempt it.

Claim: Adaptive routing maintains bandwidth under link errors and reduces variance under contention. This claim is supported by the only controlled experiments in the paper (Figure 12). The experiments are well-designed—injecting bit errors to test resilience, running concurrent NCCL rings to test contention—but are limited in scope. Only All-Reduce is tested (the most common collective in data-parallel training, but not the only one—All-Gather, Reduce-Scatter, and All-to-All are also important, especially for model parallelism). The experiments use NCCL-Tests benchmarks, which may not capture the communication patterns of real training workloads (e.g., overlapping communication with computation, varying message sizes across layers). And the 512-GPU experiment tests a single scale; it is not shown whether AR's benefits scale to larger jobs or whether AR introduces overhead (e.g., out-of-order packet delivery causing NCCL-level reordering costs) at extreme scale. These are not fatal weaknesses—the experiments convincingly demonstrate AR's benefit in the tested scenarios—but they leave open the question of how much AR contributes to the observed 0.85–0.90 ETTR of the largest job runs.

Claim: Second-order preemptions account for 16% of failure-induced goodput loss. This calculation assumes all jobs checkpoint hourly and lose an average of 30 minutes of work per preemption. The paper does not validate this checkpoint frequency assumption against actual job logs (e.g., by measuring the interval between checkpoint writes in job output). If jobs checkpoint more frequently (e.g., every 15 minutes, which would be rational for very large jobs given their high failure rate), the average lost work would be 7.5 minutes, and the 16% figure would be lower. If some small jobs don't checkpoint at all (plausible for 1-GPU development jobs that users expect to run quickly), the lost work would be the full job runtime, and the 16% figure would be higher. The checkpoint frequency assumption is a free parameter that is not empirically grounded.

Missing experiments that would strengthen the paper:

  • Controlled lemon detection experiment. As noted above, a randomized remediation experiment would provide much stronger evidence for the causal impact of lemon node removal.
  • ETTR sensitivity analysis. Showing how E[ETTR] varies with u_0 from 1 to 20 minutes and w_cp from 10 seconds to 10 minutes would reveal which parameters most constrain training efficiency, guiding investment decisions more precisely than the contour plot in Figure 10.
  • Breakdown of unattributed failures. Unattributed NODE_FAIL is the single largest failure category in Figure 4, yet the paper devotes little analysis to understanding what these failures are. Root-causing even a subset of these (e.g., by correlating with kernel logs, power events, or thermal data) could reveal new failure modes and motivate new health checks—this is flagged as future work but not attempted.
  • Network failure attribution at the switch level. The paper's failure attribution is limited to node-level health checks; switch-level failures (which could cause multi-node NCCL timeouts without any node-level health check firing) are likely a significant fraction of unattributed failures but are not analyzed.
  • Comparison with specialized LLM clusters. The paper argues that research clusters are different from specialized LLM clusters, but does not provide a direct quantitative comparison. If data from a specialized cluster (e.g., a cluster running only LLaMa-scale training jobs) were available, comparing failure rates, MTTF scaling, and ETTR between the two cluster types would directly test the paper's claim about the importance of workload diversity.

Where the paper's claims hold conditionally:

  • The MTTF ∝ 1/N_nodes scaling holds for jobs ≥32 GPUs on RSC-1 but deviates for smaller jobs due to correlated user-induced failures (Figure 7). The projections to 16k+ GPUs are valid only if independence continues to hold.
  • The ETTR model is validated for highest-priority, long-running jobs (≥48 hours) but its accuracy for lower-priority jobs with non-negligible queue time is not tested (Figure 9). The full model (Equation 1) includes queue time terms, but these are not validated.
  • The lemon detection impact (14% → 4% large job failure reduction) is demonstrated on RSC-1 and RSC-2 with their specific hardware, workload mix, and failure profiles. The ~71% relative reduction is likely cluster-specific and depends on the fraction of truly defective nodes in the population—a cluster with fewer lemons would see smaller gains.
  • Adaptive routing benefits are demonstrated for All-Reduce under injected bit errors and controlled contention, but generalization to other collectives, real training communication patterns, and larger scales is assumed rather than proven.

Overall, the experimental evidence strongly supports the paper's qualitative narrative—failures matter at scale, they are statistically predictable, and infrastructure-level mitigations can substantially improve training productivity—while leaving the precise quantitative claims subject to the measurement and attribution uncertainties inherent in any large-scale observational study. The paper is transparent about these uncertainties (the attribution noisiness, the conservative ETTR calculation, the dynamic nature of failure rates), which strengthens rather than weakens its credibility.

6. Limitations and Trade-offs

Honest Cost Accounting: The Buried Price of Difficulty Estimation

The assumption and constraint. The paper omits the cost of generating 2048 samples from the base model to estimate question difficulty from its total compute budget calculations. In Section 3.2, this is flagged explicitly:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

This is not merely an accounting technicality. Generating 2048 complete solutions per question — each potentially hundreds of tokens — likely costs more than the largest test-time compute budgets under study (256–512 generations). The claimed 4× efficiency gains over best-of-N are therefore computed given difficulty already known, without amortizing the cost of acquiring that knowledge across the test set.

The consequence. In any realistic deployment, total cost equals difficulty estimation plus strategy execution. If estimation dominates, the 4× gain shrinks dramatically or even reverses into a net loss. For batch evaluation of thousands of questions, the amortized cost per question diminishes (fixed cost of random sampling across many questions), making the approach viable. But for single-question or low-volume inference — interactive use, on-demand API calls — paying 2048 samples to decide how to allocate a 64-sample budget is clearly irrational. The headline efficiency numbers are therefore conditional on amortization over many questions, which the paper neither quantifies nor discusses, potentially misleading readers about practical applicability.

What evidence exists in the paper. The paper's Figures 4 and 8 show compute-optimal scaling outperforming best-of-N by 4× after difficulty is known, with substitution curves that largely overlap for oracle and predicted bins. Critically, there is no figure or table showing total cost (estimation + execution) versus accuracy. The predicted difficulty method, while not requiring ground-truth labels, still demands scoring 2048 samples per question with the PRM — a cost that is never added to the x-axis of any scaling plot. This means every efficiency comparison in the paper is computed on a different cost basis than a practitioner would experience, systematically favoring the compute-optimal approach.

Mitigation status. The paper explicitly frames cheaper difficulty estimation as "a key avenue for future work" (Section 3.2) and suggests training models to predict difficulty directly from question text. No such model is developed or evaluated. Until this gap is closed — via direct difficulty prediction, adaptive estimation that allocates fewer than 2048 samples per question, or amortization over question batches — the reported gains represent an upper bound on achievable efficiency, not a realized deployment improvement.

Single Benchmark, Single Model Family: The Fragility of Quantitative Generalization

The assumption and constraint. The entire experimental apparatus — MTTF characterization, ETTR modeling, lemon detection evaluation — rests on data from two clusters running identical hardware (DGX A100 servers with 8× A100 80GB GPUs) and a single scheduler (Slurm), observed over 11 months (Section III). The authors state they "believe the model is representative of the capabilities of many contemporary LLMs" (Section 4), but this belief is untested. The paper provides no evidence — not even qualitative comparisons — on whether the failure patterns, scaling relationships, or mitigation efficacies generalize to other hardware (H100/GB200, TPU v5p), other schedulers (Kubernetes, MAST), other network fabrics (RoCE, custom interconnects), or other workload compositions.

The consequence. The paper's core quantitative contributions — MTTF of 7.9 hours at 1024 GPUs, r_f = 6.50 failures per thousand node-days, 4× compute efficiency gains — may not be portable across deployments. Several specific mechanisms are hardware-dependent:

  • The co-occurrence analysis (57% of PCIe errors co-occur with XID 79 on RSC-1) is A100-specific. H100 GPUs have different XID codes, different NVSwitch implementations, and potentially different dominant failure modes. A practitioner with H100 clusters cannot assume their r_f will match.
  • The lemon detection signal set (XID counts, row-remap failures, GSP RPC timeouts) assumes NVIDIA-specific error telemetry. On TPUs or custom accelerators, entirely different signals — and different lemon detection thresholds — would be needed.
  • The adaptive routing evaluation uses Infiniband-specific features. RoCE (RDMA over Converged Ethernet) deployments may lack equivalent in-network adaptation, altering the tradeoff between network-level and node-level reliability mechanisms.
  • Workload diversity, which the paper identifies as a defining characteristic of research clusters (Observation 7), is an independent variable that shapes failure profiles. RSC-1 (LLM-heavy) and RSC-2 (vision-heavy) already show different failure rates (6.50 vs. 2.34 per thousand node-days) and different failure category distributions. A cluster with exclusively LLM workloads or exclusively small experimentation jobs would have different reliability properties, and the paper provides no framework for projecting from one workload mix to another.

What evidence exists in the paper. The comparison between RSC-1 and RSC-2 (Figures 4, 5, 7, 9) provides a limited form of cross-validation and, encouragingly, shows that the qualitative patterns — MTTF scaling inversely with GPU count, r_f serving as a sufficient statistic for ETTR prediction, lemon detection removing a disproportionate fraction of failures — are consistent across both clusters despite different workloads and absolute failure rates. However, both clusters share identical hardware generation, scheduler, and operational practices (the same health check suite, the same Slurm configuration). This is cross-validation within a single organizational and hardware ecosystem, not generalization across the diverse deployment landscape that the paper's claims implicitly address.

Mitigation status. The paper does not attempt to address this limitation. It offers no methodology for adapting r_f estimates to new hardware or new workloads, no sensitivity analysis showing how MTTF projections change if GPU failure rates shift by 2× or 10×, and no discussion of which failure categories are likely to be hardware-generation-dependent versus fundamental to distributed training. Acknowledged implicitly by the title's "Revisiting Reliability in Large-Scale Machine Learning Research Clusters" (emphasis added) — the claims are scoped to the studied clusters — but the abstract and conclusions use universal language ("the impact of job failures across different scales," "essential reliability requirements for pushing the boundaries of ML training at scale") that invites over-generalization by readers.

The 14× Larger Model Baseline Is an Uphill Opponent for Pretraining

The assumption and constraint. The FLOPs-matched comparison (Section 7) evaluates test-time compute with PaLM 2-S* against a model with ~14× more parameters trained on the same amount of data, using greedy decoding with no test-time augmentation. The paper acknowledges that this departs from compute-optimal pretraining:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

This is a two-way weakening of the pretraining baseline. First, Chinchilla-optimal training (Hoffmann et al., 2022) would allocate additional compute to both more parameters and more data, producing a better model for the same total FLOPs budget than parameter-only scaling. Second, the larger baseline model uses greedy decoding — not best-of-N, not majority voting, not beam search — meaning the comparison is fundamentally asymmetric: test-time compute gets to deploy its full suite of optimizations (search strategies, revision chains, verifier-guided selection) while the pretrained model gets none.

The consequence. The paper reports, for example, +27.8% relative improvement from test-time compute over pretraining on easy questions at low R (Figure 1, top-right bar chart). This number may be inflated relative to what a well-designed deployment would actually compare against. A practitioner choosing between (a) training a 14× larger model with Chinchilla-optimal scaling and (b) keeping the smaller model and investing in test-time compute would face a different — and likely less favorable — tradeoff than the paper reports. The +27.8% could shrink substantially or even reverse against a properly scaled baseline.

Furthermore, even under the paper's intentionally simplified baseline, the results already show sharp boundaries: on hard problems (bins 4–5) at high R, pretraining dominates (test-time compute shows −52.9% relative disadvantage under PRM search, per the bottom-right bar chart in Figure 1). If the stronger Chinchilla baseline were used, test-time compute would likely lose in strictly more regimes, potentially making the "test-time compute can substitute for pretraining" claim less practically compelling than it initially appears.

What evidence exists in the paper. Figure 9 provides the most detailed breakdown, showing per-difficulty-bin ETTR for test-time compute versus the star markers representing the greedy 14× larger model at three R values. The key observation: even with the weakened baseline, test-time compute already loses on hard problems at all R values and only wins on easy-to-medium problems when R is low. This pattern strongly suggests that a properly compute-optimal pretraining baseline would narrow or close the advantage on medium problems and potentially flip the easy-problem comparison at moderate R values. The paper does not perform the sensitivity analysis that would verify or refute this.

Mitigation status. The paper explicitly acknowledges the parameter-only scaling choice and defers compute-optimal pretraining comparison to future work. What is missing is any attempt to bound the impact of this choice — e.g., estimating what fraction of the 14× compute budget would go to data versus parameters in a Chinchilla-optimal configuration, and how that would impact the baseline's performance based on known scaling laws. Without such bounding, the reader is left to guess how much the headline substitution results depend on this design choice versus reflecting a genuine advantage of test-time compute.

Verifier Over-Optimization Exists but Is Diagnosed, Not Fixed

The assumption and constraint. The paper identifies verifier over-optimization — search algorithms exploiting the PRM's imperfections rather than genuinely improving solutions — as a central bottleneck preventing unbounded test-time compute scaling (Section 5.3). The evidence is concrete: beam search degrades performance on easy problems at high budgets (Figure 3, right), lookahead search — the most powerful optimizer — paradoxically performs worst overall (Figure 3, left), and qualitative examples (Appendix M) show search producing degenerate outputs that score highly under the PRM but are incorrect.

The consequence. The paper's compute-optimal policy (Section 3.4) mitigates over-optimization by routing easy problems away from aggressive search — using best-of-N where verifier reliability is high and beam search only where there is genuine signal — but it does not solve the underlying problem. On medium-difficulty problems where beam search is deployed, the scaling curves still flatten and begin to decline at the highest budgets (Figure 3, right, bin 3–4). This establishes a fundamental ceiling on test-time compute gains: no matter how clever the allocation policy, the verifier's reliability limits how much compute can be productively spent. The paper's 4× efficiency gains are relative to a uniform best-of-N baseline at moderate budgets (4–64 generations); they do not imply that compute-optimal scaling can continue improving indefinitely with larger budgets — the over-optimization threshold bounds the maximum useful compute per problem.

What evidence exists in the paper. The evidence for over-optimization is strong and multisource: the downward slope of beam search on bin 1 (Figure 3, right, comparing the 4-generation bar to the 256-generation bar for beam search), the systematic underperformance of lookahead search (Figure 3, left), and the qualitative failure modes (Appendix M). However, the paper provides no systematic characterization of the over-optimization threshold — at what budget does beam search peak for each difficulty bin? How does the peak shift if the PRM is improved (e.g., trained with more data, with adversarial examples, with ensemble averaging)? What is the relationship between PRM accuracy (as measured by, say, AUC on held-out solutions) and the onset of over-optimization? Without answering these questions, the practitioner cannot determine whether investing in a better PRM is more cost-effective than investing in a better allocation policy, or whether the verifier quality achieved by the Monte Carlo rollout training procedure is near-optimal or far from it.

Mitigation status. The compute-optimal policy is itself a partial mitigation — it avoids the worst over-optimization regimes by switching strategies per difficulty bin. But the paper frames verifier improvement as the key future bottleneck (Section 8: "improving verifier robustness is the key bottleneck for further scaling test-time compute") without developing or evaluating any specific verifier improvement, making this a diagnosis without a treatment plan. For a practitioner, the actionable takeaway is uncomfortable: you should invest in test-time compute, but your ceiling is determined by a verifier whose optimal quality is unknown and whose improvement path is unspecified.

Lemon Detection Is Critical but Evaluation Is Methodologically Thin

The assumption and constraint. The lemon detection pipeline (Section IV-A) is presented as a deployed system that reduced large job failures from 14% to 4% by removing 1.2% of RSC-1's nodes, with "more than 85% accuracy." The paper's evaluation of this system is absent of detail necessary to assess its reliability and reproducibility, which matters because lemon detection is not a research prototype — it is deployed on production infrastructure where false positives (removing healthy nodes) directly reduce cluster capacity and goodput.

The consequence. Several critical methodological gaps exist:

  • Threshold selection is manual and unreported. The paper states thresholds were "tuned manually based on accuracy and false positive rate of predicted lemon nodes." Without knowing the thresholds (what XID count? what failure rate? what exclusion count?), another operator cannot replicate the approach without undertaking the same manual tuning effort. Worse, if thresholds are sensitive to cluster-specific failure baselines, an operator applying the method to a new cluster may experience dramatically different accuracy.
  • The before-and-after comparison lacks controls. The reduction from 14% to 4% large-job failure rate is attributed to lemon node removal, but the failure rate evolution (Figure 5) shows that the cluster failure rate is highly dynamic — varying by an order of magnitude over 11 months, driven by driver bugs, new health check additions, and workload shifts. Without a controlled comparison (e.g., a randomized remediation experiment, or at minimum a time-series analysis showing the failure rate change coincident with lemon removal and not confounded by other trends), the causal interpretation is uncertain.
  • The 85% accuracy claim lacks a denominator. What is the false positive rate (healthy nodes incorrectly flagged as lemons)? What is the false negative rate (lemons missed)? 85% accuracy on a dataset where 98.8% of nodes are healthy and 1.2% are lemons is trivially achievable by classifying all nodes as healthy (which would achieve 98.8% accuracy while missing every lemon). The paper does not report precision, recall, or the baseline rate, making the 85% figure uninterpretable.
  • The 14% → 4% reduction is presented without statistical testing or time attribution. Over what period was this measured? Were the "before" and "after" periods contiguous? Were there other concurrent reliability improvements (e.g., Figure 5 shows new health checks added during the analysis period, which would independently reduce failure rates by catching previously undetected failure modes)?

What evidence exists in the paper. Figure 11 shows the distribution of detection signals across nodes, establishing that signals like XID counts and failure counts are sparse — the vast majority of nodes have zero values, with a long tail of elevated-signal nodes that presumably correspond to lemons. This provides a qualitative justification that statistical detection is feasible. Table II provides a root-cause breakdown of identified lemons, confirming that they exhibited real hardware issues (GPU, memory, PCIe problems) rather than being false positives. But the quantitative evidence for the system's performance — its precision, recall, impact magnitude, and robustness to cluster dynamics — is essentially absent beyond the headline 14% → 4% claim and the >85% accuracy assertion.

Mitigation status. The paper does not acknowledge these methodological limitations. The lemon detection evaluation is presented as a deployment success, not as a preliminary result requiring further validation. This is the section of the paper where a practitioner seeking to implement similar systems would want the most detail and instead finds the least — a significant gap for a paper that positions itself as sharing "operational experience" and "lessons learned."

Sequential Revision Latency: The Serial Dependency That ETTR Cannot Express

The assumption and constraint. The paper frames the sequential vs. parallel sampling tradeoff (Section 6.4) purely in terms of generation budget efficiency — how many correct answers are produced per unit compute. A 64-generation sequential chain (one solution refined 64 times) and a 64-generation parallel batch (64 independent solutions generated simultaneously) are treated as equivalent costs. This ignores a fundamental difference: latency. Sequential revisions are inherently serial — each revision depends on the output of the previous one — meaning a chain of length L takes L times longer in wall-clock time than generating L samples in parallel, assuming sufficient hardware.

The consequence. For latency-constrained settings — interactive assistants, real-time applications, any deployment where users are waiting for a response — the sequential-heavy strategies favored by the compute-optimal policy for easy problems are impractical regardless of their generation-budget efficiency. A 64-step revision chain that produces a correct answer with 90% probability but takes 64 sequential forward passes might have an unacceptable response time, whereas a 64-sample parallel best-of-N that completes in a single forward-pass duration (plus scoring) would be preferred despite lower accuracy-per-budget. The paper's compute-optimal policy optimizes a single dimension (accuracy per generation budget) but the real deployment optimization is multi-objective — accuracy, latency, throughput — and the sequential-vs-parallel tradeoff is fundamentally a latency-throughput tradeoff as much as a compute-efficiency one.

The paper's own architecture acknowledges the cost of serial operations implicitly through its analysis of restart overhead (u_0) and checkpoint write cost (w_cp) in the ETTR model. NCCL initialization is cited as an operation that "can scale poorly with the number of GPU nodes" (Section V), and it is exactly the kind of serial or coordination-bound overhead that adds to the unproductive runtime term. But sequential revision latency is never modeled, discussed, or even mentioned as a constraint — it is simply absent from the optimization framework.

What evidence exists in the paper. None directly. The revision model analysis (Section 6) reports accuracy improvements from sequential chains but never reports wall-clock time per revision step, end-to-end latency for a full chain, or any comparison between the latency of a 64-step sequential versus 8×8 sequential-parallel hybrid versus 64-sample parallel. The ETTR model (Equation 1) captures restart overhead (u_0), checkpoint write time (w_cp), and queue time (q̄) — all sources of unproductive runtime — but not the productive-time latency of the inference strategy itself, which implicitly assumes that all strategies consume the same wall-clock time per unit of productive computation. This is false for sequential vs. parallel deployments.

Mitigation status. The paper does not acknowledge this limitation. No latency-aware variant of the compute-optimal policy is proposed, no latency constraints are included in the allocation optimization, and no discussion of when latency considerations would override generation-budget efficiency appears in the discussion or future work sections. For a paper that explicitly targets "on-device deployment" and "real-time" applications in its motivation (Section 1), this omission is particularly notable — on-device inference is precisely the setting where sequential latency is most constraining.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the conversation around ML cluster reliability from binary availability thinking ("is the cluster up?") to statistical process management ("what is the failure rate, how does it scale, and what is the expected productive fraction of my training run?"). The shift is methodological rather than algorithmic—the paper does not introduce a new failure detection algorithm or a novel fault tolerance protocol. Instead, it provides the measurement infrastructure, the analytical tools, and the operational validation to make reliability a quantitative design parameter rather than a qualitative aspiration.

The specific conceptual move is the unification of three previously separate reliability concerns into a single framework:

First, MTTF as a linear function of scale (Figure 7). By validating that MTTF ∝ 1/N_nodes for jobs of 32–4,096 GPUs—with r_f = 6.50 and 2.34 failures per thousand node-days on RSC-1 and RSC-2 respectively—the paper establishes that gang-scheduled training reliability is predictable from a single cluster-specific constant. This transforms failure from an unknown risk ("failures will probably happen") into a design parameter ("expect a failure every 7.9 hours at 1024 GPUs, project 1.8 hours at 16,384 GPUs"). The implication for cluster planning is immediate: if you are designing a training run, you can compute the expected number of failures before the run begins, size your checkpoint infrastructure accordingly, and budget the restart overhead.

Second, ETTR as a deployable system-level metric (Equation 1, Figure 9). By collapsing queue time, checkpoint overhead, restart initialization, and catch-up training into a single number between 0 and 1, the paper gives operators a metric that captures the full training experience—submission to completion—rather than partial views like GPU utilization or job completion rate. The validation against actual job run data (Figure 9) demonstrates that the analytical model works with real, messy cluster telemetry, not idealized assumptions. This is significant because prior work used metrics like goodput (wasted compute per unit time, ignoring queue delays) or job slowdown (wallclock time over scheduled time, ignoring unproductive scheduled time). ETTR exposes the tradeoffs that those metrics hide: reducing checkpoint interval improves failure recovery (smaller Δt_cp/2 term) but increases write overhead (larger w_cp/Δt_cp term), and the optimal balance depends on r_f, u_0, and w_cp in ways that are not obvious without the model.

Third, lemon nodes as a distinct failure category requiring temporal analysis (Section IV-A, Table II). The finding that 1.2% of nodes cause a disproportionate fraction of large-job failures—and that removing them reduced large-job failure rates from 14% to 4%—reframes reliability investment strategy. Rather than uniformly improving all nodes (e.g., reducing the baseline failure rate through better hardware, cooling, or software), the paper shows that identifying and removing the worst outliers can yield disproportionate returns. This is a pareto-optimal insight: a 10× improvement in median node reliability would be far more expensive and less impactful than finding and replacing the 99th-percentile lemons, which account for the majority of observed failures.

The paper also reconciles a tension in the ML infrastructure literature. Prior work on LLM training reliability—Megascale (Jiang et al., 2024), LLaMa 3 (Meta, 2024), Gemini (Google, 2024)—focused on specialized clusters where essentially all resources are devoted to a single enormous training job. These papers report failure statistics but treat them as incidents overcome: "we experienced X failures and handled them by doing Y." This paper argues that in research clusters—where job sizes span four orders of magnitude (1 to 4,096+ GPUs), where workload composition changes constantly (Observation 6), and where small jobs make up >90% of job count but <10% of GPU time (Observation 7)—reliability must be workload-agnostic and statistically managed rather than tailored to any single training run. The failure cascade analysis (16% of goodput loss from second-order preemptions, Figure 8) provides the quantitative evidence: even if the goal is solely maximizing large-job throughput, the reliability of small jobs matters indirectly through scheduler dynamics. This finding pushes back against the natural instinct to optimize clusters for the largest jobs and ignore everything else.

The research directions this work makes more attractive include: cheap, online failure prediction (using the paper's MTTF framework as a baseline to beat), verifier-aware checkpoint scheduling (using ETTR optimization to decide checkpoint timing per job based on its current failure risk), and cluster-level reliability-aware scheduling (allocating jobs to nodes with different failure risk profiles based on job priority and checkpoint resilience). The directions it makes less attractive include: one-size-fits-all reliability configurations (the dynamic failure rate evolution in Figure 5 shows this is futile), application-level fault tolerance as the primary reliability mechanism (the paper argues infrastructure-level mitigations like health checks and lemon detection are more practical for diverse workloads), and single-metric optimization (e.g., maximizing utilization without accounting for failure-induced unproductive time, which ETTR captures and utilization misses).

Follow-Up Research This Work Enables

Online, per-job MTTF estimation from short-horizon telemetry. The paper's MTTF projections use a static cluster-wide r_f computed from months of historical data. But Figure 5 shows that the failure rate varies by an order of magnitude over time, with different failure modes dominating in different seasons. A natural extension is to estimate r_f online—per job or per scheduling window—using recent health check telemetry, node exclusion patterns, and known lemon node status rather than a static average. The paper's lemon detection signals (XID counts, failure rates, repair tickets over 28-day windows, Figure 11) provide candidate features. A strong follow-up would: (1) train a lightweight model to predict r_f over the next hour given the last N hours of cluster telemetry, (2) compare ETTR predictions using this dynamic r_f against the static baseline, and (3) measure whether dynamic r_f enables better per-job checkpoint interval selection (since the optimal Δt_cp depends on r_f via the Daly-Young formula, Equation 3). This is newly tractable because the paper provides both the analytical framework (Equation 1) and the validation methodology (Figure 9) for evaluating such predictions.

Attributing unattributed failures through cross-signal correlation at the cluster level. Unattributed NODE_FAIL is the single largest failure category on both clusters—1.6 × 10⁻⁵ per GPU-hour on RSC-1, 4.2 × 10⁻⁵ on RSC-2 (Figure 4). These are failures where no health check fired, likely because the node became unresponsive before any check could run. The paper observes that some NODE_FAIL events are "not associated with any health checks" but does not attempt to root-cause them. A strong follow-up would correlate unattributed NODE_FAIL events with cluster-level signals that are not per-node: switch-level telemetry (did a top-of-rack switch fail, causing multiple nodes to become unreachable simultaneously?), power distribution unit logs (did a power sag affect multiple racks?), thermal data (are unattributed failures spatially clustered in hot zones?), and job-level patterns (do unattributed failures spike during coordinated checkpoint writes, suggesting filesystem overload?). The paper's failure taxonomy (Table I) and the observation that errors have a large "blast radius" across the stack (Observation 3) motivate this investigation, and the paper's dataset of 4 million jobs provides the statistical power to detect rare spatial or temporal correlations that individual operators would miss.

Controlled lemon detection evaluation with randomized remediation. The paper reports that removing 1.2% of nodes reduced large-job failures from 14% to 4% (Observation 11), but this is an uncontrolled before-and-after comparison confounded by the dynamic failure rate evolution (Figure 5), concurrent health check additions (annotated vertical lines in Figure 5), and workload shifts. A rigorous follow-up would: (1) identify candidate lemon nodes using the paper's signal set (XID counts, failure rates, repair tickets), (2) randomly assign candidates to immediate remediation or a control group (continued scheduling for a fixed observation period), (3) measure the failure rate of jobs scheduled on lemon vs. control nodes, and (4) estimate the causal effect of lemon removal on cluster-wide goodput. This is operationally difficult—it requires temporarily leaving known-defective hardware in production—but it would transform the lemon detection claim from "plausible and consistent with observations" to "causally validated." The paper's 28-day rolling window methodology (Section IV-A) and the sparsity of lemon nodes (~1.2% of the fleet) make such an experiment feasible: the control group is small, and the observation period needed for statistical significance is known (the paper's 28-day window was chosen to balance responsiveness and noise).

ETTR-aware scheduling: allocating jobs to nodes based on failure risk profiles. The paper demonstrates that different nodes have different failure rates (lemon detection, Section IV-A) and that different jobs have different sensitivities to failure (large jobs lose more work per failure, high-priority jobs have queue-time advantages). A natural extension is to make the scheduler reliability-aware: assign high-priority, large, or checkpoint-poor jobs to nodes with low historical failure rates (non-lemons), and assign low-priority, small, or easily-checkpointed jobs to nodes with higher failure risk. This is not "fairness"—it is goodput optimization, because the ETTR cost of a failure scales with job size and checkpoint interval. The paper's analytical ETTR model (Equation 1) provides the objective function: for each job, compute the expected ETTR given the node's estimated r_f, the job's u_0 and w_cp, and the current queue depth (affecting q̄), and assign the job to maximize aggregate expected ETTR. A strong follow-up would: (1) implement ETTR-aware scheduling as a Slurm plugin that modifies job priority based on node reliability scores (extending the paper's existing lemon detection pipeline), (2) run an A/B test comparing ETTR-aware scheduling against the current priority-based scheduling over a multi-month period, and (3) measure whether aggregate goodput improves beyond the gains from lemon node removal alone.

Checkpoint strategy personalization using per-job ETTR optimization. The paper's ETTR model assumes a uniform checkpoint interval (Daly-Young optimal, Equation 3) for all jobs, but the optimal interval depends on w_cp (checkpoint write cost), u_0 (restart overhead), and r_f (failure rate), all of which vary across jobs and over time. A job training a model with 10 billion parameters has a much larger checkpoint size (and thus larger w_cp) than a job training a 100-million-parameter model. A job using asynchronous checkpointing (cited via Gemini, Wang et al., 2023) has effective w_cp of O(10s) rather than O(minutes). The paper's Figure 10 shows how sensitive ETTR is to w_cp at 12k-GPU scale. A strong follow-up would: (1) instrument the cluster to measure per-job w_cp and u_0 empirically (rather than assuming 5 minutes), (2) compute per-job optimal Δt_cp using the measured parameters and the current estimated r_f, (3) provide feedback to users (or automate via the scheduler) to adjust checkpoint frequency per job, and (4) measure the aggregate ETTR improvement over uniform checkpointing. This is newly tractable because the paper provides both the analytical optimization (Equation 3) and a validation methodology (comparing predicted and observed ETTR as in Figure 9) that can be applied per-job.

Cross-cluster and cross-hardware-generation failure rate projection. The paper's MTTF projections to 16k and 131k GPUs (Figure 7) assume r_f remains constant and failures remain independent across nodes. These assumptions may break at extreme scale: power infrastructure failures or cooling failures could cause correlated multi-node failures (violating independence), and next-generation hardware (H100, GB200) may have different per-node failure rates (due to different thermal profiles, different HBM characteristics, or different NVSwitch implementations). A strong follow-up would: (1) collect failure data from clusters with different GPU generations (A100 vs. H100) and different scales (up to ~30k GPUs, as announced by Meta and others), (2) test whether r_f is hardware-generation-dependent and whether correlated failure modes appear at larger scales, (3) develop a hierarchical failure model that includes rack-level, pod-level, and cluster-level failure rates in addition to node-level, and (4) recalibrate the MTTF and ETTR projections for exascale training runs. The paper's methodology—computing r_f from scheduler and health check logs, validating MTTF ∝ 1/N_nodes, fitting confidence intervals via Gamma distributions—is directly portable to new clusters, and the paper provides a baseline (r_f = 6.50 for general ML, r_f = 2.34 for vision on A100s) against which new measurements can be compared.

Practical Applications and Downstream Use Cases

Capacity planning for large-scale training runs. When a team plans a training run requiring, say, 8,192 GPUs for 30 days of productive training, the paper's MTTF projection (Equation: MTTF = (N_nodes × r_f)^(-1)) and ETTR model (Equation 1) allow them to estimate the actual wallclock time and total GPU-hours required before submitting the job. For RSC-1 with r_f = 6.50 failures per thousand node-days: at 8,192 GPUs (1,024 nodes), the MTTF is approximately 3.5 hours. Over 30 days of productive training (720 hours), the expected number of failures is 720 / 3.5 ≈ 206. With u_0 = 5 minutes restart overhead and w_cp = 5 minutes checkpoint cost, and assuming Daly-Young optimal checkpoint interval Δt_cp = sqrt(2 × 5 / (1024 × 6.50 / 1000 / 24)) ≈ 1.9 hours, the expected ETTR from Equation 2 is approximately 0.88. This means the 30-day productive run will actually take 30 / 0.88 ≈ 34 days of wallclock time, plus queue time. The team can budget accurately, and if the projected ETTR is too low, they can quantitatively evaluate options: invest in faster checkpointing (reduce w_cp to 10 seconds via asynchronous writes, which Figure 10 shows would improve ETTR to ~0.95), reduce restart overhead (optimize NCCL initialization to cut u_0 to 1 minute), or request higher scheduling priority (reduce q̄). Before this paper, such calculations relied on anecdotal experience or rules of thumb. The paper provides a validated model and cluster-specific parameters (r_f = 6.50 for RSC-1, r_f = 2.34 for RSC-2) that make the estimation rigorous.

Lemon detection as a standard cluster operations practice. The paper shows that a simple pipeline—tracking XID error counts, repair tickets, node exclusion counts, and failure rates over a 28-day window—can identify a small fraction of nodes (~1.2% of the fleet) that cause a disproportionate fraction of large-job failures (~71% relative reduction after removal). For any organization operating ML training clusters at scale, implementing this pipeline is a low-cost, high-impact reliability improvement. The signals are already collected by standard datacenter monitoring (XID errors from NVIDIA GPUs, repair tickets from asset management systems, job failure attribution from scheduler logs). The 28-day window is long enough to achieve statistical significance (given the sparsity of failures) and short enough to be responsive to newly degrading nodes. The manual threshold-setting methodology, while not automated, is feasible for a cluster operations team that can inspect the CDF of each signal (as in Figure 11) and set thresholds to target a desired false positive rate. The paper's root cause distribution (Table II: 28% GPU, 21% memory, 15% PCIe) helps operators prioritize repair parts and vendor engagement. And the impact—reducing large job failures from 14% to 4%—is quantified in terms that directly translate to training productivity and GPU cost savings.

Workload-agnostic reliability monitoring for multi-tenant research clusters. The paper's key architectural argument is that research clusters cannot optimize reliability for any single workload because job sizes span four orders of magnitude (Figure 6: 40% of jobs use 1 GPU, while 66% of GPU hours go to jobs ≥256 GPUs) and workload composition changes constantly (Observation 6). The health check infrastructure (Section II-C), the ETTR metric (Section II-D), and the failure attribution pipeline (Figure 4) together form a workload-agnostic reliability stack that any multi-tenant ML cluster can adopt. The specific design choices are grounded in the paper's operational experience: 5-minute health check periodicity (fast enough to catch failures before multiple jobs are affected, slow enough to avoid excessive overhead), overlapping checks with intentional redundancy (57% of PCIe failures co-occur with XID 79, providing defense in depth), severity-based response (immediate node removal for high-severity failures, deferred remediation for low-severity), and automatic job requeueing on infrastructure-attributed failures (removing the burden from users). A cluster operator adopting this stack would track: (1) the cluster-wide r_f (failures per thousand node-days for jobs >128 GPUs), (2) the per-job-size MTTF (validating that MTTF ∝ 1/N_nodes holds, alerting if correlated failures emerge), (3) the 30-day rolling failure rate by category (to detect new failure modes, as in Figure 5), and (4) the ETTR of the largest and highest-priority job runs (to ensure they maintain ≥0.85–0.90 efficiency). The paper provides baseline values against which new clusters can be compared, though the absolute r_f will differ by hardware generation and workload.

Network fabric resilience as a complement to node-level health checks. The adaptive routing evaluation (Figure 12) demonstrates that in-network adaptation can mask Infiniband link degradation from applications entirely, preventing link-level failures from becoming job-level events. For clusters using Infiniband fabrics, enabling AR is a configuration flag with no application changes required and, based on the paper's experiments, provides substantial benefits: maintaining 140–200 GB/s All-Reduce bandwidth under injected bit errors (versus ~100 GB/s without AR) and reducing performance variance under multi-tenant contention (Figure 12b). The paper's observation that "as much as 50-75% bandwidth loss" was observed during bring-up without AR underscores that this is not a marginal optimization—it is essential for making the network fabric resilient to the inevitable link degradation that occurs at scale. For clusters using RoCE (RDMA over Converged Ethernet) rather than Infiniband, equivalent mechanisms (e.g., ECMP with dynamic load balancing, packet spraying) may provide similar benefits, though the paper does not evaluate them. The broader lesson—that reliability mechanisms should operate at multiple layers (node-level health checks, network-level adaptive routing, scheduler-level lemon detection) because failures manifest at multiple layers—is directly portable to any large-scale training infrastructure.