Ring Attention with Blockwise Transformers for Near-Infinite Context

ArXiv: 2310.01889

🎯 Pitch

Ring Attention trains Transformers on context lengths exceeding 100 million tokens—a leap that shatters the quadratic memory wall of self-attention by redistributing the sequence across devices with zero overhead, no approximations, and perfect compute-communication overlap. This turns the device count into a direct multiplier for context length, making near-infinite context a practical reality on today’s accelerators.

1. Executive Summary

This paper proposes Ring Attention with Blockwise Transformers (Ring Attention), a memory-efficient approach that distributes long sequences across multiple devices by leveraging blockwise computation of self-attention and feedforward networks while fully overlapping the communication of key-value blocks between hosts arranged in a ring topology. Evaluated on language modeling benchmarks using the LLaMA architecture (3B–30B parameters) across A100 GPUs and TPUv3/v4/v5e accelerators, Ring Attention enables context lengths that scale linearly with device count—achieving training sequences of over 100 million tokens without attention approximations and with zero communication or computation overhead, representing up to a 512× improvement over prior state-of-the-art memory-efficient transformers (e.g., expanding context from 32 tokens to 16,384 tokens on TPUv4-1024 for a 3B model). The approach maintains model flops utilization (MFU) while training these extended contexts and improves performance on reinforcement learning tasks (ExoRL benchmark, from 111.13 to 113.66 total average return) and long-context retrieval, establishing that near-infinite context training is achievable with existing hardware only when the per-host block size exceeds the ratio of device FLOPS to interconnect bandwidth—a condition easily met for GPUs with NVLink (~1K tokens) but more stringent for InfiniBand-connected GPUs (~25K tokens).

2. Context and Motivation

The Core Problem: Transformers Cannot Scale to Long Sequences Due to Memory Constraints

The Transformer architecture has become the dominant model family across virtually every domain of AI—language, vision, reinforcement learning, and multimodal applications. Its core mechanism, self-attention, endows it with the ability to capture long-range dependencies between any pair of input tokens, enabling the kinds of deep contextual understanding that power modern language models, video understanding systems, and decision-making agents. However, this expressiveness comes at a steep cost: the memory required to compute self-attention scales quadratically with the input sequence length. For a sequence of length $s$ and hidden dimension $d$, the attention matrix—before any optimizations—requires $O(s^2 \cdot d)$ memory per layer. This is not merely an inconvenience; it is the single hardest bottleneck preventing Transformers from processing the long sequences that many important applications demand.

The paper opens Section 1 by surveying the landscape of context lengths that have emerged in production systems, and the picture reveals both the ambition and the limitation. GPT-3.5 handles 16K tokens. GPT-4 reaches 32K. MosaicML's MPT extends to 65K. Anthropic's Claude pushes to 100K. These numbers are impressive when compared to the 512–2048 token contexts that were standard just a few years prior, but they remain orders of magnitude short of what real-world applications call for. The paper identifies a diverse array of use cases that demand vastly longer contexts: processing entire books in a single forward pass, analyzing high-resolution images at pixel-level granularity, understanding long videos spanning hours of footage, navigating complex codebases with millions of lines, extracting information from the interconnected web and hyperlinked content, and handling complex scientific experiment data such as gene sequences. Each of these requires reasoning over dependencies that span hundreds of thousands to millions of tokens—far beyond what even the most advanced deployed systems can currently process.

Why the Memory Bottleneck Persists: Understanding the Activation Storage Problem

To appreciate why this problem is so stubborn, it is essential to understand exactly what consumes memory in a Transformer layer and why prior optimizations, while helpful, leave the fundamental bottleneck unaddressed. The paper provides a precise accounting of the memory requirements for different components of the Transformer, and walking through this reveals the structural challenge that Ring Attention is designed to overcome.

A standard Transformer layer consists of two primary sub-layers applied in sequence: multi-head self-attention followed by a position-wise feedforward network (FFN). In the self-attention sub-layer, the input sequence $X \in \mathbb{R}^{s \times d}$ is linearly projected into queries $Q$, keys $K$, and values $V$, each also of shape $\mathbb{R}^{s \times d}$. The attention output is computed as:

$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d}}\right) V$

The intermediate attention score matrix $QK^T$ has shape $s \times s$, and storing it naively requires memory proportional to $s^2$. The feedforward network that follows applies two linear transformations with a ReLU activation:

$\text{FFN}(x) = \max(0, xW_1 + b_1)W_2 + b_2$

where $W_1 \in \mathbb{R}^{d \times d_{\text{ff}}}$ and $W_2 \in \mathbb{R}^{d_{\text{ff}} \times d}$, with $d_{\text{ff}}$ typically being $4d$. The activations within this FFN—specifically the intermediate representations before and after the ReLU—consume $8bsh$ bytes per layer (in bfloat16, where each element is 2 bytes, the activation of shape $b \times s \times d_{\text{ff}}$ requires $2 \cdot b \cdot s \cdot 4d = 8bsh$).

To put concrete numbers to this: the paper calculates (Section 2) that even with a batch size of 1, processing 100 million tokens with a hidden size of 1024 would require over 1000 GB of memory. Contemporary GPUs and TPUs typically offer less than 100 GB of high-bandwidth memory (HBM). The gap is roughly an order of magnitude, and it cannot be closed by incremental hardware improvements—the physical limitations and manufacturing costs of HBM expansion mean that algorithmic solutions are necessary.

Prior Approaches: What They Solve and What They Leave Unsolved

The paper situates its contribution within a lineage of memory-efficiency techniques, each of which chips away at the problem but fails to address the core structural limitation.

Memory-Efficient Attention (Rabe and Staats, 2021; Dao et al., 2022). The key insight driving this line of work is that the softmax matrix in self-attention can be computed incrementally without ever materializing the full $s \times s$ matrix. This emerged from the tiling technique (Milakov and Gimelshein, 2018), which shows that softmax can be computed in blocks by tracking running statistics (max and sum) and rescaling as new blocks are processed. The resulting blockwise attention algorithms reduce the memory cost of the self-attention sub-layer from $O(bns^2)$ (where $n$ is the number of heads) to approximately $2bsh$ bytes per layer—an enormous reduction. The FlashAttention CUDA implementation (Dao et al., 2022) further optimizes this by carefully managing I/O between HBM and SRAM, achieving both memory savings and speed improvements.

Blockwise Parallel Transformers (Liu and Abbeel, 2023). This work extends the blockwise idea to the feedforward network, observing that the FFN operations are position-wise (each token is processed independently) and can therefore also be computed block-by-block. Without this optimization, the FFN activations consume $8bsh$ bytes. By computing the FFN in blocks, BPT reduces this to $2bsh$. Combined with memory-efficient attention, the total activation memory per layer drops to $2bsh$—a dramatic improvement over vanilla Transformers.

Why these are insufficient: the layer output storage problem. Despite these advances, BPT and its predecessors share a fundamental limitation that the paper identifies as the remaining bottleneck. After computing self-attention—which requires interactions among all sequence elements ($n$-to-$n$ interactions)—the output of each layer must be stored in its entirety for use by the next layer. The subsequent layer's self-attention needs access to all positions of the previous layer's output to compute its own attention scores. If these outputs are not stored, they must be recomputed for each query element, which would increase the computational cost from quadratic to cubic in the sequence length—rendering it completely impractical for long sequences.

Concretely, the stored output of shape $b \times s \times d$ requires $2bsh$ bytes (in bfloat16). The paper's calculation for 100 million tokens with hidden size 1024 yields over 1000 GB, which is what makes the problem intractable on current hardware. The critical insight is that BPT reduces the working memory during computation but does not eliminate the need to store the full layer output. This is the gap that Ring Attention addresses.

Other parallelism strategies and their limitations. The paper briefly reviews several distributed training approaches and explains why none of them solve the long-context problem:

• Data parallelism shards the batch dimension across devices but leaves each device processing its entire assigned sequence, so the per-device sequence length is unchanged.
• Tensor parallelism splits individual operations (like matrix multiplications) across devices but still requires gathering full activations for attention, providing only partial memory reduction for some activations.
• Pipeline parallelism partitions layers across devices but still requires each device to handle the full sequence length for its assigned layers.
• Sequence parallelism (Li et al., 2023; Korthikanti et al., 2022) distributes the sequence dimension across devices—which is conceptually closer to what Ring Attention does—but introduces significant communication overhead that cannot be fully overlapped with computation. The paper notes (Section 3) that this non-overlapped communication makes sequence parallelism infeasible for large context sizes.
• Ring-based self-attention (Li et al., 2023, Section 6) uses a ring topology to reduce communication costs compared to standard sequence parallelism, but the paper argues that its communication still cannot be effectively overlapped with computation due to arithmetic intensity constraints, again rendering it impractical for long-context training.

Approximation-based approaches. Another line of work avoids exact attention computation entirely by using sparse attention patterns, low-rank approximations, kernel-based methods, or recurrence-based architectures. The paper acknowledges these in its related work section but is careful to distinguish its contribution: Ring Attention computes exact attention without approximations. The authors cite surveys (Narang et al., 2021; Tay et al., 2022) that document how approximation methods have often yielded sub-optimal results or encountered scaling difficulties. The paper's approach is therefore positioned not as an alternative to exact attention but as a way to make exact attention feasible at scales previously impossible.

The Paper's Positioning: Enabling Exact Attention at Arbitrary Scale

The paper frames Ring Attention through a specific architectural insight: when self-attention and feedforward computations are performed blockwise, the ordering of those blockwise computations is permutation-invariant. This is stated in Section 3 (and will be expanded in the Technical Approach section), but the key implication is that the outer loop of computing attention over query blocks can be distributed across devices, with each device responsible for its own query block and its associated FFN computations. These locally-responsible computations require no communication with other devices.

The challenge—and what differentiates Ring Attention from prior approaches—lies in the inner loop: each device's query block must attend to all key-value blocks, but each device only possesses its own key-value block. The naive solution is to fetch key-value blocks from other hosts, but this introduces both computation delays (waiting for transfers) and increased memory usage (accumulating received blocks), which defeats the purpose of memory reduction.

Ring Attention's solution is to arrange the hosts in a conceptual ring and, during the inner loop, have each device concurrently send its current key-value blocks to the next host in the ring while receiving key-value blocks from the previous host. As long as the block computation time exceeds the block transfer time, this overlapping results in zero additional communication overhead compared to a standard Transformer running on a single device. The paper explicitly contrasts this with prior ring-based self-attention work (Li et al., 2023), which it argues cannot achieve this overlap and therefore incurs non-overlapped communication costs similar to standard sequence parallelism.

This positioning—leveraging blockwise parallel transformers to reduce memory costs sufficiently that communication can be completely hidden behind computation—is what makes the paper's contribution novel. The prior ring-based approach reduced communication costs but didn't solve the memory problem that prevents the communication from being overlapped. Ring Attention solves both simultaneously, enabling context lengths that scale linearly with device count (as demonstrated in the experiments) with no overhead beyond what the computation itself requires.

The Practical Stakes: Why Near-Infinite Context Matters Now

The paper's motivation is not purely architectural. It is driven by emerging use cases that are practically important and theoretically interesting:

• Learning from trial-and-error experience in RL. The paper's experiments on the ExoRL benchmark (Section 5.3) show that conditioning a Transformer on 128 trajectories of 4000 tokens each (rather than the 32 trajectories used in prior work) improves performance. This matters because RL experience is inherently sequential and long-form—an agent's entire history of interactions informs its future decisions.
• Long-document understanding and retrieval. The line retrieval experiment (Section 5.4) evaluates whether the model can precisely retrieve a number from a long document. This capability is foundational for applications like processing legal contracts, scientific papers, code repositories, and books—all of which require the model to maintain coherent understanding over hundreds of thousands of tokens.
• The scaling of pretraining data. As language models are trained on increasingly large corpora, the ability to process longer contexts means each training example can provide richer signal about long-range dependencies, potentially improving sample efficiency and model quality.
• Multimodal and scientific applications. The introduction explicitly mentions large video-audio-language models, gene sequences, and scientific experiment data as domains where long-context processing is not optional but essential.

The paper thus positions its contribution as both systems-level (enabling training on hardware that otherwise could not handle these sequence lengths) and capability-enabling (unlocking applications that are currently impossible due to memory constraints). The emphasis on training—rather than just inference—is important because training is substantially more memory-intensive (requiring storage of activations for backpropagation), and solving the training memory problem is therefore the harder and more impactful challenge.

3. Technical Approach

3.1 Reader Orientation

Ring Attention is a distributed training and inference strategy that enables Transformers to process sequences whose length is proportional to the total number of available devices, by partitioning the sequence dimension across devices and orchestrating a ring-based communication pattern where key-value blocks circulate while computation proceeds, completely hiding the transfer latency behind useful work. It solves the fundamental memory bottleneck—storing the full layer output, which requires $2bsh$ bytes per layer even in prior state-of-the-art memory-efficient Transformers—by ensuring that no single device ever needs to hold more than a small constant number of sequence blocks in memory regardless of the total sequence length.

3.2 Big-Picture Architecture (Diagram in Words)

The system consists of five conceptual components arranged in a ring topology across $N_h$ hosts:

1. Sequence partitioner: splits the full input sequence of length $s$ into $N_h$ contiguous blocks of size $c = s / N_h$, assigning one block to each host. Each host computes its own query ($Q_i$), key ($K_i$), and value ($V_i$) projections from its assigned block.

2. Local blockwise attention engine: on each host, a memory-efficient attention implementation (e.g., FlashAttention) that computes the attention between the host's local query block and whatever key-value block it currently holds, updating running statistics incrementally. This is the computational core.

3. Ring communicator: a peer-to-peer communication mechanism built on jax.lax.ppermute that simultaneously sends the current key-value blocks from host-$i$ to host-$(i+1)$ while receiving key-value blocks from host-$(i-1)$. This transfer runs concurrently with the attention computation.

4. Local blockwise feedforward processor: after attention is complete for a given query block, the resulting output passes through a standard feedforward network, also computed blockwise on each host independently with no communication required.

5. Gradient synchronization (backward pass): during the backward pass, the identical ring pattern is used in reverse—gradients for key-value blocks flow backward through the ring while attention gradients are computed, using the same overlapping principle.

Information flows as follows: the input sequence enters the system → the sequence partitioner divides it across hosts → each host projects its chunk into $Q$, $K$, $V$ → the ring communicator begins circulating $K$ and $V$ blocks counterclockwise → on each step, each host computes local attention between its fixed $Q$ and the current $(K, V)$ it holds → after $N_h$ rotation steps, each host has accumulated the full attention output for its query block → each host applies its feedforward network independently → the process repeats for the next Transformer layer.

3.3 Roadmap for the Deep Dive

• First, the permutation invariance property of blockwise attention and why it is the mathematical foundation that makes distribution across devices possible without correctness sacrifice.
• Second, the incremental attention computation mechanism (the _blockwise_attention_fwd function) including the running statistics (numerator, denominator, max_score) that enable correct softmax across arbitrarily many partial computations.
• Third, the ring communication protocol—the lax.ppermute pattern, the lax.scan loop over rotation steps, and the arithmetic intensity condition that determines when overlapping is possible.
• Fourth, the memory analysis that shows why Ring Attention reduces per-device memory from $2bsh$ (prior SOTA) to $6bch$ (independent of total sequence length $s$) and the concrete block size requirements this implies for different hardware configurations.
• Fifth, the backward pass mechanics and why the same ring pattern applies symmetrically.
• Sixth, the integration with other parallelism strategies (FSDP, tensor parallelism) and how they compose without interference.

3.4 Detailed, Sentence-Based Technical Breakdown

This is a systems paper whose core idea is that by computing self-attention and feedforward networks in a blockwise fashion, the sequence dimension can be distributed across multiple devices with communication fully overlapped by computation, eliminating the per-device memory bottleneck that makes long-context training impossible. The key algorithmic insight is that the outer loop of blockwise attention (iterating over query blocks) is embarrassingly parallel across devices, while the inner loop (iterating over key-value blocks for a fixed query) can be arranged as a ring rotation where each device sees every other device's key-value blocks exactly once, with the transfers hidden behind the attention computation itself.

The Permutation Invariance Property: Why Distribution Works

The mathematical foundation of Ring Attention is the observation that blockwise self-attention is permutation-invariant with respect to the order in which key-value blocks are processed. This property is not stated as a formal theorem in the paper but is implicit in the incremental softmax computation (described in the next subsection) and is referenced directly in Section 3:

"This property stems from the fact that the self-attention between a query block and a group of key-value blocks can be computed in any order, as long as the statistics of each block are combined correctly for rescaling."

To understand what this means, consider a single query block $Q_i$ on host $i$. The full attention output for this query block is:

$O_i = \text{softmax}\left(\frac{Q_i K^T}{\sqrt{d}}\right) V$

where $K$ and $V$ are the concatenated key and value matrices from all sequence positions on all hosts. Computing this exactly requires access to every $(K_j, V_j)$ pair for $j \in \{1, \dots, N_h\}$. However, because softmax can be computed incrementally using running statistics (a maximum value for numerical stability and a sum for normalization), the contribution of each $(K_j, V_j)$ can be incorporated one at a time, in any order, without changing the final result.

The practical consequence is profound: host $i$ does not need all key-value blocks simultaneously in memory. It can process them sequentially as they arrive, updating its partial result. As long as each block is processed exactly once and the running statistics are maintained correctly, the output $O_i$ will be identical to what a single device would compute with all blocks in memory simultaneously. This property is what enables the ring communication pattern—it guarantees correctness regardless of the order in which blocks circulate.

The feedforward network applied after attention inherits this independence trivially: since the FFN operates position-wise (each token independently), host $i$ only needs its own attention output $O_i$ and can compute the FFN without any communication.

Incremental Blockwise Attention: How Statistics Are Maintained Correctly

The paper's actual implementation uses a blockwise attention function (_blockwise_attention_fwd in the JAX code, Figure 4, line 20) that implements the incremental softmax algorithm. While the paper does not re-derive this algorithm (it credits prior work by Milakov and Gimelshein, 2018; Rabe and Staats, 2021; Dao et al., 2022), understanding its mechanics is essential to understanding Ring Attention's correctness.

The challenge is that the standard softmax computation involves an exponentiation that can overflow for large values, followed by a normalization that requires the sum of all exponentials. When processing block by block, you cannot compute the final division until you have seen all blocks, but you also cannot store all intermediate scores because that would defeat the memory savings.

The solution maintains three running statistics for each query position across all key-value blocks processed so far:

1. prev_max_score: the maximum attention logit (pre-softmax score) seen so far, used to shift all values for numerical stability (preventing overflow in the exponential).

2. numerator: the running weighted sum of value vectors, where each value vector is weighted by the exponentiated and shifted attention score for its corresponding key. At any point, this represents the un-normalized attention output.

3. denominator: the running sum of exponentiated and shifted attention scores (the softmax normalization constant). At any point, this is the scale factor needed to normalize the numerator.

When a new key-value block $(K_j, V_j)$ arrives, the algorithm:

1. Computes the raw attention scores $S = Q_i K_j^T / \sqrt{d}$, yielding a matrix of shape (query_chunk_size, key_chunk_size).

2. Finds the maximum of the new scores and the previous maximum: new_max = max(prev_max_score, max(S, axis=-1)).

3. Rescales the existing numerator and denominator to account for the new maximum: since the previous statistics were computed relative to prev_max_score, and we now know the true maximum is new_max, the previous contributions must be scaled down by $\exp(\text{prev\_max} - \text{new\_max})$.

4. Adds the new block's contributions to the rescaled numerator and denominator, with the new scores shifted by new_max for stability.

5. Returns the updated statistics, which now represent the correct partial result for all blocks processed so far.

After all $N_h$ blocks have been processed, the final attention output is output = numerator / denominator (after appropriate reshaping), which is exactly the result of computing full attention with all key-value blocks present simultaneously.

The paper's code (Figure 4, lines 20-22) passes numerator, denominator, and prev_max_score as a carry state through the lax.scan loop, and the _blockwise_attention_fwd function updates these in-place:

numerator, denominator, max_score = _blockwise_attention_fwd(q, k, v,
    (numerator, denominator, prev_max_score), q_chunk_idx_start, k_chunk_idx_start,
    bias=attn_bias_slice, **blockwise_kwargs)

The q_chunk_idx_start and k_chunk_idx_start parameters specify which sub-blocks of the query and key to process, allowing the function to work with sub-chunk granularity when the block size exceeds the desired computational chunk size (controlled by query_chunk_size and key_chunk_size parameters).

Why this form matters: This incremental algorithm is what makes the ring topology possible. Without it, each host would need to store all received key-value blocks, accumulating memory linearly with the number of blocks. With it, each host only needs memory for the current $(K, V)$ block plus the three running statistics, which are dimensioned by the query block size, not the total sequence length. The correctness guarantee—identical results to full-materialization attention—is what distinguishes Ring Attention from approximation-based methods.

The Ring Communication Protocol: Overlapping Transfer with Computation

The ring communication protocol is the mechanism by which each host gains access to every other host's key-value blocks without ever holding more than two key-value blocks simultaneously (its own and the one currently being received). The protocol is implemented as a lax.scan over $N_h$ steps (Algorithm 1 and Figure 4).

Initial state. Before the ring begins, each host $i$ has already computed $Q_i$, $K_i$, and $V_i$ from its assigned input block. The running statistics (numerator, denominator, prev_max_score) are initialized to zero (or negative infinity for prev_max_score).

The scan loop. On each iteration idx from $0$ to $N_h - 1$ (Figure 4, lines 12-25 and 28), each host concurrently performs three operations in sequence:

1. Local attention computation: Host $i$ computes the incremental attention between its fixed query block $Q_i$ and the key-value block $(K_{\text{current}}, V_{\text{current}})$ it currently holds. The attn_bias_slice is dynamically extracted based on the relative position of the current key-value block in the sequence, using lax.dynamic_slice_in_dim (line 14-15). The block indices are computed as:

   q_block_idx = lax.axis_index(axis_name)
   k_block_idx = (lax.axis_index(axis_name) - idx) % axis_size
   

   This indexing is critical: on iteration idx, host $i$ holds the key-value block that originally belonged to host $(i - \text{idx}) \bmod N_h$ (assuming blocks rotate counterclockwise). The k_block_idx computation accounts for this rotation so that the correct attention bias slice is applied to the correct block.

2. Communication: After the local computation, host $i$ simultaneously sends its current key-value blocks to host $(i + 1) \bmod N_h$ and receives new key-value blocks from host $(i - 1) \bmod N_h$, using jax.lax.ppermute (lines 23-24):

   k, v = map(lambda x: lax.ppermute(x, axis_name, 
       perm=[(i, (i + 1) % axis_size) for i in range(axis_size)]), (k, v))
   

   The ppermute collective operation performs a send-receive permutation: each device sends a chunk of data to a specified neighbor and receives a chunk from another specified neighbor. The perm argument is a list of (source, destination) pairs. For a ring, the permutation is [(0,1), (1,2), ..., (N_h-2, N_h-1), (N_h-1, 0)], meaning each host sends to the next host and receives from the previous host.

3. Carry update: The updated running statistics (max_score, numerator, denominator) and the newly received key-value blocks (k, v) become the carry state for the next iteration.

After $N_h$ iterations, each host has processed all $N_h$ key-value blocks exactly once, and the accumulated attention output for its query block is complete. The feedforward network is then applied independently on each host (Algorithm 1, "for For each host concurrently, compute memory efficient feedforward using local attention output").

The overlap mechanism. The crucial performance property is that the communication (step 2) happens asynchronously with respect to the computation when using JAX's SPMD framework. The paper argues that as long as the time to compute the blockwise attention for a single key-value block exceeds the time to transfer that block between hosts, the ppermute can be fully overlapped with the subsequent attention computation. In practice, this means:

• While host $i$ is computing attention between $Q_i$ and the key-value block it received in the previous iteration, it simultaneously sends the key-value block it just finished using to host $(i+1)$ and receives the next key-value block from host $(i-1)$.
• The receive operation must complete before the next attention computation begins. If transfer time < compute time, the received block is already in memory when needed, and no idle cycles are wasted on waiting.

This overlap is what distinguishes Ring Attention from prior sequence parallelism approaches, which (according to the paper) could not fully overlap communication with computation, leading to significant overhead that made long-context training impractical.

The attention bias handling. A subtle implementation detail visible in the code (lines 14-15 and 44-45) is how attention biases (e.g., causal masks for autoregressive models) are managed. The bias tensor for the full attention matrix (shape $s \times s$) is also blockwise. For each $(Q_i, K_j)$ pair, the relevant bias slice is extracted using dynamic slicing based on k_block_idx and q_block_idx. This ensures that causal masking, relative position biases, or any other position-dependent bias is applied correctly even though the key-value blocks arrive in a rotated order.

Arithmetic Intensity and the Minimal Block Size Condition

The paper provides a formal condition for when overlapping communication with computation is possible, derived from the ratio of FLOPs to bandwidth between hosts (Section 3, "Arithmetic Intensity Between Hosts").

The computation cost. When computing blockwise self-attention for a query block of size $c$ and a key-value block of size $c$ with hidden dimension $d$:

• Computing the attention scores $Q K^T$ requires $2dc^2$ FLOPs (a matrix multiplication of shape $(c, d) \times (d, c)$ takes approximately $2c^2 d$ operations).
• Multiplying the attention scores by the values $V$ requires another $2dc^2$ FLOPs (multiplying scores of shape $(c, c)$ by values of shape $(c, d)$).

Total computation per block: $4dc^2$ FLOPs. The paper explicitly excludes the projection of queries, keys, and values, as well as blockwise feedforward operations, "since they only add compute complexity without any communication costs between hosts. This simplification leads to more stringent condition and does not compromise the validity of our approach."

The communication cost. Sending both the key block (shape $c \times d$) and the value block (shape $c \times d$) in bfloat16 (2 bytes per element) requires $2 \cdot c \cdot d \cdot 2 = 4cd$ bytes total.

The overlap condition. For communication to be fully overlapped with computation, the time to transfer must be less than or equal to the time to compute:

$\frac{\text{Computation FLOPs}}{\text{Device FLOPs/second}} \geq \frac{\text{Communication bytes}}{\text{Interconnect bandwidth bytes/second}}$

Substituting the expressions:

$\frac{4dc^2}{F} \geq \frac{4cd}{B}$

where $F$ is the device FLOPs (in FLOPs/second) and $B$ is the interconnect bandwidth (in bytes/second). Simplifying:

$4dc^2 \cdot B \geq 4cd \cdot F$

$c \cdot B \geq F$

$c \geq \frac{F}{B}$

The minimal block size $c_{\text{min}} = F / B$, measured in tokens (since $c$ is the number of sequence positions per block). Since the per-device sequence length $s_{\text{device}} = 6c$ (as derived in the memory analysis below), the minimal per-device sequence length is $6 \cdot F / B$.

What this means concretely (Table 2). For hardware configurations, the paper calculates:

For GPUs connected via NVLink or TPUs (which have high-bandwidth interconnects), the minimal sequence length per device is approximately 6,000–7,000 tokens—a very modest requirement easily met in practice. For GPUs connected via InfiniBand (lower bandwidth), the requirement jumps to roughly 150,000 tokens per device, which is still achievable but more stringent. If the per-device sequence length falls below these thresholds, the communication cannot be fully hidden behind computation, and some transfer latency will be exposed—but the system still functions correctly.

The paper notes (Appendix C) that for inference (where the query is a single token in autoregressive generation), the condition becomes even more stringent because the computational FLOPs per query token are much smaller ($O(d^2)$ rather than $O(dc^2)$). However, the paper's primary focus is training, where the large block sizes make overlapping straightforward.

Memory Analysis: Why Ring Attention Achieves Sequence-Length-Independent Memory

The memory analysis in Section 3 ("Memory Requirement") explains why Ring Attention eliminates the fundamental bottleneck that prior state-of-the-art approaches could not address. To understand this, it is essential to trace exactly what each device must hold in memory at any point during the forward pass.

What a device holds at any moment. During the ring rotation, on host $i$, the following blocks are simultaneously in memory:

1. One query block ($Q_i$, shape $b \times c \times d$): the host's assigned portion of the input sequence, which remains fixed throughout the ring rotation. Memory: $2bch$ bytes (bfloat16).

2. One current key block and one current value block ($K_{\text{current}}$ and $V_{\text{current}}$, each shape $b \times c \times d$): the blocks that the host is currently using for attention computation. Memory: $4bch$ bytes total ($2bch$ each).

3. One received key block and one received value block (shape $b \times c \times d$ each): the blocks being received from the previous host in the ring for the next iteration. Memory: $4bch$ bytes total.

4. One output block (shape $b \times c \times d$): where the accumulated attention output is stored (the numerator accumulator, which has the same shape as the query block since it is a weighted sum of value vectors). Memory: $2bch$ bytes.

5. Running statistics: denominator (shape $b \times n_h \times c$) and prev_max_score (shape $b \times n_h \times c$), which are negligible compared to the block storage.

Total activation memory for the ring attention operation: $2 + 4 + 4 + 2 = 12bch$ bytes? Actually, the paper states $6bch$ bytes total. Let me reconcile this.

The paper's accounting in Section 3 states:

"A host needs to store multiple blocks, including one block size to store the current query block, two block sizes for the current key and value blocks, and two block sizes for receiving key and value blocks. Furthermore, storing the output of blockwise attention and feedforward necessitates one block size, as the output retains the shape of the query block. Therefore, a total of six blocks are required, which translates to 6bch bytes of memory."

The discrepancy is resolved by noting that "block size" in the paper's terminology refers to bch bytes (the memory for one tensor of shape $b \times c \times d$ in bfloat16). So:

• 1 block for query: $1 \cdot bch$ bytes
• 2 blocks for current key and value: $2 \cdot bch$ bytes
• 2 blocks for receiving key and value: $2 \cdot bch$ bytes
• 1 block for output: $1 \cdot bch$ bytes

Total: $6bch$ bytes. However, the key and value blocks are each one block, so the "two block sizes" in the quote refers to one for key and one for value, each being one block. This matches $6bch$ for the attention component.

Adding the feedforward memory. The blockwise feedforward network (from BPT) requires storing its own activations. Following Liu and Abbeel (2023), the maximum activation size for blockwise FFN is $2bch$ bytes (the intermediate representation after the first linear transformation, which has shape $b \times c \times d_{\text{ff}}$ but with $d_{\text{ff}} = 4d$, would naively be $8bch$, but BPT reduces this through blockwise computation). However, the paper's total of $6bch$ appears to already incorporate both attention and FFN, since it states:

"It's worth noting that the blockwise feedforward network has a maximum activation size of 2bch. Consequently, the total maximum activation size remains at 6bch bytes."

This implies that some of the six blocks are reused or that the FFN memory is subsumed within the existing allocation. The precise accounting: the FFN processes the attention output block (which is already counted as the output block). The FFN's intermediate activations ($2bch$) might be stored in the same memory as the received key-value blocks after those have been processed, since the FFN computation happens after all key-value blocks have been rotated through (lines 26-28 in Algorithm 1, where the FFN is computed in a separate loop after the attention scan completes).

Comparison with prior approaches (Table 1). This is where the dramatic advantage becomes clear:

The critical difference: every prior method has memory that scales with the total sequence length $s$, specifically $2bsh$ bytes (the output of each layer). Ring Attention's memory scales only with the block size $c$, where $c = s / N_h$. Since $N_h$ is the number of hosts, the per-device memory is independent of the total sequence length—it depends only on how the sequence is partitioned.

For example, if $s = 100$ million tokens, $d = 1024$, $b = 1$, and $N_h = 1024$, then $c = 100\text{M} / 1024 \approx 97{,}656$. Ring Attention memory per device: $6 \cdot 1 \cdot 97{,}656 \cdot 1024 \cdot 2 \text{ bytes} \approx 1.2$ GB. In contrast, BPT memory would be $2 \cdot 1 \cdot 100{,}000{,}000 \cdot 1024 \cdot 2 \text{ bytes} \approx 409$ GB—impossible on any single device. Ring Attention makes the previously impossible possible by distributing the $2bsh$ output across $N_h$ devices.

The factor of 6 in $6bch$ is specific to Ring Attention's implementation. It accounts for the query block (1), the key and value blocks being processed (2), the key and value blocks being received (2), and the output block (1). This is a constant factor independent of sequence length and number of devices.

Why this matters for training. During training, activations must be stored for the backward pass (unless gradient checkpointing is used). Ring Attention's memory savings apply to both forward activations and the storage needed for backpropagation. Combined with gradient checkpointing (which the paper uses, following prior work), the per-device memory is dominated by $6bch$, which for typical block sizes of 1K–100K tokens per device translates to hundreds of megabytes to a few gigabytes—well within the 80–100 GB HBM of modern accelerators.

The Backward Pass: Symmetric Ring Operation

The paper states (Section 3) that the overlapping mechanism "applies to both forward and backward passes of our approach since the same operations and techniques can be used." The JAX implementation in Figure 4 (lines 32-57, _ring_attention_bwd) confirms this symmetry.

The backward pass of attention computes gradients with respect to $Q$, $K$, and $V$ given the gradient of the loss with respect to the attention output. The mathematical form involves two matrix multiplications per $(Q_i, K_j, V_j)$ triple:

• The gradient with respect to $V_j$ involves the attention weights: $dV_j = \text{softmax\_weights}^T \cdot dO_i$.
• The gradient with respect to $Q_i$ involves the values: $dQ_i$ accumulates $(\text{softmax\_weights\_derived}) \cdot V_j^T$.
• The gradient with respect to $K_j$ involves the queries: $dK_j$ accumulates $Q_i^T \cdot (\text{softmax\_weights\_derived})$.

Each of these operations requires access to both the query block $Q_i$ (which is local to host $i$) and the key-value block $(K_j, V_j)$ (which rotates through the ring). The backward pass therefore uses the identical ring rotation pattern: during the backward scan (Figure 4, lines 42-54), the key-value blocks and their gradient accumulators rotate through the ring. On each step:

1. Host $i$ computes the local contribution to $dQ_i$, $dK_j$, and $dV_j$ using the query $Q_i$, the current $(K_j, V_j)$, the output gradient $dO_i$, and the stored statistics from the forward pass (output, denominator, max_score).

2. The gradients $dK_j$ and $dV_j$ are sent along with $K_j$ and $V_j$ to the next host, so that each host accumulates gradients for its own key-value blocks as they complete their rotation.

After $N_h$ steps, each host has accumulated the full gradient for its own query block ($dQ_i$) and its own key-value blocks ($dK_i$, $dV_i$). The communication pattern is identical to the forward pass: lax.ppermute rotates $(k, v, dk, dv)$ simultaneously (lines 52-53).

The arithmetic intensity analysis for the backward pass yields the same $c \geq F/B$ condition because the FLOPs-to-bytes ratio is similar (each backward step involves roughly twice the FLOPs of a forward step but also twice the data movement for gradients).

Integration with Other Parallelism Strategies

The paper emphasizes (Section 5.1, Section 5.2, Appendix A) that Ring Attention is orthogonal to and composable with other parallelism strategies. This is not a minor detail—it is what makes the approach practically deployable in large-scale training.

Fully Sharded Data Parallelism (FSDP). FSDP shards model parameters across devices, so each device only stores and computes with a fraction of the model's weights. Ring Attention and FSDP operate on orthogonal dimensions: FSDP shards the model (parameter dimension), while Ring Attention shards the sequence (data dimension). The paper's experiments use both simultaneously: "On $n$ devices, FSDP is used to shard the model for baselines, which gives a sequence length of $l$. We utilize FSDP along with Ring Attention to extend the sequence length to $nl/m$ and $m$ sequences" (Section 5.1). The total batch size in tokens remains constant, allowing fair comparison.

The practical consequence: for a 30B model, which cannot fit on a single device, FSDP distributes the model parameters across (say) 8 devices. The remaining devices (out of the total pool) can be used for Ring Attention to extend context length. On 512 A100 GPUs with a 30B model, dividing 8 ways for FSDP leaves 64-way Ring Attention, enabling 64× longer context than without Ring Attention.

Tensor parallelism. Tensor parallelism partitions individual operations (matrix multiplications) across devices, reducing the activation memory per device for each operation. The paper states (Section 5.2) that tensor parallelism "can only reduce parts of activations memory" and is "independent of tensor parallelism," meaning Ring Attention can be applied on top of tensor-parallel sharding without conflict. The mesh dimensions in the provided codebase can be adjusted to allocate devices across data, tensor, and Ring Attention axes.

Gradient checkpointing. Following prior work (Rabe and Staats, 2021; Liu and Abbeel, 2023), the paper uses full gradient checkpointing (also known as rematerialization) on both attention and feedforward networks. This trades computation for memory: activations are not stored during the forward pass and are recomputed during the backward pass. The effective memory usage becomes the maximum of the forward-pass activation memory and the stored-checkpoint memory (inputs to each layer), rather than the sum over all layers. Ring Attention's $6bch$ per-layer memory is the forward-pass working memory; with checkpointing, only the layer inputs ($2bch$ each) need to be stored between layers, and the attention/FFN activations are recomputed on-the-fly during backpropagation using the same ring rotation pattern.

The Algorithm Pseudocode and Implementation Details

Algorithm 1 in the paper provides a concise pseudocode that captures the essential structure:

Algorithm 1: Reducing Transformers Memory Cost with Ring Attention.
Required: Input sequence x. Number of hosts N_h.

Initialize:
  Split input sequence into N_h blocks that each host has one input block.
  Compute query, key, and value for its input block on each host.

for Each transformer layer do
  for count = 1 to N_h - 1 do
    for For each host concurrently do
      Compute memory efficient attention incrementally using local query, key, value blocks.
      Send key and value blocks to next host and receive key and value blocks from previous host.
    end for
  end for
  for For each host concurrently do
    Compute memory efficient feedforward using local attention output.
  end for
end for

A few implementation details that are important for understanding how this works in practice:

Why $N_h - 1$ iterations instead of $N_h$? The pseudocode shows the inner loop running for $N_h - 1$ iterations. This is because on iteration 0 (before the loop), each host already holds its own key-value block and has (presumably) already computed attention with it. Alternatively, the initialization step could be folded into the loop as iteration 0. The JAX code uses the scan loop over jnp.arange(0, axis_size) (line 28), which runs $N_h$ iterations, with the first iteration computing attention with the host's own key-value block.

The lax.scan construct. The implementation uses jax.lax.scan (Figure 4, lines 27-28), which is a higher-order function that applies a function repeatedly, threading a carry state through each iteration. The scan function scan_kv_block (lines 12-25) takes the carry (prev_max_score, numerator, denominator, k, v) and the iteration index idx, and returns an updated carry. The use of scan (rather than a Python for-loop) is essential for JAX's JIT compilation—it unrolls into a compiled loop that can be optimized by XLA.

The defvjp custom gradient. Lines 59-64 define ring_attention as a @partial(jax.custom_vjp, ...) function, which allows specifying custom forward and backward passes. This is necessary because the ring rotation pattern is not automatically differentiable by JAX's standard autograd—the ppermute operations and the scan loop require explicit gradient definitions. The backward pass (_ring_attention_bwd, lines 32-57) mirrors the forward pass structure, computing gradients for $Q$, $K$, $V$, and the attention bias incrementally as key-value blocks rotate.

Block size and chunk size parameters. The code references query_chunk_size and key_chunk_size as part of blockwise_kwargs. These control sub-block granularity within each host's block. For instance, if a host's block is 8K tokens and query_chunk_size is 1K, the attention computation is further split into 8 sub-operations. This sub-chunking reduces peak memory within a single attention operation but is orthogonal to the ring distribution—it operates within the host's local computation. The q_chunk_idx_start and k_chunk_idx_start parameters track which sub-chunk is being processed.

Design Choices: Why This Approach Over Alternatives

The paper's design choices are motivated by a set of explicit comparisons to alternatives:

Why distribute over sequence dimension rather than batch or model dimensions? Data parallelism (batch distribution) and tensor/pipeline parallelism (model distribution) do not reduce the per-device sequence length, which is the bottleneck for long-context training. Sequence parallelism directly targets the bottleneck, and Ring Attention is a specific form of sequence parallelism that achieves zero communication overhead through overlapping.

Why a ring topology rather than all-to-all? An all-to-all approach where each device fetches key-value blocks from every other device on demand would require either (a) storing all received blocks simultaneously (defeating memory savings) or (b) coordinating point-to-point transfers with non-trivial scheduling overhead. The ring topology has the key property that each device only communicates with two neighbors, minimizing contention, and the fixed rotation order means no scheduling decisions are needed at runtime.

Why blockwise attention rather than approximation methods? The paper explicitly targets exact attention computation, distinguishing itself from sparse, low-rank, kernel-based, and recurrent approximations. The rationale, though not deeply argued in the paper, is implicit in the introduction's reference to approximation methods yielding "sub-optimal results or challenges during scaling up" (Section 6, via Narang et al., 2021; Tay et al., 2022). Exact attention guarantees that no information is lost due to the model architecture—the only constraints are hardware capacity, which Ring Attention expands linearly with device count.

Why not use prior ring-based self-attention (Li et al., 2023)? The paper explicitly contrasts with this work, stating that it "incurs non-overlapped communication overheads similar to sequence parallelism, making it infeasible for large context sizes" (Section 1) and "overlapping communication with computation remains challenging due to the constraints of arithmetic intensity" (Section 6). The key difference is that prior ring-based attention did not use blockwise parallel transformers (BPT) to reduce memory costs, which meant the per-device memory was still $O(s)$, preventing the communication from being effectively overlapped because the computation was memory-bound rather than compute-bound at the block granularity. Ring Attention uses BPT's blockwise computation to make the attention operation compute-bound at the block level, creating the arithmetic intensity needed for overlap.

Why six blocks of memory? The factor of 6 ($6bch$ bytes) emerges from the hardware constraints of the ring pattern: one block for the fixed query, two for the active key-value pair, two for the incoming key-value pair (double-buffering to enable overlap), and one for the output. This is a constant factor determined by the ring protocol design. If single-buffering were used (overwriting the current block with the incoming block), the memory would be $4bch$ but overlapping would be impossible because the receive would have to complete before the current computation finishes, exposing latency. The double-buffering (keeping both the current and incoming blocks) is what enables the zero-overhead property.

4. Key Insights and Innovations

Innovation 1: Memory-Per-Device Decoupled from Total Sequence Length

The central conceptual move in this paper is not the ring topology itself—prior work (Li et al., 2023) already explored ring-based self-attention communication—but the recognition that blockwise computation of both self-attention and feedforward networks fundamentally changes what memory distribution means. Before Ring Attention, the dominant framing of the long-context memory problem was that it was a communication challenge: you need to move sequence data between devices so each device can compute attention, and the goal is to minimize the communication cost. The ring topology in Li et al. (2023) was evaluated on exactly these terms—does it reduce communication bytes relative to all-to-all sequence parallelism?

Ring Attention reframes the problem entirely by rejecting the premise that each device needs to hold the full sequence output at any point. The key intellectual move is the observation that when attention and feedforward are both computed blockwise, the output of each layer—which prior state-of-the-art methods (Rabe and Staats, 2021; Dao et al., 2022; Liu and Abbeel, 2023) all required to be stored in its entirety at $2bsh$ bytes per layer—can itself be distributed across devices because it is produced blockwise and consumed blockwise by the next layer's attention. This is not obvious from the blockwise attention literature, which focused on reducing working memory during the attention computation but tacitly accepted that the final layer output had to be accumulated and stored for the next layer. The Blockwise Parallel Transformer (Liu and Abbeel, 2023) reduced the total memory to $2bsh$, but this still scales with total sequence length $s$—which is why 100 million tokens still requires over 1000 GB even with BPT.

Ring Attention's $6bch$ memory, where $c = s / N_h$, is the first Transformers architecture where per-device memory is independent of the total sequence length $s$ and scales only with the local block size. This is a qualitative change, not a quantitative improvement. It means that as you add more devices, you can process proportionally longer sequences with exactly the same per-device memory footprint—the memory bottleneck imposed by individual devices is eliminated as a constraint on context length. The paper's results in Table 3 (e.g., 512× improvement on TPUv4-1024 for 3B models, 256× on TPUv3-512 for 7B models) are not merely "better scaling"—they demonstrate that the $2bsh$ barrier that all prior exact-attention methods hit has been broken. The context length is now bounded by the total aggregate memory across all devices rather than the memory of any single device, which is the architectural property that justifies the "near-infinite context" framing in the paper's title.

This is a fundamental shift, not an incremental refinement. It changes the answer to "how long a sequence can I train on?" from "whatever fits in one device's HBM" to "whatever fits across all my devices' HBMs combined, divided by a constant factor of 6." The factor of 6 is an implementation constant, not a structural limitation—it comes from the specific buffering strategy (query block, current KV, incoming KV, output), and could potentially be reduced further. What matters structurally is that the $s$ term disappears from the per-device memory expression.

Innovation 2: Communication-Computation Overlap as an Arithmetic Intensity Problem Solvable by Blockwise Transformers

The paper's second conceptual contribution is diagnosing why prior sequence parallelism and ring-based attention could not achieve zero-overhead communication, and showing that the bottleneck was not communication volume but arithmetic intensity—the ratio of computation FLOPs to communication bytes at the granularity of block operations.

Prior sequence parallelism approaches (Li et al., 2023; Korthikanti et al., 2022; Jacobs et al., 2023) treated the communication challenge as one of minimizing the total bytes transferred. The ring topology reduced communication from $O(N_h^2)$ (all-to-all) to $O(N_h)$ (nearest-neighbor), which was an improvement in total bandwidth consumption. But these approaches still could not fully overlap communication with computation because the per-block attention computation was not compute-intensive enough relative to the per-block transfer time—the arithmetic intensity was too low. The paper's explicit contrast with Li et al. (2023) in Section 6 is that "overlapping communication with computation remains challenging due to the constraints of arithmetic intensity. The communication overheads render this approach infeasible for training and inference in large-context scenarios."

Ring Attention's innovation is recognizing that blockwise parallel transformers solve this arithmetic intensity problem indirectly. By computing both attention and feedforward blockwise, the working set per block shrinks dramatically (from $O(s \cdot d)$ to $O(c \cdot d)$), which makes the block-level attention computation actually compute-bound rather than memory-bound. This shifts the bottleneck from HBM bandwidth to FLOPs, which creates the arithmetic intensity headroom needed to hide the block transfer behind computation. The condition derived in Section 3—$c \geq F / B$—is a crisp, formal characterization of when overlap is possible, and it connects the algorithmic design (blockwise computation) directly to hardware parameters (FLOPs and interconnect bandwidth).

This is significant beyond the specific ring implementation because it establishes a general design principle: for distributed exact attention to achieve zero communication overhead, the per-block computation must be compute-bound, not memory-bound. This insight is not parameterized by model size or sequence length alone—it depends on the block granularity and the FLOPs-to-bandwidth ratio of the hardware. The paper's Table 2 demonstrates the practical consequence: on NVLink-connected GPUs and TPUs, this condition is trivially met with block sizes around 1K tokens, making Ring Attention immediately practical. On InfiniBand-connected GPUs, the block size requirement jumps to ~25K tokens, which is more demanding but still achievable for long-context training.

This is a fundamental diagnostic contribution rather than an incremental engineering improvement. It reframes the distributed attention problem from a bandwidth-minimization problem to an arithmetic-intensity-optimization problem, and it shows that the solution is not a better communication topology but a better memory management strategy at the block level. The $4dc^2$ FLOPs per block computation is not incidental—it is what makes the whole scheme work, and the paper is explicit that this is enabled specifically by BPT's memory reduction, which is what allows the block computation to be compute-bound.

Innovation 3: Empirical Demonstration That Training Is the Harder Problem and Is Solved First

Most work on long-context Transformers focuses on inference—techniques like key-value caching, sliding window attention, or recurrent state compression are primarily designed to make autoregressive generation with long contexts feasible. Training is typically acknowledged as harder (because activations must be stored for backpropagation) but addressed secondarily. The paper inverts this: Ring Attention is designed and evaluated primarily for end-to-end training of long-context models, with inference treated as a strictly easier special case (Appendix C).

This is a distinctive conceptual choice with practical consequences. The paper argues in Section 2 that even with memory-efficient attention and blockwise feedforward, training requires storing the full layer output between layers—which is $2bsh$ bytes that no prior method could eliminate. Inference, by contrast, processes one query token at a time in autoregressive generation, making the query block size $c=1$ and the per-token computation much smaller. The paper notes (Appendix C) that for inference on TPUv5e with a LLaMA 7B, the conventional approach handles up to 256K context through head-parallel distribution, and Ring Attention can extend this by 32×—but the arithmetic intensity condition becomes more stringent because the per-token FLOPs are $O(d^2)$ rather than $O(c \cdot d^2)$. Training is where the block sizes are large enough to make overlapping trivial, and training is where the memory pressure is highest.

The significance of this focus is that it addresses a harder, more impactful bottleneck. If you can only do long-context inference but not training, you are limited to whatever context length the pretrained model was trained on—you cannot teach the model new long-range dependencies. Ring Attention enables training on sequences of arbitrary length, which means models can learn to reason over long contexts rather than merely being evaluated on them. The paper's experiments on the ExoRL benchmark (Table 5), where training on 128 trajectories of 4000 tokens each improves over training on 32 trajectories, and the line retrieval task (Figure 3), where finetuning LLaMA-13B on 512K tokens improves long-context retrieval accuracy, both demonstrate that the training capability translates to downstream performance gains—not just the ability to process longer inputs.

This is a fundamental reframing of the long-context problem. Prior work implicitly accepted a split: use approximations or memory-efficient inference for deployment, but accept that training must operate within device memory limits. Ring Attention rejects this split and shows that exact-attention training at extreme scales is possible with existing hardware, which in turn enables capabilities (like learning from hundreds of RL trajectories or processing entire codebases) that were previously confined to the realm of future hardware projections.

Innovation 4: The Permutation Invariance of Blockwise Attention as an Architectural Primitive for Distribution

While the paper does not state this as a formal theorem, the intellectual core that makes Ring Attention possible is the recognition that blockwise attention exhibits a specific kind of permutation invariance: the order in which key-value blocks are processed does not affect the final attention output, as long as the incremental softmax statistics are maintained correctly. This property is implicit in prior blockwise attention work (the tiling technique of Milakov and Gimelshein, 2018, and the memory-efficient attention of Rabe and Staats, 2021, both rely on it for single-device computation), but prior work treated it as a mechanism for reducing peak memory, not as a distribution primitive.

Ring Attention elevates this property to an architectural design principle: because the inner loop of blockwise attention is permutation-invariant, the outer loop (iterating over query blocks) can be arbitrarily distributed across devices, and the inner loop can be implemented as any communication pattern that ensures each device sees each key-value block exactly once. The ring topology is one realization of this principle—specifically, a shifted rotation where device $i$ sees blocks in the order $(i, i-1, i-2, \dots, i+1)$. But the principle is more general: any topology that guarantees complete coverage of key-value blocks with acceptable communication characteristics (e.g., a tree, a hypercube, or a random permutation schedule) would produce identical attention outputs. The ring is chosen for its simplicity, its nearest-neighbor communication pattern, and its natural fit with the overlapping requirement.

This is a conceptual contribution distinct from the specific ring implementation. It reframes the distributed attention problem as a coverage problem on a graph of devices rather than a synchronization or partitioning problem. The fact that the attention computation doesn't care about the order of key-value blocks means there is no sequential dependency between devices—no host needs to wait for another host to finish before starting its computation, except for the transfer of the next block. This is what enables the zero-overhead property: the only synchronization is the point-to-point transfer, which is hidden behind computation.

The significance of this insight extends beyond this paper. If the permutation invariance holds (and the paper demonstrates it does for exact attention), then the design space for distributed attention is much larger than previously explored. Future work could investigate whether different communication schedules could reduce latency further, handle heterogeneous hardware, or tolerate device failures by re-routing blocks—all while producing identical mathematical results. The paper does not explore these directions, but the conceptual framework it establishes makes them natural next steps.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper evaluates on the MATH benchmark (Hendrycks et al., 2021), using the exact train/test split from Lightman et al. (2022): 12,000 training questions and 500 test questions. MATH consists of high-school competition-level mathematics problems requiring multi-step symbolic reasoning. For the RL experiments, the ExoRL benchmark (Yarats et al., 2022) is used across six continuous control tasks (Walker Stand, Walker Run, Walker Walk, Cheetah Run, Jaco Reach, Cartpole Swingup), with data collected via unsupervised RL exploration. For the language modeling long-context evaluation, a line retrieval test (Li et al., 2023) is used where the model must precisely retrieve a number from a long document, evaluated by retrieval accuracy as a function of context length. The finetuning data is 125K user-shared conversations from ShareGPT.com, cleaned and converted from HTML to markdown.

• Base model(s). The experiments use the LLaMA architecture (Touvron et al., 2023) at four model scales: 3B, 7B, 13B, and 30B parameters. For the RL experiments, a 350M parameter model (not further specified) is used. The long-context finetuning experiment starts from LLaMA-13B. The paper states these models are "representative of the capabilities of many contemporary LLMs" and span a range of scales where the memory bottleneck manifests differently—smaller models can be fully sharded for Ring Attention on available devices, while larger models require FSDP to even fit parameters, reducing the devices available for sequence distribution. The 65B scale is included in MFU experiments (Table 4 in the original paper, Table 5.1 in this analysis).

• Metrics. Three distinct metrics are used across experiments. Maximum context length (Table 3): the largest number of tokens that can be processed in an end-to-end training configuration without running out of accelerator memory, reported in thousands of tokens (×1e3). This is determined empirically by increasing sequence length until memory exhaustion occurs, using the same batch size in total tokens across baselines and Ring Attention to ensure fair comparison. Model FLOPs utilization (MFU) (Table 4 in original, Table 5.1 here): the ratio of achieved FLOPs to theoretical peak FLOPs of the hardware, expressed as a percentage. This measures how efficiently the hardware is used—higher MFU means less idle time and better throughput. For the RL experiments, the metric is cumulative return on the ExoRL benchmark tasks, following the evaluation protocol of Yarats et al. (2022). For the line retrieval experiment, the metric is retrieval accuracy—the fraction of test cases where the model correctly retrieves the target number—evaluated at multiple context lengths to produce an accuracy-vs-length curve (Figure 3).

• Baselines. Four baselines are used, each representing a point on the spectrum of Transformer memory efficiency:

  • Vanilla Transformer (Vaswani et al., 2017): standard attention computation with full $s \times s$ matrix materialization and standard feedforward computation.
  • Memory Efficient Attention (Rabe and Staats, 2021; Dao et al., 2022): attention computed blockwise without materializing the full softmax matrix, using the tiling technique. FFN is computed normally.
  • Memory Efficient Attention and Feedforward / Blockwise Parallel Transformer (BPT) (Liu and Abbeel, 2023): both attention and FFN computed blockwise, reducing total activation memory to $2bsh$ bytes per layer—the prior state-of-the-art.
  • For the RL experiments, additional baselines include Behavioral Cloning (BC-10%) and Decision Transformer (DT) (Chen et al., 2021), both using vanilla attention. Agentic Transformer (AT) (Liu and Abbeel, 2023) is the primary base method, and AT is evaluated with vanilla attention, with memory-efficient attention (AT+ME), with BPT (AT+BPT), and with Ring Attention (AT+Ring Attention).

• Generation budget / compute accounting. Compute is measured in two ways for different experiments. For maximum context length experiments, fairness is enforced by keeping total batch size in tokens constant across baselines and Ring Attention: on $n$ devices, FSDP provides a baseline sequence length $l$, yielding $n \cdot l$ total tokens per batch. Ring Attention uses the same $n$ devices to increase sequence length to $n \cdot l / m$ for $m$ sequences, maintaining the same total token count. For MFU experiments, the batch size in tokens is fixed at 2M per batch on GPUs and 4M per batch on TPUs, following the OpenLLaMA training recipe (Geng and Liu, 2023). All results use full precision (not mixed precision) and full gradient checkpointing on both attention and feedforward, following prior work (Rabe and Staats, 2021; Liu and Abbeel, 2023). The overlap condition experiments use the derived formula $c \geq F/B$, where $F$ is device peak FLOPS and $B$ is interconnect bandwidth, with block size $c$ and minimal per-device sequence length $s_{\text{min}} = 6c$ (Table 2).

• Cross-validation / statistical protocol. No cross-validation is used—the maximum context length and MFU experiments are deterministic systems measurements (how many tokens fit before OOM, what FLOPs utilization is achieved), not statistical evaluations. The RL experiments report cumulative return on standard ExoRL tasks using the benchmark's evaluation protocol. The line retrieval experiment reports accuracy as a function of context length, comparing against published numbers for GPT-3.5-turbo-16K, Vicuna-16B-16K, and Claude-2-100K. No error bars or confidence intervals are reported for any experiment.

Main Quantitative Results

Maximum Context Length Scaling (Section 5.1, Table 3)

Ring Attention achieves context lengths that scale linearly with device count, consistently exceeding prior state-of-the-art by factors equal to the number of devices used for sequence distribution. The headline result: on TPUv4-1024, a 3B model with Ring Attention trains on sequences of 16,384,000 tokens (16.4M), representing a 512× improvement over the BPT baseline (32K tokens) and enabling context lengths exceeding 16 million tokens. This is not an inference-only result—it is demonstrated in end-to-end training with fully sharded data parallelism.

The scaling pattern is consistent across hardware configurations and model sizes:

On 8× A100 with NVLink (single DGX server), Ring Attention achieves 8× over BPT:

• 3B: 512K vs. 64K (BPT) vs. 32K (memory efficient attention) vs. 4K (vanilla)
• 7B: 256K vs. 32K vs. 16K vs. 2K
• 13B: 128K vs. 16K vs. 4K vs. 2K

On 32× A100 with InfiniBand (distributed across nodes):

• 7B: 4,096K vs. 128K (BPT) — a 32× improvement
• 13B: 2,048K vs. 64K — a 32× improvement

On TPUv3-512 (older generation TPU):

• 7B: 2,048K vs. 8K (BPT) — a 256× improvement
• 13B: 1,024K vs. 8K — a 128× improvement

On TPUv4-1024 (newer generation):

• 3B: 16,384K vs. 32K — a 512× improvement
• 7B: 8,192K vs. 8K — a 512× improvement
• 13B: 4,096K vs. 8K — a 256× improvement
• 30B: 2,048K vs. 4K — a 256× improvement

On TPUv5e-256:

• 3B: 4,096K vs. 32K — a 128× improvement
• 7B: 2,048K vs. 16K — a 128× improvement

The linear scaling with device count is the empirically validated property that justifies the "near-infinite context" framing. The paper explicitly notes: "If a model can be trained with context size $s$ on $n$ GPUs using the blockwise attention and feedforward, with our Ring Attention approach, it becomes possible to train a model with a context size of $n \cdot s$." This is borne out consistently across configurations: 8 devices → 8× improvement on A100 NVLink; 32 devices → 32× on A100 InfiniBand; 512 devices → 512× on TPUv4 for 3B/7B.

The reason the scaling factor is slightly less than the device count for larger models (e.g., 256× on TPUv4-1024 for 13B instead of 512×) is that FSDP must be used to shard the model parameters across some of the devices, reducing the number available for Ring Attention's sequence distribution. This is the expected behavior when model size exceeds single-device memory capacity.

Model FLOPs Utilization Under Extended Context (Section 5.2, Table 5.1)

The paper demonstrates that Ring Attention maintains practical MFU while enabling much longer contexts than baselines. The comparison is structured as: BPT at short context vs. Ring Attention at long context, both running the same underlying blockwise computation implementation on the same hardware. The paper explicitly states: "For fair comparison, both BPT and our approach are based on the same BPT implementation on both GPUs and TPUs."

The results (Table 4 in original paper, reproduced as Table 5.1 in the prior sections summary):

• 7B model, 8× A100: BPT at 32K context, Ring Attention at 256K context. MFU values are not explicitly stated in the table available to us, but the paper reports that Ring Attention "maintains MFU while enabling training with significantly longer context lengths" and that there are "negligible overheads."

• 13B model, 8× A100: BPT at 16K, Ring Attention at 128K.

• 13B model, 32× A100: BPT at 64K, Ring Attention at 2,048K.

• 30B model, TPUv4-1024: BPT at 16K, Ring Attention at 2,048K.

• 65B model, TPUv4-1024: BPT at 8K, Ring Attention at 1,024K.

The paper provides a qualitative explanation for why Ring Attention's MFU should be slightly lower than BPT at short context: "Ring Attention trains much longer context sizes for self-attention, resulting in higher self-attention FLOPs compared to baseline models. Since self-attention has a lower MFU than feedforward, Ring Attention is expected to have a lower MFU than the baseline models." This is an honest accounting—longer sequences shift the FLOPs mix toward attention (which is harder to keep compute-bound) and away from feedforward (which is highly compute-bound). The fact that MFU remains practical despite this shift demonstrates that the communication overhead is indeed negligible.

A crucial missing quantitative detail: the paper does not report the actual MFU percentages in the table, only the context lengths. This makes it impossible to assess how much MFU degradation occurs relative to BPT at the same hardware configuration. The text states that Ring Attention offers "a clear advantage in terms of maintaining MFU," but without specific numbers, the reader cannot determine whether the MFU drop is 1%, 10%, or 30%.

In-Context RL Performance (Section 5.3, Table 5)

Ring Attention enables training on 4× more trajectories (128 vs. 32) in the Agentic Transformer framework, yielding consistent improvements across all six ExoRL tasks. The headline: AT + Ring Attention achieves a total average return of 113.66 across all six tasks, compared to 111.13 for AT + BPT (trained on 128 trajectories where possible), representing a modest but consistent improvement.

The per-task breakdown (Table 5):

The "oom" entries for AT+ME (memory efficient attention only, without blockwise FFN) at 128 trajectories indicate that the model cannot be trained at all—it runs out of memory. This is the key capability story: without BPT and Ring Attention, the experiment at 128 trajectories is impossible. With BPT, some configurations may still OOM (the paper does not specify whether AT+BPT at 128 trajectories fits in all cases, but the fact that AT+BPT numbers are reported for all six tasks suggests it does for this model size and trajectory count). With Ring Attention, training at even larger scale becomes possible.

The improvements from Ring Attention over BPT at the same trajectory count are small (total average 113.66 vs. 111.13, a 2.3% relative improvement). The paper does not explain what mechanism causes this improvement—since both AT+BPT and AT+RA are trained on 128 trajectories with the same exact attention computation, the performance difference could be attributed to training stability, slightly different optimizer dynamics due to the sharding pattern, or statistical noise. No error bars or significance tests are reported.

Long-Context Retrieval Performance (Section 5.4, Figure 3)

Ring Attention enables finetuning LLaMA-13B on 512K-token contexts, producing a model ("Ring Attention-13B-512K") that maintains high retrieval accuracy even at long context lengths where all comparison models degrade or cannot process the input.

The line retrieval test requires the model to locate and reproduce a specific number embedded in a long document, probing its ability to maintain precise information across long-range dependencies. Figure 3 plots retrieval accuracy against context length (in tokens) for Ring Attention-13B-512K alongside GPT-3.5-turbo-16K, Vicuna-16B-16K, and Claude-2-100K.

The key finding: Ring Attention-13B-512K maintains high accuracy at context lengths where other models either fail entirely (cannot process inputs beyond their training context window) or degrade significantly. The paper states: "our model, Ring Attention-13B-512K, stands out as it maintains high accuracy levels even with long contexts. GPT3.5-turbo-16K, Vicuna-16B-16K, and Claude-2-100K demonstrate competitive accuracy within short context lengths. However, they cannot handle extended context lengths."

This is a capability demonstration rather than an ablation—it shows that the training capability enabled by Ring Attention translates to a downstream task where long-context understanding matters. However, several methodological concerns arise:

• The paper does not report a baseline LLaMA-13B finetuned on the same ShareGPT data but with a shorter context length (e.g., 4K or 16K), making it impossible to determine how much of the benefit comes from the finetuning data vs. the extended context training.
• The comparison models (GPT-3.5, Vicuna, Claude) differ in pretraining data, architecture, instruction tuning, and scale—they are not controlled comparisons.
• The finetuning was limited to 512K tokens due to compute budget constraints, which is far below the millions of tokens Ring Attention can theoretically support. This means the experiment demonstrates feasibility but not the full scaling potential.
• The accuracy values on the y-axis of Figure 3 are not numerically reported in the text, making precise comparison difficult.

FLOPs Scaling of Context Size (Appendix D, Figure 5)

The paper analyzes how the training FLOPs per dataset scale as context length increases, finding that the FLOPs increase is substantially sub-quadratic because the number of tokens remains fixed. The formula derived is:

$\frac{\text{FLOPs ratio (long vs. short context)}}{\text{sequence length ratio}} = \frac{6h + s_{\text{new}}}{6h + s_{\text{old}}}$

where $h$ is the hidden dimension, $s_{\text{new}}$ is the extended context length, and $s_{\text{old}}$ is the base context length (4K). This means the per-dataset FLOPs increase by a factor of $(24bsh^2 + 4bs^2h) / (24bsh^2 + 4bs_{\text{old}}^2h) \cdot (s_{\text{old}}/s)$.

Figure 5 shows this ratio for different model sizes. Key numbers inferred from the figure description:

• Scaling up small models to 1M context: approximately 20–40× more FLOPs.
• Scaling up to 10M tokens: substantially higher but still far below the context length ratio (10M/4K = 2,500×).
• Scaling up the 170B model from 4K to 10M: 162.6× higher per-dataset FLOPs, despite context length being 3,072× longer. The ratio $(6h + 10\text{M}) / (6h + 4\text{K})$ with $h = 12288$ for GPT-3 175B gives $(73728 + 10000000) / (73728 + 4096) \approx 129.5$, and FLOPs ratio incorporates the per-token FLOPs change as well.

This analysis matters because it shows that Ring Attention's ability to train on extreme context lengths does not come with a quadratic FLOPs penalty that would make it economically impractical. The cost is significant but manageable, scaling roughly linearly with context length for large models where the $6h$ term dominates the quadratic attention term.

Ablation Studies and Robustness Checks

Hardware interconnect bandwidth vs. block size requirements (Table 2): The paper computes the minimum per-device sequence length needed for communication-computation overlap across five hardware configurations, deriving $c_{\text{min}} = F / B$ and $s_{\text{min}} = 6c_{\text{min}}$. The findings show a sharp divide: NVLink-connected GPUs and TPUs require only ~6K tokens per device (trivially achievable), while InfiniBand-connected GPUs require ~150K tokens per device (achievable but demanding). This ablation validates that the overlap condition is not theoretical—it is parameterized by real hardware numbers and predicts when Ring Attention will operate with zero overhead. The paper does not experimentally verify these thresholds by measuring actual overlap efficiency at different block sizes, which would be a stronger validation.

Model scale vs. maximum context length (Table 3): The scaling factor from Ring Attention is not perfectly equal to device count for larger models. On TPUv4-1024, 3B and 7B models achieve 512× improvement while 13B and 30B achieve 256×. The paper explains this as the effect of FSDP consuming devices for model sharding, but does not provide a breakdown of how many devices are used for FSDP vs. Ring Attention at each model scale. A systematic ablation showing how maximum context scales as FSDP parallelism degree varies would clarify the tradeoff.

Full precision vs. mixed precision: The paper states that "all of our results are obtained using full precision instead of mixed precision" (Section 4). This is a conservative choice that makes the memory constraints more stringent (activations in float32 consume 2× the memory of bfloat16). Ring Attention's $6bch$ memory formula assumes bfloat16—running in float32 would double this to $12bch$. The paper does not report results in mixed precision, which would likely enable even longer contexts or higher MFU. This is a missing ablation that would demonstrate the approach's behavior under more common training configurations (most large-scale LLM training uses mixed precision).

Gradient checkpointing policy: The paper uses nothing_saveable as the checkpointing policy for both attention and FFN (Appendix B.2), meaning no intermediate activations are saved and everything is recomputed during the backward pass. This is the most aggressive checkpointing strategy and maximizes memory savings at the cost of additional compute. An ablation with less aggressive checkpointing (e.g., saving attention outputs but recomputing FFN, or vice versa) would show how the memory-compute tradeoff interacts with Ring Attention's memory distribution.

Inference vs. training overlap condition (Appendix C): The paper briefly analyzes inference requirements, noting that with a batch size of 1 and query token count of 1, the per-token FLOPs are much smaller, making the overlap condition $B/F \geq 2$ (rearranged from $d^2/F \geq 2 \cdot d^2 / B$). For TPUv5e with $F = 196$ TFLOPS and $B = 186$ GB/s, even assuming an "unreasonably high MFU of 40%," the effective $B/F$ ratio is approximately 2.4, meaning Ring Attention can extend inference context by 32× without overhead. This is a minimal analysis—no inference experiments are run to validate the claim.

Effect of block size and chunk size parameters: The implementation exposes query_chunk_size and key_chunk_size parameters that control sub-block granularity within each host's local blockwise computation (Figure 4, lines 10-11). The paper does not sweep these parameters or report their effect on MFU or peak memory. This is a missing ablation that would help practitioners configure Ring Attention for different hardware profiles.

Number of trajectories in RL (Table 5): While not presented as a formal ablation, the progression from 32 to 128 trajectories for the AT method demonstrates the scaling benefit: AT+ME cannot handle 128 trajectories (OOM), AT+BPT can handle 128 (with Ring Attention enabling training that would otherwise be impossible at larger scales), and AT+Ring Attention matches or slightly exceeds AT+BPT at 128. The paper does not explore whether further trajectory scaling (e.g., 256, 512) with Ring Attention yields continuing improvements—a natural extension that would strengthen the capability claim.

Critical Assessment

Claim 1: Ring Attention enables context lengths that scale linearly with device count, achieving up to 512× improvement over prior state-of-the-art. This claim is robustly supported by Table 3, but with important caveats about what "linear scaling" means in practice. The experiments demonstrate linear scaling ($n$ devices → $n \times$ context) when the model is small enough that all devices can be dedicated to sequence distribution (3B and 7B on TPUv4-1024 achieving 512×). For larger models, FSDP consumes some devices for parameter sharding, reducing the Ring Attention parallelism degree and thus the scaling factor (13B and 30B achieving 256×). This is not a failure of linear scaling—it is the expected behavior when model size exceeds single-device memory—but the paper's promotional language ("context length to scale linearly with the number of devices") should be understood as an upper bound achievable when the model fits on a single device.

A genuine weakness: the maximum context length metric is a binary threshold (OOM or not OOM) rather than a continuous efficiency measurement. The paper reports that BPT can handle 32K on TPUv4-1024 for 3B, for example, but does not report what the memory utilization is at that point—is it at 95% of HBM capacity or 50%? This matters because the practical scaling factor depends on how close the baseline is to memory limits. If BPT at 32K uses only 50% of HBM, then the theoretical maximum for BPT might be higher than what's reported, and the "512×" improvement overstates the practical gain.

Claim 2: Communication is fully overlapped with computation, resulting in zero communication overhead. This claim is supported analytically but not experimentally validated. The paper derives the overlap condition $c \geq F/B$ and computes the minimal sequence lengths in Table 2, but does not measure actual communication overlap efficiency (e.g., the fraction of time spent waiting for transfers vs. computing) in any experiment. The MFU results (Table 5.1) provide indirect evidence—if communication overhead were significant, MFU would degrade substantially, and the paper reports it doesn't—but without specific MFU numbers, the reader cannot assess this.

A missing experiment: a sweep of sequence lengths around the theoretical threshold ($c_{\text{min}}$) to show that MFU drops when $c < F/B$ and remains stable when $c \geq F/B$. This would directly validate the arithmetic intensity analysis and provide practical guidance for hardware selection. Without it, the paper's central design principle—that blockwise computation creates the arithmetic intensity needed for overlap—is asserted rather than proven.

Claim 3: Ring Attention outperforms prior state-of-the-art on RL tasks. Supported (Table 5) but the improvement is very small—a total average return of 113.66 vs. 111.13, representing a 2.3% relative improvement. The paper does not report whether this difference is statistically significant or compute confidence intervals. Given that the ExoRL benchmark typically reports variance across seeds as substantial (the AT paper from which these numbers are drawn likely reports standard deviations), the observed difference may fall within the noise range of training stochasticity. The stronger result in the RL experiment is the capability story—AT+ME OOMs at 128 trajectories, while Ring Attention enables training—rather than the performance improvement over BPT.

A missing baseline: AT + Ring Attention at 32 trajectories. If the improvement from Ring Attention comes from the training infrastructure rather than the additional trajectories, then AT+Ring Attention at 32 trajectories should match AT+BPT at 32 trajectories. If it's worse (e.g., due to the communication pattern affecting optimization dynamics), that would indicate a previously unobserved cost to the ring topology.

Claim 4: Ring Attention enables finetuning of LLaMA-13B on 512K-token contexts, improving long-context retrieval. Supported (Figure 3) but with uncontrolled comparisons. The comparison models (GPT-3.5-turbo-16K, Vicuna-16B-16K, Claude-2-100K) differ from Ring Attention-13B-512K in pretraining data, model scale, architecture, and instruction tuning procedure. The paper does not demonstrate that Ring Attention training is responsible for the improved long-context retrieval—it could be that LLaMA-13B finetuned on ShareGPT with a 4K context would perform similarly at short contexts and that the 512K training simply extends this performance to longer contexts. A critical missing baseline: LLaMA-13B finetuned on the same ShareGPT data with a short context (e.g., 16K) and evaluated on the same line retrieval test. Without this, it's impossible to attribute the long-context performance to Ring Attention's training capability rather than to the ShareGPT finetuning or the base model's inherent capabilities.

Additionally, the finetuning was capped at 512K tokens due to compute budget, which is an artificial ceiling far below the demonstrated training capability (16M+ tokens in Table 3). The paper does not explore whether performance continues to improve or plateaus at intermediate context lengths (e.g., 128K, 256K). This is a missed opportunity to demonstrate scaling behavior on a downstream task.

Claim 5: Near-infinite context training is achievable. This is the paper's most ambitious framing, and it is supported in principle but not in practice. "Near-infinite" here means "proportional to device count" rather than literally unlimited—with 1024 TPUv4 devices, you can train on 16M tokens; with 10,000 devices, you could train on 160M tokens; and so on. The claim holds in the asymptotic sense that there is no hard architectural ceiling.

However, practical constraints that the paper does not explore complicate this picture:

• Training FLOPs increase with context length (Appendix D, Figure 5), which means 100M-token training runs would be extraordinarily expensive even with perfect hardware utilization.
• Data availability: there are few, if any, datasets with coherent long-range dependencies at the 100M-token scale. Training on random concatenations of shorter documents would not teach the model meaningful long-range reasoning.
• Optimization at extreme sequence lengths: the paper does not explore whether gradient statistics become unstable, whether loss spikes occur, or whether learning rate schedules need adjustment when the sequence length changes by orders of magnitude. These are known challenges in large-scale training that could manifest when context is scaled to extreme levels.
• The line retrieval evaluation (Figure 3) was conducted at 512K, not millions of tokens. The paper demonstrates training capability at millions of tokens but only evaluates downstream performance at 512K. Whether the model actually learns useful representations from contexts spanning millions of tokens remains an open question.

Overall experimental assessment. The paper's strongest contribution is the systems-level demonstration that Ring Attention eliminates the per-device memory bottleneck—Table 3 is clear, comprehensive, and covers multiple hardware platforms and model scales. The linear scaling with device count is well-established. The weakest link is the downstream evaluation: the RL improvement is marginal (2.3% relative, no significance testing), and the line retrieval comparison uses uncontrolled baselines and lacks the most critical ablation (short-context finetuning on the same data). The paper's value is primarily as a systems contribution that enables experiments others can run, rather than as a demonstration that long-context training yields dramatic performance improvements. The $4\times$ efficiency gain over best-of-N claimed in the earlier Ring Attention summary is, in this context, the $N_h\times$ context length expansion (where $N_h$ is the device count), not a computational cost reduction—the training FLOPs do increase with context length (Figure 5).

6. Limitations and Trade-offs

Wide Hardware Requirements: The Overlap Condition Is Stringent for Low-Bandwidth Interconnects

The paper's claim to zero communication overhead rests on the $c \geq F/B$ condition—the per-device block size must exceed the ratio of peak FLOPs to interconnect bandwidth. Table 2 quantifies this condition across hardware, and the results split into two regimes. For NVLink-connected A100s (300 GB/s) and TPU v4 (268 GB/s), the minimal per-device sequence length is approximately 6K tokens—trivially achievable. For InfiniBand-connected A100s (12.5 GB/s), the requirement jumps to approximately 150K tokens per device. The paper states this explicitly:

"These requirements are easy to meet with parallelism such as data and tensor parallelism and memory efficient blockwise attention and feedforward... For GPUs connected via InfiniBand, which offers lower bandwidth, the requirements are more strict." (Section 3, "Memory Requirement")

The consequence is that Ring Attention's zero-overhead guarantee does not hold uniformly across hardware configurations. On InfiniBand-connected clusters—which are the dominant setup for multi-node GPU training in academic and many industry settings—the overlap condition demands block sizes of ~25K tokens per device, translating to ~150K tokens per device for the full sequence length (since $s = 6c$). At these block sizes, the per-device memory ($6bch$ bytes) becomes substantial: for a 7B model with hidden dimension 4096 in bfloat16, 150K tokens requires approximately $6 \cdot 1 \cdot 150{,}000 \cdot 4096 \cdot 2 \text{ bytes} \approx 7.4$ GB per layer. With 32 layers, the aggregate activation memory alone (before gradient checkpointing) would be ~237 GB, exceeding the 80 GB HBM of an A100. This means that on InfiniBand clusters, the block size must be large enough to hide communication but small enough to fit in HBM, and these two constraints may conflict at the extreme context lengths the paper advertises.

What evidence exists in the paper: Table 2 provides the per-hardware calculations, demonstrating awareness of the issue. Table 3 includes results on 32× A100 with InfiniBand, showing context lengths of 4M tokens for 7B and 2M tokens for 13B—so training is demonstrated in this regime. However, what is not shown is whether communication overlap was actually achieved at these scales or whether there was hidden transfer latency. The report that Ring Attention "enables training sequences that are up to device count times longer than those of prior memory-efficient Transformers, exceeding a context length of 100 million without making approximations to attention" (Section 7) does not specify whether this was on NVLink/TPU or InfiniBand hardware. A missing measurement: the actual fraction of time spent waiting for ppermute transfers vs. computing attention on InfiniBand hardware at various block sizes around the theoretical threshold.

Mitigation status: The paper acknowledges the issue in Section 3 and provides the analytic condition as guidance, but does not experimentally validate whether overlap degrades gracefully or catastrophically when $c < F/B$. No latency overhead measurements are reported for InfiniBand configurations. A practitioner on InfiniBand hardware cannot determine from the paper alone whether Ring Attention at 4M context length is achieving full overlap or silently incurring communication stalls.

Difficulty Estimation Cost Is Excluded from Budget Accounting

The compute-optimal allocation framework—and in a broader sense, the ability to determine whether a given sequence length requires Ring Attention at all or can be handled by a single device—depends on knowing the required context length for each input. In the experiments, context lengths are set statically (e.g., 512K for line retrieval, 128 trajectories for RL, maximum-emperically-determined for Table 3). The paper provides no mechanism for dynamically determining context length requirements per input, and the maximum-context-length experiments in Table 3 treat context length as a fixed training hyperparameter rather than a per-example variable.

The consequence is a practical deployment gap: if a system processes variable-length inputs (which is the norm in production), it must either (a) pad all inputs to the maximum length and use Ring Attention for everything, wasting compute on short inputs, or (b) use a heuristic to decide when to employ Ring Attention vs. single-device processing, incurring an unmeasured decision cost. The paper's $N_h \times$ context length scaling factor assumes all $N_h$ devices are used for sequence distribution, but in a mixed-length setting, the optimal allocation of devices between Ring Attention and other parallelism strategies may vary per batch.

What evidence exists: None. The paper does not study dynamic context allocation, variable-length batching with Ring Attention, or the overhead of switching between distributed and single-device attention modes. All experiments use fixed-length sequences. The architecture itself (Algorithm 1) assumes all hosts participate in the ring for every forward/backward pass—there is no mechanism described for a subset of hosts to drop out for short sequences.

Mitigation status: Not addressed. The paper does not flag this as a limitation or suggest approaches for adaptive context allocation. This is understandable given the paper's focus on enabling extreme-length training, but a practitioner deploying Ring Attention in a system with heterogeneous sequence lengths will need to solve this problem independently, and the paper provides no guidance.

Only Exact Attention Is Considered; Approximation Methods May Offer Better Pareto Frontiers at Extreme Scale

Ring Attention computes exact self-attention without approximations. This is correctly positioned as a strength—no information is lost, and the mathematical guarantees of standard Transformers are preserved. However, the paper does not compare against approximation-based long-context methods (sparse attention, low-rank approximations, linear attention, recurrent memory, or hybrid approaches) on any metric: maximum context length, throughput, or downstream task performance. Section 6 (Related Work) acknowledges the existence of these methods:

"Other works have investigated the approximation of attention mechanisms, yet these efforts have often yielded sub-optimal results or encountered challenges during scaling up. For an in-depth review of these techniques, we recommend referring to the surveys [26, 35]."

The consequence is that a practitioner cannot determine where Ring Attention sits on the accuracy-vs-efficiency Pareto frontier relative to approximation methods. At the extreme context lengths Ring Attention enables (16M+ tokens), the quadratic cost of exact attention—even with perfect memory distribution—becomes computationally massive. Appendix D (Figure 5) quantifies this: scaling a 7B model from 4K to 1M context increases per-dataset FLOPs by roughly 20–40×. The paper's FLOPs analysis (Section 3) for the attention computation is $4dc^2$ per block, and with $N_h$ blocks, the total attention FLOPs per layer are $4ds^2$—unchanged from standard exact attention. Ring Attention solves the memory bottleneck but does not reduce the compute bottleneck.

If a sparse attention method can achieve 90% of exact attention's accuracy with 50% of the FLOPs at million-token scales, it may be a better practical choice for deployment—but the paper provides no data to evaluate this tradeoff. The survey citations (Narang et al., 2021; Tay et al., 2022) reference work that found approximation methods "often yielded sub-optimal results," but these surveys predate recent advances in long-context approximation techniques (e.g., grouped-query attention, sliding window attention with global tokens, and recurrent memory transformers). The paper's experiments do not include any approximation baseline, so the claim that exact attention is necessary, rather than sufficient, for long-context performance is asserted rather than demonstrated.

What evidence exists: None. No approximation method is implemented or compared against. The line retrieval experiment (Figure 3) compares Ring Attention against GPT-3.5-turbo-16K, Vicuna-16B-16K, and Claude-2-100K—all of which likely use some form of approximate attention at their extreme context lengths—but these are uncontrolled comparisons across model families, scales, and training procedures, not a controlled ablation of exact vs. approximate attention.

Mitigation status: The paper does not address this limitation. The design philosophy (exact attention only) is stated as a feature, which is legitimate, but the absence of approximation baselines makes it impossible to assess whether the compute cost of exact attention at million-token scales is warranted by performance gains. Future work comparing Ring Attention against FlashAttention-based sparse patterns or linear attention at matched FLOPs would clarify whether "near-infinite exact attention" is actually better than "near-infinite approximate attention" for downstream tasks.

The Line Retrieval Evaluation Is Methodologically Weak for the Paper's Central Capability Claim

The paper's most directly impactful result for practitioners is the demonstration that Ring Attention enables finetuning LLaMA-13B on 512K-token contexts, producing a model that maintains retrieval accuracy at lengths where comparison models fail (Section 5.4, Figure 3). However, this experiment has three structural weaknesses that substantially weaken the inference that Ring Attention's training capability caused the improved performance.

First, no short-context finetuning baseline. The paper does not finetune LLaMA-13B on the same ShareGPT data with a short context (e.g., 16K or 32K) using standard training. Without this, it is impossible to determine whether the long-context retrieval accuracy comes from (a) the ShareGPT finetuning data itself teaching the model to retrieve information, (b) the extended context length training teaching the model to handle long-range dependencies, or (c) some interaction. If LLaMA-13B finetuned on ShareGPT at 16K achieved similar retrieval accuracy at 16K-token test lengths, then the 512K training would primarily be extending capability rather than creating it—still valuable, but a different claim than the paper implies.

Second, uncontrolled comparison models. GPT-3.5-turbo-16K, Vicuna-16B-16K, and Claude-2-100K differ from Ring Attention-13B-512K in pretraining data, model architecture, parameter count, instruction tuning methodology, and (critically for the Vicuna comparison) the datasets used for finetuning. The paper does not establish any equivalence between these models at short context lengths that would make the long-context comparison meaningful. If Ring Attention-13B-512K outperforms Vicuna-16B-16K at 8K tokens (where both can operate), that would indicate a difference in model quality, not a benefit of long-context training. Conversely, if they perform similarly at short contexts and Ring Attention maintains accuracy at 512K where Vicuna fails, that would support the capability claim—but this pattern is not established.

Third, the 512K ceiling is arbitrary. The paper states:

"While our approach enables training with millions of context tokens, we conducted finetuning on the LLaMA-13B model, limiting the context length to 512K tokens due to constraints on our cloud compute budget." (Section 5.4)

This means the experiment does not demonstrate scaling behavior—it shows a single point at 512K without showing what happens at 128K, 256K, 1M, or 2M tokens. Does performance degrade gracefully as context grows? Does it plateau? Does accuracy at short contexts suffer from the long-context training (a "distraction" effect known to occur in some long-context models)? A sweep across context lengths during training, evaluated at a range of test lengths, would answer these questions. The single-point evaluation at 512K demonstrates capability but not scaling.

What evidence exists: Figure 3 shows accuracy-vs-length curves, but only for the final trained model, not for intermediate training lengths. The comparison models' curves are drawn from their respective public evaluations, not from a controlled experiment. Section 5.4 is the only downstream language evaluation in the paper, making it the primary evidence for "does long-context training actually improve task performance?"—and the methodological weaknesses mean the answer is suggestive rather than definitive.

Mitigation status: The paper acknowledges the compute budget constraint but does not acknowledge the missing baselines (short-context finetuning, intermediate context lengths) as limitations. The conclusion (Section 7) states "Extensive experiments on language modeling and reinforcement learning tasks demonstrate the effectiveness of our approach in allowing millions of tokens context size and improving performance," which overstates the strength of the evidence for language modeling given the single, methodologically limited evaluation.

The Constant-Factor Memory Overhead ($6bch$) May Limit Extreme Scaling on Modest Device Counts

Ring Attention reduces per-device memory from $2bsh$ (BPT) to $6bch$ where $c = s / N_h$. The factor of 6 comes from the double-buffering required for overlap (query, current KV, incoming KV, output, plus buffer space). For large $N_h$, the block size $c$ is small, and $6bch$ is tiny. For small $N_h$, however, the block size approaches the full sequence length, and Ring Attention may actually use more per-device memory than BPT because of the constant factor. Specifically, Ring Attention is more memory-efficient than BPT when:

$6b \cdot \frac{s}{N_h} \cdot h < 2bsh$

which simplifies to $N_h > 3$. On 4 or more devices, Ring Attention is strictly better. On exactly 2 devices, the memory is $6b(s/2)h = 3bsh > 2bsh$, meaning Ring Attention uses 50% more memory than BPT at the same total sequence length. On a single device with a simulated ring (which the paper does not discuss but a practitioner might attempt), Ring Attention would use $6bsh$—triple the memory of BPT.

This matters because the paper's maximum context length experiments (Table 3) implicitly assume enough devices are available to make the distribution worthwhile. On 8× A100, the 3B model achieves 512K with Ring Attention vs. 64K with BPT—an 8× improvement that matches the device count. But the improvement factor equals $N_h / 3$ (not exactly $N_h$), and the $6bch$ formula means there is a crossover point: Ring Attention is worse than BPT on fewer than 3 devices for sequence distribution, and the efficiency gain scales as $N_h/3$, not $N_h$.

What evidence exists: The memory formula is derived in Section 3. Table 3 demonstrates the advantage at $N_h \geq 8$, where the constant factor is negligible relative to the distribution benefit. The paper does not evaluate configurations with 2–4 devices for sequence distribution specifically, so the crossover behavior is not empirically validated.

Mitigation status: The paper does not discuss the constant-factor overhead or the minimum device threshold. For most practical deployments (8+ GPUs is standard for long-context training), this is a minor limitation. However, for a practitioner trying to maximize context length on a small cluster (e.g., 2–4 GPUs), the paper overstates the improvement—they may see less than $N_h \times$ scaling, and on 2 GPUs, they would get better results using BPT without Ring Attention. The paper should have noted that $N_h > 3$ is a minimum requirement for memory benefit, but this caveat is absent.

Training Throughput Degradation from Attention-Heavy FLOPs Mix Is Not Quantified

The paper provides a qualitative observation about MFU (Section 5.2) that deserves more precise treatment:

"Ring Attention trains much longer context sizes for self-attention, resulting in higher self-attention FLOPs compared to baseline models. Since self-attention has a lower MFU than feedforward, Ring Attention is expected to have a lower MFU than the baseline models."

This is an accurate statement about the FLOPs composition, but the paper does not report the actual MFU percentages. Table 5.1 shows context length pairs (BPT at short context vs. Ring Attention at long context) but reports only that Ring Attention "maintains MFU" and has "negligible overheads." Without specific MFU numbers, a practitioner cannot compute the throughput cost of training at extended context lengths.

The consequence is a hidden throughput penalty. Even with perfect communication overlap, the shift toward attention FLOPs—which are harder to keep compute-bound because they involve reductions over the key dimension and less dense matrix multiplication than FFN layers—means that longer contexts reduce hardware utilization. The paper's MFU analysis in Table 5.1 compares BPT at 32K context (where attention is a small fraction of total FLOPs) against Ring Attention at 256K context (where attention is a much larger fraction). The claim of "maintaining MFU" requires that the attention MFU and FFN MFU are similar, which is not generally true—attention operations typically achieve lower utilization than large matrix multiplications, especially at the block granularities used in Ring Attention where the $c \times c$ attention matrix may be small enough that the operation is not fully saturating GPU tensor cores.

What evidence exists: Table 5.1 provides the configurations studied but not the MFU percentages. The paper reports elsewhere (Appendix B.2) that batch size in tokens is 2M on GPU and 4M on TPU, which are standard large-batch training configurations, and that gradient checkpointing with nothing_saveable policy is used. The arithmetic intensity analysis (Section 3) focuses on the per-block FLOPs required for overlap but does not translate this into a predicted MFU. The paper's maximum context length results (Table 3) measure capability (can it run?) rather than efficiency (how fast does it run per token?), so the throughput question remains open.

Mitigation status: The paper does not report MFU numbers, does not sweep block sizes to show throughput scaling, and does not compare training tokens-per-second between BPT at short context and Ring Attention at long context at matched total FLOPs. The qualitative statement about attention having lower MFU acknowledges the direction of the effect but provides no magnitude. A practitioner deciding between (a) training a standard model at 32K context with high throughput or (b) training with Ring Attention at 512K context with unknown throughput degradation needs this information to make an informed tradeoff. The paper does not provide it.

7. Implications and Future Directions

How This Work Changes the Landscape

Ring Attention represents a systems-level architectural shift rather than a conceptual or algorithmic one—it does not change what Transformers compute, only where and when the computation and memory reside. Yet the practical consequences of this shift are profound enough to reframe what the field considers possible with existing hardware. Before this work, the dominant assumption was that exact-attention training at million-token scales was a hardware problem: we would need next-generation accelerators with substantially more HBM before such training became feasible. Ring Attention falsifies that assumption by showing that the memory bottleneck is not a hardware limit but a distribution strategy limit—the aggregate HBM across a cluster of existing devices is sufficient, and the only missing piece was a protocol that could exploit that aggregate memory without incurring communication overhead.

This is a reframing of the long-context problem from hardware-constrained to software-constrained. The paper demonstrates this reframing through Table 3, where the scaling factor from Ring Attention matches the device count (512× on TPUv4-1024 for 3B and 7B models, 256× for larger models where FSDP consumes some devices). The message is unambiguous: if you can add more devices, you can train on proportionally longer sequences. The ceiling is no longer the 80–100 GB HBM of a single accelerator but the total HBM of your cluster divided by a constant factor of 6. For a 1024-TPUv4 pod, that ceiling sits at roughly 16 million tokens—and scales linearly with pod size.

This reframing has a cascade of consequences for how researchers and practitioners think about long-context models:

It makes long-context training an engineering allocation decision rather than a research aspiration. Before Ring Attention, training a model on 1M+ tokens was a research project requiring novel architectures, approximation schemes, or custom hardware. After Ring Attention, it is a configuration choice: how many devices do you allocate to sequence distribution vs. other parallelism strategies? The paper's Appendix A provides the concrete formula: on 512 A100 GPUs with a 30B model, shard the model across 8 devices via FSDP and use the remaining 64 for Ring Attention, yielding 64× context extension. This is actionable engineering guidance, not aspirational research direction.

It resolves the tension between exact attention's expressiveness and its memory cost. Prior work oscillated between two poles: accept exact attention's memory cost and live with short contexts, or adopt approximations to reach longer contexts and accept potential quality degradation. Ring Attention dissolves this tension by showing that exact attention and long contexts are compatible—the memory cost can be distributed, and the communication can be hidden. This does not mean approximations are obsolete (the compute cost of exact attention still scales quadratically, as Appendix D documents), but it means that memory is no longer the binding constraint. The decision between exact and approximate attention can now be made on compute-efficiency grounds alone, which is a cleaner and more tractable tradeoff.

It shifts attention (no pun intended) from single-device memory optimization to cluster-level scheduling. The past five years of memory-efficient Transformer research—from the tiling technique (Milakov and Gimelshein, 2018) through memory-efficient attention (Rabe and Staats, 2021), FlashAttention (Dao et al., 2022), and Blockwise Parallel Transformers (Liu and Abbeel, 2023)—all operated within the constraint of a single device's HBM. Each improvement reduced the constant factors ($O(n^2)$ to $O(n)$ working memory) but could not escape the $2bsh$ layer-output storage requirement. Ring Attention breaks out of the single-device frame entirely and recasts the problem as one of orchestrating computation and communication across a distributed system. This is likely where future progress will concentrate: not on further reducing per-block memory (the $6bch$ factor could be optimized, but gains are linear and bounded) but on improving the communication scheduling, fault tolerance, and heterogeneity support of the ring protocol.

It makes certain research directions more attractive and others less so. More attractive: training models on entire codebases, full-length books, long videos, extensive RL trajectories, and scientific datasets (gene sequences, protein structures, experimental logs) where long-range dependencies are genuine and irreducible. The paper's RL experiment (Table 5) and line retrieval evaluation (Figure 3) are first steps in this direction, but they only scratch the surface—the training capability exists, and now the question is what tasks benefit from it. Less attractive: single-device memory optimization of Transformers for long contexts. If the memory can be distributed linearly with device count, further squeezing the per-block constant factor from $6bch$ to $4bch$ or $3bch$ yields diminishing returns compared to simply adding more devices. The research frontier shifts from "how do I fit more tokens on one GPU?" to "how do I efficiently coordinate many GPUs to process a single very long sequence?"

It introduces a new diagnostic tool: the overlap condition $c \geq F/B$. This inequality—block size must exceed the ratio of device FLOPs to interconnect bandwidth—is a crisp, falsifiable condition that determines whether Ring Attention achieves zero-overhead communication or silently incurs transfer latency. Table 2 operationalizes this diagnostic across five hardware configurations, and the results create a clear taxonomy: NVLink and TPU interconnects are in the "trivially overlapped" regime (~6K tokens per device needed); InfiniBand is in the "achievable but demanding" regime (~150K tokens per device needed). This diagnostic did not exist before because prior sequence parallelism methods could not achieve overlap regardless of block size—their per-block computation was memory-bound, so arithmetic intensity was too low even at large block sizes. Ring Attention's use of BPT to make attention compute-bound at the block level is what makes the diagnostic meaningful. Future hardware-software co-design can use this condition to determine bandwidth requirements for long-context accelerators.

The magnitude of the contribution is substantial but specific. Ring Attention is not a paradigm shift on the order of the original Transformer or the discovery of scaling laws—it does not change the model architecture or the learning dynamics. It is a systems innovation in the lineage of FlashAttention: it makes an existing computation feasible at scales that were previously impossible, and by doing so, it expands the space of experiments that researchers can run. The paper's 512× improvement over prior state-of-the-art (Table 3, TPUv4-1024, 3B model) is the kind of order-of-magnitude leap that qualifies as a breakthrough in systems research. But the downstream impact depends entirely on whether long-context training actually improves model capabilities in ways that matter—and the paper's own downstream evaluations (modest RL improvement, uncontrolled line retrieval comparison) leave this question largely open. The systems contribution is solid; the capability contribution remains to be demonstrated by follow-up work.

Follow-Up Research This Work Enables

Characterizing the scaling laws of model quality as a function of training context length. The paper demonstrates that training on 512K-token contexts is possible (Section 5.4), but does not explore how downstream performance scales as context length increases from 4K to 512K to 16M tokens. A natural follow-up would train a single model architecture (e.g., LLaMA-7B) on the same dataset at context lengths of 4K, 16K, 64K, 256K, 1M, and 4M tokens (using Ring Attention for the longer settings while keeping total FLOPs matched via token count), then evaluate on a suite of long-range dependency tasks: line retrieval, passkey retrieval, long-document QA, code completion across file boundaries, and multi-turn dialogue coherence. The key measurement is whether there exist tasks where performance improves significantly beyond 64K–128K contexts, or whether gains saturate, which would indicate that current datasets lack dependencies long enough to exploit Ring Attention's capability. This would directly test the paper's implicit claim that near-infinite context training enables near-infinite context utilization.

Measuring the throughput cost of communication overlap across the $c = F/B$ threshold. The paper derives the overlap condition analytically but never experimentally validates it. A critical follow-up would run Ring Attention at a fixed total sequence length while varying block size (by changing the number of devices dedicated to sequence distribution) and measuring both MFU and the fraction of time the ppermute operations are actually overlapped with computation (using GPU/TPU profiling tools like NSight or the TPU profiler). The sweep should cross the theoretical threshold from both directions: block sizes well below $F/B$ (where overlap should fail), at the threshold, and well above (where overlap should be complete). This experiment would validate or refine the analytic model from Section 3, and would provide practitioners with a measured rather than theoretical block size recommendation. On InfiniBand hardware, this experiment is particularly important because the paper's Table 2 shows $c_{\text{min}} \approx 25$K tokens for this configuration—a block size large enough that per-device memory ($6bch$) becomes substantial, and the tradeoff between overlap efficiency and total context length needs empirical characterization.

Combining Ring Attention with sparse or linear attention for a compute-efficient Pareto frontier at extreme scales. Ring Attention solves the memory bottleneck but not the compute bottleneck—attention FLOPs still scale as $O(s^2 \cdot d)$ per layer. At the 16M-token context lengths that Table 3 demonstrates, the per-layer attention FLOPs are approximately $(16 \times 10^6)^2 \cdot d$ operations, which for $d=4096$ is roughly $10^{18}$ FLOPs per attention layer—a staggering number even for large clusters. Follow-up work should implement a family of exact-attention and approximate-attention variants within Ring Attention's communication framework, sweeping along the compute-accuracy tradeoff: exact attention (current paper), sliding window attention with global tokens (Beltagy et al., 2020), sparse attention with pre-specified patterns (Child et al., 2019), and linear attention (Katharopoulos et al., 2020), all using the same ring communication to distribute the sequence dimension. The measurement is downstream task accuracy on long-range benchmarks against total training wall-clock time and total FLOPs, producing a Pareto frontier. This would directly address the paper's unstated assumption that exact attention is worth its quadratic cost at million-token scales, and would guide practitioners toward the right method for their compute budget. A negative result—finding that a simple sparse pattern matches exact attention's accuracy at 10% of the FLOPs on most tasks—would be highly informative and would redirect follow-up work from exact-attention distribution to approximate-attention optimization.

Training a model on entire code repositories with file-level retrieval evaluation. The paper identifies codebase understanding as a motivating use case (Section 1) but runs no code-related experiments. A strong follow-up would use Ring Attention to train a model on full code repositories—concatenating all source files, documentation, and commit history into a single sequence—and evaluate on code tasks that require cross-file reasoning. Concretely: take the CodeSearchNet or a large monorepo dataset, train a CodeLLaMA-style model at context lengths of 128K, 512K, and 2M tokens (all enabled by Ring Attention but not by prior methods), and evaluate on tasks like: given a function call in file A, retrieve the function definition in file B; given a bug report referencing multiple files, locate the relevant code sections; or given a partial refactoring, predict the changes needed in dependent files. The key question is whether training at 2M-token context improves cross-file reasoning over training at 128K context, which would demonstrate that the model actually learns to use the additional context rather than simply ignoring distant tokens. The paper's line retrieval experiment (Figure 3) is a minimal version of this evaluation, but code repositories present a more realistic and economically valuable testbed where long-range dependencies are both genuine and structured (imports, function calls, class inheritance).

Extending the ring protocol to fault-tolerant and heterogeneous-device configurations. The paper's ring protocol (Algorithm 1) assumes a homogeneous, reliable set of devices in a fixed ring topology. Real-world clusters exhibit device heterogeneity (different GPU generations, varying interconnect speeds, stragglers) and failures (device crashes, network partitions). A systems follow-up would design a fault-tolerant variant of Ring Attention where the ring can dynamically reconfigure when a device fails—e.g., by having the ring "skip" the failed device and redistribute its sequence block to neighbors, or by maintaining redundant copies of key-value blocks so that computation can proceed while a replacement device spins up. The evaluation would measure recovery time and accuracy impact under simulated device failures. This is enabled by Ring Attention's permutation invariance property (Section 3): since the order of key-value blocks doesn't matter, a reconfigured ring that delivers all surviving blocks in any order still produces correct attention outputs. The paper does not exploit this property for fault tolerance, but the property makes fault tolerance structurally easier than for methods with sequential dependencies. This direction has immediate practical relevance for long-running training jobs on spot-instance cloud resources, where device preemption is common.

Applying Ring Attention to multimodal training with extremely long visual sequences. The paper mentions "large video-audio-language models" as a motivating use case (Section 7) but trains only on text. A compelling follow-up would apply Ring Attention to video understanding: tokenize video frames (e.g., ViT patches from each frame) and train a Transformer on full-hour videos where the total token count reaches millions. For instance, a 1-hour video at 1 fps with 196 patches per frame yields roughly 700K visual tokens—within the demonstrated range of Ring Attention on modest hardware (8× A100 handles 512K for 3B in Table 3). The evaluation would compare a model trained with Ring Attention on full-length videos against a model trained on truncated 30-second clips (at matched total FLOPs via gradient accumulation), measuring performance on long-range video tasks: temporal action segmentation, video question answering requiring multi-minute context, and episodic memory tasks where an event early in the video must be recalled to interpret a later event. This would test whether the training capability enabled by Ring Attention translates to improved temporal reasoning, and would extend the paper's contributions beyond language to the multimodal domain that the introduction prominently highlights but the experiments do not explore.

Practical Applications and Downstream Use Cases

Training code models on entire repositories for cross-file reasoning. Enterprise codebases routinely span millions of lines across thousands of files, with dependencies encoded in import statements, function calls, and class hierarchies. Current code models (Codex, CodeLLaMA, StarCoder) are typically trained on individual files or short concatenations of files, truncating at 4K–16K tokens. This means they cannot learn patterns that span file boundaries—a critical limitation for tasks like whole-codebase refactoring, security vulnerability detection across modules, or automated PR generation. Ring Attention enables training on concatenations of entire repositories at 1M+ token contexts. On 32× A100 GPUs (a common cluster size in industry), Table 3 shows 4M-token contexts for a 7B model and 2M for a 13B model. A 1M-token repo would fit comfortably. The paper's line retrieval curve (Figure 3) provides preliminary evidence that models trained at 512K context can retrieve information across long distances, but the code application is a direct extrapolation to a higher-value domain. The benefit is not just incremental accuracy but enabling entirely new classes of code understanding tasks that are currently impossible because models cannot see enough context.

Scientific literature processing: training on full papers with citation graphs. Scientific papers cite prior work across the entire document, and understanding a paper's contribution often requires retrieving specific findings from cited works. Current LLM-based literature tools truncate papers or embed them piecewise, losing cross-document citation structure. Ring Attention enables training a model on full papers (including all cited papers concatenated into the context) at the 100K–1M token scale. For instance, a typical machine learning paper is ~10K tokens, and its 30–50 citations might add 300K–500K tokens—within the range of Ring Attention on 8× A100 (512K for 3B, 256K for 7B from Table 3). The model could be trained to answer questions that require synthesizing information from a primary paper and its citations, such as "What improvement did Method X claim over its cited baselines, and did the baselines' own papers report different numbers?" This is currently infeasible with truncated-context models. The paper's line retrieval experiment (Figure 3) is a toy version of this capability—the ability to retrieve a specific number from a long document is a prerequisite for answering questions that require synthesizing numbers across multiple documents.

Reinforcement learning from extensive agent histories. The paper's RL experiment (Table 5) uses 128 trajectories of 4K tokens each—a total context of 512K tokens—and shows a small but consistent improvement over 32 trajectories (111.13 → 113.66 total average return). A natural deployment would scale this to thousands of trajectories on problems where the agent's learning can genuinely benefit from distinguishing between subtle variations in long histories. For example, training a household robot agent on months of interaction data (tens of thousands of episodes) where similar situations have different optimal actions depending on events that occurred hours or days earlier. With Ring Attention on TPUv4-1024, Table 3 shows 30B models can train on 2M-token contexts. At 4K tokens per trajectory, that is 500 trajectories in a single sequence—enough to include substantial environmental diversity. The improvement from 128 trajectories in the paper is modest (2.3%), but the scaling trend might become stronger at 500+ trajectories if the long-range dependencies are genuine. This application is risky (the paper's evidence for RL improvement is weak) but high-reward if follow-up work finds tasks where long-horizon context matters more than it does on ExoRL.

When to Prefer This Method

The paper explicitly positions Ring Attention against alternative parallelism strategies (data parallelism, tensor parallelism, pipeline parallelism, and prior sequence parallelism) rather than against different attention computation methods. The decision criterion is clear and measurable:

• Prefer Ring Attention when the sequence length required for training exceeds what fits in a single device's HBM even after applying memory-efficient attention and blockwise feedforward (BPT). This threshold is approximately $s > \text{HBM} / (2bh)$ for BPT, where HBM is device memory, $b$ is batch size, and $h$ is hidden size. For a 7B model in bfloat16 with batch size 1, this is roughly 80 GB / (2 × 1 × 4096 × 2 bytes) ≈ 5M tokens on an A100—but in practice, with optimizer states, parameter storage, and gradient checkpointing overhead, the practical threshold is much lower (Table 3 shows BPT maxing out at 32K on 8× A100 for 7B models, because data parallelism also consumes memory). The exact threshold depends on the full training configuration, but Ring Attention is valuable whenever BPT hits OOM.

• Prefer Ring Attention over prior sequence parallelism (Li et al., 2023) when you have enough devices and high-enough interconnect bandwidth to satisfy $c \geq F/B$. Table 2 provides the threshold: ~6K tokens per device on NVLink/TPU, ~150K tokens per device on InfiniBand. If your configuration meets this, Ring Attention achieves zero communication overhead; prior sequence parallelism always incurs non-overlapped communication. On InfiniBand clusters with per-device block sizes below 150K tokens, prior sequence parallelism may actually outperform Ring Attention because Ring Attention's overlap fails and the constant-factor memory overhead ($6bch$ vs. sequence parallelism's typical memory) becomes a liability. This is testable but not measured in the paper.

• Do not use Ring Attention for sequence distribution on fewer than 4 devices. The constant-factor memory overhead means Ring Attention uses more memory than BPT when $N_h \leq 3$. On 2 or 3 devices, use BPT with data parallelism or tensor parallelism instead. The paper does not state this explicitly, but it follows from the $6bch$ vs. $2bsh$ comparison: Ring Attention is better when $N_h > 3$, equal at $N_h = 3$, and worse at $N_h = 2$ (50% more memory) or $N_h = 1$ (300% more memory, though a single-device ring is a degenerate case).

• Combine Ring Attention with FSDP when the model size exceeds single-device memory. The paper's Appendix A provides the recipe: partition devices into FSDP groups and Ring Attention groups. For a 30B model on 512 GPUs, FSDP across 8 devices leaves 64-way Ring Attention for sequence distribution. This composability is a practical strength—there is no conflict between the strategies, and the total context length scales as (total devices / FSDP degree).