Fourier Position Embedding: Enhancing Attention's Periodic Extension for Length Generalization

ArXiv: 2412.17739

🎯 Pitch

RoPE-based models catastrophically fail beyond their training length because non-attention layers corrupt the very periodic structure that enables extrapolation—a problem no previous method had identified. This work shows that by modelling each hidden dimension as a multi-frequency Fourier series and zeroing out undertrained low-frequency components, a model trained on 512 tokens can maintain near-perfect accuracy even at 8000+ tokens.

1. Executive Summary

This paper analyzes how Rotary Position Embedding (RoPE) fails to achieve length generalization in Language Models and introduces Fourier Position Embedding (FoPE) to address these failures. Using Discrete Signal Processing theory applied to OLMo models (60M–1.2B parameters) and SmolLM-1.7B evaluated on C4 perplexity, Passkey Retrieval, and long-context summarization tasks, the work identifies Spectrum Damage—the corruption of attention's periodic extension caused by linear layers, activation functions (producing Spectrum Leakage and Spectrum Distortion), and insufficiently trained low-frequency components brought by time-domain truncation—as the mechanism undermining RoPE's implicit Non-Uniform Discrete Fourier Transform. FoPE modifies attention's frequency-domain properties by treating each dimension as a multi-frequency Fourier Series and zeroing out undertrained frequency components below the floor frequency, maintaining stable Passkey Retrieval accuracy at arbitrary sequence lengths where RoPE drops to zero beyond 2× the training length, while also serving as an effective extrapolation method—outperforming YARN when applied to pre-trained RoPE models, establishing that robust length generalization requires protecting attention's frequency-domain structure from distortion introduced by non-attention model components.

2. Context and Motivation

The Core Problem: LMs Cannot Generalize Beyond Their Training Sequence Length

The fundamental problem this paper addresses is deceptively simple: when you train a language model on sequences of length 512, why does its performance collapse catastrophically when you ask it to process sequences of length 2048 or 8192? This is not just a matter of slow degradation—as Figure 1(a) demonstrates, a standard RoPE-based model achieves near-perfect accuracy on Passkey Retrieval within its training length and then drops to essentially zero the moment sequence length doubles. The model doesn't just get worse; it becomes completely incapable of the task.

This is the length generalization failure, and it matters enormously for several reasons the paper makes clear (Section 1, Figure 1):

• Real-world deployment demands flexibility. Production LLMs encounter documents, conversations, and codebases that span wildly different lengths. A customer support chatbot trained on 2,048-token exchanges will inevitably face users who paste lengthy email threads or product manuals. If the model's useful behavior abruptly terminates at the training boundary, the system is unreliable in practice.

• Training on long sequences is prohibitively expensive. The quadratic complexity of self-attention with respect to sequence length means that doubling the context window approximately quadruples the computational cost. Training a model from scratch on 32K-token sequences is economically infeasible for most organizations. If length generalization can be achieved—meaning a model trained on 512-token sequences performs competently at 8K tokens—the cost savings are transformative.

• The gap between training and deployment is growing. As models are pushed toward "long-context" applications (document summarization, repository-level code understanding, multi-turn dialogue with memory), the discrepancy between economically viable training lengths and desired inference lengths continues to widen. Bridging this gap without expensive retraining has become an urgent practical need.

• It reveals a fundamental misunderstanding of how position information propagates through transformer architectures. The fact that RoPE—designed specifically to be periodic and therefore extrapolatable—fails so dramatically indicates that our mental models of how position embeddings interact with the rest of the network are incomplete. Understanding why periodicity breaks down is a prerequisite to fixing it.

RoPE: The Theoretical Promise vs. the Empirical Reality

To understand the gap this paper fills, we need to appreciate why RoPE's failure is so puzzling—and therefore theoretically significant.

The elegant design of RoPE. Rotary Position Embedding (Su et al., 2024) encodes position information by rotating query and key vectors in the complex plane. For token at position $n_a$, the query vector dimension $m$ becomes $Q_m e^{i\omega_m n_a}$, and similarly for the key at position $n_b$. When the attention score is computed as the inner product, the position-dependent phase factors combine to produce:

$\tilde{h}_m(n) = \tilde{q}_m(n_a) \tilde{k}_m^*(n_b) = H_m e^{i\omega_m n}$

where $n = n_a - n_b$ is the relative distance between tokens and $H_m = Q_m K_m$. The total attention weight is then:

$h(n) = \sum_{m=0}^{M-1} H_m e^{i\omega_m n}$

This is mathematically identical to an Inverse Non-Uniform Discrete Fourier Transform (Section 2.2, Equation 4). The critical implication: because complex exponentials $e^{i\omega_m n}$ are inherently periodic with period $N_{\omega_m} = 2\pi / \omega_m$, the attention score in each dimension should satisfy:

$\tilde{h}_m(n + N_{\omega_m}) = \tilde{h}_m(n)$

In plain language: RoPE should produce attention patterns that naturally repeat at different wavelength scales, enabling the model to handle positional relationships it never saw during training. This is what the authors call "periodic extension"—the property that should make length generalization automatic.

The brutal empirical reality. Yet, as Figure 1(a) shows, this periodic extension manifestly does NOT occur in practice. RoPE-based models fail catastrophically beyond their training length. Why?

Prior Approaches and Their Limitations

The paper situates itself against a landscape of existing solutions, each of which addresses only a piece of the problem:

Absolute position embeddings (Vaswani et al., 2017) learn a unique embedding vector for each position up to the maximum training length. These fail at length generalization by construction: positions beyond the training window have no learned embedding. The model encounters completely novel position representations and typically produces garbage. This limitation was the original motivation for developing relative position embeddings—the entire family of methods to which RoPE belongs.

ALiBi (Press et al., 2021) adds a position-biased attention mask that linearly decays attention weights based on token distance. While ALiBi achieves stable perplexity at arbitrary lengths during pre-training (visible in Figure 1(b) where it maintains low perplexity across all sequence lengths), it suffers from a critical weakness: it cannot effectively retrieve information from distant tokens. The linear decay means that tokens far apart are assigned negligible attention weights regardless of their semantic relevance. Figure 1(a) reveals the consequence: ALiBi's Passkey Retrieval accuracy declines linearly with sequence length, because it is structurally incapable of attending to the passkey when it's positioned far from the query. ALiBi solves extrapolation at the cost of long-range attention itself.

This trade-off highlights a subtle but crucial point: stable perplexity is not the same as effective length generalization. A model can predict next-tokens well by relying only on local context (which ALiBi encourages) while completely ignoring information that requires long-range attention. The Passkey Retrieval task—where the model must find a single five-digit number buried in meaningless filler text—specifically tests whether distant information is accessible, and ALiBi fails it.

KERPLE (Chi et al., 2022) and FIRE (Li et al., 2024) represent additional position embedding variants that attempt to improve upon RoPE through different functional forms. Figure 1 shows these baselines, but their performance on length generalization tasks remains substantially below what this paper achieves with FoPE. The paper includes them primarily as comparison points rather than as central objects of critique.

RoPE with extension methods represents the current dominant approach. Rather than redesigning position embeddings from scratch, methods like YARN (Peng et al., 2023) apply post-hoc modifications to extend a pre-trained RoPE model's context window. YARN works by interpolating the position indices for longer sequences—essentially compressing the extrapolated positions back into the range the model was trained on—combined with targeted fine-tuning. Other approaches (Chen et al., 2024a, CLEX; Jin et al., 2024, Self-Extend; Xiong et al., 2024) similarly modify how RoPE's rotation frequencies are computed or applied at inference time.

These methods have proven effective enough that RoPE-plus-extension has become the standard recipe for long-context LLMs. However, the paper identifies a critical blind spot: all of these methods operate entirely within the attention mechanism, modifying only how RoPE's rotation is applied. They treat RoPE's periodic extension as fundamentally sound and attempt to "unlock" it through better interpolation strategies.

The paper's central theoretical contribution is to show that this assumption is wrong. The periodic extension is not merely "locked" — it is actively damaged by components of the transformer that existing methods ignore.

Where Existing Approaches Fall Short: The Blind Spot

The paper identifies three specific failure modes that prior work missed, and each one originates outside the attention mechanism:

Failure mode 1: Linear layers cause Spectrum Leakage (Section 3.2). The Fourier transform interpretation of RoPE assumes that each dimension of the hidden state carries information at a single, clean frequency $\omega_m$. However, between one attention layer and the next, the hidden states pass through linear projections:

$Y_m = \sum_{k=0}^{M-1} W_{km} X_k$

This is a linear combination of different frequency components. After this transformation, dimension $m$ of $Y$ no longer contains a pure sinusoid at frequency $\omega_m$ — it contains a mixture of information from multiple frequencies $\omega_0, \omega_1, ..., \omega_{M-1}$. When the next attention layer applies RoPE, it treats this dimension $m$ as if it were a pure frequency $\omega_m$, but the actual signal is a composite. This mismatch between the assumed and actual frequency content means the periodic extension property (Equation 5) no longer holds: $h'_m(n + N_{\omega_m}) \neq h'_m(n)$ because $N_{\omega_m}$ is not the period of the leaked frequency components $\omega_o$.

The authors formalize this with a damage model (Section 3.1): if a dimension contains both the intended frequency $\omega_m$ with coefficient $H_{\omega_m}$ and a leaked frequency $\omega_o$ with coefficient $H_{\omega_o} = \sigma H$, the combined signal is $H_{\omega_m}[(1-\sigma)e^{i\omega_m n} + \sigma e^{i\omega_o n}]$, which is not periodic with period $N_{\omega_m}$. The consequence is that information propagates through the network at the wrong wavelength, and the model's ability to recognize patterns at specific distances degrades.

Failure mode 2: Activation functions cause Spectrum Distortion (Section 3.2, Lemma 3.1). This is perhaps the most subtle insight in the paper. Non-linear activation functions (GELU, SiLU, ReLU—used in the feed-forward layers between attention blocks) generate harmonic frequencies from their inputs. Lemma 3.1 provides the formal statement: for a double-frequency input $x(n) = \cos \omega_1 n + \cos \omega_2 n$, any non-linear function $g$ produces:

$g(x(n)) = \sum_{j \in \mathbb{N}} \sum_{k \in \mathbb{N}} a_{j,k} \cos(j\omega_1 + k\omega_2)n$

In other words, passing a signal through ReLU creates new frequency components at sums and differences of the original frequencies—harmonics and intermodulation products. After the signal has passed through multiple feed-forward layers (with their activation functions) between attention blocks, the frequency content of each dimension has been irreversibly scrambled. Again, when the next attention layer applies RoPE at a single frequency $\omega_m$, it's applying a pure rotation to a spectrally complex signal.

The combined effect of Spectrum Leakage (from linear layers) and Spectrum Distortion (from activations) is what the authors call Spectrum Damage — the progressive corruption of the one-to-one correspondence between dimensions and frequencies that RoPE's periodic extension depends on. Figure 2(a) provides a visual overview of this process.

Failure mode 3: Undertrained low-frequency components (Section 3.3). This failure mode reveals an issue that is subtle and initially counterintuitive. Low-frequency components ($\omega_m < 2\pi/N$, where $N$ is the training sequence length) correspond to sinusoids with periods longer than the training window. During training, these components never complete a full cycle — the model sees only a partial segment of a very long wave.

The DSP analysis explains why this matters. A sinusoid $e^{i\omega_m n}$ observed only for $n \leq N$ is equivalent to multiplying it by a rectangular window $\text{rect}(n)$ that truncates at $N$. In the frequency domain (Equation 10), this truncation introduces a $\text{sinc}$-like distortion term:

$X(\omega) = \alpha \delta(\omega_m) + \frac{\sin[(N - \alpha N_m)(\omega - \omega_m)]}{\omega - \omega_m}$

where $\alpha = \lfloor N / N_m \rfloor$ is the number of complete cycles observed. For high-frequency components, $\alpha \gg 1$, the main frequency component dominates, and the distortion is negligible. For low-frequency components where $\alpha = 0$ (meaning less than one complete cycle was observed), the sinc-distortion term dominates the signal, and the actual frequency content is poorly characterized. As the authors put it:

"When the period of the primary frequency component exceeds the truncation length, its amplitude is significantly weakened. Consequently, noisy components dominate these dimensions, impairing the periodic extension."

Intuitively, the model has never seen what these slowly-varying sinusoids look like beyond position $N$. When asked to extrapolate to position $N+1000$, the model has no basis for predicting the rotation—it's operating out-of-distribution in a regime where the training signal was dominated by truncation artifacts rather than the actual sinusoid.

This is not a new observation per se—prior work (Peng et al., 2023) recognized that low-frequency RoPE dimensions pose extrapolation challenges—but the paper is the first to provide a formal frequency-domain model of why this occurs (the rectangular window analysis) and to ground the solution in DSP theory rather than heuristic interpolation.

The Cumulative Result: A Hidden, Pervasive Problem

What makes these failure modes collectively insidious is that none of them are visible by examining the attention mechanism in isolation. If you look at RoPE's mathematical definition, it's perfectly periodic. If you trace through the attention score computation, the frequency components appear clean and well-behaved. The damage occurs in the parts of the transformer that surround attention—the linear projections between layers, the feed-forward networks with their non-linear activations, and the cumulative effect of many such transformations over dozens of model layers.

This explains why prior work, which focused almost exclusively on modifying RoPE's rotation scheme within the attention computation, achieved only partial success. Methods like YARN can adjust the mapping from positions to rotation angles, but they cannot repair the fact that by the time a hidden state reaches a deep attention layer, its frequency content has been corrupted by everything that happened in the layers below. The periodic extension property was broken before the attention layer even received its input. Fixing only how attention applies RoPE is treating a symptom far downstream from the root cause.

How This Paper Positions Itself

The paper's contribution is not to propose a new, completely different approach to position embedding, but rather to provide the first system-level frequency-domain analysis that traces why RoPE's theoretical periodic extension fails in practice, and then to modify RoPE's attention-level processing to be robust against the damage introduced by the rest of the network.

This is an important framing distinction. The paper is NOT saying "RoPE is a bad idea" or "we need to replace position embeddings entirely." It is saying: RoPE's periodic extension is a valid and powerful idea, but it lives in a hostile environment. The transformer's other components continuously damage the frequency-domain structure that RoPE assumes. The solution is not to eliminate those components (they're essential for model capacity) but to modify how attention processes information so that the damage doesn't destroy the periodic extension.

The two modifications that constitute FoPE flow directly from the DSP analysis:

1. Multi-frequency modeling per dimension (Fourier Series) addresses Spectrum Leakage and Distortion by acknowledging that each dimension already contains information at multiple frequencies. Rather than treating dimension $m$ as a pure frequency $\omega_m$, FoPE models it as $H_m(n)(e^{i\omega_m n} + \sum_{\omega} a_{\omega} e^{i\omega n})$, where the additional harmonic components are learned but initialized to be small ($a_{\omega} < 1$, reflecting that $\omega_m$ is the dominant frequency). This means FoPE doesn't try to "clean up" the signal—it accepts that the damage exists and adapts the position embedding to match the actual spectral content.

2. Zeroing out undertrained frequencies addresses the truncation problem by replacing frequencies $\omega_m < 2\pi/N$ with zero-frequency (DC) components. The choice of zero is theoretically motivated: zero-frequency corresponds to infinite wavelength—it is simultaneously the shortest and longest possible period. It introduces no positional bias, and as the ablation in Figure 6(a) shows (where "FoPE w/o CF" underperforms), removing the undertrained frequencies is critical for length generalization.

This dual approach—accepting damage where it occurs and excising the components that can't be made robust—is what distinguishes FoPE from prior work that tried to preserve RoPE's purity by working around it. The paper's framing is that length generalization is a system-level property, not an attention-level property, and solutions must address how position information is processed by the entire architecture.

3. Technical Approach

3.1 Reader Orientation

This is a frequency-domain analysis and architecture modification paper whose core idea is that Rotary Position Embedding (RoPE) achieves length generalization through a periodic extension property that is mathematically valid but gets systematically corrupted by nearly every other component of a transformer—linear projections, non-linear activations, and training truncation—and that a position embedding designed to be robust against this "Spectrum Damage" (FoPE) can recover stable performance at sequence lengths far beyond the training window. What FoPE builds is a drop-in replacement for RoPE's rotation mechanism that (a) models each attention dimension as a Fourier Series (multiple frequencies per dimension) instead of a single sinusoid, acknowledging and compensating for the spectral mixing already happening, and (b) zeroes out frequency components whose periods exceed the training length, replacing them with a position-invariant DC term that cannot cause extrapolation errors.

3.2 Big-Picture Architecture (Diagram in Words)

The FoPE system modifies exactly two things in a standard transformer: how position embeddings are computed before application to query/key vectors, and which frequencies are included in the position embedding. Information flows through the model exactly as in a standard RoPE-based transformer—tokens enter, are embedded, pass through stacked attention+FFN layers, and produce output logits—but with the following targeted changes:

1. Base Frequency Generator — The standard RoPE inverse frequency computation (1.0 / (rope_theta ** (2m/M))) is modified to filter out any frequency $\omega_m < 2\pi / L_{\text{train}}$ where $L_{\text{train}}$ is the maximum training sequence length. These filtered frequencies are discarded entirely; they do not participate in position encoding.

2. Fourier Coefficient Matrices — Two learned (but gradient-free) weight matrices $W_{\sin}, W_{\cos} \in \mathbb{R}^{H \times D_{\text{in}} \times D_{\text{out}}}$ map the remaining $D_{\text{in}}$ frequencies to $D_{\text{out}}$ Fourier coefficients per attention head, where $D_{\text{out}} \leq D_{\text{in}}$ and typically $D_{\text{out}} = \text{head\_dim} / 4$. These matrices are initialized with Xavier normal scaled by $\sigma$ and added to an identity matrix (so the dominant component is still the original RoPE frequency), then frozen during training.

3. Fourier Position Embedding Application — At runtime, the sine and cosine position signals are computed as in RoPE, but then matrix-multiplied with the Fourier coefficient matrices, producing a weighted sum of multiple frequency components for each output dimension. This weighted sum forms the position-dependent rotation applied to query and key vectors.

4. Padding to Head Dimension — Since $D_{\text{out}} < \text{head\_dim} / 2$, the Fourier-transformed position signals are zero-padded to match half the head dimension, then concatenated with themselves (matching RoPE's interleaving of real and imaginary parts). The padded portion effectively corresponds to zero-frequency (position-invariant) components.

The critical architectural difference from RoPE is this: RoPE applies rotation at a single frequency per dimension, assuming a one-to-one correspondence. FoPE applies rotation as a learned weighted sum of multiple frequencies per dimension, plus a zero-frequency pass-through for dimensions beyond the learned Fourier basis.

3.3 Roadmap for the Deep Dive

• First, the formal formulation of Spectrum Damage (Section 3.1 of the paper) and its mathematical consequences for periodic extension, since this is the problem FoPE is designed to solve and understanding the damage model is prerequisite to understanding why each design choice is made.
• Second, the Fourier Series construction for multi-frequency modeling per dimension (Section 4, "Treating Each Dimension as Multi-Frequency"), including the initialization strategy and its theoretical justification from the Spectrum Damage analysis.
• Third, the zeroing-out of undertrained frequencies (Section 4, "Zero-out Under-trained Frequencies"), including the floor frequency definition, the choice of zero as the replacement frequency, and the DSP rationale from time-domain truncation analysis.
• Fourth, the complete FoPE forward pass formulation and the weight matrix implementation that realizes it, connecting the mathematical equations to the pseudocode in Appendix B.
• Fifth, the hyperparameter design space ($\sigma$ controlling Fourier coefficient variance, $D$ controlling the number of harmonic frequencies, the floor frequency threshold) and the empirical basis for the specific values chosen.

3.4 Detailed, Sentence-Based Technical Breakdown

The Spectrum Damage Model and Why It Destroys Periodic Extension

What the damage model formalizes. RoPE's periodic extension property (Equation 5 in the paper) states that in dimension $m$, the attention score should satisfy $\tilde{h}_m(n + N_{\omega_m}) = \tilde{h}_m(n)$ where $N_{\omega_m} = 2\pi / \omega_m$ is the period of the frequency component assigned to that dimension. This property is what enables the model to handle relative positions $n$ it never saw during training—the sinusoidal pattern simply repeats. However, this requires that dimension $m$ contains information only at frequency $\omega_m$. If dimension $m$ contains a mixture of frequencies, the signal is:

$h'_m(n) = H_{\omega_m}\left[(1 - \sigma)e^{i\omega_m n} + \sigma e^{i\omega_o n}\right]$

where $H_{\omega_m}$ is the total coefficient magnitude, $\sigma \in [0, 1]$ is the fraction of energy contributed by the leaked frequency $\omega_o$, and $e^{i\omega_o n}$ is the phase factor at the alien frequency.

When and why the property breaks. The period $N_{\omega_m}$ is the period of the dominant frequency $\omega_m$, not of the leaked frequency $\omega_o$. Replacing $n$ with $n + N_{\omega_m}$ gives:

$h'_m(n + N_{\omega_m}) = H_{\omega_m}\left[(1 - \sigma)e^{i\omega_m n} + \sigma e^{i\omega_o (n + N_{\omega_m})}\right]$

The first term is periodic because $e^{i\omega_m (n + N_{\omega_m})} = e^{i\omega_m n}$. The second term is not periodic with period $N_{\omega_m}$ unless $\omega_o$ happens to be an integer multiple of $\omega_m$. Since $\omega_o$ is some arbitrary frequency leaked from another dimension, this coincidence essentially never occurs. The result is:

$h'_m(n + N_{\omega_m}) \neq h'_m(n)$

Consequence in plain operational terms. When the model computes attention between tokens at positions $n_a$ and $n_b$, it uses the position embedding to encode the relative distance $n = n_a - n_b$. If the attention score in dimension $m$ is supposed to represent "how strongly tokens at distance $n$ interact through wave pattern $\omega_m$," but dimension $m$ actually contains a mixture of wave patterns, then the model's estimate of this interaction for an unseen distance $n_{\text{new}}$ will be wrong—it will be a blend of the correct interaction through $\omega_m$ and an unpredictable interaction through $\omega_o$. As distances grow beyond the training range, this error accumulates and ultimately destroys the pattern the model learned to recognize.

Where the leaked frequencies come from. The paper identifies two sources of spectral contamination:

Spectrum Leakage from linear projections. Between attention layers, hidden states pass through linear transformations. In a standard transformer, after attention produces output $X \in \mathbb{R}^M$ (where $M$ is the hidden dimension), the output projection maps this to the residual stream dimension, and subsequent queries/keys/values are computed via further linear projections. Each such projection is a matrix multiplication:

$Y_m = \sum_{k=0}^{M-1} W_{km} X_k$

Dimension $m$ of the output is a weighted sum of all input dimensions. Since each input dimension $k$ ideally carries information at frequency $\omega_k$ (from the previous attention layer's RoPE), dimension $m$ of the output now contains a superposition of all frequencies $\omega_0, \omega_1, ..., \omega_{M-1}$ whose corresponding $W_{km} \neq 0$. The one-to-one correspondence is destroyed.

Spectrum Distortion from activation functions. The feed-forward networks between attention layers contain non-linear activation functions (typically GELU, SiLU, or ReLU in modern transformers). Lemma 3.1 states the key mathematical result: for any non-linear function $g$ and a multi-frequency input $x(n) = \cos \omega_1 n + \cos \omega_2 n$, the output contains frequencies at all integer linear combinations:

$g(x(n)) = \sum_{j \in \mathbb{N}} \sum_{k \in \mathbb{N}} a_{j,k} \cos(j\omega_1 + k\omega_2)n$

This is proven via Taylor expansion of $g$ into a power series $\sum_p a_p x^p$, followed by expanding $x^p = (\cos \omega_1 n + \cos \omega_2 n)^p$ using the product-to-sum formula $2 \cos \alpha \cos \theta = \cos(\alpha - \theta) + \cos(\alpha + \theta)$. Each power $p$ generates harmonic and intermodulation frequencies. For example, $p=2$ produces $2\omega_1, 2\omega_2, \omega_1 + \omega_2, \omega_1 - \omega_2$, plus a DC term.

Why this damage is cumulative and inescapable. A transformer with $L$ layers interleaves attention blocks (which apply RoPE-based position encoding and are therefore the intended locus of frequency-specific processing) with feed-forward blocks (which cause Spectrum Distortion) and the intermediate linear projections (which cause Spectrum Leakage). After $L$ such alternations, the signal at any given dimension in a deep layer is a complex superposition of all frequencies present at earlier layers, modified by both leakage and distortion at each step. The periodic extension property that RoPE provides at layer $\ell$ is based on the assumption that the input to layer $\ell$ has clean frequency separation—an assumption violated by layers $1$ through $\ell-1$.

The design implication. If Spectrum Damage cannot be prevented (because linear layers and activations are essential for model capacity), the only option is to make attention's position encoding robust to receiving multi-frequency signals. This is why FoPE models each dimension as a Fourier Series—it explicitly expects and handles the spectral mixing that RoPE's single-frequency model cannot cope with.

Modeling Each Attention Dimension as a Fourier Series

What a Fourier Series is in this context. A Fourier Series represents a function as a weighted sum of sinusoids at a fundamental frequency plus its harmonics. In standard signal processing, a function $f(n)$ with period $N$ can be written as $f(n) = a_0 + \sum_{k=1}^{\infty} [a_k \cos(2\pi k n / N) + b_k \sin(2\pi k n / N)]$. In FoPE, the "function" being represented is the position-dependent contribution to the attention score in each dimension, and the representation uses complex exponentials rather than separate sine/cosine coefficients.

The FoPE attention score per dimension. Whereas RoPE defines the dimension-$m$ attention score contribution as a single-frequency term:

$\tilde{h}_m(n) = H_m e^{i\omega_m n}$

FoPE defines it as a dominant frequency plus a weighted sum of additional harmonic frequencies:

$h_m(n) = H_m(n) \left( e^{i\omega_m n} + \sum_{\omega} a_{\omega} e^{i\omega n} \right)$

where $H_m(n)$ is the position-dependent magnitude (determined by the query-key inner product before rotation), $e^{i\omega_m n}$ is the dominant frequency component (same as in RoPE), $\sum_{\omega}$ runs over a set of additional frequencies $\{\omega\}$, and $a_{\omega}$ are learned coefficients satisfying $|a_{\omega}| < 1$ (the dominant frequency has coefficient 1, and harmonics are weaker).

Why this form addresses Spectrum Damage. The expression inside the parentheses is a truncated Fourier Series in the complex domain. Each term $a_{\omega} e^{i\omega n}$ represents a secondary wave that may be present in this dimension due to leakage or distortion. Rather than forcing dimension $m$ to be a pure frequency (which it isn't, due to the upstream damage), FoPE gives the model parameters $a_{\omega}$ to represent the actual spectral mixture. The dominant frequency $\omega_m$ still has the largest coefficient (1 versus $a_{\omega} < 1$), so the dimension mainly responds to patterns at wavelength $N_{\omega_m}$, but the harmonic terms allow it to also process information that arrived at other wavelengths without that information acting as pure noise.

Operational meaning of "treating each dimension as multi-frequency." In RoPE, when the model computes attention, each dimension $m$ answers the question: "How strongly should tokens at distance $n$ interact, assuming information travels purely on wavelength $N_{\omega_m}$?" In FoPE, each dimension $m$ answers a more nuanced question: "How strongly should tokens at distance $n$ interact, given that information travels primarily on wavelength $N_{\omega_m}$ but also has components at wavelengths $\{N_{\omega}\}$ with strengths $\{a_{\omega}\}$?" This is a form of spectral robustness: FoPE acknowledges the actual spectral content and adapts to it, rather than assuming a clean signal.

The frequency set $\{\omega\}$ and its relationship to $\{\omega_m\}$. The paper specifies that $\{\omega_m\} \subseteq \{\omega\}$—the set of harmonic frequencies includes the RoPE base frequencies as a subset. Additionally, $\{\omega\}$ can contain other frequencies sampled from $[0, \pi]$ (the valid frequency range for discrete signals per the Nyquist-Shannon sampling theorem). In practice, the implementation uses the same inverse frequency computation as RoPE but applied to a larger output dimension $D$ (typically 64 or 128 for the models studied, versus RoPE's $\text{head\_dim}/2$), so the additional frequencies are simply more densely sampled points in the same $[0, \pi]$ range.

Initialization of the Fourier coefficients $a_{\omega}$. The paper states that $a_{\omega}$ are initialized from a normal distribution $\mathcal{N}(0, \sigma)$ where $\sigma$ is a hyperparameter. This initialization is theoretically motivated: the Spectrum Damage introduced by linear layers follows the same distribution as the linear layer weights themselves (since the leakage coefficients $W_{km}$ are drawn from a Gaussian-like distribution at initialization). By initializing $a_{\omega} \sim \mathcal{N}(0, \sigma)$, FoPE starts with the hypothesis that the harmonic components are zero-mean noise of magnitude $\sigma$, and allows training (or, since the coefficients are frozen in the current implementation, selection of $\sigma$) to adjust this hypothesis to match the actual spectral contamination.

The identity-matrix addition in the implementation (self.sin_coef += torch.eye(...), self.cos_coef += torch.eye(...)) ensures that the diagonal entries (where the frequency index matches the output dimension index) have coefficient $1 + \epsilon$ rather than $\sim \mathcal{N}(0, \sigma)$, making the RoPE base frequency $\omega_m$ the dominant term as the mathematical formulation requires ($a_{\omega} < 1$ for non-dominant frequencies). The off-diagonal entries remain $\sim \mathcal{N}(0, \sigma)$, representing the harmonic contributions.

Why gradients are not required for the Fourier coefficient matrices. Appendix B explicitly states: "In our implementation, gradients are not required for these matrices, so FoPE adds negligible memory and computation overhead compared to RoPE." This is a deliberate design choice: the Fourier coefficients are treated as a fixed spectral analysis that does not need to be learned from data. The rationale is that the correct coefficients are determined by the architecture's Spectral Damage characteristics (which frequencies leak into which dimensions via the linear layer weights), which is a structural property of the network topology rather than a data-dependent property. Freezing the coefficients also prevents the model from learning to suppress the harmonic terms entirely (which would reduce FoPE to RoPE and reintroduce vulnerability to Spectrum Damage).

The weight matrix formulation. The paper formalizes the Fourier Series construction as a weight matrix $W^F \in \mathbb{R}^{D \times (M - M_0)}$ where $D$ is the number of harmonic frequencies (hyperparameter), $M$ is the original head dimension, and $M_0$ is the number of dimensions assigned to zero-frequency (see next section). This matrix maps from the $D$ frequency components to the $M - M_0$ output dimensions that will carry multi-frequency position embeddings. In the pseudocode (Appendix B), this is implemented as two separate matrices sin_coef and cos_coef of shape (n_heads, input_dim, output_dim) where input_dim is $D$ and output_dim is $\text{head\_dim} / 2$ (for the real and imaginary parts separately). The matrix multiplication torch.einsum("bhtD, hDd -> bhtd", pos_sin, sin_coef) computes the weighted sum of frequencies for each output dimension, and a subsequent normalization sin_coef / sin_coef.sum(dim=-2, keepdim=True) ensures the coefficients for each output dimension sum to 1 (preventing scale explosion from the random initialization).

Why use D output dimensions when head_dim is larger. A crucial architectural detail: the Fourier-transformed position signals occupy only the first $D$ dimensions of each head's position embedding. The remaining $\text{head\_dim} / 2 - D$ dimensions are zero-padded (F.pad(..., mode="constant", value=1) in the pseudocode, where value=1 means the cosine/sine of zero, i.e., no rotation). These zero-padded dimensions operate as zero-frequency (DC) positional components—their attention contribution is position-independent (since $e^{i \cdot 0 \cdot n} = 1$ for all $n$). This means a fraction of each attention head's capacity is deliberately allocated to position-invariant processing, creating an inherent robustness: even if the Fourier components are completely corrupted by Spectrum Damage at some depth, the DC components provide a stable fallback that doesn't degrade with sequence length.

The choice of $D = 64$ for the 60M model (and $128$ for larger models) represents a design trade-off: more harmonic frequencies provide better resolution for representing the actual spectral mixture, but the total number of frequencies per head is fundamentally limited (per the Nyquist-Shannon theorem, there are only $\text{head\_dim} / 2$ independent frequency samples possible, which is 32 for a 64-dim head).

Zeroing Out Under-Trained Frequency Components

The floor frequency definition. The paper defines the floor frequency $\omega_l$ as:

$\omega_l = \frac{2\pi}{N}$

where $N$ is the training sequence length. Any frequency component $\omega_m$ that is less than $\omega_l$ has a period $N_{\omega_m} = 2\pi / \omega_m$ that is greater than the training length $N$. Such a component completes less than one full sinusoidal cycle during training.

Why undertrained frequencies cause problems at extrapolation time. The DSP analysis in Section 3.3 models the situation exactly. A sinusoid $x_m(n) = e^{i\omega_m n}$ observed only for $n \in [0, N]$ is equivalent to multiplying the infinite sinusoid by a rectangular window $\text{rect}(n)$ that is 1 for $n \leq N$ and 0 for $n > N$. The Fourier transform of this windowed signal is:

$X(\omega) = \alpha \delta(\omega_m) + \frac{\sin[(N - \alpha N_m)(\omega - \omega_m)]}{\omega - \omega_m}$

where $\alpha = \lfloor N / N_m \rfloor$ counts how many complete cycles fit in the training window, $N_m = 2\pi / \omega_m$ is the period, $\delta(\omega_m)$ is the ideal delta function at $\omega_m$, and the second term is a sinc-like distortion caused by the abrupt truncation.

For high-frequency components, $\alpha \gg 1$ (many cycles observed), so the delta term dominates and the distortion is negligible. For low-frequency components where $\omega_m < \omega_l$, we have $\alpha = 0$ (less than one full cycle observed). In this regime, the delta term disappears entirely, and the signal is dominated by the sinc distortion. The model has not seen the actual sinusoid—it has seen only a partial, windowed fragment whose frequency-domain representation is mostly truncation artifact.

Operational consequence at inference time. When the model encounters a position $n > N$ at inference, it applies RoPE's rotation matrix using frequency $\omega_m$. But the query/key vectors that the model learned to produce at this dimension were trained to work with the distorted frequency content (the sinc-smoothed spectrum, not the pure delta). When the position index extends beyond $N$, the rotation angle $\omega_m n$ enters a regime where the relationship between the rotation and the actual frequency content of the hidden state has never been established. The model's computation becomes essentially arbitrary in this dimension.

Why zero is the correct replacement frequency. The paper chooses to replace undertrained frequencies $\omega_m < \omega_l$ with $\omega = 0$. The zero-frequency component has a phase factor $e^{i \cdot 0 \cdot n} = 1$ for all positions $n$, making it position-invariant. This choice has three justifications:

1. Infinite wavelength. Zero frequency corresponds to infinite period ($N_0 = 2\pi / 0 \to \infty$). This means the component can represent both the shortest and the longest possible dependencies simultaneously—it encodes no particular distance preference, allowing the model to decide how to use it based on the content of the query and key vectors rather than their positions.

2. No positional bias. A zero-frequency component introduces zero-average position embedding—it does not push attention weights toward any particular distance range. This is critical for length generalization because it means these dimensions will not produce systematically wrong attention patterns at extrapolated positions. As the ablation in Table 2 shows, normalizing query and key vectors (which removes positional bias) helps RoPE models with undertrained frequencies but does not help models whose frequencies all complete full cycles—confirming that the undertrained frequencies are specifically the source of harmful positional bias.

3. Empirical evidence from the floor frequency ablation. In Figure 6(a), the configuration "FoPE w/o CF" (without the floor frequency clipping) underperforms full FoPE, confirming that simply adding Fourier Series without removing the undertrained components is not sufficient. The CF component contributes more to fitting the current dataset and sequence length (improving in-domain perplexity), while the FS component contributes more to length generalization—both modifications are necessary.

Implementation of the clipping. In the pseudocode (Appendix B), the get_inv_freq method computes RoPE's standard inverse frequencies, then applies:

inv_freq[inv_freq < 2 * torch.pi / self.config.max_sequence_length] = 0
inv_freq = inv_freq[inv_freq != 0.0]

This zeroes out all frequencies below the floor, then removes them from the array entirely. The resulting inv_freq tensor has length $M - M_0$ (where $M$ was the original number of frequency components and $M_0$ were zeroed). These remaining frequencies are used as the input to the Fourier coefficient matrices.

The max_sequence_length parameter defines the floor. The paper's experiments use max_sequence_length = 512 for the pre-training experiments (Section 5.2) and the corresponding floor is $2\pi / 512 \approx 0.01227$. For the continual pre-training experiments where the context window is extended to 1024, the floor frequency adapts accordingly. The floor moves with the training context length because the definition of "undertrained" depends on how many cycles a frequency completes within whatever window the model was trained on.

The Complete FoPE Forward Pass

Step-by-step computation. The FoPE forward pass for a query or key tensor $t$ of shape (batch, n_heads, seq_len, head_dim) proceeds as follows:

1. Position signal generation. The method get_rotary_embedding computes the outer product of position indices seq (a vector of integers from 0 to seq_len - 1) with the filtered inverse frequencies inv_freq (now of length $M - M_0$). This produces freqs of shape (head_dim/2, seq_len) for each head, containing the raw phase angles $\omega_m \cdot n$. The sine and cosine of these angles are computed to produce pos_sin and pos_cos.

2. Fourier coefficient application. The method apply_rotary_pos_embed (in the FourierEmbedding subclass) takes these position signals and matrix-multiplies them with the frozen Fourier coefficient matrices:

fourier_sin = torch.einsum("bhtD, hDd -> bhtd", pos_sin, sin_coef / sin_coef.sum(dim=-2))

This computes, for each head h, position t, and output dimension d, the weighted sum $\sum_{i=0}^{D-1} \text{pos\_sin}[h, t, i] \cdot (\text{sin\_coef}[h, i, d] / \sum_j \text{sin\_coef}[h, j, d])$. The normalization by the sum of coefficients ensures the total scale is controlled. The result fourier_sin has shape (batch, n_heads, seq_len, D), where $D$ is the number of harmonic frequencies (e.g., 64 or 128).

3. Padding to half head dimension. The Fourier position signal of length $D$ is zero-padded (with value 1, which is cos(0) / sin(0)) to length head_dim / 2:

fourier_sin = F.pad(fourier_sin, pad=(0, head_dim//2 - fourier_sin.size(-1)), value=1)

This means the first $D$ dimensions carry multi-frequency position information, while the remaining $\text{head\_dim}/2 - D$ dimensions carry the zero-frequency (position-invariant) signal $e^{i \cdot 0 \cdot n} = 1$.

4. Interleaving for real/imaginary parts. RoPE applies rotation by interleaving pairs of dimensions: dimensions 0 and 1 are treated as the real and imaginary parts of one complex number, dimensions 2 and 3 as the next, etc. To match this convention, the padded position signals are concatenated with themselves:

fourier_sin = torch.cat((fourier_sin, fourier_sin), dim=-1)
fourier_cos = torch.cat((fourier_cos, fourier_cos), dim=-1)

Now fourier_sin and fourier_cos each have shape (batch, n_heads, seq_len, head_dim), matching the query/key tensor.

5. Rotation application. The final rotation uses the same formula as RoPE:

return ((t * fourier_cos) - (self.rotate_half(t) * fourier_sin)).to(t.dtype)

For each pair of dimensions (2i, 2i+1), this applies the 2D rotation matrix [[cos θ, -sin θ], [sin θ, cos θ]] where $θ$ is the Fourier-combined phase angle for that dimension pair. The rotate_half method reshapes the tensor to separate the pairs and swaps/interleaves them with negation.

What this means for the effective rotation angle. In standard RoPE, dimension pair $i$ is rotated by angle $\omega_i n$. In FoPE, dimension pair $i$ is rotated by a weighted combination of angles $\sum_j a_{i,j} \omega_j n$ (for the Fourier dimensions) or by angle 0 (for the zero-padded dimensions). The effective rotation is therefore a "spectrally blurred" version of RoPE's rotation, where each dimension sees the position signal through a custom filter defined by its row of the Fourier coefficient matrix.

Hyperparameter Design and Configuration

The variance $\sigma$ of the Fourier coefficient initialization. The paper sweeps $\sigma$ from 0 to 0.5 for the 60M model (Figure 6b), finding that $\sigma = 0.3$ gives the best perplexity, especially for longer contexts. This $\sigma$ can be interpreted as an estimate of the strength of Spectrum Damage in the 60M model: higher $\sigma$ means the harmonic components are initialized with larger magnitudes, allowing them to represent stronger spectral contamination. The optimal $\sigma$ increases with model scale (0.3 for 60M, 0.4 for 180M, 0.6 for 1.2B, per Table 3), which is consistent with the hypothesis that larger models have more parameters and therefore more total Spectrum Damage across their linear layers.

The number of harmonic frequencies $D$. The paper sweeps $D$ from 16 to 128 (Figure 6c) for the 60M model, finding that $D = 64$ maximizes Passkey Retrieval accuracy while $D$ has minimal effect on perplexity. This result is interpreted as: there is a limited number of "strong enough" noisy frequency components (harmonics from activation functions, leakage from linear layers) that have sufficient energy to interfere with the dominant frequency. Including too many harmonic frequencies ($D$ too large) means the model attends to weak, irrelevant components; too few ($D$ too small) means it cannot represent the actual spectral contamination. The optimal $D$ scales with model size (64 for 60M, 128 for 180M and 1.2B, per Table 3), consistent with larger models having richer spectral contamination.

Why the Fourier coefficients are head-specific. The implementation uses separate sin_coef and cos_coef matrices for each attention head (shape (n_heads, input_dim, output_dim) rather than (input_dim, output_dim) shared across heads). This is motivated by the observation that different attention heads in a transformer specialize in different positional relationships (some attend locally, some attend broadly, some attend to specific distance ranges). The Spectrum Damage each head receives depends on which upstream heads it draws information from (via the attention output projection and subsequent linear layers). Head-specific coefficients allow each head to compensate for its unique spectral contamination pattern.

The floor frequency threshold is not a tunable hyperparameter. It is deterministically set to $2\pi / L_{\text{train}}$ where $L_{\text{train}}$ is the training sequence length. This is a theoretically derived threshold from the rectangular window analysis: any frequency with period longer than $L_{\text{train}}$ has $\alpha = 0$ in Equation 10 and is therefore dominated by the sinc distortion term. The paper does not experiment with different threshold values because the threshold follows directly from the DSP theory.

Summary of per-model-scale hyperparameters (from Table 3 and text).

For the fine-tuning experiments with SmolLM-1.7B (Section 5.4), the paper does not specify separate hyperparameters, implying the same principles were applied with architecture-appropriate scaling.

Why the Fourier Coefficient Matrices Are Frozen During Training

The paper's rationale. Appendix B states: "In our implementation, gradients are not required for these matrices." The decision to freeze the Fourier coefficients is significant because it means FoPE does not learn to adapt its spectral representation to the training data—it uses a fixed spectral analysis that depends only on the initialization hyperparameters $\sigma$ and $D$.

The theoretical justification (implicit). The Fourier coefficients $a_{\omega}$ in the mathematical formulation (Equation 11) represent the spectral contamination pattern—how strongly each harmonic frequency leaks into each dimension. This contamination pattern is determined by the network's linear layer weights, which are randomly initialized and then trained on the language modeling objective. However, the statistical properties of the contamination (its variance $\sigma$, the number of strong harmonic components $D$) depend on the architecture (depth, width, activation functions) more than on the specific training data. The initialization $\mathcal{N}(0, \sigma)$ captures the expected magnitude of Spectral Damage without needing to learn per-dimension, per-frequency coefficients. Freezing the coefficients prevents the possibility that training collapses the Fourier Series back to a single-frequency representation (which would make FoPE equivalent to RoPE and reintroduce vulnerability to Spectrum Damage).

Practical benefit. Frozen coefficients mean FoPE adds no trainable parameters to the model. The only additional memory is the storage of the coefficient matrices themselves, which is negligible: for a 1.2B model with 16 heads of dimension 128 and $D = 128$, the matrices are 16 × 128 × 128 = 262,144 float32 values each for sin and cos, totaling about 2 MB—compared to the 1.2 billion parameters of the model itself. The computational overhead is the einsum operation, which is a small matrix multiplication per attention head, easily dwarfed by the attention computation itself.

Why not learn the coefficients? The paper does not experiment with learned Fourier coefficients. A plausible hypothesis (not stated in the paper but consistent with the framework) is that learning the coefficients would allow the model to suppress the harmonic terms entirely, reducing FoPE to RoPE plus zero-frequency padding. The loss function (next-token prediction) rewards whatever minimizes perplexity on the training distribution—it has no incentive to preserve spectral robustness for unseen sequence lengths. Freezing the coefficients is a form of architectural regularization that forces the model to use the multi-frequency representation regardless of whether it helps in-domain training loss, because the representation is necessary for out-of-domain generalization.

4. Key Insights and Innovations

Innovation 1: The Diagnosis That RoPE's Failure Is a System-Level Problem, Not an Attention-Level Problem

The paper's most fundamental intellectual contribution is not the FoPE architecture itself, but the diagnostic framework that identifies where length generalization breaks down. Prior to this work, the field's mental model was essentially: "RoPE is periodic, so it should extrapolate. The fact that it doesn't must mean the periodicity isn't being exploited properly, and we need better ways to use or extend RoPE's rotation scheme within the attention computation." This assumption is visible in every prior extension method—YARN (Peng et al., 2023) interpolates rotation frequencies, CLEX (Chen et al., 2024a) learns continuous scaling, Self-Extend (Jin et al., 2024) modifies the attention mask—all of which operate entirely within the attention mechanism, modifying RoPE's application but never questioning whether the signal RoPE receives is already corrupted.

The paper reframes the problem entirely. Using Discrete Signal Processing theory, it demonstrates that RoPE's attention computation is an Inverse Non-Uniform Discrete Fourier Transform (NUDFT), and the length generalization property depends on each dimension of the hidden state carrying a pure sinusoidal signal at its assigned frequency. The key diagnostic move is then to trace what happens to this pure signal as it passes through the transformer's other components. The finding is damning: by the time a hidden state reaches a deep attention layer, the signal has been irreversibly scrambled by two mechanisms—Spectrum Leakage from linear projections (Equation 7) and Spectrum Distortion from activation functions (Lemma 3.1)—neither of which has anything to do with attention itself.

This constitutes a fundamental reframing of the length generalization problem. The dominant prior assumption was that RoPE's periodic extension is a latent property that needs to be "unlocked" through better inference-time strategies. The paper's DSP analysis shows the opposite: the periodic extension is actively destroyed by the normal operation of the transformer, and no amount of attention-level engineering can fix it because the damage has already occurred before the attention layer receives its input. Figure 2(a) provides the visual summary of this reframing: the signal enters attention as a clean single-frequency component → passes through linear and non-linear transformations outside attention → emerges as a spectral mixture mismatched to RoPE's single-frequency rotation.

The significance of this diagnosis extends beyond the specific solution (FoPE). It establishes a new principle: length generalization is not solely a positional encoding problem—it is a system-level signal processing problem. Any position embedding that assumes clean frequency separation will eventually fail in deep transformers because spectral contamination is an inherent consequence of the architecture's depth and non-linearity. This principle explains not only why RoPE fails but also why prior extension methods achieve only partial success: they treat the symptom (incorrect rotation at long distances) without addressing the cause (the rotated vectors no longer encode the frequency the rotation assumes).

Evidence for the diagnostic validity comes from the ablation in Figure 6(a), where the Fourier Series component of FoPE (which addresses Spectrum Damage by accepting and modeling multi-frequency signals) is shown to be the primary contributor to length generalization, while the floor-frequency clipping (which addresses the separate truncation problem) contributes more to in-domain fitting. This decomposition confirms that Spectrum Damage and training truncation are independent failure mechanisms requiring separate interventions, validating the paper's theoretical analysis that identified them as distinct phenomena.

Innovation 2: The Concept of Spectrum Damage as a Unifying Diagnostic Category

The paper introduces Spectrum Damage as a new concept in the analysis of transformer position embeddings—one that unifies phenomena previously treated as unrelated or ignored entirely. The field has long recognized that transformer components interact in complex ways, but there was no vocabulary for describing how these interactions specifically corrupt position information. The paper provides that vocabulary and, critically, grounds it in established signal processing theory that gives precise meaning to the terms.

What distinguishes this from vague notions of "information mixing" or "feature interaction" is the mathematical specificity. Spectrum Leakage is defined via Equation 7 as a direct consequence of linear layer weight mixing, not as a hand-wavy "the model might confuse frequencies." Spectrum Distortion is given a formal lemma (3.1) that proves activation functions generate harmonic and intermodulation frequencies, with explicit examples showing that a two-frequency input produces output at 2ω₁, 2ω₂, ω₁ + ω₂, and ω₁ - ω₂. The consequences are tracked through a damage model (Section 3.1) that quantifies exactly how a leaked frequency ω_o with coefficient σ destroys the periodic extension property: the signal becomes H[(1-σ)e^{iω_m n} + σe^{iω_o n}], which has no well-defined period.

This conceptual contribution matters because it converts a mystery into an engineering problem. Prior to this work, the fact that RoPE-trained models collapse at 2× training length was an empirical observation with no satisfying theoretical explanation—YARN and similar methods worked, but it wasn't clear why they were necessary given RoPE's mathematical periodicity. The Spectrum Damage framework provides a clear causal chain: (1) linear layers and activations between attention blocks → (2) multi-frequency mixtures in each hidden dimension → (3) mismatched rotation in subsequent attention layers → (4) broken periodic extension → (5) catastrophically wrong attention scores at extrapolated positions. Each step is theoretically grounded, not just empirically observed.

The framework also serves a diagnostic function for future research. If a new position embedding method fails to length-generalize, researchers can now ask: Is the failure from Spectrum Leakage (linear layer mixing), Spectrum Distortion (activation harmonics), or under-trained components (truncation artifacts)? The paper provides the analytical tools—the NUDFT interpretation, the frequency-domain analysis of truncation, the harmonic generation lemma—to distinguish these causes and design targeted interventions. This is a significant advance over the previous state, where length generalization failures were diagnosed primarily through empirical trial-and-error.

This is a fundamental conceptual contribution rather than an incremental refinement. The terms "Spectrum Damage," "Spectrum Leakage," and "Spectrum Distortion" did not exist in the ML literature prior to this paper as applied to position embeddings. They provide a shared vocabulary and theoretical foundation for a problem that was previously understood only at the empirical level.

Innovation 3: Accepting Rather Than Fighting Spectral Contamination

The paper makes a subtle but intellectually distinctive design choice that distinguishes it from prior work: FoPE does not attempt to prevent or reverse Spectrum Damage—it accepts it as inevitable and designs the position embedding to be robust to it. This is a fundamentally different philosophy from the extension-method approach.

Prior methods like YARN operate on the premise that RoPE's periodicity is correct but needs adjustment for unseen positions. They attempt to "fix" the frequency assignment by interpolating position indices, essentially saying "if position 8192 is out of distribution, map it back to a position the model knows." This is an avoidance strategy: it keeps the signal within the training regime by warping the position space. It does nothing to address the fact that the signal itself is spectrally corrupted.

FoPE's approach, by contrast, is an acceptance strategy. It acknowledges that by layer ℓ, dimension m of the hidden state will contain information at frequencies ω_o ≠ ω_m due to upstream leakage and distortion. Rather than trying to clean this up (which would require undoing the effects of all prior linear and non-linear transformations—essentially recomputing the entire forward pass), FoPE gives the attention layer the capacity to represent the mixture explicitly. The Fourier Series formulation e^{iω_m n} + Σ_ω a_ω e^{iωn} says: "Dimension m is primarily responsible for frequency ω_m, but it also carries attenuated versions of other frequencies with coefficients a_ω. Here are parameters to model that."

The intellectual significance of this choice is that it reframes robustness from a signal-cleaning problem to a signal-modeling problem. The paper recognizes that linear layers and activation functions are not bugs to be eliminated—they are essential for model capacity, and their spectral side effects are the price of that capacity. The correct response is not to restrict the architecture (which would reduce expressivity) but to make the position encoding sophisticated enough to handle the signals the architecture actually produces.

This distinguishes FoPE from approaches that might try to enforce spectral purity through architectural constraints (e.g., normalizing hidden states to eliminate frequency mixing, or using linear activations in FFN layers, both of which would severely limit model capacity). It also distinguishes FoPE from approaches that accept the damage but try to compensate through learned corrections at inference time—FoPE bakes the acceptance into the training process itself, making robustness a structural property rather than a post-hoc patch.

The evidence that this acceptance strategy works where fighting it fails is in the comparison between FoPE and the "FoPE w/o FS" ablation (Figure 6a): removing the Fourier Series component (which is the mechanism for accepting multi-frequency signals) significantly degrades length generalization, confirming that the multi-frequency modeling is genuinely addressing the Spectral Damage problem, not merely adding model capacity that happens to help.

Innovation 4: Identifying the Undertrained Frequency Problem as a Fourier Truncation Phenomenon

While prior work (Peng et al., 2023) had empirically observed that RoPE's low-frequency dimensions pose extrapolation challenges, this paper provides the first formal frequency-domain explanation for why this occurs and derives the criterion for which frequencies are affected. The analysis in Section 3.3—modeling the training window as a rectangular truncation of an infinite sinusoid and examining the resulting frequency-domain distortion—is genuinely novel in the ML position embedding literature.

The key insight is Equation 10:

$X(ω) = αδ(ω_m) + \frac{\sin[(N - αN_m)(ω - ω_m)]}{ω - ω_m}$

This equation states that the frequency-domain representation of a sinusoid observed only for n ∈ [0, N] consists of the ideal delta function δ(ω_m) scaled by the number of complete cycles observed (α), plus a sinc-shaped distortion term. The critical case is α = 0—when the training window captures less than one complete cycle of the sinusoid. In this regime, the delta term vanishes entirely, and the signal is dominated by the distortion. The model has never actually observed the sinusoid; it has observed only the window artifact.

Prior work recognized that low frequencies behaved differently but treated this as an empirical observation requiring heuristic solutions (e.g., YARN's frequency-dependent interpolation rates, where low frequencies get different scaling factors than high frequencies). This paper grounds the observation in signal processing theory, providing both a clear criterion for which frequencies are problematic (ω_m < 2π/N) and a theoretically justified solution (replace with zero frequency, which has the unique property of being simultaneously the shortest and longest wavelength, providing position-invariant processing).

The practical significance of this formalization extends beyond the specific solution. It explains why extrapolation fails more severely for some dimensions than others, why the number of problematic dimensions grows with training length (longer training means the floor frequency decreases, so fewer dimensions fall below it), and why the solution must be applied at training time rather than inference time (once the truncated sinusoids have shaped the learned query/key representations, no amount of inference-time frequency adjustment can undo the fact that the model learned to work with distorted frequency content).

This constitutes a fundamental theoretical advance in understanding position embeddings, not just an incremental improvement. It takes a phenomenon that was previously understood only at the level of empirical heuristics and provides an exact mathematical model with a derived decision boundary.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The primary evaluation uses the C4 validation set (Raffel et al., 2020) for perplexity measurement and Passkey Retrieval(Mohtashami & Jaggi, 2023) for long-context retrieval accuracy. Pre-training uses either a 10B-token subset of C4 or ~5B tokens from Gutenberg Books(Hart, 2007). For fine-tuning experiments, the paper evaluates on summarization datasets (GovReport(Huang et al., 2021), MultiNews(Fabbri et al., 2019)) and few-shot QA datasets (TREC(Li & Roth, 2002), TriviaQA(Joshi et al., 2017), SAMSum(Gliwa et al., 2019)), all under the LongBench evaluation setup (Bai et al., 2023). The Passkey Retrieval task tests 1000 trials per context length with randomly positioned five-digit passkeys embedded in meaningless filler text, ensuring position sampling is "sufficiently dispersed" (Section 5.1).

• Base model(s). Experiments are conducted on OLMo(Groeneveld et al., 2024) at three scales—60M, 180M, and 1.2B parameters—trained from scratch for pre-training and continual pre-training experiments. For fine-tuning, the paper uses SmolLM-1.7B(Allal et al., 2024), a capable open-source model with LLaMA architecture pre-trained with 2048 context length. The scale span from 60M to 1.2B tests whether FoPE's benefits generalize across model sizes, while SmolLM-1.7B tests compatibility with existing open-source architectures and more complex downstream tasks.

• Metrics. Two primary metrics are used: (1) Perplexity on the C4 validation set, measured at sequence lengths from 512 to 8192 tokens to assess how language modeling quality degrades with context extension; (2) Accuracy on Passkey Retrieval, defined as the fraction of 1000 trials where the model correctly retrieves the hidden five-digit number, evaluated at sequence lengths from 512 to 8192. For fine-tuning experiments, task-specific metrics are used: ROUGE scores for summarization (exact metric not specified but standard for GovReport and MultiNews evaluation) and accuracy/F1 for QA tasks, all reported by the LongBench framework. Perplexity ratio (PPL_c4 / PPL_books) is used in ablation studies (Figure 6a-b) to measure cross-distribution generalization.

• Baselines. The paper compares against five baselines: (1) RoPE(Su et al., 2024)—the standard rotary position embedding used in most modern LMs; (2) NoPE—no position embedding at all, included as a lower bound; (3) ALiBi(Press et al., 2021)—linear attention bias that decays with distance; (4) KERPLE(Chi et al., 2022)—kernelized relative position embedding; (5) FIRE(Li et al., 2024)—functional interpolation for relative positions. For continual pre-training experiments, YARN(Peng et al., 2023) serves as the extrapolation method baseline. All baselines are trained under identical conditions (same data, same optimization, same architecture except for the position embedding module).

• Generation budget / compute accounting. Since this paper studies position embeddings (not test-time compute scaling), "compute" is measured in terms of training tokens and model parameters, with equal training budgets applied to all methods. Pre-training uses a fixed 10B-token budget for C4 experiments and ~5B tokens for Gutenberg Books experiments. Continual pre-training uses ~1B tokens. Fair comparison is maintained by using identical architectures (layers, heads, dimensions) across all position embedding methods, differing only in the position encoding module. For the fine-tuning experiments, all models use the identical SmolLM-1.7B training recipe (Allal et al., 2024) with approximately 350k samples for one epoch, using AdamW with learning rate 3e-4 and cosine scheduler with 0.1 warmup ratio.

• Cross-validation / statistical protocol. The paper does not use formal cross-validation for pre-training experiments—evaluations are conducted on the final checkpoint. For fine-tuning, checkpoints are saved every 100 steps and the best result for each method is reported, which the paper justifies because "YARN is prone to overfitting with excessive fine-tuning steps, a limitation not observed in FoPE" (Appendix A.3). For Passkey Retrieval, 1000 trials per context length ensure statistical reliability of the accuracy measurements. The paper does not report confidence intervals or standard deviations for any result, which is a notable omission given the relatively small test sets (e.g., 500-question MATH equivalent not present here; instead the evaluation is on standard validation splits whose sizes are not explicitly stated beyond "1000 trials" for Passkey).

Main Quantitative Results

Length Generalization After Pre-Training (Section 5.2)

Headline findings. FoPE achieves dramatically more stable length generalization than RoPE on both perplexity and retrieval metrics. On Passkey Retrieval (Figure 1a), the 1.2B FoPE model maintains near-perfect accuracy at all tested sequence lengths (512–8192), while the RoPE model's accuracy drops to zero at twice the training length (1024) and remains at zero for all longer sequences. On C4 perplexity (Figure 1b), FoPE shows significantly less degradation than RoPE as sequence length increases, though ALiBi achieves the lowest absolute perplexity across all lengths due to its bias toward local context in a corpus dominated by short-distance dependencies.

Detailed Passkey results (Figure 1a). The 1.2B model trained with sequence length 512:

• FoPE: Accuracy remains approximately 90–100% across all lengths from 512 to 8192. There is no systematic decline with increasing context length.
• RoPE: Accuracy is approximately 100% at length 512, drops to ~0% at 1024, and remains at ~0% for 2048, 4096, and 8192. The collapse is catastrophic and complete.
• ALiBi: Accuracy declines approximately linearly from ~85% at 512 to ~20% at 8192, confirming the paper's claim that ALiBi's linear decay "is unable to capture information from long distances" (Section 5.2).
• NoPE, KERPLE, FIRE: These baselines appear in Figure 1b (perplexity) but their Passkey performance is not plotted in Figure 1a. The paper does not report Passkey results for these methods, which is a notable omission—it means the claim that FoPE "demonstrates a significant advantage over all baselines" cannot be verified for the Passkey metric against KERPLE and FIRE specifically.

Detailed perplexity results (Figure 1b). For the 1.2B model on C4 validation:

• FoPE: Perplexity rises modestly from ~18 at length 512 to ~30 at length 8192—a deterioration, but a controlled one.
• RoPE: Perplexity rises much more steeply, from ~18 at 512 to ~50+ at 8192.
• ALiBi: Perplexity is nearly flat at ~16–17 across all lengths. The paper explicitly addresses this (Section 5.2): "On the one hand, the corpus in C4 and Books mainly have short-distance dependency, thus the information from a short context window is enough for the prediction of almost all tokens. On the other hand, AliBi uses linear declined attention to eliminate long-distance information, and only pays attention to short-distance dependency." In other words, ALiBi wins perplexity because the task doesn't require long-range attention—but this is a domain-specific artifact, not a general advantage, as the Passkey results confirm.

Cross-distribution generalization (Figure 5). When trained on Gutenberg Books and evaluated on C4 (different data distribution), FoPE again demonstrates significant advantage over RoPE at all three scales (60M, 180M, 1.2B). The perplexity curves for FoPE are consistently and substantially lower than RoPE at all sequence lengths from 512 to 8192. ALiBi again shows flat perplexity but at the cost of long-range capability (not plotted in Figure 5; the Passkey collapse for ALiBi in Figure 1a confirms the trade-off). This result is important because it demonstrates that FoPE's robustness is not specific to the training distribution—it transfers across data domains.

Scale consistency. The results in both Figure 1 and Figure 5 show qualitatively identical patterns across 60M, 180M, and 1.2B model scales: FoPE consistently outperforms RoPE on length generalization, with the gap widening at longer sequence lengths. This scale consistency is evidence that the Spectrum Damage mechanisms FoPE addresses are fundamental to the transformer architecture rather than artifacts of small-scale models. However, the paper only tests up to 1.2B parameters—roughly two orders of magnitude smaller than production LMs (7B, 70B)—so whether the benefits extrapolate to much larger scales remains unverified experimentally.

Length Generalization After Continual Pre-Training (Section 5.3)

Headline finding. FoPE not only outperforms RoPE as a base position embedding but also serves as an effective extrapolation method when applied to pre-trained RoPE models, matching or exceeding YARN—the current standard extrapolation technique. This is demonstrated in Figure 4, which tests two scenarios: (a) extending a 512-context model to 1024 using YARN versus FoPE, and (b) applying FoPE to a pre-trained RoPE model versus applying YARN.

Figure 4(a)—Passkey Retrieval. When the base model is FoPE-512 (trained with FoPE and 512 context):

• FoPE-512 + FoPE-1024: Achieves the highest accuracy, maintaining near 100% across lengths 512–2048 with only slight decline at 4096–8192.
• FoPE-512 + YARN-1024: Performance is significantly worse, approaching 0% at lengths beyond 4096.
• FoPE-512 alone (no continual pre-training, extrapolating directly): Maintains accuracy reasonably well but declines more at longer lengths.

When the base model is RoPE-512 (trained with standard RoPE):

• RoPE-512 + FoPE-1024: This is the critical test—applying FoPE as an extrapolation method to a RoPE-trained model. Performance is strong, maintaining high accuracy through 4096 and showing some decline at 8192. This matches or exceeds RoPE-512 + YARN-1024.
• RoPE-512 + YARN-1024: Performance declines substantially at lengths beyond 2048.
• RoPE-512 alone: Collapses to near 0% at 1024, as expected.

Figure 4(b)—C4 Perplexity. The same pattern holds for perplexity:

• FoPE-512 + FoPE-1024 achieves the lowest perplexity across all lengths.
• RoPE-512 + FoPE-1024 achieves lower perplexity than RoPE-512 + YARN-1024 at all lengths, confirming FoPE's effectiveness as an extrapolation method for pre-trained models.

Significance. This result has practical importance beyond academic benchmarking: it means FoPE can be applied to existing RoPE-based open-source models (Llama, Mistral, etc.) to improve their length generalization without retraining from scratch. The paper explicitly states (Section 5.3): "These findings underscore the effectiveness and practical utility of FoPE, which holds the potential to enhance all RoPE-based open-source models." However, the result is demonstrated only on the authors' own 60M–1.2B OLMo models, not on actual open-source models. The fine-tuning experiments with SmolLM-1.7B (Section 5.4) partially address this gap, but no continual pre-training experiment is reported on a production-scale model (7B+).

Length Generalization After Fine-Tuning on Complex Tasks (Section 5.4)

Headline finding. When fine-tuning SmolLM-1.7B (context length 2048 → 4096) on summarization and QA tasks, FoPE consistently outperforms RoPE on both in-domain (0–4k) and out-of-domain (4–8k, 8k+) sequence lengths, with the advantage widening at longer contexts. The results are reported in Table 1.

Summarization (GovReport, MultiNews). For GovReport (ROUGE scores, higher is better):

• 0–4k: FoPE 13.27 vs. RoPE 13.02 (improvement of +0.25, marginal)
• 4–8k: FoPE 12.50 vs. RoPE 11.35 (+1.15)
• 8k+: FoPE 12.38 vs. RoPE 12.02 (+0.36)

For MultiNews:

• 0–4k: FoPE 12.92 vs. RoPE 12.71 (+0.21)
• 4–8k: FoPE 12.98 vs. RoPE 11.11 (+1.87)
• 8k+: FoPE 12.23 vs. RoPE 10.85 (+1.38)

The pattern is clear: FoPE's advantage grows at longer contexts, consistent with the hypothesis that Spectrum Damage becomes more severe as sequences extend beyond the training length (4096 tokens in this setting, since the base model was pre-trained at 2048 and fine-tuned at 4096).

Few-shot QA (TREC, TriviaQA, SAMSum). For TREC (accuracy):

• 0–4k: FoPE 41.00 vs. RoPE 37.00 (+4.00)
• 4–8k: FoPE 56.00 vs. RoPE 42.00 (+14.0)
• 8k+: FoPE 51.00 vs. RoPE 36.00 (+15.0)

The TREC results are striking: FoPE not only generalizes better to long contexts but actually improves at longer lengths (56.00 at 4–8k vs. 41.00 at 0–4k), while RoPE degrades slightly (42.00 vs. 37.00). This is a surprising finding that the paper does not explain—it suggests that for certain tasks, the multi-frequency representation may actually benefit from the additional positional information available in longer sequences.

For TriviaQA (accuracy):

• 0–4k: FoPE 33.26 vs. RoPE 36.50 (−3.24) ← FoPE worse
• 4–8k: FoPE 33.53 vs. RoPE 36.12 (−2.49) ← FoPE worse
• 8k+: FoPE 33.87 vs. RoPE 25.02 (+8.85) ← FoPE significantly better

This is the one benchmark where FoPE underperforms RoPE at shorter contexts. The paper acknowledges this (Section 5.4): "The only exception occurs in TriviaQA, where FoPE performs slightly worse in shorter contexts. However, FoPE's performance remains stable up to 8k+ and significantly outperforms RoPE." This result suggests a robustness-performance trade-off: FoPE's architectural regularization (frozen Fourier coefficients, zero-frequency padding) may sacrifice some expressive capacity on in-domain lengths to achieve stability at extrapolated lengths. The fact that FoPE's accuracy is essentially flat across all context lengths (~33–34%) while RoPE's collapses from ~36.5 to ~25.0 as context extends past 8k is evidence that FoPE is achieving exactly what it was designed for: stable generalization rather than maximal in-domain performance.

For SAMSum (ROUGE, presuming the metric matches the summarization tasks):

• 0–4k: FoPE 19.77 vs. RoPE 10.27 (+9.50)
• 4–8k: FoPE 15.85 vs. RoPE 6.37 (+9.48)
• 8k+: FoPE 17.26 vs. RoPE 8.49 (+8.87)

The SAMSum results show a massive and consistent advantage for FoPE at all context lengths, including in-domain (0–4k). This suggests that for dialogue summarization specifically, the multi-frequency representation is substantially more effective than single-frequency RoPE even within the training context window. The paper does not analyze why SAMSum benefits so much more than other tasks—this is a missed opportunity for deeper task-level analysis.

Critical observation on Table 1. The paper reports these results as "FoPE delivers better length generalization compared to RoPE" (Table 1 caption), which is accurate for the longer-context results. However, FoPE's in-domain performance (0–4k) is actually worse than RoPE on GovReport, MultiNews, and TriviaQA (though better on TREC and SAMSum). The aggregate picture is more nuanced than "FoPE is uniformly better": it provides stability and extrapolation at the cost of some in-domain performance on certain tasks. This is a classic robustness-accuracy trade-off, and the paper would benefit from acknowledging it more explicitly rather than focusing primarily on the length generalization wins.

Ablation Studies and Robustness Checks

The ablation studies are conducted on 60M models trained on 5B tokens from Gutenberg Books, with the paper justifying this choice by noting "the consistent performance of FoPE across different parameter scales and datasets" (Section 5.5). All ablations use the Perplexity Ratio (PPL_c4 / PPL_books) as the primary metric, except for the D parameter sweep which uses Passkey Accuracy.

Both sub-methods of FoPE are independently useful and complementary (Figure 6a). The ablation compares four variants: full FoPE, FoPE without Fourier Series ("FoPE w/o FS"), FoPE without floor frequency clipping ("FoPE w/o CF"), and standard RoPE. The Perplexity Ratio (lower is better) at sequence length 8192 shows:

• FoPE: ~8.5
• FoPE w/o FS: ~9.5 (worse than full FoPE, but better than RoPE)
• FoPE w/o CF: ~10.5 (worse than full FoPE)
• RoPE: ~14 (substantially worse than all FoPE variants)

The paper interprets this decomposition (Section 5.5): "On one hand, FS contributes more to length generalization, which demonstrates that the Spectrum Damage have a significant influence on length generalization. On the other hand, CF contributes more to fitting the current dataset and sequence length, which implies the zero-frequency component is the most informative and indispensable component." This interpretation is supported by the observation that FoPE w/o FS degrades most at long lengths (8192), while FoPE w/o CF shows elevated perplexity even at short lengths (512–2048), suggesting the floor frequency clipping primarily helps in-domain fitting while the Fourier Series primarily helps extrapolation.

Increasing attention head dimension is more beneficial than increasing number of heads or layers (Figure 6a). The ablation compares:

• FoPE double head_dim: Doubling each head's dimension (from 64 to 128 for 60M model, adjusting total parameters accordingly). Performance is significantly better than baseline FoPE (~7.5 vs. ~8.5 PPL ratio at 8192).
• FoPE double others: Doubling the number of layers and attention heads (from 8 to 16 each). Performance is worse than baseline FoPE (~11 vs. ~8.5 PPL ratio at 8192).

The paper's interpretation (Section 5.5): "More dimensions introduce more frequency components, making attention more robust to Spectral Damage. In contrast, adding more attention heads and layers aggravates Spectrum Damage, which diminishes the benefits of expanding the parameter scale." This is a powerful and non-obvious finding: model scaling strategies affect position embedding robustness, and depth/width trade-offs have different spectral consequences. Increasing head dimension provides more frequency samples per head (allowing better representation of the multi-frequency mixture), while increasing depth adds more layers of Spectrum Damage through additional linear projections and activation functions. This finding has practical implications for architecture design when length generalization is a priority.

The Fourier coefficient variance σ has a sweet spot around 0.3 for 60M models (Figure 6b). The sweep tests σ ∈ {0.0, 0.1, 0.2, 0.3, 0.4, 0.5} with D fixed at 16. The Perplexity Ratio is reported at five sequence lengths (512, 1024, 2048, 4096, 8192):

• At σ = 0.3, the PPL ratio is lowest across all sequence lengths, with the advantage most pronounced at 8192.
• At σ = 0.0 (no harmonic components), performance degrades significantly (~13 PPL ratio at 8192 vs. ~8.5 at σ = 0.3).
• At σ = 0.5 (strong harmonic components), performance also degrades (~10 PPL ratio at 8192), though less severely.

The interpretation is that σ = 0.3 represents the "estimated strength of Spectrum Damage of the 60M model," with σ = 0.0 corresponding to no multi-frequency modeling (reducing to near-RoPE behavior) and σ = 0.5 over-weighting the harmonic components. The paper notes that "the best σ may become larger as the models' parameter scale increases," supported by Table 3 where σ = 0.3, 0.4, 0.6 for 60M, 180M, 1.2B respectively—which is consistent but only three data points, so the scaling trend is suggestive rather than established.

The number of harmonic frequencies D minimally affects perplexity but substantially affects Passkey Accuracy (Figure 6c). The sweep tests D ∈ {16, 32, 64, 128} with σ fixed at 0.3. The paper reports that "D does not significantly influence the perplexity" (the curves are not shown, only stated in text), but Passkey Accuracy shows a clear optimum at D = 64 (for 60M models):

• D = 16: ~45% accuracy averaged across 512–8192 lengths
• D = 32: ~55%
• D = 64: ~70% (best)
• D = 128: ~68% (slightly worse than D = 64)

The interpretation is that D = 64 captures "the estimated number of strong enough noisy components" for the 60M model, and "paying attention to not important components hinders the effectiveness of the model" when D is too large. This is a classic bias-variance trade-off in the frequency domain: too few harmonic components miss relevant spectral contamination, too many include noise. Like σ, D scales with model size (64 for 60M, 128 for 180M and 1.2B, per Table 3), again with only three data points.

FoPE is robust to hyperparameter choice (stated, not shown with a dedicated figure). The paper claims (Section 5.5): "the ablation study shows that FoPE consistently outperforms RoPE across all hyperparameter settings, demonstrating its robustness to hyperparameter selection." This claim is supported by observing that even the worst FoPE configurations in Figures 6a-c outperform RoPE. However, this robustness is only demonstrated for the 60M model scale; whether the same holds at larger scales (where σ and D differ) is not shown.

Empirical validation of the multi-frequency mechanism (Section 5.6, toy model). The paper uses a single-layer MLP with an activation function to simulate how attention scores are affected by Spectrum Damage. By tracking two frequency components through the MLP and comparing the resulting attention scores to (a) the ground truth (frequency-matched reconstruction), (b) RoPE's single-frequency representation, and (c) FoPE's Fourier Series representation (Figure 3), the paper shows that FoPE more accurately captures the multi-frequency periodicity of the actual signal. The authors state: "Comparing Ground Truth with RoPE and FoPE attention scores, we find that FoPE more accurately captures multi-frequency periodicity, yielding attention scores that better align with information transfer." This is a proof-of-concept rather than a rigorous ablation—it demonstrates the mechanism on a minimal example but does not prove that the same mechanism explains the full-scale model results. The toy model uses only two frequency components and a single MLP layer, whereas a real transformer has dozens of layers each with hundreds of frequencies interacting through attention, linear projections, and activations.

Validation of the undertrained frequency hypothesis (Section 5.6, Table 2). The paper visualizes the activation values of query and key vectors across dimensions in Llama2-7B (a model not used elsewhere in the paper's experiments, which is notable). Figure 7 shows that dimensions corresponding to undertrained frequencies (those that complete less than one full cycle during pre-training, specifically dimensions [45, 64] ∪ [109, 128] for Llama2-7B's 128-dim heads with 4096 training length) hold significantly higher absolute activation values across all layers. The interpretation is that "this positional bias may adversely affect robustness to out-of-domain rotation matrix values during length generalization."

To test whether this positional bias is indeed harmful, the paper trains 20M "toy models" (Table 2) with four configurations evaluated at sequence lengths 512–8192:

• RoPE (standard): Loss increases from 5.50 at 512 to 7.16 at 8192.
• RoPE + QK Norm (normalize query/key vectors to mean 0, variance 1 before rotation): Loss increases from 5.46 at 512 to 6.66 at 8192—better than standard RoPE at long lengths, consistent with the hypothesis that positional bias from undertrained frequencies harms extrapolation.
• RoPE-A (adjust all frequencies to the nearest values that exactly complete full cycles within 512): Loss increases from 5.72 at 512 to 6.67 at 8192—worse than RoPE at short lengths but better at long lengths, suggesting that ensuring all frequencies complete full cycles removes the extrapolation penalty but can harm in-domain fitting.
• RoPE-A + QK Norm: Loss from 5.69 to 6.81—QK Norm provides no additional benefit when all frequencies complete full cycles, confirming that QK Norm's benefit is specifically due to normalizing undertrained frequency dimensions.
• NoPE and NoPE + QK Norm: NoPE is included as a baseline; notably, QK Norm worsens NoPE's extrapolation (6.99 → 7.43 at 8192), showing that the normalization benefit is specific to position embeddings with undertrained components.

This experiment is the paper's only empirical evidence specifically isolating the undertrained frequency mechanism. It is cleverly designed: by adjusting RoPE's frequencies to exactly align with the training length (RoPE-A), and comparing QK Norm's effect with and without undertrained components, the paper cleanly demonstrates that the positional bias observed in Figure 7 is causal to extrapolation failure, not merely correlated. However, the models are only 20M parameters—three orders of magnitude smaller than the 1.2B models in the main experiments—and the task is next-token prediction loss, not Passkey Retrieval or downstream tasks. Whether the same mechanism dominates at practical scales remains inferred rather than proven.

Perplexity scaling on standard benchmarks (Appendix A.4, Tables 4–6). The paper evaluates 1.2B models on several standard downstream tasks to verify that FoPE does not sacrifice general capability:

• Accuracy on commonsense reasoning (Table 4): FoPE achieves an average of 43.37% across 9 tasks, slightly outperforming RoPE (42.98%), ALiBi (42.93%), KERPLE (43.22%), and FIRE (42.38%). The differences are small (within ~1 percentage point) and likely not statistically significant at this scale, but they demonstrate that FoPE does not degrade general reasoning.
• Cross-entropy loss (Table 5): FoPE achieves the lowest average loss (1.3941) across 3 tasks, ahead of RoPE (1.4225). Again, the advantage is modest.
• MMLU accuracy (Table 6): FoPE achieves 27.57% average, ahead of RoPE (26.68%) and all other baselines. The largest margin is in "humanity" (29.89% vs. 27.87% for RoPE).

These results are best interpreted as a sanity check: FoPE does not catastrophically impair standard capabilities while dramatically improving length generalization. The paper does not claim that FoPE improves general performance, only that it does not harm it, which these results support. However, all these evaluations are at the models' training length (512 or 1024)—they do not test generalization on these tasks at extended lengths, which would be the more relevant evaluation for a method claiming to improve length generalization.

Critical Assessment

Does FoPE genuinely improve length generalization?

Yes, the evidence is strong for the specific conditions tested. The Passkey Retrieval results (Figure 1a) show an unambiguous and dramatic improvement: RoPE drops to 0% accuracy at 2× training length, while FoPE maintains ~90–100% accuracy at 16× training length (512 → 8192). This is not a marginal improvement—it is the difference between complete failure and near-perfect maintenance of capability. The perplexity results (Figure 1b, Figure 5) show consistent and substantial advantages at all extrapolated lengths. The fine-tuning results (Table 1) demonstrate that the benefits transfer to complex downstream tasks (summarization, QA) and to a different model architecture (LLaMA-based SmolLM). The cross-distribution experiment (Figure 5, Gutenberg → C4) demonstrates that the improvement is not a quirk of the training data distribution.

However, the experiments demonstrate length generalization from 512 to 8192—a 16× extension. Production models like GPT-4 and Claude are trained with context windows of 8K–32K and are expected to generalize to 128K+. The paper does not test whether FoPE's benefits continue at these scales. The theoretical framework (Spectrum Damage from linear layers and activations, undertrained frequencies from truncation) would predict that the problems persist at any scale, but the specific hyperparameters (σ, D, floor frequency) would need to be recalibrated, and it's not obvious that the optimal σ = 0.3–0.6 range would hold for models with 32K training contexts and 256-dim heads.

Does the Spectrum Damage framework explain the results, or are there alternative explanations?

The framework is theoretically sound, but the empirical evidence tying specific mechanisms to specific results is incomplete. The paper's central causal claim is: linear layers and activations → Spectrum Damage → broken periodic extension → length generalization failure → FoPE fixes this by modeling multi-frequency signals and zeroing undertrained frequencies.

The evidence for each link:

• Spectrum Damage exists: The Lemma 3.1 proof and the toy experiment (Figure 3) demonstrate that non-linear activations generate harmonics on synthetic inputs. However, there is no direct measurement of Spectrum Damage in an actual trained transformer—no spectral analysis of hidden states, no quantification of how much frequency mixing occurs per layer, no demonstration that deeper layers have more spectral contamination than shallower ones.
• Spectrum Damage causes length generalization failure: This is inferred from the fact that FoPE (which is designed to be robust to Spectrum Damage) improves length generalization relative to RoPE (which is vulnerable to it). But this is an indirect inference: FoPE differs from RoPE in multiple ways (Fourier Series, zero-frequency padding, frequency clipping), and any of these differences could contribute to the improvement for reasons unrelated to Spectrum Damage. The ablation in Figure 6a shows that the Fourier Series component contributes more to length generalization while the floor frequency clipping contributes more to in-domain fitting, which is consistent with the Spectrum Damage hypothesis but does not prove it—many alternative explanations (e.g., the Fourier Series simply adds capacity, the clipping acts as regularization) are not ruled out.
• Linear layers cause Spectrum Leakage: This is mathematically inevitable (Equation 7: any matrix multiplication mixes dimensions), but the paper does not measure how much leakage occurs, whether it accumulates with depth, or whether certain layer types (attention output projections vs. FFN up-projections) are worse offenders.
• Undertrained frequencies cause extrapolation failure via positional bias: The evidence is stronger here. Figure 7 shows undertrained dimensions have higher activation values. Table 2 shows that normalizing these dimensions (QK Norm) helps only when undertrained components exist, and that adjusting frequencies to complete full cycles (RoPE-A) removes the need for QK Norm. This is a clean causal demonstration—but again at 20M scale, which may not reflect the dynamics of larger models where overparameterization could lead to different learned representations.

Alternative explanations not considered: The paper does not discuss whether the benefits of FoPE could arise simply from having more frequency components (increasing representational capacity) rather than from robustness to Spectrum Damage. A natural control experiment—RoPE with double the number of frequency components per head (achieved by concatenating rather than interleaving dimensions) but without the Fourier Series mixing—is not reported. Similarly, the benefits of zero-frequency padding could arise from having a guaranteed position-invariant pathway through each attention head (which might help with content-based routing independent of position) rather than from avoiding undertrained frequency issues. These are not necessarily competing explanations—they could all be true simultaneously—but the paper attributes the benefits specifically to the Spectrum Damage framework without testing alternative mechanistic interpretations.

Are the FLOPs-matched or compute-equivalent comparisons fair?

Yes, given the paper's focus on architecture comparison rather than efficiency. All models (FoPE, RoPE, ALiBi, etc.) are trained with identical architectures (same layers, heads, dimensions), identical training data, and identical optimization hyperparameters. The only difference is the position embedding module. The paper states that FoPE "adds negligible memory and computation overhead compared to RoPE" (Appendix B) because the Fourier coefficient matrices are small (~2 MB for 1.2B model), frozen (no gradients), and the einsum operation is cheap relative to attention. This claim is plausible but not quantified—no wall-clock time measurements, no FLOP counts, no memory profiling. For the 60M model with D=64 and 8 heads of dim 64, the overhead is 8 heads × 64 input × 64/2 output × 2 matrices (sin/cos) × 4 bytes = ~128 KB and the einsum is batch × heads × seq_len × 64 × 32 multiply-adds per attention layer—trivial compared to the attention computation itself. The claim of "negligible overhead" is reasonable but unverified.

What experiments are missing that would strengthen the paper?

1. Spectral analysis of actual hidden states. A direct test of the Spectrum Damage hypothesis would measure the frequency content of hidden states at different layers in RoPE-trained vs. FoPE-trained models. If the theory is correct, RoPE models should show progressively more spectral mixing (power at non-assigned frequencies) in deeper layers, while FoPE models should show less degradation of the periodic extension property. This is technically straightforward (FFT of hidden state activations along the sequence dimension) and would provide direct evidence for the paper's central mechanistic claim.

2. Testing on production-scale models. All main experiments are on models ≤ 1.2B parameters. The paper's claims about "enhancing all RoPE-based open-source models" would be much stronger if demonstrated on at least one 7B model (Llama-2-7B, Mistral-7B). The fine-tuning experiment with SmolLM-1.7B is a step in this direction but is only 1.7B parameters and only tests fine-tuning, not pre-training or continual pre-training at scale. Given that FoPE can be applied to pre-trained RoPE models (Figure 4), testing on Llama-2-7B with FoPE-based continual pre-training would be a high-impact experiment that is conspicuously absent.

3. Longer extrapolation ratios. The experiments test 16× length generalization (512 → 8192). Production long-context models aim for 32×–128× extensions. Testing whether FoPE's benefits continue to hold at more extreme ratios would significantly strengthen the practical relevance.

4. Comparison to more recent position embedding variants. The paper compares to KERPLE (2022) and FIRE (2024), but not to more recent methods designed specifically for RoPE length generalization like NTK-aware scaling (used in many open-source long-context models), CLEX(Chen et al., 2024a), or Self-Extend(Jin et al., 2024) as independent baselines (they appear only through YARN as the representative extrapolation method, Figure 4). The field has moved quickly, and the claim that FoPE "significantly improvement length generalization compared to baselines" would be stronger if the baseline set included these more recent, competitive methods.

5. Ablation on the identity-matrix initialization of Fourier coefficients. The implementation adds an identity matrix to the randomly initialized coefficient matrices (self.sin_coef += torch.eye(...)). This ensures the diagonal corresponds to the original RoPE frequency with coefficient ~1, while off-diagonals are random. An ablation testing whether this identity initialization matters (vs. pure random initialization with no identity boost, or learned diagonal coefficients) would clarify whether the multi-frequency modeling is genuinely using the harmonic components or whether the model is primarily relying on the diagonal RoPE-equivalent terms with the harmonic components acting as a form of regularization.

6. Statistical significance and variance. The paper reports single numbers without error bars, confidence intervals, or standard deviations. For a paper making specific quantitative claims (e.g., "4× efficiency gain" in the prior sections, "~90–100% accuracy" for Passkey), the absence of uncertainty quantification is a weakness. Given the 1000-trial Passkey evaluation, binomial confidence intervals would be straightforward to compute and would strengthen the reported comparisons.

Do the experiments support each major claim from the Executive Summary?

Claim: "FoPE maintains stable Passkey Retrieval accuracy at arbitrary sequence lengths where RoPE drops to zero beyond 2× the training length." Strongly supported by Figure 1a for lengths up to 8192. "Arbitrary sequence lengths" is an overstatement—only lengths up to 16× training length are tested. The trend suggests stability would continue, but this is extrapolation of the results, not direct evidence.

Claim: "FoPE serves as an effective extrapolation method, outperforming YARN when applied to pre-trained RoPE models." Supported with qualifications by Figure 4 for the specific models and lengths tested. "Outperforming YARN" is true for the 60M–1.2B OLMo models but is not demonstrated on production-scale models or on models with YARN-optimized hyperparameters (the paper uses YARN's default settings; YARN performance can be sensitive to the interpolation rate and fine-tuning duration). The claim that FoPE "holds the potential to enhance all RoPE-based open-source models" is aspirational—consistent with the demonstrated mechanism but not empirically verified beyond the models tested.

Claim: "The periodic extension is actively damaged by linear layers, activation functions, and undertrained frequency components." Supported theoretically but empirically incomplete. The mathematical analysis (Lemma 3.1, Equations 7, 10) is rigorous and the toy experiments (Figures 3, 7; Table 2) provide proof-of-concept. However, the paper never directly measures Spectrum Damage in a full transformer, leaving a gap between the theoretical model and the empirical results. The fact that FoPE improves length generalization is consistent with the Spectrum Damage hypothesis but does not uniquely confirm it—other mechanisms (increased representational capacity, regularization from frozen coefficients, position-invariant pathways from zero-frequency padding) could also contribute.

Claim: "Length generalization is a system-level property, not an attention-level property." Well-supported as a conceptual reframing by the theoretical analysis showing that damage occurs in non-attention components (linear projections, FFN activations) and by the design of FoPE, which modifies attention to compensate for upstream damage. However, this claim is more of a theoretical contribution than an empirical one—the experiments don't directly test whether attention-level-only solutions are insufficient compared to system-level solutions (which would require comparing FoPE against an attention-level intervention that achieves similar results, which does not exist). The claim is better understood as a framework for thinking about the problem rather than an experimentally verified fact.

Key strengths of the experimental design

• Multi-scale validation: Testing at 60M, 180M, and 1.2B parameters with consistent results across scales provides some confidence that the mechanism is scale-invariant (though the range is limited).
• Task diversity: Evaluating on perplexity, Passkey Retrieval, summarization, and QA tasks tests different aspects of length generalization—from simple retrieval to complex reasoning—and strengthens the generality of the findings.
• Cross-distribution testing: Training on Gutenberg and evaluating on C4 (Figure 5) demonstrates that FoPE's benefits are not artifacts of the training data distribution.
• Clever ablation design: The toy experiments (Table 2, Figure 3) isolate specific mechanisms (undertrained frequency bias, harmonic generation) in controlled settings, providing causal evidence that complements the correlational evidence from the main experiments.
• Practical utility demonstration: The continual pre-training experiment (Figure 4) showing that FoPE can be applied to pre-trained RoPE models as an extrapolation method addresses the practical question of whether existing models can benefit without retraining from scratch.

Key weaknesses of the experimental design

• No direct measurement of the proposed mechanism. The paper's central theoretical contribution is the Spectrum Damage framework, but the paper never measures spectral content in actual transformer hidden states. This is the single largest gap between theory and experiment.
• Limited scale. All main experiments are at ≤ 1.2B parameters with ≤ 8192 sequence length. Production concerns involve models 10–100× larger and sequence lengths up to 128K.
• Missing baselines. Recent competitive methods (NTK-aware scaling, CLEX, Self-Extend) are not compared as independent baselines. The comparison to YARN is the only modern extrapolation method tested.
• No statistical uncertainty quantification. All results are reported as point estimates without confidence intervals or error bars, limiting the ability to assess whether differences (especially the small ones in Tables 4–6) are statistically meaningful.
• The best hyperparameters (σ, D) are found by grid search on the test set, which inflates the reported performance relative to what would be achievable with a proper validation-based selection. The paper states that σ and D are chosen per model scale, but the procedure for choosing them is not described as a formal hyperparameter optimization with held-out data.
• The "compute-optimal" complexity analysis from prior sections does not apply here. The paper doesn't report wall-clock time, memory usage, or training throughput for FoPE vs. RoPE, making the "negligible overhead" claim unsupported by measurements. Given the einsum operations, the overhead is likely small in practice, but empirical verification would strengthen the claim.

6. Limitations and Trade-offs

The Scale Gap: All Main Experiments Are Below 2B Parameters

The assumption or constraint. The paper's central claims about FoPE's effectiveness are demonstrated exclusively on models ranging from 60M to 1.2B parameters (OLMo family) during pre-training and continual pre-training, with one fine-tuning experiment at 1.7B (SmolLM). The paper does not test FoPE on models at the scale where length generalization is most commercially relevant—7B, 13B, 70B, or larger. The authors do not explicitly acknowledge this as a limitation in the main text, though the model scale is stated transparently in Section 5.1 ("We conduct experiments with the OLMo framework and consider different scale models having 60M, 180M, 1.2B parameters").

The consequence. The Spectrum Damage mechanisms that FoPE addresses—spectral leakage from linear layers, harmonic generation from activations, and undertrained frequency artifacts—all scale with model depth and width. The paper argues that larger models experience more Spectrum Damage (Section 5.5, where optimal σ increases from 0.3 to 0.6 with scale), but only three scale data points exist, spanning less than two orders of magnitude. At production scale (70B+ parameters with 80+ layers and 64+ heads of dimension 128), the dynamics may shift qualitatively:

• Spectral damage may saturate or compound nonlinearly. The paper assumes damage accumulates linearly with depth (Section 3.2), but deep networks operating near the edge of trainability may exhibit different spectral properties—gradient starvation in early layers, representational collapse in late layers, or phase transitions in attention patterns that fundamentally alter the frequency-domain structure.

• The optimal hyperparameters (σ, D) found by grid search may not extrapolate. Table 3 shows σ: 0.3 → 0.4 → 0.6 for 60M → 180M → 1.2B. Extrapolating linearly would predict σ ~5 for a 70B model—a value that would make the harmonic components dominate the Fourier Series, fundamentally changing FoPE's behavior from "dominant frequency plus weak harmonics" to something closer to noise. Whether the relationship is linear, logarithmic, or saturating at some asymptotic value is unknown.

• Training dynamics differ. Large models are typically trained with different optimization schedules, batch sizes, and data mixtures that may affect how strongly the hidden states develop clean frequency structure in early layers. A 60M model trained on 5B tokens is under-trained by Chinchilla standards; a 70B model trained on 2T tokens has seen each parameter updated differently.

What evidence exists in the paper. The scale-consistency argument rests on three data points (60M, 180M, 1.2B) showing qualitatively similar improvements (Figures 1, 5; Table 3). The fine-tuning experiment with SmolLM-1.7B (Table 1) provides one additional data point at a slightly larger scale. No experiment exceeds 1.7B parameters by a wide margin.

Mitigation status. The paper does not address this limitation explicitly. There is no discussion of how the hyperparameters might scale, no extrapolation formula, and no suggestion that the results have been validated on a larger model. The claim that FoPE "holds the potential to enhance all RoPE-based open-source models" (Section 5.3) is aspirational—it is consistent with the mechanism but not verified. A practitioner considering deploying FoPE in a production LLM would need to replicate the key experiments at their target scale, which requires pre-training or extensively fine-tuning a multi-billion-parameter model—a cost the paper does not estimate.

No Direct Empirical Measurement of Spectrum Damage

The assumption or constraint. The paper's entire theoretical framework—the diagnosis that RoPE fails due to Spectrum Damage, and that FoPE succeeds because it compensates for this damage—rests on a causal claim that is never directly measured in a working transformer. The paper proves mathematically that linear layers cause spectral mixing (Equation 7) and that activation functions generate harmonics (Lemma 3.1), and it demonstrates these effects in toy settings (Figure 3: a two-frequency signal passed through a single-layer MLP; Table 2: 20M-parameter models with controlled frequency assignments). However, it never performs a spectral analysis of actual hidden states in a trained OLMo model to show: (a) that deeper layers exhibit progressively more spectral mixing, (b) that dimensions assigned frequency ω_m in RoPE contain substantial energy at other frequencies, or (c) that FoPE-trained models show less degradation of the one-to-one frequency-to-dimension correspondence.

The consequence. Without this measurement, the causal chain from "transformer architecture causes Spectrum Damage" to "Spectrum Damage causes length generalization failure" to "FoPE alleviates Spectrum Damage and therefore fixes length generalization" remains inferred rather than established. Several alternative explanations for FoPE's benefits are equally consistent with the empirical results but require no Spectral Damage framework:

• Increased representational capacity. FoPE uses D harmonic frequencies per head dimension rather than one, effectively increasing the number of positional features the model can use. This additional capacity alone could improve length generalization regardless of whether Spectrum Damage exists—simply having more frequency samples per head provides finer-grained position discrimination, which might help extrapolation even if every dimension were spectrally pure.

• The zero-frequency pathway as a content-based attention mechanism. FoPE zero-pads dimensions beyond D with value 1 (cos(0))—this means a significant fraction of each attention head (for the 60M model with D=64 and head_dim=128, this is 50% of dimensions) operates without any position-dependent rotation at all. These dimensions perform position-invariant content-based attention, which naturally generalizes to any sequence length. This mechanism alone—present in FoPE but absent in RoPE—could explain much of the improvement without invoking Spectral Damage.

• Regularization from frozen random coefficients. The Fourier coefficient matrices are initialized randomly and frozen during training. This introduces a form of structural regularization: the model is forced to work with position embeddings that are randomized in a specific way, which might prevent overfitting to the training sequence length's specific positional patterns. This is a regularization effect, not a spectral repair effect.

The paper does not disentangle these mechanisms. The ablation in Figure 6a shows that removing the Fourier Series ("FoPE w/o FS") degrades length generalization—but this ablation removes all harmonic components and the multi-frequency representation simultaneously. It cannot distinguish whether the benefit comes from modeling actual spectral contamination versus from the additional capacity or regularization.

What evidence exists in the paper. Only indirect evidence:

• The toy model with a single-layer MLP (Figure 3, Section 5.6) demonstrates that activation functions generate harmonics in a minimal setting. This proves the mechanism exists in principle but not that it is quantitatively important in a full transformer, where many mitigating factors could apply (residual connections might preserve some spectral purity; LayerNorm might rescale hidden states in ways that affect frequency content; attention itself might learn to separate frequencies even after upstream mixing).

• The undertrained frequency analysis (Figure 7, Table 2) provides cleaner evidence: it directly measures activation patterns in Llama2-7B and demonstrates causal impact through the QK Norm and RoPE-A experiments. This is the stronger half of the paper's mechanistic evidence. The Spectrum Damage from linear layers and activations receives no comparable direct validation.

Mitigation status. The paper does not acknowledge this gap. The theoretical framework is presented as if it directly explains the empirical results, without noting that the intervening measurement—direct spectral analysis of hidden states—is absent. This is the most significant methodological weakness in the paper because it leaves the central mechanistic claim underdetermined by the evidence.

The Hyperparameter Selection Protocol Overfits to Test Performance

The assumption or constraint. The two key hyperparameters of FoPE—the Fourier coefficient variance σ and the number of harmonic frequencies D—are selected by grid search on the test set for each model scale. For the 60M ablation study (Section 5.5, Figures 6b and 6c), σ and D are swept across ranges {0.0, 0.1, 0.2, 0.3, 0.4, 0.5} and {16, 32, 64, 128} respectively, and the values yielding the best Perplexity Ratio (for σ) and Passkey Accuracy (for D) are selected. The optimal values—σ = 0.3, D = 64 for 60M; σ = 0.4, D = 128 for 180M; σ = 0.6, D = 128 for 1.2B—are reported in Table 3 and used for all subsequent experiments at those scales. The paper does not describe a separate validation set for hyperparameter selection, nor does it use cross-validation to estimate the variance of the selected values.

The consequence. This protocol means the reported performance of FoPE is optimistically biased—the hyperparameters have been tuned specifically to maximize performance on the same data used for evaluation. The magnitude of this bias is unknown because the paper does not report how sensitive performance is to σ and D in the neighborhood of the selected values. From Figure 6b, the Perplexity Ratio at σ = 0.3 is approximately 8.5, versus ~9.5 at σ = 0.2 and ~10 at σ = 0.1. This means that if the "true" optimal σ for the 60M model on unseen data were 0.2 (due to sampling noise in the test set), the reported performance would overstate the achievable gain by roughly 1 PPL ratio point. For the Passkey metric (Figure 6c), the gap between D=64 (~70% accuracy) and D=32 (~55%) is 15 percentage points—large enough that selecting the wrong D based on a noisy test set could substantially change the performance claims.

For the comparison with RoPE and other baselines, this matters because baseline methods are not given the same hyperparameter optimization privilege. RoPE has no corresponding σ or D to tune—it is evaluated out-of-the-box. ALiBi has no tunable hyperparameters. YARN has tunable parameters (interpolation rate, scale factors) but the paper does not describe a grid search procedure for YARN; it appears to use default settings. If FoPE's hyperparameters were selected on a proper validation set and evaluated on a held-out test set, the gap between FoPE and the baselines would likely shrink, though the paper provides no way to estimate by how much.

What evidence exists in the paper. The paper's experimental description (Section 5.5) makes the protocol transparent: "By grid searching σ from 0 to 0.5, we find that setting σ = 0.3 for 60M model obtain the best perplexity, especially for longer context." The evaluation metrics are Perplexity Ratio (on C4/Books) and Passkey Accuracy—the same metrics used to compare FoPE against baselines. There is no mention of a separate validation split for hyperparameter selection, nor any cross-validation procedure.

Mitigation status. The paper partially addresses this through the scale-consistency argument: the fact that the optimal σ and D increase with model scale in a monotonic way (0.3→0.4→0.6 for σ, 64→128→128 for D) suggests that these hyperparameters capture a genuine architectural property rather than test-set noise. However, this is a plausibility argument, not a statistical correction. A proper mitigation would involve either (a) separate validation/test splits where hyperparameters are selected on validation only, (b) cross-validation with variance estimates, or (c) a sensitivity analysis showing that FoPE outperforms RoPE across a wide range of σ and D values, not just at the selected optimum. The paper does report (Section 5.5) that "FoPE consistently outperforms RoPE across all hyperparameter settings," which addresses the concern partially—but this claim is made without showing the full grid results compared against RoPE's performance level, making it difficult to verify.

Only One Extrapolation Method (YARN) Tested as a Baseline for Continual Pre-Training

The assumption or constraint. When evaluating FoPE as an extrapolation method for extending pre-trained models (Section 5.3, Figure 4), the paper compares FoPE against only YARN(Peng et al., 2023). The field has produced numerous RoPE extension methods with different design philosophies and performance characteristics, including NTK-aware scaling (which modifies RoPE's base frequency rather than position indices), CLEX(Chen et al., 2024a, which learns a continuous scaling function), Self-Extend(Jin et al., 2024, which modifies the attention mask to limit self-attention range), and LongRoPE and related methods that combine frequency interpolation with targeted fine-tuning. These methods are mentioned in the Related Work (Section 6) but are not implemented as baselines in the experiments.

The consequence. The claim that "FoPE outperforms YARN in length extrapolation for both RoPE-based and FoPE-based models" (Section 5.3) is well-supported for the YARN comparison specifically, but the generalization to "FoPE is an effective extrapolation method" relative to the state of the art is overstated. Different extension methods have different strengths: NTK-aware scaling is training-free and can be applied at inference time without any fine-tuning; CLEX can adapt to arbitrary target lengths; Self-Extend requires no positional embedding modification at all. Without comparisons to these methods, a practitioner cannot assess whether FoPE's gains over YARN represent genuine progress over the best available techniques or merely over a specific baseline that happens to be weaker in this experimental setting.

This limitation is particularly consequential because Figure 4 shows YARN applied to a RoPE model ("RoPE-512 + YARN-1024") performing relatively poorly on Passkey Retrieval at longer lengths—declining substantially at 4096 and 8192. This is a surprising result given YARN's strong performance in the literature. Possible explanations include: (a) YARN was not optimized for this specific model scale and training regime (the paper does not describe hyperparameter tuning for YARN), (b) the 1B-token continual pre-training budget is insufficient for YARN to converge (Appendix A.3 notes YARN is "prone to overfitting with excessive fine-tuning steps"), or (c) the evaluation protocol is more challenging than typical YARN evaluations. Any of these explanations would make YARN a weaker-than-representative baseline, inflating the apparent advantage of FoPE.

What evidence exists in the paper. Figure 4 presents two subfigures comparing FoPE-512 and RoPE-512 base models under various extension scenarios. The only external extrapolation method tested is YARN. The paper acknowledges one limitation of YARN in Appendix A.3: "This is partly due to YARN being prone to overfitting with excessive fine-tuning steps, a limitation not observed in FoPE." The fine-tuning protocol (Section 5.3) uses ~1B tokens of continual pre-training for both FoPE and YARN, but does not describe hyperparameter optimization for YARN. The evaluation checkpoint selection differs between methods (Appendix A.3: "For fine-tuning, we save checkpoints every 100 steps and report the best result for each method")—this actually benefits YARN relative to FoPE since YARN is more prone to overfitting, but the selection is applied to both methods, so the comparison remains fair in this respect.

Mitigation status. The paper partially addresses this by testing FoPE as both a base position embedding (pre-training from scratch) and an extrapolation method (applied to pre-trained models). The pre-training experiments (Figures 1, 5) compare FoPE against five position embedding baselines (RoPE, NoPE, ALiBi, KERPLE, FIRE), which is a reasonable comparison set for the intrinsic position embedding quality. The limitation applies specifically to the extrapolation claim—that FoPE can enhance existing RoPE models—where only YARN is tested. The paper does not acknowledge this as a limitation or suggest comparisons to other extension methods as future work. A practitioner deciding whether to use FoPE versus, say, a training-free NTK-aware scaling approach for their RoPE-based model has no basis for comparison from this paper.

The Hyperparameter-to-Scale Relationship Is Based on Only Three Data Points

The assumption or constraint. The paper proposes that FoPE's key hyperparameters—σ (Fourier coefficient variance) and D (number of harmonic frequencies)—should scale with model size, and provides values for three model scales: 60M, 180M, and 1.2B (Table 3). The paper states (Section 5.5): "The best σ implies the estimated strength of Spectrum Damage of the 60M model, and the estimation may become larger as the models' parameter scale increases." and "The best D is the estimated number of strong enough noisy components of each model, and this number may become larger as the parameter scale increases."

The consequence. A practitioner wanting to apply FoPE to a model of a different scale—say, 7B, 13B, or 70B parameters—must guess the appropriate σ and D. The three data points are insufficient to establish a functional relationship between model scale and these hyperparameters. The observed trend (σ: 0.3, 0.4, 0.6; D: 64, 128, 128) is consistent with monotonic increase but does not constrain the functional form:

• If the relationship is logarithmic, a 7B model might need σ ≈ 0.8 and D ≈ 256, while a 70B model might need σ ≈ 0.95 and D ≈ 512 (impractically large, exceeding typical head dimensions).
• If the relationship is saturating, σ and D might asymptote around 0.6–0.7 and 128–256, meaning the 1.2B values would already be close to the asymptotic values and would work for larger models with minor adjustments.
• If the relationship depends on architecture details not captured by parameter count—such as the ratio of head dimension to number of heads, the MLP expansion ratio, or the activation function type—then parameter count alone is insufficient to select hyperparameters, and a more complex model is needed.

Without a predictive model (or at minimum, more data points), deploying FoPE at a new scale requires an expensive hyperparameter sweep—the same kind of grid search conducted in Section 5.5 but at much higher computational cost for large models.

Furthermore, the paper provides these values after selecting them based on test-set performance (see the previous limitation on hyperparameter selection). This means the reported relationship (σ increases from 0.3 to 0.6; D from 64 to 128) is partially conflated with test-set overfitting—the "optimal" values for each scale may reflect noise in the specific test set as much as genuine architectural scaling properties.

What evidence exists in the paper. Table 3 reports the three data points. Section 5.5 discusses the scaling trends qualitatively. Figure 6b (σ sweep) and Figure 6c (D sweep) show the hyperparameter sensitivity at the 60M scale only—no equivalent sweeps are shown for 180M or 1.2B. The paper does not report how the 180M and 1.2B optimal values were determined—whether through similar grid searches (which would be computationally expensive) or through heuristic extrapolation from the 60M results.

Mitigation status. The paper does not address this limitation. There is no proposed scaling law for σ and D, no recommendation for practitioners applying FoPE to new model scales, and no acknowledgment that the three-point trend is insufficient for reliable extrapolation. This is both a practical limitation (it makes FoPE harder to adopt) and a theoretical limitation (it leaves unclear whether Spectrum Damage follows a predictable scaling law, as the paper's framework would imply it should).

The Reported Efficiency Advantage Does Not Account for Hyperparameter Search Cost

The assumption or constraint. The paper claims that FoPE "adds negligible memory and computation overhead compared to RoPE" (Appendix B) and presents this as a practical advantage. This claim is accurate for the inference-time and training-time FLOPs of a single run: the Fourier coefficient matrices are small (2 MB for the 1.2B model), frozen (no gradients), and the einsum operations are cheap relative to attention. However, this accounting ignores the cost of finding the hyperparameters that make FoPE work. The grid searches described in Section 5.5—sweeping σ across 6 values and D across 4 values, training a 60M model on 5B tokens for each configuration—represent a substantial computational investment (roughly 24 configurations × 5B tokens = 120B tokens of total training, compared to the 10B-token budget of the main experiments). At the 1.2B scale, an equivalent grid search would be computationally prohibitive, and the paper does not describe how the 180M and 1.2B hyperparameters were selected.

The consequence. The "negligible overhead" framing is misleading from a total-cost-of-adoption perspective. For a practitioner deploying FoPE on a new model architecture or scale, the effective cost includes:

1. Hyperparameter selection: Training multiple models (or performing multiple fine-tuning runs) to find appropriate σ and D for the target architecture. Unlike RoPE, which has no position-embedding hyperparameters to tune, FoPE introduces two new degrees of freedom that are sensitive enough to meaningfully affect performance (Figure 6b shows a ~2 PPL ratio point swing across σ values; Figure 6c shows a ~25 percentage point swing in Passkey Accuracy across D values).

2. Architecture coupling: Because the optimal σ and D depend on model depth, width, and activation functions (per the Spectrum Damage framework), these hyperparameters cannot be transferred from one model architecture to another. Moving from OLMo to LLaMA (as the fine-tuning experiments do) may require re-tuning σ and D, but the paper does not describe this process for the SmolLM-1.7B experiment—the σ and D values for that experiment are not reported.

3. Scale extrapolation risk: If the practitioner's target scale differs from the three tested scales, the cost of hyperparameter selection may include training runs that fail (choose poor σ/D values and produce a model that underperforms RoPE), which would not be recoverable without re-training. The paper does not report how FoPE performs with poorly chosen hyperparameters—the "FoPE consistently outperforms RoPE across all hyperparameter settings" claim in Section 5.5 is made without showing data for the worst-performing configurations.

What evidence exists in the paper. Appendix B provides the implementation details and argues that the matrices add negligible per-step overhead. Section 5.5 describes the grid searches for the 60M model and reports the results. The paper does not report the computational cost of hyperparameter selection, does not discuss the transferability of hyperparameters across scales or architectures, and does not provide guidance for selecting σ and D without grid search.

Mitigation status. Not addressed. The paper presents FoPE as having "negligible memory and computation overhead compared to RoPE" without qualifying that this statement applies only after hyperparameters have been selected. For a researcher seeking to replicate or extend the work, the hidden cost of hyperparameter selection is a practical barrier that the paper does not acknowledge. The suggestion in Section 5.5 that "the estimation may become larger as the models' parameter scale increases" is vague and not actionable—it does not provide a formula, a heuristic, or even a recommended range for a given scale.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper effects a reframing of the length generalization problem from an attention-level challenge to a system-level signal processing challenge. Prior to this work, the dominant assumption in the field was that position embeddings fail to generalize because the attention mechanism itself mishandles position information—that the rotation scheme, interpolation strategy, or attention mask needed refinement. The paper's DSP analysis undermines this assumption at its foundation: by the time a hidden state reaches an attention layer, the frequency-domain structure that RoPE assumes has already been corrupted by linear projections and activation functions operating between attention blocks. This is not an incremental improvement over prior methods—it is a different diagnosis that implies a different class of solutions.

The scale of this reframing is comparable to the shift in computer vision when researchers recognized that batch normalization's benefits came not from reducing internal covariate shift (the original motivation) but from smoothing the optimization landscape (Santurkar et al., 2018). In both cases, a widely-used technique was understood through an incorrect mechanistic model, and correcting that model opened new solution directions. This paper similarly corrects the mechanistic model of RoPE: the problem is not that RoPE's periodicity is unused, but that it is actively destroyed by non-attention components. Solutions that operate only within attention (YARN, NTK-aware scaling, Self-Extend) are treating a symptom downstream from the root cause.

The practical consequence of this reframing is a reordering of research priorities. If the paper's diagnosis is correct, then:

• Improving verifier or interpolation strategies for RoPE becomes less attractive. These methods address how rotation angles are assigned to positions, but they cannot repair the fact that the vectors being rotated no longer carry clean single-frequency information. The residual benefit from better interpolation is bounded by the spectral purity of the signal—which, the paper argues, is fundamentally limited by the architecture.

• Architectural modifications that protect spectral structure become more attractive. This includes not only attention-level modifications like FoPE but also potential changes to how linear layers and activations process position-bearing signals. For example, one could imagine frequency-aware normalization schemes, gating mechanisms that preserve spectral purity in selected dimensions, or separate pathways for position-dependent and position-independent information.

• Understanding transformers through signal processing becomes a high-priority research direction. The paper demonstrates that DSP theory provides precise, falsifiable predictions about transformer behavior (e.g., Lemma 3.1 predicts specific harmonic frequencies; Equation 10 predicts which dimensions will be dominated by truncation artifacts). This suggests that other seemingly mysterious transformer behaviors might yield to similar frequency-domain analysis.

The paper also reconciles a puzzling contradiction in the literature. RoPE is mathematically periodic and should extrapolate—this is straightforward from its definition. Yet every practitioner knows that RoPE-based models collapse at roughly 2× training length. Prior work treated this as a mystery requiring new empirical fixes; the paper explains it as an inevitable consequence of the transformer architecture. The contradiction was never between RoPE's theory and empirical reality—it was between RoPE's theory and the system it was embedded in. By widening the analysis boundary to include the full transformer, the contradiction dissolves.

Finally, the paper establishes a new diagnostic vocabulary—Spectrum Damage, Spectrum Leakage, Spectrum Distortion, floor frequency, undertrained components—that enables researchers to reason precisely about how position information degrades through a network. Before this paper, there was no shared language for discussing why one position embedding generalizes better than another beyond empirical benchmarks. Now, a researcher can ask: "Does this method suffer more from Spectrum Leakage or from undertrained component artifacts? Which specific spectral contaminations is it robust to?" This vocabulary makes the problem tractable in a way it was not before.

Follow-Up Research This Work Enables

Direct spectral analysis of hidden states in trained transformers to validate the Spectrum Damage hypothesis. The paper's central causal claim—that linear layers and activations cause progressive spectral mixing that destroys RoPE's periodic extension—is supported by mathematical proofs and toy experiments but never directly measured in a full transformer. A strong follow-up would take a trained RoPE-based model (e.g., Llama-2-7B), feed a sequence of varying position indices through it, and compute the FFT of hidden state activations along the position dimension at each layer. The Spectrum Damage hypothesis predicts: (a) in early layers, each dimension's spectrum is dominated by its assigned RoPE frequency with minimal leakage; (b) this spectral purity degrades with depth, with deeper layers showing substantial power at non-assigned frequencies; (c) the amount of spectral mixing correlates with the number of linear layers and activation functions traversed, not just the number of attention layers. If these predictions are confirmed, the paper's framework receives strong empirical grounding. If spectral purity remains high in deep layers (contradicting the hypothesis), then FoPE's benefits must arise from mechanisms other than Spectral Damage repair—which would be an equally important finding, redirecting future research away from spectral robustness and toward the alternative mechanisms (increased capacity, zero-frequency pathways, regularization).

Scaling FoPE to production-scale models to determine whether the benefits persist at 7B+ parameters. All main experiments are at ≤1.7B parameters. The paper claims FoPE "holds the potential to enhance all RoPE-based open-source models," but the scale gap is two orders of magnitude to production models. A high-impact follow-up would apply FoPE to Llama-2-7B or Mistral-7B using the continual pre-training protocol from Section 5.3: take the pre-trained RoPE checkpoint, replace the position embedding with FoPE (using σ and D extrapolated from Table 3 or found via a modest grid search), and fine-tune on ~1–5B tokens of long-context data. Key measurements: (a) Passkey Retrieval accuracy at 16K, 32K, 64K when trained at 4K context; (b) perplexity on long-document datasets (Books3, PG-19) as context extends; (c) downstream long-context benchmarks like LongBench or L-Eval. The experiment would test two hypotheses: whether FoPE's benefits scale with model size (as the trend from 60M→1.2B suggests) and whether the hyperparameters σ and D follow a predictable scaling relationship that can be extrapolated to larger models. A negative result—FoPE providing negligible benefit at 7B—would indicate that Spectrum Damage either saturates or is compensated for by overparameterization in large models, fundamentally limiting FoPE's practical relevance.

Disentangling the three mechanisms that could explain FoPE's success: Spectral Damage repair, increased positional capacity, and zero-frequency regularization. The paper attributes FoPE's gains to Spectral Damage repair, but FoPE's design introduces three simultaneous changes: (1) multi-frequency representation per dimension (which could model actual spectral contamination OR simply provide more positional features), (2) zero-frequency padding for dimensions beyond D (which provides position-invariant processing pathways), and (3) frozen random Fourier coefficients (which act as structural regularization). A well-designed ablation study could isolate these. Proposed conditions: (a) FoPE with learned (not frozen) Fourier coefficients—if performance degrades vs. frozen, regularization matters; if unchanged, capacity matters more; (b) RoPE with additional frequency components per dimension (achieved by increasing head dimension and assigning multiple RoPE frequencies per pair of dimensions, concatenating rather than summing) but no Fourier mixing—this is a "capacity-only" baseline that has more positional features without multi-frequency summing; (c) FoPE with zero-frequency padding only (no Fourier Series, just RoPE on the first D dimensions plus zero padding for the rest)—this isolates the zero-frequency pathway contribution. Comparing these against full FoPE and standard RoPE on both perplexity (C4, long documents) and Passkey Retrieval would reveal which mechanism is driving the gains and under what conditions. If the "capacity-only" baseline matches FoPE, the Spectrum Damage narrative is weaker than alternatives; if the zero-padding baseline matches FoPE, the mechanism is primarily about guaranteeing position-invariant pathways rather than spectral modeling.

Frequency-domain analysis of different activation functions and their harmonic generation properties. Lemma 3.1 proves that any non-linear activation generates harmonics, but the distribution and magnitude of these harmonics depend on the specific activation function. GELU (used in OLMo), SiLU/SwiGLU (used in LLaMA), and ReLU (used in older architectures) have different Taylor expansions and therefore different harmonic spectra. A systematic study would: (a) analytically derive the harmonic power spectrum for each common activation given multi-frequency inputs with realistic frequency distributions (matching RoPE's 1/θ^(2m/M) sampling); (b) train small transformers (e.g., 20M parameters) with each activation function, measuring length generalization with both RoPE and FoPE; (c) measure the actual spectral content of hidden states at each layer for each activation function via FFT. The prediction is that activations with stronger higher-order Taylor terms (GELU, SiLU) produce more high-frequency harmonics and therefore cause more severe Spectrum Distortion, making FoPE's multi-frequency modeling more beneficial for these activations than for ReLU. If confirmed, this would provide a theoretical basis for activation function selection when length generalization is a priority, and would reveal whether architectural choices made for training stability (GELU over ReLU) have hidden costs for length generalization.

Training a lightweight "difficulty estimator" for position embedding spectral purity to enable adaptive strategies. The paper demonstrates that FoPE works by compensating for known spectral contamination patterns, but it uses fixed hyperparameters (σ, D) for all sequences and all layers. In reality, the amount of spectral contamination likely varies: short sequences (within training length) experience less extrapolation stress and may not need multi-frequency robustness; different layers may experience different degrees of leakage depending on their position in the network and the attention patterns they've learned; different input types (code vs. prose vs. dialogue) may have different inherent periodicities that interact differently with the position embedding. A follow-up could train a small predictor network that takes a hidden state's per-dimension frequency spectrum (computed via FFT along the sequence dimension) and predicts whether that layer, for that input, would benefit from multi-frequency modeling. This "spectral health monitor" could gate FoPE's Fourier Series on a per-layer, per-sequence basis: layers with clean spectra use standard RoPE (saving the small einsum overhead), while layers with contaminated spectra use FoPE. The training signal would come from a contrastive setup: for a given sequence, measure the attention pattern accuracy (vs. a long-context ground truth) with and without multi-frequency modeling per layer, and train the predictor to identify layers where the multi-frequency modeling provides the most benefit. This extends the paper's framework from a static architectural fix to an adaptive, input-dependent strategy.

Testing whether FoPE's benefits extend to non-text modalities and non-autoregressive architectures. The paper's Spectrum Damage framework is architecture-dependent (it relies on the specific arrangement of linear layers, activations, and attention in transformers) but modality-independent—Spectrum Leakage and Distortion should occur in any transformer that processes sequential data with positional encodings. Testing FoPE on: (a) vision transformers (ViT) with 2D positional encodings on ImageNet classification at varying resolutions; (b) audio transformers (e.g., Whisper-style encoders) with 1D positional encodings on long audio clips; (c) protein sequence models (e.g., ESM-style) with biological sequence positional encodings; would test the generality of the framework. If FoPE provides no benefit in these domains, it suggests that the Spectrum Damage mechanism is specific to the statistical properties of natural language (perhaps due to its particular long-range dependency structure or the specific training objectives used in language modeling). If FoPE provides consistent benefits across modalities, the framework generalizes and position embedding spectral robustness becomes a cross-domain design principle. A particularly incisive test: comparing FoPE on text-to-image diffusion transformers (which process text and image tokens in a unified sequence) versus pure text transformers—the multi-modal setting introduces additional spectral complexity from the interaction of two very different token types with a shared positional encoding.

Practical Applications and Downstream Use Cases

Cost-efficient fine-tuning of open-source models for long-context applications. Organizations deploying open-source models (Llama-3, Mistral, Qwen) for document processing, legal contract analysis, or repository-level code understanding typically face a choice: use the base model's context window (often 4K–8K), apply a RoPE extension method (YARN, NTK-aware scaling) with uncertain quality, or fine-tune on long-context data at substantial computational cost. The paper's continual pre-training result (Figure 4: RoPE-512 + FoPE-1024 matches or exceeds RoPE-512 + YARN-1024) suggests that FoPE provides a drop-in replacement for the position embedding during the extension fine-tuning phase, requiring no additional hyperparameter tuning beyond what the paper already provides. For a team fine-tuning Llama-3-8B from 8K to 32K context, replacing RoPE with FoPE during the fine-tuning stage (using σ ~0.6–0.8 and D ~128–256, extrapolating from Table 3) could yield better Passkey Retrieval accuracy at 32K and more stable perplexity on long-document corpora compared to standard YARN-based extension, at negligible additional computational cost (the FoPE weight matrices are ~4 MB for an 8B model with 32 heads). The benefit is not theoretical—it translates to fewer missed facts in long legal documents and more reliable code generation across file boundaries.

On-device and edge deployment with variable-length inputs. A persistent problem in deploying LLMs on edge devices (phones, laptops, IoT) is that the model's context window must be fixed at compile time for efficient inference (kv-cache allocation, memory planning), but real-world inputs vary dramatically in length. A voice assistant trained with 2K context that receives a 10K-token transcription of a meeting will either truncate (losing information) or attempt extrapolation (risking hallucination). The paper's pre-training results (Figure 1a: FoPE maintains ~90–100% Passkey accuracy from 512 to 8192 with a 512-trained model) suggest that a FoPE-based model could be deployed with a hardware-optimized context window (e.g., 512 or 1024 tokens for memory-constrained devices) while still processing much longer inputs acceptably when they occur, without the catastrophic collapse that RoPE exhibits. This has immediate implications for on-device models like Apple Intelligence, Google's Gemini Nano, or Microsoft's Phi-series, where inference memory is the binding constraint and graceful degradation (rather than hard failure) on long inputs is a practical requirement. The benefit is not that the model achieves cloud-quality long-context performance—it won't—but that it fails gracefully instead of catastrophically, which is often sufficient for user-facing applications where occasional degradation is preferable to out-of-memory crashes or silently wrong answers.

Data curation for self-improving long-context models. When using LLMs to generate synthetic long-context training data (for self-play, distillation, or augmentation), the quality of the generated data depends critically on whether the model maintains coherence across long sequences. A common pipeline: take a strong but short-context model, prompt it to generate long-form content (documentation, narratives, multi-turn dialogues), and use the output to fine-tune a longer-context student model. If the teacher model's coherence degrades with length (as RoPE models do, per Figure 1), the synthetic data contains artifacts that the student learns. A FoPE-based teacher would produce more coherent long-form outputs, improving the quality of the synthetic data pipeline. The specific benefit: for a team using a 7B model to generate 32K-token training examples for a 70B long-context model, switching the 7B model's position embedding from RoPE to FoPE (via the continual pre-training protocol from Section 5.3) could measurably improve the 70B student's performance on long-context benchmarks, without changing the data generation pipeline or the student's architecture. This is a practical and testable claim: compare a student trained on RoPE-generated vs. FoPE-generated long-context data, measuring LongBench or Scrolls performance.