Fractal Patterns May Illuminate the Success of Next-Token Prediction

🎯 Pitch

A tiny new metric based on the fractal roughness of a language model’s predictions predicts its reasoning, math, and coding abilities far better than perplexity alone. It turns out the same word-level burstiness you see in a paragraph is mirrored across entire documents, and capturing this hidden self-similarity may be the secret to why next-token prediction produces anything that looks like general intelligence.

1. Executive Summary

This paper studies the fractal structure of language to provide a formal quantification of statistical properties that may illuminate why next-token prediction in LLMs yields remarkable capabilities. Using PaLM2-L and other models from the PaLM and T5 families on The Pile validation split, the authors establish that language is self-similar—exhibiting the same burstiness and complexity patterns at all levels of granularity, from clauses to entire documents (quantified via a Hölder exponent of S = 0.59 ± 0.08)—and long-range dependent (LRD) (quantified via a Hurst parameter of H = 0.70 ± 0.09, where H > 0.5 implies predictability over time). They further show that a combined metric of median Hurst exponent and bits-per-byte (denoted H_B) predicts downstream performance substantially better than BPB alone, increasing adjusted R² from ~0.65 to over 0.86 across 12 models on benchmarks including MMLU, BIG-Bench Hard, and GSM8K, establishing that these tiny fractal-parameter variations capture a dimension of model quality orthogonal to perplexity.

2. Context and Motivation

The Core Problem: Why Does Next-Token Prediction Produce Anything Remotely Intelligent?

The fundamental question this paper tackles is both deceptively simple and deeply ambitious: why does the seemingly trivial training objective of predicting the next token yield language models with remarkable capabilities? The authors open by citing the example of Gemini and GPT-4, models that demonstrate quantitative reasoning, summarization, and even coding—abilities that, as they note, have led some researchers to "ponder if there was more to intelligence than 'on-the-fly improvisation'" (Section 1, citing Bubeck et al., 2023). The training objective itself provides no explicit reward for understanding, reasoning, or maintaining coherence over long spans—it simply maximizes the probability of the next token given all previous tokens. Yet from this narrow signal, something that looks broadly competent emerges.

This question is important for reasons that go beyond intellectual curiosity:

Theoretical significance for AI: If we can understand why next-token prediction works so well, we gain insight into the relationship between prediction and intelligence. The authors explicitly frame their contribution as offering "a fresh perspective on language and the mechanisms underlying the success of LLMs" (Abstract). This is not merely a descriptive exercise in characterizing language statistics—it is an attempt to explain a phenomenon that has surprised even its creators.
Practical implications for model design: If the fractal structure of language is what enables next-token prediction to capture patterns at multiple granularities, then explicitly exploiting this structure—for instance, through architectural choices like parameter sharing or through training strategies like context-length selection—could yield more efficient models. The paper gestures at this in the future work section, noting that "exploiting self-similarity more directly in LLMs could lead to further optimizations" (Section 4).
Understanding the limits of current approaches: If certain domains lack the fractal properties that make next-token prediction effective, those domains may be inherently harder for LLMs to master. The paper's finding that DM-Mathematics—a dataset of independent math questions—has H ≈ 0.5 (no long-range dependence) and correspondingly weaker structure is a concrete example of this principle.

The Gap: Self-Similarity and Long-Range Dependence Have Been Suspected but Never Formally Quantified

The idea that language exhibits self-similarity and long-range dependence is not new. The paper cites a substantial body of prior work that has argued for these properties, but it identifies a critical gap: no prior study has provided a rigorous, distribution-aware quantification of these fractal parameters. Each previous attempt made simplifying assumptions that—while understandable given computational constraints of their era—fundamentally limit what can be concluded.

Prior Approach 1: Word-Level Proxies

Several studies attempted to detect fractal structure in language by replacing words with scalar proxies and analyzing the resulting time series. Ausloos (2012) modeled texts by substituting each word with its length, then studied the fractal behavior of the resulting sequence of integers. This approach is problematic for a reason the paper explicitly articulates (Section 4, citing Futrell and Hahn, 2022): "replacing a word with its length is invalid because it is not translation-independent"—meaning one could map every word to an arbitrary token, including tokens of equal length, and the analysis would change fundamentally even though the underlying linguistic structure is identical. More broadly, word length and word frequency are crude proxies that discard almost all information about semantic content, syntactic structure, and discourse coherence—precisely the features that make language what it is.

Prior Approach 2: Single-Word Recurrence

Najafi and Darooneh (2015) defined a fractal dimension for individual words by examining the recurrence pattern of a single, predetermined word as a binary time series (1 if the word appears, 0 otherwise). Altmann et al. (2012) used a similar approach. While more principled than word-length substitution, this method suffers from a different limitation: it "only applies to individual words and cannot model higher-level clauses" (Section 4). The authors give a concrete example: the word "time" in "once upon a time" versus "time" in "space and time." A binary recurrence analysis treats these identically, yet they occupy completely different syntactic and semantic roles. Moreover, analyzing one word at a time cannot capture the joint statistical structure that makes language coherent—it is like trying to understand a symphony by studying the occurrence pattern of a single note.

Prior Approach 3: ASCII Encoding

Kokol and Podgorelec (2000) estimated long-range dependence in natural language using ASCII encoding and found that its LRD was "close to that of pure noise." The authors note this surprising result and suggest it was an artifact of the encoding choice. Converting text to ASCII byte sequences introduces structure that is unrelated to linguistic content—for example, character-level encoding is dominated by the statistical properties of spelling conventions rather than the higher-level patterns that make text meaningful. This negative result actually underscores the paper's motivation: how you represent language for fractal analysis matters enormously, and prior attempts used representations that were too shallow to capture the relevant structure.

Prior Approach 4: Distribution-Agnostic Summaries

The paper also directs us toward a broader class of prior work that studied statistical regularities in language without modeling the full joint distribution. Zipf's law (word frequency follows a power law) and Heap's law (the number of unique words grows as a power law of document length) are classic examples. The authors acknowledge these contributions but point out a fundamental limitation: "both Zipf's and Heap's laws are invariant to the semantic ordering of text, so they do not capture important aspects, such as long-range dependence" (Section 4). You can shuffle the words in a document and Zipf's law will hold unchanged; the fractal parameters the paper studies, by contrast, are sensitive to ordering and dependence structure.

The Menzerath-Altmann law—which states that as the size of a language construct increases, the size of its constituents decreases, and this happens at all scales—is closer in spirit to the self-similarity the paper investigates. However, prior work on this law operated at the level of surface linguistic units (phonemes, morphemes, words) rather than on the information-theoretic content of language.

Every prior approach the paper discusses shares a common weakness: they do not model the joint probability distribution of language. They replace text with scalars (word lengths, binary indicators, ASCII codes), analyze marginal statistics (frequency distributions), or focus on individual surface features. These simplifications were necessary in an era before large language models could produce well-calibrated token-level probability estimates, but they fundamentally cannot capture the structure the paper is interested in. Fractal parameters describe how the intrinsic complexity of a process behaves across scales; to measure intrinsic complexity, you need something that approximates the true information content of the text.

How This Paper Positions Itself

The paper's key methodological innovation—and what distinguishes it from all prior work—is its use of large language models to compute the information-theoretic description of language before applying fractal analysis. Rather than analyzing text directly (or through crude proxies), the authors first pass each document through PaLM2-L to obtain the conditional probability of each token given its full prefix. By the chain rule of information theory, the negative log-probability $z_t = -\log p(w_t \mid w_{[t-1]})$ represents the number of bits required to encode token $w_t$ given all prior context—this is "an intrinsic, irreducible description of language" that captures "the minimum compute overhead to comprehend/decode it" (Section 1, citing Futrell and Hahn, 2022).

This is a fundamentally different object than any prior analysis has studied. The sequence of token surprisals $z_t$ reflects not just what words appear, but how unexpected each word is given everything that came before. A highly predictable word in context contributes few bits; a surprising word contributes many. The resulting time series encodes syntactic structure (function words tend to be low-surprisal because they are highly predictable from local context), semantic content (novel content words in new topics produce surprisal spikes), discourse structure (transitions between topics produce clusters of high-surprisal tokens), and more—all without the need for explicit linguistic annotation.

The authors ground this choice in a broader intellectual tradition. The information-theoretic approach they adopt follows recent work in psycholinguistics and computational linguistics (Hale, 2001; Levy, 2008; Gibson et al., 2019) that models language comprehension in terms of surprisal or minimal-length codes. This literature has shown that surprisal correlates with actual human reading times and processing difficulty, suggesting it captures something cognitively real about language structure. By building on this foundation rather than the crude proxies of earlier fractal analyses, the paper positions itself as providing the first distribution-aware, information-theoretic quantification of fractal properties in language.

The Conceptual Framework: What Fractal Parameters Would Explain

Beyond the methodological gap, the paper is motivated by a specific hypothesis about what fractal structure would imply for language models. This is laid out explicitly in Section 1 and deserves careful attention because it provides the intellectual bridge between "language has fractal properties" and "this explains why LLMs work."

Self-similarity addresses the multi-scale nature of language understanding. The authors argue that since LLMs are trained to predict the future of a self-similar process, they "develop proficiency in capturing patterns across multiple levels of granularity for two interconnected reasons" (Section 1):

Transfer across scales: Self-similarity implies that "patterns at the level of a paragraph are reflective of the patterns seen at the level of a whole text." This means that learning to predict well at the paragraph level automatically provides useful signal for understanding document-level structure, and vice versa. The recursive nature of language—where clauses nest inside clauses, arguments build on arguments—means that the same kinds of dependencies recur at different scales. A model that learns to track subject-verb agreement across a short clause can, in principle, apply the same tracking mechanism across a multi-clause sentence, because the statistical signature of that dependency is similar at both scales.
Necessity of multi-scale attention: Because language "displays intricate patterns at all levels of granularity, it would not be enough to rely only on the immediate context of a sentence to predict the next token." The model cannot simply memorize local n-gram statistics; it must attend to broader structure—the direction of an argument, the constraints imposed by earlier claims, the overall intent of the text. Self-similarity implies that these long-range patterns have the same kind of statistical structure as local patterns, so the same predictive mechanisms can (in principle) operate across scales.

Long-range dependence is what makes multi-scale prediction non-trivial. The authors are careful to note that self-similarity alone is insufficient. They illustrate this with a quintessential counterexample: Brownian motion, a Wiener process with independent increments. Brownian motion is self-similar—it looks the same at all scales—but it is entirely unpredictable because each increment is independent of all previous ones. A model trained to predict Brownian motion "cannot learn anything useful from data since the process itself has independent increments" (Section 1).

This is where the Hurst parameter enters. A value of H > 0.5 indicates that the process has long-range dependence—events far apart in time are statistically correlated. The paper's finding of H ≈ 0.70 means that language sits in an intermediate regime: it is neither trivially predictable (H = 1, where the process is essentially deterministic given enough history) nor pure noise (H = 0.5, where the past provides no information about the future). The authors connect this to psychological theories of curiosity, citing Kidd and Hayden (2015) who argue that "a sweet spot of complexity exists (not too simple, nor too unpredictable) that facilities or accelerates learning" (Section 1). Language, in this view, occupies a Goldilocks zone—structured enough to be learnable, unpredictable enough to require sophisticated modeling.

The practical consequence of LRD. The authors make a subtle but important point about why long-range dependence matters for model capabilities: "processes with only short-range dependence could be forecasted (somewhat trivially) with lookup tables that provide the likelihood of transitions over brief sequences. By contrast, this is not possible in LRD processes whose contexts extend indefinitely into the past" (Section 1). If language had only short-range dependence, a model with a finite context window could, in principle, capture everything that matters—expanding the window beyond the dependence length would provide no additional predictive power. The existence of LRD means that longer contexts always provide potentially useful information, which motivates the architectural trend toward ever-larger context windows and explains why models continue to improve as context length grows.

Reconciling with Conflicting Prior Results

The paper's positioning also serves to reconcile apparently contradictory findings in the prior literature. Kokol and Podgorelec (2000) found LRD in natural language to be close to noise, while they found stronger LRD in computer languages. The paper's own analysis (Table 1) finds the opposite pattern: natural language domains show H ≈ 0.65–0.74, while GitHub (code) shows the highest LRD of all at H = 0.79, and DM-Mathematics—which consists of independent questions with no narrative structure—shows H ≈ 0.50, exactly the value expected for a process without long-range dependence. The authors attribute the earlier negative result to the use of ASCII encoding, which introduces character-level structure that is unrelated to linguistic content. This resolution matters because it shows that the fractal properties of language are real and detectable with proper methodology, not artifacts of a particular encoding or analysis technique—they were simply obscured by earlier, cruder methods.

Summary of the Gap and the Paper's Intervention

To synthesize: prior work suspected that language exhibits self-similarity and long-range dependence, but every attempt to quantify these properties used representations (word lengths, binary recurrence, ASCII encoding, marginal frequency distributions) that were too impoverished to capture the relevant structure. The paper's intervention is methodological—use the learned conditional probability distribution from a large language model as the representation, thereby capturing the intrinsic information content of text—and interpretive—argue that the resulting fractal parameters illuminate why models trained on next-token prediction develop broad capabilities, because the training objective forces them to model a process whose structure at one scale mirrors its structure at all scales, and whose dependencies extend indefinitely into the past.

3. Technical Approach

This is primarily a measurement and analysis paper whose core idea is that language, when represented as a sequence of information-theoretic surprisal values derived from a large language model, exhibits quantifiable fractal properties—specifically self-similarity and long-range dependence—and that these properties have implications for understanding why next-token prediction works as a training objective and for predicting model quality.

3.1 Reader Orientation

The paper does not build a new system or model. Instead, it constructs a measurement pipeline that converts raw text into a numerical time series that captures the intrinsic information content of language, then applies well-established fractal analysis techniques from the stochastic processes literature to that time series. The core problem it addresses is: how can we rigorously quantify the long-suspected but never formally established fractal properties of language, and what do those properties tell us about language models? The shape of the solution is a three-stage pipeline: (1) use an LLM to compute token-by-token surprisal values for documents, (2) construct zero-mean unit-variance increment processes from those surprisal sequences, and (3) fit power laws to estimate the Hölder (self-similarity) exponent S, the Hurst parameter H, and the Joseph exponent J, using standard estimators from the fractal analysis literature.

3.2 Big-Picture Architecture (Diagram in Words)

The measurement pipeline has four major components:

Language Model for Surprisal Extraction — A pretrained LLM (PaLM2-L by default, but 12 models spanning T5, PaLM, and PaLM2 families are compared) that takes a document of text and outputs the conditional probability of each token given its prefix. The negative log-probability of each token becomes a real-valued surprisal score representing the information content (in bits) of that token in context.
Increment Process Construction — A normalization stage that takes the sequence of surprisal values for a document, subtracts the mean and divides by the standard deviation to produce a zero-mean, unit-variance sequence. This models language as a random walk whose steps are the normalized surprisal values, following a standard approach used in DNA sequence analysis and other complex systems research.
Integral Process Construction — A cumulative summation stage that computes the running sum of the increment process, producing the integral process. This is the representation on which self-similarity is analyzed, because self-similarity is defined on integral (accumulated) processes rather than increment processes.
Fractal Parameter Estimators — Three separate estimation procedures applied to the constructed processes: (a) a peak-probability power-law fit for the self-similarity (Hölder) exponent S, using the integral process; (b) a rescaled-range (R/S) analysis for the Hurst parameter H, using the increment process; and (c) a standard-deviation-of-increments power-law fit for the Joseph exponent J, using the integral process.

Information flows linearly: raw text → tokenized into prefix-conditional surprisal scores (via LLM) → zero-mean unit-variance normalization → cumulative summation → fractal parameter estimation via power-law fitting. The entire pipeline is applied per-document, and results are aggregated across documents within domains and across domains to produce median parameter estimates with bootstrapped error margins.

3.3 Roadmap for the Deep Dive

First, the surprisal extraction via LLMs—how token-level conditional probabilities are computed, why this representation is chosen over alternatives, and what it captures that prior proxies could not. This is the foundational representation choice that enables everything else.
Second, the construction of increment and integral processes from surprisal sequences—why normalization to zero-mean unit-variance is necessary, what these two representations encode, and how they relate to each other mathematically.
Third, the self-similarity (Hölder) exponent estimation—the peak-probability method, its mathematical foundation, how the power-law fit is performed, and why S quantifies invariance across granularity scales.
Fourth, the Hurst parameter estimation via rescaled-range analysis—the R/S statistic, its computation from the increment process, the power-law fitting procedure, and how H > 0.5 implies long-range dependence.
Fifth, the Joseph exponent estimation, the fractal dimension derivation, and the relationship between all three parameters—including when they coincide (fractional Brownian motion) and when they diverge (real language).
Sixth, the experimental setup details—which models are used, which datasets, document filtering criteria, context length choices, bootstrapping for error estimation, and the cross-validation-like framework for predicting downstream performance.

3.4 Detailed, Sentence-Based Technical Breakdown

The Choice of Representation: Why Surprisal from LLMs?

The fundamental methodological decision in this paper is what representation of language to analyze for fractal properties. Every prior study the authors cite made a different choice: word lengths, binary recurrence indicators, ASCII byte values, or marginal frequency statistics. The paper argues that all of these are inadequate because they do not capture the joint probability distribution of language—the statistical structure that relates each word to its context and that makes language coherent across multiple scales.

The authors instead adopt an information-theoretic representation: for each token in a document, they compute the negative log-probability assigned to that token by a large language model conditioned on all preceding tokens. Let $w_t$ be the $t$ -th token in the document and $w_{[t-1]} = (w_0, w_1, \ldots, w_{t-1})$ be all tokens preceding it. The surprisal (also called the information content or self-information) of token $w_t$ given its context is:

$z_t = -\log p(w_t \mid w_{[t-1]})$

where $p(w_t \mid w_{[t-1]})$ is the conditional probability assigned by the LLM.

What it computes: the number of bits required to encode token $w_t$ in an optimal code given that the decoder already knows all preceding tokens and shares the LLM's probability distribution. If the token is highly predictable from context (e.g., the word "the" after "once upon a"), $p(w_t \mid w_{[t-1]})$ is close to 1, so $z_t$ is close to 0—the token conveys almost no new information. If the token is surprising (e.g., an unexpected content word or a topic shift), $p$ is small and $z_t$ is large—the token carries many bits of new information. The full sequence $(z_1, z_2, \ldots, z_T)$ is a time series of information content that, by the chain rule of information theory, sums to the total negative log-probability (the description length) of the document: $\sum_{t=1}^T z_t = -\log p(w_1, \ldots, w_T)$ .

Why this form: the chain rule is an identity—it does not assume anything about the joint distribution of language. Any sequence of random variables can be decomposed this way. What the LLM provides is an approximation to the true conditional distribution $p(w_t \mid w_{[t-1]})$ , and the authors note that LLMs "are known to produce calibrated probability scores at the token level" (Section 2.1, citing Kadavath et al., 2022). This calibration property means that when the LLM assigns probability 0.7 to a token, that token actually occurs about 70% of the time in those contexts—the probabilities are trustworthy estimates of the true conditional distribution. This is critical because fractal analysis is being applied to the LLM's representation of language, not to language directly; if the LLM's probabilities were miscalibrated, the resulting time series would not reflect the true information structure of the text.

The authors cite prior work in psycholinguistics (Hale, 2001; Levy, 2008; Futrell and Hahn, 2022) and information theory (Shannon, 1951) to ground this choice. Shannon himself used a similar approach—estimating the entropy of English by having human subjects predict successive letters—but with crude predictive models, not modern LLMs. The key insight from this literature is that surprisal "corresponds to an intrinsic, irreducible description of language and the minimum compute overhead to comprehend/decode it" (Section 1), and empirically correlates with human reading times and processing difficulty (Hale, 2001; Levy, 2008; Smith and Levy, 2013).

The paper explicitly calls out the inadequacy of simpler proxies by articulating why translation invariance matters (Section 4): "replacing a word with its length is invalid because it is not translation-independent (i.e. one could map every word to an arbitrary token, including tokens of equal length)." In other words, if you change the vocabulary of a language but keep the syntactic and semantic structure identical, a word-length-based analysis would change completely, whereas the surprisal-based analysis would remain essentially the same (modulo changes in token-level predictability). This is a principled argument for why the representation should be invariant to arbitrary relabelings of tokens—a property that surprisal satisfies but word frequency, word length, and other lexical features do not.

From Surprisal to Stochastic Processes: The Increment and Integral Representations

Once the sequence of surprisal values $(z_1, z_2, \ldots, z_T)$ is computed for a document, the paper transforms it into the formalism of stochastic process theory. This transformation is necessary because the fractal parameters S, H, and J are defined on stochastic processes with specific properties (zero mean, unit variance, stationarity), not on arbitrary sequences of real numbers.

Step 1: Normalization to the increment process. The raw surprisal values are standardized to have zero mean and unit variance:

$x_t = \frac{z_t - \mu_z}{\sigma_z}$

where $\mu_z = \frac{1}{T}\sum_{t=1}^T z_t$ is the empirical mean of the surprisal sequence for that document, and $\sigma_z$ is its empirical standard deviation.

What it computes: a sequence $(x_t)_{t=1}^T$ where each value represents how many standard deviations above or below the mean that token's surprisal was. Positive values mean the token was more surprising than the document average; negative values mean it was less surprising (more predictable) than average. The resulting sequence satisfies $\mathbb{E}[x_t] = 0$ and $\mathbb{E}[x_t^2] = 1$ by construction (using empirical moments).

Why this form: fractal parameters are intended to measure structural properties of a process—how its complexity behaves across scales, how its dependencies decay over time—not properties that are artifacts of arbitrary scale choices. If one document has an average surprisal of 3 bits/token and another has 5 bits/token (perhaps because the second domain is inherently harder to predict), the raw surprisal sequences would have different means and variances even if they had identical dependency structures. Normalization removes these scale differences so that the fractal analysis captures only the shape of the process, not its location or spread. The authors state this explicitly: "fractal parameters are intended to measure a fundamental property of the process (e.g. LRD) that should not be affected by scale, hence the normalization" (Section 2.1, paragraph after Figure 3 description).

The authors also note that "normalizing bits (to have zero mean and unit variance) models language as a random walk" (Section 2.1). This is a conceptual framing: the increment process $(x_t)$ is analogous to the step-by-step innovations in a random walk, where each step is drawn from some distribution and the walker's position after $t$ steps is the cumulative sum. This analogy to random walks is standard in the fractal analysis of complex systems—it has been used "extensively in the literature in various contexts, such as in DNA sequences" (Section 2.1, citing multiple references).

Step 2: Cumulative summation to the integral process. The integral process is defined as the running sum of the increment process:

$X_t = \sum_{k=0}^{t} x_k$

with the convention $X_0 = 0$ (or equivalently, $X_t = \sum_{k=1}^t x_k$ for $t \geq 1$ ).

What it computes: a sequence $(X_1, X_2, \ldots, X_T)$ where $X_t$ is the cumulative deviation of surprisal from the mean up to token $t$ . If a document starts with a run of surprisingly unpredictable tokens (many positive $x_k$ values), $X_t$ will climb upward; if it then shifts to a run of highly predictable tokens (negative $x_k$ values), $X_t$ will descend. The integral process thus visualizes the "burstiness" of language—clusters of high-surprisal and low-surprisal tokens appear as upward and downward trends, respectively. This is exactly what is shown in Figure 1 (left) of the paper, where the top panel shows $X_t$ for a Wikipedia article and lower panels show magnified portions, revealing "rich, self-similar details, such as burstiness, in all of them" (Section 1, Figure 1 caption).

Why this form: self-similarity is defined on integral processes, not increment processes. A process is self-similar if scaling time by a factor $\tau$ is equivalent (in distribution) to scaling space by a factor $\tau^S$ . This definition only makes sense for processes that accumulate over time—the integral process $X_t$ is exactly such a process. The increment process $x_t$ , by contrast, does not generally satisfy self-similarity in this sense because its values at different times may have very different distributional properties depending on the granularity of observation. The integral representation "zooms out" to reveal the multi-scale structure.

The relationship between the increment and integral processes is bijective: given one, you can reconstruct the other (up to the constant offset). The increment process is recovered from the integral process by differencing: $x_t = X_t - X_{t-1}$ . Having both representations available is useful because different fractal parameters are defined on different representations—H on the increment process, S and J on the integral process—as the authors note: "self-similarity is typically studied in the context of integral processes whereas LRD is defined on increment processes" (Section 2.1).

Why the mean is subtracted before accumulation. The normalization to zero mean is not merely convenient; it is essential. Without it, the integral process would have a linear drift equal to the mean surprisal. A document with higher average surprisal would have a steeper overall trend in $X_t$ , which would dominate the fractal analysis and obscure the burstiness structure that is the actual object of interest. By removing the mean, the integral process centers around zero and its fluctuations capture only the second-order structure—how surprisal varies around its average, not the average itself.

The Self-Similarity (Hölder) Exponent S: Peak-Probability Method

The self-similarity exponent $S$ (also called the Hölder exponent) quantifies the degree to which a process looks statistically the same at different temporal granularities. Formally, an integral process $(X_t)_{t \in \mathbb{N}}$ is self-similar with exponent $S$ if for any scaling factor $\tau > 0$ , the scaled process $(X_{\tau t})_{t \in \mathbb{N}}$ has the same finite-dimensional distributions as the rescaled original $(\tau^S X_t)_{t \in \mathbb{N}}$ . In operational terms: if you observe the process at a coarser time scale (look at every $\tau$ -th time step), the resulting process is a scaled version of the original, with the scaling factor being $\tau^S$ .

The paper estimates $S$ using a method from Watkins (2019), which the authors describe as "a well-established approach for quantifying self-similarity" (Section 1). The procedure works as follows.

Step 1: Construct $\tau$ -increments of the integral process. For a chosen granularity level $\tau$ (an integer representing how many time steps to aggregate), compute the differences:

$\Delta_\tau X_t = X_{t+\tau} - X_t$

for all $t$ where $t+\tau \leq T$ . These $\tau$ -increments represent the net change in the cumulative surprisal over a window of $\tau$ tokens. In terms of the original surprisal sequence, $\Delta_\tau X_t = \sum_{k=t+1}^{t+\tau} x_k$ is the sum of $\tau$ consecutive normalized surprisal deviations. As $\tau$ increases, these increments correspond to "the number of bits used for clauses, sentences, paragraphs and longer texts" (Section 2.2), because they aggregate surprisal over increasingly large linguistic units.

Step 2: Compute the peak probability. Fix a small threshold $\epsilon \ll 1$ . The paper uses $\epsilon = 5 \times 10^{-3}$ (Section 2.2). Define:

$p_\epsilon(\tau) = \mathbb{P}\left(|\Delta_\tau X_t| \leq \epsilon\right)$

which is the probability (empirical frequency, over all $t$ in the document) that the $\tau$ -increment falls within a narrow band $[-\epsilon, \epsilon]$ around zero. In other words, $p_\epsilon(\tau)$ is the fraction of $\tau$ -length windows over which the cumulative surprisal barely changes—the process returns close to where it started after $\tau$ steps.

Step 3: Fit a power law. For a self-similar process with exponent $S$ , it can be shown (Watkins, 2019) that:

$p_\epsilon(\tau) \sim \tau^{-S}$

where $\sim$ means asymptotic proportionality up to a slowly varying function (a function $L(\tau)$ that satisfies $L(t\tau)/L(\tau) \to 1$ as $\tau \to \infty$ for all $t > 0$ —constants and logarithms are examples). The exponent $S$ is estimated by linear regression of $\log p_\epsilon(\tau)$ against $\log \tau$ , with $S$ being the absolute value of the slope.

What it computes, operationally: $S$ measures how quickly the probability of near-zero $\tau$ -increments decays as the window size increases. If $S$ is large, the probability $p_\epsilon(\tau)$ drops rapidly with $\tau$ —the process rarely stays near zero over long windows, meaning it tends to wander far from its starting point. If $S$ is small, the probability decays slowly—the process frequently returns to near-zero even at coarse granularities, meaning the burstiness structure is persistent at all scales.

Why this form: the $\tau^{-S}$ decay is a signature of self-similar processes. For a process that is not self-similar, $p_\epsilon(\tau)$ would not follow a power law—it might decay exponentially (if the process has a characteristic time scale) or follow some more complex functional form. The fact that the paper observes clean power laws in log-log plots (Figure 2) is itself evidence for self-similarity, independent of the specific value of $S$ . The threshold $\epsilon$ is chosen to be small relative to the scale of the process (which has unit variance increments, so typical $\tau$ -increments have standard deviation $\sqrt{\tau}$ ), and the authors verify that $S$ is "robust to the choice of $\epsilon \in [10^{-3}, 10^{-2}]$ " (Section 2.2, Figure 4 left).

Interpretation of S for language. The paper finds a median $S = 0.59 \pm 0.08$ across the 8 domains with more than 1000 documents in The Pile (Table 1). A value of $S > 0.5$ indicates that the process has a positive degree of persistence: large increments tend to be followed by large increments, and small increments by small ones. If $S < 0.5$ , the process would exhibit anti-persistence (mean-reversion). The observed $S \approx 0.59$ means language is mildly persistent at all scales—bursts of high surprisal tend to cluster together (as do stretches of low surprisal), and this clustering pattern is statistically similar whether you look at the level of a few tokens (phrases) or many tokens (paragraphs), as visually evident in Figure 1.

The relationship $S = 0.59$ also determines the fractal dimension of the process (discussed in Section 2.4). The fractal dimension $D \in [1, 2]$ for a continuous-time process is related to the self-similarity exponent by $D = 2 - S$ (Gneiting and Schlather, 2004, cited in Section 2.4), giving a median fractal dimension of $D = 1.41 \pm 0.08$ for language. A value of $D = 1$ would correspond to a smooth, differentiable path (like a straight line); $D = 2$ corresponds to a path so rough it fills the plane (like Brownian motion). Language at $D = 1.41$ sits between these extremes—rougher than a smooth curve but not as space-filling as pure diffusion. The authors note that prior studies sometimes gave "extremely large values of the fractal dimension, sometimes exceeding 10" (Section 4), which are "difficult to interpret because the fractal dimension D should fall in D ∈ [1, 2] for time series." Their method, by contrast, produces physically meaningful values in the expected range, which they attribute to the information-theoretic representation avoiding artifacts of surface-level proxies.

The Hurst Parameter H: Rescaled-Range (R/S) Analysis

The Hurst parameter $H \in [0, 1]$ quantifies the degree of long-range dependence (predictability over time) in a stochastic process. It was developed by the hydrologist H. E. Hurst in 1951 while studying the Nile River's flood and drought patterns, and the estimation method—rescaled-range (R/S) analysis—remains the standard approach. The authors describe $H$ as "generally considered to be a robust metric, unlike the wavelet estimator and the periodogram method that can be sensitive to errors" (Section 1, citing Pilgrim and Taylor, 2018).

The R/S procedure is applied to the increment process $(x_t)_{t \in \mathbb{N}}$ , not the integral process. For a chosen window size $n$ (an integer representing how many consecutive tokens to analyze):

Step 1: Compute the mean-adjusted series. For each $k$ in the window $\{0, 1, \ldots, n\}$ :

$y_k = x_k - \frac{1}{n}\sum_{i=0}^n x_i$

This centers the segment so that it has zero mean over the window.

Step 2: Compute the cumulative deviation. Build the running sum of the mean-adjusted series:

$Y_t = \sum_{k=0}^{t} y_k, \quad t = 0, 1, \ldots, n$

This is the partial sum of deviations from the segment mean, analogous to the integral process but computed within a finite window and with its own local mean removed.

Step 3: Compute the range. The range $R(n)$ is the difference between the maximum and minimum of the cumulative deviation within the window:

$R(n) = \max_{0 \leq t \leq n} Y_t - \min_{0 \leq t \leq n} Y_t$

This measures the total vertical extent of the cumulative deviation path—how far the process wanders from its starting point in either direction within $n$ steps.

Step 4: Compute the scale. The scale $S(n)$ is the standard deviation of the original $x_k$ values within the window:

$S(n) = \sigma(\{x_k\}_{k=0}^n)$

where $\sigma$ denotes the sample standard deviation. The division by $S(n)$ "rescales" the range so that it is comparable across different segments of the process that may have different local variance. Without this rescaling, a segment with larger-amplitude fluctuations would have a larger range purely due to scale, not due to any structural property.

Step 5: Fit a power law. For a process with Hurst parameter $H$ , the rescaled range satisfies:

$\frac{R(n)}{S(n)} \sim n^H$

where $\sim$ again means asymptotic proportionality up to a slowly varying function. The exponent $H$ is estimated by linear regression of $\log(R(n)/S(n))$ against $\log n$ across a range of window sizes $n$ .

What it computes, operationally: the rescaled range $R(n)/S(n)$ measures how the total excursion of the cumulative process scales with the window length. For a process with independent increments (pure noise, $H = 0.5$ ), it can be proven (Feller, 1951, cited in Section 2.3) that $R(n)/S(n)$ grows as $n^{1/2}$ —the range increases as the square root of time, like a standard random walk. For a process with $H > 0.5$ , the range grows faster than $n^{1/2}$ (as $n^H$ ), meaning that extreme excursions become more pronounced over longer windows—good years cluster with good years, bad with bad. For $H < 0.5$ , the range grows slower than $n^{1/2}$ , indicating anti-persistence where the process tends to reverse direction frequently.

Why this form: the rescaled range statistic has several desirable properties that make it the preferred estimator in the fractal analysis literature. First, it is robust to non-Gaussianity—it does not assume the increment distribution is normal, which matters because surprisal values are bounded below by zero and likely have a skewed distribution. Second, the rescaling by $S(n)$ makes it invariant to the local variance, which is important because the variance of surprisal may vary across different sections of a document (e.g., dialogue vs. exposition may have different predictability profiles). Third, unlike spectral methods (periodogram, wavelet), R/S analysis does not require the process to be stationary in the strict sense, only that it has stationary increments—a milder condition that is more plausible for real text where the distribution of surprisal may drift slowly as topics change.

Connection to autocovariance and LRD. The paper explains the relationship between $H$ and the autocovariance function $\rho_k = \mathbb{E}[x_t x_{t+k}]$ (Section 2.3). If the autocovariance decays as a power law $\rho_k \sim k^{-\beta}$ for large $k$ , then $H$ and $\beta$ are related by:

$H = 1 - \frac{\beta}{2}$

When $H > 1/2$ , we have $0 < \beta < 1$ , which means the autocovariances decay slowly enough that their infinite sum diverges: $\sum_{k=1}^\infty \rho_k = \infty$ . This is the definition of long-range dependence (LRD): the cumulative effect of dependencies across all time scales is infinite, even though each individual dependence becomes arbitrarily small at large lags. When the autocovariances are summable ( $\beta > 1$ or $H < 1/2$ ), the process has only short-range dependence—the total influence of the infinite past is finite, and the process can be effectively modeled with a finite memory.

Why LRD matters for the integral process variance. The authors note a critical consequence (Section 2.3, citing Samorodnitsky, 2006):

$\lim_{n \to \infty} \frac{\operatorname{Var}(X_n)}{n} = 1 + 2\sum_{i=1}^\infty \rho_i$

where $\operatorname{Var}(X_n)$ is the variance of the integral process after $n$ steps. If the autocovariances are summable ( $H < 1/2$ ), the right-hand side is finite, so $\operatorname{Var}(X_n)$ grows linearly with $n$ —the integral process behaves like a standard random walk. If $H > 1/2$ , the sum diverges, and using the Euler-Maclaurin summation formula (Apostol, 1999; Alabdulmohsin, 2018, both cited), one obtains:

$\operatorname{Var}(X_n) \sim n^{2H}$

for $H > 1/2$ . The variance grows superlinearly: the integral process diffuses faster than a random walk, creating the dramatic long-range excursions (Joseph effects) visible in Figure 1.

Interpretation of H for language. The paper finds a median $H = 0.70 \pm 0.09$ across domains (Section 2.3). This is substantially above 0.5, establishing that language exhibits statistically significant long-range dependence. The authors contextualize this value: "a Hurst parameter of about 0.75 turns out to occur commonly in nature, including in river discharges, Ethernet traffic, temperatures, precipitation, and tree rings" (Section 1, citing multiple references). They connect this to theories of curiosity and learning: a Hurst parameter around 0.75 represents "a sweet spot of complexity... that facilities or accelerates learning" (Section 1, citing Kidd and Hayden, 2015). Language is predictable enough to be learnable but unpredictable enough to require sophisticated modeling that attends to long-range structure.

The authors also note the negative case of DM-Mathematics, which has $H = 0.50 \pm 0.01$ (Table 1)—indistinguishable from pure noise. This domain consists of independent mathematical questions concatenated into documents (Figure 6), so there genuinely is no dependence from one question to the next. The fact that R/S analysis correctly identifies this lack of structure is strong validation that the method is measuring something real about text organization, not just producing numerical artifacts.

The Joseph Exponent J: Standard Deviation of Increments

The Joseph exponent $J$ (named after the biblical story of seven fat years followed by seven lean years) quantifies the degree of burstiness or clustering in the integral process. It measures how the variance of increments scales with the window size.

Procedure. For each granularity level $\tau$ , compute the standard deviation of the $\tau$ -increments:

$\sigma_\tau = \operatorname{StdDev}\left(\{X_{t+\tau} - X_t\}_t\right)$

where the standard deviation is taken over all positions $t$ in the document where $t + \tau \leq T$ . Then fit a power law:

$\sigma_\tau \sim \tau^J$

by linear regression of $\log \sigma_\tau$ against $\log \tau$ .

What it computes: $J$ measures how the volatility of cumulative surprisal grows with the observation window. If $J = 0.5$ , the standard deviation grows as $\tau^{1/2}$ —exactly the behavior of a random walk with independent increments. If $J > 0.5$ , the volatility grows faster, indicating that large deviations accumulate more rapidly than chance would predict, creating bursty behavior where the process clusters on one side of the mean for extended periods. If $J < 0.5$ , the process is anti-persistent with frequent reversals.

Why this form: the standard deviation of $\tau$ -increments is a natural measure of how much the process "spreads out" over time. For self-similar processes with Hurst parameter $H$ , $\sigma_\tau$ grows as $\tau^H$ . For fractional Brownian motion (the canonical self-similar Gaussian process), $J$ , $H$ , and $S$ all coincide: $S = H = J$ . Real-world processes, however, often deviate from this idealized case, and the relationship between these parameters provides information about the nature of the process. The seminal work of Mandelbrot and Wallis (1968) on the Noah and Joseph effects in hydrology established that these parameters can differ in practice, and "Mandelbrot himself cautioned against" equating them in general (Section 2.1).

Interpretation of J for language. The paper finds a median $J = 0.49 \pm 0.08$ (Section 2.5, Table 1). This is notably different from both $S = 0.59$ and $H = 0.70$ . The fact that $J$ is substantially lower than $H$ indicates that language is not well-modeled by fractional Brownian motion. Specifically, $J \approx 0.5$ means the volatility scaling is close to that of a random walk, even though the process exhibits both self-similarity ( $S > 0.5$ ) and long-range dependence ( $H > 0.5$ ). The authors do not provide an extensive interpretation of this discrepancy, but in the fractal analysis literature, a gap between $J$ and $H$ is often associated with multifractality—the process may have different scaling exponents at different orders of magnitude, which a single power law cannot fully capture. This is noted as an area for potential future investigation.

Relationships Among Parameters and the Fractal Dimension D

The paper's Section 2.4 provides the derivation of the fractal dimension from the self-similarity exponent. The fractal dimension $D$ of a continuous-time stochastic process is defined through the local behavior of the autocovariance function near zero lag. Specifically, if the normalized autocovariance satisfies $\rho_n \sim 1 - n^\alpha$ as $n \to 0^+$ for some $\alpha \in (0, 2]$ , then:

$D = 2 - \frac{\alpha}{2}$

It can be shown (Gneiting and Schlather, 2004) that for self-similar processes, $\alpha = 2S$ and therefore:

$D = 2 - S$

For language, this gives $D = 2 - 0.59 = 1.41 \pm 0.08$ . A process with $D$ close to 1 has smooth sample paths (few high-frequency fluctuations); a process with $D$ close to 2 has extremely rough paths (space-filling). Language at $D = 1.41$ has substantial roughness—there is meaningful variation at all frequency scales—but the paths are not as jagged as pure white noise, consistent with the presence of dependence structure that smooths out some of the variation.

The authors note that "by convention, an object is referred to as 'fractal' if D is different from its topological dimension" (Section 2.4). The topological dimension of a time series is 1 (a curve), so any $D > 1$ qualifies as fractal. Language, with $D = 1.41$ , is unambiguously fractal by this definition—its information-theoretic structure has complexity that exceeds what a smooth curve can represent. The classic example Mandelbrot used to motivate fractal geometry was the coast of Britain, with estimated $D = 1.25$ (Mandelbrot, 1967). Language has a fractal dimension in the same ballpark as natural coastlines, which is an intriguing parallel.

Experimental Setup and Document Processing Pipeline

The paper's empirical analysis involves careful choices about which models to use, which data to analyze, and how to configure the measurement pipeline. These choices are motivated by the need to establish robustness and avoid confounding factors.

Model selection. The analysis centers on PaLM2-L (Unicorn) as the primary model for computing surprisal values, but the paper also evaluates 11 additional models spanning three families (Section 2.1, Appendix A):

T5 Decoder-Only: 110M, 340M, 1B, and 5B parameter variants trained on C4 with a causal language modeling objective for 262B tokens. All use the default T5 vocabulary from Raffel et al. (2019).
PaLM: 8B, 62B, and 540B parameter variants using the checkpoints from Chowdhery et al. (2022).
PaLM2: Gecko (XXS), Otter (XS), Bison (S), M, and L (Unicorn) variants using checkpoints from Anil et al. (2023b).

This range—from 110M to over 500B parameters—allows the authors to assess whether fractal parameter estimates are robust to model architecture and scale, which is a central claim of the paper.

Training details for T5 baselines. The T5 decoder-only models are trained by the authors specifically for this analysis (Appendix A). They use the T5.1.1 decoder-only architecture from the T5X library (Roberts et al., 2022) with a causal language modeling objective. Training hyperparameters: 500k steps, sequence length 1024, batch size 512 (yielding 262B total tokens), Adafactor optimizer (Shazeer and Stern, 2018) with inverse square root learning rate schedule, 1k warmup steps, initial learning rate $1 \times 10^{-2}$ , trained on 256 TPUv5e chips (Jouppi et al., 2020). All experiments are implemented in JAX/Flax (Bradbury et al., 2018).

Dataset: The Pile validation split. All analysis is performed on The Pile validation split (Gao et al., 2020), which contains 22 subdomains including Wikipedia, GitHub, ArXiv, PubMed, FreeLaw, OpenWebText2, Pile-CC, and DM-Mathematics. The Pile is chosen because it provides diverse domains that allow the authors to test whether fractal properties are universal or domain-specific. The validation split (not the training split) is used to avoid contamination concerns.

Document filtering criteria (Section 2.1). Two filtering criteria are applied:

Minimum document length: Only documents with more than 4,000 tokens are used. This is to "sidestep potential effects of the finite length of documents" on the fractal analysis—power-law fitting requires observing the process over a sufficient range of scales, and very short documents may not provide enough data to reliably estimate asymptotic scaling behavior.
Maximum analysis length: Only the first 2,048 tokens of each document are analyzed. This constraint serves two purposes: (a) it matches the context length of the PaLM2-L model (which has a 2,048-token context window; the paper mentions "prefixes of up to 2048 tokens (≈ 8 pages of text)" in Section 2.1), and (b) it ensures that all documents are analyzed over the same length, preventing longer documents from dominating the estimates while providing enough tokens for reliable power-law fitting. The first 100 tokens of each document are trimmed before analysis (noted in Appendix B tables: "after trimming the first 100 tokens")—this likely removes boilerplate headers or title material that could be distributionally anomalous.
Minimum domain size: Only domains with more than 1,000 documents that satisfy the length criteria are included in the per-domain analysis. This is "to mitigate noise" (Section 2.1) and ensures that domain-level estimates are based on sufficient sample sizes. Eight domains meet this criterion (Table 1): OpenWebText2, GitHub, FreeLaw, Pile-CC, Wikipedia, PubMed, DM-Mathematics, and ArXiv.

Normalization scope. The normalization to zero-mean unit-variance (the $x_t = (z_t - \mu_z)/\sigma_z$ transformation) is performed per-document, not globally across all documents. This means each document's surprisal sequence is independently standardized. The motivation, as discussed earlier, is that fractal parameters should reflect the internal dependency structure of each document, not cross-document differences in average surprisal level. If two documents from different domains have different average surprisal but identical dependency structures, their normalized increment processes should be statistically indistinguishable.

Bootstrapping for error estimation. Error margins on all parameter estimates are computed using bootstrapping (Efron and Tibshirani, 1994, cited in Section 2.1). The specific bootstrap procedure is not described in detail, but the standard approach would be: resample documents with replacement from the domain, re-estimate the fractal parameter on the resampled set, repeat many times (typically 1,000–10,000), and report the standard deviation or percentile interval of the bootstrap distribution. The error margins in the tables (e.g., $S = 0.53 \pm 0.05$ for OpenWebText2 in Table 1) represent these bootstrap standard errors.

Power-law fitting details. For each parameter estimation, the power-law relationship is fit by linear regression in log-log space. The paper does not specify the range of scales ( $\tau$ for S and J, $n$ for H) over which the fit is performed, though typical practice in fractal analysis is to use a range where the log-log plot appears linear (avoiding very small scales where finite-sample effects dominate and very large scales where boundary effects become significant). The fact that the log-log plots (Figures 2, 3, 5) show clear linear trends suggests the power-law behavior holds over a substantial range, but the exact fitting range would affect the point estimates.

Downstream performance prediction framework. The second major component of the analysis (Section 3, "Downstream Performance") evaluates whether fractal parameters capture model quality information beyond what perplexity-based metrics provide. The authors compute median fractal parameters for each of the 12 models by taking the median of the parameter across the 8 qualifying domains in The Pile (described in Table 1 for PaLM2-L; equivalent tables for all 12 models are provided in Appendix B, Tables 5–7). They then evaluate each model on a battery of downstream benchmarks:

BIG-Bench Hard (BBH): 0-shot and 3-shot evaluation, both direct prompting and chain-of-thought (CoT), reporting average accuracy across all 21 BBH tasks (Suzgun et al., 2022).
MMLU: 0-shot and 5-shot evaluation, reporting average accuracy across all 57 subjects (Hendrycks et al., 2020).
GSM8K: 8-shot evaluation with chain-of-thought prompting, reporting accuracy (Cobbe et al., 2021).
Aggregate metrics: 0S BBH+MMLU (combining zero-shot results) and FS (few-shot) BBH+MMLU+GSM8K (combining few-shot results).

All prompt templates follow Chung et al. (2022) and Longpre et al. (2023). Models are prompted with a 2,048-token context length, and only pretrained checkpoints (without instruction tuning) are used for fair comparison. The authors note explicitly that "our results are not directly comparable to all previous published results for the same models... here, we only aim for a fair comparison between models" (Appendix C, Table 8 note).

The combined metric the paper introduces, $H_B$ , is defined simply as:

$H_B = \frac{1}{\text{BPB}} + H_{\text{median}}$

where BPB is the bits-per-byte score of the model computed over The Pile validation split (using the first 2,048 tokens of every document) and $H_{\text{median}}$ is the median Hurst parameter across the 8 domains. The reciprocal $1/\text{BPB}$ is used so that "higher values are better" for both components—better models have lower BPB (higher $1/\text{BPB}$ ) and higher H.

Statistical analysis for prediction. The paper reports adjusted $R^2$ values from linear regressions where the downstream benchmark score is the dependent variable and the predictor is either BPB alone, $H_{\text{median}}$ alone, or $H_B$ . The adjusted $R^2$ measures the proportion of variance in downstream performance that can be predicted by a linear function of the predictor, with a correction for the number of predictors to avoid overfitting. The key finding (Table 3, middle columns) is that $H_B$ consistently yields higher adjusted $R^2$ than BPB alone across all benchmarks—for example, 3-shot BBH CoT goes from adjusted $R^2 = 0.881$ (BPB alone) to $0.979$ ( $H_B$ ), and the few-shot aggregate (FS BBH+MMLU+GSM8K) improves from $0.717$ to $0.891$ .

The surprising correlation between H and BPB. A finding the authors describe as "surprising" (Section 3) is that "the median Hurst parameter is itself strongly correlated with the BPB scores with an absolute Pearson correlation coefficient of 0.83, even though the Hurst exponent is calculated after normalizing all token losses to zero-mean and unit variance!" In other words, a second-order statistic (how the variance of cumulative surprisal scales with window size) strongly predicts a first-order statistic (the average surprisal per token). The authors interpret this as implying that "second-order statistics on the sequence of token losses of a particular model can predict its mean"—a non-trivial relationship that would not hold for a generic time series and that suggests deep structure in how model quality manifests in the surprisal sequence. The self-similarity exponent S, by contrast, has a much weaker correlation with BPB ( $|r| = 0.23$ ).

Context length ablation (negative result). Finally, the paper reports a controlled experiment testing whether training with longer context improves downstream performance (Section 3, "Context Length at Training Time"). The hypothesis is that "self-similarity and LRD point to an intriguing possibility: the importance of training the model with extensive contexts in order to capture the fractal-nature of language." Three decoder-only T5.1.1 models with 1B parameters are pretrained from scratch on SlimPajama-627B (Soboleva et al., 2023) for 100B tokens with context lengths of 2K, 4K, and 8K tokens (converting from C4 to SlimPajama because C4 documents are too short—94% are under 2K tokens). To ensure fair comparison, batch sizes are adjusted (512, 256, and 128 respectively) so that all models see the same number of tokens per batch and per training step (100k steps total). The T5 vocabulary is modified to preserve whitespace and include byte-fallback for out-of-vocabulary tokens, following Chowdhery et al. (2022). The result (Table 3, right columns) shows "we do not observe any improvements in performance with context length in this particular setup." Accuracy is essentially flat across context lengths for all benchmarks (e.g., 0S MMLU: 25.73, 26.04, 25.81 for 2K, 4K, 8K respectively). This negative result is important because it constrains the interpretation of the fractal findings: while language has long-range dependence, it does not automatically follow that training with explicitly longer contexts improves model quality—at least not in the regime of 1B parameters and 100B training tokens. The authors present this without extensive interpretation, leaving it as a data point for future investigation.

Design Choices and Their Justifications (Summary)

LLM-based surprisal over surface-level proxies: captures the joint distribution of language, is translation-invariant, reflects intrinsic information content, and leverages well-calibrated probability estimates from modern LLMs.
Per-document normalization to zero-mean unit-variance: removes scale artifacts so that fractal parameters measure structural properties (dependency, self-similarity) rather than domain- or model-specific average surprisal levels.
Integral and increment process dual representation: self-similarity and Joseph exponent are defined on integral processes; Hurst parameter is defined on increment processes. Having both representations available enables rigorous estimation of all three parameters.
Peak-probability method for S: a well-established approach from Watkins (2019) that is robust to the choice of threshold $\epsilon \in [10^{-3}, 10^{-2}]$ and directly tests the defining property of self-similar processes.
Rescaled-range analysis for H: considered the most robust estimator for the Hurst parameter, insensitive to non-Gaussianity and mild non-stationarity, and validated by correctly identifying $H = 0.5$ for DM-Mathematics (independent questions).
Minimum document length of 4K tokens, analysis window of 2K tokens: balances the need for sufficient data for power-law fitting against the finite context window of the models and the finite length of documents in the dataset.
Bootstrapping for error estimation: provides non-parametric confidence intervals without assuming normality of the parameter estimates.
Median aggregation across domains: robust to outliers and domain-specific effects, providing a single summary statistic per model that captures the central tendency of fractal properties across diverse text types.
Combined metric $H_B$ : motivated by the observation that H and BPB are correlated but each captures partially independent information, and verified by adjusted $R^2$ improvement across all benchmarks.

4. Key Insights and Innovations

Innovation 1: Language Is a Self-Similar, Long-Range Dependent Process—and This Is Now Rigorously Quantified Using the Full Joint Distribution

The paper's most fundamental contribution is transforming a long-standing suspicion into a formal empirical fact: language, when represented as an information-theoretic surprisal process derived from a calibrated language model, exhibits both statistical self-similarity and long-range dependence with specific, well-estimated parameters ( $S = 0.59 \pm 0.08$ , $H = 0.70 \pm 0.09$ ). This may sound like an incremental refinement of prior work that already "suspected" these properties, but it is not. It is a fundamental methodological shift in how language is represented for fractal analysis, and it resolves a history of contradictory and uninterpretable results.

To understand why this matters, consider what the field had before this paper. Prior studies that argued for self-similarity in language all used surface-level proxies that discarded the joint statistical structure of text. The Menzerath-Altmann law operates on relationships between linguistic unit sizes (words, clauses) but does not engage with information content. Ausloos (2012) converted text to word-length sequences, which the present paper shows is not translation-invariant—swap the vocabulary while preserving syntax and semantics, and the analysis changes completely. Najafi and Darooneh (2015) tracked binary recurrence of individual words, which cannot distinguish between semantically distinct uses of the same word or capture multi-word dependencies. Kokol and Podgorelec (2000) used ASCII encoding and concluded that natural language's LRD was "close to that of pure noise"—a finding this paper overturns by using a representation that actually captures linguistic structure. Even more problematically, some prior studies produced fractal dimensions "sometimes exceeding 10" (Section 4), which are physically uninterpretable for time series (where $D \in [1, 2]$ ).

The paper's innovation is not that it invented fractal analysis or discovered self-similarity in language. It is that it identified the representation problem as the bottleneck and solved it by leveraging modern LLMs as calibrated probability estimators. The representation—token-by-token surprisal computed from the conditional distribution $p(w_t \mid w_{[t-1]})$ —is principled in a way that none of the prior proxies were. It is translation-invariant (relabeling tokens doesn't change the information content of their transitions). It captures the full joint distribution (through the chain rule, the surprisal sequence summarizes everything the model knows about the text's statistical structure). It connects to psycholinguistic theory (surprisal correlates with human reading times, suggesting it captures something cognitively real). And it produces physically meaningful results— $D = 1.41$ falls squarely in the interpretable range, comparable to Mandelbrot's estimate of $D = 1.25$ for the coast of Britain.

The significance of this goes beyond the specific numerical estimates. The paper demonstrates that with the right representation, language shows the same fractal signatures as many natural phenomena (river discharges, Ethernet traffic, heart rate variability). This is not a curiosity—it means that the mathematical tools developed for analyzing those phenomena (R/S analysis, peak-probability estimation, fractal dimension theory) can be applied to language with the same rigor. It also means that language's complexity is not arbitrary—it follows the same scaling laws as other complex systems that emerge from hierarchical, self-organizing processes. The fact that $H \approx 0.70$ lands in the same range as these other phenomena (Section 1 notes that " $H = 0.75$ turns out to occur commonly in nature") suggests a deep commonality in how structured, learnable signals arise from interacting components.

This innovation is fundamental, not incremental, because it changes what we can claim about language. Before this paper, one could say "language may have fractal properties." After this paper, one can say "language has a self-similarity exponent of $0.59 \pm 0.08$ and a Hurst parameter of $0.70 \pm 0.09$ , as measured on its intrinsic information-theoretic description, and these values are robust across model architectures and domains." That is a qualitative shift in the epistemic status of the claim.

The evidence anchoring this innovation is Table 1 (which shows domain-by-domain parameter estimates with bootstrapped errors), Table 2 (which shows robustness across 12 models from three families), and the log-log power-law fits in Figures 2, 3, and 5 (which visually demonstrate that the power-law relationships hold—linearity in log-log space is the signature of fractal scaling, and the clean linear trends are themselves evidence for self-similarity and LRD independent of the estimated exponents).

Innovation 2: The Fractal Structure Provides a Principled—Not Just Metaphorical—Explanation for Why Next-Token Prediction Yields Multi-Scale Capabilities

Many papers have observed that LLMs develop surprising capabilities from next-token prediction and have gestured at explanations involving "learning patterns at multiple scales" or "building world models." This paper offers something different: a mechanistic, statistically grounded argument for why the training objective works, derived from the fractal properties of the signal itself.

The argument has two linked components, and the paper is careful to present both because either one alone would be insufficient.

First, self-similarity implies transfer across scales. If the statistical patterns at the paragraph level mirror those at the document level (which is what $S > 0$ implies—the process "looks the same" at different time granularities), then optimizing next-token prediction at the local level automatically provides a learning signal that is relevant at broader scales. The model does not need a separate objective for "understand the paragraph" versus "understand the document"—the same predictive mechanisms, applied to tokens that are positioned at different points in the self-similar hierarchy, encounter statistically similar dependency structures. The paper's Figure 1 makes this visual: the integral process for a Wikipedia article shows the same kind of burstiness and clustering at all three magnification levels. A model trained to predict the next step of this process is, in effect, trained on a fractal object whose local structure is a microcosm of its global structure.

Second, long-range dependence implies that local context is never sufficient. The paper makes an elegant argument about why $H = 0.70$ matters for model design: "processes with only short-range dependence could be forecasted (somewhat trivially) with lookup tables that provide the likelihood of transitions over brief sequences. By contrast, this is not possible in LRD processes whose contexts extend indefinitely into the past" (Section 1). This is not rhetorical—it follows from the mathematics of LRD. When $H > 0.5$ , the autocovariance function decays as $\rho_k \sim k^{-\beta}$ with $\beta < 1$ , meaning the sum $\sum_{k=1}^\infty \rho_k$ diverges. Every token is correlated with all previous tokens, with no finite cutoff beyond which the correlation becomes negligible. A finite-context model always leaves predictive power on the table, and expanding the context window always helps (asymptotically). This provides a theoretical justification for the empirical trend toward ever-larger context windows in LLMs and explains why models continue to improve as context length grows—there is genuinely more information to extract from longer histories.

The paper's negative example clinches this argument. DM-Mathematics, with its concatenated independent math problems, has $H \approx 0.50$ —exactly the value for a process with no long-range dependence. And indeed, looking at the actual documents (Figure 6), each problem is independent of the next. A model that learned to predict tokens in this domain would gain nothing from attending across problem boundaries, because the Hurst parameter correctly identifies that there are no cross-problem dependencies to exploit. The contrast with natural language domains (OpenWebText2 at $H = 0.68$ , ArXiv at $H = 0.72$ ) is stark and directly interpretable.

What makes this innovation fundamental rather than incremental is that it bridges a gap in the theoretical understanding of LLMs. The standard explanation for why next-token prediction works is essentially empirical: "we tried it and it produced impressive capabilities." This paper provides a why that is rooted in the statistical structure of the training signal itself. Language is a self-similar, LRD process. Training a model to minimize surprisal on such a process forces it to learn dependencies that span multiple scales, because those dependencies are genuinely present in the data and ignoring them would incur a prediction penalty. The "remarkable capabilities" are not magic—they are the expected outcome of optimizing a well-specified objective on a signal with deep multi-scale structure.

The key pieces of evidence are: (1) the $H = 0.70$ estimate (Figure 3, Table 1) establishing LRD; (2) the $S = 0.59$ estimate (Figure 2, Table 1) establishing self-similarity; (3) the DM-Mathematics contrast ( $H = 0.50$ , Figure 6) demonstrating that the fractal parameters track meaningful differences in text organization; and (4) the visual evidence in Figure 1 showing that the burstiness structure is indeed scale-invariant. The paper does not provide a causal experiment proving that fractal structure causes model capabilities—that would require manipulating the fractal parameters of the training data and observing the effect, which is not attempted. But the correlational evidence presented is strong, theoretically grounded, and consistent across domains and model architectures.

Innovation 3: Fractal Parameters Capture a Dimension of Model Quality That Is Orthogonal to Perplexity—and Combining Them Improves Downstream Performance Prediction

The paper's most practically significant innovation is the discovery that variations in the Hurst parameter across models predict downstream performance in a way that perplexity-based metrics do not fully capture, and that a simple combined metric ( $H_B = 1/\text{BPB} + H$ ) substantially outperforms BPB alone. The adjusted $R^2$ for predicting the few-shot aggregate benchmark score (BBH + MMLU + GSM8K) jumps from 0.717 (BPB alone) to 0.891 ( $H_B$ ), and for 3-shot BBH CoT it jumps from 0.881 to 0.979. These are large improvements in explained variance, and they come from adding a single scalar parameter.

This finding is surprising for a reason the paper highlights: $H$ is computed after normalizing all token surprisals to zero mean and unit variance. That is, $H$ is a property of the second-order structure of the surprisal sequence (how variance accumulates across scales), not of the average surprisal level. Yet the median $H$ and BPB are correlated with $|r| = 0.83$ across the 12 models tested. The paper's comment that this implies "second-order statistics on the sequence of token losses of a particular model can predict its mean" understates the peculiarity. For a generic time series, there is no necessary relationship between the mean and the scaling exponent of the variance. The fact that such a relationship exists for language model surprisals suggests that improving a model's average predictive accuracy (lower BPB) and improving the fractal structure of its surprisals (higher H) are not independent processes—better models produce surprisal sequences whose dependency structure more closely resembles that of natural language.

What makes this innovation conceptually significant beyond the numbers is that it introduces a new category of model evaluation metric. Perplexity and BPB measure how well the model's probability estimates match the data on average—a first-order statistic. The Hurst parameter, by contrast, measures the temporal organization of the model's prediction errors—a second-order statistic. That this second-order metric improves downstream prediction suggests that what matters for downstream capabilities is not just how often the model is right, but what kind of wrong it is when it is wrong. Models whose surprisal sequences have fractal structure closer to natural language ( $H \approx 0.70$ ) perform better on reasoning and knowledge tasks than models with the same average surprisal but different dependency structure.

The paper does not fully explain why this should be the case, which makes this innovation more of a diagnostic discovery than a complete theoretical explanation. But several interpretations are consistent with the data: (a) $H$ may reflect how well the model has internalized the multi-scale dependency structure of language—a model that produces surprisals with $H$ close to 0.70 is one whose internal representations track long-range dependencies similarly to how human language does; (b) $H$ may capture aspects of model calibration that BPB misses—two models with the same average log-loss may have very different distributions over when they make errors, and the Hurst parameter is sensitive to the clustering of high-surprisal tokens; (c) $H$ may simply be a more stable summary statistic for model quality at the frontier, where BPB improvements become very small while downstream capabilities continue to improve. The fact that $S$ and $J$ do not provide similar predictive improvements (Appendix C, Table 9) narrows the phenomenon to $H$ specifically, ruling out a generic "all fractal parameters help" explanation.

This innovation is incremental in its practical form (a linear combination of two existing metrics) but fundamental in what it reveals: that perplexity is not a sufficient statistic for model quality, and that the temporal structure of prediction errors contains information about downstream capabilities that mean accuracy does not capture. This has implications for model evaluation (we should track fractal parameters during training, not just loss curves) and model selection (when choosing between checkpoints with similar perplexity, prefer the one with the Hurst parameter closer to 0.70). The evidence is in Table 3 (adjusted $R^2$ comparisons), Figure 7 (visualizing the joint relationship between H, BPB, and downstream performance), and Table 2 (showing that H varies systematically with model quality within each model family, while S and J are more stable).

Innovation 4: The Fractal Parameters Are Robust to Architecture and Scale—but Their Tiny Variations Track Model Quality

A finding that operates at a different level than the previous three is the robustness of fractal parameter estimates across model families, scales, and domains, combined with the observation that the small variations that do exist are meaningful. Table 2 shows that the median Hurst parameter across all domains is remarkably stable: T5-Decoder models (110M to 5B) all cluster around $H = 0.64$ , PaLM1 models (8B to 540B) sit at $H = 0.66-0.68$ , and PaLM2 models (XXS to L) range from $H = 0.66$ to $0.69$ . The self-similarity exponent S shows more variation (0.56–0.64) but still within a consistent range. The Joseph exponent J is the most stable, tracking from 0.44 in the smallest models to 0.49 in the largest.

This robustness is not foreordained—it is a non-trivial empirical finding. Since the surprisal sequences are computed from the models themselves, there is no guarantee that a 110M T5 decoder and a 540B PaLM would produce increment processes with the same fractal structure, even when evaluated on the same text. The fact that they do suggests that the fractal parameters are primarily determined by the data, not the model—they measure properties of language that even relatively weak models can detect, because the fractal structure is so deeply embedded in the text that any model that achieves non-random predictive accuracy will produce surprisal sequences with approximately the right scaling behavior.

At the same time, the tiny variations within this stable range are what drive the downstream prediction result discussed in Innovation 3. The median H values in Table 2 show a clear monotonic trend: better models have higher Hurst parameters. Within the T5 family, all four sizes have $H = 0.64$ —there is literally no variation. Within PaLM1, H increases from 0.66 (8B) to 0.68 (62B and 540B). Within PaLM2, H increases steadily from 0.66 (XXS) to 0.69 (L). This monotonicity within families, combined with the cross-family jumps (T5 at 0.64 vs. PaLM at 0.68 vs. PaLM2 at 0.69), is exactly what makes H a useful complement to BPB: it captures model-quality differences that BPB alone misses, particularly when comparing across architectures.

What makes this innovation fundamentally interesting beyond the practical metric improvement is that it establishes the Hurst parameter as a scale-invariant measure of model quality. Unlike perplexity, which depends on the tokenizer, the training data distribution, and the model's capacity, the Hurst parameter (as computed on standardized surprisal sequences) is defined in a way that is independent of the absolute scale of the predictions. Two models that assign different absolute probabilities to the same text (different BPB) can produce surprisal sequences with the same H—and when they differ in H, that difference signals something about the model's internal organization of uncertainty that perplexity does not capture. This is a new kind of model fingerprint, orthogonal to the standard loss-based metrics that dominate LLM evaluation.

The evidence for this robustness is in Tables 2, 5, 6, and 7 (Appendix B), which provide per-model, per-domain parameter estimates with bootstrapped standard errors. The domain-level breakdown in Table 1 is also critical: it shows that while some domains deviate (DM-Mathematics with $H = 0.50$ , GitHub with $H = 0.79$ ), the median across domains is stable because the domain-specific effects average out. The contextualization with prior work that found "extremely large values of the fractal dimension, sometimes exceeding 10" (Section 4) highlights that this robustness was not achieved by earlier methods—it is a consequence of the information-theoretic representation, and the fact that it holds across 12 models and 8 domains is itself a validation of the methodology.

5. Experimental Analysis

Evaluation Methodology

Dataset. The Pile validation split (Gao et al., 2020), consisting of 22 subdomains such as Wikipedia, GitHub, ArXiv, PubMed, FreeLaw, and OpenWebText2. Only documents with more than 4,000 tokens are used (to avoid finite-length effects), and only the first 2,048 tokens of each qualifying document are analyzed (matching the model context window). The first 100 tokens are trimmed before analysis. To reduce noise, only domains with more than 1,000 qualifying documents are included in the per-domain analysis, yielding eight domains for the main results: OpenWebText2, GitHub, FreeLaw, Pile-CC, Wikipedia, PubMed, DM-Mathematics, and ArXiv.
Base model(s). Three model families spanning 12 total checkpoints: T5 Decoder-Only (110M, 340M, 1B, 5B parameters, trained on C4 for 262B tokens with a causal language modeling objective), PaLM (8B, 62B, 540B, using the pretrained checkpoints from Chowdhery et al., 2022), and PaLM2 (XXS/Gecko, XS/Otter, S/Bison, M, L/Unicorn, using checkpoints from Anil et al., 2023b). The primary analysis uses PaLM2-L (Unicorn) as the default model for computing surprisal values. This range—from 110M to over 500B parameters—is chosen to assess whether fractal parameter estimates are robust to architecture and scale. The T5 decoder baselines are trained by the authors specifically for this analysis using the T5X framework (Roberts et al., 2022) with the Adafactor optimizer, inverse square root learning rate schedule, 1k warmup steps, and initial learning rate of 1 × 10⁻² on 256 TPUv5e chips (Jouppi et al., 2020). For the context length ablation, three additional T5.1.1 1B models are pretrained on SlimPajama-627B (Soboleva et al., 2023) for 100B tokens with context lengths of 2K, 4K, and 8K tokens (batch sizes adjusted to 512, 256, and 128 respectively to preserve tokens per batch).
Metrics. Three fractal parameters are estimated per document and aggregated to domain-level and cross-domain medians with bootstrapped error margins: (1) the self-similarity (Hölder) exponent S, estimated via the peak-probability method (Watkins, 2019) by fitting $p_\epsilon(\tau) \sim \tau^{-S}$ with $\epsilon = 5 \times 10^{-3}$ ; (2) the Hurst parameter H, estimated via rescaled-range (R/S) analysis by fitting $R(n)/S(n) \sim n^H$ on the increment process; and (3) the Joseph exponent J, estimated by fitting $\sigma_\tau \sim \tau^J$ on the standard deviation of $\tau$ -increments of the integral process. A derived metric, the fractal dimension D, is computed as $D = 2 - S$ . The combined predictor $H_B$ is defined as $H_B = 1/\text{BPB} + H_{\text{median}}$ , where BPB is bits-per-byte computed over The Pile validation split and $H_{\text{median}}$ is the median Hurst parameter across the eight qualifying domains. Downstream performance is evaluated on BIG-Bench Hard (0-shot and 3-shot, both direct and chain-of-thought prompting, average accuracy across 21 tasks), MMLU (0-shot and 5-shot, average accuracy across 57 subjects), and GSM8K (8-shot with chain-of-thought). Aggregate metrics combine these: 0S BBH+MMLU and FS (few-shot) BBH+MMLU+GSM8K. All downstream evaluations use a 2,048-token context length with prompt templates from Chung et al. (2022) and Longpre et al. (2023), and only pretrained checkpoints without instruction tuning are evaluated to ensure fair comparison.
Baselines. The paper's analysis is primarily descriptive—measuring and comparing fractal parameters—rather than proposing a new method that competes against baselines. However, several comparison points serve as implicit baselines: (1) BPB alone as a predictor of downstream performance, against which $H$ and $H_B$ are compared; (2) the self-similarity exponent S and Joseph exponent J as alternative fractal parameters, tested for their predictive power (Appendix C, Table 9); (3) DM-Mathematics as a "control" domain that should exhibit $H \approx 0.5$ if the methodology correctly identifies the absence of long-range dependence; (4) prior estimates from the literature (Kokol and Podgorelec, 2000; Ausloos, 2012; Najafi and Darooneh, 2015) that used surface-level proxies, against which the paper's information-theoretic estimates are implicitly compared; and (5) the three context-length variants (2K, 4K, 8K) as baselines for the hypothesis that training with longer context improves capabilities.
Generation budget / compute accounting. No generation budget in the traditional sense—the paper does not compare inference-time strategies. Instead, the computational cost is in the surprisal extraction: each document's first 2,048 tokens are processed through the LLM to obtain conditional token probabilities. All experiments are executed on TPUs. The T5 baselines are pretrained for 500k steps (262B tokens) for the main Table 2 and Table 3 analyses, and for 100k steps (100B tokens) for the context length ablation. The paper does not provide FLOP counts for the fractal parameter estimation itself, which is computationally negligible compared to the LLM inference.
Cross-validation / statistical protocol. No train/test split is used for the fractal parameter estimation—parameters are computed directly on The Pile validation split documents and reported with bootstrapped error margins (Efron and Tibshirani, 1994). The downstream performance prediction analysis uses all 12 models for linear regression and reports adjusted $R^2$ values (which inherently penalize for the number of predictors) rather than using a held-out set of models. For the context length ablation, each configuration (2K, 4K, 8K) is a single training run rather than an average over multiple seeds, so no statistical error bars are reported for those results.

Main Quantitative Results

Self-Similarity Is Confirmed Across All Domains with a Median Hölder Exponent of S = 0.59

Figure 2 (Section 2.2) plots the peak probability $p_\epsilon(\tau)$ against the granularity level $\tau$ using surprisal values computed by PaLM2-L. The log-log relationship is clearly linear across all eight qualifying domains, confirming the power-law behavior $p_\epsilon(\tau) \sim \tau^{-S}$ that is the defining signature of self-similar processes. The slopes (and thus the S estimates) vary across domains, but all exhibit the characteristic linear decay in log-log space. The median self-similarity exponent across the eight domains is S = 0.59 ± 0.08 (Section 2.2, with full per-domain breakdowns in Table 1). Individual domains range from S = 0.42 ± 0.03 (DM-Mathematics) to S = 0.70 ± 0.03 (ArXiv). The estimate is robust to the choice of threshold $\epsilon$ : Figure 4 (left) shows that S estimates are stable for $\epsilon \in [10^{-3}, 10^{-2}]$ , with the paper fixing $\epsilon = 5 \times 10^{-3}$ for all reported results. The derived fractal dimension is $D = 2 - S = 1.41 \pm 0.08$ (Section 2.4), placing language's information-theoretic complexity between a smooth curve (D = 1) and space-filling Brownian motion (D = 2), and comparable to Mandelbrot's estimate of D = 1.25 for the coast of Britain.

Long-Range Dependence Is Confirmed with a Median Hurst Parameter of H = 0.70

Figure 3 (Section 2.3) plots the rescaled range $R(n)/S(n)$ against the window size $n$ for the increment process derived from PaLM2-L surprisal values. Again, the log-log relationship is clearly linear across all domains, confirming the power-law growth $R(n)/S(n) \sim n^H$ that characterizes processes with Hurst exponent H. The median Hurst parameter across the eight domains is H = 0.70 ± 0.09 (Section 2.3), with per-domain values in Table 1 ranging from H = 0.50 ± 0.01 (DM-Mathematics) to H = 0.79 ± 0.01 (GitHub). The value H = 0.70 is substantially above the H = 0.5 threshold that would indicate independent increments (pure noise), establishing that language exhibits statistically significant long-range dependence—events far apart in a document are correlated, with no finite cutoff beyond which the correlation becomes negligible. The DM-Mathematics result (H = 0.50) serves as an internal validation: documents in this domain consist of independent mathematical questions concatenated together (Figure 6), so the methodology correctly identifies the absence of cross-question dependence.

The paper contextualizes H = 0.70 by noting that this value falls in the same range as many natural phenomena (river discharges, Ethernet traffic, temperatures, precipitation, tree rings—Section 1, citing Crovella and Bestavros, 1995; Aref, 1998; Feller, 1951), and that it represents a "sweet spot" between predictability (H = 1) and noise (H = 0.5) that psychological theories suggest facilitates learning (Kidd and Hayden, 2015). The mathematical consequence of H > 0.5 is that the autocovariance function decays slowly enough that $\sum_{k=1}^\infty \rho_k = \infty$ —the cumulative influence of the infinite past on the present is unbounded, meaning longer context windows always provide additional predictive power (Section 2.3).

The Joseph Exponent Shows Burstiness with J = 0.49

Figure 5 (Section 2.5) plots the standard deviation $\sigma_\tau$ of $\tau$ -increments of the integral process against $\tau$ , showing another power-law relationship $\sigma_\tau \sim \tau^J$ with median J = 0.49 ± 0.08. Per-domain values in Table 1 range from J = 0.28 ± 0.00 (DM-Mathematics) to J = 0.52 ± 0.00 (Wikipedia). The fact that $J \approx 0.5$ means the volatility scaling is close to that of a standard random walk, even though the process exhibits both self-similarity (S = 0.59) and long-range dependence (H = 0.70). This divergence between J, S, and H indicates that language is not well-modeled by fractional Brownian motion (where all three parameters would coincide), and suggests multifractal structure that the paper flags as a potential direction for future investigation. Operationally, J < H means that the visual burstiness (Joseph effect) is less extreme than what the long-range dependence alone would predict—the process has long memory, but the clustering of high and low values is more moderate than in, say, river discharge data where extreme multi-year droughts and floods occur.

Fractal Parameters Are Robust Across Model Architectures and Scales

Table 2 (Section 3) reports the estimated median fractal parameters for all 12 models over the entire Pile validation split. The median Hurst parameter H is remarkably stable: T5-Decoder models (110M to 5B) all cluster at H = 0.64 ± 0.08, PaLM models (8B to 540B) range from H = 0.66 ± 0.07 to H = 0.68 ± 0.07, and PaLM2 models (XXS to L) range from H = 0.66 ± 0.07 to H = 0.69 ± 0.09. The self-similarity exponent S shows more variation, from S = 0.56 ± 0.05 (PaLM2-S) to S = 0.64 ± 0.08 (PaLM1-540B), but all values fall in a consistent range. The Joseph exponent J is the most stable, ranging from J = 0.44 ± 0.06 (T5-Decoder 110M and 340M) to J = 0.49 ± 0.08 (PaLM2-L). The full per-model, per-domain tables are provided in Appendix B (Tables 4–7), allowing verification of the robustness claim at the granularity of individual domain-model pairs.

Critically, while the median parameters are robust, systematic within-family trends are visible. The median Hurst parameter shows a clear monotonic increase with model quality: within PaLM1, H goes from 0.66 (8B) to 0.68 (62B and 540B); within PaLM2, H goes from 0.66 (XXS), to 0.66 (XS), to 0.67 (S), to 0.68 (M), to 0.69 (L). This monotonicity is what enables the downstream prediction result discussed below—the tiny variations that survive the overall robustness are informative about model quality.

The Cross-Domain Analysis Reveals Meaningful Domain-Specific Variation

Table 1 (Section 2.4) provides per-domain fractal parameter estimates using PaLM2-L. The results reveal several patterns:

GitHub (code) has the highest Hurst parameter at H = 0.79 ± 0.01, indicating stronger long-range dependence in code than in natural language. The paper notes this is "in agreement with earlier findings by Kokol and Podgorelec, 2000 who estimated LRD in computer languages to be greater than in natural language" (Section 3). The self-similarity exponent for GitHub is S = 0.60, near the cross-domain median, and the Joseph exponent is J = 0.49.
DM-Mathematics is the clear outlier: S = 0.42 ± 0.03 (lowest), H = 0.50 ± 0.01 (at the noise threshold), J = 0.28 ± 0.00 (lowest). These values reflect the domain's structure as a concatenation of independent math problems (Figure 6). The Hurst parameter of exactly 0.50 validates that the R/S method correctly identifies the absence of long-range dependence. Figure 4 (right) reinforces this by showing the partial autocorrelation function: DM-Mathematics shows markedly less dependence across time lags compared to all other domains.
ArXiv has the highest self-similarity exponent at S = 0.70 ± 0.03, along with H = 0.72 ± 0.01 and J = 0.49 ± 0.00. This suggests that academic writing has particularly strong scale-invariant statistical structure, possibly reflecting the highly structured organization of scientific papers (abstract, introduction, methods, results, discussion) that creates recursive patterns at multiple levels of granularity.
Natural language domains (OpenWebText2, FreeLaw, Pile-CC, Wikipedia, PubMed) cluster in a relatively narrow range: S ≈ 0.53–0.62, H ≈ 0.65–0.74, J ≈ 0.44–0.52. The consistency across these diverse domains—ranging from informal web text (OpenWebText2) to legal documents (FreeLaw) to biomedical literature (PubMed)—supports the claim that the fractal structure is a fundamental property of natural language, not an artifact of a particular genre or register.

The Hurst Parameter and BPB Together Predict Downstream Performance Better Than BPB Alone

Table 3 (Section 3, middle columns) and Figure 7 present the downstream performance prediction results across all 12 models. The headline finding: the combined metric $H_B = 1/\text{BPB} + H_{\text{median}}$ consistently yields higher adjusted $R^2$ values than BPB alone for predicting downstream benchmark scores. Specific improvements include:

0-shot BBH Direct: Adjusted $R^2$ increases from 0.785 (BPB alone) to 0.883 ( $H_B$ ). The median Hurst alone achieves 0.841—already better than BPB alone.
3-shot BBH Direct: 0.767 → 0.895 (H alone) → 0.926 ( $H_B$ ).
3-shot BBH CoT: 0.881 → 0.892 (H alone) → 0.979 ( $H_B$ ). This is the largest improvement, with $H_B$ explaining nearly 98% of the variance in 3-shot BBH CoT performance.
Few-shot aggregate (FS BBH+MMLU+GSM8K): 0.717 → 0.890 (H alone) → 0.891 ( $H_B$ ). Here, $H$ alone already provides most of the gain, with BPB adding minimal additional predictive power for the aggregate metric.
5-shot MMLU: 0.660 → 0.853 (H alone) → 0.832 ( $H_B$ ). This is one of the few cases where $H_B$ slightly underperforms H alone, though both substantially exceed BPB alone.

Several patterns in these results are notable. First, $H$ alone is a stronger predictor than BPB alone for every single benchmark—the adjusted $R^2$ for H exceeds that for BPB across all eight metrics. This is striking because H is computed after normalizing all surprisal values to zero mean and unit variance, meaning it contains no information about the absolute scale of the model's prediction errors. Second, the improvement from adding BPB to H (i.e., going from H alone to $H_B$ ) is benchmark-dependent: it is substantial for 3-shot BBH CoT (0.892 → 0.979) but negligible for the few-shot aggregate (0.890 → 0.891). This suggests that for some capabilities (particularly chain-of-thought reasoning), the absolute quality of the model's predictions (BPB) and the fractal structure of its errors (H) capture complementary information, while for aggregate knowledge-intensive benchmarks, the fractal structure already captures most of what matters.

The paper also reports that "the median Hurst parameter is itself strongly correlated with the BPB scores with an absolute Pearson correlation coefficient of 0.83" (Section 3). This is described as "surprising" because H is computed on normalized surprisals—there is no mathematical necessity for a second-order statistic (how variance scales with window size) to predict a first-order statistic (the mean surprisal). The fact that it does suggests that improving a model's average predictive accuracy and improving the fractal structure of its surprisal sequence are deeply linked processes: better models don't just make fewer errors, they make differently structured errors whose temporal organization more closely matches natural language. The self-similarity exponent S, by contrast, has only a weak correlation with BPB ( $|r| = 0.23$ ), and the Joseph exponent J is not reported for correlation.

Figure 7 visualizes the joint relationship by plotting each benchmark as a bubble (size proportional to accuracy) in the space of median H vs. median BPB for all 12 models. The pattern shows that models in the upper-left region (higher H, lower BPB) consistently have larger bubbles (better downstream performance), while the T5 models cluster in the lower-right (lower H, higher BPB) with smaller bubbles.

The Self-Similarity and Joseph Exponents Do Not Improve Downstream Performance Prediction

Table 9 (Appendix C) reports adjusted $R^2$ values when combining BPB with S or with J. The results are consistently negative: neither S nor J provides any improvement over BPB alone, and in some cases the combined metric performs worse than BPB alone. For example:

0-shot BBH Direct: BPB alone achieves 0.785; BPB+S achieves 0.761; BPB+J achieves 0.794 (essentially unchanged).
3-shot BBH CoT: BPB alone achieves 0.881; BPB+S achieves 0.870; BPB+J achieves 0.879.
Few-shot aggregate: BPB alone achieves 0.717; BPB+S achieves 0.687 (worse); BPB+J achieves 0.686 (worse).

The fact that S and J provide no predictive power, while H does, is informative: it suggests that the specific aspect of fractal structure captured by the Hurst parameter—the scaling of long-range dependence—is what matters for downstream capabilities, not self-similarity or burstiness per se. This is consistent with the theoretical argument that LRD is what forces models to learn long-range dependencies (since short-range processes could be captured by finite-context lookup tables), while self-similarity alone (as in Brownian motion, which has S = 0.5 but H = 0.5 and independent increments) would not confer any advantage.

Training with Longer Context Does Not Improve Downstream Performance

Table 3 (Section 3, right columns) presents the negative result from the context length ablation. Three decoder-only T5.1.1 models with 1B parameters were pretrained on SlimPajama-627B for 100B tokens with context lengths of 2K, 4K, and 8K tokens. Across all downstream benchmarks, there is no consistent improvement:

0-shot MMLU: 25.73 (2K) vs. 26.04 (4K) vs. 25.81 (8K)—flat.
3-shot BBH Direct: 21.35 (2K) vs. 24.76 (4K) vs. 23.14 (8K). The 4K model performs best, but the 8K model is worse than the 4K model.
3-shot BBH CoT: 16.87 (2K) vs. 12.21 (4K) vs. 7.14 (8K)—performance declines with longer context, which is unexpected and unexplained.
8-shot GSM8K CoT: 1.06 (2K) vs. 1.21 (4K) vs. 1.74 (8K)—a slight upward trend, but all scores are near zero for these 1B models on a difficult math benchmark.

The authors present this as a negative result without extensive interpretation (Section 3): "we do not observe any improvements in performance with context length in this particular setup." This is significant because it constrains the theoretical narrative: while language has long-range dependence (H > 0.5), it does not automatically follow that explicitly training with longer contexts improves model quality—at least at the 1B parameter, 100B token scale. The paper does not explore whether the result would change with larger models, more training tokens, or different architectures.

Ablation Studies and Robustness Checks

Choice of threshold $\epsilon$ in the peak-probability method for S: Figure 4 (left) demonstrates that the self-similarity exponent estimate is stable across a range of small thresholds $\epsilon \in [10^{-3}, 10^{-2}]$ . The paper fixes $\epsilon = 5 \times 10^{-3}$ for all reported results. This robustness check matters because the peak-probability method requires choosing a band around zero within which to count $\tau$ -increments, and an arbitrary or sensitive choice would undermine confidence in the S estimates.

Model architecture and scale (T5 vs. PaLM vs. PaLM2): Table 2 and Appendix B (Tables 5–7) show that fractal parameter estimates are generally consistent across three model families spanning three orders of magnitude in parameter count (110M to 540B+). The median Hurst parameter varies from 0.64 (T5 family) to 0.69 (PaLM2-L), while the self-similarity exponent ranges from 0.56 to 0.64. This robustness to architecture is non-trivial: since the surprisal sequences are model-specific outputs, different architectures could in principle produce increment processes with different fractal structure even when evaluated on the same text. The fact that they largely agree suggests that the fractal properties are primarily determined by the data rather than the model.

Domain variation (8 domains in The Pile): Table 1 and Appendix B (Tables 4–7) provide per-domain estimates with bootstrapped standard errors for all 12 models. This granular breakdown serves as a domain-level robustness check: while DM-Mathematics is a clear outlier (expected, given its structure as independent questions) and GitHub shows systematically higher H (expected, given prior findings about structure in code), the natural language domains cluster in a relatively narrow range (S ≈ 0.53–0.62, H ≈ 0.65–0.74). The fact that domain differences are interpretable (DM-Mathematics has H = 0.50 because it lacks narrative structure; GitHub has H = 0.79 because code has strong long-range dependencies) increases confidence that the methodology is measuring something real rather than producing spurious numerical values.

Oracle vs. predicted difficulty estimation is not applicable: Unlike the paper from the reference example (which compared oracle vs. predicted difficulty bins for compute-optimal strategies), this paper does not have a comparable ablation. The fractal parameters are estimated directly from the data without any need for ground-truth labels, so there is no train/test distinction for the parameter estimation itself.

Alternative fractal parameters (S, J) for downstream prediction: Table 9 (Appendix C) tests whether the self-similarity exponent S or the Joseph exponent J can improve upon BPB for predicting downstream performance. The result is negative: adjusted $R^2$ values for BPB+S and BPB+J are essentially unchanged from BPB alone, and in some cases slightly worse. This establishes that the predictive power is specific to the Hurst parameter H, not a generic property of all fractal parameters. It also rules out the explanation that any transformation of the surprisal sequence that captures higher-order statistics would improve downstream prediction—S and J also capture higher-order structure, but they don't help.

Context length during training (2K vs. 4K vs. 8K): Table 3 (right columns) presents the negative result discussed above. This serves as an ablation on the hypothesis that training with longer contexts is important for capturing fractal structure. The finding that downstream performance does not improve (and in some cases degrades) with longer context length suggests that while language has long-range dependence, the relationship between training context length and model capabilities is not straightforward—at least for 1B parameter models trained on 100B tokens. The paper does not ablate model size or training data quantity for this experiment, leaving open the possibility that the effect might emerge at larger scales.

Choice of aggregator (median vs. mean): The paper uses the median across domains as the summary statistic for each model's fractal parameters. This choice is motivated by robustness to outliers but is not ablated against alternatives (mean, trimmed mean, weighted average by domain size). Given the relatively small number of domains (8) and the presence of at least one clear outlier (DM-Mathematics), the choice of median is reasonable, but the sensitivity of the downstream prediction results to this choice is not tested.

Critical Assessment

Claim 1: Language is self-similar and long-range dependent, with S = 0.59 and H = 0.70

The evidence for this claim is strong within the scope of the experiments performed. The power-law relationships in Figures 2, 3, and 5 are visually clear—the log-log plots show clean linear trends across all domains, which is the defining signature of fractal scaling. The estimates are reasonably precise (bootstrapped standard errors of ±0.08–0.09) and are robust to model architecture and scale (Table 2). The DM-Mathematics result (H = 0.50) provides a compelling internal validation: when the methodology is applied to text known to lack narrative structure, it correctly identifies the absence of long-range dependence.

However, the claim requires qualification in several respects:

The values are model-dependent estimates of an underlying property. The fractal parameters are computed on surprisal sequences derived from PaLM2-L (and other models for the robustness analysis), not on the "true" conditional distribution of language. If PaLM2-L has systematic biases in its probability estimates—for instance, if it underestimates the probability of rare but predictable-in-hindsight tokens, or if its calibration varies with context length—those biases would affect the fractal parameter estimates. The paper acknowledges this implicitly by testing robustness across models, but the fact that H varies from 0.64 (T5) to 0.69 (PaLM2-L) shows that model quality affects the estimates. Which model's estimates best approximate the "true" fractal parameters of language? The paper doesn't answer this—it relies on consistency across models as evidence that the parameters are real, but the variation across models of different quality could also be interpreted as evidence that better models reveal different (and presumably more accurate) fractal structure. If PaLM2-L is not yet at the frontier of language modeling, the true H for human language might be even higher than 0.70.

The analysis is limited to English and to the first 2,048 tokens of documents. The paper explicitly acknowledges the English-only limitation (Section 4): "Our analysis is currently limited to the English language so it may not apply to other languages that differ significantly. For instance, some languages such as Pirahã (spoken in the Amazon) do not have a recursive structure like most languages do." This is a genuine constraint. If the fractal structure arises from the recursive, hierarchical organization of language—as the paper's theoretical argument suggests—then languages without recursive structure might show different (possibly non-fractal) patterns. The 2,048-token window is also a limitation: the power-law fits in Figures 2, 3, and 5 necessarily extrapolate asymptotic behavior from finite samples, and while 2,048 tokens provides a reasonable range of scales for detecting power-law relationships, it is not sufficient to characterize the asymptotic behavior at very large scales (e.g., book-length texts). The claim that "no particular characteristic context length" exists (Abstract) is an inference from the fractal structure at observed scales, not a direct measurement at all possible scales.

The "self-similar" claim refers to statistical self-similarity of the surprisal process, not self-similarity of language structure in a linguistic sense. This is an important conceptual distinction. The paper establishes that the information-theoretic surprisal sequence has scale-invariant statistical properties, but this is not the same as demonstrating that, say, the syntactic structure of a paragraph mirrors the discourse structure of a chapter in a mathematically precise way. The connection between surprisal self-similarity and linguistic self-similarity (recursive clause embedding, hierarchical topic structure) is asserted as an interpretation (Section 1: "self-similarity implies that the patterns at the level of a paragraph are reflective of the patterns seen at the level of a whole text") but is not experimentally verified. It is possible that the statistical self-similarity of surprisal sequences arises from different linguistic mechanisms at different scales, and that the apparent scale-invariance is an emergent property rather than a direct reflection of recursive linguistic structure.

Missing experiment: The paper does not test whether the fractal structure persists when text structure is artificially disrupted. For instance, shuffling paragraphs within a document should destroy long-range dependence (reducing H toward 0.5) while preserving local statistical properties, providing a direct test of whether the estimated H reflects genuine document-level structure. Similarly, scrambling sentences within paragraphs while preserving word order within sentences would test which level of linguistic organization contributes most to the observed fractal properties.

Claim 2: Self-similarity and LRD explain why next-token prediction yields broad capabilities

This claim is the paper's most ambitious and also its most difficult to evaluate from the evidence presented. The paper provides a coherent theoretical argument (summarized in Section 1 and detailed in the prior sections of this analysis): self-similarity means that learning local patterns helps with global patterns; LRD means that long context always provides useful information; the specific value H ≈ 0.70 places language in a regime that is learnable but not trivially predictable. However, no experiment directly tests the causal link between fractal structure and model capabilities.

The evidence for this claim is entirely observational and interpretive: the paper establishes that language has fractal properties, and argues that this would explain why models trained on next-token prediction develop multi-scale capabilities. The gap between these two statements is large. To actually test the claim, one would need experiments like:

Manipulating the fractal structure of training data: Generate synthetic text with controlled Hurst parameters (H = 0.5, 0.6, 0.7, 0.8) by manipulating the dependency structure of token sequences, train identical models on each variant, and measure whether higher H leads to better downstream capabilities.
Measuring whether model capabilities correlate with how well the model's internal representations capture fractal structure: Do models that achieve higher accuracy on long-range dependency benchmarks also produce surprisal sequences with H closer to 0.70?
Testing the transfer-across-scales hypothesis directly: Train a model only on short sequences (e.g., 128 tokens) and test whether it transfers to longer sequences (e.g., 2,048 tokens) on tasks requiring multi-scale understanding. The self-similarity hypothesis predicts better transfer for language than for a non-self-similar process with the same local statistics.

The existing experiments provide suggestive but indirect support. The fact that the Hurst parameter predicts downstream performance better than BPB alone (Table 3) is consistent with the claim—it shows that models whose surprisal structure better matches natural language's fractal properties are also better at downstream tasks—but it does not establish that the fractal structure causes the capabilities. The negative context-length result (Table 3, right) is actually inconsistent with a strong version of the claim: if LRD means that longer context should always help, why does training with 8K context not improve over 2K context? The paper does not reconcile this tension.

The DM-Mathematics example (H = 0.50) is sometimes presented as support for the claim, but upon closer inspection it is orthogonal: DM-Mathematics lacks long-range dependence because its documents are concatenations of independent questions. A model trained on this domain would not need to attend across question boundaries because there is no cross-question structure to learn. This is a case where absence of LRD corresponds to absence of need for long-range modeling, which is consistent with the theoretical argument but does not test the positive direction (that presence of LRD causes need for and benefit from long-range modeling).

The strongest reading of the paper's contribution on this claim is that it provides a formal vocabulary and quantitative framework for stating the hypothesis precisely, backed by rigorous measurement of the fractal parameters that the hypothesis references. Whether the hypothesis is true remains an open question that the paper's experiments do not directly address.

Claim 3: Fractal parameters (specifically H) improve downstream performance prediction over BPB alone

This claim is well-supported by the adjusted $R^2$ analysis in Table 3, but with important caveats about the experimental design.

Strengths: The improvement is consistent across all eight downstream metrics tested (Table 3, middle columns). The magnitude of improvement is substantial for several benchmarks (adjusted $R^2$ from 0.785 to 0.883 for 0-shot BBH Direct; from 0.881 to 0.979 for 3-shot BBH CoT). The combined metric $H_B$ is simple—a linear combination with equal weights—which reduces concerns about overfitting from complex functional forms. The fact that H alone outperforms BPB alone on every benchmark is striking and not easily explained away.

Weaknesses:

Small sample size for regression. The analysis uses 12 models as data points. Adjusted $R^2$ penalizes for the number of predictors, but with only 12 observations, the regression is fitting a line to a very small number of points. The high $R^2$ values could be driven by a few influential points—for instance, the T5 models cluster in one region of the H-BPB space while the PaLM2 models cluster in another, and the regression is essentially fitting a line between two clusters. The paper does not report leave-one-out cross-validation or any out-of-sample prediction, which would more rigorously test whether $H_B$ generalizes to predicting the performance of a held-out model. With only 12 models, standard cross-validation would be noisy, but reporting it would still provide some indication of robustness.
The models are not a random sample. The 12 models are drawn from three families (T5, PaLM, PaLM2) with systematic differences in architecture, training data, training procedure, and scale. The relationship between H, BPB, and downstream performance might reflect family-level confounds rather than a universal relationship. For example, if PaLM2 models are simply better than T5 models in ways not fully captured by BPB (perhaps due to training data quality or architectural innovations), and PaLM2 models also happen to have higher H, then H could appear predictive without actually capturing anything fundamental about model quality. The fact that H varies monotonically within each family (Table 2) partially addresses this concern, but the within-family sample sizes are very small (4 T5 models, 3 PaLM models, 5 PaLM2 models).
The combined metric $H_B$ is not compared against other simple improvements. Adding a new scalar predictor (H) to an existing scalar predictor (1/BPB) will always improve in-sample $R^2$ unless the new predictor is perfectly collinear with the old one. The question is whether H adds more predictive power than other simple additions—for instance, model size (log parameters), training tokens, or the self-similarity exponent S. Table 9 shows that S and J do not help, which is informative, but the paper does not compare H against, say, a linear combination of BPB and log(model size). If model size also improves $R^2$ , the unique contribution of H would be less clear.
The correlation between H and BPB (|r| = 0.83) complicates interpretation. The paper acknowledges this correlation and argues that H and BPB capture complementary information (since adding H to BPB improves $R^2$ ). But with |r| = 0.83, the two variables are largely redundant. The improvement in $R^2$ from adding H to BPB means that the 17% of variance in H that is not explained by BPB is predictive of downstream performance. This is interesting, but it means the unique contribution of H is smaller than the raw $R^2$ numbers suggest. A variance partitioning analysis (how much variance does H explain beyond BPB, and vice versa) would have been informative.

Missing experiment: The paper would be strengthened by a true held-out evaluation—for instance, fitting the regression on 10 models and predicting the 11th, or fitting on all T5 and PaLM models and predicting PaLM2 models. This would directly test whether the H-BPB relationship generalizes out of distribution to new model families, which is the scenario of practical interest (predicting the performance of a new model without evaluating it on downstream benchmarks).

Claim 4: Fractal parameters are robust across architectures and domains

This claim is strongly supported for the models and domains tested. Table 2 shows that median H varies by only ±0.025 around 0.67 across 12 models from three families. The domain-level breakdowns in Table 1 and Appendix B show that the natural language domains cluster in a narrow range, with interpretable outliers (DM-Mathematics, GitHub).

Caveats:

Robustness is assessed on median values across 8 domains. Within individual domains, there is more variation across models (Appendix B, Tables 5–7). For example, the Hurst parameter for GitHub ranges from 0.79 (PaLM2-L, PaLM1-540B) to 0.83 (T5-Decoder-1B). The median aggregates out this variation, so the robustness claim applies at the level of the model's overall fractal signature, not at the level of individual domain-model pairs.
All models are English-language, Transformer-based, and trained on similar web-text corpora. The robustness claim does not extend to models trained on non-English text, non-Transformer architectures, or models trained with objectives other than causal language modeling. The paper's theoretical argument suggests that fractal structure is a property of the data (language), not the model, so any model that achieves reasonable predictive accuracy should produce similar estimates. This is plausible but untested.
The robustness of S and J is weaker than that of H. In Table 2, median S ranges from 0.56 (PaLM2-S) to 0.64 (PaLM1-540B), a range of 0.08—double the range for H (0.64 to 0.69, range of 0.05). This suggests that self-similarity estimates are more sensitive to model differences than Hurst parameter estimates, which is not explored.

Overarching Assessment

The paper's central empirical contribution—establishing and quantifying the fractal structure of language using an information-theoretic representation—is well-executed and convincing within its scope. The power-law evidence is visually clear, the parameter estimates are reasonable and consistent, and the robustness checks across models and domains are thorough. The downstream prediction result is intriguing and practically significant, though the small sample size and the high correlation between H and BPB mean that the unique predictive power of H should be interpreted cautiously.

The paper's interpretive claim—that fractal structure explains why next-token prediction works—is stimulating but not experimentally established. It is a hypothesis that the measurements make precise and testable, but the paper's experiments do not directly test it. The negative context-length result (Table 3, right) is the only experiment that bears directly on the causal link between fractal structure and model training, and it is inconsistent with a naive version of the hypothesis (longer training context → better capabilities). The paper does not fully reconcile this tension.

The most significant limitation of the experimental design, beyond those discussed above, is the absence of direct manipulation experiments. The paper takes the fractal structure of language as given, measures it, and interprets its implications. But to truly establish that fractal structure is causally important for model capabilities (rather than merely correlated with them), one would need to create training data with modified fractal parameters and measure the effect on learned capabilities. This is a challenging experiment—synthetic text with controlled long-range dependence is non-trivial to generate—but without it, the paper's most ambitious theoretical claim remains an interpretation rather than a demonstrated fact. The authors implicitly acknowledge this by framing their contribution as "offering a fresh perspective" (Abstract) and "highlighting intriguing insights" (Section 5) rather than claiming to have proven the causal mechanism.

6. Limitations and Trade-offs

The Causal Link Between Fractal Structure and Model Capabilities Is Not Experimentally Tested

The assumption or constraint. The paper's most ambitious claim—that the self-similar, long-range dependent structure of language "may illuminate the success of next-token prediction" (Abstract) and explains why LLMs "develop proficiency in capturing patterns across multiple levels of granularity" (Section 1)—is an interpretive hypothesis, not an experimentally validated mechanism. The paper establishes that language exhibits fractal properties and argues that this would explain why next-token prediction works, but it conducts no experiment that manipulates fractal structure and measures the effect on learning. The authors frame their contribution carefully as "offering a fresh perspective" (Abstract) and "highlighting intriguing insights" (Section 5), but the gap between measurement and causal demonstration is the central limitation of the work.

The consequence. Without a causal experiment, the paper's explanatory claim remains at the level of a plausible but unverified narrative. The fractal properties could be epiphenomenal—an emergent statistical signature of language that correlates with, but does not cause, the learnability that next-token prediction exploits. For example, both the Hurst parameter and model capabilities could be driven by a third factor (the hierarchical structure of human knowledge, the communicative intent behind text production, the statistical mechanics of optimal codes) without any direct causal relationship between them. A practitioner cannot conclude from this paper that training on text with specific fractal parameters will yield better models, or that engineering the fractal structure of training data is a promising direction for improving LLMs—the evidence supports neither claim. The paper's theoretical story is coherent and suggestive, but it is a post-hoc interpretation of measurements, not a predictive theory that has been tested.

What evidence exists in the paper. The only experiment that bears even indirectly on the causal question is the context-length ablation (Section 3, "Context Length at Training Time," Table 3 right columns), and it produces a negative result: training with longer contexts (4K or 8K vs. 2K) on SlimPajama-627B for 100B tokens does not improve downstream performance, and in some cases degrades it (3-shot BBH CoT declines from 16.87 to 7.14 as context grows from 2K to 8K). This is inconsistent with a naive interpretation of the LRD hypothesis—if long-range dependence means longer context always provides useful information, why doesn't longer-context training help? The paper does not reconcile this tension. The authors present the result without extensive interpretation, which is appropriate given the negative finding, but it means that the one experiment designed to probe the causal direction of the fractal hypothesis does not support it. No experiment manipulates the fractal parameters of training data, measures whether models with internal representations that better match fractal structure perform better, or tests whether transfer across scales (learning from short sequences and generalizing to long ones) is specifically enabled by self-similarity.

Mitigation status. Not addressed. The paper does not attempt a causal experiment and does not acknowledge this as a limitation in Section 4 (Limitations and Future Research) or Section 5 (Concluding Remarks). The limitations discussed in Section 4 focus on English-only scope, the absence of semantic/lexical modeling, and the potential for exploiting self-similarity in architecture design—none of which address the causal gap. The paper's contribution is explicitly observational ("quantify properties that may have been previously suspected but not formally shown," Abstract), which is valuable, but the leap from "language has fractal properties" to "this explains why LLMs work" requires causal evidence that is entirely absent. A reader should understand that the paper provides a measurement and a hypothesis, not an explanation that has been experimentally validated.

Difficulty Estimation Cost Is Not Accounted For and Would Dominate Deployment Budgets

The assumption or constraint. The paper's methodology for estimating fractal parameters requires running every document through a large language model (PaLM2-L by default) to obtain token-level conditional probability scores, then computing the surprisal sequence and fitting power laws. The authors state that "we use PaLM2-L (Unicorn) to calculate the probability of the next token $w_t$ conditioned on its entire prefix" and that these calculations are performed "for prefixes of up to 2048 tokens" (Section 2.1). Every document in the analysis dataset (thousands of documents across eight domains, filtered to >4K tokens each, using the first 2K tokens) must be processed through a model with hundreds of billions of parameters. The computational cost of this inference is substantial—comparable to running a production LLM inference workload on a large validation corpus—and the paper does not account for or discuss this cost in any quantitative way.

The consequence. For a practitioner interested in using fractal parameters as a model evaluation metric (as the paper's downstream prediction results in Table 3 suggest is valuable), the cost of computing those parameters may dominate whatever savings they enable. Unlike bits-per-byte (BPB), which can be computed as a byproduct of standard evaluation pipelines, the Hurst parameter requires: (1) full forward passes through an LLM for every document in the evaluation corpus, (2) constructing increment and integral processes per document, and (3) fitting power laws via linear regression in log-log space. Step (1) dominates the cost. If the goal is to monitor H during training to complement perplexity-based early stopping or model selection, the inference cost per checkpoint evaluation would be comparable to running a full validation pass—doubling the effective cost of validation or requiring a dedicated evaluation pipeline.

The paper also proposes using the combined metric $H_B = 1/\text{BPB} + H$ to predict downstream performance (Table 3). But if computing H requires running a large evaluation corpus through the model, the practical workflow becomes: compute BPB (cheap, part of standard validation), then decide whether the additional cost of computing H is justified by the improvement in predictive accuracy (adjusted $R^2$ from ~0.72 to ~0.89 for the few-shot aggregate). For many production settings, the cost of the H computation may exceed the value of the improved prediction—particularly since BPB alone already achieves $R^2 \approx 0.72$ , which may be sufficient for coarse model selection decisions. The paper provides no cost-benefit analysis to guide this tradeoff.

What evidence exists in the paper. The paper does not report the computational cost of the surprisal extraction pipeline. It states that "all experiments are executed on Tensor Processing Units (TPUs)" (Section 2.1) and provides training details for the T5 baselines (Appendix A: 256 TPUv5e chips), but there are no FLOP counts, wall-clock times, or dollar-cost estimates for the fractal parameter estimation itself. The number of documents, their lengths, the number of models evaluated (12), and the number of domains (8) are all specified, allowing a rough estimate, but the paper provides no guidance on whether the pipeline is practical for routine use. The fact that the analysis is conducted on a validation split (The Pile validation set, not the full dataset) helps, but for a model developer who wants to track H across many training checkpoints, the cumulative cost would be substantial.

Mitigation status. Not addressed. The paper does not discuss the computational cost of its measurement pipeline, does not propose approximations or cheaper estimators for the fractal parameters, and does not include this as a limitation in Section 4. This is a significant gap for the practical applicability of the downstream prediction result, which is otherwise the paper's most actionable contribution. A natural mitigation—training a lightweight model to predict the Hurst parameter directly from model weights or from a small sample of surprisal values, rather than requiring full-corpus inference—is not explored. The paper gestures at related future work in a different context ("exploiting self-similarity more directly in LLMs could lead to further optimizations," Section 4), but this does not address the measurement cost problem.

The Analysis Is Limited to English, Transformer Architecture, and a Single Causal Language Modeling Objective

The assumption or constraint. The paper's empirical analysis uses 12 models drawn from three families—T5 Decoder-Only, PaLM, and PaLM2—all of which are Transformer-based causal language models trained primarily or exclusively on English text. The authors explicitly acknowledge the language limitation: "Our analysis is currently limited to the English language so it may not apply to other languages that differ significantly. For instance, some languages such as Pirahã (spoken in the Amazon) do not have a recursive structure like most languages do" (Section 4). They do not, however, address the architectural and objective limitations with the same specificity.

The consequence. The paper's central empirical claim—that language, as a general phenomenon, exhibits specific fractal parameters (S = 0.59, H = 0.70)—is only tested on English text as processed by Transformer-based autoregressive models. This creates three distinct generalization gaps:

Language gap: The authors cite Pirahã as a language that "does not have a recursive structure like most languages do" (Section 4, citing Everett, 2005), and the theoretical argument explicitly links fractal structure to linguistic recursion ("reminiscent of the recursive structure of language," Section 1, citing Perfors et al., 2010). If the fractal properties arise from recursion and hierarchical organization, then non-recursive languages would show different (possibly non-fractal) patterns, which would undermine the claim that fractal structure is a universal property of language rather than a property of languages with particular structural features. The paper provides no evidence about how fractal parameters vary across typologically diverse languages, so a reader cannot assess whether the findings are specific to English (and languages with similar recursive structure) or genuinely universal.

Architecture gap: All 12 models are Transformers. It is possible that the observed fractal parameters are partially an artifact of how Transformers represent and process language, rather than a pure reflection of language structure. A recurrent architecture (LSTM, Mamba) or a state-space model might produce different surprisal sequences on the same text because its inductive biases lead to different probability estimates, particularly for long-range dependencies. The paper's robustness check across model scales (Table 2) shows that H varies from 0.64 (T5) to 0.69 (PaLM2-L)—which the paper interprets as "generally robust"—but the 0.05 gap between the weakest and strongest models could reflect architectural differences as much as quality differences. Without testing on non-Transformer architectures, it is unknown whether the fractal estimates would converge to the same values or diverge systematically.

Objective gap: All models are trained with a causal (autoregressive) language modeling objective. Masked language models (BERT, T5 encoder-decoder), which learn bidirectional representations, may produce different token-level probability estimates because they condition on both left and right context. The surprisal sequence from a bidirectional model would encode different information about predictability, and its fractal parameters could differ from those of a causal model even on the same text. The paper's reliance on causal models is natural given the research question (next-token prediction), but it means the measured fractal parameters are specific to a particular probabilistic decomposition of language, not an intrinsic property independent of modeling choices.

What evidence exists in the paper. No cross-linguistic analysis is performed. No non-Transformer architecture is tested. No non-autoregressive objective is evaluated. The paper's robustness analysis in Table 2 compares 12 models that all share these properties, so it demonstrates robustness within the Transformer-autoregressive-English envelope but cannot speak to behavior outside it. The negative context-length result (Table 3, right) uses the T5.1.1 decoder-only architecture exclusively. The paper does not ablate architecture or objective in any experiment.

Mitigation status. The language limitation is acknowledged in Section 4 ("Limitations and Future Research"), and the authors identify cross-linguistic analysis as "a promising direction for future research." The architectural and objective limitations are not acknowledged. A reader considering whether the findings generalize to, say, a Chinese BERT model or a Finnish Mamba model has no evidence to rely on. The paper's claim about "language" writ large should therefore be understood as a claim about "English as modeled by Transformer-based autoregressive models," pending replication in broader settings.

The Downstream Performance Prediction Result Uses Only 12 Models with No Out-of-Sample Validation

The assumption or constraint. The paper's claim that the combined metric $H_B$ "improves upon perplexity-based bits-per-byte (BPB) in predicting downstream performance" (Abstract, Section 3) is based on linear regressions fit to 12 data points (the 12 models in Table 2) and evaluated using in-sample adjusted $R^2$ . No out-of-sample prediction is performed, and the models are not an independent random sample—they are drawn from three model families (T5, PaLM, PaLM2) that differ systematically in architecture, training data, training procedure, and scale. The adjusted $R^2$ values reported (e.g., 0.979 for 3-shot BBH CoT using $H_B$ ) are in-sample goodness-of-fit statistics that do not guarantee predictive accuracy on held-out models.

The consequence. The headline improvement in predictive power—adjusted $R^2$ from ~0.65 to over 0.86 (Abstract)—may overstate the practical value of H as a predictor in realistic model evaluation settings. There are several reasons for concern:

Small sample instability: With 12 observations and up to 2 predictors (1/BPB and H), the regression is fitting a line to very few points. Adjusted $R^2$ corrects for the number of predictors but not for the small sample size itself. With 12 points, the regression can be heavily influenced by a single outlier or by the clustering of points (T5 models in one region, PaLM2 models in another). The paper does not report prediction intervals, standard errors on the coefficients, or any measure of uncertainty in the $R^2$ estimates.

No out-of-sample test: The claim that $H_B$ "predicts downstream performance better than BPB alone" would be much stronger if demonstrated on held-out models—for instance, fitting the regression on 10 of the 12 models and predicting the remaining 2, or fitting on one model family and predicting another. Without this, it is unclear whether $H_B$ would generalize to, say, a new model from a fourth architecture family, a model trained on non-English text, or a model of a scale not represented in the training set (the largest model is PaLM2-L at >500B parameters; predicting a 1T-parameter model would be extrapolation).

Model family confounds: The 12 models are not exchangeable observations. The T5 models (110M–5B) were trained by the authors on C4 for 262B tokens. The PaLM models (8B–540B) were trained by a different team with different data and procedures as described in Chowdhery et al. (2022). The PaLM2 models (XXS–L) were trained by yet another team as described in Anil et al. (2023b). These family-level differences could drive the correlation between H and downstream performance: PaLM2 models are better than T5 models in ways not fully captured by BPB (better training data, architectural improvements, more sophisticated training procedures), and they also happen to have higher H. The within-family trends (H increases monotonically with model size within PaLM and PaLM2 in Table 2) partially address this concern, but with only 3–5 models per family, these trends themselves could be coincidental. The paper does not report a formal test of whether H adds predictive power within families after controlling for family identity.

The high correlation between H and BPB (|r| = 0.83) means the unique contribution of H is modest: Most of the variance in downstream performance that $H_B$ explains is already explained by BPB. The improvement in adjusted $R^2$ from BPB alone to $H_B$ represents the predictive value of the ~30% of variance in H that is orthogonal to BPB. This is a genuine signal, but it is smaller than the headline numbers suggest, and with only 12 models, it could be driven by a few points where the BPB-downstream relationship is imperfect and H corrects it.

What evidence exists in the paper. Table 3 reports adjusted $R^2$ values for BPB alone, H alone, and $H_B$ across eight downstream metrics. Figure 7 visualizes the joint distribution of median H, median BPB, and downstream performance as a bubble chart. No cross-validation, held-out prediction, or uncertainty quantification is reported. The paper does include a negative control: Table 9 (Appendix C) shows that the self-similarity exponent S and the Joseph exponent J do not improve prediction when combined with BPB, and in some cases worsen it. This is useful evidence that the predictive power is specific to H rather than being a generic property of fractal parameters, but it does not address the out-of-sample generalizability concern.

Mitigation status. Not addressed. The paper does not discuss the small sample size, the lack of out-of-sample validation, or the model family confound as limitations. The concluding remarks (Section 5) state that "combining both H with BPB together yields a stronger predictor of downstream performance" as an established finding, without qualifying that this is based on in-sample fit to 12 models from three families. A reader evaluating whether to adopt H as a model selection metric should treat the result as suggestive but not validated—it demonstrates that H captures something about model quality beyond perplexity, but the magnitude of the improvement and its generalization to new model families remain uncertain without out-of-sample testing.

The Fractal Structure of Language Is Measured Through a Model-Specific Lens, and the "True" Parameters for Human Language Remain Unknown

The assumption or constraint. The paper's fractal parameter estimates are computed on surprisal sequences derived from a specific large language model (PaLM2-L by default). The choice of model is consequential: Section 3 reports that the median Hurst parameter varies from H = 0.64 (T5-Decoder-110M) to H = 0.69 (PaLM2-L) across the 12 evaluated models. The paper interprets this variation as "generally robust" and focuses on the consistency of the estimates, but it does not establish which model's estimates—if any—best approximate the "true" fractal parameters of human language processing. The underlying assumption is that LLM-derived surprisal sequences are a sufficiently good proxy for the intrinsic information content of language that the fractal parameters estimated from them reflect fundamental properties of language itself, not artifacts of the specific model used.

The consequence. There are two distinct problems this creates:

The scaling of estimates with model quality is unknown: If H = 0.64 for a 110M T5 model and H = 0.69 for a >500B PaLM2 model, does this trend continue? Would a hypothetical perfect language model (one that assigns the true conditional probabilities) yield H = 0.70, or H = 0.75, or H = 1.0? The paper's estimate of H = 0.70 for language is tied to PaLM2-L's surprisal distribution, but if better models systematically produce higher H, the "true" Hurst parameter of language (as a property of the optimal predictive distribution, not a particular model's approximation) could be substantially different from the reported value. This matters for the theoretical argument: if the true H is, say, 0.85 rather than 0.70, the assertion that language sits at a "sweet spot" of predictability (H ≈ 0.75 being common in nature) would need to be reevaluated. Conversely, if H converges to some asymptotic value as model quality improves, the paper provides no evidence for what that value is.

Model calibration errors could distort the fractal estimates: LLMs are known to be miscalibrated in various ways—they can be overconfident on some predictions and underconfident on others, and calibration can vary with token position, context length, and domain. The paper cites Kadavath et al. (2022) to support the claim that "LLMs are known to produce calibrated probability scores at the token level" (Section 2.1), but this claim is about aggregate calibration (does the model's reported probability of 0.7 correspond to ~70% empirical frequency?), not about whether the calibration is uniform across all tokens and contexts. Systematic calibration errors—for instance, if the model systematically underestimates the probability of tokens that start new topics or overestimates the probability of tokens in formulaic phrases—would introduce structure into the surprisal sequence that is not present in the true conditional distribution. This structure would be picked up by the fractal analysis and attributed to language, when it is actually an artifact of the model.

What evidence exists in the paper. The robustness check in Table 2 is the primary evidence that fractal parameters are stable across models, and it is genuinely reassuring—the fact that models from 110M to >500B parameters produce H estimates within a ~0.05 range suggests that the fractal structure is not purely a model artifact. However, the systematic trend (better models → higher H) is also visible and is not explored: why does H increase with model quality? Does it approach an asymptote? Is the increase because better models capture more of language's true long-range dependence, or because their specific error patterns create spurious long-range structure? The paper does not investigate these questions.

The domain-level analysis in Table 1 provides some indirect evidence that the measured parameters reflect real text structure: DM-Mathematics (independent questions) yields H = 0.50, which is exactly the value expected for a process with no long-range dependence, and this holds across all models (Appendix B, Table 6). This suggests that the R/S method is measuring something real about text organization, not just model noise. But the gap between "it measures something real" and "the specific numeric value of H = 0.70 is the true Hurst parameter of human language" is substantial.

Mitigation status. The paper does not address this limitation directly. The authors present the fractal parameters as properties of language that their methodology reveals, not as model-dependent estimates that may converge (or fail to converge) to true values as model quality improves. The discussion of robustness in Section 3 emphasizes consistency across models rather than the trend within them. A natural mitigation—extrapolating the H-vs-model-quality trend to estimate an asymptotic value, or using an ensemble of models to reduce model-specific biases—is not attempted. The paper's claim that "language is self-similar and long-range dependent" is well-supported in a qualitative sense (H > 0.5, S > 0, power-law relationships hold), but the specific numerical estimates (S = 0.59, H = 0.70) should be understood as PaLM2-L-derived approximations to the true parameters, with unknown bias relative to the optimal predictive distribution of human language.

The Fractal Analysis Captures Only the Information-Theoretic Structure of Language, Not Its Semantic or Lexical Content

The assumption or constraint. The paper's methodology reduces every document to a one-dimensional time series of surprisal values: $z_t = -\log p(w_t \mid w_{[t-1]})$ . This representation captures the predictability structure of the text—where surprising tokens occur relative to predictable ones—but it discards all information about what the text actually says. The authors explicitly acknowledge this: "We also do not model the semantic or lexical form of language. While our information-theoretic approach is well-founded and captures the intrinsic complexity of language, it does not account for the semantic nuances that contribute to meaning" (Section 4). They argue that the surprisal representation provides "an intrinsic, irreducible description of language" (Section 1, citing Futrell and Hahn, 2022), but this description is of language's statistical complexity, not its content.

The consequence. The fractal structure the paper measures is a property of how predictable language is across scales, not a property of what language conveys. This has several implications that limit the scope of the findings:

The connection to model capabilities is indirect: The paper argues that self-similarity in the surprisal process explains why next-token prediction yields models that can reason, summarize, and code (Section 1). But reasoning, summarization, and coding are about manipulating semantic content—tracking logical entailments, extracting gist, generating correct programs. The surprisal process captures whether a token was predictable given its context, not whether the logical argument it participates in is valid or whether the summary it contributes to is faithful. A model could, in principle, produce a perfectly fractal surprisal sequence while generating completely nonsensical text (if the nonsense followed the same predictability patterns as coherent text). The fractal structure of surprisal is a necessary but not sufficient condition for the capabilities the paper wants to explain.

Domain-specific semantic effects are invisible: The fractal parameters in Table 1 vary across domains (H from 0.50 to 0.79), but these differences are attributed to structural properties of the domains (DM-Mathematics has independent questions, GitHub has long-range dependencies in code) rather than to semantic content differences. A legal document and a scientific paper may have similar H and S values, but the kind of long-range dependency—the nature of the coherence that spans the text—is fundamentally different in law versus science. The fractal analysis treats all surprisal patterns as equivalent, regardless of whether they arise from narrative structure, logical argumentation, topic maintenance, or stylistic conventions. This is a form of information-theoretic reduction: the representation captures that there is structure, but not what kind of structure.

The theoretical link to learning is underspecified: The paper's argument that self-similarity enables transfer across scales (learning at the paragraph level helps at the document level) implicitly assumes that the same kinds of patterns recur at different scales. But the surprisal process only shows that the statistical distribution of predictability is scale-invariant, not that the mechanisms producing that predictability are the same. Paragraph-level predictability arises from local syntactic constraints, anaphora resolution, and short-range semantic coherence. Document-level predictability arises from argument structure, narrative arc, and global topic constraints. These are different linguistic mechanisms, even if their aggregate statistical signatures are similar. The fractal analysis cannot distinguish between a process where the same mechanism operates at all scales (true self-similarity in the generative sense) and a process where different mechanisms coincidentally produce similar variance scaling (emergent statistical self-similarity).

What evidence exists in the paper. The paper's entire methodology operates on the surprisal representation, so by construction, the analysis cannot speak to semantic content. The acknowledgment in Section 4 is honest but brief and does not explore the consequences in depth. No experiment compares fractal parameters for texts with identical predictability structure but different semantic content (e.g., a coherent story vs. a semantically scrambled version with the same surprisal sequence), which would directly test whether the fractal parameters capture something beyond surface predictability.

Mitigation status. The limitation is acknowledged in one sentence of Section 4, but no mitigation is proposed beyond noting it as an area for future research. The paper does not attempt to incorporate semantic representations—for instance, by analyzing surprisal in embedding space rather than token probability space, or by jointly modeling surprisal and a semantic coherence metric. The claim that the fractal structure "may illuminate the success of next-token prediction" should be understood as a claim about the information-theoretic learnability of language (the fact that the predictability structure is deep and multi-scale, forcing models to develop sophisticated internal representations) rather than a claim that explains how those internal representations support semantic reasoning. The two are related—a model that can track long-range dependencies in surprisal is likely building representations that capture semantic structure—but the paper does not establish this connection empirically, and the fractal analysis itself is blind to it.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not introduce a new algorithm, architecture, or training objective. Its contribution is methodological and diagnostic: it provides a rigorous, information-theoretic formalism for measuring properties of language that have been suspected for decades but never quantified in a way that survives scrutiny. That makes this work more akin to the introduction of a new measurement instrument than to a new model. The impact flows from what the instrument reveals and what it enables others to measure.

The primary shift is in the representation used for analyzing language structure. Prior to this work, the fractal analysis of language was stuck in a methodological dead end. Researchers used surface-level proxies—word lengths, binary recurrence indicators, ASCII byte values, marginal frequency statistics—that either destroyed the joint statistical structure of language or introduced artifacts that swamped any genuine fractal signal. The result was a literature full of contradictory findings: some studies found long-range dependence in natural language, others found it indistinguishable from noise (Kokol and Podgorelec, 2000); some estimated fractal dimensions exceeding 10 (Andres, 2009, as cited in Section 4), which are physically uninterpretable for time series. The field had no way to resolve these contradictions because there was no agreement on what the "right" representation was.

This paper resolves that impasse by identifying the representation problem as the bottleneck and solving it with a principled alternative: the surprisal sequence derived from a calibrated language model. The representation is not arbitrary—it flows from the chain rule of information theory, captures the full joint distribution of tokens in context, is translation-invariant (unlike word-length or frequency proxies), and connects to established psycholinguistic findings about human reading times and processing difficulty (Hale, 2001; Levy, 2008; Smith and Levy, 2013). The fractal parameters estimated from this representation fall in physically interpretable ranges ( $D = 1.41 \pm 0.08$ , squarely in $[1, 2]$ as required for time series). The Hurst parameter correctly identifies DM-Mathematics—a domain of independent math questions—as having $H \approx 0.50$ , exactly the value for a process without long-range dependence, while natural language domains cluster at $H \approx 0.65$ – $0.74$ . This is the kind of sanity check that prior methods failed.

This is not a paradigm shift—the idea that language might be self-similar and long-range dependent predates this work by decades. But it is a substantive methodological reframing that converts an unsettled, contradictory literature into a coherent empirical picture. After this paper, a researcher who wants to study fractal properties of language has a clear, well-justified pipeline: choose a calibrated language model, compute per-token surprisal, normalize to zero-mean unit-variance, construct increment and integral processes, fit power laws using standard estimators (peak-probability for S, R/S analysis for H, standard-deviation-of-increments for J). The choice of representation is no longer a matter of debate—it is grounded in information theory and validated by robustness across 12 models and 8 domains. This standardization is valuable even if the specific numerical estimates ( $S = 0.59$ , $H = 0.70$ ) are refined by future work using better models or broader language coverage.

The paper also introduces a new category of model evaluation metric. The finding that the median Hurst parameter H improves downstream performance prediction beyond what bits-per-byte (BPB) alone can achieve—adjusted $R^2$ from 0.717 to 0.891 for the few-shot aggregate, from 0.881 to 0.979 for 3-shot BBH CoT (Table 3)—establishes that the temporal structure of prediction errors contains information about model capabilities that mean accuracy does not capture. This is a genuinely new idea in LLM evaluation. Perplexity and its variants measure how often a model is surprised. The Hurst parameter measures how that surprise is organized over time—whether it clusters into bursts, whether it exhibits the same long-range dependence as human language, whether the model's errors have the fractal signature of natural text. That this second-order statistic predicts reasoning benchmarks, knowledge tests, and math problem-solving better than first-order statistics alone is not something the field expected, and it opens a new axis for model diagnostics.

The negative result on context length is clarifying. Three models trained with 2K, 4K, and 8K context lengths on SlimPajama-627B for 100B tokens show no consistent improvement in downstream performance (Table 3, right columns). This constrains the interpretation of the fractal findings: while language has long-range dependence, it does not automatically follow that training with explicitly longer context windows improves model capabilities—at least at the 1B parameter, 100B token scale. This negative result, while not the paper's main contribution, is practically important because it prevents naive extrapolation from "language has LRD" to "longer context training is always better." It suggests that the relationship between fractal structure and learnability is more nuanced than the simplest version of the hypothesis, and it will steer follow-up work toward investigating why the expected benefit did not materialize (model capacity constraints? insufficient training tokens? optimization difficulties with long sequences?).

Research directions that become more attractive:

Verifier and reward model evaluation using fractal diagnostics. If the Hurst parameter captures a dimension of model quality orthogonal to perplexity, it could be applied to evaluate the "language-like-ness" of verifier scores in RLHF pipelines—do reward model outputs exhibit the same fractal structure as human preference judgments? Over-optimization in search-based decoding (as studied in the companion paper analyzed in prior sections) might manifest as a deviation of the verifier's score sequence from the fractal signature of natural language.
Training data quality assessment. The domain-level variation in fractal parameters (Table 1) suggests that H and S could serve as diagnostics for whether a training corpus has appropriate long-range structure. Data mixing strategies for pretraining might benefit from balancing domains with different Hurst parameters, rather than balancing only by token count or perplexity.
Architecture design informed by fractal properties. The paper's finding that language has no characteristic context length—dependencies extend indefinitely into the past—provides theoretical motivation for architectures that can attend over variable-length histories without a hard cutoff. But it also raises the question of whether different layers or attention heads should specialize at different temporal scales, mirroring the self-similar structure of the signal.

Research directions that become less attractive:

Using surface-level proxies for fractal analysis of text. The paper's critique of prior methods (Section 4) is devastating: word-length substitution is not translation-invariant, single-word recurrence cannot capture multi-word dependencies, ASCII encoding introduces character-level artifacts, and all of these methods produced uninterpretable results (fractal dimensions exceeding 10). After this paper, there is no justification for continuing to use these proxies when calibrated LLM surprisal is a principled alternative. The field should converge on the information-theoretic representation or develop alternatives that can be shown to match its properties (translation invariance, joint distribution sensitivity, interpretable parameter ranges).
Claiming that language is non-fractal or has only short-range dependence based on ASCII or word-frequency analyses. The Kokol and Podgorelec (2000) result—that natural language's LRD is close to noise—was an artifact of their encoding choice, as the present paper demonstrates by finding $H = 0.70$ on the same type of text with a better representation. Studies that rely on primitive text encodings to make claims about the absence of fractal structure in language should be treated as methodologically obsolete.

Follow-Up Research This Work Enables

1. Does the Hurst parameter converge to a stable value as language model quality approaches the Bayes error rate for next-token prediction? The paper reports H values ranging from 0.64 (T5-Decoder-110M) to 0.69 (PaLM2-L) in Table 2, with a clear monotonic trend: better models produce higher H. Does this trend continue, or does H asymptote at some value (perhaps ~0.72–0.75) that represents the "true" Hurst parameter of human language as captured by the optimal predictive distribution? A strong follow-up would evaluate a dense grid of models along a single training trajectory—for instance, 20 checkpoints from a 7B model trained on 2T tokens, spanning from random initialization to convergence—and plot H against both training FLOPs and downstream benchmark performance. If H continues to increase with model quality, the paper's estimate of H = 0.70 for language (derived from PaLM2-L) is a lower bound, and the true Hurst parameter could be higher, which would shift the interpretation of language's position on the predictability-noise spectrum. If H asymptotes, the asymptote becomes the definitive measurement of language's long-range dependence. This experiment would also test whether the correlation between H and BPB ( $|r| = 0.83$ , Section 3) holds within a single training run or is confounded by architectural differences across model families.

2. What happens to fractal parameters when the long-range structure of text is artificially destroyed while preserving local statistics? The paper argues that the estimated H and S reflect genuine document-level structure, but no perturbation experiment verifies this. A direct test: take each document in The Pile validation split, shuffle its paragraphs (preserving paragraph-internal token order but randomizing the global order), and recompute the fractal parameters. If the measured H drops toward 0.50 and S changes substantially, this confirms that the parameters are sensitive to document-level organization and not merely an artifact of local statistical regularities. A more granular version would scramble sentences within paragraphs (preserving sentence-internal order) versus scrambling clauses within sentences, to identify which level of linguistic hierarchy contributes most to the observed fractal structure. If H remains high even after paragraph shuffling, the long-range dependence may arise from sentence-level or even lexical-level recurrence patterns rather than from document-level coherence—which would substantially weaken the paper's theoretical argument about multi-scale pattern mirroring.

3. Do non-English languages, particularly those with different recursive properties, exhibit the same fractal parameters? The paper explicitly acknowledges (Section 4) that the analysis is English-only and cites Pirahã—a language "without recursive structure" (Everett, 2005)—as a potential counterexample. A cross-linguistic study using multilingual LLMs (or per-language models of comparable quality) would test whether $S \approx 0.59$ and $H \approx 0.70$ are universal properties of human language or artifacts of English's specific structural features. Key comparisons: (a) languages with similar recursive syntax to English (Spanish, German, Japanese) should show similar parameters if the fractal structure arises from universal properties of human communication; (b) languages with different word-order typology (subject-object-verb vs. subject-verb-object) would test whether the surprisal structure is syntax-dependent; (c) languages with different discourse conventions (e.g., topic-comment structure in Chinese or Korean) would test whether the fractal structure emerges from information-theoretic constraints or from cultural conventions of text organization. A positive result (similar parameters across diverse languages) would strengthen the paper's claim that fractal structure is a deep property of language as such. A negative result (systematic variation with linguistic typology) would refine the claim to apply only to languages with particular structural features, and would open questions about which features drive the fractal signature.

4. Can the Hurst parameter be estimated cheaply enough to serve as a practical training diagnostic? The paper's measurement pipeline requires running every evaluation document through a large LLM to compute surprisal values—a cost that would dominate the evaluation budget in any production setting. A practical follow-up would develop and validate a lightweight estimator for H that does not require full LLM inference. Candidates: (a) training a small probe model that predicts the Hurst parameter of a checkpoint from a subset of its weights or activations, evaluated on whether it can track H across a training run; (b) using a fast approximate language model (n-gram, small distilled model, or even model-free estimators based on token repetition statistics) to compute a surrogate surprisal sequence and measuring whether the resulting H correlates with the full-LLM H; (c) subsampling—computing H on a small fraction of documents or a short prefix of each document—and measuring the tradeoff between estimation error and computational cost. The goal is to make H cheap enough to include in standard training dashboards alongside perplexity, so that model developers can monitor whether their training runs are producing models whose error structure approaches the fractal signature of natural language. Without a cheap estimator, the downstream prediction result in Table 3 remains a laboratory finding rather than a practical tool.

5. Does a model whose internal representations explicitly capture multi-scale structure (e.g., through hierarchical attention or multi-resolution processing) achieve a Hurst parameter closer to natural language, and does this improve downstream performance? The paper's theoretical argument is that self-similarity and LRD in language explain why next-token prediction forces models to develop multi-scale capabilities. A direct test: take two architectures with identical parameter counts—one with standard uniform-context attention (e.g., a vanilla Transformer) and one with explicit multi-scale inductive bias (e.g., hourglass transformers, multi-resolution processing, or recurrent layers operating at different timescales)—train both on the same data, and compare (a) their downstream benchmark performance, (b) their Hurst parameters as measured on the same evaluation corpus, and (c) whether the multi-scale architecture produces surprisal sequences with H closer to 0.70. If the architecture that is explicitly designed to match the fractal structure of language both achieves higher H and better downstream performance, this provides causal evidence for the paper's hypothesis. If it achieves higher H but not better performance, the fractal structure may be a statistical signature of good language models rather than a causal driver of capabilities. If it achieves better performance but H is unchanged, then H is not the right diagnostic for the phenomenon the paper is interested in. This experiment could also reveal whether certain architectural choices (e.g., the positional encoding scheme, the attention pattern, the use of recurrence) are particularly important for capturing the fractal structure of language.

6. How do fractal parameters evolve over the course of a single training run, and do they exhibit predictable phase transitions? The paper reports H for fully-trained checkpoints across 12 models, but says nothing about the trajectory of fractal parameters during training. A detailed longitudinal study would compute H, S, and J at regular intervals (every 1,000 steps) throughout pretraining, from random initialization to convergence, for a single model. Key questions: (a) Do fractal parameters emerge early in training (suggesting they reflect relatively shallow statistical regularities that are quick to learn) or late (suggesting they require sophisticated representations that only develop after the model has mastered local statistics)? (b) Does H increase monotonically, or are there phase transitions where the model suddenly begins to capture longer-range dependencies? (c) Do different domains (code vs. natural language vs. mathematics) show different emergence timelines? (d) Is there a point in training where downstream performance begins to improve without corresponding improvements in perplexity, and if so, does H improve during this phase (suggesting that H tracks the development of capabilities that perplexity misses)? The paper's finding that H captures variance in downstream performance beyond BPB (Table 3) hints that H might be a "capability emergence" indicator, but without longitudinal data, this is speculation. This experiment would directly inform training monitoring: if H plateaus while downstream performance continues to improve, H is a lagging indicator; if H leads downstream performance improvements, it could serve as an early stopping or training-duration signal.

Practical Applications and Downstream Use Cases

Model selection and early stopping during pretraining. The paper's headline practical result (Table 3) is that the combined metric $H_B = 1/\text{BPB} + H_{\text{median}}$ substantially outperforms BPB alone at predicting downstream benchmark performance across 12 models (adjusted $R^2$ of 0.891 vs. 0.717 for the few-shot aggregate). For a team training a large language model, the standard practice is to monitor validation perplexity and use it as a proxy for downstream quality—but as the paper shows, models with similar BPB can have materially different downstream performance (e.g., the PaLM and PaLM2 families in Figure 7 show variation in benchmark scores even at the same BPB level). Computing the median Hurst parameter on a validation corpus at each evaluation checkpoint would provide a second signal that captures the temporal organization of the model's prediction errors. The practical workflow: at each evaluation step, compute both BPB and H on the validation set, track the trajectory of $H_B = 1/\text{BPB} + H$ , and use it to select between checkpoints or to trigger early stopping when $H_B$ plateaus. The finding that H varies meaningfully within model families (Table 2: PaLM2 models range from H = 0.66 to H = 0.69 as quality improves) suggests that this signal has sufficient dynamic range to be useful in practice. The primary barrier is computational cost (computing H requires full LLM inference on the validation corpus, as discussed in Section 6), so this use case becomes practical only once a cheap estimator for H is developed—which the paper identifies as a direction for future research.

Training data curation and domain mixing for pretraining. The domain-level analysis in Table 1 reveals that different text types have systematically different fractal parameters: GitHub code has the highest long-range dependence (H = 0.79), natural language domains cluster around H = 0.65–0.74, and DM-Mathematics (independent questions) has H = 0.50 (no long-range dependence). This variation is interpretable and suggests that fractal parameters could serve as a data quality signal for pretraining corpus design. A team assembling a pretraining dataset could compute H and S for candidate data sources and use them to guide mixing decisions. For instance, if the goal is to train a model that excels at long-form reasoning, including a higher proportion of domains with H in the 0.70–0.79 range (ArXiv, GitHub, Wikipedia) might be beneficial compared to domains with lower H. Conversely, if a data source shows H ≈ 0.50 (like DM-Mathematics), it may contribute useful knowledge but not help the model learn long-range dependency structure—and might be de-emphasized or processed differently (e.g., by concatenating related problems rather than random ones to introduce artificial long-range structure). The self-similarity exponent S could similarly guide decisions about whether a data source provides multi-scale structure: ArXiv (S = 0.70) shows stronger self-similarity than OpenWebText2 (S = 0.53), suggesting academic writing has richer scale-invariant structure that might be particularly valuable for training. This use case does not require real-time H computation—it is a one-time analysis of candidate datasets before training begins—so the computational cost barrier discussed above is less relevant.

Verifier quality assessment in LLM-based evaluation pipelines. The paper's methodology is not limited to evaluating language models themselves—it can be applied to any system that produces a sequence of scores over text. Consider a process reward model (PRM) used for search-based decoding, as studied in the companion paper analyzed in prior sections. Such a PRM assigns a score to each step of a generated solution. One could compute the Hurst parameter of the PRM's score sequence on a corpus of natural solutions (human-written or model-generated) and compare it to the H of human language (0.70). If the PRM's score sequence deviates substantially—for instance, if it has H closer to 0.50 (noise-like, no long-range dependence) or H approaching 1.0 (excessively smooth, deterministic)—this would indicate that the PRM's assessments lack the fractal structure of natural language, suggesting miscalibration or failure to capture multi-scale coherence. This is practically relevant because verifier over-optimization is a known failure mode in test-time compute methods (as documented in the PRM search paper). A PRM whose scores are "less language-like" in their fractal structure might be more susceptible to exploitation by aggressive search, because the search algorithm can find solutions that score highly under the PRM but lack the natural multi-scale coherence that human evaluators expect. Monitoring the fractal parameters of verifier outputs could serve as an early warning system for verifier quality degradation, analogous to how the present paper uses H to predict downstream model performance. This application requires no additional infrastructure beyond what the paper already describes—it is a direct transfer of the measurement pipeline to a different score-generating process.

Diagnosing the gap between perplexity and downstream capability improvements. A recurring challenge in LLM development is that perplexity improvements do not always translate into downstream benchmark improvements, particularly for reasoning-heavy tasks. The paper's finding that H explains variance beyond BPB (Table 3) suggests a concrete diagnostic: for two checkpoints with the same validation perplexity, compute H and check which one is closer to 0.70. If downstream performance differences align with H differences, then H is flagging a dimension of model quality that perplexity misses—potentially the model's ability to track long-range dependencies or to organize its uncertainty in a human-language-like way. If H does not align, the performance gap must be driven by factors not captured by either metric (e.g., specific knowledge acquisition, reasoning strategy, calibration on particular task formats). This diagnostic is immediately actionable with the current (expensive) H computation pipeline for model development teams that can afford the evaluation compute.

When to Prefer This Method

The paper positions its contribution primarily as a measurement and diagnostic framework rather than as a method that competes against named alternatives. It does not propose, for instance, "use the Hurst parameter instead of perplexity for model selection" or "train with a fractal-aware objective instead of standard next-token prediction." It provides evidence that H captures complementary information to BPB, and it argues that the fractal structure of language illuminates why next-token prediction works, but it does not advocate for replacing existing practices with fractal-parameter-based alternatives. The downstream prediction result (Table 3) is presented as an empirical finding, not as a prescription to replace BPB with $H_B$ in production pipelines. The limitations section does not articulate a clear tradeoff between their measurement approach and competing evaluation metrics.

Because the paper does not frame its contribution as a decision between alternatives—it is an analysis paper, not a methods paper proposing a new technique to be adopted or rejected relative to baselines—a "When to Prefer This Method" decision matrix would be inappropriately forced. The practical applications discussed above (model selection, data curation, verifier diagnostics) describe settings where the fractal analysis could be applied, but the paper itself does not provide the cost-benefit analysis or comparative evaluation against alternatives that would justify a prescriptive decision rule. The barrier to adoption is not that another method is better—it is that the computational cost of measuring H is high (as discussed in Section 6) and the paper does not address how to reduce it. Until a cheap estimator is developed, the practical question is not "when to prefer H over BPB" but "when is the additional computational cost of computing H justified by the improvement in predictive accuracy," and the paper provides no data to answer this question.