MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers

🎯 Pitch

For sequences over a million bytes long, standard Transformers grind to a halt—but MEGABYTE sidesteps this by splitting the problem into a global model over patches and a tiny local model within them, cutting self-attention to O(N⁴/³) while using 98% fewer feedforward FLOPs per byte. The result: a 1.5B-parameter byte-level model that generates 40% faster than a 350M Transformer and matches tokenization-based performance on long-context language and image tasks without any tokenizer.

1. Executive Summary

This paper introduces MEGABYTE, a multiscale decoder architecture for autoregressive modeling of long byte sequences that decomposes sequences into fixed-size patches and processes them with a large global Transformer across patches and a small local Transformer within patches — enabling sub-quadratic self-attention, much larger feedforward layers for the same compute budget, and improved parallelism during decoding. The architecture is evaluated against byte-level Transformers and PerceiverAR across language modeling (PG-19, Stories, Books, arXiv, Code), image density estimation (ImageNet at resolutions up to 640×640 — over 1.2M bytes), and raw audio modeling, all under fixed compute and data budgets. MEGABYTE achieves a 40% generation speedup over a standard 350M-parameter Transformer while using a 1.5B-parameter model, reduces bits-per-byte on PG-19 from 1.057 (Transformer) and 1.104 (PerceiverAR) to 1.000 in compute-controlled experiments, and matches the ImageNet 64×64 state-of-the-art (3.40 bpb) while using roughly half the GPU hours, establishing that tokenization-free byte-level autoregressive modeling can be competitive with subword models on long-context tasks — but only when the architectural decomposition successfully trades off global modeling capacity against local sequence fidelity through careful patch-size and model-size allocation.

2. Context and Motivation

The Core Problem: Autoregressive Transformers Scale Poorly to Long Sequences

The fundamental problem MEGABYTE addresses is both simple and consequential: standard autoregressive Transformer decoders become prohibitively expensive when modeling long sequences of raw bytes. This creates a forced choice between two undesirable options — either limit context to a few thousand tokens and sacrifice long-range dependencies, or compress the input through lossy tokenization and lose information that might be important for downstream tasks.

To understand why this problem matters, we need to appreciate what long sequences means in practice. The paper points out that music, image, and video files typically consist of multiple megabytes — that's millions of individual bytes, each of which a byte-level model must predict one at a time. A single 640×640 RGB image, for instance, corresponds to a sequence of over 1.2 million bytes (Section 6.3). Standard Transformer decoders at the scale commonly used in production (hundreds of millions to billions of parameters) cannot process sequences of this length in a single forward pass without running into memory and computation constraints that make training infeasible and generation impractically slow.

The paper identifies two distinct cost bottlenecks, each of which is individually serious but which compound to make the problem especially acute:

1. Self-attention is quadratic in sequence length. This is the well-known $O(T^2)$ problem: a Transformer attending over $T$ positions requires $T^2$ attention computations. The paper acknowledges that substantial prior work has targeted this specific bottleneck (sparse attention, linear attention, recurrent memory mechanisms — Section 9), but makes a critical observation that reframes the entire discussion: self-attention is not the dominant cost in large Transformers. The paper cites the GPT-3 architecture specifically, noting that "the quadratic self-attention computation accounts for only 1.4% of FLOPS" (Section 3.1). This means that even solving the quadratic attention problem completely — achieving true $O(T)$ attention — would only yield marginal efficiency improvements for the models that most need to scale.

2. Position-wise feedforward layers dominate compute. Following the approximation from Kaplan et al. (2020), a forward pass through a Transformer with $m$ non-embedding parameters on a sequence of length $T$ uses approximately $2mT$ FLOPS (Section 3.1). The key insight is that this cost scales linearly with both model size and sequence length — every position in the sequence must pass through every feedforward layer, regardless of whether that position's prediction is difficult or trivial. For a large model, these feedforward computations account for more than 98% of total FLOPS. This is the bottleneck that matters most, and it is the one MEGABYTE directly addresses by sharing feedforward computation across positions within a patch.

These two bottlenecks together create a regime where scaling to long sequences forces an uncomfortable tradeoff: either accept a small model (limiting the expressive capacity available per position) or a short context (limiting the model's ability to capture long-range dependencies). The paper frames MEGABYTE as an architecture that breaks this tradeoff by allowing the bulk of the model's parameters to operate at a coarser temporal resolution — on patches rather than individual bytes.

Why This Problem Matters: Tokenization-Free Modeling at Scale

The motivation for solving the long-sequence problem at the byte level specifically goes beyond raw efficiency. The paper makes a case that eliminating tokenization would address several persistent challenges in autoregressive modeling:

Tokenization introduces language-specific heuristics. The most widely used tokenization methods, including Byte-Pair Encoding (BPE; Sennrich et al., 2015) and SentencePiece (Kudo & Richardson, 2018), are designed for text and make assumptions about what constitutes a useful subword unit that do not transfer well across languages or modalities. The paper specifically cites Radford et al. (2019) for requiring "language-specific heuristics" in their tokenization approach. For truly multilingual or code-switching applications, tokenization choices optimized for one language can degrade performance on others.

Tokenization loses information in non-text modalities. For images and audio, lossless tokenization is mathematically impossible — the high-frequency information in raw pixel values or audio samples cannot be compressed into a discrete vocabulary of manageable size without discarding information. The paper notes that practical approaches use clustering (Hsu et al., 2021) or discrete auto-encoders (Ramesh et al., 2021) that "lose information and likely limit generative model performance" (Section 9). This means that even state-of-the-art image and audio generation models are fundamentally bottlenecked by the compression step that precedes the autoregressive model — the model never sees the full signal, so it cannot learn to model it completely.

Tokenization complicates preprocessing and domain transfer. Every time a model is applied to a new domain or dataset, the tokenizer must be either retrained (to capture domain-specific vocabulary and patterns) or accepted as a source of distribution shift (when out-of-domain tokens appear). Byte-level models eliminate this entirely: the vocabulary is always exactly 256 values, preprocessing is identity, and domain transfer requires no vocabulary adaptation.

Tokenization creates unpredictable failure modes. Section 9 flags that tokenization "can affect prompting and truncated sampling in unpredictable ways," citing the example that whether a prompt should end in whitespace depends on implementation details of the subword algorithm. These are not theoretical concerns — they manifest as bugs that are difficult to diagnose and fix in production systems.

Prior Approaches and Their Shortcomings

The paper situates MEGABYTE relative to three broad categories of prior work, each of which addresses only part of the problem:

Efficient Encoder Models. Several related techniques have been developed for Transformer encoder architectures — ViT's patchification (Dosovitskiy et al., 2020), down- and up-sampling for text encoders (Clark et al., 2022), and Perceiver's cross-attention to a latent bottleneck (Jaegle et al., 2021). However, the paper identifies a critical obstacle that prevents these approaches from being "straightforwardly applied to decoders" (Section 9): the autoregressive causality constraint. In an encoder, you can freely pool or downsample information across positions within a block because all positions are visible simultaneously. In a decoder, predicting the next token must not leak information from future tokens. Naively applying patchification to a decoder would mean that bytes within the same patch could attend to each other bidirectionally, violating the autoregressive property. MEGABYTE's solution — offsetting the inputs to the Global and Local models so that each predicts the next patch or byte, and using autoregressive masking within the Local model — is what makes the encoder-style approach viable for decoding.

Efficient Decoder Models. The paper reviews three subcategories of prior decoder efficiency work:

Chunking with recurrence or cross-attention. Transformer-XL (Dai et al., 2019) and Block-Recurrent Transformers (Hutchins et al., 2022) segment sequences into blocks and propagate information between blocks via recurrence. PerceiverAR (Hawthorne et al., 2022) uses cross-attention from a shorter latent sequence to a longer context. These methods keep per-position computation manageable but still apply the full model to every position within each block — they address the quadratic attention problem but not the dominant feedforward cost.
Linear attention alternatives. Methods such as Linear Transformers (Katharopoulos et al., 2020), fast weight programming (Schlag et al., 2021), and state space models (Gu et al., 2021) replace quadratic attention with linear-complexity alternatives. The paper acknowledges these but notes that "we are not aware of competitive results on large scale language modeling tasks" (Section 3.1), suggesting they have not yet demonstrated the scaling properties needed to replace dense attention in production systems. Moreover, they still apply feedforward layers at every position.
Sparse attention approximations. Sparse Transformers (Child et al., 2019), Routing Transformers (Roy et al., 2020), Longformer (Beltagy et al., 2020), and Reformer (Kitaev et al., 2020) reduce attention cost by attending to only a subset of positions. These achieve complexity improvements but the paper notes that "the performance of dense attention means it is typically still chosen for large scale decoders" such as LLaMA (Touvron et al., 2023) and PaLM (Chowdhery et al., 2022). The implicit claim is that sparse attention trades away too much model quality to be the preferred solution.

The critical gap across all prior decoder efficiency work is that none of them address the feedforward layer bottleneck. They focus almost exclusively on attention, which the paper demonstrates is the wrong target when it represents only 1.4% of FLOPS in large models. MEGABYTE is distinctive because it addresses both costs simultaneously: sub-quadratic attention through the patch decomposition and amortized feedforward computation by applying large feedforward layers at the patch level rather than the byte level.

Tokenization as a workaround. The most common "solution" to the long-sequence problem is simply to avoid it: use tokenization to compress multiple bytes into single tokens, reducing sequence length by a factor of roughly 4× for text (typical BPE compression ratio) and much more for images and audio (via discrete auto-encoders). This works — the paper acknowledges that subword models achieve strong results on language modeling, including the state-of-the-art numbers in Table 3 — but it sidesteps rather than solves the underlying problem. Tokenization introduces the complications described above and, crucially, cannot be applied losslessly to images and audio. The fact that MEGABYTE achieves competitive language modeling results with subword models (Table 3) while operating on raw bytes is evidence that tokenization may not be necessary — the sequence length problem can be solved architecturally rather than being worked around through preprocessing.

How This Paper Positions Itself

MEGABYTE is positioned not as an incremental improvement along any single axis of Transformer efficiency (better attention, better recurrence, better tokenization) but as a fundamentally different decomposition of the sequence modeling problem. The core architectural insight is that predicting the next byte in a sequence involves two qualitatively different tasks that benefit from different model capacities:

Between-patch prediction (what high-level structure comes next?) requires a large model with long-range context to capture global dependencies — but can operate at a coarser temporal resolution, making predictions once per patch rather than once per byte.
Within-patch prediction (given the high-level direction, what specific bytes follow?) requires only a small model because most byte-level completions are locally predictable (completing a word given its first few characters, finishing a texture pattern given neighboring pixels) — but must operate at full resolution to produce the output sequence.

This decomposition is what enables the paper's three claimed improvements — sub-quadratic self-attention, per-patch feedforward layers, and parallel decoding — but these are better understood as consequences of the architecture rather than independently motivated design goals. The architecture reallocates model capacity from per-byte processing (where much of it is wasted on easy predictions) to per-patch processing (where it can model higher-level structure). The efficiency gains follow naturally from this reallocation.

A subtle but important aspect of the paper's positioning is its emphasis on compute-controlled experiments. The authors explicitly acknowledge that "models show consistent improvements when increasing both data and compute" (Section 4.1), meaning that an architecture can appear superior simply because it was trained with more resources. By matching forward-pass time per byte across all models and training for the same number of bytes, the paper isolates the effect of architecture from the effect of training budget. This is a methodological choice that strengthens the validity of the comparisons but also means the absolute numbers reported (e.g., 1.000 bpb on PG-19) should be understood as compute-matched results rather than the best achievable with unlimited resources. The scaling experiment in Table 3 (400B bytes, larger models) shows that MEGABYTE benefits from increased resources similarly to other architectures, suggesting the architectural advantages compound rather than saturate.

Finally, the paper draws an explicit analogy between MEGABYTE patches and traditional tokens: "Our patches are analogous to traditional lossless tokens, and the Local model performs the role of mapping a hidden state to a distribution over possible patches" (Section 9). This framing is instructive — MEGABYTE can be understood as learning an implicit, differentiable, lossless tokenization scheme through the Global model's patch representations, rather than relying on a fixed, hand-designed preprocessing step. The Local model then converts these learned "tokens" back into the original byte sequence, preserving all information that tokenization would discard. This perspective suggests that MEGABYTE is not merely an efficiency hack but a principled architectural answer to the question: what should replace tokenization in truly end-to-end sequence models?

3. Technical Approach

3.1 Reader Orientation

MEGABYTE is an autoregressive decoder architecture — a system that predicts sequences one element at a time, from left to right — that can model sequences of raw bytes (values 0–255) spanning over a million positions by decomposing the sequence into patches and processing them with two cooperating Transformers of very different sizes. The core problem it solves is the prohibitive computational cost of applying large Transformer decoders to long sequences: standard Transformers spend the vast majority of their FLOPS in position-wise feedforward layers that must execute at every single timestep, meaning that scaling up either model size or sequence length forces proportional increases in total computation. MEGABYTE's solution is to recognize that most byte-level predictions are locally easy (completing a word given its first few characters, finishing a texture pattern given neighboring pixels) and therefore do not warrant a massive model, while the difficult structural decisions (what word comes next, where the next object appears in an image) occur less frequently and benefit from a large model applied at a coarser temporal granularity — so it splits the work: a large Global Transformer operates on patches (groups of bytes) to establish high-level context, and a small Local Transformer operates on individual bytes within a patch to fill in the details autoregressively.

3.2 Big-Picture Architecture (Diagram in Words)

The MEGABYTE architecture consists of three components connected in a pipeline, plus a final output layer:

Patch Embedder: Takes a raw byte sequence of length $T$ , embeds each byte into a $D_G$ -dimensional vector using a learned embedding table, adds learned positional embeddings, then chunks the resulting sequence into $K = T/P$ patches of $P$ consecutive byte embeddings each. The chunking is lossless — it's purely a reshape operation. A trainable padding embedding is prepended to the patch sequence so that the Global model can autoregressively predict the first patch without seeing it directly. The output is a sequence of $K$ patch representations, each of dimension $P \cdot D_G$ .
Global Model: A large decoder-only Transformer (the bulk of the model's parameters) that operates on the sequence of $K$ patch representations using causal self-attention across patches. It outputs an updated sequence of $K$ patch representations of the same dimensionality. The Global model only runs once per patch, not once per byte — if the patch size $P = 8$ , the Global model executes $K = T/8$ times for a sequence of $T$ bytes, reducing its per-sequence cost by a factor of $P$ relative to a standard Transformer.
Local Model: A small decoder-only Transformer that fills in the bytes within each patch. For patch $k$ , the Global model's output representation for that patch is reshaped from $P \cdot D_G$ into $P$ vectors of dimension $D_G$ , projected down to the Local model's dimension $D_L$ via a learned linear map, and then each position within the patch is summed with an embedding of the previous byte in the original byte sequence (shifted by one, with a trainable padding embedding at position 0 to condition the first byte prediction). The Local model then applies causal self-attention within these $P$ positions and produces $P$ output vectors of dimension $D_L$ . During training, $K$ independent copies of the Local model run in parallel over all patches; during generation, the Local model runs serially for each patch's $P$ bytes.
Output Projection: The Local model output for position $p$ in patch $k$ is projected back to vocabulary size (256) by multiplying by the transpose of the Local embedding matrix, and a softmax produces the probability distribution over the next byte $x_t$ , where $t = k \cdot P + p$ .

Information flows as follows: raw bytes → byte embeddings + positional embeddings → reshape into patches → Global Transformer (causal across patches) → reshape Global outputs back to per-byte representations → project to Local dimension → add shifted byte embeddings → Local Transformer (causal within patch) → softmax over 256 bytes. The two Transformers are separated by a clear interface: the Global model's output at patch $k$ provides the conditioning context for the Local model to predict all $P$ bytes of patch $k$ , and the Local model never directly sees bytes from previous patches except through the Global model's compressed representation.

3.3 Roadmap for the Deep Dive

First, the core training and autoregressive generation procedure, because understanding how the model is used end-to-end clarifies why each component exists and how they interact during both training (fully parallel) and generation (partially sequential).
Second, the Patch Embedder in detail, because it defines how raw bytes become the structured input that the Global and Local models consume, and the padding scheme is essential for maintaining the autoregressive property.
Third, the Global Model, because it is the architectural centerpiece — the component that achieves the computational savings by operating at patch granularity while providing high-level context to the Local model.
Fourth, the Local Model and its interface to the Global Model, because this is where the two Transformers are stitched together, and the choice to add Global context to shifted byte embeddings (rather than concatenation, cross-attention, or other fusion strategies) determines the Local model's inductive bias.
Fifth, the three extensions — Convolutional Patch Encoder, Cross-Patch Attention, and Strided Inference — because they address specific weaknesses of the base architecture and reveal what the authors considered the most important failure modes.
Sixth, the efficiency analysis that motivates the architecture, including both the training FLOPs decomposition (why self-attention is not the main cost) and the generation parallelism analysis (why MEGABYTE can be faster than a smaller Transformer at inference time).

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an architectural design paper whose core idea is that decomposing long byte sequences into patches and allocating a large model to between-patch structure and a small model to within-patch detail yields substantial efficiency gains without sacrificing (and in many cases improving) predictive performance, as demonstrated through compute-controlled experiments across text, image, and audio modalities.

Training and Autoregressive Generation Procedure

The model is trained with standard next-byte prediction: given a prefix of bytes $x_{0..t-1}$ , predict the distribution over $x_t$ . The model's parameters are optimized to minimize the negative log-likelihood of the training sequences, summed (or averaged) across all positions. Because the entire architecture is a single differentiable function from input bytes to output byte probabilities, training uses standard backpropagation without any auxiliary losses or reinforcement learning components.

Training parallelism. A critical practical property of MEGABYTE is that during training, the Local model runs over all patches fully in parallel. For a training sequence of length $T$ , the Global model processes $K = T/P$ patch representations simultaneously (with causal masking ensuring each patch representation only depends on previous patches), and then the $K$ copies of the Local model process the $K$ patches simultaneously as independent sequences of length $P$ (with causal masking within each patch). The Global model executes once and the Local model executes once per patch, both in parallel across patches. This means that training throughput is comparable to training a standard Transformer on a sequence of length $K + P$ rather than length $T$ , which is much smaller when $P$ is substantially larger than 1 but much smaller than $T$ . The paper's controlled experiments use patch sizes of 8 (text), 12, 192 (images), and 32 (audio), meaning the Global model sees $T/8$ , $T/12$ , $T/192$ , and $T/32$ patches respectively — a dramatic reduction in the sequence length that must undergo full self-attention.

Autoregressive generation. During generation, the model produces one byte at a time, but the Global model only needs to be recomputed when moving to a new patch. Specifically:

The Global model is run once on all previous patches (which were already computed and can be cached) to produce the conditioning vector for the current patch. This step has cost proportional to $L_{\text{global}}$ (the number of Global model layers).
The Local model runs autoregressively within the current patch, producing one byte per step. For a patch of size $P$ , this requires $P$ forward passes through the Local model, each of cost proportional to $L_{\text{local}}$ (but the Local model is much smaller than the Global model).
When the patch is completed, the Global model is updated with the new patch's representation (requiring $L_{\text{global}}$ layers of computation again, though in practice KV-caching avoids recomputing attention for previous patches), and the cycle repeats.

The paper quantifies the generation cost as: generating each patch requires $O(L_{\text{global}} + P \cdot L_{\text{local}})$ serial operations, compared to $O(P \cdot L_{\text{global}} + P \cdot L_{\text{local}})$ for a standard Transformer of equivalent total depth (Section 3.2). When $L_{\text{global}} \gg L_{\text{local}}$ , the speedup approaches a factor of $P$ . Table 6 confirms this empirically: a MEGABYTE model with a 1.3B-parameter Global model and a 218M-parameter Local model generates 8192 bytes 40% faster than a 350M-parameter standard Transformer, despite having over 4× the total parameters. This counterintuitive result — larger model, faster generation — occurs because the Global model's expensive computations are amortized over $P = 8$ bytes.

Why this procedure rather than greedy chunked decoding? An alternative design would have the Global model predict an entire patch at once (e.g., produce a patch embedding, then have the Local model decode it independently). This would fail to capture the joint distribution within the patch: as the paper explains in Section 2.4, predicting all $P$ bytes of a patch as conditionally independent given the Global context would assign probability mass to unrealistic combinations (e.g., 50% "cat" and 50% "dog" would produce probability for "cag" and "dot"). The autoregressive Local model avoids this by conditioning each byte on previous bytes within the same patch, allowing it to model the full joint distribution over the patch's 256^P possible byte sequences without enumerating them. This is the fundamental reason the Local model must be autoregressive rather than feedforward: it factorizes an intractable categorical distribution over 256^P outcomes into $P$ tractable categorical distributions over 256 outcomes, each conditioned on the Global context and previous bytes.

Patch Embedder: Byte Embedding, Positional Encoding, and Chunking

The Patch Embedder converts a sequence of $T$ raw bytes (integers in $\{0, \dots, 255\}$ ) into $K = T/P$ patch representations that the Global model consumes. The process involves three steps: byte embedding, positional encoding, and chunking with padding.

Byte embedding. Each byte $x_t$ is mapped to a dense vector of dimension $D_G$ using a learned embedding table $E^{\text{global-embed}} \in \mathbb{R}^{V \times D_G}$ , where $V = 256$ (the vocabulary size — one entry per possible byte value). This produces a sequence of $T$ vectors:

$h_t^{\text{embed}} = E^{\text{global-embed}}_{x_t} + E^{\text{pos}}_t$

where $E^{\text{global-embed}}_{x_t} \in \mathbb{R}^{D_G}$ is the learned embedding for the byte value at position $t$ , $E^{\text{pos}}_t \in \mathbb{R}^{D_G}$ is the learned positional embedding for absolute position $t$ in the full sequence, and $t$ ranges from $0$ to $T-1$ .

Why absolute positional embeddings? The paper does not specify which positional encoding scheme is used in the Patch Embedder (rotary embeddings are mentioned specifically for the cross-patch attention extension in Section 2.3.2), but the equation shows a learned $E^{\text{pos}}$ added directly to the byte embeddings. This is a standard absolute position scheme: each position in the training sequence length gets its own trainable vector, and the model must learn to use these to distinguish positions. The Global model may additionally incorporate rotary embeddings or learned position biases in its attention layers; the paper uses the standard Transformer machinery from the Metaseq codebase.

What this embedding operation computes: for each byte in the input sequence, the model looks up a dense vector representing that byte value and adds a dense vector representing that absolute position in the sequence. The result is a single vector per byte that captures both the byte's identity and its location. This is input to the reshaping operation.

Why add rather than concatenate? Addition forces the byte identity information and the position information to share the same $D_G$ dimensions, which is both parameter-efficient (no extra projection needed) and compatible with subsequent reshaping. If position and byte identity were concatenated, the dimension would double, increasing the Global model's input dimension and cost. Addition is standard practice in Transformer architectures.

Chunking into patches with padding. The sequence of $T$ byte embeddings is reshaped into a sequence of $K$ patches, each containing $P$ consecutive byte embeddings. The reshaping operation views the sequence of shape $(T, D_G)$ as shape $(K, P, D_G)$ , then flattens the last two dimensions to produce patch representations of shape $(K, P \cdot D_G)$ . Explicitly:

$h_k^{\text{global-in}} = \begin{cases} E^{\text{global-pad}}, & \text{if } k = 0, \\ h_{((k-1) \cdot P):(k \cdot P)}^{\text{embed}}, & k \in [1, \dots, K], \end{cases}$

where $E^{\text{global-pad}} \in \mathbb{R}^{P \times D_G}$ is a learned padding embedding (a trainable tensor of the same shape as a patch), $h_k^{\text{global-in}}$ is the $k$ -th input patch representation, and $K = T/P$ .

What this operation computes: the byte embeddings for positions $(k-1) \cdot P$ through $k \cdot P - 1$ (a block of $P$ consecutive bytes) are concatenated into a single vector of length $P \cdot D_G$ . The first patch position ( $k=0$ ) receives a learned padding embedding instead of actual byte embeddings. The overall output is a sequence of $K$ vectors, each of dimension $P \cdot D_G$ , that serve as input to the Global model.

Why the padding embedding? The autoregressive property requires that when predicting patch $k$ , the model must not see any bytes from patch $k$ . In standard autoregressive modeling, predicting $x_t$ uses $x_{0..t-1}$ as input. With patches, predicting all $P$ bytes of patch $k$ should use bytes from patches $0$ through $k-1$ as input. The padding embedding serves as a placeholder for the "empty prefix" before the first patch, and the offset in the input (where $h_k^{\text{global-in}}$ contains bytes from patch $k-1$ rather than $k$ ) ensures that when the Global model processes input patch $k$ , it sees the byte embeddings for the previous patch (or padding for the first patch). This is the decoder analog of the causal masking trick: by shifting the input by one patch, each Global model position $k$ can only condition on patches $0..k-1$ , and when its output is used to condition the Local model for patch $k$ , no information leakage occurs.

Why reshape into $P \cdot D_G$ rather than use a learned projection? The reshape preserves all information from the byte embeddings exactly and losslessly. An alternative would be to apply a learned linear projection (or ConvNet) to compress the $P$ byte embeddings into a smaller patch representation, but this would be a bottleneck — information discarded at this stage could never be recovered. By keeping the full $P \cdot D_G$ dimensionality, the Global model receives the complete byte-level information for each patch and can learn its own compression through its Transformer layers. This also means that if $D_G$ is large (e.g., 1024 or 2048 in the larger models), the patch representations are very high-dimensional — $P \cdot D_G$ can be 8192 or more — which contributes to the Global model's capacity but also its cost.

Translation invariance limitation. Because the chunking uses fixed boundaries (positions 0 to $P-1$ , $P$ to $2P-1$ , etc.), the representation of a given subsequence of bytes depends on its alignment with the patch grid. A word that appears at byte positions 3–7 in one context will be chunked differently than the same word at positions 0–4 in another context, and the Global model must learn to handle both cases separately. This lack of translation invariance is the motivation for the Convolutional Patch Encoder extension (Section 2.3.1), which applies causal convolutions before chunking to provide position-invariant local context.

Global Model: Large Transformer Operating on Patches

The Global model is the computational heavyweight of the architecture. It is a standard decoder-only Transformer (following the GPT-style architecture used in the Metaseq codebase) whose input and output sequences consist of $K$ patch representations, each of dimension $P \cdot D_G$ .

Architecture. The Global model is parameterized by its number of layers $L_{\text{global}}$ , its hidden dimension (which must equal the patch representation dimension $P \cdot D_G$ in the base architecture), its number of attention heads, and its feedforward network hidden dimension. The paper does not specify a separate hyperparameter name for the Global model's hidden dimension — it directly inherits $P \cdot D_G$ from the Patch Embedder — but the feedforward dimension follows the standard Transformer convention (typically $4 \times$ hidden dimension for the intermediate layer, though the paper does not explicitly confirm this multiplier). Each layer consists of:

Masked multi-head self-attention over the $K$ patch representations, with causal masking ensuring patch $k$ only attends to patches $0..k$ .
A position-wise feedforward network (two linear projections with a ReLU activation between them, per the training details in Appendix A.1) applied independently to each of the $K$ positions.
Pre-normalization (Pre-Norm) via LayerNorm before each sublayer, with residual connections.

The output of the Global model is:

$h_{0:K}^{\text{global-out}} = \text{transformer}^{\text{global}}(h_{0:K}^{\text{global-in}})$

where $h_{0:K}^{\text{global-in}} \in \mathbb{R}^{K \times (P \cdot D_G)}$ is the input patch sequence (including the padding patch at position 0) and $h_{0:K}^{\text{global-out}} \in \mathbb{R}^{K \times (P \cdot D_G)}$ is the output.

What the Global model computes: given a sequence of patch representations, each encoding $P$ consecutive bytes from the original sequence (with the first position being a learned padding patch), the model applies causal self-attention to contextualize each patch with respect to all previous patches, producing a sequence of updated patch representations that incorporate long-range dependency information. The self-attention is "causal" in the standard autoregressive sense: patch $k$ can attend to patches $0, 1, \dots, k$ , but not to patches $k+1, k+2, \dots, K$ . This ensures the model respects the temporal ordering — when generating the bytes of patch $k$ , the Global model's representation for that patch depends only on previous patches.

Why a standard Transformer rather than a specialized architecture? The paper explicitly states in Section 2.4 that re-using established Transformer components "increases the likelihood that the architecture will inherit the desirable scaling properties of transformers." By making the Global model a standard decoder-only Transformer (just operating on patch representations rather than token embeddings), MEGABYTE can leverage all the infrastructure, optimization techniques, and scaling insights developed for GPT-style models. The only change is the granularity of the sequence.

Cost scaling. For a sequence of length $T$ and patch size $P$ , the Global model's self-attention operates on a sequence of length $K = T/P$ , so its attention cost scales as $O((T/P)^2) = O(T^2/P^2)$ . The feedforward cost scales as $O(K \cdot m_g) = O((T/P) \cdot m_g)$ , where $m_g$ is the number of parameters in the Global model. The key insight is that the feedforward cost — which dominates total FLOPS — is reduced by a factor of $P$ relative to a standard Transformer operating on all $T$ bytes. This means MEGABYTE can use a Global model that is $P$ times larger (in terms of parameters per forward pass) than a standard Transformer while incurring the same total feedforward cost, because the larger model runs $P$ times fewer times. The paper formalizes this in Section 3.1: for a Global model with $m_g$ parameters, the estimated FLOPS is approximately $2T \cdot m_g / P$ , compared to $2T \cdot m$ for a standard Transformer with $m$ parameters. When $m_g \approx P \cdot m$ , the costs are equal, but the Global model has dramatically more capacity per patch.

Global-to-Local interface: reshaping and projection. Before the Local model can use the Global model's output, the patch representations must be converted from the Global model's format (vectors of size $P \cdot D_G$ representing entire patches) to the Local model's format (sequences of $P$ vectors of size $D_L$ representing individual byte positions within a patch). This conversion has two steps:

Reshaping: The output patch representation for patch $k$ , which is a single vector of dimension $P \cdot D_G$ , is reshaped into $P$ vectors of dimension $D_G$ . Specifically, position $p$ within the patch (where $p \in [0, P-1]$ ) takes the slice of the Global output from index $p \cdot D_G$ to $(p+1) \cdot D_G$ . This is the inverse of the chunking operation in the Patch Embedder.
Linear projection: Each of these $P$ vectors of dimension $D_G$ is projected to the Local model's dimension $D_L$ using a learned weight matrix $w^{\text{GL}} \in \mathbb{R}^{D_G \times D_L}$ .

Formally, for patch $k$ and position $p$ within that patch:

$h_{k,p}^{\text{global-projected}} = w^{\text{GL}} \cdot h_{k, (p \cdot D_G):((p+1) \cdot D_G)}^{\text{global-out}}$

where $h_{k, (p \cdot D_G):((p+1) \cdot D_G)}^{\text{global-out}}$ is a slice of length $D_G$ from the Global model's output for patch $k$ , and $h_{k,p}^{\text{global-projected}} \in \mathbb{R}^{D_L}$ is the projected vector.

Why this reshaping and projection rather than learned cross-attention? A cross-attention mechanism would allow the Local model to attend to the Global output dynamically at each Local model layer, potentially using the Global context more flexibly. However, cross-attention would also add cost: each Local model layer would need to compute attention over the Global output, increasing both parameters and computation. The simple sum-based approach (project, then add to byte embeddings) is cheaper, adds no new attention mechanisms, and proved sufficient in practice — the ablation in Table 7 shows that removing the Global model entirely causes a catastrophic increase in bits-per-byte (from 3.158 to 3.181 on ImageNet256, and from 0.6871 to 1.373 on arXiv), confirming the Global context is being used even through this simple interface.

Local Model and the Full Interface

The Local model is a smaller decoder-only Transformer that autoregressively predicts the bytes within a single patch, conditioned on both the Global model's context for that patch and the preceding bytes within the patch.

Input construction. For patch $k$ , the Local model receives a sequence of $P$ input vectors. The vector at position $p$ (where $p \in [0, P-1]$ ) is the sum of two components:

Global context: The projected Global model output for patch $k$ at position $p$ , computed as described above: $w^{\text{GL}} \cdot h_{k, (p \cdot D_G):((p+1) \cdot D_G)}^{\text{global-out}}$ .
Previous byte embedding: An embedding of the byte at position $p-1$ within the same patch, using a separate embedding table $E^{\text{local-embed}} \in \mathbb{R}^{V \times D_L}$ . For $p = 0$ (the first byte of the patch), a learned padding embedding $E^{\text{local-pad}} \in \mathbb{R}^{D_L}$ is used instead, since there is no previous byte within this patch.

Formally:

$h_{k,p}^{\text{local-in}} = w^{\text{GL}} \cdot h_{k, (p \cdot D_G):((p+1) \cdot D_G)}^{\text{global-out}} + \begin{cases} E^{\text{local-pad}}, & \text{if } p = 0 \\ E^{\text{local-embed}}_{x_{(k \cdot P + p - 1)}}, & p \in [1, \dots, P-1] \end{cases}$

where $x_{(k \cdot P + p - 1)}$ is the byte at absolute position $t-1 = k \cdot P + p - 1$ in the original sequence.

What this construction computes: for each byte position $p$ within patch $k$ , the Local model receives a vector that combines (via addition) information about the high-level context for that patch and that specific position within the patch (from the Global model) with information about the immediately preceding byte (from the Local embedding table). The result is a sequence of $P$ vectors of dimension $D_L$ , where each position is conditioned on both the global structure and the local sequential context.

Why two separate embedding tables? The Global embedding table $E^{\text{global-embed}}$ maps bytes to $D_G$ -dimensional vectors for consumption by the Global model; the Local embedding table $E^{\text{local-embed}}$ maps bytes to $D_L$ -dimensional vectors for consumption by the Local model. They have different dimensions because the Global and Local models operate at different scales. An alternative would be to share a single embedding table and project, but separate tables give each model the freedom to learn byte representations suited to its role: the Global model's embeddings might capture broader semantic properties (this byte indicates the start of a new word, this byte is a digit), while the Local model's embeddings might capture finer-grained transitional properties (given the prefix "ca", the next byte is likely "t").

Why use the previous byte embedding rather than the current byte? The autoregressive constraint: when predicting byte $x_t$ , the model can condition on $x_{t-1}$ (the immediately previous byte) but not on $x_t$ itself. The Local model's input at position $p$ is used to predict $x_{k \cdot P + p}$ . To avoid leaking information about the target byte, the input must use $x_{k \cdot P + p - 1}$ — the byte before the one being predicted. For $p = 0$ , there is no $x_{k \cdot P - 1}$ within the same patch (that byte belongs to the previous patch), so the learned padding embedding serves as the conditioning signal for the first byte of each patch.

Why add rather than concatenate the Global context and byte embedding? Addition forces the two information sources to share the same $D_L$ dimensions, which is both parameter-efficient and imposes the inductive bias that the Global context should modify the byte-level representation additively (e.g., "shift the representation toward more formal vocabulary" or "steer the pixel values toward a particular color palette") rather than providing a separate channel of information. If concatenation were used, the Local model dimension would need to be $D_L + D_L = 2D_L$ , doubling the cost of all Local model layers. The paper's results suggest addition is sufficient for effective conditioning.

Local model architecture. The Local model is a standard decoder-only Transformer with $L_{\text{local}}$ layers, hidden dimension $D_L$ , and the same pre-norm, ReLU activation, and residual structure as the Global model. It applies causal self-attention over the $P$ positions within the patch. The output is:

$h_{k,0:P}^{\text{local-out}} = \text{transformer}^{\text{local}}(h_{k,0:P}^{\text{local-in}})$

where $h_{k,0:P}^{\text{local-in}} \in \mathbb{R}^{P \times D_L}$ is the input sequence for patch $k$ and $h_{k,0:P}^{\text{local-out}} \in \mathbb{R}^{P \times D_L}$ is the output.

Why a separate Transformer rather than just a linear output head? The Local model could be replaced by a simple linear projection from $D_L$ to 256 at each position, conditioned on the Global context and the previous byte embedding. This would factor the patch distribution into $P$ conditionally independent byte distributions: $p(x_{kP+0} \mid \text{global}_k) \cdot p(x_{kP+1} \mid \text{global}_k, x_{kP+0}) \cdot \dots$ . The paper argues this would severely limit expressivity (Section 2.4): the Local model's causal self-attention across the $P$ positions within a patch allows each byte prediction to depend on all previous bytes in the patch, not just the immediately previous one. This is the difference between a unigram model (each byte conditioned only on the previous byte and global context) and a proper autoregressive model (each byte conditioned on the full patch prefix). A multi-layer Transformer allows this richer dependency structure.

Output projection. The final step maps the Local model outputs to byte probabilities. For position $p$ in patch $k$ (corresponding to absolute byte position $t = k \cdot P + p$ ):

$p(x_t \mid x_{0:t}) = \text{softmax}(E^{\text{local-embed}} \cdot h_{k,p}^{\text{local-out}})_{x_t}$

where $E^{\text{local-embed}} \in \mathbb{R}^{V \times D_L}$ is the Local embedding matrix, $h_{k,p}^{\text{local-out}} \in \mathbb{R}^{D_L}$ is the Local model's output vector for that position, and the multiplication $E^{\text{local-embed}} \cdot h_{k,p}^{\text{local-out}}$ produces a vector of logits of dimension $V = 256$ . The softmax converts these logits to probabilities.

Why tie the output projection weight to the Local embedding matrix? This is the standard weight tying practice in language modeling (Press & Wolf, 2017; Inan et al., 2017): the same matrix $E^{\text{local-embed}}$ is used both to embed bytes on the input side and to project to logits on the output side. This reduces the total parameter count (saving $V \times D_L$ parameters that a separate output projection would require) and provides a useful inductive bias — the embeddings that the model learns to represent bytes during input should also be good representations for predicting bytes during output. For a vocabulary of only 256, the parameter savings are modest compared to large-vocabulary language models, but the principle applies.

Training loss. The model is trained to minimize the average negative log-likelihood over all $T$ positions:

$\mathcal{L} = -\frac{1}{T} \sum_{t=0}^{T-1} \log p(x_t \mid x_{0:t})$

where $p(x_t \mid x_{0:t})$ is computed as above. This is the standard autoregressive language modeling objective, applied to a byte-level vocabulary.

Extension: Convolutional Patch Encoder

The base Patch Embedder chunks the byte sequence at fixed boundaries (every $P$ bytes), which means that the representation of a given subsequence depends on its alignment with the patch grid. For text, this means the same word could appear at different offsets within different patches and receive different representations. The Convolutional Patch Encoder (Section 2.3.1) addresses this by applying causal convolutional layers to the byte embeddings before chunking, providing each byte position with a local context window that makes its representation more translation-invariant.

Architecture. The paper describes it minimally: "causal convolutional layers" with "filter sizes of 3, 5 and 7." This implies a stack of (likely) three 1D convolutional layers, each with a different kernel size, applied sequentially to the byte embedding sequence. The causality constraint means each convolution only looks at previous positions (standard causal padding: pad the left side with zeros so that position $t$ 's output depends on positions $t-\text{kernel\_size}+1$ through $t$ ). After the convolution stack, the resulting sequence of byte representations (still of length $T$ and dimension $D_G$ ) is chunked into patches exactly as in the base Patch Embedder.

What it computes: instead of each byte position being represented purely by its own byte embedding plus absolute position encoding, the convolutional layers blend information from local neighborhoods. A byte at position $t$ receives context from the preceding 2–6 bytes (depending on the filter stack), creating a representation that captures local patterns (common letter sequences, local pixel correlations, audio waveform shapes) in a position-invariant way — the convolution weights are shared across positions, so the same pattern at different offsets produces the same convolutional features.

Why kernel sizes 3, 5, and 7? These small odd-sized kernels capture progressively larger local contexts: a kernel of 3 sees ±1 neighboring positions (in a non-causal setting; causally, it sees 2 previous positions), a kernel of 5 sees ±2 neighbors (4 previous positions causally), and a kernel of 7 sees ±3 neighbors (6 previous positions causally). The combination provides multi-scale local features before the global model sees the patch representations. The paper does not ablate the specific kernel size choices or depth.

Impact. Table 7 shows that adding the CNN encoder provides slight improvements on audio (3.475 vs. 3.477 bpb) and ImageNet256 (3.155 vs. 3.158 bpb) but no improvement on arXiv text (0.6871 both with and without). This suggests the translation invariance problem is more acute for modalities with continuous or quasi-periodic structure (image textures, audio waveforms) than for text, where byte sequences representing words have variable length and the alignment problem may be less severe.

Extension: Cross-Patch Attention

The base architecture forces all long-range information to flow through the Global model's patch representations: the Local model only sees the Global context for its own patch and the immediately previous bytes within that patch. If the Local model needs information from bytes in the previous patch (e.g., to smoothly continue a multi-patch word or texture), it must rely on the Global model to encode that information into the patch representation. Cross-patch attention (Section 2.3.2) gives the Local model direct access to the last $r$ elements from the previous patch.

Mechanism. In each self-attention layer of the Local model, when computing attention for the $P$ positions within the current patch, the key and value sequences are augmented by concatenating the keys and values from the last $r$ positions of the previous patch:

$\text{Keys}_{\text{augmented}} = [\text{Keys}_{\text{previous-patch}}[-r:]; \text{Keys}_{\text{current-patch}}]$ $\text{Values}_{\text{augmented}} = [\text{Values}_{\text{previous-patch}}[-r:]; \text{Values}_{\text{current-patch}}]$

The Local model positions in the current patch can now attend to these $r$ additional keys and values from the previous patch, giving them a direct window into the immediately preceding byte-level context. The paper uses rotary position embeddings (Su et al., 2021) to encode the relative positions between elements across the patch boundary, which is important because the absolute positions within the Local model's sequence change when the patch boundary shifts.

Cost. The overhead is small: the attention sequence length grows from $P$ to $P + r$ , and since $r$ is typically much smaller than $P$ , the additional attention cost is negligible. The paper does not specify the value of $r$ used in experiments.

Impact. Table 7 shows that removing cross-patch attention (the "w/o cross-patch attention" row) has mixed effects: on arXiv text, performance actually improves slightly (0.6781 vs. 0.6871 bpb — lower is better), on audio it slightly degrades (3.481 vs. 3.477), and on ImageNet256 it more noticeably degrades (3.259 vs. 3.158). This suggests the value of cross-patch information is modality-dependent: images, with their continuous visual structures that cross patch boundaries, benefit more than text, where patch boundaries may align with natural linguistic boundaries more often. The paper describes the overall architecture as "robust to this modification."

Relationship to Transformer-XL. The paper explicitly notes the similarity to Transformer-XL's segment-level recurrence (Dai et al., 2019), but identifies a key difference: "differs by being fully differentiable." In Transformer-XL, the keys and values from previous segments are cached and treated as fixed (no gradient flows through them during training) to maintain computational efficiency over very long sequences. In MEGABYTE's cross-patch attention, because the Local model processes all patches in parallel during training, the gradients can flow from the current patch's attention back through the previous patch's keys and values, making the entire computation fully differentiable. This is possible because the Local model's sequence length ( $K \times P$ total positions, processed independently per patch but with cross-patch connections) is still manageable — unlike Transformer-XL, which deals with arbitrarily long sequences and can't afford to backpropagate through all previous segments.

Extension: Strided Inference

The paper observes empirically that "the per-token loss within each patch would increase towards the end of the patch, as the prediction relies more on the weaker Local model" (Section 2.3.3). This makes intuitive sense: the first byte of a patch has strong conditioning from the Global model (which captures high-level structure) and a padding embedding (which provides a clean start-of-patch signal). Later bytes within the patch rely increasingly on the Local model's own autoregressive predictions, which may drift or accumulate errors. Strided inference mitigates this by running the model twice with offset inputs and combining the best-positioned predictions.

Procedure. For a patch size $P$ :

Run the full model (Global + Local) on the input sequence as usual, producing byte probabilities for all positions. From each patch, keep only the predictions for the first $P/2$ positions.
Run the full model again, but with inputs offset by $P/2$ positions (i.e., the Global model sees patches that are shifted by half a patch, and the Local model's byte sequences within each shifted patch are correspondingly shifted). From each shifted patch, keep only the predictions for the first $P/2$ positions.
Interleave the two sets of predictions to cover the complete sequence: positions 0 through $P/2-1$ come from the first pass, positions $P/2$ through $P-1$ come from the second pass's first half-patch, positions $P$ through $3P/2-1$ come from the first pass's second half-patch, and so on.

The net effect is that every predicted byte is in the first half of some patch (either from the original or the shifted partitioning), where the per-token loss is empirically lower.

Cost. The procedure requires two full forward passes instead of one, doubling inference cost. Table 8 quantifies this: strided inference at $2\times$ cost achieves 0.8926 bpb, compared to 0.9079 for basic inference at $1\times$ cost. For comparison, a standard sliding window (where the model is run on overlapping windows and predictions from the center of each window are used) at $2\times$ cost achieves 0.8918 bpb, very similar to strided inference. Combining both — strided inference with sliding window — at $4\times$ cost achieves 0.8751 bpb, the best result reported. This shows that the strided inference improvement is additive with sliding window improvements, suggesting they address somewhat different failure modes (within-patch position dependence vs. context boundary effects).

Why not just use longer overlap during training? The problem strided inference addresses is architectural — the Local model's predictions degrade toward the end of the patch because the Global context is a single fixed vector for the entire patch, while the Local model's autoregressive predictions have limited capacity to maintain coherence over $P$ steps. Training with different patch alignments (data augmentation via random offset) could potentially teach the model to be robust to position within a patch, but the paper does not explore this. Strided inference is purely an inference-time technique that works around the limitation rather than fixing it during training.

Efficiency Analysis: Why the Architecture Saves Computation

The efficiency analysis in Section 3.1 is central to understanding why MEGABYTE's architectural choices matter. It decomposes the computational cost into attention and feedforward components and shows that MEGABYTE reduces both, but for different reasons.

Attention cost. The standard Transformer self-attention cost for a sequence of length $T$ and model dimension $d$ is $O(T^2 \cdot d)$ . MEGABYTE splits this across two models:

Global model: attention over $T/P$ patches, cost $O((T/P)^2 \cdot P \cdot D_G)$ . The dimension is $P \cdot D_G$ because each patch representation concatenates $P$ byte embeddings of size $D_G$ .
Local model: $T/P$ independent attention computations over sequences of length $P$ , each with dimension $D_L$ , total cost $O((T/P) \cdot P^2 \cdot D_L) = O(T \cdot P \cdot D_L)$ .

The total attention cost is therefore $O((T^2 / P^2) \cdot P \cdot D_G + T \cdot P \cdot D_L)$ . Simplifying and noting that $D_G$ and $D_L$ are constants relative to $T$ and $P$ , the asymptotic complexity is $O(T^2 / P + T \cdot P)$ . By choosing $P = T^{1/3}$ , this becomes $O(T^{4/3})$ , which is sub-quadratic in $T$ . More generally, for any $1 < P < T$ , the cost is less than the $O(T^2)$ of a standard Transformer.

The paper notes that even with much shorter patches of $P = T^{1/5}$ , the complexity would be $O(T^{8/5})$ , which is still sub-quadratic (since $8/5 = 1.6 < 2$ ). This means the architecture is robust to the patch size choice — as long as patches are larger than 1 and smaller than $T$ , there is an efficiency gain.

Feedforward cost — the dominant factor. This is where the paper makes its strongest argument. Following the FLOPs approximation from Kaplan et al. (2020), a standard Transformer with $m$ parameters processing a sequence of length $T$ uses approximately $2mT$ FLOPS. In GPT-3, the feedforward network accounts for the vast majority of these FLOPS (the paper doesn't give an exact percentage but states "more than 98% of FLOPS" in the Introduction and cites the 1.4% attention figure in Section 3.1).

MEGABYTE's feedforward cost decomposes as:

Global model: $m_g$ parameters, applied to $T/P$ positions, cost $\approx 2 \cdot m_g \cdot (T/P)$ FLOPS.
Local model: $m_l$ parameters, applied to $T$ positions (since the Local model runs on every byte, just in groups of $P$ ), cost $\approx 2 \cdot m_l \cdot T$ FLOPS.

Total feedforward FLOPS: $\approx 2T \cdot (m_g / P + m_l)$ . When $m_g \gg m_l$ (the Global model is much larger than the Local model), this is approximately $2T \cdot m_g / P$ . Comparing to the standard Transformer's $2mT$ : for the same total FLOPS, MEGABYTE can use a Global model with $m_g \approx P \cdot m$ parameters — $P$ times larger — while keeping $m_l$ small. This is the core efficiency argument: MEGABYTE achieves its performance improvements by deploying a much larger model at the patch level, where the difficult structural decisions are made, while using a small model at the byte level where most predictions are easy.

Concrete example from the paper. Section 3.1 provides specific model sizes for a scaling comparison: for a given total compute budget, MEGABYTE configurations use Global/Local model sizes of 452M/151M, 5.8B/604M, and 170B/3.2B parameters respectively, with $P = 8$ . The Global model is 3× to 28× larger than the Local model, and the total FLOPs are lower than equivalently-sized standard Transformers or Linear Transformers across a wide range of model sizes and sequence lengths (Figure 3). The figure shows FLOPs per token as a function of total non-embedding parameters, demonstrating that MEGABYTE's advantage grows with model size — for the largest models (100B+ parameters), the efficiency gap is substantial.

Generation parallelism. The efficiency analysis for generation (Section 3.2) focuses not on absolute FLOPs but on the number of serial operations required, which determines wall-clock latency when GPU parallelism is not saturated by single-token processing. The key observation is that generating the $P$ bytes of a patch requires one Global model forward pass (serial depth $L_{\text{global}}$ ) followed by $P$ Local model forward passes (serial depth $P \cdot L_{\text{local}}$ ). By contrast, a standard Transformer of equivalent capacity would require $P$ forward passes of depth $L_{\text{global}} + L_{\text{local}}$ (since every token goes through all layers). When $L_{\text{global}} \gg L_{\text{local}}$ , MEGABYTE's serial depth is dominated by $P \cdot L_{\text{local}}$ , while the standard Transformer's is dominated by $P \cdot L_{\text{global}}$ , giving MEGABYTE a speedup factor approaching $L_{\text{global}} / L_{\text{local}}$ in the limit.

Table 6 provides empirical confirmation: a MEGABYTE model with a 1.3B Global model (24 layers, $D_{\text{model}}=2048$ , Table 11 size S4) and a 218M Local model (15 layers, $D_{\text{model}}=1024$ , Table 10) generates 8192 bytes in 93 seconds, compared to 132 seconds for a 350M standard Transformer (24 layers, $D_{\text{model}}=1024$ ) — a 40% speedup. The paper attributes this to "the bulk of the parameters being in the Global model, which only needs to be computed once for every 8 tokens, whereas all the parameters in the baseline model are used on every token."

Why this analysis is important for understanding the design. The entire architecture — the patch decomposition, the asymmetric model sizes, the additive interface between Global and Local — is motivated by the feedforward cost dominance argument. If attention were the primary cost, MEGABYTE's design (which still has $O(T^2/P)$ attention in the Global model) would be less compelling compared to linear attention architectures. But because feedforward layers dominate, and MEGABYTE achieves a factor-of- $P$ reduction in feedforward cost for the large model, the choice to focus on the feedforward bottleneck directly — rather than on more sophisticated attention mechanisms — is the paper's key strategic insight.

4. Key Insights and Innovations

Innovation 1: Reframing Long-Sequence Efficiency as a Feedforward Bottleneck, Not an Attention Problem

The most consequential conceptual move in this paper is not the MEGABYTE architecture itself but the diagnostic that motivates it: the primary computational bottleneck in scaling Transformers to long sequences is not self-attention, but the position-wise feedforward layers. This reframes the entire conversation around efficient Transformers, which had been overwhelmingly dominated by attention-centric solutions.

Prior to this work, the field's default assumption — reflected in the sheer volume of papers on sparse attention (Child et al., 2019; Roy et al., 2020; Beltagy et al., 2020; Kitaev et al., 2020), linear attention (Katharopoulos et al., 2020), and state space models (Gu et al., 2021) — was that the quadratic cost of self-attention was the obstacle to long-sequence modeling. The paper does not dispute that attention is quadratic; it disputes that this quadratic term is what matters in practice. Section 3.1 makes the empirical claim explicit: in the GPT-3 architecture, "the quadratic self-attention computation accounts for only 1.4% of FLOPS." The remaining 98.6% comes from feedforward layers — and that percentage grows as models scale, because feedforward cost scales with 2mT while attention cost scales with T^2 * d, and m grows faster than d in large models (Kaplan et al., 2020).

This is not a small correction. It is a fundamental reframing with direct implications for where research effort should be directed. If feedforward layers dominate, then solving the attention bottleneck — even achieving true O(T) attention with zero quality degradation — would yield at most a 1.4% efficiency improvement. The paper's efficiency analysis in Figure 3 makes this visually stark: MEGABYTE with P = 8 uses fewer FLOPS than Linear Transformers (which achieve O(T) attention) across a wide range of model sizes, precisely because Linear Transformers still apply the feedforward layers at every position, while MEGABYTE amortizes them across patches.

The significance of this reframing extends beyond this paper's architecture. It suggests that the entire subfield of efficient Transformers may have been optimizing the wrong objective — attention mechanisms — when the real leverage is in reducing per-position feedforward computation. MEGABYTE achieves this through temporal downsampling (running the large model less frequently), but the conceptual insight points toward other possibilities: conditional computation (only running the large model when needed), hierarchical feedforward sharing, or learned sparsity patterns in the feedforward layers themselves. The paper doesn't pursue these, but the diagnostic opens them as research directions.

Evidence for the practical impact of this reframing comes from the generation speed results in Table 6: a 1.5B-parameter MEGABYTE model generates 40% faster than a 350M standard Transformer, despite having over 4× the parameters. This counterintuitive result — larger model, faster inference — only makes sense once you understand that the large model's parameters are concentrated in the Global model, which runs once per 8 bytes rather than once per byte. The speedup comes from amortizing feedforward cost, not from attention efficiency.

Innovation 2: Learned, Differentiable, Lossless Tokenization as an Architectural Primitive

The paper frames MEGABYTE patches as "analogous to traditional lossless tokens, and the Local model performs the role of mapping a hidden state to a distribution over possible patches" (Section 9). This framing is more than a pedagogical analogy — it represents a conceptual shift in how we think about the relationship between tokenization and architecture.

The dominant paradigm in autoregressive modeling treats tokenization as a preprocessing step: design a fixed vocabulary (via BPE, SentencePiece, or discrete auto-encoders), map the raw input to token IDs using deterministic rules, train the model on token sequences, and map token predictions back to raw outputs using a deterministic decoder. This paradigm has several well-known flaws — the tokenizer is not learned end-to-end with the model, it introduces a hard information bottleneck (especially for non-text modalities), and it creates a sharp distinction between "in-vocabulary" and "out-of-vocabulary" inputs — but it has persisted because the alternative (modeling raw bytes with standard Transformers) was computationally intractable.

MEGABYTE dissolves this distinction by making the "tokenization" an integral, differentiable part of the model. The Global model's patch representations are effectively learned token embeddings: they compress P bytes of raw input into a single high-dimensional vector that captures the information needed for downstream prediction. But unlike traditional tokenization, this compression is (a) lossless — the byte sequence can be exactly reconstructed from the autoregressive Local model, (b) differentiable — gradients flow from the Local model's byte-level predictions back through the Global model's patch representations, allowing the "tokenization" to be optimized for the end task, and (c) contextual — a given byte subsequence receives different patch representations depending on the surrounding global context, unlike fixed token embeddings.

This is a fundamental shift from "tokenize, then model" to "model the tokenization as part of modeling." The paper doesn't make this framing as explicit as it could, but it's implicit in the architecture's structure: the Global model learns what information to extract from each patch to make the Local model's autoregressive prediction task as easy as possible. The additive interface between Global and Local — where the Global context is simply added to byte embeddings rather than concatenated or attended to — reflects the inductive bias that the Global model should provide a shift to the Local model's hidden state, steering it toward the right distribution without micromanaging each byte prediction.

This perspective has implications beyond byte-level modeling. It suggests a general design pattern for hierarchical autoregressive models: learn a coarse representation that captures long-range structure, then learn a fine representation that captures local detail conditioned on the coarse representation, and train both end-to-end with a single autoregressive loss. The coarse representation becomes a learned "tokenizer" that is optimized not for reconstruction fidelity or compression ratio, but for predictive utility to the fine model.

The evidence that this learned tokenization is genuinely effective comes from the ablation in Table 7: removing the Global model entirely (the "w/o global model" row) causes bits-per-byte to increase from 0.6871 to 1.373 on arXiv, from 3.477 to 3.659 on audio, and from 3.158 to 3.181 on ImageNet256. The Global model is providing information that the Local model cannot recover from local context alone — and it's doing so through a compressed patch representation that the model learned to construct.

Innovation 3: Asymmetric Capacity Allocation as a Principle for Hierarchical Sequence Models

MEGABYTE's architecture encodes a specific hypothesis about how computational capacity should be distributed across temporal scales in sequence modeling: most capacity should be allocated to the coarsest temporal scale (the Global model), with progressively less capacity at finer scales. This is not an obvious choice — one could imagine equal-capacity models at each scale, or even more capacity at the finest scale where the output predictions are made — and the paper demonstrates through controlled experiments that the asymmetric allocation is optimal.

The field's prior approach to hierarchical sequence modeling (when it existed at all) tended toward symmetric designs. Multi-scale Vision Transformers and hierarchical text encoders typically use comparable model sizes at each scale, or even larger models at finer scales where the representation dimensionality is higher. MEGABYTE inverts this: the Global model is 3× to 53× larger than the Local model across the configurations in Tables 10 and 12. Table 10 demonstrates this explicitly for the PG-19 dataset: at equal compute, increasing the Global model from 350M to 1.3B parameters (while decreasing the Local model from 290M to 218M) improves bits-per-byte from 1.014 to 0.991. The compute budget is spent optimally when the Global model has more parameters.

Why does this work? The paper's implicit argument is that prediction difficulty is not uniform across temporal scales. Deciding what word, pixel pattern, or audio event comes next is a harder problem than filling in the bytes of a word, pattern, or event once the high-level decision is made. This is the "most byte predictions are relatively easy" observation from the Introduction — completing the bytes of "the" given "th" requires very little model capacity; deciding whether the next word should be "the" or "therefore" given the preceding paragraph requires substantial capacity. By concentrating parameters at the coarse scale, MEGABYTE allocates capacity where it has the highest marginal impact on prediction quality.

This principle — match model capacity to the information-theoretic difficulty of the prediction at each temporal scale — is more general than MEGABYTE's specific two-level hierarchy. It suggests that for any sequence modeling task, one should analyze where the predictive entropy is concentrated: if most uncertainty is at coarse scales (what to say next), use a large model at the coarse scale and a small model at fine scales; if most uncertainty is at fine scales (e.g., modeling raw audio where waveform details are highly unpredictable), the capacity allocation might need to be different. The paper's results across modalities hint at this: the optimal patch size varies from 8 (text) to 192 (images) to 32 (audio), suggesting that the "right" temporal scale for the coarse model depends on the signal's statistical structure.

The evidence for this principle is indirect — the paper doesn't measure prediction difficulty at different scales directly — but the consistent finding that a larger Global model (even at the expense of a smaller Local model) improves performance across all modalities and patch sizes (Table 9, Table 10) supports the claim that coarse-scale capacity is the binding constraint.

Innovation 4: MEGABYTE as an Existence Proof for Tokenization-Free Modeling at Scale

Perhaps the paper's most practically significant contribution is simply demonstrating that tokenization-free byte-level autoregressive modeling is viable at scales that matter. Prior to this work, byte-level language modeling was generally considered to be substantially worse than subword modeling — the paper reports their own byte-level Transformer and PerceiverAR baselines achieving word-level perplexities of 69.4 and 88.8 on PG-19 (Table 3), compared to 26.5–36.3 for state-of-the-art subword models. The implicit consensus was that tokenization was necessary to compress the sequence into a manageable length and to provide the model with meaningful linguistic units.

MEGABYTE's 36.4 test perplexity on PG-19 (Table 3) — competitive with subword models — is an existence proof that this consensus was wrong, or at least contingent on a specific architectural regime. The model achieves this without any tokenization: no BPE vocabulary, no SentencePiece model, no discrete auto-encoder for images or audio. The vocabulary is always exactly 256 bytes, preprocessing is the identity function, and the model must learn all linguistic structure from raw bytes.

This result matters for several reasons beyond the specific numbers. First, it eliminates a class of preprocessing artifacts that have bedeviled language model deployment: tokenization-related bugs in prompting, unpredictable behavior with whitespace, and domain transfer failures where out-of-vocabulary tokens degrade performance. Second, it enables truly multimodal models where text, images, and audio can be processed through the same architecture with the same byte-level vocabulary — the paper demonstrates this by applying MEGABYTE to all three modalities with minimal modality-specific adaptation (only the patch scan ordering for images and the byte reading approach for audio). Third, it simplifies the training pipeline: no separate tokenizer training step, no vocabulary size hyperparameter to tune, no decisions about how to handle characters, subwords, or bytes.

The significance of this existence proof extends beyond MEGABYTE itself. It suggests that the long-standing assumption that tokenization is necessary for competitive language modeling may have been an artifact of inefficient architectures — that standard Transformers couldn't handle byte-level sequences not because byte-level modeling is inherently harder, but because the quadratic cost structure prevented models from being large enough to learn meaningful representations over byte sequences. By removing the architectural bottleneck, MEGABYTE shows that byte-level modeling can work, opening the door to other architectures that might achieve similar or better results without tokenization.

The caveat the paper itself raises — "the scale of experiments here is far below those of state-of-the-art language models" (Section 10) — is important: this existence proof is at the 1.5B-parameter scale on PG-19, not at the 100B+ scale of GPT-4 or PaLM. Whether the byte-level approach continues to be competitive at those scales, or whether tokenization provides benefits that re-emerge with larger models and datasets, remains an open question. But the direction is now empirically validated, where previously it was largely dismissed.

Innovation 5: Strided Inference as a Diagnostic for Architectural Weakness

Strided inference (Section 2.3.3) appears at first glance to be a minor inference-time trick — run the model twice with offset patches and combine the better-positioned predictions. But the paper's presentation of this technique, combined with the empirical analysis in Figure 5 and Table 8, reveals something more valuable: strided inference is a diagnostic tool that exposes a systematic weakness in the base architecture — the Local model's predictions degrade toward the end of a patch — and the fact that the degradation exists tells us something important about the Global-Local interface.

The observation that "the per-token loss within each patch would increase towards the end of the patch, as the prediction relies more on the weaker Local model" (Section 2.3.3) is a direct empirical finding about how the architecture's inductive biases play out in practice. The first byte of a patch benefits from (a) strong Global conditioning that sets the high-level direction, and (b) a clean start-of-patch signal from the padding embedding. By the last byte of the patch, the Global conditioning is unchanged (it's the same vector for the entire patch), and the Local model has had to autoregressively maintain coherence over P - 1 steps with limited capacity. The quality degradation is a direct measure of how well the Local model can sustain predictions without refreshed Global context.

This is a negative result with diagnostic value rather than a method contribution. It reveals that the Global-Local interface — specifically, the decision to provide a single Global context vector for an entire patch rather than per-byte Global context — imposes a real performance cost. The fact that strided inference helps (0.8926 bpb vs. 0.9079 at 2× cost in Table 8) shows that the cost is non-trivial. An alternative architecture that provided per-byte Global conditioning — for example, by having the Global model output a sequence of per-byte representations rather than a single per-patch representation — might eliminate the degradation without requiring the doubled inference cost. The paper does not explore this architectural alternative, but the strided inference analysis points directly toward it.

The finding also connects to the paper's earlier claim that "most byte predictions are relatively easy." If most byte predictions are easy, why does the Local model — which is specifically designed for these easy predictions — degrade over the course of a patch? The likely answer is that the difficulty comes not from individual byte predictions but from maintaining coherence over a sequence of predictions: each step's error compounds into the next step's input, and the limited-capacity Local model accumulates drift. This suggests that the "easy" in "most byte predictions are easy" applies to isolated predictions conditioned on clean context, but autoregressive prediction with self-conditioning introduces a qualitatively different difficulty that the base architecture doesn't fully address.

This is a subtle but intellectually honest contribution: the paper identifies a limitation of its own architecture, provides a workaround, and in doing so reveals a direction for future improvement. Not all innovations are about claiming success — sometimes the most valuable contribution is clearly characterizing where the current approach falls short and why.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper evaluates on three distinct modalities. For language modeling, five long-form text datasets are used: PG-19 (10.1GB, mean document size 411,404 bytes — English books before 1919; Rae et al., 2019b), Stories (21.3GB, mean 35,265 bytes — CommonCrawl subset; Trinh & Le, 2018), Books (79.7GB, mean 509,526 bytes — English books; Gao et al., 2020), arXiv (91.5GB, mean 58,518 bytes — LaTeX technical publications), and Code (353.7GB, mean 7,461 bytes — open-source code under Apache/BSD/MIT licenses). For image modeling, ImageNet at three resolutions (64×64 producing 12,288 bytes, 256×256 producing 196,608 bytes, and 640×640 producing 1,228,800 bytes; Oord et al., 2016) is used for density estimation. For audio modeling, a proprietary 2TB dataset (approximately 18,000 hours of 16 kHz, 16-bit audio, equivalent to 32k bytes per second) is used. The language modeling experiments use the full dataset sizes listed; image experiments use the standard ImageNet training set; audio details beyond dataset size and sample rate are not disclosed.
Base model(s). All experiments use decoder-only Transformer architectures built in the Metaseq codebase (Zhang et al., 2022b). For the MEGABYTE architecture, the Global and Local models are both standard decoder Transformers with pre-normalization, ReLU activations, and learned absolute position embeddings (rotary embeddings are used specifically for the cross-patch attention extension). Model sizes range from 62M to 2.7B parameters for the Global model and from 83M to 350M for the Local model, with specific configurations listed in Tables 11 and 12. Baseline Transformer and PerceiverAR models are constructed with matching forward-pass time per byte to enable compute-controlled comparison. No pretrained models are used — all models are trained from scratch for each experiment.
Metrics. The primary metric across all experiments is bits per byte (bpb), which is the negative log-likelihood per byte divided by log(2) — equivalently, the average number of bits required to encode each byte under the model's predicted distribution. Lower bpb indicates better density estimation. For language modeling comparisons with prior subword-level work (Table 3), bpb is converted to word-level perplexity using a mapping that accounts for the different tokenization granularities. The paper does not specify the exact conversion formula, but the standard approach is to compute perplexity as 2^(bpb × bytes_per_word) where bytes_per_word is computed empirically from the dataset. For generation speed (Table 6), wall-clock time to generate 8192 bytes is reported in seconds. Image modeling follows the standard autoregressive density estimation protocol on ImageNet (Oord et al., 2016).
Baselines. Three model architectures are compared throughout: (1) a standard decoder-only Transformer (GPT-style, using the Metaseq implementation with pre-norm, ReLU, and learned position embeddings), (2) PerceiverAR (Hawthorne et al., 2022), which extends the Transformer with a single cross-attention layer over a longer context sequence through a latent bottleneck — described by the paper as "the best performing general purpose autoregressive model we are aware of," and (3) MEGABYTE itself. The authors re-implemented PerceiverAR in the Metaseq codebase and validated their implementation by reproducing the ImageNet 64×64 result from Hawthorne et al. (2022), achieving 3.53 bpb compared to the reported 3.54 (Appendix C). For the PG-19 scaling experiment (Table 3), additional published baselines are included: TransformerXL (Rae et al., 2019a), CompressiveTransformer (Rae et al., 2019a), BlockRecurrent Transformer (Hutchins et al., 2022), and two custom byte-level baselines (a standard Transformer and PerceiverAR trained at byte level). The subword baselines in Table 3 use SentencePiece tokenization with 32k vocabulary.
Generation budget / compute accounting. The paper's central methodological commitment is compute-controlled experimentation: "We conduct experiments using a fixed compute and data budget across all models to focus comparisons solely on the model architecture rather than training resources" (Section 4.1). The approach has two components. First, model hyperparameters (primarily the number of layers) are adjusted within each architecture so that the forward pass time per byte is matched across architectures. This means a MEGABYTE model with a 758M Global and 262M Local model is compared to a 320M Transformer and a 248M PerceiverAR because these configurations have approximately equal per-byte processing time. Second, all models are trained for the same number of bytes (80 billion bytes for the controlled experiments in Table 2; 400 billion bytes for the PG-19 scaling experiment in Table 3; 1.4 trillion tokens for the ImageNet 64×64 state-of-the-art comparison in Table 4). For image and audio experiments, the compute matching is done similarly: model sizes are adjusted to equalize overall training speed, and all models process the same amount of data. The paper does NOT use FLOPs directly as the compute metric — instead, wall-clock training speed matching serves as a proxy that accounts for practical implementation efficiency. For the state-of-the-art ImageNet 64×64 comparison, GPU hours are reported as an additional metric: the MEGABYTE model is estimated to use "less than half the GPU hours we would have needed to reproduce the best PerceiverAR model" (Section 6.2). For generation speed (Table 6), wall-clock time in seconds is the metric.
Cross-validation / statistical protocol. The paper does not report cross-validation, statistical significance tests, confidence intervals, or standard deviations for any result. All tables report single bpb or perplexity values without error bars. The controlled experiments (Tables 2, 5, 7, 8, 9, 10, 13, 14) use a single training run per configuration. The scaling experiment (Table 3) reports validation and test set perplexities on PG-19, following the standard test set evaluation protocol from Rae et al. (2019a). For the ImageNet 64×64 state-of-the-art result (Table 4), the paper reports test set bpb against published numbers from prior work. The absence of variance estimates means that small differences between configurations (e.g., 0.6781 vs. 0.6871 bpb in the cross-patch attention ablation for arXiv, Table 7) cannot be assessed for statistical reliability.

Main Quantitative Results

Compute-Controlled Language Modeling Across Five Datasets

Table 2 reports bits-per-byte for the three architectures trained on 80 billion bytes each, with model sizes tuned for equal per-byte forward pass time: Transformer at 320M parameters (context length 1024), PerceiverAR at 248M parameters (context length 8192, latent size 1024), and MEGABYTE with 758M Global / 262M Local parameters (context length 8192, patch size 8). MEGABYTE achieves the lowest bpb on all five datasets: PG-19: 1.000 (vs. Transformer 1.057, PerceiverAR 1.104); Stories: 0.978 (vs. 1.064, 1.070); Books: 1.007 (vs. 1.097, 1.104); arXiv: 0.678 (vs. 0.816, 0.791); Code: 0.411 (vs. 0.575, 0.546). The relative improvement over the Transformer baseline ranges from approximately 5% on PG-19 to 28% on Code. Notably, PerceiverAR underperforms the standard Transformer on three of five datasets (PG-19, Stories, Books) despite its longer context — the paper does not discuss why, but possible explanations include the latent bottleneck's information compression cost or suboptimal hyperparameter tuning in the Metaseq reimplementation.

A critical detail: the Transformer baseline uses a context length of only 1024 bytes, while MEGABYTE and PerceiverAR use 8192 bytes. This means the Transformer sees substantially less context per prediction, which could contribute to its inferior performance on the long-document datasets (PG-19 and Books have mean document lengths of 411k and 510k bytes respectively, far exceeding any model's context). The paper does not provide an ablation where the Transformer is given the same context length (which would require adjusting model size to maintain the compute budget — a Transformer with 8192-byte context at equal per-byte compute would need to be substantially smaller). This makes it difficult to disentangle whether MEGABYTE's advantage comes from the architecture per se or simply from having access to longer context. The PerceiverAR comparison partially controls for this (it also has 8192-byte context) but introduces its own architectural differences.

Scaling Experiment on PG-19: Competitive with Subword Models

Table 3 reports results from training byte-level models on 400 billion bytes of PG-19 (4× the data of the controlled experiments, with the same model configurations). The MEGABYTE model achieves test word-level perplexity of 36.4, compared to 69.4 for the byte-level Transformer and 88.8 for byte-level PerceiverAR. This is a dramatic gap: MEGABYTE is roughly 2× better than the next-best byte model. When compared to published subword-level models (which use SentencePiece tokenization with 32k vocabulary and benefit from unsupervised pretraining on different data scales), MEGABYTE's 36.4 is competitive with TransformerXL (36.3), CompressiveTransformer (33.6), and PerceiverAR with subwords (28.9). The best published subword result is BlockRecurrent at 26.5, substantially lower than MEGABYTE. However, these comparisons are confounded: the subword models may have been trained on different amounts of data, used different model sizes, or benefited from different hyperparameter tuning. The paper is explicit about this limitation: "These results may be confounded by differing amounts of compute and tuning used, but show that MEGABYTE gives results competitive with state-of-the-art models trained on subwords."

The paper also reports that extending MEGABYTE's context length from 8192 to 16384 bytes on PG-19 does not improve bpb (0.8787 vs. 0.8751, Table 14) — a negative result consistent with Hawthorne et al. (2022)'s finding that longer context does not always help. The authors speculate that "we will benefit more from longer sequence when we further scale up the model size and data" (Appendix D.3).

Image Modeling: State-of-the-Art at 64×64, Strong Scaling to 1.2M Bytes

On ImageNet 64×64 (Table 4), MEGABYTE achieves 3.40 bpb, matching the PerceiverAR result from Hawthorne et al. (2022) while using approximately half the GPU hours for training. The MEGABYTE configuration uses a 2.7B Global model and 350M Local model with patch size 12, processing the full 12,288-byte sequence in a single forward pass. This ties with the prior state-of-the-art for autoregressive density estimation on ImageNet 64×64 (Routing Transformer at 3.43, Combiner at 3.42). The GPU hour estimate is self-reported ("we estimate that training this model consumed less than half the GPU hours we would have needed to reproduce the best PerceiverAR model") without specification of hardware or precise numbers.

On multi-resolution scaling (Table 5), three architectures are compared under equal compute and data budgets at resolutions 64×64 (12,288 bytes), 256×256 (196,608 bytes), and 640×640 (1,228,800 bytes). Critically, the Transformer and PerceiverAR baselines cannot process the full sequence lengths at reasonable model sizes: the Transformer is limited to context length 1024, and PerceiverAR to context length 12,000 (with 1024 latents at 64×64, 768 latents at higher resolutions). This means these baselines must split images into segments and cannot model cross-segment dependencies. MEGABYTE processes the full sequence in all cases, with patch sizes of 12 (64×64), 192 (256×256), and 192 (640×640). Results: at ImageNet 64×64, MEGABYTE 3.52 bpb vs. PerceiverAR 3.55 vs. Transformer 3.62; at ImageNet 256×256, MEGABYTE 3.158 vs. PerceiverAR 3.373 vs. Transformer 3.801; at ImageNet 640×640 (over 1.2M tokens), MEGABYTE 2.282 vs. PerceiverAR 2.345 vs. Transformer 2.847. MEGABYTE leads at all resolutions, with the gap widening as resolution increases — at 640×640, the advantage over PerceiverAR is 0.063 bpb and over Transformer is 0.565 bpb.

An important caveat: the decreasing bpb with increasing resolution (2.282 at 640×640 vs. 3.158 at 256×256) is not because higher-resolution images are easier to model — it reflects that the bpb metric averages over all bytes, and at higher resolutions a larger fraction of bytes are "easy" predictions (e.g., smooth regions, redundant color channels) that compress well. The paper does not discuss this metric property, which is important for interpreting cross-resolution comparisons.

Audio Modeling

On the 2TB audio dataset with sequences of length 524,288 bytes and patch size 32, MEGABYTE achieves 3.477 bpb, compared to 3.543 for PerceiverAR and 3.567 for the Transformer baseline (exact numbers from the audio ablation row in Table 7, with context: Transformer at 1024 bytes, PerceiverAR at 8192 bytes with 1024 latents, MEGABYTE at 524,288 bytes with patch size 32). The paper provides only these aggregate numbers without per-dataset breakdown or generation quality evaluation (e.g., no audio samples or human evaluation). The audio dataset is not publicly described beyond size and sample rate, making reproduction impossible.

Generation Speed

Table 6 reports both bpb and generation wall-clock time for MEGABYTE versus a standard Transformer on PG-19. MEGABYTE (1.3B Global, 218M Local, context 8192, patch size 8) achieves 0.991 bpb and generates 8192 bytes in 93 seconds. The Transformer baseline (350M, context 1024) achieves 1.064 bpb and takes 132 seconds. MEGABYTE is thus both more accurate (0.991 vs. 1.064 bpb) and 40% faster at generation (93s vs. 132s) despite having over 4× the total parameter count (1.52B vs. 350M). The paper attributes the speedup to the Global model's parameters being computed once per 8 tokens rather than once per token. However, the Transformer baseline uses a shorter context (1024 vs. 8192), meaning it cannot model dependencies beyond 1024 bytes — a potential confound for both the accuracy and speed numbers. The generation speed measurement protocol (hardware, batch size, KV-caching implementation, whether the baseline Transformer also uses KV-caching) is not specified.

Context Utilization

Figure 4 shows the average log probability assigned to tokens at different positions within the 8192-byte context window, comparing MEGABYTE and the Transformer baseline on the PG-19 test set. MEGABYTE's likelihoods rise monotonically throughout the context window, indicating that later positions benefit from the full preceding context — the model "can use tokens from 8k bytes previously to improve its predictions" (Section 8.3). The Transformer baseline (context length 1024) naturally cannot use context beyond 1024 bytes, but within its window, the paper does not describe whether its likelihoods also rise. The figure serves primarily as evidence that MEGABYTE's long context is genuinely useful, not merely architectural.

Ablation Studies and Robustness Checks

Global vs. Local model necessity (Table 7, "w/o local model" and "w/o global model" rows). Removing either the Local model or the Global model causes catastrophic performance degradation across all three tested modalities. Without the Local model (meaning the Global model must directly predict byte distributions from patch representations — effectively a patch-level autoregressive model with the output space factorized into conditionally independent per-byte predictions), bpb increases from 0.6871 to 1.263 on arXiv, from 3.477 to 5.955 on audio, and from 3.158 to 4.768 on ImageNet256. Without the Global model (meaning only the Local model processes byte sequences with no cross-patch context beyond what cross-patch attention provides), bpb increases from 0.6871 to 1.373 on arXiv, from 3.477 to 3.659 on audio, and from 3.158 to 3.181 on ImageNet256. Both components are essential, but the Global model's removal is less catastrophic on ImageNet256 (3.158 to 3.181, an increase of only 0.023 bpb) than on text or audio, suggesting that for images, the Local model with its 192-byte patches already captures substantial contextual information from the patch scan ordering.

Cross-patch attention (Table 7, "w/o cross-patch attention" row). Removing cross-patch attention (the mechanism that lets the Local model attend to the last r positions of the previous patch) has mixed effects: on arXiv text, bpb actually improves slightly from 0.6871 to 0.6781 (an unexpected result — perhaps cross-patch attention introduces noise for text where patch boundaries often align with word boundaries); on audio, bpb degrades slightly from 3.477 to 3.481; on ImageNet256, bpb degrades more noticeably from 3.158 to 3.259. This modality-dependent pattern suggests that cross-patch attention is most valuable when continuous structures cross patch boundaries (image textures, audio waveforms) and least valuable when patches naturally align with discrete units (text tokens). The paper describes the architecture as "robust to this modification" given the modest degradation on most modalities.

Convolutional Patch Encoder (Table 7, "w/ CNN encoder" row). Adding causal convolutional layers (kernel sizes 3, 5, 7) before the Patch Embedder provides small improvements on audio (3.475 vs. 3.477) and ImageNet256 (3.155 vs. 3.158) but no change on arXiv text (0.6871 in both cases). The improvement is consistently small — at most 0.003 bpb — suggesting the translation invariance problem is real but modest in impact for these patch sizes. The paper does not ablate kernel sizes, number of layers, or compare against alternative solutions to the translation invariance problem (e.g., learned position biases within patches, overlapping patches).

Patch size (Table 9). On ImageNet256, varying the patch size from 48 to 192 to 768 (while adjusting Local model layers to maintain compute budget: 11, 12, and 8 layers respectively) yields bpb of 3.178, 3.158, and 3.186. The optimal patch size is 192, but the range is narrow — all three values are within 0.028 bpb of each other — demonstrating that "there is a wide range of values where MEGABYTE performs similarly" (Section 8.5). The paper claims similar robustness across modalities but only reports this ablation for ImageNet256.

Local-to-Global model size ratio (Table 10). On PG-19 with patch size 8, three configurations with equal compute are compared: Global 350M / Local 290M (bpb 1.014), Global 760M / Local 262M (bpb 1.002), and Global 1.3B / Local 218M (bpb 0.991). Increasing the Global model size at the expense of the Local model consistently improves performance, with the largest Global model achieving the best result. This confirms the paper's hypothesis that a given compute budget is "spent optimally when the Global model has more parameters than the Local model" (Section 8.5). The paper reports this trend as "consistent across all modalities and various patch sizes" but only provides this one table.

Strided inference and sliding window (Table 8). On the PG-19 test set with the best MEGABYTE model: basic inference (1× cost) achieves 0.9079 bpb; adding sliding window (2× cost) achieves 0.8918; strided inference alone (2× cost) achieves 0.8926; combining both (4× cost) achieves 0.8751. Strided inference provides an improvement of 0.0153 bpb over basic inference at doubled cost, roughly equivalent to the sliding window improvement (0.0161 bpb). The fact that the improvements are additive (strided + sliding at 4× cost is better than either alone at 2×) suggests these techniques address different weaknesses — sliding window helps with context boundary effects at the sequence level, while strided inference helps with within-patch position-dependent degradation. The paper does not report generation speed impact for any of these techniques, only the quality improvement.

Longer context (Table 14). Extending MEGABYTE context from 8192 to 16384 bytes (both with patch size 8) on PG-19 yields 0.8751 vs. 0.8787 bpb — effectively no improvement. This is a negative result consistent with prior findings (Hawthorne et al., 2022) and suggests that either the 8192-byte context is sufficient for PG-19's dependency structure, or that the Global model cannot effectively utilize longer context at this model scale.

Patch scan vs. raster scan (Table 13). For image modeling on ImageNet256, patch scan (splitting the 2D image into 2D patches and raster-scanning within and between patches) substantially outperforms raster scan (linearizing the full image row-by-row): MEGABYTE with patch scan achieves 3.158 bpb vs. 3.428 with raster scan; PerceiverAR with patch scan achieves 3.373 bpb vs. 3.552 with raster scan. This 0.27 bpb improvement for MEGABYTE and 0.179 for PerceiverAR demonstrates that respecting the 2D structure of images through patch-based scanning is important — likely because it keeps spatially adjacent bytes closer in the sequence, making the autoregressive local prediction task easier.

PerceiverAR implementation validation (Appendix C). The paper validates its PerceiverAR reimplementation by reproducing the "Standard Ordering" experiment from Hawthorne et al. (2022) on downsampled ImageNet 64×64, achieving 3.53 bpb compared to the reported 3.54. This is a 0.01 bpb difference, suggesting the reimplementation is faithful. Differences noted: the authors use standard attention dropout instead of cross-attention dropout, and did not implement chunked attention.

Critical Assessment

Claim 1: MEGABYTE "enables end-to-end differentiable modeling of sequences of over one million bytes" (Abstract) and "scales well to sequences of over 1M tokens" (Section 6.3).

What was demonstrated: Table 5 shows MEGABYTE processing ImageNet 640×640 images as a single forward pass of 1,228,800 bytes and achieving better bpb than baselines that must split the input into segments. This directly demonstrates the architectural capability: MEGABYTE can process sequences of over 1M bytes in a single differentiable forward pass, while the standard Transformer and PerceiverAR baselines cannot at comparable model sizes.

Caveats: The claim is about architectural capability, not necessarily about effective utilization of all 1.2M bytes. Table 14 shows that doubling context from 8192 to 16384 on PG-19 does not improve bpb, suggesting that having the architectural capability is not sufficient — the model must also be able to learn to use the additional context. For the 640×640 images, the paper does not report whether the full 1.2M-byte context is actually used (analogous to the Figure 4 analysis for text) or whether similar results could be achieved with shorter context. The patch size of 192 for ImageNet 640×640 means the Global model processes only 6400 patches, which is fewer than the 8192-byte context used for text (1024 patches at P=8). The sequence length that the Global model actually attends over is thus moderate (6400 patches), and the "over one million bytes" figure is realized through the Local model's within-patch processing. This is not a weakness — it's exactly how the architecture works — but the framing in the abstract is slightly misleading if one assumes the claim is about attention over 1M positions.

Claim 2: MEGABYTE "allows byte-level models to perform competitively with subword models on long context language modeling" (Abstract, Table 3).

What was demonstrated: Table 3 shows MEGABYTE achieving test word perplexity of 36.4 on PG-19, compared to 26.5–36.3 for published subword models and 69.4–88.8 for byte-level Transformer/PerceiverAR baselines. This demonstrates that byte-level modeling can be in the same performance range as subword modeling.

What was NOT demonstrated: The comparison with subword models is not controlled — different models used different amounts of training data, different model sizes, and different hyperparameter optimization. The subword models were trained by different research groups with different infrastructure, and the paper acknowledges the confound explicitly. A proper demonstration would require training subword Transformer and MEGABYTE models under the same compute and data constraints on the same dataset, which was not done. The subword models also benefit from unsupervised pretraining on large corpora before fine-tuning on PG-19, while MEGABYTE is trained from scratch on PG-19 bytes only. This means the comparison is MEGABYTE (trained on PG-19 bytes only) vs. subword models (pretrained on much larger corpora then fine-tuned), which is biased against MEGABYTE — yet MEGABYTE is still competitive, which actually strengthens the case if anything. But we cannot attribute the competitiveness to architecture alone.

Additionally, the claim is tested on only one dataset (PG-19) at one scale (400B bytes). Whether byte-level MEGABYTE remains competitive with subword models at the 1T+ byte scales used for production LLMs, or on non-book text domains, is completely unaddressed.

Claim 3: MEGABYTE achieves "state-of-the-art density estimation on ImageNet" (Abstract, Table 4).

What was demonstrated: Table 4 shows MEGABYTE at 3.40 bpb on ImageNet 64×64, matching PerceiverAR's 3.40. The paper estimates it used half the GPU hours. This is a state-of-the-art matching result achieved with lower training cost.

What was NOT demonstrated: "Half the GPU hours" is an estimate, not a measurement, and no hardware details or precise hour counts are provided. The comparison is against the PerceiverAR configuration from Hawthorne et al. (2022), which was not necessarily optimized for training speed — it's possible that a speed-optimized PerceiverAR configuration could achieve similar training throughput to MEGABYTE. More importantly, the paper does not compare against the full range of ImageNet 64×64 methods beyond Routing Transformer and Combiner, and the field has progressed since these results were published — the paper does not cite or compare against more recent methods.

Claim 4: MEGABYTE "improves generation speed" — specifically, 40% faster than a 350M Transformer despite having over 4× the parameters (Section 8.1, Table 6).

What was demonstrated: Table 6 shows wall-clock generation of 8192 bytes taking 93 seconds for MEGABYTE (1.52B total parameters) vs. 132 seconds for a 350M Transformer. This is a 40% speedup. The mechanism — amortizing the Global model's computation across 8 bytes — is sound and the result validates it.

What was NOT demonstrated: The generation speed comparison is confounded by context length: the Transformer baseline uses context 1024 while MEGABYTE uses context 8192. A fairer comparison would be a Transformer with the same context length, but this would require using a much smaller Transformer (to match the forward-pass time per byte), which would likely have much worse bpb — making the speed comparison favor MEGABYTE even more, but for reasons related to the Transformer's quality degradation rather than architectural efficiency per se. The paper also does not specify the generation protocol: whether KV-caching is used for both models, whether the baseline Transformer uses any of the standard generation optimizations, what hardware is used, whether batch size is 1, and whether the speedup measurement is averaged over multiple runs. The claim is plausible given the architectural analysis, but the empirical evidence is minimal.

Claim 5: The Local and Global models are both "critical to strong performance" (Table 7).

What was demonstrated: Table 7 shows catastrophic degradation when either model is removed — bbp increases by 0.5–1.5+ on arXiv and audio, and by smaller amounts on ImageNet256. This strongly supports the claim.

What was NOT demonstrated: The "w/o local model" and "w/o global model" ablations do not isolate the contribution of the two-model design from the contribution of increased total parameters or context length. When the Local model is removed, the architecture changes fundamentally (it becomes a patch-level autoregressive model with conditionally independent byte predictions), so the performance drop could be due to this architectural change rather than the loss of the Local model per se. An alternative ablation that kept the same total parameters but allocated them differently (e.g., a single Transformer with the same total FLOPs, or a MEGABYTE variant where capacity is shifted entirely to the Global or Local model) would more cleanly isolate the benefit of the two-level architecture. These ablations were partially done in Table 10 (varying the Global/Local ratio) but not to the extreme of fully allocating all capacity to one model.

Missing Experiments That Would Strengthen the Paper

Tokenization-equivalent baseline. A Transformer or PerceiverAR trained on byte-pair encoded text (BPE with vocabulary size ~32k) under the same compute and data constraints would enable a direct, controlled comparison between MEGABYTE's architectural efficiency and tokenization's compression efficiency. This is the most obvious missing baseline — without it, the claim that MEGABYTE enables tokenization-free modeling cannot be fully evaluated, because we don't know whether a tokenized Transformer at equal compute would outperform MEGABYTE on all tasks.

Attention vs. feedforward ablation. The paper argues that feedforward layers dominate compute cost, but never provides an ablation that isolates the benefit of amortized feedforward layers from the benefit of sub-quadratic attention. A MEGABYTE variant that uses full quadratic attention within the Global model but keeps the amortized feedforward structure, compared to one that uses linear attention but per-byte feedforward layers, would separate these contributions. The current design bundles both benefits together.

Difficulty-based analysis. The paper claims that "most byte predictions are relatively easy" but never measures prediction difficulty directly. An analysis showing the per-byte loss as a function of position within linguistic units (e.g., loss on the first byte of a word vs. subsequent bytes) would directly support the core motivation for the asymmetric capacity allocation.

Scaling curves. The paper shows one or two scale points per experiment (80B bytes, 400B bytes) but never produces scaling law plots showing how performance improves with compute and data for MEGABYTE vs. baselines. This is a notable absence given the field's emphasis on scaling behavior. The single-scale comparisons leave open the possibility that MEGABYTE's advantages are specific to the chosen compute budgets and would diminish or reverse at different scales.

Multiple random seeds. None of the experiments report variance across training runs. Given that the differences between some configurations are small (0.01–0.02 bpb), the reproducibility of the comparisons is uncertain. This is particularly relevant for the ablation results and the patch size robustness claims.

Causal contribution of patch offset and padding. The paper describes the patch offset (input is shifted by one patch relative to output) and padding embeddings as essential for maintaining the autoregressive property, but never ablates whether alternative designs (e.g., predicting patches from the Global model in a non-autoregressive order, or using different padding schemes) would work. The design is presented as a fixed consequence of the autoregressive constraint rather than as an empirically tested choice.

6. Limitations and Trade-offs

The Compute-Controlled Comparisons Are Constrained to a Single, Narrow Operating Point

The assumption or constraint. The paper's core methodological commitment is compute-controlled experimentation: "We conduct experiments using a fixed compute and data budget across all models to focus comparisons solely on the model architecture rather than training resources" (Section 4.1). This means that for each experiment, model hyperparameters are adjusted so that forward-pass time per byte is matched across architectures, and all models are trained on exactly the same number of bytes. While this isolates architectural differences cleanly, it produces comparisons at only a single compute budget per experiment — the paper never produces scaling curves showing how MEGABYTE's advantages change as compute and data increase.

The consequence. The reported improvements (e.g., MEGABYTE at 1.000 bpb vs. Transformer at 1.057 on PG-19 in Table 2; MEGABYTE at 0.991 bpb vs. Transformer at 1.064 in Table 6) are valid at the specific compute budgets tested (80B bytes for controlled experiments, 400B bytes for the scaling experiment, 1.4T tokens for ImageNet 64×64) but cannot be extrapolated. It is possible that MEGABYTE's advantages are largest at these moderate compute scales — where the Global model's extra capacity provides substantial benefit — and would diminish as compute increases and standard Transformers become large enough to model byte-level structure adequately on their own. Conversely, MEGABYTE's advantages might compound at larger scales if the feedforward bottleneck grows more severe with model size, as the paper's own analysis suggests. Without scaling curves showing bpb as a function of FLOPs or parameters for each architecture, a practitioner cannot determine whether MEGABYTE is a better investment at their specific compute budget.

What evidence exists in the paper. The scaling experiment in Table 3 provides one additional data point (80B → 400B bytes on PG-19), but the comparison is between different model configurations, not a controlled scaling sweep — the MEGABYTE model used for the 400B experiment is the same configuration (758M/262M) as the 80B experiment, so we don't see how MEGABYTE benefits from scaling model size alongside data. Table 14 shows that doubling context length from 8192 to 16384 bytes doesn't improve bpb at the tested scale, but this tests sequence length scaling, not compute scaling. The efficiency analysis in Figure 3 shows theoretical FLOPs-per-token curves for different model sizes, but these are not matched to empirical performance measurements.

Mitigation status. Not addressed. The paper acknowledges this implicitly by calling for "future work should explore scaling MEGABYTE to much larger models and datasets" (Section 10), but does not frame the single-point comparisons as a limitation for practitioners trying to decide whether to adopt the architecture at their specific scale.

The Global Model Has No Access to Fine-Grained Intra-Patch Structure, Forcing the Local Model to Maintain Coherence Without Refreshed Global Context

The assumption or constraint. The architecture provides exactly one Global context vector per patch: the Global model outputs a single representation h_k^{\text{global-out}} of dimension P · D_G for patch k, which is then reshaped into P vectors of dimension D_G and projected to condition the Local model at each byte position within the patch. Critically, this means all P bytes within a patch receive the same Global conditioning — the Global model's representation for the patch does not change as the Local model autoregressively steps through the patch's bytes. The Global model has no mechanism to provide byte-level feedback (e.g., "the first three bytes of this patch suggest a noun, so adjust the context for the remaining bytes accordingly").

The consequence. The Local model must maintain prediction coherence over P autoregressive steps with a fixed Global context vector and its own limited capacity. The paper demonstrates empirically that this creates a systematic degradation: "the per-token loss within each patch would increase towards the end of the patch, as the prediction relies more on the weaker Local model" (Section 2.3.3). Figure 5 visualizes this degradation — the probability assigned to tokens decreases with position within the patch — and this observation is what motivates strided inference, a workaround that doubles inference cost. The degradation means that MEGABYTE is effectively trading off per-byte prediction quality at later patch positions for the efficiency of amortized Global computation, and the tradeoff grows more severe as patch size increases — yet larger patch sizes are exactly what the architecture needs for greater efficiency gains.

What evidence exists in the paper. Figure 5 directly shows the within-patch probability degradation. Table 8 quantifies the cost: strided inference at 2× cost recovers 0.0153 bpb (0.9079 → 0.8926), and combining strided inference with sliding window at 4× cost recovers 0.0328 bpb (0.9079 → 0.8751). These are non-trivial improvements that reveal how much performance is left on the table by the fixed-per-patch Global context. Table 9 shows patch size sensitivity — 48, 192, and 768 all perform similarly on ImageNet256 — but this is measured on images where patches are 2D blocks and the degradation may be less severe than on 1D sequences like text. The paper does not report within-patch loss curves for text, where the degradation might be more pronounced due to the sequential nature of linguistic structure.

Mitigation status. Partially addressed through strided inference (Section 2.3.3), but this is an inference-time workaround that doubles cost, not an architectural fix. The paper acknowledges the limitation by proposing strided inference, but does not explore architectural alternatives that would address the root cause — for example, having the Global model output a sequence of per-byte conditioning vectors for each patch (increasing cost but potentially eliminating the degradation), or using a lightweight feedback mechanism within the Local model to update the "Global context" as bytes are generated. A practitioner building on MEGABYTE would need to either accept the within-patch quality degradation (and the associated tradeoff between patch size and per-byte accuracy) or pay the doubled inference cost of strided inference.

Difficulty Estimation and Dynamic Allocation Are Completely Absent — Every Byte Gets the Same Architectural Treatment

The assumption or constraint. MEGABYTE applies the same two-level processing to every byte of every sequence, regardless of how difficult or easy that byte's prediction is. The Global model provides the same amount of context for a patch containing a highly predictable byte sequence (e.g., the middle of a common word, a smooth region in an image) as it does for a patch containing a difficult structural transition (e.g., a word boundary, an object boundary in an image). The Local model expends the same computation within every patch, regardless of whether the within-patch predictions are trivial completions of obvious patterns or ambiguous choices requiring detailed reasoning.

The consequence. This is a missed opportunity for additional efficiency. The paper's own motivation rests on the observation that "most byte predictions are relatively easy (for example, completing a word given the first few characters), meaning that large networks per-byte are unnecessary" (Section 1). But this observation only informs the static allocation of capacity (large Global, small Local) — it does not lead to dynamic allocation where truly difficult bytes receive more computation than trivially predictable ones. For example, the first byte of a new word or the first byte after a punctuation mark might benefit from extra Global context or a larger Local model, while the third byte of "ing" needs almost no computation. By treating all patches and all bytes uniformly, MEGABYTE likely wastes computation on easy predictions and under-serves difficult ones, especially within patches of fixed size that may contain a mix of easy and hard byte predictions.

What evidence exists in the paper. The within-patch loss degradation in Figure 5 is indirect evidence — bytes toward the end of a patch have higher loss, but we cannot determine whether this is because those bytes are inherently harder (e.g., they happen to be at word boundaries) or because the Local model's accumulated drift makes them harder. The paper does not analyze per-byte difficulty, does not measure the variance of prediction entropy across bytes, and does not explore adaptive computation within patches. The consistent finding that larger Global models improve performance (Table 10) hints that some patches benefit substantially from Global context, but we don't know which ones.

Mitigation status. Not addressed at all. The architecture has no mechanism for per-byte or per-patch adaptive computation. This is a design choice — MEGABYTE's simplicity (uniform patches, uniform local processing) is part of what makes it implementable and parallelizable — but it leaves efficiency on the table and prevents the architecture from allocating more compute to the bytes that need it most. A practitioner building on MEGABYTE might consider adding adaptive patch sizes (larger patches for easy regions, smaller for difficult ones), dynamic depth in the Local model, or a confidence-based early-exit mechanism.

The Training Data Scales Are Far Below State-of-the-Art Language Models, and Scaling Behavior Is Unexplored

The assumption or constraint. The paper evaluates MEGABYTE on language modeling datasets totaling up to 353.7GB of code and 91.5GB of arXiv text, with the largest training run consuming 400 billion bytes (~372 GB) on PG-19 (Table 3). For comparison, production LLMs are typically trained on multiple terabytes to petabytes of text. The paper explicitly acknowledges this: "the scale of experiments here is far below those of state-of-the-art language models (Brown et al., 2020), and future work should explore scaling MEGABYTE to much larger models and datasets" (Section 10). Furthermore, the PG-19 scaling results in Table 3 show a gap between MEGABYTE (36.4 word perplexity) and the best subword model (BlockRecurrent at 26.5) that is substantial — nearly 10 perplexity points — and we don't know whether this gap would close, persist, or widen with more compute.

The consequence. A practitioner deciding whether to adopt MEGABYTE for a large-scale language modeling project has no evidence about whether the architecture continues to outperform standard Transformers (or remains competitive with subword models) at the 1T+, 10T+, or 100T+ byte scales used in production. The feedforward bottleneck argument suggests MEGABYTE's relative advantage should grow with model size (since feedforward layers become an even larger fraction of total FLOPS in larger models), but this is a theoretical argument not backed by empirical scaling evidence. It is equally plausible that with enough parameters and data, a standard byte-level Transformer learns internal representations that achieve the same kind of hierarchical processing MEGABYTE explicitly builds in — making the architectural complexity unnecessary at sufficient scale. The 40% generation speedup reported in Table 6 (93s vs. 132s for 8192 bytes) is also measured at a relatively small scale (1.52B MEGABYTE vs. 350M Transformer), and we cannot assume this speedup ratio holds at 10B or 100B parameter scales where hardware utilization characteristics change substantially.

What evidence exists in the paper. The paper provides exactly two scale points for PG-19 language modeling: 80B bytes (Table 2, bpb 1.000) and 400B bytes (Table 3, bpb 0.8751 with strided inference and sliding window, or ~0.908 with basic inference). This is a significant improvement from 4× more data, but it's only two points — not a scaling curve. For images, the state-of-the-art result at ImageNet 64×64 (Table 4) uses 1.4T tokens, which is a more substantial scale, but this is a single point with no comparison to how baselines would perform at the same scale (the PerceiverAR comparison is to their published result, which was not trained under the same data budget). The audio experiments use a 2TB dataset, which is substantial for audio but is a single data point with no scaling analysis. Figure 3 provides theoretical FLOPs-per-token curves up to 173B parameters, but these are not matched to empirical perplexity measurements.

Mitigation status. The paper explicitly flags this as future work (Section 10) and provides the FLOPs analysis in Figure 3 as a theoretical argument that advantages should persist at scale. But the absence of empirical scaling evidence is the most significant gap for practitioners considering adoption at production scale, and the paper does not attempt to partially address it through smaller-scale scaling law experiments (e.g., training multiple model sizes at multiple data scales and fitting power laws).

Performance Is Measured Exclusively Through Bits-Per-Byte — No Downstream Task Evaluation or Qualitative Assessment

The assumption or constraint. Every result in the paper is reported as bits-per-byte (or word-level perplexity derived from bpb for comparison with prior work). This is a density estimation metric: it measures how well the model predicts the next byte given the preceding context, averaged over all bytes in the test set. It does not measure the model's ability to generate coherent text, produce useful representations for downstream tasks, answer questions, follow instructions, or any of the capabilities that matter for deployed language models. For images, bpb measures compression quality but not visual quality of generated samples, fidelity to ImageNet class structure, or diversity of outputs. For audio, bpb measures waveform prediction accuracy but not perceived audio quality, intelligibility of generated speech, or musical coherence.

The consequence. A practitioner cannot determine from this paper whether MEGABYTE's byte-level modeling translates into useful downstream capabilities. It is possible that MEGABYTE achieves lower bpb than baselines but produces text with subtle coherence problems that only manifest in long-form generation — for example, the within-patch degradation observed in Figure 5 might cause locally-consistent but globally-inconsistent text, where each patch is internally coherent but patches don't connect smoothly. It is also possible that the architectural decomposition (Global model for structure, Local model for detail) produces representations that are poorly suited for transfer learning or fine-tuning, even though the density estimation performance is strong. The paper's exclusive focus on bpb means we have no information about: (1) generation quality (human evaluation or automated metrics), (2) representation quality (linear probing or fine-tuning on classification tasks), (3) sample diversity and mode coverage, (4) ability to condition on long prompts, or (5) latency and throughput at batch sizes greater than 1.

What evidence exists in the paper. The paper provides only bpb and perplexity numbers. There are no generated text samples, no generated images, and no generated audio samples anywhere in the paper or appendices. The generation speed measurement (Table 6) is the only non-perplexity evaluation, and it reports only wall-clock time without any quality assessment of the generated bytes. The paper validates its PerceiverAR implementation by matching the reported bpb (Appendix C), which is appropriate for a density estimation paper, but means we have no signal about whether MEGABYTE's lower bpb translates to measurable improvements on tasks users care about.

Mitigation status. Not addressed. The paper positions itself as a density estimation and architecture paper, and bpb is the standard metric in that literature — so this is not a failure to meet community standards, but rather a scope limitation that practitioners should be aware of. The paper does not claim applicability to downstream tasks, nor does it suggest that bpb improvements will necessarily translate to better generation or representation quality. The discussion of generation speed (Section 8.1) is purely about computational cost, not output quality. A practitioner interested in using MEGABYTE for text generation, image synthesis, or audio generation would need to conduct their own quality evaluations.

The Transformer Baselines Are Handicapped by Shorter Context Lengths, Confounding Architectural Comparison with Context Window Effects

The assumption or constraint. In the compute-controlled experiments, the baseline Transformers are given context lengths much shorter than MEGABYTE (and PerceiverAR): 1024 bytes for the standard Transformer vs. 8192 bytes for MEGABYTE (Tables 2, 6, and 12). This is because matching the forward-pass time per byte forces the Transformer to either be much smaller (to fit in the same compute budget with a longer context) or to use a shorter context (to keep the model size competitive). The paper chooses to keep the Transformer large (320M parameters) and limit its context to 1024, while MEGABYTE uses a 758M/262M model with 8192-byte context. Similarly, in the image experiments (Table 5), the Transformer uses context 1024 while MEGABYTE uses the full sequence (12,288 to 1,228,800 bytes).

The consequence. MEGABYTE's reported performance advantages over the standard Transformer — lower bpb on all five text datasets (Table 2), lower bpb on all three image resolutions (Table 5), and lower bpb on audio — are a combination of two effects: (1) MEGABYTE's architectural efficiency (amortized feedforward layers, sub-quadratic attention) and (2) MEGABYTE's longer context window (8192 vs. 1024 bytes). The paper cannot separate these effects. On long-document datasets like PG-19 (mean document length 411,404 bytes) and Books (mean 509,526 bytes), having 8× more context is a substantial advantage independent of the architecture, because the model can condition on much more preceding text. The PerceiverAR baseline partially controls for this — it also uses 8192-byte context — but PerceiverAR introduces its own architectural differences (cross-attention to latent bottleneck), so it's not a clean control for context length alone. For the image experiments in Table 5, the Transformer's 1024-byte context is so short relative to the 196,608 bytes of a 256×256 image that the baseline fundamentally cannot capture long-range image structure — the comparison is between a model that can see the whole image (MEGABYTE) and one that sees only a tiny sliver (Transformer), making the architectural comparison nearly meaningless at higher resolutions.

What evidence exists in the paper. Table 2 shows MEGABYTE's advantage over the Transformer: 1.000 vs. 1.057 on PG-19 (5.4% relative improvement), 0.978 vs. 1.064 on Stories (8.1%), 1.007 vs. 1.097 on Books (8.2%), 0.678 vs. 0.816 on arXiv (16.9%), and 0.411 vs. 0.575 on Code (28.5%). The advantage is larger on datasets where long-range dependencies might be more important (arXiv technical publications, code with long-range variable references) and smaller on Stories (where documents are shorter and local coherence may dominate). This pattern is consistent with a context-length effect confounded with the architectural effect, but the paper doesn't analyze it. Table 5 shows the gap widening from 0.10 bpb at 64×64 to 0.56 bpb at 640×640 — which is exactly what you'd expect if the Transformer's 1024-byte context becomes increasingly inadequate as image size grows. However, the PerceiverAR baseline also sees its gap to the Transformer widen (0.07 at 64×64 to 0.50 at 640×640), and PerceiverAR uses longer context (12,000 bytes), so part of the gap is genuinely about processing the full sequence vs. chunking.

Mitigation status. Not addressed. The paper acknowledges the context length difference in the model configuration table (Table 12) but does not discuss it as a confounding factor in the analysis, nor does it provide an ablation where all models use the same context length (which would require scaling model sizes to maintain the compute budget — a Transformer with 8192-byte context would need to be smaller than 320M parameters, making the comparison about architecture + scale rather than just architecture). The PerceiverAR baseline provides a partial control (same 8192-byte context as MEGABYTE for text), and MEGABYTE outperforms PerceiverAR on all datasets except arXiv where PerceiverAR is closer, suggesting MEGABYTE's advantage is not only about context length. But without a context-equalized Transformer baseline, the magnitude of the architectural contribution vs. the context-length contribution cannot be quantified, and practitioners cannot determine how much of MEGABYTE's benefit they would realize in settings where they can already afford long-context Transformers (e.g., by using efficient attention implementations that the paper's baseline does not employ).

7. Implications and Future Directions

How This Work Changes the Landscape

MEGABYTE shifts the conversation around long-sequence Transformer efficiency from an almost exclusive focus on attention mechanisms toward a more productive target: the position-wise feedforward layers that dominate compute in large models. This is not a paradigm shift in the Kuhnian sense — the basic building blocks (decoder-only Transformers, autoregressive modeling, learned embeddings) are unchanged — but it is a diagnostic reframing with significant practical consequences. Prior to this work, the field's default assumption, reflected in the sheer volume of papers on sparse attention, linear attention, and state space models, was that the quadratic cost of self-attention was the obstacle to long-sequence modeling. The paper's empirical claim that self-attention accounts for only 1.4% of FLOPS in the GPT-3 architecture (Section 3.1) — while feedforward layers account for more than 98% — effectively redirects research attention. If this claim holds across model scales and architectures (the paper argues it grows more severe with scale, since feedforward cost scales with 2mT while attention scales with T²d, and m grows faster than d), then the entire subfield of efficient attention mechanisms has been optimizing a component that, even if made perfectly efficient, would yield at most marginal total improvements. The paper's Figure 3 makes this visually explicit: MEGABYTE with P = 8 uses fewer FLOPS than Linear Transformers across a wide range of model sizes, precisely because Linear Transformers solve the wrong problem — they make attention linear but still apply full feedforward layers at every position.

This reframing has a second, subtler consequence: it elevates temporal downsampling as a first-class efficiency strategy for autoregressive models. Downsampling — processing the sequence at a coarser temporal resolution for the bulk of the computation — had been explored in encoder architectures (ViT's patchification, Perceiver's latent bottleneck) but was considered difficult or impossible for decoders because of the autoregressive causality constraint: you cannot pool information across future timesteps without leaking information. MEGABYTE's solution — offset the inputs to the Global and Local models by one patch so that each predicts the next patch or byte, and use autoregressive masking within the Local model — is not particularly complex, but its existence demonstrates that the causality constraint can be satisfied with careful input shifting. This opens the door to a family of architectures that process sequences at multiple temporal resolutions, allocating large models to coarse timescales where structural decisions are made and small models to fine timescales where local detail is filled in. Prior work that raised this possibility (e.g., the Temporal Latent Bottleneck of Didolkar et al., 2022) had not demonstrated competitive results at scale; MEGABYTE provides the first evidence that the approach can work across modalities and at model sizes where efficiency matters.

The paper also reconciles a tension that had been largely unarticulated but was implicit in the literature: the tension between the drive toward tokenization-free modeling (motivated by multimodality, domain transfer, and simplicity) and the practical impossibility of training standard Transformers on raw byte sequences of meaningful length. Before MEGABYTE, a practitioner who wanted to eliminate tokenization faced an unpleasant choice: either accept a byte-level Transformer with severely limited context (and correspondingly poor performance on long-range dependencies) or use a lossy compression scheme (discrete autoencoders for images, clustering for audio) that reintroduced the information bottleneck tokenization was meant to eliminate. MEGABYTE shows that this is a false dichotomy — that an architectural solution can provide the efficiency benefits of tokenization (shorter effective sequence lengths for the expensive model) without the information loss, language-specific heuristics, or domain transfer failures. The paper's PG-19 result (36.4 word perplexity, competitive with subword models in Table 3) is the empirical evidence that makes this reconciliation concrete rather than aspirational.

Research directions that become more attractive after this paper:

Hierarchical autoregressive architectures with more than two levels, where each level operates at a different temporal resolution and capacity allocation is learned rather than fixed. MEGABYTE's two-level design is the simplest case; a three-level architecture (e.g., paragraph-level, sentence-level, byte-level) could further amortize computation for very long sequences.
Learned, adaptive temporal downsampling where patch boundaries are not fixed at regular intervals but are predicted by the model based on content (e.g., patch boundaries at word boundaries for text, at object boundaries for images, at phoneme boundaries for audio). This would address the translation invariance limitation and potentially improve the within-patch loss degradation.
Feedforward-sharing mechanisms beyond temporal downsampling — for example, conditional computation where only a subset of feedforward parameters are active per position, or learned sparsity patterns in the feedforward layers that reduce their effective cost without reducing model dimension.
Tokenization-free multimodal models where text, images, audio, and other modalities are processed through the same byte-level architecture with minimal modality-specific adaptation. MEGABYTE demonstrates this is architecturally feasible; scaling it to production multimodal models would be a natural next step.

Research directions that become less attractive (or at least require stronger justification):

Pure attention-efficiency methods (sparse attention, linear attention, state space models) that do not also address the feedforward bottleneck. If the paper's 98%+ feedforward cost claim holds at scale, these methods are optimizing a diminishing fraction of total compute and cannot deliver order-of-magnitude efficiency improvements on their own. They remain useful as components within hierarchical architectures (e.g., linear attention in the Global model could further reduce cost), but as standalone solutions to long-sequence modeling, their value proposition is weakened.
Lossy tokenization for modalities where lossless modeling is possible. MEGABYTE's ImageNet 64×64 result (3.40 bpb, matching state-of-the-art while using raw bytes) shows that lossless byte-level modeling can be competitive with approaches that use discrete autoencoders (like VQ-VAE) to compress images into tokens. As byte-level architectures improve, the performance penalty for avoiding lossy compression shrinks, making the information-preserving approach more attractive.

Follow-Up Research This Work Enables

Scaling laws for hierarchical autoregressive models across modalities. The paper provides exactly two scale points for language modeling (80B bytes in Table 2, 400B bytes in Table 3) and single points for images and audio — not enough to fit power laws or predict performance at larger scales. A direct follow-up would train MEGABYTE models at multiple sizes (e.g., Global model sizes from 100M to 10B parameters, matched Local models) on multiple data scales (e.g., 10B to 1T bytes) on PG-19 or a similar long-document dataset, and compare the resulting scaling exponents to those of equivalently-sized standard Transformers and PerceiverAR. The key question is whether MEGABYTE's efficiency advantage grows, shrinks, or remains constant with scale. If the feedforward bottleneck argument is correct, the advantage should grow — because feedforward layers become a larger fraction of total FLOPS in larger models — producing a steeper scaling curve for MEGABYTE than for standard Transformers at equal compute. If instead MEGABYTE's advantage saturates or reverses, that would suggest the Global-Local decomposition introduces a bottleneck (perhaps the fixed per-patch Global context) that limits scaling, pointing toward the need for per-byte Global conditioning or deeper hierarchies. The experiment would require careful FLOPs-matched comparisons at each scale point, not just parameter matching, to isolate architectural effects from the benefits of additional compute.

Adaptive patch boundary prediction for text, using linguistic supervision or learned boundaries. The fixed patch boundaries in MEGABYTE create translation invariance problems (the same word at different offsets gets different representations) and may contribute to the within-patch loss degradation (Figure 5). A natural extension would replace fixed P-byte patches with variable-length patches whose boundaries are predicted by a lightweight boundary detector — for text, this could be a small model trained to predict word boundaries from byte sequences; for images, boundaries could align with edge detection outputs; for audio, boundaries could align with detected phoneme or note onsets. The Global model would then operate on variable-length patches (requiring padding or sequence packing to maintain batch efficiency), and the Local model's sequence length within each patch would vary. The research question is whether content-aware boundaries improve bpb beyond what fixed patches achieve (Table 9 shows robustness to patch size, but variable boundaries might capture a different benefit — alignment with meaningful units rather than just a different fixed size), and whether the improvement justifies the additional complexity and potential throughput reduction from variable-length sequences. A strong experiment would compare fixed-patch MEGABYTE against boundary-aware MEGABYTE on PG-19 (where word boundaries are well-defined) and on a multilingual dataset (where word boundary concepts vary across languages), measuring both bpb and the within-patch loss curve to see whether the degradation at later patch positions is reduced.

Per-byte Global conditioning to eliminate within-patch degradation. The paper identifies a clear architectural weakness: the Local model's predictions degrade toward the end of each patch because the Global context is a single fixed vector for the entire patch (Section 2.3.3, Figure 5). Strided inference is a workaround that doubles inference cost (Table 8). An architectural fix would have the Global model output a sequence of conditioning vectors for each patch — one per byte position — rather than a single patch representation that is reshaped. This could be achieved by having the Patch Embedder produce per-byte representations that are processed by the Global model with a stride (e.g., the Global model attends to every P-th byte but outputs at every byte position) or by having the Global model output a patch representation that is then processed by a small "unfolding" network that expands it to P per-byte conditioning vectors. The experiment would compare this per-byte Global conditioning variant against the base MEGABYTE architecture under equal compute (the per-byte variant would require a smaller Global model or fewer layers to fit the same budget, since it does more computation per patch). The key measurement would be the within-patch loss curve: if per-byte conditioning eliminates the degradation, the loss should be flat across byte positions within a patch. A secondary measurement would be whether bpb improves overall — even if the degradation is eliminated, the smaller Global model might produce worse patch-level representations, creating a tradeoff.

MEGABYTE as a tokenizer replacement in pretrained language model pipelines, evaluated on downstream tasks. The paper demonstrates competitive density estimation with subword models (Table 3, 36.4 vs. 26.5–36.3 word perplexity on PG-19), but never evaluates whether the byte-level representations are useful for downstream tasks. A direct follow-up would take a MEGABYTE model pretrained on a large text corpus (e.g., the Pile, or a combination of the datasets in Table 1), extract representations from the Global model's output (either the patch-level representations or the per-byte projections), and evaluate them on standard NLP benchmarks (GLUE/SuperGLUE, question answering, summarization) via fine-tuning or linear probing. The comparison would be against a comparably-sized subword Transformer (e.g., a standard GPT-2 or OPT architecture trained on the same data with BPE tokenization). The research question is whether the Global model's patch representations, learned without any explicit linguistic tokenization, capture semantic and syntactic information at a level comparable to subword-level representations. If they do, it strengthens the case for tokenization-free modeling beyond density estimation. If they don't — if the Global model's representations are too coarse-grained or task-specific to transfer well — that would reveal a limitation of the patch-level bottleneck and suggest that intermediate representations (between patch-level and byte-level) might be needed for transfer learning.

Three-level MEGABYTE for very long sequences (10M+ bytes). The paper demonstrates sequences up to 1.2M bytes (ImageNet 640×640). For longer sequences — full-length books (500K–1M+ words, ~5M+ bytes), high-resolution video (billions of bytes), or genomic sequences (chromosomes are 100M+ base pairs) — the two-level decomposition may be insufficient because the Global model's sequence length (K = T/P) becomes large enough that its quadratic attention cost becomes the bottleneck. A three-level hierarchy (e.g., super-patches of 64 patches, patches of 8 bytes) would give attention costs of O((T/(P·S))²) for the top level, O(T · S / P²) for the middle level, and O(T · P) for the bottom level, where S is the super-patch size. The optimal scaling of P and S with T would determine whether this architecture achieves O(T log T) or O(T) effective complexity. The experiment would train a three-level MEGABYTE on a dataset requiring very long context (e.g., full PG-19 books concatenated to form sequences of 5M+ bytes, or long video frame sequences represented as raw bytes) and compare against a two-level MEGABYTE with the same total compute, measuring both bpb and whether the additional level enables effective use of context beyond what two levels can capture. The diagnostic would be an analog of Figure 4 showing that probability continues to improve with context beyond the two-level model's saturation point.

Negative result: does MEGABYTE match subword models in a fully controlled comparison? The paper's Table 3 comparison with subword models is confounded by differing compute budgets, data scales, and pretraining protocols. A crucial stress-test would train a subword Transformer (BPE, vocabulary 32k) and a MEGABYTE model under exactly equal compute and data constraints on the same training corpus — for example, both trained from scratch on 400B bytes of PG-19 with matched forward-pass time per byte, both using the same optimizer, same training duration, and same evaluation protocol. The experiment would measure word-level perplexity (converting MEGABYTE's byte-level predictions to word-level via the standard product-of-byte-probabilities approach). If MEGABYTE matches or beats the subword Transformer under equal conditions, it provides the clean evidence the paper currently lacks and substantially strengthens the case for tokenization-free modeling. If MEGABYTE underperforms — if the subword model achieves, say, 30 perplexity to MEGABYTE's 36 — that would suggest that tokenization provides a genuine representational advantage beyond mere sequence compression, and that MEGABYTE's competitiveness in Table 3 was an artifact of the confounded comparisons. Either outcome is informative for the field.

Practical Applications and Downstream Use Cases

Cost-efficient training of byte-level models for multilingual or code-switching text. For organizations building language models that must handle multiple languages, code-switching (mixing languages in the same document), or technical domains with specialized vocabulary, tokenization is a persistent source of friction: BPE vocabularies optimized for one language perform poorly on others, out-of-vocabulary tokens degrade performance on domain-specific terms, and retraining the tokenizer for each new language or domain adds engineering complexity. MEGABYTE's byte-level modeling eliminates all of these problems — the vocabulary is always exactly 256 bytes, regardless of language or domain — while the paper's results (Table 2: 0.678 bpb on arXiv, 0.411 on Code) suggest the architecture handles technical text well. A deployment scenario would be a multilingual customer support system where queries and responses may contain multiple languages, code snippets, error messages, and domain-specific jargon. A MEGABYTE model trained on a multilingual corpus would process all of these through the same byte-level interface without tokenization-related failures, and the paper's demonstrated generation speed advantage (40% faster than a 350M Transformer for a 1.5B-parameter model, Table 6) could reduce latency for real-time applications.

High-resolution image generation with full-context byte-level autoregressive models. The paper demonstrates MEGABYTE processing 640×640 images (1.2M bytes) as a single differentiable forward pass (Table 5), achieving better density estimation than PerceiverAR and Transformer baselines that must split images into segments. For image generation applications where global coherence matters — generating images with consistent lighting, realistic object placement, and long-range structural dependencies — MEGABYTE's ability to condition the prediction of every pixel on the entire preceding image context (not just a local window) could produce more globally consistent outputs than approaches that generate images in segments or rely on compressed latent representations. A practical deployment would use a MEGABYTE model pretrained on ImageNet at high resolution as the backbone for class-conditional or text-to-image generation, replacing the discrete autoencoders (VQ-VAE, VQ-GAN) that current approaches use to compress images into token sequences. The benefit would be eliminating the information loss from the autoencoder compression step — the model would have direct access to raw pixel bytes throughout generation, potentially enabling finer detail and more accurate color rendering. The paper's ImageNet results provide the density estimation foundation for this, but substantial additional work on generation quality (sampling strategies, classifier-free guidance adaptation, conditioning mechanisms) would be needed.

Raw audio modeling without specialized preprocessing or lossy compression. For speech and music generation, current autoregressive approaches typically require compressing raw audio (16-bit samples at 16–48 kHz, producing 32k–96k bytes per second) into discrete tokens using clustering (HuBERT tokens) or VQ-VAE compression, losing fine spectral and temporal detail. MEGABYTE's audio result (3.477 bpb on the 2TB audio dataset, processing 524,288-byte sequences with patch size 32, Table 7) demonstrates that the architecture can model raw audio bytes directly at scale. A practical application would be a MEGABYTE-based neural vocoder or music generation system that operates on raw 16-bit audio without any preprocessing beyond reading bytes from the audio file — the paper explicitly notes this as an advantage: "we simplify the audio modeling process by directly reading the bytes (256 possible values) from the audio file and conducting an autoregressive language model on top of that" (Section 7). The benefit is streamlined training and inference pipelines (no separate autoencoder to train and maintain, no mismatch between the autoencoder's compression artifacts and the generative model's distribution), and potentially higher-fidelity output since no information is discarded in preprocessing. The generation speed advantage (the Global model runs once per 32 bytes, so once per ~1ms of audio at 16 kHz mono) would be particularly valuable for real-time audio generation applications where latency is critical.

When to Prefer This Method

The paper does not articulate an explicit decision rule or tradeoff matrix against named alternatives in different operating regimes. Its comparisons are primarily against standard byte-level Transformers and PerceiverAR under a fixed methodology (compute-controlled experiments), and while it demonstrates advantages across all tested settings, it does not characterize when a practitioner should choose MEGABYTE over alternative approaches like continued use of subword tokenization, sparse attention Transformers, linear attention models, or state space models. The conditions under which MEGABYTE is preferable can be inferred from the paper's results, but a forced "Prefer A when X, Prefer B when Y" decision matrix would impose structure the paper itself does not provide. The most honest reading is that the paper establishes MEGABYTE as a strong default choice for byte-level autoregressive modeling across modalities at the scales tested, but leaves the broader tradeoff landscape for future work to characterize.