An Empirical Study of Autoregressive Pre-training from Videos

ArXiv: 2501.05453

🎯 Pitch

A 1B-parameter transformer predicting next visual tokens in videos reaches 75.3% on ImageNet and 74.4% on Kinetics-400 with no vision-specific inductive biases—the first competitive action recognition from pure autoregressive generative modeling. Surprisingly, scaling these video models follows power-law behavior like language models, but with a slower exponent (−0.0378 vs. −0.048 for GPT-3), suggesting video redundancy intrinsically bounds the returns.

---

1. Executive Summary

This paper empirically studies autoregressive pre-training from videos by constructing a family of models called Toto that treat videos as sequences of visual tokens and train causal transformer models to predict future tokens. The authors explore different architectural, training, and inference design choices—tokenizers (dVAE vs. VQGAN vs. continuous patches), architectures (LLaMA vs. GPT2 vs. Mamba), and probing strategies (attention pooling vs. average pooling)—on a diverse dataset of videos and images comprising over 1 trillion visual tokens, evaluated across image recognition, video classification, object tracking, action forecasting, robotics, and object permanence tasks. Despite minimal inductive biases, autoregressive pre-training achieves competitive performance across all benchmarks, with the 1B-parameter model reaching 75.3% top-1 accuracy on ImageNet and 74.4% on Kinetics-400—the first demonstration of competitive action recognition with autoregressive generative modeling. The paper further shows that scaling visual next-token prediction models follows power-law behavior with compute, albeit at a slower rate than language models (with a scaling exponent of −0.0378 for Toto versus −0.048 for GPT-3), establishing that autoregressive pre-training from videos is a viable path to general visual representations even though video redundancy and tokenizer quality bound the gains achievable on tasks fundamentally outside the model's generative competence.

2. Context and Motivation

The Core Problem: We Don't Know How Well Autoregressive Pre-training Transfers to Vision

The fundamental question this paper addresses is deceptively simple: if you train an autoregressive transformer to predict the next visual token in videos, do the learned representations actually work well across a diverse range of downstream tasks? This matters because, unlike in natural language processing—where GPT-style autoregressive pre-training has become the dominant paradigm and its downstream benefits are extensively documented—the equivalent approach for visual data has remained surprisingly underexplored.

The paper opens with a historical framing that makes this gap striking. Claude Shannon proposed next-word prediction as a way to estimate the entropy of English in 1951. Nearly 70 years later, training large transformers on that exact task became the backbone of modern language AI. What is less well known—and what the paper brings to the foreground—is that Fred Attneave proposed an analogous task for images in 1954: scanning a picture pixel by pixel and having a subject guess the color of each successive element. The intellectual lineage is direct: sequential next-element prediction as a way to model visual structure was proposed essentially simultaneously with its linguistic counterpart. Yet, as the paper notes in Section 1, "despite the successes of autoregressive language and image models, their effectiveness for video modeling remains underexplored."

This gap is significant for several interconnected reasons:

• Data scale asymmetry opens an opportunity window. The paper argues that "most of the text available on the Internet has already been used by the language models," while for video "we just started on the journey of Big Data exploitation." Since autoregressive pre-training historically benefits enormously from scale (as demonstrated by GPT-3 and subsequent language models), there is a compelling hypothesis that applying the same approach to the vastly larger and largely untapped resource of internet video could yield substantial gains. But this hypothesis had not been systematically tested.

• Unified treatment of images and videos becomes natural under the token-sequence paradigm. If both images and videos are converted to sequences of discrete tokens, a single model architecture and training objective can consume both modalities simultaneously—an appealing path toward general visual understanding models. The paper explicitly frames this as a key advantage: treating "videos as sequences of tokens enables us to jointly train on videos and images using a unified format."

• Minimal inductive bias is a scientific and practical goal. Approaches like masked autoencoding (MAE) or joint-embedding methods (SimCLR, DINO) build in explicit inductive biases about what makes good representations—for instance, that representations should be invariant to augmentations, or that the task of reconstructing masked patches forces the model to learn semantic structure. Autoregressive pre-training makes no such assumptions. It simply models the data distribution. Understanding whether this minimal-assumption approach can compete with heavily engineered alternatives speaks to fundamental questions about what drives representation quality in visual models and points toward simpler, more scalable recipes if successful.

• Robotics and embodied AI need strong, general visual backbones. The paper evaluates on robot manipulation tasks precisely because embodied AI is a domain where labeled data is scarce, task-specific fine-tuning is expensive, and strong pre-trained representations that transfer broadly are essential. Showing that autoregressive video pre-training produces features useful for motor control—without any robotics-specific design—would expand the applicability of this approach beyond standard vision benchmarks.

Where Existing Approaches Fall Short

The paper surveys several families of prior work and identifies specific limitations or gaps in each.

Masked autoencoding has been the dominant self-supervised approach for images and videos, but it leaves the autoregressive alternative largely uncharacterized. The success of MAE (He et al., 2022) and its video extensions (VideoMAE, ST-MAE; Feichtenhofer et al., 2022; Wang et al., 2023a) showed that masking a large fraction of input patches and predicting the masked content yields strong representations. These methods are conceptually straightforward and computationally efficient (since only the visible patches pass through the encoder). However, they use a specific structural assumption—that reconstructing hidden patches from visible context is the right pre-training task—and they use asymmetric encoder-decoder architectures. The paper notes in Section 2 that while these masked modeling approaches showed strong results, the complementary autoregressive approach to the same data (predicting tokens sequentially rather than reconstructing masked ones) remained largely uninvestigated at scale for video. The reader should understand that masked modeling and autoregressive modeling are two different answers to the same question: "what is the right self-supervised objective for vision?" The dominance of masked approaches meant the field had little empirical evidence about where autoregressive methods stand.

Autoregressive image modeling existed but was limited in scale, scope, and transfer evaluation. The paper acknowledges a lineage of autoregressive vision work: PixelCNN and PixelRNN (Van den Oord et al., 2016) generated pixels sequentially using convolutional and recurrent architectures. ImageTransformer (Parmar et al., 2018) applied causal local attention to pixel generation. Critically, iGPT (Chen et al., 2020a) demonstrated that generative pre-training at the pixel level could learn useful representations for downstream recognition—showing that the GPT recipe could transfer to images. Henighan et al. (2020) studied scaling laws for autoregressive image and video models. However, these prior works had important limitations that the Toto paper aims to address or extend:

• iGPT operated at the pixel level (after clustering pixel patterns into a discrete vocabulary), which is computationally expensive. Toto uses learned tokenizers like dVAE and VQGAN that compress images into compact discrete codes, enabling much longer effective context lengths and higher resolution.

• Prior autoregressive video work was small-scale or generation-focused. Weissenborn et al. (2019) studied autoregressive video generation for "promotable video generations" (a term used in the paper to refer to promptable generation), but focused on generation quality, not representation learning. The field had no systematic study of how autoregressive video pre-training transfers across a broad suite of recognition, tracking, forecasting, and robotics tasks.

• Data scale and diversity were limited in prior studies. The paper constructs a dataset of over 100,000 hours of video spanning exocentric (Kinetics, HowTo100m) and egocentric (Ego4D) sources plus images (ImageNet), totaling over 1 trillion visual tokens. This represents a deliberately diverse, internet-scale training set that prior autoregressive vision studies did not use.

Joint-embedding and discriminative methods dominate current benchmarks, but their design choices may obscure the generic utility of simpler approaches. Methods like DINO (Caron et al., 2021), DINOv2 (Oquab et al., 2023), and contrastive approaches (SimCLR, BYOL) achieve strong ImageNet and downstream performance by training models to produce invariant representations under augmentation or to match representations across views. The paper's experimental results in Tables 7 and 8 confirm that these discriminative approaches generally outperform generative ones on recognition benchmarks—for instance, DINO ViT-B/8 reaches 80.1% on ImageNet versus Toto-large's 71.1%, and DINOv2 ViT-g/14 reaches 84.4% on Kinetics-400 versus Toto-1b's 74.4%. However, the paper's implicit argument is that this performance gap should not be interpreted as a fundamental limitation of autoregressive modeling, but rather as a reflection of:

1. The specific design choices (tokenizer, resolution, architecture) that have been optimized for discriminative methods but remain underexplored for autoregressive ones.
2. The fact that discriminative methods are explicitly trained to separate instances—a task aligned with classification benchmarks—while autoregressive methods optimize a more generic objective that may transfer to a broader set of behaviors, including generative tasks and domains like robotics where instance-level separation is less directly relevant.
3. That scaling behavior matters: if autoregressive models scale predictably (which the paper investigates in Section 4.9), they may close gaps or surpass discriminative approaches at larger scales or on different task distributions.

Supervision-based autoregressive vision models (AIM) used filtered data, conflating data quality effects with architectural decisions. The paper specifically contrasts its approach with AIM (El-Nouby et al., 2024), which also trains autoregressive image models but uses "Data Filtering Networks" with CLIP-filtered data. The Toto authors emphasize: "Compared to these works, we do not use any supervision during our pre-training and utilize image and videos jointly." This distinction matters because CLIP filtering introduces a form of weak supervision (text-image alignment) that may inflate the apparent effectiveness of the autoregressive objective. Toto's philosophy is to test the "minimal inductive biases" hypothesis cleanly: take raw internet data, tokenize it, predict the next token, and see what happens.

A unified framework for comparing design choices across the full autoregressive vision pipeline was missing. The paper identifies a secondary gap: even if one believes autoregressive pre-training is promising, there is no established recipe for how to do it well in vision. Which tokenizer works best? Does architecture choice (LLaMA vs. GPT2 vs. Mamba) matter? What is the best way to extract representations from a decoder-only model for downstream tasks? How should resolution be handled? The paper frames Sections 3 and 4 as an empirical study that systematically answers these questions, providing a reference point for future work.

How This Paper Positions Itself

The paper positions itself not as proposing a fundamentally new method, but as conducting the first large-scale empirical study of autoregressive pre-training from videos that simultaneously spans: (a) diverse data curation at internet scale, (b) systematic ablation of design choices, (c) evaluation across a broad suite of downstream tasks (image classification, video classification, action forecasting, semi-supervised tracking, object permanence, robotic manipulation), and (d) study of compute-optimal scaling behavior.

The framing is explicitly empirical and exploratory. The title—"An Empirical Study of Autoregressive Pre-training from Videos"—signals this intent. The paper does not claim to beat state-of-the-art discriminative methods; instead, it claims to demonstrate that autoregressive pre-training is competitive, scales predictably, and transfers broadly despite minimal inductive bias. This is a "baseline establishment" paper: it asks "what happens if you just take the LLM recipe and apply it to video tokens at scale?" and provides detailed answers across multiple dimensions.

The historical framing with Shannon and Attneave serves a specific positioning function beyond literary elegance. It suggests that autoregressive pre-training for vision is not a recent hack but rather a natural, historically anticipated approach that simply lacked the compute and data to be realized properly until now. This positions the paper as closing a 70-year loop: Attneave proposed the idea in 1954, and the paper's contribution is to finally test it at a scale that would have been unimaginable then, with modern architectures, across the "Big Visual Data" of 2024.

The paper also positions itself relative to the language model scaling literature. By studying compute-optimal scaling behavior in Section 4.9 and comparing the scaling exponent to that of language models (showing a power law of $L(C) = 7.32 \cdot C^{-0.0378}$ for Toto versus $2.57 \cdot C^{-0.048}$ for GPT-3), it situates autoregressive vision models within the broader conversation about scaling laws. The finding that vision scales with a shallower exponent suggests that while more compute helps predictably, the returns diminish faster than in language—an important empirical reality check for the field and a clear motivation for future work on more efficient video pre-training objectives.

Finally, the paper acknowledges a tension that serves as motivation for future research: video frames contain substantial redundancy that makes next-frame prediction "much easier" than next-text-token prediction, potentially limiting the quality of learned representations. This observation—supported by Figure 10 showing that validation loss drops sharply after the first frame—is not just a limitation but a research question: can we design better video pre-training objectives that retain the simplicity of autoregressive modeling while addressing this redundancy? The paper positions its empirical results as a foundation for answering this question rather than as the final word.

3. Technical Approach

3.1 Reader Orientation

The system described in this paper is a family of autoregressive transformer models (called Toto) that are pre-trained on sequences of visual tokens extracted from images and videos, using a next-token prediction objective, exactly analogous to how GPT-style language models are trained on text. The problem it addresses is: can a model trained with minimal inductive bias—simply predicting the next visual token in a sequence—learn representations that transfer effectively across a wide range of vision tasks (classification, tracking, robotics, forecasting)? The "shape" of the solution is an empirical pipeline: convert diverse visual data into discrete token sequences, train causal transformers at scale to predict future tokens, then evaluate the intermediate representations from those transformers on downstream tasks using lightweight probing or fine-tuning.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major stages, shown in Figure 1:

1. Data Collection — A diverse mixture of image and video datasets (ImageNet, Kinetics-600, Ego4D, HowTo100m) is assembled, comprising over 100,000 hours of video and approximately 2.5 trillion total visual tokens. Each mini-batch samples from datasets with fixed ratios: roughly 20% ImageNet, 10% Ego4D, 10% Kinetics, and 60% HowTo100m.

2. Frame Sampling and Tokenization — Video frames are extracted at a stride of 4 frames, resized such that the shortest side is 128 pixels, and randomly cropped to 128×128. Each frame is independently tokenized using a dVAE (Ramesh et al., 2021) with a vocabulary of 8,192 discrete tokens, producing 256 tokens per frame (a 16×16 grid). For videos, 16 frames are sampled, yielding 4,096 tokens per sequence. For images, 16 random images are concatenated to form an equivalent 4,096-token sequence. Special start tokens distinguish videos ([1]) from images ([3]), and all sequences end with a stop token ([2]).

3. Causal Transformer Pre-training — A decoder-only transformer with causal (unidirectional) attention processes the token sequence autoregressively. At each position, the model predicts the next token given all previous tokens. The objective is standard next-token prediction with cross-entropy loss. Models are trained at three scales (base: 120M parameters, large: 280M, 1B: 1.1B) on over 1 trillion visual tokens, using the LLaMA architecture with RMSNorm, SwiGLU activations, and Rotary Position Embeddings (RoPE).

4. Representation Extraction — After pre-training, intermediate representations are extracted from specific layers of the transformer. Because the model is decoder-only with causal attention, later tokens attend to more tokens than earlier tokens (a "skewed receptive field"). To handle this, the paper uses attention pooling: a learnable query vector cross-attends to all token representations at a chosen layer, producing a single fixed-dimensional representation for downstream tasks. This is contrasted with standard average pooling.

5. Downstream Transfer — The extracted representations are used for various tasks via either linear probing (training a linear classifier on frozen features), attention probing (learning an attention pooling head on frozen features, optionally with MLP layers for more capacity), or fine-tuning (continuing to train the model with both the pre-training objective and task-specific losses). Tasks span image classification (ImageNet-1K), video action recognition (Kinetics-400), action anticipation (Ego4D), semi-supervised video object tracking (DAVIS), object permanence (CATER), and robotic manipulation (simulation and real-world).

3.3 Roadmap for the Deep Dive

• First, the pre-training objective and data pipeline—how videos and images become token sequences, and what loss function drives learning. This establishes the core training mechanism that everything else depends on.
• Second, the model architecture (LLaMA-style transformer) and training hyperparameters, since the architecture choices (causal attention, RoPE, RMSNorm) affect both what the model can learn and how representations are distributed across layers.
• Third, the tokenization design space—why dVAE over VQGAN, why 8K vocabulary over other sizes, how resolution interacts with token count, and why 1-gram coverage matters. This is critical because tokenizer choice constrains everything downstream.
• Fourth, the representation extraction strategy—attention pooling vs. average pooling, which layers to probe, and why decoder-only models show peak performance at middle layers rather than the final layer.
• Fifth, the downstream evaluation protocols, since each task (classification, tracking, robotics) uses different probing or fine-tuning methods and different resolutions.
• Sixth, the compute-optimal scaling methodology using µP-parameterization, establishing how the paper measures and extrapolates the relationship between model size, compute, and validation loss.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical study paper whose core idea is that autoregressive next-token prediction applied to video tokens at scale produces representations that transfer competitively across diverse vision tasks, despite minimal task-specific inductive bias, and that these representations scale predictably with compute.

---

Pre-training: The Next-Token Prediction Objective on Visual Sequences

The pre-training objective is a direct translation of the language modeling paradigm into the visual domain. Given a collection of images and videos, every frame is independently tokenized into discrete visual tokens, and the tokens are arranged into a one-dimensional sequence using raster scan ordering: pixels are traversed left-to-right, top-to-bottom within each frame, and frames are concatenated in temporal order. This produces a dataset of token sequences where each sequence from a single video or set of images is denoted as $\{x_1^j, x_2^j, x_3^j, \ldots, x_n^j\}$, with $j$ indexing the sequence and $n$ being the total number of tokens (4,096 for all models in this paper).

The model is trained to maximize the likelihood of the correct next token at every position, which is equivalent to minimizing the negative log-likelihood summed across all positions and sequences:

$p(x^j) = \prod_{i=1}^{n} p(x_i^j \mid x_{i-1}^j, x_{i-2}^j, \ldots, x_1^j, \theta)$

where $p(x_i^j \mid x_{i-1}^j, \ldots, x_1^j, \theta)$ is the probability assigned to token $x_i^j$ by the model with parameters $\theta$, conditioned on all preceding tokens in the sequence. The full training loss is the expected negative log-likelihood over the data distribution:

$\mathcal{L}_{\text{pre-train}} = \mathbb{E}_{x^j \sim \mathcal{X}} \left[-\log p(x^j)\right]$

where $\mathcal{X}$ is the training dataset of token sequences, $x^j$ is a single sequence, and $p(x^j)$ is the product over all positions as defined above.

What it computes: the model processes the sequence token by token from left to right. At position $i$, it uses the causal transformer to produce a probability distribution over its vocabulary of 8,192 tokens, conditioned on tokens $1$ through $i-1$. The cross-entropy between this predicted distribution and the ground-truth token at position $i$ contributes one term to the loss. Summing over all tokens in all sequences gives the total pre-training objective. The model learns to predict what visual content comes next—within a frame, it learns spatial structure (predicting the bottom half of an object from its top half in raster order); across frames, it learns temporal structure (predicting what a scene will look like in the next frame given the current and past frames).

Why this form: the negative log-likelihood is the maximum-likelihood objective for modeling a categorical distribution, which is the natural choice since each target is one of 8,192 discrete tokens. It is identical to the standard next-token prediction loss used in GPT, LLaMA, and all major language models. The paper deliberately avoids alternative objectives like mask prediction (MAE), contrastive losses (SimCLR, DINO), or generative adversarial training, because the entire point of the study is to isolate the behavior of autoregressive modeling without task-specific design. Using product-of-conditionals factorization also enables a single model to handle both images and videos by treating everything as a flat token sequence, avoiding any architecture changes between modalities.

Key implementation detail: the context length is fixed at 4,096 tokens across all experiments. For videos, this corresponds to 16 frames of 16×16 tokens each (16 × 256 = 4,096). For images, 16 randomly sampled ImageNet images are concatenated to fill the same context length, with each image contributing its own 16×16 = 256 tokens. The special start tokens ([1] for video sequences, [3] for image sequences) allow the model to distinguish modalities in the first position of each sub-sequence, and the end token ([2]) marks sequence boundaries. This unified token format is what makes joint training on images and videos possible without needing separate processing pipelines.

Figure 2 shows the training loss curves for the base (120M), large (280M), and 1B models trained with dVAE tokens. All three models show smooth convergence over the course of training on 1 trillion tokens, with the larger models achieving lower final loss, which is the expected behavior for scaling model size under a fixed compute budget.

---

Model Architecture: LLaMA-Style Causal Transformer

The architecture is a decoder-only transformer with causal (autoregressive) attention, following the LLaMA design (Touvron et al., 2023). The model consists of $L$ identical layers, each containing a multi-head self-attention block and a feedforward network, both with pre-normalization using RMSNorm.

For a transformer with $L$ layers, let $H_l$ be the hidden state after layer $l$ (with $0 \leq l \leq L$, where $H_0$ is the token embedding). The computation for layer $l+1$ proceeds in two steps:

$\hat{H}_{l+1} = H_l + \text{MHSA}(\text{RMS-norm}(H_l))$

$H_{l+1} = \hat{H}_{l+1} + \text{MLP}(\text{RMS-norm}(\hat{H}_{l+1}))$

where MHSA is multi-head self-attention with causal masking (a token at position $i$ can only attend to positions $1$ through $i$), and MLP is a multi-layer perceptron with SwiGLU activations (Shazeer, 2020).

What it computes: the first equation adds the output of the self-attention block to the input (a residual connection), where the self-attention uses RMSNorm-normalized inputs. The self-attention computes weighted combinations of all previous token representations, with weights determined by learned query-key similarity. The second equation adds the output of the MLP block to the intermediate result, where the MLP uses RMSNorm-normalized inputs and SwiGLU activations. Each layer progressively transforms the token representations, building increasingly abstract features from the initial token embeddings.

Why this form: every architectural component is chosen for proven effectiveness in language modeling:

• RMSNorm (Zhang & Sennrich, 2019) is a simplification of LayerNorm that removes the mean-centering step, computing only the root-mean-square normalization. It is computationally cheaper and empirically performs equivalently or better than LayerNorm in large-scale training.
• SwiGLU activation (Shazeer, 2020) is a gated linear unit with Swish activation, which has been shown to outperform standard ReLU or GELU in transformer feedforward layers across multiple scales and training regimes.
• Rotary Position Embeddings (RoPE) (Su et al., 2024) encode positional information directly into the attention computation by rotating the query and key vectors based on their relative positions. This allows the model to generalize to sequence lengths not seen during training—which is crucial for the resolution fine-tuning strategy described later—and captures relative position information more naturally than absolute position embeddings.
• Causal attention means each token can only attend to itself and previous tokens, enforcing the autoregressive property that the model predicts the future given the past. This is the defining characteristic of GPT-style models and is what enables using the pre-trained model for both representation extraction (by reading intermediate activations) and generation (by sampling tokens sequentially).

The paper also experiments with two alternative architectures, reported in Table 6:

• GPT2 architecture (Radford et al., 2019), which uses absolute sine-cosine positional embeddings instead of RoPE. On ImageNet linear probing at 280M parameters, GPT2 achieves 48.5% top-1 accuracy, compared to 53.2% for the LLaMA architecture at the same size.
• Mamba (Gu & Dao, 2023), a state-space model that replaces attention with a selective scan mechanism, achieving 40.7% top-1 accuracy with 290M parameters.

The LLaMA architecture's superior performance is attributed to the combination of RoPE (enabling resolution generalization) and the RMSNorm/SwiGLU improvements. The Mamba result suggests that purely state-space approaches may not yet compete with attention-based models for visual autoregressive pre-training, at least at this scale.

---

Model Scales and Training Hyperparameters

The paper trains models at three sizes, detailed in Table 1:

All models are trained with a batch size of 1 million tokens (approximately 244 sequences of 4,096 tokens each). The optimizer is AdamW (Loshchilov & Hutter, 2017) with $\beta_1 = 0.9$, $\beta_2 = 0.95$, a maximum learning rate of $3 \times 10^{-4}$, and a cosine learning rate schedule after 2,000 warm-up steps. These hyperparameters follow the standard LLaMA training recipe exactly, making the training setup as close to the language model paradigm as possible.

Why these hyperparameters: the batch size of 1M tokens is large by vision standards but standard for language model pre-training. It provides stable gradient estimates for the next-token prediction objective across diverse video and image content. The learning rate of $3 \times 10^{-4}$ with cosine decay after warm-up is the default LLaMA setting and was not extensively tuned—the paper deliberately uses off-the-shelf settings to test the hypothesis that the language modeling recipe transfers directly without vision-specific optimization.

---

Tokenization: dVAE, VQGAN, and Continuous Patch Embeddings

Tokenization is the critical bridge between raw pixels and the transformer's discrete-token interface. The paper explores three families of tokenizers in Table 3, with the primary experiments using dVAE.

dVAE tokenizer (primary). The dVAE (Ramesh et al., 2021) is a discrete variational autoencoder originally developed for DALL-E. It consists of an encoder that maps a 256×256 image to a 32×32 grid of discrete codes, and a decoder that reconstructs the original image from these codes. The codebook contains 8,192 entries (vocabulary size of 8K). For Toto pre-training, the encoder processes each frame independently, producing 32×32 = 1,024 tokens per frame at full resolution. However, to fit 16 frames into the 4,096-token context window, the input resolution is reduced: frames are resized so that the shortest side is 128 pixels, then randomly cropped to 128×128. At this resolution, the dVAE produces a 16×16 grid = 256 tokens per frame. Sixteen frames × 256 tokens = 4,096 tokens, exactly filling the context window.

VQGAN tokenizer (comparison). VQGAN (Esser et al., 2020) also uses a discrete codebook but improved reconstruction quality by incorporating a perceptual loss (Johnson et al., 2016; Larsen et al., 2016) and an adversarial loss during tokenizer training. The perceptual loss uses a pre-trained VGG network (Simonyan & Zisserman, 2014) to compare features of the original and reconstructed images, which the paper notes "indirectly ingest[s] ImageNet label information via VGG-net." This makes VQGAN a contaminated tokenizer from the perspective of fair representation learning evaluation: the tokenizer itself has seen ImageNet supervision, which could inflate downstream ImageNet classification results. The paper uses VQGAN with vocabulary sizes of 1K and 16K in its ablation studies.

Continuous patch-normalized tokens (comparison). Following MAE (He et al., 2022), the paper also tests continuous embeddings where each image patch is projected through a learned linear layer to produce a continuous vector, and the training objective becomes regression to the normalized pixel values of that patch rather than classification over a discrete vocabulary.

The tokenizer comparison results are shown in Table 3 (ImageNet linear probing accuracy at the 9th layer of Toto-large, trained for 400 epochs on ImageNet-1K):

Key findings from the tokenizer comparison:

• At the same resolution (16×16 tokens), dVAE (53.2%) substantially underperforms VQGAN (61.1%) and continuous patches (60.6%). This is because dVAE at 16×16 tokens corresponds to 128×128 pixel resolution (each token covers an 8×8 pixel region), while VQGAN and patches at 16×16 tokens correspond to 256×256 pixel resolution (each token covers a 16×16 pixel region).
• At equivalent pixel resolution (dVAE 32×32 tokens vs. VQGAN/patch 16×16 tokens, both covering 256×256 pixels), all methods perform similarly: dVAE 61.2%, VQGAN 61.1%, patches 60.6%. The paper concludes that "various ways of tokenization have little effect on ImageNet linear probing accuracy" when resolution is controlled.
• The Patch-dVAE configuration (continuous patch embeddings as input, discrete dVAE tokens as prediction targets) performs worse (58.5%), suggesting that the input and target representations should be consistent—mixing continuous and discrete creates an interface mismatch.
• The paper explicitly prefers dVAE as the default tokenizer despite VQGAN's slightly higher number, because VQGAN is "contaminated with ImageNet label information via perceptual loss" and because dVAE has better 1-gram token coverage on ImageNet validation data (shown in Figure 3). Specifically, dVAE uses almost all 8,192 tokens in its vocabulary across the validation set, while VQGAN-16K and VQGAN-1K use less than 50% and 30% of their vocabularies respectively. Full vocabulary coverage means the model uses its representational capacity more efficiently (every codebook entry carries distinct information), whereas low coverage means many codebook entries are never used, wasting capacity.

---

Resolution and RoPE: Low-Resolution Pre-training with High-Resolution Fine-tuning

A central efficiency insight in the paper is the relationship between token resolution, compute cost, and RoPE embeddings. This is detailed in Table 4.

The problem: at full 256×256 pixel resolution, dVAE produces 32×32 = 1,024 tokens per frame. With 16 frames, this would require a 16,384-token context length—four times what the model supports. Conversely, at 128×128 resolution (16×16 tokens), the model can fit 16 frames into 4,096 tokens, but the lower resolution degrades representation quality: Table 4 shows dVAE at 16×16 tokens achieves only 53.2% ImageNet top-1, while 32×32 achieves 61.2%.

The solution: pre-train the model at low resolution (16×16 tokens, 128×128 pixels) to save compute, then fine-tune at high resolution (32×32 tokens, 256×256 pixels) for only one epoch on the next-token prediction objective before downstream evaluation. This is possible because RoPE embeddings are relative (encoding the distance between tokens, not their absolute positions), so the model can generalize to token grid sizes it was not trained on.

The results in Table 4 are:

• dVAE at 16×16 tokens only: 53.2% (compute: $1.42 \times 10^{17}$ MACs)
• dVAE at 32×32 tokens only: 61.2% (compute: $5.68 \times 10^{17}$ MACs)
• dVAE pre-trained at 16×16, fine-tuned at 32×32 for 1 epoch: 63.2% (compute: $2.13 \times 10^{17}$ MACs)
• Same as above, but with RoPE base frequency increased from 10,000 to 50,000 during fine-tuning: 64.4%

What this means operationally: pre-training at low resolution costs approximately 1/4 the compute of pre-training at high resolution (since the number of tokens per frame is 1/4). The one-epoch fine-tuning step adds only a small incremental cost. The total compute is $2.13 \times 10^{17}$ MACs, which is about 37% of the full-resolution cost ($5.68 \times 10^{17}$) and yet achieves better performance (63.2% vs. 61.2%). Increasing the RoPE base frequency further improves this to 64.4%.

Why RoPE base frequency matters: the RoPE base frequency controls the wavelength of the sinusoidal functions used in the rotary embeddings. A higher base frequency means the embeddings encode shorter relative distances more finely, which is appropriate for higher-resolution token grids where neighboring tokens are physically closer in pixel space. The default base frequency (10,000 in LLaMA) was designed for text where tokens are much less redundant than visual tokens. Increasing it to 50,000 adapts the positional encoding to the higher spatial resolution, giving the model finer-grained position discrimination.

---

Representation Extraction: Attention Pooling and the Layer Selection Problem

Once the model is pre-trained, the question is how to extract a fixed-dimensional representation from the variable-length sequence of token activations for downstream tasks. The paper identifies and addresses a specific problem with decoder-only models: skewed receptive fields make average pooling suboptimal.

The problem with average pooling. In an autoregressive model with causal attention, a token at position $i$ can only attend to positions $1$ through $i$. This means early tokens (at the beginning of the sequence, e.g., top-left pixels of the first frame) see very little context—they can only attend to a handful of preceding tokens. Late tokens (at the end of the sequence, e.g., bottom-right pixels of the last frame) see the full 4,096 tokens of context. If you simply average all token representations, you equally weight tokens that have rich contextual information (late tokens) and tokens that have almost no context (early tokens), diluting the representation quality. In contrast, encoder models like MAE or DINO use bidirectional attention where every token sees every other token, so averaging is unbiased.

Attention pooling as a solution. Attention pooling learns to dynamically weight tokens based on their information content. The mechanism works as follows:

1. A learnable query vector $q \in \mathbb{R}^d$ is introduced, where $d$ is the model's hidden dimension.
2. Learned key and value projection matrices $W_k, W_v \in \mathbb{R}^{d \times d}$ map each token representation $h_i \in \mathbb{R}^d$ to a key $k_i = W_k h_i$ and value $v_i = W_v h_i$.
3. The query cross-attends to all token keys, producing attention weights: $\alpha_i = \text{softmax}(q^T k_i / \sqrt{d})$.
4. The final representation is the weighted sum: $r = \sum_i \alpha_i v_i$.

This is not a linear function of the token representations (because of the softmax nonlinearity), so the paper distinguishes it from linear probing. However, as the paper notes, it has been shown to learn better representations than average pooling in related work (El-Nouby et al., 2024).

Empirical validation: Table 5 shows that at the same layer of a Toto-large model probed on ImageNet, attention pooling achieves 61.1% top-1 accuracy versus 53.2% for average pooling—a 7.9 percentage point gap. This confirms that the skewed receptive field is a real problem and that attention pooling effectively compensates by learning to up-weight informative (late) tokens and down-weight uninformative (early) tokens.

Why attention pooling works: it can learn to approximate an "information content" weighting—tokens near the end of the sequence receive higher weights because they have seen more context. Unlike a hard-coded weighting scheme (which would need to know the optimal weight per position), attention pooling learns the weighting from data, adapting to the specific task, layer, and model size.

---

Layer Selection: Why Middle Layers Are Best

A consistent empirical finding across all model sizes and tasks is that the best representations for downstream tasks come from the middle layers of the decoder-only transformer, not the final layers. This is shown in Figure 4 (ImageNet probing across layers for base, large, and 1B models) and Figure 8 (the same pattern across image classification, action recognition, object tracking, and robot manipulation).

The pattern: probing accuracy rises from low values at layer 1, peaks at approximately 50% of the model depth (layer 6 out of 12 for base, layer 8 out of 16 for large, layer 11 out of 22 for 1B), and then declines through the remaining layers. The final layer typically performs substantially worse than the peak layer.

Explanation proposed by the paper: "the first half of the model starts to behave like an encoder, and compress the information, and then rest of the model projects the compressed semantic features back to input space." In more operational terms:

• Early layers build progressively richer representations as the receptive field grows and more context is integrated. Each layer from 1 to the midpoint compresses visual information into more abstract, semantic features.
• Late layers are optimized for the next-token prediction task—they must reconstruct the target token (the next visual element). This causes them to specialize in low-level prediction-relevant features (what exactly will the next patch look like) rather than high-level semantic features (what object is present, what action is occurring). The last layer's representation is "contaminated" by the need to predict the exact next token, making it less transferable to recognition tasks that need category-level abstractions.
• Middle layers sit at the inflection point: the representation is maximally compressed and semantic before being unpacked for prediction. This is analogous to the "bottleneck" in autoencoders, where the compressed representation captures the most general features.

This behavior was first observed in iGPT (Chen et al., 2020a) for autoregressive image models, and the paper confirms it generalizes across model sizes and to video pre-training. It contrasts sharply with encoder models (MAE, DINO) where the best features come from the final encoder layer, and with encoder-decoder models (BEiT) where the best features come from the top of the encoder stack.

For robot manipulation (Figure 8, rightmost column), the pattern partially differs: the last layers perform comparably to middle layers for the "picking" tasks in simulation. The paper suggests this is because "robot manipulation has a generative nature as a task and can benefit from generative pre-training"—the prediction-oriented features in late layers may actually be useful for motor control, where predicting future states is part of the task.

---

Image Recognition Evaluation Protocol

For ImageNet-1K classification (Section 4.2), the evaluation proceeds in stages:

1. Attention probing at every layer: a learnable attention pooling head (with cross-attention and a linear classifier, keeping the embedding dimension the same as the intermediate feature dimension) is trained at each layer of the frozen pre-trained model. The layer with the highest validation accuracy is selected.

2. Fine-tuning with self-supervised auxiliary loss: the pre-trained model is further trained using both the original next-token prediction loss (Equation 2) and a cross-entropy classification loss on the attention pooling head. Importantly, stop-gradients are applied to the probing head's gradient with respect to the classification loss—the classification loss only updates the probing head, not the pre-trained backbone. The self-supervised loss updates the backbone, keeping the representations aligned with the pre-training objective while adapting to the specific image distribution.

3. Resolution increase during fine-tuning: during this fine-tuning stage, the model is trained at 32×32 token resolution (256×256 pixels) rather than the 16×16 pre-training resolution. The RoPE base frequency is increased from 10,000 to 50,000 to accommodate the higher spatial resolution, as discussed earlier.

4. Layer decay: a learning rate decay factor of 0.9 is applied per layer, meaning that earlier layers receive a lower learning rate than later layers during fine-tuning. This preserves the general features learned in early layers while allowing later layers to adapt more to the ImageNet distribution.

5. Training duration: the probing heads are trained for 90 epochs with a learning rate of $6 \times 10^{-5}$.

The final results (Table 7) show Toto-base at 64.7%, Toto-large at 71.1%, and Toto-1b at 75.3% on ImageNet top-1 accuracy. For linear probing (no fine-tuning, no self-supervised loss, single linear layer on top of attention-pooled features at the optimal layer), the paper reports in Appendix A.4 that Toto-1b achieves 66.2%, compared to iGPT-L (1.39B parameters) at 65.2%.

---

Action Recognition Evaluation Protocol

For Kinetics-400 action recognition (Section 4.3), the evaluation follows a similar protocol to ImageNet but with important video-specific differences:

1. Frame sampling: 16 frames are sampled from each video with a stride of 4 frames (meaning every 4th frame is used). This is the same configuration as pre-training, so the model sees exactly the distribution it was trained on.

2. Resolution constraint: unlike ImageNet where the model is fine-tuned at 32×32 token resolution (256×256 pixels), video evaluation is conducted at the original 16×16 token resolution (128×128 pixels) "to keep the number of tokens in a similar budget." The paper does not explore higher-resolution video evaluation, which represents a clear opportunity for future improvement given the ImageNet results showing resolution benefits.

3. Attention probing and fine-tuning: as with ImageNet, an attention pooling head with cross-attention and an optional MLP classifier is trained at each layer, the best layer is selected, and the model is fine-tuned with both the self-supervised next-token loss and the classification loss. Appendix A.7 (Table 16) shows that using a higher-capacity classification head (cross-attention followed by an MLP, rather than a single linear layer) improves performance across all models: Toto-base from 59.3% to 61.2%, Toto-large from 65.3% to 65.8%, Toto-1b from 74.4% to 74.8%.

4. Evaluation protocol: the paper follows the SlowFast (Feichtenhofer et al., 2019) evaluation protocol, which involves taking multiple temporal clips per video and averaging predictions.

The final results (Table 8) show Toto-base at 59.3%, Toto-large at 65.3%, and Toto-1b at 74.4% on Kinetics-400 top-1 accuracy. The paper emphasizes that these are "the first to show competitive performance on action recognition with autoregressive generative modeling."

---

Action Forecasting Evaluation Protocol (Ego4D)

For Ego4D short-term action anticipation (Section 4.4), the evaluation diverges from the probing approach used for classification. Instead, Toto serves as the backbone for the StillFast pyramid network (Ragusa et al., 2023):

1. Multi-layer feature extraction: rather than probing a single best layer, features are extracted from 5 different layers of the pre-trained model and fused using the pyramid network architecture from StillFast. This leverages the observation that different layers contain complementary information.

2. Full fine-tuning: the entire model (backbone + pyramid network) is fine-tuned end-to-end on the Ego4D task with task-related losses. Additionally, the self-supervised next-token prediction loss is applied during fine-tuning, which the authors report "improves overall performance."

3. Task metrics: the Ego4D short-term action anticipation task requires predicting: (a) the noun (object to be interacted with), (b) the verb (type of interaction), and (c) the time-to-contact (estimated time until object-hand contact). The evaluation uses mean average precision (mAP), with results broken down by noun prediction alone, noun+verb jointly, noun+time-to-contact, and an overall combined metric.

Why use a pyramid network rather than simple probing: action anticipation requires integrating information across multiple temporal scales—understanding what is happening now, what has been happening recently, and inferring what will happen next. Single-layer probing loses access to the hierarchical temporal structure captured across layers; the pyramid network preserves this multi-scale information.

The results (Table 9) show Toto-large achieving an overall mAP of 2.70, which is competitive with or slightly better than previous methods (StillFast at 2.48, VideoMAE-large at 2.55, MAE-ST-large at 2.60), but the absolute numbers are low because this is an extremely challenging task—predicting future actions from egocentric video with long temporal horizons.

---

Video Tracking Evaluation Protocol (DAVIS)

For semi-supervised video object tracking on DAVIS (Section 4.5), the evaluation follows the protocol from Jabri et al. (2020), which tests whether the learned features support label propagation without any fine-tuning or probing:

1. Feature extraction: given a video, the pre-trained model processes all frames and produces token-level features at the optimal layer. No task-specific training is applied—the features are used as-is.

2. Label propagation: starting from a ground-truth segmentation mask in the first frame, the mask is propagated to subsequent frames by finding nearest-neighbor matches in feature space. Specifically, for each pixel in the current frame, the algorithm finds the pixel in the previous few frames whose feature vector is most similar (using cosine similarity or Euclidean distance), and copies the label from that matched pixel.

3. Temporal context: the algorithm uses features from the last $n$ frames to match against the current frame, providing robustness to appearance changes.

4. Resolution variants: the paper evaluates at 256×256 resolution (8×8 pixel patches, since the 256-pixel frame is divided into patches that map to tokens at a specific stride) for Toto-base and Toto-large, and additionally at 512×512 resolution for Toto-large. The larger resolution uses RoPE fine-tuning to handle the increased token count.

The results (Table 10) use the J&F metric (mean of Jaccard index for region similarity and F-measure for contour accuracy). At 256×256 resolution:

• Toto-base: 42.0 J&F
• Toto-large: 44.8 J&F
• Toto-1b: 46.1 J&F

At 512×512 resolution (only evaluated for Toto-large): 62.4 J&F, which "outperforms all methods" including DINO-base at 54.3 J&F. The paper notes that this evaluation "does not require finetuning or probing of the features," making it a particularly clean test of the pre-trained representations' quality for dense correspondence tasks.

Why tracking evaluates a different capability than classification: classification requires semantic abstraction—grouping instances into categories regardless of appearance variation. Tracking requires instance-level feature consistency—the features for the same physical point should remain similar across frames despite changes in viewpoint, lighting, and occlusion. The fact that autoregressive pre-training produces features that work well for both tasks (without task-specific design) is evidence for the generality of the learned representations.

---

Robotics Evaluation Protocol

For robotic manipulation (Section 4.6), the evaluation tests whether the pre-trained representations transfer to motor control tasks in both simulation and the real world:

Simulation experiments (following Xiao et al., 2022):

1. The pre-trained model is frozen—no fine-tuning on robot data.
2. For a given camera observation, the model processes it and extracts token features at an intermediate layer (the optimal layer is determined by probing on the robot task validation set).
3. Average pooling (not attention pooling, since the paper uses simple average pooling for robotics in simulation) produces a fixed-dimensional embedding from the token features.
4. A learned linear layer projects this embedding to a state representation used by the DAgger policy.
5. The DAgger policy (a behavior cloning algorithm with online data collection) is trained from scratch using these frozen visual embeddings.

Four tasks are evaluated: Franka-pick, Kuka-pick, Franka-cabinet, and Kuka-cabinet, spanning two robot platforms and two task types (picking and cabinet opening). The metric is mean success rate over training steps, shown in Figure 6.

Real-world experiments (following Radosavovic et al., 2022):

1. The pre-trained Toto-base model is frozen and used to extract vision features from camera observations of a Franka robot.
2. A controller is trained on top of frozen representations using behavior cloning on 240 expert demonstrations of a cube-picking task.
3. The evaluation is the success rate over 16 trials with variations in object position and orientation.

The simulation results (Figure 6) show that Toto-base learns the tasks faster and with better sample efficiency than the MAE baseline (Radosavovic et al., 2022) across all four tasks. The real-world results (Table 11) show Toto-base achieving a 63% success rate compared to 75% for the MVP baseline. The paper notes this is competitive with "a vision encoder pre-trained for robotics" (referring to MVP, which was specifically designed with robotics in mind, using datasets including robot-relevant data like 100DOH hand-object interactions). Toto was not designed for robotics and was pre-trained only on general internet images and videos, making the competitive performance notable.

Why robotics is an important evaluation: it tests transfer to a domain with fundamentally different data distributions (robot egocentric views vs. internet videos), different output requirements (continuous control vs. discrete classification), and strict real-time constraints. The fact that autoregressive video pre-training produces representations useful for motor control without any task-specific adaptation strengthens the paper's claim of learning general visual representations.

---

Object Permanence Evaluation Protocol (CATER)

For object permanence (Section 4.7), the evaluation uses the CATER dataset (Girdhar & Ramanan, 2019), which tests a specific cognitive capability: can the model track an object even when it is temporarily occluded or hidden?

1. The task: a ball moves in a 3D scene, sometimes becoming hidden behind or inside other objects. The model must predict the ball's coarse location in a 6×6 spatial grid at the end of the video, requiring it to maintain a representation of the ball's position even through occlusions.
2. Toto-large is fine-tuned on this task at two temporal resolutions: 16 frames and 32 frames. The 32-frame evaluation demonstrates that the model can process longer sequences than it was pre-trained on (pre-training used 16 frames).
3. The paper compares against V3D and TFC-V3D (Zhang, 2022), which are architectures specifically designed for temporal reasoning on CATER.

The results (Table 12) show Toto-large achieving 62.8% accuracy with 16 frames and 72.9% with 32 frames, compared to V3D's 55.2% and 69.7% respectively. This suggests that autoregressive pre-training learns representations that naturally support temporal reasoning and object tracking through occlusions, without explicit architectural mechanisms for object permanence.

---

Compute-Optimal Scaling: µP-Parameterization and Scaling Laws

Section 4.9 studies how the validation loss scales with training compute for autoregressive video models, using µP-parameterization (Yang et al., 2022) to enable efficient hyperparameter transfer across model scales.

µP-parameterization setup. The paper trains six model variants (a1 through a6) with linearly increasing hidden sizes and layer counts, using the VQGAN tokenizer (Table 15):

The key hyperparameter is the learning rate. Using µP-parameterization ensures that the optimal learning rate is the same across all model widths, avoiding the need for expensive per-model hyperparameter tuning. Figure 11 confirms this: all models from a1 to a6 show optimal performance at a learning rate of $2^{-7} = 1/128 \approx 0.0078$.

Scaling law results. Once the optimal learning rate is determined, all six models are trained on the full data mixture (Table 2), and their validation losses are plotted against total training compute measured in MACs (multiply-accumulate operations). Figure 9 shows the resulting scaling curve, which is well-fit by a power law:

$L(C) = 7.32 \cdot C^{-0.0378}$

where $L$ is the validation loss and $C$ is the training compute in MACs.

What it computes: for any given total training compute budget $C$, this power law predicts the expected validation loss. The coefficient $7.32$ is the intercept (the loss when $C = 1$ in the appropriate units), and the exponent $-0.0378$ determines how much the loss decreases for each doubling of compute.

Why this form: power laws are the standard functional form for neural scaling laws, empirically observed across domains from language modeling (Hoffmann et al., 2022; Brown et al., 2020) to image modeling (Henighan et al., 2020). They encode the intuition that each multiplicative increase in compute yields a multiplicative decrease in loss, with the exponent controlling the rate of improvement. A more negative exponent means faster improvement with scale.

Comparison with language models. The paper compares this scaling law to GPT-3's reported relationship: $L(C) = 2.57 \cdot C^{-0.048}$ (Brown, 2020). The Toto exponent ($-0.0378$) is shallower than GPT-3's ($-0.048$). In practical terms, this means that doubling the compute budget for Toto reduces the validation loss by approximately $2^{-0.0378} \approx 0.974\times$ (a 2.6% reduction), while doubling compute for GPT-3 reduces loss by approximately $2^{-0.048} \approx 0.967\times$ (a 3.3% reduction).

The paper's interpretation: "visual next token prediction models, such as Toto, scale but at a slower rate than language models." The suggested reason, discussed in Appendix A.1, is that video frames contain substantial redundancy—many tokens are predictable by interpolation rather than requiring genuine extrapolation or semantic understanding—which makes the next-token prediction task easier and provides a weaker training signal for learning high-level representations. The scaling curve reflects this: more compute helps, but the marginal benefit per unit of compute is smaller than in text, where tokens carry more distinct information.

Practical implications of the scaling result:

1. Autoregressive video pre-training is predictably improvable with scale—investing more compute will yield better representations, following a known curve. This is the same property that made language model scaling a reliable engineering strategy.
2. However, the shallower exponent means that catching up to language model performance on vision tasks will require disproportionately more compute, or improvements to the pre-training objective that reduce video redundancy (the paper explicitly identifies this as a direction for future work).
3. The scaling laws are specific to the autoregressive objective and the data mixture used—changing the tokenizer, data composition, or pre-training task would change the scaling curve. The paper's contribution is establishing the baseline scaling behavior for this particular recipe.

4. Key Insights and Innovations

Innovation 1: Autoregressive Pre-training Works for Video Despite the "Video Redundancy" Problem

The field's dominant assumption has been that video data is fundamentally problematic for autoregressive pre-training because adjacent frames are highly redundant—a model can predict the next frame trivially by copying the previous one, providing weak training signal. Masked autoencoding methods (VideoMAE, ST-MAE) sidestepped this by forcing the model to reconstruct heavily masked inputs, creating an artificial information bottleneck. The implicit belief was that autoregressive modeling on video would learn lazy representations that fail to capture semantic structure.

This paper falsifies that assumption empirically. Despite the acknowledged redundancy (documented in Figure 10 and Appendix A.1, where validation loss drops sharply after the first frame), autoregressive pre-training on video tokens at scale produces representations that transfer competitively across a remarkably diverse suite of tasks—image recognition (75.3% top-1 on ImageNet with the 1B model), action classification (74.4% on Kinetics-400), video tracking (62.4 J&F on DAVIS at 512 resolution, outperforming DINO), action anticipation, object permanence, and robotic manipulation. The model was given no architectural mechanisms to combat redundancy, no masking strategy, no contrastive objective—just raw next-token prediction. And yet it learns features that support instance-level correspondence (tracking), semantic categorization (classification), temporal reasoning (forecasting, object permanence), and motor control (robotics).

What makes this a conceptual contribution rather than merely a performance report is the breadth of the evaluation. Prior autoregressive vision work (iGPT, AIM) evaluated almost exclusively on ImageNet classification. A single benchmark tells you whether a model learns something useful, but not how general that something is. By testing on tasks that require fundamentally different competencies—dense correspondence without fine-tuning (DAVIS tracking), future prediction from egocentric video (Ego4D forecasting), physical reasoning through occlusion (CATER object permanence), and real-time motor control (robotics)—the paper demonstrates that the representations are genuinely general, not just well-suited to one type of supervised probe. Table 10 is particularly striking: the tracking results require no fine-tuning at all, yet Toto-large at 512 resolution exceeds DINO-base, a discriminatively trained model. This is the strongest evidence that autoregressive video pre-training learns something broadly useful about visual structure, not just a hidden classifier.

The comparison with iGPT (Chen et al., 2020a) sharpens the insight. iGPT showed that autoregressive pre-training on pixels could learn ImageNet-relevant features, but it used a 7B-parameter model to reach 72% top-1. Toto-1b reaches 75.3% with roughly one-seventh the parameters, and with linear probing (fairer comparison) Toto-1b achieves 66.2% versus iGPT-L's 65.2% at comparable scale (Table 14). The difference is tokenization: iGPT operated on clustered pixel patterns, which are extremely low-level and computationally expensive per token. Toto uses learned discrete tokenizers (dVAE) that compress semantic information into compact codes, enabling longer effective context and higher resolution at the same compute cost. This is a fundamentally different answer to "what is the right level of abstraction for visual autoregressive modeling?"—the tokenizer matters enormously, and modern learned tokenizers unlock scale efficiencies that pixel-level approaches could not achieve.

Innovation 2: Predictable Scaling Laws With a Revealingly Shallow Exponent

The paper establishes that autoregressive video pre-training follows a power-law relationship between compute and validation loss: $L(C) = 7.32 \cdot C^{-0.0378}$, validated across six model sizes from 15M to 1.9B parameters using µP-parameterization for consistent hyperparameter transfer (Figure 9, Section 4.9). Prior work (Henighan et al., 2020) had studied scaling laws for autoregressive image and video models, but this paper provides the first scaling characterization for the specific recipe of transformer-based next-token prediction on discrete visual tokens at scale, with careful control of learning rates and architecture. The existence of predictable scaling—not just that the 1B model does better than the 120M model—is what matters practically: it means investment in autoregressive video pre-training follows known, diminishing-returns dynamics, enabling informed resource allocation.

But the genuinely insightful finding is the comparison of the scaling exponent to language. GPT-3's reported scaling law gives an exponent of −0.048; Toto's is −0.0378. The shallower exponent means visual autoregressive models improve more slowly per unit of additional compute than text autoregressive models. This is not just an incidental number—it is a diagnostic measurement of a fundamental property of the data modality. The paper connects this directly to the redundancy hypothesis (Appendix A.1, Figure 10): because many video tokens are predictable by simple interpolation or frame-to-frame copying, the learning task is effectively "easier" than next-word prediction in text, where each token typically carries substantial new information. An easier task provides weaker gradient signal for learning high-level abstractions, which translates into a shallower scaling curve.

What distinguishes this from a routine scaling-law measurement is that it reconceptualizes the video pre-training challenge. The field had framed the problem as "how do we make autoregressive pre-training work for video despite redundancy?" This paper's scaling result reframes it as "the redundancy imposes a measurable ceiling on scaling efficiency, and the question is whether we can design objectives that steepen the curve." The scaling exponent becomes a metric for evaluating pre-training objective quality: if a future method (e.g., masking, contrastive auxiliary losses, smarter frame sampling) can achieve a steeper exponent on the same data, it would represent a genuine improvement in learning efficiency, not just a one-time performance boost. The paper does not solve the redundancy problem, but it gives the field a quantitative language for measuring progress against it.

Innovation 3: Decoder-Only Vision Models Have a Characteristic "Bottleneck" Layer Architecture

The paper's systematic layer-wise probing analysis (Figures 4 and 8) reveals a consistent architectural property of decoder-only autoregressive vision models: the best representations for downstream tasks consistently emerge from the middle layers (~50% of depth), with performance declining through the final layers. This pattern generalizes across all three model sizes (base, large, 1B) and across four task families (image classification, action recognition, video tracking, robot manipulation). The paper offers an interpretation: "the first half of the model starts to behave like an encoder, and compress the information, and then rest of the model projects the compressed semantic features back to input space."

This is a diagnostic insight that changes how practitioners should use these models. In encoder-only architectures like ViT (used by DINO, MAE) or encoder-decoder architectures (BEiT), the natural assumption—and the one baked into prior evaluation protocols—is that the final encoder layer contains the best features. This paper shows that assumption is wrong for decoder-only autoregressive models, and that probing the final layer can leave substantial performance on the table. The 7.9 percentage point gap between attention pooling and average pooling at the same layer (Table 5) reinforces that the combination of a skewed receptive field (early tokens see little context) and prediction-oriented late layers creates a feature extraction challenge that requires specific mitigation strategies.

The contribution here is not just the empirical observation (which was first noted in iGPT) but the cross-task generalization of the finding and its practical codification. Figure 8 demonstrates that the middle-layer peak holds for tasks as different as recognizing object categories from still images and controlling a robot arm from video—suggesting it is a property of the architecture and training objective, not of a specific downstream task. The interesting partial exception is robot manipulation, where late layers perform comparably to middle layers for picking tasks. The paper speculates that this is because motor control has a "generative nature" and benefits from the prediction-oriented features in later layers. This exception reveals something about what different layers encode: early layers compress, middle layers semanticize, late layers reconstruct. If your downstream task requires reconstruction-like capabilities (predicting future states for control), late layers may be useful. For semantic tasks (classification, tracking), middle layers are optimal.

For practitioners, this insight has immediate implications: when building systems on top of autoregressive vision models, always probe across layers rather than defaulting to the final output, and in decoder-only models specifically, budget feature extraction around the midpoint of the model. This is not an obvious design choice—in language models, the final layer representations are standard for downstream use—and the paper provides systematic evidence for why vision diverges.

Innovation 4: The "Compute-Optimal Resolution" Strategy via RoPE Fine-Tuning

The paper's finding that pre-training at low resolution (16×16 tokens, 128×128 pixels) followed by a single epoch of fine-tuning at high resolution (32×32 tokens, 256×256 pixels) outperforms pre-training at high resolution from scratch (64.4% vs. 61.2% on ImageNet, at roughly 37% of the compute cost; Table 4) is more than a computational efficiency trick. It reveals that Rotary Position Embeddings (RoPE), originally designed for length generalization in language models, serve a qualitatively different purpose in vision: they enable resolution generalization that would be impossible with absolute position embeddings.

The key intellectual move is recognizing that the token grid in visual autoregressive models is analogous to the sequence length in language models—and that just as RoPE allows a language model trained on 2K-token sequences to generalize to 4K tokens at inference, it allows a vision model trained on 16×16 token grids to generalize to 32×32 grids. The additional finding that increasing RoPE's base frequency from 10,000 to 50,000 improves high-resolution performance (boosting accuracy from 63.2% to 64.4%) provides a tuning knob for resolution adaptation that has no analog in absolute position embedding schemes. This explains why the GPT2 architecture (which uses absolute sine-cosine embeddings) performs substantially worse than LLaMA (which uses RoPE) in Table 6 (48.5% vs. 53.2%): GPT2 cannot benefit from this resolution fine-tuning strategy.

This is an incremental but practically important innovation because it decouples the pre-training budget from the target evaluation resolution. In MAE or DINO, the model must be trained at the target resolution (or use interpolation schemes that degrade quality). In Toto, the model can be trained cheaply at low resolution and then adapted to high resolution with minimal additional cost. For practitioners, this changes the economics of scaling: you do not need to commit to training at the final target resolution from the start. For researchers, it opens a line of investigation into what other forms of "resolution generalization" are possible—can a model trained at 128 pixels generalize to 512 pixels? To what resolution? What is the relationship between RoPE base frequency, pre-training resolution, and maximal generalizable resolution? The paper provides a single data point (128→256 works), but the principle is what matters.

Innovation 5: Attention Pooling as a Necessary Feature Extraction Primitive for Causal Vision Models

The 7.9 percentage point gap between attention pooling (61.1%) and average pooling (53.2%) on ImageNet (Table 5) is not a routine "we found a better hyperparameter" result. It diagnoses a structural problem with using decoder-only models for representation extraction that does not exist in encoder or encoder-decoder architectures. Because causal attention creates a skewed receptive field—tokens at the beginning of the sequence attend to almost nothing, tokens at the end attend to the full context—uniformly averaging all token representations weights information-poor and information-rich tokens equally. Attention pooling solves this by learning to dynamically weight tokens, effectively learning to favor late-sequence tokens that have integrated more context.

What makes this an innovation rather than an implementation detail is that it establishes attention pooling as the default extraction method for autoregressive vision models, analogous to how global average pooling became the default for CNNs and ViTs. The paper provides the systematic evidence for why the default from other architectures fails and what should replace it. This matters for future work: any evaluation of autoregressive vision models that uses average pooling will systematically underestimate representation quality, potentially leading to false negative conclusions about the method's effectiveness. The field needed this calibration to fairly compare autoregressive approaches against methods where average pooling is standard and appropriate.

The connection to the layer-wise probing results (the middle-layer peak) reveals a deeper property: the optimal layer for extraction and the optimal pooling strategy interact. Middle layers have already accumulated substantial context while still retaining semantic content (before the late-layer specialization in reconstruction). Combined with attention pooling that weights tokens by their effective receptive field, the middle-layer peak becomes even more pronounced. The paper does not explore this interaction explicitly, but the data implies it: probing strategies and layer selection should be jointly optimized, and the default recipe of "take the final layer, average pool" will substantially understate what autoregressive video models have learned.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper uses five primary datasets: ImageNet-1K (Deng et al., 2009) for image classification (1.28M training images, 50K validation), Kinetics-400 (Kay et al., 2017) for action recognition (~240K training videos, 400 action classes), Ego4D (Grauman et al., 2022) for short-term action anticipation (3,750 hours of egocentric video), DAVIS 2017 (Pont-Tuset et al., 2017) for semi-supervised video object tracking (60 validation videos with pixel-level annotations), and CATER (Girdhar & Ramanan, 2019) for object permanence (synthetic videos with occluded object localization). For robotics, two simulation benchmarks (Franka and Kuka robots across pick and cabinet tasks) and a real-world Franka cube-picking task are used, following the protocols from Xiao et al. (2022) and Radosavovic et al. (2022) respectively. The pre-training dataset (Table 2) mixes samples from ImageNet, Kinetics-600, Ego4D, and HowTo100M at approximate ratios of 20%/10%/10%/60% per batch, totaling over 100,000 hours of video and ~2.5 trillion visual tokens, with actual training using ~1 trillion tokens.

• Base model(s). All experiments use the Toto family of autoregressive transformer models trained from scratch at three scales: Toto-base (120M parameters), Toto-large (280M), and Toto-1b (1.1B). The architecture is a LLaMA-style (Touvron et al., 2023) decoder-only transformer with causal attention, RMSNorm, SwiGLU activations, and RoPE positional embeddings. The authors argue that LLaMA is "representative of the capabilities of many contemporary" architectures (Section 4.1). For the architecture ablation, GPT2 (Radford et al., 2019, 280M parameters) and Mamba (Gu & Dao, 2023, 290M parameters) are also trained with comparable settings.

• Metrics. For ImageNet-1K and Kinetics-400, the metric is top-1 classification accuracy (%), computed by standard single-crop evaluation with the protocol from SlowFast (Feichtenhofer et al., 2019) for video. For Ego4D action anticipation, the metric is mean average precision (mAP) for noun prediction, noun+verb (action) prediction, verb+time-to-contact (ttc), and an overall combined metric. For DAVIS tracking, the metrics are Jaccard index (J), F-measure (F), and their mean (J&F), measuring region similarity and contour accuracy during label propagation. For CATER, the metric is top-1 localization accuracy (%) for predicting the grid location of a hidden ball. For robotics simulation, the metric is mean success rate over training steps for DAgger policies. For real-world robotics, the metric is success rate over 16 trials with varied object positions.

• Baselines. For image recognition, baselines include discriminative methods (SimCLR, BYOL, SwAV, DINO, DINOv2) and generative methods (BEiT, AIM, MAE, iGPT-L, iGPT-XL), all with citations to their original papers (Table 7). For action recognition, baselines include I-JEPA, OpenCLIP, DINOv2, InternVideo (discriminative) and Hiera, MVD, VideoMAE (generative), as shown in Table 8. For Ego4D action anticipation, baselines include FRCNN+Rnd, FRCNN+SF (Grauman et al., 2022), Hiera-large, StillFast, VideoMAE-large, and MAE-ST-large (Table 9). For DAVIS tracking, baselines are DINO-base and MAE-base (Table 10). For CATER, baselines are V3D and TFC-V3D (Table 12). For robotics simulation, the baseline is MAE-base from MVP (Radosavovic et al., 2022), and for real-world robotics, the baseline is the MVP vision encoder (Table 11).

• Generation budget / compute accounting. For pre-training, compute is measured in MACs (multiply-accumulate operations), with specific values reported for resolution experiments (Table 4: dVAE/16 at 1.42 × 10¹⁷ MACs, dVAE/32 at 5.68 × 10¹⁷ MACs). For the scaling law analysis, training compute is measured in total MACs, and validation loss is plotted against compute to fit the power law. For fair comparison across methods (Table 4), resolution variants are compared at equivalent total compute. For downstream tasks, compute is not reported in detail beyond model parameter counts. All pre-training uses a fixed context length of 4,096 tokens and a batch size of 1 million tokens.

• Cross-validation / statistical protocol. For layer-wise probing (determining which layer's features are optimal), the model is probed at every layer, and the best-performing layer on the validation set is selected—no explicit k-fold cross-validation is reported. For the scaling law experiments, µP-parameterization (Yang et al., 2022) is used to ensure a consistent optimal learning rate across model widths, avoiding per-scale hyperparameter tuning. For real-world robotics, results are averaged over 16 trials. Most other experiments report single-run results without confidence intervals or multiple seeds.

---

Main Quantitative Results

Image Recognition (ImageNet-1K)

The headline result from Table 7: Toto-1b achieves 75.3% top-1 accuracy on ImageNet-1K, with Toto-large at 71.1% and Toto-base at 64.7%. Among autoregressive generative models, this represents the highest reported accuracy—iGPT-XL at 6.8B parameters reached 72.0%, while Toto-1b surpasses it with roughly one-seventh the parameters. However, discriminatively trained models still substantially outperform generative approaches: DINO ViT-B/8 achieves 80.1%, DINOv2 ViT-g/14 reaches 86.4%, and MAE ViT-H/14 achieves 80.9%, all with comparable or smaller parameter counts.

The scaling with model size is clear: moving from base (120M) to large (280M) yields a +6.4 percentage point improvement, and from large to 1b (1.1B parameters) yields a +4.2 point improvement. The gap to discriminative methods narrows with scale—1b is within ~5 points of the best discriminative models at similar parameter counts—but does not close, consistent with the paper's framing that discriminative objectives are better aligned with classification benchmarks.

In the linear probing comparison (Table 14, Appendix A.4), where both iGPT and Toto are evaluated with a frozen backbone and single linear layer: Toto-1b achieves 66.2% vs. iGPT-L's 65.2% at comparable parameter counts (1.1B vs. 1.39B). This is a fairer comparison since iGPT was originally evaluated with linear probing. The result confirms that Toto's architectural improvements (discrete tokenization via dVAE, LLaMA architecture, RoPE) yield genuine gains over pixel-level autoregressive pre-training, not just improvements from probing methodology.

Full fine-tuning (Table 13, Appendix A.3) narrows the gap further: Toto achieves 82.6%, compared to DINO at 82.8%, MoCo v3 at 83.2%, BEiT at 83.2%, and MAE at 83.6%. This suggests that when allowed to adapt all parameters, the representations learned by autoregressive pre-training are competitive with those from dedicated self-supervised methods, with the remaining gap being small (~1 percentage point).

Action Recognition (Kinetics-400)

The headline result from Table 8: Toto-1b achieves 74.4% top-1 accuracy on Kinetics-400, with Toto-large at 65.3% and Toto-base at 59.3%. The paper emphasizes this as "the first to show competitive performance on action recognition with autoregressive generative modeling."

The comparison with generative baselines is strong: Toto-1b is competitive with VideoMAE ViT-L/14 (79.8%) and MVD ViT-H/14 (79.4%), though still ~5 points behind. The gap to discriminative methods is larger on Kinetics than on ImageNet—DINOv2 reaches 84.4% and OpenCLIP reaches 83.3%—suggesting that video action recognition benefits more from discriminative training objectives. However, the scaling trend from base to 1b (+15.1 percentage points) is steeper than on ImageNet (+10.6 points from base to 1b), suggesting that autoregressive video pre-training may benefit disproportionately from scale on temporal understanding tasks.

The higher-capacity probing head (cross-attention + MLP, Table 16 in Appendix A.7) provides a modest boost: Toto-base improves from 59.3% to 61.2%, Toto-large from 65.3% to 65.8%, and Toto-1b from 74.4% to 74.8%. The diminishing returns from additional probing capacity at larger scales suggest that the representations themselves, rather than probing architecture, are the primary bottleneck.

All Kinetics evaluations are conducted at 128×128 resolution (16×16 tokens per frame, 16 frames), matching the pre-training configuration. The paper does not evaluate at higher spatial resolution for video, unlike the ImageNet experiments that benefit from 256×256 resolution. This represents an unquantified headroom—given that ImageNet improves substantially with resolution (Table 4), it is plausible that Kinetics performance would also improve with higher-resolution evaluation.

Action Forecasting (Ego4D)

The headline result from Table 9: Toto-large achieves an overall mAP of 2.70 on Ego4D short-term action anticipation, serving as the backbone for the StillFast pyramid network. This is the highest among reported methods, marginally ahead of MAE-ST-large (2.60), VideoMAE-large (2.55), and StillFast (2.48).

The breakdown by sub-metrics: Toto-large achieves 15.20 on noun prediction (vs. 17.55 for FRCNN-based methods, which have a specialized object detection backbone), 6.75 on noun+verb joint prediction (vs. 7.47 for StillFast), and 5.41 on noun+ttc (vs. 5.37 for FRCNN+SF). The overall mAP of 2.70 being the best result, despite not leading in every sub-metric, indicates balanced performance across the three prediction dimensions.

The absolute numbers are very low (2.70 mAP overall), which the paper does not emphasize but which reflects the extreme difficulty of this task: predicting from egocentric video what object a person will interact with, what action they will perform, and when contact will occur. The fact that Toto's features transfer to this task at all—despite being pre-trained without any action-anticipation-specific design—is notable, and achieving state-of-the-art performance suggests the learned temporal representations are genuinely useful for forecasting.

Video Tracking (DAVIS)

The headline result from Table 10: at 512×512 resolution, Toto-large achieves 62.4 J&F on DAVIS semi-supervised tracking, outperforming DINO-base (54.3 J&F) and all other reported methods. At 256×256 resolution, Toto-1b achieves 46.1 J&F, Toto-large 44.8 J&F, and Toto-base 42.0 J&F.

This result is particularly significant because tracking evaluation uses no fine-tuning and no probing—the pre-trained features are used directly for nearest-neighbor label propagation. This is a cleaner test of representation quality than classification probing, where a learned head can compensate for weak features. The fact that Toto-large at 512 resolution exceeds DINO-base, a discriminatively trained model explicitly optimized for instance-level feature consistency, demonstrates that autoregressive pre-training learns features that support dense spatial correspondence without being trained to do so.

The resolution effect is dramatic: Toto-large improves from 44.8 J&F at 256 resolution to 62.4 J&F at 512 resolution—a +17.6 point gain. This is consistent with the ImageNet finding that higher resolution significantly improves feature quality (Table 4), and suggests that DAVIS tracking benefits from fine-grained spatial features that the low-resolution pre-training alone cannot provide. The RoPE fine-tuning strategy (pre-train at 128 pixels, fine-tune at 256 or 512 with increased base frequency) is what enables this resolution scaling without retraining from scratch.

Scaling with model size at the same resolution (256 pixels) shows modest gains: base at 42.0, large at 44.8, 1b at 46.1. The ~2 point improvement per scale step suggests that for dense correspondence tasks, resolution matters more than parameter count within the range tested.

Robotics

Simulation results (Figure 6): Toto-base learns all four tasks (Franka-pick, Kuka-pick, Franka-cabinet, Kuka-cabinet) faster and with better final performance than the MAE-base baseline from MVP (Radosavovic et al., 2022). The mean success rate curves show Toto-base (orange) consistently above MAE-base (blue) throughout training for all tasks. The paper notes that Toto was "able to learn these tasks faster than MAE, across two robots and two tasks."

The MAE baseline is trained on ImageNet, Ego4D, and 100DOH (a hand-object interaction dataset), giving it domain-specific advantages for manipulation. Toto was trained only on general internet images and videos without any robotics-specific data, yet outperforms this strong baseline in simulation. This is evidence that autoregressive pre-training on diverse video data yields features that transfer to embodied control without domain alignment.

Real-world results (Table 11): Toto-base achieves a 63% success rate on Franka cube-picking over 16 trials, compared to 75% for the MVP baseline. The paper characterizes this as performing "favorably" and "comparably to the state-of-the-art vision backbone for robotics, despite not being designed with the robotic application in mind." The gap (63% vs. 75%) is non-trivial—a 12-point difference—but the evaluation uses only 16 trials with unstated variance, making it difficult to assess statistical significance. The real-time requirement (Toto-base is used "to run the robot at real time") constrains model size; it is unknown whether a larger Toto model would close the gap with MVP.

The real-world experiment is limited to a single task (cube-picking), a single robot platform (Franka), and a single baseline (MVP). This is substantially narrower than the simulation evaluation and provides only weak evidence for real-world transfer. The paper acknowledges implicitly that this is preliminary by noting that Toto-base is used "despite being a small model."

Object Permanence (CATER)

The headline result from Table 12: Toto-large achieves 62.8% localization accuracy with 16 frames and 72.9% with 32 frames on CATER snitch localization, compared to V3D's 55.2%/69.7% and TFC-V3D's 54.6%/70.2%. The CATER task requires the model to track a ball through occlusions and predict its final location in a 6×6 grid—a direct test of whether the model has learned to maintain object representations over time through visual obstruction.

Toto-large outperforms both baselines at both temporal resolutions, with the 16-frame gap (62.8% vs. 55.2% for V3D, a +7.6 point advantage) being larger than the 32-frame gap (72.9% vs. 70.2% for TFC-V3D, a +2.7 point advantage). The narrowing gap at higher frame counts is expected: with more frames, the tracking task becomes easier (more observations of the ball's trajectory reduce uncertainty), and specialized architectures catch up.

The improvement from 16 to 32 frames for Toto-large (+10.1 points) is larger than for the baselines, suggesting that Toto's autoregressive pre-training on video sequences makes it particularly effective at leveraging additional temporal context. The paper notes that Toto was pre-trained with 16 frames, so the 32-frame evaluation demonstrates generalization to longer sequences.

---

Ablation Studies and Robustness Checks

Tokenizer choice and resolution (Table 3): At equivalent pixel resolution (dVAE 32×32 tokens vs. VQGAN/patch 16×16 tokens, all covering ~256×256 pixels), Toto-large achieves similar ImageNet linear probing accuracy across all tokenizer types: dVAE 61.2%, VQGAN-16K 61.3%, VQGAN-1K 61.1%, continuous patches 60.6%. However, at the same number of tokens (16×16), dVAE (53.2%) substantially underperforms because it operates at half the pixel resolution. The mix-and-match configuration (patch input with dVAE targets, 58.5%) performs worse than consistent configurations, suggesting input-target consistency matters. The practical takeaway is that tokenizer type is not a strong determinant of representation quality when pixel resolution is controlled, but that dVAE is preferred because VQGAN is "contaminated with ImageNet label information via perceptual loss" (the VGG network used for perceptual loss was trained on ImageNet) and because dVAE has better vocabulary utilization (Figure 3 shows dVAE uses nearly all 8,192 tokens while VQGAN uses <50% of its codebook). The tokenizer comparison is conducted only on ImageNet linear probing with Toto-large—it is unknown whether the findings generalize to other tasks, model sizes, or probing methods.

Resolution and fine-tuning strategy (Table 4): Pre-training at low resolution (dVAE 16×16 tokens, 128×128 pixels) followed by one epoch of high-resolution fine-tuning (32×32, 256×256 pixels) achieves 63.2% top-1 accuracy, outperforming full-resolution pre-training from scratch (61.2%) while using only ~37% of the compute. Increasing the RoPE base frequency from the default 10,000 to 50,000 during fine-tuning further improves this to 64.4%. This demonstrates that RoPE enables resolution generalization that would be impossible with absolute position embeddings, and that the combination of low-resolution pre-training + RoPE-tuned high-resolution fine-tuning is both more compute-efficient and more accurate than training at full resolution throughout. The ablation is conducted only on ImageNet classification; the DAVIS tracking results (Table 10) show a related resolution benefit (512 resolution outperforms 256), but without the controlled comparison of pre-training strategies.

Pooling method (Table 5): At the same layer of a Toto-large model on ImageNet, attention pooling achieves 61.1% top-1 accuracy versus 53.2% for average pooling—a +7.9 percentage point gap. This confirms that the skewed receptive field in causal decoder-only models substantially degrades average-pooled representations, and that attention pooling learns to up-weight informative (late) tokens. The paper evaluates only the large model on ImageNet; it is unknown whether the gap changes with model size or task.

Architecture comparison (Table 6): At ~280M parameters on ImageNet linear probing, LLaMA achieves 53.2%, GPT2 with absolute positional embeddings achieves 48.5%, and Mamba (a state-space model) achieves 40.7%. The LLaMA advantage over GPT2 is consistent with the resolution fine-tuning findings—GPT2 cannot generalize to 32×32 tokens because absolute embeddings are fixed to the pre-training sequence length, while RoPE in LLaMA enables resolution adaptation. The Mamba result shows that state-space models underperform attention for visual autoregressive pre-training at this scale, though the paper does not explore whether this gap closes at larger sizes. This comparison is limited to a single model size (280M) and ImageNet classification, so architecture rankings may not generalize.

Probing layer selection (Figures 4, 8): Across all model sizes (base, large, 1b) and four task categories (image classification, action recognition, object tracking, robot manipulation), peak downstream performance consistently occurs at approximately 50% of model depth. For image classification (Figure 4), the peak is at layer 6/12 (base), layer 8/16 (large), and layer 11/22 (1b). The final layers underperform the peak substantially—on ImageNet with 1b, the last layer is roughly 10 points below the peak. For robot manipulation (Figure 8, right), the pattern partially differs: the last layers perform comparably to middle layers for picking tasks, which the paper interprets as robot control benefiting from the generative, prediction-oriented features in late layers. This ablation establishes layer-wise probing as necessary for fair evaluation of decoder-only vision models, and codifies the middle-layer heuristic for future work.

Scaling law parameterization (Figure 9, Figure 11): Across six model sizes (15M–1.9B parameters) using µP-parameterization, a single optimal learning rate of 2⁻⁷ works for all widths (Figure 11). The validation loss follows a power law L(C) = 7.32 · C⁻⁰·⁰³⁷⁸ (Figure 9). The exponent of −0.0378 is shallower than the −0.048 reported for GPT-3, implying that autoregressive video pre-training scales more slowly with compute than text pre-training. The scaling law is fit using VQGAN tokenization, not the default dVAE, so direct translation to the main experimental results requires caution. The paper does not report scaling laws for downstream task performance (only validation loss), so the relationship between loss reduction and accuracy improvement on benchmarks remains unquantified.

Full attention during fine-tuning (Appendix A.2): Although the model is pre-trained with causal attention, switching to full (bidirectional) attention during fine-tuning on ImageNet yields 82.6% with full attention vs. 82.2% with causal attention—a small +0.4 point gap. The paper attributes this to the video pre-training approximating full attention: "a middle token... won't see the rest half of the 8th frame, but have seen all the tokens from 7th frame, which are similar because of video." This suggests that the causal masking constraint is partially circumvented by temporal redundancy, and that fine-tuning can safely use full attention without architectural mismatch.

Prefix attention and multi-layer probing variants (Appendices A.2, A.7, A.9): Additional probing architectures (cross-attention with MLP for Kinetics, Table 16) provide small gains (+1–2 points), and probing across layers shows consistent middle-layer peak behavior for models trained at 128 and 256 resolutions (Appendix A.9, Figure 14). These confirm robustness to probing head capacity and resolution.

1-gram, 2-gram, and 3-gram token distributions (Figures 3, 12, 13): The dVAE tokenizer shows near-complete vocabulary coverage on ImageNet validation data for 1-gram statistics, while VQGAN variants use less than 50% of their vocabularies. For 2-gram distributions (Figure 12), dVAE uses a larger set of token combinations than VQGAN. For 3-gram (Figure 13), all tokenizers show similarly flat distributions. This analysis supports the claim that dVAE makes more efficient use of its representational capacity, though no downstream ablation directly tests whether vocabulary utilization causally affects representation quality.

Video frame redundancy (Figure 10, Appendix A.1): The average validation loss per token on Kinetics-400 shows that the first frame has substantially higher loss than subsequent frames. The paper interprets this as evidence that "frames following the first frame [have] redundancy, and hinders the learning, since these tokens are relatively easy to predict." This is the empirical basis for the claim that video redundancy limits autoregressive pre-training effectiveness. However, the paper does not ablate video frame sampling rate (using a stride of 4 throughout) or compare training on videos vs. images alone at equivalent token counts—experiments that would directly test whether redundancy is causal in limiting representation quality.

---

Critical Assessment

Does the paper demonstrate that autoregressive pre-training from videos leads to competitive performance across all benchmarks?

The claim is supported with important nuance. The paper evaluates on six task categories and shows that Toto models are competitive with or exceed prior generative methods on all of them, and approach discriminative methods on several. However, "competitive" means different things across tasks:

• On ImageNet (Table 7), Toto-1b at 75.3% is the best autoregressive generative model, but trails DINOv2 (86.4%) and MAE (80.9%) by substantial margins. The paper acknowledges this gap and attributes it to the alignment between discriminative objectives and classification benchmarks. The full fine-tuning result (82.6%, Table 13) narrows the gap to DINO (82.8%) and BEiT (83.2%), showing that when all parameters are adapted, the representations are genuinely competitive—but this is shown only in an appendix and at one model size (likely base, though unstated).

• On Kinetics-400 (Table 8), Toto-1b at 74.4% is significantly behind DINOv2 (84.4%) and OpenCLIP (83.3%). The paper's claim of being the "first to show competitive performance on K400 with autoregressive pre-training" is accurate relative to prior autoregressive work (which effectively had no K400 results), but masks the 10-point gap to state-of-the-art. The paper does not report full fine-tuning results on Kinetics as it does for ImageNet.

• On DAVIS tracking (Table 10), Toto-large at 512 resolution (62.4 J&F) genuinely outperforms all reported baselines, including DINO-base (54.3). This is the strongest single result in the paper—a task where the pre-trained features are used without any adaptation and yet exceed a discriminatively trained model. However, only Toto-large is evaluated at 512 resolution; it is unknown how discriminative methods would perform at equivalent resolution.

• On Ego4D (Table 9), Toto-large achieves 2.70 overall mAP, marginally best among all methods. The absolute numbers are very low, and the "state-of-the-art" claim rests on a 0.10–0.22 mAP advantage—a fragile margin given the small test set and high variance typical of action anticipation benchmarks.

• On CATER (Table 12), Toto-large achieves 62.8%/72.9% at 16/32 frames, outperforming specialized temporal reasoning architectures. This is a clean win and provides evidence for temporal reasoning capabilities learned from autoregressive pre-training. The baselines (V3D, TFC-V3D) are from a single prior work; broader comparison is absent.

• On robotics (Figures 6, Table 11), Toto-base outperforms MAE in simulation across all four tasks but trails MVP by 12 points in the real-world evaluation (63% vs. 75%). The simulation result is strong; the real-world result is preliminary (single task, 16 trials, no error bars) and does not clearly demonstrate superiority.

Bottom line: The claim of "competitive performance across all benchmarks" holds, but the degree of competitiveness varies substantially by task. The paper is strongest on tracking and CATER (where autoregressive pre-training genuinely excels), solid on ImageNet and Ego4D, and weakest on Kinetics and real-world robotics. A more precise characterization would be "autoregressive pre-training produces representations that transfer to diverse tasks, with task-dependent gaps to specialized methods ranging from negligible (tracking, CATER) to substantial (Kinetics classification, real-world manipulation)."

Does the paper demonstrate that scaling video models results in similar scaling curves to language models, albeit with a different rate?

The scaling law finding is supported but its comparison to language models is loose. The paper reports a power law L(C) = 7.32 · C⁻⁰·⁰³⁷⁸ for Toto and cites GPT-3's L(C) = 2.57 · C⁻⁰·⁰⁴⁸ for comparison. The comparison has several issues:

1. The scaling laws are fit to different data and tokenizers. Toto's scaling law uses VQGAN tokens and a specific data mixture; GPT-3's scaling law uses text tokens and a different data distribution. The exponents are not directly comparable without controlling for data complexity, tokenizer entropy, and vocabulary size.

2. The scaling law is validated on only six model sizes (15M–1.9B parameters), spanning roughly two orders of magnitude in compute. While this is a reasonable range, scaling law extrapolation over longer ranges is sensitive to the functional form and could change with additional data points at larger scales.

3. The scaling law measures validation loss, not downstream performance. The paper does not establish the relationship between validation loss reduction and downstream accuracy improvement, so the practical implication of a shallower exponent (that "more compute helps but returns diminish faster") remains qualitative.

4. The comparison to GPT-3 uses a single reported fit from Brown et al. (2020). GPT-3's scaling behavior itself has been refined by subsequent work (e.g., Chinchilla scaling laws from Hoffmann et al., 2022), and the paper does not engage with this more recent literature.

What the paper does demonstrate is that autoregressive video pre-training follows a power law—a predictable, smooth relationship between compute and loss—which is sufficient to claim "similar scaling curves" in the sense of functional form. The "different rate" claim is likely directionally correct (visual tokens are more redundant, implying a shallower curve), but the specific numerical comparison to GPT-3 should be treated as illustrative rather than precise.

Does the paper demonstrate that minimal inductive biases are sufficient?

This claim is well-supported by the experimental design but has boundaries. The paper deliberately avoids vision-specific architectural components (no spatial convolutions, no tubelet embeddings, no patchification), uses a standard language modeling architecture (LLaMA), and applies a generic next-token prediction objective. The fact that this recipe works at all across classification, tracking, forecasting, and robotics is evidence that inductive biases beyond the transformer architecture itself are not necessary for learning transferable visual representations.

However, the paper's own ablations reveal that several "minimal" choices actually encode substantial inductive biases:

• The tokenizer (dVAE) is not minimal. The dVAE is a pre-trained discrete autoencoder that compresses 256×256×3 pixels into 32×32 discrete codes. This tokenizer was trained with reconstruction loss and a uniform prior—no ImageNet labels—so it is unsupervised, but it still imposes a specific compression strategy and patch-based spatial decomposition. The paper shows that tokenizer choice matters less than expected (Table 3), but "minimal inductive bias" would more naturally mean operating on raw pixels (as iGPT did) or on simple patch embeddings without discrete quantization.

• The raster scan ordering imposes a 2D-to-1D mapping that creates artificial boundaries at frame edges. A token at the end of one row is treated as preceding the token at the start of the next row, despite being spatially distant. This is a strong inductive bias about the structure of images, inherited from the scanning metaphor in Attneave's 1954 proposal. The paper does not ablate ordering strategies (e.g., random, Hilbert curve, spiral) to test whether the raster scan assumption matters.

• The data mixture (Table 2) is curated. While the individual datasets are "unfiltered" internet data, the choice to mix ImageNet (images), Kinetics (exocentric video), Ego4D (egocentric video), and HowTo100M (instructional video) at specific ratios (20/10/10/60) encodes domain knowledge about what constitutes a diverse visual pre-training corpus. The paper does not ablate this mixture to determine whether, say, HowTo100M alone would suffice.

These are not weaknesses of the paper—the point of an empirical study is to test a specific recipe—but they qualify the "minimal inductive biases" framing. A more accurate description is "inductive biases limited to those present in standard language model architecture and a learned discrete tokenizer, with no vision-specific architectural modifications."

Genuine weaknesses and missing experiments

• No combination of masked modeling with autoregressive modeling. The paper positions itself against MAE-style masked methods but never trains a hybrid model (e.g., autoregressive prediction with some frames masked). Given the finding that video redundancy limits the autoregressive objective (Figure 10, Appendix A.1), a natural experiment would be to mask random frames or patches and train the model to predict them autoregressively, potentially increasing the difficulty and improving representation quality. This is not run.

• No single-dataset pre-training ablation. The paper trains on a mixture of four datasets but does not report results from training on any single dataset alone. It is unknown whether the diverse mixture is essential for the strong downstream transfer, or whether HowTo100M alone (which constitutes 60% of batches) would achieve similar performance. This matters for understanding whether the approach requires careful data curation or can work with any large video corpus.

• Kinetics evaluation is resolution-limited. The paper demonstrates substantial gains on ImageNet from increasing resolution from 128×128 to 256×256 pixels (Table 4, +8 points), and on DAVIS from increasing resolution to 512 pixels (+17.6 points for Toto-large). Yet Kinetics evaluation is conducted only at 128×128 resolution. The paper acknowledges this as a constraint—"to keep the number of tokens in a similar budget"—but this means Kinetics results likely understate what the model is capable of with resolution adaptation. Running even a single-resolution-increased Kinetics experiment would have calibrated this headroom.

• No generation quality evaluation. The paper briefly mentions generation in Appendix A.8 ("we can generate up to 64 frames... shows 3D rotation... shows object permanence") but provides no quantitative generation metrics (FID, FVD, IS, human evaluation). For a model trained explicitly on next-token prediction, generation is the primary capability the objective optimizes for. The absence of any generation quality measurement means the paper cannot assess whether the representations good enough for downstream tasks come at the cost of generation quality, or whether the two track together.

• Small test sets with unknown variance. The real-world robotics evaluation uses 16 trials; the CATER evaluation uses an unstated test set size (the original CATER paper uses ~5,000 test sequences, but the paper does not specify its split); the Ego4D task has a standard validation set but the paper does not report the number of test samples. The scaling law experiments appear to be single-run (no multiple seeds reported). Overall, statistical reliability is difficult to assess for most results.

• No comparison to video autoregressive baselines at equal scale. The paper compares extensively against masked autoencoding methods (MAE, VideoMAE) and discriminative methods (DINO, SimCLR), but does not train a controlled baseline using the same data, architecture, and scale with a different pre-training objective. For example, a MAE-trained version of Toto-large on the same data mixture would isolate the effect of the autoregressive objective versus the data and architecture. Without this, observed performance differences confound the pre-training objective with data scale, architecture details, and probing methodology.

Where the claims hold conditionally

Autoregressive pre-training is competitive with masked modeling: holds for tracking (Toto exceeds MAE at 512 resolution) and CATER (Toto exceeds specialized architectures); partially holds for classification (Toto approaches MAE/BEiT at comparable scale with full fine-tuning but trails with linear probing); unknown for other tasks where no direct MAE vs. Toto comparison at equivalent data/scale exists.

Scaling is predictable: holds within the tested range (15M–1.9B parameters) with the specific tokenizer and data used; extrapolation beyond this range is uncertain; downstream task scaling is unquantified.

Minimal inductive biases suffice: holds in the sense that no vision-specific architectural modifications are needed to achieve competitive transfer; does NOT hold in the sense that the tokenizer, data mixture, and scanning order themselves constitute inductive biases that the paper does not ablate.

Resolution generalization via RoPE works: strongly supported on ImageNet (Table 4) and DAVIS (Table 10); not tested on Kinetics or Ego4D; mechanism is well-understood (RoPE encodes relative positions, enabling length/resolution generalization), so this claim is credible even without exhaustive task coverage.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Unaccounted for and Potentially Prohibitive

The assumption or constraint. The paper's pre-training pipeline treats all video frames and images as equally valuable tokens, sampled uniformly from the data mixture at fixed ratios. However, the paper explicitly identifies in Appendix A.1 and Figure 10 that video frames beyond the first are substantially more predictable—the average validation loss drops sharply after the first frame, meaning the model is learning less from most of the tokens it processes. The paper states:

"frames following the first frame has some redundancy, and hinders the learning, since these tokens are relatively easy to predict"

This is an acknowledged feature of the data, not an oversight. The paper operates under the assumption that this redundancy is an acceptable cost of using video data at scale, and that the volume and diversity of the data compensate for the weaker per-token training signal.

The consequence. The training compute budget is spent predominantly on tokens that provide weak learning signal. Of the 4,096 tokens in each video sequence, tokens from the first frame (256 tokens, ~6% of the sequence) carry the bulk of the prediction difficulty; the remaining 3,840 tokens from subsequent frames are substantially easier to predict because they can be interpolated from prior frames. This means that training on video sequences may be significantly less compute-efficient per unit of learned representation than training on a curated set of diverse still images (where every token is maximally unpredictable given its context), or training with an objective that explicitly filters or reweights tokens based on their information content. The paper does not quantify how much of the total training compute is effectively "wasted" on easy tokens, nor does it experiment with masking, frame skipping, or token reweighting strategies that might mitigate this inefficiency. A practitioner deciding between video pre-training and image-only pre-training at a fixed compute budget cannot determine from this paper which is more efficient for a target downstream task.

What evidence exists in the paper. Figure 10 (Appendix A.1) directly shows the per-token validation loss on Kinetics-400, with the first frame having markedly higher loss than subsequent frames. The paper interprets this as "the redundancy in videos" and notes it "hinders the learning." The scaling law in Section 4.9 provides indirect supporting evidence: the shallower exponent (−0.0378 vs. −0.048 for GPT-3) is consistent with a weaker average training signal per token, which would be expected if many tokens are trivially predictable. However, the paper does not ablate the frame sampling rate (a stride of 4 is used throughout), does not compare training on videos vs. images alone at equivalent token counts, and does not experiment with token-level importance sampling or loss weighting.

Mitigation status. The paper does not attempt to mitigate this limitation. It identifies the issue in Appendix A.1 and flags it as a direction for future work: "to learn efficiently from videos, need further research in this direction." The acknowledgment is honest, but the headline results—which show competitive performance—should be understood as achievements despite this inefficiency, not evidence that the inefficiency is negligible. The unstated implication is that a method which addressed the redundancy could achieve equivalent or better downstream performance with substantially less compute, or better performance at the same compute budget.

---

The Scaling Law is Fit to Validation Loss, Not Downstream Performance, and Uses a Different Tokenizer Than the Main Experiments

The assumption or constraint. The compute-optimal scaling analysis in Section 4.9 fits a power law relating training compute (in MACs) to validation loss (next-token prediction cross-entropy). The paper then compares the exponent of this power law to GPT-3's reported exponent and concludes that "visual next token prediction models, such as Toto, scale but at a slower rate than language models." The implicit assumption is that improvements in validation loss translate predictably to improvements in downstream task performance—the quantity practitioners actually care about.

The consequence. The paper provides no mapping from validation loss reduction to downstream accuracy improvement for any task. A practitioner reading the scaling law cannot answer the question: "If I double my training compute budget for Toto, how many points of ImageNet accuracy do I expect to gain?" This makes the scaling law a diagnostic of the pre-training objective rather than a practical resource allocation tool. It is entirely possible that the relationship between validation loss and downstream accuracy is non-linear or saturating—for instance, that improvements in predicting redundant video tokens (which dominate the loss) provide diminishing returns for semantic representation quality, while improvements in predicting informative tokens provide larger returns. If so, the scaling law's exponent would overstate or understate the practical benefit of additional compute depending on which regime the model is in.

Furthermore, the scaling law experiments use the VQGAN tokenizer (Table 15, Figure 9), while the main experimental results throughout Sections 4.1–4.8 use the dVAE tokenizer. The paper provides no evidence that the scaling behavior transfers across tokenizers. Given that dVAE and VQGAN have different vocabulary utilization (Figure 3), different 2-gram distributions (Figure 12), and potentially different entropy per token, the fitted power law may not describe the behavior of the dVAE-based models that constitute the paper's primary contributions.

What evidence exists in the paper. Figure 9 shows the scaling law fit across six model sizes (a1–a6, 14.8M–1.9B parameters) using VQGAN tokens, with a clean power-law relationship. Figure 11 confirms that µP-parameterization enables consistent optimal learning rates across scales. The paper does not report a single downstream accuracy-vs.-compute curve for any task, nor does it fit scaling laws for any downstream metric. The tokenizer mismatch between the scaling law experiments and the main experiments is stated in the text ("All models use the VQGAN tokenizer") but its implications are not discussed.

Mitigation status. Not addressed. The paper treats the scaling law as a standalone contribution describing the behavior of "visual next token prediction models" in general, without validating that the specific findings (exponent, coefficient) transfer to the dVAE-based models that are the paper's main focus. A simple additional experiment—training a3 and a6 with dVAE tokens and checking whether they lie on the same power law—would have substantially strengthened the claim that the scaling behavior is tokenizer-agnostic.

---

No Direct Comparison Between Autoregressive and Masked Modeling at Equivalent Data Scale and Architecture

The assumption or constraint. The paper's central claim—that autoregressive pre-training from videos "leads to competitive performance across all benchmarks"—is evaluated against prior published results from masked autoencoding methods (MAE, VideoMAE, Hiera) and discriminative methods (DINO, DINOv2, SimCLR). However, these comparisons confound the pre-training objective (autoregressive vs. masked vs. contrastive) with data scale and composition (Toto's specific 100K-hour mixture vs. ImageNet-1K or custom datasets used by baselines), model architecture (LLaMA vs. ViT), tokenization (dVAE vs. patch embeddings), and evaluation protocol (attention probing at mid-layer vs. average pooling at final layer). The paper never trains a controlled baseline—for instance, a MAE-trained model using the identical architecture, tokenizer, and data mixture—that would isolate the effect of the autoregressive objective.

The consequence. A practitioner cannot determine from this paper whether the autoregressive objective is the right choice for their application, or whether the competitive performance comes from the data scale, the LLaMA architecture, or the attention pooling evaluation protocol. This is particularly acute for tasks where the paper reports strong results: DAVIS tracking (62.4 J&F at 512 resolution, exceeding DINO-base at 54.3) could be driven by the high resolution rather than the autoregressive objective (since discriminative models were not evaluated at 512 resolution), and CATER object permanence (72.9% at 32 frames, exceeding V3D's 70.2%) could be driven by the model's temporal context length rather than the generative training. Without controlled comparisons, the evidence for the specific value of autoregressive pre-training (as opposed to "training on lots of video data with a good architecture") is correlational rather than causal.

What evidence exists in the paper. The paper's comparisons are exclusively against published numbers from prior work (Tables 7–12). The architecture ablation (Table 6) compares LLaMA vs. GPT2 vs. Mamba within the autoregressive framework, but never varies the pre-training objective. The tokenizer ablation (Table 3) compares discrete vs. continuous tokenization within the autoregressive framework. The resolution ablation (Table 4) and pooling ablation (Table 5) similarly stay within the autoregressive paradigm. Nowhere does the paper train a non-autoregressive model on its own data and evaluate it with its own protocols, which is the standard approach for establishing the causal contribution of a specific design choice.

Mitigation status. Not addressed. The paper positions itself as "an empirical study of autoregressive pre-training from videos," which implies its scope is characterization of the autoregressive approach rather than comparative optimization. However, the repeated framing of results as "competitive" with alternatives implies a comparison that the experimental design does not rigorously support. A future study that trains MAE, DINO, and Toto variants on the identical data mixture at identical scale with matched evaluation protocols would be needed to determine the marginal contribution of the autoregressive objective to representation quality.

---

All Scaling Analysis and Most Ablations Use Only ImageNet Classification as the Downstream Metric

The assumption or constraint. The design choice ablations that form the paper's prescriptive recommendations—which tokenizer to use (Table 3), which resolution strategy to adopt (Table 4), which pooling method to apply (Table 5), which architecture to choose (Table 6), and which layer to probe (Figure 4)—are all evaluated on a single downstream task: ImageNet-1K linear or attention probing classification accuracy. The paper then applies these design choices uniformly to all other downstream tasks (Kinetics, Ego4D, DAVIS, CATER, robotics), assuming that what is optimal for ImageNet classification is also optimal for action recognition, tracking, forecasting, and motor control.

The consequence. This assumption is plausible but unverified, and there is reason to doubt it. The paper's own data show task-specific deviations from the ImageNet-derived heuristics. The optimal probing layer for robot manipulation (Figure 8, right column) does not follow the same mid-layer peak as classification—late layers are comparably effective for picking tasks. The paper interprets this as robot control benefiting from "generative" features in late layers, implying that the optimal layer is task-dependent. Similarly, the DAVIS tracking results (Table 10) show that resolution matters enormously for dense correspondence (+17.6 J&F points from 256 to 512 resolution), while the ImageNet ablations suggested a +3.2 point gain from resolution increase (Table 4, 61.2% vs. 64.4%)—the effect size is task-dependent. The architectural comparisons (LLaMA vs. GPT2 vs. Mamba) were evaluated only on ImageNet; it is unknown whether the LLaMA advantage holds for tasks like tracking or forecasting that make different demands on positional understanding and long-range context integration.

A practitioner using this paper to guide their own autoregressive video pre-training setup would adopt the tokenizer, architecture, resolution strategy, pooling method, and probing layer that maximize ImageNet accuracy. If their target task is, say, video object tracking or robotic manipulation, this recipe may be suboptimal—they might benefit from a different architecture that excels at fine-grained spatial correspondence or a different layer that retains more predictive features. The paper provides no guidance for navigating this task-dependence.

What evidence exists in the paper. All ablations in Section 4.1 (Tables 3–6) report only ImageNet top-1 accuracy. The probing layer analysis is extended to multiple tasks in Figure 8, which partially validates the mid-layer heuristic across tasks but also reveals exceptions (robotics). The resolution effect is shown on both ImageNet (Table 4, +3.2 points from 128 to 256) and DAVIS (Table 10, +17.6 J&F from 256 to 512), demonstrating task-dependence but not evaluating whether the ImageNet-derived resolution fine-tuning strategy (1 epoch at higher resolution with RoPE base 50K) is also optimal for tracking. No ablation is re-run on Kinetics, Ego4D, CATER, or robotics to verify transfer.

Mitigation status. The paper does not acknowledge this as a limitation. It presents the ImageNet-derived design choices as general recommendations (e.g., "attention pooling performs much better than average pooling" with no task qualification in Section 4.1) and applies them uniformly across all tasks without discussion. The cross-task probing layer analysis in Figure 8 is a step toward validation but does not extend to other design dimensions (tokenizer, architecture, resolution strategy). A more comprehensive study would ablate key design choices on at least one non-classification task (e.g., DAVIS tracking) to establish whether the heuristics transfer.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not introduce a new method that beats state-of-the-art on any benchmark. It is not a paradigm shift in the sense that it proposes a fundamentally new way to learn visual representations. What it does—and this is a genuine contribution—is establish autoregressive pre-training from video tokens as a viable, general-purpose representation learning strategy with predictable scaling behavior, after years where the field had largely written off pure autoregressive modeling for vision in favor of masked autoencoding and contrastive objectives.

The shift is primarily diagnostic and baseline-setting. Before this paper, a researcher deciding how to pre-train a vision model had strong evidence that masked modeling (MAE, VideoMAE) works, that contrastive methods (SimCLR, DINO) work, and that joint-embedding methods work—but essentially no large-scale evidence about whether simply predicting the next visual token, with no masking, no negative pairs, no augmentation invariance, and no reconstruction bottleneck, could produce representations that transfer broadly. The few prior data points (iGPT, AIM) were either pixel-level (and thus computationally crippled), supervised-filtered (AIM used CLIP-filtered data), or evaluated only on ImageNet classification. This paper fills that gap with a systematic empirical characterization across six task categories, three model scales, and multiple design axes.

The paper also resolves a latent tension in the literature that is more subtle than the explicit contradictions often addressed in methods papers. On one side, the language community had demonstrated that autoregressive pre-training at scale produces remarkably general representations (GPT-3, LLaMA) with minimal inductive bias. On the other side, the vision community had converged on methods that build in strong assumptions—masking to create information bottlenecks (MAE), augmentation invariance (SimCLR), or multi-view consistency (DINO). The implicit narrative was that vision data is different: spatial redundancy, lack of clear sequential structure, and the difficulty of defining "next" in 2D meant that the language recipe would not transfer. This paper's results partially falsify that narrative. Autoregressive pre-training does transfer—not perfectly, and with important caveats, but it produces representations that are competitive across a remarkably broad evaluation suite. The "minimal inductive bias" hypothesis holds up better than a skeptical reader in 2023 would have predicted.

A second, more specific shift is the paper's reconceptualization of video redundancy from a fatal flaw to a measurable scaling inefficiency. The finding that video frames after the first are substantially easier to predict (Figure 10, Appendix A.1) is not new—anyone who has watched consecutive video frames knows they are redundant. But the paper's contribution is to quantify this redundancy's effect on scaling behavior: the power-law exponent of −0.0378 for Toto versus −0.048 for GPT-3. This gives the field a metric for evaluating pre-training objective quality, not just a complaint about video data. A future method that reduces redundancy—through smarter frame sampling, masking, or token reweighting—can be evaluated by whether it steepens the scaling curve, not just by whether it improves a downstream benchmark. This reframes video pre-training research around information-theoretic efficiency rather than benchmark chasing, which is a healthier scientific framing.

The paper also establishes several architectural and procedural heuristics that lower the barrier to entry for future work on autoregressive vision models. The finding that middle layers contain the best representations (Figures 4, 8), that attention pooling substantially outperforms average pooling (Table 5, +7.9 points), that RoPE enables resolution generalization (Table 4), and that LLaMA architecture outperforms GPT2 and Mamba (Table 6) are not individually earth-shattering, but collectively they provide a recipe that future researchers can adopt without re-discovering these lessons. Without this paper, a team starting autoregressive video pre-training would likely probe the final layer with average pooling, use a weaker architecture, and train at fixed resolution—and then conclude the approach does not work well, when in fact they were evaluating it suboptimally.

Finally, the paper expands the evaluation landscape for self-supervised video models. By including tracking (DAVIS), action anticipation (Ego4D), object permanence (CATER), and robotics alongside the standard ImageNet/Kinetics benchmarks, the paper implicitly argues that representation quality should be measured by transfer breadth, not just classification accuracy. Toto-1b's 75.3% on ImageNet is not competitive with DINOv2's 86.4%, but Toto-large's 62.4 J&F on DAVIS tracking at 512 resolution exceeds DINO-base's 54.3. These results suggest that classification accuracy is an incomplete and potentially misleading measure of representation quality—a model that excels at category-level semantic abstraction (DINO) may underperform on instance-level correspondence (tracking) relative to a model trained on a more generic objective. This is a useful corrective to the field's ImageNet-centric evaluation culture, though the paper does not make this argument explicitly.

Follow-Up Research This Work Enables

Controlled comparison of autoregressive vs. masked pre-training on identical data and architecture. The single most important missing experiment in this paper is a direct head-to-head between autoregressive (Toto) and masked (MAE-style) pre-training using the same LLaMA architecture, the same dVAE tokenizer, and—crucially—the same 100K-hour video + image data mixture. The paper reports Toto results and compares them to published MAE/VideoMAE numbers from prior work (Tables 7, 8, 9), but these comparisons confound the pre-training objective with differences in data, architecture, and evaluation protocol. A clean ablation would train two 280M-parameter LLaMA models on the identical tokenized dataset: one with causal next-token prediction (Toto), one with random token masking and reconstruction (MAE-style, but in the same decoder-only architecture with causal masking removed during pre-training). Both would be evaluated with identical attention probing at their respective optimal layers on ImageNet, Kinetics, DAVIS tracking, and CATER. The outcome would definitively answer: does the autoregressive objective provide any advantage over masked modeling for representation learning, or are Toto's competitive results entirely attributable to data scale, architecture improvements (LLaMA vs. ViT), and better probing methods? The paper's DAVIS tracking result (62.4 J&F exceeding DINO) hints that autoregressive pre-training might have an edge on dense correspondence tasks—but without a controlled comparison, this remains speculation.

Token-level difficulty weighting to steepen the scaling curve. The paper identifies video frame redundancy as a likely cause of the shallow scaling exponent (−0.0378 vs. −0.048 for text) but does not attempt to fix it. A natural next experiment is to modify the pre-training loss to weight tokens by their empirical difficulty: compute per-position loss statistics on a held-out set, identify that tokens in the first frame (or at frame boundaries, or after scene cuts) are substantially harder to predict than tokens deep inside homogeneous temporal regions, and apply inverse-difficulty or uncertainty-based weighting during training. Concretely, after pre-training a small reference model, one could estimate the variance or entropy of the predicted distribution at each spatial-temporal position, then train a larger model with loss weights inversely proportional to predictability. The success metric would not be another ImageNet point, but rather the scaling exponent itself—if token reweighting steepens the power-law slope from −0.0378 to, say, −0.045, that would be a genuine improvement in learning efficiency per unit of compute, directly addressing the paper's central diagnosed limitation.

Resolution scaling laws for downstream tasks, especially tracking and robotics. The paper demonstrates that resolution substantially improves both ImageNet classification (Table 4, +3.2 points from 128 to 256) and DAVIS tracking (Table 10, +17.6 J&F from 256 to 512 for Toto-large), but only at two data points per task. A systematic study would pre-train Toto models at multiple base resolutions (e.g., 64, 128, 192, 256 pixels), fine-tune each to a range of higher resolutions using RoPE frequency adaptation, and measure downstream performance on ImageNet, Kinetics, DAVIS, and a robotics task across the full resolution grid. This would produce resolution scaling laws analogous to the compute scaling laws in Figure 9—is there a power-law relationship between pixel resolution and downstream accuracy? Does it saturate? Is the optimal pre-training resolution different for semantic tasks (classification) versus dense tasks (tracking, manipulation)? The paper's finding that 128→256 with 1 epoch of fine-tuning outperforms full 256 pre-training (Table 4, 64.4% vs. 61.2%) is a single surprising data point that demands systematic replication and extension.

Video frame sampling as a trainable or learned policy. The paper uses a fixed stride of 4 frames throughout all pre-training and evaluation, with no ablation of sampling rate or strategy. Given the finding that the first frame carries disproportionate information (Figure 10), a fixed stride likely samples many redundant frames. A follow-up would treat frame sampling as an optimizable component of the pre-training pipeline. One concrete experiment: during pre-training, dynamically select frames based on motion magnitude, optical flow, or the model's own prediction uncertainty—sampling more frames during high-motion segments and fewer during static scenes—and measure whether this improves downstream transfer at fixed token budget. Alternatively, one could train a lightweight "frame importance" predictor (a small convolutional network or even a heuristic based on frame-to-frame feature dissimilarity from a frozen reference model) and use it to subsample videos before tokenization. The metric is downstream task performance at equivalent total training tokens, with the hypothesis that information-weighted frame sampling recovers some of the scaling efficiency lost to redundancy.

Generation quality evaluation and the representation-generation tradeoff. The paper evaluates only on recognition, tracking, and control tasks—all of which use the model as a frozen feature extractor or fine-tuned classifier. Yet the model is trained on a generative objective (next-token prediction), and the paper briefly demonstrates generation in Appendix A.8 ("periodic motion," "object permanence," "3D rotation") without any quantitative metrics. A critical follow-up would evaluate Toto models on standard video generation benchmarks (e.g., FVD on UCF-101, FID on frame generation, or human preference studies) and ask: is there a tradeoff between representation quality and generation quality? The paper's finding that middle layers (not final layers) contain the best representations for recognition suggests that the features optimized for generation (the final layer) may be suboptimal for discrimination, and vice versa. A systematic study would measure generation quality (FVD, IS, human evaluation) and representation quality (ImageNet probing, DAVIS tracking) across model scales, training duration, and data mixtures, looking for correlations or anti-correlations. If generation quality and representation quality are positively correlated, the autoregressive objective is genuinely multitask. If they trade off (improving generation degrades representation, or vice versa), that would motivate hybrid training objectives that balance both.

Hybrid autoregressive-masked video pre-training objectives. The paper studies autoregressive pre-training in isolation from masked modeling, but the two objectives are complementary: masking creates an artificial information bottleneck that forces the model to learn semantic structure, while autoregressive prediction leverages natural temporal structure. A hybrid approach could, for example, randomly mask some fraction of tokens within each frame (as in MAE) while keeping the temporal ordering autoregressive (future frames can attend to present, present cannot attend to future). Or, it could use autoregressive prediction for the first few frames (where prediction is hardest and most informative) and switch to a masked reconstruction objective for later frames (where redundancy makes prediction trivial). The paper's own Appendix A.1 observes that the first frame has much higher loss than subsequent frames, suggesting that a non-uniform objective—harder for frame 1, easier for later frames—might be more efficient. Such a hybrid would need to beat both pure autoregressive (Toto) and pure masked (VideoMAE) baselines on the same data and architecture at matched compute. The failure mode—finding that hybrid objectives do not outperform the better of the two pure objectives—would itself be informative about whether the benefits of each objective are additive or redundant.

Practical Applications and Downstream Use Cases

On-device video understanding with frozen features for tracking and AR. The DAVIS tracking results (Table 10) are the paper's strongest practical signal. Toto-large at 512 resolution achieves 62.4 J&F using frozen features with no fine-tuning and no task-specific training—nearest-neighbor label propagation is the entire algorithm. This is directly applicable to augmented reality and video editing scenarios: a user annotates an object in one frame, and the system propagates the mask through the video without any per-video training. The fact that Toto-base at 256 resolution achieves 42.0 J&F on a 120M-parameter model suggests this approach is feasible on mobile devices, where running a 120M-parameter transformer for feature extraction is plausible given modern on-device ML accelerators. A deployment scenario would pre-compute Toto features for each frame of a user's video on-device, then run nearest-neighbor propagation at interactive rates for mask tracking, enabling applications like object removal, background replacement, or gaze-guided highlighting without sending video to the cloud. The resolution scaling result (512 resolution dramatically outperforms 256) suggests that for maximum quality, features could be extracted at higher resolution when latency permits, with the RoPE fine-tuning strategy enabling this adaptation without retraining.

Universal visual backbone for robot learning across embodiments. The simulation robotics results (Figure 6) show that Toto-base, frozen and used as a feature extractor with a single learned linear layer, enables DAgger policies to learn four manipulation tasks (across two robot platforms and two task types) faster than a MAE baseline that was trained on robot-relevant data (including 100DOH hand-object interactions). This matters because the dominant bottleneck in robot learning is sample efficiency—collecting real robot data is expensive and slow. A frozen visual backbone that transfers zero-shot to new robots, new camera viewpoints, and new tasks without any domain-specific fine-tuning reduces the per-task data requirement and enables rapid prototyping. The competitive real-world result on Franka cube-picking (63% success rate, Table 11), while preliminary (16 trials, single task), suggests the simulation transfer is not purely a sim-to-sim artifact. A practical deployment would pre-train Toto once on a large, diverse video corpus (which can be continuously expanded as more internet video becomes available), then freeze it as the vision module for any new robot manipulation task, training only a lightweight policy head on task-specific demonstrations. The paper's finding that Toto outperforms a robotics-specific pre-training method (MVP) on simulation tasks without using any robot data in pre-training is the key evidence that general video pre-training can substitute for domain-specific pre-training in embodied settings—potentially eliminating the need for expensive robot data collection during the pre-training phase.

Cost-efficient pre-training for video understanding startups and research labs. The resolution fine-tuning strategy (Table 4) changes the economics of video model pre-training. Pre-training a model at 128×128 resolution (16×16 tokens per frame) costs approximately one-quarter the compute of pre-training at 256×256 (32×32 tokens), and a single epoch of high-resolution fine-tuning with increased RoPE frequency yields better accuracy than full-resolution pre-training from scratch (64.4% vs. 61.2%) at ~37% of the total compute. For a startup or academic lab with a limited GPU budget, this means they can pre-train a competitive video model for roughly one-third the cost they would have budgeted using the naive approach of "train at target resolution." The implication is not just cost savings but accessibility: a team that could previously afford to train only a 120M-parameter model at full resolution might instead train a 280M-parameter model at low resolution and fine-tune it to high resolution for the same compute budget, getting the benefits of scale without the full cost. Combined with the LLaMA architecture (which is well-supported in open-source frameworks and has efficient inference implementations), this makes autoregressive video pre-training more practical for resource-constrained settings than the headline "1 trillion tokens" might suggest.