Timer-S1: A Billion-Scale Time Series Foundation Model with Serial Scaling

🎯 Pitch

Time series foundation models have resisted the billion-parameter scaling that revolutionized NLP—until now. Timer-S1 shatters this ceiling by pairing a 8.3B-parameter Mixture-of-Experts architecture with a novel “Serial-Token Prediction” objective that mimics the serial nature of forecasting, avoiding the deadly error accumulation of standard next-token approaches. The result is a pretrained model that claims the top MASE and CRPS spots on the large-scale GIFT-Eval leaderboard, dropping forecasting error by 7.6% over its predecessor on a trillion-timestep corpus.

1. Executive Summary

This paper introduces Timer-S1, a billion-scale Mixture-of-Experts (MoE) time series foundation model with 8.3B total parameters (0.75B activated per token) that addresses scalability bottlenecks through what the authors term Serial Scaling across three dimensions—model architecture, dataset, and training pipeline. The core architectural innovation is Serial-Token Prediction (STP), a generic training objective implemented via dedicated TimeSTP blocks that progressively refine predictions through serial computations across forecasting horizons, avoiding both the error accumulation of autoregressive rolling and the parallel-only limitations of multi-token prediction. Evaluated on the GIFT-Eval leaderboard—spanning 24 datasets, 144,000 time series, and 177 million data points—Timer-S1 achieves state-of-the-art performance as a pre-trained model (CRPS: 0.485, MASE: 0.693), with a 7.6% lower MASE and 13.2% lower CRPS than its predecessor Timer-3 (Sundial) when trained on the same trillion-point TimeBench corpus, establishing that respecting the serial nature of forecasting through dedicated architectural blocks yields substantial gains on medium- and long-term horizons while enabling adaptive inference depth.

2. Context and Motivation

The Core Problem: Time Series Foundation Models Have Hit a Scaling Ceiling

The central question this paper addresses is deceptively straightforward: why haven't time series foundation models scaled to the billion-parameter level that has become standard in language and vision, and what architectural change would unlock that scaling? While large language models routinely deploy hundreds of billions of parameters with clear scaling laws relating compute, data, and performance, the time series domain has remained stubbornly stuck at substantially smaller scales. The paper's timeline in Figure 3 makes this visible: across the field's rapid recent evolution—from Timer through Chronos, Moment, Moirai, TimesFM, and others—parameter counts have largely plateaued well below the billion-parameter threshold. Timer-S1's 8.3B total parameters represents a deliberate attempt to break through this ceiling.

This gap is not merely an academic curiosity. Time series data underpins critical decisions across finance (portfolio allocation, risk assessment), industrial operations (predictive maintenance, supply chain optimization), energy (load forecasting, grid management), healthcare (patient monitoring, epidemic tracking), and climate science (extreme weather prediction). In each of these domains, more accurate long-term forecasts translate directly into economic value and societal benefit. The inability to scale time series models means that these applications cannot benefit from the same scaling-driven performance improvements that have transformed natural language processing.

Why Scaling Time Series Models Is Fundamentally Harder

The paper identifies several structural properties of time series data that make naive scaling ineffective, and understanding these is essential to appreciating why Timer-S1's approach is motivated:

Distributional heterogeneity across domains. Unlike natural language, which follows relatively stable grammatical rules and draws from shared world knowledge, time series from different domains exhibit fundamentally different statistical properties. A financial tick series (irregularly sampled, driven by market microstructure noise), an IoT sensor stream (high-frequency, periodic, with sensor drift), a meteorological record (seasonal, with long-range spatial dependencies), and an ECG waveform (quasi-periodic, with specific morphological features) share almost no common structure beyond being ordered sequences. This extreme heterogeneity means that a model architecture or training objective that works well for one domain may fail catastrophically for another—a problem the MoE architecture in Timer-S1 is explicitly designed to address through adaptive expert selection per token.

Multi-scale and multi-dimensional dependencies. Real-world time series exhibit patterns at multiple temporal scales simultaneously: a retail sales series contains intra-day patterns, weekly seasonality, monthly cycles, and annual trends superimposed on each other. Capturing these nested dependencies requires the model to simultaneously attend to immediate neighbors (short-range correlations), distant historical points (seasonal patterns), and intermediate contexts (trend estimation). Furthermore, multivariate time series introduce cross-variate interactions—how does temperature affect electricity demand?—that add an exponential dimension of complexity. The paper's focus on univariate pre-training with serial computations is a deliberate design choice to master temporal dependencies before addressing cross-variate structure.

Non-stationarity and stochasticity. Unlike images (where a cat is always a cat) or text (where word meanings are relatively stable), time series distributions can shift abruptly due to external shocks—a pandemic disrupting transportation patterns, a regulatory change altering market dynamics, a sensor recalibration changing measurement characteristics. This non-stationarity means that a model trained on historical patterns may face data at inference time drawn from a meaningfully different distribution. The paper cites this as a key reason that standard next-token prediction—which implicitly assumes the data-generating process is stationary—can fail for time series, and why Timer-S1's architecture includes explicit mechanisms (re-normalization, adaptive serial computation) to handle distributional shifts.

The serial nature of forecasting. This is perhaps the paper's most important motivating insight, and it warrants careful explanation. When one forecasts 100 steps into the future, the prediction at step 100 depends on the prediction at step 99, which depends on step 98, and so on. Forecasting is therefore an inherently serial computation—uncertainty compounds multiplicatively across the horizon, and errors at earlier steps cascade forward. The paper visualizes this in Figure 1: parallel-forecasting models that predict all future steps simultaneously from the same shared representation lack the serial computations needed to model this error propagation structure. Autoregressive models that predict step-by-step do capture the serial nature, but at the cost of requiring $H$ sequential forward passes through the entire model for an $H$ -step forecast, each pass compounding the errors from the previous one.

This frame—that scaling must respect the serial nature while avoiding autoregressive costs—motivates the entire STP architecture. The goal is not simply to make a bigger model, but to design an architecture where longer-horizon predictions undergo strictly more serial computation, mirroring the structure of the problem itself.

Where Prior Approaches Fall Short

The paper situates its contributions against four categories of prior work, each of which addresses some aspect of the scaling problem but falls short on others.

Decoder-only Transformers with next-token prediction. The Timer family's own lineage—Timer, Timer-XL, and other decoder-only architectures—adopted the standard language modeling recipe: a causal Transformer trained to predict the next patch given all previous patches. This approach has several virtues. It respects the temporal ordering of data through causal masking, it enables unified pre-training across diverse datasets by treating everything as a sequence, and it has a proven scaling track record in language. However, for time series forecasting specifically, next-token prediction forces an uncomfortable choice at inference time. To generate a 100-step forecast, the model must be rolled out autoregressively: predict step 1, append it to the input, predict step 2, and so on through 100 sequential forward passes. Each pass incurs the full computational cost of the model, and—critically—each step's prediction becomes the input for the next, meaning any errors in early predictions contaminate all subsequent ones. The paper identifies this error accumulation as a key scaling bottleneck: as models get larger and forecasts get longer, the gap between what the model could predict with clean inputs and what it does predict with its own noisy outputs grows wider.

Multi-token prediction approaches. A natural solution to the autoregressive bottleneck is multi-token prediction: instead of predicting one patch at a time, predict multiple future patches simultaneously. This has been adopted in both language models (DeepSeek, Gloeckle et al.) and time series models (Timer-3/Sundial, Moirai). The training objective forces the model's internal representations to be useful for predicting at multiple horizons simultaneously, which can improve representation learning. However, the paper argues that this approach misses something fundamental. When all future steps are predicted in parallel from a single shared representation, the model lacks any mechanism for step 50's prediction to explicitly depend on step 49's predicted value—the serial dependency that characterizes real forecasting is collapsed into a single feedforward computation. The paper's key claim is that this parallel-only approach imposes a scaling bottleneck: adding more parameters or more data cannot compensate for the missing serial computations, because the model architecture itself prevents the kind of progressive reasoning that long-term forecasting requires.

Specialized architectures for time series (non-foundation models). Before the foundation model paradigm took hold, the field developed a rich set of architectures tailored for specific forecasting challenges: TCNs for capturing long-range temporal dependencies, LSTMs and GRUs for sequential modeling with gating, and numerous Transformer variants with custom attention mechanisms (Informer, Autoformer, FEDformer, PatchTST, and many others). These models achieve strong results on individual benchmarks but are typically trained from scratch per dataset, making them inapplicable to the cold-start, data-scarce scenarios that motivate foundation models. The paper does not position Timer-S1 as competing with these specialized approaches on individual datasets, but rather as advancing the foundation model paradigm to a scale where it can match or exceed specialized models while retaining zero-shot generality.

Prior attempts at scaled time series foundation models. The paper directly references Moirai and Chronos as prior attempts to scale time series models that "may lead to inferior performance or fail to achieve a significant breakthrough in model scale" (Section 2). Moirai introduced a masked encoder architecture with variable-sized patches, while Chronos tokenized time series into discrete bins and applied language model training. Both represent serious scaling efforts, but the paper implies—without detailed head-to-head ablations—that their performance did not improve proportionally with model size, suggesting that architectural choices, not just parameter count, determine whether scaling yields benefits. This is the gap Timer-S1 aims to fill: an architecture explicitly designed so that additional parameters translate into additional serial computation where it matters most.

The Central Insight: Serial Computation as a First-Class Architectural Primitive

The paper's motivating insight—and the intellectual thread connecting its architectural, data, and training innovations—is that the amount of serial computation should scale with the forecasting horizon. Short-term predictions require less serial processing because there is less uncertainty to compound; long-term predictions require more because each step must explicitly condition on the evolving predictions that precede it.

This insight leads to a clean architectural decomposition. The TimeMoE blocks (the "main" blocks) extract a rich, contextualized representation of the historical input—this is the shared computation that benefits all forecasting horizons equally. The TimeSTP blocks then perform horizon-specific serial computation: the first STP block produces the one-step-ahead prediction, the second STP block takes that prediction's embedding and produces the two-step-ahead prediction, and so on. Each additional block adds exactly one more round of serial computation, mirroring the one-step error propagation in real forecasting.

The elegance of this design is that it simultaneously solves three problems. It avoids the error accumulation of autoregressive rollout (because predictions are generated in a single forward pass, each STP block receiving the previous block's internal representation rather than its discretized output). It avoids the serial-computation deficiency of parallel multi-token prediction (because deeper predictions undergo strictly more Transformer blocks). And it avoids the computational waste of fixed-depth models (because inference depth can be adapted to the required horizon—predicting 10 steps requires only 10 STP blocks, not all 16).

The Data and Training Gaps

Beyond architecture, the paper motivates its data and training contributions by identifying two additional scaling bottlenecks.

Predictive bias from imbalanced training data. Real-world time series datasets are not uniformly distributed across domains, frequencies, or trend characteristics. A model trained predominantly on trending financial data may learn to always predict upward drift, performing poorly on stationary or mean-reverting signals. The paper's value-flipping augmentation—multiplying both input and output by −1 to invert trends while preserving temporal structure—directly counteracts this specific failure mode by forcing the model to learn patterns that are sign-agnostic.

The short-term/long-term training conflict. Single-stage pre-training treats all forecasting horizons equally, but short-term and long-term forecasting make different demands on the model. Short-term forecasting benefits from precise local pattern matching; long-term forecasting requires capturing broader structural regularities but tolerates higher variance. The paper argues that optimizing both simultaneously in a single training stage creates conflicting gradients—improving short-term accuracy may come at the cost of long-term representations, or vice versa. The post-training stage with its horizon-decaying weighted STP objective ( $1/\sqrt{j}$ decay) is designed to resolve this conflict: first learn representations that serve all horizons, then fine-tune with emphasis on the short-term predictions that underpin long-term accuracy.

Context length as a scaling dimension. Many time series exhibit long-range dependencies (annual seasonality in monthly data requires looking back 12+ periods; business cycle effects may span years). A foundation model with insufficient context length is fundamentally limited in what patterns it can capture, regardless of parameter count. The paper's context extension from 2,880 to 11,520 time points during post-training is motivated by this observation, and is implemented via RoPE-based position interpolation rather than architectural changes—a pragmatic choice that enables longer context without retraining from scratch.

How Timer-S1 Positions Itself

The paper explicitly frames itself not as proposing a single novel method, but as systematic serial scaling across three interdependent dimensions:

Architecture: TimeMoE for heterogeneous representation learning + TimeSTP for horizon-dependent serial computation
Data: TimeBench (1 trillion points) with augmentation addressing frequency and directional bias
Training: Multi-stage pipeline decoupling general representation learning from horizon-specific optimization

The paper positions its approach as a natural evolution of the Timer family (Figure 3), with each generation addressing a specific limitation: Timer introduced unified pre-training via decoder-only architecture, Timer-XL added structured attention for multi-dimensional data, Timer-3 introduced generative forecasting via flow matching, and Timer-S1 breaks the scaling bottleneck by making serial computation a first-class architectural primitive.

The paper also positions itself relative to the broader foundation model literature by drawing explicit—and deliberately incomplete—parallels to language models. The analogy to next-token prediction is acknowledged, but the paper argues that direct transplantation fails because (a) time series have stronger non-stationarity, making autoregressive error accumulation more severe, and (b) the availability of future ground truth during training creates a train-test gap for multi-token prediction that doesn't exist in the same way for language. The LLM practice of discarding auxiliary prediction heads after training (as in DeepSeek's MTP) is explicitly rejected for time series because the train-test distribution gap is too severe to ignore—a design choice validated in the ablation comparing Timer-S1 against Shift-Token and Remove-STP variants (Figure 15).

3. Technical Approach

3.1 Reader Orientation

Timer-S1 is a time series forecasting system—a single pre-trained neural network that, given a historical sequence of numerical values, predicts a probabilistic distribution over future values without any task-specific training. The system solves the long-standing problem that time series foundation models have failed to scale to billion-parameter sizes with proportional performance gains, by introducing an architecture where the depth of computation applied to each predicted future step is proportional to how far into the future that step lies—short-term predictions get less processing, long-term predictions get more, mirroring the compounding-uncertainty structure of real forecasting.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major stages through which a raw time series flows:

Re-Normalization and Patch Embedding — takes a raw univariate time series, normalizes it to zero mean and unit variance using only input statistics (so the absolute scale doesn't matter), and splits it into fixed-length patch tokens that serve as the "words" of the Transformer vocabulary. These tokens carry both the normalized values and a binary mask indicating padding positions.
TimeMoE Encoder (Main Blocks) — a stack of 24 decoder-only Transformer blocks, each containing causal multi-head self-attention with Rotary Position Embedding (RoPE) and a sparse Mixture-of-Experts (MoE) feedforward layer. This stage processes all input patches simultaneously, producing a rich, context-aware representation of the entire history. Crucially, this computation is shared across all forecasting horizons—it runs once per input sequence regardless of how many future steps are needed.
TimeSTP Decoder (Serial Prediction Blocks) — a stack of 16 specialized Transformer blocks that perform horizon-specific serial computation. The first TimeSTP block takes the final hidden state from the main blocks, fuses it with the original raw patch embeddings, passes it through another TimeMoE module, and projects the result to predict the one-step-ahead future patch. The second TimeSTP block takes that refined representation, fuses it again with the original embeddings, and predicts two steps ahead. This continues through all 16 blocks, so the 16-step-ahead prediction has undergone 16 more rounds of serial processing than the one-step-ahead prediction—exactly mirroring the error-propagation structure of real forecasting.
Shared Quantile Forecasting Head — a single projection layer (PatchProject) applied identically to the output of every TimeMoE and TimeSTP block. It maps each token embedding to $Q = 9$ quantile predictions (the 10th through 90th percentiles), each spanning a full patch of $P = 16$ time points. Because the head is shared across all blocks, the model learns a consistent embedding-to-forecast mapping that is supervised densely at every horizon during training.
De-Normalization — applies the inverse of the input normalization (using the saved mean $\mu$ and standard deviation $\sigma$ ) to restore the original data scale to all predicted quantiles, ensuring the final output is directly interpretable in the original units.

Information flow in one sentence: A raw univariate series enters → normalized and patched → encoded by 24 shared TimeMoE blocks → progressively refined through 16 horizon-indexed TimeSTP blocks, each producing one quantile patch prediction → all predictions de-normalized to original scale → a full probabilistic forecast spanning 272 time points emerges from a single forward pass.

3.3 Roadmap for the Deep Dive

First, the normalization and embedding pipeline (Section 3.1), because every subsequent component operates on the tokenized, normalized representation and understanding this step explains why the model is robust to varying data scales and frequencies.
Second, the TimeMoE encoder (Section 3.2, first half), which is the shared computation engine—this is where the Mixture-of-Experts mechanism addresses domain heterogeneity and where the causal attention mechanism captures temporal dependencies across the input.
Third, the TimeSTP serial prediction mechanism (Section 3.2, second half), which is the core architectural innovation—this is where the horizon-dependent serial computations happen, and understanding it requires knowing what representations it receives from TimeMoE.
Fourth, the quantile forecasting head and loss function (Section 3.3), which defines what the model actually predicts and how it is trained—this connects the architectural output to the evaluation metrics (MASE, CRPS) on GIFT-Eval.
Fifth, the data curation and augmentation pipeline (Section 4.1), which is essential to understanding why the architecture works—the trillion-point TimeBench corpus and the specific augmentations that counteract predictive bias.
Sixth, the multi-stage training pipeline—pre-training, continued pre-training with weighted STP, and context extension (Sections 4.2–4.3)—because the training strategy is as important as the architecture for achieving the reported results, and the decoupling of general representation learning from horizon-specific fine-tuning is a key methodological contribution.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an architectural innovation paper whose core idea is that time series forecasting is an inherently serial problem—long-horizon accuracy depends on progressive step-by-step reasoning—and that an architecture which allocates strictly more computation to longer-horizon predictions will scale better than either parallel-only multi-token prediction or autoregressive next-token prediction. The technical approach realizes this idea through a decoder-only Transformer where the main body extracts shared representations and specialized TimeSTP blocks perform horizon-indexed serial refinement, trained on a trillion-point corpus with a multi-stage objective that decouples general representation learning from short-term accuracy optimization.

Re-Normalization: Making the Model Scale-Agnostic

Time series from different domains can differ in absolute magnitude by orders of magnitude—a stock price in the hundreds of dollars, an ECG voltage in millivolts, a temperature in tens of degrees. If a model is trained on raw values, it must learn to handle these scale discrepancies internally, which wastes representational capacity and creates brittle dependencies on input range. Timer-S1 addresses this through instance re-normalization: for each univariate input sequence independently, the model computes its own mean and standard deviation, normalizes to zero mean and unit variance, performs all forecasting in this normalized space, and then de-normalizes the predictions back to the original scale.

Specifically, for an input sequence of $T$ time points $\{x_1, \dots, x_T\}$ :

$\mu = \frac{1}{T} \sum_{t=1}^{T} x_t, \quad \sigma^2 = \frac{1}{T} \sum_{t=1}^{T} (x_t - \mu)^2, \quad \tilde{x}_t = \frac{x_t - \mu}{\sigma}, \quad t = 1, \dots, T$

where $\mu$ is the empirical mean of the input window (not the full dataset), $\sigma$ is the empirical standard deviation of the input window, and $\tilde{x}_t$ is the normalized value at time $t$ .

What it computes: for each input sequence independently, the empirical mean and standard deviation of that specific window of $T$ points, then a z-score normalization of every point in the window. The same $\mu$ and $\sigma$ are saved and later used to invert the normalization on the predictions:

$\hat{x}_t = \sigma \cdot \tilde{x}_t + \mu, \quad t = T+1, \dots, T+F$

where $\tilde{x}_t$ is the model's raw normalized prediction for time $t$ , and $\hat{x}_t$ is the final output in the original data scale.

Why this form: instance normalization (computed per input window rather than globally across the dataset) ensures that the model learns purely local temporal patterns—shapes, trends, seasonality—independent of absolute magnitude. A sine wave of amplitude 0.1 and a sine wave of amplitude 1000 become identical after this transformation, so the model only needs to learn the sinusoidal pattern once. The alternative—global normalization using dataset-wide statistics—would couple the model's internal representations to the specific datasets seen during training, degrading zero-shot generalization to new domains with different typical scales. The paper states this enables the model to "concentrate on learning local temporal patterns" by removing value-shifting and scaling as confounding factors.

A subtle detail: the normalization uses only input statistics ( $\mu$ and $\sigma$ are computed from $\{x_1, \dots, x_T\}$ only, not from future values), which is critical because during inference the future is unavailable. The de-normalization step then assumes that the future sequence will have the same mean and standard deviation as the recent past—a reasonable local-stationarity assumption that holds for most short-to-medium forecasting horizons but could fail during regime changes.

Patch Embedding: From Point Sequences to Token Sequences

Raw time series operate at the level of individual time points, but Transformers operate on token sequences. A naive approach would embed each time point as a separate token, but this creates sequence lengths proportional to the number of time points (potentially thousands), making self-attention—which scales quadratically in sequence length—prohibitively expensive. Timer-S1 instead adopts patch tokenization: consecutive time points are grouped into fixed-length patches, and each patch becomes a single token.

Formally, with patch length $P = 16$ , the $i$ -th patch is:

$\tilde{\mathbf{x}}_i = \{\tilde{x}_{1 + (i-1)P}, \dots, \tilde{x}_{iP}\}$

The full normalized input of length $T$ is divided into $N = \lceil T / P \rceil$ patches. When $T$ is not perfectly divisible by $P$ , left-padding is applied (zeros are prepended to the beginning of the sequence), and a binary mask $\mathbf{m}_i \in \mathbb{R}^P$ per patch explicitly marks which positions are real data (1) versus padding (0).

Each patch is embedded through a residual network that concatenates the normalized values with the mask and projects to dimension $D = 1024$ :

$\mathbf{h}_i^0 = \operatorname{PatchEmbed}\big(\operatorname{Concat}(\tilde{\mathbf{x}}_i, \mathbf{m}_i)\big), \quad i = 1, \dots, N$

where $\mathbf{h}_i^0 \in \mathbb{R}^{1024}$ is the initial embedding of the $i$ -th patch token, and $\operatorname{PatchEmbed}$ is a residual feedforward network mapping from $\mathbb{R}^{2P} = \mathbb{R}^{32}$ to $\mathbb{R}^{1024}$ .

What it computes: a 32-dimensional input vector (16 normalized time points concatenated with 16 mask bits) is projected through a learned nonlinear transformation to a 1024-dimensional dense embedding that serves as the patch's initial representation in the Transformer. The mask concatenation ensures the model knows which time points in the patch are real versus padded, preventing it from learning spurious patterns from padding zeros.

Why this form: the fixed patch size of $P = 16$ is a design tradeoff. Too small (e.g., $P = 1$ , single-point tokens) would create sequence lengths of 2,880 for the full context window, making self-attention too expensive and losing the local structure within patches. Too large would make patches too coarse-grained, reducing the temporal resolution of predictions (since the forecasting head predicts one quantile set per patch). The choice of 16 yields a maximum input of $N = 180$ tokens for context length $T = 2880$ and a maximum output of 17 patches (1 from the last main-block token + 16 from TimeSTP blocks) for a prediction horizon of $17 \times 16 = 272$ time points.

A crucial implementation detail: the same PatchEmbed layer is shared across all input patches and all blocks (both TimeMoE and TimeSTP receive the same initial embeddings $\mathbf{h}_i^0$ as conditioning). This parameter sharing enforces a consistent token vocabulary across the entire model.

TimeMoE Blocks: The Shared Representation Engine

The first $L = 24$ Transformer blocks of Timer-S1 form the main encoder—the shared computation that processes the entire input sequence once and produces a rich, contextualized representation of the historical data. These blocks are called TimeMoE because they combine standard causal self-attention with a sparse Mixture-of-Experts feedforward layer, where the MoE mechanism is specifically motivated by the domain heterogeneity of time series data.

Each TimeMoE block (indexed by $l = 1, \dots, 24$ ) consists of two sub-layers with Pre-RMSNorm residual connections:

$\mathbf{u}_i^l = \operatorname{MHA}\big(\operatorname{RMSNorm}(\mathbf{h}_i^{l-1})\big) + \mathbf{h}_i^{l-1}$ $\mathbf{h}_i^l = \operatorname{MoE}\big(\operatorname{RMSNorm}(\mathbf{u}_i^l)\big) + \mathbf{u}_i^l$

where $\mathbf{h}_i^{l-1} \in \mathbb{R}^{1024}$ is the $i$ -th token embedding entering block $l$ , $\mathbf{u}_i^l$ is the intermediate embedding after multi-head attention but before the MoE, and $\mathbf{h}_i^l$ is the output embedding of block $l$ . Both sub-layers use Pre-RMSNorm (normalization applied before the sub-layer, not after), which improves training stability for very deep Transformers by preventing the residual stream from accumulating excessive variance.

The Multi-Head Attention Component. Timer-S1 uses standard causal (decoder-only) self-attention, meaning each patch token can only attend to itself and earlier tokens—never to future tokens. This causal masking is essential because during inference, future tokens (patches representing time points the model hasn't seen yet) are not available, and violating causality during training would create a train-test gap where the model learns to cheat by peeking at futures.

The attention mechanism incorporates two stabilizing modifications beyond standard Transformers:

QK-Norm. Before computing attention scores, the query and key vectors are $\ell_2$ -normalized:

$\hat{\mathbf{q}}_i = \frac{\mathbf{W}_{\mathbf{q}}^{\top} \mathbf{h}_i}{\|\mathbf{W}_{\mathbf{q}}^{\top} \mathbf{h}_i\|}, \quad \hat{\mathbf{k}}_i = \frac{\mathbf{W}_{\mathbf{k}}^{\top} \mathbf{h}_i}{\|\mathbf{W}_{\mathbf{k}}^{\top} \mathbf{h}_i\|}$

where $\mathbf{W}_{\mathbf{q}}, \mathbf{W}_{\mathbf{k}} \in \mathbb{R}^{1024 \times d}$ project patch embeddings to $d$ -dimensional queries and keys respectively, $\|\cdot\|$ denotes the $\ell_2$ norm, and $\hat{\mathbf{q}}_i, \hat{\mathbf{k}}_i \in \mathbb{R}^d$ are the normalized query and key vectors.

What this computes: each query and key vector is divided by its Euclidean length, constraining them to lie on the unit hypersphere. The dot product $\hat{\mathbf{q}}_i^{\top} \hat{\mathbf{k}}_j$ then becomes the cosine similarity between the two vectors, bounded in $[-1, 1]$ rather than potentially growing unbounded as sequence length increases.

Why this form: without QK-Norm, the softmax attention scores can saturate—as the dot products grow large in magnitude, the softmax becomes near-deterministic (attending almost exclusively to a single token), which prevents the model from learning distributed attention patterns. This saturation is a known issue identified in prior work on non-parametric attention. QK-Norm prevents saturation by bounding the logits, ensuring the softmax always produces a meaningful distribution over attended positions.

Rotary Position Embedding (RoPE). After normalization, the attention score between token $i$ and token $j$ is computed as:

$\mathcal{A}_{i,j} = \hat{\mathbf{q}}_i^{\top} \mathbf{R}_{\Theta, i-j} \hat{\mathbf{k}}_j$

where $\mathbf{R}_{\Theta, i-j} \in \mathbb{R}^{d \times d}$ is a rotation matrix parameterized by the relative position $(i-j)$ and a set of rotation frequencies $\Theta$ . The matrix applies a position-dependent rotation to the query and key vectors such that their dot product depends only on their relative distance, not their absolute positions.

A learnable scalar temperature $\tau$ scales the attention logits, and a triangular causal mask $\operatorname{Mask}(\cdot)$ sets $\mathcal{A}_{i,j} = -\infty$ for $j > i$ (preventing attention to future tokens):

$\operatorname{Attention}(\mathbf{H}) = \operatorname{Softmax}\big(\tau \cdot \operatorname{Mask}(\mathcal{A})\big) \cdot \mathbf{H}\mathbf{W}_{\mathbf{v}}$

where $\mathbf{W}_{\mathbf{v}} \in \mathbb{R}^{1024 \times d}$ projects embeddings to value vectors, and the softmax produces a distribution over the $N$ input tokens (only the non-masked, i.e., past and current, ones).

Why RoPE instead of learned absolute position embeddings: RoPE encodes position information directly into the attention computation through rotation, which has two key advantages for time series. First, it naturally handles variable sequence lengths—the model can process sequences longer than any seen during training because the rotation matrix is defined for any relative distance. This is critical for the context extension from 2,880 to 11,520 time points during post-training. Second, it encodes relative position (how far apart two patches are) rather than absolute position (where in the sequence a patch appears), which aligns better with time series where patterns repeat at characteristic intervals (e.g., daily, weekly) regardless of absolute position in the sequence.

After multi-head attention, all heads are concatenated and projected back to dimension 1024, added to the residual stream to produce $\mathbf{u}_i^l$ .

The Mixture-of-Experts Component. The feedforward sub-layer of each TimeMoE block is not a single dense network but a sparse Mixture-of-Experts with $E = 32$ total experts, of which only $K = 2$ are activated per token:

$\operatorname{MoE}(\mathbf{u}_i) = \sum_{j=1}^{E} g_{j,i} \cdot \operatorname{FFN}_j(\mathbf{u}_i)$

where $\operatorname{FFN}_j$ is the $j$ -th expert (an independent feedforward network), and $g_{j,i}$ is a gating weight that is non-zero only for the top- $K$ experts according to a learned router:

$g_{j,i} = \begin{cases} a_{j,i}, & \text{if } a_{j,i} \in \operatorname{Topk}(\{a_{j,i} \mid 1 \leq j \leq E\}, K) \\ 0, & \text{otherwise} \end{cases}$

$a_{j,i} = \operatorname{Softmax}_j(\mathbf{W}_j \mathbf{u}_i)$

where $\mathbf{W}_j \in \mathbb{R}^{1024}$ is a learned weight vector for expert $j$ (the router), $a_{j,i}$ is the softmax-normalized affinity between token $i$ and expert $j$ , and $\operatorname{Topk}(\cdot, K)$ selects the $K$ largest affinities.

What this computes: for each token independently, a learned router computes 32 affinity scores (one per expert), softmax-normalizes them into a probability distribution, selects the top 2, and sets the remaining 30 to zero. The token is then processed only by the 2 winning experts, and their outputs are summed (weighted by the router probabilities). The total parameters in the 32 FFNs are large (contributing to the 8.3B total), but only 2/32 of them are used per token (giving 0.75B activated parameters), so inference cost stays manageable while the model has access to diverse specialized computation.

Why sparse MoE for time series: the paper argues that time series data exhibits "global heterogeneity yet local simplicity." Across different domains (finance, weather, healthcare), the underlying patterns are qualitatively different—requiring different processing—but within a single patch, the pattern is relatively simple. A dense feedforward layer must learn a single set of weights that works for all domains, creating interference (updates that improve financial forecasting may degrade weather forecasting). The MoE allows different experts to specialize in different types of temporal patterns without interfering. The configuration $E = 32, K = 2$ reflects this: many experts for broad coverage across domains, but only a few activated per patch to keep computation sparse and to encourage each patch to pick the most relevant experts rather than averaging over many.

Load balancing. To prevent the router from collapsing to always selecting the same few experts (which would defeat the purpose of having 32), an auxiliary loss encourages uniform expert utilization:

$\mathcal{L}_{\text{aux}} = E \sum_{j=1}^{E} f_j P_j$

$f_j = \frac{1}{KN} \sum_{i=1}^{N} \mathbf{1}\big(a_{j,i} \in \operatorname{Topk}(\{a_{j,i} \mid 1 \leq j \leq E\}, K)\big)$

$P_j = \frac{1}{N} \sum_{i=1}^{N} a_{j,i}$

where $f_j$ is the fraction of tokens in the batch that were routed to expert $j$ (the empirical selection frequency), $P_j$ is the average router probability assigned to expert $j$ across all tokens, and $\mathbf{1}(\cdot)$ is the indicator function.

What this computes: the product $f_j P_j$ measures alignment between selection frequency and router confidence. If an expert is selected frequently (high $f_j$ ) but with low router probability (low $P_j$ ), the product is moderate. If an expert is selected frequently with high probability, the product is high. The sum over all experts, scaled by $E$ , reaches its minimum $1$ when all experts are equally utilized (each $f_j = K/E = 2/32$ , each $P_j = 1/E = 1/32$ ) and grows when utilization becomes imbalanced. The auxiliary loss is added to the main training objective with weight $\alpha$ .

Why this form: the auxiliary loss is the standard load-balancing loss from the MoE literature (Shazeer et al., 2017). The multiplicative form $f_j P_j$ is differentiable with respect to the router weights (unlike a hard constraint), allowing gradient-based optimization to push the router toward balanced assignment. The scaling by $E$ normalizes the loss so the weight $\alpha$ is comparable across different numbers of experts.

After passing through all $L = 24$ TimeMoE blocks, the model produces a set of $N$ token embeddings $\{\mathbf{h}_i^{24}\}_{i=1}^{N}$ that encode the full historical context—each token's embedding has aggregated information from all previous tokens through the causal attention mechanism across 24 layers of processing.

Next-Token Prediction: The Base Training Objective

The main TimeMoE blocks are trained with a standard next-token prediction (NTP) objective: each token's output embedding predicts the next patch in the sequence. Specifically, the embedding $\mathbf{h}_i^{24}$ of the $i$ -th token is projected through a shared PatchProject layer to predict the $(i+1)$ -th patch:

$\hat{\mathbf{x}}_{i+1} = \operatorname{PatchProject}(\mathbf{h}_i^{24})$

$\mathcal{L}_{\text{NTP}} = \sum_{i=1}^{N} \mathcal{L}_{\text{pred}}(\mathbf{x}_{i+1}, \hat{\mathbf{x}}_{i+1})$

where $\mathbf{x}_{i+1}$ is the ground-truth $(i+1)$ -th patch (containing 16 actual future time points), $\hat{\mathbf{x}}_{i+1}$ is the predicted quantile distribution over that patch, and $\mathcal{L}_{\text{pred}}$ is the quantile loss defined in Section 3.3 (discussed below). The sum runs over all $N$ input tokens, meaning every position contributes a prediction loss—this is the "dense supervision" the paper emphasizes.

What this computes: for each position in the input sequence, the model attempts to predict the patch that immediately follows. If the input contains patches 1 through 180, then $\mathbf{h}_1^{24}$ predicts patch 2, $\mathbf{h}_2^{24}$ predicts patch 3, and so on up to $\mathbf{h}_{180}^{24}$ predicting patch 181 (the first future patch beyond the input window). Every token's representation is explicitly optimized to be useful for one-step-ahead forecasting.

Why dense supervision (every token predicts the next patch) rather than only the last token: training only the last token to predict would waste most of the sequence for learning—the model would only get one supervision signal per training example. Dense supervision extracts $N$ training signals per example, dramatically improving sample efficiency. It also forces the model to learn representations that are useful for forecasting from any point in the sequence, not just from the end, which improves robustness to variable-length inputs at inference time.

The limitation of NTP alone: after training, only the last token $\mathbf{h}_N^{24}$ can be used to generate future predictions beyond the input window. To forecast $H$ patches ahead, the model would need to predict patch $N+1$ , append it to the input, re-encode the entire sequence, predict patch $N+2$ , and so on— $H$ sequential forward passes through the full 24-block model. Each pass incurs the full computational cost, and each step's prediction error contaminates all subsequent steps. This is the "error accumulation" problem that TimeSTP is designed to solve.

TimeSTP Blocks: Horizon-Dependent Serial Computation

The core architectural innovation of Timer-S1 is the TimeSTP block, which enables multi-step forecasting in a single forward pass while ensuring that longer-horizon predictions undergo strictly more serial computation than shorter-horizon ones. This section requires careful attention because the mechanism's elegance lies in the precise way each block conditions on both the evolving prediction and the original input.

The problem TimeSTP solves. With only the main TimeMoE blocks, multi-step forecasting requires autoregressive rollout: $\mathbf{h}_N^{24}$ predicts patch $N+1$ , which is appended to the input, the whole sequence is re-encoded, and the new $\mathbf{h}_{N+1}^{24}$ predicts patch $N+2$ . This is computationally expensive ( $H$ full forward passes for $H$ patches) and propagates errors. Multi-token prediction alternatives—having one block predict multiple patches in parallel—avoid the rollout cost but lack serial computation between horizons. TimeSTP resolves both problems by adding dedicated blocks that perform progressive refinement.

Architecture. After the $L = 24$ main TimeMoE blocks, the model appends $H = 16$ TimeSTP blocks (indexed by $j = 1, \dots, 16$ ). Each TimeSTP block contains:

A projection layer $\mathbf{M}_j \in \mathbb{R}^{1024 \times 2048}$ that fuses two sources of information: the token embeddings from the preceding block (either the last main block for $j=1$ , or the previous TimeSTP block for $j>1$ ) and the original input patch embeddings $\mathbf{h}_i^0$ (the output of PatchEmbed from Section 3.1).
A full TimeMoE module (the same architecture as the main blocks) that processes the fused embeddings.

The fusion step. For each token position $i$ and each TimeSTP block $j$ :

$\bar{\mathbf{h}}_i^{24+j} = \mathbf{M}_j \cdot \operatorname{Concat}\big(\operatorname{RMSNorm}(\mathbf{h}_i^{24+j-1}), \operatorname{RMSNorm}(\mathbf{h}_i^{0})\big)$

where $\mathbf{h}_i^{24+j-1} \in \mathbb{R}^{1024}$ is the embedding of token $i$ from the preceding block (for $j=1$ , this is $\mathbf{h}_i^{24}$ from the last main block; for $j>1$ , this is $\mathbf{h}_i^{24+j-1}$ from the previous TimeSTP block), $\mathbf{h}_i^{0} \in \mathbb{R}^{1024}$ is the original patch embedding of token $i$ , and $\operatorname{Concat}$ produces a 2048-dimensional vector by stacking the two normalized embeddings.

What this computes: the projection $\mathbf{M}_j \in \mathbb{R}^{1024 \times 2048}$ is a learned matrix that compresses the concatenated 2048-dimensional vector back to the 1024-dimensional hidden dimension. This is essentially a learned linear combination of two sources: the "evolving representation" from the previous block (which has undergone increasing amounts of serial processing for deeper $j$ ) and the "original input" representation (frozen from the initial embedding). The fusion allows each TimeSTP block to both carry forward the refined representations from earlier predictions and continually re-ground itself in the raw input data.

Why condition on the original embeddings $\mathbf{h}_i^0$ : without this connection, the TimeSTP blocks would operate in a purely autoregressive fashion—each block's input depends only on the previous block's output, and errors would accumulate across blocks the same way they accumulate across autoregressive rollout steps. By re-injecting the original input embeddings at every TimeSTP block, the model has a direct "skip connection" to the raw data that is not contaminated by prediction errors from intermediate blocks. This is similar in spirit to how U-Nets use skip connections to preserve fine-grained spatial information, but applied temporally: the original data provides a clean reference that prevents the serial refinement from drifting.

The TimeMoE processing. After fusion, the projected embeddings pass through a standard TimeMoE block (causal self-attention with RoPE, QK-Norm, and sparse MoE), identical in structure to the main blocks:

$\mathbf{h}_i^{24+j} = \operatorname{TimeMoE}(\bar{\mathbf{h}}_i^{24+j})$

The prediction step. The output embedding $\mathbf{h}_i^{24+j}$ from each TimeSTP block is projected through the shared PatchProject layer to produce a prediction. Critically, the offset of the prediction depends on the block depth $j$ —the $j$ -th TimeSTP block predicts the patch shifted by $j+1$ positions ahead:

$\hat{\mathbf{x}}_{i+j+1} = \operatorname{PatchProject}(\mathbf{h}_i^{24+j})$

So for a given token position $i$ , block $j=1$ predicts patch $i+2$ , block $j=2$ predicts patch $i+3$ , and block $j=16$ predicts patch $i+17$ .

The serial-token prediction objective. The training loss for the TimeSTP blocks averages over all blocks and all token positions:

$\mathcal{L}_{\text{STP}} = \frac{1}{H} \sum_{j=1}^{H} \sum_{i=1}^{N} \mathcal{L}_{\text{pred}}(\mathbf{x}_{i+j+1}, \hat{\mathbf{x}}_{i+j+1})$

where $\mathbf{x}_{i+j+1}$ is the ground-truth patch at that horizon and position, and $\hat{\mathbf{x}}_{i+j+1}$ is the predicted quantile distribution. The outer factor $1/H$ ensures the STP loss is on the same scale as the NTP loss regardless of the number of TimeSTP blocks.

What this computes: every TimeSTP block at every token position produces a prediction for a specific future patch offset by $j+1$ from the current token. The model is supervised to make accurate predictions at all horizons simultaneously, with the deeper $j$ blocks naturally receiving more gradient signal for longer-horizon predictions. The block depth $j$ thus becomes a proxy for forecasting horizon—longer horizons involve strictly more serial computation because they pass through more TimeSTP blocks.

Inference behavior. At inference time, only the last token's embeddings from each block are used for forecasting beyond the input window:

$\mathbf{h}_N^{24}$ (from the main blocks) predicts patch $N+1$ (the first future patch)
$\mathbf{h}_N^{25}$ (from TimeSTP block 1) predicts patch $N+2$
$\mathbf{h}_N^{26}$ (from TimeSTP block 2) predicts patch $N+3$
...
$\mathbf{h}_N^{40}$ (from TimeSTP block 16) predicts patch $N+17$

All $1 + 16 = 17$ future patches are produced in a single forward pass. Moreover, if the required forecasting horizon is shorter than 17 patches, the model can stop early—only the needed TimeSTP blocks are executed, providing adaptive inference depth. This is in contrast to multi-token prediction approaches where a fixed number of output patches are generated and excess predictions must be truncated (wasting computation) or autoregressive approaches where every horizon requires a full model pass.

Why this is not just multi-token prediction. In standard multi-token prediction, multiple future outputs are predicted from a single shared representation—typically the last hidden state of the encoder. The predictions at horizons 1, 2, and 17 all emerge from the same vector, with no mechanism for horizon 17's prediction to explicitly depend on horizon 16's predicted value. This is fundamentally a parallel computation. In Timer-S1, the prediction at horizon 17 depends on the representation $\mathbf{h}_N^{40}$ , which has passed through 16 additional TimeMoE blocks beyond the representation $\mathbf{h}_N^{24}$ used for horizon 1. Each block includes self-attention that can attend to all previous token representations (including the refined representations from earlier TimeSTP blocks), creating an explicit computational path where later predictions condition on the intermediate representations of earlier ones. This serial computation is what the paper argues is missing from prior multi-token prediction approaches and is essential for modeling the compounding-uncertainty structure of real forecasting.

Why TimeSTP blocks are retained during inference. In language models that use auxiliary prediction heads (e.g., DeepSeek's MTP), the extra blocks are typically discarded after training—they serve only as a representation-learning regularizer. Timer-S1 explicitly retains the TimeSTP blocks at inference. The paper argues this is necessary because of the "distributional variability of time series, where the train-test gap has a pronounced impact." In language, the tokens at training and test time come from the same discrete vocabulary, so an auxiliary head trained to predict shifted tokens learns transferable representations even if the head is later removed. In time series, the continuous-valued predictions have no fixed vocabulary—the model's own predicted values at inference time come from a different distribution than the ground-truth values seen during training (because predictions contain errors). Removing the TimeSTP blocks and falling back to autoregressive rollout would mean the model generates its own inputs during inference (the predicted patches fed back as input), which come from a distribution it never saw during training (where ground-truth patches were always used as input). Keeping the TimeSTP blocks eliminates this train-test gap entirely—the model never consumes its own predictions as input.

Quantile Forecasting Head and Loss Function

All predictions in Timer-S1—from both the main TimeMoE blocks (NTP) and the TimeSTP blocks (STP)—are produced by a shared PatchProject layer that maps a 1024-dimensional token embedding to $Q = 9$ quantile predictions, each spanning a full patch of $P = 16$ time points:

$\operatorname{PatchProject}: \mathbb{R}^{1024} \mapsto \mathbb{R}^{9 \times 16}$

The nine quantile levels are $q_k \in \{0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9\}$ , following the GIFT-Eval benchmark configuration. Let $\hat{\mathbf{x}}^{(k)} \in \mathbb{R}^{16}$ denote the $k$ -th quantile prediction (all 16 time points of the patch predicted at quantile $q_k$ ). Let $\mathbf{x} \in \mathbb{R}^{16}$ be the ground-truth patch values.

The prediction loss for one patch is the weighted Quantile Loss (wQL), a common approximation to the Continuous Ranked Probability Score (CRPS):

$\mathcal{L}_{\text{pred}}(\mathbf{x}, \hat{\mathbf{x}}) = \frac{1}{Q} \sum_{k=1}^{Q} \text{wQL}_{q_k}(\mathbf{x}, \hat{\mathbf{x}}^{(k)})$

$\text{wQL}_{q}(\mathbf{x}, \hat{\mathbf{x}}) = 2 \cdot \frac{\sum_{t=1}^{P} \rho_q(x_t, \hat{x}_t)}{\sum_{t=1}^{P} |x_t|}$

$\rho_q(x, \hat{x}) = \begin{cases} (1 - q) \cdot (\hat{x} - x), & \text{if } x < \hat{x} \\ q \cdot (x - \hat{x}), & \text{if } x \geq \hat{x} \end{cases}$

where $\rho_q(x, \hat{x})$ is the pinball loss at quantile level $q$ , $x$ is the ground-truth value at a single time point, and $\hat{x}$ is the predicted value at that quantile and time point.

What this computes, step by step:

Per time point, per quantile: the pinball loss $\rho_q(x, \hat{x})$ penalizes the discrepancy between the prediction $\hat{x}$ and the truth $x$ , but asymmetrically—if the truth is below the prediction ( $x < \hat{x}$ ), the penalty is $(1-q)(\hat{x}-x)$ , which is small for high $q$ (e.g., $q=0.9$ gives penalty $0.1 \times \text{error}$ ) because a 90th-percentile prediction should be above the truth 90% of the time. If the truth is above the prediction ( $x \geq \hat{x}$ ), the penalty is $q \cdot (x - \hat{x})$ , which is large for high $q$ (e.g., $q=0.9$ gives penalty $0.9 \times \text{error}$ ) because a 90th-percentile prediction that is below the truth is a severe under-estimate. This asymmetric penalty is what makes the model learn proper quantiles rather than just the mean.
Per patch: the pinball losses are summed over all $P = 16$ time points in the patch and divided by $\sum_{t} |x_t|$ —the sum of absolute ground-truth values. This normalization by the patch's total absolute magnitude makes the loss scale-invariant (comparable across series with different amplitudes), which is critical for training on the heterogeneous TimeBench corpus. The factor of 2 is a convention that makes wQL approximate CRPS.
Per quantile set: the wQL values for all $Q = 9$ quantile levels are averaged, giving a single scalar loss per patch that measures the quality of the full predictive distribution.

Why quantile loss (CRPS approximation) instead of MSE: mean squared error optimizes only point predictions (the conditional mean) and provides no information about predictive uncertainty. In real-world forecasting, knowing the uncertainty is often as important as knowing the point estimate—a forecast of "temperature will be 20°C ± 1°C" carries very different information than "temperature will be 20°C ± 10°C." The CRPS (approximated here by the weighted quantile loss) is a proper scoring rule for probabilistic forecasts: it evaluates how well the entire predictive distribution matches the observation, penalizing both biased point predictions and overconfident or underconfident uncertainty estimates. The use of quantile loss specifically aligns with the GIFT-Eval benchmark, which reports CRPS as its primary probabilistic metric.

Why nine quantile levels: the set $\{0.1, 0.2, \dots, 0.9\}$ follows the GIFT-Eval protocol and provides a reasonable discrete approximation to a continuous predictive distribution. The 0.5 quantile corresponds to the median (point forecast), and the spread between e.g., 0.2 and 0.8 provides an 60% prediction interval. Finer quantile grids (e.g., percentiles) would give more detailed distribution information but increase computational cost; coarser grids would lose distributional fidelity. The paper notes that the architecture is general and can accommodate other forecasting heads (linear projection, parametric probabilistic heads, diffusion-based heads), suggesting the MoE+STP architecture is independent of the specific probabilistic formulation.

The Pre-Training Objective: Unified Dense Supervision

The full pre-training loss combines the next-token prediction loss (for the main TimeMoE blocks), the serial-token prediction loss (for the TimeSTP blocks), and the MoE load-balancing auxiliary loss:

$\mathcal{L}_{\text{Pre-train}} = \mathcal{L}_{\text{NTP}} + \mathcal{L}_{\text{STP}} + \alpha \mathcal{L}_{\text{aux}}$

where $\alpha$ is a hyperparameter controlling the strength of load balancing (the paper does not specify its exact value, but it is typically small, e.g., 0.01, to avoid the auxiliary loss dominating the prediction task).

What this computes: a single scalar loss that simultaneously trains the model to (1) make accurate one-step-ahead predictions from every input position (NTP), (2) make accurate multi-step-ahead predictions with increasing serial computation for longer horizons (STP), and (3) maintain balanced expert utilization across the 32 MoE experts (auxiliary). The NTP and STP losses are equally weighted ( $1:1$ ), and within STP, all 16 TimeSTP blocks are equally weighted.

Why equal weighting at pre-training: the paper states this maximizes "sample efficiency from the raw series and ensures the TimeMoE module (for contextual representation) and the TimeSTP module (for multi-patch prediction) are fully trained." The equal weighting means the model is not yet biased toward any particular horizon—it learns general representations that serve all horizons equally well. The horizon-specific fine-tuning happens in the post-training stage, where the weighting becomes asymmetric.

Post-Training: Weighted STP and Context Extension

The pre-training stage produces a model that can forecast at all horizons, but the paper identifies a key limitation: "short-term forecasting, as the initial step of long-term forecasting, should be further enhanced." This is because long-term forecast accuracy fundamentally depends on short-term accuracy—errors at step 1 compound through all subsequent steps. The post-training stage therefore shifts emphasis to short-term performance while maintaining long-term capability.

Weighted Serial-Token Prediction (wSTP). The post-training loss replaces the uniform STP weighting with a horizon-decaying weight:

$\mathcal{L}_{\text{Post-train}} = \mathcal{L}_{\text{NTP}} + \mathcal{L}_{\text{wSTP}} + \alpha \mathcal{L}_{\text{aux}}$

$\mathcal{L}_{\text{wSTP}} = \frac{1}{H} \sum_{j=1}^{H} \frac{1}{\sqrt{j}} \sum_{i=1}^{N} \mathcal{L}_{\text{pred}}(\mathbf{x}_{i+j+1}, \hat{\mathbf{x}}_{i+j+1})$

where $j$ is the TimeSTP block index ( $j=1$ is the shallowest, predicting one step ahead) and $1/\sqrt{j}$ is the decay weight.

What this computes: the shallow TimeSTP blocks (small $j$ , short horizons) receive higher weight in the loss. Specifically, block $j=1$ gets weight 1.0, block $j=4$ gets weight $1/\sqrt{4} = 0.5$ , block $j=16$ gets weight $1/\sqrt{16} = 0.25$ . The model is still trained on all horizons, but gradient updates are dominated by short-term prediction errors.

Why $1/\sqrt{j}$ specifically: the paper derives this from "the linear growth of variance in a standard first-order Markov process." For a random walk $X_t = X_{t-1} + \epsilon_t$ with independent innovations of variance $\sigma^2$ , the variance of the $j$ -step-ahead forecast is $j\sigma^2$ , so the standard deviation grows as $\sqrt{j}$ . The inverse $\sqrt{j}$ weighting thus down-weights longer-horizon predictions in proportion to their inherent uncertainty—predictions that are inherently more uncertain (because of compounding variance) contribute less to the loss, preventing the optimization from being dominated by noisy long-horizon gradients.

Continued Pre-Training (CPT) strategy. Rather than fine-tuning on downstream datasets (which would degrade zero-shot generalization), the post-training continues pre-training on a mixture of two data sources: the GIFT-Eval Pretrain dataset (which contains short-term forecasting tasks) and the original TimeBench corpus. This data revisiting mechanism "mitigates overfitting to the distribution of the post-training dataset and enhances generalization across other data." The model sees the short-term-focused data that drives the weighted STP improvement, but the interleaved TimeBench data prevents catastrophic forgetting of the diverse patterns learned during pre-training.

Why CPT instead of fine-tuning: standard fine-tuning on a specific downstream dataset often leads to "catastrophic forgetting"—the model's performance on other datasets degrades as it specializes. CPT, by maintaining a mixture of the original pre-training data and the new data, preserves the general forecasting capability while improving the targeted short-term performance. This is analogous to instruction tuning in language models, where the model is fine-tuned on a mixture of task-specific and general data to improve capabilities without losing breadth.

Context length extension. The pre-trained model operates with a maximum context of $T = 2880$ time points ( $N = 180$ patches). During post-training, the context window is extended to $T = 11520$ time points ( $N = 720$ patches) through RoPE-based position interpolation. The paper does not detail the specific interpolation method (e.g., whether they use linear interpolation, NTK-aware scaling, or YaRN), but the key idea is that RoPE's rotation frequencies are adjusted so that the position embeddings that were trained for relative distances up to 180 patches generalize to relative distances up to 720 patches. This is possible because RoPE encodes relative position—the model learns that patches 100 positions apart have a certain relationship, and by rescaling the rotation frequencies, that same relationship can be applied to patches 400 positions apart.

Why extend context post-training rather than training with full context from the start: training with 11,520-length sequences from scratch would be computationally prohibitive (self-attention cost scales quadratically with sequence length, so $720^2 / 180^2 = 16\times$ more expensive per sequence). By pre-training with shorter context and extending post-training, the model learns the fundamental temporal patterns efficiently and then adapts to longer-range dependencies with a relatively small amount of additional training. This two-stage approach is standard in the LLM literature (e.g., Llama, GPT-4) and is applied here to time series for the same computational efficiency reasons.

Data Curation and Augmentation

While the architecture and training objectives are the paper's primary technical contributions, the data pipeline is essential infrastructure that enables the scaling. Timer-S1 is pre-trained on TimeBench, a curated corpus of 1,032 billion regularly sampled time points (approximately 1 trillion). The curation involves three stages: collection, preprocessing, and augmentation.

Collection. TimeBench draws from multiple sources:

Real-world data from domains including finance, IoT, meteorology, and healthcare, plus publicly released time series from the Chronos and LOTSA projects.
Synthetic data including canonical signals (linear, sinusoidal, exponential, power, impulse, step functions) and their additive/multiplicative combinations, plus KernelSynth-generated data (randomly instantiated temporal causal models that produce realistic-looking synthetic time series with known ground-truth structure).
Variate selection guided by a proxy criterion: a variate is selected if it exhibits a strong autoregressive property, measured by the statistical significance of a fitted ARIMA model. This filters out variates that are essentially white noise (no learnable temporal structure) and improves the signal-to-noise ratio of the training data.

The inclusion of synthetic data is strategically important. Real-world time series, while abundant, may underrepresent certain pattern types (e.g., exponential growth, sharp regime changes, specific frequency combinations). Synthetic data can fill these gaps, exposing the model to a wider range of temporal dynamics than any finite collection of real datasets could provide.

Preprocessing. The raw data undergoes rigorous cleaning designed to prevent the model from learning spurious patterns from artifacts:

Causal mean imputation: missing values are filled using only past information (no "peeking" at future values), maintaining the causal structure that the model must respect at inference.
Outlier removal: values exceeding $k$ - $\sigma$ thresholds or IQR (interquartile range) thresholds, computed over a shifting window, are identified and removed. The shifting window ensures that what counts as "outlier" adapts to the local context—a value that would be extreme for one regime might be normal for another.
Timestamp preservation: for data with timestamps, the original time information is retained; for data without timestamps, default numeric indices starting from 0 are assigned. This allows the model to potentially learn frequency-dependent patterns (e.g., daily vs. hourly vs. monthly periodicity) if temporal metadata is available, though the paper notes that Timer-S1 does not natively incorporate exogenous covariates like timestamps in its current form.

Test data leakage prevention. All instances that could lead to overlap with GIFT-Eval test datasets are "carefully removed." This is critical for foundation model evaluation—if the pre-training corpus accidentally includes sequences that appear (even in modified form) in the downstream evaluation, the reported zero-shot performance would be inflated.

Data assessment metrics. Each dataset in TimeBench is characterized by two metrics that together define a "complexity plane":

$\text{ADF-Statistic}(\mathcal{D}) = \sum_{i=1}^{C} \frac{T_i}{T} \cdot \text{ADF-Statistic}(\mathbf{S}^{(i)})$

$\text{Forecastability}(\mathcal{D}) = \sum_{i=1}^{C} \frac{T_i}{T} \cdot \left(1 - \operatorname{Entropy}\left(\mathcal{F}(\mathbf{S}^{(i)})\right)\right)$

where $C$ is the number of variates, $T_i$ is the length of variate $i$ , $T = \sum_i T_i$ is the total length, $\mathbf{S}^{(i)}$ is the $i$ -th variate, $\text{ADF-Statistic}$ is the Augmented Dickey-Fuller test statistic (measuring stationarity—more negative means stronger evidence against a unit root, i.e., the series is more stationary), and $\mathcal{F}(\mathbf{S}^{(i)})$ is the Fourier decomposition whose entropy measures spectral concentration (a pure sinusoid has low entropy, white noise has high entropy, so $1 - \text{entropy}$ measures "forecastability" from spectral structure).

What this computes: each dataset is placed as a point in a 2D space where the x-axis measures stationarity (via ADF) and the y-axis measures forecastability (via spectral concentration). Datasets with diverse characteristics populate different regions of this plane, and the paper can potentially use these coordinates for data mixing strategies (though this is not explicitly detailed as part of the training recipe).

Data augmentation. The paper identifies and addresses predictive bias—the tendency of models trained on imbalanced real-world data to develop stereotypical forecasting behaviors. Two augmentation techniques are applied:

Resampling: the sampling rate of original series is varied through down-sampling and Fourier-based interpolation. This exposes the model to the same underlying pattern at multiple temporal resolutions. For example, a daily temperature cycle might appear as a period-24 pattern in hourly data, a period-7 in 4-hourly data, and a period-1 in daily data—resampling forces the model to recognize the pattern itself rather than memorizing specific frequencies.
Value-flipping: the input and output series are both multiplied by $-1$ , inverting all trends while preserving temporal dependencies. A series trending upward becomes trending downward; a spike becomes a dip. This directly counteracts "the model's tendency to latch onto persistent directional trends," which the paper identifies as a bias from prior work (Timer-3/Sundial). If the training data is dominated by series with upward trends (as is common in financial and economic data), the model may learn to always predict increases—value-flipping ensures the model sees equal numbers of upward and downward trends for statistically identical temporal structures.

Why these augmentations matter for scaling: as model size increases, the capacity to memorize spurious dataset-specific correlations also increases. A small model might be forced to learn generalizable patterns because it lacks the capacity to memorize; a billion-parameter model has ample capacity to memorize that "most series in the training set trend upward, so always predict upward." The augmentations actively combat this memorization by forcing the model to rely on the actual temporal structure (which is preserved under flipping and resampling) rather than surface-level statistical biases.

Training Infrastructure and Data Loading

The paper provides brief but important details about the computational infrastructure supporting the billion-scale training, which is relevant to understanding the practical feasibility of the approach.

Framework. Training is supported by VeOmni, a unified framework for pre-training and post-training foundation models. The framework enables distributed training across multiple devices with BF16 (Brain Floating Point 16-bit) precision, which halves memory usage compared to FP32 while maintaining sufficient numerical range for stable training.

Data storage and loading. Raw data in TimeBench is stored as compressed Parquet files (approximately 4 TB total). A key challenge is efficient sampling: the training procedure requires random access to sliding windows from arbitrary positions in arbitrary series, but loading the full 4 TB corpus into memory is impractical. The solution is a hybrid memory-disk loading strategy:

The dataset is partitioned into 50 MB shards, a size chosen to balance I/O concurrency (many shards can be loaded in parallel) and sampling randomness (larger shards would limit the diversity of series accessible simultaneously).
Each shard serves as the basic unit for in-memory sliding-window sampling.
An in-memory queue manages active shards, loading new shards from disk as needed and evicting processed ones, avoiding the need to load the entire corpus simultaneously.

This design enables random access to the trillion-point corpus without requiring a machine with multiple terabytes of RAM, making the training infrastructure practical for research labs without extreme hardware resources.

Sequence format. TimeBench is loaded in a single-sequence series format: multivariate time series are split into their constituent univariate series, and each univariate series is treated as an independent training instance. This means the model learns univariate evolving patterns during pre-training—cross-variate interactions are explicitly ignored. The paper justifies this by noting that variate semantics and correlations are "dataset-specific and unstable in cross-domain generalization," so a general-purpose pre-trained model should focus on the universal skill of temporal pattern recognition that transfers across domains. Multivariate structure can be re-introduced during task-specific fine-tuning.

Dense task construction. From each univariate series, the pre-training pipeline constructs a dense set of forecasting tasks: arbitrary contiguous segments of the series serve as input, and the immediately following segments serve as output. This means a single series of length $L$ contributes approximately $O(L^2)$ training examples (all possible input-output segmentations), maximizing sample efficiency from the raw data. The NTP and STP losses then provide supervision at every token position within each example, further multiplying the effective number of training signals.

4. Key Insights and Innovations

Innovation 1: Serial Computation as an Architectural Primitive for Forecasting

The paper's most fundamental conceptual contribution is reframing the forecasting problem from a representation-learning challenge to a computation-allocation challenge. The key insight is that the amount of serial computation applied to a prediction should scale with how far into the future that prediction lies—short-term predictions require less processing, long-term predictions require more. This is not merely an architectural tweak; it is a diagnostic recognition that forecasting is an inherently serial process where uncertainty compounds multiplicatively across the horizon, and that an architecture which mirrors this structure will scale better than one that collapses all horizons into a single parallel computation.

What the field did before this. Prior time series foundation models fell into two camps, each making an implicit architectural bet about how to handle multi-horizon forecasting. The next-token prediction (NTP) camp—including the Timer family's own predecessors, Chronos, and TimesFM—used decoder-only Transformers with causal masking, trained to predict one patch ahead. At inference, multi-step forecasts required autoregressive rollout: predict step 1, append it to the input, re-encode the entire sequence, predict step 2, and so on. This preserves serial computation—each step conditions on all previous predictions—but at prohibitive computational cost (H full model passes for H steps) and with severe error accumulation (errors at step 1 contaminate all subsequent steps). The multi-token prediction (MTP) camp—including Timer-3/Sundial, Moirai, and approaches adapted from LLMs like DeepSeek—predicted multiple future steps in parallel from a single shared representation. This is computationally efficient (one forward pass) but collapses all serial dependencies into a single feedforward computation: the prediction at step 50 has no explicit computational path through which it can depend on the predicted value at step 49. The dominant assumption was that a sufficiently expressive shared representation could capture horizon-specific information implicitly; the paper argues this assumption is fundamentally wrong for a task whose error structure is explicitly serial.

How this paper reframes the problem. Timer-S1's TimeSTP architecture decomposes forecasting into two computational stages with fundamentally different properties. The TimeMoE main blocks perform shared computation—extracting a rich contextualized representation of the entire historical input, which benefits all forecasting horizons equally. The TimeSTP blocks then perform horizon-indexed serial computation: the first block produces the one-step-ahead representation, the second block takes that refined representation and produces the two-step-ahead representation, and so on through all 16 blocks. The 16-step-ahead prediction thus passes through 16 more Transformer blocks than the one-step-ahead prediction. This is not an incremental improvement over multi-token prediction—it is a fundamentally different computational structure where the depth of processing is the mechanism by which the model models compounding uncertainty. The paper's evidence for this is the scaling analysis (Figures 10–11): under matched total block counts, Timer-S1 with 24 TimeMoE + 16 TimeSTP blocks substantially outperforms both Timer-NTP (40 TimeMoE blocks, autoregressive inference) and Timer-MTP (40 TimeMoE blocks, parallel multi-token prediction), demonstrating that it is not more parameters that drive the gain, but the specific allocation of those parameters to horizon-dependent serial computation.

Significance beyond raw performance. This reframing has implications for how the field should think about scaling time series models. Prior work implicitly assumed that scaling laws for time series would mirror language: more parameters + more data → better representations → better performance, with architecture playing a secondary role. Timer-S1 demonstrates that architecture is a primary determinant of whether scaling yields benefits, because the serial nature of forecasting imposes a structural constraint that parallel-only architectures cannot satisfy regardless of parameter count. This is a diagnostic insight rather than merely a performance claim: it explains why prior scaling attempts (e.g., Moirai, Chronos) may have hit diminishing returns despite increasing model size. The bottleneck was not insufficient parameters but insufficient serial computation in the architecture. The paper supports this with the model configuration scaling analysis (Figures 13–14), which shows continued improvements as both TimeMoE and TimeSTP blocks are scaled up to the billion-parameter level—evidence that the serial computation structure enables, rather than merely accompanies, the scaling.

Innovation 2: Retaining Auxiliary Prediction Blocks to Close the Train-Test Gap

A second conceptual contribution, more subtle but equally important, is the paper's explicit rejection of the LLM practice of discarding auxiliary prediction heads after training, and its argument that the train-test distribution gap in time series is severe enough to require architectural identity between training and inference. This is a negative finding about the transferability of LLM training techniques to time series, with direct implications for future model design.

What the field did before this. In language models, multi-token prediction has been used as a representation-learning regularizer: during training, auxiliary heads predict future tokens from intermediate representations, forcing the model to learn features useful at multiple horizons. After training, these auxiliary heads are discarded—the model reverts to standard next-token autoregressive generation. This works in language because the training and inference distributions are similar: at both training and test time, the model receives tokens from a shared discrete vocabulary, and the representations learned under the auxiliary objective transfer to the standard generation setting. DeepSeek's MTP is the canonical example cited by the paper.

Why this fails for time series. The paper's ablation (Figure 15) tests two variants of this LLM-inspired approach. Timer-S1-Shift-Token uses shifted future embeddings during training (the auxiliary block receives the ground-truth future patch as input, shifted by one position) but falls back to autoregressive inference—this is the direct analog of DeepSeek's MTP and performs worse than the retained-block design. Timer-S1-Remove-STP trains with the TimeSTP blocks but discards them at inference, relying on autoregressive rollout from the last main-block token—this performs "significantly worse" than the retained-block design. The paper's diagnosis is that time series data exhibits "distributional variability" that makes the train-test gap particularly pronounced: at inference, the model's own predicted values serve as input to subsequent steps, and these predictions come from a meaningfully different distribution than the ground-truth values seen during training (because predictions contain errors). In language, a slightly-off predicted token is still a valid vocabulary item; in time series, a slightly-off predicted value can shift the input distribution enough to degrade subsequent predictions in a way that compounds across horizons.

Significance beyond raw performance. This finding is significant as a boundary condition on technique transfer from LLMs to time series. The paper demonstrates that a practice considered standard and effective in language modeling (auxiliary prediction as representation-learning regularization) is actively harmful in time series forecasting. This has implications for the entire pipeline of adapting LLM innovations to time series—architectural choices, training objectives, and inference procedures must be validated in the time series context rather than assumed to transfer. The paper's decision to retain TimeSTP blocks at inference is not an implementation detail but a principled response to a fundamental difference between discrete-token and continuous-value sequence modeling.

Innovation 3: Multi-Stage Training with a Theoretically-Motivated Horizon-Weighted Objective

The paper's training methodology introduces two innovations that together address a previously unrecognized conflict in time series pre-training: short-term and long-term forecasting make different demands on the model, and optimizing both simultaneously in a single stage creates conflicting gradient signals. The solution—a two-stage pipeline with a theoretically-motivated horizon-weighting scheme—represents a conceptual advance in how to structure foundation model training for tasks with inherent horizon-dependent structure.

What the field did before this. Standard practice in time series foundation model pre-training was single-stage: train on a diverse corpus with a uniform objective that treats all forecasting horizons equally. The implicit assumption was that a sufficiently general representation learned under uniform supervision would serve all horizons well. The paper argues this assumption fails because short-term forecasting benefits from precise local pattern matching (requiring the model to attend closely to recent dynamics), while long-term forecasting requires capturing broader structural regularities but tolerates higher variance (requiring the model to ignore high-frequency noise that dominates short-term signals). Optimizing both simultaneously creates tension—gradients that improve short-term accuracy may degrade long-term representations, and vice versa.

The two-stage solution and its theoretical grounding. The pre-training stage uses uniform horizon weighting: all 16 TimeSTP blocks receive equal weight in the STP loss. This ensures the model learns representations that are useful at all horizons, maximizing sample efficiency and preventing premature specialization. The post-training stage then shifts to a weighted STP (wSTP) objective where the loss weight for block $j$ decays as $1/\sqrt{j}$ . The paper derives this specific decay rate from first principles: for a standard first-order Markov process, the variance of the $j$ -step-ahead forecast grows linearly with $j$ , so the standard deviation grows as $\sqrt{j}$ . The inverse $\sqrt{j}$ weighting thus down-weights longer-horizon predictions in proportion to their inherent uncertainty, preventing noisy long-horizon gradients from dominating the optimization while still maintaining signal from all horizons.

Why this is not just curriculum learning or fine-tuning. The post-training stage is explicitly continued pre-training (CPT), not fine-tuning. It maintains a mixture of the original TimeBench data and the new short-term-focused data, which the paper argues "mitigates overfitting to the distribution of the post-training dataset and enhances generalization across other data." This is a deliberate departure from standard fine-tuning approaches (e.g., Chronos-X, AdaPTS, Cora) that adapt pre-trained models to specific downstream datasets at the cost of degraded zero-shot performance on other domains. The CPT strategy preserves the general forecasting capability while improving the targeted short-term performance—a capability-specific enhancement rather than a task-specific specialization.

Evidence and significance. Figure 9 shows that the post-training stage (CPT + context extension) yields clear improvements over the single-stage pre-trained model on the GIFT-Eval leaderboard. This validates the paper's claim that single-stage pre-training is suboptimal because it "may overlook the task discrepancy." More broadly, this training methodology establishes a template for how to handle tasks with inherent horizon-dependent structure: decouple general representation learning (pre-training with uniform weighting) from horizon-specific optimization (post-training with theoretically-motivated weighting), using data revisiting to prevent catastrophic forgetting. This is a conceptual framework applicable beyond time series to any sequential prediction task where the relationship between input and output horizons is non-uniform.

Innovation 4: Targeted Data Augmentation as Predictive Bias Correction

While data augmentation is not itself novel, the paper's contribution here is diagnostic rather than methodological: it identifies specific, measurable predictive biases that emerge from imbalanced real-world training data, and proposes augmentations targeted at those specific biases rather than generic diversity enhancement. This represents a shift from "augment to increase data diversity" (the standard motivation) to "augment to correct identified failure modes," which is a more principled and potentially more sample-efficient approach to data curation.

The specific biases identified. The paper identifies two failure modes that emerge when models are trained on imbalanced real-world time series. First, directional trend bias: the model learns to predict persistent trends in the direction that dominates the training data (typically upward, since many real-world series—stock prices, economic indicators, population counts—exhibit secular growth). The value-flipping augmentation directly counteracts this by multiplying both input and output by −1, inverting trends while preserving the temporal structure. Second, frequency overfitting: the model becomes tuned to the specific sampling frequencies prevalent in the training data and performs poorly on series with different temporal resolutions. The resampling augmentation addresses this by varying sampling rates through down-sampling and Fourier-based interpolation.

Evidence that these are genuine failure modes, not hypothetical concerns. The ablation in Figure 16 shows that removing data augmentation degrades performance on GIFT-Eval. More revealing is Figure 17, which shows performance on sinusoidal signals of varying frequencies: the model without augmentation exhibits an "error spike" at a period of approximately 16, which is exactly the configured patch size. This is a concrete, measurable artifact of frequency overfitting—the model has learned to rely on patch-boundary-aligned patterns that fail when the dominant frequency does not align with the patch grid. The resampling augmentation smooths out this spike by exposing the model to the same pattern at multiple temporal resolutions.

Significance beyond Timer-S1. This diagnostic approach to data augmentation—identify specific biases through analysis of failure cases, design augmentations targeted at those biases, verify through controlled experiments—is a template for data curation in any domain where training data distributions are imbalanced in identifiable ways. The paper's value-flipping technique is particularly elegant because it is nearly cost-free (just a sign flip) yet addresses a bias that would otherwise require collecting an entirely new dataset with balanced trend directions. This kind of targeted augmentation is likely to become more important as foundation models scale: the larger the model, the greater its capacity to memorize dataset-specific biases rather than learning generalizable patterns, making active bias correction through augmentation increasingly necessary rather than optional.

5. Experimental Analysis

Evaluation Methodology

Dataset. All experiments use the GIFT-Eval benchmark (Aksu et al.), which comprises 24 datasets spanning 144,000 time series and 177 million data points, covering diverse domains, frequencies, and forecasting horizons. The paper does not describe a custom train/validation/test split—GIFT-Eval is a standardized leaderboard with a fixed evaluation protocol. The pre-training corpus TimeBench is a separate dataset of 1,032 billion time points used only for model training, not evaluation.
Base model(s). All experiments use the Timer family of decoder-only Transformers, with Timer-S1 being the primary model (24 TimeMoE blocks, 16 TimeSTP blocks, 8.3B total parameters, 0.75B activated per token). Scaling ablations vary the number of TimeMoE blocks (Figure 13) and TimeSTP blocks (Figure 14). Training objective comparisons use matched backbone variants: Timer-NTP (40 TimeMoE blocks, next-token prediction only) and Timer-MTP (40 TimeMoE blocks, multi-token prediction). The predecessor Timer-3 (Sundial) is included as a direct lineage comparison, trained on the same TimeBench corpus.
Metrics. Two primary metrics are used, following GIFT-Eval protocol:
- MASE (Mean Absolute Scaled Error) for point forecasting: scales the forecast error by the in-sample naive forecast error, making it comparable across series with different scales. Lower is better.
- CRPS (Continuous Ranked Probability Score) for probabilistic forecasting: a proper scoring rule that evaluates the full predictive distribution against the observed value. Lower is better. The paper's quantile loss (Equation 13) directly approximates CRPS, but GIFT-Eval reports the exact CRPS computed from the quantile predictions.
Both metrics are computed per dataset and then averaged (the paper does not specify whether this is a macro-average or weighted by series length, but GIFT-Eval's standard protocol uses macro-averaging across datasets).
Baselines. The paper compares against multiple categories of prior work on the GIFT-Eval leaderboard (Figures 6–8):
- Statistical methods: ARIMA, Exponential Smoothing (ETS), Theta.
- Deep learning models (trained per dataset): DeepAR, N-BEATS, TFT, PatchTST, TimesNet, DLinear, and others.
- Time series foundation models (pre-trained, zero-shot): Timer (the original), Timer-XL, Timer-3 (Sundial), Chronos (both original and Chronos-2), Moirai, Moment, TimesFM, TTM, and others visible in the GIFT-Eval leaderboard figures.
The paper does not enumerate all baselines in a dedicated table; rather, Figure 6 displays a subset of the GIFT-Eval leaderboard with Timer-S1 highlighted. For scaling and ablation comparisons, the primary baselines are Timer-NTP, Timer-MTP, and the pre-trained Timer-S1 without post-training.
Generation budget / compute accounting. The paper uses number of Transformer blocks as the primary unit of compute comparison for architectural scaling analysis. In Figures 10–11, models are matched on total block count (40 blocks total for NTP and MTP variants vs. 24+16=40 for Timer-S1). For inference time comparison (Figure 12), wall-clock time is measured for a single forward pass with input sequence length 11,520. The paper does not report FLOP counts, training wall-clock time, or GPU-hours for any experiment—this is a notable omission for a paper about scaling. The $4\times$ storage requirement for TimeBench (4 TB Parquet files) is mentioned but not factored into any cost analysis.
Cross-validation / statistical protocol. The paper does not describe any cross-validation procedure. GIFT-Eval is a fixed benchmark with a pre-defined test set; the paper reports results from a single evaluation run. No confidence intervals, standard deviations, or statistical significance tests are reported for any metric. This is a significant methodological gap—with 24 datasets of varying sizes, the variability in per-dataset performance could meaningfully affect the overall average, and without uncertainty quantification, it is difficult to assess whether reported differences (e.g., 7.6% MASE reduction over Timer-3) are statistically reliable. For the scaling analyses (Figures 13–14), the paper does not specify whether results are from a single training run or averaged across multiple random seeds, making it impossible to distinguish genuine scaling trends from training noise.

Main Quantitative Results

GIFT-Eval Benchmark Performance

Headline results. Timer-S1 achieves MASE = 0.693 and CRPS = 0.485 on the GIFT-Eval leaderboard (Figure 6), placing it as the best-performing pre-trained model at the time of evaluation. The paper reports these as state-of-the-art, though Figure 6 shows that Chronos-2 (which explicitly models multivariate interactions, unlike Timer-S1's univariate pre-training) achieves competitive results. The specific advantage over Chronos-2 is not quantified in the text—readers must estimate from the bar chart, which is a limitation of the presentation.

Comparison to Timer-3 (Sundial). When both models are trained on the same TimeBench corpus, Timer-S1 achieves a 7.6% lower MASE and a 13.2% lower CRPS than Timer-3. This is the most controlled comparison in the paper because it isolates the effect of the STP architecture and the post-training pipeline (both models share the same pre-training data). The paper attributes this gain to "the serial scaling effect of our foundation model," but note that Timer-3 differs from Timer-S1 in multiple ways simultaneously (different architecture, different training objective, different training pipeline), so the 7.6%/13.2% improvement cannot be uniquely attributed to serial-token prediction—it represents the cumulative effect of all design changes.

Horizon-dependent performance analysis (Figures 7–8). The paper breaks down MASE and CRPS by forecasting term length, revealing a pattern that directly supports the core architectural motivation:

On short-term forecasting tasks, Timer-S1 performs competitively but does not dominate—other models achieve similar performance.
On medium-term tasks, a clear advantage emerges for Timer-S1.
On long-term tasks, Timer-S1 achieves "substantially better performance" (the exact numerical advantage is not quoted in the text, requiring readers to estimate from the grouped bar charts in Figures 7–8).

This pattern—where the performance advantage grows with the forecasting horizon—is the key evidence that the serial computations in TimeSTP blocks provide genuine benefit for long-term forecasting, as opposed to being an architectural change that helps uniformly across all horizons. The paper's interpretation is that "serial forecasting improves the performance on challenging long-term forecasting tasks through crucial serial computations."

What the GIFT-Eval results demonstrate and what they do not. The leaderboard placement establishes that Timer-S1 is competitive with or superior to existing pre-trained models. However, several caveats apply:

GIFT-Eval is not an independent test set for Timer-S1 because the post-training stage explicitly uses "datasets for short-term tasks in GIFT-Eval Pretrain." The paper states that instances with potential test leakage were removed, but the domain overlap between GIFT-Eval Pretrain and GIFT-Eval test may still give Timer-S1 an advantage over models that were not post-trained on GIFT-Eval-adjacent data. The paper does not report results for Timer-S1 without post-training on the full GIFT-Eval, which would quantify the magnitude of this advantage.
The comparison to Chronos-2 is confounded by architectural differences (univariate vs. multivariate pre-training), training data differences (TimeBench vs. Chronos's proprietary corpus), and training pipeline differences.
Without error bars, the reliability of small performance differences between closely-ranked models is unknown.

Post-Training Analysis

Headline result. The post-training stage (continued pre-training with weighted STP + context extension from 2,880 to 11,520) improves performance over the single-stage pre-trained model (Figure 9). The specific numerical improvement is visible in the bar chart but not quoted in the text—readers must estimate the MASE and CRPS deltas from Figure 9.

Interpretation. This result validates the paper's claim that "single-stage pre-training may overlook the task discrepancy, i.e., short-term and long-term forecasting tasks require different training objectives and training data." The post-training stage, which emphasizes short-term performance through the $1/\sqrt{j}$ horizon weighting and uses a mixture of GIFT-Eval Pretrain data with TimeBench, improves overall performance on the benchmark. However, because GIFT-Eval Pretrain data is used in post-training, the improvement partially reflects better alignment with the evaluation distribution rather than a fundamental improvement in forecasting capability—this is the standard domain-adaptation vs. generalization tension that affects any post-training/fine-tuning evaluation.

Scaling Analysis: Serial-Token Prediction vs. Baselines

Headline comparisons (Figures 10–12). Under a matched total block count of 40 Transformer blocks:

Timer-S1 (24 TimeMoE + 16 TimeSTP) outperforms Timer-NTP (40 TimeMoE, autoregressive inference), as shown in Figure 10. The specific accuracy values must be estimated from the bar chart—the paper does not quote exact numbers in the text.
Timer-S1 outperforms Timer-MTP (40 TimeMoE, multi-token prediction), as shown in Figure 11. Again, exact numbers are not quoted.

Inference time (Figure 12). The paper reports that for a single inference pass with input length 11,520:

Timer-NTP requires passing through all 40 blocks for each predicted patch (autoregressive rollout), making multi-step inference proportionally expensive.
Timer-MTP uses a larger forecasting head (predicting all horizons at once) and must "truncate redundant predictions," incurring additional computation despite the single forward pass.
Timer-S1 requires only one pass through the 24 main blocks plus one TimeSTP block per additional predicted patch, and can adapt inference depth to the required horizon.

The specific inference times are shown as a bar chart in Figure 12 but not quoted numerically. This comparison is somewhat unfair to Timer-MTP because the multi-token prediction model could be designed without redundant predictions for a known horizon—the truncation waste is an implementation choice, not an architectural necessity.

What the scaling analysis demonstrates. These results show that under matched parameter/computation budgets, allocating some blocks to horizon-indexed serial computation (TimeSTP) is more effective than allocating all blocks to either autoregressive encoding (NTP) or parallel multi-horizon prediction (MTP). This supports the paper's central claim that serial computation is a more efficient use of parameters for multi-horizon forecasting. However, the analysis only tests one specific allocation ratio (24:16). The paper does not systematically vary the TimeMoE-to-TimeSTP ratio at fixed total block count to find the optimal split, which would strengthen the claim that the 24:16 ratio is not arbitrary but genuinely optimal or near-optimal.

Model Configuration Scaling (Figures 13–14)

Scaling TimeMoE blocks (Figure 13). With TimeSTP fixed at 16 blocks, the pre-trained Timer-S1's performance (measured by what appears to be MASE on GIFT-Eval) continues to improve as TimeMoE blocks increase, up to the 24-block configuration. The paper interprets this as evidence that the model "continues to benefit from scaling up to the billion level."

Scaling TimeSTP blocks (Figure 14). With TimeMoE fixed at 24 blocks, performance improves as TimeSTP blocks increase from low values up to approximately 16. The paper does not specify whether the curve plateaus or continues rising beyond 16 (the x-axis in Figure 14 appears to stop at 16, so this is the maximum tested).

Interpretation. These curves are the paper's scaling law evidence—they suggest that increasing either the shared representation capacity (TimeMoE) or the serial computation depth (TimeSTP) yields diminishing but still positive returns at the scale tested. However, several limitations apply:

The curves show performance at one training duration per configuration—these are not full training budget scaling curves (performance vs. total FLOPs), which would require training each configuration to convergence and plotting final performance against total compute. Without this, it is unclear whether the 24+16 configuration outperforms an 18+18 configuration, for example, or simply received more effective training under the fixed training schedule.
The paper does not report whether these results are from single runs or averaged across seeds, making it unclear whether the apparently smooth upward trends are genuine or partly noise.
The parameter counts for different configurations are not reported, so readers cannot assess whether the performance gains are proportional to the increase in parameters (i.e., whether scaling is compute-efficient or simply adds parameters for marginal gains).

Ablation Studies and Robustness Checks

TimeSTP design variants (Figure 15): The paper tests two alternatives to the retained TimeSTP block design.

Timer-S1-Shift-Token: uses shifted future embeddings during training (the TimeSTP block receives ground-truth future patches as input, shifted by one position) but falls back to standard inference without these blocks. This performs worse than the retained-block design, supporting the paper's claim that the train-test gap in time series (where the model's own predictions differ in distribution from ground-truth inputs) makes the LLM practice of discarding auxiliary heads inappropriate.
Timer-S1-Remove-STP: trains with TimeSTP blocks but discards them at inference, relying on autoregressive rollout from the last main-block token. This performs "significantly worse" than the retained-block design, confirming that the serial computation provided by TimeSTP blocks during inference is essential, not merely a training regularizer.

What this ablation establishes. The retained-block design is not an arbitrary choice—both alternative approaches that follow LLM conventions (discard auxiliary heads after training) or that remove serial computation at inference lead to degraded performance. This is one of the more convincing ablations in the paper because it isolates a specific architectural decision (retain vs. remove) and shows clear practical consequences.

Data augmentation (Figures 16–17):

Figure 16 shows overall GIFT-Eval performance with and without data augmentation (resampling + value-flipping). The augmented model performs better—the exact numerical difference must be estimated from the bar chart.
Figure 17 provides a more granular analysis: performance of the non-augmented model on sinusoidal signals of varying frequencies shows an error spike at period ≈ 16, which is exactly the patch size. The resampling augmentation smooths this artifact. This is a compelling diagnostic: the model without augmentation has learned a spurious dependency on patch-boundary alignment, and the augmentation directly addresses this by exposing the model to the same patterns at varying temporal resolutions.

What this ablation establishes. Data augmentation is genuinely corrective, not merely diversifying. The sinusoidal error spike (Figure 17) is a concrete, measurable failure mode that the augmentation eliminates. This is stronger evidence than an overall performance improvement because it demonstrates that a specific, predicted bias was present and was removed by the targeted intervention.

Pre-training on TimeBench (Figure 18): When Timer-S1's architecture is trained only on the post-training dataset (GIFT-Eval Pretrain) from scratch, without the TimeBench pre-training, performance degrades significantly compared to the full pre-trained + post-trained model. This confirms that the trillion-point TimeBench pre-training provides transferable knowledge that cannot be replicated by training the same architecture solely on the downstream-relevant data. The exact numerical difference is visible in the bar chart but not quoted in the text.

What this ablation establishes. Pre-training on the diverse TimeBench corpus provides a genuine benefit beyond what can be achieved by task-specific training with the same model architecture. This validates the foundation model approach (pre-train broadly, then adapt) as opposed to training large models from scratch on specific benchmarks. However, the ablation is an extreme comparison: TimeBench (1 trillion points) vs. GIFT-Eval Pretrain (presumably much smaller, though size is not reported). The performance gap may reflect data quantity rather than data diversity—a fairer ablation would compare TimeBench pre-training against pre-training on an equivalently-sized but less diverse corpus.

Training objective comparison (Figures 10–11, described in scaling analysis above): Under matched block count, STP outperforms both NTP and MTP. This is the paper's core empirical claim about the STP architecture, and it is tested under controlled conditions (matched total blocks).

Critical Assessment

Claim 1: Timer-S1 achieves state-of-the-art forecasting performance on GIFT-Eval.

What the experiments show. Figure 6 places Timer-S1 at the top of the GIFT-Eval leaderboard among pre-trained models for both MASE and CRPS. Figures 7–8 show this advantage is concentrated in medium- and long-term forecasting.