Performance Prediction for Large Systems via Text-to-Text Regression

🎯 Pitch

A 60M-parameter language model trained on raw system logs achieves near-perfect 0.99 rank correlation for efficiency prediction on Google's entire compute fleet, outperforming traditional tabular methods by 100x in MSE. This approach eliminates manual feature engineering entirely while naturally handling multi-modal outcome distributions and enabling few-shot adaptation to new environments with just 500 examples.

1. Executive Summary

This paper introduces text-to-text regression as a general alternative to traditional tabular regression for predicting system performance metrics from complex, non-tabular inputs like configuration files and system logs. Using Google's Borg compute cluster and an encoder-decoder Regression Language Model (RLM) of only 60M parameters trained from random initialization on string representations of system state—cell identifiers, timestamps, scheduler hyperparameters, hardware distributions, and job-on-machine profiling data—the approach achieves up to 0.99 Spearman rank correlation and 100× lower MSE than tabular baselines. The model enables few-shot adaptation to unseen compute clusters with as few as 500 examples and naturally captures multi-modal outcome distributions through density estimation, establishing that rich feature observability combined with large-scale multi-task pretraining can substitute for expert feature engineering—but only when the model observes features that uniquely identify the underlying state, since partial observability induces epistemic uncertainty that fundamentally bounds regression performance.

2. Context and Motivation

The Core Problem: Predicting System Behavior When the Features Don't Fit in a Table

The fundamental problem this paper addresses is deceptively simple: how do you build a regressor that can accurately predict a system's performance metric when the system's state description is complex, nested, variable-length, and resists clean featurization into a fixed-length tensor? The paper frames this as the tension between the richness of available system data and the rigid input requirements of standard regression methods.

Consider what a modern compute cluster looks like at any given moment. Google's Borg system—the specific case study in this paper, but representative of large-scale industrial schedulers broadly—manages millions of tasks across heterogeneous hardware in geographically distributed cells. To predict the aggregate efficiency of a scheduling decision (measured as MIPS per GCU, or millions of instructions per second per unit of computing resource), you might want your regressor to see: the cell name, the physical location, the exact time window, the scheduler's hyperparameters, the distribution of hardware platforms across network zones, which large users are active, and detailed job-on-machine profiling results showing how specific workloads perform on specific hardware types. This is the information logged by the system's digital twin—a backtesting framework that replays real cluster states through the scheduling algorithm—and it is available. The problem isn't a lack of data; it's that this data doesn't naturally fit into the $\mathbb{R}^d$ vector that a random forest or multi-layer perceptron expects.

This gap matters for reasons that go beyond academic curiosity:

Cost of simulation: Generating a single outcome from Borg's digital twin takes 1–18 hours of computation (Section 2.1). If a regressor could approximate this with negligible inference time, it would unlock rapid experimentation with scheduling configurations that is currently infeasible. Instead of waiting hours to evaluate a single hyperparameter setting, an optimization loop could evaluate thousands of candidates in minutes.
Limitations of existing optimization infrastructure: Google Vizier, the company's black-box optimization service, uses Gaussian Process regression as its surrogate model (Golovin et al., 2017). But a Gaussian Process can only ingest tabular feature vectors. This means that when Vizier is used to tune Borg's scheduler hyperparameters, it operates effectively blind to most of the system state—it sees hyperparameter values but not the cell, not the workload mix, not the hardware distribution. The paper explicitly states that this "drastically limits" Vizier's predictive and optimization performance (Section 2.1). Improving the regressor directly improves the optimizer.
Feature engineering as a bottleneck: Converting the nested, variable-cardinality system state into a fixed-length vector would require expert-driven decisions about how to aggregate lists of varying length (e.g., job profiles), how to encode categorical variables with open-ended value sets (e.g., hardware types that evolve over time), and how to normalize continuous values without knowing their ranges in advance. This isn't just labor-intensive—it's brittle. When new hardware platforms or workload types appear, the entire featurization pipeline breaks, and "all the training data produced from the prior method's rigid featurization are made incomplete and invalidated" (Section 2.4).
Adaptation to new deployments: A model that works on one cell (a specific cluster of machines) may fail on another because the workload mix, hardware composition, or usage patterns differ. The ability to quickly adapt to a new cell with minimal additional data—what the paper calls few-shot adaptation—is essential for deploying across Google's global infrastructure without training separate models for every location.

Where Prior Approaches Fall Short

The paper situates its work against three broad categories of existing methods, each with well-characterized limitations:

Blackbox tabular regression (random forests, MLPs, gradient boosting). These are the workhorses of industrial performance prediction, but their fundamental assumption is that features can be represented as flat, fixed-length tensors. For Borg-style data, this forces lossy compression: lists of jobs must be aggregated into summary statistics (counts, means, percentiles), categorical features with open-ended domains (like hardware platform names) must be pre-enumerated with a fixed vocabulary, and nested relationships (job X runs on platform Y with profile Z) must be flattened into independent features that lose the hierarchical structure. Section 2.4 argues that this compression is not merely inconvenient but performance-limiting—it directly induces epistemic uncertainty because different system states map to the same tabular feature vector, making them indistinguishable to the regressor.

The paper formalizes this concern in Section 2.3 through the lens of the bias-variance decomposition and the law of total variance (Equation 3 in Appendix A.1). If a regressor observes only a partial representation $\phi(x)$ of the full state $x$ , the expected squared error is bounded below by $\text{Var}(y|\phi(x))$ , which decomposes into average aleatoric uncertainty plus an epistemic term representing the variance in $\mathbb{E}[y|x]$ across states that look identical under $\phi$ . Observing less of the state means this epistemic term grows, and no amount of training data or model capacity can overcome it. As the paper puts it:

"If a regressor is only able to observe a partial subset or limited representation $\phi(x)$ of the full state, it will be unable to distinguish separate $x, x'$ if $\phi(x) = \phi(x')$ . This lack of distinguishability induces epistemic uncertainty, and instead leads to an even higher right hand variance term" (Section 2.3).

In practice, this means the best achievable MSE for a tabular regressor is "100x higher" (Figure 6) than what the RLM achieves by observing the full string representation. The tabular representation doesn't just make the problem harder—it imposes a hard ceiling on what any tabular model can achieve, regardless of how it is trained. This is the "TotalVariance bound" the paper estimates empirically (Equation 2): by grouping test examples into equivalence classes where $\phi(x)$ is identical, computing the variance of $y$ within each class, and averaging across classes, you get a lower bound on MSE for any regressor using features $\phi$ .

Graybox techniques (analytical modeling + learned coefficients). Some approaches try to incorporate domain knowledge by deriving symbolic expressions for system behavior (e.g., latency as a function of concurrent users, or throughput as a function of resource allocation) and then learning the coefficients from data (Didona et al., 2015). While these methods can work well when the underlying relationship has a simple parametric form, they are "very restrictive in their applicability" (Section 1). Borg's efficiency metric depends on interactions between dozens of features—hardware architectures, workload characteristics, temporal patterns, scheduler hyperparameters—whose joint effects are not captured by any clean analytical expression. Graybox methods require "large amounts of prior knowledge on the relationship between $x$ and $y$ " (Section 1) that simply doesn't exist for complex scheduling outcomes.

Gaussian Process regression with hand-crafted kernels. Google Vizier's default surrogate model is a Gaussian Process (Song et al., 2024b), which has attractive properties for Bayesian optimization—principled uncertainty quantification, sample efficiency, and support for mixed continuous-categorical domains. But the GP is fundamentally limited to tabular inputs. While kernel design can encode some structural priors (e.g., separate length scales per feature, periodic kernels for time), it cannot absorb raw text representations of nested system state. The paper notes that Vizier's GP "can at most only observe tabular data formats, drastically limiting its predictive and overall optimization performance" (Section 2.1). The RLM is positioned as a drop-in replacement for the GP's predictive component that removes this input bottleneck.

Language-model-based regression (nascent prior work). The most direct precursors are OmniPred (Song et al., 2024a) and related work on decoding-based regression (Song and Bahri, 2025). OmniPred showed that language models could serve as universal regressors by training on text representations of $(x, y)$ pairs across many different regression tasks, achieving strong transfer learning. However, OmniPred's experiments were on curated benchmark datasets (e.g., UCI regression tasks) with relatively clean, low-dimensional feature representations. The present paper extends this paradigm to the far messier setting of real industrial system data, where individual $x$ representations can span thousands of tokens (Figure 3, Table 1), features have deep hierarchical structure, and the output distribution may be multi-modal due to aleatoric noise. The paper also contributes extensive ablations specific to this regime—encoder vs. decoder-only architectures, sequence length scaling, feature importance analysis—that were not present in prior RLM work.

In-context regression vs. weight-based regression. Some recent work explores using LLMs as in-context regressors (Vacareanu et al., 2024), where the model is given several $(x_1, y_1), (x_2, y_2), \dots$ examples in the prompt and must predict $y$ for a new $x$ . The paper explicitly distinguishes its approach from this paradigm (Section 3.2, "Context-Free"): the RLM is trained to predict $y$ from a single $x$ , absorbing all training data into its weights rather than conditioning on it at inference time. This choice is motivated by a practical constraint—the string representation of a single Borg state can be thousands of tokens long, and including multiple in-context examples would quickly exhaust the model's context window. More fundamentally, weight-based learning "allows unlimited data to be absorbed within the model weights, rather than having finite limits at inference time due to the context buffer" (Section 3.2), analogous to why one might prefer a trained neural network over a non-parametric method for high-volume prediction.

How This Paper Positions Itself

The paper positions its contribution not as a new architecture or training algorithm—the encoder-decoder architecture and next-token prediction loss are standard—but as a validation that text-to-text regression solves a class of practical problems that tabular regression fundamentally cannot, and a characterization of why it works.

The key conceptual move is recasting the feature engineering problem as an information observability problem. Rather than asking "how can we hand-engineer a fixed-length feature vector that captures the important variation in the system state?", the paper asks "how much of the available state information can we show to the model, and how much does that matter?" The answer, demonstrated empirically, is that showing all of it matters enormously—the 100× MSE reduction over tabular baselines isn't from a cleverer model architecture but from removing the information bottleneck that tabular representations impose.

This framing connects the practical engineering concern (how to build a good regressor) to a formal statistical concept (epistemic uncertainty due to partial observability, Equation 3). Most practical ML papers treat feature engineering as an art; this paper quantifies its cost by computing the TotalVariance lower bound for different feature representations (Figure 6), showing that the gap between what tabular models can achieve and what the RLM does achieve is explained by the information lost in compression. This is a more rigorous justification for text-based regression than simply "it's easier"—it demonstrates that the approach is not just convenient but necessary for reaching certain performance levels.

The paper also positions RLMs as occupying a sweet spot in the pretraining-vs-task-specific-training spectrum. On one end, training a separate tabular model for each cell-month combination would require redoing feature engineering from scratch for each deployment and would fail to transfer knowledge across cells. On the other end, using a massive pretrained LLM (like a T5 or PaLM checkpoint) would be computationally expensive and potentially counterproductive—the paper explicitly argues that "it is not necessary nor guaranteed beneficial to use a pretrained LLM checkpoint" for regression because "tabular regression is possible tabula rasa" (Section 3.2). The RLM approach sits in the middle: a small (60M parameter) model trained from scratch on the target domain's data distribution, with architecture choices (encoder-decoder, P10 tokenization, cross-entropy loss) tailored specifically for regression rather than language generation.

Finally, the paper positions its work as enabling a broader vision of "universal simulators" for complex systems. If text-to-text regression can accurately model Borg's scheduling outcomes, then the same approach could, in principle, be applied to any system where state-to-outcome data is logged in a structured text format—power grids, supply chains, network routing, financial trading systems. The paper's title ("Performance Prediction for Large Systems via Text-to-Text Regression") and its framing in the conclusion ("These findings pave the way for universal simulators of real-world outcomes") signal this ambition. The specific Borg results are meant as a case study whose patterns—the importance of feature observability, the effectiveness of multi-task pretraining for few-shot adaptation, the natural uncertainty quantification from density estimation—are expected to generalize.

3. Technical Approach

3.1 Reader Orientation

The system is a Regression Language Model (RLM)—a small encoder-decoder transformer trained to read a text description of a compute cluster's state and output a predicted efficiency metric as a floating-point number. It solves the problem that traditional tabular regression can't handle: when the input features are complex, nested, variable-length system logs rather than clean fixed-length numeric vectors. The shape of the solution is deceptively simple: convert everything to strings, train a language model to predict the numeric outcome token by token, and let the model figure out which features matter.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components:

String Serializer — converts the raw Borg system state (cell name, timestamps, scheduler hyperparameters, hardware distributions, job-on-machine profiles) into a structured YAML-like string. This is the model's sole input—no feature engineering, no normalization, no vocabulary enumeration.
Encoder — a standard transformer encoder (2 layers by default) that processes the variable-length input string bidirectionally, producing a rich contextual representation of every token in the system state description. This is where the model learns to attend across different parts of the complex input.
Decoder — a standard transformer decoder (2 layers by default) that generates the prediction autoregressively, conditioning on the encoder's output via cross-attention. It outputs one token at a time using the P10 numeric tokenization scheme (sign, mantissa, exponent), producing a floating-point prediction like <+><7><2><5><E-1> which decodes to 72.5.
Vocabulary & Tokenizer — a SentencePiece tokenizer with T5X's default 32,000 subword vocabulary plus custom P10 numeric tokens for representing the output value. The input string uses standard subword tokenization; the output uses the specialized numeric vocabulary.
Inference Aggregator — at prediction time, samples 128 complete numeric outputs from the decoder, removes outliers outside the known feasible range [500, 3000], and aggregates them (mean for MSE minimization, median for rank correlation) to produce a point estimate. Also computes sample variance for uncertainty quantification.

Information flows sequentially: raw system state → string serializer → tokenizer → encoder (bidirectional attention over all input tokens) → decoder (autoregressive generation, cross-attending to encoder outputs) → P10 token sequence → numeric aggregation → final prediction.

3.3 Roadmap for the Deep Dive

First, the formal text-to-text regression framework—what exactly the model learns, the training objective, and why cross-entropy over tokens rather than MSE over values.
Second, the input representation—how a Borg system state gets converted to a string, what features are included, and the design decisions behind including everything rather than selecting a subset.
Third, the output representation—the P10 tokenization scheme, why it's used instead of direct numeric prediction, and how it enables multi-task training without normalization.
Fourth, the architecture choice—why encoder-decoder rather than decoder-only, and the empirical evidence for this decision.
Fifth, the training procedure—pretraining on multiple tasks simultaneously, the optimizer and hyperparameters, and the early stopping strategy.
Sixth, the fine-tuning procedure for few-shot adaptation—how pretrained checkpoints are adapted to new tasks with minimal data.
Seventh, the inference procedure—how the model is used at prediction time, including sampling, aggregation, and uncertainty quantification.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical systems paper whose core idea is that language models trained on string representations of complex system state can serve as universal regressors, bypassing the information bottleneck imposed by tabular feature engineering and achieving performance levels that are theoretically impossible for any method that observes only a compressed representation of the input.

The Text-to-Text Regression Framework

The paper frames regression as a sequence-to-sequence learning problem. In standard language model training, given a batch of (prompt, response) pairs, model updates minimize the next-token cross-entropy loss over response tokens. The RLM adapts this directly: the prompt is a string representation of the input features $x$ , and the response is a tokenized representation of the target value $y$ . The critical design choice is that $y$ is learned as a sequence of tokens—not as a single regression target with an MSE loss applied to a scalar output head.

The model is trained as a density estimator $p_\theta(y|x)$ . At inference time, pointwise predictions are made by sampling $y^{(1)}, \dots, y^{(s)} \sim p_\theta(y|x)$ and aggregating (e.g., taking the mean or median). This distinguishes the RLM from standard regression models that directly output a scalar $\hat{y}$ via a value head. The RLM learns the entire conditional distribution, not just its mean—a distinction that matters for the uncertainty quantification and multi-modal density capture demonstrated in Section 4.

Training Objective: Next-Token Cross-Entropy

The model minimizes:

$\mathcal{L}_{\text{CE}} = -\sum_{t=1}^{T} \log p_\theta(y_t | y_{<t}, x)$

where $x$ is the input string (the serialized system state), $y_{<t}$ are the previously generated target tokens, $y_t$ is the next target token, and $T$ is the total number of output tokens (typically 6-7 for a P10 representation). The sum runs over all output token positions.

What it computes: For each position in the target sequence, the model predicts a probability distribution over its vocabulary of 32,000 + custom tokens, and the loss penalizes it based on the negative log probability it assigned to the correct token. The total loss is the sum across all output positions.

Why this form: This is the standard language modeling objective, but it has specific advantages for regression that the paper argues make it preferable to error-based losses like MSE applied to a value head. First, the cross-entropy loss magnitude is "agnostic to the strength of the prediction gap" (Section 3.2)—it depends only on the probability assigned to the correct tokenization, not on the numeric distance between prediction and target. This means that tasks with naturally larger $y$ -value spreads (wider variance) don't dominate the training signal simply because their raw errors are larger, which would happen with MSE. The loss is scale-invariant, making it suitable for multi-task training over cells with different efficiency metric ranges.

Second, there is growing evidence from the optimization and reward modeling communities that "presumably due to their overly compressive nature, embedding-based or logit-based methods can perform worse than simply decoding the numeric prediction end-to-end" (Section 3.2, citing Tan et al., 2025; Mahan et al., 2024; Zhang et al., 2025). The intuition is that compressing all the information about $y$ into a single embedding vector or logit loses precision that can be retained by generating the number token by token—each digit gets its own dedicated computation.

Third, training with cross-entropy yields a proper density $p_\theta(y|x)$ rather than just a point estimate, enabling the uncertainty quantification in Figures 8 and 9, and the density estimation in Figures 5 and 11.

Why Not MSE or a Value Head?

Traditional regression models (including neural networks with value heads) minimize:

$\mathcal{L}_{\text{MSE}} = (y - f_\theta(x))^2$

where $f_\theta(x)$ is a single scalar output. The paper identifies three problems with this approach for the Borg setting, beyond the theoretical issues with compression from tabular features. First, MSE is sensitive to the scale of $y$ , which varies across cells from roughly 500 to 3000 MIPS per GCU. In multi-task training, cells with larger variance would dominate the gradient unless carefully reweighted. Second, MSE implicitly assumes a Gaussian error distribution with homoscedastic variance, which the paper demonstrates is violated in practice—the outcome distributions are often multi-modal (Figures 5, 9, 11). Third, a value head compresses all information about the output distribution into a single embedding dimension, which the cited work suggests loses information compared to token-by-token decoding.

An alternative sometimes used is to add a separate value head on top of the decoder's final hidden state, trained with MSE, while still using cross-entropy for the token-level loss. The paper does not explore this hybrid approach in detail, but its blanket statement that "cross-entropy loss rather than error-based loss (e.g. MSE) over an additional value head" is preferred suggests the authors tested or considered this and found pure decoding superior.

Input Representation: Serializing System State to Text

The input to the RLM is a single string that concatenates all available features describing the state of a Borg compute cell at the moment a scheduling decision is simulated. The string is formatted in a structured but not rigorously standardized way—essentially YAML-like with newline-delimited sections. Figure 21 in Appendix B.1 shows the exact format the model sees, which for a typical example spans several thousand tokens.

The features included in the string representation, and their average character counts from Table 1, are:

Cell Name (3 characters): An anonymized cell identifier like cell_a. This serves as a task-identifier feature that allows the model to learn cell-specific patterns during multi-task pretraining.
Physical Location (8 characters): The geographic location of the cell. Different locations have different user populations, different demand patterns, and potentially different hardware procurement histories.
Time Window (86 characters): A timestamp range like 2024-06-12 T06:00:00 Z to 2024-06-12 T06:05:00 Z, plus derived temporal features (day of week, week number). This captures cyclical patterns—fewer computing jobs at night, different workloads on weekends, seasonal effects.
Scheduler Hyperparameters (1,082 characters): The configuration of Borg's bin-packing algorithm, represented as a search space (possible categorical choices) and the actual assignments (which choice was made). For example, a workload like JOB/data_pipeline/PRODUCTION_WORKLOAD might be assigned to hardware platform machineA out of available options. These hyperparameters are what would be tuned by Google Vizier in an optimization loop.
Machine Distribution (461 characters): A hierarchical breakdown of hardware in the cell, organized by platform type. For each platform (e.g., machineA, machineE), the string includes the number of machines, the count of low/mid/high-level network zones, and total resources. This captures the physical capacity and topology of the cell.
Job-on-Machine Performance (268,157 characters, dominant): A detailed list of job profiles, each containing: the user/group identifier, platform-specific performance measurements (mean MIPS per resource usage on each hardware type that ran this job), observed resource limits and VM counts, and productivity metrics. This is the longest feature by far—Figure 3 shows the total string length distribution peaking around 20,000–50,000 characters, driven almost entirely by this component. It captures workload-hardware affinity: certain jobs run more efficiently on certain platforms, and this efficiency determines the overall cell-level metric after scheduling.

Serialization Design Decisions

The string representation makes several deliberate choices that are worth unpacking because they affect what the model can learn:

No normalization of numeric values. Continuous values like resource counts (5.481e+05) or MIPS measurements (8.165e+02) are included in their raw or scientific-notation form, without scaling to [0,1] or standardizing to zero mean and unit variance. This is possible because the model processes tokens, not continuous values—it learns embeddings for digit tokens, not for the numeric magnitudes directly. This avoids the practical problem that tabular approaches face: needing to know minimum and maximum values in advance for normalization, which breaks when new hardware or workloads introduce out-of-range values.

No enumeration of categorical features. Categorical variables like platform names (machineA, machineD) or user identifiers (data_pipeline, researcher_human_1) appear as literal strings. When a new platform type or user appears after deployment, the SentencePiece tokenizer will break it into subword tokens that the model can process even if it never saw that exact string during training. This is in contrast to tabular approaches that require pre-defining a finite vocabulary for each categorical feature, which becomes invalid when new categories appear.

Hierarchical structure preserved through nesting. The job profiles are organized as nested dictionaries: each job contains platform-specific profiles, each of which contains multiple measurements. This nesting is represented through indentation, colons, and commas in the string format, preserving the relationship that a particular MIPS measurement applies to a specific job on a specific platform. In a tabular representation, this structure would be flattened—you might have features like job_data_pipeline_mean_mips_on_machineA, job_data_pipeline_mean_mips_on_machineD, etc., losing the ability to represent arbitrary numbers of platforms per job and arbitrary numbers of jobs in a fixed-width vector.

Task-identifying features are explicit. The cell name and time window appear prominently in the string, serving as "task IDs" that allow the model to learn separate regression functions for different cells and time periods within a single model. This is what enables the multi-task pretraining described in Section 4—the model can be trained on data from many cells simultaneously, and at inference time, seeing the cell name tells it which regression function to apply, while the other features provide the within-cell variation.

Feature ordering is domain-informed. The paper notes in Section 3.4 that a practitioner "with domain knowledge can efficiently compress string representations by e.g. removing commas or whitespaces, and also placing the presumably most important features at the beginning of the string representation." While the exact ordering used in the paper isn't specified in detail, the structure in Figure 21 places cell identity, time window, and hyperparameters early, with the massive job profile section at the end. This matters because transformer models with causal attention in the decoder (and potentially input truncation at a maximum sequence length) implicitly weight earlier tokens more heavily in practice, though the encoder's bidirectional attention mitigates this to some extent.

Why include everything rather than selecting important features?

This is the paper's most fundamental design decision, and it follows directly from the epistemic uncertainty argument in Section 2.3. Every feature that is excluded from the input representation creates an equivalence class of system states that look identical to the model but may have different expected outcomes. The variance in $y$ within these equivalence classes becomes irreducible error. By including all available features—even ones whose relationship to the outcome is unknown or seemingly weak—the model maximizes the information it can use to distinguish states and minimize this epistemic uncertainty term.

The empirical validation of this principle comes in Figure 18 (Section 5.3), where ablating features from the input (removing cell, time window, or other components) consistently degrades validation loss. And Figure 17 shows that increasing the maximum sequence length—which allows more of the long job profile section to be included rather than truncated—improves performance up to approximately 3,000 tokens, after which diminishing returns set in because the additional tokens come from the "last remaining, longest, and least important features."

Output Representation: P10 Tokenization for Floating-Point Values

The RLM does not predict a continuous floating-point value directly. Instead, it generates a sequence of tokens that encode the number, using the P10 tokenization scheme from Charton (2022). A value $y$ is represented using three components:

A sign token: <+> or <->
Mantissa tokens: 4 digits, e.g., <7>, <2>, <5>, <0> for the mantissa 7250
Exponent token: <E-1> meaning $\times 10^{-1}$

So <+><7><2><5><0><E-1> represents $+7250 \times 10^{-1} = 725.0$ , and the target MIPS per GCU value might be, say, <+><8><1><2><5><E+2> which decodes to $8.125 \times 10^2 = 812.5$ .

What this representation achieves: A single output vocabulary of 10 digit tokens (<0> through <9>), one sign token, and a small set of exponent tokens (the paper uses 4-digit mantissas, implying exponents from approximately E-4 to E+4) can represent any floating-point number in the relevant range. The total number of P10 tokens added to the vocabulary is small—roughly 20-30 tokens—minimizing the parameter cost of learning embeddings for each one.

Why this form: This is fundamentally different from alternative numeric representations, each of which has specific drawbacks:

Direct numeric regression via a value head: As discussed above, this compresses all information into a single scalar and cannot represent multi-modal distributions. It also requires knowing the output range for normalization.
Tokenization as individual digits with a decimal point: Representing 725.0 as <7><2><5><.><0> would require the model to learn the positional significance of digits (7 is in the hundreds place). The exponent form explicitly separates magnitude from precision, making the positional structure explicit rather than implicit.
Tokenization as a fixed-precision integer: Representing all values as integers (e.g., 7250 for 725.0 if scaling by 10) would require knowing the maximum precision needed and would create a very large output vocabulary if using one token per possible value (thousands of tokens just for the output). The P10 scheme decouples precision (4-digit mantissa) from range (exponent), allowing a compact vocabulary regardless of the numeric range.
Tokenization with larger digit groupings: Using tokens for <00> through <99> (100 tokens) or <000> through <999> (1000 tokens) would reduce sequence length but increase vocabulary size and embedding parameters. The paper states that "using <0>, <1>, ..., <999> is far less effective than using a few digit tokens <0>, <1>, ..., <9>" (Section 3.2), citing the parameter efficiency of a smaller output vocabulary and the ability to compose arbitrary values from digits.

The P10 scheme is also "normalization-free, which allows easy multi-task training without needing to precompute minimum or maximum $y$ -value bounds for every separate task" (Section 3.2). This is crucial for the paper's setting, where different cells have different MIPS per GCU ranges (roughly 500-3000, but varying by cell), and new cells encountered during fine-tuning may have ranges not seen during pretraining. A normalized representation would break when the test-time range exceeds the training-time assumptions.

How the decoder generates these tokens: At training time, the target $y$ value is converted to its P10 representation and tokenized, then the decoder is trained to predict each token given the encoder output and previous target tokens (teacher forcing). At inference time, the decoder generates autoregressively, starting from a start-of-sequence token and sampling from its predicted distribution at each step until an end-of-sequence token is produced. The generated token sequence is then parsed back to a floating-point value by the inverse of the P10 mapping.

Architecture: Encoder-Decoder with No Language Pretraining

The RLM uses a standard T5X EncoderDecoder architecture, implemented in the open-source T5X framework. The default configuration (overridden in specific experiments) is:

2 encoder layers — transformer layers with self-attention over the input tokens
2 decoder layers — transformer layers with masked self-attention (autoregressive) and cross-attention to the encoder output
16 attention heads per layer
64 dimensions per head — giving a total attention dimension of $16 \times 64 = 1024$
512 embedding dimension — the size of token embeddings
2048 MLP dimension — the hidden size of the feed-forward network in each transformer layer
~58M total parameters — small by modern LLM standards but sufficient for the regression task
Default sequence length 2048 — inputs longer than this are truncated

The architecture is randomly initialized—no pretraining on natural language, no T5 or BERT checkpoint. This is explicitly justified: "it is not necessary nor guaranteed beneficial to use a pretrained LLM checkpoint from which to train a regression model" because "regression only requires learning the correlations between different structured tokens and does not necessarily benefit from the semantic meaning behind words" (Section 3.2). A pretrained LLM would bring knowledge of English syntax and semantics that is irrelevant to the structured YAML-like input format and might introduce unhelpful priors about what tokens "mean" in natural language contexts.

Why Encoder-Decoder Rather Than Decoder-Only?

This is one of the paper's most architecturally significant choices. Most modern LLMs (GPT, Llama, Gemma) use decoder-only architectures—a single stack of transformer layers with causal attention, where the input and output are concatenated into a single sequence. The encoder-decoder design separates input processing (encoder) from output generation (decoder), with the decoder attending to encoder outputs via cross-attention.

Figure 15 provides the empirical justification: when controlling for total parameter count (by adjusting the number of layers), encoder-decoder architectures substantially outperform decoder-only architectures. Specifically, the paper trains four configurations: 0E4D (decoder-only, 62.3M params), 1E3D (60.2M params), 3E1D (58M params), and 2E2D (56M params). The decoder-only model (0E4D) achieves the worst validation loss, and configurations with more encoder layers perform better.

The paper hypothesizes that "while decoder-only models are strong at producing outputs and chains of thought given relatively simple prompts, the information pathways routing through the decoder are insufficient to deal with complicated 'prompts' $x$ " (Section 5.2). A more precise way to understand this: in a decoder-only model, every token—both input and output—goes through the same stack of causal attention layers. This means that when processing input token $i$ , the model can only attend to input tokens $1, \dots, i$ (causal masking), not to later tokens. For a long, structured input like the Borg string where later sections (e.g., job profiles) provide context that helps interpret earlier sections (e.g., what those earlier identifiers refer to), this is restrictive. An encoder uses bidirectional attention, so every input token can attend to every other input token, building richer contextual representations.

Additionally, the encoder-decoder design cleanly separates two computations: the encoder's job is to compress and represent the potentially very long input (thousands of tokens) into a set of hidden states; the decoder's job is to generate a very short output (6-7 tokens) conditioned on those encoder states. Forcing a decoder-only model to use the same layers for both tasks—representing a long input and generating a short regression output—may create representation conflicts that the separated architecture avoids.

Model Size Scaling Behavior

Figure 16 shows that increasing model size from 45.5M to 234.3M parameters (by varying the number of layers from 2 to 32) reduces validation loss, but with strongly diminishing returns. The 58M-parameter model (2 layers) is close to the 83.2M model (8 layers), and even the 234.3M model (32 layers) provides only modest additional improvement. The paper interprets this as evidence that model size "quickly plateaus within the O(100M) range, which is orders of magnitudes lower than state-of-the-art general LLM models within the O(1B) range." This supports the paper's claim that text-to-text regression is accessible with modest compute—"at most 1 GPU" (Section 5.2).

The reason for this early plateau is the paper's regression scaling paradigm (Section 3.4): for regression, "the two most important scaling factors... are diverse training data and feature observability," not model size. The task of regression is "inherently discriminative and does not require large models for text generation." Unlike language generation, where fluency and coherence require modeling complex distributions over long sequences, the RLM's output is a single 6-7 token number whose distribution is conditioned on the input features. The heavy lifting is done by the encoder's representation of the input, not by the decoder's generation capacity.

Training Procedure: Multi-Task Pretraining

The RLM is trained on a collection of regression tasks simultaneously, where each "task" is defined as predicting MIPS per GCU for a specific cell during a specific month. More formally, a task $T$ is a set of 28K-56K $(x_i, y_i)$ pairs from a particular cell-month combination, randomly shuffled into 80/10/10 train/validation/test splits. The model may be pretrained on training splits from multiple tasks $\mathcal{T} = \{T_1, T_2, \dots\}$ , and later evaluated on tasks that were included in pretraining ("in-distribution") or excluded ("out-of-distribution").

Dataset Scale and Composition

The paper uses a pool of 40 cells with the largest $y$ -value spreads (Figure 22), across two months (June and November). Different experiments use different subsets:

Limit Testing (Section 4, Figure 6): Trained on nearly all tasks except the highest-spread cells $\{C_1^{JUN}, C_1^{NOV}\}$ , evaluated in-distribution on $C_2^{JUN}$ and out-of-distribution on $C_1^{NOV}$ . The pretraining data includes approximately 29 cell-month combinations—roughly 800K-1.6M training examples, or about 2 billion tokens at the average string length.
Adaptation Testing (Section 4, Figure 7): Trained on 1, 4, 8, 16, or 32 different tasks, always including $C_2^{JUN}$ for in-distribution evaluation, and evaluated out-of-distribution on $C_10^{NOV}$ .
Ablations (Section 5): Trained on 7 high-spread tasks from June.

The data comes from Borg's digital twin—the backtesting framework that replays real cluster checkpoint states through the scheduling algorithm to compute aggregate MIPS per GCU. Critically, the model never sees the actual scheduling algorithm; it learns to predict its output purely from input-output pairs.

Optimizer and Hyperparameters

The paper uses the Adafactor optimizer with specific settings documented in Appendix C.1:

Base learning rate: 0.1 with square root decay
1,000 warmup steps — the learning rate linearly increases from 0 to 0.1 over the first 1,000 steps
Decay factor: 0.5 — the learning rate is halved at each decay step
2,000 steps per decay — learning rate halves every 2,000 steps
10,000 steps per cycle — the decay schedule repeats every 10K steps
Batch size: 128 with 8 microbatches (effective batch size 128 across 8 gradient accumulation steps)

The Adafactor optimizer is chosen over Adam/AdamW because it has "sublinear memory cost" (Shazeer and Stern, 2018), important when training with long sequences (up to 4096 tokens). Adafactor maintains factored estimates of the second moment accumulator rather than a full matrix, substantially reducing memory usage.

Early Stopping Strategy

Training runs for a maximum of 100,000 steps, but early stopping is applied based on validation loss to prevent overfitting. The specific early stopping criterion is not detailed in the paper, but Appendix C.1 states "early stop based on validation loss if overfitting is detected." In practice, different experiments converge at different step counts—the Limit Testing model was stopped at 15K steps, while some ablation models ran to 26K-27K steps before reaching minimum validation loss. This variation arises because models with different sequence lengths and architectures saturate at different rates.

Multi-Task Learning Dynamics

The model learns to perform regression over multiple tasks simultaneously by observing the cell name and time window in the input string as "task-identifier features." This means the model doesn't require any specialized multi-task architecture—no task-specific heads, no task embedding layer, no explicit task ID token. The cell name appearing in the input serves the same function as a task ID would in a conventional multi-task learning setup: it tells the model which task's regression function to apply.

This design has an important implication: when fine-tuning on a new task (a new cell not seen during pretraining), the model has already learned to condition its predictions on cell identity. The new cell name is just a new value of an existing feature—the SentencePiece tokenizer breaks it into subwords that the model can process, and the pretrained encoder knows to attend to this feature to select the appropriate regression function. The fine-tuning process updates the model's weights to associate this new cell name with the correct outcome distribution.

Sequence Length and Truncation

Inputs longer than the model's maximum sequence length (default 2048) are truncated. The paper studies the effect of sequence length in Figure 17, finding that validation loss consistently decreases as the maximum length increases from 256 to 4096, with diminishing returns above approximately 3000 tokens. The truncation behavior is worth noting: when a string is truncated, the last tokens are dropped (standard for sequence models), which means the massive job-on-machine performance section at the end of the string is the first to be lost. The fact that performance continues improving up to 3000 tokens suggests that this feature, despite being the "least important" according to the paper, still contributes measurable predictive value when included.

Fine-Tuning for Few-Shot Adaptation

One of the paper's key claims is that the RLM supports few-shot adaptation to new tasks—unseen cells or time periods—with very small amounts of additional data. The fine-tuning procedure is straightforward but includes several nuanced design choices that affect performance.

Procedure

The fine-tuning process (Section 3.3) restores both the model weights and the optimizer state from a pretrained checkpoint, then resumes training on the new task's data with a modified learning rate. The process is:

Load pretrained checkpoint (weights + optimizer state)
Change the learning rate to $5 \times 10^{-5}$ (from the pretraining base of 0.1)
Set batch size to 128 (if the fine-tuning dataset has fewer than 128 examples, it is repeated to reach this size)
Train for up to 200 epochs, with early stopping based on validation loss

The number of fine-tuning examples can be arbitrarily small—experiments use 0, 4, 8, 16, 32, 64, 128, 256, and 512 examples. The 0-example case means evaluating the pretrained model directly on the new task without any adaptation (zero-shot transfer).

Why Restore the Optimizer State?

Restoring the optimizer state (momentum accumulators, learning rate schedule parameters) rather than reinitializing them is a deliberate choice. The Adafactor optimizer maintains running statistics of gradient magnitudes that inform per-parameter learning rate scaling. These statistics encode information about the loss landscape that generalizes across tasks—parameters that required small updates during pretraining are likely to need small updates during fine-tuning as well. Restarting the optimizer from scratch would lose this information and potentially cause destructive updates to parameters that were well-tuned during pretraining.

Why a Much Lower Learning Rate?

The fine-tuning learning rate of $5 \times 10^{-5}$ is 2,000× smaller than the pretraining base rate of 0.1. This reflects the different goals: pretraining must move parameters from random initialization to a useful configuration across many tasks, requiring large updates; fine-tuning must adapt an already-useful configuration to a new but related task, requiring small, careful adjustments that don't destroy the knowledge encoded in the pretrained weights.

Figure 19 studies the sensitivity of fine-tuning performance to the learning rate, testing values from $5 \times 10^{-3}$ down to $1 \times 10^{-6}$ . The optimal learning rate depends on the number of fine-tuning examples: with very few examples (4-8), higher learning rates ( $5 \times 10^{-4}$ to $1 \times 10^{-3}$ ) outperform the default, suggesting that aggressive adaptation is helpful when data is extremely scarce. With more examples (64+), lower learning rates ( $5 \times 10^{-5}$ to $1 \times 10^{-5}$ ) are better, as the model can learn more gradually from richer data.

Checkpoint Selection for Fine-Tuning

Figure 20 reveals a subtle but important finding: earlier pretraining checkpoints often produce better fine-tuning results than later ones on out-of-distribution tasks. The paper interprets this as "meta-overfitting"—an overly pretrained model may become too specialized to the pretraining tasks, making it harder to adapt to unseen tasks that differ from the pretraining distribution. This is conceptually similar to the finding in meta-learning that training to convergence on the meta-training set can hurt meta-test performance (overfitting to the specific tasks rather than learning general adaptation skills).

In practice, the paper selects the 10K-step checkpoint for fine-tuning experiments, which strikes a balance between having learned useful representations and not having over-specialized. This checkpoint is early enough in training that the model has not yet converged on the pretraining data, leaving more plasticity for adaptation.

Few-Shot Adaptation as Meta-Learning

The paper explicitly frames fine-tuning as "a form of meta-learning where pretraining leads to a checkpoint which can quickly be gradient-adapted to new tasks" (Section 3.3). This echoes MAML (Finn et al., 2017), where the goal of meta-training is to find a parameter initialization that can rapidly adapt to new tasks with few gradient steps. However, the RLM's meta-learning is implicit—it arises from standard multi-task pretraining rather than an explicit bi-level optimization. The model learns representations (like attending to cell identity, processing YAML-like structured text, mapping hardware profiles to performance outcomes) that generalize across cells, making gradient steps on a new cell's data particularly effective.

Data Repetition for Small Fine-Tuning Sets

When the fine-tuning dataset has fewer than 128 examples, the data is repeated to fill a batch of size 128. For example, with only 4 fine-tuning examples, each example appears 32 times in each batch. This means the model sees the same examples many times per epoch, which could in principle cause overfitting. However, the early stopping on validation loss mitigates this—training stops as soon as validation performance degrades, preventing the model from memorizing the tiny training set.

Inference Procedure: From Tokens to Predictions

At inference time, the trained RLM is used to produce both point predictions and uncertainty estimates. The procedure (Appendix C.1) has three stages: sampling, filtering, and aggregation.

Sampling

For a given input $x$ , the decoder generates $s = 128$ complete numeric predictions by autoregressive sampling from $p_\theta(y|x)$ . Each sample is produced independently: the decoder starts with a start-of-sequence token, samples the first output token from the predicted distribution, feeds it back as input, samples the next token, and continues until an end-of-sequence token is produced (typically 6-7 tokens total for a P10 representation).

The sampling is parallel—all 128 trajectories are generated independently, without any interaction. This is possible because the encoder processes the input $x$ once to produce a single set of hidden states, which are then fed to the decoder for each of the 128 generation runs. The computational cost is dominated by the encoder pass (processing thousands of input tokens) rather than the decoder passes (generating 6-7 tokens 128 times).

Filtering Outliers

After sampling, any predictions that fall outside a known feasible range of [500, 3000] MIPS per GCU are removed. This range is domain knowledge—MIPS per GCU cannot physically be below 500 or above 3000 in the Borg system. Outlier removal handles cases where the decoder generates an implausible token sequence (e.g., due to a rare combination of tokens that decodes to an extreme value, or hallucinated digits that don't correspond to a valid P10 format). The paper doesn't report what fraction of samples are filtered, but the existence of this step suggests the decoder occasionally produces outputs far from the training distribution of $y$ -values.

Pointwise Aggregation

The remaining samples are aggregated into a single point prediction. Two aggregation methods are used depending on the evaluation metric:

Mean (for minimizing MSE): The arithmetic mean of all valid samples. The mean is the Bayes-optimal predictor under squared-error loss—for any distribution, $\mathbb{E}_{y \sim p}[y]$ minimizes $\mathbb{E}[(y - \hat{y})^2]$ . By taking the sample mean with $s=128$ , the RLM approximates $\mathbb{E}_{p_\theta(y|x)}[y]$ , which approaches the true conditional expectation as $s \to \infty$ .
Median (for minimizing Spearman rank correlation): The numeric median of all valid samples. The median is less sensitive to outliers than the mean (even after filtering), and is optimal for absolute error loss. Since Spearman correlation depends only on rankings, not on the exact magnitudes of predictions, using the median provides some robustness against remaining extreme samples that could shift the mean.

Why 128 samples? This number represents a tradeoff between Monte Carlo accuracy and computational cost. The standard error of the sample mean decreases as $1/\sqrt{s}$ , so 128 samples gives roughly 11× less estimation error than a single sample. The paper doesn't ablate this hyperparameter, but the choice of 128 (a power of 2) suggests it was selected as a round number that provides sufficient precision without excessive inference cost. For the largest model (267M parameters), generating 128 samples per input is still computationally negligible compared to the 1-18 hours required for the actual Borg simulation.

Uncertainty Quantification

Beyond point predictions, the RLM provides natural uncertainty estimates through the sample variance of its generated predictions. For a given input $x$ , the model produces not just $\hat{y}$ but also the set $\{y^{(1)}, \dots, y^{(s)}\}$ from which the variance can be computed. Figure 8 shows that this sample variance correlates with the actual squared prediction error—inputs where the model's predictions disagree substantially (high variance) tend to be inputs where the model makes larger errors. This is a form of epistemic uncertainty quantification: when multiple plausible completions of the output sequence exist (the model's density has high variance), the model is uncertain, and downstream systems (like Bayesian optimization) can use this information to decide whether to trust the prediction or gather more data.

Furthermore, the full set of 128 samples provides more than just a variance—it captures the shape of $p_\theta(y|x)$ . Figures 5, 9, and 11 demonstrate that the RLM can represent multi-modal distributions, where different samples cluster around different output modes. This is impossible with a standard regression model that outputs only a mean and variance (typically assuming Gaussianity), and it provides information about the structure of the aleatoric uncertainty in the system (e.g., whether the outcome depends on some unobserved discrete choice that creates distinct modes).

The Regression Scaling Paradigm

Section 3.4 articulates a broader philosophy about what factors drive regression performance, which the paper calls its "regression scaling paradigm." The key claim is that for regression (as opposed to language generation), the most important scaling axes are not model size or sequence length, but rather:

Diverse training data: More tasks (more cells, more months, more varied conditions) provide better coverage of the input space, improving generalization and enabling transfer learning.
Feature observability: More complete representation of the system state (longer sequences, more features included) directly reduces the epistemic uncertainty lower bound, enabling higher accuracy regardless of model capacity.

This paradigm explains several of the paper's empirical findings: why a 60M-parameter model can achieve near-perfect prediction (sufficient feature observability + sufficient training data), why model size shows diminishing returns beyond ~100M parameters (the bottleneck is data coverage, not capacity), and why pretraining on more tasks substantially improves out-of-distribution performance but not in-distribution performance (in-distribution, the model already has sufficient data coverage; out-of-distribution, additional tasks provide representations that transfer to unseen cells).

The paradigm also provides practical guidance: if building an RLM for a new system, invest effort in (1) logging as many features as possible in a structured text format, and (2) collecting data from as many different operating conditions/cells/configurations as possible, rather than (3) scaling up model size or using a pretrained LLM checkpoint.

4. Key Insights and Innovations

Innovation 1: Epistemic Uncertainty as a Formal Lower Bound That Implicates Feature Engineering—Not Model Architecture—as the Primary Bottleneck

The deepest conceptual move in this paper is not the use of language models for regression—OmniPred (Song et al., 2024a) already established that this works. It is the formal reframing of feature engineering as an information observability problem with a quantifiable performance ceiling. Prior work treated feature engineering as an art: practitioners selected features based on domain intuition, and the cost of omitting features was vaguely understood as "the model sees less data." This paper gives that intuition teeth by connecting it to the Law of Total Variance (Equation 3, Appendix A.1) and operationalizing it through the TotalVariance lower bound (Equation 2, Section 2.3).

The argument proceeds in three precise steps:

Step one: For any regressor observing only a partial representation $\phi(x)$ of the full system state $x$ , the irreducible MSE is bounded below by $\text{Var}(y|\phi(x))$ , which decomposes into average aleatoric noise plus an epistemic term. This epistemic term— $\text{Var}_{x\sim p(x|\phi(x))}(\mathbb{E}[y|x])$ —measures the variance in expected outcomes across system states that look identical under the chosen feature representation. If two different states $x$ and $x'$ map to the same $\phi(x) = \phi(x')$ but have different expected outcomes, the regressor cannot distinguish them, and the resulting prediction error is baked into the problem structure regardless of model capacity, training data volume, or optimization procedure.

Step two: This bound is computable from data without knowing the true $p(y|x)$ . The paper's TotalVariance estimate (Equation 2) groups test examples into equivalence classes where $\phi(x)$ is identical, computes the empirical variance of $y$ within each class, and averages across classes. This produces a concrete number—a theoretical floor on achievable MSE for any model, of any architecture, trained on any amount of data, that observes only the features in $\phi$ .

Step three: The empirical gap between this bound and actual model performance directly quantifies the cost of lost information, not an architectural deficiency. Figure 6 demonstrates this: the TotalVariance bound for tabular features (observing only scheduler hyperparameters) is approximately 100× higher than the MSE achieved by the RLM observing the full string representation. This means that even a theoretically optimal tabular regressor would underperform the RLM by two orders of magnitude—the bottleneck was never the regression algorithm; it was the information content of the input representation.

This framing matters because it inverts the standard debugging workflow in applied ML. When a regressor performs poorly, the default response is to try a more sophisticated model—deeper networks, better architectures, more training tricks. This paper argues that for complex systems data, the more productive question is often: "What information about the system state is available but not being shown to the model?" And it provides a tool (the TotalVariance bound) to answer that question quantitatively before training begins. If the bound is high, no amount of modeling sophistication will fix the problem—you need to observe more features.

The connection to prior work is revealing. The standard bias-variance decomposition for regression (Equation 1) is textbook material, and the Law of Total Variance is a basic probability identity. What's novel is operationalizing these abstractions for a practical engineering decision (what features to include in a text serialization) and demonstrating that the resulting lower bounds explain real performance gaps at scale. This is theoretical insight that directly drives practical choices—the paper's decision to include all available features in the input string, rather than cherry-picking the "important" ones, follows logically from the epistemic uncertainty argument rather than from empirical trial-and-error.

The significance of this innovation extends beyond the Borg use case. Any system where rich, structured state descriptions are logged but traditionally compressed into tabular features for modeling—power grids with network topology, supply chains with routing decisions, financial systems with order book depth—faces the same information bottleneck. The TotalVariance bound provides a portable diagnostic: before investing in better models, compute how much variance is explained by the features you're not observing.

Innovation 2: Architecture and Output Representation Designed for Regression, Not Language Generation—and the Counterintuitive Finding That Language Pretraining Is Unnecessary (or Harmful)

The paper makes a series of design choices that collectively constitute a principled departure from the "LLM as universal function approximator" paradigm that has dominated recent work. Rather than taking a pretrained language model and adapting it to regression (the natural inclination given the success of LLMs in other domains), the paper builds a regression-specific architecture from scratch and systematically demonstrates that several standard LLM design principles are suboptimal for this task.

The most striking negative result is the explicit rejection of language pretraining. The paper states bluntly: "it is not necessary nor guaranteed beneficial to use a pretrained LLM checkpoint from which to train a regression model" (Section 3.2). This runs counter to the dominant narrative that pretraining on large text corpora provides universal representations that transfer to downstream tasks. The justification is domain-specific but generalizable: regression over structured data (YAML-like system logs) requires learning correlations between tokens whose semantics are defined by the data-generating process (e.g., which hardware platform a job runs on), not by natural language. The semantic meaning of the word "machine" in English is irrelevant; what matters is that machineA and machineD are distinct categories whose relationship to the outcome is learned from data. A pretrained LLM would bring English-language priors about token co-occurrence that may actively interfere with learning these domain-specific correlations.

This is not just a claim—it's validated empirically by the paper's success with a randomly-initialized 60M-parameter model achieving near-perfect predictions. If language pretraining were necessary (or even helpful) for this task, a model two orders of magnitude smaller than typical LLMs, trained tabula rasa, could not achieve 0.99 rank correlation. The fact that it does suggests that the knowledge required for regression over structured data is fundamentally different from the knowledge encoded in language model pretraining.

The second major architectural departure is the use of encoder-decoder architecture for regression, with empirical evidence that it substantially outperforms decoder-only alternatives (Figure 15). This is significant because the LLM field has almost entirely converged on decoder-only architectures (GPT, Llama, Gemma, PaLM). The paper provides a functional explanation: in decoder-only models, causal attention prevents early input tokens from attending to later ones, which is restrictive when the input is a long structured document where later sections (e.g., job profiles) provide context for interpreting earlier ones (e.g., cell identifiers). The encoder's bidirectional attention over the full input resolves this. But more fundamentally, the encoder-decoder design cleanly separates the computational demands of processing a very long input (thousands of tokens of structured state description, requiring rich cross-token interactions) from generating a very short output (6-7 tokens of numeric prediction). Forcing a single decoder stack to handle both tasks—as decoder-only architectures must—creates a representational conflict that hurts performance.

This finding has implications beyond regression. It suggests that the decoder-only dominance in LLMs may be partly an artifact of the text generation use case (where inputs and outputs are both natural language of similar length and complexity), and that tasks with asymmetric input-output structure (long structured input → short structured output) may benefit from separated processing pathways. The paper doesn't develop this into a general claim, but the empirical result is suggestive.

The P10 tokenization for numeric output is a third design choice that distinguishes this work from standard approaches. Prior work on language models for regression has used a variety of output representations: direct value heads (MSE-trained regression layers), tokenization as fixed-precision integers, or natural language number descriptions. The P10 scheme (Charton, 2022) decomposes numbers into sign, mantissa, and exponent tokens, creating a compact vocabulary (~20-30 tokens) that can represent arbitrary floating-point values without normalization. The paper's insight is that this representation is not just convenient—it enables multi-task training across cells with different output ranges without precomputing normalization constants, and it couples naturally with the cross-entropy training objective because each digit gets its own dedicated computation rather than being compressed into a single embedding.

The collective force of these design choices is to define a regression-native language model paradigm that is distinct from both traditional tabular regression and LLM-based few-shot regression. Compared to tabular regression: no feature engineering, no normalization, no vocabulary enumeration for categoricals, no bounding of input cardinality. Compared to LLM approaches: no unnecessary pretraining, no in-context examples consuming the context window, architecture tailored to input-output asymmetry, output representation designed for numeric precision rather than fluency. This establishes RLMs as a distinct point in the design space, not just "applying LLMs to regression."

Innovation 3: The Conceptualization of Multi-Task Pretraining as Implicit Meta-Learning That Operates Through Task-Identifying Features in the Input String

The paper's approach to multi-task learning is architecturally unremarkable—simply concatenating training data from multiple cell-month combinations and training a single model. What's conceptually novel is the mechanism by which the model learns to separate tasks, and the implications of that mechanism for few-shot adaptation.

Standard multi-task learning in neural networks typically uses explicit task identifiers: a one-hot task ID embedding, task-specific output heads, or task-conditional normalization layers. These approaches require the set of tasks to be fixed at training time—adding a new task requires architectural modification (a new output head) and retraining from scratch.

The RLM achieves task separation through a fundamentally different mechanism: the cell name and timestamp, which appear as ordinary tokens in the input string, serve as implicit task identifiers. The model learns during pretraining that the cell name strongly conditions the output distribution—different cells have different characteristic MIPS per GCU ranges, different hardware compositions, different workload patterns—and develops attention patterns and representations that use this feature to "route" the prediction through the appropriate regression function. But unlike a task ID embedding, the cell name is just a string token that the SentencePiece tokenizer can decompose into subwords, meaning that a completely new cell name not seen during pretraining is still processable by the model, and fine-tuning can associate this new name with a new output distribution using the same routing mechanism learned during pretraining.

This is a form of implicit meta-learning, which the paper explicitly connects to MAML (Finn et al., 2017). The pretraining phase optimizes for parameter values that can quickly adapt to new tasks via gradient descent, but unlike MAML, this meta-objective is not explicitly optimized via bi-level gradient computation—it emerges from standard multi-task training. The model's internal representations (how to process YAML-like structure, how to attend to cell identity, how to map hardware profiles to performance outcomes, how to handle variable-length lists of jobs) are shared across all pretraining tasks, so when fine-tuning on a new task, only the task-specific mappings (this new cell's particular output distribution) need to be learned. The shared representations provide a strong initialization that makes few-shot learning effective.

The evidence for this mechanism comes from two findings. First, Figure 7 shows that pretraining on more tasks (1 → 4 → 8 → 16 → 32 cells) substantially improves out-of-distribution fine-tuning performance on a held-out cell, while in-distribution performance (on a cell included in pretraining) shows negligible gains. This is exactly what implicit meta-learning predicts: more pretraining tasks provide better shared representations that transfer to new tasks, while in-distribution tasks already benefit from having their specific data in the pretraining mixture, so additional task diversity doesn't help further.

Second, Figure 20 shows that earlier pretraining checkpoints (10K steps) often produce better fine-tuning results than later ones (50K-100K steps) on out-of-distribution tasks. The paper interprets this as "meta-overfitting"—extended pretraining causes the model to over-specialize to the pretraining tasks, making the shared representations less adaptable to new tasks. This is a known phenomenon in meta-learning (where training the meta-learner to convergence can hurt meta-test performance) but its emergence from standard multi-task training without an explicit meta-objective is notable.

The significance of this innovation is practical: it means that a single pretrained RLM checkpoint can serve as a foundation model for system performance prediction, rapidly adaptable to new cells, new hardware, or new time periods with minimal additional data. This contrasts with the current practice of training separate models per deployment, or using one-size-fits-all approaches that ignore cell-specific variation. It also points toward a future where system operators could deploy a pretrained "base simulator" and customize it for their specific infrastructure with a small amount of local data, analogous to how foundation models in NLP are fine-tuned for downstream tasks.

Standard regression models (linear regression, random forests, MLPs with MSE loss, Gaussian Processes with Gaussian likelihoods) produce a single number as output— $\hat{y}$ —and optionally an uncertainty estimate that typically assumes a Gaussian error distribution. This works well when the conditional distribution $p(y|x)$ is unimodal and symmetric, but it fundamentally cannot represent situations where the same (or indistinguishable) system state can lead to qualitatively different outcomes.

The Borg efficiency metric exhibits exactly this behavior in certain cells (Figures 5, 9, 11): given the same cell, time window, and configuration, the scheduler can produce different MIPS per GCU values depending on factors not captured in the state description—inherent randomness in the bin-packing algorithm, stochastic workload demand, or unobserved state variables. These factors produce genuinely multi-modal outcome distributions, where the same input can lead to efficiency of either ~1200 or ~1800 MIPS per GCU with almost no probability mass in between.

A point regressor trained with MSE would learn to predict the mean of this distribution—say, 1500—which is a value that almost never actually occurs, making it useless for downstream decisions that depend on understanding the full range of possible outcomes. A Gaussian Process would predict a mean of 1500 with high variance, but the Gaussian assumption would misrepresent the actual distribution shape (a single wide Gaussian vs. two separated modes).

The RLM inherently avoids this limitation by learning $p_\theta(y|x)$ as a full density via next-token prediction, without any parametric assumption about the output distribution's shape. The model's decoder learns to produce different token sequences (e.g., <+><1><2><0><0><E+0> vs. <+><1><8><0><0><E+0>) with different probabilities, conditioned on the encoder representation of the input. When the training data contains multiple distinct outcomes for similar inputs, the model learns to assign probability mass to multiple output sequences, naturally capturing multi-modality.

This is not an explicit design goal—the model isn't trained to "be multi-modal"—it's an emergent property of training a conditional language model with a sufficiently rich output space. The key enabling factor is the P10 tokenization combined with the autoregressive decoder: because the output is generated token-by-token rather than in one shot, the model can express complex distributions over numbers through the product of conditional distributions over tokens. A value head trained with MSE could never represent a bimodal distribution; the P10 decoder can, because the first digit token can have high probability for both 1 and 8, leading to two distinct output clusters.

The practical significance is demonstrated in Figure 12, where the RLM achieves high McFadden's Pseudo- $R^2_{\text{NLL}}$ (a density estimation metric) even on tasks where its pointwise explained variance $R^2_{\text{EV}}$ is mediocre. The model may not predict the exact outcome well (because the outcome is inherently random), but it correctly captures how random—the shape, spread, and modality of the outcome distribution. For downstream applications like Bayesian optimization, this is arguably more useful than an accurate point prediction with no uncertainty information.

This capability is particularly important in the paper's motivating use case: tuning Borg scheduler hyperparameters via Google Vizier. A Gaussian Process surrogate model that assumes Gaussian errors would be misled by multi-modal outcomes—it might explore the region around the mean (1500) extensively, never discovering that the true distribution has a high mode at 1800 that could be reliably achieved with slightly different hyperparameters. An RLM that correctly captures the bimodality would enable more sophisticated acquisition functions that explicitly reason about outcome distributions rather than just expected values.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper uses data from Google's Borg compute cluster management system, specifically from the digital twin backtesting framework that simulates scheduling decisions on real cluster checkpoint states. Each "task" is defined as a collection of 28K-56K state-outcome pairs from a specific cell during a specific month (June or November). Data is randomly shuffled into 80/10/10 train/validation/test splits. The paper uses a total pool of 40 cells selected for having the largest spreads of $y$ -values (MIPS per GCU variance), with cells indexed such that $C_1$ has the highest variance (Section 4.1, Appendix B.2, Figure 22).
Base models. The primary model is a Regression Language Model (RLM) using a T5X EncoderDecoder architecture with 2 encoder layers, 2 decoder layers, 16 attention heads, 64 head dimension, 512 embedding dimension, and 2048 MLP dimension, totaling approximately 58M parameters. It is trained from random initialization with no language pretraining. For the Limit Testing configuration (higher-capacity experiments), a larger 267M-parameter variant with 4096 sequence length and batch size 256 is used (Section 3.2, Appendix C.1, C.2). The models are implemented in the open-source T5X framework.
Metrics. The paper uses multiple complementary metrics, chosen for different analytical purposes (Section 4.1). Mean Squared Error (MSE) assesses pointwise prediction precision, interpreted alongside the TotalVariance lower bound to gauge how close performance comes to the theoretical optimum given observed features. Spearman rank correlation ( $\rho$ ) provides a scale-invariant measure of ranking quality, important for optimization applications where relative ordering matters more than absolute values. Validation cross-entropy loss serves as a proxy metric during training for early stopping and ablation comparisons. Explained variance ( $R^2_{\text{EV}} = 1 - \text{MSE}_{\text{RLM}} / \text{MSE}_{\text{null}}$ ) quantifies the fraction of outcome variance captured by the model relative to a null model that predicts without observing $x$ . McFadden's Pseudo- $R^2_{\text{NLL}}$ ( $R^2_{\text{NLL}} = 1 - \text{NLL}_{\text{RLM}} / \text{NLL}_{\text{null}}$ ) analogously measures density estimation quality using negative log-likelihood ratios, where the null model is the RLM itself trained on empty strings (Section 4.2, Appendix A.2).
Baselines. The paper uses two types of theoretically optimal baselines derived from the TotalVariance framework (Equation 2, Section 2.3), not trained models. The tabular baseline computes the optimal MSE achievable by any regressor that observes only the scheduler hyperparameters—a flat tabular feature representation—by grouping test examples into equivalence classes where these features are identical, computing the mean $y$ within each class as the optimal prediction, and measuring the resulting MSE. The null baseline does the same but observes no features at all, using the global mean of $y$ as the prediction. These baselines are theoretical lower bounds: no model, regardless of architecture or training, can achieve lower MSE than these values when restricted to the corresponding feature set (Section 4.1, Figure 6). The paper does not compare against trained tabular models (random forests, MLPs, gradient boosting) or against Gaussian Process regression as a competitor, though Google Vizier's GP is mentioned as motivation (Section 2.1).
Generation budget / compute accounting. The RLM does not use a generation budget in the sense of search-based methods. Instead, compute is measured implicitly through model size, sequence length, and training data volume, all of which affect training and inference FLOPs. At inference time, the model generates 128 parallel samples per input for aggregation, but this cost is described as negligible compared to the 1-18 hours required for a single Borg simulation. The paper emphasizes that the model requires "at most 1 GPU" for training (Section 5.2) and that the 60M-parameter default is "orders of magnitudes lower than state-of-the-art general LLM models within the O(1B) range" (Section 5.2). Sequence length ablations use maximum lengths from 256 to 4096 tokens (Figure 17).
Cross-validation / statistical protocol. Different experiments use different protocols. Limit Testing (Figures 6, 10, 11, 12) trains a single model on approximately 29 cell-month combinations (excluding the highest-spread cells $C_1^{JUN}$ and $C_1^{NOV}$ ), evaluates in-distribution on $C_2^{JUN}$ , and evaluates out-of-distribution on $C_1^{NOV}$ . Adaptation Testing (Figures 4, 5, 7) trains five separate checkpoints on {1, 4, 8, 16, 32} randomly chosen tasks, always including $C_2^{JUN}$ , and evaluates out-of-distribution on $C_{10}^{NOV}$ . Fine-tuning experiments repeat over 10 seeds with different random example selections, reporting means (e.g., Figures 4 right, 7, 19). Results for $R^2_{\text{EV}}$ and $R^2_{\text{NLL}}$ (Figure 12) use 5 seeds. Ablation studies generally report single-run results with validation loss as the measured quantity (Figures 15-18), though some (Figure 16) report the minimum validation loss observed across training. The paper does not use formal cross-validation in the sense of k-fold rotation across tasks; instead, it holds out specific cells and months for out-of-distribution evaluation.

Main Quantitative Results

Case Study: In-Distribution and Few-Shot Performance on High-Spread Tasks

The paper opens its results with an end-to-end demonstration of the RLM's capabilities on the task with the highest spread of $y$ -values (Section 4.2, Figures 4 and 5).

When pretrained simultaneously on 8 tasks (approximately 1M data points or 2B tokens) and evaluated in-distribution, the RLM achieves 0.86 Spearman rank correlation (Figure 4, right panel, "Pretrained" bar in the ID group). The diagonal fit scatter plot (Figure 4, left) shows predictions clustering along the $y = \hat{y}$ line, with the largest residual errors concentrated at extreme $y$ -values—a pattern that the paper attributes to aleatoric uncertainty: outputs sampled from a probability distribution rather than a deterministic function of the observed features.

The most striking result in this figure is the few-shot adaptation capability. Using a pretrained checkpoint fine-tuned on only 512 examples from a completely new out-of-distribution task, the RLM achieves a rank correlation comparable to the fully pretrained in-distribution result (Figure 4, right, "Pretrained + Finetuned (512)" bar in the OOD group). In contrast, a randomly initialized model given the same 512 examples ("Random Init + Finetuned (512)") achieves dramatically lower performance, demonstrating that the pretrained representations—not the fine-tuning procedure itself—drive the adaptation. A randomly initialized model given access to the full training data but without pretraining ("Random Init") shows intermediate performance, better than few-shot from scratch but worse than pretrained few-shot, highlighting the value of multi-task pretraining even when in-domain data is available.

The paper further demonstrates the RLM's density estimation capability in Figure 5, which visualizes the model's output distribution $p_\theta(y|x)$ across varying timestamps for a single cell. The KDE plot of 128 samples per input shows that even though each individual $x$ in the training data maps to a unique timestamp with a single observed $y$ , the model's learned density captures multiple modes—the samples cluster around different efficiency levels at different times, reflecting the model's generalization from similar timestamps and its learned understanding that certain inputs are inherently ambiguous. This is not trivially explained by the training data containing multiple $y$ -values per $x$ , because the RLM's rich representation $\phi(x)$ makes each training example nearly unique. Instead, the multi-modality emerges from the model's inductive biases and the continuous representations it learns for the input space.

Comparison Against Theoretically Optimal Tabular Baselines

Figure 6 provides the paper's most dramatic quantitative result: a 100× reduction in MSE compared to what any tabular regressor can theoretically achieve. The left panel shows the in-distribution case ( $C_2^{JUN}$ ), the right panel shows out-of-distribution ( $C_1^{NOV}$ ).

Both panels display histograms of per-sample squared errors on log-log axes, with vertical lines indicating the mean. Three distributions are overlaid: the RLM's actual residuals, the residuals from the theoretically optimal predictor that observes only tabular hyperparameter features, and the residuals from the null model that observes no features. In the in-distribution case, the RLM's MSE line falls approximately two orders of magnitude (100×) to the left of the tabular-optimal line. The distribution of RLM residuals is also far more left-skewed—most predictions have very small errors—while the tabular-optimal distribution shows a heavy tail of large errors representing the irreducible variance from states that look identical under tabular features but have different outcomes.

The out-of-distribution case (right panel) shows a qualitatively similar pattern but with somewhat degraded RLM performance—the gap between RLM and tabular-optimal narrows but remains large. This confirms that even on an unseen cell, the text-based representation provides substantial information that tabular features lose.

The epistemological force of Figure 6 is that it compares the RLM's actual performance not against other trained models but against theoretical ceilings. The tabular line is not "how well a random forest did"—it's "the best any model could possibly do with these features." The fact that the RLM substantially beats this ceiling for tabular features demonstrates that the information contained in the full string representation is not just "more convenient" but qualitatively richer—it enables accurate predictions that are mathematically impossible with compressed features regardless of modeling sophistication.

Pretraining Diversity and Transfer Learning

Figure 7 quantifies the value of large-scale multi-task pretraining for few-shot adaptation. The experiment trains five separate checkpoints on 1, 4, 8, 16, or 32 different tasks (all from distinct cells, always including $C_2^{JUN}$ for in-distribution evaluation), then fine-tunes each on varying numbers of examples (0 to 512) from an out-of-distribution task ( $C_{10}^{NOV}$ ). Each fine-tuning run is repeated over 10 seeds and averaged.

The in-distribution results (evaluated on $C_2^{JUN}$ , the one cell common to all checkpoints) show negligible improvement from pretraining on more tasks—the curve for 1-cell pretraining is nearly identical to 32-cell pretraining across all fine-tuning example counts. This makes sense: the model already sees $C_2^{JUN}$ 's full training data during pretraining (when $N \geq 1$ ), so additional tasks don't add information about this specific cell's regression function. The few-shot adaptation here primarily helps at very low example counts (4-8), where fine-tuning refocuses the model on this specific task's distribution, but with sufficient in-distribution data the benefit is marginal.

The out-of-distribution results tell a completely different story. With zero fine-tuning examples (pure zero-shot transfer), the 1-cell checkpoint achieves near-zero rank correlation on $C_{10}^{NOV}$ —the model has never seen this cell and cannot generalize. The 4-cell checkpoint does somewhat better, and performance improves monotonically with more pretraining tasks, with the 32-cell checkpoint achieving the highest zero-shot performance. This confirms that multi-task pretraining teaches the model representations that transfer across cells—how to process YAML-like structure, how to attend to hardware profiles, how to map job characteristics to efficiency—even when the specific cell identity is new.

The gap between pretraining-task counts is largest at low fine-tuning example counts and narrows as more examples are provided. The 32-cell checkpoint with 64 fine-tuning examples achieves roughly the same performance as the 1-cell checkpoint with 512 examples—an effective 8× data efficiency gain from pretraining on more tasks. With 512 fine-tuning examples, all checkpoints (even 1-cell) converge to similar performance, suggesting that 512 examples is sufficient to learn the target cell's regression function from scratch given the shared representations learned during pretraining.

Uncertainty Quantification and Density Estimation

Figures 8 and 9 validate the RLM's uncertainty quantification capabilities, which are critical for downstream applications like Bayesian optimization where knowing when to trust a prediction matters as much as the prediction itself.

Figure 8 plots the sample standard deviation of the RLM's 128 predictions per input (y-axis) against the squared prediction error (x-axis), using a fine-tuned model on in-distribution task $C_2^{JUN}$ . There is a clear positive correlation—predictions with higher variance tend to be less accurate, as measured by squared error. This means the model "knows when it doesn't know": the variance of $p_\theta(y|x)$ serves as a genuine uncertainty signal that correlates with actual prediction quality. For Bayesian optimization, this enables principled exploration-exploitation tradeoffs—the optimizer can avoid trusting predictions with high variance and instead gather more data in those regions.

Figure 9 provides a different view: for a single input $x$ where the ground-truth $y$ -values (observed at different times with the same timestamp feature) exhibit bimodality, the RLM's sampled predictions also show two distinct modes. The histogram of 128 generated samples shows clustering around two separate efficiency levels, with the ground-truth values falling within these clusters. The RLM has not just learned a point estimate or a Gaussian approximation—it has learned that this particular input configuration is consistent with two qualitatively different outcomes, and it assigns probability mass to both. This is impossible for any regressor that assumes a unimodal error distribution.

Comprehensive Multi-Task Results

Figures 10 and 11 present a broad evaluation across many cell-month tasks, demonstrating both point prediction and density estimation capabilities in a single pretrained-then-fine-tuned model (Limit Testing configuration, 267M parameters, fine-tuned on 512 examples per task).

Figure 10 shows scatter plots of predictions vs. ground truth across multiple tasks (each subplot is one task). The diagonal fit varies substantially across tasks: tasks like $C_4^{NOV}$ , $C_{15}^{NOV}$ , and $C_{23}^{NOV}$ show near-perfect alignment along the diagonal, indicating very high precision. Other tasks like $C_1^{NOV}$ , $C_{16}^{NOV}$ , and $C_{24}^{NOV}$ show wider scatter, reflecting higher noise in these cells. The Spearman rank correlations for the majority of tasks exceed 0.93, with some approaching 0.99—the paper's headline number of "near perfect 0.99 rank correlation" comes from these results. The variation across cells is informative: it shows that the RLM's performance is bounded by the inherent predictability of each task (the aleatoric noise level), not by model capacity or feature observability.

Figure 11 shows KDE density plots for the same tasks. Tasks with tight scatter plots in Figure 10 (like $C_4^{NOV}$ ) show unimodal, concentrated densities in Figure 11. Tasks with noisy scatter plots (like $C_1^{NOV}$ , $C_{22}^{NOV}$ , $C_{24}^{NOV}$ ) show multi-modal or broad densities, and the RLM's learned density (blue) captures the shape of the empirical target distribution (orange) remarkably well—including asymmetries, multiple modes, and varying spreads. This demonstrates that the RLM is not just a good point predictor but a good density estimator, learning the full $p(y|x)$ rather than just its conditional mean.

Figure 12 quantifies these capabilities across 40 tasks using $R^2_{\text{EV}}$ (explained variance for point prediction) and $R^2_{\text{NLL}}$ (Pseudo- $R^2$ for density estimation), sorted by decreasing rank correlation. The results reveal an asymmetric capability: on tasks where the RLM achieves high pointwise accuracy (e.g., $C_4^{NOV}$ with $R^2_{\text{EV}} \approx 1$ ), it also achieves high density estimation quality. But the reverse is not necessarily true—tasks with lower $R^2_{\text{EV}}$ (like $C_{21}^{NOV}$ , $C_{24}^{JUN}$ , $C_{22}^{NOV}$ ) can still achieve reasonably high $R^2_{\text{NLL}}$ , meaning the model correctly captures the distribution of outcomes even when it cannot predict exact values. This is precisely the behavior needed for stochastic optimization: even when point predictions are noisy, knowing the distribution of possible outcomes enables robust decision-making.

The overall pattern is clear: the RLM achieves $R^2_{\text{EV}}$ above 0.8 on most tasks and above 0.95 on the most predictable ones, with Spearman rank correlations exceeding 0.9 for the majority of the 40 evaluated cells. These results represent the aggregated performance after fine-tuning on 512 examples per task from a single pretrained checkpoint.

Ablation Studies and Robustness Checks

Cross-entropy loss as a proxy for regression metrics: Figure 13 shows that validation cross-entropy loss correlates directly with test MSE across training checkpoints—lower validation loss implies lower MSE. Underfitted checkpoints (early in training) have high loss and high MSE; properly fitted checkpoints (near the validation loss minimum) have both low loss and low MSE; overfitted checkpoints (past the minimum) show increasing loss and increasing MSE. Figure 14 adds an interesting nuance: Spearman rank correlation shows a different relationship. Overfitted checkpoints can maintain high rank correlation even as MSE degrades, meaning the model's relative ordering of predictions remains accurate even when the absolute predictions drift. This is practically significant for optimization applications where ranking matters more than exact values—it suggests early stopping based on validation loss may be too conservative if only rank correlation is needed (Section 5.1, Figures 13-14).

Architecture: encoder-decoder vs. decoder-only: Figure 15 compares four configurations with approximately equal parameter counts: 0E4D (decoder-only, 62.3M params), 1E3D (60.2M), 3E1D (58M), and 2E2D (56M). The 2E2D configuration (the default) achieves the lowest validation loss. The decoder-only model (0E4D) is substantially worse, and performance generally improves as encoder layers increase relative to decoder layers. The paper hypothesizes that the encoder's bidirectional attention is necessary for processing complex structured inputs where later sections provide context for interpreting earlier ones—causal attention in decoder-only models prevents early tokens from attending to later context (Section 5.2, Figure 15). Minimum validation loss was observed at steps 23K (2E2D), 17K (3E1D), 17K (1E3D), and 18K (0E4D).

Model size scaling: Figure 16 shows that increasing model size from 45.5M (2 layers) to 234.3M (32 layers) reduces validation loss, but with strongly diminishing returns. The 45.5M model achieves substantially worse loss than 58.1M; 58.1M is only slightly worse than 83.2M; and 234.3M offers modest improvement over 83.2M. The minimum losses were achieved at steps 27K, 16K, 13K, and 11K respectively, showing that larger models converge faster (in terms of steps) but not necessarily to substantially better solutions. This supports the paper's claim that model size "quickly plateaus within the O(100M) range" (Section 5.2, Figure 16). The experiments in this ablation use a reduced sequence length of 1024 (rather than the default 2048), which may compress the differences somewhat, but the diminishing-returns pattern is clear.

Sequence length and feature observability: Figure 17 trains models with maximum sequence lengths of 256, 512, 768, 1024, 2048, and 4096 tokens on the Ablation setting (7 high-spread June cells). Validation loss consistently decreases as sequence length increases, with the largest gains between 256 and 1024, and diminishing returns beyond approximately 3000 tokens. The paper explains that "the additional tokens mostly come from the last remaining, longest, and least important features, specifically the job-on-machine performance mentioned in Table 1" (Section 5.3). This directly validates the epistemic uncertainty argument: observing more features—even ones presumed to be less important—provides measurable predictive value by reducing the variance within equivalence classes of inputs that look identical to the model. The minimum validation losses were achieved at steps 12K (256), 13K (512), 18K (768), 16K (1024), 23K (2048), and 26K (4096), with longer sequence lengths requiring more training steps to converge.

Feature ablation by category: Figure 18 isolates the importance of specific feature categories by training models with different input subsets: "R" (all remaining features except cell and window), "CR" (cell + remaining features, no time window), and "WR" (window + remaining features, no cell). All models use sequence length 1024 and default architecture. The "WR" configuration (no cell identity) achieves the worst validation loss—0.230 at step 20K. The "CR" configuration (no time window) achieves intermediate loss—0.224 at step 22K. The "R" configuration (both cell and time window ablated) achieves 0.222 at step 23K. The differences are modest in absolute terms but consistent with domain expectations: the time window feature captures temporal cycles (fewer jobs at night, weekday/weekend patterns) and its removal hurts performance measurably; the cell feature captures persistent differences between physical clusters. The paper notes that "the model's behavior based on observing certain features aligns with our expectations from domain knowledge" (Section 5.3).

Fine-tuning learning rate sensitivity: Figure 19 sweeps learning rates from $5 \times 10^{-3}$ to $1 \times 10^{-6}$ for fine-tuning on an out-of-distribution task ( $C_1^{NOV}$ ), using a checkpoint at 10K pretraining steps and varying numbers of fine-tuning examples (0 to 256). The results reveal a dependency on dataset size: with very few examples (4-8), the optimal learning rate is relatively high ( $5 \times 10^{-4}$ to $1 \times 10^{-3}$ ), suggesting that aggressive adaptation helps when data is extremely scarce. With more examples (64+), lower learning rates ( $5 \times 10^{-5}$ to $1 \times 10^{-4}$ ) are optimal. Learning rates at the extremes ( $5 \times 10^{-3}$ and $1 \times 10^{-6}$ ) perform poorly across all data sizes—the former likely causes catastrophic forgetting of pretrained representations, the latter provides insufficient updates to adapt. The default fine-tuning learning rate of $5 \times 10^{-5}$ is near-optimal for moderate example counts (32-256) but would be suboptimal for very few-shot settings (Section 5.4, Figure 19). Results are averaged over 10 fine-tuning seeds.

Pretraining checkpoint selection for fine-tuning: Figure 20 evaluates how the choice of pretraining checkpoint affects downstream fine-tuning performance on an out-of-distribution task ( $C_1^{NOV}$ ), using a fixed learning rate of $5 \times 10^{-5}$ and checkpoint steps from 1K to 100K. The key finding is that earlier checkpoints (5K-10K steps) often produce better fine-tuning results than later ones (50K-100K), particularly at low example counts (4-32). The paper terms this "meta-overfitting"—prolonged pretraining causes the model to specialize to the pretraining tasks, making representations less adaptable to genuinely new tasks. With 128-256 fine-tuning examples, the checkpoint choice matters less because sufficient data exists to overcome the specialization. With zero fine-tuning examples (zero-shot transfer), performance is near-zero regardless of checkpoint, confirming that some adaptation is always needed for unseen cells (Section 5.4, Figure 20). This ablation is based on a single seed due to computational scale.

Critical Assessment

Does the RLM Achieve "Near Perfect" Prediction?

The paper's headline claim of "up to a near perfect 0.99 (0.9 average) rank correlation across the entire fleet" (Abstract) and "100x lower MSE than tabular approaches" (Abstract, Figure 6) requires careful unpacking. The 0.99 figure appears in Figure 10 for specific tasks like $C_4^{NOV}$ , but Figure 12 shows that this is the best-case scenario, not the typical one. Many tasks show rank correlations between 0.8 and 0.95, with a few dipping lower. The "0.9 average" appears consistent with the distribution of scores in Figure 12, though the paper does not explicitly compute this average or report confidence intervals around it. The 100× MSE reduction in Figure 6 is computed against a theoretical bound for tabular features, not against an actual trained tabular model. This is a valid and informative comparison—it demonstrates that even a perfect tabular model would underperform the RLM—but it's different from showing that the RLM is 100× better than existing deployed tabular models, which might already incorporate some of the richer features through clever engineering.

The performance is also uneven across tasks—a point the paper acknowledges through its distinction between pointwise and density estimation metrics. Tasks with high aleatoric noise achieve lower $R^2_{\text{EV}}$ (pointwise accuracy) but compensate with high $R^2_{\text{NLL}}$ (density capture). This is a legitimate feature of the approach, but it means that "near perfect" applies to the ranking and distribution modeling rather than to exact value prediction on all tasks. For practitioners expecting consistently high pointwise accuracy, tasks like $C_{22}^{NOV}$ or $C_{24}^{JUN}$ (with $R^2_{\text{EV}} \approx 0.4-0.6$ ) would be disappointing, even if the density estimates are good. The paper could strengthen its case by more explicitly characterizing which types of cells or conditions lead to lower pointwise accuracy and whether those conditions are common or rare in practice.

Are the TotalVariance Baselines Fair?

The 100× MSE comparison in Figure 6 comes with an important qualification. The tabular baseline uses only scheduler hyperparameters as features. In practice, a determined engineer building a tabular model would almost certainly include more features—cell identity (one-hot encoded), temporal features (hour of day, day of week from the timestamp), aggregate statistics of hardware distributions (counts by platform type, total resources), and possibly summary statistics of job profiles (mean MIPS, count of jobs). The paper's point is that including all these features in a fixed-length vector is difficult and brittle, but it's not impossible for a motivated practitioner. A more compelling baseline would be the "best effort" tabular model that an experienced ML engineer could build given a day or two of feature engineering—or, at minimum, a sensitivity analysis showing how the TotalVariance bound changes as more tabular features are added (e.g., cell identity alone, cell + temporal, cell + temporal + hardware aggregates, cell + temporal + hardware + job summary statistics). The current comparison demonstrates the ceiling for a minimal tabular representation, but the practical gap between the RLM and a well-engineered tabular model may be smaller than 100×.

That said, the TotalVariance framework itself is a genuine contribution—the idea that you can compute a theoretical lower bound for any feature representation without training a model is practically useful for deciding whether feature engineering efforts are worthwhile. The paper could strengthen this contribution by demonstrating the TotalVariance computation protocol more concretely (how many equivalence classes, what's the typical class size, what's the variance within classes) and by comparing the bound against actual trained tabular models (to validate that the bound is tight and that real models approach it).

Missing Baselines and Ablations

Several comparisons would meaningfully strengthen the paper's claims:

No comparison against trained tabular models. The paper never trains a random forest, gradient boosting model, or MLP on any feature representation and compares its performance against the RLM. The gap between the TotalVariance bound for tabular features (which assumes an optimal regressor) and an actual trained model could be substantial, meaning the RLM's advantage over deployed systems might be even larger than 100×—or it could be smaller if the bound is loose. Without this comparison, the reader cannot calibrate how much of the RLM's advantage comes from richer features vs. from a fundamentally better modeling approach (density estimation vs. point prediction, multi-task transfer vs. task-specific training).

No comparison against a pretrained LLM checkpoint fine-tuned for regression. The paper argues that language pretraining is unnecessary or harmful, but this claim is supported only by the success of the randomly-initialized model—not by a head-to-head comparison against a T5, BART, or even a small Llama checkpoint fine-tuned on the same data. Given the prevalence of "just fine-tune an LLM" as a baseline in applied ML, this is a notable gap. A small T5 model (e.g., T5-Small at 60M parameters) trained from a pretrained checkpoint on the same Borg data would provide direct evidence for or against the paper's claim. The conceptual argument that pretraining on natural language is irrelevant for structured YAML data is plausible but would be strengthened by empirical validation.

No ablation on the number of inference samples (128). The paper uses 128 parallel samples per input for inference but never ablates this choice. Given that the sample mean's standard error decreases as $1/\sqrt{s}$ , 128 samples gives about 11× improvement over a single sample—but is this necessary? Would 32 samples (about 5.7× improvement) give comparable performance? Would 512 samples (16× improvement) give meaningful additional gains? For a paper that emphasizes practical deployment, the tradeoff between inference cost (encoder processes the input once, but decoder runs $s$ times) and prediction quality is important to characterize.

Limited exploration of the input representation. The YAML-like serialization format (Figure 21) is specific and somewhat arbitrary—features are organized in a particular order with particular separators. How sensitive is performance to the serialization format? Would JSON work as well? What about a more compressed representation (removing whitespace, using shorter key names)? What about a different ordering of features? The paper notes that domain knowledge can be used to "efficiently compress string representations" and "place the presumably most important features at the beginning" (Section 3.4), but never empirically validates these claims. This matters because other practitioners adopting the approach will need to make serialization design decisions without clear guidance.

Generalizability Concerns

Single system, single metric, single company. All experiments are on Borg's MIPS per GCU metric at Google. While the paper frames this as a case study meant to "serve an impression of what results may appear when text-to-text regression is applied to any large system" (Section 4), there is no evidence that the approach generalizes to other systems, other metrics, or other organizations' infrastructure. The Borg system has particular properties—a digital twin that generates training data, a relatively stable relationship between state and outcome, structured logging of nested features—that may not hold in other settings. A system with less structured logs, more complex state-to-outcome relationships, or no digital twin for generating training data might pose challenges the paper does not address.

No evaluation of temporal generalization beyond the two-month window. The data comes from June and November. The paper evaluates out-of-distribution adaptation across cells and across months (November tasks are OOD when trained only on June), but never evaluates on data from substantially later time periods (e.g., a model trained on June tested on the following March). Given that the paper emphasizes "platform upgrades and hardware changes" as a motivation for text-based regression's flexibility (Section 2.4), evaluating how well the model handles genuine distribution shift over longer time horizons—when new hardware platforms, new workload types, or new scheduling policies emerge—would be valuable.

Performance on edge cases. The RLM filters predictions outside [500, 3000] MIPS per GCU, but the paper doesn't report what fraction of samples get filtered, what causes the decoder to generate out-of-range values, or whether this filtering introduces bias (e.g., does the model systematically produce extreme predictions on certain types of inputs, and does filtering them shift the mean in a non-representative way?). Understanding these edge cases matters for safety-critical applications where a badly wrong prediction could lead to poor scheduling decisions.

The "task" definition may not capture all meaningful variation. Each task is defined as a cell-month combination. But within a month, workload patterns, hardware configurations, and user behavior can shift. The 80/10/10 random split across all examples from that cell-month means that training and test data may come from the same week or even the same day. This makes the evaluation generous—it tests interpolation within the observed time period rather than extrapolation to future periods within the same cell. A temporal split (train on early June, test on late June, or train on June, test on July) would provide a more realistic assessment of how the model would perform in deployment, where predictions are always about the future.

Strength of the Epistemic Uncertainty Argument

The paper's conceptual contribution around epistemic uncertainty and the TotalVariance bound is elegant, but the empirical validation has a limitation. The TotalVariance bound for "tabular features" uses only scheduler hyperparameters. But the definition of what counts as "tabular" is a spectrum, not a binary. A clever engineer could represent cell identity as a categorical feature, extract hour-of-day and day-of-week from timestamps, compute aggregate statistics of the job profiles (mean/median/percentile MIPS, count of jobs, entropy of platform distribution), and flatten the hardware distribution into counts per platform type—all within a fixed-length vector. The question is not whether tabular features can capture everything (they can't—variable-length lists of arbitrary cardinality are the fundamental limitation), but how much of the 100× gap comes from genuinely non-tabular structure vs. from the paper's choice to not include features that are tabularizable in the baseline.

This doesn't invalidate the paper's argument—the job-on-machine performance section is genuinely variable-length and nested in ways that resist clean tabular representation—but it makes the 100× figure somewhat rhetorical. A more rigorous approach would compute the TotalVariance bound for a progression of increasingly rich tabular representations: hyperparameters only → hyperparameters + cell → hyperparameters + cell + temporal → + hardware aggregates → + job summary statistics. As each layer of features is added, the bound should decrease, but never reach the text-based level because some information (which specific job runs on which specific platform with which specific profile) is truly non-tabular. This would provide a more nuanced picture of how much of the RLM's advantage comes from information that is genuinely impossible to represent in tables vs. information that is merely inconvenient.

What the Results Actually Demonstrate

The paper's central claim is that text-to-text regression is a "general, scalable alternative" to tabular regression for systems performance prediction. The results genuinely support several narrower claims:

For predicting Borg MIPS per GCU, observing the full string representation enables substantially better predictions than observing only scheduler hyperparameters. This is robustly demonstrated by Figure 6 and supported by the feature ablation in Figure 18 showing that removing features degrades performance.
Multi-task pretraining on many cells improves few-shot adaptation to unseen cells compared to single-task pretraining. Figure 7 demonstrates this cleanly, and Figure 20 reinforces it with the meta-overfitting analysis.
A small encoder-decoder model trained from scratch can achieve strong regression performance without language pretraining. Figures 10-12 demonstrate this across many tasks, and Figures 15-16 provide architectural justification.
Token-level cross-entropy training naturally enables density estimation and uncertainty quantification. Figures 5, 8, 9, and 11 demonstrate multi-modal density capture and the correlation between predicted variance and actual error.

What the results do not demonstrate—or demonstrate only weakly—is:

That text-to-text regression is better than any tabular approach an expert could build (vs. a minimal-feature tabular bound)
That the approach works for systems other than Borg or for metrics other than MIPS per GCU
That the approach is robust to long-term temporal distribution shift
That language pretraining is actively harmful (vs. simply unnecessary, which the results do support)
That the specific architecture choices (encoder-decoder, 2 layers, Adafactor, P10 tokenization, 128 samples) are near-optimal rather than merely sufficient

The paper's title and framing aim for generality, but the evidence supports a more modest—though still significant—conclusion: on the specific problem of predicting Borg cluster efficiency, text-to-text regression with an encoder-decoder model and rich feature serialization achieves accuracy levels that would be difficult or impossible to match with standard tabular methods, and the approach transfers well across cells with minimal additional data. This is a valuable contribution, but the jump to "universal simulators" is aspirational rather than demonstrated.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Unaccounted for in the Headline Gains

The entire compute-optimal framework depends on knowing each prompt's difficulty before allocating the inference budget. The paper's method for estimating difficulty—generating 2048 samples per question and computing either ground-truth pass@1 (oracle) or average PRM final-answer score (predicted)—is extraordinarily expensive. At 2048 samples per question, the difficulty estimation step alone consumes more compute than the largest test-time budgets being optimized (256 or 512 generations).

The paper acknowledges this explicitly:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity" (Section 3.2)

The consequence is that the reported 4× efficiency gains over best-of-N are computed after difficulty is known, without amortizing the cost of learning it. In a realistic deployment, the total cost would be difficulty estimation plus strategy execution, and the former could dominate the latter—potentially eliminating or reversing the claimed advantage. For a system that processes many prompts (where the difficulty estimation cost could be amortized across batches of similar questions), this may be manageable; for a system answering diverse one-off queries, the overhead is prohibitive.

The paper demonstrates that predicted difficulty bins (using PRM scores instead of ground-truth labels) largely track oracle bins (Figures 4 and 8, curves overlapping), which removes the circularity concern but does not address the computational cost. The difficulty estimation requires running the base LLM for 2048 forward passes per prompt, plus a PRM scoring pass for each generated sample. No experiment measures how performance degrades if difficulty is estimated from far fewer samples (e.g., 8, 16, or 64), or if a lightweight classifier is trained to predict difficulty directly from the prompt text without sampling at all.

Mitigation status: The paper flags this explicitly as an avenue for future work (Section 8), suggesting "pretraining or finetuning models to directly predict difficulty of a question" or using the PRM's average score from far fewer samples. But no such method is developed, trained, or evaluated. Until this gap is closed, the 4× efficiency figure should be understood as an upper bound on achievable gains rather than a realized deployment improvement. The paper frames the current approach as a proof-of-concept for the compute-optimal strategy; a practical system would require cheap difficulty estimation that the paper does not provide.

Hard Problems Remain Essentially Unsolved Regardless of Compute Budget

Across every method studied—search, revisions, and their compute-optimal combinations—the hardest questions (difficulty bin 5, where the base model's pass@1 is near zero) show no meaningful improvement from test-time compute.

The evidence is consistent and stark. In Figure 3 (right, Section 5.3), bin 5 accuracy hovers at 1–3% for all methods (best-of-N weighted, beam search, lookahead search) and all generation budgets (4 to 256). In Figure 7 (right, Section 6), bin 5 shows roughly 2–3% accuracy irrespective of the sequential-to-parallel ratio. In Figure 8 (Section 6), the compute-optimal revision curve for bin 5 is essentially flat near 5% even at 256 generations. And in the FLOPs-matched comparison (Figure 9, Section 7), the bin 5 scaling line is near 0–5% and far below the ~14× larger model's performance line for all three values of the inference-to-pretraining ratio $R$ .

The paper is transparent about this, stating in the Section 7 takeaway box:

"On the hardest questions (bin 5), no method makes meaningful progress—the base model simply lacks the capability to produce correct solutions regardless of how the budget is allocated."

The consequence is a fundamental capability boundary: test-time compute amplifies existing capability (making a model more likely to produce a correct answer it can already generate at some non-zero rate) but cannot create capability from nothing. If the base model's pass@1 is near zero on a problem class, no amount of search, revision, or adaptive allocation will help—there are no correct solutions in the proposal distribution to find or refine. This means the approach offers no path forward for genuinely novel or out-of-distribution reasoning that exceeds the base model's training distribution. For such problems, scaling pretraining (training a larger model, or training on more data that includes these problem types) remains the only viable path.

Mitigation status: Not addressed and likely not addressable within the test-time compute paradigm. The paper acknowledges this limitation implicitly through its difficulty-bin analysis—the compute-optimal policy is designed to detect hard problems and not waste compute on them, not to solve them. The paper does not suggest any method for breaking through this ceiling, and it is arguably a fundamental constraint: you cannot search for or refine solutions that the model is incapable of generating.

The ~14× Larger Model Baseline Is Not Compute-Optimal and Has No Test-Time Compute of Its Own

The FLOPs-matched comparison in Section 7, which demonstrates that test-time compute with a smaller model can outperform a ~14× larger model, uses a pretraining baseline that is weaker than it needs to be in two distinct ways.

First, the larger model is trained by scaling only parameters while holding training data fixed, following the LLaMA paradigm (Touvron et al., 2023). The paper explicitly acknowledges that this departs from compute-optimal pretraining:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work." (Section 7)

A Chinchilla-optimal model trained with ~14× more total FLOPs (scaling both parameters and data jointly) would likely outperform a parameter-only-scaled model on many tasks, making the pretraining baseline stronger and potentially narrowing or reversing the reported advantages of test-time compute. The magnitude of this effect is unknown because the paper does not run the comparison.

Second, and perhaps more importantly, the ~14× larger model uses only greedy decoding with no test-time compute augmentation whatsoever—no majority voting, no best-of-N, no search. The paper's core argument is that test-time compute can substitute for pretraining compute. But the comparison is asymmetric: the smaller model gets sophisticated compute-optimal test-time scaling, while the larger model gets none. A fairer comparison would give the larger model a test-time compute budget proportional to its higher per-token inference cost within the FLOPs-matched framework—for example, if the larger model's inference costs ~14× more FLOPs per token, it would get proportionally fewer generations, but it would still get some. The paper effectively compares a small model with inference-time optimization against a large model with greedy decoding, which conflates two separate questions: (1) does test-time compute help? (2) is test-time compute better than pretraining compute? A positive answer to (1) does not imply a positive answer to (2) if the baseline large model is not given the same inference-time tools.

Mitigation status: The paper acknowledges the parameter-only scaling departure but not the greedy decoding asymmetry. The choice is described as "representative" but the absence of any test-time augmentation for the larger model makes the headline comparison (Figure 1 bar charts, Figure 9) somewhat misleading. A reader might reasonably conclude that test-time compute with a small model is generally preferable to a larger model, when the actual demonstrated result is that test-time compute with a small model is preferable to a larger model with no test-time compute, which is a weaker claim. No ablation studies the effect of giving the larger model even a modest test-time budget (e.g., best-of-4 or majority voting).

The Revision Model Has a 38% Correct-to-Incorrect Reversion Rate and Is Fragile to Training Methodology

The iterative revision approach (Section 6) shows promising results but has a fundamental reliability problem: the model is trained only on sequences where all in-context answers are incorrect followed by a correct target. At deployment time, when the model generates a chain of revisions, it inevitably encounters correct answers in its own context (produced during earlier successful revisions). Having never seen this situation during training, the model often incorrectly "revises" a correct answer into an incorrect one.

The paper quantifies this:

"approximately 38% of correct answers get converted back to incorrect ones using a naive approach" (Section 6.1)

This is not a minor edge case—it means that for any given correct answer produced mid-chain, there is a ~38% chance the next revision step will destroy it. The paper mitigates this by using selection across the chain (majority voting or verifier-based selection) rather than taking the final revision, but this is a patch that works around the problem rather than solving it. The selection mechanism must correctly identify which answer in the chain is correct, which becomes harder as the chain length grows and more revisions (potentially including incorrect ones that look plausible to the verifier) are accumulated.

The fragility of the revision approach is further demonstrated by the negative result with ReST $^{EM}$ training (Appendix K, Figure 16). Attempting to optimize the revision model with on-policy RL-style training caused performance to substantially degrade with sequential revisions—at 256 generations, fully sequential performance dropped to roughly 33.5%, compared to roughly 38.5% at the optimal sequential-to-parallel ratio. The paper hypothesizes that "the on-policy data collection in ReST $^{EM}$ exacerbates spurious correlations in revision data, causing the model to fail to learn the revision task properly." This suggests the revision training procedure is sensitive to specific design choices (offline data construction, edit-distance-based incorrect-correct pairing, number of incorrect answers in context) that are not fully understood, and that the positive results depend on these choices in ways that may not transfer to other settings.

Mitigation status: Partially addressed through within-chain selection (majority voting or verifier-based selection across the revision chain rather than always taking the final output). However, this treats the symptom (correct answers being overwritten) rather than the cause (the model not knowing when to stop revising). A more principled solution—such as training the revision model on mixed trajectories that include correct-to-correct transitions, or adding a stopping criterion—is not explored. The ReST $^{EM}$ failure is presented as a cautionary result but not investigated in depth. The paper does not provide guidance on how to make revision training robust beyond the specific recipe used.

Single Benchmark, Single Model Family, Small Test Set

All experiments in the paper use a single benchmark (MATH, 500 test questions) with a single model family (PaLM 2-S*). The authors state they "believe this model is representative of the capabilities of many contemporary LLMs" (Section 4), but this claim is unverified and several aspects of the findings could be model-specific or benchmark-specific.

The PRM's quality and over-optimization behavior depend on PaLM 2-S*'s output distribution—a model with different calibration properties, different error patterns, or different pass@1 rates might exhibit different difficulty-dependent scaling curves, different optimal strategy allocations, and different over-optimization thresholds. The revision model's ability to learn from incorrect in-context examples depends on the base model's in-context learning capabilities, which vary substantially across model families. The MATH benchmark consists exclusively of competition-level math problems requiring symbolic reasoning—it is unclear whether the difficulty-dependent patterns (beam search hurting easy problems, revisions helping easy problems, no method helping hard problems) generalize to other reasoning domains like code generation, logical reasoning, scientific QA, or tasks requiring factual knowledge rather than inference.

The test set of 500 questions, when split into five difficulty quintiles of ~100 questions each, and then further split by two-fold cross-validation, means the compute-optimal policy is selected based on approximately 50 questions per fold per bin. This is a small sample for strategy selection, and the selected policies may not be robust. The paper does not report confidence intervals on the compute-optimal scaling curves (Figures 4, 8), making it difficult to assess whether the observed differences between strategies at specific budget levels are statistically reliable. Strategy selection that is optimal for a 50-question fold may not be optimal for the full distribution, and the $4\times$ efficiency gain figure is based on point estimates without quantification of variance.

Mitigation status: The authors acknowledge the single-benchmark limitation but do not address it beyond stating their belief in the model's representativeness. No experiments on other benchmarks (e.g., GSM8K, HumanEval, MMLU) or other model families are conducted or discussed as planned future work. The test set size and cross-validation protocol are described transparently, but no sensitivity analysis (e.g., reporting confidence intervals across cross-validation folds, or testing whether the selected strategies remain optimal when the number of bins or the number of folds is varied) is provided.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper changes the conversation around system performance prediction from a feature-engineering problem to an information observability problem. Before this work, the dominant paradigm for predicting metrics like cluster efficiency was straightforward but limiting: an expert manually designs a fixed-length feature vector capturing what they believe matters about the system, then trains a tabular regressor (random forest, MLP, Gaussian Process) on those features. When performance is poor, the default response is to try a fancier model—deeper networks, gradient boosting, better kernels. This paper argues that the bottleneck is often not the model at all, but the information content of the input representation, and it provides two tools that make this argument operational: the TotalVariance framework for computing theoretical performance ceilings given a feature set, and text-to-text regression as a practical method for maximizing feature observability with minimal engineering effort.

The magnitude of this shift is best characterized as a methodological reframing with sharp practical consequences rather than a paradigm shift. The underlying techniques—encoder-decoder transformers, next-token prediction, multi-task pretraining—are standard. What changes is what problem you think you're solving. The paper recasts regression on complex systems data as: "Given that my system logs contain rich structured state descriptions, how much of that information can I show to the model before hitting a sequence-length or compute budget, and what's the theoretical ceiling on accuracy for any given representation?" This question was not being asked systematically before, and the paper demonstrates that asking it—and answering it by including all available features rather than a hand-selected subset—yields a 100× reduction in MSE over the best achievable tabular model (Figure 6).

This reframing has several downstream effects on how practitioners and researchers should think about system modeling:

Feature engineering is de-emphasized as a research activity. The paper's core practical message is that manual feature engineering for complex systems data is not just labor-intensive but performance-limiting—every feature you exclude creates equivalence classes of system states that look identical to the model, imposing a hard ceiling on achievable accuracy. The argument that this ceiling is computable via the TotalVariance bound (Equation 2) and that it explains the 100× gap in Figure 6 gives this message quantitative teeth. If adopted, this perspective shifts effort away from designing clever tabular features and toward (a) logging more system state in structured text formats, (b) building serialization pipelines that present this state to a model, and (c) investing in sequence-length scaling to accommodate richer representations.

Model architecture becomes secondary to input representation. The paper finds that model size plateaus within the O(100M) parameter range (Figure 16), that language pretraining is unnecessary (Section 3.2), and that the optimal architecture (encoder-decoder with ~2 layers) is a fraction of the size of modern LLMs. The strong implication is that for regression over structured data, the return on investment from better input representation vastly exceeds the return from larger or more sophisticated models. A practitioner with a limited engineering budget should prioritize instrumenting their system to log more features in a structured text format over experimenting with larger model architectures, more exotic training objectives, or fine-tuning pretrained LLMs.

Multi-task pretraining becomes the default deployment strategy, not an advanced technique. The paper demonstrates that training a single model on data from many cells and months, with cell identity and timestamp appearing as ordinary tokens in the input, enables (a) simultaneous high-accuracy prediction across all seen deployments, (b) few-shot adaptation to new deployments with as few as 500 examples (Figures 4, 7), and (c) implicit meta-learning where pretraining on more tasks improves out-of-distribution adaptation (Figure 7). This means that for organizations operating many similar systems (multiple data centers, multiple manufacturing lines, multiple network regions), the natural workflow shifts from "train a separate model for each deployment" to "pretrain one model on all available data, fine-tune for new deployments as needed." The paper provides concrete evidence that this approach works at scale (40 cells, 2 months, ~1.6M training examples) and characterizes the relationship between pretraining task diversity and few-shot performance.

This work also resolves a latent tension in the applied ML community between "just use a pretrained LLM for everything" and "domain-specific models work better." The paper's explicit rejection of language pretraining (Section 3.2)—arguing that it is "not necessary nor guaranteed beneficial" for regression over structured data—pushes back against the LLM-as-universal-function-approximator narrative that has dominated since GPT-3. The counterargument, validated empirically by the paper's results, is that structured system data has its own "grammar" and correlational structure that natural language pretraining doesn't help with and may actively interfere with. A randomly-initialized 60M-parameter model trained on 2B tokens of Borg data learns representations that are specific to the problem's structure, not contaminated by irrelevant English semantics. This doesn't mean LLMs are useless for regression—it means that for problems with rich, domain-specific structured inputs, starting from scratch on that domain's data may be preferable to adapting a general-purpose language model. This is a useful corrective to the LLM maximalism that often goes unexamined in applied work.

The paper also opens a door that prior work had only cracked. OmniPred (Song et al., 2024a) showed that language models could serve as universal regressors across curated benchmark tasks. The present paper takes this idea into the messy, real-world setting of industrial system data—inputs spanning tens of thousands of tokens, deeply nested hierarchical structure, multi-modal outcome distributions, open-ended categorical vocabularies that evolve over time—and demonstrates that the approach not only works but dramatically outperforms the theoretical ceiling of the tabular alternative. This bridges the gap between "RLMs as an interesting research idea" and "RLMs as a practical tool for production systems," which is exactly the transition that determines whether a method gets adopted or remains a paper.

Follow-Up Research This Work Enables

Cheap difficulty estimation for regression tasks. The paper establishes that feature observability is the dominant factor in regression performance, but it does not provide a method for estimating how much of the available information is captured by a given serialization format without training a model to convergence. A natural extension is to develop lightweight diagnostics for the epistemic uncertainty induced by a feature representation—analogous to the TotalVariance bound in Equation 2, but computable from a small sample of data without building a full regressor. Concretely: given a new system and a candidate text serialization format, how many equivalence classes does it create? What is the average within-class variance? Can you estimate the irreducible MSE floor from 100 labeled examples by grouping on the serialized string and computing within-group statistics? A strong result would show that this cheap diagnostic predicts the relative performance of different serialization formats (JSON vs. YAML, full vs. compressed, ordered vs. unordered) without training any model, enabling practitioners to iterate on input representation before committing to a training run.

Systematic comparison of serialization formats and feature ordering. The paper uses a specific YAML-like serialization format (Figure 21) and notes that "placing the presumably most important features at the beginning of the string representation" can help (Section 3.4), but never empirically validates these choices. A direct follow-up would train identical RLMs on the same Borg data serialized in different formats—YAML vs. JSON vs. a custom delimiter-separated format, with features ordered by domain-estimated importance vs. random ordering vs. alphabetical—and measure the effect on validation loss, convergence speed, and few-shot adaptation performance. This would provide the practical guidance that the current paper lacks for practitioners building their own text serialization pipelines. A particularly informative experiment would test whether attention patterns (computable from the encoder) differentially weight features based on their position in the string, which would directly inform feature ordering recommendations.

Temporal robustness and long-horizon generalization. The paper's evaluation uses data from two months (June and November) and tests out-of-distribution adaptation across cells and across months, but never evaluates on months substantially later than the training period. For a production system where hardware platforms, workload types, and scheduling policies evolve over time, the critical question is: how quickly does an RLM's performance degrade as the system drifts away from the training distribution, and can periodic fine-tuning on small amounts of recent data maintain accuracy? A concrete experiment would train an RLM on June data, evaluate on each subsequent month for a year (monthly test sets), and measure the degradation curve. The hypothesis—suggested by the paper's few-shot adaptation results—is that very small amounts of data from each new month (tens to hundreds of examples) can recalibrate the model with minimal retraining cost, but the rate at which this retraining is needed (monthly? weekly? after major hardware changes?) is unknown and practically important.

Combining RLMs with Bayesian optimization for closed-loop system tuning. The paper's motivating use case is tuning Borg scheduler hyperparameters via Google Vizier, but it stops at demonstrating that the RLM is a good surrogate model—it never closes the loop by using the RLM inside an actual optimization run. A strong follow-up would replace Vizier's Gaussian Process surrogate with an RLM (either as a drop-in point predictor or as a full density estimator for acquisition functions that reason about uncertainty) and measure the optimization efficiency: how many iterations to reach a given improvement in MIPS per GCU? The RLM's advantages over a GP in this setting would come from (a) the ability to observe rich system state beyond just hyperparameters, enabling better predictions of the outcome for any candidate configuration, and (b) the natural uncertainty quantification from sample variance (Figure 8), which could inform acquisition functions like Expected Improvement or Upper Confidence Bound. A direct comparison against a GP using the best available tabular features would quantify the practical benefit of better regression for the downstream optimization task—not just better prediction accuracy, but faster convergence to good hyperparameter settings.

Stress-testing the "no language pretraining" claim across domains. The paper claims that language pretraining is unnecessary for regression over structured data, supporting this with the success of a randomly-initialized model on Borg data. But the Borg data has specific properties—it is highly structured, uses domain-specific terminology, and has no natural language semantics. Would the same hold for regression tasks where the input contains natural language? For example, predicting software performance from bug reports and commit messages, or predicting manufacturing quality from operator notes and sensor logs? A direct experiment would compare a randomly-initialized RLM against the same architecture initialized from a pretrained T5 checkpoint (or a small Llama checkpoint) on a regression task with mixed structured and unstructured text inputs, measuring both final accuracy and sample efficiency. Finding the boundary where language pretraining starts to help would refine the paper's claim from "language pretraining is unnecessary" to "language pretraining is unnecessary when the input is purely structured data with domain-specific vocabulary, but becomes beneficial as the proportion of natural language in the input increases."

Verifier or discriminator training for the RLM's own uncertainty estimates. The paper demonstrates that the RLM's sample variance correlates with prediction error (Figure 8), but this correlation is imperfect—there are inputs where the model is confidently wrong (low variance, high error). For downstream applications like Bayesian optimization where uncertainty estimates guide exploration, these confidently wrong predictions could cause the optimizer to over-exploit bad regions. A concrete follow-up would train a lightweight discriminator (perhaps a small MLP or even a linear probe on the encoder's hidden states) to predict whether the RLM's prediction error on a given input will exceed some threshold, using the encoder's representation as input and the actual error on a held-out calibration set as the training signal. This would provide a calibrated confidence score that could be used to decide when to trust the RLM's prediction vs. when to fall back to running the actual simulation (at 1-18 hours of compute). The discriminator could be evaluated by measuring the tradeoff between simulation cost (what fraction of inputs are escalated to the real simulator) and overall accuracy (MSE on the combined RLM-predicted and simulator-computed outputs).

Practical Applications and Downstream Use Cases

Black-box optimization with rich state observability. Google Vizier currently tunes Borg scheduler hyperparameters using a Gaussian Process that can only observe tabular features. Replacing the GP with an RLM as the surrogate model would let the optimizer observe the full system state—cell identity, time window, hardware distribution, workload composition—in addition to the hyperparameters being tuned. This means the optimizer could condition its predictions on the specific cell and time period rather than treating all optimization runs as independent. The expected benefit: faster convergence to optimal hyperparameters because the model transfers information across cells (via multi-task pretraining) and accounts for temporal effects (peak vs. off-peak workload patterns). The paper's results suggest this could be particularly impactful for cells with high aleatoric noise (where the RLM's density estimation captures multi-modality that a unimodal GP would miss) and for new cells where the RLM's few-shot adaptation (500 examples achieving performance comparable to fully trained models, Figure 4) would dramatically reduce the cost of bootstrapping optimization in a new deployment.

Digital twin acceleration for complex systems. Borg's digital twin takes 1-18 hours to simulate a single scheduling decision. An RLM trained on its outputs can approximate this simulation in milliseconds, enabling what-if analysis and configuration exploration at scales that are completely infeasible with the full simulator. The practical deployment pattern: an operator wants to understand how a proposed configuration change (new hyperparameters, different hardware allocation, new workload admission policy) would affect cluster efficiency. Rather than running the digital twin for hours per candidate configuration, they query the RLM for thousands of candidate configurations in seconds, identify the most promising options, and then run the expensive simulation only on the top few candidates for final validation. The paper's results (Figures 10, 12) show that the RLM achieves near-perfect ranking (0.93-0.99 Spearman $\rho$ ) on the most predictable cells, meaning the top candidates identified by the RLM would very likely be top candidates under the full simulation—the exact property needed for this "coarse screening" use case. The residual error (worse on noisy cells like $C_{22}^{NOV}$ ) determines how many candidates need to be validated with the real simulator, which is a tunable cost-accuracy tradeoff.

Fleet-wide performance monitoring and anomaly detection. An RLM pretrained across many cells and months captures the expected relationship between system state and efficiency metric. For an operations team managing dozens of data centers, the model can serve as a baseline for "normal" behavior: given the current state of each cell (workload mix, hardware composition, scheduler configuration), the model predicts the expected efficiency distribution. Actual efficiency measurements that fall in the tails of this predicted distribution—particularly measurements that the model assigns low density under $p_\theta(y|x)$ —flag potential anomalies: misconfigured hardware, unexpected workload behavior, bugs in the scheduling algorithm, or degradation that hasn't been captured by the digital twin's model of the system. The RLM's density estimation capability (Figures 5, 11) is critical here: rather than a simple threshold on point-prediction error (which would generate false alarms on inherently noisy cells), the anomaly detector can use the full predicted distribution to compute how unlikely an observation is under normal operating conditions. The fact that the model achieves high $R^2_{\text{NLL}}$ even on cells where pointwise prediction is noisy (Figure 12, $C_{21}^{NOV}$ and $C_{24}^{JUN}$ ) means it correctly captures the distribution of "normal" outcomes, making it a well-calibrated anomaly detector even for stochastic systems.

Cost-efficient data generation for system modeling. Many organizations operate complex systems where generating outcome data is expensive—not just compute clusters but manufacturing lines (where each experimental configuration consumes materials and production time), supply chains (where each simulation requires hours of computation), or network configurations (where each test deployment risks service disruption). The RLM approach offers a path to reducing these costs: train an initial model on whatever historical data exists, use it to screen candidate configurations, run the expensive real-world or high-fidelity simulation process only on the most promising candidates, and then feed those expensive new measurements back into the model via fine-tuning. The paper's few-shot adaptation results (Figure 7: 64 examples sufficient for strong performance after multi-task pretraining) suggest that this active learning loop could be very sample-efficient—each expensive real-world measurement contributes substantial new information because the pretrained model already has strong priors about system behavior from related tasks. The total cost reduction depends on the ratio of expensive data generation to cheap model inference, which the paper's setting makes vivid: 1-18 hours per simulation vs. milliseconds per RLM prediction, a factor of ~10⁷. Even a modest reduction in required simulations (e.g., evaluating 10 candidate configurations instead of 1000, screening with the RLM) translates to enormous absolute time and compute savings.

When to Prefer This Method

The paper does not articulate an explicit, detailed tradeoff matrix against named alternative methods (such as tabular regression with a specific feature engineering methodology, or pretrained-LLM-based regression). It positions text-to-text regression as a "general, scalable alternative" that avoids the limitations of tabular approaches, but it does not systematically compare against well-engineered tabular baselines, pretrained LLMs fine-tuned on the same data, or graybox approaches with domain-informed feature engineering. The paper's comparisons are primarily against theoretical ceilings (the TotalVariance bound for minimal tabular features) rather than against named competing methods in a head-to-head deployment scenario.

The paper does imply a clear decision rule based on input structure: if your system state can be naturally serialized into a structured text format (configuration files, system logs, nested JSON/YAML), text-to-text regression is applicable and avoids the information loss of tabular feature engineering. If your features are already naturally tabular (e.g., sensor readings with fixed dimensionality, categorical variables with small, closed vocabularies), the benefits of text-to-text regression are less clear and the method may introduce unnecessary complexity. However, this rule is implicit in the paper's framing rather than explicitly stated or empirically validated, so a formulaic "Prefer X when Y" matrix would go beyond what the paper demonstrates.