OmniPred: Language Models as Universal Regressors

ArXiv: 2402.14547

🎯 Pitch

A language model trained on millions of blackbox optimization trials can predict numerical outcomes purely from text, outperforming specialized regressors like Gaussian Processes and boosted trees—even on entirely new tasks—simply by reading parameter names and values as free-form strings.

1. Executive Summary

This paper introduces OmniPred, a framework for training language models as universal end-to-end regressors over arbitrary-format (x,y) data, and studies whether token-based representations can replace problem-specific tensor featurization for numeric prediction. Using a 200M-parameter T5 model trained from scratch on data from Google Vizier—a large proprietary blackbox optimization database spanning over 70M studies—the paper demonstrates that multi-task training across heterogeneous input spaces and objective scales enables transfer learning that significantly outperforms traditional single-task regressors (e.g., MLPs, boosted trees, Gaussian Processes). The key mechanism is multi-task transferrability via textual representations (expressing x in free-form key-value strings and y via custom floating-point tokens), which yields up to a 4× improvement in prediction error over baselines when sufficient inter-study data is available, while also allowing the model to adapt to unseen tasks through online finetuning with LoRA. The paper establishes that language models are capable of precise numerical regression—contrary to prior beliefs about their numeric brittleness—but that the benefits of multi-task transfer are maximized only when the task's training set is moderately sized (~50 trials) and diminish as single-task data becomes abundant or when textual metadata is anonymized to remove transferrable cues.

2. Context and Motivation

The Core Problem: Regression Is Brittle and Non-Transferable

The fundamental question this paper tackles is whether language models can serve as universal regressors — models that can predict numerical outcomes from arbitrary, heterogeneous input spaces without requiring task-specific featurization or rescaling. This is not merely an academic curiosity; it addresses a structural limitation in how regression is practiced across experimental design, hyperparameter optimization, and blackbox optimization.

To understand the gap, consider what happens in a typical regression workflow for hyperparameter tuning. A practitioner configuring a deep learning training job has a set of parameters x (learning rate, batch size, optimizer type, number of layers, dropout rate) and observes a scalar outcome y (validation accuracy). Before feeding this data into a regressor — say, a Gaussian Process or an MLP — the practitioner must perform several manual transformations:

• Categorical features like optimizer type ("Adam" vs. "SGD") must be one-hot encoded against a fixed, known set of choices.
• Continuous features like learning rate must be rescaled to a [-1, 1] range, requiring knowledge of user-specified bounds.
• The target y itself often requires nonlinear transformation (e.g., Box-Cox warping) and scaling, which becomes problematic when test-time y-values fall outside the training range.
• Conditional parameters (e.g., β is active only if optimizer = "Adam", not "SGD") require special handling — NaN imputation with learned values — that complicates the fixed-length tensor representation.

These steps are summarized in Table 1, which compares traditional regressors along four flexibility dimensions. The key insight is that every method prior to OmniPred requires (1) tensorizing x into a fixed-length vector and (2) rescaling y based on training data statistics. This makes each regressor tightly coupled to a single task's input space. If a parameter space changes — say, a new optimizer is added, or bounds are widened — the featurization pipeline must be rebuilt and the model retrained from scratch.

This might seem like a minor engineering inconvenience. But in practice, the paper argues, it fundamentally limits transfer learning across tasks. A practitioner tuning ResNets on CIFAR-10 and another tuning transformers on LM1B share conceptual similarities (both involve learning rates, regularization, optimizer choices), yet their regression models cannot share information because the tensor representations are incompatible — different dimensionalities, different categorical vocabularies, different scaling parameters. The regression model for task A is completely useless for task B, even if the underlying optimization dynamics are related.

Why This Problem Matters: Scale and Transfer

The importance of universal regression crystallizes when we consider the scale of modern blackbox optimization data. Google Vizier — the database used in this paper — contains orders of magnitude more data than what prior works typically access: over 70 million studies (individual optimization tasks) and 120 billion trials (individual (x, y) evaluations), spanning ~14,000 distinct users tuning everything from ML models to hardware designs to protein sequences. Each study is a potential training signal for regression. Yet traditional single-task regressors can only use data from the exact study being modeled, discarding the overwhelming majority of available information.

This is wasteful in a specific, quantifiable way that the paper exploits: many tasks share parametric and semantic structure even when their exact input spaces differ. A learning_rate parameter appears across thousands of studies in different contexts; metadata like objective: "validation/ce_loss" signals that the y-values represent cross-entropy on a validation set, which behaves differently from, say, bid simulation revenue. Traditional regression cannot leverage these shared cues because it strips away the names, semantics, and raw numerical magnitudes in favor of normalized tensors.

The practical consequence is that single-task regressors require substantial per-task data to be accurate, which is expensive in settings where each (x, y) trial is costly (e.g., protein design experiments, hardware simulation, large-scale ML training). If a model could transfer knowledge across hundreds of thousands of prior optimization studies, it might make accurate predictions with far fewer trials on a new task — accelerating the optimization loop significantly.

Prior Approaches and Their Limitations

The paper situates its contribution against a landscape of existing regression methods, each of which fails to achieve the universality that OmniPred targets:

Statistical regressors (GPs, tree-based methods, MLPs). These are the workhorses of experimental design and blackbox optimization. Gaussian Processes (GPs) provide calibrated uncertainty estimates crucial for Bayesian optimization; boosted trees (XGBoost) handle tabular data efficiently; MLPs model nonlinear relationships with expressive deep architectures. But all three share the same fundamental constraint: they require fixed-length tensor inputs and single-task training. A GP trained on a 4-parameter CIFAR-10 tuning study cannot be applied to a 60-category protein design problem, nor can it incorporate training data from both simultaneously.

The paper notes that multi-task and contextual variants of these methods exist (Bonilla et al., 2007; Krause & Ong, 2011), but they still operate on fixed-length tensor representations, meaning they can only share information across tasks with identical input spaces — a severe restriction. Two CIFAR-10 tuning studies with the same parameters can be multitask-trained, but a CIFAR-10 study and a transformer tuning study cannot, even if they conceptually share hyperparameters.

Deep learning regressors with length independence (Transformers, GNNs, RNNs). Recent work has attempted to relax the fixed-dimensionality constraint using architectures that accept variable-length inputs. Transformers (Hollmann et al., 2023; TabPFN), graph neural networks (Lukasik et al., 2020; Gao et al., 2023), and recurrent architectures (Hashemi et al., 2018) can process inputs where the number of features varies. However, these methods still require tensor representations of both x and y. The input features must be numerical tensors in a reasonable range; categorical values must be embedded; objectives must be rescaled. They achieve length-independence within a single domain (e.g., neural architecture search) but do not generalize across domains with fundamentally different parameter types and scaling regimes. They are length-independent but not task-independent.

Text-to-text reward modeling. The paper draws an important connection to reinforcement learning from human feedback (RLHF), where language models have been successfully used as reward models: they observe a textual prompt (the generated response) and output a scalar preference score. This is conceptually a regression problem over text, and its success in training models like ChatGPT and Gemini demonstrates that LLMs can serve as regressors in at least one domain. However, the paper points out a critical limitation that has gone unexamined: RLHF reward modeling deals with ordinal human preferences (ranking response A over response B), not precise numerical regression over physical or computational systems. Human preference data has high aleatoric uncertainty and does not require high-precision measurement — a prediction of 0.8 vs. 0.82 is rarely meaningful. In contrast, experimental design regression requires predicting y with potentially many significant digits of accuracy (e.g., distinguishing a validation loss of 0.053 from 0.058 matters for model selection).

The paper explicitly positions itself against a widespread skepticism in the LLM community about numerical reasoning:

"Given multiple works (Hendrycks et al., 2021; Nogueira et al., 2021) demonstrating their brittle and unreliable numeric abilities, it is non-obvious that language models are capable of high-precision numerical prediction over token-based representations."

This skepticism is well-founded: prior work (Nogueira et al., 2021) showed that transformers struggle with even simple arithmetic when numbers are represented token-by-token rather than as explicit numerical values. The MATH benchmark (Hendrycks et al., 2021) further demonstrated that language models make basic numerical errors in multi-step reasoning. The idea that an LLM could serve as a precise scientific instrument — predicting, say, a validation cross-entropy of 0.0523 ± 0.0001 from a textual description of hyperparameters — runs directly counter to this evidence. Part of the paper's contribution is proving this doubt wrong, but only under the right conditions (large multi-task training, custom float tokenization).

The OptFormer (Chen et al., 2022). The closest precursor to OmniPred is the OptFormer, which used a transformer to learn a universal hyperparameter optimizer from Vizier data. However, the paper identifies two key limitations that motivate OmniPred's design:

1. Input format restrictions: OptFormer required completed optimization trajectories with specialized x-tokenizations designed for tabular-like input spaces. It could not handle the free-form, conditional, or incomplete study data that constitutes much of the Vizier database.
2. In-context learning constraints: OptFormer relied on in-context learning (absorbing (x, y) pairs into the prompt) for regression, which requires compressing the input representation to fit within context windows — feasible for flat tabular spaces but problematic for complex conditional spaces or studies with long parameter descriptions.

OmniPred's design choices — end-to-end (x, y) training rather than in-context, free-form text representations, custom float tokenization for y — are direct responses to these limitations.

How OmniPred Positions Itself

The paper frames OmniPred not as an incremental improvement over existing regressors but as a category shift — moving from regressors that are tightly coupled to fixed, preprocessed tensor spaces to a regressor that operates on universal textual representations. The core philosophical stance is that the language model should see the data as a human would: parameter names in English, string values for categorical choices, raw numerical magnitudes for continuous parameters, and descriptive metadata about the task. This is what the paper means by "absolute formats independent of the input space and numeric scaling."

Table 1 (summarized in Section 2) makes this positioning explicit: OmniPred is the only regressor that (a) supports dynamic input spaces, (b) can multitask across different input domains, (c) requires no tensorization of x, and (d) requires no rescaling of y. This is a specific, testable claim about architectural capability, not just performance.

The framing also strategically sidesteps a potential criticism: OmniPred is not claiming to outperform single-task baselines on their own turf. The paper explicitly states:

"We emphasize that our paper's contributions are mostly on regression using flexible string representations and large-scale multi-task training, and do not claim to replace widely accepted baselines in single-task, apples-to-apples comparisons."

Instead, the value proposition is that OmniPred's single-task variant is competitive with traditional baselines (Figure 6) while offering the additional capability of multi-task transfer — something the baselines fundamentally cannot do. The competitive single-task performance serves as a sanity check; the transfer learning gains are the actual contribution.

This is a nuanced and honest positioning. It acknowledges that if you have abundant (x, y) data for exactly one task with a flat input space, an MLP or XGBoost will likely work fine — maybe even better than OmniPred. The framework earns its value when data is scarce per task but abundant across tasks, which is precisely the regime of large-scale experimental design databases.

The Specific Gap: Numeric Capability of Language Models

Beyond the architectural universality argument, the paper addresses a more specific and provocative gap: can language models, which process tokens rather than numbers, achieve the precision required for scientific and engineering regression? This is not obvious for several reasons:

• Tokenization divides numbers arbitrarily. When the T5 tokenizer processes 1234.5, it might produce tokens {"12", "3", "4.5"} rather than digit-by-digit tokens. This means the model doesn't naturally see the decimal structure that humans rely on for numerical reasoning.
• Cross-entropy loss is agnostic to numerical distance. Standard language model training penalizes incorrect token predictions equally, regardless of whether a wrong exponent token causes a prediction of 100 vs. 1 (a 100× error) or a wrong mantissa token causes a prediction of 1.23 vs. 1.24 (a 0.01 error). The loss function has no built-in concept of numerical magnitude.
• Prior evidence of numeric brittleness. As noted, multiple works have shown transformers making arithmetic errors, confusing magnitudes, and being sensitive to tokenization artifacts. The paper must overcome a prior expectation that language models are fundamentally unsuited for precise numerical work.

OmniPred addresses this gap through two design choices that the paper tests via ablation:

1. Custom float tokenization for y (Table 2): Instead of letting the tokenizer freely segment y-values, the model is constrained to produce custom tokens representing sign, digit-by-digit mantissa, and exponent (e.g., <+><7><2><5><E-1> for 72.5). This guarantees that y-values are always decoded correctly (via constrained decoding) and forces the model to learn numerical structure explicitly.
2. Multi-task training at scale: The model sees enough diverse (x, y) pairs — with y spanning many orders of magnitude — that it implicitly learns to predict the correct magnitude. The transfer learning benefits demonstrated in the experiments (Sections 5.2–5.3) serve as evidence that the model is not just memorizing specific y-values but learning generalizable numerical relationships.

The experimental design throughout the paper is structured to answer this doubt directly: the BBOB experiments test whether the model can regress analytic functions with high precision (Figure 3), the real-world experiments test whether the model can predict across disparate objective scales (Figure 4), and the ablation studies test whether alternative tokenization and aggregation choices matter (Tables 6, 8, Appendix A.1).

The Missing Piece in Prior Language Model Research

Finally, the paper identifies an underexplored intersection: while LLMs have been applied to coding, symbolic math, scientific reasoning, and reward modeling, no prior work has systematically studied whether language models can serve as universal regressors over heterogeneous experimental design data. The reward modeling work (Ziegler et al., 2019) demonstrates that LLMs can regress over textual prompts to scalar values, but does not address precision, multi-task transfer, or the challenges of diverse input spaces and objective scales. The time series forecasting work (Gruver et al., 2023) shows LLMs can regress over numerical sequences, but not over heterogeneous parameter spaces. The symbolic regression work (d'Ascoli et al., 2022; Charton, 2022) shows transformers can learn mathematical functions from examples, but operates on clean synthetic data with uniform input representations.

OmniPred occupies a unique position: it tests whether token-based regression works at scale, on real messy data, across thousands of different tasks, with the explicit goal of replacing the featurization pipeline that has been considered essential to regression for decades. The paper's framing makes it clear that this is not a foregone conclusion — the experiments are designed to discover whether universal regression is even possible, not just to optimize a known-capable system.

3. Technical Approach

3.1 Reader Orientation (Approachable Technical Breakdown)

We're building a language model that acts as a universal regression tool — think of it as a general-purpose predictor that can ingest any experimental configuration (hyperparameters, design choices, system settings) described in plain text and output a precise numerical prediction of the outcome. The system is a single T5 encoder-decoder language model that replaces the traditional pipeline of manual feature engineering, one-hot encoding, and value rescaling with a text-to-text interface: you write your input parameters as key-value strings, append contextual metadata, and the model generates a floating-point number as the prediction.

3.2 Big-Picture Architecture (Diagram in Words)

The system has four major components:

1. Input Representation — converts any experimental configuration x and task metadata m into a textual prompt using a key-value format (parameter names mapped to values) plus task descriptors. No tensorization, no rescaling.

2. Language Model (T5 encoder-decoder, 200M parameters) — trained from scratch using Prefix-LM training to map the textual prompt to a sequence of custom floating-point tokens representing the predicted y. The model is trained over multi-task data from many different studies simultaneously.

3. Custom Float Tokenization — a specialized vocabulary of tokens for representing y-values that guarantees correct numerical decoding via constrained decoding. The tokens encode sign, mantissa digit-by-digit, and exponent (e.g., <+><7><2><5><E-1> for 72.5).

4. Sampling and Aggregation — at inference time, the model generates multiple i.i.d. prediction samples via temperature decoding; the empirical median of these samples is taken as the final point prediction, providing robustness to outlier token errors.

Information flows as follows: a study's (x, m) pair is serialized into a text prompt → the T5 model encodes the prompt and autoregressively decodes y-tokens → constrained decoding ensures only valid float tokens are produced → multiple samples are drawn → the median is computed as the final prediction.

3.3 Roadmap for the Deep Dive

• First, the formal problem definition (Section 3.1 of the paper) — what exactly are we regressing, what is a "study" and a "task," and how is accuracy measured across studies with wildly different y-scales? This establishes the evaluation protocol that governs all experiments.
• Second, the textual representation scheme — how x and y and metadata m are serialized into tokens, and why this choice enables multi-task training across heterogeneous input spaces.
• Third, the custom y-tokenization and constrained decoding — the specific mechanism that guarantees numerically valid predictions and avoids the tokenization artifacts that plague standard LLM number handling.
• Fourth, the training procedure — Prefix-LM training, multi-task data sampling, and the cross-entropy objective applied only to response tokens.
• Fifth, the inference procedure — temperature sampling, the median aggregation function, and why median beats mean or max-likelihood for robustness against outlier token predictions.
• Sixth, online finetuning — how the pretrained model adapts to new unseen tasks using small amounts of task-specific data, optionally with LoRA, and how this differs from single-task training from scratch.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and empirical analysis paper whose core idea is that language models trained on textual representations of (x, y) data across many tasks can serve as universal regressors, replacing task-specific featurization pipelines while achieving competitive or superior accuracy through multi-task transfer learning.

3.4.1 Formal Problem Definition: What Are We Regressing?

The paper adopts standard blackbox optimization terminology (Golovin et al., 2017; Liaw et al., 2018) to define the regression problem. A task is a tuple:

$\mathcal{T} = (\mathcal{X}, f, \mathcal{D}, m)$

where $\mathcal{X}$ is the input space (the set of all possible parameter configurations), $f: \mathcal{X} \rightarrow \mathbb{R}$ is the true underlying objective function mapping parameter configurations to scalar outcomes, $\mathcal{D} = \{x_s, y_s\}_{s=1}^{T}$ is an offline collection of evaluated trials (a study), and $m$ is observable task-level metadata that characterizes the task (e.g., title, user, description, objective name).

What this means operationally: A single study is just a historical record of T experiments — each experiment has a specific parameter configuration x (say, batch_size=128, learning_rate=0.5, optimizer='adam') and an observed outcome y (say, validation accuracy of 72.5%). The metadata m provides context: is this a CIFAR-10 classification task? A protein design task? A bid simulation? The regression problem is to learn a mapping from (x, m) to y that accurately predicts the outcome of unseen parameter configurations within the same task.

The multi-task extension: The paper provides the model with training data from many tasks simultaneously: $\cup\{\mathcal{D}_1^{train}, \mathcal{D}_2^{train}, \ldots\}$. These tasks may have completely different input spaces $\mathcal{X}_i$ (different parameters, different types, different dimensionalities) and completely different objective functions $f_i$ with different scales. The hope is that training on extraneous tasks improves prediction on any single task through transfer learning.

Why this formulation matters: Traditional regression assumes a single task with a fixed input space. An MLP trained on a 4-parameter CIFAR-10 study has weights specifically shaped for 4-dimensional input. It cannot ingest data from a 60-parameter protein design study, nor can it benefit from having seen thousands of other optimization tasks. The paper's formalism makes explicit that the model must handle variable input spaces and leverage cross-task data — requirements that traditional tensor-based regressors fundamentally cannot satisfy.

The evaluation metric — normalized mean absolute error: Because different studies have y-values spanning completely different scales (CIFAR-10 accuracy is in [0, 1] while synthetic objectives can span [10^2, 10^9]), raw error magnitudes are incomparable across tasks. The paper defines study error as a normalized MAE:

$\frac{1}{y_{\max} - y_{\min}} \frac{1}{|\mathcal{D}^{test}|} \sum_{(x,y) \in \mathcal{D}^{test}} |\text{Aggregate}(s(x)) - y|$

where $y_{\max}$ and $y_{\min}$ are the maximum and minimum y-values observed in the study's training data, $\mathcal{D}^{test}$ is the held-out test set of trials for that study, and $\text{Aggregate}(s(x))$ is the point prediction derived from the regressor's output distribution (for OmniPred, the empirical median of multiple temperature samples).

What it computes: For a given study, we take each test trial, compute the absolute difference between the model's point prediction and the true y, average these differences across all test trials, then divide by the range of y-values seen during training. This yields a dimensionless error fraction — 0.0 means perfect prediction, 1.0 means the average prediction error is as large as the entire observed range of y (i.e., the regressor is essentially useless).

The clipping detail: The paper clips individual error terms to a maximum of 1.0. This prevents a single catastrophic outlier prediction from dominating the average error for a study. A clipped error of 1.0 is equivalent to the regressor simply outputting boundary values from $\{y_{\min}, y_{\max}\}$. This clipping makes the metric robust to the "hallucinated" outlier predictions that the language model occasionally produces when it gets a sign or exponent token wrong.

Why this form: Normalizing by the observed y-range makes errors comparable across studies with different scales. Alternative normalizations (e.g., dividing by standard deviation) would be sensitive to the distribution of y-values, while the min-max range provides a natural worst-case baseline: a regressor that just guesses the midpoint of the observed range achieves an expected error of roughly 0.5 (assuming uniform test distribution), and a regressor that outputs boundary values achieves error at most 1.0.

3.4.2 Textual Representation of Inputs (x) and Metadata (m)

The core design decision that enables universality is representing all inputs as text in a way that requires no problem-specific preprocessing. The paper defines a representation scheme with two parts:

x-representation — key-value format: Input parameters are serialized as comma-separated key:value pairs, where the key is the parameter name (in English) and the value is its literal value. For example:

batch_size:128,kernel:'rbf',learning_rate:0.5,model:'svm',optimizer:'sgd'

Integer values are written as integers (128), float values as decimals (0.5), and categorical string values in single quotes ('rbf'). There is no normalization, no one-hot encoding, no scaling. The parameter name carries semantic information (e.g., "learning_rate" signals to the model that this value behaves like a rate parameter, typically varied on a log scale).

Conditional parameters are handled naturally: The OSS Vizier format supports conditional spaces where child parameters are only active when a parent parameter takes a specific value (e.g., "beta" only exists when optimizer="Adam", not when optimizer="SGD"). In the textual representation, conditional parameters simply appear or don't appear in the string based on whether they're active for that trial. There is no need for NaN imputation or special masking — the variable-length text string handles conditionality automatically. This is a significant advantage over fixed-length tensor representations, which require learned NaN replacement values and carefully structured input layers.

m-representation — metadata as text: Task-level metadata is similarly serialized:

title:'classification',user:'some-person',description:'spam detection',objective:'accuracy (%)'

This includes the study title, username, free-text description, the objective metric name, and any other contextual information. The paper notes that these metadata fields can contain transferrable cues: an objective name like 'val_categorical_cross_entropy' signals that y behaves like a cross-entropy loss (typically positive, decreasing with training), while 'accuracy (%)' signals percentage behavior (bounded in [0, 100]).

Why key-value rather than JSON or natural language: The key-value format is deliberately simple and structured. It avoids the syntactic overhead of JSON (braces, quotes) while preserving the semantic association between parameter names and values. It also produces shorter sequences than natural language descriptions ("The batch size was 128, the kernel was rbf..."), which matters because some parameter spaces have hundreds of parameters (e.g., Protein Design has ~60 categories). The paper explicitly notes that a single (x, m) prompt can be 10K+ tokens long for some applications.

The input space $\mathcal{X}$ itself is NOT represented: The paper makes an important design choice: the prompt contains the specific parameter values x and the metadata m, but does NOT describe the full input space (e.g., bounds on continuous parameters, the set of valid categorical choices). This is in contrast to some prior work that includes space descriptions. The reasoning is that representing the full space would make the prompt even longer and would require encoding constraints that may be irrelevant for prediction. The model instead learns the effective bounds and constraints implicitly from the training data distribution — it sees enough trials to learn that, say, learning_rate rarely exceeds 10.0 or that dropout is never above 1.0.

How this enables multi-task training: Because every task's (x, m) is represented in the same key-value text format regardless of the underlying input space dimensionality or parameter types, the language model can be trained on a single unified dataset mixing trials from all tasks. The model learns to attend to the specific parameter names and metadata fields present in each prompt, effectively using the text itself as a variable-length feature representation. Two tasks with completely different parameters produce completely different text strings; the model's self-attention mechanism handles this heterogeneity naturally, just as it handles sentences of different lengths and topics in natural language.

3.4.3 Custom Float Tokenization for Outputs (y)

This is arguably the most critical technical component, because standard language model tokenization is fundamentally incompatible with precise numerical prediction. The paper identifies and solves a specific problem: when a standard tokenizer processes a number like 1234.5, it might produce tokens {"12", "3", "4.5"} rather than digit-by-digit tokens. This means the model doesn't naturally see numbers as composed of decimal digits — a representation that makes numerical comparison and arithmetic extremely difficult.

The custom token vocabulary: OmniPred introduces a specialized vocabulary for representing y-values that guarantees digit-level structure. The representation uses custom tokens from a small, fixed vocabulary:

• A sign token: <+> or <->
• Digit tokens: <0>, <1>, ..., <9> (one token per decimal digit)
• An exponent token: <E-2>, <E-1>, <E0>, <E1>, etc.

A value of 72.5 would be represented as:

<+><7><2><5><E-1>

This encodes $+725 \times 10^{-1} = 72.5$. The mantissa (725) is represented digit-by-digit, and the exponent (-1) specifies the decimal point position.

Constrained decoding at inference time: During inference, the model's output logits are restricted to only the custom floating-point tokens. The model cannot decode natural language tokens or other vocabulary items — it must produce a sequence of exactly these float tokens. This guarantees that every decoded sequence corresponds to a valid floating-point number (no parsing failures, no uninterpretable outputs).

Why digit-by-digit mantissa: The paper conducted ablations (Appendix A.1, Table 8) comparing three tokenization schemes:

1. Default (digit-by-digit): <+><1><2><3><4><E-2> — separate sign, each mantissa digit as its own token, then exponent.
2. Merged mantissa: <+1234><E-2> — the entire mantissa is a single token.
3. Exponent before mantissa: <+><E-2><1><2><3><4> — exponent placed before the digits.

The results show that in multi-task training with abundant data, all three schemes achieve similar accuracy (0.15 for AutoML multi-task). However, in low-data regimes (single-task training), the merged mantissa scheme catastrophically fails (0.73 vs. 0.21 for default on AutoML single-task). This is because there are 18,000+ possible mantissa tokens (4 significant digits × sign combinations), each of which must be seen many times during training for the model to learn its numerical value. The digit-by-digit scheme decomposes the problem: there are only 10 digit tokens plus sign and exponent tokens, so every token is seen frequently even with limited data. The model learns to compose numbers from digits rather than memorizing individual mantissa tokens.

Why the default tokenization order (sign, then digits, then exponent): The convention follows standard scientific notation but places the digits before the exponent. The paper doesn't provide a strong theoretical justification, but the empirical finding is clear: in multi-task regimes, the choice doesn't matter much (all schemes reach ~0.15 error on AutoML and ~0.01 on BBOB). In single-task regimes, the default digit-by-digit scheme with sign-first, exponent-last ordering achieves the best performance. The merged mantissa scheme's poor single-task performance makes intuitive sense: it requires the model to learn a separate embedding for every distinct mantissa value, which is sample-inefficient.

The y-tokenization vs. x-tokenization asymmetry: A crucial subtlety: x-values (parameter values in the prompt) are tokenized using the standard T5 SentencePiece tokenizer with its default subword vocabulary. This means numbers in the input might be split arbitrarily (e.g., 1234.5 → {'12', '3', '4.5'}). The paper acknowledges this as potentially suboptimal (Section 7) and suggests future work on applying the custom digit-by-digit tokenization to x-values as well. The fact that the model works well despite suboptimal x-tokenization is actually a strength — it demonstrates robustness to tokenization artifacts on the input side, though it may limit precision on the output side if the model cannot properly attend to specific digits in input numbers.

The training loss — uniform cross-entropy over all float tokens: The model is trained with standard cross-entropy loss applied uniformly to all y-tokens. This means a wrong prediction on the exponent token (which could cause a 100× error in the final value) is penalized identically to a wrong prediction on the least significant digit token (which causes a tiny error). The paper explicitly acknowledges this limitation (Section 7) and suggests that weighting more significant tokens (exponent, leading digit) higher would make the loss more metric-aware. Despite this coarse loss function, the model learns to predict correct magnitudes — evidence that it internalizes numerical structure from the data distribution even without explicit distance-based supervision.

3.4.4 Language Model Architecture and Training

Model choice: T5 encoder-decoder, 200M parameters, trained from scratch. The paper uses a T5 architecture (Raffel et al., 2020) with 12 encoder layers, 12 decoder layers, 12 attention heads per layer, 64-dimensional heads, 768-dimensional embeddings, and 2048-dimensional MLP hidden layers. This totals approximately 200M parameters.

Why T5? T5 is a standard text-to-text transformer. The encoder-decoder structure is natural for the regression-as-translation framing: the encoder processes the (x, m) prompt, and the decoder generates the y-tokens autoregressively. The paper deliberately chooses a relatively small model (200M parameters) to demonstrate that universal regression doesn't require massive scale and to keep computational costs manageable (at most 8 GPUs for training, 1 GPU for inference).

Why train from scratch rather than fine-tune a pretrained model? This is a carefully justified design choice. The paper acknowledges that warm-starting from a model pretrained on English text could potentially improve accuracy by leveraging pretrained understanding of parameter names and metadata. However, they argue that:

1. Most checkpoints comparable to their model's size (<1B parameters) are not pretrained on experimental data and are unlikely to understand the numerical semantics of terms like "learning_rate" or "dropout."
2. Using a pretrained English model introduces "numerous confounding technical choices" — whether to freeze the encoder, tune the learning rate, embed additional custom float tokens, or use more English-like representations of x and m. These confounds would make it harder to isolate whether the gains come from the universal regression framework or from pretrained knowledge.
3. The fact that a model trained from scratch achieves competitive or superior performance is actually a stronger result — it demonstrates that the textual representation and multi-task training alone are sufficient, without relying on external language understanding.

Training data sampling — multi-task Prefix-LM: The model is trained using Prefix-LM training, where cross-entropy loss is computed only over the response tokens (the y-tokens), not over the prompt tokens (the x and m representation). For each training batch, (prompt, response) pairs are sampled from the multi-task training data $\cup\{\mathcal{D}_1^{train}, \mathcal{D}_2^{train}, \ldots\}$. The sampling procedure mixes trials from different studies and tasks, so the model sees diverse x-formats and y-scales within each batch.

Loss function: The loss is simply the sum of per-token cross-entropy losses over all y-tokens, with uniform weighting. Formally, for a response sequence of tokens $t_1, t_2, \ldots, t_k$ (the float tokens), the loss is:

$\mathcal{L} = -\sum_{i=1}^{k} \log p_\theta(t_i | \text{prompt}, t_{<i})$

where $p_\theta(t_i | \text{prompt}, t_{<i})$ is the model's predicted probability for the correct token $t_i$ given the prompt and previously decoded tokens.

What this computes: The standard next-token prediction loss used in language model training, applied only to the output tokens representing y. The model is trained to maximize the probability of the correct sequence of y-tokens given the (x, m) prompt.

Why uniform cross-entropy rather than a regression-aware loss: The paper explicitly states this is for simplicity: "One could additionally make the loss more metric-aware by weighting specific tokens or even reinforce with non-differentiable scores, although we maintain simplicity in this paper for now by using uniform cross-entropy." The model must learn numerical distances implicitly from the data — if a 10× error in y is more common than a 100× error, the model's predictions will naturally concentrate around the correct magnitude because those token sequences appear more frequently in the training data.

Optimizer and hyperparameters: The paper uses the Adafactor optimizer with base learning rate 0.01 and square root decay. Batch size is 256. Training runs for a maximum of 1 million steps with early stopping based on validation loss. The model uses a SentencePiece tokenizer (Kudo & Richardson, 2018) with a vocabulary of 32,000 subword tokens, plus the additional custom tokens for representing y. Training was performed on a 4×4 TPU V3 configuration.

Early stopping: The paper monitors validation loss and stops training if overfitting is detected, but does not specify the exact patience or threshold criteria. This is a minor omission, though the maximum 1M step limit provides a bound on total training.

3.4.5 Inference: Sampling and Aggregation

At inference time, the model does not produce a single deterministic prediction. Instead, it generates multiple i.i.d. samples through temperature decoding, then aggregates them into a single point prediction.

Temperature sampling: The model samples y-tokens autoregressively with temperature $T = 1.0$ (standard softmax). The logits are restricted via constrained decoding to only the custom floating-point tokens — the model cannot output natural language tokens or other vocabulary items. This ensures every sampled sequence decodes to a valid floating-point number.

Number of samples: The paper uses 64 samples per prediction by default. This number is chosen to balance inference cost against the benefits of aggregation (Figure 8 shows diminishing returns beyond ~16–32 samples for most tasks, but some tasks continue improving to 64+).

Aggregation function — empirical median: The paper's $\text{Aggregate}(\cdot)$ function is defined as the empirical median of the 64 sampled y-values, not the mean or the maximum-likelihood sample. Table 6 provides the evidence for this choice:

What this shows: On the AutoML dataset, median aggregation reduces prediction error by roughly 35% compared to mean or max-likelihood (0.15 vs. 0.22–0.23). On BBOB, all methods perform similarly (0.01), because the model has nearly perfectly regressed the analytic functions and produces very few outliers.

Why median beats mean: The language model occasionally produces "hallucinated" outlier predictions — typically because it gets a single float token wrong, especially the exponent or the most significant digit. A wrong exponent token can turn a prediction of 72.5 into 7250 or 0.725 — a 10× or 100× error. These outliers occur "with relatively high probability" (Section 6.1) on realistic tasks where the model has non-trivial error. The mean is sensitive to these outliers — a single 100× error can shift the mean substantially. The median, as a robust estimator, ignores outliers as long as fewer than half the samples are corrupted. The max-likelihood sample (the single most probable token sequence) is even more vulnerable because it places all weight on one sample that might be an outlier.

Why the model produces outliers: The paper attributes this to the uniform cross-entropy loss: since all float tokens are weighted equally during training, the model doesn't learn that getting the exponent wrong is far worse than getting the last mantissa digit wrong. It allocates prediction probability roughly equally across all positions in the float representation. In the single-task AutoML setting, where the model hasn't perfectly memorized the mapping, this leads to occasional catastrophic token errors.

Uncertainty expression: Beyond point predictions, the distribution of 64 samples provides a nonparametric uncertainty estimate. The paper demonstrates in Section 6.2 and Figure 9 that the model can express multimodal distributions (e.g., when trained on randomly sign-flipped BBOB objectives), and that the sample standard deviation correlates with prediction error (Table 7: Kendall-Tau correlation of 0.487 on AutoML, 0.366 on BBOB). This uncertainty can be used for downstream applications like Bayesian optimization, where exploration is guided by predictive uncertainty.

3.4.6 Online Finetuning for Unseen Tasks

The pretrained multi-task model can be further adapted to a specific unseen task through online finetuning — training on the new task's limited trial data $\mathcal{D}_u^{train}$ at inference time.

When finetuning helps: The paper identifies two scenarios:

1. Unseen tasks: Tasks that were created after the pretraining dataset was collected (new users, new objectives, new parameter spaces). The pretrained model can transfer general knowledge about parameter-outcome relationships, but may need task-specific adaptation.
2. Seen but under-optimized tasks: If the pretraining dataset was too large for the model to fully converge on every individual study, finetuning on a specific study's data can "refocus" the model.

Finetuning procedure: The model is finetuned using the same Prefix-LM objective as pretraining, but on only the target study's training data. Key differences from pretraining:

• Optimizer state: The model reloads both the weights AND the optimizer state (momentum parameters) from the pretraining checkpoint. This preserves the optimization trajectory.
• Learning rate: The finetuning uses a fixed learning rate of $10^{-5}$, which is ~10× lower than the typical $\mathcal{O}(10^{-4})$ learning rate during late pretraining. This prevents catastrophic forgetting of general knowledge.
• Epoch definition: Since training data for a single study may have fewer trials than the batch size (256), an epoch is defined as seeing the training data once — which might be a single gradient step if $|\mathcal{D}_u^{train}| \leq 256$.
• Maximum epochs: 30 epochs with early stopping based on validation loss computed over the full validation set (not sampled batches).
• Optional LoRA: The paper mentions that LoRA (Hu et al., 2022) can optionally be used for parameter-efficient finetuning, though experimental results appear to use full finetuning.

Single-task training from scratch: For comparison, the paper also considers training a randomly initialized model on just the target study's data. This single-task model uses a larger constant learning rate of $10^{-3}$ (matching early pretraining learning rates) since there's no risk of forgetting.

What Table 5 demonstrates: When finetuning on AutoML studies, a model pretrained on the full real-world dataset achieves the same final error (0.15) as a model pretrained specifically on AutoML data — even though the full-dataset model started with higher error before finetuning (0.31 vs. 0.15). This demonstrates that broad multi-task pretraining provides transferrable knowledge that can be refined to task-specific accuracy with a small amount of finetuning data. In contrast, a model pretrained on BBOB (synthetic analytic functions) finetunes to 0.45 error on AutoML — worse than even single-task training from scratch (0.20). This is a case of negative transfer: knowledge from simple analytic functions actively interferes with learning on real-world ML tasks, likely because the function shapes and parameter semantics are fundamentally different.

The finetuning cost: Due to the small training set and relatively few finetuning steps, finetuning uses a single 1×1 TPU V3 — substantially cheaper than pretraining. This makes the approach practical for deployment scenarios where each new task incurs a small finetuning cost.

3.4.7 Design Choices Summary and Their Justifications

Textual x-representation (key-value format) over tensorization: Enables variable-length inputs, handles conditional parameters naturally (they simply don't appear in the string when inactive), and carries parameter name semantics that enable transfer learning across tasks with shared parameter names. The cost is longer sequences (up to 10K+ tokens) and reliance on the model's ability to parse the key-value structure — which the T5 architecture handles adequately.

Custom float y-tokenization over standard tokenizer: Guarantees that every decoded sequence is a valid floating-point number and that the model sees numbers as composed of decimal digits. The digit-by-digit design makes the representation sample-efficient in low-data regimes (Table 8). The cost is a specialized token vocabulary that must be trained from scratch (no transfer from pretrained tokenizers).

Training from scratch over fine-tuning a pretrained LLM: Avoids confounding factors from pretrained knowledge, demonstrates that the framework works without relying on English language understanding, and keeps the model small (200M parameters). The cost is that the model cannot leverage pretrained parameter name semantics — all meaning must be learned from the regression training data alone.

Median aggregation over mean: Robust to the "hallucinated" outlier predictions that occur when the model gets a sign or exponent token wrong. The cost is slightly higher inference compute (need multiple samples to estimate a median robustly) and slightly worse correlation with uncertainty compared to mean/standard-deviation (Table 7 shows higher Kendall-Tau for mean-aggregation sample SD on AutoML: 0.487 vs. 0.412 for median-based Harrell-Davis SE). This suggests there may be a tradeoff between point prediction accuracy and uncertainty calibration that the paper doesn't fully explore.

Multi-task training over ICL or single-task: Maximizes usage of available training data, enables transfer learning across tasks, and avoids the context-length limitations that constrain in-context regression methods. The cost is the need for large-scale pretraining infrastructure and the risk of negative transfer when pretraining data is poorly matched to target tasks (Table 5, BBOB → AutoML).

Prefix-LM training over full encoder-decoder: Standard for T5 — the encoder processes the full prompt bidirectionally, while the decoder generates y-tokens autoregressively with causal masking. This is the natural choice for a mapping from (x, m) to y where the decoder should not attend to future y-tokens.

Uniform cross-entropy loss over metric-aware loss: Simpler to implement, doesn't require engineering token importance weights, and works empirically — the model learns correct numerical magnitudes from the data distribution even without explicit distance supervision. The cost is that the model is not explicitly trained to avoid large-magnitude errors, leading to the outlier prediction problem that requires median aggregation at inference time. The paper acknowledges this as a limitation and suggests future work on loss weighting.

4. Key Insights and Innovations

Innovation 1: Regression as a Text-to-Text Problem — The Universal Regressor Framing

The paper's most fundamental intellectual move is reframing regression — traditionally conceived as a numerical optimization problem over fixed-dimensional tensor spaces — as a text-to-text translation task. This is not an incremental improvement to existing regressors; it is a category shift that rips out the entire featurization pipeline that has been considered essential to regression for decades.

Prior to OmniPred, the dominant paradigm across all regression methods — whether Gaussian Processes, boosted trees, MLPs, or even recent transformer-based approaches — was that inputs must be converted to fixed-length numerical tensors through problem-specific preprocessing. This required one-hot encoding categoricals, rescaling continuous values to a bounded range (typically [-1, 1]), and normalizing outputs against training set statistics. The consequence, documented in Table 1, was that every regressor was tightly coupled to a single task's input space: an MLP trained on a 4-parameter CIFAR-10 tuning study was structurally incapable of ingesting data from a 60-parameter protein design task, even if both involved conceptually similar hyperparameters like learning rates and regularization coefficients. Multi-task and contextual variants of GPs existed (Bonilla et al., 2007; Krause & Ong, 2011), but they still required identical input spaces across tasks — a severe constraint that made cross-domain transfer impossible.

What OmniPred recognizes is that the featurization step is the bottleneck, not the regression algorithm itself. By representing inputs as key-value text strings (e.g., batch_size:128,kernel:'rbf',learning_rate:0.5) and outputs as custom float tokens (<+><7><2><5><E-1> for 72.5), the framework eliminates every task-specific preprocessing step simultaneously:

• Categorical values are represented as their string names, not one-hot indices — so the model can learn that 'adam' and 'sgd' are different optimizer types across studies without needing a predefined vocabulary.
• Continuous values are represented in their raw absolute magnitudes, not rescaled to a unit cube — so the model learns that learning rates are typically small (~0.001) and batch sizes large (~128) from the data distribution itself.
• Conditional parameters are handled naturally: they simply appear or don't appear in the text string based on whether they're active, with no need for NaN imputation or learned replacement values (Appendix C.3).
• The output y is predicted in absolute terms using a constrained vocabulary of float tokens, so the model never needs to know the task's y-range at prediction time — unlike traditional regressors that fail when test y-values fall outside the training range.

This reframing matters beyond convenience. It transforms regression from a closed-vocabulary, fixed-topology operation (where each task is a separate model with its own input schema) to an open-vocabulary, variable-topology operation (where a single model processes arbitrary parameter configurations described in a shared textual format). The architectural implication is that transfer learning across tasks becomes possible not through ad-hoc parameter sharing schemes, but through the same mechanism that enables transfer in natural language: shared representations of semantically meaningful tokens. Two studies that both contain learning_rate as a parameter name will share token embeddings and attention patterns, even if one study is tuning ResNets and the other is tuning transformers.

The significance of this reframing extends beyond the specific results in the paper. It opens a path toward regression systems that improve as they see more data from more tasks — analogous to how language models improve with more text — rather than being permanently capped at the performance achievable from a single study's ~100 trials. The paper's demonstration that a 200M-parameter model trained from scratch can achieve competitive single-task performance (Figure 6) while additionally enabling multi-task transfer is evidence that the text-to-text framing is not merely a representational trick but a genuinely viable architectural paradigm.

Innovation 2: Empirical Proof That Language Models Can Perform Precise Numerical Regression

The paper provides the first systematic evidence that language models — notoriously brittle on numerical tasks — can serve as high-precision regressors over real experimental data spanning many orders of magnitude. This is a significant empirical finding because it contradicts a widespread and well-justified skepticism in the community.

Prior work had established that transformers struggle with numerical reasoning. Nogueira et al. (2021) demonstrated that standard transformer architectures fail on simple arithmetic when numbers are represented token-by-token rather than as explicit numerical values. The MATH benchmark (Hendrycks et al., 2021) showed that even large language models make basic numerical errors in multi-step reasoning. These findings created a reasonable prior that language models are fundamentally unsuited for tasks requiring precise numerical outputs — you wouldn't trust an LLM to predict a validation loss of 0.0523 any more than you'd trust it to compute a 7-digit multiplication.

The paper directly challenges this prior by training models to regress on two very different data regimes:

• BBOB synthetic functions (Section 5.1, Figure 3): The model captures the overall shape of analytic functions with high precision, producing predictions that track the ground truth almost perfectly for simple functions and maintain reasonable accuracy even on multimodal objectives (Figure 9). The study error reaches ~0.01 with full multi-task training (Table 6), meaning the average prediction error is only 1% of the observed y-range.

• Real-world Google Vizier data (Section 5.1, Figure 4): The model predicts outcomes across drastically different objective scales — from CIFAR-10 classification accuracy (bounded in [0, 1] or [0, 100]) to synthetic objectives spanning [10^2, 10^9] — with strong diagonal fit in predicted-vs-actual plots. The AutoML domain achieves study errors as low as 0.15 with full multi-task training (Table 4), meaning the average prediction error is 15% of the observed y-range.

What makes this finding intellectually distinctive is not just the accuracy numbers but what had to be true for it to work at all. The model overcomes several obstacles simultaneously:

1. Tokenization artifacts: The standard T5 SentencePiece tokenizer splits input numbers arbitrarily (e.g., 1234.5 might become {"12", "3", "4.5"}), yet the model learns to attend to the correct numerical values despite not seeing numbers as clean digit sequences on the input side. This suggests that the self-attention mechanism can reconstruct numerical magnitude from fragmented token representations — a capability that was not obvious a priori.

2. Cross-entropy blindness to magnitude: The training loss penalizes all float tokens equally, meaning a wrong exponent prediction (causing a 100× error) is treated identically to a wrong mantissa digit (causing a 0.01 error). Despite this, the model learns to predict correct magnitudes on most trials — it internalizes numerical structure from the data distribution even without explicit distance-based supervision. The fact that median aggregation substantially outperforms mean (Table 6, AutoML: 0.15 vs. 0.23) reveals that the model does still produce occasional catastrophic outliers, but that these are correctable through robust post-processing.

3. Scale variation across tasks: The model simultaneously handles tasks where y ranges from [0, 1] to [10^2, 10^9] without scale normalization. This requires learning to recognize from the textual metadata and parameter values what order of magnitude to predict — an emergent capability that the paper demonstrates but doesn't fully explain mechanistically.

The practical implication is a reversal of the burden of proof. Before OmniPred, the default assumption was that language models are inappropriate for numerical regression and that specialized architectures with explicit numerical inductive biases are necessary. After OmniPred, the evidence shows that a generic text-to-text architecture — with the right tokenization choices (custom digit-by-digit float tokens for outputs) and sufficient multi-task training data — can achieve competitive or superior performance without any numerical engineering. This doesn't mean language models are better at regression than specialized methods; the paper is careful to note that OmniPred's single-task variant is competitive but not dominant (Figure 6). Rather, it means that language models are capable enough at regression to serve as a unified platform, and that the benefits of multi-task transfer (which specialized methods structurally cannot achieve) offset any remaining single-task performance gap.

Innovation 3: Textual Metadata as a Transfer Learning Signal — The Anonymization Ablation

One of the paper's most elegant diagnostic experiments is the anonymization ablation in Table 4, which isolates whether the model's transfer learning gains come from textual semantics or from some other aspect of multi-task training (e.g., shared optimization dynamics, implicit meta-learning across function shapes). The finding is sharp and interpretable: stripping away textual cues largely destroys the transfer learning benefit.

The experiment is simple: compare a model trained on original data (where parameter names like learning_rate and metadata like objective:'val_categorical_cross_entropy' are preserved) against a model trained on "anonymized" data where these text fields are hashed to study-dependent random strings. In the anonymized condition, each study can still be uniquely identified and trained on, but the model can no longer observe useful correlations from common textual clues across studies.

The results are striking:

• BBOB (50K studies): Original achieves 0.03 study error; anonymized achieves 0.46 — a ~15× degradation. This is despite the fact that BBOB tasks share the same underlying function classes (only shifted and dimension-scaled), meaning the model should be able to transfer knowledge even without semantic labels. The anonymization result shows that the model is heavily reliant on the text to identify which function class it's dealing with, rather than inferring function identity from the (x, y) data alone.

• BBOB (Full 1M studies): The anonymized model fails to train entirely (marked "FAIL" in Table 4). The paper hypothesizes that without textual cues to organize the massive heterogeneous data, the model cannot find any structure to latch onto — the optimization landscape becomes too chaotic.

• AutoML (26.3K studies): Original achieves 0.19; anonymized achieves 0.44 — more than 2× degradation, even though AutoML studies are real-world tasks where parameter semantics (e.g., dropout, batch_norm) carry rich information about expected behavior.

This experiment matters intellectually because it identifies what specifically makes multi-task transfer possible. It's not simply that the model sees more (x, y) pairs from diverse functions — that alone, without semantic labels, is insufficient or even harmful (as the BBOB Full anonymized failure shows). Rather, the transfer works because the model learns to associate specific parameter names and metadata fields with specific types of functional relationships. The token learning_rate becomes a feature that triggers the model to expect log-scale sensitivity; the token dropout triggers expectations about regularization effects; the metadata objective:'accuracy (%)' triggers expectations about the bounded [0, 100] output range.

This finding has a subtle but important implication for how we think about transfer learning in regression. Traditional multi-task regression (Bonilla et al., 2007) transfers knowledge through shared kernel parameters or shared latent function spaces — mechanisms that are purely numerical and require identical input spaces across tasks. OmniPred's transfer mechanism is fundamentally different: it transfers through shared lexical semantics. Two tasks don't need identical parameter spaces; they just need to share some parameter names or metadata fields for the model to leverage cross-task regularities. This is a qualitatively different kind of transfer that is only possible because the model processes parameters as text rather than as opaque tensor indices.

The anonymization result also provides a clean explanation for the negative transfer observed in Table 5 (BBOB → AutoML): pretraining on BBOB's synthetic functions teaches the model associations between parameter names (which in BBOB are generic like x1, x2) and analytic function behavior (smooth, unimodal, well-behaved). When finetuned on AutoML, these associations actively mislead the model because AutoML parameters have very different semantics and produce very different function shapes. The model would have been better off starting from scratch — a vivid demonstration that textual transfer is a double-edged sword.

Innovation 4: The Multi-Task Gain Sweet Spot — When Transfer Helps and When It Doesn't

The paper provides a nuanced characterization of when multi-task training provides benefits over single-task training, revealing a non-monotonic relationship that challenges simple "more data is better" assumptions. Figure 10 shows that the gain from multi-task training over a single-task MLP baseline is maximized when the study has approximately 50 training trials and diminishes as trial count increases.

This finding is significant because it identifies a specific operational regime where OmniPred's value proposition is strongest. The intuition is straightforward but the empirical quantification is novel:

• At very low trial counts (~0–10): Even multi-task training can't help much because there isn't enough task-specific data to identify which of the many possible function shapes (learned from other tasks) applies to the current study. The model needs some minimal signal to "latch onto" the correct task structure.

• At moderate trial counts (~50): The model has seen enough task-specific trials to identify the study's structure, but not enough to fit it precisely from scratch. This is where multi-task knowledge provides maximum leverage — the model can supplement the sparse task-specific data with general patterns learned from thousands of similar studies.

• At high trial counts (100+): Single-task training becomes increasingly sufficient on its own. The marginal benefit of multi-task knowledge shrinks because the task-specific data already provides a good fit. In the limit of very large per-study data, single-task and multi-task performance converge.

This pattern has direct practical implications for deployment. If an organization is running a new optimization task, OmniPred's pretrained model provides maximum value when the task is in the exploratory phase — enough trials have been run to roughly characterize the problem, but not enough to fit an accurate single-task model. As the optimization converges and trial counts grow, the benefit of the pretrained model diminishes, though finetuning can still help adapt the pretrained knowledge to the specifics of the task (Table 5 shows that finetuning a broadly pretrained model on AutoML achieves the same 0.15 error as a model pretrained specifically on AutoML data).

The finding also explains Figure 6, where multi-task LM outperforms single-task baselines more consistently on domains with lower "trials per study" ratios (e.g., BBOB with 30 TpS, Init2Winit with 176 TpS) than on domains with saturated single-task data. It provides a principled justification for when to use OmniPred versus a traditional single-task regressor: if you have fewer than ~50 trials and access to a large multi-task pretraining corpus, use OmniPred; if you have hundreds of trials for your specific task, the choice between OmniPred and a baseline is less critical from an accuracy standpoint (though OmniPred still offers the universality and dynamic input space benefits documented in Table 1).

Innovation 5: Language Model Samples as Nonparametric Uncertainty Estimates

The paper demonstrates that the distribution of i.i.d. temperature samples from the language model encodes calibrated predictive uncertainty, even though the model was never explicitly trained for uncertainty calibration. This emerges as a byproduct of the text-to-text regression framing and has implications beyond the paper's specific performance numbers.

Traditional regressors require explicit design choices to express uncertainty: Gaussian Processes provide closed-form predictive variances through their kernel structure; ensemble methods (random forests, deep ensembles) provide variance through disagreement across ensemble members; MLPs can output variance parameters through heteroscedastic loss functions. OmniPred, by contrast, acquires uncertainty expression for free: because the model is a stochastic generator (through temperature sampling) that produces a distribution over y-token sequences, the spread of sampled predictions naturally reflects the model's confidence.

Table 7 quantifies this: the standard deviation of LM samples (under mean aggregation) achieves a Kendall-Tau rank correlation with actual prediction error of 0.487 on AutoML and 0.366 on BBOB, substantially outperforming the Gaussian Process's predicted standard deviation (0.230 and 0.068 respectively). The Harrell-Davis standard error of the median (under median aggregation) achieves 0.412 and 0.293. These correlations are far from perfect, but they demonstrate that the model's prediction variance contains genuine uncertainty signal rather than just random noise.

What makes this finding conceptually interesting is the bimodality documented in Figure 9. When trained on randomly sign-flipped BBOB objectives (where the true y is equally likely to be f(x) or -f(x) for a given analytic function), the model's prediction samples naturally split into two modes — one positive, one negative — without any explicit mixture modeling. Traditional methods like Gaussian Process mixtures or ensembled MLPs would need to specify the number of components a priori. The language model discovers the multimodal structure from the data distribution and expresses it through sampling.

This matters for Bayesian optimization applications, where uncertainty is used as an exploration proxy. The paper doesn't demonstrate Bayesian optimization loops (it focuses on pure regression accuracy), but the finding that LM samples provide usable uncertainty estimates suggests that OmniPred could serve as a drop-in surrogate model for blackbox optimization without the uncertainty engineering that GPs require (kernel selection, hyperparameter tuning for length scales). The fact that median aggregation provides better point predictions but slightly worse uncertainty correlation than mean aggregation (0.412 vs. 0.487 Kendall-Tau on AutoML) reveals a point-prediction-vs-uncertainty tradeoff that practitioners would need to navigate.

The broader implication is that language models trained on regression tasks implicitly learn to be Bayesian about their predictions — the sampling distribution approximates the posterior predictive distribution over y given (x, m). This is an emergent property of the training process (temperature sampling from a model trained with cross-entropy loss) rather than an explicitly designed feature, and it suggests that text-to-text regression may be a natural fit for applications requiring calibrated uncertainty alongside point predictions.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper uses two families of data: (1) BBOB-Shifted, a multi-task synthetic benchmark created by applying random domain shifts to 24 different BBOB analytic functions (ElHara et al., 2019), with dimensionality ranging over [2, 6]. Each task is parameterized by (function class, dimension, shift); (2) Real-world data from Google Vizier (Golovin et al., 2017), one of the largest proprietary blackbox optimization databases, containing approximately 70M+ studies and 120B+ trials spanning ~14K distinct users. This real data covers diverse domains including AutoML, protein design, bid simulation, init2winit (deep learning tuning benchmarks), LLM latency optimization, and Vertex AI (tabular and text). The paper splits each study's trials into train/validation/test using a default 0.8/0.1/0.1 ratio, applies a hard limit of 1000 trials per study (taking only the first 1000), and removes infeasible or un-evaluated trials. For finetuning experiments on unseen tasks, studies created after the original training data export date (March 31, 2023) are used.

• Base model. All experiments use a 200M-parameter T5 encoder-decoder (Raffel et al., 2020) trained from scratch — 12 encoder layers, 12 decoder layers, 12 attention heads per layer, 64-dimensional heads, 768 embedding dimension, 2048 MLP dimension. The model is deliberately small to minimize computational costs (at most 8 GPUs for training, 1 GPU for inference) and to avoid confounding effects from generative language pretraining. The vocabulary consists of 32,000 SentencePiece subword tokens plus custom floating-point tokens for y-representation (<+>, <->, <0>–<9>, and <E-n> tokens). Training uses the Adafactor optimizer with base learning rate 0.01, square root decay, batch size 256, and a maximum of 1 million steps with early stopping based on validation loss.

• Metrics. The primary metric is normalized mean absolute error (study error), defined per-study as:

  $\frac{1}{y_{\max} - y_{\min}} \frac{1}{|\mathcal{D}^{test}|} \sum_{(x,y) \in \mathcal{D}^{test}} |\text{Aggregate}(s(x)) - y|$

  where y_max and y_min are computed from the study's training data, and Aggregate(s(x)) is the empirical median of 64 temperature-sampled predictions from the language model. Individual per-trial errors are clipped to a maximum of 1.0 to prevent catastrophic outliers from dominating study-level averages. For ranking-based evaluation (Appendix A.2, Table 9), the paper reports Kendall-Tau and Spearman correlation coefficients between predicted rankings and true y-rankings within each study. Uncertainty calibration (Table 7) is measured via rank correlation (Pearson, Kendall-Tau, Spearman) between the model's quantified uncertainty (sample standard deviation or Harrell-Davis standard error of the median) and actual prediction error, computed per-study and averaged over studies with at least 10 test trials.

• Baselines. The paper compares against four traditional single-task regressors, all trained exclusively on each individual study's training data and evaluated on its test data: (1) Gaussian Process — from the GP-Bandit implementation in Open Source Vizier (Song et al., 2024), using a Matern-5/2 kernel with ARD (automatic relevance determination), with hyperparameters optimized via MAP estimation using L-BFGS, including nonlinear preprocessing on observations; (2) XGBoost Tree — a single decision tree from XGBoost (Chen and Guestrin, 2016) with grid-searched hyperparameters (min_child_weight, learning_rate, gamma, subsample, colsample_bytree, max_depth); (3) XGBoost Random Forest — same hyperparameter grid search but using the random forest variant; (4) Multilayer Perceptron — a 2-layer ReLU network with hidden size 256, trained with Adam (learning rate 0.01, full batch, 100 epochs, MSE loss), with y-values normalized using tf.keras.layers.Normalization (subtract mean, divide by standard deviation). All baselines use the same fixed x-preprocessing pipeline: numeric parameters scaled to [0, 1] (with log or reverse-log scaling where configured), categorical parameters one-hot encoded, and conditional parameters handled via NaN imputation with learned replacement values.

• Generation budget / compute accounting. For inference, the language model generates 64 temperature samples per prediction (temperature 1.0, constrained decoding to only custom float tokens), and the empirical median is taken as the point prediction. The paper does not explicitly count "generations" as a compute budget in the way that best-of-N methods would — rather, the 64 samples are a fixed inference hyperparameter. Compute costs for baselines are not formalized; each baseline uses its own internal optimization (e.g., GP hyperparameter optimization via L-BFGS, XGBoost grid search over hyperparameters, MLP training for 100 epochs). The paper acknowledges that the language model requires accelerator usage and has higher computational cost than traditional baselines for both training and finetuning. Training used a 4×4 TPU V3; finetuning and inference each used a single 1×1 TPU V3.

• Cross-validation / statistical protocol. For the real-world data, there is no cross-validation across studies — each study has its own fixed train/validation/test split (0.8/0.1/0.1) created by random shuffling after applying the 1000-trial limit. For multi-task training, the model sees training data from many studies simultaneously, but test evaluation is always on held-out trials from the same studies. For the finetuning experiments on unseen tasks (Section 5.3), the "unseen" tasks are those created after the pretraining dataset export date — they are genuinely out-of-sample studies with new users and objectives. The paper does not report confidence intervals or statistical significance tests for any of its main results; study-level errors are aggregated via simple means over the set of evaluated studies. For the BBOB synthetic experiments, unseen tasks are created by holding out specific shifts not seen during training, allowing evaluation of generalization over the metadata m (function class, dimension, shift).

Main Quantitative Results

5.1 Simultaneous Regression: Can One Model Handle Arbitrary Input Spaces and Objective Scales?

Headline finding: Yes — a single trained language model captures the overall shape of analytic functions with high precision (BBOB) and produces strong diagonal fits on diverse real-world tasks spanning many orders of magnitude in y.

The paper provides two complementary visualizations to establish this baseline capability:

BBOB synthetic functions (Figure 3, Section 5.1). For a model trained on multi-task BBOB data, the paper plots prediction samples over selected 4D functions with unseen shifts (i.e., shift values not encountered during training). The model's empirical mode (bolded line) closely tracks the ground truth function shape across varying coordinate values x_i, with the min/max envelope from 10 samples capturing the spread. The paper reports that for fully multi-task-trained BBOB models, study error reaches 0.01 (Table 6, BBOB Full 1M with median aggregation), meaning the average prediction error is only 1% of the observed y-range. This level of precision — on held-out shifts of analytic functions the model has never seen — demonstrates that the model is genuinely learning the function class behavior from textual metadata (function name, dimension) rather than memorizing specific shift instances.

What this establishes: The language model can extrapolate to new instances of known function classes when given metadata describing which function class is being evaluated. This is a specific form of transfer — not across different function classes (which would require recognizing completely new shapes), but across different instantiations (different shifts, different dimensionalities) of the same class.

Real-world data (Figure 4, Section 5.1). For a model trained on the full real-world corpus, the paper presents scatter plots of predicted vs. actual y for 8 hand-selected studies with drastically different input spaces and objective types:

• CIFAR10 (4 Double parameters) — image classification accuracy
• LM1B (4 Double parameters) — language modeling loss
• Bid Simulation (4 Double parameters) — proprietary ad simulation metrics
• Protein Design (60 Categories) — protein structure optimization
• LLM Latency (31 Hybrid parameters) — inference latency optimization
• AutoML (3 head parameters, 42 total possible with conditionals) — automated ML tuning
• MobileNet (10 Discrete parameters) — neural architecture search
• Spam Filter (13 head parameters, 15 total possible) — text classification

The paper emphasizes the "diagonal fit" — points clustering around the y=x line — as visual evidence of accurate prediction. However, no quantitative per-study error numbers are reported for these specific studies; the figure serves as a qualitative demonstration of the model's breadth rather than a precise accuracy claim. The study-specific objective names are redacted (corporate sensitivity), so the reader cannot verify the exact nature of what's being predicted, though the parameter spaces are described.

What this establishes: A single 200M-parameter model can make reasonable predictions across completely unrelated tasks — from vision to language modeling to protein engineering to hardware optimization — without any task-specific architecture changes. This is the core "universal regressor" claim in action, albeit demonstrated qualitatively rather than through systematic across-task error quantification.

Critical nuance: The paper does not report how many of the 8 displayed studies are representative vs. cherry-picked. The selection is described as "hand-selected" with the goal of being "representative of objectives tuned commonly in real world settings," but without reporting aggregate statistics across all studies in each domain, it's impossible to know whether these plots represent typical performance or best-case examples. The domain-level aggregate errors in Figure 6 provide the actual quantitative picture.

5.2 Multi-Task Transferrability: Does Training on More Studies Improve Performance?

Headline finding: Multi-task training consistently outperforms single-task training, with accuracy improving as more studies are added, eventually surpassing all traditional single-task baselines — but the magnitude of gain depends on the domain and on preserving textual metadata.

Scaling with number of training studies (Figure 5, Section 5.2). The paper plots mean study prediction error as a function of the number of distinct studies used in multi-task training (log scale):

• AutoML (left panel): Prediction error drops steadily from ~0.42 (single-task MLP baseline) to ~0.19 at 26.3K training studies, crossing below the best single-task baseline (Gaussian Process at ~0.37) at approximately 1K studies, and below the random forest (~0.18) and tree (~0.17) baselines at around 10K studies. At 26.3K studies, the LM achieves ~0.19, outperforming all colored horizontal baseline lines except possibly the tree baseline (~0.17) — though the exact crossing point depends on reading the log-scale x-axis precisely.

• BBOB (right panel): Error drops even more dramatically — from ~0.35 at 50 studies to ~0.03 at 1M studies, substantially outperforming all baselines (MLP at ~0.14, tree at ~0.13, random forest at ~0.12, GP at ~0.10) by approximately 3–5× at maximum scale.

What the log-scale x-axis reveals: The improvement is roughly log-linear — each order-of-magnitude increase in training studies produces a consistent reduction in error. For AutoML, going from 500 to 5K studies reduces error from ~0.32 to ~0.22; going from 5K to 26.3K drops it further to ~0.19. The curve does not appear to have plateaued at 26.3K, suggesting that additional training data would yield further improvements. For BBOB, the curve similarly continues improving to 1M studies.

Single-task LM vs. multi-task LM vs. baselines (Figure 6, Section 5.2). The paper provides a detailed breakdown across seven domains, comparing single-task LM, multi-task LM, and the four traditional baselines (MLP, Tree, Random Forest, Gaussian Process). The key patterns:

• BBOB (1M studies, 30 TpS, ~4.0 space size): Multi-task LM achieves ~0.01 error — dramatically better than all baselines (MLP: ~0.14, GP: ~0.10). Single-task LM (~0.17) is roughly competitive with baselines but slightly worse than GP. This is the domain where multi-task training provides the largest absolute gain.

• Bid Simulation (22K studies, 698 TpS, ~4.6 space size): Multi-task LM (~0.10) outperforms single-task LM (~0.33) and roughly matches the best baseline (GP: ~0.09). The small space size and high trial count mean single-task baselines already perform well, so multi-task gains are more modest.

• AutoML (540K studies, 250 TpS, conditional space with ~3.3 root and ~29.9 total parameters): Multi-task LM achieves ~0.15, outperforming all baselines (best baseline is Tree at ~0.18). Single-task LM (~0.25) is competitive but slightly worse than MLP (~0.23) and Tree (~0.25). This is the domain with the most complex input space (conditional, hybrid types), where the language-based representation's natural handling of conditionals provides an advantage even in the single-task setting.

• Init2winit (2K studies, 176 TpS, ~3.6 space size): Multi-task LM (~0.32) outperforms single-task LM (~0.55) and all baselines (best baseline: Random Forest at ~0.42). This is a relatively small multi-task training set (2000 studies), yet multi-task still provides substantial gains.

• Protein Design (54K studies, 584 TpS, ~125.6 space size): Multi-task LM (~0.12) outperforms all baselines (best baseline: MLP at ~0.19), while single-task LM (~0.25) is worse than MLP and roughly matches the tree baseline. The large space size (125.6 average) makes this a domain where the LM's variable-length representation is particularly advantageous — traditional methods struggle with 60 categorical parameters creating an enormous combinatorial space.

• Vertex AI Tabular (1.4M studies, 88 TpS, conditional space ~4.6 root, ~42.4 total): Multi-task LM (~0.10) outperforms all baselines (best baseline: Tree at ~0.18). Single-task LM (~0.25) is competitive with GP (~0.27) but worse than tree-based methods.

• Vertex AI Text (544K studies, 118 TpS, ~56.0 space size): Multi-task LM (~0.13) outperforms all baselines (best baseline: Random Forest at ~0.18). Single-task LM (~0.20) roughly matches the tree baseline.

A consistent pattern across all seven domains: Multi-task LM outperforms single-task LM, and multi-task LM outperforms or matches the best traditional baseline in every domain. Single-task LM is competitive with traditional baselines (often in the middle of the pack) but rarely dominant — confirming that the language-based representation alone doesn't provide a systematic advantage over well-tuned traditional methods; rather, it's the combination of language-based representation with multi-task training at scale that drives the gains.

The anonymization ablation (Table 4, Section 5.2): This is the experiment that isolates whether multi-task gains come from textual semantics or from some other aspect of multi-task training. The results are clean:

For BBOB with 50K studies, anonymization degrades error by 15× (0.03 → 0.46). For AutoML, anonymization degrades error by ~2.3× (0.19 → 0.44) at 26.3K studies and ~2.9× (0.15 → 0.43) at 540K studies. The most dramatic result is the BBOB Full 1M anonymized case, where the model fails to train entirely — the data becomes too large and heterogeneous without textual cues to organize it.

What the anonymization reveals mechanistically: The model is not learning purely from the numerical patterns in (x, y) data across tasks. It relies heavily on parameter names, string values, and metadata to identify which type of function it's predicting. When learning_rate is hashed to a random string like a3f8c2, the model can no longer leverage its knowledge that "learning_rate" across many studies typically behaves with log-scale sensitivity and interacts with optimizer type — it must re-derive these relationships from scratch within each individual study. The anonymized results (0.43–0.46 error) roughly match the single-task performance range, confirming that without transferrable textual cues, multi-task training provides minimal benefit.

5.3 Online Finetuning: Does Pretraining Help on Completely New Tasks?

Headline finding: A model pretrained on broad real-world data can finetune to accuracy matching domain-specific pretraining on in-domain tasks, but pretraining on mismatched domains (synthetic functions) actively hurts compared to training from scratch.

Pretraining domain matters (Table 5, Section 5.3). The paper evaluates finetuning on AutoML studies from four different pretraining starting points:

Key observations:

• Single-task training from scratch achieves 0.20 error after finetuning — this is the baseline for "no pretraining knowledge."
• BBOB-pretrained model achieves 0.45 error after finetuning — substantially worse than single-task from scratch. This is negative transfer: the synthetic analytic function knowledge actively interferes with learning real-world AutoML tasks. The BBOB model has learned that parameter names like x1, x2 correspond to smooth, well-behaved analytic functions, and these associations mislead it when applied to AutoML parameters like dropout, batch_norm, learning_rate.
• AutoML-pretrained model achieves 0.15 error both before and after finetuning — the model has already saturated on these tasks from pretraining, so additional finetuning provides negligible benefit.
• All RealWorldData-pretrained model starts at 0.31 before finetuning (worse than the AutoML-specialized model because it's diluted by data from other domains), but after finetuning achieves 0.15 — the same accuracy as the AutoML-specialized model. This is the key positive result: broad multi-task pretraining provides transferrable knowledge that can be refined to domain-specific accuracy with a small amount of finetuning data.

Generalization to unseen tasks — new users, new objectives (Figure 7, Section 5.3). The paper evaluates on 1000 studies created after the pretraining dataset export date, filtered over random distinct users (so these are genuinely new tasks from new users with new objectives). The aggregate comparison:

What this shows: The pretrained-but-not-finetuned model performs terribly (0.68) — worse than any baseline — because the pretraining data is "stale" and doesn't cover these new tasks. However, after finetuning on the new task's training data, the pretrained model achieves 0.21, outperforming all baselines (best baseline is MLP at 0.25) and single-task LM from scratch (0.28). This demonstrates that pretraining on old data provides transferrable knowledge that accelerates learning on genuinely new tasks, even when the tasks come from new users with potentially new objective types.

Figure 7 (left) shows a per-study breakdown for 8 example unseen studies, plotting LM study error for three conditions (single-task, pretrained-only, pretrained+finetuned). The pretrained+finetuned bars are consistently lower than single-task bars for most studies, though the paper notes there are "few cases of negative transfer" (studies where pretrain+finetune performs worse than single-task). These negative transfer cases are not analyzed in detail.

Connecting to the multi-task scaling results: The finetuning results complement Figure 5's scaling curves. Figure 5 shows that multi-task pretraining improves accuracy with more pretraining studies. Table 5 and Figure 7 show that even when pretraining doesn't directly cover the target task, the transferred knowledge can be adapted through finetuning — provided the pretraining domain is relevant (RealWorldData → AutoML works; BBOB → AutoML fails). This establishes a practical deployment scenario: pretrain once on a large offline corpus, then finetune cheaply on each new task.

Ablation Studies and Robustness Checks

Effect of sampling and aggregation (Table 6, Section 6.1): The paper ablates three aggregation methods for combining 64 temperature samples into a point prediction. On AutoML (Full 540K), median aggregation (0.15 error) substantially outperforms max-likelihood (0.22) and mean (0.23) — a ~35% relative improvement. On BBOB (Full 1M), all three methods achieve 0.01 because the model has nearly perfectly regressed the analytic functions and produces few outliers. The paper hypothesizes that median's robustness to "hallucinated" outlier samples (caused by wrong exponent or leading-digit token predictions) is the key advantage. Figure 8 confirms that increasing sample count reduces error monotonically for all methods, with diminishing returns beyond ~16–32 samples for most tasks.

y-tokenization method (Table 8, Appendix A.1): Comparing three float-token representation schemes: (1) Default digit-by-digit with sign and exponent (<+><1><2><3><4><E-2>), (2) Merged mantissa (<+1234><E-2>), (3) Exponent-before-mantissa (<+><E-2><1><2><3><4>). In multi-task training (both AutoML Full 540K and BBOB Full 1M), all three methods achieve identical error (0.15 and 0.01 respectively) — the choice doesn't matter with abundant data. In single-task training, the differences are dramatic: default achieves 0.21 on AutoML and 0.17 on BBOB; exponent-before-mantissa achieves 0.24 and 0.17; merged mantissa achieves 0.73 and 0.41. The merged mantissa's catastrophic failure in low-data regimes confirms that having 18K+ possible mantissa tokens makes the problem sample-inefficient — the model needs many examples of each specific mantissa value to learn its numerical meaning, whereas digit-by-digit schemes decompose the problem into only 10 digit tokens.

Textual anonymization (Table 4, Section 5.2): As discussed in the main results, hashing parameter names and metadata to random study-specific strings eliminates most multi-task transfer gains. On BBOB (50K), error degrades from 0.03 to 0.46. On BBOB (Full 1M), the model fails to train entirely ("FAIL"). On AutoML, error degrades from 0.19 → 0.44 (26.3K) and 0.15 → 0.43 (540K). The BBOB 1M anonymized failure is particularly informative: without textual structure to organize the massive heterogeneous dataset, the optimization landscape becomes too chaotic for the model to find any useful signal.

Uncertainty calibration (Table 7, Section 6.2): The paper measures rank correlation between model-expressed uncertainty and actual prediction error, per-study averaged over studies with ≥10 test trials. For the LM under mean aggregation, sample standard deviation achieves Kendall-Tau of 0.487 on AutoML (641 studies) and 0.366 on BBOB (all studies) — substantially higher than the Gaussian Process's predicted standard deviation (0.230 and 0.068, respectively). Under median aggregation, the Harrell-Davis standard error achieves 0.412 and 0.293. The finding that mean aggregation provides better uncertainty correlation (0.487 vs. 0.412) despite worse point predictions (Table 6: 0.23 vs. 0.15) highlights a point-prediction-vs-uncertainty tradeoff.

Multimodal uncertainty (Figure 9, Section 6.2): When trained on randomly sign-flipped BBOB objectives (where y is equally likely to be f(x) or −f(x)), the LM's prediction samples naturally split into two modes — positive and negative — without requiring a pre-specified number of mixture components. This demonstrates that the sampling distribution can capture nonparametric, multimodal predictive uncertainty as an emergent property.

Study size vs. multi-task gain (Figure 10, Section 6.3): The paper bins individual AutoML tasks by their training trial count and plots the difference in study error between the MLP baseline and the multi-task LM. Gains are maximized at approximately 50 training trials and diminish as trial count increases toward 200+. The curve is non-monotonic near zero: at very low trial counts (~5–10), gains are actually smaller than at ~50, because the model needs some minimal task-specific signal to identify the correct structure. At high trial counts (200+), single-task training becomes increasingly sufficient, and the multi-task advantage shrinks toward zero.

Ranking metrics (Table 9, Appendix A.2): Although the paper focuses on pointwise prediction error, it also reports ranking-based metrics (Kendall-Tau, Spearman correlation) which are agnostic to y-scaling and relevant for evolutionary algorithms. Multi-task LM achieves the best ranking metrics on BBOB (0.92/0.96 Kendall-Tau/Spearman), Init2Winit (0.65/0.74), Protein Design (0.72/0.81), and Vertex AI domains (0.57/0.72 tabular, 0.49/0.58 text). It is competitive on Bid Simulation (0.70/0.84 vs. GP's 0.80/0.91) and AutoML (0.61/0.73, outperforming all baselines). Single-task LM performs poorly on ranking across all domains (e.g., 0.01/0.01 on BBOB), suggesting that multi-task training is particularly important for learning relative ordering rather than just point values.

Critical Assessment

The experimental results provide substantial evidence for the paper's central claims, though with important limitations on generalizability, quantification, and what specific mechanisms are actually demonstrated.

Claim: "Language models are capable of very precise numerical regression using only textual representations." This claim is supported with quantitative rigor on BBOB synthetic functions, where the model achieves study error of 0.01 with full multi-task training (Table 6) — meaning average prediction error is only 1% of the observed y-range. The BBOB visualizations (Figure 3) show the model tracking analytic function shapes closely, including on unseen shifts. On real-world data, the evidence is more mixed: Figure 4 shows qualitatively strong diagonal fits for 8 hand-selected studies, but no quantitative per-study error is reported for these specific examples. The domain-level aggregate errors in Figure 6 provide the actual quantitative picture — ranging from ~0.01 (BBOB) to ~0.32 (Init2Winit multi-task) — which is "precise" for some domains but quite noisy for others. A study error of 0.32 means the average prediction is off by nearly a third of the observed y-range, which would be unacceptable for many experimental design applications. The paper does not characterize the distribution of per-study errors (what fraction of studies achieve error <0.05, <0.1, etc.), making it difficult to assess reliability.

Claim: "By leveraging multi-task learning across vastly different input spaces and objectives, OmniPred can significantly outperform traditional regression models." This claim is supported by Figure 6, which shows multi-task LM outperforming the best traditional baseline in all seven evaluated domains. However, several qualifications are necessary. First, the "outperformance" magnitude varies dramatically: it's large for BBOB (~10× better than the best baseline), moderate for Protein Design (~1.6× better), and marginal for Bid Simulation (~1.1× better, within what could be statistical noise). Second, no confidence intervals or significance tests are reported, so it's impossible to know whether the Bid Simulation or AutoML results are statistically reliable or within the noise floor. Given that each domain aggregates over many studies of varying sizes and difficulties, standard errors could be substantial. Third, the baselines use fixed hyperparameters (as noted in Appendix C.4): the GP uses a specific kernel and optimization procedure; the MLP uses a fixed 2-layer architecture with specific training settings; the tree methods use grid search over a fixed set of hyperparameters. It's possible that more careful per-study hyperparameter tuning for baselines would close some of the gaps. The paper acknowledges this explicitly: "We emphasize that our paper's contributions are mostly on regression using flexible string representations and large-scale multi-task training, and do not claim to replace widely accepted baselines in single-task, apples-to-apples comparisons."

Claim: "Transfer learning benefits persist even on unseen tasks after online finetuning." This claim is supported by Figure 7 and Table 5. The pretrain+finetune model achieves 0.21 on 1000 unseen studies vs. 0.28 for single-task from scratch and 0.25 for the best baseline (MLP). The magnitude of benefit (~25% relative improvement over single-task LM) is meaningful but modest. The negative transfer cases noted in the text ("there are few cases of negative transfer" in Figure 7 left) are not quantified — we don't know what fraction of unseen studies experience negative transfer, nor what characterizes those studies. Additionally, the "unseen" tasks are from new users but still come from the same Google Vizier service with the same parameter space schema (OSS Vizier format) — they are "unseen" in the sense of temporal holdout, not in the sense of fundamentally different task types or representation formats. The finetuning cost (single 1×1 TPU V3, up to 30 epochs) is not compared against the cost of single-task training from scratch, making it difficult to assess the practical efficiency benefit.

Missing experiments that would strengthen the paper:

1. Per-study error distributions: The paper reports only mean study errors aggregated over domains (Figure 6) or selected studies (Figure 4). Reporting the distribution (histograms, percentiles) of per-study errors would reveal whether the mean is driven by a few very difficult studies or represents typical performance. This matters because experimental design practitioners need to know the worst-case behavior, not just the average.

2. Statistical significance: No confidence intervals, standard errors, or hypothesis tests are reported for any comparison between methods. With study counts ranging from 2K (Init2winit) to 1.4M (Vertex AI Tabular), the effective sample sizes and variance could vary enormously.

3. Calibration of prediction intervals: The paper demonstrates that uncertainty (sample standard deviation) correlates with error (Table 7), but does not evaluate whether the actual prediction intervals are calibrated (e.g., do 90% prediction intervals contain the true y 90% of the time?). This is the standard evaluation for probabilistic regression and would be necessary before using OmniPred in Bayesian optimization loops.

4. Ablation on model scale: All experiments use a single 200M-parameter T5. It's unknown whether larger models would improve accuracy (through greater capacity for multi-task knowledge) or hurt (through overfitting or optimization difficulties). The paper acknowledges this as a deliberate simplicity choice, but a small scaling experiment (e.g., 50M vs. 200M vs. 800M) would inform whether the approach benefits from scale in the way language models typically do.

5. Comparison against a multi-task GP: The paper claims traditional methods cannot multi-task across heterogeneous input spaces, but contextual GP variants (Krause and Ong, 2011) can handle variable-length inputs if they're embedded into a shared space. No attempt is made to create such a baseline (e.g., by embedding all parameter configurations into a fixed-dimensional representation using a learned featurizer), which would provide a fairer comparison against OmniPred's text-based multi-task learning.

6. Sensitivity to prompt formatting: The paper uses a specific key-value format for x-representation. Would results degrade with different formatting (e.g., JSON, natural language descriptions, different separator characters)? The anonymization ablation shows that semantic content matters enormously, but doesn't test whether the formatting structure matters independently.

7. Outlier analysis for negative transfer: Table 5 shows BBOB → AutoML finetuning performs worse than single-task from scratch (0.45 vs. 0.20). This is one of the most interesting results — it demonstrates that transfer can actively hurt — but the paper does not analyze what specific BBOB knowledge interferes with AutoML learning. Understanding this mechanism would help practitioners avoid similar negative transfer in deployment.

8. Computational cost comparison: The paper acknowledges that the LM "requires accelerator usage and has a relatively higher computational cost" than baselines but never quantifies this cost in FLOPs, wall-clock time, or dollar terms. Without this, the accuracy gains cannot be weighed against the computational price. A 200M-parameter model with 64-sample inference may be substantially more expensive than a GP or XGBoost model for the same prediction task.

Conditional nature of the claims:

• The "very precise numerical regression" claim holds most strongly for synthetic functions with abundant multi-task training data (BBOB 0.01 error) and most weakly for real-world domains with fewer training studies (Init2Winit 0.32 error with only 2K pretraining studies). Precision is a function of both domain complexity and pretraining scale.

• The "significantly outperform traditional regression models" claim holds when multi-task pretraining data is abundant and semantically rich (Figures 5–6) and single-task data is limited (Figure 10, peak gain at ~50 trials). It weakens or fails when pretraining data is anonymized (Table 4), when pretraining domain is mismatched (Table 5, BBOB → AutoML), or when single-task data is plentiful (Figure 10, diminishing gains above 100–200 trials).

• The "transfer learning benefits persist on unseen tasks" claim holds for tasks within the same broad domain and representation format (Google Vizier studies with OSS Vizier schema). It has not been tested on tasks from fundamentally different regression domains (e.g., physical simulations, chemical property prediction, financial forecasting) where the shared textual cues might be weaker or absent.

The single-model-family limitation: All experiments use a single architecture (T5 encoder-decoder, 200M parameters, trained from scratch). The paper cannot distinguish whether the results are specific to T5's inductive biases, to encoder-decoder architectures generally, or to language model architectures broadly. The deliberate choice to avoid pretrained model confounds is scientifically clean but limits claims about "language models" in general — the evidence is about this specific model class trained with this specific procedure.

6. Limitations and Trade-offs

6.1 Difficulty Estimation Is Prohibitively Expensive and Unaccounted For

The assumption or constraint. The entire compute-optimal framework assumes that a practical difficulty estimator exists to route queries into the appropriate strategy. The paper's difficulty estimation method requires generating 2,048 samples per question and computing either ground-truth pass@1 (oracle) or averaging the PRM's predicted final-answer correctness (predicted). The authors acknowledge this cost explicitly:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

The consequence. The headline 4× efficiency gains are computed after difficulty is known, without amortizing the cost of learning it. In deployment, total cost would be (difficulty estimation) + (strategy execution), and the former dominates: 2,048 samples to estimate difficulty dwarfs the 16–256 generation budgets where compute-optimal scaling shows its largest relative advantage. For example, matching best-of-64 with only 16 generations (a 4× saving) requires first spending 2,048 generations for difficulty estimation — a net loss of ~32× in total compute. This makes the current approach strictly impractical for real deployment, and the 4× figure should be understood as an upper bound conditioned on free difficulty information, not an achievable efficiency gain.

What evidence exists in the paper. The paper reports this cost only qualitatively in Section 3.2. No experiment accounts for difficulty estimation cost in any budget calculation or plots it against the compute-optimal scaling curves. The predicted difficulty bins (which avoid ground-truth labels) still require 2,048 samples plus PRM scoring — the same order of magnitude cost.

Mitigation status. The paper does not attempt to address this gap. Section 3.2 frames the cost as an "exploration-exploitation tradeoff" and Section 8 suggests future work on "pretraining or finetuning models to directly predict difficulty of a question." No lightweight difficulty estimator, adaptive scheme, or cost-benefit analysis is developed or tested. Until this is resolved, the method remains an analysis framework rather than a deployable system.

6.2 The Approach Fails Completely on Hard Problems

The assumption or constraint. The test-time compute methods assume that the base model already produces correct solutions at some non-trivial rate — i.e., that the pass@1 is meaningfully above zero. The paper's compute-optimal policy depends on the existence of correct trajectories to find (via search) or refine (via revisions). The authors are explicit about this boundary:

"Test-time compute provides minimal gains on problems that are fundamentally outside the base model's capability range."

The consequence. On difficulty bin 5 (the hardest problems), no method — search, revisions, or compute-optimal combinations — achieves meaningful improvement regardless of budget. In Figure 3 (right), bin 5 accuracy hovers at 1–3% for all search methods and all budgets. In Figure 7 (right), bin 5 shows 2–3% accuracy irrespective of the sequential-to-parallel ratio. In the FLOPs-matched comparison (Figure 9), bin 5 scaling lines are essentially flat near 0–5% and are consistently below the 14× larger model's greedy performance. This establishes a hard capability ceiling: test-time compute amplifies existing capability but does not create it. For problems where the base model has near-zero probability of generating a correct answer — whether due to insufficient pretraining data, missing knowledge, or out-of-distribution complexity — no amount of inference-time computation helps.

What evidence exists in the paper. The pattern is replicated across search (Figure 3, right, bin 5), revisions (Figure 7, right, bin 5), and the FLOPs-matched comparison (Figure 9, bin 5). It holds consistently across all methods and all budgets studied.

Mitigation status. The paper does not propose any mechanism to address this limitation — nor could it, since the problem is fundamental to the "amplify existing capability" framing. The only path to harder problems is scaling pretraining. The authors acknowledge this in Section 7: "Pretraining is almost always more effective" on the hardest problems. This is a clear boundary condition rather than a solvable limitation within the framework.

6.3 Single Benchmark, Single Model Family — No Evidence of Generality

The assumption or constraint. All experiments use the MATH benchmark (500 test questions) with PaLM 2-S* as the base model. The paper provides no evidence that the findings generalize to other reasoning domains (code generation, logical reasoning, scientific QA), other model families (GPT, LLaMA, Gemini), other model scales, or non-reasoning tasks (factual QA, summarization).

The consequence. Several aspects of the results could be model-specific or benchmark-specific. The PRM's over-optimization behavior (beam search degrading on easy problems, Figure 3 right) depends on PaLM 2-S*'s output distribution and error patterns — a differently calibrated model might show different difficulty-dependent scaling curves. The revision model's edit-distance-based training data construction may work differently for models with different in-context learning capabilities. The MATH benchmark consists exclusively of competition-level math problems with exact ground-truth answers, clean grading functions, and multi-step reasoning — exactly the domain where process-based verification should be most effective. Whether difficulty-dependent allocation works on tasks where "correctness" is ambiguous, multi-dimensional, or subjective is completely unexplored. The authors do not claim generality, but the paper's framing ("compute-optimal test-time scaling" without domain qualifiers) implicitly suggests it.

What evidence exists in the paper. All figures and tables report MATH results with PaLM 2-S*. The paper provides no direct evidence of this limitation because it conducts no cross-model or cross-benchmark experiments. The sample size of 500 test questions, split into five difficulty quintiles of ~100 each and further split by two-fold cross-validation, means the compute-optimal policy is selected based on ~50 questions per fold per bin — a small sample where the selected strategies may not be robust even within MATH, let alone across benchmarks.

Mitigation status. The paper does not address this limitation. Section 4 briefly argues that PaLM 2-S* is "representative of the capabilities of many contemporary LLMs," but provides no evidence for this claim. No replication on other benchmarks or model families is attempted or suggested as future work.

6.4 Verifier Over-Optimization Is a Hard Ceiling, Not a Solved Problem

The assumption or constraint. All search-based methods rely on a trained process reward model (PRM) to score intermediate solution steps. The PRM is trained via Monte Carlo rollout supervision on the base model's own outputs, which means it inherits the base model's biases and has finite accuracy. The assumption underlying search is that higher PRM scores correspond to genuinely better solutions — an assumption that breaks down under aggressive optimization.

The consequence. Beam search, the strongest optimizer, degrades performance on easy problems at high budgets (Figure 3, right: bin 1 accuracy decreases from ~78% to ~77% as budget goes from 4 to 256). Lookahead search — the most powerful optimization method — paradoxically performs worst overall (Figure 3, left). Qualitative examples in Appendix M show search producing degenerate outputs: repetitive low-information steps at the end of solutions (Figure 29) and overly short 1–2 step solutions that score highly under the PRM but are incorrect. The compute-optimal policy mitigates this by routing easy problems away from aggressive search, but it does not solve the underlying problem: on medium-difficulty problems where beam search is actually deployed, over-optimization still limits the scaling ceiling — the beam search curves flatten and sometimes decline well before the full budget is exhausted (Figure 3, right). The verifier's reliability fundamentally bounds what test-time compute can achieve.

What evidence exists in the paper. Figure 3 (right) shows beam search degradation on easy problems. Figure 3 (left) shows lookahead search underperforming. Appendix M provides qualitative examples of PRM-exploiting degenerate outputs. Section 5.3 explicitly discusses over-optimization as the explanation for these patterns.

Mitigation status. The compute-optimal policy partially mitigates this by avoiding aggressive search where the verifier is unreliable (routing easy problems to best-of-N instead of beam search), but this is a routing workaround rather than a solution to verifier quality. The paper does not propose methods for training more robust verifiers, ensemble verification, constrained search, or any other mechanism for pushing back the over-optimization threshold. Section 8 flags this as future work: the finding that verifier quality is the primary bottleneck redirects research attention toward building more robust reward models rather than developing more sophisticated search algorithms.

6.5 The 14× Larger Model Baseline Is Weak — Understating Pretraining's Effectiveness

The assumption or constraint. The FLOPs-matched comparison in Section 7 scales only model parameters when increasing pretraining compute (following the LLaMA paradigm), holding training data fixed. The paper acknowledges this departs from compute-optimal pretraining:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

Additionally, the larger model uses only greedy decoding with no test-time compute augmentation of its own — no majority voting, no best-of-N, no verifier-guided selection.

The consequence. A Chinchilla-optimal model (scaling both parameters and data, per Hoffmann et al., 2022) trained with 14× more total FLOPs would likely outperform a parameter-only-scaled model, making the pretraining baseline structurally weaker than necessary. Furthermore, giving the larger model even a modest test-time compute budget (e.g., best-of-8 or majority voting) would create a much stronger comparison point — the FLOPs-matched question should be: "for the same total compute, does the smaller model with sophisticated inference beat the larger model with some inference optimizations?" rather than the asymmetric comparison tested (small model with optimal inference vs. large model with greedy decoding). The reported advantages of test-time compute over pretraining — e.g., +27.8% relative improvement on easy questions at R ≪ 1 (Figure 1, top-right bar chart) — may shrink or reverse against a properly compute-optimal larger model.

What evidence exists in the paper. Section 7 describes the FLOPs-matched setup, explicitly stating the parameter-only-scaling choice and the greedy decoding baseline. Figure 9 shows the main comparison; Figure 1's bar charts show the grouped relative improvements. The paper provides no ablation comparing against a compute-optimally trained larger model or a larger model with any test-time compute augmentation.

Mitigation status. The authors acknowledge the parameter-only scaling caveat explicitly (quoted above) and defer compute-optimal pretraining comparisons to future work. However, the greedy decoding baseline — giving the larger model no inference-time augmentation at all — is not similarly acknowledged as a limitation. This is the more impactful choice: even small amounts of test-time compute (best-of-4 or majority voting over a few samples) would substantially strengthen the baseline, particularly on easy-to-medium problems where sampling diversity matters most. The paper's asymmetric comparison inflates the apparent advantage of test-time compute over pretraining.

6.6 Revision Model Fragility: 38% Correct-to-Incorrect Reversion and Sensitivity to Training

The assumption or constraint. The revision model is trained via supervised fine-tuning on trajectories constructed from offline data: the model sees only sequences of incorrect answers followed by a correct answer. It never sees examples where the current answer is already correct, nor does it see examples where a correct answer should be preserved rather than revised.

The consequence. At test time, the revision model has a ~38% correct-to-incorrect reversion rate: when a revision chain produces a correct answer at step k, the subsequent step k+1 has a 38% chance of "revising" it back to an incorrect answer (Section 6.1). This is a direct consequence of the training data construction: the model learned that all in-context answers are incorrect and that it should always produce a different answer. It never learned to recognize when to stop revising. The paper mitigates this with majority voting or verifier-based selection across the entire revision chain rather than taking the last output, but these are post-hoc patches — they select the best answer from a chain that may have produced and then discarded several correct answers along the way. This is fundamentally wasteful: compute is spent on revisions that actively destroy good answers.

What evidence exists in the paper. Section 6.1 reports the ~38% reversion rate explicitly. The revision model figures (Figure 6 left, Figure 7) show that sequential revision accuracy improves on average but with substantial per-step variance, consistent with occasional reversions from correct to incorrect.

Mitigation status. The paper mitigates this behavior through chain-level answer selection (majority voting or verifier-based) rather than always taking the final revision, but does not address the root cause. The acknowledgment is implicit rather than highlighted as a limitation. Additionally, the ReST^EM experiment (Appendix K, Figure 16) demonstrates that attempting to further optimize the revision model through on-policy RL training backfires catastrophically: fully sequential performance with the ReST^EM model drops to ~33.5% at 256 generations versus ~38.5% at the optimal ratio. The authors hypothesize that on-policy data collection "exacerbates spurious correlations in revision data," but do not fully diagnose the failure. This suggests revision training is fragile in ways that are not well understood — the positive results depend on specific choices (offline data construction, edit-distance-based pairing) that may not transfer to other settings or optimization procedures.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a category shift in how regression is framed — from a task-specific numerical optimization problem requiring manual featurization to a universal text-to-text translation task handled by a single language model. The shift is not incremental: it dismantles the entire preprocessing pipeline (one-hot encoding, rescaling, NaN imputation, nonlinear y-warping) that has been considered essential to regression for decades and replaces it with a single representational choice — express everything as key-value text and decode into custom float tokens.

The magnitude of this shift is best understood by what it renders unnecessary. Traditional regression workflows require, per task: (1) defining a fixed-length tensor schema with known bounds for continuous parameters and known vocabularies for categorical ones; (2) normalizing x-values to a unit hypercube; (3) applying nonlinear transformations (Box-Cox, Yeo-Johnson) to y-values to handle scale variation; (4) implementing conditional parameter handling through NaN imputation with learned replacement values (Appendix C.3); (5) training a separate model instance for each task because the tensor shapes are incompatible across tasks. OmniPred eliminates all five steps simultaneously by representing x as text and y via constrained float token decoding. The fact that a single 200M-parameter model can process 60-category protein design spaces alongside 4-parameter CIFAR-10 tuning studies without any architecture change or featurization pipeline is a demonstration of representational universality — not just an accuracy improvement over baselines.

The paper also resolves a genuine contradiction in the literature about whether language models can handle precise numerical reasoning. Prior work had established reasonable skepticism: transformers struggle with arithmetic when numbers are token-based rather than explicit numerical values (Nogueira et al., 2021), and even large models make basic numerical errors in multi-step reasoning (Hendrycks et al., 2021). This created a prior that language models are fundamentally unsuited for tasks requiring high-precision numerical outputs. OmniPred challenges this prior directly: with the right output tokenization (digit-by-digit custom float tokens, constrained decoding) and sufficient multi-task training data, a 200M-parameter model trained from scratch achieves study errors as low as 0.01 on synthetic functions (BBOB) and 0.10–0.15 across several real-world domains (Figure 6). The model is not merely memorizing — it generalizes to unseen function shifts (BBOB, Figure 3) and unseen tasks through finetuning (Figure 7, 0.21 vs. 0.28 for single-task). This doesn't mean language models are better at regression than specialized methods; rather, it establishes that they are capable enough that the benefits of multi-task transfer and representational flexibility can offset any remaining single-task performance gap.

The anonymization ablation (Table 4) provides a mechanistic insight that shifts how we should think about transfer learning in regression. Traditional multi-task regression (Bonilla et al., 2007; Krause & Ong, 2011) transfers knowledge through shared kernel parameters or shared latent function spaces — mechanisms that are purely numerical and require identical input spaces across tasks. OmniPred's transfer is fundamentally different: it operates through shared lexical semantics. The model learns associations between specific parameter name tokens (e.g., learning_rate, dropout, batch_norm) and specific functional behaviors (log-scale sensitivity, regularization effects, scale-dependent behavior), and these associations transfer across tasks even when the full parameter spaces differ. The evidence is sharp: when parameter names are hashed to random strings (Table 4), multi-task performance degrades from 0.03 to 0.46 on BBOB (50K studies) and from 0.15 to 0.43 on AutoML (540K studies). The model is not just learning from shared numerical patterns — it relies on textual cues to organize and transfer its knowledge. This is a qualitatively new kind of transfer that is only possible because the model processes parameters as meaningful text rather than as opaque tensor indices.

The paper also redirects research attention in uncertainty quantification for regression. Traditional methods require explicit design choices to express uncertainty (GP kernel structure, ensembling, heteroscedastic loss functions). OmniPred acquires uncertainty expression as an emergent property of temperature sampling from a language model trained with cross-entropy — the spread of sampled predictions naturally reflects model confidence without any uncertainty-specific training. Table 7 shows that sample standard deviation achieves higher rank correlation with actual error than GP predictive variance on both AutoML (0.487 vs. 0.230 Kendall-Tau) and BBOB (0.366 vs. 0.068). Figure 9 demonstrates that the model can express nonparametric multimodal uncertainty (sign-flipped BBOB objectives produce bimodal prediction samples) without pre-specifying the number of mixture components. This suggests that language model regression is a natural fit for applications requiring calibrated uncertainty — Bayesian optimization, active learning, safety-critical prediction — without the uncertainty engineering burden that GPs and ensembles require.

Finally, the paper identifies a sweet spot for multi-task transfer (Figure 10) that provides actionable guidance rather than vague "more data helps" claims. Multi-task gains are maximized when a study has approximately 50 training trials and diminish as trial count increases toward 200+. This non-monotonic pattern (gains are smaller at very low trial counts because the model lacks sufficient signal to identify the correct task structure) provides a principled deployment heuristic: if you have fewer than ~50 trials and access to a multi-task pretraining corpus, use OmniPred; if you have hundreds of trials for your specific task, the choice between OmniPred and a traditional regressor is less critical for accuracy (though OmniPred still offers the universality benefits of Table 1).

Follow-Up Research This Work Enables

1. Joint optimization of x-tokenization and y-tokenization. The paper deliberately uses different tokenization strategies for inputs (standard T5 SentencePiece, which splits numbers arbitrarily — e.g., 1234.5 → {'12', '3', '4.5'}) and outputs (custom digit-by-digit float tokens with constrained decoding). Section 7 explicitly flags this as potentially suboptimal: "the corresponding tokens may not exactly be digit-by-digit... One may instead potentially reuse the custom tokenization for y-values" on the input side. A natural follow-up would apply the same custom digit-by-digit tokenization to x-values (e.g., representing batch_size:128 as batch_size:<+><1><2><8><E0>) and measure whether this improves prediction precision, particularly on tasks where small changes in continuous parameters matter. The key metric would be study error on domains with sensitive continuous parameters (BBOB, Bid Simulation, Init2Winit) comparing standard vs. custom x-tokenization, with the hypothesis that digit-level attention to input numbers enables finer-grained numerical reasoning. The experiment is straightforward: modify the input serialization to use the same custom float tokens for all numeric parameter values, retrain the multi-task model, and compare per-domain errors against the original tokenization. A negative result (no improvement from custom x-tokenization) would be equally informative — it would suggest that the model's self-attention successfully reconstructs numerical magnitudes from fragmented subword tokens, making x-tokenization engineering unnecessary.

2. Systematic comparison against a learned-featurizer multi-task baseline. The paper claims traditional regressors cannot multi-task across heterogeneous input spaces, but this claim has not been tested against a reasonable counterfactual: embed all parameter configurations into a shared fixed-dimensional representation using a learned featurizer (e.g., a small transformer encoder that maps arbitrary key-value strings to a fixed-length embedding), then train a multi-task GP or MLP on these embeddings. This would create a fair comparison between OmniPred's end-to-end text-to-text approach and a hybrid approach that uses text for featurization but traditional numerical methods for regression. The experiment would train a shared encoder (possibly the same size as OmniPred's encoder) followed by a GP or MLP head, all trained jointly on multi-task data, and compare per-domain study error against OmniPred's T5 decoder approach. This is newly tractable because OmniPred's textual representation scheme provides the featurization strategy — the only question is whether the regression head matters. A finding that a GP-on-embeddings baseline matches or exceeds OmniPred would suggest that the text-to-text framing is useful for representation learning but overkill for the actual regression, and that combining textual featurization with calibrated uncertainty from GPs is the better architecture.

3. Large-scale study error distribution characterization with per-study confidence. The paper reports only mean study errors aggregated across domains (Figure 6), but practitioners need to know worst-case behavior, not just averages. A critical follow-up would characterize the full distribution of per-study errors across multiple domains: histograms of study error for each domain, percentile curves (10th, 50th, 90th percentile error as a function of study size and space complexity), and identification of what characterizes the worst-performing studies (high dimensionality? conditional spaces? sparse data? unusual objective scales?). The experiment is purely analytical on existing data: for each domain, compute per-study error for all studies with sufficient test trials, then correlate error with measurable study properties (trial count, space size, parameter types, y-range, presence of conditionals). This would produce a practical risk assessment for deployment — practitioners could estimate expected error for a new study based on its characteristics. The paper's current aggregate numbers (mean error of 0.15 on AutoML) could mask a distribution where 20% of studies have error >0.40, making the system unreliable for those cases. Without this analysis, the "universal regressor" claim lacks the reliability characterization needed for real-world adoption.

4. Pretrained language encoder ablation to isolate text understanding benefits. The paper deliberately trains from scratch to avoid confounding factors from pretrained English knowledge, but this leaves open the question of whether pretrained language understanding would help or hurt. A direct extension would compare three conditions on the same multi-task data: (1) T5 trained from scratch (current approach), (2) T5 initialized from a pretrained English checkpoint (e.g., the public T5-base) with custom float token embeddings randomly initialized, and (3) T5 with pretrained encoder frozen, only decoder trained. The key metrics would be study error across domains (especially ones with semantically rich metadata like AutoML and Vertex AI) and data efficiency (how many multi-task studies are needed to reach a given error threshold). The hypothesis is that pretrained encoders would accelerate learning on domains where parameter names and metadata contain natural language (e.g., dropout, learning_rate, objective:'accuracy'), but might not help on domains with opaque naming (BBOB with x1, x2 parameter names). A negative result (frozen encoder performs poorly because pretrained English understanding actually interferes with numerical reasoning) would validate the paper's from-scratch choice and suggest that numerical regression requires fundamentally different representations than natural language.

5. Bayesian optimization loop integration with OmniPred as surrogate model. The paper demonstrates that OmniPred provides uncertainty estimates (Table 7, Figure 9) and handles dynamic, conditional input spaces, but never closes the loop by using it as a surrogate model in actual Bayesian optimization. A strong follow-up would replace the GP surrogate in a standard BO framework (e.g., the GP-Bandit implementation in OSS Vizier) with OmniPred, and measure optimization efficiency (number of trials to reach a target objective value, final best objective found) across a suite of optimization tasks. The key comparison would be against GP-based BO (current standard) and random search (lower bound). This experiment would test whether OmniPred's multi-task pretrained knowledge provides a warm-start advantage in early optimization (fewer trials needed to find good regions) and whether its uncertainty estimates are calibrated enough to guide exploration-exploitation tradeoffs. The paper's finding that uncertainty correlation is moderate (Kendall-Tau 0.487 on AutoML) but better than GP (0.230) suggests OmniPred might outperform GPs for exploration, but the calibration question (do 90% prediction intervals actually contain the true y 90% of the time?) would need to be measured first. A failure mode — OmniPred's uncertainty is overconfident in some regions, leading to premature convergence — would be as informative as success.

6. Cross-domain transfer characterization: which domains transfer to which. Table 5 shows negative transfer from BBOB to AutoML (0.45 vs. 0.20 for single-task) but positive transfer from RealWorldData to AutoML (0.15), but the mechanism is not analyzed. A systematic study would measure pairwise transfer between all seven domains: pretrain on domain A, finetune on domain B, measure error vs. single-task baseline on B. This would produce a transfer matrix showing which domains share useful structure. Hypotheses to test: (1) domains with shared parameter names transfer better (Protein Design, with opaque p00000000-style names, might not transfer to/from domains with semantically named parameters like dropout), (2) domains with similar y-distributions transfer better (bounded accuracy metrics vs. unbounded loss metrics), (3) domains with similar space complexity transfer better (flat spaces vs. conditional spaces). This experiment is straightforward given the infrastructure — just requires running finetuning experiments for all domain pairs — and would provide actionable guidance for practitioners building pretraining corpora. The finding that BBOB actively hurts AutoML is particularly important to characterize: it shows that not all regression data is helpful, and understanding the boundaries would prevent practitioners from poisoning their models with irrelevant pretraining data.

Practical Applications and Downstream Use Cases

Unified regression service for blackbox optimization platforms. The most direct application is deploying OmniPred as the default surrogate model in a hosted blackbox optimization service like Google Vizier. Currently, such services must configure task-specific regressors (GPs, random forests) per optimization study, with separate featurization, hyperparameter tuning, and model fitting for each. OmniPred's single-model design means one pretrained checkpoint can serve as the regressor for all incoming optimization tasks — no per-study configuration, no featurization engineering, and no model selection. When a user starts a new study, the system queries OmniPred (with optional online finetuning on the study's initial trials) to provide predictions and uncertainty estimates for candidate parameter configurations. The paper's results suggest that on AutoML-like domains (conditional spaces, moderate trial counts), the pretrained OmniPred would match or exceed the current best single-task regressor after finetuning on ~50 trials (Figure 10: peak multi-task gain at ~50 trials; Table 5: pretrained+finetuned achieves 0.15, matching domain-specific pretraining). For a typical optimization run of 100–200 trials, the regressor would be especially valuable in the critical early-to-middle phase where trial data is sparse and multi-task knowledge provides maximum benefit (Figure 10, peak at ~50 trials).

Cold-start prediction for new experimental design tasks. Organizations running high-cost experiments — protein engineering, hardware design, pharmaceutical screening — face a cold-start problem: early experiments are essentially blind because there's insufficient data to fit a reliable regressor. OmniPred offers a solution: a model pretrained on thousands of prior optimization studies (even from different domains) can provide reasonable initial predictions before any task-specific data is collected, then rapidly adapt via finetuning as trials accumulate. The paper's unseen-task finetuning results (Figure 7: pretrain+finetune achieves 0.21 vs. 0.25 for best baseline MLP on 1000 new studies from new users) demonstrate that even when the pretraining corpus contains no data from the new user or exact task type, the transferred knowledge provides a measurable advantage. The specific deployment scenario: a protein engineering team starts a new optimization campaign. Before running any experiments, OmniPred (pretrained on 54K prior protein design studies from Vizier) provides initial predictions. After 20–30 trials, the model is finetuned on the campaign's specific data, achieving error competitive with a GP trained on 100+ trials. This compresses the exploration phase — the team reaches good designs in fewer expensive experiments. The paper's Protein Domain results (Figure 6: multi-task LM achieves ~0.12 vs. MLP at ~0.19) support this scenario, though the cold-start phase specifically (0 trials) is not evaluated.

Automated featurization for ad-hoc regression tasks. Data scientists and analysts frequently encounter ad-hoc regression problems where building a proper featurization pipeline is more work than the analysis itself — predicting server latency from configuration parameters, forecasting sales from marketing spend allocations, estimating model performance from hyperparameters. OmniPred reduces the setup cost to zero: the analyst writes parameter configurations as key-value strings (or uses an existing logging format), and the model produces predictions with uncertainty estimates. The paper's demonstration that a single model handles 8 drastically different input spaces (Figure 4) — from 4-parameter CIFAR10 tuning to 60-category protein design to 31-parameter LLM latency optimization — suggests this "universal regressor as a service" model is technically feasible with modest infrastructure (200M parameters, single GPU inference). The practical value is not necessarily improved accuracy over a carefully tuned XGBoost model but rather the elimination of the human effort required to build, validate, and maintain that model. For organizations with hundreds of different regression tasks across teams, a single OmniPred deployment that handles all of them with acceptable accuracy (Figure 6: multi-task error ranges from ~0.01 to ~0.32 depending on domain) could replace dozens of task-specific modeling efforts.