Language Model Embeddings Can Be Sufficient for Bayesian Optimization

ArXiv: 2410.10190

🎯 Pitch

A single frozen LLM embedder, paired with a generic Transformer regressor, matches Google Vizier’s specialized Gaussian Process across synthetic, combinatorial, and hyperparameter optimization tasks—eliminating per-domain feature engineering. The same T5-XL model and regressor weights work universally, turning any black-box search space into a string-embedding problem.

1. Executive Summary

This paper introduces the embed-then-regress framework, a simple recipe that uses frozen LLM embeddings to convert arbitrary string representations of optimization candidates into fixed-length feature vectors for in-context regression in Bayesian Optimization—enabling a single general-purpose regressor to operate across diverse search spaces without domain-specific modeling. The authors pretrain a Transformer Neural Process regressor on large-scale offline evaluation data from varied tasks and pair it with standard explore-exploit acquisition techniques, evaluating on synthetic (BBOB), combinatorial (Travelling Salesman, Quadratic Assignment, N-Queens), and hyperparameter optimization benchmarks against Google Vizier's industry-grade Gaussian Process Bandit algorithm. Across these settings, embed-then-regress achieves optimization performance comparable to the specialized GP-based method—matching Vizier's median log-efficiency score on hyperparameter optimization surrogates and significantly outperforming it on several BBOB functions—while using the same frozen T5-XL embedder and regressor weights for all domains. The results establish that LLM embeddings possess sufficient structure for competitive black-box optimization across tabular, permutation, and choice-based search spaces, though the approach is bounded by the embedder's ability to produce Lipschitz-continuous representations of the underlying parameters.

2. Context and Motivation

The Core Problem: Regression Methods Are Shackled to Task-Specific Input Formats

The fundamental problem this paper addresses is a structural limitation in how regression models represent their inputs in black-box optimization. In a typical Bayesian Optimization (BO) loop, a regressor predicts the performance of unevaluated candidates based on a history of observed (input, objective-value) pairs. The quality of these predictions — and therefore the efficiency of the optimization — depends critically on how the regressor represents the candidates it receives. Yet despite decades of progress in regression methods, most regressors are inherently tied to fixed-dimensional, structured, tabular input features that must be defined separately for each search space.

This is not a minor implementation detail. It means that a Gaussian Process regressor configured for hyperparameter tuning (say, a 7-dimensional continuous space with a few categorical parameters) cannot be reused for a combinatorial problem like the Travelling Salesman Problem, where inputs are permutations of city indices. A random forest trained to predict objective values over a specific set of hyperparameters cannot accept candidates from a different search space with different parameter names, types, or dimensionalities. Every new optimization task requires a new regressor specification, including defining the kernel, the feature representation, and often the acquisition optimizer. This brittleness stands in stark contrast to the vision of general-purpose optimization systems that can be deployed across arbitrary black-box functions without per-task engineering.

The paper frames this as a representation gap: existing regressors are powerful within their predefined input format but fundamentally inflexible across formats. The authors situate this gap within the broader challenge of building universal optimizers — systems that can ingest a description of a search space and an objective function and efficiently locate good solutions, without human intervention to design domain-specific components. The regression model is the bottleneck: if the regressor cannot consume arbitrary input formats, the optimizer cannot be universal.

Why This Matters: Practical and Conceptual Stakes

The practical significance of this problem is immediate. Industry-grade optimization services — Google Vizier (Golovin et al., 2017), Optuna (Akiba et al., 2019), Ax/Botorch (Balandat et al., 2020a) — are deployed across enormously diverse use cases: tuning deep learning hyperparameters, optimizing hardware configurations, selecting experimental conditions in materials science, configuring production system parameters. Each of these domains presents different search spaces with different structures (continuous, discrete, categorical, combinatorial, nested, conditional). In current practice, each domain requires its own regressor configuration, often with manually specified kernels, feature engineering, or embedding strategies. The engineering cost of maintaining and adapting these regressors across domains is substantial, and the inability to transfer learned regression knowledge across tasks means every new optimization problem starts from scratch — a form of cold-start inefficiency that pretraining could potentially eliminate.

There is also a deeper conceptual stake. The field of Bayesian Optimization has long been dominated by Gaussian Processes, which encode assumptions about function behavior — smoothness, periodicity, additivity — through kernel functions defined over the input space. These assumptions are both a strength (they provide principled uncertainty estimates with few data) and a weakness (they lock the model into a specific input geometry). The rise of learned regressors based on Transformers and neural processes (Garg et al., 2022; Müller et al., 2022; Nguyen and Grover, 2022) demonstrated that purely learned models can match or exceed GPs on prediction quality when pretrained on relevant data, and that they can perform implicit Bayesian inference without explicit kernel design. But these learned regressors inherited the same fundamental constraint: they operate on fixed-dimensional feature vectors. The paper thus identifies a crucial missing piece: a learned regressor that is simultaneously flexible in its input representation and capable of meta-learning from diverse offline data. Closing this gap would mean that a single pretrained model could serve as the regression backbone for optimization across arbitrary domains, amortizing training cost and enabling transfer of learned priors about function landscapes.

Where Existing Approaches Fall Short

The paper organizes prior work along a spectrum of how they handle the input representation and sequence length constraints of Transformers, which are the dominant learned regressor architecture. This taxonomy (Section 2) reveals that no existing approach simultaneously satisfies all requirements for a universal optimization regressor: pretrainability over offline evaluations, flexible string-based input representation, long-range in-context regression over many trials, and highly precise numeric predictions with uncertainty quantification across diverse objective scales.

Gaussian Processes (the dominant paradigm). GPs remain the workhorse of Bayesian Optimization due to their sample efficiency, principled uncertainty estimates, and well-understood behavior. However, they are fundamentally tied to fixed input dimensions and kernel definitions. A GP configured for a 5-dimensional continuous hyperparameter space cannot accept a permutation of length 20, nor can it gracefully handle search spaces whose dimensionality varies across tasks. Recent work has attempted to improve GP flexibility by learning kernel hyperparameters from offline data (Fan et al., 2024; Wang et al., 2024) or by manually designing embeddings for specific structured spaces like permutations (Deshwal et al., 2022, 2023) and graphs (Ru et al., 2021). But these approaches remain domain-specific: the permutation kernel designed by Deshwal et al. (2022) requires custom acquisition optimizers using semi-definite programming, making it difficult to reproduce and impossible to reuse across unrelated search space types. The kernel itself encodes assumptions about the geometry of permutations — it cannot be repurposed for set-based choices or mixed continuous-categorical spaces. Each new domain demands new feature engineering, new kernel design, and potentially new acquisition optimization machinery.

Raw Transformers as in-context regressors. A parallel line of work (Garg et al., 2022; Nguyen and Grover, 2022; Müller et al., 2022) demonstrated that Transformer models can function as in-context regressors: given a sequence of (input, output) pairs and a query input, they can predict the query output without parameter updates, effectively performing meta-learned regression. These "Transformer Neural Processes" or "Prior-Fitted Networks" offer several advantages over GPs — they are fully learned, can capture complex non-stationary function behavior, and benefit from pretraining on diverse function data. Müller et al. (2023) and Nguyen et al. (2023) subsequently applied these models to Bayesian Optimization with promising results. But the input representation problem remains unsolved: these Transformers still require fixed-dimensional feature vectors as inputs. A Transformer trained on 5-dimensional BBOB functions cannot process a permutation of length 12 for the Travelling Salesman Problem, because the input dimensionality is baked into the architecture. This limits these models to the same tabular input format as GPs, merely replacing the kernel with learned attention — a significant advance in modeling flexibility but not in input flexibility.

Token-based representations with custom tokenization. To expand beyond tabular inputs, some works have represented optimization candidates as sequences of tokens rather than fixed feature vectors. Chen et al. (2022) used custom tokenizations to represent hyperparameter configurations as short token sequences, minimizing the per-trial token length so that many trials could fit within the Transformer's context window. This approach enables meta-learning across different hyperparameter search spaces and achieved strong results on hyperparameter optimization benchmarks. But the approach is fragile: the custom tokenization is designed for flat hyperparameter spaces and does not extend naturally to combinatorial structures like permutations, graphs, or sets. The tokenization scheme encodes a specific assumption about the search space structure that limits its generality. The paper notes that this approach "lacks flexibility in utilizing arbitrary forms of data" (Section 2).

Text-to-text in-context regression with LLMs. A more recent direction uses large language models directly as in-context regressors by formatting the optimization history as natural language text and querying a chat-based LLM for predictions. Liu et al. (2024) and Vacareanu et al. (2024) demonstrated that models like ChatGPT and Gemini exhibit emergent regression capabilities when presented with (input, output) pairs in text form. The string-based input representation is maximally flexible — any search space can be described in text — and this approach requires no pretraining or model modification. However, this flexibility comes at a steep cost. The sequence length constraint (Equation 1 in the paper) is punishing: the total context length equals the number of trials multiplied by the average token length per trial. When trials are represented in natural language (e.g., "The hyperparameters were learning_rate=0.001, batch_size=32, ... and the validation accuracy was 0.873"), each trial consumes dozens to hundreds of tokens. With current context windows, this limits the number of in-context trials to perhaps a few dozen — far fewer than the hundreds of trials needed for effective Bayesian Optimization in moderate-dimensional spaces. Furthermore, these models are used off-the-shelf; they cannot be pretrained on domain-specific offline evaluation data to improve their regression accuracy or calibration. The paper identifies this as a critical limitation: "such methods lack the ability to pretrain over large amounts of offline evaluations" (Section 2).

Single-trial regression with inference-time fine-tuning. A fourth paradigm avoids in-context regression entirely. Song et al. (2024a) and Akhauri et al. (2025) represent each trial as arbitrary text, use an LLM to embed the text, and train a regression head on top — but they use only a single trial at a time, avoiding context-length constraints. To incorporate the optimization history, they must resort to inference-time fine-tuning: as new evaluations arrive, the model is fine-tuned on the accumulating data before making predictions for the next candidate. The paper characterizes this as "tedious" and computationally expensive compared to in-context methods, where the history is simply presented as input tokens without weight updates. Fine-tuning at every step of the optimization loop introduces latency (gradient computation, weight updates) and hyperparameter sensitivity (learning rate, number of fine-tuning steps) that in-context methods avoid.

Embedding-based regression for specific domains. The approach closest to this paper's method uses language model embeddings as fixed feature vectors for regression, but only in narrow domains. Kristiadi et al. (2024) and Ranković and Schwaller (2023) applied LLM embeddings for Bayesian Optimization over chemical reactions, where the "string" is a chemical formula (e.g., SMILES strings). Hu et al. (2024) used embeddings for prompt optimization, where the string is a natural language prompt. These works demonstrated the viability of embedding-based regression within their specific domains, but none evaluated the approach in the most competitive and widely studied setting of standard black-box optimization over tabular-like search spaces, where GP-based methods are highly tuned and well-understood. The paper explicitly positions itself in this gap: "no work has assessed their use in the most competitive and widely studied field of standard black-box optimization over tabular-like search spaces, despite evidence (Tang et al., 2025) demonstrating LLM embeddings to possess promising traits such as Lipschitz continuity over tabular features" (Section 2).

The Missing Combination: A Requirements Analysis

The paper synthesizes this landscape by enumerating five requirements for a regressor suitable for general-purpose Bayesian Optimization (Section 2, final paragraph):

1. Pretrainable over offline evaluations to enable meta-learning of function priors across tasks. This rules out pure off-the-shelf LLM approaches (Liu et al., 2024; Vacareanu et al., 2024), which cannot benefit from domain-specific offline data.

2. Flexible representation of inputs with raw strings for application across multiple domains without per-task feature engineering. This rules out traditional GPs, raw Transformer regressors, and custom-tokenized approaches (Chen et al., 2022), all of which require fixed-dimensional or specially formatted inputs.

3. Long-range in-context regression using many previous evaluations. This rules out single-trial methods (Song et al., 2024a; Akhauri et al., 2025) that require inference-time fine-tuning to incorporate history, and natural-language in-context methods (Liu et al., 2024; Vacareanu et al., 2024) whose context windows are consumed by lengthy text representations.

4. Precise numeric predictions and uncertainty quantification over diverse objective scales. This rules out approaches that produce only coarse categorical scores (e.g., Process Reward Models producing {-1, 0, 1} ratings; Lightman et al., 2024), since Bayesian Optimization requires fine-grained discrimination between candidates and calibrated uncertainty for exploration-exploitation tradeoffs.

5. Computational efficiency during inference, since the regressor may be called thousands of times by the acquisition optimizer per candidate proposal. This imposes practical constraints on model size and latency that large-scale LLMs used as direct regressors may violate.

The paper's central argument is that the embed-then-regress architecture — a frozen LLM embedder converting arbitrary strings to fixed-length vectors, paired with a pretrained Transformer Neural Process regressor performing in-context regression over those vectors — satisfies all five requirements simultaneously. The embedder handles #2 (string generality), the Neural Process handles #1 (pretrainability) and #3 (in-context regression), the Gaussian output head handles #4 (precise numeric predictions with uncertainty), and the relatively small embedder and regressor sizes handle #5 (inference efficiency). The key empirical question — which the rest of the paper addresses — is whether this architecture achieves competitive optimization performance against state-of-the-art GP-based methods, or whether the embedding compression and domain-generality sacrifices too much in prediction accuracy.

How This Paper Positions Itself

The paper explicitly positions itself not as proposing a new regression architecture, but as validating a simple design principle: that frozen LLM embeddings, when paired with standard in-context regression models, are already sufficient for competitive Bayesian Optimization across diverse search spaces. The contribution is primarily empirical: demonstrating that this straightforward combination works, and works well enough to match industry-grade GP systems, despite using the same model weights across all domains.

This is an important nuance. The authors are not claiming that their specific choices — T5-XL embedder, Transformer Neural Process regressor, UCB acquisition with evolutionary search — are optimal or novel individually. Each component is drawn from existing literature. Rather, the contribution is the integration and validation of these components into a system that achieves what prior work could not: a single pretrained regressor that competes with specialized GP-based methods across synthetic, combinatorial, and hyperparameter optimization tasks without per-domain adaptation.

The paper also positions itself as a foundational step toward more ambitious goals. Section 5 explicitly calls out the vision of "a unified in-context regression model broadly over multiple different domains including prompt optimization and code search, in order to obtain a 'universal' in-context regressor which can speed up search over evolutionary algorithms." The current work establishes the baseline viability of string embeddings for optimization; future work can refine the embedder, the regression architecture, and the pretraining data mixture. The paper also notes connections to LLM reasoning, suggesting that embedding-based regression might eventually serve as reward models for tree-search-based approaches (Yao et al., 2023) in stateful language modeling environments — extending beyond the stateless black-box optimization setting studied here.

A critical theoretical anchor for the approach comes from prior work by Tang et al. (2025), which the paper cites as demonstrating that LLM embeddings possess Lipschitz continuity over tabular features. This property is essential: if small changes in the underlying parameters produce small changes in the embedding vectors, then the embedding space preserves the geometric structure of the original search space, and a regressor that assumes smoothness (as the Transformer Neural Process implicitly does) can make meaningful predictions. Without this continuity property, the embedding would scramble the input geometry, and no amount of pretraining would recover competitive regression performance. The paper's positive results can thus be understood as empirical confirmation that current LLM embeddings indeed preserve sufficient input structure for optimization purposes.

Finally, the paper emphasizes rigor in comparison methodology. For synthetic and hyperparameter optimization tasks, they use Google Vizier's GP-Bandit algorithm (Song et al., 2024c) as the baseline, keeping all other components of the optimization pipeline identical — the same acquisition optimizer (Firefly), the same UCB acquisition function, the same budget and initialization. This isolates the effect of the regressor, avoiding the confounding that arises when comparing entire optimization systems with different acquisition functions, optimizers, or initialization strategies. As the paper notes, "it is well known that other components in the Bayesian Optimization pipeline (e.g. choice of acquisition and acquisition optimizer) also strongly affect performance and can lead to confounding factors" (Section 4.1). By swapping only the regressor while holding the rest of the Vizier pipeline constant, the comparison directly tests the central hypothesis: that LLM embeddings + a pretrained Transformer regressor can replace a GP without degrading optimization performance.

3. Technical Approach

3.1 Reader Orientation

The system being built is a general-purpose regression model for Bayesian Optimization that can predict the performance of unevaluated candidates in any search space — whether continuous hyperparameters, permutations, or categorical choices — using a single set of pretrained weights and no per-task feature engineering. The core problem it solves is the input representation bottleneck: by converting arbitrary search space candidates into strings, embedding those strings with a frozen language model, and feeding the resulting fixed-length vectors into a standard in-context Transformer regressor, the system achieves competitive optimization performance against specialized Gaussian Process methods while maintaining the flexibility to operate across entirely different search space types without modification.

3.2 Big-Picture Architecture (Diagram in Words)

The system consists of five major components connected in a pipeline:

1. String Serializer — converts a search space candidate $x \in \mathcal{X}$ into a standardized string representation (e.g., JSON for tabular parameters, index lists for permutations). This is the only component that varies across search space types, and even this variation is minimal (different JSON schemas).

2. Frozen Language Model Embedder (T5-XL Encoder) — takes the string representation, tokenizes it, runs a forward pass through the pretrained encoder, average-pools across the token dimension, and outputs a single fixed-length vector $\bar{x} \in \mathbb{R}^d$. This embedder is never fine-tuned — its weights are frozen from pretraining on general text corpora.

3. Trainable Projection Layers — apply a learned linear transformation to the raw embedding $\bar{x}$ to produce the final candidate feature vector $x \in \mathbb{R}^d$ used by the regressor. A separate projection maps scalar objective values $y \in \mathbb{R}$ into the same $d$-dimensional space via a trainable nonlinear mapping, producing $\tilde{y} \in \mathbb{R}^d$.

4. In-Context Transformer Regressor (Neural Process) — an 8-layer Transformer with 16 attention heads and 1024-dimensional features that takes as input a sequence of trials $(x_1 \oplus \tilde{y}_1), \ldots, (x_t \oplus \tilde{y}_t)$ where $\oplus$ denotes concatenation, plus a query trial $(x_{t+1} \oplus \mathbf{0})$ where $\mathbf{0}$ is a dummy value vector. The Transformer processes this sequence with a custom attention mask that allows the query to attend to all history trials but prevents history trials from attending to the query. It outputs a Gaussian distribution $\mathcal{N}(\mu_{t+1}(x), \sigma^2_{t+1}(x))$ via dedicated mean and standard deviation prediction heads.

5. Acquisition Function + Optimizer — during inference, the regressor's mean and uncertainty estimates form a UCB acquisition $a_{t+1}(x) = \mu_{t+1}(x) + \sqrt{\beta} \cdot \sigma_{t+1}(x)$, which is maximized by a zeroth-order optimizer (Firefly evolutionary algorithm for continuous/tabular spaces, Regularized Evolution for combinatorial spaces) to propose the next candidate for evaluation.

Information flows sequentially: candidate → string → embedding → feature vector → Transformer regressor (conditioned on history) → Gaussian prediction → acquisition score → optimizer → next candidate.

3.3 Roadmap for the Deep Dive

• First, the formal problem setting and the string-to-embedding pipeline — because the entire architecture hinges on how string representations are constructed and how embeddings preserve input geometry. Understanding this step explains why a single regressor can work across BBOB, Travelling Salesman, and hyperparameter tuning.

• Second, the in-context Transformer regressor architecture — because this is the learned component that consumes embeddings, processes optimization histories, and produces predictions. This includes the input format, the attention mechanism, the output distribution, and critical stabilization techniques (parallel predictions, y-normalization, metadata encoding).

• Third, the pretraining procedure — because the regressor's ability to generalize across domains comes from meta-learning over diverse offline evaluation data. This covers the task sampling strategy, the loss function, the training hyperparameters, and the data augmentation choices that enable transfer to unseen search spaces.

• Fourth, the inference-time acquisition loop — because competitive optimization requires not just accurate predictions but effective exploration-exploitation balance. This covers the UCB acquisition function, the acquisition optimizer choices for different search space types, and the specific budgets and hyperparameters used.

• Fifth, the full model specifications (embedder size, regressor size, training hyperparameters, inference budgets) — because these concrete choices determine the computational cost and practical deployability that the paper claims as a strength.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical validation paper whose core idea is that frozen LLM embeddings of string-serialized optimization candidates provide sufficiently structured feature representations to enable a single pretrained Transformer Neural Process to serve as a competitive general-purpose regressor for Bayesian Optimization across diverse search spaces, without requiring per-task kernel design, feature engineering, or model fine-tuning.

The Formal Problem Setting and the Embedding Pipeline

Every Bayesian Optimization problem is defined by an objective function $f: \mathcal{X} \to \mathbb{R}$ over a search space $\mathcal{X}$. The goal is to find $x^* = \arg\max_{x \in \mathcal{X}} f(x)$ using as few evaluations of $f$ as possible. At step $t$, the optimizer has observed a history of trials $\{x_s, y_s\}_{s=1}^t$ where $y_s = f(x_s)$, and must propose the next candidate $x_{t+1}$. The regressor's job is to estimate $p(y \mid x, \{x_s, y_s\}_{s=1}^t)$ — the predictive distribution over $f(x)$ given the history.

The paper's central design decision is to sever the regressor from the search space geometry by interposing a frozen embedding function between the search space and the regression model. Rather than feeding $x$ directly into the regressor (which would require the regressor to understand the structure of $\mathcal{X}$), the system first maps $x$ to a string $s(x)$, then maps $s(x)$ to an embedding vector $\bar{x} = \phi(s(x))$ using a pretrained language model, then (optionally) applies a learned projection $x = g(\bar{x})$ to produce the final feature vector consumed by the regressor. The regressor thus never sees the original input — it only sees fixed-length vectors in $\mathbb{R}^d$.

String representation design. The paper uses JSON as the universal string format, with the specific schema varying minimally across search space types. For tabular search spaces (continuous, discrete, categorical parameters), each candidate is serialized as a JSON dictionary mapping parameter names to values:

{"p0": 0.3, "p1": "category_1"}

For permutation spaces of size $n$, each candidate is serialized as a JSON dictionary mapping position indices to elements:

{"[0]": 2, "[1]": 0, "[2]": 3, "[3]": 1}

For choice spaces (selecting $k$ elements from $n$), the JSON dictionary lists the selected indices:

{"[0]": 1, "[1]": 3}

The crucial property is that the regressor never needs to know the search space structure. The same JSON parser handles all cases, and the language model embedder processes the resulting string identically regardless of whether it describes hyperparameters, permutations, or choices. The paper provides additional examples in Appendix C, including optional metadata $m$ that can be prepended to distinguish between task types (e.g., task:"Permutation" size:4). However, the main experiments do not use metadata, demonstrating that the embedding alone carries sufficient information for the regressor to infer the task type implicitly.

Embedding computation. The paper uses the standard definition of a language model embedding from the T5 family (Raffel et al., 2020). Given a tokenized string of length $L$, the T5-XL encoder (1 billion parameters) produces a sequence of hidden states $H \in \mathbb{R}^{L \times d}$ where $d = 4096$ for the XL variant. The embedding is then computed as average pooling across the token dimension: $\bar{x} = \frac{1}{L} \sum_{\ell=1}^L H_{\ell} \in \mathbb{R}^d$. This operation collapses the variable-length string representation into a fixed-length vector, discarding positional information.

The paper notes that this choice (average pooling) is "predefined rather than learned" and flags it as a potential target for architectural improvement in future work (Section 5). Alternative aggregation methods — attention pooling, using the first token, learning a weighted combination — might better preserve task-relevant information. The average pooling choice is conservative and simple, making the positive results more surprising: even this minimal aggregation preserves sufficient structure for competitive regression.

The Lipschitz continuity property. A critical theoretical justification for why this approach works comes from prior work by Tang et al. (2025), which the paper cites as demonstrating that LLM embeddings are Lipschitz continuous over tabular features. Formally, there exists some constant $K$ such that $\|\phi(s(x_1)) - \phi(s(x_2))\|_2 \leq K \|x_1 - x_2\|$ for nearby parameter values $x_1, x_2$ in the original search space. This means that the embedding space preserves the local geometry of the search space: two candidates with similar parameter values will map to nearby embedding vectors, and thus a regressor that assumes smoothness in embedding space (as the Transformer Neural Process implicitly does through its attention mechanisms) can meaningfully interpolate between observed trials. Without this property, the embedding would be an arbitrary scrambling of the input, and no amount of pretraining could recover meaningful regression.

The Lipschitz continuity property is empirically surprising because T5-XL was pretrained on natural language text (largely English sentences from the C4 corpus), not on JSON representations of optimization parameters. The fact that its embeddings preserve the metric structure of numeric parameter spaces suggests that the model has learned general representational capacities (e.g., encoding numeric magnitude, ordering, and similarity) that transfer to entirely new input formats. The paper's ablation in Figure 7 — showing that larger T5 encoders produce monotonically better regression predictions despite being pretrained on the same text data — supports this interpretation: larger models learn richer structural representations that happen to be useful for numeric regression as a byproduct.

The trainable projection. After embedding, the raw $d$-dimensional vector $\bar{x}$ passes through a learned linear projection (or potentially a small MLP — the paper is not explicit about the exact architecture but states that "the embedding is projected using a trainable projection layer") to produce the final trial feature vector $x \in \mathbb{R}^{d_{\text{model}}}$ where $d_{\text{model}} = 1024$ is the hidden dimension of the ICL Transformer. This projection serves two purposes: it adapts the embedder's output dimension (4096 for T5-XL) to the regressor's input dimension (1024), and it allows the regressor to learn which dimensions of the raw embedding are relevant for regression versus which encode irrelevant linguistic information. The projection weights are trained jointly with the regressor during pretraining.

Why strings over raw features? The paper's central claim is not that strings are inherently better representations than raw feature vectors — indeed, for tabular search spaces where the parameters are already structured as fixed-dimensional vectors, one could simply feed those vectors directly into the regressor (as prior work like Müller et al., 2023 does). The string approach is motivated by generality: the same embedding pipeline works for any search space that can be serialized as text, including esoteric spaces (permutations, graphs, sets, conditional structures, variable-length inputs) where defining a fixed-dimensional feature vector would require ad-hoc engineering. The paper's experiments on combinatorial optimization (Section 4.1) demonstrate this generality: the same regressor weights that predict BBOB function values also predict Travelling Salesman tour lengths and Quadratic Assignment costs, despite these spaces having fundamentally different mathematical structures.

The In-Context Transformer Regressor (Neural Process)

The regression model is a Transformer Neural Process (TNP) as introduced by Nguyen and Grover (2022), adapted to consume embedding-derived feature vectors rather than raw inputs. A Neural Process is a meta-learning architecture that performs in-context regression: given a set of observed (input, output) pairs (the "context") and a query input, it predicts the distribution over the query output without any gradient updates, by processing the entire context+query sequence through a Transformer.

Input format. Each trial in the history is represented as a single vector $t_s = x_s \oplus \tilde{y}_s \in \mathbb{R}^{2d_{\text{model}}}$, where $x_s = g(\phi(s(x_s)))$ is the projected embedding of the candidate and $\tilde{y}_s = h(y_s)$ is a trainable nonlinear projection of the scalar objective value into $\mathbb{R}^{d_{\text{model}}}$. The concatenation $\oplus$ simply stacks the two $d_{\text{model}}$-dimensional vectors into a $2d_{\text{model}}$-dimensional vector. For the query point $x_{t+1}$, the objective value is unknown, so a dummy value vector $\mathbf{0} \in \mathbb{R}^{d_{\text{model}}}$ is concatenated instead: $t_{t+1} = x_{t+1} \oplus \mathbf{0}$.

The full input to the Transformer is a sequence of length $t + 1$:

$T = [t_1, t_2, \ldots, t_t, t_{t+1}]$

where $t$ is the number of observed trials. The paper uses a maximum sequence length of $T_{\text{max}} \geq 100$ during training, with the number of history trials $t'$ sampled uniformly from $[10, T_{\text{max}} - 10]$, and the remaining $T_{\text{max}} - t'$ positions filled with dummy tokens or used for parallel target predictions (see below).

Attention mechanism with parallel predictions. A standard Transformer would allow every token to attend to every other token. However, this would be problematic for regression: the query point $t_{t+1}$ should be able to attend to all history trials (to learn from observed data) but should not influence the history representations (the history is fixed, the query is just requesting a prediction). Moreover, during training, the model needs to make predictions at multiple held-out points simultaneously for computational efficiency.

The paper implements a custom attention mask to handle these requirements. When predicting over a set of $k$ target points $x_{t+1}, \ldots, x_{t+k}$, the model forms an input sequence of length $t + k$ where the first $t$ positions are history trials and the last $k$ positions are query trials (each with dummy objective values). The attention mask has shape $(t + k) \times t$:

• History tokens attend to all history tokens (standard self-attention within the first $t$ positions), allowing the model to build rich representations of observed trials by comparing them against each other.
• Query tokens attend to all history tokens (cross-attention from positions $t+1, \ldots, t+k$ to positions $1, \ldots, t$), allowing each query to extract relevant information from the full history.
• No tokens attend to query tokens (the query positions are masked out as attention targets). This prevents the query's dummy objective value from contaminating the history representations and ensures that predictions for different query points are conditionally independent given the history.

This parallel prediction design enables efficient training: the model processes all $k$ query points in a single forward pass (rather than $k$ separate passes), and the loss is computed as the sum of individual prediction losses over all query points. During inference, the model typically predicts for one query point at a time (the candidate being scored by the acquisition function), though the paper notes that batched scoring is also possible when the acquisition optimizer proposes multiple candidates.

Output distribution. At the final layer of the Transformer, the representation at each query position is passed through two separate linear output heads:

• A mean head that predicts $\mu_{t+1}(x) \in \mathbb{R}$, the expected value of $f(x)$.
• A standard deviation head that predicts $\sigma_{t+1}(x) \in \mathbb{R}_{>0}$, the predictive uncertainty.

Together, these form a Gaussian predictive distribution:

$p(y \mid x, \{x_s, y_s\}_{s=1}^t) = \mathcal{N}\left(y \mid \mu_{t+1}(x), \sigma_{t+1}^2(x)\right)$

The Gaussian assumption is standard in Neural Process literature and allows closed-form computation of the negative log-likelihood loss during training. The paper does not explore alternative output distributions (e.g., Student-t for heavier tails, mixture models for multi-modal predictions), which could be beneficial for objectives with highly non-Gaussian behavior.

Transformer architecture. The paper uses an 8-layer Transformer with 16 attention heads and a hidden dimension of $d_{\text{model}} = 1024$, with feedforward projection outputs of 4096 (a standard 4× expansion). The total parameter count of the regressor is not explicitly stated but is modest by modern standards — the paper emphasizes that the maximum training budget is "approximately 16 GPUs for training and 1 GPU for inference, possible with most academic budgets" (Section 3.4). The inference cost per prediction scales as $\mathcal{O}(t)$ with the number of history trials due to the quadratic attention complexity, which the paper notes could be reduced using efficient Transformer variants (Tay et al., 2022) for linear scaling.

Why a Transformer over a Gaussian Process? The paper's design choice of a learned Transformer regressor over a GP is driven by the requirement for pretrainability across diverse task distributions. A GP requires specifying a kernel function that encodes prior assumptions about function smoothness, periodicity, and correlation structure. While a single GP with a universal kernel (e.g., RBF) could theoretically be applied to any task given a feature vector, its hyperparameters (length scales, output variance) would need to be optimized per-task, and its fixed kernel structure might be poorly suited to the specific geometry of embedding space. In contrast, the Transformer Neural Process learns its prior from data: by training on millions of offline trials from thousands of different objective functions, it implicitly learns what kinds of function behaviors are likely, how uncertainty should scale with distance in embedding space, and how to combine information from multiple observed trials. This meta-learned prior is what enables the model to make calibrated predictions on unseen functions without per-task adaptation.

Why a Neural Process over other learned regressors? The choice of a TNP specifically is motivated by the requirement for in-context regression with many trials. Alternative learned regressors might require inference-time fine-tuning (Song et al., 2024a) or might be limited to fixed context lengths that constrain the number of usable trials (Liu et al., 2024). The TNP's architecture is designed precisely for the few-shot regression setting: it processes variable-length histories and produces per-query predictions without any weight updates, making it suitable for the online optimization loop where new trials arrive sequentially and predictions must be generated quickly.

Stabilization Techniques

The paper describes several techniques that are explicitly marked as "optional but may stabilize training and prediction" (Section 3.2, end). However, given the complexity of training on diverse objective functions with widely varying scales, these techniques are likely essential for the reported performance.

Parallel predictions. As described above, this technique allows the model to predict over multiple query points simultaneously during training, reducing the number of forward passes and enabling larger effective batch sizes. The key implementation detail is the custom attention mask that prevents information leakage between query points. This is not merely a speed optimization — it also forces the model to learn representations that are useful for predicting at arbitrary locations, rather than overfitting to the specific query position used during sequential training.

y-normalization. The paper's objective functions span enormously different scales: a BBOF function might output values in $[-10^3, 10^3]$, while a hyperparameter tuning objective (e.g., validation accuracy) might be in $[0, 1]$, and a production metric might range over $[-10^7, 10^7]$. Training a single model to make precise predictions across such diverse scales would be extremely difficult without normalization.

The paper adopts the normalization procedure from Google Vizier (Song et al., 2024c), modified to handle incoming target values (i.e., new query points whose values are outside the observed range). The procedure consists of three sequential steps:

1. Standardization: Given the history of objective values $\{y_s\}_{s=1}^t$, compute the mean $\mu_Y$ and standard deviation $\sigma_Y$, then transform $y_s \leftarrow (y_s - \mu_Y) / \sigma_Y$ for all observed values. This ensures the history has zero mean and unit variance.

2. Outlier mitigation: Fit a normal distribution to the "bad half" of objectives — specifically, those $y_i$ where $y_i \leq y_{\text{median}}$. Use percentiles from this fitted normal as z-scores to transform extreme negative values, reducing the harmful effects of outliers that would otherwise distort the standardization. This step is particularly important for objectives with heavy-tailed distributions or occasional catastrophic failures (e.g., a training run that diverges and produces an extremely negative loss).

3. Range normalization and damping: Linearly scale all historical $y$-values to $[0, 1]$ via $y \leftarrow (y - y_{\text{min}}) / (y_{\text{max}} - y_{\text{min}})$, where $y_{\text{min}}$ and $y_{\text{max}}$ are the minimum and maximum observed values. For target (query) values that fall significantly outside the observed range, apply additional damping via a sigmoid or log transform to prevent extreme predictions. This ensures the model never needs to predict values far outside $[0, 1]$ during training, while still allowing it to handle genuinely out-of-distribution objective values at test time.

The paper notes that this normalization is applied per-task at both training and inference time. During training, each sampled task's history is normalized independently. During inference on a new optimization problem, the history is normalized using only the observed trials so far, and the normalization is updated as new observations arrive. This is standard practice in Bayesian Optimization and ensures that the model's predictions are invariant to the absolute scale of the objective function.

Encoding metadata. When multiple optimization tasks with different search spaces or objective function types are present in the training data (or when test-time tasks differ from training tasks), the regressor benefits from knowing which task it is currently operating on. The paper proposes encoding task metadata $m$ as an additional embedding vector that is concatenated to every trial representation: $t_s = x_s \oplus m \oplus \tilde{y}_s$. The metadata might include the search space type (e.g., "BBOB with 5 continuous parameters"), the task description (e.g., "permutation of size 12"), or any other information that helps the model distinguish between different function families.

However, the paper's main experiments do not use metadata (Appendix C shows example metadata formats but notes they were not used in the reported results). This is a deliberate choice: it demonstrates that the model can infer task-relevant structure purely from the observed (input, output) pairs, without explicit task identification. This is a stronger test of the embedding approach, since it means the model must simultaneously figure out what kind of function it is modeling AND make accurate predictions about it. The fact that this works suggests that the embeddings carry sufficient information about search space structure that the Transformer can implicitly cluster tasks by their embedding statistics.

Why not simply standardize to $\mathcal{N}(0, 1)$? A natural question is why the paper uses this three-step normalization rather than simply standardizing each task's objectives to zero mean and unit variance (z-score normalization). The answer lies in the outlier sensitivity of simple standardization. An optimization trajectory might encounter a few extremely bad evaluations (e.g., a model that fails to train entirely, producing a loss orders of magnitude worse than typical), and these outliers would dominate the mean and standard deviation, compressing all other values into a tiny range. The three-step procedure first standardizes, then specifically models and mitigates the "bad half" of the distribution (where such outliers typically live), and finally maps to a bounded range. This bounded output is particularly important for the Transformer, whose output heads produce values that are easier to train when targets lie in a predictable range.

Pretraining

The regressor's ability to generalize across domains comes from pretraining on a large corpus of offline evaluation data from diverse tasks. The pretraining procedure is designed to meta-learn a regression prior that captures the statistical regularities of black-box optimization landscapes.

Task definition and data sources. A task $\mathcal{T} = (f, \mathcal{X})$ is defined as a specific objective function $f$ over a particular search space $\mathcal{X}$. The pretraining corpus consists of offline evaluation trajectories from many such tasks, each containing a sequence of $T$ evaluated trials $\{x_s, y_s\}_{s=1}^T$ where $T \geq 100$ (the paper uses a fixed $T \geq 100$ for all tasks during training). The paper uses the following data sources:

• Synthetic BBOB functions: 1 million tasks generated by sampling from the training split of BBOB functions (Sphere, Ellipsoidal, Rastrigin, etc.), applying randomized transformations (shifting, rotation, discretization), and evaluating on uniformly sampled candidates. Each task represents a different instantiation of a BBOB landscape with different dimensionality, parameter types, and transformations.

• Combinatorial problems: 1 million tasks generated by randomizing the coefficients of combinatorial objectives (city locations for TSP, cost matrices for Flowshop, weight matrices for Quadratic Assignment, etc.) and evaluating on randomly sampled permutations or subsets. Each task represents a different problem instance with different underlying parameters.

The paper does not use real hyperparameter optimization data for pretraining, instead training exclusively on synthetic data (Section 4.1, HPO section: "we use the same regressor model trained only on synthetic BBOB trajectories"). This is a crucial design choice that tests the out-of-distribution generalization of the approach: the model sees only synthetic function landscapes during training, yet must perform regression on real-world hyperparameter optimization objectives at test time. The positive results on HPO surrogates (Figures 5 and 6) demonstrate that the embedding-based regressor transfers effectively from synthetic to real-world data.

Training data construction. For each training task, the paper generates a full trajectory of $T$ trials by sampling candidates uniformly from the search space and evaluating them. This uniform sampling strategy means the training data consists entirely of non-adaptive trajectories — there is no Bayesian Optimization loop generating the data, just random exploration. Despite this, the model learns to make predictions that are useful for adaptive optimization at test time, because the Transformer's in-context learning mechanism does not depend on how the training trajectories were generated; it only needs diverse examples of (history, query point, true value) triplets to learn the conditional predictive distribution.

Training example construction. Each training example is constructed by:

1. Sampling a task $\mathcal{T}$ uniformly from the pretraining corpus.
2. Sampling a cutoff point $t'$ uniformly from $[10, T - 10]$ — this is the number of "history" trials provided to the model. The minimum of 10 ensures the model always has sufficient context; the maximum of $T - 10$ ensures there are always at least 10 held-out target points for loss computation.
3. The first $t'$ trials form the history: $\{x_s, y_s\}_{s=1}^{t'}$.
4. The remaining $T - t'$ trials form the target points: $\{x_{t'+i}, y_{t'+i}\}_{i=1}^{T-t'}$.

Loss function. The loss for a single training example is the average negative log-likelihood over all target points under the model's predictive distribution, conditioned on the history:

$\mathcal{L}_{\text{task}} = \frac{1}{T - t'} \sum_{i=1}^{T-t'} \ell_\theta(x_{t'+i}, y_{t'+i}; \{x_s, y_s\}_{s=1}^{t'})$

where $\ell_\theta(x, y; \text{history})$ is the negative log-likelihood of observing $y$ given $x$ and the history, under the model's Gaussian output distribution:

$\ell_\theta = -\log \mathcal{N}\left(y \mid \mu_\theta(x, \text{history}), \sigma^2_\theta(x, \text{history})\right)$

$= \frac{1}{2} \log(2\pi \sigma^2_\theta) + \frac{(y - \mu_\theta)^2}{2\sigma^2_\theta}$

What this loss computes: For each target point, the model produces a predicted mean $\mu$ and variance $\sigma^2$. The Gaussian negative log-likelihood penalizes both inaccurate means (large squared error $(y - \mu)^2$) and miscalibrated uncertainties (if $\sigma^2$ is too small for the actual error, the loss is large; if $\sigma^2$ is too large, the $\log(\sigma^2)$ term penalizes unnecessary uncertainty). The total loss is the average over all target points, encouraging the model to make accurate and well-calibrated predictions across the entire trajectory.

Why this form: The average over target points ensures the model learns to predict at arbitrary points in the trajectory, not just at the end. The negative log-likelihood is the proper scoring rule for probabilistic predictions — minimizing it encourages both accuracy (mean prediction) and calibration (uncertainty estimation) simultaneously. Using a Gaussian likelihood (rather than, say, MSE) is essential because the downstream Bayesian Optimization acquisition function requires uncertainty estimates to balance exploration and exploitation. An MSE-trained model might produce accurate means but would have no incentive to produce useful uncertainty estimates; the NLL loss directly penalizes both poor means and poor variances.

Training hyperparameters. The paper provides the full set in Appendix A:

• Model size: 1024 feature dimension, feedforward projection 4096, 8 attention layers with 16 heads each.
• Embedder: T5-XL encoder (1B parameters, frozen), SentencePiece tokenizer with vocabulary 32,000, maximum 400 tokens per string.
• Optimization: AdamW optimizer, effective batch size 16, learning rate $5 \times 10^{-4}$, weight decay $10^{-5}$, gradient clipping 0.5.
• Data sampling: Fixed total trials $T \geq 100$ per task in the context window, number of history trials $t' \sim \text{Uniform}(10, T-10)$, rest are targets.

Why AdamW with these hyperparameters? The learning rate of $5 \times 10^{-4}$ is relatively high for a Transformer, suggesting the model is trained with a warmup schedule (not explicitly mentioned but standard in Transformer training). The weight decay of $10^{-5}$ is modest, providing light regularization. The gradient clipping of 0.5 is a standard stabilization technique for Transformers. The effective batch size of 16 (with gradient accumulation if needed across GPUs) is typical for models of this scale.

No test-time fine-tuning. A key property of the approach is that after pretraining, the regressor weights $\theta$ are frozen. All adaptation to new optimization tasks happens through the in-context mechanism: presenting the observed history as input tokens. This is in contrast to methods that require inference-time fine-tuning (Song et al., 2024a), which would add latency and hyperparameter complexity to each optimization run.

Why pretrain instead of using off-the-shelf LLMs directly? The paper explicitly argues against pure off-the-shelf LLM regression (Liu et al., 2024; Vacareanu et al., 2024) on the grounds that (a) the context length limits the number of usable trials, (b) the models cannot benefit from domain-specific offline data, and (c) the inference cost of large LLMs would be prohibitive when the acquisition function is called thousands of times per candidate proposal. The embedding + small Transformer approach addresses all three: the embeddings compress each trial to a single token, the small Transformer (8 layers) is cheap to run, and the model can be pretrained on as much offline data as available.

Data augmentation for distribution shift. The paper mentions (Section 3.3) that "since there may be distributional shifts for parameter names encountered between pretraining and inference, we may either apply data augmentation by randomizing parameter names during pretraining, or transform the search space during inference to match those encountered in pretraining." The BBOB pretraining uses arbitrarily named parameters (e.g., p0, p1, etc.), while hyperparameter optimization tasks might use meaningful names (e.g., learning_rate, batch_size). By randomizing parameter names during pretraining, the model learns to ignore the semantic content of parameter names and focus on the numeric patterns in the embedding space, enabling transfer to unseen parameter naming conventions.

Inference-Time Acquisition Loop

During online Bayesian Optimization, the pretrained regressor is integrated into a standard explore-exploit loop.

Acquisition function. The paper uses the Upper Confidence Bound (UCB) acquisition function:

$a_{t+1}(x) = \mu_{t+1}(x) + \sqrt{\beta} \cdot \sigma_{t+1}(x)$

where $\mu_{t+1}(x)$ is the predicted mean, $\sigma_{t+1}(x)$ is the predicted standard deviation, and $\sqrt{\beta}$ is a problem-dependent exploration coefficient.

What it computes: UCB scores a candidate $x$ by adding the predicted performance (mean) and an exploration bonus proportional to the predictive uncertainty (standard deviation). A candidate with high predicted mean is promising; a candidate with high uncertainty is unexplored. The coefficient $\sqrt{\beta}$ controls the tradeoff: larger $\beta$ encourages more exploration (preferring uncertain candidates), smaller $\beta$ encourages exploitation (preferring known good candidates).

Why this form: UCB is chosen for its simplicity and compatibility with the Vizier GP-Bandit baseline, which also uses UCB (Song et al., 2024c). This ensures a clean comparison: both methods use the same acquisition function form, so performance differences can be attributed to the regressor's predictive quality rather than the acquisition strategy. The paper uses a fixed $\sqrt{\beta} = 1.8$ across all experiments (Appendix A), which is described as a "problem-dependent constant." In principle, $\beta$ should be tuned per-task or adjusted dynamically based on the optimization progress, but the paper uses a fixed value to avoid complicating the comparison.

Acquisition optimizer (tabular and continuous spaces). For synthetic (BBOB) and hyperparameter optimization tasks, the paper uses the Firefly acquisition optimizer — the same algorithm used by Google Vizier's GP-Bandit. Firefly is an evolutionary algorithm that searches over the acquisition function surface to find the most promising candidate to evaluate next. The paper uses a maximum budget of 1,000 evaluations for Firefly (Appendix A), meaning that for each candidate proposal, up to 1,000 possible candidates are scored by the regressor and the best according to UCB is selected. This is the setting where inference efficiency matters: the regressor must be fast enough to score thousands of candidates per proposal.

The paper emphasizes that using the same acquisition optimizer as Vizier is critical for clean comparison: "it is well known that other components in the Bayesian Optimization pipeline (e.g. choice of acquisition and acquisition optimizer) also strongly affect performance and can lead to confounding factors" (Section 4.1). By swapping only the regressor, the experiment isolates the effect of the regressor on optimization performance.

Acquisition optimizer (combinatorial spaces). For combinatorial optimization tasks (permutations, choices), the paper uses Regularized Evolution (Real et al., 2019) rather than Firefly. The reason is practical: evolutionary algorithms over permutation spaces require domain-specific mutation operators (e.g., swapping two elements, inverting a subsequence) that are not available in the general-purpose Firefly optimizer. Regularized Evolution maintains a population of candidates, selects the best according to UCB, and applies random mutations to generate new candidates.

Rather than fully optimizing the acquisition function (which would require running evolution for many generations), the paper uses a simplified best-of-many sampling approach: evolution proposes 5 candidate solutions, the UCB acquisition scores all 5, and the highest-scoring candidate is selected for evaluation. This is computationally cheaper than running evolution to convergence on the acquisition surface, and the paper found it sufficient to provide meaningful exploration guidance beyond the baseline Regularized Evolution (which selects purely based on objective value, without exploration bonuses).

The Regularized Evolution configuration: population size 50, tournament size 7 (approximately $\sqrt{\text{population size}}$ as prescribed in the original paper), and the UCB-based ranking begins after 20 initial random trials (to give the regressor a minimal history before making predictions).

Why Regularized Evolution for combinatorial spaces? The paper notes that prior work on Bayesian Optimization over permutation spaces (Deshwal et al., 2022; Oh et al., 2022) requires "constructing very domain-specific kernels and complex acquisition optimizers (e.g. semi-definite programming) making them difficult to reproduce." The embedding approach avoids this entirely: the regressor operates in embedding space, not permutation space, so it doesn't need a permutation-specific kernel. The acquisition optimizer (Regularized Evolution) operates in the original permutation space using standard mutation operators, and only uses the regressor's UCB scores to rank candidates. This decouples the regression model from the search space structure — exactly the flexibility advantage the paper claims.

Full Model Specifications and Computational Cost

The paper provides concrete model sizes and computational requirements to demonstrate that the approach is practical for academic research settings (Section 3.4 and Appendix A).

Embedder: T5-XL Encoder. The embedder is the encoder portion of the T5-XL model (Raffel et al., 2020), containing approximately 1 billion parameters. It processes input strings of up to 400 tokens (clipped from potentially longer strings) using the SentencePiece tokenizer (Kudo and Richardson, 2018) with a vocabulary of 32,000 subword tokens. The encoder output has dimension $d = 4096$, which is then projected down to $d_{\text{model}} = 1024$ for the regressor. The embedder's weights are frozen — no fine-tuning is performed on optimization data. This means the embedding computation is a pure forward pass, and the embedder's knowledge comes entirely from its original pretraining on the C4 text corpus.

Why T5-XL specifically? The paper states they "intentionally use relatively smaller language model embedder sizes in comparison to the larger and significantly more expensive GPT or Gemini family of models" (Section 3.4). T5-XL is chosen as a balance point: it is large enough to produce useful representations (as shown by the ablation in Figure 7, where larger T5 variants produce monotonically better predictions) but small enough that its embeddings can be computed cheaply during inference. The paper also notes that "faster embedders lead to large constant factor reductions" in overall cost, since the embedder is called for every candidate scored by the acquisition optimizer.

Regressor: 8-layer Transformer. The ICL Transformer has 8 attention layers, 16 heads per layer, a hidden dimension of 1024, and feedforward projections of 4096. The total parameter count is not explicitly stated but is modest — the paper claims a training budget of "approximately 16 GPUs" and inference on "1 GPU," which is feasible for most academic groups. The paper suggests that even this cost could be reduced using efficient Transformer architectures (Tay et al., 2022) that achieve sub-quadratic attention complexity, though such variants are not explored.

Training cost. The paper uses an effective batch size of 16, with gradient accumulation across GPUs if needed. Training is performed on 1 million BBOB tasks and 1 million combinatorial tasks (presumably separate models, though the paper is not explicit about whether a single model is trained on both data sources jointly or separate models are trained per domain). Each task contributes a trajectory of at least 100 trials. The maximum number of trials in the context window is $T \geq 100$, with $t' \sim \text{Uniform}(10, T-10)$ history trials and the remainder as targets. This means each training example requires processing a sequence of at least 100 tokens (one per trial), though the Transformer's quadratic attention means the computational cost per example scales as $\mathcal{O}(T^2)$.

Inference cost. The paper emphasizes that "the cheap inference cost is also necessary when the acquisition function may be called thousands of times by a zeroth-order acquisition optimizer per candidate proposal" (Section 3.4). With a 1,000-evaluation budget for Firefly and, say, 200 optimization trials, the regressor is called roughly 200,000 times over the course of an optimization run. This is feasible with the small Transformer (single GPU) but would be prohibitive with larger models.

Why these sizes? The paper's design philosophy is to demonstrate sufficiency with conservative choices rather than to maximize performance with expensive models. The ablation studies (Figures 7 and 8) show that both larger embedders and larger regressors improve predictive performance, suggesting that using a larger LLM embedder (e.g., PaLM, Gemini) and a larger regressor (more layers, larger hidden dimension) would likely yield even better optimization performance. The paper deliberately leaves these improvements to future work, focusing instead on establishing that even modestly-sized models achieve competitive results.

Summary of Design Choices and Their Justifications

• Frozen embedder over learned embeddings: Freezing the embedder decouples representation learning (handled by the LLM's pretraining on text) from regression learning (handled by the Transformer Neural Process). This reduces the number of trainable parameters and allows the model to benefit from the LLM's general-purpose representational capacities without requiring optimization-specific fine-tuning. The ablation in Figure 7 supports this: larger frozen embedders produce better regression even without optimization-aware training.

• Average pooling over learned aggregation: Average pooling is the simplest possible method for converting variable-length token sequences to fixed-length vectors. The paper explicitly identifies this as a target for improvement (Section 5: "the aggregation method over Transformer outputs may be learned rather than predefined using average pooling"). The choice of average pooling makes the positive results more surprising: even this lossy compression preserves sufficient structure.

• JSON string format over natural language: JSON is chosen for its simplicity, token efficiency, and deterministic structure. Natural language descriptions (e.g., "The learning rate is 0.001 and the batch size is 32") would consume many more tokens per trial, reducing the number of trials that fit in the context window. JSON is also easily parsed and generated programmatically, which matters for the acquisition optimizer (which needs to propose new candidates as JSON strings).

• UCB acquisition with fixed $\sqrt{\beta} = 1.8$ over more sophisticated acquisition functions: UCB is used for direct comparability with the Vizier GP-Bandit baseline. The fixed exploration coefficient avoids per-task tuning while still providing meaningful exploration bonuses. More sophisticated acquisition functions (Expected Improvement, Thompson Sampling, entropy search) might improve performance but would complicate the comparison.

• Synthetic-only pretraining over mixed synthetic-real pretraining: Using only synthetic BBOB data for pretraining (even for HPO experiments) is a deliberate choice to test out-of-distribution generalization. It also avoids the practical challenge of obtaining large-scale real-world optimization data, which is typically proprietary or expensive to generate. The positive HPO results validate that synthetic pretraining transfers to real tasks.

• Regularized Evolution for combinatorial acquisition optimization over domain-specific solvers: The paper explicitly avoids the complex acquisition optimizers required by prior combinatorial BO work (e.g., semi-definite programming for permutation kernels). Regularized Evolution with simple mutation operators is domain-agnostic and requires only the ability to randomly modify a candidate, making it applicable to any search space where mutations can be defined. This aligns with the paper's theme of generality and ease of reproduction.

4. Key Insights and Innovations

Innovation 1: Embeddings as a Task-Independent Regression Prior — Separating Representation from Prediction in Bayesian Optimization

The dominant assumption in Bayesian Optimization has always been that the regressor and the input representation are a single indivisible system: a Gaussian Process is defined by its kernel over a specific input space, a Transformer Neural Process is architected to accept a specific input dimension, and even text-to-text approaches entangle the representation (how trials are described) with the prediction mechanism (the LLM's internal reasoning). This paper breaks that coupling in a way that is both conceptually simple and practically powerful.

The key move is relegating representation entirely to a frozen external component — a pretrained language model embedder that converts arbitrary strings to fixed-length vectors — while the regression model operates purely on those vectors. The regressor never sees the original input format, never needs to know whether it's modeling hyperparameters or permutations, and never adapts its architecture to the search space. This is fundamentally different from prior work on flexible BO, which either (a) designed hand-crafted kernels or embeddings for each new domain (Deshwal et al., 2022, 2023; Ru et al., 2021), or (b) used token-based representations that still required the regression model to process variable-length, format-dependent sequences (Chen et al., 2022; Liu et al., 2024), or (c) performed inference-time fine-tuning to adapt a fixed model to new data (Song et al., 2024a; Akhauri et al., 2025). All of these approaches bind the regressor to the search space at some level — through kernel design, tokenization schemes, or parameter updates. The embed-then-regress framework eliminates this binding entirely: the regressor lives in embedding space, and embedding space is always $\mathbb{R}^d$.

This is not merely an architectural convenience. It represents a conceptual reframing of what a regressor for optimization needs to know. The paper shows that a Transformer Neural Process trained on synthetic BBOB data can be dropped directly into hyperparameter optimization tasks (Figures 5 and 6) and combinatorial optimization tasks (Figure 4) without any retraining, fine-tuning, or even task identification metadata — it simply processes the embeddings of whatever trials it observes, and produces calibrated predictions. The fact that the same model weights achieve this across such fundamentally different mathematical structures (continuous landscapes, permutation groups, subset selection) is the empirical signature of a genuine separation between representation and prediction.

The theoretical underpinning comes from Tang et al. (2025), which the paper cites to establish Lipschitz continuity of LLM embeddings over tabular features. But the paper's contribution goes beyond citing this result: it demonstrates that this continuity property is sufficient for competitive optimization, not just for reasonable regression accuracy. The GP-Bandit baseline in Figure 3 represents decades of engineering and theoretical work on Gaussian Process regression specifically tailored to continuous and tabular spaces. Matching or exceeding it with a method that treats these spaces identically to permutations and choices — using the same embedder with no per-space adaptation — is evidence that the embedding space preserves not just local geometry but the kind of global structure that enables effective exploration-exploitation tradeoffs.

This is a fundamental advance, not an incremental improvement. It changes the question from "how do we design a good regressor for this search space?" to "can we find an embedding space where a single regressor works across all search spaces?" The answer appears to be yes, using off-the-shelf LLM embeddings with no optimization-specific training.

Innovation 2: Demonstrating That Synthetic-Only Pretraining Transfers to Real-World Optimization Tasks

A persistent concern in learned optimization methods is data availability: real-world optimization trajectories — especially from expensive evaluations like hyperparameter tuning or hardware design — are scarce, proprietary, and expensive to generate. Most prior work on learned regressors for BO either trained on the same type of data they would be tested on (e.g., Chen et al., 2022 trained on hyperparameter optimization data for hyperparameter optimization; Müller et al., 2023 trained on synthetic functions similar to their test functions), or used off-the-shelf LLMs that didn't require training data at all but couldn't benefit from pretraining (Liu et al., 2024; Vacareanu et al., 2024).

The paper makes a striking empirical claim that cuts through this dilemma: a regressor pretrained exclusively on synthetic BBOB data — randomized, transformed mathematical functions — transfers effectively to real-world hyperparameter optimization tasks without any exposure to real HPO data during training. This is stated explicitly in Section 4.1: "we use the same regressor model trained only on synthetic BBOB trajectories as found previously in Figure 3" for the hyperparameter optimization experiments. The HPO results in Figures 5 and 6 are produced by a model that has never seen a learning rate, a batch size, or a validation accuracy.

This finding matters because it challenges the assumption that learned regressors need domain-matched pretraining data. If synthetic function data — which is infinite, free to generate, and covers a wide variety of landscape types — produces regressors that work on real tasks, then the data bottleneck for learned BO largely disappears. The key enabling factor is the string embedding: the synthetic BBOB data trains the regressor to map embedding-space geometries to predictions, and the LLM embedder ensures that real HPO parameters project into embedding space with similar structural properties (Lipschitz continuity, smoothness, meaningful distances) as the synthetic parameters.

This is an empirical discovery rather than a theoretical advance, but its practical implications are substantial. It means organizations can pretrain a general-purpose BO regressor on cheap synthetic data and deploy it on proprietary optimization problems without ever exposing the regressor to their real evaluation data during training — a form of zero-shot transfer that wasn't previously demonstrated for regression-based BO at this level of competitiveness.

The violin plot in Figure 6 provides the aggregate evidence: across 20 diverse HPO tasks (image classification, hardware metrics, production metrics), embed-then-regress achieves the same median log-efficiency as GP-Bandit, with the two distributions largely overlapping. Log-efficiency being a scale-invariant metric (roughly, the factor of trials needed to reach equivalent performance) means this isn't an artifact of objective scaling or cherry-picked tasks. The model genuinely matches an industry-grade GP system on problems it was never trained to solve.

Innovation 3: Best-of-Many UCB-Guided Selection as a Lightweight Alternative to Full Acquisition Optimization on Combinatorial Spaces

The paper introduces a pragmatic mechanism for deploying embedding-based regressors on combinatorial optimization that avoids the complexity and domain-specificity of prior Bayesian Optimization approaches over structured spaces. The standard paradigm for combinatorial BO (as exemplified by Deshwal et al., 2022; Oh et al., 2022) requires two tightly coupled components: a kernel that defines a valid covariance function over the combinatorial structure (e.g., a permutation kernel based on Kendall-tau distance or an assignment kernel), and an acquisition optimizer that can efficiently search over that space guided by the GP's predictions (often requiring semi-definite programming, quadratic programming, or specialized graph algorithms). Changing the search space — from permutations to subsets to graphs — means redesigning both components from scratch.

The paper's approach is to decouple the regressor from the acquisition optimizer entirely. The regressor operates in embedding space and is agnostic to the search space structure. The acquisition optimizer operates in the original combinatorial space using a domain-agnostic evolutionary algorithm (Regularized Evolution) that only needs mutation operators — swapping two elements in a permutation, toggling an element in a subset. The regressor's UCB scores are used only to rank the candidates proposed by evolution, selecting the best out of a small batch (5 samples) rather than fully optimizing the acquisition surface.

This "best-of-many" approach is not theoretically optimal — it doesn't find the global maximum of the acquisition function, which is what standard BO theory prescribes. But it is practically effective in a way that challenges the necessity of sophisticated acquisition optimization. Figure 4 shows that this lightweight combination of Regularized Evolution + UCB-guided selection consistently outperforms the baseline Regularized Evolution (which selects purely on objective value with no exploration bonuses), and the gains appear early (within 100–200 trials) and persist across permutation problems (Travelling Salesman, Quadratic Assignment, Flowshop Scheduling, N-Queens) and choice problems (Coverage, Log Determinant, Modular).

The significance of this innovation is methodological rather than theoretical: it demonstrates that the overhead of domain-specific kernel design and acquisition optimization — which has been a major barrier to applying BO to combinatorial problems — may be unnecessary if the regressor is sufficiently flexible. The embedding-based regressor provides the exploration guidance (via UCB uncertainty estimates) that Regularized Evolution lacks, while the evolutionary algorithm provides the search mechanism that doesn't need to be reformulated for each problem type. This separation of concerns — regressor handles where to explore, evolution handles how to generate candidates — is a practical template for applying BO to new combinatorial domains without deep expertise in kernel design.

The paper explicitly positions this against the complexity of prior work: "While there are few previous works using GPs for e.g. permutation spaces, they require constructing very domain-specific kernels and complex acquisition optimizers (e.g. semi-definite programming) making them difficult to reproduce" (Section 4.1). The embed-then-regress approach requires only mutation operators, which exist for virtually any combinatorial structure, and a regressor that is already trained and frozen.

Innovation 4: Architecture Scaling Laws for Embedding-Based Regression — Larger Frozen Embedders Produce Monotonically Better Predictors

The paper provides what amounts to a scaling law for embedding-based regression in Figure 7, showing that as the frozen T5 embedder size increases from Small to Large to XL, predictive performance improves monotonically across all metrics (NLL, MAE, R², MACE) and all context sizes. This result is significant not because it's surprising that larger models produce better embeddings — that's a well-established trend in NLP — but because of what it implies about the nature of the representations being used.

The T5 models were pretrained on English text (the C4 corpus), not on JSON-formatted optimization parameters. There is no obvious reason that a model trained to predict masked spans in Wikipedia articles and web pages should develop incrementally better representations of {"p0": 0.3, "p1": "category_1"} as it scales up. The monotonic improvement suggests that the linguistic knowledge acquired during text pretraining transfers to structured numeric representations in a smooth, capacity-dependent way — larger models don't just know more about language; they develop richer geometric representations that happen to be useful for encoding the similarity structure of arbitrary token sequences, including those far outside their training distribution.

This has practical implications for model selection in embedding-based BO: it provides evidence that investing in larger embedders yields reliable returns in regression quality, even if those embedders are never fine-tuned on optimization data. It also connects to the broader literature on scaling laws for language models (Kaplan et al., 2020) but in a novel context — the downstream task is not language modeling or NLP, but black-box optimization, and the metric is not perplexity but negative log-likelihood on held-out function evaluations.

Figure 8 provides a complementary scaling result for the ICL Transformer regressor: more layers (2 → 4 → 6 → 8) produce better predictions. This is expected (larger models have higher capacity) but validates that the pretraining data volume (1M tasks) is sufficient to support learning in larger architectures — a non-trivial finding given that overfitting on synthetic data is a common failure mode.

Together, Figures 7 and 8 constitute an empirical justification for scaling both the embedder and the regressor, providing a roadmap for future work: using GPT-scale or Gemini-scale embedders and larger Transformers would likely yield further improvements, with the primary constraint being inference-time compute rather than a fundamental ceiling on what embeddings can capture. The paper notes this explicitly in Section 3.4: "we intentionally use relatively smaller language model embedder sizes in comparison to the larger and significantly more expensive GPT or Gemini family of models," implying that the current results represent a lower bound on what is achievable.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper evaluates on three distinct benchmark families, each representing a different optimization domain. For synthetic optimization, the authors use the Black-Box Optimization Benchmarking (BBOB) suite (ElHara et al., 2019), containing 24 objective functions over continuous search spaces, split by landscape type into training and test functions with randomized transformations (shifting, rotation, discretization) applied to prevent overfitting and introduce categorical parameters. For combinatorial optimization, the paper implements five permutation-based problems (Travelling Salesman, Flowshop Scheduling, Linear Ordering, Quadratic Assignment, N-Queens) and three choice-based problems (Modular Function, Coverage Function, Log Determinant Function), with problem coefficients repeatedly randomized to generate distinct task instances. For hyperparameter optimization, the authors use 20 representative surrogate-based tasks drawn from real-world tuning studies (image classification, hardware metrics, production metrics), with tree-based XGBoost surrogates (Chen and Guestrin, 2016) interpolating offline evaluation data to simulate realistic online objective functions without incurring the wall-clock cost of live evaluation. The synthetic and combinatorial pretraining corpora each contain 1 million randomly generated tasks; the HPO tasks serve only as test problems and are never used for training.

• Base model(s). The embedder is a frozen T5-XL encoder (Raffel et al., 2020) with approximately 1B parameters, pretrained on the C4 text corpus and never fine-tuned on optimization data. This choice is deliberate: the paper explicitly aims to use a model "representative of the capabilities of many contemporary LLMs" while remaining "relatively smaller" than GPT- or Gemini-scale models to keep inference costs manageable (Section 3.4). The regression model is an 8-layer Transformer Neural Process (Nguyen and Grover, 2022) with 1024-dimensional features, 16 attention heads, and 4096-dimensional feedforward projections, trained from scratch on the synthetic pretraining data. The paper also ablates over smaller T5 variants (Small, Large) as embedders and smaller Transformer depths (2, 4, 6 layers) as regressors to establish scaling trends.

• Metrics. For optimization performance, the primary metric is best-so-far objective value plotted against the number of trials (function evaluations), with curves showing mean and ±0.5 standard deviation over 10 or 20 random seeds depending on the benchmark. For synthetic BBOB functions, where the global optimum is known, the paper reports optimality gap (difference between best-so-far and global optimum) on log-scaled y-axes to emphasize separation between methods. For combinatorial and HPO tasks where global optima are unknown, the paper reports raw or normalized best-so-far objective values (higher is better). For hyperparameter optimization, the paper additionally uses log-efficiency (Song et al., 2024c), a scale-invariant metric defined as $\text{LogEfficiency}(y \mid f, A, A_{\text{ref}}) = \log\left(\frac{\text{RequiredBudget}(y \mid f, A)}{\text{RequiredBudget}(y \mid f, A_{\text{ref}})}\right)$, where $\text{RequiredBudget}(y \mid f, A)$ is the minimum number of trials for algorithm $A$ to reach performance $y$ on function $f$, and $A_{\text{ref}}$ is embed-then-regress. A log-efficiency of $c$ means the baseline requires $\exp(-c)$ times as many trials as the reference method. This is aggregated via the median over varying $y$ levels spanning the averaged best-so-far curves. For regression quality assessment (ablation studies), the paper uses four predictive metrics: negative log-likelihood (NLL), mean absolute error (MAE), R-squared, and mean absolute calibration error (MACE) (Chung et al., 2021), all computed on normalized $y$-values over held-out BBOB test functions.

• Baselines. The paper uses four baseline methods. (1) Google Vizier GP-Bandit (Song et al., 2024c): the industry-grade Gaussian Process-based Bayesian Optimization system, using UCB acquisition with the Firefly acquisition optimizer (for tabular spaces) and reported to be near-optimal across five major industry methods including Optuna (Akiba et al., 2019) and Ax (Balandat et al., 2020a). This is the primary competitive baseline and the paper goes to significant lengths to ensure clean comparison by keeping all other pipeline components (acquisition function form, acquisition optimizer, initial random trials) identical between GP-Bandit and embed-then-regress, swapping only the regressor. (2) Random search: uniformly random sampling without any regression model or acquisition guidance, serving as a lower bound. (3) Quasi-random search: low-discrepancy sampling (e.g., Sobol sequences) providing better coverage than pure random sampling but still without learned guidance. (4) Regularized Evolution (Real et al., 2019): an evolutionary algorithm maintaining a population of 50 candidates with tournament size 7, used for combinatorial problems where GP-based methods are impractical. For combinatorial experiments, Regularized Evolution serves as both the baseline optimizer and the acquisition optimizer for embed-then-regress (where it is augmented with UCB-guided selection rather than pure fitness-based selection).

• Generation budget / compute accounting. The paper measures computation in terms of number of function evaluations (trials), which is the standard unit in black-box optimization where the objective function is assumed to be expensive. All methods are compared at equal trial counts, typically ranging from 20–300 depending on the benchmark. For synthetic and HPO tasks, the acquisition optimizer (Firefly) uses a maximum of 1,000 internal evaluations per candidate proposal for both GP-Bandit and embed-then-regress; these are cheap function evaluations of the learned acquisition surface, not the expensive true objective, and the paper keeps this budget equal across methods. For combinatorial tasks, Regularized Evolution uses 5 acquisition-ranked candidate proposals per trial for embed-then-regress versus pure fitness selection for the baseline; the additional computational cost of the regressor forward passes (8-layer Transformer, single GPU) is noted as negligible relative to the true objective evaluations. The paper explicitly does not include the cost of the initial 20 random trials (for combinatorial tasks) or the pretraining cost in optimization efficiency comparisons, consistent with standard BO benchmarking practice where pretraining is a one-time offline cost amortized over many deployments.

• Cross-validation / statistical protocol. The BBOB train-test split is performed at the landscape type level to ensure test functions are structurally distinct from training functions (e.g., training includes Sphere, Rastrigin, BentCigar; test includes BuecheRastrigin, RosenbrockRotated, SharpRidge). All reported optimization curves use 20 random seeds (Appendix B states "For every algorithm and objective pair, we run 20 seeds"), with mean and 0.5 standard deviation error bars displayed. For hyperparameter optimization, the 20 HPO tasks are "representative and diverse" and were "randomly chosen" (Section 4.1); the paper reports both per-task curves for 8 randomly selected tasks (Figure 5) and aggregate violin plots over all 20 tasks (Figure 6). The combinatorial experiments use 10 repeats per problem instance (inferred from curve smoothness and the "10 repeats" statement in Section 4.1). Predictive ablation studies (Figures 7 and 8) use 10 test BBOB functions with mean and standard deviation reported. No formal statistical significance tests (e.g., t-tests, bootstrap confidence intervals) are reported; the paper relies on visual separation of error bars and aggregate metrics like median log-efficiency.

Main Quantitative Results

Synthetic Optimization (BBOB)

The paper reports results on 9 randomized test functions from the BBOB suite (Figure 3), with embed-then-regress compared against GP-Bandit, Random, and Quasi-Random baselines. The headline finding is that embed-then-regress is generally comparable to GP-Bandit and can significantly outperform it in several cases, with the performance gap persisting across all trial budgets (25–100+ trials).

On the majority of functions (BuecheRastrigin 4D, NegativeSphere 5D, SharpRidge 5D, SchaffersF7IllConditioned 8D, RosenbrockRotated 4D, GriewankRosenbrock 4D), the two methods track closely, with overlapping error bars throughout the optimization trajectory. The paper characterizes this as "generally comparable" performance, though careful inspection of the log-scaled plots reveals that GP-Bandit holds a slight advantage on approximately 4–5 functions (e.g., NegativeSphere 5D, SharpRidge 5D) while the methods are near-identical on others (RosenbrockRotated 4D, GriewankRosenbrock 4D).

However, on two functions, embed-then-regress achieves substantially better performance that persists even after 100+ trials:

• Gallagher21Me 7D: Embed-then-regress reaches an optimality gap of roughly $2 \times 10^1$ at 100 trials, while GP-Bandit plateaus around $4 \times 10^1$. The gap is approximately a factor of 2 in optimality gap and widens at higher trial counts, with the GP-Bandit curve appearing to saturate while embed-then-regress continues improving.

• Lunacek 2D: Embed-then-regress reaches roughly $10^1$ at 100 trials versus GP-Bandit at approximately $3-4 \times 10^1$. The embed-then-regress curve consistently lies below the GP-Bandit curve across all trial counts from roughly 20 onward.

These results are notable because Gallagher21Me is a multi-modal function with 21 randomly located optima — precisely the kind of landscape where GP-based methods typically excel due to their principled uncertainty estimates guiding exploration. The fact that an embedding-based regressor trained on synthetic data (with entirely different functions in its training set) can outperform a well-tuned GP on such problems suggests that the meta-learned regression prior captures exploration-relevant structure beyond what the GP's stationary kernel assumptions provide.

The paper emphasizes that these results were obtained with "non-continuous parameters" introduced via random discretization (Section 4.1), meaning the search spaces include both continuous and discrete/categorical dimensions — a realistic setting that challenges GP methods which typically perform best on purely continuous spaces. The BBOB transformations appendix (Appendix B.1) details that each parameter is randomly designated as continuous, DISCRETE, or CATEGORICAL with 2–16 feasible points, breaking the continuous smoothness assumptions that Gaussian Processes rely on.

Combinatorial Optimization

The paper evaluates embed-then-regress on 8 randomized combinatorial problems (Figure 4), spanning permutation spaces (Queen Placement sizes 10 and 11, Quadratic Assignment size 14, Linear Ordering size 9) and choice spaces (Coverage sizes 7-choose-3 and 9-choose-4, Modular size 12-choose-4, Log Determinant size 8-choose-4). The baseline is Regularized Evolution (population 50, tournament size 7) without any regression model. Embed-then-regress uses the same evolutionary algorithm but replaces pure fitness-based selection with UCB-guided best-of-5 selection starting at trial 20 (after 20 initial random trials to provide the regressor with a minimal history).

The headline finding is that embed-then-regress consistently outperforms Regularized Evolution across all 8 problems, with the performance gap emerging early (by trial 50–100) and persisting or widening through 300 trials. Specific notable results:

• Queen Placement (10): Embed-then-regress reaches roughly $-5$ best-so-far attacks (fewer attacks = better, since $f(x)$ is negative of the number of diagonal attacking pairs) by trial 100, while Regularized Evolution stagnates around $-20$. This represents nearly a 4× improvement in the raw objective and is particularly significant because the N-Queens landscape is highly deceptive — many configurations have the same number of attacks, creating large plateaus that are difficult for evolutionary search without an exploration model.

• Queen Placement (11): A similar pattern with embed-then-regress reaching roughly $-3$ vs. Regularized Evolution at $-12$ by trial 200.

• Quadratic Assignment (14): Embed-then-regress reaches roughly $-295$ vs. Regularized Evolution at $-305$ by trial 300. The gap is smaller here (roughly 3% relative improvement), but the problem is NP-hard with a permutation space of size $14! \approx 8.7 \times 10^{10}$, and even small improvements in objective value are practically meaningful.

• Coverage (9, 4): Embed-then-regress reaches roughly $33.0$ vs. Regularized Evolution at $33.5$ by trial 250. The gap is modest but consistent.

• Log Determinant (8, 4): Embed-then-regress reaches roughly $9.30$ vs. Regularized Evolution at $9.23$ by trial 200. This is a choice problem (selecting 4 items from 8) where the objective is the log-determinant of a PSD matrix minor — a submodular function where greedy algorithms perform well, yet the UCB-guided regressor still provides a visible improvement.

A critical observation from Figure 4 is that the performance curves for Regularized Evolution frequently plateau (visible in Queen Placement, Quadratic Assignment, Coverage, and Modular functions), indicating that the evolutionary search alone gets stuck in local optima. Embed-then-regress continues to improve, suggesting that the regressor's UCB uncertainty estimates are guiding the search toward underexplored regions of the search space that would otherwise be missed. This is precisely the role Bayesian Optimization is designed to play, and the results demonstrate that embedding-based regression can fulfill this role even on highly structured combinatorial spaces.

The paper acknowledges that prior work on BO over permutations (Deshwal et al., 2022; Oh et al., 2022) uses domain-specific kernels and complex acquisition optimizers that are difficult to reproduce. The embed-then-regress approach achieves better-than-evolutionary performance with zero domain-specific modeling — the same regressor weights and embedder that were used for BBOB are applied directly to permutations and choices. This is a strong validation of the central flexibility claim.

Hyperparameter Optimization (HPO Surrogates)

The paper evaluates on 20 HPO surrogate tasks (Figure 5 shows 8 randomly selected individual curves; Figure 6 shows the aggregate violin plot over all 20). The regressor used here is the same model trained only on synthetic BBOB trajectories — it has never seen real hyperparameter optimization data. This is a deliberate test of cross-domain transfer.

The 8 individually plotted tasks (Figure 5) cover a diverse range: ImageNet-ResNet50 (7 parameters), Fashion MNIST-CNN (4 parameters), CIFAR10-WideResNet (4 parameters), ImageNet-ViT (3 parameters), Phone Hardware (4 parameters), and three Production Metrics (5, 6, and 3 parameters respectively). The normalized objective values displayed are meaningful only for relative comparison (raw values span $[-10^7, 10^7]$ for production functions, which would lack context).

The headline finding is that embed-then-regress is generally comparable to GP-Bandit across these tasks, with the two methods tracking closely on most functions:

• On ImageNet-ResNet50 (7P), Fashion MNIST-CNN (4P), and CIFAR10-WideResNet (4P), the curves are nearly identical, with GP-Bandit holding a very slight advantage (approximately 1–2% in normalized objective) at most trial counts. The differences are within the error bars throughout.

• On ImageNet-ViT (3P), embed-then-regress appears to slightly outperform GP-Bandit after roughly 60 trials, reaching normalized values around 2.5 vs. 2.3 for GP-Bandit — a gap of approximately 8%.

• On Phone Hardware (4P), GP-Bandit holds a consistent advantage of roughly 2–3% that persists through 100 trials.

• On the three Production Metrics (5P, 6P, 3P), the methods are nearly indistinguishable, with curves overlapping throughout.

The paper notes that normalized objective values are displayed "since raw objective values over large ranges, e.g. $y \in [-10^7, 10^7]$ from private functions would lack meaningful context." This is standard practice but means the reader cannot assess the absolute magnitude of improvements — only relative comparisons between methods on the same task are interpretable.

Aggregate results (Figure 6, violin plots): The paper aggregates performance across all 20 HPO tasks using log-efficiency scores, with embed-then-regress as the reference method (horizontal line at Log-Efficiency = 0). The violin plot shows:

• GP-Bandit achieves a median log-efficiency of approximately 0.0, with the distribution centered around zero and roughly symmetric, spanning from approximately −0.5 to +0.5. This means GP-Bandit requires roughly the same number of trials as embed-then-regress to reach equivalent performance — sometimes slightly fewer (negative log-efficiency), sometimes slightly more (positive log-efficiency), but the median is equal.

• Quasi-Random search has a median log-efficiency of roughly −0.5, with the distribution shifted negative (worse than embed-then-regress) and spanning approximately −1.0 to 0.0. This corresponds to requiring roughly $\exp(0.5) \approx 1.65\times$ as many trials.

• Random search has a median log-efficiency of roughly −1.0, spanning approximately −1.5 to −0.5, corresponding to roughly $\exp(1.0) \approx 2.7\times$ as many trials.

The paper characterizes this as embed-then-regress "achieving the same median log-efficiency as GP-Bandit." The overlapping distributions indicate that the two methods are statistically indistinguishable in aggregate, despite embed-then-regress having been trained exclusively on synthetic BBOB data with no exposure to hyperparameter optimization tasks.

This is the paper's strongest result in terms of practical impact: it demonstrates that a single pretrained model, using a frozen general-purpose embedder, can match an industry-grade GP system that has been extensively engineered for hyperparameter tuning specifically. The implication is that organizations could replace per-domain GP configurations with a single pretrained regressor, amortizing the training cost across many optimization tasks.

Predictive Ablation Studies: Scaling the Embedder and Regressor

The paper includes two ablation studies on regression quality (rather than optimization performance) to understand how model scale affects predictive accuracy. These use held-out BBOB test functions and measure NLL, MAE, R², and MACE across varying numbers of context points (40–100).

Embedder size ablation (Figure 7): The paper compares T5-Small, T5-Large, and T5-XL as frozen embedders, keeping the ICL Transformer regressor fixed at 8 layers. The results show a clear monotonic improvement with embedder size across all metrics and all context sizes:

• NLL: T5-XL achieves roughly −1.0 at 100 context points vs. T5-Large at −1.2 and T5-Small at −1.4. The gaps are consistent across context sizes, with larger embedders providing proportionally better predictions.

• MAE: T5-XL achieves roughly $6 \times 10^{-2}$ at 100 context points vs. T5-Large at $8 \times 10^{-2}$ and T5-Small at $9 \times 10^{-2}$.

• R²: T5-XL achieves roughly 0.55 at 100 context points vs. T5-Large at 0.45 and T5-Small at 0.35. This is a substantial gap — T5-XL explains roughly 20 percentage points more variance than T5-Small.

• MACE: T5-XL achieves roughly $7 \times 10^{-2}$ vs. T5-Large at $8 \times 10^{-2}$ and T5-Small at $9 \times 10^{-2}$.

The paper notes that this trend is observed despite the embedders being "frozen" and pretrained "over mostly English text" — the BBOB representations contain no English words, only JSON-formatted parameter-value pairs. The monotonic improvement "potentially suggests that larger language models inherently provide better features even for numeric data formats" (Section 4.2). This is a non-obvious finding: scaling up a model trained on natural language yields better representations of structured numeric data, even though that data format was never seen during the embedder's pretraining.

ICL Transformer depth ablation (Figure 8): The paper compares regressors with 2, 4, 6, and 8 attention layers, keeping the embedder fixed at T5-XL. The results show improvement with more layers:

• NLL: 8 layers achieves roughly −1.0 at 100 context points vs. 6 layers at −1.1, 4 layers at −1.3, 2 layers at −1.7. The improvement from 2 to 4 layers is larger than from 6 to 8, suggesting diminishing returns but no saturation at 8 layers.

• MAE: 8 layers achieves roughly $6 \times 10^{-2}$ vs. 2 layers at $9 \times 10^{-2}$.

• R²: 8 layers achieves roughly 0.55 vs. 2 layers at 0.35.

• MACE: 8 layers achieves roughly $7 \times 10^{-2}$ vs. 2 layers at $9 \times 10^{-2}$.

Critically, Figure 8 also verifies in-context regression behavior: across all layer depths, the predictive metrics improve monotonically as the number of context points increases from 40 to 100. The improvement is most pronounced for deeper models (the 8-layer slope is steeper), suggesting that larger regressors are better able to extract and combine information from more observed trials. This is the expected signature of effective in-context learning — more data leads to better predictions — and its presence confirms that the regressor is genuinely learning from the provided history rather than relying on some context-independent prior.

The combined message of Figures 7 and 8 is that both the embedder capacity and the regressor capacity contribute to predictive performance, with no evidence of saturation at the scales tested. This provides empirical justification for the paper's claim that "further investigation into different regression bodies could lead to e.g. string-based GPs using kernels over string embeddings" and that larger embedders (GPT-scale, Gemini-scale) would likely yield further gains.

Ablation Studies and Robustness Checks

Effect of using BBOB-only pretraining for HPO tasks: The paper does not explicitly ablate this as a separate study, but the HPO results themselves (Figure 5 and 6) serve as a de facto ablation of the pretraining data source. The regressor used for HPO was trained exclusively on synthetic BBOB trajectories — the paper explicitly states "we use the same regressor model trained only on synthetic BBOB trajectories as found previously in Figure 3" (Section 4.1, HPO section). The fact that this model achieves median log-efficiency equal to GP-Bandit on 20 real-world HPO tasks demonstrates that synthetic-only pretraining transfers effectively. However, the paper does not compare against a model trained on HPO data or on a mixture of BBOB + HPO data, so it is impossible to determine whether domain-matched pretraining would outperform the transfer setting. The 1M synthetic tasks provide vastly more data than could be practically collected for HPO, suggesting a data-volume-vs-domain-match tradeoff that is empirically resolved in favor of data volume.

Effect of UCB-guided selection in combinatorial optimization: The comparison between Regularized Evolution (with pure fitness selection) and embed-then-regress (with UCB-guided best-of-5 selection from the same evolutionary algorithm) serves as an implicit ablation of the acquisition function's contribution. Since the evolutionary algorithm is identical in both cases (population 50, tournament size 7, same mutation operators), the only difference is the selection criterion: fitness vs. UCB score. The consistent improvement across all 8 problems (Figure 4) demonstrates that the regressor's uncertainty estimates provide genuine exploration value beyond what the evolutionary algorithm's diversity mechanisms achieve on their own. This is corroborated by the qualitative observation that Regularized Evolution frequently plateaus (e.g., Queen Placement, Quadratic Assignment) while the UCB-guided variant continues improving. The paper does not ablate the number of candidates sampled per step (fixed at 5), the tournament size (fixed at 7), or the population size (fixed at 50), so the sensitivity of results to these hyperparameters is unknown.

Effect of embedder size on optimization (not just prediction): The ablation in Figure 7 measures predictive metrics (NLL, MAE, R², MACE) but does not measure downstream optimization performance for different embedder sizes. The paper assumes that better predictive metrics translate to better optimization, which is generally true in Bayesian Optimization but not guaranteed — factors like calibration quality and uncertainty estimation interact nonlinearly with acquisition function behavior. A direct optimization comparison between T5-Small and T5-XL on the same set of BBOB or HPO tasks would strengthen the claim that larger embedders yield better optimization. This is left to future work, though the positive scaling trend in prediction quality makes it plausible.

Effect of regressor depth on optimization: Similarly, Figure 8 shows that deeper regressors produce better predictions, but the paper does not evaluate whether 2-layer vs. 8-layer regressors produce different optimization performance. The 8-layer variant is used for all main experiments, and the paper does not report whether shallower regressors would suffice for competitive optimization (i.e., whether the predictive gains from depth actually matter for the downstream task).

Effect of string representation format: The paper uses JSON strings throughout ("{"p0":0.3,"p1":"category_1"}") but does not ablate alternative string formats — for example, key-value pairs separated by newlines ("p0:0.3\np1:category_1"), natural language descriptions ("The value of parameter p0 is 0.3 and p1 is category_1"), or tabular representations. The choice of JSON is motivated by its token efficiency and standardization, but the paper does not evaluate whether the embedder's regression-relevant properties (Lipschitz continuity, smoothness) depend on the string format. This is a significant gap because the framework's generality claim hinges on the embedder being format-agnostic; if performance degrades significantly with different string representations, the approach would be fragile to formatting choices.

Effect of the number of history trials on optimization: The ablation studies (Figures 7 and 8) show that more context points improve predictive metrics, but the paper does not ablate the number of history trials used during optimization. All main experiments presumably use the full observed history up to the current trial count, but the regressor was trained with contexts of 10–100+ trials. The optimization curves extend to 100–300 trials, and it is unclear whether the regressor's performance degrades when the history exceeds the training context length. The paper notes that the training always places $T \geq 100$ total trials in the context window, so the model may extrapolate poorly beyond this range — a potential concern for the combinatorial experiments running to 300 trials.

Effect of the projection layer: The paper mentions that the raw embedder output ($d = 4096$ for T5-XL) is projected to $d_{\text{model}} = 1024$ via a trainable linear layer, but does not ablate whether this projection is necessary or whether feeding the raw 4096-dimensional embeddings directly into the regressor (with appropriate dimension matching) would work as well or better. The projection serves both dimensionality reduction and learned feature selection; understanding which of these functions matters would inform model design.

Effect of UCB exploration coefficient $\sqrt{\beta}$: All experiments use $\sqrt{\beta} = 1.8$ (Appendix A), but the paper does not ablate this value. In principle, the optimal exploration-exploitation tradeoff should depend on the function landscape, the trial budget, and the regressor's calibration quality. The fact that a single fixed value works across synthetic, combinatorial, and HPO tasks suggests either that the regressor's uncertainties are well-calibrated across domains, or that optimization performance is robust to the choice of $\beta$. Without an ablation, it is impossible to distinguish these explanations or to determine whether task-specific tuning of $\beta$ would yield improvements.

Effect of the number of offline pretraining tasks: The paper uses 1M tasks for both BBOB and combinatorial pretraining. No ablation is provided varying the number of pretraining tasks (e.g., 100K, 500K, 1M, 5M) to assess whether the performance saturates or continues improving. The scaling trends in Figures 7 and 8 address model capacity but not data quantity; it is possible that 1M tasks are excessive and 100K would achieve similar performance, or that 5M tasks would yield further gains. This matters for practical adoption because generating 1M synthetic tasks with 100+ trials each is computationally expensive during pretraining.

Negative result: no combination of BBOB and combinatorial pretraining: The paper trains separate models for BBOB and combinatorial optimization (with 1M tasks each) but does not report results from a single model trained on both data sources jointly. The HPO experiments use the BBOB-only model, which means the paper never demonstrates a single set of regressor weights that simultaneously performs well on synthetic, combinatorial, and HPO tasks. The HPO transfer demonstrates generalization from BBOB to hyperparameter spaces, but it is unknown whether the same model that achieved competitive combinatorial results in Figure 4 would also achieve competitive HPO results in Figure 5, or whether pretraining on combinatorial data degrades synthetic/HPO performance due to catastrophic interference. This is a significant gap relative to the paper's vision of a "unified in-context regression model broadly over multiple different domains" (Section 5).

Critical Assessment

Claim from the executive summary: "Embed-then-regress achieves optimization performance comparable to state-of-the-art Gaussian Process-based methods such as Google Vizier."

This claim is supported by the BBOB results (Figure 3: performance comparable or better on 8 of 9 functions, significantly better on 2) and the HPO aggregate results (Figure 6: same median log-efficiency as GP-Bandit). However, several qualifications are necessary.

First, "comparable" means track-close-with-overlapping-error-bars on the majority of functions, which is genuinely impressive given that the regressor is a single pretrained model with no per-task adaptation. But "comparable" also means embed-then-regress does not consistently outperform GP-Bandit — on several BBOB functions (SharpRidge 5D, RosenbrockRotated 4D) GP-Bandit appears slightly better, and on the HPO individual curves (Figure 5) GP-Bandit holds small advantages on several tasks. The paper's claim is appropriately measured: it does not assert superiority, only parity.

Second, the comparison is against GP-Bandit, which is one specific instantiation of Gaussian Process-based Bayesian Optimization. GP-Bandit was reported by Song et al. (2024c) to be near-optimal among five industry methods, but other GP configurations — with different kernels (Matérn, spectral mixtures), different acquisition functions (Expected Improvement, entropy search), or different hyperparameter optimization strategies for the GP itself — might perform differently. The paper is comparing against a well-engineered but specific baseline, not against the theoretical ceiling of what GP-based methods can achieve. This is a reasonable choice for a clean comparison (since the paper swaps only the regressor while keeping the rest of the pipeline identical), but it means the claim should be interpreted as "comparable to a specific state-of-the-art GP system" rather than "comparable to the best possible GP-based approach."

Third, the HPO comparison uses a regressor trained exclusively on synthetic BBOB data. While this demonstrates impressive transfer, it also means the comparison is asymmetric: GP-Bandit benefits from being designed specifically for tabular hyperparameter spaces (with domain-appropriate kernels, length scale priors, etc.), while embed-then-regress is handicapped by having never seen any hyperparameter optimization data. The fact that embed-then-regress achieves parity under this handicap is a stronger result than if it had been trained on HPO data, but it also leaves open the question: would embed-then-regress trained on HPO data significantly outperform GP-Bandit? The paper does not answer this.

Claim: "Representing inputs as strings enables general-purpose regression across diverse domains."

This claim is well-supported by the breadth of evaluation: the same architecture (T5-XL embedder, 8-layer TNP regressor) and training methodology are applied to continuous/discrete tabular spaces (BBOB), permutation spaces (TSP, QAP, etc.), choice spaces (Coverage, Log Determinant), and real-world hyperparameter tuning (HPO surrogates). The regressor architecture and pretrained weights are identical for BBOB and HPO; the paper does not explicitly state whether the combinatorial model shares weights with the BBOB model, but the architecture is the same.

However, the claim of generality is tempered by several observations:

• All search spaces are ultimately represented as strings, but the string format is domain-dependent — JSON for tabular spaces, JSON-with-indices for permutations and choices. The embedder processes these uniformly, but the serialization scheme (what information goes into the string and how it is structured) is implicitly part of the method. The paper does not demonstrate robustness to alternative string representations for the same search space (e.g., representing a permutation as "[0]:2, [1]:0, [2]:3, [3]:1" vs. "[2, 0, 3, 1]" vs. a natural language description), limiting the evidence for "arbitrary string representations."

• The evaluation covers three major domain types (tabular, combinatorial, HPO) but within each, the tasks share structural similarities. All BBOB functions are continuous-like with added discretization; all permutation problems use the same evolutionary mutation operators; all HPO tasks are tabular hyperparameter spaces. The paper does not test on more exotic search spaces — variable-length sequences, tree-structured spaces, conditional/graph-structured parameters — that would more strongly test the "general-purpose" claim.

• The paper does not evaluate on search spaces where the correspondence between string representation and embedding geometry might break down, such as highly symbolic or code-like representations, or spaces where the "right" string representation is unclear (e.g., neural architecture search, where the search space is a computational graph).

Claim: "Our approach achieves optimization performance comparable to ... Google Vizier" (on combinatorial problems).

This claim requires careful interpretation. For combinatorial problems, the baseline is Regularized Evolution, not Vizier GP-Bandit. The paper explicitly notes that GP-based methods for permutations "require constructing very domain-specific kernels and complex acquisition optimizers ... making them difficult to reproduce" (Section 4.1). So embed-then-regress is not being compared against state-of-the-art Bayesian Optimization on combinatorial spaces; it is being compared against an evolutionary algorithm that was not designed to be an optimization benchmark but rather serves as the acquisition optimizer for embed-then-regress itself.

The appropriate framing is: embed-then-regress improves upon Regularized Evolution by adding UCB-guided selection, demonstrating that embedding-based regression provides exploration value in combinatorial spaces where GP-based BO is difficult to implement. This is a valid and useful result, but it does not establish that embed-then-regress is comparable to specialized combinatorial BO methods (Deshwal et al., 2022; Oh et al., 2022) — that comparison is simply not made. The paper's contribution on combinatorial spaces is better characterized as: demonstrating that a generic embedding-based regressor can augment evolutionary search in combinatorially structured spaces without any domain-specific modeling, not "matching state-of-the-art combinatorial BO."

Weaknesses in the experimental design:

• No formal statistical testing: All conclusions rely on visual inspection of mean curves with ±0.5 standard deviation error bars. While this is common in optimization benchmarking (where the number of seeds is typically small due to computational cost), the paper does not report confidence intervals, effect sizes, or hypothesis tests. The log-efficiency violin plot (Figure 6) is the closest to a formal aggregation, but even this does not include statistical tests comparing the distributions. The overlap between GP-Bandit and embed-then-regress distributions is visually clear, but a Kolmogorov-Smirnov or Mann-Whitney test would provide a more rigorous basis for the "same median" claim.

• Single model family for the embedder: All experiments use the T5 family (Small, Large, XL). The paper claims that T5 was chosen as "relatively smaller ... in comparison to the larger and significantly more expensive GPT or Gemini family of models" (Section 3.4), but this means the paper provides no evidence that the approach works with decoder-only models (GPT, LLaMA), models with different pretraining objectives (contrastive, autoregressive), or models from different architectural families. The T5 encoder uses bidirectional attention and was trained on a span-corruption objective; it is unknown whether the findings transfer to decoder-only architectures where token representations are computed autoregressively with causal masking.

• No evaluation on model-free optimization baselines: The paper uses Random, Quasi-Random, and Regularized Evolution as baselines, but does not compare against other model-free methods that are standard in combinatorial optimization — Simulated Annealing, Ant Colony Optimization, Genetic Algorithms with different selection mechanisms, or modern methods like particle swarm optimization. Regularized Evolution is a specific evolutionary algorithm; its performance relative to other model-free methods is not established in this paper.

• Fixed architecture across all experiments: The 8-layer Transformer with 1024-dimensional features and 16 heads is used for all main experiments. The paper does not evaluate whether different architectural choices (e.g., more layers, larger hidden dimension, different attention patterns) would improve performance, nor whether the optimal architecture depends on the domain (e.g., combinatorial tasks might benefit from different inductive biases than tabular tasks). The ablation in Figure 8 shows that performance improves with depth up to 8 layers, but does not explore whether even deeper models would continue improving or would overfit on the 1M synthetic tasks.

• No evaluation of inference time or wall-clock latency: The paper claims that inference efficiency is important ("the regressor may be called thousands of times ... per candidate proposal," Section 3.4) and that the chosen model sizes enable "1 GPU for inference," but it does not report actual wall-clock times for a full optimization run, nor does it compare inference latency between embed-then-regress and GP-Bandit. GP predictions scale as $\mathcal{O}(t^3)$ with the number of trials (due to Cholesky decomposition) while Transformer predictions scale as $\mathcal{O}(t)$ with efficient attention or $\mathcal{O}(t^2)$ with standard attention. At large trial counts (300+ for combinatorial tasks), this could be a significant practical advantage for the Transformer, but it is not quantified.

• Test set size and diversity: The BBOB test set includes 9 functions (from the 24-function suite), the combinatorial evaluation includes 8 specific problem instances with particular sizes, and the HPO evaluation includes 20 tasks. These are reasonable numbers for an initial validation study, but they limit the generalizability of the findings. For BBOB, the 9 test functions are explicitly selected to be from different landscape types than the training functions, but the test is still within the BBOB distribution. For combinatorial problems, the sizes tested are relatively small (permutations of size 9–14, choices of 7-choose-3 to 12-choose-4); it is unknown whether the approach scales gracefully to larger problem instances (e.g., permutation size 100 for TSP) where the embedding would need to encode much longer strings.

• No evaluation of the pretraining data's influence on optimization: The paper uses 1M BBOB tasks for pretraining but does not ablate the number of pretraining tasks, the diversity of transformations applied, or the distribution of search space dimensionalities. It is possible that the strong HPO transfer results depend critically on the specific BBOB transformations (random discretization, shifting, rotation) which were designed to create diverse and challenging search spaces, and that a different pretraining data distribution would transfer less effectively. The combinatorial results use 1M combinatorial tasks pretrained on randomized problem coefficients; no transfer from combinatorial pretraining to other domains is evaluated.

Missing experiments that would strengthen the paper:

• Joint training on BBOB + combinatorial + HPO data: The paper's vision of a "unified" regressor requires evidence that a single model can perform well across all three domains simultaneously. Currently, the BBOB model handles synthetic and HPO tasks; the combinatorial model handles combinatorial tasks. Training one model on all data sources and evaluating on all three benchmarks would directly test the unified regressor claim.

• Comparison against LLM-as-regressor baselines: The paper argues that text-to-text in-context regression (Liu et al., 2024; Vacareanu et al., 2024) is limited by context length and inability to pretrain, but never empirically compares against these methods. A comparison against, say, Gemini or GPT used as a direct regressor with natural language trial descriptions would quantify the benefit of the embedding approach over the pure text-to-text approach.

• Gradient-based acquisition optimization for continuous spaces: The paper uses zeroth-order Firefly for acquisition optimization but notes that "gradient-based acquisition maximization is possible with soft-prompt optimization techniques" (Section 3.3). Since the regressor is a differentiable Transformer, the acquisition function is differentiable with respect to the input embedding, and gradient-based optimization in embedding space followed by decoding to the nearest valid string could be more efficient than evolutionary search. An empirical comparison would reveal whether gradient-based acquisition optimization improves sample efficiency.

• Evaluation on search spaces with variable numbers of parameters: All BBOB and HPO tasks have fixed numbers of parameters within each task. The paper's string representation should naturally handle variable-length inputs (since strings can have arbitrary length), but this capability is never tested. An experiment where the regressor must predict across search spaces with different dimensionalities within the same task distribution would test the flexibility claim more strongly.

• Out-of-distribution search space dimensions: The BBOB training uses randomized transformations with dimensions presumably drawn from some distribution. The paper does not report the dimensionality range for training vs. test functions. If test functions have dimensionalities within the training distribution, the generalization is in-distribution; if they exceed the training range, the paper is testing extrapolation. This distinction matters for the transfer claim.

• Ablation of the y-normalization procedure: The three-step normalization (standardize → outlier mitigation → range normalization with damping) is described as optional but stabilizing. An ablation comparing this procedure to simple z-score normalization or to no normalization would reveal how much of the model's robustness to diverse objective scales depends on this preprocessing.

Summary of what the experiments demonstrate vs. what the paper claims:

The experiments convincingly demonstrate that a pretrained Transformer Neural Process using frozen T5-XL embeddings of JSON-serialized candidates can serve as the regressor in a Bayesian Optimization pipeline and achieve optimization performance that is competitive with a well-engineered Gaussian Process system on tabular search spaces, particularly when pretrained on large amounts of diverse synthetic data. The experiments further demonstrate that this approach can be applied to combinatorial search spaces and improve upon baseline evolutionary algorithms, and that the synthetic-pretrained model transfers effectively to real-world hyperparameter optimization tasks without domain-specific training.

What the experiments do not demonstrate: (1) that a single unified model works across all three domain types simultaneously; (2) that embed-then-regress is competitive against the best possible GP configurations or against specialized combinatorial BO methods; (3) that the approach scales to much larger combinatorial problems, exotic search space structures, or applications beyond black-box optimization (e.g., LLM reasoning reward models); (4) that the string representation format is truly arbitrary — the paper tested JSON only, and the serialization scheme implicitly encodes assumptions about which features are represented and how. These are directions for future work that the paper appropriately flags in Section 5, but they represent genuine limitations of the current experimental evidence.

6. Limitations and Trade-offs

6.1 Pretraining Data Distribution Mismatch: Synthetic-Only Training Limits Transfer to Real-World Tasks

The assumption or constraint. The paper's regressor for hyperparameter optimization is pretrained exclusively on synthetic BBOB trajectories — randomized mathematical functions with artificially introduced transformations (shifting, rotation, discretization). The authors explicitly state this: "we use the same regressor model trained only on synthetic BBOB trajectories as found previously in Figure 3" (Section 4.1, HPO section). No real hyperparameter optimization data, combinatorial data, or mixed-domain data is used during pretraining for the HPO experiments. The paper assumes that the statistical patterns learned from randomized mathematical landscapes transfer adequately to real-world optimization problems with fundamentally different structure — HPO objectives are typically non-stationary, may have threshold effects (e.g., learning rate cliffs), and involve parameter interactions that synthetic BBOB functions were not designed to capture.

The consequence. While the HPO results (Figures 5 and 6) demonstrate that transfer is possible — the BBOB-pretrained model achieves median log-efficiency equal to GP-Bandit — there is no evidence that this transfer is optimal or that it would hold across a broader range of real-world tasks. The 20 HPO tasks tested are all drawn from specific domains (image classification, hardware, production metrics), and the paper provides no characterization of how representative these are of the full distribution of real-world optimization problems. A practitioner deploying this method on, say, materials science experiments, drug discovery, or reinforcement learning hyperparameter tuning would have no guarantee that BBOB pretraining provides an adequate prior. Worse, the paper does not compare against a model trained on HPO data or a mixture of synthetic and real data, so it is impossible to know whether the synthetic-only approach leaves performance on the table relative to domain-matched pretraining. If a practitioner has access to even a modest amount of real-world optimization data (e.g., from previous tuning studies), should they use it for pretraining? Should they mix it with synthetic data? The paper provides no guidance.

What evidence exists in the paper. The HPO experiments (Figures 5 and 6) serve as the sole evidence for transfer. Figure 5 shows 8 individual HPO task curves where embed-then-regress is generally comparable to GP-Bandit, and Figure 6 shows the aggregate log-efficiency distribution across all 20 tasks with overlapping medians. However, these results only demonstrate that transfer worked for these specific 20 tasks; they do not establish that the transfer is robust to different task distributions. The paper also notes that the BBOB pretraining includes 1 million tasks with randomized transformations (Appendix B.1), which is an enormous amount of diverse data — but the diversity is entirely within the synthetic function family, not across real-world problem types. The combinatorial optimization results (Figure 4) use a separate model pretrained on 1 million combinatorial tasks, which means even the paper's own experiments do not demonstrate a single pretrained model working across all three domain types (tabular, combinatorial, HPO) simultaneously. The gap between "separate models per domain type" and "single unified model" is significant and unaddressed.

Mitigation status. The paper partially acknowledges this limitation through its framing: the HPO experiments are explicitly described as testing whether "the BBOB-trained model has learned a general regression capability which can be transferred to very different unseen tasks at inference time" (Section 4.1). The positive result is presented as a surprising discovery rather than an expected outcome. However, the paper does not discuss what properties of BBOB pretraining enable transfer, what types of real-world tasks might violate the transfer assumptions, or how practitioners should assess whether their target domain is "close enough" to BBOB for the approach to work. Section 5 mentions future work on pretraining "a unified in-context regression model broadly over multiple different domains including prompt optimization and code search" but does not address the fundamental question of what data mix is needed for robust cross-domain transfer.

6.2 Difficulty Estimation and Cold-Start Vulnerability: No Mechanism for Identifying When the Regressor Will Fail

The assumption or constraint. The embed-then-regress framework provides no mechanism for assessing a priori whether the pretrained regressor is suitable for a given optimization task, nor for detecting during optimization that the model's predictions are unreliable and the search should fall back to a safer strategy. The approach assumes that the embedding space preserves sufficient structure for regression across all tasks in the deployment distribution, and that the pretrained prior is adequate. When this assumption fails — for example, on search spaces where the embedder produces embeddings that are not Lipschitz continuous with respect to the true objective, or on objective functions with landscapes radically different from the pretraining distribution — the regressor may produce confidently wrong predictions that actively mislead the acquisition function.

The consequence. Bayesian Optimization regressors can exhibit pathological exploitation when their uncertainty estimates are miscalibrated: the acquisition function may repeatedly sample candidates that the regressor confidently (but incorrectly) predicts will be good, wasting the evaluation budget on unpromising regions while ignoring genuinely better parts of the search space. Unlike Gaussian Processes, which have well-understood failure modes (e.g., kernel misspecification leading to over-smoothing or over-fitting) and diagnostic tools (e.g., marginal likelihood for model selection), the Transformer Neural Process provides no built-in mechanism for detecting when its predictions are unreliable. The regressor always produces a mean and standard deviation — but whether those statistics are meaningful for a given task depends on whether the task is in-distribution. The paper's experimental results on the hardest BBOB functions provide indirect evidence of this vulnerability: functions that are fundamentally different from the training distribution might show flat or even degrading optimization performance, but the paper does not evaluate on tasks specifically chosen to stress the regressor's out-of-distribution behavior. In a production deployment, a practitioner might waste dozens or hundreds of expensive function evaluations before realizing the regressor is not providing useful guidance — and they would have no diagnostic signal to detect this early.

What evidence exists in the paper. The paper does not explicitly evaluate regressor failure modes or miscalibration on out-of-distribution tasks. The ablation studies (Figures 7 and 8) measure predictive metrics (NLL, MAE, R², MACE) on held-out BBOB test functions, which are in-distribution relative to the BBOB training data — they come from the same BBOB suite with the same transformation distribution, just different base functions. These metrics establish that the regressor is well-calibrated on BBOB-like tasks, but they do not assess calibration on HPO tasks (where the model was not trained) or entirely novel search space types. The HPO optimization results (Figures 5 and 6) provide indirect evidence that the model's predictions are useful enough for competitive optimization, but these are aggregate performance metrics — they do not reveal whether the model occasionally produces catastrophically bad predictions on particular tasks or at particular points in the optimization trajectory. The paper includes no analysis of per-task prediction quality, no correlation between predictive metrics and optimization performance, and no detection mechanism for when the model is out of its depth.

Mitigation status. Not addressed. The paper does not discuss out-of-distribution detection, uncertainty calibration diagnostics, or fallback strategies. The UCB acquisition function with fixed $\sqrt{\beta} = 1.8$ provides some robustness (if the model's uncertainties are inflated, exploration will dominate; if they are too narrow, exploitation may be overly aggressive), but $\beta$ is not tuned per-task and there is no adaptive mechanism. A practitioner concerned about regressor reliability would need to implement their own monitoring and fallback logic — for example, comparing the regressor's predictions against a simpler baseline (like a random forest or distance-weighted average) on held-out data within the optimization history, or reverting to random search if the regressor's predictive likelihood on recently observed trials degrades. None of this is provided or tested in the paper.

6.3 The String Representation Design Is Implicitly Domain-Specific and Not Empirically Validated

The assumption or constraint. A core claim of the paper is that representing inputs as strings enables general-purpose regression — that the embed-then-regress framework is flexible across arbitrary search spaces because "strings are significantly more flexible representation formats of different data types" (Section 5). However, the paper tests only one specific string format (JSON with a particular schema) and provides no evidence that the approach is robust to alternative string representations. The implicit assumption is that the T5 embedder's Lipschitz continuity properties (established for tabular features by Tang et al., 2025) transfer to arbitrary string formats, and that the JSON schema faithfully preserves the metric structure of the underlying search space.

The consequence. Without validation of alternative string representations, the "general-purpose" claim is an architectural aspiration rather than an empirically demonstrated property. A practitioner applying embed-then-regress to a novel search space — say, neural architecture specifications, chemical reaction conditions, or structured queries — must make design decisions about how to serialize candidates as strings. Should parameters be named or positional? Should categorical values use numeric codes or descriptive strings? Should the string include metadata about the search space? These choices could substantially affect the embedder's ability to produce meaningful representations, yet the paper provides no guidance and no sensitivity analysis. In the worst case, a poorly chosen string representation could break the Lipschitz continuity property — for example, if small changes in the underlying parameters produce large lexical changes in the string (due to tokenization artifacts, inconsistent ordering, or verbose natural language), the embedding geometry might not reflect the true search space geometry, and the regressor would be unable to learn meaningful patterns. Practitioners would discover this only through trial and error on their specific domain.

What evidence exists in the paper. The paper uses exactly one string format per domain type: JSON dictionaries for tabular spaces (e.g., {"p0":0.3,"p1":"category_1"}), JSON-with-bracket-indices for permutations (e.g., {"[0]":2,"[1]":0,"[2]":3,"[3]":1}), and JSON-with-bracket-indices for choices (e.g., {"[0]":1,"[1]":3}). Appendix C provides examples with optional metadata fields, but these are not used in the main experiments. The paper does not ablate: (1) natural language representations vs. JSON, (2) ordered vs. unordered JSON keys, (3) including vs. excluding parameter type information in the string, (4) representing categorical values as strings (e.g., "category_1") vs. integer codes (e.g., 1), or (5) the effect of string length and tokenization on embedding quality. The ablation studies (Figures 7 and 8) investigate embedder and regressor sizes, but assume a fixed string representation throughout. The Lipschitz continuity property from Tang et al. (2025) is cited as theoretical motivation, but the paper does not empirically measure whether the JSON-over-tabular-features representation actually satisfies Lipschitz continuity in the T5 embedding space, nor whether the JSON-over-permutations or JSON-over-choices representations preserve meaningful geometric structure.

Mitigation status. Partially acknowledged. Section 5 mentions that "the aggregation method over Transformer outputs may be learned rather than predefined using average pooling" and that "further investigation into different regression bodies could lead to e.g. string-based GPs using kernels over string embeddings," but these are suggestions about the regression architecture, not about string representation design. The paper does not explicitly call for future work on validating alternative string formats or developing principles for string representation design. The practical implication — that a practitioner must design the string serialization and hope it works — is not discussed.

6.4 Computational Overhead of the Embedding Step: A Constant-Factor Cost Multiplier Not Accounted for in Efficiency Claims

The assumption or constraint. The embed-then-regress framework adds a constant-factor computational overhead relative to methods that work directly with raw feature vectors: every candidate scored by the regressor must first be serialized to a string, tokenized, and passed through a T5-XL encoder (1 billion parameters) before the regression Transformer can process it. The paper acknowledges that "faster embedders lead to large constant factor reductions" in overall cost (Section 3.4) and that "the cheap inference cost is also necessary when the acquisition function may be called thousands of times by a zeroth-order acquisition optimizer per candidate proposal" (Section 3.4), but it does not quantify this overhead or include it in any efficiency comparison against GP-Bandit.

The consequence. For tabular search spaces — where GP methods operate directly on the raw parameter vectors — the embedding step is pure overhead: the GP can compute its kernel function in microseconds, while the embed-then-regress pipeline must tokenize a JSON string, run a forward pass through a 1B-parameter encoder, average-pool the output, and project to the regressor dimension before the regression Transformer can even begin. The paper reports using the Firefly acquisition optimizer with a maximum budget of 1,000 evaluations per candidate proposal (Appendix A), meaning that for each new trial proposed, up to 1,000 candidates must be scored. With a typical optimization run of 100–200 trials, this amounts to 100,000–200,000 embedding forward passes — each through a 1B-parameter model. While the paper claims inference is possible on "1 GPU" (Section 3.4), the wall-clock time for 100,000 T5-XL encoder passes (even with batching) could be substantial — potentially minutes per optimization step, making the approach impractical for interactive or time-sensitive optimization. The comparison against GP-Bandit, which does not incur this embedding cost, is therefore not a fair wall-clock-time comparison, even though it is fair in terms of function evaluations (which is the standard metric in BO literature). A practitioner choosing between GP-Bandit and embed-then-regress for tabular optimization would need to weigh this constant-factor latency increase against the flexibility benefits, but the paper provides no data to inform this decision.

What evidence exists in the paper. The paper provides hardware requirements (approximately 16 GPUs for training, 1 GPU for inference, Section 3.4) and model sizes (T5-XL with 1B parameters, 8-layer Transformer regressor, Appendix A), but no wall-clock timing measurements, no latency comparisons against GP-Bandit, and no throughput analysis. The ablation in Figure 7 shows that smaller T5 variants (Small, Large) produce worse predictions, establishing that the large embedder is necessary for competitive performance — so practitioners cannot simply downsize the embedder to reduce latency without sacrificing regression quality. The paper also does not evaluate whether the embedding computation can be amortized across candidates (e.g., by caching embeddings for previously scored candidates across acquisition optimizer iterations) or whether batched embedding computation on GPU significantly reduces per-candidate cost. The claim that inference is "possible with most academic budgets" (Section 3.4) refers to hardware availability, not to practical runtime for a full optimization loop.

Mitigation status. The paper acknowledges the cost implicitly by noting that "faster embedders lead to large constant factor reductions" (Section 3.4) and that efficient Transformer variants could reduce the regressor's $\mathcal{O}(t)$ complexity, but the embedder cost itself is not addressed. There is no discussion of techniques to reduce embedding overhead — for example, distilling the T5-XL encoder into a smaller model, pre-computing embeddings for a grid of candidates, or using cached embeddings from similar previously-evaluated candidates. The paper also does not consider the possibility that for tabular spaces specifically, the embedding step could be skipped entirely (using raw feature vectors directly with the same Transformer regressor, which would require a different architecture but avoid the embedder cost) — this would provide a more relevant baseline for tabular optimization but is not explored.

6.5 Single Model Family and Single Benchmark Bias: No Evidence That Results Generalize Beyond T5 and MATH-Like Search Spaces

The assumption or constraint. All experiments use the T5 family of encoder-decoder language models (Small, Large, XL) as the frozen embedder, with embeddings computed via average pooling over the encoder's output tokens. The paper selects T5 because it is "relatively smaller ... in comparison to the larger and significantly more expensive GPT or Gemini family of models" (Section 3.4), but this choice means the entire empirical evaluation is conditioned on one specific model architecture, pretraining objective (span corruption on C4 text corpus), and embedding extraction method. The paper assumes that the key property enabling effective regression — Lipschitz continuity of embeddings over tabular features — is a general property of LLMs, not specific to T5's encoder-decoder architecture or span-corruption pretraining.

The consequence. A practitioner who wants to use embed-then-regress with a different embedder — for example, a decoder-only model like LLaMA or GPT (which are more widely available and have stronger ecosystems), a contrastively trained embedding model, or a domain-specific encoder — has no evidence that the approach will work. Decoder-only models compute token representations autoregressively with causal masking, meaning the representation of early tokens depends only on preceding context, while T5's encoder uses bidirectional attention that can incorporate information from the entire string. This architectural difference could affect the geometric properties of the embedding space in ways that matter for regression. Similarly, models pretrained on code rather than natural language might produce embeddings with different continuity properties over structured data. The ablation in Figure 7 shows that larger T5 variants produce better regression, but this trend may not hold for other model families — it is possible that a smaller model from a different family with a different pretraining objective could outperform T5-XL, or that a larger decoder-only model could underperform T5-Small due to fundamental architectural differences in how embeddings are computed. Without cross-family comparisons, the paper's claim that "LLM embeddings can be sufficient for Bayesian Optimization" is more accurately stated as "T5 encoder embeddings with average pooling can be sufficient."

What evidence exists in the paper. The paper uses exclusively T5 variants for the embedder (Figure 7) and provides no comparison against decoder-only models (GPT, LLaMA, Gemini), contrastively trained models, or non-LLM string embedding methods. The link to Tang et al. (2025) provides theoretical motivation for Lipschitz continuity, but that work also studied specific encoder architectures; the paper does not cite evidence that this property generalizes across model families. The BBOB, combinatorial, and HPO benchmarks all involve search spaces that can be serialized as relatively short, structured strings (JSON with numeric and categorical values). The paper does not test on search spaces requiring longer, more complex string representations (e.g., full configuration files, code snippets, natural language descriptions), where tokenization effects and the embedder's context-window limits might matter more. The combinatorial experiments use permutation sizes up to 14 and choice sizes up to 12-choose-4, which produce relatively short strings; the approach's behavior on larger instances (permutation size 100, choice size 50-choose-10) where the JSON string would be much longer (potentially exceeding the 400-token limit mentioned in Appendix A) is unknown.

Mitigation status. The paper frames the use of T5 as a deliberate choice for accessibility ("possible with most academic budgets," Section 3.4) and explicitly states that larger embedders from the GPT or Gemini families would be "larger and significantly more expensive" — implying that the current results represent a lower bound and that scaling up the embedder would yield improvements. However, this framing conflates model size with model family: it assumes that "larger GPT" would provide better embeddings than "larger T5" without accounting for architectural differences. Section 5 suggests future work on "universal in-context regression" but does not specifically call for evaluation across embedder architectures. The paper also does not evaluate whether the 400-token string limit (Appendix A) was ever reached in practice or how performance degrades when strings are truncated.

6.6 The Method Provides No Path Forward for Hard Problems Where the Base Model's Capability Is Insufficient

The assumption or constraint. The embed-then-regress framework assumes that the search space can be meaningfully embedded into a vector space where the regression Transformer can learn to interpolate between observed trials. This assumption breaks down when the search space contains fundamentally discrete or non-smooth structure that cannot be preserved through the string → tokenization → embedding pipeline, or when the objective function exhibits behavior (e.g., extreme non-stationarity, chaotic sensitivity to parameters, phase transitions) that the regressor's meta-learned prior does not capture. More fundamentally, the approach inherits a limitation common to all surrogate-based optimization: if the regressor cannot learn a useful model of the objective function given the observed data, the optimization degrades to random search. The paper's results on synthetic and HPO tasks demonstrate competitive performance, but do not reveal which types of problems cause the approach to fail relative to GP-based methods or other alternatives.

The consequence. A practitioner deploying embed-then-regress encounters two risks. First, on problems where the embedding geometry is fundamentally misaligned with the objective function's structure, the method may waste the initial evaluation budget producing confidently wrong predictions (as discussed in limitation 6.2) and then fail to recover, ending up worse than random search or a simple evolutionary algorithm. Second, on problems where the meta-learned prior is inappropriate — for example, objectives with sharp discontinuities that the BBOB-pretrained model has never encountered, or objectives defined over search spaces with conditional/hierarchical parameter structures that JSON serialization flattens in ways that mislead the embedder — the regressor may produce predictions that are systematically biased, causing the acquisition function to consistently favor unpromising regions. The paper provides no characterization of the problem classes where embed-then-regress is likely to outperform, match, or underperform GP-based methods, making it difficult for practitioners to assess whether their specific use case is suitable.

What evidence exists in the paper. The paper's main experiments show that embed-then-regress is competitive with GP-Bandit on the tested benchmarks, but these benchmarks are selected to be favorable to regression-based methods — they involve relatively smooth objective functions (BBOB), structured combinatorial spaces with clear distance metrics (permutations, subsets), and real-world HPO tasks that are known to be amenable to GP-based BO. The paper does not include "adversarial" test cases designed to stress the embedding approach: for example, objective functions defined directly on the string representation (where the embedder's internal geometry is the ground truth rather than a proxy), search spaces with deliberately misleading JSON serializations (where lexically similar strings map to semantically different candidates), or objectives with extreme sensitivity to specific parameter combinations (where the smoothness assumptions of both GPs and Transformer regressors break down). The combinatorial results (Figure 4) show consistent improvement over Regularized Evolution, but the absolute performance gains are modest on several problems (e.g., Quadratic Assignment shows only a ~3% relative improvement) — it is unclear whether a domain-specific combinatorial BO method (Deshwal et al., 2022; Oh et al., 2022) would substantially outperform embed-then-regress on these problems. Without head-to-head comparisons against specialized methods, the paper cannot distinguish between "embed-then-regress is competitive with the best available methods" and "embed-then-regress is better than naive baselines but far from the specialized state-of-the-art."

Mitigation status. The paper does not directly address this limitation. The framing emphasizes flexibility and generality over peak performance: "we focus on demonstrating the general applicability of LLM embeddings across a variety of tasks rather than achieving the best possible result against very domain-specific baselines" (Section 4.1). This is a reasonable scope for an initial validation study, but it means the paper does not provide the information a practitioner needs to decide whether to adopt the method for a specific high-stakes optimization problem versus investing in domain-specific BO engineering. Section 5 suggests future work on combining embeddings with "different regression bodies" and extending to "prompt optimization and code search," but does not propose systematic studies of failure modes or problem characteristics that predict embedding-based regression success.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not propose a new algorithm, architecture, or theoretical framework — it demonstrates that a simple design principle (frozen LLM embeddings + standard in-context regression) is already sufficient to match industry-grade Gaussian Process systems across diverse optimization domains. This is a methodological contribution, not a theoretical one, but its implications for how the field thinks about regression in Bayesian Optimization are substantial.

The primary shift is decoupling representation from regression in a way that enables amortization. Before this work, the dominant mental model for BO was: choose a regressor appropriate to your search space (GP with a suitable kernel for continuous spaces, domain-specific kernels for combinatorial spaces, or a custom embedding for exotic spaces), configure its hyperparameters, and run the optimization. Each new search space required a new regressor design. The embed-then-regress framework shows that a single representation function — a frozen LLM embedder — can map any string-serialized search space to a geometry where a single pretrained regressor works across continuous, discrete, categorical, permutation, and choice-based spaces without modification. This changes the question from "how do I build a regressor for my search space?" to "how do I serialize my search space so the embedder captures its structure?" — and the paper's results suggest that even naive JSON serialization is often sufficient.

This is not a paradigm shift on the order of the GP-to-Transformer transition — GPs remain competitive and well-understood, and the paper shows parity, not dominance. But it is a reframing of the regression problem in BO from "design" to "amortize." If a single pretrained regressor can handle the range of search spaces tested here, then the engineering effort shifts from per-task kernel design to one-time pretraining of a general-purpose model, analogous to how foundation models in NLP and vision shifted effort from task-specific architecture design to pretraining + adaptation. The paper explicitly positions itself as a foundational step toward this vision (Section 5: "a 'universal' in-context regressor which can speed up search over evolutionary algorithms"), and the empirical evidence — particularly the BBOB-to-HPO transfer in Figures 5 and 6 — provides the first concrete validation that this direction is viable.

The work also reconciles a tension in the literature between flexibility and performance in learned BO regressors. Prior approaches faced a tradeoff: text-to-text LLM regression (Liu et al., 2024; Vacareanu et al., 2024) offered maximal flexibility (any string input) but suffered from context-length limits blocking long optimization histories and from the inability to benefit from offline pretraining. Custom-tokenized Transformer regressors (Chen et al., 2022) offered pretrainability and long histories but were brittle to search space structure. Single-trial embedding methods (Song et al., 2024a; Akhauri et al., 2025) offered flexibility and pretrainability but required inference-time fine-tuning to incorporate history. The embed-then-regress framework resolves this tension by showing that all four desiderata — flexibility, pretrainability, long in-context history, and efficient inference — can be satisfied simultaneously with a frozen embedder and a modestly-sized Transformer regressor. The key architectural insight is that the embedder compresses each trial to a single token, sidestepping the context-length bottleneck that plagued text-to-text approaches.

Several research directions become more attractive in light of these results:

• Scaling up the embedder becomes an obvious path to improvement. The ablation in Figure 7 shows monotonic gains from T5-Small to T5-XL, with no sign of saturation. Using GPT-scale or Gemini-scale embedders — which the paper explicitly avoids for cost reasons — would likely yield further improvements. The implication is that progress in LLM embedding quality (driven by the NLP community for unrelated reasons) will automatically improve embedding-based BO with no changes to the regression architecture.

• Pretraining data engineering emerges as the critical lever. The paper's most surprising result — that BBOB-only pretraining transfers to HPO — suggests that the distribution of pretraining tasks may matter less than their diversity and scale. Generating 1M synthetic tasks is feasible for any research group (the paper used BBOB with random transformations), meaning the data bottleneck for learned BO is largely artificial.

• Domain-specific kernel design for BO becomes less attractive as a research direction, at least for the types of structured spaces tested here (permutations, choices). If frozen LLM embeddings provide sufficient structure for competitive optimization on these spaces, the marginal benefit of hand-designing kernels for each new combinatorial structure — which requires deep expertise and is difficult to reproduce (as the paper notes for Deshwal et al., 2022; Oh et al., 2022) — is harder to justify. Research effort is better spent on improving the general-purpose regression pipeline than on specialized kernels.

• Inference-time fine-tuning approaches (Song et al., 2024a) become harder to motivate, since the embed-then-regress framework achieves competitive performance with pure in-context regression and no per-task weight updates. The "tedious" fine-tuning step (as characterized by the paper) adds latency and hyperparameter complexity that may be unnecessary given a sufficiently well-pretrained regressor.

However, the work also clarifies the boundaries of what test-time compute can substitute for, in a sense analogous to the pretraining-vs-inference tradeoff in LLMs. The regressor here is pretrained once and deployed frozen; it cannot acquire fundamentally new regression capabilities at test time. For search spaces or objective functions that fall outside the pretraining distribution, the approach offers no mechanism for adaptation beyond what in-context learning provides. The paper's HPO transfer results suggest this in-context capacity is surprisingly broad, but there will inevitably be problem classes where the pretrained prior is misaligned, and the paper provides no diagnostic for detecting this. This is a genuine limitation that bounds the "universal regressor" vision.

Follow-Up Research This Work Enables

Joint pretraining on synthetic, combinatorial, and real-world HPO data to produce a single unified regressor. The paper trains separate models for BBOB and combinatorial optimization, and uses the BBOB model for HPO transfer. The natural next step is to train a single regressor on all three data sources simultaneously (1M BBOB tasks + 1M combinatorial tasks + whatever HPO data is available) and evaluate on all three benchmark families. The key question is whether the regressor can share representations across such different search space structures — or whether interference between domains degrades performance below the separate-model baseline. A strong result would show that the unified model matches or exceeds the specialized models on each domain, demonstrating positive transfer. A negative result (e.g., combinatorial training hurts BBOB performance) would reveal important limits on the universality claim and suggest that separate models per broad domain type are necessary.

Robustness to string representation format — does JSON matter, or does any reasonable serialization work? The paper uses a single JSON schema per domain type but never ablates the string format. A critical stress-test would compare optimization performance across multiple string representations of the same search space: JSON with named parameters vs. positional arrays vs. natural language descriptions vs. key-value text files. If performance is robust across formats, the "arbitrary string" claim is validated; if it degrades significantly for some formats, then string representation design becomes a crucial engineering step that the paper currently does not address. The experiment would use the same pretrained regressor and embedder, varying only the string serialization, on a subset of BBOB and HPO tasks. Additionally, measuring the Lipschitz constant of the embedding function for each format would ground the empirical results in the theoretical justification from Tang et al. (2025) — do formats that produce lower Lipschitz constants (smoother embedding spaces) yield better optimization?

Direct comparison against domain-specific combinatorial BO methods (Deshwal et al., 2022; Oh et al., 2022). The paper's combinatorial experiments compare only against Regularized Evolution, not against the specialized GP-based methods with permutation kernels and custom acquisition optimizers. A head-to-head comparison on standard permutation benchmarks (TSP, QAP, Flowshop Scheduling at sizes used in the combinatorial BO literature) would establish whether embed-then-regress is genuinely competitive with the specialized state-of-the-art or merely better than a naive evolutionary baseline. Given the difficulty of reproducing the specialized methods (which the paper itself notes), this comparison would be high-value even if embed-then-regress underperforms — it would quantify the "generality tax" (how much performance is sacrificed for flexibility) and identify problem characteristics where domain-specific modeling is worth the engineering investment.

Dynamic difficulty estimation and acquisition function adaptation. The paper uses a fixed UCB exploration coefficient (\sqrt{\beta} = 1.8) across all tasks, which implicitly assumes the regressor's uncertainty calibration is consistent across domains. A valuable extension would measure the regressor's calibration quality online during optimization (e.g., by tracking the empirical coverage of the predictive distributions on newly observed trials) and adaptively tune \beta or switch acquisition functions (UCB vs. Expected Improvement vs. Thompson Sampling) based on calibration diagnostics. This would address the cold-start vulnerability discussed in limitation 6.2 and could be implemented with no changes to the pretrained regressor — only the acquisition logic would be modified. The experiment would compare adaptive \beta tuning against the fixed 1.8 baseline on the same BBOB and HPO benchmarks, measuring both final optimization performance and the frequency of catastrophic prediction failures.

Gradient-based acquisition optimization in embedding space. The paper uses zeroth-order evolutionary optimizers (Firefly for tabular spaces, Regularized Evolution for combinatorial spaces) to maximize the UCB acquisition function, noting that gradient-based optimization "is possible with soft-prompt optimization techniques" (Section 3.3) but not evaluating it. Since the regressor is a differentiable Transformer, the acquisition function a(x) = \mu(x) + \sqrt{\beta} \cdot \sigma(x) is differentiable with respect to the embedding x, and gradients could be used to guide the search for high-acquisition candidates. The natural experiment is to compare: (1) zeroth-order Firefly (current approach), (2) gradient ascent in embedding space with nearest-neighbor decoding to valid candidates (via a library of pre-computed embeddings or a learned decoder), and (3) hybrid approaches that use gradients to propose candidates which are then refined by evolutionary search. This would test whether the differentiability of the learned regressor — a capability that GPs also have but that is rarely exploited in combinatorial spaces — provides a meaningful advantage for acquisition optimization.

Scaling laws for pretraining data volume and diversity. The paper uses 1M tasks for both BBOB and combinatorial pretraining but never ablates this number. A systematic study varying the number of pretraining tasks (10K, 100K, 500K, 1M, 5M) and the diversity of transformations (number of discrete parameter values, number of dimensions, types of landscapes included) would establish whether the current results are near saturation or whether substantially more data would yield further gains. This is the analog of scaling laws for foundation models (Kaplan et al., 2020) applied to regression meta-learning, and would provide practical guidance for practitioners on how much pretraining data to generate. The key metrics would be downstream optimization performance (on held-out BBOB and HPO tasks) as a function of pretraining data scale, with separate curves for different regressor sizes to assess whether larger models benefit more from more data.

Practical Applications and Downstream Use Cases

Cost-efficient hyperparameter tuning for organizations without BO expertise. The paper demonstrates that a single pretrained regressor, deployed frozen, matches Google Vizier's GP-Bandit on 20 diverse HPO tasks (Figure 6). For a mid-sized ML team that tunes models regularly but lacks the expertise to configure GP kernels, length-scale priors, and acquisition hyperparameters per task, the embed-then-regress model could be served as a drop-in regressor: serialize the hyperparameter space as JSON, point the pretrained model at it, and run BO with no per-task configuration. The model's BBOB-only pretraining means the team doesn't need to collect their own historical tuning data — the synthetic pretraining transfers. The compute cost (single GPU inference) is within reach of any cloud budget, and the optimization performance matches an industry system that required years of engineering. The primary adoption barrier is the string serialization step (which hyperparameters to include, how to represent categorical values), but the paper's JSON format is straightforward to implement for any tabular search space.

Augmenting evolutionary algorithms with exploration bonuses in combinatorial optimization. The paper's combinatorial results (Figure 4) show that UCB-guided selection consistently improves upon Regularized Evolution across all 8 tested problems, including TSP, QAP, and N-Queens. This requires no domain-specific modeling — the same pretrained regressor and embedder used for BBOB are applied directly, with Regularized Evolution serving as the proposal mechanism and the regressor providing exploration guidance. For operations research practitioners who currently use evolutionary algorithms, simulated annealing, or other model-free methods for combinatorial optimization, adding the embed-then-regress UCB filter is a lightweight bolt-on improvement: the evolutionary algorithm generates candidates as before, and the regressor ranks them by predicted performance plus uncertainty bonus rather than by raw fitness. The paper's best-of-5 sampling approach (evaluating only 5 candidates per step) keeps the additional computational cost negligible relative to the true objective evaluations.

Pretrained regression models as a service for black-box optimization platforms. The paper's architecture — frozen embedder serving multiple downstream regressors, or a single regressor serving multiple optimization domains — maps naturally to a model-as-a-service deployment. An optimization platform (analogous to Google Vizier, Optuna, or Ax) could host a pretrained embed-then-regress model and expose it as a general-purpose regressor backend. Users would provide their search space definition and objective function; the platform would handle string serialization, embedding, and UCB scoring transparently. The key value proposition is eliminating the regression modeling expertise currently required to use BO effectively: no kernel selection, no GP hyperparameter tuning, no feature engineering. The paper's results suggest this is viable today for tabular, permutation, and choice-based spaces — covering a large fraction of practical optimization use cases. The main engineering challenge would be scaling the embedder inference to handle multiple concurrent optimization clients (each requiring thousands of embedding forward passes per trial), which the paper's single-GPU inference claim suggests is manageable with batching.

Rapid prototyping of Bayesian Optimization on novel search spaces. For research groups exploring optimization on new or exotic search spaces (e.g., chemical reaction conditions represented as SMILES strings, neural architecture specifications represented as JSON graphs, or prompt templates for LLMs), the traditional BO pipeline requires either designing a domain-specific kernel (months of work) or falling back to model-free methods (random/evolutionary search). The embed-then-regress framework provides a third option: serialize the search space as strings in whatever format is natural, use a frozen off-the-shelf LLM embedder, and apply the pretrained regressor from this paper (or train a similar one on synthetic BBOB-like data) with no domain-specific adaptation. The paper's demonstration that synthetic pretraining transfers to real HPO tasks (Figures 5, 6) and that the same architecture handles permutations and choices (Figure 4) suggests this is likely to work for many novel spaces without requiring the research group to collect optimization data first. This lowers the barrier to entry for using BO in new domains from "requires kernel design expertise" to "requires identifying a reasonable string serialization."

When to Prefer This Method

The paper articulates a clear tradeoff between flexibility and peak performance: "we focus on demonstrating the general applicability of LLM embeddings across a variety of tasks rather than achieving the best possible result against very domain-specific baselines" (Section 4.1). This implies a decision rule that the paper supports empirically:

• Prefer embed-then-regress when you operate across diverse search space types (continuous, discrete, categorical, permutations, choices) and want a single regression model without per-domain customization. The paper demonstrates this works for BBOB synthetic functions (Figure 3), combinatorial optimization (Figure 4), and HPO surrogates (Figure 5, 6) using the same architecture and, for BBOB-to-HPO transfer, the same model weights. The approach is also preferable when you lack domain-specific optimization data for pretraining but can generate synthetic data (BBOB-style randomized functions) — the synthetic-to-real transfer results suggest this is viable.

• Prefer embed-then-regress when you need to deploy BO in a setting where GP expertise is unavailable. Configuring GP kernels, length-scale priors, and acquisition hyperparameters requires specialized knowledge; the pretrained regressor works out of the box with a fixed UCB coefficient (\sqrt{\beta} = 1.8) and no per-task tuning.

• Prefer domain-specific GP-based methods when you are optimizing on a single well-understood search space type (e.g., purely continuous hyperparameters) and peak performance matters more than flexibility. GP-Bandit holds a slight advantage on several BBOB functions (SharpRidge 5D, RosenbrockRotated 4D in Figure 3) and several HPO tasks (Figure 5), and GPs provide well-understood uncertainty diagnostics that the Transformer regressor currently lacks.

• Prefer specialized combinatorial BO methods (Deshwal et al., 2022; Oh et al., 2022) when you need the best possible performance on a specific combinatorial structure (e.g., large-scale permutation optimization) and can invest in implementing domain-specific kernels and acquisition optimizers. The paper does not compare against these methods, so the performance gap is unknown, but the specialized methods leverage structure that the embedding approach ignores.

The paper does not provide sufficient evidence to choose between embed-then-regress and LLM-as-regressor approaches (Liu et al., 2024; Vacareanu et al., 2024) — it argues for the advantages of pretrainability and context-length efficiency but never compares them empirically. A practitioner choosing between these approaches would need to weigh the embed-then-regress requirement for pretraining (significant offline compute, but amortized) against the LLM-as-regressor requirement for a large proprietary LLM API (per-query cost, no pretraining benefit, context-length limits on history size).