Yi: Open Foundation Models by 01.AI

🎯 Pitch

When inferring a weighted sum from examples, only massive models like LLaMA2 70B achieve accurate predictions—and only when the function has few coefficients. With five coefficients, exact-match performance abruptly collapses for smaller models, revealing that in-context learning of complex functions is an emergent ability tied to scale.

1. Executive Summary

This paper studies how large language models infer the underlying functional relationship from in-context examples, operationalized as learning the linear coefficients of a weighted sum from demonstration sequences. The core contribution is a systematic evaluation of in-context learning capability across model scales—ranging from Yi-34B to LLaMA2 70B and Mixtral 8x7B—under both a continuous error metric (difference to target) and a discontinuous metric (exact match), with the number of linear coefficients systematically varied to probe complexity scaling. When the function is simple (two coefficients), Yi-34B achieves the lowest prediction error, but when complexity increases to five coefficients, only the largest models (LLaMA2 70B and Mixtral 8x7B) achieve meaningful exact match performance, establishing that in-context learning of complex functions is an emergent ability that manifests only when models surpass a critical scale threshold.

2. Context and Motivation

The Core Problem: We Don't Know How Model Scale Relates to In-Context Learning of Complex Functions

The fundamental question this paper tackles is deceptively simple: when large language models learn from in-context examples, what happens to that capability as the function being learned becomes more complex, and how does this interact with model scale? In-context learning (ICL)—the ability of LLMs to infer patterns from a handful of demonstrations provided in the prompt and apply those patterns to new inputs—is one of the most striking and practically important behaviors of modern language models. Yet, despite extensive study of ICL for natural language tasks (translation, classification, question answering), we lack a systematic understanding of how this capability scales when the underlying function is purely mathematical and its complexity can be precisely controlled.

This gap matters for several reasons that the paper implicitly raises through its experimental design:

Scientific understanding of emergent abilities: The debate over whether emergent abilities in LLMs are real or measurement artifacts (citing Schaeffer et al., 2024) creates a need for experimental paradigms that can cleanly distinguish genuine capability emergence from metric-induced illusions. By using both a continuous metric (prediction error) and a discontinuous metric (exact match) on the same task, this paper provides a framework for directly testing that debate.
Understanding the limits of in-context learning: If ICL is to be relied upon for deploying LLMs in novel domains without fine-tuning, we need to know what kinds of functions models can and cannot learn from examples alone. A weighted-sum task with parameterizable complexity provides a clean testbed: we know the ground-truth function, so we can precisely measure how close the model's inferred function is to reality.
Guiding model selection and deployment: Organizations deciding which model size to deploy for tasks requiring in-context reasoning need evidence about whether smaller models can handle complex ICL tasks or whether larger models are genuinely necessary. This paper's findings—that Yi-34B is best on simple functions but only 70B+ models handle complex ones—provide actionable guidance.

The Emergent Ability Debate Creates a Measurement Crisis

The paper explicitly positions itself within the ongoing discussion sparked by Schaeffer et al. (2024) about whether emergent abilities are fundamental properties of scaling or artifacts of how we measure performance. The concern is this: if you use a discontinuous metric like exact match (did the model get the answer exactly right?), you might observe an apparent "emergence" when performance jumps from near-zero to above-chance at some scale. But this could simply be an artifact of thresholding a smoothly improving continuous capability. A model might be getting progressively better at a task (its internal representations are improving continuously with scale), but exact match would report zero improvement until the model crosses the threshold of getting answers perfectly correct.

The paper addresses this directly by the dual-metric design: the continuous difference-to-target metric captures any improvement in the model's ability to approximate the function, while exact match captures the discontinuous threshold of perfect inference. If emergent abilities are purely measurement artifacts, then the continuous metric should improve smoothly with scale even as exact match shows a jump. If they are genuine, then both metrics should show that some capability dimension only appears above a critical scale. The paper's finding that only the largest models achieve meaningful exact match on 5-coefficient tasks—while the continuous metric presumably shows progressive improvement (though the exact data is not shown in the provided snippet)—has direct bearing on this debate.

Prior Approaches and Where They Fall Short

The paper does not extensively cite prior ICL literature in the provided snippet, but we can infer the landscape from its experimental choices and the debate it engages:

Natural language ICL benchmarks conflate multiple skills. Most prior work on ICL evaluates on natural language tasks—sentiment classification, translation, question answering—where the "function" being learned from examples is a mixture of linguistic pattern recognition, factual recall, and reasoning. In these settings, it is impossible to isolate the model's pure function inference capability from its language understanding, its memorized knowledge, and its ability to reason about the specific domain. A model might succeed at ICL for sentiment analysis because it already understands sentiment from pretraining, not because it inferred the sentiment function from the provided examples. The weighted-sum task strips away domain knowledge: the model must infer a mathematical relationship from numerical examples, with no possibility of relying on memorized patterns from training data.

Scaling studies of ICL have focused on natural language metrics. While prior work has studied how ICL performance varies with model size (and found, for example, that larger models are better few-shot learners), these studies use natural language task accuracy as the metric. The Schaeffer et al. (2024) critique suggests that task accuracy—particularly when binarized into correct/incorrect—may not reveal the true underlying scaling relationship. A systematic scaling study using a controlled mathematical function with both continuous and discontinuous metrics has, to this paper's apparent contribution, not been done before.

Complexity-controlled function learning is understudied. The specific choice of a linear weighted sum as the target function is deliberate and reveals a gap in prior work. A weighted sum is parameterizable: we can vary the number of coefficients (the dimensionality of the function being learned) to precisely control the complexity of the inference problem. With one coefficient, the model needs to infer a single scalar multiplier. With five coefficients, it must infer a 5-dimensional linear function from examples—a substantially harder credit assignment problem. Prior work on ICL has not systematically varied the complexity of the to-be-learned function while holding all else constant, leaving the complexity-scaling relationship unexplored.

The "scaling laws of ICL" are incomplete without complexity as a variable. The broader scaling laws literature (Kaplan et al., 2020; Hoffmann et al., 2022) characterizes how model performance improves with compute, parameters, and data—but treats the task as fixed. This paper's experimental design implies that task complexity is a critical missing variable: scaling behavior depends not just on model size but on how complex the ICL problem is. A model that appears to have "saturated" ICL capability on simple tasks may reveal untapped potential when tested on more complex functions, and conversely, a model that excels on complex ICL may be overkill for simple ones.

How This Paper Positions Itself

The paper positions itself as a measurement-focused contribution to the emergent abilities debate, using a carefully controlled experimental paradigm rather than a new method or architecture. Its contribution is not a technique but an empirical finding about how in-context learning of mathematical functions scales with model size and function complexity.

The weighted-sum task serves as a "model organism" for studying ICL: it is simple enough to be fully understood (we know the ground-truth function, we can analytically compute the optimal inference), complex enough to be non-trivial (5-coefficient inference requires genuine pattern extraction from examples), and parameterizable (we can dial the complexity knob by changing the number of coefficients). This positions the paper as a bridge between two research traditions: the scaling laws literature (which studies how performance changes with scale) and the mechanistic interpretability / capabilities literature (which studies what models can and cannot do), contributing a clean experimental paradigm that both communities can use.

The explicit citation of Schaeffer et al. (2024) in the figure caption signals that this paper is entering the emergent abilities conversation with empirical ammunition. The dual-metric design (continuous difference + discrete exact match) is a direct response to the measurement artifact hypothesis. If emergent abilities are purely artifacts of metric choice, then the continuous metric should reveal smooth improvement across all scales. If the paper finds that even the continuous metric shows a qualitative change at some scale threshold for complex functions, that would be evidence for genuine emergence.

The selection of models—Yi-34B, LLaMA2 70B, Mixtral 8x7B—covers a range of scales and architectures. Yi-34B represents the "large but not largest" category, while LLaMA2 70B and Mixtral 8x7B (a mixture-of-experts model with ~47B active parameters but 8×7B total) represent the frontier scale. Finding that Yi-34B excels on simple tasks but not complex ones, while the larger models handle complex tasks, would establish a scale-dependent capability boundary for ICL that prior work has not quantified. This finding has theoretical significance for understanding what fundamentally changes as models grow, and practical significance for deciding which model to deploy for ICL-heavy applications.

3. Technical Approach

3.1 Reader Orientation (Approachable Technical Breakdown)

The system is a controlled experimental paradigm for measuring how well large language models can infer a mathematical function—specifically, a linear weighted sum—solely from examples provided in the prompt, with no additional training or fine-tuning. It solves the problem of isolating and measuring pure in-context learning capability in a way that cleanly separates genuine capability emergence from measurement artifacts, by testing models across a range of parameterized function complexities and using both continuous and discrete evaluation metrics.

3.2 Big-Picture Architecture (Diagram in Words)

The experimental apparatus has five major components:

Target Function Generator — a deterministic mathematical procedure that produces linear weighted-sum functions with a specified number of coefficients ( $k = 2$ or $k = 5$ ). This component encodes the "ground truth" that models must infer. It generates input vectors (the numbers being weighted) and computes the correct output sum.
In-Context Prompt Constructor — takes the generated input-output pairs and formats them into a text prompt following a consistent template, where each line shows an input vector and its corresponding summed output. The prompt provides the model with demonstration examples (the "in-context" part) followed by a new input vector for which the model must predict the output.
Model Inference Pipeline — passes the constructed prompt to each candidate language model (Yi-34B, LLaMA2 70B, Mixtral 8x7B) and extracts the model's numeric prediction from its generated response. This component is the "subject under test."
Dual-Metric Evaluation — computes two separate performance measurements for each model prediction: a continuous difference metric (target - prediction) that captures how close the model's answer is to the correct answer in real-valued space, and a discontinuous exact match metric (target == prediction) that captures whether the model got the answer perfectly right.
Complexity-Scale Analyzer — cross-tabulates performance metrics against model scale (34B through 70B+ parameters) and function complexity (2 vs. 5 coefficients) to characterize scaling behavior and identify emergent thresholds.

Information flows as follows: the function generator creates a set of demonstration examples → the prompt constructor formats these into a text string → each model receives the prompt and generates a prediction → the evaluation module computes both the continuous difference and the exact match flag → the analyzer aggregates results across many trials and plots performance as a function of model identity for each complexity level (as shown in Figure 1, panels A and B).

3.3 Roadmap for the Deep Dive

First, the target function specification—what exactly the model must learn, how complexity is parameterized, and why a linear weighted sum was chosen over alternatives—because understanding the "task" is prerequisite to understanding the evaluation.
Second, the prompt construction protocol—the exact format of demonstrations, how many examples are provided, and how the target input is presented—because the prompt is the model's only source of information about the function.
Third, the model candidates and inference procedure—which models are tested, why these specific ones, and how predictions are extracted from generated text—because model selection determines what scaling comparisons are possible.
Fourth, the dual-metric evaluation design—the continuous difference-to-target metric, the exact match metric, and how they address the emergent abilities measurement debate—because this is the paper's primary methodological contribution.
Fifth, the complexity-scaling experimental logic—how varying the number of coefficients from 2 to 5 probes the relationship between function complexity and model scale—because this is the independent variable that makes the scaling analysis possible.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical measurement paper whose core idea is that in-context learning of mathematical functions reveals scale-dependent capability boundaries, and that using both continuous and discrete metrics on the same task provides a direct test of whether emergent abilities are measurement artifacts or genuine properties of model scaling.

Target Function Specification: What the Model Must Learn

The model's task is to infer the coefficients of a linear weighted sum from a set of input-output examples. The function being learned has the form:

$y = \sum_{i=1}^{k} w_i \cdot x_i$

where $k$ is the number of linear coefficients (the dimensionality of the function), $w_i \in \mathbb{R}$ are the true weight coefficients that the model must infer, and $x_i \in \mathbb{R}$ are the input values provided in the prompt.

What it computes: This equation computes a scalar output $y$ as the dot product between a weight vector $\mathbf{w} = [w_1, w_2, \ldots, w_k]^\top$ and an input vector $\mathbf{x} = [x_1, x_2, \ldots, x_k]^\top$ . For each input vector shown in the demonstration examples, the model sees the input values $x_1$ through $x_k$ and the corresponding output $y$ . The model must infer the hidden weights $w_i$ that generated those outputs, then apply those inferred weights to a novel input vector to predict its output.

Why this form: The linear weighted sum is the simplest nontrivial parametric function class—it has a clear algebraic structure, a well-defined solution (given enough linearly independent examples, the weights are uniquely determined by solving a linear system), and its complexity can be precisely controlled by varying $k$ . Crucially, a weighted sum requires genuine multivariate function inference: the model cannot succeed by memorizing a single scalar trend but must disentangle the contributions of multiple independent input dimensions. This distinguishes it from simpler ICL probes like "continue the arithmetic sequence," where the function is univariate and trivial to infer. The linearity also means there is no hidden nonlinearity for the model to discover—failure can be attributed purely to difficulty in inferring the coefficient structure, not to insufficient representational capacity for nonlinear relationships.

Complexity parameterization: The paper uses two complexity levels: $k = 2$ coefficients (the simple case) and $k = 5$ coefficients (the complex case). With two coefficients, the model sees pairs $(x_1, x_2)$ mapped to $w_1 x_1 + w_2 x_2$ —a credit assignment problem across two input dimensions. With five coefficients, the model must disentangle five independent weight contributions from the demonstration examples. This five-dimensional setting is substantially harder: the model needs enough examples to constrain a 5-dimensional weight space, must accurately track the contribution of each input position, and cannot rely on simple heuristics (like "the output is roughly proportional to the first number") that might suffice in low-dimensional settings. The jump from 2 to 5 coefficients is specifically chosen as a complexity threshold that probes the limits of ICL capability in the tested model size range.

Prompt Construction Protocol: How the Function is Communicated to the Model

The prompt is constructed as a text string containing demonstration examples formatted in a consistent structure. While the paper does not provide a verbatim prompt template in the snippet, we can infer its structure from the task description: each example shows an input vector and its corresponding output, and the sequence of examples constitutes the model's only source of information about the underlying function.

What happens first: For each trial, the function generator randomly samples the true weight coefficients $w_i$ (the paper does not specify the sampling distribution in the provided snippet, but weights are presumably drawn from a reasonable numeric range to produce outputs within the model's token-level numeric reasoning capability). It then generates a set of input vectors $\mathbf{x}$ (again, the sampling distribution is not detailed) and computes the corresponding outputs $y$ using the weighted sum formula. These input-output pairs form the demonstration set.

What happens second: The demonstration pairs are serialized into text. Each pair likely follows a format conceptually similar to:

Input: [x1, x2, x3, x4, x5]
Output: y

or perhaps a more condensed tabular format. The critical design constraint is that the model sees multiple such pairs—enough to theoretically determine the weights—followed by a single new input vector for which it must predict the output. The number of demonstration examples must be sufficient for the inference problem to be well-posed: with $k$ unknown coefficients, at minimum $k$ linearly independent examples are needed to uniquely determine the weights. The paper may provide more than the minimum to give the model redundant information and reduce the impact of any individual example being misinterpreted.

What happens third: The prompt ends with a query input—a new vector $\mathbf{x}_{\text{new}}$ without its output—and the model is expected to generate the corresponding numeric prediction. The model's raw text output is then parsed to extract the numeric answer.

Design choices and their justifications: Several non-obvious decisions are embedded in this protocol. First, the weights vary across trials: the model cannot succeed by learning a single fixed function; it must genuinely infer the specific weights of the current trial's function from the provided examples. This makes ICL measurement genuine rather than a test of whether the model can reproduce a memorized mapping. Second, the input-output format is intentionally simple: there is no natural language framing, no domain-specific vocabulary, no reasoning chain expectation. The model receives raw numeric mappings and must output a raw number. This strips away confounds—any difficulty the model experiences is due to the mathematical inference problem itself, not to linguistic complexity. Third, the weights are linear coefficients applied to provided inputs: the model does not need to guess the function form (it is demonstrated to be a weighted sum through the examples) but only the specific parameter values. This tests parameter inference rather than structure discovery, which is the core of in-context learning as typically studied.

Model Candidates and Inference Procedure

The paper evaluates three large language models spanning a range of parameter counts and architectural families:

Yi-34B: A ~34 billion parameter dense transformer model, representing the "large but sub-frontier" scale category. Its inclusion tests whether mid-scale models can handle complex ICL tasks or whether this capability requires frontier-scale models.
LLaMA2 70B: A ~70 billion parameter dense transformer model, representing the largest dense model in the comparison. Its inclusion establishes a frontier dense-model baseline for ICL capability.
Mixtral 8x7B: A mixture-of-experts (MoE) model with 8 expert sub-networks of ~7 billion parameters each, totaling ~47 billion active parameters during any forward pass (though the architecture holds ~56 billion total parameters). Its inclusion tests whether sparse activation architectures confer any advantage for mathematical ICL—since MoE models route inputs to different experts, they might develop specialized sub-networks better suited to numeric reasoning.

Why these specific models: The selection spans a parameter range from 34B to 70B+ (dense equivalent), which is precisely the regime where prior work on emergent abilities has observed qualitative capability shifts. If ICL of complex functions genuinely emerges at some scale threshold, the 34B-to-70B range is exactly where one would expect to find it. The inclusion of Mixtral also tests whether parameter count alone drives the effect or whether architectural innovations can shift the threshold. The paper explicitly does not test smaller models (e.g., 7B or 13B parameter models), which implies that pilot experiments may have shown near-zero performance at those scales, or that the research question is specifically about the 34B+ regime where ICL is already non-trivial.

Inference procedure: The paper's approach to extracting numeric predictions from model outputs is simple but critical. The model receives the entire prompt as a text string and generates a continuation using standard autoregressive decoding (presumably with greedy or low-temperature sampling, though the paper does not specify the decoding strategy). The generated text is then parsed to extract a numeric answer. The parsing must handle variability in how models format numbers—some models might output just the number, others might include explanatory text before or after the prediction, and formatting conventions (decimal points, digit grouping, negative number notation) may vary across models. The paper's ability to compute both difference-to-target (requiring a parsed number) and exact match (requiring string-level comparison) suggests a robust parsing pipeline, though the specific parsing rules are not detailed in the provided snippet.

A subtle point about model capability scope: The weighted-sum task tests a very specific cognitive operation—multivariate linear regression from examples—that is not a natural language task and not obviously present in standard pretraining data. The models' ability to perform this task reflects either (a) an emergent capability to simulate mathematical reasoning processes from their training on mathematical content, or (b) a more fundamental circuit for pattern extraction that operates over token sequences regardless of domain. The paper's design cannot distinguish these mechanisms, but the fact that the capability is being tested at all presumes that LLMs trained primarily on natural language can, at sufficient scale, develop representations that support numerical function inference—a claim that, if supported, has implications for the generality of representations learned during language modeling.

Dual-Metric Evaluation Design: Addressing the Emergent Abilities Debate

The paper's central methodological contribution is the simultaneous use of two complementary evaluation metrics on the same model outputs, designed to directly test the measurement artifact hypothesis for emergent abilities.

Metric 1: Difference to Target (Continuous)

The difference metric is defined as:

$\Delta = |y_{\text{target}} - y_{\text{prediction}}|$

where $y_{\text{target}} \in \mathbb{R}$ is the correct output computed by the true weighted-sum function and $y_{\text{prediction}} \in \mathbb{R}$ is the numeric value extracted from the model's generated text.

What it computes: The absolute difference between the ground-truth output and the model's predicted output, expressed in the same numeric units as the output values (which depend on the magnitude of the weights and inputs, so the absolute scale of $\Delta$ is task-dependent and not normalized in the presented figure). A difference of zero indicates perfect prediction; larger differences indicate proportionally worse approximation. When aggregated across many trials, the mean or median difference provides a continuous measure of approximation quality.

Why this form: Absolute difference is the most natural continuous metric for a regression task because it preserves the magnitude of errors—a prediction that is off by 100 is penalized twenty times more than a prediction off by 5, which correctly reflects that the model's internal representation of the function is much worse in the former case. This is the critical property for testing the measurement artifact hypothesis: if the model's internal function representation improves smoothly with scale (the model gets progressively better at approximating the weights, producing predictions with ever-smaller errors), then the continuous metric should show steady improvement. If exact match shows a sudden jump at some scale while difference continues improving smoothly, that jump is a measurement artifact—the model didn't suddenly become capable; it just crossed a threshold of the discontinuous metric. If, however, difference also shows a qualitative change (e.g., the error drops from large-and-random to small-and-systematic), that would be evidence for genuine emergence.

Metric 2: Exact Match (Discontinuous)

The exact match metric is defined as:

$\text{EM} = \mathbb{1}[y_{\text{target}} = y_{\text{prediction}}]$

where $\mathbb{1}[\cdot]$ is the indicator function, returning 1 if the model's prediction is exactly equal to the target and 0 otherwise (including both completely wrong answers and answers that are close but not exact).

What it computes: A binary flag indicating whether the model produced the numerically exact correct answer. Over many trials, the exact match rate is the fraction of predictions that are perfectly correct. This is the metric that prior work on emergent abilities has predominantly used (e.g., "model can perform arithmetic" defined as getting the answer exactly right), and it is the metric that Schaeffer et al. (2024) argued may create the illusion of emergence.

Why this form—and why compare against difference: Exact match is the "discontinuous measure" that the paper explicitly contrasts with difference. In the figure caption, the paper states that this dual-metric design is motivated by "discussions of whether emergent ability is an artifact of measurement" (citing Schaeffer et al., 2024). The logical structure of this comparison is: if both metrics tell the same qualitative story (e.g., both show that ICL of 5-coefficient functions requires 70B+ models), then the emergence is genuine and not an artifact of the metric. If the continuous metric showed that even small models produce reasonable approximations to 5-coefficient functions (low difference) but exact match remained zero (because they never get it perfectly right), the emergence would be an artifact of the metric's all-or-nothing threshold.

Complexity-Scaling Experimental Logic: Probing the Scale-Complexity Interaction

The experiment involves a 3 × 2 factorial design (three model sizes × two complexity levels), with the key analyses being:

Within-complexity scaling: For $k = 2$ , how does performance vary across Yi-34B, LLaMA2 70B, and Mixtral 8x7B? For $k = 5$ , how does it vary? The paper's finding that Yi-34B achieves the lowest difference-to-target at $k = 2$ (Figure 1A) but only the 70B+ models achieve meaningful exact match at $k = 5$ (Figure 1B) is the core empirical result.
Between-complexity scaling: How does the performance gap between models change when complexity increases? If the gap widens—smaller models degrade more severely when going from 2 to 5 coefficients than larger models do—that suggests a scale-dependent capability for handling complexity. The paper's results show qualitatively that Yi-34B's advantage at $k = 2$ disappears or reverses at $k = 5$ , supporting this interpretation.
Metric concordance/discordance: Does the difference metric tell the same story as exact match? If both metrics agree that 70B+ models dominate at $k = 5$ , the emergence is robust to metric choice. If they disagree, the emergence narrative requires qualification.

The implicit statistical model: The experiment treats each trial (a randomly generated set of weights and input-output demonstrations) as an independent draw from a distribution of possible functions. The model's prediction on each trial is compared against that specific function's ground truth. Aggregating across many trials estimates the model's expected ICL capability over the space of linear functions with $k$ coefficients, controlling for any trial-specific idiosyncrasies (e.g., unusually large weights, input vectors that are nearly collinear and make the inference problem ill-conditioned). The paper presumably uses enough trials per condition to achieve reasonable statistical precision, though exact trial counts are not specified in the provided snippet.

How this design advances beyond prior work: Prior ICL evaluations typically present models with a small number of natural language tasks (often drawn from standard benchmarks) and report task-level accuracy. This conflates multiple sources of variation—the specific examples chosen, the wording of the prompt, the domain of the task—and does not systematically vary task complexity. By using a parameterized mathematical function as the target, this paper controls all task features except complexity ( $k$ ), isolates ICL as the sole capability being measured (no memorized knowledge helps), and directly addresses the emergent abilities measurement debate through its dual-metric design. The trade-off is ecological validity: weighted-sum inference is not a natural language task that users would deploy LLMs for. The paper implicitly accepts this trade-off in favor of measurement precision and theoretical clarity.

The interpretation challenge: The finding that only 70B+ models achieve meaningful exact match at $k = 5$ does not, by itself, prove that ICL of complex functions is emergent. As Schaeffer et al. (2024) would argue, it could be that all models are improving smoothly in their ability to approximate 5-coefficient functions (captured by the difference metric), and only the largest models happen to cross the exact-match threshold. The paper's figure caption claims that exact match at $k = 5$ shows "in-context learning complex functions is an emergent ability," but this claim depends on what the difference metric shows at $k = 5$ for the smaller models—data which is not fully visible in the provided snippet. If Yi-34B achieves a relatively low difference at $k = 5$ but zero exact match, that would support the measurement artifact interpretation. If Yi-34B's difference at $k = 5$ is very large (essentially random guessing), while the 70B+ models show both low difference and non-zero exact match, that would more strongly support genuine emergence. The paper's framing leans toward the latter interpretation, but evaluating this claim requires seeing the full difference-metric results at both complexity levels, which the provided snippet only partially resolves.

4. Key Insights and Innovations

Innovation 1: Complexity-Scale Interaction as a Diagnostic for Genuine Emergence

The paper's most distinctive intellectual contribution is not the finding that larger models perform better—which would be unsurprising—but rather the demonstration that the relationship between model scale and in-context learning capability is qualitatively modulated by the complexity of the function being learned. Specifically, the paper shows that Yi-34B achieves the lowest prediction error when the function is simple (k = 2 coefficients, Figure 1A), but when complexity increases to k = 5 coefficients, only the largest models (LLaMA2 70B and Mixtral 8x7B) achieve meaningful exact match performance (Figure 1B). This is not merely a "bigger is better" story; it is a cross-over interaction where rank-ordering of models depends on task complexity.

What makes this intellectually distinctive is that prior work on scaling laws (Kaplan et al., 2020; Hoffmann et al., 2022) and in-context learning treated task difficulty as an implicit, uncontrolled variable. The typical finding was that larger models are better few-shot learners, with performance improving monotonically with scale. This paper's contribution is to show that monotonicity can break down when task complexity is systematically varied: the mid-scale model outperforms the largest models on simple tasks but is incapable of handling complex ones. This implies that scaling model size does not produce a uniform improvement in ICL capability—it specifically enables the learning of more complex functions, while introducing no benefit (or even slight degradation, in the case of the difference metric for simple functions) on simpler ones. This is a fundamentally different picture of scaling than the smooth power-law improvements documented in prior scaling literature.

The significance extends beyond the specific models tested. This finding establishes complexity as a critical confound in ICL evaluation—any ICL benchmark that does not control for or report task complexity may produce misleading comparisons between models, because the ranking depends on complexity. A model that appears inferior on one benchmark may actually be superior on more complex tasks, or vice versa. This is an important methodological caution for the field, analogous to how the psychometrics literature recognizes that test difficulty modulates the information a test provides about individual differences.

The cross-over pattern also speaks directly to the emergent abilities debate in a way more nuanced than the paper's surface-level framing suggests. If ICL capability were simply a smooth function of scale, we would expect the same model to dominate at all complexity levels, just with different absolute performance. The fact that the best model at k = 2 is not the best at k = 5 suggests that different scaling regimes enable qualitatively different capabilities, not just more of the same capability. This is evidence for genuine emergence in a more subtle sense than the usual "performance jumps from zero to above-chance" framing: the capability to handle high-dimensional function inference appears to be a distinct cognitive skill that mid-scale models lack entirely, not a continuous extension of the skill that enables low-dimensional inference.

Innovation 2: The Parametric Function Probe as a "Model Organism" for ICL Research

The paper introduces a new experimental paradigm—using parameterized mathematical functions as the target of in-context learning—that provides a level of experimental control previously unavailable in ICL research. This is not merely a new benchmark; it is a conceptual framework for studying in-context learning that treats the to-be-learned function as an independent variable whose properties (dimensionality, linearity, noise level) can be systematically manipulated.

Prior work on ICL has predominantly used natural language tasks—classification, translation, question answering, reasoning—where the "function" being learned from examples is a mixture of linguistic patterns, factual knowledge, and reasoning strategies. In those settings, it is impossible to answer basic scientific questions about ICL: Is the model genuinely inferring the relationship from examples, or is it recognizing a pattern it memorized during pretraining? What properties of the underlying function make it easy or hard to learn from context? How does the number of examples needed scale with function complexity? The parametric function probe addresses all of these by using purely synthetic, mathematically specified functions that the model cannot have encountered during training (since the weights are randomly generated per trial) and whose ground truth is fully known to the experimenter.

This is a significant methodological advance because it enables decomposition of ICL into its constituent inference operations. With a linear weighted sum, the experimenter knows that success requires: (1) parsing numeric values from text, (2) tracking which input position corresponds to which coefficient, (3) estimating the weight for each coefficient from the demonstration examples, and (4) computing the dot product for a novel input. If a model fails, the experimenter can design follow-up probes to isolate which sub-operation failed—something impossible with natural language ICL tasks where the "correct" inference strategy is unknown. The parametric function probe thus serves as a "model organism" for ICL in the same way that C. elegans serves as a model organism for developmental biology: it is simple enough to be fully understood, yet exhibits the core phenomenon of interest in a way that generalizes.

The intellectual lineage of this approach traces to cognitive science, where carefully controlled artificial learning tasks (category learning with parameterized category structures, function learning with parameterized function families) have been used for decades to study human inductive inference. The paper imports this methodological rigor into LLM research, and in doing so, opens a bridge between the LLM capabilities literature and the cognitive science of human learning—two fields that study essentially the same phenomenon (inferring structure from examples) but have historically used very different experimental methods. This bridge is valuable because it allows LLM researchers to draw on decades of cognitive science findings about how function complexity, example diversity, and prior knowledge interact to determine learning difficulty.

Innovation 3: Dual Continuous-Discrete Measurement as a Diagnostic Instrument

While the paper explicitly frames its dual-metric design as motivated by the emergent abilities debate (citing Schaeffer et al., 2024), the deeper intellectual contribution is the operationalization of that debate into a concrete experimental diagnostic. The simultaneous use of a continuous error metric (difference to target) and a discrete accuracy metric (exact match) on the same model outputs is not just a robustness check—it is a principled method for distinguishing between genuine capability emergence and measurement artifact, and the paper demonstrates how this diagnostic can produce nuanced conclusions.

The core logic is this: if a capability genuinely emerges at some scale threshold, both the continuous and discrete metrics should show a qualitative change at that threshold—the continuous metric should show a transition from large, essentially random errors to small, systematic errors, and the discrete metric should transition from near-zero to above-chance accuracy, at roughly the same scale. If instead the continuous metric shows smooth improvement across all scales but the discrete metric shows a sudden jump, the apparent emergence is an artifact of thresholding a continuously improving latent capability.

This is intellectually significant because it transforms the emergent abilities debate from a philosophical argument about definitions into an empirically testable question with a clear experimental design. Prior to this work, the debate largely consisted of Schaeffer et al. pointing out that discontinuous metrics can create illusions of emergence, and defenders of emergence arguing that some capabilities really do appear qualitatively at scale, with no clear way to adjudicate between these positions for any specific capability. The dual-metric design provides that adjudication method, and it can be applied to any capability of interest, not just the weighted-sum task evaluated here.

The paper's finding—that only 70B+ models achieve meaningful exact match at k = 5—is more informative when interpreted through this diagnostic lens. If the continuous metric for smaller models at k = 5 showed reasonable approximation quality (low difference) but zero exact match (because approximation errors, while small, never hit exactly zero), the emergence would be primarily a measurement artifact—smaller models can approximate complex functions but can't achieve perfection. If the continuous metric for smaller models at k = 5 showed very large errors (essentially random), the emergence would be genuine—smaller models lack any meaningful capability for complex function inference, and only at ~70B+ parameters does this capability appear. The paper's interpretation leans toward the latter, but the full continuous-metric data would be needed to confirm. Regardless, the diagnostic framework itself—the idea that emergence claims should be evaluated by comparing continuous and discrete performance trajectories—is a conceptual contribution that generalizes beyond this paper.

This innovation also carries a practical implication for model evaluation: the field should routinely report both continuous and discrete metrics for capabilities where emergence is claimed. Current benchmarks almost exclusively report discrete accuracy metrics (exact match, F1, pass@k), which are susceptible to the measurement artifact critique. Including a continuous metric—even a simple one like log-probability of the correct answer under the model's distribution—would provide the data needed to distinguish measurement artifacts from genuine emergence in published results.

5. Experimental Analysis

Evaluation Methodology

Dataset. The dataset is synthetically generated by the authors rather than drawn from an existing benchmark. Each trial consists of a randomly sampled set of linear weight coefficients $w_1, \ldots, w_k$ and corresponding input-output pairs $(\mathbf{x}, y)$ where $y = \sum_{i=1}^{k} w_i x_i$ . The number of coefficients $k$ is set to either 2 (simple condition) or 5 (complex condition). The sampling distributions for the weights and input vectors are not specified in the provided snippet. This synthetic construction is essential because it guarantees the model cannot have memorized the function from pretraining data—the weights vary per trial, so success requires genuine in-context inference.
Base model(s). Three large language models are evaluated: Yi-34B (~34B parameters, dense transformer), LLaMA2 70B (~70B parameters, dense transformer), and Mixtral 8x7B (mixture-of-experts architecture with 8 experts of ~7B parameters each, totaling ~47B active parameters per forward pass). These models are chosen to span the scale range (34B–70B+) where prior work on emergent abilities has documented qualitative capability shifts. The inclusion of Mixtral additionally tests whether sparse activation architectures shift the scale threshold for complex ICL capability.
Metrics. The paper evaluates each model prediction using two complementary metrics: (1) difference to target, a continuous metric computed as $|\text{target} - \text{prediction}|$ , which captures how far the model's prediction is from the ground-truth output in real-valued space; and (2) exact match, a discontinuous metric defined as $\mathbb{1}[\text{target} = \text{prediction}]$ , which captures whether the model produced the numerically exact correct answer. The dual-metric design is explicitly motivated by the debate over whether emergent abilities are artifacts of discontinuous measurement (Schaeffer et al., 2024): comparing continuous and discrete metrics on the same task provides a diagnostic for distinguishing genuine emergence from thresholding effects.
Baselines. The paper does not describe external baselines or compare against alternative approaches. The comparisons are internal to the experiment: across model scales (34B vs. 70B vs. 8x7B MoE) and across complexity levels (k = 2 vs. k = 5). The experimental logic treats smaller models as implicit baselines for larger ones, and the simple (k = 2) condition as a baseline for the complex (k = 5) condition. There is no comparison against non-LLM methods (e.g., explicit linear regression solvers), no comparison against fine-tuned models, and no comparison against alternative prompting strategies (chain-of-thought, scratchpad, etc.).
Generation budget / compute accounting. The paper does not explicitly define a compute budget or generation budget metric. Each model presumably receives the same number of in-context demonstration examples and generates a single prediction per trial. There is no explicit accounting for the number of output tokens generated, the FLOPs consumed per forward pass, or any comparison of compute-matched conditions across models of different sizes. The experiment is a capability probe rather than an efficiency study—the question is what each model can do, not how efficiently it does it.
Cross-validation / statistical protocol. The paper does not describe a cross-validation procedure, statistical significance testing, confidence intervals, or trial count per condition. The provided snippet contains only the figure (Figure 1) with two panels showing the main results, without reporting standard errors, sample sizes, or any quantification of variability across trials. This constitutes a significant gap in the reported methodology: without knowing the number of trials per condition or the variance in model performance, the reader cannot assess whether observed differences between models (e.g., Yi-34B's apparent advantage at k = 2) are statistically reliable or could be explained by sampling noise.

Main Quantitative Results

Simple Function Learning (k = 2 Coefficients)

The headline finding at the two-coefficient complexity level (Figure 1A) is that Yi-34B achieves the lowest difference-to-target among all tested models, outperforming both the larger LLaMA2 70B and the mixture-of-experts Mixtral 8x7B. This result is reported in the figure caption as: "when there is two linear coefficients, Yi-34B performs the best when measuring by the difference to the target number."

This finding is notable because it violates the simple "larger is better" expectation that dominates the scaling literature. If in-context learning capability were a monotonic function of model size, LLaMA2 70B (with roughly twice the parameters of Yi-34B) should achieve equal or lower prediction error. The observed reversal—the smaller model outperforms the larger one on this specific complexity level—suggests that model scale does not uniformly improve ICL fidelity and that other factors (potentially training data composition, architecture details, or optimization choices) can dominate.

The paper does not report exact match rates for the k = 2 condition in the provided snippet. It is unclear whether Yi-34B's advantage in the continuous difference metric translates to an advantage in exact match, or whether all models achieve similar exact match rates at this low complexity level (since a function with only two coefficients may be simple enough that all tested models can infer it perfectly in a non-trivial fraction of trials).

Complex Function Learning (k = 5 Coefficients)

When the complexity is increased to five coefficients, the pattern shifts dramatically (Figure 1B). The figure caption states: "increasing the number of linear coefficients to 5, only models that are large enough (LLaMA2 70B and Mixtral 8x7B) can achieve meaningful exact match, showing that in-context learning complex functions is an emergent ability."

This is the paper's central empirical claim and its primary evidence for emergence. The key observations are:

Yi-34B fails entirely: At k = 5, Yi-34B achieves zero or near-zero exact match. Despite being the strongest model at k = 2, the mid-scale model cannot produce a single perfectly correct prediction when the function dimensionality increases to five.
LLaMA2 70B succeeds: The largest dense model achieves "meaningful exact match" (the paper does not quantify the exact rate in the provided snippet, but the term implies substantially above chance and practically useful).
Mixtral 8x7B succeeds: The mixture-of-experts model also achieves meaningful exact match, demonstrating that the capability is not exclusive to dense architectures and that MoE models at sufficient total scale can also handle complex ICL.

The threshold nature of this result—one model produces essentially zero exact match while the larger models produce non-trivial rates—is what the authors interpret as evidence for emergence. The capability to learn complex functions from in-context examples does not gradually improve with scale; instead, it appears to be absent below some critical scale threshold (somewhere between 34B and ~47B+ active parameters) and present above it.

The paper does not report the continuous difference metric for the k = 5 condition in the provided snippet. This is a critical missing piece of evidence: without knowing whether Yi-34B's difference-to-target at k = 5 is (a) very large (suggesting the model has no meaningful grasp of the 5-dimensional function) or (b) moderately low but never exactly correct (suggesting the model approximates the function well but cannot achieve perfection due to, e.g., tokenization noise or insufficient numerical precision), the claim of emergence cannot be fully evaluated. The Schaeffer et al. (2024) critique argues precisely that scenario (b) would mean the apparent emergence in exact match is an artifact of thresholding a continuously improving capability. The paper's figure caption asserts emergence, but the provided data does not resolve whether the difference metric tells the same story.

Cross-Complexity Comparison

Combining the two complexity levels reveals a cross-over interaction: Yi-34B dominates at k = 2 (by the continuous metric) but fails entirely at k = 5 (by exact match), while LLaMA2 70B and Mixtral 8x7B are slightly worse at k = 2 but uniquely capable at k = 5. This is not a main effect of model scale (which would predict the same ranking at both complexity levels) but a scale × complexity interaction, implying that what model size buys is not uniformly better ICL but specifically the ability to handle more complex functions.

The practical implication is that model selection for ICL-heavy applications should depend on the expected complexity of the functions being learned from context. A 34B model may be preferable (lower error, lower inference cost) if functions are simple; only if functions are expected to be high-dimensional does the case for 70B+ models become compelling.

Ablation Studies and Robustness Checks

The provided paper snippet does not contain ablation studies. There is no systematic variation of:

The number of in-context examples provided (does providing more demonstrations help smaller models at k = 5, or is the capability fundamentally absent regardless of how much information is given?)
The format of the prompt (would natural language framing or chain-of-thought encouragement change any results?)
The decoding strategy (greedy vs. temperature sampling, effect of multiple samples per trial)
The specific weight or input distributions (are results robust to the range of numeric values used, or do models fail when coefficients are very large/small?)
The number of trials or statistical power (are the observed cross-over patterns stable across different random seeds for weight generation?)
The parsing pipeline for extracting numeric predictions (how sensitive are exact match rates to formatting variations in model outputs?)

The absence of these ablations means the results represent a single point in a large experimental design space, and the generality of the findings (to different prompt formats, example counts, or numeric ranges) is unknown.

Implicit ablation: model architecture (dense vs. MoE). The inclusion of both LLaMA2 70B (dense) and Mixtral 8x7B (MoE) among the models that succeed at k = 5 serves as an informal architecture ablation. The fact that both succeed suggests that the critical factor is total effective capacity (~47B+ active parameters), not the specific architectural choice of dense vs. mixture-of-experts. However, this comparison is confounded by the fact that these models are trained on different data with different optimization procedures—they differ in many ways beyond architecture, so attributing the similar performance to architecture-agnostic scaling is speculative.

Critical Assessment

The central claim of this paper is that in-context learning of complex functions is an emergent ability, manifesting only when models surpass a critical scale threshold—specifically, that models in the ~34B parameter range fail at k = 5 coefficient inference while models at ~70B (dense) and ~47B active (MoE) succeed (Figure 1B). The experiments provide suggestive but incomplete evidence for this claim, and several important gaps prevent a definitive conclusion.

What the experiments demonstrate: The experiments convincingly show that Yi-34B, which performs well (indeed, best) at k = 2, produces zero or near-zero exact match at k = 5, while LLaMA2 70B and Mixtral 8x7B achieve non-zero exact match. This establishes that something changes between the 34B and 70B scale that affects exact-match performance on high-dimensional ICL tasks. The cross-over interaction—Yi-34B being best at k = 2 but worst at k = 5—strengthens the case that the effect is not trivially "larger is always better" but involves a qualitative shift in what the models can do.

What the experiments do not demonstrate: The claim of "emergence" depends critically on what the continuous difference metric reveals at k = 5 for Yi-34B. If Yi-34B produces reasonably low difference-to-target at k = 5 (approximating the function well but never perfectly), then the "emergence" in exact match is plausibly a measurement artifact of thresholding a smoothly improving capability—exactly the concern Schaeffer et al. (2024) raised. The paper reports the difference metric at k = 2 (Figure 1A) and exact match at k = 5 (Figure 1B), but the crucial combination—difference metric at k = 5 for all models—is not visible in the provided snippet. Without this data, the emergence claim is supported by only half the necessary evidence. The dual-metric design is methodologically sound, but the paper appears to apply it incompletely in the reported results.

Statistical reliability is unquantified. The paper does not report the number of trials per condition, any measure of variance (standard errors, confidence intervals), or any statistical test of the differences between models. If the trial count is small, the observed cross-over (Yi-34B best at k = 2, worst at k = 5) could be noise. If Yi-34B's exact match at k = 5 is, say, 2% while LLaMA2 70B's is 8%, the qualitative pattern holds (one is "meaningful" and one is not), but the statistical significance of that difference depends on trial count and variance. The paper's presentation as a simple bar or line chart without error bars or sample size reporting makes the results visually suggestive but not statistically evaluable.

Only three models are tested. The experiment evaluates exactly three models at two discrete scale points: ~34B and ~70B (with Mixtral at ~47B active as an intermediate architecture variant). This is a very sparse sampling of the model scale axis. The claim that there is a "threshold" between 34B and 70B would be substantially stronger if intermediate models were tested—does a 50B dense model show partial exact match at k = 5? Does a 13B model already fail at k = 2? The sparsity of scale points means the results are consistent with a threshold interpretation but also consistent with a smooth sigmoidal improvement that happens to cross the exact-match detectability threshold in the 34B–70B range. Testing 4–5 models spanning 7B to 70B would dramatically clarify the scaling curve shape.

Only two complexity levels are tested. The experiment uses k = 2 and k = 5 as the simple and complex conditions. This binary complexity manipulation cannot reveal the shape of the complexity-scaling relationship. Does Yi-34B fail at k = 3? Does it partially succeed at k = 4? Does LLaMA2 70B eventually fail at k = 10? The binary design identifies that a difficulty threshold exists somewhere between 2 and 5 coefficients, but cannot characterize where each model's capability boundary lies or whether the boundary is sharp or gradual. A more informative design would sweep k continuously (e.g., k = 1, 2, 3, 4, 5, 6, 8, 10) for each model to map out the full complexity-capability frontier.

No experimental control for confounds beyond parameter count. Yi-34B, LLaMA2 70B, and Mixtral 8x7B differ in many dimensions beyond parameter count: training data composition and quantity, tokenizer design, training duration, optimization hyperparameters, and architectural details (dense vs. MoE). The observation that Mixtral succeeds at k = 5 despite having fewer active parameters than LLaMA2-70B could reflect MoE-specific advantages, differences in training data quality, or simply that total parameter count (including inactive experts) is the relevant scaling variable. Without controlling for these confounds—for example, by comparing multiple model families at similar scales—attributing the k = 5 capability gap purely to scale is premature.

Missing baselines undermine claims about in-context learning specifically. The experiment shows that models produce outputs when given input-output examples, but it does not demonstrate that the models are actually performing in-context learning (inferring the function from the provided examples) as opposed to some other process. A critical missing baseline is zero-shot performance: what do the models predict when given only the query input without any demonstration examples? If models can guess the output distribution well without examples (e.g., by learning during pretraining that outputs in this format tend to be certain magnitudes), then the observed performance at k = 5 may not reflect ICL at all. Similarly, a scrambled-label baseline (providing demonstration pairs where the outputs are randomly permuted relative to inputs) would test whether models genuinely use the input-output correspondence or simply reproduce plausible-looking numbers. The paper's task is designed to make zero-shot success unlikely (since weights are random), but without verifying this assumption, the attribution of performance to in-context learning is an inference rather than an established fact.

No comparison against explicit algorithms. A natural baseline for the weighted-sum task is an explicit linear regression solver: given the demonstration examples, solve the normal equations or apply gradient descent to recover the weights, then compute the prediction. This is the optimal inference procedure. Comparing model predictions against this optimal baseline would contextualize model performance—are the models approximating the optimal inference (just with some noise), or are they using a fundamentally different and suboptimal strategy? If LLaMA2 70B achieves, say, 40% exact match while the optimal solver achieves 100% (which it should, given noiseless data and sufficient examples), the remaining 60% error represents the inference gap that scale has not closed.

Experiments that would strengthen the paper:

Full dual-metric reporting at all conditions: Report both difference-to-target and exact match for all three models at both k = 2 and k = 5, ideally in a 2×2 panel or a single figure with both metrics overlaid. This is the minimum additional analysis needed to evaluate the emergence claim.
Sweep k continuously: Test models at k = 1, 2, 3, 4, 5, 6, 8 for a fine-grained view of the complexity-scaling curve. Does each model show a sharp dropoff at some k, or does performance degrade gradually?
Test more model scales: Include models at 7B, 13B, and intermediate scales (e.g., 50B) to resolve the shape of the scale-complexity interaction and determine whether a genuine threshold exists.
Zero-shot and scrambled-label controls: Verify that performance depends on valid input-output demonstrations by comparing against conditions where no examples or misleading examples are provided.
Vary the number of demonstrations: Test whether providing more examples (e.g., 10, 20, 50 instead of the default) allows smaller models to close the gap at k = 5, which would reveal whether the scale threshold is about sample efficiency or fundamental inference capability.
Optimal-solver baseline: Report the exact match and difference-to-target achievable by an explicit linear regression solver given the same demonstration examples, to establish the ceiling and quantify the inference gap.
Statistical reporting: Include error bars, trial counts, and ideally statistical tests for all between-model comparisons.

In summary, the paper presents an intriguing and well-motivated experimental paradigm, and the observed cross-over interaction (Yi-34B best at simple ICL, incapable at complex ICL) is suggestive of a genuine scale-dependent capability boundary. However, the evidence for the specific claim of "emergence" is incomplete without the continuous metric data at k = 5, and the experimental design—only two complexity levels, only three models, no control conditions, no statistical quantification—limits the strength of the conclusions that can be drawn. The contribution is best viewed as a promising demonstration of the parametric function probe methodology and a preliminary finding that warrants more systematic investigation rather than a definitive resolution of the emergent abilities debate.

6. Limitations and Trade-offs

Limitation 1: The Emergence Claim Depends on Incomplete Dual-Metric Reporting

The paper's central theoretical claim is that in-context learning of complex functions is an emergent ability—a qualitative capability that appears only when models surpass a critical scale threshold. This claim is explicitly stated in the figure caption: "only models that are large enough (LLaMA2 70B and Mixtral 8x7B) can achieve meaningful exact match, showing that in-context learning complex functions is an emergent ability." The claim is structured as a direct response to Schaeffer et al. (2024), whose critique is that emergent abilities may be measurement artifacts arising from discontinuous metrics rather than genuine capability discontinuities. The paper's dual-metric design—continuous difference to target alongside discrete exact match—is the intended diagnostic for distinguishing between these interpretations.

The constraint: The provided paper snippet reports the continuous difference metric at k = 2 (Figure 1A) and the exact match metric at k = 5 (Figure 1B), but does not present the continuous difference metric at k = 5 for any model, and particularly not for Yi-34B. This is precisely the data point that would adjudicate between genuine emergence and measurement artifact. If Yi-34B produces a reasonably low difference-to-target at k = 5 (indicating it approximates the 5-coefficient function decently but never perfectly), the emergence in exact match would be an artifact of thresholding a continuously improving capability—exactly the Schaeffer et al. concern. If Yi-34B produces a very large difference at k = 5 (comparable to random guessing), the capability is genuinely absent at that scale. The paper does not provide this data.

The consequence: Without the continuous metric at k = 5, the claim of emergence is supported by only half of the paper's own diagnostic framework. A practitioner evaluating this work cannot determine whether the observed exact-match failure of Yi-34B at k = 5 reflects a genuine capability gap or a measurement artifact. This matters for model selection: if Yi-34B approximates 5-coefficient functions reasonably well (low difference) but never exactly (zero exact match), it may still be practically useful for applications where approximate answers suffice, and the paper's framing of Yi-34B as incapable at k = 5 would be misleading. Conversely, if Yi-34B's difference is very large, the paper's emergence claim is well-supported but the practical implication is that 34B models are genuinely useless for complex ICL—a different deployment consideration.

What evidence exists: The paper provides no continuous metric data at k = 5, no ablation testing this specific comparison, and no discussion acknowledging that this data is missing. The dual-metric framework is clearly articulated in the figure caption, but the execution applies it asymmetrically: continuous metric for the simple condition, discrete metric for the complex condition. The critical cell of the 2 (complexity) × 2 (metric) matrix is empty.

Mitigation status: Not addressed. The paper does not acknowledge this gap or suggest that future work should report the full cross of complexity and metric. This is a fundamental evidentiary gap in the paper's primary claim.

Limitation 2: Only Three Models at Two Discrete Scale Points Prevents Characterizing the Scaling Curve

The experiment evaluates exactly three language models—Yi-34B, LLaMA2 70B, and Mixtral 8x7B—at two discrete points on the parameter-count axis (~34B and ~70B dense, with ~47B active for MoE). The claim that there exists a "threshold" between these points, with capability absent at 34B and present at 70B+, requires the assumption that the scaling relationship is step-like rather than smooth and sigmoidal.

The constraint: With only two scale points (34B and 70B) that show different qualitative behavior, the data is consistent with at least three interpretations: (1) a genuine step function—capability is absent below some critical scale and present above it; (2) a smooth sigmoidal improvement—performance increases gradually with scale and happens to cross the detectability threshold (for exact match) in the 34B–70B range, with models at 50B showing partial success; or (3) a model-family effect—Yi-34B, LLaMA2 70B, and Mixtral 8x7B differ in training data, tokenization, optimization, and architecture, and the observed difference may reflect those confounds rather than parameter count per se. The sparse sampling of the scale axis makes these interpretations indistinguishable.

The consequence: A practitioner deciding between deploying a 34B model, a 50B model, or a 70B model for ICL-heavy applications cannot determine from this paper where the capability boundary lies. If the relationship is sigmoidal, a 50B model might achieve, say, 15% exact match at k = 5—not as good as 70B but practically useful and cheaper to run. If the relationship is a step function with the threshold at exactly 70B, then anything below 70B is useless regardless of how close it is to the threshold. The paper provides no guidance for this decision. Furthermore, the sparsity of scale points, combined with the lack of statistical quantification (no confidence intervals, no trial counts), means that even the observed ordering (Yi-34B best at k = 2, LLaMA2-70B and Mixtral at k = 5) could be unstable across different random seeds for weight generation.

What evidence exists: Figure 1 presents the only model-scale data. Three models at two scale points (with Mixtral as an intermediate architecture variant rather than a clean intermediate scale point). No intermediate scales (e.g., 7B, 13B, 50B) are tested. No multiple model families at the same scale are compared to control for confounds.

Mitigation status: Not addressed. The paper does not discuss the sparsity of scale sampling, does not test intermediate-scale models, does not control for model-family confounds, and does not qualify its threshold claim with the caveat that only two discrete scale points are observed. A richer scale sweep with 5+ models spanning 7B–70B would be needed to characterize the scaling curve shape.

Limitation 3: Binary Complexity Manipulation Cannot Characterize Each Model's Capability Boundary

The experiment uses exactly two complexity levels: k = 2 coefficients (simple) and k = 5 coefficients (complex). This binary manipulation can establish that a difficulty threshold exists somewhere between 2 and 5 coefficients, but it cannot reveal where each model's capability boundary lies, whether the boundary is sharp or gradual, or how far beyond the tested maximum the largest models can go.

The constraint: The design treats function complexity as a two-level categorical variable rather than a continuous parameter. At k = 2, Yi-34B succeeds (indeed outperforms larger models on difference-to-target). At k = 5, Yi-34B fails entirely on exact match. This tells us Yi-34B's capability boundary lies somewhere at k = 3 or k = 4, but we do not know which. Similarly, LLaMA2 70B succeeds at k = 5, but we do not know whether it would fail at k = 8 or k = 10. The paper therefore identifies that a scale-dependent complexity boundary exists but cannot characterize it quantitatively.

The consequence: A practitioner facing a specific deployment task where the effective function complexity is, say, k = 3 or k = 6 has no basis for model selection from this paper. If the target complexity is k = 3, is Yi-34B sufficient (its boundary might be at k = 4) or is a larger model needed (its boundary might be at k = 3)? The binary design provides no resolution between these possibilities. More fundamentally, the claim that "in-context learning complex functions is an emergent ability" is underspecified without quantifying what "complex" means: the paper establishes that k = 5 is complex relative to k = 2 for models at the tested scales, but provides no mapping from coefficient count to a more general notion of function complexity, and no evidence that the 2-to-5 gap generalizes to other measures of complexity.

What evidence exists: Two complexity levels are tested and reported in Figure 1. No intermediate levels (k = 3, k = 4), no levels beyond k = 5 (e.g., k = 8, k = 10), and no other axis of complexity (e.g., nonlinear functions, noisy outputs, varying input dimensionality independently of coefficient count) are explored.

Mitigation status: Not addressed. The paper does not discuss the binary complexity design as a limitation, does not suggest future work sweeping k continuously, and does not attempt to estimate where capability boundaries lie for individual models.

Limitation 4: No Controls Confirm That Performance Reflects In-Context Learning Rather Than Other Processes

The paper interprets model performance on the weighted-sum task as evidence for in-context learning—the ability to infer the underlying function from the provided demonstration examples. However, the experimental design lacks the control conditions necessary to establish that ICL, rather than alternative processes, is driving performance. Several alternative explanations for the observed behavior are not ruled out.

The constraint: The experiment shows that models produce predictions when given input-output examples, but does not include a zero-shot baseline where the model receives only the query input without demonstrations. If models can produce reasonable predictions without any examples—for instance, by learning during pretraining that numbers in this format tend to fall within certain ranges, or by exploiting token-level statistical regularities—then the examples are not necessary for performance, and calling the observed behavior "in-context learning" is misleading. Critically, at k = 5, the models that succeed (LLaMA2 70B, Mixtral 8x7B) might already produce plausible-looking numbers without demonstrations, and the paper cannot distinguish between ICL and zero-shot capability. Also missing is a scrambled-label control: providing demonstration pairs where the outputs are randomly permuted relative to their corresponding inputs. If model performance is unaffected by scrambling, the model is not using the input-output mapping at all—it is simply reproducing plausible output values from its prior.

The consequence: The paper's central interpretation—that this experiment measures in-context learning specifically—is an inference rather than an established fact. If larger models succeed at k = 5 primarily because they have better zero-shot numerical priors (e.g., they have seen more numerical data during pretraining and have better-calibrated expectations about plausible output ranges), then the paper's findings reflect pretraining data scale effects rather than ICL capability emergence. A practitioner evaluating whether to rely on ICL for their application needs to know whether the model is genuinely learning from their provided examples or simply has good priors—the former suggests the model will adapt to novel function families the practitioner provides; the latter suggests it will only work well for function families statistically similar to those seen during pretraining.

What evidence exists: The paper provides no zero-shot baseline, no scrambled-label control, and no analysis of whether the demonstration examples actually affect model predictions beyond providing formatting cues. The task is designed such that zero-shot success is unlikely (since weights are randomly generated per trial), but this design assumption is not empirically verified.

Mitigation status: Not addressed. The paper does not discuss the need for control conditions to isolate ICL, does not report any ablations testing whether example quality or ordering affects performance, and does not acknowledge this as a limitation.

Limitation 5: Missing Statistical Quantification Prevents Assessment of Result Reliability

The paper reports results as point estimates in a figure (Figure 1) without any quantification of statistical uncertainty. No trial counts, no standard errors, no confidence intervals, and no statistical tests are reported. The reader cannot determine whether observed differences between models—particularly the cross-over interaction where Yi-34B is best at k = 2 but worst at k = 5—are reliable or could reflect sampling variability across a small number of randomly generated function trials.

The constraint: Each trial involves randomly sampling the true weight coefficients and generating input-output pairs. The number of trials per condition (model × complexity level) is unknown from the provided snippet. If the trial count is small (e.g., 20–50 trials per condition), the observed differences could be noise. Yi-34B's apparent advantage at k = 2 on the difference metric might not replicate with a different random seed; similarly, the exact-match failure at k = 5 might be less absolute than it appears (e.g., 0% in a small sample vs. 2% in expectation). The paper's lack of statistical reporting makes it impossible to evaluate these possibilities.

The consequence: A practitioner cannot make deployment decisions based on point estimates of unknown reliability. If Yi-34B's advantage at k = 2 is within the margin of error, the cost savings of deploying the smaller model might not be justified by a genuine performance advantage. If the exact-match difference at k = 5 between Yi-34B (0%) and LLaMA2 70B (e.g., 15% with a standard error of ±8%), the difference might not be statistically significant, and the paper's claim of a qualitative capability gap would be unsupported. Without replication data or uncertainty quantification, the paper's contributions are suggestive but not actionable for practitioners who need reliable estimates.

What evidence exists: Figure 1 presents only point estimates with no error bars, no sample size annotations, and no reporting of variance across trials. The figure caption makes qualitative claims ("performs the best," "meaningful exact match") without quantification.

Mitigation status: Not addressed. The paper does not mention trial counts, does not report any variance measures, does not present statistical tests, and does not discuss the reliability of the observed model rankings. This is a significant methodological gap that undermines the strength of all empirical claims in the paper.

Limitation 6: Single Model Family, Single Architect, and Single Task Family Limit Generality

The paper's findings are derived from three specific models (Yi-34B, LLaMA2 70B, Mixtral 8x7B) on a single task family (linear weighted sum inference) with two complexity levels (k = 2, k = 5). The extent to which the observed scale-complexity interaction generalizes—to other model families, other architectures, or other task types—is entirely unexplored.

The constraint: The three models tested share broad architectural similarities: they are all transformer-based autoregressive language models trained primarily on text. They differ in parameter count, training data, and architectural details (dense vs. MoE), but they represent a narrow slice of the model design space. There is no evidence about whether the scale threshold for complex ICL would be similar for models from different families (e.g., GPT-4 class models, Gemini, Claude), for models using different architectures (e.g., state-space models like Mamba), or for models trained with different objectives or data mixtures. Similarly, the weighted-sum task probes a specific cognitive operation—multivariate linear function inference from numerical examples—that may not be representative of in-context learning more broadly. Tasks requiring different inference operations (nonlinear functions, logical rules, linguistic patterns, visual patterns) may show entirely different scale-complexity relationships.

The consequence: A practitioner deploying an LLM for ICL tasks cannot generalize from this paper's findings to their specific model or task. If they use a model from a different family (e.g., GPT-4, Claude, Gemini), the paper provides no basis for predicting whether a 34B-equivalent model will succeed or fail on tasks with effective complexity analogous to k = 5. Similarly, if their ICL task is not a linear weighted sum—for instance, learning a classification boundary, a translation mapping, or a logical rule—the paper's specific complexity threshold has no clear mapping to their domain. The paper's contribution is thus a demonstration of the phenomenon in one setting rather than a general characterization of ICL scaling.

What evidence exists: All experiments use the linear weighted-sum task with three models from the LLaMA/Yi/Mixtral ecosystem. No other model families, architectures, or task types are tested. The paper does not include experiments on standard ICL benchmarks, natural language tasks, or other mathematical function families that would establish cross-task generality.

Mitigation status: Not addressed. The paper does not discuss generalizability as a limitation, does not suggest that the scale threshold may vary across model families or tasks, and does not propose cross-model or cross-task replication as future work. The findings are presented as general claims about in-context learning emergence without qualification about scope.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the discourse around in-context learning from a qualitative capability description to a quantitative, complexity-parameterized scaling phenomenon. Prior to this work, the field knew that larger models are generally better few-shot learners, but the relationship between model scale, task complexity, and ICL capability was treated as a monotonic, unexamined background assumption. This paper's contribution is to introduce complexity as a first-class variable in the study of in-context learning, and in doing so, it reveals that the scaling relationship is not simply "larger is better" but involves a cross-over interaction where the best model at simple tasks (Yi-34B at k = 2) is incapable of complex tasks (k = 5), while larger models show the reverse pattern. This reframes ICL as a multivariate capability whose expression depends jointly on scale and complexity, not a unidimensional trait that improves uniformly with parameter count.

This is best understood as an incremental but important methodological reframing rather than a paradigm shift. The paper does not propose a new theory of ICL, a new architecture, or a new training procedure. What it does is introduce a parametric function probe paradigm—using synthetic, complexity-parameterized mathematical functions as the target of ICL—that provides experimental control previously unavailable in the ICL literature. This is analogous to how controlled artificial grammar learning paradigms transformed the study of human language acquisition in cognitive science: by stripping away natural language confounds (prior knowledge, semantic content, domain familiarity) and replacing them with a fully specified synthetic system, researchers could isolate the learning mechanism itself. The paper's weighted-sum probe serves the same role for LLM ICL research. This methodological contribution is likely to outlast the specific empirical findings about Yi-34B and LLaMA2 70B, because it provides a template for systematically studying how ICL capability varies with any function property the experimenter chooses to manipulate.

The paper also provides a concrete, operational resolution to a tension in the emergent abilities literature. The debate crystallized by Schaeffer et al. (2024)—whether emergent abilities are genuine or measurement artifacts—had largely been conducted at the level of statistical argument and re-analysis of published benchmark curves. This paper shows how to test the hypothesis experimentally rather than argue about it post hoc: by measuring both a continuous approximation metric and a discrete exact-match metric on the same model outputs, the experimenter can distinguish between (a) smooth improvement that crosses a detectability threshold (measurement artifact) and (b) qualitative capability change (genuine emergence). The diagnostic framework is general: it can be applied to any capability where emergence is claimed, using any continuous metric that captures graded improvement (log-probability, distance to target, partial credit) alongside the standard discrete accuracy metric. This transforms the emergence debate from a philosophical disagreement about definitions into an empirical research program.

The cross-over interaction finding—Yi-34B best at k = 2, incapable at k = 5—has specific implications that redirect research attention. It suggests that mid-scale models may be underestimated by standard benchmarks that only test simple tasks, and conversely, that frontier-scale models may be overkill for simple ICL tasks. A research program that characterizes the full scale × complexity interaction surface—mapping out which model sizes are optimal for which function complexity regimes—would enable principled model selection that current benchmark-centric evaluation cannot provide. This also implies that scaling laws research, which has focused on how loss scales with compute and parameters for fixed task distributions, is incomplete without a complexity dimension: the scaling exponent itself may depend on task complexity.

Follow-Up Research This Work Enables

1. Map the full scale × complexity surface with continuous k. The paper's binary complexity manipulation (k = 2 vs. k = 5) identifies that a threshold exists but cannot locate it. A natural follow-up sweeps the number of coefficients continuously—k = 1, 2, 3, 4, 5, 6, 8, 10, 12—for each of 5+ models spanning 7B to 70B+ parameters, reporting both difference-to-target and exact match at every cell. This would reveal whether each model shows a sharp phase transition at some critical k (sudden drop from high exact match to zero) or a gradual degradation (exact match declining smoothly as k increases). The shape of the degradation curve has direct theoretical implications: a sharp transition suggests a representational bottleneck—the model either has the circuit for k-dimensional credit assignment or it doesn't—while a gradual decline suggests a continuous resource-limitation model where more complex functions consume more of a limited inference budget. The experiment requires generating many trials per (model, k) cell (at minimum hundreds, to get stable exact-match estimates at intermediate k where rates may be low) and should control for the number of demonstration examples (e.g., always provide 2k examples so the inference problem is equally well-posed across k).

2. Disentangle parameter count from training data scale and architecture. The paper compares three models that differ simultaneously in parameter count, training data, tokenizer, and architecture (dense vs. MoE). A rigorous follow-up would isolate parameter count by testing multiple model families at matched scales—for example, LLaMA2 7B, 13B, 70B alongside Mistral 7B, Mixtral 8×7B, and Yi-6B, Yi-34B—and looking for consistency in the k = 5 capability threshold across families. If all ~30B models fail at k = 5 and all ~70B models succeed, the threshold is robustly scale-driven. If some ~30B models succeed and others fail, training data or architecture matters more than raw parameter count. A particularly informative comparison would pit a 70B model trained on 2T tokens against a 34B model trained on 4T tokens (roughly matched total pretraining FLOPs) to test whether ICL emergence depends on parameters-per-se or total training compute. The paper's Mixtral result (succeeding at k = 5 with ~47B active parameters) hints that active parameters during inference, not total parameters including inactive experts, may be the relevant scale variable—testing dense models at ~47B (e.g., LLaMA2 40B if available) would test this directly.

3. Add zero-shot, scrambled-label, and optimal-solver baselines to establish that performance reflects ICL. The paper's interpretation that models are performing in-context learning—genuinely inferring the function from provided examples—is an assumption rather than an established fact. A critical follow-up adds three control conditions to the k = 2 and k = 5 experiments: (a) a zero-shot baseline where the model receives only the query input without any demonstration examples, testing whether models can produce plausible predictions from their pretrained priors alone; (b) a scrambled-label baseline where demonstration pairs are provided but the outputs are randomly shuffled relative to inputs, testing whether models actually use the input-output correspondence or just reproduce plausible output magnitudes; and (c) an optimal-solver ceiling where an explicit linear regression solver (least squares on the demonstration pairs) computes the exact weights and produces the optimal prediction, establishing the maximum achievable performance given the provided information. If model performance at k = 5 is substantially above the zero-shot and scrambled baselines but below the optimal-solver ceiling, the attribution to ICL is validated and the remaining gap quantifies the inference inefficiency. If performance at k = 5 is similar to the scrambled baseline, models are not learning from the examples at all—they are generating plausible guesses that happen to sometimes match. This control experiment is low-cost (it only adds prompt variants, no new models or training) and would dramatically strengthen or qualify the paper's conclusions.

4. Test generalization across function families: linear, polynomial, nonlinear, logical. The paper tests only linear weighted sums. A systematic follow-up would determine whether the observed scale-complexity interaction is specific to linear function inference or reflects a domain-general ICL capability. Test at least four function families: (a) polynomial functions (e.g., y = w₁x₁² + w₂x₂ + w₃x₁x₂), testing whether models can handle nonlinear feature combinations; (b) discontinuous functions (e.g., a threshold function where y depends on whether Σwᵢxᵢ exceeds some value, or a sorting/ranking task where the output is the index of the largest weighted input), testing whether the emergence is tied to continuous-valued regression; (c) logical functions (e.g., the output is a Boolean combination of thresholded inputs, akin to learning a decision tree from examples), testing whether the scale threshold differs for symbolic vs. numeric inference; and (d) compositional functions (e.g., y = f(g(x₁), h(x₂)) where f, g, h are independently weighted sums), testing whether the model can infer hierarchical structure from flat examples. For each family, sweep complexity (e.g., number of terms, depth of composition) and report continuous + discrete metrics. If the scale threshold is consistent across families at the same "effective complexity," that supports a domain-general ICL capability. If thresholds vary widely, ICL capability is function-family-specific and the paper's weighted-sum findings do not generalize.

5. Test whether additional in-context examples can substitute for model scale. The paper's k = 5 finding shows that Yi-34B fails at exact match given some fixed number of demonstration examples (unknown from the snippet, but presumably the minimum needed to constrain the weights). A critical follow-up tests whether sample efficiency—not capability—is the bottleneck: provide Yi-34B with substantially more demonstration examples at k = 5 (e.g., 20, 50, 100 examples instead of the default, presumably 5–10), and measure whether exact match improves. If Yi-34B achieves meaningful exact match when given many more examples, the scale threshold is about data efficiency (smaller models need more examples to reach the same inference quality) rather than a hard capability boundary. This would reframe the emergence narrative as "larger models are more sample-efficient at complex ICL" rather than "only large models can do complex ICL." If Yi-34B's exact match remains at zero regardless of example count, the capability is genuinely absent—the model lacks the representational or algorithmic machinery for high-dimensional credit assignment—and the emergence claim is strengthened. The experiment should also test the inverse: how few examples can LLaMA2 70B use at k = 5 and still succeed? If LLaMA2 70B achieves non-zero exact match with only 5 examples (the theoretical minimum), while Yi-34B fails even with 100, the capability gap is stark.

6. Mechanistic analysis: do models learn weights or memorize examples? The parametric function probe paradigm enables a type of analysis impossible with natural language ICL tasks: probing the model's internal representation of the inferred function. A mechanistic follow-up would design probes to test whether the model's hidden states encode the inferred weight vector ŵ = [ŵ₁, ..., ŵₖ] in a linearly decodable format. Specifically: (a) train linear probes on model hidden states at the position of each demonstration example to predict the true weight vector, testing whether the model builds up a weight representation incrementally; (b) at the query input position, decode the weight vector and compare the probe's predicted output (ŷ_probe = Ŵ_decoded · x_query) against the model's actual output, measuring whether the model's prediction is consistent with its internal weight representation; (c) perform activation patching (as in Wang et al., 2023; Meng et al., 2022) to identify which attention heads and MLP layers are causally responsible for weight inference, and whether these circuits are present-but-weak in Yi-34B or absent entirely. This would transform the paper's behavioral finding (models succeed/fail at k = 5) into a mechanistic account (the circuit for k-dimensional credit assignment emerges at ~70B parameters). A negative result—no linearly decodable weight vector, or inconsistent relationship between decoded weights and predictions—would suggest models are using a non-algorithmic, pattern-matching strategy for ICL even when they succeed, which would qualify the interpretation that they are "learning the function" in any meaningful sense.

Practical Applications and Downstream Use Cases

Cost-efficient model selection for ICL-heavy production pipelines. The paper's cross-over finding—that Yi-34B achieves lower prediction error than LLaMA2 70B on simple (k = 2) ICL tasks—has direct implications for deployment decisions. An organization that uses in-context learning for a task where the effective complexity is low (e.g., extracting structured fields from templated documents, where the mapping from input text to output format requires tracking only a few contextual cues) should prefer the 34B model over the 70B model. The 34B model is cheaper per token (roughly half the inference cost in FLOPs), lower latency, and—if the paper's difference-metric finding generalizes—more accurate. The paper's results suggest that deploying the 70B model for simple ICL tasks may be actively worse than deploying the cheaper model. Conversely, for tasks requiring high-dimensional pattern inference (e.g., learning a complex scoring rubric from a handful of annotated examples, where many input features contribute independently to the output), the paper's k = 5 finding provides evidence that the 70B model is not merely better—it is necessary, because the 34B model achieves approximately zero success rate. This creates a concrete decision rule: profile the target task's effective complexity (e.g., by measuring how many independent input dimensions affect the output, or by testing a small model on a pilot set), and route queries below the complexity threshold to the smaller model.

Benchmark design with continuous metrics to avoid emergence artifacts. The paper's dual-metric diagnostic (continuous difference + discrete exact match) provides a practical template for benchmark designers who want to avoid the measurement artifact problem identified by Schaeffer et al. (2024). When constructing a new benchmark for evaluating LLM capabilities—especially capabilities where emergence is anticipated—include both a continuous performance measure and a discrete accuracy measure, and report both at every model scale tested. For a math reasoning benchmark, the continuous measure might be the fraction of intermediate steps completed correctly (partial credit) or the negative log-likelihood of the correct answer under the model's output distribution. For a code generation benchmark, it might be the edit distance from the generated code to a correct solution, or the fraction of test cases passed (continuous from 0 to 1). Reporting both metrics preempts the emergence-as-artifact critique: if the continuous metric improves smoothly across scales while the discrete metric jumps, the benchmark's authors can explicitly note that the apparent emergence may be a threshold effect on a gradually improving capability. This shifts benchmark design from a binary "did the model get it right?" mindset to a graded "how close did the model get?" framework that is more informative for tracking progress and comparing models.

Synthetic data generation for fine-tuning ICL capability. The parametric function probe is not only an evaluation tool but also a data generation engine for fine-tuning. A practitioner who finds that their deployed model fails at the required complexity level (e.g., a 34B model failing at k = 5 equivalent tasks) could use the weighted-sum task—or more complex synthetic function families—to generate a large dataset of ICL training examples with known ground truth. Fine-tuning the model on this synthetic ICL data, where the model receives demonstration examples and must predict the output, could teach the model the underlying inference algorithm (multivariate credit assignment) that it failed to learn during pretraining. The advantage of synthetic data is scalability: millions of unique functions can be generated programmatically with zero labeling cost, and complexity can be curriculum-controlled (start fine-tuning on k = 2, then progress to k = 3, 4, 5). This is a direct application of the paper's paradigm: what the paper uses to diagnose a capability gap, a practitioner can use to train away that gap. The effectiveness of this approach—whether synthetic ICL fine-tuning transfers to natural language ICL tasks—would itself be a valuable research contribution, but the paper's framework makes the experiment straightforward to design.