Online Experiential Learning for Language Models

ArXiv: 2603.16856

🎯 Pitch

Language models can now keep learning after deployment without any reward signals or environment access. By distilling transferable rules and strategies from raw text feedback during real-world use, this method doubled task success rates while cutting response length by 30%.

1. Executive Summary

This paper introduces Online Experiential Learning (OEL), a reward-free framework that enables language models to continuously improve from their own deployment experience by iterating over two stages: extracting transferable experiential knowledge from interaction trajectories (accumulating insights like game rules and winning strategies from raw text feedback) and consolidating that knowledge into model parameters via on-policy context distillation (training the model to match a knowledge-conditioned teacher using reverse KL divergence, without requiring server-side environment access). Evaluated on text-based game environments (Frozen Lake and Sokoban in TextArena) using Qwen3 models at 1.7B, 4B, and 8B scales across both thinking and non-thinking variants, OEL achieves consistent improvements over successive online learning rounds—for example, more than doubling pass rate on Frozen Lake from the initial model to Round 2 consolidation—while also improving token efficiency (response lengths decreasing to roughly 70% of the initial length by the third iteration) and preserving out-of-distribution performance on IF-Eval, establishing that extracted experiential knowledge is significantly more effective than raw trajectories and that on-policy consistency between the knowledge source and the policy model is critical for effective learning.

2. Context and Motivation

The Core Problem: Models Stop Learning at Deployment

The fundamental problem this paper addresses is structural: current large language models are static artifacts—they are trained on pre-constructed data and then frozen at deployment, gaining nothing from the vast stream of real-world interactions that occur afterward. The paper articulates this directly in Section 1:

"Once deployed, the model encounters a vast, ever-evolving landscape of real-world tasks and user needs, yet gains nothing from these interactions. The rich stream of experience accumulated during deployment is simply discarded."

This is not merely an inefficiency—it represents a ceiling on what models can become. The dominant paradigm (illustrated in Figure 2, left) relies entirely on offline training with pre-constructed data: supervised fine-tuning with human annotations, or reinforcement learning with verifiable rewards or reward models in simulated environments. The model's capabilities are bounded by the data and environments curated before deployment. No matter how broadly the model is deployed or how many interactions it has, it cannot improve unless someone explicitly constructs new training data.

This gap matters for several reasons the paper develops:

• Coverage impossibility: Real-world deployment exposes models to an open-ended stream of environments, tasks, and user demands that "far exceed what any controlled training setting can anticipate" (Section 2). No amount of offline data curation can fully cover this diversity, because the space of possible interactions is effectively infinite and evolves over time.

• Annotation bottleneck: Scaling offline training requires increasingly costly human annotations or increasingly faithful simulations, creating an economic ceiling. Every new domain, every new capability, every adaptation to shifting user behavior requires a fresh annotation campaign or simulation pipeline. This does not scale.

• Wasted signal: The model's own interactions with real environments generate rich textual feedback—natural language descriptions of outcomes, errors, state changes, successes—that contains information about what works and what doesn't. This signal is generated for free during normal usage but is completely unused for learning.

The Server-User Disconnect and the Reward Problem

The paper identifies two specific technical barriers that make online learning from deployment experience difficult in practice (Section 1):

Barrier 1: The server side cannot access user-side environments. Training happens on the server, but the environments the model interacts with (applications, tools, games, user interfaces) live on the user side. The server cannot run these environments directly—it cannot execute code in a user's browser, replay their game state, or interact with their tools. This means standard online learning approaches that require environment interaction during training (e.g., rolling out a policy to collect new trajectories, getting environment rewards) are infeasible. The model's training process must somehow learn from user-side experience without being able to access the environment itself.

Barrier 2: Real-world interactions provide textual feedback, not scalar rewards. When a language model interacts with a real environment—a game, a tool, a dialogue—the environment rarely returns a clean scalar reward signal. Instead, it returns natural language: "you hit a wall," "the file failed to compile with error X," "the box is now on the target." Standard reinforcement learning algorithms are designed to consume scalar reward signals and cannot directly process unstructured text. The paper states:

"Constructing verifiable reward functions or training reward models for every new deployment scenario is impractical."

If every new deployment required building a reward model, the annotation or engineering burden would defeat the purpose. The challenge is to enable learning from textual environment feedback alone—the natural output of whatever environment the model happens to be deployed in.

These two barriers together define the paper's core research question: can a language model continuously improve from its own deployment experience using only the textual feedback that environments naturally provide, without requiring server-side environment access, reward models, or human annotations?

Where Prior Approaches Fall Short

The paper positions its work against three categories of prior approaches, each of which addresses some aspect of the problem but falls short of providing a complete solution for online learning from deployment experience.

Offline Training with Pre-Constructed Data (The Dominant Paradigm)

The prevailing approach to improving LLMs is entirely offline: collect human annotations for supervised fine-tuning (SFT), or construct simulated environments with verifiable rewards for reinforcement learning (RL). This paradigm is effective for targeted optimization—the paper cites DeepSeekMath (Shao et al., 2024) and DAPO (Yu et al., 2025) as examples of successful offline RL for reasoning—but it has a fundamental scalability ceiling: "performance saturates on the curated training distribution, and further scaling requires increasingly costly annotations or increasingly faithful simulations" (Section 2). The trained model is a static artifact; it cannot adapt to new domains without a new offline training campaign.

What's missing: These methods make no use of the experience the model accumulates during deployment. They treat deployment as the endpoint of training, not as a source of learning signal.

Off-Policy Context Distillation (Compressing In-Context Knowledge)

Context distillation methods (Askell et al., 2021; Snell et al., 2022; Cao et al., 2025) aim to compress in-context knowledge into model parameters. The typical approach: a teacher model is given useful context (documents, examples, instructions) and generates responses; a student model is trained via forward KL divergence to imitate those responses without seeing the context. This removes the need to provide lengthy contexts at inference time.

What's missing (Section 5, Figure 6): These off-policy methods suffer from two limitations. First, they use mode-covering forward KL divergence, which encourages the student to cover all modes of the teacher's distribution—including modes the student lacks capacity to represent well, leading to degraded performance. Second, and more critically for the online learning setting, off-policy training on teacher-generated data causes a distribution mismatch between training and inference. The paper's experiments (Figure 6) directly demonstrate this: off-policy context distillation causes clear degradation on out-of-distribution tasks (IF-Eval accuracy drops from ~66% to ~65% over training), whereas on-policy context distillation largely preserves OOD performance.

These methods also presuppose that useful context already exists. They do not address the problem of generating that context from raw deployment experience—the extraction stage that OEL introduces.

Learning from Experience (Reflection, Memory, Self-Play)

Several prior works have explored aspects of learning from interaction experience, but each addresses only part of the full online learning loop. The Reflexion framework (Shinn et al., 2023) prompts models to reflect on past failures to guide future attempts—a form of in-context learning from experience. Expel (Zhao et al., 2024) extracts insights from trajectories and stores them in external memory for retrieval. Cogito, Ergo Ludo (Wang et al., 2025) demonstrates that reasoning-based agents can discover game strategies through self-play and reflection.

What's missing: These methods operate entirely at inference time—the extracted knowledge is used in-context (prepended to the prompt) but is never internalized into model parameters. As the paper demonstrates (Table 1, "In-Context" column), in-context use of extracted knowledge provides substantial gains over no experience (18.2% pass rate vs. 7.5% on Sokoban), but these gains are bounded by the context window and do not compound across deployment rounds. Furthermore, the paper shows that in-context accumulation of experiential knowledge eventually saturates (the transparent curves in Figure 4): "as the experiential knowledge accumulates, the context window becomes increasingly occupied, limiting the model's capacity to absorb and leverage additional knowledge through in-context learning alone." Reflection-based and memory-based methods hit this saturation ceiling because they never convert experience into permanent parameter updates.

The position paper by Silver and Sutton (2025), which the paper cites in Section 5, argues that "agents should primarily learn from their own interaction with the world rather than from human-curated data, heralding an era of experience." This provides conceptual motivation for OEL but does not provide a concrete technical framework for doing so with language models.

On-Policy Distillation (Closing the Distribution Gap)

On-policy distillation methods (Gu et al., 2024; Agarwal et al., 2024; Lu, 2025) train student models on their own generated trajectories rather than on teacher-produced data, minimizing reverse KL divergence instead of forward KL divergence. Reverse KL encourages mode-seeking behavior—the student focuses on matching the teacher's most likely outputs rather than covering all modes—which is more appropriate when the student has limited capacity relative to the teacher.

What's missing: On-policy distillation addresses the distribution mismatch problem but does not provide a mechanism for generating the teacher's knowledge from deployment experience. It assumes a teacher with useful knowledge already exists. OEL builds directly on the on-policy context distillation framework from Part I of this Experiential Learning series (Ye et al., 2026) but adds the crucial extraction stage that converts raw trajectories into the experiential knowledge that the teacher uses.

How This Paper Positions Itself

The paper positions OEL as filling a specific gap in this landscape: it provides the first end-to-end framework for online learning from deployment experience that works with textual environment feedback alone, requires no reward models or human annotations, and can be iterated to compound improvements over successive deployment rounds.

The key conceptual move is the extraction-consolidation decomposition (Section 3, Figure 3). By separating the problem into two stages—first extract transferable knowledge from raw trajectories, then consolidate that knowledge into parameters—OEL addresses both barriers identified above:

• The server-user disconnect is resolved because consolidation uses on-policy context distillation with single-turn rollouts from partial trajectory prefixes, requiring only the pre-collected trajectories from the user side, not live environment access (Section 3.2: "the entire training procedure can be carried out on the server side without access to the user-side environment").

• The textual-feedback problem is resolved because the extraction stage uses the language model itself (π_extract) to process raw environment text into structured experiential knowledge, and the consolidation teacher is conditioned on this knowledge to provide dense token-level training signal—no scalar reward is needed (Section 3.2: "the experiential-knowledge-conditioned teacher provides dense, token-level training signal derived solely from textual environment feedback").

The paper explicitly contrasts its vision with the offline paradigm in Figure 2: offline training operates in a "closed world" with pre-constructed data, while online experiential learning forms "a virtuous cycle during deployment" where "deployment and learning are thus connected—the broader the deployment, the richer the signal for continued improvement." This is not merely an incremental improvement to existing methods but a proposed shift in the fundamental relationship between deployment and learning.

The paper also builds explicitly on Part I of the Experiential Learning series (Ye et al., 2026), which introduced on-policy context distillation. Part I focused on the consolidation mechanism—how to internalize in-context knowledge into parameters using reverse KL divergence with on-policy sampling. Part II (this paper) adds the extraction stage and the iterative online learning loop, making the framework capable of generating its own knowledge from raw deployment experience rather than relying on pre-existing context.

Importantly, the paper does not claim to have solved all challenges of online learning. It acknowledges several limitations implicitly through its experimental design: the environments are text-based games with clear success/failure signals (though presented as text, not scalar rewards), the model collects trajectories in batches between training rounds rather than truly continuously, and the extraction and consolidation stages require hyperparameter choices (number of trajectories, training steps, knowledge format) that are fixed across rounds. These represent practical simplifications for a first demonstration rather than fundamental constraints of the framework.

3. Technical Approach

3.1 Reader Orientation

Online Experiential Learning (OEL) is a two-stage training loop that runs on the server side, consuming only the text logs that a language model naturally produces during user-facing deployment, and outputting an improved version of that model. The system solves the problem of learning from deployment without environment access or reward signals by first converting raw interaction transcripts into concise "experiential knowledge" (a distillation stage that is purely text-to-text), and then baking that knowledge into the model weights (a consolidation stage that teaches the model to behave as if it had that knowledge in-context, without actually needing it at inference time).

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components arranged in a cycle (Figure 3):

1. The deployed model $\pi_\theta$ — the language model being improved. It interacts with the user-side environment, producing multi-turn trajectories of (environment feedback, model action) pairs.
2. The knowledge extractor $\pi_{\text{extract}}$ — a language model (by default, the deployed model itself) that processes the raw trajectories and produces concise "experiential knowledge" summarising what was learned (rules, strategies, failure modes). It operates accumulatively: each new trajectory is processed in the context of previously extracted knowledge.
3. The experiential knowledge accumulator — a persistent store of extracted knowledge items, built up over multiple trajectories. It provides the conditioning context for the teacher model during consolidation.
4. The teacher model $\pi_{\text{teacher}}$ — a frozen copy of the initial model, given the accumulated experiential knowledge as context. It provides a dense, token-level supervision signal showing what the model should do if it had access to that knowledge.
5. The consolidation trainer — an on-policy context distillation procedure that takes partial trajectory prefixes (without environment access), has the student model $\pi_\theta$ generate responses, and trains it to match the knowledge-conditioned teacher via token-level reverse KL divergence.

Information flows cyclically: Deployment → Trajectory collection → Extraction (trajectories → experiential knowledge) → Consolidation (knowledge + partial prefixes → updated model) → Re-deployment → Higher-quality trajectories → Richer extraction → ...

3.3 Roadmap for the Deep Dive

• First, the formal trajectory and extraction formalism (Equation 1) — this defines what "experiential knowledge" means, how it is accumulated, and how the extraction process scales with the number of trajectories.
• Second, the consolidation objective (Equations 2–4) — this is the core learning mechanism, so we need to understand the on-policy context distillation loss, the reverse KL divergence, and the teacher-student setup before we can discuss the full loop.
• Third, the online learning loop (Algorithm 1) — with extraction and consolidation defined, we walk through the full iterative procedure, showing how it naturally forms a virtuous cycle.
• Fourth, the knowledge formats and hyperparameter configurations — the concrete engineering choices (structured vs. unstructured extraction, accumulation counts, training steps) that make the framework work in practice.
• Fifth, the partial rollout prefix mechanism — the specific trick that enables on-policy training without server-side environment access, which is the key solution to the server-user disconnect barrier.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an algorithmic systems paper whose core idea is that language models can improve continuously from deployment experience by decomposing the learning problem into two stages — a pure text-to-text extraction stage that converts raw interaction logs into reusable knowledge, and a consolidation stage that internalizes that knowledge into model parameters using on-policy context distillation — and then iterating these stages to form an online learning loop that compounds gains.

The Trajectory Formalism and What OEL Works With

Before explaining the extraction mechanism, we need to define what the raw data looks like. The paper assumes the model $\pi_\theta$ is deployed to interact with a user-side environment $E$. This environment could be a game, a tool, a code interpreter, a dialogue interface — anything that accepts the model's text output as an action and returns text feedback. Crucially, the environment is opaque to the server: the server cannot instantiate it, query it, or reset it. The only thing the server receives is a collection of trajectories — logs of the model's interactions.

Each trajectory $\tau_i$ is defined as an alternating sequence:

$\tau_i = (f_i^1, a_i^1, f_i^2, a_i^2, \ldots)$

where $f_i^j$ is the textual environment feedback at turn $j$ of episode $i$ (e.g., "You moved up. The path ahead is clear. Current position: (2, 1)"), and $a_i^j$ is the model's action at turn $j$ (e.g., "[up]" or a longer reasoning-plus-action response).

What this represents: a complete multi-turn interaction history. There is no scalar reward, no success/failure label attached by the environment — just the raw text that the model saw (environment output) and produced (its response). The trajectory is purely observational.

Why this matters: OEL must work with exactly this data format — no more — because this is what real-world deployments naturally produce. The framework cannot assume access to ground-truth labels, reward signals, or environment resets. If the learning mechanism required anything beyond $\{ \tau_i \}$, it would be impractical for most deployment scenarios.

For training, the paper collects two separate sets of trajectories from the same environment, both using the current model $\pi_\theta$:

• $T = \{\tau_1, \tau_2, \ldots, \tau_n\}$: used in the extraction stage to produce experiential knowledge (Section 3.1). Typically $n = 15$, $25$, or $50$ depending on configuration (Table 4).
• $T' = \{\tau_1, \tau_2, \ldots, \tau_m\}$: used in the consolidation stage to create training prefixes (Section 3.2). The number of trajectories $m$ is determined by the required number of training samples: each consolidation round uses $20$ or $100$ training steps with $64$ game samples per step, requiring $1280$ or $6400$ total trajectory samples (Appendix B.4). Each trajectory spans up to $5$ turns.

Both sets are collected during the same deployment window, from the same environment, with the same model. The $T$ set is used for knowledge generation; the $T'$ set is used for training data. They serve different purposes and have different size requirements.

The Extraction Mechanism: From Raw Trajectories to Accumulated Experiential Knowledge

The extraction stage (Section 3.1) converts the raw trajectory set $T$ into a set of experiential knowledge $C = \{e_1, e_2, \ldots, e_K\}$ where each $e_k$ is one accumulated knowledge sequence produced by running the extraction process with a different random seed. This is the paper's first key innovation: instead of feeding raw trajectories directly into training (which Table 1 shows is ineffective — only 10.9% pass rate in-context vs. 18.2% for extracted knowledge), the system uses the language model itself to distill trajectories into concise, transferable knowledge.

The extraction model. By default, $\pi_{\text{extract}} = \pi_\theta$ — the deployed model itself serves as the extractor. This means the model that generated the trajectories is also the one that interprets them, creating an on-policy knowledge extraction loop. If the model is a thinking model (e.g., Qwen3-1.7B with reasoning traces), thinking mode is enabled during extraction, but only the final answer part is retained as experiential knowledge while the reasoning chains are removed (Section 4.1, Extraction Stage). This filtering is important because the experiential knowledge needs to be concise enough to fit in the teacher's context window during consolidation, and reasoning traces would consume that budget without adding transferable value.

The accumulation process. Extraction is not done independently per trajectory. Instead, the system processes trajectories sequentially, accumulating knowledge as it goes. Let $e_i$ be the accumulated experiential knowledge after processing trajectory $\tau_i$, with $e_0 = \emptyset$ (empty at the start). The recursive definition is:

$e_i' \sim \pi_{\text{extract}}(\cdot \mid \tau_i, e_{i-1})$

$e_i = [e_{i-1}; e_i']$

where $\pi_{\text{extract}}(\cdot \mid \tau_i, e_{i-1})$ denotes sampling from the extraction model conditioned on the current trajectory $\tau_i$ and all previously accumulated knowledge $e_{i-1}$, and $[e_{i-1}; e_i']$ denotes string concatenation.

What this computes: For each trajectory in order, the extraction model produces new knowledge items $e_i'$ by reasoning about what can be learned from $\tau_i$ that hasn't already been captured in $e_{i-1}$. The prompt explicitly instructs: "Do not repeat the previous experience. Make sure the newly generated experience is different from the previous experience" (Figure 10). The new items are appended to the accumulated store, which then conditions the extraction for the next trajectory.

Why this recursive form over batch extraction: Processing trajectories one at a time with accumulated context serves two purposes. First, it encourages minimal redundancy: each new extraction builds on what's already known, so the extractor can focus on identifying novel insights rather than re-stating rules it already captured. Second, it enables scaling to many trajectories: if the system processes $n$ trajectories (where $n = 15$, $25$, or $50$ in the experiments), the final accumulated knowledge $e_n$ can contain insights synthesized across all of them, potentially capturing patterns that no single trajectory would reveal. A batch approach that processes all trajectories simultaneously would hit the context window limit much sooner and couldn't build cross-trajectory insights iteratively.

Knowledge formats. The paper supports two formats for extracted knowledge (Section 4.1, Extraction Stage):

• Structured format: The extraction model is prompted to produce knowledge as a markdown list where each item is prefixed with "- EXPERIENCE ITEM:" (Figure 10). Only lines matching this prefix are retained; everything else is discarded. A typical item reads: "- EXPERIENCE ITEM: The game's core mechanic is 'axis-aligned convergence': at every move, the player must reduce the Manhattan distance to the goal by progressing directly along the shared row or column" (Figure 13). This format enforces compactness and parseability, and works well when the knowledge can be expressed as a discrete list of rules or strategies. The experiments use $n = 25$ or $n = 50$ trajectories with a maximum generation length $L_{\max} = 8192$ tokens for both each extraction step and the accumulated knowledge (content exceeding this limit is truncated).

• Unstructured format: The extraction model generates knowledge freely without formatting constraints (Figure 11). This is more flexible — the model can produce prose, structured explanations, or any format that best captures the insight. The prompt still instructs against repetition but doesn't enforce a specific output template. This format is used with $n = 15$ trajectories and $L_{\max} = 2048$ tokens. The specific configurations per experiment are listed in Table 4: Qwen3-1.7B on Frozen Lake uses unstructured, while the larger models on Frozen Lake and Sokoban use structured.

Why two formats exist: The smaller model (1.7B) uses unstructured because "smaller models lack sufficient capacity to effectively leverage long contextual information" (Appendix B.4, Table 4 footnote) — for Qwen3-1.7B, the system even omits previously accumulated experiential knowledge from the extraction context entirely for this reason. The structured format with longer context is effective for larger models that can self-regulate their output within formatting constraints.

Multiple accumulation seeds. The accumulation process is repeated $K = 10$ times with different random seeds (Section 4.1). Each run produces a potentially different accumulated knowledge sequence $e_n$, because the extraction model's sampling is stochastic (temperature is used during extraction) and the trajectory ordering within $T$ may vary. This yields a set $C = \{e_1, e_2, \ldots, e_{10}\}$ of accumulated knowledge variants. During consolidation, experiential knowledge $e$ is randomly sampled from $C$ for each training prefix. The paper explicitly notes that "since the extraction process is performed server-side and we do not require scalar reward signals from the environment, we do not select the optimal experiential knowledge and instead retrieve the knowledge at the fixed accumulation step across OEL rounds" (Section 4.1). This avoids the need for a validation signal to pick the "best" knowledge — the diversity across seeds provides a form of data augmentation.

What the extraction prompt contains. For the structured format (Figure 10), the prompt includes:

• The full multi-turn interaction history for the current trajectory (latest_experience): all environment outputs and model responses, including thinking traces when available.
• The previously accumulated experiential knowledge (previous_experience): the concatenation of all items from trajectories 1 through $i-1$.
• An explicit instruction to "conduct a deep, comparative analysis to infer the game rules and the fundamental principles behind winning and losing," and to generate "1-2 concise, high-level, and widely applicable experience items."
• Formatting rules requiring the "- EXPERIENCE ITEM:" prefix.

Design choice — extraction model as the deployed model: Using $\pi_\theta$ as $\pi_{\text{extract}}$ means the model that generated the trajectories is analyzing its own behavior. This is a form of self-reflection, but crucially, it's structured: rather than asking the model to "reflect on what went wrong" (as in Reflexion, Shinn et al., 2023), the extractor is given the full trajectory history and asked to produce general, transferable knowledge. The paper's analysis (Table 2) demonstrates that this on-policy extraction is superior to using knowledge from a different model: experiential knowledge extracted from Qwen3-1.7B's own trajectories yields a 31.1% consolidated pass rate, while knowledge from a larger Qwen3-4B model yields only 22.7% — even though the 4B model is objectively stronger. The hypothesis is that larger-model knowledge "may encode strategies beyond the smaller model's capabilities," making it less actionable. On-policy extraction ensures the knowledge is matched to what the deployed model can actually execute.

The Consolidation Mechanism: On-Policy Context Distillation

After extraction produces the experiential knowledge set $C$, the consolidation stage (Section 3.2) internalizes this knowledge into the model parameters. The mechanism is on-policy context distillation, which builds on the framework from Part I (Ye et al., 2026) and is described in detail in Appendix A.

The core idea: We have a student model $\pi_\theta$ (the model being trained) and a teacher model $\pi_{\text{teacher}}$ (a frozen copy of the initial model before training). The teacher is given the experiential knowledge $e$ as context and can therefore produce better responses. The student is trained to produce those same better responses without seeing the knowledge context, by minimizing the reverse KL divergence between its output distribution and the teacher's knowledge-conditioned output distribution.

Why distillation and not supervised learning on teacher outputs? Standard (off-policy) context distillation works by: (1) have the teacher generate responses with knowledge in context, (2) train the student via forward KL divergence to imitate those responses. The paper's Figure 6 demonstrates that this approach is inferior on two fronts: it achieves lower in-distribution performance (pass rate around 35% vs. 45% at 2500 training samples) and causes out-of-distribution degradation (IF-Eval accuracy drops from ~66% to ~65%). The on-policy approach — where the student generates its own responses and is trained to match the teacher's token-level probabilities on those responses — avoids the distribution mismatch between training data (teacher-generated) and inference data (student-generated).

The partial rollout prefix mechanism. The key practical challenge is: how do we train on-policy without access to the user-side environment? The answer is the partial rollout prefix construction. From each trajectory $\tau_i$ in $T'$, the system extracts all prefixes of the form:

$x_i^j = (f_i^1, a_i^1, f_i^2, a_i^2, \ldots, f_i^{j-1}, a_i^{j-1}, f_i^j)$

This is the interaction history up to but not including the $j$-th model response. The full set across all trajectories and all turn positions forms the training dataset $D = \{x_i^j\}$.

What this represents operationally: A prefix $x_i^j$ is exactly the text the model would see at turn $j$ of episode $i$ — all prior environment feedback and model actions, plus the new environment feedback from the current turn. When we feed this prefix to the model during training, it produces a response $y \sim \pi_\theta(\cdot \mid x_i^j)$ — a single-turn generation conditioned on the history. This is on-policy because the current model $\pi_\theta$ is generating the response. It requires no environment access because we use the environment feedback recorded in the trajectory as text; we don't need to re-query the environment.

Why this works for any deployment: The only requirement on the user side is to collect and send back the trajectory text. The server can reconstruct every partial state of the interaction from the text log alone and use those as training inputs. This is the paper's solution to Barrier 1 (the server-user disconnect).

The teacher model setup. For each training step, we sample a prefix $x$ from $D$ and experiential knowledge $e$ from $C$ (randomly, from among the $K = 10$ accumulated variants). The teacher is $\pi_{\text{teacher}}$ — a frozen copy of the initial model $\pi_\theta$ before any OEL training in the current round. The teacher receives the experiential knowledge prepended to the prefix: its conditioning context is $(e, x)$. The student $\pi_\theta$ receives only $x$.

The training objective. The loss function for on-policy context distillation is the token-level reverse KL divergence:

$\mathcal{L}(\theta) = \mathbb{E}_{x \sim D, e \sim C, y \sim \pi_\theta(\cdot \mid x)} \left[ \frac{1}{|y|} \sum_{t=1}^{|y|} D_{\text{KL}} \left( \pi_\theta(\cdot \mid x, y_{<t}) \;\|\; \pi_{\text{teacher}}(\cdot \mid e, x, y_{<t}) \right) \right]$

where:

• $x \sim D$ is a partial rollout prefix sampled from the training dataset.
• $e \sim C$ is an experiential knowledge sequence sampled uniformly from the accumulated set.
• $y \sim \pi_\theta(\cdot \mid x)$ is a full response generated by the current student model, sampled token by token from its own distribution.
• $|y|$ is the length of the generated response in tokens.
• $\pi_\theta(\cdot \mid x, y_{<t})$ is the student's predicted next-token distribution at position $t$, conditioned on the prefix $x$ and the tokens generated so far $y_{<t}$.
• $\pi_{\text{teacher}}(\cdot \mid e, x, y_{<t})$ is the teacher's predicted next-token distribution at position $t$, conditioned on the experiential knowledge $e$, the prefix $x$, and the tokens generated so far.
• $D_{\text{KL}}(P \| Q)$ is the Kullback-Leibler divergence from $Q$ to $P$.

What this loss computes in operation: For each training step:

1. Sample a prefix $x$ and knowledge $e$.
2. Have the student generate a full response $y$ autoregressively, collecting the sequence of generated tokens.
3. For each position $t$ in the generated response:
   • Compute the student's full next-token probability distribution $\pi_\theta(\cdot \mid x, y_{<t})$ over the vocabulary.
   • Compute the teacher's full next-token probability distribution $\pi_{\text{teacher}}(\cdot \mid e, x, y_{<t})$ over the vocabulary, crucially with $e$ in its context.
   • Compute the KL divergence between these two distributions: $\sum_{v \in \mathcal{V}} \pi_\theta(v \mid ...) \cdot (\log \pi_\theta(v \mid ...) - \log \pi_{\text{teacher}}(v \mid ...))$.
4. Average these token-level divergences across all positions in the response.
5. Average across the sampled $(x, e)$ pairs in the batch.

The result is a single scalar loss. Minimizing it pushes the student's output distribution toward the teacher's knowledge-conditioned distribution, at every token position, on the tokens the student itself chose to generate.

Why this form — the token-level reverse KL decomposition: The paper uses the sequence-level reverse KL divergence $D_{\text{KL}}(\pi_\theta(\cdot \mid x) \| \pi_{\text{teacher}}(\cdot \mid e, x))$, decomposed into a sum of token-level divergences. This decomposition is valid because the KL divergence between two autoregressive distributions factorizes into the expected sum of token-level KLs when both distributions are defined over the same token sequence. Formally:

$D_{\text{KL}}(\pi_\theta(y \mid x) \| \pi_{\text{teacher}}(y \mid e, x)) = \mathbb{E}_{y \sim \pi_\theta} \left[ \sum_{t=1}^{|y|} D_{\text{KL}}(\pi_\theta(y_t \mid x, y_{<t}) \| \pi_{\text{teacher}}(y_t \mid e, x, y_{<t})) \right]$

Why this decomposition matters for on-policy training: If we used the sequence-level KL directly, the student's generation $y$ would be treated as a black box, and we could only compute a single divergence score per sequence. The token-level decomposition provides a dense training signal: at every token position, the student gets a gradient pushing its distribution toward the teacher's. This is substantially more informative than a single scalar per sequence, especially for long responses. The paper explicitly notes that "the experiential-knowledge-conditioned teacher provides dense, token-level training signal" (Section 3.2) — the "dense" refers to this per-token granularity.

The top-k approximation. Computing the full KL divergence over the entire vocabulary $\mathcal{V}$ (typically 50K–250K tokens) at every position would be prohibitively expensive. The paper uses a practical approximation (Appendix A):

$D_{\text{KL}}(\pi_\theta(\cdot) \| \pi_{\text{teacher}}(\cdot)) \approx \sum_{v \in \mathcal{V}_{\text{top-}k}} \pi_\theta(v) (\log \pi_\theta(v) - \log \pi_{\text{teacher}}(v))$

where $\mathcal{V}_{\text{top-}k}$ is the set of $k$ vocabulary tokens with the highest probability under the student model $\pi_\theta$. Throughout all experiments, $k = 256$.

What this approximation does: Instead of summing over all ~150K tokens (for a typical Qwen vocabulary), we sum only over the 256 tokens the student considers most likely. The student's probability mass outside these top-256 tokens is typically very small (the tail is diffuse), and the teacher's probabilities for those rare tokens are similarly small, so the contribution to the KL divergence is negligible. This reduces the computational cost by roughly $|\mathcal{V}| / k \approx 500\times$ while introducing minimal approximation error.

Why reverse KL and not forward KL: The choice of KL direction is mathematically critical. Forward KL is $D_{\text{KL}}(\pi_{\text{teacher}} \| \pi_\theta)$, which corresponds to mode-covering behavior: it heavily penalizes the student for assigning low probability to any token the teacher considers likely. This forces the student to spread its probability mass across all teacher modes, including modes the student may lack capacity to represent well, leading to degraded quality. Reverse KL is $D_{\text{KL}}(\pi_\theta \| \pi_{\text{teacher}})$, which corresponds to mode-seeking behavior: it penalizes the student most for assigning high probability to tokens the teacher considers unlikely. This allows the student to focus on a subset of the teacher's modes that it can represent well, producing higher-quality outputs. The paper's comparison in Figure 6 confirms that reverse KL (on-policy) achieves both higher in-distribution performance and better OOD preservation than forward KL (off-policy).

The teacher is frozen. The paper states: "We use the frozen initial $\pi_\theta$ before training as $\pi_{\text{teacher}}$ in this work" (Section 3.2). This means the teacher is a snapshot of the model before the current round's consolidation, given knowledge context. The teacher never updates during consolidation. This is standard distillation practice: if the teacher also updated, the target would be a moving one, leading to training instability. The teacher's role is to provide a stable, knowledge-augmented reference distribution.

Connection to RL. The reverse KL objective can be interpreted as a form of reward-free reinforcement learning. The teacher model acts as a dense reward function: at each token position, the log-ratio $\log \pi_{\text{teacher}}(v) - \log \pi_\theta(v)$ (inside the KL) gives an implicit reward for generating tokens the teacher considers likely. Minimizing the reverse KL is equivalent to maximizing this expected log-ratio under the student's distribution, which is a policy gradient objective with the teacher as the reward model. The paper does not frame it this way, but the connection is important: OEL achieves RL-like behavior (improving the policy from environment experience) without ever requiring an explicit reward function.

The Online Learning Loop: Iterating Extraction and Consolidation

With extraction and consolidation defined, the full OEL algorithm (Algorithm 1) is a straightforward iteration. The pseudocode is reproduced and explained below, with details filled in from the text.

Algorithm 1 walkthrough:

while Online Learning do
    [User Side]
    Collect trajectories T = {τ₁, ..., τₙ} and T' = {τ₁, ..., τₘ} from E using π_θ
    
    [Server Side]
    // Stage 1: Extract Experiential Knowledge
    Set π_extract = π_θ
    Accumulate experiential knowledge C = {e₁, ..., e_K} on T using π_extract
    // Stage 2: Consolidate Experiential Knowledge into Model Weights
    Construct partial rollout prefixes D = {x_i^j} from T'
    Set π_teacher = π_θ and keep it frozen
    for batch x ~ D, e ~ C do
        Sample response y ~ π_θ(· | x)
        L(θ) ← 1/|y| Σ_t D_KL(π_θ(· | x, y_{<t}) || π_teacher(· | e, x, y_{<t}))
        Update θ by minimizing L(θ)
    end for
    Transfer updated π_θ to user side
end while

What changes across iterations: The critical dynamic is that $T$ and $T'$ are collected using the current model $\pi_\theta$. After the first round of consolidation, $\pi_\theta$ is improved — it solves more tasks successfully and does so more efficiently (shorter responses, Figure 5). When this improved model is redeployed to collect the next round's trajectories:

• $T$ contains higher-quality interactions: more successful episodes, fewer repeated mistakes, more efficient solution paths.
• Extraction from these higher-quality trajectories yields richer experiential knowledge: the model has explored more of the task space, discovered better strategies, and can articulate more sophisticated rules.
• Consolidation with this richer knowledge further improves $\pi_\theta$.

This is what the paper means by "a virtuous cycle where better models produce better trajectories, which in turn yield more informative experiential knowledge" (Section 3.3).

Concrete example of the loop in action (Frozen Lake, Qwen3-1.7B, Figure 4):

• Initial model: pass rate around 12–15% (the starting point of the leftmost transparent curve).
• Round 1 extraction: as experiential knowledge accumulates over $n = 15$ trajectories, in-context pass rate rises from the initial ~12% to around 35% at accumulation step 15 (transparent curve, leftmost segment). This is pure in-context learning — the model sees the knowledge but hasn't internalized it.
• Round 1 consolidation: on-policy context distillation for 20 steps internalizes this knowledge. The consolidated model's pass rate (opaque dot at accumulation step 15) jumps above the in-context curve, reaching approximately 39%. The paper explains: "the teacher model augmented with experiential knowledge serves as an effective reward model, providing dense token-level training signal that enables the student model to learn from consolidation training data that the teacher itself never accessed. In other words, the student can generalize beyond the teacher's in-context capabilities by distilling the knowledge directly into its parameters."
• Round 2: The consolidated model (~39% pass rate) is redeployed. It collects new trajectories $T$ and $T'$. Extraction from these higher-quality trajectories starts from a higher baseline (the leftmost point of the second transparent curve is around 39%) and rises further to around 47% at accumulation step 15. Consolidation pushes this to approximately 49%.
• Round 3: Extraction reaches around 53% in-context; consolidation pushes to approximately 54%.

The gains are diminishing but consistent: +27 percentage points from initial to Round 1 consolidation, +10 points from Round 1 to Round 2, +5 points from Round 2 to Round 3. This is expected — the model is approaching the performance ceiling for its size on this task. The key point is that the process never requires new human annotations or environment access.

Why the student can surpass the teacher's in-context performance: This is a subtle but important phenomenon visible in Figure 4: the consolidation dots are consistently above the corresponding in-context extraction curves. The teacher model $\pi_{\text{teacher}}$ with knowledge in context achieves a certain pass rate (the transparent curve). The student $\pi_\theta$ after consolidation achieves a higher pass rate than the teacher's in-context performance. The paper explains this as generalization: the student is trained on many prefixes with randomly sampled knowledge variants, and learns to internalize the knowledge in a way that transfers across prefixes. The teacher, by contrast, can only use the knowledge as context and may not fully leverage it — there's a gap between "having knowledge available" and "effectively using that knowledge," and parameterization through distillation can bridge that gap.

Training Data Construction: The Partial Rollout Prefix Mechanism in Detail

The partial rollout prefix construction deserves its own detailed treatment because it is the enabling mechanism for server-side on-policy training without environment access.

Step-by-step construction from a trajectory:

Given a trajectory $\tau_i = (f_i^1, a_i^1, f_i^2, a_i^2, f_i^3, a_i^3, \ldots)$, where each episode has up to 5 turns (as configured in the experiments), the system extracts:

• Prefix 1 (turn 1): $x_i^1 = (f_i^1)$ — the initial environment feedback (the starting game state). The model hasn't acted yet.
• Prefix 2 (turn 2): $x_i^2 = (f_i^1, a_i^1, f_i^2)$ — the initial state, the model's first action, and the environment's response to that action. This is what the model sees before making its second move.
• Prefix 3 (turn 3): $x_i^3 = (f_i^1, a_i^1, f_i^2, a_i^2, f_i^3)$.
• ...up to prefix 5 (the maximum turn count).

For a single 5-turn trajectory, this yields 5 training prefixes. Across $m$ trajectories (where $m$ is the number needed to fill the training budget), the total number of prefixes in $D$ is approximately $m \times 5$, though some trajectories may end early (success or failure before turn 5).

What each prefix represents during training: When we feed $x_i^j$ to the model during consolidation, the model sees exactly the same text it would see during live deployment at that same point in the interaction. Its response $y \sim \pi_\theta(\cdot \mid x_i^j)$ is a single-turn generation — the model produces just the response for this turn, not a full multi-turn rollout. This single-turn roll-out is what makes the procedure environment-free: we don't need the environment to process the model's action and return feedback, because that feedback is already recorded in the trajectory for the next prefix $x_i^{j+1}$.

Training budget and trajectory requirements: Each consolidation round uses a fixed number of training steps (20 or 100, per Table 4) with 64 prefixes per batch. This means:

• 20 steps × 64 = 1,280 prefixes per round (lighter training).
• 100 steps × 64 = 6,400 prefixes per round (heavier training, used for Sokoban and larger Frozen Lake models).

Since each trajectory yields up to 5 prefixes, the number of trajectories $m$ needed is approximately $\text{steps} \times \text{batch_size} / 5$. For 20 steps: $m \approx 256$ trajectories. For 100 steps: $m \approx 1280$ trajectories. These are collected fresh in each round using the current model.

Why single-turn rollouts and not full multi-turn rollouts: If the system attempted to do multi-turn rollouts during training (generate action, get environment feedback, generate next action, ...), it would need to query the user-side environment, violating the server-side constraint. Single-turn rollouts from pre-recorded prefixes completely avoid this. The prefix already contains the recorded environment feedback for the next turn, so the model can be trained to generate the appropriate action in response to that recorded feedback. This is a form of off-policy evaluation on historical data combined with on-policy generation: the data (prefix + recorded feedback) comes from a previous policy's trajectory, but the response is generated by the current policy, and the loss is computed against the knowledge-conditioned teacher.

**Design choice — using $T'$ separately from $T$:** The extraction stage uses trajectories `$T$`; the consolidation stage uses a separate set `$T'$`. Why not use the same trajectories for both? The paper doesn't explicitly justify this, but the reasoning is likely: extraction benefits from seeing diverse, representative trajectories to build general knowledge, and doesn't require a large number (15–50 trajectories suffice). Consolidation requires many training prefixes to avoid overfitting (1,280–6,400 per round). Using the same small set of trajectories for both would provide insufficient training data for consolidation. Collecting a larger `$T'$` is straightforward — it's just more deployment interactions logged on the user side.

Hyperparameter Configurations and Design Choices

The paper's experimental setup involves several hyperparameters that are fixed across OEL rounds for each model-task pair (Table 4). Here we enumerate them with their rationale.

Extraction hyperparameters:

• **Number of trajectories for accumulation $n$:** 15 (unstructured, 1.7B), 25 (structured, 4B), or 50 (structured, 8B and 4B-Instruct). Larger models can handle more trajectories before the accumulated knowledge exceeds the context window or becomes redundant. The paper notes that "content exceeding `$L_{\max}$` is truncated" — so with `$n = 50$`, the accumulated knowledge is frequently truncated, meaning earlier items may be lost but the model has seen enough diversity to extract high-quality patterns.

• Maximum generation/extraction length $L_{\max}$: 2048 tokens (unstructured) or 8192 tokens (structured). This limits both each extraction step's output and the total accumulated knowledge length. The larger window for structured extraction allows more accumulated knowledge items to be retained without truncation, which matters for $n = 50$ trajectories.

• Number of accumulation seeds $K$: Always 10. This provides diversity in the knowledge set $C$ for consolidation training. Each training prefix randomly samples from among these 10 variants.

• Knowledge format choice: Unstructured for Qwen3-1.7B (the smallest model, which can't handle the structured prompt's complexity well); structured for all larger models. The structured format is preferred when feasible because it produces parseable, non-redundant items that are easier for the teacher to use in-context.

• Context window management for small models: For Qwen3-1.7B, "we do not include previously accumulated experiential knowledge in the extraction context, as we find that smaller models lack sufficient capacity to effectively leverage long contextual information" (Appendix B.4). This means for the 1.7B model, $e_{i-1}$ is omitted from the extraction prompt — each trajectory is processed independently, and the accumulated knowledge is simply the concatenation of independently extracted items. For all other models, the full recursive context is used.

Consolidation hyperparameters:

• Training steps per round: 20 or 100 (Table 4). The choice appears to depend on task difficulty: Sokoban (harder spatial reasoning) uses 100 steps; Frozen Lake with larger models also uses 100 steps; Frozen Lake with 1.7B uses 20 steps. More training steps allow more thorough internalization but cost more trajectory collection and compute.

• Batch size: Always 64 prefixes per step. This is a standard training batch size chosen for GPU memory efficiency and gradient stability.

• Learning rate: Either $1 \times 10^{-6}$ or $5 \times 10^{-6}$ (Table 3 search range). The specific values per configuration (Table 4): 1.7B Frozen Lake uses $5 \times 10^{-6}$; all other configurations use $1 \times 10^{-6}$. The learning rate is fixed across all rounds for each configuration — no annealing or decay.

• Sampling temperature: 0.7 for both trajectory collection and training-time student generation. This provides some diversity in generated responses while keeping the distribution reasonably focused.

• Top-k for KL approximation: $k = 256$ (Appendix A). This is fixed across all experiments. The paper likely chose 256 as a balance between computational cost and approximation accuracy — 256 is large enough to cover the vast majority of the student's probability mass (>99% in practice for most token positions).

• Maximum response length per turn: 1024 tokens. This caps the training sequences but is ample for the game actions (which are typically short: "[up]" or a brief reasoning paragraph).

• Maximum turns per episode: 5. This limits trajectory length for both collection and training. Games that require more than 5 moves to solve are impossible under this constraint — this is a limitation of the experimental setup, not the algorithm.

Checkpoint selection: The paper states: "We fix the number of training steps across all OEL rounds and adopt the final-step checkpoint without any checkpoint selection" (Section 4.1, Consolidation Stage). This is an important design choice: no validation-based early stopping or checkpoint selection. The system simply trains for a fixed number of steps and uses the result. This is practical because it avoids needing a validation environment (which would require server-side environment access, violating the constraint) and simplifies the pipeline. The implicit assumption is that the fixed number of steps is sufficient for convergence without overfitting — which the OOD results in Figure 6 support.

Why the Two-Stage Design Is Necessary

Having walked through the full mechanism, we can now articulate why the extraction-consolidation decomposition is not just convenient but necessary for the problem setting.

Without extraction (raw trajectories → consolidation): If we tried to consolidate directly from raw trajectories, the teacher would need to condition on the full trajectory text to provide useful signal. But raw trajectories are long, noisy, and contain environment-specific details (exact board states, move sequences) that don't generalize. Table 1 confirms this: consolidating from raw trajectories yields only a 7.8% pass rate (barely above the 7.5% baseline without any experience), compared to 21.4% from extracted knowledge. The extraction stage serves as a compression and generalization step: it converts noisy, instance-specific trajectories into concise, transferable rules and strategies that the teacher can effectively use and the student can effectively internalize.

Without consolidation (extraction only → in-context use): If we only extracted knowledge and used it in-context (as in Reflexion, Expel, or the transparent curves in Figure 4), we'd get improvements bounded by the context window. The paper demonstrates this saturation: in Figure 4, the in-context pass rate rises as knowledge accumulates but then plateaus. Moreover, every inference call would need to include the full accumulated knowledge as context, increasing latency and token costs (Figure 5 shows that consolidated models generate shorter responses, indicating internalized efficiency). Consolidation converts context-dependent improvements into parameter-level capabilities that persist without context.

Why the two stages must alternate: The online learning loop requires both stages to alternate because each depends on the other's output. Extraction needs the model to generate trajectories — and the quality of those trajectories determines the quality of extracted knowledge. Consolidation needs extracted knowledge to train from. Iterating means: consolidated model generates better trajectories → extraction from those produces richer knowledge → consolidation with richer knowledge produces an even better model. Breaking the loop (e.g., extracting once and retraining indefinitely from the same knowledge) would saturate, as the knowledge would become stale relative to the model's improving capabilities.

The extraction model as the deployed model (on-policy extraction): The paper's default of $\pi_{\text{extract}} = \pi_\theta$ means the model that analyzes the trajectories is the same model that generated them. This is not arbitrary — Table 2 shows that off-policy extraction from a different model degrades performance: Qwen3-1.7B achieves 31.1% consolidated pass rate when using its own extracted knowledge, but only 22.7% when using knowledge extracted by the larger Qwen3-4B (despite the 4B model being better at the game itself). The interpretation is that experiential knowledge encodes strategies at a level of abstraction matched to the extractor's capabilities; a stronger model may produce knowledge that assumes capabilities the weaker model lacks. On-policy extraction ensures the knowledge is actionable for the model that will use it.

Connecting the Components: A Concrete Training Step Walkthrough

To make the full pipeline concrete, here is what happens in one training step during consolidation (say, step 7 of 20 in Round 2 of OEL on Frozen Lake with Qwen3-1.7B):

1. Sampling the prefix: A prefix $x$ is randomly drawn from $D$. For example, it might be: "You are the player... [initial board state]. Your action: [up] Environment: You moved up. The goal is two steps to the right. Current board: ..." — this is the history up to the start of turn 3.

2. Sampling the knowledge: An experiential knowledge sequence $e$ is randomly drawn from $C$ (one of the 10 accumulated variants from the Round 2 extraction). It might contain items like: "- EXPERIENCE ITEM: Always identify the goal position ('G') before moving and compute Manhattan distance..."

3. Student generation: The current model $\pi_\theta$ (being trained, updated from the previous 6 steps) generates a response $y$ token by token, conditioned only on $x$. It might produce: "Based on the board state, the goal is at position (3,2). I am at (1,2). I need to move right twice. My action: [right]".

4. Teacher forward pass: The frozen teacher $\pi_{\text{teacher}}$ (the initial Round 2 model, before any consolidation) computes its next-token distribution at each position of $y$, conditioned on $(e, x, y_{<t})$. With knowledge in context, the teacher might assign higher probability to the token "right" and lower probability to "up" or "down" at the action position.

5. Loss computation: For each token position $t$, compute $D_{\text{KL}}(\pi_\theta(\cdot \mid x, y_{<t}) \| \pi_{\text{teacher}}(\cdot \mid e, x, y_{<t}))$ using the top-256 approximation. Sum across positions, divide by $|y|$. This yields a single scalar loss for this $(x, e)$ pair.

6. Averaging and gradient step: The losses for all 64 prefixes in the batch are averaged. The gradient of the average loss with respect to $\theta$ is computed and applied via the optimizer (AdamW with the configured learning rate).

After 20 such steps (consuming 1,280 prefixes total), the consolidated model is the checkpoint from step 20 — no validation, no selection. This model is then deployed to collect trajectories for Round 3.

What the gradient is doing: At each token position, the gradient pushes $\pi_\theta$ to increase the probability of tokens the teacher considers likely (given knowledge) and decrease the probability of tokens the teacher considers unlikely. Because the student generated the response $y$ itself (on-policy), this is a form of self-supervised correction: the model is learning to adjust its own generation distribution to better match what it would produce if it had access to the accumulated experiential knowledge. Over many prefixes and many training steps, this causes the model's unconditional (context-free) behavior to converge toward its knowledge-conditioned behavior.

4. Key Insights and Innovations

Innovation 1: Decomposing Online Learning from Text into Extraction and Consolidation as Separable, Stackable Stages

The paper's deepest conceptual move is not the specific extraction prompt or the distillation loss — it's the recognition that learning from deployment experience can be cleanly factored into two independent, optimizable stages: a text-to-text extraction stage (trajectories → experiential knowledge) and a consolidation stage (knowledge → parameters). This decomposition is what makes the entire framework practical, because each stage solves a different barrier.

Before OEL, the field treated "learning from experience" as a monolithic problem. Reflexion (Shinn et al., 2023) folded extraction and application into a single in-context loop: reflect on failures, then use those reflections in the next attempt. Expel (Zhao et al., 2024) extracted insights but stored them externally for retrieval — again, everything happened at inference time. Neither approach separated the generation of reusable knowledge from its internalization into model weights. The result was that all learning remained in-context: bounded by the context window, non-compounding across deployment rounds, and vulnerable to saturation (the transparent curves in Figure 4).

What OEL recognizes is that these two functions have fundamentally different requirements. Extraction benefits from seeing complete multi-turn trajectories and reasoning globally across episodes — it's a synthesis task that produces concise, structured output. Consolidation benefits from many diverse training examples and dense token-level feedback — it's a compression task that requires statistical repetition. By separating them, each can be optimized independently: the extraction prompt can be engineered for knowledge quality without worrying about training dynamics, and the consolidation procedure can be tuned for parameter efficiency without worrying about knowledge coverage. The decomposition also creates a natural modularity: if someone develops a better extraction method (e.g., using a separate critic model, or incorporating environment-specific heuristics), it can be dropped into the OEL framework without changing the consolidation stage. Conversely, improvements to the distillation procedure (different KL objectives, different teacher configurations) work with any extraction method.

This is a fundamental shift in how to think about online learning for language models — not an incremental refinement. Prior work conflated two problems that have different computational signatures and different scaling properties. OEL demonstrates that separating them enables a virtuous cycle (Section 3.3) where each stage amplifies the other: better extraction yields better consolidation targets; better consolidation yields a model that collects higher-quality trajectories for the next extraction. The evidence for this compounding is in Figure 4: each round's extraction starts from a higher baseline than the previous round's, and consolidation pushes further each time. This would not be possible if extraction and consolidation were coupled — the extraction would always operate on trajectories from the same-quality model, and consolidation would plateau.

A subtle but important aspect of the decomposition is that it makes the framework reward-free by construction. Neither stage requires scalar rewards. Extraction operates purely on text — the model reads trajectories and writes knowledge. Consolidation's teacher provides token-level probability targets derived from the knowledge text, not from any external reward function. This means the framework can be applied to any environment that produces text feedback, without ever building a reward model. Prior approaches to learning from environment interaction — whether RL-based (needing scalar rewards or learned reward models) or reflection-based (needing the model to generate useful reflections, which may not work in domains where the model struggles to diagnose its own failures) — had implicit requirements on the nature of the feedback. OEL's decomposition eliminates those requirements: if the environment produces text, and the model can process that text into general insights, learning is possible.

Innovation 2: The Concept of "Experiential Knowledge" as a Sharable, Compressible Intermediate Representation Separate from Raw Experience

The paper introduces a new abstraction — experiential knowledge — that sits between raw trajectories and model parameters. This is not just a relabeling of "reflections" or "insights." It is a specific claim about what kind of representation enables effective online learning: transferable, general, and compressible rules extracted from instance-specific interaction logs.

Prior work on learning from experience typically used one of two representations: raw trajectories (replaying past interactions, as in experience replay for RL) or in-situ reflections (generating self-critique immediately after a failure, as in Reflexion). Table 1 demonstrates that neither works well for language model improvement. Raw trajectories in context improve pass rate only modestly (10.9% vs. 7.5% baseline on Sokoban), and consolidation from raw trajectories is essentially ineffective (7.8%). The paper's interpretation is that "unprocessed trajectories introduce noise that obscures useful information" — they contain environment-specific details (exact board states, move sequences for particular maps) that don't generalize and consume context budget that could be used for signal.

What makes experiential knowledge distinctive as an abstraction:

First, it is explicitly transferable. The extraction prompt instructs the model to produce "possible rules, instructions or winning strategies for the game" that are "generally useful rather than only applicable for the current map" (Figure 10). The resulting knowledge items (Figure 13) are abstract principles — "axis-aligned convergence," "progressive alignment through convergence" — not move-by-move plans. This transferability is what enables consolidation to work: if the knowledge were specific to particular trajectories, training on it would lead to overfitting. The knowledge needs to encode patterns that hold across episodes, which is why the extraction stage processes multiple trajectories accumulatively, synthesizing cross-episode regularities.

Second, it is accumulative. The recursive extraction process (Equation 1) builds knowledge progressively, with each new trajectory conditioned on previously accumulated knowledge. This creates a representation that grows richer with more experience without becoming redundant — the prompt explicitly instructs "Do not repeat the previous experience." This accumulative property is what enables the online learning loop: each round's extraction starts from the previous round's consolidated knowledge (implicitly, through the improved model that generates better trajectories), and adds new layers of insight that the model couldn't have discovered before because it lacked the capability to explore those regions of the task space.

Third, it is compressible into parameters. This is the property that distinguishes experiential knowledge from memory-based representations (Expel) or in-context reflections (Reflexion). Experiential knowledge is designed to be internalized — it's concise enough to fit in a teacher's context window (2K–8K tokens) but informative enough to guide token-level behavior across many situations. The consolidation stage demonstrates that this compression is effective and even surpasses in-context use: Figure 4 shows consolidation dots consistently above the in-context extraction curves, meaning the parameterized knowledge outperforms the same knowledge used as context.

This is a fundamental conceptual contribution rather than an incremental one. The paper is essentially arguing that there exists a level of abstraction — between raw sensorimotor experience (trajectories) and compiled skill (parameters) — that is both learnable from experience and usable for training. Identifying this level and operationalizing it through the extraction mechanism is what makes online learning for language models tractable. Without this intermediate representation, you're stuck with either noisy raw data (ineffective for training) or the model's own parametric knowledge (which doesn't improve from experience). The representation exists in the "sweet spot" where language models are good at both producing it (text synthesis from examples) and consuming it (in-context learning for distillation).

Innovation 3: On-Policy Consistency as a Critical Constraint for Knowledge Transfer Between Models

The paper's third major contribution is an empirical finding with significant implications: experiential knowledge does not transfer well between models of different capabilities; there is an on-policy consistency requirement that constrains whose experience can be used to train whom. Table 2 is the key evidence: Qwen3-1.7B trained on its own extracted knowledge achieves 31.1% pass rate after consolidation; the same model trained on knowledge extracted by Qwen3-4B (a strictly more capable model at the task, achieving 18.0% in-context pass rate vs. 1.7B's 23.8%) achieves only 22.7%.

This is a counterintuitive finding that challenges a natural assumption. One might expect that knowledge from a stronger model — one that solves the task more effectively — would be more valuable for training, since it encodes better strategies. The data shows the opposite. The paper's interpretation: larger-model knowledge "may encode strategies beyond the smaller model's capabilities." In other words, the knowledge assumes a level of reasoning, planning, or precision that the smaller model cannot execute. When the teacher conditions on this knowledge and provides token-level targets, it pushes the student toward a behavior distribution the student fundamentally cannot match, leading to degraded performance.

This finding has diagnostic significance beyond the specific experiments. It suggests that:

• Online learning is inherently model-specific. The knowledge a model extracts from its own experience is matched to its own capability frontier — the strategies it can discover and execute. Using another model's experience introduces a capability mismatch that degrades learning. This means deployment experience is not fungible: a fleet of models at different scales cannot pool their experience for mutual improvement unless the capability gap is bridged.

• Self-improvement loops are more robust than cross-model distillation loops for experiential learning. The OEL framework's default of π_extract = π_θ is not incidental but essential — it ensures the extraction stays on-policy relative to the model that will consume the knowledge. If the extraction model were a separate, potentially stronger model, the knowledge might encode strategies beyond the deploying model's reach, creating a training signal that pushes the model toward behaviors it cannot reliably produce.

• The teacher model in distillation should be capability-matched to the student. In OEL, the teacher during consolidation is "the frozen initial π_θ before training" — the same model, just with knowledge in context. This ensures the teacher's knowledge-conditioned behavior is achievable by the student. If the teacher were a fundamentally stronger model, the reverse KL objective would push the student toward an unachievable target, causing degradation rather than improvement.

This insight connects to broader discussions in the distillation literature about the importance of capacity gap between teacher and student. Standard distillation works best when the teacher is strong but the gap is not too large; if the teacher is dramatically more capable, the student's limited capacity forces it into a mode-covering compromise that reduces quality. OEL's finding extends this principle to the knowledge representation level: the gap is not just in model capacity but in the actionability of the knowledge itself.

This is an incremental finding in the sense that it emerges from a specific experimental comparison rather than a new theoretical framework, but it has fundamental implications for how online learning systems should be architected. It provides a principled reason for keeping extraction on-policy and for matching the teacher to the student during consolidation — design choices that might otherwise seem arbitrary.

Innovation 4: Demonstrating That Parameterized Experiential Knowledge Outperforms In-Context Experiential Knowledge (The Consolidation Surplus)

A recurring but insufficiently emphasized pattern in the paper's results is that consolidation consistently produces a model that outperforms the teacher's in-context performance. In Figure 4, every consolidation dot (opaque) sits above the corresponding point on the extraction curve (transparent). For Frozen Lake Round 1: in-context extraction reaches ~35% at step 15; consolidation reaches ~39%. Round 2: extraction reaches ~47%; consolidation reaches ~49%. Round 3: extraction reaches ~53%; consolidation reaches ~54%. This pattern holds across all rounds and both environments.

The paper offers an explanation: "the student can generalize beyond the teacher's in-context capabilities by distilling the knowledge directly into its parameters." But this deserves closer examination as a distinct finding, because it reveals something non-obvious about the relationship between in-context knowledge use and parameterized knowledge.

Why should consolidation outperform in-context use? The teacher model during extraction has access to the same knowledge as the teacher during consolidation. In both cases, the knowledge is prepended to the context. The difference is what happens next: during extraction, the model uses that knowledge to solve new game instances directly (in-context). During consolidation, the model uses that knowledge to provide token-level targets for training, and the student learns to produce similar behavior without the knowledge context.

The consolidation surplus suggests that having knowledge in context is not the same as effectively using it. Several mechanisms could explain this:

• Attention dilution: In long contexts (the extraction teacher sees up to 8192 tokens of accumulated knowledge plus the game prompt), the model's attention may not focus effectively on the most relevant knowledge items for each decision. Parameterization through distillation forces the model to internalize the knowledge, making it available at every token position without attention competition.

• Generalization across knowledge variants: Consolidation trains on multiple knowledge seeds (K = 10), randomly sampling different accumulated knowledge sequences for each training prefix. This exposes the student to diverse formulations of the same underlying strategies, potentially helping it extract the invariant core. The extraction teacher, by contrast, sees only one accumulated knowledge sequence at a time.

• Compression as regularization: Distilling knowledge into parameters forces lossy compression that may actually improve generalization by discarding knowledge-instance-specific noise and retaining only the patterns that recur across many training examples. This is analogous to how model compression can sometimes improve generalization by acting as an implicit regularizer.

Prior work did not establish this gap. Standard context distillation (Askell et al., 2021; Snell et al., 2022) trains students to match teacher outputs but typically evaluates whether the student can recover the teacher's performance, not whether it can exceed it. The finding that parameterized knowledge outreasons in-context knowledge is significant because it inverts the usual assumption that in-context learning is an upper bound — that giving the model direct access to knowledge should be at least as good as baking it into weights. OEL suggests the opposite: parameters are a better home for reusable knowledge than context windows, at least for the kinds of strategic, rule-based knowledge that extraction produces.

This is a diagnostic finding with implications for system design. It suggests that in a deployed online learning system, the goal of consolidation should not be merely to recover in-context performance without the context cost, but to exceed it. The consolidation stage is not just cost-saving (removing the need for knowledge context at inference) — it's genuinely capability-increasing. This reframes the value proposition of the extraction-consolidation loop: it's not just about maintaining what was learned in context, but about amplifying it.

The finding also raises a deeper question the paper doesn't fully explore: what is the nature of the capability that consolidation adds? Is the student learning to apply knowledge more effectively, or is it learning something the knowledge didn't explicitly encode (e.g., when and how to apply different rules, or how to resolve conflicts between rules)? The fact that the consolidation surplus persists across rounds (Figure 4) suggests it's not just a one-time effect of cleaning up noisy knowledge, but rather a genuine enhancement of the model's ability to leverage the extracted insights.

5. Experimental Analysis

Evaluation Methodology

• Dataset. Two text-based game environments, both implemented within TextArena (Guertler et al., 2025): Frozen Lake (3×3 grid with two holes; agent navigates to a goal while avoiding holes) and Sokoban (6×6 grid with one box; spatial reasoning puzzle requiring the model to push a box onto a target without falling into holes or getting stuck). Neither game provides explicit rules—the original rules from TextArena are replaced with a general task description (Figures 8–9), forcing the model to discover mechanics through exploration. This setup simulates real-world deployment where minimal prior knowledge about new environments is available. Evaluation uses a held-out test split of size-128 game maps, with results averaged over 10 random seeds. For out-of-distribution evaluation, the paper reports prompt-level strict accuracy on IF-Eval (Zhou et al., 2023).

• Base model(s). Qwen3 family (Yang et al., 2025) at three scales: Qwen3-1.7B, Qwen3-4B, and Qwen3-8B (all thinking models), plus one non-thinking variant Qwen3-4B-Instruct-2507. The paper chooses these models to demonstrate OEL across a range of capacities, from small models with limited in-context learning ability (1.7B cannot effectively leverage long contextual information during extraction—Appendix B.4 explicitly notes that accumulated knowledge from previous steps is omitted from the extraction context for this model) to larger models that can handle structured knowledge extraction with 50 trajectories and 8192-token context windows. The inclusion of both thinking and non-thinking variants tests whether OEL's extraction mechanism (which retains only answer parts and removes reasoning traces for thinking models) generalizes across model types.

• Metrics. Pass rate on the held-out 128-map test split, averaged over 10 random seeds—this measures the fraction of episodes where the model successfully completes the game objective (reaching the goal in Frozen Lake, pushing the box onto the target in Sokoban) within the 5-turn episode limit. Normalized response length (Figure 5) measures token efficiency: the average per-turn response length is normalized relative to the initial model's length, showing how reasoning becomes more concise as experiential knowledge is internalized. IF-Eval strict accuracy (Figure 6, right) measures out-of-distribution instruction-following capability to track catastrophic forgetting.

• Baselines. The paper compares against several alternative configurations within its own framework rather than against external methods:

  • In-context experiential knowledge (extraction only): The accumulated experiential knowledge is prepended to the model's prompt at inference time, representing what prior reflection-based methods (Reflexion, Shinn et al., 2023; Expel, Zhao et al., 2024) achieve—knowledge used in context but never internalized into parameters.
  • In-context raw trajectories: The raw interaction trajectories (not extracted knowledge) are prepended to the context, testing whether extraction is necessary (Table 1).
  • Off-policy context distillation: The teacher model with experiential knowledge generates responses, and the student is trained via forward KL divergence to imitate them (standard approach from Askell et al., 2021; Snell et al., 2022; Cao et al., 2025). Compared against OEL's on-policy reverse KL consolidation in Figure 6.
  • Off-policy experiential knowledge: Knowledge extracted by a different (larger) model than the one being trained (Table 2), testing the on-policy consistency requirement.
  • w/o Experience: The model without any experiential knowledge, establishing the floor (Table 1).

• Generation budget / compute accounting. The paper measures compute indirectly through the number of training samples and trajectories collected, not through FLOPs or wall-clock time. Each OEL round consists of two budgeted phases. Extraction: n = 15, 25, or 50 trajectories are processed accumulatively, with each extraction step consuming up to L_max = 2048 or 8192 tokens of generation. This accumulation is repeated K = 10 times with different random seeds. Consolidation: 20 or 100 training steps per round with 64 game samples per step, requiring 1,280 or 6,400 trajectory samples per training round (each trajectory spans up to 5 turns with max 1024 response tokens per turn). The total "experience seen" includes both the extraction trajectories (small, tens of episodes) and the consolidation trajectories (large, thousands of partial prefixes). The x-axis in Figure 1 and Figure 4 shows "Accumulation Step" during extraction, and consolidation points are plotted at the corresponding accumulation step where knowledge was internalized—this represents logical progress through the online learning loop, not total FLOPs. No attempt is made to equate computational cost across methods (e.g., in-context vs. consolidated use of knowledge has vastly different inference costs per episode, but this is not accounted for in the comparisons).

• Cross-validation / statistical protocol. The paper uses fixed random seeds for reproducibility (10 seeds for evaluation, 10 seeds for knowledge accumulation) but does not employ cross-validation for model selection. There is no validation-based hyperparameter tuning, early stopping, or checkpoint selection: "We fix the number of training steps across all OEL rounds and adopt the final-step checkpoint without any checkpoint selection" (Section 4.1, Consolidation Stage). This is a deliberate simplification justified by the constraint of no server-side environment access (which would be needed for validation-based model selection). Hyperparameters are searched once per model-task pair (Table 3) and then fixed across all OEL rounds (Table 4). The 10-seed averaging for evaluation provides some statistical robustness, but no confidence intervals or standard deviations are reported, making it difficult to assess whether differences between methods are statistically significant.

Main Quantitative Results

OEL Enables Continuous Online Improvement Across Successive Rounds

Figure 4 presents the central result: OEL achieves consistent pass rate improvements over three successive rounds on both Frozen Lake (Qwen3-1.7B) and Sokoban (Qwen3-4B-Instruct-2507).

On Frozen Lake (Figure 4, left):

• The initial model pass rate sits approximately at the leftmost point of the Round 1 extraction curve (transparent), around 12–15%.
• During Round 1 extraction, as experiential knowledge accumulates over n = 15 trajectories, in-context pass rate rises to approximately 35% at accumulation step 15. This is pure in-context learning—the knowledge is prepended but not parameterized.
• Round 1 consolidation at step 15 (opaque dot) achieves approximately 39%, surpassing the in-context ceiling by roughly 4 percentage points.
• Round 2 extraction starts from the consolidated model (~39% baseline), and in-context use of newly extracted knowledge reaches approximately 47% at step 15—a ~8-point in-context gain over the Round 2 baseline. Consolidation pushes this to approximately 49%.
• Round 3 extraction reaches approximately 53% in-context; consolidation reaches approximately 54%.
• The cumulative gain from initial model to Round 3 consolidation is roughly +39–42 percentage points of pass rate, more than a 3× improvement over the starting performance.

On Sokoban (Figure 4, right), the pattern is similar but with lower absolute numbers due to the harder task:

• The extraction curves (transparent) show in-context pass rate improving across accumulation steps within each round.
• Consolidation dots consistently sit above the extraction curves.
• Successive rounds shift the entire extraction curve upward, demonstrating that the improved model collects higher-quality trajectories that yield richer knowledge.

Key dynamic visible in both plots: The transparent extraction curves within each round show diminishing returns as accumulation steps increase—performance rises quickly at first and then plateaus, consistent with the paper's claim that "the context window becomes increasingly occupied, limiting the model's capacity to absorb and leverage additional knowledge through in-context learning alone" (Section 4.2). Consolidation resets this saturation by parameterizing the knowledge, and the next round's extraction starts from a higher baseline, enabling new knowledge to be extracted that couldn't be discovered before.

OEL Improves Token Efficiency

Figure 5 shows the normalized per-turn response length of Qwen3-1.7B on Frozen Lake across OEL rounds. The y-axis represents response length normalized to the initial model's length (100% baseline).

• During Round 1 extraction, response length decreases from 100% to roughly 85% by accumulation step 15—the model becomes more concise as it internalizes rules through in-context knowledge.
• After Round 1 consolidation, the consolidated model (opaque dot at step 15) produces responses at roughly 85% length.
• Round 2 extraction starts near 85% and decreases further to approximately 77% at step 15. Consolidation locks this in.
• Round 3 extraction reaches roughly 70% of the initial length by step 15.

The simultaneous pass rate improvement (Figure 4) and response length reduction (Figure 5) means the model is solving problems both more accurately and with less reasoning effort. This is an underappreciated benefit of the OEL framework: it doesn't just make the model better at the task—it makes it more efficient, which matters for deployment latency and cost. The paper frames this as evidence that "successive iterations of OEL progressively internalize experiential knowledge" (Section 4.3).

OEL Mitigates Catastrophic Forgetting Compared to Off-Policy Alternatives

Figure 6 compares on-policy context distillation (OEL's consolidation) against off-policy context distillation over 2,500 training samples (concatenating Round 1 and Round 2 consolidation stages of 20 steps each, with saturated portions omitted).

In-distribution pass rate (left subplot):

• Both methods start from the initial model performance (the y-axis intercept).
• On-policy (OEL) rises more steeply and reaches approximately 45–48% pass rate after 2,500 samples.
• Off-policy reaches approximately 35–38% at the same point—a gap of roughly 10 percentage points.
• The off-policy curve appears to saturate earlier and at a lower plateau than on-policy.

Out-of-distribution IF-Eval accuracy (right subplot):

• The initial model achieves approximately 66% accuracy (the horizontal dashed line).
• On-policy training largely preserves this: the blue curve stays within approximately 0.5 percentage points of the initial model throughout training, ending near 66%.
• Off-policy training shows a clear degradation: accuracy drops from ~66% to approximately 64.5–65% over the training period—a loss of 1–1.5 percentage points.

This result is critical for the viability of online learning in deployment: if each round of improvement on the target task degraded general capabilities, the framework would be self-defeating. The paper attributes the OOD preservation to the on-policy nature of training: "consistent with prior work showing that on-policy training mitigates catastrophic forgetting" (Section 4.4, citing Shenfeld et al., 2025; Chen et al., 2025; Ye et al., 2026).

OEL Scales with Model Size

Figure 7 reports Frozen Lake pass rates for Qwen3-1.7B, 4B, and 8B across initial model, Round 1, and Round 2.

• Initial model pass rates are relatively flat across scales: all three models start around 5–7% (reading from the bar heights). The task is hard enough that raw capability differences are compressed at the floor.
• Round 1 consolidation yields substantial gains at all scales: 1.7B reaches roughly 30–35% (consistent with Figure 4's ~39% at the specific checkpoint), 4B reaches roughly 40–45%, 8B reaches roughly 50–55%.
• Round 2 adds consistent further gains across all scales: the incremental improvement from Round 1 to Round 2 appears similar in absolute terms (~5–10 percentage points) for all three model sizes.
• Larger models benefit more from OEL: the gap between 1.7B and 8B widens from near-zero initially to roughly 20–25 percentage points by Round 2.

The paper interprets this as a "virtuous cycle where greater capacity and better experience compound to amplify performance gains" (Section 4.5). Larger models generate higher-quality trajectories during deployment, from which more effective experiential knowledge can be extracted, which when consolidated yields a stronger model that generates even better trajectories. The consistent Round 1 → Round 2 gain across scales also demonstrates that "experiential knowledge continues to accumulate meaningfully beyond the first round regardless of model capacity"—the online learning loop doesn't saturate after a single iteration, even for the largest model tested.

Ablation Studies and Robustness Checks

• Extracted experiential knowledge vs. raw trajectories (Table 1): Using Qwen3-4B-Instruct-2507 on Sokoban in Round 1, raw trajectories used in-context yield only 10.9% pass rate (vs. 7.5% without any experience)—a modest gain. Consolidation from raw trajectories is essentially ineffective at 7.8% (no better than the no-experience baseline of 7.5%). In contrast, extracted experiential knowledge achieves 18.2% in-context and 21.4% after consolidation—more than double the in-context gain and nearly 3× the consolidation gain compared to raw trajectories. This confirms that the extraction stage is essential: unprocessed trajectories contain too much instance-specific noise to serve as effective training signal, while extracted knowledge distills cross-episode patterns into a form that both in-context learning and consolidation can leverage. An important nuance: the "Consolidate" column for raw trajectories (7.8%) being essentially equal to the no-experience baseline (7.5%) suggests that raw trajectory consolidation may be actively harmful or neutral—any useful signal is swamped by noise.

• On-policy vs. off-policy experiential knowledge source (Table 2): Qwen3-1.7B on Frozen Lake is trained with experiential knowledge extracted either from its own trajectories (on-policy) or from the larger Qwen3-4B model's trajectories (off-policy source). Self-extracted knowledge achieves 23.8% in-context and 31.1% after consolidation. Knowledge from the larger model achieves only 18.0% in-context and 22.7% after consolidation—despite the 4B model being objectively stronger at the task. The paper's interpretation (Section 4.6.2): "experiential knowledge from a stronger model does not necessarily transfer well, as it may encode strategies beyond the smaller model's capabilities." This is a non-obvious finding—one might expect stronger-model knowledge to be a better teacher—and it establishes on-policy consistency between knowledge source and policy model as a critical requirement. This ablation justifies the default design choice of π_extract = π_θ.

• Structured vs. unstructured knowledge format (Table 4 and Appendix B): The paper does not present a direct ablation comparing these formats head-to-head on the same model-task pair. Instead, format is chosen per configuration: unstructured for Qwen3-1.7B (the smallest model, which "lack[s] sufficient capacity to effectively leverage long contextual information"), structured for all larger models. The structured format enforces a specific output template ("- EXPERIENCE ITEM:") and retains only conforming lines, providing parseability and non-redundancy. The unstructured format allows free-form generation, which may be necessary when the model cannot reliably follow formatting instructions. The absence of a direct comparison leaves open the question of whether the structured format's constraints actually improve knowledge quality or merely provide engineering convenience.

• Training steps per round (Table 4): Frozen Lake with Qwen3-1.7B uses 20 training steps per consolidation round; Frozen Lake with 4B and 8B, and Sokoban with 4B-Instruct, use 100 steps. The paper notes that in-distribution performance "tends to saturate within each stage after 20 steps" and omits saturated portions when plotting Figure 6. This implies that 20 steps may be sufficient for convergence on the easier task with the smallest model, but 100 steps provide benefit for harder tasks or larger models. No systematic study of the optimal number of steps is presented, and the fixed-step checkpoint without validation means the system may be under-training (leaving performance on the table) or over-training (risking OOD degradation, though Figure 6 suggests this risk is low with on-policy distillation).

• Context window and accumulation length limits: The paper explicitly truncates accumulated experiential knowledge when it exceeds L_max (2048 for unstructured, 8192 for structured). With n = 50 trajectories and structured extraction, accumulated knowledge frequently exceeds this limit, meaning earlier knowledge items are lost. The paper does not study how truncation affects knowledge quality or whether more sophisticated knowledge selection (beyond simple concatenation and truncation) would improve results. This is a practical limitation: the extraction process generates knowledge linearly with the number of trajectories, but the usable context window is fixed, creating a ceiling on how much experience can be leveraged at once.

• ReST^EM negative result (not present in this paper): Unlike the reference paper on test-time compute scaling, OEL does not present a specific negative result from trying an alternative training objective or data generation pipeline. The paper does not experiment with alternative extraction methods (e.g., using a separate critic model, filtering knowledge by usefulness), alternative consolidation objectives (e.g., forward KL, distillation with auxiliary losses), or alternative teacher configurations (e.g., using the best extracted knowledge rather than random sampling). The closest to a negative result is the off-policy distillation comparison (Figure 6), which confirms that off-policy training is inferior—but this is an expected result consistent with prior work, not a surprising empirical finding.

• No ablation on the teacher model selection: The teacher is always "the frozen initial π_θ before training" (Section 3.2). The paper does not test alternatives such as using the previous round's consolidated model as teacher, using a separate stronger model as teacher (with its own extracted knowledge), or updating the teacher during consolidation. This leaves open whether the choice of teacher matters and whether the initial model is optimal.

• No ablation on the reverse KL vs. forward KL for the consolidation objective beyond Figure 6: Figure 6 compares on-policy (reverse KL) vs. off-policy (forward KL) context distillation, but this confounds two differences simultaneously: (1) on-policy vs. off-policy data generation, and (2) reverse KL vs. forward KL divergence. The paper cannot attribute the performance difference solely to the KL direction—the on-policy/off-policy distinction may be the dominant factor. A cleaner ablation would fix the data source (both on-policy) and compare reverse KL vs. forward KL objectives.

Critical Assessment

The experiments in this paper demonstrate a specific and genuinely interesting capability: a language model can improve at a text-based game over multiple rounds by extracting reusable knowledge from its own deployment trajectories and distilling that knowledge into its parameters, without any reward model or environment access during training. The experimental evidence for this core claim is solid within the tested domain. Figure 4 shows consistent, compounding improvements across three rounds on two different games and two different model types (thinking and non-thinking). The gains are substantial—roughly 3–4× improvement from initial to Round 3 performance on Frozen Lake—and the pattern of extraction saturation followed by consolidation surplus is consistent.

However, the experiments also reveal important boundaries and limitations that constrain how broadly the claims should be interpreted.

What the experiments actually demonstrate: The system works on two specific text-based grid-world games (Frozen Lake and Sokoban) implemented in TextArena, with Qwen3 models at 1.7B–8B scales. These are environments with clear, deterministic success/failure conditions (reach the goal, push the box to target) and relatively small state spaces (3×3 and 6×6 grids). The text feedback is simple and structured—each turn returns predictable textual descriptions like "You moved up. Current board: ..." The model interactions span at most 5 turns. The extracted knowledge (Figure 13) captures strategic principles ("axis-aligned convergence," "progressive alignment through convergence") that are genuinely abstract and transferable across game instances.

What has not been demonstrated: The paper does not test on:

• Environments with stochastic transitions or partial observability, where the relationship between action and outcome is probabilistic and the same action may have different results on different attempts.
• Environments where success is ambiguous, multi-dimensional, or subjective (dialogue quality, creative writing, open-ended assistance)—the extraction stage relies on the model being able to infer correct strategies from trajectories, which requires some notion of what constitutes a good outcome.
• Long-horizon tasks where 5 turns is insufficient (many real-world tasks require dozens or hundreds of interaction steps).
• Continuous online learning: in the experiments, the model collects trajectories in discrete batches (all of Round 1's T and T' at once) rather than continuously streaming experience. The paper's "online learning" is batched and episodic, not truly continuous.

The generation budget accounting is incomplete. The consolidation stage uses 1,280–6,400 trajectory samples per round, each spanning up to 5 turns. The extraction stage uses 15–50 trajectories. These numbers are not trivial—collecting 6,400 5-turn trajectories requires 32,000 model-environment interactions per round. For comparison, the evaluation metric (pass rate on 128 maps) covers only 128 episodes. The ratio of training-to-evaluation data is enormous (~50–250:1), and the paper never discusses the sample efficiency of OEL relative to alternatives. Could a simpler approach—say, few-shot prompting with carefully constructed examples—achieve similar performance with far fewer interactions? The paper doesn't provide this comparison.

The baseline comparisons are internal to OEL, not against external methods. The paper's baselines are all variants of its own framework (raw trajectories vs. extracted knowledge, on-policy vs. off-policy distillation, self-extracted vs. other-model-extracted knowledge). There is no comparison against:

• Standard few-shot prompting: Give the model 5–10 examples of successful game completions in context. How does this compare to OEL's extracted knowledge?
• Reinforcement learning with a learned reward model: If one were willing to build a reward model from the same trajectory data (e.g., train a classifier on success/failure from the 6,400 consolidation trajectories), how would standard RL (PPO, GRPO, etc.) compare to OEL's reward-free distillation?
• Simple imitation learning: Use the successful trajectories from the extraction set as supervised fine-tuning data (behavioral cloning). How does this compare to the more complex extraction-consolidation loop?

The absence of these comparisons makes it difficult to assess whether OEL's specific architecture (extraction → knowledge accumulation → on-policy reverse KL consolidation) is genuinely more effective than simpler alternatives, or whether the gains come primarily from having access to large amounts of deployment data that any reasonable learning algorithm could exploit.

The pass rate metric has an implicit ceiling. On Frozen Lake, a 3×3 grid with two holes, optimal play should achieve near-100% pass rate (the game is fully solvable). The best model in Figure 4 (Round 3 consolidated) reaches only ~54%. This means that after three full rounds of online learning involving thousands of trajectory samples and extensive computation, the model is still failing nearly half the time on a very simple grid-world task. This is both a strength of the paper (honest reporting) and a limitation: OEL improves the model substantially from its starting point, but doesn't come close to optimal performance. The framework amplifies existing capability but may have a low ceiling when the base model is fundamentally limited—consistent with the finding in the reference paper on test-time compute scaling that hard problems (difficulty bin 5) see near-zero improvement regardless of budget.

The catastrophic forgetting result (Figure 6) is important but limited. The paper shows that on-policy distillation preserves IF-Eval accuracy over 2,500 training samples, while off-policy distillation degrades it by 1–1.5 points. This is encouraging but leaves several questions:

• Does OOD preservation hold across multiple rounds (3+) or does forgetting accumulate?
• Is IF-Eval a sufficiently sensitive measure of general capability? A model could lose knowledge in domains not covered by IF-Eval (coding, factual knowledge, multilingual ability) that wouldn't show up in this metric.
• The training regime in Figure 6 concatenates only Round 1 and Round 2 (2,500 samples total). In a production deployment running 10+ rounds with larger models, the total training samples would be much higher. Does the OOD degradation scale sublinearly or linearly with training steps?

The difficulty of the environments may be artificially constrained by the 5-turn limit. Both games (especially Sokoban, which is a full puzzle game) could require more than 5 moves for optimal solutions. The 5-turn cap truncates trajectories, meaning the model never sees full solutions for harder instances. The experiential knowledge extracted from truncated trajectories may be incomplete or misleading—it captures strategies for partial progress but not for complete solutions. This is acknowledged only implicitly in the experimental setup description.

The paper's central innovation claim—that OEL is "reward-free"—is accurate but potentially narrower than presented. It's true that OEL requires no scalar reward function, no reward model, and no verifiable reward. But the extraction stage does require that the model can infer what constitutes good behavior from textual feedback. In the tested environments, the feedback clearly indicates success ("You reached the goal!") or failure ("You fell into a hole! Game over."). The model's extraction of "winning strategies" depends on being able to distinguish successful from unsuccessful trajectories based on this textual signal. In environments where textual feedback is ambiguous, contradictory, or uninformative about outcome quality, extraction would fail. The framework is reward-free in the technical sense but outcome-aware in practice.

The on-policy consistency finding (Table 2) is genuinely important but needs more investigation. The result that Qwen3-1.7B learns better from its own extracted knowledge than from Qwen3-4B's knowledge is striking and has implications for multi-model deployment fleets. But the experiment only tests one pair of models (1.7B and 4B) on one task. It doesn't establish how the capability gap affects transfer—would knowledge from a 0.5B model also underperform for a 1.7B model? Is there a threshold where off-policy knowledge becomes useful if the gap is small enough? Does the finding hold for models from different families (e.g., knowledge from Llama-3-8B used to train Qwen3-4B)?

Missing experiments that would have strengthened the paper:

1. Learning curves for the consolidation stage: Figure 4 shows before-and-after consolidation points but doesn't show how pass rate evolves during the 20 or 100 training steps. The Figure 6 in-distribution plot combines two rounds and omits saturated portions—a per-round breakdown would reveal whether early stopping could be effective and whether different rounds converge at different rates.

2. Comparison of different extraction model configurations: The paper sets π_extract = π_θ by default, but what if a separate, fine-tuned extractor model were used? Or what if extraction were performed by the same model but with a different prompt (e.g., asking for "mistakes to avoid" rather than "winning strategies")? The extraction prompt (Figures 10–11) is complex and task-specific—how sensitive are results to the exact prompt wording?

3. Ablation on the number of accumulation seeds K: The paper uses K = 10 throughout, randomly sampling from among these variants during consolidation. Does this diversity matter? Would a single accumulated knowledge sequence work nearly as well? Would 100 seeds provide additional benefit? This parameter directly affects the diversity of the consolidation training signal.

4. Scaling the number of OEL rounds beyond 3: Figure 4 shows diminishing returns (Round 1: +27 points, Round 2: +10 points, Round 3: +5 points). Would Round 4 and 5 continue to provide gains, or does performance saturate? The answer determines whether OEL is a temporary bootstrap or a sustainable improvement mechanism.

5. Environment complexity scaling: The environments are simple grid-worlds. Would OEL work on more complex text-based environments (larger grids, more objects, more complex objectives) or on non-game tasks (tool use, web navigation, code debugging)? The knowledge extraction mechanism assumes the model can formulate general rules from a small number of trajectories—this may break down as task complexity increases and the space of possible strategies grows.

Conditional assessment of the headline claims:

• "OEL enables language models to continuously improve from their own deployment experience": Supported for the tested environments and model scales, with the caveat that "continuous" means three discrete rounds, not truly continuous streaming learning. The improvement is genuine and compounding, but the experiments don't demonstrate indefinite improvement—the diminishing returns in Figure 4 suggest a ceiling may be approaching.

• "The entire process is reward-free: no reward model, no verifiable reward function, and no human annotation is needed": Technically true—no explicit reward function is constructed. But the environments provide clear textual outcome signals that the model uses to infer what constitutes success. Whether this counts as truly "reward-free" depends on whether one considers the implicit success signal in environment text as a reward proxy. In environments without clear outcome text, extraction would likely fail, so the "reward-free" claim is conditional on the environment providing informative textual feedback about outcomes.

• "OEL improves not only task accuracy but also inference efficiency": Supported by Figure 5, showing response length decreasing to ~70% of initial. This is a genuine and important finding—the model becomes both more accurate and more concise—but it's measured only for Qwen3-1.7B on Frozen Lake. Whether efficiency gains generalize to other models and tasks is not established.

• "On-policy context distillation mitigates catastrophic forgetting compared to off-policy alternatives": Supported by Figure 6 for 2,500 training samples on IF-Eval. The effect is clear but modest (1–1.5 points difference). Whether this holds for longer training, more rounds, or broader OOD evaluations is untested.

• "Extracted experiential knowledge is significantly more effective than raw trajectories": Strongly supported by Table 1. The gap is large—21.4% vs. 7.8% consolidated pass rate—and the result makes intuitive sense given the noise in raw trajectories. This is one of the most robust findings in the paper.

• "On-policy consistency between the knowledge source and the policy model is critical": Supported by Table 2 for one model pair (1.7B vs. 4B) on one task, with a substantial gap (31.1% vs. 22.7%). The finding is important but the evidence base is thin—a single pairwise comparison.

Overall, the experimental section establishes OEL as a viable framework for a specific class of problems (text-based environments with clear textual outcome feedback) and provides evidence that the extraction-consolidation decomposition is both necessary and effective. The paper would be strengthened by broader environment testing, external baselines, sample efficiency analysis, and investigation of the ceiling behavior that the diminishing returns in Figure 4 suggest. The current experiments prove that OEL works, but not yet how widely, how efficiently, or for how long the improvements continue.

6. Limitations and Trade-offs

Difficulty Estimation Cost Is Unaccounted for in the Online Learning Loop

The assumption or constraint. The entire OEL framework depends on collecting interaction trajectories from the user-side environment during deployment. The paper acknowledges this requirement implicitly through its experimental design but does not account for the cost or feasibility of trajectory collection in the reported performance numbers. Specifically, each OEL extraction round requires n = 15–50 trajectories for knowledge extraction, and each consolidation round requires 1,280–6,400 trajectory prefixes (derived from ~256–1,280 full trajectories, since each trajectory yields up to 5 prefixes) for training (Section 4.1). In the experiments, these trajectories are collected by deploying the model to play the game in batches. The paper does not discuss the cost of this deployment in terms of environment interactions, latency, or compute, nor does it amortize this cost into the reported pass rate improvements. The x-axis of Figure 4 and Figure 1 tracks "Accumulation Step" during extraction—a logical progress metric—rather than total environment interactions or FLOPs. A practitioner reading the headline "OEL achieves consistent improvements" would not immediately realize that each round of improvement requires thousands of new deployment interactions.

The consequence. For real-world deployment, the trajectory collection cost dominates the learning budget. Consider a production LLM deployed in a customer-facing application. Collecting 6,400 multi-turn trajectories (the consolidation training budget for Sokoban and larger Frozen Lake models, per Table 4) would require 32,000+ user interactions—each involving the model generating responses and waiting for the environment to return feedback. If each interaction takes even a few seconds, the total wall-clock time for data collection would be measured in hours or days. This is a practical barrier: OEL's learning loop can only advance as fast as the model can accumulate deployment experience. The paper's claim that "deployment and learning are thus connected in a virtuous cycle—the broader the deployment, the richer the signal for continued improvement" (Section 2) is correct in principle, but the experiments use a controlled game environment where trajectories can be collected on demand at essentially zero cost—a setting that does not reflect the friction of real-world deployment.

Furthermore, the extraction and consolidation phases use different trajectory sets (T and T'), meaning the total deployment interaction budget is the sum of both. A deployment with limited user traffic (a niche application, a new product with few users) may not generate enough trajectories to trigger an OEL round within a reasonable timeframe, effectively stalling the learning loop.

What evidence exists in the paper. The paper provides the raw numbers needed to calculate the trajectory cost: Table 4 specifies n = 15, 25, or 50 extraction trajectories per round, and Appendix B.4 states that consolidation uses 20 or 100 training steps with 64 game samples per step, requiring 1,280 or 6,400 trajectory samples per training round. Since each trajectory yields up to 5 prefixes, the number of full trajectories needed is approximately training_steps × 64 / 5. For the heaviest configuration (100 steps, 64 batch size), this is ~1,280 full trajectories for consolidation, plus 15–50 for extraction. The evaluation uses a held-out test split of size-128 game maps (Section 4.1), meaning the training-to-evaluation ratio is roughly 10–50:1. The paper expresses this implicitly in the configuration tables but never aggregates the numbers or discusses the practical implications.

Mitigation status. The paper does not address this limitation. Section 8 (Conclusion) frames OEL as "a promising direction" but does not discuss data efficiency, trajectory collection cost, or how the framework would scale when trajectories are scarce. There is no ablation studying how performance changes with fewer consolidation trajectories—all experiments use the fixed budgets in Table 4 without sweeping the number of training samples. A practitioner cannot determine from the paper whether OEL would still work with, say, 100 instead of 1,280 consolidation trajectories per round, or whether the gains would vanish below some threshold. The paper also does not explore whether extraction and consolidation could share trajectory sets to reduce the total deployment burden. This is a significant gap because data efficiency is often the binding constraint in online learning, not asymptotic performance.

Only Three Online Learning Rounds Are Tested, and Gains Are Diminishing

The assumption or constraint. The paper demonstrates OEL over exactly three successive rounds on Frozen Lake and two rounds on Sokoban (Figure 4, Figure 7). The authors do not claim that improvement is indefinite, but they also do not characterize the scaling behavior of OEL with more rounds. The framework is presented as a "virtuous cycle" (Section 3.3) that "can be iterated to progressively improve performance, forming an online learning loop"—language that implies sustained improvement, not a fixed number of rounds.

The consequence. The diminishing returns visible in Figure 4 raise the question of whether OEL converges to a ceiling and, if so, how quickly. On Frozen Lake (Figure 4, left), the Round 1 consolidation adds roughly +24–27 percentage points over the initial model (from ~12–15% to ~39%), Round 2 adds roughly +10 points (from ~39% to ~49%), and Round 3 adds roughly +5 points (from ~49% to ~54%). The incremental gain is halving each round. Extrapolating linearly, Round 4 would add ~2.5 points, Round 5 ~1.25 points, and the model would asymptote around ~60%. This is substantially below optimal performance—a 3×3 Frozen Lake grid with two holes is fully solvable with near-100% pass rate by a competent agent—suggesting that OEL is hitting a capability ceiling well short of the task's true ceiling.

If this diminishing-returns pattern holds generally, OEL provides a one-time bootstrap from a model's initial capability to a moderately higher plateau, rather than a mechanism for sustained, open-ended improvement. A practitioner deploying OEL would see strong gains in the first 1–2 rounds and then rapidly diminishing returns, potentially not justifying the ongoing trajectory collection and training cost of later rounds.

The paper does not investigate why gains diminish. Possible explanations include: (a) the model exhausts the learnable strategies in the task, and further rounds just re-extract and re-consolidate the same knowledge; (b) the extraction mechanism cannot produce novel insights beyond what the model can already discover, so later rounds are bounded by the extraction model's capability; (c) the consolidation mechanism saturates—the student cannot internalize additional knowledge beyond what the first round already captured. Without diagnosing the cause, it is difficult to know whether the ceiling could be raised (e.g., by using a better extraction prompt, or more training steps, or a larger model as extractor) or is fundamental to the approach.

What evidence exists in the paper. Figure 4 (left) provides the three-round data for Frozen Lake. The diminishing returns are visible in the decreasing vertical gaps between successive consolidation dots. The paper briefly acknowledges this pattern implicitly when noting that "each new iteration starts from a stronger baseline" (Section 4.2), but treats it as a positive feature (the model is improving) rather than a potential ceiling. For Sokoban (Figure 4, right), only the first round is shown in detail with a clear extraction curve and consolidation dot; the second round is partially visible, making it difficult to assess whether the same diminishing-returns pattern holds on the harder task. Figure 7 shows two rounds across model sizes for Frozen Lake, confirming that the Round 1 → Round 2 gain is consistent across scales but providing no information about Round 3 or beyond for the larger models.

Mitigation status. The paper does not address this limitation. Section 8 (Conclusion) states that "online experiential learning represents a promising direction" but does not discuss the scaling behavior, ceiling effects, or how many rounds would be practical. A key missing experiment is running OEL to convergence—continuing rounds until the pass rate stabilizes—to establish whether the framework has a natural ceiling and what that ceiling is relative to optimal performance. The paper also does not ablate whether the ceiling is caused by extraction quality or consolidation capacity (e.g., by testing whether a stronger extraction model or more training steps in later rounds can break through the plateau).

Environments Are Simple Grid-Worlds with Clear Textual Outcome Signals

The assumption or constraint. The paper evaluates OEL exclusively on two text-based game environments—Frozen Lake (3×3 grid with two holes, up to 5 turns) and Sokoban (6×6 grid with one box, up to 5 turns)—both implemented in TextArena (Section 4.1). Neither game provides explicit rules; the model must discover mechanics through exploration. However, both environments share characteristics that materially affect OEL's viability: (1) the state space is small and fully observable (the grid is small enough to fit in the model's context window), (2) the action space is minimal (four directional moves), (3) the outcome signal is binary and clearly communicated in text ("You reached the goal!" or "You fell into a hole! Game over."), (4) successful strategies are expressible as a small number of abstract rules (axis-aligned convergence, Manhattan distance reduction), (5) the maximum episode length is only 5 turns, and (6) the environments are deterministic.

The consequence. Several of these characteristics are likely necessary for OEL's extraction stage to work as demonstrated, but the paper provides no evidence on which ones are essential and which are incidental. Consider the possible failure modes in more realistic settings:

• Stochastic environments: If the same action sometimes succeeds and sometimes fails (e.g., a move that might slip on ice in a stochastic Frozen Lake variant), the extraction model may struggle to infer reliable rules from trajectories. The knowledge items in Figure 13 are deterministic principles ("at every move, the player must reduce the Manhattan distance to the goal"). A stochastic environment would require probabilistic or risk-aware knowledge ("when ice is present, prefer paths with lower slip probability"), which the extraction prompt does not explicitly elicit and which may be harder for the model to infer from a small number of trajectories.

• Ambiguous outcome signals: In these games, the environment text unambiguously indicates success or failure. In real-world deployments—code debugging (where "it compiled" doesn't mean "it's correct"), dialogue (where user satisfaction is implicit), or tool use (where partial progress is common)—the relationship between environment text and outcome quality is much murkier. The extraction model's ability to distill "winning strategies" depends on being able to identify which trajectories represent success. If that identification is unreliable, the extracted knowledge may encode spurious or harmful strategies. The paper does not test whether extraction quality degrades when success signals are noisy, delayed, or absent.

• Complex long-horizon tasks: The 5-turn limit truncates trajectories before any real planning depth develops. In Sokoban especially, 5 turns is insufficient to solve most puzzle instances—the box may need to be pushed multiple times, often requiring the player to circle around to push from different directions. The extracted knowledge (Figure 13) captures high-level principles but not the multi-step planning that Sokoban typically requires. In a longer-horizon setting, extraction would need to synthesize knowledge from partial trajectories that may not reach completion, making it harder to distinguish "good partial progress" from "dead-end actions."

• Larger state/action spaces: Both games have tiny grids (3×3 and 6×6) and four actions. The full state description fits easily in the model's context, so extraction can reference specific board configurations. In environments with larger state spaces (e.g., a 20×20 grid, or a web navigation task with hundreds of possible elements), the trajectory text would be longer, the space of possible strategies larger, and the number of trajectories needed to cover the state space would grow combinatorially. The extraction mechanism's accumulation (Equation 1) assumes a linear growth of knowledge with trajectory count, but in large state spaces, the relationship may be sublinear—each new trajectory may reveal entirely novel situations rather than reinforcing known patterns.

What evidence exists in the paper. The paper provides no experiments varying environment complexity. All results are on Frozen Lake and Sokoban at the specific grid sizes and turn limits described. The environments were deliberately simplified by replacing TextArena's original rules with a general task description (Figures 8–9), which tests whether OEL can discover rules from scratch—a valid test. But the simplicity of the resulting environments is not acknowledged as a limitation. Section 8 (Conclusion) mentions "real-world deployment" and "real-world environments" but these terms refer to the conceptual motivation, not the experimental domain. The knowledge examples in Figure 13 demonstrate that extraction produces genuinely abstract and transferable rules for these specific games, but the paper does not test whether this abstraction capability generalizes to tasks where the space of valid strategies is larger or more nuanced.

Mitigation status. The paper does not address this limitation. No text-based environments beyond these two grid-worlds are tested. The paper does not discuss what properties an environment must have for OEL to be effective (determinism? small state space? clear outcomes? short horizons?). This leaves a practitioner with no principled way to determine whether OEL would work for their specific deployment domain. The paper's title claims "Online Experiential Learning for Language Models"—an unqualified scope—but the experiments cover only a narrow slice of the space of possible "experiences." Future work on this limitation would require testing OEL on environments with systematic variation in stochasticity, horizon length, outcome signal clarity, and state space size, to map out the conditions under which the extraction-consolidation loop provides reliable gains.

No Comparison Against Simpler Methods That Also Use Deployment Data

The assumption or constraint. The paper compares OEL against variants of its own framework (raw trajectories vs. extracted knowledge, on-policy vs. off-policy distillation, self-extracted vs. other-model-extracted knowledge) but never against simpler, well-established methods that could also leverage the same deployment trajectory data. The baselines in Table 1 and Figure 6 are all internal to OEL's architecture. This implicitly assumes that OEL's specific combination of extraction-then-consolidation is necessary to achieve improvement from deployment data, rather than being one of several possible approaches.

The consequence. A practitioner reading this paper cannot determine whether OEL's architectural complexity—the extraction stage with accumulative prompting, the multiple accumulation seeds, the on-policy reverse KL distillation—is justified relative to simpler alternatives that would be easier to implement and maintain. Several natural baselines are missing:

• Supervised fine-tuning on successful trajectories (behavioral cloning): The deployment trajectories T and T' contain both successful and unsuccessful episodes. A straightforward baseline would be to filter for trajectories where the environment text indicates success (e.g., "You reached the goal!") and fine-tune the model on those trajectories using standard next-token prediction. This would use the same deployment data as OEL but without extraction or distillation. How does OEL compare?

• Few-shot prompting with curated examples: Instead of extracting abstract experiential knowledge, one could select a small number of successful trajectories and include them as few-shot examples in the model's prompt at inference time. This is the standard approach for adapting LLMs to new tasks and requires no training at all—just trajectory selection. How does OEL's consolidation compare to simply giving the model 5–10 good examples in context?

• Reinforcement learning with a learned reward model: The deployment trajectories contain clear success/failure signals in the environment text. A reward model could be trained to predict success from trajectory prefixes (a simple binary classifier on the textual outcome description), and then used with a standard RL algorithm (PPO, GRPO) to improve the policy. This would also be reward-free in the sense of requiring no human annotation—just automatic extraction of the binary outcome from environment text. How does OEL's reward-free distillation compare to reward-based RL using the same trajectory data?

The paper's Table 1 shows that raw trajectories in context (10.9%) underperform extracted knowledge in context (18.2%), which is evidence that extraction matters for in-context use. But this doesn't address whether extraction matters for training—behavioral cloning on successful raw trajectories might work well without any explicit knowledge extraction, because the training process itself would implicitly learn the patterns that extraction makes explicit.

What evidence exists in the paper. None of these baselines are tested. The paper's closest comparison is "raw trajectory" consolidation (Table 1, Consolidate column: 7.8%), but this is described as using raw trajectories as context for the teacher during on-policy distillation—it still uses the full OEL consolidation machinery. It is not a test of simple behavioral cloning. The paper also does not compare against few-shot prompting or RL with a learned reward, despite both being obvious alternatives that use the same deployment data.

Mitigation status. The paper does not address this limitation. The experiments establish that OEL works, and that within OEL's design space, certain choices (extracted knowledge over raw trajectories, on-policy over off-policy distillation, self-extracted over other-model-extracted knowledge) are better than others. But the paper cannot claim that OEL is the best way to learn from deployment data without comparing against simpler, more standard approaches. This is a significant gap because the paper is proposing a new paradigm ("Online Experiential Learning") and arguing that the extraction-consolidation decomposition is the right architecture for that paradigm. A practitioner deciding whether to adopt OEL wants to know not just that it works, but that it works better than simpler alternatives that would be cheaper to implement and debug. The paper does not provide this evidence.

On-Policy Consistency Finding Is Based on a Single Model Pair

The assumption or constraint. Section 4.6.2 and Table 2 present the finding that on-policy experiential knowledge (extracted from the model's own trajectories) significantly outperforms off-policy knowledge (extracted from a different model) for consolidation. The experiment compares Qwen3-1.7B trained with its own extracted knowledge (31.1% pass rate after consolidation) against the same model trained with knowledge extracted by Qwen3-4B (22.7%). This is a single pairwise comparison—one student model, one alternative knowledge source.

The consequence. The paper draws a strong conclusion from this single comparison: "experiential knowledge from a stronger model does not necessarily transfer well, as it may encode strategies beyond the smaller model's capabilities" and "on-policy consistency between experiential knowledge and the policy model is critical" (Section 4.6.2). This conclusion is stated as a general principle, but the evidence base is too thin to support that level of generality. Several critical questions are unanswered:

• Does the finding hold in the reverse direction? If Qwen3-4B used knowledge extracted by Qwen3-1.7B, would performance also degrade? Or is the asymmetry only in one direction (larger-to-smaller hurts, but smaller-to-larger might help)? The paper's interpretation—that larger-model knowledge encodes strategies beyond the smaller model's capabilities—predicts asymmetry: larger-model knowledge should be too complex for the smaller model, but smaller-model knowledge might still be useful for the larger model (since the larger model can execute simple strategies as well as complex ones). This prediction is not tested.

• What is the relationship between capability gap and transfer degradation? The models differ by roughly 2.4× in parameter count (1.7B vs. 4B). Would knowledge from a 0.5B model also underperform for a 1.7B model? Would knowledge from a 2B model (a much smaller gap) work nearly as well as self-extracted? Without characterizing the relationship, a practitioner cannot determine how close the knowledge source needs to be to the learning model for transfer to be effective.

• Does the finding hold across model families? Both models are from the Qwen3 family, sharing architecture, training data, and tokenizer. Would the same degradation occur if the knowledge source were from a different family (e.g., Llama-3-8B knowledge used to train Qwen3-4B)? The paper's interpretation suggests the degradation is due to capability mismatch rather than architectural mismatch, but this is untested.

• Is the degradation due to the knowledge content or the knowledge format? The Qwen3-4B extraction uses structured format with "- EXPERIENCE ITEM:" prefixes (Table 4). The Qwen3-1.7B extraction uses unstructured format. If the 1.7B model was trained on the 4B's structured knowledge, the format difference (not just the capability difference) could contribute to the degradation. The paper does not control for format when comparing knowledge sources.

What evidence exists in the paper. Only Table 2. The table reports "In-Context" and "Consolidate" pass rates for the two knowledge sources. No error bars, no statistical tests, no additional model pairs, no ablations on the capability gap. The paper's text in Section 4.6.2 presents the finding as a clear conclusion ("This suggests that experiential knowledge from a stronger model does not necessarily transfer well") without qualification about the limited evidence.

Mitigation status. The paper does not acknowledge the thinness of the evidence for this claim. The on-policy consistency finding is presented as one of the paper's key insights, but it rests on a single experiment. The paper does not suggest future work to characterize the capability gap effect or test the finding across more model pairs. This is a notable weakness because the implication—that deployment experience should not be pooled across models of different sizes—would have significant practical consequences for organizations running multiple model variants. If the finding does not generalize, practitioners might unnecessarily constrain their learning pipelines to use only self-extracted knowledge, missing opportunities to leverage experience from larger or more capable models in their fleet.

No Detection or Handling of Knowledge Quality Deterioration Across Rounds

The assumption or constraint. OEL iterates extraction and consolidation without any quality control on the extracted experiential knowledge. The extraction model is the deployed model itself (Section 4.1: "π_extract = π_θ"), and the paper explicitly states that "since the extraction process is performed server-side and we do not require scalar reward signals from the environment, we do not select the optimal experiential knowledge and instead retrieve the knowledge at the fixed accumulation step across OEL rounds." There is no mechanism to verify that the extracted knowledge is accurate, useful, or non-redundant. The knowledge is used as-is for consolidation, with random sampling from K = 10 accumulation seeds.

The consequence. In early OEL rounds, when the model's pass rate is low (12–15% initially on Frozen Lake, Figure 4), the majority of trajectories in T are likely failures—episodes where the model fell into a hole, hit a wall, or timed out. The extraction stage is supposed to learn from these failures: "conduct a deep, comparative analysis to infer the game rules and the fundamental principles behind winning and losing" (Figure 10). But the extraction model itself is the same low-performing model, which may not be capable of diagnosing its own failures accurately. If the extraction model hallucinates incorrect rules (e.g., "always move up" when that happened to work on one map but is generally harmful), or fails to identify the true cause of failure (attributing a loss to the wrong action), the extracted knowledge would encode misconceptions rather than insights. Consolidating this incorrect knowledge into parameters would then degrade the model rather than improve it, potentially creating a feedback loop where bad knowledge → worse model → worse trajectories → worse knowledge.

The paper provides no evidence that this failure mode does not occur. The consistent pass rate improvements in Figure 4 suggest that, in these specific environments, the extraction model is able to extract useful knowledge from mixed-quality trajectories. But this may be a favorable property of the simple grid-worlds tested—the relationship between actions and outcomes is straightforward, and failure modes are obvious (hole = bad, goal = good). In more complex environments where failure attribution is harder (e.g., "the code compiled but produced wrong output"—was the bug in the algorithm or in a specific implementation detail?), the extraction model may be equally likely to extract spurious or misleading "knowledge."

What evidence exists in the paper. The paper provides no direct evidence on extraction quality or error rates. The experiential knowledge examples in Figure 13 are selected illustrative examples—the paper does not state whether these are representative or cherry-picked, and does not report the frequency of incorrect, contradictory, or redundant knowledge items. The extraction prompt instructs the model to "hypothesize the game rules and effective winning strategies" (Figure 10)—the word "hypothesize" acknowledges that the extracted knowledge may be incorrect, but the paper provides no mechanism for validating these hypotheses before using them for training. The accumulation seeds (K = 10) provide some robustness through diversity (if some seeds produce bad knowledge, others may compensate), but there is no systematic filtering or quality assessment.

The ReST^EM negative result mentioned in the reference paper (where RL-based revision training caused performance to degrade) provides a cautionary analogy: when the training data generation process is on-policy and the model is imperfect, errors can compound. OEL's extraction stage has the same vulnerability—the extraction model's errors can propagate into the consolidation training signal and potentially degrade the model. The paper does not test for this failure mode (e.g., by measuring whether any OEL rounds produce knowledge contradictions or whether pass rate ever decreases after a round).

Mitigation status. The paper does not address this limitation. The extraction process is treated as a black box that reliably produces useful knowledge. There is no discussion of how to detect low-quality knowledge, how to filter it, or whether the K = 10 accumulation seeds are sufficient to dilute any individual seed's errors. This is a significant gap because, in a deployed system running OEL autonomously without human oversight, there is no safety net to catch extraction failures. A practitioner would need to build their own quality monitoring—perhaps by tracking whether extracted knowledge items are consistent across seeds, or whether the consolidated model's pass rate on a held-out validation set improves—but the paper provides no guidance on how to do this effectively or what thresholds to use.

7. Implications and Future Directions

How This Work Changes the Landscape

OEL introduces a genuinely new category of learning signal for language models—deployment experience, in the form of raw text interaction logs—that the dominant offline paradigm treats as waste. This is not an incremental improvement to an existing training recipe; it is a paradigm proposal that redefines the relationship between deployment and learning. Under the prevailing view (Section 2, Figure 2 left), deployment is the endpoint—models are trained offline on pre-constructed data, frozen, and shipped. Any capability beyond that point requires a new annotation campaign or simulation pipeline. OEL argues, and demonstrates empirically, that deployment itself can be the training pipeline, with the model's own trajectories serving as the raw material for continued improvement.

The shift in thinking this demands is fundamental: we should stop treating inference as purely consumptive and start treating it as generative of training data. Every user interaction, every environment response, every success and failure—these are not just outputs to be served and forgotten, but data points that can be mined for reusable knowledge. The paper's extraction-consolidation decomposition provides the first concrete technical framework for doing this mining at scale, without reward models or environment access. If this direction proves general, it reframes the economics of LLM deployment: the value of a deployed model is not just in the services it provides today, but in the improvement signal it generates for tomorrow.

What kind of shift is this? It is a paradigm proposal, not yet a paradigm shift—the evidence is confined to two simple grid-world games, and the broader applicability is untested. But the conceptual architecture (extraction → consolidation → iteration) is sufficiently general that if it transfers to more complex domains, it would change how practitioners think about the deployment lifecycle. The paper explicitly draws this contrast in Figure 2: offline training operates in a "closed world" of pre-constructed data, while online experiential learning operates in an "open world" of real deployment experience. The closed-world assumption is so deeply embedded in current LLM training practice (SFT on human annotations, RL on simulated environments) that challenging it qualifies as a methodological reframing, even if the experimental validation is preliminary.

Reconciling prior contradictions. The paper resolves a tension that has existed implicitly in the LLM community between two observations:

• On one hand, self-reflection and self-critique methods (Reflexion, Shinn et al., 2023; Expel, Zhao et al., 2024) have shown that models can extract useful insights from their own interaction histories and use those insights in-context to improve subsequent attempts. This suggests deployment experience contains learnable signal.
• On the other hand, these improvements are bounded by the context window and do not compound across deployment rounds. The model "learns" for the duration of a session but forgets everything when the context is cleared. This suggests that in-context learning from experience is a dead end for sustained improvement.

OEL reconciles these observations by showing that the bottleneck is not the quality of the experience signal but the storage mechanism. Extracted experiential knowledge is useful—Table 1 shows it provides an 18.2% in-context pass rate versus 7.5% without experience, nearly a 2.5× improvement. But in-context storage saturates (the transparent curves in Figure 4 plateau) and doesn't compound. Consolidation converts that transient, context-bound improvement into permanent, parameter-level capability that becomes the foundation for the next round of learning. The reconciliation is: self-reflection works, but only when paired with a parameterization mechanism that prevents the gains from evaporating. This explains why the reflection literature produced mixed results—papers that found benefits were measuring in-context performance within a single session, while papers that found no sustained benefit were looking for permanent improvement. Both were right, but they were measuring different things.

Directions that become more attractive. OEL's success makes online learning from user interactions a credible research direction for LLMs, not just a speculative vision. Several specific research agendas become newly tractable:

• Deployment-scale data flywheels: If a deployed model's interactions can be automatically converted into training data, the distinction between "training time" and "inference time" blurs. Organizations with large deployed user bases (chat platforms, coding assistants, search engines) suddenly have a continuously renewing training data source that is perfectly matched to their actual usage distribution. This makes the "data moat" argument for large-scale deployment much stronger—it's not just about market share, but about accumulating an ever-growing proprietary dataset of experiential knowledge that competitors cannot replicate.

• Domain-adaptive models that improve with use: A model deployed in a specialized domain (medical diagnosis, legal research, industrial control) could gradually accumulate experiential knowledge about that domain's specific failure modes, edge cases, and best practices—without requiring domain experts to annotate data or build simulators. The model would get better at its specific deployment context simply by being used.

• Self-improving agents in open-ended environments: The "era of experience" argument (Silver and Sutton, 2025) posits that agents should primarily learn from their own interaction with the world. OEL provides a concrete technical pathway for language-model-based agents to do this without reward engineering. An agent deployed in a novel environment (a new game, a new software tool, a new web platform) could explore, extract experiential knowledge from its exploration trajectories, consolidate that knowledge, and become competent—all without the environment designer providing a reward function.

Directions that become less attractive. OEL's results also cast doubt on several alternative approaches:

• Pure in-context reflection without parameterization (Reflexion-style methods) now appears as a local maximum—useful within a session but fundamentally bounded. The extraction curves in Figure 4 show clear saturation, and the consolidation dots consistently exceed the in-context ceiling. A researcher building an agent that "learns from experience" through reflection alone should expect diminishing returns and should consider adding a consolidation mechanism.

• Off-policy context distillation with forward KL for internalizing knowledge is shown to be strictly worse than on-policy reverse KL distillation on two axes simultaneously: lower in-distribution performance and worse OOD preservation (Figure 6). This doesn't mean off-policy distillation is useless—it may be more data-efficient or computationally cheaper—but OEL's results shift the burden of proof: a researcher choosing off-policy distillation now needs to justify why they're not using on-policy, given the demonstrated gaps in both task performance and forgetting.

• Building bespoke reward models for every new deployment environment is shown to be unnecessary for a certain class of problems. The paper's entire framework is reward-free—if the OEL paradigm generalizes, the labor-intensive process of constructing verifiable reward functions or training reward models for each new domain may be avoidable for many applications. This doesn't eliminate the need for reward models in general (especially for tasks where correct behavior is complex to specify textually), but it carves out a large space where they're not needed.

The central open question the paper leaves for the field. All of these implications are contingent on the generality of OEL beyond simple grid-world games. The paper demonstrates feasibility but does not establish generality. The field's task is now to determine the envelope: for what classes of environments and tasks does the extraction-consolidation loop actually work? The paper provides strong evidence that the answer is "at least simple text-based games with clear outcome signals." Whether the answer includes "complex open-ended tasks with ambiguous feedback" is the question that will determine whether OEL is remembered as a clever prototype or as the seed of a genuinely new training paradigm.

Follow-Up Research This Work Enables

Stress-testing OEL on environments with systematically varied properties. The most urgent follow-up is to map the boundary conditions of the extraction mechanism. The paper tests two deterministic, fully-observable grid-worlds with clear textual success/failure signals and short horizons (5 turns). A systematic study would vary one property at a time: (a) Stochasticity: introduce probabilistic transitions (e.g., Frozen Lake on slippery ice where the agent moves in the intended direction with only 80% probability). Does extraction still produce reliable knowledge when the same action sometimes succeeds and sometimes fails? One would measure whether the extracted knowledge shifts from deterministic rules ("always move toward the goal") to probabilistic or risk-aware strategies, and whether this shift happens automatically or requires prompt engineering. (b) Horizon length: increase the maximum turns from 5 to 20 or 50. Does extraction quality degrade when trajectories are longer and each individual action contributes less to the outcome? Does the accumulation mechanism (Equation 1) scale to processing 50-turn trajectories without the context being overwhelmed? (c) Outcome signal ambiguity: replace the explicit "You reached the goal!" / "You fell into a hole!" feedback with subtler cues (e.g., in a dialogue task, user satisfaction is implicit in their responses rather than explicitly stated). Does extraction still produce useful knowledge when the model has to infer success rather than being told? This line of work would produce a phase diagram for OEL—a characterization of which environment properties are necessary, which are merely helpful, and which are irrelevant. The paper's current evidence maps only a single point in this space.

Combining OEL with reinforcement learning for environments where rewards are available. The paper positions OEL as reward-free, but many deployment environments do have access to scalar rewards or verifiable outcomes (code execution passes tests, game scores, task completion metrics). A natural hybrid would use OEL's extraction stage to generate dense, token-level knowledge for the teacher, and combine the reverse KL distillation loss with a standard RL objective (PPO, GRPO) that uses the scalar reward. The hypothesis: experiential knowledge provides useful shaping—it tells the model how to succeed (strategies, rules, failure modes), while the scalar reward provides what to optimize (task completion). The ablation would compare: (1) RL alone, (2) OEL alone, (3) RL + OEL combined. If the combination outperforms either alone, it would establish that textual experiential knowledge and scalar rewards are complementary signals that capture different aspects of the task. The Sokoban environment in particular is a natural testbed—it's a puzzle where RL from sparse rewards is notoriously difficult (the box may need to be pushed away from the target to eventually reach it), and OEL's extracted strategic knowledge (Figure 13 shows items like "axis-aligned convergence") could provide the intermediate guidance that pure RL lacks.

Dynamic, per-episode extraction and consolidation rather than batched rounds. The paper's OEL operates in discrete rounds: collect a batch of trajectories, extract knowledge, consolidate, redeploy. This is a practical simplification, but it means the model's knowledge lags behind its experience—trajectories collected early in a round use an outdated model, and the extracted knowledge doesn't benefit from the model's improvements until the next round. A dynamic version would extract and consolidate continuously: after each episode (or small batch of episodes), extract new knowledge items, immediately consolidate them into the model (perhaps with a small number of gradient steps), and use the updated model for the very next episode. This is more challenging operationally (it requires training to happen concurrently with deployment) but could dramatically accelerate the learning rate—the model would benefit from its own improvements within a single deployment session rather than waiting for the next batched round. The key experiment would compare batched OEL (the paper's current approach) against continuous OEL on the same total number of trajectories, measuring pass rate as a function of episodes experienced rather than rounds. If continuous OEL achieves the same performance with fewer total episodes, it would demonstrate that the batching introduces inefficiency. The paper's consolidation procedure (single-turn rollouts from partial prefixes) is already compatible with continuous updates—the question is whether the extraction quality would suffer when processing trajectories one-at-a-time without the benefit of seeing many episodes before extracting knowledge.

Extraction model ablations: separate critic, ensemble, and capability gap characterization. The paper sets π_extract = π_θ by default and shows that off-policy extraction from a larger model degrades performance (Table 2). But the design space of extraction models is barely explored. Several specific experiments would clarify the role of the extractor: (a) Separate critic model: train a dedicated extraction model (fine-tuned specifically for the knowledge extraction task on a diverse set of environments) and use it instead of the deployed model. The hypothesis: a model specialized for extraction might produce higher-quality knowledge (more accurate, less redundant, better structured) than a general-purpose model, potentially breaking through the performance ceiling that Figure 4 shows. The experiment would compare self-extraction versus critic-extraction at each OEL round, measuring both extraction quality (human evaluation of knowledge items) and downstream consolidation performance. (b) Ensemble extraction: use multiple extraction models (different sizes, different architectures, or the same model with different prompts) and aggregate their knowledge outputs (e.g., by retaining only items that appear in multiple extracts). The hypothesis: ensemble extraction would be more robust to individual model errors and hallucinations, producing cleaner knowledge for consolidation. (c) Capability gap sweep: the on-policy consistency finding (Table 2) compares only one model pair. A systematic sweep would extract knowledge from models at multiple scales (0.5B, 1.7B, 4B, 8B, 14B, 32B) and consolidate each knowledge source into each model scale, producing a matrix of pass rates. This would reveal the functional form of the capability gap effect—does degradation set in at a threshold ratio, or is it monotonic with the parameter difference?—and provide practical guidance for multi-model deployment fleets.

Measuring and mitigating knowledge contamination across rounds. The paper acknowledges implicitly that the extraction model can produce incorrect knowledge (the prompt uses the word "hypothesize," Figure 10), but provides no mechanism for detecting or filtering bad knowledge. A crucial follow-up would measure extraction error rates and their downstream effects. The experiment: for a task where ground-truth rules are known (e.g., Frozen Lake's actual rules: move in four directions, holes end the game, goal wins), compare the extracted knowledge items against the ground truth. What fraction of items are (a) factually correct, (b) partially correct but incomplete, (c) incorrect, (d) contradictory with other items in the same accumulated knowledge sequence? Then correlate extraction error rates with consolidation outcomes—does a round with many incorrect knowledge items lead to degraded or stagnant performance? If so, filtering mechanisms become essential. Candidates include: (1) Consistency filtering: retain only knowledge items that appear in multiple accumulation seeds (K = 10 is already used; cross-seed consistency could be a quality signal without requiring ground truth). (2) Outcome-conditioned extraction: modify the extraction prompt to explicitly reference which trajectories succeeded and which failed, and instruct the extractor to generate knowledge only from successful trajectories or to contrast successful versus failed patterns. (3) Consolidation validation: after consolidation, test the new model on a small set of held-out game instances (collected server-side if possible, or from a small validation deployment) and reject the update if pass rate doesn't improve. The paper currently has no such safety net—it uses the final-step checkpoint without validation (Section 4.1). Building one would be critical for any production deployment of OEL where model regression is unacceptable.

Extending OEL to multi-modal environments and non-game tasks. The paper's environments are text-in, text-out games. The extraction mechanism relies on the model's ability to read environment feedback and write strategic knowledge—both purely textual operations. But many real deployments involve non-text modalities: a model controlling a robot receives sensor readings; a model generating images receives visual feedback; a model interacting with a GUI receives screenshots. A natural extension would test whether the extraction stage can work with multi-modal inputs—e.g., can a vision-language model extract experiential knowledge from trajectories that include both text feedback and image observations? The extraction prompt would need to reference the images (perhaps with descriptions or by passing them through the vision encoder), and the knowledge format might need to include spatial or visual concepts that are hard to express purely textually. A concrete experiment: deploy a vision-language model in a simple visual navigation environment (e.g., a 3D maze where the model sees rendered images), collect trajectories, and test whether extraction produces useful spatial knowledge ("when you see a red door, turn left") that consolidation can internalize. This would test whether OEL's textual knowledge abstraction is sufficient for non-textual experience or whether the knowledge representation needs to be fundamentally richer.

Practical Applications and Downstream Use Cases

Self-improving coding assistants deployed in IDEs. Consider a code completion or code generation model deployed inside an IDE (like GitHub Copilot). The model suggests code; the user accepts, modifies, or rejects the suggestion; the code either compiles, passes tests, or fails. These interaction trajectories—the model's suggestion, the user's action, the compiler output—are natural OEL input: they contain textual environment feedback (compiler errors, test results, user edits) and clear outcome signals (did the code work?). An OEL pipeline could extract experiential knowledge from these trajectories—e.g., "when generating a sorting function, always handle the empty-list edge case," or "in Django views, import render from django.shortcuts, not from django.http." Consolidation would bake these patterns into the model, so the next version of the assistant makes fewer of the same mistakes. The benefit: the assistant improves continuously from its own usage data, automatically adapting to the specific coding patterns, libraries, and error modes of its user base, without requiring the development team to curate new training examples or build reward models. The paper's results on token efficiency (Figure 5: response length decreasing to ~70% of initial) suggest an additional benefit—the assistant might generate more concise, targeted suggestions as experiential knowledge accumulates, reducing the cognitive load on developers reading the completions.

Domain-specific chatbots that improve from customer interactions. A customer support chatbot deployed for a specific product (e.g., a SaaS platform, an e-commerce site) generates responses to user queries. The interaction trajectories contain the user's question, the bot's response, and the user's follow-up (which may indicate satisfaction, confusion, or escalation to a human agent). These are textual environment feedback—the user's next message is effectively the "environment response" to the bot's action. OEL's extraction stage could process these trajectories to extract experiential knowledge: "when users ask about refund policies, always include the link to the refund form," or "users who mention 'billing cycle' are usually confused about proration—explain proration explicitly." Consolidation internalizes this knowledge, making the bot more helpful over time. The benefit: the chatbot adapts to the actual questions and confusion patterns of real users, which may differ substantially from the scenarios anticipated during offline training. The paper's on-policy consistency finding (Table 2) implies that each deployment of the chatbot should use its own extracted knowledge—a chatbot for Product A shouldn't be trained on knowledge extracted from interactions with Product B's users, even if Product B's model is more capable, because the knowledge may encode product-specific assumptions that don't transfer.

Game-playing agents that discover strategies through self-play and deployment. A language model agent deployed in a complex strategy game (board games, puzzle games, multi-player games) interacts with the game environment over many episodes. The game returns textual descriptions of game state and outcomes. OEL enables the agent to extract strategic knowledge—"controlling the center of the board in the opening yields a positional advantage," "in this puzzle type, solving the corners first reduces the search space"—and consolidate it into parameters. The paper's Figure 4 shows that on Frozen Lake and Sokoban, three OEL rounds more than triple the initial pass rate (from ~12–15% to ~54%). In a more complex game, the absolute numbers would be lower, but the relative improvement pattern might hold: the agent bootstraps from random exploration to competent play through self-extracted knowledge, without the game developers needing to provide a reward function or strategy guide. The virtuous cycle described in Section 3.3 is especially relevant here: as the agent improves, it explores more advanced game states, from which it can extract more sophisticated strategic knowledge, enabling further improvement. This is a form of curriculum learning driven by the agent's own improving capability rather than by an external curriculum designer.

Continuous fine-tuning of personal AI assistants from user interaction history. A personal AI assistant (calendar manager, email drafter, task prioritizer) interacts with a single user over weeks or months, accumulating a rich history of the user's preferences, communication style, and recurring tasks. The interaction trajectories contain the user's requests, the assistant's responses, and the user's corrections or confirmations. OEL's extraction stage could process these to extract personalized experiential knowledge: "the user prefers morning meetings to be scheduled after 9:30 AM," or "when summarizing email threads, the user wants action items highlighted at the top." Consolidation internalizes these preferences into the model's parameters, making the assistant increasingly personalized over time. The key practical advantage over in-context personalization (storing user preferences in a persistent prompt prefix) is the token efficiency gain demonstrated in Figure 5: the consolidated model generates shorter, more targeted responses because the personalization is in the weights, not consuming context budget. The paper's catastrophic forgetting result (Figure 6) is critical here—the assistant must improve on the personalized task without degrading its general language capabilities, and OEL's on-policy consolidation achieves this whereas off-policy alternatives cause OOD degradation.

When to Prefer This Method

The paper does not explicitly position OEL against named alternative methods with a clear tradeoff analysis. It compares OEL against internal variants (raw trajectories vs. extracted knowledge, on-policy vs. off-policy distillation) but does not articulate decision rules for when a practitioner should choose OEL over, say, supervised fine-tuning on human annotations, reinforcement learning with a learned reward model, or few-shot prompting. The experimental baselines in Figure 6 and Tables 1–2 are ablations of OEL's own components rather than external competitors. A forced "Prefer OEL when / Prefer alternative when" matrix would therefore be fabricating a comparison the paper does not make.

What the paper does establish is a set of necessary conditions for OEL to be applicable, based on its architecture and experimental design, and a set of empirical advantages of specific design choices within OEL:

OEL is only applicable when: (1) the deployment environment returns textual feedback that the model can process (the extraction stage requires text-in, text-out), (2) trajectories can be collected and sent to the server side (the user-side/server-side split in Figure 3 must be feasible), (3) the model can distinguish successful from unsuccessful outcomes from the text alone (extraction of "winning strategies" requires identifying which trajectories represent success), and (4) the task can be decomposed into partial rollout prefixes for server-side training (the environment's turn-by-turn feedback structure must be capturable in text prefixes).

Within OEL, the paper's evidence supports: using on-policy reverse KL consolidation rather than off-policy forward KL (Figure 6: higher in-distribution performance and better OOD preservation); using extracted experiential knowledge rather than raw trajectories (Table 1: 21.4% vs. 7.8% consolidated pass rate); and using self-extracted knowledge rather than knowledge from a different model when the capability gap is large (Table 2: 31.1% vs. 22.7%). These are design prescriptions for building an OEL system, not tradeoffs against entirely different learning paradigms. The paper leaves the comparison against external methods—behavioral cloning, few-shot prompting, reward-based RL—as future work.