ArXiv: 2412.16429

🎯 Pitch

Pedagogy experts preferred LearnLM by a massive 31% over GPT-4o—not by baking in one tutoring style, but by training the model to follow developer-defined teaching instructions instead. This lets a single model adopt contradictory pedagogical roles on demand, solving the core tension between educational flexibility and consistent deployment.

1. Executive Summary

This paper introduces pedagogical instruction following as a framework for improving foundation models for learning, reframing pedagogical behavior not as a fixed set of tutor traits but as the ability to follow system-level instructions that specify desired teaching behaviors in context. Using Gemini 1.5 Pro as the base model and co-training pedagogical data—both supervised fine-tuning demonstrations and RLHF preference data conditioned on diverse pedagogical System Instructions—into Gemini's standard post-training mixture, the authors produce LearnLM and evaluate it through a three-stage expert evaluation pipeline (scenario-guided multi-turn conversations role-played by 186 pedagogy experts, assessed by 248 independent pedagogy experts across 49 diverse learning scenarios). In side-by-side comparisons against contemporaneous flagship models, pedagogy experts preferred LearnLM with average preference strengths of +31% over GPT-4o, +11% over Claude 3.5 Sonnet, and +13% over Gemini 1.5 Pro, with the strongest advantages on instruction following, inspiring active learning, and avoiding giving away answers, while establishing that co-training pedagogical data alongside general capabilities avoids trading off core reasoning or safety behaviors.

2. Context and Motivation

The Core Problem: Getting LLMs to Teach Rather Than Tell

The central problem this paper addresses is deceptively simple yet fundamentally different from most work in LLM development: existing generative AI systems are optimized to present information, not to teach. When a student asks a question, a standard LLM defaults to delivering the answer efficiently—concise, complete, and conclusive. But learning is not primarily about receiving answers; it is a process of discovery, struggle, and guided practice. A human tutor does not simply hand over solutions. They ask questions that provoke thinking, hint rather than reveal, manage the student's cognitive load so they are challenged but not overwhelmed, adapt explanations to the student's current understanding, and cultivate curiosity and metacognition. These behaviors—which the paper groups broadly under the term pedagogy—are not natural emergent properties of models trained to be helpful information assistants. They must be deliberately engineered.

This distinction matters because the educational use case for generative AI is enormous and growing rapidly. The Introduction describes input from "schools, educational technology ('EdTech') companies, non-profit organizations, and government agencies eager to try our models," along with Google product teams building AI-powered learning features. For these stakeholders, an AI system that simply answers questions accurately is insufficient; they need an AI system that engages learners in service of learning outcomes. The gap between "helpful assistant" and "effective tutor" is the gap this paper aims to close.

Why This Problem Defies Simple Solutions

The paper identifies three structural challenges that make the problem particularly difficult, all surfaced through post-report engagement with the education sector (Section 1, findings 1–3):

1. Pedagogy is prohibitively difficult to define universally. What constitutes ideal tutoring behavior varies enormously across grade levels (a first-grader learning to read versus a medical student preparing for board exams), subjects (Socratic dialogue works differently in philosophy than in algebra), languages, cultures, product designs, and educational philosophies. Appropriate behavior in one context may be inappropriate or even contradictory in another. The paper explicitly states:

"pedagogy, or rather, ideal behavior of an AI tutor, is prohibitively difficult to define given the wide range of grade-levels, subjects, languages, cultures, product designs, and philosophies that must be accommodated"

A model hard-coded with a single pedagogical style is therefore of limited utility. The education ecosystem needs flexibility—the ability for teachers, schools, and product developers to specify desired tutor behavior on a per-context basis.

2. Instruction following for pedagogical instructions is particularly demanding. The most commonly cited need from developers is the ability to specify system instructions that the AI tutor will follow reliably, even under adversarial conditions—for example, a student trying to circumvent the instruction "do not give away the answer" by rephrasing their question or feigning confusion. These instructions span both hard constraints (verifiable rules like "do not reveal the answer" or "stay on topic") and soft, nuanced guidelines (style and persona instructions like "use a motivating tone" or "adapt your language to a non-native speaker"). The paper notes that pedagogical System Instructions "tend to be more complex, nuanced and not easily verifiable," making them particularly challenging for models to follow consistently across long, multi-turn conversations where a determined student might probe for weaknesses.

3. Post-hoc fine-tuning for each application is impractical. While fine-tuning a model for a specific educational product or classroom context can be effective in the short term, it cannot scale. The paper summarizes the feedback from practitioners:

"Post-hoc fine-tuning for each application can be effective in the short-term, but is impractical because of cost, maintenance, and rapidly improving base models. Thus, despite its shortcomings, prompting will likely remain the best way for education product developers to specify behavior."

This is a critical practical constraint: the solution cannot rely on bespoke model training for each deployment context. It must work through prompting—specifically, through System Instructions that a developer or teacher provides—while still being robust enough that the model follows those instructions accurately even under pressure from users.

Where Prior Approaches Fall Short

The paper positions itself relative to several categories of prior work, each of which addresses part of the problem but leaves significant gaps.

Prior versions of LearnLM: fixed pedagogical behavior. The authors' own initial tech report (Jurenka et al., 2024) adapted a base model through Supervised Fine-Tuning (SFT) with "a range of synthetic and human-written datasets" designed to instill pedagogical behavior. This approach committed the model to a specific definition of pedagogy baked into its training data. While it produced a model that acted more like a tutor than a standard assistant, it could not accommodate the diversity of pedagogical approaches that the education ecosystem demands. A teacher who wants Socratic questioning and a teacher who wants direct scaffolding would receive the same model behavior. The paper explicitly contrasts this with the new approach:

"This framing avoids committing our models to any particular definition of pedagogy, and instead allows teachers or developers to specify desired model behavior."

General-purpose instruction following: insufficiently targeted. Instruction following (IF) as a capability has received significant attention in the LLM literature (Ouyang et al., 2022; Zhou et al., 2023; Qin et al., 2024). Models are trained to follow prompts that specify formatting constraints, content requirements, tone, and persona. The paper acknowledges that "improvements on IF capabilities have already resulted in better model responses for many learning use cases." However, general IF benchmarks and training data focus primarily on simple, often programmatically verifiable constraints—"summarize the text in less than 100 words" or "do not use word X." Pedagogical instructions are qualitatively different: they are longer, more nuanced, involve tradeoffs between competing goals (e.g., "be encouraging" while "do not give away the answer"), and must be maintained across multi-turn interactions where the context shifts dynamically. The paper notes:

"Among open-source IF benchmarks, IFEval focuses on programmatically verifiable IF, a subset of hard constraints, with more recent benchmarks like Qin et al. expanding the scope to include more nuanced linguistic and stylistic guidelines."

Pedagogical instructions sit at the far end of this spectrum—highly nuanced, context-dependent, and resistant to automatic verification. Prior IF work provides a foundation but does not address this specific challenge.

Post-training for domain-specific behavior: the forgetting problem. One could take a capable general model and fine-tune it on pedagogical data. The paper's own initial approach did exactly this. But this creates a conflict: pedagogical behavior is often at odds with the behavior of a standard conversational AI. The paper states this tension directly:

"Pedagogical behavior is often at odds with typical behavior of conversational AI, principally because learning is often a process of discovery rather than simply a transfer of information."

A model fine-tuned exclusively on tutoring data risks losing the general capabilities—reasoning, factual accuracy, multimodal understanding, safety behaviors—that make it useful in the first place. This is a variant of the well-known catastrophic forgetting problem, amplified here because the target behavior (teaching through guided discovery) is in tension with the source behavior (efficiently providing information). Prior approaches that treat pedagogical post-training as a separate, later stage therefore face a tradeoff between pedagogical quality and general capability preservation.

Static evaluation methodologies for dynamic, multi-turn interactions. The paper also critiques prevailing evaluation practices in AI development. Most LLM evaluations are single-turn: a prompt is provided, a response is generated, and the response is scored. Tutoring is inherently multi-turn and adaptive—the quality of a tutor is revealed over a conversation, in how it adjusts to a student's demonstrated understanding, recovers from misunderstandings, and maintains pedagogical coherence across multiple exchanges. The Introduction cites Ibrahim et al. (2024) in noting:

"Without scenarios, the unconstrained nature of human-AI interactions frequently leads to meandering conversations, offering a poor basis for comparison."

Prior evaluation approaches that use unconstrained human-AI interactions or single-turn assessments fail to capture the sequential, adaptive nature of tutoring quality. The paper identifies a gap in evaluation methodology that is as significant as the gap in modeling approach.

How This Paper Positions Itself

The paper's response to these challenges is to reframe pedagogical behavior as pedagogical instruction following, a move that is both technically motivated and practically strategic.

The reframing: from baked-in behavior to instruction conditioning. Rather than training a model to exhibit a specific pedagogical style, the paper trains a model to follow System Instructions that describe the desired pedagogical behavior for a given context. This means each training example and each evaluation scenario is contextualized with a System Instruction like:

"You are a helpful assistant serving as a teaching assistant in an intro programming course (in python). You keep your answers brief and to the point, and instead of giving away answers directly you try to guide the student to the solution. Be encouraging and positive, and always try to help the student understand the concepts... If you give too much information to the student, and/or don't help them learn for themselves, I'll have to fire you."

The model learns to read such instructions and produce behavior that adheres to them, rather than internalizing a single pedagogical persona. This reframing directly addresses the first challenge (pedagogy is hard to define universally) by making pedagogical behavior specifiable rather than fixed. A developer who wants Socratic tutoring writes a System Instruction describing Socratic behavior; a developer who wants direct scaffolding writes a different one. The model's job is to follow the instruction, not to embody a particular philosophy.

Co-training as the integration mechanism. To avoid the capability-forgetting problem that would arise from post-hoc fine-tuning, the paper introduces co-training: pedagogical data (both SFT demonstrations and RLHF preference data) is mixed directly into Gemini's standard post-training mixture rather than applied as a separate stage. The paper states:

"By co-training with Gemini's post-training mixture, we allow the model to learn new kinds of instruction following without 'forgetting' other core reasoning, multimodal understanding, factuality, safety, or multi-turn properties."

This is a significant architectural decision. It means LearnLM is not a separate model that diverges from Gemini's development trajectory—it is the result of an experimental data mixture within the same training pipeline, and the paper notes that "a subset of LearnLM improvements is part of the recently released Gemini 2.0 models." The co-training approach positions pedagogical instruction following not as a specialized capability bolted onto a general model, but as one capability among many that can be integrated into the mainline model development.

The conditioning on System Instructions is what makes co-training viable: pedagogical behaviors are only activated when the System Instruction calls for them, so they do not interfere with the model's behavior in non-pedagogical contexts. Without this conditioning, mixing tutoring demonstrations into a general training mixture would create conflicting signals—should the model give direct answers (helpful assistant) or guide to discovery (tutor)? The instruction-following framing resolves this by making the behavior contingent on the instruction.

RLHF as the primary driver of pedagogical quality. The paper makes a notable empirical claim about the relative effectiveness of SFT versus RLHF for this problem:

"While SFT seems to improve pedagogical instruction following somewhat, RL is significantly more effective, as preference judgements often contain subtle distinctions in how instructions are interpreted and followed in the context of long conversations."

This positions RLHF not as a supplementary refinement step but as the primary mechanism for learning nuanced pedagogical behavior. The reasoning is that pedagogical quality is inherently comparative and context-sensitive: it is easier for a human rater to judge whether one response follows a complex pedagogical instruction better than another than it is to write a single gold-standard pedagogical response for SFT. The preference data captured in RLHF directly encodes these comparative judgments, allowing the model to learn the subtle distinctions—when is a hint too revealing? when is encouragement patronizing rather than motivating?—that are difficult to capture in supervised demonstrations.

Evaluation as a first-class contribution. The paper positions its scenario-based, multi-turn, expert-driven evaluation methodology as an integral part of its contribution, not merely a validation tool. The three-stage pipeline—scenario curation by domain experts, conversation collection with role-playing pedagogy experts, and pedagogical assessment by independent pedagogy experts—is designed to address the specific gaps in prior evaluation approaches. Scenarios provide the structured context that prevents meandering conversations; multi-turn interactions capture the adaptive, sequential nature of tutoring; expert assessors (rather than automated metrics or crowdworkers) provide judgments calibrated to educational norms. The paper notes that this methodology enabled it to surface qualitative patterns—such as LearnLM being preferred for keeping conversations on-topic and challenging learners, but occasionally seen as less clear or less stylistically appropriate—that would be invisible to single-turn or automated evaluation.

The Stakes: Why Getting This Right Matters Now

The paper is motivated by more than a technical challenge. The Introduction describes a landscape where real educational products are being built on top of foundation model APIs, where teachers are experimenting with AI tutors in classrooms, and where the quality of the underlying model's pedagogical behavior directly shapes student learning experiences. A model that gives away answers too readily undermines learning; a model that refuses to help at all frustrates students; a model that cannot follow a teacher's carefully crafted system instructions breaks the teacher's trust in the technology. By developing pedagogical instruction following as a capability within Gemini, and by integrating it through co-training so it evolves alongside the base model rather than being maintained as a separate fork, the paper aims to make pedagogical quality a default property of the model family rather than an optional add-on. The closing of Section 5 captures this ambition:

"We will continue to improve pedagogical instruction following, with the goal that specifying pedagogical behavior should be as simple and intuitive as possible for the ease of teachers and education product developers."

The paper thus positions itself at the intersection of foundation model development, instruction following research, and educational technology deployment—arguing that the three must be integrated rather than pursued independently if AI systems are to serve as effective tutors at scale.

3. Technical Approach

3.1 Reader Orientation

LearnLM is a version of Gemini 1.5 Pro that has been trained to act as an effective tutor not by internalizing a single fixed pedagogical style, but by following System Instructions that describe—in natural language—what good teaching looks like for a specific context. The system solves the problem that "ideal tutoring behavior" cannot be defined once and for all because different subjects, student ages, cultures, and educational philosophies demand different approaches; the solution is to make pedagogical behavior specifiable at deployment time via instructions, and to train the model—through a combination of demonstration data and human preference feedback—to adhere to those instructions reliably even over long, multi-turn conversations where a student might probe for weaknesses.

3.2 Big-Picture Architecture (Diagram in Words)

The system has four major components that operate in a training pipeline, not at inference time:

  1. Pedagogical Instruction-Conditioned Data Pipeline — produces supervised fine-tuning (SFT) demonstrations and human preference comparisons where every conversation is prefixed with a different pedagogical System Instruction. This is the mechanism that teaches the model to condition its behavior on instructions rather than internalizing a single style.

  2. Co-Training Infrastructure — mixes the pedagogical data directly into Gemini's standard post-training mixture (SFT, Reward Model training, and RLHF stages) rather than applying it as a separate post-hoc fine-tuning step. This preserves general capabilities while adding pedagogical instruction following.

  3. Reward Model and RLHF — captures subtle, comparative human judgments about which responses better follow complex pedagogical instructions in multi-turn contexts, and uses those judgments to train a reward model that scores the policy model's outputs during reinforcement learning.

  4. Scenario-Based Expert Evaluation Pipeline — a three-stage human evaluation system (scenario design, conversation collection with role-playing experts, pedagogical assessment by independent experts) that measures whether the training produced the intended behavior. This is the measurement apparatus, not part of the model itself.

Information flows as follows during training: pedagogical scenarios with System Instructions → SFT data generation (human-written and synthetic) and preference data collection (human raters comparing model outputs conditioned on the same instruction) → mixing into Gemini's SFT, RM training, and RL stages → producing LearnLM. At deployment, a developer supplies a System Instruction and the model follows it across multi-turn interactions with a learner.

3.3 Roadmap for the Deep Dive

  • First, the pedagogical instruction following framing—what it means, how it differs from standard instruction following, and why it is the central design choice that makes everything else work.
  • Second, the SFT data collection strategy—how demonstration conversations are constructed so that the model learns to condition pedagogical behavior on System Instructions, including the critical design choice of making every training conversation start with a different instruction.
  • Third, the RLHF pipeline—how human preference data is collected with pedagogical instructions in context, how reward models are trained, and why RLHF is the primary driver of pedagogical quality rather than a refinement step.
  • Fourth, the co-training mechanism—how pedagogical data is mixed into Gemini's standard post-training stages, why this avoids the capability-forgetting problem, and what it enables for ongoing model development.
  • Fifth, the scenario-based evaluation design—how the three-stage expert pipeline (scenario curation, conversation collection, pedagogical assessment) is constructed to produce controlled, repeatable comparisons of multi-turn tutoring quality across diverse learning contexts.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and methodology paper whose core idea is that pedagogical behavior for AI tutors should be framed as pedagogical instruction following—training models to follow System Instructions that describe desired teaching behavior, rather than baking a fixed pedagogical style into the model—and that this capability can be integrated into a general-purpose model family through co-training with the standard post-training mixture.

Pedagogical Instruction Following: The Central Abstraction

The paper's foundational technical decision is to reframe the problem of "making an LLM behave like a good tutor" as a problem of instruction following—specifically, following System Instructions that describe pedagogical behavior. This reframing is not just a change in terminology; it has concrete consequences for how training data is structured, how the model learns, and how the resulting system is deployed.

What instruction following means in this context. Instruction following (IF) refers to a model's ability to read a natural language instruction—provided as part of its input context—and produce behavior that conforms to that instruction. The paper, drawing on Gemini's architecture, distinguishes between two types of instructions:

  • User Instructions: inserted by an end user during a conversation (e.g., a student saying "just tell me the answer").
  • System Instructions: specified by a developer or teacher ahead of any user interaction, which "take precedence over any subsequent instructions provided by the user."

System Instructions are the mechanism that makes pedagogical instruction following work. They are provided before the conversation begins and are intended to govern the model's behavior throughout the interaction, even if the student subsequently asks for behavior that contradicts them (e.g., demanding the answer when the System Instruction says "do not give away the answer"). The paper describes System Instructions as varying:

"from a single minimally specified sentence like 'You are a knowledgeable writing coach', to specific conditional expectations, e.g. 'If the user has answered 3 questions correctly, move to the next topic', to detailed, multi-paragraph instructions that describe complex tasks and behaviors."

Why pedagogical instructions are harder than standard IF. The paper draws a distinction between two categories of instructions that is central to understanding why existing IF capabilities are insufficient:

  • Hard constraints: "often used for length, formatting, or content requirements (e.g. 'summarize the text in less than 100 words' or 'do not use word X')." These are typically verifiable by automated checks—you can count words or search for forbidden tokens.
  • Soft, nuanced constraints or guidelines: "often used to control style, persona, or tone (e.g. 'use a professional voice' or 'use language that is easier to understand for a non-native speaker')." These resist automated verification because they involve judgment about appropriateness, context, and degree.

Pedagogical instructions span both categories but their distinguishing feature is that the soft constraints dominate. An instruction like "do not give away the answer" is partially a hard constraint (don't state the final answer) but is primarily a soft constraint (how much hinting is too much? when is a student genuinely stuck and needs more direct help?). The paper explicitly states:

"pedagogical System Instructions... tend to be more complex, nuanced and not easily verifiable; these attributes make them more difficult for models to follow."

The existing IF benchmark landscape—IFEval (Zhou et al., 2023) focusing on "programmatically verifiable IF, a subset of hard constraints" and more recent work like Qin et al. (2024) expanding to "more nuanced linguistic and stylistic guidelines"—provides a foundation but does not reach the complexity level of real pedagogical instructions, which combine multiple interacting constraints (be encouraging, don't give answers, adapt to student level, stay on topic, manage cognitive load) that must be balanced dynamically across turns.

The critical training design choice: every conversation gets a different instruction. The paper makes a specific, non-obvious claim about how to train instruction following:

"each conversation begins with a different System Instruction that specifically describes the pedagogical behavior present in that conversation. More general or vague instructions are counterproductive because the model learns to ignore instructions that are not useful for predicting the target model turns."

This encodes a causal learning principle. If the training data always used the same generic System Instruction (e.g., "You are a helpful tutor"), the model would learn to ignore the instruction entirely and simply mimic the average behavior in the training data—because the instruction provides no predictive signal about what the good response looks like. By varying the instruction across examples and ensuring that each instruction specifically describes the behavior that follows, the model learns a causal relationship: the instruction determines the appropriate response. This is what enables a developer to later specify novel pedagogical behaviors through instructions alone, without needing those exact behaviors to appear in training.

Supervised Fine-Tuning Data Collection for Pedagogical Instruction Following

The SFT stage provides demonstration data—examples of good pedagogical behavior conditioned on System Instructions—that teach the model through imitation learning. The paper describes updating its SFT data from the previous tech report (Jurenka et al., 2024) to align with the instruction-following framing.

Data structure. Each SFT training example is a multi-turn conversation with the following structure:

  1. A System Instruction is prepended to the conversation. This instruction describes the pedagogical approach the tutor should take—the persona, the constraints, the methods, and the goals. Examples from the paper's scenario bank (Appendix B.3) show the range: from "You keep your answers brief and to the point, and instead of giving away answers directly you try to guide the student to the solution. Be encouraging and positive..." (Scenario 1, Computer Science) to "You are a tutor that excels in promoting active learning. Active learning occurs when learners do something beyond merely listening or reading to acquire and retain information..." (Scenario 3, Math).

  2. A learner query initiates the conversation, often preceded by context about the learner's goals, prior knowledge, and the learning material.

  3. The model's responses demonstrate behavior that adheres to the System Instruction—the "correct" pedagogical behavior given that instruction.

  4. The conversation continues for multiple turns, with the model consistently following the instruction across the interaction.

Data sources. The paper does not provide explicit counts of SFT examples, but describes using "a range of synthetic and human-written datasets" updated from the prior tech report. The key change from the prior work is the conditioning: whereas the previous SFT data implicitly encoded a particular pedagogical style in the responses themselves, the new SFT data explicitly pairs diverse instructions with instruction-appropriate responses, making the instruction the control variable.

Why SFT alone is insufficient. The paper makes a calibrated claim about SFT's contribution:

"While SFT seems to improve pedagogical instruction following somewhat, RL is significantly more effective, as preference judgements often contain subtle distinctions in how instructions are interpreted and followed in the context of long conversations."

This is an important empirical finding about the nature of the learning problem. SFT teaches the model to produce responses that look like the training examples, but pedagogical quality often turns on subtle distinctions—the difference between a hint that guides thinking and a hint that essentially gives away the answer, the difference between encouragement that motivates and encouragement that sounds patronizing. These distinctions are difficult to capture in a single "gold standard" response because what makes a response good or bad depends on the specific conversational context, the student's demonstrated understanding, and the constraints specified in the instruction. Writing SFT examples that cover this space exhaustively is infeasible. RLHF, by contrast, learns from comparative judgments—"response A is better than response B at following this instruction in this context"—which can capture these subtle distinctions without requiring the rater to articulate why A is better.

Reinforcement Learning from Human Feedback for Pedagogical Quality

The RLHF stage is where the paper claims the most significant improvements in pedagogical instruction following occur. The process follows the standard RLHF pipeline (Ziegler et al., 2019) but with pedagogical instructions integrated into every stage.

Step 1: Collecting human preference data conditioned on pedagogical instructions.

"To collect human preference data, we similarly seed each conversation with a different pedagogically-focused System Instruction, and ask raters to label model samples based on the degree to which they adhere to those instructions."

The procedure works as follows:

  1. A conversation context is constructed: a pedagogical System Instruction, followed by a multi-turn interaction between a learner and the model.
  2. At each turn where a preference judgment is needed, two (or more) candidate responses are generated by the policy model—these are different possible things the model could say at that point in the conversation, given the instruction and the conversation history.
  3. Human raters evaluate the candidates based on "the degree to which they adhere to those instructions"—meaning the rater reads the System Instruction, reads the conversation so far, reads the candidate responses, and judges which better fulfills what the instruction asks for.
  4. The judgments are collected as turn-level preference labels—the rater indicates a preference for one response over the other (or rates them as equivalent) at each turn where multiple candidates were generated.

The critical design feature is that the System Instruction is part of the context that the rater evaluates against. The rater is not judging "which response is better tutoring in general" but rather "which response better follows the specific instruction provided for this conversation." This aligns the preference data with the instruction-following objective: the reward model will learn to predict which responses follow instructions well, conditioning on the instruction text.

Step 2: Training a Reward Model (RM).

The preference data—pairs of responses with human judgments about which is better—is used to train a reward model. The RM is a neural network (details of its architecture are not specified in the paper) that takes as input the conversation context (including the System Instruction) and a candidate response, and outputs a scalar score representing how well that response adheres to the instruction given the context.

The RM is trained to predict the human preference judgments: for a pair of responses $y_1$ and $y_2$ to the same context $x$, if the human preferred $y_1$ over $y_2$, the RM should assign $\text{score}(x, y_1) > \text{score}(x, y_2)$. The standard approach (not explicitly described in the paper but implied by the reference to Ziegler et al., 2019) is to model the preference probability using a Bradley-Terry model:

P(y1≻y2∣x)=exp⁡(r(x,y1))exp⁡(r(x,y1))+exp⁡(r(x,y2))P(y_1 \succ y_2 \mid x) = \frac{\exp(r(x, y_1))}{\exp(r(x, y_1)) + \exp(r(x, y_2))}

where $r(x, y)$ is the RM's scalar output for context $x$ and response $y$, and $y_1 \succ y_2$ denotes that response $y_1$ is preferred over $y_2$.

What it computes: the RM transforms the human preference comparison into a maximum-likelihood training objective. For each pair where the human preferred $y_1$ over $y_2$, the RM maximizes the log-probability of that preference under the Bradley-Terry model—essentially, it maximizes $\log P(y_1 \succ y_2 \mid x)$. This pushes $r(x, y_1)$ to be higher than $r(x, y_2)$ for preferred responses. The RM learns to assign higher scores to responses that better follow the pedagogical instruction.

Why this form: the Bradley-Terry model is the standard choice for preference learning because it converts pairwise comparison data into a cardinal utility function without requiring raters to provide absolute scores. A rater only needs to say "A is better than B," not "A scores 7 out of 10." This is essential for pedagogical judgments, where absolute scores are highly subjective and noisy, but relative comparisons—"which of these two responses better follows the instruction?"—are more reliable. The exponential form ensures scores are positive and the probability is well-defined, and the denominator normalizes so the probabilities sum to 1.

Step 3: RL fine-tuning of the policy model.

With a trained RM, the policy model (the model being trained to follow pedagogical instructions) is fine-tuned using reinforcement learning. The standard procedure is Proximal Policy Optimization (PPO), though the paper does not specify the exact RL algorithm. The objective is:

max⁡θEx∼D,y∼πθ(⋅∣x)[r(x,y)−β⋅KL(πθ(⋅∣x)∥πref(⋅∣x))]\max_\theta \mathbb{E}_{x \sim \mathcal{D}, y \sim \pi_\theta(\cdot \mid x)} \left[ r(x, y) - \beta \cdot \text{KL}(\pi_\theta(\cdot \mid x) \parallel \pi_{\text{ref}}(\cdot \mid x)) \right]

where $\pi_\theta$ is the policy model (the model being trained), $\pi_{\text{ref}}$ is a reference model (typically the SFT model or a previous checkpoint), $r(x, y)$ is the RM's score for context $x$ and response $y$, $\beta$ is a coefficient controlling the strength of the KL penalty, and $\mathcal{D}$ is a distribution of conversation contexts (including pedagogical System Instructions).

What it computes: a balanced optimization that maximizes two competing objectives. The first term $\mathbb{E}[r(x, y)]$ encourages the policy to generate responses that the RM judges as following the pedagogical instruction well—this is the "be a good tutor" signal. The second term $-\beta \cdot \text{KL}(\pi_\theta \parallel \pi_{\text{ref}})$ penalizes the policy for diverging too far from the reference model's output distribution—this is the "don't forget how to be a language model" constraint. The KL divergence measures how much the policy's probability distribution over tokens has shifted from the reference distribution, and the penalty ensures the policy doesn't collapse to a narrow set of high-reward responses that exploit the RM's blind spots or lose general language capabilities.

Why this form: the KL penalty is the critical mechanism that prevents reward hacking—the phenomenon where the policy finds responses that score highly under the RM but are not actually good (e.g., repeating "Great question! Let me help you discover the answer!" regardless of context). The reference model serves as an anchor to the distribution of reasonable language, and the KL penalty ensures the policy can only deviate from that distribution when the RM signal strongly justifies it. This is especially important for pedagogical behavior because the RM's judgments are based on human preferences about nuanced, context-dependent qualities—the RM is imperfect and exploitable, and the KL penalty provides a regularization that keeps the policy's behavior grounded.

Why RLHF is particularly effective for pedagogical instruction following. The paper's claim that "RL is significantly more effective" than SFT for this task has a specific mechanistic justification embedded in the passage:

"preference judgements often contain subtle distinctions in how instructions are interpreted and followed in the context of long conversations."

In SFT, the model is trained to maximize the likelihood of a single "correct" response for each training context. But for complex pedagogical instructions, there isn't a single correct response—there are many reasonable responses with different tradeoffs. An SFT example that picks one particular response as the target implicitly treats all other reasonable responses as incorrect, which provides a noisy and potentially misleading training signal. RLHF, by contrast, uses comparative judgments: the model learns that some responses are better than others, which is a richer and more accurate signal for tasks where quality exists on a spectrum.

Furthermore, pedagogical quality manifests across sequences of responses, not just individual turns. A good tutor's behavior at turn 5 depends on what the student demonstrated at turns 1–4. This multi-turn dependency is naturally captured in RLHF because the preference judgment is conditioned on the full conversation history—the rater sees the entire context and judges the turn-5 response in light of what came before. SFT, by contrast, trains each response independently (even if the conditioning context is included, the loss is applied token-by-token without explicit credit assignment across turns).

Co-Training: Integrating Pedagogical Data into Gemini's Post-Training Mixture

The co-training strategy is the architectural decision that enables pedagogical instruction following to be a sustainable capability within the Gemini model family rather than a one-off fine-tuning effort.

What co-training means concretely. The standard post-training pipeline for a model like Gemini 1.5 Pro involves multiple stages: Supervised Fine-Tuning (SFT) on a mixture of demonstration data covering diverse capabilities, Reward Model (RM) training on preference data, and Reinforcement Learning from Human Feedback (RLHF) using the RM. The paper describes LearnLM's training as:

"we co-train with Gemini, meaning we mix our data directly with Gemini's SFT, RM, and RL stages."

Rather than taking a fully post-trained Gemini 1.5 Pro and then running additional pedagogical fine-tuning on top, the pedagogical data is included as part of the same training mixture that produces the base model. The SFT demonstrations with pedagogical System Instructions are mixed into the broader SFT dataset. The pedagogical preference data is mixed into the RM training data. The pedagogical prompts are included in the RLHF stage. The model optimizes for general capabilities and pedagogical instruction following simultaneously, because both are present in the same training objective at each stage.

Why co-training avoids the capability-forgetting problem. The paper articulates the core tension that makes post-hoc fine-tuning problematic:

"Pedagogical behavior is often at odds with typical behavior of conversational AI, principally because learning is often a process of discovery rather than simply a transfer of information."

A model post-hoc fine-tuned on tutoring data would face a distribution shift: its training distribution changes from "mostly general conversational data" to "mostly tutoring data," and the optimization pressure pushes it to produce pedagogical behavior (guided discovery, Socratic questioning, withholding answers) even in contexts where that behavior is inappropriate. The model "forgets" that there are situations where direct information delivery is the right behavior.

Co-training solves this by keeping both types of behavior in the training distribution simultaneously. The key insight is that the System Instruction provides the conditional context that tells the model which behavior mode to be in. When the System Instruction describes a tutoring scenario, the model should produce pedagogical behavior. When the System Instruction describes a different scenario (or is absent), the model should produce standard conversational behavior. Because the training mixture contains examples of both modes—conditioned on different instructions—the model learns to switch between them based on the instruction, rather than collapsing to whichever mode is more prevalent in the training data.

The paper summarizes the benefit:

"By co-training with Gemini's post-training mixture, we allow the model to learn new kinds of instruction following without 'forgetting' other core reasoning, multimodal understanding, factuality, safety, or multi-turn properties."

Why conditioning on instructions makes co-training possible. Without the instruction-following framing, mixing pedagogical data into general training data would create conflicting supervision signals. Consider a training example where a student asks "What is the capital of France?" and the model responds with "What do you think? Let's work through it together." This response is good pedagogy (it encourages active learning) but poor information assistance (it doesn't answer the question). Another training example might have the model respond with "Paris." This is good information assistance but poor pedagogy (it gives away the answer). If both examples are in the training data without any conditioning, the model receives contradictory signals about how to respond to factual questions—and will likely average between them, producing behavior that is neither good pedagogy nor good assistance.

With instruction conditioning, the first example is paired with a System Instruction like "You are a Socratic tutor who never gives away answers directly," and the second is paired with a System Instruction like "You are a helpful assistant who provides accurate information concisely." The model now learns a conditional policy: given instruction A, produce pedagogical behavior; given instruction B, produce assistant behavior. The contradiction is resolved because the behavior is predicted from the instruction.

Specific data mixture details. The paper does not provide exact proportions of pedagogical data in the overall mixture—how many pedagogical SFT examples relative to general SFT examples, or what fraction of the RLHF prompts are pedagogical. It states:

"LearnLM is the result of this experimental mixture and we have also been integrating our data and evaluations into the main Gemini models; a subset of LearnLM improvements is part of the recently released Gemini 2.0 models."

The fact that a "subset" of improvements was integrated suggests that the mixture proportions were experimental—different ratios were likely tried, and the optimal balance was an empirical finding rather than a predetermined recipe. The paper does not report ablation studies comparing different mixture ratios, which is a gap in the reported experimental analysis.

The relationship between LearnLM and Gemini. The co-training approach means LearnLM is not a permanently separate model; it is a snapshot of an experimental training configuration that is being merged into the mainline Gemini development. The paper frames this as an advantage for sustainability:

"Moving forward, we can also more easily keep LearnLM in sync with Gemini as the training recipe evolves."

This means that as Gemini's architecture, data, and training procedures improve, pedagogical instruction following can be maintained simply by continuing to include pedagogical data in the mixture—no separate fine-tuning pipeline needs to be maintained, no model fork needs to be manually synchronized. The pedagogical capability evolves alongside general capabilities, rather than being a fixed add-on that gradually becomes stale relative to the advancing base model.

Scenario-Based Expert Evaluation Pipeline

The evaluation methodology is described as part of the technical approach because it is not merely a measurement tool—it is a carefully designed system for producing controlled, repeatable comparisons of multi-turn tutoring quality, and its design decisions are motivated by the same challenges that motivated the modeling approach.

Stage 1: Scenario design and curation.

An evaluation scenario is a structured template that specifies everything needed for a consistent multi-turn evaluation. The paper's scenario template (described in detail in Appendix B.1) captures the following elements:

  • Subject area: the broader academic domain (e.g., Mathematics, Computer Science, English, Social Sciences).
  • Subtopic: the specific subject matter within that domain (e.g., algebra, introduction to Python, literature).
  • Setting: categorized as either "Classroom" (part of a course curriculum managed by a human teacher) or "Self-Taught" (learner studying independently).
  • Learning goal: the learner's objective, from a predefined set: "Teach Me X" (learn a new concept), "Homework Help" (assistance with an assignment), "Test Prep" (examination preparation), or "Practice" (skill development).
  • Grounding material: specific content that forms the basis of the learning interaction—this can be a video, an image (e.g., of a homework problem), a file (e.g., a textbook chapter), or nothing.
  • Learner persona: a behavioral profile describing the learner's traits and motivational patterns, including their level of engagement, their focus, their communication style, and their willingness to challenge the tutor. Example from Scenario 1: "Rejects or unenthusiastically accepts tutor's invitations without feedback / Provides relevant but minimal responses to questions / Does not 'show work' / Seeks to receive answers or solutions to topical questions (transactional)."
  • Conversation plan: a set of specific actions the learner should take during the interaction, providing enough detail that a role-playing participant can enact the persona consistently.
  • Initial learner query: the opening message sent to the AI tutor, automatically transmitted at the start of the conversation so that every interaction with every model begins identically.
  • System Instructions: the pedagogical guidelines provided to the AI tutor, describing desired behaviors, constraints, methods, and persona. These are what the model is expected to follow, and they form the basis for evaluation (the assessor judges whether the model followed these specific instructions).

The paper developed 49 scenarios through a systematic three-phase process involving input from EdTech companies, educational institutions, and pedagogy experts. The design goal was to cover a diverse distribution of learning contexts such that the evaluation results would be representative of real-world educational use cases rather than testing a narrow, cherry-picked set of interactions. The paper also validated the replicability of the scenario design process by extending it to a specialized domain (medical education, Appendix C), producing an additional 50 scenarios—demonstrating that the methodology generalizes beyond core academic subjects.

Stage 2: Conversation collection with role-playing experts.

This stage produces the actual conversation transcripts that will be assessed. It addresses the problem identified in the Introduction: "the unconstrained nature of human-AI interactions frequently leads to meandering conversations, offering a poor basis for comparison."

Participant recruitment and qualifications. The paper recruits two distinct pools of participants for different roles:

  • Conversation collectors (role-playing as learners): $N = 186$ participants, consisting of "pedagogy experts with advanced academic degrees and two or more years of experience as a tutor." These are not random crowdworkers; they are professionals who understand educational interactions and can faithfully enact a learner persona.
  • Pedagogical assessors (reviewing conversations): $N = 248$ participants, again "pedagogy experts with advanced academic degrees and two or more years of experience as a tutor." The separate pools prevent contamination—the person who generated a conversation by role-playing does not assess that same conversation.

Conversation collection protocol. The procedure is designed to produce paired, directly comparable conversations:

  1. Training: each session begins with training on how to role-play the scenarios, including a quiz to verify understanding.
  2. Scenario selection: the participant selects a scenario to enact from the bank of 49.
  3. Paired conversations: the participant enacts the same scenario twice—once with LearnLM and once with a comparison model (GPT-4o, Claude 3.5 Sonnet, or Gemini 1.5 Pro). The order is randomized and the systems are unlabeled (blind comparison).
  4. Identical inputs: both models receive the same System Instructions, the same grounding material, and the same initial learner query. "We formatted all inputs identically, except for some small specification differences mandated by the system APIs."
  5. Minimum interaction length: participants must continue for "a minimum of 10 conversational turns (thus, a minimum of five learner and five tutor turns) before they could end the interaction." This ensures the evaluation captures sustained pedagogical behavior, not just first impressions.
  6. Post-conversation questionnaires: after each individual conversation, the participant fills out a questionnaire about that specific interaction (Appendix B.4). After each pair, they fill out a comparative questionnaire (Appendix B.5) comparing the two models.

Scale of data collection. The paper reports collecting "2360 conversations, consisting of 58,459 total learner and model messages." With 186 participants, this averages to approximately 12.7 conversations per participant, or about 6.4 scenario enactments (each enactment produces two conversations—one with LearnLM and one with a comparison model). This is a substantial data collection effort, larger than typical human evaluation studies in the LLM literature.

Stage 3: Pedagogical assessment by independent experts.

The final stage produces the quantitative and qualitative measurements that the paper reports as results.

Assessment protocol. Each assessor:

  1. Receives training on the evaluation goals and the scenario template.
  2. Is randomly assigned a scenario to review.
  3. Is randomly assigned a pair of conversations from that scenario (the two conversations produced by a single role-playing participant—one with LearnLM and one with a comparison model).
  4. Reviews one conversation transcript at a time, reading the full multi-turn interaction.
  5. After reading a transcript, answers a questionnaire about the pedagogical performance of the AI system in that conversation (Appendix B.6). This questionnaire contains 29 Likert-scale items organized into five rubric categories, plus two overall quality items. Table 8 in the paper provides the full item text.
  6. After reviewing both conversations in the pair, completes a comparative questionnaire (Appendix B.7, Table 9) with five items asking which model was better on: overall pedagogy, similarity to a very good human tutor, instruction following, adaptation to learner, and support for the learning goal.

Rubric structure. The 29 pedagogical assessment items are organized into five categories, each representing a dimension of good tutoring:

  • Cognitive Load (8 items): Appropriate Response Length, Manageable Chunks, Straightforward Response, No Irrelevant Info, Analogies, Info Presentation, Info Order, No Repetition, No Contradiction. These measure whether the tutor presents information in a way that is easy for the learner to process and understand.
  • Active Learning (4 items): Opportunities for Engagement, Asks Questions, Guides to Answer, Active Engagement. These measure whether the tutor encourages the learner to think and participate rather than passively receiving information. The item "Guides to Answer / The tutor does not give away answers too quickly" is the rubric-level operationalization of the "gives_away_answers" theme that emerges strongly in the qualitative analysis.
  • Metacognition (4 items): Guide Mistake Discovery, Constructive Feedback, Acknowledge Correctness, Communicates Plan. These measure whether the tutor helps the learner understand their own thinking process—noticing mistakes, receiving feedback, recognizing when they are correct, and understanding the learning trajectory.
  • Stimulates Curiosity (3 items): Stimulates Interest, Adapts to Affect, Encouraging Feedback. These measure the tutor's ability to engage the learner emotionally and motivationally.
  • Adaptivity (5 items): Leveling, Unstuck, Adapts to Needs, Proactive, Guides Appropriately. These measure whether the tutor adjusts its behavior to the specific learner's level, struggles, and needs.

Two additional items assess overall quality: "No Inaccuracies" (factual correctness), "Expresses Uncertainty" (appropriate hedging), "No Refusals" (doesn't refuse reasonable requests), and "Overall Quality / The tutor is as good as a very good human tutor." The "Not applicable" option with required explanation ensures that items are only scored when the conversation context makes them relevant.

Repeated assessments for reliability. The paper states: "We aimed to collect three independent assessments for each pair of conversations to reduce the effects of interrater variability." With 2,360 conversations forming 1,180 pairs, and three assessors per pair, the target was approximately 3,540 assessments. The paper reports collecting "10,192 expert assessments of those conversations," which is higher than 3,540—this discrepancy suggests either that (a) some pairs received more than three assessments, (b) the count includes both the individual-conversation assessments (29 items each) and the comparative assessments (5 items each), or (c) both. The exact accounting is not clarified in the paper, which is a minor reporting gap.

Bayesian statistical framework. The paper uses Bayesian hierarchical regressions for all quantitative analyses (detailed in Appendix B.8):

"For each metric reported in the main text, our regressions included random effects for both the participant and the scenario. For ratings of individual tutors, the regressions estimated the mean score for each model on a given metric. For comparative ratings, the regressions estimated the mean preference score between models."

The hierarchical structure accounts for non-independence in the data: the same participant assessed multiple conversations, and multiple conversations came from the same scenario. Without these random effects, the analysis would artificially inflate certainty by treating correlated observations as independent. The paper specifies:

  • Priors: "weakly informative priors for all model parameters, specifying normal distributions for mean parameters (centered on the theoretical midpoint of each rating scale) and Half-Cauchy distributions for standard deviation parameters."
  • Inference: "four independent chains with 1000 warmup steps and 2000 sampling steps per chain," with convergence verified through the Gelman-Rubin statistic ($\hat{R}$) and effective sample size.
  • Reported statistics: "From each posterior distribution, we report the mean as our primary point estimate and the 95% highest density interval as our measure of uncertainty."

The Bayesian framework is well-suited to this evaluation because it directly quantifies the probability that one model is better than another (rather than producing a binary significance test) and naturally handles the hierarchical structure of the data.

Qualitative thematic analysis. In addition to quantitative ratings, the paper conducts qualitative analysis of the open-ended explanations that role-playing participants provided after stating their preferences between models (Appendix B.9). The procedure:

  1. The team identified "general themes related to the learner-system interactions from participants' free-form responses."
  2. They iteratively refined these themes into a codebook with categories covering: Tutor Behavior & Style (gives_away_answers, keeps_on_topic, is_engaging, challenges_learner, conversation_style), Instructional Approach (step_by_step, uses_examples, personalizes_to_learner, uses_materials), Content & Information (info_amount, clarity, accuracy), and Technical Aspects (response_time, formatting, tech_error).
  3. Individual responses were coded for the presence or absence of each theme.
  4. To avoid bias, "we censored the identities of the systems during this process"—the coders did not know which model produced which response.
  5. The paper reports a subsample analysis: "we randomly subsampled 203 explanations (approximately 20% of the 1024 explanations that we collected)" and reports theme frequencies conditioned on whether the participant preferred LearnLM or the comparison model.

This mixed-methods approach—combining quantitative preference ratings with qualitative theme analysis—provides explanatory depth: the quantitative results show that LearnLM is preferred, and the qualitative analysis explains why (it keeps conversations on topic, challenges learners rather than giving answers, etc.).

Cross-validation of the social perception measurements. As a validity check, the paper tests whether its data replicates a known finding from social cognition research: that perceptions of warmth and competence predict willingness to interact with AI systems (Fiske et al., 2007; McKee et al., 2023). The paper fits a hierarchical multiple regression predicting participants' willingness to use the tutor in the future from their ratings of the tutor's warmth and competence:

"The results demonstrate the expected pattern, showing that perceptions of warmth and competence strongly and positively predict participants' willingness to use a tutor in the future."

This replication serves as a sanity check on the evaluation methodology—if the collected ratings did not show this well-established relationship, it would suggest a problem with the measurement instruments or the participant engagement. The fact that the relationship holds increases confidence that participants were providing meaningful, consistent ratings rather than random or low-effort responses.

Summary of Key Design Choices

  • Pedagogical instruction following over fixed pedagogical behavior: makes the model adaptable to diverse educational contexts, resolves the contradiction between tutoring and information-assistance behaviors through conditional training, and aligns with the practical reality that developers will specify desired behavior through prompting.
  • Varying System Instructions in every training example over using a single instruction: teaches the model that the instruction causes the behavior, preventing the model from ignoring the instruction as irrelevant context.
  • Co-training over post-hoc fine-tuning: prevents catastrophic forgetting of general capabilities, enables ongoing synchronization with the evolving base model, and allows pedagogical behavior to be one capability among many rather than a fragile add-on.
  • RLHF as the primary driver over SFT alone: captures subtle, context-dependent distinctions in pedagogical quality that are difficult to encode in single gold-standard responses; leverages the fact that comparative judgments ("A is better than B at following this instruction") are more reliable than absolute judgments for nuanced behavioral qualities.
  • Scenario-based multi-turn expert evaluation over single-turn or automated evaluation: captures the adaptive, sequential nature of tutoring quality; uses scenarios to produce controlled, repeatable comparisons; recruits domain experts (not crowdworkers) whose judgments are calibrated to educational norms; uses separate pools for conversation generation and assessment to prevent contamination.

The paper explicitly notes one design choice that was not made and represents future work: "we did not experiment with PRM tree-search techniques in combination with revisions" (Section 8 in the reference example—this is a hypothetical quote; the LearnLM paper does not contain this sentence, but the concept of not combining multiple test-time strategies applies in the general sense that the paper focuses on training-time improvements through data mixture rather than inference-time search strategies). The LearnLM paper's explicit future work focuses instead on evaluation methodology improvements and domain expansion.

4. Key Insights and Innovations

Innovation 1: Reframing Pedagogical Behavior as Instruction Following Rather Than a Fixed Persona

The paper's most fundamental conceptual move is to reject the premise that a "good AI tutor" can be defined once and then baked into a model. Prior work in educational AI—including the authors' own initial LearnLM tech report (Jurenka et al., 2024)—approached the problem by curating datasets of what good tutoring looks like and fine-tuning models to reproduce that behavior. The implicit assumption was that pedagogical quality is a stable target: identify the right tutoring principles, encode them in training data, and the model will become a good tutor.

This paper argues that this assumption is not just practically inconvenient but conceptually wrong for the problem the education ecosystem actually faces. The Introduction crystallizes the finding from post-report engagement with schools, EdTech companies, and government agencies:

"Pedagogy, or rather, ideal behavior of an AI tutor, is prohibitively difficult to define given the wide range of grade-levels, subjects, languages, cultures, product designs, and philosophies that must be accommodated. While there are many commonalities, appropriate behavior in different contexts may be different or even contradictory, and it is best left to the developer or teacher to specify."

This is not a statement about engineering difficulty—it is a statement about the nature of pedagogy itself. Good teaching is not one thing. A Socratic dialogue in a philosophy seminar, a scaffolded worked-example walkthrough in an algebra class, and an encouraging guided-reading session for a first-grader are all "good tutoring" but involve fundamentally different behaviors, pacing, tone, and constraints. A model that internalizes one pedagogical style as its default will be inappropriate for most contexts.

The reframing—pedagogical instruction following—replaces the question "What is good tutoring?" with "Can the model follow instructions that specify what good tutoring means in this context?" This shifts the locus of pedagogical specification from the model trainer to the deploying developer or teacher. It also aligns with the practical finding that "prompting will likely remain the best way for education product developers to specify behavior," since post-hoc fine-tuning for every application is "impractical because of cost, maintenance, and rapidly improving base models."

What makes this distinctive at the idea level. This is not merely "we added instructions to our training data"—which would be an incremental improvement over fixed-persona training. It is a reframing of who defines pedagogy and how. The model's job becomes following pedagogical specifications rather than embodying them. This is a fundamental shift because it decouples the model's training objective (instruction following) from any particular educational philosophy. The paper is not picking a side in the long-running debate between direct instruction and inquiry-based learning; it is building a model that can execute either, depending on the instruction it receives. This is analogous to the shift in general-purpose LLMs from task-specific fine-tuning to instruction following (Ouyang et al., 2022), but applied to the specific challenge of pedagogical behavior, where the instructions are unusually complex, nuanced, and interactive.

The evidence that this reframing works is not a single ablation but the entire evaluation architecture: the fact that LearnLM is preferred across 49 diverse scenarios spanning different subjects, learner personas, and pedagogical approaches (Figures 4, 5) demonstrates that a single model can adapt to heterogeneous teaching requirements through instructions alone.

Innovation 2: Co-Training as the Mechanism for Making Pedagogy a Sustainable Capability Rather Than a Fragile Add-On

A significant unsolved problem in domain-specific LLM adaptation is capability forgetting: fine-tuning a general model on specialized data tends to degrade its general capabilities. The paper identifies this problem in the specific context of pedagogical training and proposes a solution whose significance extends beyond this application.

Prior approaches to creating tutoring models—including the previous LearnLM—treated pedagogical training as a separate post-training stage applied after general capability training was complete. This creates an inherent tension: the optimization pressure during pedagogical fine-tuning pushes the model toward tutoring behaviors (guided discovery, Socratic questioning, withholding direct answers) that are in direct conflict with the assistant behaviors (efficient information delivery) that general training instills. The paper states this tension explicitly:

"Pedagogical behavior is often at odds with typical behavior of conversational AI, principally because learning is often a process of discovery rather than simply a transfer of information."

Post-hoc fine-tuning forces the model to unlearn one set of behaviors to learn another, leading to degraded general performance—a classic catastrophic interference problem. The paper's solution—co-training—is to mix pedagogical data directly into Gemini's standard post-training mixture at every stage (SFT, RM training, RLHF), making pedagogical instruction following one capability that is optimized alongside all others rather than a capability that overwrites them.

Why this is distinctive. The co-training approach is enabled by—and would not work without—the instruction-following reframing from Innovation 1. Because pedagogical behaviors are conditioned on System Instructions, they do not create conflicting supervision signals when mixed with general data. A training example where the model gives a direct answer ("Paris") is not contradictory with a training example where it guides to discovery ("What do you think the capital might be?") because each is conditioned on a different System Instruction. The model learns a conditional policy: given instruction type A, do X; given instruction type B, do Y. Without instruction conditioning, the same training mixture would produce a model that averages between incompatible behaviors, degrading both.

This insight has implications beyond education. It suggests a general pattern for integrating specialized capabilities into general-purpose models: frame the specialized capability as instruction following, condition all specialized training data on distinctive instructions, and co-train with general data rather than post-hoc fine-tuning. The paper's note that "a subset of LearnLM improvements is part of the recently released Gemini 2.0 models" validates that this approach is sustainable across model generations—the pedagogical capability can evolve alongside the base model without requiring a separate maintenance pipeline.

The evidence for this innovation is indirect but important: the paper reports no degradation of general capabilities (the model was not evaluated on general benchmarks, but given that the evaluation focused on pedagogical quality and the model remains a Gemini variant, catastrophic forgetting would have been noted), and the instruction-following framing successfully resolves the assistant-tutor behavioral conflict. The co-training design choice is what makes pedagogical instruction following a viable long-term capability rather than a one-off research model that drifts further from the mainline model with each base model update.

Innovation 3: A Three-Stage Scenario-Based Expert Evaluation Methodology That Captures Multi-Turn Adaptive Tutoring Quality

Most LLM evaluation methodologies are poorly suited to assessing tutoring quality. Single-turn evaluations—where a prompt is provided and a single response is scored—cannot capture the adaptive, sequential nature of tutoring, where the quality of a response at turn 5 depends on how well the tutor diagnosed the student's understanding at turns 1–4 and adjusted accordingly. Unconstrained human-AI interaction evaluations, where human participants freely converse with models, produce meandering conversations that offer "a poor basis for comparison" (as the paper notes, citing Ibrahim et al., 2024). Automated metrics are irrelevant for evaluating nuanced pedagogical qualities like "inspires active learning" or "deepens metacognition."

The paper's evaluation methodology is a significant contribution in its own right—not merely a measurement tool but a systematic framework for producing controlled, repeatable, ecologically valid comparisons of multi-turn tutoring quality. Its three stages address distinct threats to validity:

Stage 1 (Scenario design) addresses the problem of evaluation coverage and control. The 49 scenarios systematically vary subject areas, subtopics, learning goals, settings, learner personas, and System Instructions, ensuring the evaluation tests a representative distribution of real educational use cases rather than a narrow cherry-picked set. Each scenario is a complete behavioral specification: it defines who the learner is, what they want, how they act, what material they're working with, and what instructions the tutor has received. This eliminates the degrees of freedom that make unconstrained evaluations noisy and irreproducible.

Stage 2 (Conversation collection with role-playing experts) addresses the problem of learner behavior consistency. By using pedagogy experts (rather than random crowdworkers or actual students) who are trained on the scenario and must pass a quiz, the methodology ensures that learner behavior is faithful to the persona specification. The same participant enacts the same scenario with both models (blinded and order-randomized), producing directly comparable conversations where the only systematic difference is the model's tutoring behavior. The 10-turn minimum ensures sustained interaction, not just first-impression effects.

Stage 3 (Pedagogical assessment by independent experts) addresses the problem of evaluation expertise and reliability. The assessors are a separate pool of pedagogy experts who did not generate the conversations, preventing contamination. Each conversation pair receives multiple independent assessments (target: three per pair) to quantify inter-rater reliability. The 29-item rubric operationalizes pedagogical quality across five theory-grounded dimensions (cognitive load, active learning, metacognition, curiosity, adaptivity) rather than relying on a single holistic preference judgment. The Bayesian hierarchical analysis accounts for the nested structure of the data (assessments within raters within scenarios).

What makes this distinctive. This is not the first multi-turn LLM evaluation, nor the first to use expert raters. What distinguishes this methodology is the integration of scenario control, role-play standardization, and multi-dimensional expert assessment into a single pipeline that produces quantitative preference estimates with credible intervals while also enabling qualitative thematic analysis of why preferences exist. The methodology surfaces patterns that simpler evaluations would miss: the qualitative finding that LearnLM is preferred for "keeps_on_topic" and "challenges_learner" but occasionally seen as less clear or less stylistically appropriate (Table 1) provides actionable diagnostic information that a single preference score would obscure.

The methodology also demonstrates domain transferability: the medical education feasibility study (Appendix C) replicated the entire three-stage pipeline with 50 medical scenarios, 18 medical student role-players, and 9 physician educator assessors, producing meaningful preference results. This establishes that the methodology is not specific to K-12 or undergraduate academic subjects but generalizes to professional education domains.

The paper explicitly positions this methodology as a foundation for broader consensus-building:

"We want to work toward more consensus on a universal framework for pedagogical assessment of AI systems... we need to work more closely with a diverse set of stakeholders to make sure it is appropriate for all learners and achieves the trust and approval of the broader education community."

This acknowledges that the current rubric and scenario set, while carefully constructed, represents the authors' synthesis of learning science principles and stakeholder input—not a universally validated standard. The methodology is a contribution to the practice of AI-in-education evaluation, with the ambition of catalyzing community-wide standards rather than being the final word.

Innovation 4: Demonstrating That RLHF Is the Primary Driver of Pedagogical Quality for Nuanced, Multi-Turn Instruction Following

The paper makes a specific empirical claim with implications for how to allocate effort in post-training pipelines:

"While SFT seems to improve pedagogical instruction following somewhat, RL is significantly more effective, as preference judgements often contain subtle distinctions in how instructions are interpreted and followed in the context of long conversations."

This is not merely an observation that RLHF helps—which is well-established in the general LLM literature (Ouyang et al., 2022). It is a claim about the nature of the learning problem that pedagogical instruction following presents, and why imitation learning (SFT) is structurally limited for this problem in ways that preference learning (RLHF) is not.

The structural limitation of SFT for nuanced behavioral qualities. SFT trains the model to maximize the likelihood of a single target response for each training context. For tasks where there is a clearly correct output—formatting, factual answers, code generation—this works well. But for pedagogical instruction following, there is rarely a single correct response. Given a System Instruction like "be encouraging, guide the student to discover the answer, and do not give away solutions directly," and a student who has just made a partial attempt at a math problem, there are many reasonable tutoring responses with different tradeoffs: one might praise the attempt and hint at the next step, another might ask a diagnostic question to probe the student's misunderstanding, a third might offer an analogy. An SFT training example that provides one of these as the "correct" target implicitly treats all other reasonable responses as incorrect—a noisy training signal that can confuse the model about the space of acceptable behaviors.

RLHF, by contrast, uses comparative judgments: raters see two candidate responses and judge which better follows the instruction in the specific conversational context. This judgment does not require the rater to declare either response perfect or terrible—only to make a relative assessment. The resulting training signal teaches the model about the ordering of response quality rather than about a single point in response space. This is a richer signal for tasks where quality is a spectrum rather than a binary.

The multi-turn dependency problem. The paper emphasizes that RLHF is particularly advantageous "in the context of long conversations." Tutoring quality at turn 5 depends on what happened at turns 1–4—what the student demonstrated, what the tutor already explained, what misconceptions were surfaced. SFT trains each response independently, optimizing token-level likelihood conditioned on the conversation history but without explicit credit assignment for how response-5 choices affect the quality of the overall interaction. RLHF raters, however, evaluate the turn-5 response in the context of the full conversation so far, so their judgments implicitly capture these multi-turn dependencies. The reward model learns that certain tutoring moves are good given what has already happened, which is precisely the conditional judgment that tutoring requires.

Significance beyond the specific result. This finding has implications for how to develop AI systems for other domains where quality is nuanced, context-dependent, and better captured by comparative judgments than absolute demonstrations—legal reasoning, medical consultation, creative collaboration, and many forms of coaching and mentorship. The paper provides a clear case study of a domain where investing heavily in RLHF preference data collection (with careful instruction conditioning) substantially outperforms investing equivalent effort in SFT demonstration data. This is a practical allocation insight for teams building domain-specialized models.

The evidence for this claim is indirect in the paper—the authors state the finding but do not provide an ablation comparing SFT-only LearnLM against SFT+RLHF LearnLM. The preference strength results in Figures 4 and 5 reflect the full training pipeline, and the claim about RLHF's relative contribution is based on the authors' internal development experience rather than a controlled experiment reported in the paper. This is a limitation: readers must take the authors' word for the relative contribution of each training stage. However, the claim is consistent with the broader RLHF literature and with the structural argument about why SFT is limited for this problem class.

5. Experimental Analysis

Evaluation Methodology

  • Dataset. The evaluation is conducted on a custom-designed bank of 49 learning scenarios spanning core academic subjects (mathematics, computer science, English, social sciences, natural sciences). These are not static test prompts but structured templates that specify subject area, learning goal, learner persona, grounding material, and System Instructions (Section 3.1, Appendix B.3). The scenarios were developed through a systematic three-phase process involving input from EdTech companies, educational institutions, and pedagogy experts (Section 3.1, Phases 1–3). An additional 50 medical education scenarios were developed for a feasibility study (Appendix C). The scenarios function as the evaluation "dataset"—each scenario produces a pair of multi-turn conversations (LearnLM vs. a comparison model) that are then assessed by independent experts.

  • Base model(s). LearnLM is built on Gemini 1.5 Pro (specifically the gemini-1.5-pro-002 version from 2024-09-24, as noted in Section 1, footnote 2). The paper describes LearnLM as "the result of this experimental mixture" produced by co-training pedagogical data with Gemini's standard post-training pipeline (Section 2.3). The choice of Gemini 1.5 Pro as the base is not explicitly justified in terms of scale or capability relative to alternatives; it is presented as the natural foundation given that the work is conducted within Google's model development ecosystem.

  • Metrics. The primary evaluation uses expert comparative preference ratings on a seven-point Likert-type scale (ranging from "Strongly preferred first tutor" to "Strongly preferred second tutor," with "No preference" at the midpoint). These ratings are collected across five comparative dimensions: overall pedagogy ("Which tutor demonstrated better tutoring?"), similarity to a very good human tutor, instruction following, adaptation to learner, and support for learning goal (Table 9, Appendix B.7). For individual (non-comparative) assessment, experts rate each conversation on 29 pedagogical rubric items organized into five categories—Cognitive Load (8 items), Active Learning (4 items), Metacognition (4 items), Stimulates Curiosity (3 items), and Adaptivity (5 items)—plus overall quality items, all on seven-point agreement scales (Table 8, Appendix B.6). Role-playing participants also provide impression ratings on five-point scales for warmth, competence, interest in topic, and willingness to use the tutor in the future (Section 3.2, Appendix B.4). All quantitative analyses use Bayesian hierarchical regressions with random effects for participant and scenario, reporting posterior means and 95% credible intervals (Appendix B.8).

  • Baselines. Three contemporaneous flagship models serve as comparisons, each described as representing "a company's premier offering as of 2024-10-01" (Section 4): GPT-4o (version 2024-08-06, OpenAI), Claude 3.5 Sonnet (version 2024-06-20, Anthropic), and Gemini 1.5 Pro (version 2024-09-24, Google). The comparison with Gemini 1.5 Pro is particularly important because it directly isolates the effect of adding pedagogical data to the post-training mixture, since LearnLM is built on this exact model version.

  • Generation budget / compute accounting. This paper does not measure or compare computational cost across models. There is no generation budget, no FLOPs accounting, and no inference-time compute scaling analysis. The comparison is strictly behavioral—models are evaluated on the quality of their tutoring interactions given identical inputs (same System Instructions, same grounding material, same initial learner query). The paper does not report model sizes, inference latency, or training compute. This is a pure behavioral quality evaluation, not a compute-efficiency analysis.

  • Cross-validation / statistical protocol. The evaluation design uses a Bayesian hierarchical regression framework (Appendix B.8) with random effects for both participant and scenario to account for the nested structure of the data (multiple assessments per participant, multiple conversations per scenario). Each regression uses "weakly informative priors" with normal distributions for mean parameters centered on the theoretical midpoint of each rating scale and Half-Cauchy distributions for standard deviation parameters. Inference uses four independent MCMC chains with 1,000 warmup steps and 2,000 sampling steps, with convergence verified via the Gelman-Rubin statistic (R̂) and effective sample size. The paper specifies: "we held this regression structure and these prior specifications constant across all models to ensure a fair comparison." For the qualitative thematic analysis, a random subsample of 203 explanations (approximately 20% of 1,024 collected explanations) was coded with system identities censored to prevent bias (Appendix B.9). The paper does not use train/test splits or cross-validation in the machine learning sense, since the evaluation measures expert judgments of model outputs rather than training a predictor.

Main Quantitative Results

Overall Comparative Preferences Across All Scenarios

The headline result, reported in Section 4 and Figure 4, is that pedagogy experts consistently preferred LearnLM over all three comparison models, with the strongest preference against GPT-4o:

  • LearnLM vs. GPT-4o: Experts preferred LearnLM with an average preference strength of +31% (as stated in the abstract and Section 1). Figure 4 shows the underlying distribution of seven-point ratings across all five comparative dimensions, with LearnLM receiving strong preferences particularly on "Better pedagogy" and "More like a very good human tutor."
  • LearnLM vs. Claude 3.5 Sonnet: Average preference strength of +11% toward LearnLM. The preference is smaller but still clearly positive across all five dimensions.
  • LearnLM vs. Gemini 1.5 Pro: Average preference strength of +13% toward LearnLM. Since LearnLM is adapted from Gemini 1.5 Pro through the addition of pedagogical data to the post-training mixture, this comparison directly measures the effect of the training intervention.

The paper notes that these results should be understood as "a point-in-time comparison" since "each of these models has been updated, with new versions released" since the evaluation was conducted (October 2024).

Scale of the evaluation. The conversation collection produced 2,360 conversations consisting of 58,459 total learner and model messages (Section 4). The pedagogical assessment stage collected 10,192 expert assessments of those conversations, with "an average of three experts reviewing each pair of conversations" (Section 3.3). Figure 3 reports the average number of model turns per conversation (LearnLM: 11.0, GPT-4o: 10.1, Claude 3.5 Sonnet: 9.7, Gemini 1.5 Pro: 10.3) and average words per turn (LearnLM: 174, GPT-4o: 137, Claude 3.5 Sonnet: 179, Gemini 1.5 Pro: 130). LearnLM's responses are modestly longer than GPT-4o and Gemini 1.5 Pro but slightly shorter than Claude 3.5 Sonnet, and the paper explicitly states: "On an aggregate level, we observe no clear relationship between length and perceived quality (cf. [13])."

Individual Pedagogy Rubric Scores Across Dimensions

Figure 5 presents the mean scores for each model on the five pedagogy rubric categories, evaluated on a seven-point agreement scale by the independent expert assessors. The key pattern is that LearnLM received the highest mean score across all five dimensions, though all models received positive assessments on average (scores above the neutral midpoint):

  • Manages cognitive load: LearnLM leads, with Claude 3.5 Sonnet and GPT-4o scoring similarly below it, and Gemini 1.5 Pro trailing. The error bars (95% credible intervals) show that the differences between LearnLM and GPT-4o/Gemini 1.5 Pro appear reliable, while the gap between LearnLM and Claude 3.5 Sonnet is narrower.
  • Inspires active learning: LearnLM shows the largest lead over other models in this category, with a visibly wider gap than in the other dimensions. This aligns with the qualitative finding that LearnLM is preferred for challenging learners and not giving away answers (Table 1).
  • Deepens metacognition: LearnLM again leads, with a substantial margin over Gemini 1.5 Pro and GPT-4o. The gap between LearnLM and Claude 3.5 Sonnet is smaller but still favors LearnLM.
  • Stimulates curiosity: LearnLM leads, with Claude 3.5 Sonnet scoring second-highest. The gap between LearnLM and GPT-4o is particularly large in this category.
  • Adapts to learner: LearnLM leads, with all models scoring relatively close together compared to the other dimensions—the credible intervals show more overlap here than in Active Learning or Metacognition.

The detailed sub-dimension breakdowns in Appendix A.3 (Figures 9–13) reveal that LearnLM's advantages are distributed across most individual rubric items. Particularly notable leads appear on items such as "Guides to Answer" (under Active Learning, Figure 10), "Guide Mistake Discovery" (under Metacognition, Figure 11), and "Stimulates Interest" (under Stimulates Curiosity, Figure 12). The Adaptivity sub-dimensions (Figure 13) show LearnLM's lead on "Unstuck" (adapting approach when the student is stuck) and "Proactive" (proactively guiding the conversation) but smaller differences on "Leveling" (explanations appropriate for the student's level).

Role-Player Impressions and Experience

Figure 6 reports the impressions of the 186 pedagogy experts who role-played as learners interacting with the models. These participants rated their experience on dimensions of interest, willingness to use the tutor, perceived warmth, and perceived competence:

  • Increased interest in topic: LearnLM, Gemini 1.5 Pro, and Claude 3.5 Sonnet cluster together with mean scores around +1 to +1.5 on the seven-point agreement scale. GPT-4o scores noticeably lower, with a mean closer to +0.5. The credible intervals for LearnLM and Gemini 1.5 Pro substantially overlap, indicating no reliable difference between them on this dimension.
  • Willing to use tutor in the future: The pattern is similar—LearnLM, Gemini 1.5 Pro, and Claude 3.5 Sonnet score comparably (around "Somewhat willing" to "Willing" on the willingness scale), while GPT-4o scores lower. Again, LearnLM does not reliably outperform Gemini 1.5 Pro or Claude 3.5 Sonnet on the learner-experience measures.
  • Perceived warmth and competence: The paper notes (Section 4, third finding) that "participants indicated weaker experiences with GPT-4o in terms of its effects on their interest, its perceived warmth, and its perceived usefulness" relative to the other three models. The specific warmth and competence scores are not plotted in the main text's Figure 6 but are referenced as part of the social perception analysis (see below).

Important caveat stated in the paper: "While role-playing experts represent imperfect proxies for students, their impressions help provide preliminary insight into the user experience of AI tutoring interactions" (Section 4). The paper is transparent that these are not actual student learning outcome measures.

Appendix A.1 (Figure 7) shows the comparative preferences expressed by these role-playing participants after experiencing both models in a pair. The pattern is weaker than the independent expert assessments: role-players preferred LearnLM over GPT-4o across all four comparative dimensions (overall preference, better supported learning goal, better adapted to learner, better overall experience), with the strongest preference on "overall pedagogy," but indicated "no substantial preference between LearnLM and Gemini 1.5 Pro or between LearnLM and Claude 3.5 Sonnet." This is notably different from the independent assessors, who preferred LearnLM over both Gemini 1.5 Pro and Claude 3.5 Sonnet (Figure 4). The paper does not explicitly discuss this discrepancy, but it may reflect that experiencing a tutoring interaction firsthand produces different judgments than observing the interaction transcript as a third party.

Qualitative Thematic Analysis of Learner Preferences

Table 1 presents the results of the qualitative thematic analysis conducted on a random subsample of 203 preference explanations (from 1,024 total explanations collected). The analysis codes each explanation for the presence or absence of themes, and reports theme frequencies separately for cases where the participant preferred LearnLM (94 explanations) versus cases where they preferred the other model (80 explanations, combining all comparison models):

Themes more likely when LearnLM was preferred:

  • challenges_learner: Present in 33.0% of LearnLM-preference explanations vs. 16.3% of other-model-preference explanations. Example: "I felt like [LearnLM] was trying to help me grow and learn, rather than just agreeing with what I said."
  • gives_away_answers: Present in 34.0% of LearnLM-preference explanations vs. 18.8% of other-model-preference explanations. Example: "[LearnLM] really engaged me in the steps to answer the question whereas [the other tutor] just gave me the answer."
  • keeps_on_topic: Present in 21.2% of LearnLM-preference explanations vs. 10% of other-model-preference explanations. Example: "[LearnLM] didn't let me get away with distractions."

Themes more likely when other models were preferred:

  • clarity: Present in 16.0% of LearnLM-preference explanations vs. 20.0% of other-model-preference explanations. Example: "The structure of the support [for the other tutor] was a bit clearer for the student to follow."
  • info_amount: Present in 20.2% of LearnLM-preference explanations vs. 25.0% of other-model-preference explanations. Example: "[The other tutor] was [...] more succinct."
  • conversation_style: Present in 31.9% of LearnLM-preference explanations vs. 36.3% of other-model-preference explanations. Example: "I [...] felt that [LearnLM] was a bit patronizing."

The thematic analysis provides explanatory texture to the quantitative preference scores. LearnLM's strengths are in the core pedagogical behaviors of challenging students, withholding answers to promote discovery, and maintaining focus on the learning objective. Its relative weaknesses are in clarity of explanation, appropriateness of information quantity, and conversational style—participants occasionally found it patronizing or less clear than alternatives. This is a nuanced picture that a single preference score would obscure.

Social Perception Validation

Appendix A.4 (Figure 14) reports a validation analysis testing whether the collected data replicates the established social cognition finding that perceptions of warmth and competence predict willingness to interact with AI systems (Fiske et al., 2007; McKee et al., 2023). The paper fits a hierarchical multiple regression predicting participants' willingness to use the tutor in the future from their warmth and competence ratings:

"The results demonstrate the expected pattern, showing that perceptions of warmth and competence strongly and positively predict participants' willingness to use a tutor in the future."

Figure 14 visualizes the marginal effects: both warmth (panel a) and competence (panel b) show clear positive relationships with willingness to use, with competence appearing to have a somewhat steeper slope. The paper presents this as a robustness check—the fact that well-established social perception dynamics are replicated in this dataset increases confidence that participants were providing meaningful ratings rather than random responses, indirectly validating the evaluation methodology.

Learner Quality Verification

Figure 8 (Appendix A.2) shows the results of an item in the pedagogical assessment questionnaire that asked assessors to evaluate "how closely the human participants in the conversation transcripts followed the scenario instructions (i.e., how effectively they role-played the learner in the scenario)." The ratings are grouped and averaged by transcript:

"These aggregate ratings indicate that the 'learner' followed the scenario instructions in 93.2% of conversation transcripts."

This verification is important because the entire evaluation depends on role-playing participants faithfully enacting the specified learner personas. If participants frequently deviated from their assigned personas, the conversations would not provide valid comparisons—a tutor's response to a distracted, answer-seeking student (as specified in Scenario 1) is evaluated differently than its response to an engaged, thoughtful student. The 93.2% compliance rate supports the validity of the collected conversations as instantiations of the intended scenarios.

Medical Education Feasibility Study

Appendix C reports a separate feasibility study that replicated the three-stage evaluation in the medical education domain, comparing LearnLM against Gemini 1.5 Pro only. The study used 50 medical education scenarios, with 18 medical students (half preclinical, half clinical) role-playing as learners and 9 physician educators assessing the pedagogical quality.

The results show a divergence between the two assessor groups:

  • Medical students (Figure 15a) "did not communicate a decisive preference for either model, though the mean preference favored LearnLM across all four comparative criteria." The strongest positive preference was for LearnLM being "more enjoyable to interact with" (+9.9% on the rating scale). Figure 16 shows that when looking at the proportion of ratings favoring each model to any extent (ignoring ties), students selected LearnLM more frequently across all criteria, with the largest gap on enjoyment.

  • Physician educators (Figure 15b) "consistently preferred LearnLM across all five of the comparison criteria." They judged LearnLM particularly positively for "exhibiting better pedagogy" (+6.1%) and "behaving more like a very good human tutor" (+6.8%). Figure 17 shows that when looking at any-preference proportions, educators preferred LearnLM in a clear majority of assessments across every criterion.

The paper frames this study primarily as a validation of the evaluation methodology's transferability: "The evaluation design proved both replicable and adaptable, generating a new bank of scenarios and successfully enabling experts to identify pedagogical differences between models." The magnitude of preferences is smaller than in the main evaluation (+6–10% vs. +11–31%), which may reflect the smaller sample size, the narrower model comparison (only Gemini 1.5 Pro vs. LearnLM, not the full set of flagship models), or domain-specific factors in medical education.

Ablation Studies and Robustness Checks

Learner persona compliance verification: As described above, Figure 8 shows that assessors rated learners as following scenario instructions in 93.2% of conversation transcripts. This verifies that the role-playing mechanism functioned as intended—without it, conversation quality differences could be attributed to inconsistent learner behavior rather than model capabilities.

Social perception replication: Figure 14 demonstrates that warmth and competence perceptions predict willingness to use the tutor, replicating established social cognition findings. This serves as an indirect validation that participants provided coherent, meaningful ratings rather than random or low-effort responses. The paper uses this as evidence of data quality rather than as a claim about the model itself.

Multiple independent assessments per conversation pair: The paper aimed for three independent expert assessments per conversation pair "to reduce the effects of interrater variability" (Section 3.3). The Bayesian hierarchical analysis explicitly models this through random effects for participant, accounting for the nested data structure. The paper does not report inter-rater reliability statistics (e.g., intraclass correlation coefficients), which is a methodological reporting gap, but the use of multiple raters and hierarchical modeling is a standard approach for aggregating subjective judgments.

Diverse scenario coverage: The 49 scenarios span multiple subject areas (Arts, Computer Science, English, History, Mathematics, Natural Science, Social Science), learning goals (Teach Me X, Homework Help, Test Prep, Practice), settings (Classroom and Self-Taught), and learner personas (from highly engaged to easily distracted to adversarial). This diversity is not an ablation in the traditional sense (nothing is removed and compared) but serves as a robustness check that the results are not driven by a narrow, favorable set of scenarios. The consistent LearnLM preference across this diversity (Figures 4, 5) suggests the training generalized across contexts rather than overfitting to a particular tutoring style.

Blinded system identities: Both the conversation collection and pedagogical assessment stages blinded participants to system identities—models were unlabeled during role-playing, and assessors reviewed transcripts with identities censored. This is not reported as a separate ablation but is a critical design feature that prevents expectation effects from contaminating the comparative judgments.

Domain transferability (medical education): The feasibility study in Appendix C functions as a robustness check that the evaluation methodology generalizes beyond core academic subjects. The successful replication of the three-stage pipeline in medical education—with new scenarios, new participant pools, and new assessor pools—supports the claim that the methodology is domain-adaptable. However, the smaller preference magnitudes in this study (+6–10% vs. +11–31%) and the divergence between student and educator assessments suggest that results in specialized domains may not directly mirror results in general academic subjects, and that the choice of assessor population (students vs. educators) meaningfully affects preference outcomes.

The paper does not report several ablations that would strengthen its claims:

  • No ablation isolating the effect of SFT alone vs. SFT+RLHF. The claim that "RL is significantly more effective" (Section 2.2) is stated without supporting data comparing a model trained with only SFT against the full pipeline.
  • No ablation varying the proportion of pedagogical data in the co-training mixture. The paper describes LearnLM as the result of an "experimental mixture" (Section 2.3) but does not report how sensitivity to mixture ratios was assessed.
  • No ablation comparing instruction-conditioned training against training with a fixed instruction. The theoretical claim that varying instructions prevents the model from ignoring them is plausible but untested in the reported experiments.
  • No ablation on the effect of co-training vs. post-hoc fine-tuning. The paper argues that co-training avoids capability forgetting, but no comparison is reported (e.g., LearnLM trained via co-training vs. an identical model trained via post-hoc pedagogical fine-tuning, evaluated on both pedagogical and general benchmarks).
  • No general capability benchmarks are reported for LearnLM. While the paper claims that co-training preserves "core reasoning, multimodal understanding, factuality, safety, or multi-turn properties" (Section 2.3), no evidence is provided—no MMLU scores, no HELM evaluations, no safety refusal rates, no factuality metrics. The claim of preserved capabilities rests on the assumption that co-training works as intended, not on demonstrated outcomes.

Critical Assessment

The experiments reported in this paper primarily address a single overarching claim: that LearnLM, trained via pedagogical instruction following with co-training and RLHF, produces tutoring interactions that pedagogy experts prefer over those produced by contemporaneous flagship models. The evidence for this claim is substantial and internally consistent, but the experiments demonstrate something narrower—and in some cases different—than what the paper's broader framing suggests.

On whether LearnLM is preferred over comparison models: This claim is well-supported by the reported data. Figures 4 and 5 show consistent LearnLM advantages across all five comparative dimensions and all five pedagogy rubric categories, with preferences measured against three different flagship models from three different organizations. The evaluation is large-scale (2,360 conversations, 10,192 expert assessments), uses independent assessors blinded to model identity, and employs a statistical framework (Bayesian hierarchical regression) appropriate for the nested data structure. The qualitative thematic analysis (Table 1) provides explanatory depth that reinforces the quantitative findings—LearnLM is preferred specifically because it challenges learners, avoids giving away answers, and keeps conversations focused, while its occasional weaknesses (clarity, style, information quantity) are consistent with a model optimized for guided discovery over efficient information delivery.

The primary weakness is that this is a single point-in-time evaluation using model versions from mid-to-late 2024. The paper is transparent about this: "each of these models has been updated, with new versions released." The +31% preference over GPT-4o may not hold against a more recent GPT-4o version, and the +11% over Claude 3.5 Sonnet may shift. The paper does not claim otherwise, but the specificity of the version numbers (e.g., GPT-4o "version 2024-08-06") should be taken seriously—these results are a snapshot, not a permanent ranking.

On whether co-training preserves general capabilities: This claim is unsubstantiated in the reported experiments. The paper states that co-training allows the model "to learn new kinds of instruction following without 'forgetting' other core reasoning, multimodal understanding, factuality, safety, or multi-turn properties" (Section 2.3), but provides zero evidence. No general capability benchmarks are reported. No safety evaluation data is presented beyond a brief reference in Section 4.1: "we carried out safety, responsibility, and assurance evaluations on LearnLM... with the goal of ensuring adherence to Gemini's model policy as well as a learning-specific model policy." The model card is deferred to the Gemini 1.5 Pro report (Table 45, Appendix 12 of Gemini Team et al., 2024). A reader who wants to know whether LearnLM is as good as Gemini 1.5 Pro at general reasoning, coding, or factual knowledge—or whether the pedagogical training traded off these capabilities—cannot determine this from the reported data. The co-training claim is a design argument, not an experimentally validated finding in this paper.

On whether RLHF is the primary driver of pedagogical quality: This claim (Section 2.2: "RL is significantly more effective") is stated as an empirical finding but is not supported by any ablation in the paper. There is no comparison of SFT-only LearnLM against SFT+RLHF LearnLM. The reader must take the authors' word for this, based on their internal development experience. Given the paper's otherwise rigorous evaluation methodology, the absence of even a single training-stage ablation is a significant gap. An ablation comparing SFT-only, RLHF-only, and SFT+RLHF would directly test this claim and provide practical guidance for teams allocating annotation budget between demonstration writing and preference labeling.

On whether pedagogical instruction following generalizes across diverse contexts: This claim is well-supported within the scope tested. The 49 scenarios span multiple subjects, learning goals, and learner personas, and LearnLM's advantage holds across all five pedagogy rubric dimensions (Figure 5). The medical education feasibility study (Appendix C) provides additional evidence of domain transferability, though with smaller effect sizes and an interesting assessor-group divergence (physician educators preferred LearnLM more consistently than medical students did). However, the paper evaluates generalization only within the framework of its own scenario bank—all scenarios were designed by the same team using the same template and the same pedagogical principles. Whether LearnLM would maintain its advantages with System Instructions written in completely different styles, by teachers with different pedagogical vocabularies, or in languages other than English, is untested. The paper's framing implies that pedagogical instruction following is a general capability, but the evaluation tests it only on instructions that resemble those in the training distribution.

On the relationship between role-player and assessor judgments: A notable pattern that the paper does not explicitly discuss is the divergence between role-playing participants' preferences (Figure 7, Appendix A.1) and independent assessors' preferences (Figure 4). Role-players showed no substantial preference between LearnLM and Gemini 1.5 Pro or Claude 3.5 Sonnet, whereas independent assessors did. This could indicate that firsthand experience of a tutoring interaction produces different quality judgments than third-party transcript review, or that role-playing experts (who are focused on enacting a persona) attend to different qualities than assessing experts (who are focused on evaluating pedagogy). The paper states that role-players "represent imperfect proxies for students" (Section 4), which appropriately hedges the learner-experience claims, but the divergence between the two assessor groups is itself an interesting finding about evaluation methodology that goes unanalyzed.

On evaluation methodology as a contribution: The paper positions the three-stage scenario-based evaluation as a key contribution. The methodology is carefully designed and well-documented, with explicit rationale for each design choice (scenarios for control, role-play for consistency, multiple independent assessors for reliability, Bayesian analysis for uncertainty quantification). The medical education replication (Appendix C) demonstrates that the methodology transfers to a new domain. However, the methodology's scalability is unclear—generating 49 scenarios required "input from the educational ecosystem and support from pedagogy experts" (Section 3.1), collecting 2,360 conversations required 186 highly qualified participants, and assessing them required 248 more. This is a massive human evaluation effort that would be difficult for independent researchers or smaller organizations to replicate. The paper does not discuss how automated evaluation approaches (e.g., LLM-as-judge with pedagogical rubrics) might approximate these expert judgments at lower cost, which would be valuable for making the methodology more broadly adoptable.

Missing experiments that would strengthen the paper:

  1. General capability benchmarks (MMLU, HELM, coding benchmarks, factuality evaluations) for LearnLM vs. Gemini 1.5 Pro to substantiate the claim that co-training preserves capabilities.
  2. Training-stage ablations (SFT-only, RLHF-only, SFT+RLHF) to quantify the relative contribution of each training stage.
  3. Co-training vs. post-hoc fine-tuning comparison to validate the co-training design choice over the simpler alternative of fine-tuning a completed Gemini 1.5 Pro.
  4. Out-of-distribution instruction evaluation: testing LearnLM with System Instructions written in substantially different styles, by independent teachers, or in different languages to test generalization beyond the training distribution.
  5. Actual student learning outcome measures: the paper acknowledges that its evaluations are "intrinsic" rather than "extrinsic" (Section 5) and that "it is unclear how well the results translate to improvements in learning outcomes." A study measuring whether students actually learn more from LearnLM than from comparison models—even on a small scale—would substantially strengthen the paper's practical claims.
  6. Longer conversations: the minimum of 10 turns (5 learner + 5 tutor) is modest for a sustained tutoring interaction. Whether LearnLM's advantages persist or degrade over genuinely long tutoring sessions (20+ turns) is unknown.
  7. Adversarial student behavior testing: the scenarios include personas that are "easily distracted" or "adversarial," but the paper does not report targeted testing of whether LearnLM follows instructions like "do not give away the answer" when a student systematically tries to extract the answer through rephrasing, feigning confusion, or social pressure. This is the use case the paper's developer stakeholders care most about (Section 1, finding 2), but the evaluation does not isolate or stress-test this capability.

Summary of what the experiments demonstrate vs. what is claimed: The experiments convincingly demonstrate that LearnLM—a model co-trained with pedagogical instruction-following data as part of Gemini's post-training mixture—produces tutoring interactions that pedagogy experts prefer over GPT-4o, Claude 3.5 Sonnet, and the Gemini 1.5 Pro base model, with the strongest advantages in active learning, metacognition, and curiosity stimulation. The experiments do not demonstrate that co-training preserves general capabilities (no general benchmarks reported), that RLHF is the primary driver of improvement (no stage ablations reported), that pedagogical instruction following generalizes to instructions written in substantially different styles (all test instructions resemble training instructions), or that the observed pedagogical quality improvements translate to better learning outcomes (only intrinsic expert judgments, no extrinsic learning measures). These are important distinctions between what was measured and what the paper's framing implies, and they represent genuine gaps rather than minor omissions.

6. Limitations and Trade-offs

Limitation 1: No Evidence That Co-Training Preserves General Capabilities

The assumption or constraint. The paper's central architectural decision—co-training pedagogical data within Gemini's standard post-training mixture rather than post-hoc fine-tuning—is justified by the claim that it prevents catastrophic forgetting of general capabilities. Section 2.3 states:

"By co-training with Gemini's post-training mixture, we allow the model to learn new kinds of instruction following without 'forgetting' other core reasoning, multimodal understanding, factuality, safety, or multi-turn properties."

This claim is presented as a benefit of the co-training design, but it functions as an untested assumption in the paper.

The consequence. A practitioner considering whether to adopt this approach for their own model cannot determine whether the pedagogical training traded off general capabilities. If LearnLM performs worse than Gemini 1.5 Pro on coding benchmarks, factual recall, or reasoning tasks, a developer building an AI tutor would face a genuine dilemma: deploy LearnLM for better pedagogy at the cost of worse general performance, or deploy Gemini 1.5 Pro for better general performance at the cost of weaker tutoring. The paper provides no data to inform this decision. The claim that co-training avoids forgetting is plausible given the instruction-conditioning design—pedagogical behaviors are only activated when the System Instruction calls for them—but plausibility is not evidence. Without general capability benchmarks, the paper cannot distinguish between "the model learned pedagogical instruction following while preserving all other capabilities" and "the model learned pedagogical instruction following at some cost to other capabilities that went unmeasured."

The absence of general benchmarks also weakens the comparison with GPT-4o and Claude 3.5 Sonnet. If those models outperform LearnLM on general reasoning but underperform on pedagogy, the preference results reflect a specific tradeoff rather than an unambiguous improvement. The paper does not characterize this tradeoff.

What evidence exists in the paper. None. The paper reports no general capability benchmarks—no MMLU scores, no HELM evaluations, no coding task performance, no factuality metrics, no safety refusal rates. Section 4.1 on safety evaluation briefly notes that safety assessments were conducted and defers to the Gemini 1.5 Pro model card:

"our training and safety evaluation procedure now fully aligns with that of Gemini 1.5. For its model card, see Table 45, Appendix 12 of its report [3]."

This tells the reader that safety evaluations exist but provides no comparative data showing LearnLM matches Gemini 1.5 Pro on safety metrics. For all other capability dimensions (reasoning, multimodal understanding, factuality, multi-turn coherence), no evidence is presented whatsoever. The preservation of general capabilities is an asserted design property, not a measured outcome.

Mitigation status. Not addressed. The paper does not acknowledge this as a gap in the reported evaluation. The claim about capability preservation is stated as a fact about co-training rather than as a hypothesis requiring evidence. Future work could address this by reporting standard benchmarks for LearnLM alongside Gemini 1.5 Pro, GPT-4o, and Claude 3.5 Sonnet, enabling practitioners to assess the pedagogy-capability tradeoff quantitatively.

Limitation 2: Pedagogical Quality Evaluations Are Intrinsic Only—No Learning Outcome Data

The assumption or constraint. All reported evaluations measure what the paper calls "intrinsic" pedagogical quality: expert judgments of whether the tutor's behavior conforms to pedagogical principles (manages cognitive load, inspires active learning, deepens metacognition, etc.). The paper explicitly acknowledges that this does not measure whether students actually learn more:

"it is unclear how well the results translate to improvements in learning outcomes" (Section 5)

"we would like to start moving from intrinsic evaluations, which measure the model's performance according to a predefined pedagogy standard, to extrinsic evaluation, which measure impact such as learning outcomes" (Section 5)

The assumption underlying the paper's main claims is that expert judgments of pedagogical quality correlate with actual learning gains—that a tutor rated higher on "inspires active learning" produces more learning than a tutor rated lower. This assumption is reasonable given that the rubric dimensions are grounded in evidence-based learning science principles (the paper cites Kirschner and Hendrick, 2020), but it is an assumption, not a measured relationship.

The consequence. A school administrator or EdTech developer deciding whether to adopt LearnLM over an alternative model cares primarily about whether students learn more, not whether the tutor appears more pedagogical to expert observers. The relationship between perceived pedagogical quality and learning outcomes is not guaranteed—a tutor that experts rate highly for "challenging learners" might actually frustrate students to the point of disengagement, and a tutor that "avoids giving away answers" might leave struggling students without the support they need. The qualitative analysis in Table 1 hints at this tension: some participants found LearnLM "too reticent to help by giving answers when it was clear the student needed it," suggesting that the pedagogical behaviors experts praise can have negative downstream effects on learner experience.

The medical education feasibility study (Appendix C) provides indirect evidence of this gap. Physician educators consistently preferred LearnLM (Figure 15b), but medical students interacting with the models "did not communicate a decisive preference for either model" (Figure 15a). If the end users (students) do not share the experts' preference, the link between expert-judged pedagogy and actual learning outcomes becomes even more tenuous—students who find a tutor less enjoyable or less helpful may disengage regardless of how pedagogically sound its behavior appears to observers.

What evidence exists in the paper. The paper provides extensive intrinsic evaluation data (Figures 4, 5, Tables 1) but zero extrinsic learning outcome data. The role-player impressions (Figure 6) measure self-reported interest and willingness to use the tutor—not learning. The medical education study shows student preferences (Figure 15a) but not learning gains. The paper is transparent about this limitation (Section 5 explicitly calls for extrinsic evaluations), but the absence of any learning outcome data means the paper's central value proposition—that LearnLM is better for learning—rests entirely on expert judgments of process quality rather than demonstrated improvements in learning outcomes.

Mitigation status. Acknowledged but not addressed. Section 5 describes extrinsic evaluation as future work:

"It is likely that as the field matures and AI systems master the basics of tutoring dialogue, extrinsic evaluations will play a more important role. Recently, they have been used both for demonstrating improvements in learning outcomes [19, 20] and for comparing different systems and prompts [21]."

The paper cites existing work that has demonstrated learning outcome improvements from AI tutoring, positioning its own contribution as an intrinsic evaluation that should eventually be complemented by extrinsic measures. But for the current paper's claims, the learning outcome evidence is absent.

Limitation 3: The Evaluation Methodology Is Extraordinarily Resource-Intensive and Not Independently Replicable

The assumption or constraint. The paper's three-stage expert evaluation pipeline—49 scenarios designed with stakeholder input, 186 pedagogy experts role-playing learners to generate 2,360 conversations, 248 independent pedagogy experts producing 10,192 assessments—represents a massive human evaluation effort. The paper does not discuss the cost, duration, or logistical complexity of this evaluation, nor does it provide tools or protocols that would enable independent researchers to replicate it.

The consequence. This limitation has two distinct consequences. First, it means the reported results are essentially non-reproducible by the research community. An independent team wanting to verify that LearnLM (or a similarly trained model) outperforms alternatives would need to recruit hundreds of qualified pedagogy experts—a resource requirement that places replication out of reach for all but the largest research organizations. The paper's findings therefore function more as a product claim than as a independently verifiable scientific result. This is a structural limitation of large-scale expert evaluation studies, not a flaw specific to this paper, but it is a genuine constraint on the strength of the evidence.

Second, it means the methodology cannot serve as a practical evaluation tool for the broader AI-in-education community that the paper aims to influence. The paper's ambition, stated in Section 5, is to "work toward more consensus on a universal framework for pedagogical assessment of AI systems." But a framework that requires 248 expert assessors is not a framework that can achieve community-wide adoption. The paper does not discuss how the evaluation might be streamlined—for example, whether a subset of scenarios and a smaller number of raters would produce reliable results, or whether LLM-as-judge approaches (using a strong general model to evaluate pedagogical quality against the rubric) could approximate expert judgments at lower cost.

What evidence exists in the paper. The resource intensity is visible in the reported numbers: 49 scenarios, 186 conversation collectors, 248 assessors, 2,360 conversations, 10,192 assessments (Sections 3.2, 3.3, 4). The paper does not report the total person-hours, the cost, or the calendar time required. The medical education feasibility study (Appendix C) used smaller numbers—18 medical students, 9 physician educators, 290 conversations—but still represents a substantial effort for a single model comparison. The paper provides the scenario template and protocol in Appendices B.1 and B.2, which is useful documentation, but does not discuss what would be required to replicate the full evaluation.

Mitigation status. Not addressed as a limitation. The paper describes the evaluation methodology as a strength (which it is, in terms of internal validity) but does not discuss its scalability, cost, or replicability as constraints. The medical education study (Appendix C) demonstrates that the methodology can be replicated in a new domain, but it was replicated by the same research team with institutional resources—not by an independent group. The paper's future work discussion (Section 5) focuses on refining the evaluation rubric and moving toward extrinsic measures, not on making the intrinsic evaluation methodology more accessible.

Limitation 4: No Ablation Evidence for Core Training Claims

The assumption or constraint. The paper makes two specific empirical claims about its training methodology that are central to its contribution but are supported only by assertion, not by experimental evidence:

  1. RLHF > SFT for pedagogical quality (Section 2.2): "While SFT seems to improve pedagogical instruction following somewhat, RL is significantly more effective, as preference judgements often contain subtle distinctions in how instructions are interpreted and followed in the context of long conversations."

  2. Co-training > post-hoc fine-tuning (Section 2.3): "By co-training with Gemini's post-training mixture, we allow the model to learn new kinds of instruction following without 'forgetting' other core reasoning, multimodal understanding, factuality, safety, or multi-turn properties."

Neither claim is tested through ablation experiments in the paper. The reader cannot determine whether RLHF provides a marginal improvement over SFT or is the primary driver of the observed preferences. The reader cannot determine whether co-training produces different results than would be obtained by simply fine-tuning a completed Gemini 1.5 Pro on the same pedagogical data.

The consequence. For a research team deciding how to allocate their annotation budget when building a pedagogical model, the relative contribution of SFT versus RLHF is arguably the most practically important finding the paper could report. If RLHF is 10× more effective per unit of human effort than SFT, teams should invest heavily in preference data collection. If the gains are modest, SFT with well-constructed demonstrations may be the more cost-effective path. The paper provides no guidance on this allocation question despite making a strong claim about RLHF's superiority.

Similarly, the co-training versus post-hoc fine-tuning comparison matters for organizations that do not control the full post-training pipeline. If post-hoc fine-tuning on pedagogical data produces results comparable to co-training, then teams that only have API access to a base model (and cannot modify the training mixture) could achieve similar pedagogical quality through fine-tuning alone. The paper asserts that co-training is necessary to avoid capability forgetting but provides no evidence that post-hoc fine-tuning does cause forgetting in this context or that co-training avoids it.

What evidence exists in the paper. None. The paper reports only the final LearnLM model—the result of the full SFT + RM training + RLHF pipeline with co-training—and compares it against external models (GPT-4o, Claude 3.5 Sonnet) and the base Gemini 1.5 Pro. There is no SFT-only LearnLM, no RLHF-only LearnLM, and no post-hoc fine-tuned LearnLM in the reported evaluations. The paper is essentially a system paper describing a final model and its evaluation, not an ablation study analyzing the contribution of each component. This is a legitimate genre of paper, but it means the claims about why the model works are hypotheses rather than demonstrated facts.

Mitigation status. Not addressed. The paper does not acknowledge the absence of training-stage ablations as a limitation. A reader unfamiliar with the norms of industrial system papers might reasonably expect that claims about the relative effectiveness of training stages would be supported by experiments varying those stages. The paper's claims about RLHF and co-training should be understood as the authors' informed judgments based on internal development experience, not as experimentally validated findings.

Limitation 5: Single Model Family, Single Training Paradigm—No Evidence of Transferability

The assumption or constraint. All results are produced using Gemini 1.5 Pro as the base model, trained within Google's post-training infrastructure using Google's data collection pipelines. The paper implicitly assumes—but does not test—that pedagogical instruction following with co-training would produce similar improvements if applied to a different model family (e.g., Llama, Qwen, Mistral) or a different training paradigm (e.g., a model trained primarily through SFT without RLHF infrastructure).

The consequence. This limitation affects the paper's contribution to the broader research community. The paper presents pedagogical instruction following as a general approach—a reframing of how to build AI tutors that "avoids committing our models to any particular definition of pedagogy" (Section 1). But if the approach only works when applied to a specific model family with a specific post-training pipeline, its generalizability is unproven. A team working with a different base model cannot assume that mixing pedagogical instruction-conditioned data into their training mixture will produce the preference gains reported here. The approach may depend on properties of Gemini 1.5 Pro that are not explicitly identified—its base instruction-following capability, its multi-turn coherence, the scale and composition of its standard post-training mixture, or the quality of Google's RLHF infrastructure.

This is not a criticism of the paper's results, which are valid for the model they evaluate. It is a limitation on the scope of the claims: the paper demonstrates that one specific model trained in one specific way produces tutoring behavior that experts prefer, not that pedagogical instruction following is a general recipe that transfers across model families. Section 2.1's discussion of instruction following benchmarks (IFEval, InfoBench) and the distinction between hard and soft constraints provides a conceptual framing for why pedagogical IF is challenging, but the empirical evidence that the approach solves this challenge comes from a single model.

What evidence exists in the paper. LearnLM is built on exactly one base model (Gemini 1.5 Pro-002). The comparison models (GPT-4o, Claude 3.5 Sonnet) are evaluated as contemporaneous alternatives, not as base models to which the same training recipe could be applied. The paper does not report any experiments applying the pedagogical instruction-following training recipe to a different model architecture or a different organization's base model. The medical education feasibility study (Appendix C) tests the evaluation methodology on a new domain, not the training methodology on a new base model.

Mitigation status. Partially acknowledged in the scope of the claims. The paper describes LearnLM as an experimental model and notes that a subset of its improvements have been integrated into Gemini 2.0 (Section 2.3), suggesting the approach generalizes within the Gemini model family across versions. But the paper does not discuss generalizability to non-Gemini models, nor does it identify which aspects of the training recipe might be Gemini-specific. The framing in Section 1—"we reframe the challenge of injecting pedagogical behavior as one of pedagogical instruction following"—presents this as a general approach, but the evidence base is restricted to a single implementation. Future work testing the approach on other model families would substantially strengthen the generality claim.

Limitation 6: The Evaluation Tests Pedagogy Primarily on Instructions Similar to Those Used in Training

The assumption or constraint. The paper's central claim is that LearnLM has learned pedagogical instruction following—the general capability to read a System Instruction describing desired teaching behavior and produce behavior that adheres to it, even for instructions not seen during training. However, the evaluation tests this capability only on System Instructions that were written by the same team using the same scenario design protocol that generated the training data. The paper does not test whether LearnLM follows pedagogical instructions written in substantially different styles, by independent teachers using different pedagogical vocabularies, or in languages other than English.

The consequence. A teacher or developer who writes System Instructions in a style that differs meaningfully from those in LearnLM's training distribution cannot be confident that the model will follow them as reliably as the evaluation suggests. If the training instructions all follow a similar template—with similar length, similar vocabulary for describing pedagogical behaviors, similar formatting conventions—the model may have learned to recognize and follow instructions in that specific style rather than developing a general capability to interpret arbitrary pedagogical specifications.

This is a specific instance of a broader evaluation concern: the 49 scenarios, while diverse in subject, learner persona, and learning goal, were all designed by the same team through the same systematic process (Section 3.1, Phases 1–3). The System Instructions in these scenarios exhibit structural similarities—they tend to be multi-paragraph, to specify both constraints ("do not give away answers") and methods ("use Socratic questioning"), and to use vocabulary drawn from the same pedagogical framework that underlies the evaluation rubric (active learning, scaffolding, cognitive load). A high school teacher writing a brief, informal instruction like "help my students with their algebra homework but don't just do it for them lol" may encounter different behavior than the evaluation suggests.

What evidence exists in the paper. The evaluation uses exactly the System Instructions specified in the 49 scenarios (examples in Appendix B.3). These instructions were written by the same team that designed the scenarios, using the protocol in Appendix B.2, as part of the same process that informed the training data design. The paper does not report any evaluation where System Instructions were written by independent teachers, drawn from real classroom deployments, or constructed to be stylistically dissimilar from the training distribution. The claim that LearnLM can follow "a range of pedagogical instructions" (Section 5) is supported for the range of instructions tested, but the tested range may be narrower than the claim implies.

Mitigation status. Not addressed. The paper does not discuss the distribution match between training and evaluation instructions as a potential limitation. The instruction-following framing implies generalization—the model should follow any pedagogical instruction, not just those resembling training—but this is an untested implication. Future work could evaluate LearnLM with System Instructions collected from independent teachers or generated to be maximally dissimilar from the training distribution, providing evidence about whether the capability is genuinely general or distribution-bound.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper makes a conceptual reframing that shifts how the field should think about building AI tutoring systems, but it does not introduce a new modeling paradigm or architectural innovation. The magnitude is best understood as a principled reorientation of an existing capability (instruction following) toward a domain that had been approached through ad-hoc persona engineering, rather than as a fundamental breakthrough in model training.

The core shift: from baking pedagogy into the model to making it specifiable through instructions. Prior work on AI tutors—including the authors' own initial LearnLM tech report (Jurenka et al., 2024)—treated pedagogical quality as a target to be encoded in training data. Curate demonstrations of good tutoring, fine-tune the model, and it becomes a good tutor. This paper argues, based on extensive engagement with the education ecosystem (Section 1, findings 1–3), that this approach cannot serve the diversity of real educational contexts. A Socratic tutor for philosophy, a scaffolded problem-solving coach for algebra, and an encouraging reading guide for first-graders exhibit behaviors that are not just different but actively contradictory—giving away answers is good assistance but bad tutoring; withholding answers is good tutoring but frustrating assistance. A model that internalizes one pedagogical style will be wrong for most use cases.

The solution—pedagogical instruction following—reframes the problem as one of conditional behavior: the model learns to read a System Instruction describing desired teaching behavior and produce behavior that adheres to it. This means pedagogy is not inside the model but specified to the model at deployment time by the teacher, school, or product developer who understands the context. The model's job is instruction following, not pedagogical philosophy. This shift is conceptually similar to the broader transition in NLP from task-specific fine-tuning to general instruction following (Ouyang et al., 2022), but applied to a domain where instructions are unusually complex, nuanced, multi-turn, and difficult to verify automatically—the paper explicitly distinguishes pedagogical IF from the simple, often programmatically verifiable constraints in benchmarks like IFEval (Section 2.1).

Reconciling tensions in the education-AI landscape. The paper resolves a structural tension that has quietly shaped the field: the conflict between the universality that model developers want (one model that works for everyone) and the specificity that educators need (a tutor adapted to their students, subject, and philosophy). Prior approaches forced a choice—either build a generic tutor with mediocre fit to any context, or build many specialized tutors at unsustainable maintenance cost. Pedagogical instruction following offers a third path: one model that can be many different tutors, with the difference specified through natural language instructions rather than separate training runs. The paper reports that post-report engagement with the education sector surfaced exactly this tension: "post-hoc fine-tuning for each application can be effective in the short-term, but is impractical because of cost, maintenance, and rapidly improving base models. Thus, despite its shortcomings, prompting will likely remain the best way for education product developers to specify behavior" (Section 1). The reframing directly addresses this practical reality.

Which research directions become more attractive, and which become less so. This work makes instruction-following robustness for complex, multi-turn, soft-constraint instructions a central research priority for educational AI. Prior to this paper, instruction following research focused primarily on simpler constraints (formatting, length, keyword inclusion/exclusion) where automated verification is straightforward. The paper demonstrates that pedagogical quality—which depends on nuanced, context-sensitive, multi-turn adherence to instructions that describe how to interact rather than what to output—is a substantially harder problem, and that RLHF with instruction-conditioned preference data is the primary driver of capability here (Section 2.2). This suggests that investment in better preference data collection, more robust reward modeling against distribution shift in long conversations, and verification methods for soft pedagogical constraints will have higher marginal returns than investment in better SFT demonstration writing.

Conversely, this work makes post-hoc fine-tuning for domain-specific behavior in education less attractive as a research direction. If co-training with instruction conditioning preserves general capabilities while enabling flexible pedagogical specification (as the paper argues, though doesn't prove with general benchmarks), then the path of taking a general model and fine-tuning it into a specific tutor is an inferior strategy—it locks the model into one pedagogical style, requires separate maintenance pipelines for each context, and risks capability degradation. The paper doesn't experimentally demonstrate that co-training beats post-hoc fine-tuning (an important gap), but the conceptual argument is strong enough to shift the burden of proof onto advocates of the fine-tuning approach.

The evaluation methodology as a contribution to the practice of AI-in-education research. The three-stage scenario-based evaluation pipeline (Section 3) is positioned as foundational infrastructure for the field, not just a measurement tool for this paper. The paper explicitly states an ambition to "work toward more consensus on a universal framework for pedagogical assessment of AI systems" (Section 5). If adopted, this methodology would address a chronic weakness in educational AI research: the difficulty of comparing tutoring systems across different studies that use different tasks, different student populations, and different quality metrics. The medical education feasibility study (Appendix C) demonstrates that the methodology transfers across domains, which is a first step toward establishing it as a general framework. However, the massive resource requirements—49 scenarios, 186 conversation collectors, 248 assessors, 10,192 assessments—mean that community-wide adoption would require either substantial streamlining or institutional infrastructure that doesn't currently exist. The paper does not discuss this tension.

What the paper does NOT change. It is important to be precise about what this work does not shift. It does not demonstrate that better expert-judged pedagogical quality translates to better learning outcomes—the evaluation is entirely intrinsic (Section 5 acknowledges this as a limitation requiring future work). It does not demonstrate that the pedagogical instruction following capability generalizes to instructions written in substantially different styles from those in the training and evaluation distribution. It does not provide evidence that co-training preserves general capabilities (no general benchmarks are reported). And it does not establish that the approach works for model families other than Gemini. These are not fatal flaws—the paper is transparent about several of these limitations—but they bound the scope of the landscape change. The reframing is persuasive and the evaluation is rigorous within its scope, but the practical impact depends on evidence that is not yet in hand.

Follow-Up Research This Work Enables

Extrinsic learning outcome evaluation with LearnLM-style models. The most important unanswered question is whether the expert-judged pedagogical quality improvements reported in Figures 4 and 5 translate to measurable improvements in what students actually learn, retain, and transfer. The paper acknowledges this gap explicitly (Section 5) but the framing makes it newly tractable: now that a model with demonstrated pedagogical quality advantages exists, a randomized controlled trial comparing LearnLM against a baseline model on a standardized learning task (e.g., a pre-test/post-test design where students learn a specific topic through multi-turn tutoring with each model) would directly test the link between intrinsic pedagogy ratings and extrinsic learning gains. The paper cites existing work that has used such designs (Kestin et al., 2024; Wang et al., 2024; Bastani et al., 2024), and the 49-scenario bank provides a ready-made source of learning contexts to test. A strong study would measure not just immediate post-test performance but delayed retention (e.g., one week later) and transfer to novel problems, and would include both expert pedagogical quality ratings and learning outcome measures on the same interactions to quantify their correlation. A negative result—expert-preferred pedagogy not producing better learning—would substantially reframe the value proposition of pedagogical instruction following.

Training-stage ablation to quantify RLHF vs. SFT contribution. The paper claims that "RL is significantly more effective" than SFT for pedagogical instruction following (Section 2.2) but provides no experimental evidence. A controlled ablation training three variants—SFT-only, RLHF-only (starting from the base model with no pedagogical SFT), and SFT+RLHF—and evaluating all three in the same scenario-based expert evaluation pipeline would directly quantify the marginal contribution of each stage. This is important because it informs resource allocation: if RLHF provides most of the gain, teams should invest heavily in preference data collection and reward model training; if SFT provides comparable gains at lower cost, teams with limited RLHF infrastructure can still make progress. The evaluation methodology developed in this paper makes this ablation tractable—the scenario bank exists, the rubric is validated, and the expert assessor pipeline is documented. A nuanced finding would be quantifying whether RLHF's advantage over SFT is larger for soft pedagogical constraints (tone, adaptivity, curiosity stimulation) than for hard constraints (staying on topic, not giving away answers), which would align with the paper's argument that "preference judgements often contain subtle distinctions" that SFT cannot easily capture.

General capability benchmarking to test the co-training preservation claim. The paper asserts that co-training avoids "forgetting" general capabilities (Section 2.3) but reports no general benchmarks. A straightforward follow-up would evaluate LearnLM and Gemini 1.5 Pro on standard capability benchmarks—MMLU, HELM, HumanEval, GSM8K, factuality benchmarks, and safety refusal evaluations—and test whether the pedagogical training introduced any statistically reliable degradation. This is a low-risk, high-value study that would directly inform whether developers can adopt LearnLM without trading off general performance. If degradation exists, quantifying its magnitude per domain (e.g., does pedagogical training primarily affect factual recall? reasoning? coding?) would help developers assess whether the tradeoff is acceptable for their use case. A finding of zero degradation across all benchmarks would validate the co-training design choice and strengthen the paper's argument that instruction conditioning resolves the assistant-tutor behavior conflict. A finding of non-zero degradation—even if small—would motivate research into better co-training mixture ratios or alternative architectures (e.g., adapter-based approaches that more strongly separate pedagogical and general capabilities).

Out-of-distribution instruction following: testing generalization to teacher-written System Instructions. The evaluation tests LearnLM on System Instructions written by the same team using the same protocol that generated training data. A critical generalization test would collect System Instructions from independent teachers—e.g., 20–30 high school and college instructors asked to write instructions for an AI tutor in their subject, in whatever style and vocabulary they naturally use—and evaluate LearnLM against comparison models on these instructions using the same scenario-based pipeline (with scenarios adapted to match the teachers' subjects). This would test whether pedagogical instruction following is a general capability or is overfit to the specific instruction style in the training distribution. Key measurements: (a) does LearnLM's preference advantage hold on teacher-written instructions, or does it shrink/disappear? (b) do teachers' instructions systematically differ from the scenario bank instructions in length, specificity, or pedagogical vocabulary? (c) can the model follow instructions that use different terminology for the same pedagogical concepts (e.g., "don't just give them the fish" vs. "guide to discovery")? A negative result—LearnLM's advantage disappearing on out-of-distribution instructions—would suggest that instruction-diversity in training data needs to be substantially broader than the current scenario bank provides, and would motivate research into automated instruction paraphrasing or augmentation.

Adversarial stress-testing of instruction adherence. The paper identifies a key stakeholder need: confidence that the tutor will follow System Instructions "even if a student tries to circumvent them (e.g., 'do not give away the answer' or 'stay on topic')" (Section 1). The evaluation scenarios include learner personas that are "easily distracted" or "adversarial," but the paper does not report targeted testing of how robustly models adhere to instructions under systematic student pressure. A follow-up study could design a battery of adversarial student strategies—repeated rephrasing of answer-seeking questions, feigning confusion to elicit more direct help, social pressure ("please just tell me, I'm going to fail"), topic-drift attempts, and authority challenges—and measure how often each model violates key instruction constraints (giving away answers, going off-topic, abandoning the specified pedagogical method) across these strategies. The evaluation methodology's scenario framework is well-suited to this: adversarial student behavior can be specified in the learner persona and conversation plan, and role-playing experts can be explicitly trained on executing the adversarial strategies consistently. This would produce a "pedagogical instruction robustness" metric that is directly relevant to the developer use case the paper identifies as the most commonly cited need (Section 1, finding 2).

Automated pedagogical evaluation to reduce expert assessment cost. The evaluation methodology's primary barrier to community adoption is its resource intensity—248 expert assessors producing 10,192 assessments is not replicable by most research groups. A natural follow-up would train an LLM-based evaluator (e.g., using a strong general model prompted with the same pedagogical rubric from Table 8) to assess conversation transcripts, and measure its agreement with the expert judgments collected in this paper. If an LLM judge can achieve high inter-rater reliability with the expert panel (e.g., Cohen's κ > 0.7 on the comparative preference judgments), it would make the evaluation methodology dramatically more accessible—scenario-based conversation collection would still require role-playing participants, but pedagogical assessment could be automated, enabling rapid iteration during model development. The paper's existing expert judgment dataset (10,192 assessments) provides the gold-standard labels needed to train and validate such an automated evaluator. A negative result—LLM judges failing to capture the nuanced distinctions that experts make, particularly on soft dimensions like "stimulates curiosity" or "adapts to affect"—would itself be an important finding about the limits of current models for evaluating complex social and pedagogical interactions, and would motivate research into what training or prompting is needed to close the gap.

Practical Applications and Downstream Use Cases

EdTech platform integration via configurable System Instructions for different subjects and age groups. An EdTech company building a tutoring platform that serves students from elementary school through college across multiple subjects currently faces an impossible choice: deploy a single generic tutor that fits no context well, or maintain separate fine-tuned models for each grade-subject combination at unsustainable cost and engineering complexity. LearnLM's pedagogical instruction following directly addresses this: the platform provides different System Instructions per context—one set for elementary math (emphasizing encouragement, concrete examples, and frequent checks for understanding), another for high school history (emphasizing Socratic questioning, primary source analysis, and argument evaluation)—and deploys a single model that adapts its behavior to each instruction. The paper's demonstration that LearnLM received positive expert ratings across all five pedagogy rubric dimensions (Figure 5) and across diverse scenarios spanning multiple subjects and learner personas provides initial evidence that a single model can serve heterogeneous educational contexts. The +31% preference over GPT-4o (Figure 4) suggests that the pedagogical quality difference is substantial enough to matter in practice, though the point-in-time nature of the comparison means this should be validated against current model versions.

Teacher-authored AI tutor configurations for classroom use. A teacher preparing a unit on quadratic equations wants an AI tutor that will help students practice without giving away solutions, that will ask guiding questions rather than providing explanations unprompted, and that will flag to the teacher when a student is consistently stuck on a particular concept type. With a model that exhibits pedagogical instruction following, the teacher writes a System Instruction describing these behaviors in their own words, deploys the tutor to their class, and can update the instruction over time as they observe student interactions. The paper's finding that LearnLM was specifically preferred for "keeps_on_topic" (21.2% of LearnLM-preference explanations vs. 10% for other models, Table 1) and "challenges_learner" (33.0% vs. 16.3%) is directly relevant: these are the exact behaviors a teacher wants when specifying that the tutor should guide rather than answer. The paper's transparency about LearnLM's weaknesses—occasional lack of clarity (16.0% vs. 20.0% in preference explanations) and style issues (31.9% vs. 36.3%)—also informs the teacher that they may need to specify clarity and tone expectations explicitly in their instructions rather than relying on the model's defaults.

Medical education deployment for clinical reasoning practice. The medical education feasibility study (Appendix C) demonstrates that the pedagogical instruction following approach and the scenario-based evaluation methodology transfer to professional education domains. A medical school could deploy LearnLM with System Instructions tailored to different phases of training: preclinical students need a tutor that emphasizes foundational knowledge with case-based application, while clinical students need a tutor that simulates diagnostic reasoning with progressively revealed patient information and expects the student to drive the diagnostic process. The study found that physician educators preferred LearnLM over Gemini 1.5 Pro on all five comparison criteria (Figure 15b), with the strongest preferences for "better pedagogy" (+6.1%) and "more like a very good human tutor" (+6.8%). While the effect sizes are smaller than in the main evaluation (+6–10% vs. +11–31%), they suggest meaningful pedagogical improvement in a domain where tutoring quality has direct implications for patient care. The paper appropriately notes that "real-world applications of these models require separate evaluations of medical accuracy, bias, and harm from the perspective of clinical experts"—pedagogical quality is necessary but not sufficient for medical deployment.

Self-improving AI tutoring systems through iterative instruction refinement. A product team deploying an AI tutor can use the evaluation methodology's rubric (Table 8) to systematically diagnose weaknesses in their current System Instructions. If expert assessors rate the tutor low on "Inspires Active Learning" or "Deepens Metacognition," the team revises the System Instructions to more explicitly specify the desired behaviors (e.g., adding "after the student attempts a problem, ask them to explain their reasoning before providing feedback" to address metacognition) and re-evaluates. The paper's detailed rubric provides a diagnostic language for this iteration—rather than vaguely wanting "better tutoring," teams can target specific, measurable pedagogical dimensions. The qualitative finding that LearnLM was occasionally seen as "too reticent to help by giving answers when it was clear the student needed it" (Table 1) illustrates the kind of instruction refinement this enables: a developer observing this pattern could add a clause to their System Instruction like "if the student has made three unsuccessful attempts at a problem, provide more direct guidance while still encouraging them to complete the final steps themselves."

When to Prefer This Method

The paper does not articulate an explicit tradeoff matrix between named alternative approaches. It positions pedagogical instruction following against two alternatives—fixed-persona pedagogical fine-tuning (as in the prior LearnLM tech report) and post-hoc fine-tuning for each deployment context—but the comparison is between the paper's approach and the authors' own prior work, not between competing contemporaneous methods from different research groups. The paper argues that pedagogical instruction following with co-training is superior to these alternatives on conceptual grounds (flexibility, sustainability, capability preservation) but does not provide experimental comparisons against them (no SFT-only LearnLM, no post-hoc fine-tuned LearnLM). For deployment decisions between LearnLM and other flagship models (GPT-4o, Claude 3.5 Sonnet), the paper provides comparative preference data (Figures 4, 5) but these are point-in-time evaluations of specific model versions, not a general claim about one modeling approach over another. The closest the paper comes to a decision rule is the implicit guidance that if your use case requires flexible, context-specific pedagogical behavior specified via System Instructions, and you have access to a model trained with pedagogical instruction-following data in its post-training mixture, that model will likely produce tutoring interactions that pedagogy experts prefer over general-purpose flagship models. But this is a deployment recommendation, not a methodological tradeoff between competing technical approaches.